Ocypod is an open source job queue server implemented using Rust/Actix/Redis. It’s been used production for a few months now, handling millions of jobs a week, with a few hundred workers.
It was mostly written to deal with gaps found in existing queuing systems for very long running jobs, without requiring complex setup/configuration or client protocols (it’s not really aimed at the super high throughput use case, plenty of systems handle that very well already).
- simple setup - only requirement is Redis
- language agnostic - uses HTTP/JSON protocol, clients/workers can be implemented in any language
- long running jobs - handle jobs that may be running for hours/days, detect failure early using heartbeats
- simple HTTP interface - no complex binary protocols or client/worker logic
- flexible job metadata - allows for different patterns of use (e.g. progress tracking, partial results, etc.)
- job inspection - check the status of any jobs submitted to the system
- tagging - custom tags allow easy grouping and searching of related jobs
- automatic retries - re-queue jobs on failure or timeout