Ocypod: language-agnostic, Redis-backed job queue server implemented using Rust/Actix

#1

Ocypod is an open source job queue server implemented using Rust/Actix/Redis. It’s been used production for a few months now, handling millions of jobs a week, with a few hundred workers.

It was mostly written to deal with gaps found in existing queuing systems for very long running jobs, without requiring complex setup/configuration or client protocols (it’s not really aimed at the super high throughput use case, plenty of systems handle that very well already).

Some features:

  • simple setup - only requirement is Redis
  • language agnostic - uses HTTP/JSON protocol, clients/workers can be implemented in any language
  • long running jobs - handle jobs that may be running for hours/days, detect failure early using heartbeats
  • simple HTTP interface - no complex binary protocols or client/worker logic
  • flexible job metadata - allows for different patterns of use (e.g. progress tracking, partial results, etc.)
  • job inspection - check the status of any jobs submitted to the system
  • tagging - custom tags allow easy grouping and searching of related jobs
  • automatic retries - re-queue jobs on failure or timeout
5 Likes