Context
The platform runs a steady stream of background jobs: enrichment, re-pricing, imports, notifications. The original scheduler worked, but it was single-instance by assumption and spent too much time contending for threads under load.
Constraints
- Several scheduler instances must be able to run at once without executing the same job twice.
- Jobs are stored in MongoDB; the scheduler cannot assume anything about the database beyond the official driver.
- Throughput had to improve without starving the rest of the service.
Approach
- Rebuilt the execution loop on the Tokio runtime, with each job kind owning a worker pool bounded by an
Arc<Semaphore>. Backpressure is a permit count, not a thread count. - Made claiming a job an atomic database operation, so two instances racing for the same job can only ever have one winner.
- Separated scheduling (when a job becomes due) from execution (who runs it) so each could be reasoned about and tested alone.
Outcome
The scheduler now runs across multiple instances, thread contention is gone, and the worker-pool pattern became the template for later async work in the codebase.