Hey everyone, just sharing a project I’ve been working on called DocMQ. It’s a visibility based queueing system optimized for run times set in the future as opposed to a traditional FIFO. Before going too much further, I wanted to explain the pitch/anti-pitch.
Why wouldn’t you use this?
The first, and most important reason you wouldn’t use DocMQ is that you have a need for high-performance FIFO queues. While DocMQ supports the queue style, you’ll consistently see better performance on a tool such as Kafka or Rabbit or even a Redis based solution.The second reason you may not need DocMQ is that you are not requiring at-least-once delivery. Several MQ systems, especially in node, do not require an
ackoperation in order to confirm the job’s completion. For example, sending a push notification does not require at-least-once guarantees, as the missed notification does not have an outsized negative impact on the product.
Why would you use this?
Queryable future jobs. Because DocMQ is built around document based databases including Mongo, it’s trivial to execute a query and get a snapshot of upcoming and in-process jobs and their payload. Mongo Compass is already an excellent explorer that can tell you everything about your queue’s health using simple aggregation pipelines!At least once delivery. DocMQ operates on visibility windows. Similar to SQS/ASQ/CPS, DocMQ relies on a timestamp of when a job is eligible to be attempted. When jobs are claimed, this window is set N seconds in the future. If a job completes the
completedflag is set, but if a job never completes, it becomes automatically eligible to run again when the timestamp N seconds in the future is reached. Coupled with Mongo’s transaction, once a job is in DocMQ it will run until its exhausted all attempts. Recovering a failed job is a simple$match+$setoperation.
Internally, we’re using DocMQ at work and a single worker is doing about 500 ops/s on a single core, limited mainly by our concurrency settings since other things also run on the machine. I’d love feedback about the tool or even key features you think it should have. The goal is to have a solid core that’s easy to extend and build on without requiring an entire ecosystem of custom tooling.