Chapman: Building a High-Performance Distributed Task Service with MongoDB

June 02, 2015

Facebook ShareLinkedin ShareReddit ShareTwitter ShareGoogle+ Share


Rick Copeland, Co-founder & CTO at

When you're building a web application, you want to respond to every request as quickly as possible. The usual approach is to use an asynchronous job queue like Sidekiq, Resque, Celery, RQ, or a number of other frameworks to handle those tasks outside the request/response cycle in a separate 'worker' process. Unfortunately, many of these frameworks either require the deployment of Redis, RabbitMQ, or some other request broker, or they resort to polling a database for new work to do. Chapman is a distributed task queue built on MongoDB that avoids gratuitous polling, using tailable cursors with the oplog to provide notifications of incoming work. Inspired by Celery, Chapman also supports task graphs, where multiple tasks that depend on each other can be executed by the system asynchronously. Come learn how is using MongoDB and Chapman to handle its core data processing needs.

Try MongoDB Atlas, our fully-managed
database as a service.

Available on AWS, Azure, and GCP.