Building Real Time Systems on MongoDB Using the Oplog at Stripe

Building Real Time Systems on MongoDB Using the Oplog at Stripe

Evan Broder

June 24, 2014

Technical topics covered: Replication, Querying, Integration | Languages used: Ruby

MongoDB's oplog is possibly its most underrated feature. The oplog is vital as the basis on which replication is built, but its value doesn't stop there. Unlike the MySQL binlog, which is poorly documented and not directly exposed to MySQL clients, the oplog is a well-documented, structured format for changes that is query-able through the same mechanisms as your data. This allows many types of powerful, application-driven streaming or transformation. At Stripe, we've used the MongoDB oplog to create PostgresSQL, HBase, and ElasticSearch mirrors of our data. We've built a simple real-time trigger mechanism for detecting new data. And we've even used it to recover data. In this talk, we'll show you how we use the MongoDB oplog, and how you can build powerful reactive streaming data applications on top of it.

MongoDB World for Giant Ideas
June 20-21 Chicago, IL

MongoDB World is where the world’s fastest growing database community comes to connect, explore, and learn. Join us for hands-on and deep-dive technical sessions, giving you the tools you need to build and deploy your giant ideas.