MongoDB live at Craigslist

MongoDB
May 16, 2011
#Releases

Update: You can view a video of Jeremy Zawodny’s talk at MongoSF on 10gen.com.

MongoDB is now live at Craigslist, where it is being used to archive billions of records.

Craiglist has kept every post anyone has ever made in a large MySQL cluster. A few months ago, they began looking for alternatives: schema changes were taking forever (Craigslist’s schema has changed a couple times since 1995) and it wasn’t really relational information. They wanted to be able to add new machines without downtime (which sharding provides) and route around dead machines without clients failing (which replica sets provide), so MongoDB was a very strong candidate. After looking into a few of the most popular non-relational database systems, they decided to go with MongoDB.

Jeremy Zawodny is a software engineer at Craigslist and an author of High Performance MySQL (O'Reilly). He kindly agreed to answer some questions about their MongoDB cluster (editor’s comments in italics).

Any numbers you can give us?

We’re sizing the install for around 5 billion documents. That’s from the initial 2 billion document import we need to do plus room to grow for a few years to come. Average document size is right around 2KB. (Five billion 2KB documents is 10TB of data.) We’re getting our feet wet with MongoDB so this particular task isn’t high throughput or growing in unpredictable ways.

We can put data into MongoDB faster than we can get it out of MySQL during the migration.

What does your cluster topology look like?

We have several three-machine replica sets, each set serving a shard of our “archive” database cluster. The configuration is three replica sets in each colo (two total) to handle our initial build out. Obviously there will be a set of config servers and routing processes in the mix as well.

Craigslist is using the MongoDB Perl driver.

Did you find any stumbling blocks relational database developers should watch out for?

Oh, yeah. :-) I’m planning to write up a blog post on that and I talked about it a bit at MongoSV (watch the video), too. The short version is that you have to think differently about indexing and do a bit more bookkeeping of your own. But on the plus side, you don’t have to pay the join penalty, so you can get your data back a lot faster.

Character set issues come up as well, since we’re a Latin-1 or Windows-1252 shop currently (but really need to go UTF-8 across the board). That means some upfront work, but it’s good that MongoDB is UTF-8 end-to-end already.

Got future plans for your MongoDB cluster?

Too soon to tell! But I have a few ideas about ways we can use MongoDB to supplement other needs and possibly replace other data stores. But I really need to think more about them before I go spouting off.

Looking into the future, Zawodny hopes for more polish on sharding and replica sets and that some good front-end admin tools will be developed. “But that will come in time, I’m sure.”

If you’re interested in more details, check out the video’s from Jeremy’s talks at MongoSV or MongoSF.

← Previous

MongoDB Powering MTV's Web Properties

An interview with Jeff Yemin, Director of Content Management Systems at MTV Networks and a presenter at the upcoming MongoNYC conference. What do you work on at MTV Networks? I manage the backend development of our next-generation content management system. Tell us how MTV Networks is using MongoDB. What is the size and scope of the CMS application that you have built? MongoDB is the repository that powers our CMS, which will eventually be used to manage and serve content for all of MTV Networks’ major websites. Can you tell us why you chose MongoDB? I have to save something for my actual talk ! But briefly, MongoDB is a good fit for a lot of reasons. The schema-less document structure provides a lot of flexibility, while the search capabilities provide a lot more value than you get with key-value stores. At the same time, from an operational perspective, it feels very much like MySQL, so our systems and DBA groups are quite comfortable managing the deployment. What were some of the challenges you faced using MongoDB for a large enterprise application? The biggest was probably convincing people that it was a good idea. Like most enterprises that have been around for a while, we have a long history of using relational databases, so this is a pretty big change for us. What’s next for MongoDB at MTV? Right now MongoDB is powering the recently re-launched spike.com , and we are going to be rolling it out on many other major sites within the next year, most likely including gametrailers.com , thedailyshow.com , comedycentral.com , nick.com , and numerous international properties. We’ll also be rolling out new MongoDB-based applications for social activity tracking and online polls. Learn more at Jeff’s talk at MongoNYC on June 7. Register before May 17 to take advantage of early bird pricing, which is only $50.

May 10, 2011
Next →

Accelerating to T+1 - Have You Got the Speed and Agility Required to Meet the Deadline?

On May 28, 2024, the Securities and Exchange Commission (SEC) will implement a move to a T+1 settlement for standard securities trades , shortening the settlement period from 2 business days after the trade date to one business day. The change aims to address market volatility and reduce credit and settlement risk. The shortened T+1 settlement cycle can potentially decrease market risks, but most firms' current back-office operations cannot handle this change. This is due to several challenges with existing systems, including: Manual processes will be under pressure due to the shortened settlement cycle Batch data processing will not be feasible To prepare for T+1, firms should take urgent action to address these challenges: Automate manual processes to streamline them and improve operational efficiency Event-based real-time processing should replace batch processing for faster settlement In this blog, we will explore how MongoDB can be leveraged to accelerate manual process automation and replace batch processes to enable faster settlement. What is a T+1 and T+2 settlement? T+1 settlement refers to the practice of settling transactions executed before 4:30pm on the following trading day. For example, if a transaction is executed on Monday before 4:30 pm, the settlement will occur on Tuesday. This settlement process involves the transfer of securities and/or funds from the seller's account to the buyer's account. This contrasts with the T+2 settlement, where trades are settled two trading days after the trade date. According to SEC Chair Gary Gensler , “T+1 is designed to benefit investors and reduce the credit, market, and liquidity risks in securities transactions faced by market participants.” Overcoming T+1 transition challenges with MongoDB: Two unique solutions 1. The multi-cloud developer data platform accelerates manual process automation Legacy settlement systems may involve manual intervention for various tasks, including manual matching of trades, manual input of settlement instructions, allocation emails to brokers, reconciliation of trade and settlement details, and manual processing of paper-based documents. These manual processes can be time-consuming and prone to errors. MongoDB (Figure 1 below) can help accelerate developer productivity in several ways: Easy to use: MongoDB is designed to be easy to use, which can reduce the learning curve for developers who are new to the database. Flexible data model: Allows developers to store data in a way that makes sense for their application. This can help accelerate development by reducing the need for complex data transformations or ORM mapping. Scalability: MongoDB is highly scalable , which means it can handle large volumes of trade data and support high levels of concurrency. Rich query language: Allows developers to perform complex queries without writing much code. MongoDB's Apache Lucene-based search can also help screen large volumes of data against sanctions and watch lists in real-time. Figure 1: MongoDB's developer data platform Discover the developer productivity calculator . Developers spend 42% of their work week on maintenance and technical debt. How much does this cost your organization? Calculate how much you can save by working with MongoDB. 2. An operational trade store to replace slow batch processing Back-office technology teams face numerous challenges when consolidating transaction data due to the complexity of legacy batch ETL and integration jobs. Legacy databases have long been the industry standard but are not optimal for post-trade management due to limitations such as rigid schema, difficulty in horizontal scaling, and slow performance. For T+1 settlement, it is crucial to have real-time availability of consolidated positions across assets, geographies, and business lines. It is important to note that the end of the batch cycle will not meet this requirement. As a solution, MongoDB customers use an operational trade data store (ODS) to overcome these challenges for real-time data sharing. By using an ODS, financial firms can improve their operational efficiency by consolidating transaction data in real-time. This allows them to streamline their back-office operations, reduce the complexity of ETL and integration processes, and avoid the limitations of relational databases. As a result, firms can make faster, more informed decisions and gain a competitive edge in the market. Using MongoDB (Figure 2 below), trade desk data is copied into an ODS in real-time through change data capture (CDC), creating a centralized trade store that acts as a live source for downstream trade settlement and compliance systems. This enables faster settlement times, improves data quality and accuracy, and supports full transactionality. As the ODS evolves, it becomes a "system of record/golden source" for many back office and middle office applications, and powers AI/ML-based real-time fraud prevention applications and settlement risk failure systems. Figure 2: Centralized Trade Data Store (ODS) Managing trade settlement risk failure is critical in driving efficiency across the entire securities market ecosystem. Luckily, MongoDB integration capabilities (Figure 3 below) with modern AI and ML platforms enable banks to develop AI/ML models that make managing potential trade settlement fails much more efficient from a cost, time, and quality perspective. Additionally, predictive analytics allow firms to project availability and demand and optimize inventories for lending and borrowing. Figure 3: Event-driven application for real time monitoring Summary Financial institutions face significant challenges in reducing settlement duration from two business days (T+2) to one (T+1), particularly when it comes to addressing the existing back-office issues. However, it's crucial for them to achieve this goal within a year as required by the SEC. This blog highlights how MongoDB's developer data platform can help financial institutions automate manual processes and adopt a best practice approach to replace batch processes with a real-time data store repository (ODS). With the help of MongoDB's developer data platform and best practices, financial institutions can achieve operational excellence and meet the SEC's T+1 settlement deadline on May 28, 2024. In the event of T+0 settlement cycles becoming a reality, institutions with the most flexible data platform will be better equipped to adjust. Top banks in the industry are already adopting MongoDB's developer data platform to modernize their infrastructure, leading to reduced time-to-market, lower total cost of ownership, and improved developer productivity. Looking to learn more about how you can modernize or what MongoDB can do for you? Zero downtime migrations using MongoDB’s flexible schema Accelerate your digital transformation with these 5 Phases of Banking Modernization Reduce time-to-market for your customer lifecycle management applications MongoDB’s financial services hub

May 25, 2023