MongoDB 3.4.0-rc3 is out and is ready for testing. This is the culmination of the 3.3.x development series.
Fixed in this release candidate:
- SERVER Configurable connection pools size for mongos
- SERVER Limit total memory utilization for bulk index builds
- SERVER Additional tests for views on sharded collections
- SERVER Over 25% regression on mongodb using YCSB workload
- SERVER Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
- TOOLS A single invocation of mongoreplay replays the ops twice
- TOOLS Connections never closed during replay
As always, please let us know of any issues.
-- The MongoDB Team
Leaf in the Wild: How Loopd uses MongoDB to Power its Advanced Location-Tracking Platform for Conferences
Conferences can be incredibly hectic experiences for everyone involved. You have attendees wanting to meet and exchange information, sponsors and exhibitors looking to maximize foot traffic to their booths, and the conference hosts trying to get a sense of how they can optimize their event and if it was all worth it in the end. While sponsors usually do get a lead list immediately after an event for their troubles, attendees often struggle to remember who they actually spoke to and event hosts are often left in the dark about what they can do to maximize the returns on their investments. Enter Loopd, an advanced events engagement platform. I sat down with their CEO, Brian Friedman, to understand how they’re using MongoDB to help conference attendees and event hosts separate the signal from the noise. Tell us about Loopd. Loopd provides physical intelligence for corporate events. We help corporate marketers learn how people interact with each other, with their company, and with their company's products. The Loopd event engagement system is the industry's only bi-directional CRM solution that enables the exchange of content and contact information passively and automatically. We equip conference attendees with Loopd wearable badges, which can be used to easily exchange contact information or gain entry into sessions. Through our enterprise IoT analytics and sensors, we then gather and interpret rich data so that marketers have a more sophisticated understanding of business relationships and interactions at conferences, exhibits and product activation events. Some of our clients include Intel, Box, Twilio, and MongoDB. Bluetooth LE Loopd Badges How are you using MongoDB? We use MongoDB to store millions of datapoints from connected advertising and Bluetooth LE Loopd Badges on the conference floor. All of the attendee movement data captured by the Loopd Badge at an event can be thought of as time series data associated with location information. We track each Loopd Badge’s location and movement path in real time during the event. As a result, we handle heavy write operations during an event to make sure any and all calculations are consistent, timely, and accurate. We also use the database for real-time analysis. For example, we calculate the number of attendee visits & returns, and average time durations in near real time. We use the aggregation framework in MongoDB to make this happen. What did you use before MongoDB? Before MongoDB, we used PostgreSQL as our main data store. We used Redis as a temporary data buffer queue for storing new movement data. The data was dumped, inserted, and updated into rows in the SQL database once per second. The raw location data was read and parsed from the SQL database into a user-readable format. We needed a temporary buffer because the high volume of insert and update requests drained available resources. What challenges did you face with PostgreSQL? With PostgreSQL, we needed a separate Redis caching server to buffer write and update operations before storing them in the database, which added architectural and operational complexity. It also wasn’t easy to scale as it’s not designed to be deployed across multiple instances. How did MongoDB help you resolve those challenges? When we switched to MongoDB from PostgreSQL, our write throughput significantly increased, removing the need for a separate caching server in between the client and the database. We were able to halve our VM resource consumption (CPU power and memory), which translated to significant cost savings. As a bonus, our simplified underlying architecture is now much easier to manage. Finally, one of the great things about MongoDB is its data model flexibility, which allows us to rapidly adapt our schema to support new application demands, without the need to incur downtime or manage complex schema migrations. Please describe your MongoDB deployment. We typically run one replica set per event. The database size depends on the event — for MongoDB World 2016, we generated about 2 million documents over the course of a couple of days. We don’t shard our MongoDB deployments yet but having that ability in our back pocket will be very important for us going forward. At the moment, all of our read queries are executed on the secondaries in the replica set, which means write throughput isn’t impacted by read operations. The smallest analytics window in our application is a minute, which means we can tolerate any eventual consistency from secondary reads. Our MongoDB deployments are hosted in Google Cloud VM instances. We’re exploring using containers but they’re currently not in use for any production environments. We’re also evaluating Spark and Hadoop for doing some more interesting things with the data we have in MongoDB. What version of MongoDB are you running? We use MongoDB 3.2. We find the added document validation feature very valuable for checking data types. While we will still perform application-level error validation, we appreciate this added level of security. What advice do you have for other companies looking to start with MongoDB? MongoDB is flexible, scalable, and quite developer and DBA friendly, even if you’re used to RDBMS. We would recommend familiarizing yourself with the basic concepts of MongoDB first, heavily leaning on the community during learning. I’d also recommend reading the production notes to optimize system configuration operational parameters. Brian, thanks for taking the time to share your story with the MongoDB community. Thinking of migrating to MongoDB from a relational database? Learn more from our guide: Download RDBMS Migration Guide
Considering NoSQL? Let's Break Down Your Options
Non-relational alternatives to relational databases — usually referred to as NoSQL databases — have been rapidly gaining popularity over the past decade. In 2013, MongoDB published one of our most popular white papers, “Top 5 Considerations When Evaluating NoSQL Databases.” We have since updated that paper as the technology has evolved. MongoDB is now offering a major update, which adds two new issues organizations should include in their thinking: how a database handles data generated at the edge by mobile devices and how a database fits into a broader data platform that includes search and analytics. If you’re testing the waters of NoSQL databases, then you’re probably familiar with how they’re different from traditional relational databases. The list of things you already know about NoSQL probably looks something like this: They use a different data model and query language. They have dynamic schemas. They scale horizontally. Beyond those common features, there are significant differences among NoSQL databases. The seven areas of significant differences among your options are: Data model (document, graph, key-value, etc.) Query model Consistency and transactional model APIs Mobile data Data platform Commercial support, community strength, and lock-in From MongoDB’s point of view, the most important consideration is the data model. We popularized the document model , which supports a superset of all data models, making it useful for a wide variety of applications. Key features include the ability to index and query in any field, and the natural mapping of document data structures to objects in modern programming languages. Recent shifts in how modern applications are developed and deployed — and in the experiences they offer customers — highlight the two new considerations. Mobile use cases: Mobile applications introduce the added challenge of not always being connected to the network. Developers need a solution for keeping all their customers’ apps in sync with the back-end database, no matter where they are in the world and what kind of network connection they have. The solution also needs to scale easily and quickly as more users download an app, and support the cutting edge of mobile development technologies as they evolve. Data platform: MongoDB’s application data platform provides developers a unified interface to serve transactional and operational applications alongside search, real-time, and data lake application needs. It eliminates the overhead and friction of developers having to stitch together multiple discrete technologies into a complex architecture, each creating its own duplicated data silo — connected by fragile ETL pipelines — and accessed, secured, governed, and operationalized by different APIs and tools. For a deep dive into all the differences among NoSQL databases, download our white paper, “ Top 7 Considerations When Evaluating NoSQL Databases .”