MongoDB 3.0.5-rc2 is out and is ready for testing. This is a release candidate containing only fixes since 3.0.4. The next stable release 3.0.5 will be a recommended upgrade for all 3.0 users.
Fixed in this release:
- SERVER-17254 WT: drop collection while concurrent oplog tailing may greatly reduce throughput
- SERVER-17689 Server crash during initial replication sync
- SERVER-18875 Oplog performance on WT degrades over time after accumulation of deleted items
- SERVER-18902 Retrieval of large documents slower on WiredTiger than MMAPv1
- SERVER-18926 Full text search extremely slow and uses a lot of memory under WiredTiger
- SERVER-19255 Listener::waitUntilListening may return before listening has started
- SERVER-19489 Assertion failure and segfault in WorkingSet::free in 3.0.5-rc0
- SERVER-19513 Truncating a capped collection may not unindex deleted documents in WiredTiger
As always, please let us know of any issues.
– The MongoDB Team
MongoDB Pluggable Storage Engines: State of the Union, Storage Engine Summit
Back at the inaugural MongoDB World user conference in June 2014, our co-founder and CTO, Eliot Horowitz, announced that MongoDB would incorporate a storage engine API into the next major version of the database – MongoDB 3.0 . The goal of the storage engine API was to make it fast and easy for MongoDB and the community to build new pluggable storage engines that allow the database to be extended with new capabilities, and to be configured for specific hardware architectures. 12 months on, following both the MongoDB 3.0 release and the MongoDB World 2015 conference (keynotes and sessions from the event are now online ), we hosted our first Storage Engine Summit, bringing together storage engine developers from across both MongoDB and the community. The summit’s goals were to: Review the status of current MongoDB storage engines Provide developers with visibility into the roadmap for both the storage engine API itself, and the database roadmap Collect storage engine developer’s feature requirements Develop best practices for community collaboration and development of the API In this blog, I’ll provide an overview of the progress happening around MongoDB storage engines, and key outcomes from the summit. Why Pluggable Storage Engines? First, a bit of background. With users building increasingly complex data-driven apps, there is no longer a "one size fits all" database storage technology capable of powering every type of application built by the business. Modern applications need to support a variety of workloads with different access patterns and price/performance profiles – from low latency, in-memory read and write applications, to real-time analytics, to highly compressed "active" archives. The storage engine API allows MongoDB to be configured with a choice of storage engines, each configured for specific workloads. This “pluggable” approach significantly reduces developer and operational complexity compared to running multiple databases. Now users can leverage the same MongoDB query language, data model, scaling, security and operational tooling across different applications, each powered by pluggable MongoDB storage engines that are optimized for specific workloads. So what progress has been made so far? Quite a bit, and in a very short time frame. MongoDB 3.0 shipped with two supported storage engines: The default MMAPv1 engine, an improved version of the engine used in prior MongoDB releases, enhanced with collection level concurrency control. The new WiredTiger storage engine. For many applications, WiredTiger's more granular concurrency control and native compression provide significant benefits in terms of lower storage costs, greater hardware utilization, higher throughput, and more predictable performance. Benchmarks show MongoDB 3.0 configured with WiredTiger delivers 7-10x higher performance than MongoDB 2.6 using the original MMAP storage engine. In addition, storage compression rates of up to 80% are fairly common. To demonstrate how developers could start to innovate, several experimental engines also shipped with MongoDB 3.0, including the in-memory and /dev/null storage engines. Beyond the walls of MongoDB engineering teams, the community also started releasing its own engines. Within one quarter of 3.0’s availability: Facebook released mongo-rocks , a MongoDB storage engine based on its RocksDB embedded database project. Mongo-rocks is already in use for some production workloads at Facebook’s Parse mobile backend-as-a-service division, which today supports over 500,000 applications on MongoDB. Percona published a release candidate of its TokuMXse storage engine. What’s Next? Held in our New York offices on June 4th, the summit drew diverse representation from the community. Some attendees already had storage engines for MongoDB; some were in varying stages of development; while others were just starting to evaluate the opportunities. Attendees included: Facebook, who already has RocksDB Percona, readying TokuMXse SanDisk, developing an engine optimized for its Fusion-io SSD storage devices Deep , who have developed a MySQL storage engine featuring an adaptive database kernel 8k Data , currently building a JSON database on top of Postgres There was also representation from MongoDB’s storage engine teams, who discussed additional engines planned for the MongoDB 3.2 timeframe, including: An encrypted storage engine protecting data at-rest, with keys secured by the industry standard Key Management Interoperability Protocol (KMIP). At-rest encryption is especially critical for regulated industries such as healthcare, financial services, retailers and certain government agencies. And with so many high profile breaches of sensitive data at high profile companies over the past five years, increasingly all data is being encrypted. An in-memory engine designed to serve ultra high-throughput, low-latency apps typical in finance, ad-tech, gaming, real-time analytics, session management and general cache use cases. A new option for insert only workloads (e.g., streaming IoT sensor data, log file analysis, social media feed ingestion), based on the LSM option in the WiredTiger storage engine. Much more than just sharing feeds and speeds, the intent of the storage engine summit was to bring together developers to share best practices in how to collaborate: Mark Callaghan, one of our esteemed MongoDB masters and member of Facebook’s technical staff, gave great insight into his experiences working as part of the MySQL storage engine community. He shared the good, the bad, and the downright ugly. Dr Michael Cahill, co-founder of WiredTiger, and now Director of Storage Engineering at MongoDB, shared his experiences implementing the WiredTiger storage engine, both while outside MongoDB, and within the company. Mathias Stearn & Geert Bosch, senior database kernel engineers, provided a deep dive walkthrough into the storage engine API implementation, and plans for the future. This gave all those present an opportunity to provide feedback on issues they had faced, solutions to move forward, and future requirements to shape the evolution of the API All of the talks gave a great frame-of-reference in how the MongoDB storage engine community should move forward. So what did the summit achieve in one short day? The formation of a focused community committed to building MongoDB storage engines. The establishment of a framework for communications, contributions, collaboration & future summits. It connected engine developers directly with MongoDB engineers designing & implementing API functionality. Not only has feedback already helped evolve the storage engine API in the current 3.0 release, additional enhancements will be reflected in MongoDB 3.2’s API. Wrapping Up Huge progress has been made, and the MongoDB storage engine ecosystem is continuing to grow. This diversity helps MongoDB users solve new classes of use cases with a single database framework. The storage engine API helps developers innovate faster. And it helps storage engine partners get their technology in front of a whole new audience. To learn more of the promise of multiple storage engines, download the What’s New in MongoDB 3.0 guide. What’s New in MongoDB 3.0
Australian Start-Up Ynomia Is Building an IoT Platform to Transform the Construction Industry and its Hostile Environments
The trillion dollar construction industry has not yet experienced the same revolution in technology you might have expected. Low levels of R&D and difficult working environments have led to a lack of innovation and fundamental improvements have been slow. But one Australian start-up is changing that by building an Internet of Things (IoT) platform to harness construction and jobsite data in real time. “Productivity in construction is down there with hunting and fishing as one of the least productive industries per capita in the entire world. It's a space that's ripe for people to come in and really help,” explains Rob Postill , CTO at Ynomia. Ynomia has already been closely involved with many prestigious construction projects, including the residential N06 development in London’s famous 2012 Olympic Village. It was also integral to the construction of the Victoria University Tower in Australia. Link to Podcast Episode Here “These projects involve massive outflow of money: think about glass facades on modern buildings, which can represent 20-30 percent of the overall project cost. They are largely produced in China and can take 12 weeks to get here,” says Postill. “Meanwhile, the plasterer, the plumber, the electrician are all waiting for those glass facades to be put on so it is safe for them to work. If you get it wrong, you can go in the deep red very quickly.” To tackle these longstanding challenges, Ynomia aims to address the lack of connectivity, transparency and data management on construction sites, which has traditionally resulted in the inefficient use of critical personnel, equipment and materials; compressed timelines; and unpredictable cash flows. To optimize productivity, Ynomia offers a simple end-to-end technology solution that creates a Connected Jobsite. Helping teams manage materials, tools, and people across the worksite in real time. IOT in a Hostile Environment The deployment of technology in construction is often fraught with risk. As a result, construction sites are still largely run on paper, such as blueprints, diagrams and models as well as the more traditional invoices and filing. At the same time, there is a constant need to track progress and monitor massive volumes of information across the entire supply chain. Engineers, builders, electricians, plumbers, and all the other associated professionals need to know what they need to do, where they need to be, and when they need to start. “The environment is hostile to technology like GPS, computers, and mobile phone reception because you have a lot of Faraday cages and lots of water and dust,” explains Postill. “You can't have somebody wandering around a construction site with a laptop; it'll get trashed pretty quickly." Enter MongoDB Atlas “On a site, you might be talking about materials, then if you add to that plant & equipment, or bins, or tools etc, you're rapidly getting into thousands and thousands of tags, talking all the time, every day,” said Postill. That means thousands of tags now send millions of readings on Ynomia building sites around the world. All these IoT data packets must be stored efficiently and accurately so Ynomia can reassemble the history of what has happened and track tagged inventory, personnel, and vehicles around the site. Many of the tag events are also safety critical so accuracy is a vital component and packets can't be missed. To address these needs Ynomia was looking for a database that was scalable, flexible, resilient and could easily handle a wide variety of fast-changing sensor data captured from multiple devices. The final component Postill was looking for in a database layer was freedom: a database that didn't lock them into a single cloud platform as they were still in the early stages of assessing cloud partners. The Commonwealth Scientific and Industrial Research Organisation , which Postill had worked with in the past, suggested MongoDB , a general purpose, document-based database built for modern applications. “The most important factor was that the database is event-driven, which I knew would be difficult in the traditional relational model. We deal with millions of tag readings a day, which is a massive wall of data,” said Postill. A Cloud Database Ynomia is using MongoDB Atlas , the global cloud database service, now hosted on Microsoft Azure. Atlas offers best-in-class automation and proven practices that combine availability, scalability, and compliance with the most demanding data security and privacy standards. “When we started we didn't know enough about the problem and we didn't want to be constrained," explained Postill. "MongoDB Atlas gives us a cloud environment that moves with us. It allows us to understand what is happening and make changes to the architecture as we go." Postill says this combination of flexibility and management tooling also allows his developers to focus on business value not undifferentiated code. One example Postill gave was cluster administration: "Cluster administration for a start-up like us is wasted work," he said. "We’re not solving the customer's problem. We're not moving anything on. We’re focusing on the wrong thing. For us to be able to just make that problem go away is huge. Why wouldn’t you?" Atlas also gives Ynomia the option to spin out new clusters seamlessly anywhere in the world. This allows customers to keep data local to their construction site, improving latency and helping solve for regional data regulations. Real Time Analytics The company has also deployed MongoDB Charts, which takes this live data and automatically provides a real time view. Charts is the fastest and easiest way to visualize event data directly from MongoDB in order to act instantly and decisively based on the real-time insights generated by event-driven architecture. It allows Ynomia to share dashboards so all the right people can see what they need to and can collaborate accordingly. “Charts enables us to quickly visualize information without having to build more expensive tools, both internally and externally, to examine our data,” comments Postill. “As a startup, we go through this journey of: what are we doing and how are we doing it? There's a lot of stuff we are finding out along the way on how we slice and re-slice our data using Charts.” A Platform for Future Growth Ynomia is targeting a huge market and is set for ambitious growth in the coming years. How the platform, and its underlying architecture, can continue to scale and evolve will be crucial to enabling that business growth. “We do anything we can to keep things simple,” concluded Postill. “We pick technology partners that save us from spending time we shouldn't spend so we can solve real problems. We pick technologies that roll with the punches and that's MongoDB.” When we started we didn't know enough about the problem and we didn't want to be constrained," explained Postill. "MongoDB Atlas gives us a cloud environment that moves with us. It allows us to understand what is happening and make changes to the architecture as we go. Rob Postill, CTO, Ynomia