Another release of MMS is here!
In the realm of automation, today we are releasing the ability to provision on ephemeral SSD drives for all EC2 types.
Within the MMS UI, users can now search by replica set name, which is particularly helpful if you have hundreds of replica sets.
A new monitoring agent (220.127.116.11) and a new backup agent (18.104.22.168) were released - agents will now identify themselves to the MMS servers using the FQDN of the servers on which they are running.
The outlook for 2015 is “busy” – we’re just starting to roll out 2.8 across all of MMS Systems here internally, and we are sure customers are going to love all the new functionality.
On day one, we plan to have functionality for customers to easily and quickly upgrade to MongoDB, including choosing their storage engine. There will also be some changes to performance metrics in the UI - some metrics have been removed, including the old favorite “Lock %”. This is due to the fact that some metrics are no longer supported in MongoDB 2.8, and the metrics supported also depend on the storage engine chosen. 2.8 will also mean new metrics, so stay tuned for a list of those. In summary, this release was primarily pre-release prep work, which is not yet available in the UI.
Leaf in the Wild: Scaling China’s Largest Car Service App with MongoDB
Leaf in the Wild posts highlight real world MongoDB deployments. Read other stories about how companies are using MongoDB for their mission-critical projects. Kuaidi uses MongoDB at the heart of its taxi hailing service, connecting drivers with passengers up to 6 million times a day, and managing nearly half a billion orders. Kuaidi has scaled MongoDB across 4 geographic regions, serving thousands of reads and writes every second. Following his presentation at last month’s MongoDB Day in Beijing, I sat down with Ouyang Kang, Chief Architect at Kuaidi, to learn more about how China’s leading taxi booking application is using MongoDB, and his recommendations for those getting started with the database. Smartphone based taxi-calling and ride-sharing services are growing at an astounding rate – attracting significant investment (and huge company valuations). They are also intensely competitive. The choice of technology will ultimately drive success or failure in the market. In the world’s most populous country – and one suffering the most severe traffic congestion – the importance of using agile and scalable technology for transportation services is magnified. Please start by telling us a little bit about your company. Kuaidi was founded in 2012 and has grown to become Greater China’s largest car service application 1 , attracting investment from Alibaba and Matrix Partners. In just 2 years, we have attracted 100 million users who place up to 6 million ride requests every day via our smartphone app, connecting them to 3 million drivers in more than in 300 cities across China. And we are continuing to grow fast. The goal of Kuaidi Group is to improve the efficiency of urban transportation and the population’s quality of life. We currently operate 2 branded services – Kuaidi Taxi and Kuaidi ONE – which provide taxi and chauffeured limousine services respectively. Our long term plan is to offer services for every facet of passenger transportation combining location-based mobile technologies, data mining of our huge user base and intelligent routing algorithms. Tell us how you use MongoDB. At heart of our taxi booking application is the location based service, and we rely on MongoDB for this. Using MongoDB’s geospatial indexes and queries we can track the location of our drivers in real time, using it to connect users with their closest taxi, and displaying updates directly to the customer’s app. The location data is constantly being updated and queried. We also use MongoDB as an active archive of our order data. Each time a customer requests a taxi, the journey’s start and end points, the driver identity and fare are stored in a single record. We initially built our archive on top of MySQL, but once our order volume exceeded 100 million records, we hit scaling limits. We knew MongoDB scaled, so we migrated the archive to get the cost and performance benefits of horizontal scale out. What other databases do you use? We use Redis for caching and MySQL to store operational customer and order data. We also replicate data from MongoDB and MySQL into Hadoop for data mining and analytics. Did you consider other databases for your app? What made you select MongoDB? We considered three options for our location based service: Relational solutions based on MySQL and Postgres SOLR (for the search element of the application) MongoDB We evaluated each on multiple criteria, including Performance. We measure performance on multiple dimensions: latency, which is critical for good user experience on mobile apps; and speed of real time updates, so we are always working from the freshest data Scalability. We were confident that the service would quickly gain traction, so knowing we could scale our database on demand was paramount Ease-of-Use. We needed to achieve our performance and scalability goals without burdening our developer and operations team with complexity We evaluated all of the options on this criteria, and found MongoDB to be the best choice for us. It met the performance objectives. We found it easy to develop against. What was really important was that it proved easy to deploy and easy to run at scale . Please describe your MongoDB deployment Our MongoDB database is sharded across four geographic regions. A 7-node replica set is deployed in each region (6 data-bearing nodes and an arbiter). This deployment enables us to place data physically closer to local users for low latency access, as well as provide the scalability and resilience our application needs. We cannot tolerate downtime at all. We use Nagios for monitoring the application and database. Geo-Distributed MongoDB Deployment at Kuaidi We are running MongoDB 2.6 with the Java driver. Are there any metrics you can share? Yes. MongoDB is serving 50,000 operations per second (split 80:20 between reads and writes) Our database has grown to just under half a billion documents and continues to scale Do you have plans to use MongoDB for other applications? Our marketing team stores all of its promotions and messaging in MySQL, but is starting to hit scaling limits. As a result, it is not keeping pace with their demands. We are evaluating migrating this to MongoDB as well. What feature of the forthcoming MongoDB 3.0 release are you most looking forward to? It has to be document level concurrency control. As our service continues to grow, we need to scale to keep pace – especially writes. This is something we believe MongoDB 3.0 with its new WiredTiger storage engine will allow us to do. What advice would you give someone who is considering using MongoDB for their next project? Don’t just follow the crowd. Don’t just choose the same technology you have always chosen. There is so much innovation happening today, and the databases of the last decade are not always the right choice. Once you have a short-list of potential technologies, test them with your app, your queries and your data. It is the only way to be sure you are choosing the right technology going forward. Ouyang, thank you for your time, and sharing your experiences with the MongoDB community. Thinking about migrating from a relational database? Read the MongoDB white paper to get started: Migrating from RDBMS to MongoDB 1 Based on market share and transaction volume About the Author - Mat Keep Mat is part of the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies. < Read About Our William Zola Award for Community Excellence
Australian Start-Up Ynomia Is Building an IoT Platform to Transform the Construction Industry and its Hostile Environments
The trillion dollar construction industry has not yet experienced the same revolution in technology you might have expected. Low levels of R&D and difficult working environments have led to a lack of innovation and fundamental improvements have been slow. But one Australian start-up is changing that by building an Internet of Things (IoT) platform to harness construction and jobsite data in real time. “Productivity in construction is down there with hunting and fishing as one of the least productive industries per capita in the entire world. It's a space that's ripe for people to come in and really help,” explains Rob Postill , CTO at Ynomia. Ynomia has already been closely involved with many prestigious construction projects, including the residential N06 development in London’s famous 2012 Olympic Village. It was also integral to the construction of the Victoria University Tower in Australia. Link to Podcast Episode Here “These projects involve massive outflow of money: think about glass facades on modern buildings, which can represent 20-30 percent of the overall project cost. They are largely produced in China and can take 12 weeks to get here,” says Postill. “Meanwhile, the plasterer, the plumber, the electrician are all waiting for those glass facades to be put on so it is safe for them to work. If you get it wrong, you can go in the deep red very quickly.” To tackle these longstanding challenges, Ynomia aims to address the lack of connectivity, transparency and data management on construction sites, which has traditionally resulted in the inefficient use of critical personnel, equipment and materials; compressed timelines; and unpredictable cash flows. To optimize productivity, Ynomia offers a simple end-to-end technology solution that creates a Connected Jobsite. Helping teams manage materials, tools, and people across the worksite in real time. IOT in a Hostile Environment The deployment of technology in construction is often fraught with risk. As a result, construction sites are still largely run on paper, such as blueprints, diagrams and models as well as the more traditional invoices and filing. At the same time, there is a constant need to track progress and monitor massive volumes of information across the entire supply chain. Engineers, builders, electricians, plumbers, and all the other associated professionals need to know what they need to do, where they need to be, and when they need to start. “The environment is hostile to technology like GPS, computers, and mobile phone reception because you have a lot of Faraday cages and lots of water and dust,” explains Postill. “You can't have somebody wandering around a construction site with a laptop; it'll get trashed pretty quickly." Enter MongoDB Atlas “On a site, you might be talking about materials, then if you add to that plant & equipment, or bins, or tools etc, you're rapidly getting into thousands and thousands of tags, talking all the time, every day,” said Postill. That means thousands of tags now send millions of readings on Ynomia building sites around the world. All these IoT data packets must be stored efficiently and accurately so Ynomia can reassemble the history of what has happened and track tagged inventory, personnel, and vehicles around the site. Many of the tag events are also safety critical so accuracy is a vital component and packets can't be missed. To address these needs Ynomia was looking for a database that was scalable, flexible, resilient and could easily handle a wide variety of fast-changing sensor data captured from multiple devices. The final component Postill was looking for in a database layer was freedom: a database that didn't lock them into a single cloud platform as they were still in the early stages of assessing cloud partners. The Commonwealth Scientific and Industrial Research Organisation , which Postill had worked with in the past, suggested MongoDB , a general purpose, document-based database built for modern applications. “The most important factor was that the database is event-driven, which I knew would be difficult in the traditional relational model. We deal with millions of tag readings a day, which is a massive wall of data,” said Postill. A Cloud Database Ynomia is using MongoDB Atlas , the global cloud database service, now hosted on Microsoft Azure. Atlas offers best-in-class automation and proven practices that combine availability, scalability, and compliance with the most demanding data security and privacy standards. “When we started we didn't know enough about the problem and we didn't want to be constrained," explained Postill. "MongoDB Atlas gives us a cloud environment that moves with us. It allows us to understand what is happening and make changes to the architecture as we go." Postill says this combination of flexibility and management tooling also allows his developers to focus on business value not undifferentiated code. One example Postill gave was cluster administration: "Cluster administration for a start-up like us is wasted work," he said. "We’re not solving the customer's problem. We're not moving anything on. We’re focusing on the wrong thing. For us to be able to just make that problem go away is huge. Why wouldn’t you?" Atlas also gives Ynomia the option to spin out new clusters seamlessly anywhere in the world. This allows customers to keep data local to their construction site, improving latency and helping solve for regional data regulations. Real Time Analytics The company has also deployed MongoDB Charts, which takes this live data and automatically provides a real time view. Charts is the fastest and easiest way to visualize event data directly from MongoDB in order to act instantly and decisively based on the real-time insights generated by event-driven architecture. It allows Ynomia to share dashboards so all the right people can see what they need to and can collaborate accordingly. “Charts enables us to quickly visualize information without having to build more expensive tools, both internally and externally, to examine our data,” comments Postill. “As a startup, we go through this journey of: what are we doing and how are we doing it? There's a lot of stuff we are finding out along the way on how we slice and re-slice our data using Charts.” A Platform for Future Growth Ynomia is targeting a huge market and is set for ambitious growth in the coming years. How the platform, and its underlying architecture, can continue to scale and evolve will be crucial to enabling that business growth. “We do anything we can to keep things simple,” concluded Postill. “We pick technology partners that save us from spending time we shouldn't spend so we can solve real problems. We pick technologies that roll with the punches and that's MongoDB.” When we started we didn't know enough about the problem and we didn't want to be constrained," explained Postill. "MongoDB Atlas gives us a cloud environment that moves with us. It allows us to understand what is happening and make changes to the architecture as we go. Rob Postill, CTO, Ynomia