Every month, we’ll be publishing the best community blog posts from the month. Here is the digest for July:
Every month, we’ll be publishing the best community blog posts from the month. Here is the digest for July:
- Mike O'Brien, node.js and python engineer at 10gen, wrote an overview of tips and tricks for using mongo, the MongoDB shell
- Jesse Jiryu Davis published an overview of Motor, the Asynchronous python driver for MongoDB.
- Kristina Chodorow wrote an overview of Shard Tagging, a new feature in the 2.2.0 series.
- Juhi Bhatia, a member of the Pune MongoDB User Group, wrote an article on how she improved on her Schema Design for Entrib’s EMG PaaS.
- Tobias Trelle, organizer of the Dusseldorf MongoDB User group, published a post on using GridFS with Spring Data.
- Rick Copeland wrote an overview of the Ming toolkit for Python with tips on how it can accelerate your development.
- Stephen Bronstein shared some tactics in continuous deployment in his blog post on Feature flipping with MongoDB and Node.js.
- 10gen interns, Samantha Ritter and Kaushal Parikh released their log visualization tool, Edda.
- Database as a service company ObjectRocket launched their blog with the post In Memory MongoDB concurrency testing on ObjectRocket
- Comsysto’s Johannes Brandstetter showcased how to use MongoDB to create a real-time Twitter Heatmap using MongoDB’s capped collections.
Want your blog post to be included in the next update? Tweet it out with the #mongodb hashatag or send it to us directly.
MongoDB at ShowClix
We recently spoke with Nate Good, Director of Software Engineering a t ShowClix , about ShowClix's technology stack and use of MongoDB. WHAT IS SHOWCLIX? ShowClix was founded in 2007 as a live event search engine and has evolved into the full-service event ticketing system. Millions of tickets, three offices and two cereal walls later, ShowClix is the preferred ticketing solution for thousands of performing arts theatres, live music venues, and festivals Ã¢â‚¬â€ù as well as museums, non-profit fundraisers, nightclubs, even circuses and rodeos. WHAT WERE SOME OF THE REASONS THAT LED YOU TO ADOPT MONGODB? We use MongoDB as our Swiss army knife of data stores. It supplements our primary relational database, filling in the gaps that are far better suited for MongoDB’s document oriented nature. The three main ways we use it are: Â· Membership Data: Our clients have the ability to import customer data into ShowClix to be used to authenticate against. The challenge is that each client may have a very different schema than the next. MongoDB’s schemaless nature lends itself very nicely to this problem by allowing us to store diverse data in a consistent and manageable way for our application. Â· Auto Complete: ShowClix uses an auto-complete or “suggest” feature in several areas of the application, which must quickly scan millions of records to provide relevant search results to our clients. MongoDB serves as the index for this feature and leverage its left-anchored regex index support to make things snappy. Â· Log Aggregation and Indexing: We use Scribe to aggregate the many types of logs across our many servers. However, once that data is aggregated, it sits in huge nasty flat files that can be a pain to navigate/search when they need to be accessed (grep/awk can only get you so far). We turned to MongoDB to help us make sense of our most important log files. These files are parsed and imported into MongoDB upon aggregation. This allows us to use MongoDB’s query engine to help us find the answers we need quickly. Additionally, we’re looking into using MongoDB’s newer aggregation framework and Scribe’s Thrift support to take this process one step further. YOU ARE USING MONGODB WITH A RELATIONAL DATABASE. HOW DO YOU DECIDE WHAT GETS STORED IN MONGODB VS THE RDBMS? For us, ACID compliancy, namely atomicity, is pretty important. We deal with financial transactions and finite resources like available seats in a venue. For this reason, much of our ticket reservation process lives in a traditional RDBMS. However, there are definitely areas where RDBMS falls short. The rigidness of RDBMS schemas can lead to some awkward “round peg in a square hole” situations. When we’re modeling data that is a little less predictable, MongoDB’s schemalessness is a natural choice. It provides the flexibility needed, with a feature rich query engine to boot. HOW IS THE DATA MODEL FOR YOUR MEMBERSHIP DATA DIFFERENT FROM HOW YOU WOULD HAVE MODELLED IT IN A RELATIONAL DATABASE? The data our clients collect on their members can vary drastically from one client to the next. Had we stuck with an RDBMS for modeling membership data, we would have had to choose from two awkward options: a) De-normalize the data into blobs and sacrifice the very features that make RDBMS’ great or b) Try to normalize the data and end up with a multi-table, key-value mess. With Mongo, we’re able to get the best of both worlds: a DB that models unique membership data in a natural, cohesive “document” and still provides a brilliant query engine. YOU MENTIONED THAT YOU ARE LOOKING FORWARD TO 2.2. WHAT TYPES OF ANALYSES ARE YOU PLANNING TO DO WITH THE NEW AGGREGATION FRAMEWORK? The 'GROUP BY’ clause in SQL is arguably one of its most useful features from an analytics standpoint. While we’re big fans of MongoDB’s map/reduce support, we’re very excited to see a more familiar analog to SQL’s aggregate support. We hope this will be a more natural transition into analytics in MongoDB for those on our team who are more accustomed to the SQL world. We’ll likely use it the most with our log dataset in mongo to get answers about user and system behavior. Want to learn more? Visit ShowClix to learn about their innovative event ticketing software for managing every aspect of ticketing operations. AndÃ¢â‚¬Â_ they're hiring ! Tagged with: showclix, search, tickets, MongoDB, Mongo, NoSQL, Polyglot persistence, 10gen
Australian Start-Up Ynomia Is Building an IoT Platform to Transform the Construction Industry and its Hostile Environments
The trillion dollar construction industry has not yet experienced the same revolution in technology you might have expected. Low levels of R&D and difficult working environments have led to a lack of innovation and fundamental improvements have been slow. But one Australian start-up is changing that by building an Internet of Things (IoT) platform to harness construction and jobsite data in real time. “Productivity in construction is down there with hunting and fishing as one of the least productive industries per capita in the entire world. It's a space that's ripe for people to come in and really help,” explains Rob Postill , CTO at Ynomia. Ynomia has already been closely involved with many prestigious construction projects, including the residential N06 development in London’s famous 2012 Olympic Village. It was also integral to the construction of the Victoria University Tower in Australia. Link to Podcast Episode Here “These projects involve massive outflow of money: think about glass facades on modern buildings, which can represent 20-30 percent of the overall project cost. They are largely produced in China and can take 12 weeks to get here,” says Postill. “Meanwhile, the plasterer, the plumber, the electrician are all waiting for those glass facades to be put on so it is safe for them to work. If you get it wrong, you can go in the deep red very quickly.” To tackle these longstanding challenges, Ynomia aims to address the lack of connectivity, transparency and data management on construction sites, which has traditionally resulted in the inefficient use of critical personnel, equipment and materials; compressed timelines; and unpredictable cash flows. To optimize productivity, Ynomia offers a simple end-to-end technology solution that creates a Connected Jobsite. Helping teams manage materials, tools, and people across the worksite in real time. IOT in a Hostile Environment The deployment of technology in construction is often fraught with risk. As a result, construction sites are still largely run on paper, such as blueprints, diagrams and models as well as the more traditional invoices and filing. At the same time, there is a constant need to track progress and monitor massive volumes of information across the entire supply chain. Engineers, builders, electricians, plumbers, and all the other associated professionals need to know what they need to do, where they need to be, and when they need to start. “The environment is hostile to technology like GPS, computers, and mobile phone reception because you have a lot of Faraday cages and lots of water and dust,” explains Postill. “You can't have somebody wandering around a construction site with a laptop; it'll get trashed pretty quickly." Enter MongoDB Atlas “On a site, you might be talking about materials, then if you add to that plant & equipment, or bins, or tools etc, you're rapidly getting into thousands and thousands of tags, talking all the time, every day,” said Postill. That means thousands of tags now send millions of readings on Ynomia building sites around the world. All these IoT data packets must be stored efficiently and accurately so Ynomia can reassemble the history of what has happened and track tagged inventory, personnel, and vehicles around the site. Many of the tag events are also safety critical so accuracy is a vital component and packets can't be missed. To address these needs Ynomia was looking for a database that was scalable, flexible, resilient and could easily handle a wide variety of fast-changing sensor data captured from multiple devices. The final component Postill was looking for in a database layer was freedom: a database that didn't lock them into a single cloud platform as they were still in the early stages of assessing cloud partners. The Commonwealth Scientific and Industrial Research Organisation , which Postill had worked with in the past, suggested MongoDB , a general purpose, document-based database built for modern applications. “The most important factor was that the database is event-driven, which I knew would be difficult in the traditional relational model. We deal with millions of tag readings a day, which is a massive wall of data,” said Postill. A Cloud Database Ynomia is using MongoDB Atlas , the global cloud database service, now hosted on Microsoft Azure. Atlas offers best-in-class automation and proven practices that combine availability, scalability, and compliance with the most demanding data security and privacy standards. “When we started we didn't know enough about the problem and we didn't want to be constrained," explained Postill. "MongoDB Atlas gives us a cloud environment that moves with us. It allows us to understand what is happening and make changes to the architecture as we go." Postill says this combination of flexibility and management tooling also allows his developers to focus on business value not undifferentiated code. One example Postill gave was cluster administration: "Cluster administration for a start-up like us is wasted work," he said. "We’re not solving the customer's problem. We're not moving anything on. We’re focusing on the wrong thing. For us to be able to just make that problem go away is huge. Why wouldn’t you?" Atlas also gives Ynomia the option to spin out new clusters seamlessly anywhere in the world. This allows customers to keep data local to their construction site, improving latency and helping solve for regional data regulations. Real Time Analytics The company has also deployed MongoDB Charts, which takes this live data and automatically provides a real time view. Charts is the fastest and easiest way to visualize event data directly from MongoDB in order to act instantly and decisively based on the real-time insights generated by event-driven architecture. It allows Ynomia to share dashboards so all the right people can see what they need to and can collaborate accordingly. “Charts enables us to quickly visualize information without having to build more expensive tools, both internally and externally, to examine our data,” comments Postill. “As a startup, we go through this journey of: what are we doing and how are we doing it? There's a lot of stuff we are finding out along the way on how we slice and re-slice our data using Charts.” A Platform for Future Growth Ynomia is targeting a huge market and is set for ambitious growth in the coming years. How the platform, and its underlying architecture, can continue to scale and evolve will be crucial to enabling that business growth. “We do anything we can to keep things simple,” concluded Postill. “We pick technology partners that save us from spending time we shouldn't spend so we can solve real problems. We pick technologies that roll with the punches and that's MongoDB.” When we started we didn't know enough about the problem and we didn't want to be constrained," explained Postill. "MongoDB Atlas gives us a cloud environment that moves with us. It allows us to understand what is happening and make changes to the architecture as we go. Rob Postill, CTO, Ynomia