Performance Tips: MongoDB at Firescope

MongoDB
December 19, 2012
#Releases

Guest post by Pete Whitney

Starting to work with any new technology or new API is always challenging at first. You’re often not quite sure of the best ways to get things done or if you’re are using the new technology in the most efficient manner. Furthermore, the early learning process is often littered with trial and error improvements that unfortunately take time to rework and reengineer into more optimal solutions. It sure would be nice if we could short circuit this learning process and simply arrive at nirvana on the first cut. While I won’t proclaim that the destination of the blog is nirvana, I will try to short circuit the learning process by sharing four specific performance related tips that we learned at FireScope Inc. when we transitioned from MySQL to MongoDB for our improved cloud based Stratis product. This blog will share the shorthand version of these tips and point the reader to a more in depth rendering if further understanding is desired.

  1. Through a comparative analysis FireScope found that accessing MongoDB via the MongoDB java driver was three times faster than doing the same access using SpringData. While SpringData yields many benefits it accomplishes its job using a reflection based solution that is performed on top of the native MongoDB java driver. So for FireScope’s performance centric considerations paying a 3X performance penalty for its benefits was not a tradeoff we were willing to make.
  2. When retrieving (querying) for a document consider returning only the fields that are needed to fulfill the current use case. In our analysis FireScope found that using this approach delivered an astounding nine times (9X) performance improvement.
  3. Aggregate data via nesting as much as makes sense for your application. By aggregating data, in particular historical related data, the future access of this data is improved because the accessed data, due to nesting, is always proximally close. Also index size can often be significantly reduced by aggregating historical data due to a reduction in the number of entries in the index. Smaller indexes yield smaller memory footprint, and faster index access times.
  4. When possible allocate documents early with the appropriate amount of space and update them later. If your updates do not change the size of the document than the document will not be moved. If the document is not moved then index updates are not required. This approach provides that most optimal approach for document creation with subsequent updates.

As noted earlier these are the shorthand tips. If you are interested in learning more about the research and analysis that produced these recommendations, you can read the following article on real-world application performance with MongoDB. 

Pete serves as Vice President of Cloud Development at FireScope Inc. Pete’s primary role at FireScope is overseeing the architecture and development of FireScope Stratis and ensuring that our product line is the envy of the IT world. The architectural cornerstones of the Stratis product are unlimited scalability, built in redundancy, and no single point of failure. In the advertising industry Pete designed and delivered DG Fastchannel’s internet based advertising distribution architecture. Pete also excelled in other areas including design enhancements in robotic machine vision systems for FSI International Inc. These enhancements included mathematical changes for improved accuracy, improved speed, and automated calibration. Pete also designed a narrow spectrum light source, and a narrow spectrum band pass camera filter for controlled machine vision imaging. Pete graduated Cum Laude from the University of Texas at Dallas, and holds a Bachelor’s of Science in Computer Science. Pete can be contacted via Email at pwhitney@firescope.com.

← Previous

3D Repo Runs MongoDB

If you’re an architectural or engineering firm, you’ve undoubtedly confronted the difficulty of managing and collaborating on 3D assets like CAD drawings.  The act of sharing massive files is hard but feasible, but it is significantly complicated by the inability to determine that you’re using the latest version.  For the CAD-inclined, there’s hope.  Jozef Dobos , a doctoral student at University College London (UCL), has applied the geospatial indexing capabilities of MongoDB a version control system for 3D assets called  3D Repo .  Sponsored by  Arup Foresight , the built environment innovation division of Arup Group Limited, a global design and business consulting firm with offices in over 30 countries, 3D Repo leverages the flexibility of MongoDB’s data model, not to mention its geospatial capabilities, to make collaboration on 3D assets easy. The Problem Whether an architectural firm or a product design company, collaboration on large (500GB+) 3D assets is a critical but difficult task, one not solved simply by sending links around to these assets. The maintenance of assets in a large 3D visualisation can involve authors that are numerous, geographically dispersed, and diverse in their skillsets and tools. This presents problems including, but not limited to, maintaining consistency of the models and dealing with concurrent edits in the same part of a 3D scene. For example, in the industry today, each user loads a particular 3D scene into a modeling tool, modifies it, and then re-saves the entire file again, making any version tracking and sharing unnecessarily unclear. By the time a simulation is completed, the design might progress so rapidly that the results are often inapplicable.  An SVN for 3D Assets Looking for a better way to collaborate on 3D files, Dobos developed a unified and integrated framework, called  3D Repo , that supports collaborative editing and distribution of 3D assets.  Think of it like Subversion (SVN), but for 3D assets, not code.   The 3D Repo framework tracks multiple revisions of 3D assets so that they can be integrated later on. It thus provides similar functionality to file-based revision control systems like SVN, but is built around MongoDB, thereby avoiding the constraints of a file-based system. The framework also supports distributed editing over the Internet and additional lightweight clients in web-browsers and mobile devices. Such an approach is expected to improve the engineering as well as public engagement and considerably reduce the costs of future industrial development. Why MongoDB? While there are other proprietary and open-source databases that support spatial data directly - e.g. Oracle Spatial and PostGIS for PostgreSQL - these alternatives focus on 2D geometry, not 3D models and their properties. They also do not support the type of revision histories that is commonly required in 3D asset management. MongoDB, on the other hand, avoids rigid data models and is optimized for large read-write operations. It is therefore possible to store 3D models in a database and, due to MongoDB’s schema flexibility, also track other associated data such as semantic relationships and even individual revisions, as is the case in 3D Repo.  As Dobos explains: Basically, relational databases can in no way cut it. Rigid table structures are not suitable for highly diverse and large 3D data. In our case, a single 3D model or, better said, a “3D scene” is represented as a scene graph, where each node can be, and most of the time is, something different. It can be a node for a mesh, hence a large binary array, or a transformation, where transformation matrix is simply 16 numbers, or animation, bone, material, texture etc. Each of these is represented and stored differently. So key-value pairs are a match made in heaven for this type of data. In MongoDB, 3D Repo assigns two collections (tables) per 3D scene, one for all the scene graph constituents and one for all the documents that belong to a revision history. Hence, each of these collections stores a directed acyclic graph, making the access implementation reusable. Once the data is in a database, access is implicitly supported in a distributed manner via a dedicated query language.  In order to offer a scalable visualization platform, 3D Repo streams decomposed scene graph components from MongoDB onto client devices, reconstructs their 3D representation and displays them for viewing. This explains why MongoDB is superior to an RDBMS like Oracle for this kind of application, but it doesn’t address the possibility that Dobos could have turned to another NoSQL database, and particularly a graph database, given that a scene graph is a directed acyclic graph.  Surely this should be a fit for a graph database? Not really, as Dobos articulates:  Mostly simply stated, we don’t need any complicated graph traversal operations. We opted for MongoDB because of its efficiency with binary storage, BSON rather than ASCII (hence a clear winner over CouchDB, too), and due to its flexibility, not to mention its web-ready approach. MongoDB is proven to scale well, has sharding built-in and there is a thriving community to support it. Most of the design inspiration in our system actually comes from webinars by Wordnik and their experience.  We are really grateful for those! In sum, MongoDB delivers the flexibility and 3D geospatial capabilities necessary to making 3D Repo a reality. Results The 3D Repo approach offers significant benefits over alternative techniques like geographic information systems (GIS), given the direct MongoDB database connection, the scalability of MongoDB, and its built-in support for geospatial indexing. In addition, MongoDB enables version control for 3D assets in a way impossible to proprietary or open-source RDBMS.  Geographic information systems such as Oracle Spatial or PostGIS may be acceptable for 2D assets such as lines, but they cannot match MongoDB for representing 3D data efficiently and carrying metadata on top. Importantly, 3D Repo’s MongoDB-based solution can be extended to support various data types including building information modelling (BIM) or other representations.  For more information on the research Dobos is doing on 3D Repo, please see “ Revision Control Framework for 3D Assets ” (PDF) or review his  related research .  Note: MongoDB does not currently support 3D geospatial indexing. 

December 12, 2012
Next →

Building a Culture of Growth: SVP Simon Eid on MongoDB's Massive Opportunity in APAC

Simon Eid is Senior Vice President Asia-Pacific (APAC) at MongoDB and leads the sales teams across Australia and New Zealand, India, ASEAN, and Japan. Simon's go-to-market organisation in APAC is growing rapidly and has nearly tripled in size in the past three years. They are hiring in all regions . In this article, Simon discusses MongoDB’s opportunity in APAC and how he builds a culture of growth and accountability. Simon Eid, SVP APAC, MongoDB (left) and Anoop Dhankar, RVP ANZ, MongoDB (right) MongoDB's opportunity in Asia-Pacific Out of the top 13 economies by GDP in the world , five of them are located in APAC: China, Japan, Australia, India, and South Korea. And that's to say nothing of the ASEAN countries which alone have more than 650 million inhabitants. Combine this with the worldwide database market, one of the largest markets in the software industry. IDC estimates that it will grow to $137B in 2027, and MongoDB has just reached $1B in ARR. This gives you a sense of the massive market opportunity we have globally. Regardless of industry, product, or service, almost every company is becoming a technology company, which means that every company is becoming a data company. We believe MongoDB is the Developer Data Platform that is best placed to support and accelerate that trend. We’ve already captured thousands of customers around the globe, but it’s important to keep in mind that our world is still in the early stages of shifting to the cloud and changing how applications are built and run. Compared to other software, what's special about the market we play in is that the database is not a “nice-to-have”; it’s mission-critical for organisations. As our world continues to undergo this digital transformation, we have the opportunity to transform how our customers use software and data to innovate, create, and disrupt industries. For example, look at Cathay Pacific , Hong Kong's home airline carrier operating in more than 60 destinations worldwide. The company's digital team turned to MongoDB on their journey to become one of the first airlines to create a truly paperless flight deck. Flight Folder, their application built on MongoDB, consolidates dozens of different information sources into one place. Since the Flight Folder launch, Cathay Pacific has completed more than 340,000 flights with full digital integration in the flight deck. Their innovation is enabled by MongoDB. Building a team across regions and cultures Our team in APAC is unique because of the different markets and cultures within the region. What this means is that we go to market differently in India than we do in Australia, in Singapore than we do in South Korea, and so on. Each market is completely different, but within all of them, there is a huge opportunity. Different from many of our peers, in APAC we've established business leaders who run regionalized teams in India, ASEAN, and ANZ with all functions reporting to them. These teams essentially operate as their own business and implement local best practices into their strategy. But, it doesn’t mean they’re operating in a silo. At the leadership level, there is an immense amount of collaboration and sharing of experiences to identify what’s working and what isn’t within each region. We also have a fantastic global sales organisation that rolls out extensive training and best practices to help enable our local teams to best help our customers and grow the business. Members of our APAC team at a recent offsite in Phuket Culture The most important thing is culture. We have a very high standard around everything we do and how we interact with each other. We don’t entertain politics. You can teach someone new skills and coach them on how to be successful in a new role, but if they’re not aligned with the culture, they will not be a fit. It’s a non-negotiable for me and why the most important aspect of the hiring process is the cultural aspect. If you get the culture right, everything else starts to fall into place. What I hear at MongoDB and from the teams I've built at other companies is that this is the kind of culture they can really thrive and grow. At MongoDB, our culture is defined and shaped by six core values . One of the values that’s most important to my team is “Embrace the Power of Differences”. Within APAC, there are a variety of cultural identities and nuances that can often be difficult to navigate, whether it is cultural values, beliefs, or go-to-market strategy. It’s important that everyone who joins my team is respectful of each other’s regional culture. What we’ve done within the APAC region, and with teams across the globe, is take everyone on a journey to understand and embrace these cultural differences. Our role as leaders is to develop our teams, from the bottom all the way up, which is part of MongoDB’s BDR to CRO career development initiative. We need to develop the next wave of leaders so that they’re prepared to step up when the time comes. For APAC, this means that regardless of where someone is from, each team member has been coached and developed on the cultural nuances so that they can lead people and go to market in each of the different regions. It’s also important that each team member contributes to a culture of psychological safety. Being part of a high-growth tech company requires taking risks and making mistakes. We have a high standard and we hold each other accountable, but it never comes at the cost of creating an environment where people are afraid to fail. When someone faces setbacks, I encourage them to share those experiences so that we can collectively learn. Through mutual support, we foster a stronger team capable of delivering exceptional results. The future of MongoDB in Asia-Pacific For any organisation to be successful, I believe it’s critically important for the entire ecosystem to act as one. As I mentioned earlier, at MongoDB the whole country ecosystem is aligned around one set of goals, so it's not a case of different teams running off in different directions. The teams are willing to lean in and do what's required to help each other build a great business. I can confidently say that in APAC, we are one team. This means sales, marketing, customer success, solutions consulting, and professional services all working together to focus on three things: making customers successful, building technical champions, and driving new workloads. As we continue to grow our team and MongoDB’s footprint in the region, these are the three things that will drive our success. As I mentioned earlier, there's a huge opportunity for MongoDB in APAC. Despite hiring slowing down or stopping completely at many other organisations, we're continuing to invest heavily in the region. To give you a sense of that - we've nearly tripled the size of our APAC go-to-market team in the past three years, and we've got more open roles across the different functions and regions. If you want to be part of this journey, there are three things I want to reiterate: First, we are extremely passionate about our culture, from the field level up to the leadership level. As a team, this is the brand we bring to the market. Second, the opportunity here is massive based on the total addressable market and our current share. And third, we place critical importance on development. By joining this team, I can promise that you’ll be provided with countless opportunities to develop your career and make an impact. I’m confident in my team and the leadership we have in place who are ready to take MongoDB APAC to the next level. Join us !

September 28, 2023