GIANT Stories at MongoDB

Designing the Perfect Prototyping, Workflow and Collaboration Platform in the Cloud

InVision was recognized at MongoDB World 2017 as the winner of the MongoDB Innovation Award: Atlas category. We had the opportunity to sit down with Dana Lawson, InVision’s VP of Platform Engineering to learn more about their innovation.

Tell us about InVision

InVision is the world’s leading design collaboration platform. Helping companies like IBM, Airbnb, Visa, Netflix and Evernote unlock the power behind design-driven product development, InVision makes it easy for teams to prototype, manage their workflow, and control their entire design process all in one place.

The goal of InVision has always been to create a highly collaborative design platform in the cloud that would allow people around the world to have access to design, review, and user test products—all without a single line of code.

Why did you build InVision on MongoDB?

When originally building InVision, we looked to MongoDB right out of the gate because of its uptime and scalability. We needed to be able to provide our clients with a platform that is as reliable as we are.

In addition, MongoDB helps us easily build new features. You can imagine that with designs, requirements are very fluid and having a restrictive data model is a limiting factor. MongoDB’s document data model has help us innovate quickly. An example of a project that uses MongoDB is Inspect. For designers that build in Sketch, they can send their design to Inspect, which breaks down the Sketch file into different layers and allows front-end developers to get assets, CSS tags, and exact pixel dimensions for how the designs should actually look on a live site.

When we started with MongoDB, we were using Chef to automate AWS instances and the database. We had 28 replica sets spread across 4 different environments, some of them in different AWS accounts.

While we were having success, once MongoDB Atlas was released, we immediately took the leap. Moving to a database-as-a-service offering increased our team’s productivity as we were able to focus on our product, rather than managing the infrastructure. As a turnkey cloud database, MongoDB Atlas provides us with the flexibility needed to develop in a secure, robust environment while our customers continue to have access to a highly collaborative design platform in the cloud.

How do you use MongoDB Atlas and what impact does it have on your day to day?

We use MongoDB Atlas on the AWS cloud because of the ease of use and flexibility of the cloud platform. Together we are able to build a multi-tenant solution for Javascript Object Notation (JSON) messages received. Essentially, we use MongoDB as a transactional data store for any data, like our design artifacts, that benefit from not having a pre-defined schema.

The design data for a prototype can have any number of layers and graphics. Anyone who has seen or worked with a design tool will have familiarity with layers and the layers pane where objects can be nested, grouped, and inherited. MongoDB's flexible schema allows our backend services to store the data model for a prototype with minimal effort.

The most common queries we run are simple .find()s on indexed fields. At our peak, we read thousands of documents per second; we’re able to retrieve documents usually within 2 milliseconds.

By integrating Atlas with all of our provisioning, we have avoided the dreaded DevOps bottleneck. Without training, our engineers can “self-serve” by simply going to the InVision platform and define the instance they need; MongoDB Atlas configures it automatically and they’re ready to go.

What does your technology stack look like?

Our technology stack is primarily focused on Node and Google Go, allowing us to run a microservices architecture to create independent feature sets on data stores and significantly reduce dependencies. These reductions allow us to quickly spin up data stores and automatically add clusters as needed. Right now we use Kubernetes with MongoDB Atlas, though we are moving everything over to Atlas so we do not have to worry about uptime, EC2 clusters, anything. We’ve incorporated the MongoDB Atlas REST API into our Ansible scripts, which makes it incredibly easy for us to add new replica sets and users for different environments. We simply call out to the API and MongoDB Atlas spins up the replica sets so we don’t have to.

Some other technologies we use include AWS Lambda for short-term execution, Amazon SNS and Rabbit MQ for messaging and mobile notifications, and Amazon S3 for storing assets.

To learn more about InVision, watch their talk from MongoDB World 2017 here.

How Our FinTech Startup Migrated to MongoDB’s Database-as-a-Service to Save Time and Money

aBorrow is a Thailand-based fintech platform that matches loan products with borrowers’ profiles and needs. Launched in 2016, aBorrow needed a database they could trust to provide the high quality security and scalability required to handle their growing business Co-Founder and CTO, Cholathit Khueankaew, shares how MongoDB Atlas satisfied that need and transformed aBorrow’s business quickly, easily, and affordably.


As CTO and co-founder of a small business in Thailand, I spent hours managing production database configuration and monitoring until MongoDB Atlas was introduced. MongoDB now has multiple AWS regions available to host databases. This all-inclusive cloud-based solution was exactly what our business needed. Plus, it’s available at a perfect price point. Here is how MongoDB Atlas helped shape our startup and increase our efficiency.

I’m the co-founder and CTO of a startup, aBorrow, which is based out of Thailand. We launched our business about a year ago. At that time, we were using MongoDB for development, but we didn't have expertise in monitoring a production cluster in AWS. This created a perplexing issue for us until MongoDB Atlas was introduced. Its new cloud-based platform was the answer to our questions and exactly what we were looking for in a solution. MongoDB now has multiple AWS regions available to host databases. Although we are based in Thailand, we actually use cloud services in the Singapore region. With MongoDB Atlas, we can offload our database management to the cloud, which saves us time and money.

Essentially, we get a complete cloud-based database solution with optimized monitoring out of the box; the service is built on AWS but powered by MongoDB. It’s what we needed for staging, beta, and production because it comes with a dashboard where we can configure everything— adding new servers, monitoring uptime and downtime, and how the cloud database is operating.

Implementation was easier than I expected, too. I thought I would need several days setting up MongoDB Atlas and getting it ready for our developers. In the end, setup and the move to production took only a few hours. It was amazing.

Plus, we work in finance, so scalability and security is essential. aBorrow matches loan products with borrower’s profiles and needs. Being a loan consultant platform, our overall goal is to increase a borrower’s chance of being approved for a loan, and on the other hand, increase efficiency on bank's end and decrease acquisition cost. As our business has matured, we have learned who the best customers are to target, so we’ve doubled-down on our marketing expenses. We have jumped from 100 applications per day to 400 coming into our database. MongoDB has handled the increase in volume without issue.

MongoDB Atlas’ dashboard provides me with the transparency I need to feel confident that we are secure and running seamlessly. I can monitor security directly on the dashboard and MongoDB Atlas does the work. It monitors the database and provide backups with the data always being replicated in different availability zones. It provides end-to-end encryption of all the data and it uses Amazon’s own tools to keep databases safe. The solution also allows us to add new servers quickly and efficiently, which means we’re never inconveniencing our customer base.

Having visibility in the dashboards is among my favorite features. Other cloud solutions require logging into a third-party vendor to view the dashboard. With MongoDB, visibility is a click away. It is fast and easy and that is crucial, especially for maintaining data integrity.

MongoDB: A Developer’s Software

I like MongoDB Atlas because they’re looking to the future. They’re transitioning to providing infrastructure as service, which is vital. In addition, the MongoDB community is one of the biggest reasons we use the platform. They treat developers well and work diligently to build tools that developers want.

aBorrow is still growing—and quickly—and we’re still learning about cloud-based database solutions as we go. Our team is still quite small, and I don’t consider us super users for this type of platform yet; however, our experience with MongoDB Atlas is that the solution is quite complete. Even better, the minds at MongoDB are constantly releasing updates. Currently, we are enjoying a new feature called Data Explorer. We love it. We can query data from the database right on the admin dashboard or monitor how it is working. The solution and its capabilities are evolving. As a career developer, it is what I look for in a solution. I want a feature-rich product that will continuously get better.

Cost-Effective Startup Solution

Since we’ve made the switch to MongoDB Atlas, we are also saving money. Where other solutions charge their clients for use of the database, MongoDB is designed to be used as a database from the ground up.

Even better, with MongoDB Atlas, aBorrow is not being charged for services that we’re not using. Take for example, our setup. We currently use two database instances. Each cluster has three nodes, so we use six total nodes. We’ve spent only $40 in the past three months. Anybody who wants to use MongoDB can quickly provision it in the cloud and get platform support from the MongoDB team.

Not only is this impressive, but it’s invaluable to a startup like aBorrow.

Building a Business with MongoDB Atlas

Building, monitoring and maintaining our database is integral to the operation of aBorrow, but—to be honest—we have so much more on our plate than worrying about the functionality and performance of our database solution. That’s why MongoDB Atlas is so perfect for us. The company has yet to experience a failure or downtime of any kind. Even better, MongoDB Atlas is always providing usable statistics and up-to-the-minute feedback via its dashboard.

Ultimately, it works for our needs and resources. We are a team of four and it saves us time. With MongoDB Atlas, we do not need to do a lot of deploying, monitoring or orchestrating instances on the cloud because everything is done by MongoDB. The solution is built with my company’s needs in mind. Its easy-to-use interface ensures that I can get what I need at a moment’s notice and then get back to the business of running my business. It’s ideal for any entrepreneur who is looking for an all-in-one cloud-based service.


About Cholathit Khueankaew, Chief Technology Officer at aBorrow Company Limited

Aspired to be a software engineer since childhood. Spent years in Seattle, WA, studying, interning, and gaining experience in software development. Worked at an international software agency in Thailand, before starting own software house as Managing Director and Project Manager. Introverted, but a good team player and leader. Practical, technical, and analytical. Love to explore and experiment with new technologies and web stacks. Located in Chiang Mai and Bangkok, Thailand.

How to Configure LDAP Authentication for MongoDB

This is a guest post from Tom Spitzer, Vice President, Engineering of EC Wise, Inc. To expand on the MongoDB LDAP documentation, the objective of this post is to elaborate on configuring LDAP authentication for MongoDB. We will use the Mini-Clinic application presented at MongoDB World ‘17 as the illustrative example.

Mini-Clinic Windows Active Directory (AD) Users and Groups

We start by creating AD users and groups for Mini-Clinic.

The SYSInternals ADExplorer tool is a useful utility enabling one to view these objects in AD. This seems to work from Windows accounts that should not have privileges on the AD database, so be careful in how you use it!

User objects

dn: CN=scott,CN=Users,DC=ecwise,DC=local
memberof: CN=Mongo Scheduler,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
dn: CN=page,CN=Users,DC=ecwise,DC=local
memberof: CN=Mongo Practitioner,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
dn: CN=parker,CN=Users,DC=ecwise,DC=local
memberof: CN=Mongo Pharmacist,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
dn: CN=Adam,CN=Users,DC=ecwise,DC=local
memberof: CN=Mongo Auditor,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
dn: CN=Duke,CN=Users,DC=ecwise,DC=local
memberof: CN=Mongo DBA,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local

Group objects

dc: CN=Mongo Scheduler,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
member: CN=scott,CN=Users,DC=ecwise,DC=local
dc: CN=Mongo Practitioner,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
member: CN=page,CN=Users,DC=ecwise,DC=local
dc: CN=Mongo Pharmacist,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
member: CN=parker,CN=Users,DC=ecwise,DC=local
dc: CN=Mongo Auditor,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
member: CN=Adam,CN=Users,DC=ecwise,DC=local
dc: CN=Mongo DBA,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local
member: CN=Duke,CN=Users,DC=ecwise,DC=local

Procedure

Configure TLS/SSL for the server running MongoDB

By default, the mongod process connects to AD via TLS/SSL. You can configure ldap.transportSecurity in the MongoDB configuration file to none to disable TLS/SSL.

ldap:
transportSecurity: none

This is not recommended for production, but it can be useful for debugging when running into configuration issues.

Ensure AD server is enabled with TLS/SSL

Use ldp.exe (part of the Windows Server Remote Server Admin toolkit) to verify if AD server is actively listening for SSL. Refer to this article on LDAP over SSL verification for in-depth guidance. For example:

server: cdcorpwindc01.ecwise.local
port: 636
SSL: true

If the response includes "Host supports SSL......Established connection", then the AD server is responding to requests coming in over SSL.

Configure LDAPS (LDAP over SSL)

Export the Root CA certificates from the AD server.

  1. Click Start, Administrative Tools, Certification Authority
  2. Right-click on your CA, and select Properties
  3. In the CA Properties window, click on View Certificate
  4. In the Certificate window, click the Details tab and click Copy to File
  5. In the Certificate Export Wizard window, click Next
  6. Select Base-64 encoded X.509 (.CER), and click Next
  7. Enter the export name (e.g., c:\corpRootCa.cer), and click Next 8 Click Finish
  8. Copy certificate to the Linux server (assuming that MongoDB is running on a Linux server), for example, to /etc/openldap/certs

Note: step 6 is very important. If the certificate is not encoded with Base-64, it won't work for LDAPS. Edit /etc/openldap/ldap.conf, add a line.

TLS_CACERT /etc/openldap/certs/ecwise-root.cer

ecwise-root.cer is stored in /resource of the repo.

Specifying the TLS_CACERTDIR line is not essential here; TLS_CACERT takes the priority.

Use ldapsearch from the Linux MongoDB server to verify that LDAPS is working

ldapsearch -x -H ldaps://cdcorpwindc01.ecwise.local -b "DC=ecwise,DC=local" -D "CN=TM-EM, OU=Accounts,OU=Chengdu,OU=EC Wise Users,DC=ecwise,DC=local" -W

-b starting point to search
-D specifies the DC with which to authenticate to the server

Create User Administrative Role

MongoDB grants privileges to AD groups instead of AD users! We create MongoDB roles corresponding to AD groups, and add AD users into the AD groups. When the AD user logs into MongoDB they will be granted the role to which their AD group is assigned. Disable LDAP authentication, and execute the scripts below, which are in /init-script/create-user-administrative-role.js on the MongoDB server.

mongo create-user-administrative-role.js

var admin = db.getSiblingDB("admin")
admin.createRole(
    {
        role: "CN=Mongo DBA,OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local",
        privileges: [],
        roles: [ "userAdminAnyDatabase" ]
    }
)

Edit the MongoDB configuration file

The configuration should look like

security:
  authorization: "enabled"
  ldap:
  servers: "cdcorpwindc01.ecwise.local"
  userToDNMapping:
  '[
    {
     match: "(.+)",
     ldapQuery: "CN=Users,dc=ecwise,dc=local??sub?(sAMAccountName={0})"
     }
   ]'
  authz:
    queryTemplate: "OU=Groups,OU=EC Wise Users,DC=ecwise,DC=local??sub?(&(objectClass=group)(member={user_DN}))"
    bind:
      queryUser: "duke"
      queryPassword: "ecwise@123"
      setParameter:
      authenticationMechanisms: 'PLAIN'
DN=Distinguished name, see the <a href="https://msdn.microsoft.com/en-us/library/aa366101(v=vs.85).aspx" target="_blank">Active Directory reference</a> for an explanation. (Of course, you will substitute your OU and DC designators for our EC Wise designators.)

Configure LDAP query template for authorization

Use queryTemplate to identify which groups the user belongs to. The queryTemplate pattern is:

(AD groups DN to search)??sub?(query condition)

So in our configuration, it means search all DNs under "OU=EC Wise Users,dc=ecwise,dc=local" and its property objectClass equals “group” and member equals the user’s DN.

** Transform incoming usernames for authentication via AD**

Usually, the user DN is not something a user enters directly. So userToDNMapping helps transform the username to a full LDAP DN, which the system will use as the user’s logical identifier.

The ldapQuery pattern is:

(AD users DN to search)??sub?(match condition)

So in our configuration, we are trying to find the user where property sAMAccountName equals user name under "CN=User,dc=ecwise,dc=local".

Configure query credentials

MongoDB requires credentials for performing a query on your Windows AD server. We’ll set up a single use account with this permission. To do so, we configure queryUser and queryPassword in the bind section to specify the user who has permission to perform query.

Create roles and users for Mini-clinic

Execute the script(init_script/init_ldap_role_user.js) in the repo with DBA user to create roles and users for Mini-clinic. 'mongo -u duke -p --authenticationMechanism PLAIN --authenticationDatabase $external init_ldap_role_user.js'

Hopefully this article has helped you to better understand how Windows Directory and LDAP can be configured with MongoDB


Download the MongoDB Security Architecture to learn more about the security capabilities of MongoDB

About the Author: Tom is a software industry and IT services veteran, having been the lead technologist at both consulting and software development companies for over twenty-five years. As part of the EC Wise senior team, he leads the company’s secure development and secure database implementation practices, which have built products and services for leading business services providers, casinos and other security sensitive industries. Over the years, Tom has made presentations at many technical conferences and written for a variety of industry publications. Prior to joining EC Wise in 1999, Tom was CTO at a venture backed Silicon Valley e-commerce startup.

Looking at MongoDB for Content Management

We went to see our good friends Nuxeo and ask them what’s the best way to manage content at enterprise scale. They told us about how customers use MongoDB and their content services platform to deliver business applications with high scalability. Let’s hear about Thierry Delprat, Nuxeo CTO.

Please tell us a bit about Nuxeo. What does your company offer its customers?
Nuxeo provides purpose built digital content applications focused on business outcomes. Created on the industries most modern content services platform, leveraging the latest innovation in Machine Learning and Artificial Intelligence to deliver unparalleled insight across the content spread throughout the enterprise.

How would you describe your specific use case for MongoDB?
Nuxeo uses MongoDB because it is the best storage option for scalability and speed and it allows developers to build content intensive applications that have:

  1. Complex data structures
  2. A requirement for massive scalability
  3. The need for rapid retrieval for content
  4. A requirement for heavy write access

Nuxeo provides the infrastructure and the tools for efficiently addressing these requirements.

Did you start with MongoDB or migrate from a different database?
Initially Nuxeo only provided a SQL binding. However, we started seeing the limitations of the SQL model:

  • Difficulty to scale to very large volume
  • Performance issues when having a lot of Write operations
  • Complex architecture to be able to leverage Read replicas

These were the main reasons for us to start the work on a MongoDB Connector and indeed leveraging MongoDB allows us to:

  • Improve the performance and scalability of the application
  • Simplify the deployment architecture since we no longer need complex SQL replicas and distributed caching

More details can be found here: http://slides.com/thierrydelprat/mongodb-impact

Can you describe your application architecture and design approach?
Nuxeo is a software platform that was designed to be very modular and pluggable. This architecture allows us to choose what storage backend can be used. We use this pluggable mechanism to leverage MongoDB for scalable meta-data storage, Search or Blob storage.

How has MongoDB Atlas allowed you to scale your app as well as the business?
When deploying our application on AWS, MongoDB Atlas allows us to use MongoDB as a service. Nuxeo can then focus on delivering the Content Services Platform and let MongoDB provide the data safety and SLA required.

What programming languages are you using to build your app?
The server side is mainly built using Java but some extensions can be contributed using JavaScript. The client side use HTML5 / JS (Web Components & Polymer)

What are your thoughts about the current database trend of moving to the cloud?
Looks like all our customers are indeed moving to the Cloud, both private and public. In making this move, it is essential for the database also to be hosted in the Cloud.

This is one of the reasons why we use MongoDB Atlas to delivery content applications at scale.

Using MongoDB allowed us to simplify our deployment architecture, and to offer to our customer a big improvement in terms of scalability. We’re now exploring MongoDB Atlas for new ways for our customers to deliver their ECM applications.

MongoDB helps IHS Markit achieve up to 250x Higher Performance for Data Delivery Service

As data volumes grow, and business agility becomes the differentiator between success and failure, enterprises are increasingly re-evaluating legacy technology choices. In order to meet customer demands for higher performance and to develop the new, data-rich services demanded by the business, IHS Markit made the decision to migrate to MongoDB for its Data Delivery Service.

I had the opportunity to meet with Sander Van Loo, Director of Data Delivery & Index Administration Services at IHS Markit, to discuss why his team selected MongoDB, and the results of the company’s migration.

Can you start by telling us a little bit about your company?
IHS Markit, headquartered in London, is a world leader in delivering next-generation information, analytics, and solutions to customers in business, finance, and government. Through our information products and services, we provide our customers with 360-degree views of risk, opportunity, and financial impact. IHS Markit has more than 50,000 key business and government customers, including 85% of the Fortune Global 500 and the world’s leading financial institutions.

What is the role of the Data Delivery group?
We generate market and pricing data used by our financial services customers for the valuation and risk management of derivatives. We aggregate and analyze data from both our own internal data products and from third party data sources, which we then present to customers across multiple delivery channels and APIs.

All market and pricing data was stored in a legacy relational database, but we are now migrating that across to MongoDB.

What factors are driving your migration to MongoDB?
Application agility, performance, and scalability.

Can you tell me more about your challenges around application agility, and how MongoDB has helped?
Our data sources don’t have a fixed schema. Each source uses a different structure, and those structures can rapidly change as new attributes are added. We had to develop a schema generator to transform the data dictionaries that defined the data structures into a relational database schema. However, as the number of data sources increased, and as structures themselves changed, maintaining the schema generator consumed too much development resource. Rather than focusing on building application and analytics functionality, my team were increasingly occupied with schema management tasks.

MongoDB’s flexible document data model means we don’t need to predefine schemas. In addition, all of the data we ingest is formatted as JSON objects, so we can store it much more quickly without transformations or flattening it into rows and columns.

As a result of MongoDB, my developers are much more productive, working on projects that differentiate our products and improve customer experience.

How about performance and scalability?
The performance of our legacy relational database wasn’t predictable enough to provide performance SLAs on our deliveries, and its scaling model was not particularly suited to maintain performance with continuously growing data sets. We used eager in-memory caching frameworks to work around these challenges. This approach works well for current market data but it will not scale to years of historical data across millions of time-series data points.

The trigger to evaluate non-relational databases came when we started to develop enriched application functionality. We wanted to allow our customers to request not just current market data, but also provide access to historical time series data and support point in time snapshots on that data, totalling over 30TB. We constantly strive to reduce latency so that our data reaches customers sooner, and enable on-demand delivery of historical data. Any delay in presenting data makes our services less competitive.

Why did you choose MongoDB?
For our use case, MongoDB delivered higher scalability and better price/performance. We also found the document data model to provide richer schema flexibility than alternatives. It was more natural for our developers to work with, which increased their productivity.

How would you rate MongoDB’s performance?
From our proof of concept with 7 billion records, we’ve found that:

  • Reading end-of-day market data collected over a week takes 7 milliseconds on MongoDB compared to 758 milliseconds previously.
  • Reading end-of-day market data collected over a year takes 10 milliseconds on MongoDB compared to 2,523 milliseconds previously.

Our testing concluded that using MongoDB reads were up 250x faster, writes were 10x faster, and the database required only 35% of the storage space.

Now we can ingest a GB of data in less than 4 seconds, process it, and serve it out to our customers in less than 2.5 seconds. Performance improvements of this magnitude helps us deliver market leading SLAs and increase our customers’ opportunities to act on the data ahead of their competitors.

What does your deployment look like?
We have deployed a 4-shard MongoDB cluster across three sites located in London, Amsterdam, and New York. This configuration provides resilience to data center failures and allows us to co-locate data closer to users. As we onboard new data sets, we expect to scale the cluster by a factor of 3x to 4x over the next couple of years. MongoDB enables us to support our expected business growth with a scalable business model, and an architecture that is ready for the cloud.

How do you manage MongoDB?
Ops Manager is used for provisioning, configuring, and backing up MongoDB. We can drive these operations from the GUI, and so we don’t need to invest in developing complex scripts. Continuous backups and cross-shard snapshots across multiple regions provide the data protection demanded by the business. The Ops Manager API pushes all monitoring telemetry into our enterprise management platform so we can gain complete visibility of the application stack. We also use MongoDB Compass for schema and query optimization.

We have taken advantage of MongoDB's consulting services – our Dedicated Consulting Engineer acts as a trusted advisor in ongoing development, and helps build out our team’s operational capability.

You recently upgraded to MongoDB 3.4? What most excites you about this release?

The features we’re most interested in include:

  • Intra-cluster network compression. Our data is time-sensitive. We get lots of traffic bursts as batches of data are ingested into our platform, and then need to be processed across the cluster and released to customers around the world. Reducing network traffic by up to 70% is helping to deliver further latency improvements.
  • The new Decimal data type is helping to simplify our complex financial data.
  • Multi-faceted aggregations provides multiple filters and categorizations to guide data browsing and analysis.
  • Ops Manager server pools will allow more seamless provisioning of new MongoDB instances into our database services.

Are you considering MongoDB for other applications?
We are always looking for opportunities to use MongoDB for new applications. One area of interest is advanced risk factor computations. We plan on using Apache Spark R and Scala APIs to process market data stored in MongoDB, using the Apache Spark Connector for MongoDB.

How are you measuring the impact of MongoDB on your business?

  • Higher revenue. In our business, seconds matter. Faster data ingest, processing, and serving translates directly to more revenue for our customers, and improved customer acquisition and retention for us.
  • New applications. MongoDB allows our developers to innovate faster and deliver new functionality. We are better able to upsell value-added products to our customers.

Sander, thanks for sharing your experiences with the community.

Want to break free from the constraints of relational databases? Download our Relational Database to MongoDB migration guide.

2017 MongoDB Innovation Award Winners

MongoDB received an overwhelming number of nominations for the fourth annual MongoDB Innovation Awards, recognizing companies who are using MongoDB to dream big and deliver incredibly bold, innovative solutions for their business and industry.

We are thrilled to announce our 13 winners who will be honored at MongoDB World 2017, June 20-21 in Chicago.

See the full list and read a bit more about how they are disrupting the status quo here:

Healthcare Category: Experian Health


Experian Health helps customers make smarter business decisions, build a better bottom line and foster stronger patient relationships for more than 3,200 hospitals and 7,100 healthcare organizations, representing more than 500,000 healthcare providers. The team recently launched the Universal Identity Manager (UIM), a patient identifier system built on top of MongoDB that brings together data from disparate databases, systems and data formats to create a single view of a patient. The UIM improves the safety, speed, quality, and cost of care for patients and caregivers alike.

The Startup Enterprise: Nuxeo


Working with MongoDB, Nuxeo built an entirely modern, cloud native, Content Services Platform - that enables customers to easily transform data into highly valuable digital assets, supporting their Digital Transformation initiatives. Native integration into business and productivity applications means customers can work the way they want without disrupting their operations.

MongoDB is helping Nuxeo's largest customers exceed their most demanding scalability, performance and high availability requirements. Using MongoDB Enterprise Advanced, Nuxeo has seen a 10x improvement in bulk import throughput, stored more than 1B documents on a single cluster, and secured an import rate of more than 30K documents per second.

Internet of Things: Orbita



Orbita is an innovator of connected home healthcare solutions that increase patient engagement, improve patient outcomes, and reduce costs for the patient and healthcare provider alike. Building on top of voice-powered assistants such as Amazon Echo and Google Home, the team created Orbita Voice™, the first healthcare-centric solution designed to create interactive voice experiences that can survey patients, provide medical education, share activity reminders, monitor compliance, and more. Furthermore, Orbita Engage™, a first-of-its-kind IoT platform built for comprehensive care management, allows organizations to create, deploy, and manage HIPAA-compliant applications that combine data from wearables and other sensor-embedded devices into collaborative care experiences. Features include:

  • Rule-based triggers for rapid intervention based on real-time data and insights
  • User engagement tracking
  • Visualizations and built-in analytics that help shape responses to user inputs

Leveraging MongoDB Enterprise Advanced, Orbita securely ingests massive volumes of IoT data, adapts to changing data requirements, and helps organizations provide patients with high quality, cost-effective, and personalized care and support.

The Enterprise Startup: Pivotal


Pivotal Cloud Foundry, Pivotal's flagship software product, is the world's most powerful cloud-native platform used by organizations to rapidly develop and run modern and legacy applications at startup speeds. As a result, Pivotal works with over one-third of the Fortune 100, and a rapidly growing portion of the Fortune 2000. Moreover, Pivotal's continued customer momentum is accelerated by its growing partner ecosystem—highlighted by collaborations with companies such as MongoDB. MongoDB Enterprise is integrated as a service broker on Pivotal Cloud Foundry via Ops Manager, allowing dynamic on-demand provisioning of MongoDB service instances across hybrid clouds in standalone, replica set, or sharded cluster configurations.

The joint solution drives rapid application development by reducing the friction between development and IT Operation teams. Developers are able to instantly provision MongoDB instances in a self service manner, without worrying about the complexities of configuring the underlying infrastructure. Simultaneously, IT Operation teams can seamlessly monitor, manage and scale MongoDB instances to ensure high service levels. Together, Pivotal and MongoDB offer the promise of an automated platform to build and run cloud-native applications, at scale.

MongoDB Atlas: InVision



InVision is the world’s leading design collaboration platform. Leveraging MongoDB Atlas as the multi-tenant solution for JSON messages, InVision built a highly collaborative design platform in the cloud that allows people around the world to have access to their designs at the speed of light and without a single line of code. Companies like IBM, Airbnb, Visa, Netflix and Evernote trust InVision to unlock the power behind design-driven product development.

Panera Bread: Launch Fast


Panera Bread uses MongoDB to help power U.S. e-commerce applications. This has helped Panera ensure their apps are available and response times are optimized.

Partner of the Year: Infosys


Infosys, a global leader in consulting, technology and next-generation services, partnered with MongoDB to launch a 1000+ consultant practice coupled with a robust center of excellence to help enterprises accelerate their digital transformation and application modernization initiatives. Infosys and MongoDB not only help organizations address the key pain points of cost, performance and scalability across legacy applications, but also respond to changing business needs by building new digital capabilities. As part of the partnership, Infosys and MongoDB are already helping many large enterprises with renewing and modernizing their IT landscape.

Customer Experience: Cisco



Cisco is the worldwide leader in IT that helps companies seize the opportunities of tomorrow. One important part of the business is Cisco’s eCommerce Platform which serves 140,000 global users across Cisco’s sales team, partner employees, and customer employees. This is a mission critical platform supporting $40+B of Cisco’s business. In order to tap into the agility of the cloud and improve the customer experience Cisco migrated to MongoDB. This change is helping developers get new applications live quicker, providing better scalability and giving customers a greatly improved experience by improving response times to users by 5x. Cisco’s commitment to excellence and embrace of cloud-native technologies has dramatically improved how its most important clients interact online.

Scale: Phreesia


Phreesia is at the forefront of improving the patient experience and organizational efficiency by helping healthcare providers better manage the patient intake process so they can focus on what they do best: rich doctor-patient interactions and high-quality treatment. MongoDB powers one of Phreesia’s most important applications – the eligibility application – that covers the gamut of the ‘run-up appointment’ process and allows patients to proactively pre-register, manage copayments, and ensure all their personal contact, medical history, and insurance information is accurate, up to date, and secure. Phreesia manages intake for over 50 million patients annually and has checked in more than 15% of the U.S. population.

Data driven business: HSBC




A digital revolution is happening at HSBC. As one of the world's largest banking and financial services organizations, the company is building global digital solutions that will help millions of customers worldwide to manage their personal and business finances. The drive for digital innovation comes as part of a series of strategic actions set out by HSBC’s board to adopt technology that makes HSBC simpler, better and faster. An example of that is how two HSBC trading IT teams are migrating away from relational databases to MongoDB as a technology to help simplify, migrate and manage complex data change. This technology shift is happening in tandem with a philosophical shift to an agile development and devops culture, enabled in part by MongoDB’s flexibility and scalability.

Modern Enterprise Architecture: IHS Markit


IHS Markit (Nasdaq:INFO), whose customers include 85% of the Fortune Global 500, is a world leader in information, analytics and solutions for the major industries and markets that drive economies worldwide. The company’s Data Delivery Service gives businesses access to more than 50 different streams of data including bonds pricing, commodities data and securities information. The service is powered by a complex infrastructure originally built on a relational database. However, growing data volumes and new requirements mandated a faster and more scalable solution. To solve that challenge and get new applications to market quickly, IHS Markit migrated its Data Delivery Service to MongoDB. The results have been impressive, with some financial data being delivered as much as 250x faster than it was previously.

Single View: Barclays




Barclays is solving one of the trickiest challenges facing any enterprise: a true 360 degree view of the customer with a platform that gives all support staff a complete single-view of every interaction a customer has had with the bank – including branch interactions, telephone banking, fraud detection alerts, outbound transactions, and more. This is helping Barclays drive customer interactions to new digital channels and improve the customer experience.

Zola Award Winner: Robert Fehrmann

Robert Fehrmann is a Senior Architect at Snagajob, a site that aims to put people in the right fit positions so they can maximize their potential and live more fulfilling lives. Robert has been working with Database technologies for over 30 years. Back then the newest kid on the block was Oracle. Over the years Robert has worked on countless applications, big and small, development and administration, always staying close to the data tier.

Since bringing MongoDB to Snagajob in 2011, Robert has taken almost all the courses at MongoDB University, has been a founding member of the Richmond MUG (MongoDB User Group), which celebrated it's 4th anniversary in October of last year. And when Snagajob opened an office in Arlington, VA, Robert joined the Washington DC MUG, became a co-organizer and convinced Snagajob to provide the community with the same support for the DC community that the Richmond community has enjoyed for years.

Connecting Patients to Medical Providers Quicker with MongoDB – Phreesia Wins MongoDB 2017 Innovation Award for Scale

This year, we received nominations across dozens of industries for the annual MongoDB Innovation Awards that recognize organizations and individuals that took a giant idea and made a tangible impact on the world. The winners will be honored at MongoDB World 2017 , in Chicago on June 20 and 21. But we’re so excited about some of the winners we wanted to share their stories before the event. This year, Phreesia won the MongoDB Innovation Award for Scale.

Here’s why:

Improving Patient Experience

Patient care and organizational efficiency are vastly improved today, thanks to technology innovation that is connecting hospitals, insurance providers, doctors’ offices, and patients. Phreesia, a company at the heart of this movement, is helping healthcare providers better manage, access, and store patient information so the healthcare organization can focus on what they do best: doctor-patient interactions, diagnosis, and treatment. Phreesia’s software as a service (SaaS) is also pushing the power of self-service to improve the patient experience, as it enables patients to complete their registration on their home computer or mobile device before they arrive at a clinic. Patients can also pay copays and outstanding balances through their mobile device or Phreesia hardware in the waiting room.

For over 10 years, healthcare providers all over the country have integrated Phreesia’s software into their electronic medical record/practice management systems. Today, Phreesia manages intake for over 50 million patients annually and has checked in more than 15% of the U.S. population. On the payment side alone, Phreesia processes tens of millions of transactions per year, significantly streamlining healthcare organizations’ workflow. In addition to improving efficiency and profitability, Phreesia also uses its database to administer clinical screeners to patients and alert clinical staff about the need for a test or treatment.

Diagnosis and Treatment

Phreesia was built on relational databases, but in recent years the volume and variety of patient data being managed on some parts of the platform led the company to look for alternatives. Certain workloads – particularly the patient intake information – were posing problems because of the wide variety of formats and data types that would need to be processed. Alongside that variety, there was another challenge: volume.

Securely storing and managing several million insurance claims for every major insurer is no small feat. To get there, Phreesia migrated its eligibility application away from relational databases and on to MongoDB, to take advantage of the flexibility of the document model to better handle this rapidly expanding volume and variety of data.

To ensure the migration process was smooth and no downtime affected customers, Phreesia called on MongoDB’s consulting engineers. The professional services team helped Phreesia avoid common pitfalls and equipped them with best practices from the start of their journey with MongoDB, helping the developers focus on building value for patients and not wrangling infrastructure.

Now running on MongoDB 3.4, Phreesia’s eligibility application covers the gamut of the ‘run-up appointment’ processes, which allows patients to proactively pre-register, manage copayments, and ensure all their personal contact, medical history, and insurance information is accurate and up to date. This information is seamlessly pushed to their physicians quickly and in advance of the doctor visit. Importantly, insurance providers are contacted directly to make certain all patient information is accurate and current.

Reliability, Security, Scalability – With a Lower Total Cost

Performance and ease of development aren’t enough in the healthcare sector. Security and regulatory requirements are always the first priority. To address those concerns and ensure the SaaS platform had best-in-class data layer security, Phreesia used MongoDB’s encrypted storage engine, available with MongoDB Enterprise Advanced. This native encryption, along with the other advanced security features of MongoDB Enterprise Advanced, help Phreesia meet the regulatory requirements of HIPAA (Health Insurance Portability and Accountability Act of 1996) and PCI DSS (Payment Card Industry Data Security Standards). To monitor, automate, and back up the deployment, as well as support disaster recovery, Phreesia relies on MongoDB’s cloud-hosted management tool, Cloud Manager.

Dan Nathan, VP of Engineering at Phreesia, said: “Downtime or data loss isn’t an option when you’re dealing with this kind of information. MongoDB has given us the confidence that our customers have secure, always-on, and reliable access to data.”

“MongoDB does what we need it to do by scaling independently of the application, no matter how many millions of patients we’re processing,” added Nathan. “The fantastic thing is we get this scale and flexibility, all at a lower total cost of ownership.”

Phreesia manages the administrative tasks of patient intake so medical professionals can focus on the important stuff – managing the health and wellbeing of their patients. In much the same way, MongoDB’s document data model is helping Phreesia developers focus on what’s most important – building new applications and features so they can continue delivering critical services for millions of patients and practitioners. Phreesia is a clear and deserving winner in the Scale category for the MongoDB Innovation Awards 2017.

Orbita Engage: Optimizing in-home healthcare with real-time data

The annual MongoDB Innovation Awards recognize those organizations and individuals that took a giant idea and made a tangible impact in the world. The winners will be honored at MongoDB World 2017, in Chicago on June 20 and 21.

Orbita, Inc. is the 2017 winner of the MongoDB Innovation Award: Internet of Things category. We spoke to Bill Rogers, CEO of Orbita, to learn more about his business and how they are utilizing MongoDB to power their innovation.

Tell us about Orbita, Inc.

Orbita is an innovator of connected home healthcare solutions that dramatically increase patient engagement, optimize care coordination, improve outcomes, and reduce costs.

Orbita Voice™ is the first healthcare-centric solution designed to easily create and manage intelligent, conversational skills for voice-powered assistants such as Amazon Echo and Google Home. Healthcare professionals, including clinicians and non-clinicians, can readily use Orbita’s intuitive tools to create interactive voice experiences to survey patients, provide medical education, share care activity reminders, monitor compliance and more. Intelligent, conversational interfaces can bring significant value to patients, providers and caregivers.

Orbita Engage™ is a first-of-its-kind IoT platform built from the ground up for comprehensive care management. Organizations use this SaaS solution to create, deploy and manage HIPAA-compliant applications that combine data from wearables and other sensor-embedded devices -- including conversational experiences such as those created with Orbita Voice™ -- into collaborative care experiences. Orbita Engage provides unique user experiences optimized for needs of clinical and non-clinical professionals, patients and family members. Rule-based triggers and visualizations empower rapid intervention based on real-time data and insights. Built-in analytics enable tracking of user engagement and response to user input.

According to researchers at Gartner, by 2020, the average person will have more conversations with bots than with their spouse. Gartner analysts also say they are seeing more progress in voice technology in the last 30 months than over the past 30 years.

While change has come quickly in certain markets, the healthcare industry has been slower to adopt information-technology-driven approaches for patient care, including voice-first solutions. When leveraged within a care management platform, digital voice assistants have the potential to substantially impact the Triple Aim of healthcare: improve patient satisfaction, health, and healthcare affordability. They improve patient satisfaction through more natural experiences that eliminate the “monitoring” stigma of other connected healthcare solutions. They improve health outcomes by ensuring patients are engaged in their care, while providing data-driven insights for optimizing provider and caregiver response in moments of need and intervention. And they reduce costs by keeping patients at home and away from costly clinical care settings. Voice assistants also meet the unique needs of patients who cannot use mobile, tablet or PC devices.

Orbita, now just over two years old, was founded by individuals, myself included, who were frustrated with navigating the healthcare system, both as patients and as caretakers for family members. We asked ourselves, ‘How can we make information technology, especially IoT-focused technology, more meaningful in healthcare?’ A basic requirement, we believe, is to make it easy for healthcare professionals, patients, and family members to create and use applications – whether Web, mobile, or voice. And what’s necessary for this? A scalable, secure and flexible database environment.

How does it work?

Our IoT platform is used to engage patients in many ways – including voice. Our solutions are used to help patients get healthy and stay as healthy as possible by receiving the right care at the right time in the home, thereby reducing the need for more expensive care settings.

Think about this — normally, if someone told their Amazon Echo what their blood pressure was, the device would likely respond back saying that it does not understand the request. However, if a patient utilized an Orbita-developed conversational interface within the Orbita Engage platform, not only would the information be recognized and captured, it would also be pushed to an electronic health record system and sent to a health care professional as needed.

Orbita’s solutions support healthcare professionals with powerful analytics that track treatments for multiple conditions and provide guidance through data-driven insights and visualizations, all based on real-time data and rule-based triggers. In the home, the patient experience, along with the experience of family, friends, and home care agents, is improved.

Why did you select MongoDB as your database?

When building Orbita Engage, we needed a system that could easily collect and understand data from a variety of different applications, users, and organizations. Originally, we tried to build our application on Microsoft SQL Server. However, we quickly found that we did not have the flexibility to dynamically adapt to the data coming in. We turned to MongoDB knowing that it would be able to ingest massive volumes of IoT data and that its dynamic schema would allow us to easily adjust to new data requirements. While we cannot make an apples to apples comparison based on hardware, Orbita saw a 300% improvement in both reads and writes using MongoDB instead of Microsoft SQL Server. Additionally, we liked that we could scale horizontally using MongoDB versus more expensive vertical scaling with SQL Server.

Another consideration was security. Because we are dealing with sensitive patient data, we needed a system that would allow us to iterate quickly while ensuring HIPAA compliance. MongoDB Enterprise Advanced and MongoDB Ops Manager provided us with the security features and oversight we needed, such as network encryption, encryption of data at rest, and auditing.

What’s next for Orbita?

Orbita is growing exponentially. As we do, we have no qualms about our database. I’m as confident in MongoDB today as when we first started. As we continue to grow, we will ensure we are running on the most up-to-date versions of MongoDB. When we first updated to MongoDB 3.4, we saw a 50% improvement in performance for some of our queries, and the database keeps getting better. This is key when developing a platform that drives improvements in healthcare.

Delivering Thoughtfulness at Scale Using MongoDB Atlas & AWS

On the third floor of a pre-war building in Manhattan’s Chelsea neighborhood, you might not expect to stumble upon a fleet of hundreds of handwriting robots. However, in the offices of Bond, that’s exactly what you’ll find. Bond began in 2013 as a gifting company, adorning each of their gifts with a handwritten note. It soon became clear that the note (and not the gift) would be the kickstart to Bond’s success.

Bond’s notes are generated with proprietary machine learning algorithms that mimic the way we write letters. The team examines the way different letters of the alphabet relate to each other and recreate that effect using NodeJS and their purpose-built robotic fleet. It’s one of the few companies where you’ll find calligraphers sitting alongside software engineers.

Selecting MongoDB over MySQL

While novelty may be part of the reason Bond’s notes catch the attention of millions of senders and recipients across the world, the company’s mission is more elegant: to equip anyone with the technology to be more thoughtful to the important people in their lives.

This mission resonated with thousands of new Bond customers, who quickly pushed the limits of Bond’s existing technical infrastructure. Originally built on MySQL running in Amazon Relational Database Service (RDS), the platform through which customers create and order notes was seeing upwards of 1,000 read operations per second. This read workload came at the expense of write consistency. The business was scaling exponentially but their database wasn’t keeping pace.

Before long, the engineering team was spending more cycles troubleshooting issues with the datastore rather than building out the core product offering.

Bond’s CTO began evaluating other options with a particular focus on NoSQL databases for their horizontal scalability. However, the team quickly realized that most NoSQL databases weren’t ready for primetime—they either lacked the required querying capabilities or were too infrastructure-intensive for their rapidly-growing requirements.

MongoDB was ultimately selected for its robust ecosystem, expressive query language, and scalability.

Migrating to MongoDB

Initially, Bond chose to continue to route write operations to MySQL and pass them to a hosted MongoDB instance where the data could be read at a much higher frequency. However, the team has since migrated completely to MongoDB as their database of record.

Ensuring a more stable IOPS load enabled the platform to scale, and therefore allowed Bond to process more orders. In the 6 months after migrating to MongoDB, Bond fulfilled twice as many orders than in the previous 2 years on MySQL.

Throughout the process, the team also transitioned from working with PHP to building predominantly in Node with Python for machine learning.

Having used a managed service on AWS for MySQL, Bond's team was eager to hand over the day-to-day management of the database so they turned to Compose.io, a third party MongoDB service provider. While offloading their MongoDB management to a Compose-hosted deployment on AWS enabled the team to return focus to the consumer-facing portions of their app, it became apparent that the lack of encryption and features in the most recent releases of MongoDB were becoming a security and operational hurdle.

Finding MongoDB Atlas

Prompted by their need for end-to-end encryption and the upcoming support for the Decimal 128 data type in MongoDB 3.4, Bond began migrating their data from Compose to MongoDB Atlas shortly after its debut in the summer of 2016. MongoDB Atlas exposed all of the latest functionality of the underlying database, allowing Bond’s technology to not only keep pace with their rapidly-growing business, but to also accelerate to the point where innovation is now driving their business growth.

The team has since built a machine data analytics platform to understand and optimize the performance of their robotic fleet, allowing them to fulfill more orders with the same proprietary infrastructure.

Using the Connector for Apache Spark, Bond is also using machine learning to extract usage data from MongoDB to anticipate the needs of their many types of customers.

To see Bond in action, watch our video with Chief Product Officer, Sam Broe:

Customer Testimonial: Bond



Leaf in the Wild: Appsee Shapes the Mobile Revolution with Real-Time Analytics Service Powered by MongoDB

20 billion documents, 15TB of data scaled across a MongoDB cluster in the cloud. New apps each adding 1 billion+ data points per month

With mobile now accounting for more than 90% of all time spent online in some countries, delivering rich app experiences is essential. Appsee is a new generation of mobile analytics company providing business owners with deep insights into user behavior, enabling them to increase engagement, conversion, and monetization. Customers include eBay, AVIS, Virgin, Samsung, Argos, Upwork, and many more. Appsee is also featured in Google’s Fabric platform. Appsee relies on MongoDB to ingest the firehose of time-series session data collected from its customers’ mobile apps, and then makes sense of it all.

I met with Yoni Douek, CTO and co-founder of Appsee, to learn more.

Can you start by telling us a little bit about your company?

Appsee is a real-time mobile app analytics platform that provides what we call “qualitative analytics”. Our goal is to help companies understand exactly how users are using their app with a very deep set of tools, so that they can improve their app. Traditional analytics help you see numbers – we also provide reasons behind these numbers.

One of Appsee's key qualitative tools is session recordings and replay, which enable our customers to obtain the most accurate and complete picture of each user's in-app experience. We also offer touch heatmaps, which highlight exactly where users are interacting with each screen, and where users are encountering any usability issues. Our platform also provides real-time analytics, user flows, UX insights, crash statistics, and many more insights that help companies in optimizing their app.

Appsee session replay
Figure 1: Appsee session replay provides developers with deep customer experience insights

Please describe how you’re using MongoDB

Our service has two distinct data storage requirements, both of which are powered by MongoDB.

Session database: storing activity from each user as they interact with the mobile application. Session data is captured as a time-series data stream into MongoDB, including which screens they visit, for how long, what they are tapping or swiping, crashes, and so on. This raw session data is modeled in a single document, typically around 6KB in size. Complete user sessions can be reconstructed from this single MongoDB document, allowing mobile app professionals to watch the session as a video, and to optimize their app experiences.

Real-time mobile analytics database: session data is aggregated and stored in MongoDB to provide in-depth analysis of user behavior. Through 50+ charts and graphs, app owners can track a range of critical metrics including total installs, app version adoption, conversion funnels and cohorts, user flows, crash analytics, average session times, user retention rates, and more.

We make extensive use of MongoDB’s aggregation pipeline, relying on it for matching, grouping, projecting, and sorting of raw session data, combined with the various accumulator and array manipulation expressions to transform and analyze data. MongoDB’s secondary indexes allow us to efficiently access data by any dimension and query pattern. We typically enforce three or four secondary indexes on each collection.

Appsee dashboard
Figure 2: Appsee dashboard provides qualitative analytics to mobile app owners

Why did you select MongoDB?

When we began developing the Appsee service, we had a long list of requirements for the database:

  • Elastic scalability, with almost infinite capacity for user and data growth
  • High insert performance required for time-series data, coupled with low latency read performance for analytics
  • Reliable data storage so we would never lose a single user session
  • Flexible data model so we could persist highly complex, rapidly changing data generated by new generations of mobile apps
  • Developer ease-of-use, to allow us to maximize the productivity of our team, and shorten time to market. This was especially important in the early days of the company, as at the time, we only had two developers
  • Support for rich, in-database analytics so we could deliver real-insights to app owners, without having to move data into dedicated analytics nodes or data warehousing infrastructure.

As we came from a relational database background, we initially thought about MySQL. But the restrictive data model imposed by its relational schema, and inability to scale writes beyond a single node made us realize it wouldn’t meet our needs.

We reviewed a whole host of NoSQL alternatives, and it soon became clear that the best multi-purpose database that met all of our requirements was MongoDB. Its ability to handle high velocity streams of time series data, and analyze it in place was key. And the database was backed by a vibrant and helpful community, with mature documentation that would reduce our learning curve.

Can you tell us what your MongoDB deployment looks like?

We currently run a 5-shard MongoDB cluster, with a 3-node replica set provisioned for each shard, providing self-healing recovery. We run on top of AWS, with CPU-optimized Linux-based instances.

We are on MongoDB 3.2, using the Python and C# drivers, with plans to upgrade to the latest MongoDB 3.4 release later in the year. This will help us take advantage of parallel chunk migrations for faster cluster balancing as we continue to elastically scale-out.

Can you share how MongoDB is performing for you?

MongoDB is currently storing 20 billion documents, amounting to 15TB of data, which we expect to double over the next 12 months. The cluster is currently handling around 50,000 operations per second, with a 50:50 split between reads and inserts. Through our load testing, we know we can support 2x growth on the same hardware footprint.

Can you share best practices for scaling?

My top recommendation would be to shard before you actually need to – this will ensure you have plenty of capacity to respond to sudden growth requirements when you need to. Don’t leave sharding until you are close to maximum utilization on your current setup.

To put our scale into context, every app that goes live with Appsee can send us 1 billion+ data points per month as soon it launches. Every few weeks we run a load test that simulates 2x of the data we are currently processing. From those tests, we adapt our shards, collections and servers to be able to handle that doubling in load.

How do you monitor your cluster?

We are using Cloud Manager to monitor MongoDB, and our own monitoring system based on Grafana, Telegraf, and Kapacitor for the rest of the application stack.

Appsee heat maps
Figure 3: Appsee heat maps enable app designed to visualize complex data sets

How are you measuring the impact of MongoDB on your business?

Speed to market, application functionality, customer experience, and platform efficiency.

  • We can build new features and functionality faster with MongoDB. When we hire new developers, MongoDB University training and documentation gets them productive within days.
  • MongoDB simplifies our data architecture. It is a truly multi-purpose platform – supporting high speed data ingest of time-series data, coupled the ability to perform rich analytics against that data, without having to use multiple technologies.
  • Our service is able to sustain high uptime. Using MongoDB’s distributed, self-healing replica set architecture, we deploy across AWS availability zones and regions for fault tolerance and zero downtime upgrades.
  • Each generation of MongoDB brings greater platform efficiency. For example, upgrading to the WiredTiger storage engine cut our storage consumption by 30% overnight.

MongoDB development is open and collaborative – giving us the opportunity to help shape the product. Through the MongoDB Jira project, we engage directly with the MongoDB engineering team, filing feature requests and bug reports. It is as though MongoDB engineers are an extension of our team!

Yoni, thanks for taking the time to share your experience with the company

To learn more about best practices for deploying and running MongoDB on AWS, download our guide.