Why Testim Left Google Firebase
Over the last three months we have migrated our service database from Firebase to MongoDB. In this post we will explore the reasons why we initially selected Firebase, the challenges we experienced as our needs evolved, and the reasons why we decided to make the transition.
How Open Listings is Using MongoDB Atlas to Make Buying Your Dream Home a Reality
This is a guest post by Alex Farrill, Co-Founder and CTO, OpenListings.
At Open Listings, our mission is to make buying a home simple and more affordable.
For many people, buying a home can be a frustrating, slow, and expensive experience. This was the experience my wife and I had when we bought our first home in Brooklyn, NY a few years ago and discovered how antiquated the process seemed. Unlike many other industries, technology just hadn’t made home buying easier—we actually had to use a fax machine.
My co-founders and I looked to the best purchasing experience on the internet: Amazon. You can press a button and get a package as soon as an hour later; and because it's efficient, you get a great price. We knew that there had to be a way to use technology to adapt this process for the real estate market to create the all-in-one homebuying app—one that’s so efficient we could give buyers back 50% of their agent fees. So, that’s when we started Open Listings.
To date, we’ve helped thousands of customers buy over $500 million worth of homes and save over $4 million in unnecessary agent fees.
Because of today’s competitive real estate market, our buyers need speed, reliability, and an individualized experience tailored to their unique needs. MongoDB helps us behind the scenes to provide our customers with the home buying experience they deserve.
Power Behind the Scenes
With over 2.5 million home listings, 300+ home tours a week, and an average of 2 home closings per day on the Open Listings platform, MongoDB has played a big role in our ability to ship what matters most to our buyers.
In the past, I’ve had a lot of experience running MySQL and other relational databases. At my last company, an ad-tech company where I was Chief Data Scientist, we built a real-time bidding platform that handled hundreds of thousands of ad requests per second, and the results of those requests—viewer actions, clicks, etc.—would all go into MongoDB. That translated to tons and tons of writes with upserts.
I’d seen the benefits of using MongoDB to handle native data types like arrays, hashes, and embedded documents. Based on MongoDB’s ability to manage the massive amounts of data we were throwing at it then, I knew it could handle the scale we needed with Open Listings. Now, we rely heavily on MongoDB Atlas, MongoDB’s Database as a Service.
We’re running on AWS, using high memory EC2 instances, giving us more performance and storage. It allows us to keep more indexes in memory and prevent our most common queries from hitting the disk.
For our dataset, we’ve needed to upgrade to larger instance sizes and MongoDB Atlas has made scaling up much simpler. If we were using AWS alone, it wouldn’t be as simple as clicking a button. But now, it’s a process we can do in a few minutes rather than over the course of a few days.
Even more importantly to our buyers, we can do it with no service downtime. We always think first about our customers and their experience. By reducing the upgrade process to minutes instead of days, we’re able to reroute those periods of administrative work into time spent shipping features that will make our platform better.
Expanding faster with Atlas
Going forward, we’re looking to use Atlas to enable microservices for property tour requests on the Open Listings platform. Much like booking a ride through a ridesharing program, we allow buyers to book private home tours by connecting them with an agent close to the property. Since the system doesn’t necessarily need access to every single collection that’s in our database, it makes sense to segment this out using microservices.
That also helps with reliability. By separating our platform out into different services, we can minimize risk for the rest of our application. If we have an outage with our showing requests service, that won't affect our homepage, for example. One failure isn’t going to bring everything else down.
Atlas is also allowing us to expand faster. We just launched Open Listings in Washington state, bringing our entire service online there in a matter of weeks. Part of that process was plugging into the Seattle-area multiple listing service (MLS), which includes over a million properties that have been listed for sale over the years. Normally, we add about 7,000 properties to the system per day —using fully managed MongoDB allowed us to accommodate write workloads an order of magnitude greater over the same time period.
MongoDB Atlas was crucial to making this possible in a short time frame. First, we were able to scale up the database to a larger instance size with a few clicks, instead of the time-consuming process of bringing up new EC2 servers and migrating manually via a rolling upgrade process across our replica set. We didn't have to create a migration plan with an in-house DBA or review our backup strategy. All of that was handled for us in Atlas.
Second, because of MongoDB's flexible data model, we were easily able to accommodate the nuances of this particular MLS's property data without having to go back through all our existing data and painstakingly adjust the schema. Adding additional fields to our property records was as simple as writing it as we processed the new data.
Finally, writing this amount of data is well within the bounds of what MongoDB can handle. WiredTiger's strategy of flushing writes to disk periodically is very performant and is a huge asset for us as we continue to grow.
All in all, we were able to complete the Seattle MLS integration in only two weeks, which allowed us to bring our superior service and 50% commission refund to our next planned state ahead of schedule.
Helping Us Connect the Dots
A few months ago, we joined MongoDB Startup Accelerator. One of the best advantages of the six-month accelerator is the priority support that we get. Early on, we migrated part of our existing MongoDB infrastructure to Atlas for one of our microservices. The Atlas technical services team was incredible in assisting us with migrating this data without downtime, and we were able to demo the new setup in a couple of days.
Getting Atlas credits as part of the program has also been terrific. When you’re a startup, it’s nice to be able to test drive a platform without any financial commitment. It let us defer thinking about the expenses and just focus on figuring out if the fit was right. The credits allowed us to get moving with a great platform at a time when we didn’t have a lot of resources.
Do What Matters
We’ve built Open Listings to be the best way to buy a home. MongoDB helps us move faster, and ship features faster so we can give our customers what they need. The problem is that many things—like data migration or scaling your database—still need to be done, but the bright side is that we’re able to get those out of the way because of MongoDB Atlas.
Because we’re able to be so efficient, we can refund our customers 50% of their buying agent fees at close. This is real savings coming back to people when they need it the most. Some use it to renovate their houses, but for others, it can mean the difference between getting the home of their dreams or settling on something that is “good enough.”
The difference isn't just in the cost savings, though. Open Listings offers the best all around customer experience because we focus closely on the customer's needs. We don't waste time sending faxes and we don't waste time on manual migrations either.
Want to get started? Try building a sample app with Atlas on the free tier.
How MongoDB Atlas Helps TripCloud Move at Startup Speed
This is a guest post by Nag Palavalli, CEO and Co-Founder, TripCloud.
I love to travel, but while working as a consultant and bouncing from city to city, I noticed a gap in the business travel market. Businesses are starting to spend tremendous amounts of money on travel, but with no real way to track and manage those expenses, many companies are unnecessarily bleeding cash. This can have a huge impact on a business' bottom line, and for small or mid-sized businesses, that cash can have a dramatic impact.
That’s why I created TripCloud, a travel and expense management platform for small and midsize businesses in the US. We’ll help you book travel, manage expenses, and understand exactly how that money is being spent.
At the core, I started the company because expenses are something that everybody universally hates, and hate is a great motivator for change.
Supporting Our Team
At TripCloud, we’re a small team of four. Given that our time is limited, we need to work on the projects that will have a big impact on our business. As a result, we wanted a database-as-a-service to help us manage our data platform and associated ops work so we can focus on building and managing the app.
We started Tripcloud on mLab, but we were looking for a database-as-a-service with the best in class support, performance, and security for our customers. Now that we’re using the MongoDB Atlas service, we get unparalleled, reliable performance, which is incredibly important because today’s users are accustomed to almost instantaneous response times. It doesn’t matter if you have the best product; if it’s slow or crashes, people won’t use it. MongoDB Atlas allows me to run a report every Friday reviewing all of our cloud resources. I am able to tweak our system every week to ensure it’s running the best it can—looking for any anomalies or alerts. It only takes me 15 minutes to do this, but it’s incredibly impactful. Having a platform optimized to get these rapid insights on our database operations is a huge differentiator for MongoDB Atlas.
With Atlas, all of our data is automatically replicated three times and geographically distributed, so our platform is always on. We also invest in Atlas’s fully managed backups so we can move fast and minimize the risk of failure. It’s crucial because when you’re a small team moving as fast as we are, the chances of breaking something or running into other challenges are higher. At the same time, we’re dealing with mission-critical information and can’t afford to lose any data. It’s comforting knowing that if we do break stuff, we’ve got the ability to restore to any point in time. If I tried to do all this on my own, it would be a full-time job just managing this infrastructure.
Accelerating Our Progress
Another offering from MongoDB that’s helped us build our company is the Startup Accelerator. It’s a six month program that gives startups Atlas credits, priority support, as well as access to their community network. Being in the accelerator has been like a gift for us. It’s given us the chance to get closer to a company that played such a huge role in our business.
I’ve been able to leverage the expertise of that network through Slack, where I’m connected directly with the people building MongoDB. We were building a marketing site, BusinessTravelTools.com, and needed to find the answer to a problem that wasn’t documented online. Within 15 minutes of talking to them, I had solved the issue. I wouldn’t have been able to solve the problem that quickly without this connection, which was crucial since I needed to deploy that night.
These connections mean everything. Especially in the beginning when no one in the industry knows who you are. If you can find a way to get visibility and gain these connections, you’re going to be able to get a lot more important work done. It’s really important and beneficial to us that AWS and MongoDB are working together to deliver the best in class database as a service in the cloud through the Startup Accelerator program.
The Right Tools
With the speed and reliability that we get from MongoDB Atlas, we can now solve the big problems. We know our users will have a consistently positive experience, and we don’t have to worry if our database will scale. We only have to focus on building the future of business travel management. Distractions are momentum killers, and in the startup world, speed is king. Just like how MongoDB is removing obstacles for good work, that’s what we’re doing at TripCloud.
The tools you pick will decide how fast you can move. Make sure you’re choosing the best tools for your company—your success is riding on it.
How SteppeChange used MongoDB Zones to build a Global Mobile Customer Engagement Platform for 220 million users in the hybrid cloud
Using MongoDB, SteppeChange was able to shave approximately six months off of the development schedule of their application.
SteppeChange is a big data analytics technology firm that designs and implements client-tailored, fast-to-market data science and technology solutions. They work with clients around the world to find innovative answers to challenging problems and allocate analytical effort where it will create the most value.
Gregory Rayzman, CTO and Chief Data Architect at SteppeChange shares how and why the company relies on MongoDB for a variety of solutions, including an extendable mobile customer engagement platform for 220 million global users.
A Complex Task
A global technology company hired us to build a mobile customer engagement platform to be used by mobile operators around the world. The second we were assigned the project, we knew we were up for a challenge as different countries have vastly different data management laws. While one country might require that all data be encrypted at rest, another might require that all data is stored within their country boundaries.
Our goal was to build a platform with a single code-base, all while balancing multiple data management requirements and meeting the needs of an expected user base of 220 million subscribers globally.
Finding a system that would meet varying data management requirements governed by multiple countries was of utmost importance when evaluating database options. After surveying a number of different options including relational players like MySQL and PostgreSQL, and NoSQL players like Cassandra and Couchbase, we quickly realized that MongoDB Enterprise Advanced provided the flexibility, scalability, and agility we needed.
The Zones feature in MongoDB is critical to our application. With it, we can break data from MongoDB collections into multiple shards and assign each shard to a zone associated with a specific geographic location. Zones are part of the same cluster and can be queried globally, but the data resides at sovereign locations where local laws prevail. Not only is latency reduced with MongoDB Zones, but we are also able to scale and grow each zone independently of others.
MongoDB Cloud Manager was also a major asset in setting up and monitoring our MongoDB deployment. It allows us to visualize the ongoing state and status of all systems, troubleshoot issues, and easily perform point-in-time restores.
With MongoDB Zones, we separate user data for regulatory purposes and keep it under local jurisdiction. More specifically, user data resides in data centers physically located in the appropriate country, so that the application’s access to user data complies with local regulation boundaries.
We designed and set up a multi-sharded MongoDB cluster consisting of three Zones. Each Shard has three voting replicas, in addition to hidden non-voting replicas for reporting purposes, allowing the system to spread the load based on node functionality. We do this so the data pertinent to a specific jurisdiction is deployed at data centers inside the respective jurisdictional boundaries, while data that is not subject to the same regulations is deployed on AWS.
For MongoDB Zones deployed on AWS, we distribute replica set nodes across multiple AWS Availability Zones (AZs) to increase application availability and protect against AWS outages. In addition, we leverage a similar design for configuration servers — they reside in multiple AZs as well.
To guarantee compliance with security and privacy standards, we also leverage MongoDB’s native encryption. To satisfy regulations around data access, we use the auditing framework to record and log all administrative and non-administrative actions performed against the database.
Accelerated Delivery with MongoDB
With MongoDB, we were able to quickly bring our application to market by shaving approximately six months off of our development schedule. Our team has been able to take advantage of MongoDB BSON based document storage, Binary Serializable JSON objects. It was a perfect native match to the JSON-based underlying data structure used in our app, which provides an agile approach to adding new features rapidly. We were also able to simplify our data management and remove the complexities of data migration, increasing developer productivity and allowing our engineering team to concentrate on the task at hand.
As we work towards the future, we are looking to expand our use of other features in MongoDB, like the geospatial capabilities — such as geo-fencing and geo-based offer management — and add them to our Mobile Customer Engagement Platform.
eBay: Building Mission-Critical Multi-Data Center Applications with MongoDB
As a top 10 global retail brand with 170+ million active buyers and 1 billion live listings across 190 markets around the world, eBay cannot afford systems downtime. This is why the company relies on MongoDB as one of its core enterprise data platform standards, powering multiple, customer-facing applications that run ebay.com.
At this year’s MongoDB World conference, Feng Qu, eBay’s lead NoSQL DBA, presented Practical Design Patterns for Resilient Applications – a set of architectural blueprints his team has developed to support enterprise-class MongoDB deployments.
Mr. Qu began his session discussing how the concept of availability has changed over the years. In the past, it was acceptable for sites to take scheduled downtime for weekly maintenance events. With the global nature of today’s services, neither users, or the business, are quite so accepting! In addition, most organizations now build out their services on commodity hardware platforms, rather than the exotic Sun Solaris / Sparc servers of yesteryear. While commodity hardware is much less costly, it also fails much more regularly. Both of these factors radically alter how engineering teams consider availability, and has led eBay to create its “Resiliency Design Patterns” to institute database best practices that maximize Mean Time To Failure (MTTF) and minimize Mean Time To Recovery (MTTR).
To build their apps, eBay developers can choose from five corporate-approved database standards. Alongside MongoDB, teams also have the option to use Oracle or MySQL relational databases, and two NoSQL options. Mr. Qu’s DBA team provide guidance on the appropriate database choice, qualifying the selection against the application’s data access patterns, user load, data types, and more.
eBay currently runs over 3,000 non-relational database instances powering a range of applications, managing multiple petabytes of data between them. In the past Oracle was the System of Record, while the non-relational databases handled transient data used in “systems of engagement”. However, the non-relational database landscape has matured. With consistent, point-in-time backup and recovery, MongoDB now also serves System of Record use cases at eBay.
While all of eBay’s non-relational database choices offer built in resilience to failure, they make different design tradeoffs that can impact application behavior. The DBA team assesses these differences across six dimensions: availability, consistency, durability, recoverability, scalability, and performance. For example, those NoSQL databases using peer-to-peer, masterless designs have expensive data repair and rebalancing processes that must be initiated following a node failure. This rebalancing process impacts both application throughput and latency, and can cause connection stacking as clients wait for recovery, which can lead to application downtime. To mitigate these affects, eBay has had to layer an application-level sharding solution, originally developed for its Oracle estate, on top of those masterless databases. This approach enables the DBA team to divide larger clusters into a series of sub-clusters, which isolates rebalancing overhead to a smaller set of nodes, impacting just a subset of queries. It is against these different types of database behaviors that the eBay DBA team builds its Resiliency Design Patterns.
Mr. Qu presented eBay’s standard “MongoDB Resilience Design Pattern”, as shown in Figure 1 below.Figure 1: eBay design pattern for its MongoDB Resilience Architecture. (Image courtesy of eBay’s MongoDB World presentation).
In this design pattern, a 7-node MongoDB replica set is distributed across eBay’s three US data centers. This pattern ensures that in the event of the primary data center failing, the database cluster can maintain availability by establishing a quorum between remaining data centers. MongoDB’s replica set members can be assigned election priorities that control which secondary members are considered as candidates for promotion in the event of a primary failure. For example, the nodes local to DC1 are prioritized for election if the primary replica set member fails. Only if the entire DC1 suffers an outage are the replica set members in DC2 considered for election, with the new primary member selected on the basis of which node has committed the most recent write operations. This design pattern can be extended by using MongoDB’s majority write concern to enable writes that are durable across data centers.
The standard MongoDB design pattern is used as the basis for eBay’s “Read Intensive / Highly Available Read Pattern” discussed in the presentation, which is used to power the eBay product catalog. For the catalog workload, the MongoDB replica set is scaled out to 50 members, providing massive data distribution for both read scalability and resilience.
For more write-intensive workloads, eBay has developed its “Extreme High Read / Write Pattern”, which distributes a sharded MongoDB cluster across its US data centers.Figure 2: eBay design pattern for the MongoDB Extreme High Read / Write Pattern. (Image courtesy of eBay’s MongoDB World presentation).
Again, eBay developers can configure this design pattern with specific MongoDB write and read concerns to tune the levels of durability and consistency that best meet the needs of different applications.
Mr. Qu noted that with recent product enhancements, MongoDB is being deployed to serve a greater range of application needs:
- The addition of zone sharding to MongoDB 3.4 now enables eBay to serve applications that demand distributed, always-on write availability across multiple data centers.
- Retryable writes, targeted for the forthcoming MongoDB 3.6 release, will allow eBay to reduce application-side exception handling code.
Review the recording of Feng Qu’s presentation at MongoDB World to learn more about eBay’s Design Patterns.
Download the MongoDB Multi-Data Center Deployments guide to get deeper insight into enabling active/active data center deployments and global data distribution with MongoDB.
Methodist Le Bonheur Healthcare: Transforming Hospital Operations and Improving Patient Care with MongoDB
When it comes to healthcare, the patient experience is always a top priority. Organizations must coordinate a variety of services from room cleaning to patient transport fast and efficiently. Methodist Le Bonheur Healthcare (MLH), the largest healthcare system in Memphis, Tennessee, relies on MongoDB to make this happen.
We connected with David Deas, MLH’s Corporate Director, Innovation and Knowledge Analytics, to learn more about their MongoDB-powered hospitality application.
Can you tell us a little bit about your company?
Methodist Le Bonheur Healthcare (MLH) was founded in 1918 and is the largest health care system in Memphis, consisting of six hospitals and dozens of smaller facilities and clinics, with nearly 14,000 employees. Our biggest hospital has over 400 beds and operates at near-capacity on a regular basis.
Why did you start using MongoDB?
Coordinating hospitality on a continuous schedule requires dozens of people to ensure the patient remains as comfortable as possible during their stay. Historically, we have been using a legacy, relational database system to manage the flow of patients in, out, and around the hospital, and it was on its last legs. When the time came for a new license and a software upgrade, we were hit with a $1.6 million price tag, which gave hospital administrators some pause.
Can you explain how Melodi Flow works and how it uses MongoDB?
Melodi Flow gives the central dispatch office a real-time master view of every room, along with the status of room cleaners and patient transporters. Unlike with our legacy relational system, using MongoDB ensured that there was no delay between status changes, requests, and raised issues. And those antiquated pagers carried around by room cleaners and patient transporters? Replaced by brand new iOS devices with custom mobile apps to easily manage the daily workflow.
The real-time capabilities of the new Melodi Flow system are powered by reading updates directly from MongoDB's replica set oplog by connected browsers and mobile clients. The real-time aspect of the new system is a huge win, but an additional, not-so-obvious differentiator is the mountain of data being collected each and every day. Every interaction, from acknowledging reservations, to a completed room cleaning, to fetching a patient for daily treatments is recorded for later analysis.
We are confident in MongoDB Enterprise Advanced. Replica sets give us peace of mind for resilience and performance, and backups are a breeze. Since we designed our schema with reporting in mind, we can spend less time worrying about complicated JOINs and sub-queries, and more time analyzing our data. After only a few weeks of use, we’ve already examined patient complaints, identified peaks, valleys, and bottlenecks in patient flow throughout the day, and created the beginnings for benchmarking employee performance.
What’s next for Melodi Flow?
Our future plans include:
- The creation of single-view dashboards for upper management to get a snapshot view of the facility at any given time (in comparison to recent history)
- More advanced aggregations and analytics to find hidden efficiencies
- Utilizing the MongoDB Connector for BI with Tableau for rich reporting and analysis
- Scaling out the software to the remaining 5 hospitals in the Methodist Le Bonheur system
- Coalescing data from our Electronic Medical Records (EMR) system to give better real-time views into the current state of the hospitals and their patients
MongoDB has been a crucial tool in managing one of the busiest hospitals in Memphis. Its utility will only grow as the Process Improvement & Innovation team continues delivering solutions for improving patient care.
Index the Planet: How MongoDB is powering Bluedot, the world’s most accurate geofencing service
Bluedot Innovation provides a location services platform that enables location-based commerce and social innovation across any industry where mobility and geospatial awareness can add value. With customers like News Corp, Shell and Samsung and a solution on Salesforce Marketing Cloud, common use cases include driving foot traffic and sales in retail stores, increasing sales at restaurants and bars through automated offers, and more.
The following guest post is by Andrew Stirling, Head of Engineering (Infrastructure) at Bluedot Innovation, who highlights how Bluedot is at the forefront of location-based services.
Planet Earth is big. Really, really big. So when Bluedot Innovation was building a new mobile SDK to provide extremely accurate geofencing, with very little overhead in cellphone battery usage, we needed our back-end platform to bear as much of the load as possible. Specifically, we needed a solution that could quickly and scalably report only the nearest geofences, where each could be a different shape and dimension.
We’ve been working with MongoDB for over three years now, and haven’t looked back. After some analysis, we found that MongoDB’s geospatial capabilities fulfilled our requirements perfectly, with fast geospatial index lookup and a variety of geospatial query operators available including $geoWithin, $geoIntersects, and $near / $geoNear. The flexibility granted by the GeoJSON standard means that we aren’t limited to the circle and square geofences that most of our competitors are offering. Instead, we are able to offer polygons that directly fit the contours of a building or outdoor feature. Where our competitors offer geofences no smaller than 50 meters across, we can offer infinitely thin lines drawn across, say, the entrance to a drive-in to detect the arrival of a new customer. Thanks to the flexibility that MongoDB’s native document data structures provide, we aren’t limited to the basic approaches that the incumbent providers offer and are instead able to act fast and disrupt the market.The GeoJSON format allows Bluedot Innovation to accurately index a variety of different geofence shapes.
The implementation of these capabilities is remarkably simple, as long as the database schema complies with established standards. GeoJSON was recently fully formalized in RFC 7946, and its implementation in MongoDB adheres to the specification. We have two stacks that interface with the MongoDB database – a Java/Spring Data stack and a NodeJS/Mongoose stack. Support for geospatial tools in these frameworks is now quite complete, and the implementation process is straightforward.
In our experience, geographic searches in MongoDB are a heavily CPU-bound process. The indexing takes only a little more memory than a standard index, but the search query takes more CPU cycles. This is because performing a search on two-dimensional proximity is not a simple operation by itself, let alone when the lookup needs to be performed on a sphere rather than a flat Cartesian plane. We’ve experimented with a variety of data structures to optimize the lookup, but the naïve one performed well enough that we decided to keep it for the sake of programmatic simplicity. We added a solid cache layer to reduce load on the back-end.
MongoDB has continued to improve on their geospatial queries. The 3.2 release saw some very nice performance improvements which brought down our cost-per-query significantly. We’re excited that the geospatial component will continue to be enhanced and anticipate further expansions in functionality and performance.
In all, we’re very satisfied with our choice of MongoDB for our product’s back-end database, and we look forward to working with the MongoDB platform as it continues to grow.
To learn more about geospatial data and MongoDB, view our on-demand webinar.
Vyopta: Shipping fast and reducing operational costs with MongoDB Atlas
This is a guest post from Brent Mills, Principal Infrastructure Engineer at Vyopta, an analytics company that enterprises use to monitor, support, and optimize their video and unified collaboration systems.
MongoDB has always been very attractive to developers in part due to the ease with which you can set up, manage, and interact with the database. Time-to-market is increasingly important, as businesses require increasing levels of speed and flexibility in how they deliver new products and features to their customers.
But as with any other back-end system, you have to manage monitoring and alerting, the clusters (and all of the idiosyncrasies of every distributed system), backups, encryption, provisioning for scale, etc.; These are all things operations teams have to consider in order to deliver production-ready applications, but they’re also things that can delay innovation velocity and can present operational overhead.
By moving our database to the MongoDB cloud service, we’ve been able to remove much of the operational load, saving us over 15 hours of work per month, enabling us to increase efficiency as we’ve moved apps from development to production.
Our Use Case
Below is a simplified version of our pre-MongoDB Atlas setup. I would imagine this looks familiar to many of you:
First off, there’s nothing wrong with this infrastructure, in fact it’s served us pretty well we’ve experienced very few issues. Even so, we were spending a non-trivial amount of time each month on operational overhead. Troubleshooting backup failures, relocating instances because of a network revamp, spinning up new environments for testing, configuring encryption for SOC2 compliance and additional requirements for customers storing data outside of the United States. All of this maintenance work adds up, slowing us down. We’d rather have our development team pushing out new features, and our DevOps team running our CI/CD process, or supporting new initiatives like our move to a microservices architecture.
MongoDB Atlas to the Rescue
Personally I’ve never been a big fan of managed cloud services. They always seem to be lacking important features, have frustrating usability issues, and/or monetarily punish you in creative ways. MongoDB Atlas, however, was very attractive to us for the following reasons:
- Fully functional MongoDB instance. With the ability to upgrade versions in a few clicks you can always have the newest feature set. It’s also close to identical to what you would deploy on-premise, not some specific “hosted” variation that can’t do x, y, or z.
- Easy backup and restore. Click to enable, edit the schedule in one pane with cost estimation and options for separate daily, weekly, and monthly retention periods. Straight-forward restore with option to download or query data in place, without having to restore the complete backup.
- Integrated metrics and alerts. No more setting up agents on every node. Built in alert integrations with services like PagerDuty.
- Easy scale up/down. Need more RAM? Edit the cluster configuration and select a bigger size. Need more disk space? Drag a slider.
- Easy to provision new clusters. A few mouse clicks or an API call and we can have a cluster with encryption, sharding, and backups. The GUI also shows you a direct cost estimation.
- No nickel and diming. The folks at MongoDB could have easily added extra charges for each cluster modification, encryption, restores, alerts, and various other things to muddy the water, but they did not. You pay for the managed instances and backups you use and that’s it.
- No lock-in. Migrating data in and out of Atlas is extremely simple.
After experimenting with the various features it became pretty obvious this would eliminate the vast majority of our overhead
With Atlas it now takes about 10 minutes to provision a full cluster with monitoring and any other features we need. We can even create a new group, which is sort of like a linked account, and give developers the ability to create their own clusters in an isolated environment without waiting on us.
This is Going to Hurt...Oh Wait...
So with all of this the only potential downside has to be the price, right? It turns out this was the most pleasant surprise of all.
Since our Atlas clusters are running on AWS and the cluster sizes are based on EC2 instances, we can make some rough comparisons on price:
- M30 cluster w/ 8gb RAM and 100gb storage - ~$430/mo
- 3 m4.large AWS instances w/ 8gb RAM and 100gb storage - ~265/mo
This does not include storage cost for backups or data transfer but gives us a good idea. If we’re running on an M30 instance in production, we don’t have to save many man hours to justify a ~$165/month difference. Another thing to note is that most people will over-provision their own clusters because it’s not easy to scale. With Atlas, we can eliminate this waste.
As we continue to build out our continuous deployment systems we keep looking for ways to dynamically provision environments and test for scale. Another thing Atlas offers in this respect is an API that encompasses most of their GUI features. This allows us to programmatically spin up a cluster as part of a test run:
We can also use a Docker container for this, but if we’re trying to replicate our production environment or test performance it can be quite difficult with containers.
The one thing missing from the API at the moment is access to metrics. It would be a killer feature if we could spin up a cluster, run some tests, then include production-like database performance stats in our test results. I know after speaking with the Atlas team, that this is coming in the next few months.
We have been using MongoDB Atlas in production for around 6 months and have been very happy with it. Our migration consisted of stopping MongoDB, running a mongodump, then simply restoring it with mongorestore using the new Atlas connection strings. We were able to schedule a maintenance window to accomplish this but there are also now options for doing a live migration if you require the uptime. Because of the simplicity and minimal effort involved in migrating the data, we were able to see cost savings in the first month. Clear, upfront cost estimates have been especially nice since I seem to spend a lot of my time in convoluted pricing calculators. Hopefully our experience inspires others to to give Atlas a try (which you can do without cost using the MongoDB Atlas free tier).
How DevOps, Microservices, and MongoDB are Making HSBC “Simpler, Better, and Faster”
A digital revolution is happening at HSBC. As one of the world's largest banking and financial services organizations, the company is building global digital solutions that will help customers worldwide to manage their personal and business finances.
The drive for digital innovation comes as part of a series of strategic actions set out by HSBC’s board to adopt technology that makes HSBC “simpler, better and faster”. To achieve that goal HSBC has embraced agile ways of working. Technology teams are championing a DevOps culture and microservices design approach, while harnessing the next generation of technologies like MongoDB – which is helping HSBC to simplify, migrate and manage complex data change.
Data is at the core of HSBC’s innovation. Nick Maybin, Development Manager, and Andrew Matthews, Equities Architect, HSBC, shared some powerful statistics in their “Bye Bye Legacy: Simplifying the Journey” session at this year’s MongoDB World conference. From data assets totaling 56PB in 2014, the HSBC team expects to see these grow to 93PB by the end of this year.
One area of HSBC that is particularly data driven is the trading technology teams. As part of a collaborative initiative the trading teams in London are using MongoDB to build an Operational Data Store for trades to enable, equip and engage trading desks, salespeople and operations teams across the world, rapidly serving up data consistently and reliably from a single source of truth For example, the Operational Data Store will help users access data on complex markets like Equities and Fixed Income. This includes simple instruments like bonds and shares, but also many other types of more intricate products such as futures, forwards, options and derivatives trading.
Previously this data would have been stored in relational databases, but in the quest to better serve users the team are migrating to MongoDB. The reasons align with the board’s goal of finding technology that would make HSBC simpler, better and faster:
- Simpler – Compared to relational tables, MongoDB’s document data model mean it is significantly simpler to model bonds, swaps and equity products. While each of these assets are instruments, they all need to store different attributes – something known as polymorphism. MongoDB’s flexible document data model makes it simple to manage this type of data variability. With a dynamic schema, it is also easier to make changes for new data types or for regulatory reasons. Finally, MongoDB makes life simpler for the Ops part of DevOps by providing high availability, scale and security, right out of the box. HSBC operates multiple data centers around the world. The existing relational databases required manual failover to recover service in the event of an outage – a process that is difficult to configure. MongoDB’s self-healing replica sets largely eliminate the complexity in creating a resilient, always-on database infrastructure.
- Better – HSBC’s Operational Data Store is being built with numerous microservices using native end-to-end JSON data structures that extend from the client, through to the apps, and the market data itself. The flexibility of MongoDB’s data model allows HSBC to manage complex data modeling better, including bitemporal data, providing a way of storing historical data across two separate timelines. Take a look at the slides to learn more about how the HSBC developers model bitemporal data with MongoDB.
- Faster – this is not just faster response times, but a perhaps even more important type of speed: organisational change. The Operational Data Store is demonstrating the value of adopting new technology and development processes. HSBC has a large DevOps Programme underway and application teams are changing their approaches, enabled by more flexible technologies. Shifting from traditional siloed teams, segregated by management hierarchy, into much more cross functional groups with self-sufficient, multi-disciplinary pods. All of which is helping the company move faster with a more entrepreneurial approach. Review the slides to learn more about the roles of technical staff, communicators, and influencers in driving technology change.
HSBC believes the technology ideas of today, will be the intuitive habits of tomorrow. The trading teams’ migration away from relational databases, is another demonstration of how impactful that can be, with MongoDB enabling fast iteration for new applications and helping to support a shift in culture. It’s no surprise HSBC was a clear winner in this year’s MongoDB Innovation Award for Data Driven Business.
To learn more about its journey to a data-driven business, take a look at HSBC’s talk: “Bye Bye Legacy: Simplifying the Journey”
Download our whitepaper to learn more about how microservices are enabling modern application development.
How Alight Solutions (Aon Hewitt) Improved Customer Experience by over 250x with Mainframe Offload and Single View
Fortune 500 companies are increasingly seeking ways to innovate in customer experience, while reducing reliance on legacy technologies. Alight Solutions, formerly part of Aon PLC, was attempting to integrate customer data stored in multiple source systems hosted on its zOS mainframe and in cloud services, while at the same time improving developer productivity, and reducing costs.
Alight’s solution was to build a customer 360 single view platform using MongoDB and Apache Kafka, and was presented by Christopher Roberts, the company’s VP of Enterprise Architecture, during this year’s MongoDB World ‘17 conference. You can view a recording of “Aon: Taking Big Iron to Ludicrous Speed” along with Christopher’s slides at the MongoDB World ‘17 site.
Alight Solutions is the leading provider of outsourced benefits administration, cloud-based HR and financial solutions, serving close to 40 million employees from over 1,400 organizations, including nearly 50% of the Fortune 500.
To meet the demands of the business, the architecture team was tasked with building a cross-platform single customer view to unlock greater data insights and improve application responsiveness, while also reducing mainframe costs. Retrieving customer data from frontend consumer interfaces such as Salesforce, the UPOINT HR portal, and mobile apps meant accessing multiple backend source systems running on a mainframe platform and Workday. Each request added to the ongoing mainframe MIPS costs, and scaling multiple systems to support customer growth was proving difficult. Query latency was typically 0.5 of a second, which negatively impacted customer experience.
As the project was scoped, consideration was given to copying all customer data into Alight’s existing Hadoop system, and serving the single view from there. However, Hadoop didn’t offer the OLTP or low latency demands of the app, so the business needed an alternative approach.
After researching multiple options, the Alight team decided MongoDB was the best choice for its single view database. With its flexible document data model, MongoDB offered the schema flexibility needed to accommodate multi-structured, polymorphic customer data from the source systems. Its expressive query language and secondary indexes enabled business users to access customer data in whichever way they needed. MongoDB’s distributed, self-healing architecture enabled Alight to operationalize the single view, ensuring high scalability and always-on availability with minimal overhead to the ops team.
Application performance was a key driver for the project, and MongoDB delivered. With customer data aggregated into a single document, query latency was reduced from 500 milliseconds to less than 2 milliseconds – a 250x improvement!
Delivering the Single View on MongoDBIn his session, Christopher walked through the three phases of the project, from the initial phase delivering a working solution in less than six months, through to the current phase that provides more granular insights into each customer’s benefit classes. He shared tips and patterns for providing low latency solutions that can be leveraged effectively on legacy mainframe and modern cloud platforms.
In phase 1, data was extracted from the mainframe in batches, transformed in Hadoop, and then loaded into MongoDB. Phase 1 was a vital first step in demonstrating the possibilities of building a single view, but Hadoop’s fragility and reliance on multiple independent technologies complicated the data transform process. In addition, change data capture was slow, and the data merging and matching rules were too simplistic, affecting data quality.
In phase 2, Alight replaced Hadoop with Kafka, enabling modifications to source data to be sent as messages in both batches and in real time streams. Data could then be merged through the application of more sophisticated Master Data Management (MDM) rules, before being loaded into MongoDB, where it was exposed to consuming systems via a REST API.
In its current third phase, Alight engineers are redesigning their data structures to model customer data by benefit class, enabling greater deployment flexibility for scalability and high availability. Data versioning is also being added to track changes to each customer record over time.
Christopher wrapped up his session by talking through the benefits the single view project has delivered to the business:
- Customer experience: faster responsiveness, and ease in delivering new functionality
- Lower costs: fewer calls to the mainframe, and reductions in peak consumption
- Unlocked innovation: greater insight into customer data, and improved agility in delivering new services
To learn more, take a look at the slides and recording.
To get started in customer experience innovation, download our 10-Step Methodology to Building a Single View of Your Business whitepaper.