MongoDB 3.4.0-rc2 is out and is ready for testing. This is the culmination of the 3.3.x development series.
Fixed in this release candidate:
- SERVER-7306 Mongod as windows service should not claim to be 'started' until it is ready to accept connections
- SERVER-18908 Secondaries unable to keep up with primary under WiredTiger
- SERVER-26420 Make internal clients identify themselves in the isMaster handshake
- SERVER-26514 Create command should take idIndex option
- SERVER-26648 Tolerate bad collection metadata produced on version 2.4 or earlier
- SERVER-26652 Invalid definitions in systemd configuration for debian
- WT-1592 Dump detailed cache information via statistics
- WT-2954 Inserting multi-megabyte values can cause large in-memory pages
As always, please let us know of any issues.
-- The MongoDB Team
Microservices Webinar Recap
Recently, we held a webinar discussing microservices, and how two companies, Hudl and UPS i-parcel, leverage MongoDB as the database powering their microservices environment. There have been a number of theoretical and vendor-led discussions about microservices over the past couple of years. We thought it would be of value to share with you real world insights from companies who have actually adopted microservices, as well as answers to questions we received from the audience during the live webinar. Jon Dukulil is the VP of Engineering from Hudl and Yursil Kidwai is the VP of Technology from UPS i-parcel. How are Microservices different from Service Oriented Architectures (SOAs) utilizing SOAP/REST with an Enterprise Service Bus (ESB)? Microservices and SOAs are related in that both approaches distribute applications into individual services. Where they differ though, is the scope of the problem they address today. SOAs aim for flexibility at the enterprise IT level. This can be a complex undertaking as SOAs only work when the underlying services do not need to be modified. Microservices represent an architecture for an individual service, and aim at facilitating continous delivery and parallel development of multiple services. The following graphic highlights some of the differences. One significant difference between SOAs and microservices revolves around the messaging system, which coordinates and synchronizes communication between different services in the application. Enterprise service buses (ESB) emerged as a solution for SOAs because of the need for service integration and a central point of coordination. As ESBs grew in popularity, enterprise vendors packaged more and more software and smarts into the middleware, making it difficult to decouple the different services that relied on the ESB for coordination. Microservices keep the messaging middleware focused on sharing data and events, and enabling more of the intelligence at the endpoints. This makes it easier to decouple and separate individual services. How big should a microservice be? There are many differing opinions about how large a microservice should be, thus it really depends on your application needs. Here is how Hudl and UPS i-parcel approach that question. Jon Dukulil (Hudl) : We determine how big our microservice should be the amount of work that can be completed by a squad. For us, a squad is a small completely autonomous team. It consists of 4 separate functions: product manager, developer, UI designer, and QA. When we are growing headcount we are not thinking of growing larger teams, we are thinking of adding more squads. !(https://webassets.mongodb.com/_com_assets/cms/Microservices_MongoDB_Blog2-a6l74owk23.png) Yursil Kidwai (UPS i-parcel) : For us, we have defined microservice as a single verb (e.g. Billing), and are constantly challenging ourselves on how that verb should be defined. We follow the “two pizza” rule, in which a team should never be larger than what you can feed with two large pizzas. Whatever our “two pizza” team can deliver in one week is what we consider to be the right size for a microservice. Why should I decouple databases in a microservices environment? Can you elaborate on this? One of the core principles behind microservices is strong cohesion (i.e. related code grouped together) and loose coupling (i.e. a change to one service should not require a change to another). With a shared database architecture both these principles are lost. Consumers are tied to a specific technology choice, as well as particular database implementation. Application logic may also be spread among multiple consumers. If a shared piece of information needs to be edited, you might need to change the behavior in multiple places, as well as deploy all those changes. Additionally, in a shared database architecture a catastrophic failure with the infrastructure has the potential to affect multiple microservices and result in a substantial outage. Thus, it is recommended to decouple any shared databases so that each microservice has its own database. Due to the distributed nature of microservices, there are more failure points. Because of all these movable parts in microservices, how do you deal with failures to ensure you meet your SLAs? Jon Dukulil (Hudl) : For us it’s an important point. By keeping services truly separate where they share as little as possible, that definitely helps. You’ll hear people working with microservices talk about “minimizing the blast radius” and that’s what I mean by the separation of services. When one service does have a failure it doesn’t take everything else down with it. Another thing is that when you are building out your microservices architecture, take care of the abstractions that you create. Things in a monolith that used to be a function call are now a network call, so there are many more things that can fail because of that: networks can timeout, network partitions, etc. Our developers are trained to think about what happens if we can’t complete the call. For us, it was also important to find a good circuit breaker framework and we actually wrote our own .NET version of a framework that Netflix built called Hystrix. That has been pretty helpful to isolate points of access between services and stop failures from cascading. Yursil Kidwai (UPS i-parcel) : One of the main approaches we took to deal with failures and dependencies was the choice to go with MongoDB. The advantage for us is MongoDB’s ability to deploy a single replica set across multiple regions. We make sure our deployment strategy always includes multiple regions to create that high availability infrastructure. Our goal is to always be up, and the ability of MongoDB’s replica sets to very quickly recover from failures is key to that. Another approach was around monitoring. We built our own monitoring framework that we are reporting on with Datadog. We have multiple 80 inch TVs displaying dashboards of the health of all our microservices. The dashboards are monitoring the throughput of the microservices on a continual basis, with alerts to our ops team configured if the throughput for a service falls below an acceptable threshold level. Finally, it’s important for the team to be accountable. Developers can’t just write code and not worry about, but they own the code from beginning to end. Thus, it is important for developers to understand the interdependencies between DevOps, testing, and release in order to properly design a service. Why did you choose MongoDB and how does it fit in with your architecture? Jon Dukulil (Hudl) : One, from a scaling perspective, we have been really happy with MongoDB’s scalability. We have many small databases and a couple of very large databases. Our smallest database today is serving up just 9MB of data. This is pretty trivial so we need these small databases to run on cost effective hardware. Our largest database is orders of magnitude larger and is spread over 8 shards. The hardware needs of those different databases are very different, but they are both running on MongoDB. Fast failovers are another big benefit for us. It’s fully automated and it’s really fast. Failovers are in the order of 1-5 seconds for us, and the more important thing is they are really reliable. We’ve never had an issue where a failover hasn’t gone well. Lastly, since MongoDB has a dynamic schema, for us that means that the code is the schema. If I’m working on a new feature and I have a property that last week was a string, but this week I want it to be an array of strings, I update my code and I’m ready to go. There isn’t much more to it than that. Yursil Kidwai (UPS i-parcel) : In many parts of the world, e-commerce rules governing cross border transaction are still changing and thus our business processes in those areas are constantly being refined. To handle the dynamic environment that our business operates in, the requirement to change the schema was paramount to us. For example, one country may require a tax identification number, while another country may suddenly decide it needs your passport, as well as some other classification number. As these changes are occurring, we really need something behind us that will adapt with us and MongoDB’s dynamic schema gave us the ability to quickly experiment and respond to our ever changing environment. We also needed the ability to scale. We have 20M tracking events across 100 vendors processed daily, as well as tens of thousands of new parcels that enter into our system every day. MongoDB’s ability to scale-out on commodity hardware and its elastic scaling features really allowed us to handle any unexpected inflows. Next Steps To understand more about the business level drivers and architectural requirements of microservices, read Microservices: Evolution of Building Modern Apps Whitepaper . For a technical deep dive into microservices and containers, read Microservices: Containers and Orchestration Whitepaper
MACH Aligned for Retail (Microservices, API-First, Cloud Native SaaS, Headless)
Across the Retail industry, MACH principles and the Mach Alliance are becoming increasingly common. What is MACH and why is it being embraced for Retail? The MACH Alliance is a non-profit organization fostering the adoption of composable architecture principles. It stands for Microservices, API-First, Cloud-Native SaaS and Headless. The MACH Alliance’s Manifesto is to: “Future proof enterprise technology and propel current and future digital experiences" The MACH Alliance and the creation of this set of principles originated in the Retail Industry. Several of the 5 co-founders of the MACH Alliance are technology companies building for retail use cases: for example commercetools is a composable commerce platform for retail (built completely on MongoDB). MongoDB has been a member of the MACH Alliance since 2020, as an “enabler” member, meaning use of our technology can enable the implementation of the MACH principles in application architectures. This is because a data layer built on MongoDB is ideal as the basis for a MACH architecture. Members of our Industry Solutions team sit on the MACH technology, growth and marketing councils, and actively are involved with furthering the adoption of MACH across the Retail Industry What is MACH, why is it important for retail? The retail industry has long been a fast adopter of technology and a forerunner in technology trends. This is because of the competitive nature of the business leading a drive towards innovation- its vital that retails are able to react quickly to new technologies (e.g. NFTs, VR, AI) to capture market share and stay ahead of the competitors. Retailers have realized that to be able to deliver new and value-add experiences to their customers, they have to cut back on operational overhead that leads to increased cost and build standard functionality that can either be bought or re-used. This is where the benefits of MACH comes in- it's all about increasing the ability to deliver innovation quickly while lowering operational costs & risk. Microservices: An approach to building applications in which business functions are broken down into smaller, self-contained components called services. These services function autonomously and are usually developed and deployed independently. This means the failure or outage of one microservice will not affect another and teams can develop in parallel, increasing efficiency. API-First: A style of development where the sharing and use of the data via API (application programming interface) is considered first and foremost in the development process. This means that services are designed to aid the easy sharing of information across the organization and simple interconnectivity of systems. Cloud-Native SaaS: Cloud-native SaaS solutions are vendor-managed applications developed in and for the cloud, and leveraging all the capabilities the cloud has to offer, such as fully managed hosting, built-in security, auto-scaling, cross-regional deployment and automatic updates. These are a good fit for a MACH architecture as adopting them can reduce operational costs and frees up developers for value-add work like new unique customer experiences. Headless: Decoupling the front end from the back-end so that front ends (or “heads”) can be created or iterated on with no dependencies on the back end. The fact that the layers are loosely coupled decreases time to market for new front ends, and encourages the re-use back-end services for multiple purposes. It also de-risks change in the long term as services can function independently. Where does MongoDB come in? MongoDB is an enabler for MACH, meaning that using MongoDB as your data layer helps retailers and retail software companies. achieve MACH compliance. Our data model, architecture and functionality empower IT organizations to build in line with these architecture principles. During a digital transformation, where a retailer is modernizing a monolith into a microservices based architecture, they're looking for a data layer which will enable speed of development & change. MongoDB is the "most wanted" database 4 years running on Stack Overflow's developer survey- this is because our document model maps to the way developers are thinking & coding, and the flexibility allows for iterative change of the data layer. When looking at API based communication, the standard format for APIs is JSON, which again maps to MongoDB's document model. The idea with API-first development is to develop with the API in mind- why not store the data the way you're going to serve it by API. This reduces complexity and increases performance. Cloud Native and SaaS products have become the norm as retailers wish to reduce maintenance and management work. MongoDB Atlas, provides a database-as-a-service, guaranteeing 99.995% uptime, automatic failover and self-healing and allowing DevOps engineers to spin up databases in minutes or by API/ script. Many retail software companies are also built on MongoDB Atlas- for example commercetools, which provides an ecommerce solution as a SaaS product. Headless architectures require a data layer that is able to adapt and change for new workloads. The ability to change the schema at runtime, with no downtime, makes MongoDB's document model ideal for this. Performance and the ability to scale for new "heads" is also important. MongoDB is known as a high performance database and can scale vertically automatically or scale out horizontally seamlessly. So MongoDB becomes a great choice for retailers choosing to adopt a MACH architecture (see figure 1 below). As a general purpose database with high performance, a rich expressive query language and secondary indexing, MongoDB is a really good fit as a data layer as it is capable of handling operational and analytical needs of the application. FIgure 1: Example of a MACH architecture Want to know more? Are you interested in a transition to MACH? Dive into our four part blog series exploring each topic in detail and how MongoDB supports each of these principles: Microservices API-First Cloud-Native SaaS Headless