Improved Experience for Saved Aggregations and Queries in MongoDB Compass
April 1, 2022
Tens of thousands of MongoDB users take advantage of MongoDB Compass to query their data and build sophisticated aggregation pipelines. As an easy-to-use GUI, Compass lets you seamlessly connect to and interact with your data, including using our powerful Query API. You just connect to your cluster, navigate to your chosen database and collection, and start building your queries.
Many Compass users want to come back again and again to their best queries — or make a query repeatedly available to all database users — but the experience of working with saved queries and aggregations has created some challenges for users in the past. Previously, saved aggregations and queries were bound to a specific database and collection, making it harder to integrate those saved queries and aggregations into the standard software-development lifecycle. If, for example, you built an aggregation pipeline against a staging database and saved it, you’d still have to build that same pipeline again if you wanted to use it for your production database. Users have also reported difficulty finding their favorites after saving them.
That’s why we’ve released a new-and-improved experience for saved aggregations and queries in MongoDB Compass. It includes a new “My Queries” screen you can navigate to from the left sidebar or from a tab at the top, next to the “Database” and “Performance” tabs.
Once on the “My Queries” screen, you can search across all your saved queries/aggregations and sort or filter by database or collection. And you can apply your saved queries/aggregations across namespaces.
To learn more about working with queries and aggregations in Compass, visit our documentation on the Aggregation Pipeline Builder or queries.
We’re confident this new experience will make it easier than ever to build, save, and reuse your favorite aggregations and queries, and ultimately remove friction with integrating them into the application development process.
Head over to your Compass instance and check it out. (If you’re not yet a Compass user, you can download it for free.) Happy querying!
Consider Your Cloud Backup Strategy on World Backup Day, March 31
World Backup Day , marked every March 31, reminds all of us of the risk and vulnerability of the data stored on our devices and systems. Data loss happens every day for a variety of reasons. While human error is the most common cause, ransomware is fast becoming the most expensive one. A big reason is because ransomware criminals have moved on from targeting individuals to attacking businesses, where data is far more sensitive and valuable. Businesses have more to lose if their data is compromised or exposed, and they have deeper pockets to pay larger ransoms. Having a backup copy of data can help a business recover clean copies of data after it’s been corrupted by malware, accidentally deleted, or destroyed by a fire or flood. Backup could also save the day during cloud outages. For businesses, any type of data loss is extremely costly. So having a backup plan as part of an overall disaster recovery strategy is critical for the survival of every business. Backup benefits A backup and disaster recovery strategy is necessary to protect your mission critical data against these types of risks. With such a strategy in place, you'll gain peace of mind knowing that if your data ever becomes accidentally deleted or infected by malware, you'll be able to recover it and avoid the cost and consequences of data loss. You'll also satisfy important regulatory and compliance requirements by demonstrating that you've taken a proactive approach towards data safety and business continuity. Taking regular backups offers other advantages as well. The backups can be used to create new environments for development, staging, or QA without impacting production. This practice enables development teams to quickly and easily test new features, accelerating application development and ensuring smooth product launches. Backup and the cloud factor Back when business systems were mostly on-premises, there was a simple framework for disaster recovery planning: the 3-2-1 backup rule. It essentially recommends keeping three copies of your data, in at least two different form factors, with one of them kept at an offsite or remote location. In practice, this might mean creating a bare metal image of an entire server and storing it on a secondary server in the same server room. In addition, you would also make a copy of all of your server data on magnetic tape and transport it to an offsite, secure location. So that’s three copies of data (the original, the bare metal image, and the tape backup) in two different form factors (disc and tape), with one stored offsite. The 3-2-1 backup rule has worked well for a lot of businesses for decades. But systems have changed. First, barely anyone uses tape anymore. The time it takes to transport a tape backup to where it can be used for disaster recovery is more than most businesses can tolerate. The criticality of systems has shrunk recovery time objectives (RTO) to just hours or minutes. Tape backups are simply not practical for most modern IT environments. But the cloud is. Hyperscale cloud providers can provide service levels that are just as reliable, if not moreso, than traditional on-premises environments. But customers shouldn’t make the mistake of thinking that by deploying in the cloud, they’re off the hook when it comes to planning and implementing a disaster recovery strategy. Businesses can and do lose data in the cloud. And hyperscale cloud providers do experience outages. Businesses must have a backup strategy, but it needs to be updated to factor in the cloud workloads that have become ubiquitous. Cross-cloud data protection An outage at any of the major cloud providers could take your databases offline. Keeping extra backup copies with different cloud providers is a good way to ensure you can still access your data in the event of an outage with a single cloud provider. Recent outages at hyperscale cloud providers have underscored the need for cross-cloud backup. “Most cloud outages are related to software bugs rather than physical catastrophes,” says Chris Shum, Atlas product lead at MongoDB. “We've always protected ourselves against physical catastrophes by distributing across data centers or regions, but no one really protects themselves against the software bug.” Shum says by backing up workloads on different clouds, you could tolerate a cloud provider going down and your database and your application would still be up. Getting around the cloud backup skills gap Data gravity keeps many businesses locked into a single cloud provider. Becoming fluent with a particular cloud provider is a skill to acquire just like anything else. And once you become comfortable with one provider, its hardware configurations, operational tools, and its offerings, it can be hard to venture out to a different provider with different hardware and pricing. But developing flexibility in the cloud is critical if you hope to leverage the best features, functionality, and cost efficiencies from each cloud provider. MongoDB has made it possible to do just that. “With Atlas, we’ve made it our mission to abstract as much of the management away as possible,” Shum says. “It’s all available as a fully managed service. So things like hardware asymmetry between different cloud providers, offerings being different, prices being different, how you’d set up networking infrastructure, all of those things that to you as a consumer might be different, we’ve abstracted it away for you. And because of the abstraction, you are then free to move nodes to whichever cloud provider or region you want.” Data protection with Atlas MongoDB Atlas provides point in time recovery of replica sets and cluster-wide snapshots of sharded clusters. It’s simple to restore to precisely the moment you need, quickly and safely. Backups can be restored automatically to the existing MongoDB Atlas cluster or downloaded to be manually archived or restored on a different infrastructure. MongoDB Atlas provides: Security features to protect access to your data Built in replication for always-on availability, tolerating complete data center failure Backups and point in time recovery to protect against data corruption Fine-grained monitoring to let you know when to scale — additional instances can be provisioned with the push of a button Automated patching and one-click upgrades for new major versions of the database, enabling you to take advantage of the latest and greatest MongoDB features A choice of cloud providers, regions, and billing options Backup wrap-up If you’re still depending on a single cloud provider to keep your critical workloads online and available, consider World Backup Day as your reminder to identify and close the gaps in your cloud disaster recovery strategy. For information on how to back up and restore cluster data in MongoDB read this article in our documentation.
Dissecting Open Banking with MongoDB: Technical Challenges and Solutions
Thank you to Ainhoa Múgica for her contributions to this post. Unleashing a disruptive wave in the banking industry, open banking (or open finance), as the term indicates, has compelled financial institutions (banks, insurers, fintechs, corporates, and even government bodies) to embrace a new era of transparency, collaboration, and innovation. This paradigm shift requires banks to openly share customer data with third-party providers (TPPs), driving enhanced customer experiences and fostering the development of innovative fintech solutions by combining ‘best-of-breed’ products and services. As of 2020, 24.7 million individuals worldwide used open banking services, a number that is forecast to reach 132.2 million by 2024. This rising trend fuels competition, spurs innovation, and fosters partnerships between traditional banks and agile fintech companies. In this transformative landscape, MongoDB, a leading developer data platform, plays a vital role in supporting open banking by providing a secure, scalable, and flexible infrastructure for managing and protecting shared customer data. By harnessing the power of MongoDB's technology, financial institutions can lower costs, improve customer experiences, and mitigate the potential risks associated with the widespread sharing of customer data through strict regulatory compliance. Figure 1: An Example Open Banking Architecture The essence of open banking/finance is about leveraging common data exchange protocols to share financial data and services with 3rd parties. In this blog, we will dive into the technical challenges and solutions of open banking from a data and data services perspective and explore how MongoDB empowers financial institutions to overcome these obstacles and unlock the full potential of this open ecosystem. Dynamic environments and standards As open banking standards continue to evolve, financial institutions must remain adaptable to meet changing regulations and industry demands. Traditional relational databases often struggle to keep pace with the dynamic requirements of open banking due to their rigid schemas that are difficult to change and manage over time. In countries without standardized open banking frameworks, banks and third-party providers face the challenge of developing multiple versions of APIs to integrate with different institutions, creating complexity and hindering interoperability. Fortunately, open banking standards or guidelines (eg. Europe, Singapore, Indonesia, Hong Kong, Australia, etc) have generally required or recommended that the open APIs be RESTful and support JSON data format, which creates a basis for common data exchange. MongoDB addresses these challenges by offering a flexible developer data platform that natively supports JSON data format, simplifies data modeling, and enables flexible schema changes for developers. With features like the MongoDB Data API and GraphQL API , developers can reduce development and maintenance efforts by easily exposing data in a low-code manner. The Stable API feature ensures compatibility during database upgrades, preventing code breaks and providing a seamless transition. Additionally, MongoDB provides productivity-boosting features like full-text search , data visualization , data federation , mobile database synchronization , and other app services enabling developers to accelerate time-to-market. With MongoDB's capabilities, financial institutions and third-party providers can navigate the changing open banking landscape more effectively, foster collaboration, and deliver innovative solutions to customers. An example of a client who leverages MongoDB’s native JSON data management and flexibility is Natwest. Natwest is a major retail and commercial bank in the United Kingdom based in London, England. The bank has moved from zero to 900 million API calls per month within years, as open banking uptake grows and is expected to grow 10 times in coming years. At a MongoDB event on 15 Nov 2022, Jonathan Haggarty, Natwest’s Head of “Bank of APIs” Technology – an API ecosystem that brings the retail bank’s services to partners – shared in his presentation titled Driving Customer Value using API Data that Natwest’s growing API ecosystem lets it “push a bunch of JSON data into MongoDB [which makes it] “easy to go from simple to quite complex information" and also makes it easier to obfuscate user details through data masking for customer privacy. Natwest is enabled to surface customer data insights for partners via its API ecosystem, for example “where customers are on the e-commerce spectrum”, the “best time [for retailers] to push discounts” as well insights on “most valuable customers” – with data being used for problem-solving; analytics and insight; and reporting. Performance In the dynamic landscape of open banking, meeting the unpredictable demands for performance, scalability, and availability is crucial. The efficiency of applications and the overall customer experience heavily rely on the responsiveness of APIs. However, building an open banking platform becomes intricate when accommodating third-party providers with undisclosed business and technical requirements. Without careful management, this can lead to unforeseen performance issues and increased costs. Open banking demands high performance of the APIs under all kinds of workload volumes. OBIE recommends an average TTLB (time to last byte) of 750 ms per endpoint response for all payment invitations (except file payments) and account information APIs. Compliance with regulatory service level agreements (SLAs) in certain jurisdictions further adds to the complexity. Legacy architectures and databases often struggle to meet these demanding criteria, necessitating extensive changes to ensure scalability and optimal performance. That's where MongoDB comes into play. MongoDB is purpose-built to deliver exceptional performance with its WiredTiger storage engine and its compression capabilities. Additionally, MongoDB Atlas improves the performance following its intelligent index and schema suggestions, automatic data tiering, and workload isolation for analytics. One prime illustration of its capabilities is demonstrated by Temenos, a renowned financial services application provider, achieving remarkable transaction volume processing performance and efficiency by leveraging MongoDB Atlas. They recently ran a benchmark with MongoDB Atlas and Microsoft Azure and successfully processed an astounding 200 million embedded finance loans and 100 million retail accounts at a record-breaking 150,000 transactions per second . This showcases the power and scalability of MongoDB with unparalleled performance to empower financial institutions to effectively tackle the challenges posed by open banking. MongoDB ensures outstanding performance, scalability, and availability to meet the ever-evolving demands of the industry. Scalability Building a platform to serve TPPs, who may not disclose their business usages and technical/performance requirements, can introduce unpredictable performance and cost issues if not managed carefully. For instance, a bank in Singapore faced an issue where their Open APIs experienced peak loads and crashes every Wednesday. After investigation, they discovered that one of the TPPs ran a promotional campaign every Wednesday, resulting in a surge of API calls that overwhelmed the bank's infrastructure. A scalable solution that can perform under unpredictable workloads is critical, besides meeting the performance requirements of a certain known volume of transactions. MongoDB's flexible architecture and scalability features address these concerns effectively. With its distributed document-based data model, MongoDB allows for seamless scaling both vertically and horizontally. By leveraging sharding , data can be distributed across multiple nodes, ensuring efficient resource utilization and enabling the system to handle high transaction volumes without compromising performance. MongoDB's auto-sharding capability enables dynamic scaling as the workload grows, providing financial institutions with the flexibility to adapt to changing demands and ensuring a smooth and scalable open banking infrastructure. Availability In the realm of open banking, availability becomes a critical challenge. With increased reliance on banking services by third-party providers (TPPs), ensuring consistent availability becomes more complex. Previously, banks could bring down certain services during off-peak hours for maintenance. However, with TPPs offering 24x7 experiences, any downtime is unacceptable. This places greater pressure on banks to maintain constant availability for Open API services, even during planned maintenance windows or unforeseen events. MongoDB Atlas, the fully managed global cloud database service, addresses these availability challenges effectively. With its multi-node cluster and multi-cloud DBaaS capabilities, MongoDB Atlas ensures high availability and fault tolerance. It offers the flexibility to run on multiple leading cloud providers, allowing banks to minimize concentration risk and achieve higher availability through a distributed cluster across different cloud platforms. The robust replication and failover mechanisms provided by MongoDB Atlas guarantee uninterrupted service and enable financial institutions to provide reliable and always-available open banking APIs to their customers and TPPs. Security and privacy Data security and consent management are paramount concerns for banks participating in open banking. The exposure of authentication and authorization mechanisms to third-party providers raises security concerns and introduces technical complexities regarding data protection. Banks require fine-grained access control and encryption mechanisms to safeguard shared data, including managing data-sharing consent at a granular level. Furthermore, banks must navigate the landscape of data privacy laws like the General Data Protection Regulation (GDPR), which impose strict requirements distinct from traditional banking regulations. MongoDB offers a range of solutions to address these security and privacy challenges effectively. Queryable Encryption provides a mechanism for managing encrypted data within MongoDB, ensuring sensitive information remains secure even when shared with third-party providers. MongoDB's comprehensive encryption features cover data-at-rest and data-in-transit, protecting data throughout its lifecycle. MongoDB's flexible schema allows financial institutions to capture diverse data requirements for managing data sharing consent and unify user consent from different countries into a single data store, simplifying compliance with complex data privacy laws. Additionally, MongoDB's geo-sharding capabilities enable compliance with data residency laws by ensuring relevant data and consent information remain in the closest cloud data center while providing optimal response times for accessing data. To enhance data privacy further, MongoDB offers field-level encryption techniques, enabling symmetric encryption at the field level to protect sensitive data (e.g., personally identifiable information) even when shared with TPPs. The random encryption of fields adds an additional layer of security and enables query operations on the encrypted data. MongoDB's Queryable Encryption technique further strengthens security and defends against cryptanalysis, ensuring that customer data remains protected and confidential within the open banking ecosystem. Activity monitoring With numerous APIs offered by banks in the open banking ecosystem, activity monitoring and troubleshooting become critical aspects of maintaining a robust and secure infrastructure. MongoDB simplifies activity monitoring through its monitoring tools and auditing capabilities. Administrators and users can track system activity at a granular level, monitoring database system and application events. MongoDB Atlas has Administration APIs , which one can use to programmatically manage the Atlas service. For example, one can use the Atlas Administration API to create database deployments, add users to those deployments, monitor those deployments, and more. These APIs can help with the automation of CI/CD pipelines as well as monitoring the activities on the data platform enabling developers and administrators to be freed of this mundane effort and focus on generating more business value. Performance monitoring tools, including the performance advisor, help gauge and optimize system performance, ensuring that APIs deliver exceptional user experiences. Figure 2: Activity Monitoring on MongoDB Atlas MongoDB Atlas Charts , an integrated feature of MongoDB Atlas, offers analytics and visualization capabilities. Financial institutions can create business intelligence dashboards using MongoDB Atlas Charts. This eliminates the need for expensive licensing associated with traditional business intelligence tools, making it cost-effective as more TPPs utilize the APIs. With MongoDB Atlas Charts, financial institutions can offer comprehensive business telemetry data to TPPs, such as the number of insurance quotations, policy transactions, API call volumes, and performance metrics. These insights empower financial institutions to make data-driven decisions, improve operational efficiency, and optimize the customer experience in the open banking ecosystem. Figure 3: Atlas Charts Sample Dashboard Real-Timeliness Open banking introduces new challenges for financial institutions as they strive to serve and scale amidst unpredictable workloads from TPPs. While static content poses fewer difficulties, APIs requiring real-time updates or continuous streaming, such as dynamic account balances or ESG-adjusted credit scores, demand capabilities for near-real-time data delivery. To enable applications to immediately react to real-time changes or changes as they occur, organizations can leverage MongoDB Change Streams that are based on its aggregation framework to react to data changes in a single collection, a database, or even an entire deployment. This capability further enhances MongoDB’s real-time data and event processing and analytics capabilities. MongoDB offers multiple mechanisms to support data streaming, including a Kafka connector for event-driven architecture and a Spark connector for streaming with Spark. These solutions empower financial institutions to meet the real-time data needs of their open banking partners effectively, enabling seamless integration and real-time data delivery for enhanced customer experiences. Conclusion MongoDB's technical capabilities position it as a key enabler for financial institutions embarking on their open banking journey. From managing dynamic environments and accommodating unpredictable workloads to ensuring scalability, availability, security, and privacy, MongoDB provides a comprehensive set of tools and features to address the challenges of open banking effectively. With MongoDB as the underlying infrastructure, financial institutions can navigate the ever-evolving open banking landscape with confidence, delivering innovative solutions, and driving the future of banking. Embracing MongoDB empowers financial institutions to unlock the full potential of open banking and provide exceptional customer experiences in this era of collaboration and digital transformation. If you would like to learn more about how you can leverage MongoDB for your open banking infrastructure, take a look at the below resources: Open banking panel discussion: future-proof your bank in a world of changing data and API standards with MongoDB, Celent, Icon Solutions, and AWS How a data mesh facilitates open banking Financial services hub