Spring + MongoDB: Two Leaves of the Same Tree
May 8, 2015 | Updated: June 4, 2015
Norberto Leite spoke at Spring.io last week. In this blog post, he will discuss his talk: Spring + MongoDB: Two Leaves of the Same Tree.
Spring Framework and MongoDB
At MongoDB, we believe that integration is key. It is important to integrate with a wide variety of languages and ecosystems, and it is important to expose all the features and functionality that make MongoDB great. We want you to be able to build all kinds of applications on all sorts of different environments using the tools that are best suited for your purpose. As part of this effort of enabling and boosting developer productivity, we support a large variety of programming languages through our official and community drivers and enabling existing frameworks and applications stacks to integrate correctly with MongoDB.
Spring Framework, and in particular, Spring Data, is a very good example of how one can consolidate the development experience using familiar or well understood tools to build their applications
Spring Projects and MongoDB
Spring is one of the most prominent frameworks used across Java enterprise projects. Many applications across a variety of businesses, environments, and stacks rely on Spring projects for many of the integrations and general implementation of functionality. Some Spring projects are widely used like Spring Batch which offers a generic approach to batch processing, Spring Boot where we can automatize a large set of application processes so developers can focus on the business logic and differentiated algorithms of their apps and services. Spring Data offers applications a very powerful ODM to support not only application level abstraction of the persistence layer but also an integrated approach to handle all the data manipulation and common impedance mismatches that application logic provokes.
This presentation discusses a set of features that make this integration "gel" well: Spring Data abstraction layer The way that Spring Data covers Object Mapping Optimizations that Spring Data enables to make the most out of MongoDB Batch processing and indexing will also be covered, with particular emphasis around the method overriding and query optimization.
Use your tools wisely
There are significant benefits of using ODMs, especially for large complex projects:
- Assures integration with existing components
- Abstraction layers allow architects to delay decisions and avoid pre-optimizations
- Common patterns that are recurrent across different data stores
But also bear in mind that many of the existing ORMs/ODMs do not have a "Document Oriented Database" first policy but have been evolving to adjust to today’s database industry revolution. Many of the implementations are based on an architecture that is oriented to relational technology, and they make significant tradeoffs to accommodate several different systems.
Spring Data is one of the most popular and best-designed ORM technologies out there. MongoDB is committed to making sure the integration between these technologies is great.
View Norberto's presentation here.
Artificial Intelligence Goes Beyond Kittens Playing the Piano
In March, Facebook announced the ability to recognize different types of actions in videos. This is one of many recent examples of powerful and interesting innovations in the area of Artificial Intelligence. Being able to draw context out of text, images and rich media types will allow Facebook to be more effective in content curation. In other words, animal lovers may start seeing a lot more videos of kittens playing pianos in their feeds. This technology should help Facebook better target their audience, suggest friends and sell more ads. But the possibilities go far beyond kitten videos. In addition to delivering more personalized content, AI can lengthen our lives, make our businesses more efficient, and protect our citizens. We can already see the huge impact of predictive analytics in metropolitan services. For example, the City of Chicago relies on their WindyGrid service to collect and make sense of the millions of pieces of information gathered daily from Chicago’s 15 most crucial departments, including police, transportation, and fire. It’s an ever-changing view of what makes the city tick. Roadwork updates, trash pickup delays, 911 health emergencies, 311 complaints about noise, public tweets about the minutia of the city’s workings, bus locations along their route, traffic light patterns, and much more. WindyGrid analyzes trends across multiple data sources to make predictions about what will happen next. Now imagine what the City of Chicago could do with AI technology similar to Facebook’s. We move beyond understanding trends to being able to develop solutions that automatically understand and respond to specific events as they’re happening in real-time. Chicago health officials could know if an elderly citizen is experiencing a health emergency when they are no longer able to call for help. Emergency responders could be notified when an infant is in need health attention before he first cries out. Firemen could be deployed when the first ash ignites, and an intelligent system could recognize the severity of the fire to recommend an appropriate response. Video surveillance could recognize a burglary as it is happening and send alerts to the property authorities. The possibilities are endless. Building a practical system of this kind hasn’t been easy. Natural language processing, machine learning and reasoning require the processing of high volume, variant data which had overwhelmed traditional data stores. New technologies and new databases allow for associations, patterns, and vectors to be recognized. MongoDB has made the real-world deployment of massive, integrated machine learning systems a practical reality. With new AI technologies developing, we will be able to make predictions about future human actions and better respond to everyday issues. Users will have more personalized experiences and better quality of life than ever before. But kittens playing pianos are important too. If you're interested in learning more about how the City of Chicago leveraged real-time analytics for their WindyGrid service, read the customer case study, or come to MongoDB World this June and hear for yourself! Read the Case Study
Dissecting Open Banking with MongoDB: Technical Challenges and Solutions
Thank you to Ainhoa Múgica for her contributions to this post. Unleashing a disruptive wave in the banking industry, open banking (or open finance), as the term indicates, has compelled financial institutions (banks, insurers, fintechs, corporates, and even government bodies) to embrace a new era of transparency, collaboration, and innovation. This paradigm shift requires banks to openly share customer data with third-party providers (TPPs), driving enhanced customer experiences and fostering the development of innovative fintech solutions by combining ‘best-of-breed’ products and services. As of 2020, 24.7 million individuals worldwide used open banking services, a number that is forecast to reach 132.2 million by 2024. This rising trend fuels competition, spurs innovation, and fosters partnerships between traditional banks and agile fintech companies. In this transformative landscape, MongoDB, a leading developer data platform, plays a vital role in supporting open banking by providing a secure, scalable, and flexible infrastructure for managing and protecting shared customer data. By harnessing the power of MongoDB's technology, financial institutions can lower costs, improve customer experiences, and mitigate the potential risks associated with the widespread sharing of customer data through strict regulatory compliance. Figure 1: An Example Open Banking Architecture The essence of open banking/finance is about leveraging common data exchange protocols to share financial data and services with 3rd parties. In this blog, we will dive into the technical challenges and solutions of open banking from a data and data services perspective and explore how MongoDB empowers financial institutions to overcome these obstacles and unlock the full potential of this open ecosystem. Dynamic environments and standards As open banking standards continue to evolve, financial institutions must remain adaptable to meet changing regulations and industry demands. Traditional relational databases often struggle to keep pace with the dynamic requirements of open banking due to their rigid schemas that are difficult to change and manage over time. In countries without standardized open banking frameworks, banks and third-party providers face the challenge of developing multiple versions of APIs to integrate with different institutions, creating complexity and hindering interoperability. Fortunately, open banking standards or guidelines (eg. Europe, Singapore, Indonesia, Hong Kong, Australia, etc) have generally required or recommended that the open APIs be RESTful and support JSON data format, which creates a basis for common data exchange. MongoDB addresses these challenges by offering a flexible developer data platform that natively supports JSON data format, simplifies data modeling, and enables flexible schema changes for developers. With features like the MongoDB Data API and GraphQL API , developers can reduce development and maintenance efforts by easily exposing data in a low-code manner. The Stable API feature ensures compatibility during database upgrades, preventing code breaks and providing a seamless transition. Additionally, MongoDB provides productivity-boosting features like full-text search , data visualization , data federation , mobile database synchronization , and other app services enabling developers to accelerate time-to-market. With MongoDB's capabilities, financial institutions and third-party providers can navigate the changing open banking landscape more effectively, foster collaboration, and deliver innovative solutions to customers. An example of a client who leverages MongoDB’s native JSON data management and flexibility is Natwest. Natwest is a major retail and commercial bank in the United Kingdom based in London, England. The bank has moved from zero to 900 million API calls per month within years, as open banking uptake grows and is expected to grow 10 times in coming years. At a MongoDB event on 15 Nov 2022, Jonathan Haggarty, Natwest’s Head of “Bank of APIs” Technology – an API ecosystem that brings the retail bank’s services to partners – shared in his presentation titled Driving Customer Value using API Data that Natwest’s growing API ecosystem lets it “push a bunch of JSON data into MongoDB [which makes it] “easy to go from simple to quite complex information" and also makes it easier to obfuscate user details through data masking for customer privacy. Natwest is enabled to surface customer data insights for partners via its API ecosystem, for example “where customers are on the e-commerce spectrum”, the “best time [for retailers] to push discounts” as well insights on “most valuable customers” – with data being used for problem-solving; analytics and insight; and reporting. Performance In the dynamic landscape of open banking, meeting the unpredictable demands for performance, scalability, and availability is crucial. The efficiency of applications and the overall customer experience heavily rely on the responsiveness of APIs. However, building an open banking platform becomes intricate when accommodating third-party providers with undisclosed business and technical requirements. Without careful management, this can lead to unforeseen performance issues and increased costs. Open banking demands high performance of the APIs under all kinds of workload volumes. OBIE recommends an average TTLB (time to last byte) of 750 ms per endpoint response for all payment invitations (except file payments) and account information APIs. Compliance with regulatory service level agreements (SLAs) in certain jurisdictions further adds to the complexity. Legacy architectures and databases often struggle to meet these demanding criteria, necessitating extensive changes to ensure scalability and optimal performance. That's where MongoDB comes into play. MongoDB is purpose-built to deliver exceptional performance with its WiredTiger storage engine and its compression capabilities. Additionally, MongoDB Atlas improves the performance following its intelligent index and schema suggestions, automatic data tiering, and workload isolation for analytics. One prime illustration of its capabilities is demonstrated by Temenos, a renowned financial services application provider, achieving remarkable transaction volume processing performance and efficiency by leveraging MongoDB Atlas. They recently ran a benchmark with MongoDB Atlas and Microsoft Azure and successfully processed an astounding 200 million embedded finance loans and 100 million retail accounts at a record-breaking 150,000 transactions per second . This showcases the power and scalability of MongoDB with unparalleled performance to empower financial institutions to effectively tackle the challenges posed by open banking. MongoDB ensures outstanding performance, scalability, and availability to meet the ever-evolving demands of the industry. Scalability Building a platform to serve TPPs, who may not disclose their business usages and technical/performance requirements, can introduce unpredictable performance and cost issues if not managed carefully. For instance, a bank in Singapore faced an issue where their Open APIs experienced peak loads and crashes every Wednesday. After investigation, they discovered that one of the TPPs ran a promotional campaign every Wednesday, resulting in a surge of API calls that overwhelmed the bank's infrastructure. A scalable solution that can perform under unpredictable workloads is critical, besides meeting the performance requirements of a certain known volume of transactions. MongoDB's flexible architecture and scalability features address these concerns effectively. With its distributed document-based data model, MongoDB allows for seamless scaling both vertically and horizontally. By leveraging sharding , data can be distributed across multiple nodes, ensuring efficient resource utilization and enabling the system to handle high transaction volumes without compromising performance. MongoDB's auto-sharding capability enables dynamic scaling as the workload grows, providing financial institutions with the flexibility to adapt to changing demands and ensuring a smooth and scalable open banking infrastructure. Availability In the realm of open banking, availability becomes a critical challenge. With increased reliance on banking services by third-party providers (TPPs), ensuring consistent availability becomes more complex. Previously, banks could bring down certain services during off-peak hours for maintenance. However, with TPPs offering 24x7 experiences, any downtime is unacceptable. This places greater pressure on banks to maintain constant availability for Open API services, even during planned maintenance windows or unforeseen events. MongoDB Atlas, the fully managed global cloud database service, addresses these availability challenges effectively. With its multi-node cluster and multi-cloud DBaaS capabilities, MongoDB Atlas ensures high availability and fault tolerance. It offers the flexibility to run on multiple leading cloud providers, allowing banks to minimize concentration risk and achieve higher availability through a distributed cluster across different cloud platforms. The robust replication and failover mechanisms provided by MongoDB Atlas guarantee uninterrupted service and enable financial institutions to provide reliable and always-available open banking APIs to their customers and TPPs. Security and privacy Data security and consent management are paramount concerns for banks participating in open banking. The exposure of authentication and authorization mechanisms to third-party providers raises security concerns and introduces technical complexities regarding data protection. Banks require fine-grained access control and encryption mechanisms to safeguard shared data, including managing data-sharing consent at a granular level. Furthermore, banks must navigate the landscape of data privacy laws like the General Data Protection Regulation (GDPR), which impose strict requirements distinct from traditional banking regulations. MongoDB offers a range of solutions to address these security and privacy challenges effectively. Queryable Encryption provides a mechanism for managing encrypted data within MongoDB, ensuring sensitive information remains secure even when shared with third-party providers. MongoDB's comprehensive encryption features cover data-at-rest and data-in-transit, protecting data throughout its lifecycle. MongoDB's flexible schema allows financial institutions to capture diverse data requirements for managing data sharing consent and unify user consent from different countries into a single data store, simplifying compliance with complex data privacy laws. Additionally, MongoDB's geo-sharding capabilities enable compliance with data residency laws by ensuring relevant data and consent information remain in the closest cloud data center while providing optimal response times for accessing data. To enhance data privacy further, MongoDB offers field-level encryption techniques, enabling symmetric encryption at the field level to protect sensitive data (e.g., personally identifiable information) even when shared with TPPs. The random encryption of fields adds an additional layer of security and enables query operations on the encrypted data. MongoDB's Queryable Encryption technique further strengthens security and defends against cryptanalysis, ensuring that customer data remains protected and confidential within the open banking ecosystem. Activity monitoring With numerous APIs offered by banks in the open banking ecosystem, activity monitoring and troubleshooting become critical aspects of maintaining a robust and secure infrastructure. MongoDB simplifies activity monitoring through its monitoring tools and auditing capabilities. Administrators and users can track system activity at a granular level, monitoring database system and application events. MongoDB Atlas has Administration APIs , which one can use to programmatically manage the Atlas service. For example, one can use the Atlas Administration API to create database deployments, add users to those deployments, monitor those deployments, and more. These APIs can help with the automation of CI/CD pipelines as well as monitoring the activities on the data platform enabling developers and administrators to be freed of this mundane effort and focus on generating more business value. Performance monitoring tools, including the performance advisor, help gauge and optimize system performance, ensuring that APIs deliver exceptional user experiences. Figure 2: Activity Monitoring on MongoDB Atlas MongoDB Atlas Charts , an integrated feature of MongoDB Atlas, offers analytics and visualization capabilities. Financial institutions can create business intelligence dashboards using MongoDB Atlas Charts. This eliminates the need for expensive licensing associated with traditional business intelligence tools, making it cost-effective as more TPPs utilize the APIs. With MongoDB Atlas Charts, financial institutions can offer comprehensive business telemetry data to TPPs, such as the number of insurance quotations, policy transactions, API call volumes, and performance metrics. These insights empower financial institutions to make data-driven decisions, improve operational efficiency, and optimize the customer experience in the open banking ecosystem. Figure 3: Atlas Charts Sample Dashboard Real-Timeliness Open banking introduces new challenges for financial institutions as they strive to serve and scale amidst unpredictable workloads from TPPs. While static content poses fewer difficulties, APIs requiring real-time updates or continuous streaming, such as dynamic account balances or ESG-adjusted credit scores, demand capabilities for near-real-time data delivery. To enable applications to immediately react to real-time changes or changes as they occur, organizations can leverage MongoDB Change Streams that are based on its aggregation framework to react to data changes in a single collection, a database, or even an entire deployment. This capability further enhances MongoDB’s real-time data and event processing and analytics capabilities. MongoDB offers multiple mechanisms to support data streaming, including a Kafka connector for event-driven architecture and a Spark connector for streaming with Spark. These solutions empower financial institutions to meet the real-time data needs of their open banking partners effectively, enabling seamless integration and real-time data delivery for enhanced customer experiences. Conclusion MongoDB's technical capabilities position it as a key enabler for financial institutions embarking on their open banking journey. From managing dynamic environments and accommodating unpredictable workloads to ensuring scalability, availability, security, and privacy, MongoDB provides a comprehensive set of tools and features to address the challenges of open banking effectively. With MongoDB as the underlying infrastructure, financial institutions can navigate the ever-evolving open banking landscape with confidence, delivering innovative solutions, and driving the future of banking. Embracing MongoDB empowers financial institutions to unlock the full potential of open banking and provide exceptional customer experiences in this era of collaboration and digital transformation. If you would like to learn more about how you can leverage MongoDB for your open banking infrastructure, take a look at the below resources: Open banking panel discussion: future-proof your bank in a world of changing data and API standards with MongoDB, Celent, Icon Solutions, and AWS How a data mesh facilitates open banking Financial services hub