Applications

Customer stories, use cases, and experiences of MongoDB

Microservices: Realizing the Benefits Without the Complexity

The microservice architecture has emerged as the preferred, modern approach for developers to build and deploy applications on the cloud. It can help you deliver more reliable applications, and address the scale and latency concerns for System Reliability Engineers (SREs) and operations. But microservices aren't without their hangups. For developers, microservices can lead to additional complexity and cognitive overhead, such as cross-service coordination, shared states across multiple services, and coding and testing failure logic across disconnected services. While the monolith was suboptimal for compute and scale efficiencies, the programming model was simple. So the question is, can we get the best of both worlds? In addition, how do we make the individual services easier to build and adapt to changing requirements? Since, at their core, microservices provide access to and perform operations on data, how do we architect services so that developers can easily work with data? How can we make it easier for developers to add new types of data and data sources and perform a wide variety of data operations without the complexity of managing caches and using multiple query languages (SQL, full-text and vector search, time series, geospatial, etc.) The development complexity associated with microservice architectures occurs at two levels: service orchestration and service data management. The diagram below depicts this complexity. At the orchestration level, a typical application may support tens or hundreds of processes, and each may have thousands or millions of executions. To make this work, services are often connected by a patchwork of queues. Developers spend quite a bit of time tracking and managing all the various workflows. The sheer scale necessitates a central mechanism to manage concurrent tasks and sharded databases to manage the state of millions of concurrent workflow instances. To add more complexity, each microservice is deployed using a set of data platforms including RDBMS, caches, search engines, and vector and NoSQL databases. Developers must work with multiple query languages, write code to keep data in sync among these platforms and write code to deal with edge cases when invariably data or indexes are not in sync. Finally, developer productivity is inhibited by the brittleness of RDBMS, which lacks flexibility when trying to incorporate new or changing data types. As a result, microservice applications often end up with complex architectures that are difficult to develop against and maintain in terms of both the individual microservices and the service orchestration. Realizing the benefits without the complexity One approach to address these microservice challenges is to combine two technologies: Temporal and MongoDB. Both give you the benefits of microservices while simplifying the implementation of service orchestration. Together, they allow developers to build services that can easily handle a wide variety of data, eliminate the need to code for complex infrastructure and reduce the likelihood of failure. They simplify the data model and your code. In one real-world example, open-source indexing company Mixpeek leverages the combination of MongoDB and Temporal to provide a platform enabling organizations to easily incorporate multi-modal data sources in AI applications. Mixpeek’s CEO Ethan Steininger states, “Temporal’s durable execution guarantees and MongoDB's flexible data model are core components of Mixpeek’s multimodal data processing and storage. Combined, they enable our users to run high volume ML on commodity hardware without worrying about dropped jobs.” MongoDB and Temporal: Build like a monolith with durable microservices Both MongoDB and Temporal were built by developers, for developers. They both use a code-first approach to solving the complex infrastructure needs of our modern applications within our application code and empower developers to be more productive. They are part of an emerging development stack that greatly simplifies data and all the cross-functional coordination we need in our cloud applications. Ultimately, the combination of these two developer-focused platforms allows you to simplify design, development, and testing of microservice-based applications. With the document model of MongoDB, you model data as real world objects and not tables, rows, and columns. With Temporal, you design your end-to-end service flows as workflows as described by domain experts without having to explicitly identify every edge case and exception (Temporal handles those implicitly). Temporal and MongoDB provide the same benefits that, when combined, multiply in value. You become more agile, as not only can everyone understand your code better, but you are no longer challenged by the cognitive overload of trying to coordinate, comprehend, and test a web of disconnected and complex services. Together, they allow us to reliably orchestrate business processes within apps that are all speaking the language of the data itself. Combining Temporal and MongoDB results in the simplified architecture shown below. Temporal enables orchestration to be implemented at a higher level of abstraction, eliminating much of the event management and queuing complexity. MongoDB, in turn, provides a single integrated data platform with a unified query language thereby eliminating much of the data management complexity. Let’s examine MongoDB and Temporal in more depth to better understand their capabilities and why they facilitate the rapid development of microservices-based applications. MongoDB: Simplifying microservice data MongoDB's features align well with the principles of microservices architectures. It reduces the need for niche databases and the associated costs of deploying and maintaining a complicated sprawl of data technologies. More explicitly, MongoDB delivers key benefits for microservice development: Flexible schema, flexible services: Unlike relational databases with rigid schemas, MongoDB's document model allows microservices to easily evolve as data requirements change. Distributed scale for data-heavy, distributed services: MongoDB scales horizontally by adding more partitions to distribute the load. This aligns with the modular nature of microservices, where individual services can scale based on their specific needs. Unified query language reduces microservice sprawl: MongoDB supports a diverse set of data operations without requiring multiple data platforms (caches, vector, and text search engines, time series, geospatial, etc.) Operational efficiency: MongoDB Atlas, the cloud-based version of MongoDB, simplifies managing databases for microservices. It handles provisioning, backups, and patching, freeing developers to focus on core responsibilities. Integrated developer data platform: The integrated developer data platform delivers an intuitive set of tools to build services that support mobile clients, real-time analytics, data visualization, and historical analysis across many service databases. With MongoDB, development teams use one interface for all their services and run it anywhere, even across clouds. Also, it provides a data foundation for your microservices that is highly available, scalable, and secure. It greatly simplifies microservices development so that you can focus on your business problems and not data. Temporal: Don't coordinate services, orchestrate them Temporal delivers an open-source, durable execution solution that removes the complexity of building scalable distributed microservices. It presents a development abstraction that preserves the complete application state so that in the case of a host or software failure, it can seamlessly migrate execution to another machine. This means you can develop applications as if failures—like network outages or server crashes—do not exist. Temporal handles these issues, allowing you to focus on implementing business logic rather than coding complex failure detection and recovery routines. Here's how Temporal simplifies application development: Durable workflows: Temporal maintains the state and progress of a defined workflow across multiple services, even in the face of server crashes, network partitions, and other types of failures. This durability ensures that your application logic can resume where it left off, making your overall application more resilient. Simplifies failure handling: Temporal abstracts away the complex error handling and retry logic that developers typically have to implement in distributed systems. This abstraction allows developers to focus on business logic rather than the intricacies of ensuring their end-to-end services can handle failures gracefully. Scale: Temporal applications are inherently scalable and capable of handling billions of workflow executions. Long-running services: Temporal supports long-running operations, from seconds to years, with the same level of reliability and scalability. By providing a platform that handles the complexities of distributed systems, Temporal allows developers to concentrate on implementing business logic in their services. This focus can lead to faster development times and more reliable applications, as developers are not bogged down by the intricacies of state management, retries, and error handling. The next generation of microservices development is here Developers want to code. They want to solve business problems. They do not want to be bogged down by the complexity of infrastructure failures. They want to model their apps and data so that it is aligned with the real-world entities and domains they are solving for. Using MongoDB and Temporal together solves these complexities. Together, they simplify design, development, and testing of microservice-based applications so that you can focus on business problems and deliver more features faster. Getting started with Temporal and MongoDB Atlas We can help you design the best architecture for your organization’s needs. Feel free to connect with your MongoDB and Temporal account teams or contact us to schedule a collaborative session and explore how Temporal and MongoDB can optimize your AI development process.

June 3, 2024
Applied

Payments Modernization and the Role of the Operational Data Layer

To stay relevant and competitive, payment solution providers must enhance their payment processes to adapt to changing customer expectations, regulatory demands, and advancing technologies. The imperative for modernization is clear: payment systems must become faster, more secure, and seamlessly integrated across platforms. Driven by multiple factors—real-time payments, regulatory shifts like Payment Services Directive 2 (PSD2), heightened customer expectations, the power of open banking, and the disruptive force of fintech startups—the need for payment modernization has never been more pressing. But transformation is not without its challenges. Complex systems, industry reliance on outdated technology, high upgrade costs, and technical debt all pose formidable obstacles. This article will explore modernization approaches and how MongoDB helps smooth transformations. Approaches to modernization As businesses work to modernize their payment systems, they need to overcome the complexities inherent in updating legacy systems. Forward-thinking organizations embrace innovative strategies to streamline their operations, enhance scalability, and facilitate agile responses to evolving market demands. Two such approaches gaining prominence in the realm of payment system modernization are domain-driven design and microservices architecture : Domain-driven design: This approach focuses on a business's core operations to develop scalable and easier-to-manage systems. Domain-driven design ensures that technology serves strategic business goals by aligning system development with business needs. At its core, this approach seeks to break down complex business domains into manageable components, or "domains," each representing a distinct area of business functionality. Microservices architecture: Unlike traditional monolithic architectures, characterized by tightly coupled and interdependent components, a microservices architecture decomposes applications into a collection of loosely coupled services, each of which is responsible for a specific business function or capability. It introduces more flexibility and allows for quicker updates, facilitating agile responses to changing business requirements. Discover how Wells Fargo launched their next-generation card payments by building an operational data store with MongoDB . Modernizing with an operational data layer In the payments modernization process, the significance of an operational data layer (ODL) cannot be overstated. An ODL is an architectural pattern that centrally integrates and organizes siloed enterprise data, making it available to consuming applications. The simplest representation of this pattern looks something like the sample reference architecture below. Figure 1: Operational Data Layer structure An ODL is deployed in front of legacy systems to enable new business initiatives and to meet new requirements that the existing architecture can’t handle—without the difficulty and risk of fully replacing legacy systems. It can reduce the workload on source systems, improve availability, reduce end-user response times, combine data from multiple systems into a single repository, serve as a foundation for re-architecting a monolithic application into a suite of microservices, and more. The ODL becomes a system of innovation, allowing the business to take an iterative approach to digital transformation. Here's why an ODL is considered ideal for payment operations: Unified data management: Payment systems involve handling a vast amount of diverse data, including transaction details, customer information, and regulatory compliance data. An ODL provides a centralized repository for storing and managing this data, eliminating silos and ensuring data integrity. Real-time processing: An ODL enables real-time processing of transactions, allowing businesses to handle high numbers of transactions swiftly and efficiently. This capability is essential for meeting customer expectations for instant payments and facilitating seamless transactions across various channels. Scalability and flexibility: Payment systems must accommodate fluctuating transaction volumes and evolving business needs. An ODL offers scalability and flexibility, allowing businesses to scale their infrastructure as demand grows. Enhanced security: An ODL incorporates robust security features —such as encryption, access controls, and auditing capabilities—to safeguard data integrity and confidentiality. By centralizing security measures within the ODL, businesses can ensure compliance with regulatory requirements and mitigate security risks effectively. Support for payments data monetization: Payment systems generate a wealth of data that can provide valuable insights into customer behavior, transaction trends, and business performance. An ODL facilitates real-time analytics and reporting by providing a unified platform for collecting, storing, and analyzing this data. Transform with MongoDB MongoDB’s fundamental technology principles ensure companies can reap the advantages of microservices and domain-driven design—specifically, our flexible data model and built-in redundancy, automation, and scalability. Indeed, the document model is tailor-made for the intricacies of payment data, ensuring adaptability and scalability as market demands evolve. Here’s how MongoDB helps with domain-driven design and microservice implementation to adopt industry best practices: Ease of use: MongoDB’s document model makes it simple to model or remodel data to fit the needs of payment applications. Documents are a natural way of describing data. They present a single data structure, with related data embedded as sub-documents and arrays, making it simpler and faster for developers to model how data in the application will be mapped to data stored in the database. In addition, MongoDB guarantees the multi-record ACID transactional semantics that developers are familiar with, making it easier to reason about data. Flexibility: MongoDB’s dynamic schema is ideal for handling the requirements of microservices and a domain-driven design. Domain-driven design emphasizes modeling the domain to reflect the business requirements, which may evolve over time. MongoDB's flexible schema allows you to store domain objects as documents without rigid schema constraints, facilitating agile development and evolution of the domain model. Speed: Using MongoDB for an ODL means you can get better performance when accessing data, and write less code to do so. A document is a single place for the database to read and write data for an entity. This locality of data ensures the complete document can be accessed in a single database operation that avoids the need internally to pull data from many different tables and rows. Data access and microservice-based APIs: MongoDB integrates seamlessly with modern technologies and frameworks commonly used in microservices architectures. MongoDB's flexible data model and ability to handle various data types, including structured and unstructured data, is a great fit for orchestrating your open API ecosystem to make data flow between banks, third parties, and consumers possible. Scalability: Even if an ODL starts at a small scale, you need to be prepared for growth as new source systems are integrated, adding data volume, and new consuming systems are developed, increasing workload. MongoDB provides horizontal scale-out on low-cost, commodity hardware or cloud infrastructure using sharding to meet the needs of an ODL with large data sets and high throughput requirements. High availability: Microservices architectures require high availability to ensure that individual services remain accessible even in the event of failures. MongoDB provides built-in replication and failover capabilities, ensuring data availability and minimal downtime in case of server failures. Payment modernization is not merely a trend but a strategic imperative. By embracing modern payment solutions and leveraging the power of an ODL with MongoDB, organizations can unlock new growth opportunities, enhance operational efficiency, and deliver superior customer experiences. Learn how to build an operational data layer with MongoDB using this Payments Modernization Solution Accelerator . Learn more about how MongoDB is powering industries on our solution library .

May 15, 2024
Applied

Five Languages, One Goal: A Developer's Path to Certification Mastery

MongoDB Community Creator Markandey Pathak has become a certified developer in five different programming languages: C#, Java, Node.JS, PHP, and Python. Pursuing multiple certifications equips developers with a diverse skill set, making them invaluable team members. Fluency across different programming languages enables them to foster platform-agnostic solutions and promote adaptability, collaboration, and informed decision-making, which are crucial for success in the global tech landscape. To understand what led Markandey to take on so many certifications while managing a busy and successful career, we spoke with him to gain insights into the challenges and triumphs he faced. What motivated you to pursue certification in multiple programming languages, and how has achieving such a diverse set of skills impacted your career? C was the first programming language I learned, followed by C# and the .NET ecosystem a few years later. Transitioning to a new language like C# after knowing one was straightforward. I then delved into ASP.NET, JAVA, and subsequently PHP. Despite the differing syntax of these languages, I found that fundamental programming concepts remained consistent. This enlightening realization led me to explore JavaScript and, later, Python. Such a diverse skill set made me a go-to resource for many senior leaders seeking insights. This versatility allowed me to transcend categorization based on programming ecosystems in the workplace, evolving my mindset to develop platform-agnostic solutions. I believe in the adage of being a jack of all trades while still mastering one or more. I took on the challenge of discovering MongoDB drivers available for various platforms. I created sample applications to practice basic MongoDB concepts using specific drivers, and soon, everything fell into place effortlessly. What tips or advice would you share with someone who looks up to your achievement and aspires to become a certified developer in multiple languages like C#, Java, Node.JS, PHP, and Python? How can they effectively approach learning and mastering these languages? Before attempting proficiency in MongoDB across multiple languages, it's crucial to prioritize understanding fundamental concepts such as data modeling practices, CRUD operations, and indexes. Mastering MongoDB's shell, MongoSh, is essential to grasp the workings of MongoDB's read and write operations. Following this, individuals should select a programming environment they're most adept in and practice executing MongoDB operations within that ecosystem. Constructing a personal project can aid in practically observing various MongoDB concepts in action. Utilizing resources such as MongoDB Certification Learning Paths , practice tests, and MongoDB Documentation is vital for excelling in certification exams. Additionally, it's advisable to undertake the initial certification in the programming language one feels most comfortable with. Reflection is key; saving or emailing exam scores enables individuals to identify areas needing improvement for future attempts. With proficiency in C#, Java, Node.JS, PHP, and Python, how do you perceive the role of versatility in today's tech industry, especially regarding job opportunities and project flexibility? Programming languages, very much like spoken languages, are merely a medium. The most important thing is knowing what to say. The tech industry depends on problems, and developers seek solutions to them. Once they have a solution, programming languages help make those solutions a reality. It’s not hard to learn different programming languages or even to master them. Knowing the basics of different programming ecosystems can give developers an edge regarding job opportunities. It makes them flexible and enables them to make crucial and informed decisions in choosing the correct tech stack or defining good architecture for solutions. In your experience, how does fluency in multiple languages enhance collaboration and innovation within development teams, particularly in today's globalized tech landscape? Fluency or even practical awareness about programming languages or ecosystems promotes versatility in problem-solving, facilitates cross-functional collaboration, supports agile development, enables integration with legacy systems, fosters global collaboration, reduces dependency, and empowers informed decision-making, all of which are crucial for staying competitive in today's globalized tech landscape. As a MongoDB Community Creator, how do you leverage your expertise in these five languages to contribute to and engage with the broader tech community? What advice would you offer aspiring developers seeking to expand their skill set? I aim to open-source my MongoDB-focused projects across various ecosystems, accompanied by detailed articles outlining their construction. Since these projects were designed with exams in mind, they serve as skill-testing tools for developers and comprehensive guides to the various components comprising certification exams. I advocate for developers to choose a favorite language and compare others to it, as this approach facilitates a quicker and more efficient understanding of concepts. Relating new information to familiar concepts makes learning easier and more effective. The MongoDB Community Advocacy Program is a vibrant global community designed for MongoDB enthusiasts who are passionate about advocating for the platform. Our Community Creators Program welcomes members of all skill levels eager to deepen their involvement in advancing MongoDB's community and technology. We empower our members to expand their expertise, visibility, and leadership by actively engaging with and advocating for MongoDB technologies among users worldwide. Join us and amplify your impact within the MongoDB community! Elevate your career with MongoDB University 's 1,000+ learning assets. Access free courses and hands-on labs, and earn certifications to boost your skills and stand out in tech.

April 24, 2024
Applied

The Journey of MongoDB with COVESA in the Connected Vehicle Landscape

There’s a popular saying: “If you want to go fast, go alone; if you want to go far, go together.” I would argue The Connected Vehicle Systems Alliance (COVESA) in partnership with their extensive member network, turns this saying on its head. They have found a way to go fast, together and also go far, together. COVESA is an industry alliance focused on enabling the widespread adoption of connected vehicle systems. This group aims to accelerate the development of these technologies through collaboration and standardization. It's made up of various stakeholders in the automotive and technology sectors, including car manufacturers, suppliers, and tech companies. COVESA’s collaborative approach allows members to accelerate progress. Shared solutions eliminate the need for individual members to reinvent the wheel. This frees up their resources to tackle new challenges, as the community collectively builds, tests, and refines foundational components. As vehicles become more connected, the data they generate explodes in volume, variety, and velocity. Cars are no longer just a mode of transportation, but a platform for advanced technology and data-driven services. This is where MongoDB steps in. MongoDB and COVESA As the database trusted for mission-critical systems by enterprises such as Cathay Pacific , Volvo Connect , or Cox Automotive ; MongoDB has gained expertise in automotive, along with many other industries, building cross-industry knowledge in handling large-scale, diverse data sets. This in turn enables us to contribute significantly to vehicle applications and provide a unique view, especially in the data architecture discussions within COVESA. MongoDB solutions support these kinds of innovations, enabling automotive companies to leverage data for advanced features. One of the main features we provide is Atlas Device SDKs : a low-footprint, embedded database directly living on ECUs. It can synchronize data automatically with the cloud using Atlas Device Sync , our data transfer protocol that compresses the data handles conflict resolution, and only syncs delta changes, making it extremely efficient in terms of operations and maintenance. VSS: The backbone of connected vehicle data An important area of COVESA’s work is the Vehicle Signal Specification (VSS). VSS is a standardized framework used to describe data of a vehicle, such as speed, location, and diagnostic information. This standardization is essential for interoperability between different systems and components within a vehicle, as well as for external communication with other vehicles and infrastructure. VSS has been gaining more and more adoption, and it’s backed by ongoing contributions from BMW, Volvo Cars, Jaguar LR, Robert Bosch and Geotab, among others. MongoDB’s BSON and our Object-oriented Device SDKs uniquely position us to contribute to VSS implementation. The VSS data structured maps 1 to 1 to documents in MongoDB and objects in Atlas Device SDKs , which simplifies development, and speeds up applications by completely skipping any Relational Mapper layer. For every read or write, there is no need to transform the data between relational and VSS. Our insights into data structuring, querying, and management can help optimize the way data is stored and accessed in connected vehicles, making it more efficient and robust. Where MongoDB contributes MongoDB, within COVESA, finds its most meaningful contributions in areas where data complexities and community collaboration intersect. First, we can share insights into managing vast and varied data emerging from connected vehicles generating data on everything from engine performance to driver behavior. Second, we have an important role in supporting the standardization efforts, crucial for ensuring different systems within vehicles can communicate seamlessly. Our inputs can help ensure these standards are robust and practical, considering the real-world scenarios of data usage in vehicles. Some of our contributions include an Over the Air update architectural review presented at Troy COVESA’s AMM in October 2023; sharing insights about the Data Middleware PoC with BMW; and weekly contributions at the Data Expert Group. You can find some of our contributions on COVESA’s Wiki page . In essence, MongoDB's role in COVESA is about providing a unique perspective from the database management point of view, offering our understanding from different industries and use cases to support the developments towards more connected and intelligent vehicles. MongoDB, COVESA, and AWS together at CES2024 MongoDB’s most recent collaboration with COVESA was at the Consumer Electronics Show CES 2024 during which MongoDB’s Connected Vehicle solution was showcased. This solution leverages Atlas Device SDKs, such as the SDK for C++ , which enables local data storage, in-vehicle data synchronization, and also uni and bi-directional data transfer with the cloud. Below is a schematic illustrating the integration of MongoDB within the software-defined vehicle: Schema 1: End to end integration for the connected vehicle At CES 2024, MongoDB also teamed up with AWS for a compelling presentation, " AI-powered Connected Vehicles with MongoDB and AWS " led by Dr. Humza Akhtar and Mohan Yellapantula, Head of Automotive Solutions & Go To Market at AWS. The session delved into the intricacies of building connected vehicle user experiences using MongoDB Atlas. It showcased the combined strengths of MongoDB's expertise and AWS's generative AI tools, emphasizing how Atlas Vector Search unlocks the full lifecycle value of connected vehicle data. During the event, MongoDB also engaged in a conversation with The Six Five, exploring various aspects of mobility, self-driving vehicles (SDVs), and the MongoDB and AWS partnership. This discussion extended to merging IT and OT, GenAI, Atlas Edger Server, and Atlas Device SDK. Going forward At the end of the road, it’s all about enhancing the end-user experience and providing unique value propositions. Defect diagnosis based on the acoustics of the engine, improved crash assistance with mobile and vehicle telemetry data, just-in-time food ordering while on the road, in-vehicle payments, and much, much more. What all these experiences have in common is the combination of interconnected data from different systems. At MongoDB, we are laser-focused on empowering OEMs to create, transform, and disrupt the automotive industry by unleashing the power of software and data. We enable this by: Partnering with alliances such as COVESA to build a strong ecosystem or collaboration. Having one single API for In-vehicle Data Storage, Edge to Cloud Synchronization, Time Series storage, and more, improves the developer experience. Focusing on having a robust, scalable, and secure suite of services trusted by tens of thousands of customers in more than 100 countries. Together with COVESA’s vision for connected vehicles, we’re driving a future where this industry is safer, more efficient, and seamlessly integrated into the digital world. The journey is just beginning. To learn more about MongoDB-connected mobility solutions, visit the MongoDB for Manufacturing & Motion webpage . Achieving fast, reliable and compressed data exchange is one of the pillars of Software Defined Vehicles, learn how MongoDB Atlas and Edge Server can help in this short demo .

April 15, 2024
Applied

Enabling Commerce Innovation with the Power of MongoDB and Google Cloud

Across all industries, business leaders are grappling with economic uncertainty, cost concerns, disruption to supply chains, and pressure to embrace new technologies like generative AI. In this dynamic landscape, having a performant and future-proofed technology foundation is critical to your business’s success. Kin + Carta, a Premier Google Cloud Partner and MongoDB Systems Integrator Partner, recently launched the Integrated Commerce Network . The Integrated Commerce Network is an Accelerator that enables clients to modernize to a composable commerce platform and create value with their commerce data on Google Cloud with a pre-integrated solution in as little as six weeks. This article explains the concept of composable commerce and explores how MongoDB and Google Cloud form a powerful combination that enables innovation in commerce. Finally, it explains how Kin + Carta can help you navigate the complexity facing businesses today with their approach to digital decoupling. Unraveling the complexity: What is composable commerce? Why microservices and APIs? The evolution of commerce architecture Traditional monolithic architectures, once the cornerstone of commerce platforms, are facing challenges in meeting the demands of today's fast-paced digital environment. Microservices, a paradigm that breaks down applications into small, independent services, offer a solution to the limitations of monoliths. This architectural shift allows for improved agility, scalability, and maintainability. Defining composable commerce Composable commerce is a component-based, API-driven design approach that gives businesses the flexibility to build and run outstanding buying experiences free of constraints found in legacy platforms. To be truly composable, the platform must support key tenets: Support continuous delivery without downtime at the component level Have API as the contract of implementation between services, with open, industry-standard protocols providing the glue between components Be SaaS based, or portable to run on any modern public cloud environment Allow the open egress and ingress of data — no black-boxes of vendor data ownership Defining APIs and microservices APIs play a pivotal role in connecting microservices, enabling seamless communication and data exchange. This modular approach empowers businesses to adapt quickly to market changes, launch new features efficiently, and scale resources as needed. Enhanced scalability, resilience, and agility Taking a microservices approach provides businesses with options and now represents a mature and battle-tested approach with commoditized architectures, infrastructure-as-code, and open-source design patterns to enable robust, resilient, and scalable commerce workloads at lower cost and risk. Additionally, the decoupled nature of microservices facilitates faster development cycles. Development teams can work on isolated components, allowing for parallel development and quicker releases. This agility is a game-changer in the competitive e-commerce landscape, where rapid innovation is essential for staying ahead. Microservices and API-based commerce solutions (like commercetools, which is powered by MongoDB) have begun to dominate the market with their composable approach, and for good reason. These solutions remove the dead-end of legacy commerce suite software and enable a brand to pick and choose to enhance its environment on its own terms and schedule. MongoDB Atlas: The backbone of intelligent, generative AI-driven experiences As e-commerce has developed, customers are expecting more from their interactions — flat, unsophisticated experiences just don’t cut it anymore and brands need to deliver on the expectation of immediacy and contextual relevance. Taking a microservices approach enables richer and more granular data to be surfaced, analyzed, and fed back into the loop, perhaps leveraging generative AI to synthesize information that previously would have been difficult or impossible without huge computing capabilities. However, to do this well you need core data infrastructure that underpins the platform and provides the performance, resilience, and advanced features required. MongoDB Atlas on Google Cloud can play a pivotal role in this enablement. Flexible data models: Microservices often require diverse data models. MongoDB Atlas, a fully managed database service, accommodates these varying needs with its flexible schema design, which allows businesses to adapt their data structures without compromising performance. Horizontal scalability: Modern commerce moves a lot of data. MongoDB Atlas excels in distributing data across multiple nodes, ensuring that the database can handle increased loads effortlessly. Real-time data access: Delivering on expectations relies on real-time data access. MongoDB Atlas supports real-time, event-driven data updates, ensuring you are using the most up-to-date information about your customers. Serverless deployment: Rather than spend time and money managing complex database infrastructure, MongoDB Atlas can leverage serverless deployment, allowing developers to focus on building features that delight customers and impact the bottom line. Unleashing generative AI with MongoDB and Google Cloud Generative AI applications thrive on massive datasets and require robust data management. MongoDB effortlessly handles the complex and ever-evolving nature of gen AI data. This includes text, code, images, and more, allowing you to train your models on a richer data tapestry. MongoDB Atlas: Streamlined gen AI development on Google Cloud MongoDB Atlas, the cloud-based deployment option for MongoDB, integrates seamlessly with Google Cloud. Atlas offers scalability and manageability, letting you focus on building groundbreaking gen AI applications. Here's how this powerful duo functions together: Data ingestion and storage: Effortlessly ingest your training data, regardless of format, into MongoDB Atlas on Google Cloud. This data can include text for natural language processing, code for programming tasks, or images for creative generation. AI model training: Leverage Google Cloud's AI services like Vertex AI to train your gen AI models using the data stored in MongoDB Atlas. Vertex AI provides pre-built algorithms and tools to streamline model development. Operationalization and serving: Once trained, deploy your gen AI model seamlessly within your application. MongoDB Atlas ensures the smooth data flow to and from your model, enabling real-time generation. Vector search with MongoDB Atlas: MongoDB Atlas Vector Search allows for efficient retrieval of similar data points within your gen AI dataset. This is crucial for tasks like image generation or recommendation systems. Advantages of this open approach By leveraging a microservices architecture, APIs, and the scalability and flexibility of Atlas, businesses can build agile and adaptable composable platforms. Atlas seamlessly integrates with Google Cloud, providing a streamlined environment for developing and deploying generative AI models. This integrated approach offers several benefits: Simplified development: The combined power of MongoDB Atlas and Google Cloud streamlines the development process, allowing you to focus on core gen AI functionalities. Scalability and flexibility: Both MongoDB Atlas and Google Cloud offer on-demand scalability, ensuring your infrastructure adapts to your gen AI application's growing needs. Faster time to market: The ease of integration and development offered by this combination helps you get your gen AI applications to market quickly. Cost-effectiveness: Both MongoDB Atlas and Google Cloud offer flexible pricing models, allowing you to optimize costs based on your specific gen AI project requirements. Digital decoupling, a legacy modernization approach With so much digital disruption, technology leaders are constantly being challenged. Existing legacy architectures and infrastructure can be extremely rigid and hard to unravel. Over 94% of senior leaders reported experiencing tech anxiety . So how do you manage this noise, meet the needs of the business, stay relevant, and evolve your technology so that you can deliver the kinds of experiences audiences expect? Digital decoupling is a legacy modernization approach that enables large, often well-established organizations to present a unified online experience to their users, take full advantage of their data, innovate safely, and compete effectively with digital natives. Technology evolves rapidly, and an effective microservices solution should be designed with future scalability and adaptability in mind. Kin + Carta helps to ensure that your solution is not only robust for current requirements but also capable of evolving with emerging technologies and business needs. It all starts with a clear modernization strategy that allows you to iteratively untangle from legacy systems, while also meeting the needs of business stakeholders seeking innovation. Navigating commerce complexity with Kin + Carta on Google Cloud Commerce is undergoing a significant transformation, and businesses need a future-proof technology foundation to handle the demands of complex models and massive datasets. That’s why Kin + Carta launched their Integrated Commerce Network , the first commerce-related solution that’s part of Google’s Industry Value Network . With the right tools and partners, your business can be at the forefront of innovation with generative AI, through automating tasks in revolutionary new ways, creating entirely new content formats, and delivering more personalized customer experiences. The complexities of commerce transformation can be daunting. But you can master the art of digital decoupling and leverage the strengths of the Integrated Commerce Network to unlock limitless possibilities and gain an edge over your competition. Check out Kin + Carta’s guide: Flipping the script — A new vision of legacy modernization enabled by digital decoupling . Get started with MongoDB Atlas on Google Cloud today.

April 9, 2024
Applied

A Smarter Factory Floor with MongoDB Atlas and Google Cloud's Manufacturing Data Engine

The manufacturing industry is undergoing a transformative shift from traditional to digital, propelled by data-driven insights, intelligent automation, and artificial intelligence. Traditional methods of data collection and analysis are no longer sufficient to keep pace with the demands of today's competitive landscape. This is precisely where Google Cloud’s Manufacturing Data Engine (MDE) and MongoDB Atlas come into play, offering a powerful combination for optimizing your factory floor. Unlock the power of your factory data MDE is positioned to transform the factory floor with the power of cloud-driven insights. MDE simplifies communication with your factory floor, regardless of the diverse protocols your machines might use. It effortlessly connects legacy equipment with modern systems, ensuring a comprehensive data stream. MDE doesn't just collect data, it transforms it. By intelligently processing and contextualizing the information, you gain a clearer picture of your production processes in real-time with a historical pretext. It offers pre-built analytics and AI tools directly addressing common manufacturing pain points. This means you can start finding solutions faster, whether it's identifying bottlenecks, reducing downtime, or optimizing resource utilization. Conveniently, it also offers great support for integrations that can further enhance the potential of the data (e.g. additional data sinks). The MongoDB Atlas developer data platform enhances MDE by providing scalability and flexibility through automated scaling to adapt to evolving data requirements. This makes it particularly suitable for dynamic manufacturing environments. MongoDB’s document model can handle diverse data types and structures effortlessly while being incredibly flexible because of its native JSON format. This allows for enriching MDE data with other relevant data, such as supply chain logistics, for a deeper understanding of the factory business. You can gain immediate insights into your operations through real-time analytics, enabling informed decision-making based on up-to-date data. While MDE offers a robust solution for collecting, contextualizing, and managing industrial data, leveraging it alongside MongoDB Atlas offers tremendous advantages Inside the MDE integration Google Cloud’s Manufacturing Data Engine (MDE) acts as a central hub for your factory data. It not only processes and enriches the data with context, but also offers flexible storage options like BigQuery and Cloud Storage. Now, customers already using MongoDB Atlas can skip the hassle of application re-integration and make this data readily accessible for applications. Through this joint solution developed by Google Cloud and MongoDB, you can seamlessly move the processed streaming data from MDE to MongoDB Atlas using Dataflow jobs. MDE publishes the data via a Pub/Sub subscription, which is then picked up by a custom Dataflow job built by MongoDB. This job transforms the data into the desired format and writes it to your MongoDB Atlas cluster. Google Cloud’s MDE and MongoDB Atlas utilize compatible data structures, simplifying data integration through a shared semantic configuration. Once the data resides in MongoDB Atlas, your existing applications can access it seamlessly without any code modifications, saving you time and effort. The flexibility of MDE, combined with the scalability and ease of use of MongoDB Atlas, makes this a powerful and versatile solution for various data-driven use cases such as predictive maintenance and quality control, while still providing factory ownership of the data. Instructions on how to set up the dataflow job are available in the MongoDB github repository. Conclusion If you want to level up your manufacturing data analytics, pairing MDE with MongoDB Atlas provides a proven, easy-to-implement solution. It's easy to get started with MDE and MongoDB Atlas .

April 9, 2024
Applied

Unleashing Developer Potential–and Managing Costs–with MongoDB Atlas

In today's business landscape, where unpredictability has become the norm, engineering leaders have to balance the dual challenges of managing uncertainty while optimizing IT costs. Indeed, the 2024 MarginPLUS Deloitte survey—which draws on insights from over 300 business leaders—emphasizes a collective pivot towards growth initiatives and cost transformations amidst the fluctuating global economic climate. MongoDB Atlas: A developer's ally for cost-effective productivity Executives across industries want to cut costs without impacting innovation; based on the Deloitte survey , 83% of companies are looking to change how they run their business margin improvement efforts. This is where MongoDB Atlas , the most advanced cloud database service on the market, comes in. An integrated suite of data services that simplify how developers build with data, MongoDB Atlas helps teams enhance their productivity without compromising on cost efficiency by offering visibility and control over spending—balancing developer freedom with data governance and cost management. This helps organizations escape the modernization hamster wheel—or the vicious cycle of continuously updating technology without making real progress, and draining resources while failing to deliver meaningful improvements. Put another way, MongoDB gives teams more time to innovate, instead of just maintaining the status quo. Outlined below are the built-in features of MongoDB Atlas, which enable customers to get the most out of their data while also focusing on budget optimization. Strategic features for cost optimization with MongoDB Atlas Right-sizing your cluster Use MongoDB Atlas’s Cluster Sizing Guide or auto-scalability to match your cluster with your workload, optimizing resource use with options for every requirement, including Low CPU options for lighter workloads. Pausing clusters and global distribution Save costs by pausing your cluster , and securely storing data for up to 30 days with auto-resume. Furthermore, Global Clusters improve performance across regions while maintaining cost efficiency and compliance. Index and storage management Enhance performance and reduce costs with MongoDB Atlas’s Performance Advisor , which provides tailored index and schema optimizations for better query execution and potential reductions in cluster size. Strategic data management Reduce storage expenses using Online Archive for infrequently accessed data and TTL indexes for efficient Time Series data management, ensuring only essential data is stored. Securely backup data before deletion with mongodump. Enhanced spend management Use spend analysis, billing alerts , and usage insights via the Billing Cost Explorer for detailed financial management and optimization. Resource tagging and customizable dashboards provide in-depth financial reporting and visual expense tracking, supporting effective budgeting and cost optimization. Additionally, Opt for serverless instances to adjust to your workload's scale, offering a pay-for-what-you-use model that eliminates overprovisioning concerns. Transforming uncertainty into advancement MongoDB Atlas equips IT decision-makers and developers with the features and tools to balance developer productivity with strategic cost management, transforming economic uncertainty into a platform for strategic advancement. MongoDB Atlas is more than a database management solution; it’s a strategic partner in optimizing your IT spending, ensuring that your organization remains agile, efficient, and cost-effective in the face of change. Need expert assistance in taking control of your MongoDB Atlas costs? MongoDB’s Professional Services team can provide a deep-dive assessment of your environment to build a tailored optimization plan—and to help you execute. Reach out to learn how we can support your cost optimization goals! If you haven't yet set up your free cluster on MongoDB Atlas , now is a great time to do so. You have all the instructions in this DevCenter article .

April 8, 2024
Applied

From Relational Databases to AI: An Insurance Data Modernization Journey

Imagine you’re a data architect, a developer, or a data engineer at an insurance company. Management has asked you and your team to build a new AI claim adjustment system, a customer-facing LLM-powered chatbot, and an application to streamline the underwriting process. However, doing so is far from straightforward due to the challenges you face on a daily basis. The bulk of your time is spent navigating your company’s outdated legacy systems, which were built in the 1970s and 1980s. Some of these legacy platforms were written in COBOL and CICS, and today very few people on your team know how to develop and maintain those technologies. Moreover, the data models you work with are another source of frustration. Every interaction with them is a reminder of the intricate structures that have evolved over time, making data manipulation and analysis a nightmare. In sum, legacy systems are preventing your team—and your company—from innovating and keeping up with both your industry and customer demands. Whether you’re trying to modernize your legacy systems to improve operational efficiency, boost developer productivity, or if you want to build AI-powered apps that integrate with large language models (LLMs), MongoDB has a solution for that. In this post, we’ll walk you through a journey that starts with a relational data model refactored into MongoDB collections, vectorization and querying of unstructured data and, finally, retrieval augmented generation (RAG) : asking large language models (LLMs) questions about data in natural language. Identifying, modernizing, and storing the data Our journey starts with an assessment of the data sources we want to work with. As shown below, we can bucket the data into three different categories: Structured legacy data: Tables of claims, coverages, billings, and more. Is your data locked in rigid relations schemas? This tutorial is a step-by-step guide on how to migrate a real-life insurance relational model with the help of MongoDB Relational Migrator , refactoring 21 tables to only five MongoDB collections. Structured data (JSON): You might have files of policies, insurance products, or forms in JSON format. Check out our docs to learn how to insert those into a MongoDB collection. Unstructured data (PDFs, Audios, Images, etc.): If you need to create and store a numerical representation (vector embedding) of, for instance, claim-related photos of accidents or PDFs of policy guidelines, you can have a look at this blog that will walk you through the process of generating embeddings of pictures of car crashes and persisting them alongside existing fields in a MongoDB collection. Figure 1: Storing different types of data into MongoDB Regardless of the original format or source, our data has finally landed into MongoDB Atlas into what we call a Converged AI Data Store, which is a platform that centrally integrates and organizes enterprise data, including vectors, that enable the development of ML- and AI-powered applications. Accessing, experimenting, and interacting with the data It’s time to put the data to work. The Converged AI Data Store unlocks a plethora of use cases and efficiency gains, both for the business and for developers. The next step of the journey is about the different ways we can interact with our data: Database and Full Text Search: Learn how to run database queries, start from the basics and move up to advanced features such as facets, fuzzy search, autocomplete, highlighting, and more with Atlas Search . Vector Search: We can finally leverage unstructured data. The Image Search blog we mentioned earlier also explains how to create a Vector Search index and run vector queries against embeddings of photos. RAG: Combining Vector Search and the power of LLMs, it is possible to interact in natural language with our data (see Figure 2 below), asking complex questions and getting detailed answers. Follow this tutorial to become a RAG expert. Figure 2: Retrieval augmented generation (RAG) diagram where we dynamically combine our custom data with the LLM to generate reliable and relevant outputs Having explored all the different ways we can ask questions of the data, we made it to the end of our journey. You are now ready to modernize your company’s systems and finally be able to keep up with the business’ demands. What will you build next? If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases Build a ML-Powered Underwriting Engine in 20 Minutes with MongoDB and Databricks

March 14, 2024
Applied

How MongoDB Enables Digital Twins in the Industrial Metaverse

The integration of MongoDB into the metaverse marks a pivotal moment for the manufacturing industry, unlocking innovative use cases across design and prototyping, training and simulation, and maintenance and repair. MongoDB's powerful capabilities — combined with Augmented Reality (AR) or Virtual Reality (VR) technologies — are reshaping how manufacturers approach these critical aspects of their operations, while also enabling the realization of innovative product features. But first: What is the metaverse, and why is it so important to manufacturers? We often use the term, "digital twin" to refer to a virtual replication of the physical world. It is commonly used for simulations and documentation. The metaverse goes one step further: Not only is it a virtual representation of a physical device or a complete factory, but the metaverse also reacts and changes in real time to reflect a physical object’s condition. The advent of the industrial metaverse over the past decade has given manufacturers an opportunity to embrace a new era of innovation, one that can enhance collaboration, visualization, and training. The industrial metaverse is also a virtual environment that allows geographically dispersed teams to work together in real-time. Overall, the metaverse transforms the way individuals and organizations interact to produce, purchase, sell, consume, educate, and work together. This paradigm shift is expected to accelerate innovation and affect everything from design to production across the manufacturing industry. Here are some of the ways the metaverse — powered by MongoDB — is having an impact on manufacturing. Design and prototyping Design and prototyping processes are at the core of manufacturing innovation. Within the metaverse, engineers and designers can collaborate seamlessly using VR, exploring virtual spaces to refine and iterate on product designs. MongoDB's flexible document-oriented structure ensures that complex design data, including 3D models and simulations, is efficiently stored and retrieved. This enables real-time collaboration, accelerating the design phase while maintaining the precision required for manufacturing excellence. Training and simulation Taking a digital twin and connecting it to physical assets enables training beyond traditional methods and provides immersive simulations in the metaverse that enhance skill development for manufacturing professionals. VR training, powered by MongoDB's capacity to manage diverse data types — such as time-series, key-values and events — enables realistic simulations of manufacturing environments. This approach allows workers to gain hands-on experience in a safe virtual space, preparing them for real-world challenges without affecting production cycles. Gamification is also one of the most effective ways to learn new things. MongoDB's scalability ensures that training data, including performance metrics and user feedback, is efficiently handled to continuously enlarge the training modules and the necessary resources for the ever-increasing amount of data. Maintenance and repair Maintenance and repair operations are streamlined through AR applications within the metaverse. The incorporation of AR and VR technologies into manufacturing processes amplifies the user experience, making interactions more intuitive and immersive. Technicians equipped with AR devices can access real-time information overlaid onto physical equipment, providing step-by-step guidance for maintenance and repairs. MongoDB's support for large volumes of diverse data types, including multimedia and spatial information, ensures a seamless integration of AR and VR content. This not only enhances the visual representation of data from the digital twin and the physical asset but also provides a comprehensive platform for managing the vast datasets generated during AR and VR interactions within the metaverse. Additionally, MongoDB's geospatial capabilities come into play, allowing manufacturers to manage and analyze location-based data for efficient maintenance scheduling and resource allocation. The result is reduced downtime through more efficient maintenance and improved overall operational efficiency. From the digital twin to metaverse with MongoDB The advantages of a metaverse for manufacturers are enormous, and according to Deloitte many executives are confident the industrial metaverse “ will transform research and development, design, and innovation, and enable new product strategies .” However, the realization is not easy for most companies. Challenges include managing system overload, handling vast amounts of data from physical assets, and creating accurate visualizations. The metaverse must also be easily adaptable to changes in the physical world, and new data from various sources must be continuously implemented seamlessly. Given these challenges, having a data platform that can contextualize all the data generated by various systems and then feed that to the metaverse is crucial. That is where MongoDB Atlas , the leading developer data platform, comes in, providing synchronization capabilities between physical and virtual worlds, enabling flexible data modeling, and providing access to the data via a unified query interface as seen in Figure 1. Figure 1: MongoDB connecting to a physical & virtual factory Generative AI with Atlas Vector Search With MongoDB Atlas, customers can combine three systems — database, search engine, and sync mechanisms — into one, delivering application search experiences for metaverse users 30% to 50% faster . Atlas powers use cases such as similarity search, recommendation engines, Q&A systems, dynamic personalization, and long-term memory for large language models (LLMs). Vector data is integrated with application data and seamlessly indexed for semantic queries, enabling customers to build easier and faster. MongoDB Atlas enables developers to store and access operational data and vector embeddings within a single unified platform. With Atlas Vector Search , users can generate information for maintenance, training, and all the other use cases from all possible information that is accessible. This information can come from text files such as Word, from PDFs, and even from pictures or sound streams from which an LLM then generates an accurate semantic answer. It’s no longer necessary to keep dozens of engineers busy, just creating useful manuals that are outdated at the moment a production line goes through first commissioning. Figure 2: Atlas Vector Search Transforming the manufacturing industry with MongoDB In the digital twin and metaverse-driven future of manufacturing, MongoDB emerges as a linchpin, enabling cost-effective virtual prototyping, enhancing simulation capabilities, and revolutionizing training processes. The marriage of MongoDB with AR and VR technologies creates a symbiotic relationship, fostering innovation and efficiency across design, training, and simulation. As the manufacturing industry continues its journey into the metaverse, the partnership between MongoDB and virtual technologies stands as a testament to the transformative power of digital integration in shaping the future of production. Learn more about how MongoDB is helping organizations innovate with the industrial metaverse by reading how we Build a Virtual Factory with MongoDB Atlas in 5 Simple Steps , how IIoT data can be integrated in 4 steps into MongoDB, or how MongoDB drives Innovations End-To-End in the whole Manufacturing Chain .

March 12, 2024
Applied

RegData & MongoDB: Streamline Data Control and Compliance

While navigating the requirements of keeping data secure in highly regulated markets, organizations can find themselves entangled in a web of costly and complex IT systems. Whether it's the GDPR safeguarding European personal data or the Monetary Authority of Singapore's guidelines on outsourcing and cloud computing , the greater the number of regulations organizations are subjected to, particularly across multiple geographical locations, the more intricate their IT infrastructure becomes, and organizations today face the challenge of adapting immediately or facing the consequences. In addition to regulations, customer expectations have become a major driver for innovation and modernization. In the financial sector, for example, customers demand a fast and convenient user experience with real-time access to transaction info, a fully digitized mobile-first experience with mobile banking, and personalization and accessibility for their specific needs. While these sorts of expectations have become the norm, they conflict with the complex infrastructures of modern financial institutions. Many financial institutions are saddled with legacy infrastructure that holds them back from adapting quickly to changing market conditions. Established financial institutions must find a way to modernize, or they risk losing market share to nimble challenger banks with cost-effective solutions. The banking market today is increasingly populated with nimble fintech companies powered by smaller and more straightforward IT systems, which makes it easier for them to pivot quickly. In contrast, established institutions often operate across borders, meaning they must adhere to a greater number of regulations. Modernizing these complex systems requires the simultaneous introduction of new, disruptive technology without violating any regulatory constraints, akin to driving a car while changing a tire. The primary focus for established banks is safeguarding existing systems to ensure compliance with regulatory constraints while prioritizing customer satisfaction and maintaining smooth operations as usual. RegData: Compliance without risk Multi-cloud application security platform, RegData embraces this challenge head-on. RegData has expertise across a number of highly regulated markets, from healthcare to public services, human resources, banking, and finance. The company’s mission is clear—delivering a robust, auditable, and confidential data protection platform within their comprehensive RegData Protection Suite (RPS), built on MongoDB. RegData provides its customers with more than 120 protection techniques , including 60 anonymization techniques, as well as custom techniques (protection of IBANs, SSNs, emails, etc), giving them total control over how sensitive data is managed within each organization. For example, by working with RegData, financial institutions can configure their infrastructure to specific regulations, by masking, encrypting, tokenizing, anonymizing, or pseudonymizing data into compliance. With RPS, company-wide reports can be automatically generated for the regulating authorities (i.e., ACPR, ECB, EU-GDPR, FINMA, etc.). To illustrate the impact of RPS, and to debunk some common misconceptions, let’s explore before and after scenarios. Figure 1 shows the decentralized management of access control. Some data sources employ features such as Field Level Encryption (FLE) to shield data, restricting access to individuals with the appropriate key. Additionally, certain applications implement Role-Based Access Control (RBAC) to regulate data access within the application. Some even come with an Active Directory (AD) interface to try and centralize the configuration. Figure 1: Simplified architecture with no centralized access control However, each of these only addresses parts of the challenge related to encrypting the actual data and managing single-system access. Neither FLE nor RBAC can protect data that isn’t on their data source or application. Even centralizing efforts like the AD interface excludes older legacy systems that might not have interfacing functionalities. The result in all of these cases is a mosaic of different configurations in which silos stay silos, and modernization is risky and slow because the data may or may not be protected. RegData, with its RPS solution, can integrate with a plethora of different data sources as well as provide control regardless of how data is accessed, be it via the web, APIs, files, emails, or others. This allows organizations to configure RPS at a company level. All applications including silos can and should interface with RPS to protect all of the data with a single global configuration. Another important aspect of RPS is its functions with tokenization, allowing organizations to decide which columns or fields from a given data source should be encrypted according to specific standards and govern the access to corresponding tokens. Thanks to tokenization, RPS can track who accesses what data and when they access it at a company level, regardless of the data source or the application. This is easy enough to articulate but quite difficult to execute at a data level. To efficiently manage diverse data sources, fine-grained authorization, and implement different protection techniques, RegData builds RPS on top of MongoDB's flexible and document-oriented database. The road to modernization As noted, to fully leverage RegData’s RPS, all data sources should go through the RPS. RPS works like a data filter, putting in all of the information and extracting protected data on the other side, to modernize and innovate. Just integrating RegData means being able to make previously siloed data available by masking, encrypting, or anonymizing it before sending it out to other applications and systems. Together, RegData and MongoDB form a robust and proven solution for protecting data and modernizing operations within highly regulated industries. The illustration below shows the architecture of a private bank utilizing RPS. Data can only be seen in plain text to database admins when the request comes from the company’s headquarters. This ensures compliance with regulations, while still being able to query and search for data outside the headquarters. This bank goes a step further by migrating their Customer Relationship Management (CRM), core banking, Portfolio Management System (PMS), customer reporting, advisory, tax reporting, and other digital apps into the public cloud. This is achieved while still being compliant and able to automatically generate submittable audit reports to regulating authorities. Figure 2: Private bank business care Another possible modernization scheme—given RegData’s functionalities—is a hybrid cloud Operational Data Layer (ODL), using MongoDB Atlas . This architectural pattern acts as a bridge between consuming applications and legacy solutions. It centrally integrates and organizes siloed enterprise data, rendering it easily available. Its purpose is to offload legacy systems by providing alternative access to information for consuming applications, thereby breaking down data silos, decreasing latency, allowing scalability, flexibility, and availability, and ultimately optimizing operational efficiency and facilitating modernization. RegData integrates, protects, and makes data available, while MongoDB Atlas provides its inherent scalability, flexibility, and availability to empower developers to offload legacy systems. Figure 3: Example of ODL with both RegData and MongoDB In conclusion, in a world where finding the right solutions can be difficult, RegData provides a strategic solution for financial institutions to securely modernize. By combining RegData's regulatory protection and modern cloud platforms such as MongoDB Atlas, the collaboration takes on the modernizing challenge of highly regulated sectors. Are you prepared to harness these capabilities for your projects? Do you have any questions about this? Then please reach out to us at industry.solutions@mongodb.com or info@regdata.ch You can also take a look at the following resources: Hybrid Cloud: Flexible Architecture for the Future of Financial Services Implementing an Operational Data Layer

February 29, 2024
Applied

Reducing Bias in Credit Scoring with Generative AI

This post is also available in: Deutsch , Français , Español , Português , Italiano , 한국어 , 简体中文 . Credit scoring plays a pivotal role in determining who gets access to credit and on what terms. Despite its importance, however, traditional credit scoring systems have long been plagued by a series of critical issues, from biases and discrimination, to limited data consideration and scalability challenges. For example, a study of US loans showed that minority borrowers were charged higher interest rates (+8%) and rejected loans more often (+14%) than borrowers from more privileged groups. The rigid nature of credit systems means that they can be slow to adapt to changing economic landscapes and evolving consumer behaviors, leaving some individuals underserved and overlooked. To overcome this, banks and other lenders are looking to adopt artificial intelligence to develop increasingly sophisticated models for scoring credit risk. In this article, we'll explore the fundamentals of credit scoring, the challenges current systems present, and delve into how artificial intelligence (AI), in particular, generative AI (genAI) can be leveraged to mitigate bias and improve accuracy. From the incorporation of alternative data sources to the development of machine learning (ML) models, we'll uncover the transformative potential of AI in reshaping the future of credit scoring. Check out our AI resource page to learn more about building AI-powered apps with MongoDB. What is credit scoring? Credit scoring is an integral aspect of the financial landscape, serving as a numerical gauge of an individual's creditworthiness. This vital metric is employed by lenders to evaluate the potential risk associated with extending credit or lending money to individuals or businesses. Traditionally, banks rely on predefined rules and statistical models often built using linear regression or logistic regression. The models are based on historical credit data, focusing on factors such as payment history, credit utilization, and length of credit history. However, assessing new credit applicants poses a challenge, leading to the need for more accurate profiling. To cater to the underserved or unserved segments traditionally discriminated against, fintechs and digital banks are increasingly incorporating information beyond traditional credit history with alternative data to create a more comprehensive view of an individual's financial behavior. Challenges with traditional credit scoring Credit scores are integral to modern life because they serve as a crucial determinant in various financial transactions, including securing loans, renting an apartment, obtaining insurance, and even sometimes in employment screenings. Because the pursuit of credit can be a labyrinthine journey, here are some of the challenges or limitations with traditional credit scoring models that often cloud the path to credit application approval. Limited credit history: Many individuals, especially those new to the credit game, encounter a significant hurdle – limited or non-existent credit history. Traditional credit scoring models heavily rely on past credit behavior, making it difficult for individuals without a robust credit history to prove their creditworthiness. Roughly 45 million Americans lack credit scores simply because those data points do not exist for them. Inconsistent income: Irregular income, typical in part-time work or freelancing, poses a challenge for traditional credit scoring models, potentially labeling individuals as higher risk and leading to application denials or restrictive credit limits. In 2023 in the United States , data sources differ on how many people are self-employed. One source shows more than 27 million Americans filed Schedule C tax documents, which cover net income or loss from a business – highlighting the need for different methods of credit scoring for those self-employed. High utilization of existing credit: Heavy reliance on existing credit is often perceived as a signal of potential financial strain, influencing credit decisions. Credit applications may face rejection or approval with less favorable terms, reflecting concerns about the applicant's ability to judiciously manage additional credit. Lack of clarity in rejection reasons: Understanding the reasons behind rejections hinders applicants from addressing the root causes – in the UK, a study between April 2022 and April 2023 showed the main reasons for rejection included “poor credit history” (38%), “couldn’t afford the repayments” (28%), “having too much other credit" (19%) and 10% said they weren’t told why. The reasons even when given are often too vague which leaves applicants in the dark, making it difficult for them to address the root cause and enhance their creditworthiness for future applications. The lack of transparency is not only a trouble for customers, it can also lead to a penalty for banks. For example, a Berlin bank was fined €300k in 2023 for lacking transparency in declining a credit card application. Lack of flexibility: Shifts in consumer behavior, especially among younger generations preferring digital transactions, challenge traditional models. Factors like the rise of the gig economy, non-traditional employment, student loan debt, and high living costs complicate assessing income stability and financial health. Traditional credit risk predictions are limited during unprecedented disruptions like COVID-19, not taking this into account in scoring models. Recognizing these challenges highlights the need for alternative credit scoring models that can adapt to evolving financial behaviors, handle non-traditional data sources, and provide a more inclusive and accurate assessment of creditworthiness in today's dynamic financial landscape. Credit scoring with alternative data Alternative credit scoring refers to the use of non-traditional data sources (aka. alternative data) and methods to assess an individual's creditworthiness. While traditional credit scoring relies heavily on credit history from major credit bureaus, alternative credit scoring incorporates a broader range of factors to create a more comprehensive picture of a person's financial behavior. Below are some of the popular alternative data sources: Utility payments: Beyond credit history, consistent payments for utilities like electricity and water offer a powerful indicator of financial responsibility and reveal a commitment to meeting financial obligations, providing crucial insights beyond traditional metrics. Rental history: For those without a mortgage, rental payment history emerges as a key alternative data source. Demonstrating consistent and timely rent payments paints a comprehensive picture of financial discipline and reliability. Mobile phone usage patterns: The ubiquity of mobile phones unlocks a wealth of alternative data. Analyzing call and text patterns provides insights into an individual's network, stability, and social connections, contributing valuable information for credit assessments. Online shopping behavior: Examining the frequency, type, and amount spent on online purchases offers valuable insights into spending behaviors, contributing to a more nuanced understanding of financial habits. Educational and employment background: Alternative credit scoring considers an individual's educational and employment history. Positive indicators, such as educational achievements and stable employment, play a crucial role in assessing financial stability. These alternative data sources represent a shift towards a more inclusive, nuanced, and holistic approach to credit assessments. As financial technology continues to advance, leveraging these alternative data sets ensures a more comprehensive evaluation of creditworthiness, marking a transformative step in the evolution of credit scoring models. Alternative credit scoring with artificial intelligence Besides the use of alternative data, the use of AI as an alternative method has emerged as a transformative force to address the challenges of traditional credit scoring for a number of reasons: Ability to mitigate bias: Like traditional statistical models, AI models, including LLMs, trained on historical data that are biased will inherit biases present in that data, leading to discriminatory outcomes. LLMs might focus on certain features more than others or may lack the ability to understand the broader context of an individual's financial situation leading to biased decision-making. However, there are various techniques to mitigate the bias of AI models: Mitigation strategies: Initiatives begin with the use of diverse and representative training data to avoid reinforcing existing biases. Inadequate or ineffective mitigation strategies can result in biased outcomes persisting in AI credit scoring models. Careful attention to the data collected and model development is crucial in mitigating this bias. Incorporating alternative data for credit scoring plays a critical role in reducing biases. Rigorous bias detection tools, fairness constraints, and regularization techniques during training enhance model accountability: Balancing feature representation and employing post-processing techniques and specialized algorithms contribute to bias mitigation. Inclusive model evaluation, continuous monitoring, and iterative improvement, coupled with adherence to ethical guidelines and governance practices, complete a multifaceted approach to reducing bias in AI models. This is particularly significant in addressing concerns related to demographic or socioeconomic biases that may be present in historical credit data. Regular bias audits: Conduct regular audits to identify and mitigate biases in LLMs. This may involve analyzing model outputs for disparities across demographic groups and adjusting the algorithms accordingly. Transparency and explainability: Increase transparency and explainability in LLMs to understand how decisions are made. This can help identify and address biased decision-making processes. Trade Ledger , a lending software as a service (SaaS) tool, uses a data-driven approach to make informed decisions with greater transparency and traceability by bringing data from multiple sources with different schemas into a single data source. Ability to analyze vast and diverse datasets: Unlike traditional models that rely on predefined rules and historical credit data, AI models can process a myriad of information, including non-traditional data sources, to create a more comprehensive assessment of an individual's creditworthiness, ensuring that a broader range of financial behaviors is considered. AI brings unparalleled adaptability to the table: As economic conditions change and consumer behaviors evolve, AI-powered models can quickly adjust and learn from new data. The continuous learning aspect ensures that credit scoring remains relevant and effective in the face of ever-changing financial landscapes. The most common objections from banks to not using AI in credit scoring are transparency and explainability in credit decisions. The inherent complexity of some AI models, especially deep learning algorithms, may lead to challenges in providing clear explanations for credit decisions. Fortunately, the transparency and interpretability of AI models have seen significant advancements. Techniques like SHapley Additive exPlanations (SHAP) values and Local Interpretable Model-Agnostic Explanations (LIME) plots</a,> and several other advancements in the domain of Explainable AI (XAI) now allow us to understand how the model arrives at specific credit decisions. This not only enhances trust in the credit scoring process but also addresses the common critique that AI models are "black boxes." Understanding the criticality of leveraging alternative data that often comes in a semi or unstructured format, financial institutions work with MongoDB to enhance their credit application processes with a faster, simpler, and more flexible way to make payments and offer credit: Amar Bank, Indonesia's leading digital bank , is combatting bias by providing microloans to people who wouldn’t be able to get financial services from traditional banks (unbanked and underserved). Traditional underwriting processes were inadequate for customers lacking credit history or collateral so they have streamlined lending decisions by harnessing unstructured data. Leveraging MongoDB Atlas, they developed a predictive analytics model integrating structured and unstructured data to assess borrower creditworthiness. MongoDB's scalability and capability to manage diverse data types were instrumental in expanding and optimizing their lending operations. For the vast majority of Indians, getting credit is typically challenging due to stringent regulations and a lack of credit data. Through the use of modern underwriting systems, Slice, a leading innovator in India’s fintech ecosystem , is helping broaden the accessibility to credit in India by streamlining their KYC process for a smoother credit experience. By utilizing MongoDB Atlas across different use cases, including as a real-time ML feature store, slice transformed their onboarding process, slashing processing times to under a minute. slice uses the real-time feature store with MongoDB and ML models to compute over 100 variables instantly, enabling credit eligibility determination in less than 30 seconds. Transforming credit scoring with generative AI Besides the use of alternative data and AI in credit scoring, GenAI has the potential to revolutionize credit scoring and assessment with its ability to create synthetic data and understand intricate patterns, offering a more nuanced, adaptive, and predictive approach. GenAI’s capability to synthesize diverse data sets addresses one of the key limitations of traditional credit scoring – the reliance on historical credit data. By creating synthetic data that mirrors real-world financial behaviors, GenAI models enable a more inclusive assessment of creditworthiness. This transformative shift promotes financial inclusivity, opening doors for a broader demographic to access credit opportunities. Adaptability plays a crucial role in navigating the dynamic nature of economic conditions and changing consumer behaviors. Unlike traditional models, which struggle to adjust to unforeseen disruptions, GenAI’s ability to continuously learn and adapt ensures that credit scoring remains effective in real-time, offering a more resilient and responsive tool for assessing credit risk. In addition to its predictive prowess, GenAI can contribute to transparency and interpretability in credit scoring. Models can generate explanations for their decisions, providing clearer insights into credit assessments, and enhancing trust among consumers, regulators, and financial institutions. One key concern however in making use of GenAI is the problem of hallucination, where the model may present information that is either nonsensical or outright false. There are several techniques to mitigate this risk and one approach is using the Retrieval Augment Generation (RAG) approach. RAG minimizes hallucinations by grounding the model’s responses in factual information from up-to-date sources, ensuring the model’s responses reflect the most current and accurate information available. Patronus AI , for example, leverages RAG with MongoDB Atlas to enable engineers to score and benchmark large language models (LLMs) performance on real-world scenarios, generate adversarial test cases at scale, and monitor hallucinations and other unexpected and unsafe behavior. This can help to detect LLM mistakes at scale and deploy AI products safely and confidently. Another technology partner of MongoDB is Robust Intelligence . The firm’s AI Firewall protects LLMs in production by validating inputs and outputs in real-time. It assesses and mitigates operational risks such as hallucinations, ethical risks including model bias and toxic outputs, and security risks such as prompt injections and personally identifiable information (PII) extractions. As generative AI continues to mature, its integration into credit scoring and the broader credit application systems promises not just a technological advancement, but a fundamental transformation in how we evaluate and extend credit. A pivotal moment in the history of credit The convergence of alternative data, artificial intelligence, and generative AI is reshaping the foundations of credit scoring, marking a pivotal moment in the financial industry. The challenges of traditional models are being overcome through the adoption of alternative credit scoring methods, offering a more inclusive and nuanced assessment. Generative AI, while introducing the potential challenge of hallucination, represents the forefront of innovation, not only revolutionizing technological capabilities but fundamentally redefining how credit is evaluated, fostering a new era of financial inclusivity, efficiency, and fairness. If you would like to discover more about building AI-enriched applications with MongoDB, take a look at the following resources: Digitizing the lending and leasing experience with MongoDB Deliver AI-enriched apps with the right security controls in place, and at the scale and performance users expect Discover how slice enables credit approval in less than a minute for millions

February 20, 2024
Applied

Safety Champion Builds the Future of Safety Management on MongoDB Atlas, with genAI in Sight

Safety Champion was born in Australia in 2015, out of an RMIT university project aiming to disrupt the safety management industry, still heavily reliant on paper-based processes and lagging in terms of digitisation, and bring it to the cloud. Most companies today need to comply with strict workplace safety policies. This is true for industries reliant on manual workers, such as manufacturing, transport and logistics, construction, and healthcare, but also for companies dealing with digital workers, both working in the office or remotely. To do this, organisations rely on safety management processes and systems that help them comply with government and industry-led regulations, as well as keep their employees safe. Whether it’s legal obligations about safety reporting, management of employees and contractors, or ways to implement company-wide safety programs, Safety Champion’s digital platform provides customers more visibility and tracking over safety programs, and a wealth of data to help make evidence-based safety decisions. "Data is core to our offering, as well as core to how next-generation safety programs are being designed and implemented. With paper-based processes, you simply can’t get access to rich data, connect data sets easily, or uncover organisation-wide insights and patterns that can help drive efficiencies and improve safety outcomes," explains Craig Salter, Founder of Safety Champion. MongoDB Atlas: Unlocking the power of data and analytics to improve safety outcomes for customers Safety Champion started using the self-managed version of MongoDB, and shortly after that in 2017 moved onto MongoDB Atlas which was more cost-effective, meant less overhead and not having to manage the day-to-day tasks required to keep a database up and running. The main challenge is that industry standards and policies around safety vary significantly for every company - the safety risks of an office-based business of digital workers are widely different from the risks workers on a manufacturing plant are exposed to, making data collection and itemisation for deeper insights very complex. MongoDB’s document model, its flexibility, and its ability to handle complex sets of data combined with Atlas’ ease of use for developers made it the perfect fit for Safety Champion. "It was very easy to get started on MongoDB, but also super easy and quick to get applications developed and brought to market," says Sid Jain, Solution Architect for Safety Champion. "The performance optimisation we saw using MongoDB Atlas was significant, and it freed up a lot of time from our developers so they could focus on what matters most to our business, instead of worrying about things like patching, setting up alerts, handling back-ups, and so on." The use of MongoDB Charts also gives Safety Champions’ customers access to important analytics that can be presented in visual forms, fitting very specific use cases and internal audiences. This helps organisations using Safety Champion improve decision-making by presenting very concrete data and graphs that can fuel evidence-based safety decisions. "MongoDB Atlas helps drive efficiencies for our clients, but it also helps safety and operations managers to have a voice in making important safety decisions because they are backed by strong data-led evidence. Connecting data sets means the ability to have a much deeper, richer view of what’s happening and what needs to be done," says Salter. Managing exponential growth: 2024, the year of scaling up, generative AI, Search, and much more Before 2020, Safety Champions was still a small start-up, with its platform managing about 5,000 documents a month - these include incident records, checklists, inspection reports, actionable tasks, task completion reports, and more. The COVID pandemic forced organisations to move their safety processes online and comply with a whole new set of safety measures and policies, which saw the company’s business explode: triple-digit annual growth between 2021 and 2023, a dev team that tripled in size, over 2,000 customers, and now up to 100,000 documents handled per month. "As our company kept growing, with some of our new customers handling tens of thousands of safety documents every month - we knew we needed to enable even more scale and future proof ourselves for the next years to come," explains Salter. "We also knew that if we wanted to take advantage of MongoDB’s capabilities in generative AI, search, multi-region, and more, which a lot of our customers were asking for, we needed to set some strong data foundations." Safety Champion is now in the process of upgrading to MongoDB 6.0, which will offer its clients more speed, especially when handling larger and more complex queries. MongoDB Search is now also available system-wide, allowing search queries to be performed across all the modules a client has added records for. "Since many modules allow linking records to each other, allowing a single search query to find and return records from multiple modules makes a world of difference. Developers no longer have to maintain other data systems and the extraction, transformation, and sync of data between MongoDB and search index happens seamlessly, greatly reducing the Ops burden on dev teams," explains Jain. The use of multi-region functionalities within MongoDB Atlas means customers, especially global ones operating in multiple geographic regions, will be able to segregate data and ensure they meet regulatory requirements around data hosting and security. Lastly, Safety Champion is exploring the potential of generative AI with plans to start using MongoDB Vector Search , later in 2024. Some of the use cases the company is already investigating include semantic insights, understanding textual data that employees enter in forms, applying LLMs to that data, and extracting helpful information from it. "Every client wants more analytics, more insights, and more high-level meaning out of data. It’s not just about making it easier to enter data and seeing safety incidents, it’s about what it means and decisions that can be made from a safety perspective," says Salter. "The new version of the Safety Champion platform powered by MongoDB Atlas means we are fully ready to dive into the next phase of our evolution as a business and offer features such as generative AI which will take both Safety Champions and our customers to the next era of safety management."

February 14, 2024
Applied

Ready to get Started with MongoDB Atlas?

Start Free