rust

2985 results

Goodnotes Finds Marketplace Success Using MongoDB Atlas

In the fast-paced world of app development, creating a feature-rich digital marketplace that scales effectively can be challenging. Goodnotes was founded in 2010 with the aim of replacing traditional paper notebooks with a digital alternative that reimagines the note-taking experience. Since then, the app has gone through several iterations and grown in popularity, now with more than 24 million monthly active users and 2.5 billion notes. The team behind Goodnotes spoke at MongoDB.local Hong Kong in September 2024. They shared their journey of using MongoDB Atlas and MongoDB Atlas Search to build and run a comprehensive marketplace that expands the company’s offerings, catering to its growing number of content creators. “At the beginning of 2023, we launched a pop-up shop, which was a very simple version of the marketplace, to test the water, and we realized it got really popular,” said Xing Dai, Principal Backend Engineer at Goodnotes. The full Goodnotes Marketplace launched in August 2024 as a space where content creators can enhance their note-taking experience by purchasing additional content, such as planners, stickers, and textbooks. Building a robust digital marketplace with MongoDB Atlas “The first and the most difficult challenge [was] that we are a multiplatform app, and if you want to launch on multiple platforms, you need to support different app stores as well as [the] web,” said Dai. Using MongoDB Atlas, Dai’s team created a fully configurable marketplace that would be accessible on different mobile and desktop platforms and the web. The initial pop-up shop’s infrastructure consisted of a Payload content management system connected to a MongoDB database. However, building a full-fledged marketplace was more challenging. The architecture needed to be scalable and include search, ordering, and customization capabilities. “With [MongoDB] Atlas, it was really easy to add the in-app purchase and build the subscription infrastructure to manage the purchase workflow,” said Dai. The Goodnotes team introduced NestJS—a JavaScript API framework—to build client APIs. It then developed a user-friendly portal for the operations team and for creators who want to upload new products. Finally, the team built a full event-based data pipeline on top of MongoDB. “What’s nice is that everything on the marketplace is actually configurable in the backend,” said Dai. “We don’t need to do anything other than configuring what we need to store in the database, and the iOS client will connect it to the backend.” “When we want to extend the marketplace to other platforms, nothing needs to be changed,” Dai added. “We only need to configure different shops for different platforms.” This means that Goodnotes can easily make its marketplace available on different app platforms, such as Apple and Android, and on the web. Adding searches, charts, and soon AI As Goodnotes added more products to its marketplace, users had difficulty finding what they wanted. Despite having limited resources, the Goodnotes team endeavored to build a comprehensive search function. Using MongoDB Atlas Search and MongoDB Atlas Triggers , the team built a search function that would generate the search view collection by-products and attributes, combining them into one collection. The team then added an Atlas Search index for the search field with an API exposing the search. “We also added an auto-complete function, which is very similar to search, in the sense that we just had to create a function to generate aggregated collections, trigger it using [MongoDB] Atlas Triggers, and add the index and expose it in the marketplace,” said Dai. The search function is now popular among marketplace users, making it quick and easy for them to find what they are looking for. Goodnotes also regularly uses MongoDB Atlas Charts . For example, it creates charts showing how many products there are in the system over time. One of the key next steps for Goodnotes involves using generative AI to translate product descriptions and content into different languages (the app is currently available in 11 languages). The team also wants to introduce personalized recommendations for a more tailored experience for each user. Ending the MongoDB.local presentation, Dai said: “MongoDB made it very fast and easy to build the whole marketplace and our search feature on top of the database using [MongoDB] Atlas Search. The solution scales, and so far we haven’t had any performance issues.” Visit our product page to learn more about MongoDB Atlas .

December 10, 2024

Managing Data Corruption in the Cloud

Key Takeaways Silent data corruption—rare events in which data becomes corrupt in a way that is not readily detected—can impact systems of all kinds across the software industry. MongoDB Atlas, our global cloud database service, operates at petabyte scale, which requires sophisticated solutions to manage the risk of silent data corruption. Because MongoDB Atlas itself relies on cloud services, the systems we have engineered to safeguard customer data have to account for limited access to the physical hardware behind our infrastructure. The systems we have implemented consist of software-level techniques for proactively detecting and repairing instances of silent data corruption. These systems include monitoring for checksum failures and similar runtime evidence of corrupt data, methods of identifying corrupt documents by leveraging MongoDB indexes and replication, and processes for repairing corrupt data by utilizing the redundant replicas. Introduction: Hardware corruption in the cloud As a cloud platform, MongoDB Atlas promises to free its customers from the work of managing hardware. In our time developing Atlas, however, some hardware problems have been challenging to abstract away. One of the most notable of these is silent data corruption. No matter how well we design our software, in rare cases hardware can silently fail in ways that compromise data. Imagine, for example, a distance sensor that detects an obstacle 10 meters away. Even if the many layers of software handling the sensor’s data function flawlessly (application, network, database, operating system, etc.), if an unlucky memory chip has a single bit flipped by radiation the value of this measurement could mutate to something like 26. 1 The consequences of this botched measurement would depend on how this data is used: in some cases it may introduce a blip in a vast amount of research data, 2 but in the wrong system it could be dangerous. Despite the rarity of events like this, global cloud systems like MongoDB Atlas operate at such a scale that these events become statistically inevitable over time, even in the presence of existing manufacturer defect screening. Our platform currently stores petabytes of data and operates nearly half a million virtual machines in cloud datacenters in dozens of countries; even random failures with odds as low as one in a hundred thousand become likely to appear in such an environment. Complicating this further is the reality that silent data corruption has many possible causes beyond rare, random failures like the example of radiation above. Recent research has identified notable rates of data corruption originating from defective CPUs in large-scale data centers, 3 and corruption can just as easily originate from bugs in operating systems or other software. Considering this scope, and with limited levels of access to the cloud hardware we use to power MongoDB Atlas, how can we best stay ahead of the inevitability of silent data corruption affecting customers on our platform? Our response to this problem has been to implement features both in the MongoDB database and the orchestration software that powers MongoDB Atlas for continuously detecting and repairing silent data corruption. Our systems are designed to be proactive, identifying potential issues before they ever affect customer data or operations. The way we use these systems can be described in three steps. First, Atlas proactively monitors for signals of corrupt data from across our fleet of databases by checking for certain logical truths about data in the course of runtime operations. Then, in the case that evidence of corruption is identified, we utilize features in MongoDB for scanning databases to pinpoint the location of corrupt data, narrowing down the site of corruption to specific documents. Finally, once we have enough information to identify a remediation plan, we repair corruption in coordination with our customers by leveraging the redundancy of MongoDB’s replica set deployment model. As a whole this approach gives us early visibility into new types of data corruption that emerge in our fleet, as well as the tools we need to pinpoint and repair corruption when it occurs. Proactively monitoring for evidence of corruption Fortunately for anyone interested in managing silent data corruption at scale, databases tend to tell you a lot about what they observe. In the course of a single day, the hundreds of thousands of database processes in the Atlas fleet generate terabytes of database logs describing their operations: connections received from clients, the details of startup and shutdown procedures, queries that are performing poorly. At their worst, logs at this scale can be expensive to store and difficult to parse, but at their best they are an indispensable diagnostic tool. As such, the first step of our strategy for managing corruption in Atlas is strategic log management. There are several points in the course of a MongoDB database’s operations where we can proactively validate logical assumptions about the state of data and emit messages if something appears to be corrupt. The most fundamental form of this validation we perform is checksum validation. A checksum is a very small piece of information that is deterministically generated from a much larger piece of information by passing it through a mathematical function. When the storage engine for an Atlas cluster writes data to disk, each block–or individual unit of data written–is accompanied by a checksum of the data. When that block is later read from disk, the storage engine once again passes the data through the checksum function and verifies that the output matches the checksum that was originally stored. If there is a mismatch, we have a strong reason to suspect that the stored information is corrupt; the database process then emits a log line indicating this and halts further execution. You can see this behavior in the MongoDB source code here . Figure 1: Checksum validation fails after corruption is introduced on the disk level. In addition to checksums, there are other opportunities for checking basic assumptions about the data we are handling during routine MongoDB operations. For example, when iterating through a list of values that is expected to be in ascending order, if we find that a given value is actually less than the one that preceded it we can also reasonably suspect that a piece of information is corrupt. Similar forms of validation exist in dozens of places in the MongoDB database. Successfully leveraging these types of runtime errors in the context of the entire Atlas fleet, however, comes with some challenges. We need to quickly detect these messages from among the flood of logs generated by the Atlas fleet, and, importantly, do so in a way that maintains data isolation for the many individual customer applications running on Atlas. Database logs, by design, reveal a lot about what is happening in a system; as the creators of a managed database service, exposing the full contents of database logs to our employees for corruption analysis is a non-starter, and so we need a more nuanced method of detecting these signals. To solve this problem, we implemented a system for detecting certain classes of error as Atlas database logs are generated and emitting high-level metadata that can be analyzed by our teams internally without revealing sensitive information about the content or operations of a given database. To describe this system, it is first useful to understand a pair of concepts that we often reference internally, and which play an important role in the development of the Atlas platform, the data plane and the control plane. The data plane describes the systems that manage the data that resides in a given Atlas cluster. It consists of the virtual network containing the cluster’s resources, the virtual machines and related resources hosting the cluster’s database processes, and storage for diagnostic information like database logs. As a whole, it consists of many thousands of individual private networks and virtual machines backing the Atlas fleet of database clusters. The control plane, on the other hand, is the environment in which the Atlas management application runs. It consists of separate networks hosting our own application processes and backing databases, and stores all of the operational metadata required for running Atlas including, for example, metadata about the configurations of the clusters that constitute the Atlas fleet. Figure 2: An Agent observes log line indicative of data corruption and communicates this to to the Atlas Control Plane. The flow of information between the two planes only occurs on a limited set of vectors, primary among these being the MongoDB Agent, a background process that runs locally on virtual machines in the Atlas data plane. The Agent serves as the primary orchestrator of a cluster’s database processes. Whenever an Atlas customer requests a change to their cluster–for example, upgrading the version of their database–their request results in some modification to metadata that resides in the control plane which is then picked up by the Agents in the data plane. The Agents then begin to interact with the individual database processes of the cluster to bring them to the desired state. The Agent, with its ability to access database logs inside the data plane, provides the tool we need to detect critical logs in a controlled manner. In fact, at the time we implemented the feature for ingesting these logs, the Agent was already capable of tailing MongoDB logs in search of particular patterns. This is how the Performance Advisor feature works, in which the Agent looks for slow query logs above a certain operation duration threshold to alert users of potentially inefficient data queries. For the purposes of corruption detection we introduced a new feature for defining additional log patterns for the Agent to look for: for example, a pattern that matches the log line indicating an invalid checksum when data is read from disk. If the Agent observes a line that matches a specified pattern it will send a message to the control plane reporting when the message was observed, in which process, along with high-level information–such as an error code–but without further information about the activity of the cluster. The next step of this process, once evidence of corruption is detected, is to assess the extent of the problem and gather additional information to inform our response. This brings us to the next piece of our corruption management system: once we become aware of the possibility of corruption, how do we pinpoint it and determine a remediation plan? Scanning databases to pinpoint identified corruption So far, we have outlined our system for detecting runtime errors symptomatic of corrupt data. However, the detection of these errors by itself is not sufficient to fully solve the problem of data corruption in a global cloud database platform. It enables early awareness of potential corruption within the Atlas fleet, but when it is time to diagnose and treat a specific case we often need more exhaustive tools. The overall picture here is not unlike the treatment of an illness: so far, what we have described is our system for detecting symptoms. Once symptoms have been identified, further testing may be needed to determine the correct course of treatment. In our case, we may need to perform further scanning of data to identify the extent of corruption and the specific information affected. The ability to scan MongoDB databases for corruption relies on two of the most fundamental concepts of a database, indexes and replication . These two basic features of MongoDB each come with certain simple logical assumptions that they adhere to in a healthy state. By scanning for specific index entries or replicated data that violate these assumptions, we are able to pinpoint the location of corrupt data, a necessary step towards determining a remediation path. Indexes–data structures generated by a database to allow for quick lookup of information–relate to the contents of a database following specific logical constraints. For example, if a given collection is using an index on the lastName field and contains a document with a lastName value of “Turing,” the value “Turing” should be present in that collection’s index keys. A violation of this constraint, therefore, could point to the location of corrupt data; the absence of an in-use lastName value in the index would indicate that either the index has become corrupt or the lastName value on the document itself has become corrupt. Because almost all indexes are specified by the customer, Atlas does not have control over how the data in a cluster is indexed. In practice, though, the most frequently-accessed data tends to be highly indexed, making index scanning a valuable tool in validating the most critical data in a cluster. Replicated data, similarly, adheres to certain constraints in a healthy state: namely, that replicas of data representing the same point in time should be identical to one another. As such, replicated data within a database can also be scanned at a common point in time to identify places where data has diverged as a result of corruption. If two of three replicas in a cluster show a lastName value of “Turing” for a given document but the third shows “Toring” 4 , we have a clear reason to suspect that this particular document’s data has become corrupt. Since all Atlas clusters are deployed with at least three redundant copies of data, replication is always available to be leveraged when repairing corruption on Atlas. This is, of course, easier said than done. In practice, performing integrity scanning of indexes and replicated data for a very large database requires processing a large amount of complex data. In the past, performing such an operation was often infeasible on a database cluster that was actively serving reads and writes. The db.collection.validate command was one of the first tools we developed at MongoDB for performing integrity scans of index data, but it comes with certain restrictions. The command obtains an exclusive lock on the collection it is validating, which means it will block reads and writes on the collection until it is finished. We still use this command as part of our corruption scanning strategy, but because of its limitations this means it is often only feasible to run on an offline copy of a database restored from a backup snapshot. This can be expensive, and comes with the overhead of managing additional hardware for performing validations on offline copies. With this in mind, we have been developing new tools for scanning for data corruption that are more feasible to run in the background of a cluster actively serving traffic. Our latest tools for detecting inconsistencies in replica sets utilize the replication process to perform background validation of data while a cluster is processing ordinary operations, and can be rate-limited based on the available resources on the cluster. When this process is performed, the primary node will begin by iterating through a collection in order, pulling a small batch of data into memory that stays within the bounds of a specified limit. It will make note of the range of the data being analyzed and produce an MD5 hash , writing this information to an entry in the oplog , a transaction log maintained by MongoDB that is replayed by secondary nodes in the database. When the secondaries of the cluster encounter this entry in their copies of the oplog, they perform the same calculation based on their replicas of the data, generating their own hash belonging to the indicated range of data at the specified point in time. By comparing a secondary’s hash with the original hash recorded by the primary, we can determine whether or not this batch of data is consistent between the two nodes. The result of this comparison (consistent or inconsistent) is then logged to an internal collection in the node’s local database . In this manner small batches of data are processed until the collection has been fully scanned. Figure 3: Data consistency between replicas is validated by leveraging the oplog. This form of online data consistency scanning has allowed us to scan our own internal databases without interruption to their ordinary operations, and is a promising tool for expanding the scale of our data corruption scanning without needing to manage large amounts of additional hardware for performing offline validations. We do, nonetheless, recognize there will be some cases where running an online validation may be unviable, as in the cases of clusters with very limited available CPU or memory. For this reason, we continue to utilize offline validations as part of our strategy, trading the cost of running additional hardware for the duration of the validation for complete isolation between the validation workload and the application workload. Overall, utilizing both online and offline approaches in different cases gives us the flexibility we need to handle the wide range of data characteristics we encounter. Repairing corruption Once the location of corrupt data has been identified, the last step in our process is to repair it. Having several redundant copies of data in an Atlas cluster means that more often than not it is straightforward to rebuild the corrupt data. If there is an inconsistency present on a given node in a cluster, that node can be rebuilt by triggering an initial sync and designating a known healthy member as the sync source. Triggering this type of resync is sufficient to remediate both index inconsistencies and replication inconsistencies 5 as long as there is at least one known, healthy copy of data in a cluster. While it is typically the case that it is straightforward to identify a healthy sync source when repairing corruption–truly random failures would be unlikely to happen on more than one copy of data–there are some additional considerations we have to make when identifying a sync source. A given node in an Atlas cluster may have already had its data resynced at some point in the past in the course of ordinary operations. For example, if the cluster was migrated to a new set of hardware in the past, some or all nodes in the cluster may have already been rebuilt at least once before. For this reason, it is important for us to consider the history of changes in the cluster in the time after corruption may have been introduced to rule out any nodes that may have copied corrupt data, and to separately validate the integrity of the sync source before performing any repair. Once we are confident in a remediation plan and have coordinated any necessary details with our customers, we leverage internal features for performing automated resyncs on Atlas clusters to rebuild corrupt data. Very often, these repairs can be done with little interruption to a database’s operations. Known healthy nodes can continue to serve application traffic while data is repaired in the background. Internal-facing functionality for repairing Atlas clusters has existed since the early days of the platform, but in recent years we have added additional features and levels of control to facilitate corruption remediation. In particular, in many cases we are able to perform optimized versions of the initial sync process by using cloud provider snapshots of healthy nodes to circumvent the sometimes-lengthy process of copying data directly between replica set members, reducing the overall time it takes to repair a cluster. In the rarer event that we need to perform a full logical initial sync of data, we can continue to perform this mode of data synchronization as well. After repair has completed, we finish by performing follow-up scanning to validate that the repair succeeded. We are still hard at work refining our systems for detecting and remediating data corruption. At the moment, much of our focus is on making our scanning processes as performant and thorough as possible and continuing to reduce the time it takes to identify new instances of corruption when they occur. With these systems in place it is our intention to make silent data corruption yet another detail of hardware management that the customers of Atlas don’t need to lose any sleep over, no matter what kinds of rare failures may occur. Join our MongoDB Community to learn about upcoming events, hear stories from MongoDB users, and connect with community members from around the world. Acknowledgments The systems described here are the work of dozens of engineers across several teams at MongoDB. Significant developments in these areas in recent years were led by Nathan Blinn, Xuerui Fa, Nan Gao, Rob George, Chris Kelly, and Eric Sedor-George. A special thanks to Eric Sedor-George for invaluable input throughout the process of writing this post. 1 Instances of alpha radiation, often in the form of cosmic rays, introducing silent data corruption have been explored in many studies since the 1970s. For a recent overview of literature on this topic, see Reghenzani, Federico, Zhishan Guo, and William Fornaciari. "Software fault tolerance in real-time systems: Identifying the future research questions." ACM Computing Surveys 55.14s (2023): 1-30 . 2 Five instances of silent data corruption introducing inaccurate results in scientific research were identified in a review of computing systems at Los Alamos National Laboratory in S. E. Michalak, et al , "Correctness Field Testing of Production and Decommissioned High Performance Computing Platforms at Los Alamos National Laboratory," SC '14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2014 3 A recent survey of the CPU population of Alibaba Cloud identified a corruption rate of 3.61 per 10,000 CPUs, see Wang, Shaobu, et al. "Understanding Silent Data Corruptions in a Large Production CPU Population." Proceedings of the 29th Symposium on Operating Systems Principles. 2023. Research by Google on its own datacenters identified CPU corruption on “the order of a few mercurial cores per several thousand machines,” see Hochschild, Peter H., et al. "Cores that don't count." Proceedings of the Workshop on Hot Topics in Operating Systems. 2021. Research by Meta on its own datacenters found that “hundreds of CPUs” demonstrated silent data corruption across “hundreds of thousands of machines,” see Dixit, Harish Dattatraya, et al. "Silent data corruptions at scale." arXiv preprint arXiv:2102.11245 (2021). 4 In practice, it is rare that corrupt data results in a legible value; more often data in this state would be simply illegible. 5 There are other methods of rebuilding indexes in a MongoDB database beyond what is described here; see db.collection.reIndex for information on rebuilding indexes without triggering an initial sync.

December 9, 2024

What’s New From MongoDB at AWS re:Invent 2024

As thousands of attendees make their way home after a week in Vegas—a week packed with learning, product launches, and round-the-clock events—we thought we’d reflect on the show’s highlights. MongoDB was excited to showcase our latest integrations and solutions with Amazon Web Services (AWS), which range from new ways to optimize generative AI, to faster, more cost-effective methods for modernizing applications. But first, we want to thank our friends at AWS for recognizing MongoDB as the AWS Technology Partner of the Year NAMER! This prestigious award recognizes AWS Technology Partners that are using AWS to lower costs, increase agility, and innovate faster. Announced during the re:Invent Partner Awards Gala, the Technology Partner of the Year Award is a testament to the specialization, innovation, and cooperation MongoDB and AWS have jointly brought to customers this year. In addition, MongoDB also received AWS Partner of the Year awards for Italy, Turkey, and Iberia. These awards follow wins in the ASEAN Global Software Partner of the Year and Taiwan Technology Partner of the Year categories earlier in the year, further demonstrating the global reach and popularity of the world’s most popular document database! Harnessing the potential of gen AI If 2024 (and 2023, and 2022…) was the year of gen AI excitement, then 2025 may turn out to be marked by realistic gen AI implementation. Indeed, we’ve seen customers shift their 2025 AI focus toward optimizing resource-intensive gen AI workloads to drive down costs—and to get the most out of this groundbreaking technology. Retrieval-augmented generation (RAG), one of the main ways companies use their data to customize the output of foundation models, has become the focus of this push for optimization. Customers are looking for easier ways to fine-tune their RAG systems, asking questions like, “How do I evaluate the efficiency and accuracy of my current RAG workflow?” To that end, AWS and MongoDB are introducing new services and technologies for enterprises to optimize RAG architecture compute costs, while also maintaining accuracy. First up is vector quantization. By reducing vector storage and memory requirements while preserving performance, these capabilities empower developers to build AI-enriched applications with more scale—and at a lower cost. Leading foundation models like Amazon Titan are already compatible with vector quantization, helping to maintain high accuracy of generated responses while simultaneously reducing costs. You can read more about vector quantization on our blog. As for RAG evaluation, AWS has launched a new feature for Amazon Bedrock called, naturally, RAG Evaluator. This tool allows Bedrock users to evaluate and monitor RAG Apps natively within the Bedrock environment, eliminating the need for third-party frameworks to run tests and comparisons. As a Knowledge Base for Amazon Bedrock, MongoDB Atlas is ready on day one to take advantage of Bedrock RAG Evaluator, allowing companies to gauge and compare the quality of their RAG Apps across different applications. The RAG Evaluator, built on several joint integrations and solutions AWS and MongoDB released in 2024, and vector quantization together can streamline the deployment of enterprise generative AI. For example, in October MongoDB, Anthropic, and AWS announced a joint solution to create a memory-enhanced AI agent . Together, the three partners offer enterprise-grade, trusted, secure technologies to build generative AI apps quickly and flexibly using a family of foundation models in a fully managed environment. Overall, MongoDB and AWS are making it easier—and more cost-effective—for developers to build innovative applications that harness the full potential of generative AI on AWS. From cars to startups to glue MongoDB and AWS have been hard at work on a number of other solutions for developers across industries. Here’s a quick roundup: AWS Amplify + AppSync + MongoDB For startups, or for any organization looking to quickly test and launch applications, speed is everything. That’s why MongoDB teamed up with AWS to create a full-stack solution that provides developers with the same high standards of performance and scalability they would demand for any app. By combining AWS Amplify, AWS AppSync, and MongoDB Atlas, AWS and MongoDB have created a full-stack solution that enables seamless front-end development, robust and scalable backend services, out-of-the-box CI/CD, and a flexible and powerful database solution, allowing developers to drastically reduce the coding time required to launch new applications. Check out this tutorial and repository for a starter template . Digital twins on AWS CMS For those in the automotive sector, MongoDB and AWS have developed a connected mobility solution to help remove the undifferentiated integration, or “technical plumbing” work, of connecting vehicles to the cloud. When used together, Connected Mobility Solution (CMS) on AWS and MongoDB Atlas help accelerate the development of next-generation digital twin use cases and applications, including connected car use cases. MongoDB’s document model allows easy and flexible modeling and storage of connected vehicle sensor data. Read our joint blog with AWS to learn how the MongoDB Atlas document model helps with data modeling of connected vehicles data and how this capability can be leveraged via AWS Automotive Cloud Developer Portal (ACDP). AWS Glue + MongoDB Atlas Speaking of undifferentiated plumbing, MongoDB Atlas is now integrated into AWS Glue’s visual interface. The new integration simplifies data integration between MongoDB Atlas and AWS, making it easy to build efficient ETL (Extract, Transform, Load) pipelines with minimal effort. With its visual interface, AWS Glue allows users to seamlessly transfer, transform, and load data to and from MongoDB Atlas without needing deep technical expertise in Spark or SQL. In this blog post , we look at how AWS Glue and MongoDB Atlas can transform the way you manage data movement. Buy with AWS In the spirit of making things easier for our joint customers, in early 2025 MongoDB will also join the AWS ‘Buy with AWS’ program. Once up and running, Buy With AWS will allow customers to pay for Atlas using their AWS account directly from the Atlas UI, further reducing friction for customers wanting to get started with Atlas on AWS. New Atlas Updates Announced at re:Invent Aside from our joint endeavors with AWS, MongoDB has also been hard at work on improving the core Atlas platform. Here’s an overview of what we announced: Asymmetrical sharding support for Terraform Atlas Provider Customers are constantly seeking ways to optimize costs to ensure they get the best value for their resources. With Asymmetrical Sharding, now available in the Terraform MongoDB Atlas Provider, MongoDB Atlas users can customize the Cluster Tier and IOPS for each shard, encouraging better resource allocation, improved operational efficiency, and cost savings as customer needs evolve. Atlas Flex Tier Our new Atlas Flex tier offers the scaled flexibility of serverless, with the cost-capped assurance of shared tier clusters. With Atlas Flex Tier, developers can build and scale applications cost-effectively without worrying about runaway bills or resource provisioning. New test bench feature in Query Converter At MongoDB, we firmly believe that the document model is the best way for customers to build applications with their data. In our latest update to Relational Migrator , we’ve introduced Generative AI to automatically convert SQL database objects and validate them using the test bench in a fraction of the time, producing deployment-ready code up to 90% faster. This streamlined approach reduces migration risks and manual development effort, enabling fast, efficient, and precise migrations to MongoDB. For more about MongoDB’s work with AWS—including recent announcements and the latest product updates—please visit the MongoDB Atlas on AWS page ! Visit our product page to learn more about MongoDB Atlas .

December 5, 2024

The MongoDB AI Applications Program: Delivering Customer Value

When people ask me about MongoDB, I tell them that they’ve probably interacted with MongoDB without realizing it. In fact, many of the world’s leading companies—including 70% of the Fortune 100—are powered by MongoDB. Everything we do at MongoDB is about serving our customers, but that often happens in the background, where our work is invisible to many users. In my case, that means building an ecosystem of partners who enable customer innovation. A recent example is how MongoDB teamed up with Amazon Web Services (AWS) and Amazon Bedrock to help Base39 —a Brazilian fintech provider—automate loan analysis, decreasing decision time from three days to one hour, and reducing cost per loan analysis by 96%. And there’s the Indian company IndiaDataHub, which joined the MongoDB AI Applications Program (MAAP) to access AI expertise, in-depth support, and a full spectrum of technologies to enhance AI functionality within IndiaDataHub’s analytics platform. This includes connecting relevant data in MongoDB with Meta's AI models to perform sentiment analysis on text datasets. I could go on and on—after all, tens of thousands of MongoDB’s customers have success stories like these. Enabling customer success is precisely why we launched MAAP last summer, and why the program has evolved since. Customers tell us that they want to take advantage of AI, but they’re unsure how to navigate a fast-moving market, how to control costs, and how to unlock business value from their AI investments. So with MAAP, MongoDB offers customers a full AI stack and an integrated set of professional services to help them keep pace with the latest innovations, identify the best AI use cases, and to help them future-proof AI investments. With today’s announcement , Capgemini, Confluent, IBM, QuantumBlack, AI by McKinsey, and Unstructured have joined the 22 companies that now comprise the MAAP partner network. Which means that the MAAP ecosystem (which was founded with Accenture, Anthropic, Anyscale, Arcee AI, AWS, Cohere, Credal, Fireworks AI, Google Cloud, gravity9, LangChain, LlamaIndex, Microsoft Azure, Nomic, PeerIslands, Pureinsights, and Together AI) offers additional cutting-edge AI integration and solutions to customers—and more ways to set them on the path to AI success. CentralReach: Making an impact on autism with AI More than 150 customers have already gotten involved with MAAP, but I’m particularly excited to share the work of CentralReach . CentralReach provides an AI-powered electronic medical record (EMR) platform that is designed to improve outcomes for children and adults diagnosed with autism and related intellectual and developmental disabilities (IDD). Prior to working with MongoDB and MAAP, CentralReach was looking for an experienced partner to further connect and aggregate its more than 4 billion financial and clinical data points across its suite of solutions. CentralReach leveraged MongoDB’s document model to aggregate the company’s diverse forms of information from assessments to clinical data collection, so the company could build rich AI-assisted solutions on top of its database. Meanwhile, MAAP partners helped CentralReach to design and optimize multiple layers of its comprehensive buildout. All of this will enable CentralReach to support initiatives such as value-based outcome measurement, clinical supervision, and care delivery efficacy. With these new data layers in place, providers will be able to make substantial improvements to their clinical delivery to optimize care for all those they serve. “As a mission-driven organization, CentralReach is always looking to innovate on behalf of the clinical professionals—and the more than 350,000 autism and IDD learners—that we serve globally,” said Chris Sullens, CEO of CentralReach. “So being able to lean on MongoDBs database technology and draw on the collective expertise of the MAAP partner network—in addition to MongoDB’s tech expertise and services—to help us improve outcomes for our customers and their clients worldwide has been invaluable.” Working backward from customer needs The addition of Capgemini, Confluent, IBM, QuantumBlack, AI by McKinsey, and Unstructured to the MAAP partner network offers customers additional technology and AI support options. It also builds on MongoDB’s larger partner ecosystem , which is designed to give customers flexibility and choice. By working closely with our partners on product launches, integrations, and real-world challenges, MongoDB has been able to bring a better understanding of the challenges facing customers—and to give them the resources and confidence to move forward with groundbreaking technology like AI . Examples of support MAAP has offered customers include: Guidance on chunking strategies for an AI-native healthcare provider providing patient recommendations based on complex data sources Collaboration on advanced retrieval techniques to improve response accuracies for a large consultancy to automate manual research Evaluation of embedding models for multi-modal data stores for a well-known automaker developing diagnostic applications Guidance on architectures for complex agentic workflows for a mature enterprise technology provider augmenting customer service workflows One way we offer this support is through the MAAP Center of Excellence (CoE). The MAAP CoE comprises AI technical experts from across MongoDB and the MAAP partner ecosystem who collaborate with customers to understand their challenges, technical requirements, and timelines. The MAAP CoE can then recommend custom full-stack architectures and implementation best practices, optimized for the customer’s specific use case and requirements. Indeed, customization is intrinsic to MAAP: MongoDB and our MAAP partners will meet customers wherever they are to help them achieve their goals. For example, if an organization wants to fully own its AI application development, MongoDB and partners can provide guidance and expertise. And in cases where customers want hands-on support, we can help speed projects with professional services. Ultimately, we want MAAP customers—and anyone who works with MongoDB’s partner ecosystem at large—to feel empowered to own their application development, and to transform challenges into opportunities. Let’s build the next big thing together! To learn more about building AI-powered apps with MongoDB, see MongoDB’s AI Resources Hub , the Partner Ecosystem Catalog , or visit the MAAP page . And check out our partner Confluent’s own blog post about MAAP!

December 2, 2024

New Course for Building AI Applications with MongoDB on AWS

Developers everywhere want to expand the limits of what they can build with new generative AI technologies. But the AI market and its offerings have evolved so quickly that for many developers, keeping up can feel overwhelming. As we’ve entered the AI era, MongoDB and Amazon Web Services (AWS) have built upon our eight year partnership to deliver technology integrations—like MongoDB Atlas’s integrations with Amazon Bedrock and Amazon Q Developer (formerly CodeWhisperer)—that simplify the process of building and deploying gen AI applications. By combining MongoDB’s integrated operational and vector database capabilities with AWS’s AI infrastructure solutions, our goal is to make it easier for our developer community to innovate with AI. So, to help developers get started, we’re launching a new, free MongoDB Learning Badge focused on Building AI Applications with MongoDB on AWS . Building AI with MongoDB on AWS This is MongoDB University’s first AWS Learning Badge, and with it, we’ve focused on teaching developers how Amazon Bedrock and Atlas work together—including how to create a knowledge base in Amazon Bedrock, configure a knowledge base to use Atlas, inspect how a query is answered, create an Agent to answer questions based on data in Atlas, and configure guardrails that support responsible agentic behavior. In short, developers will learn how to remove the heavy lifting of infrastructure configuration and integration so they can get up and running with innovative new semantic search and RAG applications faster. Amazon Bedrock is a fully managed service from AWS that offers a choice of high-performing foundation models from leading AI companies via a single API, along with a broad set of capabilities organizations need to build secure, high-performing AI applications. Developers can connect Bedrock to MongoDB Atlas for blazing-fast vector searches and secure vector storage with minimal coding. With the integration, developers’ can use their proprietary data alongside industry-leading foundation models to launch AI applications that deliver hyper-intelligent and hyper-relevant results. Tens of thousands of customers are running MongoDB Atlas on AWS, and many have already embarked successfully on cutting-edge AI journeys. Take Scalestack for example, which used MongoDB Atlas Vector Search to build a RAG-powered AI copilot, named Spotlight, and is now using Bedrock’s customizable models to enhance Spotlight’s relevance and performance. Meanwhile, Base39 —a Brazilian fintech provider—used MongoDB Atlas and Amazon Bedrock to automate loan analysis, decreasing decision time from three days to one hour and reducing cost per loan analysis by 96%. Badge up with MongoDB MongoDB Learning Badges are a powerful way to demonstrate your dedication to continuous learning. These digital credentials not only validate your educational accomplishments but also stand as a testament to your expertise and skill. Whether you're a seasoned developer, an aspiring data scientist, or an enthusiastic student, earning a MongoDB badge can elevate your professional profile and unlock new opportunities in your field. Learn, prepare, and earn Complete the Learning Badge Path and pass a brief assessment to earn your badge. Upon completion, you'll receive an email with your official Credly badge and digital certificate, ready to share on social media, in email signatures, or on your resume. Additionally, you'll gain inclusion in the Credly Talent Directory, where you will be visible to recruiters from top employers. Millions of builders have been trained through MongoDB University courses—join them and get started building your AI future with MongoDB Atlas and AWS. And if you’re attending AWS re:Invent 2024, come find MongoDB at Booth #824. The first 100 people to receive their learning badge will receive a special gift! Start learning today

December 2, 2024

AI-Powered Call Centers: A New Era of Customer Service

Customer satisfaction is critical for insurance companies. Studies have shown that companies with superior customer experiences consistently outperform their peers. In fact, McKinsey found that life and property/casualty insurers with superior customer experiences saw a significant 20% and 65% increase in Total Shareholder Return , respectively, over five years. A satisfied customer is a loyal customer. They are 80% more likely to renew their policies, directly contributing to sustainable growth. However, one major challenge faced by many insurance companies is the inefficiency of their call centers. Agents often struggle to quickly locate and deliver accurate information to customers, leading to frustration and dissatisfaction. This article explores how Dataworkz and MongoDB can transform call center operations. By converting call recordings into searchable vectors (numerical representations of data points in a multi-dimensional space), businesses can quickly access relevant information and improve customer service. We'll dig into how the integration of Amazon Transcribe, Cohere, and MongoDB Atlas Vector Search—as well as Dataworkz's RAG-as-a-service platform— is achieving this transformation. From call recordings to vectors: A data-driven approach Customer service interactions are goldmines of valuable insights. By analyzing call recordings, we can identify successful resolution strategies and uncover frequently asked questions. In turn, by making this information—which is often buried in audio files— accessible to agents, they can give customers faster and more accurate assistance. However, the vast volume and unstructured nature of these audio files make it challenging to extract actionable information efficiently. To address this challenge, we propose a pipeline that leverages AI and analytics to transform raw audio recordings into vectors as shown in Figure 1: Storage of raw audio files: Past call recordings are stored in their original audio format Processing of the audio files with AI and analytics services (such as Amazon Transcribe Call Analytics ): speech-to-text conversion, summarization of content, and vectorization Storage of vectors and metadata: The generated vectors and associated metadata (e.g., call timestamps, agent information) are stored in an operational data store Figure 1: Customer service call insight extraction and vectorization flow Once the data is stored in vector format within the operational data store, it becomes accessible for real-time applications. This data can be consumed directly through vector search or integrated into a retrieval-augmented generation (RAG) architecture, a technique that combines the capabilities of large language models (LLMs) with external knowledge sources to generate more accurate and informative outputs. Introducing Dataworkz: Simplifying RAG implementation Building RAG pipelines can be cumbersome and time-consuming for developers who must learn yet another stack of technologies. Especially in this initial phase, where companies want to experiment and move fast, it is essential to leverage tools that allow us to abstract complexity and don’t require deep knowledge of each component in order to experiment with and realize the benefits of RAG quickly. Dataworkz offers a powerful and composable RAG-as-a-service platform that streamlines the process of building RAG applications for enterprises. To operationalize RAG effectively, organizations need to master five key capabilities: ETL for LLMs: Dataworkz connects with diverse data sources and formats, transforming the data to make it ready for consumption by generative AI applications. Indexing: The platform breaks down data into smaller chunks and creates embeddings that capture semantics, storing them in a vector database. Retrieval: Dataworkz ensures the retrieval of accurate information in response to user queries, a critical part of the RAG process. Synthesis: The retrieved information is then used to build the context for a foundational model, generating responses grounded in reality. Monitoring: With many moving parts in the RAG system, Dataworkz provides robust monitoring capabilities essential for production use cases. Dataworkz's intuitive point-and-click interface (as seen in Video 1) simplifies RAG implementation, allowing enterprises to quickly operationalize AI applications. The platform offers flexibility and choice in data connectors, embedding models, vector stores, and language models. Additionally, tools like A/B testing ensure the quality and reliability of generated responses. This combination of ease of use, optionality, and quality assurance is a key tenet of Dataworkz's "RAG as a Service" offering. Diving deeper: System architecture and functionalities Now that we’ve looked at the components of the pre-processing pipeline, let’s explore the proposed real-time system architecture in detail. It comprises the following modules and functions (see Figure 2): Amazon Transcribe , which receives the audio coming from the customer’s phone and converts it into text. Cohere ’s embedding model, served through Amazon Bedrock , vectorizes the text coming from Transcribe. MongoDB Atlas Vector Search receives the query vector and returns a document that contains the most semantically similar FAQ in the database. Figure 2: System architecture and modules Here are a couple of FAQs we used for the demo: Q: “Can you explain the different types of coverage available for my home insurance?” A: “Home insurance typically includes coverage for the structure of your home, your personal belongings, liability protection, and additional living expenses in case you need to temporarily relocate. I can provide more detailed information on each type if you'd like.” Q: “What is the process for adding a new driver to my auto insurance policy?" A: “To add a new driver to your auto insurance policy, I'll need some details about the driver, such as their name, date of birth, and driver's license number. We can add them to your policy over the phone, or you can do it through our online portal.” Note that the question is reported just for reference, and it’s not used for retrieval. The actual question is provided by the user through the voice interface and then matched in real-time with the answers in the database using Vector Search. This information is finally presented to the customer service operator in text form (see Fig. 3). The proposed architecture is simple but very powerful, easy to implement, and effective. Moreover, it can serve as a foundation for more advanced use cases that require complex interactions, such as agentic workflows , and iterative and multi-step processes that combine LLMs and hybrid search to complete sophisticated tasks. Figure 3: App interface, displaying what has been asked by the customer (left) and how the information is presented to the customer service operator (right) This solution not only impacts human operator workflows but can also underpin chatbots and voicebots, enabling them to provide more relevant and contextual customer responses. Building a better future for customer service By seamlessly integrating analytical and operational data streams, insurance companies can significantly enhance both operational efficiency and customer satisfaction. Our system empowers businesses to optimize staffing, accelerate inquiry resolution, and deliver superior customer service through data-driven, real-time insights. To embark on your own customer service transformation, explore our GitHub repository and take advantage of the Dataworkz free tier .

November 27, 2024

Better Digital Banking Experiences with AI and MongoDB

Interactive banking represents a new era in financial services where customers engage with digital platforms that anticipate, understand, and meet their needs in real-time. This approach encompasses AI-driven technologies such as chatbots, virtual assistants, and predictive analytics that allow banks to enhance digital self-service while delivering personalized, context-aware interactions. According to Accenture’s 2023 consumer banking study , 44% of consumers aged 18-44 reported difficulty accessing human support when needed, underscoring the demand for more responsive digital solutions that help bridge this gap between customers and financial services. Generative AI technologies like chatbots and virtual assistants can fill this need by instantly addressing inquiries, providing tailored financial advice, and anticipating future needs. This shift has tremendous growth potential; the global chatbot market is expected to grow at a CAGR of 23.3% from 2023 to 2030 , with the financial sector experiencing the fastest growth rate of 24.0%. This shift is more than just a convenience; it aims to create a smarter, more engaging, and intuitive banking journey for every user. Simplifying self-service banking with AI Navigating daily banking activities like transfers, payments, and withdrawals can often raise immediate questions for customers: “Can I overdraft my account?” “What will the penalties be?” or “How can I avoid these fees?” While the answers usually lie within the bank’s terms and conditions, these documents are often dense, complex, and overwhelming for the average user. At the same time, customers value their independence and want to handle their banking needs through self-service channels, but wading through extensive fine print isn't what they signed up for. By integrating AI-driven advisors into the digital banking experience, banks can provide a seamless, in-app solution that delivers instant, relevant answers. This removes the need for customers to leave the app to sift through pages of bank documentation in search of answers, or worse, endure the inconvenience of calling customer service. The result is a smoother and user-friendly interaction, where customers feel supported in their self-service journey, free from the frustration of navigating traditional, cumbersome information sources. The entire experience remains within the application, enhancing convenience and efficiency. Solution overview This AI-driven solution enhances the self-service experience in digital banking by applying Retrieval-Augmented Generation (RAG) principles, which combine the power of generative AI with reliable information retrieval, ensuring that the chatbot provides accurate, contextually relevant responses. The approach begins by processing dense, text-heavy documents, like terms and conditions, often the source of customer inquiries. These documents are divided into smaller, manageable chunks vectorized to create searchable data representations. Storing these vectorized chunks in MongoDB Atlas allows for efficient querying using MongoDB Atlas Vector Search , making it possible to instantly retrieve relevant information based on the customer’s question. Figure 1: Detailed solution architecture When a customer inputs a question in the banking app, the system quickly identifies and retrieves the most relevant chunks using semantic search. The AI then uses this information to generate clear, contextually relevant answers within the app, enabling a smooth, frustration-free experience without requiring customers to sift through dense documents or contact support. Figure 2: Leafy Bank mock-up chatbot in action How MongoDB supports AI-driven banking solutions MongoDB offers unique capabilities that empower financial institutions to build and scale AI-driven applications. Unified data model for flexibility: MongoDB’s flexible document model unifies structured and unstructured data, creating a consistent dataset that enhances the AI’s ability to understand and respond to complex queries. This model enables financial institutions to store and manage customer data, transaction history, and document content within a single system, streamlining interactions and making AI responses more contextually relevant. Vector search for enhanced querying: MongoDB Atlas Vector Search makes it easy to perform semantic searches on vectorized document chunks, quickly retrieving the most relevant information to answer user questions. This capability allows the AI to find precise answers within dense documents, enhancing the self-service experience for customers. Scalable integration with AI models: MongoDB is designed to work seamlessly with leading AI frameworks, allowing banks to integrate and scale AI applications quickly and efficiently. By aligning MongoDB Atlas with cloud-based LLM providers, banks can use the best tools available to interpret and respond to customer queries accurately, meeting demand with responsive, real-time answers. High performance and cost efficiency: MongoDB’s multi-cloud, developer-friendly platform allows financial institutions to innovate without costly infrastructure changes. It’s built to scale as data and AI needs to grow, ensuring banks can continually improve the customer experience with minimal disruptions. MongoDB’s built-in scalability allows banks to expand their AI capabilities effortlessly, offering a future-proof foundation for digital banking. Building future-proof applications Implementing generative AI presents several advantages, not only for end-users of the interactive banking applications but also for financial institutions: Enhanced user experience encourages customer satisfaction, ensures retention, boosts reputation, and reduces customer turnover while unlocking new opportunities for cross-selling and up-selling to increase revenue, drive growth and elevate customer value. Moreover, adopting AI-driven initiatives prepares the groundwork for businesses to develop innovative, creative, and future-proof applications to address customer needs and upgrade business applications with features that are shaping the industry and will continue to do so, here are some examples: Summarize and categorize transactional information by powering applications with MongoDB’s Real-Time Analytics . Understand and find trends based on customer behavior that could positively impact and leverage fraud prevention , anti-money laundering (AML) , and credit card application (just to mention a few). Offering investing, budgeting, and loan assessments through AI-powered conversational banking experience. In today’s data-driven world, companies face increasing pressure to stay ahead of rapid technological advancements and ever-evolving customer demands. Now more than ever, businesses must deliver intuitive, robust, and high-performing services through their applications to remain competitive and meet user expectations. Luckily, MongoDB provides businesses with comprehensive reference architectures for building generative AI applications, an end-to-end technology stack that includes integrations with leading technology providers, professional services, and a coordinated support system through the MongoDB AI Applications Program (MAAP) . By building AI-enriched applications with the leading multi-cloud developer data platform, companies can leverage low-cost, efficient solutions through MongoDB’s flexible and scalable document model which empowers businesses to unify real-time, operational, unstructured, and AI-related data, extending and customizing their applications to seize upcoming technological opportunities. Check out these additional resources to get started on your AI journey with MongoDB: How Leading Industries are Transforming with AI and MongoDB Atlas - E-book Our Solutions Library is where you can learn about different use cases for gen AI and other interesting topics that are applied to financial services and many other industries.

November 26, 2024

网易游戏分享游戏场景中MongoDB运行和分析实践

在游戏行业中,数据库的稳定和性能直接影响了游戏质量和用户满意度。在竞争激烈的游戏市场中,一个优秀的数据库产品无疑能为游戏的开发和后期的运营奠定良好的基础。伴随着MongoDB在不同类型游戏场景中的应用越来越广泛,许多知名的游戏公司都在使用MongoDB来处理他们的游戏数据。 网易游戏自成立以来与广大游戏热爱者一同成长。经过20年的快速发展,网易已跻身全球七大游戏公司之一。作为中国领先的游戏开发公司,网易一直处于网络游戏自主研发领域的前端。目前在网易游戏中,包括手游、业务数据中心以及其他内部产品等在内的多项产品都已广泛应用MongoDB。 综合考虑游戏生态、架构、稳定性和成本等因素,网易游戏的数据库架构采用了多类型服务架构,根据游戏在不同的生命周期进行选型适配。MongoDB支持副本集和分片集群模式,可根据需求进行水平扩展,选择合适的分片策略和副本集设置,为网易游戏提供了灵活性和高可用性的帮助。 如何有计划地分析MongoDB数据的过程行为以便产品决策? 随着游戏行业的不断发展,游戏数据的规模和复杂性也在不断增加。在游戏开发过程中还需要进行大量的数据分析,用以了解玩家行为、优化游戏体验、制定营销策略等。 网易互娱数据库业务负责人郑良榉表示:“我们希望有计划地对MongoDB的过程行为数据进行体系化分析,并应用到常态化管理中,以便在与MongoDB数据库交互的过程中,能够对按需触发的业务进行实时响应。” 灵活的文档模型,让数据分析更从容 MongoDB的灵活文档模型在处理游戏中的复杂数据结构时表现尤为出色。游戏存在不同场景下大量的全量更新和增量更新行为,游戏中包括玩家信息、游戏状态、物品信息等在内的数据结构会频繁发生变化,与此同时,优化占比大的字段还可能带来工作负载的更新。MongoDB采用文档存储的方式,使用JSON格式来存储数据。每个文档可以有不同的字段和结构,这使得它非常适合处理游戏中的复杂数据。通过使用MongoDB,游戏开发人员可以随时添加或修改字段而不需要进行复杂的数据库迁移操作。 当运营需要查证玩家反馈时,比如游戏中出现装备丢失、属性不对、需要查证奖励发放情况等问题,网易游戏内部对文档历史分析提出了时效性(周期性)、可查性(可用性)、对比性(变更差异)和及时性(oplog不带索引)的要求。这是一个既要又要的选择。MongoDB使用副本集来实现数据的冗余和高可用性,而操作日志(oplog)是副本集的核心机制之一。网易游戏能够通过副本集中的oplog实现了与原始文档数据联动管理,在主节点发生故障时,可以根据oplog中的信息快速恢复数据,审计并排查问题。 多场景多应用,也能游刃有余 在一些特定的游戏场景中,历史榜单会保存所有场次的数据,周期性活动的副本战斗需要结算,单一玩家的数据玩法会不断叠加,玩家的装备属性也会越来越复杂……诸如此类的大文档数据会持续增加,从而导致批量更新慢、cache使用异常、队列等待长等问题。网易游戏通过MongoDB的定期、自动巡检,帮助产品及时发现大文档潜在问题,例如数据库负载过高、请求超时、异常错误等,从而采取相应措施,规避线上运营风险。 在游戏运营期间,索引对于服务的访问效率至关重要,无索引行为会导致实例负载变高,严重时会导致整个请求阻塞,影响游戏体验和服务稳定性。依托MongoDB,网易游戏对于人工日常的数据查询行为,在访问入口代码层面实现了实时索引检测与报警,用户可以根据提示是否继续查询行为。同时在系统层面,用户访问可按实例级别配置是否允许无索引查询,以进一步减少非预期内的误操作行为。除了使用索引外,MongoDB还可以通过查询优化来加速热点数据的查询,进一步提升QPS(每秒查询推理响应速度)。 游戏运营中会有一些突发状况,比如高峰期单一玩法异常导致请求突增、某个服务卡住导致整个实例请求阻塞等,此时分析库表级别的访问QPS(非mongostat和mongotop结果)对于找到引起问题的根因点至关重要。借助于 MongoDB 的内置功能,网易游戏实现了实时的库表 QPS 和访问延时采集以帮助产品及时发现瓶颈点。此外,客户端行为的跟踪和分析也能帮助业务发现问题来源,借助MongoDB的CurrentOP功能,网易游戏能够统计业务客户端实时会话情况,并针对当前实例节点操作进行行为检索,还能追踪某个连接操作行为。 与此同时,在数据迁移和数据分布异常的情况下,MongoDB也能有效解决阻碍技术负责人执行的最大障碍——大数据量的数据同步工作及落地后的数据管理,从而提升运维效率。 郑良榉表示,MongoDB的接入只是一个开始,如何更好地贴近业务的使用需要长期的多场景实践、探索与总结,这也是当前持续发展的方向。对MongoDB数据行为的分析能够帮助网易游戏在数据决策方面提供有效参考,从而产生更大的收益和价值。 点击注册,免费开始使用 MongoDB Atlas

November 26, 2024

Influencing Product Strategy at MongoDB with Garaudy Etienne

Garaudy Etienne joined MongoDB as a Product Manager in October of 2019. Since then, he’s experienced tremendous growth. Successful deliveries of MongoDB 4.4 features and MongoDB 5.0 sharding features helped fuel Garaudy’s career development, as did his work establishing a long-term sharding vision, mentoring others, and successfully managing interns. Now, as a Director of Product, he’s defining the strategic direction across multiple products and helping grow our product management organization and culture. Read on to learn more about Garaudy’s experience at MongoDB and his expanding team. A team with impact My team focuses on distributed systems within MongoDB's core database functions, also known as the database engine. Our team ensures the database is reliable and scalable for our most demanding customers. We ensure the product consistently performs as promised, especially at scale. MongoDB's dependability drives greater usage, which enhances our revenue and brand perception. The problems my team works on are vast and relatively undefined. These include revamping our Go-To-Market strategy for new and existing features, guiding the engineering team on architectural decisions driven by customer demands, identifying target markets, and assisting customers in challenging situations. MongoDB and AI We’re in the early stages of the AI boom. MongoDB’s document model is particularly well-suited for this era, as it excels in handling unstructured data, which makes up the majority of today’s information. As AI increasingly relies on diverse formats like text, images, and videos, our flexible schema enables efficient storage and retrieval of unstructured data, enabling applications to extract valuable insights. Our vector search capability enables fast, complex data matching and retrieval, making it ideal for AI-powered applications. This synergy between MongoDB’s document model plus Vector Search and the needs of AI-driven applications positions us as a powerful foundation for companies looking to enable AI into their workflows. The beauty of working in the core database is that it has to support every workload, including the new and expanding Vector Search applications. This means we need to ensure the database remains robust and scalable as AI demands evolve. Some examples are helping develop a more scalable architecture for Search or a new networking stack for Search. No matter what new capabilities MongoDB decides to deliver or the new markets we enter, everything must pass through the core database. This also allows you to meet lots of people and understand everything the company is doing instead of working in a silo. A rewarding career in product MongoDB is committed to career development, something I’ve experienced first-hand. The company has provided me with development opportunities through product management-specific training with Reforge, conferences, direct engagement with critical customers, and leadership training. As a product manager, I was offered mentorship and coaching with multiple experienced product leaders who provided guidance and support as I worked toward promotions. The company clearly communicates the expectations and requirements for advancement within the product management organization. Reflecting on my journey at MongoDB, I still remember the first two features I PM’d: Hedged Reads and Mirrored Reads. One of my first major highlights was presenting at the MongoDB 5.0 keynote to showcase resharding. Seeing genuine excitement from customers and internal teams about this new feature was incredibly fulfilling and reinforced its value. While the keynote was a public milestone, another personal highlight came when I finally visited one of my engineering teams in Barcelona after nearly two years of remote collaboration. This in-person time was invaluable and helped us bring the groundbreaking sharding changes for MongoDB 6.0 to the finish line. Most recently, defining the key strategic pillars for MongoDB 8.0 and allowing other product managers to take ownership of key initiatives has been more rewarding than I imagined. MongoDB’s engineering team is extremely talented, and collaborating with them always brings me tremendous joy. The most recent highlight of my career has been building a diverse product team and helping other product managers make a larger impact than they previously envisioned. Why MongoDB What keeps me at MongoDB is the opportunity to tackle significant challenges, make autonomous decisions, own multiple products, and take on greater leadership responsibilities. MongoDB also rewards and recognizes product managers who drive meaningful impact across the organization and its products. If these opportunities excite you, you'll thrive as part of MongoDB’s product management team! For my team, I’m committed to providing the right balance of guidance and autonomy. Your decisions will have a lasting impact at the executive and organizational levels, creating continuous opportunities to excel and deliver meaningful results. Plus, I always try to make the job fun. Head to our careers site to apply for a role on Garaudy’s team and join our talent community to stay in the loop on all things #LifeAtMongoDB!

November 25, 2024

Innovazione e salute: MongoDB a supporto della Piattaforma Nazionale di Telemedicina

La medicina del futuro sarà molto più “tele” e meno fisica? Potremo monitorare la nostra salute senza spostamenti e senza sale d’attesa? Scopriamolo insieme ai protagonisti italiani di questa rivoluzione, costruita su importanti investimenti e tecnologie innovative. PNT Italia è la società di progetto costituita da Engineering e Almaviva, a cui AGENAS (Agenzia Nazionale per i Servizi Sanitari Regionali) ha affidato in concessione la progettazione, realizzazione e gestione della Piattaforma Nazionale di Telemedicina, che integra i servizi regionali per migliorare la qualità e l’accesso alle cure in tutto il Paese, in linea con gli obiettivi indicati dal Piano di Ripresa e Resilienza in ambito Sanità Digitale. Formalmente, PNT Italia è una società costituita dal 60% da Engineering e dal 40% da Almaviva, due tra i più grandi Gruppi tecnologici italiani. Su concessione di AGENAS PNT Italia ha creato la Piattaforma Nazionale di Telemedicina, che gestirà per 10 anni con il compito di assicurare la governance e il monitoraggio centralizzati dei processi di telemedicina attuati a livello regionale, mettendo in comunicazione l’Amministrazione Centrale con le Amministrazioni locali. Per capire meglio il percorso che ci porterà alla telemedicina abbiamo intervistato tre protagonisti di questa iniziativa: Stefano Zema, responsabile sviluppo e delivery PNT del Gruppo Engineering, Daniele Fortuna, PNT Infrastructure Architect di Almaviva e Angelo Immediata, PNT Solution Architect di Engineering. “Il progetto, che ha un respiro decennale, è diviso in tre macro fasi” dice Stefano Zema, responsabile sviluppo e delivery PNT del Gruppo Engineering, “la prima, che è durata da maggio a dicembre 2023, è stata la progettazione e implementazione, conclusa con il collaudo della piattaforma. Ora ci aspettano due anni di avvio e consolidamento (2024-25) in cui la piattaforma dovrà adattarsi e integrare le soluzioni regionali di telemedicina, che sono anch’esse in via di sviluppo. La terza fase comprende la gestione vera e propria della piattaforma”. “Tra i team di Engineering e Almaviva”, prosegue Zema, “si è subito creato un clima coeso, favorito anche dall’affiancamento dei tecnici italiani di MongoDB e dal supporto internazionale. Gli obiettivi del progetto, aiutare la transizione digitale del Paese e consentire ai malati cronici di stare vicino ai propri familiari, erano talmente alti e sfidanti da creare un clima di entusiasmo e voglia di fare bene in tutta l’organizzazione. Le persone, il metodo, la coesione e l’entusiasmo sono stati ingredienti fondamentali per la riuscita di questa prima fase”. “L’obiettivo”, aggiunge Angelo Immediata, “è anche quello di aiutare il Governo a far capire a operatori e pazienti l’importanza della telemedicina. Per raggiungerlo ci sono tante strade. Quelle più efficaci sono la definizione di standard comuni, e l’implementazione di soluzioni di telemedicina efficaci per fare in modo che gli operatori aderiscano in modo naturale e rispettino le indicazioni previste. Per raggiungere questo scopo, PNT impiega tra le 100 e le 150 persone che risiedono in 14 diverse regioni italiane, con Engineering e Almaviva che conferiscono risorse al progetto in modo dinamico”. Un progetto complesso per un obiettivo alto La sfida che il gruppo si trova ad affrontare è articolata: nel breve periodo bisogna creare il nucleo di una piattaforma flessibile, sicura e interoperabile (per mettere in comunicazione i sistemi regionali e quello centrale). Nel medio periodo è necessario garantire stabilità, scalabilità e resilienza. Nel lungo periodo, invece, sarà fondamentale supportare gli operatori a diffondere la cultura di un’assistenza sanitaria centrata sui bisogni degli assistiti e dei loro familiari. “Abbiamo scelto”, spiega Fortuna, “di preferire un’infrastruttura cloud e un’architettura orientata agli eventi, oltre che pensare a una piattaforma aperta, che evitasse i lock-in e che permettesse a PNT di trarre il maggior vantaggio dal cloud pubblico”. “Abbiamo scelto MongoDB Atlas come database per diversi motivi”, dice Angelo Immediata, PNT Solution Architect di Engineering, “primo tra tutti la tipologia di dati da trattare, in standard HL7 FHIR (Fast Healthcare Interoperability Resources), perfettamente gestibili da MongoDB, poi la natura multicloud della tecnologia, la gestione della sicurezza e la scalabilità orizzontale per trattare grandi moli di dati.” I vantaggi di un ambiente aperto e scalabile PNT sceglie di realizzare una piattaforma orientata agli eventi, semplice da espandere e adattare, scalabile e flessibile. “Tutto lo stack tecnologico”, spiega Immediata, “è basato sui framework Java Spring per il back-end e Angular per il front-end. Nei flussi di dati dalle infrastrutture regionali al sistema centrale, le informazioni vengono anonimizzate ma non perdono il loro valore, assicurando così sicurezza e privacy, con l’obiettivo di una vulnerabilità pari a zero”. “La piattaforma è tutta in cloud su infrastrutture AWS”, dice Fortuna, “ma è stata progettata per non avere nessun lock-in. Siamo convintamente cloud native, ma non abbiamo voluto usare servizi nativi del provider; questo è stato uno dei motivi per cui abbiamo scelto MongoDB Atlas, che vive in SaaS e che è per definizione multi-ambiente”. Tra i servizi associati a MongoDB già utilizzati da PNT c’è Online Archive, che permette di sfruttare in modo intelligente gli spazi di archiviazione dei Terabyte di dati provenienti dalle strutture sanitarie, favorendo la sostenibilità economica e ambientale del progetto, ed Encryption at Rest, che proteggendo i dati a livello di DB consente di abbattere la complessità e favorire la compliance. Nel mirino per il prossimo futuro c’è poi la tecnologia Vector Search, associata all’adozione dell’AI Generativa. “Conclusa la prima fase”, dice Fortuna, “ora dedicheremo le nostre energie alla verifica e alla validazione delle piattaforme regionali” e, come rinforza Immediata, “Partiamo dal dato incoraggiante che con MongoDB l’effort per creare le istanze è circa la metà di quello necessario con altre tecnologie e che i tempi si calcolano in ore invece che in giorni.” Il team è pronto alle imminenti sfide, e a sfruttare la flessibilità del database considerando che alla fine del 2025 ci saranno oltre 300mila pazienti assistiti in telemedicina contemporaneamente. In un progetto lungo 10 anni, nessuno può sapere come evolverà la tecnologia; per questo è importante aver scelto un partner che ci permette di scegliere sempre le soluzioni migliori. MongoDB ci fa dormire tranquilli. È una società giovane, dinamica ma nello stesso tempo leader di mercato. Stefano Zema, responsabile sviluppo e delivery PNT, Gruppo Engineering Guarda come gestire in sicurezza i dati sanitari con MongoDB Atlas .

November 25, 2024

Customer Service Expert Wati.io Scales Up on MongoDB

Wati.io is a software-as-a-service (SaaS) platform that empowers businesses to develop conversation-driven strategies to boost growth. Founded by CEO Ken Yeung in 2019, Wati started as a chatbot solution for large enterprises, such as banks and insurance companies. However, over time, Yeung and his team noticed a growing need among small and medium-sized businesses (SMBs) to manage customer conversations more effectively. To address this need, Wati used MongoDB Atlas and built a solution based on the WhatsApp Business API. It enables businesses to manage and personalize conversations with customers, automate responses, improve commerce functions, and enhance customer engagement. Speaking at MongoDB.local Hong Kong in September 2024, Yeung said, “The current solutions on the market today are not good enough. Especially for SMBs [that] don’t have the same level of resources as enterprises to deal with the number of conversations and messages that need to be handled every day.” Supporting scale: From MongoDB Community Edition to MongoDB Atlas “From the beginning, we relied on MongoDB to handle high volumes of messaging data and enable businesses to manage and scale their customer interactions efficiently,” said Yeung. Wati originally used MongoDB Community Edition , as the company saw the benefits of a NoSQL model from the beginning. As the company grew, it realized it needed a scalable infrastructure, so Wati transitioned to MongoDB Atlas. “When we started reaching the 2 billion record threshold, we started having some issues. Our system slowed down, and we were not able to scale it,” said Yeung. Atlas has now become an essential part of Wati’s infrastructure, helping the company store and process millions of messages each month for over 10,000 customers in 165 countries. “Transitioning to a new platform—MongoDB Atlas—seamlessly was critical because our messaging system needs to be on 24/7,” said Yeung. Wati collaborated closely with the MongoDB Professional Services and MongoDB Support teams, and in a few months it was able to rearchitect the deployment and data model for future growth and demand. The work included optimizing Wati’s database by breaking it down into clusters. Wati then focused on extracting connections, such as conversations, and dividing and categorizing data within the clusters—for example, qualifying data as cold or hot based on the read and write frequencies. This architecture underpins the platform’s core features, including automated customer engagement, lead qualification, and sales management. Deepening search capabilities with MongoDB Atlas Search For Wati’s customers, the ability to search through conversation histories and company documents to retrieve valuable information is a key function. This often requires searching through millions of records to rapidly find answers so that they can respond to customers in real-time. By using MongoDB Atlas Search , Wati improved its search capabilities, ultimately helping its business customers perform more advanced analytics and improve their customer service agents’ efficiency and customer reporting. “[MongoDB] Atlas Search is really helpful because we don’t have to do a lot of technical integration, and minimal programming is required,” said Yeung. Looking ahead: Using AI and integrating more channels Wati expects to continue collaborating with MongoDB to add more features to its platform and keep innovating at speed. The company is currently exploring to build more AI capabilities of Wati KnowBot , as well as how it can expand its integration with other conversation platforms and channels such as Instagram and Facebook. To learn more about MongoDB Atlas, visit our product page . To get started with MongoDB Atlas Search, visit the Atlas Search product page .

November 25, 2024

Hanabi Technologies Uses MongoDB to Power AI Assistant, Hana

For all the hype surrounding generative AI, cynics tend to view the few real-world implementations as little more than “fancy chatbots.” But for Abhinav Aggarwal, CEO of Hanabi Technologies , the idea of a generative AI-powered bot that is more than just an assistant was intriguing. “I’d been using ChatGPT since it launched,” said Aggarwal. “That got me thinking: How could we make a chatbot that was like a team member?” And with that concept, Hana was born. The problem with bots “Most generative AI chatbots do not act like people; they wait for a command and give a response,” said Aggarwal. “We wanted to create a human-like chatbot that would proactively help people based on what they wanted—automating reminders, for example, or fetching time zones from your calendar to correctly schedule meetings.” Hanabi’s flagship product, Hana, is an AI assistant designed to enhance team collaboration within Google Chat, working in concert with Google Workspace and its suite of products. “Our target customers are smaller companies of between 10 and 50 people. At this size you’re not going to build your own agent from scratch,” he said. Hana integrates with Google APIs to deliver a human-like assistant that chimes in with helpful interventions, such as automatically setting reminders and making sure meetings are booked in the right time zone for each participant. “Hana is designed to bring AI to smaller companies and help them collaborate in a space where they are already working—Google Workspace,” Aggarwal explained. The MongoDB Atlas solution For Hana to act like a member of the team, Hanabi needed to process massive amounts of data to support advanced features like retrieval-augmented generation (RAG) for better information retrieval across Google Docs and many other sources. And with a rapidly growing user base of over 600 organizations and 17,000+ installs, Hanabi also required a secure, scalable, and high-performing data storage solution. MongoDB Atlas provided a flexible document model, built-in vector database, and scalable cloud-based infrastructure, freeing Hanabi engineers to build new features for Hana rather than focusing on rote tasks like data extract, transform, and load processes or manual scaling and provisioning. Now, MongoDB Atlas handles a variety of responsibilities: Scalability and security: MongoDB Atlas’s auto-scaling and automatic backup features have enabled Hanabi to seamlessly grow its user base without the need for manual database management. RAG: MongoDB Atlas plays a critical role in Hana’s RAG functionality. The platform enables Hanabi to split Google Docs into small sections, create embeddings, and store these sections in Atlas’s vector database. Development Processes: According to Aggarwal, MongoDB’s flexibility in managing changing schemas has been essential to the company’s fast-paced development cycle. Data Visualization: Using MongoDB Atlas Charts has enabled Hanabi to create comprehensive dashboards for real-time data visualization. This has helped the team track usage, set reminders, and optimize performance without needing to build a manual dashboard. Impact and results With MongoDB Atlas, Hanabi can successfully scale Hana to meet the demands of its rapidly expanding user base. The integration is also enabling Hana to offer powerful features like automatic interactions with customers, advanced information retrieval from Google Docs, and manually added memory snippets, making it an essential tool for teams around the world. Next steps Hanabi plans to continue integrating more tools into Hana while expanding its reach to personal Gmail users. The company is also rolling out a new automatic-interaction feature, further enhancing Hana’s ability to proactively assist users without direct commands. MongoDB Atlas remains a key component of Hanabi’s stack, alongside Google Kubernetes Engine, NestJS, and LangChain, enabling Hanabi to focus on innovating to improve the customer experience. Tech Stack MongoDB Atlas Google Kubernetes Engine NestJS LangChain Are you building AI apps? Join the MongoDB AI Innovators Program today! Successful participants gain access to free MongoDB Atlas credits, technical enablement, and invaluable connections within the broader AI ecosystem. If your company is interested in being featured, we’d love to hear from you. Connect with us at ai_adopters@mongodb.com.

November 21, 2024