Applications
Customer stories, use cases, and experiences of MongoDB
Mobile and Edge Solutions with MongoDB and Ditto
September 3, 2024
Applied
Saving Energy, Smarter: MongoDB and Cedalo for Smart Meter Systems
The global energy landscape is undergoing a significant transformation, with energy consumption rising 2.2% in 2023, surpassing the 2010-2019 average of 1.5% per year. This increase is largely due to global developments in BRICS member countries—Brazil, Russia, India, China, and South Africa. As renewable sources like solar power and wind energy become more prevalent (in the EU, renewables accounted for over 50% of the power mix in the first quarter of 2024 ), ensuring a reliable and efficient energy infrastructure is crucial. Smart meters, the cornerstone of intelligent energy networks, play a vital role in this evolution. According to IoT analyst firm Berg Insight, the penetration of smart meters is skyrocketing, with the US and Canada expected to reach nearly 90% adoption by 2027, whereas China is expected to account for as much as 70–80% of smart electricity meter demand across Asia in the next few years. This surge is indicative of a growing trend towards smarter, more sustainable energy solutions. In Central Asian countries, the Asian Development Bank is supporting the fast deployment of smart meters to save energy and improve the financial position of countries' power utilities. This article will delve into the benefits of smart meters, the challenges associated with managing their data, and the innovative solutions offered by MongoDB and Cedalo. The rise of smart meters Smart meters, unlike traditional meters that require manual readings, collect and transmit real-time energy consumption data directly to energy providers. This digital transformation offers numerous benefits, including: Accurate Billing: Smart meters eliminate the need for estimations, ensuring that consumers are billed precisely for the energy they use. Personalized Tariffs: Energy providers can offer tailored tariffs based on individual consumption patterns, allowing consumers to take advantage of off-peak rates, special discounts, and other cost-saving opportunities. Enhanced Grid Management: Smart meter data enables utilities to optimize grid operations, reduce peak demand, and improve overall system efficiency. Energy Efficiency Insights: Consumers can gain valuable insights into their energy usage patterns, identifying areas for improvement and reducing their overall consumption. With the increasing adoption of smart meters worldwide, there is a growing need for effective data management solutions to harness the full potential of this technology. Data challenges in smart meter adoption Despite the numerous benefits, the widespread adoption of smart meters also presents significant data management challenges. To use smart metering, power utility companies need to deploy a core smart metering ecosystem that includes the smart meters themselves, the meter data collection network, the head-end system (HES), and the meter data management system (MDMS). Smart meters collect data from end consumers and transmit it to the data aggregator via the Local Area Network (LAN). The transmission frequency can be adjusted to 15 minutes, 30 minutes, or hourly, depending on data demand requirements. The aggregator retrieves the data and then transmits it to the head-end system. The head-end system analyzes the data and sends it to the MDMS. The initial communications path is two-way, signals or commands can be sent directly to the meters, customer premise, or distribution device. Figure 1: End-to-end data flow for a smart meter management system / advanced metering infrastructure (AMI 2.0) When setting up smart meter infrastructure, power, and utility companies face several significant data-related challenges: Data interoperability: The integration and interoperability of diverse data systems pose a substantial challenge. Smart meters must be seamlessly integrated with existing utility systems and other smart grid components often requiring extensive upgrades and standardization efforts. Data management: The large volume of data generated by smart meters requires advanced data management and analytics capabilities. Utilities must implement robust data storage, processing, and analysis solutions to handle real-time time series data streams storage, analysis for anomaly detection, and trigger decision-making processes. Data privacy: Smart meters collect vast amounts of sensitive information about consumer energy usage patterns, which must be protected against breaches and unauthorized access. Addressing these challenges is crucial for the successful deployment and operation of smart meter infrastructure. MQTT: A cornerstone of smart meter communication MQTT , a lightweight publish-subscribe protocol, shines in smart meter communication beyond the initial connection. It's ideal for resource-constrained devices on low-bandwidth networks, making it perfect for smart meters. While LoRaWAN or PLC handle meter-to-collector links, MQTT bridges Head-End Systems (HES) and Meter Data Management Systems (MDMS). Its efficiency, reliable delivery, and security make it well-suited for large-scale smart meter deployments. Cedalo MQTT platform and MongoDB: A powerful combination Cedalo , established in 2017, is a leading German software provider specializing in MQTT solutions. Their flagship product, the Cedalo MQTT Platform, offers a comprehensive suite of features, including the Pro Mosquitto MQTT broker and Management Center . Designed to meet the demands of large enterprises, the platform delivers high availability, audit trail logging, persistent queueing, role-based access control, SSO integration, advanced security, and enhanced monitoring. To complement the platform's capabilities, MongoDB's Time Series collections provide a robust and optimized solution for storing and analyzing smart meter data. These collections leverage a columnar storage format and compound secondary indexes to ensure efficient data ingestion, reduced disk usage, and rapid query processing. Additionally, window functions enable flexible time-based analysis, making them ideal for IoT and analytical applications. Figure 2: MongoDB as the main database for the meter data management system where it receives meter data via Pro Mosquitto MQTT broker. Let us revisit Figure 1 and leverage both the Cedalo MQTT Platform and MongoDB in our design. In Figure 2, the Head-end System (HES) can use MQTT to filter, aggregate, and convert data before storing it in MongoDB. This data flow can be established using the MongoDB Bridge plugin provided by Cedalo. Since the MQTT payload is JSON, it is ideal to store it in MongoDB as the database stores data in BSON (Binary JSON). The MongoDB Bridge plugin offers advanced features such as flexible data import settings (specifying target databases and collections, choosing authentication methods, and selecting specific topics and message fields to import) and advanced collection mapping (mapping multiple MQTT topics to one or more collections with the ability to choose specific fields for insertion). MongoDB's schema flexibility is crucial for adapting to the ever-changing structures of MQTT payloads. Unlike traditional databases, MongoDB accommodates shifts in data format seamlessly, eliminating the constraints of rigid schema requirements. This helps with interoperability challenges faced by utility companies. Once the data is stored in MongoDB, it can be analyzed for anomalies. Anomalies in smart meter data can be identified based on various criteria, including sudden spikes or drops in voltage, current, power, or other metrics that deviate significantly from normal patterns. Here are some common types of anomalies that we might look for in smart meter data: Sudden spikes or drops: These include voltage, current, or power spikes or drops. A sudden increase or decrease in voltage beyond expected limits. Outliers: Data points that are significantly different from the majority of the data. Unusual patterns: Unusually high or low energy consumption compared to historical data or inconsistent power factor readings. Frequency anomalies: Frequency readings that deviate from the normal range. MongoDB's robust aggregation framework can aid in anomaly detection. Both anomaly data and raw data can be stored in time series collections, which offer reduced storage footprint and improved query performance due to an automatically created clustered index on timestamp and _id. The high compression offered addresses the challenge of data management at scale. Additionally, data tiering capabilities like Atlas Online Archive can be leveraged to push cold data into cost-effective storage. MongoDB also provides built-in security controls for all your data, whether managed in a customer environment or MongoDB Atlas, a fully managed cloud service. These security features include authentication, authorization, auditing, data encryption (including Queryable Encryption ), and the ability to access your data security with dedicated clusters deployed in a unique Virtual Private Cloud (VPC). End-to-end solution Figure 3: End-to-end data flow Interested readers can clone this repository and set up their own MongoDB-based smart meter data collection and anomaly detection solution. The solution follows the pattern illustrated in Figure 3, where a smart meter simulator generates raw data and transmits it via an MQTT topic. A Mosquitto broker receives these messages and then stores them in a MongoDB collection using the MongoDB Bridge. By leveraging MongoDB change streams , an algorithm can retrieve these messages, transform them according to MDMS requirements, and perform anomaly detection. The results are stored in a time series collection using a highly compressed format. The Cedalo MQTT Platform with MongoDB offers all the essential components for a flexible and scalable smart meter data management system, enabling a wide range of applications such as anomaly detection, outage management, and billing services. This solution empowers power distribution companies to analyze trends, implement real-time monitoring, and make informed decisions regarding their smart meter infrastructure. We are actively working with our clients to solve IoT challenges. Take a look at our Manufacturing and Industrial IoT page for more stories.
The Dual Journey: Healthcare Interoperability and Modernization
Interoperability in healthcare isn’t just a buzzword; it’s a fundamental necessity. It refers to the ability of IT systems to enable the timely and secure access, integration, and use of electronic health data. However, integrating data across different applications is a pressing challenge, with 48% of US hospitals reporting a one-sided sharing relationship in which they share patient data with other providers who do not, in turn, share patient data with the hospital. The ability to share electronic health data seamlessly across various healthcare systems can revolutionize patient care, enhance operational efficiency, and drive innovation. In this post, we’ll explore the challenges of healthcare data sharing, the role of interoperability , and how MongoDB can be a game-changer in this landscape. The challenge of data sharing in healthcare Today's consumers have high expectations for accessing information, and many now anticipate quick and continuous access to their health and care records. One of the biggest IT challenges faced by healthcare organizations is sharing data effectively and creating seamless data integrations to build patient-centric healthcare solutions. Healthcare data has to be shared in multiple ways: Between internal applications to ensure seamless data flow across various internal systems. Between primary and secondary care , to coordinate care across healthcare providers. To patient portals and telemedicine to enhance patient engagement and remote care. To payers, institutions, and patients themselves , to streamline interactions with insurance companies and regulatory bodies. To R&D units to accelerate medical research and pharmaceutical developments. The complexity of healthcare data is staggering, and hospitals regularly need to integrate dozens of different applications—all of which means that there are significant barriers to healthcare data sharing and integration. A vision for patient-centric healthcare Imagine a world where patient data is shared in real-time with all relevant parties—doctors, hospitals, labs, pharmacies, and insurance companies. This level of interoperability would streamline the flow of information, reduce errors, and improve patient outcomes. Achieving this, however, is no easy feat as healthcare data is immensely complex, involving various types of data such as unstructured clinical notes, lab tests, medical images, medical devices, and even genomic data. Furthermore, types of data mean different things depending on where and where it was collected. Achieving seamless data sharing also involves overcoming barriers to data sharing between different healthcare providers and systems, all while adapting to evolving regulations and standards. Watch the " MongoDB and FHIR: Navigating Healthcare Data " session from MongoDB.local NYC on YouTube. The intersection of modernization and interoperability Modernization of healthcare IT systems and achieving interoperability are two sides of the same coin. Both require significant investments and a focus on transitioning from application-driven to data-driven architecture. By focusing first on data and then connecting applications with a developer data platform like MongoDB Atlas , healthcare organizations can avoid data silos and achieve vendor-neutral data ownership. As healthcare interoperability standards define a common language, organizations might question whether, instead of reinventing the wheel with their own data domains, they can use the interoperability journey (and its high investments) to modernize their applications. MongoDB’s document data model supports the JSON format, just like FHIR (Fast Healthcare Interoperability Resources) and other interoperability standards, making it a more efficient and flexible data platform for developing healthcare applications beyond the limitations of external APIs. FHIR for storing healthcare data? The most implemented standard worldwide, HL7 FHIR , treats each piece of data as a self-contained resource with external links, similar to web pages. HL7 adopted a pragmatic approach: there was no need to define a complete set of resources for all the clinical data, but they wanted to get the 80% that most electronic health records (EHR) share. For the 20% of non-standardized data, they created FHIR Extensions to extend every resource to specific needs. However, FHIR is not yet fully developed, with only 15 of the 158 resources it defines having reached the highest level of maturity. The constant changes can be as simple as a name change or can be so complex that data has to be rearranged. FHIR is designed for the exchange of data but can also be used for persistence. Figure 1: Using FHIR for persistence depending on the complexity of the use case For specific applications with no complex data, such as integrating data from wearables, you can leverage FHIR. However, building a primary operational repository for broader applications like a patient summary or even a healthcare information system presents a significant challenge. This is because the data model required goes beyond the capabilities of FHIR, and solving that through FHIR extensions is an inefficient approach. OpenEHR as an alternative approach In Catalonia, Spain—for about 8 million people and roughly 60 public hospitals—there are 29 different hospital EHR systems. Each hospital maintains a team of developers exclusively focused on building interoperability interfaces. Due to an increased demand for data sharing, the cost of maintaining this data will only grow. Rather than implementing interoperability interfaces, why not create new applications that are implicitly interoperable? This is what openEHR proposes, defining the clinical information model from the maximal perspective and developing applications that consume a subset of the clinical information system using an open architecture. However, while FHIR is very versatile—offering data models for administrative and operational data efficiently—openEHR focuses exclusively on clinical data. So while a combination of FHIR and openEHR can solve part of the problem, future healthcare applications need to integrate a wide variety of data, including medical images, genomics, proteomics, and data from complex medical devices—which could be complicated by the lack of a single standard. Overcoming this challenge with the document model and MongoDB Now, let’s discover the power of the document model to advance interoperability while modernizing systems. MongoDB Atlas features a flexible document data model, which provides a flexible way of storing and organizing healthcare data using JSON-like documents. With a flexible data schema, healthcare organizations can accommodate any data structure, format, or source into one platform, providing seamless third-party integration capabilities necessary for interoperability. While different use cases will have different solutions, the flexibility of the document model means MongoDB is able to adapt to changes. Figure 2 below shows a database modeled in MongoDB, where each collection stores a FHIR resource type (e.g., patients, encounters, conditions). These documents mirror the FHIR objects, conserving its complex hierarchy. Let's imagine our application requires specific fields not supported by FHIR, and there is no need for an FHIR extension because it won’t be shared externally. We can add a metadata field containing all this information that is as flexible as needed. It can be used to track the standard evolution of the resource, the version of the document itself, tenant ID for multi-tenant applications, and more. Figure 2: Data modeled in MongoDB Another possibility is to add the searchable fields of the resource as key-value pairs so that you can retrieve data with a single index. We can maintain the indexes by automating this with the FHIR search parameters. In a single repository, we can combine FHIR data with custom application data. Additionally, data in other protocols can be integrated, providing unparalleled flexibility in accessing your data. This setup permits access through different endpoints within one repository. Using MQL (the MongoDB Query Language), you can build FHIR APIs or use the MongoDB SQL interface to provide SQL querying capabilities to connect your preferred business intelligence tools. Figure 3: Unified and flexible data store and flexible data retrieval MongoDB’s developer data platform At the center of MongoDB’s developer data platform is MongoDB Atlas, the most advanced cloud database service on the market. It provides integrated full-text search capabilities, allowing applications to perform when making complex queries without the need to maintain a separate system. With generative AI, multidimensional vectors that represent data are becoming a necessity. MongoDB Atlas stores vectors along the operational data and provides vector search, which enables fast data retrieval. Therefore, you can store metadata in your vector embeddings, as shown in Figure 4. Figure 4: Vector embeddings These capabilities transform a single database into a unique, powerful, and easy-to-use interface capable of handling diverse use cases without the need for any single-purpose databases. Solving the dual challenge with MongoDB Achieving interoperability and modernization in healthcare IT are challenging but essential. MongoDB provides a powerful platform that meets organizations’ modern data management needs. By embracing MongoDB, healthcare organizations can unlock the full potential of their data, leading to improved patient outcomes and operational efficiency. Figure 5: Closing the gap between interoperability and modernization journey with MongoDB Refactoring applications to incorporate interoperability resources as part of documents—and extending them with all the requirements for your modern needs—will ensure organizations’ data layers remain robust and adaptable. By doing so, organizations can create a flexible architecture that can seamlessly integrate diverse data types and accommodate future advancements. This approach not only enhances data accessibility and simplifies data management but also supports compliance with evolving standards and regulations. Furthermore, it enables real-time data analytics and insights, fostering innovation and driving better decision-making. Ultimately, this strategy positions healthcare organizations to effectively manage and leverage their data, leading to improved patient outcomes and operational efficiencies. For more detailed information and resources on how MongoDB can transform organizations’ healthcare IT systems, we encourage you to apply for an exclusive innovation workshop with MongoDB's industry experts to explore bespoke modern app development and tailored solutions for your organization. Additionally, check out these resources: MongoDB and FHIR: Navigating Healthcare Data with MongoDB How Leading Industries are Transforming with AI and MongoDB Atlas The MongoDB Solutions Library is curated with tailored solutions to help developers kick-start their projects For developers: From FHIR Synthea data to MongoDB
CTF Life Leverages MongoDB Atlas to Deliver Customer-Centric Service
Hong Kong-based Chow Tai Fook Life Insurance Company Limited (CTF Life) is proud of its rich, nearly 40-year history of providing a wide range of insurance and financial planning services. The company provides life, health, accident, savings, and investment insurance to its customers, helping them and their loved ones navigate life’s journey with personalized planning solutions, lifelong protection, and diverse lifestyle experiences. A wholly-owned subsidiary of NWS Holdings Limited and a member of Chow Tai Fook Group, CTF Life consistently strengthens its collaboration with the diverse conglomerate of the Cheng family (Chow Tai Fook Group) and draws on the Group’s robust financial strength, strategic investments across the globe, and advanced customer-focused digital technology with the aspiration of becoming a leading insurance company in the Greater Bay Area. To achieve this goal, CTF Life modernized its on-premises infrastructure to provide the speed and flexibility required to offer customers personalized experiences. To turn their vision into reality, CTF Life decided to adopt MongoDB Atlas . By modernizing their systems and processes with the world’s most versatile developer data platform, CTF Life knew they’d be able to meet customer expectations, offering improved customer service, faster response times, and more convenient access to their products and services. Data-driven customer service The insurance industry is undergoing a significant shift, from traditional data management to near-real-time data-driven insights, driven by strong consumer demand and the urgent need for companies to process large amounts of data efficiently. As insurance companies strive to provide personalized and real-time products, the move towards sophisticated and real-time data-driven customer service is inevitable. CTF Life is on its digital transformation journey to modernize its relational database management system (RDBMS) infrastructure to empower its agents, known as Life Planners, to provide enhanced customer experiences. The company faced obstacles to legacy systems and siloed data. Life Planners were spending a lot of time looking up customer information from various systems and organizing this into useful customer insights. Not having a holistic view of customer data also made it challenging to recommend personalised products and services within CTF Life, the Group, and beyond. Reliance on legacy RDBMS systems presented a major challenge in CTF Life’s pursuit of leveraging real-time customer information to enhance customer experiences and operational efficiency. For their modernization efforts, CTF Life was looking for the following required capabilities: A modernized application with agile development No downtime for changing schema, new modules, or feature updates A single way of centralizing and organizing data from a number of sources (backend, CRM, etc.) into a standardized format ready for a front-end mobile application A future-proof data platform with extensible capability for analytics across CTF Life, their diverse conglomerate collaboration, and their strategic partners to support the company’s digital solutions Embracing the operational data layer for enhanced experiences CTF Life knew they had to build a solution for the Life Planners to harness the wealth of useful information available to them, making it easier to engage and connect with customers. The first project identified was their clienteling system, which is designed to establish long-term relationships with customers based on data about their preferences, behaviors, and needs. To overcome their legacy systems and siloed data, CTF Life built their clienteling system on MongoDB Atlas . Atlas serves as the digital data store for Life Planners, creating a single view of the customer (SVOC) with a flexible document model that enables CTF Life to handle large volumes of customer data in real-time efficiently. By integrating their operational data into one platform with MongoDB Atlas on Microsoft Azure, CTF Life’s revamped clienteling system provides their Life Planners with a comprehensive view of customer profiles, which allows them to share targeted content with customers. Additionally, CTF Life is using Atlas Search to build relevance-based search capabilities directly into the application, making it faster and easier to search for customer data across the company’s system landscape. These benefits helped improve customer service with faster access to data with an SVOC so Life Planners can provide more accurate and timely information to their customers. Atlas Search is now the foundation of the clienteling system, which powers data analytics and machine learning capabilities to support various use cases. For example, the clienteling app's smart reminder feature recognizes key moments in a customer's life, like the impending arrival of a newborn child. Based on these types of insights, the app can help Life Planners make personalized recommendations to the customer about relevant services and products that may be of interest to them as new parents. Because of its work with MongoDB, CTF Life can now analyze customer profiles and use smart reminders to engage customers at the right time in the right context. This has made following up with customers and leads faster and easier. And, contacting prospects, scheduling appointments, setting reminders, sharing relevant content, running campaigns and promotions, recommending products and services, and tracking lead progress can all be performed in one system. Moreover, access to real-time data enables Life Planners to streamline their work and reduce manual processes. And data-driven insights empower Life Planners to make informed decisions quickly. They can analyze customer information, identify trends, and tailor their recommendations to meet individual needs more effectively. With MongoDB Atlas Search, Life Planners can use advanced search capabilities to identify opportunities to serve customers better. Continuing to create value beyond insurance CTF Life strives to provide its customers with value beyond insurance. Through a range of collaborations with Chow Tai Fook Group, and strategic partnerships with technology partners like MongoDB, CTF Life has created a customer-centric approach and continues to advance its digital transformation strategy to enhance a well-rounded experience for customers that goes beyond insurance with a sincere and deep understanding of their diverse needs in every chapter of their life journey. In the future, CTF Life will continue to build upon its strategic partnership with MongoDB and expand the use of its digital data store on MongoDB Atlas by creating new client servicing modules on the mobile app their Life Planners use. CTF Life will also be expanding its search capabilities with Atlas Vector Search to accelerate their journey to building advanced search and generative AI applications for more automated servicing. Partnering with MongoDB helped us prioritize technology that accelerates our digital transformation. The integration between generative AI and MongoDB as a medium for information search can be leveraged to further support front-line Life Planners as well as mid/back-office operations. Derek Ip, Chief Digital and Technology Officer of CTF Life Learn how to tap into real-time data with MongoDB Atlas .
Built With MongoDB: Atlas Helps Team-GPT Launch in Two Weeks
Team-GPT enables teams large and small to collaborate on AI projects. When OpenAI released GPT-4, it turned out to be a game-changer for the startup. Founded in 2023, the company has been helping people train machine learning (ML) models, in particular natural language processing (NLP) models. But when OpenAI launched GPT-4 in March 2023, the team was blown away by how much progress had been made on large language models (LLMs). So Team-GPT dropped everything they were doing and started experimenting with it. Many of those early ideas are still memorialized on a whiteboard in one of the office's meeting rooms: The birth of an idea. Like many startups, Team-GPT began with a brainstorm on a whiteboard. Evolving the application Of all the ideas they batted around, there was one issue in particular the team wanted to solve—the need for a shared workspace where they could experiment with LLMs together. What they found was that having to work with LLMs in the terminal was a major point of friction. Plus, there weren't any sharing abilities. So they set out to create a UI consisting of chat sharing, in-chat team collaboration, folders and subfolders, and a prompt library. The whole thing came together in an incredibly short period of time. This was due, in large part, to their initial choice of MongoDB Atlas, which allowed them to build with speed and scalability. "MongoDB made it possible for us to launch in just two weeks," said Team-GPT Founder and CTO, Ilko Kacharov. "With the MongoDB Atlas cloud platform, we were able to move rapidly, focusing our efforts on developing innovative product features rather than dealing with the complexities of infrastructure management." Before long, the team realized there was a lot more that could be built around LLMs than simply chat, and set out to add more advanced capabilities. Today, users can integrate any LLM of their choice and add custom instructions. The platform also supports multimodality like ChatGPT Vision and DALL-E. Users use any GPT model to turn chat responses into a standalone document that can then be edited. All these improvements are meant to unify teams' AI workflows in a single, AI-powered tool. A platform built for developers Diving deeper into more technical aspects of the solution, Team-GPT CEO Iliya Valchanov acknowledges the virtues of the document data model, which underpins the Atlas developer data platform. "We wanted the ability to quickly update and create new collections, add more data, and expand the existing database setup without major hurdles or time consumption," he said. "That's something that relational databases often struggle with." A developer data platform consists of integrated data infrastructure components and services for quick deployment. With transactional, analytical, search, and stream processing capabilities, it supports various use cases, reduces complexity, and accelerates development. Valchanov's team leverages a few key elements of the platform to address a range of application needs. "We benefited from Atlas Triggers , which allow automatic execution of specified database operations," he said. "This greatly simplified many of our routine tasks." It's not easy to build truly differentiated applications without a friction-free developer experience. Valchanov cites Atlas' user-friendly UI as a key advantage for a startup where time is of the essence. And he said that Atlas Charts has been instrumental for the team, who use it every day, even their less technical people. Of course one of the biggest reasons why developers and tech leaders choose MongoDB, and why so many are moving away from relational databases, is its ability to scale—which Valchanov said is one of the most critical requirements for supporting the company's growth. "With MongoDB handling the scaling aspect, we were able to focus our attention entirely on building the best possible features for our customers." Team-GPT deployment options Accelerating AI transformation Team-GPT is a collaborative platform that allows teams of up to 20,000 people to use AI in their work. It's designed to help teams learn, collaborate, and master AI in a shared workspace. The platform is used by over 2,000 high-performing businesses worldwide, including EY, Charles Schwab, Johns Hopkins University, Yale University, and Columbia University, all of which are also MongoDB customers. The company's goal is to empower every person who works on a computer to use AI in a productive and safe manner. Valchanov fully appreciates the rapid change that accompanies a product's explosive growth. "We never imagined that we would eventually grow to provide our service to over 40,000 users," he said. "As a startup, our primary focus when selecting a data platform was flexibility and the speed of iteration. As we transitioned from a small-scale tool to a product used by tens of thousands, MongoDB's attributes like flexibility, agility, and scalability became necessary for us." Another key enabler of Team-GPT's explosive growth has been the MongoDB for Startups program . It offers valuable resources such as free Atlas credits, technical guidance, co-marketing opportunities, and access to a network of partners. Valchanov makes no secret of how instrumental the program has been for his company's success. "The startup program made it free! It offered us enough credits to build out the MVP and cater to all our needs," he said. "Beyond financial aid, the program opened doors for us to learn and network. For instance, my co-founder, Yavor Belakov, and I participated in a MongoDB hackathon in MongoDB's office in San Francisco." Team-GPT co-founders Yavor Belakov (l) and Iliya Valchanov (r) participated in a MongoDB hackathon at the San Francisco office Professional services engagements are an essential part of the program, especially for early-stage startups. "The program offered technical sessions and consultations with MongoDB staff, which enriched our knowledge and understanding, especially for Atlas Vector Search , aiding our growth as a startup," said Valchanov. The roadmap ahead for the company includes the release of Team-GPT 2.0, which will introduce a brand-new user interface and new, robust functionalities. The company encourages anyone looking to learn more or join their efforts to ease adoption of AI innovations to reach out on LinkedIn . Are you part of a startup and interested in joining the MongoDB for Startups program? Apply to the program now . For more startup content, check out our Built With MongoDB blog collection.
Agnostiq & MongoDB: High-Performance Computing for All
Material scientists, computational biologists, and AI researchers all have at least one thing in common; the need for huge amounts of processing power, or ‘compute’, to turn raw data into results. But here’s the problem. Many researchers lack the skills needed to build the workflows that move data through huge networks of distributed servers, CPUs, and GPUs that actually do the number crunching. And that’s where Agnostiq comes in. Since the company’s inception in 2018, Agnostiq has put the power of high-performance computing (HPC) in the hands of researchers, bypassing the need for development expertise to build these essential data and compute pipelines. Power to the people “We started research on high-performance computing needs in fields like finance and chemistry, and through the process of onboarding, researchers quickly realized how hard it was for [researchers] to access and scale up on the cloud, or tap into HPC and GPU resources,'' said Santosh Kumar Radha, Agnostiq’s Head of Product. “If you wanted to scale up, there were not many tools available in the major cloud providers to do this.” To address this bottleneck, the team at Agnostiq built Covalent, a Python-based framework that allows researchers to easily design and run massive compute jobs on cloud platforms, on-prem clusters, and HPC services. With Covalent, startups and enterprises can build any AI or HPC application in a simple, scalable, and cost-effective way using a Python notebook, negating the need to interact with underlying infrastructure. One of the hardest challenges the Covalent team faced was combining traditional HPC with modern cloud technology. Because traditional HPC infrastructure was never designed to run in the cloud, the team spent considerable resources marrying techniques like GPU and CPU parallelization, task parallelization, and graph optimization with distributed cloud computing environments. As a result, researchers can use Covalent to quickly create a workflow that combines the convenience of cloud computing with specialized GPU providers and other HPC services. Everything, everywhere, all at once As the name suggests, Agnostiq has always focused on making their platform as open and resource neutral as possible. MongoDB Atlas , with its native multi-cloud capability, was a perfect complement. “At Agnostiq, everything we build has to be technology and vendor neutral. Interoperability is key for us,” said Radha. “We do all the mapping for our customers, so our platform has to perform a seamless transition from cloud to cloud.” The ability to move data between clouds became even more critical following the release of ChatGPT. With an explosion in generative AI research and development, the availability of GPU resources plummeted. “Resource scarcity in the ‘GPT era’ means you couldn’t get access to GPUs anywhere,” Radha added. “If you didn’t have a default cloud posture, you were nowhere, which is why we doubled down on multi-cloud and MongoDB Atlas to give our clients that optionality.” Open source opening doors Since the beginning, the team at Agnostiq has chosen MongoDB as their default NoSQL database. At first, the team adopted MongoDB’s free, open source product. “We didn’t have any DBAs as a small agile team. MongoDB gave us the freedom to build and manage our data workflows without the need for a specialist,” said William Cunningham, Head of HPC at Agnostiq. As their customer base grew along with the demand for cloud computing access, Agnostiq moved to MongoDB Atlas, gaining the freedom to move data seamlessly between AWS, Google Cloud, and Microsoft Azure. This gave Covalent the flexibility to reach multi-cloud compatibility at a faster rate than with standard tooling. Covalent provides a workflow management service by registering jobs, dispatching IDs, and collecting other metadata that allows fellow researchers and developers to reproduce the original work. MongoDB is used in the front-end, allowing a high volume of metadata and other assets to be published and cached in accordance with an event-driven architecture. This near real-time experience is key to a product aimed at delivering a unified view over distributed resources. MongoDB Atlas further provided the autoscaling required to grow with the user base and the number of workloads while keeping costs in check. “MongoDB Atlas helps us provide an ideal foundation for modern HPC and AI applications which require serverless compute, autoscaling resources, distributed workloads, and rapidly reconfigurable infrastructure,” added Radha. The future Looking to the future, Agnostiq is focused on servicing the huge demand for gen AI modeling and workflow building. To that end, the company released its own inference service called Function Serve within Covalent. Function Serve offers customers a complete, enterprise-grade solution for AI development and deployment, supporting serverless AI model training and fine-tuning. With Function Serve, customers can fine-tune, host, and serve any open-source or proprietary model with full infrastructure abstraction, all with only a few additional lines of code. MongoDB Atlas was used to rapidly develop a minimal service catalog while remaining cloud-agnostic. Looking ahead, the team plans to leverage MongoDB Atlas for enterprise and hybrid-cloud deployments in order to quickly meet customers in their existing cloud platforms. Agnostiq is a member of the MongoDB AI Innovators program , providing their team with access to Atlas credits and technical best practices. You can get started with your AI-powered apps by registering for MongoDB Atlas and exploring the tutorials available in our AI resources center . Additionally, if your company is interested in being featured, we'd love to hear from you. Reach out to us at ai_adopters@mongodb.com .
Meeting the UK’s Telecommunications Security Act with MongoDB
Emerging technologies like AI, IoT, and 5G have transformed the value that telecommunications companies provide the world. However, these new technologies also present new security challenges. As telcos continue to amass large amounts of sensitive data, they become an increasingly attractive target for cybercriminals — making both companies and countries vulnerable to cyberattacks. Fortunately, developers can protect user data which comes with strong security requirements on a developer data platform. By offering features to meet stringent requirements with robust operational and security controls, telcos can protect their customers’ private information. The UK Telecommunications Security Act Amid growing concerns about the vulnerability of telecom infrastructure, and its increasing digital dependency, the UK Telecommunications (Security) Act (TSA) was enacted on November 17, 2021. It was designed to bolster the security and resilience of the UK’s telecommunications networks. The TSA mandates that telecom operators implement rigorous security measures such as end-to-end encryption as well as identity and access management to protect their networks from a broad spectrum of threats, ensuring the integrity and continuity of critical communication services. The act allows the government to compel telecom providers to meet specific security directives. The United Kingdom’s Office of Communications (Ofcom) is a regulatory body responsible for overseeing compliance, conducting inspections, and enforcing penalties on operators that fail to meet the standards. The comprehensive code of practice included in the act offers detailed guidance on the security measures that should be implemented, covering risk management, network architecture, incident response, and supply chain security. The TSA tiering system The TSA establishes a framework for ensuring the security of public electronic communications networks and services. It categorizes telecoms providers into different tiers, with specific security obligations for each tier. The Act outlines three main tiers: Tier 1: These are the largest and most critical providers. They have the most extensive obligations due to their significant role in the UK's telecoms infrastructure. Tier 1 providers must comply with the full set of security measures outlined in the Act. Tier 2: These providers have a considerable role in the telecoms network but are not as critical as Tier 1 providers. They have a reduced set of obligations compared to Tier 1 but still need to meet substantial security requirements. Tier 3: These are smaller providers with a limited impact on the overall telecoms infrastructure. Their obligations are lighter compared to Tiers 1 and 2, reflecting their smaller size and impact. The specific obligations for each tier include measures related to network security, incident reporting, and supply chain security. The aim is to ensure a proportional approach to securing the telecoms infrastructure, with the highest standards applied to the most critical providers. Non-compliance may result in fines Under the TSA, non-compliance with security obligations can result in substantial fines. The fines are designed to be significant enough to ensure compliance and deter breaches. The significance of the fines imposed under the TSA underscores the importance the UK government places on telecom security and the serious consequences of failing to meet the established standards. How MongoDB can help MongoDB offers built-in security controls for all your data—whether your databases are managed on-premises with MongoDB Enterprise Advanced or with MongoDB Atlas , our fully managed cloud service. MongoDB enables enterprise-grade security features and simplifies deploying and managing your databases. Encrypting sensitive data The TSA emphasizes securing telecom networks against cyber threats. While specific encryption requirements are not detailed, the focus is on robust security practices, including encryption to protect data integrity and confidentiality. Operators must implement measures that prevent unauthorized access and ensure data security throughout transmission and storage. Compliance may involve regular risk assessments and adopting state-of-the-art technologies to safeguard the network infrastructure. MongoDB data encryption offers robust features to protect your data while it’s in the network, being stored, in memory, in transit (network), at rest (storage), and in use (memory, logs). Customers can use automatic encryption of key data fields like personally identifiable information (PII) or any data deemed sensitive—ensuring data is encrypted through its use. Additionally, with our industry-first Queryable Encryption , MongoDB offers a fast, searchable encryption scheme that supports equality searches, with additional query types such as range, prefix, suffix, and substring planned for future releases. Authentication and Authorization The TSA contemplates stringent identity and access management requirements to enhance network security. Regular audits and reviews of access permissions should be designed to prevent unauthorized access and to quickly identify and respond to potential security breaches. These measures aim to protect the integrity and confidentiality of telecommunications infrastructure. MongoDB enables users to authenticate to their Atlas UI with their Atlas credentials or via single sign-on with their GitHub or Google accounts. Atlas also supports MFA with various options, including OTP authenticators, push notifications, FIDO2 (hardware security keys or biometrics), SMS, and e-mail. MongoDB Enterprise Advanced users can authenticate to the MongoDB database using mechanisms including SCRAM, x.509 certificates, LDAP, OIDC, and passwordless authentication with AWS-IAM. Auditing Under the TSA, providers must implement logging mechanisms to detect and respond to security incidents effectively. Logs should cover access to sensitive systems and data, including unsuccessful access attempts, and must be comprehensive, capturing sufficient detail to facilitate forensic investigations. Additionally, logs should be kept for a specified minimum period and to be protected against unauthorized access, tampering, and loss. MongoDB offers granular auditing that monitors actions in your MongoDB environment and is designed to prevent and detect any unauthorized access to data, including CRUD operations, encryption key management, authentication, role-based access controls, replication, and sharding cluster operations. Additionally, MongoDB’s Atlas Organization Activity Feed displays select events that occurred for a given Atlas organization, such as billing or access events. Likewise, the Atlas Project Activity Feed displays select events that occurred for a given Atlas project. Network security The TSA outlines several network security requirements to ensure the protection and resilience of telecommunications networks. These requirements encompass various aspects of network security, including risk management, protection measures, incident response, and compliance with standards and best practices. Atlas offers many options to securely access your data with dedicated clusters deployed in a unique virtual private cloud (VPC) to isolate your data and prevent inbound network access from the internet. You can also allow a one-way connection from your AWS, Azure, or Google Cloud VPC/VNet to Atlas Clusters via Private Endpoints . Additionally, you can enable peering between your MongoDB Atlas VPC or VNet to your own dedicated application tier VPN with the cloud provider of your choice or enable only specific network segments to connect to your Atlas clusters via the IP Access list . In summary, the UK TSA is a critical regulatory framework aimed at protecting the nation’s telecommunications infrastructure from cyber threats. For telecom companies, compliance isn’t just a legal obligation but a business imperative. Failure to comply can mean significant financial penalties, reputational harm, and long-term operational challenges, underscoring the importance of adopting robust security measures and maintaining continuous adherence to the Act’s requirements. Visit MongoDB’s Strong Security Defaults page for more information on protecting your data with strong security defaults on the MongoDB developer data platform, as well as how to meet stringent requirements with robust operational and security controls.
Leveraging Database Observability at MongoDB: Real-Life Use Case
This post is the second in our three-part series, Leveraging Database Observability at MongoDB. Welcome back to the Leveraging Database Observability at MongoDB series. In our last discussion, we explored MongoDB's unique observability strategy using out-of-the-box tools designed to automatically monitor and optimize customer databases. These tools provide continuous feedback to answer critical questions such as what is happening, where is the issue, why is it occurring, and how do I fix it? This ensures enhanced performance, increased productivity, and minimized downtime. So let’s dive into a real-life use case, illustrating how different tools in MongoDB Atlas come together to address database performance issues. Whether you're a DBA, developer, or just a MongoDB enthusiast, our goal is to empower you to harness the full potential of your data using the MongoDB observability suite. Why is it essential to diagnose a performance issue? Identifying database bottlenecks and pinpointing the exact problem can be daunting and time-consuming for developers and DBAs. When your application is slow, several questions may arise: Have I hit my bandwidth limit? Is my cluster under-provisioned and resource-constrained? Does my data model need to be optimized, or cause inefficient data access? Do my queries need to be more efficient, or are they missing necessary indexes? MongoDB Atlas provides tools to zoom in, uncover insights, and detect anomalies that might otherwise go unnoticed in vast data expanses. Let’s put it into practice Let's consider a hypothetical scenario to illustrate how to track down and address a performance bottleneck. Setting the context Imagine you run an online e-commerce store selling a popular item. On average, you sell about 500 units monthly. Your application comprises several services, including user management, product search, inventory management, shopping cart, order management, and payment processing. Recently, your store went viral online, driving significant traffic to your platform. This surge increased request latencies, and customers began reporting slow website performance. Identifying the bottleneck With multiple microservices, finding the service responsible for increased latencies can be challenging. Initial checks might show that inventory loads quickly, search results are prompt, and shopping cart updates are instantaneous. However, the issue might be more nuanced and time-sensitive, potentially leading to a full outage if left unaddressed. The five-step diagnostic process To resolve the issue, we’ll use a five-step diagnostic process: Gather data and insights by collecting relevant metrics and data. Generate hypotheses to formulate possible explanations for the problem. Prioritize hypotheses to use data to identify the most likely cause. Validate hypotheses by confirming or disproving the top hypothesis. Implement and observe to make changes and observe the results. Applying the five-step diagnostic process for resolution Let’s see how this diagnostic process unfolds: Step 1: Gather Data and Insights Customers report that the website is slow, so we start by checking for possible culprits. Inefficient queries, resource constraints, or network issues are the primary suspects. Step 2: Generate Hypotheses Given the context, the application could be making inefficient queries, the database could be resource-constrained, or network congestion could be causing delays. Step 3: Prioritize Hypotheses We begin by examining the Metric Charts in Atlas. Since our initial check revealed no obvious issues, we will investigate further. Step 4: Validate Hypotheses Using Atlas' Namespace Insights , we break down the host-level measurements to get collection-level data. We notice that the transactions.transactions collection has much higher latency than others. By increasing our lookback period to a week, the latency increased just over 24 hours ago when customers began reporting slow performance. Since this collection stores details about transactions, we use the Atlas Query Profiler to find that the queries are inefficient because they’re scanning through the whole transaction documents. This validates our hypothesis that application slowness was due to query inefficiency. Figure 1: New Query Insights Tab Step 5: Implement and Observe We need to create an index to resolve the collection scan issue. The Atlas Performance Advisor suggests an index on the customerID field. Adding this index enables the database to locate and retrieve transaction records for the specified customer more efficiently, reducing execution time. After creating the index, we return to our Namespace Insights page to observe the effect. We see that the latency on our transactions collection has decreased and stabilized. We can now follow up with our customers to update them on our fix and assure them that the problem has been resolved. Conclusion By gathering the correct data, working iteratively, and using the MongoDB observability suite , you can quickly resolve database bottlenecks and restore your application's performance. In our next post in the "Leveraging Database Observability at MongoDB" series, we’ll show how to integrate MongoDB metrics seamlessly into central observability stacks and workflows. This 'plug-and-play' experience aligns with popular monitoring systems like Datadog, New Relic, and Prometheus, offering a unified view of application performance and deep database insights in a comprehensive dashboard. Sign up for MongoDB Atlas , our cloud database service, to see database observability in action. For more information, see our Monitor Your Database Deployment docs page.
Inside MongoDB’s Early Access Programs
In tech, staying ahead of the curve is essential. Tech companies must continually innovate and release new developments to meet the ever-evolving needs of users and global market demands. One successful strategy is the use of Early Access Programs (EAPs) and Preview Features. These initiatives offer unique opportunities for both companies and users, fostering mutually beneficial relationships that drive product excellence. You may have heard them referred to as "Beta Programs," "Pilot Programs," or "Feature Previews," but they all fall under the same category of early user engagement aimed at refining products before general release. So what are Early Access Programs and Preview Features? EAPs and Preview Features allow a select group of users to test and provide feedback on new features before general release. These programs are aimed at loyal customers, power users, or those who’ve shown interest in the company's products. EAPs grant users early access to upcoming features or products still in development. Participants engage closely with the product team, providing valuable insights and feedback that shape the final product. Preview Features are specific features in a product made available for users to try before the official release. Unlike beta testing, which may involve a full product, preview features are often isolated components of a larger system. What are the Benefits of Early Access Programs? They offer numerous advantages for both companies and users. Direct feedback from real users helps identify bugs, usability issues, and areas for improvement that may not be evident during internal testing. This actionable feedback is crucial in enhancing product quality, allowing companies to release more polished and reliable features. This means higher user satisfaction and fewer post-release problems. EAPs also increase customer engagement and appreciation among participants by offering a hands-on exclusive experience to them. This can foster loyalty and strengthen the relationship between the company and its customers. These programs also provide market validation, offering an opportunity to gauge market interest and demand for new features. This enables companies to make data-driven decisions about which features to prioritize and further invest in. By identifying and resolving potential issues early in the development process, EAPs reduce the risk of major failures post-launch and allow for better resource allocation and planning. How does MongoDB approach Early Access Programs? At MongoDB, EAPs are classified into two categories: private preview and public preview. Private preview entails an invite-only white-glove experience for a select few participants who closely test and provide feedback. Public preview implies the feature is available to the public to try, either as downloadable tools or features in Atlas. Key Elements Selective Participation: MongoDB’s EAPs are typically invitation-only, targeting power users, those who’ve shown significant interest in new features, or are part of our MongoDB Champions community. This selective approach ensures that feedback comes from experienced users who can provide valuable insights. Direct Collaboration: Participants engage directly with MongoDB’s product and engineering teams. This direct line of communication allows for real-time feedback, in-depth discussions, and a collaborative approach to feature development. Structured Feedback Collection: We use a variety of methods to collect feedback, including surveys, structured interviews, and feedback forms. This structured approach ensures that feedback is actionable and can be effectively integrated into the development process. Iterative Development: Feedback is used to make iterative improvements to the features being tested. This agile approach allows us to quickly address issues, refine functionalities, and enhance the overall user experience. Transparency and Updates: We maintain open and transparent communication with our EAP participants, providing regular updates on the status of the features, changes made based on their feedback, and future development plans. This transparency fosters trust and keeps participants engaged throughout the program. Rewards and Recognition: Participants can share their stories on our social channels, be part of global events, and win MongoDB swag! Be Part of MongoDB’s Innovation MongoDB’s Early Access Programs offer participants the chance to gain early access to innovative features, influence product development, and join an exclusive, engaged community. As we continue to innovate and expand our product offerings, our Early Access Programs will help us deliver high-quality, user-centric solutions that empower our customers to achieve their strategic objectives. Join us in shaping the future of MongoDB by enrolling in our early access programs today!
magicpin Builds India's Largest Hyperlocal Retail Platform on MongoDB
Despite its trillion-dollar economy, 90% of retail consumption in India still takes place offline . While online retail in India has grown in recent years, much of it still consists of dark stores (a retail outlet or distribution center that exists exclusively for online shopping) and warehouses, the majority of retail establishments—fashion, food, dining, nightlife, and groceries—thrive as physical stores. What’s more, businesses looking to transition to online models are hindered by major platforms that focus primarily on clicks rather than encouraging transactions. This opportunity was the inspiration for the founders of magicpin , India’s largest hyperlocal retail platform. magicpin has revolutionized the conventional pay-per-click model, where businesses bid on keywords or phrases related to their products or services and then pay a fee each time someone clicks on an ad, with a new pay-per-conversion strategy. In a pay-per-conversion model, businesses only pay when they make an actual sale of a product or item. magicpin does not rely on dark stores, warehouses, or deep discounting; instead, it collaborates with local retailers, augmenting foot traffic and preserving the essence of local economies. This unique model ensures that consumers not only enjoy existing in-store benefits, but also receive additional perks when opting to transact through magicpin. “We enable the discovery of those merchants,” says Kunal Gupta, senior vice president at magicpin. “Which merchants in your local neighborhood are selling interesting stuff? What’s their inventory? What savings can we offer to buyers? We have data for everything.” Effectively three SaaS platforms in one, magicpin is a seller app, a buyer app, and a developing logistics app on the Open Network for Digital Commerce ( ONDC ), which is backed by the Indian government. With over 10 million users on its platform (covering the majority of Indian cities and over 100 localities), magicpin has established itself as a leading offline retail discovery and savings app. magicpin currently has 250,000 merchants in categories ranging from food to fashion to pharmacy. The power behind magicpin has always been MongoDB's flexibility and scalability. And from the company’s start in 2015, it became clear that magicpin was on to something special. “In the first week of March 2023 when we onboarded ONDC, we hit almost 10,000 transactions a day. In October last year, we peaked at 50,000 orders in a single day, which is a huge milestone,” says Kunal. “When an ONDC order is placed, it flows through us. We manage the entire process—from sending the order to the merchant, assigning logistics personnel for pickup and delivery, to handling any customer support tickets that may arise. It's the seamless integration of these elements that defines our contribution to the intricate framework of ONDC." Having launched using the community version of MongoDB , Kunal realized that magicpin needed to make better use of its relatively lean tech team and allow them to focus more on building the business. He also saw that a managed service would be a more effective way of handling maintenance and related tasks. “We realized there had to be a better solution. We can’t afford to have all the database expertise tied up with a team that’s focusing on creating businesses and building applications,” said Kunal. “That’s when we started to use MongoDB Atlas." magicpin uses a multitude of technologies, to store over 600 million SKUs, and handle its SaaS platform, session cache, card, and order management, and MongoDB Atlas sits at the heart of the business. “For our operational and scaling needs, it’s seamless,” Kunal concludes. “Availability is high, and monitoring and routing are super-good. Our lives have become much easier.” Watch the full presentation on YouTube to learn more.
Teach & Learn with MongoDB: Professor Abdussalam Alawini, University of Illinois at Urbana-Champaign
In this series of interviews, we talk to students and educators around the world who are using MongoDB to make their classes more engaging and relevant. By exploring their stories, we uncover how MongoDB’s innovative platform and resources are transforming educational landscapes and empowering the next generation of tech-savvy professionals. From creative teaching approaches to advanced classroom solutions, the MongoDB for Educators program can help you transform your classroom with cutting-edge technology and free resources. It can help you provide students with an interactive and dynamic learning environment that bridges the gap between theoretical knowledge and practical application. The program includes a variety of free resources for educators crafted by MongoDB experts to prepare learners with in-demand database skills and knowledge. Program participants have access to MongoDB Atlas credits, curriculum materials, certifications, and membership in a global community of educators from over 700 universities. From theory to practice: Hands-on MongoDB Teaching Professor Abdussalam Alawini is known for his creative use of MongoDB in his courses. He heavily uses MongoDB's free cluster to demonstrate MongoDB concepts during classes, and his students also use the free cluster for their projects, giving them hands-on experience with real-world applications. Currently, a Teaching Associate Professor at the University of Illinois Urbana-Champaign, Professor Alawini’s research interests span databases, applied machine learning, and education. He is particularly focused on applying machine learning methods to enhance classroom experiences and education. His work also includes developing next-generation data management systems, such as data provenance, citation, and scientific management systems. He recently received the U of I’s 2024 Campus Excellence in Undergraduate Education award, which highlights his commitment to teaching and the impact he’s had on his students. Professor Alawini is currently collaborating with colleagues on research to map how databases, data systems, data management, and related courses are taught in introductory computer science undergraduate courses worldwide. Professor Alawini’s story offers valuable insights for educators eager to enhance their teaching and prepare students for a tech-driven future. Check out how MongoDB Atlas has revolutionized his teaching by simplifying database deployment, management, and scaling, allowing students to focus more on learning MongoDB concepts. Tell us about your educational journey and what sparked your interest in databases. My educational journey began with a bachelor's degree in Computer Science from the University of Tripoli in 2002. I then spent over six years in the industry as a database administrator, lead software developer, and IT Manager. In 2011, I returned to academia and earned two master's degrees in Computer Science and Engineering and Technology Management from Portland State University, followed by a Ph.D. in Computer Science in 2016. Subsequently, I joined the University of Pennsylvania for a two-year postdoctoral training. My interest in databases was sparked during my time as a database administrator at PepsiCo, where I enjoyed maintaining the company's databases and building specialized reports to improve business operations. I was particularly fascinated by database systems’ ability to optimize queries and handle millions of concurrent user requests seamlessly. This experience led me to focus my doctoral studies on building data management systems for scientific applications. What courses are you currently teaching at the University of Illinois Urbana-Champaign? Currently, I teach Database Systems and Data Management in the Cloud courses at the University of Illinois Urbana-Champaign. In addition, I also teach a course to University High School students to introduce them to data management and database basics. My intention with teaching databases to high schoolers is to use data management as a gateway to lower entry barriers into computing fields for non-computer science students and to recruit underrepresented minorities to computing. What inspired you to start teaching MongoDB? I was inspired to start teaching MongoDB after seeing several surveys indicating that it is the most used database in web development and one of the leading document-oriented databases. MongoDB offers several unique features that set it apart from other databases, including the aggregation pipeline, which simplifies data processing and transformation. Additionally, MongoDB's flexible schema design allows for easier handling of unstructured data, and its horizontal scalability ensures robust performance as data volumes grow. These features make MongoDB an essential tool for modern web development, and I wanted to equip my students with the skills to leverage this powerful technology. How do you design your course content to effectively integrate MongoDB and engage students in practical learning? In all my data management courses, I focus on teaching students the concept of data models, including relational, document, key-value, and graph. In my Database Systems course, I teach MongoDB alongside SQL and Neo4J to highlight the unique features and capabilities of each data model. This comparative approach helps students appreciate the importance and applications of different databases, ultimately making them better data engineers. In my Data Management in the Cloud course, I emphasize the system's side of MongoDB, particularly its scalability. Understanding how MongoDB is built to handle large volumes of data efficiently provides students with practical insights into managing data in a cloud environment. To effectively integrate MongoDB and engage students in practical learning, I use a hybrid flipped-classroom approach. Students watch recorded lectures before class, allowing us to dedicate class time to working through examples together. Additionally, students form teams to work on various data management scenarios using a collaborative online assessment tool called PrairieLearn. This model fosters peer learning and collaboration, enhancing the overall educational experience. How has MongoDB supported you in enhancing your teaching methods and upskilling your students? I would like to sincerely thank MongoDB for Academia for the amazing support and material they provided to enhance my course design. The free courses offered at MongoDB University have significantly improved my course delivery, allowing me to provide more in-depth and practical knowledge to my students. I heavily use MongoDB's free cluster to demonstrate MongoDB concepts during classes, and my students also use the free cluster for their projects, which gives them hands-on experience with real-world applications. MongoDB Atlas has been a game-changer in my teaching methods. As a fully managed cloud database, it simplifies the process of deploying, managing, and scaling databases, allowing students to focus on learning and applying MongoDB concepts without getting bogged down by administrative tasks. The flexibility and reliability of MongoDB Atlas make it an invaluable tool for both educators and students in the field of data management. Could you elaborate on the key findings from your ITiCSE paper on students' experiences with MongoDB and how these insights can help other educators? In my ITiCSE paper, we conducted an in-depth analysis of students' submissions to MongoDB homework assignments to understand their learning experiences and challenges. The study revealed that as students use more advanced MongoDB operators, they tend to make more reference errors, indicating a need for a better conceptual understanding of these operators. Additionally, when students encounter new functionalities, such as the $group operator, they initially struggle but generally do not repeat the same mistakes in subsequent problems. These insights suggest that educators should allocate more time and effort to teaching advanced MongoDB concepts and provide additional support during the initial learning phases. By understanding these common difficulties, instructors can better tailor their teaching strategies to improve student outcomes and enhance their learning experience. What advice would you give to fellow educators who are considering implementing MongoDB in their own courses to ensure a successful and impactful experience for their students? Implementing MongoDB in your courses can be highly rewarding. Here’s some advice to ensure success: Foundation in Data Models: Teach MongoDB alongside other database types to highlight unique features and applications, making students better data engineers. Utilize MongoDB Resources: Leverage support from MongoDB for Academia, free courses from MongoDB University, and free clusters for hands-on projects. Practical Learning: Use MongoDB Atlas to simplify database management and focus on practical applications. Focus on Challenges: Allocate more time for advanced MongoDB concepts. Address common errors and use tools like PrairieLearn that capture students' interactions and learning progress to identify learning patterns and adjust instruction. Encourage Real-World Projects: Incorporate practical projects to enhance skills and relevance. Continuous Improvement: Gather feedback to iteratively improve course content and share successful strategies with peers. MongoDB is always evolving so make sure to stay tuned with their updates and new features. These steps will help create an engaging learning environment, preparing students for real-world data management. Apply to MongoDB for Educators program and explore free resources for educators crafted by MongoDB experts to prepare learners with in-demand database skills and knowledge.
Nokia Corteca Scales Wi-Fi Connectivity to Millions of Devices With MongoDB Atlas
Nokia’s home Wi-Fi connectivity cloud platform was launched in 2019 as the Nokia WiFi Cloud Controller (NWCC). In 2023, it was renamed and relaunched as the Corteca Home Controller, becoming part of the Corteca software suite that delivers smarter broadband for a better experience. The Corteca Home Controller can be hosted on Amazon Web Services, Google Cloud, or Microsoft Azure, and is the industry’s first platform to support three management services—device management, Wi-Fi management, and application management. Supporting TR-369 (a standardized remote device management protocol) also allows the Home Controller to work in a multi-vendor environment, managing both Nokia broadband devices and third-party broadband devices. By solving connectivity issues before the end-user detects them, and by automatically optimizing Wi-Fi performance, the Home Controller helps deliver excellent customer experiences to millions of users, 24/7. During the five years that Nokia Corteca has been a MongoDB Atlas customer, the Home Controller has successfully scaled from 500,000 devices to over 4.5 million. There are now 75 telecommunications customers of Home Controller spread across all regions of the globe. Having the stability, efficiency, and performance to scale Nokia Corteca's solution is end-to-end, from applications embedded in the device, through the home, and into the cloud. Algorithms assess data extracted from home networks, based on which performance parameters automatically adjust as needed—changing Wi-Fi channels to avoid network interference, for example—thereby ensuring zero downtime. The Home Controller processes real-time data sent from millions of devices, generating massive volumes of data. With a cloud optimization team tasked with deploying the solution across the globe to ever more customers, the Home Controller needed to store and manage its vast dataset and to onboard new telecommunication organizations more easily without incurring any downtime. Prior to Nokia Corteca moving to MongoDB Atlas, its legacy relational database lacked stability and required both admin and application teams to manage operations. A flexible model with time series capabilities That's where MongoDB Atlas came in. Nokia was familiar with the MongoDB Atlas database platform, having already worked with it as part of a previous company acquisition and solution integration. As Nokia's development team had direct experience with the scalability, manageability, and ease of use offered by MongoDB Atlas, they knew it had the potential to address the Home Controller’s technical and business requirements. There was another key element: Nokia wanted to store time-series data—a sequence of data points in which insights are gained by analyzing changes over time. MongoDB Atlas has the unique ability to store operational and time series data in parallel and provides robust querying capabilities on that data. Other advantages include MongoDB's flexible schema, which helps developers store data to match the application's needs and adapt as data changes over time. MongoDB Atlas also provides features such as Performance Advisor that monitors the performance of the database and makes intelligent recommendations to optimize and improve the performance and resource consumption Fast real time data browsing and scalability made easy Previously, scaling the database had been time-consuming and manual. With MongoDB Atlas, the team can easily scale up as demand increases with very little effort and no downtime. This also means it is much more straightforward to add new clients, such as large telecommunications companies. Having started with 100GB of data, the team now has more than 1.3 terabytes, and can increase the disc space in a fraction of a second, positioning the team to be able to scale with the business. As the Home Controller grows and onboards more telcos, the team anticipates a strengthening relationship with MongoDB. “We have a very good relationship with the MongoDB team,” said Jaisankar Gunasekaran, Head of Cloud Hosting and Operations at Nokia. “One of the main advantages is their local presence—they’re accessible, they’re friendly, and they’re experts. It makes our lives easier and lets us concentrate on our products and solutions.” To learn more about how MongoDB can help drive innovation and capture customer imaginations, check out our MongoDB for Telecommunications page.
Unlock PDF Search in Insurance with MongoDB & SuperDuperDB
As industries go, the insurance industry is particularly document-driven. Insurance professionals, including claim adjusters and underwriters, spend considerable time handling documentation with a significant portion of their workday consumed by paperwork and administrative tasks. This makes solutions that speed up the process of reviewing documents all the more important. Retrieval-augmented generation (RAG) applications are a game-changer for insurance companies, enabling them to harness the power of unstructured data while promoting accessibility and flexibility. This is especially true for PDFs, which despite their prevalence are difficult to search, leading claim adjusters and underwriters to spend hours reviewing contracts, claims, and guidelines in this common format. By combining MongoDB and SuperDuperDB you can build a RAG-powered system for PDF search, thus bringing efficiency and accuracy to this cumbersome task. With a PDF search application, users can simply type a question in natural language and the app will sift through company data, provide an answer, summarize the content of the documents, and indicate the source of the information, including the page and paragraph where it was found. In this blog, we will dive into the architecture of how this PDF search application can be created and what it looks like in practice. Why should insurance companies care about PDF Search? Insurance firms rely heavily on data processing. To make investment decisions or handle claims, they leverage vast amounts of data, mostly unstructured. As previously mentioned, underwriters and claim adjusters need to comb through numerous pages of guidelines, contracts, and reports, typically in PDF format. Manually finding and reviewing every piece of information is time-consuming and can easily lead to expensive mistakes, such as incorrect risk estimations. Quickly finding and accessing relevant content is key. Combining Atlas Vector Search and LLMs to build RAG apps can directly impact the bottom line of an insurance company. Behind the scenes: System architecture and flow As mentioned, MongoDB and SuperDuperDB underpin our information retrieval system. Let’s break down the process of building it: The user adds the PDFs that need to be searched. A script scans them, creates the chunks, and vectorizes them (see Figure 1). The chunking step is carried out using a sliding window methodology, which ensures that potentially important transitional data between chunks is not lost, helping to preserve continuity of context. Vectors and chunk metadata are stored in MongoDB, and an Atlas Vector Search index is created (see Figure 3). The PDFs are now ready to be queried. The user selects a customer, asks a question, and the system returns an answer, where it was found and highlights the section with a red frame (see Figure 3). Figure 1: PDF chunking, embedding creation, and storage orchestrated with SuperDuperDB Each customer has a guidelines PDF associated with their account based on their residency. When the user selects a customer and asks a question, the system runs a Vector Search query on that particular document, seamlessly filtering out the non-relevant ones. This is made possible by the pre-filtering field included in the search query. Atlas Vector Search also takes advantage of MongoDB’s new Search Nodes dedicated architecture, enabling better optimization for the right level of resourcing for specific workload needs. Search Nodes provide dedicated infrastructure for Atlas Search and Vector Search workloads, allowing you to optimize your compute resources and fully scale your search needs independent of the database. Search Nodes provide better performance at scale, delivering workload isolation, higher availability, and the ability to optimize resource usage. Figure 2: PDF querying flow, orchestrated with SuperDuperDB SuperDuperDB SuperDuperDB is an open-source Python framework for integrating AI models and workflows directly with and across major databases for more flexible and scalable custom enterprise AI solutions. It enables developers to build, deploy, and manage AI on their existing data infrastructure and data, while using their preferred tools, eliminating data migration and duplication. With SuperDuperDB, developers can: Bring AI to their databases, eliminate data pipelines and moving data, and minimize engineering efforts, time to production, and computation resources. Implement AI workflows with any open and closed source AI models and APIs, on any type of data, with any AI and Python framework, package, class or function. Safeguard their data by switching from APIs to hosting and fine-tuning your own models, on your own existing infrastructure, whether on-premises or in the cloud. Easily switch between embedding models and LLMs, to other API providers as well as hosting your own models, on HuggingFace, or elsewhere just by changing a small configuration. Build next-generation AI apps on your existing database SuperDuperDB provides an array of sample use cases and notebooks that developers can use to get started, including vector search with MongoDB, embedding generation, multimodal search, retrieval-augmented generation (RAG), transfer learning, and many more. The demo showcased in this post is adapted from an app previously developed by SuperDuperDB. Let's put it into practice To show you how this could work in practice, let’s look at, an underwriter handling a specific case. The underwriter is seeking to identify the risk control measures as shown in Figure 3 below but needs to look through documentation. Analyzing the guidelines PDF associated with a specific customer helps determine the loss in the event of an accident or the new premium in the case of a policy renewal. The app assists by answering questions and displaying relevant sections of the document. Figure 3: Screenshot of the UI of the application, showing the question asked, the LLM’s answer, and the reference document where the information is found By integrating MongoDB and SuperDuperDB, you can create a RAG-powered system for efficient and accurate PDF search. This application allows users to type questions in natural language, enabling the app to search through company data, provide answers, summarize document content, and pinpoint the exact source of the information, including the specific page and paragraph. If you would like to learn more about Vector Search powered apps and SuperDuperDB, visit the following resources: PDF Search in Insurance Github repository Search PDFs at Scale with MongoDB and Nomic SuperDuperDB Github, includes notebooks and examples