The MongoDB Web Shell

MongoDB
August 14, 2013
#Releases

About

The MongoDB Web Shell is a web application designed to emulate some of the features of the mongo terminal shell. This project has three main uses: try.mongodb.org, 10gen Education online classes, and the MongoDB API documentation.

In these three different contexts, users will be able to familiarize themselves with the MongoDB interface and basic commands available both independently and as part of a 10gen education homework assignment in the education program.

See a screenshot of the state of the browser shell prior to this summer below:

Architecture

The user interfaces with the new MongoDB Web Shell through any web browser. A majority of the codebase for the project is written in Javascript executed in the browser. It provides a responsive command line interface similar to the MongoDB desktop shell. Because the MongoDB console is also implemented in Javascript, it is convenient to evaluate user-written code locally.

The browser-based shell interacts with a backing mongod instance through a RESTful interface implemented on top of Flask, a Python web application microframework. The backend provides

1) sandboxing capability to manage sessions and access to resources on the backing database 2) a framework for preloading data into a given resource 3) ability to verify data state for use in the online classes.

One interesting problem we ran into is allowing the blocking paradigm of code used by the mongo shells without actually blocking in the web browser. Browser tabs run on a single thread, so if Javascript code blocks on that thread, you can’t scroll or select things or interact with the tab at all. However, running code such as length = db.foo.aggregate(pipeline).results.length traditionally requires that the aggregate call is a blocking request so that it can fetch the remote data and return it in a single function call. Normally, we should just rewrite out function calls to take callbacks, but since this is a web shell, we don’t have control over the code being written. Our solution was to write a Javascript evaluator that could pause execution at any time, similar to coroutines. This solution manifested itself in a library called suspend.js evaluates Javascript program strings and can pause execution, perform some arbitrary, possibly asynchronous actions, then resume execution with the result of those actions. Thus users can still write code in the blocking paradigm that the mongo shell uses without locking up the browser.

1) try.mongodb.org

One of the primary goals of the project is to replace the existing browser shell on try.mongodb.org. The newer version provides better support for javascript syntax and mongoDB features in a cleaner UI. Among the features that will be present in the shell are syntax highlighting (replacing the green on black styling), fully-featured Javascript, and a new tutorial system that integrates beyond the shell window in the containing page.

A screenshot of the new shell:

One of the challenges in developing a fully featured shell was the ability to execute user-written code in a sandboxed environment within the browser. Originally, the shell relied on parsing the code into an abstract syntax tree (AST) and mutating the source to namespace variables in a manner where var x would become mongo.shells[index].vars['x']. We quickly realized that this implementation would be difficult to maintain given potential feature growth and the number of edge case scenarios. Instead, the latest version of the shell delegates the scoping of variables to the browser by executing in an iframe element that intrinsically provides many sandboxing features.

2) 10gen Education

The MongoDB Web Shell will also make its debut on the 10gen Education platform where users learn to develop for and administer MongoDB quickly and efficiently. Currently, the assignments for the online classes must be downloaded and solved locally on an individual installation of MongoDB. With the web shell, students will be able to complete and submit assignments entirely within the education platform without needing to download and install their own copy of MongoDB or sample data. This online solution makes it more convenient for users to take courses wherever they may be and makes it easier to validate the correct completion of an assignment. Furthermore, users will no longer be required to load their own datasets using mongorestore since it is handled directly by the shell. Finally, since all the work is stored on their online account, students will be able to continue their work from where they left off from any computer.

3) MongoDB Documentation

Finally, the MongoDB Web Shell will be embedded into the API reference pages as part of the database’s documentation. This will allow a user to immediately test the commands detailed on the page and to evaluate its effects against provided sample data sets.

Resources

This project is Apache licensed open source software and is freely available through this Github repository. We have provided documentation using the Github wiki and welcome any feedback or contributions through issues and pull requests. The mentioned suspend.js library can be found at this Github repository.

About Us

The MongoDB Web Shell driver was developed by Ryan Chan and Danny Weinberg, who are currently interns at 10gen.

Acknowledgements

Special thanks to 10gen for hosting us as interns, Stacy Ferranti and Ian Whalen for managing the internship program, and our mentors Ian Bentley and Emily Stolfo for their help and support throughout the project.

← Previous

Do Hadoop and the Cloud Mean the End of the Closed Data Warehouse?

The data warehouse business has been great for at least a decade as almost every sizable company has been finding itself awash in data that it could not bring itself to part with, and was willing to pay a lot to have it stored, organized, and analyzed. But in an era of open tools – such as Hadoop - plus a push to lower costs by using commodity hardware, has the clock started to tick down on closed data warehouses? Data warehouses are fairly simple. They are built to accommodate a business’s data, often its structured data, although with a little tweaking unstructured data can be stored in them too. Everything is usually kept inside a relational database and - as always with relational databases - the upside is speed of processing and the downsides are some inflexibility about kinds of data that can be handled along with heavy demands for costly hardware to run the SQL databases. But as businesses look to cut costs, particularly with open source, and as commodity servers in the cloud become a new standard for running IT operations, this basic structure is beginning to be questioned. Over the long term, private warehouses will continue to thrive, but they likely will make themselves open to manipulation by open tools such as Hadoop and NoSQL databases. It’s increasingly said that enterprise needs a new data fabric that accommodates the growing diversity of data types. Hybrid kinds of arrangements - where relational databases blend in with NoSQL databases - will likely emerge as solutions, at least in the short term. But the other reality is that ours is an era of humongous data - really, really big data - and inevitably that will break many, perhaps not all, data warehouses as presently conceived. What’s keenly needed is big and fast data and the latter of course is not always easy to achieve with Hadoop, but there’s a stampede of plug-ins and tweaks designed to coax Hadoop into realtime, fast processing. MongoDB of course already has announced integration with Hadoop that allows MongoDB data sets also to be manipulated by Hadoop and a plus is that data set size - for all practical purpose - ceases to be much of an issue. However big the data, bring it on. The key of course is that - in their essences - Hadoop and MongoDB are fundamentally very different. MongoDB was written for data storage, not processing, and also data retrieval, while Hadoop was written to process data, particularly massive data sets. But those differences are why they can work together. Processing can be done in Hadoop, then sent for storage in MongoDB, building on the strengths of each. And that is exactly the kind of strengths needed in a world where the new normal is really big data. Ask yourself this, do you envision data warehousing needs diminishing, holding the same, or growing? If the last, get to know Hadoop because it will become a part of your life. At least you will hope it does.

August 13, 2013
Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024