Using SuperDuperDB to Accelerate AI Development on MongoDB Atlas Vector Search
Rate this article
Are you interested in getting started with vector search and AI on MongoDB Atlas but don’t know where to start? The journey can be daunting; developers are confronted with questions such as:
- Which model should I use?
- Should I go with an open or closed source?
- How do I correctly apply my model to my data in Atlas to create vector embeddings?
- How do I configure my Atlas vector search index correctly?
- Should I chunk my text or apply a vectorizing model to the text directly?
- How and where can I robustly serve my model to be ready for new searches, based on incoming text queries?
SuperDuperDB is an open-source Python project designed to accelerate AI development with the database and assist in answering such questions, allowing developers to focus on what they want to build, without getting bogged down in the details of exactly how vector search and AI more generally are implemented.
SuperDuperDB includes computation of model outputs and model training which directly work with data in your database, as well as first-class support for vector search. In particular, SuperDuperDB supports MongoDB community and Atlas deployments.
You can follow along with the code below, but if you prefer, all of the code is available in the SuperDuperDB GitHub repository.
Once you’ve installed SuperDuperDB, you’re ready to connect to your MongoDB Atlas deployment:
The trailing characters after the last “/” denote the database you’d like to connect to. In this case, the database is called "documents." You should make sure that the user is authorized to access this database.
The variable
db is a connector that is simultaneously:- A database client.
- An artifact store for AI models (stores large file objects).
- A meta-data store, storing important information about your models as they relate to the database.
- A query interface allowing you to easily execute queries including vector search, without needing to explicitly handle the logic of converting the queries into vectors.
Let’s see this in action.
With SuperDuperDB, developers can import model wrappers that support a variety of open-source projects as well as AI API providers, such as OpenAI. Developers may even define and program their own models.
For example, to create a vectorizing model using the OpenAI API, first set your
OPENAI_API_KEY as an environment variable:Now, simply import the OpenAI model wrapper:
To check this is working, you can apply this model to a single text snippet using the
predictmethod, specifying that this is a single data point with
one=True.Alternatively, we can also use an open-source model (not behind an API), using, for instance, the
sentence-transformers library:This code snippet uses the base
Model wrapper, which supports arbitrary model class instances, using both open-sourced and in-house code. One simply supplies the class instance to the object parameter, optionally specifying preprocess and/or postprocess functions. The encoder argument tells Atlas Vector Search what size the outputs of the model are, and the batch_predict=True option makes computation quicker.As before, we can test the model:
Let’s add some data to MongoDB using the
db connection. We’ve prepared some data from the PyMongo API to add a meta twist to this walkthrough. You can download this data with this command:You’ll see from this command that, in contrast to
pymongo, superduperdbincludes query objects (
Other than this fact,
Collection(...)...). This allows superduperdb to pass the queries around to models, computations, and training runs, as well as save the queries for future use.Other than this fact,
superduperdb supports all of the commands that are supported by the core pymongo API.Here is an example of fetching some data with SuperDuperDB:
You can see that the usual data from MongoDB is wrapped with the
Document class.You can recover the unwrapped document with
unpack:The reason
superduperdb uses the Document abstraction is that, in SuperDuperDB, you don't need to manage converting data to bytes yourself. We have a system of configurable and user-controlled types, or "Encoders," which allow users to insert, for example, images directly. (This is a topic of an upcoming tutorial!)Now you have chosen and tested a model and inserted some data, you may configure vector search on MongoDB Atlas using SuperDuperDB. To do that, execute this command:
This command tells
superduperdb to do several things:- Search the "documents" collection
- Set up a vector index on our Atlas cluster, using the text in the "value" field (Listener)
- Use the model variable to create vector embeddings
After receiving this command, SuperDuperDB:
- Configures a MongoDB Atlas knn-index in the "documents" collection.
- Saves the model object in the SuperDuperDB model store hosted on gridfs.
- Applies model to all data in the "documents" collection, and saves the vectors in the documents.
- Saves the fact that the model is connected to the "pymongo-docs" vector index.
If you’d like to “reload” your model in a later session, you can do this with the
load command:To look at what happened during the creation of the VectorIndex, we can see that the individual documents now contain vectors:
The outputs of models are always saved in the
"_outputs.<key>.<model>" path of the documents. This allows MongoDB Atlas Vector Search to know where to look to create the fast vector lookup index.You can verify also that MongoDB Atlas has created a
knn vector search index by logging in to your Atlas account and navigating to the search tab. It will look like this:
The green
ACTIVE status indicates that MongoDB Atlas has finished comprehending and “organizing” the vectors so that they may be searched quickly.
If you navigate to the “...” sign on Actions and click edit with JSON editor*,* then you can inspect the explicit index definition which was automatically configured by
superduperdb:
You can confirm from this definition that the index looks into the
"_outputs.<key>.<model>" path of the documents in our collection.Now that our index is ready to go, we can perform some “search-by-meaning” queries using the
db connection:🚀 So that’s it! 🚀
You’ve now queried a vector search index on MongoDB Atlas Vector Search using a model and setup installed with SuperDuperDB. This required only a few key commands in Python, utilizing model libraries and API clients from the Python open-source ecosystem!
superduperdb has lots more to offer:- Developers can bring their own model, or install arbitrary models from the open-source ecosystem.
- The cohere and anthropic APIs are also supported, in addition to openai.
- Developers may also search through images and videos.
- Other use cases, in addition to vanilla vector search, are supported:
- Chat with your docs
- Classical machine learning
- Transfer learning
- Vector search with arbitrary data-types
- Much, much more…
SuperDuperDB is open source and permissively licensed under the Apache 2.0 license. We would like to encourage developers interested in open-source development to contribute to our discussion forums and issue boards and make their own pull requests. We'll see you on GitHub!
We are looking for visionary organizations we can help to identify and implement transformative AI applications for their business and products. We're offering this absolutely for free. If you would like to learn more about this opportunity, please reach out to us via email: partnerships@superduperdb.com.
Rate this article