Cloud Native Enterprise Data Management with Tardigrade for MongoDB

MongoDB and Tardigrade are excited to collaborate and demonstrate managing data backups and migrations across multiple computing and storage platforms with enterprise security and SLAs. Tardigrade, the new S3-compliant storage solution from Storj Labs, can easily be configured to store all your database backups and snapshots from MongoDB Ops Manager, the management platform for enterprise MongoDB deployments. In this post, we’ll give an overview of how all this works in cloud-native environments. For this, we’ll install the entire MongoDB Enterprise stack into a Kubernetes cluster along with a small gateway app from Tardigrade and we’re ready to go!

Most companies have multiple engineering teams supporting multiple apps. These apps need data services and specifically require dynamic provisioning of new environments along with integrated platform management in order to continuously support their businesses. Each application, then, in turn, has its own requirements for security, high-availability and disaster recovery scenarios. We can combine tech from MongoDB and Tardigrade to support all these variable requirements.

The best part of Tardigrade is the service includes features by default that would otherwise cost organizations a large premium. For example, all data stored on Tardigrade is end-to-end encrypted by default, meaning you don’t have to do any special configuration or run any special software to keep your database backups and snapshots safe, private, and out of the hands of would-be hackers.

Every file uploaded to Tardigrade also has cross-geography redundancy by default, which is similar to a multi-region cloud backup architecture. This means your cloud storage backups will always be accessible, and a single outage won’t cripple your operations like it would in a centralized cloud storage architecture. Tardigrade backups are available everywhere, and with hot availability, developers can quickly rehydrate their MongoDB instance from anywhere in the world.

In addition to being half the cost of centralized cloud storage services, Tardigrade is also more affordable should you need to access your data. Egress bandwidth is half the price of Google Cloud, Amazon S3, and Azure and even more affordable than centralized backup architectures, which generally charge a hefty premium should you need to access your data.

MongoDB Enterprise supports a variety of cloud-native deployment options. This gives you and your apps access to locally deployed MongoDB clusters alongside direct access to MongoDB clusters running in the MongoDB Atlas cloud. Enterprises can now offer controlled access to advanced MongoDB data platform features directly to developers and CI/CD pipelines. Figure 1 depicts the overall architecture.

Demonstration

To see how all this works, we’ll spin up a deployment on GKE with everything. With this setup, we can create MongoDB clusters, managed by MongoDB Ops Manager. The entire deployment automates the configuration of an S3-compliant bucket to store backups.

Key Technologies

  • MongoDB Enterprise Kubernetes Operator
  • MongoDB Enterprise Ops Manager

  • S3-gateway - Tardigrade
  • Kubernetes

This example presents a v3 Helm chart which installs a complete MongoDB Enterprise database-as-a-service platform. There is a subchart for installing the Tardigrade S3-gateway to store backups and snapshots along. There are a number of components in this deployment. The workloads view in GCP shows the main ones:

This entire setup and more can be found in this repository https://github.com/jasonmimick/total-cluster. Note the special Helm chart in total-cluster/addons/tardigrade-gateway to set up communication between our private Kubernetes cluster and the Tardigrade decentralized cloud storage service.

Your first step is to procure a Kubernetes cluster. A couple of examples can be found in the total-cluster/quickstarts folder, but most any cluster should work. This full example will require 5 to 7 worker nodes with 2 nodes having at least 8 GB of ram.

Next, set up your Tardigrade Gateway. Check out the details for that over in the gateway chart in total-cluster. Installing the chart (Note: requires Helm V3),

➜  addons git:(master) ✗ helm install s3-gateway tardigade-gateway 
NAME: s3-gateway
LAST DEPLOYED: Sat May 30 07:52:43 2020
NAMESPACE: total-cluster
STATUS: deployed
REVISION: 1
TEST SUITE: None

Here, we can inspect the cluster and see our local gateway running. This service provides a local secure connection for MongoDB Ops Manager to write database backups.

➜  addons git:(master) ✗ kubectl get all --selector='app=s3-gateway-tardigrade-gateway'
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/s3-gateway-tardigrade-gateway-68fbf4b4d7-4qbvt   1/1     Running   0          104s

NAME                                        TYPE        CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/s3-gateway-tardigrade-gateway-svc   ClusterIP   10.43.65.192 <none> 7777/TCP   104s

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/s3-gateway-tardigrade-gateway   1/1     1            1           104s

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/s3-gateway-tardigrade-gateway-68fbf4b4d7   1         1         1       104s

Once your gateway is set up and running, install the entire MongoDB data platform with the total-cluster chart.

git clone https://github.com/jasonmimick/total-cluster
helm install mongodb total-cluster

The entire chart takes between 5 to approx 10 mins to finish installing. This is because it’s installing quite a few components.

Ops Manager needs to be configured to use the gateway. You can connect to your Ops Manager with a basic port-forward command like this,

kubectl port-forward mongodb-ops-manager-0 8080:8080

Connect your browser to localhost:8080 and use admin@mongodb.com and MongoDB12345% to login.

Note: These credentials are stored in a secret, called mongodb-ops-manager-secret. You can update this in the chart. Also note the “mongodb” in the secret name comes directly from the Helm release name.

You should see your MongoDB cluster once connected:

Setting up the S3 Blockstore pointing to local gateway

You can configure backups for your app databases following this guide: Backup Overview.

Screen shot of backup taking place.

Finally, you can see the backup data blocks stored in the Tardigrade bucket. You can even use the local tardigrade-gateway with command s3 command line tools:

$kubectl port-forward tardigrade-gateway-tardigrade-gateway-75cfdb889-nz2n4 7777:7777 &
$export AWS_ACCESS_KEY_ID=XXX
$export AWS_SECRET_ACCESS_KEY=XXX 
$aws --endpoint-url http://localhost:7777 s3 ls test

The Ops Manager extensive API can be used to automate backup restoration along with many devops tasks.

Conclusion

We look forward to seeing what MongoDB users do with Tardigrade. We regularly hear from users who would like cloud backup services that provide end-to-end encryption, multi-region redundancy, as well as lower and more predictable costs compared to centralized services. While relatively new, Tardigrade delivers on all of these needs, while still offering the same enterprise service level agreements users absolutely require.

Learn more

← Previous

Peek at your MongoDB Clusters like a Pro with Keyhole: Part 3

Performance evaluation and tuning is an important part of application development life cycle. Keyhole is an open source project tool for MongoDB.

June 11, 2020
Next →

Building AI With MongoDB: Integrating Vector Search And Cohere to Build Frontier Enterprise Apps

Cohere is the leading enterprise AI platform, building large language models (LLMs) which help businesses unlock the potential of their data. Operating at the frontier of AI, Cohere’s models provide a more intuitive way for users to retrieve, summarize, and generate complex information. Cohere offers both text generation and embedding models to its customers. Enterprises running mission-critical AI workloads select Cohere because its models offer the best performance-cost tradeoff and can be deployed in production at scale. Cohere’s platform is cloud-agnostic. Their models are accessible through their own API as well as popular cloud managed services, and can be deployed on a virtual private cloud (VPC) or even on-prem to meet companies where their data is, offering the highest levels of flexibility and control. Cohere’s leading Embed 3 and Rerank 3 models can be used with MongoDB Atlas Vector Search to convert MongoDB data to vectors and build a state-of-the-art semantic search system. Search results also can be passed to Cohere’s Command R family of models for retrieval augmented generation (RAG) with citations. Check out our AI resource page to learn more about building AI-powered apps with MongoDB. A new approach to vector embeddings It is in the realm of embedding where Cohere has made a host of recent advances. Described as “AI for language understanding,” Embed is Cohere’s leading text representation language model. Cohere offers both English and multilingual embedding models, and gives users the ability to specify the type of data they are computing an embedding for (e.g., search document, search query). The result is embeddings that improve the accuracy of search results for traditional enterprise search or retrieval-augmented generation. One challenge developers faced using Embed was that documents had to be passed one by one to the model endpoint, limiting throughput when dealing with larger data sets. To address that challenge and improve developer experience, Cohere has recently announced its new Embed Jobs endpoint . Now entire data sets can be passed in one operation to the model, and embedded outputs can be more easily ingested back into your storage systems. Additionally, with only a few lines of code, Rerank 3 can be added at the final stage of search systems to improve accuracy. It also works across 100+ languages and offers uniquely high accuracy on complex data such as JSON, code, and tabular structure. This is particularly useful for developers who rely on legacy dense retrieval systems. Demonstrating how developers can exploit this new endpoint, we have published the How to use Cohere embeddings and rerank modules with MongoDB Atlas tutorial . Readers will learn how to store, index, and search the embeddings from Cohere. They will also learn how to use the Cohere Rerank model to provide a powerful semantic boost to the quality of keyword and vector search results. Figure 1: Illustrating the embedding generation and search workflow shown in the tutorial Why MongoDB Atlas and Cohere? MongoDB Atlas provides a proven OLTP database handling high read and write throughput backed by transactional guarantees. Pairing these capabilities with Cohere’s batch embeddings is massively valuable to developers building sophisticated gen AI apps. Developers can be confident that Atlas Vector Search will handle high scale vector ingestion, making embeddings immediately available for accurate and reliable semantic search and RAG. Increasing the speed of experimentation, developers and data scientists can configure separate vector search indexes side by side to compare the performance of different parameters used in the creation of vector embeddings. In addition to batch embeddings, Atlas Triggers can also be used to embed new or updated source content in real time, as illustrated in the Cohere workflow shown in Figure 2. Figure 2: MongoDB Atlas Vector Search supports Cohere’s batch and real time workflows. (Image courtesy of Cohere) Supporting both batch and real-time embeddings from Cohere makes MongoDB Atlas well suited to highly dynamic gen AI-powered apps that need to be grounded in live, operational data. Developers can use MongoDB’s expressive query API to pre-filter query predicates against metadata, making it much faster to access and retrieve the more relevant vector embeddings. The unification and synchronization of source application data, metadata, and vector embeddings in a single platform, accessed by a single API, makes building gen AI apps faster, with lower cost and complexity. Those apps can be layered on top of the secure, resilient, and mature MongoDB Atlas developer data platform that is used today by over 45,000 customers spanning startups to enterprises and governments handling mission-critical workloads. What's next? To start your journey into gen AI and Atlas Vector Search, review our 10-minute Learning Byte . In the video, you’ll learn about use cases, benefits, and how to get started using Atlas Vector Search.

April 25, 2024