Google’s Datastream Powers Seamless MongoDB Integration into BigQuery

Venkatesh Shanbhag, Yang Li, and Dhirendra Sinha
July 21, 2025

Google’s Datastream service now offers public preview support for MongoDB as a source, marking an exciting expansion of its data streaming capabilities. This new feature enables users to seamlessly ingest data from MongoDB databases into Google’s BigQuery and Cloud Storage for real-time insights and enhanced data-driven decision-making.

MongoDB Atlas has emerged as a cornerstone of modern application development, and is celebrated for its flexible document model, horizontal scalability, and high performance. As a leading NoSQL database, it's the go-to choice for applications requiring agile schema evolution, handling diverse data types, and supporting rapid iteration cycles. From real-time analytics dashboards to content management systems and IoT data ingestion, MongoDB Atlas's versatility allows developers to build robust, scalable, and responsive applications that can easily adapt to changing business needs and data structures. Its ability to store semi-structured and unstructured data makes it particularly powerful for dynamic datasets that don't fit neatly into traditional relational tables, which is one of the reasons MongoDB was recognized as a leader in the Gartner Magic Quadrant.

Supercharging MongoDB with BigQuery analytics

MongoDB shines as an operational database, perfectly suited for transactional workloads and providing efficient, application-specific data access. For deep analytical insights, complex querying, and leveraging the power of machine learning and generative AI, moving this valuable data into a dedicated data warehouse like Google BigQuery becomes paramount. BigQuery offers petabyte-scale analytics, a serverless architecture, and powerful SQL capabilities, making it ideal for running complex queries across massive datasets, joining data from various sources, and performing advanced analytics.

Generative AI thrives on rich data, making the MongoDB operational insights invaluable. Structuring this data in BigQuery empowers you to train powerful AI models, build recommendation engines, perform sentiment analysis, and unlock entirely new revenue streams from your existing data.

Datastream helps to integrate MongoDB into BigQuery

Datastream is a serverless Change Data Capture (CDC) service that enables real-time data replication from various sources, including MongoDB, directly into BigQuery. It captures changes (inserts, updates, deletes) as they happen in your MongoDB database and streams them continuously and seamlessly to BigQuery, ensuring your analytical data warehouse is always up-to-date. For now, data destined for BigQuery will be delivered in JSON

This eliminates the need for complex batch processing, custom scripts, or manual data transfers, significantly reducing operational overhead and data latency. With Datastream, organizations can unlock immediate insights from their MongoDB data, fuel real-time dashboards, and empower their gen AI initiatives with the freshest possible information, all with minimal effort and maximum reliability.

Figure 1. MongoDB as a source connector on Google Datastream.

Screen grab of the Google Cloud dashboard showing MongoDB as the source connector.

The key benefits of Datastream

Better decisions and actionable Intelligence: With Datastream's low-latency replication, you can empower your business with up-to-the-minute insights from your MongoDB data.
Scalability and reliability: Datastream scales to handle large volumes of data and ensures reliable replication.
Fully managed: No need to manage infrastructure or worry about maintenance, freeing your team to focus on core tasks.
Wide support matrix: The MongoDB connectivity in Datastream supports Replica Sets and Sharded Clusters, as well as self-hosted and fully-managed Atlas databases.
Support for backfill and CDC: Datastream supports both backfill and CDC (change data capture) from a MongoDB source.
Secure by design: Datastream supports multiple secure, private connectivity methods to protect data in transit and encrypts it in transit and at rest.

With Datastream's new MongoDB connector, you can effortlessly integrate your MongoDB data. This means greater data flexibility and the ability to make smarter, data-driven decisions. Start leveraging your MongoDB information to innovate and boost business growth today. Connecting your MongoDB databases to Datastream is a simple process—just follow the easy steps in the Datastream documentation to begin data replication.

Ready to get started with MongoDB and Google Cloud? Check out the Google Cloud Marketplace.

← Previous

Innovating with MongoDB | Customer Successes, July 2025

How time flies! Summer is in full swing, and it’s already time for another MongoDB customer success roundup. This month, we’re focusing on customers who have combined the flexibility of MongoDB Atlas with cutting-edge AI advancements to unlock insights and fuel innovation. Let’s be honest—AI is everywhere. We’re intrigued, inspired, and maybe a little overwhelmed by its possibilities. But the hype exists for a good reason: AI is a groundbreaking technology that’s poised to transform every industry, job, and task, and it’s fundamentally changing how software interacts with data. We’re quickly learning, though, that delivering meaningful outcomes with AI requires the right infrastructure. With MongoDB Atlas , companies are leveraging vector search, seamless document modeling, and large language model (LLM) integrations to make smarter use of their data in real time. Whether that means enhancing engagement, simplifying decision-making, or enabling more efficient processes, MongoDB is helping organizations redefine how they leverage AI to solve critical challenges and create lasting impact. In this issue, I’m particularly excited to share the highlight of an impactful platform developed by CentralReach in the Autism care space – a cause near and dear to my family. They, along with customers like the Financial Times, Ubuy, and Base39, are demonstrating AI’s possibilities and transforming how data powers success. Ubuy Ubuy , an e-commerce platform serving customers in over 180 countries, needed a faster, more scalable solution to manage its catalog of over 300 million products. They were facing significant search performance bottlenecks, which impacted user experience and limited growth potential. By migrating from MySQL to MongoDB Atlas and leveraging Atlas Search and Atlas Vector Search , Ubuy reduced search response times from 4–5 seconds to milliseconds and enabled intent-driven product discovery powered by AI. Now, Ubuy easily handles over 150 million searches annually while delivering personalized recommendations and seamless scalability. AI-driven search enhancements have boosted customer engagement and SEO visibility, transforming global e-commerce and redefining how Ubuy customers access international products. Financial Times The Financial Times (FT) , a global leader in business journalism, wanted to deliver a hybrid search experience that combined traditional keyword precision with AI-driven discovery. With over a million daily searches, scaling this innovative solution quickly was critical. Using MongoDB Atlas—including Atlas Vector Search—the FT developed its AI-powered hybrid search in just 18 weeks. By blending full-text and semantic search capabilities, the solution delivers relevant recommendations instantly, enhancing content discovery for time-strapped readers. Partnering with MongoDB streamlined deployment, enabling the FT to surface hyper-relevant results while positioning itself as a leader in media innovation. With plans to roll out hybrid search across mobile apps and specialist titles next, the FT continues redefining how readers engage with trusted journalism in an AI-enabled world. Through our partnership with The Stack, learn how our customers are achieving extraordinary results with MongoDB. This exclusive content could spark the insights you need to drive your business forward. CentralReach CentralReach , a global leader in autism and intellectual and developmental disability (IDD) care technology, faced the challenge of managing 4 billion clinical data points annually while reducing administrative burdens for behavioral analysts. By building its Care360 platform on MongoDB Atlas and joining the MongoDB AI Applications Program (MAAP), CentralReach unified data across 62 million service appointments per year. With flexible document modeling, vector search, and advanced AI pipelines, the platform enables seamless access to patient records and intelligent querying, reducing manual workflows and improving care consistency. CentralReach’s AI-powered solution has streamlined processes, reduced documentation errors, and helped expand access to care for hundreds of thousands globally. With MongoDB Atlas’s scalability and powerful AI integrations, CentralReach is redefining autism care delivery. Base39 Base39 , a Brazil-based fintech, set out to streamline complex credit analysis using AI-driven insights. Manual processes and data scarcity limited efficiency and accuracy, often delaying loan assessments by up to 10 days. By leveraging MongoDB Atlas on AWS , as well as Atlas Vector Search and LLM integrations, Base39 transformed its workflow. With agentic AI and predictive algorithms, loan applications are now assessed in minutes, achieving 96% cost reductions and improved data insights. MongoDB’s flexible schema and native vector search capabilities helped boost productivity while cutting infrastructure costs by 84%. By empowering developers to focus on innovation instead of management, Base39 has set a new standard in AI-powered credit analysis. Video spotlight: Cisco Before you go, check out how Cisco is redefining innovation with generative AI while prioritizing security. Omar Santos, Distinguished Engineer at Cisco, shares how MongoDB Atlas Vector Search accelerated development and saved millions through smarter, safer AI applications. Want to get inspired by your peers and discover all the ways we empower businesses to innovate for the future? Visit MongoDB’s Customer Success Stories hub to see why these customers, and so many more, build modern applications with MongoDB.

July 17, 2025

Next →

3 “Lightbulb Moments” for Better Data Modeling

We all know that feeling: the one where you have been wrestling with a complex problem for hours, maybe even days, and suddenly, it just clicks. It is a rush of clarity, a "lightbulb moment" that washes away the frustration and replaces it with pure, unadulterated excitement. It's the moment you realize a solution is not just possible, it is elegant. This blog post is dedicated to that feeling—the burst of insight when you discover a more intuitive, performant, and powerful way to work. We have spoken with hundreds of developers new to MongoDB to understand where they get stuck, and we have distilled their most common "Lightbulb Moments" to help you get there faster. We found that the key to getting the best performance from MongoDB is to adjust the way you think about data. Once developers recognize that the flexible document model gives them more control, not less, they become unstoppable. In this inaugural post of our new “Lightbulb Moments” blog series, we will walk you through three essential data modeling tips on schema validation and versioning, the aggregation pipeline framework, and the the Single Collection Pattern. These concepts will help you structure your data for optimal performance in MongoDB and lead to your own "Lightbulb Moments," showing you how to build fast, efficient, and scalable applications. 1. Schema validation and versioning: Flexibility with control A common misconception about MongoDB is that its underlying document model is “schemaless.” With MongoDB, your schema is flexible and dependent on the needs of your application. If your workload demands a more structured schema, you can create validation rules for your fields with schema validation. If your schema requires more flexibility to adapt to changes over time, you can apply the schema versioning pattern. db.createCollection("students", { validator: { $jsonSchema: { bsonType: "object", title: "Student Object Validation", required: [ "address", "major", "name", "year" ], properties: { name: { bsonType: "string", description: "'name' must be a string and is required" }, year: { bsonType: "int", minimum: 2017, maximum: 3017, description: "'year' must be an integer in [ 2017, 3017 ] and is required" }, gpa: { bsonType: [ "double" ], description: "'gpa' must be a double if the field exists" } } } } } ) MongoDB has offered schema validation since 2017, providing as much structure and enforcement as a traditional relational database. This feature allows developers to define validation rules for their collections using the industry-standard JSON Schema. Schema validation gives you the power to: Ensure every new document written to a collection conforms to a defined structure. Specify required fields and their data types, including for nested documents. Choose the level of strictness, from off to strict , and whether to issue a warning or an error when a document fails validation. The most powerful feature, however, is schema versioning . This pattern allows you to: Gradually evolve your data schema over time without downtime or the need for migration scripts. Support both older and newer document versions simultaneously by defining multiple schemas as valid within a single collection using the oneOf operator. db.contacts.insertOne( { _id: 2, schemaVersion: 2, name: "Cameron", contactInfo: { cell: "903-555-1122", work: "670-555-7878", instagram: "@camcam9090", linkedIn: "CameronSmith123" } } ) Performance problems can stem from poor schema design—not the database. One example from a financial services company showed that proper schema design improved query speed from 40 seconds to 377 milliseconds. The ability to enforce a predictable structure while maintaining the flexibility of the document model gives developers the best of both worlds. For a deeper dive: See an example of schema optimization in MongoDB Design Reviews: how applying schema design best practices resulted in a 60x performance improvement by Staff Developer Advocate, Graeme Robinson. Learn how to use MongoDB Compass to analyze, export, and generate schema validation rules. 2. Aggregation pipeline framework: Simplifying complex data queries In SQL, developers often use JOINs to aggregate data across multiple tables. As joins stack up, queries can become slow and operationally expensive. Some may attempt a band-aid solution by querying each table separately and manually aggregating the data in their programming language, but this can introduce additional latency. MongoDB's Aggregation Framework provides a much simpler alternative. Instead of a single, complex query, you can break down your logic into an Aggregation Pipeline , or a series of independent pipeline stages. This approach offers several advantages: Easier to debug: Each pipeline stage can be developed and tested independently. Visual query building: MongoDB's stage-based Aggregation Pipeline simplifies query visualization compared to SQL. Tools like the Pipeline Builder in Atlas and Compass make it easy to build complex queries with stages like $match , $group , and $sort , while analyzing performance in real-time. Fewer joins: The framework's design encourages a schema that reduces the need for joins in the first place, as data that is accessed together should be stored together. When you do need to pull data from another collection, the $lookup operator handles it easily. Figure 1. MongoDB Compass aggregation pipeline builder. The ability to decompose complex queries into independent stages is a powerful new experience that long-time SQL users will find far smoother. For more information and case studies, see: MongoDB official resources on Aggregation and Aggregation Operations . Take a free, one-hour MongoDB Skills course on Fundamentals of Data Transformation to learn how to build aggregation pipelines to process, transform, and analyze data efficiently in MongoDB. Aggregation Optimization in MongoDB: A Case Study From the Field (Part 1) by Staff Developer Advocate, Graeme Robinson. 3. The Single Collection Pattern: Storing data together for faster queries Many developers new to MongoDB create a separate collection for each entity. If someone were building a new book review app, they might have collections for books , reviews , and users . While this seems logical at first, it can lead to slow queries that require expensive $lookup operations or multiple queries to gather all the data for a single view, which can slow down your overall app and increase your database costs. A more efficient approach is to use the Single Collection Pattern. This pattern helps model many-to-many relationships when embedding is not an option. It lets you store all related data in a single collection, avoiding the need for data duplication when the costs outweigh the benefits. This approach adheres to three defining characteristics: All related documents that are frequently accessed together are stored in the same collection. Relationships between the documents are stored as pointers or other structures within the document. An index is built on a field or array that maps the relationships between documents. Such an index supports retrieving all related documents in a single query without database join operations. When using this pattern, you can add a docType field (e.g., book, review, user) and use a relatedTo array to link all associated documents to a single ID. This approach offers significant advantages: Faster queries: A single query can retrieve a book, all of its reviews, and user data. No joins: This eliminates the need for expensive joins or multiple database trips. Improved performance: Queries are fast because all the necessary data lives in the same collection. For developers struggling with slow MongoDB queries, understanding this pattern is a crucial step towards better performance. For more details and examples, see: Building with Patterns: The Single Collection Pattern by Staff Senior Developer Advocate, Daniel Coupal. Take a free, one-hour MongoDB Skills course on Schema Design Optimization . Get started with MongoDB Atlas for free today. Start building your MongoDB skills through the MongoDB Atlas Learning Hub .

September 15, 2025