4 Ways MongoDB Enhances Your Google BigQuery Experience

Cara Heimbaugh
December 1, 2022
#Cloud

MongoDB and Google Cloud continue to build on their partnership, with MongoDB enhancing Google Cloud with pay-as-you-go abilities, unified billing, and integrations with multiple different GC features, including BigQuery. And, when it comes to data architecture, BigQuery and MongoDB are two products that are better together.

Google BigQuery and MongoDB are better together

Google’s serverless data warehouse, BigQuery, was launched in 2011 with an aim to enhance business agility as their cloud-native data warehouse. BigQuery allows for fast queries that can uncover insights using familiar SQL.

When MongoDB is added to the database technology stack as a complementary technology, it enhances the breadth of capabilities for the developer across a variety of use cases, including the following four examples.

Combined impact of the Enterprise Data Warehouse and the Operational Data Store

BigQuery is best suited as an Enterprise Data Warehouse (EDW), meaning it is designed to optimize long-running analytics. MongoDB Atlas, on the other hand, is best suited as an Operational Data Store (ODS), designed to optimally support high throughput and highly concurrent real-time operational applications that demand random access to an entity’s data in native JSON.

This combination means that BigQuery and MongoDB are complementary technologies that can jointly deliver more value — each delivering on their strongest qualities. BigQuery excels at long-running queries, while Atlas handles the real-time operational application needs with thousands of concurrent sessions and millisecond response times.

Enriched end-customer experiences

BigQuery enables data scientists and analysts with machine learning (ML) models and BI tools for structured and semi-structured data at scale. For roles that need results with a turnaround time of a day or more, BigQuery is a strong tool for big data queries. With MongoDB Atlas, engineers and development teams can build applications faster and handle highly diverse schema, query, and update patterns, adapting to demanding user needs and competition. Atlas can also deliver the real-time or less than 24-hour queries that are necessary to keep your business operational. Additionally, data can easily move back and forth between the two platforms, creating a prime combination for running analytics on operational data.

Being able to unlock the full potential of your data across your organization means that everyone has the insight into the business metrics they need, when they need it. This allows quicker decision making, as well as stronger and more accurate reporting.

Extensibility to MongoDB Atlas features

On top of the value and synergy that can be realized by a BigQuery+Atlas combination, other Atlas features can help enhance the usefulness and sophistication of a data architecture, such as:

  • Atlas Charts can be leveraged to create rich visualizations of any data stored within Atlas.

  • Atlas Triggers and Alerts can apply database logic in response to events or on a predefined schedule.

  • Atlas Search brings full-text search at scale to all data across MongoDB and BigQuery alike.

  • Atlas Data Federation enables aggregating data across multiple data sources, such as Atlas clusters and HTTPS endpoints, and transforming it into analytical formats (e.g., Parquet).

This means you can not only access data in real-time, but you can also analyze it in a visual, user-friendly way. This functionality makes your data more actionable, allowing you not only to answer questions about your business data but also make better predictions and future adjustments based on it. Furthermore, being alerted to certain data-based events and triggering new actions based on that information means you can have your data working more efficiently for you, freeing up time to innovate and focus on core business competencies. Lastly, this approach simplifies your data lifecycle, so JSON data from various applications and endpoints can easily be transformed and consumed for rich analytics.

Deeper understanding of your customer

Businesses can use fully managed MongoDB Atlas to store customer 360 profiles. A 360-degree view of a customer allows businesses to track an individual customer’s journey across multiple channels, devices, purchases, and interactions, and improves customer satisfaction. With the combination of Atlas and BigQuery, businesses can also use compiled data — such as, transactional data, behavioral data, user profile and segmentations, and business analytics — to match user profiles with products and services using Artificial Intelligence (AI). Vertex AI, a managed machine learning platform, provides all the Google cloud services in one place to deploy and maintain AI models.

Being able to easily access a 360 view for each customer and have automation around their customer journey helps with customer engagement and loyalty by improving customer satisfaction and retention through personalization and targeted marketing communications. It also enables retailers to aggregate customer interactions across all channels and identify valuable new customers.

Google BigQuery and MongoDB Atlas in the real world

Current, a leading U.S. challenger bank, uses innovative approaches, services, and technologies to serve people overlooked by traditional banks, regardless of age or income level, to help improve their financial outcomes. To help create customer experiences that cannot exist in traditional systems, Current chose to leverage Google Cloud, including BigQuery, with MongoDB layering the platform to achieve their goals.

Are you a Google BiqQuery customer that is curious about how MongoDB Atlas can amplify your existing data warehouse or data lake architecture? Try MongoDB Atlas for free today and spin up your first workload in minutes.

Try pay-as-you-go Atlas on GC Marketplace

← Previous

Achieving Industrial Connectivity at Scale with Wimera and MongoDB

Industry 4.0 (I4.0) represents the beginning of the Fourth Industrial Revolution. It includes the current trend of automation technologies in the manufacturing industry as well as disruptive technologies and concepts, such as cyber-physical systems (CPS), Industrial Internet of Things (IIoT), cloud computing, and immersive visualization. Through Industry 4.0, embedded systems, semantic machine-to-machine communication, IIoT, and CPS technologies are integrating the virtual space with the physical world. These technologies are enabling a new generation of industrial systems, such as smart factories, to deal with the complexity of fast-paced and hyper-personalized production. In this article, we’ll explore Wimera’s unique solutions to the challenges of I4.0 and IIoT, built with MongoDB. Information and insights With IIoT, existing industrial systems will be modernized to drive digital transformation and unlock tomorrow's smart enterprise. IIoT has been finding its way into products and sensors while revolutionizing existing manufacturing systems; thus, it is considered a key enabler for the next generation of advanced manufacturing. Industry 4.0 generally comprises many complex components and has broad applications in all manufacturing sectors. The first challenge faced by manufacturing companies when embarking on the I4.0 journey is to sensorize and connect their manufacturing equipment in order to collect, store, and analyze data for information and insights. Wimera Systems is solving this challenge as an I4.0 enablement company offering IIoT solutions using their unique hardware, software application, and AI/ML-based analytics engine. Wimera’s Smart Factory Suite has seen tremendous growth, with 2500+ global installations across 50+ customers. MongoDB has been pivotal to that growth, acting as the core component of the IIoT suite and enabling the company to offer its services at scale without having to worry about managing the complexity of an IIoT database. Bringing AI-powered IIoT to the manufacturing shop floor Manufacturing companies are emerging from the pandemic with a renewed focus on digital transformation and smart factories investment. COVID-19 has heightened the need for IIoT technology and innovation, forcing manufacturers to compete in a digitalized business environment. Many manufacturers still operate using legacy technologies and systems; on most shop floors, equipment and operator efficiency are manually calculated and tracked using spreadsheets. The machines are maintained using time-based rather than condition-based maintenance strategies. And, no real-time visibility exists on consumables and tools usage. All these practices result in increased maintenance costs, suboptimal production, and ultimately, customer dissatisfaction. Wimera understands these challenges all too well, which is why they created the Smart Factory Suite supporting both on-premise and cloud deployments. The Smart Factory Suite provides insights for managing the entire production landscape through interconnected devices and machines, operations, and facilities. It can predict and make real-time adjustments for increased production efficiency and less downtime. The suite is primarily utilized for empowering manufacturing operations, equipment maintenance, warehouse operations, and inventory management. With Smart Factory Suite, Wimera serves a wide range of manufacturing industry sectors including, but not limited to, automotive, electronics, chemical, and food processing companies. Deploy and run anywhere with MongoDB MongoDB, with its freedom to run anywhere, lets Wimera offer both on-premises and cloud deployment options for its customers. In both cases, the suite is directly connected with machine controllers using Wimera libraries for all popular Programmable Logic Controller (PLC) brands. The suite is also connected to legacy machines through external sensors installed by the Wimera team. Data is extracted via the Wimera ReMON Data Acquisition (DAQ) device (Figure 1) that utilizes the MongoDB database as the persistent data storage. MongoDB’s flexible data model makes it easy to combine and enrich this data and enables live dashboards and instant alerts for factory personnel. The data collected and optimized by ReMON DAQ is further fed to ReMON AI , an advanced analytics engine. ReMON AI provides advanced analytics through AI/ML models and leverages MongoDB to deliver application-driven analytics in real time. Figure 1: ReMON DAQ and ReMON AI (source: Wimera ReMON ). Whether through on-premises or cloud deployment (Figures 2 and 3), Wimera’s customers have benefited from MongoDB’s capabilities that are critical for IIoT applications, such as time series collections and the flexible, intuitive document data model. Figure 2: Wimera IoT architecture on premises. Figure 3: Wimera IoT architecture on cloud (using MongoDB on AWS). In one customer example, while deploying IIoT at a multinational CNC machine shop, the customer preferred to use their existing production monitoring application enriched with IoT data coming from Wimera’s Smart Factory Suite. In this case, MongoDB enabled easy and seamless integration of the IoT application with the customer's application via a simple API. Additionally, high-speed data coming from a vibration sensor was handled effectively by MongoDB time series collections, resulting in real-time alerts sent to maintenance teams for instant corrective actions on the shop floor. In another example, a multinational automotive manufacturer wanted a single platform that could collect and combine data coming from vendors in different formats and contexts. MongoDB's flexible document model helped manage the varied data types easily, allowing the customer to benefit from a single application capable of managing multiple vendors in parallel. This flexibility offered by MongoDB enables the customer to keep adding new vendors instantly without changing the underlying cloud infrastructure or tweaking schemas. Interested readers can check out additional case studies on Wimera’s website. Building better together Wimera and MongoDB’s partnership gives customers confidence with validated architectures to ensure successful, optimized, and scalable deployments at their facilities. Wimera’s continued partnership with MongoDB also helps guide the company’s product roadmap as we expand in the IIoT, Smart Factory market together. MongoDB is the only enterprise grade database chosen by the Wimera development team due to easy handling of the large volume of data generated from machines and sensors while maintaining a high performance… If we want to insert thousands of records in a second, then MongoDB is the best choice for that given our solutions are for Industrial IoT. Also, horizontal scaling (adding new columns) is not an easy process in any RDBMS system. But in the case of MongoDB, it is very easy Nagarajan Narayanasamy, CEO, Wimera Systems Private Limited A bright future ahead Since 2019, Wimera has been an early adopter of MongoDB for their Industrial IoT application for discrete manufacturing industries and process industries on multiple domains. “Currently, Narayanasamy says, “Wimera’s Industrial IoT solutions are matured, and we are focused on scaling globally.” Wimera now targets expansion in India, APAC, EU, and USA for the discrete manufacturing and process industries and also for select OEMs and machine builders. “As MongoDB continues to scale itself globally through its multi-cloud data distribution strategy, we see a good synergy partnering with MongoDB for the mutual benefit of both companies and the community as a whole. We also would like to work with MongoDB on the technology roadmap and solve some of the real-life challenges faced by manufacturing industries,” Narayanasamy says. Wimera has recently started their MongoDB Atlas journey, and the adoption will grow as their customers demand more cloud solutions compared to current on-premises deployments. MongoDB will continue to help IoT companies like Wimera take their product offering to the next level and enable their customers to digitally transform their manufacturing operations. Thank you to Karolina Ruiz Rogelj for her contributions to this post. To learn more about MongoDB’s role in industrial connectivity and IIoT, please visit our Manufacturing and Industrial IoT page.

December 1, 2022
Next →

VertexAI and MongoDB for Intelligent Retail Pricing

In today’s competitive retail environment, the ability to quickly adjust pricing in response to market trends, consumer demand, and competitors’ moves is not just an advantage — it's essential for survival. This is where dynamic pricing comes into play, serving as a strategic tool for businesses to pull in their quest for market dominance. Dynamic pricing goes beyond changing numbers; it’s a strategic approach that reflects the dynamic nature of the market, powered by data-driven insights that enable prices to be adjusted in real-time for maximum effectiveness. This shift towards a more agile, data-driven pricing strategy underscores a broader trend in the business world: the recognition of data as a foundational element in decision-making processes. By leveraging real-time data, businesses can ensure their pricing strategies are not only responsive to market fluctuations but also strategically aligned with their overall business objectives, thus driving retail competitiveness to new heights. Let’s uncover how integrating both platforms empowers developers when it comes to delivering best-in-class, data-driven applications. MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More Google Cloud: A platform for real-time analytics and AI Google Cloud stands out as a powerhouse in real-time analytics and artificial intelligence (AI), offering the infrastructure necessary for dynamic pricing strategies and other data-driven business approaches. It's designed to facilitate big data analysis, machine learning, and operational agility. Built-in tools form the backbone of an effective dynamic pricing strategy. These include Vertex AI for advanced machine learning models following best- in- class MLOps practices, and Pub/Sub for real-time messaging to solve real- time data ingestion. By harnessing the power of Google Cloud, retailers can analyze vast quantities of data in real-time, from current market trends to customer behavior and competitor pricing. This enables businesses to make informed decisions swiftly, adjusting their pricing strategies to reflect the ever-changing market conditions. MongoDB: Flexible data modeling and rapid application development MongoDB complements Google Cloud by offering a high performance document- based database with a flexible data model that allows rapid application development. For pricing data in particular, where there may be different variants for different sizes of store or country, the flexibility allows for the ease of storage of complex or hierarchical data. In addition, polymorphic capabilities allow you to use a single interface to represent different types, making your system more flexible. It also supports scalability as new types can be easily integrated. Lastly, it enhances efficiency by allowing the same operation to behave differently based on the object, reducing code redundancy. This flexible schema also enables seamless integration with AI models. MongoDB Atlas supports workload isolation , ensuring dedicated resources for AI tasks and smooth operation alongside core application workloads. Additionally, change streams and triggers can be utilized to capture real-time updates in the pricing data, allowing the AI model to be called upon for immediate analysis and adaptation and enabling in-app analytics for retailers to gain a competitive edge. Figure 1: MongoDB replica set: Workload Isolation In the dynamic pricing reference architecture, Atlas collections function as an ML feature store. By leveraging the capabilities of MongoDB Atlas as a developer data platform, we are able to embed real-time automated decision-making into our e-commerce applications and reduce operational overhead for both business operations and MLOps model fine-tuning. This is achieved through implementing a streamlined approach to data management, incorporating real-time, automated decision making, workload isolation, change streams, triggers for immediate updates, and seamless integration with AI models. Dynamic prising microservice overview Building an event-driven AI architecture leveraging MongoDB Atlas in Google Cloud is straightforward. We can summarize our dynamic pricing microservice by first describing the different components of its architecture, what they are used for, and how they interact with each other: Figure 2: Description of the different technology components of a dynamic pricing microservice and what they are used for. Handling data sources The proposed solution uses Google Cloud Pub/Sub to ingest data sources like customer behavior events in JSON format. Using a technology like Pub/Sub allows for scaling to handle a large number of messages and efficiently distribute them to many subscribers. This is partly because it allows for parallel processing of messages and can be distributed across multiple servers or instances. It is often a fundamental pattern in event-driven architectures, where the flow of the program is determined by events or messages, supporting reactive programming and making the system more responsive and efficient. Data federation We’ll use Vertex AI Notebooks to clean the data and train a TensorFlow model. This model will learn the non-linear relation between customer events, products names, and prices, enabling it to calculate the optimal predicted price. Orchestrating Using Cloud Functions, we orchestrate the customer events coming from the Pub/Sub topic to be converted into tensors, which are then stored in a MongoDB Atlas collection. This collection acts as a feature store serving as a centralized repository designed to store, manage, and serve features for machine learning (ML) models. Features represent individual measurable properties or characteristics used by ML models to make predictions or decisions. MongoDB’s document model flexibility paired with the document versioning pattern will allow us to design time-sensitive chunks of events and granularly manage the training datasets for our models. Serving The Cloud Function will use the event tensor to invoke our trained model that is served in a Vertex AI endpoint. The model will provide a predicted price score that can then be inserted into our product catalog stored in MongoDB so our e-commerce application can read the price change in real time. Dynamic pricing architecture: Putting it all together In the following architecture diagram, the blue data flow illustrates how customer event data is ingested into a Pub/Sub topic. This allows us to make a push subscription to a Cloud Function from the topic. This function orchestrates the data transformation from raw event into a tensor and calls an endpoint to then update the predicted price into our MongoDB product catalog collection. By using this architectural approach, we can isolate raw events threads and build different services around them, reacting in real time for dynamic pricing or asynchronously for model training. With every component loosely coupled, we prevent the system from crashing completely. Moreover, publishers and subscribers can continue to process their logic without the need for the other components to receive or publish messages. Figure 3: Dynamic pricing architecture integrating different Google Cloud components and MongoDB Atlas as a Feature Store For businesses, this translates into more precise and responsive pricing strategies. In the model building and optimization phase, by utilizing TensorFlow within Google Cloud Vertex AI notebooks, retailers can harness the power of deep learning capabilities. The neural network model is capable of analyzing intricate patterns and relationships within large datasets. This is how businesses may capture nuanced market dynamics, customer behavior, and pricing elasticity with greater accuracy, leading to more optimized pricing decisions. But even the best of the models should be consistently optimized. Maintaining model effectiveness requires continuous adaptation. Regularly evaluating accuracy and performing feature engineering ensures your models stay sensitive to market changes. This underscores the importance of retraining as a core principle in a continuous improvement data science approach. Using MongoDB Atlas as your operational data layer means that your feature store is always accessible, reducing downtime and improving the efficiency of machine learning operations. On the other hand, cross-region deployments can bring features closer to where machine learning models are being trained or served, reducing latency and improving model performance. Get started The integration of Google Cloud and MongoDB presents an easy approach to modernizing dynamic pricing strategies. Leveraging real-time analytics, flexible data modeling, and reactive microservices architecture, it empowers businesses to achieve operational efficiencies and gain a competitive advantage in their pricing strategies. For retailers looking to elevate their pricing strategies, considering a strategic partnership with both technologies is essential. For a deeper dive into integrating the different components of this architecture, make sure to check our GitHub repository.

April 17, 2024