Introducing Snapshot Distribution in MongoDB Atlas

Ashley Kuhwilm
September 29, 2022
#Atlas

Data is at the heart of everything we do and in today’s digital economy has become an organization's most valuable asset. But sometimes the lengths that need to be taken to protect that data can present added challenges and result in manual processes that ultimately slow development, especially when it comes to maintaining a strict backup and recovery strategy.

MongoDB Atlas aims to ease this burden by providing the features needed to help organizations not only retain and protect their data for recovery purposes, but to meet compliance regulations with ease. Today we’re excited to announce the release of a new backup feature, Snapshot Distribution. Snapshot Distribution allows you to easily distribute your backup snapshots across multiple geographic regions within your primary cloud provider with the click of a button.

You can configure how snapshots are distributed directly within your backup policy and Atlas will automatically distribute them to other regions as selected—no manual process necessary.

How to distribute your snapshots

To enable Snapshot Distribution, navigate to the backup policy for your cluster and select the toggle to copy snapshots to other regions. From there, you can add any number of regions within your primary cloud provider—including regions you are not deployed in—to store snapshot copies. You can even customize your configuration to copy only specific types of snapshots to certain regions.

Screenshot of the field to enable Snapshot Distribution in your backup policy
Copy snapshots to other regions

Restore your cluster faster with optimized, intelligent restores

If you need to restore your cluster, Atlas will intelligently decide whether to use the original snapshot or a copied snapshot for optimal restore speeds. Copied snapshots may be utilized in cases where you are restoring to a cluster in the same region as a snapshot copy, including multi-region clusters if the snapshots are copied to every cluster region. Alternatively, if the original snapshot becomes unavailable due to a regional outage within your cloud provider, Atlas will utilize a copy in the nearest region to enable restores regardless of the cloud region outage.

Screenshot of the point in time restore option in MongoDB Atlas
Perform point in time restore

Get started with Snapshot Distribution

Although storing additional snapshot copies in varying places may not always be required, this can be extremely useful in several situations, such as:

  • For organizations who have a compliance requirement to store backups in different geographical locations from their primary place of operation

  • For organizations operating multi-region clusters looking for faster direct-attach restores for the entire cluster

If you fall into either of these categories, Snapshot Distribution may be a valuable feature addition to your current backup policy, allowing you to automate prior manual processes and free up development time to focus on innovation. Check out the documentation to learn more or navigate to your backup policy to enable this feature.

← Previous

Built With MongoDB: Satori Streamlines Secure Data Access

Handling data imposes contradictory responsibilities upon organizations. On one hand, they need to protect data from unauthorized access. On the other hand, they need to extract value from data; otherwise, why collect it in the first place? The contradiction lies in the fact that to extract value from data, you have to grant access to it, but unregulated access to data can lead to its misuse. Data access service provider Satori enables organizations to accelerate their data use by simplifying and automating access policies while helping to ensure compliance with data security and privacy requirements. In addition to being a member of the MongoDB for Startups program, Satori has just added support for MongoDB workloads, so organizations running MongoDB can now take advantage of Satori's secure data access service. Balancing act Despite the immense volume of sensitive personal, financial, or health-related data within most organizations, managing access to that data is often a manual process handled by a small team struggling with other competing interests. Satori chief scientist Ben Herzberg says this task of managing data access at companies is slowing down innovation. "The majority of organizations are still managing access to data in a manual way," Herzberg says. "Everyone is feeling the bottleneck. The data analyst who wants to do their job in a meaningful way just wants to understand what data sets they can use and get access to it fast." Getting access to data can be an uphill battle, however. "Sometimes you have to go through three or four different teams to get access to data," Herzberg says. "It can take a week or two." Meanwhile, the data engineers who are primarily responsible for managing access to data are getting pulled away from their core responsibilities. "This places the company in an uncomfortable position of having time-intensive processes implemented by teams who would prefer to be working on other tasks," Herzberg says. Simple, fast, secure As a data access service, Satori streamlines access to data, accelerates time-to-value, improves engineering productivity, and reduces complexity and operational risk, all while protecting sensitive data and maintaining compliance with relevant data privacy regulations. The first job of protecting sensitive data is identifying it, but according to Satori's research , few companies have a system in place that continuously monitors for and discovers sensitive data. Organizations that do monitor sensitive data typically do so only quarterly or annually. Herzberg says Satori continuously discovers sensitive data as it's being accessed. "As one of our customers said: I want to remain continuously compliant. I want to know where my sensitive data is at all times. We do that," Herzberg says. Data users can request access to data over Slack, the Satori data portal, and through other integrations to get immediate access to data without any engineering effort, changes to infrastructure, schemas, or tables, or creating objects on the database. "When a lot of people want access to data, you need a simple, fast, and secure way to do it without exposing yourself to risk," Herzberg says. Instead of taking days or weeks to process data access requests, with Satori, it takes just minutes. Build the next big thing with MongoDB Satori chose MongoDB early on because of the inherent flexibility of the document data model. "We chose MongoDB to move quickly and without limitations," Satori software engineering manager Oleg Toubenshlak says. "We didn't know what type of data we would be storing or how we might want to extend objects, so we chose MongoDB because of the flexibility of the data model." "MongoDB is a core component of our infrastructure where we keep customer configurations," Toubenshlak says. "We started with MongoDB deployed on-prem and moved to MongoDB Atlas." Toubenshlak cites continuous backups, easy deployment, and scalability as additional Atlas capabilities he finds valuable. "MongoDB allows us to move fast with development so we can focus on other areas. It's very simple in terms of security and network access. In terms of clients, MongoDB Atlas helps us provide extended capabilities in order to map our Java objects to BSON. It's very compatible and does this very quickly. Once we moved to Atlas, all our problems were solved," he says. Toubenshlak also appreciates the help he received as a member of the startup program. "We had startup credits, and we used professional services to make sure everything was configured properly," he says. "Satori is a small cluster for MongoDB, but I'm very surprised at the time investment we've received." The company is also excited about adding MongoDB Atlas to its list of supported platforms. "Adding MongoDB support is very exciting for us," Herzberg says. "We're already working with some design partners in different industries and helping them with their deployment. It's a meaningful step for us in NoSQL databases. We're seeing a lot of traction with existing customers that want to expand their MongoDB deployments and with new customers." If you're running MongoDB and are interested in simplifying data access, visit Satori and set up a demo or test drive. Are you part of a startup and interested in joining the MongoDB for Startups program? Apply now .

September 28, 2022
Next →

VertexAI and MongoDB for Intelligent Retail Pricing

In today’s competitive retail environment, the ability to quickly adjust pricing in response to market trends, consumer demand, and competitors’ moves is not just an advantage — it's essential for survival. This is where dynamic pricing comes into play, serving as a strategic tool for businesses to pull in their quest for market dominance. Dynamic pricing goes beyond changing numbers; it’s a strategic approach that reflects the dynamic nature of the market, powered by data-driven insights that enable prices to be adjusted in real-time for maximum effectiveness. This shift towards a more agile, data-driven pricing strategy underscores a broader trend in the business world: the recognition of data as a foundational element in decision-making processes. By leveraging real-time data, businesses can ensure their pricing strategies are not only responsive to market fluctuations but also strategically aligned with their overall business objectives, thus driving retail competitiveness to new heights. Let’s uncover how integrating both platforms empowers developers when it comes to delivering best-in-class, data-driven applications. MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More Google Cloud: A platform for real-time analytics and AI Google Cloud stands out as a powerhouse in real-time analytics and artificial intelligence (AI), offering the infrastructure necessary for dynamic pricing strategies and other data-driven business approaches. It's designed to facilitate big data analysis, machine learning, and operational agility. Built-in tools form the backbone of an effective dynamic pricing strategy. These include Vertex AI for advanced machine learning models following best- in- class MLOps practices, and Pub/Sub for real-time messaging to solve real- time data ingestion. By harnessing the power of Google Cloud, retailers can analyze vast quantities of data in real-time, from current market trends to customer behavior and competitor pricing. This enables businesses to make informed decisions swiftly, adjusting their pricing strategies to reflect the ever-changing market conditions. MongoDB: Flexible data modeling and rapid application development MongoDB complements Google Cloud by offering a high performance document- based database with a flexible data model that allows rapid application development. For pricing data in particular, where there may be different variants for different sizes of store or country, the flexibility allows for the ease of storage of complex or hierarchical data. In addition, polymorphic capabilities allow you to use a single interface to represent different types, making your system more flexible. It also supports scalability as new types can be easily integrated. Lastly, it enhances efficiency by allowing the same operation to behave differently based on the object, reducing code redundancy. This flexible schema also enables seamless integration with AI models. MongoDB Atlas supports workload isolation , ensuring dedicated resources for AI tasks and smooth operation alongside core application workloads. Additionally, change streams and triggers can be utilized to capture real-time updates in the pricing data, allowing the AI model to be called upon for immediate analysis and adaptation and enabling in-app analytics for retailers to gain a competitive edge. Figure 1: MongoDB replica set: Workload Isolation In the dynamic pricing reference architecture, Atlas collections function as an ML feature store. By leveraging the capabilities of MongoDB Atlas as a developer data platform, we are able to embed real-time automated decision-making into our e-commerce applications and reduce operational overhead for both business operations and MLOps model fine-tuning. This is achieved through implementing a streamlined approach to data management, incorporating real-time, automated decision making, workload isolation, change streams, triggers for immediate updates, and seamless integration with AI models. Dynamic prising microservice overview Building an event-driven AI architecture leveraging MongoDB Atlas in Google Cloud is straightforward. We can summarize our dynamic pricing microservice by first describing the different components of its architecture, what they are used for, and how they interact with each other: Figure 2: Description of the different technology components of a dynamic pricing microservice and what they are used for. Handling data sources The proposed solution uses Google Cloud Pub/Sub to ingest data sources like customer behavior events in JSON format. Using a technology like Pub/Sub allows for scaling to handle a large number of messages and efficiently distribute them to many subscribers. This is partly because it allows for parallel processing of messages and can be distributed across multiple servers or instances. It is often a fundamental pattern in event-driven architectures, where the flow of the program is determined by events or messages, supporting reactive programming and making the system more responsive and efficient. Data federation We’ll use Vertex AI Notebooks to clean the data and train a TensorFlow model. This model will learn the non-linear relation between customer events, products names, and prices, enabling it to calculate the optimal predicted price. Orchestrating Using Cloud Functions, we orchestrate the customer events coming from the Pub/Sub topic to be converted into tensors, which are then stored in a MongoDB Atlas collection. This collection acts as a feature store serving as a centralized repository designed to store, manage, and serve features for machine learning (ML) models. Features represent individual measurable properties or characteristics used by ML models to make predictions or decisions. MongoDB’s document model flexibility paired with the document versioning pattern will allow us to design time-sensitive chunks of events and granularly manage the training datasets for our models. Serving The Cloud Function will use the event tensor to invoke our trained model that is served in a Vertex AI endpoint. The model will provide a predicted price score that can then be inserted into our product catalog stored in MongoDB so our e-commerce application can read the price change in real time. Dynamic pricing architecture: Putting it all together In the following architecture diagram, the blue data flow illustrates how customer event data is ingested into a Pub/Sub topic. This allows us to make a push subscription to a Cloud Function from the topic. This function orchestrates the data transformation from raw event into a tensor and calls an endpoint to then update the predicted price into our MongoDB product catalog collection. By using this architectural approach, we can isolate raw events threads and build different services around them, reacting in real time for dynamic pricing or asynchronously for model training. With every component loosely coupled, we prevent the system from crashing completely. Moreover, publishers and subscribers can continue to process their logic without the need for the other components to receive or publish messages. Figure 3: Dynamic pricing architecture integrating different Google Cloud components and MongoDB Atlas as a Feature Store For businesses, this translates into more precise and responsive pricing strategies. In the model building and optimization phase, by utilizing TensorFlow within Google Cloud Vertex AI notebooks, retailers can harness the power of deep learning capabilities. The neural network model is capable of analyzing intricate patterns and relationships within large datasets. This is how businesses may capture nuanced market dynamics, customer behavior, and pricing elasticity with greater accuracy, leading to more optimized pricing decisions. But even the best of the models should be consistently optimized. Maintaining model effectiveness requires continuous adaptation. Regularly evaluating accuracy and performing feature engineering ensures your models stay sensitive to market changes. This underscores the importance of retraining as a core principle in a continuous improvement data science approach. Using MongoDB Atlas as your operational data layer means that your feature store is always accessible, reducing downtime and improving the efficiency of machine learning operations. On the other hand, cross-region deployments can bring features closer to where machine learning models are being trained or served, reducing latency and improving model performance. Get started The integration of Google Cloud and MongoDB presents an easy approach to modernizing dynamic pricing strategies. Leveraging real-time analytics, flexible data modeling, and reactive microservices architecture, it empowers businesses to achieve operational efficiencies and gain a competitive advantage in their pricing strategies. For retailers looking to elevate their pricing strategies, considering a strategic partnership with both technologies is essential. For a deeper dive into integrating the different components of this architecture, make sure to check our GitHub repository.

April 17, 2024