MongoDB Updates
The newest releases and freshest updates
Introducing Multi-Cloud Clusters on MongoDB Atlas
October 20, 2020
Updates
1Data - PeerIslands Data Sync Accelerator
Today’s enterprises are in the midst of digital transformation, but they’re hampered by monolithic, on-prem legacy applications that don’t have the speed, agility, and responsiveness required for digital applications. To make the transition, enterprises are migrating to the cloud. MongoDB has partnered with PeerIslands to develop 1Data, a reference architecture and solution accelerator that helps users with their cloud modernization. This post details the challenges enterprises face with legacy systems and walks through how working with 1Data helps organizations expedite cloud adoption. Modernization Trends As legacy systems become unwieldy, enterprises are breaking them down into microservices and adopting cloud native application development. Monolith-to-microservices migration is complex, but provides value across multiple dimensions. These include: Development velocity Scalability Cost-of-change reduction Ability to build multiple microservice databases concurrently One common approach for teams adopting and building out microservices is to use domain driven design to break down the overall business domain into bounded contexts first. They also often use the Strangler Fig pattern to reduce the overall risk, migrate incrementally, and then decommission the monolith once all required functionality is migrated. While most teams find this approach works well for the application code, it’s particularly challenging to break down monolithic databases into databases that meet the specific needs of each microservice. There are several factors to consider during transition: Duration. How long will the transition to microservices take? Data synchronization. How much and what types of data need to be synchronized between monolith and microservice databases? Data translation in a heterogeneous schema environment. How are the same data elements processed and stored differently? Synchronization cadence. How much data needs syncing, and how often (real-time, nightly, etc.)? Data anti-corruption layer. How do you ensure the integrity of transaction data, and prevent the new data from corrupting the old? Simplifying Migration to the Cloud Created by PeerIslands and MongoDB, 1Data helps enterprises address the challenges detailed above. Migrate and synchronize your data with confidence with 1Data Schema migration tool. Convert legacy DB schema and related components automatically to your target MongoDB instance. Use the GUI-based data mapper to track errors. Real-time data sync pipeline. Sync data between monolith and microservice databases nearly in real time with enterprise grade components. Conditional data sync. Define how to slice the data you’re planning to sync. Data cleansing. Translate data as it’s moved. DSLs for data transformation. Apply domain-specific business rules for the MongoDB documents you want to create from your various aggregated source system tables. This layer also acts as an anti-corruption layer. Data auditing. Independently verify data sync between your source and target systems. Go beyond the database. Synchronize data from APIs, Webhooks & Events. Bidirectional data sync. Replicate key microservice database updates back to the monolithic database as needed. Get Started with Real-Time Data Synchronization With the initial version of 1Data, PeerIslands addresses the core functionality of real-time data sync between source and target systems. Here’s a view of the logical architecture: Source System. The source system can be a relational database like Oracle, where we’ll rely on CDC, or other sources like Events, API, or Webhooks. **Data Capture & Streaming.**Captures the required data from the source system and converts them into data streams using either off-the-shelf DB connectors or custom connectors, depending on the source type. 1Data implements data sharding and throttling, which enable data synchronization at scale, in this phase. Data Transformation. The core of the accelerator, when we convert the source data streams into target MongoDB document schemas. We use LISP-based Domain Specific Language to enable simple, rule-based data transformation, including user-defined rules. Data Sink & Streaming. Captures the data streams that need to be updated into the MongoDB database through stream consumers. The actual update into the target DB is done through sink connectors. Target system. The MDB database used by the microservices. Auditing. Most data that gets migrated is enterprise-critical; 1Data audits the entire data synchronization process for missed data and incorrect updates. Two-way sync. The logical architecture enables data synchronization from the MongoDB database back to the source database. We used MongoDB, Confluent Kafka and Debezium to implement this initial version of 1Data: The technical architecture is cloud agnostic, and can be deployed on-prem as well. We’ll be customizing it for key cloud platforms as well as fleshing out specific architectures to adopt for common data sync scenarios. Conclusion The 1Data solution accelerator lends itself to multiple use cases, from single view to legacy modernization. Please reach out to us for technical details and implementation assistance, and watch this space as we develop the 1Data accelerator further.
Announcing Azure Private Link Integration for MongoDB Atlas
We’re excited to announce the general availability of Azure Private Link as a new network access management option in MongoDB Atlas . MongoDB Atlas is built to be secure by default . All dedicated Azure clusters on Atlas are deployed in their own VNET. For network security controls, you already have the options of an IP Access List and VNET Peering . The IP Access List in Atlas offers a straightforward and secure connection mechanism, and all traffic is encrypted with end-to-end TLS. But it requires that you provide static public IPs for your application servers to connect to Atlas, and to list all such IPs in the Access List. And if your applications don’t have static public IPs or if you have strict requirements on outbound database access via public IPs, this won’t work for you. The existing solution to this is VNET Peering, with which you configure a secure peering connection between your Atlas cluster’s VNET and your own VNET(s). This is easy, but the connections are two way. While Atlas never has to initiate connections to your environment, some customers perceive VNET peering as extending the perceived network trust boundary anyway. Although Access Control Lists (ACLs) and security groups can control this access, they require additional configuration. MongoDB Atlas and Azure Private Link Now, you can use Azure Private Link to connect a VNET to MongoDB Atlas. This brings two major advantages: Unidirectional: connections via Private Link use a private IP within the customer’s VNET, and are unidirectional such that the Atlas VNET cannot initiate connections back to the customer's VNET. Hence, there is no extension of the network trust boundary. Transitive: connections to the Private Link private IPs within the customer’s VNET can come transitively from another VNET peered to the Private Link-enabled VNET, or from an on-prem data center connected with ExpressRoute to the Private Link-enabled VNET. This means that customers can connect directly from their on-prem data centers to Atlas without using public IP Access Lists. Azure PrivateLink offers a one-way network peering service between an Azure VNET and a MongoDB Atlas VNET Meeting Security Requirements with Atlas on Azure Azure Private Link adds to the security capabilities that are already available in MongoDB Atlas, like Client Side Field-Level Encryption , database auditing , BYO key encryption with Azure Key Vault integration , federated identity , and more. MongoDB Atlas undergoes independent verification of security and compliance controls , so you can be confident in using Atlas on Azure for your most critical workloads. Ready to try it out? Get started with MongoDB Atlas today! Sign up now
Getting started with MongoDB, PySpark, and Jupyter Notebook
Learn how to leverage MongoDB data in your Jupyter notebooks via the MongoDB Spark Connector and PySpark. We will load financial security data from MongoDB, calculate a moving average, and then update the data in MongoDB with the new data.
Finer Grained Database User Authorization in MongoDB Atlas
We’re happy to announce that it is now possible to create database users with privileges scoped to a specific set of clusters or Atlas Data Lakes in a MongoDB Atlas project. Traditionally, database users have always been created on a project level in Atlas. This provided a centralized interface for database user management as user privileges were scoped to all clusters in your Atlas project. Customers could manage and revoke user permissions confidently, without fear that the updated permissions would be applied to one cluster but not another. However, this abstraction created limitations for use cases that required creating database users scoped to specific clusters or Data Lakes. After much anticipation, we now offer the flexibility to refine database access on a more granular level. Restrict privileges for different environments of the same application One reason for scoping database users to the resource level is to restrict privileges by environment. For example, you may have a dev cluster, a test/qa cluster, and a prod cluster in the same project, but you don’t want all users to have equal access to all three. Isolate teams working on different microservices in the same project Another scenario is to be able to split up users so that certain developers only have access to specific clusters within a project. This is especially useful for customers who have networking restrictions that require them to only use one or two Atlas projects, and therefore co-locate multiple microservices or applications within the same project. With database users per cluster, those customers can now scope team A to only accessing cluster A, and team B to cluster B. Grant other users to access Atlas Data Lakes for analytics Finally, this capability also allows admins to restrict a user’s privileges to only Atlas Data Lakes within the project. This is helpful for analysts, data engineers, and other employees who need access to those Data Lakes and not live operational data in a production cluster, for example. Finer-grained authorization for database users Today, Atlas customers can already use built-in roles or custom roles to grant privileges to database users. With this update, admins get additional flexibility in the database user authorization model while keeping the authentication model exactly the same––we made sure to maintain all of the great features that database users already have. Customers can continue to pick from any of the four authentication mechanisms (SCRAM, X.509, LDAP, AWS IAM) supported by Atlas, and choose to create temporary database users that automatically expire within a user-configurable 7-day period, which can be used in conjunction with database auditing to audit any activity performed by a user with elevated privileges. Database user settings can be modified at any point, so existing users can now be scoped to the cluster or Data Lake-level. Database users with the default authorization settings will continue to have the same access to all resources within an Atlas project. If you have feedback, please add or upvote requests to our feedback portal . Already managing users programmatically? You can create and edit database users with the Atlas API or the Terraform MongoDB Atlas Provider (use this example configuration for a head start!). Ready to try it out? Get started with MongoDB Atlas today! Sign up now