MongoDB Updates
The newest releases and freshest updates
Introducing: Atlas Operator for Kubernetes
The MongoDB Enterprise Operator serves to automate and manage MongoDB clusters on self-managed infrastructure. While this integration has provided complete control over self-managed MongoDB deployments from a single Kubernetes control plane, we’re taking it a step further by extending this functionality to our fully-managed database—MongoDB Atlas. We’re excited to introduce the trial version of the Atlas Operator for Kubernetes. The Atlas Operator will allow you to manage all your MongoDB Atlas clusters without ever having to leave Kubernetes. Keep your workflow as seamless and optimized as possible by managing the lifecycle of your cloud-native applications from where you want most. With the trial version of this Atlas Operator, you can provision and deploy fully-managed MongoDB Atlas clusters on the cloud provider of your choice through Kubernetes. This provider is especially important for those seeking to unlock the power of multi-cloud with unique tools and services native to AWS, Google Cloud, and Azure without any added complexity to the data management experience. With this new Atlas Operator, you get the best of all clouds with multi-cloud clusters on Atlas , coupled with the freedom to run your entire stack anywhere, all while managed in one central location. The “trial version” simply means it has all the core functionality to provision fully-managed Atlas clusters, but the bells and whistles are yet to come. In addition to encapsulating core Atlas functionality, it ensures Kubernetes Secrets are created for each database user which allows for easier management of sensitive data. The Atlas Operator also allows you to create IP Bindings so your applications can securely access clusters. If you’re interested in using the trial version of the Atlas Operator today, follow our quickstart guide below to get started! Quickstart Below you’ll find the steps to create your first cluster in Atlas using the Atlas Operator. Note that you need to have a running Kubernetes cluster before deploying the Atlas Operator. Register/Login to Atlas and create API Keys for your Organization. This information together with the Organization ID will be used to configure the Atlas Operator access to Atlas. Deploy the Atlas Operator kubectl apply -f \ https://raw.githubusercontent.com/mongodb/mongodb-atlas-kubernetes/main/deploy/all-in-one.yaml Create a Secret containing connection information from step one. This Secret will be used by the Atlas Operator to connect to Atlas: kubectl create secret generic mongodb-atlas-operator-api-key \ --from-literal="orgId=<the_atlas_organization_id>" \ --from-literal="publicApiKey=<the_atlas_api_public_key>" \ --from-literal="privateApiKey=<the_atlas_api_private_key>" \ -n mongodb-atlas-system Create AtlasProject Custom Resource: cat <<EOF | kubectl apply -f - apiVersion: atlas.mongodb.com/v1 kind: AtlasProject metadata: name: my-project spec: name: Test Atlas Operator Project projectIpAccessList: - ipAddress: "0.0.0.0/0" comment: "Allowing access to database from everywhere (only for Demo!)" EOF Create AtlasCluster Custom Resource cat <<EOF | kubectl apply -f - apiVersion: atlas.mongodb.com/v1 kind: AtlasCluster metadata: name: my-atlas-cluster spec: name: "Test-cluster" projectRef: name: my-project providerSettings: instanceSizeName: M10 providerName: AWS regionName: US_EAST_1 EOF (You'll have to wait until the cluster is ready - "status" field shows "ready:true":) kubectl get atlasclusters my-atlas-cluster -o=jsonpath='{.status.conditions[?(@.type=="Ready")].status}' True Create a Secret for the password that will be used to log into Atlas Cluster Database kubectl create secret generic the-user-password \ --from-literal="password=P@@sword%" Create AtlasDatabaseUser Custom Resource (references the password Secret) cat <<EOF | kubectl apply -f - apiVersion: atlas.mongodb.com/v1 kind: AtlasDatabaseUser metadata: name: my-database-user spec: roles: - roleName: "readWriteAnyDatabase" databaseName: "admin" projectRef: name: my-project username: theuser passwordSecretRef: name: the-user-password EOF Shortly the Secret will be created by the Atlas Operator containing the data necessary to connect to the Atlas Cluster. You can mount it into your application Pod and read the connection strings from the file or from the environment variable. kubectl get secrets/test-atlas-operator-project-test-cluster-theuser \ -o=jsonpath="{.data.connectionString.standardSrv}} | base64 -d mongodb+srv://theuser:P%40%40sword%25@test-cluster.peqtm.mongodb.net Stay Tuned for More Be on the lookout for updates in future blog posts! The trial version of the MongoDB Atlas Operator is currently available on multiple marketplaces, but we’ll be looking to make enhancements in the near future. For more information, check out our MongoDB Atlas & Kubernetes GitHub page and our documentation .
MongoDB Connector for Apache Kafka 1.5 Available Now
Today, MongoDB has released version 1.5 of the MongoDB Connector for Apache Kafka! This article highlights some of the key features of this new release in addition to continuing to improve the overall quality & stability of the Connector . DeleteOne write model strategy When messages arrive on Kafka topics, the MongoDB Sink Connector reads them and by default will upsert them into the MongoDB cluster specified in the sink configuration. However, what if you didn’t want to always upsert them? This is where write strategies come in and provide you with the flexibility to define what you want to do with the document. While the concept of write strategies is not new to the connector, in this release there is a new write strategy available called DeleteOneBusinessKeyStrategy . This is useful for when a topic contains records identifying data that should be removed from a collection in the MongoDB sink. Consider the following: You run an online store selling fashionable face masks. As part of your architecture, the website sends orders to a Kafka topic, “web-orders” which upon message arrival kicks off a series of actions such as sending an email confirmation, and inserting the order details into an “Orders” collection in a MongoDB cluster. A sample Orders document: { _id: ObjectId("6053684f2fe69a6ad3fed028"), 'customer-id': 123, 'order-id': 100, order: { lineitem: 1, SKU: 'FACE1', quantity: 1 } } This process works great, however, when a customer cancels an order, we need to have another business process to update our inventory, send the cancellation, email and remove the order from our MongoDB sink. In this scenario a cancellation message is sent to another Kafka topic, “canceled-orders”. For messages in this topic, we don’t just want to upsert this into a collection, we want to read the message from the topic and use a field within the document to identify the documents to delete in the sink. For this example, let’s use the order-id key field and define a sink connector using the DeleteOneBusinessKeyStrategy as follows: "connector.class": "com.mongodb.kafka.connect.MongoSinkConnector", "topics":"FaceMaskWeb.OrderCancel", "connection.uri":"mongodb://mdb1", "database":"FaceMaskWeb", "collection":"Orders", "writemodel.strategy": "com.mongodb.kafka.connect.sink.writemodel.strategy.DeleteOneBusinessKeyStrategy", "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy", "document.id.strategy.partial.value.projection.type": "AllowList", "document.id.strategy.partial.value.projection.list": "order-id", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable":false, "document.id.strategy.overwrite.existing": true Now when messages arrive in the “FakeMaskWeb.OrderCancel” topic, the “order-id” field is used to delete documents in the Orders collection. For example, using the sample document above, if we put this value into the OrderCancel topic { “order-id”: 100 } It would cause the document in the Orders collection with order-id and value 100 to be deleted. For a complete list of write model strategies check out the MongoDB Kafka Connector Sink documentation . Qlik Replicate Qlik Replicate is recognized as an industry leader in data replication and ingestion. With this new release of the Connector, you can now replicate and stream heterogeneous data from data sources like Oracle, MySQL, PostGres and others to MongoDB via Kafka and the Qlik Replicate CDC handler . To configure the MongoDB Connector for Apache Kafka to consume Qlik Replicate CDC events, use “com.mongodb.kafka.connect.sink.cdc.qlik.rdbms.RdbmsHandler” as the value for the change data capture handler configuration parameter. The handler supports, insert, refresh, read, update and delete events. Errant Record Reporting Kafka Connect, the service which manages connectors that integrate with a Kafka deployment, has the ability to write records to a dead letter queue (DLQ) topic if those records could not be serialized or deserialized. Starting with Apache Kafka version 2.6, there was added support for error reporting within the sink connectors. This gives sink connectors the ability to send individual records to the DLQ if the connector deems the records to be invalid or problematic. For example, if you are projecting fields in the sink that do not exist in the kafka message or if your sink is expecting a JSON document and the message arrives in a different format. In these cases an error is written to the DLQ versus failing the connector. Various Improvements As with every release of the connector, we are constantly improving the quality and functionality. This release is no different. You’ll also see pipeline errors now showing up in the connect logs, and the sink connector can now be configured to write to the dead letter queue! Next Steps Download the latest MongoDB Connector for Apache Kafka 1.5 from the Confluent Hub ! Read the MongoDB Connector for Apache Kafka documentation . Questions/Need help with the connector? Ask the Community . Have a feature request? Provide Feedback or a file a JIRA .
Global, Multi-Cloud Security at Scale with MongoDB Atlas
In October 2020, we announced the general availability of multi-cloud clusters on MongoDB Atlas . Since then, we’ve made several key improvements that allow customers to take advantage of the full breadth of MongoDB Atlas ’ best-in-class data security and privacy capabilities across clouds on a global scale. Cross-Cloud Security with MongoDB Atlas A common question we get from customers about multi-cloud clusters is how security works. Each cloud provider offers protocols and controls to ensure that data within its ecosystem is securely stored and accessed. But what happens when your data is distributed across different clouds? Don’t worry–we have you covered. MongoDB Atlas is designed to ensure that our built-in best practices are enforced regardless of which cloud providers you choose to use, from dedicated network peering connections to customer-managed keys for data encryption-at-rest and client-side field-level encryption. Private Networking to Multiple Clouds You can now create multiple network peering connections and/or private endpoints for a multi-cloud cluster to access data securely within each cloud provider. For example, say your operational workload runs on Azure, but you want to set up analytics nodes in Google Cloud and AWS so you can compare the performance of Datalab and SageMaker for machine learning. You can set up network peering connections for all three cloud providers in Atlas to allow each of your cloud environments to access cluster data in their respective nodes using private networks. For more details, take a look at our documentation on network peering architecture . Integrate with Cloud KMS for Additional Control Over Encryption Any data stored in Atlas can be encrypted with an external key from AWS KMS, Google Cloud KMS, or Azure Key Vault for an extra layer of encryption on top of MongoDB’s built-in encrypted storage engine . You can also configure client-side field level encryption (client-side FLE) with any of the three cloud key management services to further protect sensitive data by encrypting document fields before it even leaves your application ( support for Azure Key Vault and Google Cloud KMS is available in beta with select drivers ). This means data remains encrypted even while it is in memory and in-use within your live database. Even though the data is encrypted, it remains queryable by the application but is inaccessible to any administrators running the database or underlying cloud infrastructure for you. Beyond security, client-side FLE is also a great way to comply with right to erasure requests that are part of modern privacy regulations such as the GDPR or the CCPA. You simply destroy the user’s encryption key and their PII is unreadable and irrecoverable in memory, on disk, in logs, and in backups. For multi-cloud clusters, this means you can take advantage of multiple layers of encryption that use keys from different clouds. For example, you can have PII data encrypted client-side with AWS KMS keys, then stored in both an AWS and Google Cloud region on Atlas and further encrypted at rest with a key managed via Azure Key Vault. Global, Multi-Cloud Clusters on MongoDB Atlas For workloads that reach users across continents, our customers leverage Global Clusters . This gives you the unique ability to shard clusters across geographic zones and pin documents to a specific zone. Now that Atlas is multi-cloud, you can now choose from the nearly 80 available regions across all three providers, expanding the potential reach of your client applications while making it easy to comply with data residency regulations. Consider a sample scenario where you’re based in the US and want to expand to reach audiences in Europe. To comply with GPDR , you must store EU customer data within that region. With Global Clusters, you can configure a multi-cloud cluster with a US zone and an EU zone. In the US, you choose to run on AWS, but in Europe, you decide to go with Azure because it has more available regions. All of this can be configured in minutes using the Atlas UI: simply define your zones and ensure that your documents contain a location field that dictates which zone they should be stored in. For more details, follow our tutorial for how to configure a multi-cloud Global Cluster on Atlas . Future-Proof Your Applications with Multi-Cloud Clusters There are many reasons why companies are considering a multi-cloud strategy , from cross-cloud resiliency to geographical reach to being able to leverage the latest tools and services on the market. With MongoDB Atlas, you get best-in-class data security and operations and intuitive admin controls, regardless of how many cloud providers you want to use. To learn more about how to deploy a multi-cloud cluster on MongoDB Atlas, check out our step-by-step tutorial , which includes best practices for node distribution, instructions for how to test failing over to another cloud, and more. Safe Harbor The development, release, and timing of any features or functionality described for our products remains at our sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality.
Dive Deeper into Chart Data with New Drill-Down Capability
With the latest release of MongoDB Charts, you’re now able to dive deeper into the data that’s aggregated in your visualizations. At a high level, we generally create charts, graphs and visualizations of our data to answer questions about our business or products. Oftentimes, we need to “double click” on those visualizations to get insight into each individual data point that makes up the line, bar, column, etc. How the drill-down functionality works: Step 1: Right click on the data point you are interested in drilling down into Step 2: Click "show data for this item" Step 3: View the data in tabular or document format Each view can be better for different circumstances. For data without too many fields or no nested arrays, it might be quicker and more easily viewed in a table. On the other hand, the JSON view allows you to explore the structure of documents and click into arrays. Scenarios where more detailed information can help: Data visualization use cases are relatively broad spanning, but oftentimes they fall into 3 main categories: monitoring data, finding insights, and embedding analytics into applications. I’ll be focusing on the first two of these three as there are many different ways you could potentially build drilling-down into data via embedded charts. (Read more about our click events and embedded analytics ). For data or performance monitoring purposes , we're not speaking so much about the performance of your actual database and its underlying infrastructure, but the performance of the application or system built on top of the database. Imagine I have an application or website that takes reviews, if I build a chart like the one below where I want to easily see when an interaction hits a threshold that I want to dive deeper into, I now have the ability to quickly see the document that created that data point. This chart shows app ratings given after a user session in an app. For this example, we want to dive into any rating that was below a 3 (out of 5). This scatter plot shows I have two such ratings that cross that threshold. With the drill-down capability, I can easily see all the details captured in that user session. For finding new insights, let’s imagine I’m tracking how many transactions happen on my ecommerce site over time. In the column chart below, you can see I have purchases by month for the last year and a half (note, there’s a gap because this example is for a seasonal business!). Just by glancing at the chart, I can quickly see purchases have increased over time, and my in-app purchases have increased my overall sales. However, I want to see more about the documents that were aggregated to create those columns, so I can quickly see details about the transaction amount and location without needing to create another chart or dashboard filter. In both examples, I was able to answer a deeper level question that the original chart couldn’t answer on it’s own. We hope this new feature helps you and your stakeholders get more out of MongoDB Charts, regardless if you’re new to it or have been visualizing your Atlas data with it for months, if not years! If you haven’t tried Charts yet, you can get started for free by signing up for a MongoDB Atlas and deploying a free tier cluster.
New MongoDB Shell now supports Client-side Field-level Encryption
Last summer, we introduced mongosh , the new MongoDB Shell with an enhanced user experience and a powerful, Node.js-based scripting environment . Since then, we have been adding new functionality and APIs to close the gap with the legacy mongo shell, on the path to making it the default shell for MongoDB. In addition to the set of CRUD and other commands that we supported in the first release we recently added: Bulk operations Change Streams Sessions and Transactions Logging and profiling commands Replica set and Sharding configuration commands Plus some other minor things and utility commands here and there. This week, we released mongosh 0.8 with support for Client-side Field-level Encryption (FLE). Support for Client-side Field-level Encryption MongoDB Client-Side Field-level Encryption (FLE) allows developers to selectively encrypt individual fields of a document using the MongoDB drivers (and now with mongosh as well) on the client before it is sent to the server. This keeps data encrypted (but still queryable) while it is in-use in database memory, and protects it from the providers hosting the database, as well as from any user that has direct access to the database. Back in November, we announced that in addition to AWS’ KMS, Client-side FLE now supports key management systems in Azure and Google Cloud in beta. The most recent version of the MongoDB Shell makes it easy to test this functionality in a few easy steps: Create a free Atlas Cluster Install mongosh . Check out our documentation to set up your KMS in Azure or GCP. Start encrypting! To make it easier to get started with Client-Side FLE, here are two simple scripts that you can edit and copy-paste into mongosh: mongosh-fle-gcp-kms to set up Client-side FLE with Google Cloud and mongosh-fle-local-kms to use a local key. In the screenshot below, you can see a document that was encrypted on the client with automatic encryption before it was sent across the wire and inserted into MongoDB. Fields are in clear text in the shell but then are shown as encrypted when connecting with Compass to the same Atlas cluster. A Powerful Scripting Environment As mongosh is built on top of Node.js, it’s a great environment for scripting , no matter if it’s about checking the health status of your replica set or if you want to take a quick look at the data to make sure it’s coming in from your application as you are expecting. With modules from npm , the experience becomes much more rich and interactive. For example, if I want to look at the sample_mflix collection available in the Atlas sample datasets and check the distribution of thriller movies over the years, I can put together a simple script that includes running an aggregation and visually formatting the results with an open source library called babar This is just one of many ways you can extend the functionality of the MongoDB Shell by taking advantage of the great ecosystem of JavaScript libraries and modules that the community has built over the years and keeps on building every day. Start Scripting and Let Us Know How it's Working for You! As we added new functionality to the MongoDB Shell, we tried as much as possible to keep backwards compatibility with mongo, and we were mostly able to do that. In a limited number of cases, however, we took the opportunity to clean up the API and address some unexpected behaviors. Wondering what’s coming next in mongosh? We are working on adding support for load() and rc files to make it easy to load your scripts into the shell. If you find something that does not work as expected, please let us know! Simply create a bug in our JIRA project or reach out on Twitter .
MongoDB Connector for Apache Kafka 1.4 Available Now
As businesses continue to embrace event-driven architectures and tackle Big Data opportunities, companies are finding great success integrating Apache Kafka and MongoDB. These two complementary technologies provide the power and flexibility to solve these large scale challenges. Today, MongoDB continues to invest in the MongoDB Connector for Apache Kafka releasing version 1.4! Over the past few months, we’ve been collecting feedback and learning how to best help our customers integrate MongoDB within the Apache Kafka ecosystem. This article highlights some of the key features of this new release. Selective Replication in MongoDB Being able to track just the data that has changed is an important use case in many solutions. Change Data Capture (CDC) has been available on the sink since the original version of the connector. However, up until version 1.4, the source for CDC events could only be sourced from MongoDB via the Debezium MongoDB Connector. WIth the latest release you can specify the MongoDB Change Stream Handler on the sink to read and replay MongoDB events sourced from MongoDB using the MongoDB Connector for Apache Kafka. This feature enables you to record insert, update, and delete activities on a namespace in MongoDB and replay them on a destination MongoDB cluster. In effect you have a lightweight way to perform basic replication of MongoDB data via Kafka. Let’s dive in and see what is happening under the hood. Recall that when the connector is used as a source to MongoDB, it starts a change stream on a specific namespace. Depending on how you configure the source connector, documents are written into a Kafka topic based on this namespace and pipeline that match your criteria. These documents are by default in the change stream event format . Here is a partial message in the Kafka topic that was generated from the following statement: db.Source.insert({proclaim: "Hello World!"}); { "schema": { "type": "string", "optional": false }, "payload": { "_id": { "_data": "82600B38...." }, "operationType": "insert", "clusterTime": { "$timestamp": { "t": 1611348141, "i": 2 } }, "fullDocument": { "_id": { "$oid": "600b38ad6011ef6265c3acd1" }, "proclaim": "Hello World!" }, "ns": { "db": "Tutorial3", "coll": "Source" }, "documentKey": { "_id": { "$oid": "600b38ad6011ef6265c3acd1" } } } } Now that our change stream message is in the Kafka topic, we can use the connector as a sink to read the stream of messages and replay them at the destination cluster. To set up the sink to consume these events, set the “change.data.capture.handler" to the new com.mongodb.kafka.connect.sink.cdc.mongodb.ChangeStreamHandler property. Notice that one of the fields is “operationType”. The sink connector will only support insert, update and delete operations on the namespace and does not support actions like creation of database objects such as users, namespaces, indexes, views, and other metadata that occurs in more traditional replication solutions. In addition this capability is not intended as a replacement for a full featured replication system as it can not guarantee transactional consistency between the two clusters. That said, if all you are looking to do is move data and can accept its lack of consistency then you have a simple solution using the new ChangeStreamHandler. To work through a tutorial on this new feature, check out Tutorial 3 of the MongoDB Connector for Apache Kafka Tutorials in GitHub . Dynamic Namespace Mapping When we use the MongoDB connector as a sink we take data that resides on a Kafka Topic and insert it into a collection. Prior to 1.4, once this mapping is defined it isn’t possible to route topic data to another collection. In this release we added the ability to dynamically map a namespace to the contents of the kafka topic message. For example, consider a Kafka Topic “Customers.Orders” that contains the following messages: {"orderid":1,"country":"ES"} {"orderid":2,"country":"US"} We would like these messages to be placed in their own collection based upon the country value. Thus, the message with the field “orderid” that has a value of 1 will be copied in a collection called, “ES”. Likewise, the message with the field “orderid” that has a value of 2 will be copied to a collection called, “US”. To see how we configure this scenario, we will define a sink using the new namespace.mapper property configured with a value of “ com.mongodb.kafka.connect.sink.namespace.mapping.FieldPathNamespaceMapper ”. Using this mapper, we can use a key or value field to determine the database and collection respectively. In our example above let’s define our config using the value of the country field as the collection name to sink to: '{"name": "mongo-dynamic-sink", "config": { "connector.class":"com.mongodb.kafka.connect.MongoSinkConnector", "topics":"Customers.Orders", "connection.uri":"mongodb://mongo1:27017,mongo2:27017,mongo3:27017", "database":"Orders", "collection":"Other" "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable":"false", "namespace.mapper":"com.mongodb.kafka.connect.sink.namespace.mapping.FieldPathNamespaceMapper", "namespace.mapper.value.collection.field":"country" }} Messages that do not have a country value will by default be written to the namespace defined in the configuration just like they would have been without the mapping. However, If you want messages that do not conform to the map to generate an error simply set the property namespace.mapper.error.if.invalid to true. This will raise an error and stop the connector when messages can not be mapped to a namespace due to missing fields or fields that are not strings. If you’d like to have more control over the namespace you can use the new “getNamespace” method of the interface com.mongodb.kafka.connect.sink.namespace.mapping.NamespaceMapper . Implementations of this method can implement more complex business rules and can access the SinkRecord or SinkDocument as part of the logic to determine the destination namespace. Dynamic Topic Mapping Once the source connector is configured, change stream events flow from the namespace defined in the connector to a Kafka Topic. The name of the Kafka Topic is made up of three configuration parameters: topic.prefix, database and collection. For example, if you had as part of your source connector configuration: “topic.prefix”:”Stocks”, “database”:”Customers”, “collection”:”Orders” The Kafka topic that would be created would be “Stocks.Customers.Orders”. However, what if you didn’t always want the events in the Orders collection to always go to this specific topic? What if you wanted to determine at run-time which topic a specific message should be routed to? In 1.4 you can now specify a namespace map that defines which kafka topic a namespace should be written to. For example, consider the following map: {"Customers": "CustomerTopic", "Customers.Orders": "Orders"} This will map all change stream documents from the Customers database to CustomerTopic.<collectionName> apart from any documents from the Customers.Orders namespace which map to the Orders topic. If you need to use complex business logic to determine the route, you can implement the getTopic method in the new TopicMapper class to handle this mapping logic. Also note that 1.4 introduced a topic.suffix configuration property in addition to the topic.prefix. Using our example above, you can configure “topic.prefix”:”Stocks”, “database”:”Customers”, “collection”:”Orders”, topics.suffix:”US” This will define the topic to write to as “Stocks.Customers.Orders.US” Next Steps Download the latest MongoDB Connector for Apache Kafka 1.4 from the Confluent Hub ! Read the MongoDB Connector for Apache Kafka documentation Questions/Need help with the connector? Ask the Community Have a feature request? Provide Feedback or a file a JIRA
MongoDB Atlas Online Archive for Data Tiering is now GA
We’re thrilled to announce that MongoDB Atlas Online Archive is now Generally Available. Online Archive allows you to seamlessly tier your data across Atlas clusters and fully managed cloud object stores, while retaining the ability to query it through a single endpoint. Reduce storage costs. Set the perfect price to performance ratio on your data. Automate data tiering. Eliminate the need to manually migrate or delete valuable data. Queryable archives. Easily federate queries across live and archival data using a unified connection string. With Online Archive, you can bring new use cases to MongoDB Atlas that were previously cost-prohibitive such as high volume time-series workloads, data archival for auditing purposes, historical log keeping and more. Manage your entire data lifecycle on MongoDB Atlas without replicating or migrating it across multiple systems. What is Atlas Online Archive? Online Archive is a fully managed data tiering solution that allows you to tier data across your "hot" database storage layer and "colder" cloud object storage to maintain queryability while optimizing on cost and performance. Online Archive is a good fit for many different use cases, including: Insert heavy workloads, where data is immutable and has lower performance requirements as it ages Historical log keeping and time-series datasets Storing valuable data that would have otherwise been deleted using TTL indexes We’ve received amazing feedback from the community over the past few months while the feature was in beta and we’re now confident in supporting your production workloads. Our users have put the feature through a variety of use cases in production and development workloads which has enabled us to make a wide range of improvements. Online Archive gives me the flexibility to store all of my data without incurring high costs, and feel safe that I won't lose it. It's the perfect solution. Ran Landau, CTO, Splitit Autonomous Archival Management It's easy to get started with Online Archive and it requires no ongoing maintenance once it’s been set up. In order to activate the feature, you can follow these simple steps: Navigate to the “Online Archive” tab on your cluster card and begin the setup flow. Set an archiving rule by selecting a date field, with dot-notation if it’s nested, or creating a custom filter. Choose commonly queried fields that you want your archival queries to be optimized for, with a few things in mind: Your data will always be “partitioned” by the date field in your archive, but can be partitioned by up to two additional fields as well. The fields that you most commonly query should be towards the top of the list (date can be moved to the top or bottom). Query fields should be chosen carefully as they cannot be changed after the fact and will have a large impact on query performance. Avoid choosing a field that has unique values as it will have negative performance impacts for queries that need to scan lots of data. And you’re done! MongoDB Atlas will automatically move data off of your cluster and into a more cost-effective storage layer that can still be queried with a single connection string that combines cluster and archive data, powered by Atlas Data Lake . What's Next? Along with announcing Online Archive as Generally Available, we’re excited to share a few additional product enhancements which should be available in the coming months: Private Link support when querying your archive Incremental deletes of data from your archive Support for BYO key encryption on your archival data Improved performance and stability Try Atlas Online Archive Online Archive allows you to right-size your Atlas clusters by storing hot data that is regularly accessed in live storage and moving colder data to a cheaper storage tier. Billing for this feature will include the cost to store data in our fully managed cloud object storage and usage based pricing for querying archive data. We can’t wait to see what new workloads you’ll bring onto MongoDB Atlas with the new flexibility provided by Online Archive! To get started, sign up for an Atlas account and deploy any dedicated cluster (M10 or higher). Have questions? Check out the documentation or head over to our community forums to get answers from fellow developers. And if we’re missing a feature you’d like to see, please let us know ! Safe Harbor Statement The development, release, and timing of any features or functionality described for MongoDB products remains at MongoDB's sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality. Except as required by law, we undertake no obligation to update any forward-looking statements to reflect events or circumstances after the date of such statements.
New Ways to Customize Your Charts
When it comes to building charts, we know that details matter. Small differences in layout, styling or composition can make a big difference in how well your chart communicates the story behind your data. That’s why we’ve just released a whole bunch of new capabilities in MongoDB Charts , giving you more control than ever. Here’s what’s new: Secondary Y Axis: Charts can be a great way to show correlation between two different datasets, but when their scales differ greatly it can be hard to see the correlation. By choosing to plot one more series on a secondary Y Axis, you can allow them to make the most of the available space and highlight any interesting relationships. Secondary Y Axis can be enabled on Grouped Column, Discrete Line, Continuous Line and Continuous Area charts. Legend Position: Chart legends can now be moved to the top, right or bottom of your chart, or hidden altogether. “All Others” Group: Charts has long allowed you to limit a chart to show, say, just the top 10 values. The new “All Others” option allows you to add an additional bar or donut segment that shows the value of all other categories not included in the limit. “Count by Value” aggregation: Building multi-series charts is now easier than ever, with the new “Count by Value” aggregation option. This will automatically create series from each distinct value found in a field. String binning with Regular Expressions: Last month we introduced binning of string values, allowing you to choose the exact values to go into each bin. This month we’ve extended this further by allowing you to use Regular Expressions to assign values to a bin based on powerful patterns. Scatter Mark formatting: We’ve ramped up the customization options available on Scatter charts, allowing you to control the size, border thickness and opacity of each plotted mark. Line Dash Styles: A new option on Discrete and Continuous Line charts results in a different dash style for each series, making it easier to differentiate the series and improve the accessibility of your charts. Here’s one example of a chart that shows off the secondary Y axis, custom legend position and line dash styles: And here’s another, showing the effect you can get by customizing your scatter chart’s mark style: We hope you enjoy these new charting capabilities, but we’re not done yet! Over the next couple of months, we’ll be moving our focus to Table charts, adding options like conditional formatting, text wrapping and column pinning. If you have any other ideas for new customization features, please let us know using the MongoDB Feedback Engine . If you haven’t tried Charts yet, you can get started for free by signing up for MongoDB Atlas and deploying a free tier cluster.
Client-Side Field Level Encryption is now on Azure and Google Cloud
We’re excited to announce expanded key management support for Client-Side Field Level Encryption (FLE). Initially released last year with Amazon’s Key Management Service (KMS), native support for Azure Key Vault and Google Cloud KMS is now available in beta with support for our C#/.Net, Java, and Python drivers. More drivers will be added in the coming months. Client-Side FLE provides amongst the strongest levels of data privacy available today. By expanding our native KMS support, it is even easier for organizations to further enhance the privacy and security of sensitive and regulated workloads with multi-cloud support across ~80 geographic regions. My databases are already encrypted. What can I do with Client-Side Field Level Encryption? What makes Client-Side FLE different from other database encryption approaches is that the process is totally separated from the database server. Encryption and decryption is instead handled exclusively within the MongoDB drivers in the client, before sensitive data leaves the application and hits the network. As a result, all encrypted fields sent to the MongoDB server – whether they are resident in memory, in system logs, at-rest in storage, and in backups – are rendered as ciphertext. Neither the server nor any administrators managing the database or cloud infrastructure staff have access to the encryption keys. Unless the attacker has a compromised DBA password, privileged network access, AND a stolen client encryption key, the data remains protected, securing it against sophisticated exploits. MongoDB’s Client-Side FLE complements existing network and storage encryption to protect the most highly classified, sensitive fields of your records without: Developers needing to write additional, highly complex encryption logic application-side Compromising your ability to query encrypted data Significantly impacting database performance By securing data with Client-Side FLE you can move to managed services in the cloud with greater confidence. This is because the database only works with encrypted fields, and you control the encryption keys, rather than having the database provider manage the keys for you. This additional layer of security enforces an even finer-grained separation of duties between those who use the database and those who administer and manage the database. You can also more easily comply with “right to erasure” mandates in modern privacy legislation such as the GDPR and the CCPA . When a user invokes their right to erasure, you simply destroy the associated field encryption key and the user’s Personally Identifiable Information (PII) is rendered unreadable and irrecoverable to anyone. Client-Side FLE Implementation Client-Side FLE is highly flexible. You can selectively encrypt individual fields within a document, multiple fields within the document, or the entire document. Each field can be optionally secured with its own key and decrypted seamlessly on the client. To check-out how Client-Side FLE works, take a look at this handy animation. Client-Side FLE uses standard NIST FIPS-certified encryption primitives including AES at the 256-bit security level, in authenticated CBC mode: AEAD AES-256-CBC encryption algorithm with HMAC-SHA-512 MAC. Data encryption keys are protected by strong symmetric encryption with standard wrapping Key Encryption Keys, which can be natively integrated with external key management services backed by FIPS 140-2 validated Hardware Security Modules (HSMs). Initially this was with Amazon’s KMS, and now with Azure Key Vault and Google Cloud KMS in beta. Alternatively, you can use remote secure web services to consume an external key or a secrets manager such as Hashicorp Vault. Getting Started To learn more, download our Guide to Client-Side FLE . The Guide will provide you an overview of how Client-Side FLE is implemented, use-cases for it, and how it complements existing encryption mechanisms to protect your most sensitive data. Review the Client-Side FLE key management documentation for more details on how to configure your chosen KMS. Safe Harbor The development, release, and timing of any features or functionality described for our products remains at our sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality.