MongoDB Updates

The newest releases and freshest updates

Introducing the MongoDB Atlas Data API, Now Available in Preview

As the leading application data platform , we’re hyper-focused on accelerating and simplifying how developers leverage their application data. This has led to the introduction of features like serverless instances and the Atlas Triggers that minimize the operational burden associated with traditional database workloads. Today, we’re excited to announce the next step forward in this mission with the introduction of the MongoDB Atlas Data API – a fully managed, REST-like API for accessing your Atlas data. The Data API makes it easy to perform CRUD and aggregations on your data in minutes and allows you to query MongoDB from your backend in any language, without the need for drivers. The next level of data access Organizations are increasingly relying on operational data layers to build distributed architectures like microservices for their modern applications to speed-up development and stay competitive in rapidly changing markets. These stacks often require scalable, highly available, and secure access to the data layer. The most popular way to architect these data services is to build APIs that communicate with MongoDB data over HTTPS using REST or similar protocols. However, creating a custom-built API typically takes a lot of time and effort. It's a painful process that introduces unnecessary operational burdens like provisioning additional servers, connection management, and scaling. With the Atlas Data API, customers can generate a fully managed, REST-like API for their Atlas data in seconds. Developers no longer need to worry about the underlying infrastructuring of their APIs, and instead can enjoy the efficiency of intuitive, out-of-the box data access, while still being able to leverage the always-on and highly available qualities of Atlas as the underlying database. This unlocks a whole new level of developer productivity for use cases that were previously time consuming to accomplish – such as building data-centric microservices, simplifying access from serverless functions, and integrating with third party services . The API even has built-in support for aggregation pipelines to use with services like Atlas Search . Try the Atlas Data API All customers now have the ability to enable the Data API for their Atlas deployment. We invite you to try it out today with a new or existing Atlas account. It’s incredibly easy to get started: simply choose the cluster you’d like to connect to and generate an API key. That’s all it takes to set up and start accessing Atlas data. Have questions? Check out our documentation or head over to our community forums to get answers from fellow developers. What's next for the Atlas Data API This preview release is just the beginning. Support for services like Data Lake and Serverless Instances will be added over the coming months. And, long term, we see the Data API as the next step in our journey to abstract and automate infrastructure decisions – to help developers build the future faster. Atlas Data API documentation can be found here

November 18, 2021
Updates

Announcing Google Private Service Connect (PSC) Integration for MongoDB Atlas

We’re excited to announce the general availability of Google Cloud Private Service Connect (PSC) as a new network access management option in MongoDB Atlas . Announced alongside the availability of MongoDB 5.1 , Google Cloud PSC is GA for use with Altas. See the documentation for instructions on setting up Google Cloud PSC for Atlas, or read on for more information. MongoDB Atlas is secure by default . All dedicated Google Cloud clusters on Atlas are deployed in their own VPC. To set up network security controls, Atlas customers already have the options of an IP Access List and VPC Peering . The IP Access List in Atlas is a straightforward and secure connection mechanism, and all traffic is encrypted with end-to-end TLS. But you must be able to provide static public IPs for your application servers to connect to Atlas, and to list those IPs in the Access List. If your applications don’t have static public IPs or if you have strict requirements on outbound database access via public IPs, this won’t work for you. The existing solution to this is VPC Peering, which allows you to configure a secure peering connection between your Atlas cluster’s VPC and your own Google Cloud VPC(s). This is easy, but the connections are two way. Atlas never has to initiate connections to your environment, but some Atlas users don’t want to use VPC peering because it extends the perceived network trust boundary. Access Control Lists (ACLs) and IAM Groups can control this access, but they require additional configuration. MongoDB Atlas and Google Cloud PSC Now, you can use Google Cloud Private Service Connect to connect a VPC to MongoDB Atlas. Private Service Connect allows you to create private and secure connections from your Google Cloud networks to MongoDB Atlas. It creates service endpoints in your VPCs that provide private connectivity and policy enforcement, allowing you to easily control network security in one place. This brings two major advantages: Unidirectional: connections via PSC use a private IP within the customer’s VPC, and are unidirectional. Atlas cannot initiate connections back to the customer's VPC. This means that there is no extension of the perceived network trust boundary. Transitive: connections to the PSC private IPs within the customer’s VPC can come transitively from an on-prem data center connected to the PSC-enabled VPC with Cloud VPN . Customers can connect directly from their on-prem data centers to Atlas without using public IP Access Lists. Google Cloud Private Service Connect offers a one-way network peering service between a Google Cloud VPC and a MongoDB Atlas VPC Meeting security requirements with Atlas on Google Cloud Google Cloud PSC adds to the security capabilities that are already available in MongoDB Atlas, like Client Side Field-Level Encryption , database auditing , BYO key encryption with Google Cloud KMS integration , federated identity , and more. MongoDB Atlas undergoes independent verification of security and compliance controls , so you can be confident in using Atlas on Google Cloud for your most critical workloads. To learn more about configuring Google PSC with MongoDB Atlas, visit our docs . If you’re already managing your Atlas clusters with our API, you can add a private endpoint with the documentation here . For more information about Google Cloud Private Service Connect, visit the Google Cloud docs or read the Introducing Private Service Connect release announcement. Try MongoDB Atlas for free today!

November 11, 2021
Updates

Introducing the MongoDB 5.1 Rapid Release

Arriving just a few months after the General Availability of 5.0, MongoDB 5.1 is our first Rapid Release which brings more native time series enhancements, richer analytics, new security options, and overall improvements to platform resilience and developer productivity. Launching alongside MongoDB 5.1 are new capabilities in Atlas Search which will make it easier for users to build fast and rich application search experiences. MongoDB 5.1 marks our accelerated release cadence designed to get new database features and improvements into your hands faster than ever before. MongoDB 5.1 and all future rapid releases will be fully supported on MongoDB Atlas and are available as development releases from our Download Center. Native Time Series Enhancements With optimized time series collections, clustered indexes, and window functions, MongoDB 5.0 made it faster, easier, and lower cost to serve the industry’s fastest growing, data intensive use cases such as IoT platforms and real-time financial analytics. Now with MongoDB 5.1, you can globally distribute your time series applications and further simplify their development: More developer velocity Time series collections can now take advantage of MongoDB’s native sharding to horizontally distribute massive data sets and co-locate nodes with data producers to support local write operations and to enforce the data sovereignty controls. It is common for time series data to be uneven, for example a sensor goes offline and several readings are missed. But in order to perform analytics and ensure correctness of results data needs to be continuous. With densification you can now handle missing data better and build time series apps and analytics faster putting less burden on the developer. Time series collections now also support delete operations . While most time series applications are append-only, users need to be able to invoke their right to erasure so we are giving developers an easy way to comply with modern data privacy regulations. Complete data lifecycle From medical sensors to market data fluctuations, time series means hundreds of millions data points per day. You need to process these massive volumes fast, distill valuable insights then continue to retain the full data set for regulatory purposes - possibly for years - all without incurring skyrocketing costs and data movement complexity. With Atlas Online Archive support for time series, now available in preview, you can do exactly that and seamlessly and economically manage your entire time series data lifecycle. Simply define your own archiving policy, and Atlas handles all data movement for you by tiering aged time series data out of your database into lower cost, fully managed cloud object storage. Rather than delete anything, you can retain all your time series data, preserving the ability to query it at any time alongside your live data for long term trend analytics and machine learning, or for compliance purposes. Support for online archiving is available for MongoDB 5.0 and above. Broader platform support for Time Series Data Our native time series capabilities are supported across the entire MongoDB application data platform making it easy to work with time series data in any context. You can now create time series collections directly from Atlas Data Explorer, MongoDB Compass or MongoDB for VS Code. With support for date binning, date filtering options, and value comparison, Atlas Charts lets you create graphs and dashboards from any Atlas times series collection, easily share insights, and embed visualizations into your applications for a rich user experience. Richer and More Flexible Analytics and Full-Text Search Many developers start out with MongoDB for their operational use cases, and then expand to leverage our platform's versatility in powering analytics and search as well. MongoDB 5.1 includes new features and enhancements that make it easier to unlock insights from your data and improve user experience. Cross-shard joins and graph traversals For most transactional and operational workloads, the document data model largely eliminates the need to join data from different collections. This is because related data can be embedded in sub-documents and arrays within a single, richly structured document – following the principle that what is accessed together is often best stored together. However analytical applications can sometimes require joins to be executed – for example bringing together customers and orders from separate collections. Through the $lookup aggregation pipeline stage, you can have the database join collections for you. The $graphLookup stage gives you the ability to traverse related data, performing “friend-of-friend” type queries to uncover patterns and surface previously unidentified connections in your data. In MongoDB 5.1 we now allow you to use $lookup and $graphLookup to combine and analyze data that is distributed across shards which was not previously possible. Our design gives you even more precision in your code by enabling you to target individual shards as needed. However you don’t need to understand sharding or even know your collection is sharded to run these queries as there is no new syntax for developers to learn. Materializing results for operational analytics The $merge and $out aggregation stages can be used to write the results of an aggregation pipeline in order to create a new collection or create/update an on-demand materialized view . These stages enable users to reduce processing overhead by reading pre-computed results instead of re-running the aggregation each time, and by writing only incremental results when the aggregation results change. Users often want to run resource-intensive analytical queries on secondary nodes in order to avoid performance impacts on the primary — but since only primaries can serve writes, aggregations including $out or $merge could not previously run on a secondary node. Soon, such pipelines will run, performing their query execution work on a secondary node, then automatically directing any writes to the primary. This allows you to offload computationally expensive analytics work to secondary nodes while still being able to materialize the results of that work. This will be accessible via drivers in their upcoming releases. Full-Text Search Facets: now in public preview Faceted search allows users to filter and quickly navigate search results by categories and see the total number of results per category for at-a-glance statistics. With our new facet operator , facet and count operations are pushed down into Atlas Search’s embedded Lucene index and processed locally, taking advantage of 20+ years of Lucene optimizations. This makes workloads such as ecommerce product catalogs, content libraries, and counts run up to 100x faster . Learn more from our Atlas Search facets blog post . New and Enhanced Security Options End-to-end encryption for confidential computing Extending beyond cloud provider Key Management Services (KMS), MongoDB’s unique Client-Side Field Level Encryption will support any KMIP-compliant KMS . This functionality is being released in new versions of drivers that will be available soon. Client-Side FLE delivers some of the strongest privacy and security controls available anywhere today. By using the MongoDB drivers to encrypt the most sensitive fields in your documents before they leave the application you can do three things that are not possible with in-flight or at-rest encryption alone: Protect data while it is in-use, in the memory of your active database instance. The database never sees plaintext, but data remains queryable. Make data unreadable to anyone running the database for you, or who has access to the underlying database infrastructure — this includes MongoDB SREs running the Atlas services as well as cloud provider personnel. Simplify the process of enforcing right to erasure (sometimes called right to be forgotten) mandates in modern privacy regulations such as the GDPR or the CCPA. This is because you simply destroy the key encrypting a user’s PII, and their data is rendered unreadable and unrecoverable — in-memory, at-rest, in backups, and in logs. Google Cloud Private Service Connect We’ve also added a new network security option to MongoDB Atlas with the availability of Google Private Service Connect (PSC). Private Service Connect allows you to create private and secure connections from your Google Cloud networks to MongoDB Atlas. It creates service endpoints in your VPCs that provide private connectivity and policy enforcement, allowing you to easily control network security in one place. Along with VPC Peering, Google Cloud PSC makes it easy to connect your applications and services in Google Cloud to Atlas. Platform Resilience MongoDB 5.1 continues to build out controls for reliability and availability with the following enhancements: We've made a number of changes to WiredTiger internals that improve backups, including minimizing the checkpoints pinned while a backup cursor is open and improving handling of backup cursors that are open for long periods. These improvements will reduce both the operational overhead and storage consumption on the replica node from which the backup is taken. This improvement is available for backups taken from MongoDB Atlas and from self-hosted deployments controlled by Ops Manager or Cloud Manager, and has been backported to MongoDB 4.2 and above. In addition to enhancements affecting backups, WiredTiger checkpointing and locking have been improved to enhance performance when MongoDB is managing many concurrently active collections in a single instance. This is especially useful to multi-tenant applications built on MongoDB. We'll also be adding improvements in upcoming versions of our drivers that support mongos controls to mitigate connection storms in sharded clusters, especially during failover events. These include preferentially connecting to nodes that have existing idle connections that can be reused, improving the matching of connection pool sizing across replica set members, limiting the rate of new connections, and adding a mechanism to limit the number of mongos servers used when connecting to sharded clusters via SRV records. Improved Productivity for C# Developers Making it easier for developers to query and manipulate data is at the core of our mission as the modern application data platform. For C# developers the LINQ API serves as the main gateway between the language and database. In MongoDB 5.1 we are improving developer productivity for our C# community with a completely redesigned LINQ interface that lets developers write all of their MongoDB queries as well as build sophisticated aggregation pipelines natively in C#. Getting Started with MongoDB 5.1 You can learn more about all of the new features and enhancements in MongoDB 5.0 and 5.1 from our Guide to What’s New . MongoDB 5.1 is available now. If you are running Atlas Serverless instances or have opted in to receive Rapid Releases in your dedicated Atlas cluster, then your deployment will be automatically updated to 5.1 starting today. For a short period after upgrade, the Feature Compatibility Version (FCV) will be set to 5.0; certain 5.1 features will not be available until we increment the FCV. MongoDB 5.1 is also available as a Development Release for evaluation purposes only from the MongoDB Download Center. Consistent with our new release cadence announced last year, the functionality available in 5.1 and the subsequent Rapid Releases will all roll up into MongoDB 6.0, our next Major Release scheduled for delivery in 2022. I really look forward to hearing what you think about MongoDB 5.1, and can’t wait to tell you what’s new in the 5.2 Rapid Release scheduled for next quarter. Safe Harbour Statement The development, release, and timing of any features or functionality described for our products remains at our sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality.

November 9, 2021
Updates

MongoDB Atlas for Government Achieves "FedRAMP In-process"

We are pleased to announce that MongoDB Atlas for Government has achieved the FedRAMP designation of “ In-process ”. This status reflects MongoDB’s continued progress toward a FedRAMP Authorized modern data platform for the US Government. Earlier this year, MongoDB Atlas for Government achieved the designation of FedRAMP Ready . MongoDB is widely used across the Federal Government, including the Department of Veterans Affairs, the Department of Health & Human Services (HHS), the General Services Administration, and others. HHS is also sponsoring the FedRAMP authorization process for MongoDB. What is MongoDB Atlas for Government? MongoDB Atlas for Government is an independent environment of our flagship cloud product MongoDB Atlas. Atlas for Government has been built for US government needs. It allows federal, state, and local governments as well as educational institutions to build and iterate faster using a modern database-as-a-service platform. The service is available in AWS GovCloud (US) and AWS US East/West regions. MongoDB Atlas for Government Highlights: Atlas for Government clusters can be created in AWS GovCloud East/West or AWS East/West regions. Atlas for Government clusters can span regions within AWS GovCloud or within AWS. Atlas core features such as automated backups, AWS PrivateLink, AWS KMS, federated authentication, Atlas Search, and more are fully supported Applications can use client-side field level encryption with AWS KMS in GovCloud or AWS East/West. Getting started and pricing MongoDB Atlas for Government is available to Government customers or companies that sell to the US Government. You can buy Atlas for Government through AWS GovCloud or the AWS marketplace . Please fill out this form and a representative will get in touch with you. To learn more about Atlas for Government, visit the product page , check out the documentation , or read the FedRAMP FAQ .

September 22, 2021
Updates

Serverless Instances Now Offer Extended Regional and Cloud Provider Support

Today’s applications are expected to just work, regardless of time of day, user traffic, or where in the world they are being accessed from. But in order to achieve this level of performance and scale, developers have to meticulously plan for infrastructure needs, sometimes before they even know what the success of their application may be. In many cases, this is not feasible and can lead to over provisioning and over paying. But what if you could forgo all of this planning and the database would seamlessly scale for you? Well, now you can - with serverless instances on MongoDB Atlas. Since we announced serverless instances in preview at MongoDB.live we have been actively working toward implementing new functionality to make them more robust and widely available. With our most recent release, serverless instances now offer expanded cloud providers and regions, and support MongoDB tools. Deploy a serverless instance on the cloud provider of your choice With our dedicated clusters on MongoDB Atlas, you have the flexibility to run anywhere with global reach on the cloud provider of your choice, so you can deliver responsive and reliable applications wherever your users are located. Our goal is to provide this same flexibility for serverless instances. We’re happy to announce that you can now deploy a serverless instance in ten regions on AWS, Google Cloud, and Azure. You’ll see when deploying a serverless instance there are now more regions supported on AWS, as well as two available regions on both Google Cloud and Azure - so you can get started with the cloud provider that best suits your needs or the region that’s closest to you. We will be continuing to add new regions over time to ensure coverage where you need it most. Easily import your data with MongoDB tools With this release, we have also made it easier to work with your data. You can now easily import data from an existing MongoDB deployment using the MongoDB Tools including mongodump, mongorestore, mongoexport , and mongoimport . In order to use MongoDB tools with serverless instances, you will need to be using the latest version . If you have additional feature requests that would make your developer experience better, share them with us in our feedback forums . Database deployment made simple With serverless instances, you can get started with almost no configuration needed - MongoDB Atlas will automatically scale to meet your workload needs, whether you have variable traffic patterns or you’re looking for a sandbox database for your weekend hobby project. If you haven’t yet given serverless instances a try, now is a great time to see what they can offer. If you have feedback or questions, we’d love to hear them! Join our community forums to meet other MongoDB developers and see what they’re building with serverless instances. Create your own serverless instance on MongoDB Atlas. Try the Preview .

September 16, 2021
Updates

Highlight What Matters with the MongoDB Charts SDK

We're proud to announce that with the latest release of the MongoDB Charts SDK you can now apply highlights to your charts. These allow you to emphasize and deemphasize your charts with our MongoDB query operators . Build a richer interactive experience for your customers by highlighting with the MongoDB Charts embedding SDK . By default, MongoDB Charts allows for emphasizing parts of your charts by series when you click within a legend. With the new highlight capability in the Charts Embedding SDK, we put you in control of when this highlighting should occur, and what it applies to. Why would you want to apply highlights? Highlighting opens up the opportunity for new experiences for your users. The two main reasons why you may want to highlight are: To show user interactions: We use this in the click handler sandbox to make it obvious what the user has clicked on. You could also use this to show documents affected by a query for a control panel. Attract the user’s attention: If there's a part of the chart you want your users to focus on, such as the profit for the current quarter or the table rows of unfilled orders. Getting started With the release of the Embedding SDK , we've added the setHighlight method to the chart object, which uses MQL queries to decide what gets highlighted. This lets you attract attention to marks in a bar chart, lines in a line chart, or rows in a table. Most of our chart types are already supported, and more will be supported as time goes on. If you want to dive into the deep end, we've added a new highlighting example and updated the click event examples to use the new highlighting API: Highlighting sandbox Click events sandbox Click events with filtering sandbox The anatomy of a click In MongoDB Charts, each click produces a wealth of information that you can then use in your applications , as seen below: In particular, we generate an MQL expression that you can use called selectionFilter , which represents the mark selected. Note that this filter uses the field names in your documents, not the channel names. Before, you could use this to filter your charts with setFilter , but now you can use the same filter to apply emphasis to your charts. All this requires is calling setHighlight on your chart with the selectionFilter query that you get from the click event, as seen in this sandbox . Applying more complex highlights Since we accept a subset of the MQL language for highlighting, it's possible to specify highlights which target multiple marks, as well as multiple conditions. We can use expressions like $lt and $gte to define ranges which we want to highlight. And since we support the logical operators as well, you can even use $and / $or . All the Comparison , Logical and Element query operators are supported, so give it a spin! Conclusion This ability to highlight data will make your charts more interactive and help you better emphasize the most important information in your charts. Check out the embedding SDK to start highlighting today! New to Charts? You can start now for free by signing up for MongoDB Atlas , deploying a free tier cluster and activating Charts. Have an idea on how we can make MongoDB Charts better? Feel free to leave an idea at the MongoDB Feedback Engine .

September 2, 2021
Updates

Fine-Tune Relevance in MongoDB Atlas Search with Function Scoring and Synonyms

MongoDB Atlas Search is an embedded full-text search solution in MongoDB Atlas that gives developers a seamless and scalable experience for building fast, relevance-based application features. We announced its general availability last year at MongoDB.live 2020 and over the past year we’ve introduced many new features, including a visual index builder, search query tester, custom analyzers , and wildcard path queries . This year at MongoDB.live 2021 , we’re excited to highlight two new capabilities that help developers tune the relevance of search results. See how easy it is to get started with MongoDB Atlas Search in this demo video by Marcus Eagan, Senior Product Manager for Atlas Search. Building relevance into search results Understanding the behavior of your users is essential when thinking about search result relevance. People don’t always tell you what they want, and they sometimes use words or phrases that don’t match your content exactly. To cover these scenarios, you can use full-text search features like function scoring and synonyms. Influence search rankings with function scoring There are often multiple factors that influence how search results should be ranked. For example, let’s say you have a restaurant finder application. The explicit inputs are things like the user’s location and what they’re searching for, but what’s implied is that they likely want to see highly rated restaurants or ones with more reviews. What’s Cooking: a sample restaurant finder application using MongoDB Atlas Search Function scoring allows you to influence the order of results returned by manipulating the score of each result. In Atlas Search, that means you can use a numeric field in a document and apply a mathematical expression to it. For example, you might want to increase the score of restaurants that are sponsored or have higher star ratings. This can easily be accomplished within the same search query by simply adding the function option to the score parameter of your query. Learn more about how to use function scores in our developer tutorial . Show results for more search queries with synonyms Synonyms are often used to define terms that are semantically similar to each other to improve search results. For example, someone searching for “noodles” might want to find results for “spaghetti”, “chow mein”, or “pad thai”. Synonyms can also help with typos, especially on mobile and small keyboards. In Atlas Search, you can define collections of synonyms for a search index via the API. Synonyms can be explicit (one-way) or equivalent (two-way). Explicit synonyms are good for defining relationships between terms that are subsets of each other, like the noodle example above: “spaghetti”, “chow mein”, and “pad thai” are all explicit synonyms for “noodles”, but not each other (you don’t want results for “chow mein” in a search for “spaghetti”). Equivalent synonyms are often used for terms that have regional variations or are otherwise interchangeable both ways, like soda and pop, or Kleenex and tissues. What's next for Atlas Search Developers are increasingly turning to full-text search to make content more discoverable and relevant for application end users. With Atlas Search, we hope to not only make building full-text search easier, but also more powerful and expressive. Join our community to ask questions and find out what other developers are building with Atlas Search and let us know what you think we should build next in our feedback forums .

July 13, 2021
Updates

Introducing Serverless Instances on MongoDB Atlas, Now Available in Preview

Since we first launched MongoDB Atlas in June 2016, we’ve been working towards building a cloud database that not only delivers a first-class developer experience, but also simply just works: no setup, tuning, or maintenance required. Over the years, this has led to features like auto-scaling and click-to-create index suggestions , along with numerous optimizations to our automation engine. We’re excited to announce that we’re one more step closer to realizing this vision with the introduction of serverless databases on MongoDB Atlas . Think less about your database, and more about your data Serverless computing and NoOps have emerged as popular trends in modern application development. Cloud functions are commonly used to power business logic in applications, and many teams rely on completely automated IT operations. The appeal of serverless technology is hard to deny: elastic scaling eliminates the need for upfront resource provisioning and ongoing maintenance, and consumption-based pricing means paying only for resources that are used. It abstracts and automates away many of the lower-level infrastructure decisions that developers don’t want to have to learn or manage so they can focus on building differentiated features. When it comes to databases, compute and storage resources have traditionally been tightly coupled. Applying a serverless model to databases means decoupling them and changing the way engineering teams think about infrastructure. Rather than asking a developer to predict an application’s future workload patterns, break them down into individual resource requirements, and then map them to arbitrary units of database instance sizes, serverless databases offer a much simpler experience: define where your data lives, and get a database endpoint you can use. This not only streamlines the database deployment process, it also eliminates the need to monitor and adjust capacity on an ongoing basis. Developers are free to focus on thinking about their data rather than their databases, and leave the lower-level infrastructure decisions to intelligent, behind-the-scenes automation. Serverless instances on MongoDB Atlas All customers now have the ability to create a serverless database on MongoDB Atlas with the introduction of serverless instances , announced at MongoDB.live 2021 . It’s incredibly easy to get started: simply choose a cloud region and you’ll receive an on-demand database endpoint for your application. Serverless instances always run on the latest MongoDB version so you never have to worry about backwards compatibility or upgrades. You can view and manage them using the same UI and API as your existing database deployment on Atlas (i.e., clusters), and they come with end-to-end security, continuous uptime, metrics, alerts, and backups. Watch this demo of how to create a serverless instance on MongoDB Atlas This new deployment type will be available in preview, so it doesn’t yet support all of the features and capabilities available on clusters today. It’s ideal for infrequent or sparse workloads, or development and testing workloads in the cloud. If you’re running a high-throughput production workload, dedicated clusters are still the recommended deployment option. A hands-free database experience This is the first of many releases, and we have an ambitious roadmap ahead. We will continue to invest in making working with data ever more seamless and delightful for developers, from adding support for newer Atlas capabilities like full-text search and native visualizations , to even more intelligent automation and optimization. Create your own serverless instance on MongoDB Atlas. Try the Preview If you have feedback or questions, we’d love to hear them! Join our community forums to meet other MongoDB developers and see what they’re building with serverless instances. What's next for MongoDB Atlas Serverless instances are just one of many new additions to Atlas that we hope will make developers’ lives easier. Earlier this year, we added index removal suggestions to Performance Advisor and released a quick start for creating and managing clusters via the command line with the MongoDB CLI . We are also working on integrations with Vercel and Netlify , two popular serverless application platforms, to give developers an easy way to get started on MongoDB Atlas. What would make your development experience better on MongoDB Atlas? Share your feature requests in our feedback forums .

July 13, 2021
Updates

Streaming Time-Series Data Using Apache Kafka and MongoDB

There is one thing the world agrees on and it is the concept of time. Many applications are heavily time-based. Consider solar field power generation, stock trading, and health monitoring. These are just a few of the plethora of applications that produce and use data that contains a critical time component. In general, time-series data applications are heavy on inserts, rarely perform updates and are even more unlikely to delete the data. These applications generate a tremendous amount of data and need a robust data platform to effectively manage and query data. With MongoDB, you can easily: Pre-aggregate data using the MongoDB Query language and window functions Optimally store large amounts of time-series data with MongoDB time-series collections Archive data to cost effective storage using MongoDB Atlas Online Archive Apache Kafka is often used as an ingestion point for data due to its scalability. Through the use of the MongoDB Connector for Apache Kafka and the Apache Kafka Connect service, it is easy to transfer data between Kafka topics and MongoDB clusters. Starting in the 1.6 release of the MongoDB Connector for Apache Kafka, you can configure kafka topic data to be written directly into a time-series collection in MongoDB. This configuration happens in the sink. Configuring time series collections in the sink With MongoDB, applications do not need to create the database and collection before they start writing data. These objects are created automatically upon first arrival of data into MongoDB. However, a time-series collection type needs to be created first before you start writing data. To make it easy to ingest time-series data into MongoDB from Kafka, these collection options are exposed as sink parameters and the time-series collection is created by the connector if it doesn’t already exist . Some of the new parameters are defined as follows: timeseries.timefield Name of the top level field used for time. timeseries.expire.after.seconds This optional field determines the amount of time the data will be in MongoDB before being automatically deleted. Omitting this field means data will not be deleted automatically. If you are familiar with TTL indexes in MongoDB, setting this field provides a similar behavior. timeseries.timefield.auto.convert This optional field tells the connector to convert the data in the field into a BSON Date format. Supported formats include integer, long, and string. For a complete list of the new time-seris parameters check out the MongoDB Sink connector online documentation . When data is stored in time-series collections, MongoDB optimizes the storage and bucketization of your data behind the scenes. This saves a tremendous amount of storage space compared to the typical one document per data point data structure in regular collections. You can also explore the many new time and window functionalities within the MongoDB Query Language. For example, consider this sample document structure: { tx_time: 2021-06-30T15:47:31.000Z, _id: '60dc921372f0f39e2cd6cba5', company_name: 'SILKY CORNERSTONE LLC', price: 94.0999984741211, company_symbol: 'SCL' } You can use the new $setWindowFields pipeline to define the window of documents to perform an operation on then perform rankings, cumulative totals, and other analytics of complex time series data. For example, using the data generated in the tutorial, let’s determine the rolling average to the data as follows: db.StockDataTS.aggregate( [ { $match: {company_symbol: 'SCL'} }, { $setWindowFields: { partitionBy: '$company_name', sortBy: { 'tx_time': 1 }, output: { averagePrice: { $avg: "$price", window: { documents: [ "unbounded", "current" ] } } } } } ]) A sample of the result set is as follows: { tx_time: 2021-06-30T15:47:45.000Z, _id: '60dc922172f0f39e2cd6cbeb', company_name: 'SILKY CORNERSTONE LLC', price: 94.06999969482422, company_symbol: 'SCL', averagePrice: 94.1346669514974 }, { tx_time: 2021-06-30T15:47:47.000Z, _id: '60dc922372f0f39e2cd6cbf0', company_name: 'SILKY CORNERSTONE LLC', price: 94.1500015258789, company_symbol: 'SCL', averagePrice: 94.13562536239624 }, { tx_time: 2021-06-30T15:47:48.000Z, _id: '60dc922472f0f39e2cd6cbf5', company_name: 'SILKY CORNERSTONE LLC', price: 94.0999984741211, company_symbol: 'SCL', averagePrice: 94.13352966308594 } Notice the additional “averagePrice” field is now populated with a rolling average. For more information on time-series collection in MongoDB check out the online documentation . Migrating existing collections To convert an existing MongoDB collection to a time-series collection you can use the MongoDB Connector for Apache Kafka. Simply configure the source connection to your existing collection and configure the sink connector to write to a MongoDB time series collection by using the “timeseries.timefield” parameter. You can configure the source connector to copy existing data by setting the “copy.existing” parameter to true. This will create insert events for all existing documents in the source. Any documents that were inserted during the copying process will be inserted once the copying process has finished. While not always possible, it is recommended to pause writes to the source data while the copy process is running. To see when it finishes, you can view the logs for the message, “Finished copying existing data from the collection(s).”. For example, consider a source document that has this structure: { company_symbol: (STRING), company_name: (STRING), price: (DECIMAL), tx_time: (STRING) } For the initial release of MongoDB Time series collections, the field that represents the time is required to be stored as a Date. In our example, we are using a string to showcase the ability for the connector to automatically convert from a string to a Date. If you chose to perform the conversion outside of the connector you could use a Single Message Transform in Kafka Connect to convert the string into a Date at the Sink. However, certain SMTs like Timestampconverter require schemas to be defined for the data in the Kafka topic in order to work. This may add some complexity to the configuration. Instead of using an SMT you can automatically convert into Dates using the new timeseries.timefield.auto.convert, and timeseries.timefield.auto.convert.date.format options. Here is a sample source configuration that will copy all the existing data from the StockData collection then continue to push data changes to the stockdata.Stocks.StockData topic: {"name": "mongo-source-stockdata", "config": { "tasks.max":"1", "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector", "key.converter":"org.apache.kafka.connect.storage.StringConverter", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "publish.full.document.only": true, "connection.uri":(MONGODB SOURCE CONNECTION STRING), "topic.prefix":"stockdata", "database":"Stocks", "collection":"StockData", "copy.existing":"true" }} This is a sample configuration for the sink to write the data from the stockdata.Stocks.StockData topic to a MongoDB time series collection: {"name": "mongo-sink-stockdata", "config": { "connector.class":"com.mongodb.kafka.connect.MongoSinkConnector", "tasks.max":"1", "topics":"stockdata.Stocks.StockData", "connection.uri":(MONGODB SINK CONNECTION STRING), "database":"Stocks", "collection":"StockDataMigrate", "key.converter":"org.apache.kafka.connect.storage.StringConverter", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "timeseries.timefield":"tx_time", "timeseries.timefield.auto.convert":"true", "timeseries.timefield.auto.convert.date.format":"yyyy-MM-dd'T'HH:mm:ss'Z'" }} In this sink example, the connector will convert the data in the “tx_time” field into a Date and parse it expecting the string format yyyy-MM-ddTHH:mm:ssZ (e.g. '2021-07-06T12:25:45Z') Note that in the initial version of time-series collections, only insert into a time-series collection is supported. Updating or deleting documents on the source will not propagate to the destination. Also, you can not use the MongoDB CDC Handler in this scenario because the handler uses ReplaceOne which is a type of update command. These are limitations of the initial release of time-series in MongoDB and may be irrelevant by the time you read this post. Check the online documentation for the latest information. The MongoDB Connector for Apache Kafka version 1.6 is available to download from GitHub . Look for it on the Confluent Hub later this week!

July 13, 2021
Updates

Ready to get Started with MongoDB Atlas?

Start Free