Atlas Stream Processing is Now Generally Available!

We're thrilled to announce that Atlas Stream Processing—the MongoDB-native way to process streaming data—is now generally available, empowering developers to quickly build responsive, event-driven applications!

Our team spent the last two years defining a vision and building a product that leans into MongoDB’s strengths to overcome the hard challenges in stream processing. After a decade of building stream processing products outside of MongoDB, we are using everything that makes MongoDB unique and differentiated—the Query API and powerful aggregation framework, as well as the document model and its schema flexibility—to create an awesome developer experience.

It’s a new approach to stream processing, and based on the feedback of so many of you in our community, it’s the best way for most developers using MongoDB to do it.

Let’s get into what’s new.

This post is also available in: Deutsch, Français, Español, Português, Italiano, 한국어, 简体中文.

What's new in general availability?

Graphic calling out the new features in general availability. Under the Dev Experience Improvements category, there is
  1. Production Readiness
    Ready to support your production workloads, ensuring reliable and scalable stream processing for your mission-critical applications.

  2. Time Series Collection Support
    Emit processor results into Time Series Collections. Pre-process data continuously while saving it for historical access later in a collection type available in MongoDB Atlas built to efficiently store and query time series data.

  3. Development and Production Tiers
    Besides the SP30 cluster tier available during the public preview, we’re introducing an SP10 tier to provide flexibility and a cost-effective option for exploratory use cases and low-traffic stream processing workloads.

  4. Improved Kafka Support
    Added support for Kafka headers allows applications to provide additional metadata alongside event data. They are helpful for various stream processing use cases (e.g., routing messages, conditional processing, and more).

  5. Least Privilege Access
    Atlas Database Users can grant access to Stream Processing Instances and enable access to only those who need it. Read our tutorial for more information.

  6. Stream Processor Alerting
    Gain insight and visibility into the health of your stream processors by creating alerts for when a failure occurs. Supported methods for alerting include email, SMS, monitoring platforms like Datadog, and more.

Why Atlas Stream Processing?

Atlas Stream Processing brings the power and flexibility of MongoDB's document model and Query API to the challenging stream processing space. With Atlas Stream Processing, developers can:

  • Effortlessly handle complex and rapidly changing data structures

  • Use the familiar MongoDB Query API for processing streaming data

  • Seamlessly integrate with MongoDB Atlas

  • Benefit from a fully managed service that eliminates operational overhead

Customer highlights

Read what developers are saying about Atlas Stream Processing:

At Acoustic, our key focus is to empower brands with behavioral insights that enable them to create engaging, personalized customer experiences. To do so, our Acoustic Connect platform must be able to efficiently process and manage millions of marketing, behavioral, and customer signals as they occur. With Atlas Stream Processing, our engineers can leverage the skills they already have from working with data in Atlas to process new data continuously, ensuring our customers have access to real-time customer insights.

John Riewerts, EVP, Engineering at Acoustic

Atlas Stream Processing enables us to process, validate, and transform data before sending it to our messaging architecture in AWS powering event-driven updates throughout our platform. The reliability and performance of Atlas Stream Processing has increased our productivity, improved developer experience, and reduced infrastructure cost.

Cody Perry, Software Engineer, Meltwater

What's ahead for Atlas Stream Processing?

Graphic calling out some of the things that are ahead for Atlas Stream Processing, including more regions, advanced networking, additional sources & sinks, and richer metrics & observability

We’re rapidly introducing new features and functionality to ensure MongoDB delivers a world-class stream processing experience for all development teams. Over the next few months, you can expect to see:

  1. Advanced Networking Support
    Support for VPC Peering to Kafka Clusters for teams requiring additional networking capabilities

  2. Expanded Cloud Region Support
    Support for all cloud regions available in Atlas Data Federation

  3. Expanded Cloud Provider Support
    Support for Microsoft Azure

  4. Expanded Data Source and Sink Support
    We have plans to expand beyond Kafka and Atlas databases in the coming months. Let us know which sources and sinks you need, and we will factor that into our planning

  5. Richer Metrics & Observability
    Support for expanded visibility into your stream processors to help simplify monitoring and troubleshooting

  6. Expanded Deployment Flexibility
    Support for deploying stream processors with Terraform. This integration will help to enable a seamless CI/CD pipeline, enhancing operational efficiency with infrastructure as code. Look out for a dedicated blog in the near future on how to get started with Atlas Stream Processing and Terraform.

So whether you're looking to process high-velocity sensor data, continuously analyze customer data to deliver personalized experiences, or perform predictive maintenance to increase yields and reduce costs, Atlas Stream Processing has you covered. Join the hundreds of development teams already building with Atlas Stream Processing. Stay tuned to hear more from us soon, and good luck building!

Login today or check out our introductory tutorial to get started.