Atlas Stream Processing is Now in Public Preview

Clark Gates-George and Joe Niemiec
February 13, 2024 | Updated: March 12, 2024
#Atlas Stream Processing

This post is also available in: Deutsch, Français, Español, Português, Italiano, 한국인, 简体中文.

Today, we’re excited to announce that Atlas Stream Processing is now in public preview. Any developer on Atlas interested in giving it a try has full access. Learn more in our docs or get started today.

Listen to the MongoDB Podcast to learn about the Atlas Stream Processing public preview from Head of Streaming Products, Kenny Gorman.

Developers love the flexibility and ease of use of the document model, alongside the Query API, which allows them to work with data as code in MongoDB Atlas. With Atlas Stream Processing, we are bringing these same foundational principles to stream processing.

A report covering the topic published by S&P Global Market Intelligence 451 Research had this to say, “A unified approach to leveraging data for application development — the direction of travel for MongoDB — is particularly valuable in the context of stream processing where operational and development complexity has proven a significant barrier to adoption."

First announced at .local NYC 2023, Atlas Stream Processing is redefining the experience of aggregating and enriching streams of high velocity, rapidly changing event data, and unifying how to work with data in motion and at rest.

How are developers using the product so far? And what have we learned?

During the private preview, we saw thousands of development teams request access and we have gathered useful feedback from hundreds of engaged teams. One of those engaged teams is marketing technology leader, Acoustic:

"At Acoustic, our key focus is to empower brands with behavioral insights that enable them to create engaging, personalized customer experiences. To do so, our Acoustic Connect platform must be able to efficiently process and manage millions of marketing, behavioral, and customer signals as they occur. With Atlas Stream Processing, our engineers can leverage the skills they already have from working with data in Atlas to process new data continuously, ensuring our customers have access to real-time customer insights."
John Riewerts, EVP, Engineering at Acoustic

Other interesting use cases include:

A leading global airline using complex aggregations to rapidly process maintenance and operations data, ensuring on-time flights for their thousands of daily customers,
A large manufacturer of energy equipment using Atlas Stream Processing to enable continuous monitoring of high volume pump data to avoid outages and optimize their yields, and
An innovative enterprise SaaS provider leveraging the rich processing capabilities in Atlas Stream Processing to deliver timely and contextual in-product alerts to drive improved product engagement.

These are just a few of the many use case examples that we’re seeing across industries. Beyond the use cases we’ve already seen, developers are giving us tons of insight into what they’d like to see us add to in the future.

In addition to enabling continuous processing of data in Atlas databases through change streams, it’s exciting to see developers using Atlas Stream Processing with their Kafka data hosted by valued partners like Confluent, Amazon MSK, Azure Event Hubs, and Redpanda. Our aim with developer data platform capabilities in Atlas has always been to make for a better experience across the key technologies relied on by developers.

What’s new in the public preview?

That brings us to what’s new. As we scale to more teams, we’re expanding functionality to include the most requested feedback gathered in our private preview. From the many pieces of feedback received, three common themes emerged:

Refining the developer experience
Expanding advanced features and functionality
Improving operations and security

Refining the developer experience

In private preview, we established the core of the developer experience that is essential to making Atlas Stream Processing a natural solution for development teams. And in public preview, we’re doubling down on this by making two additional enhancements:

VS Code integration
The MongoDB VS Code plugin has added support for connecting to Stream Processing instances. For developers already leveraging the plugin, teams can create and manage processors in a familiar development environment. This means less time switching between tools and more time building your applications!
Improved dead letter queue (DLQ) capabilities
DLQ support is a key element for powerful stream processing and in public preview, we’re expanding DLQ capabilities. DLQ messages will now display themselves when executing pipelines with sp.process() and when running .sample() on running processors, allowing for a more streamlined development experience that does not require setting up a target collection to act as a DLQ.

Expanding advanced features and functionality

Atlas Stream Processing already supported many of the key aggregation operators developers are familiar with in the Query API used with data at rest. We've now added powerful windowing capabilities and the ability to easily merge and emit data to an Atlas database or to a Kafka topic. Public preview will add even more functionality demanded by the most advanced teams relying on stream processing to deliver customer experiences:

$lookup
Developers can now enrich documents being processed in a stream processor with data from remote Atlas clusters, performing joins against fields from the document and the target collection.
Change streams pre- and post-imaging
Many developers are using Atlas Stream Processing to continuously process data in Atlas databases as a source through change streams. We have enhanced the change stream $source in public preview with support for pre- and post-images. This enables common use cases where developers need to calculate deltas between fields in documents as well as use cases requiring access to the full contents of a deleted document.
Conditional routing with dynamic expressions in merge and emit stages
Conditional routing lets developers use the value of fields in documents being processed in Atlas Stream Processing to dynamically send specific messages to different Atlas collections or Kafka topics. The $merge and $emit stages also now support the use of dynamic expressions. This makes it possible to use the Query API for use cases requiring the ability to fork messages to different collections or topics as needed.
Idle stream timeouts
Streams without advancing watermarks due to a lack of inbound data can now be configured to close after a period of time emitting the results of the windows. This can be critical for streaming sources that have inconsistent flows of data.

Improving operations and security

Finally, we have invested heavily over the past few months in improving other operational and security aspects of Atlas Stream Processing. A few of the highlights include:

Checkpointing
Atlas Stream Processing now performs checkpoints for saving a state while processing. Stream processors are continuously running processes, so whether due to a data issue or infrastructure failure, they require an intelligent recovery mechanism. Checkpoints make it easy to resume your stream processors from wherever data stopped being collected and processed.
Terraform provider support
Support for the creation of connections and stream processing instances (SPIs) is now available with Terraform. This allows for infrastructure to be authored as code for repeatable deployments.
Security roles
Atlas Stream Processing has added a project-level role, giving users just enough permission to perform their stream processing tasks. Stream processors can run under the context of a specific role, supporting a least privilege configuration.
Auditing
Atlas Stream Processing can now audit authentication attempts and actions within your Stream Processing Instance giving you insight into security-related events.
Kafka consumer group support
Stream processors in now use Kafka consumer groups for offset tracking. This allows users to easily change the position of the processor in the stream for operations and easily monitor for potential processor lag.

A final note on what’s new is that in public preview, we will begin charging for Atlas Stream Processing, using preview pricing (subject to change). You can learn more about pricing in our documentation.

Build your first stream processor today

Public preview is a huge step forward for us as we expand the developer data platform and enable more teams with a stream processing solution that simplifies the operational complexity of building reactive, responsive, event-driven applications, while also offering an improved developer experience.

We can’t wait to see what you build!

Login today or get started with the tutorial, view our resources, or follow the Learning Byte on MongoDB University.

← Previous

Atlas Stream Processing ahora está en vista previa pública.

Hoy, nos complace anunciar que Atlas Stream Processing ya está en versión preliminar pública. La vista previa pública abre el acceso a cualquier desarrollador en Atlas interesado en probarlo. Obtenga más información en nuestros docs o comienza hoy mismo. Escuche el podcast de MongoDB para obtener más información sobre la versión preliminar pública de Atlas Stream Processing del jefe de productos de transmisión, Kenny Gorman. A los desarrolladores les encanta la flexibilidad y facilidad de uso del modelo de documentos, junto con la API de consultas, que les permite trabajar con datos como código. Con Atlas Stream Processing, trasladamos estos mismos principios fundamentales al procesamiento de flujos. Anunciado por primera vez en .local NYC 2023 , Atlas Stream Processing está redefiniendo la experiencia de agregar y enriquecer flujos de datos de transmisiones de alta velocidad y que cambian con rapidez, y unificando cómo trabajar con datos en movimiento y en reposo. ¿Cómo usan el producto los desarrolladores hasta ahora? ¿Y qué aprendimos? Durante la vista previa privada, vimos que miles de equipos de desarrollo solicitan acceso y recopilamos comentarios útiles de cientos de equipos comprometidos. Entre los casos de uso se incluyen los siguientes: Una aerolínea líder mundial que utiliza agregaciones complejas para procesar rápidamente los datos de mantenimiento y operaciones, lo que garantiza la puntualidad de los vuelos para sus miles de clientes diarios Un gran fabricante de equipos de energía que utilizan Atlas Stream Processing para permitir el monitoreo continuo de datos de bombas de alto volumen para evitar interrupciones y optimizar sus rendimientos Un innovador proveedor de SaaS empresarial que aprovecha las capacidades de procesamiento enriquecidas en Atlas Stream Processing para ofrecer alertas oportunas y contextuales en el producto para impulsar una mejor interacción con el producto Estos son solo algunos de los muchos ejemplos de casos de uso que vemos en todas las industrias para Atlas Stream Processing. Más allá de los casos de uso que ya vimos, los desarrolladores nos dan toneladas de información sobre lo que les gustaría que agregáramos a Atlas Stream Processing en el futuro. Además de permitir el procesamiento continuo de los datos en las bases de datos de Atlas a través de change stream, fue impactante ver a los desarrolladores utilizar Atlas Stream Processing con sus datos de Kafka alojados por socios valiosos como Confluent , Amazon MSK , Azure Event Hubs y Redpanda . Nuestro objetivo con las capacidades de la plataforma de datos para desarrolladores siempre es lograr una mejor experiencia en todas las tecnologías clave en las que confían los desarrolladores. ¿Qué hay de nuevo en la vista previa pública? Eso nos trae a las novedades. Y a medida que escalamos a más equipos, expandimos la funcionalidad para incluir los comentarios más solicitados recopilados en nuestra vista previa privada. De los muchos comentarios recibidos, surgieron tres temas comunes: Cómo refinar la experiencia del desarrollador Ampliar las características y funcionalidades avanzadas, y Mejorar las operaciones y la seguridad Cómo refinar la experiencia del desarrollador En la vista previa privada, establecimos el núcleo de la experiencia del desarrollador esencial para hacer de Atlas Stream Processing una solución natural para los equipos de desarrollo. Y en la vista previa pública, publicamos esto haciendo 2 mejoras adicionales: Integración de VS Code El complemento MongoDB VS Code agregó soporte para conectarse a instancias de procesamiento de flujo. Para los desarrolladores que ya utilizan el complemento, esto permite a los equipos crear y gestionar procesadores en un entorno de desarrollo familiar. ¡Pase menos tiempo cambiando entre herramientas y más tiempo creando sus aplicaciones! Capacidades mejoradas de cola de mensajes fallidos (DLQ) La compatibilidad con DLQ es un elemento clave para un potente procesamiento de transmisiones y, en vista previa pública, estamos expandiendo las capacidades de DLQ en Atlas Stream Processing. Los mensajes de la cola de letras muertas ahora se mostrarán al ejecutar tuberías con sp.proceso() y al ejecutar .muestra() en procesadores en ejecución, lo que permite una experiencia de desarrollo más optimizada que no requiere configurar una collection objetivo para actuar como DLQ. Ampliación de características y funcionalidades avanzadas Atlas Stream Processing ya admitía muchos de los operadores de agregación clave con los que los desarrolladores están familiarizados en la API de consulta utilizada con datos en reposo, y agregaba potentes funciones de ventanas y la capacidad de fusionar y emitir datos fácilmente a la base de datos Atlas o a un tema Kafka. La vista previa pública agregará aún más funcionalidades exigidas por los equipos más avanzados que confían en el procesamiento de stream para ofrecer experiencias de cliente: $lookup Los desarrolladores ahora pueden enriquecer los documentos que se procesan en un procesador de transmisión con datos del Atlas cluster remoto, y realizar uniones con campos del documento y la collection de destino. change stream antes y después de la creación de imágenes Muchos desarrolladores están utilizando Atlas Stream Processing para procesar continuamente datos en bases de datos Atlas como fuente a través de change streams. Mejoramos el $source de change stream en vista previa pública con asistencia técnica para imágenes previas y posteriores. Esto permite casos de uso comunes en los que los desarrolladores necesitan calcular deltas entre campos en documentos, así como casos de uso que requieren acceso al contenido completo de un documento eliminado. Enrutamiento condicional con expresiones dinámicas en las etapas de fusión y emisión El enrutamiento condicional permite a los desarrolladores utilizar el valor de los campos de los documentos que se procesan en Atlas Stream Processing para enviar dinámicamente mensajes específicos a diferentes collections de Atlas o temas de Kafka. Las etapas $merge y $emit ahora también admiten el uso de expresiones dinámicas. Esto permite utilizar la API de consultas para casos de uso que requieren la capacidad de bifurcar mensajes a diferentes collections o temas, según sea necesario. Tiempos de espera de transmisión inactiva Las reproducciones sin avanzar en las marcas de agua debido a la falta de datos entrantes ahora se pueden configurar para cerrar después de un período de tiempo que emita los resultados de las ventanas. Esto puede ser fundamental para las fuentes de transmisión que tienen flujos de datos inconsistentes. Mejorar las operaciones y la seguridad Por último, invertimos mucho en los últimos meses en mejorar otros aspectos operativos y de seguridad de Atlas Stream Processing. Entre los aspectos destacados se incluyen los siguientes: Punto de control Atlas Stream Processing ahora realiza puntos de control para guardar el estado durante el procesamiento. Los procesadores de flujos son procesos en ejecución continua, por lo que, ya sea por un problema de datos o por un fallo de la infraestructura, requieren un mecanismo de recuperación inteligente. Los puntos de control facilitan la reanudación de sus procesadores de transmisión desde cualquier lugar donde los datos dejaron de recopilarse y procesarse. Asistencia técnica para proveedores de Terraform El soporte para la creación de Conexiones e Instancias de Procesamiento de Transmisión (SPI) ahora está disponible con Terraform. Esto permite crear la infraestructura como código para implementaciones repetibles. Funciones de seguridad Atlas Stream Processing agregó una función a nivel de proyecto, que otorga a los usuarios los permisos necesarios para realizar sus tareas de procesamiento de transmisiones. Los procesadores de transmisiones pueden ejecutarse bajo el contexto de una función específica, lo que permite una configuración de mínimos privilegios. Asistencia técnica del grupo de consumidores de Kafka Los procesadores de transmisión en Atlas Stream Processing ahora usan grupos de consumidores Kafka para el seguimiento compensatorio. Esto permite a los usuarios cambiar fácilmente la posición del procesador en la transmisión de operaciones y controlar fácilmente el posible retraso del procesador. Una nota final sobre las novedades es que, en la vista previa pública, comenzaremos a cobrar por Atlas Stream Processing, utilizando precios promocionales hasta nuestra versión de disponibilidad general. Puede obtener más información sobre los precios de Atlas Stream Processing en nuestra documentación. Crea tu primer procesador de transmisión hoy La vista previa pública es un gran avance para nosotros a medida que ampliamos la plataforma de datos para desarrolladores y permitimos más equipos con una solución de procesamiento de transmisiones que simplifica la complejidad operativa de crear aplicaciones reactivas, receptivas y basadas en eventos, al tiempo que también ofrece una mejor experiencia de desarrollador. ¡Estamos ansiosos por ver lo que desarrollas! Inicia sesión hoy mismo para comenzar u obtener más información sobre Atlas Stream Processing en nuestra documentación , recursos , tutoriales o Learning Byte en MongoDB University.

February 13, 2024

Next →

Building AI With MongoDB: Integrating Vector Search And Cohere to Build Frontier Enterprise Apps

Cohere is the leading enterprise AI platform, building large language models (LLMs) which help businesses unlock the potential of their data. Operating at the frontier of AI, Cohere’s models provide a more intuitive way for users to retrieve, summarize, and generate complex information. Cohere offers both text generation and embedding models to its customers. Enterprises running mission-critical AI workloads select Cohere because its models offer the best performance-cost tradeoff and can be deployed in production at scale. Cohere’s platform is cloud-agnostic. Their models are accessible through their own API as well as popular cloud managed services, and can be deployed on a virtual private cloud (VPC) or even on-prem to meet companies where their data is, offering the highest levels of flexibility and control. Cohere’s leading Embed 3 and Rerank 3 models can be used with MongoDB Atlas Vector Search to convert MongoDB data to vectors and build a state-of-the-art semantic search system. Search results also can be passed to Cohere’s Command R family of models for retrieval augmented generation (RAG) with citations. Check out our AI resource page to learn more about building AI-powered apps with MongoDB. A new approach to vector embeddings It is in the realm of embedding where Cohere has made a host of recent advances. Described as “AI for language understanding,” Embed is Cohere’s leading text representation language model. Cohere offers both English and multilingual embedding models, and gives users the ability to specify the type of data they are computing an embedding for (e.g., search document, search query). The result is embeddings that improve the accuracy of search results for traditional enterprise search or retrieval-augmented generation. One challenge developers faced using Embed was that documents had to be passed one by one to the model endpoint, limiting throughput when dealing with larger data sets. To address that challenge and improve developer experience, Cohere has recently announced its new Embed Jobs endpoint . Now entire data sets can be passed in one operation to the model, and embedded outputs can be more easily ingested back into your storage systems. Additionally, with only a few lines of code, Rerank 3 can be added at the final stage of search systems to improve accuracy. It also works across 100+ languages and offers uniquely high accuracy on complex data such as JSON, code, and tabular structure. This is particularly useful for developers who rely on legacy dense retrieval systems. Demonstrating how developers can exploit this new endpoint, we have published the How to use Cohere embeddings and rerank modules with MongoDB Atlas tutorial . Readers will learn how to store, index, and search the embeddings from Cohere. They will also learn how to use the Cohere Rerank model to provide a powerful semantic boost to the quality of keyword and vector search results. Figure 1: Illustrating the embedding generation and search workflow shown in the tutorial Why MongoDB Atlas and Cohere? MongoDB Atlas provides a proven OLTP database handling high read and write throughput backed by transactional guarantees. Pairing these capabilities with Cohere’s batch embeddings is massively valuable to developers building sophisticated gen AI apps. Developers can be confident that Atlas Vector Search will handle high scale vector ingestion, making embeddings immediately available for accurate and reliable semantic search and RAG. Increasing the speed of experimentation, developers and data scientists can configure separate vector search indexes side by side to compare the performance of different parameters used in the creation of vector embeddings. In addition to batch embeddings, Atlas Triggers can also be used to embed new or updated source content in real time, as illustrated in the Cohere workflow shown in Figure 2. Figure 2: MongoDB Atlas Vector Search supports Cohere’s batch and real time workflows. (Image courtesy of Cohere) Supporting both batch and real-time embeddings from Cohere makes MongoDB Atlas well suited to highly dynamic gen AI-powered apps that need to be grounded in live, operational data. Developers can use MongoDB’s expressive query API to pre-filter query predicates against metadata, making it much faster to access and retrieve the more relevant vector embeddings. The unification and synchronization of source application data, metadata, and vector embeddings in a single platform, accessed by a single API, makes building gen AI apps faster, with lower cost and complexity. Those apps can be layered on top of the secure, resilient, and mature MongoDB Atlas developer data platform that is used today by over 45,000 customers spanning startups to enterprises and governments handling mission-critical workloads. What's next? To start your journey into gen AI and Atlas Vector Search, review our 10-minute Learning Byte . In the video, you’ll learn about use cases, benefits, and how to get started using Atlas Vector Search.

April 25, 2024