Each Stream Processing Instance stores every configured connection to a streaming data source or sink in its connection registry. The procedures on this page walk you through creating and configuring your Atlas Stream Processing connections.
Prerequisites
To create and configure connections, you must have the following:
An Atlas project
An Atlas user with the
Project Owneror theProject Stream Processing Ownerrole to manage a connection registryNote
While both the
Project OwnerandProject Stream Processing Ownerroles grant the privileges to create, view, edit and delete connections in the connection registry, theProject Ownerrole also allows users to perform tasks such as creating database deployments, managing project access, and more.See Project Roles to learn more about the differences between these roles.
An Atlas cluster
Considerations
When configuring Atlas Stream Processing connections, ensure you select a connection type suitable to your use case. Atlas Stream Processing supports the following providers:
Provider | Description | Connection Types |
|---|---|---|
Kafka Broker | Configurable server that receives and stores event streams, organizing them into partitioned topics. | Sink and Source |
Atlas Clusters | Cloud-hosted MongoDB server providing both permanent storage and event-driven Change Streams. | Sink and Source |
AWS S3 | Object storage container hosted on AWS. | Only Sink |
HTTPS | HTTPS request to retrieve data through an API endpoint each time a document passes through. Atlas Stream Processing then uses this data to enrich the document that triggered the request. | Supplementary Source; must follow a |
External Function Connections | Atlas Stream Processing supports embedding calls to AWS Lambda resources within your Atlas Stream Processing pipeline. You can execute these calls synchronously or asynchronously, and you can pass responses from synchronous calls to subsequent stages in your pipeline. To learn more, see the Atlas Stream Processing External Functions documentation. | Trigger |
Add a Connection to the Connection Registry
Each Atlas Stream Processing connection type has distinct configuration requirements and procedures. To learn more, see Add an Atlas Stream Processing Connection.
View Connections in the Connection Registry
To view a connection to your stream processing instance:
To list all connections from one stream processing instance you specify using the Atlas CLI, run the following command:
atlas streams connections list [options]
To return the details for one stream processing connection you specify using the Atlas CLI, run the following command:
atlas streams connections describe <streamConnectionName> [options]
To learn more about the syntax and parameters for the previous commands, see the Atlas CLI documentation for atlas streams connections list and atlas streams connections describe.
In Atlas, go to the Stream Processing page for your project.
WARNING: Navigation Improvements In Progress
We're currently rolling out a new and improved navigation experience. If the following steps don't match your view in the Atlas UI, see the preview documentation.
If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.
Go to the Connection Registry page.
Locate the overview panel of the stream processing instance containing the connection registry you want to view.
Click Configure.
Select the Connection Registry tab.
Atlas Stream Processing displays all of the connections available in your instance connection registry.
The Atlas Administration API provides different endpoints for retrieving one or all connections in a connection registry.
To list all available connections using mongosh, use the
sp.listConnections() method. It returns a list of documents
describing each connection in the stream processing instance's connection
registry. It has the following syntax:
sp.listConnections()
Modify a Connection from the Connection Registry
To modify a connection from a connection registry:
To modify the details of one connection on the stream processing instance you specify using the Atlas CLI, run the following command:
atlas streams connections update <connectionName> [options]
To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections update.
In Atlas, go to the Stream Processing page for your project.
WARNING: Navigation Improvements In Progress
We're currently rolling out a new and improved navigation experience. If the following steps don't match your view in the Atlas UI, see the preview documentation.
If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.
Modify the connection.
For the connection that you want to modify in the table, click the pencil icon in the Actions column.
You can edit a connection only if it is not currently in use by a stream processor. If you want to edit a connection in use, stop all stream processors that use the connection first.
Modify the fields you want to modify.
Click Save changes.
The Atlas Administration API provides an endpoint for editing a stream processing instance.
Delete a Connection from the Connection Registry
To delete a connection from a connection registry:
To remove one connection from the stream processing instance you specify using the Atlas CLI, run the following command:
atlas streams connections delete <connectionName> [options]
To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas streams connections delete.
In Atlas, go to the Stream Processing page for your project.
WARNING: Navigation Improvements In Progress
We're currently rolling out a new and improved navigation experience. If the following steps don't match your view in the Atlas UI, see the preview documentation.
If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar.
If it's not already displayed, select your project from the Projects menu in the navigation bar.
In the sidebar, click Stream Processing under the Services heading.
The Stream Processing page displays.
The Atlas Administration API provides an endpoint for editing a stream processing instance.
List Private Link Connections
The Atlas Administration API provides an endpoint for viewing all Private Link connections configured for Atlas Stream Processing within a specified project.
View One Private Link Connection
The Atlas Administration API provides an endpoint for viewing the details of one Private Link connection configured for Atlas Stream Processing within a specified project.
Delete a Private Link Connection
The Atlas Administration API provides an endpoint for deleting a Private Link connection configured for Atlas Stream Processing within a specified project.