Introducing Atlas Stream Processing Support Within the MongoDB for VS Code Extension
Rate this tutorial
Across industries, teams are building applications that need access to low-latency data to deliver compelling experiences and gain valuable business insights. Stream processing is a fundamental building block powering these applications. Stream processing lets developers discover and act on streaming data (data in motion), and combine that data when necessary with data at rest (data stored in a database). MongoDB is a natural fit for streaming data with its capabilities around storing and querying unstructured data and an effective query API. MongoDB Atlas Stream Processing is a service within MongoDB Atlas that provides native stream processing capabilities. In this article, you will learn how to use the MongoDB for VS Code extension to create and manage stream processors in MongoDB Atlas.
MongoDB support for VS Code is provided by the MongoDB for VS Code extension. To install the MongoDB for VS Code extension, launch VS Code, open the Extensions view, and search for MongoDB to filter the results. Select the MongoDB for VS Code extension.
Once installed, you can connect to the Atlas Stream Processing instance by clicking on the “Connect” button, as shown below.
This will pop up a connection string dialog box.
To get the MongoDB Atlas Stream Processing instance connection string, navigate to the Atlas portal and then “Stream Processing” in the menu, and click on the “Connect” button of the Stream Processing Instance you’d like to connect from VS Code.
This will launch the Connect dialog. Select “MongoDB for VS Code,” as shown below.
The connection string will be displayed in Step 3 of this dialog.
Copy the “mongodb://…” connection string to the clipboard and paste it into the connection string box in VS Code as shown in Figure 3. Make sure to modify the username : password with your own credentials. Note the colon separation of the username and password.
Once connected, you can explore the Atlas Stream Processor instance and the associated Stream Processors. Since this is a new Stream Processing instance, there are no stream processors. Let’s create one using the Playground.
Click on the “Create New Playground” button. This will launch a window with an Atlas Stream Processing template. The template provides two commands, sp.listConnections() and sp.process().
By highlighting text within the Playground, you can selectively execute parts of the script as shown in the figure above. In our example, by highlighting the sp.listConnections() statement and clicking the “Run” button, the Playground will display the results as shown in Figure 8. In this case, we have two connection registry entries defined for this Stream Processor instance. For commands that return a single result, such as listConnections, the results will be displayed in the Playground Result tab. For commands that stream results, such as with sp.process(), these results will be written to the output tab as shown below.
Note that each execution of the Playground is independent so if your script has variables, for example, make sure you include those definitions each time you run the Playground.
The .process() command runs the stream processor continually, sending results to the console. This is great for developers since they can build out their stream processors and test them before creating them to run in perpetuity in the background.
In this example, notice when .process is run, there is a popup within VS Code that enables you to cancel/stop the .process. While you might be tempted to hit Control-C to stop the stream, clicking “Cancel” is the way to stop the output. Control-C will not stop the processing.
Now that we have explored the Playground, let’s create a simple stream processor that reads data from the solar sample stream, computes a tumbling window, and writes the results out to an Atlas collection. Assume two connection registry entries were created in the Atlas portal, one for sample_stream_solar and one for our Atlas cluster, called “LBSolar.” To develop the new stream processor, the following code was entered in the Playground.
Here we use .process to verify that the results are what we expect. The .process command enables developers to build out their stream processing pipelines and see the results of the stream in the console versus having to write it out to a collection or other stream processing source.
Next, we can create the stream processor with the createStreamProcessor command. This will add the metadata of the stream processor to the Atlas Stream Processing back end. Once created, users will need to issue a .start() to start the processor or launch “Start Stream Processor” from the side bar menu, as shown in Figure 11.
Some things to note: The MongoDB extension for VS Code supports auto-complete. Simply start the “sp.” and hit Tab to get a list of available functions.
In addition, you can also launch the MongoDB Shell right from the context menu of the Stream Processing instance. This command leverages the credentials used in the connection string, offering quick access to the MongoDB Shell in the VS Code Terminal.
Support for Atlas Stream Processing begins with version 1.15 of the MongoDB extension for VS Code. The extension provides basic functionality to connect, build, and test stream processors. Over time, additional features and functionality will become available. Learn more about Atlas Stream Processing or visit the online documentation.
Top Comments in Forums
There are no comments on this article yet.