Java Aggregation Expression Builders in MongoDB
Rate this tutorial
MongoDB aggregation pipelines allow developers to create rich document retrieval, manipulation, and update processes expressed as a sequence — or pipeline — of composable stages, where the output of one stage becomes the input to the next stage in the pipeline.
With aggregation operations, it is possible to:
- Group values from multiple documents together.
- Reshape documents.
- Perform aggregation operations on the grouped data to return a single result.
- Apply specialized operations to documents such as geographical functions, full text search, and time-window functions.
- Analyze data changes over time.
The aggregation framework has grown since its introduction in MongoDB version 2.2 to — as of version 6.1 — cover over 35 different stages and over 130 different operators.
Working with the MongoDB shell or in tools such as MongoDB Compass, aggregation pipelines are defined as an array of BSON[1] objects, with each object defining a stage in the pipeline. In an online-store system, a simple pipeline to find all orders placed between January 1st 2023 and March 31st 2023, and then provide a count of those orders grouped by product type, might look like:
Expressions give aggregation pipeline stages their ability to manipulate data. They come in four forms:
Operators: expressed as objects with a dollar-sign prefix followed by the name of the operator. In the example above, {$sum : 1} is an example of an operator incrementing the count of orders for each product type by 1 each time a new order for a product type is found.
Field Paths: expressed as strings with a dollar-sign prefix, followed by the field’s path. In the case of embedded objects or arrays, dot-notation can be used to provide the path to the embedded item. In the example above, "$productType" is a field path.
Variables: expressed with a double dollar-sign prefix, variables can be system or user defined. For example, "$$NOW" returns the current datetime value.
Literal Values: In the example above, the literal value ‘1’ in {$sum : 1} can be considered an expression and could be replaced with — for example — a field path expression.
In Java applications using the MongoDB native drivers, aggregation pipelines can be defined and executed by directly building equivalent BSON document objects. Our example pipeline above might look like the following when being built in Java using this approach:
The Java code above is perfectly functional and will execute as intended, but it does highlight a couple of issues:
- When creating the code, we had to understand the format of the corresponding BSON documents. We were not able to utilize IDE features such as code completion and discovery.
- Any mistakes in the formatting of the documents being created, or the parameters and data types being passed to its various operators, would not be identified until we actually try to run the code.
- Although our example above is relatively simple, in more complex pipelines, the level of indentation and nesting required in the corresponding document building code can lead to readability issues.
As an alternative to building BSON document objects, the MongoDB Java driver also defines a set of “builder” classes with static utility methods to simplify the execution of many operations in MongoDB, including the creation and execution of aggregation pipeline stages. Using the builder classes allows developers to discover more errors at compile rather than run time and to use code discovery and completion features in IDEs. Recent versions of the Java driver have additionally added extended support for expression operators when using the aggregation builder classes, allowing pipelines to be written with typesafe methods and using fluent coding patterns.
Using this approach, the above code could be written as:
In the rest of this article, we’ll walk through an example aggregation pipeline using the aggregation builder classes and methods and highlight some of the new aggregation expression operator support.
Our aggregation pipeline example is based on a database collecting and analyzing Air Traffic Control data transmitted by aircraft flying in and out of Denver International Airport. The data is collected using a receiver built using a Raspberry Pi and USB Software Defined Radios (SDRs) using software from the rather excellent Stratux open-source project.
These cheap-to-build receivers have become popular with pilots of light aircraft in recent years as it allows them to project the location of nearby aircraft within the map display of tablet and smartphone-based navigation applications such as Foreflight, helping to avoid mid-air collisions.
In our application, the data received from the Stratux receiver is combined with aircraft reference data from the Opensky Network to give us documents that look like this:
The “tailNum” field provides the unique registration number of the aircraft and doesn’t change between position reports. The position reports are in an array[2], with each entry giving the geographical coordinates of the aircraft, its altitude, speed (horizontal and vertical), heading, and a timestamp. The position reports also give the callsign of the flight the aircraft was operating at the time it broadcast the position report. This can vary if the aircraft’s position reports were picked up as it flew into Denver, and then again later as it flew out of Denver operating a different flight. In the sample above, aircraft N8620H, a Boeing 737, was operating flight SWA962 — a Southwest Airlines flight. It was flying at a speed of 283 knots, on a heading of 345 degrees, descending through 12,600 feet at 1344 ft/minute.
Using data collected over a 36-hour period, our collection contains information on over 500 different aircraft and over half a million position reports. We want to build an aggregation pipeline that will show the number of different aircraft operated by United Airlines grouped by aircraft type.
The aggregation pipeline that we will run on our data will consist of three stages:
The first — a match stage — will find all aircraft that transmitted a United Airlines callsign between two dates.
Next, we will carry out a group stage that takes the aircraft documents found by the match stage and creates a new set of documents — one for each model of aircraft found during the match stage, with each document containing a list of all the tail numbers of aircraft of that type found during the match stage.
Finally, we carry out a project stage which is used to reshape the data in each document into our final desired format.
A match stage carries out a query to filter the documents being passed to the next stage in the pipeline. A match stage is typically used as one of the first stages in the pipeline in order to keep the number of documents the pipeline has to work with — and therefore its memory footprint — to a reasonable size.
In our pipeline, the match stage will select all aircraft documents containing at least one position report with a United Airlines callsign (United callsigns all start with the three-letter prefix “UAL”), and with a timestamp between falling within a selected date range. The BSON representation of the resulting pipeline stage looks like:
The $elemMatch operator specifies that the query criteria we provide must all occur within a single entry in an array to generate a match, so an aircraft document will only match if it contains at least one position report where the callsign starts with “UAL” and the timestamp is between 12:00 on January 31st and 00:00 on February 1st in the Mountain time zone.
In Java, after using either Maven or Gradle to add the MongoDB Java drivers as a dependency within our project, we could define this stage by building an equivalent BSON document object:
As we saw with the earlier online store example, whilst this code is perfectly functional, we did need to understand the structure of the corresponding BSON document, and any mistakes we made in constructing it would only be discovered at run-time.
As an alternative, after adding the necessary import statements to give our code access to the aggregation builder and expression operator static methods, we can build an equivalent pipeline stage with the following code:
There’s a couple of things worth noting in this code:
Firstly, the expressions operators framework gives us access to a method current() which returns the document currently being processed by the aggregation pipeline. We use it initially to get the array of position reports from the current document.
Next, although we’re using the match() aggregation builder method to create our match stage, to better demonstrate the use of the expression operators framework and its associated coding style, we’ve used the expr()[3] filter builder method to build an expression that uses the any() array expression operator to iterate through each entry in the positionReports array, looking for any that matches our predicate — i.e., that has a callsign field starting with the letters “UAL” and a timestamp falling within our specified date/time range. This is equivalent to what the $elemMatch operator in our original BSON document-based pipeline stage was doing.
Also, when using the expression operators to retrieve fields, we’ve used type-specific methods to indicate the type of the expected return value. callsign was retrieved using getString(), while the timestamp variable ts was retrieved using getDate(). This allows IDEs such as IntelliJ and Visual Studio Code to perform type checking, and for subsequent code completion to be tailored to only show methods and documentation relevant to the returned type. This can lead to faster and less error-prone coding.
Finally, note that in building the predicate for the any() expression operator, we’ve used a fluent coding style and idiosyncratic coding elements, such as lambdas, that many Java developers will be familiar with and more comfortable using rather than the MongoDB-specific approach needed to directly build BSON documents.
Having filtered our document list to only include aircraft operated by United Airlines in our match stage, in the second stage of the pipeline, we carry out a group operation to begin the task of counting the number of aircraft of each model. The BSON document for this stage looks like:
In this stage, we are specifying that we want to group the document data by the “model” field and that in each resulting document, we want an array called “aircraftSet” containing each unique tail number of observed aircraft of that model type. The documents output from this stage look like:
The corresponding Java code for the stage looks like:
As before, we’ve used the expressions framework current() method to access the document currently being processed by the pipeline. The aggregation builders addToSet() accumulator method is used to ensure only unique tail numbers are added to the “aircraftSet” array.
- Rename the “_id” field introduced by the group stage back to “model.”
- Swap the array of tail numbers for the number of entries in the array.
- Add a new field, “airline,” populating it with the literal value “United.”
- Add a field named “manufacturer” and use a $cond conditional operator to populate it with:
- “AIRBUS” if the aircraft model starts with “A.”
- “BOEING” if it starts with a “B.”
- “CANADAIR” if it starts with a “C.”
- “EMBRAER” if it starts with an “E.”
- “MCDONNELL DOUGLAS” if it starts with an “M.”
- “UNKNOWN” in all other cases.
The BSON document for this stage looks like:
The resulting output documents look like:
The Java code for this stage looks like:
Note again the use of type-specific field accessor methods to get the aircraft model type (string) and aircraftSet (array of type MqlDocument). In determining the aircraft manufacturer, we’ve again used a fluent coding style to conditionally set the value to Boeing or Airbus.
With our three pipeline stages now defined, we can now run the pipeline against our collection:
If all goes to plan, this should produce output to the console that look like:
In this article, we shown examples of how expression operators and aggregation builder methods in the latest versions of the MongoDB Java drivers can be used to construct aggregation pipelines using a fluent, idiosyncratic style of Java programming that can utilize autocomplete functionality in IDEs and type-safety compiler features. This can result in code that is more robust and more familiar in style to many Java developers. The use of the builder classes also places less dependence on developers having an extensive understanding of the BSON document format for aggregation pipeline stages.
More information on the use of aggregation builder and expression operator classes can be found in the official MongoDB Java Driver documentation.
The example Java code, aggregation pipeline BSON, and a JSON export of the data used in this article can be found in Github.
More information
[1] MongoDB uses Binary JSON (BSON) to store data and define operations. BSON is a superset of JSON, stored in binary format and allowing data types over and above those defined in the JSON standard. Get more information on BSON.
[2] It should be noted that storing the position reports in an array for each aircraft like this works well for purposes of our example, but it’s probably not the best design for a production grade system as — over time — the arrays for some aircraft could become excessively large. A really good discussion of massive arrays and other anti patterns, and how to handle them, is available over at Developer Center.
[3] The use of expressions in Aggregation Pipeline Match stages can sometimes cause some confusion. For a discussion of this, and aggregations in general, Paul Done’s excellent eBook, “Practical MongoDB Aggregations,” is highly recommended.