For AI agents: a documentation index is available at https://www.mongodb.com/docs/llms.txt — markdown versions of all pages are available by appending .md to any URL path.

$iceberg Aggregation Stage

The $iceberg stage specifies a connection in the Connection Registry to an AWS S3 bucket or an Azure Blob Storage container where you can write data to an Apache Iceberg table.

$iceberg must be the last stage of any pipeline in which it appears. You can use only one $iceberg stage per pipeline.

The $iceberg pipeline stage has the following prototype form:

{ "$iceberg": { "connectionName": "<registered-connection>", "bucket": "<target-bucket>", "container": "<target-container>", "databaseName": "<database>", "tableName": "<string>" | <expression>, "path": "<key-prefix>", "region": "<target-region>", "mode": "cdc" | "insert", "idFieldName": "<field-name>", "partitionedBy": { "<column-name>": "<partition-transform>", . . . }, "catalog": { "type": "hadoop" | "glue" | "unity" | "horizon", "connectionName": "<data-catalog-connection>", "warehouse": "<warehouse>" }, "schemaInference": { "mode": "json" | "nested" } }}

The $iceberg stage takes a document with the following fields:

Field
Type
Necessity
Description

connectionName

string

Required

Name of the AWS S3 or Azure Blob Storage connection to use for reads and writes. This must match the name of the connection in the Connection Registry.

bucket

string

Conditional

Required for S3 connections.

Name of the S3 bucket containing the target Apache Iceberg database. If you set bucket, you must set region, but don't set container.

container

string

Conditional

Required for Azure Blob Storage connections.

Name of the Azure Blob Storage container containing the target Apache Iceberg database. If you set container, don't set bucket or region.

databaseName

string

Required

Name of the Apache Iceberg database containing the target table.

tableName

string | expression

Required

Name of the target Apache Iceberg table. Must be either a string or an expression that evaluates to a string. Use expressions for dynamic routing per-document.

path

string

Required

Prefix key of the path to your Apache Iceberg database.

region

string

Conditional

AWS region of the bucket. Required for stream processors not running on AWS.

Does not apply to Azure Blob Storage connections.

mode

string

Optional

Strategy for determining which operation to perform per input document.

  • "cdc" causes Atlas Stream Processing to determine operation type by reading the stream.source.operationType metadata field.

  • "insert" causes Atlas Stream Processing to append each document to the target table as a new row, ignoring operation type declarations in the stream.source.operationType metadata field.

Defaults to cdc. The "unity" catalog type doesn't support cdc mode. If catalog.type is "unity", set mode to "insert".

idFieldName

string

Optional

Field and column name used as the row key in cdc mode.

Defaults to "_id".

partitionedBy

document

Optional

Partitioning specification. If you don't set this field, $iceberg sets a default partition transform for the idFieldName column.

Must be a document containing between one and three key-value pairs. Each key must be the name of a column against which to perform a partition transform, and each value must be the partition transform to use. The first partition transform must be against idFieldName.

To partition on a field nested inside a struct column, specify the dot-qualified path to the field, for example outer.inner.key_field. The path can't include a list column at any level.

The partition transform value of a given field must be one of the following:

  • "identity"

  • "year"

  • "month"

  • "day"

  • "hour"

  • { truncate: int }

  • { bucket: int }

For more information about Apache Iceberg partition transforms, see the Apache Iceberg documentation.

catalog

document

Optional

Document defining the Iceberg catalog to use. Must be a document containing the type field with a value of "hadoop", "glue", "unity", or "horizon". The "unity" and "horizon" catalog types also require the connectionName and warehouse fields. To learn more, see catalog Fields.

The "unity", "horizon", and "hadoop" catalog types support both S3 and Azure Blob Storage connections.

The "glue" catalog type supports only S3 connections.

schemaInference

document

Optional

Document that configures how $iceberg infers the table schema. Must be a document containing the mode field with one of the following values:

  • "json" causes Atlas Stream Processing to write object and array fields as JSON string columns.

  • "nested" causes Atlas Stream Processing to write object fields as struct columns and array fields as list columns.

Defaults to "json". To learn more, see Type Conversion.

The catalog document takes the following fields:

Field
Type
Necessity
Description

type

string

Optional

Type of Apache Iceberg catalog to use. Must be one of the following values:

  • "hadoop" for data catalogs in AWS and Azure

  • "glue" for data catalogs in AWS

  • "unity" for Databricks Unity catalogs in AWS and Azure

  • "horizon" for Snowflake Horizon catalogs in AWS and Azure

connectionName

string

Conditional

Name of the data catalog connection to use. This must match the name of a data catalog connection in the Connection Registry.

Required if type is "unity" or "horizon".

Atlas Stream Processing rejects this field if type is "hadoop" or "glue".

warehouse

string

Conditional

Location in the catalog that contains the target schema.

Required if type is "unity" or "horizon".

Atlas Stream Processing rejects this field if type is "hadoop" or "glue".

  • For "unity", the name of the Databricks catalog.

  • For "horizon", the name of the Snowflake database.

For "unity" and "horizon" catalogs, $iceberg resolves the target table from the warehouse, databaseName, and tableName fields:

Catalog Type
warehouse
databaseName
tableName

"unity"

Databricks catalog

Databricks schema

Table name

"horizon"

Snowflake database

Snowflake schema

Table name

Note

If Atlas Stream Processing can't find the warehouse or schema, confirm that you specified the warehouse and databaseName values in the correct fields. Snowflake database names are usually uppercase.

If you use the $iceberg stage, it must be the last stage in your stream processor.

Atlas Stream Processing supports the $iceberg stage for only SP10, SP30, and SP50 stream processors. Your processor tier determines the maximum number of tables supported for dynamic routing:

Tier
Max Tables

SP10

5

SP30

10

SP50

50

Important

If a stream processor with dynamic routing exceeds the maximum number of tables supported at its tier, the processor enters a FAILED state. To learn about failure causes and recovery, see Error Handling and Retry Policies.

The $iceberg stage infers the schema of the resulting Apache Iceberg table from the schema of the stream processor's output data. As Atlas Stream Processing observes new fields in the stream, including fields within struct columns, the table schema evolves accordingly.

If you specify a table that doesn't already exist, Apache Iceberg creates the table when it receives the first message that targets it.

Atlas Stream Processing guarantees at-least-once processing for output to Apache Iceberg tables.

You can use a dynamic expression as the value of the tableName field. By using a dynamic expression to capture document-specific values, you can route input documents to different tables according to these values. The expression must evaluate to a string. For an example, see Dynamic Routing. To learn more see expression operators.

If you specify a topic with a dynamic expression, but Atlas Stream Processing cannot evaluate the expression for a given message, Atlas Stream Processing sends that message to the dead letter queue, if it's configured, and processes subsequent messages. If there's no dead letter queue configured, then Atlas Stream Processing skips the message completely and processes subsequent messages.

Atlas Stream Processing performs type conversion from BSON to Iceberg primitive types when writing to tables in the $iceberg stage.

BSON
Apache Iceberg Primitive
Details

string

string

int

int

long

long

double

double

bool

boolean

ObjectId

string

Hexadecimal-encoded

UUID

string

Stringified UUID

BinData

binary

Doesn't apply to UUID

date

timestamptz

UTC time, measured to the microsecond

timestamp

timestamptz

UTC time, measure to the microsecond

object

string or struct

Serialized as a Basic JSON string by default. In nested mode, becomes a struct column. To learn more, see Object and Array Fields.

array

string or list

Serialized as a Basic JSON string by default. In nested mode, becomes a list column. To learn more, see Object and Array Fields.

Other BSON types are unsupported. Atlas Stream Processing sends documents with unsupported BSON types to the DLQ.

The schemaInference.mode field determines how Atlas Stream Processing writes object and array fields to Apache Iceberg tables:

  • In json mode, Atlas Stream Processing serializes object and array fields as Basic JSON strings and writes them to string columns. json is the default mode.

  • In nested mode, Atlas Stream Processing infers Iceberg nested types from object and array fields:

    • Each object field becomes a struct column. Atlas Stream Processing infers the type of each field in the struct from the corresponding field in the object. This applies at every level of nesting.

    • Each array field becomes a list column. Atlas Stream Processing infers the list element type from the non-null elements of the array. Primitive elements must all be the same type. If the elements are objects, the element type is a struct that contains the fields of all the objects.

The mode applies only to fields that don't have a matching column in the target table. If the column already exists, Atlas Stream Processing ignores the mode and writes the field based on the column type:

  • If the column is a string, Atlas Stream Processing serializes the field as a Basic JSON string.

  • If the column is a struct or list, Atlas Stream Processing writes the field as that nested type.

If the input contains new object or array fields, Atlas Stream Processing uses the mode to infer the types of the new columns.

The follow examples illustrate various applications of the $iceberg stage.

The following example demonstrates how to write the initial contents and changestream of an Atlas database to an Apache Iceberg table in an append-only fashion, creating a durable archive of that database's operational history. This aggregation has two stages:

  1. The $source stage establishes a connection with the Atlas database, specifically targeting the orders collection in the db database. It enables initial sync to capture documents in the database at the time of processor activation, and ensures that the full document is captured by each changestream event.

  2. The $iceberg stage establishes a connection to the AWS S3 bucket, writing to a table named myTable in the iceberg-warehouse/ path. By specifying only insert operations, it ensures an append-only, log-style write flow.

{ "$source": { "connectionName": "atlas1", "db": "db", "coll": "orders", "initialSync": { "enable": true }, "config": { "fullDocument": "required" } }, "$iceberg": { "connectionName": "myS3Connection", "bucket": "myData", "path": "iceberg-warehouse/", "tableName": "myTable", "mode": "insert" }}

The following example demonstrates how to mirror an Atlas collection in its entirety to an Apache Iceberg table.

Before defining the aggregation, set the following variable:

const isDeleteExpr = {$eq: [{$meta: "stream.source.operationType"}, "delete"]};

The following aggregation adds, updates, and deletes Apache Iceberg table entries in sync with changes to the Atlas source collection. It has four stages:

  1. The $source stage establishes a connection with the Atlas database, specifically targeting the orders collection in the db database. It enables initial sync to capture documents in the database at the time of processor activation, and ensures that the full document is captured by each changestream event.

  2. The $match stage filters on operationType so that only documents with valid operation type declarations are processed.

  3. The $replaceRoot stage changes the document root depending on the type of operation.

    • For delete operations, it changes the document root to the document's key. This results in a record that the document was deleted but excludes its content from further processing.

    • For all other operations, it changes the document root to the fullDocument, passing the content of the document along for further processing while excluding changestream metadata.

  4. The $iceberg stage establishes a connection to the AWS S3 bucket, writing to an Apache Iceberg table named myTable in the iceberg-warehouse/ path. In cdc mode, this stage determines the operation to perform against the Apache Iceberg table by reading from the stream.source.operationType metadata field of each document.

{ "$source": { "connectionName": "atlas1", "db": "db", "coll": "orders", "initialSync": { "enable": true }, "config": { "fullDocument": "required" } }, "$match": { "operationType": { "$in": ["insert", "update", "delete", "replace"] } }, "$replaceRoot": { "newRoot": { "$cond": { "if": isDeleteExpr, "then": "$documentKey", "else": "$fullDocument" } } } "$iceberg": { "connectionName": "myS3Connection", "bucket": "myData", "path": "iceberg-warehouse/", "tableName": "myTable", "mode": "cdc" }}

In the following example, Atlas Stream Processing uses dynamic expressions to dynamically route documents to various output targets.

  1. The $source stage establishes a connection with the Atlas database, specifically targeting the collections a, b, and c in the db database. It enables initial sync to capture documents in the database at the time of processor activation, and ensures that the full document is captured by each changestream event.

  2. The $match stage filters for documents whose operationType is one of "insert", "update", "delete", or "replace".

  3. The $replaceRoot stage changes the document root depending on the type of operation.

    • For delete operations, it changes the document root to the document's key. This results in a record that the document was deleted but excludes its content from further processing.

    • For all other operations, it changes the document root to the fullDocument, passing the content of the document along for further processing while excluding changestream metadata.

  4. The $iceberg stage establishes a connection to the AWS S3 bucket named myData, writing to an Apache Iceberg table in the iceberg-warehouse/ path. It determines the name of the table according to the name of the source collection as retrieved from document metadata. It also determines the operation to perform according to document metadata.

{ "$source": { "connectionName": "atlas1", "db": "db", "coll": ["a", "b", "c"], "initialSync": { "enable": true }, "config": { "fullDocument": "required" } }},{ "$match": { "operationType": { "$in": ["insert", "update", "delete", "replace"] } }},{ "$replaceRoot": { "newRoot": { "$cond": { "if": { "$eq": [{ "$meta": "stream.source.operationType" }, "delete"] }, "then": "$documentKey", "else": "$fullDocument" } } }},{ "$iceberg": { "connectionName": "myS3Connection", "databaseName": "iceberg-db", "bucket": "myData", "path": "iceberg-warehouse/", "tableName": { "$meta": "stream.source.ns.coll" }, "mode": "cdc" }}

The following example demonstrates how to write the initial contents and changestream of an Atlas collection to an Apache Iceberg table that a data catalog manages. Select the tab for your catalog type. The example sets mode to insert to append each document to the table as a new row. This aggregation has two stages:

  1. The $source stage establishes a connection with the Atlas database, specifically targeting the orders collection in the db database. It enables initial sync to capture documents in the database at the time of processor activation, and ensures that the full document is captured by each changestream event.

  2. The $iceberg stage writes to the AWS S3 bucket named myData through the myS3Connection connection. It uses a data catalog connection to write to the orders table of the catalog.

The $iceberg stage uses the myUnityCatalog data catalog connection to write to the orders table in the sales schema of the Databricks main catalog. The unity catalog type doesn't support cdc mode, so you must set mode to insert.

[ { "$source": { "connectionName": "atlas1", "db": "db", "coll": "orders", "initialSync": { "enable": true }, "config": { "fullDocument": "required" } } }, { "$iceberg": { "connectionName": "myS3Connection", "bucket": "myData", "path": "iceberg-warehouse/", "databaseName": "sales", "tableName": "orders", "mode": "insert", "catalog": { "type": "unity", "connectionName": "myUnityCatalog", "warehouse": "main" } } }]