> This page location: /docs.md > /docs/development.md > /docs/atlas/atlas-stream-processing.md > /docs/atlas/atlas-stream-processing/stream-aggregation-stages.md

> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt

# `$emit` Aggregation Stage (Stream Processing)

## Definition

The `$emit` stage specifies a connection in the [Connection Registry](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-connection-registry.md#std-label-atlas-sp-manage-connections) to emit messages to. The following connection types are supported:

- [Apache Kafka](https://kafka.apache.org/) broker

- MongoDB [time series collection](https://www.mongodb.com/docs/manual/core/timeseries-collections/)

- AWS (Amazon Web Services) Kinesis data stream

- AWS (Amazon Web Services) S3 (Simple Storage Service) bucket

- Azure (Microsoft Azure) Blob Storage

- Google Cloud Pub/Sub

## Placement

`$emit` must be the last stage of any pipeline it appears in. You can use only one `$emit` stage per pipeline.

## Syntax

### Apache Kafka Broker

To write processed data to an [Apache Kafka](https://kafka.apache.org/) broker, use the `$emit` pipeline stage with the following prototype form:

```json
{
  "$emit": {
    "connectionName": "<registered-connection>",
    "topic": "<target-topic>" | <expression>,
    "schemaRegistry": {
      "connectionName": "<schema-registry-name>",
      "valueSchema": {
        type: "<schema-type>",
        schema: <schema-name>,
        options: {
          subjectNameStrategy: "<topic-name-strategy>",
          autoRegisterSchemas: true
        }
      }
    },
    "config": {
      "acks": <number-of-acknowledgements>,
      "compression_type": "<compression-type>",
      "dateFormat": "default" | "ISO8601",
      "headers": "<expression>",
      "key": "<key-string>" | { key-document },
      "keyFormat": "<serialization-type>",
      "outputFormat": "basicJson" | "canonicalJson" | "relaxedJson",
      "tombstoneWhen": <expression>
    }
  }
}
```

The `$emit` stage takes a document with the following fields:

| Field | Type | Necessity | Description |
| --- | --- | --- | --- |
| `connectionName` | string | Required | Name, as it appears in the [Connection Registry](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-connection-registry.md#std-label-atlas-sp-manage-connections), of the connection to ingest data from. |
| `topic` | string or expression | Required | Name of the [Apache Kafka](https://kafka.apache.org/) topic to emit messages to. |
| `schemaRegistry` | document | Optional | Document enabling the use of a [Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html) to support writing to an [Avro-serialized](https://avro.apache.org/docs/) source. To enable this feature, you must create a [Schema Registry connection.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/add-sp-connection.md#std-label-atlas-sp-add-connection) |
| `schemaRegistry.connectionName` | string | Conditional | Name of the Schema Registry connection to use for Avro deserialization. |
| `schemaRegistry.valueSchema` | document \| expression | Conditional | Document defining the properties of your serialization schema, or an expression that evaluates to such. |
| `schemaRegistry.valueSchema.type` | string | Conditional | The type of serialization for which to use the Schema Registry. Atlas Stream Processing currently supports `"avro"` serialization through Schema Registry connections. |
| `schemaRegistry.valueSchema.schema` | document | Conditional | Document defining your [Schema Declaration.](https://avro.apache.org/docs/1.12.0/specification/#schema-declaration) |
| `schemaRegistry.valueSchema.options` | document | Optional | Document defining optional configuration parameters for your schema registry connection. |
| `schemaRegistry.valueSchema.options.autoRegisterSchemas` | boolean | Optional | Toggle determining whether to automatically register schemas when processing documents with unrecognized schemas. If set to false, documents with unrecognized schemas are sent to the [dead letter queue.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/architecture.md#std-label-atlas-sp-dlq) Defaults to `true`. |
| `schemaRegistry.valueSchema.options.subjectNameStrategy` | string | Conditional | Method of determining the subject name of automatically registered schemas. Must be one of the following: `"TopicNameStrategy"`: Uses the Kafka `{topic}` name as the subject name.; `"RecordNameStrategy"`: Uses the Avro record name as the subject name.; `"TopicRecordNameStrategy"`: Uses a composite of the Kafka `{topic}` name and Avro record name as the subject name. Defaults to `"TopicNameStrategy"`. You can set this parameter only if `schemaRegistry.valueSchema.options.autoRegisterSchemas` is set to `true`. |
| `config` | document | Optional | Document containing fields that override various default values. |
| `config.acks` | string | Optional | Number of acknowledgements required from the [Apache Kafka](https://kafka.apache.org/) cluster for a successful `$emit` operation. The default value is `"all"`. Atlas Stream Processing supports the following values: `"-1"`; `"0"`; `"1"`; `"all"` |
| `config.compression_type` | string | Optional | Compression type for all data generated by the producer. The default is none (i.e no compression). Valid values are: `none`; `gzip`; `snappy`; `lz4`; `zstd` Compression is used for full batches of data, so the efficacy of batching impacts the compression ratio; more batching results in better compression. |
| `config.dateFormat` | string | Optional | Date format for the date value. Valid values are: `default` - to use the default of the `outputFormat`.; `ISO8601` - to convert dates to strings in the `ISO8601` format, which includes millisecond precision (`YYYY-MM-DDTHH:mm:ss.sssZ`). For example: Consider the following input. `{ "flightTime" :  ISODate('2025-01-10T20:17:38.387Z') }` If `$emit.config.dateFormat` is set to `default`, output looks similar to the following: `{ "flightTime" :  {$date :"2025-01-10T20:17:38.387Z"}}` If `$emit.config.dateFormat` is set to `ISO8601`, output looks similar to the following: `{ "flightTime" :  "2025-01-10T20:17:38.387Z" }` |
| `config.headers` | expression | Optional | Headers to add to the output message. You must specify `config.headers` as an aggregation expression or a field reference. You can't supply a raw inline object that contains dynamic expressions, or a raw array literal, as the value of `config.headers`. At runtime, the expression must evaluate to either the complete headers object or an array of `{ k, v }` documents. If the expression evaluates to an object, Atlas Stream Processing constructs a header from each key-value pair in that object, where the key is the header name, and the value is the header value. If the expression evaluates to an array, it must take the form of an array of key-value pair objects. For example: `[
  {k: "name1", v: ...},
  {k: "name2", v: ...},
  {k: "name3", v: ...}
]` Atlas Stream Processing constructs a header from each object in the array, where the key is the header name, and the value is the header value. Atlas Stream Processing supports header values of the following types: `binData`; `string`; `object`; `int`; `long`; `double`; `null` To learn how to build headers from field values or stream metadata, see [Dynamic Kafka Headers.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-sp-emit-dynamic-headers) |
| `config.key` | object \| string \| expression | Optional | Expression that evaluates to a [Apache Kafka](https://kafka.apache.org/) message key. If you specify `config.key`, you must specify `config.keyFormat`. |
| `config.keyFormat` | string | Conditional | Data type used to serialize [Apache Kafka](https://kafka.apache.org/) key data. Must be one of the following values: `"binData"`; `"string"`; `"json"`; `"int"`; `"long"`; `"double"` Defaults to `binData`. If you specify `config.key`, you must specify `config.keyFormat`. If the `config.key` of a document does not serialize successfully to the specified data type, Atlas Stream Processing sends it to your [dead letter queue.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/architecture.md#std-label-atlas-sp-dlq) |
| `config.outputFormat` | string | Optional | JSON format to use when emitting messages to [Apache Kafka](https://kafka.apache.org/). Must be one of the following values: `"basicJson"`; `"canonicalJson"`; `"relaxedJson"` Defaults to `"relaxedJson"`. To learn more, see [Basic JSON.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit-basic-json) |
| `config.tombstoneWhen` | expression | Optional | Expression that determines when to emit `null` to Kafka. The expression must evaluate to either boolean `true` or `false`. When the expression evaluates to `true` for a given document, Atlas Stream Processing emits a `null` in its place to your Kafka sink. When the expression evaluates to false, Atlas Stream Processing emits the document as it exists when it reaches the `$emit` stage. If the expression fails to evaluate to a boolean value, or can't be evaluated, Atlas Stream Processing writes the document to the [DLQ.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/architecture.md#std-label-atlas-sp-dlq) This setting can be used to enable topic compaction if you provide `$emit.config.key` and `$emit.config.keyFormat` values. If you don't provide these values, Atlas Stream Processing still emits `null` when this expression evaluates to `true`, but these don't trigger Kafka topic compaction. |

### Atlas Time Series Collection

To write processed data to an Atlas time series collection, use the `$emit` pipeline stage with the following prototype form:

```json
{
  "$emit": {
    "connectionName": "<registered-connection>",
    "db": "<target-db>" | <expression>,
    "coll": "<target-coll>" | <expression>,
    "parallelism": <number>,
    "timeseries": {
      <options>
    }
  }
}
```

The `$emit` stage takes a document with the following fields:

| Field | Type | Necessity | Description |
| --- | --- | --- | --- |
| `connectionName` | string | Required | Name, as it appears in the [Connection Registry](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-connection-registry.md#std-label-atlas-sp-manage-connections), of the connection to ingest data from. |
| `db` | string \| expression | Required | Name of, or expression that resolves to the Atlas database that contains the target time series collection. |
| `coll` | string \| expression | Required | Name of, or expression that resolves to the Atlas time series collection to write to. |
| `parallelism` | integer | Optional | Number of internal write threads that the timeseries sink operator uses to write data to the time series collection. This improves throughput for high-volume workloads by distributing write operations across multiple threads. Defaults to `1`. The maximum value you can set depends on the tier of your stream processor. To learn more, see [Atlas Stream Processing Tier Selection Guide.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/tier-guide.md#std-label-atlas-sp-tier-guide) Each stream processor has a maximum cumulative parallelism value determined by its tier. The cumulative parallelism of a stream processor is calculated as follows: `parallelism total - parallelized stages` Where `parallelism total` is the sum of all `parallelism` values greater than 1 across the [`$source`](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#mongodb-pipeline-pipe.-source), [$lookup](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-lookup.md#std-label-atlas-sp-agg-lookup), [$merge](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-merge.md#std-label-atlas-sp-agg-merge), [$emit](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit), and [$externalFunction](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-externalfunction.md#std-label-atlas-sp-agg-external-function) stages, and `parallelized stages` is the number of these stages with `parallelism` values greater than `1`. For example, if your `$source` stage sets a `parallelism` value of `4`, your `$lookup` stage sets no `parallelism` value (thus defaulting to `1`), and your `$merge` stage sets a `parallelism` value of `2`, then you have two `parallelized stages`, and the cumulative parallelism of your stream processor is calculated as `(4 + 2) - 2`. If a stream processor exceeds the maximum cumulative parallelism for its tier, Atlas Stream Processing throws an error and advises you of the minimum processor tier required for your intended level of parallelism. You must either scale the processor up to a higher tier or lower the parallelism values of your stages to resolve the error. To learn more, see [Stream Processing.](https://www.mongodb.com/docs/atlas/billing/stream-processing-costs.md#std-label-stream-processing-costs) |
| `timeseries` | document | Optional | Document defining the [time series fields](https://www.mongodb.com/docs/manual/core/timeseries/timeseries-procedures/#time-series-field-reference) for the collection. |

**Note:**

The maximum size for documents within a time series collection is 4 MB. To learn more, see [Time Series Collection Limitations.](https://www.mongodb.com/docs/manual/core/timeseries/timeseries-limitations.md#std-label-manual-timeseries-collection-limitations)

**Important:**

**Dynamic Expression Requirements**

When using expressions for the `db` or `coll` fields, you must specify the `timeField` option in the `timeseries` configuration. This ensures that Atlas Stream Processing can properly create time series collections when the target collection does not exist.

For example:

```json
{
  "$emit": {
    "connectionName": "atlas1",
    "db": "$targetDatabase",
    "coll": "$targetCollection",
    "timeseries": {
      "timeField": "timestamp"
    }
  }
}
```

### AWS (Amazon Web Services) Kinesis

Atlas Stream Processing supports creating Private Link connections to AWS Kinesis streams. To learn more, see [Add a Kinesis Private Link Connection.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/kinesis-private-connection.md#std-label-atlas-sp-pl-kinesis-add)

To write processed data to AWS (Amazon Web Services) Kinesis, use the `$emit` pipeline stage with the following prototype form:

```json
{
  "$emit": {
    "connectionName": "<registered-connection>",
    "stream": "<stream-name>",
    "region": "<aws-region>",
    "partitionKey": "<key>" | <field> | <expression>
    "config": {
      "outputFormat": "basicJson" | "canonicalJson" | "relaxedJson",
      "dateFormat": "default" | "ISO8601",
    }
  }
}
```

The `$emit` stage takes a document with the following fields:

| Field | Type | Necessity | Description |
| --- | --- | --- | --- |
| `connectionName` | string | Required | Name, as it appears in the [Connection Registry](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-connection-registry.md#std-label-atlas-sp-manage-connections), of the connection to ingest data from. |
| `stream` | string | Required | Name of the Kinesis data stream to which to connect. |
| `region` | string | Optional | Region in which the Kinesis Data Stream operates. AWS (Amazon Web Services) supports multiple streams with the same name, each in a distinct region. This parameter allows Atlas Stream Processing to distinguish between such streams. |
| `partitionKey` | string \| expression | Required | Key used to determine which Kinesis shard receives each record. Must be a literal string or an expression that evaluates to a string. |
| `config` | document | Optional | Document containing fields that override various default values. |
| `config.outputFormat` | string | Optional | JSON format to use when emitting messages to Kinesis. Must be one of the following values: `"basicJson"`; `"canonicalJson"`; `"relaxedJson"` Defaults to `"relaxedJson"`. To learn more, see [Basic JSON.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit-basic-json) |
| `config.dateFormat` | string | Optional | Date format for the date value. Valid values are: `default` - to use the default of the `outputFormat`.; `ISO8601` - to convert dates to strings in the `ISO8601` format, which includes millisecond precision (`YYYY-MM-DDTHH:mm:ss.sssZ`). For example: Consider the following input. `{ "flightTime" :  ISODate('2025-01-10T20:17:38.387Z') }` If `$emit.config.dateFormat` is set to `default`, output looks similar to the following: `{ "flightTime" :  {$date :"2025-01-10T20:17:38.387Z"}}` If `$emit.config.dateFormat` is set to `ISO8601`, output looks similar to the following: `{ "flightTime" :  "2025-01-10T20:17:38.387Z" }` |

### AWS (Amazon Web Services) S3

To write processed data to an AWS (Amazon Web Services) S3 (Simple Storage Service) bucket sink connection, use the `$emit` pipeline stage with the following prototype form:

```json
{
  "$emit": {
    "connectionName": "<registered-connection>",
    "bucket": "<target-bucket>" | <expression>,
    "region": "<target-region>",
    "path": "<key-prefix>" | <expression>,
    "parallelism": <number>,
    "partitionBy": <expression>,
    "config": {
      "writeOptions": {
        "count": <doc-count>,
        "bytes": <threshold>,
        "interval": {
          "size": <unit-count>,
          "unit": "<time-denomination>"
        }
      },
      "delimiter": "<delimiter>",
      "outputFormat": "basicJson" | "canonicalJson" | "relaxedJson",
      "dateFormat": "default" | "ISO8601",
      "compression": "gzip" | "snappy",
      "compressionLevel": <level>
    }
  }
}
```

The `$emit` stage takes a document with the following fields:

| Field | Type | Necessity | Description |
| --- | --- | --- | --- |
| `connectionName` | string | Required | Name, as it appears in the [Connection Registry](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-connection-registry.md#std-label-atlas-sp-manage-connections), of the connection to write data to. |
| `bucket` | string \| expression | Required | Name of the S3 (Simple Storage Service) bucket to which to write data, specified as a literal string or an expression that evaluates to a string. |
| `region` | string | Optional | Name of the AWS (Amazon Web Services) region in which the target bucket resides. If you host your stream processing workspace in an AWS (Amazon Web Services) region, this parameter defaults to that region. Otherwise, it defaults to the AWS (Amazon Web Services) region nearest your stream processing workspace host region. |
| `path` | string \| expression | Required | Prefix of the key of objects written to the S3 (Simple Storage Service) bucket. Must either be a literal [prefix](http://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) string or an expression that evaluates to a string. |
| `parallelism` | integer | Optional | Number of internal write threads that the S3 sink operator uses to write data to S3. This improves throughput for high-volume workloads by distributing write operations across multiple threads. Defaults to `1`. The maximum value you can set depends on the tier of your stream processor. To learn more, see [Atlas Stream Processing Tier Selection Guide.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/tier-guide.md#std-label-atlas-sp-tier-guide) Each stream processor has a maximum cumulative parallelism value determined by its tier. The cumulative parallelism of a stream processor is calculated as follows: `parallelism total - parallelized stages` Where `parallelism total` is the sum of all `parallelism` values greater than 1 across the [`$source`](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#mongodb-pipeline-pipe.-source), [$lookup](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-lookup.md#std-label-atlas-sp-agg-lookup), [$merge](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-merge.md#std-label-atlas-sp-agg-merge), [$emit](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit), and [$externalFunction](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-externalfunction.md#std-label-atlas-sp-agg-external-function) stages, and `parallelized stages` is the number of these stages with `parallelism` values greater than `1`. For example, if your `$source` stage sets a `parallelism` value of `4`, your `$lookup` stage sets no `parallelism` value (thus defaulting to `1`), and your `$merge` stage sets a `parallelism` value of `2`, then you have two `parallelized stages`, and the cumulative parallelism of your stream processor is calculated as `(4 + 2) - 2`. If a stream processor exceeds the maximum cumulative parallelism for its tier, Atlas Stream Processing throws an error and advises you of the minimum processor tier required for your intended level of parallelism. You must either scale the processor up to a higher tier or lower the parallelism values of your stages to resolve the error. To learn more, see [Stream Processing.](https://www.mongodb.com/docs/atlas/billing/stream-processing-costs.md#std-label-stream-processing-costs) |
| `partitionBy` | expression | Optional | Expression that determines how documents are partitioned across write threads when `parallelism` is greater than `1`. The hash of the evaluated expression determines which thread processes each document. If `parallelism` is greater than `1` but `partitionBy` is not specified, Atlas Stream Processing uses a round-robin strategy to assign documents to threads. |
| `config` | document | Optional | Document containing additional parameters that override various default values. |
| `config.writeOptions` | document | Optional | Document containing additional parameters governing write behavior. These parameters trigger write behavior according to which threshold is met first. For example, if the ingested documents reach the `config.writeOptions.count` threshold without reaching the `config.writeOptions.interval` threshold, the stream processor still emits these documents to S3 (Simple Storage Service) according to the `config.writeOptions.count` threshold. |
| `config.writeOptions.count` | integer | Optional | Number of documents to group into each file written to S3 (Simple Storage Service). |
| `config.writeOptions.bytes` | integer | Optional | Specifies the minimum number of bytes that must accumulate before a file is written to S3 (Simple Storage Service). The byte count is determined by the size of the BSON documents ingested by the pipeline, not the size of the final output file. Defaults to 32 MB. |
| `config.writeOptions.interval` | document | Optional | Specifies a timer for bulk writing documents as a combination of `size` and `unit`. Defaults to 1 minute. You can't set the `size` to 0 for any `unit`. The maximum interval is 7 days. |
| `config.writeOptions.interval.size` | integer | Conditional | The number of units specified by `writeOptions.interval.unit` after which the stream processor bulk writes documents to S3 (Simple Storage Service). Defaults to `1`. You can't set a `size` of 0. If you define `writeOptions.interval`, you must also define this parameter. |
| `config.writeOptions.interval.unit` | string | Conditional | The denomination of time in which to count the bulk write timer. This parameter supports the following values: `ms`; `second`; `minute`; `hour`; `day` Defaults to `minute`. If you define `writeOptions.interval`, you must also define this parameter. |
| `config.delimiter` | string | Optional | Delimiter between each entry in the emitted file. Defaults to `\n`. |
| `config.outputFormat` | string | Optional | Specifies the output format of the JSON written to S3 (Simple Storage Service). Must be one of the following values: `"basicJson"`; `"canonicalJson"`; `"relaxedJson"` Defaults to "`relaxedJson`". To learn more, see [Basic JSON](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit-basic-json) and [Extended JSON.](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json.md#std-label-mongodb-extended-json-v2) |
| `config.dateFormat` | string | Optional | Date format for the date value. Valid values are: `default` - to use the default of the `outputFormat`.; `ISO8601` - to convert dates to strings in the `ISO8601` format, which includes millisecond precision (`YYYY-MM-DDTHH:mm:ss.sssZ`). For example, if you add the following record to the pipeline: `{ "flightTime" :  ISODate('2025-01-10T20:17:38.387Z') }` then if `$emit.config.dateFormat` is set to `default`, output looks similar to the following: `{ "flightTime" :  {$date :"2025-01-10T20:17:38.387Z"}}` If `$emit.config.dateFormat` is set to `ISO8601`, output looks similar to the following: `{ "flightTime" :  "2025-01-10T20:17:38.387Z" }` |
| `config.compression` | string | Optional | Name of the compression algorithm to use. Must be one of the following values: `"none"`; `"gzip"`; `"snappy"` |
| `config.compressionLevel` | integer | Conditional | Level of compression to apply to the emitted message. Supports values `1-9` inclusive; higher values mean more compression. Defaults to `6`. This parameter is valid only when `config.compression` is set to `gzip`. Setting it with any other compression type returns an error. |

### Azure (Microsoft Azure) Blob Storage

To write processed data to Azure (Microsoft Azure) Blob Storage, use the `$emit` pipeline stage with the following prototype form:

```json
{
  "$emit": {
    "connectionName": "<registered-connection>",
    "container": "<storage-container>",
    "path": "<prefix-key>" | <expression>,
    "parallelism": <int>,
    "partitionBy": "<key>" | <expression>,
    "config": {
      "dateFormat": "default" | "ISO8601",
      "compression": "none" | "gzip" | "snappy",
      "compressionLevel": [1-9],
      "outputFormat": "relaxedJson" | "canonicalJson" | "basicJson",
      "delimiter": "<delimiter>",
      "writeOptions": {
        "count": <int>,
        "interval": {
          "size": <int>,
          "unit": "ms" | "second" | "minute" | "hour" | "day"
        },
        "bytes": <int>
      }
    }
  }
}
```

The `$emit` stage takes a document with the following fields:

| Field | Type | Necessity | Description |
| --- | --- | --- | --- |
| `connectionName` | string | Required | Name, as it appears in the [Connection Registry](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-connection-registry.md#std-label-atlas-sp-manage-connections), of the connection to which to write data. |
| `container` | string | Required | Storage container to which to write blobs. |
| `path` | string \| expression | Required | Prefix of the key of objects written to Azure (Microsoft Azure) Blob Storage. Must either be a literal string or an expression that evaluates to a string. |
| `config` | document | Optional | Document containing additional parameters that override various default values. |
| `config.dateFormat` | string | Optional | Date format for the date value. Valid values are: `default` - to use the default of the `outputFormat`.; `ISO8601` - to convert dates to strings in the `ISO8601` format, which includes millisecond precision (`YYYY-MM-DDTHH:mm:ss.sssZ`). For example, if you add the following record to the pipeline: `{ "flightTime" :  ISODate("2025-01-10T20:17:38.387Z") }` then if `$emit.config.dateFormat` is set to `default`, output looks similar to the following: `{ "flightTime" :  {$date :"2025-01-10T20:17:38.387Z"}}` If `$emit.config.dateFormat` is set to `ISO8601`, output looks similar to the following: `{ "flightTime" :  "2025-01-10T20:17:38.387Z" }` |
| `config.compression` | string | Optional | Name of the compression algorithm to use. Must be one of the following values: `"none"`; `"gzip"`; `"snappy"` If you don't specify a value for this field, Atlas Stream Processing writes data without compression. |
| `config.compressionLevel` | integer | Conditional | Level of compression to apply to the emitted message. Supports values `1-9` inclusive; higher values mean more compression. Defaults to `6`. This parameter is valid only when `config.compression` is set to `gzip`. Setting it with any other compression type returns an error. |
| `config.outputFormat` | string | Optional | Specifies the output format of the JSON written to Azure (Microsoft Azure) Blob Storage. Must be one of the following values: `"basicJson"`; `"canonicalJson"`; `"relaxedJson"` Defaults to `"relaxedJson"`. To learn more, see [Basic JSON](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit-basic-json) and [Extended JSON.](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json.md#std-label-mongodb-extended-json-v2) |
| `config.delimiter` | string | Optional | Delimiter between each entry in the emitted file. Defaults to `\n`. |
| `config.writeOptions` | document | Optional | Document containing additional parameters governing write behavior. These parameters trigger write behavior according to which threshold is met first. For example, if the ingested documents reach the `config.writeOptions.count` threshold without reaching the `config.writeOptions.interval` threshold, the stream processor still emits these documents to Azure (Microsoft Azure) Blob Storage according to the `config.writeOptions.count` threshold. |
| `config.writeOptions.count` | integer | Optional | Number of documents to group into each file written to Azure (Microsoft Azure) Blob Storage. |
| `config.writeOptions.bytes` | integer | Optional | Specifies the minimum number of bytes that must accumulate before a file is written to Azure (Microsoft Azure) Blob Storage. The byte count is determined by the size of the BSON documents ingested by the pipeline, not the size of the final output file. Defaults to 32 MB. |
| `config.writeOptions.interval` | document | Optional | Specifies a timer for bulk writing documents as a combination of `size` and `unit`. Defaults to 1 minute. You can't set the `size` to `0` for any `unit`. The maximum interval is 7 days. |
| `config.writeOptions.interval.size` | integer | Conditional | The number of units specified by `writeOptions.interval.unit` after which the stream processor bulk writes documents to Azure (Microsoft Azure) Blob Storage. Defaults to `1`. You can't set a `size` of `0`. If you define `writeOptions.interval`, you must also define this parameter. |
| `config.writeOptions.interval.unit` | string | Conditional | The denomination of time in which to count the bulk write timer. This parameter supports the following values: `ms`; `second`; `minute`; `hour`; `day` Defaults to `minute`. If you define `writeOptions.interval`, you must also define this parameter. |
| `parallelism` | integer | Optional | Number of internal write threads that Atlas Stream Processing uses to write data to Azure (Microsoft Azure) Blob Storage. This improves throughput for high-volume workloads by distributing write operations across multiple threads. Defaults to `1`. Maximum value is `16`. |
| `partitionBy` | string \| expression | Optional | String or expression that determines how Atlas Stream Processing partitions documents across write threads when `parallelism` is greater than `1`. The hash of the evaluated expression determines which thread processes each document. If `parallelism` is greater than `1` but `partitionBy` is not specified, Atlas Stream Processing uses a round-robin strategy to assign documents to threads. |

### Google Cloud Pub/Sub

To write processed data to Google Cloud Pub/Sub, use the `$emit` pipeline stage with the following prototype form:

```json
{
  "$emit": {
    "connectionName": "<registered-connection>",
    "topic": "<topic>" | <expression>,
    "projectId": "<project-id>",
    "region": "<pubsub-region>",
    "orderingKey": "<key>" | <expression>,
    "attributes": {
      "<key1>": "<value1>" | <expression>,
      . . .
      "<keyN>": "<valueN>" | <expression>
    },
    "config": {
      "dateFormat": "default" | "ISO8601",
      "outputFormat": "relaxedJson"| "canonicalJson" | "basicJson",
    }
  }
}
```

The `$emit` stage takes a document with the following fields:

| Field | Type | Necessity | Description |
| --- | --- | --- | --- |
| `connectionName` | string | Required | Name, as it appears in the [Connection Registry](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-connection-registry.md#std-label-atlas-sp-manage-connections), of the connection to which to write data. |
| `projectId` | string | Required | ID of the project to which to publish. |
| `topic` | string \| expression | Required | Name of the topic within the given project to which to publish messages. Must be a string, or an expression that evaluates to a string. |
| `region` | string | Required | The locational endpoint to which to send requests. For available regions, see the [Pub/Sub documentation.](https://docs.cloud.google.com/pubsub/docs/reference/service_apis_overview#list_locational_endpoints) |
| `orderingKey` | string \| expression | Optional | Key in published data for which to order message publication. If you set an `orderingKey`, Atlas Stream Processing ensures that the subscriber sink receives all published messages containing that key in the order in which the source publishes them. Use a string value for this field to set a static `orderingKey`. Use an expression that evaluates to a string for a dynamic `orderingKey`. An expression that evaluates specifically to a field path sets the `orderingKey` to the value of a field in a document. |
| `attributes` | document | Optional | Document defining attributes to add to messages. Attributes take the form of key value pairs where: Each key is a string or a field path expression that determines the key Atlas Stream Processing adds to messages.; Each value is either a string or a field path expression that evaluates to a string. |
| `config` | document | Optional | Document containing additional parameters that override various default values. |
| `config.dateFormat` | string | Optional | Date format for the date value. Valid values are: `default` - to use the default of the `outputFormat`.; `ISO8601` - to convert dates to strings in the `ISO8601` format, which includes millisecond precision (`YYYY-MM-DDTHH:mm:ss.sssZ`). For example, if you add the following record to the pipeline: `{ "flightTime" :  ISODate("2025-01-10T20:17:38.387Z") }` then if `$emit.config.dateFormat` is set to `default`, output looks similar to the following: `{ "flightTime" :  {$date :"2025-01-10T20:17:38.387Z"}}` If `$emit.config.dateFormat` is set to `ISO8601`, output looks similar to the following: `{ "flightTime" :  "2025-01-10T20:17:38.387Z" }` |
| `config.outputFormat` | string | Optional | Specifies the output format of the JSON written to Google Cloud Pub/Sub. Must be one of the following values: `"basicJson"`; `"canonicalJson"`; `"relaxedJson"` Defaults to `"relaxedJson"`. To learn more, see [Basic JSON](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit-basic-json) and [Extended JSON.](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json.md#std-label-mongodb-extended-json-v2) |

### Basic JSON

To ease ingestion of messages, the `$emit` stage supports writing processed data to sinks in the Basic JSON format, which simplifies the Extended (`canonicalJson`) and Relaxed Extended (`relaxedJson`) JSON formats. Basic JSON does not use MongoDB Extended JSON wrappers and therefore does not preserve all BSON types.

You can specify the Basic JSON format by setting the `config.outputFormat` field to `"basicJson"` in your `$emit` stage.

The following table provides examples of these simplifications for all affected fields.

| Field Type | relaxedJson | basicJson |
| --- | --- | --- |
| Binary | `{ "binary": { "$binary": { "base64": "gf1UcxdHTJ2HQ/EGQrO7mQ==", "subType": "00" }}}` | `{ "binary": "gf1UcxdHTJ2HQ/EGQrO7mQ=="}` |
| Date | `{ "date": { "$date": "2024-10-24T18:07:29.636Z"}}` | `{ "date": 1729625275856}` |
| Decimal | `{ "decimal": { "$numberDecimal": "9.9" }}` | `{ "decimal": "9.9" }` |
| Timestamp | `{ "timestamp": { "$timestamp": { "t": 1729793249, "i": 1 }}}` | `{ "timestamp": 1729793249000}` |
| ObjectId | `{ "_id": { "$oid": "671a8ce1497407eff0e17cba" }}` | `{ "_id": "6717fcbba18c8a8f74b6d977" }` |
| Negative Infinity | `{ "negInf": { "$numberDouble": "-Infinity" }}` | `{ "negInf": "-Infinity" }` |
| Positive Infinity | `{ "posInf": { "$numberDouble": "Infinity" }}` | `{ "posInf": "Infinity" }` |
| Regular Expressions | `{ "regex": { "$regularExpression": { "pattern": "ab+c", "options": "i" }}}` | `{ "regex": { "pattern": "ab+c", "options": "i" }}` |
| UUID | `{ "uuid": { "$binary": { "base64": "Kat+fHk6RkuAmotUmsU7gA==", "subType": "04" }}}` | `{ "uuid": "420b7ade-811a-4698-aa64-c8347c719cf1"}` |

## Behavior

You can only write to a single Atlas time series collection per stream processor. If you specify a collection that doesn't exist, Atlas creates the collection with the time series fields you specified. You must specify an existing database.

Atlas Stream Processing doesn't support writing BSON documents larger than `16MB` with the [$emit](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit) stage to an AWS (Amazon Web Services) or Azure blob storage sink. You can increase this limit to up to `125MB` if your stream processor's source isn't a MongoDB deployment, or if earlier pipeline stages such as [`$lookup`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/lookup.md#mongodb-pipeline-pipe.-lookup) increase the size of your documents.

The [Azure Blob Storage](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-sp-emit-azure-blob) `$emit` variant also supports writing to Azure (Microsoft Azure) Datalake Storage v2 containers.

The [GCP Pub/Sub](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-sp-emit-gcp-pubsub) `$emit` variant supports defining values through field path expressions for both the `orderingKey` field and the key-value pairs in the `attributes` field. On expression failure or if the expression doesn't resolve to a string, Atlas Stream Processing sends the affected message to the [dead letter queue.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/architecture.md#std-label-atlas-sp-dlq)

You can use a field or [dynamic expression](https://www.mongodb.com/docs/manual/reference/mql/expressions.md#std-label-aggregation-expressions) as the value of the `topic` field to enable your stream processor to write to different targets on a message-by-message basis. The expression must evaluate to a string.

**Example:**

You have a stream of transaction events that generates messages of the following form:

```json
{
  "customer": "Very Important Industries",
  "customerStatus": "VIP",
  "tenantId": 1,
  "transactionType": "subscription"
}

{
  "customer": "N. E. Buddy",
  "customerStatus": "employee",
  "tenantId": 5,
  "transactionType": "requisition"
}

{
  "customer": "Khan Traktor",
  "customerStatus": "contractor",
  "tenantId": 11,
  "transactionType": "billableHours"
}
```

To sort each of these into a distinct [Apache Kafka](https://kafka.apache.org/) topic, you can write the following `$emit` stage:

```json
{
  "$emit": {
    "connectionName": "kafka1",
    "topic": "$customerStatus"
  }
}
```

This `$emit` stage:

- Writes the `Very Important Industries` message to a topic named `VIP`.

- Writes the `N. E. Buddy` message to a topic named `employee`.

- Writes the `Khan Traktor` message to a topic named `contractor`.

For more information on dynamic expressions, see [expression operators.](https://www.mongodb.com/docs/manual/reference/mql/expressions.md#std-label-aggregation-expression-operators)

If you specify a topic that doesn't already exist, [Apache Kafka](https://kafka.apache.org/) automatically creates the topic when it receives the first message that targets it.

If you specify a topic with a dynamic expression, but Atlas Stream Processing cannot evaluate the expression for a given message, Atlas Stream Processing sends that message to the [dead letter queue](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-dead-letter-queue) if configured and processes subsequent messages. If there is no [dead letter queue](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-dead-letter-queue) configured, then Atlas Stream Processing skips the message completely and processes subsequent messages.

### Dynamic Kafka Headers

To construct [Apache Kafka](https://kafka.apache.org/) headers dynamically, build the complete header object in an earlier pipeline stage, then reference that object from `$emit.config.headers`. Use [$setStreamMeta](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-setstreammeta.md#std-label-atlas-sp-agg-setStreamMeta) when the headers are transport metadata that shouldn't appear in the body of the emitted message:

```json
{
  "$setStreamMeta": {
    "stream.kafka_headers": {
      "$arrayToObject": [
        {
          "k": "test_header",
          "v": "$myField"
        }
      ]
    }
  }
},
{
  "$emit": {
    "connectionName": "kafka",
    "topic": "my-topic",
    "config": {
      "headers": {
        "$meta": "stream.kafka_headers"
      }
    }
  }
}
```

## Examples

To run the examples in this section, create a stream processing workspace and add a connection to your Atlas Stream Processing workspace. To learn how, see [Get Started with Atlas Stream Processing](https://www.mongodb.com/docs/atlas/atlas-stream-processing/quickstart.md#std-label-atlas-sp-quickstart) and [Add an Atlas Stream Processing Connection.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/add-sp-connection.md#std-label-atlas-sp-add-connection)

### Write to a Kafka Topic

A streaming data source generates detailed weather reports from various locations, conformant to the schema of the [Sample Weather Dataset](https://www.mongodb.com/docs/manual/sample-data/sample-weather.md#std-label-sample-weather). The following aggregation has three stages:

1. The [`$source`](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#mongodb-pipeline-pipe.-source) stage establishes a connection with the [Apache Kafka](https://kafka.apache.org/) broker collecting these reports in a topic named `my_weatherdata`, exposing each record as it is ingested to the subsequent aggregation stages. This stage also overrides the name of the timestamp field it projects, setting it to `ingestionTime`.

2. The [`$match`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/match.md#mongodb-pipeline-pipe.-match) stage excludes documents that have an `airTemperature.value` of greater than or equal to `30.0` and passes along the documents with an `airTemperature.value` less than `30.0` to the next stage.

3. The [`$addFields`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/addFields.md#mongodb-pipeline-pipe.-addFields) stage enriches the stream with metadata.

4. The `$emit` stage writes the output to a topic named `stream` over the `weatherStreamOutput` Kafka broker connection.

```json
{
  "$source": {
    "connectionName": "sample_weatherdata",
    "topic": "my_weatherdata",
    "tsFieldName": "ingestionTime"
  }
},
{
  "$match": {
    "airTemperature.value": {
      "$lt": 30
    }
  }
},
{
  "$addFields": {
    "processorMetadata": {
      "$meta": "stream"
    }
  }
},
{
  "$emit": {
    "connectionName": "weatherStreamOutput",
    "topic": "stream"
  }
}
```

Documents in the `stream` topic take the following form:

```json
{
  "st": "x+34700+119500",
  "position": {
    "type": "Point",
    "coordinates": [122.8, 116.1]
  },
  "elevation": 9999,
  "callLetters": "6ZCM",
  "qualityControlProcess": "V020",
  "dataSource": "4",
  "type": "SAO",
  "airTemperature": {
    "value": 6.7,
    "quality": "9"
  },
  "dewPoint": {
    "value": 14.1,
    "quality": "1"
  },
  "pressure": {
    "value": 1022.2,
    "quality": "1"
  },
  "wind": {
    "direction": {
      "angle": 200,
      "quality": "9"
    },
    "type": "C",
    "speed": {
      "rate": 35,
      "quality": "1"
    }
  },
  "visibility": {
    "distance": {
      "value": 700,
      "quality": "1"
    },
    "variability": {
      "value": "N",
      "quality": "1"
    }
  },
  "skyCondition": {
    "ceilingHeight": {
      "value": 1800,
      "quality": "9",
      "determination": "9"
    },
    "cavok": "N"
  },
  "sections": ["AA1", "AG1", "UG1", "SA1", "MW1"],
  "precipitationEstimatedObservation": {
    "discrepancy": "0",
    "estimatedWaterDepth": 999
  },
  "atmosphericPressureChange": {
    "tendency": {
      "code": "4",
      "quality": "1"
    },
    "quantity3Hours": {
      "value": 3.8,
      "quality": "1"
    },
    "quantity24Hours": {
      "value": 99.9,
      "quality": "9"
    }
  },
  "seaSurfaceTemperature": {
    "value": 9.7,
    "quality": "9"
  },
  "waveMeasurement": {
    "method": "M",
    "waves": {
      "period": 8,
      "height": 3,
      "quality": "9"
    },
    "seaState": {
      "code": "00",
      "quality": "9"
    }
  },
  "pastWeatherObservationManual": {
    "atmosphericCondition": {
      "value": "6",
      "quality": "1"
    },
    "period": {
      "value": 3,
      "quality": "1"
    }
  },
  "skyConditionObservation": {
    "totalCoverage": {
      "value": "02",
      "opaque": "99",
      "quality": "9"
    },
    "lowestCloudCoverage": {
      "value": "00",
      "quality": "9"
    },
    "lowCloudGenus": {
      "value": "00",
      "quality": "1"
    },
    "lowestCloudBaseHeight": {
      "value": 1750,
      "quality": "1"
    },
    "midCloudGenus": {
      "value": "99",
      "quality": "1"
    },
    "highCloudGenus": {
      "value": "00",
      "quality": "1"
    }
  },
  "presentWeatherObservationManual": {
    "condition": "52",
    "quality": "1"
  },
  "atmosphericPressureObservation": {
    "altimeterSetting": {
      "value": 1015.9,
      "quality": "9"
    },
    "stationPressure": {
      "value": 1026,
      "quality": "1"
    }
  },
  "skyCoverLayer": {
    "coverage": {
      "value": "08",
      "quality": "1"
    },
    "baseHeight": {
      "value": 2700,
      "quality": "9"
    },
    "cloudType": {
      "value": "99",
      "quality": "9"
    }
  },
  "liquidPrecipitation": {
    "period": 12,
    "depth": 20,
    "condition": "9",
    "quality": "9"
  },
  "extremeAirTemperature": {
    "period": 99.9,
    "code": "N",
    "value": -30.4,
    "quantity": "1"
  },
  "ingestionTime": {
    "$date": "2024-09-26T17:34:41.843Z"
  }
}
```

**Note:**

The preceding is a representative example. Streaming data are not static, and each user sees distinct documents.

### Write to a Time Series Collection

The following aggregation reads reports from the sample streaming data source `sample_stream_solar` and writes them to a time series collection. This aggregation has three stages:

1. The [`$source`](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#mongodb-pipeline-pipe.-source) stage connects to the `sample_stream_solar` source.

2. The [`$set`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/set.md#mongodb-pipeline-pipe.-set) stage converts the `timestamp` field of each report from a string to a date using [`$dateFromString`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/dateFromString.md#mongodb-expression-exp.-dateFromString), so that the time series collection can use it as its `timeField`.

3. The `$emit` stage writes each report to the `solar_ts` time series collection in the `solarDb` database over the `solar-cluster-connection` connection. The `metaField` option is set to `device_id`, so reports from the same device are stored together in a bucket. If the `solar_ts` collection does not exist, Atlas Stream Processing creates it as a time series collection.

```json
{
  "$source": {
    "connectionName": "sample_stream_solar"
  }
},
{
  "$set": {
    "timestamp": {
      "$dateFromString": { "dateString": "$timestamp" }
    }
  }
},
{
  "$emit": {
    "connectionName": "solar-cluster-connection",
    "db": "solarDb",
    "coll": "solar_ts",
    "timeseries": {
      "timeField": "timestamp",
      "metaField": "device_id"
    }
  }
}
```

**Note:**

The `sample_stream_solar` source is a test-only connection that produces sample documents every second for rapid prototyping.

Documents in the `solar_ts` collection take the following form:

```json
{
  "timestamp": {
    "$date": "2026-10-22T15:07:49.299Z"
  },
  "device_id": "device_0",
  "max_watts": 450,
  "_id": {
    "$oid": "6ada26c5ad54068b03089593"
  },
  "obs": {
    "watts": 135,
    "temp": 5
  },
  "event_type": 0,
  "group_id": 8
}
```

**Note:**

The preceding is a representative example. Streaming data are not static, and each user sees distinct documents.
