> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt # `$out` Stage (Atlas Data Federation) [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) takes documents returned by the aggregation pipeline and writes them to a specified collection. The [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) operator must be the last [stage](https://www.mongodb.com/docs/manual/reference/mql/aggregation-stages.md#std-label-aggregation-pipeline-operator-reference) in the [aggregation pipeline](https://www.mongodb.com/docs/manual/core/aggregation-pipeline.md#std-label-aggregation-pipeline). In Atlas Data Federation, you can use [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) to write data from any one of the [supported](https://www.mongodb.com/docs/atlas/data-federation/config/config-data-stores.md#std-label-config-adf) federated database instance stores or multiple [supported](https://www.mongodb.com/docs/atlas/data-federation/config/config-data-stores.md#std-label-config-adf) federated database instance stores when using [federated queries](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md#std-label-federated-queries) to any one of the following: - Atlas cluster [namespace](https://www.mongodb.com/docs/manual/reference/limits.md#std-label-faq-dev-namespace) - AWS (Amazon Web Services) S3 (Simple Storage Service) buckets with read and write permissions - Azure (Microsoft Azure) Blob Storage containers with read and write permissions - Google Cloud Storage (GCS) You must [connect](https://www.mongodb.com/docs/atlas/data-federation/tutorial/connect.md#std-label-fdi-connect) to your federated database instance to use [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out). ### S3 ## Permissions Required ### S3 You must have: - A federated database instance configured for an S3 (Simple Storage Service) bucket with read and write permissions or [s3:PutObject](http://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-objects) permissions. - A MongoDB user with the [atlasAdmin](https://www.mongodb.com/docs/atlas/security-add-mongodb-users/#database-user-privileges) role or a custom role with the [`outToS3`](https://www.mongodb.com/docs/atlas/data-federation/config/config-data-stores.md#mongodb-authaction-outToS3) privilege. ## Syntax ### S3 ```json { "$out": { "s3": { "bucket": "", "region": "", "filename": "", "format": { "name": "", "maxFileSize": "", "maxRowGroupSize": "", "columnCompression": "" }, "errorMode": "stop"|"continue" } } } ``` ## Fields ### S3 | Field | Type | Description | Necessity | | --- | --- | --- | --- | | `s3` | object | Location to write the documents from the aggregation pipeline. | Required | | `s3.bucket` | string | Name of the S3 (Simple Storage Service) bucket to write the documents from the aggregation pipeline to. The generated call to S3 (Simple Storage Service) inserts a `/` between `s3.bucket` and `s3.filename`. Don't append a `/` to your `s3.bucket` string. For example, if you set `s3.bucket` to `myBucket` and `s3.filename` to `myPath/myData`, Atlas Data Federation writes the output location as follows: `s3://myBucket/myPath/myData.[n].json` | Required | | `s3.region` | string | Name of the AWS (Amazon Web Services) region in which the bucket is hosted. If omitted, uses the federated database instance [configuration](https://www.mongodb.com/docs/atlas/data-federation/config/config-data-stores.md#std-label-config-adf) to determine the region where the specified `s3.bucket` is hosted. | Optional | | `s3.filename` | string | Name of the file to write the documents from the aggregation pipeline to. Filename can be constant or [created dynamically](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/out.md#std-label-adf-out-stage-egs) from the fields in the documents that reach the [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) stage. Any filename expression you provide must evaluate to a `string` data type. If there are any files on S3 (Simple Storage Service) with the same name and path as the newly generated files, [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) overwrites the existing files with the newly generated files. The generated call to S3 (Simple Storage Service) inserts a `/` between `s3.bucket` and `s3.filename`. Don't prepend a `/` to your `s3.filename` string. For example, if you set `s3.filename` to `myPath/myData` and `s3.bucket` to `myBucket`, Atlas Data Federation writes the output location as follows: `s3://myBucket/myPath/myData.[n].json` | Required | | `s3.format` | object | Details of the file in S3 (Simple Storage Service). | Required | | `s3``.format``.name` | enum | Format of the file in S3 (Simple Storage Service). Value can be one of the following: `bson`; `bson.gz`; `csv`; `csv.gz`; `json` 1; `json.gz` 1; `parquet`; `tsv`; `tsv.gz` 1 For this format, [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) writes data in [MongoDB Extended JSON](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json.md#std-label-mongodb-extended-json-v2) format. To learn more, see [Limitations.](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/out.md#std-label-adf-out-stage-limitations) | Required | | `s3``.format``.maxFileSize` | bytes | Maximum size of the uncompressed BSON (Binary Javascript Object Notation) file in S3 (Simple Storage Service). When converting from BSON (Binary Javascript Object Notation) to your preferred format, the resulting output file might be smaller (for example, when converting to Parquet) or larger (for example, when converting to CSV). If a document is larger than the `maxFileSize`, Atlas Data Federation writes the document to its own file. The following suffixes are supported: Base 10, scaling in multiples of 1000: `B`, `KB`, `MB`, `GB`, `TB`, `PB`; Base 2, scaling in multiples of 1024: `KiB`, `MiB`, `GiB`, `TiB`, `PiB` If omitted, defaults to `200MiB`. When the file size limit for the current file is reached, Atlas Data Federation creates a new file in S3 (Simple Storage Service). It appends a `1` before the filename extension for the first file. For each subsequent file, Atlas Data Federation increments the appended number by one. For example, `.1.`, and `.2.`. | Optional | | `s3``.format``.maxRowGroupSize` | string | Supported for Parquet file format only. Maximum row group size to use when writing to Parquet file. If omitted, defaults to `128 MiB` or the value of the `s3.format.maxFileSize`, whichever is smaller. The maximum allowed value is `1 GB`. | Optional | | `s3``.format``.columnCompression` | string | Supported for Parquet file format only. Compression type to apply for compressing data inside a Parquet file when formatting the Parquet file. Valid values are: `gzip`; `snappy`; `uncompressed` If omitted, defaults to `snappy`. To learn more, see [Supported Data Formats.](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/supported-data-formats.md#std-label-adf-data-formats) | Optional | | `errorMode` | enum | Specifies how Atlas Data Federation should proceed if there are errors when processing a document. For example, if Atlas Data Federation encounters an array in a document when Atlas Data Federation is writing to a CSV file, Atlas Data Federation uses this value to determine whether or not to skip the document and process other documents. Valid values are: `continue` to skip the document and continue processing the remaining documents. Atlas Data Federation also writes the document that caused the error to an error file.To learn more see, [Errors.](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/out.md#std-label-adf-out-stage-errors); `stop` to stop at that point and not process the remaining documents. If omitted, defaults to `continue`. | Optional | ## Options | Option | Type | Description | Necessity | | --- | --- | --- | --- | | `background` | boolean | Flag to run aggregation operations in the background. If omitted, defaults to `false`. When set to `true`, Atlas Data Federation runs the queries in the background. `{ "background" : true }` Use this option if you want to submit other new queries without waiting for currently running queries to complete or disconnect your federated database instance connection while the queries continue to run in the background. | Optional | ## Examples ### S3 **Create a Filename** The following examples show [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) syntaxes for dynamically creating a filename from a constant string or from the fields of the same or different data types in the documents that reach the [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) stage. ### Simple String Example You want to write 1 GiB of data as compressed BSON (Binary Javascript Object Notation) files to an S3 (Simple Storage Service) bucket named `my-s3-bucket`. Using the following [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) syntax: ```json { "$out": { "s3": { "bucket": "my-s3-bucket", "filename": "big_box_store/", "format": { "name": "bson.gz" } } } } ``` The `s3.region` is omitted and so, Atlas Data Federation determines the region where the bucket named `my-s3-bucket` is hosted from the storage configuration. [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) writes five compressed BSON (Binary Javascript Object Notation) files: 1. The first 200 MiB of data to a file that [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) names `big_box_store/1.bson.gz`. - The value of `s3.filename` serves as a constant in each filename. This value doesn't depend upon any document field or value. - Your `s3.filename` ends with a delimiter, so Atlas Data Federation appends the counter after the constant. - If it didn't end with a delimiter, Atlas Data Federation would have added a `.` between the constant and the counter, like `big_box_store.1.bson.gz` - Because you didn't change the maximum file size using `s3.format.maxFileSize`, Atlas Data Federation uses the default value of 200 MiB. 2. The second 200 MiB of data to a new file that [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) names `big_box_store/2.bson.gz`. 3. Three more files that [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) names `big_box_store/3.bson.gz` through `big_box_store/5.bson.gz`. ### Single Field from Documents You want to write 90 MiB of data to JSON (Javascript Object Notation) files to an S3 (Simple Storage Service) bucket named `my-s3-bucket`. Using the following [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) syntax: ```json { "$out": { "s3": { "bucket": "my-s3-bucket", "region": "us-east-1", "filename": {"$toString": "$saleDate"}, "format": { "name": "json", "maxFileSize": "100MiB" } } } } ``` [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) writes 90 MiB of data to JSON (Javascript Object Notation) files in the root of the bucket. Each JSON (Javascript Object Notation) file contains all of the documents with the same `saleDate` value. [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) names each file using the documents' `saleDate` value converted to a string. ### Multiple Fields from Documents You want to write 176 MiB of data as BSON (Binary Javascript Object Notation) files to an S3 (Simple Storage Service) bucket named `my-s3-bucket`. Using the following [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) syntax: ```json { "$out": { "s3": { "bucket": "my-s3-bucket", "region": "us-east-1", "filename": { "$concat": [ "persons/", "$name", "/", "$uniqueId", "/" ] }, "format": { "name": "bson", "maxFileSize": "200MiB" } } } } ``` [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) writes 176 MiB of data to BSON (Binary Javascript Object Notation) files. To name each file, [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) concatenates: - A constant string `persons/` and, from the documents: - The string value of the `name` field, - A forward slash (`/`), - The string value of the `uniqueId` field, and - A forward slash (`/`). Each BSON (Binary Javascript Object Notation) file contains all of the documents with the same `name` and `uniqueId` values. [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) names each file using the documents' `name` and `uniqueId` values. ### Multiple Types of Fields from Documents You want to write 154 MiB of data as compressed JSON (Javascript Object Notation) files to an S3 (Simple Storage Service) bucket named `my-s3-bucket`. Consider the following [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) syntax: ```json { "$out": { "s3": { "bucket": "my-s3-bucket", "region": "us-east-1", "filename": { "$concat": [ "big-box-store/", { "$toString": "$storeNumber" }, "/", { "$toString": "$saleDate" }, "/", "$partId", "/" ] }, "format": { "name": "json.gz", "maxFileSize": "200MiB" } } } } ``` [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) writes 154 MiB of data to compressed JSON (Javascript Object Notation) files, where each file contains all documents with the same `storeNumber`, `saleDate`, and `partId` values. To name each file, [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) concatenates: - A constant string value of `big-box-store/`, - A string value of a unique store number in the `storeNumber` field, - A forward slash (`/`), - A string value of the date from the `saleDate` field, - A forward slash (`/`), - A string value of part ID from the `partId` field, and - A forward slash (`/`). ### Run a Query in the Background The following example shows [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) syntax for running an aggregation pipeline that ends with the [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) stage in the background. ### S3 ```json db.runCommand({ "aggregate": "my-collection", "pipeline": [ { "$out": { "s3": { "bucket": "my-s3-bucket", "filename": { "$toString": "$saleDate" }, "format": { "name": "json" } } } } ], "background": true }) ``` [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) writes to JSON (Javascript Object Notation) files in the root of the bucket in the background. Each JSON (Javascript Object Notation) file contains all of the documents with the same `saleDate` value. [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) names each file using the documents' `saleDate` value converted to a string. ## Limitations ### S3 ### String Data Type Atlas Data Federation interprets empty strings (`""`) as `null` values when parsing filenames. If you want Atlas Data Federation to generate parseable filenames, wrap the field references that could have `null` values using [`$convert`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/convert.md#mongodb-expression-exp.-convert) with an empty string `onNull` value. This example shows how to handle null values in the `year` field when creating a filename from the field value. ```json { "$out": { "s3": { "bucket": "my-s3-bucket", "region": "us-east-1", "filename": { "$concat": [ "big-box-store/", { "$convert": { "input": "$year", "to": "string", "onNull": "" } }, "/" ] }, "format": { "name": "json.gz", "maxFileSize": "200MiB" } } } } ``` ### Number of Unique Fields When writing to [CSV](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/data-formats/csv-tsv-data-files.md#std-label-adf-csv-tsv-data), [TSV](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/data-formats/csv-tsv-data-files.md#std-label-adf-csv-tsv-data), or [Parquet](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/data-formats/parquet-data-files.md#std-label-adf-parquet-data) file format, Atlas Data Federation doesn't support more than 32000 unique fields. ### CSV and TSV File Format When writing to CSV or TSV format, Atlas Data Federation does not support the following data types in the documents: - Arrays - DB pointer - JavaScript - JavaScript code with scope - Minimum or maximum key data type In a CSV file, Atlas Data Federation represents nested documents using the dot (`.`) notation. For example, Atlas Data Federation writes `{ x: { a: 1, b: 2 } }` as the following in the CSV file: ```shell x.a,x.b 1,2 ``` Atlas Data Federation represents all other data types as strings. Therefore, the data types in MongoDB read back from the CSV file may not be the same as the data types in the original BSON (Binary Javascript Object Notation) documents from which the data types were written. ### Parquet File Format For Parquet, Atlas Data Federation reads back fields with null or undefined values as missing because Parquet doesn't distinguish between null or undefined values and missing values. Although Atlas Data Federation supports all data types, for BSON (Binary Javascript Object Notation) data types that do not have a direct equivalent in Parquet, such as JavaScript, regular expression, etc., it: - Chooses a representation that allows the resulting Parquet file to be read back using a non-MongoDB tool. - Stores a MongoDB schema in the Parquet file's key/value metadata so that Atlas Data Federation can reconstruct the original BSON (Binary Javascript Object Notation) document with the correct data types if the Parquet file is read back by Atlas Data Federation. Consider the following BSON (Binary Javascript Object Notation) documents: ```json { "clientId": 102, "phoneNumbers": ["123-4567", "234-5678"], "clientInfo": { "name": "Taylor", "occupation": "teacher" } } { "clientId": "237", "phoneNumbers": ["345-6789"], "clientInfo": { "name": "Jordan" } } ``` If you write the preceding BSON (Binary Javascript Object Notation) documents to Parquet format using [$out to S3](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/out.md#std-label-adf-out-stage), the Parquet file schema for your BSON (Binary Javascript Object Notation) documents would look similar to the following: ```shell message root { optional group clientId { optional int32 int; optional binary string (STRING); } optional group phoneNumbers (LIST) { repeated group list { optional binary element (STRING); } } optional group clientInfo { optional binary name (STRING); optional binary occupation (STRING); } } ``` Your Parquet data on S3 (Simple Storage Service) would look similar to the following: ```shell clientId: .int = 102 phoneNumbers: .list: ..element = "123-4567" .list: ..element = "234-5678" clientInfo: .name = "Taylor" .occupation = "teacher" clientId: .string = "237" phoneNumbers: .list: ..element = "345-6789" clientInfo: .name = "Jordan" ``` The preceding example demonstrates how Atlas Data Federation handles complex data types: - Atlas Data Federation maps documents at all levels to a Parquet group. - Atlas Data Federation encodes arrays using the `LIST` logical type and the mandatory three-level list or element structure. To learn more, see [Lists.](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#nested-types) - Atlas Data Federation maps polymorphic BSON (Binary Javascript Object Notation) fields to a group of multiple single-type columns because Parquet doesn't support polymorphic columns. Atlas Data Federation names the group after the BSON (Binary Javascript Object Notation) field. In the preceding example, Atlas Data Federation creates a Parquet group named `clientId` for the polymorphic field named `clientId` with two children named after its BSON (Binary Javascript Object Notation) types, `int` and `string`. ## Error Output ### S3 Atlas Data Federation uses the error handling mechanism described below for documents that enter the [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) stage and cannot be written for one of the following reasons: - The `s3.filename` does not evaluate to a string value. - The `s3.filename` evaluates to a file that cannot be written to. - The `s3.format.name` is set to `csv`, `tsv`, `csv.gz`, or `tsv.gz` and the document passed to [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) contains data types that are not supported by the specified file format. For a full list of unsupported data types, see [CSV and TSV File Format.](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/out.md#std-label-csv_and_tsv_format) If [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) encounters one of the above errors while processing a document, Atlas Data Federation writes to the following three special error files in the path `s3:///atlas-data-lake-/`: | Error File Name | Description | | --- | --- | | out-error-docs/.json | Atlas Data Federation writes the document that encountered an error to this file. `i` begins with `1` and increments whenever the file being written to reaches the `maxFileSize`. Then, any further documents are written to the new file `out-error-docs/.json`. | | out-error-index/.json | Atlas Data Federation writes an error message to this file. Each error message contains a description of the error and an index value `n` that begins with `0` and increments with each additional error message written to the file. `i` begins with `1` and increments whenever the file being written to reaches the `maxFileSize`. Then, any further error messages are written to the new file `out-error-docs/.json`. | | out-error-summary.json | Atlas Data Federation writes a single summary document for each type of error encountered during an aggregation operation to this file. Each summary document contains a description of the type of error and a count of the number of documents that encountered that type of error. | This example shows how to generate error files using [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) in a federated database instance. The following aggregation pipeline sorts documents in the `analytics.customers` sample dataset collection by descending customer birthdate and attempts to write the `_id`, `name` and `accounts` fields of the youngest three customers to the file named `youngest-customers.csv` in the S3 (Simple Storage Service) bucket named `customer-data`. ```json db.customers.aggregate([ { "$sort": { "birthdate": -1 } }, { "$unset": [ "username", "address", "email", "tier_and_details", "birthdate" ] }, { "$limit": 3 }, { "$out": { "s3": { "bucket": "customer-data", "region": "us-east-2", "filename": "youngest-customers", "format": { "name": "csv" } } } } ]) ``` Because `accounts` is an array field, [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) encounters an error when it tries to write a document to `s3.format.name` `csv`. To handle these errors, Atlas Data Federation writes to the following three error files: - The following output shows the first of three documents written to the `out-error-docs/1.json` file: ```json { "_id" : {"$oid":"5ca4bbcea2dd94ee58162ba7"}, "name": "Marc Cain", "accounts": [{"$numberInt":"980440"}, {"$numberInt":"626807"}, {"$numberInt":"313907"}, {"$numberInt":"218101"}, {"$numberInt":"157495"}, {"$numberInt":"736396"}] } ``` - The following output shows the first of three error messages written to the `out-error-index/1.json` file. The `n` field starts at 0 and increments for each error written to the file. ```json { "n" : {"$numberInt": "0"}, "error" : "field accounts is of unsupported type array" } ``` - The following output shows the error summary document written to the `out-error-summary` file. The `count` field represents the number of documents passed to [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) that encountered an error due to the `accounts` array field. ```json { "errorType": "field accounts is of unsupported type array", "count": {"$numberInt":"3"} } ``` **See also:** - [`$out Aggregation Stage Reference`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) - [Query a Federated Database Instance](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md#std-label-query-fdi)