> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt # Define Path Syntax for Data Stores **Note:** For documentation simplicity, the examples found on this page are based on data stores in AWS (Amazon Web Services) S3 (Simple Storage Service), however, they apply to Google Cloud and Azure Private Link too. ## Overview When you query documents in your data store, the Atlas Data Federation [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) value allows Data Federation to map the data inside your document to the filename of the document. [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) supports parsing filenames into your data stores into computed fields. Data Federation can add the computed fields to each document generated from the parsed file. Data Federation can target queries on those computed field values to only those files with a matching file name. See [Supported Parsing Functions](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-supported-parsing-funcs) and [Examples](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-path-syntax-egs) for more information. [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) also supports creating partitions using partition attributes in the path to the file. Data Federation can target queries on the parameter defined in the partition attribute to only those files that contain the query in the filename or partition prefix. **Example:** Consider the following files in your S3 (Simple Storage Service) bucket: ```sh /users/26/1234.json /users/26/5678.json ``` The JSON (Javascript Object Notation) document `1234.json` contains the following: ```json { "name": "jane doe", "age": 26, "userID": "1234" } ``` your federated database instance configuration for the files in your S3 (Simple Storage Service) bucket defines the following `path`: ```sh "path": "/users/{age int}/{userID string}" ``` The following shows how Data Federation maps a query to the partitions created from the `path` definition: ```json db.users.findOne( /users { /40 "age": 26 -----------------> /26 "userID": "1234" ----------> /1234.json } /5678.json ) ``` Data Federation adds the computed field for the partition attribute to the document. In the above example, Data Federation adds the `age` field and value to `1234.json`. Atlas Data Federation adds partition attributes to documents even when a field with the same name already exists in the document, which creates duplicate field names. MongoDB doesn't support [duplicate field names](https://www.mongodb.com/docs/manual/reference/limits/#mongodb-does-not-support-duplicate-field-names). To avoid duplicate field names, set `omitAttributes` to `true` or use partition attribute names that don't appear in your documents. ## Supported Parsing Functions | You can specify a single parsing function on the filename. | `/path/to/files/{ }` | | --- | --- | | You can specify multiple parsing functions on the filename. | `/path/to/files/{ }-{ }` | | You can specify parsing functions alongside static strings in the filename: | `/path/to/files/prefix-{ }-suffix` | | You can specify dot (i.e. `.`) along the path to the filename. | `/path/to/files/{. }` | | You can enclose a field name in single quotes (`'`) if the field name contains special characters, such as `:` or `#`. | `/path/to/files/{'' }` | | You can specify `ObjectIds` in the path to the files to create partitions. | `/path/to/files/{objid objectid}` | | You can specify a range of `ObjectIds` in the path to the files to create partitions. | `/path/to/files/{min(obj) objectid}-{max(obj) objectid}` | | You can specify parsing functions along the path to the filename. | `/path/{ }/{ }/{ }/*` | | You can specify regex for `ISODate` along the path to the filename. | `/path/to/files/{ isodate(""):\\}` | **Note:** When specifying the [`path`:](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) - Specify the data type for the partition attribute. - Ensure that the partition attribute type matches the data type to parse. - Use the delimiter specified in [`delimiter`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.delimiter) When specifying attributes of the same type, do any of the following: - Add a constant separator between the attributes. - Use regular expressions to describe the search pattern. To learn more, see [Unsupported Parsing Functions.](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-unsupported-parsing-funcs) ### Parsing Null Values from Filenames Data Federation automatically parses an empty string (`""`) in the place of an attribute in the file path as the BSON null value for all the Atlas Data Federation attribute types except `string`. With a `string`, empty string could either represent a BSON null value or a BSON empty string value. Atlas Data Federation does not parse any BSON value for `string` attribute type. This avoids adding a BSON value with a conflicting type to documents read from your data store. **Example:** Consider the following S3 (Simple Storage Service) federated database instance store: ```none /records/january/1.json /records/february/1.json /records//1.json For the path ``/records/{month string}/*``, Data Federation does not add any computed fields for the ``month`` attribute to documents generated from the third record in the above store. ``` **Note:** When writing files to S3 (Simple Storage Service), write BSON (Binary Javascript Object Notation) null values as empty strings in filenames for all Atlas Data Federation attribute types. ### Parsing Padded Numbers from Filenames File path can include numeric values that are padded with leading zeros. For Data Federation to correctly parse padded numeric values for attribute types like `int`, `epoch_millis`, and `epoch_secs`, specify the number of digits in the value using regular expressions. **Example:** Consider a S3 (Simple Storage Service) store with the following files: ```text |--users |--001.json |--002.json ... ``` The following `path` syntax uses a regular expression to specify the number of digits in the filename. Data Federation identifies the portion of the path that corresponds to the partition attribute and then maps that partition attribute to a type `int`: ```sh /users/{user_id int:\\d{3}} ``` ## Unsupported Parsing Functions Specifying consecutive parsing functions on the same filename with no way of differentiating them could lead to inconsistent results. **Example:** Consider a S3 (Simple Storage Service) store with the following files: ```shell |--food |--icecream |--peanutbutter ``` Now consider the following path to the files in the `food` directory: ```shell /food/{first string}{second string}/ ``` There is no way to determine which part of the filename substring should match the `first` versus `second` string attributes. Therefore, Atlas Data Federation matches everything to `first`. Consider using attributes with regex similar to the following instead: - ```shell /food/{first string:\\D{3}}{second string:\\D{5}} ``` The preceding path specifies that `first` matches 3 non-digit characters and `second` matches the next 5 non-digit characters. - ```shell /food/{first string:[A-Z]{3}}{second string:[a-z]{5}} ``` The preceding path matches `first` to 3 uppercase letters and `second` to 5 lowercase letters. ## Examples The following examples demonstrate how to parse filenames into computed fields: - [Parse Single Field from Filename](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-parse-single) - [Parse Multiple Fields from Filename](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-parse-multiple) - [Use Regular Expression to Parse Fields from Filename](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-parse-regex) - [Identify Ranges of Queryable Data from Filename](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-parse-range) - [Identify Nested Fields from Filename](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-parse-nested-fields) - [Identify Fields with Special Characters from Filename](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-special-chars) - [Create Partitions from ObjectIds](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-objectids) - [Create Partitions from File Path](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-create-partition) - [Create Partitions from ISODate](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-isodate) - [Generate Dynamic Collection Names from File Path](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-generate-collection) ### Parse Single Field from Filename Consider a federated database instance store `accountingArchive` containing files where the filename describes an invoice date. For example, the filename `/invoices/1564671291998.json` contains the invoices for the UNIX timestamp `1564671291998`. The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object generates a field `invoiceDate` by parsing the filename as a UNIX timestamp: ```none "databases" : [ { "name" : "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{invoiceDate epoch_millis}" } ] } ] } ] ``` Data Federation adds the computed field and value to each document generated from the filename. Documents generated from the example filename includes a field `invoiceDate: ISODate("2019-08-01T14:54:51Z")`. Queries on the `invoiceDate` field can be targeted to only those files that match the specified value. ### Parse Multiple Fields from Filename Consider a federated database instance store `accountingArchive` containing files where the filename describes an invoice number and invoice date. For example, the filename `/invoices/MONGO12345-1564671291998.json` contains the invoice `MONGODB12345` for the UNIX timestamp `1564671291998`. The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object generates: - A field `invoiceNumber` by parsing the first segment of the filename as a string. - A field `invoiceDate` by parsing the second segment of the filename as a UNIX timestamp. ```none "databases" : [ { "name": "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{invoiceNumber string}-{invoiceDate epoch_millis}" } ] } ] } ] ``` Data Federation adds the computed fields and values to each document generated from the filename. Documents generated from the example filename include the following fields: - `invoiceNumber : "MONGODB12345"` - `invoiceDate : ISODate("2019-08-01T14:54:51Z")` Queries that include *both* the `invoiceNumber` and `invoiceDate` fields can be targeted to only those files that match the specified values. ### Use Regular Expression to Parse Fields from Filename Consider a federated database instance store `accountingArchive` containing files where the filename describes an invoice number and invoice date. For example, the filename `/invoices/MONGODB12345-20190102.json` contains the invoice `MONGODB12345` for the date `20190102`. The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object generates: - A field `invoiceNumber` by parsing the first segment of the filename as a string - A field `year` by using a regular expression to parse only the first 4 digits of the second segment of the filename as an int. - A field `month` by using a regular expression to parse only the next 2 digits of the second segment of the filename as an int. - A field `day` by using a regular expression to parse only the next 2 digits of the second segment of the filename as an int. ```none "databases" : [ { "name" : "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{invoiceNumber string}-{year int:\\d{4}}{month int:\\d{2}}{day int:\\d{2}}" } ] } ] } ] ``` Data Federation adds the computed fields and values to each document generated from the filename. Documents generated from the example filename include the following fields: - `invoiceNumber : "MONGODB12345"` - `year : 2019` - `month: 01` - `day: 02` **Important:** You must escape the regex string specified in the `path`. For example, if the regex string includes double quotes, you must escape those values. Data Federation supports the [Package Syntax](https://golang.org/pkg/regexp/syntax/) for regular expressions in the storage configuration. Queries that include *all* generated fields can be targeted to only those files that match the specified values. **See also:** [Parsing Padded Numbers from Filenames](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-parse-padded-numeric-values) ### Identify Ranges of Queryable Data from Filename Consider a federated database instance store `accountingArchive` containing files where the filename describes the range of data contained in the file. For example, the filename `/invoices/1546367712000-1549046112000.json` contains invoices for the time period between 2019-01-01 and 2019-01-02 with the date range represented as milliseconds elapsed since the UNIX epoch. The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object identifies the minimum time range as the first segment of the filename and the maximum time range as the second segment of the filename: ```none "databases" : [ { "name: "accounting", "collections" : [ { "name: "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{min(invoiceDate) epoch_millis}-{max(invoiceDate) epoch_millis}" } ] } ] } ] ``` When Data Federation receives a query on the `"invoiceDate"` field, it uses the specified path to identify which files contain the data that matches the query. Queries on the `invoiceDate` field can be targeted to only those files whose range captures the specified value, including the `min` and `max` date. **Important:** The field specified for the min and max ranges *must* exist in every document contained in the file to avoid unexpected or undesired behavior. Data Federation does *not* perform any validation that the underlying data conforms to this constraint. **See also:** [Parsing Padded Numbers from Filenames](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-parse-padded-numeric-values) ### Identify Nested Fields from Filename Data Federation supports querying nested data when the nested data value is also the filename. You can use the dot operator (i.e. `.`) in your [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) to map the partitions attributes in your storage configuration to nested fields in your documents. Consider a federated database instance store `accountingArchive`. The federated database instance store contains files with names that match values of nested fields in the documents. For example: ```text accountingArchive |--invoices |--January.json |--February.json ... ``` Suppose the `January.json` file contains a document with the following fields: ```text { "invoice": { "invoiceNumber" : "MONGODB12345", "year" : 2019, "month": "January", //value matches filename "date": 02 }, "vendor": "MONGODB", ... } ``` The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object identifies `month` as a nested field inside a document. The [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object also identifies the value of `month` as the name of the file that contains the document. ```text "databases" : [ { "name" : "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{invoice.month string}" } ] } ] } ] ``` When Atlas Data Federation receives a query on a specific month such as `January`, it uses the specified path to identify which file contains the data that matches the query. ### Identify Fields with Special Characters from Filename If a field name contains special characters, such as a colon (`:`) or a number sign (`#`), enclose the field name in single quotes (`'`) in the [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path). Otherwise, Atlas Data Federation considers the storage configuration invalid. Enclose the entire field name, including any dots that separate nested fields, in a single pair of quotes. Consider a federated database instance store `accountingArchive` that contains files with names that match values of nested fields in the documents. For example, suppose a file contains a document with the following fields: ```text { "m:invoice": { "m:asOfDate": { "#value": "2026-08-01T14:54:51Z" } }, "vendor": "MONGODB", ... } ``` The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object encloses the `m:invoice.m:asOfDate.#value` field name in single quotes to identify the range of dates that each file contains: ```text "databases" : [ { "name" : "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{min('m:invoice.m:asOfDate.#value') isodate}-{max('m:invoice.m:asOfDate.#value') isodate}/*" } ] } ] } ] ``` ### Create Partitions from ObjectIds You can specify [ObjectIds](https://www.mongodb.com/docs/manual/reference/bson-types.md#std-label-objectid) in the path to the files. For files that contain `ObjectId` in the filename, Atlas Data Federation creates partitions for each `ObjectId`. Consider the following federated database instance store, `accountingArchive`. This data store contains files that include `ObjectId` in the filename: ```text accountingArchive |--invoices |--507f1f77bcf86cd799439011.json |--507f1f77bcf86cd799439012.json |--507f1f77bcf86cd799439013.json |--507f1f77bcf86cd799439014.json |--507f1f77bcf86cd799439015.json ``` The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object creates partitions for the `ObjectIds`. ```text "databases" : [ { "name" : "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{objid objectid}" } ] } ] } ] ``` Or, suppose the federated database instance store `accountingArchive` contains files that include a range of `ObjectIds` in the filename. For example: ```text accountingArchive |--invoices |--507f1f77bcf86cd799439011-507f1f77bcf86cd799439020.json |--507f1f77bcf86cd799439021-507f1f77bcf86cd799439030.json |--507f1f77bcf86cd799439031-507f1f77bcf86cd799439040.json ``` The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object creates partitions for the given range of `ObjectIds`. ```text "databases" : [ { "name" : "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{min(obj) objectid}-{max(obj) objectid}" } ] } ] } ] ``` When Data Federation receives a query on the `ObjectId`, it uses the specified path to identify which file contains the data that matches the query. ### Create Partitions from File Path You can specify parsing functions on any path leading up to the filename. Each computed field is based on a parsing function along the path. When querying the data, Atlas Data Federation converts each computed field to a partition. Partitions, which are synonymous with subdirectories, are then used to filter files with increasing precision. Consider a federated database instance store `accountingArchive` with the following directory structure: ```text invoices |--MONGO12345 |--2019 |--01 |--02 ``` The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object creates the `invoiceNumber`, `year`, `month`, and `day` partitions with a small set of filtered files: ```text "databases" : [ { "name" : "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{invoiceNumber string}/{year int}/{month int:\\d{2}}/{day int:\\d{2}}/*" } ] } ] } } ``` **See also:** [Parsing Padded Numbers from Filenames](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-parse-padded-numeric-values) ### Create Partitions from ISODate You can specify [ISODate](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-ISODate) in the path to the files. For files that contain `ISODate` in the filename, Atlas Data Federation creates partitions for each `ISODate`. Consider the following federated database instance store, `accountingArchive`. This data store contains files that include `ISODate` in the filename: ```text accountingArchive |--invoices |--01_02_2022_2301.json |--02_02_2022_2301.json |--03_02_2022_2301.json |--04_02_2022_2301.json |--05_02_2022_2301.json ``` The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object creates partitions for the `ISODate`. The `path` demonstrates how to use regex for the date format if the date is not in [RFC 3339](https://tools.ietf.org/html/rfc3339) format. ```text "databases" : [ { "name" : "accounting", "collections" : [ { "name" : "invoices", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/creationDate isodate('01_02_2006_1504'):\\d{2}_\\d{2}_\\d{4}_\\d{4}.json" } ] } ] } ] ``` To learn more about other supported formats, see [Use Partition Attribute Types.](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/supported-partition-attributes.md#std-label-adf-path-attribute-types) ### Generate Dynamic Collection Names from File Path Consider a federated database instance store `accountingArchive` with the following directory structure: ```text invoices |--SuperSoftware |--UltraSoftware |--MegaSoftware ``` The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object generates a dynamic collection name from the file path: ```json "databases" : [ { "name" : "invoices", "collections" : [ { "name" : "*", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/invoices/{collectionName()}/" } ] } ] } ] ``` When applied to the example directory structure, the path results in the following collections: - SuperSoftware - UltraSoftware - MegaSoftware Or, consider a federated database instance store `accountingArchive` with the following files: ```text /orders/MONGODB/invoices/January.json /orders/MONGODB/purchaseOrders/January.json /orders/MONGODB/invoices/February.json ... ``` The following [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) object generates a dynamic collection name from the file path: ```json "databases" : [ { "name" : "invoices", "collections" : [ { "name" : "*", "dataSources" : [ { "storeName" : "accountingArchive", "path" : "/orders/MONGODB/{collectionName()}/{invoiceMonth string}.json/" } ] } ] } ] ``` When applied to the example filenames, the path results in the following collections: - `invoices` - `purchaseOrders` **Note:** When you dynamically generate collections from filenames, the number of collections is not accurately reported in the Data Federation view.