How to Index String Fields for Efficient Filtering and Sorting
On this page
You can use the Atlas Search token
type to index string fields for sorting the Atlas Search results.
You can then use the $search
sort
option in your query
to sort the results by the indexed field. To learn more, see
Sort Atlas Search Results. You can also use the Atlas Search token
type to index
string fields for
pre-filtering the data that the $vectorSearch
queries analyze. To learn more, see Perform Semantic Search with Atlas Vector Search.
Review the Behavior of the token
Type
When you index a field as token
type, Atlas Search indexes the terms in
the string as a single token (searchable term) and stores them in a
columnar storage format for efficient filtering or sort operations. You
can use a normalizer to transform the
token. By default, the normalizer is set to none
and so Atlas Search
indexes strings in their original form.
The major difference between the Atlas Search string
and token
types is
that Atlas Search creates one or more tokens for fields indexed as string
type whereas Atlas Search creates only a single token for fields indexed as the
token
type.
If a string being indexed as a token
field type exceeds 8181
characters, Atlas Search truncates it to 8181 characters before indexing.
Review token
Type Limitations
When you index a field as the token
type, you can only sort by
that field. To also query that field, you must index that field as
string
type also. You can't query a field indexed only as the
token
type.
You can't index children of fields indexed as the embeddedDocuments type as the token
type.
Define the Index for the token
Type
To define the index for the token
type, choose your preferred
configuration method in the Atlas UI and then select the
database and collection.
Configure token
Field Properties
The Atlas Search token
type takes the following parameters:
Option | Type | Necessity | Description | Default |
---|---|---|---|---|
type | string | Required | Human-readable label that identifies this field type.
Value must be token . | |
normalizer | string | Optional | Type of transformation to perform on the field value. Value can be one of the following:
If you don't set this option explicitly, it defaults to | none |
Try an Example for the token
Type
The following index definition example uses the sample_mflix.movies collection. If you have the sample data already loaded on your cluster, you can use the Visual Editor or JSON Editor in the Atlas UI to configure the index. After you select your preferred configuration method, select the database and collection, and refine your index to add field mappings.