Vector search : filter doesn't works / token index

Hi,

I’ve a collection sources like this interface:

export interface SourceDocument {
_id: ObjectId;
  caseType: "DEFENDER" | "ASKER";
  argumentId: ObjectId;
  mindmapId: ObjectId;
  vector?: Vector;
}

I’ve created a search vector index named “sources_vector_index” with this definition:

{
  "fields": [
    {
      "numDimensions": 1536,
      "path": "vector",
      "similarity": "euclidean",
      "type": "vector"
    }
  ]
}

I’ve created a caseType_token_index for the attribute casetype for the collection sources with this definition:

{
  "mappings": {
    "dynamic": false,
    "fields": {
      "caseType": {
        "normalizer": "none",
        "type": "token"
      }
    }
  }
}

I’m trying to implement a search functionality in my app with filters by the filter tag in the aggregation:

    const agg = [
      {
        $vectorSearch: {
          index: "sources_vector_index",
          path: "vector",
          filter,
          queryVector: queryVector,
          numCandidates: 2000,
          limit: 10,
        },
      },
      {
        $project: {
          _id: 1,
          score: {
            $meta: "vectorSearchScore",
          },
        },
      },
    ];

    const sourcesArr = await sources.aggregate(agg).toArray();

When I execute the code, I got this error:
error: PlanExecutor error during aggregation :: caused by :: Path 'caseType' needs to be indexed as token code: "8"

I’ve tried to make my search with a { $match } and it works, but why can I not use the filter tag in the $vectorSearch object instead of I’ve an index on the caseType attribute ?

In future, I want add more filters for fields which have ObjectId as values, will I have to create a token index for all of them ?

Are there any solutions to this issue?

You wiill need to create a Vector Search index in the following way:

{
  "fields": [
    {
      "numDimensions": 1536,
      "path": "vector",
      "similarity": "euclidean",
      "type": "vector"
    },
   { "type": "filter",
      "path": "caseType"}
  }
  ]
}

This pattern of using pre-filtering alongside the vector search is described here - https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-tutorial/#define-the-atlas-vector-search-index.-1

3 Likes

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.