$vectorSearch is not allowed

I’ve kind of “solved” it for me by using an older version of langchain.

This problem occurs since version "langchain": "^0.0.165",
So just use version “langchain”: “^0.0.164”,

Hi everybody, I think it might not be related to LangChain, as I tried the $vectorSearch tutorial provided my MongoDB using their own movies dataset, and I’m facing the exact same issue.

1 Like

Same here, @Victor_Carrara , It is not langchain issue.

It is true that knnBeta is deprecated in favor of $vectorSearch. However, the $vectorSearch itself is a Preview feature which means it is not stable. So switching to the knnBeta operator is a temporary solution until the $vectorSearch starts working properly.

Here is my implementation on Github of the vector search with knnBeta if anybody is looking for an example.

1 Like

I’m not using LangChain, but running into this problem as well. Deeply disappointing, with all the “you don’t need a vector database” messaging Mongo’s been using.

Hello all, apologies for the error and for the delayed response. To clarify, the error that you’re seeing here “$vectorSearch is not allowed or the syntax is incorrect, see the Atlas documentation for more information” is due to an issue in M0-M5 Clusters which will be resolved after our deployment tomorrow, it is not related to LangChain.

The origin of this error stems from a slight difference in how our shared tier clusters work when compared to dedicated tier, hence clusters M10 and above are not affected.

To provide a bit more context, $vectorSearch is our target interface for Vector Search in the long term, and as such we’ve updated LangChain to utilize this new syntax. That said, we do not currently have plans to remove support in Atlas for knnBeta, and will not do so until every Cluster in Atlas is running a version that supports $vectorSearch. We will provide additional guidance ahead of any changes to support for knnBeta.

And one last clarification, the entire Vector Search service is in Preview, whether it be through $vectorSearch or knnBeta.

Finally, for anyone who would like to chat in more depth please feel free to follow up here or reach out to me directly at Benjamin.flast@mongodb.com.

4 Likes

Thank you so much, Benjamin!
I’d much prefer to keep my company’s vector search needs here with our preferred vendor, so a near-future resolution of this issue is very good news.
(BTW, our production cluster is M10, but our dev cluster, where I was working when I encountered the error, is on a shared tier.)

Hey Benjamin! thanks for the update. I still have the problem and before to restart everything, I would like to know if the fix has been deployed?
if so, how can i know / check if my cluster has been updated?

Thanks

2 Likes

Hey Benjamin, do you know when this $vectorSearch gonna be released? It has been a blocker for our project. We have vector DB in pinecone but were planning to move to MongoDB for vector search capability and now we are basically stuck. Would really appreciate it if you let us know the timeline.

This was hyped but now has been a real pain for developers like us who trusted the MongoDB ecosystem. Would appreciate it if Mongo could communicate it better.

Hello All,

Just to provide a quick update, the update has been made, it’s just making it through all of the testing environments. This should be available on shared tier clusters (M0-M5) by the end of day Monday.

As a quick reminder, this can be accessed on dedicated cluster M10 and above today. Or you can continue to query utilizing the knnBeta operator until Monday.

2 Likes

Hi Everyone!

Thank you all for your patience, this issue is now resolved. You should be able to use the $vectorSearch stage on shared tier clusters (M0-M5) running on MongoDB versions 6.0.11 and 7.0.2.

I am still getting this same error. I am on version 7.0.2 M30. What am I missing? Your previous message 5 days ago says M10 and above can access today. This is massively frustrating …

You’ll have to upgrade to 7.0.2

I just updated my message please review it.

It is working for me now

Hey @Dustin_Dier apologies for the challenges and delay here!

The issue that was resolved on Monday was for M0-M5, if you’re having trouble with an M30 on 7.0.2 this must be a separate issue.

Please reach out to me directly at benjamin.flast@mongodb.com or schedule some time directly with Calendly - Benjamin Flast so we can look into what’s going on. I’m sure we can get this sorted out quickly!

(If anyone else is having separate or different challenges please feel free to throw some time on my calendar!)

Best,
Ben

I am trying to query multiple Vector Embeddings with $unionWith

Code:

const documents = await collection
          .aggregate([
            {
              $unionWith: {
                coll: "SearchLeads",
                pipeline: [
                  {
                    $vectorSearch: {
                      queryVector: embedding,
                      path: "industry_embedding",
                      numCandidates: 10000,
                      limit: 30,
                      index: "default",
                    },
                  },
                  {
                    $vectorSearch: {
                      queryVector: embedding,
                      path: "city_embedding",
                      numCandidates: 10000,
                      limit: 30,
                      index: "default",
                    },
                  },
                ],
              },
            },
          ])
          .toArray();

        return documents;

and I am getting this error while searching $vectorSearch is not allowed within a $unionWith’s sub-pipeline reference.

Cluster version: 6.0.13
Tier: M10 (General)

Hi Pratham,

I’m sorry you ran into this issue. We don’t currently support running $vectorSearch within a subpipeline, as noted in the limitations section of our documentation. I believe that previous comment I left that you linked to was mistaken, and have provided a clarification on that thread.

Since this is being actively worked on, I will provide an update directly on this thread once this is supported. In the meantime you should be able to achieve similar behavior by using $search with the knnBeta operator in a subpipeline instead (note that it has different syntax from the $vectorSearch stage but very similar functionality under the hood).

Sorry again for the confusion and inconvenience here, and please let me know if I can help with any other questions you have.

2 Likes

Hi Henry,

Thanks for your help. I am trying to use $search with the knnBeta operator in a sub-pipeline instead with the following syntax

const documents = await collection
          .aggregate([
            {
              $unionWith: {
                coll: "SearchLeads",
                pipeline: [
                  {
                    $search: {
                      index: "default",
                      knnBeta: {
                        vector: embedding,
                        path: "city_embedding",
                        k: 150,
                      },
                    },
                  },
                  {
                    $limit: 50,
                  },
                  {
                    $search: {
                      index: "default",
                      knnBeta: {
                        vector: embedding,
                        path: "industry_embedding",
                        k: 150,
                      },
                    },
                  },
                  {
                    $limit: 50,
                  },
                ],
              },
            },
          ])
          .toArray();

But I am getting the following error:

$_internalSearchMongotRemote is only valid as the first stage in a pipeline

What I want to do:
I have created two embedding in my SearchLeads collection industry_embedding and city_embedding. I want to match the results that contain both industry and city based on the query. How can I achieve this? Appreciate your help.

Hi Pratham,

Thank you for sharing. You must have $search as the first stage in the pipeline, even if executed within a subpipeline. The fix I would suggest for you is to have a first $search stage, then execute a $unionWith which executes the second $search stage as a subpipeline:

const documents = await collection
          .aggregate([
            {
              $search: {
                index: "default",
                knnBeta: {
                  vector: embedding,
                  path: "city_embedding",
                  k: 150,
                },
              },
            },
            {
              $limit: 50,
            },
            {
              $unionWith: {
                coll: "SearchLeads",
                pipeline: [
                  {
                    $search: {
                      index: "default",
                      knnBeta: {
                        vector: embedding,
                        path: "industry_embedding",
                        k: 150,
                      },
                    },
                  },
                  {
                    $limit: 50,
                  },
                ],
              },
            },
          ]).toArray();

Hope this helps and sorry for the delay in response!