您可以将 MongoDB Vector Search 与 LangChain 集成以执行自查询检索。本教程演示如何使用自查询检索器运行元数据筛选的自然语言MongoDB Vector Search 查询。
自查询检索使用 LLM进程搜索查询以识别可能的元数据筛选器,使用筛选器形成结构化向量搜索查询,然后运行查询以检索最相关的文档。
例子
对于类似“What istrunk movie from after 2010 with ratings over 8?”这样的查询,检索器可以识别 genre、year 和 rating 字段上的筛选器,并使用这些筛选器来检索与查询匹配的文档。
先决条件
如要完成本教程,您必须具备以下条件:
以下MongoDB 集群类型之一:
使用 Python 和 Docker 创建的本地 Atlas 部署。安装
atlas-local-lib-py(pip install atlas-local-lib-py),以编程方式创建和管理本地部署。要了解更多信息,请参阅 atlas-local-lib-py 存储库。A MongoDB Community cluster with Search and Vector Search installed.
Voyage AI API密钥。要学习;了解更多信息,请参阅 Voyage AI文档。
OpenAI API密钥。您必须拥有一个具有可用于API请求的积分的 OpenAI 帐户。要学习;了解有关注册 OpenAI 帐户的更多信息,请参阅 OpenAI API网站。
使用MongoDB作为向量存储
在本部分中,您将使用MongoDB 集群作为向量数据库来创建向量存储实例。
设置环境。
为此教程设置环境。 通过保存扩展名为 .ipynb 的文件来创建交互式Python笔记本。 此 Notebook 允许您单独运行Python代码片段,并且您将使用它来运行本教程中的代码。
要设立笔记本环境,请执行以下操作:
在笔记本中运行以下命令:
pip install --quiet --upgrade langchain-mongodb langchain-voyageai langchain-openai langchain langchain-core lark 设置环境变量。
Run the following code to set the environment variables for this tutorial. Provide your Voyage API key, OpenAI API Key, and MongoDB cluster's SRV connection string.
import os os.environ["OPENAI_API_KEY"] = "<openai-key>" os.environ["VOYAGE_API_KEY"] = "<voyage-key>" MONGODB_URI = "<connection-string>" 注意
将
<connection-string>替换为您的 Atlas 集群或本地部署的连接字符串。连接字符串应使用以下格式:
mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net 重要: 确保连接字符串包含数据库用户的凭证。要学习有关查找连接字符串的更多信息,请参阅通过客户端库连接到集群。
实例化向量存储。
在 Notebook 中运行以下代码,使用MongoDB中的 langchain_db.self_query命名空间创建一个名为 vector_store 的向量存储实例:
from langchain_mongodb import MongoDBAtlasVectorSearch from langchain_voyageai import VoyageAIEmbeddings # Use the voyage-3-large embedding model embedding_model = VoyageAIEmbeddings(model="voyage-3-large") # Create the vector store vector_store = MongoDBAtlasVectorSearch.from_connection_string( connection_string = MONGODB_URI, embedding = embedding_model, namespace = "langchain_db.self_query", text_key = "page_content" )
将数据添加到向量存储中。
在笔记本中粘贴并运行以下代码,以将一些带有元数据的示例文档提取到MongoDB中的集合中。
from langchain_core.documents import Document docs = [ Document( page_content="A bunch of scientists bring back dinosaurs and mayhem breaks loose", metadata={"year": 1993, "rating": 7.7, "genre": "action"}, ), Document( page_content="A fight club that is not a fight club, but is a fight club", metadata={"year": 1994, "rating": 8.7, "genre": "action"}, ), Document( page_content="Leo DiCaprio gets lost in a dream within a dream within a dream within a ...", metadata={"year": 2010, "genre": "thriller", "rating": 8.2}, ), Document( page_content="A bunch of normal-sized women are supremely wholesome and some men pine after them", metadata={"year": 2019, "rating": 8.3, "genre": "drama"}, ), Document( page_content="Three men walk into the Zone, three men walk out of the Zone", metadata={"year": 1979, "rating": 9.9, "genre": "science fiction"}, ), Document( page_content="A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea", metadata={"year": 2006, "genre": "thriller", "rating": 9.0}, ), Document( page_content="Toys come alive and have a blast doing so", metadata={"year": 1995, "genre": "animated", "rating": 9.3}, ), Document( page_content="The toys come together to save their friend from a kid who doesn't know how to play with them", metadata={"year": 1997, "genre": "animated", "rating": 9.1}, ), ] # Add data to the vector store, which automaticaly embeds the documents vector_store.add_documents(docs)
如果您使用 Atlas,您可以通过导航到 Atlas 用户界面中的 langchain_db.self_query 命名空间来验证您的向量嵌入。
使用筛选器创建MongoDB Vector Search索引。
运行以下代码,为向量存储创建带有筛选器的MongoDB Vector Search索引,从而对数据启用向量搜索和元数据筛选:
# Use LangChain helper method to create the vector search index vector_store.create_vector_search_index( dimensions = 1024, # The dimensions of the vector embeddings to be indexed filters = [ "genre", "rating", "year" ], # The metadata fields to be indexed for filtering wait_until_complete = 60 # Number of seconds to wait for the index to build (can take around a minute) )
The index should take about one minute to build. While it builds, the index is in an initial sync state. When it finishes building, you can start querying the data in your collection.
创建自查询检索器
在本部分中,您将初始化自查询检索器以查询向量存储中的数据。
描述文档和元数据字段。
要使用自查询检索器,您必须描述集合中的文档以及要过滤的元数据字段。此信息有助于法学硕士了解数据结构以及如何根据用户查询过滤结果。
from langchain.chains.query_constructor.schema import AttributeInfo # Define the document content description document_content_description = "Brief summary of a movie" # Define the metadata fields to filter on metadata_field_info = [ AttributeInfo( name="genre", description="The genre of the movie", type="string", ), AttributeInfo( name="year", description="The year the movie was released", type="integer", ), AttributeInfo( name="rating", description="A 1-10 rating for the movie", type="float" ), ]
初始化自查询检索器。
运行以下代码,使用 MongoDBAtlasSelfQueryRetriever.from_llm 方法创建自查询检索器。
from langchain_mongodb.retrievers import MongoDBAtlasSelfQueryRetriever from langchain_openai import ChatOpenAI llm = ChatOpenAI(model="gpt-4o") retriever = MongoDBAtlasSelfQueryRetriever.from_llm( llm=llm, vectorstore=vector_store, metadata_field_info=metadata_field_info, document_contents=document_content_description )
使用自查询检索器运行查询
运行以下查询,查看自查询检索器如何执行不同类型的查询:
# This example specifies a filter (rating > 9) retriever.invoke("What are some highly rated movies (above 9)?")
# This example specifies a semantic search and a filter (rating > 9) retriever.invoke("I want to watch a movie about toys rated higher than 9")
# This example specifies a composite filter (rating >= 9 and genre = thriller) retriever.invoke("What's a highly rated (above or equal 9) thriller film?")
# This example specifies a query and composite filter (year > 1990 and year < 2005 and genre = action) retriever.invoke( "What's a movie after 1990 but before 2005 that's all about dinosaurs, " + "and preferably has the action genre" )
# This example only specifies a semantic search query retriever.invoke("What are some movies about dinosaurs")
在 RAG 管道中使用检索器
您可以在 RAG管道中使用自查询检索器。在笔记本中粘贴并运行以下代码,以实现执行自查询检索的示例RAG管道。
此代码还会将检索器配置为使用 enable_limit 参数,从而允许 LLM 在必要时限制检索器返回的文档数量。生成的响应可能会有所不同。
import pprint from langchain_core.output_parsers import StrOutputParser from langchain_core.prompts import PromptTemplate from langchain_core.runnables import RunnablePassthrough from langchain_openai import ChatOpenAI llm = ChatOpenAI(model="gpt-4o") # Configure self-query retriever with a document limit retriever = MongoDBAtlasSelfQueryRetriever.from_llm( llm, vector_store, document_content_description, metadata_field_info, enable_limit=True ) # Define a prompt template template = """ Use the following pieces of context to answer the question at the end. {context} Question: {question} """ prompt = PromptTemplate.from_template(template) # Construct a chain to answer questions on your data chain = ( { "context": retriever, "question": RunnablePassthrough()} | prompt | llm | StrOutputParser() ) # Prompt the chain question = "What are two movies about toys after 1990?" # year > 1990 and document limit of 2 answer = chain.invoke(question) print("Question: " + question) print("Answer: " + answer) # Return source documents documents = retriever.invoke(question) print("\nSource documents:") pprint.pprint(documents)
Question: What are two movies about toys after 1990? Answer: The two movies about toys after 1990 are: 1. The 1995 animated movie (rated 9.3) where toys come alive and have fun. 2. The 1997 animated movie (rated 9.1) where toys work together to save their friend from a kid who doesn’t know how to play with them. Source documents: [Document(id='686e84de13668e4048bf9ff5', metadata={'_id': '686e84de13668e4048bf9ff5', 'year': 1995, 'genre': 'animated', 'rating': 9.3}, page_content='Toys come alive and have a blast doing so'), Document(id='686e84de13668e4048bf9ff6', metadata={'_id': '686e84de13668e4048bf9ff6', 'year': 1997, 'genre': 'animated', 'rating': 9.1}, page_content="The toys come together to save their friend from a kid who doesn't know how to play with them")]