Realizar la recuperación de documentos principales con MongoDB y LangChain

Segundo plano

Larecuperación de documentos principales es una técnica que implica fragmentar documentos grandes en subdocumentos más pequeños. En esta técnica, se consultan los fragmentos más pequeños antes de devolver el documento principal completo a la base de datos. LLM. Esto puede mejorar las respuestas de sus agentes y aplicaciones RAG al permitir búsquedas más granulares en fragmentos más pequeños, a la vez que proporciona a los LLMel contexto completo del documento principal.

La recuperación de documentos principales con MongoDB permite almacenar tanto documentos principales como secundarios en una única colección. Esta técnica facilita la recuperación eficiente al requerir únicamente calcular e indexar los embeddings de los documentos secundarios.

Requisitos previos

Para completar este tutorial, debes tener lo siguiente:

• Uno de los siguientes tipos de clúster de MongoDB:

  • An Clúster Atlas con MongoDB 6.0.11 versión, 7.0.2 o posterior. Asegúrese de que su dirección IP esté incluida en la lista de acceso de su proyecto Atlas.

  • Una implementación local de Atlas creada usando Atlas CLI. Para obtener más información, consulta Crear una Implementación local de Atlas.

  • Un clúster de MongoDB Community o Enterprise con Search y Vector Search instalados.

• Una clave API de Voyage AI. Para crear una clave API, consulta Claves API de modelos.

• Una llave de API de OpenAI. Debes tener una cuenta de OpenAI con créditos disponibles para las solicitudes de API. Para obtener más información sobre cómo registrar una cuenta de OpenAI, consulta el sitio web de la API de OpenAI.

• Un entorno para ejecutar cuadernos interactivos de Python como Colab.

Configurar el entorno

Configura el entorno para este tutorial. Crea un cuaderno interactivo de Python guardando un archivo con el .ipynb Extensión. Este cuaderno te permite ejecutar fragmentos de código Python individualmente y lo usarás para ejecutar el código en este tutorial.

Para configurar el entorno de su portátil:

pip install --quiet --upgrade langchain langchain-community langchain-core langchain-mongodb langchain-voyageai langchain-openai pymongo pypdf

import os
os.environ["VOYAGE_API_KEY"] = "<voyage-api-key>"
os.environ["OPENAI_API_KEY"] = "<openai-api-key>"
MONGODB_URI = "<connection-string>"

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

mongodb://localhost:<port-number>/?directConnection=true

Prepara los datos

Pegue y ejecute el siguiente código en su cuaderno para cargar y dividir un PDF de muestra que contiene un informe de ganancias reciente de MongoDB.

Este código utiliza un divisor de texto para fragmentar los datos PDF en documentos principales más pequeños. Especifica el tamaño del fragmento (número de caracteres) y la superposición de fragmentos (número de caracteres superpuestos entre fragmentos consecutivos) para cada documento.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader
# Load the PDF
loader = PyPDFLoader("https://investors.mongodb.com/node/12881/pdf") 
data = loader.load()
# Chunk into parent documents
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=20)
docs = parent_splitter.split_documents(data)
# Print a document
docs[0]

Document(metadata={'source': 'https://investors.mongodb.com/node/12881/pdf', 'page': 0, 'page_label': '1'}, page_content='MongoDB, Inc. Announces Third Quarter Fiscal 2025 Financial Results\nDecember 9, 2024\nThird Quarter Fiscal 2025 Total Revenue of $529.4 million, up 22% Year-over-Year\nContinued Strong Customer Growth with Over 52,600 Customers as of October 31, 2024\nMongoDB Atlas Revenue up 26% Year-over-Year; 68% of Total Q3 Revenue\nNEW YORK , Dec. 9, 2024 /PRNewswire/ -- MongoDB, Inc. (NASDAQ: MDB) today announced its financial results for the third quarter ended October\n31, 2024.\n\xa0\n  \xa0\n"MongoDB\'s third quarter results were significantly ahead of expectations on the top and bottom line, driven by better-than-expected EA performance\nand 26% Atlas revenue growth.\xa0 We continue to see success winning new business due to the superiority of MongoDB\'s developer data platform in\naddressing a wide variety of mission-critical use cases," said Dev Ittycheria, President and Chief Executive Officer of MongoDB .\n"We continue to invest in our legacy app modernization and AI offerings as our document model and distributed architecture are exceptionally well\nsuited for customers looking to build highly-performant, modern applications.\xa0 MongoDB  is in a great position to be a central pillar of the emerging AI\ntech stack and benefit from the next wave of application development in the years ahead."\nThird Quarter Fiscal 2025 Financial Highlights\nRevenue: Total revenue was $529.4 million for the third quarter of fiscal 2025, an increase of 22% year-over-year.\nSubscription revenue was $512.2 million, an increase of 22% year-over-year, and services revenue was $17.2 million, an\nincrease of 18% year-over-year.\nGross Profit: Gross profit was $394.0 million for the third quarter of fiscal 2025, representing a 74% gross margin\ncompared to 75% in the year-ago period. Non-GAAP gross profit was $405.7 million, representing a 77% non-GAAP gross\nmargin, consistent with a non-GAAP gross margin of 77% in the year-ago period.')

Crear una instancia del recuperador

En esta sección, crea una instancia del recuperador de documentos principal y lo usa para ingerir datos en MongoDB.

MongoDBAtlasParentDocumentRetriever Divide los documentos principales en documentos secundarios más pequeños, los incrusta y luego los incorpora a la misma colección en MongoDB. En esencia, este recuperador crea lo siguiente:

• Una instancia de MongoDBAtlasVectorSearch, un almacén de vectores que maneja consultas de búsqueda de vectores a los documentos secundarios.

• Una instancia de MongoDBDocStore, un almacén de documentos que maneja el almacenamiento y la recuperación de los documentos principales.

from langchain_mongodb.retrievers import MongoDBAtlasParentDocumentRetriever
from langchain_voyageai import VoyageAIEmbeddings
# Define the embedding model to use
embedding_model = VoyageAIEmbeddings(model="voyage-3-large")
# Define the chunking method for the child documents
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)
# Specify the database and collection name
database_name = "langchain_db"
collection_name = "parent_document"
# Create the parent document retriever
parent_doc_retriever = MongoDBAtlasParentDocumentRetriever.from_connection_string(
    connection_string = MONGODB_URI,
    child_splitter = child_splitter,
    embedding_model = embedding_model,
    database_name = database_name,
    collection_name = collection_name,
    text_key = "page_content",
    relevance_score_fn = "dotProduct",
    search_kwargs = { "k": 10 },
)

parent_doc_retriever.add_documents(docs)

Cree el índice de búsqueda vectorial de MongoDB

Para habilitar consultas de búsqueda vectorial en la colección langchain_db.parent_document, debes crear un índice de búsqueda vectorial de MongoDB. Puedes usar el método asistente de LangChain o el método del driver PyMongo. Ejecuta el siguiente código en tu notebook para el método preferido:

# Get the vector store instance from the retriever
vector_store = parent_doc_retriever.vectorstore
# Use helper method to create the vector search index
vector_store.create_vector_search_index(
   dimensions = 1024 # The number of dimensions to index
)

from pymongo import MongoClient
from pymongo.operations import SearchIndexModel
# Connect to your cluster
client = MongoClient(MONGODB_URI)
collection = client[database_name][collection_name]
# Create your vector search index model, then create the index
vector_index_model = SearchIndexModel(
   definition={
      "fields": [
         {
         "type": "vector",
         "path": "embedding",
         "numDimensions": 1024,
         "similarity": "dotProduct"
         }
      ]
   },
   name="vector_index",
   type="vectorSearch"
)
collection.create_search_index(model=vector_index_model)

El índice debería tardar aproximadamente un minuto en crearse. Mientras se crea, el índice está en un estado de sincronización inicial. Cuando termine de crearse, se pueden empezar a realizar los query en los datos de la colección.

Utilice el recuperador en su tubería RAG

Una vez que MongoDB construya su índice, podrá ejecutar consultas de búsqueda vectorial en sus datos y usar el recuperador en su pipeline RAG. Pegue y ejecute el siguiente código en su notebook para implementar un pipeline RAG de ejemplo que realice la recuperación del documento principal:

parent_doc_retriever.invoke("AI technology")

[Document(metadata={'_id': '492a138c-1309-4791-a0d0-282d34ea1e55', 'source': 'https://investors.mongodb.com/node/12881/pdf', 'page': 1, 'page_label': '2'}, page_content='downturns and/or the effects of rising interest rates, inflation and volatility in the global economy and financial markets on our business and future\noperating results; our potential failure to meet publicly announced guidance or other expectations about our business and future operating results; our\nlimited operating history; our history of losses; failure of our platform to satisfy customer demands; the effects of increased competition; our\ninvestments in new products and our ability to introduce new features, services or enhancements; our ability to effectively expand our sales and\nmarketing organization; our ability to continue to build and maintain credibility with the developer community; our ability to add new customers or\nincrease sales to our existing customers; our ability to maintain, protect, enforce and enhance our intellectual property; the effects of social, ethical and\nregulatory issues relating to the use of new and evolving technologies, such as artificial intelligence, in our offerings or partnerships; the growth and\nexpansion of the market for database products and our ability to penetrate that market; our ability to integrate acquired businesses and technologies\nsuccessfully or achieve the expected benefits of such acquisitions; our ability to maintain the security of our software and adequately address privacy\nconcerns; our ability to manage our growth effectively and successfully recruit and retain additional highly-qualified personnel; and the price volatility of'),
 Document(metadata={'_id': 'a937204a-0e85-4827-ac63-124735529d51', 'source': 'https://investors.mongodb.com/node/12881/pdf', 'page': 1, 'page_label': '2'}, page_content='that it obtained the AWS Modernization Competency designation and launched a MongoDB University course focused on\nbuilding AI applications with MongoDB  and AWS. At Microsoft Ignite, MongoDB  announced new technology integrations for\nAI, data analytics, and automating database deployments across on-premises, cloud, and edge environments.\nLaunched in July 2024, the MongoDB AI Applications Program (MAAP) is designed to help companies unleash the power\nof their data and to take advantage of rapidly advancing AI technologies. We recently announced that Capgemini,\nConfluent, IBM, Unstructured, and QuantumBlack, AI by McKinsey have joined the MAAP ecosystem, offering customers\nadditional integration and solution options.\nExecutive Leadership Update\nMichael Gordon, MongoDB\'s Chief Operating Officer and Chief Financial Officer, will be stepping down at the end of the Company\'s fiscal year on\nJanuary 31, 2025, and afterwards will serve as an advisor to ensure a smooth transition. The Company has commenced an executive search process\nfor a new CFO and will evaluate internal and external candidates.\xa0 Serge Tanjga, MongoDB\'s Senior Vice President of Finance, will serve as interim\nCFO starting February 1st if a permanent successor has not been named by that date.\nDev Ittycheria commented, "On behalf of everyone at MongoDB , I want to thank Michael for everything he has done to contribute to our success in his\nnearly 10 years with the company.\xa0 In Michael\'s time here, MongoDB  had a successful IPO, has grown revenue nearly 50x and has successfully\nscaled the business model to generate meaningful operating leverage. Michael has also built out a world-class finance team that I am confident will\ndeliver a smooth transition to a new CFO in the coming months."\nMichael Gordon said, "I am incredibly proud of what we have accomplished as a team in my almost ten years with the company.\xa0 While we have')]

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import  RunnablePassthrough
from langchain_openai import ChatOpenAI
# Define a prompt template
template = """
   Use the following pieces of context to answer the question at the end.
   {context}
   Question: {query}?
"""
prompt = PromptTemplate.from_template(template)
model = ChatOpenAI()
# Construct a chain to answer questions on your data
chain = (
   {"context": parent_doc_retriever, "query": RunnablePassthrough()}
   | prompt
   | model
   | StrOutputParser()
)
# Prompt the chain
query = "In a list, what are MongoDB's latest AI announcements?"
answer = chain.invoke(query)
print(answer)

1. MongoDB obtained the AWS Modernization Competency designation.
2. MongoDB launched a MongoDB University course focused on building AI applications with MongoDB and AWS.
3. MongoDB announced new technology integrations for AI, data analytics, and automating database deployments across various environments.
4. MongoDB launched the MongoDB AI Applications Program (MAAP) to help companies harness the power of data and future AI technologies.
5. Capgemini, Confluent, IBM, Unstructured, and QuantumBlack joined the MAAP ecosystem to offer customers additional integration and solution options.

Aprende observando

Siga este video sobre la recuperación de documentos principales con LangChain y MongoDB.

Duración: 27 minutos