Realice una búsqueda híbrida con MongoDB y LangChain

Requisitos previos

Para completar este tutorial, debes tener lo siguiente:

• Uno de los siguientes tipos de clúster de MongoDB:

  • Un clúster de Atlas que ejecuta la versión 6.0.11, 7.0.2 o posterior de MongoDB. Es necesario garantizar que la dirección IP esté incluida en la lista de acceso del proyecto Atlas.

  • A local Atlas deployment created using Python and Docker. Install atlas-local-lib-py (pip install atlas-local-lib-py) to programmatically create and manage local deployments. To learn more, see the atlas-local-lib-py repository.

  • A MongoDB Community cluster with Search and Vector Search installed.

• Una clave de API de Voyage AI. Para crear una clave de API, consulta Gestionar claves de API de modelos de Voyage AI.

• Una llave de API de OpenAI. Debes tener una cuenta de OpenAI con créditos disponibles para las solicitudes de API. Para obtener más información sobre cómo registrar una cuenta de OpenAI, consulta el sitio web de la API de OpenAI.

• Un entorno para ejecutar cuadernos interactivos de Python como Colab.

Configurar el entorno

Configura el entorno para este tutorial. Cree un cuaderno interactivo de Python guardando un archivo con la extensión .ipynb. Este cuaderno te permite ejecutar snippets de código Python de forma individual y lo utilizarás para ejecutar el código en este tutorial.

Para configurar tu entorno de notebook:

pip install --quiet --upgrade langchain langchain-community langchain-core langchain-mongodb langchain-voyageai langchain-openai pymongo pypdf

import os
os.environ["VOYAGE_API_KEY"] = "<voyage-api-key>"
os.environ["OPENAI_API_KEY"] = "<openai-api-key>"
MONGODB_URI = "<connection-string>"

mongodb+srv://<db_username>:<db_password>@<clusterName>.<hostname>.mongodb.net

mongodb://localhost:<port-number>/?directConnection=true

Usa MongoDB como Vector Store

Debes utilizar MongoDB como una memoria vectorial para tus datos. Puede instanciar un almacén vectorial usando una colección existente en MongoDB.

from langchain_mongodb import MongoDBAtlasVectorSearch
from langchain_voyageai import VoyageAIEmbeddings
# Create the vector store
vector_store = MongoDBAtlasVectorSearch.from_connection_string(
   connection_string = MONGODB_URI,
   embedding = VoyageAIEmbeddings(model = "voyage-3-large", output_dimension = 2048),
   namespace = "sample_mflix.embedded_movies",
   text_key = "plot",
   embedding_key = "plot_embedding_voyage_3_large",
   relevance_score_fn = "dotProduct"
)

Cree los índices

Para habilitar consultas de búsqueda híbrida en tu almacén de vectores, crea un MongoDB Vector Search y un índice MongoDB Search en la colección. Puedes crear los índices usando los métodos auxiliares de LangChain o el método del Controlador PyMongo:

# Use helper method to create the vector search index
vector_store.create_vector_search_index(
   dimensions = 2048 # The dimensions of the vector embeddings to be indexed
)

from langchain_mongodb.index import create_fulltext_search_index
from pymongo import MongoClient
# Connect to your cluster
client = MongoClient(MONGODB_URI)
# Use helper method to create the search index
create_fulltext_search_index(
   collection = client["sample_mflix"]["embedded_movies"],
   field = "plot",
   index_name = "search_index"
)

from pymongo import MongoClient
from pymongo.operations import SearchIndexModel
# Connect to your cluster
client = MongoClient(MONGODB_URI)
collection = client["sample_mflix"]["embedded_movies"]
# Create your vector search index model, then create the index
vector_index_model = SearchIndexModel(
   definition={
      "fields": [
         {
         "type": "vector",
         "path": "plot_embedding_voyage_3_large",
         "numDimensions": 2048,
         "similarity": "dotProduct"
         }
      ]
   },
   name="vector_index",
   type="vectorSearch"
)
collection.create_search_index(model=vector_index_model)

1
# Create your search index model, then create the search index
2
search_index_model = SearchIndexModel(
3
   definition={
4
      "mappings": {
5
            "dynamic": False,
6
            "fields": {
7
               "plot": {
8
                  "type": "string"
9
               }
10
            }
11
      }
12
   },
13
   name="search_index"
14
)
15
collection.create_search_index(model=search_index_model)

The indexes should take about one minute to build. While they build, the indexes are in an initial sync state. When they finish building, you can start querying the data in your collection.

Ejecutar una consulta de búsqueda híbrida

Once MongoDB builds your indexes, you can run hybrid search queries on your data. The following code uses the MongoDBAtlasHybridSearchRetriever retriever to perform a hybrid search for the string "time travel". It also specifies the following parameters:

• vectorstoreNombre de la instancia del almacén de vectores.

• search_index_nameNombre del índice de búsqueda de MongoDB.

• top_kLa cantidad de documentos a devolver.

• fulltext_penaltyLa penalización por la búsqueda de texto completo.

  Cuanto menor es la penalización, mayor es la puntuación de la búsqueda de texto completo.

• vector_penalty: La sanción para la búsqueda vectorial.

  Cuanto menor sea la penalización, mayor será la puntuación de búsqueda vectorial.

El recuperador devuelve una lista de documentos ordenados por la suma de la puntuación de búsqueda de texto completo y la puntuación de búsqueda vectorial. El resultado final del ejemplo de código incluye el título, la gráfica y las diferentes puntuaciones para cada documento.

Para aprender más sobre los resultados de las consultas de búsqueda híbridas, consulta Acerca de la query.

from langchain_mongodb.retrievers.hybrid_search import MongoDBAtlasHybridSearchRetriever
# Initialize the retriever
retriever = MongoDBAtlasHybridSearchRetriever(
    vectorstore = vector_store,
    search_index_name = "search_index",
    top_k = 5,
    fulltext_penalty = 50,
    vector_penalty = 50,
    post_filter=[
        {
            "$project": {
                "plot_embedding": 0,
                "plot_embedding_voyage_3_large": 0
            }
        }
    ])
# Define your query
query = "time travel"
# Print results
documents = retriever.invoke(query)
for doc in documents:
   print("Title: " + doc.metadata["title"])
   print("Plot: " + doc.page_content)
   print("Search score: {}".format(doc.metadata["fulltext_score"]))
   print("Vector Search score: {}".format(doc.metadata["vector_score"]))
   print("Total score: {}\n".format(doc.metadata["fulltext_score"] + doc.metadata["vector_score"]))

Title: Timecop
Plot: An officer for a security agency that regulates time travel, must fend for his life against a shady politician who has a tie to his past.
Search score: 0.019230769230769232
Vector Search score: 0.018518518518518517
Total score: 0.03774928774928775
Title: A.P.E.X.
Plot: A time-travel experiment in which a robot probe is sent from the year 2073 to the year 1973 goes terribly wrong thrusting one of the project scientists, a man named Nicholas Sinclair into a...
Search score: 0.018518518518518517
Vector Search score: 0.018867924528301886
Total score: 0.0373864430468204
Title: About Time
Plot: At the age of 21, Tim discovers he can travel in time and change what happens and has happened in his own life. His decision to make his world a better place by getting a girlfriend turns out not to be as easy as you might think.
Search score: 0
Vector Search score: 0.0196078431372549
Total score: 0.0196078431372549
Title: The Time Traveler's Wife
Plot: A romantic drama about a Chicago librarian with a gene that causes him to involuntarily time travel, and the complications it creates for his marriage.
Search score: 0.0196078431372549
Vector Search score: 0
Total score: 0.0196078431372549
Title: Retroactive
Plot: A psychiatrist makes multiple trips through time to save a woman that was murdered by her brutal husband.
Search score: 0
Vector Search score: 0.019230769230769232
Total score: 0.019230769230769232

Pasar resultados a una pipeline RAG

Puedes pasar los resultados de tu búsqueda híbrida a tu pipeline RAG para generar respuestas sobre los documentos recuperados. El código de muestra hace lo siguiente:

• Defines a LangChain prompt template to instruct the LLM to use the retrieved documents as context for your query. LangChain passes these documents to the {context} input variable and your query to the {query} variable.

• Construye una cadena que especifica lo siguiente:

  • El buscador híbrido que definiste para recuperar documentos relevantes.

  • La plantilla de prompt que definiste.

  • Un LLM de OpenAI para generar una respuesta consciente del contexto. Por defecto, este es el modelo gpt-3.5-turbo.

• Solicita a la cadena una query de muestra y devuelve la respuesta. La respuesta generada puede variar.

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import PromptTemplate
from langchain_core.runnables import  RunnablePassthrough
from langchain_openai import ChatOpenAI
# Define a prompt template
template = """
   Use the following pieces of context to answer the question at the end.
   {context}
   Question: Can you recommend some movies about {query}?
"""
prompt = PromptTemplate.from_template(template)
model = ChatOpenAI()
# Construct a chain to answer questions on your data
chain = (
   {"context": retriever, "query": RunnablePassthrough()}
   | prompt
   | model
   | StrOutputParser()
)
# Prompt the chain
query = "time travel"
answer = chain.invoke(query)
print(answer)

Certainly! Here are some movies about time travel from the context provided:
1. **Timecop (1994)**
   Genre: Action, Crime, Sci-Fi
   Plot: A law enforcement officer working for the Time Enforcement Commission battles a shady politician with a personal tie to his past.
   IMDb Rating: 5.8
2. **A.P.E.X. (1994)**
   Genre: Action, Sci-Fi
   Plot: A time-travel experiment gone wrong thrusts a scientist into an alternate timeline plagued by killer robots.
   IMDb Rating: 4.3
3. **About Time (2013)**
   Genre: Drama, Fantasy, Romance
   Plot: A young man discovers he can time travel and uses this ability to improve his life, especially his love life, but learns the limitations and challenges of his gift.
   IMDb Rating: 7.8
4. **The Time Traveler's Wife (2009)**
   Genre: Drama, Fantasy, Romance
   Plot: A Chicago librarian with a gene causing him to involuntarily time travel struggles with its impact on his romantic relationship and marriage.
   IMDb Rating: 7.1
5. **Retroactive (1997)**
   Genre: Action, Crime, Drama
   Plot: A woman accidentally time-travels to prevent a violent event, but her attempts to fix the situation lead to worsening consequences due to repeated time cycles.
   IMDb Rating: 6.3
Each movie covers time travel with unique perspectives, from action-packed adventures to romantic dramas.