LangChain integration

Use Meilisearch as a LangChain vector store for semantic search with OpenAI embeddings.

LangChain is a framework for building applications powered by language models. Meilisearch integrates with LangChain as a vector store, letting you import documents with embeddings and perform similarity searches.

Requirements

This guide assumes a basic understanding of Python and LangChain. Beginners to LangChain will still find the tutorial accessible.

• Python (LangChain requires >= 3.8.1 and < 4.0) and the pip CLI
• A Meilisearch >= 1.6 project
• An OpenAI API key

Creating the application

Create a folder for your application with an empty setup.py file.

Before writing any code, install the necessary dependencies:

pip install langchain openai meilisearch python-dotenv

First create a .env to store our credentials:

# .env

MEILI_HTTP_ADDR="your Meilisearch host"
MEILI_API_KEY="your Meilisearch API key"
OPENAI_API_KEY="your OpenAI API key"

Now that you have your environment variables available, create a setup.py file with some boilerplate code:

# setup.py

import os
from dotenv import load_dotenv # remove if not using dotenv
from langchain.vectorstores import Meilisearch
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.document_loaders import JSONLoader

load_dotenv() # remove if not using dotenv

# exit if missing env vars
if "MEILI_HTTP_ADDR" not in os.environ:
    raise Exception("Missing MEILI_HTTP_ADDR env var")
if "MEILI_API_KEY" not in os.environ:
    raise Exception("Missing MEILI_API_KEY env var")
if "OPENAI_API_KEY" not in os.environ:
    raise Exception("Missing OPENAI_API_KEY env var")

# Setup code will go here 👇

Importing documents and embeddings

Now that the project is ready, import some documents in Meilisearch. First, download this small movies dataset:

movies-lite.json

Download movies-lite.json

Then, update the setup.py file to load the JSON and store it in Meilisearch. You will also use the OpenAI text search models to generate vector embeddings.

To use vector search, we need to set the embedders index setting. In this case, you are using an userProvided source which requires to specify the size of the vectors in a dimensions field. The default model used by OpenAIEmbeddings() is text-embedding-ada-002, which has 1,536 dimensions.

# setup.py

# previous code

# Load documents
loader = JSONLoader(
    file_path="./movies-lite.json",
    jq_schema=".[] | {id: .id, overview: .overview, title: .title}",
    text_content=False,
)
documents = loader.load()
print("Loaded {} documents".format(len(documents)))

# Store documents in Meilisearch
embeddings = OpenAIEmbeddings()
embedders = { 
        "custom": {
            "source": "userProvided",
            "dimensions": 1536
        }
    }
embedder_name = "custom" 
vector_store = Meilisearch.from_documents(documents=documents, embedding=embeddings, embedders=embedders, embedder_name=embedder_name)

print("Started importing documents")

Your Meilisearch instance will now contain your documents. Meilisearch runs tasks like document import asynchronously, so you might need to wait a bit for documents to be available. Consult the asynchronous operations explanation for more information on how tasks work.

Performing similarity search

Your database is now populated with the data from the movies dataset. Create a new search.py file to make a semantic search query: searching for documents using similarity search.

# search.py

import os
from dotenv import load_dotenv
from langchain.vectorstores import Meilisearch
from langchain.embeddings.openai import OpenAIEmbeddings
import meilisearch

load_dotenv()

# You can use the same code as `setup.py` to check for missing env vars

# Create the vector store
client = meilisearch.Client(
    url=os.environ.get("MEILI_HTTP_ADDR"),
    api_key=os.environ.get("MEILI_API_KEY"),
)
embeddings = OpenAIEmbeddings()
vector_store = Meilisearch(client=client, embedding=embeddings)

# Make similarity search
embedder_name = "custom"
query = "superhero fighting evil in a city at night"
results = vector_store.similarity_search(
    query=query,
    embedder_name=embedder_name,
    k=3,
)

# Display results
for result in results:
    print(result.page_content)

Run search.py. If everything is working correctly, you should see an output like this:

{"id": 155, "title": "The Dark Knight", "overview": "Batman raises the stakes in his war on crime. With the help of Lt. Jim Gordon and District Attorney Harvey Dent, Batman sets out to dismantle the remaining criminal organizations that plague the streets. The partnership proves to be effective, but they soon find themselves prey to a reign of chaos unleashed by a rising criminal mastermind known to the terrified citizens of Gotham as the Joker."}
{"id": 314, "title": "Catwoman", "overview": "Liquidated after discovering a corporate conspiracy, mild-mannered graphic artist Patience Phillips washes up on an island, where she's resurrected and endowed with the prowess of a cat -- and she's eager to use her new skills ... as a vigilante. Before you can say \"cat and mouse,\" handsome gumshoe Tom Lone is on her tail."}
{"id": 268, "title": "Batman", "overview": "Batman must face his most ruthless nemesis when a deformed madman calling himself \"The Joker\" seizes control of Gotham's criminal underworld."}

Congrats 🎉 You managed to make a similarity search using Meilisearch as a LangChain vector store.

Going further

Using Meilisearch as a LangChain vector store allows you to load documents and search for them in different ways:

• Import documents from text
• Similarity search with score
• Similarity search by vector

For additional information, consult:

Meilisearch Python SDK docs

Finally, should you want to use Meilisearch’s vector search capabilities without LangChain or its hybrid search feature, refer to the dedicated tutorial.