LangChain RAG

In the rapidly evolving landscape of Large Language Model (LLM) application development, Retrieval-Augmented Generation

In the rapidly evolving landscape of Large Language Model (LLM) application development, Retrieval-Augmented Generation (RAG) has emerged as one of the most critical architectural patterns. While LLMs are incredibly powerful, they suffer from inherent limitations: they are prone to "hallucinations," their knowledge is frozen at the point of their last training cutoff, and they lack access to your organization's private, proprietary data.

RAG solves these challenges by providing the model with a "search engine" for your specific data. Instead of relying solely on its internal weights, the model retrieves relevant information from an external knowledge base to ground its answers in fact. LangChain, the industry-leading framework for LLM orchestration, provides a highly abstracted and modular toolchain that makes implementing RAG both efficient and scalable.

In this guide, we will walk through a hands-on implementation of a complete RAG pipeline using Python and LangChain.
Before we dive into the code, we need to prepare our development environment. LangChain is modular, so we need to install the core library along with specific dependencies for OpenAI integration, vector storage, and text processing.

Run the following command in your terminal:

pip install langchain langchain-openai chromadb sentence-transformers tiktoken

Next, ensure your OpenAI API key is configured in your environment. This key will be used for both generating embeddings and driving the LLM.

import os

# Replace with your actual API key
os.environ["OPENAI_API_KEY"] = "your-openai-api-key-here"

Phase 1: Document Ingestion and Chunking

The first step in the RAG workflow is "ingestion"—the process of making your private data readable by the system. LangChain offers a diverse array of Document Loaders capable of handling various formats, including PDFs, Markdown, Notion pages, and web URLs.

For this demonstration, we will load a simple local text file:

from langchain_community.document_loaders import TextLoader

# Load the local knowledge base
loader = TextLoader("knowledge.txt", encoding="utf-8")
documents = loader.load()

print(f"Successfully loaded {len(documents)} document(s).")

However, simply loading a document is not enough. LLMs have a finite context window (the maximum amount of text they can process at once). To handle large datasets, we must break long documents into smaller, manageable pieces called chunks.

We use the RecursiveCharacterTextSplitter, which is highly effective because it attempts to split text at logical boundaries (like paragraphs and sentences) to maintain semantic coherence.

from langchain.text_splitter import RecursiveCharacterTextSplitter

# Initialize the splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,      # Maximum number of characters per chunk
    chunk_overlap=50,    # Overlap to maintain context between adjacent chunks
)

# Split the documents into smaller chunks
docs = text_splitter.split_documents(documents)
print(f"Created {len(docs)} text chunks.")

Phase 2: Vectorization and Vector Storage

Computers do not understand the "meaning" of words; they understand numbers. To enable semantic search, we must convert our text chunks into high-dimensional dense vectors using an Embedding Model. Once converted, these vectors are stored in a Vector Database, which allows us to perform lightning-fast similarity searches.

In this implementation, we use OpenAI's embedding models and Chroma, a lightweight, developer-friendly vector database.

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

# Initialize the embedding model
embeddings = OpenAIEmbeddings()

# Create the vector store and persist it locally
vectorstore = Chroma.from_documents(
    documents=docs, 
    embedding=embeddings, 
    persist_directory="./chroma_db"
)

# Transform the vector store into a retriever
# We configure it to return the top 3 most relevant chunks
retriever = vectorstore.as_retriever(
    search_type="similarity", 
    search_kwargs={"k": 3}
)

Phase 3: Constructing the RAG Chain

The "brain" of our application is the RAG Chain. When a user asks a question, the system follows a specific logic:

  1. Retrieve: Find the most relevant document chunks from the vector database based on the user's query.
  2. Augment: Inject those chunks into a predefined prompt template.
  3. Generate: Pass the augmented prompt to the LLM to produce a factual response.

We will use LangChain Expression Language (LCEL) to compose these components into a seamless pipeline.

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_openai import ChatOpenAI

# 1. Define the Prompt Template
# We instruct the model to use ONLY the provided context.
template = """You are a professional assistant. Answer the question based ONLY on the following context. 
If you do not know the answer, simply say that you do not know; do not try to make up an answer.

Context:
{context}

Question: {question}

Answer:"""

prompt = ChatPromptTemplate.from_template(template)

# 2. Initialize the LLM
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

# 3. Helper function to format retrieved documents into a single string
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# 4. Build the RAG Chain using LCEL
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Phase 4: Testing the Application

With the pipeline constructed, we can now test our application by asking a question that requires knowledge from our private knowledge.txt file.

# Example query based on your private data
query = "What is our company's return policy?"

# Execute the chain
response = rag_chain.invoke(query)

print("--- Response ---")
print(response)

When you run this, LangChain automatically performs the semantic search, pulls the relevant snippets regarding "return policy," feeds them into the prompt, and the LLM generates a response grounded in your specific data.

Summary and Future Optimizations

In this practical guide, we have successfully implemented the end-to-end RAG lifecycle: Document Loading $\rightarrow$ Text Splitting $\rightarrow$ Embedding $\rightarrow$ Vector Storage $\rightarrow$ Retrieval $\rightarrow$ Generation.

While this setup provides a solid foundation, production-grade enterprise systems often require further refinement. To take your RAG application to the next level, consider exploring these advanced techniques:

  • Hybrid Search: Combining semantic (vector) search with keyword (BM25) search to improve accuracy for specific terms.
  • Reranking: Using a specialized "Reranker" model to re-evaluate the relevance of retrieved chunks before passing them to the LLM.
  • Conversation Memory: Integrating ChatMessageHistory so the model can handle follow-up questions in a multi-turn dialogue.
  • Query Transformation: Using an LLM to rewrite user queries into more effective search terms before retrieval.