Building Production RAG Systems with LangChain

What Makes RAG "Production-Ready"?

Most tutorials stop at the prototype — a chatbot that answers questions from PDFs. Production RAG means:

Reliable, consistent answers (not 80% correct)

Sub-2-second response times

Graceful handling of out-of-scope questions

Monitoring and alerting for quality degradation

Cost management at scale

Architecture Overview

User Query
    ↓
Query preprocessing & expansion
    ↓
Retrieval (vector + keyword hybrid)
    ↓
Reranking
    ↓
Context construction
    ↓
LLM generation
    ↓
Response validation
    ↓
User

Component 1: Document Processing

Chunking Strategy (Critical)

Poor chunking = poor retrieval. Strategies:

Recursive character splitting (default, but not always best):

python
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "]
)

Semantic chunking (better for complex docs):

python
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai.embeddings import OpenAIEmbeddings
chunker = SemanticChunker(OpenAIEmbeddings())
Groups semantically similar sentences together

Best practice: Test both with your specific documents using retrieval evaluation.

Component 2: Vector Store Selection

StoreBest ForScaleCost

PineconeManaged, no ops100M+ vectors$$$ WeaviateHybrid search10M+ vectors$$ QdrantSelf-hostedAny$ pgvectorAlready on Postgres<1M vectors$ ChromaDevelopment/small<500K vectorsFree

Hybrid Search Implementation

python
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
bm25_retriever = BM25Retriever.from_documents(docs, k=5)
hybrid_retriever = EnsembleRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    weights=[0.7, 0.3]  # Vector search weighted higher
)

Component 3: Reranking

Reranking dramatically improves precision:

python
from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank
reranker = CohereRerank(top_n=3)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=hybrid_retriever
)

Impact: Reranking typically improves answer accuracy by 15-30%.

Evaluation Framework

Using RAGAS

python
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
result = evaluate(
    dataset=evaluation_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision]
)
faithfulness: Are claims supported by context?
answer_relevancy: Does answer address the question?
context_precision: Is retrieved context relevant?

Target Scores

Faithfulness: >0.90 (critical for factual accuracy)

Answer relevancy: >0.85

Context precision: >0.80

Production Monitoring

python
import langsmith
with langsmith.trace("rag-query") as run:
    result = rag_chain.invoke(query)
    run.add_metadata({
        "retrieval_score": result.retrieval_score,
        "response_time_ms": result.time_ms,
        "user_feedback": None  # Updated when received
    })

Alerts to configure:

Retrieval score < 0.7 for >10% of queries

Response time P95 > 3 seconds

LLM errors > 1%

Unexpected topic drift

Cost Optimization

OptimizationSavings

Semantic caching30-60% on repeated queries Smaller embedding model (ada-002 → text-3-small)15% Reduce chunk count retrieved20% Model tiering (Haiku for simple, Opus for complex)40%