LLM Application Architecture Patterns: From Simple to Complex Systems

Building production-grade LLM applications requires moving beyond a single prompt call. As complexity grows, you need architectural patterns that handle context, tool use, reliability, and cost. This guide walks through the key patterns—from basic retrieval to full agent systems—with practical trade-offs and selection advice.

1. The Foundation: Single LLM Call

The simplest pattern: one prompt, one response. Suitable for classification, summarization, or simple generation where no external data is needed.

User → Prompt → LLM → Response

When to use: Prototyping, stateless tasks, no need for context or tools. Limitation: No memory, no external knowledge, no multi-step reasoning.

2. Retrieval-Augmented Generation (RAG)

RAG addresses the LLM's lack of up-to-date or domain-specific knowledge by injecting relevant documents into the prompt context.

Architecture:

User Query → Embedding Model → Vector DB (retrieve top-k chunks)
         ↓
Prompt = System + Retrieved Chunks + User Query
         ↓
         LLM → Response

Key components:

Embedding model: Converts text to vectors (e.g., text-embedding-3-small, BGE).

Vector database: Stores and indexes embeddings (e.g., Pinecone, Weaviate, Qdrant, pgvector).

Chunking strategy: How you split documents (semantic chunking, fixed-size with overlap).

Retrieval: Typically top-k by cosine similarity; can add hybrid search (BM25 + vector).

Trade-offs:

Chunk size: Larger chunks give more context but dilute relevance; smaller chunks risk missing context.

Retrieval quality: Low recall means the LLM lacks needed info; low precision wastes tokens on noise.

Latency: Embedding + retrieval adds 100-500ms; caching can help.

Common pitfalls:

Retrieving chunks that don't answer the query (use reranking or query rewriting).

Not handling cases where no relevant chunk exists (fallback to "I don't know").

Token overflow from too many chunks (limit to 3-5 chunks, or use sliding window).

3. Tool Calling / Function Calling

LLMs can request external actions—like database queries, API calls, or calculations—by outputting structured function calls.

Architecture:

User → LLM (with tool definitions) → Tool call request
         ↓
         Execute tool (e.g., SQL query, weather API)
         ↓
         Tool result → LLM → Final response

Real tools:

Database query: get_user_orders(user_id) → returns order history.

Calculator: calculate(expression) → returns numeric result.

Web search: search(query) → returns top results.

Code interpreter: Execute Python and return output.

Pattern variants:

Single-turn tool use: One tool call, then respond.

Multi-turn: LLM calls tools sequentially, building on previous results (e.g., first search, then summarize).

Trade-offs:

Tool definition quality: Poor descriptions cause wrong tool selection.

Error handling: Tool failures (timeout, invalid input) must be caught and reported to the LLM.

Cost: Each tool call round-trip adds latency and token usage.

4. Routing (Model Router / Intent Router)

Route different types of queries to specialized handlers—different prompts, models, or even entirely different systems.

Architecture:

User Query → Classifier (LLM or lightweight model)
         ↓
         ├── "math" → Calculator tool
         ├── "summarize" → RAG pipeline
         ├── "chat" → General LLM
         └── "code" → Code interpreter

Routing methods:

LLM-based: Ask the LLM to classify intent (e.g., "You are a router. Output one of: math, summarize, chat, code").

Embedding + classifier: Embed query, compare to known intent centroids.

Regex/keyword: Simple but brittle.

When to use:

Different tasks need different prompts or models (e.g., cheap model for simple Q&A, expensive model for complex reasoning).

You want to isolate failure modes (e.g., math errors don't affect chat).

Trade-offs:

Router accuracy: Misrouting leads to poor responses. Use confidence thresholds with fallback.

Latency overhead: Router adds 50-200ms.

Maintenance: Each route is a separate pipeline to test and monitor.

5. Caching

Reduce cost and latency by caching LLM responses for identical or similar queries.

Types:

Exact match cache: Same input → same output. Simple but low hit rate.

Semantic cache: Embed query, find nearest neighbor above similarity threshold. Higher hit rate but risk of stale or incorrect matches.

Prefix cache: Cache responses for common prefixes (e.g., "What is the capital of...").

Implementation:

Use Redis or Memcached for exact cache.

Use vector DB for semantic cache (store query embedding + response).

Set TTL (time-to-live) to avoid stale data.

Trade-offs:

Semantic cache: Tuning similarity threshold is critical—too low returns wrong answers, too high misses valid matches.

Cache invalidation: When underlying data changes, cached responses become stale.

Cost vs. benefit: Cache only for high-frequency, deterministic queries.

6. Guardrails (Input/Output Validation)

Prevent harmful, off-topic, or malformed inputs and outputs.

Input guardrails:

Content moderation: Block profanity, PII, or malicious prompts.

Topic restriction: Reject queries outside allowed domain (e.g., "I can only answer about company policy").

Format validation: Ensure input matches expected schema (e.g., JSON).

Output guardrails:

Factuality check: Compare output against retrieved documents (e.g., "Did the LLM hallucinate?").

Safety filter: Block toxic, biased, or dangerous content.

Format compliance: Ensure output is valid JSON, follows instructions.

Implementation:

Use dedicated guardrail libraries (e.g., Guardrails AI, NVIDIA NeMo Guardrails).

Or build custom: regex, LLM-as-judge, embedding similarity to banned phrases.

Trade-offs:

False positives: Overly strict guardrails block legitimate use.

Latency: Each guardrail check adds time (especially LLM-as-judge).

Maintenance: Guardrails need regular updates as new edge cases emerge.

7. Evaluation (Evals)

Measure LLM output quality systematically, not by gut feeling.

Key metrics:

Correctness: Exact match, F1, or LLM-as-judge (e.g., "Rate answer accuracy 1-5").

Faithfulness: Does the answer contradict the provided context?

Relevance: Is the answer on-topic?

Safety: Does the output contain harmful content?

Evaluation pipeline:

Test Dataset (queries + expected answers)
         ↓
         Run LLM pipeline → Get outputs
         ↓
         Compare outputs to expected (automated or human)
         ↓
         Aggregate metrics → Report

Tools: LangSmith, Weights & Biases, MLflow, custom scripts.

Trade-offs:

LLM-as-judge: Cheap but can be biased (prefers longer answers, agrees with its own style).

Human evaluation: Gold standard but expensive and slow.

Test coverage: Hard to cover all edge cases; focus on critical paths.

8. Observability (Monitoring & Tracing)

Understand what your LLM app is doing in production.

What to track:

Latency: Per component (embedding, retrieval, LLM call, tool execution).

Token usage: Input/output tokens per request.

Cost: Estimated cost per request.

Error rates: Tool failures, guardrail blocks, LLM timeouts.

User feedback: Thumbs up/down, ratings.

Tracing: See the full chain of events for a single request (e.g., "User query → router → RAG → LLM → guardrail → response").

Tools: LangSmith, Langfuse, Helicone, Datadog, OpenTelemetry.

Trade-offs:

Overhead: Tracing adds latency and storage cost.

Sampling: Trace only a fraction of requests (e.g., 1%) to reduce cost.

Privacy: Avoid logging PII in traces.

9. Monolith vs. Pipeline (Microservices)

Monolith: All components in one service (e.g., FastAPI app with embedding, vector DB client, LLM call, guardrails).

Pipeline: Separate services for each component (e.g., embedding service, retrieval service, LLM service, guardrail service), communicating via message queue or HTTP.

AspectMonolithPipeline

ComplexityLowHigh LatencyLower (no network hops)Higher (inter-service calls) ScalabilityScale entire appScale individual components Fault isolationOne bug takes down allComponent failure isolated Development speedFast initiallySlower, more coordination TestingEasier (single process)Harder (integration tests)

When to choose:

Monolith: Prototyping, low traffic (<100 req/s), small team.

Pipeline: High traffic, need to scale components independently, multiple teams.

10. Putting It All Together: A Realistic Architecture

For a production customer support chatbot:

User → Load Balancer
         ↓
    API Gateway (auth, rate limit)
         ↓
    Router (classify intent: billing, technical, general)
         ↓
    ├── Billing → RAG (policy docs) + Guardrails (no financial advice)
    ├── Technical → Tool calling (DB lookup, log search) + Guardrails (no code execution)
    └── General → Chat LLM + Guardrails (safety filter)
         ↓
    Cache (exact + semantic, TTL 1 hour)
         ↓
    LLM Call (with context from RAG/tools)
         ↓
    Output Guardrails (factuality, safety)
         ↓
    Response → User
         ↓
    Observability (trace, log, metrics)
         ↓
    Evals (offline, on sampled production data)

Selection Guide

PatternWhen to UseWhen to Skip

RAGNeed domain knowledge, up-to-date infoSimple Q&A with no external data Tool callingNeed real-time data, actions, calculationsPure text generation RouterMultiple distinct tasks, cost optimizationSingle use case CacheHigh repeat query rate, deterministic responsesUnique queries, dynamic data GuardrailsSafety-critical, regulated domainsInternal tools, low-risk use EvalsProduction deployment, quality monitoringPrototyping ObservabilityAny production systemPrototyping

Common Pitfalls to Avoid

Over-engineering: Start with a single LLM call, add patterns only when needed.

Ignoring latency: Each pattern adds time; measure end-to-end before optimizing.

Neglecting error handling: Tools fail, LLMs hallucinate, caches miss. Plan for all.

Skipping evals: Without metrics, you can't improve.

Not monitoring cost: Token usage adds up fast; set budgets and alerts.

FAQ

Q: Should I use RAG or fine-tuning? A: RAG is better for dynamic knowledge (docs change frequently) and when you need to cite sources. Fine-tuning is better for style, tone, or domain-specific behavior that doesn't change often. Many systems use both: fine-tune for behavior, RAG for facts.

Q: How do I handle LLM hallucination in production? A: Combine RAG (ground in facts), output guardrails (check against context), and evals (measure faithfulness). No single solution is perfect; use layered defenses.

Q: What's the best vector database for RAG? A: Depends on scale and infrastructure. For small apps, pgvector (PostgreSQL extension) is simple. For large scale, Pinecone or Weaviate. For self-hosted, Qdrant or Milvus. Always benchmark with your data.

Q: How do I decide between monolith and microservices? A: Start monolith. Split when you need to scale components independently, have multiple teams, or hit latency bottlenecks. Premature microservices add complexity without benefit.

Q: What's the minimum observability I need? A: Log every request (query, response, latency, tokens used). Track error rates and user feedback. Add tracing when debugging complex multi-step flows. Start simple, expand as needed.

*Last updated: July 2026. Always verify against each tool's official docs.*