AI Agents in Production: Architecture Patterns and Reliability Engineering

The Agent Reliability Challenge

AI agents introduce a new category of production engineering problems. Unlike deterministic software where the same input always produces the same output, agents:

Make probabilistic decisions that can be different each run

Execute long action sequences where early errors compound

Use external tools that can fail, return unexpected results, or change behavior

Can get into loops, dead ends, or produce unexpected results

May cause real-world side effects (sending emails, modifying databases, making API calls)

Building reliable agents requires rethinking several software engineering fundamentals.

Fundamental Agent Architecture

The Perception-Decision-Action Loop

All agents follow the same basic loop:

Perceive current state (context, previous actions, tool results)

Decide next action (LLM reasoning over available tools)

Execute action (call tool, record result)

Update state (add action + result to context)

Check completion (goal reached? Max steps? Human review needed?)

Loop or return

Minimal Footprint Principle

Anthropic's guideline for safe agents: take the minimum necessary actions, prefer reversible over irreversible actions, escalate to humans when uncertain.

Implementation: design tools to be reversible where possible (move to trash vs. delete permanently), require explicit confirmation for high-impact actions (send email? confirm before sending), implement "dry run" mode for debugging.

Tool Design for Reliability

Tool Interface Design

Well-designed agent tools:

Clear, specific function name (search_customer_records not query_db)

Descriptive docstring explaining exactly what the tool does and doesn't do

Typed parameters with validation

Idempotent where possible (calling twice has same effect as calling once)

Returns structured data (not free-form text that requires parsing)

Includes error information in return (don't just raise exceptions)

python
from pydantic import BaseModel, Field
from typing import Optional
class CustomerSearchResult(BaseModel):
    found: bool
    customer_id: Optional[str] = None
    name: Optional[str] = None
    email: Optional[str] = None
    error: Optional[str] = None
def search_customer_by_email(email: str) -> CustomerSearchResult:
    """
    Search for a customer by their email address.
    Returns customer details if found, or found=False if no customer with that email exists.
    Does NOT create new customers or modify any data.
    """
    try:
        customer = db.customers.find_one({"email": email})
        if customer:
            return CustomerSearchResult(
                found=True,
                customer_id=str(customer["_id"]),
                name=customer["name"],
                email=customer["email"]
            )
        return CustomerSearchResult(found=False)
    except Exception as e:
        return CustomerSearchResult(found=False, error=str(e))

Error Handling in Tools

Tools will fail. Design for it:

Return errors as data (not exceptions) so agent can reason about them

Include enough error context for the agent to decide next action

Distinguish retryable vs. non-retryable errors

Implement timeouts on all external calls

Tool Permissions Model

Not all agents need all tools. Use least-privilege principle:

Define tool sets per agent role (customer service agent gets read tools + send email; billing agent gets read + write billing tools)

Separate read and write tools explicitly

Require human approval for high-risk tool categories (deletes, large financial transactions)

State Management for Long-Running Agents

Why State Management Matters

LLM context windows are limited. A 100-step agent workflow can't fit in a single context. Long-running agents need persistent state.

State components:

Working memory: current task, recent actions, tool results

Long-term memory: facts learned during execution, user preferences

Episodic memory: log of all actions taken (for audit and debugging)

External state: changes made to external systems (so agent can reason about current state)

Checkpointing

Save agent state after each significant action. Benefits: resume from checkpoint if interrupted, audit trail of all decisions, debugging (reproduce agent state at any point).

LangGraph: built-in checkpointing via SQLite, Redis, or custom backends. Agents can be interrupted, resumed, and forked from any checkpoint.

python
from langgraph.checkpoint.sqlite import SqliteSaver
memory = SqliteSaver.from_conn_string("agent_checkpoints.db")
graph = StateGraph(AgentState)
... add nodes and edges ...
app = graph.compile(checkpointer=memory)
Resume agent from previous checkpoint
config = {"configurable": {"thread_id": "task-123"}}
result = app.invoke(input, config=config)

Human-in-the-Loop Patterns

When to Insert Human Review

Not all agent decisions should be autonomous. Insert human review for:

Irreversible high-impact actions (delete customer data, send mass email, large financial transaction)

Low-confidence decisions (agent indicates uncertainty, multiple valid paths)

Escalated customer service (frustrated user, complex edge case)

Scheduled review gates (complete phase A autonomously, review before phase B)

Implementation

LangGraph interrupt: built-in mechanism to pause execution, serialize state, await human decision, resume.

python
Agent pauses here awaiting human approval
tool_result = interrupt({
    "type": "approval_required",
    "action": "send_email",
    "details": {"to": customer_email, "subject": "...", "body": "..."},
    "risk_level": "medium"
})
Execution resumes when human approves/rejects/modifies

User interface: build a simple approval dashboard that shows: pending decisions, context, recommended action, approve/reject/modify options.

Observability for Agents

What to Trace

Standard APM (Application Performance Monitoring) doesn't capture agent-specific information. You need:

Every LLM call: input prompt, output, model, token usage, latency

Every tool call: tool name, parameters, result, latency, success/failure

Agent decisions: what the agent decided and why

State changes: what changed in agent state

Errors and recovery attempts

Tracing Stack

LangSmith: purpose-built for LLM application observability. Full trace tree for every agent run. Token usage, latency, model calls, tool calls. Production monitoring with alerts.

OpenTelemetry: standard observability protocol. LangChain and LangGraph support OTEL export. Send to Jaeger, Zipkin, Datadog, or any OTEL backend.

AgentOps: newer observability platform specifically for agent deployments.

Agent-Specific Metrics

Task completion rate: % of agent runs that reach goal

Step count distribution: how many steps does the agent typically take? (outliers = problems)

Tool failure rate by tool: which tools fail most?

Human escalation rate: % of runs requiring human review

Cost per task: tokens + time

Testing Non-Deterministic Agents

The Testing Challenge

You can't write deterministic unit tests for agents. The same input may produce different output. How do you test?

Behavioral testing: test that agent achieves the goal, not the specific path. "Agent should successfully complete order refund" with multiple valid execution paths.

Mock tool testing: replace real tools with mocks. Test agent reasoning in isolation from external services. Test error handling by injecting tool failures.

Trajectory analysis: record successful agent traces. Test that new versions follow similar high-level patterns (visit same tools, reach same decision points).

Eval suites: define 50-100 representative tasks with clear success criteria. Run regularly. Measure pass rate across versions.

Chaos testing: inject random tool failures, timeouts, unexpected responses. Verify agent handles gracefully without catastrophic failure.

Security Considerations

Prompt injection: user-controlled input that redirects agent to attacker's instructions. Mitigation: clearly separate system and user content, validate agent actions against original task scope.

Tool abuse: agent with broad permissions can cause damage if manipulated. Mitigation: least-privilege tool permissions, action confirmation for high-risk operations.

Data exfiltration: agent that can read and write could be manipulated to exfiltrate data. Mitigation: separate read and write permissions, audit all write operations.

Resource exhaustion: agents can loop indefinitely. Mitigation: step limits, cost limits, time limits with graceful termination.