LangSmith provides the observability and evaluation infrastructure that production LLM apps require. Core features: 1) Trace collection: every LLM call, tool use, and chain step is logged with inputs, outputs, latency, and tokens. Use @traceable decorator or LANGCHAIN_TRACING_V2=true env var. 2) Dataset management: annotate production traces as ground truth, build evaluation datasets from real usage. Crucial for catching regressions. 3) Evaluations: define evaluators (LLM-as-judge, exact match, regex) and run against datasets. Compare model versions, prompts, or configurations on same dataset. 4) Online evaluation: run evaluators on production traffic sample for continuous quality monitoring. 5) Annotation workflows: route uncertain examples to human annotators, collect thumbs up/down feedback from users into LangSmith. Evaluation workflow: 1) Collect 100-200 representative queries from production. 2) Annotate expected outputs. 3) Run evaluations on current model version - establishes baseline. 4) When changing prompts or models, run evaluations on same dataset. 5) Only deploy if metrics maintain or improve. LLM-as-judge evaluator: define rubric (relevance 1-5, faithfulness 1-5, helpfulness 1-5), use GPT-4o or Claude to score, aggregate scores per dataset.