AI Evaluation Frameworks: How to Measure What Actually Matters

Why Evaluation Is the Foundation

"Vibe checking" AI outputs—trying a few prompts and saying "this seems good"—is how most teams start. It's also why most teams are surprised when production performance disappoints. Systematic evaluation is the engineering discipline that separates amateur and professional AI development.

Evaluation serves multiple purposes: catching regressions before deployment, measuring improvement from prompt changes, comparing model options objectively, understanding failure modes, demonstrating system quality to stakeholders.

Defining What to Measure

Task-Specific Metrics

Different tasks need different metrics:

Classification: accuracy, precision, recall, F1, AUC. Standard ML metrics apply directly.

Question Answering: exact match (EM), F1 over token overlap, BERTScore (semantic similarity), factuality (does answer contain correct information?).

Summarization: ROUGE (n-gram overlap with reference), BERTScore, factuality (does summary contain only facts from source?), coverage (does summary include key points?), conciseness.

Code Generation: functional correctness (does code pass test cases?), compilation success, code quality metrics (complexity, style).

Dialogue: task completion rate, human-judged coherence, factuality, helpfulness, safety.

RAG-Specific: faithfulness (answer grounded in context?), context precision (retrieved context relevant to query?), context recall (relevant documents retrieved?), answer relevance.

Calibrating Metrics to Business Value

The most important evaluation question: "does this metric predict whether users are satisfied?"

Proxy metrics (ROUGE, BERTScore) can diverge from user satisfaction. Always validate that your automatic metrics correlate with human judgment before relying on them for production decisions.

Building Your Eval Dataset

Dataset Construction Principles

Quality > Quantity: 100 carefully curated examples beats 1,000 automatically generated ones.

Coverage: represent the full distribution of real inputs:

Common cases (80%): most frequent query types

Edge cases (15%): unusual but valid inputs

Adversarial cases (5%): tricky inputs designed to reveal failures

Maintenance: eval datasets need updates. What you evaluate today becomes training signal, not eval signal. Refresh with new examples quarterly.

Sourcing Eval Examples

Real user queries: gold standard. Sample from production logs (with privacy protections). Curated by domain experts: for specialized domains (medical, legal), expert-created examples are essential. Adversarial generation: use LLM to generate challenging inputs. "Generate 50 tricky questions about [domain] that might confuse an AI assistant." Existing benchmarks: BIG-Bench, MMLU, HellaSwag, etc. Use as a component, not replacement for task-specific evals.

Annotation Guidelines

For human-labeled eval sets:

Write explicit annotation guidelines with examples of each label

Adjudicate disagreements (not just majority vote)

Track inter-annotator agreement (Cohen's kappa > 0.7 acceptable)

Regular annotator calibration sessions

Automated Evaluation

LLM-as-Judge

Use a powerful LLM (GPT-4o, Claude 3.5 Sonnet) to evaluate outputs from a system under test. Enables:

Evaluation of open-ended responses (no reference answer needed)

Rapid evaluation at scale

Nuanced quality assessment

python
def evaluate_response(question: str, response: str, rubric: str) -> dict:
    eval_prompt = f"""
    Evaluate the following AI response on these criteria:
    {rubric}
    
    Question: {question}
    Response: {response}
    
    Score each criterion 1-5 and provide brief justification.
    Return as JSON: {{"criterion1": {{"score": X, "reason": "..."}}, ...}}
    """
    
    result = llm.complete(eval_prompt)
    return json.loads(result)

Limitations: LLM judges have biases (prefer longer responses, own-style outputs, confident tone). Mitigate: use multiple evaluator models, test for known biases, calibrate against human judgments.

Reference-Based Metrics

When you have a reference (expected answer): automated metrics are cheap and fast.

For text: BERTScore (semantic similarity), ROUGE (n-gram overlap), BLEU (for translation). For factuality: NLI-based entailment checking (does response entail the correct facts?). For code: execution against test suite.

RAG Evaluation with RAGAS

RAGAS (Retrieval Augmented Generation Assessment) is the standard framework for evaluating RAG pipelines:

Faithfulness: are claims in the answer supported by retrieved context? (0-1 score) Answer Relevance: does the answer address the question asked? (0-1 score) Context Precision: are retrieved chunks relevant to the question? (0-1 score) Context Recall: were all relevant documents retrieved? (0-1 score)

python
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset
Your RAG test data
data = {
    "question": ["What is RAG?"],
    "answer": ["RAG stands for..."],
    "contexts": [["Retrieved chunk 1...", "Retrieved chunk 2..."]],
    "ground_truth": ["The correct answer..."]
}
dataset = Dataset.from_dict(data)
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision, context_recall])
print(result)

Human Evaluation Methodology

When to use human eval: output quality is subjective (tone, style, helpfulness), automatic metrics diverge from user satisfaction, high-stakes deployment requiring human sign-off.

Side-by-Side Comparison

Show evaluators two outputs (from current and new system), ask which is better. Reduces absolute judgment difficulty.

Randomize order to prevent position bias. Use 3+ evaluators per comparison. Calculate win rate for statistical significance.

Absolute Rating

Rate output on 1-5 scale per dimension. Requires calibration (what does "3" mean?).

Use: when comparing to absolute standard (minimum acceptable quality), or when tracking improvement over time.

User Study

Real users completing real tasks, measured by: task completion rate, time on task, error rate, satisfaction survey.

Most expensive but most valid for production quality decisions.

Regression Testing and CI/CD Integration

Regression Test Suite

For every significant model or prompt change: run standard eval suite, compare to previous baseline, fail if metrics degrade beyond threshold.

yaml
Example GitHub Actions eval check
name: Run AI Eval Suite
  run: |
    python eval_suite.py --model=gpt-4o --output=results.json
    python check_regression.py --baseline=baseline.json --results=results.json --threshold=0.02
  # Fails if any metric degrades >2% from baseline

Eval-Driven Development

Workflow: measure current performance → identify failure modes → hypothesis for improvement → implement → run eval → measure improvement → iterate.

Never change prompts or models without running evals. The improvement in one dimension you targeted may cause regression in another.

Evaluation Tools and Platforms

RAGAS: RAG-specific evaluation. Python library, open source.

DeepEval: general LLM evaluation framework. Multiple metrics, CI/CD integration, test management.

Promptfoo: prompt testing and evaluation. Config-file-based test definitions, multiple providers, diff view between versions.

HELM (Stanford): comprehensive LLM benchmark suite. Academic standard for model comparison.

LangSmith: evaluation integrated with LangChain tracing. Annotation queues for human eval, automated evals, regression tracking.

Braintrust: evaluation platform with dataset management, human review workflow, CI integration.