AI Application Testing and Evaluation

The Testing Challenge for AI Systems

Traditional software testing relies on deterministic outputs. AI systems introduce:

Non-deterministic responses

Hallucination risk

Quality degradation over time

Prompt injection vulnerabilities

Unit Testing LLM Calls

Mock LLM calls for fast, reliable unit tests:

python
from unittest.mock import patch, MagicMock
def test_summarization():
    mock_response = MagicMock()
    mock_response.choices[0].message.content = "Test summary"
    
    with patch('openai.chat.completions.create', return_value=mock_response):
        result = summarize_document("Long document text...")
        assert len(result) < 100
        assert isinstance(result, str)

Evaluation with RAGAS

python
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
dataset = {
    "question": ["What is RAG?", "How does fine-tuning work?"],
    "contexts": [["RAG is..."], ["Fine-tuning involves..."]],
    "answer": ["RAG stands for...", "Fine-tuning works by..."],
    "ground_truth": ["Retrieval Augmented Generation", "Training on domain data"]
}
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
print(result)

DeepEval for Comprehensive Testing

python
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, HallucinationMetric
from deepeval.test_case import LLMTestCase
def test_qa_system():
    test_case = LLMTestCase(
        input="What is the capital of France?",
        actual_output=qa_system.answer("What is the capital of France?"),
        expected_output="Paris",
        retrieval_context=["France is a country in Europe. Its capital is Paris."]
    )
    
    assert_test(test_case, [
        AnswerRelevancyMetric(threshold=0.8),
        HallucinationMetric(threshold=0.1)
    ])

Regression Testing

Track quality metrics over time and alert on degradation:

Store evaluation results in a database

Compare against baseline

Alert when metrics drop below threshold

CI/CD Integration

Run automated evaluations on every PR to catch quality regressions before deployment.