Claude Vision: Image and Document Analysis Implementation Guide

Claude's vision capability takes images (and PDFs) directly in the messages API — screenshots, charts, photos, scanned documents — and reasons over them with the same model that handles your text. The current generation handles high-resolution input well (recent Opus models accept up to ~2576px on the long edge with pixel-accurate coordinate understanding, per Anthropic's docs), which unlocked the chart-reading and document-QA workloads this guide implements.

The basic call

python
import base64
from anthropic import Anthropic
client = Anthropic()
with open('dashboard.png', 'rb') as f:
    image_data = base64.standard_b64encode(f.read()).decode()
response = client.messages.create(
    model='claude-opus-4-8',
    max_tokens=16000,
    messages=[{
        'role': 'user',
        'content': [
            {'type': 'image',
             'source': {'type': 'base64', 'media_type': 'image/png', 'data': image_data}},
            {'type': 'text',
             'text': 'What changed in this metrics dashboard vs normal? List anomalies with the panel each appears in.'}
        ],
    }],
)
print(response.content[0].text)

URL sources work too ('source': {'type': 'url', 'url': 'https://...'}), and multiple images per message are supported — useful for "compare these two screenshots" tasks. Supported formats: JPEG, PNG, GIF, WebP.

The workloads it's genuinely good at

Document understanding: invoices, receipts, forms — extraction *with layout awareness* (which label belongs to which value), where classic OCR gives you a bag of strings. For multi-page PDFs, pass the document directly (PDF support) rather than rasterizing pages yourself.

Chart/graph reading: pulling values, trends, and outliers out of plotted data — ask for the data *as a table* and you've inverted a chart back into numbers.

UI screenshots: bug-report triage ("what error is shown, what state is the form in"), accessibility review, and the perception layer of computer-use agents.

Real-world photos: damage assessment, shelf audits, equipment identification — anywhere a junior analyst would describe a photo.

Production patterns

1. Structured extraction, validated. Vision + structured output is the production combo — define the schema, force JSON, validate:

python
prompt = '''Extract from this invoice as JSON:
{"vendor": str, "invoice_number": str, "date": "YYYY-MM-DD",
 "line_items": [{"description": str, "amount_cents": int}],
 "total_cents": int, "confidence": "high|medium|low"}
If a field is unreadable, use null and set confidence accordingly. JSON only.'''

Schema-validate the response (Zod vs Pydantic) and route confidence != high to human review — vision extraction belongs in the human-in-the-loop pattern until your measured error rate says otherwise.

2. The self-check trick for numbers. For high-stakes numeric extraction (totals, meter readings), ask the model to also quote *where* each number appears ("bottom right, below the subtotal line") — location-grounding measurably cuts misreads, the same cite-your-source principle as in contract analysis.

3. Cost control. Images bill as input tokens scaling with resolution (a full-res image on current Opus can run a few thousand tokens). Levers: downscale to the minimum resolution your task needs (receipts don't need 4K), crop to the region of interest before sending, and batch low-urgency volume through the Batches API at 50% off (batch patterns — Anthropic's equivalent works the same way). But don't reflexively downscale chart/document work — if fidelity drives accuracy, resolution is what you're paying *for*.

4. OCR replacement decision. Pure dense-text digitization at massive scale: dedicated OCR is still cheaper per page. Anything needing *understanding* (layout, classification, extraction logic, handwriting tolerance): vision-LLM wins on total system complexity — comparison in OCR with large vision models.

Limits to design around

Counting and fine spatial precision are weak spots (many small similar objects, exact pixel measurements) — don't build inventory counting on raw vision calls without verification.

Hallucinated plausibility: a blurry field becomes a *plausible* invoice number, not a refusal — hence confidence fields, location grounding, and validation gates above.

People identification is refused by design — identity workflows need different tooling.

FAQ

Vision vs fine-tuned classifier for one fixed task? High-volume single-label classification (defect/no-defect) eventually justifies a small trained model; vision-LLM wins for long-tail variety and anything needing language output.

Can it handle handwriting? Usefully yes for reasonably legible writing, with the same confidence-gating advice doubled.

Multi-provider? GPT and Gemini vision take near-identical message shapes — the gateway pattern covers vision routes too; just re-run your eval per provider, vision quality varies more than text.

---

*Last updated: June 2026. Model capabilities per Anthropic's docs — verify current limits there.*