Vector Databases for Production: Architecture, Performance, and Scaling

What Is a Vector Database?

Traditional databases store exact values—you query for records where field = value. Vector databases store high-dimensional numerical representations (embeddings) of data objects and support similarity search: find items most similar to a query vector.

This enables: semantic search (find documents with similar meaning, not just matching keywords), recommendation (find items similar to what a user liked), anomaly detection (find vectors far from normal cluster), duplicate detection, multimodal search (search images using text description).

How Vector Similarity Search Works

The Embedding Space

Embeddings are numerical representations where semantic similarity maps to geometric proximity. Word2Vec, Sentence-BERT, OpenAI embeddings convert text → vectors (1536 dimensions for OpenAI's ada-002). Similar concepts cluster together in embedding space.

Similarity metrics:

Cosine similarity: measures angle between vectors. Standard for normalized text embeddings.

Euclidean distance: measures straight-line distance. Better for absolute magnitude matters.

Dot product: optimized for vectors already normalized. Fastest computation.

Approximate Nearest Neighbor (ANN) Algorithms

Exact nearest neighbor search (check every vector) is O(n) per query—impractical at scale. ANN algorithms trade small accuracy loss for orders-of-magnitude speed improvement.

HNSW (Hierarchical Navigable Small World): the dominant ANN algorithm. Builds multi-layer navigable small world graph. Query traversal: start at top layer (few nodes, long-range connections), navigate down to bottom layer (all nodes, local connections).

Performance: sub-millisecond queries on millions of vectors. Recall@10: 99%+. Memory-intensive (stores full vectors + graph structure).

IVF (Inverted File Index): clusters vectors into Voronoi cells, searches only nearby cells. Lower memory than HNSW, lower recall. Good for high-dimensional vectors where HNSW graph becomes expensive.

Product Quantization (PQ): compresses vectors by quantizing sub-spaces. Dramatic memory reduction (32x or more), moderate recall reduction. Used when dataset doesn't fit in RAM.

IVF+PQ: combines both—cells for coarse navigation, quantization for compression. Standard approach for billion-scale vector search.

Vector Database Comparison

pgvector (PostgreSQL Extension)

What it is: Vector search built into PostgreSQL. HNSW and IVF-Flat indexing.

When to use: team already uses Postgres, simpler infra is valuable, dataset < 10M vectors, can tolerate slightly higher query latency vs. dedicated solutions.

Performance: with HNSW index, <10ms for million-scale queries. Significantly slower than dedicated vector DBs for large datasets.

Scaling: vertical scaling only. Single-node architecture limits to ~500M vectors practically.

Cost: free (Postgres hosting costs only). Significant savings vs. managed vector DB services.

Pinecone

What it is: Managed vector database as a service. No infrastructure to manage.

When to use: rapid development without infra expertise, production SLA requirements, team lacks time/expertise for self-hosted setup.

Performance: sub-millisecond P99 latency (serverless), configurable throughput (pods).

Scaling: automatic scaling (serverless) or manual pod scaling. Handles billions of vectors.

Cost: $0.00/query for serverless (storage-based pricing), pod pricing for performance-sensitive workloads. Can become expensive at high query volume.

Weaviate

What it is: Open source vector database with native hybrid search (vector + BM25 keyword).

When to use: hybrid search is required (combines semantic and keyword), need self-hosted option, multi-modal data (text + images + audio).

Performance: comparable to Pinecone for equivalent hardware. Hybrid search is a standout feature.

Scaling: horizontal scaling with Kubernetes. Production deployments handle billions of vectors.

Cost: free self-hosted, cloud offering available.

Qdrant

What it is: Rust-based open source vector database. Optimized for performance and on-premise deployment.

When to use: performance-critical applications (Rust's zero-cost abstractions deliver benchmark-leading performance), self-hosted deployment required (regulated data, air-gapped environments), cost-sensitive scale.

Performance: fastest in benchmarks for high-QPS scenarios. Efficient memory usage.

Scaling: built-in distributed mode. Production-ready for billion-scale.

Cost: free self-hosted, cloud offering available.

Milvus / Zilliz

What it is: Distributed vector database designed for massive scale. CNCF graduated project.

When to use: 100M+ vectors, need proven distributed architecture, enterprise Kubernetes deployment.

Performance: designed for billion-scale. Strong batch insertion performance.

Scaling: distributed architecture with automatic sharding and replication.

Cost: free self-hosted (complex), Zilliz Cloud managed offering.

Production Architecture Patterns

Caching Layer

Vector search is expensive. Cache frequent queries:

Application-level cache: Redis stores query → result pairs. TTL-based expiration.

Embedding cache: store embeddings for common queries. Avoid re-embedding on every request.

Result cache: for static collections, most queries repeat. Cache hit rates of 30-50% common.

Metadata Filtering

Real applications need filtered vector search: "find the most semantically similar product from the electronics category under $50."

Pre-filtering: apply metadata filter first, then vector search only similar items. Problem: if filter is very selective, remaining index is small and ANN accuracy degrades.

Post-filtering: vector search first, then filter results. Problem: top-k results after filtering may be fewer than requested.

Weaviate's approach: ACORN algorithm—integrates metadata filtering into the graph traversal. Best quality for filtered search.

Qdrant's approach: native payload filtering with indexed fields. Fast filtered search without quality degradation.

Real-Time Updates

Vector databases handle updates differently:

Inmutable vectors: Pinecone, Qdrant support real-time upserts efficiently. Index rebuild: some configurations require periodic index rebuilding for optimal performance. Schedule during low-traffic periods. Write amplification: insertions into HNSW graphs are expensive. Batch insertions significantly more efficient than individual.

Benchmarking Your Vector DB

Standard benchmark: ANN Benchmarks (ann-benchmarks.com). But benchmark on YOUR data:

Your specific vector dimensionality

Your query distribution (uniform vs. clustered)

Your filter selectivity

Your QPS requirements

Key metrics: recall@k (what % of true nearest neighbors are returned), queries per second (QPS), build time (how long to index a dataset), memory usage, cost per query.

Never choose a vector DB based on vendor benchmarks alone. Run your own.

Scaling Considerations

Small scale (< 1M vectors): pgvector on Postgres. Simple, free, good enough.

Medium scale (1M - 100M vectors): Pinecone serverless, Qdrant single-node, or Weaviate single-node. Dedicated vector DB performance, manageable infrastructure.

Large scale (100M - 1B vectors): Milvus distributed, Weaviate cluster, Qdrant cluster, Pinecone pods. Requires careful capacity planning and distributed systems expertise.

Extreme scale (> 1B vectors): purpose-built solutions or FAISS on custom infrastructure. Spotify (Annoy), Facebook (FAISS), Google (ScaNN) all built custom solutions at this scale.

Practical advice: optimize for developer productivity at small scale. Switch to dedicated vector DB when search latency > 100ms or queries per second requirements exceed single-node capacity. Don't over-engineer early.