Semantic Search Implementation: A Complete Developer Guide for 2026

Semantic search finds results based on meaning rather than exact keywords. You convert text into embeddings (dense vectors), store them in a vector index, embed the query at search time, and retrieve the nearest vectors. It's the retrieval part of RAG and the reason "find docs about canceling subscriptions" can match a page titled "How to terminate your plan."

Pipeline

Chunking: Split documents into passages (a few hundred tokens each).

Embedding: Embed each chunk using an embedding model.

Storage: Store vectors and metadata in a vector database.

Query: Embed the user's text, retrieve top-k nearest chunks.

(Optional) Re-ranking: Re-rank the top-k results with a cross-encoder before passing to the LLM for better precision.

python
pip install openai
from openai import OpenAI
client = OpenAI()
def embed(texts):
    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [d.embedding for d in r.data]
docs = ["How to cancel your subscription", "Resetting your password", "Billing FAQ"]
doc_vecs = embed(docs)
import numpy as np
def search(query, k=2):
    q = np.array(embed([query])[0])
    sims = [float(np.dot(q, v) / (np.linalg.norm(q) * np.linalg.norm(v))) for v in doc_vecs]
    return sorted(zip(docs, sims), key=lambda x: -x[1])[:k]
print(search("end my plan"))  # → "How to cancel your subscription" ranks first

The numpy version is for illustration only—in production, use a vector database to keep search fast at scale.

Choosing a Vector Store

Prototyping / local: Chroma or pgvector—see Chroma vs Qdrant and pgvector guide.

Production / large scale: Qdrant, Pinecone, or Weaviate—see Pinecone vs Weaviate.

Quality Levers

Chunking matters more than the embedding model. Too large chunks → imprecise retrieval; too small → lost context. Try 200–500 tokens with a small overlap.

Hybrid search (combining semantic + keyword/BM25) catches exact terms (code, names) that pure vectors miss.

Re-ranking: Use a cross-encoder on a larger top-k set to significantly boost precision before the LLM sees the results.

Metadata filtering (date, source, language) narrows the candidate set first.

The whole pipeline is the retrieval stage of RAG—to build the full system, see LangChain vs LlamaIndex for RAG and LlamaIndex Production RAG.

FAQ

Embeddings vs keyword search? Keywords match exact terms; embeddings match meaning. Hybrid search uses both. Which embedding model to choose? Current general-purpose models are fine to start; bigger gains come from chunking and re-ranking. How many results to retrieve? Retrieve a larger top-k (e.g., 20), re-rank, then pass the best few to the LLM.

Summary

Semantic search = embed, store, retrieve by nearest neighbor. Chunk well, add hybrid search and re-ranking for precision, and pick a vector store that fits your scale. It's the backbone of every RAG system.

---

*Last updated: June 2026. Verify embedding API against OpenAI documentation.*