Vector Database Internals

How Vector Databases Work: Indexing, Distance Metrics, and Performance

How Vector Databases Work

Vector databases are specialized storage systems designed to efficiently store and query high-dimensional embedding vectors. Unlike traditional databases that use exact matching, vector databases find the most similar vectors using approximate nearest neighbor (ANN) algorithms.

Understanding the internals helps you choose the right database and configuration for your use case—whether optimizing for speed, accuracy, or cost.

ANN

Approximate Search

HNSW

Graph Index

IVF

Inverted File

Quantization

Distance Metrics

Distance metrics define how similarity between vectors is measured. The choice affects both accuracy and performance.

Cosine Similarity

Most Common

Measures the angle between vectors, ignoring magnitude. Perfect for text embeddings where direction matters more than length. Range: -1 (opposite) to 1 (identical).

cosine(A, B) = (A · B) / (||A|| × ||B||)

Euclidean Distance (L2)

Geometric

Straight-line distance between points. Good for when magnitude matters, like image embeddings or physical coordinates. Range: 0 (identical) to ∞.

euclidean(A, B) = √Σ(Aᵢ - Bᵢ)²

Dot Product (Inner Product)

Normalized

Equivalent to cosine similarity when vectors are normalized. Faster to compute (no division). Many embedding models output normalized vectors.

dot(A, B) = Σ(Aᵢ × Bᵢ)

Indexing Algorithms

HNSW (Hierarchical Navigable Small World)

Best Quality

Graph-based index with multiple layers. Upper layers provide fast navigation, lower layers provide precision. Best recall/speed trade-off for most use cases.

O(log n) search High recall (99%+) Memory intensive

M (connections)

More connections = better recall, more memory. Default: 16

efConstruction

Build quality. Higher = better index, slower build. Default: 200

IVF (Inverted File Index)

Scalable

Partitions vectors into clusters using k-means. Search only checks nearby clusters, not the entire database. Good for very large datasets.

O(√n) search Lower memory Training required

nlist (clusters)

Number of clusters. Rule of thumb: √n vectors

nprobe

Clusters to search. More = better recall, slower

Flat (Brute Force)

100% Recall

Compares query against every vector. Perfect recall but O(n) complexity. Only practical for small datasets (<100K vectors) or as ground truth.

No index needed Exact results Slow at scale

HNSW Architecture

flowchart TB subgraph Layer2["Layer 2 (Sparse)"] A2((Entry)) --> B2((Hub)) end subgraph Layer1["Layer 1 (Medium)"] A1((Node)) --> B1((Node)) B1 --> C1((Node)) C1 --> D1((Node)) end subgraph Layer0["Layer 0 (Dense - All Vectors)"] A0((●)) --> B0((●)) B0 --> C0((●)) C0 --> D0((●)) D0 --> E0((●)) E0 --> F0((●)) A0 --> C0 B0 --> D0 end B2 -.->|"descend"| B1 C1 -.->|"descend"| D0

Search starts at top layer, navigates to closest node, descends to next layer, repeats until Layer 0.

Vector Quantization

Quantization compresses vectors to reduce memory and speed up search, with some accuracy trade-off.

Technique	Compression	Recall Impact	Best For
Scalar (INT8)	4x	~1%	General production
Product Quantization (PQ)	8-64x	3-10%	Large datasets
Binary	32x	5-15%	First-stage retrieval
Half-precision (FP16)	2x	<0.1%	Memory savings

FAISS Index Types

Python - Common FAISS Index Configurations

import faiss
import numpy as np

dimension = 1536  # OpenAI embedding dimension
n_vectors = 1_000_000

# 1. Flat Index (Brute Force) - 100% recall, slow
index_flat = faiss.IndexFlatL2(dimension)

# 2. HNSW - Best recall/speed trade-off
index_hnsw = faiss.IndexHNSWFlat(dimension, 32)  # M=32
index_hnsw.hnsw.efConstruction = 200  # Build quality
index_hnsw.hnsw.efSearch = 64  # Search quality

# 3. IVF + PQ - Best for 10M+ vectors
nlist = 1000  # Number of clusters
m = 64  # PQ subquantizers
quantizer = faiss.IndexFlatL2(dimension)
index_ivfpq = faiss.IndexIVFPQ(quantizer, dimension, nlist, m, 8)
index_ivfpq.nprobe = 10  # Clusters to search

# 4. HNSW + Scalar Quantization - Good balance
index_hnsw_sq = faiss.IndexHNSWSQ(dimension, faiss.ScalarQuantizer.QT_8bit, 32)

# Add vectors (IVF needs training first)
vectors = np.random.random((n_vectors, dimension)).astype('float32')

# Train IVF index
index_ivfpq.train(vectors[:100000])  # Train on subset
index_ivfpq.add(vectors)

# Search
query = np.random.random((1, dimension)).astype('float32')
distances, indices = index_hnsw.search(query, k=10)

Pgvector Index Options

SQL - Creating Indexes in Pgvector

-- Create table with vector column
CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding vector(1536)
);

-- HNSW Index (best for most cases)
CREATE INDEX ON documents 
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- IVFFlat Index (for larger datasets, needs ANALYZE first)
CREATE INDEX ON documents 
USING ivfflat (embedding vector_l2_ops)
WITH (lists = 100);

-- Search with HNSW
SET hnsw.ef_search = 100;  -- Increase for better recall

SELECT id, content, 
       1 - (embedding <=> '[0.1, 0.2, ...]') AS similarity
FROM documents
ORDER BY embedding <=> '[0.1, 0.2, ...]'
LIMIT 10;

Performance Tuning

Memory vs Disk

HNSW is memory-resident. For huge datasets, use IVF+PQ or move to disk-based solutions like DiskANN.

Recall vs Latency

Increase efSearch (HNSW) or nprobe (IVF) for better recall. Decrease for lower latency.

Pre-filtering

Filter metadata before vector search to reduce search space. Critical for multi-tenant systems.

Oversampling + Rerank

Retrieve more candidates (e.g., 100) then rerank to top-k. Improves final quality.

Vector Database Comparison

Database	Index Types	Best For
Pinecone	Proprietary	Managed, serverless
Qdrant	HNSW, PQ	Self-hosted, filtering
Weaviate	HNSW, PQ	GraphQL, modules
Milvus	IVF, HNSW, DiskANN	Billion-scale
Pgvector	HNSW, IVFFlat	PostgreSQL users
Chroma	HNSW	Development, prototyping

Test Your Knowledge

Score 8/10 or higher to pass