GenAIHub
EN | PT
← Back to Technical Section

Embedding Optimization

Techniques for Improving Embedding Quality, Retrieval Performance & Efficiency

Why Optimize Embeddings?

Embeddings are the foundation of semantic search, RAG systems, and recommendation engines. Optimizing embeddings can dramatically improve retrieval accuracy, reduce latency, lower costs, and enable better downstream LLM performance.

Whether you're using OpenAI, Cohere, or open-source models, the techniques on this page will help you get the most out of your embedding pipeline.

40%
Better Retrieval
50%
Reduced Storage
3x
Faster Search
30%
Lower Costs

Optimization Techniques

Matryoshka Embeddings (MRL)

Dimensionality

Train embeddings where truncating to smaller dimensions still preserves semantic meaning. Use 256 or 512 dimensions instead of 1536 with minimal quality loss.

50% Storage Reduction Faster Similarity OpenAI Supported

Binary Quantization

Compression

Convert float32 embeddings to binary (1 bit per dimension). Reduces storage by 32x and enables ultra-fast Hamming distance computation. Use with reranking for best results.

32x Compression Hamming Distance Two-stage Retrieval

Scalar Quantization (INT8)

Precision

Reduce float32 to int8, achieving 4x compression with minimal accuracy loss. Better quality than binary but still significant storage savings.

4x Compression Better Quality Pgvector Support

Domain-Specific Fine-tuning

Quality

Fine-tune embedding models on your domain data using contrastive learning. Dramatically improves retrieval for specialized vocabularies and concepts.

Contrastive Learning Domain Vocabulary Sentence Transformers

Hybrid Search (BM25 + Dense)

Retrieval

Combine sparse (BM25/keyword) and dense (embedding) retrieval with Reciprocal Rank Fusion. Captures both exact matches and semantic similarity.

RRF Fusion Best of Both Weaviate/Qdrant

Matryoshka Embeddings with OpenAI

Python - Reduced Dimension Embeddings
from openai import OpenAI

client = OpenAI()

def get_embedding(text: str, dimensions: int = 256) -> list:
    """Get embedding with reduced dimensions (Matryoshka)."""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
        dimensions=dimensions  # 256, 512, or 1536
    )
    return response.data[0].embedding

# Compare storage requirements
full_embedding = get_embedding("Hello world", dimensions=1536)
small_embedding = get_embedding("Hello world", dimensions=256)

print(f"Full: {len(full_embedding)} dims = {len(full_embedding) * 4} bytes")
print(f"Small: {len(small_embedding)} dims = {len(small_embedding) * 4} bytes")
# Output: Full: 1536 dims = 6144 bytes
#         Small: 256 dims = 1024 bytes (6x smaller!)

Binary Quantization

Python - Binary Quantization with Reranking 
import numpy as np
from typing import List

def quantize_binary(embedding: List[float]) -> np.ndarray:
    """Convert float embeddings to binary (1 bit per dimension)."""
    arr = np.array(embedding)
    # Pack 8 dimensions into 1 byte
    binary = np.packbits((arr > 0).astype(np.uint8))
    return binary

def hamming_distance(a: np.ndarray, b: np.ndarray) -> int:
    """Ultra-fast similarity using XOR and popcount."""
    return np.unpackbits(a ^ b).sum()

def two_stage_retrieval(query_emb, documents, top_k=10, rerank_k=100):
    """Stage 1: Fast binary search, Stage 2: Rerank with full embeddings."""
    
    # Stage 1: Binary quantized search (very fast)
    query_binary = quantize_binary(query_emb)
    candidates = []
    for doc in documents:
        dist = hamming_distance(query_binary, doc["binary_emb"])
        candidates.append((doc, dist))
    
    # Get top rerank_k candidates
    candidates.sort(key=lambda x: x[1])
    top_candidates = candidates[:rerank_k]
    
    # Stage 2: Rerank with full precision embeddings
    reranked = []
    for doc, _ in top_candidates:
        score = cosine_similarity(query_emb, doc["full_emb"])
        reranked.append((doc, score))
    
    reranked.sort(key=lambda x: x[1], reverse=True)
    return reranked[:top_k]

Domain Fine-tuning with Sentence Transformers

Python - Fine-tune for Your Domain
from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

# Load pretrained model
model = SentenceTransformer("all-MiniLM-L6-v2")

# Prepare training data (query, positive_doc pairs)
train_examples = [
    InputExample(texts=["what is kubernetes?", "K8s is a container orchestration platform"]),
    InputExample(texts=["how to scale pods", "Use kubectl scale deployment --replicas=N"]),
    InputExample(texts=["ingress vs service", "Ingress manages external access, Service handles internal"]),
    # Add your domain-specific pairs...
]

# Create dataloader
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)

# Use Multiple Negatives Ranking Loss (MNR)
train_loss = losses.MultipleNegativesRankingLoss(model)

# Fine-tune
model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=3,
    warmup_steps=100,
    output_path="./my-domain-embeddings"
)

# Use fine-tuned model
embeddings = model.encode(["my domain query"])

Technique Comparison

Technique Compression Quality Loss Speed Gain Use Case
Matryoshka (256d) 6x ~1-2% 3-4x General use
Scalar (INT8) 4x ~1% 2-3x Production systems
Binary (1-bit) 32x ~5-10% 10-20x First-stage retrieval
Fine-tuning None +10-40% gain Same Domain-specific
Hybrid Search None +5-15% gain Varies Mixed queries

Vector Database Support

Pinecone
Binary, Scalar
Qdrant
Binary, Scalar, MRL
Weaviate
PQ, Hybrid Search
Pgvector
Halfvec, INT8
Milvus
IVF, PQ, SQ
Chroma
HNSW, Hybrid
Elasticsearch
Dense + BM25
FAISS
All techniques

Best Practices

Benchmark First

Measure baseline performance before optimizing. Use MTEB, BEIR, or custom evaluation sets.

Stack Techniques

Combine Matryoshka + Binary for extreme compression, or Fine-tuning + Hybrid for best quality.

Use Reranking

With aggressive quantization, add a cross-encoder reranker to recover quality.

A/B Test Changes

Validate optimizations in production with real user queries and feedback.

Related Topics

Embeddings Voyage AI Vector Databases RAG Semantic Search Reranking

Test Your Knowledge

Score 8/10 or higher to pass