Embedding Models

What Are Embeddings?

Embeddings are dense vector representations of data (text, images, audio) that capture semantic meaning in a continuous, high-dimensional space. Similar concepts are mapped to nearby points, enabling machines to understand relationships, perform similarity searches, and support downstream ML tasks.

In the context of LLMs, text embeddings convert words, sentences, or documents into fixed-size vectors (typically 384 to 4096 dimensions) that encode semantic content. These vectors are the foundation for Retrieval-Augmented Generation (RAG), semantic search, clustering, and classification.

How Text Embeddings Work

The Embedding Process

1. Tokenization: Split text into tokens (words, subwords, or characters)
2. Token Embedding: Map each token to a learned vector
3. Contextual Encoding: Pass through transformer layers to capture context
4. Pooling: Aggregate token embeddings into a single vector (mean, CLS, max)
5. Normalization: L2-normalize for cosine similarity comparisons

# Example using OpenAI embeddings
from openai import OpenAI
client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Machine learning is transforming industries"
)

embedding = response.data[0].embedding
print(f"Dimensions: {len(embedding)}")  # 1536
print(f"First 5 values: {embedding[:5]}")
# [0.023, -0.045, 0.012, 0.078, -0.034]
            

Similarity Metrics

Once you have embeddings, you need a way to measure how similar two vectors are. The choice of metric affects retrieval quality and performance.

cos(A,B) = A·B / (|A||B|)

d(A,B) = √Σ(Aᵢ-Bᵢ)²

A·B = Σ(Aᵢ × Bᵢ)

Popular Embedding Models

How to Choose an Embedding Model

Vector Databases

To search embeddings efficiently at scale, you need a vector database with Approximate Nearest Neighbor (ANN) indexing. Key options:

# Example: Storing and searching with Chroma
import chromadb
from chromadb.utils import embedding_functions

# Initialize
client = chromadb.Client()
ef = embedding_functions.OpenAIEmbeddingFunction(
    model_name="text-embedding-3-small"
)

# Create collection
collection = client.create_collection(
    name="documents",
    embedding_function=ef
)

# Add documents (automatically embedded)
collection.add(
    documents=["Machine learning basics", "Deep learning fundamentals"],
    ids=["doc1", "doc2"]
)

# Query (automatically embedded and searched)
results = collection.query(
    query_texts=["What is neural network?"],
    n_results=2
)
            

ANN Indexing Algorithms

Exact nearest neighbor search is O(n)—too slow for large datasets. Approximate algorithms trade accuracy for speed:

Common Use Cases

Best Practices

• Use the same model for queries and documents: Different models produce incompatible embedding spaces.
• Normalize your embeddings: Most modern models do this automatically, but verify for your specific model.
• Chunk intelligently: Split long documents into meaningful segments (paragraphs, sections) before embedding.
• Consider asymmetric embeddings: Some models have separate modes for short queries vs. long documents (e.g., Cohere input_type).
• Batch your API calls: Embedding APIs support batching—use it for better throughput and lower costs.
• Monitor embedding drift: If your model or data changes, re-embed your corpus to maintain consistency.
• Store metadata: Always store original text and metadata alongside vectors for context in retrieval results.

Related Topics

• Retrieval-Augmented Generation (RAG)
• Chunking Strategies
• Transformers Architecture
• Semantic Search
• 🔧 RAG Visualizer Tool — Interactive visualization of embeddings and retrieval