Hybrid Approach

Small Models + Tools

Why use 70B parameters when 3B + a calculator can do better? The compound AI systems approach.

The Core Insight

LLMs are generalists—they try to do everything with neural weights. But many tasks (math, search, code execution, API calls) are better solved by specialized tools.

A small model (1-7B) that can orchestrate tools often outperforms a large model (70B+) working alone—at 1/10th the cost and latency.

Best Small Models (2024-2025)

Model	Size	Strengths	VRAM (Q4)
Phi-3.5 Mini	3.8B	Reasoning, code (trained on synthetic data)	~3 GB
Qwen2.5-3B	3B	Multilingual, math, coding	~2.5 GB
Llama 3.2 3B	3B	Edge/mobile optimized, tool use	~2.5 GB
Gemma 2 2B	2B	Efficient, good instruction following	~1.5 GB
Mistral 7B	7.3B	Best 7B class, sliding window attention	~5 GB
Llama 3.1 8B	8B	128K context, tool use, strong baseline	~6 GB

Types of Tools

Computation

• Calculator / math engine
• Python interpreter
• Date/time calculations

Information Retrieval

• Web search (Bing, Google)
• Vector database (RAG)
• Wikipedia / knowledge bases

Code Execution

• Python sandbox
• SQL databases
• Shell commands

External APIs

• Weather, stocks, news
• CRMs, ticketing systems
• Custom business APIs

How It Works

# ReAct Pattern: Reason + Act
User: What's 23% of the revenue if we made $4.2M last quarter?

Model (Think): I need to calculate 23% of $4,200,000
Model (Act): calculator 4200000 * 0.23
Tool Result: 966000

Model (Think): The result is $966,000
Model (Answer): 23% of $4.2M is $966,000

💡 Key: The model doesn't do math—it recognizes when math is needed and delegates to a calculator that's 100% accurate.

Quick Example with LangChain

from langchain_community.llms import Ollama
from langchain.agents import create_react_agent, Tool
from langchain_community.tools import DuckDuckGoSearchRun

# Small model (runs on laptop)
llm = Ollama(model="llama3.2:3b")

# Define tools
tools = [
    Tool(name="search", func=DuckDuckGoSearchRun().run, 
         description="Search the web for current information"),
    Tool(name="calculator", func=lambda x: eval(x),
         description="Calculate math expressions"),
]

# Create agent
agent = create_react_agent(llm, tools, prompt=...)

# Now 3B model + tools can answer:
# "What's the current Bitcoin price times 1.5?"

When to Use This Approach

✓ Great For

• Tasks needing real-time data
• Math/calculations
• Structured data queries (SQL)
• Edge/mobile deployment
• Cost-sensitive applications

⚠️ Consider Alternatives

• Complex multi-step reasoning
• Creative/nuanced writing
• Tasks requiring deep knowledge
• When latency is critical (tool calls add overhead)