Model Compression

Quantization

Reduce model size by 4-8x with minimal quality loss. The key to running large models on consumer hardware.

What is Quantization?

Quantization reduces the precision of model weights from 32/16-bit floats to 8-bit, 4-bit, or even 2-bit integers. A 70B model at FP16 needs ~140GB VRAM. With 4-bit quantization, it fits in ~35-40GB.

Precision	Bits	70B Size	Quality
FP16	16	~140 GB	Baseline
INT8	8	~70 GB	~1% loss
INT4	4	~35-40 GB	~2-5% loss

Popular Methods

GGUF (llama.cpp/Ollama)

Standard for local inference. Best CPU/GPU hybrid support.

ollama run llama3.1:70b-instruct-q4_K_M

GPTQ / AWQ

GPU-optimized. AWQ often better quality than GPTQ. Best for vLLM/TGI.

from vllm import LLM
llm = LLM(model="TheBloke/Llama-2-70B-AWQ", quantization="awq")

BitsAndBytes (QLoRA)

Dynamic quantization for training. Enables 4-bit fine-tuning.

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")

VRAM Requirements

Model	FP16	INT4	Runs On
Llama 8B	16 GB	5 GB	RTX 3060 ✓
Mixtral 8x7B	90 GB	26 GB	RTX 4090 ✓
Llama 70B	140 GB	40 GB	2x RTX 4090