Compression Is the Family. Quantisation Is One Cousin.

There’s a vocabulary problem quietly distorting self-hosting decisions. Ask a team how they shrank their model and the answer is usually “we quantised it” — said as if quantisation and compression were the same word. They aren’t. Compression is the umbrella term. Quantisation is one family within it, alongside pruning, distillation, and low-rank factorisation. They do different things, cost different things, and — this is the part that matters — they compose. A modern self-hosted small model has typically been touched by two or three of them.

The distinction isn’t pedantry. Each technique buys memory back on a different curve, and damages quality on a different curve. If you only know one word, you only have one knob.

The four techniques, and when you reach for each

Quantisation stores the same network at lower numeric precision — snapping continuous weights to a coarser grid. FP16 has an effectively continuous range; INT8 has 256 values; INT4 has sixteen. The arithmetic is brutally simple: weights-on-disk is roughly parameters × bits ÷ 8, so an 8B model at FP16 is 16 GB of weights and the same model at INT4 is 4 GB. Quality holds up remarkably well down to INT4 — the tooling around post-training quantisation (GPTQ, AWQ, bitsandbytes) has made 4-bit weight-only quantisation the de facto default for self-hosted models in 2026. The cliff lives below that: INT2 without quantisation-aware training is aggressive and fragile.

Reach for it: always, first. It’s the cheapest memory you’ll ever buy back.

Pruning removes parts of the network — individual weights, attention heads, whole layers — and stitches the rest back together. Structured pruning gives you a dense, fast model on any hardware; unstructured pruning gets higher sparsity but needs specialist kernels to see the speed. The trade-off curve is the nastiest of the four: the first 30% of pruning costs almost nothing, and then the damage accelerates sharply — past roughly half the network you typically need fine-tuning just to claw quality back.

Reach for it: when quantisation alone leaves you 20–30% over your VRAM budget. It’s the gap-closer, not the headline act.

Distillation trains a small student model to imitate a large teacher — learning from the teacher’s output distributions, not just the right answers, so it absorbs the confidence structure as well as the labels. It’s how most credible small models are actually made (synthetic-data distillation is the pattern behind the Phi and Gemma small-model families). You can get a model 5–10× smaller that keeps 90–95% of the capability on the distribution you targeted — with weaker generalisation outside it.

Reach for it: when you control the training pipeline and know the deployment target. It’s an architectural decision made upstream, not a knob you turn at inference time.

Low-rank factorisation replaces one big weight matrix with the product of two thin ones. For LLMs in 2026 it’s niche — useful for embedding tables and specific bottlenecks. Its famous relatives, LoRA and QLoRA, get mentioned in every compression conversation but solve a different problem: they’re additive adapters for parameter-efficient fine-tuning, not compression of the base model. Adjacent territory, permanently confused.

Reach for it: rarely, knowingly.

Three curves worth memorising

If you take one model of the trade-offs away, take this: quantisation saves memory linearly, pruning damages quality cubically past a threshold, and distillation loses a small fixed amount per halving of size. That’s why the standard recipe is quantise first (cheap, predictable), distil when you can (efficient, but upstream), prune to close a stubborn gap (dangerous in large doses) — and why the techniques stack multiplicatively, in quality loss as well as in savings. Two “acceptable” compromises compose into a product that falls faster than either term.

One hedge I want to carry over faithfully from the full insight: the specific quality constants you’ll see quoted — INT4 retaining ~94%, a 4× student keeping ~92% — are rules of thumb calibrated from published benchmarks, not laws. The shapes of the curves are the reliable part. Anyone who wants to argue about the decimal points is having exactly the right conversation. The Compression Lab exists so you can compose the techniques yourself and watch the footprint, VRAM bill, and hardware tier move — build intuition with it, don’t commit a procurement on it.

The situations, mapped

The practical decision guide compresses to five rows:

1. Self-hosting an open model on consumer hardware → INT4 weight-only quantisation. If that’s still too big, pick a smaller model rather than dropping below INT4.
2. Production inference on datacentre GPUs → FP8/INT8 through a serving stack that supports it; quality holds and throughput improves materially.
3. Edge and mobile deployment → distil to a 1–3B model, then quantise to INT4. This is the recipe that makes edge inference tiers viable at all.
4. Long-context workloads → quantise the KV cache, not just the weights. Past 32K tokens the cache dominates VRAM, and weight-only quantisation stops moving the number that’s actually killing you.
5. Adapting to a domain → LoRA/QLoRA, and be honest that you’re now doing parameter-efficient training, not compression.

Why this is a strategy question at all

Because the economics of self-hosting — the whole cost-curves argument — depend on what hardware the model fits on. An 8B model at FP16 needs a workstation-class GPU; the same model at INT4 runs on a consumer card, and a distilled-then-quantised 3B runs at the edge. Every step down that ladder changes the per-query cost, the placement options, and the build-vs-rent calculus. Compression isn’t an optimisation you apply after the infrastructure decision. It largely is the infrastructure decision.

The insight closes on a line I keep coming back to: a model is not really a thing — it’s a budget, of memory, arithmetic, and attention. Compression is the practice of spending that budget more cheaply for almost the same behaviour. Quantisation is the most efficient way to do it. It is not the only way.