# Model Compression

Source: John Bradshaw — https://bradshaw.cloud
URL: https://bradshaw.cloud/strategy/model-compression/
Type: insight
Date: 2026-05-27
Updated: 2026-08-01
Tags: AI, Infrastructure, Inference

> Compression is the family. Quantisation is one cousin. The techniques that shrink language models for self-hosting — what they do, what they cost in quality, and which ones you actually reach for.

## Frequently asked questions

### What is model compression, and how is it different from quantisation?

Compression is the family; quantisation is one member of it, alongside pruning, distillation, and low-rank factorisation. Most people use the words interchangeably — they are not the same thing. The techniques compose rather than compete, and a modern self-hosted small model has typically been touched by two or three of them. A model is really a budget of memory, arithmetic, and attention; compression is spending that budget more cheaply for almost the same behaviour.

### What is quantisation and how much quality does it cost?

Quantisation runs the same network at lower numeric precision — the shape is unchanged, the arithmetic just happens on coarser numbers. FP32 offers around four billion possible values per weight, INT8 offers 256, INT4 offers sixteen. Quality is near-lossless at INT8 and holds up well at INT4; the cliff lives between INT4 and INT2, where models break down without quantisation-aware training. Expect roughly a twofold to eightfold reduction in footprint.

### When should I reach for pruning?

When quantisation alone leaves you 20–30% over your VRAM budget. Pruning removes weights, neurons, heads, or whole layers and stitches the rest back together. Quality holds to about 50% sparsity and then drops sharply without fine-tuning: the first third of pruning costs almost nothing, the second costs a little, and the last third is where the model degrades faster than it shrinks. Structured pruning stays fast on any hardware; unstructured needs specialist kernels.

### What is knowledge distillation?

Training a small student model to imitate a large teacher, learning from the teacher's output distributions rather than raw labels — so it absorbs the teacher's confidence structure, not just its answers. A well-executed student can be five to ten times smaller while keeping 90–95% of capability on the task distribution it was trained for, with worse generalisation outside it. It is an architectural decision upstream, not an inference-time knob you turn later.

### Which compression technique should I try first?

Quantisation, always. Weight-only INT4 is the de facto default for self-hosted small models, and it is the cheapest large win available — an 8B model at FP16 is 16 GB of weights and roughly 4 GB at INT4. Only once that has been spent should you consider pruning to close a remaining gap. And if INT4 still is not enough, picking a smaller model usually beats pushing precision below it.

### Why does long-context inference need a different approach?

Because the KV cache dominates VRAM once contexts run long, and it scales with context length rather than parameter count. Quantising the weights alone is not enough — a model that fits comfortably at 4K tokens can fall off a hardware tier at 32K on cache alone. For long-context serving, quantise the KV cache itself as well as the weights, and size the hardware against the context you actually intend to run.

## Interactive companion

**Model Compression: LLM Sizing Lab** — Compose precision, pruning, distillation, and context length for a self-hosted language model. Watch the footprint, VRAM bill, and hardware tier shift as you go.

Live calculator: https://bradshaw.cloud/strategy/model-compression/calculator/

## Related

- [AI cost curves](https://bradshaw.cloud/strategy/ai-cost-curves/) — the economics that make self-hosting worth the engineering.
- [Frontier models](https://bradshaw.cloud/strategy/frontier-models/) — the upstream choice that decides whether you ever need to compress.
- [AI fundamentals](https://bradshaw.cloud/strategy/ai-fundamentals/) — the building blocks that sit under every model choice.
- [All strategy insights](https://bradshaw.cloud/strategy/)

_This insight has additional narrative content on the HTML page. Read the full version at https://bradshaw.cloud/strategy/model-compression/._