Glossary

A plot of the total cost of serving AI inference against request volume, drawn once per infrastructure option (per-token API, self-hosted GPU, hybrid). API pricing is a straight line through the origin; self-hosting is a high fixed cost plus a near-flat marginal cost. Comparing the curves reveals where each option is cheapest as volume grows.

See: AI Cost Curves

NVIDIA's reference design for distributing AI inference across edge locations — in effect, CDN architecture applied to AI. Rather than routing every request to a centralised GPU cluster, GPU compute is placed in the edge locations that already serve web content, and each request is routed to the nearest node capable of handling it. Practical deployments use three tiers (edge, regional, core) with an orchestrator choosing the cheapest tier that meets each request's constraints.

See: AI Grid

The request volume at which a fixed-cost self-hosted setup becomes cheaper than per-token API pricing. Found by dividing the fully-loaded monthly cost of self-hosting (hardware plus the engineering time to run it) by the per-request API price. Below it, APIs win on total cost; above it, self-hosting does — provided the hardware stays busy.

See: AI Cost Curves

Running model inference at points of presence close to users instead of in a central data centre, cutting the network round trip out of the response time. Made economic by quantised small models, which fit the modest GPU capacity an edge site can host — the combination that turns sub-50ms global response times from impossible to routine.

See: AI Grid

An architecture that routes the bulk of requests to a cheap self-hosted small model and reserves a frontier API for the minority that need advanced reasoning. Because most workloads are long-tailed, hybrid routing captures the majority of self-hosting savings while preserving quality on the hard cases — commonly cutting spend by 80% or more versus an all-frontier approach.

See: AI Cost Curves

Running a trained model to produce an output (a completion, classification, or embedding) in response to a request. Inference is the recurring, per-request cost of operating AI in production — as opposed to training, which is a one-off up-front cost — and is the cost that AI cost curves are drawn against.

The routing layer of a distributed inference deployment. For each request it weighs the latency budget, the capability the task needs, and any jurisdiction or sovereignty constraint, then sends the request to the cheapest node that satisfies all three. The orchestrator is what turns a set of scattered GPU sites into an architecture — without it, distribution is just fragmentation.

See: AI Grid

The maximum end-to-end delay an interaction can tolerate before its user experience or business value degrades — a frame time in gaming, a conversational beat for a customer agent, milliseconds for a trading decision. In a distributed inference architecture the latency budget is the primary routing constraint: it determines which tiers and regions are even eligible to serve a request.

See: AI Grid

The usage-based billing model of hosted inference APIs, where you pay a fixed amount per thousand input and output tokens with no upfront commitment. It has zero fixed cost and a constant marginal cost, which makes it unbeatable at low volume and progressively expensive as request volume scales.

Reducing the numerical precision of a model’s weights (for example from 16-bit to 4-bit) so it needs less memory and runs faster, usually with only a small quality loss. Quantisation is what makes self-hosted and edge inference economic: a quantised small model can run on a single GPU that a full-precision model would not fit.

See: Model Compression