The AI Grid: Why Inference Is Becoming a CDN Problem

The first wave of AI infrastructure was built for training, and it shows. Massive GPU clusters, a handful of mega data centres, power contracts measured in gigawatts. Training genuinely needs that concentration — the workload is one enormous, tightly-coupled computation that wants every accelerator in the same room.

Then the model ships, and everything about the problem changes. Inference — the unglamorous work of actually serving predictions to users — is latency-sensitive, geographically distributed, and bursty. It follows your users around the planet and around the clock. Pointing that workload at the same centralised cluster you trained on is how you end up with an AI product that feels instant in one region and treacle everywhere else.

I keep making the same observation in these conversations: we have solved this exact problem before. Video buffering from a server 3,000 miles away was unacceptable, so the industry spent twenty-five years building content delivery networks that cache content close to users. The physics that made CDNs inevitable for content applies, unchanged, to AI inference. A 200ms round trip to a central GPU cluster is fine for an internal summarisation tool. It is lethal for real-time video, gaming, or a customer-facing agent mid-conversation.

The grid, in one sentence

That’s the idea behind what NVIDIA calls the AI Grid reference design, and what I’d describe more generally as CDN architecture for inference: distribute GPU compute across the edge locations that already serve web content, video, and security traffic, and route each AI request to the nearest node capable of handling it — instead of hauling every request back to a central cluster.

Put GPUs where the users are. Route to the nearest one that can do the job. It’s the CDN playbook with matrix multiplications instead of video segments.

Three tiers, because one size doesn’t fit anything

A practical deployment isn’t “everything at the edge” any more than a CDN is “everything in cache”. The working shape is three tiers:

• Edge — thousands of points of presence running small, quantised models for latency-critical inference. Sub-50ms answers, close to the user.
• Regional — heavier models and batch workloads, where a 100–200ms budget is acceptable and utilisation matters more than proximity.
• Core — training, fine-tuning, and frontier-model inference, where the workload genuinely needs the big iron.

The piece that makes this an architecture rather than a diagram is the orchestrator: something that looks at each request’s latency budget, capability requirement, and — increasingly — sovereignty constraint, and routes it to the cheapest tier that satisfies all three. Latency, capability, jurisdiction: pick the cheapest node inside the intersection. (The routing calculator on the insight page lets you play with exactly this: set the constraints and watch the eligible-region set collapse around them.)

The economic logic is blunt. You stop paying for hyperscale GPU time when what the request actually needed was a 3B-parameter model returning a sub-50ms answer to a user far from your core region. The model-size question and the placement question turn out to be the same question — which is why model compression and the grid belong in the same conversation: quantised small models are what make edge tiers viable at all.

Where it actually changes outcomes

The honest qualifier: most enterprise AI workloads today don’t need this. Internal copilots, overnight document processing, batch classification — a central region serves them fine, and the grid adds complexity they’ll never repay. The grid matters where latency is part of the product:

• Real-time video and media personalisation — decisions made per viewer, per moment, inside the delivery path.
• Gaming — AI behaviour that has to land inside a frame budget, not a network round trip.
• Customer-facing agents — a support agent that responds instantly in Tokyo and London, not “instantly in us-east-1 and 400ms everywhere else”.
• Financial services and retail — recommendations and risk decisions where the value decays in milliseconds.

The pattern across all of these: the user is global, the interaction is synchronous, and the tolerable delay is smaller than the speed-of-light tax to your core region. When those three line up, centralised inference isn’t just slower — it’s structurally more expensive, because you end up over-provisioning frontier capacity to compensate for a problem that a small model at the edge solves outright.

The strategic read

Two things follow if this architecture wins.

First, the commercial advantage tilts toward whoever already owns a distributed footprint. Building thousands of edge locations from scratch is a decade of capital expenditure; adding GPUs to an existing global network is an upgrade cycle. That’s true for the edge-first networks and CDN providers as a category, and it inverts the usual cloud logic where scale meant concentration.

Second, inference economics become a placement decision, not just a model decision. The questions I walk through in AI Cost Curves — API vs self-hosted, where the crossover sits — gain a third axis: where the self-hosted option runs. A workload that doesn’t pencil out in a central region at frontier-model prices can pencil out very comfortably as a compressed model on regional or edge capacity.

None of this makes the centralised AI factory obsolete — training still wants concentration, and frontier inference still needs the core. What it ends is the assumption that inference defaults there. Compute follows content, content follows users, and users are everywhere. The CDN people have been saying that for twenty-five years. This time the payload is a model.