Model Routing: The Cheapest Model Is the One You Didn't Call

Most AI estates I look at made their biggest cost decision before anyone wrote a line of product code. Someone picked a model, wired it into the client library, and every feature since has called it. That choice sets the cost floor for the whole estate, and no amount of prompt tuning or committed-spend negotiation moves a floor. The cheapest model call is the frontier call you didn’t make.

Routing is the practice of making that call less often: sending each request to the smallest model that can answer it well enough, and escalating only when it can’t. I have touched on it before, in AI Cost Curves as a hybrid architecture and in FinOps for AI as a budget control. Once routing decides which model answers, it is also deciding cost, latency, residency and quality for every request in the estate. That makes it the control plane of an AI platform, whether or not anyone designed it as one.

One model everywhere is a procurement decision, not an architecture

The single-model estate is rarely chosen on its merits. It is what you get when the first proof of concept used the most capable model available, the evaluation passed, and production inherited the configuration. Nobody decided that sentiment classification, field extraction and multi-step contract analysis should cost the same per token. It just happened, because the client library had one model name in it.

The reason it persists is that it is easy to evaluate. One model means one evaluation suite, one set of failure modes, one line on the invoice. Routing breaks that simplicity on purpose. But the default is not neutral. A frontier model answering a question a small model would have answered identically is pure margin handed to whoever runs the frontier model.

The useful split is not “frontier versus open-weight” as a matter of taste. It is a question about each task: what is the smallest model that clears the quality bar for this request, and what does it cost to find out? The frontier models and system-one models insights cover the two ends of that spectrum. Routing is the layer that lets you use both.

A worked example, with illustrative numbers

Every number in this section is illustrative — made up to show the shape of the calculation, not drawn from any real price list or workload. Plug in your own.

Assume three tiers, priced in relative units per request: a small model at 1, a mid-sized model at 5, a frontier model at 25. Assume a traffic mix of 70% simple requests, 20% medium, 10% hard.

Three things fall out of that table, and they hold whatever numbers you substitute.

Perfect routing is a ceiling you will not reach. It assumes you know the difficulty of a request before you have answered it. You don’t. Every real router either guesses up front with a classifier, or finds out by trying the small model and checking the result.

The escalation rate is the number that decides it. A cascade pays for the small model on every request and for the frontier model on every escalated one. In this example, a cascade stops paying for itself once it escalates more than 96% of traffic, which is one minus the small-model price divided by the frontier-model price (1 - 1/25). Long before that point it is paying a latency penalty: an escalated request makes two round trips, and its user waits for both.

The price ratio sets the share you save; your spend decides whether that share is worth having. If the gap between the small and frontier tiers narrows, routing saves a smaller share of the bill. If it widens, a larger one. But a ratio says nothing about cash. If both tiers halve in price, the ratio and the percentage saving stay the same while the saving in money halves, and it can fall below what the router costs to build and run. Watch the ratio to see how much routing could save, and your absolute frontier spend to see whether that saving pays for the router.

If you want to run your own volumes, the FinOps for AI calculator models tier routing alongside caching and agent budget controls.

The routing maturity ladder

Routing is not one design. In practice I see estates sit on one of five rungs, and most of the trouble comes from believing you are higher up than you are.

1. One model everywhere. The default described above. Cost floor set by the most expensive model.
2. Static per-feature assignment. Each feature is pinned to a model at build time. Summaries go to a small model; the analyst assistant goes to a frontier one. Cheap to run and easy to reason about, and it captures a large share of the saving, because difficulty correlates strongly with feature.
3. Rules-based routing. Requests are routed on attributes you can read without a model: prompt length, presence of attachments, user tier, the tool being called. Transparent and auditable, and it fails quietly when the attributes stop predicting difficulty.
4. Classifier or cascade. A small model, or a dedicated classifier, decides the tier per request; or the small model answers first and a confidence or validation check decides whether to escalate. This is where the larger savings live, and where evaluation becomes non-optional.
5. Policy-driven control plane. Routing reads more than difficulty. It knows each request’s latency budget, its data classification and permitted jurisdictions, the remaining budget for its team, and the health of each model endpoint, and it picks the cheapest model that satisfies all of them.

The fifth rung is the reason I call routing a control plane. Once every request passes through one decision point, that point becomes the natural home for policy you would otherwise scatter across every application. A request carrying regulated data can only go to models hosted in permitted jurisdictions, or to a self-hosted open-weight model inside your own boundary. That is the same distinction I drew in Data Residency Is Not Sovereign AI: where the model runs, and under whose control, is a routing decision as much as a hosting one. A degraded endpoint fails over to an equivalent model rather than to an error page. A team past its budget gets the small tier, or a refusal, rather than an invoice surprise at month end.

The AI Grid insight describes a router too, but a different one: it picks where to run a request across locations. Model routing picks which model answers it. A mature platform ends up with both, and they read many of the same inputs.

What goes wrong

Routing moves cost out of the invoice and into operations. Three failure modes account for most of that cost.

Silent quality drift. A router’s aggregate quality score can hold steady while one slice of traffic gets markedly worse. The small model handles 95% of billing questions well and quietly mishandles the 5% that involve a refund dispute, and the average hides it. Worse, the boundary moves without you touching anything. When a model behind an endpoint is updated, the set of requests it can answer well changes, and a threshold tuned last quarter is now routing the wrong traffic. The defence is per-route, per-slice evaluation that runs continuously, not a launch-day benchmark.

Evaluation debt. One model needs one evaluation suite. A router with three tiers needs evidence that each tier is good enough for the traffic it receives, plus evidence that the router is sending the right traffic to each. Every tier you add multiplies the evaluation work, and teams that add tiers faster than they add evaluation are borrowing against quality they can no longer see. I treat a routing change without an evaluation change as unreviewed, in the same way a schema migration without a rollback is unreviewed.

Routing logic as a hidden product decision. An escalation threshold decides which customers get the better answer. A rule that sends free-tier users to the small model is a pricing decision. A residency rule that refuses a request is a product behaviour. These decisions end up in a configuration file maintained by a platform engineer, with no product owner reviewing them. Each one should have a named owner, and that owner should be able to explain it to a customer. If the person who can explain a threshold is the one who typed it, it is not owned.

When routing isn’t worth it

Routing has a fixed cost: a classifier or a cascade to build, an evaluation set per route, monitoring per slice, and a new component on the request path that can fail. It only pays when the variable saving clears that.

It usually doesn’t when volume is low. A few thousand requests a day to a frontier model is a rounding error next to the engineering time a router consumes. It doesn’t when the traffic is uniformly hard, because a cascade that escalates nearly everything costs more than calling the frontier model directly. And it doesn’t when you cannot measure quality per request, because a router you cannot evaluate is a cost cut you cannot defend when someone asks why answers got worse.

The test I apply is one question about the existing estate: of last week’s frontier calls, what fraction would a small model have answered acceptably? If nobody can answer that, the first piece of work is not a router. It is a labelled sample of real traffic and an agreed definition of “acceptably”. Both are useful even if you never build the router, and without them you cannot build one safely.

Own the policy, not the plumbing

The routing mechanism itself is not where the value sits. Sending a request to one of several endpoints, retrying on failure and recording what happened is commodity plumbing.

The value is in the policy and the evidence: your traffic mix, your quality bar per task, your evaluation sets, your residency rules and your budget owners. Those are yours however the mechanism is run. Keep the policy as configuration you own and version, and keep the evaluation sets in your own repository.

The bottom line

Picking a cheaper model is a one-off saving. Routing is a standing one, and it compounds as the tiers diverge in price. It also adds a component that decides quality, cost and compliance for every request, and that component needs what any control plane needs: an owner, an evaluation suite, monitoring per slice, and a change process. Route the traffic, but govern the router. A router without those things will save money in the first quarter and lose the business’s trust after that.

---

Read AI Cost Curves for the API versus self-hosting crossover that decides where each tier runs, and FinOps for AI Metrics for the scorecard that tells you whether routing is moving the numbers that matter. The glossary defines model routing and hybrid routing alongside the rest of the inference vocabulary.