AI Inference Economics

The short version

Inference cost behaves nothing like the rest of the infrastructure budget, and that is what catches teams out. Training is a capital event you plan for. Inference is a per-query tax you pay forever, and it scales with exactly the thing you were hoping for: usage. The proof-of-concept that cost a few thousand a month becomes a six-figure line item on the same code, because nothing changed except volume.

The shape of the problem is two curves crossing. Token price makes API inference linear — double the queries, double the bill, with no volume discount that meaningfully bends the line. Self-hosting is a step function: you buy capacity in chunks, and inside a chunk the marginal query is close to free. On a fixed worked example — a retrieval-augmented answer with 4,000 tokens in and 600 out, about $0.016 through a frontier API — the lines meet somewhere near 25,000 queries a day, where roughly $12,000 a month of self-hosted capacity starts to undercut the metered bill. Your crossover will sit somewhere else. The point is that you have one, and that most teams have never worked out where.

Almost nobody should land on either end of that curve. The durable answer is a tiered one: route the easy majority of traffic to a small or mid-tier model you run yourself, and reserve the frontier API for the queries that genuinely need it. Most production traffic is repetitive, and a mid-tier model handles it at a fraction of the token cost — the gap between what a 7B-class model costs to serve and what the same volume costs through a frontier endpoint is the entire prize. Getting that routing right is worth more than any negotiation over list pricing.

Agentic workflows change the arithmetic again, and mostly for the worse. A single user request that used to be one call becomes a chain of them — plan, retrieve, call a tool, check the result, retry — so token costs multiply by a factor nobody modelled at design time. A ten-step agent on a query you priced as one is a 10x surprise, and it arrives after launch, when the usage curve is already climbing. The number to watch is cost per task, not cost per call.

Then there is placement, which is the part that gets treated as an afterthought. Where inference physically runs sets your latency floor, your egress bill, and increasingly your regulatory position — data that cannot leave a jurisdiction cannot be inferred over in another one. Centralised inference is the simple default and the right answer for plenty of workloads. It stops being right the moment round-trip latency is part of the product, or the data has gravity of its own. That is an architecture decision made early and expensively revisited late.