FinOps for AI: Why Your LLM Bill Is Exploding — and How to Stop It

There’s a new flavour of invoice shock doing the rounds, and it isn’t the one I wrote about in AI Cost Curves. That one was volume: the PoC costs $3,000 a month, production costs $300,000, and nothing changed except the number of users. Painful, but at least it’s arithmetic. The new one is worse, because the bill went up while usage stayed flat.

I’ve watched finance teams try to reconcile that and fail. Traffic: unchanged. Users: unchanged. Spend: tripled. The culprit, almost every time, is autonomy. Somewhere in the estate, an agent — or a fleet of them — started making its own decisions about how much compute to consume. AI spend no longer scales with traffic. It scales with how much freedom you’ve given the software.

FinOps grew up on infrastructure that sits still

The discipline has certainly arrived. The FinOps Foundation’s State of FinOps 2026 survey found that 98% of practitioners now manage AI spend, up from 31% in 2024. In two years, AI went from a curiosity on the cost dashboard to the thing the dashboard is for.

The toolkit hasn’t kept pace. Classic FinOps was built for infrastructure that sits still: right-size the instances, buy the commitments, tag the idle VMs, delete the orphaned volumes. It works because cost is a proxy for capacity and capacity is a proxy for demand. AI inference breaks that chain. Token spend moves with prompt design, context size, model choice, retry behaviour and — once agents arrive — the number of steps the software decides to take on your behalf. Two requests with identical business value can differ in cost by two orders of magnitude, and nothing on a utilisation graph will tell you why.

I made a related point about cloud repatriation: a structural cost gap and a FinOps failure look identical on an invoice, and you need to know which one you have before you act. AI adds a third possibility — the software behaving exactly as designed, and the design having no brakes.

The agentic trap

An agent differs from a chatbot in one financially important way: it acts in a loop. Perceive, reason, act, repeat — until it decides the task is finished. A chatbot answers and stops. A confused agent rarely concludes that it’s finished, because from the inside, trying again always looks like progress.

The arithmetic is the nasty part. Most model APIs bill the full conversation history on every call, so each step of a loop resends everything that came before it. An agent that takes thirty steps doesn’t cost three times one that takes ten — it costs closer to nine times, because the context grows with every pass. Loop cost doesn’t grow linearly with step count. It grows roughly with the square of it.

The war stories circulating — an agent retrying a failing API call millions of times over a weekend, two agents politely asking each other to verify their work for eleven days straight — are mostly unverifiable, and I’d treat the dollar figures attached to them as folklore. The mechanism behind them is not. Give an agent an API key and broad instructions and you’ve handed an intern a corporate credit card with “do whatever you think is best” written on the envelope. The intern isn’t malicious. They’re diligent, uncapped, and unsupervised — and the diligence is exactly the problem.

The choice of agent architecture compounds this: pattern selection alone carries a 2–5x cost multiplier, which I covered in Agentic AI Patterns. Whatever pattern you pick, though, it needs brakes.

Three circuit breakers every agent needs

None of this requires exotic engineering. It’s the discipline distributed systems learned decades ago — timeouts, circuit breakers, bulkheads — applied to a component that spends money instead of threads. Three controls, in the order I’d add them:

1. A hard step cap. Most tasks an agent can solve at all, it solves in the first handful of steps. In my experience, an agent that hasn’t converged by step ten or fifteen isn’t persevering — it’s looping or hallucinating, and every further step is spent proving that. Treat the cap like a timeout: generous enough for real work, tight enough that failure is cheap. That’s a rule of thumb, not a law; tune it per task type and revisit it when the traces tell you to.

2. A repetition check. Step caps miss oscillation. An agent bouncing between two states — A, B, A, B — stays comfortably under any generous cap while burning money the entire time. The fix is to compare each reasoning step against the last few (embedding similarity is the usual trick) and interrupt when the agent is saying essentially the same thing twice. It isn’t thinking at that point. It’s pacing.

3. A session budget that actually kills. Never give an agent an open-ended credential. Issue each session a token allowance, meter it at a proxy or gateway — never inside the agent itself, which is asking the runaway process to police itself — and when the allowance is spent, the credential dies. The task fails loudly. Loud failure is the point: a dead agent is a bug report; an undead one is a budget line.

The test I apply: if this agent’s prompt were subtly broken tomorrow, what is the maximum it could spend before a human found out? If the honest answer is “whatever’s on the card”, you don’t have a cost model. You have exposure.

Route by complexity — the maths lives elsewhere

The second-biggest source of AI waste is much duller: sending every query to the most expensive model you have. Using a frontier model to classify sentiment is couriering a Post-it note.

The fix is routing — send each task to the cheapest tier that clears production quality. In practice that means 80–90% of traffic handled by a small, cheap model and frontier reasoning reserved for the 10–20% that genuinely needs it. I’ve written up the crossover economics in full, and there’s an interactive calculator if you want to run your own volumes, so I won’t repeat the maths here.

From a FinOps perspective, the point is simpler: routing is a budget control, not just an architecture choice. An estate without model tiering has a cost floor set by its most expensive model, and no amount of dashboarding will lower it.

The cost of context

Output tokens are the expensive ones. Input tokens are the ones that get you.

Every time an agent retrieves context — documents, records, prior conversation — it bloats the prompt, and you pay for that bloat on every subsequent call. Analyses from the data-governance world put retrieval-augmented queries at 3–5x the tokens of a plain query, and poor upstream data quality makes it worse: duplicates and stale records force bigger retrievals to compensate. Data hygiene has quietly become a FinOps line item.

The counterweight is semantic caching — serving a cached answer when a new query means the same thing as one you’ve already paid for. Redis’s engineering team reports savings of up to 73% on high-repetition workloads like customer support and FAQ flows; on genuinely diverse traffic, the figure is closer to 10%. That spread is the honest number. Cache the workloads that repeat, and don’t build the budget on the ceiling.

Seeing the bill isn’t governing it

Measurement is not governance. Knowing you spent $100,000 with a model provider last month is trivia unless you also know which team spent it, on which use case, and which of those use cases would survive a business-case review.

What that takes is attribution: every call tagged with a team, a use case, and a data domain, enforced at a central gateway — because at enterprise scale, API-key-level tracking produces noise, not insight. API-gateway vendors describe a four-level maturity path for what happens next, and it matches what I see in the field:

1. Basic showback. Request volumes by team. Enough to know who’s calling.
2. Advanced showback. Token counts, provider costs, model versions. Enough to know who’s spending.
3. Entitlement enforcement. Usage limits per team or product, enforced at the gateway before the overrun — not reported after it.
4. Chargeback. Usage drives real internal invoices, and AI cost becomes a defensible input to budget planning rather than a surprise in arrears.

Most organisations I meet are at level one and believe they’re at three. The tell is a simple question: who gets paged when a team blows through its AI budget? If the answer is “nobody, we review it monthly”, then the circuit breakers from earlier have no owner — and an unowned control is a decoration.

The bottom line

Caching, routing, and circuit breakers deliver real savings — I’ve seen each cut spend by double-digit percentages, though treat every number in this post as a rule of thumb rather than a law; your workload will move the decimals. The catch is that they’re one-off wins unless the operating model holds them in place. Attribution is what makes them durable, because attribution creates owners, and owners notice drift.

So stop treating AI inference as a single line item and start managing it as a portfolio of capabilities — each with an owner, a budget, and a kill switch. Your cloud bill used to measure how much you built. Your AI bill measures how much you’ve delegated. Govern it like that.

---

Run your own numbers in the FinOps for AI cost governance calculator — tier routing, caching, and agentic budget controls against your workload. Then explore the interactive AI Cost Curves framework for the API vs self-hosting crossover maths, and the Agentic AI Patterns guide for choosing agent architectures that don’t spend like they’re on commission.