# FinOps for AI

Source: John Bradshaw — https://bradshaw.cloud
URL: https://bradshaw.cloud/strategy/finops-for-ai/
Type: insight
Date: 2026-07-06
Updated: 2026-08-15
Tags: Economics, AI, FinOps, Governance

> AI spend scales with autonomy, not traffic. The circuit breakers, tier routing, caching, and chargeback discipline that keep LLM and agent spend answerable to somebody.

## Where AI spend goes wrong

### AI spend scales with autonomy, not traffic

An agent differs from a chatbot in one financially important way: it acts in a loop — perceive, reason, act, repeat — until it decides the task is finished. Most model APIs bill the full conversation history on every call, so each step of a loop resends everything that came before it. An agent that takes thirty steps doesn't cost three times one that takes ten — it costs closer to nine times, because the context grows with every pass.

**Business take:** Give an agent an API key and broad instructions and you've handed an intern a corporate credit card with "do whatever you think is best" written on the envelope. The intern isn't malicious — they're diligent, uncapped, and unsupervised. Traffic unchanged, users unchanged, spend tripled: the culprit is almost always the freedom you've given the software.

### Three circuit breakers every agent needs

None of this requires exotic engineering — it's the discipline distributed systems learned decades ago, applied to a component that spends money instead of threads. A hard step cap catches wandering. A repetition check catches oscillation — an agent bouncing between two states stays comfortably under any generous cap while burning money the entire time. A session budget that actually kills the credential catches everything else.

**Business take:** The test to apply: if this agent's prompt were subtly broken tomorrow, what is the maximum it could spend before a human found out? If the honest answer is "whatever's on the card", you don't have a cost model — you have exposure. Meter the budget at a gateway, never inside the agent itself: asking the runaway process to police itself is not a control.

### Seeing the bill isn't governing it

Measurement is not governance. Knowing you spent $100,000 with a model provider last month is trivia unless you also know which team spent it, on which use case, and which of those would survive a business-case review. The maturity path runs from basic showback (request counts by team), through advanced showback (tokens and cost), to entitlement enforcement (limits applied at the gateway before the overrun), and finally chargeback (usage drives real internal invoices).

**Business take:** Most organisations are at level one and believe they're at three. The tell is a simple question: who gets paged when a team blows through its AI budget? If the answer is "nobody, we review it monthly", the circuit breakers have no owner — and an unowned control is a decoration. Attribution is what makes savings durable, because attribution creates owners, and owners notice drift.

## Decision framework

### If an agent's prompt broke tomorrow, what's the maximum it could spend?

**If the answer is "whatever's on the card", add circuit breakers before anything else.**

Three controls, in order: a hard step cap (an agent that hasn't converged by step ten or fifteen is looping, not persevering), a repetition check (embedding similarity against the last few steps catches oscillation), and a session token budget metered at a gateway that kills the credential when spent — never inside the agent itself.

### Is every query going to your most expensive model?

**Route by complexity: 80–90% of traffic to small, cheap tiers; frontier reasoning for the rest.**

Using a frontier model to classify sentiment is couriering a Post-it note. Routing is a budget control, not just an architecture choice — an estate without model tiering has a cost floor set by its most expensive model, and no amount of dashboarding will lower it.

### Does your traffic repeat?

**Cache what repeats: up to ~70% savings on support/FAQ flows, closer to 10% on diverse traffic.**

Semantic caching serves a paid-for answer when a new query means the same thing as a previous one. The honest number is the spread — cache the workloads that repeat, and don't build the budget on the ceiling. Add context compaction for RAG workloads, where retrieval bloat runs 3–5x the tokens of a plain query.

### Who gets paged when a team blows through its AI budget?

**If nobody, you have showback, not governance. Climb the maturity ladder to enforcement.**

The path runs basic showback → advanced showback → entitlement enforcement (limits applied at the gateway before the overrun) → chargeback (usage drives real internal invoices). Attribution creates owners, and owners notice drift — an unowned control is a decoration.

## Frequently asked questions

### Why is AI spend different from ordinary cloud spend?

Because it scales with autonomy, not traffic. An agent acts in a loop until it decides the task is finished, and most model APIs bill the full conversation history on every call — so each step resends everything before it. An agent that takes thirty steps does not cost three times one that takes ten; it costs closer to nine times. Users unchanged, traffic unchanged, spend tripled.

### What circuit breakers does every agent need?

Three, in order. A hard step cap, because an agent that has not converged by step ten or fifteen is looping rather than persevering. A repetition check — embedding similarity against recent steps — because an agent oscillating between two states stays comfortably under any generous cap while burning money the whole time. And a session token budget that actually kills the credential when spent. Meter all of it at a gateway, never inside the agent: asking a runaway process to police itself is not a control.

### How much does model tier routing save?

Enough that it is a budget control rather than an architecture preference. Routing 80–90% of traffic to small, cheap tiers and reserving frontier reasoning for the remainder attacks the largest single line item in most AI estates. An estate without tiering has a cost floor set by its most expensive model, and no amount of dashboarding will lower it — using a frontier model to classify sentiment is couriering a Post-it note.

### Does semantic caching actually work?

It works where traffic repeats, and the honest number is a spread rather than a headline. Serving a previously paid-for answer when a new query means the same thing can reach around 70% savings on support and FAQ flows, but lands closer to 10% on genuinely diverse traffic. Cache the workloads that repeat and do not build the budget on the ceiling. For RAG workloads, add context compaction — retrieval bloat commonly runs three to five times the tokens of a plain query.

### What is the difference between seeing AI spend and governing it?

One question: who gets paged when a team blows through its AI budget? If the answer is "nobody, we review it monthly", you have showback, not governance. The path runs from basic showback through advanced showback to entitlement enforcement — limits applied at the gateway before the overrun — and finally chargeback, where usage drives real internal invoices. Attribution is what makes savings durable, because attribution creates owners, and owners notice drift.

### When should I self-host a model rather than use an API?

When the workload is sufficiently predictable and sustained that fully loaded capacity plus operations is cheaper than the managed API path — and when the value of control, residency or latency justifies the operating responsibility. Compare unit cost at the required quality, then include utilisation, reliability, security, evaluation, on-call and transition cost. A cheap GPU list price is not a deployment model.

## Diagrams

Standalone SVG diagrams, licensed CC BY 4.0 with attribution to John Bradshaw — link back to https://bradshaw.cloud/strategy/finops-for-ai/ when embedding.

- [AI spend scales with autonomy, not traffic (SVG)](https://bradshaw.cloud/strategy/finops-for-ai/diagrams/loop-cost.svg)
- [Three circuit breakers every agent needs (SVG)](https://bradshaw.cloud/strategy/finops-for-ai/diagrams/breakers.svg)
- [Seeing the bill isn't governing it (SVG)](https://bradshaw.cloud/strategy/finops-for-ai/diagrams/maturity.svg)

## Interactive companion

**FinOps for AI: Cost Governance Calculator** — Model your AI spend across tier routing, semantic caching, agentic budget controls, and API-versus-self-hosted deployment TCO — find where governance saves money before the invoice arrives.

Live calculator: https://bradshaw.cloud/strategy/finops-for-ai/calculator/

## Related

- [Read the full article](https://bradshaw.cloud/writing/finops-for-ai/) — why the LLM bill explodes, and how to stop it.
- [AI FinOps metrics](https://bradshaw.cloud/writing/finops-for-ai-metrics/) — the scorecard that turns spend into decisions.
- [Agentic AI control planes](https://bradshaw.cloud/writing/agentic-ai-control-plane/) — the guardrails that keep autonomy and cost operable.
- [AI cost curves](https://bradshaw.cloud/strategy/ai-cost-curves/) — the API vs self-hosting crossover economics.
- [Typed decision models](https://bradshaw.cloud/strategy/system-one-models/) — typed decisions cheap enough to sit on every fork.
- [Agentic AI patterns](https://bradshaw.cloud/strategy/agentic-ai/) — choosing agent architectures that don't spend like they're on commission.
- [Cloud repatriation](https://bradshaw.cloud/strategy/cloud-repatriation/) — the same portfolio discipline applied to infrastructure placement.
- [All strategy insights](https://bradshaw.cloud/strategy/)