AI Cost Curves: When API Pricing Becomes Your Biggest Infrastructure Decision

There’s a moment in every AI deployment where someone opens the monthly invoice and has a very uncomfortable meeting. The PoC cost $3,000 a month. The production rollout costs $300,000. Nothing changed except volume.

I’ve watched this happen at multiple organisations. The economics of AI inference are not intuitive, and the cost curves don’t behave like anything else in enterprise IT. Understanding where the lines cross is one of the most consequential infrastructure decisions a technology leader makes today.

The API trap: elegant simplicity, linear cost

API pricing for AI models is beautifully simple. You pay per token in, per token out. At low volumes, there’s nothing to beat. No infrastructure. No GPUs. No ops team. You’re renting someone else’s compute by the millisecond.

But the cost scales linearly. Double your queries, double your bill. There’s no economy of scale, no volume discount that matters when you’re processing 100,000+ queries per day.

Take one concrete query and hold it fixed for the rest of this piece: a retrieval-augmented answer with 4,000 tokens of context going in and 600 tokens coming out. At frontier list pricing — call it $2.50 per million input tokens and $10 per million output — that query costs $0.016. Every figure below is that query on a 30-day month, so the arithmetic is yours to check.

At 1,000 queries per day, that’s $480 a month. Manageable. At 100,000 queries per day, it’s $48,000 a month for the same query. At that point, you’re not paying for AI. You’re paying rent on someone else’s GPUs.

The analogy I use: API pricing is like taking a taxi. Cheap for short trips. Ruinous for a daily commute.

Self-hosting: the car purchase

Self-hosting means GPU servers. A100s, H100s, or their cloud equivalents. Matching frontier quality means a 70B-class open-weight model, which means four H100-class GPUs at roughly $2,250 each per month, plus about $3,000 a month of platform engineering to keep it serving. Call it $12,000 a month — the low end of the $10K-30K range a capable cluster lands in — and it will handle around 40,000 of our 4,000-in / 600-out queries a day. Once you’re paying that fixed cost, the marginal cost per query approaches zero. Your 40,000th query costs the same as your first.

The economics are a step function. You pay for capacity in chunks, and within each chunk, additional queries are effectively free. The second chunk takes you to $21,000 a month and about 80,000 queries a day; the third to $30,000 and 120,000.

Drop the quality bar and the numbers change completely. A 7B model on a single A10G-class GPU handles roughly 20,000 of these queries a day for about $700 a month of hardware — call it 40,000 with 4-bit weights. The same 20,000 a day through a frontier API is $9,600. That is not a like-for-like comparison — a 7B model is not a frontier model — and that gap between the two is precisely what the hybrid pattern below exists to harvest.

But self-hosting brings operational complexity. You need MLOps expertise, model management, monitoring, scaling, and on-call support. The question isn’t whether it’s cheaper at scale. It’s whether the operational overhead is worth the savings.

The crossover point

There’s a specific volume where the lines cross. API cost rising linearly meets self-hosted cost sitting flat. Below the crossover, API wins on simplicity. Above it, self-hosting wins on economics — with one caveat worth knowing about, which is that a step function can cross a straight line more than once.

That caveat is the first capacity boundary. The four-GPU chunk runs out at 40,000 queries a day, where the API bill is $19,200 — and buying the next four GPUs costs $21,000, so the API is briefly the cheaper option again until about 43,750 a day. It is a narrow window, roughly 3,750 queries a day wide, and it’s the only one: by the time the second chunk fills at 80,000 a day the API is at $38,400 and the third chunk’s $30,000 has been the cheaper option since 62,500. Worth knowing if you’re sizing a cluster right at a boundary; not worth redrawing the decision over.

That first crossing is arithmetic, not judgement — and the only variable in it that moves much is the price of a query. Change the query shape and the crossover moves with it. A long agentic call, 8,000 tokens in and 2,000 out, costs $0.040, and crosses the same $12,000 floor at 10,000 queries a day. A short classification query, 1,600 in and 400 out, costs $0.008, and doesn’t cross until 50,000 a day.

That is where the 10,000–50,000 band comes from. It isn’t a rule of thumb — it’s one fixed cost divided by the range of per-query prices that real production workloads actually run at.

The variables that shift the crossover:

• Model size: A 7B parameter model is cheap to self-host. A 70B parameter model requires serious GPU investment.
• Query complexity: Simple classification tasks cost less per query than long-form generation.
• Latency requirements: Edge inference (close to users) shifts the equation toward self-hosting because API round-trips add 200-500ms you can’t optimise away.
• Data sensitivity: If your data can’t leave your infrastructure, self-hosting isn’t a cost decision. It’s a compliance requirement.

My advice: don’t guess. Run your actual production workload on a self-hosted model for a week. Compare cost, latency, and output quality side by side. The crossover point is different for every use case.

The hybrid answer

The pragmatic answer is rarely pure API or pure self-hosted. The approach I recommend and have seen work best in production:

Route by complexity. Simple, high-volume queries go to a cheap self-hosted small language model (SLM). Complex, nuanced queries that need frontier reasoning go to the API.

A typical production split: 80-90% of queries handled by a 7B-14B self-hosted model at near-zero marginal cost. The remaining 10-20% routed to a frontier API for the hard cases.

At 100,000 queries a day, that’s 85,000 on a small model — five A10G-class GPUs at about $700 a month each, plus the same $3,000 ops floor, so $6,500 — and 15,000 a day still going to the frontier API at $0.016, which is $7,200. Total $13,700 a month, against $48,000 for pure API and $30,000 for a frontier-quality cluster you run yourself. Roughly 70% off the API bill, with the frontier model still on hand for the queries that need it.

The routing logic doesn’t need to be sophisticated. Task classification works well: simple lookups, summaries, and classification go to the SLM. Multi-step reasoning, creative generation, and ambiguous queries go to the API. You get most of the saving without lowering the quality ceiling.

What this means for technology leaders

The AI cost curve decision isn’t just an infrastructure choice. It’s a strategic one.

If you’re running at low volume (under 1,000 queries per day), use APIs. $480 a month of tokens against a $12,000 floor isn’t a close call, and the infrastructure overhead of self-hosting costs more than the tokens do.

If you’re between 1,000 and 50,000 queries per day, you’re in the benchmark zone. Run the numbers on your own query shape. The crossover might be closer than you think.

If you’re above 50,000 queries per day and still on pure API, the gap is real and it widens with every increment: at 100,000 a day it’s $48,000 against $30,000 self-hosted, or $13,700 hybrid. Self-host the bulk workload. Keep the API for frontier-quality edge cases.

And if latency matters, if you’re processing at the edge, if your users expect sub-100ms responses, self-hosting with quantised models isn’t optional. It’s the only architecture that works.

The organisations getting this right are treating AI inference as a portfolio, not a binary choice. Different models for different jobs, at different price points, all orchestrated through a single routing layer.

---

Explore the interactive AI Cost Curves framework for visual diagrams showing the crossover point, cost curves, and the hybrid routing architecture.