A/B Testing Models: Your Model Choice Is a Hypothesis, Not a Decision

· · 9 min read

Controlled experiments are routine for interface changes and rare for the model behind a product. With model prices and capability moving every few weeks, an unchallenged incumbent is a standing cost, and a staged production experiment with a written non-inferiority margin is how to measure it.

Listen, 9:12 Synthetic voice cloned from John, with his consent

Audio edition

Chapters
  1. Chapters load when you press play.

Read by a synthetic voice cloned from John with his consent. About the audio edition

A marketing team will not change the wording on a checkout button without a controlled experiment. The model behind the product was picked at the proof of concept and has answered every request since without facing a challenger. The model in production is a hypothesis that nobody has re-tested.

Model Routing argued for sending each request to the smallest model that can answer it well. Routing gives you a table that maps traffic to models, and the table starts going stale the day you write it. A/B testing is how the table stays right: a challenger model takes a measured share of real traffic, and the incumbent keeps its place only if the evidence says so.

The method is thirteen years old, and the model estate has not adopted it

Running many experiments at once on live traffic stopped being novel a long time ago. A 2013 KDD paper, Online Controlled Experiments at Large Scale (opens in a new tab), described one web search engine doing it as routine, and its authors’ own summary (opens in a new tab) put the figure at more than 250 experiments running on a typical day. The same summary carried the finding that matters here: fewer than a third of ideas moved the metric they were designed to improve.

Those ideas came from informed people with data in front of them, and they were wrong more than two times in three. A model picked by reading benchmark tables and trying a week of prompts is the same kind of informed guess. It may be the right one. The experiment is what tells you.

The evidence that AI teams have not carried the habit across is thinner, and I would rather say so than dress it up. LangChain surveyed (opens in a new tab) 1,340 practitioners in late 2025: 57.3% had agents in production, 52.4% ran offline evaluations, and 37.3% scored live traffic at all. More than three quarters used several models. That is a vendor’s survey of people already building agents, and it measures evaluation, not randomised comparison between models. I know of no survey that measures that directly. But if barely a third of teams score production traffic, the share running a controlled test between two models on it is smaller again.

The pattern I see most often is a single general-purpose model baselined early as good enough and then applied to everything that arrives: catalogue copy, customer service chat, whatever the next feature needs. Nobody set a quality bar per task or a cost per successful task, so there is nothing to test a challenger against, and “good enough” was a judgement made once.

The control arm has an expiry date

A button test has a stable control. The existing button stays the same price and keeps working. A model test has neither property.

The price moves. Stanford’s 2025 AI Index (opens in a new tab) put the fall in the price of a fixed level of capability at 280-fold between November 2022 and October 2024. Epoch AI measured (opens in a new tab) declines of between 9 and 900 times a year depending on the task, and a later academic estimate (opens in a new tab) for frontier-level performance came in at 5 to 10 times a year. Those are prices for reaching a benchmark score, not for doing your task, so treat the direction as well-evidenced and the multiples as decoration.

The supply moves too. One public release tracker (opens in a new tab) counted 103 notable model releases in the year to 21 July 2026, about one every three and a half days, with just under half of them open-weight. The same tracker showed 335 for the year to 13 November 2026; it is a live count, so the figure has moved since. The industry did not triple its output in four months; more likely, the tracker widened its net. Take the low count and the point survives.

A falling price is an offer, and you collect it only by switching. You can only switch safely with evidence. That leaves an untested estate carrying two costs. The first is on the invoice: paying the incumbent’s price for work a cheaper model now does to the same standard. The second has no invoice at all. It is the feature that failed evaluation a year ago and was shelved, and has not been retried against models that could now do it.

Hosted model versions are also retired on their provider’s schedule. You will run the migration regardless. You can run it as an experiment with a control, or as a forced cutover with a deadline.

A model test is not a button test

The statistics transfer from marketing. Four other things do not.

The arm is a configuration. A model is tested with its prompt, its parameters and its output handling. The production prompt was tuned against the incumbent, so running the challenger on that prompt unchanged is a rigged test that the incumbent wins.

The outcome is not a click. You need a graded result per request: the task completed, the answer accepted without edits, the conversation not escalated to a person. Alongside it sit cost per successful task and p95 latency. Cost per token is the wrong unit, for the reasons in FinOps for AI Metrics. If an automated grader does the scoring, it needs calibrating against human labels first, and it should not share a lineage with either arm.

Most model tests are non-inferiority tests. A marketing test asks whether B beats A. A cost-driven model test asks whether B is no worse than A by more than a margin you will tolerate, at a fraction of the price. The margin has to be written down before the test starts, and it is a product decision. Someone who can explain it to a customer should sign it.

The unit is the user or the conversation. Randomise per request and one user gets two models in the same session, and an agent changes model mid-task.

Three stages, each buying different evidence

StageWhat it answersWhat it costsWhat it cannot tell you
Offline replayDoes the challenger pass the evaluation set?Engineering time and a small inference billWhether the set still resembles this month’s traffic
ShadowHow does it behave on real requests, with its output discarded?Extra inference on the mirrored shareHow users respond to its answers
Live splitDo outcomes hold when users receive its answers?Exposure of real users to the challengerEffects that take longer than the test window to appear

Figure 1The Challenger Path

Each stage admits the challenger to the next only on evidenceOffline replayevaluation setno live trafficpassfailShadowmirrored requestsoutput discardedpassfailLive splitsmall share of usersreceives its answerspassfailPromotechallenger isthe new incumbentStop the testthe incumbent keeps the traffic, and the result is recordedOld incumbentkept for rollback
A challenger reaches users only after passing replay and shadow. A fail at any stage ends the test with the incumbent still serving, and a promotion keeps the old incumbent available for rollback.

Shadow traffic is cheaper than it sounds. Using the illustrative units from the routing piece, with an incumbent at 25 per request and a challenger at 5, mirroring 5% of traffic adds 0.25 units per request. That is 1% on the bill, to find out whether 80% of it is avoidable.

Sample size sets the experiment budget

The numbers here are illustrative. Substitute your own.

Suppose the incumbent completes 90% of tasks and you want to detect a fall to 88%. At the conventional 5% significance and 80% power, that takes about 3,800 graded requests per arm. Halve the gap to one point and the requirement rises to roughly 14,750, because sample size grows with the square of the precision you ask for.

Figure 2Sample Size vs Detectable Gap

Graded requests per armSmallest quality gap to detect (percentage points)04k8k12k16k12345One point: 14,752Two points: 3,843Three: 1,776Five: 687Halve the gap and the sample roughly quadruples
Illustrative: a 90% baseline, 5% significance and 80% power. Detecting a two-point fall takes 3,843 graded requests per arm; a one-point fall takes 14,752.

A feature with 2,000 graded requests a day, sending 10% to the challenger, collects 200 challenger samples a day and finishes in about 19 days. That is around 19 tests a year on that feature, against a market shipping a notable release every few days.

So the constraint is experiment capacity, and the work is deciding which releases earn a slot. I would give one to four triggers and ignore the rest: a price change in a tier you spend heavily on, a retirement notice for a model you run, a traffic slice whose quality has slipped, and a shelved feature worth retrying.

When a live split is the wrong tool

Low volume is the first case. A route with a few hundred requests a day cannot power a two-point comparison in any useful time, and a human reviewing a shadow sample will get there faster.

The second is an output you cannot grade. If nobody can say what a good answer is per request, a test produces a number without a meaning, and the missing piece is the definition, as it was for routing.

The third is regulated output, where exposing users to an unvalidated arm is itself the risk. Stay in shadow and promote through review.

Agent workloads need more samples on the cost side than the quality side. Cost per task is heavy-tailed, and a handful of long loops can decide the average.

And a winning test does not end the work. A new model can mean a new hosting location and a new data-processing review, and that cost belongs in the comparison before the test starts.

Three questions for every model in production

Routing without testing is a snapshot of last year’s market. Give every production route a challenger slot, a named owner and a review date.

The test I apply is three questions for each model in production: when was it last challenged, by what, and on what evidence did it keep its place? If the answer to the first is the date of the proof of concept, the model was inherited, and it is being paid on reputation.


Read Model Routing for the decision point these experiments feed, and AI Cost Curves for the API versus self-hosting crossover that a successful challenger can move.