Glossary

A hard limit that stops an autonomous agent spending unboundedly: a step cap (an agent that has not converged by step ten or fifteen is looping, not persevering), a repetition check against its own recent actions, and a budget ceiling that pages a human. Necessary because agent spend scales with autonomy, not traffic — each loop step typically re-bills the whole conversation history.

See: FinOps for AI

A system that receives a goal, breaks it into steps, and uses tools to accomplish them — reading data, calling APIs, writing files, checking its own work — in a loop of think, act, observe, repeat. A chatbot answers questions; an agent takes actions. That difference is what makes agents useful, and what makes their cost and safety controls matter.

See: Agentic AI Patterns

A plot of the total cost of serving AI inference against request volume, drawn once per infrastructure option (per-token API, self-hosted GPU, hybrid). API pricing is a straight line through the origin; self-hosting is a high fixed cost plus a near-flat marginal cost. Comparing the curves reveals where each option is cheapest as volume grows.

See: AI Cost Curves

NVIDIA's reference design for distributing AI inference across edge locations — in effect, CDN architecture applied to AI. Rather than routing every request to a centralised GPU cluster, GPU compute is placed in the edge locations that already serve web content, and each request is routed to the nearest node capable of handling it. Practical deployments use three tiers (edge, regional, core) with an orchestrator choosing the cheapest tier that meets each request's constraints.

See: AI Grid

Actually billing internal teams for their infrastructure consumption — the step that changes behaviour rather than just awareness. It only works once allocation is comfortably past 90%: turn it on while a large slice of spend still lands in a "shared" bucket nobody trusts, and the year is spent disputing the bill instead of cutting it.

See: FinOps Maturity

Moving a workload out of public cloud onto owned or leased infrastructure — colocation, on-premises, or a hybrid. It is a workload-by-workload placement decision, not an ideology, and almost never wholesale: most organisations repatriate some workloads while only a small single-digit share move off cloud entirely. For genuinely suitable workloads, savings cluster in the 30–60% range on a fully loaded three-to-five-year view.

See: Cloud Repatriation

How much a team has to hold in its head to do its job — the services, languages, datastores, and domain rules it owns. Applying John Sweller's cognitive load theory to team design, it is this rather than headcount that sets a team's ceiling: past the limit, adding people adds communication overhead to a comprehension problem and makes the team slower.

See: Team Topologies

Melvin Conway's 1968 observation that a system's design mirrors the communication structure of the organisation that built it. The reverse Conway manoeuvre inverts it as a tactic: decide the architecture you want, shape the teams and their interactions to match, and let the architecture follow — which is why decoupling a system rarely survives an org chart that keeps the builders coupled.

See: Team Topologies

The first cut in any build-vs-buy decision. Core is what makes you different — the thing customers actually pay for. Context is everything else: payroll, email, logging, CI/CD. A restaurant cooks its own food but does not grow its own wheat; the strategic question is never "can we build this?" but "is this ours to build?".

See: Build vs Buy

Knowing which team, application, and environment every unit of spend belongs to — the foundation under every other cost practice. Until the unallocated remainder is too small to argue about, showback, chargeback, and unit economics all stand on guesses. In practice it is a tagging-discipline problem dressed up as a finance problem: enforce tags at provision time, not in a quarterly cleanup.

See: FinOps Maturity

The request volume at which a fixed-cost self-hosted setup becomes cheaper than per-token API pricing. Found by dividing the fully-loaded monthly cost of self-hosting (hardware plus the engineering time to run it) by the per-request API price. Below it, APIs win on total cost; above it, self-hosting does — provided the hardware stays busy.

See: AI Cost Curves

The tendency of workloads, services, and analytics to accrete around where data already sits, because moving the data is the expensive part. The larger the estate grows, the stronger the pull — which is why data gravity and egress exposure together carry the most weight when scoring workloads for repatriation.

See: Cloud Repatriation

The ability to change your infrastructure arrangements when the economics change — move a workload, renegotiate a contract, replace a service — without rewriting your product. Distinct from data sovereignty, which is about jurisdiction and residency: you can be fully compliant on where data sits and still have no practical ability to leave. The point is leverage, not exit — renewal conversations go differently when the provider knows you have somewhere to go.

See: Economic Sovereignty

Running model inference at points of presence close to users instead of in a central data centre, cutting the network round trip out of the response time. Made economic by quantised small models, which fit the modest GPU capacity an edge site can host — the combination that turns sub-50ms global response times from impossible to routine.

See: AI Grid

The charges a provider levies for moving data out of its cloud — the toll booth on the exit road. Egress exposure is a heavyweight factor in repatriation scoring and a dominant line in any exit-cost model, because it converts the size of your data estate directly into the price of leaving.

See: Cloud Repatriation

What leaving a provider would actually cost: data egress fees, re-architecture effort for every proprietary service you depend on, retraining staff, and contract penalties — modelled at one, three, and five years, because the number climbs every quarter you stay. Most organisations underestimate it by a factor of three to five, which is why it belongs in the commitment decision, not the crisis.

See: Economic Sovereignty

The four layers under any streaming service: Protect (enforcement that makes piracy harder and traceable), Deliver (efficient global distribution), Personalise (context-aware delivery that knows a lunch break from a Friday night), and Monetise (the revenue side of that personalisation). Protect and Deliver are table stakes — necessary, expensive, invisible when they work. The margin lives in Personalise and Monetise.

See: Content Wars to Experience Wars

The competitive shift in streaming from catalogue spend to experience quality. Streaming's original promise solved access; fragmentation broke it again — more apps, higher combined cost, worse discovery. The next round is not won on content budgets but by whoever makes it easiest to find and watch what you actually want.

See: Content Wars to Experience Wars

Specialising a pre-trained model by continuing its training on your own examples, changing the weights themselves. It suits stable knowledge and consistent style or format; unlike retrieval-augmented generation, updating fine-tuned knowledge means training again, which is why RAG usually comes first and fine-tuning is reserved for what genuinely will not change.

See: AI Fundamentals

The operating practice that makes cloud and AI spend an engineering concern rather than a finance afterthought — shared cost visibility, allocation, and efficiency KPIs that engineers own. It works when both sides read the same scoreboard and cost becomes a non-functional requirement like latency or security; it fails when it is finance policing engineering.

See: FinOps for AI

One of the most capable general-purpose AI models available at a given moment — trained on trillions of tokens at a cost north of $100M. A frontier model earns its price when the cost of a wrong answer exceeds the cost of the call: broad reasoning, novel situations, work that resists reduction to a well-defined pattern. Using one to classify sentiment is couriering a Post-it note.

See: Frontier Models

Two ways to control risk in a delivery path. Gates stop everyone to catch the few who would drive off the cliff — a human approval step in every journey. Guardrails let everyone travel at speed and intervene only when someone is about to leave the road — policy as a pipeline check. Moving controls from gates to guardrails is most of what separates a platform that accelerates teams from one that queues them.

See: Platform Operating Model

An agent-supervision pattern where the agent pauses and waits for approval before a defined action — sending the email, making the payment, deleting the data. Its lighter-weight sibling, human-on-the-loop, lets the agent act autonomously while a human supervises the output and can intervene. The choice between them is set by whether a failure can be safely reversed.

See: Agentic AI Patterns

An architecture that routes the bulk of requests to a cheap self-hosted small model and reserves a frontier API for the minority that need advanced reasoning. Because most workloads are long-tailed, hybrid routing captures the majority of self-hosting savings while preserving quality on the hard cases — commonly cutting spend by 80% or more versus an all-frontier approach.

See: AI Cost Curves

Running a trained model to produce an output (a completion, classification, or embedding) in response to a request. Inference is the recurring, per-request cost of operating AI in production — as opposed to training, which is a one-off up-front cost — and is the cost that AI cost curves are drawn against.

The routing layer of a distributed inference deployment. For each request it weighs the latency budget, the capability the task needs, and any jurisdiction or sovereignty constraint, then sends the request to the cheapest node that satisfies all three. The orchestrator is what turns a set of scattered GPU sites into an architecture — without it, distribution is just fragmentation.

See: AI Grid

The compounding cost of stitching vendors together: every product arrives with its own auth model, data format, error conventions, and upgrade cycle, and engineers become glue-code specialists translating between them. When one system changes its API, the blast radius is every dependency on it — a cost that rarely appears in the purchase decision and always appears in the run cost.

See: Build vs Buy

The self-service layer a central platform team provides so dozens of autonomous teams can provision environments, ship services, and operate safely without queueing on tickets. Run well, it is a product with a roadmap, adoption metrics, and someone accountable for whether developers succeed — not a shared-services queue with a newer name.

See: Platform Operating Model

Training a small student model to imitate a large teacher, learning from the teacher's output distributions rather than raw labels — so it absorbs the teacher's confidence structure, not just its answers. A well-executed student can be five to ten times smaller while retaining most of the capability that matters for its task.

See: Model Compression

The per-conversation memory an inference server keeps so it does not recompute attention over the whole context on every token. It scales with context length rather than parameter count, so it dominates VRAM once contexts run long — a model that fits comfortably at 4K tokens can fall off a hardware tier at 32K, which is why long-context serving needs cache-aware compression, not just smaller weights.

See: Model Compression

The maximum end-to-end delay an interaction can tolerate before its user experience or business value degrades — a frame time in gaming, a conversational beat for a customer agent, milliseconds for a trading decision. In a distributed inference architecture the latency budget is the primary routing constraint: it determines which tiers and regions are even eligible to serve a request.

See: AI Grid

The family of techniques that shrink language models for cheaper serving — quantisation, pruning, distillation, and low-rank factorisation. The terms are often used interchangeably; they are not the same thing. The techniques compose rather than compete, and the composition is what makes self-hosted and edge inference economic.

See: Model Compression

An open standard for connecting AI applications to tools and data sources — USB-C for AI. An application implements the protocol once and can reach any tool that also speaks it, turning the N × M integration tax (every app × every tool, each connector bespoke) into N + M. Launched by Anthropic in November 2024, adopted across the major labs, and donated to the Linux Foundation in December 2025.

See: MCP

Sending each request to the cheapest model that can handle it, rather than picking one model for everything — simple classification to a small model at a fraction of a penny, complex reasoning to a frontier model at many times the price. Routing 80–90% of traffic to small, cheap tiers attacks the largest single line item in most AI estates, which makes it a budget control rather than an architecture preference.

See: Frontier Models

The supported, opinionated way to do a common thing — spin up an environment, ship a service, wire up logging — documented, automated, and patched for you. Also called a golden path. It is not the only road, but it should be the path of least resistance: paved roads beat mandates because they win on convenience rather than compliance.

See: Platform Operating Model

The usage-based billing model of hosted inference APIs, where you pay a fixed amount per thousand input and output tokens with no upfront commitment. It has zero fixed cost and a constant marginal cost, which makes it unbeatable at low volume and progressively expensive as request volume scales.

Improving a model's output by improving the brief it is given — system prompts, a few worked examples, asking the model to show its reasoning. It is free, gives instant feedback, and resolves the large majority of enterprise use cases on its own, which makes it the cheapest lever to pull before reaching for retrieval, fine-tuning, or heavier infrastructure.

See: AI Fundamentals

Removing weights, neurons, attention heads, or whole layers from a model and stitching the rest back together — the technique to reach for when quantisation alone leaves you 20–30% over your VRAM budget. Quality holds to roughly 50% sparsity and then drops sharply without follow-up fine-tuning.

See: Model Compression

Reducing the numerical precision of a model’s weights (for example from 16-bit to 4-bit) so it needs less memory and runs faster, usually with only a small quality loss. Quantisation is what makes self-hosted and edge inference economic: a quantised small model can run on a single GPU that a full-precision model would not fit.

See: Model Compression

Giving a model access to your data at query time instead of training it in — relevant documents are retrieved and supplied alongside the prompt. Because the data stays out of the model weights, it is easy to update, audit, and delete: drop a new document in and the knowledge is current immediately, with no retraining cost. For giving a model knowledge of a business, RAG is the right starting point in almost every case.

See: AI Fundamentals

Serving a previously paid-for answer when a new query means the same thing, judged by embedding similarity rather than exact text match. The honest number is a spread: savings can reach around 70% on repetitive support and FAQ traffic, but land closer to 10% on genuinely novel queries — it works where traffic repeats.

See: FinOps for AI

Reporting each team's share of infrastructure cost without moving any money — visibility that changes awareness. It is the rung before chargeback on the maturity ladder, and the right place to start: teams first need to believe the numbers before the numbers can carry consequences.

See: FinOps Maturity

A compact language model — typically single-digit billions of parameters — that runs on a single GPU or modest hardware, especially once quantised. SLMs are the workhorse of the bulk tier: routed the high-volume, well-defined traffic in a hybrid architecture at near-zero marginal cost, while frontier models are reserved for the minority of requests that need them.

See: AI Fundamentals

The model for organising software teams published by Matthew Skelton and Manuel Pais (IT Revolution, 2019): four fundamental team types — stream-aligned, platform, enabling, and complicated-subsystem — and three interaction modes between them — collaboration, X-as-a-Service, and facilitating. The underlying claim is that team boundaries should be drawn to keep each team's cognitive load within its capacity.

See: Team Topologies

Ward Cunningham's metaphor (1992) for shipping something you know is not quite right in order to learn faster: the borrowing is deliberate, the interest is the extra effort every subsequent change costs while the loan stands, and the principal is the one-off cost of putting it right. He was explicit that it never meant writing bad code and calling it a loan — and debt only charges interest when somebody touches the code, which is why stable, ugly modules are often not debt at all.

See: Technical Debt

Martin Fowler's 2009 classification of debt on two axes: deliberate or inadvertent, prudent or reckless. Deliberate-prudent is a real decision with understood consequences; deliberate-reckless is skipping design under schedule pressure; inadvertent-prudent is debt only visible once you understand the domain, and largely unavoidable; inadvertent-reckless is not knowing the difference.

See: Technical Debt

The elapsed time from a new engineer starting to their code running in production — the internal platform's first impression, and the metric that sets the tone for every later interaction. A multi-day chain of tickets and approvals means every subsequent journey starts from distrust.

See: Developers as Customers

The full cost of an option over its life, not its sticker price — for bought software, everything beyond the licence fee; for built software, hiring, onboarding, infrastructure, and maintenance; for infrastructure moves, compute, storage, egress, facilities, staff, lost elasticity, and one-time migration. Honest comparisons model it over three to five years, because the hidden layers are where build and buy actually diverge.

See: Build vs Buy

Expressing infrastructure cost per unit of business value — cost per request, per customer, per transaction — rather than as a monthly total. It is the top rung of the cost-maturity ladder because it turns "spend went up" into a question with an answer: did cost per unit go up, or did the business grow?

See: FinOps Maturity