AI Governance Operating Models: Who Decides, Who Enforces, Who Gets Paged

An organisation can have an AI policy, a risk committee and a model register, and still be unable to answer who approves a new agent with access to customer data and the ability to send external emails. Legal assumes product owns it. Product assumes security signed it off. Security assumes the model team assessed it. The system goes live because everyone has a plausible reason to think someone else is responsible.

That is an operating-model failure, rather than a policy failure.

AI governance breaks when organisations organise it around documents rather than decisions. A policy can say “use AI responsibly”, but it cannot decide whether a particular agent may act autonomously, which controls it needs or who is accountable when its behaviour changes in production.

I start with a harder question: which decisions must be made, when, by whom, on what evidence, how are they enforced — and what does the organisation run every day to keep the answers true? The first four make a governance model. The last two decide whether it survives production.

One workflow runs through everything below, because these tables read as abstractions until something specific goes through them. It is invented for this piece rather than taken from any organisation — a constructed workflow can be built to touch every row of every table, which is exactly what is wanted here. Take an internal agent that answers “has supplier invoice 4471 been paid, and if not, why not” for an accounts-payable team. It retrieves from the ERP and replies in the team’s chat channel. Deliberately unremarkable: the interesting part is not how it is built, but what happens six weeks later when somebody asks whether it could reply to the supplier directly.

Separate decision rights from delivery work

I separate governance from delivery at decision rights. A governance team should not become a second engineering department, and delivery teams should not define their own acceptable risk.

This is a map from decision to evidence and enforcement, rather than a committee chart. Without the last column, governance becomes a recommendation the runtime can ignore.

Run the invoice agent down the first column and it is clear which decisions are settled and which only look settled. Permitted use: yes, the AP lead sponsors it. Acceptable data use: less obvious than it looks, because supplier invoices carry bank details and, for sole traders, personal contact data. Proportionate architecture: retrieval only, no writes, modest. Autonomy: none. Can it remain in production: nobody has been named. That last row is blank on most systems at launch, and it is the row the rest of this piece is about.

Govern the lifecycle, not just the launch

Most governance attention arrives before release, when the system is still a diagram. The larger risk often appears afterwards: a new data source changes retrieval, a model version changes output behaviour, a tool gains a permission, or a product team broadens the workflow without revisiting the original risk assessment.

I use lifecycle gates to make the obligation recurring.

The change gate is usually the missing one. Treat a material permission, tool or data change as a new risk decision even if the product name did not change.

The invoice agent shows why it is the one that goes missing. It clears every gate to release, runs quietly for six weeks, and then somebody proposes letting it reply to the supplier by email rather than posting in the team channel. Almost nothing about the model, the retrieval or the code changes. The action class changes completely: from answering a colleague who can sanity-check it, to making an external statement about a payment on the company’s behalf, to a recipient with no idea they are reading a generated answer. That is a new risk decision on an unchanged product name, and if the only gate is at launch, nobody takes it.

The RACI that prevents governance theatre

I would not copy this table blindly. Its job is to expose the role nobody assigned. If an agent has no business sponsor, it lacks an owner for the value it is supposed to create. If it has no operations owner, it lacks an owner for the risk it creates at 03:00.

Use proportional governance

I would not put a knowledge assistant that summarises approved internal documents through the same process as an agent that changes a customer record. Risk turns on data, autonomy, consequence, scale and reversibility.

Proportional does not mean optional. It means matching the control burden to the consequence.

The invoice agent starts moderate. The email change moves it to high, and that is a different control set rather than a stricter version of the same one — human approval on the outbound action, monitoring of what was actually sent rather than whether the service was up, and a change gate passed before the first email goes out rather than after the first complaint. The tier is a property of the deployed system and its current use, which is why it gets re-asked rather than stamped once.

Governed AI delivery: a gate is a code path or it is a meeting

Every row of the first table ends at an enforcement point, and that column is where most operating models quietly fail. A gate named in a policy is a gate someone has to remember. A gate in the delivery pipeline is a gate that has to be actively removed.

The distinction is not stylistic. A gate implemented as a meeting is schedulable, so it slips into a quarter when nobody is looking; unevidenced, so the only artefact is minutes nobody will accept as an audit trail eighteen months later; and linear in delivery volume, so the busier the organisation gets the more certainly it becomes the bottleneck and the more reliably it is routed around. A control that is routed around is not slow governance. It is no governance, with reporting attached.

The useful question about each gate is therefore physical: which system executes it, and what happens to the change when it fails.

Three of those six are checks and automate outright. Three are decisions and do not: whether a use case is permitted, whether an architecture is proportionate, and whether a service may stay in production are judgements about acceptable risk, and a judgement does not delegate to CI. What the pipeline does for those three is make the decision unavoidable and its evidence automatic. That is a smaller claim than “automate governance”, and it is the one that survives a real delivery organisation.

Make the governed properties a file

The hardest gate to implement is the change gate, because enforcing “reassess on material change” needs a machine-readable answer to what counts as material. Put every governed property of the service into one declared manifest in its own repository — model identifier and version, tool and connector list, data sources and retrieval scope, action classes with their approval policy, proportionality tier, named owners, and active exceptions with expiry dates. A material change then stops being a judgement. It is a diff on that file.

The rest comes from tooling every delivery organisation already runs. Code-ownership rules on the path pull in the risk owner, so broadening an agent’s tool list cannot merge on an engineer’s approval alone. The runtime reads the same manifest to configure the policy gateway, so the document and the enforcement cannot drift — there is only one of them. The review pack is generated from it. And an exception is a dated entry in it, which turns “every exception expires” from an aspiration into enforcement on a known date. That needs one detail right: CI runs on commits, not on calendars, so a build check alone lets an exception outlive its expiry on any service nobody happens to push to. The gateway checks the expiry on every action it governs, so a lapsed exception stops applying on the day it lapses, and a scheduled job flags the ones about to. A register in a spreadsheet decays silently, because nothing is ever paged by a spreadsheet, whereas a register the runtime reads fails loudly and is cheap to renew if the exception still stands.

For the invoice agent the diff that matters is one line: an action class moving from replying in a channel to sending external email. Nobody has to remember that this is material. The gateway refuses the outbound call until the policy granting it is merged, and merging it requires the risk owner. Evidence assembled at audit time is reconstruction, which is what an auditor is trained to distrust; a versioned file with a review history is the record, produced as a by-product of the work rather than as a project after it.

Exceptions are part of the model

A policy without an exception route creates shadow delivery. Teams with a legitimate urgent need will work around it. I want the exception path visible: what is requested, who accepts the residual risk, how long it lasts, which compensating controls apply and when it expires.

Every exception should expire. A permanent exception is an undeclared policy change.

The invoice agent’s realistic exception is not an exotic architectural one. It is the request to switch outbound approval off during month-end close, when the queue is long and approvals are visibly slowing everyone down. That deserves a real answer: a bounded exception, a named risk acceptor, an expiry at the end of the close window, a compensating control that is not the same control weakened — a sampled review of what was sent, rather than approval of each item — and an entry in the manifest, so outbound approval switches back on at the expiry whether or not anyone remembers to close it. The unbounded version of that request is how a high-risk workflow ends up running permanently on a moderate-risk control set.

AI operational governance: what you actually run

Almost everything written about AI governance stops at the release gate, which is roughly where the interesting part starts. A model in production is a system whose behaviour changes without a deployment: the retrieval corpus shifts, the provider updates a model behind a stable endpoint, users find a workflow nobody designed, and traffic drifts away from the distribution the evaluation set was built on. None of that arrives as a change request.

Operational governance is the set of loops that keep the decisions in the first table true after go-live. There are three, they run at different speeds, and conflating them is the commonest failure I see.

That shape is not mine. The NIST AI Risk Management Framework (opens in a new tab) packs most of it into one subcategory, MANAGE 4.1: “post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management”. ISO/IEC 42001 (opens in a new tab) says it through structure — internal audit, management review, corrective action — which is legible from its contents page. Both describe something you run, not something you pass.

The runtime loop is the only one that scales with usage, and the only one that should. Put a human in the per-action path for anything short of a consequential action and volume defeats them: they approve on autopilot, which yields the worst outcome available — an approval record carrying a signature and no scrutiny.

The exception loop is the one most organisations never staff. It is a triage queue fed by signals the platform already emits: step-limit and budget breaches, repeated tool failures, evaluation failures, permission denials, and spikes in human rejection of proposed actions. Its job is daily classification, not investigation. A defect goes to the delivery backlog, a policy gap to the manifest, and normal variation nowhere — but the threshold that produced it gets adjusted so it stops arriving.

What a governance review actually reads

A standing review that opens with a slide deck is a reporting exercise, and it will be skipped the first quarter somebody is busy. Generate the pack and fix the agenda, so the meeting spends its time on decisions rather than on establishing facts. Six inputs, all of which the controls above already produce:

• The manifest diff since the last review — what changed, who approved it, whether the tier still fits.
• The exception register with expiry dates. A twice-renewed exception is a policy question wearing a temporary label.
• Evaluation results over time, not a single pass rate. A stable 94% failing a different category each month is not stable.
• Incidents and interventions. The override count is the more sensitive of the two, and a sharp fall usually means the reviewers stopped reviewing rather than that quality improved.
• Spend against budget, by workflow. Unexplained cost growth is often the first visible symptom of a behavioural change nobody has noticed.
• Denials and refusals from the gateway. A rise means either the workflow has outgrown its permissions or something is attempting an action it was never designed to take — both worth knowing, and they need opposite responses.

The output is one of the four outcomes on the Operate row of the lifecycle table. A review whose only available answer is “continue” is not a gate, and the outcome omitted in practice is retire. Registers of AI systems only ever grow, every entry takes a permanent share of a finite review budget, and a model nobody uses steadily consumes attention a model people depend on is not getting. NIST puts decommissioning in its GOVERN function rather than treating it as an operational afterthought (GOVERN 1.7), which is the right altitude for it. Escalate out of the review on four triggers: an incident that reached someone outside the organisation, an action taken outside its declared class, a data boundary crossed, or a second renewal of the same exception. Everything else is a service-owner decision, and routing more than that upward teaches people to stop routing anything upward.

What the on-call engineer gets at 03:00

This question exposes whether an operating model is real, because at 03:00 there is no committee, no risk owner and no time.

The governance model’s job at 03:00 is to have already made the decision. The engineer who is paged must not be weighing residual risk against business impact for a system they did not build — that decision has an owner, and the owner is asleep. What they need is a pre-authorised action and explicit permission to take it, in a runbook readable in under two minutes, because a longer one will not be read at 03:00.

1. What this system is allowed to do — the action classes, one line each, so the engineer can judge whether the observed behaviour sits inside them.
2. The kill condition and the kill switch — the exact command that stops it, and an explicit statement that on-call may use it without calling anyone.
3. The fallback — what happens to the work while the system is off. If there is no answer, the system is a single point of failure that was approved as an efficiency.
4. The trace query — one query returning what the system actually did, correlated across request, retrieved context, model call, tool calls and policy decisions. Assembled during an incident, this is guesswork.
5. Who owns the decision to turn it back on, by name and rota.

The asymmetry between points two and five is the whole design. Stopping is cheap, reversible and pre-authorised. Restarting is a decision with an owner, taken in daylight, with the trace in front of them. For the invoice agent the 03:00 failure is mundane: it starts quoting payment dates a week stale because a scheduled ERP export failed. The engineer needs no view on whether that is an acceptable risk. They need to know they may stop it, how, and that the questions go back to the shared inbox they came from meanwhile — slower, and entirely survivable. Written down in advance, that sentence is what makes suspension a two-minute action rather than a thirty-minute negotiation with somebody’s voicemail.

One anti-pattern is worth naming because it is so common: paging on quality. Degraded output is a ticket, not a page. Pages are for actions outside the declared action class, breached budgets and kill-switch conditions — events where stopping is right and delay makes things worse. Paging a human for a falling evaluation score teaches them to acknowledge alerts without reading them, and what that costs you is the page that mattered.

The cadence has an external floor

Choosing a review cadence feels like taste until you notice regulation has already set a floor under it. Under the EU AI Act (opens in a new tab), a serious incident involving a high-risk system must be reported no later than 15 days after the provider or deployer becomes aware of it, 10 days where a person has died, and two days for a widespread infringement or a serious disruption to critical infrastructure (Article 73). Deployers must keep automatically generated logs, to the extent under their control, for a period appropriate to the system’s intended purpose and of at least six months, unless applicable Union or national law — particularly Union law on personal data protection — provides otherwise (Article 26(6)). And the system must be designed so an overseer can “interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state” (Article 14(4)(e)).

Read as operating requirements rather than compliance line items, those describe the machine. A 15-day outer limit is only survivable if detection is materially faster than 15 days — a quarterly review detects nothing, it reviews what detection already found, which is the argument for a daily exception loop without needing to believe in governance at all. “Becomes aware” starts the clock, which gives not-looking a short shelf life as a strategy. Six months of logs is a statement about what the trace must still answer half a year later, so it constrains the runbook’s trace query rather than a storage policy. The stop button is the kill switch, written as a design obligation.

Whether any of it binds a given system turns on classification and on a timetable already amended once — Regulation (EU) 2026/1744 moved the high-risk dates back — so check the current Article 113, not any summary, this one included. That amendment is itself the argument for building the cadence rather than the deadline: a detection loop needs no rework when a date moves, and a compliance project starts again.

Test the model against one live workflow

Use the AI governance operating-model worksheet for one live or proposed workflow. Name the business sponsor, data owner, engineering owner, action class and ongoing review cadence. If any role is blank, the workflow is not governed yet.

Then read AI Governance Is Enforcement, Not Intent for why a control has to bite somewhere. Each enforcement point in the first table has a piece behind it: the runtime guards and the exception queue in Agentic AI Control Planes, and the approval policy and trust boundary for tool access in MCP Security and Governance.

I do not measure governance by the number of approvals. I measure it by whether the decision to move quickly is explicit, bounded and reversible where possible. That is how an organisation accelerates without letting an incident discover who owned the decision.