Your Internal Developer Platform Will Not Survive AI Unchanged

A real scenario: standardised deployments, unknown AI behaviour

The platform team has done the hard work. Kubernetes clusters are standardised, golden paths exist, and CI/CD pipelines push application services reliably to production. Developers use self‑service templates. Costs are visible per service. On‑call is predictable.

Then the first AI agent hits production.

Support asks which foundation model the agent used last night when it drafted customer emails. No answer. Product asks what context the agent saw from the knowledge base. Unknown. Security asks which tools the agent was authorised to call, and whether it called a payment API. The logs show an opaque chain‑of‑thought redacted by the SDK and a token count, but no tool call inventory. Finance asks why monthly spend spiked. The team can see GPU hours and an API line item, yet cannot attribute it to the agent’s versions or a specific customer segment. Quality has regressed, but nobody can reproduce a run with the exact prompt, context window, model version and tool responses to evaluate the change.

The platform has done its job for applications. It is not yet built for AI. That is an addressable gap.

Evidence from platform teams shows the job is expanding from CPU‑centric application delivery to AI‑native responsibilities: accelerators, inference runtimes, data movement and more opinionated self‑service for developers adopting AI workloads. The direction is towards modular capabilities inside the existing platform rather than a separate AI silo (PlatformEngineering.com (opens in a new tab)). Current coverage of AI platform scope reinforces this: model access, agent and tool governance, cost governance, safety guardrails and developer self‑service belong in a coherent platform experience (TrueFoundry (opens in a new tab)).

The argument here is simple. Your internal developer platform remains the right foundation. AI will expand its responsibilities: model access, data context, agent runtime, evaluation, serving, observability, cost attribution and policy enforcement. The better choice is a modular extension of the current platform, not a separate AI empire.

This piece lays out what changes, the seven‑service catalogue you need, the operating model options, and a 90‑day path to get moving. It assumes you already think in platform product terms. If you do not, start there: see Platform Operating Model and Developers as Customers.

What changes for AI versus application platforms

Application platforms focus on code artefacts, container runtimes, service discovery, secrets, and traffic management. Observability captures process metrics, request traces and logs. FinOps ties cloud resources to services. Pipelines manage build, test, deploy. The unit of control is the service.

AI workloads add and shift responsibilities:

• Compute and runtimes. From general CPUs to accelerators and inference runtimes. Scheduling, right‑sizing and placement strategies differ (PlatformEngineering.com (opens in a new tab)).

• Models as dependencies. Models, whether external or hosted, become first‑class dependencies with versions, capabilities and operational constraints. “Model access” replaces “library import”.

• Context as data product. Prompt inputs are rarely enough. Agents need retrieval from knowledge bases, feature stores or lakes. Data movement and freshness matter (PlatformEngineering.com (opens in a new tab)).

• Agents and tools. Agents orchestrate calls to tools (APIs, functions, workflows). Tool governance, registration and capability exposure are platform responsibilities (TrueFoundry (opens in a new tab)).

• Evaluation. Quality must be tested beyond unit tests: prompts, datasets, human‑in‑the‑loop checks, safety probes and regression suites (TrueFoundry (opens in a new tab)).

• Observability shifts left. Token usage, model choice, prompt/response metadata, tool call traces and safety events become first‑class signals (TrueFoundry (opens in a new tab)).

• Policy and guardrails. Safety, privacy, data residency, tool entitlements and rate governance must be enforced across models and agents (TrueFoundry (opens in a new tab)).

• Cost attribution. AI costs accrue per token, per call and per accelerator minute more than per container hour. Attribution needs to tie back to agent, model, dataset and customer cohort. FinOps practices extend accordingly (TrueFoundry (opens in a new tab)). See also FinOps for AI and the practical notes in FinOps for AI.

The design implication is direct. Keep your core IDP assumptions—self‑service golden paths, paved roads, sensible defaults—and extend them to these AI‑specific units of control. Add a catalogue of services that give teams opinionated, safer, observable ways to build with AI. That is the modular path described below.

The seven‑service catalogue your platform will need

The platform product should add seven coherent services. Each can be adopted independently, but they compound in value together. Each has a clear developer API, a back‑office control plane, and observable outputs that feed governance and FinOps.

• Model access

• Data context

• Agent runtime

• Evaluation

• Serving

• Observability

• Policy

These are not complete reinventions. They are extensions that align with the existing platform operating model. For a structured view of platform scope, see AI Platform Operating Model and the grid of lifecycle controls in AI Grid.

1) Model access

What it is. A brokered, policy‑aware interface for developers to call foundation and fine‑tuned models. It abstracts provider APIs, hosts internal models where needed, and normalises request/response metadata.

Why it matters. It centralises entitlements, isolates credentials, exposes model capabilities and versions, and feeds attribution. It is also where you shape developer experience: SDKs, Terraform modules, CLI and templates.

Key capabilities.

• Model catalogue with versions and metadata.

• Routing and fallbacks across providers and internal runtimes.

• Cost tags on every call.

• Quotas and rate plans per team, environment and use case (TrueFoundry (opens in a new tab)).

• Accelerator scheduling where hosting internally (PlatformEngineering.com (opens in a new tab)).

2) Data context

What it is. A service that defines and delivers the context an AI workload can see: retrieval pipelines, embeddings, prompts, feature stores and caching. It supplies the grounding data for queries and agents.

Why it matters. Relevance, freshness and security determine quality. Without a standard way to declare and deliver context, each team invents its own pipelines, and you lose reproducibility and control.

Key capabilities.

• Context source registry and schemas.

• Retrieval patterns (RAG) with filters for tenancy and sensitivity.

• Data freshness SLAs and lineage hooks.

• Prompt templates and versioning.

• Access controls enforced at retrieval time (PlatformEngineering.com (opens in a new tab)).

3) Agent runtime

What it is. An opinionated way to run agents that call tools. It hosts orchestrators, registers tools, and enforces contracts and timeouts. It is not an app server; it is closer to a workflow engine tuned for AI agents.

Why it matters. Tool sprawl and unmanaged capabilities are risk multipliers. A runtime with a registry lets the platform enforce who can call what, record tool calls, and simulate flows.

Key capabilities.

• Tool registration with scopes, schemas and approval.

• Sandboxed tool execution contexts.

• Step‑level tracing and reproduction via captured state.

• Safety checks before and after tool calls (TrueFoundry (opens in a new tab)).

4) Evaluation

What it is. A testing service for prompts, models, datasets and agents. It runs regression suites, safety probes and A/Bs; aggregates human feedback; and reports quality trends.

Why it matters. Without evaluation, you ship unknown behaviour. With it, you can isolate regressions to model versions, prompt changes or context drift.

Key capabilities.

• Dataset curation and versioning.

• Offline and online evaluation runners.

• Safety and policy test packs.

• Baselines and acceptance gates for CI/CD (TrueFoundry (opens in a new tab)).

5) Serving

What it is. A layer that exposes models and agents to applications and channels. It provides APIs, streaming endpoints, traffic shaping and canary controls.

Why it matters. You need a stable interface for clients while you move behind the scenes—model upgrades, prompt tweaks, new tools. Serving is the change‑management layer for AI endpoints.

Key capabilities.

• Endpoint lifecycle and versioning.

• Canary, shadow and rollback.

• Multi‑tenant request shaping.

• Backpressure, retries and circuit breakers (PlatformEngineering.com (opens in a new tab)).

6) Observability

What it is. End‑to‑end visibility of prompts, context, model choice, tokens, tool calls, latencies, safety events and costs. It extends traces from application requests into AI steps.

Why it matters. You cannot debug behaviour or govern spend without rich traces. Observability also feeds evaluation and FinOps.

Key capabilities.

• Prompt/response/event capture with PII controls.

• Token accounting and per‑unit cost.

• Tool call inventories.

• Dashboards and alerts for quality and spend (TrueFoundry (opens in a new tab)).

7) Policy

What it is. Guardrails and governance across the stack: model approvals, tool entitlements, data residency, rate limits, jailbreak defences and audit.

Why it matters. AI raises exposure. Central policy as code gives you a single place to express and enforce constraints while keeping developer flow usable.

Key capabilities.

• Policy packs and exceptions workflow.

• Pre‑request safety checks and post‑response filters.

• Data access boundaries in retrieval.

• Audit trails tied to identities, models and tools (TrueFoundry (opens in a new tab)).

For each service, treat the developer as customer. Provide paved paths, default templates and a support channel. The product work mirrors your existing platform operating model.

Operating model choices: silo, enablement, or platform extension

There are three common responses to AI demand. Only one scales with control and developer experience.

• AI silo: build a separate AI platform and team, with its own backlogs and tooling.

• Enablement overlay: keep the platform unchanged; add an AI coaching team and a few templates.

• Platform extension: expand the internal developer platform with modular AI services and golden paths.

Industry analysis argues against separate AI silos and towards modular, AI‑native capabilities inside the enterprise platform (PlatformEngineering.com (opens in a new tab)). Current AI platform scope also points to integrated functions—model access, tool governance, cost governance, guardrails and self‑service—that are better delivered as part of the platform experience (TrueFoundry (opens in a new tab)).

Comparison of operating models

The platform extension is the pragmatic centre. It keeps delivery paths familiar, adds AI‑specific services, and avoids a split brain. It lines up with the enterprise OS direction described for platform engineering’s next stage (PlatformEngineering.com (opens in a new tab)).

Mapping services to outcomes and controls

Your seven services exist to achieve outcomes and enforce controls. Make this traceable. The table below gives a starting map. Adapt it to your risk profile and business context. Use it to explain trade‑offs to stakeholders and to design dashboards that show value.

For cost governance specifics, align this mapping with your FinOps for AI approach and educate service owners using practical guidance in FinOps for AI.

Ownership, team topology and product thinking

Do not bolt these services onto “whoever is free”. Treat them as platform products with customers, backlogs and SLAs. The operating model is familiar if you already run a modern platform.

• Product ownership. Assign a product manager for AI platform services. Their customers are internal developers. Their outcomes are adoption, quality and governance signals, not vanity features. Reference Developers as Customers.

• Service ownership. Each of the seven services has a clear owner, on‑call rota and runbooks. They expose roadmaps and APIs. They integrate with platform identity and billing.

• Team topology. Keep the AI platform engineers inside the platform organisation. Embed them with application teams for discovery and feedback but maintain a single platform backlog. This avoids the silo dynamic and supports reuse (PlatformEngineering.com (opens in a new tab)).

• Golden paths. Extend existing templates to include AI use cases: promptable microservice, RAG service, batch evaluation job, agent with tool registration. Keep defaults sensible and documented.

• Governance by design. Policies live in code within the services, not in Confluence. Exceptions use the same workflow across services. Signals flow to a central observability layer.

• FinOps embedded. Every service emits cost and usage tags. Budgeting and showback are first‑class. Extend your cost allocation model to tokens, model calls and accelerator minutes (TrueFoundry (opens in a new tab)). See FinOps for AI.

• Developer feedback loops. Instrument adoption and friction. Hold regular clinics. Publish updates on changes to defaults, approved models and tool registration.

This is platform product work. It requires the same discipline you used to create your IDP. The payoff is that AI does not fracture your operating model.

A 90‑day sequence to extend your platform

You do not need a year‑long programme to start. Sequence pragmatic steps in three 30‑day sprints. Keep scope tight, deliver visible value, and set up the control points you will need later. Use the AI Platform Operating Model and the AI Grid to plan scope and controls across the lifecycle.

Days 0–30: establish foundations and reduce risk

Objectives

• Stop uncontrolled model access.

• Capture observability you will need for governance.

• Prove one golden path end to end.

Actions

1. Model access minimal viable service.

• Stand up a brokered interface for two approved models. Hide provider keys. Expose a single SDK and Terraform module.

• Implement entitlements by team and environment. Add quotas.

• Tag every call with service, environment, team and use case. Record model version.

2. Observability spine.

• Extend tracing to include prompt/response metadata, token counts, model choice and tool call events. Redact PII at source.

• Build a basic dashboard: requests by model, tokens by service, cost per team (even if rough). Alert on quota breaches and anomalous token spikes (TrueFoundry (opens in a new tab)).

3. Golden path: promptable service template.

• Ship a template that uses the model access SDK, emits observability, and includes basic error handling and retries. Add CI scaffolding.

4. Policy starter pack.

• Define an initial policy set: approved models, maximum context window, tool call timeouts, and data residency defaults. Enforce via the model access layer and service templates (TrueFoundry (opens in a new tab)).

5. FinOps hooks.

• Establish a cost ledger for model calls and, if applicable, accelerator time. Link tags to your cost allocation model. Start weekly showback to teams. See FinOps for AI.

Deliverables

• Model access service v0.1.

• Observability dashboards.

• Promptable service golden path.

• Policy pack v0.1 and exception workflow.

• Basic AI showback report.

Days 31–60: add context, agent control and evaluation

Objectives

• Make AI quality reproducible and testable.

• Prevent tool sprawl.

• Deliver one agent use case end to end.

Actions

1. Data context v0.1.

• Inventory top context sources for target use cases. Register two sources with schemas.

• Implement a retrieval pattern (RAG) with tenancy filters and freshness SLAs.

• Add prompt template versioning. Capture template IDs in traces.

2. Agent runtime v0.1.

• Stand up an agent orchestrator with tool registration. Define tool schemas, scopes and approval workflow.

• Implement sandboxed tool execution. Enforce timeouts and capture step‑level traces and tool outputs.

• Deliver a golden path for “agent with two tools”.

3. Evaluation v0.1.

• Curate a small evaluation dataset for the target use case.

• Implement offline evaluation runs in CI with baselines and acceptance thresholds.

• Add a safety test pack and run it pre‑deploy (TrueFoundry (opens in a new tab)).

4. Serving v0.1.

• Expose stable endpoints for the promptable service and the agent. Add versioning and a simple canary.

5. Policy expansion.

• Add policies for tool entitlements, data residency in retrieval, and safety checks pre‑ and post‑tool calls.

Deliverables

• Data context service v0.1 and two registered sources.

• Agent runtime v0.1 with tool registry.

• Evaluation runners with first datasets and baselines.

• Serving endpoints with canary support.

• Updated dashboards showing tool calls, template versions, evaluation results.

Days 61–90: scale adoption and tighten governance

Objectives

• Onboard more teams without loss of control.

• Improve cost accuracy and rate governance.

• Harden quality and rollback.

Actions

1. Model access v0.2.

• Add routing and fallbacks across two providers or internal/hosted models.

• Introduce per‑use‑case rate plans and burst controls.

• Publish a catalogue page with approved models and recommended use cases (TrueFoundry (opens in a new tab)).

2. Data context v0.2.

• Introduce lineage tracking and context freshness dashboards.

• Add caching with bounded staleness controls.

3. Evaluation v0.2.

• Add online evaluation probes and human feedback capture for low‑confidence responses.

• Gate production deploys on offline baselines and staged canaries.

4. Observability v0.2.

• Expand dashboards to quality and spend per model, per agent, per customer cohort.

• Add alerts on quality regression and spend anomalies.

5. FinOps maturity.

• Move from showback to chargeback or budget enforcement where appropriate.

• Publish monthly “cost and quality” reviews to service owners with agreed actions. Align with FinOps for AI.

6. Policy hardening.

• Implement exceptions workflow with time‑bounded approvals.

• Add audit reports linking identities, models, tools and policy outcomes.

Deliverables

• Updated golden paths for promptable service and agent.

• Adoption playbook for new teams.

• Governance dashboards and monthly review cadence.

This sequence gives you a functioning platform extension in 90 days. It also gives you the minimum controls to answer the questions from the opening scenario.

Design notes: keep it modular, keep it opinionated

A few principles help the extension succeed.

• Modular services, not a monolith. Each of the seven services can ship and evolve independently. Integration points are clear: identities, tags, traces, policy hooks.

• Opinionated defaults with escape hatches. Give developers a fast path that is safe by default. Provide documented ways to extend or opt out with approvals.

• One identity, one bill, one trace. Unify developer login, cost allocation and tracing across application and AI layers. This keeps incident response and FinOps coherent.

• Integrate with the platform backlog. Treat AI service requests like any platform feature. Publish roadmaps and changelogs. Avoid side channels.

• Testability as a first‑class constraint. If a capability cannot be evaluated and observed, it is not ready for general use.

• Minimise bespoke SDKs. Where you provide SDKs, focus on thin wrappers that emit the right signals and enforce policy. Reduce lock‑in, increase portability.

These principles mirror the “enterprise OS” view of platform engineering’s next stage: modular capabilities, integrated experience and opinionated self‑service (PlatformEngineering.com (opens in a new tab)).

How this changes developer experience

Developers should see continuity, not upheaval.

• The same portal and catalogue, now with AI golden paths.

• The same IaC workflow, now with modules for model access, data context and serving.

• The same CI/CD gates, now with evaluation steps.

• The same tracing dashboards, now with prompt and tool events.

• The same budgeting conversations, now with tokens and model calls.

The new elements are discoverable, documented and supported. The platform team keeps the mental load manageable by curating defaults and giving clear examples.

What you deprecate and what you standardise

Platform work is as much subtraction as addition.

Deprecate

• Direct use of provider keys in application code.

• Ad‑hoc tool execution without registration.

• Unversioned prompt strings embedded in services.

• Opaque SDKs that do not emit traces or cost tags.

Standardise

• A single way to call approved models.

• A single way to register tools and declare scopes.

• A single retrieval pattern with tenancy and residency enforcement.

• A single evaluation runner and results format.

• A single policy workflow for exceptions.

Standardisation buys you safety and speed. It also gives developers clarity. See Developers as Customers for how to roll out changes without friction.

Answering the questions from day one

Return to the opening scenario. With the platform extension in place:

• Which model did the agent use? The model access service records model and version per call. The serving layer ties endpoint versions to models.

• What context did it see? The data context service registry and traces record context sources, retrieval filters and freshness at request time.

• Which tools did it call? The agent runtime captures a step‑level trace and tool call inventory with inputs and outputs.

• Why did costs spike? Observability shows tokens and per‑call costs tagged by service and use case. Routing and quotas cap runaway workloads. FinOps showback covers responsibility.

• What regressed? Evaluation baselines and online probes show where quality changed. Canary and rollback in serving reduce blast radius.

These answers are table stakes for operating AI in production. They are the outcome of productised platform capabilities, not of manual triage.

Risks and mitigations

A few common pitfalls and how to avoid them.

• Building an AI silo. Mitigation: place AI engineers inside the platform org; publish shared roadmaps; integrate identity, tracing and billing from day one (PlatformEngineering.com (opens in a new tab)).

• Over‑abstracting models. Mitigation: expose model capabilities and constraints in the catalogue. Avoid a “lowest common denominator” that hides useful features. Provide routing with explicit opt‑in.

• Ignoring evaluation. Mitigation: make evaluation a gate in CI before any AI endpoint goes live. Provide templates and datasets. Tie OKRs to quality signals (TrueFoundry (opens in a new tab)).

• Weak cost attribution. Mitigation: enforce tags at the SDK and gateway. Treat missing tags as errors. Publish weekly showback during adoption. Align with FinOps for AI.

• Policy as paperwork. Mitigation: codify guardrails in the model access, data context and agent runtime services. Automate exceptions with time limits and audit.

• Over‑promising. Mitigation: ship the 90‑day sequence. Declare scope plainly. Expand only after adoption and signals show value.

Governance reporting: what to show to leadership

Leaders do not need a tour of toolchains. They need proof that AI is controlled, costed and delivering value. Your dashboards should cover:

• Adoption: teams using AI golden paths; endpoints live by environment.

• Quality: evaluation pass rates; regression incidents; safety events.

• Cost: spend by model, team and use case; unit costs (per request, per customer); accelerator utilisation.

• Risk: exceptions granted and expired; policy denials; data residency adherence.

• Efficiency: time from idea to first safe deploy on the golden paths.

These measures use the signals designed into the seven services. They also link to financial and risk governance without creating new committees. What they cannot tell you is who is entitled to act on them: a rising exception count is a fact, not a decision. Pair this reporting with the decision rights, lifecycle gates and review cadence in AI Governance Operating Models, so every number on the dashboard has a named owner and a gate it feeds.

Where to take it next

Beyond 90 days, expand in measured steps:

• Add more models and capabilities to the catalogue as demand grows.

• Integrate with more data domains in the context service, prioritising those with clear ownership.

• Enrich the agent runtime with simulations and scenario testing.

• Extend evaluation to include business outcomes where attribution is possible.

• Optimise serving for latency and cost with workload‑aware routing.

• Mature FinOps for AI into forecasting and budget planning with product managers.

• Iterate the AI Grid of controls as regulations and enterprise policies evolve.

The direction is consistent: modular capabilities inside the platform, product‑managed and adopted through paved paths.

Takeaway

AI does not replace your internal developer platform. It stretches it. The gap between standardised application delivery and AI in production is filled by seven modular services: model access, data context, agent runtime, evaluation, serving, observability and policy. Build them into your platform, avoid a separate AI fiefdom, and sequence delivery over 90 days. You will keep developer velocity, gain control over cost and safety, and have answers when behaviour changes.

For operating model patterns and control design, see AI Platform Operating Model, Platform Operating Model and AI Grid. For cost governance, start with FinOps for AI and the practical notes in FinOps for AI. The industry direction is clear: modular, AI‑native capabilities inside the platform, not a separate AI empire (PlatformEngineering.com (opens in a new tab); TrueFoundry (opens in a new tab)).