The Accountability Gap: Governance Frameworks Meet the Autonomous Agent
NIST's AI RMF, ISO 42001, and the EU AI Act were written for a model that produces an output a human then acts on. Agents act directly — and the owner of record, the audit trail, and the kill switch are all still being retrofitted.
Most AI governance in production today inherited its shape from a period when the deliverable was a prediction. A model scored a loan application, flagged a transaction, or drafted a paragraph, and a person downstream decided what to do with that output. The governance artifacts we standardized around — a model card, a risk classification, a human-review checkpoint, a documented owner — all assume that a human decision sits between the model and any consequence in the world.
Agentic systems removed that seat. When an agent reads a ticket, calls an internal API, moves a record, and closes the loop without a human in the middle, the review checkpoint the framework assumed was there is gone — and with it the point at which accountability was supposed to attach. The controls did not fail so much as get bypassed by a workflow they were never positioned to see.
This is not an argument that the major frameworks are wrong. NIST's AI Risk Management Framework, ISO/IEC 42001, and the EU AI Act are each sound about the thing they describe. It is an argument that the thing they describe — a governed model — is no longer the unit that acts. The unit that acts is a governed identity with standing access, and almost nothing in the current stack was designed to answer the two questions that identity raises: who owns what it does, and how do you stop it.
The frameworks govern the model, not the actor
Read the control catalogs closely and the assumption is everywhere. NIST's RMF organizes work around Govern, Map, Measure, and Manage functions that are anchored to an AI system and its intended context of use. ISO/IEC 42001 certifies an AI management system — the organizational process around building and operating models. The EU AI Act classifies obligations by the risk of the use case and layers separate rules for general-purpose models on top. All three are coherent, and all three take the model, or the process that produces it, as the object of governance.
An autonomous agent is not fully described by any of those objects. It is a loop that pairs a model with a set of tools, a memory, a trigger, and — critically — a credential that lets it act. The same underlying model, wired to a read-only reporting tool, is a low-risk assistant; wired to a payments API with a service account, it is something a regulator would recognize as consequential. The governance-relevant properties live in the wiring, not in the model card, and the wiring is exactly what a model-centered framework does not have a field for.
The EU AI Act's general-purpose provisions and its August 2025 code of practice sharpened obligations on model providers — documentation, systemic-risk evaluation, transparency. Necessary, and largely aimed upstream. The deployer who composes that model into an agent with production access is operating in the space between "provider of a model" and "operator of a high-risk system," and that space is where most real agent deployments actually sit.
The agent is a non-human identity, and it outnumbers you
Security teams already had a name for what an agent is before the AI vocabulary arrived: a non-human identity. Service accounts, API keys, workload identities, and bot credentials have outnumbered human identities in most enterprises for years, and they have long been the harder half to govern — created for a task, granted broad access for convenience, and rarely retired. Agents are non-human identities with a language model deciding, at runtime, how to use their access.
That inheritance is the security story. The failure modes are the ones identity teams already know — over-provisioned standing credentials, secrets that never rotate, permissions granted once and never reviewed — now attached to an actor whose next action is decided by a model reading untrusted text. OWASP's LLM Top Ten named this convergence directly: Excessive Agency sits on the list precisely because the blast radius of a compromised or misled agent is set by the scope of the identity it runs as, not by the cleverness of the prompt that misled it.
An agent granted a single narrow, short-lived, auditable credential is a bounded problem even when it is fully compromised. An agent running as a standing service account with write access across three systems is an unbounded one the moment a poisoned document convinces it to act. The difference is entirely a matter of identity governance, and it is decided at provisioning time — long before any prompt is ever injected.
The audit trail records the call, not the intent
When something goes wrong and governance asks the accountability question — who decided this, and on what basis — the artifacts an agent leaves behind answer a narrower question than the one being asked. The logs show that the agent called refund.issue with a set of arguments at a timestamp. They rarely show the reasoning that selected that action, the document whose text steered it there, or the human who owns the outcome.
A traditional access log answers "which identity did what." An agent needs a decision log that answers "which inputs, which reasoning, which policy in force at the time" — and most deployments capture the tool call without the context that made it a governance event rather than a line in a metrics dashboard. Reconstructing intent after the fact, from a trace that recorded only the effect, is the recurring pain of every agent incident review.
The ownership question is thornier than the logging one. A model card names the team that trained a model. It does not name the person answerable when an agent composed from that model takes an irreversible action in production. Frameworks call for a designated owner; org charts rarely have a row for "accountable human for autonomous agent number forty-one," and the agents proliferate faster than the ownership records that are supposed to track them.
What changes when you govern the actor
Register the agent, not just the model. The unit of governance is the deployed loop — model plus tools plus credential plus trigger — and it needs its own record: what it can touch, which identity it runs as, what it is allowed to do without a human, and who is accountable when it does. A model card describing the base model does not answer any of those, and the answers change every time the wiring changes.
Scope the identity, then trust the prompt less.An agent's worst possible action is bounded by its credential, not by its instructions. Short-lived, narrowly-scoped, per-task credentials turn a full prompt-injection compromise into a contained one; standing broad access turns the same compromise into an incident. Provision for the adversarial case, because a model reading untrusted text will eventually be given adversarial text.
Log decisions, not only calls. An audit trail that captures the tool invocation but not the inputs, the reasoning, and the policy in force cannot answer the accountability question when it is finally asked. Capture enough of the decision context that an incident review can reconstruct why, not merely confirm what.
Name the human, and keep the kill switch reachable. Every autonomous agent needs a person answerable for its actions and a tested way to revoke its access immediately — not a redeploy, not a code change, a switch. Both tend to be assumed present and discovered absent during the first incident, which is the most expensive moment to go looking for them.
None of this replaces NIST, ISO 42001, or the EU AI Act. It fills the gap they leave: those frameworks govern whether you should have built the system, and how well the model behaves. Governing the agent as an identity with standing access is what governs whether you can answer for it once it is running.