A growing number of enterprises are running into the same uncomfortable discovery. The AI agent works – it books the shipment, refunds the customer, flags the compliance exception – right up until the moment it doesn't and nobody in the building can quite explain why it did what it did, or who was supposed to be watching.
That's not a bug report. It's a governance gap and it's opening up faster than most organisations, and most regulators, have noticed.
Why the old rules stop applying
Conventional AI oversight was built around a simple assumption: a model produces an output and a human decides what to do with it. That assumption quietly disappears the moment a company starts building agent systems that can plan their own steps and act across systems without waiting for a person to sign off on each one. In an IBM study, organisations reported an average of 54 AI agent incidents a year and a third of those caused cascading failures — one agent's bad call becoming the input another agent trusted and built on.
The reasoning an agent displays isn't always the reasoning that actually drove its action, either, which means oversight built on reading an agent's stated logic rather than logging what it actually did is oversight built on the wrong evidence. And because agents increasingly hand work to each other, a flawed assumption made early in a chain doesn't stay contained — it becomes the next agent's starting point and the one after that.
Two-thirds of technology leaders now say they're held accountable for outcomes produced by systems they don't fully control. That's a strikingly direct admission from the people whose job it is to be in control and it points to something regulators haven't caught up to: these systems are being deployed well ahead of any clear answer to the question of who is responsible when one of them gets it wrong.
The oversight math doesn't work
Here's the arithmetic that makes "just have a human check it" an increasingly hollow safeguard. Take an agent carrying out 10,000 actions an hour, reviewed by a person who can realistically assess perhaps 50 of them in that same hour. That reviewer is seeing 0.5% of what the system actually did. Under that kind of load, meaningful review degrades into habitual approval — a person clicking "confirm" because the last fifty actions were fine, not because this one was actually checked.
That's the quiet failure mode nobody budgets for: oversight that exists on paper and as a job title, but has stopped functioning as an actual check by the time anyone notices.
The honest response to that problem isn't more people reviewing more actions — that doesn't scale and everyone building these systems already knows it. It's being explicit about which decisions genuinely require a human before they happen (an irreversible payment, a credit decision, anything that can't be unwound), which decisions can be monitored after the fact with room to roll back and which low-stakes, reversible tasks can run with rules set in advance and audited later rather than supervised in real time. Treating all three categories the same way – either full human sign-off on everything, which collapses under volume, or blanket trust, which is how a single bad assumption turns into a cascading failure – is how organisations end up with oversight that looks rigorous in a policy document and does almost nothing in production.
Accountability gets blurry exactly where it matters most
When an autonomous agent causes real harm, the fault rarely sits with one party. There's the company that built the underlying model, the team that built the tool the agent called, the integrator who connected it to a live system and the person nominally supervising the agent's work. Current regulation doesn't do much to sort that out. The EU's AI Act requires human oversight and transparency for high-risk systems, with its obligations for those systems expected to start applying from December 2027 under recent amendments — but it offers comparatively little technical specificity about what oversight actually has to look like for a system that acts autonomously at machine speed, as opposed to a model that simply returns an output for a person to act on.
Australia has its own version of this gap. Proposed mandatory guardrails for AI used in high-risk settings have been sitting in consultation since 2024, alongside a voluntary safety standard meant to hold the line until something binding exists. Voluntary standards are useful for organisations already inclined to act responsibly. They do nothing for the ones that aren't and agentic systems are being deployed into production well ahead of any enforceable requirement that someone has to answer for what they do.
Cost compounds the opacity problem. Separately, 85% of organisations report they lack full real-time visibility into what their AI systems are actually spending and doing - which means a meaningful share of enterprises deploying autonomous agents can't fully account for their own exposure even in purely financial terms, let alone in terms of what those agents did and why.
What accountability would actually require
None of this is unsolvable, but the fixes are more structural than most organisations currently treat them as being. An agent needs a named human owner - not a team, a specific person who answers for it - the same way a company car has a registered driver rather than a shared set of keys. Its access to systems and data should expire with the task it was given, rather than persisting as a standing credential that outlives the job it was created for. Every consequential action needs a durable, tamper-evident record of what the agent did and why, because a reasoning trace a company can't produce after the fact isn't oversight, it's a liability waiting to be discovered during an incident review rather than before one.
And the emergency stop has to sit genuinely outside the system it's meant to stop — not a setting the agent itself could theoretically influence, any more than a car's brake pedal should run through the same computer that's doing the driving.
Much of this comes down to a shift a lot of engineering organisations are only just starting to make: moving from a scattered collection of individual agents, each bolted onto a different tool with its own ad-hoc rules, toward one governed delivery pipeline where every agent's actions are visible, logged and answerable to the same set of controls. That's a harder, slower thing to build than simply switching an agent on, which is precisely why so few organisations have done it yet.
Teams that build these systems for a living have started treating governance as an engineering problem rather than a policy document, for a straightforward reason: a rule that only exists on paper doesn't stop an agent from acting on a bad assumption at 2am, but a permission that was never granted does.
The part worth asking out loud
What's largely missing from the public conversation is any expectation that companies disclose when an autonomous system causes harm, in the way airlines are required to report incidents or banks are required to report breaches. Gartner expects automated governance tooling to cut the cost of regulatory compliance by 20 to 70% by 2028, which is a reasonable argument for why the technical side of this problem is becoming more tractable. It says nothing about whether the public will actually find out when an agent gets it wrong, how often that happens, or who was supposed to be watching when it did.
Enterprises are racing to deploy autonomous systems that act faster than any person can meaningfully supervise, inside a regulatory environment that is still, by its own admission, working out what oversight is supposed to mean once the human in the loop stops being able to see most of what's happening in it. That gap between deployment speed and accountability infrastructure is the actual story here - not whether the agents work, which increasingly they do, but whether anyone will be able to say, after the fact, who was supposed to be watching.







