Human in the Loop Is Not Enough for AI Agents

The assumption almost every AI governance plan rests on

Ask most executives how their company controls its AI and you will hear four words.

A human reviews it.

For the systems enterprises deployed until recently, that was a reasonable answer. Ray Eitel-Porter, who spent years building responsible AI governance at global scale, is careful not to dismiss it. Human in the loop was a good and effective form of AI governance for the AI we had.

The problem is that the AI changed.

As agents move from single predictions to long chains of autonomous action, the human in the loop paradigm runs into limits that have nothing to do with how good the technology is. They have to do with how humans behave when they are asked to check things.

Why human oversight gets worse as AI gets better

The failure mode is not the one most people expect.

If a system is wrong three or four times out of ten, the person reviewing it stays alert. Errors are frequent enough to keep attention high. But when the error rate drops and the system is only occasionally wrong, staying attuned to catch that rare mistake becomes far harder, not easier.

Add volume to that. An agent can execute more decisions in an hour than a reviewer can meaningfully read in a day. The reviewer relaxes. The check becomes a formality.

Researchers call the underlying effect automation bias, and it has been measured.

The radiology study that put a number on automation bias

Eitel-Porter pointed to a study published in Radiology in 2023 by Thomas Dratsch. Twenty-seven radiologists read fifty mammograms alongside what they were told was a newly developed AI system. To establish the system’s credibility, the readers first saw ten cases where it was always correct. Then some of its suggestions were deliberately wrong.

Among inexperienced radiologists, accuracy fell from close to 80 percent to under 20 percent when the purported AI suggested the wrong category.

Among readers with more than fifteen years of experience, accuracy still fell from 82 percent to 45.5 percent.

These were trained specialists reviewing images inside their own domain of expertise. The human was in the loop. The human deferred to the label.

The researchers noted something else. Inexperienced readers rated the AI system a median nine out of ten for accuracy, while rating their own assessment skills a median two out of ten. Low confidence in your own judgment makes you more likely to follow the machine.

Now apply that to a procurement agent, a claims agent, or a customer service agent operating at volumes no radiologist ever faces.

Can AI check AI?

The obvious response is to put another model in the reviewer’s seat. Use an LLM as a judge.

Eitel-Porter’s assessment is that this helps and is already widely used, but it does not close the gap. Large language models are probabilistic. Run the same inputs repeatedly and you get slightly different outputs. One probabilistic system checking another probabilistic system improves the odds without ever making them certain.

That is why he expects growing emphasis on deterministic software controls, and eventually controls baked into hardware, particularly in high stakes domains where regulators require certainty rather than likelihood. Financial services. Healthcare. Anywhere safety is on the line.

Jeff Carson recognized the pattern from his own years inside large banks, where teams built deterministic checks around probabilistic models not because they wanted to, but because regulatory exposure demanded it.

What happens when nobody is at the keyboard

Carson raised a case that had been published days before filming. On July 1, 2026, the cloud security firm Sysdig documented what it assessed as the first ransomware operation run end to end by an autonomous AI agent.

The agent gained access through CVE-2025-3248, a missing-authentication flaw in Langflow, an open-source framework widely used to build AI applications. It harvested credentials, mapped the network, moved to a production database server, encrypted 1,342 configuration records, deleted the originals, and left a ransom note.

The detail researchers found most telling was a failure. An early attempt to insert a backdoor administrator account failed a login check. Within thirty-one seconds, with no human involvement, the agent diagnosed the cause as a subprocess path issue preventing the password hash from generating correctly, rewrote its own code, and logged in. No person reads an error message, identifies the real cause, and ships a fix at that speed.

One clarification matters here, because the early coverage overstated it. Sysdig’s threat research lead has since confirmed that a human selected the target and launched the operation. What ran without supervision was the technical execution, start to finish, once it was underway. That is a meaningfully different threat from an agent that also picks its own targets, and it is the version that exists today.

Eitel-Porter’s response was direct. It is concerning, and it reflects something structural about the technology. AI is dual use. The same autonomy that makes agents valuable inside a business makes them effective in the hands of someone attacking one.

He also flagged the trajectory. Open weight models can be downloaded, run privately, and modified without the original developer’s involvement, which means refusal training can be stripped from the weights. The UK AI Security Institute’s July 2026 benchmark found leading open weight models now trail the closed frontier on cyber capability by four to seven months, down from six to ten at the start of 2025. The window in which defenders hold a capability advantage is narrowing.

The defense he describes is not a pause. It is that AI governance teams and cybersecurity teams stop operating as separate functions. Cybersecurity organizations have decades of accumulated practice. AI opens new attack surfaces those teams have not had to defend before, from prompt-level attacks through public chat interfaces to agents transacting with third parties outside the company perimeter.

Which decisions still need a human

If you cannot put a person on every action, you need a way to decide where a person is required. Eitel-Porter uses three criteria.

•      Consequence. How serious is the outcome if this goes wrong? Severe consequences mean a human makes the final decision, every time.

•      Reversibility. How easily can the damage be undone? If a mistake rolls back cleanly, an organization can absorb the occasional error. If it cannot, the calculus changes entirely.

•      Contestability. Does the person on the receiving end have a right to an explanation or to redress? Under GDPR, automated decisions can carry exactly that entitlement.

The framing underneath all three is risk-appropriate control. Governance applied uniformly is governance applied badly. Low-stakes, straightforward uses of AI should not carry the same guardrails as decisions that cannot be reversed, because over-controlling the simple cases slows the business down for no protective benefit.

Carson noted what happens when this triage never occurs. In Moffatt v. Air Canada, decided in February 2024, British Columbia’s Civil Resolution Tribunal held the airline liable for a bereavement fare policy its website chatbot had invented. Air Canada argued the chatbot was a separate legal entity responsible for its own actions. The tribunal called that a remarkable submission and ruled that a company is responsible for all the information on its website, whether it comes from a static page or a chatbot.

The agent acted. The company paid.

Onboarding an AI agent like a new employee

Carson brought his own working analogy to the conversation. You do not hand a new hire keys to every door in the building on their first day.

Eitel-Porter endorsed the discipline and mapped it directly.

•      Policies. A new employee gets the rulebook. An agent gets governing documents that define what it should and should not do.

•      Context. You explain the task and the situation around it. Same for an agent: clear instruction, clear context.

•      Permissions. Employees get scoped access to specific systems, not everything. Agents get defined database access, read-only where read-only is sufficient.

•      Approved tools. Employees use the tools the organization sanctions. Agents should not be discovering their own tooling on the internet.

•      Monitoring. A manager watches a new hire closely, then loosens as trust is earned. Agents warrant the same trajectory.

Then he drew a line Carson had not drawn, and it is the most important caution in the conversation.

There is a growing tendency to describe agents as team members. Eitel-Porter considers that risky, because the moment you treat an agent as a colleague, you begin attributing accountability to it. A colleague who makes a bad call bears the consequence. An agent cannot bear anything. A human is accountable for an agent, always, and the language an organization uses quietly moves that line.

Borrow the onboarding discipline. Do not extend it to accountability.

When one agent talks to another, who is it?

Agents increasingly transact with other agents. They exchange information, make purchases, and pass along data that may be confidential. Which raises a question most organizations have not asked: how does your agent know who it is dealing with?

Eitel-Porter’s comparison is to the early web. Domain names, security certificates, and browser warnings exist because the internet needed a way for people to know that a site genuinely belonged to the organization it claimed to represent. That infrastructure is what made financial transactions online possible.

Agents have no equivalent yet, and work is underway to build one. In the meantime, a well-governed agent should be demanding credentials from any third-party agent before interacting with it, rather than transacting with whatever presents itself.

The second requirement is logs. Detailed records of what your agents and third-party agents actually did, ideally tamper-proof and timestamped so that alterations are detectable. When a transaction goes wrong between two agents, reconstructing which one made the error is impossible without them, and that reconstruction has to be designed into the system in advance rather than assembled after the fact.

The agents you cannot see are inside your building

Ask most leaders about shadow agents and they picture something external.

Eitel-Porter redirected that. The harder governance problem is internal. Platforms like Microsoft Copilot now let ordinary employees create agents for themselves. An agent built by a data science or IT team can be governed centrally. Agents appearing across departments, built by people with no governance training, cannot be governed the same way.

He identified a structural reason this is difficult. AI governance programs typically calibrate control based on the intended use of a system. Open-ended tools break that model, because a single tool can be used for almost anything. An employee can upload HR records, personal data, or company IP into the same interface. Filters and guardrails help, but they are far harder to design when the tool itself has no fixed purpose.

The regulatory clock is already running

Carson asked about the EU AI Act, and Eitel-Porter’s answer was that it arrives in stages rather than all at once. That staging is now settled. On June 29, 2026, the Council of the EU gave final approval to the Digital Omnibus on AI, deferring high-risk obligations for stand-alone Annex III systems from August 2, 2026 to December 2, 2027, and for AI embedded in regulated products to August 2, 2028.

What did not move is Article 50. The transparency obligations, which require organizations to disclose when people are interacting with AI, apply from August 2, 2026 as originally scheduled. Only the watermarking requirement for systems already on the market received a four-month extension, to December 2, 2026.

The deferral is a reprieve, not a repeal. The EU delayed the high-risk timeline because national authorities and harmonized technical standards were not ready, not because the requirements went away. Organizations reading the headlines as a pause will find themselves sixteen months behind on the one piece of work that takes longest.

Governance is what lets a company move

The conversation closed on the framing Eitel-Porter believes gets lost most often.

Picture a robot with no safety controls moving freely through public space. Everyone would object, and reasonably. It is precisely because organizations build in governance, controls, and safety mechanisms that society becomes willing to accept self-driving cars, delivery robots, and autonomous systems operating around people.

Governance is not the brake. It is the thing that makes moving forward possible.

Carson’s close: the fear is understandable, but the answer is not to slow down or look away. It is to govern well and on purpose, because the companies that get this right do not merely stay safe. They get to use the technology with confidence.

Risk avoided is not value created. The point was never just to be safe. It is to capture the upside safely.