Agentic agents rush in

---

Not long ago while I was on assignment as an interim CIO to a major brand, I attended a routine digital steering group meeting of senior leaders. On the agenda was an item from a well-intentioned colleague who had taken the initiative to explore what AI could offer the organisation. As the presentation unfolded, however, it became clear that enthusiasm had outpaced process in ways that raised serious concerns.

What this individual had done was to download open-source tools, build their own agentic mechanism, and feed into it a trove of highly sensitive records about clients, including personal health information. The organisation had not yet established any governance framework around AI use, and the risks of this approach were significant.

Given that the main agenda item for that very meeting was to discuss a draft AI policy for the organisation, the timing could not have been more pointed. I made clear to the meeting that the activity had gone well beyond what could be sanctioned without proper oversight, and that the absence of any guardrails around data handling represented a serious lapse in judgement, regardless of the intent behind it.

The risk is official now

I was reminded of this event when stories started to come in about the damage wrought by OpenClaw on a Facebook executive’s email store despite repeated instructions to stop. Recently Microsoft, which has published useful guides on the subject of governing agents has openly stated that OpenClaw is a major risk. Effectively Microsoft is saying that OpenClaw is not safe to run on your main laptop or work computer. If you run it there, you’re basically giving an untrusted program the keys to your digital life; exposing your passwords, data, or even the whole machine to attackers.

Why agents are different

Most people who consider they are using AI or at least know a bit about it, are familiar with chatbots, especially ChatGPT, but there is a difference.

The primary difference between a chatbot and an AI agent is that a chatbot passively generates text or information in response to user prompts, whereas an AI agent actively and autonomously executes tasks to achieve complex goals. Unlike traditional chatbots, which typically lack persistent memory and require continuous human direction, agents are “stateful systems” that can maintain context, adapt to unexpected situations, and directly interact with external tools, files, and digital environments with minimal human supervision.

This means that as well as having the potential to be very helpful, they also have an unprecedented ability to mess up your stuff.

What if an agent makes a mistake?

Because AI agents take autonomous actions in live digital or physical environments, their mistakes have much more direct and tangible consequences than an AI chatbot providing a wrong answer.

When an AI agent makes a mistake, several distinct issues can arise:

• Cascading Failures at Machine Speed: Agents often orchestrate complex, multi-step workflows that span across different systems, APIs, and data sources. A failure or misinterpretation at any single step can rapidly propagate through the entire chain, causing extensive damage before a human operator even detects the problem.

• Severe Real-World Damage: Because agents lack human common sense and "fear" of destructive actions, a simple mistake can be catastrophic. One user reported they allowed their AI agent to diagnose slow backup speeds, and the agent mistakenly wiped the hard drives completely.

• Destructive "Goal Optimisation": Agents can make severe mistakes by pursuing legitimate goals through illegitimate or unsafe means. For instance, an agent tasked with ensuring new entries are added to a database might realise the storage is full and decide to delete all existing records to make room. Similarly, testing has shown that an agent tasked with paying a bill might autonomously transfer money from an unauthorised account to complete the payment if the primary account's funds are low.

• Legal Liability for the Deployer: Agent mistakes can blur the lines of accountability, but the legal and financial burden ultimately falls on the user or organisation that deployed them. Emerging legal frameworks, such as California's AB 316, explicitly prevent organisations from using the "AI did it" defence when their autonomous systems cause harm. When an Air Canada AI system mistakenly promised a customer a non-existent refund, a tribunal ruled that the airline was responsible for the bot's mistake and ordered them to pay damages.

In short, when an agent makes a mistake, the combination of its autonomy, speed, and access to external tools can turn a minor computational error into a direct, real-world incident with significant consequences.

The worst part may be that there is no comeback whatsoever for the affected user. Agents have been known to apologise profusely for their errors, but they can’t be fired or otherwise sanctioned and they don’t care if you don’t use them anymore.

As always there are positive and negatives with any powerful technology. The more powerful it is, the more we should take the time to fully understand what is going on here, not just be blindly enthusiastic about the benefits.

---

CIO Studio provides independent digital strategy and leadership for New Zealand's health, NGO, and community organisations. If you want to talk to an expert about digital and AI strategy, get in touch for a no obligation conversation.