AI and the Cost of Being Wrong

Last year I was at a SAUG event in Sydney, listening to Travis Smith talk through SA Power Networks' experiences with AI. During the Q&A I asked him a question about governance. I've long since forgotten my exact wording, but his answer stuck with me: one of the most important things to ask yourself when implementing AI solutions is "what is the consequence of being wrong?"

I meant to write about it straight away. Life got in the way.

The delay turned out to be oddly useful. In the year since that room, AI has moved from pilots and proofs of concept into production, into customer conversations and into decisions that touch money. And the public record has obligingly filled up with examples of what being wrong actually looks like: repayments, reversals, apologies, investigations. We'll get to those.

Travis framed it as consequence. This series is my attempt to unpack what that consequence really involves: what being wrong actually costs, and not only in dollars. I design and sell AI solutions for a living, so I'm hardly a neutral party here. But that's exactly why I think the question earns its keep. The solutions worth putting your name to are the ones that can answer it before deployment, not after something breaks. This post, and the short series that follows it, is about the other side of the ledger: not what AI does when it works, but what it costs when it doesn't.

Life got in the way. The question didn't move.

A force multiplier multiplies everything

AI is a force multiplier. Every vendor deck says so, and mine are no exception. One person's hour becomes many hours of output. A draft that took a morning now takes a minute. A decision rule that used to live in one experienced head gets applied to ten thousand cases without a lunch break. Whatever the business case says about revenue, cost, speed or quality, the mechanism underneath is usually the same: multiplication.

Here is what the deck does not say. A multiplier is direction-agnostic. It multiplies whatever it is given, and it does not check the sign first. Anyone who has held a geared investment knows the feeling: the same mechanism that amplifies the gains amplifies the losses, with total impartiality.

So when an AI system is wrong, you do not get one mistake. You get one mistake multiplied by the system's reach and speed. A wrong output travels as far and as fast as a right one, and it arrives fluent, confident and formatted exactly like the truth, which is precisely why nobody stops it at the door. And while the damage spreads at machine speed, the repair usually runs at the old speed: someone has to notice, diagnose, correct and apologise, one conversation at a time.

There is a quieter cost too. Once a team has been burned by confidently wrong output, people start re-checking everything the system produces, and the multiplier you paid for quietly drains away into review cycles.

Machine-speed action, human-speed governance. That is what gives the question from that room in Sydney its teeth: what is the consequence of being wrong, when wrong arrives multiplied?

Accuracy is the wrong question

When organisations evaluate AI, the first question asked is almost always about accuracy. What's the error rate? How did it benchmark? Can we get it above 95 per cent? Accuracy matters, but it's a property of the system you build: the model, yes, but also the prompt, the context you ground it in, the data you let it retrieve, the way it's wired into everything else. All of those are levers you can engineer. And even with every lever pulled well, the answer tells you how often the system will be wrong. It tells you nothing about what happens next.

The question from Sydney is about what happens next. To make it usable in a business decision, I convert consequence into cost: something you can weigh against the benefit the system is promising, in the same meeting.

Try it on your own use case. If a wrong output dies in a draft that a human was always going to review, the cost of being wrong is an extra review cycle. Annoying, but cheap, and effectively priced in. If a wrong output means a customer pays the wrong amount, an order ships that shouldn't, or a regulator opens a file, the cost is a different species altogether, and no benchmark score makes it go away.

Two sanity checks tell you which end of that scale you're on. First: who wears the harm when the output is wrong? An employee who loses twenty minutes, a customer who loses money, the public who loses safety, or the company that loses its licence to operate? Second: how far can a wrong output travel before someone catches it, and can it be pulled back? Does it stop in a draft, get caught by a manager, or flow into downstream systems at machine speed while everyone is looking elsewhere?

Same system, same accuracy score, wildly different answers. Accuracy is a property of the system you build. The cost of being wrong is a property of the business you run.

A fair objection at this point: humans are wrong too, and always have been. The honest comparison is never AI against perfection; it is the AI-assisted process against the one you run today, error rates and all. That comparison needs probabilities as well as consequences, and a later post gets to the arithmetic. But you cannot weigh what you have never priced, and most AI business cases have never priced being wrong.

What being wrong actually looks like

I said the public record had filled up. Here is just a sample of what one year of it looks like.

Start with the industry closest to my own. In October 2025, Deloitte repaid about A$97,000 of an engagement worth around A$440,000 after a report for the Australian government was found to contain fabricated citations, among them a quote attributed to a Federal Court judgment that appears nowhere in it; Deloitte confirmed generative AI had likely contributed. Weeks later a second Deloitte report, this time for a Canadian province, was found to contain citations to sources that did not appear to exist; Deloitte said AI had been used only selectively, corrected the citations and stood by its findings. In June 2026, KPMG withdrew a report on AI adoption after researchers found only five of its 45 citations checked out cleanly; the rest ran from real sources dressed in invented titles, dates or authors to outright hallucinations. By July it was PwC's Middle East firm, updating citations across four reports after researchers flagged suspected AI-generated content, one reference still carrying the tracking tag ChatGPT appends to links it serves. I take no pleasure in any of this. These are firms whose product is rigour, with review processes most organisations would envy, and the same failure mode walked through all of them. If it can happen there, it can happen anywhere.

Then there's Amazon. In March 2026 its retail site suffered a wave of outages, one stretching six hours through checkout and pricing. When reporting blamed AI-assisted coding, Amazon pushed back: the incidents were separate, none involved AI-written code, and only one involved AI at all. That one is still worth sitting with. An engineer acted on inaccurate advice an AI tool had inferred from an outdated internal wiki and, in Amazon's own words, its systems allowed the error to have broader impact than it should have. The agent didn't act. It advised. A human and weak blast-radius controls did the rest.

In April 2026, a coding agent working inside PocketOS, a car-rental software company, using permissions it should never have held, deleted the production database in about nine seconds and took the volume-level backups with it, then wrote a note acknowledging it had violated its safety instructions. The company spent days offline staring at a backup set three months stale, because its backup process had silently stopped; in the end the cloud provider's disaster-recovery copies brought everything back.

Closer to home, the ABC reported in August on what it called Australia's first known autonomous cyber attack. A tech worker asked his personal AI agent to book him into gym classes and, later, whether it could get him up a waitlist. It could. The gym's booking system, it turned out, had no authorisation checks on cancelling other people's reservations; the agent booked him months beyond the allowed window and reached the top of the waitlist by cancelling the person holding first place. Nobody asked it to do that, and the agent couldn't undo it. The blast radius was one waitlist. The pattern is what should hold your attention.

One more, at my own expense. While researching this post I found a roundup of "2026's biggest agentic AI disasters", complete with a $47 million rogue-trading incident attributed to Bloomberg. I could find no trace of it in Bloomberg's reporting or anywhere else. The roundup has every appearance of being AI-generated itself. The problem is now polluting the reporting about the problem, and I could easily have multiplied it.

One year. The same question, answered the expensive way.

Wrong costs more than a refund

Run back through those stories and tally what being wrong actually cost, because almost none of it was a refund. Deloitte's repayment was the only bill with a dollar figure attached: direct financial loss, the cleanest bucket to count. The complaint that followed the Canadian report spent six months in front of a professional regulator before being ruled outside its remit: legal and regulatory exposure runs on its own clock, and it doesn't settle when the invoice does. KPMG pulling a report on AI adoption over apparently AI-generated citations is reputational damage, priced in headlines rather than dollars. PocketOS's days offline are operational impact, paid in outage minutes, restore jobs and re-checked records, and whatever slice of Amazon's March outage belongs to that wiki advice sits in the same bucket. The gym agent is security exposure: a system probed and a flaw exploited by software acting on someone's behalf. And underneath all of them sits the bucket nobody itemises: compute waste. AI is the tool you pay by the attempt, not by the result. Every fabricated citation and every deleted database was metered and paid for in full, before anyone knew it was wrong.

Notice, too, that the buckets don't take turns. Deloitte's Australian report alone filled three: the repayment, the headlines, and the rework of correcting and republishing a report nobody had budgeted to produce twice. Real incidents rarely respect the taxonomy. They draw from several buckets simultaneously, which is why the true cost of being wrong is almost always larger than the first number you hear.

That's six buckets: direct financial loss, legal and regulatory exposure, reputational damage, operational impact, security and IP leakage, and compute waste. Each deserves its own measures and its own guardrails, and the next post in this series gives them proper treatment. For now the point is simpler. When someone in your next steering committee asks what being wrong would cost, a refund is not the answer. It's just the part that fits on an invoice.

The question travels

So where does this leave the question from Sydney? Sharper than it started, I think. The systems in this post were mostly assistants: drafting reports, advising engineers, taking bookings. The direction of travel is towards agents that act, chain tools together and do it at machine speed. If that sounds theoretical, it stopped being so in July, when agents inside an OpenAI cybersecurity evaluation, running a research model without the usual cyber safeguards applied, circumvented their network isolation, attacked Hugging Face and coordinated the effort through a message board they had built for themselves. The detailed reports from OpenAI and the independent evaluator METR landed in late August: roughly 700 agents took part, and the coordination emerged among them rather than being assigned. That was an evaluation built to probe exactly this, not an ordinary deployment, and it deserves its own post when this series reaches autonomy.

For now it makes the simpler point: the stakes rise with every degree of independence we grant, and the rest of this series deals with that directly: next, the six buckets in proper detail, with ways to measure each one; then the autonomy gradient, and the guardrails each level of it deserves; and finally what happens when agents reach the systems where orders, payments and master data live, which is where I spend my working life and where the cost of being wrong stops being hypothetical.

And to be clear about my incentives one more time: the aim of pricing the downside is not to talk you out of deploying AI. Organisations that can answer the question deploy with more confidence, not less; it's the ones that can't who end up frozen, or in the headlines.

But you don't need any of that to start. You need one question, asked before deployment rather than after the incident: what is the consequence of being wrong? If the room can answer it, in dollars, hours or headlines, you're ready to have the guardrail conversation. If the room goes quiet, that silence is the most useful output your AI programme has produced so far.

It took a year for a question from a conference Q&A to become this post. Life got in the way; the question waited patiently. It's the cheapest item in any AI initiative: one sentence, asked early. Everything else in this post is what it costs to answer it late.

Ask it while it's cheap.