I found the same failure four times, in four unrelated places, before I noticed it was the same failure.
First in a flight simulator, six years after two planes crashed. Then in a car park in Aurora, Colorado, where a license plate reader matched a family’s SUV to a stolen motorcycle. Then in an Indian courtroom, where a tribunal cited court cases that didn’t exist. Then on a two-lane road in Key Largo, Florida, where a car’s cameras saw a pedestrian 116 feet away and a jury spent weeks deciding who was actually driving.
Four industries. Four decades apart in origin, months apart in publication.
I didn’t build these stories to prove a theory. I found each one separately, wrote about it, then moved on to the next thing that caught my attention.
At some point they stopped looking like four stories and started looking like one.
Connecting the dots
When we implement systems whose output we rely on to make real decisions, we build in safeguards — approval steps, audit trails, sign-offs, a person whose job is to check the work before it goes out the door. Often we call these guardrails, the rules or boundaries. We then put in place governance to check we are staying inside these guardrails. When a human did that work, a lot of the checking happened without anyone planning it — someone paused, glanced back over it, or turned to a colleague and asked, “does this look right to you?”.
AI removed that informal checking. It just executes, which is how a gap that was always there finally became visible.
Readers who have been reading my published articles over the last few weeks will have spotted the pattern: A date, a protagonist, an event where automation went unchecked.
In this article I wanted to stop and connect the dots on why I keep finding this, and why I think it’s about to matter a great deal more than it has so far.
The house
I like to simplify the complex by using analogies. Think about a house.
Every house has a structure – a foundation, core framework, load-bearing walls. Let’s call that the spine. An architect designs the spine first, then builds everything else around it: rooms, partitions, windows etc.
Builders know the order. Foundation first, then everything built off it.
Guardrails and governance map onto this directly. One guardrail might be a wall. Governance is the inspector, walking through after the house building is complete, checking the walls are standing where they should be.
But here is the part most people miss. Not every wall needs to hold weight. A decorative partition doesn’t need to, but it’s still nailed into the frame, top and bottom, even though it is not load bearing.
But a guardrail is not decoration. Like a wall, it has to do two things at once: be nailed to the spine, and actually serve a purpose like separating one room from another — not just stand there.
Too many guardrails today do neither. They stand in open space, not attached to anything, not marking the boundary of anything. They look like a wall. Nobody decided they should be there, or what they were meant to divide.
An inspector can walk straight past a wall like that and tick the box. It looks just like every other wall in the building. Nobody finds out until someone actually tests it.
Let me colour that picture with a real example.
Aurora, in this language
Take the number plate recognition problem in Aurora, Colorado. On paper, the guardrail – rule – was simple: an officer must confirm an automated plate-reader hit before acting on it.
Notice what that policy got right. It named a who. Most organisations never get that far. They build the system and assume someone, somewhere, will catch it if something’s wrong. Aurora didn’t assume. They wrote it down: an officer confirms.
But confirm what, exactly?
The plate numbers had already matched, the machine had already done that part. Confirming the digits again confirms nothing new. Confirming the actual vehicle, the make, model, colour, state, would catch a problem in five seconds:
A Colorado SUV isn’t a Montana motorcycle, whatever the plate says.
Nobody had ever specified which one “confirm” meant. So on a hot Sunday afternoon in August 2020, an officer did the only confirmation the system had made easy; checked what the reader had already told him that the digits matched. He then authorised a felony stop on a woman, her sister, and four children.
The who was named. The what was never decided.
On paper, that word ‘confirm’ checked the box. That meant an officer could check the plate, confirm the digits, and never ask the deeper question: is this the same vehicle.
And that is exactly what a guardrail without a spine looks like.
Let’s step back. Think of the plate-reader program as a wall. All houses need walls. But walls built within what? Imagine a house with no plan; nothing that starts by laying out the spine first: load-bearing walls, foundation, the order everything else has to follow. You can place walls wherever you like when there’s no plan. They attach to nothing substantial. They can exist without serving any purpose at all.
That is a house that is destined to fall down.
Aurora lacked a spine. That meant guardrails like the confirm plate policy simply checked a box. The guardrail was not missing. It was not being ignored. It was not attached to anything load-bearing. So in the end served little practical purpose.
A defence that is going away
For a long time, none of what we have discussed here mattered much, legally. You could build the visible, auditable layer, a policy document, a training slide, a compliance dashboard, and call it done. A written policy that sounds reasonable has generally been enough to satisfy “reasonable care,” even when nothing underneath it actually holds.
That’s not an accident. It’s an incentive. Vendors get funded for shipping the visible layer, because that’s the part a buyer can see and a board can point to. Nobody’s selling the invisible part, in other words the upstream decision about who actually has standing to intervene, and what they’re authorised to do about it. It doesn’t demo well. So it doesn’t get built, and for a long time, nobody paid for skipping it.
That defence is going away.
More than twenty US states have adopted some version of the NAIC’s Model AI Bulletin, requiring a written, governed program for AI-influenced decisions, one that can be shown to actually function, not just exist. The EU AI Act’s human-oversight provisions became enforceable this month. China’s own framework for tiering AI agent authority took effect in July.
Three regulatory bodies, three continents, independently converging on the same question Aurora’s policy never answered: who, specifically, is authorised to do what?
Regulation is only half of it. Every case above already cost someone real money, decided by a court, not a regulator. $243 million against Tesla. $1.9 million against Aurora. Two grounded aircraft and a redesigned certification process. A tribunal ruling thrown out by India’s Supreme Court. Nobody got to stand up in front of a jury and say “we had a policy” and have that be the end of it. The policy existing was never the question. Whether anything real stood behind it was.
As more organisations get this wrong, in public, with a dollar figure attached, “we had a guardrail” stops being a defence and starts being an admission.
Where to actually start
You don’t need a framework to check your own organisation for this. You need one honest walk-through.
Picture your own building. Somewhere inside it, a wall exists that was never actually part of the plan. Someone put it up because a wall seemed sensible there, not because anyone decided what it needed to divide. It looks exactly like every other wall. It just isn’t attached to anything, and it isn’t sealing off anything either.
Now pick the most consequential AI-influenced decision already running in your organisation, and ask, plainly: who – by name, not by job title – has the standing to say this one needs a second look before it acts? And was that decided on purpose, or does it just sound like the kind of thing that should exist?
If you can answer that in one sentence, you likely have a spine, and a wall properly attached to it. If you can’t, you probably have a wall standing in the middle of a room, nailed to nothing, dividing nothing, and it’s only a matter of time before something tests whether it’s actually there for a reason.
Follow-Up
If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”
If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.
If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io