In 2023, a police algorithm in Aurora, Colorado matched a licence plate to a stolen vehicle. Only one of three checks agreed — the plate. The state and the vehicle description didn’t. A patrol car pulled the vehicle over at gunpoint.
It was the wrong car.
In October 2025, an AI-enhanced surveillance camera at a Baltimore County high school flagged a crumpled Doritos bag in a student’s pocket as a gun. Within moments, police arrived, drew their weapons, and forced 16-year-old Taki Allen to his knees.
It was a bag of chips.
Two different systems. Two different cities. Two different technologies. But the same failure, in the same place, both times:
A machine produced a signal, nobody had decided in advance what that signal was allowed to authorise on its own, and armed officers acted on it as though someone had.
That is not a coincidence, and it’s not a technology problem. It’s a design problem, and it’s one I’ve spent the past several years working on.
The gap people are starting to name
Others have begun to talk about this. The language itself varies: “decision ownership,” “authority gap,” “judgment gap”, but the underlying observation is becoming common:
Organisations are using AI and computer technology to automate decisions before they have clearly defined which decisions can, and which decisions cannot, be automated.
It is all well and good pointing out the gap, but quite another thing to do something about it.
Three parts, and the one everyone skips
With the arrival of AI too many have rushed in and implemented an AI powered solution. Many have failed.
Why?
Let’s step back for a moment. I think about AI decision systems as having three parts.
The spine. This is the blueprint of any decision process. It defines the boundaries. Let’s use Aurora to help illustrate. If the number plate, state and vehicle type match – 3-of-3 – automate the process. Any other condition needs human oversight. Using an analogy – this is the boundary or the fence line.
The guardrail. This is the infrastructure that holds and enforces a boundary once it’s set — checking a proposed action against a rule, letting it through, blocking it, or escalating it. Extending the analogy, this is putting up the fence and defining what sits inside (automation) and what sits outside (human judgment) that fence. Some of the best engineering happening in AI right now lives here. A number of innovative companies are now solving this problem.
The automation. This is the work that runs inside that boundary (or fence using our analogy) once it’s marked – fast, repeatable, and safe precisely because someone already decided it was safe.
Neither Aurora nor Baltimore lacked a guardrail. Both systems had rules, checks, thresholds — the plate-matching logic ran, the image classifier ran.
What neither had was a spine:
Nobody had decided, ahead of time, that a one-of-three match, or a single flagged image with no second signal, wasn’t enough to authorise an armed response on its own.
Where the spine comes from
This is the part that is easy to miss: the spine isn’t a smarter algorithm, and it isn’t a stricter rule bolted onto the existing process.
It comes from designing the decision process itself.
That does not mean documenting how the process works today, but working through, with someone – a subject matter expert or SME – who actually knows the workflow and its failure modes, what the process should be now that AI can do things the old process couldn’t.
From there designing the decision process itself – think of this as a set of instructions.
Much of my thinking here came from aviation’s model. Before a new aircraft type ever flies, engineers sit down and build a checklist specific to that plane; critically that is not one copied from a different aircraft. Most items on that checklist can be legitimately cleared with more evidence: a second sensor confirms a reading, another source resolves it.
But some can’t.
A crack in a wing strut doesn’t get cleared by a second scan, however clean it comes back. It gets cleared by an engineer physically inspecting it, no matter what any recheck shows.
And this needs clearly defining, before the aircraft ever takes off.
Aviation didn’t arrive at that distinction by accident. It was built deliberately, item by item, because the cost of getting it wrong is measured in lives. Almost no other industry has to do the same work, not because the stakes are always lower, but because nobody has forced the reckoning the way flight safety did decades ago.
AI is forcing that reckoning now.
The practice
This is at the heart of the work I have been exploring at SpatialNext:
- Sitting down with a subject-matter expert (SME) who knows a given decision process — its steps, its failure history, where a system’s confidence shouldn’t be trusted alone.
- Next, working with the SME to redesign the decision process itself, given what’s now possible, rather than digitising the workflows.
- Lastly, classifying what comes out of that redesign: which decisions are safe to hand to evidence and automation, and which must always end with a person, regardless of how strong the automated case looks.
SpatialNext is focused on spine building — defining that boundary. This is what a guardrail then enforces, and what automation then runs inside of. Not a technical fix.
A design discipline that’s currently missing in most AI implementations.
Baltimore and Aurora weren’t failures of AI. They were failures of a step nobody took before the AI was ever switched on.
And that is the gap I fill. Not after the stop. But before it.
Follow-Up
If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”
If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.
If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io