Insurance Regulations by State (Nearmap) Nearmap, an aerial imagery provider, tracks how US states regulate insurers’ use of aerial images in underwriting and nonrenewal. More than a dozen states now have rules, and a recurring theme is that imagery should be one input among several, never the sole basis for dropping a policy. It’s a useful reference from a geospatial firm mapping the limits placed on its own product. https://www.nearmap.com/solutions/insurance-regulations-by-state
2. SB 5928: Concerning wildfire risk models and score disclosure (Washington State Legislature) A 2026 Washington bill, requested by the state’s insurance commissioner. It would require insurers that use a wildfire risk score to tell policyholders their score, the range of possible scores, who created it and when, and to give them a right to appeal. Similar bills were introduced this year in Hawaii, Oregon and New Mexico. https://lawfilesext.leg.wa.gov/biennium/2025-26/Pdf/Bills/Senate%20Bills/5928.pdf
3. New Records Obtained by EFF Expose More Problems with Medicare’s AI Experiment (Electronic Frontier Foundation, 27 October 2026) More than 800 pages of records on Medicare’s AI prior-authorisation pilot. They show CMS privately flagging problems with four of its six AI vendors, including denials that appeared inaccurate. The key section argues that a human in the loop only protects anyone if the reviewer has the time and practical authority to override the AI. https://www.eff.org/deeplinks/2026/10/new-records-obtained-eff-expose-more-problems-medicares-ai-experiment
4. California’s “No Robo Bosses Act” (SB 947): What Employers Need to Know (Sweta Patel, Potomac Law Group, 23 October 2026) A clear guide to California’s new law on AI in discipline and termination. Patel makes the point that corroboration isn’t a rubber stamp: a reviewer who finds no support for an output has disqualified it. She advises giving reviewers written authority to reject a result, and notes that monitoring flags and quality scores may be covered. https://www.potomaclaw.com/news-PLG-Californias-No-Robo-Bosses-Act-SB-947-What-Employers-Need-to-Know
5. California to Require Human Corroboration of AI-Driven Discipline and Termination (Nicholas Spencer, AALRR, 23 October 2026) Spencer opens with a scenario every manager will recognise: a dashboard ranks an employee bottom of the team, a manager acts, and “the system identified the problem” becomes the explanation. From July 2027 that won’t be enough in California. He also notes that the law gives no percentage or safe harbour for when an employer has “primarily relied” on the system. https://www.aalrr.com/Labor-Employment-Law-Blog/california-to-require-human-corroboration-of-ai-driven-discipline-and-termination
Where this is heading
These five pieces come from different industries, and they point the same way: when a score feeds a decision about a person, someone has to be able to overrule it, and show that they could.
If your organisation, or one of your customers, is facing one of these deadlines, I’d like to hear what it looks like from the inside. Email me at mattsheehan@spatialnext.io.
In autumn 2021, Nick van Terheyden had a $350 vitamin D blood test. He himself is a physician, so he knew how doctor’s order such test. Cigna refused to pay. The test was not “medically necessary”, the letter said, and it was signed by a Cigna medical director, Dr. Cheryl Dopke.
It took Dr van Terheyden seven months to get the denial overturned. In late June 2022, an independent reviewer found the test was medically necessary, and Cigna paid.
Van Terheyden’s denial is not an isolated case. In March 2023, ProPublica reported that Cigna doctors had denied more than 300,000 claims over two months in 2022, spending an average of 1.2 seconds on each. One doctor alone, Dr. Dopke, denied around 121,000 claims in two months. Cigna’s PXDX system checked each claim against a list of tests the company would pay for with each diagnosis. If the test wasn’t on the list, the claim was flagged for denial. Medical directors then signed the denials off in batches:
“We literally click and submit,” one former Cigna doctor told ProPublica. “It takes all of 10 seconds to do 50 at a time.”
Cigna called ProPublica’s reporting “biased and incomplete”, and and denied that its doctors rejected claims without examining them. The class action that followed, Kisting-Leung v. Cigna, partly survived a motion to dismiss in March 2025 and is still running, with class certification briefing due to start later this month.
This is the part anyone running automated decisions should notice. Every one of those denials had a ‘human-in-the-loop’. In this case a named, qualified doctor.
The human was in the loop
Last week I wrote about Robodebt, the Australian scheme that raised around 470,000 wrong welfare debts. It wasn’t AI. It was a computer acting on a rule. The system took a person’s yearly income from the tax office, spread it evenly across every fortnight, and compared that with what they’d actually reported each fortnight. Where the two didn’t match, it assumed they’d been overpaid and issued a debt.
Robodebt’s problem: it took the human checks out.
PXDX wasn’t AI either. But PXDX did the opposite of Robodebt. It kept the human in. A doctor signed every denial.
But it made almost no difference.
At 1.2 seconds a claim, the doctor isn’t deciding anything. The system has decided, and the doctor is the signature at the bottom – the rubber stamp or a person who is in the loop on paper and absent from it in practice.
This is the uncomfortable lesson for anyone who thinks “keep a human in the loop” settles the question:
Robodebt took people out and failed. Cigna, on these allegations, kept people in and failed the same way.
In both cases, nobody had written down which decisions the system could make by itself, and which needed a person to actually check them first.
A signature is not a review
With the rise of AI, the law is moving fast in this area. Since 1 October, Alabama has required that any prior-authorisation denial on medical-necessity grounds is made or reviewed by a licensed physician or other qualified professional. Iowa and Washington already have their versions. Colorado, Georgia and Minnesota follow on 1 January.
Now read that rule again – prior-authorisation denial on medical-necessity grounds is made or reviewed by a licensed physician or other qualified professional – and think about PXDX. A licensed physician reviewed every denial. On the face of it, the rule is met. And that is the gap.
A rule about who must decide says nothing about what deciding involves.
California has gone a step further, and it is a critical step that narrows this gap. Its new rules on automated decision-making, which apply to hiring and other significant decisions from 1 January, do not simply ask whether a person was involved.
For the review to mean anything, the reviewer has to:
know how to interpret what the system produced
weigh it against the other relevant information
have the authority to make or change the decision
If you run the PXDX 1.2 second denial process through that test, it fails on every line. Nobody interprets a claim they haven’t opened. Nobody weighs evidence in a batch of fifty. And authority to change a decision means little if the volume makes changing it impossible.
This is where most organisations are today, whether they’re screening job applicants or processing claims. The vendors all say a person makes the final call. LinkedIn, Workday and others put the recruiter or reviewer in charge of the decision. What almost nobody has written down is what that person needs for the call to be real: the time to look, the information to judge, and the standing to say no.
Where the line should have been
This is not an argument against automation, or for a person reading every claim. Insurers handle millions of them, and automation is the only way that works. The question is which decisions the system gets to make alone.
Take the van Terheyden’s vitamin D test and walk it through.
Does the diagnosis code match the tests the plan accepts for it? That’s a lookup. Let the system do it, in milliseconds.
It matches. Pay the claim? Yes. Paying a routine claim that fits the rules is exactly what automation is for. That can be batched; no person needed.
It doesn’t match. Is that a denial? No. A mismatch is a question, not an answer. The system can go further on its own: check the patient’s history, the notes that came with the claim, earlier tests. Many mismatches will resolve there.
It still doesn’t resolve. Deny it? That decision is passed on to a real person, it should never be closed by the system alone. It goes to a named doctor, who opens the file and applies a written standard for what has to be looked at before a denial can go out.
That fourth rule above is the boundary.
Once it’s written down, the technology has something real to enforce – no denial without an opened file, and no batch approval of denials.
It also needs a record, which needs to be kept outside the system, about that decision. Who reviewed it, what they opened, how long they spent, and why they decided as they did.
A system should never be the only witness to its own decisions.
If the review is real, the record shows it. If it is not, the record shows that too. That happens long before a class action does.
This also deals with the volume problem that made batch signing tempting in the first place. Doctors stop rubber-stamping hundreds of thousands of denials. And this is critical to understand – they make the calls that are needed, on a far smaller number of cases.
One appeal is evidence about every case
A reader of my Robodebt piece, Warren Seeto, made a very important point.
A system makes the same mistake for everyone in the same situation.
So when van Terheyden won his appeal, that wasn’t one wrong decision put right. It was evidence that every vitamin D claim with the same codes had probably been denied the same way.
A well-designed process treats it like that. An overturned denial should send someone back to check the rule, not just pay the one claim. Aviation learned this decades ago. Near misses are reported, without blame, and acted on before there is an actual crash.
Automated decisions need to be treated similarly.
The new state laws say who must make the decision, and California’s go a step further: the reviewer must understand the system’s output and have the authority to overturn it. What none of them say is how much time a real review needs, or how you prove one took place. Each insurer, and each employer using AI in hiring, has to work that out for themselves.
Having a human in the loop isn’t the test. Cigna had a doctor sign every denial. The test is the one Robodebt failed too: which decisions need a person, which person, to what standard, and how you’ll prove it.
Settle it before launch, and a person is actually deciding. Leave it, and all you have is a doctor, a signature and 1.2 seconds.
If you run a system that approves, denies or flags people, try this: pick one decision it makes and ask who opens the file before it goes out, and how you’d prove they did. If the answer is ‘nobody’ or ‘we couldn’t’, that boundary hasn’t been drawn yet. That’s the work I do: mattsheehan@spatialnext.io
I keep reading articles about AI and having a “Human-in-the-loop” (HITL) versus automation with no human. At HUMANX in Amsterdam recently one speaker discussed how HITL in practice means asking a person to watch a sophisticated system work and click “approve”. That is what I call the rubber stamp trap. But that same speaker then went on to say: humans should not be the guardrails, that we should let the technology provide the security, controls and boundaries, and let people provide the goals, the judgement and the humanity. That sounds good on paper, letting the technology hold the boundaries, but there is one major flaw here:
It assumes the boundaries already exist.
The Case of Robodebt
On 23 June 2026, Australia’s Federal Court approved a A$548.5 million settlement for people wrongly chased for welfare debts by a government computer. It is the largest class action settlement in Australian legal history. It comes on top of A$112 million paid out in 2021, and payments will run until 2028.
The scheme was called Robodebt. From July 2016, it compared what people had told Centrelink they earned every two weeks with their annual income on record at the tax office. The computer spread that annual figure evenly across the year, as if the person had earned the same amount every fortnight. Wherever that average was higher than what they had reported, it assumed they had been overpaid benefits and sent a debt notice.
But, casual workers, students and seasonal workers earn money in bursts. Two weeks of earnings do not reflect a years worth of earnings. A fundamental flaw in Robodebt. It was found that around 470,000 debts had been wrongly raised. The Royal Commission called the scheme a “crude and cruel mechanism” that was “neither fair nor legal”.
Robodebt is usually told as a story about a bad algorithm. But this is far more a story about a defined boundary and automation versus oversight by a human with authority.
Three phases, three answers
Before 2016 & Robodebt: People in the loop, unable to keep up. The department was struggling finding around 300,000 income discrepancies a year. Staff had the capacity to investigate around 20,000 of the highest-risk ones, gathering payslips and evidence before making any decisions. The checks were careful. They were also slow, and most discrepancies were never looked at.
2016: Robodebt automation or removing the human. The Online Compliance Intervention removed the manual step. The aim was to raise as many debts in a week as staff had previously raised in a year. Notices went out without anyone gathering evidence, and the burden of proof moved to the recipient. If a recipient disagreed, they had to find their own payslips, sometimes from years earlier. The Royal Commission found that removing human intervention was “a key factor in the harm Robodebt did”.
After the Robodebt backlash: People put back. The Commonwealth Ombudsman called for clearer notices and help for people to gather evidence, and a Senate committee recommended restoring manual checking based on actual fortnightly earnings. In November 2019, after conceding in court that income averaging was unlawful, the government stopped raising debts without first gathering evidence such as payslips. The Royal Commission’s recommendations on automation focused on a clear path for people to seek review, and on transparency about how automated decisions are made.
Robodebt ended up being a hugely expensive mistake, causing considerable distress to those wrongly sent a debt notice. The intention behind its introduction was, on paper good – speed and efficiency, but its application was a disaster.
And one we are repeating with the emergence of AI.
The answer we’re hearing now
Returning to the comment made at HumanX in Amsterdam. A HITL who simply approves the output from AI is not oversight; it’s a rubber stamp. A 2021 study from the University of Washington and Microsoft Research found this isn’t solved by better AI explanations: explanations made people more likely to accept the AI’s answer, whether it was right or wrong. Robodebt’s pre-2016 staff show the other problem; that careful human checking simply does not scale.
But suggesting we solve that by “letting the technology hold the boundaries”, or by taking humans out of the loop entirely, as Robodebt did, assumes the boundaries we need already exist: a line drawn in advance between what the system may decide on its own and what must go to a person with the authority to stop it.
It is fair to say that the Robodebt technology did have a boundary. That boundary was the gap between averaged income and reported income. The rule it enforced was simple: any gap was a debt. And the system applied that rule perfectly, hundreds of thousands of times.
The problem was the rule.
Nobody had decided, before the system went live, which of its outputs it was allowed to act on alone. Taking people out didn’t answer that question. Putting them back didn’t answer it either. Handing the boundary to technology won’t answer it either, because technology can only enforce a line somebody has already drawn.
The step everyone skips
So what broke Robodebt and invalidates the view shared at HumanX?
This is what I call the AI Decision Spine Model:
AI Decision Spine Model
Deciding, in writing and before any live case, which decisions an AI’s output may close on its own (automated execution), which needs a named person working to a defined standard (human judgement), and which are never delegated.
Run Robodebt through this and the design almost writes itself. Imagine Sarah, a university student. She receives payments for nine months while studying and correctly reports earning nothing. Over the summer she comes off payments and works full time in a café, earning $12,000.
Comparing the two income records? That’s arithmetic. Let the machine do it. The tax office says Sarah earned $12,000 last year. Centrelink has her reporting nothing. The machine spots the difference in a second.
Spotting a gap worth a closer look? Also arithmetic. Let the machine do it. A $12,000 gap is well over any sensible threshold, so Sarah’s case is flagged for checking.
Working out what someone actually earned? That needs evidence. If the café’s payroll is digital, the machine can pull the records and see that every dollar was earned in summer, when Sarah wasn’t on payments. If it isn’t, a person asks Sarah for her payslips.
Telling someone they owe the government money? That decision has consequences for a real person. A gap in averaged income opens up a question, but it can never, on its own, become a debt. Robodebt spread Sarah’s $12,000 evenly across the year, that is around $460 a fortnight, including the months she was studying. Robodebt would have decided she owed money. The evidence shows she owes nothing.
That last rule is the spine. Once it’s written down, the technology finally has something real to enforce: no debt notice without proof of what the person actually earned, and no debt in any unclear or disputed case unless a named officer signs it off.
People no longer check every case, as they tried to before 2016, or no case at all, as under Robodebt. They make the one call that needs them, on the cases that need it.
This also solves the capacity problem that tempted the department to automate in the first place. Before 2016, staff spent their limited hours on the whole investigation. In this design, the system does the matching and the evidence gathering, and people handle only the adverse decisions the data can’t settle. That’s far fewer than 300,000 a year, and each one gets real attention rather than a click.
Neither “keep the humans in” nor “take them out” gets you there. Both treat the loop as a single on/off setting. The real question is which decisions need a person, and which don’t.
The law is starting to draw the line
Some legislators are now drawing this line themselves. The clearest examples are in US health insurance.
Since 1 October 2026, Alabama has required that any decision to deny, delay or modify a prior-authorisation request on medical-necessity grounds is made or reviewed by a licensed physician or other qualified health care professional. Iowa’s and Washington’s versions are already in force, and Colorado, Georgia and Minnesota follow on 1 January 2027.
AI may assist. It may not be the sole basis for a denial.
Look at the nature of that rule. An approval can be automated. A denial, the adverse decision against a person, cannot. That is a spine, written into statute. It’s the line Robodebt never drew.
California’s ADMT rules, from 1 January, ask the same question of AI used in hiring and other employment decisions.
What the laws don’t do is finish the job. They say who must make the denial. They don’t say which cases the AI can send straight to approval, what standard the clinician applies when they disagree with it, or how an insurer shows the review was real rather than a click. Each insurer has to design that for themselves, and the sensible time to do it is before the deadline, not after the first complaint.
Otherwise they end up where Robodebt ended up after 2019: a rule about who is allowed to decide, with nobody having designed what deciding actually involves.
Not in or out, but where
Robodebt has cost Australia more than half a billion dollars in this year’s settlement alone, years of public trust, and far worse for some of the families involved. Nobody chose that outcome. It came from a question nobody asked:
What was this system’s output allowed to decide on its own?
“Human in the loop” and “human out of the loop” both answer the wrong question. The right one is simpler: which decisions need a person, which person, and to what standard? Answer that before the system goes live, and technology can hold the boundary you chose. Skip it, and the system sets its own boundary by default. Robodebt’s default was “any gap is a debt”, and it stayed in place, despite years of warnings, until 470,000 debts had been wrongly raised.
If you’re about to put an AI-influenced decision into production, or a regulator has just put a date on one you already run, that’s the question the Decision Spine Diagnostic maps — one workflow, in writing, in two weeks. If it would help to talk it through, I offer a free 30-minute conversation: mattsheehan@spatialnext.io.
Nature has not been kind to Nepal over the last few months.
On 16 August 2024, a rock avalanche struck a small glacial lake above the village of Thame, in Nepal’s Everest region. It burst into a second lake below, releasing 460,000 cubic metres of water and debris. The flood reached the village in 22 to 32 minutes. It destroyed a school, a health post, five hotels, seven homes and a hydropower plant. Around 135 people were displaced. Losses ran to $6.18 million. Thankfully, no one died.
In the region Imja Tsho, is called the “doomsday lake”. Thanks to an injection of $3.5 million from the UN a drainage project was started; the Nepal Army team lowered the lake by 3.5 metres from 2016, and sirens went up to warn six villages and the 60,000 tourists who pass through each year if there was a flooding threat.
But Imja Tsho was not the lake that flooded Thame.
They were two different lakes; too small to be named on the maps scientists use to track glacial risk. They had never been assessed.
They were literally off the map – so nobody had been asked to watch them.
Not a bad prediction
It is tempting to call the flooding of Thame a monitoring failure. That would presume that someone assessed these lakes and got it wrong. But that is not what happened. In reality nobody assessed them at all. They sat outside the list of what counted as a risk ..
and nobody had gone back to ask whether that list was still up to date.
A bad prediction means the analysis was done and came out wrong. Here, the analysis never took place.
The lake on the list isn’t much better off
Money was used to mitigate the risk of Imja Tsho. But, years on, the sirens that were installed are rusting. Villagers in Dingboche report batteries stolen from the towers. The monitoring agency describes its own satellite data reception as unreliable. In April 2026, a senior meteorologist at Nepal’s Department of Hydrology and Meteorology told reporters: “We cannot say for sure if the early warning sirens work.”
So Nepal is failing in two ways at once.
The work done in 2016 on Imja Tsho was left to decay. It was never revisited, never re-funded, never re-inspected. Secondly, the other two lakes. They were ignored, because the thing that needed watching didn’t look dangerous enough until the day it was.
Don’t blame the mountain
When something like this happens, the instinct is to blame the terrain or the climate. When an AI denies a claim or flags the wrong person, the same instinct kicks in, to blame the algorithm instead of whoever decided how far to trust it.
In September, FTC Chair Andrew Ferguson pushed back on exactly this at the Reuters Momentum AI conference in Austin. He said he would keep resisting the habit of talking about AI tools as if they were people: “If someone tells a tool to do something, and the tool does it, I don’t think we would say, ‘Oh, what do we do about the tool?'” In other words, the responsibility sits with whoever gave the instruction.
He was talking about software, but the same goes for a lake. A lake doesn’t choose whether it’s watched. People do, and not watching it is a choice too.
The same is true inside any organisation using AI. Someone decides which of its outputs a person checks, and which go straight through. Until now, most organisations have never had to say who made that call. In California, that’s about to change.
California has just made this a legal question
From 1 January 2027, California’s rules on automated decision-making technology (ADMT) apply to any business using AI or automated tools to make significant decisions about people. In practice that mostly means employers: decisions on promotion, pay, discipline and termination.
The rules are centred on the same question as Thame. For each of those decisions, is a human really involved? The regulation sets a clear test. The reviewer must know how to interpret and use the system’s output, look at the other relevant information, and have the authority to change the decision. If that isn’t in place, the business must offer an appeal to someone who meets the test, or let people opt out and explain in advance how the decision will be made instead.
What the rules don’t tell you is which decisions need that person, who it should be, or on what basis they can overturn the AI. Each business has to work that out for itself. It’s the equivalent of Nepal’s list of lakes worth watching: a list of which AI decisions need a person, and which person. And like Imja Tsho, it can’t be drawn once and forgotten. Every new system deployed after 1 January has to meet the test from its first day.
So in California, the useful question to ask before January isn’t what to build or buy. It’s whether you can point to your own version of that list: which decisions your AI influences, who reviews each one and with what authority, who decided that, and when it was last checked. A system with excellent sensors, watching the wrong list, will pass every audit right up until the thing that was never on it breaks.
Drawing that list is what the Decision Spine Diagnostic does: one workflow, in writing, in two weeks. If you’d like to talk it through, I’m at mattsheehan@spatialnext.io.
1. Australia Just Paid A$548 Million for Robodebt. It Wasn’t AI. The Lesson… Source: Matt Sheehan, SpatialNext (LinkedIn), October 2026 Summary: Robodebt happened in three phases: people checking and overwhelmed, people taken out, people put back. None of the three ever decided which decisions the system could make on its own. The piece walks through how a student with a summer job would have been treated if that line had been drawn first. Link: https://www.linkedin.com/pulse/australia-just-paid-a548-million-robodebt-wasnt-ai-lesson-sheehan-qgtzc/
2. Using new imaging technologies to monitor the Common Agricultural Policy (Special Report 04/2020) Source: European Court of Auditors, 2020 Summary: This is the report that set the scene for the system in this week’s newsletter. The auditors found that satellite “checks by monitoring” could cover every farmer instead of a small sample, and let farmers correct claims before they were finalised. They also flagged the weak spot: many parcels, especially small plots and grazing land, can’t be settled by imagery alone, so someone has to decide what happens next. Link:https://op.europa.eu/webpub/eca/special-reports/new-tech-in-agri-monitoring-4-2020/en/index.html
3. Xenophobic Machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal Source: Amnesty International, October 2021 Summary: The Dutch tax authority’s risk algorithm flagged families for possible fraud, and flagged families had their benefits suspended. The civil servants reviewing the flags were given no explanation for the scores. Tens of thousands of parents were wrongly accused, and the government resigned. It’s the clearest case of a flag carrying a financial consequence before anyone had properly looked. Link:https://www.amnesty.org/en/latest/news/2021/10/xenophobic-machines-dutch-child-benefit-scandal/
4. The flaws of policies requiring human oversight of government algorithms Source: Ben Green, Computer Law & Security Review, 2022 Summary: Green reviewed 41 policies that require a human to oversee government algorithms. He found that people often can’t provide the oversight these policies assume, and that the requirement can make flawed systems look legitimate. His answer is to make agencies justify, with evidence and before adoption, both the algorithm and the human role. That’s close to your argument for drawing the line before launch. Link:https://arxiv.org/abs/2109.05067
If your organisation runs a system that flags, scores or approves, and you’re not sure where its line sits, that’s the work I do. I help teams decide, before launch, which decisions the system can make alone and which need a person. Get in touch at mattsheehan@spatialnext.io.
A woman and four children, the youngest was only six years old, were placed face down on a hot pavement in an Aurora, Colorado car park. Their hands were zip-tied behind their backs, while officers checked a licence plate that had matched a “stolen vehicle”. The plate number was correct. But everything else about the vehicle was wrong – a different make, a different colour, registered in a different state to the stolen motorcycle the system was hunting. But none of that prevented the stop.
By the time anyone checked that only the number plate matched, the children had already been handcuffed.
But here is the part that is so often missed each time this story is told: the plate reader was not broken. It did in fact do exactly what it was built to do: identify a number match which it then passed on. In other words, the system worked.
What failed was something that was never built at all: a rule saying a number match alone is not enough to justify a stop. Nobody had decided, in advance, what a partial match was allowed to trigger and what it wasn’t. So the decision was made by default, in the moment.
This is not a technology failure. It is a missing boundary. And the fix itself is surprisingly simple. Most look to better data, better AI, better governance.
And they are all looking in all the wrong places.
The Six-Step Diagnostic
I run a six-step diagnostic with organisations to find exactly this kind of gap before it produces a scenario like Aurora. Let me to show you what it would have found here, step by step, using what’s been made public about this case.
Step 1 — Redesign the workflow. Before touching the AI, map what the officer’s decision actually was, working from what they already know how to do. An experienced officer doesn’t stop a vehicle on a plate number alone; they are trained to cross-check make, colour, state, before they act. The AI’s job should have been to do that cross-check automatically and present a single (verified) answer, not to hand the officer a raw license plate number match for the police to go into arrest mode.
Step 2 — Decision inventory. List every decision this workflow actually contains. Not one decision: “is this the stolen vehicle”, but several. That might include: does the plate match, does the vehicle description match, does the state match, and only after all three, what level of response is warranted. Aurora treated this as one decision, where it reality it is actually four.
Step 3 — Measurement vs. judgement. Of those four, which are things a system can simply measure (automation), and which require a person to weigh-up consequence (judgement)? Plate-number matching is measurement: a database lookup, no judgement needed. Vehicle-description matching is also measurement: the system had the data to check make, colour, and state, and didn’t. What to do about a partial match, with children in the vehicle, in public, is judgement, and it’s the one part of this that genuinely needed a human standard, not just a human present.
Step 4 — The decision boundary. Here’s the rule that was missing: a plate match alone doesn’t authorise a felony stop. All three elements: plate, description, and state, have to agree before the system treats the vehicle as the one it’s looking for. If all three agree, the system executes the standard stop protocol. If only the plate matches, and the other two elements contradict it, the case routes to a named supervisor, and no stop happens without their sign-off. Writing that boundary is not complex. But it absolutely has to happen before any system goes live, and not improvised by an officer at the roadside, under pressure, with the decision already half-made for them (by the AI).
Step 5 — The authority map. Who is actually allowed to override a match that clears the plate check but fails the rest? Not “any officer,” which is what effectively existed here. That override authority needs to be a named role, with a defined standard for how much disagreement between the plate, the vehicle description, and the registration state is enough to stop a match being treated as confirmed, and clear authority to downgrade or cancel the stop before units are committed.
Step 6 — Failure-mode stress test. Before deployment, ask three questions. What happens when the plate matches but the vehicle description and registration state don’t? What happens when the system reports a match with high confidence and that confidence turns out to be wrong? What happens when time pressure pushes an officer to act on the plate match alone, before the other two checks come back? Each of those questions has a clear, writable answer. In Aurora, none of them had been written down before the stop happened.
As I mentioned, none of this requires new technology. Aurora’s system already had the data to check vehicle type and state, it simply wasn’t required to before triggering a stop. The gap wasn’t in what the AI could do.
It was in the decision nobody had made about what the AI’s output was allowed to authorise on its own.
Run those six steps before a system like this goes live, and here’s what would have happened instead: the plate matches, the description doesn’t, the system holds, a supervisor gets a flag instead of officers getting a ‘hit,’ and a family safely park their car instead of being forced to be face down on the pavement. No distress, no resulting court cases.
That is exactly what the boundary is actually for.
This isn’t an Aurora problem, or even a policing problem. It’s what happens anywhere an AI system gets trusted to decide when its own output is good enough. I’ve written about many similar cases in insurance, hiring, lending, healthcare, aviation, and more. The instinct everywhere is to fix it with more data, better models, tighter confidence thresholds. None of that closes the gap, because the gap was never about accuracy. It was about a decision nobody made in advance. The fix is not rocket science. It is six questions, documenmted, before the system goes live. Simple, sensible, obvious .. which is exactly why it keeps getting skipped.
Follow-up
If you have an AI workflow in production or pilot, ask your team one question: what is this system explicitly authorised to do below a certain confidence score?
If you can’t get a written answer in ten minutes, the boundary doesn’t exist yet — it just hasn’t met the wrong case yet.
In June 2026, the ACLU filed suit on behalf of Robert Dillon, a Florida man arrested after a facial recognition system returned a 93 percent confidence match against a photo purported to be him. Ninety-three percent is pretty decisive. Dillon was arrested and spent a night in jail.
For a crime he didn’t commit.
Nothing about this failure was a modelling problem. The facial recognition system worked as designed, it returned candidates, ranked by similarity. What was missing was a decision no one had made in advance: at what point does a confidence score become an action, and who is accountable for drawing that line?
That gap – not the models accuracy, nor the data quality – is what the AI Decision Spine Model is built to close.
Six stages, one fork
The diagram above walks through each of the AI maturity stages leading to the AI Decision Spine Model. Most organisations move through the first three stages without much friction.
Stage 0 to Stage 1 is digital transformation — consolidating data into one connected digital environment.
Stage 1 to Stage 2 adding an AI assistant (LLM) to that environment.
Stage 2 to Stage 3 moving to the rapidly advancing AI and data layers: grounding, automation, and AI-augmented human judgment. This layer includes simulation and world models.
Stage 3.5 is the trap, this is the rubber-stamp state; a human is nominally “in the loop” approving the AI’s output, but with no real standard, time, or standing to say no. It’s more dangerous than having no human at all, because it looks like oversight but functions as theatre. In Aurora: a plate-reader system returned an exact number match to a stolen vehicle — but nobody had a standard requiring the vehicle itself to be checked too, so a family was stopped and held at gunpoint, mistaken for a motorcycle in a different state.
Stage 3 to Stage 4 is the entry point to the AI Decision Spine Model. Stage 4 is centred on Decision Authority, it has 2 parts. Firstly, the spine; this defines the decision boundary (think of this as the law) written in advance, before any live case, setting out which decisions an AI’s output may settle on its own, which need a named human’s approval, and which are never delegated at all. Secondly, the guardrail; this applies the boundary to each live case (think of this as the judge): the mechanism that checks a specific decision against the spine and routes it – automate, escalate, or block – rather than trusting the AI’s confidence score to decide that for itself.
Stage 4 to Stage 5 is Decision Execution. Guardrail enforcement splits this between automation, where a decision has cleared the spine cleanly enough that the system can act on its own, no human required and augmentation, where the guardrail routes the decision to a named human instead, because it falls short of the automation bar and needs their judgement before it becomes an action.
The Importance of Stages 4 and 5 – and dangers of Stage 3.5
Stage 3.5 “Human in the loop” (HITL) is usually offered as the answer to AI risk. In reality it is most often the mechanism that hides it.
The academic term for what happens to that human is the “moral crumple zone,” coined by researcher Madeleine Clare Elish in a 2019 paper on accountability in automated systems. Just as a car’s crumple zone is engineered to absorb the physical force of a collision, a human placed nominally “in the loop” of an automated system can end up engineered – whether anyone intends it or not – to absorb the moral and legal force of its failures. The system gets to claim it has oversight. The human gets the blame when the oversight turns out to have been theoretical. Elish’s own case studies are aviation automation accidents and driverless-car incidents where a human was technically in control and functionally unable to be.
The clearest documented example is Uber’s autonomous test vehicle in Tempe, Arizona, in March 2018, which struck and killed a pedestrian, Elaine Herzberg. There was a human in the loop: a safety driver, present specifically to intervene if the system failed. The National Transportation Safety Board’s final report found the vehicle’s sensors detected Herzberg six seconds before impact and never braked, because Uber’s engineers had suppressed emergency braking to cut down on false-alarm stops, that was a design choice that quietly handed the actual decision back to the human. The NTSB named the underlying failure directly: Uber had built “inadequate mechanisms for addressing operators’ automation complacency.” The driver wasn’t reckless. She was doing exactly what a human put in that role for that many hours, watching a system that rarely needs intervening on, predictably does — she stopped actively monitoring it. The role was designed to fail this way; it just took time for it to.
This isn’t a one-off human failing. It’s what happens by default whenever “human in the loop” is treated as a design decision rather than a designed decision. A 2024 Harvard Business School study of 228 evaluators found reviewers were 19 percentage points more likely to defer to an AI’s recommendation when it came with an explanation, and a further 5 points more likely when that explanation was narrative rather than a bare score — meaning the more persuasive an AI system sounds, the less real scrutiny a human gives it. And the underlying constraint isn’t fixable by training or willpower: Caltech researchers Zheng and Meister found human conscious thought processes information at roughly 10 bits per second, against sensory intake running around a billion bits per second. Ask a person to meaningfully review AI output arriving faster than that gap allows, and there are only two outcomes: the organisation slows down and loses the speed it built the AI system to gain, or the human review becomes a formality that exists to be passed. Almost every organisation, under that pressure, ends up with the second without ever deciding to.
That’s the HITL cop-out: naming a human as the control without asking whether that human has the standard, the time, or the standing to actually exercise it. It satisfies a checklist. It survives an audit, until the audit asks the right question. And it has now caused fatalities (Uber/Herzberg), wrongful arrests at gunpoint (Aurora), and a teenager handcuffed over a bag of Doritos (Baltimore County) — different industries, different systems, the same structural gap. Stage 4 exists specifically to close it: not by adding another human to the loop, but by deciding, in writing, before the live case arrives, exactly what that human is there to do and what authority they actually have to do it.
The regulators just caught up
Two governments have now written this exact distinction into law, arriving at it independently, from liability law rather than systems design.
China’s Cyberspace Administration, NDRC, and MIIT jointly issued the Implementation Opinions on Intelligent Agents, effective 15 July 2026, the first national policy treating AI agents as their own regulated category. Article 6 requires that before an agent is deployed, its decisions need to be sorted into the same three tiers as the spine: human-only, approval-required, autonomous.
That is Stage 4, legislated.
Article 7 goes further, and it’s the more useful part. Chinese legal commentary has summarised the liability principle as “look at control, not code”, responsibility falling on whoever controlled the agent’s behaviour, not whoever built it. But control can only be judged after the fact if there is a record of which tier a decision fell into and whether approval actually happened. A tiering policy with no such trail is, in the regulation’s own framing, unreconstructable, so paper, not proof. That’s Article 7 legislating against exactly the HITL cop-out: a declared boundary that nobody can show actually held.
Illinois answered the same pressure differently. On 6 July 2026, Governor JB Pritzker signed the Artificial Intelligence Safety Measures Act, making Illinois the first US state to mandate annual independent third-party audits of frontier AI developers with audit fees structured so they can’t be tied to the findings, just to keep the auditor honest.
So we have different subjects, and different mechanisms. One tiers and traces an agent’s own behaviour, the other checks a developer’s safety practices from the outside. But there is a shared demand: a policy is not evidence. Only a record, audited or traceable, closes that gap.
Which is, in miniature, the entire case for Stage 4.
Follow-up
If you have an AI workflow in production or pilot, ask your team one question: “What is this system explicitly authorised to do below a certain confidence score?”
If you can’t find that answer in writing within ten minutes, the boundary doesn’t exist yet — it just hasn’t been caught out by the wrong case yet.
“One Out of Three Checks Matched. The AI Authorised the Stop Anyway.” — Matt Sheehan, LinkedIn, September 21, 2026. A licence-plate match felt conclusive enough to authorise an armed stop on a family. It wasn’t — the same failure this newsletter is making about evidence, playing out as a decision-boundary problem instead. Read it here
“You’re Not Doing Customer Validation. Stop Calling It That.” — Paul O’Brien, June 30, 2026. Draws the line between customer validation and market validation, and lands the point in one sentence: “‘Sounds interesting’ is the most useless signal in the history of entrepreneurship.”Read it here
“The Validation Trap: Why Early Positive Feedback Kills More Startups Than It Saves.” — Medium, April 10, 2026. A sharper edge on the same failure: founders confirm desire and stop there, never testing whether the thing can actually survive contact with the harder constraints. Read it here
“Focus On What Customers Do, Not What They Say.” — Startup Strategies (Substack). The plainest version of the same argument: people are nice, and being nice isn’t evidence. Paying is. Read it here
This round-up runs every week alongside Decision Layer Weekly — sources on AI decision authority that keep independently landing in the same place. Get it in your inbox → https://genaimarketers.kit.com/03d60cdef4
A machine produced a signal. That signal became an action. Somewhere in between, the part that should have decided whether the signal was enough was missing.
Today, I want to show you where that missing part sits.
Every AI system that touches a real-world decision has three layers. And most organisations build only two.
Layer 1: Data/Grounding Layer
This is the evidence.
In Aurora: a licence-plate number, a registration state, a vehicle description. In a claims workflow: photographs, weather data, repair estimates. In a control room: sensor readings, forecasts, asset condition.
These are facts, signals, and observations. They are the decision’s raw materials.
Layer 2: AI Layer
AI examines the evidence and returns an assessment:
“This appears to be a match.” “This claim may be fraudulent.” “This circuit may be at elevated risk.” “Here is my confidence level.”
That can be extremely useful.
But an assessment is not a decision.
Nobody would allow a detective to handcuff someone purely because they had a hunch. The detective gathers evidence and forms a view. Another process decides what that view is allowed to authorise. And in too many AI deployments, that process does not exist.
The model’s conclusion quietly becomes the action because nobody designed a place for it to go in between.
There is a missing layer, Layer 3 or the decision layer.
The courtroom
I like to fall back on analogies to help colour the picture. You can think of the decision layer as the courtroom process between an AI assessment and a real-world action.
That layer has three parts:
1. The spine: the law
Think of this as the law. Defined (designed) in advance. In the case of the decision process it defines:
What evidence is sufficient for a particular action.
Which discrepancies must stop or slow the process.
Which decisions can be automated.
Which require verification.
Which always require accountable human judgment, regardless of a model’s confidence.
Who has standing to override, escalate, or halt the process.
In Aurora, the spine would have specified that a plate match alone could never authorise a high-consequence stop — any mismatch in vehicle type or state required a person to verify it directly, and no action could proceed until they had.
The law is not written while the case is unfolding. It is written beforehand. In the case of building the decision spine, this is done by people who understand the workflow, its failure modes, and the consequence of being wrong.
2. The guardrail: the judge
Once the spine exists, a guardrail or the judge using our court room analogy, enforces it.
It checks a proposed action against the boundary already defined: allow, block, route for more evidence, or escalate to an authorised person.
The judge does not decide whether the law is wise. The judge holds it, consistently, case after case.
This is where much of the excellent engineering in AI safety, policy enforcement, and workflow orchestration belongs. But a judge can only enforce a law someone has actually written.
3. The automation: the bailiff
Only then comes automation.
The bailiff carries out an action that has already been authorised: send the customer notification, route the claim, log the event, dispatch the work order, or trigger an approved response.
It is safe to move quickly precisely because the judgment about when it may act was made upstream.
Where systems break
Aurora and Baltimore did not lack data. Nor did they lack technology capable of detecting a plate match or flagging a possible weapon.
They lacked a decision spine.
No one had clearly defined what a single match, a conflicting signal, or an uncertain classification was allowed to authorise. The result was predictable: a model output travelled directly from detection to action, with a person nominally present but no meaningful decision process for them to apply.
A courtroom with evidence and enforcement but no law is not a courtroom.
The work before AI goes live
This is not primarily a model problem.
The model can improve. The data can improve. The guardrail can become more sophisticated.
But none of those answers the first question:
Which decisions should never be closed by an AI’s confidence alone, however strong the case appears?
That question has to be designed into the operating process before the system goes live—not discovered in front of a customer, an operator, a regulator, or a jury.
Follow-Up
If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”
If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.
If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io
In 2023, a police algorithm in Aurora, Colorado matched a licence plate to a stolen vehicle. Only one of three checks agreed — the plate. The state and the vehicle description didn’t. A patrol car pulled the vehicle over at gunpoint.
It was the wrong car.
In October 2025, an AI-enhanced surveillance camera at a Baltimore County high school flagged a crumpled Doritos bag in a student’s pocket as a gun. Within moments, police arrived, drew their weapons, and forced 16-year-old Taki Allen to his knees.
It was a bag of chips.
Two different systems. Two different cities. Two different technologies. But the same failure, in the same place, both times:
A machine produced a signal, nobody had decided in advance what that signal was allowed to authorise on its own, and armed officers acted on it as though someone had.
That is not a coincidence, and it’s not a technology problem. It’s a design problem, and it’s one I’ve spent the past several years working on.
The gap people are starting to name
Others have begun to talk about this. The language itself varies: “decision ownership,” “authority gap,” “judgment gap”, but the underlying observation is becoming common:
Organisations are using AI and computer technology to automate decisions before they have clearly defined which decisions can, and which decisions cannot, be automated.
It is all well and good pointing out the gap, but quite another thing to do something about it.
Three parts, and the one everyone skips
With the arrival of AI too many have rushed in and implemented an AI powered solution. Many have failed.
Why?
Let’s step back for a moment. I think about AI decision systems as having three parts.
The spine. This is the blueprint of any decision process. It defines the boundaries. Let’s use Aurora to help illustrate. If the number plate, state and vehicle type match – 3-of-3 – automate the process. Any other condition needs human oversight. Using an analogy – this is the boundary or the fence line.
The guardrail. This is the infrastructure that holds and enforces a boundary once it’s set — checking a proposed action against a rule, letting it through, blocking it, or escalating it. Extending the analogy, this is putting up the fence and defining what sits inside (automation) and what sits outside (human judgment) that fence. Some of the best engineering happening in AI right now lives here. A number of innovative companies are now solving this problem.
The automation. This is the work that runs inside that boundary (or fence using our analogy) once it’s marked – fast, repeatable, and safe precisely because someone already decided it was safe.
Neither Aurora nor Baltimore lacked a guardrail. Both systems had rules, checks, thresholds — the plate-matching logic ran, the image classifier ran.
What neither had was a spine:
Nobody had decided, ahead of time, that a one-of-three match, or a single flagged image with no second signal, wasn’t enough to authorise an armed response on its own.
Where the spine comes from
This is the part that is easy to miss: the spine isn’t a smarter algorithm, and it isn’t a stricter rule bolted onto the existing process.
It comes from designing the decision process itself.
That does not mean documenting how the process works today, but working through, with someone – a subject matter expert or SME – who actually knows the workflow and its failure modes, what the process should be now that AI can do things the old process couldn’t.
From there designing the decision process itself – think of this as a set of instructions.
Much of my thinking here came from aviation’s model. Before a new aircraft type ever flies, engineers sit down and build a checklist specific to that plane; critically that is not one copied from a different aircraft. Most items on that checklist can be legitimately cleared with more evidence: a second sensor confirms a reading, another source resolves it.
But some can’t.
A crack in a wing strut doesn’t get cleared by a second scan, however clean it comes back. It gets cleared by an engineer physically inspecting it, no matter what any recheck shows.
And this needs clearly defining, before the aircraft ever takes off.
Aviation didn’t arrive at that distinction by accident. It was built deliberately, item by item, because the cost of getting it wrong is measured in lives. Almost no other industry has to do the same work, not because the stakes are always lower, but because nobody has forced the reckoning the way flight safety did decades ago.
AI is forcing that reckoning now.
The practice
This is at the heart of the work I have been exploring at SpatialNext:
Sitting down with a subject-matter expert (SME) who knows a given decision process — its steps, its failure history, where a system’s confidence shouldn’t be trusted alone.
Next, working with the SME to redesign the decision process itself, given what’s now possible, rather than digitising the workflows.
Lastly, classifying what comes out of that redesign: which decisions are safe to hand to evidence and automation, and which must always end with a person, regardless of how strong the automated case looks.
SpatialNext is focused on spine building — defining that boundary. This is what a guardrail then enforces, and what automation then runs inside of. Not a technical fix.
A design discipline that’s currently missing in most AI implementations.
Baltimore and Aurora weren’t failures of AI. They were failures of a step nobody took before the AI was ever switched on.
And that is the gap I fill. Not after the stop. But before it.
Follow-Up
If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”
If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.
If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io
