Category: Uncategorized

I found the same failure four times, in four unrelated places, before I noticed it was the same failure.

First in a flight simulator, six years after two planes crashed. Then in a car park in Aurora, Colorado, where a license plate reader matched a family’s SUV to a stolen motorcycle. Then in an Indian courtroom, where a tribunal cited court cases that didn’t exist. Then on a two-lane road in Key Largo, Florida, where a car’s cameras saw a pedestrian 116 feet away and a jury spent weeks deciding who was actually driving.

Four industries. Four decades apart in origin, months apart in publication.

I didn’t build these stories to prove a theory. I found each one separately, wrote about it, then moved on to the next thing that caught my attention.

At some point they stopped looking like four stories and started looking like one.

Connecting the dots

When we implement systems whose output we rely on to make real decisions, we build in safeguards — approval steps, audit trails, sign-offs, a person whose job is to check the work before it goes out the door. Often we call these guardrails, the rules or boundaries. We then put in place governance to check we are staying inside these guardrails. When a human did that work, a lot of the checking happened without anyone planning it — someone paused, glanced back over it, or turned to a colleague and asked, “does this look right to you?”.

AI removed that informal checking. It just executes, which is how a gap that was always there finally became visible.

Readers who have been reading my published articles over the last few weeks will have spotted the pattern: A date, a protagonist, an event where automation went unchecked.

In this article I wanted to stop and connect the dots on why I keep finding this, and why I think it’s about to matter a great deal more than it has so far.

The house

I like to simplify the complex by using analogies. Think about a house.

Every house has a structure – a foundation, core framework, load-bearing walls. Let’s call that the spine. An architect designs the spine first, then builds everything else around it: rooms, partitions, windows etc.

Builders know the order. Foundation first, then everything built off it.

Guardrails and governance map onto this directly. One guardrail might be a wall. Governance is the inspector, walking through after the house building is complete, checking the walls are standing where they should be.

But here is the part most people miss. Not every wall needs to hold weight. A decorative partition doesn’t need to, but it’s still nailed into the frame, top and bottom, even though it is not load bearing.

But a guardrail is not decoration. Like a wall, it has to do two things at once: be nailed to the spine, and actually serve a purpose like separating one room from another — not just stand there.

Too many guardrails today do neither. They stand in open space, not attached to anything, not marking the boundary of anything. They look like a wall. Nobody decided they should be there, or what they were meant to divide.

An inspector can walk straight past a wall like that and tick the box. It looks just like every other wall in the building. Nobody finds out until someone actually tests it.

Let me colour that picture with a real example.

Aurora, in this language

Take the number plate recognition problem in Aurora, Colorado. On paper, the guardrail – rule – was simple: an officer must confirm an automated plate-reader hit before acting on it.

Notice what that policy got right. It named a who. Most organisations never get that far. They build the system and assume someone, somewhere, will catch it if something’s wrong. Aurora didn’t assume. They wrote it down: an officer confirms.

But confirm what, exactly?

The plate numbers had already matched, the machine had already done that part. Confirming the digits again confirms nothing new. Confirming the actual vehicle, the make, model, colour, state, would catch a problem in five seconds:

A Colorado SUV isn’t a Montana motorcycle, whatever the plate says.

Nobody had ever specified which one “confirm” meant. So on a hot Sunday afternoon in August 2020, an officer did the only confirmation the system had made easy; checked what the reader had already told him that the digits matched. He then authorised a felony stop on a woman, her sister, and four children.

The who was named. The what was never decided.

On paper, that word ‘confirm’ checked the box. That meant an officer could check the plate, confirm the digits, and never ask the deeper question: is this the same vehicle.

And that is exactly what a guardrail without a spine looks like.

Let’s step back. Think of the plate-reader program as a wall. All houses need walls. But walls built within what? Imagine a house with no plan; nothing that starts by laying out the spine first: load-bearing walls, foundation, the order everything else has to follow. You can place walls wherever you like when there’s no plan. They attach to nothing substantial. They can exist without serving any purpose at all.

That is a house that is destined to fall down.

Aurora lacked a spine. That meant guardrails like the confirm plate policy simply checked a box. The guardrail was not missing. It was not being ignored. It was not attached to anything load-bearing. So in the end served little practical purpose.

A defence that is going away

For a long time, none of what we have discussed here mattered much, legally. You could build the visible, auditable layer, a policy document, a training slide, a compliance dashboard, and call it done. A written policy that sounds reasonable has generally been enough to satisfy “reasonable care,” even when nothing underneath it actually holds.

That’s not an accident. It’s an incentive. Vendors get funded for shipping the visible layer, because that’s the part a buyer can see and a board can point to. Nobody’s selling the invisible part, in other words the upstream decision about who actually has standing to intervene, and what they’re authorised to do about it. It doesn’t demo well. So it doesn’t get built, and for a long time, nobody paid for skipping it.

That defence is going away.

More than twenty US states have adopted some version of the NAIC’s Model AI Bulletin, requiring a written, governed program for AI-influenced decisions, one that can be shown to actually function, not just exist. The EU AI Act’s human-oversight provisions became enforceable this month. China’s own framework for tiering AI agent authority took effect in July.

Three regulatory bodies, three continents, independently converging on the same question Aurora’s policy never answered: who, specifically, is authorised to do what?

Regulation is only half of it. Every case above already cost someone real money, decided by a court, not a regulator. $243 million against Tesla. $1.9 million against Aurora. Two grounded aircraft and a redesigned certification process. A tribunal ruling thrown out by India’s Supreme Court. Nobody got to stand up in front of a jury and say “we had a policy” and have that be the end of it. The policy existing was never the question. Whether anything real stood behind it was.

As more organisations get this wrong, in public, with a dollar figure attached, “we had a guardrail” stops being a defence and starts being an admission.

Where to actually start

You don’t need a framework to check your own organisation for this. You need one honest walk-through.

Picture your own building. Somewhere inside it, a wall exists that was never actually part of the plan. Someone put it up because a wall seemed sensible there, not because anyone decided what it needed to divide. It looks exactly like every other wall. It just isn’t attached to anything, and it isn’t sealing off anything either.

Now pick the most consequential AI-influenced decision already running in your organisation, and ask, plainly: who – by name, not by job title – has the standing to say this one needs a second look before it acts? And was that decided on purpose, or does it just sound like the kind of thing that should exist?

If you can answer that in one sentence, you likely have a spine, and a wall properly attached to it. If you can’t, you probably have a wall standing in the middle of a room, nailed to nothing, dividing nothing, and it’s only a matter of time before something tests whether it’s actually there for a reason.

Follow-Up

If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”

If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.

If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io

On the evening of April 25, 2019, George McGee was driving his Tesla Model S down Card Sound Road in Key Largo, Florida, with Autopilot engaged.

He dropped his phone and bent down to find it.

The cars autopilot did not stop the car for the stop sign ahead, or the flashing red light beside it. It went through both at roughly 62 miles per hour and struck a parked Chevrolet Tahoe. The impact pushed the Tahoe into two people standing nearby. Naibel Benavides Leon, 22, was killed. Her boyfriend, Dillon Angulo, 26, was severely injured.

Six years later, a Miami jury tried to answer a critical question about the crash:

Who was actually in control of that car?

A verdict that split the difference

On August 1, 2025, the jury delivered its answer, and it was not a clean one.

McGee was 67% responsible. Tesla, 33%.

Twelve people, presented with the full evidence, didn’t put the blame entirely on the man who looked away from the road, and didn’t put it entirely on the system that let him. They split it, and then added something a simple negligence finding doesn’t usually carry:

$200 million in punitive damages against Tesla – meaning the jury believed Tesla knew about the risk and didn’t fix it – on top of $42.6 million in compensatory damages to the victims.

A federal judge upheld the verdict in February 2026, rejecting Tesla’s bid to have it thrown out. In July, Florida’s Attorney General asked the court to erase it entirely. That fight is still live.

The measurement was never attention. It was a proxy for it.

The cars autopilot monitoring system tracked steering wheel torque, in other words how much pressure a driver’s hands were applying. That’s a measurement, and it has a single black and white answer:

Either your hands are exerting force on the wheel, or they aren’t.

Whether a driver is actually watching the road is a different question entirely, and torque doesn’t answer that question. A hand resting on the wheel registers the same as a hand actively steering. McGee’s hands may well have satisfied the system’s measurement in the moments before the crash, even as his eyes were on the floor of the car.

The system was built to measure grip not a judgment about attention – grip and attention aren’t the same thing. They just happened to correlate often enough that the difference went unnoticed until disaster struck.

Three questions the case turns on

What decisions needed making? Before Autopilot shipped, someone needed to define exactly which roads it was fit to operate on, and build the system to enforce that boundary rather than trust the driver to self-select appropriately. Card Sound Road – a two-lane county road with cross-traffic and stop signs – was, according to the plaintiffs’ case, outside the conditions Autopilot was designed to handle safely. But the system engaged anyway.

Where does the system measure, and where does a human judge? Steering torque is a measurement. Whether a driver is paying enough attention to react is a judgement call, and it is the kind two reasonable engineers could draw different conclusions from, looking at the exact same data. Treating torque as a stand-in for attention meant that judgement call never got made at all.

Who has the authority to act on that judgement, and could they actually use it? On paper, McGee held full authority. Autopilot is a driver-assistance system, not autonomous driving, and the driver is meant to stay ready to take over at any moment. The plaintiffs argued something more specific:

That Tesla’s own marketing had described Autopilot’s capabilities in terms that quietly eroded how seriously drivers treated that responsibility.

Authority that exists in the owner’s manual but is undermined by the company’s own messaging is authority weakened at exactly the moment it matters most.

What the verdict actually decided, and what it didn’t

The jury’s 67/33 split isn’t a loophole or an inconsistency. It is an honest reflection of a case where both a human failure and a design failure were real, and where responsibility genuinely didn’t belong entirely to either one. McGee looked away. The system that was supposed to require his attention couldn’t actually verify it was there.

That’s a harder, more useful finding than a full verdict against either party would have been. A full finding against McGee alone would suggest Autopilot’s design had nothing to do with it. A full finding against Tesla alone would suggest human distraction is beside the point once a driver-assist system is engaged. Neither is true, and the jury, which deliberated for weeks, weighing this same tension, made a decision which sat somewhere in between.

The open question

This case is still moving. Tesla is appealing. Florida’s own Attorney General is now asking the court to erase a verdict a jury spent weeks reaching. Whatever happens next will say something real about how much weight the legal system is willing to put on the gap between what a system measures and what it needs to know.

The real question isn’t whether Autopilot failed, or whether McGee did. It’s whether anyone, before this car was ever sold, asked what the system could actually verify about the driver, and built its limits around the honest answer, not the convenient one.

Follow-Up

If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”

If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.

If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io

On August 2, 2020, Brittney Gilliam drove her sister, her 6-year-old daughter, and two nieces to get their nails done in Aurora, Colorado.

The salon was closed. She sat in the car park, looking up another salon.

That’s when the police surrounded the car.

Guns drawn, officers ordered everyone out and onto the pavement. It was 87 degrees that afternoon. Gilliam, her sister, and her 12-year-old niece were handcuffed. The cuffs didn’t fit her 6-year-old daughter, so officers had her lie face-down with her hands raised instead.

They held the family there for nearly two hours.

A license plate reader (ALPR) had flagged Gilliam’s SUV as a stolen vehicle. The plate numbers matched.

And therein lay the problem.

The stolen vehicle was actually a motorcycle. Registered in Montana. Gilliam’s SUV (and plates) were in Colorado.

The measurement was never the problem

The license plate reader did exactly what it was built to do. It read the plate correctly, character for character, and matched it against a police “hot list.”

That is purely a measurement question, and the system got it right.

But what it could not do, something it was never asked to do, was confirm that the vehicle attached to that plate was the vehicle being searched for. A Colorado SUV and a Montana motorcycle sharing a string of digits is not evidence of anything. It is a coincidence a five-second glance from a human would have caught.

Aurora’s own written policy says as much: no action gets taken on ALPR data alone. An officer has to confirm it first.

That is not a measurement problem. It is an authority problem, and one that only existed on paper.

The gap the policy didn’t close

The question that never went answered before this unfortunate incident was: Confirm what, exactly?

“Confirming” the plate number is trivial, that had already been done by the machine. But confirming the vehicle, its make, model, color, state, is a quite different, slower, more deliberate check ..

And it’s the one that would have stopped this distressing incident from ever happening.

Nothing in the record suggests anyone made that distinction explicit. So officers did the only confirmation the system had made easy: they confirmed the digits matched, and treated that as enough to authorize a felony stop.

Critically: A measurement was allowed to stand in for a judgement nobody had actually made.

The consequences

The city settled Gilliam’s lawsuit for $1.9 million in 2024. This was the first case filed under a new Colorado law that stripped officers of qualified immunity; the legal shield that had made it difficult to hold officers personally accountable in court. The district attorney’s office cleared the officers of criminal charges but called what happened “disturbing.” The children, per the family’s attorney, spent years in weekly therapy afterward.

This wasn’t a one-off. This same failure – a plate match mistaken for a vehicle match, has cost San Francisco $495,000 and Oakland $49,500 in separate, unrelated ALPR cases.

Same mechanism. Different departments, different years, same missing step.

The three questions this needed, before that afternoon

What decisions needed making? Somebody needed to write down, in advance, every category of ALPR alert and ask: which of these can trigger action on the match alone, and which require a human to verify something the machine can’t see. This would include whether the vehicle itself, not just its plate, matches what’s being searched for.

Where does the system measure, and where does a human judge? Reading a plate is measurement. It has a right answer, and the system correctly found it. Whether that match is strong enough to justify guns drawn on a family is not measurement. This is a threshold call, and it is exactly the kind of call two reasonable officers, given the same alert, might handle differently.

Treating a plate match as sufficient on its own erased that judgement call entirely.

Who has the authority to act on that judgement, and can they actually use it? Aurora’s policy says a human must confirm before acting. But authority without a defined process is authority in name only. Nobody had specified what confirmation required. So officers confirmed the only thing left to confirm; the plate number, which had already been matched by the reader.

The moment of design

This is not a story about bad technology. The reader worked. This is a story about what happens when a system is trusted to answer a question it was never built to answer, because nobody drew the line between what it could confirm and what a person still had to.

Somewhere in your organisation, right now, a match is being treated as a decision. The honest question isn’t whether a human is technically in the loop. It’s whether anyone has ever specified, in writing, what that human is actually required to check before they act — and whether that specification exists anywhere outside a policy page nobody consulted on a hot Sunday afternoon in August.

Follow-Up

If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”

If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.

If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io

In June 2019, Chesley “Sully” Sullenberger – yes that Sully, the guy who safely landed an aeroplane on the Hudson River in 2009 – sat in a 737 MAX simulator and flew the exact scenario that killed 346 people twice over; the Lion Air and Ethiopian Airlines crashes.

A stick for stick recreation.

He knew what was coming. He knew the failure, the false warnings, the nose-down commands.

Testifying to the House Transportation Committee afterward, he described how close he came to running out of time to react — even with foreknowledge of what was coming. No US pilot, he said, had trained for this exact failure before the crashes.

That’s one of the most experienced pilots in America forewarned, in a simulator, with every advantage a pilot could ask for. He still nearly ran out of time.

The 737 MAX’s automated trim system, called MCAS, pushed the nose down again and again, on the real flights and in the simulator recreation alike. It didn’t fail because its sensors were wrong.

It failed because no one had the authority to override it before it acted.

An industry that already knew the answer

This is what makes Boeing’s failure different from most of the ones I have so far written about. Aviation isn’t a young industry discovering this problem for the first time. It learned it decades ago, the hard way, through a string of crashes in the 1970s and 80s that forced a generation of engineers to build a doctrine most other industries still don’t have:

Automation can assist, but a human must always retain the authority, protected, trained, and real, to override it.

That doctrine is why a plane is one of the most heavily automated machines a human operates, and also one of the few when “the pilot says stop” the system still reliably stops.

Which makes the 737 MAX story harder to explain away as ignorance.

Three layers, one quietly skipped

If you break MCAS down in the way I have broken every AI-adjacent failure down in past articles, you will see a repeating structure.

The sensing layer was a single Angle of Attack (AOA) sensor – a sensor on an aircraft that measures the angle between the wing and the oncoming airflow. Boeing built MCAS to run off a reading from only one of the AOA sensor rather than cross-checking against a second, which meant one faulty vane, with no independent confirmation, was enough to set the whole failure in motion.

The reasoning layer was MCAS itself, and this is the part that should sound familiar to anyone who has read my newsletter. MCAS didn’t just measure an angle. It acted on that measurement, repeatedly, on its own authority. The original design let a single sensor reading trigger the system again and again, nose-down each time, with no limit written in for when the pattern should stop and control should hand back to the crew. A measurement was allowed to become a decision, on a loop, because nobody had drawn the line for where it should end.

The decision layer was supposed to be the pilots. It didn’t exist, because they didn’t know there was anything to decide. Boeing never disclosed MCAS to the pilot community, so the training that would have let a crew recognise and fight it was never built. You cannot exercise protected authority over a system you were never told was flying the plane.

This isn’t an aviation story. It’s what happens whenever a measurement is allowed to become a decision, on a loop, with no human positioned to stop it — and MCAS just happens to be the case where the black box was recovered.

The same three-part failure shows up everywhere I’ve looked. Not sensing, reasoning, and deciding, but a version of it that applies before a single line of automation gets written.

Before you build: three questions

1) What decisions needed making?

Before any of the 737 MAX engineering, there was a decision inventory nobody wrote down. Somewhere in the certification process, someone needed to list out every way MCAS could act on the aeroplane and ask, plainly:

Which of these can the software be trusted to settle on its own, and which of these need a human positioned to catch it if the software is wrong?

That inventory is a design question, not a technology one, and it comes before the software is written, not after it ships.

2) Where does AI measure, and where does a human judge?

An Angle of Attack reading is a measurement. It has a right answer, and a good sensor gets it. That part, correctly, belonged to the machine.

Whether to push the nose down, how far, how many times, and at what point a disagreement between instruments should be handed to the person in the seat …

That is not a measurement. That’s a threshold question, the kind where reasonable engineers, given the same data, could land in different places. MCAS treated it as settled. It wasn’t.

Sullenberger named this exact confusion in his testimony, in language close enough to the framework I use here that it’s worth quoting directly. Boeing, he said, had added a computer-controlled feature to a human-controlled aeroplane without giving it “the integrity, reliability and redundancy that a computer-controlled system requires.”

Integrity and redundancy are what you build once you’ve correctly separated the measurement from the judgement.

Boeing built the measurement. It skipped the judgement.

3) Who has the authority to act on that judgement and can they actually use it?

Even a correctly identified judgement call is worthless without someone positioned to make it. That’s authority, and authority requires three things most organisations quietly skip: the person has to know the system exists, has to be trained on what it does, and has to have protected standing to override it without that override counting against them.

Pilots had none of the three. They didn’t know MCAS existed. The pilots’ union told the same committee that without disclosure, meaningful training on the system had never happened. And because they didn’t know what they were fighting, there was no way to exercise an authority they didn’t know they had.

The fix, and what it rebuilt

Everything the FAA mandated after the grounding maps directly onto the three layers that failed.

1) Sensing was fixed with redundancy: MCAS now requires agreement between both Angle of Attack sensors before it will activate at all. One bad vane can no longer make the decision alone.

2) Reasoning was capped: the system now fires once per event at most, and was redesigned so it can never override a pilot’s ability to control the aircraft through the control column. That is the guardrail turned back into a guardrail; adjustable, bounded, no longer able to overpower the spine.

3) The decision layer was rebuilt from nothing: a new cockpit alert now flags disagreement between the sensors so the uncertainty is visible rather than hidden, and simulator training was made mandatory so a pilot encountering the failure would recognise it and know exactly what authority they held.

Uncertainty made visible. Authority made real.

Protection built in before the system flies again, not after.

The moment of design

Aviation didn’t need to invent the discipline of protected human authority in 2019. It already had it, built over decades, and paid for in an earlier generation of crashes. MCAS is what happens when a mature industry, one that already knows better, lets a single new feature ship without asking the same question it forces every other system in the cockpit to answer.

If an industry with the deepest experience in this exact problem can still forget to ask it, the odds that your organisation is asking it by default are not good.

Somewhere in your organisation, a system is measuring something, deciding something, and acting on it. The honest question is whether anyone, by name, has the standing to say not this one, not like this.

If you can’t answer that in one sentence, you don’t have a gap in your AI strategy. You have an MCAS you haven’t found yet.

Follow-Up

If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”

If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.

If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io

1. “Where is Accountability When Governments Deploy AI?”
Tech Policy Press, March 2026
Names what it calls “the oversight fallacy” — accountability gets attached to visible outcomes, while the decisive judgments are actually made earlier, upstream, in design and authorization. Argues that “human in the loop” only works when a human reviews each discrete decision; it breaks down once systems optimize across many variables without a person present at each step.
https://www.techpolicy.press/where-is-accountability-when-governments-deploy-ai/

2. “Enterprise AI Governance, Part 2”
Trust Insights, August 2026
A sharp warning that human-in-the-loop is becoming “one of the biggest governance illusions in enterprise AI” — if the system decides when to escalate, and the reviewer just rubber-stamps the escalation, and the audit log only captures what the model chose to log, that isn’t oversight. It’s theater.
https://www.trustinsights.ai/blog/2026/08/enterprise-ai-part-2/

3. “Agentic AI at Scale: Redefining Management for a Superhuman Workforce”
MIT Sloan Management Review, 2026
Draws on a global survey of 1,221 executives to argue that governing agentic AI means defining, in advance, where AI can be trusted to act on its own and where human intervention must be reserved for higher-risk decisions — with those boundaries agreed at the senior leadership level, not discovered after something goes wrong.
https://sloanreview.mit.edu/article/agentic-ai-at-scale-redefining-management-for-a-superhuman-workforce/

4. Regulators are starting to mandate the spine directly
On July 15, 2026, China became the first country to legally require AI agent decisions be sorted into three tiers before deployment: human-only, approval-required, and autonomous. The EU AI Act’s high-risk obligations, in force since August 2, 2026, require human oversight and traceability for systems like credit scoring and insurance pricing. Different jurisdictions, same demand: provable, enforced authority over what a system is allowed to do — not just a policy saying it was considered.
https://blog.pebblous.ai/blog/china-ai-agent-decision-tiers/en/

Four different sources, four different angles, one converging signal: the industry is starting to name what I’ve been calling the spine. If you’re already seeing this gap in your own AI deployments, that’s usually where my conversations start: mattsheehan@spatialnext.io

This Week’s Signal Scanner

Four stories that landed this week, each showing the same unanswered authority question in a different room — a hospital billing system, a robotaxi, and two governments trying to legislate the gap before it costs someone their life.

1. AI Is Now Denying Health Insurance Claims With a 90% Error Rate
Source: Outsource Accelerator
UnitedHealth Group faces a federal lawsuit after its naviHealth AI platform had its coverage-denial predictions overturned on appeal nine times out of ten, with a court-ordered compliance deadline now set. A separate finding in the same reporting shows an AI scheduling system produced wait times 33% longer for Black patients — a second, quieter version of the same authority gap.
🔗 https://news.outsourceaccelerator.com/ai-idenying-health-insurance/ Bloomberg LawBloomberg Law

2. US Safety Regulator Investigates Avride Self-Driving Crashes
Source: Zag Daily
NHTSA opened a probe into robotaxi company Avride after a series of Texas crashes where the vehicles changed lanes into other cars’ paths, failed to stop for traffic, and struck stationary objects — all sixteen incidents occurring with a trained safety monitor seated behind the wheel. The monitor was there. The question is why that didn’t matter.
🔗 https://zagdaily.com/connected/us-safety-regulator-investigates-avride-self-driving-crashes/ TechCrunchTechCrunch

3. EU AI Act’s High-Risk Rules Become Enforceable
Source: EVE AI Core
As of August 2, 2026, high-risk AI systems — including credit scoring and insurance pricing — are legally required to build in mechanisms that let a human actually monitor and override the system’s output, not just document that oversight was considered. It’s the closest any government has come to writing the spine question directly into law.
🔗 https://eveaicore.com/blog/ai-regulation-2026-what-changed

4. Meta Sued Over AI-Driven Layoff Selection
Source: Butzel Long
On July 13, 2026, 26 current and former Meta employees sued the company in federal court, alleging Meta used AI systems to select them for layoff in a discriminatory manner. Same question as everywhere else this week: not whether the model was accurate, but who had standing to check its output before it decided someone’s job.
🔗 https://www.butzel.com/alert-ai-on-the-layoff-list-meta-lawsuit-and-workday-rulings-raise-the-stakes-for-algorithmic-employment-decisions

---

Most organisations deploying AI have checked whether the model is accurate. Far fewer have checked whether the human and the machine, together, actually decide better than either would alone — especially when the call is physical, fast, and can’t be undone.

If that question is live in your world, it’s usually where my conversations start: ​mattsheehan@spatialnext.io

Rita Kohli sits as a Judicial Member on the National Company Law Tribunal (NCLT) in Mumbai. Jammu & Kashmir Bank had lent ₹200 crore to a company called Pan India Utilities. A separate company – Essel Infraprojects – had guaranteed that loan, if the borrower couldn’t pay. That put Essel on the hook.

The borrower defaulted on ₹87.43 crore.

The bank came after Essel to collect, asking Kohli’s tribunal to open insolvency proceedings to recover the money. Kohli’s bench sided with the bank – Essel owed the debt. An appeals tribunal agreed the following year.

Then, on July 2, 2026, the Supreme Court set the whole thing aside.

Not because anyone disputed the debt. Because the old court cases the tribunal had leaned on to make its decision didn’t exist. They had been generated by AI, without anyone checking if they were real.

This isn’t a story about legal AI specifically. It’s what happens anywhere a machine hands someone an answer that looks finished enough to skip checking.

Let’s be clear – nobody involved in this case set out to file fake law. The old court case citations read like the real thing because that’s what the AI tool was built to produce. Somewhere between drafting and filing, a plausible-looking citation was trusted without verification.

The Supreme Court did not wave this off as a technical slip. The judges hearing the appeal – Justices P.S. Narasimha and Alok Aradhe – said it went to the heart of whether the courts could be trusted at all, and ordered India’s Bar Council to write new rules for AI use in courts. Their finding was blunt:

A lawyer who cites an AI-generated case without checking it isn’t making an honest mistake. That’s misconduct.

The part everyone will skip past

You will hear the same response every time this happens:

Better AI will catch the fake citations before they cause harm.

No it won’t. Not because the tools won’t improve … they will. The problem is that the tool checking for fakes and the tool making them up are stuck in the same race. By the time a checker gets good enough to catch today’s fakes, the tool generating them has already moved on to fakes that look even more convincing.

AI checking AI will compound the problem.

The fix was never going to be a smarter research tool. It was always going to be a person, named in advance, whose job includes actually reading and validating the citation before it’s filed. Not skimming it. Not trusting that it looks right.

A courtroom has always assumed a human read the law being cited. That assumption held for centuries because a human always had to physically find the case first. AI removed the friction that used to guarantee the check happened.

But it didn’t remove the need for the check.

The court that had the same failure and kept its order anyway

Five months earlier, a different Indian court ran into the identical problem.

A trial judge in Vijayawada was hearing a property dispute. In her ruling, she cited four court cases to back up her decision. All four were made up. When the Andhra Pradesh High Court asked her to explain, she admitted she’d used AI to find the cases, and never checked if they were real.

Same failure. Same tool, doing the same thing.

Justice Ravi Nath Tilhari heard the challenge and asked a simple question: take the fake cases out, does the judge’s actual reasoning still hold up? It did. She had genuinely worked through the legal question herself. The fake citations weren’t holding the decision up. They were just extra decoration sitting on top of one.

That’s why the ruling survived. The mistake was real, but it hadn’t done any of the actual work.

The High Court was blunt about the lesson: AI can help organise and summarise, it said, but it can’t do the judge’s actual reasoning for her. That has to stay a human job, not something AI does while a human nods along.

Kohli’s tribunal didn’t have that same backup. Nothing suggests her bench worked through the legal question on its own and only slipped up on the citation. Here, the fake cases weren’t decoration, they were what the tribunal pointed to as its reason for deciding the way it did. Take them away, and there’s nothing left holding the ruling up.

Same disease, different immune response

Put these two cases side by side and one question stands out.

Fabricated citations entering a judicial order is not, by itself, what determines the outcome.

What determines it is whether a human’s own independent judgment was ever actually exercised underneath the citation, or whether the citation was standing in for judgment that never happened.

In Vijayawada, the judge’s reasoning existed with or without the fake cases. In Mumbai, remove the fabricated precedent and there’s a hole where the reasoning was supposed to be.

In both cases, the fake citations got past the court before anyone caught them. What decided the outcome wasn’t whether the mistake got caught. It was whether a human had actually done the reasoning underneath it.

No fact-checking tool would have caught this difference. Run both rulings through a citation-checker, and they’d both fail the same way; fake cases, flagged … done. Only one of them fails the question that actually matters:

Did a person do their job, or not.

Who reads it before it’s filed

India’s Supreme Court now has the Bar Council of India drafting rules for AI use in courts. That is the all important rule-writing layer.

That is not the same as the real question underneath it: is any actual lawyer, on any actual filing, reading every citation the AI hands them? Or has “the AI found it” quietly started to feel like “someone checked it” … when nobody has?

Two courts ran into the same failure in the same year. One had a human doing real work underneath the citation, and the fabrication cost nothing. One didn’t, and it cost a signed order, an appellate ruling, and two years of litigation, all undone in a single afternoon in Delhi.

The technology was identical in both rooms. This isn’t really a story about legal AI, or about courts. It’s what happens anywhere an AI hands someone an answer that looks finished, whether it be a citation, a score, a recommendation, and it becomes easier to trust it than to check it.

The question worth asking isn’t whether the AI gets better at catching its own mistakes. It’s whether anyone is still required to do the checking themselves, or whether that requirement quietly disappeared the moment it got easy to skip.

References

1. Bar and Bench – “Supreme Court sets aside NCLT judgment for relying on hallucinated AI-generated citations”: https://www.barandbench.com/news/litigation/supreme-court-sets-aside-nclt-judgment-for-relying-on-hallucinated-ai-generated-citations
2. LiveLaw – “AI-Generated Citations Cannot Replace Judicial Mind, But Order Will Stand If Law Applied Is Correct: Andhra Pradesh High Court”: https://www.livelaw.in/high-court/andhra-pradesh-high-court/andhra-pradesh-high-court-ruling-mere-mention-wrong-citation-cannot-vitiate-order-if-law-correctly-applied-cautionary-note-unregulated-ai-tools-use-520160

Has your organisation sorted which decisions are measurement and which are judgment — or built oversight on top of a map nobody drew? If you’re not sure, that’s usually the answer.

I write Decision Layer Weekly for people working on exactly this gap: Subscribe here. And if it’s live for you right now, reach me directly: mattsheehan@spatialnext.io

---

Matt Sheehan

Matt is an AI strategist with 25 years at the intersection of geospatial intelligence and decision-making. He maps the architecture connecting three layers most organisations haven’t yet seen together: the sensing layer the geospatial industry has built, the causal reasoning layer now arriving, and the human decision layer nobody is designing. The third layer is where most deployments fail. And where Matt has his primary focus.

On June 13, 2020, a hailstorm tore through northeast Calgary with tennis-ball-sized stones, 70,000 insurance claims, an estimated $1.2 billion in damage.

This was the costliest hail event in Canadian history.

Jason Fischer’s two-story home was among the thousands hit. Fifty days later, he was still waiting.

No timeline. No word on when repairs would start. Roofing crews worked their way street by street through Saddleridge and Taradale while homeowners watched from their driveways, unsure where they sat in the queue. The Insurance Bureau of Canada told residents to expect a full year before all the damage would be fixed.

Nobody disputed the scale of the storm. What was missing was an answer to a much smaller question: when claim volume jumps overnight, who decides how fast Jason Fischer gets a callback?

The measurement was never the hard part

There is something very easily missed in this story – the actual detection problem was trivial. A hailstorm of this size is not subtle. Radar systems saw it coming, damage reports came in fast, and within days insurers knew roughly how many homes were hit and roughly how bad.

That is measurement, and it was solved almost immediately.

So what was not solved? It took fifty days to even begin resolving for people like Fischer. This was a judgement problem. Given the scale of this event, who has standing to trigger surge staffing, prioritize outreach, and put a real number on “how long will this take”. In this case, unfortunately, the silence itself became the story? Nobody had decided that in advance. So by default, nothing happened until public frustration forced it into the open.

This is the same shape as a guardrail auditing an outcome after the fact, not a spine built to act before one.

The exception that proves it’s a design choice, not an inevitability

Let’s now contrast that with Liberty Mutual Insurance‘s hail alert program, built by its Solaria Labs innovation team. Customers opt in and get a text message the moment hail is forecast for their location, in other words before it falls. In a survey of users, 44% said the alert prompted them to actually move their car to a garage or covered area, avoiding the damage altogether.

So what is different here?

This still began as a measurement problem: a hailstorm is coming. But Liberty Mutual also built a judgement layer. Someone at the company decided, well before any particular storm, that “hail forecast for this customer” should automatically trigger a specific action: notify them, now, with enough lead time to act.

And that is the distinction.

Calgary’s insurers had the measurement (a historic hailstorm, clearly underway) and no pre-built judgement layer to act on it at scale. Liberty Mutual built the judgement layer for a much smaller, quieter version of the same problem … an individual car, a single approaching storm.

How is it benefiting them and their customers?

Retention, trust, fewer claims filed in the first place.

The test, applied without any AI in the loop at all

So let’s go back to one of my foundational questions – Is this a measurement problem or a judgement problem?

Knowing a hailstorm hit Calgary, or knowing hail is forecast for a specific zip code tomorrow … that is measurement, and in both cases it was solved cleanly, with good data, and in good time.

Deciding what happens next, and who’s authorized to make it happen without waiting for someone to notice and remember .. now that is judgement.

And it is the only real variable separating a customer who gets a text before the damage happens from a customer who spends fifty days watching roofing crews work their way toward his street.

It is worth pausing here. My focus is on measurement versus judgement as we begin to adopt and apply AI. In what we have described so far there is no model, no algorithm, and no automation-bias risk anywhere. This is a pure test of whether an organization has built the checkpoint – who decides, at what threshold, with what standing to act – before the event, or ..

is discovering the gap in real-time, in public, one frustrated customer at a time.

Why this matters even where there’s no model in sight

It would be easy to read this as a story about staffing and communication, completely unrelated to anything I have been writing about before.

It is not.

This is the cleanest version of the design problem I discuss. Sure, there is no AI model accuracy to argue about and no automation bias to muddy the picture — just the bare structure of measurement versus judgement.

And that is exactly what makes this a useful blueprint. Insurers are now building AI into both halves of this story — forecasting models that predict hail paths and claim surges before they happen, and generative tools that draft the outreach, triage the queue, and recommend which customers to prioritize.

Adding AI to Calgary’s problem without fixing the design gap first just moves the failure downstream.

A model that predicts “this zip code will file 4,000 claims by Thursday” is still only measurement. If nobody has pre-decided who’s authorized to act on that number — trigger surge staffing, greenlight proactive outreach, commit to a communicated timeline, the prediction just sits in a dashboard while the queue backs up the same way it did in 2020.

Liberty Mutual’s alert program is the model for getting this right, and the reason it works has nothing to do with how sophisticated the forecast is. The forecast is almost incidental — hail prediction isn’t new.

What is designed is the handoff.

Someone decided in advance what the threshold is and what action it triggers. When the forecast crosses that threshold, the action fires. The judgement was already made; nobody has to remake it live, in the moment, under pressure.

Building an AI system on this pattern means separating two questions at the design stage, before a single model gets deployed:

• What is the model measuring, and how good does it need to be before anyone trusts its output? That’s the forecasting or triage layer — hail probability, claim volume, likely severity. It’s a measurement question, and it’s the one AI is actually good at.
• Given that measurement, who has pre-authorized standing to act on it, at what threshold, without waiting for a human to catch it in real time? That’s the judgement layer, and no amount of model accuracy substitutes for someone having answered it in advance. A perfect hail forecast is worthless to Jason Fischer’s neighbors if the surge-staffing decision still waits on a Monday morning meeting.

The design principle carries over directly: build the AI to answer the measurement question as well as it can. Build the organization, separately and in advance, to answer the judgement question before the model ever produces an output that needs acting on.

If organizations blur the two, by treating a good forecast as if it already made the call on what to do, then they have quietly let the model decide by default.

Same failure as before, just a different storm.

The moment of design

Every insurer serving a hail-prone region already has the measurement problem solved. Radar, forecasts, claims counts.

None of that is the bottleneck anymore, and hasn’t been for years.

The bottleneck is whether someone, in the calm before the season starts, sat down and decided exactly who gets to say “surge staffing starts now” or “this customer gets a proactive text”, and gave them the standing to act on it without a committee meeting once the storm’s already on radar.

Liberty Mutual answered that question in advance for one small piece of the problem: a car, a driveway, a few hours of lead time. Calgary’s insurers didn’t answer it at the scale that mattered.

Jason Fischer paid for that gap in the thing that actually costs a carrier something … fifty days of silence, and lost renewals.

References

• CBC News — https://www.cbc.ca/news/canada/calgary/calgary-hail-storm-insurance-delay-1.5670692
• PR Newswire (Liberty Mutual) — https://www.prnewswire.com/news-releases/liberty-mutual-helps-auto-insurance-customers-stay-safe-and-protect-vehicles-with-real-time-hailstorm-alerts-301312721.html

Has your organisation sorted which decisions are measurement and which are judgment — or built oversight on top of a map nobody drew? If you’re not sure, that’s usually the answer.

I write Decision Layer Weekly for people working on exactly this gap: Subscribe here. And if it’s live for you right now, reach me directly: mattsheehan@spatialnext.io

More than a week before the Eaton Fire, Ariel Cohen already knew something was building.

Cohen is the National Weather Service’s (NWS) chief meteorologist for the Los Angeles office. In the days before January 7, 2025, she and her team watched the signals stack up in the weather models, these were the kind of pattern that, in her own words, pointed to “the possibility of a highly volatile fire weather event.” Their confidence didn’t spike overnight. It built, day by day, as the models converged on the same answer:

A historic Santa Ana wind event was coming, into a landscape that hadn’t seen meaningful rain in eight months.

By January 6, that confidence level had crossed a threshold that the NWS reserves for its most extreme warnings. Los Angeles was placed under a Particularly Dangerous Situation Red Flag Warning (PDS), this was an alert type used only a handful of times in the four years since it was introduced for Southern California fire weather.

NWS meteorologists describe it, plainly, as “about as bad as it gets.”

Southern California Edison’s (SCE) transmission circuits running through Eaton Canyon stayed energized anyway.

On the evening of January 7 a fire ignited in the canyon. Driven by the wind event Cohen had been tracking for a week, with wind gusting up to 100 mph, it tore into the town of Altadena, killing 19 people and destroying more than 9,400 structures.

It stands as one of the most destructive fires in California history.

Two signals, one decision

Cohen’s warning wasn’t a guess. It was the product of a week of rising model confidence, escalating through the NWS’s own alert hierarchy – from watch, to warning, to PDS – precisely because that escalation is designed to tell downstream decision-makers ..

This one is different.

Utilities like SCE run their own internal risk-scoring models to help decide when to call a Public Safety Power Shutoff, inputs include fuel moisture, live weather feeds, and vegetation conditions circuit by circuit. Those models exist because de-energizing a canyon full of transmission lines is expensive and disruptive, in other words not a call to make on every red flag day. Based on the output from its model –

SCE did not shut off the Eaton Canyon circuits, despite Cohen’s warning already being in effect.

The cause of the fire has not been formally determined. SCE’s own investigators, along with CEO Pedro Pizarro, have pointed to a working theory that an idle, decommissioned line in the canyon may have become unexpectedly energized, in other words not a fault on one of the active lines. Multiple lawsuits, including one from the Department of Justice, allege SCE equipment caused the fire; SCE hasn’t conceded liability, and the county’s investigation is ongoing.

The exposure is already real on the balance sheet. Edison International has recorded roughly a billion dollars in Eaton Fire liabilities through settlements and its compensation program, and Jefferies equity analysts estimate SCE’s total potential liability could run as high as $13.5 billion.

So was this a forecasting problem?

Not on Cohen’s side. Her team’s forecast held. The confidence she flagged a week out proved correct almost to the hour.

If there was a forecasting problem, it sat inside the localized, circuit-level models utilities like SC Edison use to decide when a Public Safety Power Shutoff is warranted. Canyons are brutal environments for this kind of statistical model: they funnel and accelerate wind in ways that are very hard to capture at the resolution most operational risk-scoring runs at.

That’s a real, and very difficult technical limitation. If we stop the story here we would come to the conclusion that a decision failure was caused by a technology failure ..

.. and that is the trap.

Whatever SCE’s internal model said that night, the NWS had already issued an independent, human-legible signal that conditions were historically dangerous; one of the rarest alert types NWS issues, for a reason. Two signals existed. One of them was a regional warning designed explicitly to say “we don’t need higher resolution to know this is bad.” The other was a circuit-level risk score.

The Illusion of Compliance: Guardrails vs. a Spine

In the wake of disasters like this, the corporate reflex is predictable:

throw more money at the data.

Update the sensors, increase the compute, polish the predictive mirror. That is an optimization trap which treats a structural decision failure as a technology problem.

When organizations try to handle these gaps, they default to standard corporate governance: risk committees, ethics charters, vendor audit checklists, in other words expensive administrative workflows designed to ensure “compliance” on paper. But there’s a distinction heavy industry keeps missing:

Governance gives a system guardrails. A decision architecture gives it a spine.

Guardrails only activate after a number crosses a preset line – the model provides a score of 10 for example which activates a specific rule – they’re reactive, built to catch problems after the fact. A spine determines behavior before the crisis hits. It establishes who holds ultimate authority, what specific contextual signals they’re empowered to act on, and what organizational protection exists when a human commander decides to shut down an asset against a model’s recommendation; including a model built in-house.

The “Decide Who Decides” Test

The fix isn’t a more complex model. It’s a step that has to happen before the system ever goes live. Run every core operational workflow through a simple, binary test:

Is this a Measurement Problem? (The Machine Owns It) – If a question has a single, definitive, verifiable answer given good data, the machine should settle it. Calling an offside in football or measuring real-time voltage frequency on a wire are math problems. Autonomy belongs here.

Is this a Judgment Problem? (The Human Commands It) – If a question involves chaotic variables, shifting physical conditions, and asymmetric risk, where two reasonable experts looking at the same data could honestly disagree, it’s a judgment call.

An internal model’s risk score against a regional National Weather Service PDS warning, in a canyon known to funnel Santa Ana winds unpredictably, was a judgment call.

Nothing in the public record shows SCE had a structured doctrine for resolving that conflict. That means the system defaulted to treating a model output as sufficient on its own, rather than as one input a human was empowered to override.

That is backwards.

The Moment of Design

Control room operators, incident commanders, and risk underwriters are being handed sophisticated predictive models and left to manage the handoff under running clocks.

The centre of control doesn’t live at the moment of crisis. It lives at the moment of design.

As regulation, see Article 14 of the EU AI Act as one example, and tort litigation start closing the door on corporate deniability, executives won’t be able to point to an automated threshold as the reason a decision got made.

The organizations that get this right won’t be the ones with the most expensive models. They’ll be the ones with the discipline to decide, in advance, what happens when a chief meteorologist and their algorithm disagree .. before a jury has to ask why nobody did.

Has your organisation sorted which decisions are measurement and which are judgment — or built oversight on top of a map nobody drew? If you’re not sure, that’s usually the answer.

I write Decision Layer Weekly for people working on exactly this gap: Subscribe here. And if it’s live for you right now, reach me directly: mattsheehan@spatialnext.io

---

Matt Sheehan

Matt is an AI strategist with 25 years at the intersection of geospatial intelligence and decision-making. He maps the architecture connecting three layers most organisations haven’t yet seen together: the sensing layer the geospatial industry has built, the causal reasoning layer now arriving, and the human decision layer nobody is designing. The third layer is where most deployments fail. And where Matt has his primary focus.