According to ProPublica’s reporting, Cigna doctors signed off claim denials at an average of 1.2 seconds each.
Every one had a human in the loop.
Nobody had decided what a real review involved.
Most organisations using AI on decisions about people are in the same place. A person signs off, so the decision feels safe. The law now asks whether that review was real.
That gap, between what a system produces and what the person signing it off actually does, is where most AI deployments quietly fail. Not because the technology is wrong. Because nobody decided before launch which decisions need a person, what that person has to check, and what they’re authorised to do when the system is confident and wrong.
Think of it as a fence. Before anyone builds it, someone has to mark where it goes. In AI, that marked boundary is the spine: it decides what’s safe to automate and what isn’t. The fence itself is the guardrail that enforces it. Most organisations put up the fence without ever marking the boundary. There are three ways to run AI on a decision, and each one needs a boundary:
- A person signs off every decision. This is where most organisations are, and at volume it slides into what I call the rubber-stamp trap. The boundary sets what a real review has to involve, so the sign-off still means something.
- A deliberate split between the two. The boundary sets which decisions go to the system and which go to a person, and why.
- The system decides on its own. The boundary sets exactly what it’s allowed to decide, because nobody is there to catch a mistake, and someone has to check it still holds.
The moment of decision is no time to work this out. The boundary has to be set before the system runs, or it isn’t set at all.
Where this matters now
Across the US, states are now putting rules in place for automated decisions about people. California is moving first on employment:
- 1 January 2027, California. New rules cover employers using automated systems for hiring, promotion, termination and other significant decisions, unless a person genuinely reviews the decision. Not a click: someone who understands what the system produced and has the authority to change it.
- 1 July 2027, California (SB 947). Employers can’t rely solely on an automated system to discipline or fire someone. A person has to corroborate the output against supporting evidence, and the employee must be told in writing that this happened.
Most companies will put a person in the loop, and these laws now say what that person has to do. They don’t say which decisions that person should see, how much time a real review takes, or what record will hold up if it’s challenged. Once a regulator or a lawyer asks, it’s too late to write it down.
That’s the work I do. Whichever way you run your AI, I help you set the boundary before launch: who holds authority at each decision, what the system can decide alone, and the record that proves it. Then I stay to put it in place.
