I keep reading articles about AI and having a “Human-in-the-loop” (HITL) versus automation with no human. At HUMANX in Amsterdam recently one speaker discussed how HITL in practice means asking a person to watch a sophisticated system work and click “approve”. That is what I call the rubber stamp trap. But that same speaker then went on to say: humans should not be the guardrails, that we should let the technology provide the security, controls and boundaries, and let people provide the goals, the judgement and the humanity. That sounds good on paper, letting the technology hold the boundaries, but there is one major flaw here:
It assumes the boundaries already exist.
The Case of Robodebt
On 23 June 2026, Australia’s Federal Court approved a A$548.5 million settlement for people wrongly chased for welfare debts by a government computer. It is the largest class action settlement in Australian legal history. It comes on top of A$112 million paid out in 2021, and payments will run until 2028.
The scheme was called Robodebt. From July 2016, it compared what people had told Centrelink they earned every two weeks with their annual income on record at the tax office. The computer spread that annual figure evenly across the year, as if the person had earned the same amount every fortnight. Wherever that average was higher than what they had reported, it assumed they had been overpaid benefits and sent a debt notice.
But, casual workers, students and seasonal workers earn money in bursts. Two weeks of earnings do not reflect a years worth of earnings. A fundamental flaw in Robodebt. It was found that around 470,000 debts had been wrongly raised. The Royal Commission called the scheme a “crude and cruel mechanism” that was “neither fair nor legal”.
Robodebt is usually told as a story about a bad algorithm. But this is far more a story about a defined boundary and automation versus oversight by a human with authority.
Three phases, three answers
Before 2016 & Robodebt: People in the loop, unable to keep up. The department was struggling finding around 300,000 income discrepancies a year. Staff had the capacity to investigate around 20,000 of the highest-risk ones, gathering payslips and evidence before making any decisions. The checks were careful. They were also slow, and most discrepancies were never looked at.
2016: Robodebt automation or removing the human. The Online Compliance Intervention removed the manual step. The aim was to raise as many debts in a week as staff had previously raised in a year. Notices went out without anyone gathering evidence, and the burden of proof moved to the recipient. If a recipient disagreed, they had to find their own payslips, sometimes from years earlier. The Royal Commission found that removing human intervention was “a key factor in the harm Robodebt did”.
After the Robodebt backlash: People put back. The Commonwealth Ombudsman called for clearer notices and help for people to gather evidence, and a Senate committee recommended restoring manual checking based on actual fortnightly earnings. In November 2019, after conceding in court that income averaging was unlawful, the government stopped raising debts without first gathering evidence such as payslips. The Royal Commission’s recommendations on automation focused on a clear path for people to seek review, and on transparency about how automated decisions are made.
Robodebt ended up being a hugely expensive mistake, causing considerable distress to those wrongly sent a debt notice. The intention behind its introduction was, on paper good – speed and efficiency, but its application was a disaster.
And one we are repeating with the emergence of AI.
The answer we’re hearing now
Returning to the comment made at HumanX in Amsterdam. A HITL who simply approves the output from AI is not oversight; it’s a rubber stamp. A 2021 study from the University of Washington and Microsoft Research found this isn’t solved by better AI explanations: explanations made people more likely to accept the AI’s answer, whether it was right or wrong. Robodebt’s pre-2016 staff show the other problem; that careful human checking simply does not scale.
But suggesting we solve that by “letting the technology hold the boundaries”, or by taking humans out of the loop entirely, as Robodebt did, assumes the boundaries we need already exist: a line drawn in advance between what the system may decide on its own and what must go to a person with the authority to stop it.
It is fair to say that the Robodebt technology did have a boundary. That boundary was the gap between averaged income and reported income. The rule it enforced was simple: any gap was a debt. And the system applied that rule perfectly, hundreds of thousands of times.
The problem was the rule.
Nobody had decided, before the system went live, which of its outputs it was allowed to act on alone. Taking people out didn’t answer that question. Putting them back didn’t answer it either. Handing the boundary to technology won’t answer it either, because technology can only enforce a line somebody has already drawn.
The step everyone skips
So what broke Robodebt and invalidates the view shared at HumanX?
This is what I call the AI Decision Spine Model:
Deciding, in writing and before any live case, which decisions an AI’s output may close on its own (automated execution), which needs a named person working to a defined standard (human judgement), and which are never delegated.
Run Robodebt through this and the design almost writes itself. Imagine Sarah, a university student. She receives payments for nine months while studying and correctly reports earning nothing. Over the summer she comes off payments and works full time in a café, earning $12,000.
- Comparing the two income records? That’s arithmetic. Let the machine do it. The tax office says Sarah earned $12,000 last year. Centrelink has her reporting nothing. The machine spots the difference in a second.
- Spotting a gap worth a closer look? Also arithmetic. Let the machine do it. A $12,000 gap is well over any sensible threshold, so Sarah’s case is flagged for checking.
- Working out what someone actually earned? That needs evidence. If the café’s payroll is digital, the machine can pull the records and see that every dollar was earned in summer, when Sarah wasn’t on payments. If it isn’t, a person asks Sarah for her payslips.
- Telling someone they owe the government money? That decision has consequences for a real person. A gap in averaged income opens up a question, but it can never, on its own, become a debt. Robodebt spread Sarah’s $12,000 evenly across the year, that is around $460 a fortnight, including the months she was studying. Robodebt would have decided she owed money. The evidence shows she owes nothing.
That last rule is the spine. Once it’s written down, the technology finally has something real to enforce: no debt notice without proof of what the person actually earned, and no debt in any unclear or disputed case unless a named officer signs it off.
People no longer check every case, as they tried to before 2016, or no case at all, as under Robodebt. They make the one call that needs them, on the cases that need it.
This also solves the capacity problem that tempted the department to automate in the first place. Before 2016, staff spent their limited hours on the whole investigation. In this design, the system does the matching and the evidence gathering, and people handle only the adverse decisions the data can’t settle. That’s far fewer than 300,000 a year, and each one gets real attention rather than a click.
Neither “keep the humans in” nor “take them out” gets you there. Both treat the loop as a single on/off setting. The real question is which decisions need a person, and which don’t.
The law is starting to draw the line
Some legislators are now drawing this line themselves. The clearest examples are in US health insurance.
Since 1 October 2026, Alabama has required that any decision to deny, delay or modify a prior-authorisation request on medical-necessity grounds is made or reviewed by a licensed physician or other qualified health care professional. Iowa’s and Washington’s versions are already in force, and Colorado, Georgia and Minnesota follow on 1 January 2027.
AI may assist. It may not be the sole basis for a denial.
Look at the nature of that rule. An approval can be automated. A denial, the adverse decision against a person, cannot. That is a spine, written into statute. It’s the line Robodebt never drew.
California’s ADMT rules, from 1 January, ask the same question of AI used in hiring and other employment decisions.
What the laws don’t do is finish the job. They say who must make the denial. They don’t say which cases the AI can send straight to approval, what standard the clinician applies when they disagree with it, or how an insurer shows the review was real rather than a click. Each insurer has to design that for themselves, and the sensible time to do it is before the deadline, not after the first complaint.
Otherwise they end up where Robodebt ended up after 2019: a rule about who is allowed to decide, with nobody having designed what deciding actually involves.
Not in or out, but where
Robodebt has cost Australia more than half a billion dollars in this year’s settlement alone, years of public trust, and far worse for some of the families involved. Nobody chose that outcome. It came from a question nobody asked:
What was this system’s output allowed to decide on its own?
“Human in the loop” and “human out of the loop” both answer the wrong question. The right one is simpler: which decisions need a person, which person, and to what standard? Answer that before the system goes live, and technology can hold the boundary you chose. Skip it, and the system sets its own boundary by default. Robodebt’s default was “any gap is a debt”, and it stayed in place, despite years of warnings, until 470,000 debts had been wrongly raised.
If you’re about to put an AI-influenced decision into production, or a regulator has just put a date on one you already run, that’s the question the Decision Spine Diagnostic maps — one workflow, in writing, in two weeks. If it would help to talk it through, I offer a free 30-minute conversation: mattsheehan@spatialnext.io.