Paula submitted the file for the third time that week.
Her credit was solid – steady income, manageable debt, a job her employer confirmed in writing was not at risk. But the AI model had flagged her file as high risk on the first pass. The second submission came back the same, as did the third. Each time, the note was the same. Not a flat denial. A “refer to underwriting.” A status that implied a human would look again, weigh what the model couldn’t, and decide.
At some point in that process, she asked the loan officer handling her file what was happening. The answer sounded less like reassurance and more like an honest shrug: it seemed like her application was getting rejected by an algorithm, and that there was a person who should be able to step in and override. The loan officer wasn’t being evasive. He was working with the same uncertainty.
Neither of them knew what the model actually knew – that a single variable had become a proxy for race inside the model’s logic. This was not by intent, it was by optimization. The model found it predictive, so it kept it. And nobody was in a position to catch it.
That is the entire problem. Not that the model discriminated; that nobody was given the authority to question the model.
A human in the loop, on paper
Automated underwriting has been the default across consumer lending since the late 1990s. But the systems running today are not the rules engines Fannie Mae and Freddie Mac introduced then. Modern AI underwriting models find patterns in data that no human would think to look for. They pull credit history, income data, debt ratios, and increasingly a wider net of behavioral and alternative data, and return a decision in seconds. Most applications go straight through. The harder cases get flagged for a person to review
On paper, that’s exactly the design my articles have been arguing for. A human in the loop, positioned at the point where the model’s confidence runs out.
In practice, that loop is often missing. The loan officer has a job title that implies authority. What he frequently lacks is a defined basis for using it: what evidence is sufficient to override the model, what protection he has if he’s wrong, and what happens to his numbers if he overrides too often. Nobody designed those answers. The override exists in the policy manual and nowhere else.
This matters because the failure isn’t the loan officers. It is what he was never given. That does not mean just a human in the room, but a human with a defined scope, defined authority, protection from the career cost of using it. Those are three different things. Most organizations provide the first and skip the other two entirely.
I discussed Zillow in my last article – Zillow Offers Didn’t Fail Because Its AI Was Bad. It Failed Because the AI Was the Only One in the Room. What we are discussing here is a quite different failure. Zillow removed the human deliberately, by design, because the model felt accurate enough not to need one. Here, nobody was removed. The human was simply never given real power to begin with. Nobody decided that. Which is exactly why nobody is accountable for it, and why it persists.
The mechanism, made concrete
In July 2025, the Massachusetts Attorney General announced a settlement with a student loan servicer over allegations that its AI underwriting model produced unlawful disparate impact based on race and immigration status. The case is notable not for the size of the fine, but for the precision of the finding.
Investigators didn’t allege the algorithm was generally biased. They traced the mechanism. The Cohort Default Rate — a US Department of Education metric tracking the average loan default rate associated with specific colleges — had functioned as a pipeline for discrimination embedded directly in the model’s logic.
That detail matters more than the dollar figure. A single input, several steps removed from anything resembling race, produced a discriminatory pattern at scale, and it ran for years before anyone with the authority to ask “what is this variable actually doing” did so.
The model wasn’t broken. It was built to predict default risk. A variable that happened to proxy for race improved that prediction, so it stayed. The model had no way to ask whether it should be kept, it was simply doing exactly what it was designed to do. The failure was that nobody was positioned to interrogate the logic before it ran.
That question was available to ask at any point. It didn’t require better AI. It required someone positioned to interrogate the model’s logic, with the authority to say stop while the answer remained unclear. That person, functionally, did not exist inside the process until a state attorney general’s office built the case from the outside.
The three layers, again
The sensing layer or the data which feeds the system, is large and growing. Credit bureau data, income verification, increasingly bank transaction history and behavioral signal. Lenders have more data about applicants than at any point in the industry’s history.
The reasoning layer is more sophisticated here than in Zillow’s case. Zillow’s model couldn’t reason about changing conditions. Modern underwriting models aren’t simple rules engines. They find patterns in historical data that no human would think to look for. That sophistication is exactly what makes a variable like Cohort Default Rate dangerous. The model isn’t reasoning about fairness. It’s optimizing for predictive power.
The human layer is where this should have been caught. Not after a regulator builds a discovery case. Before the model goes live, with someone whose actual job is to interrogate what each variable is doing and why. Someone with real authority to pull a variable or halt a deployment. Someone who isn’t quietly penalized in their own performance numbers for using that authority.
That last condition is the one organizations consistently skip. Authority on paper means nothing if the person who exercises it pays a career price for doing so – see the UnitedHealthcare case in my article: Remember the Fire Chief? His AI Just Overruled Him.
Designing the human layer means designing the protection too.
Why this is the harder version
Zillow’s failure in 2021 was impossible to miss. A decision to take the human out of the loop resulting in losses exceeding $500 million, and a CEO’s public admission. The kind of collapse that triggers a postmortem.
Most human layer failures don’t look like that. They look like the one I discuss in this article. No single dramatic event. A loan officer with an ambiguous override. A variable nobody questioned because the model’s accuracy numbers looked fine in aggregate. Years of quietly wrong decisions, made at scale, by people who were never given the standing to challenge what the model decided.
That’s called drift, not collapse. It doesn’t trigger a postmortem. It triggers, eventually, a regulator’s letter, years after the pattern was set.
The difference matters. Collapse is recoverable. Drift compounds and is invisible until it isn’t.
The shift already underway
Regulators are starting to close this gap from the outside. Lenders using AI can no longer tell a rejected applicant their application was declined without explaining specifically why the model made that decision. That’s the law demanding, after the fact, what the human decision layer should have been designed to provide from the start:
A real, defensible answer for why the model decided what it decided, and who inside the organisation is positioned to challenge it.
The institutions that wait for the regulator’s letter to design that layer will keep finding out about their own Cohort Default Rate variable years too late. The ones that design it now, before the next model goes live, are doing the same work Zillow could have done in 2021 and chose not to.
The question isn’t whether your underwriting model is accurate.
It’s what happens to the person inside your organization who asks what a specific variable is actually doing … before it runs for three years undetected.
Has your organisation designed the human layer — or just assumed it’s there?
If you’re not sure, that’s usually your answer: mattsheehan@spatialnext.io
Matt Sheehan
Matt is a geographer and AI strategist with 25 years at the intersection of geospatial intelligence and decision-making. He maps the architecture connecting three layers most organisations haven’t yet seen together: the sensing layer the geospatial industry has built, the causal reasoning layer now arriving, and the human decision layer nobody is designing. The third layer is where most deployments fail. And where Matt has his primary focus.