In June 2019, Chesley “Sully” Sullenberger – yes that Sully, the guy who safely landed an aeroplane on the Hudson River in 2009 – sat in a 737 MAX simulator and flew the exact scenario that killed 346 people twice over; the Lion Air and Ethiopian Airlines crashes.
A stick for stick recreation.
He knew what was coming. He knew the failure, the false warnings, the nose-down commands.
Testifying to the House Transportation Committee afterward, he described how close he came to running out of time to react — even with foreknowledge of what was coming. No US pilot, he said, had trained for this exact failure before the crashes.
That’s one of the most experienced pilots in America forewarned, in a simulator, with every advantage a pilot could ask for. He still nearly ran out of time.
The 737 MAX’s automated trim system, called MCAS, pushed the nose down again and again, on the real flights and in the simulator recreation alike. It didn’t fail because its sensors were wrong.
It failed because no one had the authority to override it before it acted.
An industry that already knew the answer
This is what makes Boeing’s failure different from most of the ones I have so far written about. Aviation isn’t a young industry discovering this problem for the first time. It learned it decades ago, the hard way, through a string of crashes in the 1970s and 80s that forced a generation of engineers to build a doctrine most other industries still don’t have:
Automation can assist, but a human must always retain the authority, protected, trained, and real, to override it.
That doctrine is why a plane is one of the most heavily automated machines a human operates, and also one of the few when “the pilot says stop” the system still reliably stops.
Which makes the 737 MAX story harder to explain away as ignorance.
Three layers, one quietly skipped
If you break MCAS down in the way I have broken every AI-adjacent failure down in past articles, you will see a repeating structure.
The sensing layer was a single Angle of Attack (AOA) sensor – a sensor on an aircraft that measures the angle between the wing and the oncoming airflow. Boeing built MCAS to run off a reading from only one of the AOA sensor rather than cross-checking against a second, which meant one faulty vane, with no independent confirmation, was enough to set the whole failure in motion.
The reasoning layer was MCAS itself, and this is the part that should sound familiar to anyone who has read my newsletter. MCAS didn’t just measure an angle. It acted on that measurement, repeatedly, on its own authority. The original design let a single sensor reading trigger the system again and again, nose-down each time, with no limit written in for when the pattern should stop and control should hand back to the crew. A measurement was allowed to become a decision, on a loop, because nobody had drawn the line for where it should end.
The decision layer was supposed to be the pilots. It didn’t exist, because they didn’t know there was anything to decide. Boeing never disclosed MCAS to the pilot community, so the training that would have let a crew recognise and fight it was never built. You cannot exercise protected authority over a system you were never told was flying the plane.
This isn’t an aviation story. It’s what happens whenever a measurement is allowed to become a decision, on a loop, with no human positioned to stop it — and MCAS just happens to be the case where the black box was recovered.
The same three-part failure shows up everywhere I’ve looked. Not sensing, reasoning, and deciding, but a version of it that applies before a single line of automation gets written.
Before you build: three questions
1) What decisions needed making?
Before any of the 737 MAX engineering, there was a decision inventory nobody wrote down. Somewhere in the certification process, someone needed to list out every way MCAS could act on the aeroplane and ask, plainly:
Which of these can the software be trusted to settle on its own, and which of these need a human positioned to catch it if the software is wrong?
That inventory is a design question, not a technology one, and it comes before the software is written, not after it ships.
2) Where does AI measure, and where does a human judge?
An Angle of Attack reading is a measurement. It has a right answer, and a good sensor gets it. That part, correctly, belonged to the machine.
Whether to push the nose down, how far, how many times, and at what point a disagreement between instruments should be handed to the person in the seat …
That is not a measurement. That’s a threshold question, the kind where reasonable engineers, given the same data, could land in different places. MCAS treated it as settled. It wasn’t.
Sullenberger named this exact confusion in his testimony, in language close enough to the framework I use here that it’s worth quoting directly. Boeing, he said, had added a computer-controlled feature to a human-controlled aeroplane without giving it “the integrity, reliability and redundancy that a computer-controlled system requires.”
Integrity and redundancy are what you build once you’ve correctly separated the measurement from the judgement.
Boeing built the measurement. It skipped the judgement.
3) Who has the authority to act on that judgement and can they actually use it?
Even a correctly identified judgement call is worthless without someone positioned to make it. That’s authority, and authority requires three things most organisations quietly skip: the person has to know the system exists, has to be trained on what it does, and has to have protected standing to override it without that override counting against them.
Pilots had none of the three. They didn’t know MCAS existed. The pilots’ union told the same committee that without disclosure, meaningful training on the system had never happened. And because they didn’t know what they were fighting, there was no way to exercise an authority they didn’t know they had.
The fix, and what it rebuilt
Everything the FAA mandated after the grounding maps directly onto the three layers that failed.
1) Sensing was fixed with redundancy: MCAS now requires agreement between both Angle of Attack sensors before it will activate at all. One bad vane can no longer make the decision alone.
2) Reasoning was capped: the system now fires once per event at most, and was redesigned so it can never override a pilot’s ability to control the aircraft through the control column. That is the guardrail turned back into a guardrail; adjustable, bounded, no longer able to overpower the spine.
3) The decision layer was rebuilt from nothing: a new cockpit alert now flags disagreement between the sensors so the uncertainty is visible rather than hidden, and simulator training was made mandatory so a pilot encountering the failure would recognise it and know exactly what authority they held.
Uncertainty made visible. Authority made real.
Protection built in before the system flies again, not after.
The moment of design
Aviation didn’t need to invent the discipline of protected human authority in 2019. It already had it, built over decades, and paid for in an earlier generation of crashes. MCAS is what happens when a mature industry, one that already knows better, lets a single new feature ship without asking the same question it forces every other system in the cockpit to answer.
If an industry with the deepest experience in this exact problem can still forget to ask it, the odds that your organisation is asking it by default are not good.
Somewhere in your organisation, a system is measuring something, deciding something, and acting on it. The honest question is whether anyone, by name, has the standing to say not this one, not like this.
If you can’t answer that in one sentence, you don’t have a gap in your AI strategy. You have an MCAS you haven’t found yet.
Follow-Up
If you currently have an AI workflow in production or pilot, ask your team one question: “What is the system explicitly authorised to do below a certain confidence score?”
If you can’t find that answer in writing within 10 minutes, your boundary isn’t set.
If you’d like to stress-test that workflow before an edge case does it for you, let’s run a 30-minute boundary review: mattsheehan@spatialnext.io