Every organization deploying AI in consequential work has answered this question, usually in a single line of a governance document: a human reviews and approves the output, so a human remains accountable. The phrase "human in the loop" appears, everyone nods, and the matter is considered settled.

It is worth examining how much unearned work that phrase is doing. The claim is that because a person clicks approve, that person exercised judgment, and therefore responsibility sits with them. But approval and judgment are not the same act. A person can approve something they had no realistic capacity to evaluate, no time to examine, and no practical freedom to reject, and the approval will look identical in the audit log to one that involved genuine scrutiny. When an organization treats the click as the accountability, it has not retained human control over the decision. It has designated someone to absorb the consequences of a decision the system effectively made.

The moral crumple zone

There is a precise name for this arrangement. The cultural anthropologist Madeleine Clare Elish, studying accountability in automated systems, called it the moral crumple zone: just as a car's crumple zone is engineered to absorb the force of an impact, the human operator in a highly automated system can become the component that absorbs the moral and legal penalty when the overall system fails.

Her point is structural rather than cynical. Control in these systems is genuinely distributed across designers, vendors, data, the model, the process, and the operator. But our legal and social instincts about responsibility are built around individuals, so when something goes wrong we look for a person, and the person nearest the failure is the one who was holding the mouse. The result is that responsibility gets attributed to a human who had limited ability to influence the outcome, while the parties who designed the system, chose to deploy it, and set the conditions under which the operator worked remain at a comfortable distance.

Elish drew this from aviation, where decades of automation produced a consistent pattern: when automated systems failed, investigations and public accounts tended to credit the technology for successes and assign the failures to pilot error. The pattern is not an accident of any single case. It follows from the mismatch between distributed control and individual blame.

The oversight does not work the way the policy assumes

The whole arrangement rests on an empirical assumption that the human reviewer will actually catch the AI's errors. That assumption is testable, and the results are not encouraging.

In a prospective study published in Radiology, researchers had 27 radiologists read 50 mammograms with the assistance of what they believed was an AI system, where the suggestions were in fact controlled by the researchers. When the purported AI suggested an incorrect category, accuracy collapsed. Inexperienced readers fell from 79.7% correct to 19.8%. Moderately experienced readers fell from 81.3% to 24.8%. And very experienced radiologists, the ones you would most expect to hold their ground, fell from 82.3% to 45.5%.

Sit with that last number. These are trained specialists, reviewing images in their own domain, and an incorrect machine suggestion cut their accuracy roughly in half. The lead author noted that it was surprising to find even highly experienced radiologists adversely affected, though to a lesser degree than their less seasoned colleagues. The human was in the loop. The human did not catch it.

If expert clinicians reviewing images in their specialty are pulled this far off by a wrong suggestion, it is not reasonable to assume that a busy analyst approving the two-hundredth AI-generated recommendation of the day is providing meaningful oversight. The oversight is nominal, and the accountability built on top of it is nominal too.

Approval is not accountability unless three things are true

The useful way to think about this is to ask what genuine accountability actually requires, because approval only constitutes it when specific conditions hold.

The person must have the capacity to evaluate the output. Not access to it, capacity to judge it. If the reasoning behind a recommendation cannot be reconstructed, or the reviewer lacks the domain depth to know when it is wrong, they cannot exercise judgment no matter how carefully they read.

They must have the time. Oversight has a real cost in minutes, and if the volume of decisions has been set on the assumption that AI made review fast, the reviewer has been given a workload that presumes they will not look closely. A throughput target is a policy statement about how much scrutiny is expected, whatever the governance document says.

And they must be genuinely free to dissent. If overriding the AI requires justification that agreeing does not, if disagreement is slower, if the system is described internally as more accurate than the people reviewing it, then rejection carries a cost that acceptance does not. Under those conditions, approval is the path of least resistance and tells you nothing about the reviewer's actual assessment.

Where all three hold, a human approval is meaningful and accountability can honestly rest there. Where they do not, the organization has created a role whose function is to be blamed. That is worth naming plainly, because it is usually not anyone's intention. It is the default that emerges when nobody checks whether the oversight they specified is physically possible.

The switch institutions flip

There is a second-order version of this that leaders should recognize, because it is the pattern that turns a design flaw into something closer to a strategy. An organization can hold two contradictory positions about the same system depending on which is convenient.

When promoting or justifying the AI, the system is described as fast, consistent, and better than human judgment, which is why it was worth deploying. When the system produces a bad outcome, the account shifts: a human reviewed and approved this, so the failure lies with the operator's judgment. The technology's competence is invoked to justify adoption and the human's responsibility is invoked to absorb failure, and both claims are made by the same institution about the same decision.

You do not have to attribute bad faith to anyone for this to happen. Each statement is made at a different time, by different people, for different purposes, and each sounds reasonable in isolation. But the combined effect is an accountability structure where the benefits of automation accrue upward and the liability settles downward onto whoever was closest to the output.

The honest part: this is not an argument against human oversight

It would be a serious misreading to conclude that human review is theater and should be abandoned. Genuine oversight is valuable, and there are many settings where a human reviewer with adequate context, time, and authority catches errors that would otherwise cause real harm. The problem is not oversight. It is oversight specified on paper without the conditions that make it function.

It is also true that someone has to be accountable. A decision nobody owns is worse than a decision owned by the wrong person, and organizations cannot resolve this by concluding that AI-assisted decisions simply have no responsible party. The vendor is not accountable for how you deployed their tool. The model cannot be questioned, disciplined, or held to account in any meaningful sense. Something has to sit with a person.

The argument is narrower: accountability should be placed where control actually exists. Often that is not the person who clicked approve. It is the executive who decided to deploy the system in this context, the leader who set the review throughput, the team that chose not to build the interface that would have made errors visible. Those are the decisions that determined whether the operator could succeed, and they were made by people with genuine agency over the outcome.

Designing accountability instead of assigning it

The practical shift is from asking who is responsible to asking who could actually have prevented this, and then making sure those are the same person.

For any AI-assisted decision that matters, work through the three conditions honestly. Can the reviewer evaluate this output, with the information and expertise available to them? Does the workload leave time for genuine examination, or has the volume been set on the assumption that review is a formality? Is disagreeing with the system as easy, fast, and career-safe as agreeing with it? Where any answer is no, you have found a moral crumple zone in your own organization, and the fix is either to change the conditions or to move the accountability upward to where the real decision was made.

It is also worth writing down, in advance, who owns the outcome when the AI is wrong and the human agreed with it. Deciding that after a failure guarantees it lands on the person nearest the keyboard, because that is where blame flows by default. Deciding it beforehand forces the uncomfortable and clarifying question of whether the oversight you designed was ever capable of doing the job you assigned it.

This is the organizational counterpart to the individual problem of protecting yourself when you rely on AI output, covered in the AI that makes you look bad. The individual version is about personal vigilance. This one cannot be solved by vigilance, because the evidence suggests vigilance is not sufficient even among experts. It has to be solved by design.

FAQs

Q1. What is a moral crumple zone?
It is a term from Madeleine Clare Elish's research on automated systems, describing how a human operator can become the component that absorbs the moral and legal penalty when a complex automated system fails, much as a car's crumple zone absorbs impact. Control in such systems is distributed across designers, vendors, and operators, but blame tends to concentrate on the individual nearest the failure, often someone with limited ability to influence the outcome.

Q2. Doesn't "human in the loop" solve AI accountability?
Only when the human genuinely has the capacity to evaluate the output, the time to do so, and real freedom to dissent. Without those conditions, the approval is nominal and the accountability built on it is nominal too. The phrase is frequently treated as a complete answer when it is really a claim that needs to be verified against the actual working conditions of the reviewer.

Q3. What does the evidence say about whether humans catch AI errors?
It is sobering. In a study published in Radiology, 27 radiologists read mammograms alongside what they believed was an AI system. When the suggestions were incorrect, accuracy fell from 79.7% to 19.8% among inexperienced readers, and from 82.3% to 45.5% among very experienced ones. Domain experts reviewing work in their own specialty were substantially misled, which makes strong assumptions about routine business review hard to justify.

Q4. If oversight is unreliable, should we remove the human?
No. Genuine oversight catches real errors and remains valuable. The finding is not that humans are useless in the loop, it is that oversight has to be designed with the conditions that let it function: adequate expertise, realistic time, visible reasoning, and a genuine ability to say no. Removing the human because oversight is imperfect would be the wrong lesson.

Q5. Where should accountability actually sit?
Where control actually existed. Often that is not the person who approved the output but the executive who chose to deploy the system in that context, the manager who set review volumes that made scrutiny impossible, or the team that designed an interface where errors were invisible. Those decisions determined whether the reviewer could succeed, and accountability tracks agency.

Q6. What is the institutional double-move to watch for?
Describing the AI as superior to human judgment when justifying its adoption, then attributing failures to the human reviewer's judgment when something goes wrong. Both claims can be made sincerely by different people at different times, but together they create a structure where automation's benefits flow upward and its liabilities flow down to whoever was nearest the output.

Q7. How do I tell whether our oversight is real or nominal?
Ask three questions about any AI-assisted decision. Can the reviewer actually evaluate the output with the expertise and information available? Does their workload leave time for genuine examination? Is disagreeing with the system as fast and as safe as agreeing? If any answer is no, the oversight is nominal, and the accountability resting on it is misplaced.

Q8. What should we do before deploying AI into a consequential decision?
Write down in advance who owns the outcome when the AI is wrong and the human agreed with it. Deciding this after a failure almost guarantees blame lands on the person nearest the keyboard. Deciding it beforehand forces the question of whether the oversight you specified was ever capable of the job, which is the question most governance documents quietly skip.