Listen carefully to how AI value gets described in most companies and you will notice something odd about the most confident claims. "It's made us more strategic." "It future-proofs the business." "It's freed our people to do higher-value work." "We're making better decisions." These statements sound like strong endorsements, and they are delivered as if they settle the question of whether the AI is worth it. But they share a hidden property that should trouble anyone approving the budget: there is no result, no outcome, no possible future, that would prove any of them false.
Ask yourself what would have to happen for "it's made us more strategic" to be wrong. Nothing would. Whatever the business does next, the claim survives. Grow, and the AI made you strategic. Shrink, and imagine how much worse it would have been without it. That immunity from disproof feels like strength. It is actually the tell of a claim that contains no information at all, and learning to spot it is one of the more useful things a leader can do before writing another AI check, because the claims that can never be wrong are precisely the ones that can never be trusted.
A test borrowed from science
The tool for seeing this clearly is almost a century old and comes from the philosophy of science. Karl Popper, in The Logic of Scientific Discovery in 1934, proposed a criterion for telling claims that carry real information from claims that merely sound like they do. He called it falsifiability. A statement conveys genuine information about the world, Popper argued, only if some possible observation could contradict it. A theory that forbids nothing, that is compatible with every possible outcome, tells you nothing, precisely because it rules nothing out.
His examples are famous. A horoscope that says "something of consequence will happen in your life tomorrow" cannot be falsified, because no possible day would contradict it, and so it tells you nothing despite sounding meaningful. Popper leveled the same charge at systems that could explain any outcome after the fact: if every result, and its opposite, confirms the theory, the theory is not being tested by reality, it is floating free of it. The power of a real claim lies in what it prohibits. A claim that prohibits no possible result is not a strong claim. It is an empty one dressed as a strong one.
This is not only a rule for scientists. It is a razor for any claim that asks you to believe something, and AI ROI claims ask you to believe quite a lot.
Most AI value claims fail the test
Run the common AI ROI claims through Popper's razor and most of them fall apart, not because the AI is necessarily worthless, but because the claims were constructed so that nothing could ever count against them.
"It made us more strategic." What measurable result, if it failed to appear, would prove this false? None is ever specified, so the claim can never be checked. "It future-proofs us." Against a future that has not arrived and against which no result can yet be compared, this is unfalsifiable by construction. "It freed our people for higher-value work." Did the freed time actually go to higher-value work, and did that work produce a measurable result? If those questions are never asked, the claim asserts a benefit no outcome could contradict. "We make better decisions now." Better by what measure, compared to what baseline, and how would we know if they were actually worse? Without answers, the claim is a horoscope: agreeable, unfalsifiable, and empty.
The pattern is consistent. The claims are phrased in terms that admit no disproof, and that is exactly why they feel safe to make and impossible to challenge. Notice that this is a deeper problem than measuring the wrong thing. A claim can survive the discipline of measuring an outcome and still fail here, if the outcome was defined so loosely that no result could ever have contradicted it. Falsifiability is the prior question: before you ask whether a claim was measured well, ask whether it was ever the kind of claim that could be tested at all.
Why the empty claims are the popular ones
There is a reason the unfalsifiable claims are the ones that thrive, and it is not stupidity. It is incentive. An unfalsifiable claim is the safest possible thing to assert, because it can never be shown to be wrong. The manager who says "the AI made us more strategic" has taken on no risk, because no future audit can contradict them. The one who says "the AI will cut our invoice-processing errors by 30% within two quarters" has made a real claim, a falsifiable one, and has therefore exposed themselves to being proven wrong. Human incentive quietly rewards the first person and punishes the second, even though the second is the only one saying anything of substance.
So the vagueness is not accidental. It is protective. In an environment where AI spend has to be justified and no one wants to be the person whose initiative visibly failed, the unfalsifiable claim is the rational move, and it propagates because it is comfortable for everyone: the person making it cannot be wrong, and the person hearing it gets to feel reassured. The whole exchange has the shape of accountability without any of the substance, and it will continue until someone insists that claims be made in a form that could actually fail.
The honest part: not everything real is falsifiable
It would be an overcorrection to conclude that only falsifiable, immediately measurable benefits are real, and Popper himself did not claim that unfalsifiable statements are meaningless or worthless, only that they are not scientific, not the kind of thing empirical evidence can settle. Some genuine value is legitimately hard to quantify. Morale, optionality, learning, and long-term positioning are real and can matter enormously, and a crude insistence that everything reduce to a falsifiable number this quarter would be its own kind of foolishness.
The point is not to banish the intangible. It is to stop laundering the intangible into the language of proven ROI. There is nothing wrong with saying "we believe this positions us well for a future we can't yet measure, and here is our reasoning." That is an honest, acknowledged bet. What is not honest is dressing that same bet in the costume of demonstrated return, presenting an unfalsifiable hope as though it were established fact. Call a bet a bet and a proof a proof. The damage comes from the confusion between them, when an unfalsifiable claim is granted the authority of evidence it has not earned, and capital gets allocated as though a hope had been proven.
The test to apply
The discipline reduces to a single question you can ask of any AI ROI claim before you accept it: what result, if we observed it, would prove this claim false? If there is a clear answer, a specific outcome that would falsify the claim, then the claim is real, it is being tested against reality, and it is worth taking seriously whether it turns out true or false. If there is no such answer, if no conceivable result could contradict the claim, then you are not looking at evidence. You are looking at a horoscope with a budget attached, and it should carry exactly as much weight in your capital allocation as a horoscope would.
This pairs directly with the practical discipline of measurement, defining a baseline, tracking outcomes rather than activity, and controlling for confounds, covered in how to tell if your AI is actually working. Falsifiability is the entry criterion and honest measurement is the method: first insist the claim be the kind of thing that could be proven wrong, then actually go and test it. The organizations that get real value from AI are not the ones with the most confident claims. They are the ones whose claims were specific enough to have failed, and then didn't.
FAQs
Q1. What does falsifiability mean in plain terms?
It means a claim can only tell you something real if some possible result could prove it false. The philosopher Karl Popper argued that claims which are compatible with every possible outcome, that nothing could ever contradict, convey no actual information, no matter how meaningful they sound. Applied to AI, it means a value claim is only worth trusting if you can name what result would have shown it to be wrong.
Q2. Why is an unfalsifiable ROI claim a problem if the AI might really be helping?
Because you have no way to know whether it is helping, and no way to allocate capital rationally on the basis of a claim that no outcome could ever contradict. The AI may genuinely be adding value, but an unfalsifiable claim gives you no evidence of it, so you are believing on faith while calling it ROI. The problem is not the AI, it is treating an untestable assertion as proof.
Q3. What do unfalsifiable AI claims sound like?
They tend to use language that admits no disproof: "made us more strategic," "future-proofs the business," "freed people for higher-value work," "better decisions." None of these specifies a result that, if it failed to appear, would show the claim was wrong. That missing specification is the tell. A falsifiable version would name a concrete outcome and a timeframe against which it could fail.
Q4. Doesn't this dismiss real but intangible benefits?
No. Some genuine value, like morale, optionality, or long-term positioning, is legitimately hard to quantify, and Popper did not consider unfalsifiable statements meaningless, only unproven. The point is to stop presenting those intangible bets as demonstrated ROI. It is honest to say "this is a bet on a future we can't yet measure, and here's our reasoning." It is misleading to dress that bet as established financial return.
Q5. How is this different from just measuring AI properly?
Measurement is the method; falsifiability is the prior question of whether the claim can be tested at all. A claim can even survive shallow measurement and still be unfalsifiable if its target was defined so vaguely that no result could contradict it. You first insist the claim be the kind of thing that could be proven wrong, then apply honest measurement to actually test it. Both steps are needed.
Q6. Why do vague, unfalsifiable claims spread so easily?
Because they are safe to make. A claim that can never be proven wrong exposes no one to the risk of being caught out, while a specific, falsifiable claim puts the person making it on the hook. Incentives quietly reward the vague claim and punish the specific one, even though only the specific one carries information. The vagueness is protective, not accidental.
Q7. What single question should I ask about any AI ROI claim?
Ask what result, if you observed it, would prove the claim false. If someone can name a specific outcome that would falsify it, the claim is real and testable. If no possible result could contradict it, the claim is empty regardless of how confident it sounds, and it should carry no weight in a budget decision. That one question filters most productivity theater on contact.
Q8. Can a claim be falsifiable and still be wrong?
Yes, and that is a feature, not a flaw. Falsifiable claims can turn out false, which is exactly what makes them valuable: they can be tested, and a wrong one can be discovered and discarded. The goal is not to make only claims that turn out true, it is to make claims specific enough that reality could correct them. An organization that makes falsifiable claims and occasionally proves itself wrong is learning; one that makes only unfalsifiable claims never can.
.svg)
.svg)
.svg)
