A CFO signs their name to numbers. That is the whole job, stripped to its center. And every number a CFO stands behind rests on something underneath it: a source document, a calculation a reasonable person could reproduce, an approval, a trail that lets someone else arrive at the same figure by the same road. This is not bureaucracy. It is the entire basis on which a financial statement is trusted. A number without evidence behind it is not a number. It is an assertion.

Which is why the arrival of AI in the finance function raises a problem that has nothing to do with whether the AI is any good. An AI can be accurate and still be unexplainable, and for a CFO those are two completely different properties. Accuracy is whether the answer is right. Explainability is whether you can show why it is right. Finance has always required the second one, and a model that cannot provide it is asking you to sign your name to the one figure on the page that has no evidence underneath it.

"The model said so" is the same answer you would never accept

Picture a junior analyst handing you a number and, when asked how they got it, saying "trust me." You would not accept it, and not because you distrust the analyst. You would reject it because an unsupported figure cannot go on a statement no matter who produced it. The number has to be reconstructable by someone other than the person who made it, or it is not yet real.

An unexplainable AI gives you exactly that answer, dressed in more impressive clothing. "The model calculated it" is "trust me" with a technology budget behind it. The dressing is doing a lot of work, because a model carries an aura of objectivity that a nervous junior analyst does not. But the underlying gap is identical. If no one can trace how the figure was produced, you have not gained an efficient analyst. You have introduced an authoritative one whose work you cannot check, which is a more dangerous thing to have than a slow one whose work you can.

Where the exposure actually lands

This is not a philosophical worry. It shows up in the specific rooms a CFO is accountable in.

The first is the audit. Auditors work under standards that are not optional and not new. PCAOB standards on audit evidence require sufficient, appropriate evidence to support any conclusion, and the profession has been direct about what that means once AI enters the workflow: an output that feeds a financial statement has to be traceable back to its source data, explainable to a reviewer, and defensible under inspection. Practitioners have put the point bluntly, that "the platform calculated it" is not an acceptable answer on a provided-by-client list or in response to a management representation request. COSO, which frames the control standards these audits lean on, has warned that generative AI's opaque reasoning can jeopardize reporting integrity outright. So an unexplainable model does not just create a technical debt. It creates an audit finding waiting to happen, on a schedule you do not control.

The second room is the board and the lenders. Covenants often require clean annual audits. An unauditable figure is not a private inconvenience between you and your model vendor. It can become a qualified opinion, a covenant conversation, and a higher cost of capital, which is a strange price to pay for a tool you bought to save money.

The third is the regulator, and here the direction of travel is clear even if the dates are still moving. The EU AI Act classifies a range of financial models, credit scoring prominent among them, as high-risk and subject to documentation, transparency, and human-oversight obligations. Those high-risk obligations were set to apply from August 2026, and under the Digital Omnibus agreement reached in May 2026 they are being deferred toward December 2027, pending formal adoption. The exact date is genuinely unsettled as of this writing. The direction is not. Explainability is becoming a legal expectation for financial AI, and building for the requirement is the safe move regardless of which deadline finally sticks.

The honest part: not every AI has to explain itself

It would be overreach to claim every use of AI in a finance function must clear the explainability bar, and pretending otherwise would just get the argument dismissed by anyone who has actually deployed these tools. Plenty of valuable AI in finance never touches a reported number. A model that drafts a first version of a memo, routes an invoice to the right approver, or summarizes a contract for a human who then does the real work is an assistant, not a source of financial assertions. Holding those uses to full audit-grade explainability would be a tax with no benefit.

The distinction that matters is not how sophisticated the AI is. It is whether its output lands in the financials. That gives a clean test.

The Materiality Test

Materiality to the financials is the practical test: If an AI influences a figure that reaches your statements, an estimate, a classification, an allocation, an accrual, a forecast the board or an auditor will see, it has to meet the explainability bar. If it does not touch a reported number, it can be judged on ordinary usefulness like any other productivity tool. The test is not "is this AI important." It is "does this AI's output become something I sign." Run every proposed use through that one question and the governance sorts itself into two piles: the assistants, which you can adopt freely, and the ones producing financial assertions, which you cannot adopt until they can show their work.

This also keeps the discipline proportionate. You are not trying to make every model in the building interpretable. You are drawing a bright line around the outputs that become your representations, and refusing to let anything cross it that cannot be traced, reproduced, and defended.

What this asks of a CFO

The requirement, then, is not to fear AI or to slow its adoption in the parts of the function where it carries no reporting risk. It is to hold anything that touches the financials to the same standard you have always held every other input: show the work. For an AI that means the output has to trace to identifiable source data, the logic has to be documented and stable rather than silently shifting between runs, and a competent person has to be able to walk an auditor from input to output without arriving at "the system did it." An explanation that changes each time you ask for it, which some popular interpretability techniques produce, is not an explanation you can put in a workpaper.

This is the ordinary discipline finance already applies to every other input it relies on, extended to a new and persuasive one. The CFOs who get value from AI over the next several years will be the ones who insisted, early and without apology, that anything touching the financials be able to explain itself. The rest will find out what an unexplainable number costs at the worst possible moment to find out, which is in an audit, in front of a board, with their name already on it.

FAQs

Q1. What is the difference between an accurate AI and an explainable one?
Accuracy is whether the output is correct. Explainability is whether you can demonstrate why it is correct, by tracing it to source data and reproducible logic. Finance requires both, because a figure you cannot support cannot go on a statement even if it happens to be right. An accurate but unexplainable model gives you a number you are not able to defend.

Q2. Isn't "the model calculated it" good enough if the model is reliable?
No, and the audit profession has said so directly. Reliability is not the same as evidence. An auditor testing your numbers needs to see how a figure was produced, and "the platform calculated it" does not satisfy an audit-evidence standard. The reliability of the model does not transfer to you as documentation you can show under inspection.

Q3. Where is the line between AI that needs to be explainable and AI that doesn't?
Materiality to the financials is the practical test. If an AI influences a figure that reaches your statements, an estimate, a classification, a forecast the board sees, it must meet the explainability bar. If it does not touch a reported number, it can be judged on ordinary usefulness like any other tool.

Q4. Does the EU AI Act apply to us if we are not in Europe?
It can, because the regulation reaches AI systems whose outputs affect the EU market regardless of where the provider sits, similar to how GDPR operates. Even setting jurisdiction aside, the Act signals where financial-AI expectations are heading everywhere. The specific high-risk deadline is in flux, having moved from August 2026 toward late 2027, but the explainability expectation itself is not in doubt.

Q5. Our auditors haven't asked about AI yet. Is this really urgent?
The question is coming whether or not your auditors have raised it yet. Audit standards on evidence already apply to any AI output that feeds a financial statement, and inspection priorities are moving toward AI explicitly. Building traceability in while adoption is still small is far cheaper than retrofitting it after an unexplainable model is already embedded in your close.

Q6. Doesn't requiring explainability mean we can't use the most powerful models?
Not for most finance uses. It means the most opaque models should not be the ones producing reported figures without compensating controls. You can still use powerful models as assistants that support human work, and reserve the explainability requirement for outputs that become financial assertions. The constraint is on placement, not on capability.

Q7. What does an acceptable explanation actually look like?
It traces the output to identifiable source data, rests on logic that is documented and stable rather than shifting invisibly between runs, and can be walked through by a competent person from input to result. A useful warning sign: if the explanation changes each time you request it, it is not something you can rely on as evidence, because reproducibility is part of what makes an explanation defensible.

Q8. Who owns this risk inside the company?
The CFO, because the CFO owns the numbers. Data science may build or select the models and IT may run them, but the person who signs the statements owns the requirement that everything on them be supportable. That makes explainability a finance governance question with a finance owner, not a technical decision to be delegated and forgotten.

Q9. How do we start without stalling our AI plans?
Inventory where AI currently touches reported figures, apply the materiality test to sort those uses from the harmless ones, and require traceability only where outputs become financial assertions. That lets adoption continue everywhere it carries no reporting risk while protecting the specific places where an unexplainable number would actually cost you.