A model that is right most of the time, and cannot tell you which times, is not a decision tool. It is a suggestion engine. The distinction matters enormously once a human being is accountable for what happens next.

This is the uncomfortable position a lot of AI deployments find themselves in. The output is fluent, specific and confident. It is also unfalsifiable at the point of use — the person reading it has no way to check it without redoing the work themselves, which defeats the purpose entirely.

Interpretability and explanation are not the same thing

These two words get used interchangeably, and conflating them causes real design mistakes.

Interpretability is a property of the model. A linear regression with a handful of coefficients is interpretable: you can read the mechanism directly. A decision tree of modest depth is interpretable. A large neural network is not, in any practical sense, and no amount of engineering will make its internal weights legible to the person using the product.

Explanation is a property of the output. It is the account that travels alongside a particular answer, saying what that answer rests on. Crucially, you can build good explanations on top of a model that is not itself interpretable — and for most useful systems, that is the only option available.

The practical consequence: stop waiting for models to become transparent, and start designing the explanation layer as a first-class part of the system.

What an explanation actually has to contain

A genuinely useful explanation answers four questions, and most systems that claim explainability answer only the first:

  • What did you conclude? The answer itself, stated plainly.
  • What is it based on? The specific inputs — documents, records, events — that drove this particular output, not a general description of the training corpus.
  • How confident are you, and why? A calibrated signal, with the reason for uncertainty named: thin evidence, conflicting sources, an input outside the range the system handles well.
  • What would change your mind? The conditions under which the conclusion would flip. This is the one almost nobody ships, and it is the one that most rewards the reader's attention.

Retrieval-based architectures make the second question tractable in a way that older approaches did not. When an answer is assembled from specific retrieved passages, those passages are the citation — they can be surfaced, linked and checked. That is a meaningful structural advantage, and it is worth designing around.

Where explainability goes wrong

Three failure modes recur.

Explanations that are generated rather than traced. If the system produces its answer by one route and its justification by another, the justification is a plausible-sounding narrative that may have no causal relationship to the output. This is worse than no explanation, because it manufactures confidence. The explanation has to be a record of what actually happened, not a story written afterwards.

Confidence presented as precision. A score rendered to two decimal places implies a measurement. If the underlying number is not calibrated — if a stated 0.9 does not correspond to being right nine times in ten — the precision is decorative and actively misleading.

Explanations nobody can act on. A feature-attribution chart is an explanation for a data scientist. It is noise for an operations manager deciding whether to act before end of day. The explanation has to be legible to the person actually holding the decision.

A product requirement, not a feature

The case for treating this as foundational rather than additive is straightforward: explainability determines whether the product can be adopted at all in the settings where it would be most valuable.

In any domain with professional accountability — clinical, financial, legal, safety-critical — someone signs off on the decision. That person cannot delegate responsibility to a system that will not account for itself. Faced with an unexplained output, they will either ignore it or verify it manually, and both outcomes destroy the value the system was supposed to create.

There is a regulatory direction of travel here too. Across multiple jurisdictions, rules governing automated decision-making are converging on similar themes: people affected by a decision should be able to learn the basis for it, and organisations deploying these systems should be able to demonstrate how they work. The specifics differ and continue to evolve, but the direction is consistent enough to design for.

The engineering implication is that explainability is not something to retrofit. Traceability has to be carried through the pipeline from ingestion onward — which sources were used, which transformations were applied, which records contributed to which output. Systems that did not record this from the start generally cannot reconstruct it later.

Build it in at the foundation, and the explanation is close to free. Bolt it on afterwards, and you are usually writing fiction.