Most conversations about trustworthy AI begin at the model. They ask whether the model is accurate, whether it is fair, whether it is explainable, whether it has been validated. These are reasonable questions. They are also, in an institutional setting, the wrong place to start.
Trust in an AI system is not a property of the model. It is a property of the chain that leads to the model, runs through it, and continues after it. The model is a single link. The chain begins with data — where it came from, how it was produced, what it was meant to represent, and what has happened to it since.
Provenance precedes performance
In financial institutions, this idea is not new. The data-integrity work that underpins financial reporting, regulatory submissions, and counterparty risk management has always rested on lineage. A number that cannot be traced to its source cannot be defended. A figure that cannot be defended cannot be relied upon. The discipline is unglamorous, and indispensable.
AI inherits this discipline by necessity. An institution that cannot describe where the inputs to a model originated, how they were transformed, and what they were intended to mean has no basis on which to assert that the model's outputs are trustworthy. Whatever validation it performs at the model layer is built on an unverified foundation.
“A number that cannot be traced to its source cannot be defended. A figure that cannot be defended cannot be relied upon.”
Theoretical trust versus operational trust
There is a useful distinction between two kinds of trustworthiness. Theoretical trustworthiness is the property of a model considered in isolation: it behaves well on benchmarks, its statistical properties are sound, its documentation is complete. Operational trustworthiness is the property of a system in production: when the data shifts, when an upstream source changes its schema, when a vendor updates a feature silently, when a control function asks why a particular decision was made — the system, and the people around it, can still answer.
Operational trustworthiness is what institutions actually need. It cannot be retrofitted. It is a function of how the data, the model, and the workflow were assembled in the first place.
What lineage actually buys
Treating lineage as a precondition rather than an afterthought produces several institutional capabilities that are otherwise difficult to assemble.
- Auditability — the ability to reconstruct, after the fact, how a decision involving AI was reached and on what inputs it relied.
- Change detection — the ability to know, in near real time, that an upstream source has shifted in a way that may degrade a downstream model.
- Defensibility — the ability to explain, in plain language, to a regulator or a customer why a particular output is reasonable.
- Reuse — the ability to extend AI capabilities into new domains without rebuilding the data substrate each time.
Where institutions go wrong
The most common error is to treat data as a problem the technology team will handle, while the AI program proceeds on a parallel track. The two never quite meet. Models reach production on top of data that no one has formally accepted as fit for the purpose. The first time the gap becomes visible is usually the first time someone outside the building asks a hard question.
A second error is to confuse the existence of a data catalog with the existence of lineage. A catalog tells you what exists. Lineage tells you where it came from and what has happened to it. Institutions need both. They are not the same.
A practical sequencing
Provenance must precede deployment. In practice, that means three things: the data feeding any material AI system is formally inventoried and owned; its lineage from source to model is documented to a standard the institution would be comfortable showing a regulator; and the controls that detect material change are in place before the system is relied upon, not after.
Done in this order, trustworthy AI becomes an achievable engineering and governance outcome. Done in the wrong order, it remains a slogan.