The phrase 'human-in-the-loop' has done a great deal of work in the last two years. It has reassured boards, satisfied policy documents, and given comfort to regulators. It has also, in many institutions, come to mean very little.
Putting a human at the end of an AI workflow is not oversight. It is, more often, the appearance of oversight — a checkbox at the terminus of a process whose substance has already been decided by the system. Meaningful human oversight of generative AI requires a more honest design.
Why the old frameworks strain
Model-risk management as it is practised today was built for a different kind of model. The classical paradigm assumes a model with a stable purpose, a defined input space, a measurable output, and a validation regime that can be run once and refreshed periodically. Generative systems fit none of these assumptions cleanly. They are general-purpose. Their inputs are open-ended. Their outputs are linguistic. Their behaviour shifts when the underlying foundation model is updated, when a prompt is changed, when a tool is added.
Existing frameworks are not wrong; they are insufficient. They need to be extended, not replaced — but the extension is non-trivial, and pretending it is not is the source of most of today's oversight theatre.
“Putting a human at the end of an AI workflow is not oversight. It is, more often, the appearance of oversight.”
Where institutions get it wrong today
Three failure patterns are common, and worth naming plainly.
- Oversight as ratification — a reviewer signs off on an output they had no meaningful basis to challenge, because the cost of overruling the system is higher than the cost of accepting it.
- Oversight as bottleneck — humans are inserted at every step, the workflow grinds, and the institution quietly removes them later in the name of efficiency, leaving nothing behind.
- Oversight as documentation — the substantive judgement happens informally, while the formal record is constructed afterwards to satisfy the policy.
Each of these patterns looks like governance from the outside and is hollow from the inside. None of them survives an examiner who asks to see the reviewer's reasoning.
What meaningful oversight looks like
Real oversight of generative systems is designed into four layers at once.
- Workflow design — the human is positioned at a step where their judgement is genuinely additive, with the information and time required to exercise it.
- Decision rights — the institution has specified which classes of output a human may approve, which require escalation, and which the system is not permitted to produce at all.
- Review quality — the institution measures whether reviewers actually catch errors, not merely whether they signed.
- Management reporting — oversight outcomes are surfaced to senior leadership in a form that allows pattern detection, not just attestation.
Taken together, these layers produce something that an outsider can recognise as oversight. They also produce something an insider can defend.
Oversight as an operating discipline
The redefinition we propose is straightforward. Oversight is not a person; it is a property of a system. It is the set of design choices, control points, and management practices that together preserve human judgement over consequential outcomes, even as the technology underneath those outcomes becomes more capable and more autonomous.
Designed this way, oversight becomes compatible with the speed and scale that generative systems promise. Designed the old way, it becomes the first thing the institution sacrifices when the pressure to scale arrives — and the first thing it wishes it had kept when something goes wrong.