The quality chain — the hard lesson about model output
The agents produced fine prose and wrong figures. Every deviation sounded plausible. The hard lesson: model output must never reach a user unchecked.
The chain ran. Reports were readable. And on a Wednesday morning there was an amount that was simply wrong. Not slightly off — entirely wrong. The justification was convincingly written, referencing the right contract and the right invoice. But the number was a fabrication.
What we discovered
The model was excellent at summarising what it saw. It was equally excellent at summarising what it did not see: missing figures were filled in with plausible values, in the same confident tone as the rest. In one case the model double-counted a correction. In another it read a percentage as an absolute amount. In both cases the surrounding text was flawless.
Each error was — in isolation — understandable. Added up, they were unacceptable. A controller can manually check one report. Not a thousand.
What we built
We introduced a quality chain that ran before publication. Not a filter on style or tone — a filter on numbers.
- Figure-exact. Every quoted amount had to appear verbatim in the source data. Otherwise: blocked.
- Elimination. Every conclusion had to explicitly rule out an alternative plausible explanation. No "probably" without the counter-hypothesis.
- Miss check. The chain actively searched for what should have been there and was not — no silent omission.
- Maturity. New signal types went through a learning period: first shadow, then recommendation, only then report.
And the most important rule: model output that did not pass this gate did not reach the user. Not filtered, not corrected — blocked. Better no answer than a wrong answer that sounds convincing.
The real lesson
The greatest risk in an agent system is not that the model fails. It is that the model fails convincingly. Every reporting layer you add without verification widens the gap between tone and truth. The quality chain was the most expensive part of the platform and at the same time the most indispensable.
THE LESSON A model that lies convincingly is more dangerous than one that fails visibly. Verification is not a luxury — it is the permission to publish at all.