Evaluating agents honestly

Demo success rate and production success rate are different numbers, and the gap between them is where most agent projects quietly die.

Why the demo number is always better

Demos are run by the person who built the thing, on inputs they chose, with the failures they already know about mentally filtered out. None of that is dishonest. It is just not measurement.

What we measure instead

• Success on a frozen set of real inputs, collected before the system existed.
• Cost per completed task, including retries, not cost per model call.
• Failure mode distribution. Ten failures of one kind is a fix; ten failures of ten kinds is a rethink.
• How often a human had to intervene, and how long it took them to notice they needed to.

The last one is the one teams skip and the one that decides whether anyone still trusts the system in month three.

Unglamorous, and that is the point

None of this needs a bespoke evaluation platform. A frozen input set, a scoring rubric a human agrees with, and a scheduled run is enough to catch the drift that matters. Teams skip it because it is boring, then spend far longer arguing about whether the thing works.