Horizon Research · · 3 min read
A clean verdict is a claim about what was tested. If a task was not tested, there is no claim to make, so the task is not clean. Horizon Audit holds to that in every case: a task that was not checked is never counted as clean.
This sounds obvious. It is also the rule that is easiest to break without anyone seeing it.
Three answers, not two
An audit that can only say "pass" or "fail" has a problem with every task it could not run. It must put the task somewhere. If it puts the task with the failures, the author is told about a flaw that nobody found. If it puts the task with the passes, a reader is told that the task was tested.
The second mistake is the common one, because it is quiet. A report that says "412 tasks, 9 flawed" reads as "403 clean". It may mean "380 clean and 23 that never ran".
So there are three answers.
- Clean. Every promise of a clean verdict held.
- Flawed. A flaw was proved, and the proof is attached.
- Needs from you. The task cannot be checked until something is supplied. This is not a verdict.
The third is shown as what it is. It is never added to the first.
What "checked" means
A task is clean only when all of this was proved.
- No reward without the work. An agent that does nothing, or only looks busy, earns nothing.
- The task can be solved as written. The task's own solution earns full reward in the task's own environment.
- A wrong answer is refused. An answer that is close but wrong does not pass.
- A right answer is accepted. A correct answer passes, also when it does not look like the author's.
- The answer cannot be found without the work. The answer is not there to be read from the environment.
If one of these could not be established, the task is not clean. Code enforces this. It is not left to the judgement of whoever is looking at the task that day.
A draft is not a verdict
A verdict rests on what was measured, or on what an engineer has confirmed. An engineer confirms every flawed verdict before you see it. You never see a draft.
Clean has a scope
A clean verdict holds for the exact files that were audited. Each task is pinned to a digest of its files when it arrives. If one file changes, the digest changes, and the earlier verdict says nothing about the new files. The changed task is sent again and audited again.
It also holds only for what was tested. Anything else worth a look is listed beside the verdict as a flag, so that a reader can see what was proved and what was only noted. A flag never decides a verdict.
Why this matters to a reader of scores
A model's score on a benchmark is trusted as far as its tasks are. When a set says its tasks were audited, the useful question is not how many were flawed. It is how many were checked, and what "checked" covered.
A verdict that answers that question is worth more than a larger number of tasks that passed by default. What each verdict proves is on what an audit returns.