What an audit returns
You send tasks. Each one comes back clean, or flawed with its proof, or with a line that says what it needs from you. This page says what each result means.
What you send
Harbor tasks, as one archive. A task is an instruction, a task.toml, an environment, a reference solution and a verifier. Each task is held to the published intake standard: nine rules, IS-1 to IS-9. Each task is pinned to a digest of its files, and nothing runs until you confirm the list.
What comes back
Each task comes back with one of three results.
Clean
Every promise of a clean verdict held. The verdict holds for the exact files that were audited.
Flawed
A flaw that can change a realistic score was proved. You get the flaw, where it is, and the proof.
Needs from you
The task cannot be audited until you supply something. This is not a verdict, and it is never counted as clean.
What a clean verdict proves
Clean is a claim about what was tested. A clean verdict makes these promises about the task.
No reward without the work.
An agent that does nothing, or only looks busy, earns nothing.
The task can be solved as written.
The task's own solution earns full reward in the task's own environment.
A wrong answer is refused.
An answer that is close but wrong does not pass.
A right answer is accepted.
A correct answer passes, also when it does not look like the author's.
The answer cannot be found without the work.
The answer is not there to be read from the environment.
Anything else worth a look is listed beside the verdict as a flag. A flag never decides a verdict.
Tasks that say the same thing stand out.
Each task shows how close its instruction is to the closest other task you sent with it. It is information only: it never changes a verdict.
What you get when a task is flawed
A flawed verdict comes with a finding. No fix is suggested: you change the task and send it again.
- Where the flaw is
- The part of the task that holds the flaw.
- How it was established
- Steps you can repeat, with the saved output of the run as the proof.
- What it does not claim
- The limits of the finding, so that it is not read as more than was proved.
Who stands behind the verdict
The verdict is computed from the saved output of runs. An engineer confirms every flawed verdict before you see it. You never see a draft.