Questions and answers

Horizon Audit audits benchmark tasks before they are released. You submit tasks, and each comes back with a verdict: clean, for the exact files that were audited, or flawed, with the flaw and its proof.

Dataset vendors who prepare tasks for model labs, benchmark publishers, and teams that turn workflows into environments for agents. Model labs use the verdicts to decide which tasks to trust.

A task is flawed when its score does not measure the work. The common cases: an agent that does nothing passes, the reference solution fails, the tests check less than the instruction asks, the answer can be read from the environment, or the grading depends on something that can break.

Harbor tasks. A task is an instruction, a task.toml, an environment, a reference solution and a verifier. You send your tasks as one archive.

The published list of nine rules a task must meet before it can be audited, IS-1 to IS-9. A task that misses a rule is not judged. It comes back as "Needs from you", with what is missing.

Whether the score of the task measures the work. A clean verdict proves that no reward is given without the work, that the task can be solved as written, that a wrong answer is refused and a right answer is accepted, and that the answer cannot be found without the work. Anything else worth a look is listed beside the verdict as a flag.

Clean is a claim about what was tested. Every promise of a clean verdict held. The verdict holds for the exact files that were audited. A changed task is sent again, and it is audited again.

A finding: what the flaw is, where it is, how it was established as steps you can repeat, and the proof from the run. An engineer confirms every flawed verdict before you see it.

No. The audit says what is wrong and proves it. It does not suggest a fix, and it never edits a task to make it run. You change the task and send it again.

No. Not checked is never clean. A task that cannot be audited shows "Needs from you" and names what is needed.

The verdict is computed from the saved output of runs. An engineer confirms every flawed verdict before you see it. You never see a draft.

Yes. Your tasks and results are visible to your organisation and to the engineers who audit them. Each organisation sees only its own tasks. Runs happen in an isolated sandbox.

Access is by invitation. Write to contact@horizonanalyticslabs.com.

Horizon Analytics Labs LLC makes Horizon Audit. The company audits AI datasets and environments as a service, and Horizon Audit is its product for auditing tasks before release.