Skip to content
Horizon Audit

The intake standard

The intake standard is the published list of what a task must have before it can be audited. It has nine rules, IS-1 to IS-9. A task that misses a rule is not judged: it comes back as "Needs from you", with what is missing.

Version v0 · The ids are never renumbered. A rule that changes keeps its id.

  1. IS-1

    An instruction

    Requires
    instruction.md exists and has text.
    Why
    With no instruction there is nothing to hold the grader against.
  2. IS-2

    A verifier

    Requires
    tests/test.sh exists. A Windows task has tests/test.bat instead.
    Why
    The verifier is the grader. A task with none cannot be scored.
  3. IS-3

    A configuration that parses

    Requires
    task.toml parses.
    Why
    The configuration says how the task runs. A file that does not parse says nothing.
  4. IS-4

    An environment

    Requires
    One of environment/Dockerfile, environment/docker-compose.yaml, or docker_image under [environment] in task.toml.
    Why
    Every run of the audit happens inside the environment the task declares.
  5. IS-5

    A reference solution

    Requires
    solution/solve.sh exists, and runs something.
    Why
    The reference solution shows that the task can be solved as written.
  6. IS-6

    An environment that builds

    Requires
    The environment builds today, from what the task declares.
    Why
    A task that no longer builds cannot be run by anyone, model or auditor.
  7. IS-7

    Every secret declared and supplied

    Requires
    Every key or outside account the task needs is declared in task.toml as ${NAME}, and a value is supplied for each name.
    Why
    An undeclared secret makes a run fail for a reason that is not in the task.
  8. IS-8

    No access to the host

    Requires
    environment/docker-compose.yaml, when the task has one, uses only settings that keep the task inside its sandbox.
    Why
    A task is run as it is written, so it must be safe to run as it is written.
  9. IS-9

    Resources within the limits

    Requires
    cpus, memory_mb and storage_mb under [environment] are within the limits of the audit sandbox.
    Why
    A task that asks for more than the sandbox has cannot be run as written.
Read how it works