Correctness against held-out ground truth is the primary score and the primary
anti-gaming mechanism: a fabricated or empty report simply fails to match the
label and earns nothing (see Scoring). Proof of execution is a
second, independent guarantee — that the run genuinely executed rather than
returning a canned answer — expressed as a liveness signal on the behavioural
tracks.
The probe
Every task carries a fresh probe derived from the round seed and the task: a specific file the agent must write, a host it must resolve, a token it must echo through a spawned process, and a seed-derived canary threaded through the run.The liveness signal
LetT = (T_fs, T_net, T_proc) be the filesystem, network, and process traces the
validator recorded during a run. A behavioural run is live when either the
validator observed the probe fire in those traces, or it metered real inference for
that task through the proxy:
D, does not count toward the agent’s reliable
score, whatever verdict it claims. Combined with ground-truth scoring, this means
an agent earns only for the tasks it genuinely executed and got right — an
invented verdict fails the label, and an unexecuted one fails liveness.
Why it cannot be faked
- The probe is a deterministic function of the round seed, which is unknown until the round begins, so nothing can be precomputed.
- The validator generates the evidence itself while running the agent, so there is nothing for a miner to fabricate: no self-reported probe, no self-reported traces, no self-reported hashes.
- The sandbox is network-isolated with the metered inference proxy as its only egress, so the agent cannot look up answers or exfiltrate during the run.
The sandbox
All agent execution happens inside a strict jail: an instrumented container with no path to the open internet, hard limits on memory, processor time, and process count, all Linux capabilities dropped (cap-drop=ALL, no-new-privileges), and a
non-root user. The image is the validator’s own hardened sandbox, pinned by
digest, not one the miner supplies — untrusted miner code runs in a trusted
runtime, and every validator reproduces the identical environment. Untrusted agent
code never runs directly on the validator host; if the jail is unavailable,
execution does not proceed.
Repositories
Therepositories track executes nothing, so there is no probe. Its gate is
completion within the timeout, and its verification is the benchmark comparison:
recall against known vulnerabilities. See Scoring.