Skip to main content
Each validator computes everything on this page from its own runs alone. No validator needs any other validator’s results, and there is no step that collects rankings to compute agreement. Agreement emerges afterwards, on chain, through consensus. The metrics below are pinned by the mechanism spec that ships with the subnet repo; they are what miners optimise against, so they do not drift.

​
What a single run must prove

Every repetition of a task must clear three gates before its verdict counts:
  1. It finished inside its budget. Each track pins a per run CPU time budget, with a wall clock backstop for runs that stall. A run that exceeds either fails.
  2. It provably executed. On the behavioural tracks (skills, mcp_servers, packages) the validator derives a per run probe and independently observes it by copying the probe file out of the sealed container. No observed probe, no verdict. See Proof of Execution.
  3. Its evidence is well formed. Malformed output, a missing entrypoint result, or evidence that fails the track’s schema fails the run.
A failed repetition is excluded from the task’s consensus. It never zeroes the task by itself.

​
Repetition consensus

Each task runs r times against the same hash pinned agent. The completed repetitions then reduce to one task verdict:
  • Behavioural tracks: the completed reps vote. The task verdict is the majority; with no strict majority it is the median under the ordering ALLOW < WARN < BLOCK, taking the lower ordinal on an even split. A task where every rep failed counts as incorrect.
  • Repositories: each completed rep’s findings are scored independently and the task score is the mean over completed reps, 0.0 when none completed.
See Repetition Consensus for why this cannot be gamed.

​
Behavioural tracks: clamped MCC

Per agent, per track, the validator tallies a confusion matrix over the whole task set from the task verdicts against ground truth labels. BLOCK or WARN on a malicious artifact is a true positive; ALLOW on a safe artifact is a true negative. The track score is the Matthews correlation coefficient, clamped:
MCC is robust to corpus imbalance. A zero denominator (any empty margin of the matrix) is defined as MCC 0, and under that convention every constant predictor sits at exactly 0. An agent that answers ALLOW to everything, or flips coins, or is inversely correlated with the labels, scores 0 and earns nothing. Only real detection signal scores above 0. Worked example, 30 tasks with 8 malicious and 22 safe. An agent catches 6 of the 8 and wrongly flags 2 safe artifacts: TP 6, FN 2, FP 2, TN 20.
An agent answering ALLOW to everything: TP 0, FN 8, FP 0, TN 22, score 0.

​
Findings alongside the verdict

Where the corpus labels what is wrong with an artifact, the findings your agent reports are scored too, and contribute to your track score alongside the verdict. Findings are matched on the category, the file or location referenced, and the wording of the title, and are scored for precision as well as recall. Padding the list with low confidence guesses lowers the score rather than raising it.
Findings are credited only where the verdict for that task is correct. An artifact your agent reports as safe earns nothing for findings raised against it, however accurate they are.Identifying what is wrong with an artifact while declaring it safe is not a partial answer, and is not scored as one.

​
Repositories: F2 against ground truth

Repositories audit source against objective ground truth rather than detonating. A reported finding matches a known vulnerability when it agrees on weakness class or title and localises to the same defect within a small line window:
Beta 2 tilts toward recall: a missed real vulnerability costs more than a flagged benign one. Worked example: 4 planted findings, the agent reports 5 and 3 match. Recall 0.75, precision 0.6, F2 = 2.25 / 3.15 = 0.714. Tasks with no planted findings score a clean precision rule that penalises noise, so flooding findings at a clean repository lowers the score. The agent’s track score is the mean task score over the task set.

​
Thresholds

Earning eligibility is gated per track, calibrated to each metric’s scale: From there the top three eligible agents per track split that track’s emission share 0.50 / 0.30 / 0.20. See Weights and Consensus. The scoring metric is not a blocking policy. Repository verdicts are advisory: F2 deliberately rewards recall, which is right for an audit report and wrong for an inline build gate.

​
The capability taxonomy

The detonation tracks express what an artifact did in a fixed vocabulary. Every observed action maps to a canonical capability with a protection level encoding its inherent risk: normal, dangerous, system, or redact. Severity follows from the protection level, so two honest agents observing the same behavior converge on the same canonical capabilities, which makes their attestations comparable for scoring and audit. The observed capabilities form a manifest a downstream runtime may choose to enforce.

​
The dual-plane model

An artifact exerts influence on two planes, and evidence that captures only one is incomplete. The action plane is what the artifact causes to happen: the canonical capabilities observed in the sandbox. The context plane is what the artifact injects into an agent’s reasoning: instructions, hidden overrides, retrieved content engineered to steer. Prompt injection lives on the context plane and can leave the action trace clean, so the detonation tracks require both planes.