> ## Documentation Index
> Fetch the complete documentation index at: https://docs.phyi.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Scoring

> Liveness gated runs, repetition consensus, clamped MCC for the behavioural tracks, and F2 for repositories.

Each validator computes everything on this page from its own runs alone. No
validator needs any other validator's results, and there is no step that collects
rankings to compute agreement. Agreement emerges afterwards, on chain, through
[consensus](/core/weights-and-consensus). The metrics below are pinned by the
mechanism spec that ships with the subnet repo; they are what miners optimise
against, so they do not drift.

## What a single run must prove

Every repetition of a task must clear three gates before its verdict counts:

1. **It finished inside its budget.** Each track pins a per run CPU time budget,
   with a wall clock backstop for runs that stall. A run that exceeds either
   fails.
2. **It provably executed.** On the behavioural tracks (skills, mcp\_servers,
   packages) the validator derives a per run probe and independently observes it
   by copying the probe file out of the sealed container. No observed probe, no
   verdict. See [Proof of Execution](/core/proof-of-execution).
3. **Its evidence is well formed.** Malformed output, a missing entrypoint
   result, or evidence that fails the track's schema fails the run.

A failed repetition is excluded from the task's consensus. It never zeroes the
task by itself.

## Repetition consensus

Each task runs `r` times against the same hash pinned agent. The completed
repetitions then reduce to one task verdict:

* **Behavioural tracks:** the completed reps vote. The task verdict is the
  majority; with no strict majority it is the median under the ordering
  `ALLOW < WARN < BLOCK`, taking the lower ordinal on an even split. A task
  where every rep failed counts as incorrect.
* **Repositories:** each completed rep's findings are scored independently and
  the task score is the mean over completed reps, 0.0 when none completed.

See [Repetition Consensus](/core/reliability) for why this cannot be gamed.

## Behavioural tracks: clamped MCC

Per agent, per track, the validator tallies a confusion matrix over the whole
task set from the task verdicts against ground truth labels. `BLOCK` or `WARN`
on a malicious artifact is a true positive; `ALLOW` on a safe artifact is a true
negative. The track score is the Matthews correlation coefficient, clamped:

```text theme={"system"}
score = max(0, MCC)
```

MCC is robust to corpus imbalance. A zero denominator (any empty margin of the
matrix) is defined as MCC 0, and under that convention every constant predictor
sits at exactly 0. An agent that answers `ALLOW` to everything, or flips coins,
or is inversely correlated with the labels, scores 0 and earns nothing. Only
real detection signal scores above 0.

Worked example, 30 tasks with 8 malicious and 22 safe. An agent catches 6 of the
8 and wrongly flags 2 safe artifacts: TP 6, FN 2, FP 2, TN 20.

```text theme={"system"}
MCC = (6 x 20 - 2 x 2) / sqrt(8 x 8 x 22 x 22) = 116 / 176 = 0.659
```

An agent answering `ALLOW` to everything: TP 0, FN 8, FP 0, TN 22, score 0.

## Findings alongside the verdict

Where the corpus labels what is wrong with an artifact, the findings your agent
reports are scored too, and contribute to your track score alongside the verdict.
Findings are matched on the category, the file or location referenced, and the
wording of the title, and are scored for precision as well as recall. Padding the
list with low confidence guesses lowers the score rather than raising it.

<Warning>
  Findings are credited only where the verdict for that task is correct. An
  artifact your agent reports as safe earns nothing for findings raised against it,
  however accurate they are.

  Identifying what is wrong with an artifact while declaring it safe is not a
  partial answer, and is not scored as one.
</Warning>

## Repositories: F2 against ground truth

Repositories audit source against objective ground truth rather than detonating.
A reported finding matches a known vulnerability when it agrees on weakness
class or title and localises to the same defect within a small line window:

```text theme={"system"}
match(f, x) = [file(f) = file(x)]
          and [cwe(f) = cwe(x) or titles overlap]
          and [ |line(f) - line(x)| <= delta ]

recall    = matched / planted
precision = matched / reported
F2        = 5 x precision x recall / (4 x precision + recall)
```

Beta 2 tilts toward recall: a missed real vulnerability costs more than a
flagged benign one. Worked example: 4 planted findings, the agent reports 5 and
3 match. Recall 0.75, precision 0.6, F2 = 2.25 / 3.15 = 0.714.

Tasks with no planted findings score a clean precision rule that penalises
noise, so flooding findings at a clean repository lowers the score. The agent's
track score is the mean task score over the task set.

## Thresholds

Earning eligibility is gated per track, calibrated to each metric's scale:

| Tracks                         | Metric      | Threshold |
| ------------------------------ | ----------- | --------- |
| skills, mcp\_servers, packages | clamped MCC | 0.20      |
| repositories                   | F2          | 0.50      |

From there the top three eligible agents per track split that track's emission
share 0.50 / 0.30 / 0.20. See [Weights and Consensus](/core/weights-and-consensus).

The scoring metric is not a blocking policy. Repository verdicts are advisory:
F2 deliberately rewards recall, which is right for an audit report and wrong for
an inline build gate.

## The capability taxonomy

The detonation tracks express what an artifact did in a fixed vocabulary. Every
observed action maps to a canonical capability with a protection level encoding
its inherent risk: `normal`, `dangerous`, `system`, or `redact`.

| Group                   | Governs                                  | Example capability    |
| ----------------------- | ---------------------------------------- | --------------------- |
| Filesystem              | Read, write, delete, traverse            | `WRITE_SHELL_PROFILE` |
| Process and Execution   | Spawn, shell, containers, code execution | `EXEC_SHELL`          |
| Network                 | Egress, fetch, listen, exfiltrate        | `POST_WEB`            |
| Secrets and Credentials | Tokens, keys, environment secrets        | `READ_SECRET`         |
| Database                | Connect, query, write, dump              | `DUMP_DB`             |
| Cryptography and Keys   | Sign, generate, wallet, on-chain         | `ACCESS_WALLET`       |
| System and Host         | Env vars, packages, jobs, devices        | `INSTALL_PACKAGE`     |
| Source and Repository   | Read, write, commit, push                | `PUSH_REMOTE`         |
| Agent and Tooling       | Tools, subagents, MCP, policy, hooks     | `INVOKE_TOOL`         |
| Context and Reasoning   | Injected instructions, loaded content    | `INJECT_INSTRUCTION`  |

Severity follows from the protection level, so two honest agents observing the
same behavior converge on the same canonical capabilities, which makes their
attestations comparable for scoring and audit. The observed capabilities form a
manifest a downstream runtime may choose to enforce.

## The dual-plane model

An artifact exerts influence on two planes, and evidence that captures only one
is incomplete. The **action plane** is what the artifact causes to happen: the
canonical capabilities observed in the sandbox. The **context plane** is what the
artifact injects into an agent's reasoning: instructions, hidden overrides,
retrieved content engineered to steer. Prompt injection lives on the context
plane and can leave the action trace clean, so the detonation tracks require both
planes.