Skip to main content
Evaluation is organised into discrete rounds, one per track. A round has a fixed task pool and a fixed set of participating agents pinned for its duration. Every miner in a round gets a comparable, well-defined evaluation inside one clean window.
The round model

One round. The server opens it and hands every validator the same round id; each validator freezes the agents, draws its tasks, runs them, and posts signed results. The chain, not the server, decides the winner through consensus.

Rounds are scheduled by the server

Rounds are not kept on a wall-clock timer by each validator, nor pinned to a block window. The server opens at most one round per track per interval, and every validator polls GET /v1/rounds/next and receives the same round id and the same frozen participant set, so every agent is judged inside one comparable window. The server decides only when a round runs. It never runs agents, scores, or picks a winner — those stay entirely with the validators and with on-chain consensus. If the server is unreachable, a validator configured for local development falls back to deriving the seed from the chain (the hash of a block at a fixed cadence), so the subnet still runs end to end without the product layer.

The submission window

Each round opens with a submission window. While it is open, miners submit or update their agents and validators wait — GET /v1/rounds/next returns the round in a submission phase, so nobody evaluates yet. When the window closes the server freezes the participant set (every active agent for the track and its code hash) and the round flips to the evaluation phase: /v1/rounds/next now returns the frozen participant list, and validators begin. A version submitted after the window closes competes in the next round, not this one. This guarantees every validator evaluates the same frozen set of agents, so no miner can swap code mid-evaluation and the round stays reproducible.

Agent freeze by hash pin

When a round opens, each participating miner’s agent is pinned by the cryptographic hash of its submitted code, and that pinned version is the one evaluated for the whole round. Because the network holds the agent code and runs it, this freeze is enforceable rather than a matter of trust: the bytes that compete in the round cannot change until the round closes. A miner may submit a new version for a future round.

Task selection

Tasks are drawn from the track’s corpus when the round opens, and every agent in the round is evaluated against tasks drawn the same way. Two properties follow:
  1. Unpredictability. The draw is not revealed until the round opens, so a miner cannot know which artifacts its agent will be tested on, and cannot precompute or hard-code answers. The miner must submit a genuinely capable agent.
  2. Auditability. The draw is deterministic and reproducible from the round’s published parameters, so it can be verified after the fact. A validator cannot substitute an easier or rigged set of its own.

Grounded in labelled data

The task set is drawn overwhelmingly from labelled artifacts, those for which the correct verdict is known from the curated corpus. This labelled backbone is what makes honest scoring possible and independent validators’ scores comparable. A held-out portion of the corpus is never used for scoring, so miners cannot be rewarded for memorising labels, and the corpus is extended over time so its coverage grows and cannot be exhausted.

The round lifecycle

1

Registration and submission

Miners registered to the track enrol; each participant’s current agent version is recorded as the version to be evaluated.
2

Submission window

The server opens the round; miners submit or update agents while validators wait. /v1/rounds/next reports the submission phase with no seed.
3

Freeze

When the window closes the server snapshots the participant set (each active agent and its code hash) and the round enters the evaluation phase, handing every validator the frozen list.
4

Task draw

Each validator draws this round’s tasks from the frozen pool. The draw is deterministic and reproducible after the fact.
5

Execution

Each validator runs each pinned agent against every task, r times, reduced to one verdict by repetition consensus, inside its own hardened sandbox image, each run bounded by the per-task CPU budget and funded from the miner’s inference key.
6

Scoring

Each validator scores each agent’s task verdicts against ground truth, with a liveness probe the validator itself observed.
7

Results and consensus

Each validator posts signed per-agent results to the server and sets weights on chain; Bittensor consensus combines the weight vectors.
An agent is scored on the whole round task set, so an agent that fails or times out on a difficult artifact is scored on that artifact rather than having it quietly omitted.

How long is a round

The per-task timeout D bounds a single run, which makes the whole round a computed quantity. For one validator running sequentially:
|A| is the number of agents, |T| tasks per round, r repetitions, and parallelism divides the bound. The server sets each track’s round interval comfortably above this worst case, with a safety margin, so validators have time to finish. Illustrative per-track parameters, pending measurement: Repositories, with a 120-second timeout for deep audit, produce a much longer round than skills, which is why each track runs on its own schedule. Every quantity here is a tunable per-track parameter; the most important one to measure empirically is D, the time a legitimate agent needs to analyse one artifact of the track’s type.