ScoreBench API Protocol¶

This is the technical contract for ScoreBench's worker API: scoped access, candidate submissions, official results, usage measurements, and audit evidence. Start with the ScoreBench Guide or launch runbook for the user workflow.

ScoreBench owns venue credentials and official submission transport. Workers still use their model provider, local tools, and permitted public problem reads; the API contract is not a blanket prohibition on external network access.

Roles¶

Owner: creates and monitors experiments in the browser.
Coordinator: prepares and supervises workers on the user's computer.
Worker supervisor: establishes one assigned run, native-session accounting, execution-gate access, and final lifecycle evidence around the coding agent.
scorebenchd (legacy alias harnessd): HTTP service, web UI, and server-side credential and submission boundary.
scorebench (legacy alias harness): scoped worker CLI.
Adapter: translates submissions and results. The hosted composer uses paradigm_puzzles for vliw and prop-amm. Other configured adapters are covered in Integrations and Adapters.
expctl: operator CLI for YAML setup, credentials, usage snapshots, and reports.

Agent Context¶

The supported flow starts with an experiment and its single-use launch prompt. The launcher provisions an isolated assignment per worker; the supervisor redeems it and starts the assigned run before releasing the model. Workers inspect scorebench context, exercise, run current, and run progress. They do not log in to the owner's account or mint replacement run tokens.

Manual handoffs use the following connection values:

export SCOREBENCH_URL=https://scorebench.dev/
export SCOREBENCH_RUN_TOKEN=hrun_...
export SCOREBENCH_TIMING_OBSERVER_TOKEN=hobs_...

The CLI also supports legacy HARNESS_URL, HARNESS_RUN_TOKEN, and HARNESS_ARM_TOKEN values and discovery through .harness/context.json. scorebench context does not print the token. Compatibility paths, package names, and headers such as X-Harness-Trace-Id retain their literal spelling.

The dedicated hobs_... timing-observer token can append allowlisted evidence for one run; it cannot submit, read venue credentials, or attest verified blockage. The ordinary run token cannot write timing evidence. See Active-Time Accounting.

Venue credentials stay on the ScoreBench server. Model-provider authentication is separate: isolated workers receive the minimum credential copy or Claude access-only relay, as documented in the launch runbook. A worker never receives the owner's ScoreBench browser or CLI session.

Agent Commands¶

The following includes manual setup commands. Generated Docker workers already have an assigned, started run; use run current instead of creating one. Adapter-specific examples apply only to their configured scope.

scorebench context
scorebench exercise
scorebench exercise --exercise sum_of_prime_numbers
scorebench run start --id run001 \
  --skills scorebench \
  --model gpt-5-codex \
  --coding-harness Codex \
  --effort high \
  --autonomy autonomous
scorebench run current
scorebench run progress
scorebench run gate --wait
scorebench submit path/to/solution --notes "short result note" --total-tokens 126000 --tokens-total-source agent_claim
scorebench submit path/to/solution --exercise sum_of_prime_numbers --notes "short result note" --total-tokens 126000 --tokens-total-source agent_claim
scorebench invalidate --reason "exploit: caches venue inputs instead of solving generally"
scorebench invalidate <candidate_id> --reason "bug: undefined behavior produced an invalid score"
scorebench best
scorebench history

--exercise also accepts the alias --problem. It is request scoped: it tells the connector which problem to read or submit for that one call, without mutating the run configuration or any other arm. If omitted, the middleware uses the run's configured default exercise.

Manual run access can be created under Account > Exercise API Keys. The run name can either be pre-bound in the web UI or left blank for the agent to choose. A blank run name creates an exercise-scoped token whose server-side scope includes:

{
  "user_name": "admin",
  "connector": "highload",
  "exercise": "sum_of_prime_numbers",
  "credential_profile": "skill-research",
  "needs_run_name": true
}

For an unbound manual token with needs_run_name: true, start or continue a named run before the first submission:

scorebench run start --id run001 \
  --strategy "progress logging skill with perf access" \
  --hypothesis "perf-guided changes should reduce score faster" \
  --skills scorebench,problem-agnostic-optimization \
  --model gpt-5-codex \
  --coding-harness Codex \
  --effort high \
  --autonomy autonomous

--model and --effort are required: scorebench run start fails without them (supplied on the command line, or inherited from a pre-bound exercise API key created with --model/--effort) so every run is attributable in reports. --coding-harness records the actual agent interface separately from the model: normally Claude uses Claude Code, GPT/OpenAI uses Codex, DeepSeek uses Deep Code, Grok uses Grok Build, Kimi (including model ID k3) uses Kimi Code, and GLM uses ZCode. Known families are inferred for legacy clients; crossed/custom setups must set it explicitly. A Claude model inside Codex records Codex. --strategy, --hypothesis, --label, --skills, and --autonomy are optional; --skills and --autonomy are strongly encouraged and produce non-fatal warnings when missing.

Use --prompt or --prompt-file to persist the original run instructions for the chart view. Pre-bound exercise API keys may carry the same original_prompt metadata, and scorebench admin launch records its goal and additional prompt automatically. Prompt text is stored once on the logical run, not duplicated on every candidate.

On GPU-backed connectors, scorebench run start --gpu <gpu> (for example --gpu H100 on tensara) declares the GPU the whole run targets. Submissions in that run inherit it as their connector gpu option, and a conflicting scorebench submit --gpu is rejected so a run never mixes GPUs. Without a run-level GPU the connector default applies (T4 for tensara connectors), and the GPU actually used is still recorded per candidate for report filtering.

If a previous run already exists for the same user/profile/exercise, ScoreBench rejects the start request and names the previous run. Continue it with that run id, or retry with --confirm-new-run only when the user explicitly wants a new independent run.

If the web UI creates the token with a run name, the token is already scoped to that run and needs_run_name is false. The agent should use scorebench run current to inspect it rather than switching it to a different run.

scorebench submit requires a cumulative token snapshot in the same call. total_tokens is not optional, and it must be run-relative unless a supervised runner records a run-start baseline. Before optimization work starts or resumes, the agent must send a run ping so reports can derive elapsed time from server timestamps and remove dead time between sessions:

scorebench run ping --event start --note "starting work"
scorebench run ping --event resume --note "resuming work"

The ping records a middleware timestamp only. It is not an agent claim about thinking time.

At the end of a run, or before handing control back to the user, record the final run-level usage measurement. Reports prefer this final run usage over per-submission inference:

scorebench run usage \
  --total-tokens 10643192 \
  --input-tokens 301811787 \
  --cached-input-tokens 292001024 \
  --output-tokens 832429 \
  --reasoning-output-tokens 329446 \
  --usage-source codex_usage \
  --usage-confidence exact \
  --tokens-total-source final_goal_usage

If only total_tokens is available, send that for providers whose aggregate is a working-token count. Do not invent missing input/output breakdown fields. Grok is an explicit exception: its aggregate includes cached reads, so a Grok run must use the ScoreBench skill's token_usage.py --grok-jsonl parser and send fresh input, output, and cache-read counters.

scorebench submit sol.cu \
  --exercise sum_of_prime_numbers \
  --label baseline \
  --notes "first viable kernel" \
  --idempotency-key baseline-001 \
  --input-tokens 120000 \
  --output-tokens 6000 \
  --total-tokens 126000 \
  --tokens-delta 126000 \
  --usage-source agent_reported \
  --usage-confidence estimated \
  --tokens-total-source agent_claim \
  --meta session_id=abc123 \
  --meta hypothesis=baseline

total_tokens is mandatory. Token deltas and cost are accepted only as measurements, not as trusted facts. Agent-reported timing fields are rejected; use scorebench run ping instead.

The installed ScoreBench skill adds --accounting-version to snapshots it generates. Do not set this flag by hand. For accounting version 3 and newer, the server recomputes the working total from available components, requires trusted provenance, and rejects any cumulative token component, total, or cost that falls below an earlier checkpoint in the same run. Transcript-backed Codex, Claude Code, and Grok Build snapshots must include their complete component breakdown. Older clients retain warning-only regression handling so an upgrade does not rewrite or block an in-progress legacy run.

Working token total¶

Reports and charts use a working token total — input_tokens + output_tokens + cache_creation_tokens — as the headline token count. Cache reads are re-reads of already-counted context and are deliberately excluded; counting them inflates the total by roughly conversation-length times turn-count. Send the categories distinctly:

scorebench submit sol.cu \
  --input-tokens 57578 \
  --output-tokens 437510 \
  --cache-creation-tokens 1191658 \
  --cache-read-tokens 998452654 \
  --total-tokens 1686746 \
  --tokens-total-source agent_claim

When the input/output breakdown is present ScoreBench computes the working total itself (recording tokens_total_source: harness_working_tokens), so cache reads in --total-tokens never inflate what the chart shows; the cache-read count is retained for the per-candidate breakdown and API-equivalent cost. If you can supply only --total-tokens, ScoreBench uses it as-is — it cannot un-inflate a total it did not break down or recover cached reads for pricing. For that reason, aggregate-only usage is rejected when the active run model is Grok.

The canonical fields are provider-agnostic: with --cache-read-tokens, --input-tokens must be the uncached, non-cache-write input. Anthropic already reports disjoint categories. OpenAI/Codex instead reports an inclusive input_tokens count. Pass that raw total with --cached-input-tokens and any cache-write subset with --cache-creation-tokens; the CLI converts them to the canonical disjoint fields. Do not pass both cache-read flags. In every case the working total excludes cache reads, while API-equivalent cost includes them at the model's cached-input rate. Grok also reports inclusive input, but its native field names and per-turn semantics differ; use the skill's --grok-jsonl parser rather than translating totalTokens manually.

Submission resource limits¶

Every run is protected by server-side candidate, artifact, token, and rate budgets. GET /context returns the effective submission_limits so workers and supervisors can inspect the contract before submitting. Defaults are:

submission_limits:
  max_candidates_per_run: 1000
  max_bundle_bytes_per_run: 2147483648
  max_total_tokens_per_run: 100000000
  rate_window_seconds: 600
  max_submissions_per_window: 20
  same_content_window_seconds: 600
  max_same_content_submissions_per_window: 1
  token_warning_floor: 5000000
  token_warning_per_elapsed_hour: 10000000

An experiment can override these positive values at the top level of experiment.yaml. Candidate, artifact, and token exhaustion returns HTTP 409; the rolling run and same-content rate limits return HTTP 429 with Retry-After and a structured rate_limit object. Both windows are derived from persisted candidate timestamps, so daemon restarts do not reset them. Checks run under the same service lock as candidate allocation. An exact retry with a previously accepted idempotency key is resolved before the budgets, so a client can safely recover a lost response without consuming another candidate or venue submission. The same-content window applies only when a new idempotency key would create a new candidate for an unchanged bundle. When a connector records an explicit cooldown with nextSubmissionAt, that venue timestamp releases the bundle instead of adding a second middleware wait.

The token ceiling applies to the normalized working total. When input/output categories are supplied, cache reads remain recorded but do not count against the token budget. Aggregate-only totals cannot be corrected this way. Totals that are below the hard limit but unusually high for the run's elapsed time are accepted with token_usage: suspect and a trust warning instead of silently distorting comparisons.

HTTP Endpoints¶

Worker bootstrap endpoints do not use a run bearer token because they issue that token. They require a short-lived, single-use launch or pairing credential; they are not anonymous access to run data. Generated launchers own this flow.

The legacy interactive pairing endpoint remains available:

POST /api/worker-launch/redeem exchanges the single-use launch token for the protected worker manifest. This is called by the generated installer.
POST /api/pair with JSON { "code": "PZL-..." } redeems a short-lived, single-use code and returns one pre-bound scoped run token. The server stores only the code hash, rate-limits attempts, and returns the token only in the successful response. The scorebench pair CLI command saves it in the local user config with mode 0600, scoped to the selected workspace.

Creating, listing, monitoring, and revoking pairings requires the owning browser session. First-class experiment monitoring uses GET /puzzles/scorebench/experiment/status?id=<experiment-id>; adding trajectory=1 includes candidate trajectories. The owner can stop every unfinished pairing with POST /puzzles/scorebench/experiment/stop and a valid CSRF token. POST /puzzles/scorebench/experiment/exclusion excludes or restores one run from analysis and requires an audit reason for either action. Once paired, the worker uses the standard bearer-token protocol below; pairing does not weaken or broaden its scope.

All arm endpoints require:

Authorization: Bearer <SCOREBENCH_RUN_TOKEN>

Legacy arm tokens use the same bearer header.

Endpoints:

GET /exercise: exercise statement and connector metadata visible to this arm. For HighLoad this includes languages, defaultLanguage, and defaultFilenames.
GET /exercise?exercise=<problem_id>: request-scoped exercise statement override for connectors that support multiple problems.
GET /context: token-bound user, credential profile, connector, exercise, run context, and effective submission limits.
POST /run/start: legacy arm-token endpoint to start or reactivate the current agent run.
POST /run/ping: record a server timestamp for run start/resume/heartbeat; required before submit. A finish or failure ping may carry an idempotency_key (up to 180 letters, numbers, _, ., : or -). Repeating that ping with the same key returns the recorded heartbeat with "replayed": true instead of adding a later terminal event.
GET /run/current: read the active agent run for this arm.
GET /run/progress: canonical latest-submission active time, elapsed time, token total, sources, measurement timestamps, and current submission allowance for the exact run-token scope. This read does not create an activity heartbeat. During timing-v2 shadow evaluation it also contains progress.timing_v2; the existing top-level timing remains authoritative.
GET /run/execution-gate: read whether this run's recipe stage may start. Parallel runs are immediately ready. Sequential workers wait until earlier recipes satisfy the recorded gate policy. Failed predecessors can keep the gate closed; see the execution-gate runbook. This observational read does not create an activity heartbeat or start the run budget.
POST /run/timing-evidence: append a bounded batch of passive local timing spans. This endpoint accepts only the dedicated timing-observer bearer token, has a strict metadata allowlist, and never counts the request as run activity.
POST /run/trace: upload one sanitized gzip NDJSON trace artifact for the exact run-token scope.
GET /run/traces: list trace artifacts for the exact run-token scope.
GET /run/trace?trace_id=<id>: download one scoped gzip trace artifact. Trace uploads and reads do not create activity heartbeats.
GET /best: best scored candidate visible to this arm.
GET /history: candidates for this arm only.
GET /team, GET /team/log?after=<seq>, POST /team/post, GET /team/candidates, GET /team/candidate?candidate_id=<id> (a candidate or snap_... snapshot id), and, when the swarm's level enables snapshots, POST /team/share and GET /team/snapshots: swarm trials only. The findings log and teammates' submitted candidates for the caller's own trial; every call is audited. Other runs, including single-agent and isolated best_of_k workers, receive 401. For swarm workers, GET /run/progress also carries a team object: unread_messages, latest_seq, read_after, new_teammate_candidates, teammate_candidate_count, teammate_best_score, the owner's instructions when set, and a hint command when messages are unread. See agent teams.
GET /solutions: connector-visible solution list for the scoped exercise.
GET /leaderboard: connector-visible leaderboard for the scoped exercise.
GET /public-solution?solution_id=<id>: connector-visible metadata for a public or website-visible solution id. This is distinct from GET /solution.
GET /challenge-page?section=<section>: connector-visible challenge section, such as HighLoad leaderboard or generators.
GET /solve-form: redacted connector solve-form defaults, such as HighLoad compiler, compiler flags, availableLanguages, and defaultFilenames.
GET /solution?solution_id=<id>: solution details only for connector ids submitted by the current run.
POST /submit: immutable candidate submission.
POST /refresh: reconcile pending candidates visible to this run; it does not create a replacement candidate or start another submission.
POST /invalidate: mark a visible candidate invalid without deleting its bundle, score payload, logs, or connector evidence.
POST /reinstate: restore a visible invalidated candidate by appending a linked audit event.

GET /run/progress¶

GET /run/progress requires a run-scoped bearer token. Legacy arm tokens are rejected because they do not identify one authoritative run. Existing run tokens with own-history:read remain compatible; newly created tokens also carry run-progress:read.

The response is designed for supervisors and agents that need accounting progress without parsing chart HTML or treating /best as the latest measurement:

{
  "scope": {
    "kind": "run_token",
    "run_id": "run001"
  },
  "progress": {
    "schema_version": 1,
    "run_id": "run001",
    "active_seconds": 14764.6,
    "elapsed_seconds": 15120.7,
    "tokens_total": 4780000,
    "active_seconds_source": "server_timestamps_with_run_pings",
    "elapsed_seconds_source": "server_timestamps",
    "tokens_total_source": "claude_code_jsonl",
    "measured_at": "2026-07-29T10:20:30Z",
    "tokens_measured_at": "2026-07-29T10:21:00Z",
    "latest_candidate_measured_at": "2026-07-29T10:15:00Z",
    "idle_gap_seconds": 900,
    "idle_gap_removed_seconds": 0,
    "latest_candidate_id": "candidate-id",
    "latest_candidate_status": "accepted",
    "candidate_count": 33
  },
  "submission": {
    "can_submit": true,
    "blocked_by": null,
    "retry_after_seconds": 0,
    "next_submission_at": null,
    "general": {
      "window_seconds": 600.0,
      "limit": 20,
      "used": 2,
      "remaining": 18,
      "can_submit": true
    },
    "same_content": {
      "window_seconds": 600.0,
      "limit": 1,
      "checked": false
    }
  }
}

active_seconds and elapsed_seconds are measured through the latest trusted candidate or run ping, identified by measured_at; they do not advance merely because a watcher polls this endpoint. latest_candidate_measured_at remains separate so callers can see when timing extends beyond the last submission. Active time uses the same 15-minute unsupported-gap cap and run-ping heuristic as reports. A later run usage event may provide a newer token measurement, so tokens_measured_at is also separate.

This endpoint is intentionally different from /best. /best describes the best-scoring candidate and therefore exposes time-to-best, which can be older than the run's latest trusted progress. A run with start/activity pings can accumulate measured time before its first candidate. With neither candidates nor trusted pings, active and elapsed time are zero, measured_at is null, and their source is no_submitted_candidates.

Run trace artifacts¶

Run traces are post-run observability artifacts. Discovery, normalization, redaction, size limiting, and compression belong to the installed ScoreBench skill; the server only validates and stores the completed gzip. Trace operations require a bound, started run token. Legacy arm tokens are rejected.

Upload the gzip body directly:

POST /run/trace HTTP/1.1
Authorization: Bearer hrun_...
Content-Type: application/gzip
Content-Length: ...
X-Scorebench-Trace-Id: trace_<first-32-hex-of-sha256>
X-Scorebench-Trace-Sha256: <full-sha256>

The first uncompressed NDJSON record is a bounded manifest:

{
  "format": "scorebench-run-trace",
  "schema_version": 1,
  "provider": "codex",
  "session_id": "session-id",
  "event_count": 128,
  "normalization": {
    "private_reasoning_included": false
  }
}

The trace ID is content-derived, so retrying the same upload is idempotent. Compressed content is limited to 50 MiB and uncompressed content to 64 MiB. Each run can retain at most three traces and 128 MiB of compressed trace data in aggregate. Replaying an already stored trace is still allowed after those limits are reached. The server validates the complete gzip stream, its hash, and the manifest. It rejects artifacts claiming to contain private reasoning. Trace files live outside SQLite, candidate bundles, and generated report payloads.

Master-token endpoint:

POST /usage_snapshot: external launcher usage snapshot. Requires X-Harness-Admin-Token.

POST /submit Schema¶

Required fields:

{
  "run_id": "run001",
  "bundle_b64": "<base64 tar.gz>",
  "bundle_sha256": "<sha256 hex>",
  "manifest": {
    "root": ".",
    "files": []
  }
}

run_id is mandatory for raw HTTP clients. The CLI fills it from GET /run/current. With SCOREBENCH_RUN_TOKEN, the value must match the server-side run bound to the token.

Optional fields:

{
  "exercise": "sum_of_prime_numbers",
  "notes": "short human note",
  "label": "baseline",
  "connector_options": {
    "compiler": "gcc_cpp",
    "compiler_options": "-O3 -march=native",
    "language": "cpp",
    "system": "raptor_cove_p"
  },
  "idempotency_key": "baseline-001",
  "metadata": {
    "session_id": "abc123",
    "hypothesis": "baseline"
  },
  "usage": {
    "input_tokens": 120000,
    "output_tokens": 6000,
    "total_tokens": 126000,
    "tokens_delta": 126000,
    "cost_usd": 1.25,
    "source": "agent_reported",
    "confidence": "estimated",
    "tokens_total_source": "agent_claim"
  }
}

exercise may also be sent as problem, metadata.exercise, or metadata.problem by minimal clients. The middleware copies it into candidate metadata as both exercise and problem_id, includes it in the immutable request hash, and passes it to the connector for that submission only.

connector_options is optional and narrowly scoped to the selected connector for this submission. Supported keys are compiler, compiler_options, language, solution_file, system, gpu, leaderboard, submission_mode, profile_brev, and benchmark_index; top-level aliases with the same names are accepted for simple clients. Unknown connector option keys are rejected. For CPU.mode, valid compiler ids are currently rustc, clang_cpp, gcc_cpp, and clang_asm. For HighLoad, compiler, compiler_options, and language map directly to the website solve form fields compiler, compilerArgs, and lang. The language is the venue language id for the current candidate, such as CPP, RUST, GO, CSHARP, or ZIG; callers should inspect scorebench solve-form instead of assuming C++. Changing only compiler flags is a legitimate new candidate and should use a new idempotency key.

HighLoad's built-in language ids are CPP, GO, RUST, CSHARP, and ZIG. Connector config may add additional safe ids through languages, langs, or language_ids. scorebench solve-form --language RUST requests the upstream /solve/RUST form for the scoped exercise and returns the selected language, available language ids, default filenames, compiler id, and compiler arguments.

The middleware also injects normalized run metadata into every accepted candidate:

{
  "agent_run_id": "run001",
  "agent_run_label": "montgomery p31 baseline",
  "agent_run_strategy": "deterministic Miller-Rabin with Montgomery multiplication",
  "agent_run_hypothesis": "small-prime filtering plus Montgomery arithmetic lowers score without correctness loss",
  "agent_run_candidate_seq": 1,
  "agent_run_candidate_id": "run001-c001",
  "agent_run_candidate_timestamp": "2026-06-22T12:00:00Z"
}

candidate_seq and the immutable candidate storage id remain arm-global so database rows and artifacts never change identity. agent_run_candidate_seq restarts at 1 for each run, and reports use it for human-facing candidate labels such as c001-baseline. Older candidates without run metadata fall back to the arm-global sequence.

If a request tries to provide conflicting run metadata, the middleware rejects it instead of silently rewriting it.

Top-level usage keys are accepted as aliases for usage.* for simple clients. usage.total_tokens or top-level total_tokens is required; submissions without it are rejected before candidate allocation and connector submission.

Trust Model¶

ScoreBench records three categories of data:

Authoritative middleware data: timestamp, arm, candidate sequence, bundle hash, artifact path, connector stdout/stderr, raw score payload, and status.
Reported data: token counts, cost, and arbitrary metadata.
Derived middleware data: elapsed time from server timestamps, run pings, candidate submission timestamps, and API-equivalent cost estimates from the versioned public list-price table.

Reported data is never silently promoted to authoritative. The submit response and reports include trust fields:

trusted: exact or parsed data from a launcher/middleware/API meter.
agent_reported: supplied by the solving agent.
suspect: accepted for auditability, but outside the elapsed-time sanity envelope and visibly warned in reports.
missing: not supplied. New submissions are rejected when total token usage is missing, so this value only appears in older runs or non-token fields.

Sanity checks either reject unsafe requests before candidate allocation or add warnings without discarding the candidate:

missing or mismatched run id is rejected before candidate allocation.
missing idempotency key.
agent-reported timing fields are rejected.
total_tokens lower than the previous recorded token total.
tokens_delta inconsistent with the previous token total.
total_tokens unusually high for elapsed run time (accepted as suspect).
same content hash submitted under a new candidate.
candidate label reused with different content.

If an idempotency key is reused with the same content hash and the same exercise scope, the middleware returns the original candidate instead of resubmitting. If the key is reused with different content or a different exercise, the request is rejected.

Negative numeric usage values are rejected.

POST /invalidate Schema¶

Agents can invalidate only candidates visible to their current token scope. This is for submissions that were accepted by the connector but later found to be exploity, invalid, or based on a false assumption. Invalidation is not deletion: the middleware keeps the immutable bundle, score rows, raw connector payload, usage metadata, and trace evidence, and appends an audit row.

The candidate status becomes invalidated. Invalidated candidates remain in history and report exports for auditability, but are excluded from best, best-so-far curves, promotion decisions, and chart winner calculations.

Request:

{
  "candidate_id": "profile_local_tensara_josu_cand_0533",
  "reason": "exploit: memoizes exact matrix inputs instead of general multiplication",
  "metadata": {
    "class": "exploit"
  }
}

candidate_id is optional. If omitted, ScoreBench invalidates the latest candidate visible to the token's run scope. reason is required and should state why the candidate should no longer count.

CLI:

scorebench invalidate \
  profile_local_tensara_josu_cand_0533 \
  --reason "exploit: memoizes exact matrix inputs instead of general multiplication" \
  --meta class=exploit

Response:

{
  "candidate": {"id": "profile_local_tensara_josu_cand_0533", "status": "invalidated"},
  "invalidation": {
    "invalidated_at": "2026-07-03T12:00:00Z",
    "invalidated_by": "admin",
    "reason": "exploit: memoizes exact matrix inputs instead of general multiplication"
  },
  "best": {}
}

Invalidated candidates remain in history, TSV exports, logs, and charts with status INVALIDATED, but they are excluded from best, best-so-far curves, promotion decisions, and winner calculations.

GET /history includes an audit object on every candidate row. For an invalidated candidate it contains the reason, actor, timestamp, metadata, and any later reinstatement, so a worker can distinguish a venue failure from an explicit correctness decision.

Concrete operational example:

scorebench invalidate profile_local_tensara_josu_cand_0533 \
  --reason "device-side exact input comparison and cached output reuse is not a valid general square matrix multiplication submission"

For the regenerated local_tensara / square-matmul chart, that candidate is shown as INVALIDATED, and the run falls back to profile_local_tensara_josu_cand_0529 at 8078.938681941922 us.

POST /reinstate Schema¶

Reinstatement reverses the candidate's current exclusion without removing or editing the invalidation record. The candidate must be visible to the current token and currently invalidated.

{
  "candidate_id": "profile_local_tensara_josu_cand_0533",
  "reason": "contract correction: the documented input domain permits this specialization",
  "metadata": {
    "class": "contract_correction"
  }
}

ScoreBench restores the status captured in the invalidation metadata. For a legacy invalidation that predates status capture, it conservatively infers scored, failed, or submitted from preserved score evidence. An operator can pass restore_status explicitly when that inference is insufficient.

scorebench reinstate profile_local_tensara_josu_cand_0533 \
  --reason "contract correction: the documented input domain permits this specialization" \
  --meta class=contract_correction

The response contains the restored candidate, the append-only reinstatement event, and the newly computed scoped best.

Deterministic Outputs¶

expctl report <run_dir> writes:

reports/report.json: complete structured data.
reports/report.csv: candidate table for spreadsheets.
reports/progress.tsv: canonical all-arm progress log:

timestamp   experiment_id   run_id  agent_id    variant exercise    problem_id  candidate   score   score_unit  decision    status  tokens_total    tokens_delta    elapsed_seconds server_elapsed_seconds  exercise_submission_id  code_sha256 label   warnings

reports/progress-details.tsv: all-arm debug log with token, elapsed-time source, and trust fields.
reports/<arm>-perf.tsv: per-arm deterministic log with the reference columns:

timestamp   candidate   score   decision    tokens_total    tokens_delta    wall_seconds    label

reports/report.html: static run-scoped summary page.
reports/strategy-compare.html: primary exercise-level comparison charts. It aggregates sibling runs with the same connector and exercise, plus optional external baselines from runs/external_players/<connector>/<exercise>.json. It includes exercise picking, strategy toggles, best-score trajectories, failed/rejected markers, wall idle-gap compression, token expenditure, API-equivalent cost, and a summary table. Cost derivation and uncertainty are documented in API Cost Accounting.
Web route /ui/dashboards: Charts with owner-scoped filtering.
Web route /ui/reports/: compatibility redirect to the deployment's default report.
Web route /ui/runs: searchable personal Runs index with isolated chart links and two independent controls. Hide locally removes a run from its owner's normal comparisons without deleting evidence; Delete permanently removes the run, submissions, bundles, and ledger rows after confirmation. Chart requests always require login and remain owner-scoped.
Web route /ui/candidates/code?candidate=<experiment>/<candidate>: bounded, syntax-highlighted source from the immutable submitted bundle. It is visible only to the submitting owner unless that owner explicitly selects Make public. Source publication is candidate-specific and reversible; it does not expose the owning run or charts.
Web route /ui/keys: create and manage scoped exercise API keys.
Web route /ui/docs/: generated MkDocs documentation with search.
reports/strategy-compare-<connector>-<exercise>.html: one comparison page per discovered exercise under the same runs root.
reports/comparison.svg and reports/<arm>_progress.svg: lightweight charts.

The TSV files are generated from SQLite and immutable artifacts, never from agent prose logs.

For new rows, the canonical progress.tsv run_id column is the web/token-bound run id. Legacy rows without run metadata fall back to the arm id. Exercise-level strategy comparison charts split ScoreBench data by agent_run_id when present, so multiple independent runs from the same arm can be compared objectively. Run and candidate payloads also expose coding_harness; charts can filter it and Export Studio can group or split comparisons by it independently of model.

GitHub PR Venues¶

This and the CPU.mode/GPU Mode sections describe retained adapters for configured deployments, not hosted experiment choices.

PR-based exercises are modeled as a transport, not as special logic in every connector. A github_pr submission still uses the same POST /submit schema and the same report fields. The normalized remote id is stored in exercise_submission_id as:

github:owner/repo#123@<head_sha>

Default mode: dry_run is deterministic and side-effect free. It computes the server-generated branch, PR title/body marker, PR URL shape, head SHA surrogate, and state-machine events without touching GitHub. This is useful for tests and for checking PR policy without creating public noise.

mode: gh_cli is opt-in and requires server-side config:

connector: github_pr
connector_config:
  mode: gh_cli
  base_repo: owner/exercise-repo
  base_ref: main
  repo_path: /srv/harness/checkouts/exercise-repo
  target_path: submissions/agent-a
  head_owner: github-user-or-fork-owner
  draft: true

The daemon can load a per-arm connectors.github_pr.env_file containing GH_TOKEN=.... Agents never receive that token. The generated branch name is server-controlled:

arena/<connector>/<problem>/<experiment>/<arm>/<candidate>-<content_sha12>

For public exercise repos, use a conservative policy such as pr_only_final or pr_only_on_promote. If both A/B arms open public PRs during the run, record the experiment as public-contaminated because agents may observe each other's work.

For ECDSA.fail specifically, the official scoring flow should remain the ECDSA.fail CLI/API-key path unless the organizers require PRs for official scoring. The GitHub PR transport is best treated as an optional publish step for that connector.

CPU.mode Venue¶

The cpumode connector uses https://cpu.mattstuchlik.com/api:

GET /api/challenges/{exercise} for exercise metadata.
POST /api/challenges/{exercise}/submissions for source submissions.
GET /api/jobs/{job_id} for benchmark results.

The daemon stores CPU.mode auth in connector env files. The web UI accepts a regular cpumode_... API token or an authenticated cpu_mode_session=... cookie. When a parent credential is provided, it can use it once to mint a scoped cpumode_... API token and store that token. Existing and scripted credentials can still use any supported shape:

CPUMODE_API_TOKEN=...
CPU_MODE_TOKEN=...
CPU_MODE_TOKEN_FILE=/secure/cpumode-token.txt
CPUMODE_SESSION_COOKIE=cpu_mode_session=...
CPUMODE_AUTH=Bearer cpumode_...

CPU.mode agent tokens are stored through CPUMODE_API_TOKEN; the web UI keeps the setup path simple by collecting the authenticated cookie and doing the mint server-side.

The normalized score is the selected job's result_time_ns with score type cpu_time_ns. Set connector_config.system to choose the target system, for example raptor_cove_p; all raw job payloads are preserved in the candidate artifact.

GPU Mode Venue¶

The gpumode connector submits through the official Popcorn CLI. The daemon stores the Popcorn identity, creates an isolated connector home per credential profile/run, writes that home's .popcorn.yaml, and runs:

popcorn submit --no-tui --leaderboard <exercise> --gpu <gpu> --mode <mode> --output <artifact> <submission.py>

The stored credential is the cli_id value from ~/.popcorn.yaml after running popcorn register discord or popcorn register github:

POPCORN_CLI_ID=...

Submissions must contain a single Python file, usually submission.py. If a bundle contains multiple Python files, pass --solution-file or connector_options.solution_file so the connector knows which file to send. Popcorn directives such as #!POPCORN leaderboard ... and #!POPCORN gpu ... can still live in the source, but ScoreBench scope remains the source of truth for the exercise/run/credential profile.

Useful connector options:

{
  "gpu": "B200",
  "submission_mode": "test",
  "leaderboard": "qr_v2",
  "profile_brev": "true",
  "benchmark_index": "0",
  "solution_file": "submission.py"
}

submit and refresh are Popcorn proxy calls. The normalized score is the selected Popcorn run score in seconds with score type gpumode_score_seconds, but the response also preserves the visible Popcorn CLI payload and normalized public case results:

connector_response.raw.popcorn.command
connector_response.raw.popcorn.stdout
connector_response.raw.popcorn.stderr
connector_response.raw.popcorn.output
connector_response.raw.popcorn.text
connector_response.raw.popcorn.parsed
connector_response.case_results
connector_response.case_summary

The connector performs an authenticated read of /user/submissions/<submission_id> after submit, refresh, and solution fetches because Popcorn's submissions show text omits runs[].result. Public benchmark/test cases are normalized with typed parameters and nanosecond timing fields. Secret runs retain their aggregate run/score metadata, but their detailed result bodies are stripped before storage and response serialization. If structured details are temporarily unavailable, submit output in Popcorn's rendered benchmark format is parsed as a bounded fallback.

refresh uses popcorn submissions show <id> --no-code. To fetch the same view directly through ScoreBench, run:

scorebench solution <submission_id> --no-code

The full source-including Popcorn view remains available through:

scorebench solution <submission_id>

Agents must use these ScoreBench calls instead of calling popcorn directly. The run token can only fetch submissions recorded under its scoped run. Running scorebench refresh <candidate_id> after this connector update can enrich an older candidate with case details if Popcorn still retains the submission.

Paradigm Puzzles Venue¶

paradigm_puzzles stores one server-side PARADIGM_PUZZLES_API_KEY=pp_... credential and exposes the active API-documented exercises through run-token scope. Workers submit ordinary bundles through POST /submit; they never receive the bearer key.

The hosted product uses ScoreBench accounts and accepts the owner's Paradigm Puzzles key through its credential form. The optional native integration uses /api/integrations/paradigm/v1/* from Paradigm's backend, with trusted identity and server-to-server credentials. It is disabled unless configured and is not the default browser authentication flow. See Optional Paradigm-native Integration.

The canonical ScoreBench product at https://scorebench.dev/puzzles/scorebench, with its review deployment at https://staging.scorebench.dev/puzzles/scorebench, composes an Anthropic Take-Home or Prop AMM experiment from a hypothesis, explicit run cart, shared per-run budget, constraints, and recipe-specific instructions. The exercise remains an explicit protocol field so more exercises can be added without changing the experiment model. ScoreBench records the cart as an immutable launch batch and creates one independently scoped worker credential for every run. The page returns one coordinator prompt with a single-use launch token; the launcher retrieves the protected worker assignments directly from ScoreBench. A batch may start all recipes in parallel or gate model startup by recipe order while retaining concurrent replicates inside each recipe.

The canonical, versioned experiment specification is hashed with SHA-256. It records outcome direction, run recipes, repetitions, launch order, manual retry policy, and reason-required exclusion policy. Experiment id, specification hash, recipe, replicate coordinates, and launch order are persisted in worker metadata and the generated assignment. Account and worker credentials are excluded from the specification. A budget can target active time, working tokens, or estimated API cost; the normalized target is copied into run scope metadata and returned as progress.budget by GET /run/progress.

Agents and coding harnesses continue to run on the user's computer. Named profiles and HTTPS reference URLs are private to their owning user; ScoreBench does not fetch or execute those URLs or coding-harness parameters. Supported Docker launches build an image from the latest releases and use a single-use launch token; they do not require a reusable host CLI login. Manual launches use the separate owner-profile flow in the launch runbook. The owner-only monitor can be closed and revisited while workers continue.

The connector selects the upstream codec from the scoped exercise: JSON source, structured packing JSON, or multipart file upload. It preserves SSE progress and result events, normalizes each challenge's metric, and records an authoritative lower or higher direction with the score. Report grouping uses the persisted direction, so maximizing Paradigm challenges are not interpreted using the host experiment's lower-is-better default.

AMM, Prop AMM, QEC, and Packing are validated before submission. SSE exercises check authenticated cooldown state before submitting. Lean Semantics returns a remote submission id in submitted state and supports POST /refresh; the other documented endpoints normally return a terminal score in the initial response. Paradigm leaderboard, submission-list, and submission-detail reads are proxied through the existing connector read endpoints.

See Paradigm Puzzles for file names, payloads, metrics, and directions.

Credential Storage¶

The web UI stores connector credentials as named profiles under the shared runs root:

runs/credentials/<connector>/<credential-name>.env
runs/credentials/<connector>/<credential-name>.json

These profiles are selectable when creating exercise API keys. A run token binds one user, one credential profile, one connector, one exercise, and optionally one run; agents cannot use that token to switch to other profiles or connectors. Exercise API keys can be archived, revoked, or deleted from the Account page. Only active run tokens authenticate.

The same credential profile may back several independent run tokens in parallel. Those tokens intentionally do not share a read surface. For a scoped run token, best, history, refresh, and solution are filtered to rows with the same user, credential profile, connector, exercise, and run id. In particular, solution only fetches a connector source when the requested remote solution id was recorded by a submission from that same run. This lets the owner compare runs in the web charts while preventing agents from inspecting sibling runs through the ScoreBench API, even when the connector credential itself could see them.

New web UI credential env files are encrypted at rest:

# harness-secret-env:v1
HARNESS_SECRET_ENV=<fernet payload>

The local decrypt key is stored at runs/credentials/.harness_secret_key with 0600 permissions. This is local at-rest protection; the daemon can decrypt the secrets while running. Production deployments can set SCOREBENCH_SECRET_ENV_KEY_FILE (legacy alias: HARNESS_SECRET_ENV_KEY_FILE) to an absolute path outside the run-data tree so database or credential-directory snapshots do not contain both ciphertext and key. Copy the existing key to that path before setting the variable; the daemon fails closed if an existing encrypted credential cannot be decrypted. Connector loaders still support legacy plaintext env files for scripted and older runs.

The older YAML-oriented CLI flow remains available:

expctl credentials setup experiment.yaml

The interactive flow supports any number of agents and any configured connector. It writes YAML-bound credential env files under:

CONFIG_DIR/.harness_credentials/<experiment_id>/users/<user>/<connector>.env

All credential env files are written with 0600 permissions. The web UI never renders secret values back to the browser, and the YAML flow stores only env-file paths in experiment.yaml.