Monobench Reference - Monolex Docs

Published contract: monobench 0.2.7, OpenCLIs build 2026-07-12-042014. This page covers both the native instance harness and the complete public bench-set grammar embedded in that payload. Inventory proof: 35 canonical top-level commands, nine global bench estate/data actions, and 48 canonical bench <id> actions. memo aliases note; contamination aliases integrity. Counts exclude aliases but include help/version as top-level commands.

Commands

version

Print the installed binary version.

monobench version

help

Print the full command reference. Running bare monobench with no arguments prints the same text.

monobench help

root

Print the active benchmark root plus candidate roots — use when installed-vs-source or working-directory inference is confusing.

monobench root

list

List the instance corpus.

monobench list [options]

Option	Description
`--catalog`	Show source/cohort/tool-profile/language columns
`--source S`	Filter by problem source
`--cohort C`	Filter by curated cohort
`--tool-profile T`	Filter by intended target tool profile
`--semantics-supported`	Only languages currently covered by monogram-semantics

tools

List available tool adapters (baseline, monogram, monogram-mcp, codegraph, or a custom one).

monobench tools

show

Print the task given to the agent — the symptom, never the answer.

monobench show <id> [--catalog] [--spoil]

--spoil reveals the gated ground truth. Never paste its output into anything that will also solve the instance.

run

Run ONE arm of ONE instance: clone the repo at the pinned tag, index per the tool adapter, invoke the model, grade.

monobench run <id> <arm> [note...]

prepare

Pre-index a tool once and snapshot the result, so matrix --prepared skips per-run indexing.

monobench prepare <id> [--tools a,b]

matrix

Run a multi-run comparison for one instance — one CLI+model combination per command; repeat the command for the next model.

monobench matrix <id> --tools a,b --cli <cli> --model <model> [options]

Option	Description
`--tools a,b`	Arms to compare
`--cli`	`claude`, `codex`, `agy`, `grok`, or `opencode`
`--model`	Full model label for that CLI
`--via`	`direct` (default) or `niia`
`--effort`	Effort label, mapped per CLI
`--prepared`	Reuse a `prepare`d index
`--isolate`	`worktree` (default) or single-lane `shared`
`--runs N`	Repeats per arm
`--jobs N`	Parallel workers
`--tag T` / `--note TXT`	Human-readable experiment metadata
`--semantics-supported`	Refuse languages outside current monogram-semantics coverage

sweep

Run matrix-style jobs across multiple instances at once.

monobench sweep <ids|--all> [--source S] [--cohort C] [--tool-profile T]

Add --semantics-supported to filter/refuse instances outside the current semantic extractor language set.

absorb

Project external Harbor tasks into the native default+ catalog without copying their tests or solution.

monobench absorb <deepswe|swe-bench-verified|swe-bench-pro> \
  (--task <id> | --all) [--cohort C] [--tool-profile T] [--dry-run]

--overwrite replaces an existing projection; --allow-duplicate disables the default same-bench-task duplicate guard. Preview first.

stop

Gracefully stop an active matrix/sweep after in-flight runs finish.

monobench stop

clean

Delete recorded runs for an instance — all of them, or one arm’s.

monobench clean <id> [arm-prefix]

Deletes result data. Use before a fresh re-bench, not casually.

tui

A keyboard-driven wrapper over the same read-only views: browse instances, arms, and runs. Introduces no new judgment logic.

monobench tui
monobench tui --probe 0,1,2   # non-TTY deterministic path probe

grade

Automatic keyword grade for all runs of an instance, or one run.

monobench grade <id> [run]

judge

Build an answer-key-aware review prompt for a final grader. It does not call a model itself — the solver process stays separate from the reviewer that sees the key.

monobench judge <id> <run> [--model m] [--write]

review

Record the checked final grade.

monobench review <id> <run> --final <GRADE> [--reason TEXT] [--judge-model m]

GRADE is one of FULL, NAME_ONLY, DECOY, MISS, NO_RESULT, INVALID, FORFEIT.

status

Per-instance run state, chronologically, with an optional live view.

monobench status <id> [--live] [--every N] [--detail]

watch

Global status across every instance.

monobench watch [--live] [--every N]

report

Per-CLI/model comparison for one instance: FULL hit-rate, review status, median cost/tokens/time, tool-adoption share.

monobench report <id> [--since DUR]

--since 9h isolates one session — all-time totals conflate old arms.

summary

Cross-instance leaderboard: FULL hit-rate and median wall time per arm.

monobench summary [--since DUR]

gate

Evaluate a native-result signal arm against its baseline. Promotion requires a minimum sample count, pooled margin, one-sided two-proportion significance, multi-instance coverage, and no per-instance regression beyond tolerance. This is a read-only report over recorded results.

monobench gate <signal-arm> [baseline-arm] \
  [--min-n N] [--margin PCT] [--z Z] \
  [--min-insts K] [--strat-tol PCT]

The output is PROMOTE or HOLD with pooled and stratified reasons. It does not create runs or change result weight.

ladder

Run three consecutive read-only gates over the four native layer-isolation rungs: baseline → monogram-own → monogram-nologos → monogram. The view attributes own-index, semantic, and Logos net lift without rewriting grades.

monobench ladder [--cli C] [--model M] [--version V] \
  [--instances id,id] [--min-n N] [--margin PCT] [--z Z] \
  [--min-insts K] [--strat-tol PCT]

column

One arm’s per-instance grade breakdown, plus judged-vs-unreviewed coverage.

monobench column <arm> [--since DUR]

adoption

Per-run tool-call breakdown: mono% share, first-use point, and git attempts (each must read as denied).

monobench adoption <id>

monogram-audit

Diagnose Monogram command/result failure patterns in solver telemetry and surface maker-state recommendations.

monobench monogram-audit <id> [--tag T | --run RUN] [--json]

inspect

Quick peek at one run: grade/review status, artifact sizes and age, event counts, and a next action.

monobench inspect <id> <run>

integrity

Heuristic contamination-risk scan. A high score means “rerun before counting this,” not an automatic verdict.

monobench integrity <id> [run]

monobench contamination ... is an accepted alias.

evidence

Search one run’s transcript for a pattern; without a run, rank every run by whether it concluded on that pattern.

monobench evidence <id> [run] [--pattern P]

trace

Ordered tool-call timeline for one run.

monobench trace <id> <run>

export

Render one run’s full transcript to markdown for later search and comparison.

monobench export <id> <run>

note

Attach human metadata to an existing run (also available as memo).

monobench note <id> <run> [--tag T] [--note TXT]

meter

Summarize tokens, cache breakdown, and cost for a raw model-session file.

monobench meter <session.jsonl>

add

Scaffold a native benchmark instance from instances/_TEMPLATE/.

monobench add <id>

This writes instance files. Keep symptom.md spoiler-free and the answer key in the gated ground_truth.md and grading fields.

Bench-Set Commands

monobench bench <set> ... plugs into DeepSWE and SWE-Bench Verified/Pro, each graded through its own external verifier (see the overview). This is a public CLI surface, so its complete command inventory is documented below.

Estate Setup and Data

Command	What it does	Effect or proof boundary
`bench setup`	Prints the ordered new-machine runbook and completion checks for native and external harnesses.	Read-only guide.
`bench doctor`	Checks Pier, Docker/Colima, architecture support, auth, tool host, and served-versus-local versions.	Required preflight; it does not create benchmark truth.
`bench health`	Reports disk, containers, VM, and active benchmark capacity.	Read-only resource gate.
`bench stage-tools`	Stages portable Monogram/NIIA and optional Monograph payloads, writes `VERSIONS`, and starts the `:8899` tool host.	Executable/cache/process mutation; run after reboot or a stale doctor row.
`bench list`	Lists registered bench sets and tool arms.	Read-only discovery.
`bench watch`	Shows the state of every bench-set estate.	Read-only overview; `bench <id> watch` drills into one set.
`bench fetch-baselines swe-bench-verified`	Imports published per-instance solved sets and computes empirical solve rates.	Network/data write; use the field’s headroom rather than inventing a local baseline survey.
`bench import <set>`	Converts SWE-Bench Verified/Pro data into Pier tasks.	Data write; supports ids/limits/rate/difficulty and Pro language filters. Keep `--include-spec` off for reportable Pro runs.
`bench prune-cache`	Reclaims Docker build cache and optionally stopped containers or Colima sparse-disk blocks.	Destructive maintenance; dry-run first and confirm no live sweep. `--force` overrides the live-work refusal.

doctor proves the served Linux artifacts rather than only the macOS/host PATH. Every versioned tool arm records that served version, and cache reuse is valid only when the cached tool version matches it.

Per-Set Measurement and State

Here <id> is deepswe, swe-bench-verified, or swe-bench-pro.

Command	What it does	Effect or truth boundary
`bench <id> tasks`	Lists imported tasks and current filtering metadata.	Read-only.
`bench <id> run <task> --tool <arm>`	Solves one task through Pier and invokes the set’s verifier.	Expensive Docker/model mutation; supports `--cli`, `--model`, and `--via`.
`bench <id> sweep`	Runs a task × arm × repeat experiment.	Writes jobs, cache, inflight claims, and archive; use `--sample`, `--tools`, `--repeats`, `--batch-size`, `--jobs`, and `--no-cache` deliberately.
`bench <id> status [--json]`	Shows canonical live markers, cache/archive axes, pause state, and current jobs.	Read-only; JSON is the full automation state.
`bench <id> agent-status [--json]`	Emits a compact AI-first subset with liveness, latest version-honest axes, active jobs, and next commands.	Read-only schema `monobench.bench-agent-status.v1`.
`bench <id> agent-tail`	Reads a bounded active or selected job agent-log tail.	Read-only; `--lines` default 80/max 500, optional `--job`, structured JSON.
`bench <id> report <task>`	Counts every physical job attempt for a task, including retries hidden by cache deduplication.	Read-only physical-run truth.
`bench <id> lift --tool <arm> --model <m>`	Measures tool-solved rate only where baseline never solved and the tool ran.	Reportable admission-gated lift; `--version` or `--all-versions` controls version scope.
`bench <id> schemes [--model M] [--json]`	Compares binary, `g95`, `g90`, `g85`, and guarded-mean scoring per task/arm/tool version.	Read-only, P2P-guarded what-if view computed from cache rows; it does not rewrite reward or select policy.
`bench <id> checkpoint`	Writes canonical state to `latest-checkpoint.json` plus history.	Explicit handoff write; optional `--note` and `--json`.
`bench <id> pause`	Requests safe stop before a new task claim.	Writes pause state; already-running old binaries may not honor it.
`bench <id> resume`	Removes the pause request.	State mutation; does not resurrect a dead solver.
`bench <id> snapshot`	Writes a dated cache subset/full copy plus manifest for one tool/model axis.	Audit artifact write.
`bench <id> archive`	Verifies/resynchronizes durable aggregate axes.	Archive write/repair; `--versions` exposes every version rather than only latest clean rate.
`bench <id> watch`	Explores one set by axis → task → run → normalized commands.	Read-only; filters by tool/model/run.

Never launch a sweep through | tail: tail waits for EOF and makes a healthy long-running sweep look frozen. Launch it directly or redirect logs, then use status, agent-status, and agent-tail.

Recent-Session Cohort Commands

Command	What it does	Main filters
`jobs`	Lists recent physical Pier sessions with state, reward, provider/model, steps, task, and exact path.	`--max`, `--scan`, task/provider/text/reward/running filters, `--json`
`session-scan`	Batch-runs the single-session audit logic and summarizes status, flags, command mix, and Monogram use.	Same cohort filters plus `--flag` and `--audit-contains`
`session-queue`	Ranks concrete sessions by status and risk/warn/info gaps, then prints the next drill-down command.	Same cohort filters
`session-latest`	Selects the newest sessions per provider/model/schema/status/task cohort.	`--group-by`, `--top`, and cohort filters
`session-trends`	Aggregates cross-session status, command, query, path, and repeated-pattern motifs.	`--top` and cohort filters
`session-compare --left <job> --right <job>`	Compares normalized audits, flags, command mixes, Monogram subcommands, and first differing command.	`--max`, `--json`

All six are read-only routing surfaces. They select the job path to pass to the single-session commands below; they do not rerun a solver or verifier.

Single-Session Analysis Commands

Every command in this table is read-only and requires --job <jobs/stamp-or-trial> unless noted.

Command	What it explains	Important controls
`session-brief`	One-page continuation dashboard: verdict, audit flags, artifacts, signals, checkpoints, anchors, next commands.	`--max`, `--full`, `--json`
`session-gaps`	Missing cache/reward/verifier/patch/test/routing evidence and risky rows.	`--max`, `--full`, `--json`
`session-plan`	Deduplicated ordered runbook assembled from the other review views.	`--max`, `--full`, `--json`
`session-outline`	Chunked turn ranges before opening raw text.	`--chunk`, `--max`, `--full`, `--json`
`session-evidence`	Effective verdict, cache/reward, bounded verifier, patch, result, and checkpoint evidence.	`--artifact-lines`, `--contains`, `--max`, `--full`
`session-actions`	Ranked next analysis commands from gaps, signals, checkpoints, and evidence.	`--max`, `--full`, `--json`
`session-scorecard`	Compact status/count/signal hub with original turn/step coordinates.	`--max`, `--full`, `--json`
`session-windows`	Bounded turn contexts around deduplicated scorecard centers.	`--radius`, `--max`, `--full`
`session-handoff`	Transfer summary for another AI or human reviewer.	`--max`, `--full`, `--json`
`session-checkpoints`	Planning, tool-routing, search, test, verifier, patch, error, and final milestones.	Turn/range/around/radius/text/kind/flag filters
`session-artifacts`	Bounded, line-numbered raw cache/verifier/reward/stdout/stderr/patch/agent/result/log files.	`--artifact`, `--contains`, `--max-lines`, `--head`
`session-digest`	First message/tool/Monogram/sequence/reindex/test/error anchors.	`--max`, `--json`
`session-map`	Dense normalized command table retaining original turn and step ids.	Turn/range/around/text/kind/subcommand/flag filters
`session-queries`	Extracted search phrases with tool, scope path, and flags.	Map filters plus `--query` and `--path`
`session-patterns`	Repeated command, query, scope, path, and warning motifs.	Turn/range/around/text/kind/flag filters
`session-paths`	Touched path counts, first/last turns, sources, and artifact/repo/verifier flags.	Turn/range/around/text/path/flag filters
`inspect-job`	Provider-neutral Pier artifact dossier.	`--json`
`session`	Normalized ATIF turns with raw-agent fallback for active jobs.	`--full`, `--max`, `--tail`, turn/range/around/source/text, tools/messages/Monogram/error filters
`session-audit`	Command mix, Monogram routing, stale-index/reindex, raw fallback, reward, and running state.	`--json`
`timeline`	Merges normalized turns with cache/reward/verifier/patch/result checkpoints.	Same range/content filters as `session`
`verdict`	Classifies effective outcome as pass, partial, fail, running, or unknown.	Reads cache first for externally graded SWE-Bench; otherwise `reward.txt`.
`tests`	Reads raw per-test verifier rows from `verifier/ctrf.json`, including F2P/P2P suite tags.	`--status`, `--suite`, `--contains`, `--max`, `--json`; CTRF-era jobs only, read-only.
`commands`	Lists normalized CLI commands from archived job telemetry.	`--tool`, `--model`, `--task`, `--job`, `--all`, `--command`, `--contains`, `--max`, `--json`

Recovery and Pinpoint Cleanup

Command	What it does	Safety contract
`recover-job <task> --tool <arm> --model <m> --job <path>`	Records one explicit completed DeepSWE `reward.txt` into cache, archive, and checkpoint.	DeepSWE only; never scan arbitrary job dirs or use in-job reward as SWE-Bench external truth.
`drop-job --job <path>`	Previews removal of one contaminated/interrupted physical job.	Dry-run by default; `--apply` needs exactly one cache match. `--delete-raw` removes only that exact path; broad `jobs/` targets are refused.
`scrub-harness-errors`	Finds cached reward-zero rows whose logs prove setup failed before the solver started.	Dry-run first; `--apply` removes matched rows and re-archives only affected axes.
`rescore`	Backfills F2P/P2P score-detail fields from each existing cache row’s own persisted artifacts.	Plan-only by default. `--apply` changes only six score-detail fields, never reward/confounded/run_quality, and cannot resurrect removed rows.

monobench bench <id> rescore [--apply] [--from-stdout] \
  [--tool <arm>] [--task <task>] [--model <model>] [--json]

DeepSWE’s in-job reward.txt is its grade. SWE-Bench Verified and Pro report the external cache/archive outcome. A raw pass rate pools easy tasks and is not the same claim as admission-gated lift.

Examples

# Compare a tool against baseline
monobench run bun-1.3.10-toThreadSafe baseline
monobench run bun-1.3.10-toThreadSafe monogram
monobench report bun-1.3.10-toThreadSafe

# Multi-run comparison across a tool and a model
monobench matrix bun-1.3.10-toThreadSafe \
    --tools baseline,monogram --cli claude --model haiku --runs 3 --jobs 2

# Investigate a MISS
monobench evidence bun-1.3.10-toThreadSafe <run> --pattern ROOTCAUSE
monobench trace    bun-1.3.10-toThreadSafe <run>
monobench export   bun-1.3.10-toThreadSafe <run>

​Commands

​version

​help

​root

​list

​tools

​show

​run

​prepare

​matrix

​sweep

​absorb

​stop

​clean

​tui

​grade

​judge

​review

​status

​watch

​report

​summary

​gate

​ladder

​column

​adoption

​monogram-audit

​inspect

​integrity

​evidence

​trace

​export

​note

​meter

​add

​Bench-Set Commands

​Estate Setup and Data

​Per-Set Measurement and State

​Recent-Session Cohort Commands

​Single-Session Analysis Commands

​Recovery and Pinpoint Cleanup

​Examples

Commands

version

help

root

list

tools

show

run

prepare

matrix

sweep

absorb

stop

clean

tui

grade

judge

review

status

watch

report

summary

gate

ladder

column

adoption

monogram-audit

inspect

integrity

evidence

trace

export

note

meter

add

Bench-Set Commands

Estate Setup and Data

Per-Set Measurement and State

Recent-Session Cohort Commands

Single-Session Analysis Commands

Recovery and Pinpoint Cleanup

Examples