monobench 0.2.7, OpenCLIs build
2026-07-12-042014. The executing binary, registry, and public CDN metadata
agree on this release.
The public dispatcher is larger than the usual quick-start surface:
35 canonical top-level commands, nine global bench estate/data actions,
and 48 canonical per-bench-set actions. The five commands most easily lost
from a short guide are top-level gate and ladder, plus per-set tests,
schemes, and plan-first rescore.
Quick Start
[NEXT] hint, so the CLI is self-guiding — no
command dead-ends.
Published Platforms
The Windows artifact includes
monobench.exe. Actual benchmark execution has
additional runtime dependencies — model CLI/auth, Pier, Docker or Colima, and
the staged Linux tool host — so an artifact existing is not proof that
bench doctor will pass on that machine.
The Admission Gate
Most “tool helps” claims are tested on grep-solvable problems, where a strong model with plain grep wins and the tool adds nothing. Monobench admits an instance only if ALL six criteria hold:
C5 is the core safeguard: monobench always runs the baseline arm first. If a
baseline — given the identical “dig deep” directive every tool arm gets —
solves the bug cheaply more than half the time (default threshold), the
instance is judged non-discriminating and down-weighted rather than counted
as a real test of the tool. This is the single mechanism that keeps the
benchmark from being a grep-solvable toy.
Two Roles: Fast Signal vs. the Number You Report
Monobench measures the same comparison two different ways, and the difference matters for how much to trust a number. Native instances — a corpus of 50+ pinned real bugs shipped with the tool, graded by keyword match against a gated answer key. Fast, and the right harness for tuning a tool’s prompt injection and verifying the agent actually uses the tool — but the keyword grade can leak if a symptom happens to name the fix symbol. This is the tool’s own development signal, not the number to publish. Bench-sets — monobench also plugs into external, independently graded benchmarks: DeepSWE (113 tasks, graded by hidden in-container tests) and SWE-Bench Verified / SWE-Bench Pro (graded through their own external harness). Same tool arms, same fairness rules below — only the verifier changes, and it is never monobench’s own keyword match. This is the number you report.Fairness Rules
Six rules, each born from a real failure mode:- Every arm gets the identical “dig deep” directive — only the tool differs. If only the tool arm were told to dig deep, the benchmark would measure the prompt, not the tool.
- Admission gate — baseline runs first; a cheaply solved instance is down-weighted (see above). Only hard problems count.
- Symptom ≠ cause, no text bridge — instances where a grep on the symptom finds the cause are rejected outright.
- The answer key is never shown to the solver — it receives only the task description; the grader alone holds the fix and the mechanism keywords.
- Git access is blocked for every arm. The on-demand clone carries full
history — including the fix commit — so without this rule an agent can
simply search the git log for the symptom and copy the answer from the
fix commit. Monobench caught baselines doing exactly this: runs posting
3/3 “FULL” that turned out to be pure git-history cheating (8 and 4
un-denied
gitcalls, exposed bymonobench trace). All arms run git-blocked, andmonobench adoptionreports every git attempt and confirms each was denied. - The tool is forced into the prompt, not merely offered. Its full documentation is placed directly in the prompt, and the agent is told to run it first. A tool only mentioned in passing sees roughly 5-13% adoption; forcing the docs in-prompt with a run-first directive lifts both adoption and correctness sharply.
Arms: Native Adapters vs Bench-Set Arms
The native harness ships adapter directories such asbaseline, monogram,
monogram-mcp, and codegraph, and accepts custom adapters. baseline is the
control: no index step, no injected skill, builtins only. A tool that cannot
index a repository is recorded FORFEIT rather than silently scored as a
miss.
The external bench-set engine has its own fixed public arm enum. bench list
in 0.2.7 reports nine arms:
These diagnostic arms are not interchangeable with native adapter names, and
their version axes include the served tool/lead identity used in the sandbox.
CLI & Model Axes
The model invocation is split into two axes, so the CLI environment and the model are measured separately:--cli agy --model claude-opus-4.1 and
--cli claude --model claude-opus-4.1 are distinct runs.
--model is free-form text passed straight to the chosen CLI. --via
chooses how it’s launched: directly (default), or through an already-running
session (--via niia). Gemini has no direct CLI runner today — reach Gemini
models through --cli agy instead.
Grading & Two-Stage Review
Every run must end its answer withROOTCAUSE: <file>::<fn> and
FIX: <one sentence>. That answer is graded against the gated key:
Grading is two stages.
grade is deterministic and automatic — a keyword
match against the answer key, fast enough to run after every run. judge
builds a prompt that does see the answer key, for an orchestrating LLM or a
human to read — it never calls a model itself, on purpose, so the solver
process stays separate from the reviewer that sees the key. review records
that checked decision. The automatic grade is a fast dev signal; the reviewed
grade is the trustworthy one.
Tool Adoption
A tool the agent never called was not tested. Monobench reports the share of tool calls that went to the tool under test (mono%) and where in the run the first call happened — the fact that keeps “the model just didn’t use it” from silently masquerading as a real MISS.Reading a Report
report, summary, and column show medians, not means — these bugs
have high variance, so run at least 3 repeats per arm — and a per-instance
breakdown, because a pooled win across the whole corpus can still hide a loss
on one instance. Cost and token figures show an em-dash (”—”), never a fake
$0.00, for CLIs that structurally can’t report them today (grok, opencode,
and agy) — an unmeasurable number is never presented as zero.
Bench-Sets: DeepSWE and SWE-Bench
Beyond the native corpus, monobench plugs into two independently graded external benchmarks: DeepSWE (113 tasks, hidden in-container tests) and SWE-Bench Verified / SWE-Bench Pro (graded through their own external harness). Both run the identical tool arms and fairness rules above — only the verifier changes, and it is never monobench’s own keyword match.Instances
The native corpus ships with the tool — 50+ pinned, real, merged-fix bugs across a dozen-plus languages (Zig, Rust, C++, Python, Go, TypeScript, Dart, Swift, and more), each with a task description and a gated ground-truth file the solver never sees. Some instances also carry a decoy — a plausible-looking wrong answer that punishes shallow pattern-matching.Design Notes
Every run is isolated in its own throwaway git worktree and cleaned up automatically when it ends — including its temporary code-search index — so results don’t cross-contaminate and disk usage doesn’t grow unbounded across a long benchmarking session. A corrupt or partial clone fails loudly rather than silently grading a broken checkout. Monobench also feeds Monolex’s own tool-development process: the same evidence that grades a run is what tells monogram’s makers which primitive to improve next.Install
monobench list. Monobench pairs naturally with
monogram (the tool most often benchmarked), monometer (an independent
token/cost cross-check), and niia (an optional execution path via
--via niia).
Relationships and Ownership
Release source is owned by the public
monolex/monobench repository. Source,
installed binary, CDN metadata, benchmark cache, and verifier output remain
separate proof layers.
See the command reference for all commands and options.