> ## Documentation Index
> Fetch the complete documentation index at: https://docs.monolex.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Monobench

> Root-cause benchmark for code-intelligence tools — does monogram (or any tool) actually help an AI agent find a real bug's cause, versus a baseline told to dig equally hard?

Monobench measures whether giving an AI coding agent a code-intelligence tool
— monogram, codegraph, or one you write — actually helps it find the ROOT
CAUSE of a real bug, compared to a baseline agent told to dig just as hard.
It only admits bugs where the symptom and the cause are linked by *structure*
(a call graph, an ownership edge, an FFI boundary) rather than shared text,
because a strong baseline with plain grep already wins the textual case.
Think "SWE-bench, but for comparing code-intelligence tools."

*Published contract: `monobench 0.2.7`, OpenCLIs build
`2026-07-12-042014`. The executing binary, registry, and public CDN metadata
agree on this release.*

<Warning>
  Release and development source are intentionally different right now. The
  installed `0.2.7` payload exposes the **35 / 9 / 48** command inventory
  documented here and rejects both `monobench docs` and `monobench verify` as
  unknown commands. The current checkout still carries version `0.2.7`, but adds
  embedded `docs`, a `verify` command, and benchmark-owned process adapters such
  as OSWorld V2. Those additions are development-only until a new artifact and
  registry contract are published. The immutable release guide hashes to
  `c4266eb68d309c3ac092eaf84f85054261ed944c56c1dde25e00c3bdb716a18f`;
  the current source guide hashes to
  `9b2bb3d407470ac94dfb0827e82c8763034111aedf5ae13028ce9ae8077466aa`.
  Do not infer shipped commands from the source version string alone.
</Warning>

The public dispatcher is larger than the usual quick-start surface:
**35 canonical top-level commands**, **nine global `bench` estate/data actions**,
and **48 canonical per-bench-set actions**. The five commands most easily lost
from a short guide are top-level `gate` and `ladder`, plus per-set `tests`,
`schemes`, and plan-first `rescore`.

***

## Quick Start

```bash theme={null}
openclis install monobench

monobench list                 # browse the pinned bug corpus
monobench show <id>            # read the task (no spoilers)
monobench run <id> baseline    # the control run
monobench run <id> monogram    # the tool arm
monobench report <id>          # FULL hit-rate, cost, tokens, tool-adoption
```

Every command ends with a `[NEXT]` hint, so the CLI is self-guiding — no
command dead-ends.

## Published Platforms

| Target                       | Published archive SHA-256                                          |
| ---------------------------- | ------------------------------------------------------------------ |
| macOS Arm (`darwin-aarch64`) | `800f79ed3c85e2d0a9f9e844a12c9bae41a34311a3017be3271e47b83ce02ed3` |
| Linux x86\_64                | `63547bb0f480d2c9dbee00154c6fba33317fb3fd49b8d2acd2f54d77543c7d70` |
| Linux Arm64                  | `b872ab574c157195c1d6e43f5117f02f2e76463f4a899e940cd2a84df9c4ad0e` |
| Windows x86\_64              | `6a00ccd73e686d538ef7491949b9d9823daa0ae7c00f14e552a337c677da4d2a` |
| Intel macOS / Windows Arm64  | Not published                                                      |

The Windows artifact includes `monobench.exe`. Actual benchmark execution has
additional runtime dependencies — model CLI/auth, Pier, Docker or Colima, and
the staged Linux tool host — so an artifact existing is not proof that
`bench doctor` will pass on that machine.

***

## The Admission Gate

Most "tool helps" claims are tested on grep-solvable problems, where a strong
model with plain grep wins and the tool adds nothing. Monobench admits an
instance only if ALL six criteria hold:

| #  | Criterion                                                                                       |
| -- | ----------------------------------------------------------------------------------------------- |
| C1 | Symptom location ≠ root-cause location (ideally different file *and* language)                  |
| C2 | No text token in the symptom greps directly to the cause — the link is a call or ownership edge |
| C3 | Objective ground truth (a merged fix commit)                                                    |
| C4 | Contamination controlled (recent or obscure bugs, risk recorded)                                |
| C5 | **Discriminating** — the admission gate itself                                                  |
| C6 | The tool's claimed capability is on the critical path (e.g. cross-language call chain)          |

C5 is the core safeguard: monobench always runs the baseline arm first. If a
baseline — given the identical "dig deep" directive every tool arm gets —
solves the bug cheaply more than half the time (default threshold), the
instance is judged non-discriminating and down-weighted rather than counted
as a real test of the tool. This is the single mechanism that keeps the
benchmark from being a grep-solvable toy.

***

## Two Roles: Fast Signal vs. the Number You Report

Monobench measures the same comparison two different ways, and the difference
matters for how much to trust a number.

**Native instances** — a corpus of 50+ pinned real bugs shipped with the
tool, graded by keyword match against a gated answer key. Fast, and the right
harness for tuning a tool's prompt injection and verifying the agent actually
uses the tool — but the keyword grade can leak if a symptom happens to name
the fix symbol. This is the tool's own development signal, not the number to
publish.

**Bench-sets** — monobench also plugs into external, independently graded
benchmarks: **DeepSWE** (113 tasks, graded by hidden in-container tests) and
**SWE-Bench Verified / SWE-Bench Pro** (graded through their own external
harness). Same tool arms, same fairness rules below — only the verifier
changes, and it is never monobench's own keyword match. **This is the number
you report.**

***

## Fairness Rules

Six rules, each born from a real failure mode:

1. **Every arm gets the identical "dig deep" directive** — only the tool
   differs. If only the tool arm were told to dig deep, the benchmark would
   measure the prompt, not the tool.
2. **Admission gate** — baseline runs first; a cheaply solved instance is
   down-weighted (see above). Only hard problems count.
3. **Symptom ≠ cause, no text bridge** — instances where a grep on the
   symptom finds the cause are rejected outright.
4. **The answer key is never shown to the solver** — it receives only the
   task description; the grader alone holds the fix and the mechanism
   keywords.
5. **Git access is blocked for every arm.** The on-demand clone carries full
   history — *including the fix commit* — so without this rule an agent can
   simply search the git log for the symptom and copy the answer from the
   fix commit. Monobench caught baselines doing exactly this: runs posting
   3/3 "FULL" that turned out to be pure git-history cheating (8 and 4
   un-denied `git` calls, exposed by `monobench trace`). All arms run
   git-blocked, and `monobench adoption` reports every git attempt and
   confirms each was denied.
6. **The tool is forced into the prompt, not merely offered.** Its full
   documentation is placed directly in the prompt, and the agent is told to
   run it first. A tool only *mentioned* in passing sees roughly 5-13%
   adoption; forcing the docs in-prompt with a run-first directive lifts both
   adoption and correctness sharply.

***

## Arms: Native Adapters vs Bench-Set Arms

The native harness ships adapter directories such as `baseline`, `monogram`,
`monogram-mcp`, and `codegraph`, and accepts custom adapters. `baseline` is the
control: no index step, no injected skill, builtins only. A tool that cannot
index a repository is recorded **FORFEIT** rather than silently scored as a
miss.

The external bench-set engine has its own fixed public arm enum. `bench list`
in `0.2.7` reports nine arms:

| Arm                      | Role                                        |
| ------------------------ | ------------------------------------------- |
| `baseline`               | No injected Monolex code-intelligence tool. |
| `monogram`               | Standard Monogram delivery.                 |
| `monogram-penalty`       | Penalty experiment family.                  |
| `monogram-penalty-exp1`  | Explicit first penalty experiment.          |
| `monogram-penalty-pen03` | Explicit `pen03` penalty variant.           |
| `monogram-monograph`     | Monogram plus Monograph fix-shape Books.    |
| `monogram-siblings-5`    | Five-sibling context experiment.            |
| `monogram-siblings-8`    | Eight-sibling context experiment.           |
| `niia`                   | NIIA-delivered tool workflow.               |

These diagnostic arms are not interchangeable with native adapter names, and
their version axes include the served tool/lead identity used in the sandbox.

***

## CLI & Model Axes

The model invocation is split into two axes, so the CLI environment and the
model are measured separately: `--cli agy --model claude-opus-4.1` and
`--cli claude --model claude-opus-4.1` are distinct runs.

| `--cli`    | What it runs                                                                                         |
| ---------- | ---------------------------------------------------------------------------------------------------- |
| `claude`   | Claude Code CLI, direct print mode                                                                   |
| `codex`    | Codex CLI, direct exec mode                                                                          |
| `agy`      | Antigravity CLI — reads its model from local settings and refuses the run if `--model` doesn't match |
| `grok`     | Grok Build CLI (OAuth subscription)                                                                  |
| `opencode` | opencode with a `provider/model` label                                                               |

`--model` is free-form text passed straight to the chosen CLI. `--via`
chooses how it's launched: directly (default), or through an already-running
session (`--via niia`). Gemini has no direct CLI runner today — reach Gemini
models through `--cli agy` instead.

***

## Grading & Two-Stage Review

Every run must end its answer with `ROOTCAUSE: <file>::<fn>` and
`FIX: <one sentence>`. That answer is graded against the gated key:

| Grade       | Meaning                                                     |
| ----------- | ----------------------------------------------------------- |
| `FULL`      | Names the root-cause function **and** the correct mechanism |
| `NAME_ONLY` | Right function, mechanism not demonstrated                  |
| `DECOY`     | Named the adjacent trap function instead                    |
| `MISS`      | Neither                                                     |
| `FORFEIT`   | The tool couldn't even index the repository                 |

Grading is two stages. `grade` is deterministic and automatic — a keyword
match against the answer key, fast enough to run after every run. `judge`
builds a prompt that *does* see the answer key, for an orchestrating LLM or a
human to read — it never calls a model itself, on purpose, so the solver
process stays separate from the reviewer that sees the key. `review` records
that checked decision. The automatic grade is a fast dev signal; the reviewed
grade is the trustworthy one.

***

## Tool Adoption

A tool the agent never called was not tested. Monobench reports the share of
tool calls that went to the tool under test (mono%) and where in the run the
first call happened — the fact that keeps "the model just didn't use it" from
silently masquerading as a real MISS.

***

## Reading a Report

`report`, `summary`, and `column` show **medians, not means** — these bugs
have high variance, so run at least 3 repeats per arm — and a per-instance
breakdown, because a pooled win across the whole corpus can still hide a loss
on one instance. Cost and token figures show an em-dash ("—"), never a fake
`$0.00`, for CLIs that structurally can't report them today (grok, opencode,
and agy) — an unmeasurable number is never presented as zero.

***

## Bench-Sets: DeepSWE and SWE-Bench

Beyond the native corpus, monobench plugs into two independently graded
external benchmarks: **DeepSWE** (113 tasks, hidden in-container tests) and
**SWE-Bench Verified / SWE-Bench Pro** (graded through their own external
harness). Both run the identical tool arms and fairness rules above — only
the verifier changes, and it is never monobench's own keyword match.

***

## Instances

The native corpus ships with the tool — 50+ pinned, real, merged-fix bugs
across a dozen-plus languages (Zig, Rust, C++, Python, Go, TypeScript, Dart,
Swift, and more), each with a task description and a gated ground-truth file
the solver never sees. Some instances also carry a **decoy** — a
plausible-looking wrong answer that punishes shallow pattern-matching.

***

## Design Notes

Every run is isolated in its own throwaway git worktree and cleaned up
automatically when it ends — including its temporary code-search index — so
results don't cross-contaminate and disk usage doesn't grow unbounded across
a long benchmarking session. A corrupt or partial clone fails loudly rather
than silently grading a broken checkout.

Monobench also feeds Monolex's own tool-development process: the same
evidence that grades a run is what tells monogram's makers which primitive to
improve next.

***

## Install

```bash theme={null}
openclis install monobench
```

First run: `monobench list`. Monobench pairs naturally with
**monogram** (the tool most often benchmarked), **monometer** (an independent
token/cost cross-check), and **niia** (an optional execution path via
`--via niia`).

## Relationships and Ownership

| Tool                                   | Boundary                                                                                          |
| -------------------------------------- | ------------------------------------------------------------------------------------------------- |
| [Monogram](/ai-clis/monogram/overview) | Primary code-intelligence arm; Monobench measures use and lift but does not own Monogram ranking. |
| [Monograph](/ai-clis/monograph)        | Supplies optional fix-shape Books to the `monogram-monograph` arm.                                |
| [Monometer](/ai-clis/monometer)        | Independent token/cost cross-check; it does not determine benchmark correctness.                  |
| [NIIA](/ai-clis/niia)                  | Optional host-session transport via `--via niia`; Pier still owns task setup and verifier.        |
| [Monomento](/ai-clis/monomento)        | Indexes exported run transcripts for later evidence search.                                       |

Release source is owned by the public
[`monolex/monobench`](https://github.com/monolex/monobench) repository. Source,
installed binary, CDN metadata, benchmark cache, and verifier output remain
separate proof layers.

See the [command reference](/ai-clis/monobench/reference) for all commands and options.