SWE-bench-lite Provisional Score — Funnel A/B + Raw %
Status: PRELIMINARY · N=15 · operator-run · scored by the OFFICIAL swebench harness Spec: specs/129-swebench-provisional-score.md
What this measures
Weftra reports two numbers from its SWE-bench-lite evaluation:
Headline — A/B lift:
funnel_resolved_rate − bare_resolved_rate
This is the FA-specific number: how much does running the bare problem statement through the FA clarify→spec→approve funnel improve resolution over submitting it directly?
This is defensible because both arms use the same resolver loop — the only variable is whether the funnel pre-processed the request.Anchor — raw resolved rate: per-arm absolute resolved % on the 15-instance subset
Familiar SWE-bench metric; context for comparing FA to other agents on the same task.
Both numbers are from a N=15 provisional run and are labeled accordingly.
Scoring: the OFFICIAL harness, not a hand-rolled script
resolved in every report comes from the official swebench package (python -m swebench.harness.run_evaluation) — never a hand-rolled apply-and-test script. Every report JSON names its scorer in scored_by:
swebench-official@<pinned version>— a real, harness-verified number.- an explicit
unscored — …placeholder — a live FA run that has not yet been scored. Itsresolved/resolved_ratefields are meaningless placeholders; do not publish them.
Why this matters — read before trusting any older number. The first version of this scorer (src/services/eval/swebench/eval-runner.ts, now retired) applied the candidate patch and the gold test_patch itself and ran pytest inside the official per-instance eval image by hand. It looked canonical — same image, same tests — but diverged from the real harness in three ways: it assumed a conda env layout that didn't always hold, it could clobber the gold test_patch with the agent's own test edits, and --network none silently failed instances whose graded tests need network (e.g. requests' httpbin tests). The first published number (9/15 = 60%, recorded 2026-07-20) had to be hand-run through the real harness to catch this — a scorer that looks canonical but isn't invites trust it hasn't earned. It has been deleted rather than kept around as a second scorer that could drift from the harness it was meant to approximate.
Running the evaluation
Prerequisites
- A running FA instance (
FA_EVAL_BASE_URL) and an FA admin API key (FA_EVAL_ADMIN_KEY) — only needed to run the FA arms themselves, not to score an existing predictions file. - A local Docker daemon, with
/var/run/docker.sockreachable — the harness step mounts the socket into apython:3.11container so the harness can spawn its own per-instance eval containers as siblings of that one. This is host-root-equivalent. Treat it exactly like runningdockeryourself: operator-only, never automated from a server process, never exposed over HTTP. FA enforces this structurally — the scorer/harness modules are importable only from the eval CLI (tests/swebench-scoring-isolation.test.tsfails the build otherwise). - ~2 USD per arm × 15 instances estimated agent spend, plus harness runtime (pulls the official per-instance eval images on first use).
Command — run the FA arms and score them in one step
FA_EVAL_BASE_URL=http://localhost:3100 \
FA_EVAL_ADMIN_KEY=your-admin-key \
npm run eval:swebench -- --arm both --score official \
--out data/swebench-provisional-report.jsonOptions:
| Flag | Description |
|---|---|
--arm bare|funnel|governed|both | Which arm(s) to run (required). both = bare+funnel; governed (spec 277) = funnel + implement + review + bounded fix — requested on its own, never combined with both |
--out <path> | Write JSON report to file (default: stdout) |
--run-id <id> | Unique run identifier for project naming |
--engine <label> | Engine/model label in the report |
--limit <N> | Run only the first N instances — cheap pipeline smoke test |
--concurrency <N> | Max concurrent (instance, arm) runs in flight (default: AGENT_MAX_CONCURRENCY, else 1 — serial, identical output to omitting the flag) |
--code-discipline off|lite|full | Set code_discipline on the created project (admin path, at creation) — pair with --arm funnel to produce the D (lite) / E (full) arms; see OPERATIONS.md §10a |
--reviewer-model <id> | governed arm only — model for the spec-conformance review round. Default: unset (platform default) — see "The governed arm" below |
--fixer-model <id> | governed arm only — model for the bounded fix round(s). Default: unset |
--fix-round-cap <N> | governed arm only — max bounded fix rounds after REQUEST_CHANGES (default: $REVIEWER_MAX_ROUNDS, else 2 — the same cap/env var FA's own auto-revise loop uses). 0 disables the fix loop |
--baseline-report <path> | governed arm only, requires --score official — a prior --arm funnel report's JSON file. Merges its per-instance funnel results into this run's report (by instance_id) so governed_funnel_paired_delta is computable from a single governed invocation. The baseline must be comparable (same official scorer, dataset, default model, and arm config outside review/fix) or the run is refused — see below |
--score official|none | Score via the official harness after the run (default: none) |
--dataset-name <name> | Dataset name passed to the harness (default: princeton-nlp/SWE-bench_Lite) |
--score-workdir <dir> | Host dir bind-mounted at /work for scoring (default: --out's dir, or cwd) |
The governed arm (spec 277) — measuring the loop FA actually sells
bare and funnel both stop at implement: run_mode: 'patch' produces no PR, and both reviewers skip on a missing pr_url. So neither arm has ever measured the independent reviewer + bounded fixer that FA sells as its differentiated half. --arm governed closes that: funnel → an explicit review request (spec 276's opt-in manual-run path — never an automatic side effect of patch mode) → a bounded fix loop when review returns REQUEST_CHANGES, scored by the same official harness.
Same-model is the default, and it is the stronger claim, not a fallback to be corrected later. The first governed runs use the platform default at implement, review, and fix alike — only --funnel-model (the clarify phase) differs, exactly as for funnel. This removes both "you spent more compute" and "you used a better model" as objections: if the same model produces the defect, catches it on review, and fixes it correctly, the gain can't be attributed to capability — only to structure (a different role, a fresh context, a rubric it didn't write). --reviewer-model / --fixer-model exist for a second experiment (an independently-modelled reviewer/fixer) and must stay unset for the first.
The headline is a paired delta, never an unpaired absolute. Report field governed_funnel_paired_delta = governed_resolved − funnel_resolved, computed only over instances carrying a result for both arms in the same run — never governed's raw resolved rate presented alone. A raw governed percentage is an anchor and must carry its N. The computation refuses its own inputs when they are not verdicts: a paired instance carrying scoring_infra_failure on either side (a crashed harness leaves placeholder resolved fields) suppresses the delta to null (computeGovernedFunnelPairedDelta, governed.ts) — the same both-sides rule that suppresses the bare/funnel lift for a scoring-failed arm.
governed is requested on its own — it never re-runs funnel itself, so pair it with --baseline-report. Because governed is "requested on its own, never combined with both" (above), a single --arm governed invocation carries no funnel per-instance results of its own — without more, governed_funnel_paired_delta stays null on every real run. Run --arm funnel once (or reuse an existing one over the same instances) and save its --out JSON, then pass that path to a later --arm governed run via --baseline-report: the CLI merges the baseline's per-instance funnel results into this run's report by instance_id (never overwriting a funnel result this run produced itself, which normally does not happen), and --score official then stamps the paired delta over the merged instances before printing/writing. Instances present in the governed run but missing from the baseline are named on stderr, never silently dropped from the count.
The pairing is CHECKED, not assumed — a baseline that isn't comparable is refused and no delta is published. A delta is only about review-and-fix if everything else about the two runs matches, so before merging anything the CLI asserts, against the two report artifacts themselves (validateBaselinePairing, governed.ts — called from cli.ts):
--score officialis on for this run, and the baseline carries the samescored_by(the same pinned official-harness version). Without this, a funnel baseline saved without--score officialcarriesresolved: falseeverywhere as a placeholder, and pairing against it would credit review-and-fix with the whole of governed's resolution count — a large fabricated "governance lift".- No scoring-infra failure on the baseline's funnel side — neither the arm-level
scoring_failedflag nor a per-instancescoring_infra_failure(a crashed harness leaves the same all-zero placeholder counts). - Same
datasetand samedefault_model— §4's claim is "same instances, same models", and anullrole model in an arm config means "the platform default", so two runs with differentdefault_modelvalues ran different models. - The resolved arm configs differ only in the reviewer/fixer fields (§8-5) —
autonomy_mode,force_clarify,code_discipline,analyze_model,answerer_modelmust match. A funnel baseline run with--code-discipline fullpaired against a governed run without it would produce a delta that measures code discipline. A config that wasn't captured on either side is also a refusal: comparability can't be asserted, so it isn't assumed.
Each unmet precondition is printed on stderr with the differing values, and the run exits non-zero without printing or writing the report at all.
# 1. establish (or reuse) a funnel baseline over the same instances — note --score
# official: an unscored baseline is refused at step 2, by design
npm run eval:swebench -- --arm funnel --score official \
--out data/swebench-funnel-baseline.json
# 2. pair the governed run against it (same dataset, same --default-model, and no
# per-arm flag — e.g. --code-discipline — that the baseline run didn't also have)
npm run eval:swebench -- --arm governed --score official \
--baseline-report data/swebench-funnel-baseline.json \
--out data/swebench-governed-report.jsonEvery governed instance record carries, never inferred: whether review ran and its verdict (review.ran/decision/unmet_requirements/skip_reason, plus rereview_error when a completed review drove a fix round but the re-review of the overwritten with ran: false, and validation rejects a record claiming ran: false fix rounds ran and whether the patch changed (fix_rounds/patch_changed_by_fix), and whether the fixer could execute the project's tests (tests_executable + tests_executable_reason — spec 276 §5a: no image today carries both the claude CLI and the SWE-bench test environment, so this is false today; this spec does not resolve §5a for the benchmark, it only refuses to hide which applied). A report missing any of these three fields on any governed instance fails validation and is neither printed nor written — see docs/OPERATIONS.md §10e for the full mechanism (how the bounded fix loop is orchestrated without a new FA route, the resolved arm-config fields, and where each surface renders them).
FA_EVAL_BASE_URL=http://localhost:3100 \
FA_EVAL_ADMIN_KEY=your-admin-key \
npm run eval:swebench -- --arm governed --score official \
--baseline-report data/swebench-funnel-baseline.json \
--out data/swebench-governed-report.json(Omitting --baseline-report still runs the full governed pipeline and produces a valid report — every §4 per-instance field is still required and validated — it just leaves governed_funnel_paired_delta null, since there is nothing to pair against yet. That null is the honest state: prefer it to a delta the comparability checks above would have refused.)
Every report — JSON and CLI text summary — also carries per-arm LOC (exact, always present once an arm has a captured patch) and complexity (best-effort, labeled [EXPERIMENTAL], populated only when --score official runs). See OPERATIONS.md §10a for the full breakdown of both fields.
Omitting --score official still runs the FA arms and captures patches, but the report's resolved fields are placeholders (scored_by says so explicitly) — run the score step below (or re-run with --score official) before publishing a number.
Concurrency. By default the runner is serial — one (instance, arm) run at a time, same as before this flag existed. --concurrency N (N > 1) bounds up to N (instance, arm) runs in flight at once, so wall-clock is governed by the pool size instead of the sum of every run — this is what makes a larger N (e.g. a governed A/B) affordable to run at all. The produced report is identical either way: instance order always matches the input dataset order (results are assembled by index, not completion order), and each arm's config (the read-back-from-the-project stamp, spec 168) is still captured at most once per arm even when several of that arm's runs start together. When --concurrency is omitted, the CLI falls back to AGENT_MAX_CONCURRENCY (the same env var the agent engine itself reads), then 1.
Arm-distinct predictions. --arm both scores bare and funnel as two separate harness invocations against the same --score-workdir. Each arm writes its own predictions file — predictions-bare.jsonl / predictions-funnel.jsonl — with its own model_name_or_path (<engine>-bare / <engine>-funnel), so both arms' patches for the same instance_id land in separate files instead of one arm's predictions overwriting the other's. This only applies to the --arm both path; the standalone score subcommand below always writes a single predictions.jsonl (it has no arm to distinguish).
Scoring can fail loudly — never a fake 0%. If the harness itself crashes or never writes a readable report for an arm (e.g. a transient PyPI/Hugging Face network blip), that arm's report carries scoring_failed: true and its instances carry the scoring_infra_failure reason instead of a real resolved/unresolved verdict. The text report prints *** SCORING FAILED (infra) — NOT a 0% *** for that arm (and withholds the A/B lift headline, since it can't be computed from a real number), and the CLI exits non-zero. Treat a non-zero exit from --score official as "scoring did not complete", not "the arm scored 0%" — re-run scoring once the transient issue clears.
Environment variables:
| Variable | Description | Default |
|---|---|---|
FA_EVAL_BASE_URL | FA server base URL | required |
FA_EVAL_ADMIN_KEY | FA admin API key | required |
FA_EVAL_ENGINE | Engine/model label | sonnet/claude-sonnet-4-6 |
FA_EVAL_POLL_INTERVAL_MS | Poll interval | 30000 |
FA_EVAL_TIMEOUT_MS | Per-instance-arm timeout | 3600000 |
FA_SWEBENCH_REPORT_PATH | Path to report file served by FA | data/swebench-provisional-report.json |
Command — score an existing predictions.jsonl (no FA server needed)
If you already have patches captured (a prior FA run, or predictions from anywhere else) in the standard SWE-bench predictions.jsonl format ({instance_id, model_name_or_path, model_patch} per line), score them directly:
npm run eval:swebench -- score \
--predictions data/predictions.jsonl \
--out data/swebench-score-report.json| Flag | Description |
|---|---|
--predictions <path> | predictions.jsonl to score (required) |
--out <path> | Write JSON verdict report to file (default: stdout) |
--run-id <id> | Harness run id (default: timestamp) |
--dataset-name <name> | Dataset name passed to the harness (default: princeton-nlp/SWE-bench_Lite) |
--workdir <dir> | Host dir bind-mounted at /work (default: predictions.jsonl's directory) |
--max-workers <N> | Harness parallelism (default: 4) |
This still requires the local-Docker prerequisite above — no FA server or admin key. Unlike the --arm both path above, this subcommand scores one arm-less predictions file, so it keeps writing a plain predictions.jsonl under --workdir with whatever model_name_or_path the input file already names — there's no arm to make distinct. It carries the same loud-failure behavior: a harness crash / unreadable report sets scoring_failed: true on the JSON report, prints the SCORING FAILED (infra) banner to stderr instead of a resolved percentage, and exits non-zero.
Publishing the result
After a run with --score official (or a score run merged back in) writes data/swebench-provisional-report.json:
- Dashboard: Available immediately at
/eval-report.html(admin-gated, SWE-bench tab) viaGET /api/eval/swebench-report, which now also showsscored_by. - Docs site: Copy the report to
data/swebench-provisional-report.json, then rebuild:bashThe build script picks up the JSON and publishes it tonpm run docs:builddocs-site/public/.
Before publishing any number, check scored_by in the report JSON. If it isn't swebench-official@<version>, it's an unscored placeholder — score it first.
N=15 instance subset
The 15 instances are real rows from princeton-nlp/SWE-bench_Lite (test split), restricted to pytest-runner repositories (requests, flask, pytest, pylint, xarray) so the derived test_command is a correct pytest invocation. Each runtime_image is the official eval image, whose Docker tag encodes the instance_id's __ as _1776_.
| Instance ID | Repository | Docker image |
|---|---|---|
psf__requests-1963 | psf/requests | sweb.eval.x86_64.psf_1776_requests-1963 |
psf__requests-2148 | psf/requests | sweb.eval.x86_64.psf_1776_requests-2148 |
psf__requests-2317 | psf/requests | sweb.eval.x86_64.psf_1776_requests-2317 |
pallets__flask-4045 | pallets/flask | sweb.eval.x86_64.pallets_1776_flask-4045 |
pallets__flask-4992 | pallets/flask | sweb.eval.x86_64.pallets_1776_flask-4992 |
pallets__flask-5063 | pallets/flask | sweb.eval.x86_64.pallets_1776_flask-5063 |
pytest-dev__pytest-11143 | pytest-dev/pytest | sweb.eval.x86_64.pytest-dev_1776_pytest-11143 |
pytest-dev__pytest-11148 | pytest-dev/pytest | sweb.eval.x86_64.pytest-dev_1776_pytest-11148 |
pytest-dev__pytest-5103 | pytest-dev/pytest | sweb.eval.x86_64.pytest-dev_1776_pytest-5103 |
pylint-dev__pylint-5859 | pylint-dev/pylint | sweb.eval.x86_64.pylint-dev_1776_pylint-5859 |
pylint-dev__pylint-6506 | pylint-dev/pylint | sweb.eval.x86_64.pylint-dev_1776_pylint-6506 |
pylint-dev__pylint-7080 | pylint-dev/pylint | sweb.eval.x86_64.pylint-dev_1776_pylint-7080 |
pydata__xarray-4248 | pydata/xarray | sweb.eval.x86_64.pydata_1776_xarray-4248 |
pydata__xarray-5131 | pydata/xarray | sweb.eval.x86_64.pydata_1776_xarray-5131 |
pydata__xarray-3364 | pydata/xarray | sweb.eval.x86_64.pydata_1776_xarray-3364 |
The raw records (verbatim dataset fields) are in src/services/eval/swebench/instances.dataset.ts, generated by scripts/fetch-swebench-instances.ts — never hand-authored. instances.ts derives repo_url, runtime_image, and test_command from them.
Resolution criterion
An instance is resolved when the official harness's resolved_ids for that run contains its instance_id — i.e. the candidate patch applies AND every FAIL_TO_PASS
PASS_TO_PASStest the harness itself runs, in the harness's own per-instance eval image, passes. FA never re-implements this check:
src/services/eval/swebench/predictions.tsbuilds a standardpredictions.jsonlfrom FA's captured patches (empty patches are excluded up front and reportedempty_patch— no reason to spend a harness run on an instance with no patch).src/services/eval/swebench/harness.tsinvokespython -m swebench.harness.run_evaluationinside apython:3.11container.src/services/eval/swebench/report-parser.tsmaps the harness's ownresolved_ids/unresolved_ids/error_idsback onto FA's per-instance report. An instance_id missing from all three of the harness's sets — e.g. its sibling eval container crashed or timed out — is surfaced aserror, never silently counted as a test failure.
error vs. scoring_infra_failure. error (above) means the harness ran to completion and wrote a real report that itself names this instance as errored — a harness-attested verdict. scoring_infra_failure is different and more severe: the harness invocation never produced a readable report at all (crashed before finishing, or its report file is missing/corrupt) — there is no verdict for any instance in that arm, so src/services/eval/swebench/official-scorer.ts marks the whole arm's SweBenchArmScore.scoring_failed and refuses to let report-format.ts render its resolved_rate as a percentage. See "Loud scoring-infra failure" above.
Interpreting results
- The lift is the primary FA claim. A positive lift (funnel better than bare) validates that the clarify→spec pre-processing step adds resolution value, not just PR volume.
- The raw % is a familiar anchor — compare with caution since this is N=15, not 300.
- The wasted-run count shows how many instances each arm spent credits on without resolving.
- Per-instance details in the report show exactly which instances resolved and at what cost.
- Always check
scored_bybefore quoting a number — see "Scoring" above.
This is a provisional run. The full N=300 evaluation (inc-2) is planned after this design is validated.