SWE-bench-lite Provisional Score — Funnel A/B + Raw %

Status: PRELIMINARY · N=15 · operator-run · scored by the OFFICIAL swebench harness Spec: specs/129-swebench-provisional-score.md

---

What this measures

Weftra reports two numbers from its SWE-bench-lite evaluation:

1. Headline — A/B lift: funnel_resolved_rate − bare_resolved_rate
   This is the Weftra-specific number: how much does running the bare problem statement through the Weftra clarify→spec→approve funnel improve resolution over submitting it directly?
   This is defensible because both arms use the same resolver loop — the only variable is whether the funnel pre-processed the request.

2. Anchor — raw resolved rate: per-arm absolute resolved % on the 15-instance subset
   Familiar SWE-bench metric; context for comparing Weftra to other agents on the same task.

Both numbers are from a N=15 provisional run and are labeled accordingly.

---

Scoring: the OFFICIAL harness, not a hand-rolled script

resolved in every report comes from the official swebench package (python -m swebench.harness.run_evaluation) — never a hand-rolled apply-and-test script. Every report JSON names its scorer in scored_by:

• swebench-official@<pinned version> — a real, harness-verified number.
• an explicit unscored — … placeholder — a live Weftra run that has not yet been scored. Its resolved/resolved_rate fields are meaningless placeholders; do not publish them.

Why this matters — read before trusting any older number. The first version of this scorer (src/services/eval/swebench/eval-runner.ts, now retired) applied the candidate patch and the gold test_patch itself and ran pytest inside the official per-instance eval image by hand. It looked canonical — same image, same tests — but diverged from the real harness in three ways: it assumed a conda env layout that didn't always hold, it could clobber the gold test_patch with the agent's own test edits, and --network none silently failed instances whose graded tests need network (e.g. requests' httpbin tests). The first published number (9/15 = 60%, recorded 2026-07-20) had to be hand-run through the real harness to catch this — a scorer that looks canonical but isn't invites trust it hasn't earned. It has been deleted rather than kept around as a second scorer that could drift from the harness it was meant to approximate.

---

Running the evaluation

Prerequisites

• A running Weftra instance (FA_EVAL_BASE_URL) and a Weftra admin API key (FA_EVAL_ADMIN_KEY) — only needed to run the Weftra arms themselves, not to score an existing predictions file.
• A local Docker daemon, with /var/run/docker.sock reachable — the harness step mounts the socket into a python:3.11 container so the harness can spawn its own per-instance eval containers as siblings of that one. This is host-root-equivalent. Treat it exactly like running docker yourself: operator-only, never automated from a server process, never exposed over HTTP. Weftra enforces this structurally — the scorer/harness modules are importable only from the eval CLI (tests/swebench-scoring-isolation.test.ts fails the build otherwise).
• ~2 USD per arm × 15 instances estimated agent spend, plus harness runtime (pulls the official per-instance eval images on first use).

Command — run the Weftra arms and score them in one step

FA_EVAL_BASE_URL=http://localhost:3100 \
FA_EVAL_ADMIN_KEY=your-admin-key \
npm run eval:swebench -- --arm both --score official \
  --out data/swebench-provisional-report.json

Options:

Flag	Description
--arm bare|funnel|governed|both	Which arm(s) to run (required). both = bare+funnel; governed (spec 277) = funnel + implement + review + bounded fix — requested on its own, never combined with both
--out <path>	Write JSON report to file (default: stdout)
--run-id <id>	Unique run identifier for project naming
--engine <label>	Engine/model label in the report
--limit <N>	Run only the first N instances — cheap pipeline smoke test
--concurrency <N>	Max concurrent (instance, arm) runs in flight (default: AGENT_MAX_CONCURRENCY, else 1 — serial, identical output to omitting the flag)
--code-discipline off|lite|full	Set code_discipline on the created project (admin path, at creation) — pair with --arm funnel to produce the D (lite) / E (full) arms; see OPERATIONS.md §10a
--reviewer-model <id>	governed arm only — model for the spec-conformance review round. Default: unset (platform default) — see "The governed arm" below
--fixer-model <id>	governed arm only — model for the bounded fix round(s). Default: unset
--fix-round-cap <N>	governed arm only — max bounded fix rounds after REQUEST_CHANGES (default: $REVIEWER_MAX_ROUNDS, else 2 — the same cap/env var Weftra's own auto-revise loop uses). 0 disables the fix loop
--baseline-report <path>	governed arm only, requires --score official — a prior --arm funnel report's JSON file. Merges its per-instance funnel results into this run's report (by instance_id) so governed_funnel_paired_delta is computable from a single governed invocation. The baseline must be comparable (same official scorer, dataset, default model, and arm config outside review/fix) or the run is refused — see below
--score official|none	Score via the official harness after the run (default: none)
--dataset-name <name>	Dataset name passed to the harness (default: princeton-nlp/SWE-bench_Lite)
--score-workdir <dir>	Host dir bind-mounted at /work for scoring (default: --out's dir, or cwd)

The governed arm (spec 277) — measuring the loop Weftra actually sells

bare and funnel both stop at implement: run_mode: 'patch' produces no PR, and both reviewers skip on a missing pr_url. So neither arm has ever measured the independent reviewer + bounded fixer that Weftra sells as its differentiated half. --arm governed closes that: funnel → an explicit review request (spec 276's opt-in manual-run path — never an automatic side effect of patch mode) → a bounded fix loop when review returns REQUEST_CHANGES, scored by the same official harness.

Same-model is the default, and it is the stronger claim, not a fallback to be corrected later. The first governed runs use the platform default at implement, review, and fix alike — only --funnel-model (the clarify phase) differs, exactly as for funnel. This removes both "you spent more compute" and "you used a better model" as objections: if the same model produces the defect, catches it on review, and fixes it correctly, the gain can't be attributed to capability — only to structure (a different role, a fresh context, a rubric it didn't write). --reviewer-model / --fixer-model exist for a second experiment (an independently-modelled reviewer/fixer) and must stay unset for the first.

The headline is a paired delta, never an unpaired absolute. Report field governed_funnel_paired_delta = governed_resolved − funnel_resolved, computed only over instances carrying a result for both arms in the same run — never governed's raw resolved rate presented alone. A raw governed percentage is an anchor and must carry its N. The computation refuses its own inputs when they are not verdicts: a paired instance carrying scoring_infra_failure on either side (a crashed harness leaves placeholder resolved fields) suppresses the delta to null (computeGovernedFunnelPairedDelta, governed.ts) — the same both-sides rule that suppresses the bare/funnel lift for a scoring-failed arm.

governed is requested on its own — it never re-runs funnel itself, so pair it with --baseline-report. Because governed is "requested on its own, never combined with both" (above), a single --arm governed invocation carries no funnel per-instance results of its own — without more, governed_funnel_paired_delta stays null on every real run. Run --arm funnel once (or reuse an existing one over the same instances) and save its --out JSON, then pass that path to a later --arm governed run via --baseline-report: the CLI merges the baseline's per-instance funnel results into this run's report by instance_id (never overwriting a funnel result this run produced itself, which normally does not happen), and --score official then stamps the paired delta over the merged instances before printing/writing. Instances present in the governed run but missing from the baseline are named on stderr, never silently dropped from the count.

A paired governed run starts from the funnel's own patch, and only ever swaps it for a patch that is no worse. With --baseline-report, round 0 of each instance whose baseline funnel result reached a patch IS that patch, byte for byte; nothing is re-authored, and the funnel's cost and duration count toward the governed totals. That patch cannot be put through review (review always reads a feature's own diff, never one supplied by the caller), so the only thing that starts a fix round on it is a regression measured in the project's own base suite. A fix round's patch replaces the scored one only if it has no more base-suite regressions, does not turn a repro that was fixed back into an unfixed one, and was judged by the suite; otherwise it is recorded in rejected_fix_rounds and the next round builds on the patch that stays scored. The hidden-test fail_to_pass set never takes part. The report names the result: round0_source (funnel_patch or authored), scored_round, adopted_fix_rounds, rejected_fix_rounds, patch_changed_by_fix (the scored patch differs from round 0), and governed_ge_funnel_guaranteed: true on a seeded instance. A standalone governed run still authors its own round 0, with the same monotonic selection across its fix rounds. This shared round 0 is an A/B isolation device, not a correctness claim: it removes author sampling variance from the comparison.

The pairing is CHECKED, not assumed — a baseline that isn't comparable is refused and no delta is published. A delta is only about review-and-fix if everything else about the two runs matches, so before merging anything the CLI asserts, against the two report artifacts themselves (validateBaselinePairing, governed.ts — called from cli.ts):

• --score official is on for this run, and the baseline carries the same scored_by (the same pinned official-harness version). Without this, a funnel baseline saved without --score official carries resolved: false everywhere as a placeholder, and pairing against it would credit review-and-fix with the whole of governed's resolution count — a large fabricated "governance lift".
• No scoring-infra failure on the baseline's funnel side — neither the arm-level scoring_failed flag nor a per-instance scoring_infra_failure (a crashed harness leaves the same all-zero placeholder counts).
• Same dataset and same default_model — §4's claim is "same instances, same models", and a null role model in an arm config means "the platform default", so two runs with different default_model values ran different models.
• The resolved arm configs differ only in the reviewer/fixer fields (§8-5) — autonomy_mode, force_clarify, code_discipline, analyze_model, answerer_model must match. A funnel baseline run with --code-discipline full paired against a governed run without it would produce a delta that measures code discipline. A config that wasn't captured on either side is also a refusal: comparability can't be asserted, so it isn't assumed.

Each unmet precondition is printed on stderr with the differing values, and the run exits non-zero without printing or writing the report at all.

# 1. establish (or reuse) a funnel baseline over the same instances — note --score
#    official: an unscored baseline is refused at step 2, by design
npm run eval:swebench -- --arm funnel --score official \
  --out data/swebench-funnel-baseline.json

# 2. pair the governed run against it (same dataset, same --default-model, and no
#    per-arm flag — e.g. --code-discipline — that the baseline run didn't also have)
npm run eval:swebench -- --arm governed --score official \
  --baseline-report data/swebench-funnel-baseline.json \
  --out data/swebench-governed-report.json

Every governed instance record carries, never inferred: whether review ran and its verdict (review.ran/decision/unmet_requirements/skip_reason, plus rereview_error when a completed review drove a fix round but the re-review of the overwritten with ran: false, and validation rejects a record claiming ran: false fix rounds ran and whether the patch changed (fix_rounds/patch_changed_by_fix), and whether the fixer could execute the project's tests (tests_executable + tests_executable_reason — spec 276 §5a: no image today carries both the claude CLI and the SWE-bench test environment). Since spec 490 inc-2 tests_executable is derived from the run-events ledger — true only when every round's test_gate_ran event shows the suite executed in the instance's own evaluation image — and its reason names the first failing round (no_test_gate_ran_event, not_executed, ran_in_agent_runtime, image_mismatch, or on SWE-bench Pro harness_command_failed when the round exited with the harness wrapper's own code for "no repository root" or "overlay failed", so the suite never ran). Each governed instance also reports claim_mode: "leaderboard", test_command_declared and suite_rounds (per round: executed, image, exit_code, timed_out, and suite_feedback_withheld when the suite output named a fail_to_pass id and was kept from the fixer). A multi-instance governed run is refused unless --fix-path-evidence <report.json> names a prior governed report showing review, a fix round and tests_executable: true on one instance; the new report records it as fix_path_evidence. Each governed instance also reports repro ({base, base_exit_code, command_source, files, confirmed_by} — what the agent's repro file(s) showed at base, run on their own via the table's repro_command: reproduced for pytest exit 1, not_reproduced for 0, unjudgeable for 2-5, no_command when none was captured; plus confirmed_by: "benchmark" when the repro gate parked and the arm confirmed the symptom, else null) and repro_fixed (true/false/null, from the SCORED round's own repro review, with repro_fixed_reason and repro_fixed_round; null means unknown, never a miss) — see docs/OPERATIONS.md §10h and §10j. Each governed instance also reports suite_base (the repository's own suite run once at the base commit in the instance image, --network none) and regressions_at_final ({count, tests}: tests that passed at base and fail in the SCORED patch, or null with regressions_at_final_reason when it could not be judged); a round with a regression gets a fix round even on APPROVE — see docs/OPERATIONS.md §10i. A report missing review, fix_rounds/patch_changed_by_fix, tests_executable(_reason), claim_mode, suite_rounds, repro, repro_fixed, suite_base or regressions_at_final on any governed instance fails validation and is neither printed nor written — see docs/OPERATIONS.md §10e for the full mechanism (how the bounded fix loop is orchestrated without a new Weftra route, the resolved arm-config fields, and where each surface renders them).

FA_EVAL_BASE_URL=http://localhost:3100 \
FA_EVAL_ADMIN_KEY=your-admin-key \
npm run eval:swebench -- --arm governed --score official \
  --baseline-report data/swebench-funnel-baseline.json \
  --out data/swebench-governed-report.json

(Omitting --baseline-report still runs the full governed pipeline and produces a valid report — every §4 per-instance field is still required and validated — it just leaves governed_funnel_paired_delta null, since there is nothing to pair against yet. That null is the honest state: prefer it to a delta the comparability checks above would have refused.)

Every report — JSON and CLI text summary — also carries per-arm LOC (exact, always present once an arm has a captured patch) and complexity (best-effort, labeled [EXPERIMENTAL], populated only when --score official runs). See OPERATIONS.md §10a for the full breakdown of both fields.

Omitting --score official still runs the Weftra arms and captures patches, but the report's resolved fields are placeholders (scored_by says so explicitly) — run the score step below (or re-run with --score official) before publishing a number.

Concurrency. By default the runner is serial — one (instance, arm) run at a time, same as before this flag existed. --concurrency N (N > 1) bounds up to N (instance, arm) runs in flight at once, so wall-clock is governed by the pool size instead of the sum of every run — this is what makes a larger N (e.g. a governed A/B) affordable to run at all. The produced report is identical either way: instance order always matches the input dataset order (results are assembled by index, not completion order), and each arm's config (the read-back-from-the-project stamp, spec 168) is still captured at most once per arm even when several of that arm's runs start together. When --concurrency is omitted, the CLI falls back to AGENT_MAX_CONCURRENCY (the same env var the agent engine itself reads), then 1.

Arm-distinct predictions. --arm both scores bare and funnel as two separate harness invocations against the same --score-workdir. Each arm writes its own predictions file — predictions-bare.jsonl / predictions-funnel.jsonl — with its own model_name_or_path (<engine>-bare / <engine>-funnel), so both arms' patches for the same instance_id land in separate files instead of one arm's predictions overwriting the other's. This only applies to the --arm both path; the standalone score subcommand below always writes a single predictions.jsonl (it has no arm to distinguish).

Scoring can fail loudly — never a fake 0%. If the harness itself crashes or never writes a readable report for an arm (e.g. a transient PyPI/Hugging Face network blip), that arm's report carries scoring_failed: true and its instances carry the scoring_infra_failure reason instead of a real resolved/unresolved verdict. The text report prints *** SCORING FAILED (infra) — NOT a 0% *** for that arm (and withholds the A/B lift headline, since it can't be computed from a real number), and the CLI exits non-zero. Treat a non-zero exit from --score official as "scoring did not complete", not "the arm scored 0%" — re-run scoring once the transient issue clears.

Environment variables:

Variable	Description	Default
FA_EVAL_BASE_URL	Weftra server base URL	required
FA_EVAL_ADMIN_KEY	Weftra admin API key	required
FA_EVAL_ENGINE	Engine/model label	sonnet/claude-sonnet-4-6
FA_EVAL_POLL_INTERVAL_MS	Poll interval	30000
FA_EVAL_TIMEOUT_MS	Per-instance-arm timeout	3600000
FA_SWEBENCH_REPORT_PATH	Path to report file served by Weftra	data/swebench-provisional-report.json

Command — score an existing predictions.jsonl (no Weftra server needed)

If you already have patches captured (a prior Weftra run, or predictions from anywhere else) in the standard SWE-bench predictions.jsonl format ({instance_id, model_name_or_path, model_patch} per line), score them directly:

npm run eval:swebench -- score \
  --predictions data/predictions.jsonl \
  --out data/swebench-score-report.json

Flag	Description
--predictions <path>	predictions.jsonl to score (required)
--out <path>	Write JSON verdict report to file (default: stdout)
--run-id <id>	Harness run id (default: timestamp)
--dataset-name <name>	Dataset name passed to the harness (default: princeton-nlp/SWE-bench_Lite)
--workdir <dir>	Host dir bind-mounted at /work (default: predictions.jsonl's directory)
--max-workers <N>	Harness parallelism (default: 4)

This still requires the local-Docker prerequisite above — no Weftra server or admin key. Unlike the --arm both path above, this subcommand scores one arm-less predictions file, so it keeps writing a plain predictions.jsonl under --workdir with whatever model_name_or_path the input file already names — there's no arm to make distinct. It carries the same loud-failure behavior: a harness crash / unreadable report sets scoring_failed: true on the JSON report, prints the SCORING FAILED (infra) banner to stderr instead of a resolved percentage, and exits non-zero.

Publishing the result

After a run with --score official (or a score run merged back in) writes data/swebench-provisional-report.json:

1. Dashboard: Available immediately at /eval-report.html (admin-gated, SWE-bench tab) via GET /api/eval/swebench-report, which now also shows scored_by.
2. Docs site: Copy the report to data/swebench-provisional-report.json, then rebuild:

   npm run docs:build

   The build script picks up the JSON and publishes it to docs-site/public/.

Before publishing any number, check scored_by in the report JSON. If it isn't swebench-official@<version>, it's an unscored placeholder — score it first.

---

SWE-bench Pro (Python, Go, JavaScript and TypeScript tasks)

--benchmark pro runs the same arms on SWE-bench Pro instead of Lite. The default is --benchmark lite, so existing commands and reports are unchanged.

# a reproducible random sample of 10 Python tasks, funnel arm, graded by each task's own verifier
npm run eval:swebench -- --arm funnel --benchmark pro --language python \
  --sample 10 --seed 42 --score official --out data/pro-funnel-seed42.json

# one Go task end to end
npm run eval:swebench -- --arm bare --benchmark pro --language go --instance <instance_id> \
  --score official --out data/pro-bare-go-one.json

• Language. --language python|go|js|ts selects which tasks run (default python). The sample, --instance, --limit and --seed all work inside the chosen language. An unknown language, or an --instance that belongs to another language, is refused with the languages the snapshot holds. The report's language is the selected language and every instance row carries repo_language.

• The agent works in the task's own image. For every Pro task, in every language including Python, the arm's project is created with the task's docker_image as its runtime base image. Weftra builds a derived image on top of it (the task image plus Weftra's agent layer) and the agent works there, so the Go, Node or Python toolchain the task needs is present. The base image is added to the operator's runtime-image allowlist and set on the project with the admin key before the project is created; the per-project key never sends it. A refused image stops that instance with runtime_image_not_allowlisted; there is no fallback to the default runtime. The arm result records runtime_base_image, and, read back from the run's log, runtime_base_digest, derived_image and derived_image_id (null when the log line could not be read; the run itself is unaffected). Lite results carry none of these fields.

• The task's repository root is writable by the sandbox user. A Pro image owns its repository (/app, else /testbed) as root, so the governed arm's overlay step and run_script.sh, which run as the non-root sandbox user, could not write there and exited 98. Every Pro arm project therefore declares that root as its single runtime_writable_paths entry (see docs/RUNTIME_IMAGES.md), so Weftra runs the agent and the test container from a copy of the task image with that directory owned by the sandbox user (no RUN, no network). The root is found by the same /app-then-/testbed lookup the scorer uses, by running a fixed script in the task image on the operator's Docker before the project is created, and is set with the admin key; the per-project key never sends it. An image with neither directory stops that instance with pro_repo_root_not_found, and one that cannot be read with pro_repo_root_unreadable; there is no empty list. Lite projects declare none and are unchanged. Operator-pending (spec 517 AC1/AC6/AC7): not yet validated on a host with Docker. Still to record: the governed test command on a Go task (flipt) and a JavaScript task (NodeBB), each exiting with the tests' own status rather than 98; tests_executable: yes with a suite at base and at final; and claude --print returning PONG under both api and oauth auth in the writable image. Run tests/spec-517-writable-image-live.test.ts on that host first, since it checks the re-owning mechanism itself.

• No test command on bare and funnel. Pro arm projects declare an empty test command on the bare and funnel arms; the task's own verifier decides resolution.

• The governed arm on Pro. --arm governed --benchmark pro works for every language (--arm both still means bare plus funnel). The governed project's test command is the task's own run_script.sh, run with no arguments, fetched at the verifier commit the report records. Nothing else from the verifier directory (the hidden test patch, test.sh, config.json, the fail-to-pass and pass-to-pass lists) is fetched for the project or sent to it. The script travels in the create request encoded and is decoded inside the container; the shell text around it is fixed. A Pro image keeps its repository and dependencies at /app (else /testbed), so each command first copies the agent's workspace onto that root (modified, added and deleted files; never .git, never ignored files) and runs there; the workspace itself is only read. If the script cannot be fetched, or the task id or image is unsafe, the run stops with no_pro_test_command before any project is created, and nothing falls back to a pytest call. The test gate stays informational (warn), as on Lite.

  • Repro. Python tasks run python -m pytest -rA over the repro files, and Go tasks run go test over the distinct package directories of the repro's .go files. Both report exit 1 for a repro that ran and failed, 0 for one that ran and passed, and 2 or more for one that could not run (no repro files, a Go package that did not build or has no tests, no repository root in the image); never 1 for something that could not be judged.
  • JavaScript and TypeScript run the repro through the repository's own runner. The command looks for vitest, then jest, then mocha in node_modules/.bin and runs the repro files with it; it reports exit 1 for a repro that ran and failed, 0 for one that ran and passed, and 3 when no runner is installed or the output shows no test ran (a per-test summary with a non-zero passed or failed count; a Test Suites or Test Files line alone does not count). A repository with no discoverable runner reports cannot_verify, not a guess. A language with no repro runner at all still creates the project with the repro gate off and reports repro_fixed: null with the reason no_repro_runner_for_language:<language>. Either way repro_cannot_verify is true on the arm result, and the conformance reviewer is told the fix is undemonstrated.
  • Pairing a governed run with a funnel baseline. --baseline-report on Pro also requires the baseline's benchmark, dataset revision, verifier commit, language and sample (size and seed) to equal this run's; each one that differs is refused, naming both values, and no delta is published. Lite pairings are checked exactly as before.
  • Not yet probed against a real image. The overlay and the commands are contract-tested locally with stub runners and temporary directories; a run against a real task image per language has not been done on a host with Docker, and should be before a Pro governed number is published.

• Tasks. A committed snapshot of ScaleAI/SWE-bench_Pro tasks: 30 Python, 8 Go, 8 JavaScript and 4 TypeScript tasks. Each language is a seeded sample (the seed, the pool size it was drawn from and the dataset revision are recorded per language in the snapshot; the report records the revision), generated by scripts/fetch-swebench-pro-instances.ts (never hand-edited), which regenerates any sample deterministically. The snapshot carries only what the agent is shown and the task image — no reference patch and no test patch; both come from the task's directory in scaleapi/SWE-bench_Pro-os at the verifier commit. --sample <n> --seed <s> draws a reproducible sample (the same pair always draws the same tasks; the seed is recorded in the report; --seed alone is refused). --instance <id> and --limit <n> still work.

• What the agent sees. The problem statement, the requirements and the new interfaces. Never the reference patch, the hidden tests or the fail-to-pass and pass-to-pass lists.

• Grading. --score official runs each task's own verifier (its test.sh, run_script.sh and parser.py, fetched from scaleapi/SWE-bench_Pro-os at the verifier commit the report records) on a fresh container of the task's docker_image, with local Docker and no Modal. The diff is applied to the image's untouched repository; the container gets no host mount, no Docker socket and no environment variables. Runs are serialised per host with the same lock the Lite harness uses. Like the Lite harness, this is operator-only tooling that needs a local Docker daemon, and it executes model-written code in a container with Docker's default network (the verifier may install requirements).

• Outcomes. resolved, tests_failed (including a verifier time cap), patch_failed (the diff does not apply), empty_patch, or scoring_infra_failure (no verdict could be obtained: verifier files unreachable, the image would not run, the verifier ended without a verdict). A scoring failure is never reported as unresolved: the arm carries the same SCORING FAILED banner as Lite and the CLI exits non-zero.

• Report. Names the benchmark, dataset revision, verifier commit, language, and the sample size and seed (or "all matching instances").

• Checking the scorer itself. Before trusting a run, grade one task's reference patch (fetched from the task's solution/gold_patch.diff at the verifier commit) and its untouched repository:

  npm run eval:swebench -- score --benchmark pro --reference <instance_id>   # should resolve
  npm run eval:swebench -- score --benchmark pro --nop <instance_id>         # should not
  npm run eval:swebench -- score --benchmark pro --predictions <path>        # grade a predictions.jsonl

  Then run one funnel task end to end:

  npm run eval:swebench -- --arm funnel --benchmark pro --instance <instance_id> \
    --score official --out data/pro-funnel-one.json

• The verdict contract, and how it is pinned. The scorer does not decide pass or fail itself. It reads two things from the task's own test.sh: an exact RESULT: PASSED or RESULT: FAILED line, and the script's exit code (0 on pass). It applies the diff in the repository root that test.sh itself looks up (/app, else /testbed). A real task's test.sh and config.json, fetched byte-for-byte at the verifier commit, are committed under tests/fixtures/swebench-pro-verifier/ (their hashes are in its PROVENANCE.md). tests/swebench-pro-verifier-contract.test.ts runs that script's own verdict block on a pass result and a fail result, and feeds what it prints through the scorer's detection lines. If Pro changes what its verifier prints, that test fails; Pro runs do not silently turn into scoring failures.

• Status of the end-to-end check. The three commands above (reference resolves, untouched repository does not, one funnel task gives a scored report) need a local Docker daemon. They have not been run yet: the environment that built this change has no Docker. Until an operator runs them and records each report's scored_by, verifier_commit and resolved/reason here, treat Pro numbers as unvalidated.

• Not in this version:

  • The repository path is looked up, not assumed. Pro's dataset rows have no repository-path field. The agent clones the task repository at base_commit into its own workspace, and the scorer uses the same /app-then-/testbed lookup as Pro's own test.sh, so the diff always lands in the tree the verifier grades. An image with neither path is reported as a scoring failure, never as unresolved. The same grader is used for every language; one the verifier cannot grade is a scoring failure too.

Paired multi-seed protocol (SWE-bench Pro)

One funnel-versus-governed comparison on one sample is a single draw. The paired protocol repeats it over several seeds and reports the spread, so a lift is not read off one lucky sample.

npm run eval:swebench -- --arm paired --benchmark pro --score official \
  --language python,go,js --sample 2 --seeds 1,2 \
  --fix-path-evidence data/governed-fix-path-evidence.json \
  --out-dir data/pro-paired-1-2

• What it requires. --benchmark pro, --score official, at least two distinct --seeds (integers from 0 to 4294967295), --sample and --out-dir. --fix-path-evidence is required whenever a seed's selection holds more than one instance, exactly as for a single governed run. --seed, --instance, --limit, --baseline-report and --out are refused (exit 2) before any request is made, and so is an --out-dir that already holds protocol output, so a stale file can never pass for this run's.
• Selection. --language takes a comma-separated list or all. With --sample N --seed S it draws N tasks per language, each draw exactly the single-language draw for that seed, joined in the order the languages were given (all is sorted: go, js, python, ts). One language behaves as it always has. The report adds languages only when several are selected. Every seed draws its own selection; within a seed the funnel and governed arms run over the identical instance set.
• What runs. For each seed S, in order: the funnel arm, written to <dir>/seed-<S>/funnel.json, then the governed arm, run with --baseline-report set to that file, written to <dir>/seed-<S>/paired.json. Each arm is an ordinary run of this CLI with its own run id (<run-id>-s<S>-funnel, <run-id>-s<S>-governed). --engine, --default-model, --funnel-model, --concurrency, --suite-timeout-ms, --force-clarify and --code-discipline go to both arms. --reviewer-model, --fixer-model, --fix-round-cap and --fix-path-evidence go to the governed arm only. Each seed's official-scorer work directory is <dir>/seed-<S> unless --score-workdir is given.
• The summary. <dir>/summary.json and <dir>/summary.md are computed only from the per-seed paired.json files read back from disk, never from anything held in memory by the arms.
  • Provenance: benchmark, dataset revision, verifier commit, FA commit, default model, subset, languages, sample size per language, seeds and each seed's report paths. Seeds whose reports disagree on any of these are refused rather than averaged.
  • Per seed: n, funnel_resolved, governed_resolved, delta (copied from the report's governed_funnel_paired_delta, the governed resolve rate minus the funnel's over the paired instances), and the cost of each arm. A cost is null, shown as n/a, when any instance's cost is unknown; it is never a partial sum.
  • Across seeds: the mean, minimum, maximum and sample standard deviation (n-1) of the per-seed deltas. summary.md shows deltas in percentage points and prints no figure that is not in summary.json.
  • By language, per seed and pooled over the seeds: n, funnel and governed resolved counts and the delta, from each instance's repo_language.
  • Per seed, the ids of instances left out of the pairing.
• --subset hard. --subset all|hard (Pro only; refused with Lite). hard restricts each language's pool to the rows marked hard BEFORE sampling. It draws from the hard rows in the committed snapshot (5 today: python 4, ts 1), not from the 51 hard tasks in SWE-bench Pro. A language with no hard rows in the snapshot (go and js today) is refused by name, so --language all --subset hard is refused. The report records the subset and each language's hard pool size.
• Failure semantics. If any arm of any seed fails, the protocol stops there: no later arm or seed starts, no summary is written, and the CLI exits non-zero naming the seed and the arm. Files already written for earlier seeds, and for the failed seed's earlier arm, stay on disk. A scorer infrastructure failure on any instance fails its arm, as in a single run: it is never counted as unresolved.
• Not validated live. This protocol has been exercised offline only. No paired run against a real FA instance and Docker daemon has been done, so treat the first live summary as unvalidated.

Running on a declared engine

--engine is only a label stamped on the report. To measure an engine declared in FA_ENGINE_PROFILES (for example an OpenAI-compatible model such as Flint, see docs/ENGINES.md §4e), pass --engine-profile <id> (env FA_EVAL_ENGINE_PROFILE). When --engine is not given, the label defaults to the profile id.

• What it sets. Every arm's project is created with engine: <id>, which is the engine the author turn and the governed fix round resolve. Nothing else changes: the analyze and answerer models (--funnel-model), the reviewer (--reviewer-model), the security reviewer and every other role keep their defaults. Without the flag the create body is unchanged.

• Refusal. Before any project is created the CLI reads GET /api/admin/engines with the admin key. An id that is not listed exits non-zero naming the id and the listed ids; no project is created, no feature is submitted and no report is written. The flag selects only among host-declared profiles and never carries a command, URL or key.

• Report fields. The report, each paired funnel.json / paired.json, and summary.json / summary.md gain engine_profile: { id, model }. model is read back from the engine list (null when the profile declares none), never copied from the command line. Each arm's config also shows the stored engine. Seeds measured on different profiles refuse to combine.

• Honest cost. An arm with any instance whose cost is unknown gains cost_unmeasured: N; total_cost_usd stays the sum of the known costs. The markdown shows ≥ $X (N of M unmeasured), or unknown when every cost is unknown. With all costs known the output is unchanged.

• Paired Pro run (operator-run). The same flag is forwarded to both arms:

  npx tsx src/services/eval/swebench/cli.ts --arm paired --benchmark pro --score official \
    --seeds 1,2,3 --sample 15 --language python,go,js --out-dir ./out/flint-pro \
    --engine-profile <id> --fix-path-evidence <path>

  This has not been run live; see specs/520-openai-compatible-engine/verification-inc-3.md.

N=15 instance subset

The 15 instances are real rows from princeton-nlp/SWE-bench_Lite (test split), restricted to pytest-runner repositories (requests, flask, pytest, pylint, xarray) so the derived test_command is a correct pytest invocation. Each runtime_image is the official eval image, whose Docker tag encodes the instance_id's __ as _1776_.

Instance ID	Repository	Docker image
psf__requests-1963	psf/requests	sweb.eval.x86_64.psf_1776_requests-1963
psf__requests-2148	psf/requests	sweb.eval.x86_64.psf_1776_requests-2148
psf__requests-2317	psf/requests	sweb.eval.x86_64.psf_1776_requests-2317
pallets__flask-4045	pallets/flask	sweb.eval.x86_64.pallets_1776_flask-4045
pallets__flask-4992	pallets/flask	sweb.eval.x86_64.pallets_1776_flask-4992
pallets__flask-5063	pallets/flask	sweb.eval.x86_64.pallets_1776_flask-5063
pytest-dev__pytest-11143	pytest-dev/pytest	sweb.eval.x86_64.pytest-dev_1776_pytest-11143
pytest-dev__pytest-11148	pytest-dev/pytest	sweb.eval.x86_64.pytest-dev_1776_pytest-11148
pytest-dev__pytest-5103	pytest-dev/pytest	sweb.eval.x86_64.pytest-dev_1776_pytest-5103
pylint-dev__pylint-5859	pylint-dev/pylint	sweb.eval.x86_64.pylint-dev_1776_pylint-5859
pylint-dev__pylint-6506	pylint-dev/pylint	sweb.eval.x86_64.pylint-dev_1776_pylint-6506
pylint-dev__pylint-7080	pylint-dev/pylint	sweb.eval.x86_64.pylint-dev_1776_pylint-7080
pydata__xarray-4248	pydata/xarray	sweb.eval.x86_64.pydata_1776_xarray-4248
pydata__xarray-5131	pydata/xarray	sweb.eval.x86_64.pydata_1776_xarray-5131
pydata__xarray-3364	pydata/xarray	sweb.eval.x86_64.pydata_1776_xarray-3364

The raw records (verbatim dataset fields) are in src/services/eval/swebench/instances.dataset.ts, generated by scripts/fetch-swebench-instances.ts — never hand-authored. instances.ts derives repo_url, runtime_image, and test_command from them.

---

Resolution criterion

An instance is resolved when the official harness's resolved_ids for that run contains its instance_id — i.e. the candidate patch applies AND every FAIL_TO_PASS

• PASS_TO_PASS test the harness itself runs, in the harness's own per-instance eval image, passes. Weftra never re-implements this check:

1. src/services/eval/swebench/predictions.ts builds a standard predictions.jsonl from Weftra's captured patches (empty patches are excluded up front and reported empty_patch — no reason to spend a harness run on an instance with no patch).
2. src/services/eval/swebench/harness.ts invokes python -m swebench.harness.run_evaluation inside a python:3.11 container.
3. src/services/eval/swebench/report-parser.ts maps the harness's own resolved_ids / unresolved_ids / error_ids back onto Weftra's per-instance report. An instance_id missing from all three of the harness's sets — e.g. its sibling eval container crashed or timed out — is surfaced as error, never silently counted as a test failure.

error vs. scoring_infra_failure. error (above) means the harness ran to completion and wrote a real report that itself names this instance as errored — a harness-attested verdict. scoring_infra_failure is different and more severe: the harness invocation never produced a readable report at all (crashed before finishing, or its report file is missing/corrupt) — there is no verdict for any instance in that arm, so src/services/eval/swebench/official-scorer.ts marks the whole arm's SweBenchArmScore.scoring_failed and refuses to let report-format.ts render its resolved_rate as a percentage. See "Loud scoring-infra failure" above.

---

Interpreting results

• The lift is the primary Weftra claim. A positive lift (funnel better than bare) validates that the clarify→spec pre-processing step adds resolution value, not just PR volume.
• The raw % is a familiar anchor — compare with caution since this is N=15, not 300.
• The wasted-run count shows how many instances each arm spent credits on without resolving.
• Per-instance details in the report show exactly which instances resolved and at what cost.
• Always check scored_by before quoting a number — see "Scoring" above.

---

This is a provisional run. The full N=300 evaluation (inc-2) is planned after this design is validated.