How ShallowSWE scores routine software work

What ShallowSWE measures

Most software-engineering benchmarks ask whether agents can solve difficult tasks. ShallowSWE targets the opposite regime: original, routine, functionally verifiable work that a frontier agent should usually complete — regression fixes, compatibility changes, report transformations, configuration repairs, branch operations, bounded tool workflows. In this regime many configurations eventually succeed, but they differ sharply in premature completion, repair cycles, context appetite, and spend.

The central question is:

For a declared workload basket, how much reference-budget cost is charged per verified completion when one fixed agent policy handles every ticket?

Each scored row binds one immutable model configuration and one fixed agent policy — model, provider route, sampling, scaffold, prompt, tools, and continuation behavior stay fixed for the whole trajectory. The result is properly read as model configuration × agent policy × provider path × snapshot. It is not a universal intelligence ranking, a comparison of vendors' native coding products, or a task-value estimate.

full treatment: paper §2 — scope and design principles

One bounded repair loop

A row starts in a clean, isolated sandbox. The agent works normally and declares completion; only then does a hidden programmatic verifier run. If verification fails and the undisclosed allowance remains, the same agent continues in the same conversation and filesystem state with only coarse feedback — never the failing check itself:

The requested behavior is not yet satisfied. Continue working.
The run failed because the required artifact is missing.
The submitted result could not be executed.
The submitted output does not match the required contract.

The contract is deliberately naturalistic. The agent is never told:

• that it is in a benchmark, or that verification is hidden;
• how many submissions, steps, or dollars remain;
• whether a submission is final.

The loop ends at verified success or a scored terminal condition: exhausting the undisclosed verifier -submission cap, agent-step guard, or dollar budget; context exhaustion after meaningful progress; or giving up. Provider outages, network errors, and harness failures are excluded and retried, not scored. No fallback, model switching, ensembling, or fresh retry happens inside a scored row.

full treatment: paper §7 — bounded repair-loop protocol

Three views of one metric

The headline metric is reference-budget cost per successful completion, CPSC^B. Successful rows are charged what they actually spent. Failed rows are charged the task's frozen reference budget B_t — a task-only failure price, independent of how the row happened to stop. For solve probability p and mean successful spend μ^S:

A worked example with B_t = $0.20: configuration A succeeds 90% of the time at $0.04 per success, so it scores $0.04 + (0.10 / 0.90) · $0.20 ≈ $0.062. Configuration B always succeeds but costs $0.08. A is cheaper under the reference-budget policy — though a separate reliability floor may still make it ineligible for recommendation.

Two companion views are always published beside the headline, so a recommendation can be checked against the failure convention it depends on:

Aggregation is a weighted ratio — weighted spend over weighted successes — never an average of per-task CPSC values. Cells with zero successes are shown as no verified successes, not dropped. A provisional 90% observed solve-rate floor gates recommendations.

full treatment: paper §8 — metric family

Category, pressure, budget, scope

ShallowSWE separates four properties that are usually collapsed into “task size”:

• Category — what kind of work: code (change software behavior), artifact (transform inputs into checked outputs), workflow (operate on repository, tool, or system state).
• Floor pressure — measured, not manufactured: how often a frozen economical reference configuration solves the task on the first submission. Low ≈ 70–100%, medium ≈ 30–70%, high ≈ 0–40%.
• Reference budget B_t — a calibrated dollar allowance per task, and the headline failure charge.
• Structural scope — how much state must be traversed: files, context, artifacts, steps. Metadata, not pressure.

A task can be broad but easy, or narrow but punishing — the benchmark does not force these axes to correlate. The suite is organized by category × pressure; budget and scope stay continuous. Coverage is a target, not a quota: if credible routine tasks cannot fill a cell, the cell is reported undercovered rather than filled with manufactured pressure.

full treatment: paper §4 — benchmark overview

Calibration in four phases

Every constant a score depends on is frozen in order, before the runs it governs. Construct review comes before any model calibration; caps freeze before budgets; fresh runs confirm the final policy.

1. Permissive collection. Frozen anchor and floor panels run under generous undisclosed limits, recording spend, submissions, steps, and first success.
2. Freeze supervision and liveness. The hidden verifier-submission cap K is the smallest value after which extra feedback adds negligible legitimate success; the step cap is a pooled liveness guard that should almost never bind.
3. Select the reference budget. B_t is the smallest pre-registered dollar band in which the anchor reaches the coverage target, checked on a held-out split with at most one adjacent band increase.
4. Fresh confirmation. New anchor replicates run under the exact frozen policy — no budget increases allowed — and estimate the replacement cost R_t: total anchor spend divided by anchor successes.

full treatment: paper §6 — calibration

Task quality and admission

Tasks are written from scratch — no adaptations of public issues, patches, or benchmark instances. Before any model calibration, every candidate passes a routine-work construct gate: at least one qualified reviewer who is not the task author rates realism, frequency, delegability, ambiguity, and expected effort against a predeclared rubric, and the review is published.

Each task must then execute — not merely declare — its quality evidence:

• prompt–verifier consistency checks;
• a reference solution and a materially different alternate solution that both pass;
• no-op and negative controls that fail;
• feedback-leakage review and clean-sandbox reproducibility.

Verifiers are deterministic and implementation-flexible: every hidden assertion maps to a visible requirement or public contract, so materially different valid implementations pass while hardcoded fixtures and partial solutions fail. Admission requires the frozen frontier anchor to succeed in one-shot mode (provisionally 12/16). Every snapshot publishes the full candidate funnel — authored, rejected at each gate, accepted — with reasons.

full treatment: paper §5 — task sampling and quality assurance

Claim tiers

Releases come in two classes, and the tier is part of every artifact:

• Protocol-validation pilot. Six construct-reviewed tasks exercise all four calibration phases at small scope. It proposes constants and demonstrates machinery; it supports no global leaderboard claim. This is the current stage.
• Report-grade snapshot. A calibrated suite and frozen panel with fresh anchor confirmation, all three metric views, provider reconciliation, eligibility extensions, and pre-registered rank-stability analysis.

full treatment: paper §10.4 — claim tiers and rollout counts

What this site shows today

full treatment: paper §1.3 — implementation boundary

Limitations

• Within-suite claims only. The rank belongs to a declared, curated, currently Python-heavy basket — not a probability sample of software engineering.
• Anchor and convention dependence. B_t and R_t depend on the chosen anchor, provider route, and price sheet; sensitivity to them is reported as part of the result.
• Supervision shapes behavior. Even coarse hidden rejection is supervision, and the submission cap shapes the repair process.
• Small N. Initial rollout counts support protocol validation and within-suite comparisons; near-threshold solve rates and budget tails stay noisy.
• Tasks age. Public tasks lose value for future rankings and require new versions or held-out extensions.

full treatment: paper §13 — limitations