π OpenHands: AI-Driven Development
feat(verify-openhands): executable known bugs and a repeatable fix proof (#18348)
Engel Nystd7e14e25bcf1
diff --git a/.agents/skills/verify-openhands/SKILL.md b/.agents/skills/verify-openhands/SKILL.mdindex 52864f2177..fd9fca07fa 100644--- a/.agents/skills/verify-openhands/SKILL.md+++ b/.agents/skills/verify-openhands/SKILL.md@@ -26,7 +26,7 @@ together:lists every user-facing behavior with stable IDs, user entry points, exact`control-openhands` recipes, observable results and gotchas.3. **Evidence**: screenshots, ARIA snapshots and a pass/fail/blocked/not-run- ledger that survive cleanup.+ ledger (plus xfail/xpass for known bugs) that survive cleanup.Neither this skill nor the map authorizes external writes, paid models beyondthe budget you were given, or product fixes the user did not ask for.@@ -188,6 +188,19 @@ prerequisite; never substitute a mock and call it a pass.`src/` paths no family owns (a map gap) and non-user-facing paths. Drive every entry point those features list atdesktop and phone viewports, and report with[the report contract](references/report.md).+- **Verify a fix for a known bug** (an open PR or a merged commit that claims+ to fix a bug the map records): launch the baseline and the fix as two runs+ (`launch --new` from each worktree, same backend pins), then run the note's+ known-bug block twice on each with this checkout's CLI:+ `control-openhands bug run <ID> --attempts 2 --label baseline --run $BASE_RUN`+ must report `xfail` both times, and `--label fix --run $FIX_RUN` `xpass` both+ times. `control-openhands bug proof <ID> --baseline $BASE_RUN --baseline-label+ baseline --fix $FIX_RUN --fix-label fix` gives the verdict: `verified`,+ `not-fixed`, or `inconclusive`. A baseline that does not reproduce twice is+ inconclusive, never "fixed". When the note has no block yet, write one first+ ([mapping.md](references/mapping.md)) and prove it reproduces on the+ baseline. Every attempt runs the same recipe, viewport and fixtures, and+ screenshots or recordings stay part of the proof.- **Run the daily pass**: follow [references/daily.md](references/daily.md):static checks (`map check`, `map coverage`, `map testids`), the changedfamilies since the `Maintenance baseline` line of the map index@@ -217,7 +230,10 @@ prerequisite; never substitute a mock and call it a pass.in the map, which should know every open bug: a `Known failure (reproduced<date>, #<issue>)` note on the bullet that reproduces it, next to theexpected result, or a `Known issue #<issue>` Gotchas entry when no recipe can- drive it. Passes re-drive these notes once the issue closes.+ drive it. Make the note executable with a `known-bug` block where a recipe+ can drive it, so `control-openhands bug run <ID>` reports it as an expected+ failure (`xfail`) and flags it (`xpass`) the day it stops reproducing.+ Passes re-drive these notes once the issue closes.- **Blocked**: name the missing prerequisite (account, entitlement, OS, binary)and the route you attempted. Unreachable is never a pass.diff --git a/.agents/skills/verify-openhands/references/daily.md b/.agents/skills/verify-openhands/references/daily.mdindex 27d4962eea..f398ecf269 100644--- a/.agents/skills/verify-openhands/references/daily.md+++ b/.agents/skills/verify-openhands/references/daily.md@@ -72,7 +72,9 @@ cheaper than the next and catches a different kind of rot.results, as [maintenance.md](maintenance.md) step 2 describes. The deltapass has a second half: every bullet whose Gotchas or Known failure notelinks an issue that closed since `BASE` (check each linked issue's state)- is re-driven too, its row recorded and its note updated. A fix that landed+ is re-driven too, its row recorded and its note updated (a note with a+ `known-bug` block is re-driven with `control-openhands bug run <ID>`; an+ `xpass` goes to map review per [maintenance.md](maintenance.md) step 5). A fix that landedin the Agent Server or the automation service shows up only this way. So isevery open `bug` issue opened since `BASE` that names a mapped feature: itis reproduced and gets its note, so bugs filed outside a pass reach the map.diff --git a/.agents/skills/verify-openhands/references/feature-map/F25-backends-and-cloud.md b/.agents/skills/verify-openhands/references/feature-map/F25-backends-and-cloud.mdindex 7b6601588c..7ce98e0b69 100644--- a/.agents/skills/verify-openhands/references/feature-map/F25-backends-and-cloud.md+++ b/.agents/skills/verify-openhands/references/feature-map/F25-backends-and-cloud.md@@ -112,7 +112,21 @@ Preconditions:- The device-flow popup survives errors: every failed attempt leaves an `about:blank` tab (#17953). Cancel closes it; after an error close it with `browser close-tab`.- With the default host the device flow talks to the real `app.all-hands.dev` (`POST /oauth/device/authorize`, then token polling). Starting and cancelling is harmless; never approve a code.- While the second stack's agent server is stopped or restarting, its ingress answers without CORS headers, so the browser logs `Access to fetch at '<second-url>/api/settings' ... blocked by CORS policy` (also `/server_info`, `/api/conversations/search`, `/api/llm/models/verified`). That is the dead upstream, not a CORS bug; a healthy second stack sends `access-control-allow-origin` for this origin.-- Expected: Escape never clears the backend selector. Known failure (reproduced 2026-10-09, #18233): Escape on the focused backend selector while its menu is closed (a second Escape after the one that closes the menu, and sometimes the Escape that closes Manage Backends opened from it) clears the selection: the closed selector shows the active backend's name as an italic placeholder (`browser value 'testid=backend-selector >> role=combobox'` is `""`) and its status dot is gone, so `browser wait 'testid=backend-selector >> role=status[name="Disconnected"]'` times out although the backend is down; a reload restores it. Reload before a closed-selector status check, as the Edit bullet does. Repro: `browser click 'testid=backend-selector'`, `browser press Escape` twice (#18233; fix PR #18235).+- Expected: Escape never clears the backend selector. Known failure (reproduced 2026-10-09 and 2026-10-12, #18233): Escape on the focused backend selector while its menu is closed (a second Escape after the one that closes the menu, and sometimes the Escape that closes Manage Backends opened from it) clears the selection: the closed selector shows the active backend's name as an italic placeholder (`browser value 'testid=backend-selector >> role=combobox'` is `""`) and its status dot is gone, so `browser wait 'testid=backend-selector >> role=status[name="Disconnected"]'` times out although the backend is down; a reload restores it. Reload before a closed-selector status check, as the Edit bullet does. Repro: `browser click 'testid=backend-selector'`, `browser press Escape` twice (#18233). On 2026-10-12 `bug proof` verified fix PR #18238 at 77de0ad24 against main f07aebdf5 (two `xfail`, two `xpass`, desktop). `control-openhands bug run F25.selector-dropdown` replays it (desktop, after a reload with the mouse off the selector: a reload under a mouse left on it opens its menu and empties its field, which would look like the bug) and reports `xfail` while the bug lasts:++ ```sh known-bug F25.selector-dropdown escape-closed+ control-openhands browser viewport desktop+ control-openhands browser goto /+ control-openhands browser hover 'testid=root-layout'+ control-openhands browser reload+ control-openhands browser wait 'testid=backend-selector >> role=status[name="Connected"]' --timeout 15000+ control-openhands assert --field value --equals Local -- browser value 'testid=backend-selector >> role=combobox'+ control-openhands browser click 'testid=backend-selector'+ control-openhands browser press Escape+ control-openhands browser press Escape+ control-openhands browser screenshot 'testid=backend-selector' --feature F25.selector-dropdown --name escape-closed+ control-openhands assert --field value --equals Local --bug-shows '' -- browser value 'testid=backend-selector >> role=combobox' # bug+ ```- Local and remote backends are probed every 30 s (settings + server_info), Cloud backends every 5 min (#18128), so allow up to 40 s for a local dot to change. `browser network --clear`, wait about 70 s, then `browser network --filter server_info` shows two probes per local origin 30 s apart. While the active backend is down, any reload lands on the recovery gate: drive the disconnected rows on the page that was already loaded.- `/oauth/device/verify` is a Cloud page that also ships in Canvas: on a local stack its POST target does not exist, so Authorize always ends in `Error`. The "Authentication Required" state is unreachable (`useIsAuthed` always resolves `true`).- `/shared/conversations/<id>` on a local backend also raises a raw toast `HTTP request failed (404 Not Found): {"detail":"Not Found"}` next to `Conversation not found` (#17953).diff --git a/.agents/skills/verify-openhands/references/maintenance.md b/.agents/skills/verify-openhands/references/maintenance.mdindex b8ed321190..dad9611647 100644--- a/.agents/skills/verify-openhands/references/maintenance.md+++ b/.agents/skills/verify-openhands/references/maintenance.md@@ -65,7 +65,7 @@ issue acceptance criteria, relevant review discussion and file list.Record direct commits and unresolvable PRs as unknown intent. Keep intent(`documented`, `undocumented`, `contradictory`) independent of runtime-(`pass`, `fail`, `blocked`, `not-run`). A working change can still be+(`pass`, `fail`, `blocked`, `not-run`, and `xfail`/`xpass` for known bugs). A working change can still beundocumented. Never edit a PR description (the `HUMAN:` section is human-only)to justify a change after the fact.@@ -132,7 +132,23 @@ For an apparent regression, run the same minimal recipe on BASE and TARGET.Claim introduced-in-range only when BASE passes and TARGET fails undercomparable conditions (same browser, viewport, fixture state, backend pins).Otherwise classify as reproduced-on-both, environment/harness gap, intent gap,-or origin-unconfirmed. Use screenshots side by side; a pixel difference is not+or origin-unconfirmed. Once the regression is a known bug with a `known-bug`+block, the same rule is executable: `bug run <ID> --attempts 2` on each run and+`bug proof` (introduced-in-range is the proof read the other way: BASE `xpass`+twice, TARGET `xfail` twice).++To attribute a **fix** (a PR that closes a recorded bug, or an issue closed+since BASE), reproduce twice on the baseline and confirm twice on the fix,+with the same recipe, viewport, entry point and backend pins:+`control-openhands bug run <ID> --attempts 2 --label baseline --run "$BASE_RUN"`,+the same with `--label fix --run "$FIX_RUN"`, then+`control-openhands bug proof <ID> --baseline "$BASE_RUN" --baseline-label baseline --fix "$FIX_RUN" --fix-label fix`.+Each attempt records its revision, backend versions, viewport, recipe hash,+failing line and assertion in `evidence/known-bugs/attempts.jsonl`. Only+`verified` is a fix. A baseline that reproduces fewer than two times out of+two, or attempts under different inputs, are `inconclusive`: say so and do not+remove the note. A setup failure, a stopped backend, a timeout or another+symptom is a `fail`, never a reproduction. Use screenshots side by side; a pixel difference is notautomatically a bug and a zero difference does not prove behavior.## 5. Triage@@ -155,7 +171,13 @@ automatically a bug and a zero difference does not prove behavior.- **Undocumented or contradictory change**: cite the PR and the observeddifference; ask the maintainer instead of rewriting the expectation.- **Known issues** linked in Gotchas are repro candidates: retest, link fresh- results, never file duplicates or silently waive the check.+ results, never file duplicates or silently waive the check. Run every+ `known-bug` block (`control-openhands bug list`, then `bug run <ID>`): an+ `xfail` keeps the note, with its date refreshed. An `xpass` is a map review,+ not a pass: find what fixed it (a merged PR, a backend release), prove it+ with `bug proof` where a baseline is at hand, then drop the note and keep the+ block's commands as the bullet's normal steps. A `fail` is a broken recipe or+ a new symptom: triage it like any other failure.## 6. Report and hand offdiff --git a/.agents/skills/verify-openhands/references/mapping.md b/.agents/skills/verify-openhands/references/mapping.mdindex 421cb40c2e..aa566684c9 100644--- a/.agents/skills/verify-openhands/references/mapping.md+++ b/.agents/skills/verify-openhands/references/mapping.md@@ -124,6 +124,23 @@ sections, in this order:(reproduced <date>, #<issue>): <what happens>`; list it here as`Known issue #<issue>: ...` only when no recipe can drive it.+ Make the note executable when a recipe can drive it: indent a fenced block+ under it, opened with `` ```sh known-bug <ID> [name] `` (the name tells+ entry points or viewports apart, such as `desktop` and `phone`). Its+ commands set their own viewport and starting page and assert the *correct*+ behavior; exactly one line, the one that fails because of the bug, is a+ `control-openhands assert --field <path> --equals <correct> --bug-shows+ <what the bug shows> -- <command>` ending in `# bug`. Arrange steps go+ before it and must be able to fail on their own (assert the starting value+ too), so a broken setup is never read as the bug. Keep a `browser+ screenshot` before the assertion where the symptom is visual.+ `control-openhands bug run <ID>` then reports `xfail` while the bug lasts,+ `xpass` once it stops, and `fail` for anything else; `map check` rejects a+ block without exactly one such line. F25's `F25.selector-dropdown` note is+ the worked example. When a fix is proven, the PR that drops the note turns+ the block's commands into the bullet's normal steps, so the same assertion+ guards the corrected behavior.+Treat commands as literal: quote selectors, keep names unique to the run (prefixfixtures with `QA_`), and restore shared state after mutations (delete the secret,deactivate the profile) unless the next recipe depends on it, in which case say so.diff --git a/.agents/skills/verify-openhands/references/report.md b/.agents/skills/verify-openhands/references/report.mdindex 6a294388d7..41e390f008 100644--- a/.agents/skills/verify-openhands/references/report.md+++ b/.agents/skills/verify-openhands/references/report.md@@ -22,11 +22,30 @@ one passing example into a pass for its whole family. Record rows as you go with| Feature/check + entry point | Revision/backend/viewport | Action or command | Expected β actual | Result | Evidence ||---|---|---|---|---|---|-Use `pass`, `fail`, `blocked`, or `not-run`. For blocked rows name the missing+Use `pass`, `fail`, `blocked`, or `not-run`; `bug run` adds `xfail` (a known+bug reproduced at its marked assertion) and `xpass` (it stopped reproducing,+which needs map review). Keep both apart from `pass` and `fail` in the counts:+an `xfail` is neither a regression nor a pass, and an `xpass` is never folded+into a family pass. For blocked rows name the missingprerequisite, attempted action, and next step. Identify checks outside the declaredscope separately; do not count them as passed. Report denominator and counts forboth feature families and individual checks. A layout-only inspection must say so.+## Fix verification++For each fix claimed, list one row per known-bug block (ID and name, which+names the entry point or viewport):++| Feature/block | Baseline revision β attempts | Fix revision β attempts | Inputs (viewport, backend) | Verdict | Evidence |+|---|---|---|---|---|---|++Copy the attempt counts and verdict from `control-openhands bug proof`, and+link each attempt's transcript and screenshots. Only `verified` (two `xfail`+on the baseline, two `xpass` on the fix, same inputs) is reported as fixed;+`inconclusive` names its reason and `not-fixed` says the fix still+reproduces. A partial proof (one entry point verified, another not run) says+so per row.+## Change / intent ledger (weekly)| Commit / merged PR | Paths β feature IDs | Documented intent (quote/link) | Baseline β target observation | Intent status | Classification / issue |diff --git a/.agents/skills/verify-openhands/scripts/control-openhands.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.mjsindex 556d9f731d..0488b11f61 100755--- a/.agents/skills/verify-openhands/scripts/control-openhands.mjs+++ b/.agents/skills/verify-openhands/scripts/control-openhands.mjs@@ -34,6 +34,12 @@ import {} from "./lib/launcher-env.mjs";import { baselineLines, parseBaseline, withBaseline } from "./lib/baseline.mjs";import { affectedFamilies, familyHead } from "./lib/map-sources.mjs";+import {+ classifyAttempt,+ judgeAssertion,+ knownBugBlocks,+ proofVerdict,+} from "./lib/known-bug.mjs";import { routePattern } from "./lib/route-pattern.mjs";import {citedTestids,@@ -3102,6 +3108,10 @@ async function cmdBrowser({ positional, flags }) {// Commands: evidence ledger.// ---------------------------------------------------------------------------const RESULTS = ["pass", "fail", "blocked", "not-run"];+// `bug run` adds two results of its own: xfail (a known bug reproduced at+// its marked assertion) and xpass (a known bug no longer reproduces).+const BUG_RESULTS = ["xfail", "xpass"];+const REPORT_RESULTS = [...RESULTS, ...BUG_RESULTS];// The ledger is append-only: the report shows each check's latest row,// keyed by feature and entry point, minus retractions.@@ -3228,14 +3238,14 @@ async function cmdEvidence({ positional, flags }) {}if (sub === "report") {const latest = latestRows(rows);- const counts = Object.fromEntries(RESULTS.map((r) => [r, 0]));+ const counts = Object.fromEntries(REPORT_RESULTS.map((r) => [r, 0]));for (const row of latest.values()) counts[row.result] += 1;// Per family: the denominator the report contract asks for.const families = new Map();for (const row of latest.values()) {const id = row.feature.slice(0, 3);if (!families.has(id))- families.set(id, Object.fromEntries(RESULTS.map((r) => [r, 0])));+ families.set(id, Object.fromEntries(REPORT_RESULTS.map((r) => [r, 0])));families.get(id)[row.result] += 1;}// --baseline PATH: another run's ledger.jsonl (or run directory). Rows@@ -3280,7 +3290,7 @@ async function cmdEvidence({ positional, flags }) {"",`Revision \`${run.revision}\`, ${run.mode} mode, Agent Server ${run.versions?.agentServer ?? "?"}, base ${run.baseUrl}.`,"",- `Checks: ${latest.size} β ${RESULTS.map((r) => `${r} ${counts[r]}`).join(", ")}. Families: ${families.size}.`,+ `Checks: ${latest.size} β ${REPORT_RESULTS.map((r) => `${r} ${counts[r]}`).join(", ")}. Families: ${families.size}.`,"",];if (changes) {@@ -3309,14 +3319,14 @@ async function cmdEvidence({ positional, flags }) {lines.push("## Families","",- "| Family | pass | fail | blocked | not-run |",- "|---|---|---|---|---|",+ "| Family | pass | fail | blocked | not-run | xfail | xpass |",+ "|---|---|---|---|---|---|---|",);for (const [id, c] of [...families.entries()].sort(([a], [b]) =>a.localeCompare(b),))lines.push(- `| ${id} | ${c.pass} | ${c.fail} | ${c.blocked} | ${c["not-run"]} |`,+ `| ${id} | ${c.pass} | ${c.fail} | ${c.blocked} | ${c["not-run"]} | ${c.xfail} | ${c.xpass} |`,);lines.push("",@@ -3325,8 +3335,16 @@ async function cmdEvidence({ positional, flags }) {"| Feature/check | Entry point | Expected β actual | Result | Note | Evidence |","|---|---|---|---|---|---|",);- // Fail and blocked rows first (report.md), then by feature ID.- const order = { fail: 0, blocked: 1, "not-run": 2, pass: 3 };+ // Fail, xpass (a known bug stopped reproducing: review the map) and+ // blocked rows first (report.md), then by feature ID.+ const order = {+ fail: 0,+ xpass: 1,+ blocked: 2,+ "not-run": 3,+ xfail: 4,+ pass: 5,+ };for (const row of [...latest.values()].sort((a, b) =>order[a.result] - order[b.result] || a.feature.localeCompare(b.feature),@@ -3344,6 +3362,420 @@ async function cmdEvidence({ positional, flags }) {);}+// ---------------------------------------------------------------------------+// Commands: assert and known bugs (executable expected failures).+// ---------------------------------------------------------------------------+const cliPath = fileURLToPath(import.meta.url);++// Runs another control-openhands command in a child process, as a recipe+// would, and returns its exit code and parsed JSON output.+function runSelf(args, { env = process.env, timeout = 300_000 } = {}) {+ const child = spawnSync(process.execPath, [cliPath, ...args], {+ env,+ encoding: "utf8",+ timeout,+ maxBuffer: 64 * 1024 * 1024,+ });+ let json;+ try {+ json = JSON.parse(child.stdout);+ } catch {+ json = undefined;+ }+ return { code: child.status ?? 1, json, stdout: child.stdout ?? "" };+}++async function cmdAssert({ positional, flags }) {+ const hasEquals = flags.equals !== undefined && flags.equals !== true;+ const hasMatches = flags.matches !== undefined && flags.matches !== true;+ if (+ !flags.field ||+ flags.field === true ||+ hasEquals === hasMatches ||+ !positional.length+ )+ usage(+ "assert needs --field PATH, one of --equals VALUE or --matches REGEX, and -- <control-openhands command>",+ "control-openhands assert --field enabled --equals true --bug-shows false -- browser enabled 'testid=submit-button'",+ );+ const env = { ...process.env };+ if (flags.run && flags.run !== true) env.OH_VERIFY_RUN = resolve(flags.run);+ const child = runSelf(positional.map(String), { env });+ const judged = judgeAssertion({+ exitCode: child.code,+ json: child.json,+ field: flags.field,+ equals: hasEquals ? String(flags.equals) : undefined,+ matches: hasMatches ? String(flags.matches) : undefined,+ bugShows:+ flags["bug-shows"] === undefined || flags["bug-shows"] === true+ ? undefined+ : String(flags["bug-shows"]),+ });+ const result = {+ ok: judged.kind === "hold",+ ...judged,+ field: flags.field,+ expected: hasEquals ? String(flags.equals) : `/${flags.matches}/`,+ bugShows: flags["bug-shows"] === true ? undefined : flags["bug-shows"],+ command: positional.join(" "),+ output:+ judged.kind === "error"+ ? (child.json ?? child.stdout.slice(-2000))+ : undefined,+ };+ // Under `bug run` every assertion is logged, so the runner can tell the+ // marked assertion's mismatch from an error at the same line.+ if (process.env.OH_VERIFY_ASSERT_LOG)+ appendFileSync(+ process.env.OH_VERIFY_ASSERT_LOG,+ `${JSON.stringify({ ts: new Date().toISOString(), ...result, output: undefined })}\n`,+ );+ out(result);+ if (judged.kind === "error") process.exitCode = child.code || 1;+ else if (judged.kind !== "hold") process.exitCode = 1;+}++function knownBugsIn(file) {+ const files = file+ ? [resolve(String(file))]+ : featureFiles().map((f) => join(mapDir, f));+ return files.flatMap((path) =>+ knownBugBlocks(readFileSync(path, "utf8")).map((b) => ({+ ...b,+ file: relative(skillDir, path),+ })),+ );+}++function selectBlocks(id, flags) {+ const blocks = knownBugsIn(+ flags.file === true ? undefined : flags.file,+ ).filter(+ (b) =>+ b.id === id &&+ (!flags.name || flags.name === true || b.name === flags.name),+ );+ if (!blocks.length)+ throw new CliError(+ `No known-bug block for ${id}${flags.name && flags.name !== true ? ` named ${flags.name}` : ""}`,+ { code: 2, hint: "control-openhands bug list" },+ );+ const broken = blocks.filter((b) => b.problems.length);+ if (broken.length)+ throw new CliError(`Malformed known-bug block for ${id}`, {+ code: 2,+ extra: {+ problems: broken.map(+ (b) => `${b.file}:${b.line} ${b.problems.join("; ")}`,+ ),+ },+ });+ return blocks;+}++function runBlock(run, block, { timeoutMs, logFile, failureFile, scriptFile }) {+ const preamble = [+ "set -eo pipefail",+ `cd ${JSON.stringify(repoRoot)}`,+ `trap 'printf "%s\\t%s\\n" "$LINENO" "$BASH_COMMAND" > ${JSON.stringify(failureFile)}' ERR`,+ ];+ writeFileSync(scriptFile, `${preamble.join("\n")}\n${block.script}\n`);+ const env = {+ ...process.env,+ OH_VERIFY_RUN: run.dir,+ OH_VERIFY_ASSERT_LOG: logFile,+ PATH: `${here}:${dirname(process.execPath)}:${process.env.PATH ?? ""}`,+ };+ return new Promise((done) => {+ const started = Date.now();+ const child = spawn("bash", [scriptFile], {+ env,+ detached: true,+ stdio: ["ignore", "pipe", "pipe"],+ });+ let stdout = "";+ let stderr = "";+ child.stdout.on("data", (d) => (stdout += d));+ child.stderr.on("data", (d) => (stderr += d));+ let timedOut = false;+ const timer = setTimeout(() => {+ timedOut = true;+ try {+ process.kill(-child.pid, "SIGKILL");+ } catch {}+ }, timeoutMs);+ child.on("close", (code) => {+ clearTimeout(timer);+ // Nothing a recipe started outlives it.+ try {+ process.kill(-child.pid, "SIGKILL");+ } catch {}+ let failedLine = null;+ let failedCommand;+ if (code !== 0 && existsSync(failureFile)) {+ const [line, ...cmd] = readFileSync(failureFile, "utf8")+ .trimEnd()+ .split("\t");+ if (/^\d+$/.test(line)) failedLine = Number(line) - preamble.length;+ failedCommand = cmd.join("\t").slice(0, 500);+ }+ done({+ exitCode: timedOut ? null : code,+ timedOut,+ failedLine,+ failedCommand,+ stdout,+ stderr,+ elapsedMs: Date.now() - started,+ });+ });+ });+}++function readJsonl(path) {+ return existsSync(path)+ ? readFileSync(path, "utf8")+ .split("\n")+ .filter(Boolean)+ .map((l) => JSON.parse(l))+ : [];+}++const attemptsFile = (dir) =>+ join(dir, "evidence", "known-bugs", "attempts.jsonl");++async function cmdBug({ positional, flags }) {+ const [sub, id] = positional;+ if (sub === "list") {+ const blocks = knownBugsIn(flags.file === true ? undefined : flags.file);+ out({+ ok: blocks.every((b) => !b.problems.length),+ count: blocks.length,+ blocks: blocks.map(({ script, ...b }) => b),+ });+ return;+ }+ if (sub === "run") {+ if (!id)+ usage(+ "bug run needs a sub-feature ID",+ "control-openhands bug run F05.prefill-send --attempts 2 --label baseline",+ );+ const run = loadRun(flags);+ if (!run) throw new CliError("No run: launch one first", { code: 3 });+ const blocks = selectBlocks(id, flags);+ const attempts = intFlag(flags.attempts, 1);+ const timeoutMs = intFlag(flags.timeout, 180) * 1000;+ const label = flags.label === true ? undefined : flags.label;+ const skipDoctor = Boolean(flags["no-doctor"]);+ const dir = join(run.dir, "evidence", "known-bugs");+ const work = join(run.dir, "private", "known-bugs");+ mkdirSync(dir, { recursive: true });+ mkdirSync(work, { recursive: true });+ const env = { ...process.env, OH_VERIFY_RUN: run.dir };+ const doctor = () =>+ skipDoctor+ ? undefined+ : runSelf(["doctor"], { env, timeout: 180_000 }).code === 0;+ const results = [];+ for (const block of blocks) {+ for (let n = 1; n <= attempts; n += 1) {+ const stamp = `${id}-${block.name}-${Date.now()}-${n}`;+ const logFile = join(work, `${stamp}.asserts.jsonl`);+ const failureFile = join(work, `${stamp}.failed`);+ const doctorBefore = doctor();+ let exec = {+ exitCode: null,+ timedOut: false,+ failedLine: null,+ stdout: "",+ stderr: "",+ elapsedMs: 0,+ };+ if (doctorBefore !== false)+ exec = await runBlock(run, block, {+ timeoutMs,+ logFile,+ failureFile,+ scriptFile: join(work, `${stamp}.sh`),+ });+ const asserts = readJsonl(logFile);+ const assertion = asserts.at(-1);+ const doctorAfter =+ exec.exitCode !== 0 && doctorBefore !== false ? doctor() : undefined;+ const { status, reason } = classifyAttempt({+ doctorBefore,+ doctorAfter,+ exitCode: exec.exitCode,+ timedOut: exec.timedOut,+ failedLine: exec.failedLine,+ bugLines: block.bugLines,+ assertion,+ });+ const page = daemonInfo(run)+ ? await browserCall(run, "url").catch(() => ({}))+ : {};+ const transcript = join(dir, `${stamp}.json`);+ writeFileSync(+ transcript,+ `${JSON.stringify(+ {+ id,+ name: block.name,+ file: block.file,+ script: block.script,+ ...exec,+ stdout: exec.stdout.slice(-8000),+ stderr: exec.stderr.slice(-8000),+ asserts,+ },+ null,+ 2,+ )}\n`,+ );+ const attempt = {+ ts: new Date().toISOString(),+ id,+ name: block.name,+ label,+ attempt: n,+ status,+ reason,+ sha: block.sha,+ file: `${block.file}:${block.line}`,+ exitCode: exec.exitCode,+ failedLine: exec.failedLine,+ failedCommand: exec.failedCommand,+ assertion: assertion+ ? {+ kind: assertion.kind,+ expected: assertion.expected,+ actual: assertion.actual,+ bugShows: assertion.bugShows,+ }+ : undefined,+ doctorBefore,+ doctorAfter,+ elapsedMs: exec.elapsedMs,+ revision: run.revision,+ backend: `${run.mode} agent-server ${run.versions?.agentServer ?? "?"} automation ${run.pins?.automation ?? "?"}`,+ viewport: page.viewport+ ? `${page.viewport.width}x${page.viewport.height}`+ : undefined,+ transcript: relative(run.dir, transcript),+ };+ appendFileSync(attemptsFile(run.dir), `${JSON.stringify(attempt)}\n`);+ appendFileSync(+ join(run.dir, "evidence", "ledger.jsonl"),+ `${JSON.stringify({+ ts: attempt.ts,+ feature: id,+ entry: `known bug: ${block.name}`,+ result: status,+ expected: assertion?.expected,+ actual: assertion?.actual,+ artifacts: [attempt.transcript],+ note: [+ status === "xfail"+ ? "known bug reproduced at its marked assertion"+ : status === "xpass"+ ? "known bug no longer reproduces: review the map note"+ : reason,+ label && `${label} attempt ${n}`,+ ]+ .filter(Boolean)+ .join("; "),+ revision: attempt.revision,+ backend: attempt.backend,+ viewport: attempt.viewport,+ })}\n`,+ );+ results.push(attempt);+ }+ }+ const xpassed = results.some((r) => r.status === "xpass");+ out({+ ok: results.every((r) => r.status === "xfail"),+ run: run.runId,+ revision: run.revision,+ summary: Object.fromEntries(+ ["xfail", "xpass", "fail"].map((s) => [+ s,+ results.filter((r) => r.status === s).length,+ ]),+ ),+ attempts: results,+ hint: xpassed+ ? "The known bug did not reproduce. On a fix under verification that is the goal: compare with `bug proof`. Otherwise review the map note: once the fix is confirmed, drop the note and keep the assertion as the bullet's normal expected result."+ : results.some((r) => r.status === "fail")+ ? "A fail is not a reproduction: read the reason and the transcript, doctor the run, and retry or fix the recipe."+ : undefined,+ });+ if (!results.every((r) => r.status === "xfail")) process.exitCode = 1;+ return;+ }+ if (sub === "proof") {+ if (+ !id ||+ !flags.baseline ||+ flags.baseline === true ||+ !flags.fix ||+ flags.fix === true+ )+ usage(+ "bug proof needs a sub-feature ID, --baseline RUN_DIR and --fix RUN_DIR",+ "control-openhands bug proof F05.prefill-send --baseline $BASE_RUN --fix $FIX_RUN",+ );+ const blocks = selectBlocks(id, flags);+ const read = (dir) => {+ const path = attemptsFile(resolve(String(dir)));+ if (!existsSync(path))+ throw new CliError(`No known-bug attempts at ${path}`, {+ code: 2,+ hint: "Run `control-openhands bug run ID --attempts 2` on that run first.",+ });+ return readJsonl(path).filter((r) => r.id === id);+ };+ // --baseline-label / --fix-label keep only the attempts given that+ // --label, so a deliberate negative check on the same run is set aside.+ const labelled = (rows, label) =>+ label && label !== true ? rows.filter((r) => r.label === label) : rows;+ const baseline = labelled(read(flags.baseline), flags["baseline-label"]);+ const fix = labelled(read(flags.fix), flags["fix-label"]);+ const verdicts = blocks.map((block) => ({+ name: block.name,+ sha: block.sha,+ ...proofVerdict({+ baseline: baseline.filter((r) => r.name === block.name),+ fix: fix.filter((r) => r.name === block.name),+ sha: block.sha,+ min: intFlag(flags.min, 2),+ }),+ }));+ const verified = verdicts.every((v) => v.verdict === "verified");+ out({+ ok: verified,+ id,+ verdict: verified+ ? "verified"+ : verdicts.some((v) => v.verdict === "not-fixed")+ ? "not-fixed"+ : "inconclusive",+ blocks: verdicts,+ hint: verified+ ? "Drop the Known failure note and the block's `# bug` marker in the fix's PR: the assertion now guards the corrected behavior as a normal recipe step."+ : "Only a verified verdict proves the fix. An inconclusive baseline never counts as fixed.",+ });+ if (!verified) process.exitCode = 1;+ return;+ }+ usage(+ "Usage: control-openhands bug list | run ID [--attempts N] [--label L] | proof ID --baseline RUN --fix RUN",+ "control-openhands bug run F05.prefill-send --attempts 2 --label baseline",+ );+}+// ---------------------------------------------------------------------------// Commands: feature-map tooling (index hygiene and coverage).// ---------------------------------------------------------------------------@@ -3371,6 +3803,8 @@ const KNOWN_COMMANDS = new Set(["workspace","browser","evidence",+ "assert",+ "bug","map","help",]);@@ -3506,6 +3940,25 @@ function mapCheck({ only } = {}) {if (m[1] === "browser" && m[2] && !BROWSER_VERBS.has(m[2]))problems.push(`${file}: unknown browser verb \`${m[2]}\``);}+ // Known-bug blocks: one `# bug` assertion naming the bug's value, a+ // declared ID, one block per ID and name.+ const blockNames = new Set();+ for (const block of knownBugBlocks(text)) {+ for (const problem of block.problems)+ problems.push(+ `${file}:${block.line}: known-bug ${block.id}: ${problem}`,+ );+ if (!declared.includes(block.id))+ problems.push(+ `${file}:${block.line}: known-bug ${block.id} is not a sub-feature of this file`,+ );+ const key = `${block.id} ${block.name}`;+ if (blockNames.has(key))+ problems.push(+ `${file}:${block.line}: duplicate known-bug block ${key}`,+ );+ blockNames.add(key);+ }for (const m of text.matchAll(/\]\((\.\.\/[^)#]+)/g)) {const target = resolve(mapDir, m[1]);if (!existsSync(target)) problems.push(`${file}: dead link ${m[1]}`);@@ -4061,6 +4514,8 @@ Drive and observeEvidence and mapevidence add | list | report β the run's pass/fail/blocked/not-run ledger+ assert --field F --equals V [--bug-shows B] -- <command> β check one value of a command's output+ bug list | run ID --attempts 2 | proof ID β known bugs as executable expected failuresmap check | coverage | ids | routes | affected | testids | baseline β keep the feature map honestRun state lives in $OH_VERIFY_HOME (default: $TMPDIR/openhands-verify); the@@ -4306,6 +4761,46 @@ table, then every check with fail and blocked rows first and the --note column(where blocked rows name their missing prerequisite). --baseline compares withanother run's ledger and lists newly failing, newly passing and unchecked rowsfirst (a daily pass against yesterday's run). --json prints the same data.+Rows from \`bug run\` add xfail (a known bug reproduced) and xpass (it stopped+reproducing); xpass rows sort right after fail rows.+`,+ assert: `control-openhands assert --field PATH (--equals VALUE | --matches REGEX) [--bug-shows VALUE] -- <control-openhands command...>++Runs the command, reads PATH (dotted, e.g. value, viewport.width) from its JSON+output and compares it: strings as they are, other values as JSON. Exit 0 when+it holds; 1 on a mismatch; the command's own code when it fails or has no such+field (kind "error": no value was compared). --bug-shows names the value a known+bug produces: a mismatch with any other value is kind "other-symptom".++ control-openhands assert --field enabled --equals true -- browser enabled 'testid=submit-button'+`,+ bug: `control-openhands bug list [--file PATH]+control-openhands bug run ID [--name N] [--attempts N] [--label L] [--timeout SEC] [--no-doctor] [--file PATH]+control-openhands bug proof ID --baseline RUN_DIR --fix RUN_DIR [--baseline-label L] [--fix-label L] [--min 2]++A known bug in the map can carry a fenced block, opened with+"\`\`\`sh known-bug <ID> [name]", whose commands assert the correct behavior. Its one+line ending in "# bug" is a "control-openhands assert ... --bug-shows <value>".++'run' executes the block in bash (set -eo pipefail) against the run, each+attempt after a passing doctor, and classifies it:+ xfail the "# bug" assertion failed and saw the bug's value (reproduced)+ xpass every command held (the bug did not reproduce: review the map note)+ fail anything else: doctor failed before (setup) or after (backend), a+ timeout, an earlier line (setup) or a later one, the bug line's command+ erroring, or a different value (wrong symptom)+Each attempt is appended to evidence/known-bugs/attempts.jsonl (revision,+backend, viewport, recipe hash, failing line, assertion, transcript) and to the+ledger as an xfail/xpass/fail row. Exit 0 only when every attempt is xfail.++'proof' compares the attempts of the current recipe on a baseline run and a fix+run: verified only when the baseline has at least two attempts, all xfail, the+fix at least two, all xpass, on different revisions with the same viewport and+backend. A fix that still reproduces is not-fixed; anything else is inconclusive.++ control-openhands bug run F25.selector-dropdown --attempts 2 --label baseline --run $BASE_RUN+ control-openhands bug run F25.selector-dropdown --attempts 2 --label fix --run $FIX_RUN+ control-openhands bug proof F25.selector-dropdown --baseline $BASE_RUN --baseline-label baseline --fix $FIX_RUN --fix-label fix`,map: `control-openhands map check [--file Fnn-name.md] lint the map (or one entry, skipping index links): four H2s, unique IDs, links, known commands, index countscontrol-openhands map check --fix-counts rewrite the index's sub-feature counts and total from the files, then lint@@ -4349,6 +4844,8 @@ const COMMANDS = {workspace: cmdWorkspace,browser: cmdBrowser,evidence: cmdEvidence,+ assert: cmdAssert,+ bug: cmdBug,map: cmdMap,};diff --git a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjsindex 9690da8fea..6825643183 100755--- a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs+++ b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs@@ -53,6 +53,12 @@ import {import { MAX_PAGES, collectEvents, countImages } from "./lib/events-paging.mjs";import { redactBody, redactUrl } from "./lib/network-bodies.mjs";import { buildLocator, parseRole, toCss } from "./lib/selectors.mjs";+import {+ classifyAttempt,+ judgeAssertion,+ knownBugBlocks,+ proofVerdict,+} from "./lib/known-bug.mjs";const here = dirname(fileURLToPath(import.meta.url));const cli = join(here, "control-openhands.mjs");@@ -1844,3 +1850,406 @@ test.skipIf(process.platform === "win32")(}},);++// A run directory with no stack behind it: enough for evidence and bug runs+// whose recipes only call offline commands.+function fakeRun(revision = "abc") {+ const dir = mkdtempSync(join(tmpdir(), "bug-run-"));+ mkdirSync(join(dir, "private"));+ mkdirSync(join(dir, "evidence"));+ writeFileSync(+ join(dir, "run.json"),+ JSON.stringify({+ runId: `r-${revision}`,+ revision,+ mode: "local",+ baseUrl: "http://127.0.0.1:9",+ ports: { ingress: 9 },+ launcherPgid: 0,+ versions: { agentServer: "1.54.0" },+ pins: { automation: "1.19.3" },+ }),+ );+ return dir;+}++const block = (body, head = "```sh known-bug F14.create desktop") =>+ `# F14\n\n- Known failure (#1):\n\n ${head}\n${body+ .split("\n")+ .map((l) => ` ${l}`)+ .join("\n")}\n \`\`\`\n`;++test("known-bug blocks parse with their bug line, and malformed ones say why", () => {+ const [ok] = knownBugBlocks(+ block(+ "control-openhands browser goto /\ncontrol-openhands assert --field value --equals Local --bug-shows '' -- browser value 'testid=x' # bug",+ ),+ );+ assert.equal(ok.id, "F14.create");+ assert.equal(ok.name, "desktop");+ assert.deepEqual(ok.bugLines, [2]);+ assert.deepEqual(ok.problems, []);+ assert.match(ok.script, /^control-openhands browser goto \/\n/);+ assert.match(ok.sha, /^[0-9a-f]{16}$/);+ const [unnamed] = knownBugBlocks(+ block(+ "control-openhands assert --field a --equals 1 --bug-shows 0 -- status # bug",+ "```sh known-bug F14.create",+ ),+ );+ assert.equal(unnamed.name, "default");+ const problems = (body) => knownBugBlocks(block(body))[0].problems.join("; ");+ assert.match(+ problems("control-openhands browser goto /"),+ /exactly one line ending in `# bug` \(found 0\)/,+ );+ assert.match(+ problems("false # bug"),+ /must be a `control-openhands assert`/,+ );+ assert.match(+ problems("control-openhands assert --field a --equals 1 -- status # bug"),+ /must name the bug's value with --bug-shows/,+ );+ assert.match(+ knownBugBlocks(+ "```sh known-bug F14.create\ncontrol-openhands status\n",+ )[0].problems.join(),+ /never closed/,+ );+});++test("an assertion tells the bug's value from another value and from an error", () => {+ const json = { ok: true, value: "", viewport: { width: 390 } };+ const judge = (o) =>+ judgeAssertion({+ exitCode: 0,+ json,+ field: "value",+ equals: "Local",+ bugShows: "",+ ...o,+ });+ assert.equal(judge({}).kind, "mismatch");+ assert.equal(judge({ bugShows: "Remote" }).kind, "other-symptom");+ assert.equal(judge({ bugShows: undefined }).kind, "mismatch");+ assert.equal(judge({ equals: "" }).kind, "hold");+ assert.equal(judge({ field: "viewport.width", equals: "390" }).kind, "hold");+ assert.equal(judge({ field: "nope" }).kind, "error");+ assert.equal(judge({ exitCode: 1 }).kind, "error");+ assert.equal(judge({ equals: undefined, matches: "^$" }).kind, "hold");+});++test("only a mismatch at the bug line on a healthy stack counts as reproducing", () => {+ const base = {+ doctorBefore: true,+ doctorAfter: true,+ exitCode: 1,+ timedOut: false,+ failedLine: 4,+ bugLines: [4],+ assertion: { kind: "mismatch" },+ };+ const c = (o) => classifyAttempt({ ...base, ...o });+ assert.deepEqual(c({}), { status: "xfail" });+ assert.equal(c({ exitCode: 0, failedLine: null }).status, "xpass");+ // Negative cases: none of these is the bug.+ assert.match(+ c({ doctorBefore: false }).reason,+ /^setup: doctor failed before/,+ );+ assert.match(+ c({ doctorAfter: false }).reason,+ /^backend: doctor failed after/,+ );+ assert.equal(c({ timedOut: true, exitCode: null }).reason, "timeout");+ assert.match(c({ failedLine: 2 }).reason, /^setup: line 2 failed before/);+ assert.match(c({ failedLine: 5 }).reason, /after the `# bug` assertion held/);+ assert.match(+ c({ assertion: { kind: "error" } }).reason,+ /command itself failed/,+ );+ assert.match(+ c({ assertion: { kind: "other-symptom" } }).reason,+ /^wrong symptom/,+ );+ assert.match(c({ assertion: undefined }).reason, /no assertion result/);+ assert.match(c({ failedLine: null }).reason, /unknown/);+ for (const o of [+ { doctorBefore: false },+ { doctorAfter: false },+ { timedOut: true },+ { failedLine: 2 },+ { assertion: { kind: "error" } },+ { assertion: { kind: "other-symptom" } },+ ])+ assert.equal(c(o).status, "fail", JSON.stringify(o));+});++test("a fix is verified only by two reproductions and two passes under the same inputs", () => {+ const row = (status, o = {}) => ({+ status,+ sha: "s1",+ revision: "base",+ viewport: "1440x1000",+ backend: "local agent-server 1.54.0 automation 1.19.3",+ ...o,+ });+ const fixRow = (status, o = {}) => row(status, { revision: "fix", ...o });+ const v = (baseline, fix) => proofVerdict({ baseline, fix, sha: "s1" });+ const good = v(+ [row("xfail"), row("xfail")],+ [fixRow("xpass"), fixRow("xpass")],+ );+ assert.equal(good.verdict, "verified", good.reasons.join());+ // A baseline that does not reproduce reliably is inconclusive, never fixed.+ assert.equal(+ v([row("xfail"), row("xpass")], [fixRow("xpass"), fixRow("xpass")]).verdict,+ "inconclusive",+ );+ assert.equal(+ v([row("xfail"), row("fail")], [fixRow("xpass"), fixRow("xpass")]).verdict,+ "inconclusive",+ );+ assert.equal(+ v([row("xfail")], [fixRow("xpass"), fixRow("xpass")]).verdict,+ "inconclusive",+ );+ assert.equal(+ v([row("xfail"), row("xfail")], [fixRow("xpass")]).verdict,+ "inconclusive",+ );+ assert.equal(+ v([row("xfail"), row("xfail")], [fixRow("xpass"), fixRow("xfail")]).verdict,+ "not-fixed",+ );+ assert.equal(+ v([row("xfail"), row("xfail")], [fixRow("xpass"), fixRow("fail")]).verdict,+ "inconclusive",+ );+ // Inputs must match.+ assert.match(+ v(+ [row("xfail"), row("xfail")],+ [fixRow("xpass"), fixRow("xpass", { viewport: "390x844" })],+ ).reasons.join(),+ /different viewports/,+ );+ assert.match(+ v(+ [row("xfail"), row("xfail")],+ [fixRow("xpass", { backend: "x" }), fixRow("xpass", { backend: "x" })],+ ).reasons.join(),+ /different backend/,+ );+ assert.match(+ v(+ [row("xfail"), row("xfail")],+ [row("xpass"), row("xpass")],+ ).reasons.join(),+ /same revision/,+ );+ // Attempts of an older recipe do not count.+ const old = v(+ [row("xpass", { sha: "s0" }), row("xfail"), row("xfail")],+ [fixRow("xpass"), fixRow("xpass")],+ );+ assert.equal(old.verdict, "verified");+ assert.equal(old.ignored, 1);+});++test("bug run classifies live attempts and bug proof turns two runs into a verdict", () => {+ const dir = mkdtempSync(join(tmpdir(), "bug-map-"));+ const write = (name, body, timeout) => {+ const file = join(dir, `${name}.md`);+ writeFileSync(file, block(body));+ return file;+ };+ // `evidence list` on an empty ledger prints {"count": 0}: the "bug" is a+ // count of 0 where 1 is expected.+ const bugLine = (o = "--equals 1 --bug-shows 0") =>+ `control-openhands assert --field count ${o} -- evidence list --feature QA # bug`;+ const cases = {+ reproduced: [`true\n${bugLine()}`, "xfail"],+ fixed: [`true\n${bugLine("--equals 0 --bug-shows 1")}`, "xpass"],+ setup: [`false\n${bugLine()}`, "fail", /^setup: line 1/],+ symptom: [+ `true\n${bugLine("--equals 1 --bug-shows 7")}`,+ "fail",+ /^wrong symptom/,+ ],+ broken: [+ "true\ncontrol-openhands assert --field count --equals 1 --bug-shows 0 -- evidence nope # bug",+ "fail",+ /command itself failed/,+ ],+ later: [+ `${bugLine("--equals 0 --bug-shows 1")}\nfalse`,+ "fail",+ /after the `# bug` assertion held/,+ ],+ timeout: [`sleep 5\n${bugLine()}`, "fail", /^timeout$/, ["--timeout", "1"]],+ };+ for (const [name, [body, status, reason, extra = []]] of Object.entries(+ cases,+ )) {+ const runDir = fakeRun();+ const r = run(+ [+ "bug",+ "run",+ "F14.create",+ "--file",+ write(name, body),+ "--no-doctor",+ ...extra,+ ],+ { OH_VERIFY_RUN: runDir },+ );+ const [attempt] = r.json.attempts;+ assert.equal(attempt.status, status, `${name}: ${r.stdout}`);+ if (reason) assert.match(attempt.reason, reason, name);+ assert.equal(r.status, status === "xfail" ? 0 : 1, name);+ // Every attempt lands in the ledger as its own result.+ const ledger = readFileSync(+ join(runDir, "evidence", "ledger.jsonl"),+ "utf8",+ );+ assert.match(ledger, new RegExp(`"result":"${status}"`), name);+ assert.ok(existsSync(join(runDir, attempt.transcript)), name);+ }+ // Two attempts on each side, then the verdict.+ const reproduced = write("proof-base", `true\n${bugLine()}`);+ const base = fakeRun("base");+ const fix = fakeRun("fix");+ const twice = (runDir, file, label) =>+ run(+ [+ "bug",+ "run",+ "F14.create",+ "--file",+ file,+ "--no-doctor",+ "--attempts",+ "2",+ "--label",+ label,+ ],+ {+ OH_VERIFY_RUN: runDir,+ },+ );+ assert.deepEqual(twice(base, reproduced, "baseline").json.summary, {+ xfail: 2,+ xpass: 0,+ fail: 0,+ });+ // The fix run sees the same recipe pass: a ledger row makes count 1.+ writeFileSync(+ join(fix, "evidence", "ledger.jsonl"),+ `${JSON.stringify({ feature: "QA", result: "pass", artifacts: [] })}\n`,+ );+ const fixed = twice(fix, reproduced, "fix");+ assert.equal(fixed.json.summary.xpass, 2, fixed.stdout);+ assert.match(fixed.json.hint, /bug proof/);+ const proof = run([+ "bug",+ "proof",+ "F14.create",+ "--file",+ reproduced,+ "--baseline",+ base,+ "--fix",+ fix,+ ]);+ assert.equal(proof.json.verdict, "verified", proof.stdout);+ assert.equal(proof.status, 0);+ // The same run on both sides proves nothing; a one-sided rerun is short.+ assert.equal(+ run([+ "bug",+ "proof",+ "F14.create",+ "--file",+ reproduced,+ "--baseline",+ base,+ "--fix",+ base,+ ]).json.verdict,+ "inconclusive",+ );+ // A negative attempt on the baseline makes it inconclusive unless set aside+ // by label.+ run(+ [+ "bug",+ "run",+ "F14.create",+ "--file",+ reproduced,+ "--no-doctor",+ "--timeout",+ "0",+ "--label",+ "negative",+ ],+ {+ OH_VERIFY_RUN: base,+ },+ );+ const args = [+ "bug",+ "proof",+ "F14.create",+ "--file",+ reproduced,+ "--baseline",+ base,+ "--fix",+ fix,+ ];+ assert.equal(run(args).json.verdict, "inconclusive");+ assert.equal(+ run([...args, "--baseline-label", "baseline", "--fix-label", "fix"]).json+ .verdict,+ "verified",+ );+ // The report keeps xfail and xpass apart from pass and fail.+ const report = run(["evidence", "report"], { OH_VERIFY_RUN: fix });+ assert.match(+ report.stdout,+ /\| Family \| pass \| fail \| blocked \| not-run \| xfail \| xpass \|/,+ );+ assert.match(report.stdout, /\| F14 \| 0 \| 0 \| 0 \| 0 \| 0 \| 1 \|/);+ assert.match(+ report.stdout,+ /\| xpass \| known bug no longer reproduces: review the map note; fix attempt 2 \|/,+ );+ // bug list reports malformed blocks.+ const bad = run(["bug", "list", "--file", write("bad", "true")]);+ assert.equal(bad.json.ok, false);+ assert.match(bad.json.blocks[0].problems[0], /exactly one line/);+});++test("help documents assert and bug, and the shipped map's known-bug blocks are well formed", () => {+ const help = run(["--help"]).stdout;+ assert.match(help, /assert +--field/);+ assert.match(help, /bug +list \| run ID/);+ const bug = run(["help", "bug"]).stdout;+ for (const word of [+ "xfail",+ "xpass",+ "proof",+ "--bug-shows",+ "--attempts",+ "inconclusive",+ ])+ assert.ok(bug.includes(word), word);+ assert.match(run(["help", "assert"]).stdout, /--bug-shows/);+ const list = run(["bug", "list"]);+ assert.equal(list.json.ok, true, list.stdout);+ assert.ok(list.json.count >= 1);+});diff --git a/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs b/.agents/skills/verify-openhands/scripts/lib/known-bug.mjsnew file mode 100644index 0000000000..a9ae195217--- /dev/null+++ b/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs@@ -0,0 +1,235 @@+// Known bugs as executable expected failures.+//+// A feature-map note that records an open bug may carry a fenced block+//+// ```sh known-bug F05.example desktop+// control-openhands browser goto /+// control-openhands assert --field enabled --equals true --bug-shows false -- browser enabled 'testid=submit-button' # bug+// ```+//+// whose commands assert the correct behavior. The one line ending in `# bug`+// is a `control-openhands assert` that names the value the bug shows+// (`--bug-shows`). `bug run` executes the block and classifies each attempt:+// the bug reproduced (xfail) only when that assertion is the command that+// failed and it saw the bug's own value; the correct behavior held (xpass);+// anything else (setup, timeout, an unhealthy stack, another symptom) is a+// fail. `bug proof` turns two runs' attempts into a verdict on a fix.+import { createHash } from "node:crypto";++const OPEN = /^(\s*)```sh known-bug (F\d{2}\.[a-z0-9-]+)(?: ([a-z0-9-]+))?\s*$/;+const BUG_LINE = /#\s*bug\s*$/;++/**+ * Every known-bug block in a map file's text.+ * @returns {{ id: string, name: string, line: number, script: string, bugLines: number[], sha: string, problems: string[] }[]}+ */+export function knownBugBlocks(text) {+ const lines = text.split("\n");+ const blocks = [];+ for (let i = 0; i < lines.length; i += 1) {+ const open = OPEN.exec(lines[i]);+ if (!open) continue;+ const [, indent, id, name = "default"] = open;+ const body = [];+ let j = i + 1;+ while (j < lines.length && lines[j].trim() !== "```") {+ body.push(+ lines[j].startsWith(indent) ? lines[j].slice(indent.length) : lines[j],+ );+ j += 1;+ }+ const problems = [];+ if (j >= lines.length) problems.push("the block is never closed");+ const bugLines = body+ .map((l, n) => (BUG_LINE.test(l) ? n + 1 : 0))+ .filter(Boolean);+ if (bugLines.length !== 1)+ problems.push(+ `needs exactly one line ending in \`# bug\` (found ${bugLines.length})`,+ );+ for (const n of bugLines) {+ const l = body[n - 1];+ if (!/^control-openhands assert\b/.test(l.trim()))+ problems.push(+ `line ${n}: the \`# bug\` line must be a \`control-openhands assert\``,+ );+ else if (!/--bug-shows\b/.test(l))+ problems.push(+ `line ${n}: the \`# bug\` assertion must name the bug's value with --bug-shows`,+ );+ }+ const script = body.join("\n");+ blocks.push({+ id,+ name,+ line: i + 1,+ script,+ bugLines,+ sha: createHash("sha256").update(script).digest("hex").slice(0, 16),+ problems,+ });+ i = j;+ }+ return blocks;+}++/**+ * The value at a dotted path of a command's JSON output, as a string:+ * strings as they are, everything else as JSON.+ */+export function fieldValue(json, path) {+ let value = json;+ for (const key of String(path).split(".")) {+ if (value === null || typeof value !== "object" || !(key in value))+ return undefined;+ value = value[key];+ }+ return typeof value === "string" ? value : JSON.stringify(value);+}++/**+ * What an assertion saw.+ * kind: hold | mismatch (the bug's own value, or any mismatch when no+ * --bug-shows is given) | other-symptom (a mismatch, but not the bug's value)+ * | error (the command failed or printed no such field).+ */+export function judgeAssertion({+ exitCode,+ json,+ field,+ equals,+ matches,+ bugShows,+}) {+ if (exitCode !== 0 || !json)+ return { kind: "error", reason: `the command exited ${exitCode}` };+ const actual = fieldValue(json, field);+ if (actual === undefined)+ return { kind: "error", reason: `the output has no field ${field}` };+ const holds =+ equals !== undefined+ ? actual === String(equals)+ : new RegExp(matches).test(actual);+ if (holds) return { kind: "hold", actual };+ if (bugShows !== undefined && actual !== String(bugShows))+ return { kind: "other-symptom", actual };+ return { kind: "mismatch", actual };+}++/**+ * One attempt's result.+ * @param {object} a+ * @param {boolean} a.doctorBefore false when the stack was unhealthy before the attempt+ * @param {boolean|undefined} a.doctorAfter false when it was unhealthy after a failure+ * @param {number} a.exitCode+ * @param {boolean} a.timedOut+ * @param {number|null} a.failedLine 1-based line of the block that failed+ * @param {number[]} a.bugLines+ * @param {{kind: string, line?: number}|undefined} a.assertion the last assertion the attempt made+ * @returns {{ status: "xfail"|"xpass"|"fail", reason?: string }}+ */+export function classifyAttempt(a) {+ if (a.doctorBefore === false)+ return {+ status: "fail",+ reason: "setup: doctor failed before the attempt",+ };+ if (a.timedOut) return { status: "fail", reason: "timeout" };+ if (a.exitCode === 0) return { status: "xpass" };+ if (a.failedLine == null)+ return { status: "fail", reason: "the failing line is unknown" };+ const bugLine = a.bugLines[0];+ if (a.failedLine < bugLine)+ return {+ status: "fail",+ reason: `setup: line ${a.failedLine} failed before the \`# bug\` line`,+ };+ if (a.failedLine > bugLine)+ return {+ status: "fail",+ reason: `line ${a.failedLine} failed after the \`# bug\` assertion held`,+ };+ if (a.doctorAfter === false)+ return {+ status: "fail",+ reason: "backend: doctor failed after the attempt",+ };+ const kind = a.assertion?.kind;+ if (kind === "mismatch") return { status: "xfail" };+ if (kind === "other-symptom")+ return {+ status: "fail",+ reason:+ "wrong symptom: the assertion saw neither the expected nor the bug's value",+ };+ if (kind === "error")+ return {+ status: "fail",+ reason: "the `# bug` command itself failed; no value was compared",+ };+ return {+ status: "fail",+ reason: "no assertion result was recorded at the `# bug` line",+ };+}++const same = (values) => new Set(values).size <= 1;++/**+ * Verdict on a fix from the attempts recorded on a baseline run and a fix+ * run for one block. Only attempts of the given recipe (sha) count.+ * verified: every baseline attempt (at least two) reproduced the bug and+ * every fix attempt (at least two) passed, under matching inputs.+ * not-fixed: the fix still reproduces it. inconclusive: anything else.+ */+export function proofVerdict({ baseline, fix, sha, min = 2 }) {+ const reasons = [];+ const pick = (rows) => rows.filter((r) => r.sha === sha);+ const b = pick(baseline);+ const f = pick(fix);+ const ignored = baseline.length + fix.length - b.length - f.length;+ const tally = (rows) =>+ rows.reduce((t, r) => ({ ...t, [r.status]: (t[r.status] ?? 0) + 1 }), {});+ const bx = b.filter((r) => r.status === "xfail").length;+ const fp = f.filter((r) => r.status === "xpass").length;+ if (b.length < min)+ reasons.push(+ `baseline has ${b.length} attempt(s) of this recipe; need ${min}`,+ );+ else if (bx !== b.length)+ reasons.push(+ `baseline reproduced the bug in ${bx} of ${b.length} attempts (flaky or not reproduced)`,+ );+ let notFixed = false;+ if (f.length < min)+ reasons.push(`fix has ${f.length} attempt(s) of this recipe; need ${min}`);+ else if (fp !== f.length) {+ notFixed = f.some((r) => r.status === "xfail");+ reasons.push(`fix passed in ${fp} of ${f.length} attempts`);+ }+ const all = [...b, ...f];+ const inputs = [];+ if (!same(all.map((r) => r.viewport ?? "")))+ inputs.push("attempts ran at different viewports");+ if (!same(all.map((r) => r.backend ?? "")))+ inputs.push("attempts ran against different backend versions");+ if (b.length && f.length && same(all.map((r) => r.revision)))+ inputs.push("baseline and fix ran the same revision");+ if (!same(b.map((r) => r.revision)) || !same(f.map((r) => r.revision)))+ inputs.push("attempts on one side ran different revisions");+ reasons.push(...inputs);+ // Mismatched inputs or an unreliable baseline make any fix reading moot.+ const baselineReproduced = b.length >= min && bx === b.length;+ const verdict = !reasons.length+ ? "verified"+ : notFixed && baselineReproduced && !inputs.length+ ? "not-fixed"+ : "inconclusive";+ return {+ verdict,+ reasons,+ baseline: tally(b),+ fix: tally(f),+ ignored,+ };+}