π OpenHands: AI-Driven Development
feat(verify-openhands): adopt the shared known-bug contract (#18355)
Engel Nyst0e9ce191390f
diff --git a/.agents/skills/verify-openhands/references/mapping.md b/.agents/skills/verify-openhands/references/mapping.mdindex aa566684c9..57d5c3939d 100644--- a/.agents/skills/verify-openhands/references/mapping.md+++ b/.agents/skills/verify-openhands/references/mapping.md@@ -128,15 +128,18 @@ sections, in this order:under it, opened with `` ```sh known-bug <ID> [name] `` (the name tellsentry points or viewports apart, such as `desktop` and `phone`). Itscommands set their own viewport and starting page and assert the *correct*- behavior; exactly one line, the one that fails because of the bug, is a+ behavior; exactly one command, the one that fails because of the bug, is a`control-openhands assert --field <path> --equals <correct> --bug-shows- <what the bug shows> -- <command>` ending in `# bug`. Arrange steps go+ <what the bug shows> -- <command>` ending in `# bug` (a long one may+ continue over lines ending in `\`). When the bug's value varies, or+ needs more than one field to pin down, name it with `--bug 'PATH OP+ [VALUE] && ...'` instead (see the known-bug contract below). Arrange steps gobefore it and must be able to fail on their own (assert the starting valuetoo), so a broken setup is never read as the bug. Keep a `browserscreenshot` before the assertion where the symptom is visual.`control-openhands bug run <ID>` then reports `xfail` while the bug lasts,`xpass` once it stops, and `fail` for anything else; `map check` rejects a- block without exactly one such line. F25's `F25.selector-dropdown` note is+ block without exactly one such command. F25's `F25.selector-dropdown` note isthe worked example. When a fix is proven, the PR that drops the note turnsthe block's commands into the bullet's normal steps, so the same assertionguards the corrected behavior.@@ -232,3 +235,43 @@ parallel, each edits only its own file and checks it with `--file`; onecoordinator updates the index afterwards.Then hand the evidence ledger (`control-openhands evidence report`) to whoeverasked, with fail/blocked rows first.++## The known-bug contract++Keep these rules in step with the sibling verification skills+(verify-agent-server in OpenHands/software-agent-sdk, verify-automation in+OpenHands/automation), so a proof means the same thing in each. The code stays+separate; only the syntax for naming the bug differs, noted below.++- The recipe asserts the correct behavior. The command that fails because of+ the bug ends its line with `# bug` and names the bug's own symptom: the+ value or error it produces, not just any wrong value. Here that command is+ always a `control-openhands assert`, which carries the symptom as+ `--bug-shows V` or `--bug PREDICATE`; the sibling skills write+ `# bug: PREDICATE` on the marked line instead.+- Three outcomes. `xfail`: the marked command failed and its symptom held (the+ documented bug reproduced). `xpass`: every command passed, so the correct+ behavior was observed; `bug run` exits 1 so the note gets reviewed and the+ marker removed. `fail`: anything else, including a different value at the+ marked command, an arrange step that breaks first, a timeout, an unhealthy+ stack before or after, a failure after the marked command, or a failing line+ the runner could not locate.+- The predicate is `PATH OP [VALUE]` clauses joined by `&&`, matched against+ the JSON the asserted command printed: `--bug 'status eq 500 &&+ body.error contains "has no attribute"'`. OP is `eq`, `ne`, `contains`,+ `not-contains`, `matches` (a regex), `exists` or `missing`. `--bug-shows V`+ is `--bug '<field> eq V'`. Prefer `contains` or `matches` on the stable part+ of an error message over pinning its whole text.+- `xpass` must mean the correct value was observed, so the marked command+ asserts positively. Negative shapes (`! cmd`, `if cmd; then false; fi`,+ `test !`) cannot be marked: the marked command must be an `assert`, and+ `map check` rejects anything else.+- `map check` rejects a missing or malformed predicate.+- The observation is the marked command's own. Every assertion is logged with+ the recipe line that ran it and the first failure is kept, so a later step+ or an `EXIT` trap's cleanup cannot replace it. A marked command continued+ over several lines counts as one, from its first line to the `# bug` line.+- A fix is proven with the baseline reproducing (`xfail`) twice and the fix+ passing (`xpass`) twice, with the same recipe, viewport and backend:+ `control-openhands bug proof` computes that verdict (`verified`,+ `not-fixed` or `inconclusive`).diff --git a/.agents/skills/verify-openhands/scripts/control-openhands.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.mjsindex 0488b11f61..ed27a53b0c 100755--- a/.agents/skills/verify-openhands/scripts/control-openhands.mjs+++ b/.agents/skills/verify-openhands/scripts/control-openhands.mjs@@ -38,6 +38,7 @@ import {classifyAttempt,judgeAssertion,knownBugBlocks,+ parseBugPredicate,proofVerdict,} from "./lib/known-bug.mjs";import { routePattern } from "./lib/route-pattern.mjs";@@ -3398,6 +3399,16 @@ async function cmdAssert({ positional, flags }) {"assert needs --field PATH, one of --equals VALUE or --matches REGEX, and -- <control-openhands command>","control-openhands assert --field enabled --equals true --bug-shows false -- browser enabled 'testid=submit-button'",);+ const bug =+ flags.bug === undefined || flags.bug === true+ ? undefined+ : String(flags.bug);+ const bugProblems = bug === undefined ? [] : parseBugPredicate(bug).problems;+ if (bugProblems.length)+ usage(+ `--bug ${bugProblems.join("; ")}`,+ `control-openhands assert --field status --equals 200 --bug 'status eq 500 && body.error contains "has no attribute"' -- api GET /api/...`,+ );const env = { ...process.env };if (flags.run && flags.run !== true) env.OH_VERIFY_RUN = resolve(flags.run);const child = runSelf(positional.map(String), { env });@@ -3411,6 +3422,7 @@ async function cmdAssert({ positional, flags }) {flags["bug-shows"] === undefined || flags["bug-shows"] === true? undefined: String(flags["bug-shows"]),+ bug,});const result = {ok: judged.kind === "hold",@@ -3418,18 +3430,20 @@ async function cmdAssert({ positional, flags }) {field: flags.field,expected: hasEquals ? String(flags.equals) : `/${flags.matches}/`,bugShows: flags["bug-shows"] === true ? undefined : flags["bug-shows"],+ bug,command: positional.join(" "),output:judged.kind === "error"? (child.json ?? child.stdout.slice(-2000)): undefined,};- // Under `bug run` every assertion is logged, so the runner can tell the- // marked assertion's mismatch from an error at the same line.+ // Under `bug run` every assertion is logged with the recipe line that ran+ // it, so the runner reads the marked command's own result, never one a+ // later cleanup step logged.if (process.env.OH_VERIFY_ASSERT_LOG)appendFileSync(process.env.OH_VERIFY_ASSERT_LOG,- `${JSON.stringify({ ts: new Date().toISOString(), ...result, output: undefined })}\n`,+ `${JSON.stringify({ ts: new Date().toISOString(), line: process.env.OH_VERIFY_LINE, ...result, output: undefined })}\n`,);out(result);if (judged.kind === "error") process.exitCode = child.code || 1;@@ -3475,10 +3489,15 @@ function selectBlocks(id, flags) {}function runBlock(run, block, { timeoutMs, logFile, failureFile, scriptFile }) {+ // The DEBUG trap tags every command (and the asserts it runs) with its+ // line. The ERR trap keeps the first failure only, and retags what runs+ // after it (an EXIT trap's cleanup), so neither overwrites the observation.const preamble = ["set -eo pipefail",`cd ${JSON.stringify(repoRoot)}`,- `trap 'printf "%s\\t%s\\n" "$LINENO" "$BASH_COMMAND" > ${JSON.stringify(failureFile)}' ERR`,+ "export OH_VERIFY_LINE=",+ "trap 'OH_VERIFY_LINE=$LINENO' DEBUG",+ `trap '[ -e ${JSON.stringify(failureFile)} ] || printf "%s\\t%s\\n" "$LINENO" "$BASH_COMMAND" > ${JSON.stringify(failureFile)}; OH_VERIFY_LINE=after-failure' ERR`,];writeFileSync(scriptFile, `${preamble.join("\n")}\n${block.script}\n`);const env = {@@ -3521,6 +3540,7 @@ function runBlock(run, block, { timeoutMs, logFile, failureFile, scriptFile }) {failedCommand = cmd.join("\t").slice(0, 500);}done({+ lineOffset: preamble.length,exitCode: timedOut ? null : code,timedOut,failedLine,@@ -3600,8 +3620,16 @@ async function cmdBug({ positional, flags }) {failureFile,scriptFile: join(work, `${stamp}.sh`),});- const asserts = readJsonl(logFile);- const assertion = asserts.at(-1);+ const asserts = readJsonl(logFile).map((a) => ({+ ...a,+ line: /^\d+$/.test(a.line ?? "")+ ? Number(a.line) - exec.lineOffset+ : a.line,+ }));+ const [start, end] = block.bugRanges[0];+ const assertion = asserts+ .filter((a) => a.line >= start && a.line <= end)+ .at(-1);const doctorAfter =exec.exitCode !== 0 && doctorBefore !== false ? doctor() : undefined;const { status, reason } = classifyAttempt({@@ -3611,6 +3639,7 @@ async function cmdBug({ positional, flags }) {timedOut: exec.timedOut,failedLine: exec.failedLine,bugLines: block.bugLines,+ bugRanges: block.bugRanges,assertion,});const page = daemonInfo(run)@@ -3653,6 +3682,8 @@ async function cmdBug({ positional, flags }) {expected: assertion.expected,actual: assertion.actual,bugShows: assertion.bugShows,+ bug: assertion.bug,+ bugChecks: assertion.bugChecks,}: undefined,doctorBefore,@@ -4514,7 +4545,7 @@ Drive and observeEvidence and mapevidence add | list | report β the run's pass/fail/blocked/not-run ledger- assert --field F --equals V [--bug-shows B] -- <command> β check one value of a command's output+ assert --field F --equals V [--bug-shows B | --bug P] -- <command> β check one value of a command's outputbug list | run ID --attempts 2 | proof ID β known bugs as executable expected failuresmap check | coverage | ids | routes | affected | testids | baseline β keep the feature map honest@@ -4764,15 +4795,20 @@ first (a daily pass against yesterday's run). --json prints the same data.Rows from \`bug run\` add xfail (a known bug reproduced) and xpass (it stoppedreproducing); xpass rows sort right after fail rows.`,- assert: `control-openhands assert --field PATH (--equals VALUE | --matches REGEX) [--bug-shows VALUE] -- <control-openhands command...>+ assert: `control-openhands assert --field PATH (--equals VALUE | --matches REGEX) [--bug-shows VALUE] [--bug PREDICATE] -- <control-openhands command...>Runs the command, reads PATH (dotted, e.g. value, viewport.width) from its JSONoutput and compares it: strings as they are, other values as JSON. Exit 0 whenit holds; 1 on a mismatch; the command's own code when it fails or has no such-field (kind "error": no value was compared). --bug-shows names the value a known-bug produces: a mismatch with any other value is kind "other-symptom".+field (kind "error": no value was compared). --bug-shows and --bug name what a+known bug produces; a mismatch that does not match them is kind "other-symptom".+--bug-shows V is short for "--bug 'PATH eq V'". --bug takes "PATH OP [VALUE]"+clauses joined by "&&", on any field of the output, with OP one of eq, ne,+contains, not-contains, matches (regex), exists, missing; quote a VALUE with+spaces. The result lists each clause under bugChecks.control-openhands assert --field enabled --equals true -- browser enabled 'testid=submit-button'+ control-openhands assert --field text --equals Saved --bug 'text contains "An error occurred"' -- browser text 'testid=toast'`,bug: `control-openhands bug list [--file PATH]control-openhands bug run ID [--name N] [--attempts N] [--label L] [--timeout SEC] [--no-doctor] [--file PATH]@@ -4780,11 +4816,12 @@ control-openhands bug proof ID --baseline RUN_DIR --fix RUN_DIR [--baseline-labeA known bug in the map can carry a fenced block, opened with"\`\`\`sh known-bug <ID> [name]", whose commands assert the correct behavior. Its one-line ending in "# bug" is a "control-openhands assert ... --bug-shows <value>".+command ending in "# bug" (continued lines ending in "\\" count as one) is a+"control-openhands assert ... --bug-shows <value>" or "... --bug <predicate>".'run' executes the block in bash (set -eo pipefail) against the run, eachattempt after a passing doctor, and classifies it:- xfail the "# bug" assertion failed and saw the bug's value (reproduced)+ xfail the "# bug" assertion failed and saw the bug's symptom (reproduced)xpass every command held (the bug did not reproduce: review the map note)fail anything else: doctor failed before (setup) or after (backend), atimeout, an earlier line (setup) or a later one, the bug line's command@@ -4792,6 +4829,9 @@ attempt after a passing doctor, and classifies it:Each attempt is appended to evidence/known-bugs/attempts.jsonl (revision,backend, viewport, recipe hash, failing line, assertion, transcript) and to theledger as an xfail/xpass/fail row. Exit 0 only when every attempt is xfail.+Each assertion is logged with the recipe line that ran it, and the first+failure is the one kept, so a cleanup step (an EXIT trap) cannot replace the+marked command's observation.'proof' compares the attempts of the current recipe on a baseline run and a fixrun: verified only when the baseline has at least two attempts, all xfail, thediff --git a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjsindex 6825643183..5dd9170713 100755--- a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs+++ b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs@@ -57,6 +57,7 @@ import {classifyAttempt,judgeAssertion,knownBugBlocks,+ parseBugPredicate,proofVerdict,} from "./lib/known-bug.mjs";@@ -1909,8 +1910,37 @@ test("known-bug blocks parse with their bug line, and malformed ones say why", ();assert.match(problems("control-openhands assert --field a --equals 1 -- status # bug"),- /must name the bug's value with --bug-shows/,+ /must name what the bug shows with --bug-shows or --bug/,);+ assert.match(+ problems(+ "control-openhands assert --field a --equals 1 --bug 'a is 0' -- status # bug",+ ),+ /--bug `a is 0` is not PATH OP \[VALUE\]/,+ );+ assert.match(+ problems(+ "control-openhands assert --field a --equals 1 --bug 'a eq' -- status # bug",+ ),+ /needs a value/,+ );+ assert.deepEqual(+ knownBugBlocks(+ block(+ "control-openhands assert --field a --equals 1 --bug 'a eq 0 && b.c contains \"no such\"' -- status # bug",+ ),+ )[0].problems,+ [],+ );+ // A command continued over several lines is one marked command.+ const [continued] = knownBugBlocks(+ block(+ "true\ncontrol-openhands assert --field a --equals 1 \\\n --bug-shows 0 \\\n -- status # bug\ntrue",+ ),+ );+ assert.deepEqual(continued.problems, []);+ assert.deepEqual(continued.bugLines, [4]);+ assert.deepEqual(continued.bugRanges, [[2, 4]]);assert.match(knownBugBlocks("```sh known-bug F14.create\ncontrol-openhands status\n",@@ -1938,6 +1968,56 @@ test("an assertion tells the bug's value from another value and from an error",assert.equal(judge({ field: "nope" }).kind, "error");assert.equal(judge({ exitCode: 1 }).kind, "error");assert.equal(judge({ equals: undefined, matches: "^$" }).kind, "hold");+ // --bug clauses name the symptom on any field; all must hold.+ const api = {+ status: 500,+ response: { body: { exception: "'int' object has no attribute 'pop'" } },+ };+ const bug = (predicate) =>+ judgeAssertion({+ exitCode: 0,+ json: api,+ field: "status",+ equals: "422",+ bug: predicate,+ });+ const reproduced = bug(+ "status eq 500 && response.body.exception contains 'has no attribute'",+ );+ assert.equal(reproduced.kind, "mismatch");+ assert.deepEqual(+ reproduced.bugChecks.map((c) => c.ok),+ [true, true],+ );+ assert.equal(+ bug("status eq 500 && response.body.exception contains timeout").kind,+ "other-symptom",+ );+ assert.equal(bug("status eq 503").kind, "other-symptom");+ assert.equal(bug("response.body.exception matches ^'int'").kind, "mismatch");+ assert.equal(bug("response.body.detail missing").kind, "mismatch");+ assert.equal(bug("status ne 500").kind, "other-symptom");+});++test("bug predicates parse PATH OP [VALUE] clauses joined by &&", () => {+ assert.deepEqual(+ parseBugPredicate(+ `status eq 500 && body.error contains "no such file" && body.x exists`,+ ).clauses,+ [+ { path: "status", op: "eq", value: "500" },+ { path: "body.error", op: "contains", value: "no such file" },+ { path: "body.x", op: "exists" },+ ],+ );+ assert.deepEqual(parseBugPredicate("value eq ''").clauses, [+ { path: "value", op: "eq", value: "" },+ ]);+ assert.match(parseBugPredicate("value matches (").problems[0], /Invalid/);+ assert.match(+ parseBugPredicate("value exists 1").problems[0],+ /takes no value/,+ );});test("only a mismatch at the bug line on a healthy stack counts as reproducing", () => {@@ -1975,6 +2055,11 @@ test("only a mismatch at the bug line on a healthy stack counts as reproducing",);assert.match(c({ assertion: undefined }).reason, /no assertion result/);assert.match(c({ failedLine: null }).reason, /unknown/);+ // A continued marked command spans lines; bash reports its first one.+ const span = { bugLines: [6], bugRanges: [[4, 6]] };+ assert.deepEqual(c(span), { status: "xfail" });+ assert.match(c({ ...span, failedLine: 3 }).reason, /^setup: line 3/);+ assert.match(c({ ...span, failedLine: 7 }).reason, /after the `# bug`/);for (const o of [{ doctorBefore: false },{ doctorAfter: false },@@ -2089,6 +2174,22 @@ test("bug run classifies live attempts and bug proof turns two runs into a verdi/after the `# bug` assertion held/,],timeout: [`sleep 5\n${bugLine()}`, "fail", /^timeout$/, ["--timeout", "1"]],+ predicate: [`true\n${bugLine("--equals 1 --bug 'count eq 0'")}`, "xfail"],+ "predicate-symptom": [+ `true\n${bugLine("--equals 1 --bug 'count eq 0 && count ne 0'")}`,+ "fail",+ /^wrong symptom/,+ ],+ continued: [+ `true\ncontrol-openhands assert --field count \\\n --equals 1 --bug-shows 0 \\\n -- evidence list --feature QA # bug`,+ "xfail",+ ],+ // A cleanup trap's later assertions, passing or failing, do not replace+ // the marked command's observation.+ cleanup: [+ `trap 'control-openhands assert --field count --equals 0 -- evidence list --feature QA; control-openhands assert --field count --equals 5 -- evidence list --feature QA; false' EXIT\ntrue\n${bugLine()}`,+ "xfail",+ ],};for (const [name, [body, status, reason, extra = []]] of Object.entries(cases,diff --git a/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs b/.agents/skills/verify-openhands/scripts/lib/known-bug.mjsindex a9ae195217..57c9f20f59 100644--- a/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs+++ b/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs@@ -7,9 +7,12 @@// control-openhands assert --field enabled --equals true --bug-shows false -- browser enabled 'testid=submit-button' # bug// ```//-// whose commands assert the correct behavior. The one line ending in `# bug`-// is a `control-openhands assert` that names the value the bug shows-// (`--bug-shows`). `bug run` executes the block and classifies each attempt:+// whose commands assert the correct behavior. The one command ending in+// `# bug` (it may continue over lines ending in `\`) is a+// `control-openhands assert` that names what the bug shows: `--bug-shows V`+// for the asserted field's value, or `--bug 'PATH OP [VALUE] && ...'` for+// clauses on any fields of the command's output. `bug run` executes the+// block and classifies each attempt:// the bug reproduced (xfail) only when that assertion is the command that// failed and it saw the bug's own value; the correct behavior held (xpass);// anything else (setup, timeout, an unhealthy stack, another symptom) is a@@ -18,10 +21,92 @@ import { createHash } from "node:crypto";const OPEN = /^(\s*)```sh known-bug (F\d{2}\.[a-z0-9-]+)(?: ([a-z0-9-]+))?\s*$/;const BUG_LINE = /#\s*bug\s*$/;+const BUG_OPS = [+ "eq",+ "ne",+ "contains",+ "not-contains",+ "matches",+ "exists",+ "missing",+];+const CLAUSE = new RegExp(`^(\\S+)\\s+(${BUG_OPS.join("|")})(?:\\s+(.*))?$`);+const unquote = (v) =>+ /^(['"]).*\1$/s.test(v) && v.length >= 2 ? v.slice(1, -1) : v;++/**+ * A `--bug` predicate: `PATH OP [VALUE]` clauses joined by `&&`, the grammar+ * the sibling verification skills share. OP is one of eq, ne, contains,+ * not-contains, matches (a regex), exists, missing; a VALUE with spaces may+ * be quoted.+ * @returns {{ clauses: {path: string, op: string, value?: string}[], problems: string[] }}+ */+export function parseBugPredicate(text) {+ const clauses = [];+ const problems = [];+ for (const raw of String(text).split("&&")) {+ const part = raw.trim();+ const m = CLAUSE.exec(part);+ if (!m) {+ problems.push(+ `\`${part}\` is not PATH OP [VALUE] with OP in ${BUG_OPS.join(", ")}`,+ );+ continue;+ }+ const [, path, op, value] = m;+ const takesValue = op !== "exists" && op !== "missing";+ if (takesValue && value === undefined) {+ problems.push(`\`${part}\` needs a value (quote an empty one: '')`);+ continue;+ }+ if (!takesValue && value !== undefined) {+ problems.push(`\`${part}\`: ${op} takes no value`);+ continue;+ }+ const clause = takesValue+ ? { path, op, value: unquote(value.trim()) }+ : { path, op };+ if (op === "matches")+ try {+ new RegExp(clause.value);+ } catch (e) {+ problems.push(`\`${part}\`: ${e.message}`);+ continue;+ }+ clauses.push(clause);+ }+ return { clauses, problems };+}++/** Whether one clause holds on a command's JSON output. */+export function clauseHolds(json, { path, op, value }) {+ const actual = fieldValue(json, path);+ if (op === "exists") return actual !== undefined;+ if (op === "missing") return actual === undefined;+ if (actual === undefined) return false;+ if (op === "eq" || op === "ne") {+ const same =+ actual === value ||+ (value.trim() !== "" &&+ actual.trim() !== "" &&+ Number.isFinite(Number(actual)) &&+ Number(actual) === Number(value));+ return op === "eq" ? same : !same;+ }+ if (op === "contains") return actual.includes(value);+ if (op === "not-contains") return !actual.includes(value);+ return new RegExp(value).test(actual);+}++// The value of a `--bug` flag on a recipe line, for static checks.+function bugFlag(command) {+ const m = /--bug(?:=|\s+)('[^']*'|"[^"]*"|\S+)/.exec(command);+ return m ? unquote(m[1]) : undefined;+}/*** Every known-bug block in a map file's text.- * @returns {{ id: string, name: string, line: number, script: string, bugLines: number[], sha: string, problems: string[] }[]}+ * @returns {{ id: string, name: string, line: number, script: string, bugLines: number[], bugRanges: [number, number][], sha: string, problems: string[] }[]}*/export function knownBugBlocks(text) {const lines = text.split("\n");@@ -47,16 +132,32 @@ export function knownBugBlocks(text) {problems.push(`needs exactly one line ending in \`# bug\` (found ${bugLines.length})`,);- for (const n of bugLines) {- const l = body[n - 1];- if (!/^control-openhands assert\b/.test(l.trim()))+ // A command continued with `\` starts at the first line of the+ // continuation; bash reports that line when it fails.+ const bugRanges = bugLines.map((n) => {+ let start = n;+ while (start > 1 && /\\$/.test(body[start - 2])) start -= 1;+ return [start, n];+ });+ for (const [start, n] of bugRanges) {+ const command = body+ .slice(start - 1, n)+ .map((l) => l.replace(/\\$/, ""))+ .join(" ");+ if (!/^control-openhands assert\b/.test(command.trim())) {problems.push(`line ${n}: the \`# bug\` line must be a \`control-openhands assert\``,);- else if (!/--bug-shows\b/.test(l))+ continue;+ }+ const predicate = bugFlag(command);+ if (predicate === undefined && !/--bug-shows\b/.test(command))problems.push(- `line ${n}: the \`# bug\` assertion must name the bug's value with --bug-shows`,+ `line ${n}: the \`# bug\` assertion must name what the bug shows with --bug-shows or --bug`,);+ else if (predicate !== undefined)+ for (const p of parseBugPredicate(predicate).problems)+ problems.push(`line ${n}: --bug ${p}`);}const script = body.join("\n");blocks.push({@@ -65,6 +166,7 @@ export function knownBugBlocks(text) {line: i + 1,script,bugLines,+ bugRanges,sha: createHash("sha256").update(script).digest("hex").slice(0, 16),problems,});@@ -89,9 +191,10 @@ export function fieldValue(json, path) {/*** What an assertion saw.- * kind: hold | mismatch (the bug's own value, or any mismatch when no- * --bug-shows is given) | other-symptom (a mismatch, but not the bug's value)- * | error (the command failed or printed no such field).+ * kind: hold | mismatch (the bug's own symptom: every --bug-shows/--bug+ * clause holds, or any mismatch when none is given) | other-symptom (a+ * mismatch, but not the bug's) | error (the command failed or printed no such+ * field).*/export function judgeAssertion({exitCode,@@ -100,6 +203,7 @@ export function judgeAssertion({equals,matches,bugShows,+ bug,}) {if (exitCode !== 0 || !json)return { kind: "error", reason: `the command exited ${exitCode}` };@@ -111,9 +215,23 @@ export function judgeAssertion({? actual === String(equals): new RegExp(matches).test(actual);if (holds) return { kind: "hold", actual };- if (bugShows !== undefined && actual !== String(bugShows))- return { kind: "other-symptom", actual };- return { kind: "mismatch", actual };+ const clauses = [+ ...(bugShows !== undefined+ ? [{ path: field, op: "eq", value: String(bugShows) }]+ : []),+ ...(bug !== undefined ? parseBugPredicate(bug).clauses : []),+ ];+ if (!clauses.length) return { kind: "mismatch", actual };+ const bugChecks = clauses.map((c) => ({+ clause: `${c.path} ${c.op}${c.value === undefined ? "" : ` ${JSON.stringify(c.value)}`}`,+ actual: fieldValue(json, c.path),+ ok: clauseHolds(json, c),+ }));+ return {+ kind: bugChecks.every((c) => c.ok) ? "mismatch" : "other-symptom",+ actual,+ bugChecks,+ };}/**@@ -125,7 +243,8 @@ export function judgeAssertion({* @param {boolean} a.timedOut* @param {number|null} a.failedLine 1-based line of the block that failed* @param {number[]} a.bugLines- * @param {{kind: string, line?: number}|undefined} a.assertion the last assertion the attempt made+ * @param {[number, number][]} [a.bugRanges] the marked command's first and last line+ * @param {{kind: string, line?: number}|undefined} a.assertion the assertion logged at the marked command* @returns {{ status: "xfail"|"xpass"|"fail", reason?: string }}*/export function classifyAttempt(a) {@@ -138,13 +257,13 @@ export function classifyAttempt(a) {if (a.exitCode === 0) return { status: "xpass" };if (a.failedLine == null)return { status: "fail", reason: "the failing line is unknown" };- const bugLine = a.bugLines[0];- if (a.failedLine < bugLine)+ const [start, end] = a.bugRanges?.[0] ?? [a.bugLines[0], a.bugLines[0]];+ if (a.failedLine < start)return {status: "fail",reason: `setup: line ${a.failedLine} failed before the \`# bug\` line`,};- if (a.failedLine > bugLine)+ if (a.failedLine > end)return {status: "fail",reason: `line ${a.failedLine} failed after the \`# bug\` assertion held`,