Skip to content

πŸ™Œ OpenHands: AI-Driven Development

feat(verify-openhands): adopt the shared known-bug contract (#18355)

Engel Nyst0e9ce191390f
diff --git a/.agents/skills/verify-openhands/references/mapping.md b/.agents/skills/verify-openhands/references/mapping.md
index aa566684c9..57d5c3939d 100644
--- a/.agents/skills/verify-openhands/references/mapping.md
+++ b/.agents/skills/verify-openhands/references/mapping.md
@@ -128,15 +128,18 @@ sections, in this order:
under it, opened with `` ```sh known-bug <ID> [name] `` (the name tells
entry points or viewports apart, such as `desktop` and `phone`). Its
commands set their own viewport and starting page and assert the *correct*
- behavior; exactly one line, the one that fails because of the bug, is a
+ behavior; exactly one command, the one that fails because of the bug, is a
`control-openhands assert --field <path> --equals <correct> --bug-shows
- <what the bug shows> -- <command>` ending in `# bug`. Arrange steps go
+ <what the bug shows> -- <command>` ending in `# bug` (a long one may
+ continue over lines ending in `\`). When the bug's value varies, or
+ needs more than one field to pin down, name it with `--bug 'PATH OP
+ [VALUE] && ...'` instead (see the known-bug contract below). Arrange steps go
before it and must be able to fail on their own (assert the starting value
too), so a broken setup is never read as the bug. Keep a `browser
screenshot` before the assertion where the symptom is visual.
`control-openhands bug run <ID>` then reports `xfail` while the bug lasts,
`xpass` once it stops, and `fail` for anything else; `map check` rejects a
- block without exactly one such line. F25's `F25.selector-dropdown` note is
+ block without exactly one such command. F25's `F25.selector-dropdown` note is
the worked example. When a fix is proven, the PR that drops the note turns
the block's commands into the bullet's normal steps, so the same assertion
guards the corrected behavior.
@@ -232,3 +235,43 @@ parallel, each edits only its own file and checks it with `--file`; one
coordinator updates the index afterwards.
Then hand the evidence ledger (`control-openhands evidence report`) to whoever
asked, with fail/blocked rows first.
+
+## The known-bug contract
+
+Keep these rules in step with the sibling verification skills
+(verify-agent-server in OpenHands/software-agent-sdk, verify-automation in
+OpenHands/automation), so a proof means the same thing in each. The code stays
+separate; only the syntax for naming the bug differs, noted below.
+
+- The recipe asserts the correct behavior. The command that fails because of
+ the bug ends its line with `# bug` and names the bug's own symptom: the
+ value or error it produces, not just any wrong value. Here that command is
+ always a `control-openhands assert`, which carries the symptom as
+ `--bug-shows V` or `--bug PREDICATE`; the sibling skills write
+ `# bug: PREDICATE` on the marked line instead.
+- Three outcomes. `xfail`: the marked command failed and its symptom held (the
+ documented bug reproduced). `xpass`: every command passed, so the correct
+ behavior was observed; `bug run` exits 1 so the note gets reviewed and the
+ marker removed. `fail`: anything else, including a different value at the
+ marked command, an arrange step that breaks first, a timeout, an unhealthy
+ stack before or after, a failure after the marked command, or a failing line
+ the runner could not locate.
+- The predicate is `PATH OP [VALUE]` clauses joined by `&&`, matched against
+ the JSON the asserted command printed: `--bug 'status eq 500 &&
+ body.error contains "has no attribute"'`. OP is `eq`, `ne`, `contains`,
+ `not-contains`, `matches` (a regex), `exists` or `missing`. `--bug-shows V`
+ is `--bug '<field> eq V'`. Prefer `contains` or `matches` on the stable part
+ of an error message over pinning its whole text.
+- `xpass` must mean the correct value was observed, so the marked command
+ asserts positively. Negative shapes (`! cmd`, `if cmd; then false; fi`,
+ `test !`) cannot be marked: the marked command must be an `assert`, and
+ `map check` rejects anything else.
+- `map check` rejects a missing or malformed predicate.
+- The observation is the marked command's own. Every assertion is logged with
+ the recipe line that ran it and the first failure is kept, so a later step
+ or an `EXIT` trap's cleanup cannot replace it. A marked command continued
+ over several lines counts as one, from its first line to the `# bug` line.
+- A fix is proven with the baseline reproducing (`xfail`) twice and the fix
+ passing (`xpass`) twice, with the same recipe, viewport and backend:
+ `control-openhands bug proof` computes that verdict (`verified`,
+ `not-fixed` or `inconclusive`).
diff --git a/.agents/skills/verify-openhands/scripts/control-openhands.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.mjs
index 0488b11f61..ed27a53b0c 100755
--- a/.agents/skills/verify-openhands/scripts/control-openhands.mjs
+++ b/.agents/skills/verify-openhands/scripts/control-openhands.mjs
@@ -38,6 +38,7 @@ import {
classifyAttempt,
judgeAssertion,
knownBugBlocks,
+ parseBugPredicate,
proofVerdict,
} from "./lib/known-bug.mjs";
import { routePattern } from "./lib/route-pattern.mjs";
@@ -3398,6 +3399,16 @@ async function cmdAssert({ positional, flags }) {
"assert needs --field PATH, one of --equals VALUE or --matches REGEX, and -- <control-openhands command>",
"control-openhands assert --field enabled --equals true --bug-shows false -- browser enabled 'testid=submit-button'",
);
+ const bug =
+ flags.bug === undefined || flags.bug === true
+ ? undefined
+ : String(flags.bug);
+ const bugProblems = bug === undefined ? [] : parseBugPredicate(bug).problems;
+ if (bugProblems.length)
+ usage(
+ `--bug ${bugProblems.join("; ")}`,
+ `control-openhands assert --field status --equals 200 --bug 'status eq 500 && body.error contains "has no attribute"' -- api GET /api/...`,
+ );
const env = { ...process.env };
if (flags.run && flags.run !== true) env.OH_VERIFY_RUN = resolve(flags.run);
const child = runSelf(positional.map(String), { env });
@@ -3411,6 +3422,7 @@ async function cmdAssert({ positional, flags }) {
flags["bug-shows"] === undefined || flags["bug-shows"] === true
? undefined
: String(flags["bug-shows"]),
+ bug,
});
const result = {
ok: judged.kind === "hold",
@@ -3418,18 +3430,20 @@ async function cmdAssert({ positional, flags }) {
field: flags.field,
expected: hasEquals ? String(flags.equals) : `/${flags.matches}/`,
bugShows: flags["bug-shows"] === true ? undefined : flags["bug-shows"],
+ bug,
command: positional.join(" "),
output:
judged.kind === "error"
? (child.json ?? child.stdout.slice(-2000))
: undefined,
};
- // Under `bug run` every assertion is logged, so the runner can tell the
- // marked assertion's mismatch from an error at the same line.
+ // Under `bug run` every assertion is logged with the recipe line that ran
+ // it, so the runner reads the marked command's own result, never one a
+ // later cleanup step logged.
if (process.env.OH_VERIFY_ASSERT_LOG)
appendFileSync(
process.env.OH_VERIFY_ASSERT_LOG,
- `${JSON.stringify({ ts: new Date().toISOString(), ...result, output: undefined })}\n`,
+ `${JSON.stringify({ ts: new Date().toISOString(), line: process.env.OH_VERIFY_LINE, ...result, output: undefined })}\n`,
);
out(result);
if (judged.kind === "error") process.exitCode = child.code || 1;
@@ -3475,10 +3489,15 @@ function selectBlocks(id, flags) {
}
function runBlock(run, block, { timeoutMs, logFile, failureFile, scriptFile }) {
+ // The DEBUG trap tags every command (and the asserts it runs) with its
+ // line. The ERR trap keeps the first failure only, and retags what runs
+ // after it (an EXIT trap's cleanup), so neither overwrites the observation.
const preamble = [
"set -eo pipefail",
`cd ${JSON.stringify(repoRoot)}`,
- `trap 'printf "%s\\t%s\\n" "$LINENO" "$BASH_COMMAND" > ${JSON.stringify(failureFile)}' ERR`,
+ "export OH_VERIFY_LINE=",
+ "trap 'OH_VERIFY_LINE=$LINENO' DEBUG",
+ `trap '[ -e ${JSON.stringify(failureFile)} ] || printf "%s\\t%s\\n" "$LINENO" "$BASH_COMMAND" > ${JSON.stringify(failureFile)}; OH_VERIFY_LINE=after-failure' ERR`,
];
writeFileSync(scriptFile, `${preamble.join("\n")}\n${block.script}\n`);
const env = {
@@ -3521,6 +3540,7 @@ function runBlock(run, block, { timeoutMs, logFile, failureFile, scriptFile }) {
failedCommand = cmd.join("\t").slice(0, 500);
}
done({
+ lineOffset: preamble.length,
exitCode: timedOut ? null : code,
timedOut,
failedLine,
@@ -3600,8 +3620,16 @@ async function cmdBug({ positional, flags }) {
failureFile,
scriptFile: join(work, `${stamp}.sh`),
});
- const asserts = readJsonl(logFile);
- const assertion = asserts.at(-1);
+ const asserts = readJsonl(logFile).map((a) => ({
+ ...a,
+ line: /^\d+$/.test(a.line ?? "")
+ ? Number(a.line) - exec.lineOffset
+ : a.line,
+ }));
+ const [start, end] = block.bugRanges[0];
+ const assertion = asserts
+ .filter((a) => a.line >= start && a.line <= end)
+ .at(-1);
const doctorAfter =
exec.exitCode !== 0 && doctorBefore !== false ? doctor() : undefined;
const { status, reason } = classifyAttempt({
@@ -3611,6 +3639,7 @@ async function cmdBug({ positional, flags }) {
timedOut: exec.timedOut,
failedLine: exec.failedLine,
bugLines: block.bugLines,
+ bugRanges: block.bugRanges,
assertion,
});
const page = daemonInfo(run)
@@ -3653,6 +3682,8 @@ async function cmdBug({ positional, flags }) {
expected: assertion.expected,
actual: assertion.actual,
bugShows: assertion.bugShows,
+ bug: assertion.bug,
+ bugChecks: assertion.bugChecks,
}
: undefined,
doctorBefore,
@@ -4514,7 +4545,7 @@ Drive and observe
Evidence and map
evidence add | list | report β€” the run's pass/fail/blocked/not-run ledger
- assert --field F --equals V [--bug-shows B] -- <command> β€” check one value of a command's output
+ assert --field F --equals V [--bug-shows B | --bug P] -- <command> β€” check one value of a command's output
bug list | run ID --attempts 2 | proof ID β€” known bugs as executable expected failures
map check | coverage | ids | routes | affected | testids | baseline β€” keep the feature map honest
@@ -4764,15 +4795,20 @@ first (a daily pass against yesterday's run). --json prints the same data.
Rows from \`bug run\` add xfail (a known bug reproduced) and xpass (it stopped
reproducing); xpass rows sort right after fail rows.
`,
- assert: `control-openhands assert --field PATH (--equals VALUE | --matches REGEX) [--bug-shows VALUE] -- <control-openhands command...>
+ assert: `control-openhands assert --field PATH (--equals VALUE | --matches REGEX) [--bug-shows VALUE] [--bug PREDICATE] -- <control-openhands command...>
Runs the command, reads PATH (dotted, e.g. value, viewport.width) from its JSON
output and compares it: strings as they are, other values as JSON. Exit 0 when
it holds; 1 on a mismatch; the command's own code when it fails or has no such
-field (kind "error": no value was compared). --bug-shows names the value a known
-bug produces: a mismatch with any other value is kind "other-symptom".
+field (kind "error": no value was compared). --bug-shows and --bug name what a
+known bug produces; a mismatch that does not match them is kind "other-symptom".
+--bug-shows V is short for "--bug 'PATH eq V'". --bug takes "PATH OP [VALUE]"
+clauses joined by "&&", on any field of the output, with OP one of eq, ne,
+contains, not-contains, matches (regex), exists, missing; quote a VALUE with
+spaces. The result lists each clause under bugChecks.
control-openhands assert --field enabled --equals true -- browser enabled 'testid=submit-button'
+ control-openhands assert --field text --equals Saved --bug 'text contains "An error occurred"' -- browser text 'testid=toast'
`,
bug: `control-openhands bug list [--file PATH]
control-openhands bug run ID [--name N] [--attempts N] [--label L] [--timeout SEC] [--no-doctor] [--file PATH]
@@ -4780,11 +4816,12 @@ control-openhands bug proof ID --baseline RUN_DIR --fix RUN_DIR [--baseline-labe
A known bug in the map can carry a fenced block, opened with
"\`\`\`sh known-bug <ID> [name]", whose commands assert the correct behavior. Its one
-line ending in "# bug" is a "control-openhands assert ... --bug-shows <value>".
+command ending in "# bug" (continued lines ending in "\\" count as one) is a
+"control-openhands assert ... --bug-shows <value>" or "... --bug <predicate>".
'run' executes the block in bash (set -eo pipefail) against the run, each
attempt after a passing doctor, and classifies it:
- xfail the "# bug" assertion failed and saw the bug's value (reproduced)
+ xfail the "# bug" assertion failed and saw the bug's symptom (reproduced)
xpass every command held (the bug did not reproduce: review the map note)
fail anything else: doctor failed before (setup) or after (backend), a
timeout, an earlier line (setup) or a later one, the bug line's command
@@ -4792,6 +4829,9 @@ attempt after a passing doctor, and classifies it:
Each attempt is appended to evidence/known-bugs/attempts.jsonl (revision,
backend, viewport, recipe hash, failing line, assertion, transcript) and to the
ledger as an xfail/xpass/fail row. Exit 0 only when every attempt is xfail.
+Each assertion is logged with the recipe line that ran it, and the first
+failure is the one kept, so a cleanup step (an EXIT trap) cannot replace the
+marked command's observation.
'proof' compares the attempts of the current recipe on a baseline run and a fix
run: verified only when the baseline has at least two attempts, all xfail, the
diff --git a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs
index 6825643183..5dd9170713 100755
--- a/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs
+++ b/.agents/skills/verify-openhands/scripts/control-openhands.test.mjs
@@ -57,6 +57,7 @@ import {
classifyAttempt,
judgeAssertion,
knownBugBlocks,
+ parseBugPredicate,
proofVerdict,
} from "./lib/known-bug.mjs";
@@ -1909,8 +1910,37 @@ test("known-bug blocks parse with their bug line, and malformed ones say why", (
);
assert.match(
problems("control-openhands assert --field a --equals 1 -- status # bug"),
- /must name the bug's value with --bug-shows/,
+ /must name what the bug shows with --bug-shows or --bug/,
);
+ assert.match(
+ problems(
+ "control-openhands assert --field a --equals 1 --bug 'a is 0' -- status # bug",
+ ),
+ /--bug `a is 0` is not PATH OP \[VALUE\]/,
+ );
+ assert.match(
+ problems(
+ "control-openhands assert --field a --equals 1 --bug 'a eq' -- status # bug",
+ ),
+ /needs a value/,
+ );
+ assert.deepEqual(
+ knownBugBlocks(
+ block(
+ "control-openhands assert --field a --equals 1 --bug 'a eq 0 && b.c contains \"no such\"' -- status # bug",
+ ),
+ )[0].problems,
+ [],
+ );
+ // A command continued over several lines is one marked command.
+ const [continued] = knownBugBlocks(
+ block(
+ "true\ncontrol-openhands assert --field a --equals 1 \\\n --bug-shows 0 \\\n -- status # bug\ntrue",
+ ),
+ );
+ assert.deepEqual(continued.problems, []);
+ assert.deepEqual(continued.bugLines, [4]);
+ assert.deepEqual(continued.bugRanges, [[2, 4]]);
assert.match(
knownBugBlocks(
"```sh known-bug F14.create\ncontrol-openhands status\n",
@@ -1938,6 +1968,56 @@ test("an assertion tells the bug's value from another value and from an error",
assert.equal(judge({ field: "nope" }).kind, "error");
assert.equal(judge({ exitCode: 1 }).kind, "error");
assert.equal(judge({ equals: undefined, matches: "^$" }).kind, "hold");
+ // --bug clauses name the symptom on any field; all must hold.
+ const api = {
+ status: 500,
+ response: { body: { exception: "'int' object has no attribute 'pop'" } },
+ };
+ const bug = (predicate) =>
+ judgeAssertion({
+ exitCode: 0,
+ json: api,
+ field: "status",
+ equals: "422",
+ bug: predicate,
+ });
+ const reproduced = bug(
+ "status eq 500 && response.body.exception contains 'has no attribute'",
+ );
+ assert.equal(reproduced.kind, "mismatch");
+ assert.deepEqual(
+ reproduced.bugChecks.map((c) => c.ok),
+ [true, true],
+ );
+ assert.equal(
+ bug("status eq 500 && response.body.exception contains timeout").kind,
+ "other-symptom",
+ );
+ assert.equal(bug("status eq 503").kind, "other-symptom");
+ assert.equal(bug("response.body.exception matches ^'int'").kind, "mismatch");
+ assert.equal(bug("response.body.detail missing").kind, "mismatch");
+ assert.equal(bug("status ne 500").kind, "other-symptom");
+});
+
+test("bug predicates parse PATH OP [VALUE] clauses joined by &&", () => {
+ assert.deepEqual(
+ parseBugPredicate(
+ `status eq 500 && body.error contains "no such file" && body.x exists`,
+ ).clauses,
+ [
+ { path: "status", op: "eq", value: "500" },
+ { path: "body.error", op: "contains", value: "no such file" },
+ { path: "body.x", op: "exists" },
+ ],
+ );
+ assert.deepEqual(parseBugPredicate("value eq ''").clauses, [
+ { path: "value", op: "eq", value: "" },
+ ]);
+ assert.match(parseBugPredicate("value matches (").problems[0], /Invalid/);
+ assert.match(
+ parseBugPredicate("value exists 1").problems[0],
+ /takes no value/,
+ );
});
test("only a mismatch at the bug line on a healthy stack counts as reproducing", () => {
@@ -1975,6 +2055,11 @@ test("only a mismatch at the bug line on a healthy stack counts as reproducing",
);
assert.match(c({ assertion: undefined }).reason, /no assertion result/);
assert.match(c({ failedLine: null }).reason, /unknown/);
+ // A continued marked command spans lines; bash reports its first one.
+ const span = { bugLines: [6], bugRanges: [[4, 6]] };
+ assert.deepEqual(c(span), { status: "xfail" });
+ assert.match(c({ ...span, failedLine: 3 }).reason, /^setup: line 3/);
+ assert.match(c({ ...span, failedLine: 7 }).reason, /after the `# bug`/);
for (const o of [
{ doctorBefore: false },
{ doctorAfter: false },
@@ -2089,6 +2174,22 @@ test("bug run classifies live attempts and bug proof turns two runs into a verdi
/after the `# bug` assertion held/,
],
timeout: [`sleep 5\n${bugLine()}`, "fail", /^timeout$/, ["--timeout", "1"]],
+ predicate: [`true\n${bugLine("--equals 1 --bug 'count eq 0'")}`, "xfail"],
+ "predicate-symptom": [
+ `true\n${bugLine("--equals 1 --bug 'count eq 0 && count ne 0'")}`,
+ "fail",
+ /^wrong symptom/,
+ ],
+ continued: [
+ `true\ncontrol-openhands assert --field count \\\n --equals 1 --bug-shows 0 \\\n -- evidence list --feature QA # bug`,
+ "xfail",
+ ],
+ // A cleanup trap's later assertions, passing or failing, do not replace
+ // the marked command's observation.
+ cleanup: [
+ `trap 'control-openhands assert --field count --equals 0 -- evidence list --feature QA; control-openhands assert --field count --equals 5 -- evidence list --feature QA; false' EXIT\ntrue\n${bugLine()}`,
+ "xfail",
+ ],
};
for (const [name, [body, status, reason, extra = []]] of Object.entries(
cases,
diff --git a/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs b/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs
index a9ae195217..57c9f20f59 100644
--- a/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs
+++ b/.agents/skills/verify-openhands/scripts/lib/known-bug.mjs
@@ -7,9 +7,12 @@
// control-openhands assert --field enabled --equals true --bug-shows false -- browser enabled 'testid=submit-button' # bug
// ```
//
-// whose commands assert the correct behavior. The one line ending in `# bug`
-// is a `control-openhands assert` that names the value the bug shows
-// (`--bug-shows`). `bug run` executes the block and classifies each attempt:
+// whose commands assert the correct behavior. The one command ending in
+// `# bug` (it may continue over lines ending in `\`) is a
+// `control-openhands assert` that names what the bug shows: `--bug-shows V`
+// for the asserted field's value, or `--bug 'PATH OP [VALUE] && ...'` for
+// clauses on any fields of the command's output. `bug run` executes the
+// block and classifies each attempt:
// the bug reproduced (xfail) only when that assertion is the command that
// failed and it saw the bug's own value; the correct behavior held (xpass);
// anything else (setup, timeout, an unhealthy stack, another symptom) is a
@@ -18,10 +21,92 @@ import { createHash } from "node:crypto";
const OPEN = /^(\s*)```sh known-bug (F\d{2}\.[a-z0-9-]+)(?: ([a-z0-9-]+))?\s*$/;
const BUG_LINE = /#\s*bug\s*$/;
+const BUG_OPS = [
+ "eq",
+ "ne",
+ "contains",
+ "not-contains",
+ "matches",
+ "exists",
+ "missing",
+];
+const CLAUSE = new RegExp(`^(\\S+)\\s+(${BUG_OPS.join("|")})(?:\\s+(.*))?$`);
+const unquote = (v) =>
+ /^(['"]).*\1$/s.test(v) && v.length >= 2 ? v.slice(1, -1) : v;
+
+/**
+ * A `--bug` predicate: `PATH OP [VALUE]` clauses joined by `&&`, the grammar
+ * the sibling verification skills share. OP is one of eq, ne, contains,
+ * not-contains, matches (a regex), exists, missing; a VALUE with spaces may
+ * be quoted.
+ * @returns {{ clauses: {path: string, op: string, value?: string}[], problems: string[] }}
+ */
+export function parseBugPredicate(text) {
+ const clauses = [];
+ const problems = [];
+ for (const raw of String(text).split("&&")) {
+ const part = raw.trim();
+ const m = CLAUSE.exec(part);
+ if (!m) {
+ problems.push(
+ `\`${part}\` is not PATH OP [VALUE] with OP in ${BUG_OPS.join(", ")}`,
+ );
+ continue;
+ }
+ const [, path, op, value] = m;
+ const takesValue = op !== "exists" && op !== "missing";
+ if (takesValue && value === undefined) {
+ problems.push(`\`${part}\` needs a value (quote an empty one: '')`);
+ continue;
+ }
+ if (!takesValue && value !== undefined) {
+ problems.push(`\`${part}\`: ${op} takes no value`);
+ continue;
+ }
+ const clause = takesValue
+ ? { path, op, value: unquote(value.trim()) }
+ : { path, op };
+ if (op === "matches")
+ try {
+ new RegExp(clause.value);
+ } catch (e) {
+ problems.push(`\`${part}\`: ${e.message}`);
+ continue;
+ }
+ clauses.push(clause);
+ }
+ return { clauses, problems };
+}
+
+/** Whether one clause holds on a command's JSON output. */
+export function clauseHolds(json, { path, op, value }) {
+ const actual = fieldValue(json, path);
+ if (op === "exists") return actual !== undefined;
+ if (op === "missing") return actual === undefined;
+ if (actual === undefined) return false;
+ if (op === "eq" || op === "ne") {
+ const same =
+ actual === value ||
+ (value.trim() !== "" &&
+ actual.trim() !== "" &&
+ Number.isFinite(Number(actual)) &&
+ Number(actual) === Number(value));
+ return op === "eq" ? same : !same;
+ }
+ if (op === "contains") return actual.includes(value);
+ if (op === "not-contains") return !actual.includes(value);
+ return new RegExp(value).test(actual);
+}
+
+// The value of a `--bug` flag on a recipe line, for static checks.
+function bugFlag(command) {
+ const m = /--bug(?:=|\s+)('[^']*'|"[^"]*"|\S+)/.exec(command);
+ return m ? unquote(m[1]) : undefined;
+}
/**
* Every known-bug block in a map file's text.
- * @returns {{ id: string, name: string, line: number, script: string, bugLines: number[], sha: string, problems: string[] }[]}
+ * @returns {{ id: string, name: string, line: number, script: string, bugLines: number[], bugRanges: [number, number][], sha: string, problems: string[] }[]}
*/
export function knownBugBlocks(text) {
const lines = text.split("\n");
@@ -47,16 +132,32 @@ export function knownBugBlocks(text) {
problems.push(
`needs exactly one line ending in \`# bug\` (found ${bugLines.length})`,
);
- for (const n of bugLines) {
- const l = body[n - 1];
- if (!/^control-openhands assert\b/.test(l.trim()))
+ // A command continued with `\` starts at the first line of the
+ // continuation; bash reports that line when it fails.
+ const bugRanges = bugLines.map((n) => {
+ let start = n;
+ while (start > 1 && /\\$/.test(body[start - 2])) start -= 1;
+ return [start, n];
+ });
+ for (const [start, n] of bugRanges) {
+ const command = body
+ .slice(start - 1, n)
+ .map((l) => l.replace(/\\$/, ""))
+ .join(" ");
+ if (!/^control-openhands assert\b/.test(command.trim())) {
problems.push(
`line ${n}: the \`# bug\` line must be a \`control-openhands assert\``,
);
- else if (!/--bug-shows\b/.test(l))
+ continue;
+ }
+ const predicate = bugFlag(command);
+ if (predicate === undefined && !/--bug-shows\b/.test(command))
problems.push(
- `line ${n}: the \`# bug\` assertion must name the bug's value with --bug-shows`,
+ `line ${n}: the \`# bug\` assertion must name what the bug shows with --bug-shows or --bug`,
);
+ else if (predicate !== undefined)
+ for (const p of parseBugPredicate(predicate).problems)
+ problems.push(`line ${n}: --bug ${p}`);
}
const script = body.join("\n");
blocks.push({
@@ -65,6 +166,7 @@ export function knownBugBlocks(text) {
line: i + 1,
script,
bugLines,
+ bugRanges,
sha: createHash("sha256").update(script).digest("hex").slice(0, 16),
problems,
});
@@ -89,9 +191,10 @@ export function fieldValue(json, path) {
/**
* What an assertion saw.
- * kind: hold | mismatch (the bug's own value, or any mismatch when no
- * --bug-shows is given) | other-symptom (a mismatch, but not the bug's value)
- * | error (the command failed or printed no such field).
+ * kind: hold | mismatch (the bug's own symptom: every --bug-shows/--bug
+ * clause holds, or any mismatch when none is given) | other-symptom (a
+ * mismatch, but not the bug's) | error (the command failed or printed no such
+ * field).
*/
export function judgeAssertion({
exitCode,
@@ -100,6 +203,7 @@ export function judgeAssertion({
equals,
matches,
bugShows,
+ bug,
}) {
if (exitCode !== 0 || !json)
return { kind: "error", reason: `the command exited ${exitCode}` };
@@ -111,9 +215,23 @@ export function judgeAssertion({
? actual === String(equals)
: new RegExp(matches).test(actual);
if (holds) return { kind: "hold", actual };
- if (bugShows !== undefined && actual !== String(bugShows))
- return { kind: "other-symptom", actual };
- return { kind: "mismatch", actual };
+ const clauses = [
+ ...(bugShows !== undefined
+ ? [{ path: field, op: "eq", value: String(bugShows) }]
+ : []),
+ ...(bug !== undefined ? parseBugPredicate(bug).clauses : []),
+ ];
+ if (!clauses.length) return { kind: "mismatch", actual };
+ const bugChecks = clauses.map((c) => ({
+ clause: `${c.path} ${c.op}${c.value === undefined ? "" : ` ${JSON.stringify(c.value)}`}`,
+ actual: fieldValue(json, c.path),
+ ok: clauseHolds(json, c),
+ }));
+ return {
+ kind: bugChecks.every((c) => c.ok) ? "mismatch" : "other-symptom",
+ actual,
+ bugChecks,
+ };
}
/**
@@ -125,7 +243,8 @@ export function judgeAssertion({
* @param {boolean} a.timedOut
* @param {number|null} a.failedLine 1-based line of the block that failed
* @param {number[]} a.bugLines
- * @param {{kind: string, line?: number}|undefined} a.assertion the last assertion the attempt made
+ * @param {[number, number][]} [a.bugRanges] the marked command's first and last line
+ * @param {{kind: string, line?: number}|undefined} a.assertion the assertion logged at the marked command
* @returns {{ status: "xfail"|"xpass"|"fail", reason?: string }}
*/
export function classifyAttempt(a) {
@@ -138,13 +257,13 @@ export function classifyAttempt(a) {
if (a.exitCode === 0) return { status: "xpass" };
if (a.failedLine == null)
return { status: "fail", reason: "the failing line is unknown" };
- const bugLine = a.bugLines[0];
- if (a.failedLine < bugLine)
+ const [start, end] = a.bugRanges?.[0] ?? [a.bugLines[0], a.bugLines[0]];
+ if (a.failedLine < start)
return {
status: "fail",
reason: `setup: line ${a.failedLine} failed before the \`# bug\` line`,
};
- if (a.failedLine > bugLine)
+ if (a.failedLine > end)
return {
status: "fail",
reason: `line ${a.failedLine} failed after the \`# bug\` assertion held`,