Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
test(e2e): fix the routine gaps its first full run found (#21645)
zchpeter762e4ed5eae0
diff --git a/.claude/skills/e2e-routine/ROUTINE_PROMPT.md b/.claude/skills/e2e-routine/ROUTINE_PROMPT.mdindex bf6157a9ab..0438b04741 100644--- a/.claude/skills/e2e-routine/ROUTINE_PROMPT.md+++ b/.claude/skills/e2e-routine/ROUTINE_PROMPT.md@@ -21,15 +21,6 @@ Policy for this task:- Finish with Phase 6 even when an earlier phase failed, and end your final message with the report.```-## Bootstrap prompt (until the skill is on `main`)--The cloud clone starts on `main`. While the skill lives on a branch, prepend-this to the prompt and drop it once the PR merges:--```-First: git fetch origin worktree-e2e-routine && git checkout origin/worktree-e2e-routine -- .claude/skills/e2e-routine frontend/tests/e2e (stay on main's commit; overlay only the routine's files). Then read .claude/skills/e2e-routine/SKILL.md. For this bootstrap run use FORCE=1 on Phase 1, run Phases 0 to 4 and 6, skip the adaptation PR of Phase 5 (list what it would change instead), and report build and suite timings.-```-## Environment `bytebase-e2e` (create once at claude.ai/code, Environments)| Setting | Value |diff --git a/.claude/skills/e2e-routine/SKILL.md b/.claude/skills/e2e-routine/SKILL.mdindex 5020374a1e..9a62dab570 100644--- a/.claude/skills/e2e-routine/SKILL.md+++ b/.claude/skills/e2e-routine/SKILL.md@@ -67,7 +67,7 @@ issue body is `references/issue-template.md`; the ledger format is| `E2E_ROUTINE_UMBRELLA` | none | the umbrella issue (parent of filed issues, target of the run comment) || `E2E_ROUTINE_LINEAR_TEAM` | `BOT` | the only Linear team the routine may create issues in || `E2E_ROUTINE_SLACK_CHANNEL` | none | when set and the Slack connector is attached, post the report summary there |-| `E2E_RESULTS_ROOT` | `/tmp/e2e-routine` | run directories |+| `E2E_RESULTS_ROOT` | `/tmp/e2e-routine/<checkout>-<path hash>` | run directories, one root per checkout so two worktrees never share `run-dir.sh`'s current run || `E2E_LEDGER_BRANCH` | `claude/e2e-ledger` | ledger branch || `E2E_LEDGER_PUSH` | `1` | `0` commits the ledger entry to the local ref `refs/e2e-routine/ledger-dry-run` without pushing, and ledger reads use it (local dry runs) || `E2E_GLOBAL_TIMEOUT_MS` | `7200000` | wall-clock cap for one suite run; tests not reached count as skipped and the run is reported as such |@@ -116,12 +116,12 @@ DIR=$(bash .claude/skills/e2e-routine/scripts/run-dir.sh) && bash .claude/skills```Build from `origin/main` checked out at `head_sha` (`git checkout -q-<head_sha>` in the clone is fine; the ledger records `HEAD`, so it must be-main's commit). While this skill lives on a branch, overlay its files onto that-checkout instead of cherry-picking:-`git checkout origin/<branch> -- .claude/skills/e2e-routine frontend/tests/e2e`.-A build failure is INFRA unless the failing step is the Go or frontend compile-of a product file.+<head_sha>` in the clone is fine). The skill and the suite are both on `main`,+so nothing is overlaid: copying files in from another branch reverts every+`main` change to them since that branch forked. The ledger records the main+commit under test, never a local commit on top (`references/ledger.md`), and+refuses a tree whose product files are not on `main`. A build failure is INFRA+unless the failing step is the Go or frontend compile of a product file.- INFRA (dependency download, toolchain, disk): record it with`summarize.py --pre-run-failure "$(bash .claude/skills/e2e-routine/scripts/run-dir.sh)" build "<failing step and first error line>"`@@ -143,15 +143,18 @@ python3 .claude/skills/e2e-routine/scripts/summarize.py "$(bash .claude/skills/e`start` detaches the suite (its own session, so a tool timeout cannot kill it)and prints the run directory. Call `wait` with the Bash tool's maximum timeout;if the tool returns before the suite ends, call `wait` again: it resumes. Do not-run anything else that boots a Bytebase while it runs. The suite takes roughly-40 to 60 minutes.+run anything else that boots a Bytebase while it runs. A green suite took+43 minutes on the 4-vCPU cloud VM and 15 minutes on a 12-core Mac; red tests+add their retries on top.`summarize.py` writes `summary.json` and `summary.md` (buckets: `failed`,-`flaky`, `lock_flipped`, `lock_holding`, `not_run`, `skipped`, `infra`,-`incomplete`). Each entry's `evidence` names the screenshots (one per open-page), aria snapshot (`error-context.md`) and trace of the failed attempt it-reports; `summary.md` prints the screenshot paths under each failed and flaky-row. When the server was lost and `server.log` holds a Go crash, the run is+`flaky`, `lock_flipped`, `lock_unstable`, `lock_holding`, `not_run`, `skipped`, `infra`,+`incomplete`; the truth table is in `references/triage.md`). Each entry's+`evidence` names the screenshots (one per open page), aria snapshot+(`error-context.md`) and trace of the anomalous attempt it reports: a failure+of an ordinary test, a pass of a `test.fail()` lock. `summary.md` prints the+screenshot paths under each row, and a `hint` when the error itself names the+class (a strict-mode violation is a spec defect). When the server was lost and `server.log` holds a Go crash, the run isred, not infra: one `failed` row "Bytebase server crashed" carries the crashline and backend frame, and the tests that died with it go to `lost_to_crash`(SERVER-CRASH in `triage.md`). `not_run` lists tests that never ran because an earlier test in@@ -165,11 +168,14 @@ say how much wall clock the build and the suite took.## Phase 4: Classify and verify-Work `references/triage.md` for every entry of `failed`, `flaky` and-`lock_flipped`. For each `failed` entry:+Work `references/triage.md` for every entry of `failed`, `flaky`,+`lock_unstable` and `lock_flipped`. For each `failed`, `flaky` and+`lock_unstable` entry:-1. Isolate: `run-suite.sh start "$(bash .claude/skills/e2e-routine/scripts/run-dir.sh id)-iso-<n>" -- <spec file> --retries=0 --repeat-each=2`,- then `wait` on the directory it prints, one file after another, always in the foreground. Never park+1. Isolate: `run-suite.sh start "$(bash .claude/skills/e2e-routine/scripts/run-dir.sh id)-iso-<n>" -- <spec file> --retries=0 --repeat-each=2`+ (a `lock_unstable` entry: `-- <spec file> -g "<title>" --retries=0 --repeat-each=10`, and count the passes).+ A `flaky` entry that is red here is not a flake; work it as `failed`.+ Then `wait` on the directory it prints, one file after another, always in the foreground. Never parkthe reruns in a background loop and go idle: a cloud session with nothingin the foreground pauses, and the first run lost 25 minutes that way.2. Open each of the entry's `evidence.screenshots` with the Read tool and look@@ -196,7 +202,7 @@ check it in Linear (`get_issue`).### Phase 4 gate-The decision table, with every `failed` row showing the isolated rerun result,+The decision table, with every `failed`, `flaky` and `lock_unstable` row showing the isolated rerun result,the screenshot path and what it shows, and a commit or the words "no commitexplains it", and every PRODUCT row naming the violated expectation.diff --git a/.claude/skills/e2e-routine/references/ledger.md b/.claude/skills/e2e-routine/references/ledger.mdindex 69c8798adb..4b4bdf9e18 100644--- a/.claude/skills/e2e-routine/references/ledger.md+++ b/.claude/skills/e2e-routine/references/ledger.md@@ -31,9 +31,24 @@ flakes.json # {"runs": [sha7 ...last 20], "tests": {"<file>::<titleRules:-- The attribution baseline (`ledger.sh last-green`) is the newest `runs/*.json` with `"result": "green"`;- when none exists, the last run's `sha`; when the ledger is empty, 60 days- back on `main`. A spec can have been red for longer than any window, so+- `sha` is always a commit on `main`: the one whose product files (`frontend`,+ `backend`, `proto`, `go.mod`, `go.sum`, outside `frontend/tests/e2e`) the+ binary was built from. Local commits that touch only the e2e tree or+ `.claude/` (an adaptation, an overlay) do not change it, so `run-suite.sh`+ records the merge base with `origin/main` in that case. A tree whose+ product files match no `main` commit is a branch, not main: `ledger.sh+ record` refuses it (exit 1) instead of writing a SHA no later run can use.+- "On `main`" is checked against `origin/main` as fetched now, else the+ local `origin/main` when the fetch fails (an older main is still main),+ never against HEAD: that fallback would let any branch pass. A SHA this+ clone lacks is fetched before it is judged, and a shallow clone is first+ deepened to 120 days (`E2E_HISTORY_DAYS`), so a missing object never reads+ as "not on main". Picking the commit to test (`should-run.sh`) is different:+ there HEAD is the fallback.+- The attribution baseline (`ledger.sh last-green`) is the newest `runs/*.json` with `"result": "green"`+ whose `sha` is on `main`; when none exists, the last such run's `sha`; when+ there is none, 60 days back on `main`. Entries whose `sha` is not on `main`+ (written before this rule) are skipped. A spec can have been red for longer than any window, so`attribute.py` widens on its own to the spec file's last change when nothinginside the window scores.- A run records itself exactly once, after `summarize.py`, whatever the result.diff --git a/.claude/skills/e2e-routine/references/triage.md b/.claude/skills/e2e-routine/references/triage.mdindex 776810db0c..615d9b93d2 100644--- a/.claude/skills/e2e-routine/references/triage.md+++ b/.claude/skills/e2e-routine/references/triage.md@@ -4,6 +4,27 @@ A red test is a signal, not a finding. The routine turns each one into exactlyone class with evidence, and each class has one action and one owner. Work thetable top to bottom; the first row whose signal and verification both hold wins.+## Buckets: how `summarize.py` sorts a test++A test's evidence is its anomalous attempts: those whose status differs from+the test's `expectedStatus` (an ordinary test expects `passed`, a `test.fail()`+lock expects `failed`; a timeout or interruption is never the expected+status). The bucket follows from which attempts were anomalous:++| Expected | Attempts (3 at most) | Bucket |+|---|---|---|+| passed | all passed | passed |+| passed | every attempt anomalous | `failed` |+| passed | anomalous, then a pass | `flaky` |+| failed (lock) | all failed | `lock_holding` |+| failed (lock) | all passed | `lock_flipped` |+| failed (lock) | at least one pass and at least one failure | `lock_unstable` |+| failed (lock) | no pass, at least one timeout or interruption | `failed` if no attempt failed as locked, else `flaky`: the test now hangs, which is not the locked bug |++The error and the screenshots come from the latest anomalous attempt. An+unexpected pass has no error message, so its error reads "Expected to fail,+but passed".+## Classes| Class | Signal (from summary.json) | Verification that must hold | Action | Owner |@@ -12,8 +33,9 @@ table top to bottom; the first row whose signal and verification both hold wins.| SERVER-CRASH | a `failed` row "Bytebase server crashed": the client lost the server (what INFRA calls `server-lost` or boot) and `server.log` holds a Go crash, a `panic:` or `fatal error:` line followed by a goroutine dump. The tests that died with it are in `lost_to_crash`, not triaged | the crash line and its first backend frame (`frame`) in `server.log`; `first_test` is the first test that lost the server, so the crash happened during it or the test before | PRODUCT issue, priority Urgent (every later test is blocked): `attribute.py --spec <frame file> --since "$(bash .claude/skills/e2e-routine/scripts/ledger.sh last-green)"` for the culprit; reproduce with `first_test` alone | culprit PR author via the frame's file; else the area owner of that file || INFRA | `infra` set: preflight or build failed (`--pre-run-failure`), no report, 0 tests ran with a boot/license/browser/psql signature, or the failed tests are environment casualties: at least 5 failures and 80% of them carry an environment signature (connection or boot is `server-lost`, unless `server.log` shows the server crashed: then SERVER-CRASH; browser, psql, toolchain or license keeps its own name), or the setup project failed with one and nothing else failed. The casualties move to `lost_to_infra` and are not triaged; failures without a signature stay in `failed` and are triaged row by row. A global-setup error with no such signature is a `failed` row named "global setup", and an error outside any test after tests ran (global teardown, a worker crash) is a `failed` row named "suite-level error"; both are triaged and attributed like any test. A nonzero Playwright exit that nothing else explains is infra (`unexplained-exit`): a run is green only when Playwright says so | `server.log` or `playwright.log` shows the boot, license, browser or psql error | No product issue. Record the ledger entry as `infra`; say what failed and the fix (environment, allowlist, setup script). Two consecutive infra runs: one BOT issue for the framework | framework owner || INCOMPLETE | `incomplete` set: the global timeout or a crash ended the run with tests never reached (nonzero exit, no failed test) | `playwright.log` ends with the timeout line or an abort; `summary.md` names how many tests never ran | Not green. Record `incomplete`; the next run retries the same SHA. Report the wall-clock split (build, suite) so the cap or a hung spec can be fixed; a hung spec that recurs becomes a BOT issue | framework owner |-| LOCK-FLIPPED | `lock_flipped`: a `test.fail()` lock passed | The describe title names an issue; that issue is Done, or its fix PR is in `since..main` | Adaptation PR removes `test.fail()`; comment on the issue when filing is enabled | none |-| FLAKE | `flaky`: failed, then passed on a retry | nothing more for this run | Append to `flakes.json` (ledger.sh does it). Issue when the same test is flaky in 3 of the last 5 runs, with the five error lines | framework owner, unless the error names a product race (then PRODUCT) |+| LOCK-UNSTABLE | `lock_unstable`: a `test.fail()` lock passed on some attempts and failed on others | rerun the one test alone, `-g "<title>" --retries=0 --repeat-each=10`, and count the passes; open the screenshot of a passing attempt and the trace of a failing one | The locked bug is intermittent, a race. Never remove the lock and never file a twin: comment on the locked issue with the pass rate (or propose the comment when filing is disabled), and reopen-worthy evidence goes there. If that issue is Done, say so: its fix is incomplete | the locked issue's assignee |+| LOCK-FLIPPED | `lock_flipped`: a `test.fail()` lock passed on every attempt | The describe title names an issue; that issue is Done, or its fix PR is in `since..main` | Adaptation PR removes `test.fail()`; comment on the issue when filing is enabled | none |+| FLAKE | `flaky`: failed, then passed on a retry | the isolated rerun (`--retries=0 --repeat-each=2`) is green. A `flaky` row that is red in isolation is not a flake: work it as a `failed` row (SPEC-ROT or PRODUCT). An error with `strict mode violation` is a locator that matches several elements, a spec defect and never timing: SPEC-ROT | Append to `flakes.json` (ledger.sh does it). Issue when the same test is flaky in 3 of the last 5 runs, with the five error lines | framework owner, unless the error names a product race (then PRODUCT) || TEST-ISOLATION | `failed`, but passes when its file runs alone (`--repeat-each=2`) | the isolated rerun is green | Adaptation PR when the leftover state is obvious (fixture not idempotent, order dependence); else a BOT issue for the framework | framework owner || SPEC-ROT | `failed`, red in isolation | a commit in `since..main` changed the asserted text, locator, route or flow **on purpose**: its PR title/body describes the new behavior, and the new screen is coherent (no error, no dead end, no permission or masking contradiction) | Adaptation PR: update the spec to the new behavior, citing the PR in the commit message. No Linear issue | the product PR's author reviews the adaptation || PRODUCT | `failed`, red in isolation | no intentional change explains it, **or** the new behavior breaks an invariant (see below) | Linear BOT issue, or a proposal when filing is disabled. In the adaptation PR, lock it with `test.describe('… (BUG BOT-n)')` + `test.fail()` only once the issue exists | culprit PR author via owners.json; else area default; else unassigned with candidates listed |@@ -25,12 +47,12 @@ page shows an error, an empty state in place of data, or an action with noeffect; export or copy policy is not enforced; the workspace or project cannotbe reached after sign-in.-## Verification procedure for a `failed` entry+## Verification procedure for a `failed`, `flaky` or `lock_unstable` entry-1. **Isolate.** `run-suite.sh start <run-id>-iso-N -- <spec file> --retries=0 --repeat-each=2`, then `wait`. Green here means TEST-ISOLATION.-2. **Read the evidence, not the title.** The entry's `evidence` in `summary.json` names the files of the failed attempt it reports: `screenshots`, one per open page (open them and look: they are what a user would have seen), `error_context`, the aria snapshot of the page at failure (the real DOM), and `trace` when a retry ran. In `playwright.log`, the attempt's error with the locator and the expected vs received text. In `server.log`, 4xx/5xx or panics around the timestamp.+1. **Isolate.** `run-suite.sh start <run-id>-iso-N -- <spec file> --retries=0 --repeat-each=2`, then `wait`. For a `failed` row, green here means TEST-ISOLATION; for a `flaky` row, red here means it is not a flake (see FLAKE). A `lock_unstable` row reruns the one test ten times instead (LOCK-UNSTABLE).+2. **Read the evidence, not the title.** The entry's `evidence` in `summary.json` names the files of the anomalous attempt it reports (see "Buckets"): `screenshots`, one per open page (open them and look: they are what a user would have seen), `error_context`, the aria snapshot of the page at failure (the real DOM), and `trace` when a retry ran. In `playwright.log`, the attempt's error with the locator and the expected vs received text. In `server.log`, 4xx/5xx or panics around the timestamp.3. **Pin the assertion.** Open the spec at the failing line. Write down the exact UI text or locator and what the test expected.-4. **Find the component.** `attribute.py --text "<quoted text>"` resolves the text through `frontend/src/locales/en*.json` to its key and the files that use it, and lists the commits in `since..origin/main` that touched them or the text. Read the top candidate's PR (`gh pr view <n> --json title,body`).+4. **Find the component.** `attribute.py --text "<quoted text>"` resolves the text through `frontend/src/locales/en*.json` to its key and the files that use it, and lists the commits in `since..HEAD` that touched them or the text. It also lists commits to the spec file itself and to `frontend/tests/e2e/framework/`, marked as test changes: a test commit that broke its own test is SPEC-ROT, adapted and reviewed by its author. It exits 2 when `--since` or `--head` is not a commit here, rather than reporting an empty range. Read the top candidate's PR (`gh pr view <n> --json title,body`).5. **Decide.** Apply the table. A decision cites: the attempt error line, the screenshot and what it shows, the spec line, and the commit (SPEC-ROT, PRODUCT) or the isolated rerun (TEST-ISOLATION).6. **Weigh before filing.** Validity (does the evidence show it), frequency (every run, or one race), return (what a fix buys). A valid, rare, cosmetic failure is still recorded in the report with the decision "not worth an issue" and why; it is never silently dropped.diff --git a/.claude/skills/e2e-routine/scripts/attribute.py b/.claude/skills/e2e-routine/scripts/attribute.pyindex e653be2e5b..d7de34c2fd 100755--- a/.claude/skills/e2e-routine/scripts/attribute.py+++ b/.claude/skills/e2e-routine/scripts/attribute.py@@ -1,15 +1,17 @@#!/usr/bin/env python3-"""Rank the product commits that could explain a red spec, and name an owner.+"""Rank the commits that could explain a red spec, and name an owner.attribute.py --spec frontend/tests/e2e/sql-editor/sql-editor-result.spec.ts \--since <last-green-sha> [--text "Request export" ...] [--json]-Candidates are the non-merge commits in <since>..origin/main that touch the-area's source paths (owners.json), the component files that render any quoted-UI text (found through the locale files), or whose diff adds/removes the text-itself (git log -S). Score = 3 per text hit + 1 per path hit. The owner is the-top candidate's author mapped through owners.json to a Linear user name, else-the area default.+Candidates are the non-merge commits in <since>..HEAD that touch the area's+source paths (owners.json), the component files that render any quoted UI+text (found through the locale files), or whose diff adds/removes the text+itself (git log -S), plus the test changes: commits to the spec itself and to+the e2e framework. Score = 3 per text hit or spec change, 2 per rendering+file, 1 per area or framework path. The owner is the top candidate's author+mapped through owners.json to a Linear user name, else the area default. Exits+with an error when --since or --head is not a commit in this clone."""import argparseimport glob@@ -114,13 +116,21 @@ def main():ap = argparse.ArgumentParser()ap.add_argument("--spec", required=True)ap.add_argument("--since", required=True)- # HEAD is the tested checkout. origin/main can move during a 40-60 minute+ # HEAD is the tested checkout. origin/main can move during a 15-45 minute# run, and a commit merged after the test cannot have caused its failure.ap.add_argument("--head", default="HEAD")ap.add_argument("--text", action="append", default=[])ap.add_argument("--json", action="store_true")a = ap.parse_args()a.spec = repo_spec_path(a.spec)+ # An unknown SHA makes every git query below come back empty, which would+ # read as "no commit explains it" and fall through to the area default.+ for name, ref in (("--since", a.since), ("--head", a.head)):+ if subprocess.run(["git", "-C", REPO, "rev-parse", "-q", "--verify", f"{ref}^{{commit}}"],+ capture_output=True).returncode != 0:+ print(f"attribute.py: {name} {ref!r} is not a commit in this clone; "+ "use `ledger.sh last-green` or a SHA on main", file=sys.stderr)+ sys.exit(2)owners = json.load(open(os.path.join(SKILL_DIR, "owners.json")))rng = f"{a.since}..{a.head}"@@ -144,6 +154,11 @@ def main():if source_mode:add(commits(rng, [a.spec]), 3, f"changes {a.spec}, the file that fails")+ else:+ # A test can break itself: a new locator or step in the spec, or a+ # framework helper it calls. These are test changes (SPEC-ROT).+ add(commits(rng, [a.spec]), 3, "test change: edits this spec")+ add(commits(rng, [f"{E2E_ROOT}/framework"]), 1, "test change: edits the e2e framework")# An empty path list would make git log return every commit in the range.if area_cfg["sources"]:add(commits(rng, area_cfg["sources"]), 1, f"touches {area} sources")diff --git a/.claude/skills/e2e-routine/scripts/build.sh b/.claude/skills/e2e-routine/scripts/build.shindex 839514d998..e9e56d4a18 100755--- a/.claude/skills/e2e-routine/scripts/build.sh+++ b/.claude/skills/e2e-routine/scripts/build.sh@@ -22,7 +22,7 @@ stamp="$(git rev-parse HEAD) $(git hash-object "$SERVER_BUILD")"# Any uncommitted change that can reach the binary invalidates the cached# build: the whole frontend tree (package.json, lockfile, Tailwind and Vite# config, plugins), not only frontend/src. The e2e specs do not reach it.-dirty="$(git status --porcelain -- frontend backend proto go.mod go.sum ':!frontend/tests/e2e' | head -1)"+dirty="$(git status --porcelain -- "${PRODUCT_PATHSPEC[@]}" | head -1)"timing="$RESULTS/build-timing.txt"if [ -x "$BIN" ] && [ -f "$STAMP" ] && [ "$(cat "$STAMP")" = "$stamp" ] && [ -z "$dirty" ] && [ "${FORCE_BUILD:-0}" != 1 ]; then@@ -42,8 +42,14 @@ phase() { # phase <name> <cmd...>printf '%-16s %5ss\n' "$name" "$dt" | tee -a "$timing" >&2}-phase pnpm-install pnpm --dir frontend install --frozen-lockfile-phase pnpm-release pnpm --dir frontend release+# pnpm from inside frontend/: corepack picks the version by the working+# directory, and only there does it see packageManager (pnpm 11). Run from+# the repo root it falls back to the global pnpm. A node_modules that another+# pnpm laid out must be rebuilt, and pnpm asks before deleting it, which+# aborts without a TTY; it is generated, so the build may purge it.+fe_pnpm() { (cd frontend && pnpm "$@"); }+phase pnpm-install fe_pnpm install --frozen-lockfile --config.confirmModulesPurge=false+phase pnpm-release fe_pnpm releasephase go-build bash "$SERVER_BUILD" "$BIN"echo "$stamp" >"$STAMP"log "built $BIN from $stamp"diff --git a/.claude/skills/e2e-routine/scripts/ledger.sh b/.claude/skills/e2e-routine/scripts/ledger.shindex 92a883ae22..32a5e9ea21 100755--- a/.claude/skills/e2e-routine/scripts/ledger.sh+++ b/.claude/skills/e2e-routine/scripts/ledger.sh@@ -32,11 +32,13 @@ last-result) git show "$(read_ref):last-run.json" 2>/dev/null | python3 -c 'imposhow) git show "$(read_ref):last-run.json" 2>/dev/null || echo '{}' ;;flakes) git show "$(read_ref):flakes.json" 2>/dev/null || echo '{}' ;;last-green)- # The attribution baseline (references/ledger.md): the newest green run's- # SHA, else the last run's, else main 60 days back. Computed on each call so- # a phase never depends on a variable an earlier shell set.+ # The attribution baseline (references/ledger.md): the newest green run on+ # main, else the newest run on main, else main 60 days back. Computed on each+ # call so a phase never depends on a variable an earlier shell set.r="$(read_ref)"- sha="$(python3 - "$r" <<'PY'+ main="$(main_ref_strict)" || die "no origin/main to check the ledger against"+ # Every entry as "<result> <sha>", newest first; on_main judges each one.+ entries="$(python3 - "$r" <<'PY'import json, subprocess, sysref = sys.argv[1]def git(*a):@@ -47,26 +49,38 @@ for path in git("ls-tree", "--name-only", f"{ref}:runs").split():runs.append(json.loads(git("show", f"{ref}:runs/{path}")))except ValueError:pass-green = [e for e in runs if e.get("result") == "green" and e.get("sha")]-if green:- print(max(green, key=lambda e: (e.get("date", ""), e.get("run_id", "")))["sha"])-else:- try:- print(json.loads(git("show", f"{ref}:last-run.json")).get("sha", ""))- except ValueError:- print("")+for e in sorted(runs, key=lambda e: (e.get("date", ""), e.get("run_id", "")), reverse=True):+ if e.get("sha"):+ print(e.get("result", "?"), e["sha"])PY)"- [ -n "$sha" ] || sha="$(git rev-list -1 --before='60 days ago' "$(main_sha)")"+ green="" newest=""+ while read -r result s; do+ [ -n "$s" ] && on_main "$s" "$main" || continue+ [ -n "$newest" ] || newest="$s"+ if [ "$result" = green ]; then green="$s"; break; fi+ done <<<"$entries"+ sha="${green:-$newest}"+ [ -n "$sha" ] || sha="$(git rev-list -1 --before='60 days ago' "$main")"+ [ -n "$sha" ] || die "no baseline: no ledger entry on main and no main commit 60 days back"echo "$sha";;record)RESULTS="${2:?results dir}"; note="${3:-}"[ -f "$RESULTS/summary.json" ] || die "$RESULTS/summary.json missing; run summarize.py first"+ sha="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("meta", {}).get("sha", ""))' "$RESULTS/summary.json")"+ # Only a main commit may enter the ledger (references/ledger.md); a dry run+ # from a branch records with a warning, since it never reaches the remote.+ if ! on_main "$sha"; then+ dry_run || die "run tested ${sha:-no SHA}, which is not a commit on main; not recorded"+ log "WARNING: ${sha:-no SHA} is not on main; recording on the dry-run ref only"+ fitmp="$(mktemp -d)"if dry_run && has_ref "$dry_ref"; thengit worktree add -q --detach "$tmp" "$dry_ref"- elif fetch_ledger; then+ elif fetch_ledger || has_ref "$remote_ref"; then+ # A failed fetch keeps the last fetched ledger rather than starting an+ # empty one; a push from it is then rejected, never a rewrite.git worktree add -q --detach "$tmp" "$remote_ref"elseroot="$(git "${ident[@]}" commit-tree "$(git hash-object -t tree /dev/null)" -m "e2e ledger root")"@@ -94,6 +108,7 @@ else:entry = {"run_id": meta.get("run_id"), "sha": sha, "date": day, "result": result, "totals": t,"failed": [f"{e['file']}::{e['title']}" for e in s.get("failed", [])],"flaky": [f"{e['file']}::{e['title']}" for e in s.get("flaky", [])],+ "lock_unstable": [f"{e['file']}::{e['title']}" for e in s.get("lock_unstable", [])],"infra": s.get("infra"), "note": note, "run_url": run_url}os.makedirs(os.path.join(dest, "runs"), exist_ok=True)# The run id keeps a retry of the same SHA on the same day from overwriting@@ -114,7 +129,7 @@ if not os.path.exists(os.path.join(dest, "README.md")):print(f"recorded {result} run {entry['run_id']} @ {sha[:7]}")PY(- cd "$tmp" && git add -A && git "${ident[@]}" commit -q -m "e2e ledger: $(date -u +%F) $(git -C "$REPO_ROOT" rev-parse --short HEAD)" \+ cd "$tmp" && git add -A && git "${ident[@]}" commit -q -m "e2e ledger: $(date -u +%F) ${sha:0:10}" \&& if dry_run; thengit update-ref "$dry_ref" HEADlog "E2E_LEDGER_PUSH=0: not pushing; recorded on $dry_ref, which ledger.sh reads while E2E_LEDGER_PUSH=0"diff --git a/.claude/skills/e2e-routine/scripts/lib.sh b/.claude/skills/e2e-routine/scripts/lib.shindex 0065fbb38c..177e7c5cf8 100755--- a/.claude/skills/e2e-routine/scripts/lib.sh+++ b/.claude/skills/e2e-routine/scripts/lib.sh@@ -5,7 +5,9 @@ set -o pipefailE2E_SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"REPO_ROOT="$(cd "$E2E_SKILL_DIR/../../.." && pwd)"-E2E_RESULTS_ROOT="${E2E_RESULTS_ROOT:-/tmp/e2e-routine}"+# One results root per checkout: run-dir.sh keeps its CURRENT pointer there,+# and two worktrees running at once must not take over each other's run.+E2E_RESULTS_ROOT="${E2E_RESULTS_ROOT:-/tmp/e2e-routine/$(basename "$REPO_ROOT")-$(printf '%s' "$REPO_ROOT" | cksum | cut -d' ' -f1)}"E2E_LEDGER_BRANCH="${E2E_LEDGER_BRANCH:-claude/e2e-ledger}"# The cloud image ships Node 20-22 on PATH; setup-env.sh installs 24 under /opt/node24.@@ -52,8 +54,9 @@ e2e_license_available() {if e2e_license >/dev/null 2>&1; then echo yes; else echo no; fi}-# origin/main as fetched now. When the fetch fails it is HEAD, never an older-# origin/main left by an earlier fetch, which would test or skip a stale SHA.+# The commit to test: origin/main as fetched now. When the fetch fails it is+# HEAD, never an older origin/main left by an earlier fetch, which would test+# or skip a stale SHA. Not for "is X on main": see main_ref_strict.main_sha() {if git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null; thengit -C "$REPO_ROOT" rev-parse origin/main@@ -63,4 +66,58 @@ main_sha() {fi}+# The main to judge membership against (references/ledger.md): fetched now,+# else the local origin/main, since an older main is still main. Never HEAD,+# which would let any branch pass. Exit 1 when there is no origin/main at all.+main_ref_strict() {+ git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null \+ || log "fetch of origin/main failed; checking against the local origin/main"+ git -C "$REPO_ROOT" rev-parse -q --verify 'origin/main^{commit}'+}++# A shallow clone lacks the history that ancestry checks and attribution+# need; deepen it once to E2E_HISTORY_DAYS of main.+ensure_history() {+ [ "$(git -C "$REPO_ROOT" rev-parse --is-shallow-repository 2>/dev/null)" = true ] || return 0+ git -C "$REPO_ROOT" fetch -q --shallow-since="${E2E_HISTORY_DAYS:-120} days ago" origin main 2>/dev/null \+ || log "could not deepen the shallow clone; older ledger entries may not resolve"+}++# on_main <sha> [main]: the SHA is a commit on main. A SHA this clone lacks is+# fetched first (GitHub serves any published commit), so an absent object is+# never read as "not on main"; one GitHub does not have fails. In a shallow+# clone a fetched commit is not yet linked to main, so main is deepened to+# just before the commit's date and the check is made again.+on_main() {+ local sha="$1" main="${2:-}" when+ [ -n "$sha" ] || return 1+ [ -n "$main" ] || main="$(main_ref_strict)" || return 1+ ensure_history+ git -C "$REPO_ROOT" cat-file -e "$sha^{commit}" 2>/dev/null \+ || git -C "$REPO_ROOT" fetch -q origin "$sha" 2>/dev/null || return 1+ git -C "$REPO_ROOT" merge-base --is-ancestor "$sha" "$main" && return 0+ [ "$(git -C "$REPO_ROOT" rev-parse --is-shallow-repository 2>/dev/null)" = true ] || return 1+ when="$(git -C "$REPO_ROOT" log -1 --format=%ct "$sha")" || return 1+ git -C "$REPO_ROOT" fetch -q --shallow-since="$((when - 86400))" origin main 2>/dev/null || return 1+ git -C "$REPO_ROOT" merge-base --is-ancestor "$sha" "$main"+}++# What the binary is built from; the e2e tree is excluded because specs are+# read at run time and an adaptation changes only them.+PRODUCT_PATHSPEC=(frontend backend proto go.mod go.sum ':!frontend/tests/e2e')++# The main commit the checkout's product files come from (references/ledger.md):+# HEAD when it is on main, the merge base when the commits above it, and any+# uncommitted change, touch only the e2e tree or .claude/. Exit 1 when the+# product files match no main commit.+tested_sha() {+ local main base+ main="$(main_ref_strict)" || return 1+ ensure_history+ base="$(git -C "$REPO_ROOT" merge-base HEAD "$main" 2>/dev/null)" || return 1+ git -C "$REPO_ROOT" diff --quiet "$base" HEAD -- "${PRODUCT_PATHSPEC[@]}" || return 1+ [ -z "$(git -C "$REPO_ROOT" status --porcelain -- "${PRODUCT_PATHSPEC[@]}" | head -1)" ] || return 1+ echo "$base"+}+run_id_now() { date -u +%Y%m%dT%H%M%SZ; }diff --git a/.claude/skills/e2e-routine/scripts/preflight.sh b/.claude/skills/e2e-routine/scripts/preflight.shindex a9bd6e8c6e..5b9a4d27e4 100755--- a/.claude/skills/e2e-routine/scripts/preflight.sh+++ b/.claude/skills/e2e-routine/scripts/preflight.sh@@ -13,8 +13,12 @@ bad() { printf 'FAIL %-28s %s\n' "$1" "$2"; fail=1; }v=$(node --version 2>/dev/null || true)case "$v" in v2[4-9].*|v[3-9]*) ok node "$v" ;; "") bad node "missing" ;; *) warn node "$v (CI uses 24.18.1; build_init.sh requires >= 24.15)" ;; esac-v=$(pnpm --version 2>/dev/null || true)-[ -n "$v" ] && ok pnpm "$v (packageManager pins 11.17.0; pnpm self-switches)" || bad pnpm "missing"+# The pnpm that build.sh runs: the one corepack picks inside frontend/.+want=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("packageManager","").split("@")[-1].split("+")[0])' frontend/package.json 2>/dev/null)+v=$(cd frontend && pnpm --version 2>/dev/null || true)+if [ -z "$v" ]; then bad pnpm "missing"+elif [ -n "$want" ] && [ "$v" != "$want" ]; then bad pnpm "$v inside frontend/, but packageManager pins $want"+else ok pnpm "$v inside frontend/ (packageManager pins ${want:-?})"; fiv=$(go version 2>/dev/null | awk '{print $3}')[ -n "$v" ] && ok go "$v (go.mod 1.27.1; GOTOOLCHAIN=$GOTOOLCHAIN)" || bad go "missing"diff --git a/.claude/skills/e2e-routine/scripts/run-suite.sh b/.claude/skills/e2e-routine/scripts/run-suite.shindex 1b4a6d41e7..ee191d6707 100755--- a/.claude/skills/e2e-routine/scripts/run-suite.sh+++ b/.claude/skills/e2e-routine/scripts/run-suite.sh@@ -10,7 +10,7 @@# JSON reporter), test-results/ (traces, screenshots, error-context.md),# exit-code, meta.json. Every test gets three attempts (--retries=2): red only# when all three fail, flaky when a retry passes. A run is capped at-# E2E_GLOBAL_TIMEOUT_MS (default 2 h; a healthy run takes about 1 h) because a+# E2E_GLOBAL_TIMEOUT_MS (default 2 h; a green run took 43 min on the cloud VM) because a# night of hung tests otherwise multiplies 3 attempts by each test's timeout.# The license comes from lib.sh's e2e_license and is exported only into the# detached process.@@ -47,12 +47,17 @@ EOF# A JSON encoder writes the metadata: Playwright arguments can hold quotes,# backslashes or newlines that raw interpolation would turn into invalid JSON.- python3 - "$RESULTS/meta.json" "$run_id" "$(git rev-parse HEAD)" "$(git rev-parse --short HEAD)" \+ # `sha` is the main commit under test (references/ledger.md), not HEAD: an+ # adaptation or overlay commit on top must not end up in the ledger.+ if sha="$(tested_sha)"; then on_main=1; else sha="$(git rev-parse HEAD)"; on_main=0+ log "product files match no main commit; recording HEAD $sha (ledger.sh record will refuse it)"+ fi+ python3 - "$RESULTS/meta.json" "$run_id" "$sha" "$on_main" "$(git rev-parse HEAD)" \"$(date -u +%FT%TZ)" "$(hostname)" "$@" <<'PY'import json, sys-path, run_id, sha, short_sha, started_at, host, *args = sys.argv[1:]-json.dump({"run_id": run_id, "sha": sha, "short_sha": short_sha, "started_at": started_at,- "host": host, "args": args}, open(path, "w"))+path, run_id, sha, on_main, head, started_at, host, *args = sys.argv[1:]+json.dump({"run_id": run_id, "sha": sha, "short_sha": sha[:10], "on_main": on_main == "1", "head": head,+ "started_at": started_at, "host": host, "args": args}, open(path, "w"))PY# Playwright arguments reach the runner shell-quoted word by word, so a@@ -110,11 +115,15 @@ status)RESULTS="${1:?results dir}"lg="$RESULTS/playwright.log"[ -f "$lg" ] || { echo "no log yet"; exit 0; }- done_n=$(grep -cE '^\s+[✓✘-]\s' "$lg" 2>/dev/null || echo 0)- fail_n=$(grep -cE '^\s+✘\s' "$lg" 2>/dev/null || echo 0)- last=$(grep -E '^\s+[✓✘-]\s' "$lg" | tail -1 | cut -c1-110)+ # A list-reporter result line carries a run number and a [project] tag;+ # Playwright's indented call-log lines ("- waiting for ...") do not.+ # grep -c prints 0 itself when nothing matches, so no fallback echo.+ result_line='^\s+[✓✘-]\s+[0-9]+ \['+ done_n=$(grep -cE "$result_line" "$lg" 2>/dev/null; true)+ fail_n=$(grep -cE '^\s+✘\s+[0-9]+ \[' "$lg" 2>/dev/null; true)+ last=$(grep -E "$result_line" "$lg" | tail -1 | cut -c1-110)state=$([ -f "$RESULTS/exit-code" ] && echo "done(exit $(cat "$RESULTS/exit-code"))" || echo running)- echo "$(date -u +%H:%M:%SZ) $state lines=$done_n red=$fail_n last: $last"+ echo "$(date -u +%H:%M:%SZ) $state results=${done_n:-0} red=${fail_n:-0} last: $last";;*)echo "usage: run-suite.sh start [run-id] [-- playwright args] | wait <dir> [max-min] | status <dir>" >&2; exit 2 ;;diff --git a/.claude/skills/e2e-routine/scripts/summarize.py b/.claude/skills/e2e-routine/scripts/summarize.pyindex f65724c2f9..1888fc39b9 100755--- a/.claude/skills/e2e-routine/scripts/summarize.py+++ b/.claude/skills/e2e-routine/scripts/summarize.py@@ -3,10 +3,11 @@summarize.py <results-dir> -> writes summary.json and summary.md there, prints summary.md-Buckets (see references/triage.md for what to do with each):+Buckets (the truth table and the action for each are in references/triage.md):failed unexpected on every attempt -> verify: product regression or spec rot- flaky failed, then passed on a retry -> flakes ledger- lock_flipped test.fail() lock that now passes -> the bug was fixed; remove the lock+ flaky failed, then passed on a retry -> rerun alone; a flake only if green there+ lock_flipped test.fail() lock passed every attempt -> the bug was fixed; remove the lock+ lock_unstable test.fail() lock passed some attempts -> the locked bug is intermittent; keep the locklock_holding test.fail() lock still failing -> not redinfra no test ran, or every test died on a boot/license/browser signature(a lost server whose server.log holds a Go crash is a failed row instead)@@ -93,15 +94,50 @@ def first_line(msg):return ""-def pick_evidence(results):- """The files that show the failure the entry reports: the screenshots (one- per open page) and aria snapshot of the latest failed attempt that has- them, and the trace of a failed attempt (Playwright traces the first retry- only). A passing retry's trace shows the pass, so it is never the evidence."""- failed = [(i, r) for i, r in enumerate(results) if r.get("status") not in ("passed", "skipped")]+def anomalous(results, expected):+ """The attempts whose status is not the test's expected one: the failures+ of an ordinary test, the passes (or timeouts) of a test.fail() lock+ (references/triage.md, "Buckets")."""+ return [(i, r) for i, r in enumerate(results) if r.get("status") not in (expected, "skipped")]+++def attempt_error(results, expected):+ """The latest anomalous attempt's error as (first line, whole message). An+ unexpected pass carries no error, so it is named for what it is."""+ odd = anomalous(results, expected)+ if not odd:+ # A holding lock: what it still fails with, as expected.+ odd = [(i, r) for i, r in enumerate(results) if (r.get("error") or {}).get("message")]+ if not odd:+ return "", ""+ r = odd[-1][1]+ msg = re.sub(r"\x1b\[[0-9;]*m", "", (r.get("error") or {}).get("message") or "")+ if msg:+ return first_line(msg), msg+ if r.get("status") == "passed":+ return f"Expected to fail, but passed (attempt {odd[-1][0] + 1} of {len(results)})", ""+ return r.get("status") or "", ""+++def hint(bucket, message, expected):+ # Playwright puts the strict-mode cause lines below the assertion's first line.+ if "strict mode violation" in message:+ return "a locator matches several elements: a spec defect, never timing (SPEC-ROT)"+ if expected == "failed" and bucket in ("failed", "flaky"):+ return "a test.fail() lock timed out: the test now hangs, which is not the locked bug"+ return None+++def pick_evidence(results, expected="passed"):+ """The files that show what the entry reports: the screenshots (one per+ open page) and aria snapshot of the latest anomalous attempt that has+ them, and the trace of an anomalous attempt (Playwright traces the first+ retry only). An attempt that went as expected shows nothing wrong, so it is+ never the evidence."""+ odd = anomalous(results, expected)def latest(name):- for i, r in reversed(failed):+ for i, r in reversed(odd):paths = sorted({a["path"] for a in r.get("attachments", [])if a.get("name") == name and a.get("path") and os.path.exists(a["path"])})if paths:@@ -129,15 +165,23 @@ def walk(suite, file_title, out):def classify(test):+ """The bucket per the truth table in references/triage.md ("Buckets")."""status = test.get("status") # expected | unexpected | flaky | skippedexpected = test.get("expectedStatus") # passed | failed | skippedif status == "skipped":return "skipped"- if status == "flaky":- return "flaky"- if expected == "failed":- return "lock_holding" if status == "expected" else "lock_flipped"- return "passed" if status == "expected" else "failed"+ if expected != "failed":+ return {"expected": "passed", "flaky": "flaky"}.get(status, "failed")+ statuses = [r.get("status") for r in test.get("results", [])]+ passes = statuses.count("passed")+ if passes and passes == len(statuses):+ return "lock_flipped"+ if passes:+ return "lock_unstable"+ if status == "expected":+ return "lock_holding"+ # No pass, yet not the expected failure on every attempt: a timeout.+ return "flaky" if "failed" in statuses else "failed"def infra_signature(text):@@ -159,8 +203,8 @@ def main(results_dir):exit_code = int(open(ep).read().strip() or 0)summary = {"meta": meta, "exit_code": exit_code, "totals": {}, "failed": [], "flaky": [],- "lock_flipped": [], "lock_holding": [], "skipped": [], "not_run": [], "infra": None,- "incomplete": None, "errors": []}+ "lock_flipped": [], "lock_unstable": [], "lock_holding": [], "skipped": [], "not_run": [],+ "infra": None, "incomplete": None, "errors": []}if not os.path.exists(report_path):log_tail = ""@@ -194,10 +238,8 @@ def main(results_dir):if bucket == "skipped":counts["not_run" if not deliberate else "skipped_deliberate"] += 1results = test.get("results", [])- # The latest attempt that carries an error: for a flaky test the final- # attempt passed and has none, and the failure is the evidence.- errored = [r for r in results if (r.get("error") or {}).get("message")]- err = first_line(errored[-1]["error"]["message"]) if errored else ""+ expected = test.get("expectedStatus") or "passed"+ err, message = attempt_error(results, expected)entry = {"file": file_title,"title": spec.get("title"),@@ -208,13 +250,14 @@ def main(results_dir):"duration_ms": sum(r.get("durationMs", r.get("duration", 0)) or 0 for r in results),"error": err,"infra_signature": infra_signature(err),- "evidence": pick_evidence(results),+ "evidence": pick_evidence(results, expected),+ "hint": hint(bucket, message, expected),"annotations": [a.get("type") for a in test.get("annotations", [])],}if bucket == "skipped" and not deliberate:entry["blocked_by"] = last_failed_in_file.get(file_title)summary["not_run"].append(entry)- elif bucket in ("failed", "flaky", "lock_flipped", "lock_holding", "skipped"):+ elif bucket in ("failed", "flaky", "lock_flipped", "lock_unstable", "lock_holding", "skipped"):summary[bucket].append(entry)stats = report.get("stats", {})@@ -225,6 +268,7 @@ def main(results_dir):"failed": counts["failed"],"flaky": counts["flaky"],"lock_flipped": counts["lock_flipped"],+ "lock_unstable": counts["lock_unstable"],"lock_holding": counts["lock_holding"],"skipped": counts["skipped"],"not_run": counts["not_run"],@@ -380,12 +424,13 @@ def write(results_dir, s):lines += [f"**INCOMPLETE**: {s['incomplete']['reason']}; {s['incomplete']['not_run']} test(s) never ran. Not a green run.", ""]if t:lines += [f"{t['passed']} passed, {t['failed']} failed, {t['flaky']} flaky, {t['lock_flipped']} lock flipped, "- f"{t['lock_holding']} lock holding, {t.get('not_run', 0)} did not run, "+ f"{t.get('lock_unstable', 0)} lock unstable, {t['lock_holding']} lock holding, {t.get('not_run', 0)} did not run, "f"{t.get('skipped_deliberate', t['skipped'])} skipped on purpose, {t['tests']} total, {t['duration_s']}s", ""]for bucket, head in (("failed", "Failed (every attempt)"), ("flaky", "Flaky (passed on retry)"),("lost_to_infra", "Lost to the environment (not triaged; they rerun once it is fixed)"),("lost_to_crash", "Lost to the server crash (not triaged; they rerun once the crash is fixed)"),("lock_flipped", "Locks that now pass"),+ ("lock_unstable", "Locks that pass only sometimes (the locked bug is intermittent)"),("not_run", "Did not run (not covered by this run)"), ("skipped", "Skipped on purpose")):if s.get(bucket):lines += [f"## {head}", ""]@@ -396,7 +441,9 @@ def write(results_dir, s):lines.append(f" - {e['error']}")if bucket == "not_run":lines.append(f" - blocked by: {e.get('blocked_by') or 'the run ending early'}")- if bucket in ("failed", "flaky", "lock_flipped"):+ if e.get("hint"):+ lines.append(f" - hint: {e['hint']}")+ if bucket in ("failed", "flaky", "lock_flipped", "lock_unstable"):lines.append(f" - screenshots: {evidence_line(e.get('evidence') or NO_EVIDENCE, results_dir)}")lines.append("")if s.get("errors"):@@ -419,9 +466,17 @@ def record_pre_run_failure(results_dir, kind, detail, cause="infra"):if os.path.exists(mp):meta = json.load(open(mp))else:- repo = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..", "..", ".."))- sha = subprocess.run(["git", "-C", repo, "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()- meta = {"run_id": os.path.basename(os.path.normpath(results_dir)), "sha": sha, "short_sha": sha[:10]}+ here = os.path.dirname(os.path.abspath(__file__))+ repo = os.path.abspath(os.path.join(here, "..", "..", "..", ".."))+ # The same rule as run-suite.sh: the main commit under test, else HEAD+ # marked off main (references/ledger.md).+ sha = subprocess.run(["bash", "-c", 'source "$1"; tested_sha', "_", os.path.join(here, "lib.sh")],+ capture_output=True, text=True).stdout.strip()+ on_main = bool(sha)+ if not sha:+ sha = subprocess.run(["git", "-C", repo, "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()+ meta = {"run_id": os.path.basename(os.path.normpath(results_dir)), "sha": sha, "short_sha": sha[:10],+ "on_main": on_main}json.dump(meta, open(mp, "w"))summary = {"meta": meta, "exit_code": None, "totals": {}, "failed": [], "flaky": [],"lock_flipped": [], "lock_holding": [], "skipped": [], "not_run": [],