Skip to content

Database governance built for humans and agents — controlling changes and access across every major database.

test(e2e): fix the routine gaps its first full run found (#21645)

zchpeter762e4ed5eae0
diff --git a/.claude/skills/e2e-routine/ROUTINE_PROMPT.md b/.claude/skills/e2e-routine/ROUTINE_PROMPT.md
index bf6157a9ab..0438b04741 100644
--- a/.claude/skills/e2e-routine/ROUTINE_PROMPT.md
+++ b/.claude/skills/e2e-routine/ROUTINE_PROMPT.md
@@ -21,15 +21,6 @@ Policy for this task:
- Finish with Phase 6 even when an earlier phase failed, and end your final message with the report.
```
-## Bootstrap prompt (until the skill is on `main`)
-
-The cloud clone starts on `main`. While the skill lives on a branch, prepend
-this to the prompt and drop it once the PR merges:
-
-```
-First: git fetch origin worktree-e2e-routine && git checkout origin/worktree-e2e-routine -- .claude/skills/e2e-routine frontend/tests/e2e (stay on main's commit; overlay only the routine's files). Then read .claude/skills/e2e-routine/SKILL.md. For this bootstrap run use FORCE=1 on Phase 1, run Phases 0 to 4 and 6, skip the adaptation PR of Phase 5 (list what it would change instead), and report build and suite timings.
-```
-
## Environment `bytebase-e2e` (create once at claude.ai/code, Environments)
| Setting | Value |
diff --git a/.claude/skills/e2e-routine/SKILL.md b/.claude/skills/e2e-routine/SKILL.md
index 5020374a1e..9a62dab570 100644
--- a/.claude/skills/e2e-routine/SKILL.md
+++ b/.claude/skills/e2e-routine/SKILL.md
@@ -67,7 +67,7 @@ issue body is `references/issue-template.md`; the ledger format is
| `E2E_ROUTINE_UMBRELLA` | none | the umbrella issue (parent of filed issues, target of the run comment) |
| `E2E_ROUTINE_LINEAR_TEAM` | `BOT` | the only Linear team the routine may create issues in |
| `E2E_ROUTINE_SLACK_CHANNEL` | none | when set and the Slack connector is attached, post the report summary there |
-| `E2E_RESULTS_ROOT` | `/tmp/e2e-routine` | run directories |
+| `E2E_RESULTS_ROOT` | `/tmp/e2e-routine/<checkout>-<path hash>` | run directories, one root per checkout so two worktrees never share `run-dir.sh`'s current run |
| `E2E_LEDGER_BRANCH` | `claude/e2e-ledger` | ledger branch |
| `E2E_LEDGER_PUSH` | `1` | `0` commits the ledger entry to the local ref `refs/e2e-routine/ledger-dry-run` without pushing, and ledger reads use it (local dry runs) |
| `E2E_GLOBAL_TIMEOUT_MS` | `7200000` | wall-clock cap for one suite run; tests not reached count as skipped and the run is reported as such |
@@ -116,12 +116,12 @@ DIR=$(bash .claude/skills/e2e-routine/scripts/run-dir.sh) && bash .claude/skills
```
Build from `origin/main` checked out at `head_sha` (`git checkout -q
-<head_sha>` in the clone is fine; the ledger records `HEAD`, so it must be
-main's commit). While this skill lives on a branch, overlay its files onto that
-checkout instead of cherry-picking:
-`git checkout origin/<branch> -- .claude/skills/e2e-routine frontend/tests/e2e`.
-A build failure is INFRA unless the failing step is the Go or frontend compile
-of a product file.
+<head_sha>` in the clone is fine). The skill and the suite are both on `main`,
+so nothing is overlaid: copying files in from another branch reverts every
+`main` change to them since that branch forked. The ledger records the main
+commit under test, never a local commit on top (`references/ledger.md`), and
+refuses a tree whose product files are not on `main`. A build failure is INFRA
+unless the failing step is the Go or frontend compile of a product file.
- INFRA (dependency download, toolchain, disk): record it with
`summarize.py --pre-run-failure "$(bash .claude/skills/e2e-routine/scripts/run-dir.sh)" build "<failing step and first error line>"`
@@ -143,15 +143,18 @@ python3 .claude/skills/e2e-routine/scripts/summarize.py "$(bash .claude/skills/e
`start` detaches the suite (its own session, so a tool timeout cannot kill it)
and prints the run directory. Call `wait` with the Bash tool's maximum timeout;
if the tool returns before the suite ends, call `wait` again: it resumes. Do not
-run anything else that boots a Bytebase while it runs. The suite takes roughly
-40 to 60 minutes.
+run anything else that boots a Bytebase while it runs. A green suite took
+43 minutes on the 4-vCPU cloud VM and 15 minutes on a 12-core Mac; red tests
+add their retries on top.
`summarize.py` writes `summary.json` and `summary.md` (buckets: `failed`,
-`flaky`, `lock_flipped`, `lock_holding`, `not_run`, `skipped`, `infra`,
-`incomplete`). Each entry's `evidence` names the screenshots (one per open
-page), aria snapshot (`error-context.md`) and trace of the failed attempt it
-reports; `summary.md` prints the screenshot paths under each failed and flaky
-row. When the server was lost and `server.log` holds a Go crash, the run is
+`flaky`, `lock_flipped`, `lock_unstable`, `lock_holding`, `not_run`, `skipped`, `infra`,
+`incomplete`; the truth table is in `references/triage.md`). Each entry's
+`evidence` names the screenshots (one per open page), aria snapshot
+(`error-context.md`) and trace of the anomalous attempt it reports: a failure
+of an ordinary test, a pass of a `test.fail()` lock. `summary.md` prints the
+screenshot paths under each row, and a `hint` when the error itself names the
+class (a strict-mode violation is a spec defect). When the server was lost and `server.log` holds a Go crash, the run is
red, not infra: one `failed` row "Bytebase server crashed" carries the crash
line and backend frame, and the tests that died with it go to `lost_to_crash`
(SERVER-CRASH in `triage.md`). `not_run` lists tests that never ran because an earlier test in
@@ -165,11 +168,14 @@ say how much wall clock the build and the suite took.
## Phase 4: Classify and verify
-Work `references/triage.md` for every entry of `failed`, `flaky` and
-`lock_flipped`. For each `failed` entry:
+Work `references/triage.md` for every entry of `failed`, `flaky`,
+`lock_unstable` and `lock_flipped`. For each `failed`, `flaky` and
+`lock_unstable` entry:
-1. Isolate: `run-suite.sh start "$(bash .claude/skills/e2e-routine/scripts/run-dir.sh id)-iso-<n>" -- <spec file> --retries=0 --repeat-each=2`,
- then `wait` on the directory it prints, one file after another, always in the foreground. Never park
+1. Isolate: `run-suite.sh start "$(bash .claude/skills/e2e-routine/scripts/run-dir.sh id)-iso-<n>" -- <spec file> --retries=0 --repeat-each=2`
+ (a `lock_unstable` entry: `-- <spec file> -g "<title>" --retries=0 --repeat-each=10`, and count the passes).
+ A `flaky` entry that is red here is not a flake; work it as `failed`.
+ Then `wait` on the directory it prints, one file after another, always in the foreground. Never park
the reruns in a background loop and go idle: a cloud session with nothing
in the foreground pauses, and the first run lost 25 minutes that way.
2. Open each of the entry's `evidence.screenshots` with the Read tool and look
@@ -196,7 +202,7 @@ check it in Linear (`get_issue`).
### Phase 4 gate
-The decision table, with every `failed` row showing the isolated rerun result,
+The decision table, with every `failed`, `flaky` and `lock_unstable` row showing the isolated rerun result,
the screenshot path and what it shows, and a commit or the words "no commit
explains it", and every PRODUCT row naming the violated expectation.
diff --git a/.claude/skills/e2e-routine/references/ledger.md b/.claude/skills/e2e-routine/references/ledger.md
index 69c8798adb..4b4bdf9e18 100644
--- a/.claude/skills/e2e-routine/references/ledger.md
+++ b/.claude/skills/e2e-routine/references/ledger.md
@@ -31,9 +31,24 @@ flakes.json # {"runs": [sha7 ...last 20], "tests": {"<file>::<title
Rules:
-- The attribution baseline (`ledger.sh last-green`) is the newest `runs/*.json` with `"result": "green"`;
- when none exists, the last run's `sha`; when the ledger is empty, 60 days
- back on `main`. A spec can have been red for longer than any window, so
+- `sha` is always a commit on `main`: the one whose product files (`frontend`,
+ `backend`, `proto`, `go.mod`, `go.sum`, outside `frontend/tests/e2e`) the
+ binary was built from. Local commits that touch only the e2e tree or
+ `.claude/` (an adaptation, an overlay) do not change it, so `run-suite.sh`
+ records the merge base with `origin/main` in that case. A tree whose
+ product files match no `main` commit is a branch, not main: `ledger.sh
+ record` refuses it (exit 1) instead of writing a SHA no later run can use.
+- "On `main`" is checked against `origin/main` as fetched now, else the
+ local `origin/main` when the fetch fails (an older main is still main),
+ never against HEAD: that fallback would let any branch pass. A SHA this
+ clone lacks is fetched before it is judged, and a shallow clone is first
+ deepened to 120 days (`E2E_HISTORY_DAYS`), so a missing object never reads
+ as "not on main". Picking the commit to test (`should-run.sh`) is different:
+ there HEAD is the fallback.
+- The attribution baseline (`ledger.sh last-green`) is the newest `runs/*.json` with `"result": "green"`
+ whose `sha` is on `main`; when none exists, the last such run's `sha`; when
+ there is none, 60 days back on `main`. Entries whose `sha` is not on `main`
+ (written before this rule) are skipped. A spec can have been red for longer than any window, so
`attribute.py` widens on its own to the spec file's last change when nothing
inside the window scores.
- A run records itself exactly once, after `summarize.py`, whatever the result.
diff --git a/.claude/skills/e2e-routine/references/triage.md b/.claude/skills/e2e-routine/references/triage.md
index 776810db0c..615d9b93d2 100644
--- a/.claude/skills/e2e-routine/references/triage.md
+++ b/.claude/skills/e2e-routine/references/triage.md
@@ -4,6 +4,27 @@ A red test is a signal, not a finding. The routine turns each one into exactly
one class with evidence, and each class has one action and one owner. Work the
table top to bottom; the first row whose signal and verification both hold wins.
+## Buckets: how `summarize.py` sorts a test
+
+A test's evidence is its anomalous attempts: those whose status differs from
+the test's `expectedStatus` (an ordinary test expects `passed`, a `test.fail()`
+lock expects `failed`; a timeout or interruption is never the expected
+status). The bucket follows from which attempts were anomalous:
+
+| Expected | Attempts (3 at most) | Bucket |
+|---|---|---|
+| passed | all passed | passed |
+| passed | every attempt anomalous | `failed` |
+| passed | anomalous, then a pass | `flaky` |
+| failed (lock) | all failed | `lock_holding` |
+| failed (lock) | all passed | `lock_flipped` |
+| failed (lock) | at least one pass and at least one failure | `lock_unstable` |
+| failed (lock) | no pass, at least one timeout or interruption | `failed` if no attempt failed as locked, else `flaky`: the test now hangs, which is not the locked bug |
+
+The error and the screenshots come from the latest anomalous attempt. An
+unexpected pass has no error message, so its error reads "Expected to fail,
+but passed".
+
## Classes
| Class | Signal (from summary.json) | Verification that must hold | Action | Owner |
@@ -12,8 +33,9 @@ table top to bottom; the first row whose signal and verification both hold wins.
| SERVER-CRASH | a `failed` row "Bytebase server crashed": the client lost the server (what INFRA calls `server-lost` or boot) and `server.log` holds a Go crash, a `panic:` or `fatal error:` line followed by a goroutine dump. The tests that died with it are in `lost_to_crash`, not triaged | the crash line and its first backend frame (`frame`) in `server.log`; `first_test` is the first test that lost the server, so the crash happened during it or the test before | PRODUCT issue, priority Urgent (every later test is blocked): `attribute.py --spec <frame file> --since "$(bash .claude/skills/e2e-routine/scripts/ledger.sh last-green)"` for the culprit; reproduce with `first_test` alone | culprit PR author via the frame's file; else the area owner of that file |
| INFRA | `infra` set: preflight or build failed (`--pre-run-failure`), no report, 0 tests ran with a boot/license/browser/psql signature, or the failed tests are environment casualties: at least 5 failures and 80% of them carry an environment signature (connection or boot is `server-lost`, unless `server.log` shows the server crashed: then SERVER-CRASH; browser, psql, toolchain or license keeps its own name), or the setup project failed with one and nothing else failed. The casualties move to `lost_to_infra` and are not triaged; failures without a signature stay in `failed` and are triaged row by row. A global-setup error with no such signature is a `failed` row named "global setup", and an error outside any test after tests ran (global teardown, a worker crash) is a `failed` row named "suite-level error"; both are triaged and attributed like any test. A nonzero Playwright exit that nothing else explains is infra (`unexplained-exit`): a run is green only when Playwright says so | `server.log` or `playwright.log` shows the boot, license, browser or psql error | No product issue. Record the ledger entry as `infra`; say what failed and the fix (environment, allowlist, setup script). Two consecutive infra runs: one BOT issue for the framework | framework owner |
| INCOMPLETE | `incomplete` set: the global timeout or a crash ended the run with tests never reached (nonzero exit, no failed test) | `playwright.log` ends with the timeout line or an abort; `summary.md` names how many tests never ran | Not green. Record `incomplete`; the next run retries the same SHA. Report the wall-clock split (build, suite) so the cap or a hung spec can be fixed; a hung spec that recurs becomes a BOT issue | framework owner |
-| LOCK-FLIPPED | `lock_flipped`: a `test.fail()` lock passed | The describe title names an issue; that issue is Done, or its fix PR is in `since..main` | Adaptation PR removes `test.fail()`; comment on the issue when filing is enabled | none |
-| FLAKE | `flaky`: failed, then passed on a retry | nothing more for this run | Append to `flakes.json` (ledger.sh does it). Issue when the same test is flaky in 3 of the last 5 runs, with the five error lines | framework owner, unless the error names a product race (then PRODUCT) |
+| LOCK-UNSTABLE | `lock_unstable`: a `test.fail()` lock passed on some attempts and failed on others | rerun the one test alone, `-g "<title>" --retries=0 --repeat-each=10`, and count the passes; open the screenshot of a passing attempt and the trace of a failing one | The locked bug is intermittent, a race. Never remove the lock and never file a twin: comment on the locked issue with the pass rate (or propose the comment when filing is disabled), and reopen-worthy evidence goes there. If that issue is Done, say so: its fix is incomplete | the locked issue's assignee |
+| LOCK-FLIPPED | `lock_flipped`: a `test.fail()` lock passed on every attempt | The describe title names an issue; that issue is Done, or its fix PR is in `since..main` | Adaptation PR removes `test.fail()`; comment on the issue when filing is enabled | none |
+| FLAKE | `flaky`: failed, then passed on a retry | the isolated rerun (`--retries=0 --repeat-each=2`) is green. A `flaky` row that is red in isolation is not a flake: work it as a `failed` row (SPEC-ROT or PRODUCT). An error with `strict mode violation` is a locator that matches several elements, a spec defect and never timing: SPEC-ROT | Append to `flakes.json` (ledger.sh does it). Issue when the same test is flaky in 3 of the last 5 runs, with the five error lines | framework owner, unless the error names a product race (then PRODUCT) |
| TEST-ISOLATION | `failed`, but passes when its file runs alone (`--repeat-each=2`) | the isolated rerun is green | Adaptation PR when the leftover state is obvious (fixture not idempotent, order dependence); else a BOT issue for the framework | framework owner |
| SPEC-ROT | `failed`, red in isolation | a commit in `since..main` changed the asserted text, locator, route or flow **on purpose**: its PR title/body describes the new behavior, and the new screen is coherent (no error, no dead end, no permission or masking contradiction) | Adaptation PR: update the spec to the new behavior, citing the PR in the commit message. No Linear issue | the product PR's author reviews the adaptation |
| PRODUCT | `failed`, red in isolation | no intentional change explains it, **or** the new behavior breaks an invariant (see below) | Linear BOT issue, or a proposal when filing is disabled. In the adaptation PR, lock it with `test.describe('… (BUG BOT-n)')` + `test.fail()` only once the issue exists | culprit PR author via owners.json; else area default; else unassigned with candidates listed |
@@ -25,12 +47,12 @@ page shows an error, an empty state in place of data, or an action with no
effect; export or copy policy is not enforced; the workspace or project cannot
be reached after sign-in.
-## Verification procedure for a `failed` entry
+## Verification procedure for a `failed`, `flaky` or `lock_unstable` entry
-1. **Isolate.** `run-suite.sh start <run-id>-iso-N -- <spec file> --retries=0 --repeat-each=2`, then `wait`. Green here means TEST-ISOLATION.
-2. **Read the evidence, not the title.** The entry's `evidence` in `summary.json` names the files of the failed attempt it reports: `screenshots`, one per open page (open them and look: they are what a user would have seen), `error_context`, the aria snapshot of the page at failure (the real DOM), and `trace` when a retry ran. In `playwright.log`, the attempt's error with the locator and the expected vs received text. In `server.log`, 4xx/5xx or panics around the timestamp.
+1. **Isolate.** `run-suite.sh start <run-id>-iso-N -- <spec file> --retries=0 --repeat-each=2`, then `wait`. For a `failed` row, green here means TEST-ISOLATION; for a `flaky` row, red here means it is not a flake (see FLAKE). A `lock_unstable` row reruns the one test ten times instead (LOCK-UNSTABLE).
+2. **Read the evidence, not the title.** The entry's `evidence` in `summary.json` names the files of the anomalous attempt it reports (see "Buckets"): `screenshots`, one per open page (open them and look: they are what a user would have seen), `error_context`, the aria snapshot of the page at failure (the real DOM), and `trace` when a retry ran. In `playwright.log`, the attempt's error with the locator and the expected vs received text. In `server.log`, 4xx/5xx or panics around the timestamp.
3. **Pin the assertion.** Open the spec at the failing line. Write down the exact UI text or locator and what the test expected.
-4. **Find the component.** `attribute.py --text "<quoted text>"` resolves the text through `frontend/src/locales/en*.json` to its key and the files that use it, and lists the commits in `since..origin/main` that touched them or the text. Read the top candidate's PR (`gh pr view <n> --json title,body`).
+4. **Find the component.** `attribute.py --text "<quoted text>"` resolves the text through `frontend/src/locales/en*.json` to its key and the files that use it, and lists the commits in `since..HEAD` that touched them or the text. It also lists commits to the spec file itself and to `frontend/tests/e2e/framework/`, marked as test changes: a test commit that broke its own test is SPEC-ROT, adapted and reviewed by its author. It exits 2 when `--since` or `--head` is not a commit here, rather than reporting an empty range. Read the top candidate's PR (`gh pr view <n> --json title,body`).
5. **Decide.** Apply the table. A decision cites: the attempt error line, the screenshot and what it shows, the spec line, and the commit (SPEC-ROT, PRODUCT) or the isolated rerun (TEST-ISOLATION).
6. **Weigh before filing.** Validity (does the evidence show it), frequency (every run, or one race), return (what a fix buys). A valid, rare, cosmetic failure is still recorded in the report with the decision "not worth an issue" and why; it is never silently dropped.
diff --git a/.claude/skills/e2e-routine/scripts/attribute.py b/.claude/skills/e2e-routine/scripts/attribute.py
index e653be2e5b..d7de34c2fd 100755
--- a/.claude/skills/e2e-routine/scripts/attribute.py
+++ b/.claude/skills/e2e-routine/scripts/attribute.py
@@ -1,15 +1,17 @@
#!/usr/bin/env python3
-"""Rank the product commits that could explain a red spec, and name an owner.
+"""Rank the commits that could explain a red spec, and name an owner.
attribute.py --spec frontend/tests/e2e/sql-editor/sql-editor-result.spec.ts \
--since <last-green-sha> [--text "Request export" ...] [--json]
-Candidates are the non-merge commits in <since>..origin/main that touch the
-area's source paths (owners.json), the component files that render any quoted
-UI text (found through the locale files), or whose diff adds/removes the text
-itself (git log -S). Score = 3 per text hit + 1 per path hit. The owner is the
-top candidate's author mapped through owners.json to a Linear user name, else
-the area default.
+Candidates are the non-merge commits in <since>..HEAD that touch the area's
+source paths (owners.json), the component files that render any quoted UI
+text (found through the locale files), or whose diff adds/removes the text
+itself (git log -S), plus the test changes: commits to the spec itself and to
+the e2e framework. Score = 3 per text hit or spec change, 2 per rendering
+file, 1 per area or framework path. The owner is the top candidate's author
+mapped through owners.json to a Linear user name, else the area default. Exits
+with an error when --since or --head is not a commit in this clone.
"""
import argparse
import glob
@@ -114,13 +116,21 @@ def main():
ap = argparse.ArgumentParser()
ap.add_argument("--spec", required=True)
ap.add_argument("--since", required=True)
- # HEAD is the tested checkout. origin/main can move during a 40-60 minute
+ # HEAD is the tested checkout. origin/main can move during a 15-45 minute
# run, and a commit merged after the test cannot have caused its failure.
ap.add_argument("--head", default="HEAD")
ap.add_argument("--text", action="append", default=[])
ap.add_argument("--json", action="store_true")
a = ap.parse_args()
a.spec = repo_spec_path(a.spec)
+ # An unknown SHA makes every git query below come back empty, which would
+ # read as "no commit explains it" and fall through to the area default.
+ for name, ref in (("--since", a.since), ("--head", a.head)):
+ if subprocess.run(["git", "-C", REPO, "rev-parse", "-q", "--verify", f"{ref}^{{commit}}"],
+ capture_output=True).returncode != 0:
+ print(f"attribute.py: {name} {ref!r} is not a commit in this clone; "
+ "use `ledger.sh last-green` or a SHA on main", file=sys.stderr)
+ sys.exit(2)
owners = json.load(open(os.path.join(SKILL_DIR, "owners.json")))
rng = f"{a.since}..{a.head}"
@@ -144,6 +154,11 @@ def main():
if source_mode:
add(commits(rng, [a.spec]), 3, f"changes {a.spec}, the file that fails")
+ else:
+ # A test can break itself: a new locator or step in the spec, or a
+ # framework helper it calls. These are test changes (SPEC-ROT).
+ add(commits(rng, [a.spec]), 3, "test change: edits this spec")
+ add(commits(rng, [f"{E2E_ROOT}/framework"]), 1, "test change: edits the e2e framework")
# An empty path list would make git log return every commit in the range.
if area_cfg["sources"]:
add(commits(rng, area_cfg["sources"]), 1, f"touches {area} sources")
diff --git a/.claude/skills/e2e-routine/scripts/build.sh b/.claude/skills/e2e-routine/scripts/build.sh
index 839514d998..e9e56d4a18 100755
--- a/.claude/skills/e2e-routine/scripts/build.sh
+++ b/.claude/skills/e2e-routine/scripts/build.sh
@@ -22,7 +22,7 @@ stamp="$(git rev-parse HEAD) $(git hash-object "$SERVER_BUILD")"
# Any uncommitted change that can reach the binary invalidates the cached
# build: the whole frontend tree (package.json, lockfile, Tailwind and Vite
# config, plugins), not only frontend/src. The e2e specs do not reach it.
-dirty="$(git status --porcelain -- frontend backend proto go.mod go.sum ':!frontend/tests/e2e' | head -1)"
+dirty="$(git status --porcelain -- "${PRODUCT_PATHSPEC[@]}" | head -1)"
timing="$RESULTS/build-timing.txt"
if [ -x "$BIN" ] && [ -f "$STAMP" ] && [ "$(cat "$STAMP")" = "$stamp" ] && [ -z "$dirty" ] && [ "${FORCE_BUILD:-0}" != 1 ]; then
@@ -42,8 +42,14 @@ phase() { # phase <name> <cmd...>
printf '%-16s %5ss\n' "$name" "$dt" | tee -a "$timing" >&2
}
-phase pnpm-install pnpm --dir frontend install --frozen-lockfile
-phase pnpm-release pnpm --dir frontend release
+# pnpm from inside frontend/: corepack picks the version by the working
+# directory, and only there does it see packageManager (pnpm 11). Run from
+# the repo root it falls back to the global pnpm. A node_modules that another
+# pnpm laid out must be rebuilt, and pnpm asks before deleting it, which
+# aborts without a TTY; it is generated, so the build may purge it.
+fe_pnpm() { (cd frontend && pnpm "$@"); }
+phase pnpm-install fe_pnpm install --frozen-lockfile --config.confirmModulesPurge=false
+phase pnpm-release fe_pnpm release
phase go-build bash "$SERVER_BUILD" "$BIN"
echo "$stamp" >"$STAMP"
log "built $BIN from $stamp"
diff --git a/.claude/skills/e2e-routine/scripts/ledger.sh b/.claude/skills/e2e-routine/scripts/ledger.sh
index 92a883ae22..32a5e9ea21 100755
--- a/.claude/skills/e2e-routine/scripts/ledger.sh
+++ b/.claude/skills/e2e-routine/scripts/ledger.sh
@@ -32,11 +32,13 @@ last-result) git show "$(read_ref):last-run.json" 2>/dev/null | python3 -c 'impo
show) git show "$(read_ref):last-run.json" 2>/dev/null || echo '{}' ;;
flakes) git show "$(read_ref):flakes.json" 2>/dev/null || echo '{}' ;;
last-green)
- # The attribution baseline (references/ledger.md): the newest green run's
- # SHA, else the last run's, else main 60 days back. Computed on each call so
- # a phase never depends on a variable an earlier shell set.
+ # The attribution baseline (references/ledger.md): the newest green run on
+ # main, else the newest run on main, else main 60 days back. Computed on each
+ # call so a phase never depends on a variable an earlier shell set.
r="$(read_ref)"
- sha="$(python3 - "$r" <<'PY'
+ main="$(main_ref_strict)" || die "no origin/main to check the ledger against"
+ # Every entry as "<result> <sha>", newest first; on_main judges each one.
+ entries="$(python3 - "$r" <<'PY'
import json, subprocess, sys
ref = sys.argv[1]
def git(*a):
@@ -47,26 +49,38 @@ for path in git("ls-tree", "--name-only", f"{ref}:runs").split():
runs.append(json.loads(git("show", f"{ref}:runs/{path}")))
except ValueError:
pass
-green = [e for e in runs if e.get("result") == "green" and e.get("sha")]
-if green:
- print(max(green, key=lambda e: (e.get("date", ""), e.get("run_id", "")))["sha"])
-else:
- try:
- print(json.loads(git("show", f"{ref}:last-run.json")).get("sha", ""))
- except ValueError:
- print("")
+for e in sorted(runs, key=lambda e: (e.get("date", ""), e.get("run_id", "")), reverse=True):
+ if e.get("sha"):
+ print(e.get("result", "?"), e["sha"])
PY
)"
- [ -n "$sha" ] || sha="$(git rev-list -1 --before='60 days ago' "$(main_sha)")"
+ green="" newest=""
+ while read -r result s; do
+ [ -n "$s" ] && on_main "$s" "$main" || continue
+ [ -n "$newest" ] || newest="$s"
+ if [ "$result" = green ]; then green="$s"; break; fi
+ done <<<"$entries"
+ sha="${green:-$newest}"
+ [ -n "$sha" ] || sha="$(git rev-list -1 --before='60 days ago' "$main")"
+ [ -n "$sha" ] || die "no baseline: no ledger entry on main and no main commit 60 days back"
echo "$sha"
;;
record)
RESULTS="${2:?results dir}"; note="${3:-}"
[ -f "$RESULTS/summary.json" ] || die "$RESULTS/summary.json missing; run summarize.py first"
+ sha="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("meta", {}).get("sha", ""))' "$RESULTS/summary.json")"
+ # Only a main commit may enter the ledger (references/ledger.md); a dry run
+ # from a branch records with a warning, since it never reaches the remote.
+ if ! on_main "$sha"; then
+ dry_run || die "run tested ${sha:-no SHA}, which is not a commit on main; not recorded"
+ log "WARNING: ${sha:-no SHA} is not on main; recording on the dry-run ref only"
+ fi
tmp="$(mktemp -d)"
if dry_run && has_ref "$dry_ref"; then
git worktree add -q --detach "$tmp" "$dry_ref"
- elif fetch_ledger; then
+ elif fetch_ledger || has_ref "$remote_ref"; then
+ # A failed fetch keeps the last fetched ledger rather than starting an
+ # empty one; a push from it is then rejected, never a rewrite.
git worktree add -q --detach "$tmp" "$remote_ref"
else
root="$(git "${ident[@]}" commit-tree "$(git hash-object -t tree /dev/null)" -m "e2e ledger root")"
@@ -94,6 +108,7 @@ else:
entry = {"run_id": meta.get("run_id"), "sha": sha, "date": day, "result": result, "totals": t,
"failed": [f"{e['file']}::{e['title']}" for e in s.get("failed", [])],
"flaky": [f"{e['file']}::{e['title']}" for e in s.get("flaky", [])],
+ "lock_unstable": [f"{e['file']}::{e['title']}" for e in s.get("lock_unstable", [])],
"infra": s.get("infra"), "note": note, "run_url": run_url}
os.makedirs(os.path.join(dest, "runs"), exist_ok=True)
# The run id keeps a retry of the same SHA on the same day from overwriting
@@ -114,7 +129,7 @@ if not os.path.exists(os.path.join(dest, "README.md")):
print(f"recorded {result} run {entry['run_id']} @ {sha[:7]}")
PY
(
- cd "$tmp" && git add -A && git "${ident[@]}" commit -q -m "e2e ledger: $(date -u +%F) $(git -C "$REPO_ROOT" rev-parse --short HEAD)" \
+ cd "$tmp" && git add -A && git "${ident[@]}" commit -q -m "e2e ledger: $(date -u +%F) ${sha:0:10}" \
&& if dry_run; then
git update-ref "$dry_ref" HEAD
log "E2E_LEDGER_PUSH=0: not pushing; recorded on $dry_ref, which ledger.sh reads while E2E_LEDGER_PUSH=0"
diff --git a/.claude/skills/e2e-routine/scripts/lib.sh b/.claude/skills/e2e-routine/scripts/lib.sh
index 0065fbb38c..177e7c5cf8 100755
--- a/.claude/skills/e2e-routine/scripts/lib.sh
+++ b/.claude/skills/e2e-routine/scripts/lib.sh
@@ -5,7 +5,9 @@ set -o pipefail
E2E_SKILL_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
REPO_ROOT="$(cd "$E2E_SKILL_DIR/../../.." && pwd)"
-E2E_RESULTS_ROOT="${E2E_RESULTS_ROOT:-/tmp/e2e-routine}"
+# One results root per checkout: run-dir.sh keeps its CURRENT pointer there,
+# and two worktrees running at once must not take over each other's run.
+E2E_RESULTS_ROOT="${E2E_RESULTS_ROOT:-/tmp/e2e-routine/$(basename "$REPO_ROOT")-$(printf '%s' "$REPO_ROOT" | cksum | cut -d' ' -f1)}"
E2E_LEDGER_BRANCH="${E2E_LEDGER_BRANCH:-claude/e2e-ledger}"
# The cloud image ships Node 20-22 on PATH; setup-env.sh installs 24 under /opt/node24.
@@ -52,8 +54,9 @@ e2e_license_available() {
if e2e_license >/dev/null 2>&1; then echo yes; else echo no; fi
}
-# origin/main as fetched now. When the fetch fails it is HEAD, never an older
-# origin/main left by an earlier fetch, which would test or skip a stale SHA.
+# The commit to test: origin/main as fetched now. When the fetch fails it is
+# HEAD, never an older origin/main left by an earlier fetch, which would test
+# or skip a stale SHA. Not for "is X on main": see main_ref_strict.
main_sha() {
if git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null; then
git -C "$REPO_ROOT" rev-parse origin/main
@@ -63,4 +66,58 @@ main_sha() {
fi
}
+# The main to judge membership against (references/ledger.md): fetched now,
+# else the local origin/main, since an older main is still main. Never HEAD,
+# which would let any branch pass. Exit 1 when there is no origin/main at all.
+main_ref_strict() {
+ git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null \
+ || log "fetch of origin/main failed; checking against the local origin/main"
+ git -C "$REPO_ROOT" rev-parse -q --verify 'origin/main^{commit}'
+}
+
+# A shallow clone lacks the history that ancestry checks and attribution
+# need; deepen it once to E2E_HISTORY_DAYS of main.
+ensure_history() {
+ [ "$(git -C "$REPO_ROOT" rev-parse --is-shallow-repository 2>/dev/null)" = true ] || return 0
+ git -C "$REPO_ROOT" fetch -q --shallow-since="${E2E_HISTORY_DAYS:-120} days ago" origin main 2>/dev/null \
+ || log "could not deepen the shallow clone; older ledger entries may not resolve"
+}
+
+# on_main <sha> [main]: the SHA is a commit on main. A SHA this clone lacks is
+# fetched first (GitHub serves any published commit), so an absent object is
+# never read as "not on main"; one GitHub does not have fails. In a shallow
+# clone a fetched commit is not yet linked to main, so main is deepened to
+# just before the commit's date and the check is made again.
+on_main() {
+ local sha="$1" main="${2:-}" when
+ [ -n "$sha" ] || return 1
+ [ -n "$main" ] || main="$(main_ref_strict)" || return 1
+ ensure_history
+ git -C "$REPO_ROOT" cat-file -e "$sha^{commit}" 2>/dev/null \
+ || git -C "$REPO_ROOT" fetch -q origin "$sha" 2>/dev/null || return 1
+ git -C "$REPO_ROOT" merge-base --is-ancestor "$sha" "$main" && return 0
+ [ "$(git -C "$REPO_ROOT" rev-parse --is-shallow-repository 2>/dev/null)" = true ] || return 1
+ when="$(git -C "$REPO_ROOT" log -1 --format=%ct "$sha")" || return 1
+ git -C "$REPO_ROOT" fetch -q --shallow-since="$((when - 86400))" origin main 2>/dev/null || return 1
+ git -C "$REPO_ROOT" merge-base --is-ancestor "$sha" "$main"
+}
+
+# What the binary is built from; the e2e tree is excluded because specs are
+# read at run time and an adaptation changes only them.
+PRODUCT_PATHSPEC=(frontend backend proto go.mod go.sum ':!frontend/tests/e2e')
+
+# The main commit the checkout's product files come from (references/ledger.md):
+# HEAD when it is on main, the merge base when the commits above it, and any
+# uncommitted change, touch only the e2e tree or .claude/. Exit 1 when the
+# product files match no main commit.
+tested_sha() {
+ local main base
+ main="$(main_ref_strict)" || return 1
+ ensure_history
+ base="$(git -C "$REPO_ROOT" merge-base HEAD "$main" 2>/dev/null)" || return 1
+ git -C "$REPO_ROOT" diff --quiet "$base" HEAD -- "${PRODUCT_PATHSPEC[@]}" || return 1
+ [ -z "$(git -C "$REPO_ROOT" status --porcelain -- "${PRODUCT_PATHSPEC[@]}" | head -1)" ] || return 1
+ echo "$base"
+}
+
run_id_now() { date -u +%Y%m%dT%H%M%SZ; }
diff --git a/.claude/skills/e2e-routine/scripts/preflight.sh b/.claude/skills/e2e-routine/scripts/preflight.sh
index a9bd6e8c6e..5b9a4d27e4 100755
--- a/.claude/skills/e2e-routine/scripts/preflight.sh
+++ b/.claude/skills/e2e-routine/scripts/preflight.sh
@@ -13,8 +13,12 @@ bad() { printf 'FAIL %-28s %s\n' "$1" "$2"; fail=1; }
v=$(node --version 2>/dev/null || true)
case "$v" in v2[4-9].*|v[3-9]*) ok node "$v" ;; "") bad node "missing" ;; *) warn node "$v (CI uses 24.18.1; build_init.sh requires >= 24.15)" ;; esac
-v=$(pnpm --version 2>/dev/null || true)
-[ -n "$v" ] && ok pnpm "$v (packageManager pins 11.17.0; pnpm self-switches)" || bad pnpm "missing"
+# The pnpm that build.sh runs: the one corepack picks inside frontend/.
+want=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("packageManager","").split("@")[-1].split("+")[0])' frontend/package.json 2>/dev/null)
+v=$(cd frontend && pnpm --version 2>/dev/null || true)
+if [ -z "$v" ]; then bad pnpm "missing"
+elif [ -n "$want" ] && [ "$v" != "$want" ]; then bad pnpm "$v inside frontend/, but packageManager pins $want"
+else ok pnpm "$v inside frontend/ (packageManager pins ${want:-?})"; fi
v=$(go version 2>/dev/null | awk '{print $3}')
[ -n "$v" ] && ok go "$v (go.mod 1.27.1; GOTOOLCHAIN=$GOTOOLCHAIN)" || bad go "missing"
diff --git a/.claude/skills/e2e-routine/scripts/run-suite.sh b/.claude/skills/e2e-routine/scripts/run-suite.sh
index 1b4a6d41e7..ee191d6707 100755
--- a/.claude/skills/e2e-routine/scripts/run-suite.sh
+++ b/.claude/skills/e2e-routine/scripts/run-suite.sh
@@ -10,7 +10,7 @@
# JSON reporter), test-results/ (traces, screenshots, error-context.md),
# exit-code, meta.json. Every test gets three attempts (--retries=2): red only
# when all three fail, flaky when a retry passes. A run is capped at
-# E2E_GLOBAL_TIMEOUT_MS (default 2 h; a healthy run takes about 1 h) because a
+# E2E_GLOBAL_TIMEOUT_MS (default 2 h; a green run took 43 min on the cloud VM) because a
# night of hung tests otherwise multiplies 3 attempts by each test's timeout.
# The license comes from lib.sh's e2e_license and is exported only into the
# detached process.
@@ -47,12 +47,17 @@ EOF
# A JSON encoder writes the metadata: Playwright arguments can hold quotes,
# backslashes or newlines that raw interpolation would turn into invalid JSON.
- python3 - "$RESULTS/meta.json" "$run_id" "$(git rev-parse HEAD)" "$(git rev-parse --short HEAD)" \
+ # `sha` is the main commit under test (references/ledger.md), not HEAD: an
+ # adaptation or overlay commit on top must not end up in the ledger.
+ if sha="$(tested_sha)"; then on_main=1; else sha="$(git rev-parse HEAD)"; on_main=0
+ log "product files match no main commit; recording HEAD $sha (ledger.sh record will refuse it)"
+ fi
+ python3 - "$RESULTS/meta.json" "$run_id" "$sha" "$on_main" "$(git rev-parse HEAD)" \
"$(date -u +%FT%TZ)" "$(hostname)" "$@" <<'PY'
import json, sys
-path, run_id, sha, short_sha, started_at, host, *args = sys.argv[1:]
-json.dump({"run_id": run_id, "sha": sha, "short_sha": short_sha, "started_at": started_at,
- "host": host, "args": args}, open(path, "w"))
+path, run_id, sha, on_main, head, started_at, host, *args = sys.argv[1:]
+json.dump({"run_id": run_id, "sha": sha, "short_sha": sha[:10], "on_main": on_main == "1", "head": head,
+ "started_at": started_at, "host": host, "args": args}, open(path, "w"))
PY
# Playwright arguments reach the runner shell-quoted word by word, so a
@@ -110,11 +115,15 @@ status)
RESULTS="${1:?results dir}"
lg="$RESULTS/playwright.log"
[ -f "$lg" ] || { echo "no log yet"; exit 0; }
- done_n=$(grep -cE '^\s+[✓✘-]\s' "$lg" 2>/dev/null || echo 0)
- fail_n=$(grep -cE '^\s+✘\s' "$lg" 2>/dev/null || echo 0)
- last=$(grep -E '^\s+[✓✘-]\s' "$lg" | tail -1 | cut -c1-110)
+ # A list-reporter result line carries a run number and a [project] tag;
+ # Playwright's indented call-log lines ("- waiting for ...") do not.
+ # grep -c prints 0 itself when nothing matches, so no fallback echo.
+ result_line='^\s+[✓✘-]\s+[0-9]+ \['
+ done_n=$(grep -cE "$result_line" "$lg" 2>/dev/null; true)
+ fail_n=$(grep -cE '^\s+✘\s+[0-9]+ \[' "$lg" 2>/dev/null; true)
+ last=$(grep -E "$result_line" "$lg" | tail -1 | cut -c1-110)
state=$([ -f "$RESULTS/exit-code" ] && echo "done(exit $(cat "$RESULTS/exit-code"))" || echo running)
- echo "$(date -u +%H:%M:%SZ) $state lines=$done_n red=$fail_n last: $last"
+ echo "$(date -u +%H:%M:%SZ) $state results=${done_n:-0} red=${fail_n:-0} last: $last"
;;
*)
echo "usage: run-suite.sh start [run-id] [-- playwright args] | wait <dir> [max-min] | status <dir>" >&2; exit 2 ;;
diff --git a/.claude/skills/e2e-routine/scripts/summarize.py b/.claude/skills/e2e-routine/scripts/summarize.py
index f65724c2f9..1888fc39b9 100755
--- a/.claude/skills/e2e-routine/scripts/summarize.py
+++ b/.claude/skills/e2e-routine/scripts/summarize.py
@@ -3,10 +3,11 @@
summarize.py <results-dir> -> writes summary.json and summary.md there, prints summary.md
-Buckets (see references/triage.md for what to do with each):
+Buckets (the truth table and the action for each are in references/triage.md):
failed unexpected on every attempt -> verify: product regression or spec rot
- flaky failed, then passed on a retry -> flakes ledger
- lock_flipped test.fail() lock that now passes -> the bug was fixed; remove the lock
+ flaky failed, then passed on a retry -> rerun alone; a flake only if green there
+ lock_flipped test.fail() lock passed every attempt -> the bug was fixed; remove the lock
+ lock_unstable test.fail() lock passed some attempts -> the locked bug is intermittent; keep the lock
lock_holding test.fail() lock still failing -> not red
infra no test ran, or every test died on a boot/license/browser signature
(a lost server whose server.log holds a Go crash is a failed row instead)
@@ -93,15 +94,50 @@ def first_line(msg):
return ""
-def pick_evidence(results):
- """The files that show the failure the entry reports: the screenshots (one
- per open page) and aria snapshot of the latest failed attempt that has
- them, and the trace of a failed attempt (Playwright traces the first retry
- only). A passing retry's trace shows the pass, so it is never the evidence."""
- failed = [(i, r) for i, r in enumerate(results) if r.get("status") not in ("passed", "skipped")]
+def anomalous(results, expected):
+ """The attempts whose status is not the test's expected one: the failures
+ of an ordinary test, the passes (or timeouts) of a test.fail() lock
+ (references/triage.md, "Buckets")."""
+ return [(i, r) for i, r in enumerate(results) if r.get("status") not in (expected, "skipped")]
+
+
+def attempt_error(results, expected):
+ """The latest anomalous attempt's error as (first line, whole message). An
+ unexpected pass carries no error, so it is named for what it is."""
+ odd = anomalous(results, expected)
+ if not odd:
+ # A holding lock: what it still fails with, as expected.
+ odd = [(i, r) for i, r in enumerate(results) if (r.get("error") or {}).get("message")]
+ if not odd:
+ return "", ""
+ r = odd[-1][1]
+ msg = re.sub(r"\x1b\[[0-9;]*m", "", (r.get("error") or {}).get("message") or "")
+ if msg:
+ return first_line(msg), msg
+ if r.get("status") == "passed":
+ return f"Expected to fail, but passed (attempt {odd[-1][0] + 1} of {len(results)})", ""
+ return r.get("status") or "", ""
+
+
+def hint(bucket, message, expected):
+ # Playwright puts the strict-mode cause lines below the assertion's first line.
+ if "strict mode violation" in message:
+ return "a locator matches several elements: a spec defect, never timing (SPEC-ROT)"
+ if expected == "failed" and bucket in ("failed", "flaky"):
+ return "a test.fail() lock timed out: the test now hangs, which is not the locked bug"
+ return None
+
+
+def pick_evidence(results, expected="passed"):
+ """The files that show what the entry reports: the screenshots (one per
+ open page) and aria snapshot of the latest anomalous attempt that has
+ them, and the trace of an anomalous attempt (Playwright traces the first
+ retry only). An attempt that went as expected shows nothing wrong, so it is
+ never the evidence."""
+ odd = anomalous(results, expected)
def latest(name):
- for i, r in reversed(failed):
+ for i, r in reversed(odd):
paths = sorted({a["path"] for a in r.get("attachments", [])
if a.get("name") == name and a.get("path") and os.path.exists(a["path"])})
if paths:
@@ -129,15 +165,23 @@ def walk(suite, file_title, out):
def classify(test):
+ """The bucket per the truth table in references/triage.md ("Buckets")."""
status = test.get("status") # expected | unexpected | flaky | skipped
expected = test.get("expectedStatus") # passed | failed | skipped
if status == "skipped":
return "skipped"
- if status == "flaky":
- return "flaky"
- if expected == "failed":
- return "lock_holding" if status == "expected" else "lock_flipped"
- return "passed" if status == "expected" else "failed"
+ if expected != "failed":
+ return {"expected": "passed", "flaky": "flaky"}.get(status, "failed")
+ statuses = [r.get("status") for r in test.get("results", [])]
+ passes = statuses.count("passed")
+ if passes and passes == len(statuses):
+ return "lock_flipped"
+ if passes:
+ return "lock_unstable"
+ if status == "expected":
+ return "lock_holding"
+ # No pass, yet not the expected failure on every attempt: a timeout.
+ return "flaky" if "failed" in statuses else "failed"
def infra_signature(text):
@@ -159,8 +203,8 @@ def main(results_dir):
exit_code = int(open(ep).read().strip() or 0)
summary = {"meta": meta, "exit_code": exit_code, "totals": {}, "failed": [], "flaky": [],
- "lock_flipped": [], "lock_holding": [], "skipped": [], "not_run": [], "infra": None,
- "incomplete": None, "errors": []}
+ "lock_flipped": [], "lock_unstable": [], "lock_holding": [], "skipped": [], "not_run": [],
+ "infra": None, "incomplete": None, "errors": []}
if not os.path.exists(report_path):
log_tail = ""
@@ -194,10 +238,8 @@ def main(results_dir):
if bucket == "skipped":
counts["not_run" if not deliberate else "skipped_deliberate"] += 1
results = test.get("results", [])
- # The latest attempt that carries an error: for a flaky test the final
- # attempt passed and has none, and the failure is the evidence.
- errored = [r for r in results if (r.get("error") or {}).get("message")]
- err = first_line(errored[-1]["error"]["message"]) if errored else ""
+ expected = test.get("expectedStatus") or "passed"
+ err, message = attempt_error(results, expected)
entry = {
"file": file_title,
"title": spec.get("title"),
@@ -208,13 +250,14 @@ def main(results_dir):
"duration_ms": sum(r.get("durationMs", r.get("duration", 0)) or 0 for r in results),
"error": err,
"infra_signature": infra_signature(err),
- "evidence": pick_evidence(results),
+ "evidence": pick_evidence(results, expected),
+ "hint": hint(bucket, message, expected),
"annotations": [a.get("type") for a in test.get("annotations", [])],
}
if bucket == "skipped" and not deliberate:
entry["blocked_by"] = last_failed_in_file.get(file_title)
summary["not_run"].append(entry)
- elif bucket in ("failed", "flaky", "lock_flipped", "lock_holding", "skipped"):
+ elif bucket in ("failed", "flaky", "lock_flipped", "lock_unstable", "lock_holding", "skipped"):
summary[bucket].append(entry)
stats = report.get("stats", {})
@@ -225,6 +268,7 @@ def main(results_dir):
"failed": counts["failed"],
"flaky": counts["flaky"],
"lock_flipped": counts["lock_flipped"],
+ "lock_unstable": counts["lock_unstable"],
"lock_holding": counts["lock_holding"],
"skipped": counts["skipped"],
"not_run": counts["not_run"],
@@ -380,12 +424,13 @@ def write(results_dir, s):
lines += [f"**INCOMPLETE**: {s['incomplete']['reason']}; {s['incomplete']['not_run']} test(s) never ran. Not a green run.", ""]
if t:
lines += [f"{t['passed']} passed, {t['failed']} failed, {t['flaky']} flaky, {t['lock_flipped']} lock flipped, "
- f"{t['lock_holding']} lock holding, {t.get('not_run', 0)} did not run, "
+ f"{t.get('lock_unstable', 0)} lock unstable, {t['lock_holding']} lock holding, {t.get('not_run', 0)} did not run, "
f"{t.get('skipped_deliberate', t['skipped'])} skipped on purpose, {t['tests']} total, {t['duration_s']}s", ""]
for bucket, head in (("failed", "Failed (every attempt)"), ("flaky", "Flaky (passed on retry)"),
("lost_to_infra", "Lost to the environment (not triaged; they rerun once it is fixed)"),
("lost_to_crash", "Lost to the server crash (not triaged; they rerun once the crash is fixed)"),
("lock_flipped", "Locks that now pass"),
+ ("lock_unstable", "Locks that pass only sometimes (the locked bug is intermittent)"),
("not_run", "Did not run (not covered by this run)"), ("skipped", "Skipped on purpose")):
if s.get(bucket):
lines += [f"## {head}", ""]
@@ -396,7 +441,9 @@ def write(results_dir, s):
lines.append(f" - {e['error']}")
if bucket == "not_run":
lines.append(f" - blocked by: {e.get('blocked_by') or 'the run ending early'}")
- if bucket in ("failed", "flaky", "lock_flipped"):
+ if e.get("hint"):
+ lines.append(f" - hint: {e['hint']}")
+ if bucket in ("failed", "flaky", "lock_flipped", "lock_unstable"):
lines.append(f" - screenshots: {evidence_line(e.get('evidence') or NO_EVIDENCE, results_dir)}")
lines.append("")
if s.get("errors"):
@@ -419,9 +466,17 @@ def record_pre_run_failure(results_dir, kind, detail, cause="infra"):
if os.path.exists(mp):
meta = json.load(open(mp))
else:
- repo = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..", "..", ".."))
- sha = subprocess.run(["git", "-C", repo, "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()
- meta = {"run_id": os.path.basename(os.path.normpath(results_dir)), "sha": sha, "short_sha": sha[:10]}
+ here = os.path.dirname(os.path.abspath(__file__))
+ repo = os.path.abspath(os.path.join(here, "..", "..", "..", ".."))
+ # The same rule as run-suite.sh: the main commit under test, else HEAD
+ # marked off main (references/ledger.md).
+ sha = subprocess.run(["bash", "-c", 'source "$1"; tested_sha', "_", os.path.join(here, "lib.sh")],
+ capture_output=True, text=True).stdout.strip()
+ on_main = bool(sha)
+ if not sha:
+ sha = subprocess.run(["git", "-C", repo, "rev-parse", "HEAD"], capture_output=True, text=True).stdout.strip()
+ meta = {"run_id": os.path.basename(os.path.normpath(results_dir)), "sha": sha, "short_sha": sha[:10],
+ "on_main": on_main}
json.dump(meta, open(mp, "w"))
summary = {"meta": meta, "exit_code": None, "totals": {}, "failed": [], "flaky": [],
"lock_flipped": [], "lock_holding": [], "skipped": [], "not_run": [],