Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

ci(migrations): re-check open PRs when a new head lands on main (CHOO-2689)

Wojciech Wojtyniakab8f51a24a6d
diff --git a/.github/workflows/pr-ci.yml b/.github/workflows/pr-ci.yml
index 7801e615b..d2e36deb6 100644
--- a/.github/workflows/pr-ci.yml
+++ b/.github/workflows/pr-ci.yml
@@ -21,6 +21,22 @@ name: PR CI
# Main is verified on push as well as per-PR, because a PR merged while red
# leaves main red with nothing to say so — the next branch to merge main
# inherits the failure and pays for it (CHOO-1430 reached main exactly this way).
+#
+# `actions/checkout` on a `pull_request` event checks out `refs/pull/<n>/merge`
+# by default (verified against a real run's logs), GitHub's own preview of the
+# PR merged into its base — so every job above already tests the merge result,
+# not the branch tip, as of whenever it last ran. That "as of" is the gap:
+# GitHub recomputes that preview as the base moves, but nothing re-runs the
+# workflow just because it did, and merging is not gated on the branch being
+# up to date. Two PRs chaining a migration off the same parent can each see a
+# single head, go green, and sit open; if the first merges and the second
+# never gets a new commit, its stale green check still satisfies the merge
+# button, and only the merge of the second one actually produces two heads
+# (CHOO-2689 — PRs #404 and #426, hand-fixed with merge revisions
+# `b47e0c39a1f5` and `c81f4a06d2b7`). The `migration-heads-on-merge` job below
+# re-tests every other open PR that touches migrations the moment a new one
+# lands on main, so that race shows up as a red status on the affected PR
+# immediately instead of at the next deploy.
on:
pull_request:
@@ -47,6 +63,7 @@ jobs:
console: ${{ steps.filter.outputs.console }}
gateway: ${{ steps.filter.outputs.gateway }}
artifacts: ${{ steps.filter.outputs.artifacts }}
+ migrations: ${{ steps.filter.outputs.migrations }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
@@ -74,6 +91,11 @@ jobs:
gateway:
- *shared
- 'gateway/**'
+ # Its own filter, narrower than `backend`, so the push-triggered
+ # `migration-heads-on-merge` job below only runs when it can
+ # actually find a new head — not on every backend-touching push.
+ migrations:
+ - 'core/switch_core/migrations/versions/**'
# Spans every tree, so it gets its own filter rather than riding
# any one of them. Includes the files whose versions the registry is
# checked against, so a release bump cannot land without the check.
@@ -237,6 +259,33 @@ jobs:
- name: Build (tsc + vite)
run: npm run build
+ migration-heads-on-merge:
+ name: Migration heads on merge
+ # Only meaningful right after a new migration lands on main — a PR still
+ # open at that moment is the one whose stale green check could let a
+ # second head slip through (see the comment at the top of this file and
+ # scripts/check_migration_heads_for_open_prs.py for the full story,
+ # CHOO-2689). Reads revision files as plain text and never imports them
+ # (see the script's own docstring for why), so it needs no database and no
+ # Python environment for the project — cheap enough to run on every push
+ # that touches the migrations directory.
+ needs: changes
+ if: ${{ github.event_name == 'push' && needs.changes.outputs.migrations == 'true' }}
+ runs-on: ubuntu-latest
+ permissions:
+ contents: read
+ pull-requests: read
+ statuses: write
+ steps:
+ - name: Checkout
+ uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+
+ - name: Re-check open PRs that touch migrations
+ env:
+ GH_TOKEN: ${{ github.token }}
+ RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+ run: python3 scripts/check_migration_heads_for_open_prs.py
+
gitleaks:
name: Secret scan (gitleaks)
# Scans a PR's commit range, which only exists on a pull_request event: on a
diff --git a/scripts/check_migration_heads_for_open_prs.py b/scripts/check_migration_heads_for_open_prs.py
new file mode 100644
index 000000000..e698e2869
--- /dev/null
+++ b/scripts/check_migration_heads_for_open_prs.py
@@ -0,0 +1,257 @@
+#!/usr/bin/env python3
+"""Re-check the migration chain for open PRs against the just-updated base.
+
+`core/tests/switch_core/test_migration_chain.py` already asserts the chain has
+one head, and `actions/checkout` already runs it against the merge-preview
+commit (`refs/pull/<n>/merge`) rather than the PR's raw branch tip — so a PR
+that adds a migration is tested against the base as it looked when the PR's
+own CI last ran.
+
+That is not the same moment as "right before merge". Two PRs opened off the
+same parent can each see one head, go green, and sit open. If the first one
+merges and nothing pushes a new commit to the second, GitHub does not re-run
+its CI just because its target moved — the merge-preview ref updates on
+GitHub's side, but the check recorded against the PR's head commit does not.
+The second PR still shows the old green result and can merge on it, and only
+then does the chain end up with two heads (CHOO-2689; see PRs #404 and #426,
+fixed by hand with merge revisions `b47e0c39a1f5` and `c81f4a06d2b7`).
+
+This script runs on every push to main that touches the migrations directory
+— i.e. right after a new head lands — and re-checks every other open,
+same-repo PR that also touches migrations: it reads that PR's added revision
+files alongside the base as it now stands and reports whether the combination
+still has exactly one head, posting the outcome as a commit status on the
+PR's own head commit. A PR that would create a second head goes red
+immediately, without needing a new commit or its own CI run.
+
+Deliberately does not use Alembic here, and imports no revision file. The
+obvious way to compute a revision graph is `alembic.script.ScriptDirectory`,
+which is what the real test file uses — but it builds the graph by
+*importing* every file under `versions/`, running whatever module-level code
+that file contains. That is fine for the file under review in a PR's own CI:
+by the time this repo's checkout step fetches it, it's the PR's own code
+being tested. It stops being fine here: this job's input is a file from some
+*other* open pull request that nobody has approved, this repo is public so
+anyone can open one, and the job runs on a push to main holding a token that
+can write commit statuses. Importing an unreviewed file to compute a graph
+would hand that token's holder's code execution to whoever opened the PR.
+
+Instead this reads `revision`/`down_revision` straight out of the file text
+with a regex and `ast.literal_eval` — never `eval`, so the value must be a
+literal (a string, `None`, or a tuple of strings) or parsing raises instead of
+running anything. `test_revision_ids_are_unique` already reads `revision` the
+same way, for the same reason, on a smaller scale; this applies it to the
+whole graph. Nothing here is executed, so nothing here needs a database, a
+Python environment for the project, or the `alembic` library itself.
+
+It shells out to `gh` (for PR discovery and posting statuses) and `git` (to
+read a PR's files without checking them out or running them), both already
+present on GitHub-hosted runners.
+"""
+
+from __future__ import annotations
+
+import ast
+import json
+import os
+import re
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+VERSIONS_DIR = REPO_ROOT / "core" / "switch_core" / "migrations" / "versions"
+VERSIONS_PATH = "core/switch_core/migrations/versions"
+STATUS_CONTEXT = "migration-heads-on-merge"
+
+# Matches the two assignments Alembic's revision template generates, with or
+# without the type annotation it has carried across template versions
+# (`revision = ...`, `revision: str = ...`, `down_revision: str | None = ...`,
+# `down_revision: str | Sequence[str] | None = ...`). Only the right-hand side
+# is captured and handed to `ast.literal_eval`.
+_ASSIGNMENT_RE = re.compile(
+ r"^(revision|down_revision)\s*(?::[^=\n]+)?=\s*(.+)$", re.MULTILINE
+)
+
+
+class RevisionFile:
+ __slots__ = ("filename", "revision", "parents")
+
+ def __init__(self, filename: str, revision: str, parents: tuple[str, ...]) -> None:
+ self.filename = filename
+ self.revision = revision
+ self.parents = parents
+
+
+def parse_revision_file(filename: str, text: str) -> RevisionFile:
+ """Extract `revision`/`down_revision` from a migration file's text.
+
+ `ast.literal_eval` only ever produces a literal (or raises) -- it cannot
+ call a function, access an attribute, or run a statement, so this is safe
+ to point at a file nobody has reviewed.
+ """
+ values: dict[str, object] = {}
+ for match in _ASSIGNMENT_RE.finditer(text):
+ name, rhs = match.group(1), match.group(2).strip()
+ values[name] = ast.literal_eval(rhs)
+ revision = values["revision"]
+ assert isinstance(revision, str), (
+ f"{filename}: revision is not a string literal: {revision!r}"
+ )
+ down = values.get("down_revision")
+ if down is None:
+ parents: tuple[str, ...] = ()
+ elif isinstance(down, str):
+ parents = (down,)
+ elif isinstance(down, tuple):
+ parents = down # a merge revision's tuple of parents
+ else:
+ raise TypeError(
+ f"{filename}: down_revision is neither a string, tuple, nor None: {down!r}"
+ )
+ return RevisionFile(filename, revision, parents)
+
+
+def read_base_revisions() -> list[RevisionFile]:
+ return [
+ parse_revision_file(path.name, path.read_text())
+ for path in sorted(VERSIONS_DIR.glob("*.py"))
+ ]
+
+
+def run(
+ *args: str, cwd: Path = REPO_ROOT, check: bool = True
+) -> subprocess.CompletedProcess[str]:
+ return subprocess.run(
+ list(args), cwd=cwd, check=check, capture_output=True, text=True
+ )
+
+
+def open_prs_touching_migrations(repo: str) -> list[dict[str, Any]]:
+ listing = run(
+ "gh",
+ "pr",
+ "list",
+ "--repo",
+ repo,
+ "--state",
+ "open",
+ "--json",
+ "number,headRefOid",
+ )
+ candidates = []
+ for pr in json.loads(listing.stdout):
+ diff = run("gh", "pr", "diff", str(pr["number"]), "--repo", repo, "--name-only")
+ if any(line.startswith(VERSIONS_PATH) for line in diff.stdout.splitlines()):
+ candidates.append(pr)
+ return candidates
+
+
+def fetch_pr_revisions(pr_number: int) -> list[RevisionFile]:
+ """The revision files as they stand on the PR's own head, read as text.
+
+ Fetched straight from the PR's head ref rather than a merge -- the
+ versions directory only ever gains files, so a plain union with the
+ base's current revisions is the merge result for the purpose these checks
+ care about, and it skips every non-migration conflict a real merge could
+ raise. `git show` only ever prints a blob's content; nothing here writes
+ the file to disk or runs it.
+ """
+ run("git", "fetch", "--depth=1", "origin", f"refs/pull/{pr_number}/head")
+ listing = run(
+ "git", "ls-tree", "-r", "--name-only", "FETCH_HEAD", "--", VERSIONS_PATH
+ )
+ revisions = []
+ for path in listing.stdout.splitlines():
+ if not path:
+ continue
+ content = run("git", "show", f"FETCH_HEAD:{path}")
+ revisions.append(parse_revision_file(Path(path).name, content.stdout))
+ return revisions
+
+
+def describe_chain_problem(revisions: list[RevisionFile]) -> str:
+ """The three graph assertions in test_migration_chain.py, over a plain
+ list of (revision, parents) pairs instead of an Alembic graph.
+
+ Returns an empty string if the chain is well-formed, otherwise a
+ description of what is wrong.
+ """
+ by_revision: dict[str, list[str]] = {}
+ for rev in revisions:
+ by_revision.setdefault(rev.revision, []).append(rev.filename)
+ duplicates = {rev: files for rev, files in by_revision.items() if len(files) > 1}
+ if duplicates:
+ return f"duplicate migration revision ids: {duplicates}"
+
+ known = set(by_revision)
+ dangling = {
+ rev.revision: [parent for parent in rev.parents if parent not in known]
+ for rev in revisions
+ if any(parent not in known for parent in rev.parents)
+ }
+ if dangling:
+ return f"migrations pointing at unknown parents: {dangling}"
+
+ all_parents = {parent for rev in revisions for parent in rev.parents}
+ heads = [rev.revision for rev in revisions if rev.revision not in all_parents]
+ if len(heads) != 1:
+ return f"expected exactly one migration head, got {heads}"
+
+ return ""
+
+
+def post_status(
+ repo: str, sha: str, state: str, description: str, run_url: str
+) -> None:
+ run(
+ "gh",
+ "api",
+ f"repos/{repo}/statuses/{sha}",
+ "-f",
+ f"state={state}",
+ "-f",
+ f"context={STATUS_CONTEXT}",
+ "-f",
+ f"description={description}",
+ "-f",
+ f"target_url={run_url}",
+ )
+
+
+def check_pr(
+ pr: dict[str, Any], base_revisions: list[RevisionFile], repo: str, run_url: str
+) -> bool:
+ number = pr["number"]
+ sha = pr["headRefOid"]
+ combined = base_revisions + fetch_pr_revisions(int(number))
+ problem = describe_chain_problem(combined)
+ passed = not problem
+ if passed:
+ description = "Migration chain stays single-headed if this PR merges now"
+ else:
+ description = "Merging this PR now would break the migration chain -- rebase onto the latest main"
+ print(
+ f"::error::PR #{number} would break the migration chain if merged now: {problem}"
+ )
+ post_status(
+ repo, str(sha), "success" if passed else "failure", description, run_url
+ )
+ return passed
+
+
+def main() -> int:
+ repo = os.environ["GITHUB_REPOSITORY"]
+ run_url = os.environ["RUN_URL"]
+ prs = open_prs_touching_migrations(repo)
+ if not prs:
+ print("No open PRs touch the migrations directory.")
+ return 0
+ base_revisions = read_base_revisions()
+ results = [check_pr(pr, base_revisions, repo, run_url) for pr in prs]
+ return 0 if all(results) else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())