Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(ci): fail PRs above the fifth open stack layer (#119650)

Raúl Negrón-Oterob3c136a7f26c
diff --git a/.agents/skills/stacking-prs/SKILL.md b/.agents/skills/stacking-prs/SKILL.md
index 9dcfac928f60..13aa8926af83 100644
--- a/.agents/skills/stacking-prs/SKILL.md
+++ b/.agents/skills/stacking-prs/SKILL.md
@@ -55,6 +55,8 @@ gh stack add -Am "add UI" my-feature-ui # stage all + commit in one step
- Link PRs that already exist on GitHub, without local tracking: `gh stack link <pr> <pr> <pr>`, bottom to top (branch names and PR URLs work too). Pass a stack number first to append to an existing stack: `gh stack link <stack> <pr>`.
- Slice by reviewable unit: migration / backend / frontend, or mechanical-rename / behavior-change. Each PR must make sense to review and merge alone.
- Keep stacks shallow (2–4 layers). Every layer multiplies CI cost and rebase churn, and deep-stack pushes can trip GitHub's dispatch cap (see AGENTS.md, "Stacked PRs").
+ The pre-push hook and the `Stack depth` check block a layer above the fifth open one (merged layers don't count).
+ For more work on top, run `gh stack init --base <top branch> <new branch>` from `master`, because gh-stack refuses to start a stack from a layer of another one. Rebase the new stack onto `master` once the lower stack lands.
- Place a CI-heavy layer by how CI diffs it:
- Most jobs pick their work from the layer's own diff against its parent, so such a layer does not make the layers above it heavy. A push to any layer below it starts its jobs again (see the duplicate-run note under "Iterate and keep in sync"). Put it low, unless it is the layer that still changes most.
- A few steps diff against master, so they repeat on layers above the change. Trunk's impacted-targets job runs on every layer and computes affected crates when `rust/` or `proto/` differs. Backend jobs build the HogQL parser from source when `common/hogql_parser/` or `rust/hogql/parser/` differs, and Playwright jobs when `common/hogql_parser/` differs. Those builds repeat only on upper layers whose own diff starts backend or Playwright jobs. If upper layers do, put the parser change in the top layer.
@@ -64,6 +66,12 @@ gh stack add -Am "add UI" my-feature-ui # stage all + commit in one step
## Publish
+When a push or the `Stack depth` check reports too many open layers, merge lower layers or split the stack.
+To split a published stack, drop the upper layers with `gh stack modify`, then run `gh stack submit` from a retained lower branch first.
+This publishes the smaller lower stack and releases the dropped pull requests from the original GitHub stack.
+Check out a dropped branch, adopt the dropped branches bottom to top with `gh stack init --base <top retained branch> <dropped branches>`, then run `gh stack submit`.
+Once the lower stack merges, rebase the upper stack onto `master`.
+
```bash
gh stack submit --auto
```
diff --git a/.github/actionlint.yaml b/.github/actionlint.yaml
index c1db5ba9c676..326bedaa255f 100644
--- a/.github/actionlint.yaml
+++ b/.github/actionlint.yaml
@@ -24,3 +24,8 @@ paths:
'.github/workflows/**/*.yaml':
ignore:
- 'property ".+" is not defined in object type \{changes: string\}'
+ # GitHub's stacked PRs added the `stacked` activity type after actionlint's event list.
+ # The broad glob also matches the copies hogli lint:workflows checks outside .github.
+ '**/*.yml':
+ ignore:
+ - 'invalid activity type "stacked" for "pull_request" Webhook event'
diff --git a/.github/scripts/check-stack-depth.mjs b/.github/scripts/check-stack-depth.mjs
new file mode 100644
index 000000000000..ffd994871cd7
--- /dev/null
+++ b/.github/scripts/check-stack-depth.mjs
@@ -0,0 +1,82 @@
+#!/usr/bin/env node
+import { execFileSync } from 'node:child_process'
+import { pathToFileURL } from 'node:url'
+
+const MAX_OPEN_DEPTH = 5
+export const EXEMPT_THROUGH_PR = 121285
+const SKIPPED = { status: 'skipped', detail: 'The current branch is not in a gh stack.' }
+
+export function checkStackDepth(pullRequests) {
+ const open = pullRequests.filter((pr) => pr.state === 'open')
+ const failed = open.slice(MAX_OPEN_DEPTH).some((pr) => !pr.number || pr.number > EXEMPT_THROUGH_PR)
+ return failed
+ ? {
+ status: 'fail',
+ detail: `${open.length} open layers; the limit is ${MAX_OPEN_DEPTH}. Merge lower layers or split the stack using /stacking-prs before publishing.`,
+ }
+ : {
+ status: 'pass',
+ detail: `${open.length} open layers; within the limit or exempt as an existing pull request.`,
+ }
+}
+
+function gh(args) {
+ return JSON.parse(
+ execFileSync('gh', args, { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'pipe'], timeout: 20_000 })
+ )
+}
+
+function checkLocalStack() {
+ let branches
+ try {
+ ;({ branches } = gh(['stack', 'view', '--json']))
+ } catch {
+ return SKIPPED
+ }
+ // On the trunk, gh stack view reports some stack with no current branch.
+ if (!branches?.some((branch) => branch.isCurrent === true)) {
+ return SKIPPED
+ }
+ // gh stack submit and push publish every layer, whichever branch is checked out.
+ return checkStackDepth(
+ branches.map((branch) => ({
+ number: branch.pr?.number,
+ state: branch.isMerged || branch.pr?.state === 'CLOSED' ? 'closed' : 'open',
+ }))
+ )
+}
+
+function checkPullRequest() {
+ const prNumber = Number(process.env.PR_NUMBER)
+ if (prNumber <= EXEMPT_THROUGH_PR) {
+ return
+ }
+ try {
+ const { pull_requests } = gh([
+ 'api',
+ `repos/${process.env.GITHUB_REPOSITORY}/stacks/${process.env.STACK_NUMBER}`,
+ ])
+ const index = pull_requests.findIndex((layer) => layer.number === prNumber)
+ if (index === -1) {
+ throw new Error(`Pull request ${prNumber} is absent from the stack response`)
+ }
+ const result = checkStackDepth(pull_requests.slice(0, index + 1))
+ if (result.status === 'fail') {
+ console.log(`::error title=Stack too deep::${result.detail}`)
+ process.exitCode = 1
+ }
+ } catch (err) {
+ console.log(`::error title=Stack depth not checked::${err.message}. Retry this check.`)
+ process.exitCode = 1
+ }
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+ if (process.argv.includes('--local')) {
+ const result = checkLocalStack()
+ console.log(JSON.stringify(result))
+ process.exitCode = result.status === 'fail' ? 1 : 0
+ } else {
+ checkPullRequest()
+ }
+}
diff --git a/.github/scripts/check-stack-depth.test.mjs b/.github/scripts/check-stack-depth.test.mjs
new file mode 100644
index 000000000000..aa3a7721468c
--- /dev/null
+++ b/.github/scripts/check-stack-depth.test.mjs
@@ -0,0 +1,103 @@
+import assert from 'node:assert/strict'
+import { spawnSync } from 'node:child_process'
+import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { describe, it } from 'node:test'
+import { fileURLToPath } from 'node:url'
+
+import { EXEMPT_THROUGH_PR, checkStackDepth } from './check-stack-depth.mjs'
+
+const layers = (...states) => states.map((state, i) => ({ number: EXEMPT_THROUGH_PR + 1 + i, state }))
+const existingLayers = (count) =>
+ Array.from({ length: count }, (_, i) => ({ number: EXEMPT_THROUGH_PR - count + 1 + i, state: 'open' }))
+
+function runWithFakeGh(t, ghOutput, args = [], env = {}) {
+ const directory = mkdtempSync(join(tmpdir(), 'stack-depth-'))
+ t.after(() => rmSync(directory, { recursive: true, force: true }))
+ const script = ghOutput === null ? 'exit 1' : `printf '%s' '${JSON.stringify(ghOutput)}'`
+ writeFileSync(join(directory, 'gh'), `#!/bin/sh\n${script}\n`, { mode: 0o755 })
+ return spawnSync(process.execPath, [fileURLToPath(new URL('./check-stack-depth.mjs', import.meta.url)), ...args], {
+ env: { PATH: directory, ...env },
+ encoding: 'utf-8',
+ timeout: 5000,
+ })
+}
+
+describe('checkStackDepth', () => {
+ for (const { name, stack, expectedStatus } of [
+ {
+ name: 'rejects the sixth open layer',
+ stack: layers('open', 'open', 'open', 'open', 'open', 'open'),
+ expectedStatus: 'fail',
+ },
+ {
+ name: 'skips merged layers below',
+ stack: layers('closed', 'closed', 'closed', 'closed', 'open', 'open', 'open'),
+ expectedStatus: 'pass',
+ },
+ {
+ name: 'accepts five open layers',
+ stack: layers('open', 'open', 'open', 'open', 'open'),
+ expectedStatus: 'pass',
+ },
+ {
+ name: 'keeps existing pull requests exempt',
+ stack: existingLayers(6),
+ expectedStatus: 'pass',
+ },
+ {
+ name: 'rejects adding an unpublished layer to an exempt stack',
+ stack: [...existingLayers(6), { state: 'open' }],
+ expectedStatus: 'fail',
+ },
+ ]) {
+ it(name, () => {
+ assert.equal(checkStackDepth(stack).status, expectedStatus)
+ })
+ }
+})
+
+describe('local stack adapter', () => {
+ for (const { name, current, closedBottom = false, expectedStatus } of [
+ { name: 'fails pushing the sixth unpublished layer', current: 5, expectedStatus: 'fail' },
+ { name: 'fails publishing from a lower layer of a deep stack', current: 1, expectedStatus: 'fail' },
+ { name: 'skips a closed pull request below', current: 5, closedBottom: true, expectedStatus: 'pass' },
+ {
+ name: 'skips the trunk, which gh stack reports with no current branch',
+ current: -1,
+ expectedStatus: 'skipped',
+ },
+ ]) {
+ it(name, (t) => {
+ const branches = Array.from({ length: 6 }, (_, i) => ({ isMerged: false, isCurrent: i === current }))
+ if (closedBottom) {
+ branches[0].pr = { number: EXEMPT_THROUGH_PR + 1, state: 'CLOSED' }
+ }
+ const result = runWithFakeGh(t, { branches }, ['--local'])
+ assert.equal(JSON.parse(result.stdout).status, expectedStatus, result.stderr)
+ assert.equal(result.status, expectedStatus === 'fail' ? 1 : 0)
+ })
+ }
+})
+
+describe('stack depth check', () => {
+ for (const { name, prNumber = EXEMPT_THROUGH_PR + 1, unavailable = false, expected } of [
+ { name: 'fails a sixth open layer', expected: 1 },
+ { name: 'keeps existing pull requests exempt', prNumber: EXEMPT_THROUGH_PR, unavailable: true, expected: 0 },
+ { name: 'fails when GitHub cannot verify the stack', unavailable: true, expected: 1 },
+ ]) {
+ it(name, (t) => {
+ const stack = {
+ pull_requests: Array.from({ length: 6 }, (_, i) => ({ number: prNumber - 5 + i, state: 'open' })),
+ }
+ const result = runWithFakeGh(t, unavailable ? null : stack, [], {
+ GITHUB_REPOSITORY: 'example/repository',
+ PR_NUMBER: String(prNumber),
+ STACK_NUMBER: '123',
+ })
+ assert.equal(result.error, undefined)
+ assert.equal(result.status, expected, result.stdout + result.stderr)
+ })
+ }
+})
diff --git a/.github/workflows/pr-housekeeping.yml b/.github/workflows/pr-housekeeping.yml
index ce7e6fe485b9..b718027a8c53 100644
--- a/.github/workflows/pr-housekeeping.yml
+++ b/.github/workflows/pr-housekeeping.yml
@@ -7,7 +7,7 @@ name: PR housekeeping
on:
pull_request:
- types: [opened, synchronize, reopened, edited, ready_for_review, review_requested, converted_to_draft]
+ types: [opened, synchronize, reopened, edited, ready_for_review, review_requested, converted_to_draft, stacked]
# Keyed by event action so a new push (synchronize) cancels the previous
# push's run without cancelling runs fired by other event types on the PR.
@@ -128,6 +128,13 @@ jobs:
secrets:
GH_APP_POSTHOG_PATHS_FILTER_PRIVATE_KEY: ${{ secrets.GH_APP_POSTHOG_PATHS_FILTER_PRIVATE_KEY }}
+ stack-depth:
+ if: contains(fromJSON('["opened", "synchronize", "reopened", "stacked"]'), github.event.action)
+ uses: ./.github/workflows/pr-stack-depth.yml
+ permissions:
+ contents: read
+ pull-requests: read
+
release-plan:
if: contains(fromJSON('["opened", "synchronize", "reopened"]'), github.event.action)
uses: ./.github/workflows/release.yml
diff --git a/.github/workflows/pr-stack-depth.yml b/.github/workflows/pr-stack-depth.yml
new file mode 100644
index 000000000000..2c2ecee74dc9
--- /dev/null
+++ b/.github/workflows/pr-stack-depth.yml
@@ -0,0 +1,28 @@
+name: PR stack depth
+
+on:
+ workflow_call:
+
+permissions:
+ contents: read
+ pull-requests: read
+
+jobs:
+ stack-depth:
+ name: Stack depth
+ if: ${{ github.event.pull_request.stack != null }}
+ runs-on: ubuntu-24.04
+ timeout-minutes: 5
+
+ steps:
+ - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+ with:
+ sparse-checkout: .github/scripts/check-stack-depth.mjs
+ sparse-checkout-cone-mode: false
+
+ - name: Check open layers below this pull request
+ env:
+ GH_TOKEN: ${{ github.token }}
+ PR_NUMBER: ${{ github.event.pull_request.number }}
+ STACK_NUMBER: ${{ github.event.pull_request.stack.number }}
+ run: node .github/scripts/check-stack-depth.mjs
diff --git a/tools/hogli-commands/hogli_commands/ci_preflight.py b/tools/hogli-commands/hogli_commands/ci_preflight.py
index 148c811211bf..6ec3ef0ddcd2 100644
--- a/tools/hogli-commands/hogli_commands/ci_preflight.py
+++ b/tools/hogli-commands/hogli_commands/ci_preflight.py
@@ -61,6 +61,7 @@ from hogli_commands.preflight_checks import (
check_merge_queue_lane,
check_semgrep_devex,
check_snapshot_baselines,
+ check_stack_depth,
prepare_semgrep,
)
from hogli_commands.projections import all_outputs as projection_outputs
@@ -880,6 +881,13 @@ def ci_preflight(do_fix: bool, strict: bool, against: str | None, as_json: bool,
click.secho(f" {_ICON[stale_status]} [staleness] branch freshness vs master", fg=_COLOR[stale_status])
click.echo(f" {stale_detail}")
+ depth_status, depth_detail = check_stack_depth()
+ failures += depth_status == "fail"
+ results.append({"check": "stack-depth", "status": depth_status, "files": 0, "detail": depth_detail})
+ if not as_json and depth_status != "skipped":
+ click.secho(f" {_ICON[depth_status]} [stack-depth] open layers in this gh stack", fg=_COLOR[depth_status])
+ click.echo(f" {depth_detail}")
+
if shadow_drift_triggered:
violations = mirror_violations(REPO_ROOT, set(files))
drift_status: Status = "fail" if violations else "pass"
diff --git a/tools/hogli-commands/hogli_commands/preflight_checks.py b/tools/hogli-commands/hogli_commands/preflight_checks.py
index 6993edc81557..a12ac5eabcc0 100644
--- a/tools/hogli-commands/hogli_commands/preflight_checks.py
+++ b/tools/hogli-commands/hogli_commands/preflight_checks.py
@@ -462,6 +462,24 @@ def check_merge_queue_lane(scope: Scope) -> Outcome:
CROSS_LANE_LABEL = "cross-lane-change"
+def check_stack_depth() -> Outcome:
+ try:
+ result = subprocess.run(
+ ["node", ".github/scripts/check-stack-depth.mjs", "--local"],
+ stdin=subprocess.DEVNULL,
+ capture_output=True,
+ text=True,
+ timeout=25,
+ cwd=REPO_ROOT,
+ )
+ match json.loads(result.stdout):
+ case {"status": "pass" | "fail" | "skipped" as status, "detail": str() as detail}:
+ return status, detail
+ except (OSError, subprocess.SubprocessError, ValueError):
+ pass
+ return "skipped", "could not run the stack-depth check"
+
+
def _first(files: list[str], total: int) -> str:
count = max(total, len(files))
more = f" (+{count - 2} more)" if count > 2 else ""
diff --git a/tools/hogli-commands/hogli_commands/tests/test_ci_preflight.py b/tools/hogli-commands/hogli_commands/tests/test_ci_preflight.py
index a6e7e58e1c8c..7c19b43d3b9e 100644
--- a/tools/hogli-commands/hogli_commands/tests/test_ci_preflight.py
+++ b/tools/hogli-commands/hogli_commands/tests/test_ci_preflight.py
@@ -27,7 +27,10 @@ runner = CliRunner()
@pytest.fixture(autouse=True)
def no_diff_reading_checks() -> Iterator[None]:
# The real ones shell out to node and semgrep on any changed path.
- with patch("hogli_commands.ci_preflight.DIFF_CHECKS", [chk for chk in DIFF_CHECKS if chk.run is None]):
+ with (
+ patch("hogli_commands.ci_preflight.DIFF_CHECKS", [chk for chk in DIFF_CHECKS if chk.run is None]),
+ patch("hogli_commands.ci_preflight.check_stack_depth", return_value=("skipped", "no local stack")),
+ ):
yield
@@ -85,6 +88,18 @@ class TestKillSwitch:
class TestStrictAndFixContracts:
+ @patch("hogli_commands.ci_preflight._emit_telemetry")
+ @patch("hogli_commands.ci_preflight._staleness", return_value=("pass", "even with master", {}))
+ @patch("hogli_commands.ci_preflight._fetch_master")
+ @patch("hogli_commands.ci_preflight.changed_files", return_value=[])
+ def test_explicit_preflight_blocks_an_oversized_stack_without_a_diff(
+ self, mock_changed: MagicMock, mock_fetch: MagicMock, mock_stale: MagicMock, mock_emit: MagicMock
+ ) -> None:
+ with patch("hogli_commands.ci_preflight.check_stack_depth", return_value=("fail", "6 open layers")):
+ result = runner.invoke(cli, ["ci:preflight", "--strict"])
+ assert result.exit_code == 1
+ assert "6 open layers" in result.output
+
@patch("hogli_commands.ci_preflight._emit_telemetry")
@patch("hogli_commands.ci_preflight._staleness", return_value=("pass", "even with master", {}))
@patch("hogli_commands.ci_preflight._fetch_master")
diff --git a/tools/hogli-commands/hogli_commands/tests/test_preflight_checks.py b/tools/hogli-commands/hogli_commands/tests/test_preflight_checks.py
index c2c9ca9796ff..71e2d0966d2e 100644
--- a/tools/hogli-commands/hogli_commands/tests/test_preflight_checks.py
+++ b/tools/hogli-commands/hogli_commands/tests/test_preflight_checks.py
@@ -19,6 +19,7 @@ from hogli_commands.preflight_checks import (
check_merge_queue_lane,
check_semgrep_devex,
check_snapshot_baselines,
+ check_stack_depth,
)
MANIFEST = """version: 1
@@ -41,6 +42,18 @@ def _scope(changed: list[str]) -> Scope:
return Scope(files=changed, changed=changed, merge_base="abc123", committed_only=True)
+class TestStackDepth:
+ @pytest.mark.parametrize(
+ "stdout,returncode,expected",
+ [(json.dumps({"status": "fail", "detail": "six open layers"}), 1, "fail"), ("not json", 1, "skipped")],
+ )
+ def test_uses_the_shared_checker_verdict(self, stdout: str, returncode: int, expected: str) -> None:
+ result = subprocess.CompletedProcess([], returncode, stdout=stdout)
+ with patch("hogli_commands.preflight_checks.subprocess.run", return_value=result):
+ status, _ = check_stack_depth()
+ assert status == expected
+
+
def _git_show(blobs: dict[str, bytes]) -> Callable[..., bytes | None]:
return lambda _show, ref, timeout=20.0: blobs.get(ref)
diff --git a/tools/hogli-commands/hogli_commands/workflow_lint/checks/pr_event_fanout.py b/tools/hogli-commands/hogli_commands/workflow_lint/checks/pr_event_fanout.py
index 4be2a9d38d15..ad1cad55dd72 100644
--- a/tools/hogli-commands/hogli_commands/workflow_lint/checks/pr_event_fanout.py
+++ b/tools/hogli-commands/hogli_commands/workflow_lint/checks/pr_event_fanout.py
@@ -37,6 +37,7 @@ PR_EVENT_FANOUT_BUDGET: Mapping[str, int] = {
"ready_for_review": 11,
"reopened": 24,
"review_requested": 1,
+ "stacked": 1,
"synchronize": 28,
}