:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(docs): read depot checks from any depot app (#121202)
Raúl Negrón-Otero495470316c5c
diff --git a/.agents/skills/debugging-ci-failures/references/master-red-incident.md b/.agents/skills/debugging-ci-failures/references/master-red-incident.mdindex 462fb36c4d4e..6289963362b7 100644--- a/.agents/skills/debugging-ci-failures/references/master-red-incident.md+++ b/.agents/skills/debugging-ci-failures/references/master-red-incident.md@@ -61,11 +61,12 @@ The alerter reads Depot when it opens the incident, so the alert identifies theDepot posts a check for each job on the master commit that the run tested.Those checks give each job's conclusion and times, with no log text.-Several hourly runs can test one commit, so keep the checks whose `details_url` holds the workflow ID from the alert's URL:+Several hourly runs can test one commit, so the command keeps the checks whose `details_url` holds the workflow ID from the alert's URL.+Depot posts these checks from more than one GitHub app, so do not filter them by app:```bash-gh api "repos/PostHog/posthog/commits/<sha>/check-runs?app_id=219785&filter=all&per_page=100" --paginate \- --jq '.check_runs[] | select(.conclusion != "success" and .conclusion != "skipped") | {name, conclusion, started_at, completed_at, details_url}'+gh api "repos/PostHog/posthog/commits/<sha>/check-runs?filter=all&per_page=100" --paginate \+ --jq '.check_runs[] | select(((.details_url // "") | contains("/workflows/<workflow-id>")) and .conclusion != "success" and .conclusion != "skipped") | {name, conclusion, started_at, completed_at, details_url}'```The logs are in the warehouse once it has synced Depot, which can trail the run by hours.@@ -81,10 +82,10 @@ With rows, pass that `run_id` and `"ci_engine": "depot_ci"` to the failure-log cIt returns failed jobs only, so a job that Depot cancelled at its timeout is not in the answer.The failed gate job prints a `Failing backend & product tests` table.-This query lists the gate checks of the hourly runs on the newest 100 master commits, newest first:+This query lists the gate checks of the hourly runs on the newest 100 master commits, newest first.```bash-gh api graphql -f query='{repository(owner:"PostHog",name:"posthog"){ref(qualifiedName:"master"){target{... on Commit{history(first:100){nodes{oid checkSuites(first:5,filterBy:{appId:219785}){nodes{checkRuns(first:30,filterBy:{checkName:"Backend CI on Depot / Django Tests Pass on Depot"}){nodes{status conclusion startedAt detailsUrl}}}}}}}}}}}' \+gh api graphql -f query='{repository(owner:"PostHog",name:"posthog"){ref(qualifiedName:"master"){target{... on Commit{history(first:100){nodes{oid checkSuites(first:5,filterBy:{checkName:"Backend CI on Depot / Django Tests Pass on Depot"}){nodes{checkRuns(first:30,filterBy:{checkName:"Backend CI on Depot / Django Tests Pass on Depot"}){nodes{status conclusion startedAt detailsUrl}}}}}}}}}}}' \--jq '[.data.repository.ref.target.history.nodes[] | {headSha: .oid} + (.checkSuites.nodes[].checkRuns.nodes[])] | sort_by(.startedAt) | reverse'```diff --git a/.agents/skills/depot-ci/SKILL.md b/.agents/skills/depot-ci/SKILL.mdindex 7dd4ba03e589..ac7b6f8100a9 100644--- a/.agents/skills/depot-ci/SKILL.md+++ b/.agents/skills/depot-ci/SKILL.md@@ -25,6 +25,8 @@ This SKILL.md covers the core workflow plus the most common commands. Detailed fPostHog addition: read [references/posthog-check-run-semantics.md](references/posthog-check-run-semantics.md) before you reason about what a Depot CI run reports to GitHub.It covers what Depot CI and GitHub Actions each post for a skipped, empty-matrix or `continue-on-error` job, how GitHub and the Trunk merge queue score those conclusions, and how the two engines differ on check naming and reruns.This repo runs Depot CI workflows out of `.depot/workflows/` alongside GitHub Actions, so both engines report on the same commit.+Depot posts each check from one of three GitHub apps.+The "Which app posts a Depot check" section of `references/posthog-check-run-semantics.md` lists the apps and how to match their checks.<!-- End PostHog-local section. -->diff --git a/.agents/skills/depot-ci/references/posthog-check-run-semantics.md b/.agents/skills/depot-ci/references/posthog-check-run-semantics.mdindex 47f913485020..3e63c50ac457 100644--- a/.agents/skills/depot-ci/references/posthog-check-run-semantics.md+++ b/.agents/skills/depot-ci/references/posthog-check-run-semantics.md@@ -5,26 +5,26 @@ PostHog-authored. Not part of the vendored upstream skill.Two engines report on the same commit in this repo, and they are easy to confuse:- **Depot runners** run GitHub Actions jobs from `.github/workflows/`. GitHub is still the engine, and checks come from the `github-actions` app. See the `depot-github-runners` skill.-- **Depot CI** is its own engine. It parses workflows under `.depot/workflows/`, which GitHub Actions ignores, and reports its own check runs from the `depot-code-access` app. That is what this skill covers.+- **Depot CI** is its own engine. It parses workflows under `.depot/workflows/`, which GitHub Actions ignores, and reports its own check runs from one of three Depot apps (see [Which app posts a Depot check](#which-app-posts-a-depot-check)). That is what this skill covers.Neither vendor documents any of what follows. It comes from running one job graph on both engines against the same commit and reading the check runs each posted. To re-derive it, put twin workflows in `.github/workflows/` and `.depot/workflows/`, give each job a shape from the table below, and have an `if: always()` job print `toJSON(needs)`.## Conclusions the two engines post-| Job shape | GHA `needs.result` | GHA check run | Depot `needs.result` | Depot check run |-| ------------------------------------------ | ------------------ | --------------------------- | -------------------- | ------------------------- |-| Plain success | `success` | `success` | `success` | `success` |-| `if: false` | `skipped` | `skipped` | `skipped` | `skipped` |-| `if:` expression that is false | `skipped` | `skipped` | `skipped` | `skipped` |-| Skipped because a dependency skipped | `skipped` | `skipped` | `skipped` | `skipped` |-| **Matrix that expands to zero cells** | **`failure`** | **none posted** | **`skipped`** | **`skipped`** |-| Two cell matrix | `success` | two checks, `(1)` and `(2)` | `success` | one check, no cell suffix |-| Job-level `continue-on-error`, step fails | `success` | **`failure`** | `success` | **`failure`** |-| Step-level `continue-on-error`, step fails | `success` | `success` | `success` | `success` |-| Hard failure | `failure` | `failure` | `failure` | `failure` |-| `if: always()` gate over all of the above | `success` | `success` | `success` | `success` |--The engines agree everywhere except the empty matrix and the matrix check granularity.+| Job shape | GHA `needs.result` | GHA check run | Depot `needs.result` | Depot check run |+| ------------------------------------------ | ------------------ | --------------------------- | -------------------- | ------------------ |+| Plain success | `success` | `success` | `success` | `success` |+| `if: false` | `skipped` | `skipped` | `skipped` | `skipped` |+| `if:` expression that is false | `skipped` | `skipped` | `skipped` | `skipped` |+| Skipped because a dependency skipped | `skipped` | `skipped` | `skipped` | `skipped` |+| **Matrix that expands to zero cells** | **`failure`** | **none posted** | **`skipped`** | **`skipped`** |+| Two cell matrix | `success` | two checks, `(1)` and `(2)` | `success` | one check per cell |+| Job-level `continue-on-error`, step fails | `success` | **`failure`** | `success` | **`failure`** |+| Step-level `continue-on-error`, step fails | `success` | `success` | `success` | `success` |+| Hard failure | `failure` | `failure` | `failure` | `failure` |+| `if: always()` gate over all of the above | `success` | `success` | `success` | `success` |++The engines agree everywhere except the empty matrix.## Two cases where the check disagrees with the run@@ -37,7 +37,7 @@ The engines agree everywhere except the empty matrix and the matrix check granul- A check run that concludes `skipped` **satisfies** a required status check, and the Trunk merge queue scores it the same way. The repo already depends on this: `Build Docker image` is a required context that concludes `skipped` on most PRs, and they merge.- A required context that **no check run ever reports** stays pending and blocks. This is the empty-matrix failure mode above, and it is also what a `paths:` filter does when it silences a whole workflow.- A check run that stays `in_progress` blocks, and GitHub keeps every completed check run on its commit. Depot creates a job's own check only when the job resolves, so a new run on an already green commit (a draft marked ready, a reopen) cannot hold the commit pending: the old success stands until the new gate job ends. That is why the required backend context is the gate job's `Backend CI on Depot / Test results` check, posted by the PostHog tests app: a passing draft run does not post it, so the context stays missing until the ready run posts the verdict. A draft shows the check only when its run failed.-- Required contexts are pinned to one app. Most entries in this repo's `master` ruleset carry `integration_id: 15368`, the `github-actions` app; the backend entry, `Backend CI on Depot / Test results`, is pinned to the PostHog tests app. Read them with:+- A ruleset entry with an `integration_id` accepts a check from that one app only. An entry without one accepts a check from any app, which GitHub's UI calls "Any source". Most entries in this repo's `master` ruleset carry `integration_id: 15368`, the `github-actions` app. `Backend CI on Depot / Test results` carries `2492437`, the `tests-posthog` app. `Backend CI on Depot / Django Tests Pass on Depot` carries none. Read them with:```bashgh api repos/PostHog/posthog/rulesets --jq '.[] | select(.name=="master") | .id'@@ -45,15 +45,31 @@ The engines agree everywhere except the empty matrix and the matrix check granul--jq '.rules[] | select(.type=="required_status_checks") | .parameters.required_status_checks'```- A check run posted by any other app does not satisfy a context pinned to `github-actions`, however exactly the name matches. Depot CI's app id is `219785`.+ A check run from any other app does not satisfy a context pinned to `github-actions`, even when the name matches exactly.- Trunk merge queue takes its required list from GitHub branch protection by default, and can override it from the Trunk UI or from `merge.required_statuses` in `.trunk/trunk.yaml`. The trunk.yaml form matches on **name only**, with no app pinning, so a repo that sets both can have Trunk and GitHub disagree about which app's check counts. This repo sets no `merge:` block, so branch protection is the source.+## Which app posts a Depot check++Depot's [GitHub checks page](https://depot.dev/docs/ci/observability/github-checks#install-the-apps) says a check can come from any of these apps, with the same name:++| App | Slug | App id |+| ----------------- | ------------------- | --------- |+| Depot Checks | `depot-checks` | `4964980` |+| Depot Checks 2 | `depot-checks-2` | `4968192` |+| Depot Code Access | `depot-code-access` | `219785` |++Depot uses the Depot Checks apps first and falls back to Depot Code Access. A check stays with the app that created it.++A required context for a check that Depot posts must use "Any source", as [Depot's GitHub checks page](https://depot.dev/docs/ci/observability/github-checks#update-required-check-sources) says. Match a Depot check by its name, or by its `details_url`, which holds the Depot workflow ID. Never match it by one app ID. GitHub's GraphQL `checkSuites` takes `filterBy: {checkName}`, which matches a suite from any app.++A check that a Depot job posts itself through the REST API comes from the app whose token the step used, so its required context can stay pinned to that app. `Backend CI on Depot / Test results` is one.+## NamingDepot names each check `<workflow name> / <job name>`. GitHub Actions names it `<job name>` alone. The workflow name is therefore part of every Depot context, and renaming a workflow renames every check it posts.-Matrix jobs differ: GitHub Actions posts one check per cell and appends the cell to the name, Depot posts one check for the whole job. Depot also posts internal expansion checks named `<file>:<job key>:_dynamicMatrix` for matrices built from `fromJSON`.+Both engines post one check per matrix cell. Depot also posts an internal expansion check named `<file>:<job key>` for a matrix built from `fromJSON`, for example `Backend CI on Depot / ci-backend.yml:django`, which concludes `skipped`.## When a check run appears@@ -130,7 +146,7 @@ The CLI needs ids that the GitHub API does not carry. Start from a check run's ````bash# Job and workflow ids for the depot checks on a commitgh api "repos/PostHog/posthog/commits/$SHA/check-runs?per_page=100" --paginate \- --jq '.check_runs[] | select(.app.slug=="depot-code-access") | "\(.name)\t\(.details_url)"'+ --jq '.check_runs[] | select((.details_url // "") | startswith("https://depot.dev/")) | "\(.name)\t\(.details_url)"'# Everything else, including the run id, from a job iddepot ci diagnose --job <job-id> -o json@@ -150,7 +166,7 @@ Depot CI checks out the pull request merge ref (`refs/pull/<n>/merge`) and postsFrom Depot's [compatibility page](https://depot.dev/docs/ci/compatibility), not measured here:- Depot does not run fork pull requests. After review, a maintainer re-creates the pull request in the main repository. See [Pull requests from forks](../../../../docs/published/handbook/engineering/fork-pull-requests.md).-- `environment:` is unsupported, and `uses:` cannot reference a workflow in another repository.+- `environment:` is unsupported. For a `uses:` call to a workflow in another repository, see `github-actions-compatibility.md`.- `secrets.GITHUB_TOKEN` is a GitHub App installation token, not the Actions token. GitHub Packages rejects it. The rate limit it draws on is the app installation's pool, which is shared across repos and is not the per-repo Actions bucket that `monitor-github-rate-limit.yml` watches.- Non-Depot `runs-on` labels are treated as `depot-ubuntu-latest`.- Secrets and variables live in Depot's own store (`depot ci secrets`, `depot ci vars`), not in GitHub's.diff --git a/.agents/skills/depot-github-runners/SKILL.md b/.agents/skills/depot-github-runners/SKILL.mdindex 6acd97831489..486e1c8f6c1a 100644--- a/.agents/skills/depot-github-runners/SKILL.md+++ b/.agents/skills/depot-github-runners/SKILL.md@@ -22,7 +22,7 @@ Depot provides managed, ephemeral, single-tenant GitHub Actions runners. Drop-in## Depot runners are not Depot CIDepot runners keep GitHub Actions as the engine, so the workflow lives in `.github/workflows/` and the checks still come from the `github-actions` app.-Depot CI is a different product: it reads `.depot/workflows/`, which GitHub Actions ignores, and posts its own check runs from the `depot-code-access` app.+Depot CI is a different product. It reads `.depot/workflows/`, which GitHub Actions ignores, and posts its own check runs from one of [three Depot apps](../depot-ci/references/posthog-check-run-semantics.md#which-app-posts-a-depot-check).Use the `depot-ci` skill for that one.Its [references/posthog-check-run-semantics.md](../depot-ci/references/posthog-check-run-semantics.md) compares what each engine reports for a skipped, empty-matrix or `continue-on-error` job, and how GitHub and the Trunk merge queue score those conclusions.