:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
ci(tasks): remove the tasks golden snapshot workflow (#121348)
Georges-Antoine Assi5fc954fe105c
diff --git a/.github/auto-assign-labels.json b/.github/auto-assign-labels.jsonindex e368131b6f38..588e70255139 100644--- a/.github/auto-assign-labels.json+++ b/.github/auto-assign-labels.json@@ -31,7 +31,6 @@".agents/skills/finding-llm-gateway-migration-candidates/",".agents/skills/migrating-llm-gateway-callers/",".github/workflows/cd-agent-proxy-image.yml",- ".github/workflows/cd-tasks-golden-snapshot.yml",".github/workflows/ci-agent-proxy.yml",".github/workflows/ci-llm-gateway.yml",".github/workflows/llm-gateway-cd.yml",diff --git a/.github/owners.yaml b/.github/owners.yamlindex 74ecaa35be63..cb69fdd643d4 100644--- a/.github/owners.yaml+++ b/.github/owners.yaml@@ -33,6 +33,5 @@ rules:- match:- '/workflows/ci-agent-proxy.yml'- '/workflows/cd-agent-proxy-image.yml'- - '/workflows/cd-tasks-golden-snapshot.yml'owners: team-agent-infrastructuresensitive: truediff --git a/.github/workflows/cd-tasks-golden-snapshot.yml b/.github/workflows/cd-tasks-golden-snapshot.ymldeleted file mode 100644index 7e0bc84e7d83..000000000000--- a/.github/workflows/cd-tasks-golden-snapshot.yml+++ /dev/null@@ -1,748 +0,0 @@-# Bake the PostHog *tasks* golden snapshot and promote it into the alias that-# HoglandSandbox.create restores every task sandbox from.-#-# HoglandSandbox.create (products/tasks/backend/logic/services/hogland_sandbox.py)-# restores every task sandbox from the alias this workflow promotes. Instead of an-# engineer running the bake from a laptop, a runner renders the agent skills,-# joins the hogland tailnet, builds the seed box from box primitives + a delivered-# payload, smoke-tests the candidate, and only then re-points the live alias —-# with a dated archive alias for rollback.-#-# Skills delivery (the crux): the golden must ship the SAME merged, rendered skills-# the Modal image ships, but WITHOUT depending on the sandbox-base image. So a-# `render_skills` job renders them itself with a database — the same-# `hogli build:skills` mechanism the image build uses — and merges in the-# context-mill skills, exactly as cd-sandbox-base-image.yml's build_skills job does.-# The bake job streams that rendered set into the seed box over the box's own SSH-# (see bake-golden.sh). No GHCR image, no public artifact host.-#-# Modal / Dockerfile.sandbox-base is deliberately untouched: this bakes the same-# contents into a hogland snapshot, it does not change the container image path.-#-# Reachability + auth (mirrors hogbox-preview-env.yml, which already reaches-# hogland from this repo):-# * both hogland APIs are tailnet-only, served by tag:hogplane devices. We join-# as tag:hogland-ci — the tag the ACL grants reach to those devices, and the-# same join hogbox-preview-env.yml uses. One join covers dev and prod-us.-# * hogplane authenticates the caller by bearer: GitHub OIDC end-to-end, no-# stored token. It verifies the Actions OIDC token and resolves the-# (repo, workflow) claim to a service-account Principal via a github_oidc-# TrustMapping. Those tokens live ~5 min while a bake runs ~90, so the CLI-# re-mints on any mid-bake 401 via HOG_TOKEN_COMMAND — a single mint-at-start-# bearer would die mid-bake.-#-# Gating: this stays a deliberate no-op until an operator provisions the per--# cluster svc-ci principal + github_oidc TrustMapping + tailnet admittance and-# flips the arming vars (see GOLDEN_CI_RUNBOOK.md). workflow_dispatch is the only-# real trigger; the nightly schedule fires but the job's `if` keeps it a no-op-# until armed, so merging this PR cannot attempt a bake before the principal-# exists.--name: Tasks Golden Snapshot CD--on:- workflow_dispatch:- inputs:- cluster:- description: 'Only bake this cluster (blank = all armed clusters)'- required: false- default: ''- ref:- description: "PostHog ref/SHA to render the skills from (blank = this workflow's checkout ref, i.e. master on schedule)"- required: false- default: ''- box_mem_mib:- description: 'Total/cap memory (MiB) for the golden. Blank = 20480 (fixed-size default).'- required: false- default: ''- box_mem_boot_mib:- description: 'Boot memory (MiB) for a pluggable-memory golden — boots this small and hot-adds up to the cap. Blank = fixed-size golden (today).'- required: false- default: ''- target_alias:- description: 'Live alias to promote the baked snapshot into. Blank = posthog-tasks-default (the golden every task box uses). Set posthog-tasks-hotplug to publish a pluggable-memory golden alongside the default, selected per team by the tasks-hogland-hotplug-golden flag.'- required: false- default: ''- schedule:- # Daily, minute offset off :00 — GitHub's scheduler is contended at the- # top of the hour and may drop those runs. No-op until armed (job `if`).- - cron: '17 8 * * *' # ~08:17 UTC, low-traffic window--permissions:- contents: read- id-token: write # mint OIDC for the tailnet join AND the hogplane bearer--# Never two bakes racing the same alias. Per-cluster group keys would need-# ${{ matrix.cluster }}, unavailable at workflow level, so serialise the whole-# workflow and rely on fail-fast=false to keep one cluster's flake from killing-# the other within a run.-concurrency:- group: cd-tasks-golden-snapshot- cancel-in-progress: false--jobs:- # Resolve and validate the workflow_dispatch `ref` input in an UNPRIVILEGED- # job, before any credentialed job checks that tree out. A blank ref resolves- # to the literal `master` branch. A non-blank value must be a full 40-char- # commit SHA that GitHub's compare API confirms is already an ancestor of- # master — anything else fails here. This stops a collaborator pointing `ref`- # at a branch that carries a modified bake-golden.sh / setup-golden.sh and- # having it run in the credentialed bake job (hogland CLI App key, tailnet- # join, hogland/prod OIDC). render_skills and bake check out- # needs.validate_ref.outputs.ref, never the raw input. The- # `github.ref == 'refs/heads/master'` gate on every privileged job below is- # defense in depth (a branch copy can strip it); the durable fix is hogland- # pinning `ref` in the github_oidc TrustMapping — see GOLDEN_CI_RUNBOOK.md.- validate_ref:- if: github.ref == 'refs/heads/master'- name: Validate the requested ref- runs-on: ubuntu-24.04- timeout-minutes: 5- permissions:- contents: read- outputs:- ref: ${{ steps.resolve.outputs.ref }}- steps:- - name: Resolve and validate ref- id: resolve- env:- REQUESTED_REF: ${{ github.event.inputs.ref }}- GH_TOKEN: ${{ github.token }}- run: |- set -euo pipefail- if [[ -z "$REQUESTED_REF" ]]; then- echo "ref=master" >> "$GITHUB_OUTPUT"- echo "::notice::No ref supplied; baking from the master branch"- exit 0- fi- if [[ ! "$REQUESTED_REF" =~ ^[0-9a-f]{40}$ ]]; then- echo "::error::ref must be a full 40-character commit SHA (got '$REQUESTED_REF')"- exit 1- fi- # compare master...<ref>: ahead_by counts commits in <ref> that- # master does not contain. Zero => <ref> is an ancestor of (or- # identical to) master. A non-existent SHA 404s and fails here.- ahead_by=$(gh api "repos/${GITHUB_REPOSITORY}/compare/master...${REQUESTED_REF}" --jq '.ahead_by')- if [[ "$ahead_by" != "0" ]]; then- echo "::error::ref ${REQUESTED_REF} is not an ancestor of master (ahead_by=${ahead_by}); only merged commits may be baked"- exit 1- fi- echo "ref=${REQUESTED_REF}" >> "$GITHUB_OUTPUT"- echo "::notice::ref ${REQUESTED_REF} confirmed an ancestor of master"-- # Render the agent skills the golden ships, the same way the sandbox-base image- # build does (cd-sandbox-base-image.yml's build_skills job): stand up a DB,- # uv sync, migrate, `hogli build:skills` to render the .md.j2 templates, then- # merge in the context-mill skills. Upload the merged set as an artifact the- # bake job delivers into the box. Rendered once here, reused by every cluster.- #- # Shares the master arm gate so an unarmed nightly does not pay for a full DB- # render; when it skips, the bake job (needs) skips too, keeping the no-op.- render_skills:- needs: validate_ref- if: vars.CD_DEPLOY_ENABLED == 'true' && vars.HOG_TASKS_GOLDEN_ENABLED == 'true' && github.ref == 'refs/heads/master'- name: Render agent skills from source- runs-on: ubuntu-24.04- # `Run migrations` needs a database: 24 skill templates call- # `render_hogql_example`, which reads `Team.objects.first()`. The schema- # cache below makes migrate a top-up, but a cache miss is a 22-29 minute- # cold migrate (see cd-sandbox-base-image.yml), so budget an hour for- # ample headroom over that worst case.- timeout-minutes: 60- permissions:- contents: read-- steps:- - name: Check out posthog- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2- with:- ref: ${{ needs.validate_ref.outputs.ref }}- # Deep + blob-less: enough history for the schema cache merge-base key.- fetch-depth: 1000- filter: blob:none-- - name: Fetch base branch for merge-base computation- if: github.event_name == 'pull_request'- env:- BASE_REF: ${{ github.event.pull_request.base.ref }}- # Scoped, blob-less, no-tags: without an explicit refspec, git fetch- # falls back to remote.origin.fetch and pulls every branch.- run: git fetch --no-tags --depth=1000 --filter=blob:none origin "$BASE_REF:refs/remotes/origin/$BASE_REF"-- - name: Compute schema cache keys- id: schema-key- # Restores the schema dump ci-backend saves on master pushes, so the- # migrate step below only tops up what is newer instead of replaying the- # full migration history. On a PR the migration-set key is the merge-base's;- # on a push or dispatch it is HEAD's. A cache miss falls through to a full- # migrate, so an absent or wrong key is never a correctness risk.- env:- BASE_REF: ${{ github.event.pull_request.base.ref }}- IS_PR: ${{ github.event_name == 'pull_request' }}- # Manual cache-bust knob. Keep in sync with SCHEMA_CACHE_EPOCH in- # ci-backend.yml, which saves these caches on master pushes.- SCHEMA_CACHE_EPOCH: v2- run: |- if [ "$IS_PR" != "true" ]; then- # Pushed or dispatched commit is checked out directly: no merge- # commit, no base ref. The per-SHA key would point at this same- # push's cache (saved concurrently by ci-backend, so racy); rely- # on the stable migration-set key below.- REF=HEAD- else- # HEAD is the synthetic merge commit; HEAD^2 is the PR branch tip.- MERGE_BASE=$(git merge-base HEAD^2 "origin/${BASE_REF}" 2>/dev/null || echo "")- if [ -z "$MERGE_BASE" ]; then- echo "key=" >> $GITHUB_OUTPUT- echo "migrations_key=" >> $GITHUB_OUTPUT- echo "::notice::merge-base not found (branch too stale?), so the schema cache is skipped"- exit 0- fi- # Routing decides which app labels reach the dump, so an edited routing- # config matches no master dump: drop both keys and migrate from scratch.- DB_ROUTING_BASE=$(git ls-tree -r --format='%(objectname) %(path)' "$MERGE_BASE" -- products/db_routing.yaml posthog/product_db_config.py posthog/product_db_router.py)- DB_ROUTING_HEAD=$(git ls-tree -r --format='%(objectname) %(path)' HEAD -- products/db_routing.yaml posthog/product_db_config.py posthog/product_db_router.py)- if [ "$DB_ROUTING_BASE" != "$DB_ROUTING_HEAD" ]; then- echo "key=" >> $GITHUB_OUTPUT- echo "migrations_key=" >> $GITHUB_OUTPUT- echo "::notice::db routing config or router differs from the merge base, so the schema cache is skipped"- exit 0- fi- echo "key=posthog-schema-master-${MERGE_BASE}" >> $GITHUB_OUTPUT- REF="$MERGE_BASE"- fi- # Last-resort restore prefix: matches the newest saved dump when the exact- # keys miss. Gated on checkout age because a re-run of an old commit could- # restore a dump newer than its code, which forward-only migrate cannot repair.- HEAD_AGE_SECONDS=$(( $(date +%s) - $(git show -s --format=%ct HEAD) ))- if [ "$HEAD_AGE_SECONDS" -lt 86400 ]; then- echo "prefix_key=posthog-schema-mig-${SCHEMA_CACHE_EPOCH}-" >> $GITHUB_OUTPUT- fi- # Must match ci-backend.yml's save-side computation byte for byte or the key never hits.- MIG_FILES=$(git ls-tree -r --format='%(objectname) %(path)' "$REF" \- | grep -E '/migrations/[^/]+\.py$|^[0-9a-f]+ uv\.lock$' \- | grep -vE '/(clickhouse|async_migrations)/migrations/' \- | LC_ALL=C sort) || true- # Routing decides which app labels are in the dump, so these inputs belong in the key.- DB_ROUTING=$(git ls-tree -r --format='%(objectname) %(path)' "$REF" -- products/db_routing.yaml posthog/product_db_config.py posthog/product_db_router.py)- PG_IMAGE=$(git show "${REF}:docker-compose.base.yml" 2>/dev/null \- | grep -m1 -E '^[[:space:]]*image:[[:space:]]*postgres:' | tr -d '[:space:]')- if [ -z "$MIG_FILES" ]; then- echo "migrations_key=" >> $GITHUB_OUTPUT- echo "::notice::no migration files matched, so restore falls back to the SHA schema key"- else- MIG_HASH=$(printf '%s\n%s\n%s' "$MIG_FILES" "$PG_IMAGE" "$DB_ROUTING" | sha256sum | cut -c1-40)- echo "migrations_key=posthog-schema-mig-${SCHEMA_CACHE_EPOCH}-${MIG_HASH}" >> $GITHUB_OUTPUT- fi-- - name: Start Docker services- env:- COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml- run: bin/ci-wait-for-docker launch-- - name: Set up Python- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0- with:- python-version-file: 'pyproject.toml'- token: ${{ github.token }}-- # Direct install: The requested historical source revision may not contain the local action.- - name: Install uv- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0- with:- version: '0.12.13'- enable-cache: true- save-cache: ${{ github.ref == 'refs/heads/master' }}-- - name: Install python dependencies- run: UV_PROJECT_ENVIRONMENT=$pythonLocation uv sync --frozen --dev-- - name: Install Rust- uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9- with:- toolchain: stable- components: cargo-- - name: Cache Rust dependencies- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1- with:- shared-key: 'v2-rust-skills'- workspaces: rust- save-if: ${{ github.ref == 'refs/heads/master' }}-- - name: Install sqlx-cli- uses: ./.github/actions/setup-sqlx-cli-- - name: Wait for Docker services- env:- COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml- run: bin/ci-wait-for-docker wait-- - name: Restore schema cache from master- if: steps.schema-key.outputs.migrations_key != '' || steps.schema-key.outputs.key != ''- uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3- with:- path: schema.sql.gz- # Prefer the shared content key; fall back to the exact merge-base SHA- # key, then to the newest dump of any migration set.- key: ${{ steps.schema-key.outputs.migrations_key || steps.schema-key.outputs.key }}- restore-keys: |- ${{ steps.schema-key.outputs.key }}- ${{ steps.schema-key.outputs.prefix_key }}-- - name: Prime posthog from cached schema- # db:restore-schema-fresh restores the dump and seeds the RunPython data a- # schema-only dump lacks; the migrate step below layers anything newer on- # top. On failure it leaves a fresh empty db, so migrate runs the full history.- env:- SECRET_KEY: '6b01eee4f945ca25045b5aab440b953461faf08693a9abbf1166dc7c6b9772da'- DATABASE_URL: 'postgres://posthog:posthog@localhost:5432/posthog'- REDIS_URL: 'redis://localhost'- OBJECT_STORAGE_ENABLED: 'False'- TEST: 1- DEBUG: 1- TARGET_DB: posthog- run: |- if [ ! -f schema.sql.gz ]; then- echo "::notice::Schema cache miss, so the migrate step below runs the full history"- exit 0- fi- mkdir -p .postgres-backups- mv schema.sql.gz .postgres-backups/schema-latest.sql.gz- if ! ./bin/hogli db:restore-schema-fresh; then- echo "::warning::Schema restore failed (stale or incompatible cached dump?), so the migrate step below runs the full history"- fi-- - name: Run migrations- env:- SECRET_KEY: '6b01eee4f945ca25045b5aab440b953461faf08693a9abbf1166dc7c6b9772da'- DATABASE_URL: 'postgres://posthog:posthog@localhost:5432/posthog'- REDIS_URL: 'redis://localhost'- OBJECT_STORAGE_ENABLED: 'False'- TEST: 1- DEBUG: 1- run: |- DATABASE_URL="postgres://posthog:posthog@localhost:5432/posthog_persons" \- sqlx database create- DATABASE_URL="postgres://posthog:posthog@localhost:5432/posthog_persons" \- sqlx migrate run --source rust/persons_migrations/- python manage.py migrate --noinput- python manage.py setup_dev --no-data-- - name: Build skills- env:- SECRET_KEY: '6b01eee4f945ca25045b5aab440b953461faf08693a9abbf1166dc7c6b9772da'- DATABASE_URL: 'postgres://posthog:posthog@localhost:5432/posthog'- REDIS_URL: 'redis://localhost'- OBJECT_STORAGE_ENABLED: 'False'- TEST: 1- DEBUG: 1- run: ./bin/hogli build:skills-- - name: Download context-mill skills- run: |- CONTEXT_MILL_ZIP_URL="https://github.com/PostHog/context-mill/releases/latest/download/skills-mcp-resources.zip"- TARGET_DIR="products/posthog_ai/dist/skills"- tmp_dir=$(mktemp -d)- trap 'rm -rf "$tmp_dir"' EXIT- echo "Downloading context-mill skills from ${CONTEXT_MILL_ZIP_URL}..."- curl -fsSL --retry 2 --retry-delay 3 -o "$tmp_dir/cm.zip" "$CONTEXT_MILL_ZIP_URL"- unzip -q -o "$tmp_dir/cm.zip" -d "$tmp_dir/outer"- mkdir -p "$TARGET_DIR"- count=0- while read -r inner_zip; do- skill_name="$(basename "$inner_zip" .zip)"- skill_name="${skill_name#omnibus-}"- skill_dir="$TARGET_DIR/$skill_name"- mkdir -p "$skill_dir"- unzip -q -o "$inner_zip" -d "$skill_dir"- find "$skill_dir" -name 'SKILL.md' -type f -exec sed -i 's/^\(name: *\)omnibus-/\1/' {} +- count=$((count + 1))- done < <(find "$tmp_dir/outer" -name 'omnibus-*.zip' -type f)- echo "Installed ${count} context-mill skills into ${TARGET_DIR}"-- - name: Upload rendered skills- uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0- with:- name: tasks-golden-skills- path: products/posthog_ai/dist/skills/- retention-days: 1-- bake:- # Arming gate: stay a deliberate no-op until the operator provisions the- # service-account principal + github_oidc TrustMapping + tailnet reach- # and sets HOG_TASKS_GOLDEN_ENABLED=true. Mirrors golden-snapshots.yml's- # HOG_GOLDEN_ENABLED gate. `needs: render_skills` — when render skips- # (unarmed), this skips too. `needs: validate_ref` to check out its- # validated ref, never the raw workflow_dispatch input.- needs: [validate_ref, render_skills]- if: vars.CD_DEPLOY_ENABLED == 'true' && vars.HOG_TASKS_GOLDEN_ENABLED == 'true' && github.ref == 'refs/heads/master'- name: Bake and promote the tasks golden snapshot- runs-on: depot-ubuntu-24.04- timeout-minutes: 240 # worst case: 30m quiet-window guard + 30m cold create + ~90m setup + snapshot push + 30m smoke cold restore + buffer- # Job-level permissions replace the workflow defaults, so re-declare the- # contents/id-token the bake needs and add actions:read for the- # quiet-deploy-window guard's Actions API query.- permissions:- contents: read- id-token: write # mint OIDC for the tailnet join AND the hogplane bearer- actions: read # read hogland rollout runs for the quiet-window guard- strategy:- fail-fast: false- matrix:- include:- - cluster: dev- hog_host: https://hogland-dev.hedgehog-kitefin.ts.net- hog_oidc_audience: hogland.dev.posthog.dev- rollout_workflow: deploy.yml- - cluster: prod-us- hog_host: https://hogland.hedgehog-kitefin.ts.net- hog_oidc_audience: hogland.prod-us.posthog.dev- rollout_workflow: promote-to-prod.yml- # prod-eu is deferred to the EU rollout. To add it, append a- # matrix entry (host https://hogland-eu... + audience- # hogland.prod-eu.posthog.dev + the EU rollout workflow file)- # once its svc-ci principal + TrustMapping exist, and extend- # the prod arming gate below.-- env:- HOG_HOST: ${{ matrix.hog_host }}- HOG_OIDC_AUDIENCE: ${{ matrix.hog_oidc_audience }}- # No stored bearer. The CLI mints one on demand — and re-mints on any- # mid-bake 401 — by hitting the Actions OIDC endpoint and jq-ing the- # token out. ACTIONS_ID_TOKEN_REQUEST_{URL,TOKEN} are ambient for the- # whole job under permissions.id-token: write.- HOG_TOKEN_COMMAND: 'curl -sf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$HOG_OIDC_AUDIENCE" | jq -re .value'-- steps:- - name: Check out posthog- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2- with:- ref: ${{ needs.validate_ref.outputs.ref }}-- # Skip clusters not requested when running manually with the input set.- - name: Filter cluster- if: ${{ github.event.inputs.cluster != '' && github.event.inputs.cluster != matrix.cluster }}- run: echo "skip=true" >> "$GITHUB_ENV"-- # Second arming gate, for prod-targeting clusters. HOG_TASKS_GOLDEN_ENABLED- # covers dev provisioning; a prod cluster additionally needs its own- # svc-ci principal + TrustMapping in that cluster (see the runbook).- # Stay a no-op until an operator flips HOG_TASKS_GOLDEN_PROD_ENABLED.- - name: Gate prod-targeting clusters- if: ${{ env.skip != 'true' && matrix.cluster != 'dev' && vars.HOG_TASKS_GOLDEN_PROD_ENABLED != 'true' }}- run: |- echo "::notice::${{ matrix.cluster }} tasks golden bake skipped — set HOG_TASKS_GOLDEN_PROD_ENABLED=true once the ${{ matrix.cluster }} principal + TrustMapping exist"- echo "skip=true" >> "$GITHUB_ENV"-- # The rendered skills the golden ships, produced by render_skills. Land- # them where build_skills / install-skills.sh expect, then tar the set- # for bake-golden.sh to stream into the box.- - name: Download rendered skills- if: env.skip != 'true'- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0- with:- name: tasks-golden-skills- path: products/posthog_ai/dist/skills/-- - name: Tar rendered skills- if: env.skip != 'true'- run: |- set -euo pipefail- SRC=products/posthog_ai/dist/skills- if [ -z "$(find "$SRC" -mindepth 1 -maxdepth 1 -type d 2>/dev/null)" ]; then- echo "no rendered skills found under $SRC" >&2- exit 1- fi- tar -czf "$RUNNER_TEMP/golden-skills.tar.gz" -C "$SRC" .- echo "SKILLS_TARBALL=$RUNNER_TEMP/golden-skills.tar.gz" >> "$GITHUB_ENV"-- # The hogland CLI lives in the private PostHog/hogland repo. Check it- # out with a scoped App token and build it, mirroring- # golden-snapshots.yml's `go build ./cmd/hogland`.- - name: Mint hogland-repo token- if: env.skip != 'true'- id: hogland-token- uses: actions/create-github-app-token@1b10c78c7865c340bc4f6099eb2f838309f1e8c3 # v3.1.1- with:- app-id: ${{ secrets.GH_APP_HOGLAND_DEPLOYER_APP_ID }}- private-key: ${{ secrets.GH_APP_HOGLAND_DEPLOYER_PRIVATE_KEY }}- owner: PostHog- repositories: hogland- # Read-only: the deployer App may hold write on hogland, but- # this job only reads it — checks out the CLI source to build,- # and the quiet-window guard lists hogland Actions runs.- permission-contents: read- permission-actions: read-- # Build the CLI from the newest released vX.Y.Z-cli tag: immutable- # and reviewed, never a moving branch like main (which would run- # unreviewed CLI code in this credentialed job). Resolved at runtime,- # so there is no HOGLAND_CLI_REF var to bump on each CLI release — the- # CLI only orchestrates the bake (create/snapshot/alias) and does not- # change the golden's contents. Still guard the minimum the scripts need.- - name: Resolve latest hogland CLI release- if: env.skip != 'true'- id: cli-ref- env:- GH_TOKEN: ${{ steps.hogland-token.outputs.token }}- run: |- set -euo pipefail- MIN_CLI_VERSION=1.5.0- ref=$(gh api --paginate repos/PostHog/hogland/releases --jq '.[].tag_name' \- | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-cli$' | sort -V | tail -1)- if [[ -z "$ref" ]]; then- echo "::error::no vX.Y.Z-cli release found on PostHog/hogland"- exit 1- fi- cli_version="${ref#v}"; cli_version="${cli_version%-cli}"- lowest=$(printf '%s\n%s\n' "$MIN_CLI_VERSION" "$cli_version" | sort -V | head -1)- if [[ "$cli_version" != "$MIN_CLI_VERSION" && "$lowest" != "$MIN_CLI_VERSION" ]]; then- echo "::error::latest hogland CLI release ${ref} is below the minimum v${MIN_CLI_VERSION}-cli"- exit 1- fi- echo "ref=${ref}" >> "$GITHUB_OUTPUT"- echo "::notice::building hogland CLI from ${ref}"-- - name: Check out hogland CLI source- if: env.skip != 'true'- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2- with:- repository: PostHog/hogland- ref: ${{ steps.cli-ref.outputs.ref }}- token: ${{ steps.hogland-token.outputs.token }}- path: hogland-src-- - name: Set up Go- if: env.skip != 'true'- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0- with:- go-version-file: hogland-src/go.mod-- - name: Build hogland CLI- if: env.skip != 'true'- run: |- set -euo pipefail- mkdir -p "$RUNNER_TEMP/bin"- (cd hogland-src && go build -o "$RUNNER_TEMP/bin/hogland" ./cmd/hogland)- echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"-- # Join the tailnet as tag:hogland-ci — the tag whose ACL reaches the- # tag:hogplane device serving the API. WIF, no stored secret; the- # client-id + audience are shared with hogbox-preview-env.yml.- - name: Connect to Tailscale (tag:hogland-ci)- if: env.skip != 'true'- uses: tailscale/github-action@306e68a486fd2350f2bfc3b19fcd143891a4a2d8 # v4.1.2- with:- oauth-client-id: ${{ vars.TS_HOGLAND_CI_CLIENT_ID }}- audience: ${{ vars.TS_HOGLAND_CI_AUDIENCE }}- tags: tag:hogland-ci-- # Ephemeral keypair for this run. bake-golden.sh bakes the PUBLIC half- # into the seed box on create and reaches it with the PRIVATE half; the- # smoke box is reached with the same key.- - name: Generate ephemeral ssh key- if: env.skip != 'true'- run: |- set -euo pipefail- ssh-keygen -t ed25519 -f "$RUNNER_TEMP/tasks_golden_key" -N "" -C tasks-golden-ci- echo "GOLDEN_SSH_KEY=$RUNNER_TEMP/tasks_golden_key" >> "$GITHUB_ENV"-- # Resolve the published @posthog/agent version so the golden pins an- # exact, reproducible agent-server — decoupled from any image. Mirrors- # cd-sandbox-base-image.yml's agent-version resolve.- - name: Resolve published agent version- if: env.skip != 'true'- id: agent-version- run: |- set -euo pipefail- version=$(curl -fsSL --retry 3 https://registry.npmjs.org/@posthog%2fagent/latest | jq -r '.version')- if [[ -z "$version" || "$version" == "null" ]]; then- echo "Could not resolve the latest @posthog/agent version" >&2- exit 1- fi- echo "version=$version" >> "$GITHUB_OUTPUT"-- # Quiet-deploy-window guard (mirrors hogland golden-snapshots.yml):- # do not bake while a hogd rollout is queued or in flight, or within- # 900s of the last one — a rollout racing the bake orphans the seed box- # and wastes the run. The rollout workflow differs per cluster- # (dev: deploy.yml, prod-us: promote-to-prod.yml), so it comes from the- # matrix, not a single shared var. Reads PostHog/hogland's Actions with- # the hogland App token (actions:read); an API error fails OPEN (a- # racing rollout only wastes a run, it cannot corrupt the alias).- - name: Wait for a quiet hogd rollout window- if: env.skip != 'true'- env:- GH_TOKEN: ${{ steps.hogland-token.outputs.token }}- ROLLOUT_WORKFLOW: ${{ matrix.rollout_workflow }}- run: |- set -euo pipefail- if [[ -z "$ROLLOUT_WORKFLOW" ]]; then- echo "::notice::no rollout workflow for this cluster; skipping the quiet-window guard"- exit 0- fi- deadline=$(( $(date +%s) + 1800 )) # wait at most 30m for the window- runs_api="repos/PostHog/hogland/actions/workflows/${ROLLOUT_WORKFLOW}/runs"- while :; do- in_progress=$(gh api "${runs_api}?status=in_progress&per_page=1" --jq '.total_count' 2>/dev/null || echo "error")- queued=$(gh api "${runs_api}?status=queued&per_page=1" --jq '.total_count' 2>/dev/null || echo "error")- if [[ "$in_progress" == "error" || "$queued" == "error" ]]; then- echo "::notice::could not read hogland rollout runs (App token may lack actions:read); proceeding"- exit 0- fi- active=$(( in_progress + queued ))- last_completed=$(gh api "${runs_api}?status=completed&per_page=1" --jq '.workflow_runs[0].updated_at // empty' 2>/dev/null || echo "")- since=999999- if [[ -n "$last_completed" ]]; then- since=$(( $(date +%s) - $(date -u -d "$last_completed" +%s 2>/dev/null || echo 0) ))- fi- if [[ "$active" == "0" && "$since" -ge 900 ]]; then- echo "::notice::quiet window: no queued/in-progress rollout, ${since}s since the last"- break- fi- if [[ "$(date +%s)" -ge "$deadline" ]]; then- echo "::error::hogd rollout window did not go quiet within 30m (active=${active}, since=${since}s); not baking"- exit 1- fi- echo "rollout active (in_progress=${in_progress}, queued=${queued}, ${since}s since last); waiting 30s"- sleep 30- done-- # Build into the CANDIDATE alias — no live consumer points at this- # name, so a bad build can't poison task runs. bake-golden.sh boots a- # bare seed box from box primitives, streams the payload (guards + cpu- # sampler + the rendered-skills tarball + install-skills.sh +- # setup-golden.sh) in over the box's SSH, runs setup-golden.sh, then- # snapshots and points the candidate alias at it. Golden shape is the- # task sandbox shape: 4 CPU / 20 GiB / 64 GiB.- - name: Bake candidate snapshot (box primitives)- if: env.skip != 'true'- env:- ALIAS: posthog-tasks-candidate- SSH_KEY: ${{ env.GOLDEN_SSH_KEY }}- IMAGES_DIR: products/tasks/backend/sandbox/images- SKILLS_TARBALL: ${{ env.SKILLS_TARBALL }}- AGENT_VERSION: ${{ steps.agent-version.outputs.version }}- BOX_CPUS: '4'- # Both blank on the schedule -> fixed-size 20 GiB golden, unchanged.- # A manual dispatch can bake a pluggable-memory golden (e.g. cap 32768- # + boot 4096) without touching the scheduled path.- BOX_MEM_MIB: ${{ inputs.box_mem_mib || '20480' }}- BOX_MEM_BOOT_MIB: ${{ inputs.box_mem_boot_mib || '' }}- BOX_DISK_GIB: '64'- run: |- set -euo pipefail- bash products/tasks/backend/sandbox/images/hogland/bake-golden.sh-- # Boot a smoke box from the candidate, run the tasks readiness- # contract, delete. Non-zero => DO NOT promote, so the live alias- # keeps pointing at the previous known-good snapshot.- - name: Smoke-test candidate- id: smoke- if: env.skip != 'true'- env:- ALIAS: posthog-tasks-candidate- SSH_KEY: ${{ env.GOLDEN_SSH_KEY }}- run: |- set -euo pipefail- bash products/tasks/backend/sandbox/images/hogland/smoke-golden.sh | tee "$RUNNER_TEMP/smoke.out"- secs=$(sed -n 's/^SMOKE_HEALTHY_SECONDS=//p' "$RUNNER_TEMP/smoke.out")- echo "healthy_seconds=${secs:-unknown}" >> "$GITHUB_OUTPUT"-- # Promote: re-point the live alias at the candidate's snapshot id. A- # single alias PUT — atomic from any restoring caller's perspective.- - name: Promote candidate to the live alias- id: promote- if: env.skip != 'true'- env:- CLUSTER: ${{ matrix.cluster }}- # Blank -> the default golden every box restores from. A pluggable- # bake sets this to a separate alias (e.g. posthog-tasks-hotplug) so- # it never touches the default; the hotplug flag selects it per team.- TARGET_ALIAS: ${{ inputs.target_alias || 'posthog-tasks-default' }}- run: |- set -euo pipefail- SNAP_ID=$(hogland snapshot resolve posthog-tasks-candidate | jq -r .snapshot_id)- # Retry the alias PUT: a concurrent repoint returns 500 (not a- # clean 409), so a transient collision must not fail the promote.- promoted=0- for attempt in 1 2 3 4 5; do- if hogland snapshot alias "$SNAP_ID" "$TARGET_ALIAS"; then- promoted=1- break- fi- echo "alias PUT attempt ${attempt} failed; retrying in $((attempt * 5))s" >&2- sleep $((attempt * 5))- done- if [[ "$promoted" != "1" ]]; then- echo "::error::failed to repoint $TARGET_ALIAS after 5 attempts"- exit 1- fi- # Confirm the live alias now resolves to our snapshot. This- # resolve targets the same HOG_HOST the PUT wrote to, so it- # catches a lost or overwritten alias write on THIS cluster — it- # does not prove the bake ran against the intended cluster (both- # the PUT and this read use the same matrix HOG_HOST).- resolved=$(hogland snapshot resolve "$TARGET_ALIAS" | jq -r .snapshot_id)- if [[ "$resolved" != "$SNAP_ID" ]]; then- echo "::error::$TARGET_ALIAS in cluster ${CLUSTER} resolves to ${resolved}, expected ${SNAP_ID}"- exit 1- fi- echo "snapshot_id=$SNAP_ID" >> "$GITHUB_OUTPUT"-- # Dated archive alias for rollback. Reuse the snapshot id the promote- # step already resolved (via env, not ${{ }} into run:) rather than- # resolving the candidate alias again — a second resolve could race a- # concurrent rebuild and archive a different snapshot than was- # promoted. Retention prune is intentionally not wired here — a- # separate cleanup cron handles it once we've watched the alias count- # for a couple of weeks.- - name: Archive dated alias- if: env.skip != 'true'- env:- SNAP_ID: ${{ steps.promote.outputs.snapshot_id }}- run: |- set -euo pipefail- DATE=$(date -u +%Y%m%d)- hogland snapshot alias "$SNAP_ID" posthog-tasks-default-"$DATE"-- # Delete the seed + smoke boxes by name, always. The seed and smoke- # boxes use an unregistered kind (no server-side TTL), so a killed- # runner cannot reap them — and their in-script EXIT traps do not fire- # on a job cancel. This always()-guarded step is the backstop that- # prevents a leaked 64 GiB box on cancel/timeout. Best-effort: the- # names are fixed in bake-golden.sh / smoke-golden.sh, and a delete of- # an absent box is a no-op. Runs only after the CLI + tailnet came up- # (env.skip guards the cluster-filtered legs, which create no box).- - name: Tear down seed + smoke boxes- if: ${{ always() && env.skip != 'true' }}- run: |- set -uo pipefail- for box_name in golden-seed-tasks golden-smoke-tasks; do- id=$(hogland box list 2>/dev/null \- | jq -r --arg n "$box_name" 'map(select(.spec.name == $n)) | .[0].id // empty' 2>/dev/null || true)- if [[ -n "$id" ]]; then- echo "deleting leaked box $box_name ($id)"- hogland box delete "$id" >/dev/null 2>&1 || true- fi- done-- # Step outputs go through env vars, not direct ${{ }} interpolation- # into run: — semgrep's shell-injection rule treats step output as- # untrusted, and env indirection is the standard fix.- - name: Step summary- if: env.skip != 'true'- env:- CLUSTER: ${{ matrix.cluster }}- HOG_HOST: ${{ matrix.hog_host }}- SNAPSHOT_ID: ${{ steps.promote.outputs.snapshot_id }}- HEALTHY_SECONDS: ${{ steps.smoke.outputs.healthy_seconds }}- run: |- {- echo "### tasks golden snapshot: $CLUSTER"- echo ""- echo "- cluster: \`$CLUSTER\` ($HOG_HOST)"- echo "- promoted snapshot \`$SNAPSHOT_ID\` -> alias \`posthog-tasks-default\`"- echo "- smoke create->contract: ${HEALTHY_SECONDS}s"- } >> "$GITHUB_STEP_SUMMARY"diff --git a/docs/published/handbook/engineering/ai/writing-skills.md b/docs/published/handbook/engineering/ai/writing-skills.mdindex 4434a13fce48..0b6aff5fe01b 100644--- a/docs/published/handbook/engineering/ai/writing-skills.md+++ b/docs/published/handbook/engineering/ai/writing-skills.md@@ -442,7 +442,6 @@ so a skill from this repo is absent there rather than overwritten.| PostHog Desktop runtime, every 30 min | `products/desktop/packages/workspace-server/src/services/posthog-plugin/update-skills-saga.ts` || Desktop harness bundle | `packages/agent/packages/harness/tsup.config.ts` – context-mill only, this repo's skills are absent rather than overwritten || Tasks sandbox base image | `.github/workflows/cd-sandbox-base-image.yml` |-| Tasks golden snapshot | `.github/workflows/cd-tasks-golden-snapshot.yml` || `PostHog/skills` mirror | that repo's `.github/workflows/sync-omnibus.yml` || `PostHog/ai-plugin` plugin | that repo's `.github/workflows/sync-skills.yml` |diff --git a/products/tasks/backend/logic/services/hogland_sandbox.py b/products/tasks/backend/logic/services/hogland_sandbox.pyindex edee6268f47e..c6f7b264a469 100644--- a/products/tasks/backend/logic/services/hogland_sandbox.py+++ b/products/tasks/backend/logic/services/hogland_sandbox.py@@ -62,15 +62,15 @@ logger = logging.getLogger(__name__)# instead of minting an immortal box.HOGLAND_TASKS_BOX_KIND = "agent"-# Every template hogland supports maps to a global snapshot alias baked in CI-# (the tasks golden-snapshot workflow). Only the plain default-template run is routed+# Every template hogland supports maps to a global snapshot alias. No CI job bakes+# these any more; the alias must already exist. Only the plain default-template run is routed# to hogland; anything else stays on Modal (see get_task_processing_context).TEMPLATE_TO_SNAPSHOT_ALIAS: dict[SandboxTemplate, str] = {SandboxTemplate.DEFAULT_BASE: "alias:posthog-tasks-default",}-# The pluggable-memory golden (baked by the tasks golden-snapshot workflow with a-# BOX_MEM_BOOT_MIB below the cap). A box restored from it boots small and hot-adds guest+# The pluggable-memory golden (boot memory below the cap). A box restored from it+# boots small and hot-adds guest# RAM up to the cap on demand. Selected per run by the tasks-hogland-hotplug-golden flag# (config.use_hotplug_golden); only the default template has a pluggable variant. The# alias must exist before the flag is enabled for a team.diff --git a/products/tasks/backend/sandbox/images/hogland/GOLDEN_CI_RUNBOOK.md b/products/tasks/backend/sandbox/images/hogland/GOLDEN_CI_RUNBOOK.mddeleted file mode 100644index fe883e4bc9bb..000000000000--- a/products/tasks/backend/sandbox/images/hogland/GOLDEN_CI_RUNBOOK.md+++ /dev/null@@ -1,297 +0,0 @@-# Tasks golden snapshot CI — ops runbook--The workflow `.github/workflows/cd-tasks-golden-snapshot.yml` bakes the task-sandbox golden snapshot and promotes the alias `posthog-tasks-default` that-`HoglandSandbox.create`-(`products/tasks/backend/logic/services/hogland_sandbox.py`) restores every task-box from.--This runbook covers the hogland-team dependencies each cluster needs **before**-the workflow can run. Until they exist and the arming variables are set, the-workflow is a deliberate no-op (the job `if` gates on `HOG_TASKS_GOLDEN_ENABLED`)-so a merged-but-unprovisioned workflow never sits red.--Modal / `Dockerfile.sandbox-base` is untouched by this workflow — it bakes the-same contents into a hogland snapshot, it does not change the container image.--## What the workflow does--0. `validate_ref` job (unprivileged, no secrets) — resolves the- `workflow_dispatch` `ref` input before any credentialed job checks that tree- out. Blank resolves to the literal `master` branch; a non-blank value must be- a full 40-char commit SHA that the GitHub compare API confirms is already an- ancestor of `master`, else the job fails. `render_skills` and `bake` check out- `needs.validate_ref.outputs.ref`, never the raw input, and every privileged- job additionally gates on `github.ref == 'refs/heads/master'`. This closes the- `ref` RCE in the workflow YAML (see "Known open security issues").-1. `render_skills` job — renders the agent skills the golden ships, the same way- `cd-sandbox-base-image.yml`'s `build_skills` job does. It stands up a DB,- restores the schema cache saved on master so `migrate` only tops up the newer- migrations, runs `setup_dev --no-data`, then `hogli build:skills` to expand the- skill `.md.j2` templates and merges in the context-mill skills. The DB is- needed because 24 skill templates call `render_hogql_example`, which reads- `Team.objects.first()`. Uploads the merged set as the `tasks-golden-skills`- artifact. Rendered once, reused by every cluster. It shares the master arm- gate, so an unarmed nightly renders nothing.-2. Joins the hogland tailnet as `tag:hogland-ci` (the tag whose ACL reaches the- `tag:hogplane` device serving the API).-3. Authenticates to hogplane as a per-cluster `svc-ci-*` service-account- principal via GitHub OIDC — no stored bearer. The CLI mints the token from the- Actions OIDC endpoint and re-mints on any mid-bake 401 via `HOG_TOKEN_COMMAND`,- because a golden bake runs ~90 min while an OIDC token lives ~5 min.-4. Builds the `hogland` CLI from a scoped checkout of `PostHog/hogland`.-5. `bake-golden.sh` — decomposes the bake into box primitives instead of- `hogland snapshot-build` (whose <=256 KiB bootstrap cannot carry the multi-MB- skills set). It `box create`s a bare seed box (cold boot, `--no-connect`),- streams the payload into the box over the box's own SSH (`cat >` — the git/gh- guards, the cpu sampler, the rendered-skills tarball, `install-skills.sh`, and- `setup-golden.sh`), runs `setup-golden.sh` in the box, then `box snapshot`s the- result and points the candidate alias at it. `setup-golden.sh` installs- node/uv/tools/agentsh fresh, npm-installs `@posthog/agent` at the version the- workflow resolved, and installs the delivered skills. No GHCR image and no- public artifact host are involved — delivery rides the box's SSH.-6. Boots a smoke box from `posthog-tasks-candidate`, runs `smoke-golden.sh`- (agent-server starts; a trivial clone + exec works; the hogpanion exec daemon- is running with the container-style env), and deletes it. A smoke failure- skips promotion, leaving `posthog-tasks-default` on the previous known-good- snapshot.-7. On success, re-points `posthog-tasks-default` at the candidate's snapshot- (retrying the alias PUT, since a concurrent repoint returns 500 not 409),- confirms the live alias resolves to that snapshot in the expected cluster, and- stamps a dated archive alias `posthog-tasks-default-YYYYMMDD` for rollback.--Before the bake step, a quiet-deploy-window guard waits for no queued or-in-progress hogd rollout and >=900s since the last one (up to 30 min), mirroring-hogland's `golden-snapshots.yml`. The rollout workflow is per cluster (see "Repo-config").--The matrix covers **dev** and **prod-us**. **prod-eu is deferred** to the EU-rollout — see "Adding prod-eu" below.--## Per-cluster prerequisites (hogland team, out-of-band)--These are hogland runtime-cluster data, not repo config. They mirror the-`preview` persona's provisioning in hogland's `docs/BUILD_TEST_DEPLOY.md`.--For **each** cluster the workflow targets (dev, then prod-us):--1. **Service-account principal.** Create a `svc-ci-tasks-golden` principal via- `POST /v1/service-accounts` on that cluster's hogplane. This endpoint is- admin-only and has no CLI; it mints a name of the form `svc-<slug>-<rand>`, so- the random suffix is expected — do **not** read it as a failure.-2. **`github_oidc` TrustMapping.** Map this repo's workflow to that principal:- `{repo: PostHog/posthog, workflow: cd-tasks-golden-snapshot.yml}` →- `svc-ci-tasks-golden`. TrustMappings are per-cluster runtime data — the dev- mapping grants nothing in prod-us, so create it again per cluster. Without it- the OIDC mint succeeds and hogplane returns 401. **Pin `ref: refs/heads/master`- once hogplane's TrustMapping schema supports it** (PostHog/hogland#414, durable- fix PostHog/hogland#424) — this is the durable fix for the branch-dispatch- class of attack. The workflow's own- `validate_ref` job + per-job `github.ref == 'refs/heads/master'` gates raise- the bar, but a branch copy of the workflow can strip in-file gates, so the- TrustMapping ref pin is the backstop that cannot be bypassed from the repo.-3. **OIDC audience.** The mint audience is the per-cluster literal in the- workflow's matrix, kept equal to `hogplane.githubOIDCAudience` in that- cluster's values file:-- | Cluster | `HOG_HOST` | Audience (`HOG_OIDC_AUDIENCE`) |- | ------- | --------------------------------------------- | ------------------------------ |- | dev | `https://hogland-dev.hedgehog-kitefin.ts.net` | `hogland.dev.posthog.dev` |- | prod-us | `https://hogland.hedgehog-kitefin.ts.net` | `hogland.prod-us.posthog.dev` |--4. **Tailnet admittance.** The workflow joins as `tag:hogland-ci`, reusing the- `TS_HOGLAND_CI_CLIENT_ID` + `TS_HOGLAND_CI_AUDIENCE` repo variables that- `hogbox-preview-env.yml` already uses. One join reaches the `tag:hogplane`- device serving each cluster's API, so if preview is already provisioned no- extra tailnet onboarding is needed for the API calls.-5. **Tailnet route to the box VPC (required for `--access-type ssh-private`).**- The bake and smoke reach the seed/smoke box over its VPC IP, not the public- internet. `tag:hogland-ci` reaches `tag:hogplane` today, but **not** a box's- VPC IP — so `ssh-private` fails at the first `ssh` until a grant is added in- `posthog-cloud-infra/tailnet-policy.hujson`:-- ```text- src: tag:hogland-ci- dst: 10.90.0.0/16:10000-59999, 10.91.0.0/16:10000-59999- ```-- This is an ops/infra prerequisite (like the svc-ci principal and arming), not- a repo change. Do not arm any cluster before it lands, or every bake fails on- the SSH reachability wait.--## Repo config (PostHog/posthog)--- **Variables** (Settings → Secrets and variables → Actions → Variables):- - `HOG_TASKS_GOLDEN_ENABLED=true` — master arm. Until set, the whole job is a- no-op. Set it once the dev prerequisites (steps 1–5, including the tailnet route to the box VPC) exist.- - `HOG_TASKS_GOLDEN_PROD_ENABLED=true` — second arm for prod-targeting- clusters. The prod-us leg stays a no-op until this is set, so it does not- fail nightly before its own principal + TrustMapping exist.- - No `HOGLAND_CLI_REF` needed — the workflow resolves the newest released- `vX.Y.Z-cli` tag from `PostHog/hogland` at runtime (immutable and reviewed,- never `main`), guarding a `v1.5.0-cli` minimum.- - The quiet-window guard's rollout workflow is **per cluster**, hardcoded in the- bake matrix (dev: `deploy.yml`, prod-us: `promote-to-prod.yml`) — not a shared- var, because the two clusters roll out through different workflows. The guard- counts `queued` + `in_progress` runs and waits up to 30 min for a quiet window.- It reads `PostHog/hogland` Actions with the hogland App token, so that App- installation needs `actions:read` on hogland; without it the guard fails open- (a racing rollout only wastes a run, it cannot corrupt the alias). Adding a- cluster means adding its rollout workflow file to the matrix.- - `TS_HOGLAND_CI_CLIENT_ID`, `TS_HOGLAND_CI_AUDIENCE` — shared with- `hogbox-preview-env.yml`; already present if preview is provisioned.-- **Secrets** — the workflow checks out `PostHog/hogland` to build the CLI with- the existing `GH_APP_HOGLAND_DEPLOYER` GitHub App (org secrets- `GH_APP_HOGLAND_DEPLOYER_APP_ID` + `GH_APP_HOGLAND_DEPLOYER_PRIVATE_KEY`),- minted read-only: `contents: read` for the checkout, `actions: read` so the- quiet-window guard can list hogland rollout runs. The App must be installed on- `PostHog/hogland` with both.--## Running it--- Manual: Actions → "Tasks Golden Snapshot CD" → Run workflow. Leave `cluster`- blank to bake every armed cluster, or set it to `dev` / `prod-us` to bake one.- Leave `ref` blank to render from `master`, or set it to a full commit SHA that- is already merged into `master`. `validate_ref` enforces this: a blank ref- resolves to `master`, a non-blank value must be a full 40-char SHA that is an- ancestor of `master`, and anything else (a branch name, an unmerged SHA) fails- the run before any credentialed job starts. Dispatch only works from `master`- (every privileged job gates on `github.ref == 'refs/heads/master'`).-- Nightly: the `schedule` cron fires daily but only bakes armed clusters,- rendering from `master`.--## Cleaning up leaked seed/smoke boxes--The seed (`golden-seed-tasks`) and smoke (`golden-smoke-tasks`) boxes use an-unregistered kind with no server-side TTL, so hogland never reaps them. The-workflow's `always()` teardown deletes them on a normal cancel or failure — but-it does **not** run if the runner itself dies (an infra kill, a spot reclaim),-which leaks a 64 GiB box. The names are fixed, so a leak is at most one seed +-one smoke per cluster (the next run pre-cleans them). After any run that ended-abnormally, sweep them (dev/prod-us need the tailnet):--```bash-for n in golden-seed-tasks golden-smoke-tasks; do- id=$(hogland box list | jq -r --arg n "$n" 'map(select(.spec.name==$n)) | .[0].id // empty')- [ -n "$id" ] && hogland box delete "$id"-done-```--## Rollback--Each successful bake stamps `posthog-tasks-default-YYYYMMDD`. To roll back, point-the live alias at a known-good archive:--```bash-hogland snapshot resolve posthog-tasks-default-<YYYYMMDD> | jq -r .snapshot_id-hogland snapshot alias <snapshot_id> posthog-tasks-default-```--After hogland#426 the alias PUT is owner-scoped: only the alias's owner (the-`svc-ci-*` principal that first promoted it) or an admin may repoint-`posthog-tasks-default`, so a hand rollback runs as an admin or as svc-ci. See-hogland's `docs/BUILD_TEST_DEPLOY.md` "Alias ownership" section for the adoption-and recovery model.--## Adding prod-eu--1. Provision the prerequisites above in the prod-eu cluster: a- `svc-ci-tasks-golden` principal, a `github_oidc` TrustMapping- `{repo: PostHog/posthog, workflow: cd-tasks-golden-snapshot.yml}` (pin- `ref: refs/heads/master` once PostHog/hogland#414 lands), and the audience- `hogland.prod-eu.posthog.dev`.-2. Append a matrix entry in the workflow (host + audience + the EU rollout- workflow file) and confirm the prod arming gate covers it.-3. Confirm `tag:hogland-ci` reaches the prod-eu `tag:hogplane` device.--## Known Phase-1 gaps (validate on a live cluster)--- **No live bake has run.** The scripts and workflow are lint- and- shellcheck-clean, but a real bake needs a cluster, the tailnet, and the- principal — none of which exist in local dev. First armed run is the real test.-- **Smoke asserts the daemon env, and best-effort the exec API.** `smoke-golden.sh`- reads the running hogpanion daemon's live `/proc/<MainPID>/environ` (via `sudo`;- the box ssh user is `hog`) — the env its hog-exec children inherit — and asserts- `IS_SANDBOX=1`, a `/opt/posthog/bin` first `PATH`, and `PYTHONPATH`. This catches- a hogpanion that never re-exec'd with the drop-in, which an SSH login shell would- hide (PAM feeds it `/etc/environment`). It also now attempts a trivial command- through `POST /v1/hogboxes/{id}/exec` — the production reach path — but that- assertion **fails open** only on `000` (genuinely unreachable). A `404` (missing- box or non-owner) is now a hard fail, since it would be a real regression. Once- PostHog/hogland#416 lands a `box exec`/`box cp` verb (durable fix- PostHog/hogland#425) and the exec contract is confirmed, make the exec assertion- a hard gate on all codes.-- **Quiet-deploy-window guard is provisional.** The guard polls `PostHog/hogland`- Actions for the per-cluster rollout workflow's queued + in-progress runs and- waits up to 30 min for a quiet window, but **fails open** when the hogland App- lacks `actions:read` on hogland. Grant the App `actions:read` to arm it. A racing- rollout without the guard only wastes a run (it orphans the seed box and skips- promotion), it cannot corrupt the alias.-- **The golden tracks the render ref (default `master`).** The `render_skills` job- checks out PostHog at the validated ref and renders the skills from it, and- `setup-golden.sh` reconstructs `Dockerfile.sandbox-base` from the same checkout —- so the ref is the one knob for what the golden tracks. On the nightly schedule- this is `master`; pass the `ref` `workflow_dispatch` input (a merged SHA) for a- reproducible golden. `@posthog/agent` is decoupled from the ref: the workflow- resolves the latest published version and pins it into the bake.-- **Cluster confirm catches a lost write, not a wrong-cluster bake.** The promote- step re-resolves `posthog-tasks-default` after the PUT, but on the same- per-cluster `HOG_HOST` it wrote to. That catches a lost or overwritten alias- write on this cluster; it does not prove the bake targeted the intended cluster.- A cheap cluster-identity query on the CLI would let it assert the expected- cluster name — not available today, so this stays a lost-write check.-- **The `tasks-golden-skills` artifact is public-downloadable.** It is a- build artifact on a public repo, so anyone can download it. Its content is- repo-derived (rendered skills + fixtures) — confirm the render emits nothing- beyond repo fixtures before treating the golden as sensitive-free.-- **Unpinned third-party installs.** `setup-golden.sh` runs the nodesource- `setup_24.x` piped to `bash`, the gh `.deb` with no sha256, and global npm- tools — the same posture as `Dockerfile.sandbox-base`. The nightly rebake- widens the window in which an upstream change lands in a golden unnoticed.-- **hogland#410 (per-box `PATH`) forces a full rebake.** When it lands it requires- a rebake of every golden, and its per-box `PATH` overrides this drop-in.--## Known open security issues--The `ref` RCE and branch-dispatch findings from PR review are now **mitigated in-`cd-tasks-golden-snapshot.yml`**: the unprivileged `validate_ref` job resolves and-ancestor-checks the `ref` input before any credentialed checkout, and every-privileged job gates on `github.ref == 'refs/heads/master'`. The residual, durable-gap is hogland-owned:--- **Branch dispatch — durable fix is hogland-side (PostHog/hogland#414, durable- fix PostHog/hogland#424).** The in-file `github.ref == 'refs/heads/master'` gates- raise the bar, but a branch copy of the workflow can strip them. The fix that- cannot be bypassed from the repo is pinning `ref: refs/heads/master` in each- cluster's `github_oidc` TrustMapping. Until #414 lands and the mapping is pinned,- treat the in-workflow gates as the only barrier and keep dispatch discipline- (dispatch only from `master`).-- **Alias namespace is global and unprotected (PostHog/hogland#415, durable fix- PostHog/hogland#426).** Alias PUT checks only new-snapshot ownership and DELETE- checks nothing, so a `svc-ci-tasks-golden` principal on prod-us could repoint or- delete `devbox-golden` / `posthog-preview-golden`. **Keep prod-us UNARMED- (`HOG_TASKS_GOLDEN_PROD_ENABLED` unset) until #415 lands.** Arming dev grants the- CI principal delete over `devbox-golden` (which lives on dev), so dev is only- acceptable to arm knowingly — #426 mitigates this once it lands.--## hogland dependencies (filed issues)--Items owned by the hogland team, filed from this review. Not blockers for merging-the workflow (it is a no-op until armed), but referenced above:--- **PostHog/hogland#414** — `github_oidc` TrustMapping cannot pin a `ref` (durable- branch-dispatch fix, PR PostHog/hogland#424). Gates arming prod-us safely.-- **PostHog/hogland#415** — alias namespace PUT/DELETE unprotected (durable fix PR- PostHog/hogland#426). With #426 merged, `svc-ci` adopts `posthog-tasks-default`- on its first PUT (a new alias is owned by its creator), so no extra- provisioning step — see hogland `docs/BUILD_TEST_DEPLOY.md` (Alias ownership).-- **PostHog/hogland#416** — no `box exec` / `box cp` verb: `hogland box exec` does- not exist today, so the smoke uses a raw `POST .../exec` (durable fix PR- PostHog/hogland#425). Also covers the batch `/exec` timeout mismatch (advertises- 6h, caps at 60s).-- **PostHog/hogland#417** — snapshot chunks are never GC'd (a daily bake leaks- ~10-25 GiB). Must be reference-aware, **not** a bucket-wide expiry rule (that- caused the 2026-08-06 preview outage).-- **PostHog/hogland#418** — hogbox env / `EnvironmentFile` ordering: an- `EnvironmentFile=` key can override the guard-critical `Environment=` vars, so- the drop-in sets no `EnvironmentFile`.diff --git a/products/tasks/backend/sandbox/images/hogland/bake-golden.sh b/products/tasks/backend/sandbox/images/hogland/bake-golden.shdeleted file mode 100755index 64d19032ef37..000000000000--- a/products/tasks/backend/sandbox/images/hogland/bake-golden.sh+++ /dev/null@@ -1,255 +0,0 @@-#!/usr/bin/env bash-# bake-golden.sh — bake the PostHog *tasks* golden snapshot from box primitives.-#-# This replaces `hogland snapshot-build`. snapshot-build can only deliver a-# <=256 KiB bootstrap (--setup-script + --inline-file heredocs), which cannot-# carry the multi-MB rendered-skills set. So we decompose the bake into the-# underlying primitives and stream the payload in over the box's own SSH:-#-# 1. `box create` a bare seed box from the base (cold boot, --no-connect).-# 2. Stream the payload into the box over SSH (cat > file): the git/gh guards,-# the cpu sampler, the rendered-skills tarball, install-skills.sh, and-# setup-golden.sh. Only `cat` is needed in the box, so this works without a-# remote scp/sftp binary and without any public artifact host.-# 3. Run setup-golden.sh in the box over SSH. It installs node/uv/tools/agentsh-# fresh, npm-installs @posthog/agent, and installs the delivered skills.-# 4. `box snapshot` the result and point the candidate alias at it.-#-# Success detection is ours now (snapshot-build appended /var/lib/hog/-# snapshot-build-ok and polled it; we do not). setup-golden.sh runs under-# `set -euo pipefail`, and we check its SSH exit code explicitly — a non-zero-# setup aborts the bake before `box snapshot`, so a broken build never promotes.-#-# Required env (set by the workflow):-# ALIAS — candidate alias to point at the new snapshot-# SSH_KEY — path to the PRIVATE key; the public half is at ${SSH_KEY}.pub-# IMAGES_DIR — path to products/tasks/backend/sandbox/images (payload source)-# SKILLS_TARBALL — path to the runner-rendered skills tarball (.tar.gz)-# Optional env:-# AGENT_VERSION — @posthog/agent version to install in-box (default: latest)-# BOX_CPUS / BOX_MEM_MIB / BOX_DISK_GIB — seed box shape (default 4 / 20480 / 64)-# BOX_MEM_BOOT_MIB — boot memory (MiB) for a pluggable-memory golden: the box-# boots this small and hot-adds guest RAM up to BOX_MEM_MIB on demand. Blank-# (default) bakes a fixed-size golden at BOX_MEM_MIB.-#-# On success prints a single parseable line to stdout:-# BAKED_SNAPSHOT_ID=<id>--set -euo pipefail--: "${ALIAS:?ALIAS is required}"-: "${SSH_KEY:?SSH_KEY is required (private key path; pub at \${SSH_KEY}.pub)}"-: "${IMAGES_DIR:?IMAGES_DIR is required (path to sandbox/images)}"-: "${SKILLS_TARBALL:?SKILLS_TARBALL is required (path to rendered-skills tarball)}"-AGENT_VERSION="${AGENT_VERSION:-latest}"-BOX_CPUS="${BOX_CPUS:-4}"-BOX_MEM_MIB="${BOX_MEM_MIB:-20480}"-BOX_MEM_BOOT_MIB="${BOX_MEM_BOOT_MIB:-}"-BOX_DISK_GIB="${BOX_DISK_GIB:-64}"--log() { printf '[tasks-bake] %s\n' "$*" >&2; }--GIT_GUARD="$IMAGES_DIR/git-guard.sh"-GH_GUARD="$IMAGES_DIR/gh-guard.sh"-HOGLI_SHIM="$IMAGES_DIR/hogli-shim.sh"-CPU_SAMPLER="$IMAGES_DIR/cpu_billing_sampler.py"-SETUP_SCRIPT="$IMAGES_DIR/hogland/setup-golden.sh"-INSTALL_SKILLS="$IMAGES_DIR/install-skills.sh"--for f in "$GIT_GUARD" "$GH_GUARD" "$HOGLI_SHIM" "$CPU_SAMPLER" "$SETUP_SCRIPT" "$INSTALL_SKILLS" "$SKILLS_TARBALL" "${SSH_KEY}.pub"; do- test -f "$f" || { log "FAIL: required file missing: $f"; exit 1; }-done--# Fixed name so a killed prior run's leftover seed box is found and cleaned-# rather than leaking a node or colliding on the next create (hogplane 409s on a-# duplicate name — matches smoke-golden.sh's fixed-name teardown).-SEED_BOX_NAME=golden-seed-tasks-SEED_BOX_ID=""--# List across ALL kinds, not just ours: hogplane's name-uniqueness check-# (ensureBoxNameUnique) scans every kind, so a same-named box of another kind-# would wedge the create. Match by name only so the pre-clean finds it.-resolve_seed_box_id() {- hogland box list 2>/dev/null \- | jq -r --arg n "$SEED_BOX_NAME" 'map(select(.spec.name == $n)) | .[0].id // empty' 2>/dev/null \- || true-}--cleanup() {- if [[ -z "$SEED_BOX_ID" ]]; then- SEED_BOX_ID=$(resolve_seed_box_id)- fi- if [[ -n "$SEED_BOX_ID" ]]; then- log "deleting seed box $SEED_BOX_ID"- hogland box delete "$SEED_BOX_ID" >/dev/null 2>&1 || true- fi-}-trap cleanup EXIT--# Defensive pre-clean: a previous killed run may have leaked a box under our-# fixed name, and every subsequent create would 409 on it. Wait for the delete-# to land (it is async) before creating.-stale_id=$(resolve_seed_box_id)-if [[ -n "$stale_id" ]]; then- log "pre-cleaning stale seed box $stale_id (name $SEED_BOX_NAME)"- hogland box delete "$stale_id" >/dev/null 2>&1 || true- gone_deadline=$(( $(date +%s) + 120 ))- while [[ -n "$(resolve_seed_box_id)" ]]; do- if [[ "$(date +%s)" -ge "$gone_deadline" ]]; then- log "FAIL: stale seed box $stale_id still present 2m after delete"- exit 1- fi- sleep 5- done- log "stale seed box gone"-fi--# Cold boot from the base (no --snapshot-id), so cpus/mem/disk are required and-# fix the snapshot's machine shape to the task sandbox shape. --no-connect keeps-# create non-interactive. --name gives teardown a handle that survives never-# learning the id.-# When BOX_MEM_BOOT_MIB is set (and below the cap), the box boots small and-# hot-adds guest RAM up to BOX_MEM_MIB via virtio-mem; the golden snapshot pins-# that geometry, so every restored task box inherits it. Blank => fixed-size.-mem_boot_args=()-if [[ -n "$BOX_MEM_BOOT_MIB" ]]; then- mem_boot_args+=(--memory-boot-mib "$BOX_MEM_BOOT_MIB")- log "creating seed box (cold boot, $BOX_CPUS cpu / boot ${BOX_MEM_BOOT_MIB}->cap $BOX_MEM_MIB MiB / $BOX_DISK_GIB GiB)"-else- log "creating seed box (cold boot, $BOX_CPUS cpu / $BOX_MEM_MIB MiB / $BOX_DISK_GIB GiB)"-fi-create_rc=0-# --kind posthog-tasks-golden: an UNREGISTERED kind carries no server-side idle-# TTL, so the box is never reaped mid-bake. A registered kind like `ci` expires-# at max(LastUsedAt, CreatedAt)+2h, and only API calls bump LastUsedAt — this-# bake is all SSH, so a 30m-create + 90m-bake would exceed 2h and get reaped.-# The leak-on-cancel that no-TTL reintroduces is covered by the workflow's-# always() teardown step, which deletes both boxes by name.-# --access-type ssh-private: keep the box off the public internet; the runner-# reaches it over the tailnet (default ssh-public DNATs a public port).-# --disk-mbps 0 --disk-iops 0: match hogland's own bakes (CLI defaults 125MB/s /-# 3000 IOPS otherwise).-# --timeout 30m: a cold 64 GiB box may wait on Karpenter to provision a node.-box_json=$(- hogland box create \- --ssh-key "${SSH_KEY}.pub" \- --name "$SEED_BOX_NAME" \- --kind posthog-tasks-golden \- --access-type ssh-private \- --cpus "$BOX_CPUS" \- --memory-mib "$BOX_MEM_MIB" \- ${mem_boot_args[@]+"${mem_boot_args[@]}"} \- --disk-gib "$BOX_DISK_GIB" \- --disk-mbps 0 \- --disk-iops 0 \- --no-connect \- --timeout 30m-) || create_rc=$?-SEED_BOX_ID=$(printf '%s' "$box_json" | jq -r '.id // empty' 2>/dev/null || true)-if [[ "$create_rc" -ne 0 ]]; then- log "FAIL: box create exited $create_rc"- printf '%s\n' "$box_json" >&2- exit 1-fi-ssh_cmd=$(printf '%s' "$box_json" | jq -r '.ssh_command // empty')-if [[ -z "$ssh_cmd" ]]; then- log "FAIL: box create returned no ssh_command; cannot reach seed box"- printf '%s\n' "$box_json" >&2- exit 1-fi-log "seed box $SEED_BOX_ID ssh_command: $ssh_cmd"--# Rebuild the ssh invocation from the returned command, injecting our key +-# non-interactive options right after the `ssh` word (mirrors smoke-golden.sh).-# IdentitiesOnly pins to our ephemeral key; BatchMode never prompts; accept-new-# trusts the fresh host key silently. ServerAlive* guards the ONE long-lived-# session the whole bake runs over: at 30s x 10 it tolerates a ~5m stall before-# dropping. A loaded 4-vCPU box building git from source can stall well past the-# old 15s x 3 (45s) window, which would drop the entire bake mid-step.-read -r -a ssh_parts <<<"$ssh_cmd"-ssh_opts=(-i "$SSH_KEY" -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=10)-ssh_base=("${ssh_parts[0]}" "${ssh_opts[@]}" "${ssh_parts[@]:1}")--log "waiting for ssh reachability (up to 5m)"-ssh_deadline=$(( $(date +%s) + 300 ))-until timeout 20 "${ssh_base[@]}" -o ConnectTimeout=5 true 2>/dev/null; do- if [[ "$(date +%s)" -ge "$ssh_deadline" ]]; then- log "FAIL: ssh never became reachable within 5m"- exit 1- fi- sleep 5-done-log "ssh reachable"--# Stream one local file to a path in the box over SSH, chmod it, and verify the-# byte count landed. The box's ssh user is `hog` (not root), so the write goes-# through passwordless sudo — target paths like /opt/posthog/bin are root-owned.-# `sudo tee` is the delivery primitive; no scp/sftp, no public host. Usage:-# deliver <local-path> <box-path> <mode>-deliver() {- local src="$1" dst="$2" mode="$3" dir size remote_size- dir=$(dirname "$dst")- if ! "${ssh_base[@]}" "sudo mkdir -p '$dir' && sudo tee '$dst' >/dev/null && sudo chmod '$mode' '$dst'" < "$src"; then- log "FAIL: could not deliver $src -> $dst"- exit 1- fi- size=$(wc -c < "$src" | tr -d '[:space:]')- remote_size=$("${ssh_base[@]}" "wc -c < '$dst' | tr -d '[:space:]'" 2>/dev/null || echo "")- if [[ "$size" != "$remote_size" ]]; then- log "FAIL: $dst is $remote_size bytes in box, expected $size"- exit 1- fi- log "delivered $dst ($size bytes, mode $mode)"-}--# Guards + cpu sampler land at their final paths with mode 0755, so setup-golden.sh-# only has to `test -x` them. The skills tarball, installer, and setup script stage-# under /tmp for setup-golden.sh to consume.-deliver "$GIT_GUARD" /opt/posthog/bin/git 0755-deliver "$GH_GUARD" /opt/posthog/bin/gh 0755-deliver "$HOGLI_SHIM" /opt/posthog/bin/hogli 0755-deliver "$CPU_SAMPLER" /usr/local/bin/posthog-cpu-billing-sampler 0755-deliver "$SKILLS_TARBALL" /tmp/golden-skills.tar.gz 0644-deliver "$INSTALL_SKILLS" /tmp/install-skills.sh 0755-deliver "$SETUP_SCRIPT" /tmp/setup-golden.sh 0755--# Run setup-golden.sh in the box. Its `set -euo pipefail` plus this explicit exit-# check are our success gate: a failing step returns non-zero and we never reach-# `box snapshot`. AGENT_VERSION / SKILLS_TARBALL / INSTALL_SKILLS reach it as env.-log "running setup-golden.sh in the box"-# %q-escape AGENT_VERSION before splicing it into the remote command string ---# it comes from an npm registry response (see the workflow's "Resolve-# published agent version" step), and unescaped single-quote interpolation-# would let a value containing a quote break out of the assignment and run-# arbitrary commands in the box.-printf -v agent_version_escaped '%q' "$AGENT_VERSION"-setup_rc=0-# The ssh user is `hog`; setup-golden.sh installs apt packages, writes-# /etc/environment and the systemd drop-in, and restarts hogpanion — all root-# work — so run it through passwordless sudo. `sudo env VAR=...` sets the child-# env explicitly rather than relying on sudo's implied SETENV for leading-# assignments (which a stricter sudoers policy would strip).-"${ssh_base[@]}" \- "sudo env AGENT_VERSION=$agent_version_escaped SKILLS_TARBALL=/tmp/golden-skills.tar.gz INSTALL_SKILLS=/tmp/install-skills.sh bash /tmp/setup-golden.sh" \- || setup_rc=$?-if [[ "$setup_rc" -ne 0 ]]; then- log "FAIL: setup-golden.sh exited $setup_rc in the box; not snapshotting"- exit 1-fi-log "setup-golden.sh completed in the box"--# Snapshot the seed box, then point the candidate alias at the new snapshot id.-# `box snapshot` pauses the box and persists it to S3, emitting the record JSON.-log "snapshotting seed box $SEED_BOX_ID"-snap_json=$(hogland box snapshot "$SEED_BOX_ID")-SNAP_ID=$(printf '%s' "$snap_json" | jq -r '.id // empty')-if [[ -z "$SNAP_ID" ]]; then- log "FAIL: box snapshot returned no id"- printf '%s\n' "$snap_json" >&2- exit 1-fi-log "pointing alias $ALIAS at snapshot $SNAP_ID"-hogland snapshot alias "$SNAP_ID" "$ALIAS"--log "bake complete: $SNAP_ID -> alias $ALIAS"-printf 'BAKED_SNAPSHOT_ID=%s\n' "$SNAP_ID"diff --git a/products/tasks/backend/sandbox/images/hogland/setup-golden.sh b/products/tasks/backend/sandbox/images/hogland/setup-golden.shdeleted file mode 100755index 4c3259779cf2..000000000000--- a/products/tasks/backend/sandbox/images/hogland/setup-golden.sh+++ /dev/null@@ -1,454 +0,0 @@-#!/usr/bin/env bash-# First-boot setup for the PostHog *tasks* golden snapshot.-#-# `bake-golden.sh` runs this once as root inside a fresh seed box (over SSH),-# then `box snapshot`s the result and points an alias at it.-# `HoglandSandbox.create` restores every task sandbox from that alias, so a-# rebuild here is the hogland equivalent of publishing a new base image tag.-#-# The steps here reconstruct products/tasks/backend/sandbox/images/Dockerfile.sandbox-base-# over exec. That Dockerfile stays the source of truth for what a task sandbox-# contains; keep the pins below in sync with its ARGs.-#-# Content delivery (the crux): the CI runner renders the skills itself with a-# database (the same `hogli build:skills` mechanism the Modal image build uses)-# and merges in the context-mill skills, then `bake-golden.sh` delivers the whole-# set into the box over SSH. Nothing is fished out of a published image. Content-# arrives two ways, both from `bake-golden.sh` before this script runs:-# * small, fixed posthog-owned files (the git/gh guards, the cpu sampler) are-# streamed to their target paths with mode 0755; this script only verifies-# them (`test -x`), so they are ready before the checks below.-# * the rendered skills tarball (SKILLS_TARBALL) and the shared installer-# (INSTALL_SKILLS, the same install-skills.sh the image uses) are streamed in-# too. This script extracts the tarball and runs the installer, so the golden-# lands the SAME merged, rendered skills at the SAME paths the image uses.-# `@posthog/agent` is npm-installed here at AGENT_VERSION (default latest; the-# workflow resolves and pins an exact version), decoupled from any image.-#-# Success detection is the caller's: `bake-golden.sh` runs this over SSH and-# checks its exit code before snapshotting. `set -e` below makes a failing step-# abort with non-zero, so a broken bake never reaches `box snapshot`.-#-# Required env (set by bake-golden.sh over SSH):-# SKILLS_TARBALL — path in the box to the rendered-skills tarball (.tar.gz)-# INSTALL_SKILLS — path in the box to install-skills.sh-# AGENT_VERSION — @posthog/agent version to install (default: latest)--set -euo pipefail-export DEBIAN_FRONTEND=noninteractive--: "${SKILLS_TARBALL:?SKILLS_TARBALL is required (path in-box to the rendered-skills tarball)}"-: "${INSTALL_SKILLS:?INSTALL_SKILLS is required (path in-box to install-skills.sh)}"-AGENT_VERSION="${AGENT_VERSION:-latest}"--# Free-RAM scrub (see scrub_free_ram): leave this much RAM untouched so the scrub-# never races the OOM killer. Tunable for a differently-sized seed box.-SCRUB_HEADROOM_MIB="${SCRUB_HEADROOM_MIB:-2048}"-# Off by default: dropping reclaimable page cache would let the scrub zero even-# more frames, but it evicts the warm tool-binary cache that keeps restores fast.-# Set to 1 only if a smaller image matters more than restore warmth.-SCRUB_DROP_PAGECACHE="${SCRUB_DROP_PAGECACHE:-0}"--# --- Version pins, mirrored from Dockerfile.sandbox-base ARGs ------------------GIT_VERSION=2.49.1-GIT_SHA256=310831de967f1c8c5e8ff55f92807dea89f83dc3d3d2a5d16c209bd01a31def1-UV_VERSION=0.12.13-RUFF_VERSION=0.14.11-TY_VERSION=0.0.29-GH_CLI_VERSION=2.97.0-AGENTSH_TAG=v0.18.3-AGENTSH_SHA256_AMD64=4ac486eea1e10600c29078a7a992d2067774edfb66be1318a2acf1fcf8b6d774-AGENTSH_SHA256_ARM64=d1393a27943d207442ea077b1d36d9561a8af5613e7b67d9f7d4fafd00626c6b-RTK_VERSION=0.43.0-RTK_SHA256_AMD64=ff8a1e7766496e175291a85aeca1dc97c9ff6df33e51e5893d1fbc78fea2a609-RTK_SHA256_ARM64=5519f7ca12e5c143a609f0d28a0a77b97413a8dce31c2681f1a41c24519a8731--APT_PACKAGES="curl wget git vim nano tree htop unzip zip jq \-build-essential pkg-config musl \-python3 python3-pip python3-venv python3-dev \-sqlite3 postgresql-client mysql-client redis-tools \-libssl-dev libcurl4-gnutls-dev libexpat1-dev libffi-dev libbz2-dev \-libreadline-dev libsqlite3-dev libncursesw5-dev xz-utils tk-dev \-libxml2-dev libxmlsec1-dev zlib1g-dev \-ca-certificates gnupg sudo"--# Dockerfile ENVs, made visible to every login/exec process. /etc/environment-# covers PAM sessions; the systemd drop-in covers the box's agent daemon, whose-# services do not read /etc/environment. Per-box create(env=...) values reach-# exec children through the adapter's per-exec env, not this drop-in.-STATIC_ENV_PATH="/opt/posthog/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"--log() { printf '[setup-golden] %s\n' "$*" >&2; }--# --- curl with retries (transient reset must not fail the whole bake) ----------fetch() {- # fetch <url> <out>- curl -fsSL --retry 5 --retry-all-errors --retry-max-time 60 --connect-timeout 10 -o "$2" "$1"-}--# --- Free-RAM scrub: shrink the snapshot's stored/faulted memory image ---------# The hogland snapshot store elides every all-zero 4 MiB chunk: a zero chunk is-# served as a zero page on restore, never stored in S3, never uploaded, never-# faulted. But Linux never zeroes a page it frees, so every frame the guest-# dirtied during this ~90-min bake (apt, npm, freed heap) keeps its stale-# non-zero content forever — mempack stores and uploads it and each restore-# faults it from S3. Writing zeros over free RAM converts those frames to zero,-# so mempack collapses them to SrcZero (not stored, not uploaded, not faulted).-#-# This is a pure optimization, so it fails SOFT: any error only logs a warning-# and returns 0, and the C helper raises its own oom_score_adj so memory pressure-# kills the scrub first, never the bake. Runs LAST, after all other guest writes,-# so no later step re-dirties the frames it just zeroed.-scrub_free_ram() {- command -v cc >/dev/null 2>&1 || { log "WARN: free-RAM scrub skipped: no cc"; return 0; }-- local src bin cc_log- src="$(mktemp --suffix=.c)"- bin="$(mktemp)"- cc_log="$(mktemp)"-- # Self-contained C: mmap N bytes of anonymous memory, write zeros through every- # page, then release it. Anonymous pages fault in zero-filled, so each backing- # frame ends up all-zero and is freed still zeroed at exit. C (no GC/interpreter)- # is the most reliable way to touch every page without leaving its own churn.- cat > "$src" <<'ZEROFREE_C'-#include <fcntl.h>-#include <stdio.h>-#include <stdlib.h>-#include <string.h>-#include <sys/mman.h>-#include <unistd.h>--int main(int argc, char **argv) {- if (argc != 2) {- fprintf(stderr, "usage: %s <bytes>\n", argv[0]);- return 2;- }-- unsigned long long bytes = strtoull(argv[1], NULL, 10);- if (bytes == 0) {- return 0;- }-- /* Die first under memory pressure: never let the OOM killer pick the bake- * or hogpanion over this optional scrub. Best effort — proceed if it fails. */- int fd = open("/proc/self/oom_score_adj", O_WRONLY);- if (fd >= 0) {- ssize_t w = write(fd, "1000\n", 5);- (void)w;- close(fd);- }-- size_t len = (size_t)bytes;- void *p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);- if (p == MAP_FAILED) {- perror("mmap");- return 1;- }-- /* Fault in and zero every page. Anonymous pages fault in zero-filled, so- * this leaves each backing frame all-zero. After we exit, the kernel frees- * them still zeroed — exactly what mempack collapses to SrcZero. */- memset(p, 0, len);-- munmap(p, len);- return 0;-}-ZEROFREE_C-- if ! cc -O2 -o "$bin" "$src" 2>"$cc_log"; then- log "WARN: free-RAM scrub skipped: compile failed: $(tr '\n' ' ' < "$cc_log")"- rm -f "$src" "$bin" "$cc_log"- return 0- fi-- # Optional, off by default: drop clean reclaimable page cache so the scrub can- # zero those frames too. It evicts the warm tool-binary cache, so it is gated.- if [ "$SCRUB_DROP_PAGECACHE" = "1" ] && [ -w /proc/sys/vm/drop_caches ]; then- sync- echo 1 > /proc/sys/vm/drop_caches 2>/dev/null || log "WARN: drop_caches failed; continuing"- log "dropped reclaimable page cache before scrub (SCRUB_DROP_PAGECACHE=1)"- fi-- # MemFree, not MemAvailable: MemAvailable counts reclaimable page cache we want- # to KEEP resident for warm restores. Touching only genuinely-free frames avoids- # evicting that cache. MemFree is in kB.- local memfree_kb headroom_bytes target_bytes- memfree_kb="$(awk '/^MemFree:/ {print $2; exit}' /proc/meminfo)"- if [ -z "$memfree_kb" ]; then- log "WARN: free-RAM scrub skipped: could not read MemFree"- rm -f "$src" "$bin" "$cc_log"- return 0- fi- headroom_bytes=$(( SCRUB_HEADROOM_MIB * 1024 * 1024 ))- target_bytes=$(( memfree_kb * 1024 - headroom_bytes ))- if [ "$target_bytes" -le 0 ]; then- log "free-RAM scrub skipped: MemFree ${memfree_kb} kB is within the ${SCRUB_HEADROOM_MIB} MiB headroom"- rm -f "$src" "$bin" "$cc_log"- return 0- fi-- log "free-RAM scrub: zeroing ${target_bytes} bytes (MemFree ${memfree_kb} kB, headroom ${SCRUB_HEADROOM_MIB} MiB)"- if "$bin" "$target_bytes"; then- log "free-RAM scrub done"- else- # A non-zero exit (mmap fail) or an OOM-kill signal lands here. Soft-fail.- log "WARN: free-RAM scrub did not complete cleanly (rc $?); continuing"- fi-- rm -f "$src" "$bin" "$cc_log"- return 0-}--log "apt packages"-apt-get update-# shellcheck disable=SC2086 # word-splitting the package list is intended-apt-get install -y --no-install-recommends $APT_PACKAGES-rm -rf /var/lib/apt/lists/*--log "git ${GIT_VERSION} from source"-fetch "https://www.kernel.org/pub/software/scm/git/git-${GIT_VERSION}.tar.xz" /tmp/git.tar.xz-echo "${GIT_SHA256} /tmp/git.tar.xz" | sha256sum -c --mkdir /tmp/git-tar -xf /tmp/git.tar.xz -C /tmp/git --strip-components=1-make -C /tmp/git prefix=/usr NO_GETTEXT=YesPlease NO_TCLTK=YesPlease -j"$(nproc)" all-make -C /tmp/git prefix=/usr NO_GETTEXT=YesPlease NO_TCLTK=YesPlease install-rm -rf /tmp/git /tmp/git.tar.xz-git --version-# Materialise `git help -a` before matching. Piping it straight into `grep -q`-# lets grep close the pipe on the first match, so git dies with SIGPIPE (141)-# and pipefail aborts the whole setup. Capture first, then grep the string.-git_help_all="$(git help -a)"-grep -q '[[:space:]]backfill' <<<"$git_help_all"--log "node 24"-# The rootfs bakes node 22 into /usr/local/bin, which precedes /usr/bin (where-# nodesource installs 24) on STATIC_ENV_PATH and in sudo's secure_path. Remove-# the baked copy first, or `npm install @posthog/agent` and the runtime resolve-# node 22, not 24.-rm -f /usr/local/bin/node /usr/local/bin/npm /usr/local/bin/npx /usr/local/bin/corepack-curl -fsSL --retry 5 --retry-all-errors --retry-max-time 60 --connect-timeout 10 \- https://deb.nodesource.com/setup_24.x | bash --apt-get install -y --no-install-recommends nodejs-rm -rf /var/lib/apt/lists/*-node_version="$(node -v)"-case "$node_version" in- v24.*) log "node ${node_version}" ;;- *) echo "expected node v24 after nodesource install, got ${node_version}" >&2; exit 1 ;;-esac--log "npm globals"-npm install -g yarn pnpm typescript ts-node nodemon--log "uv ${UV_VERSION} + ruff ${RUFF_VERSION} + ty ${TY_VERSION}"-# The Dockerfile takes uv from its pinned official image; without a container-# runtime we pin the same version's release tarball and verify its published-# checksum.-uv_arch="$(uname -m)"-uv_asset="uv-${uv_arch}-unknown-linux-gnu.tar.gz"-uv_base="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}"-fetch "${uv_base}/${uv_asset}" /tmp/uv.tar.gz-fetch "${uv_base}/${uv_asset}.sha256" /tmp/uv.tar.gz.sha256-(cd /tmp && echo "$(cut -d' ' -f1 uv.tar.gz.sha256) uv.tar.gz" | sha256sum -c -)-tar -xzf /tmp/uv.tar.gz -C /tmp-install -m 755 "/tmp/uv-${uv_arch}-unknown-linux-gnu/uv" "/tmp/uv-${uv_arch}-unknown-linux-gnu/uvx" /usr/local/bin/-rm -rf /tmp/uv.tar.gz /tmp/uv.tar.gz.sha256 "/tmp/uv-${uv_arch}-unknown-linux-gnu"-UV_TOOL_BIN_DIR=/usr/local/bin uv tool install "ruff==${RUFF_VERSION}"-UV_TOOL_BIN_DIR=/usr/local/bin uv tool install "ty==${TY_VERSION}"-ruff --version-ty --version--log "gh CLI ${GH_CLI_VERSION}"-gh_arch="$(dpkg --print-architecture)"-fetch "https://github.com/cli/cli/releases/download/v${GH_CLI_VERSION}/gh_${GH_CLI_VERSION}_linux_${gh_arch}.deb" /tmp/gh.deb-dpkg -i /tmp/gh.deb-rm /tmp/gh.deb--log "agentsh ${AGENTSH_TAG}"-agentsh_version="${AGENTSH_TAG#v}"-agentsh_arch="$(dpkg --print-architecture)"-case "$agentsh_arch" in- amd64) agentsh_sha256="$AGENTSH_SHA256_AMD64" ;;- arm64) agentsh_sha256="$AGENTSH_SHA256_ARM64" ;;- *) echo "Unsupported architecture for agentsh: $agentsh_arch" >&2; exit 1 ;;-esac-fetch "https://github.com/canyonroad/agentsh/releases/download/${AGENTSH_TAG}/agentsh_${agentsh_version}_linux_${agentsh_arch}.deb" /tmp/agentsh.deb-echo "${agentsh_sha256} /tmp/agentsh.deb" | sha256sum -c --dpkg -i /tmp/agentsh.deb-rm /tmp/agentsh.deb-agentsh --version-mkdir -p /var/lib/agentsh/sessions /var/lib/agentsh/quarantine /var/log/agentsh-chmod 777 /var/lib/agentsh /var/lib/agentsh/sessions /var/lib/agentsh/quarantine /var/log/agentsh--log "rtk ${RTK_VERSION}"-rtk_arch="$(dpkg --print-architecture)"-case "$rtk_arch" in- amd64) rtk_asset="rtk-x86_64-unknown-linux-musl.tar.gz"; rtk_sha256="$RTK_SHA256_AMD64" ;;- arm64) rtk_asset="rtk-aarch64-unknown-linux-gnu.tar.gz"; rtk_sha256="$RTK_SHA256_ARM64" ;;- *) echo "Unsupported architecture for rtk: $rtk_arch" >&2; exit 1 ;;-esac-fetch "https://github.com/rtk-ai/rtk/releases/download/v${RTK_VERSION}/${rtk_asset}" /tmp/rtk.tar.gz-echo "${rtk_sha256} /tmp/rtk.tar.gz" | sha256sum -c --tar -xzf /tmp/rtk.tar.gz -C /usr/local/bin rtk-rm /tmp/rtk.tar.gz-rtk --version--# --- Agent SDK: @posthog/agent, npm-installed at AGENT_VERSION -----------------# Mirrors Dockerfile.sandbox-base's `npm install @posthog/agent@${AGENT_VERSION}`-# into /scripts. The workflow resolves and pins an exact published version so the-# golden's agent-server is reproducible; a hand run defaults to latest.-log "@posthog/agent@${AGENT_VERSION} in /scripts"-mkdir -p /scripts-(cd /scripts && npm init -y && npm install "@posthog/agent@${AGENT_VERSION}")-test -x /scripts/node_modules/.bin/agent-server--# --- Skills: install the runner-rendered set from the delivered tarball --------# The CI runner renders the skills with a database (hogli build:skills) and merges-# in the context-mill skills, exactly as the Modal image build does, then delivers-# the result as SKILLS_TARBALL. Extract it and run the SAME install-skills.sh the-# image uses (delivered as INSTALL_SKILLS), so the golden lands the merged, rendered-# skills at the SAME paths: /scripts/plugins/posthog/skills, /root/.agents/skills,-# /root/.claude/skills, plus /scripts/plugins/posthog/plugin.json. install-skills.sh-# keys off $HOME, so pin it to /root (this script runs as root in the box).-log "skills: install runner-rendered set from ${SKILLS_TARBALL}"-test -f "$SKILLS_TARBALL" || { echo "SKILLS_TARBALL ${SKILLS_TARBALL} not found in box" >&2; exit 1; }-test -f "$INSTALL_SKILLS" || { echo "INSTALL_SKILLS ${INSTALL_SKILLS} not found in box" >&2; exit 1; }-skills_extract_dir="$(mktemp -d)"-tar -xzf "$SKILLS_TARBALL" -C "$skills_extract_dir"-HOME=/root bash "$INSTALL_SKILLS" "$skills_extract_dir"-rm -rf "$skills_extract_dir"--# Fail closed: an empty tarball or a broken install must not ship a golden that-# silently lost its skills.-for skills_target in /scripts/plugins/posthog/skills /root/.agents/skills /root/.claude/skills; do- # Capture then test. `find ... | grep -q .` lets grep close the pipe on the- # first hit, so find dies with SIGPIPE (141) and pipefail flips this guard.- found_skill="$(find "$skills_target" -name 'SKILL.md' -type f 2>/dev/null)"- if [ -z "$found_skill" ]; then- echo "no SKILL.md found under ${skills_target} after installing ${SKILLS_TARBALL}" >&2- exit 1- fi-done--log "guards + cpu sampler (delivered over ssh by bake-golden.sh)"-# The git/gh guards and the cpu sampler are streamed to their target paths with-# mode 0755 by bake-golden.sh before this script runs. The final verify below-# fails the bake if a hand run omitted them.-test -x /opt/posthog/bin/git-test -x /opt/posthog/bin/gh-test -x /opt/posthog/bin/hogli-test -x /usr/local/bin/posthog-cpu-billing-sampler--log "git identity"-git config --global user.email "[email protected]"-git config --global user.name "PostHog Desktop"--log "static env (/etc/environment + agent-daemon systemd drop-in)"-mkdir -p /tmp/workspace-cat > /etc/environment <<EOF-DEBIAN_FRONTEND="noninteractive"-TZ="UTC"-GH_TELEMETRY="false"-AGENTSH_SERVER="http://127.0.0.1:18080"-IS_SANDBOX="1"-PYTHONPATH="/tmp/workspace"-PATH="${STATIC_ENV_PATH}"-EOF-# Lay down the agent-daemon drop-in so exec processes inherit the container-style-# env, then restart the daemon so the snapshot captures a hogpanion already-# re-exec'd with the new env.-if [ -d /etc/systemd/system ]; then- dropin_dir=/etc/systemd/system/hogpanion.service.d- mkdir -p "$dropin_dir"- # No EnvironmentFile= here. A per-box /etc/hogbox-env would let an unreserved- # key such as PATH override these guard vars (EnvironmentFile settings win- # over Environment= regardless of order), routing hog-exec children through- # /usr/bin/git instead of the wrapped /opt/posthog/bin/git. Per-box env still- # reaches exec children through the adapter's per-exec env, not this drop-in.- # HOME/USER/LOGNAME: hogpanion runs as root with no User= and hands exec- # children bare os.Environ(), so nothing sets HOME. Skills resolve- # $HOME/.agents/skills and git reads /root/.gitconfig; without HOME a set -u- # task step dies. Pin them to root so exec children inherit a usable home.- cat > "$dropin_dir/posthog-env.conf" <<EOF-[Service]-Environment="DEBIAN_FRONTEND=noninteractive"-Environment="TZ=UTC"-Environment="GH_TELEMETRY=false"-Environment="AGENTSH_SERVER=http://127.0.0.1:18080"-Environment="IS_SANDBOX=1"-Environment="PYTHONPATH=/tmp/workspace"-Environment="PATH=${STATIC_ENV_PATH}"-Environment="HOME=/root"-Environment="USER=root"-Environment="LOGNAME=root"-EOF- systemctl daemon-reload- # daemon-reload does NOT re-exec a running unit, so the drop-in's new- # Environment= only reaches hogpanion on its next restart. Without a restart- # the snapshot freezes the OLD env, and restored task boxes' hog-exec children- # lack IS_SANDBOX=1, the /opt/posthog/bin-first PATH (git/gh guards), and- # PYTHONPATH. This script runs under hogpanion's cgroup, so a direct restart- # would kill it mid-bake; fire the restart from a detached transient unit and- # then poll until the daemon is back and reporting ready before returning.- if systemctl cat hogpanion.service >/dev/null 2>&1; then- old_pid="$(systemctl show hogpanion.service -p MainPID --value 2>/dev/null || echo 0)"- systemd-run --collect --unit=hogpanion-reload --on-active=2 \- systemctl restart hogpanion.service- # Gate on hogpanion reporting READY, not just on a new MainPID. hogpanion- # is Type=simple: the pid changes at execve, but `ready` only flips true- # after runBoot finishes, so a pid-only wait can snapshot a half-booted- # daemon. Require a NEW pid (proves the restart fired, not the old still-- # ready daemon) AND the status endpoint reporting ready:true.- ready=0- for _ in $(seq 1 90); do- new_pid="$(systemctl show hogpanion.service -p MainPID --value 2>/dev/null || echo 0)"- if systemctl is-active --quiet hogpanion.service \- && [ -n "$new_pid" ] && [ "$new_pid" != "0" ] && [ "$new_pid" != "$old_pid" ] \- && curl -sf http://localhost:7682/status 2>/dev/null | jq -e '.ready == true' >/dev/null 2>&1; then- ready=1- break- fi- sleep 1- done- if [ "$ready" != "1" ]; then- echo "hogpanion did not restart and report ready with the env drop-in (old pid ${old_pid})" >&2- systemctl status hogpanion.service --no-pager || true- exit 1- fi- log "hogpanion restarted (pid ${old_pid} -> ${new_pid}) and reported ready with the env drop-in"- else- log "hogpanion.service not present; skipping restart (drop-in applies on next start)"- fi-fi--log "verify"-python3 --version-node --version-npm --version-gh --version-rtk --version-agentsh --version-test -x /scripts/node_modules/.bin/agent-server-test -x /opt/posthog/bin/git-test -x /opt/posthog/bin/gh-test -x /opt/posthog/bin/hogli--# LAST step: strip the ephemeral CI ssh key so it does not persist into the-# snapshot and therefore into every restored task box. Production restores with-# access_type: none, and hogpanion returns early on an empty key list, so a-# non-empty authorized_keys from the bake would survive into live boxes. Truncate-# every user's authorized_keys the bake could have populated. This is the final-# action: no SSH step runs after it (the current session stays open; box snapshot-# is driven over the hogplane API, not SSH).-for ak in /home/hog/.ssh/authorized_keys /root/.ssh/authorized_keys; do- [ -f "$ak" ] && : >"$ak"-done-log "truncated CI ssh authorized_keys"--# Absolute last guest write before bake-golden.sh calls `box snapshot` (over the-# hogplane API, not SSH — no further guest activity runs after this returns).-# Zeroing free RAM here, after every other step, means no later step re-dirties-# the frames it just zeroed. See scrub_free_ram for why (zero-page collapse).-scrub_free_ram--log "setup-golden complete"diff --git a/products/tasks/backend/sandbox/images/hogland/smoke-golden.sh b/products/tasks/backend/sandbox/images/hogland/smoke-golden.shdeleted file mode 100755index 542027af52ed..000000000000--- a/products/tasks/backend/sandbox/images/hogland/smoke-golden.sh+++ /dev/null@@ -1,281 +0,0 @@-#!/usr/bin/env bash-# smoke-golden.sh — readiness gate for the tasks golden-snapshot rebuild.-#-# Boots a smoke box from the posthog-tasks-candidate alias, runs the task-# sandbox's readiness contract over SSH, and deletes the box. A non-zero exit-# tells the workflow to skip the promote step, which leaves the live-# posthog-tasks-default alias pointing at the previous known-good snapshot.-#-# The contract asserts the things a task run depends on:-# a. AGENT-SERVER — @posthog/agent's agent-server binary is present and starts-# (a golden that baked a broken /scripts install must not promote).-# b. CLONE + EXEC — a trivial `git clone` of a public repo and a command run-# inside it work, i.e. the toolchain (git guard, network, exec) is live.-# c. EXEC-DAEMON ENV — the running hogpanion daemon (whose env hog-exec children-# inherit) carries the container-style env: IS_SANDBOX=1, a PATH with-# /opt/posthog/bin first (the git/gh guards), and PYTHONPATH. This reads the-# daemon's live /proc environ, NOT an SSH login shell — a login shell reads-# /etc/environment via PAM and would look correct even if the daemon env-# never picked up the drop-in, certifying a broken golden green.-#-# Required env vars (set by the workflow):-# ALIAS — the posthog-tasks-candidate alias to boot from-# SSH_KEY — path to the PRIVATE key; the public half is at ${SSH_KEY}.pub-#-# On success prints a single parseable line to stdout:-# SMOKE_HEALTHY_SECONDS=<n> (create→contract-passed wall-clock)--set -euo pipefail--: "${ALIAS:?ALIAS is required}"-: "${SSH_KEY:?SSH_KEY is required (private key path; pub at \${SSH_KEY}.pub)}"--log() { printf '[tasks-smoke] %s\n' "$*" >&2; }--# Fixed name so a killed prior run's leftover box is found and cleaned rather-# than leaking a node or colliding on the next create (hogplane 409s on a-# duplicate name).-SMOKE_BOX_NAME=golden-smoke-tasks-SMOKE_BOX_ID=""--# List across ALL kinds, not just ours: hogplane's name-uniqueness check-# (ensureBoxNameUnique) scans every kind, so match by name only.-resolve_smoke_box_id() {- hogland box list 2>/dev/null \- | jq -r --arg n "$SMOKE_BOX_NAME" 'map(select(.spec.name == $n)) | .[0].id // empty' 2>/dev/null \- || true-}--cleanup() {- if [[ -z "$SMOKE_BOX_ID" ]]; then- SMOKE_BOX_ID=$(resolve_smoke_box_id)- fi- if [[ -n "$SMOKE_BOX_ID" ]]; then- log "deleting smoke box $SMOKE_BOX_ID"- hogland box delete "$SMOKE_BOX_ID" >/dev/null 2>&1 || true- fi-}-trap cleanup EXIT--# Defensive pre-clean: a previous killed run may have leaked a box under our-# fixed name, and every subsequent create would 409 on it. Wait for the delete-# to land (it is async) before creating.-stale_id=$(resolve_smoke_box_id)-if [[ -n "$stale_id" ]]; then- log "pre-cleaning stale smoke box $stale_id (name $SMOKE_BOX_NAME)"- hogland box delete "$stale_id" >/dev/null 2>&1 || true- gone_deadline=$(( $(date +%s) + 120 ))- while [[ -n "$(resolve_smoke_box_id)" ]]; do- if [[ "$(date +%s)" -ge "$gone_deadline" ]]; then- log "FAIL: stale smoke box $stale_id still present 2m after delete"- exit 1- fi- sleep 5- done- log "stale smoke box gone"-fi--log "creating smoke box from alias:$ALIAS"-create_t0=$(date +%s)-# Don't let `set -e` abort on a failed create: the box may still exist (a create-# that timed out waiting for ready left one behind), and the trap needs to run.-# --name gives teardown a handle that survives never learning the id. No sizing-# flags — restore inherits cpus/mem/disk from the snapshot.-create_rc=0-# --kind posthog-tasks-golden: an UNREGISTERED kind carries no server-side idle-# TTL, so an all-SSH smoke run is never reaped mid-check. The workflow's-# always() teardown deletes this box by name on cancel. --access-type-# ssh-private: keep the box off the public internet; the runner reaches it over-# the tailnet. --disk-mbps 0 --disk-iops 0: the bake baked the snapshot with-# uncapped disk, and production restores inherit that — but the CLI applies its-# own 125MB/s / 3000-IOPS defaults on restore rather than inheriting, so pass 0-# here too or the smoke measures a throttle production never sees.-# --timeout 30m: a 64 GiB restore may wait on Karpenter for a node.-box_json=$(- hogland box create \- --snapshot-id "alias:$ALIAS" \- --ssh-key "${SSH_KEY}.pub" \- --name "$SMOKE_BOX_NAME" \- --kind posthog-tasks-golden \- --access-type ssh-private \- --disk-mbps 0 \- --disk-iops 0 \- --no-connect \- --timeout 30m-) || create_rc=$?-SMOKE_BOX_ID=$(printf '%s' "$box_json" | jq -r '.id // empty' 2>/dev/null || true)-if [[ "$create_rc" -ne 0 ]]; then- log "FAIL: box create exited $create_rc"- printf '%s\n' "$box_json" >&2- exit 1-fi-ssh_cmd=$(printf '%s' "$box_json" | jq -r '.ssh_command // empty')-if [[ -z "$ssh_cmd" ]]; then- log "box create returned no ssh_command; cannot reach box for smoke"- printf '%s\n' "$box_json" >&2- exit 1-fi-log "smoke box $SMOKE_BOX_ID ssh_command: $ssh_cmd"--# Rebuild the ssh invocation from the returned command, injecting our key +-# non-interactive options right after the `ssh` word. IdentitiesOnly pins the-# probe to our ephemeral key (a stray agent identity would burn MaxAuthTries-# first); BatchMode never prompts; accept-new trusts the fresh host key silently.-read -r -a ssh_parts <<<"$ssh_cmd"-# ServerAlive* turns a wedged binary or dead session on the box into a dropped-# connection in ~45s, instead of hanging an assert until the job's 150m ceiling.-ssh_opts=(-i "$SSH_KEY" -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=3)-ssh_base=("${ssh_parts[0]}" "${ssh_opts[@]}" "${ssh_parts[@]:1}")--# Run one SSH assertion under a hard local ceiling, so neither a hung binary nor-# a stuck TCP session can wedge the job. Usage: ssh_assert <remote-command...>-ssh_assert() {- timeout 60 "${ssh_base[@]}" "$@"-}--log "waiting for ssh reachability (up to 5m)"-ssh_deadline=$(( $(date +%s) + 300 ))-until timeout 20 "${ssh_base[@]}" -o ConnectTimeout=5 true 2>/dev/null; do- if [[ "$(date +%s)" -ge "$ssh_deadline" ]]; then- log "ssh never became reachable within 5m"- exit 1- fi- sleep 5-done-log "ssh reachable"--# (a) agent-server: present and able to report its version. --help/--version-# exits cleanly without needing task credentials or a running control plane.-log "asserting agent-server present and starts"-if ! ssh_assert "test -x /scripts/node_modules/.bin/agent-server"; then- log "FAIL: /scripts/node_modules/.bin/agent-server missing or not executable"- exit 1-fi-if ! ssh_assert "/scripts/node_modules/.bin/agent-server --version >/dev/null 2>&1 || /scripts/node_modules/.bin/agent-server --help >/dev/null 2>&1"; then- log "FAIL: agent-server did not start (neither --version nor --help succeeded)"- exit 1-fi--# (b) clone + exec: the git guard passes through a real clone, network egress-# works, and a command runs inside the checkout. POSTHOG_ALLOW_UNSIGNED_GIT is-# irrelevant here (clone is never blocked); this exercises the everyday path.-log "asserting trivial clone + exec"-if ! ssh_assert "set -e; d=\$(mktemp -d); git clone --depth 1 https://github.com/octocat/Hello-World \"\$d/repo\"; test -f \"\$d/repo/README\"; ( cd \"\$d/repo\" && git rev-parse HEAD ); rm -rf \"\$d\""; then- log "FAIL: trivial clone/exec did not complete"- exit 1-fi--# (c) exec-daemon env: the golden must promote only if hogpanion — the daemon-# whose env hog-exec children inherit — is running with the container-style env-# from setup-golden.sh's drop-in. We read the DAEMON's live /proc environ, not a-# login shell: PAM feeds an SSH shell /etc/environment, so a shell would look-# correct even when the daemon never re-exec'd with the drop-in (the exact bug a-# missing hogpanion restart causes). Reading another process's /proc/<pid>/environ-# needs root; the box's ssh user is `hog`, which has passwordless sudo, so the-# environ read goes through sudo. `systemctl show` of the unit config is the-# secondary, weaker signal (config loaded, not necessarily applied to the live-# process). Reaching a real hog-exec child through hogpanion's exec API is the-# ideal upgrade — see GOLDEN_CI_RUNBOOK.md's known-gaps.-log "asserting exec-daemon (hogpanion) env carries the container-style env"-# shellcheck disable=SC2016 # $pid/$environ must expand on the box, not locally-daemon_env_probe='set -eu-# Query via sudo. The box runs no system D-Bus daemon, only the systemd-# root-only private socket, so a non-root systemctl fails with "Failed to-# connect to bus" and reports a false MainPID=0. The ssh user hog has-# passwordless sudo, same as the environ read below.-pid=$(sudo systemctl show hogpanion.service -p MainPID --value 2>/dev/null || echo 0)-if [ -z "$pid" ] || [ "$pid" = "0" ]; then- echo "hogpanion has no running MainPID" >&2- echo "--- hogpanion.service status ---" >&2- sudo systemctl status hogpanion.service --no-pager >&2 2>&1 || true- echo "--- hogpanion journal (tail) ---" >&2- sudo journalctl -u hogpanion.service --no-pager 2>&1 | tail -40 >&2 || true- exit 1-fi-environ=$(sudo cat "/proc/$pid/environ" | tr "\0" "\n")-printf "%s\n" "$environ" | grep -qx "IS_SANDBOX=1" || { echo "daemon env missing IS_SANDBOX=1" >&2; exit 1; }-printf "%s\n" "$environ" | grep -q "^PATH=/opt/posthog/bin:" || { echo "daemon PATH does not start with /opt/posthog/bin" >&2; exit 1; }-printf "%s\n" "$environ" | grep -q "^PYTHONPATH=" || { echo "daemon env missing PYTHONPATH" >&2; exit 1; }-# HOME must be set: hogpanion hands exec children bare os.Environ(), and skills-# resolve $HOME/.agents/skills while git reads /root/.gitconfig. Without HOME a-# set -u task step dies. The drop-in sets HOME=/root.-printf "%s\n" "$environ" | grep -qx "HOME=/root" || { echo "daemon env missing HOME=/root" >&2; exit 1; }-# The /opt/posthog/bin-first PATH is what makes git resolve to the guard; confirm-# the guard is actually there so the guarded path the daemon exposes is real.-test -x /opt/posthog/bin/git || { echo "/opt/posthog/bin/git missing" >&2; exit 1; }-grep -q POSTHOG_ALLOW_UNSIGNED_GIT /opt/posthog/bin/git || { echo "/opt/posthog/bin/git is not the git guard" >&2; exit 1; }-sudo systemctl show hogpanion.service -p Environment -p EnvironmentFiles --no-pager >&2'-if ! ssh_assert "$daemon_env_probe"; then- log "FAIL: hogpanion daemon env is not the container-style env (restart likely did not take before snapshot)"- exit 1-fi--# (d) exec API: production reaches a task box only through hogpanion's exec API-# (POST /v1/hogboxes/{id}/exec), never SSH. Assert a trivial command round-trips-# through it so a golden that works over SSH but not via exec is caught. HOG_HOST-# + HOG_TOKEN_COMMAND come from the workflow's bake-job env. The request body is-# {argv, timeout_seconds} — the schema requires `argv` (not `command`). Fail OPEN-# only when the endpoint is genuinely unreachable (000) — a 404 is what a missing-# box or a non-owner gets, i.e. a real regression that must not pass green.-if [[ -n "${HOG_HOST:-}" && -n "${HOG_TOKEN_COMMAND:-}" ]]; then- log "asserting box reachable via hogpanion exec API"- exec_token=$(eval "$HOG_TOKEN_COMMAND" 2>/dev/null || true)- if [[ -z "$exec_token" ]]; then- log "WARN: could not mint bearer for the exec assertion; relying on SSH assertions"- else- exec_out=$(mktemp)- exec_body='{"argv":["/bin/sh","-c","echo golden-exec-ok"],"timeout_seconds":30}'- # hogplane matches the auth scheme case-sensitively — "Bearer" (capital- # B), unlike the lowercase "bearer" GitHub's OIDC token endpoint wants in- # HOG_TOKEN_COMMAND. A lowercase scheme here gets a 401, which the case- # below treats as a hard failure.- # curl already prints "000" via -w on a connection failure, so append- # nothing on its non-zero exit — `|| echo 000` would concatenate to- # "000000" and never match the 000) arm below. --max-time caps a hung- # request so a wedged endpoint can't stall the job. -w also prints- # num_connects so a 000 can be routed by whether a connection was ever- # made: 0 => the endpoint was unreachable (a tailnet/infra blip, not the- # golden's fault) => fail open and rely on the SSH assertions; >=1 with a- # 000 => we connected but the request hung or was reset (a --max-time- # stall) => a real failure. So a box that stalls the exec cannot promote- # itself by forcing a fail-open.- read -r http_code num_connects <<<"$(curl -sS --connect-timeout 10 --max-time 60 \- -o "$exec_out" -w '%{http_code} %{num_connects}' \- -X POST "$HOG_HOST/v1/hogboxes/$SMOKE_BOX_ID/exec" \- -H "Authorization: Bearer $exec_token" \- -H 'Content-Type: application/json' \- -d "$exec_body" 2>/dev/null || true)"- case "$http_code" in- 2*)- if grep -q 'golden-exec-ok' "$exec_out"; then- log "exec API round-trip ok"- else- log "FAIL: exec API returned $http_code but its output lacked the marker"- cat "$exec_out" >&2 || true- rm -f "$exec_out"- exit 1- fi- ;;- 000)- if [[ "${num_connects:-0}" != "0" ]]; then- log "FAIL: exec API connected but the request did not complete (num_connects=${num_connects}) — a stall or reset, not fail-open"- cat "$exec_out" >&2 || true- rm -f "$exec_out"- exit 1- fi- log "WARN: exec API unreachable (never connected); relying on SSH assertions. Enable once the exec contract is confirmed (see runbook)."- ;;- *)- log "FAIL: exec API POST returned HTTP $http_code"- cat "$exec_out" >&2 || true- rm -f "$exec_out"- exit 1- ;;- esac- rm -f "$exec_out"- fi-fi--elapsed=$(( $(date +%s) - create_t0 ))-log "tasks golden contract passed after ${elapsed}s"-printf 'SMOKE_HEALTHY_SECONDS=%d\n' "$elapsed"