Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

ci(tasks): remove the tasks golden snapshot workflow (#121348)

Georges-Antoine Assi5fc954fe105c
diff --git a/.github/auto-assign-labels.json b/.github/auto-assign-labels.json
index e368131b6f38..588e70255139 100644
--- a/.github/auto-assign-labels.json
+++ b/.github/auto-assign-labels.json
@@ -31,7 +31,6 @@
".agents/skills/finding-llm-gateway-migration-candidates/",
".agents/skills/migrating-llm-gateway-callers/",
".github/workflows/cd-agent-proxy-image.yml",
- ".github/workflows/cd-tasks-golden-snapshot.yml",
".github/workflows/ci-agent-proxy.yml",
".github/workflows/ci-llm-gateway.yml",
".github/workflows/llm-gateway-cd.yml",
diff --git a/.github/owners.yaml b/.github/owners.yaml
index 74ecaa35be63..cb69fdd643d4 100644
--- a/.github/owners.yaml
+++ b/.github/owners.yaml
@@ -33,6 +33,5 @@ rules:
- match:
- '/workflows/ci-agent-proxy.yml'
- '/workflows/cd-agent-proxy-image.yml'
- - '/workflows/cd-tasks-golden-snapshot.yml'
owners: team-agent-infrastructure
sensitive: true
diff --git a/.github/workflows/cd-tasks-golden-snapshot.yml b/.github/workflows/cd-tasks-golden-snapshot.yml
deleted file mode 100644
index 7e0bc84e7d83..000000000000
--- a/.github/workflows/cd-tasks-golden-snapshot.yml
+++ /dev/null
@@ -1,748 +0,0 @@
-# Bake the PostHog *tasks* golden snapshot and promote it into the alias that
-# HoglandSandbox.create restores every task sandbox from.
-#
-# HoglandSandbox.create (products/tasks/backend/logic/services/hogland_sandbox.py)
-# restores every task sandbox from the alias this workflow promotes. Instead of an
-# engineer running the bake from a laptop, a runner renders the agent skills,
-# joins the hogland tailnet, builds the seed box from box primitives + a delivered
-# payload, smoke-tests the candidate, and only then re-points the live alias —
-# with a dated archive alias for rollback.
-#
-# Skills delivery (the crux): the golden must ship the SAME merged, rendered skills
-# the Modal image ships, but WITHOUT depending on the sandbox-base image. So a
-# `render_skills` job renders them itself with a database — the same
-# `hogli build:skills` mechanism the image build uses — and merges in the
-# context-mill skills, exactly as cd-sandbox-base-image.yml's build_skills job does.
-# The bake job streams that rendered set into the seed box over the box's own SSH
-# (see bake-golden.sh). No GHCR image, no public artifact host.
-#
-# Modal / Dockerfile.sandbox-base is deliberately untouched: this bakes the same
-# contents into a hogland snapshot, it does not change the container image path.
-#
-# Reachability + auth (mirrors hogbox-preview-env.yml, which already reaches
-# hogland from this repo):
-# * both hogland APIs are tailnet-only, served by tag:hogplane devices. We join
-# as tag:hogland-ci — the tag the ACL grants reach to those devices, and the
-# same join hogbox-preview-env.yml uses. One join covers dev and prod-us.
-# * hogplane authenticates the caller by bearer: GitHub OIDC end-to-end, no
-# stored token. It verifies the Actions OIDC token and resolves the
-# (repo, workflow) claim to a service-account Principal via a github_oidc
-# TrustMapping. Those tokens live ~5 min while a bake runs ~90, so the CLI
-# re-mints on any mid-bake 401 via HOG_TOKEN_COMMAND — a single mint-at-start
-# bearer would die mid-bake.
-#
-# Gating: this stays a deliberate no-op until an operator provisions the per-
-# cluster svc-ci principal + github_oidc TrustMapping + tailnet admittance and
-# flips the arming vars (see GOLDEN_CI_RUNBOOK.md). workflow_dispatch is the only
-# real trigger; the nightly schedule fires but the job's `if` keeps it a no-op
-# until armed, so merging this PR cannot attempt a bake before the principal
-# exists.
-
-name: Tasks Golden Snapshot CD
-
-on:
- workflow_dispatch:
- inputs:
- cluster:
- description: 'Only bake this cluster (blank = all armed clusters)'
- required: false
- default: ''
- ref:
- description: "PostHog ref/SHA to render the skills from (blank = this workflow's checkout ref, i.e. master on schedule)"
- required: false
- default: ''
- box_mem_mib:
- description: 'Total/cap memory (MiB) for the golden. Blank = 20480 (fixed-size default).'
- required: false
- default: ''
- box_mem_boot_mib:
- description: 'Boot memory (MiB) for a pluggable-memory golden — boots this small and hot-adds up to the cap. Blank = fixed-size golden (today).'
- required: false
- default: ''
- target_alias:
- description: 'Live alias to promote the baked snapshot into. Blank = posthog-tasks-default (the golden every task box uses). Set posthog-tasks-hotplug to publish a pluggable-memory golden alongside the default, selected per team by the tasks-hogland-hotplug-golden flag.'
- required: false
- default: ''
- schedule:
- # Daily, minute offset off :00 — GitHub's scheduler is contended at the
- # top of the hour and may drop those runs. No-op until armed (job `if`).
- - cron: '17 8 * * *' # ~08:17 UTC, low-traffic window
-
-permissions:
- contents: read
- id-token: write # mint OIDC for the tailnet join AND the hogplane bearer
-
-# Never two bakes racing the same alias. Per-cluster group keys would need
-# ${{ matrix.cluster }}, unavailable at workflow level, so serialise the whole
-# workflow and rely on fail-fast=false to keep one cluster's flake from killing
-# the other within a run.
-concurrency:
- group: cd-tasks-golden-snapshot
- cancel-in-progress: false
-
-jobs:
- # Resolve and validate the workflow_dispatch `ref` input in an UNPRIVILEGED
- # job, before any credentialed job checks that tree out. A blank ref resolves
- # to the literal `master` branch. A non-blank value must be a full 40-char
- # commit SHA that GitHub's compare API confirms is already an ancestor of
- # master — anything else fails here. This stops a collaborator pointing `ref`
- # at a branch that carries a modified bake-golden.sh / setup-golden.sh and
- # having it run in the credentialed bake job (hogland CLI App key, tailnet
- # join, hogland/prod OIDC). render_skills and bake check out
- # needs.validate_ref.outputs.ref, never the raw input. The
- # `github.ref == 'refs/heads/master'` gate on every privileged job below is
- # defense in depth (a branch copy can strip it); the durable fix is hogland
- # pinning `ref` in the github_oidc TrustMapping — see GOLDEN_CI_RUNBOOK.md.
- validate_ref:
- if: github.ref == 'refs/heads/master'
- name: Validate the requested ref
- runs-on: ubuntu-24.04
- timeout-minutes: 5
- permissions:
- contents: read
- outputs:
- ref: ${{ steps.resolve.outputs.ref }}
- steps:
- - name: Resolve and validate ref
- id: resolve
- env:
- REQUESTED_REF: ${{ github.event.inputs.ref }}
- GH_TOKEN: ${{ github.token }}
- run: |
- set -euo pipefail
- if [[ -z "$REQUESTED_REF" ]]; then
- echo "ref=master" >> "$GITHUB_OUTPUT"
- echo "::notice::No ref supplied; baking from the master branch"
- exit 0
- fi
- if [[ ! "$REQUESTED_REF" =~ ^[0-9a-f]{40}$ ]]; then
- echo "::error::ref must be a full 40-character commit SHA (got '$REQUESTED_REF')"
- exit 1
- fi
- # compare master...<ref>: ahead_by counts commits in <ref> that
- # master does not contain. Zero => <ref> is an ancestor of (or
- # identical to) master. A non-existent SHA 404s and fails here.
- ahead_by=$(gh api "repos/${GITHUB_REPOSITORY}/compare/master...${REQUESTED_REF}" --jq '.ahead_by')
- if [[ "$ahead_by" != "0" ]]; then
- echo "::error::ref ${REQUESTED_REF} is not an ancestor of master (ahead_by=${ahead_by}); only merged commits may be baked"
- exit 1
- fi
- echo "ref=${REQUESTED_REF}" >> "$GITHUB_OUTPUT"
- echo "::notice::ref ${REQUESTED_REF} confirmed an ancestor of master"
-
- # Render the agent skills the golden ships, the same way the sandbox-base image
- # build does (cd-sandbox-base-image.yml's build_skills job): stand up a DB,
- # uv sync, migrate, `hogli build:skills` to render the .md.j2 templates, then
- # merge in the context-mill skills. Upload the merged set as an artifact the
- # bake job delivers into the box. Rendered once here, reused by every cluster.
- #
- # Shares the master arm gate so an unarmed nightly does not pay for a full DB
- # render; when it skips, the bake job (needs) skips too, keeping the no-op.
- render_skills:
- needs: validate_ref
- if: vars.CD_DEPLOY_ENABLED == 'true' && vars.HOG_TASKS_GOLDEN_ENABLED == 'true' && github.ref == 'refs/heads/master'
- name: Render agent skills from source
- runs-on: ubuntu-24.04
- # `Run migrations` needs a database: 24 skill templates call
- # `render_hogql_example`, which reads `Team.objects.first()`. The schema
- # cache below makes migrate a top-up, but a cache miss is a 22-29 minute
- # cold migrate (see cd-sandbox-base-image.yml), so budget an hour for
- # ample headroom over that worst case.
- timeout-minutes: 60
- permissions:
- contents: read
-
- steps:
- - name: Check out posthog
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- with:
- ref: ${{ needs.validate_ref.outputs.ref }}
- # Deep + blob-less: enough history for the schema cache merge-base key.
- fetch-depth: 1000
- filter: blob:none
-
- - name: Fetch base branch for merge-base computation
- if: github.event_name == 'pull_request'
- env:
- BASE_REF: ${{ github.event.pull_request.base.ref }}
- # Scoped, blob-less, no-tags: without an explicit refspec, git fetch
- # falls back to remote.origin.fetch and pulls every branch.
- run: git fetch --no-tags --depth=1000 --filter=blob:none origin "$BASE_REF:refs/remotes/origin/$BASE_REF"
-
- - name: Compute schema cache keys
- id: schema-key
- # Restores the schema dump ci-backend saves on master pushes, so the
- # migrate step below only tops up what is newer instead of replaying the
- # full migration history. On a PR the migration-set key is the merge-base's;
- # on a push or dispatch it is HEAD's. A cache miss falls through to a full
- # migrate, so an absent or wrong key is never a correctness risk.
- env:
- BASE_REF: ${{ github.event.pull_request.base.ref }}
- IS_PR: ${{ github.event_name == 'pull_request' }}
- # Manual cache-bust knob. Keep in sync with SCHEMA_CACHE_EPOCH in
- # ci-backend.yml, which saves these caches on master pushes.
- SCHEMA_CACHE_EPOCH: v2
- run: |
- if [ "$IS_PR" != "true" ]; then
- # Pushed or dispatched commit is checked out directly: no merge
- # commit, no base ref. The per-SHA key would point at this same
- # push's cache (saved concurrently by ci-backend, so racy); rely
- # on the stable migration-set key below.
- REF=HEAD
- else
- # HEAD is the synthetic merge commit; HEAD^2 is the PR branch tip.
- MERGE_BASE=$(git merge-base HEAD^2 "origin/${BASE_REF}" 2>/dev/null || echo "")
- if [ -z "$MERGE_BASE" ]; then
- echo "key=" >> $GITHUB_OUTPUT
- echo "migrations_key=" >> $GITHUB_OUTPUT
- echo "::notice::merge-base not found (branch too stale?), so the schema cache is skipped"
- exit 0
- fi
- # Routing decides which app labels reach the dump, so an edited routing
- # config matches no master dump: drop both keys and migrate from scratch.
- DB_ROUTING_BASE=$(git ls-tree -r --format='%(objectname) %(path)' "$MERGE_BASE" -- products/db_routing.yaml posthog/product_db_config.py posthog/product_db_router.py)
- DB_ROUTING_HEAD=$(git ls-tree -r --format='%(objectname) %(path)' HEAD -- products/db_routing.yaml posthog/product_db_config.py posthog/product_db_router.py)
- if [ "$DB_ROUTING_BASE" != "$DB_ROUTING_HEAD" ]; then
- echo "key=" >> $GITHUB_OUTPUT
- echo "migrations_key=" >> $GITHUB_OUTPUT
- echo "::notice::db routing config or router differs from the merge base, so the schema cache is skipped"
- exit 0
- fi
- echo "key=posthog-schema-master-${MERGE_BASE}" >> $GITHUB_OUTPUT
- REF="$MERGE_BASE"
- fi
- # Last-resort restore prefix: matches the newest saved dump when the exact
- # keys miss. Gated on checkout age because a re-run of an old commit could
- # restore a dump newer than its code, which forward-only migrate cannot repair.
- HEAD_AGE_SECONDS=$(( $(date +%s) - $(git show -s --format=%ct HEAD) ))
- if [ "$HEAD_AGE_SECONDS" -lt 86400 ]; then
- echo "prefix_key=posthog-schema-mig-${SCHEMA_CACHE_EPOCH}-" >> $GITHUB_OUTPUT
- fi
- # Must match ci-backend.yml's save-side computation byte for byte or the key never hits.
- MIG_FILES=$(git ls-tree -r --format='%(objectname) %(path)' "$REF" \
- | grep -E '/migrations/[^/]+\.py$|^[0-9a-f]+ uv\.lock$' \
- | grep -vE '/(clickhouse|async_migrations)/migrations/' \
- | LC_ALL=C sort) || true
- # Routing decides which app labels are in the dump, so these inputs belong in the key.
- DB_ROUTING=$(git ls-tree -r --format='%(objectname) %(path)' "$REF" -- products/db_routing.yaml posthog/product_db_config.py posthog/product_db_router.py)
- PG_IMAGE=$(git show "${REF}:docker-compose.base.yml" 2>/dev/null \
- | grep -m1 -E '^[[:space:]]*image:[[:space:]]*postgres:' | tr -d '[:space:]')
- if [ -z "$MIG_FILES" ]; then
- echo "migrations_key=" >> $GITHUB_OUTPUT
- echo "::notice::no migration files matched, so restore falls back to the SHA schema key"
- else
- MIG_HASH=$(printf '%s\n%s\n%s' "$MIG_FILES" "$PG_IMAGE" "$DB_ROUTING" | sha256sum | cut -c1-40)
- echo "migrations_key=posthog-schema-mig-${SCHEMA_CACHE_EPOCH}-${MIG_HASH}" >> $GITHUB_OUTPUT
- fi
-
- - name: Start Docker services
- env:
- COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml
- run: bin/ci-wait-for-docker launch
-
- - name: Set up Python
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
- with:
- python-version-file: 'pyproject.toml'
- token: ${{ github.token }}
-
- # Direct install: The requested historical source revision may not contain the local action.
- - name: Install uv
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
- with:
- version: '0.12.13'
- enable-cache: true
- save-cache: ${{ github.ref == 'refs/heads/master' }}
-
- - name: Install python dependencies
- run: UV_PROJECT_ENVIRONMENT=$pythonLocation uv sync --frozen --dev
-
- - name: Install Rust
- uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9
- with:
- toolchain: stable
- components: cargo
-
- - name: Cache Rust dependencies
- uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
- with:
- shared-key: 'v2-rust-skills'
- workspaces: rust
- save-if: ${{ github.ref == 'refs/heads/master' }}
-
- - name: Install sqlx-cli
- uses: ./.github/actions/setup-sqlx-cli
-
- - name: Wait for Docker services
- env:
- COMPOSE_FILE: docker-compose.dev.yml:docker-compose.profiles.yml
- run: bin/ci-wait-for-docker wait
-
- - name: Restore schema cache from master
- if: steps.schema-key.outputs.migrations_key != '' || steps.schema-key.outputs.key != ''
- uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
- with:
- path: schema.sql.gz
- # Prefer the shared content key; fall back to the exact merge-base SHA
- # key, then to the newest dump of any migration set.
- key: ${{ steps.schema-key.outputs.migrations_key || steps.schema-key.outputs.key }}
- restore-keys: |
- ${{ steps.schema-key.outputs.key }}
- ${{ steps.schema-key.outputs.prefix_key }}
-
- - name: Prime posthog from cached schema
- # db:restore-schema-fresh restores the dump and seeds the RunPython data a
- # schema-only dump lacks; the migrate step below layers anything newer on
- # top. On failure it leaves a fresh empty db, so migrate runs the full history.
- env:
- SECRET_KEY: '6b01eee4f945ca25045b5aab440b953461faf08693a9abbf1166dc7c6b9772da'
- DATABASE_URL: 'postgres://posthog:posthog@localhost:5432/posthog'
- REDIS_URL: 'redis://localhost'
- OBJECT_STORAGE_ENABLED: 'False'
- TEST: 1
- DEBUG: 1
- TARGET_DB: posthog
- run: |
- if [ ! -f schema.sql.gz ]; then
- echo "::notice::Schema cache miss, so the migrate step below runs the full history"
- exit 0
- fi
- mkdir -p .postgres-backups
- mv schema.sql.gz .postgres-backups/schema-latest.sql.gz
- if ! ./bin/hogli db:restore-schema-fresh; then
- echo "::warning::Schema restore failed (stale or incompatible cached dump?), so the migrate step below runs the full history"
- fi
-
- - name: Run migrations
- env:
- SECRET_KEY: '6b01eee4f945ca25045b5aab440b953461faf08693a9abbf1166dc7c6b9772da'
- DATABASE_URL: 'postgres://posthog:posthog@localhost:5432/posthog'
- REDIS_URL: 'redis://localhost'
- OBJECT_STORAGE_ENABLED: 'False'
- TEST: 1
- DEBUG: 1
- run: |
- DATABASE_URL="postgres://posthog:posthog@localhost:5432/posthog_persons" \
- sqlx database create
- DATABASE_URL="postgres://posthog:posthog@localhost:5432/posthog_persons" \
- sqlx migrate run --source rust/persons_migrations/
- python manage.py migrate --noinput
- python manage.py setup_dev --no-data
-
- - name: Build skills
- env:
- SECRET_KEY: '6b01eee4f945ca25045b5aab440b953461faf08693a9abbf1166dc7c6b9772da'
- DATABASE_URL: 'postgres://posthog:posthog@localhost:5432/posthog'
- REDIS_URL: 'redis://localhost'
- OBJECT_STORAGE_ENABLED: 'False'
- TEST: 1
- DEBUG: 1
- run: ./bin/hogli build:skills
-
- - name: Download context-mill skills
- run: |
- CONTEXT_MILL_ZIP_URL="https://github.com/PostHog/context-mill/releases/latest/download/skills-mcp-resources.zip"
- TARGET_DIR="products/posthog_ai/dist/skills"
- tmp_dir=$(mktemp -d)
- trap 'rm -rf "$tmp_dir"' EXIT
- echo "Downloading context-mill skills from ${CONTEXT_MILL_ZIP_URL}..."
- curl -fsSL --retry 2 --retry-delay 3 -o "$tmp_dir/cm.zip" "$CONTEXT_MILL_ZIP_URL"
- unzip -q -o "$tmp_dir/cm.zip" -d "$tmp_dir/outer"
- mkdir -p "$TARGET_DIR"
- count=0
- while read -r inner_zip; do
- skill_name="$(basename "$inner_zip" .zip)"
- skill_name="${skill_name#omnibus-}"
- skill_dir="$TARGET_DIR/$skill_name"
- mkdir -p "$skill_dir"
- unzip -q -o "$inner_zip" -d "$skill_dir"
- find "$skill_dir" -name 'SKILL.md' -type f -exec sed -i 's/^\(name: *\)omnibus-/\1/' {} +
- count=$((count + 1))
- done < <(find "$tmp_dir/outer" -name 'omnibus-*.zip' -type f)
- echo "Installed ${count} context-mill skills into ${TARGET_DIR}"
-
- - name: Upload rendered skills
- uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
- with:
- name: tasks-golden-skills
- path: products/posthog_ai/dist/skills/
- retention-days: 1
-
- bake:
- # Arming gate: stay a deliberate no-op until the operator provisions the
- # service-account principal + github_oidc TrustMapping + tailnet reach
- # and sets HOG_TASKS_GOLDEN_ENABLED=true. Mirrors golden-snapshots.yml's
- # HOG_GOLDEN_ENABLED gate. `needs: render_skills` — when render skips
- # (unarmed), this skips too. `needs: validate_ref` to check out its
- # validated ref, never the raw workflow_dispatch input.
- needs: [validate_ref, render_skills]
- if: vars.CD_DEPLOY_ENABLED == 'true' && vars.HOG_TASKS_GOLDEN_ENABLED == 'true' && github.ref == 'refs/heads/master'
- name: Bake and promote the tasks golden snapshot
- runs-on: depot-ubuntu-24.04
- timeout-minutes: 240 # worst case: 30m quiet-window guard + 30m cold create + ~90m setup + snapshot push + 30m smoke cold restore + buffer
- # Job-level permissions replace the workflow defaults, so re-declare the
- # contents/id-token the bake needs and add actions:read for the
- # quiet-deploy-window guard's Actions API query.
- permissions:
- contents: read
- id-token: write # mint OIDC for the tailnet join AND the hogplane bearer
- actions: read # read hogland rollout runs for the quiet-window guard
- strategy:
- fail-fast: false
- matrix:
- include:
- - cluster: dev
- hog_host: https://hogland-dev.hedgehog-kitefin.ts.net
- hog_oidc_audience: hogland.dev.posthog.dev
- rollout_workflow: deploy.yml
- - cluster: prod-us
- hog_host: https://hogland.hedgehog-kitefin.ts.net
- hog_oidc_audience: hogland.prod-us.posthog.dev
- rollout_workflow: promote-to-prod.yml
- # prod-eu is deferred to the EU rollout. To add it, append a
- # matrix entry (host https://hogland-eu... + audience
- # hogland.prod-eu.posthog.dev + the EU rollout workflow file)
- # once its svc-ci principal + TrustMapping exist, and extend
- # the prod arming gate below.
-
- env:
- HOG_HOST: ${{ matrix.hog_host }}
- HOG_OIDC_AUDIENCE: ${{ matrix.hog_oidc_audience }}
- # No stored bearer. The CLI mints one on demand — and re-mints on any
- # mid-bake 401 — by hitting the Actions OIDC endpoint and jq-ing the
- # token out. ACTIONS_ID_TOKEN_REQUEST_{URL,TOKEN} are ambient for the
- # whole job under permissions.id-token: write.
- HOG_TOKEN_COMMAND: 'curl -sf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$HOG_OIDC_AUDIENCE" | jq -re .value'
-
- steps:
- - name: Check out posthog
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- with:
- ref: ${{ needs.validate_ref.outputs.ref }}
-
- # Skip clusters not requested when running manually with the input set.
- - name: Filter cluster
- if: ${{ github.event.inputs.cluster != '' && github.event.inputs.cluster != matrix.cluster }}
- run: echo "skip=true" >> "$GITHUB_ENV"
-
- # Second arming gate, for prod-targeting clusters. HOG_TASKS_GOLDEN_ENABLED
- # covers dev provisioning; a prod cluster additionally needs its own
- # svc-ci principal + TrustMapping in that cluster (see the runbook).
- # Stay a no-op until an operator flips HOG_TASKS_GOLDEN_PROD_ENABLED.
- - name: Gate prod-targeting clusters
- if: ${{ env.skip != 'true' && matrix.cluster != 'dev' && vars.HOG_TASKS_GOLDEN_PROD_ENABLED != 'true' }}
- run: |
- echo "::notice::${{ matrix.cluster }} tasks golden bake skipped — set HOG_TASKS_GOLDEN_PROD_ENABLED=true once the ${{ matrix.cluster }} principal + TrustMapping exist"
- echo "skip=true" >> "$GITHUB_ENV"
-
- # The rendered skills the golden ships, produced by render_skills. Land
- # them where build_skills / install-skills.sh expect, then tar the set
- # for bake-golden.sh to stream into the box.
- - name: Download rendered skills
- if: env.skip != 'true'
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
- with:
- name: tasks-golden-skills
- path: products/posthog_ai/dist/skills/
-
- - name: Tar rendered skills
- if: env.skip != 'true'
- run: |
- set -euo pipefail
- SRC=products/posthog_ai/dist/skills
- if [ -z "$(find "$SRC" -mindepth 1 -maxdepth 1 -type d 2>/dev/null)" ]; then
- echo "no rendered skills found under $SRC" >&2
- exit 1
- fi
- tar -czf "$RUNNER_TEMP/golden-skills.tar.gz" -C "$SRC" .
- echo "SKILLS_TARBALL=$RUNNER_TEMP/golden-skills.tar.gz" >> "$GITHUB_ENV"
-
- # The hogland CLI lives in the private PostHog/hogland repo. Check it
- # out with a scoped App token and build it, mirroring
- # golden-snapshots.yml's `go build ./cmd/hogland`.
- - name: Mint hogland-repo token
- if: env.skip != 'true'
- id: hogland-token
- uses: actions/create-github-app-token@1b10c78c7865c340bc4f6099eb2f838309f1e8c3 # v3.1.1
- with:
- app-id: ${{ secrets.GH_APP_HOGLAND_DEPLOYER_APP_ID }}
- private-key: ${{ secrets.GH_APP_HOGLAND_DEPLOYER_PRIVATE_KEY }}
- owner: PostHog
- repositories: hogland
- # Read-only: the deployer App may hold write on hogland, but
- # this job only reads it — checks out the CLI source to build,
- # and the quiet-window guard lists hogland Actions runs.
- permission-contents: read
- permission-actions: read
-
- # Build the CLI from the newest released vX.Y.Z-cli tag: immutable
- # and reviewed, never a moving branch like main (which would run
- # unreviewed CLI code in this credentialed job). Resolved at runtime,
- # so there is no HOGLAND_CLI_REF var to bump on each CLI release — the
- # CLI only orchestrates the bake (create/snapshot/alias) and does not
- # change the golden's contents. Still guard the minimum the scripts need.
- - name: Resolve latest hogland CLI release
- if: env.skip != 'true'
- id: cli-ref
- env:
- GH_TOKEN: ${{ steps.hogland-token.outputs.token }}
- run: |
- set -euo pipefail
- MIN_CLI_VERSION=1.5.0
- ref=$(gh api --paginate repos/PostHog/hogland/releases --jq '.[].tag_name' \
- | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-cli$' | sort -V | tail -1)
- if [[ -z "$ref" ]]; then
- echo "::error::no vX.Y.Z-cli release found on PostHog/hogland"
- exit 1
- fi
- cli_version="${ref#v}"; cli_version="${cli_version%-cli}"
- lowest=$(printf '%s\n%s\n' "$MIN_CLI_VERSION" "$cli_version" | sort -V | head -1)
- if [[ "$cli_version" != "$MIN_CLI_VERSION" && "$lowest" != "$MIN_CLI_VERSION" ]]; then
- echo "::error::latest hogland CLI release ${ref} is below the minimum v${MIN_CLI_VERSION}-cli"
- exit 1
- fi
- echo "ref=${ref}" >> "$GITHUB_OUTPUT"
- echo "::notice::building hogland CLI from ${ref}"
-
- - name: Check out hogland CLI source
- if: env.skip != 'true'
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- with:
- repository: PostHog/hogland
- ref: ${{ steps.cli-ref.outputs.ref }}
- token: ${{ steps.hogland-token.outputs.token }}
- path: hogland-src
-
- - name: Set up Go
- if: env.skip != 'true'
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
- with:
- go-version-file: hogland-src/go.mod
-
- - name: Build hogland CLI
- if: env.skip != 'true'
- run: |
- set -euo pipefail
- mkdir -p "$RUNNER_TEMP/bin"
- (cd hogland-src && go build -o "$RUNNER_TEMP/bin/hogland" ./cmd/hogland)
- echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"
-
- # Join the tailnet as tag:hogland-ci — the tag whose ACL reaches the
- # tag:hogplane device serving the API. WIF, no stored secret; the
- # client-id + audience are shared with hogbox-preview-env.yml.
- - name: Connect to Tailscale (tag:hogland-ci)
- if: env.skip != 'true'
- uses: tailscale/github-action@306e68a486fd2350f2bfc3b19fcd143891a4a2d8 # v4.1.2
- with:
- oauth-client-id: ${{ vars.TS_HOGLAND_CI_CLIENT_ID }}
- audience: ${{ vars.TS_HOGLAND_CI_AUDIENCE }}
- tags: tag:hogland-ci
-
- # Ephemeral keypair for this run. bake-golden.sh bakes the PUBLIC half
- # into the seed box on create and reaches it with the PRIVATE half; the
- # smoke box is reached with the same key.
- - name: Generate ephemeral ssh key
- if: env.skip != 'true'
- run: |
- set -euo pipefail
- ssh-keygen -t ed25519 -f "$RUNNER_TEMP/tasks_golden_key" -N "" -C tasks-golden-ci
- echo "GOLDEN_SSH_KEY=$RUNNER_TEMP/tasks_golden_key" >> "$GITHUB_ENV"
-
- # Resolve the published @posthog/agent version so the golden pins an
- # exact, reproducible agent-server — decoupled from any image. Mirrors
- # cd-sandbox-base-image.yml's agent-version resolve.
- - name: Resolve published agent version
- if: env.skip != 'true'
- id: agent-version
- run: |
- set -euo pipefail
- version=$(curl -fsSL --retry 3 https://registry.npmjs.org/@posthog%2fagent/latest | jq -r '.version')
- if [[ -z "$version" || "$version" == "null" ]]; then
- echo "Could not resolve the latest @posthog/agent version" >&2
- exit 1
- fi
- echo "version=$version" >> "$GITHUB_OUTPUT"
-
- # Quiet-deploy-window guard (mirrors hogland golden-snapshots.yml):
- # do not bake while a hogd rollout is queued or in flight, or within
- # 900s of the last one — a rollout racing the bake orphans the seed box
- # and wastes the run. The rollout workflow differs per cluster
- # (dev: deploy.yml, prod-us: promote-to-prod.yml), so it comes from the
- # matrix, not a single shared var. Reads PostHog/hogland's Actions with
- # the hogland App token (actions:read); an API error fails OPEN (a
- # racing rollout only wastes a run, it cannot corrupt the alias).
- - name: Wait for a quiet hogd rollout window
- if: env.skip != 'true'
- env:
- GH_TOKEN: ${{ steps.hogland-token.outputs.token }}
- ROLLOUT_WORKFLOW: ${{ matrix.rollout_workflow }}
- run: |
- set -euo pipefail
- if [[ -z "$ROLLOUT_WORKFLOW" ]]; then
- echo "::notice::no rollout workflow for this cluster; skipping the quiet-window guard"
- exit 0
- fi
- deadline=$(( $(date +%s) + 1800 )) # wait at most 30m for the window
- runs_api="repos/PostHog/hogland/actions/workflows/${ROLLOUT_WORKFLOW}/runs"
- while :; do
- in_progress=$(gh api "${runs_api}?status=in_progress&per_page=1" --jq '.total_count' 2>/dev/null || echo "error")
- queued=$(gh api "${runs_api}?status=queued&per_page=1" --jq '.total_count' 2>/dev/null || echo "error")
- if [[ "$in_progress" == "error" || "$queued" == "error" ]]; then
- echo "::notice::could not read hogland rollout runs (App token may lack actions:read); proceeding"
- exit 0
- fi
- active=$(( in_progress + queued ))
- last_completed=$(gh api "${runs_api}?status=completed&per_page=1" --jq '.workflow_runs[0].updated_at // empty' 2>/dev/null || echo "")
- since=999999
- if [[ -n "$last_completed" ]]; then
- since=$(( $(date +%s) - $(date -u -d "$last_completed" +%s 2>/dev/null || echo 0) ))
- fi
- if [[ "$active" == "0" && "$since" -ge 900 ]]; then
- echo "::notice::quiet window: no queued/in-progress rollout, ${since}s since the last"
- break
- fi
- if [[ "$(date +%s)" -ge "$deadline" ]]; then
- echo "::error::hogd rollout window did not go quiet within 30m (active=${active}, since=${since}s); not baking"
- exit 1
- fi
- echo "rollout active (in_progress=${in_progress}, queued=${queued}, ${since}s since last); waiting 30s"
- sleep 30
- done
-
- # Build into the CANDIDATE alias — no live consumer points at this
- # name, so a bad build can't poison task runs. bake-golden.sh boots a
- # bare seed box from box primitives, streams the payload (guards + cpu
- # sampler + the rendered-skills tarball + install-skills.sh +
- # setup-golden.sh) in over the box's SSH, runs setup-golden.sh, then
- # snapshots and points the candidate alias at it. Golden shape is the
- # task sandbox shape: 4 CPU / 20 GiB / 64 GiB.
- - name: Bake candidate snapshot (box primitives)
- if: env.skip != 'true'
- env:
- ALIAS: posthog-tasks-candidate
- SSH_KEY: ${{ env.GOLDEN_SSH_KEY }}
- IMAGES_DIR: products/tasks/backend/sandbox/images
- SKILLS_TARBALL: ${{ env.SKILLS_TARBALL }}
- AGENT_VERSION: ${{ steps.agent-version.outputs.version }}
- BOX_CPUS: '4'
- # Both blank on the schedule -> fixed-size 20 GiB golden, unchanged.
- # A manual dispatch can bake a pluggable-memory golden (e.g. cap 32768
- # + boot 4096) without touching the scheduled path.
- BOX_MEM_MIB: ${{ inputs.box_mem_mib || '20480' }}
- BOX_MEM_BOOT_MIB: ${{ inputs.box_mem_boot_mib || '' }}
- BOX_DISK_GIB: '64'
- run: |
- set -euo pipefail
- bash products/tasks/backend/sandbox/images/hogland/bake-golden.sh
-
- # Boot a smoke box from the candidate, run the tasks readiness
- # contract, delete. Non-zero => DO NOT promote, so the live alias
- # keeps pointing at the previous known-good snapshot.
- - name: Smoke-test candidate
- id: smoke
- if: env.skip != 'true'
- env:
- ALIAS: posthog-tasks-candidate
- SSH_KEY: ${{ env.GOLDEN_SSH_KEY }}
- run: |
- set -euo pipefail
- bash products/tasks/backend/sandbox/images/hogland/smoke-golden.sh | tee "$RUNNER_TEMP/smoke.out"
- secs=$(sed -n 's/^SMOKE_HEALTHY_SECONDS=//p' "$RUNNER_TEMP/smoke.out")
- echo "healthy_seconds=${secs:-unknown}" >> "$GITHUB_OUTPUT"
-
- # Promote: re-point the live alias at the candidate's snapshot id. A
- # single alias PUT — atomic from any restoring caller's perspective.
- - name: Promote candidate to the live alias
- id: promote
- if: env.skip != 'true'
- env:
- CLUSTER: ${{ matrix.cluster }}
- # Blank -> the default golden every box restores from. A pluggable
- # bake sets this to a separate alias (e.g. posthog-tasks-hotplug) so
- # it never touches the default; the hotplug flag selects it per team.
- TARGET_ALIAS: ${{ inputs.target_alias || 'posthog-tasks-default' }}
- run: |
- set -euo pipefail
- SNAP_ID=$(hogland snapshot resolve posthog-tasks-candidate | jq -r .snapshot_id)
- # Retry the alias PUT: a concurrent repoint returns 500 (not a
- # clean 409), so a transient collision must not fail the promote.
- promoted=0
- for attempt in 1 2 3 4 5; do
- if hogland snapshot alias "$SNAP_ID" "$TARGET_ALIAS"; then
- promoted=1
- break
- fi
- echo "alias PUT attempt ${attempt} failed; retrying in $((attempt * 5))s" >&2
- sleep $((attempt * 5))
- done
- if [[ "$promoted" != "1" ]]; then
- echo "::error::failed to repoint $TARGET_ALIAS after 5 attempts"
- exit 1
- fi
- # Confirm the live alias now resolves to our snapshot. This
- # resolve targets the same HOG_HOST the PUT wrote to, so it
- # catches a lost or overwritten alias write on THIS cluster — it
- # does not prove the bake ran against the intended cluster (both
- # the PUT and this read use the same matrix HOG_HOST).
- resolved=$(hogland snapshot resolve "$TARGET_ALIAS" | jq -r .snapshot_id)
- if [[ "$resolved" != "$SNAP_ID" ]]; then
- echo "::error::$TARGET_ALIAS in cluster ${CLUSTER} resolves to ${resolved}, expected ${SNAP_ID}"
- exit 1
- fi
- echo "snapshot_id=$SNAP_ID" >> "$GITHUB_OUTPUT"
-
- # Dated archive alias for rollback. Reuse the snapshot id the promote
- # step already resolved (via env, not ${{ }} into run:) rather than
- # resolving the candidate alias again — a second resolve could race a
- # concurrent rebuild and archive a different snapshot than was
- # promoted. Retention prune is intentionally not wired here — a
- # separate cleanup cron handles it once we've watched the alias count
- # for a couple of weeks.
- - name: Archive dated alias
- if: env.skip != 'true'
- env:
- SNAP_ID: ${{ steps.promote.outputs.snapshot_id }}
- run: |
- set -euo pipefail
- DATE=$(date -u +%Y%m%d)
- hogland snapshot alias "$SNAP_ID" posthog-tasks-default-"$DATE"
-
- # Delete the seed + smoke boxes by name, always. The seed and smoke
- # boxes use an unregistered kind (no server-side TTL), so a killed
- # runner cannot reap them — and their in-script EXIT traps do not fire
- # on a job cancel. This always()-guarded step is the backstop that
- # prevents a leaked 64 GiB box on cancel/timeout. Best-effort: the
- # names are fixed in bake-golden.sh / smoke-golden.sh, and a delete of
- # an absent box is a no-op. Runs only after the CLI + tailnet came up
- # (env.skip guards the cluster-filtered legs, which create no box).
- - name: Tear down seed + smoke boxes
- if: ${{ always() && env.skip != 'true' }}
- run: |
- set -uo pipefail
- for box_name in golden-seed-tasks golden-smoke-tasks; do
- id=$(hogland box list 2>/dev/null \
- | jq -r --arg n "$box_name" 'map(select(.spec.name == $n)) | .[0].id // empty' 2>/dev/null || true)
- if [[ -n "$id" ]]; then
- echo "deleting leaked box $box_name ($id)"
- hogland box delete "$id" >/dev/null 2>&1 || true
- fi
- done
-
- # Step outputs go through env vars, not direct ${{ }} interpolation
- # into run: — semgrep's shell-injection rule treats step output as
- # untrusted, and env indirection is the standard fix.
- - name: Step summary
- if: env.skip != 'true'
- env:
- CLUSTER: ${{ matrix.cluster }}
- HOG_HOST: ${{ matrix.hog_host }}
- SNAPSHOT_ID: ${{ steps.promote.outputs.snapshot_id }}
- HEALTHY_SECONDS: ${{ steps.smoke.outputs.healthy_seconds }}
- run: |
- {
- echo "### tasks golden snapshot: $CLUSTER"
- echo ""
- echo "- cluster: \`$CLUSTER\` ($HOG_HOST)"
- echo "- promoted snapshot \`$SNAPSHOT_ID\` -> alias \`posthog-tasks-default\`"
- echo "- smoke create->contract: ${HEALTHY_SECONDS}s"
- } >> "$GITHUB_STEP_SUMMARY"
diff --git a/docs/published/handbook/engineering/ai/writing-skills.md b/docs/published/handbook/engineering/ai/writing-skills.md
index 4434a13fce48..0b6aff5fe01b 100644
--- a/docs/published/handbook/engineering/ai/writing-skills.md
+++ b/docs/published/handbook/engineering/ai/writing-skills.md
@@ -442,7 +442,6 @@ so a skill from this repo is absent there rather than overwritten.
| PostHog Desktop runtime, every 30 min | `products/desktop/packages/workspace-server/src/services/posthog-plugin/update-skills-saga.ts` |
| Desktop harness bundle | `packages/agent/packages/harness/tsup.config.ts` – context-mill only, this repo's skills are absent rather than overwritten |
| Tasks sandbox base image | `.github/workflows/cd-sandbox-base-image.yml` |
-| Tasks golden snapshot | `.github/workflows/cd-tasks-golden-snapshot.yml` |
| `PostHog/skills` mirror | that repo's `.github/workflows/sync-omnibus.yml` |
| `PostHog/ai-plugin` plugin | that repo's `.github/workflows/sync-skills.yml` |
diff --git a/products/tasks/backend/logic/services/hogland_sandbox.py b/products/tasks/backend/logic/services/hogland_sandbox.py
index edee6268f47e..c6f7b264a469 100644
--- a/products/tasks/backend/logic/services/hogland_sandbox.py
+++ b/products/tasks/backend/logic/services/hogland_sandbox.py
@@ -62,15 +62,15 @@ logger = logging.getLogger(__name__)
# instead of minting an immortal box.
HOGLAND_TASKS_BOX_KIND = "agent"
-# Every template hogland supports maps to a global snapshot alias baked in CI
-# (the tasks golden-snapshot workflow). Only the plain default-template run is routed
+# Every template hogland supports maps to a global snapshot alias. No CI job bakes
+# these any more; the alias must already exist. Only the plain default-template run is routed
# to hogland; anything else stays on Modal (see get_task_processing_context).
TEMPLATE_TO_SNAPSHOT_ALIAS: dict[SandboxTemplate, str] = {
SandboxTemplate.DEFAULT_BASE: "alias:posthog-tasks-default",
}
-# The pluggable-memory golden (baked by the tasks golden-snapshot workflow with a
-# BOX_MEM_BOOT_MIB below the cap). A box restored from it boots small and hot-adds guest
+# The pluggable-memory golden (boot memory below the cap). A box restored from it
+# boots small and hot-adds guest
# RAM up to the cap on demand. Selected per run by the tasks-hogland-hotplug-golden flag
# (config.use_hotplug_golden); only the default template has a pluggable variant. The
# alias must exist before the flag is enabled for a team.
diff --git a/products/tasks/backend/sandbox/images/hogland/GOLDEN_CI_RUNBOOK.md b/products/tasks/backend/sandbox/images/hogland/GOLDEN_CI_RUNBOOK.md
deleted file mode 100644
index fe883e4bc9bb..000000000000
--- a/products/tasks/backend/sandbox/images/hogland/GOLDEN_CI_RUNBOOK.md
+++ /dev/null
@@ -1,297 +0,0 @@
-# Tasks golden snapshot CI — ops runbook
-
-The workflow `.github/workflows/cd-tasks-golden-snapshot.yml` bakes the task
-sandbox golden snapshot and promotes the alias `posthog-tasks-default` that
-`HoglandSandbox.create`
-(`products/tasks/backend/logic/services/hogland_sandbox.py`) restores every task
-box from.
-
-This runbook covers the hogland-team dependencies each cluster needs **before**
-the workflow can run. Until they exist and the arming variables are set, the
-workflow is a deliberate no-op (the job `if` gates on `HOG_TASKS_GOLDEN_ENABLED`)
-so a merged-but-unprovisioned workflow never sits red.
-
-Modal / `Dockerfile.sandbox-base` is untouched by this workflow — it bakes the
-same contents into a hogland snapshot, it does not change the container image.
-
-## What the workflow does
-
-0. `validate_ref` job (unprivileged, no secrets) — resolves the
- `workflow_dispatch` `ref` input before any credentialed job checks that tree
- out. Blank resolves to the literal `master` branch; a non-blank value must be
- a full 40-char commit SHA that the GitHub compare API confirms is already an
- ancestor of `master`, else the job fails. `render_skills` and `bake` check out
- `needs.validate_ref.outputs.ref`, never the raw input, and every privileged
- job additionally gates on `github.ref == 'refs/heads/master'`. This closes the
- `ref` RCE in the workflow YAML (see "Known open security issues").
-1. `render_skills` job — renders the agent skills the golden ships, the same way
- `cd-sandbox-base-image.yml`'s `build_skills` job does. It stands up a DB,
- restores the schema cache saved on master so `migrate` only tops up the newer
- migrations, runs `setup_dev --no-data`, then `hogli build:skills` to expand the
- skill `.md.j2` templates and merges in the context-mill skills. The DB is
- needed because 24 skill templates call `render_hogql_example`, which reads
- `Team.objects.first()`. Uploads the merged set as the `tasks-golden-skills`
- artifact. Rendered once, reused by every cluster. It shares the master arm
- gate, so an unarmed nightly renders nothing.
-2. Joins the hogland tailnet as `tag:hogland-ci` (the tag whose ACL reaches the
- `tag:hogplane` device serving the API).
-3. Authenticates to hogplane as a per-cluster `svc-ci-*` service-account
- principal via GitHub OIDC — no stored bearer. The CLI mints the token from the
- Actions OIDC endpoint and re-mints on any mid-bake 401 via `HOG_TOKEN_COMMAND`,
- because a golden bake runs ~90 min while an OIDC token lives ~5 min.
-4. Builds the `hogland` CLI from a scoped checkout of `PostHog/hogland`.
-5. `bake-golden.sh` — decomposes the bake into box primitives instead of
- `hogland snapshot-build` (whose <=256 KiB bootstrap cannot carry the multi-MB
- skills set). It `box create`s a bare seed box (cold boot, `--no-connect`),
- streams the payload into the box over the box's own SSH (`cat >` — the git/gh
- guards, the cpu sampler, the rendered-skills tarball, `install-skills.sh`, and
- `setup-golden.sh`), runs `setup-golden.sh` in the box, then `box snapshot`s the
- result and points the candidate alias at it. `setup-golden.sh` installs
- node/uv/tools/agentsh fresh, npm-installs `@posthog/agent` at the version the
- workflow resolved, and installs the delivered skills. No GHCR image and no
- public artifact host are involved — delivery rides the box's SSH.
-6. Boots a smoke box from `posthog-tasks-candidate`, runs `smoke-golden.sh`
- (agent-server starts; a trivial clone + exec works; the hogpanion exec daemon
- is running with the container-style env), and deletes it. A smoke failure
- skips promotion, leaving `posthog-tasks-default` on the previous known-good
- snapshot.
-7. On success, re-points `posthog-tasks-default` at the candidate's snapshot
- (retrying the alias PUT, since a concurrent repoint returns 500 not 409),
- confirms the live alias resolves to that snapshot in the expected cluster, and
- stamps a dated archive alias `posthog-tasks-default-YYYYMMDD` for rollback.
-
-Before the bake step, a quiet-deploy-window guard waits for no queued or
-in-progress hogd rollout and >=900s since the last one (up to 30 min), mirroring
-hogland's `golden-snapshots.yml`. The rollout workflow is per cluster (see "Repo
-config").
-
-The matrix covers **dev** and **prod-us**. **prod-eu is deferred** to the EU
-rollout — see "Adding prod-eu" below.
-
-## Per-cluster prerequisites (hogland team, out-of-band)
-
-These are hogland runtime-cluster data, not repo config. They mirror the
-`preview` persona's provisioning in hogland's `docs/BUILD_TEST_DEPLOY.md`.
-
-For **each** cluster the workflow targets (dev, then prod-us):
-
-1. **Service-account principal.** Create a `svc-ci-tasks-golden` principal via
- `POST /v1/service-accounts` on that cluster's hogplane. This endpoint is
- admin-only and has no CLI; it mints a name of the form `svc-<slug>-<rand>`, so
- the random suffix is expected — do **not** read it as a failure.
-2. **`github_oidc` TrustMapping.** Map this repo's workflow to that principal:
- `{repo: PostHog/posthog, workflow: cd-tasks-golden-snapshot.yml}` →
- `svc-ci-tasks-golden`. TrustMappings are per-cluster runtime data — the dev
- mapping grants nothing in prod-us, so create it again per cluster. Without it
- the OIDC mint succeeds and hogplane returns 401. **Pin `ref: refs/heads/master`
- once hogplane's TrustMapping schema supports it** (PostHog/hogland#414, durable
- fix PostHog/hogland#424) — this is the durable fix for the branch-dispatch
- class of attack. The workflow's own
- `validate_ref` job + per-job `github.ref == 'refs/heads/master'` gates raise
- the bar, but a branch copy of the workflow can strip in-file gates, so the
- TrustMapping ref pin is the backstop that cannot be bypassed from the repo.
-3. **OIDC audience.** The mint audience is the per-cluster literal in the
- workflow's matrix, kept equal to `hogplane.githubOIDCAudience` in that
- cluster's values file:
-
- | Cluster | `HOG_HOST` | Audience (`HOG_OIDC_AUDIENCE`) |
- | ------- | --------------------------------------------- | ------------------------------ |
- | dev | `https://hogland-dev.hedgehog-kitefin.ts.net` | `hogland.dev.posthog.dev` |
- | prod-us | `https://hogland.hedgehog-kitefin.ts.net` | `hogland.prod-us.posthog.dev` |
-
-4. **Tailnet admittance.** The workflow joins as `tag:hogland-ci`, reusing the
- `TS_HOGLAND_CI_CLIENT_ID` + `TS_HOGLAND_CI_AUDIENCE` repo variables that
- `hogbox-preview-env.yml` already uses. One join reaches the `tag:hogplane`
- device serving each cluster's API, so if preview is already provisioned no
- extra tailnet onboarding is needed for the API calls.
-5. **Tailnet route to the box VPC (required for `--access-type ssh-private`).**
- The bake and smoke reach the seed/smoke box over its VPC IP, not the public
- internet. `tag:hogland-ci` reaches `tag:hogplane` today, but **not** a box's
- VPC IP — so `ssh-private` fails at the first `ssh` until a grant is added in
- `posthog-cloud-infra/tailnet-policy.hujson`:
-
- ```text
- src: tag:hogland-ci
- dst: 10.90.0.0/16:10000-59999, 10.91.0.0/16:10000-59999
- ```
-
- This is an ops/infra prerequisite (like the svc-ci principal and arming), not
- a repo change. Do not arm any cluster before it lands, or every bake fails on
- the SSH reachability wait.
-
-## Repo config (PostHog/posthog)
-
-- **Variables** (Settings → Secrets and variables → Actions → Variables):
- - `HOG_TASKS_GOLDEN_ENABLED=true` — master arm. Until set, the whole job is a
- no-op. Set it once the dev prerequisites (steps 1–5, including the tailnet route to the box VPC) exist.
- - `HOG_TASKS_GOLDEN_PROD_ENABLED=true` — second arm for prod-targeting
- clusters. The prod-us leg stays a no-op until this is set, so it does not
- fail nightly before its own principal + TrustMapping exist.
- - No `HOGLAND_CLI_REF` needed — the workflow resolves the newest released
- `vX.Y.Z-cli` tag from `PostHog/hogland` at runtime (immutable and reviewed,
- never `main`), guarding a `v1.5.0-cli` minimum.
- - The quiet-window guard's rollout workflow is **per cluster**, hardcoded in the
- bake matrix (dev: `deploy.yml`, prod-us: `promote-to-prod.yml`) — not a shared
- var, because the two clusters roll out through different workflows. The guard
- counts `queued` + `in_progress` runs and waits up to 30 min for a quiet window.
- It reads `PostHog/hogland` Actions with the hogland App token, so that App
- installation needs `actions:read` on hogland; without it the guard fails open
- (a racing rollout only wastes a run, it cannot corrupt the alias). Adding a
- cluster means adding its rollout workflow file to the matrix.
- - `TS_HOGLAND_CI_CLIENT_ID`, `TS_HOGLAND_CI_AUDIENCE` — shared with
- `hogbox-preview-env.yml`; already present if preview is provisioned.
-- **Secrets** — the workflow checks out `PostHog/hogland` to build the CLI with
- the existing `GH_APP_HOGLAND_DEPLOYER` GitHub App (org secrets
- `GH_APP_HOGLAND_DEPLOYER_APP_ID` + `GH_APP_HOGLAND_DEPLOYER_PRIVATE_KEY`),
- minted read-only: `contents: read` for the checkout, `actions: read` so the
- quiet-window guard can list hogland rollout runs. The App must be installed on
- `PostHog/hogland` with both.
-
-## Running it
-
-- Manual: Actions → "Tasks Golden Snapshot CD" → Run workflow. Leave `cluster`
- blank to bake every armed cluster, or set it to `dev` / `prod-us` to bake one.
- Leave `ref` blank to render from `master`, or set it to a full commit SHA that
- is already merged into `master`. `validate_ref` enforces this: a blank ref
- resolves to `master`, a non-blank value must be a full 40-char SHA that is an
- ancestor of `master`, and anything else (a branch name, an unmerged SHA) fails
- the run before any credentialed job starts. Dispatch only works from `master`
- (every privileged job gates on `github.ref == 'refs/heads/master'`).
-- Nightly: the `schedule` cron fires daily but only bakes armed clusters,
- rendering from `master`.
-
-## Cleaning up leaked seed/smoke boxes
-
-The seed (`golden-seed-tasks`) and smoke (`golden-smoke-tasks`) boxes use an
-unregistered kind with no server-side TTL, so hogland never reaps them. The
-workflow's `always()` teardown deletes them on a normal cancel or failure — but
-it does **not** run if the runner itself dies (an infra kill, a spot reclaim),
-which leaks a 64 GiB box. The names are fixed, so a leak is at most one seed +
-one smoke per cluster (the next run pre-cleans them). After any run that ended
-abnormally, sweep them (dev/prod-us need the tailnet):
-
-```bash
-for n in golden-seed-tasks golden-smoke-tasks; do
- id=$(hogland box list | jq -r --arg n "$n" 'map(select(.spec.name==$n)) | .[0].id // empty')
- [ -n "$id" ] && hogland box delete "$id"
-done
-```
-
-## Rollback
-
-Each successful bake stamps `posthog-tasks-default-YYYYMMDD`. To roll back, point
-the live alias at a known-good archive:
-
-```bash
-hogland snapshot resolve posthog-tasks-default-<YYYYMMDD> | jq -r .snapshot_id
-hogland snapshot alias <snapshot_id> posthog-tasks-default
-```
-
-After hogland#426 the alias PUT is owner-scoped: only the alias's owner (the
-`svc-ci-*` principal that first promoted it) or an admin may repoint
-`posthog-tasks-default`, so a hand rollback runs as an admin or as svc-ci. See
-hogland's `docs/BUILD_TEST_DEPLOY.md` "Alias ownership" section for the adoption
-and recovery model.
-
-## Adding prod-eu
-
-1. Provision the prerequisites above in the prod-eu cluster: a
- `svc-ci-tasks-golden` principal, a `github_oidc` TrustMapping
- `{repo: PostHog/posthog, workflow: cd-tasks-golden-snapshot.yml}` (pin
- `ref: refs/heads/master` once PostHog/hogland#414 lands), and the audience
- `hogland.prod-eu.posthog.dev`.
-2. Append a matrix entry in the workflow (host + audience + the EU rollout
- workflow file) and confirm the prod arming gate covers it.
-3. Confirm `tag:hogland-ci` reaches the prod-eu `tag:hogplane` device.
-
-## Known Phase-1 gaps (validate on a live cluster)
-
-- **No live bake has run.** The scripts and workflow are lint- and
- shellcheck-clean, but a real bake needs a cluster, the tailnet, and the
- principal — none of which exist in local dev. First armed run is the real test.
-- **Smoke asserts the daemon env, and best-effort the exec API.** `smoke-golden.sh`
- reads the running hogpanion daemon's live `/proc/<MainPID>/environ` (via `sudo`;
- the box ssh user is `hog`) — the env its hog-exec children inherit — and asserts
- `IS_SANDBOX=1`, a `/opt/posthog/bin` first `PATH`, and `PYTHONPATH`. This catches
- a hogpanion that never re-exec'd with the drop-in, which an SSH login shell would
- hide (PAM feeds it `/etc/environment`). It also now attempts a trivial command
- through `POST /v1/hogboxes/{id}/exec` — the production reach path — but that
- assertion **fails open** only on `000` (genuinely unreachable). A `404` (missing
- box or non-owner) is now a hard fail, since it would be a real regression. Once
- PostHog/hogland#416 lands a `box exec`/`box cp` verb (durable fix
- PostHog/hogland#425) and the exec contract is confirmed, make the exec assertion
- a hard gate on all codes.
-- **Quiet-deploy-window guard is provisional.** The guard polls `PostHog/hogland`
- Actions for the per-cluster rollout workflow's queued + in-progress runs and
- waits up to 30 min for a quiet window, but **fails open** when the hogland App
- lacks `actions:read` on hogland. Grant the App `actions:read` to arm it. A racing
- rollout without the guard only wastes a run (it orphans the seed box and skips
- promotion), it cannot corrupt the alias.
-- **The golden tracks the render ref (default `master`).** The `render_skills` job
- checks out PostHog at the validated ref and renders the skills from it, and
- `setup-golden.sh` reconstructs `Dockerfile.sandbox-base` from the same checkout —
- so the ref is the one knob for what the golden tracks. On the nightly schedule
- this is `master`; pass the `ref` `workflow_dispatch` input (a merged SHA) for a
- reproducible golden. `@posthog/agent` is decoupled from the ref: the workflow
- resolves the latest published version and pins it into the bake.
-- **Cluster confirm catches a lost write, not a wrong-cluster bake.** The promote
- step re-resolves `posthog-tasks-default` after the PUT, but on the same
- per-cluster `HOG_HOST` it wrote to. That catches a lost or overwritten alias
- write on this cluster; it does not prove the bake targeted the intended cluster.
- A cheap cluster-identity query on the CLI would let it assert the expected
- cluster name — not available today, so this stays a lost-write check.
-- **The `tasks-golden-skills` artifact is public-downloadable.** It is a
- build artifact on a public repo, so anyone can download it. Its content is
- repo-derived (rendered skills + fixtures) — confirm the render emits nothing
- beyond repo fixtures before treating the golden as sensitive-free.
-- **Unpinned third-party installs.** `setup-golden.sh` runs the nodesource
- `setup_24.x` piped to `bash`, the gh `.deb` with no sha256, and global npm
- tools — the same posture as `Dockerfile.sandbox-base`. The nightly rebake
- widens the window in which an upstream change lands in a golden unnoticed.
-- **hogland#410 (per-box `PATH`) forces a full rebake.** When it lands it requires
- a rebake of every golden, and its per-box `PATH` overrides this drop-in.
-
-## Known open security issues
-
-The `ref` RCE and branch-dispatch findings from PR review are now **mitigated in
-`cd-tasks-golden-snapshot.yml`**: the unprivileged `validate_ref` job resolves and
-ancestor-checks the `ref` input before any credentialed checkout, and every
-privileged job gates on `github.ref == 'refs/heads/master'`. The residual, durable
-gap is hogland-owned:
-
-- **Branch dispatch — durable fix is hogland-side (PostHog/hogland#414, durable
- fix PostHog/hogland#424).** The in-file `github.ref == 'refs/heads/master'` gates
- raise the bar, but a branch copy of the workflow can strip them. The fix that
- cannot be bypassed from the repo is pinning `ref: refs/heads/master` in each
- cluster's `github_oidc` TrustMapping. Until #414 lands and the mapping is pinned,
- treat the in-workflow gates as the only barrier and keep dispatch discipline
- (dispatch only from `master`).
-- **Alias namespace is global and unprotected (PostHog/hogland#415, durable fix
- PostHog/hogland#426).** Alias PUT checks only new-snapshot ownership and DELETE
- checks nothing, so a `svc-ci-tasks-golden` principal on prod-us could repoint or
- delete `devbox-golden` / `posthog-preview-golden`. **Keep prod-us UNARMED
- (`HOG_TASKS_GOLDEN_PROD_ENABLED` unset) until #415 lands.** Arming dev grants the
- CI principal delete over `devbox-golden` (which lives on dev), so dev is only
- acceptable to arm knowingly — #426 mitigates this once it lands.
-
-## hogland dependencies (filed issues)
-
-Items owned by the hogland team, filed from this review. Not blockers for merging
-the workflow (it is a no-op until armed), but referenced above:
-
-- **PostHog/hogland#414** — `github_oidc` TrustMapping cannot pin a `ref` (durable
- branch-dispatch fix, PR PostHog/hogland#424). Gates arming prod-us safely.
-- **PostHog/hogland#415** — alias namespace PUT/DELETE unprotected (durable fix PR
- PostHog/hogland#426). With #426 merged, `svc-ci` adopts `posthog-tasks-default`
- on its first PUT (a new alias is owned by its creator), so no extra
- provisioning step — see hogland `docs/BUILD_TEST_DEPLOY.md` (Alias ownership).
-- **PostHog/hogland#416** — no `box exec` / `box cp` verb: `hogland box exec` does
- not exist today, so the smoke uses a raw `POST .../exec` (durable fix PR
- PostHog/hogland#425). Also covers the batch `/exec` timeout mismatch (advertises
- 6h, caps at 60s).
-- **PostHog/hogland#417** — snapshot chunks are never GC'd (a daily bake leaks
- ~10-25 GiB). Must be reference-aware, **not** a bucket-wide expiry rule (that
- caused the 2026-08-06 preview outage).
-- **PostHog/hogland#418** — hogbox env / `EnvironmentFile` ordering: an
- `EnvironmentFile=` key can override the guard-critical `Environment=` vars, so
- the drop-in sets no `EnvironmentFile`.
diff --git a/products/tasks/backend/sandbox/images/hogland/bake-golden.sh b/products/tasks/backend/sandbox/images/hogland/bake-golden.sh
deleted file mode 100755
index 64d19032ef37..000000000000
--- a/products/tasks/backend/sandbox/images/hogland/bake-golden.sh
+++ /dev/null
@@ -1,255 +0,0 @@
-#!/usr/bin/env bash
-# bake-golden.sh — bake the PostHog *tasks* golden snapshot from box primitives.
-#
-# This replaces `hogland snapshot-build`. snapshot-build can only deliver a
-# <=256 KiB bootstrap (--setup-script + --inline-file heredocs), which cannot
-# carry the multi-MB rendered-skills set. So we decompose the bake into the
-# underlying primitives and stream the payload in over the box's own SSH:
-#
-# 1. `box create` a bare seed box from the base (cold boot, --no-connect).
-# 2. Stream the payload into the box over SSH (cat > file): the git/gh guards,
-# the cpu sampler, the rendered-skills tarball, install-skills.sh, and
-# setup-golden.sh. Only `cat` is needed in the box, so this works without a
-# remote scp/sftp binary and without any public artifact host.
-# 3. Run setup-golden.sh in the box over SSH. It installs node/uv/tools/agentsh
-# fresh, npm-installs @posthog/agent, and installs the delivered skills.
-# 4. `box snapshot` the result and point the candidate alias at it.
-#
-# Success detection is ours now (snapshot-build appended /var/lib/hog/
-# snapshot-build-ok and polled it; we do not). setup-golden.sh runs under
-# `set -euo pipefail`, and we check its SSH exit code explicitly — a non-zero
-# setup aborts the bake before `box snapshot`, so a broken build never promotes.
-#
-# Required env (set by the workflow):
-# ALIAS — candidate alias to point at the new snapshot
-# SSH_KEY — path to the PRIVATE key; the public half is at ${SSH_KEY}.pub
-# IMAGES_DIR — path to products/tasks/backend/sandbox/images (payload source)
-# SKILLS_TARBALL — path to the runner-rendered skills tarball (.tar.gz)
-# Optional env:
-# AGENT_VERSION — @posthog/agent version to install in-box (default: latest)
-# BOX_CPUS / BOX_MEM_MIB / BOX_DISK_GIB — seed box shape (default 4 / 20480 / 64)
-# BOX_MEM_BOOT_MIB — boot memory (MiB) for a pluggable-memory golden: the box
-# boots this small and hot-adds guest RAM up to BOX_MEM_MIB on demand. Blank
-# (default) bakes a fixed-size golden at BOX_MEM_MIB.
-#
-# On success prints a single parseable line to stdout:
-# BAKED_SNAPSHOT_ID=<id>
-
-set -euo pipefail
-
-: "${ALIAS:?ALIAS is required}"
-: "${SSH_KEY:?SSH_KEY is required (private key path; pub at \${SSH_KEY}.pub)}"
-: "${IMAGES_DIR:?IMAGES_DIR is required (path to sandbox/images)}"
-: "${SKILLS_TARBALL:?SKILLS_TARBALL is required (path to rendered-skills tarball)}"
-AGENT_VERSION="${AGENT_VERSION:-latest}"
-BOX_CPUS="${BOX_CPUS:-4}"
-BOX_MEM_MIB="${BOX_MEM_MIB:-20480}"
-BOX_MEM_BOOT_MIB="${BOX_MEM_BOOT_MIB:-}"
-BOX_DISK_GIB="${BOX_DISK_GIB:-64}"
-
-log() { printf '[tasks-bake] %s\n' "$*" >&2; }
-
-GIT_GUARD="$IMAGES_DIR/git-guard.sh"
-GH_GUARD="$IMAGES_DIR/gh-guard.sh"
-HOGLI_SHIM="$IMAGES_DIR/hogli-shim.sh"
-CPU_SAMPLER="$IMAGES_DIR/cpu_billing_sampler.py"
-SETUP_SCRIPT="$IMAGES_DIR/hogland/setup-golden.sh"
-INSTALL_SKILLS="$IMAGES_DIR/install-skills.sh"
-
-for f in "$GIT_GUARD" "$GH_GUARD" "$HOGLI_SHIM" "$CPU_SAMPLER" "$SETUP_SCRIPT" "$INSTALL_SKILLS" "$SKILLS_TARBALL" "${SSH_KEY}.pub"; do
- test -f "$f" || { log "FAIL: required file missing: $f"; exit 1; }
-done
-
-# Fixed name so a killed prior run's leftover seed box is found and cleaned
-# rather than leaking a node or colliding on the next create (hogplane 409s on a
-# duplicate name — matches smoke-golden.sh's fixed-name teardown).
-SEED_BOX_NAME=golden-seed-tasks
-SEED_BOX_ID=""
-
-# List across ALL kinds, not just ours: hogplane's name-uniqueness check
-# (ensureBoxNameUnique) scans every kind, so a same-named box of another kind
-# would wedge the create. Match by name only so the pre-clean finds it.
-resolve_seed_box_id() {
- hogland box list 2>/dev/null \
- | jq -r --arg n "$SEED_BOX_NAME" 'map(select(.spec.name == $n)) | .[0].id // empty' 2>/dev/null \
- || true
-}
-
-cleanup() {
- if [[ -z "$SEED_BOX_ID" ]]; then
- SEED_BOX_ID=$(resolve_seed_box_id)
- fi
- if [[ -n "$SEED_BOX_ID" ]]; then
- log "deleting seed box $SEED_BOX_ID"
- hogland box delete "$SEED_BOX_ID" >/dev/null 2>&1 || true
- fi
-}
-trap cleanup EXIT
-
-# Defensive pre-clean: a previous killed run may have leaked a box under our
-# fixed name, and every subsequent create would 409 on it. Wait for the delete
-# to land (it is async) before creating.
-stale_id=$(resolve_seed_box_id)
-if [[ -n "$stale_id" ]]; then
- log "pre-cleaning stale seed box $stale_id (name $SEED_BOX_NAME)"
- hogland box delete "$stale_id" >/dev/null 2>&1 || true
- gone_deadline=$(( $(date +%s) + 120 ))
- while [[ -n "$(resolve_seed_box_id)" ]]; do
- if [[ "$(date +%s)" -ge "$gone_deadline" ]]; then
- log "FAIL: stale seed box $stale_id still present 2m after delete"
- exit 1
- fi
- sleep 5
- done
- log "stale seed box gone"
-fi
-
-# Cold boot from the base (no --snapshot-id), so cpus/mem/disk are required and
-# fix the snapshot's machine shape to the task sandbox shape. --no-connect keeps
-# create non-interactive. --name gives teardown a handle that survives never
-# learning the id.
-# When BOX_MEM_BOOT_MIB is set (and below the cap), the box boots small and
-# hot-adds guest RAM up to BOX_MEM_MIB via virtio-mem; the golden snapshot pins
-# that geometry, so every restored task box inherits it. Blank => fixed-size.
-mem_boot_args=()
-if [[ -n "$BOX_MEM_BOOT_MIB" ]]; then
- mem_boot_args+=(--memory-boot-mib "$BOX_MEM_BOOT_MIB")
- log "creating seed box (cold boot, $BOX_CPUS cpu / boot ${BOX_MEM_BOOT_MIB}->cap $BOX_MEM_MIB MiB / $BOX_DISK_GIB GiB)"
-else
- log "creating seed box (cold boot, $BOX_CPUS cpu / $BOX_MEM_MIB MiB / $BOX_DISK_GIB GiB)"
-fi
-create_rc=0
-# --kind posthog-tasks-golden: an UNREGISTERED kind carries no server-side idle
-# TTL, so the box is never reaped mid-bake. A registered kind like `ci` expires
-# at max(LastUsedAt, CreatedAt)+2h, and only API calls bump LastUsedAt — this
-# bake is all SSH, so a 30m-create + 90m-bake would exceed 2h and get reaped.
-# The leak-on-cancel that no-TTL reintroduces is covered by the workflow's
-# always() teardown step, which deletes both boxes by name.
-# --access-type ssh-private: keep the box off the public internet; the runner
-# reaches it over the tailnet (default ssh-public DNATs a public port).
-# --disk-mbps 0 --disk-iops 0: match hogland's own bakes (CLI defaults 125MB/s /
-# 3000 IOPS otherwise).
-# --timeout 30m: a cold 64 GiB box may wait on Karpenter to provision a node.
-box_json=$(
- hogland box create \
- --ssh-key "${SSH_KEY}.pub" \
- --name "$SEED_BOX_NAME" \
- --kind posthog-tasks-golden \
- --access-type ssh-private \
- --cpus "$BOX_CPUS" \
- --memory-mib "$BOX_MEM_MIB" \
- ${mem_boot_args[@]+"${mem_boot_args[@]}"} \
- --disk-gib "$BOX_DISK_GIB" \
- --disk-mbps 0 \
- --disk-iops 0 \
- --no-connect \
- --timeout 30m
-) || create_rc=$?
-SEED_BOX_ID=$(printf '%s' "$box_json" | jq -r '.id // empty' 2>/dev/null || true)
-if [[ "$create_rc" -ne 0 ]]; then
- log "FAIL: box create exited $create_rc"
- printf '%s\n' "$box_json" >&2
- exit 1
-fi
-ssh_cmd=$(printf '%s' "$box_json" | jq -r '.ssh_command // empty')
-if [[ -z "$ssh_cmd" ]]; then
- log "FAIL: box create returned no ssh_command; cannot reach seed box"
- printf '%s\n' "$box_json" >&2
- exit 1
-fi
-log "seed box $SEED_BOX_ID ssh_command: $ssh_cmd"
-
-# Rebuild the ssh invocation from the returned command, injecting our key +
-# non-interactive options right after the `ssh` word (mirrors smoke-golden.sh).
-# IdentitiesOnly pins to our ephemeral key; BatchMode never prompts; accept-new
-# trusts the fresh host key silently. ServerAlive* guards the ONE long-lived
-# session the whole bake runs over: at 30s x 10 it tolerates a ~5m stall before
-# dropping. A loaded 4-vCPU box building git from source can stall well past the
-# old 15s x 3 (45s) window, which would drop the entire bake mid-step.
-read -r -a ssh_parts <<<"$ssh_cmd"
-ssh_opts=(-i "$SSH_KEY" -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=10)
-ssh_base=("${ssh_parts[0]}" "${ssh_opts[@]}" "${ssh_parts[@]:1}")
-
-log "waiting for ssh reachability (up to 5m)"
-ssh_deadline=$(( $(date +%s) + 300 ))
-until timeout 20 "${ssh_base[@]}" -o ConnectTimeout=5 true 2>/dev/null; do
- if [[ "$(date +%s)" -ge "$ssh_deadline" ]]; then
- log "FAIL: ssh never became reachable within 5m"
- exit 1
- fi
- sleep 5
-done
-log "ssh reachable"
-
-# Stream one local file to a path in the box over SSH, chmod it, and verify the
-# byte count landed. The box's ssh user is `hog` (not root), so the write goes
-# through passwordless sudo — target paths like /opt/posthog/bin are root-owned.
-# `sudo tee` is the delivery primitive; no scp/sftp, no public host. Usage:
-# deliver <local-path> <box-path> <mode>
-deliver() {
- local src="$1" dst="$2" mode="$3" dir size remote_size
- dir=$(dirname "$dst")
- if ! "${ssh_base[@]}" "sudo mkdir -p '$dir' && sudo tee '$dst' >/dev/null && sudo chmod '$mode' '$dst'" < "$src"; then
- log "FAIL: could not deliver $src -> $dst"
- exit 1
- fi
- size=$(wc -c < "$src" | tr -d '[:space:]')
- remote_size=$("${ssh_base[@]}" "wc -c < '$dst' | tr -d '[:space:]'" 2>/dev/null || echo "")
- if [[ "$size" != "$remote_size" ]]; then
- log "FAIL: $dst is $remote_size bytes in box, expected $size"
- exit 1
- fi
- log "delivered $dst ($size bytes, mode $mode)"
-}
-
-# Guards + cpu sampler land at their final paths with mode 0755, so setup-golden.sh
-# only has to `test -x` them. The skills tarball, installer, and setup script stage
-# under /tmp for setup-golden.sh to consume.
-deliver "$GIT_GUARD" /opt/posthog/bin/git 0755
-deliver "$GH_GUARD" /opt/posthog/bin/gh 0755
-deliver "$HOGLI_SHIM" /opt/posthog/bin/hogli 0755
-deliver "$CPU_SAMPLER" /usr/local/bin/posthog-cpu-billing-sampler 0755
-deliver "$SKILLS_TARBALL" /tmp/golden-skills.tar.gz 0644
-deliver "$INSTALL_SKILLS" /tmp/install-skills.sh 0755
-deliver "$SETUP_SCRIPT" /tmp/setup-golden.sh 0755
-
-# Run setup-golden.sh in the box. Its `set -euo pipefail` plus this explicit exit
-# check are our success gate: a failing step returns non-zero and we never reach
-# `box snapshot`. AGENT_VERSION / SKILLS_TARBALL / INSTALL_SKILLS reach it as env.
-log "running setup-golden.sh in the box"
-# %q-escape AGENT_VERSION before splicing it into the remote command string --
-# it comes from an npm registry response (see the workflow's "Resolve
-# published agent version" step), and unescaped single-quote interpolation
-# would let a value containing a quote break out of the assignment and run
-# arbitrary commands in the box.
-printf -v agent_version_escaped '%q' "$AGENT_VERSION"
-setup_rc=0
-# The ssh user is `hog`; setup-golden.sh installs apt packages, writes
-# /etc/environment and the systemd drop-in, and restarts hogpanion — all root
-# work — so run it through passwordless sudo. `sudo env VAR=...` sets the child
-# env explicitly rather than relying on sudo's implied SETENV for leading
-# assignments (which a stricter sudoers policy would strip).
-"${ssh_base[@]}" \
- "sudo env AGENT_VERSION=$agent_version_escaped SKILLS_TARBALL=/tmp/golden-skills.tar.gz INSTALL_SKILLS=/tmp/install-skills.sh bash /tmp/setup-golden.sh" \
- || setup_rc=$?
-if [[ "$setup_rc" -ne 0 ]]; then
- log "FAIL: setup-golden.sh exited $setup_rc in the box; not snapshotting"
- exit 1
-fi
-log "setup-golden.sh completed in the box"
-
-# Snapshot the seed box, then point the candidate alias at the new snapshot id.
-# `box snapshot` pauses the box and persists it to S3, emitting the record JSON.
-log "snapshotting seed box $SEED_BOX_ID"
-snap_json=$(hogland box snapshot "$SEED_BOX_ID")
-SNAP_ID=$(printf '%s' "$snap_json" | jq -r '.id // empty')
-if [[ -z "$SNAP_ID" ]]; then
- log "FAIL: box snapshot returned no id"
- printf '%s\n' "$snap_json" >&2
- exit 1
-fi
-log "pointing alias $ALIAS at snapshot $SNAP_ID"
-hogland snapshot alias "$SNAP_ID" "$ALIAS"
-
-log "bake complete: $SNAP_ID -> alias $ALIAS"
-printf 'BAKED_SNAPSHOT_ID=%s\n' "$SNAP_ID"
diff --git a/products/tasks/backend/sandbox/images/hogland/setup-golden.sh b/products/tasks/backend/sandbox/images/hogland/setup-golden.sh
deleted file mode 100755
index 4c3259779cf2..000000000000
--- a/products/tasks/backend/sandbox/images/hogland/setup-golden.sh
+++ /dev/null
@@ -1,454 +0,0 @@
-#!/usr/bin/env bash
-# First-boot setup for the PostHog *tasks* golden snapshot.
-#
-# `bake-golden.sh` runs this once as root inside a fresh seed box (over SSH),
-# then `box snapshot`s the result and points an alias at it.
-# `HoglandSandbox.create` restores every task sandbox from that alias, so a
-# rebuild here is the hogland equivalent of publishing a new base image tag.
-#
-# The steps here reconstruct products/tasks/backend/sandbox/images/Dockerfile.sandbox-base
-# over exec. That Dockerfile stays the source of truth for what a task sandbox
-# contains; keep the pins below in sync with its ARGs.
-#
-# Content delivery (the crux): the CI runner renders the skills itself with a
-# database (the same `hogli build:skills` mechanism the Modal image build uses)
-# and merges in the context-mill skills, then `bake-golden.sh` delivers the whole
-# set into the box over SSH. Nothing is fished out of a published image. Content
-# arrives two ways, both from `bake-golden.sh` before this script runs:
-# * small, fixed posthog-owned files (the git/gh guards, the cpu sampler) are
-# streamed to their target paths with mode 0755; this script only verifies
-# them (`test -x`), so they are ready before the checks below.
-# * the rendered skills tarball (SKILLS_TARBALL) and the shared installer
-# (INSTALL_SKILLS, the same install-skills.sh the image uses) are streamed in
-# too. This script extracts the tarball and runs the installer, so the golden
-# lands the SAME merged, rendered skills at the SAME paths the image uses.
-# `@posthog/agent` is npm-installed here at AGENT_VERSION (default latest; the
-# workflow resolves and pins an exact version), decoupled from any image.
-#
-# Success detection is the caller's: `bake-golden.sh` runs this over SSH and
-# checks its exit code before snapshotting. `set -e` below makes a failing step
-# abort with non-zero, so a broken bake never reaches `box snapshot`.
-#
-# Required env (set by bake-golden.sh over SSH):
-# SKILLS_TARBALL — path in the box to the rendered-skills tarball (.tar.gz)
-# INSTALL_SKILLS — path in the box to install-skills.sh
-# AGENT_VERSION — @posthog/agent version to install (default: latest)
-
-set -euo pipefail
-export DEBIAN_FRONTEND=noninteractive
-
-: "${SKILLS_TARBALL:?SKILLS_TARBALL is required (path in-box to the rendered-skills tarball)}"
-: "${INSTALL_SKILLS:?INSTALL_SKILLS is required (path in-box to install-skills.sh)}"
-AGENT_VERSION="${AGENT_VERSION:-latest}"
-
-# Free-RAM scrub (see scrub_free_ram): leave this much RAM untouched so the scrub
-# never races the OOM killer. Tunable for a differently-sized seed box.
-SCRUB_HEADROOM_MIB="${SCRUB_HEADROOM_MIB:-2048}"
-# Off by default: dropping reclaimable page cache would let the scrub zero even
-# more frames, but it evicts the warm tool-binary cache that keeps restores fast.
-# Set to 1 only if a smaller image matters more than restore warmth.
-SCRUB_DROP_PAGECACHE="${SCRUB_DROP_PAGECACHE:-0}"
-
-# --- Version pins, mirrored from Dockerfile.sandbox-base ARGs -----------------
-GIT_VERSION=2.49.1
-GIT_SHA256=310831de967f1c8c5e8ff55f92807dea89f83dc3d3d2a5d16c209bd01a31def1
-UV_VERSION=0.12.13
-RUFF_VERSION=0.14.11
-TY_VERSION=0.0.29
-GH_CLI_VERSION=2.97.0
-AGENTSH_TAG=v0.18.3
-AGENTSH_SHA256_AMD64=4ac486eea1e10600c29078a7a992d2067774edfb66be1318a2acf1fcf8b6d774
-AGENTSH_SHA256_ARM64=d1393a27943d207442ea077b1d36d9561a8af5613e7b67d9f7d4fafd00626c6b
-RTK_VERSION=0.43.0
-RTK_SHA256_AMD64=ff8a1e7766496e175291a85aeca1dc97c9ff6df33e51e5893d1fbc78fea2a609
-RTK_SHA256_ARM64=5519f7ca12e5c143a609f0d28a0a77b97413a8dce31c2681f1a41c24519a8731
-
-APT_PACKAGES="curl wget git vim nano tree htop unzip zip jq \
-build-essential pkg-config musl \
-python3 python3-pip python3-venv python3-dev \
-sqlite3 postgresql-client mysql-client redis-tools \
-libssl-dev libcurl4-gnutls-dev libexpat1-dev libffi-dev libbz2-dev \
-libreadline-dev libsqlite3-dev libncursesw5-dev xz-utils tk-dev \
-libxml2-dev libxmlsec1-dev zlib1g-dev \
-ca-certificates gnupg sudo"
-
-# Dockerfile ENVs, made visible to every login/exec process. /etc/environment
-# covers PAM sessions; the systemd drop-in covers the box's agent daemon, whose
-# services do not read /etc/environment. Per-box create(env=...) values reach
-# exec children through the adapter's per-exec env, not this drop-in.
-STATIC_ENV_PATH="/opt/posthog/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
-
-log() { printf '[setup-golden] %s\n' "$*" >&2; }
-
-# --- curl with retries (transient reset must not fail the whole bake) ---------
-fetch() {
- # fetch <url> <out>
- curl -fsSL --retry 5 --retry-all-errors --retry-max-time 60 --connect-timeout 10 -o "$2" "$1"
-}
-
-# --- Free-RAM scrub: shrink the snapshot's stored/faulted memory image --------
-# The hogland snapshot store elides every all-zero 4 MiB chunk: a zero chunk is
-# served as a zero page on restore, never stored in S3, never uploaded, never
-# faulted. But Linux never zeroes a page it frees, so every frame the guest
-# dirtied during this ~90-min bake (apt, npm, freed heap) keeps its stale
-# non-zero content forever — mempack stores and uploads it and each restore
-# faults it from S3. Writing zeros over free RAM converts those frames to zero,
-# so mempack collapses them to SrcZero (not stored, not uploaded, not faulted).
-#
-# This is a pure optimization, so it fails SOFT: any error only logs a warning
-# and returns 0, and the C helper raises its own oom_score_adj so memory pressure
-# kills the scrub first, never the bake. Runs LAST, after all other guest writes,
-# so no later step re-dirties the frames it just zeroed.
-scrub_free_ram() {
- command -v cc >/dev/null 2>&1 || { log "WARN: free-RAM scrub skipped: no cc"; return 0; }
-
- local src bin cc_log
- src="$(mktemp --suffix=.c)"
- bin="$(mktemp)"
- cc_log="$(mktemp)"
-
- # Self-contained C: mmap N bytes of anonymous memory, write zeros through every
- # page, then release it. Anonymous pages fault in zero-filled, so each backing
- # frame ends up all-zero and is freed still zeroed at exit. C (no GC/interpreter)
- # is the most reliable way to touch every page without leaving its own churn.
- cat > "$src" <<'ZEROFREE_C'
-#include <fcntl.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/mman.h>
-#include <unistd.h>
-
-int main(int argc, char **argv) {
- if (argc != 2) {
- fprintf(stderr, "usage: %s <bytes>\n", argv[0]);
- return 2;
- }
-
- unsigned long long bytes = strtoull(argv[1], NULL, 10);
- if (bytes == 0) {
- return 0;
- }
-
- /* Die first under memory pressure: never let the OOM killer pick the bake
- * or hogpanion over this optional scrub. Best effort — proceed if it fails. */
- int fd = open("/proc/self/oom_score_adj", O_WRONLY);
- if (fd >= 0) {
- ssize_t w = write(fd, "1000\n", 5);
- (void)w;
- close(fd);
- }
-
- size_t len = (size_t)bytes;
- void *p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- if (p == MAP_FAILED) {
- perror("mmap");
- return 1;
- }
-
- /* Fault in and zero every page. Anonymous pages fault in zero-filled, so
- * this leaves each backing frame all-zero. After we exit, the kernel frees
- * them still zeroed — exactly what mempack collapses to SrcZero. */
- memset(p, 0, len);
-
- munmap(p, len);
- return 0;
-}
-ZEROFREE_C
-
- if ! cc -O2 -o "$bin" "$src" 2>"$cc_log"; then
- log "WARN: free-RAM scrub skipped: compile failed: $(tr '\n' ' ' < "$cc_log")"
- rm -f "$src" "$bin" "$cc_log"
- return 0
- fi
-
- # Optional, off by default: drop clean reclaimable page cache so the scrub can
- # zero those frames too. It evicts the warm tool-binary cache, so it is gated.
- if [ "$SCRUB_DROP_PAGECACHE" = "1" ] && [ -w /proc/sys/vm/drop_caches ]; then
- sync
- echo 1 > /proc/sys/vm/drop_caches 2>/dev/null || log "WARN: drop_caches failed; continuing"
- log "dropped reclaimable page cache before scrub (SCRUB_DROP_PAGECACHE=1)"
- fi
-
- # MemFree, not MemAvailable: MemAvailable counts reclaimable page cache we want
- # to KEEP resident for warm restores. Touching only genuinely-free frames avoids
- # evicting that cache. MemFree is in kB.
- local memfree_kb headroom_bytes target_bytes
- memfree_kb="$(awk '/^MemFree:/ {print $2; exit}' /proc/meminfo)"
- if [ -z "$memfree_kb" ]; then
- log "WARN: free-RAM scrub skipped: could not read MemFree"
- rm -f "$src" "$bin" "$cc_log"
- return 0
- fi
- headroom_bytes=$(( SCRUB_HEADROOM_MIB * 1024 * 1024 ))
- target_bytes=$(( memfree_kb * 1024 - headroom_bytes ))
- if [ "$target_bytes" -le 0 ]; then
- log "free-RAM scrub skipped: MemFree ${memfree_kb} kB is within the ${SCRUB_HEADROOM_MIB} MiB headroom"
- rm -f "$src" "$bin" "$cc_log"
- return 0
- fi
-
- log "free-RAM scrub: zeroing ${target_bytes} bytes (MemFree ${memfree_kb} kB, headroom ${SCRUB_HEADROOM_MIB} MiB)"
- if "$bin" "$target_bytes"; then
- log "free-RAM scrub done"
- else
- # A non-zero exit (mmap fail) or an OOM-kill signal lands here. Soft-fail.
- log "WARN: free-RAM scrub did not complete cleanly (rc $?); continuing"
- fi
-
- rm -f "$src" "$bin" "$cc_log"
- return 0
-}
-
-log "apt packages"
-apt-get update
-# shellcheck disable=SC2086 # word-splitting the package list is intended
-apt-get install -y --no-install-recommends $APT_PACKAGES
-rm -rf /var/lib/apt/lists/*
-
-log "git ${GIT_VERSION} from source"
-fetch "https://www.kernel.org/pub/software/scm/git/git-${GIT_VERSION}.tar.xz" /tmp/git.tar.xz
-echo "${GIT_SHA256} /tmp/git.tar.xz" | sha256sum -c -
-mkdir /tmp/git
-tar -xf /tmp/git.tar.xz -C /tmp/git --strip-components=1
-make -C /tmp/git prefix=/usr NO_GETTEXT=YesPlease NO_TCLTK=YesPlease -j"$(nproc)" all
-make -C /tmp/git prefix=/usr NO_GETTEXT=YesPlease NO_TCLTK=YesPlease install
-rm -rf /tmp/git /tmp/git.tar.xz
-git --version
-# Materialise `git help -a` before matching. Piping it straight into `grep -q`
-# lets grep close the pipe on the first match, so git dies with SIGPIPE (141)
-# and pipefail aborts the whole setup. Capture first, then grep the string.
-git_help_all="$(git help -a)"
-grep -q '[[:space:]]backfill' <<<"$git_help_all"
-
-log "node 24"
-# The rootfs bakes node 22 into /usr/local/bin, which precedes /usr/bin (where
-# nodesource installs 24) on STATIC_ENV_PATH and in sudo's secure_path. Remove
-# the baked copy first, or `npm install @posthog/agent` and the runtime resolve
-# node 22, not 24.
-rm -f /usr/local/bin/node /usr/local/bin/npm /usr/local/bin/npx /usr/local/bin/corepack
-curl -fsSL --retry 5 --retry-all-errors --retry-max-time 60 --connect-timeout 10 \
- https://deb.nodesource.com/setup_24.x | bash -
-apt-get install -y --no-install-recommends nodejs
-rm -rf /var/lib/apt/lists/*
-node_version="$(node -v)"
-case "$node_version" in
- v24.*) log "node ${node_version}" ;;
- *) echo "expected node v24 after nodesource install, got ${node_version}" >&2; exit 1 ;;
-esac
-
-log "npm globals"
-npm install -g yarn pnpm typescript ts-node nodemon
-
-log "uv ${UV_VERSION} + ruff ${RUFF_VERSION} + ty ${TY_VERSION}"
-# The Dockerfile takes uv from its pinned official image; without a container
-# runtime we pin the same version's release tarball and verify its published
-# checksum.
-uv_arch="$(uname -m)"
-uv_asset="uv-${uv_arch}-unknown-linux-gnu.tar.gz"
-uv_base="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}"
-fetch "${uv_base}/${uv_asset}" /tmp/uv.tar.gz
-fetch "${uv_base}/${uv_asset}.sha256" /tmp/uv.tar.gz.sha256
-(cd /tmp && echo "$(cut -d' ' -f1 uv.tar.gz.sha256) uv.tar.gz" | sha256sum -c -)
-tar -xzf /tmp/uv.tar.gz -C /tmp
-install -m 755 "/tmp/uv-${uv_arch}-unknown-linux-gnu/uv" "/tmp/uv-${uv_arch}-unknown-linux-gnu/uvx" /usr/local/bin/
-rm -rf /tmp/uv.tar.gz /tmp/uv.tar.gz.sha256 "/tmp/uv-${uv_arch}-unknown-linux-gnu"
-UV_TOOL_BIN_DIR=/usr/local/bin uv tool install "ruff==${RUFF_VERSION}"
-UV_TOOL_BIN_DIR=/usr/local/bin uv tool install "ty==${TY_VERSION}"
-ruff --version
-ty --version
-
-log "gh CLI ${GH_CLI_VERSION}"
-gh_arch="$(dpkg --print-architecture)"
-fetch "https://github.com/cli/cli/releases/download/v${GH_CLI_VERSION}/gh_${GH_CLI_VERSION}_linux_${gh_arch}.deb" /tmp/gh.deb
-dpkg -i /tmp/gh.deb
-rm /tmp/gh.deb
-
-log "agentsh ${AGENTSH_TAG}"
-agentsh_version="${AGENTSH_TAG#v}"
-agentsh_arch="$(dpkg --print-architecture)"
-case "$agentsh_arch" in
- amd64) agentsh_sha256="$AGENTSH_SHA256_AMD64" ;;
- arm64) agentsh_sha256="$AGENTSH_SHA256_ARM64" ;;
- *) echo "Unsupported architecture for agentsh: $agentsh_arch" >&2; exit 1 ;;
-esac
-fetch "https://github.com/canyonroad/agentsh/releases/download/${AGENTSH_TAG}/agentsh_${agentsh_version}_linux_${agentsh_arch}.deb" /tmp/agentsh.deb
-echo "${agentsh_sha256} /tmp/agentsh.deb" | sha256sum -c -
-dpkg -i /tmp/agentsh.deb
-rm /tmp/agentsh.deb
-agentsh --version
-mkdir -p /var/lib/agentsh/sessions /var/lib/agentsh/quarantine /var/log/agentsh
-chmod 777 /var/lib/agentsh /var/lib/agentsh/sessions /var/lib/agentsh/quarantine /var/log/agentsh
-
-log "rtk ${RTK_VERSION}"
-rtk_arch="$(dpkg --print-architecture)"
-case "$rtk_arch" in
- amd64) rtk_asset="rtk-x86_64-unknown-linux-musl.tar.gz"; rtk_sha256="$RTK_SHA256_AMD64" ;;
- arm64) rtk_asset="rtk-aarch64-unknown-linux-gnu.tar.gz"; rtk_sha256="$RTK_SHA256_ARM64" ;;
- *) echo "Unsupported architecture for rtk: $rtk_arch" >&2; exit 1 ;;
-esac
-fetch "https://github.com/rtk-ai/rtk/releases/download/v${RTK_VERSION}/${rtk_asset}" /tmp/rtk.tar.gz
-echo "${rtk_sha256} /tmp/rtk.tar.gz" | sha256sum -c -
-tar -xzf /tmp/rtk.tar.gz -C /usr/local/bin rtk
-rm /tmp/rtk.tar.gz
-rtk --version
-
-# --- Agent SDK: @posthog/agent, npm-installed at AGENT_VERSION ----------------
-# Mirrors Dockerfile.sandbox-base's `npm install @posthog/agent@${AGENT_VERSION}`
-# into /scripts. The workflow resolves and pins an exact published version so the
-# golden's agent-server is reproducible; a hand run defaults to latest.
-log "@posthog/agent@${AGENT_VERSION} in /scripts"
-mkdir -p /scripts
-(cd /scripts && npm init -y && npm install "@posthog/agent@${AGENT_VERSION}")
-test -x /scripts/node_modules/.bin/agent-server
-
-# --- Skills: install the runner-rendered set from the delivered tarball -------
-# The CI runner renders the skills with a database (hogli build:skills) and merges
-# in the context-mill skills, exactly as the Modal image build does, then delivers
-# the result as SKILLS_TARBALL. Extract it and run the SAME install-skills.sh the
-# image uses (delivered as INSTALL_SKILLS), so the golden lands the merged, rendered
-# skills at the SAME paths: /scripts/plugins/posthog/skills, /root/.agents/skills,
-# /root/.claude/skills, plus /scripts/plugins/posthog/plugin.json. install-skills.sh
-# keys off $HOME, so pin it to /root (this script runs as root in the box).
-log "skills: install runner-rendered set from ${SKILLS_TARBALL}"
-test -f "$SKILLS_TARBALL" || { echo "SKILLS_TARBALL ${SKILLS_TARBALL} not found in box" >&2; exit 1; }
-test -f "$INSTALL_SKILLS" || { echo "INSTALL_SKILLS ${INSTALL_SKILLS} not found in box" >&2; exit 1; }
-skills_extract_dir="$(mktemp -d)"
-tar -xzf "$SKILLS_TARBALL" -C "$skills_extract_dir"
-HOME=/root bash "$INSTALL_SKILLS" "$skills_extract_dir"
-rm -rf "$skills_extract_dir"
-
-# Fail closed: an empty tarball or a broken install must not ship a golden that
-# silently lost its skills.
-for skills_target in /scripts/plugins/posthog/skills /root/.agents/skills /root/.claude/skills; do
- # Capture then test. `find ... | grep -q .` lets grep close the pipe on the
- # first hit, so find dies with SIGPIPE (141) and pipefail flips this guard.
- found_skill="$(find "$skills_target" -name 'SKILL.md' -type f 2>/dev/null)"
- if [ -z "$found_skill" ]; then
- echo "no SKILL.md found under ${skills_target} after installing ${SKILLS_TARBALL}" >&2
- exit 1
- fi
-done
-
-log "guards + cpu sampler (delivered over ssh by bake-golden.sh)"
-# The git/gh guards and the cpu sampler are streamed to their target paths with
-# mode 0755 by bake-golden.sh before this script runs. The final verify below
-# fails the bake if a hand run omitted them.
-test -x /opt/posthog/bin/git
-test -x /opt/posthog/bin/gh
-test -x /opt/posthog/bin/hogli
-test -x /usr/local/bin/posthog-cpu-billing-sampler
-
-log "git identity"
-git config --global user.email "[email protected]"
-git config --global user.name "PostHog Desktop"
-
-log "static env (/etc/environment + agent-daemon systemd drop-in)"
-mkdir -p /tmp/workspace
-cat > /etc/environment <<EOF
-DEBIAN_FRONTEND="noninteractive"
-TZ="UTC"
-GH_TELEMETRY="false"
-AGENTSH_SERVER="http://127.0.0.1:18080"
-IS_SANDBOX="1"
-PYTHONPATH="/tmp/workspace"
-PATH="${STATIC_ENV_PATH}"
-EOF
-# Lay down the agent-daemon drop-in so exec processes inherit the container-style
-# env, then restart the daemon so the snapshot captures a hogpanion already
-# re-exec'd with the new env.
-if [ -d /etc/systemd/system ]; then
- dropin_dir=/etc/systemd/system/hogpanion.service.d
- mkdir -p "$dropin_dir"
- # No EnvironmentFile= here. A per-box /etc/hogbox-env would let an unreserved
- # key such as PATH override these guard vars (EnvironmentFile settings win
- # over Environment= regardless of order), routing hog-exec children through
- # /usr/bin/git instead of the wrapped /opt/posthog/bin/git. Per-box env still
- # reaches exec children through the adapter's per-exec env, not this drop-in.
- # HOME/USER/LOGNAME: hogpanion runs as root with no User= and hands exec
- # children bare os.Environ(), so nothing sets HOME. Skills resolve
- # $HOME/.agents/skills and git reads /root/.gitconfig; without HOME a set -u
- # task step dies. Pin them to root so exec children inherit a usable home.
- cat > "$dropin_dir/posthog-env.conf" <<EOF
-[Service]
-Environment="DEBIAN_FRONTEND=noninteractive"
-Environment="TZ=UTC"
-Environment="GH_TELEMETRY=false"
-Environment="AGENTSH_SERVER=http://127.0.0.1:18080"
-Environment="IS_SANDBOX=1"
-Environment="PYTHONPATH=/tmp/workspace"
-Environment="PATH=${STATIC_ENV_PATH}"
-Environment="HOME=/root"
-Environment="USER=root"
-Environment="LOGNAME=root"
-EOF
- systemctl daemon-reload
- # daemon-reload does NOT re-exec a running unit, so the drop-in's new
- # Environment= only reaches hogpanion on its next restart. Without a restart
- # the snapshot freezes the OLD env, and restored task boxes' hog-exec children
- # lack IS_SANDBOX=1, the /opt/posthog/bin-first PATH (git/gh guards), and
- # PYTHONPATH. This script runs under hogpanion's cgroup, so a direct restart
- # would kill it mid-bake; fire the restart from a detached transient unit and
- # then poll until the daemon is back and reporting ready before returning.
- if systemctl cat hogpanion.service >/dev/null 2>&1; then
- old_pid="$(systemctl show hogpanion.service -p MainPID --value 2>/dev/null || echo 0)"
- systemd-run --collect --unit=hogpanion-reload --on-active=2 \
- systemctl restart hogpanion.service
- # Gate on hogpanion reporting READY, not just on a new MainPID. hogpanion
- # is Type=simple: the pid changes at execve, but `ready` only flips true
- # after runBoot finishes, so a pid-only wait can snapshot a half-booted
- # daemon. Require a NEW pid (proves the restart fired, not the old still-
- # ready daemon) AND the status endpoint reporting ready:true.
- ready=0
- for _ in $(seq 1 90); do
- new_pid="$(systemctl show hogpanion.service -p MainPID --value 2>/dev/null || echo 0)"
- if systemctl is-active --quiet hogpanion.service \
- && [ -n "$new_pid" ] && [ "$new_pid" != "0" ] && [ "$new_pid" != "$old_pid" ] \
- && curl -sf http://localhost:7682/status 2>/dev/null | jq -e '.ready == true' >/dev/null 2>&1; then
- ready=1
- break
- fi
- sleep 1
- done
- if [ "$ready" != "1" ]; then
- echo "hogpanion did not restart and report ready with the env drop-in (old pid ${old_pid})" >&2
- systemctl status hogpanion.service --no-pager || true
- exit 1
- fi
- log "hogpanion restarted (pid ${old_pid} -> ${new_pid}) and reported ready with the env drop-in"
- else
- log "hogpanion.service not present; skipping restart (drop-in applies on next start)"
- fi
-fi
-
-log "verify"
-python3 --version
-node --version
-npm --version
-gh --version
-rtk --version
-agentsh --version
-test -x /scripts/node_modules/.bin/agent-server
-test -x /opt/posthog/bin/git
-test -x /opt/posthog/bin/gh
-test -x /opt/posthog/bin/hogli
-
-# LAST step: strip the ephemeral CI ssh key so it does not persist into the
-# snapshot and therefore into every restored task box. Production restores with
-# access_type: none, and hogpanion returns early on an empty key list, so a
-# non-empty authorized_keys from the bake would survive into live boxes. Truncate
-# every user's authorized_keys the bake could have populated. This is the final
-# action: no SSH step runs after it (the current session stays open; box snapshot
-# is driven over the hogplane API, not SSH).
-for ak in /home/hog/.ssh/authorized_keys /root/.ssh/authorized_keys; do
- [ -f "$ak" ] && : >"$ak"
-done
-log "truncated CI ssh authorized_keys"
-
-# Absolute last guest write before bake-golden.sh calls `box snapshot` (over the
-# hogplane API, not SSH — no further guest activity runs after this returns).
-# Zeroing free RAM here, after every other step, means no later step re-dirties
-# the frames it just zeroed. See scrub_free_ram for why (zero-page collapse).
-scrub_free_ram
-
-log "setup-golden complete"
diff --git a/products/tasks/backend/sandbox/images/hogland/smoke-golden.sh b/products/tasks/backend/sandbox/images/hogland/smoke-golden.sh
deleted file mode 100755
index 542027af52ed..000000000000
--- a/products/tasks/backend/sandbox/images/hogland/smoke-golden.sh
+++ /dev/null
@@ -1,281 +0,0 @@
-#!/usr/bin/env bash
-# smoke-golden.sh — readiness gate for the tasks golden-snapshot rebuild.
-#
-# Boots a smoke box from the posthog-tasks-candidate alias, runs the task
-# sandbox's readiness contract over SSH, and deletes the box. A non-zero exit
-# tells the workflow to skip the promote step, which leaves the live
-# posthog-tasks-default alias pointing at the previous known-good snapshot.
-#
-# The contract asserts the things a task run depends on:
-# a. AGENT-SERVER — @posthog/agent's agent-server binary is present and starts
-# (a golden that baked a broken /scripts install must not promote).
-# b. CLONE + EXEC — a trivial `git clone` of a public repo and a command run
-# inside it work, i.e. the toolchain (git guard, network, exec) is live.
-# c. EXEC-DAEMON ENV — the running hogpanion daemon (whose env hog-exec children
-# inherit) carries the container-style env: IS_SANDBOX=1, a PATH with
-# /opt/posthog/bin first (the git/gh guards), and PYTHONPATH. This reads the
-# daemon's live /proc environ, NOT an SSH login shell — a login shell reads
-# /etc/environment via PAM and would look correct even if the daemon env
-# never picked up the drop-in, certifying a broken golden green.
-#
-# Required env vars (set by the workflow):
-# ALIAS — the posthog-tasks-candidate alias to boot from
-# SSH_KEY — path to the PRIVATE key; the public half is at ${SSH_KEY}.pub
-#
-# On success prints a single parseable line to stdout:
-# SMOKE_HEALTHY_SECONDS=<n> (create→contract-passed wall-clock)
-
-set -euo pipefail
-
-: "${ALIAS:?ALIAS is required}"
-: "${SSH_KEY:?SSH_KEY is required (private key path; pub at \${SSH_KEY}.pub)}"
-
-log() { printf '[tasks-smoke] %s\n' "$*" >&2; }
-
-# Fixed name so a killed prior run's leftover box is found and cleaned rather
-# than leaking a node or colliding on the next create (hogplane 409s on a
-# duplicate name).
-SMOKE_BOX_NAME=golden-smoke-tasks
-SMOKE_BOX_ID=""
-
-# List across ALL kinds, not just ours: hogplane's name-uniqueness check
-# (ensureBoxNameUnique) scans every kind, so match by name only.
-resolve_smoke_box_id() {
- hogland box list 2>/dev/null \
- | jq -r --arg n "$SMOKE_BOX_NAME" 'map(select(.spec.name == $n)) | .[0].id // empty' 2>/dev/null \
- || true
-}
-
-cleanup() {
- if [[ -z "$SMOKE_BOX_ID" ]]; then
- SMOKE_BOX_ID=$(resolve_smoke_box_id)
- fi
- if [[ -n "$SMOKE_BOX_ID" ]]; then
- log "deleting smoke box $SMOKE_BOX_ID"
- hogland box delete "$SMOKE_BOX_ID" >/dev/null 2>&1 || true
- fi
-}
-trap cleanup EXIT
-
-# Defensive pre-clean: a previous killed run may have leaked a box under our
-# fixed name, and every subsequent create would 409 on it. Wait for the delete
-# to land (it is async) before creating.
-stale_id=$(resolve_smoke_box_id)
-if [[ -n "$stale_id" ]]; then
- log "pre-cleaning stale smoke box $stale_id (name $SMOKE_BOX_NAME)"
- hogland box delete "$stale_id" >/dev/null 2>&1 || true
- gone_deadline=$(( $(date +%s) + 120 ))
- while [[ -n "$(resolve_smoke_box_id)" ]]; do
- if [[ "$(date +%s)" -ge "$gone_deadline" ]]; then
- log "FAIL: stale smoke box $stale_id still present 2m after delete"
- exit 1
- fi
- sleep 5
- done
- log "stale smoke box gone"
-fi
-
-log "creating smoke box from alias:$ALIAS"
-create_t0=$(date +%s)
-# Don't let `set -e` abort on a failed create: the box may still exist (a create
-# that timed out waiting for ready left one behind), and the trap needs to run.
-# --name gives teardown a handle that survives never learning the id. No sizing
-# flags — restore inherits cpus/mem/disk from the snapshot.
-create_rc=0
-# --kind posthog-tasks-golden: an UNREGISTERED kind carries no server-side idle
-# TTL, so an all-SSH smoke run is never reaped mid-check. The workflow's
-# always() teardown deletes this box by name on cancel. --access-type
-# ssh-private: keep the box off the public internet; the runner reaches it over
-# the tailnet. --disk-mbps 0 --disk-iops 0: the bake baked the snapshot with
-# uncapped disk, and production restores inherit that — but the CLI applies its
-# own 125MB/s / 3000-IOPS defaults on restore rather than inheriting, so pass 0
-# here too or the smoke measures a throttle production never sees.
-# --timeout 30m: a 64 GiB restore may wait on Karpenter for a node.
-box_json=$(
- hogland box create \
- --snapshot-id "alias:$ALIAS" \
- --ssh-key "${SSH_KEY}.pub" \
- --name "$SMOKE_BOX_NAME" \
- --kind posthog-tasks-golden \
- --access-type ssh-private \
- --disk-mbps 0 \
- --disk-iops 0 \
- --no-connect \
- --timeout 30m
-) || create_rc=$?
-SMOKE_BOX_ID=$(printf '%s' "$box_json" | jq -r '.id // empty' 2>/dev/null || true)
-if [[ "$create_rc" -ne 0 ]]; then
- log "FAIL: box create exited $create_rc"
- printf '%s\n' "$box_json" >&2
- exit 1
-fi
-ssh_cmd=$(printf '%s' "$box_json" | jq -r '.ssh_command // empty')
-if [[ -z "$ssh_cmd" ]]; then
- log "box create returned no ssh_command; cannot reach box for smoke"
- printf '%s\n' "$box_json" >&2
- exit 1
-fi
-log "smoke box $SMOKE_BOX_ID ssh_command: $ssh_cmd"
-
-# Rebuild the ssh invocation from the returned command, injecting our key +
-# non-interactive options right after the `ssh` word. IdentitiesOnly pins the
-# probe to our ephemeral key (a stray agent identity would burn MaxAuthTries
-# first); BatchMode never prompts; accept-new trusts the fresh host key silently.
-read -r -a ssh_parts <<<"$ssh_cmd"
-# ServerAlive* turns a wedged binary or dead session on the box into a dropped
-# connection in ~45s, instead of hanging an assert until the job's 150m ceiling.
-ssh_opts=(-i "$SSH_KEY" -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new -o BatchMode=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=3)
-ssh_base=("${ssh_parts[0]}" "${ssh_opts[@]}" "${ssh_parts[@]:1}")
-
-# Run one SSH assertion under a hard local ceiling, so neither a hung binary nor
-# a stuck TCP session can wedge the job. Usage: ssh_assert <remote-command...>
-ssh_assert() {
- timeout 60 "${ssh_base[@]}" "$@"
-}
-
-log "waiting for ssh reachability (up to 5m)"
-ssh_deadline=$(( $(date +%s) + 300 ))
-until timeout 20 "${ssh_base[@]}" -o ConnectTimeout=5 true 2>/dev/null; do
- if [[ "$(date +%s)" -ge "$ssh_deadline" ]]; then
- log "ssh never became reachable within 5m"
- exit 1
- fi
- sleep 5
-done
-log "ssh reachable"
-
-# (a) agent-server: present and able to report its version. --help/--version
-# exits cleanly without needing task credentials or a running control plane.
-log "asserting agent-server present and starts"
-if ! ssh_assert "test -x /scripts/node_modules/.bin/agent-server"; then
- log "FAIL: /scripts/node_modules/.bin/agent-server missing or not executable"
- exit 1
-fi
-if ! ssh_assert "/scripts/node_modules/.bin/agent-server --version >/dev/null 2>&1 || /scripts/node_modules/.bin/agent-server --help >/dev/null 2>&1"; then
- log "FAIL: agent-server did not start (neither --version nor --help succeeded)"
- exit 1
-fi
-
-# (b) clone + exec: the git guard passes through a real clone, network egress
-# works, and a command runs inside the checkout. POSTHOG_ALLOW_UNSIGNED_GIT is
-# irrelevant here (clone is never blocked); this exercises the everyday path.
-log "asserting trivial clone + exec"
-if ! ssh_assert "set -e; d=\$(mktemp -d); git clone --depth 1 https://github.com/octocat/Hello-World \"\$d/repo\"; test -f \"\$d/repo/README\"; ( cd \"\$d/repo\" && git rev-parse HEAD ); rm -rf \"\$d\""; then
- log "FAIL: trivial clone/exec did not complete"
- exit 1
-fi
-
-# (c) exec-daemon env: the golden must promote only if hogpanion — the daemon
-# whose env hog-exec children inherit — is running with the container-style env
-# from setup-golden.sh's drop-in. We read the DAEMON's live /proc environ, not a
-# login shell: PAM feeds an SSH shell /etc/environment, so a shell would look
-# correct even when the daemon never re-exec'd with the drop-in (the exact bug a
-# missing hogpanion restart causes). Reading another process's /proc/<pid>/environ
-# needs root; the box's ssh user is `hog`, which has passwordless sudo, so the
-# environ read goes through sudo. `systemctl show` of the unit config is the
-# secondary, weaker signal (config loaded, not necessarily applied to the live
-# process). Reaching a real hog-exec child through hogpanion's exec API is the
-# ideal upgrade — see GOLDEN_CI_RUNBOOK.md's known-gaps.
-log "asserting exec-daemon (hogpanion) env carries the container-style env"
-# shellcheck disable=SC2016 # $pid/$environ must expand on the box, not locally
-daemon_env_probe='set -eu
-# Query via sudo. The box runs no system D-Bus daemon, only the systemd
-# root-only private socket, so a non-root systemctl fails with "Failed to
-# connect to bus" and reports a false MainPID=0. The ssh user hog has
-# passwordless sudo, same as the environ read below.
-pid=$(sudo systemctl show hogpanion.service -p MainPID --value 2>/dev/null || echo 0)
-if [ -z "$pid" ] || [ "$pid" = "0" ]; then
- echo "hogpanion has no running MainPID" >&2
- echo "--- hogpanion.service status ---" >&2
- sudo systemctl status hogpanion.service --no-pager >&2 2>&1 || true
- echo "--- hogpanion journal (tail) ---" >&2
- sudo journalctl -u hogpanion.service --no-pager 2>&1 | tail -40 >&2 || true
- exit 1
-fi
-environ=$(sudo cat "/proc/$pid/environ" | tr "\0" "\n")
-printf "%s\n" "$environ" | grep -qx "IS_SANDBOX=1" || { echo "daemon env missing IS_SANDBOX=1" >&2; exit 1; }
-printf "%s\n" "$environ" | grep -q "^PATH=/opt/posthog/bin:" || { echo "daemon PATH does not start with /opt/posthog/bin" >&2; exit 1; }
-printf "%s\n" "$environ" | grep -q "^PYTHONPATH=" || { echo "daemon env missing PYTHONPATH" >&2; exit 1; }
-# HOME must be set: hogpanion hands exec children bare os.Environ(), and skills
-# resolve $HOME/.agents/skills while git reads /root/.gitconfig. Without HOME a
-# set -u task step dies. The drop-in sets HOME=/root.
-printf "%s\n" "$environ" | grep -qx "HOME=/root" || { echo "daemon env missing HOME=/root" >&2; exit 1; }
-# The /opt/posthog/bin-first PATH is what makes git resolve to the guard; confirm
-# the guard is actually there so the guarded path the daemon exposes is real.
-test -x /opt/posthog/bin/git || { echo "/opt/posthog/bin/git missing" >&2; exit 1; }
-grep -q POSTHOG_ALLOW_UNSIGNED_GIT /opt/posthog/bin/git || { echo "/opt/posthog/bin/git is not the git guard" >&2; exit 1; }
-sudo systemctl show hogpanion.service -p Environment -p EnvironmentFiles --no-pager >&2'
-if ! ssh_assert "$daemon_env_probe"; then
- log "FAIL: hogpanion daemon env is not the container-style env (restart likely did not take before snapshot)"
- exit 1
-fi
-
-# (d) exec API: production reaches a task box only through hogpanion's exec API
-# (POST /v1/hogboxes/{id}/exec), never SSH. Assert a trivial command round-trips
-# through it so a golden that works over SSH but not via exec is caught. HOG_HOST
-# + HOG_TOKEN_COMMAND come from the workflow's bake-job env. The request body is
-# {argv, timeout_seconds} — the schema requires `argv` (not `command`). Fail OPEN
-# only when the endpoint is genuinely unreachable (000) — a 404 is what a missing
-# box or a non-owner gets, i.e. a real regression that must not pass green.
-if [[ -n "${HOG_HOST:-}" && -n "${HOG_TOKEN_COMMAND:-}" ]]; then
- log "asserting box reachable via hogpanion exec API"
- exec_token=$(eval "$HOG_TOKEN_COMMAND" 2>/dev/null || true)
- if [[ -z "$exec_token" ]]; then
- log "WARN: could not mint bearer for the exec assertion; relying on SSH assertions"
- else
- exec_out=$(mktemp)
- exec_body='{"argv":["/bin/sh","-c","echo golden-exec-ok"],"timeout_seconds":30}'
- # hogplane matches the auth scheme case-sensitively — "Bearer" (capital
- # B), unlike the lowercase "bearer" GitHub's OIDC token endpoint wants in
- # HOG_TOKEN_COMMAND. A lowercase scheme here gets a 401, which the case
- # below treats as a hard failure.
- # curl already prints "000" via -w on a connection failure, so append
- # nothing on its non-zero exit — `|| echo 000` would concatenate to
- # "000000" and never match the 000) arm below. --max-time caps a hung
- # request so a wedged endpoint can't stall the job. -w also prints
- # num_connects so a 000 can be routed by whether a connection was ever
- # made: 0 => the endpoint was unreachable (a tailnet/infra blip, not the
- # golden's fault) => fail open and rely on the SSH assertions; >=1 with a
- # 000 => we connected but the request hung or was reset (a --max-time
- # stall) => a real failure. So a box that stalls the exec cannot promote
- # itself by forcing a fail-open.
- read -r http_code num_connects <<<"$(curl -sS --connect-timeout 10 --max-time 60 \
- -o "$exec_out" -w '%{http_code} %{num_connects}' \
- -X POST "$HOG_HOST/v1/hogboxes/$SMOKE_BOX_ID/exec" \
- -H "Authorization: Bearer $exec_token" \
- -H 'Content-Type: application/json' \
- -d "$exec_body" 2>/dev/null || true)"
- case "$http_code" in
- 2*)
- if grep -q 'golden-exec-ok' "$exec_out"; then
- log "exec API round-trip ok"
- else
- log "FAIL: exec API returned $http_code but its output lacked the marker"
- cat "$exec_out" >&2 || true
- rm -f "$exec_out"
- exit 1
- fi
- ;;
- 000)
- if [[ "${num_connects:-0}" != "0" ]]; then
- log "FAIL: exec API connected but the request did not complete (num_connects=${num_connects}) — a stall or reset, not fail-open"
- cat "$exec_out" >&2 || true
- rm -f "$exec_out"
- exit 1
- fi
- log "WARN: exec API unreachable (never connected); relying on SSH assertions. Enable once the exec contract is confirmed (see runbook)."
- ;;
- *)
- log "FAIL: exec API POST returned HTTP $http_code"
- cat "$exec_out" >&2 || true
- rm -f "$exec_out"
- exit 1
- ;;
- esac
- rm -f "$exec_out"
- fi
-fi
-
-elapsed=$(( $(date +%s) - create_t0 ))
-log "tasks golden contract passed after ${elapsed}s"
-printf 'SMOKE_HEALTHY_SECONDS=%d\n' "$elapsed"