Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

9.9 KB
Raw Blame

hogbox-preview

Per-PR PostHog preview environments, the layer-agnostic way.

Spins PostHog up inside a disposable box and serves it at a stable URL that survives the box being recreated. Today the box is a hogland hogbox (Firecracker microVM, reachable over the tailnet). It used to be a DigitalOcean droplet (bin/hobby-ci.py), and it may be something else next.

The one idea: layer vs stack

The thing that changes when we swap providers is small and isolated:

  • Layer (backend.py, *_backend.py) — which box: how to provision it, how to run commands / write files in it, what its URL is, how to destroy it. Swapping providers = one new PreviewBackend.
  • Stack (stack.py) — what runs in the box: the PostHog docker-compose recipe (image, override, frontend, migrate, seed, health). It talks to the box only through the backend interface and never changes when the layer changes.
from hogbox_preview import HoglandBackend, PostHogPreviewStack

backend = HoglandBackend(host="https://hogland-dev.hedgehog-kitefin.ts.net", name="preview-pr-123")
url = PostHogPreviewStack(backend, branch="pull/123/head").bring_up()
print(url)  # https://pen-….boxes.hogland.dev.posthog.dev/  (stable across rebuilds)

How it actually works

  1. Restore a warm golden (alias:posthog-preview-golden) — a Firecracker snapshot of PostHog already running on the ghcr.io/posthog/posthog:master image, pg + ClickHouse migrated, demo-data seeded. A warm restore resumes a live, serving PostHog in ~30s (no cold boot, no rebuild). Sizing must match the golden (8 vCPU / 16 GiB / 100 GiB).
  2. Mount the PR's backend (posthog, ee, products) over the image's /code — so the backend runs the PR code with no per-PR image build.
  3. Frontend (only when the PR touches frontend/**): the untrusted build-frontend CI job compiles the PR's frontend/dist on the runner in parallel with the bring-up. Once the box is healthy, the swap-frontend subcommand ships the dist in (chunked past hogplane's 64 MiB write cap), re-runs collectstatic, and recreates web — so the box comes up on the golden's :master SPA first, then swaps to the PR frontend before "ready" is posted. Non-FE PRs skip the swap and keep the :master SPA. Running the swap after bring-up (not inside the single up) is what lets the FE build hide under the longer bring-up instead of serializing before it.
  4. Delta-migrate — only the PR's unapplied migrations on top of the seeded DB (--reset-db if the PR's migrations are incompatible with the baseline).
  5. Sync HogFunction templates - start the CDP service and load destination templates into the restored database before the preview becomes available.
  6. Sync feature flags - run sync_feature_flags so each flag in the PR's constants.tsx that the golden lacks exists and is on. Then warm_team_metadata_cache and warm_flags_cache fill the flags Redis (redis7 database 1), which feature-flags reads. Django and feature-flags share FLAGS_REDIS_URL, and feature-flags uses the stack's object storage for cache misses. A celery worker runs in the preview, without the scheduler, so tasks such as the flag cache refresh that a flag save queues actually run.
  7. Serve + report — the box is HTTP-exposed; the URL is posted to the PR. A Caddy container (static-proxy) takes the exposed port. It serves /static/* from staticfiles/ and sends all other requests to web, so asset requests do not use Django threads. A non-frontend PR copies staticfiles/ out of the image. A file that is not there falls through to Django.
  8. Self-telemetry - the preview sends its own data to its demo project, so its product pages show real data from the preview itself. Analytics: web runs with SELF_CAPTURE=1. posthog-js sends to the preview's own origin, the backend sends to static-proxy, and Caddy routes capture and flag paths to capture and feature-flags. ingestion-general writes the events to ClickHouse. Logs, traces and metrics: web, the temporal worker and the Node services export OTLP to otel-collector. The collector also reads every container's log file. It sends everything to capture-logs with the demo project's token, and ingestion-logs, ingestion-traces and ingestion-metrics write it to ClickHouse. The collector skips the telemetry pipeline's own containers, so their logs do not feed back into themselves.

Driven entirely by the posthog-hogland Python SDK over hogplane's HTTP API — keyless (GitHub OIDC → hogplane token over the tailnet), no hogland CLI binary and no box SSH key.

Pens: the stable URL

The box behind a preview rotates (a fresh restore per push), so the tool keys a pen (hogland's stable identity, docs/PENS.md there) on the deterministic name preview-pr-<n>: get-or-create the pen, restore a fresh box, PATCH current_box_id. The posted URL is https://<pen-id>.<edge>/, which hogland's box-http edge resolves to whatever box currently backs the pen — so a reviewer's link survives every re-push (the box swaps underneath). Requires hogland's pen-id edge routing (shipped in hogland #319).

Only the pen name is deterministic. Each box gets a unique name — a fresh preview-pr-<n>-<6hex> per create attempt, including 5xx retries — because hogland enforces per-owner name uniqueness across all box statuses (a failed placement holds its name for up to an hour), so any reused box name would make a placement retry 409 against its own corpse. Leftover boxes are reaped after the new box comes up and on teardown.

Getting a preview

Previews are opt-in and run on the dev hogland cluster. Add the hogbox-preview label to a same-repo PR, draft or ready, and hogbox-preview-env.yml builds one for any paths. While the label stays on, every push rebuilds the preview. Fork PRs never get one, even when labeled, because the build would run fork code with the hogland token in scope. GitHub cannot filter a label trigger by name, so the workflow dispatches on every labeled, unlabeled, synchronize and reopened event, and a job-level if: skips every PR without the label before a runner starts. Rapid pushes coalesce through a per-PR concurrency group on the deploy job.

Reporting & lifecycle

  • PR comment: a Hogbox preview section in the shared 🤖 CI report comment (.github/scripts/post-hogbox-preview-section.mjs), staged building → ready → failed → torn down, with the URL, login, and what's running.
  • GitHub Deployment: a preview-pr-<n> task in the shared hogbox-preview environment (in_progress → success/failure + URL), so the preview shows in the PR's Deployments UI.
  • Teardown: removing the label destroys box + pen, and so does closing a labeled PR (pr-closed.yml). A daily stale-sweep (cleanup-stale) in hogbox-preview-cleanup.yml reaps previews whose PR closed but slipped through. It is its own workflow because the OIDC mint needs id-token: write, which can't live in the reusable pr-cleanup.yml.

Hibernation & wake (why previews are affordable)

Every preview pen is created with on_idle=hibernate + wake=on-request (both re-asserted on each push). After 30 min idle (the --ttl-seconds window), hogland's reaper snapshots the box to S3 and deletes it — zero node cost while nobody's looking. The stable URL stays live: the next visit hits the box-front edge, which sees a hibernated wake-on-request pen and serves a brief "waking up" interstitial (a polling page for browser navigations, a retry-503 for XHR/asset clients) while it restores the box (30–40s, a warm restore of the ready stack) and repoints the pen. Actively-viewed previews stay awake — edge traffic touches the box, so only a genuinely idle one sleeps. See hogland's docs/PENS.md.

Access is tailnet-only (PostHog VPN) — internal reviewers, no public URL.

CLI

Pure stdlib + the SDK; run with uv run or plain python. Subcommands mirror bin/hobby-ci.py so it drops into the same CI shape.

cd tools/hogbox-preview

# one-shot: provision (pen + box) + bring PostHog up, print the URL. The CI flow
# omits --frontend-dist and swaps the frontend in afterwards (below) so the FE
# build runs in parallel; the flag stays for a single-shot local build.
python -m hogbox_preview --host "$HOG_HOST" --name "preview-pr-$PR" up \
  --branch "pull/$PR/head" [--frontend-dist /path/to/frontend-dist.tgz] [--no-seed]

# deferred frontend swap onto the already-up box (what CI runs once the parallel
# build-frontend job finishes): resolves the live box by pen name, no restore.
python -m hogbox_preview --host "$HOG_HOST" --name "preview-pr-$PR" \
  swap-frontend --frontend-dist /path/to/frontend-dist.tgz

# granular / staged (reuse a box):
python -m hogbox_preview --host "$HOG_HOST" --name "preview-pr-$PR" create
python -m hogbox_preview --host "$HOG_HOST" --name "preview-pr-$PR" health
python -m hogbox_preview --host "$HOG_HOST" --name "preview-pr-$PR" destroy   # box + pen

# cron backstop: reap previews whose PR is closed (needs GITHUB_REPOSITORY + GH_TOKEN)
python -m hogbox_preview --host "$HOG_HOST" cleanup-stale

The golden (and why it's fast)

The image-pull + full migrate is a one-time cost baked into the golden, not paid per-PR. The golden is built on the hogland side by scripts/posthog-preview-setup.sh via snapshot-build (a cold-boot seed — never box snapshot of a restored box, which bakes the source TAP name and won't re-restore) and is left warm (stack running) so restores resume a serving PostHog. It bakes JS_URL="" + wildcard CSRF so any box's edge URL serves without a per-box rewrite. The DB travels with the image it was migrated by — don't reuse a stale golden DB against a newer image; re-bake instead. Timing details + the warm-vs-cold analysis live in STARTUP.md and hogland's docs/CHUNKFS_RESTORE_PERFORMANCE.md.