Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

57.6 KB
Raw Blame

Cloud runs: local setup guide

Quickstart

  1. Create a personal dev GitHub App (see GitHub App below)
  2. Run python manage.py setup_background_agents
  3. Run hogli start
  4. Open Tasks in PostHog and create a task

The setup command is idempotent — re-run it anytime. It writes the dev JWT keys to your .env, creates the Array OAuth application, enables the tasks feature flag for every team, and builds the agent skills bundle.

To trigger runs from Slack (@PostHog <task>) instead of the UI, set up a dev Slack workspace and app once this guide is working — see slack-local-setup-guide.md.

GitHub App

Each engineer needs their own GitHub App. The setup command will print these instructions and offer to open the creation page in your browser, but for reference:

Shortcut: python manage.py create_github_app automates everything below via GitHub's App Manifest flow. It opens the browser with the manifest pre-filled; on the single "Create GitHub App" click it writes the four GITHUB_APP_* values straight to your .env and verifies the key works. Add --org <name> to create it under an organization. The manual steps below remain the fallback / reference.

Permission Access Purpose
Contents Read & Write Read files, create branches, push commits
Pull requests Read & Write Create and update PRs
Metadata Read Required for all GitHub Apps

Optional: Issues (R/W), Workflows (R/W).

Steps:

  1. GitHub → Settings → Developer Settings → GitHub Apps → New GitHub App
  2. Set the Setup URL (NOT the Callback or Homepage URL) to http://localhost:8010/integrations/github/callback
  3. Set a Callback URL — http://localhost:8010/complete/github-link/ works for the personal user-link flow used by Code. Any URL under your localhost is fine; the value just has to be a valid URL since it's required when creating the App.
  4. Set the permissions above
  5. Under "Identifying and authorizing users", check Request user authorization (OAuth) during installation — required for the personal user-link flow
  6. Generate a client secret under "Client secrets" on the App page — this is required (added a couple of releases back). If your local setup stopped working recently, this is most likely what's missing.
  7. Generate a private key
  8. Install the app on your test repositories by going to http://localhost:8010/project/1/integrations/github and installing the GitHub Integration
  9. Add to your .env:
# The OAuth Client ID (starts with Iv1 or Iv23) — NOT the numeric App ID.
# Both fields are visible on the GitHub App settings page; the App ID is the
# small grey number at the top, the Client ID is the labelled field below.
GITHUB_APP_CLIENT_ID=Iv1.xxxxxxxxxxxxxxxx
GITHUB_APP_CLIENT_SECRET=your_client_secret
GITHUB_APP_SLUG=your-app-slug
GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"

The app slug is the URL-friendly name in your App URL (github.com/apps/your-app-slug). Literal \n characters in the private key are fine — they get converted to newlines.

Temporal worker

Temporal and the temporal-django-worker start automatically via phrocs when you run hogli start.

The separate task-management implementation is not registered with the worker. It stops polling closed or merged PRs and caps background runs at two idle checks; pending CI and merge-queue checks do not consume that budget. Pending CI and merge queues have a separate limit of 96 waiting checks (24 hours at the normal cadence). Follow-ups and sandbox-session boundaries reset both budgets, and the counters persist in server-owned TaskRun.state across orchestrator restarts. With the tasks-task-management-durable-ci-checkpoints Temporal patch, queue and budget changes are staged in a generation-protected checkpoint before application. Startup applies any unfinished checkpoint before restoring the queue and counters. Follow-ups wait for their reset and queued message to persist before delivery. Exhausted persistence retries stop the orchestrator instead of continuing with an unsaved budget; a new execution can recover a staged checkpoint. A failure before staging does not create a recoverable database checkpoint. With tasks-task-management-checkpoint-recovery-status, checkpoint read or write failures leave the task run non-terminal so recovery can reattach to its sandbox. tasks-execute-sandbox-reopen-parent-delivery resumes sandbox acknowledgements when the restarted orchestrator attaches.

The process-task workflow defined in products/tasks/backend/temporal/process_task/workflow.py provisions a sandbox, starts an agent inside it, and waits for the agent to finish. The workflow orchestrates these activities:

  1. get_task_processing_context — Loads the TaskRun from the database, validates the GitHub integration and repository, and builds a TaskProcessingContext carrying all the IDs needed by later activities
  2. get_sandbox_for_repository — Creates an OAuth access token, provisions a Docker sandbox (reusing a snapshot if one exists), clones the repository, and stores the sandbox URL in TaskRun.state
  3. start_agent_server — Prepares and starts the agent server inside the sandbox, then polls /health until it responds. Modal uploads one preparation script to install the shell environment hook and GitHub CLI shim. When network enforcement is enabled, that script also installs the agentsh configuration, policy, and environment wrapper, then starts the daemon and creates its session. Files are staged with their required permissions before atomic replacement, and any preparation failure stops the launch. Failed preparation terminates and reaps its agentsh daemon, and captured errors include a bounded daemon log tail when available. The Modal launch preparation finished worker log records upload, installation, daemon/session, and total preparation times in milliseconds. This preparation runs inside the agent_server_invoke latency metric; agent_server_prepare measures credentials and MCP configuration before the sandbox call. The tasks_modal_launch_preparation_latency histogram measures the preparation upload and execution in milliseconds, excluding later launch work and failure diagnostics. Labels identify runtime, boot path, origin product, snapshot use, and COMPLETED or FAILED status; healthy-server reuse emits no preparation sample. Its _count, _sum, and _bucket series support attempt counts, mean duration, and percentiles, with finer buckets between 1 and 10 seconds.
  4. wait_condition — The workflow blocks with a 30-minute inactivity timeout, extended by heartbeat signals from the agent. Exits on a complete_task signal or when no heartbeat arrives within 30 minutes
  5. cleanup_sandbox — Destroys the sandbox container (always runs, even on failure)

The activities live in products/tasks/backend/temporal/process_task/activities/.

Repository clone activities allow six attempts in total. Failed git commands, including RPC errors and repository-not-found responses, retry after 1, 2, 16, 32, and 64 seconds. Explicit authentication failures stop immediately. Each attempt keeps the five-minute timeout and reuses the sandbox and credential. Activity timeouts and infrastructure exceptions use Temporal's default exponential backoff within the same six-attempt limit. The repository_clone_failed worker log records the run, repository, sandbox, clone mode, exit code, and whether a GitHub token was supplied. Its stdout, stderr, and execution error fields redact the supplied token and sandbox command credentials before truncation to 500 characters each; the same diagnostics are attached to the Temporal error.

The agent's finish tool marks a TaskRun terminal and triggers sandbox cleanup. ReviewHog and scout suggestion tasks do not expose it because their callers own session completion. ReviewHog receives and validates each turn's JSON before ending the session, and validation can use multiple turns in the same sandbox.

Credential refresh runs in the background. For workflow histories with the tasks-credential-refresh-propagate-cancel patch, cancellation stops the loop even during an in-flight refresh activity. Other refresh failures retry on the default cadence.

A follow-up command read timeout leaves its turn open, even if an earlier turn's completion signal arrived during delivery. This applies to user and peer messages. An open turn blocks sandbox rotation. If its sandbox disappears before completion, a workflow-origin run fails instead of reporting unfinished work as completed.

With tasks-rotation-activity-guard, active heartbeats also block rotation until the agent reports idle, including background work after a user turn ends. Pi message and tool events mark the agent active. The first activity after a turn ends bypasses heartbeat throttling so a quick follow-up cannot look idle. Activity during snapshot capture or replacement startup abandons the handoff and keeps the live sandbox. The event relay stays active until startup finishes, and an abandoned handoff with new activity requests a fresh snapshot. Directory resume snapshots cover /tmp/workspace, including nested Git worktrees and agent state. Paths outside that directory are not included. Cloud launches set POSTHOG_AGENT_STATE_DIR=/tmp/workspace/.posthog/agent-state. The agent server links Claude's projects, session environment, plans and todos, and Codex's sessions and shell snapshots into that directory before starting a session. Authentication and platform configuration stay outside these mappings. Pi restores its session history through the task-session API.

The agent stores completed prompt acknowledgements in its persistent state directory, so a resumed server skips completed prompts even when clearing their API state failed. Creating a resume snapshot preserves pending user messages until delivery handles them. If capture exceeds the file limit, it reports failure and retains the previous checkpoint without pruning the live workspace. Directory names such as .cache or .venv do not establish that their contents can be rebuilt. The legacy allow_pruning activity field remains readable but no longer enables deletion.

TASK_RUN_STREAM_GENERATION_ROTATION_ENABLED defaults to false during rollout. Deploy generation-aware token issuers and ingest handlers everywhere before enabling it. Leave rotation off until old workers drain; otherwise a resumed sandbox can receive an old-format token that the new ingest handlers reject. Already rotated streams keep their generation if rotation is disabled again. Stream completion is bound to the workflow's stream generation. A resume replaces that generation atomically with the stream reset, so late cleanup cannot close it. Dispatch saves the generation on the run and workflow input before context loading. Cancellation can tear down the sandbox without Redis and retains that generation across completion retries. Sandbox ingest tokens carry the same generation. Django and agent-proxy check it atomically when accepting events or completion, rejecting stale senders with HTTP 403 and stream_generation_mismatch. The sender discards its buffer and stops on this code. Tokens issued before generation tracking remain valid only until the stream resets.

TASKS_SANDBOX_PERSISTENT_FILESYSTEM_ENABLED enables an experimental Modal layout for default, autoresearch, and VM task templates. Task processes see /tmp, /var/tmp, and /root backed by /tmp/workspace/.posthog/task-filesystem, so ordinary files at those paths enter directory snapshots. Modal control processes keep their original filesystem view. Agentsh, the setup wizard, dev-stack bootstrap, and preview services enter the task view; Docker must start there so bind mounts resolve the same paths. Fresh credential files remain outside the saved directories. Claude's pinned session settings resolve to a private runtime directory, and the one-use Codex token is staged under /run. Base images seed the persistent home outside the workspace during build, preserving ownership. Custom images and dev-stack bakes refresh that seed after warming their caches. Activation moves the seed into the workspace without copying it; disabled tasks leave it out of directory snapshots, and versioned restores keep their saved home. Fresh tasks using older images without a home seed keep the provider view until those images are rebuilt. Image-bake sandboxes keep the provider view, and wizard output lives under /tmp/workspace/.posthog/wizard-cloud-run. Namespace entry preserves the caller's working directory. Health and capability checks run in the provider view, while bundled-skill cleanup runs in the task view. Cleanup uses a fixed system PATH and absolute system binaries. Helper uploads use the caller's execution timeout. Automatic dev-stack bootstrap uses a fresh Docker config directory under /run. Launch preparation uses a fixed system PATH and the image's absolute agentsh path.

Snapshots carry a layout version. Restoring that version initializes the task view even when the setting is disabled, while legacy workspace snapshots stay readable. Unversioned restores keep their original layout even when enabled, so existing files remain accessible without copying a home at startup. Missing namespaces and unknown versions fail startup. This layout covers files; processes, runtime sockets, system package changes, and service data outside these directories still need their own recovery. Keep it disabled until the checkpoint handoff and required service recovery are validated together.

Running via the UI

This is very minimal at the moment, but the tasks page can be used to see what is happening with a background cloud run and for debugging. You can also use PostHog Desktop to do this rather than the debug UI.

  1. Navigate to Tasks in PostHog (requires the tasks feature flag) by visiting /tasks (it will not show up in the sidebar)
  2. Create a task with a title, description, and repository (format: owner/repo)
  3. Click "Run task"
  4. Watch logs stream in the session view

Testing with local agent packages (you only need to do this if you are making changes to the agent package, otherwise ignore this)

To test changes to @posthog/agent before publishing:

Modal credentials

Both MODAL_DOCKER and modal providers require a Modal API token. Look up "Modal Development Token" in 1Password and add the values to your .env:

MODAL_TOKEN_ID=<token_id>
MODAL_TOKEN_SECRET=<token_secret>

Docker sandboxes derive the LLM gateway from SITE_URL. It reaches the sandbox as POSTHOG_API_URL, and getCloudTaskGatewayUrl maps only localhost and host.docker.internal to the local gateway on 3308. Any other host — an ngrok domain set for Slack or webhook testing, for instance — falls through to the production gateway, which a local run token cannot authenticate against. The failure is silent: the agent boots, sends its first prompt, and idles until its inactivity window closes it, with nothing in the gateway or agent logs. If SITE_URL is not localhost, set SANDBOX_LLM_GATEWAY_URL=http://host.docker.internal:3308 as well.

Tunnel gateway, API, and MCP

If you run in a docker sandbox you don't need to do this. If you are testing with Modal sandboxes, since they run in the cloud and can't reach localhost directly, you'll need to expose the Django API, LLM gateway, and MCP server publicly. We use Tailscale Funnel.

The eval harness sets Funnel up and tears it down automatically on a Modal run, so for evals you don't need to do any of this by hand — it only needs the daemon running and Funnel enabled (below). This section is for driving a Modal sandbox manually (e.g. from a task run).

First, install Tailscale, start the daemon, and sign in:

tailscale up

Then enable Funnel for the tailnet and this node in the admin console (grant the funnel node attribute) and make sure HTTPS certificates are enabled for the tailnet. Funnel only serves three public ports — 443, 8443, and 10000 — which is exactly enough for the three services. Point each one at the corresponding local port:

tailscale funnel --bg --https=443 8000    # Django API
tailscale funnel --bg --https=8443 3308   # LLM gateway
tailscale funnel --bg --https=10000 8787  # MCP server

Find your node's public hostname (the MagicDNS name) with tailscale status --json | jq -r .Self.DNSName, then set the resulting URLs in your .env (the 443 service drops the port):

SANDBOX_API_URL=https://<node>.<tailnet>.ts.net
SANDBOX_LLM_GATEWAY_URL=https://<node>.<tailnet>.ts.net:8443
SANDBOX_MCP_URL=https://<node>.<tailnet>.ts.net:10000/mcp

Funnel exposes these services to anyone on the internet who learns the URL, so turn the mappings off when you're done: tailscale funnel --https=443 off (and likewise for 8443 and 10000).

SANDBOX_MCP_URL overrides the host.docker.internal default (which only resolves from local Docker sandboxes, not Modal). Without it, sandbox agents can't reach the MCP server and lose access to the PostHog execute-sql, query, and tool-calling stack.

AI gateway token caps

Scoped AI gateway tokens use SANDBOX_AI_GATEWAY_TOKEN_CAP_USD as their default per-run dollar cap. Two JSON object settings can override it:

  • SANDBOX_AI_GATEWAY_TOKEN_CAP_USD_OVERRIDES maps team IDs to caps.
  • SANDBOX_AI_GATEWAY_TOKEN_CAP_USD_PRODUCT_OVERRIDES maps AI product names to caps. It is merged onto the built-in SANDBOX_AI_GATEWAY_TOKEN_CAP_USD_PRODUCT_DEFAULTS map in posthog/settings/temporal.py, and its entry wins per product.

A product cap takes precedence over a team override, which takes precedence over the default cap. An invalid entry, or a value that is not a JSON object, is reported to error tracking and ignored, so the product keeps its built-in cap.

Which gateway a sandbox run uses

A run reaches the Go gateway only when its ai_product is listed in SANDBOX_AI_GATEWAY_PRODUCTS and the worker minted a scoped token for it. When the worker mints, it also injects AI_GATEWAY_PRODUCT and AI_GATEWAY_AI_STAGE, naming the product the token is pinned to, and the agent routes on those in preference to what it derives itself. Both are reserved keys: a sandbox environment cannot set them.

Account audit runs use the reserved account_audit origin and AI product, and must carry the server-set internal flag before they can mint a token. Add account_audit to SANDBOX_AI_GATEWAY_PRODUCTS to route them to the Go gateway. Their Python gateway fallback uses background_agents. Signup onboarding stays on onboarding and cannot mint these cost-tracking tokens. A periodic completion task waits three minutes for cost accounting before reporting the audit result. Audits use an internal CUSTOM sandbox environment with no extra allowed domains. Each new audit reasserts this policy; only the platform service allowlist stays reachable.

When a run lands on the Python gateway unexpectedly, check those two variables first. Their absence means no token was minted, so the agent falls back to deriving the product from the task run it fetches at boot, which is the path that fails quietly.

ReviewHog project access

The boolean review-hog feature flag controls the Code review UI and all project ReviewHog APIs, including for staff. Normal membership and API scopes still apply. Set every release condition to the project group, match its id property against the allowed project IDs, and use 100% rollout. Replace broader conditions and include existing projects before deployment. Use the id property, since frontend and backend group keys differ. The property identifies the active environment: explicitly include every environment that should have access. A parent project's flag does not enable its child environments, even though ReviewHog stores their settings and reviews under the shared parent project.

Newly enabled projects get every ReviewHog part: manual Full and Flash reviews, automatic Flash reviews, the Stamphog handoff, resolution, Inbox reviews, the tiered review models, and the Stamphog Inbox switch. Each project needs a GitHub App integration covering the repository; review skills seed automatically. Automatic reviews and the Stamphog handoff also need the project to claim the repository in the Code review settings. No environment variable or shared secret picks the project.

Use resolution only for explicitly enabled trusted projects reviewing repositories their teams own. Ownership does not authenticate commenters: operators must assess repository and comment trust before enabling resolution. The current risk acceptance covers this limited manual rollout, including the missing commenter authorization and the post-push path check. Public or untrusted use still requires the hardening listed in ReviewHog's architecture.

ReviewHog Flash uses gpt-6-luna for review, blind-spot checks, and validation. The configured internal project's ReviewHog Flash - Experimental subsection under What gets reviewed holds the automatic Flash review toggle. Flash reviews use the medium reasoning effort. The shared FLASH_ARM and flash_arm_for_effort in products/review_hog/backend/reviewer/constants.py pin the Codex runtime and full-access permission mode. Flash uses the existing review_hog model allowance. Both review modes instruct the agent to fetch pinned review and validation skills through the PostHog MCP with skill-get. The agent can fetch referenced bundled files with skill-file-get. On the configured internal project, choose Review in Flash mode from the review menu to run it for one turn without changing the PR's full-review configuration. Flash requests preserve an existing report's review tier, including when they join a running review. Flash marks its status comment header as PostHog Review (flash), skips the clean-review media, and never starts comment resolution.

Review all your PRs in Flash mode is off by default and shown only on the configured internal project. Turn it on in Code review to review PRs you author in the repositories your project reviews when they open or receive new commits, including drafts. The head branch must belong to the PR's own repository; fork PRs are excluded. Enabling it does not review existing PRs immediately; an existing PR becomes eligible on its next push. Only one review runs per PR, and pushes during a review coalesce into a follow-up for the latest head. An explicit Full request also runs after an active Flash review when that head still needs a Full review. Turning the setting off stops future and pending automatic starts; a running review finishes. Automatic Flash uses your existing severity threshold and never resolves comments or changes the PR branch.

The switch sets your default_review_mode preference to flash or follow through PATCH /api/projects/<project_id>/review_hog/settings/. Project admins choose which repositories the project reviews, and who gets automatic Flash there, through the review_hog/project_settings, review_hog/installation_claims, and review_hog/repositories endpoints. Any project member can turn a repository on through review_hog/repositories: include it into the project's selection, raise its exception, or turn its draft reviews on. Turning a repository down or removing it stays with admins.

The Code review settings page calls automatic reviews Standard and requested reviews Deep. Project defaults appear beside personal preferences. Personal repository choices take priority over personal defaults and project rules. Commit fixes to my pull requests allows Deep reviews to push fixes only with the author's own opt-in. The control's help explains when fixes are paused. Review skill descriptions summarize each check; editing skill text changes it for the whole project.

To run Flash locally, use run_review --review-mode flash from the repository root:

.codex/with-flox python manage.py run_review \
  --pr-url https://github.com/PostHog/posthog/pull/PR_NUMBER \
  --team-id 1 --user-id 1 --review-mode flash

Replace PR_NUMBER and use the team and user IDs from your local instance. The command defaults to Full mode and only requests GitHub publishing when you add --publish. For isolated tests, use a fresh local report with no active review on the same PR and an original branch that still points at the reviewed commit. An active turn keeps its settings snapshot, and an existing report's status comment can still receive progress updates. If a review fails, the next attempt keeps cached reviewer results for the same commit, model, and reasoning effort. Deduplication retires superseded findings from the unfinished turn and reuses a verdict only when its finding, commit, review mode, and model configurations are unchanged. Completed turns remain in the report history. Long-running review activities refresh the active report every minute so it remains visible in Code review while an agent works without new results. The refresh stops when the activity exits; a report with no new activity still expires from the running list after 30 minutes. Review-started, completed, and failed event IDs distinguish Full and Flash retries while preserving the legacy Full IDs. When calculating completion rates, match report, turn, and mode, treating an absent mode as Full for legacy events. Flash finding-outcome events use the model configuration saved with the finding, even if the Flash defaults change before classification. Full findings retain report-level model attribution, and findings without readable saved context have an unknown review mode. When recovering a failed publish with python manage.py publish_review, the command infers Full or Flash from the completed turn's findings. Legacy findings without a stored mode default to Full, and an explicit --review-mode must match the stored mode. Recovery uses the completed turn's commit when recorded, even if a newer unfinished turn has fetched another commit.

Agent run telemetry (optional)

To ship agent-server run metadata to PostHog Logs, set both of the first two; the third additionally produces one APM trace per run (root task_run span, a turn span per prompt, a tool_call:<kind> span per tool call) with trace/span ids stamped on the log records:

SANDBOX_AGENT_OTEL_LOGS_URL=http://localhost:8000/i/v1/logs  # or https://us.i.posthog.com/i/v1/logs
SANDBOX_AGENT_OTEL_LOGS_TOKEN=<project API key of the telemetry project>
SANDBOX_AGENT_OTEL_TRACES_URL=http://localhost:8000/i/v1/traces  # optional, enables APM spans

In cloud, dispatch stamps each run as telemetry-enabled in run state, and the scout run-log mirror only mirrors stamped runs. DEBUG skips the stamp, so locally these settings are the only switch. They're injected into the sandbox as POSTHOG_AGENT_OTEL_LOGS_URL/_TOKEN/POSTHOG_AGENT_OTEL_TRACES_URL (deliberately not standard OTEL_* names, so OTel SDKs in user code don't auto-export into the telemetry project). The agent-server exports run/turn/tool lifecycle metadata (never message content or tool arguments), tagged with run_id/task_id/team_id/user_id/distinct_id resource attributes and service.name=posthog-code-agent. Telemetry stays off when either of the first two vars is unset. For local Docker sandboxes the localhost URLs are rewritten to host.docker.internal automatically; local ingestion requires the capture-logs service to be running.

The trace id of a run is its run UUID without dashes, so /tracing?trace=<run UUID without dashes> in the telemetry project opens the run. The agent-server spans, the mirrored run logs and the process-task Temporal workflow spans all carry that trace id. The workflow spans carry it only when the run has the telemetry flag on. Those runs are also marked as sampled, so a parent-based sampler on the worker keeps every one of them. They show up in the same trace only if the Temporal worker's collector forwards to the telemetry project. The workflow and the agent-server share a trace id but no parent span, so the workflow spans sit under a <parent span missing> row beside task_run.

MCP server .env

MODAL_DOCKER (and the local Docker provider) both depend on the MCP server running at localhost:8787. The server reads its config from services/mcp/.env — without it, things like POSTHOG_API_BASE_URL, the UI-apps token, and analytics keys are missing and the server will either refuse to start or return broken responses to the sandbox.

cd services/mcp && cp .env.example .env

Then fill in the secrets. POSTHOG_UI_APPS_TOKEN and POSTHOG_ANALYTICS_API_KEY are public PostHog phc_* project keys — for local dev you can paste the same key you use for analytics, or leave them as the placeholder (analytics calls will no-op). Restart the mcp phrocs process after changing .env.

Memory pressure during agent validation

The memory watchdog stops tool process trees before the sandbox reaches its memory limit. A process stop, including SIGKILL escalation, does not mean the task run died.

A run whose whole sandbox stops usually still ends as completed, with termination_reason set to sandbox_gone. The error_message property on task_run_completed carries the sandbox exit reason. Exit code 137 there usually means the sandbox ran out of memory.

Cloud Claude and Codex sessions deliver each watchdog warning separately to subagents and their parent. Codex uses native MCP tool hooks for shell calls, including the final result of an exec_command completed through write_stdin. Claude shell results with exit codes 137, 143, or 144 wait briefly for the watchdog's delayed record; an exit code alone is not treated as proof of an OOM. Codex hook payloads carry the command output without its exit status, so a Codex command counts as stopped when a kill record lands between its start and shortly after its end. A quick rerun of a long Codex command waits for that record before it starts.

Codex trusts only the PostHog memory hooks through session configuration. It does not store hook trust in the user's config file. The local tools MCP server opts out of Codex's tool catalog cache, so a subagent thread starts it before its first shell command instead of lazily on its first MCP tool call. Codex hook payloads do not include an exec_command workdir, so a Codex retry count is keyed by the thread directory and the command text. Put cd <directory> && in the command to scope a retry count to a directory.

Common build, test, and typecheck commands share a sandbox-wide lock, including commands started in the background. When another validation command holds the lock, the shell returns exit code 75 and asks the agent to wait. After the same shell command fails twice during observed watchdog interventions, the session rejects another unchanged attempt. This retry limit applies to every command, not only validation. Codex includes final background results in this limit. Claude background-result tools deliver warnings but lack the command text needed to count retries. Reduce the command's scope or concurrency, or report the memory limit. Validation recognition is best-effort and controls only the lock. It is not a resource limit for arbitrary shell programs.

The guard recognizes validation through timeout, npx, hogli, .codex/with-flox, and flox activate -- bash -c '…'. It preserves the command's directory, arguments, and inline shell body when identifying retries. Script names such as backend:test and build-storybook, shell continuations, and command substitutions are recognized. Heredoc bodies are skipped before scanning subsequent commands. It does not inspect script files: invoke validation directly or through a supported wrapper instead of hiding it in bash script.sh.

Local agent packages

Cloud tasks use the published @posthog/agent package by default. Set LOCAL_POSTHOG_CODE_MONOREPO_ROOT only when you need to test local agent changes.

For local Docker, the worker builds the packages inside the sandbox image. The first build installs the workspace dependencies. Later source changes reuse those dependencies from the Docker build cache.

# In your .env:
SANDBOX_PROVIDER=docker
# The agent workspace lives in this repo at packages/agent
LOCAL_POSTHOG_CODE_MONOREPO_ROOT=./packages/agent

Restart the temporal worker after changing .env.

For local Modal, set SANDBOX_PROVIDER=MODAL_DOCKER, build the packages, and restart the temporal worker:

pnpm --dir packages/agent build

Sandbox providers

Provider .env value When to use
modal (default) SANDBOX_PROVIDER=modal Production. Uses the published @posthog/agent npm package from the GHCR image.
MODAL_DOCKER SANDBOX_PROVIDER=MODAL_DOCKER Local development with Modal. Same as modal but uses a separate Modal app (posthog-sandbox-modal-docker-*) so local image builds don't pollute the production app cache. When LOCAL_POSTHOG_CODE_MONOREPO_ROOT is set, each local package's external runtime dependencies are installed and its compiled output is overlaid onto the image.
docker SANDBOX_PROVIDER=docker Local-only Docker containers (DEBUG=True required). No Modal account needed. Uses the published agent by default and builds local agent packages when LOCAL_POSTHOG_CODE_MONOREPO_ROOT is set. This is the recommended option for local development.

Modal apps

Every sandbox is booked against a Modal app, which is what groups it in the Modal dashboard and attributes its cost. The app is picked from the template first, and from SandboxConfig.workload when the template has no app of its own.

App Owned by
posthog-sandbox-default Everything not claimed below — user-created tasks, loops, onboarding, image builds
posthog-sandbox-self-driving The self-driving fleet: Signals report research and repo selection, Signals scouts, ReviewHog
posthog-sandbox-notebook NOTEBOOK_BASE template
posthog-sandbox-streamlit STREAMLIT_BASE template

Self-driving membership is derived from the task's origin_product (SELF_DRIVING_ORIGIN_PRODUCTS in logic/services/sandbox.py), so a product joins the fleet by adding its origin there — no caller changes. The split is for metering only: same image, same resources, same isolation, and images and snapshots are workspace-scoped in Modal, so a self-driving box still restores a snapshot baked under the default app.

Each name above is a class attribute on ModalSandbox, and the MODAL_DOCKER and MODAL_EVALS providers override all four (posthog-sandbox-modal-docker-*, posthog-sandbox-evals), so local and eval runs never land in a production app. A new app name has to be a class attribute for that to keep holding.

Task sandboxes carry task_id and task_run_id tags for exact lookup, plus a readable workflow_label. The label keeps the workflow prefix, replaces characters outside A-Z, a-z, 0-9, _, ., and - with dashes, and caps the result at 63 characters. Use TaskRun.workflow_id to recover the full Temporal workflow ID from the run.

Sandbox templates

Staff can inspect the agent release pipeline at /admin/tasks/task/infrastructure/ in each region. Release reads use the configured GitHub App installation for PostHog/posthog, with a temporary token restricted to Contents and Actions reads on that repository. No integration record or installation ID setting is needed. Deployments without GitHub App credentials use GITHUB_TOKEN when set. The read-only page compares the published package, master version pin, registry platforms, custom-image bases, and the last recorded dev-stack bake. Release evidence separates workflow status from image build and base promotion results, including skipped builds. Select a custom image to inspect its latest Temporal execution. A failed refresh can leave a ready image on an older base. The refresh sweep retries that image only when the VM base digest changes again. Missing or stale sources remain unverified. This view does not measure versions inside running sandboxes or reconstruct historical rollout completion. The Data sources tab lists each source's status and last successful read in UTC. Registry coverage remains unverified when the release source is unavailable or stale. Graph release badges compare observed versions with npm latest; cached observations say "Last seen". Select an image for its separate version-pin and base-lineage assessment. Source failures include a safe diagnostic; a rejected GitHub credential requires checking the server's App configuration or shared token. Build-history failures leave a successfully read version pin available and show a separate warning. Individual job-read failures retain run links and other build results. If the workflow list cannot be read, previous runs remain visible with their original read timestamp. Retained source values are identified as the last successful read, and shared GitHub request-budget limits have a separate diagnostic. The dev-stack graph and details share a base-adoption status. "Awaiting refresh" means the last successful bake uses a different digest from the current VM image; the details show both references. A fresh source read does not establish that a bake has finished.

Each sandbox is created from a template that determines its base image and capabilities.

Template Image Description
DEFAULT_BASE ghcr.io/posthog/posthog-sandbox-base Standard sandbox template (default).
NOTEBOOK_BASE ghcr.io/posthog/posthog-sandbox-notebook Template for notebook functionality.
VM_BASE ghcr.io/posthog/posthog-sandbox-vm Docker-in-Docker capable. Layers Docker engine, compose v2, and buildx on the base image. Includes an idempotent start-dockerd helper for on-demand dockerd.

VM_BASE uses the Modal VM runtime (real Linux kernel) instead of gVisor because dockerd cannot run under gVisor. When a sandbox is created with template=SandboxTemplate.VM_BASE, ModalSandbox.create automatically sets experimental_options={"vm_runtime": True}.

Protected dev-stack activation

Dev-stack bakes can use a durable, regional activation pointer instead of publishing over a shared Modal image name. The two infrastructure models are instance-global: they describe the shared image, not a tenant's sandbox or custom image. Explicit rebuild requests are recorded before observing inputs, so a newer routine observation can carry a pending request forward.

VM digest + repository SHA + bake recipe
  -> generation -> immutable snapshot -> boot verification
  -> atomic activation -> new task sandboxes

Activation locks the target and checks its generation, attempt owner, lease and deadline. A superseded bake can finish its snapshot but cannot change the active pointer. Candidate verification boots the exact snapshot, restores the manifest's repository revision for retained container bind mounts, and checks its manifest, agent version, Postgres, Redis and ClickHouse. The previous active image remains available if verification fails. Failed verification commands log bounded output tails with the stage and attempt, image and sandbox IDs. Cleanup retries cannot replace the original verification failure.

The two-minute sweep reconciles durable state using a fresh registry digest rather than the task-provisioning cache. A changed bake recipe requests a new generation even when the VM digest is unchanged. Dispatch retries reuse an attempt's Temporal workflow ID, and a lost activation acknowledgment reads the committed result. Workers renew two-minute leases every 30 seconds, with a separate three-hour deadline. Failed bakes retry after 2, 5 and then 15 minutes with up to 29 seconds of jitter. Each region allows three full bakes per rolling six hours across generations; snapshot verification does not consume that budget. The dashboard shows the next retry or blocking reason. Provider boot failures retry the checkpointed snapshot; confirmed candidate defects require a corrected target or a new manual request.

Roll out one region at a time:

  1. Apply the additive migration and deploy the new workers and task readers. With no target row, the existing named-image behavior remains in place.
  2. Disable the legacy tasks-dev-stack-image-bake flag and drain legacy bake executions and old task workers. All task readers must understand protected activation before initialization.
  3. Run python manage.py bake_dev_stack_image --initialize --seed-image-id <immutable-image-id> using a known existing snapshot. This boots and checks the baseline before recording its ID and a verified attempt for rollback. Save the printed attempt UUID; its base provenance remains unknown. Omitting the seed deliberately leaves tasks on the plain VM base until the first verified activation.
  4. Run python manage.py bake_dev_stack_image --enable. The durable enablement setting now controls scheduling in this region; the legacy flag no longer controls protected bakes. GitHub App Contents read access is needed to pin the repository revision. The next sweep requests a generation from the current VM digest.
  5. Verify a successful candidate and a new task's image selection before enabling another region. Existing sandboxes are unchanged.

Use python manage.py bake_dev_stack_image --request-key <request-id> for a manual refresh. Protected bakes reject --inline. --disable stops scheduling and activation while preserving the active image. A database read failure or missing active pointer falls back to the plain VM base with a recorded image-fallback reason, never to the mutable named image.

To roll back, use python manage.py bake_dev_stack_image --rollback-attempt <verified-attempt-uuid>, including the seed's UUID. This creates a new generation and reverifies that attempt's immutable candidate. Automatic target updates remain paused until --enable resumes them. Initialization, enablement, disablement, and rollback must be separate commands. Do not delete the target row or redeploy alias-only readers as a rollback strategy. Candidate snapshots are retained; this change does not add automatic image deletion.

The admin separates Base current / Base behind / Base unknown from Queued / Building / Rebuilding / Verifying / Retry scheduled / Blocked / Baking disabled. A same-base nightly rebuild can show Base current and Rebuilding together. Budget cooldowns show Retry scheduled with their reason and retry time; Blocked means intervention is required. Failed Temporal reads show retry guidance even when no execution data is available. Source timestamps describe observation time; activation time and the exact attempt's Temporal run link describe the bake.

Optional: local repository mounts (Docker only)

If you already have a repository checked out locally, you can skip cloning by bind-mounting it into the Docker sandbox:

# Format: org/repo:/local/path,org2/repo2:~/other/path
SANDBOX_REPO_MOUNT_MAP=PostHog/posthog:~/Developer/posthog

When configured, matching repositories are mounted read-write from your host into the container, and clone_repository becomes a no-op for those repositories.

Note: This only works with SANDBOX_PROVIDER=docker.

Task-run log mirroring to PostHog Logs (dogfooding)

Task-run log entries (the JSONL appended to object storage via TaskRun.append_log) are also mirrored into the PostHog Logs product, so runs can be browsed and sampled in the Logs UI instead of fetching S3 blobs.

In production there is no transport of its own: entries are emitted as structured stdout log lines (event=task_run_log), and the per-cluster OTel collector that already ships all container stdout into the region's internal PostHog project picks them up. The collector parses each JSON key into a queryable attribute and turns the emitted request_id (the run uuid) into a trace id, so one run groups as a trace and can be pulled up with an attribute filter on task_run_id.

# Which task origins to mirror (comma-separated). Defaults to signals scouts and user-created tasks.
# Set empty to disable.
TASK_RUN_LOGS_MIRROR_ORIGIN_PRODUCTS=signals_scout,user_created

The mirror also has a direct OTLP leg that ships each batch straight to a logs ingest endpoint:

TASK_RUN_LOGS_MIRROR_OTLP_URL=http://localhost:8000/i/v1/logs  # prod: https://us.i.posthog.com/i/v1/logs
TASK_RUN_LOGS_MIRROR_OTLP_TOKEN=<project API key of the internal logs project>

The token pins the destination: scout runs execute for customer teams, but their mirrored transcripts must only ever land in — and bill — PostHog's own internal logs project, never the customer's. Records arrive under service.name=task-run-log-mirror with the run uuid as the trace id.

Locally the direct leg is the only delivery path: append_log runs in the host Django process, and the dev collector (otel-collector-config.dev.yaml) only tails docker-compose container stdout, so without these settings the mirrored lines only show up in the Django phrocs pane.

Mirroring failures are logged and never break the run's log write.

How MODAL_DOCKER works

When both SANDBOX_PROVIDER=MODAL_DOCKER and LOCAL_POSTHOG_CODE_MONOREPO_ROOT are set:

  1. The selected sandbox Dockerfile is built in a temporary context
  2. External runtime dependencies from local packages/agent, packages/agent-contracts, and packages/git manifests that are missing from the published image are installed at /scripts; required system compatibility packages such as musl for Codex are installed with them, while workspace:* dependencies continue to resolve through the overlaid packages
  3. Each local package's built dist/ directory is mounted over the published package's compiled output
  4. The image runs in a separate Modal app (posthog-sandbox-modal-docker-default) so it doesn't affect production
  5. The first build takes a few minutes; subsequent builds reuse Modal's layer cache

After changing agent-server code, rebuild and restart the worker:

cd packages/agent/packages/agent && pnpm build

Note: The build context is cached for the lifetime of the worker process (lru_cache). You must restart the temporal worker to pick up new local package changes.

Troubleshooting

Problem Solution
Docker not running Start Docker Desktop or the Docker daemon
Temporal not reachable Ensure Temporal is running on 127.0.0.1:7233. Check with temporal server start-dev
Feature flag not enabled Re-run python manage.py setup_background_agents to (re-)create the tasks flag at 100% rollout
Array OAuth app missing Re-run python manage.py setup_background_agents
PostHog AI app not found for region ... The PostHog AI OAuth app is missing. Run python manage.py setup_tasks_oauth, which creates both the Array and PostHog AI dev apps. Deploys run it from bin/migrate; it no-ops in the US/EU regions
GitHub token expired Tokens from GitHub App installations expire after ~1 hour. Re-run the task to get a fresh token
"Task workflow execution blocked" The tasks feature flag is not enabled for this user/org
Sandbox image build fails Check Docker has enough disk space. Delete old images with docker system prune
Agent server health check fails Check sandbox logs: docker exec <container_id> cat /tmp/agent-server.log
SANDBOX_JWT_PRIVATE_KEY missing Re-run python manage.py setup_background_agents — it will auto-fill from .env.example
Port conflict on sandbox host port DockerSandbox maps container port 47821 to a dynamic host port. Check sandbox logs or TaskRun state for the assigned port; if another process uses it, stop that process or restart Docker
Sandbox can't reach PostHog API Don't set SANDBOX_API_URL with Docker — auto-transform handles it. If overriding, use port 8000, not 8010 (Caddy returns empty responses from inside Docker)
MCP Store connectors don't mount in local sandbox runs (agent lists only posthog and posthog-code-tools) Known local-dev gap: store-connector proxy URLs are built from SANDBOX_API_URL/SITE_URL and, unlike sandbox env vars, are not rewritten to host.docker.internal for Docker. The agent SDK drops unreachable servers silently. Affects loop and workflow connectors locally; prod URLs are public so it never applies there
DEBUG not set SANDBOX_PROVIDER=docker requires DEBUG=1. Re-run python manage.py setup_background_agents to write it
... sandbox is for local development only (RuntimeError at import) The docker / MODAL_DOCKER providers require DEBUG=1 (or TEST=1, which pytest sets). DEBUG=1 is normally injected by the flox env (.flox/env/manifest.toml [vars]) — this fires when you're outside flox activate or explicitly unset DEBUG (e.g. to escape the cloud-DEBUG guard). Keep DEBUG on and use CLOUD_DEPLOYMENT=E2E for cloud-mode dev instead. See dev-env-vars.md
git commit is disabled in PostHog Desktop A PATH shim (git-guard.sh at /opt/posthog/bin/git) blocks git commit and git push so unsigned commits can't leave the sandbox. Stage changes with git add, then use the git_signed_commit tool. To bypass during debugging, set POSTHOG_ALLOW_UNSIGNED_GIT=1