:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(desktop): run the live agent e2e against the Go AI gateway (#124920)
Jonathan Mieloo873aa828cd8c
diff --git a/.github/workflows/desktop-ci.yml b/.github/workflows/desktop-ci.ymlindex b767c3f93fab..04c14712e410 100644--- a/.github/workflows/desktop-ci.yml+++ b/.github/workflows/desktop-ci.yml@@ -52,7 +52,6 @@ jobs:secrets:GH_APP_POSTHOG_PATHS_FILTER_PRIVATE_KEY: ${{ secrets.GH_APP_POSTHOG_PATHS_FILTER_PRIVATE_KEY }}POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY: ${{ secrets.POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY }}- POSTHOG_CODE_E2E_GATEWAY_URL: ${{ secrets.POSTHOG_CODE_E2E_GATEWAY_URL }}# Desktop counterpart of the migration/service separation check (see the# script header for why). Lives here rather than pr-housekeeping so thediff --git a/.github/workflows/desktop-test.yml b/.github/workflows/desktop-test.ymlindex e036bd2f925f..19aeb8451706 100644--- a/.github/workflows/desktop-test.yml+++ b/.github/workflows/desktop-test.yml@@ -9,8 +9,6 @@ on:required: falsePOSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY:required: false- POSTHOG_CODE_E2E_GATEWAY_URL:- required: false# Master coverage comes from this cron and from the release tag job, which# dispatches this workflow on the tree it tags. Pull requests and merge queue# batches run the suite per change, so the cron is here to catch drift from@@ -269,8 +267,7 @@ jobs:# after the unit + Electron jobs pass — a red tree never reaches the# gateway — and only when a products/desktop/packages/ file changed. Each arm still# self-skips unless the POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY secret is- # present (fork PRs never see it) and POSTHOG_CODE_E2E_GATEWAY_URL points at- # a runner-reachable gateway. Drives cheap models (claude-haiku-4-5 /+ # present (fork PRs never see it). Drives cheap models (claude-haiku-4-5 /# gpt-5-mini), so a run is a handful of short turns.needs: [changes, unit-test, electron-test]# Always run on any products/desktop/packages/ change (or a change to this workflow),@@ -351,13 +348,10 @@ jobs:working-directory: packages/agentrun: pnpm --filter agent run test:e2e# The suite reads these POSTHOG_CODE_E2E_* names directly- # (packages/agent/packages/agent/e2e/config.ts). The personal API key is an org- # secret; the URL and model/environment overrides are org vars.+ # (packages/agent/packages/agent/e2e/config.ts).env:POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY: ${{ secrets.POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY }}- # Accept the URL from either an org var or an org secret — it has ended- # up in both places, and vars.* can't see secrets.- POSTHOG_CODE_E2E_GATEWAY_URL: ${{ vars.POSTHOG_CODE_E2E_GATEWAY_URL || secrets.POSTHOG_CODE_E2E_GATEWAY_URL }}+ POSTHOG_CODE_E2E_GATEWAY_URL: https://ai-gateway.us.posthog.comPOSTHOG_CODE_E2E_CLAUDE_MODEL: ${{ vars.POSTHOG_CODE_E2E_CLAUDE_MODEL }}POSTHOG_CODE_E2E_CODEX_MODEL: ${{ vars.POSTHOG_CODE_E2E_CODEX_MODEL }}# Optional: set vars.POSTHOG_CODE_E2E_ENVIRONMENT=cloud to exercise thediff --git a/packages/agent/packages/agent/e2e/README.md b/packages/agent/packages/agent/e2e/README.mdindex 48ea2b1edd97..5234316355c8 100644--- a/packages/agent/packages/agent/e2e/README.md+++ b/packages/agent/packages/agent/e2e/README.md@@ -1,7 +1,7 @@# Live agent e2e suiteDrives representative sessions **end to end** through the real adapter, the real-binary (codex `app-server` / Claude Code CLI), and the real llm-gateway on a cheap+binary (codex `app-server` / Claude Code CLI), and the real ai-gateway on a cheapmodel — parametrized across `claude` and `codex`. The only thing mocked is thehost/UI client (a recording `sessionUpdate`, an auto-allow `requestPermission`,and real file read/write against a throwaway git repo). Nothing in the@@ -72,8 +72,7 @@ includes `src/**`). They are opt-in and cost a couple of short model turns.In CI they run as the **`e2e` job in `.github/workflows/test.yml`**, on pullrequests only, after the unit + integration jobs pass. The job is opt-in and safeby default: it self-skips unless the repo variable `AGENT_E2E_ENABLED` is `true`-with an `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY` secret and an `POSTHOG_CODE_E2E_GATEWAY_URL` variable pointing at a-gateway reachable from the runner, and it never runs for fork PRs (their secrets+with a `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY` secret against `https://ai-gateway.us.posthog.com`, and it never runs for fork PRs (their secretsare withheld, which would otherwise red the fail-loud token guard). Off bydefault, so it costs nothing until explicitly enabled; the codex arm self-skips ifthe native binary isn't on the runner.@@ -86,7 +85,7 @@ bash e2e/run-e2e.shbash e2e/run-e2e.sh -t "(codex)"```-Prereqs: a local llm-gateway up (`./bin/start` in the posthog repo) and the+Prereqs: a local ai-gateway up (`bin/start-ai-gateway`) and thenative codex binary present at `apps/code/resources/codex-acp/codex` (the codexarm self-skips if it is missing).@@ -94,8 +93,8 @@ arm self-skips if it is missing).| Var | Default | Notes || --- | --- | --- |-| `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY` | — | Required. A token the gateway accepts — the `ci` product takes a personal API key (no OAuth). Without it every arm skips. `run-e2e.sh` reads the local dev key. |-| `POSTHOG_CODE_E2E_GATEWAY_URL` | `http://localhost:3308/ci` | Gateway base (codex appends `/v1`). `ci` accepts a personal API key; `posthog_code` is OAuth-only. |+| `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY` | — | Required. A `phs_` key, or a `phe_` dev token from `hogli gateway:token`. Requests carry `X-PostHog-Product: ci`. Without it every arm skips. |+| `POSTHOG_CODE_E2E_GATEWAY_URL` | `http://localhost:8080` | ai-gateway base (codex appends `/v1`). || `POSTHOG_CODE_E2E_CLAUDE_MODEL` | `claude-haiku-4-5` | Override if the gateway serves a different cheap Claude id. || `POSTHOG_CODE_E2E_CODEX_MODEL` | `gpt-5-mini` | Cheapest codex id the local gateway serves; override if needed. || `POSTHOG_CODE_E2E_CLAUDE_STRONG_MODEL` | `claude-sonnet-4-5` | Stronger Claude id for tests the cheap model can't handle (structured output). |diff --git a/packages/agent/packages/agent/e2e/config.ts b/packages/agent/packages/agent/e2e/config.tsindex ac4f375b43dd..f8fa368b072a 100644--- a/packages/agent/packages/agent/e2e/config.ts+++ b/packages/agent/packages/agent/e2e/config.ts@@ -6,16 +6,15 @@ export type { Adapter };/*** Live e2e configuration, resolved entirely from the environment so no secret is- * committed. Needs a local llm-gateway and a token in `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY`; targets- * the `ci` product, which accepts a personal API key (no OAuth mint,- * unlike prod's `posthog_code`) and keeps e2e traffic attributed to its own- * `ai_product` rather than the catch-all `llm_gateway` bucket. Without the token- * every arm self-skips.+ * committed. Needs an ai-gateway and a `phs_` key in `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY`; sends+ * `X-PostHog-Product: ci` so e2e traffic is attributed to its own product.+ * Without the token every arm self-skips.*/// `||` not `??`: CI sets unset vars to "" which should fall back to the default.const GATEWAY_URL =- process.env.POSTHOG_CODE_E2E_GATEWAY_URL || "http://localhost:3308/ci";+ process.env.POSTHOG_CODE_E2E_GATEWAY_URL || "http://localhost:8080";const TOKEN = process.env.POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY ?? "";+const GATEWAY_PRODUCT = "ci";// The desktop app downloads the codex binaries; it lives in the sibling desktop workspace.const CODEX_RESOURCES_DIR = join(@@ -92,6 +91,7 @@ export const E2E = {if (adapter === "claude") {process.env.ANTHROPIC_BASE_URL = GATEWAY_URL;process.env.ANTHROPIC_AUTH_TOKEN = TOKEN;+ process.env.ANTHROPIC_CUSTOM_HEADERS = `X-PostHog-Product: ${GATEWAY_PRODUCT}`;return;}process.env.OPENAI_BASE_URL = openAiBase();@@ -108,6 +108,7 @@ export const E2E = {binaryPath: string;apiBaseUrl: string;apiKey: string;+ httpHeaders: Record<string, string>;model: string;configOverrides?: Record<string, string | number>;} {@@ -116,6 +117,7 @@ export const E2E = {binaryPath: NATIVE_CODEX_BIN,apiBaseUrl: openAiBase(),apiKey: TOKEN,+ httpHeaders: { "X-PostHog-Product": GATEWAY_PRODUCT },model: modelOverride || this.model("codex"),...(configOverrides ? { configOverrides } : {}),};diff --git a/packages/agent/packages/agent/e2e/guard.e2e.test.ts b/packages/agent/packages/agent/e2e/guard.e2e.test.tsindex 8130aea94528..8daac1981d09 100644--- a/packages/agent/packages/agent/e2e/guard.e2e.test.ts+++ b/packages/agent/packages/agent/e2e/guard.e2e.test.ts@@ -11,8 +11,8 @@ describe("live e2e preconditions", () => {expect(E2E.hasToken,"POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY is not set — every adapter arm would skip and the run " +- "would pass without testing anything. Mint one via e2e/run-e2e.sh or " +- "set POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY against a reachable POSTHOG_CODE_E2E_GATEWAY_URL.",+ "would pass without testing anything. " ++ "Set POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY against a reachable POSTHOG_CODE_E2E_GATEWAY_URL.",).toBe(true);});@@ -25,7 +25,7 @@ describe("live e2e preconditions", () => {process.env.POSTHOG_CODE_E2E_GATEWAY_URL,"POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY is set but " +"POSTHOG_CODE_E2E_GATEWAY_URL is empty — on CI the suite would fall " +- "back to localhost:3308 (unreachable from a runner) and every model " ++ "back to localhost:8080 (unreachable from a runner) and every model " +"turn would fail with ConnectionRefused. Set the org variable to a " +"runner-reachable gateway.",).toBeTruthy();diff --git a/packages/agent/packages/agent/e2e/run-e2e.sh b/packages/agent/packages/agent/e2e/run-e2e.shindex df8a20fc7840..3785ef69c635 100755--- a/packages/agent/packages/agent/e2e/run-e2e.sh+++ b/packages/agent/packages/agent/e2e/run-e2e.sh@@ -1,12 +1,8 @@#!/usr/bin/env bash# Run the live golden-path e2e for both adapters (claude + codex).#-# Needs a local llm-gateway (run `./bin/start` in the posthog repo) and a token.-# The suite targets the gateway's `ci` product, which accepts a personal-# API key (no OAuth), so if POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY is unset this reads the repo's-# hardcoded local dev key from ee/settings.py (override the repo with POSTHOG_REPO).-# That key must be registered in the local DB — run `python manage.py-# setup_local_api_key` in the posthog repo once if auth fails.+# Needs an ai-gateway and a token in POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY: a `phs_`+# key, or a `phe_` dev token from `hogli gateway:token` in the posthog repo.## Usage:# bash e2e/run-e2e.sh # both adapters, both suites@@ -16,26 +12,13 @@ set -euo pipefailHERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"AGENT_DIR="$(cd "$HERE/.." && pwd)"-POSTHOG_REPO="${POSTHOG_REPO:-$(cd "$AGENT_DIR/../../.." && pwd)}"if [[ -z "${POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY:-}" ]]; then- SETTINGS="$POSTHOG_REPO/ee/settings.py"- if [[ ! -f "$SETTINGS" ]]; then- echo "POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY unset and posthog settings not found at $SETTINGS." >&2- echo "Set POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY, or POSTHOG_REPO to the posthog checkout." >&2- exit 1- fi- # The `ci` product accepts personal API keys, so no OAuth mint needed.- POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY="$(grep -E '^DEV_API_KEY[[:space:]]*=' "$SETTINGS" | head -1 | sed -E 's/^DEV_API_KEY[[:space:]]*=[[:space:]]*"([^"]+)".*/\1/')"-fi--if [[ -z "${POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY:-}" ]]; then- echo "Failed to obtain an POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY (no DEV_API_KEY in ee/settings.py?)." >&2- echo "If auth then fails, run 'python manage.py setup_local_api_key' in the posthog repo." >&2+ echo "Set POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY to a phs_ key or a phe_ token from 'hogli gateway:token'." >&2exit 1fiexport POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY-echo "token: ${POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY:0:8}… gateway: ${POSTHOG_CODE_E2E_GATEWAY_URL:-http://localhost:3308/ci}"+echo "gateway: ${POSTHOG_CODE_E2E_GATEWAY_URL:-http://localhost:8080}"cd "$AGENT_DIR"pnpm test:e2e "$@"