Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(desktop): run the live agent e2e against the Go AI gateway (#124920)

Jonathan Mieloo873aa828cd8c
diff --git a/.github/workflows/desktop-ci.yml b/.github/workflows/desktop-ci.yml
index b767c3f93fab..04c14712e410 100644
--- a/.github/workflows/desktop-ci.yml
+++ b/.github/workflows/desktop-ci.yml
@@ -52,7 +52,6 @@ jobs:
secrets:
GH_APP_POSTHOG_PATHS_FILTER_PRIVATE_KEY: ${{ secrets.GH_APP_POSTHOG_PATHS_FILTER_PRIVATE_KEY }}
POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY: ${{ secrets.POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY }}
- POSTHOG_CODE_E2E_GATEWAY_URL: ${{ secrets.POSTHOG_CODE_E2E_GATEWAY_URL }}
# Desktop counterpart of the migration/service separation check (see the
# script header for why). Lives here rather than pr-housekeeping so the
diff --git a/.github/workflows/desktop-test.yml b/.github/workflows/desktop-test.yml
index e036bd2f925f..19aeb8451706 100644
--- a/.github/workflows/desktop-test.yml
+++ b/.github/workflows/desktop-test.yml
@@ -9,8 +9,6 @@ on:
required: false
POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY:
required: false
- POSTHOG_CODE_E2E_GATEWAY_URL:
- required: false
# Master coverage comes from this cron and from the release tag job, which
# dispatches this workflow on the tree it tags. Pull requests and merge queue
# batches run the suite per change, so the cron is here to catch drift from
@@ -269,8 +267,7 @@ jobs:
# after the unit + Electron jobs pass — a red tree never reaches the
# gateway — and only when a products/desktop/packages/ file changed. Each arm still
# self-skips unless the POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY secret is
- # present (fork PRs never see it) and POSTHOG_CODE_E2E_GATEWAY_URL points at
- # a runner-reachable gateway. Drives cheap models (claude-haiku-4-5 /
+ # present (fork PRs never see it). Drives cheap models (claude-haiku-4-5 /
# gpt-5-mini), so a run is a handful of short turns.
needs: [changes, unit-test, electron-test]
# Always run on any products/desktop/packages/ change (or a change to this workflow),
@@ -351,13 +348,10 @@ jobs:
working-directory: packages/agent
run: pnpm --filter agent run test:e2e
# The suite reads these POSTHOG_CODE_E2E_* names directly
- # (packages/agent/packages/agent/e2e/config.ts). The personal API key is an org
- # secret; the URL and model/environment overrides are org vars.
+ # (packages/agent/packages/agent/e2e/config.ts).
env:
POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY: ${{ secrets.POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY }}
- # Accept the URL from either an org var or an org secret — it has ended
- # up in both places, and vars.* can't see secrets.
- POSTHOG_CODE_E2E_GATEWAY_URL: ${{ vars.POSTHOG_CODE_E2E_GATEWAY_URL || secrets.POSTHOG_CODE_E2E_GATEWAY_URL }}
+ POSTHOG_CODE_E2E_GATEWAY_URL: https://ai-gateway.us.posthog.com
POSTHOG_CODE_E2E_CLAUDE_MODEL: ${{ vars.POSTHOG_CODE_E2E_CLAUDE_MODEL }}
POSTHOG_CODE_E2E_CODEX_MODEL: ${{ vars.POSTHOG_CODE_E2E_CODEX_MODEL }}
# Optional: set vars.POSTHOG_CODE_E2E_ENVIRONMENT=cloud to exercise the
diff --git a/packages/agent/packages/agent/e2e/README.md b/packages/agent/packages/agent/e2e/README.md
index 48ea2b1edd97..5234316355c8 100644
--- a/packages/agent/packages/agent/e2e/README.md
+++ b/packages/agent/packages/agent/e2e/README.md
@@ -1,7 +1,7 @@
# Live agent e2e suite
Drives representative sessions **end to end** through the real adapter, the real
-binary (codex `app-server` / Claude Code CLI), and the real llm-gateway on a cheap
+binary (codex `app-server` / Claude Code CLI), and the real ai-gateway on a cheap
model — parametrized across `claude` and `codex`. The only thing mocked is the
host/UI client (a recording `sessionUpdate`, an auto-allow `requestPermission`,
and real file read/write against a throwaway git repo). Nothing in the
@@ -72,8 +72,7 @@ includes `src/**`). They are opt-in and cost a couple of short model turns.
In CI they run as the **`e2e` job in `.github/workflows/test.yml`**, on pull
requests only, after the unit + integration jobs pass. The job is opt-in and safe
by default: it self-skips unless the repo variable `AGENT_E2E_ENABLED` is `true`
-with an `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY` secret and an `POSTHOG_CODE_E2E_GATEWAY_URL` variable pointing at a
-gateway reachable from the runner, and it never runs for fork PRs (their secrets
+with a `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY` secret against `https://ai-gateway.us.posthog.com`, and it never runs for fork PRs (their secrets
are withheld, which would otherwise red the fail-loud token guard). Off by
default, so it costs nothing until explicitly enabled; the codex arm self-skips if
the native binary isn't on the runner.
@@ -86,7 +85,7 @@ bash e2e/run-e2e.sh
bash e2e/run-e2e.sh -t "(codex)"
```
-Prereqs: a local llm-gateway up (`./bin/start` in the posthog repo) and the
+Prereqs: a local ai-gateway up (`bin/start-ai-gateway`) and the
native codex binary present at `apps/code/resources/codex-acp/codex` (the codex
arm self-skips if it is missing).
@@ -94,8 +93,8 @@ arm self-skips if it is missing).
| Var | Default | Notes |
| --- | --- | --- |
-| `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY` | — | Required. A token the gateway accepts — the `ci` product takes a personal API key (no OAuth). Without it every arm skips. `run-e2e.sh` reads the local dev key. |
-| `POSTHOG_CODE_E2E_GATEWAY_URL` | `http://localhost:3308/ci` | Gateway base (codex appends `/v1`). `ci` accepts a personal API key; `posthog_code` is OAuth-only. |
+| `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY` | — | Required. A `phs_` key, or a `phe_` dev token from `hogli gateway:token`. Requests carry `X-PostHog-Product: ci`. Without it every arm skips. |
+| `POSTHOG_CODE_E2E_GATEWAY_URL` | `http://localhost:8080` | ai-gateway base (codex appends `/v1`). |
| `POSTHOG_CODE_E2E_CLAUDE_MODEL` | `claude-haiku-4-5` | Override if the gateway serves a different cheap Claude id. |
| `POSTHOG_CODE_E2E_CODEX_MODEL` | `gpt-5-mini` | Cheapest codex id the local gateway serves; override if needed. |
| `POSTHOG_CODE_E2E_CLAUDE_STRONG_MODEL` | `claude-sonnet-4-5` | Stronger Claude id for tests the cheap model can't handle (structured output). |
diff --git a/packages/agent/packages/agent/e2e/config.ts b/packages/agent/packages/agent/e2e/config.ts
index ac4f375b43dd..f8fa368b072a 100644
--- a/packages/agent/packages/agent/e2e/config.ts
+++ b/packages/agent/packages/agent/e2e/config.ts
@@ -6,16 +6,15 @@ export type { Adapter };
/**
* Live e2e configuration, resolved entirely from the environment so no secret is
- * committed. Needs a local llm-gateway and a token in `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY`; targets
- * the `ci` product, which accepts a personal API key (no OAuth mint,
- * unlike prod's `posthog_code`) and keeps e2e traffic attributed to its own
- * `ai_product` rather than the catch-all `llm_gateway` bucket. Without the token
- * every arm self-skips.
+ * committed. Needs an ai-gateway and a `phs_` key in `POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY`; sends
+ * `X-PostHog-Product: ci` so e2e traffic is attributed to its own product.
+ * Without the token every arm self-skips.
*/
// `||` not `??`: CI sets unset vars to "" which should fall back to the default.
const GATEWAY_URL =
- process.env.POSTHOG_CODE_E2E_GATEWAY_URL || "http://localhost:3308/ci";
+ process.env.POSTHOG_CODE_E2E_GATEWAY_URL || "http://localhost:8080";
const TOKEN = process.env.POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY ?? "";
+const GATEWAY_PRODUCT = "ci";
// The desktop app downloads the codex binaries; it lives in the sibling desktop workspace.
const CODEX_RESOURCES_DIR = join(
@@ -92,6 +91,7 @@ export const E2E = {
if (adapter === "claude") {
process.env.ANTHROPIC_BASE_URL = GATEWAY_URL;
process.env.ANTHROPIC_AUTH_TOKEN = TOKEN;
+ process.env.ANTHROPIC_CUSTOM_HEADERS = `X-PostHog-Product: ${GATEWAY_PRODUCT}`;
return;
}
process.env.OPENAI_BASE_URL = openAiBase();
@@ -108,6 +108,7 @@ export const E2E = {
binaryPath: string;
apiBaseUrl: string;
apiKey: string;
+ httpHeaders: Record<string, string>;
model: string;
configOverrides?: Record<string, string | number>;
} {
@@ -116,6 +117,7 @@ export const E2E = {
binaryPath: NATIVE_CODEX_BIN,
apiBaseUrl: openAiBase(),
apiKey: TOKEN,
+ httpHeaders: { "X-PostHog-Product": GATEWAY_PRODUCT },
model: modelOverride || this.model("codex"),
...(configOverrides ? { configOverrides } : {}),
};
diff --git a/packages/agent/packages/agent/e2e/guard.e2e.test.ts b/packages/agent/packages/agent/e2e/guard.e2e.test.ts
index 8130aea94528..8daac1981d09 100644
--- a/packages/agent/packages/agent/e2e/guard.e2e.test.ts
+++ b/packages/agent/packages/agent/e2e/guard.e2e.test.ts
@@ -11,8 +11,8 @@ describe("live e2e preconditions", () => {
expect(
E2E.hasToken,
"POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY is not set — every adapter arm would skip and the run " +
- "would pass without testing anything. Mint one via e2e/run-e2e.sh or " +
- "set POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY against a reachable POSTHOG_CODE_E2E_GATEWAY_URL.",
+ "would pass without testing anything. " +
+ "Set POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY against a reachable POSTHOG_CODE_E2E_GATEWAY_URL.",
).toBe(true);
});
@@ -25,7 +25,7 @@ describe("live e2e preconditions", () => {
process.env.POSTHOG_CODE_E2E_GATEWAY_URL,
"POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY is set but " +
"POSTHOG_CODE_E2E_GATEWAY_URL is empty — on CI the suite would fall " +
- "back to localhost:3308 (unreachable from a runner) and every model " +
+ "back to localhost:8080 (unreachable from a runner) and every model " +
"turn would fail with ConnectionRefused. Set the org variable to a " +
"runner-reachable gateway.",
).toBeTruthy();
diff --git a/packages/agent/packages/agent/e2e/run-e2e.sh b/packages/agent/packages/agent/e2e/run-e2e.sh
index df8a20fc7840..3785ef69c635 100755
--- a/packages/agent/packages/agent/e2e/run-e2e.sh
+++ b/packages/agent/packages/agent/e2e/run-e2e.sh
@@ -1,12 +1,8 @@
#!/usr/bin/env bash
# Run the live golden-path e2e for both adapters (claude + codex).
#
-# Needs a local llm-gateway (run `./bin/start` in the posthog repo) and a token.
-# The suite targets the gateway's `ci` product, which accepts a personal
-# API key (no OAuth), so if POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY is unset this reads the repo's
-# hardcoded local dev key from ee/settings.py (override the repo with POSTHOG_REPO).
-# That key must be registered in the local DB — run `python manage.py
-# setup_local_api_key` in the posthog repo once if auth fails.
+# Needs an ai-gateway and a token in POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY: a `phs_`
+# key, or a `phe_` dev token from `hogli gateway:token` in the posthog repo.
#
# Usage:
# bash e2e/run-e2e.sh # both adapters, both suites
@@ -16,26 +12,13 @@ set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
AGENT_DIR="$(cd "$HERE/.." && pwd)"
-POSTHOG_REPO="${POSTHOG_REPO:-$(cd "$AGENT_DIR/../../.." && pwd)}"
if [[ -z "${POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY:-}" ]]; then
- SETTINGS="$POSTHOG_REPO/ee/settings.py"
- if [[ ! -f "$SETTINGS" ]]; then
- echo "POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY unset and posthog settings not found at $SETTINGS." >&2
- echo "Set POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY, or POSTHOG_REPO to the posthog checkout." >&2
- exit 1
- fi
- # The `ci` product accepts personal API keys, so no OAuth mint needed.
- POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY="$(grep -E '^DEV_API_KEY[[:space:]]*=' "$SETTINGS" | head -1 | sed -E 's/^DEV_API_KEY[[:space:]]*=[[:space:]]*"([^"]+)".*/\1/')"
-fi
-
-if [[ -z "${POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY:-}" ]]; then
- echo "Failed to obtain an POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY (no DEV_API_KEY in ee/settings.py?)." >&2
- echo "If auth then fails, run 'python manage.py setup_local_api_key' in the posthog repo." >&2
+ echo "Set POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY to a phs_ key or a phe_ token from 'hogli gateway:token'." >&2
exit 1
fi
export POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY
-echo "token: ${POSTHOG_CODE_E2E_GATEWAY_PERSONAL_API_KEY:0:8}… gateway: ${POSTHOG_CODE_E2E_GATEWAY_URL:-http://localhost:3308/ci}"
+echo "gateway: ${POSTHOG_CODE_E2E_GATEWAY_URL:-http://localhost:8080}"
cd "$AGENT_DIR"
pnpm test:e2e "$@"