:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(evals): route sandboxed eval agents through the go ai-gateway (#118445)
Daniel Parkerb92cad71d864
diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.mdindex 755d2b98c1ca..4527f4517ec1 100644--- a/products/posthog_ai/eval_harness/README.md+++ b/products/posthog_ai/eval_harness/README.md@@ -7,11 +7,11 @@ The harness builds product skills from the current checkout and uses native bund`--skill-delivery exec` instead packages those skills for MCP distribution, enables the exec skill prompt, and removes native skills from every sandbox so the two paths cannot satisfy the same case.This is the eval harness for PostHog AI 2.0. It does **not** run under pytest.-It runs on a standalone harness that boots the shared infrastructure once (test database, Django live server, LLM gateway, MCP server, Temporal) and then runs every selected suite concurrently.+It runs on a standalone harness that boots the shared infrastructure once (test database, Django live server, AI gateway, MCP server, Temporal) and then runs every selected suite concurrently.See [`harness/README.md`](harness/README.md) for how that works internally.The harness also hosts non-sandboxed suite kinds — see [Suite kinds](#suite-kinds).-A run boots only the infrastructure its selected suites need, so a one-shot-only run starts no sandbox provider, Temporal, live server, LLM gateway, or MCP server.+A run boots only the infrastructure its selected suites need, so a one-shot-only run starts no sandbox provider, Temporal, live server, AI gateway, or MCP server.Two other docs sit next to this one:@@ -37,7 +37,7 @@ python -m products.posthog_ai.eval_harness.harness [SELECTOR ...] [flags]No manual env sourcing is needed on either path.The harness loads the repo-root `.env` itself (shell values win), and `hogli evals` additionally layers in `.env.local` / `.env.development` / `.env.services` through hogli's standard env loading — including 1Password resolution when `.env.local` holds `op://` references.Before any infrastructure boots, a preflight validates that the required variables are set and fails with a one-line fix per missing variable.-Which variables are required depends on the eval engine and the selected suites' kinds: the braintrust engine (the default, and only one today) requires `BRAINTRUST_API_KEY` on every run — it is the engine's own `required_env()`, not a core harness variable; sandboxed suites add `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`; one-shot suites add `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).+Which variables are required depends on the eval engine and the selected suites' kinds: the braintrust engine (the default, and only one today) requires `BRAINTRUST_API_KEY` on every run — it is the engine's own `required_env()`, not a core harness variable; sandboxed suites add `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`, and need a [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway) clone plus `go` on `PATH` (see [AI gateway](#ai-gateway)); one-shot suites add `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).Selectors are substrings matched against a suite id of the form `<domain>/<module>::<fn>`, for example `experiments`, `sql`, or `eval_lifecycle_skills`.Omit them to run every suite.@@ -117,6 +117,21 @@ Dependency warnings, SDK progress bars, and direct setup prints stay in the tran`--verbose` also shows Python warnings, SDK output, and detailed logs in the console.Native subprocess output always goes to the transcript.+### AI gateway++Sandboxed agents call their model through the Go [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway), as PostHog AI runs do in production.+The harness builds and starts it from a local clone at `~/Development/ai-gateway`, the same path `bin/start-ai-gateway` uses. Set `AI_GATEWAY_REPO` to use a clone at a different path.++```bash+git clone [email protected]:PostHog/ai-gateway.git ~/Development/ai-gateway+```++The gateway runs on `:18080` with its own Postgres database and Valkey DB index in the clone's compose containers, so it does not touch a dev-stack gateway's ledger or blobs.+It uses `LLM_GATEWAY_ANTHROPIC_API_KEY` and `LLM_GATEWAY_OPENAI_API_KEY` as its provider keys.+The harness adds test credits to the master demo team in this separate ledger. Scoped tokens require a positive balance and cannot use an overspend allowance.+Each run gets a capped `phe_` token minted with a key on that team, so spend is per run and never reaches a real wallet. The harness refreshes the mint key during long runs. It retries temporary database or Redis errors until the credential expires. Retry delays increase to at most one minute.+If token minting fails, the fallback URL stays on the eval gateway, which rejects requests without a run token.+### Interactive viewThe view is `phrocs evals`, the same Bubble Tea program that renders `hogli start`.@@ -170,7 +185,7 @@ The experiment metadata records `skill_delivery`; compare the shared `expected_s### Codex runtime`--agent-runtime codex` runs the same agent-server with OpenAI's Codex harness instead of Claude, defaulting the model to `gpt-6.1-sol`.-It requires `LLM_GATEWAY_OPENAI_API_KEY` in the environment (checked by preflight), which the harness's LLM gateway uses to proxy the agent's OpenAI calls.+It requires `LLM_GATEWAY_OPENAI_API_KEY` in the environment (checked by preflight), which the harness's AI gateway uses to proxy the agent's OpenAI calls.Experiment names don't change with the runtime or skill delivery. Each Braintrust experiment and every PostHog event records `agent_model`, `trials`, `git_sha` and `git_dirty`. Sandboxed runs add `agent_runtime`, `skill_delivery` and `reasoning_effort`. Use these fields to separate runs. A comparison across runtimes measures the model and runtime together.### Claude and Codex smoke comparison@@ -226,7 +241,7 @@ The first modal run pays a one-time remote image build; later runs reuse the cacModal prerequisites, all checked by preflight before anything boots:-- `tailscale` on `PATH`, with tailscaled running and this node logged in (`tailscale up`). Funnel must be enabled for the tailnet and this node in the admin console's ACLs, with HTTPS certificates enabled. The harness serves Django, the LLM gateway, and MCP on Funnel's three public ports (443, 8443, 10000).+- `tailscale` on `PATH`, with tailscaled running and this node logged in (`tailscale up`). Funnel must be enabled for the tailnet and this node in the admin console's ACLs, with HTTPS certificates enabled. The harness serves Django, the AI gateway, and MCP on Funnel's three public ports (443, 8443, 10000).- Modal credentials: `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`, or a `~/.modal.toml` from `modal token new`.- `SANDBOX_JWT_PRIVATE_KEY` in the environment. The dev key ships in `.env.example`; the harness auto-loads it from `.env`.diff --git a/products/posthog_ai/eval_harness/harness/AGENTS.md b/products/posthog_ai/eval_harness/harness/AGENTS.mdindex 61d61e0ee416..d0ef3b0bb86c 100644--- a/products/posthog_ai/eval_harness/harness/AGENTS.md+++ b/products/posthog_ai/eval_harness/harness/AGENTS.md@@ -90,7 +90,7 @@ Braintrust's `EvalAsync` always dispatches through `eval_async`, and the base `SBootstrap registers teardown on an `ExitStack` as each resource comes up, so a failure halfway through still unwinds what already started.`atexit` hooks plus the subprocess manager's signal handlers cover Ctrl-C, where neither stack unwinds.-After any change, a Ctrl-C mid-run must leave no listeners on 18000 / 13308 / 18787 / 18788 / 15051 / 15052, no `task-sandbox-*` containers, no Temporal dev server, and no Tailscale Funnel mappings left enabled on 443 / 8443 / 10000.+After any change, a Ctrl-C mid-run must leave no listeners on 18000 / 18080 / 18787 / 18788 / 15051 / 15052, no `task-sandbox-*` containers, no Temporal dev server, and no Tailscale Funnel mappings left enabled on 443 / 8443 / 10000.A case can also own a sandbox the harness never created: a notebook python or duckdb cell provisions the notebook kernel through the notebook Temporal workflow.Nothing upstream reclaims it — the docker backend ignores `SandboxConfig.ttl_seconds`, and the sweeps below match agent containers by task id, which a kernel container's name never carries.diff --git a/products/posthog_ai/eval_harness/harness/README.md b/products/posthog_ai/eval_harness/harness/README.mdindex 340822493973..40aa514da76a 100644--- a/products/posthog_ai/eval_harness/harness/README.md+++ b/products/posthog_ai/eval_harness/harness/README.md@@ -7,7 +7,7 @@ This file explains how it fits together.## Why it exists-The sandboxed evals need heavy shared infrastructure: a committed test database, an in-process Django server the sandbox can call back into, an LLM gateway, an MCP server, and a Temporal server plus worker.+The sandboxed evals need heavy shared infrastructure: a committed test database, an in-process Django server the sandbox can call back into, the Go AI gateway, an MCP server, and a Temporal server plus worker.Under pytest that infrastructure lived in session fixtures, and suites ran strictly one at a time, with Braintrust parallelizing only _within_ a suite.Wall-clock was dominated by suites queueing behind each other while sandbox capacity sat idle.@@ -16,32 +16,32 @@ Braintrust remains the eval engine and reporting backend; it just no longer contThe `EvalAsync` call itself lives behind an `EvalEngine` seam (`../engines/`): `_BaseEvalRun.run()` hands a neutral `ExperimentSpec` to the engine resolved by `engines/registry.resolve_engine()` (shared via `EvalContext.engine`) and gets back a neutral `ExperimentResult` (`engines/types.py`), so a future PostHog-native engine can slot in behind the same interface — implementing the `EvalEngine` protocol and passing the conformance suite — without touching the run base or any suite.Not every suite is sandboxed anymore: each suite declares a `SUITE_KIND` (`requirements.py`), the kind maps to a set of `Infra` requirements, and the harness boots only the union of what the selected suites need.-A run of one-shot suites never pays for — or fails preflight on — the sandbox provider, Temporal, the live server, the LLM gateway, or the MCP server.+A run of one-shot suites never pays for — or fails preflight on — the sandbox provider, Temporal, the live server, the AI gateway, or the MCP server.## Modules-| Module | Role |-| ------------------ | --------------------------------------------------------------------------------------------------------------------------------- |-| `__main__.py` | Entry point. Parses args, starts the transcript, configures Django, then hands off to `lifecycle`. |-| `cli.py` | `HarnessOptions` and the argparse builder. Resolves per-provider defaults. |-| `env_preflight.py` | Loads the repo-root `.env` (stdlib parser, shell wins) and validates per-kind env vars via pydantic models. |-| `ports.py` | The seven port constants. Deliberately free of Django imports. |-| `providers.py` | `SandboxProviderStrategy` and its docker/modal implementations: preflight, settings overrides, sandbox TTL, cleanup. |-| `tunnels.py` | `TailscaleFunnel`. Modal only: enables Tailscale Funnel on the host services, waits for the public relays, and resolves the URLs. |-| `requirements.py` | `SuiteKind`, `Infra`, and the kind → infrastructure mapping (with implication closure). |-| `django_env.py` | `setup_django()`, the `NullDbBlocker` shim, and `EvalDatabase` (test database lifecycle). |-| `live_server.py` | `EvalLiveServer`, a session-lifetime Uvicorn server for PostHog's full ASGI application. |-| `services.py` | Starts the LLM gateway, MCP server, skill archive server, and personhog subprocesses; builds local skills. |-| `temporal_env.py` | Local Temporal dev server, stale-workflow cleanup, and the worker thread. |-| `demo_data.py` | `SandboxedDemoData`: seeds the master Hedgebox team once, then mints an isolated team per case. |-| `discovery.py` | Walks the tree for `eval_*.py` and collects `eval_*` coroutines into `EvalSuite` objects. |-| `context.py` | `EvalContext`, the single object every suite receives. |-| `reporting.py` | `ProgressReporter` and the final summary table. The quiet Braintrust reporter lives with the engine (`../engines/braintrust.py`). |-| `transcript.py` | Mirrors stdout and stderr to one run log, then publishes its absolute path as the final line. |-| `dashboard.py` | Starts the `phrocs evals` view process and streams run progress to it over a socket pair. |-| `progress.py` | Immutable trial progress shared by the reporter and interactive view. |-| `terminal.py` | Opens terminal streams without sharing the descriptors used by transcript capture. |-| `lifecycle.py` | `SandboxedEvalHarness`: orchestrates bootstrap, the run, and teardown. |+| Module | Role |+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |+| `__main__.py` | Entry point. Parses args, starts the transcript, configures Django, then hands off to `lifecycle`. |+| `cli.py` | `HarnessOptions` and the argparse builder. Resolves per-provider defaults. |+| `env_preflight.py` | Loads the repo-root `.env` (stdlib parser, shell wins) and validates per-kind env vars via pydantic models. |+| `ports.py` | The seven port constants. Deliberately free of Django imports. |+| `providers.py` | `SandboxProviderStrategy` and its docker/modal implementations: preflight, settings overrides, sandbox TTL, cleanup. |+| `tunnels.py` | `TailscaleFunnel`. Modal only: enables Tailscale Funnel on the host services, waits for the public relays, and resolves the URLs. |+| `requirements.py` | `SuiteKind`, `Infra`, and the kind → infrastructure mapping (with implication closure). |+| `django_env.py` | `setup_django()`, the `NullDbBlocker` shim, and `EvalDatabase` (test database lifecycle). |+| `live_server.py` | `EvalLiveServer`, a session-lifetime Uvicorn server for PostHog's full ASGI application. |+| `services.py` | Starts the AI gateway, MCP server, skill archive server, and personhog subprocesses; provisions the gateway mint key; builds local skills. |+| `temporal_env.py` | Local Temporal dev server, stale-workflow cleanup, and the worker thread. |+| `demo_data.py` | `SandboxedDemoData`: seeds the master Hedgebox team once, then mints an isolated team per case. |+| `discovery.py` | Walks the tree for `eval_*.py` and collects `eval_*` coroutines into `EvalSuite` objects. |+| `context.py` | `EvalContext`, the single object every suite receives. |+| `reporting.py` | `ProgressReporter` and the final summary table. The quiet Braintrust reporter lives with the engine (`../engines/braintrust.py`). |+| `transcript.py` | Mirrors stdout and stderr to one run log, then publishes its absolute path as the final line. |+| `dashboard.py` | Starts the `phrocs evals` view process and streams run progress to it over a socket pair. |+| `progress.py` | Immutable trial progress shared by the reporter and interactive view. |+| `terminal.py` | Opens terminal streams without sharing the descriptors used by transcript capture. |+| `lifecycle.py` | `SandboxedEvalHarness`: orchestrates bootstrap, the run, and teardown. |## Boot sequence@@ -52,14 +52,16 @@ The sequence as written is the sandboxed (full) path; a one-shot-only run perforThe bootstrap is deliberately synchronous and runs before any event loop exists, because it is ORM-heavy and Django's async-safety guard rejects sync ORM calls from an async context:1. `__main__` creates the run transcript, then loads the repo-root `.env` (never overriding what the shell — or hogli's own env loading, under `hogli evals` — already exported), then `setup_django()` sets `DEBUG` / `TEST` / `IN_EVAL_TESTING`, forces `SELF_CAPTURE=0`, and runs `django.setup()` and `setup_test_environment()`. Eval telemetry still uses the explicit regional client configured by the harness.- The env preflight (required variables, one-line fix per missing one) and provider preflight then run, followed by the personhog binary build (`cargo build`, incremental after the first run) — a missing key or toolchain fails here, before any database work.+ The env preflight (required variables, one-line fix per missing one) and provider preflight then run, followed by the personhog binary build (`cargo build`, incremental after the first run) and the AI gateway build (`go build` in the `PostHog/ai-gateway` clone) — a missing key or toolchain fails here, before any database work.2. `EvalDatabase.setup()` creates the `default` test database and drives PostHog's own eval database setup (persons database, ClickHouse).3. `personhog-replica` (`:15051`) and `personhog-router` (`:15052`) start against the test persons database — before anything can query, so a dead router never poisons the negative group-types cache.4. `EvalLiveServer` serves PostHog's ASGI application on `0.0.0.0:18000`, including the sandbox event-ingest route.-5. The LLM gateway (`:13308`) starts and local skills are built. In bundled mode, the MCP server (`:18787`) starts with exec skill distribution forced off. In exec mode, the skills are packaged, the local archive server (`:18788`) starts, and MCP receives its content-addressed URL with distribution forced on.+5. The Go AI gateway (`:18080`) starts in resolver mode, on the gateway compose Postgres (database `ai_gateway_evals`) and Valkey (DB 5), and local skills are built. In bundled mode, the MCP server (`:18787`) starts with exec skill distribution forced off. In exec mode, the skills are packaged, the local archive server (`:18788`) starts, and MCP receives its content-addressed URL with distribution forced on.6. Docker only: the `posthog-sandbox-base` image freshness check runs (rebuilding when the Dockerfile or its agent pin changed). Modal only: Tailscale Funnel comes up, exposing the callback services publicly. Setup then waits until every public Funnel relay serves every service, because the relays start serving at different times and can take minutes.7. Bundled mode bind-mounts the built skills on Docker and bakes them into the Modal image. Exec mode suppresses those skills and clears native skill directories before every case; `disable_bundled_skills=True` remains available as a per-case override in other modes.-8. The master Hedgebox team is seeded.+8. The master Hedgebox team is seeded. It then gets the AI gateway mint key: a `phs_` key, gateway admission, and test credits in the separate eval ledger. One model listing and one token mint check that path before any case starts. The harness refreshes the same mint credential before its cache TTL expires during the run.++Gateway cache writes stay disabled until the gateway starts. A one-shot run leaves them disabled.The async phase then starts the Temporal dev server on the main loop, applies the provider's settings overrides, terminates stale workflows, starts the Temporal worker on its own thread and loop, and fans the suites out with `asyncio.gather`.diff --git a/products/posthog_ai/eval_harness/harness/__init__.py b/products/posthog_ai/eval_harness/harness/__init__.pyindex 85e1721612a9..f478439f869d 100644--- a/products/posthog_ai/eval_harness/harness/__init__.py+++ b/products/posthog_ai/eval_harness/harness/__init__.py@@ -1,7 +1,7 @@"""Standalone runner for the sandboxed agent evals.Replaces pytest as the driver: boots the shared session infrastructure once-(test DB, Django live server, LLM gateway, MCP server, Temporal), then runs+(test DB, Django live server, AI gateway, MCP server, Temporal), then runsevery selected eval suite concurrently on one event loop with a single globalsandbox semaphore bounding total load. Braintrust stays as the eval engine.diff --git a/products/posthog_ai/eval_harness/harness/__main__.py b/products/posthog_ai/eval_harness/harness/__main__.pyindex de613775f5ce..a951b5d80b93 100644--- a/products/posthog_ai/eval_harness/harness/__main__.py+++ b/products/posthog_ai/eval_harness/harness/__main__.py@@ -9,7 +9,7 @@ from typing import TYPE_CHECKINGfrom .cli import HarnessOptions, parse_argsfrom .django_env import setup_djangofrom .env_preflight import load_env_file-from .ports import PERSONHOG_ROUTER_PORT+from .ports import AI_GATEWAY_REDIS_URL, PERSONHOG_ROUTER_PORTfrom .providers import SANDBOX_PROVIDER_SETTING, PreflightErrorfrom .terminal import open_terminalfrom .transcript import RunTranscript, write_console@@ -47,6 +47,10 @@ def _run(# would otherwise silently point eval reads at the dev persons DB.os.environ["PERSONHOG_ADDR"] = f"127.0.0.1:{PERSONHOG_ROUTER_PORT}"+ # HyperCache selects its cache alias at import time. Bootstrap enables+ # gateway projections only after the selected suites start the gateway Redis.+ os.environ["AI_GATEWAY_REDIS_URL"] = AI_GATEWAY_REDIS_URL+setup_django(keep_logging=transcript is not None)if transcript is not None:_configure_logging(transcript, verbose=options.verbose)diff --git a/products/posthog_ai/eval_harness/harness/env_preflight.py b/products/posthog_ai/eval_harness/harness/env_preflight.pyindex cc7a81148a1e..f8ba39603da1 100644--- a/products/posthog_ai/eval_harness/harness/env_preflight.py+++ b/products/posthog_ai/eval_harness/harness/env_preflight.py@@ -34,7 +34,7 @@ class SandboxEvalEnv(BaseModel):)LLM_GATEWAY_ANTHROPIC_API_KEY: str = Field(min_length=1,- description="Anthropic API key the LLM gateway proxies the agent's model calls with",+ description="Anthropic API key the AI gateway proxies the agent's model calls with",)@@ -56,7 +56,7 @@ class CodexEvalEnv(BaseModel):LLM_GATEWAY_OPENAI_API_KEY: str = Field(min_length=1,- description="OpenAI API key the LLM gateway proxies the codex agent's model calls with",+ description="OpenAI API key the AI gateway proxies the codex agent's model calls with",)diff --git a/products/posthog_ai/eval_harness/harness/lifecycle.py b/products/posthog_ai/eval_harness/harness/lifecycle.pyindex ebac910a0768..f13387707326 100644--- a/products/posthog_ai/eval_harness/harness/lifecycle.py+++ b/products/posthog_ai/eval_harness/harness/lifecycle.py@@ -42,7 +42,7 @@ from .env_preflight import validate_eval_envfrom .kernel_sandboxes import reclaim_kernelsfrom .live_server import EvalLiveServerfrom .offline_delivery import OfflineEvalDelivery-from .ports import DJANGO_LIVE_PORT+from .ports import AI_GATEWAY_REDIS_URL, DJANGO_LIVE_PORTfrom .providers import PreflightError, SandboxProviderStrategy, build_providerfrom .reporting import ProgressReporter, SuiteRunResultfrom .requirements import Infra, infra_union@@ -50,11 +50,15 @@ from .requirements import Infra, infra_unionif TYPE_CHECKING:from .dashboard import EvalDashboardfrom .services import (+ EVAL_AI_PRODUCT,LONG_LIVED_SUBPROCESSES,build_local_skills,+ ensure_ai_gateway_binary,ensure_personhog_binaries,+ maintain_ai_gateway_mint_key,package_local_skills_archive,- start_llm_gateway,+ provision_ai_gateway_mint_key,+ start_ai_gateway,start_mcp_server,start_personhog,start_skill_archive_server,@@ -121,6 +125,7 @@ class SandboxedEvalHarness:self._posthog_client: Posthog | None = Noneself._demo_data: SandboxedDemoData | None = Noneself._offline_delivery: OfflineEvalDelivery | None = None+ self._ai_gateway_mint_key: str | None = None# Capture the revision before any eval starts so every result describes the same run.self._git_sha = _git("rev-parse", "HEAD")self._git_dirty = _worktree_dirty()@@ -189,6 +194,9 @@ class SandboxedEvalHarness:if Infra.PERSONHOG in required:reporter.setup_started("build personhog")ensure_personhog_binaries()+ if Infra.AI_GATEWAY in required:+ reporter.setup_started("build ai-gateway")+ ensure_ai_gateway_binary()reporter.setup_started("start shared services and prepare demo data")self._bootstrap(required, reporter)if self._dashboard:@@ -262,6 +270,7 @@ class SandboxedEvalHarness:def _bootstrap(self, required: frozenset[Infra], reporter: ProgressReporter) -> None:"""Stand up everything the selected suites share. Teardown is registered inreverse on the stack, so what wasn't started is never torn down."""+ self._stack.enter_context(override_settings(AI_GATEWAY_REDIS_URL=None))LONG_LIVED_SUBPROCESSES.startup_retries = self.options.infrastructure_retries# Belt and braces for the SIGINT / SIGTERM paths, where the ExitStack never unwinds.atexit.register(stop_all_subprocesses)@@ -289,10 +298,10 @@ class SandboxedEvalHarness:self._stack.callback(live_server.stop)self._live_server = live_server- if Infra.LLM_GATEWAY in required:- reporter.setup_started("start LLM gateway")- assert self._live_server is not None- self._stack.callback(start_llm_gateway(self._live_server.url))+ if Infra.AI_GATEWAY in required:+ reporter.setup_started("start AI gateway")+ self._stack.callback(start_ai_gateway())+ self._stack.enter_context(override_settings(AI_GATEWAY_REDIS_URL=AI_GATEWAY_REDIS_URL))if Infra.MCP_SERVER in required:reporter.setup_started("build skills and start MCP server")assert self._live_server is not None@@ -341,10 +350,22 @@ class SandboxedEvalHarness:),)+ if Infra.AI_GATEWAY in required:+ assert self._demo_data is not None and self._database is not None+ with self._database.blocker.unblock():+ self._ai_gateway_mint_key = provision_ai_gateway_mint_key(+ self._demo_data.master_team_id, agent_model=self.options.agent_model+ )+async def _run_suites(self, suites: Sequence[EvalSuite], required: frozenset[Infra], reporter: ProgressReporter) -> list[SuiteRunResult]:async with AsyncExitStack() as stack:+ if Infra.AI_GATEWAY in required:+ assert self._demo_data is not None and self._ai_gateway_mint_key is not None+ await stack.enter_async_context(+ maintain_ai_gateway_mint_key(self._demo_data.master_team_id, self._ai_gateway_mint_key)+ )overrides: dict[str, object] = {}if Infra.LIVE_SERVER in required:overrides.update(@@ -375,6 +396,10 @@ class SandboxedEvalHarness:# The harness never starts run_task_workflow_dispatcher, so an outbox dispatch would# leave every case parked in QUEUED until the case's poll budget runs out.TASKS_WORKFLOW_DISPATCH_OUTBOX_ENABLED=False,+ # Only the eval product routes to the Go gateway. The worker mints each+ # run a capped scoped token with this key, as it does in production.+ SANDBOX_AI_GATEWAY_PRODUCTS=EVAL_AI_PRODUCT,+ SANDBOX_AI_GATEWAY_MINT_KEY=self._ai_gateway_mint_key,**self.provider.settings_overrides(),)diff --git a/products/posthog_ai/eval_harness/harness/ports.py b/products/posthog_ai/eval_harness/harness/ports.pyindex 59ac1ebc7833..9b1c70b702e1 100644--- a/products/posthog_ai/eval_harness/harness/ports.py+++ b/products/posthog_ai/eval_harness/harness/ports.py@@ -9,8 +9,15 @@ SKILL_ARCHIVE_PORT = 18788DJANGO_LIVE_PORT = 18000"""Non-default port for the in-process Django server."""-LLM_GATEWAY_PORT = 13308-"""Non-default port to avoid conflicts with a running dev LLM gateway."""+AI_GATEWAY_PORT = 18080+"""Non-default port so a dev-stack Go ai-gateway on 8080 never collides."""++AI_GATEWAY_REDIS_URL = "redis://localhost:6381/5"+"""The ai-gateway compose valkey, on its own DB index.++Django publishes the credential and team policy blobs here, and the eval gateway+reads them. Policy blobs are keyed by team id, and eval team ids overlap with dev+team ids, so DB 0, which a dev-stack gateway reads, must stay untouched."""PERSONHOG_REPLICA_PORT = 15051"""Non-default port so a dev-stack personhog-replica on 50051 never collides."""diff --git a/products/posthog_ai/eval_harness/harness/providers.py b/products/posthog_ai/eval_harness/harness/providers.pyindex 8dae5236b06c..eeae2a6a4006 100644--- a/products/posthog_ai/eval_harness/harness/providers.py+++ b/products/posthog_ai/eval_harness/harness/providers.py@@ -9,7 +9,7 @@ from contextlib import ExitStackfrom pathlib import Pathfrom typing import Any, ClassVar, Literal-from .ports import DJANGO_LIVE_PORT, LLM_GATEWAY_PORT, MCP_PORT+from .ports import AI_GATEWAY_PORT, DJANGO_LIVE_PORT, MCP_PORTfrom .tunnels import TailscaleFunnel, funnel_preflight_errorlogger = logging.getLogger(__name__)@@ -219,11 +219,14 @@ class DockerProviderStrategy(SandboxProviderStrategy):ensure_template_image(SandboxTemplate.NOTEBOOK_BASE)def settings_overrides(self) -> dict[str, Any]:- # Docker containers reach the host via host.docker.internal.+ # Docker containers reach the host via host.docker.internal. The gateway URL is the+ # exception: the worker on the host mints run tokens through it, and DockerSandbox+ # rewrites localhost to host.docker.internal in the container's env.return {"SANDBOX_PROVIDER": SANDBOX_PROVIDER_SETTING[self.name],"SANDBOX_API_URL": f"http://host.docker.internal:{DJANGO_LIVE_PORT}",- "SANDBOX_LLM_GATEWAY_URL": f"http://host.docker.internal:{LLM_GATEWAY_PORT}",+ "SANDBOX_AI_GATEWAY_URL": f"http://localhost:{AI_GATEWAY_PORT}",+ "SANDBOX_LLM_GATEWAY_URL": f"http://host.docker.internal:{AI_GATEWAY_PORT}","SANDBOX_MCP_URL": f"http://host.docker.internal:{MCP_PORT}/mcp",}@@ -293,7 +296,7 @@ class ModalProviderStrategy(SandboxProviderStrategy):tunnels = TailscaleFunnel({"django": DJANGO_LIVE_PORT,- "gateway": LLM_GATEWAY_PORT,+ "gateway": AI_GATEWAY_PORT,"mcp": MCP_PORT,})@@ -307,6 +310,7 @@ class ModalProviderStrategy(SandboxProviderStrategy):return {"SANDBOX_PROVIDER": SANDBOX_PROVIDER_SETTING[self.name],"SANDBOX_API_URL": self._tunnels.url_for("django"),+ "SANDBOX_AI_GATEWAY_URL": self._tunnels.url_for("gateway"),"SANDBOX_LLM_GATEWAY_URL": self._tunnels.url_for("gateway"),"SANDBOX_MCP_URL": f"{self._tunnels.url_for('mcp')}/mcp",}diff --git a/products/posthog_ai/eval_harness/harness/requirements.py b/products/posthog_ai/eval_harness/harness/requirements.pyindex b9143f4ffa53..543c8da6c348 100644--- a/products/posthog_ai/eval_harness/harness/requirements.py+++ b/products/posthog_ai/eval_harness/harness/requirements.py@@ -22,8 +22,8 @@ class Infra(enum.Enum):LIVE_SERVER = "live_server""""The Django live server sandboxes and the MCP server call back into."""- LLM_GATEWAY = "llm_gateway"- """The gateway proxying the sandboxed agent's model calls."""+ AI_GATEWAY = "ai_gateway"+ """The Go ai-gateway proxying the sandboxed agent's model calls."""MCP_SERVER = "mcp_server""""The MCP server the sandboxed agent's tools talk to."""@@ -47,13 +47,13 @@ class SuiteKind(enum.Enum):_IMPLIES: dict[Infra, frozenset[Infra]] = {Infra.LIVE_SERVER: frozenset({Infra.DATABASE}),- # Both services take the live server URL as their upstream.- Infra.LLM_GATEWAY: frozenset({Infra.LIVE_SERVER}),+ # The gateway's mint credential belongs to the master demo team.+ Infra.AI_GATEWAY: frozenset({Infra.DEMO_DATA}),Infra.MCP_SERVER: frozenset({Infra.LIVE_SERVER}),# Demo seeding's person/group reads go through the personhog router; a dead# router would poison its 30s negative group-types cache for the whole run.Infra.DEMO_DATA: frozenset({Infra.DATABASE, Infra.PERSONHOG}),- Infra.SANDBOX: frozenset({Infra.LLM_GATEWAY, Infra.MCP_SERVER, Infra.DEMO_DATA}),+ Infra.SANDBOX: frozenset({Infra.AI_GATEWAY, Infra.MCP_SERVER, Infra.DEMO_DATA}),}INFRA_BY_KIND: dict[SuiteKind, frozenset[Infra]] = {diff --git a/products/posthog_ai/eval_harness/harness/services.py b/products/posthog_ai/eval_harness/harness/services.pyindex 93cde2a0f4be..50cea1bf1264 100644--- a/products/posthog_ai/eval_harness/harness/services.py+++ b/products/posthog_ai/eval_harness/harness/services.py@@ -1,20 +1,33 @@from __future__ import annotations+import ioimport osimport sysimport json+import timeimport shutilimport signal+import asyncioimport hashlibimport logging+import secretsimport zipfileimport contextlibimport subprocess-from collections.abc import Callable+from collections.abc import AsyncIterator, Callablefrom pathlib import Pathfrom django.conf import settings-from django.db import connections+from django.core.management import call_command+from django.db import InterfaceError, OperationalError, close_old_connections++import requests+from redis.exceptions import (+ ConnectionError as RedisConnectionError,+ TimeoutError as RedisTimeoutError,+)++from posthog.storage.gateway_credential_cache import GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTLfrom products.posthog_ai.eval_harness.long_lived_subprocess import LongLivedSubprocessManager, SubprocessStartupErrorfrom products.tasks.backend.facade.agents import (@@ -23,7 +36,14 @@ from products.tasks.backend.facade.agents import (LocalSkillsCache,)-from .ports import LLM_GATEWAY_PORT, MCP_PORT, PERSONHOG_REPLICA_PORT, PERSONHOG_ROUTER_PORT, SKILL_ARCHIVE_PORT+from .ports import (+ AI_GATEWAY_PORT,+ AI_GATEWAY_REDIS_URL,+ MCP_PORT,+ PERSONHOG_REPLICA_PORT,+ PERSONHOG_ROUTER_PORT,+ SKILL_ARCHIVE_PORT,+)from .providers import PreflightErrorlogger = logging.getLogger(__name__)@@ -31,69 +51,238 @@ logger = logging.getLogger(__name__)LONG_LIVED_SUBPROCESSES = LongLivedSubprocessManager()_ZIP_FIXED_TIME = (2020, 1, 1, 0, 0, 0)+EVAL_AI_PRODUCT = "posthog_ai"+"""The gateway product eval tasks resolve to through their ``posthog_ai`` origin."""-def start_llm_gateway(live_server_url: str) -> Callable[[], None]:- """Start the LLM gateway as a subprocess.+AI_GATEWAY_EVAL_DATABASE = "ai_gateway_evals"+AI_GATEWAY_EVAL_CREDITS_USD = "10000"+_AI_GATEWAY_INTERNAL_TOKEN = secrets.token_urlsafe(32)+_AI_GATEWAY_CREDIT_GRANT_ID = secrets.token_hex(16)+AI_GATEWAY_CREDENTIAL_REFRESH_SECONDS = max(1, min(3600, GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL // 3))- Mirrors ``bin/start-llm-gateway``: runs uvicorn on a non-default port.- The sandbox's agent-server uses this to proxy LLM calls to Anthropic, through the- ``posthog_ai`` product that eval tasks select with their origin.- """- gateway_dir = Path(settings.BASE_DIR) / "services" / "llm-gateway"- venv_dir = gateway_dir / ".venv"- uvicorn_bin = venv_dir / "bin" / "uvicorn"- if not uvicorn_bin.exists():- raise SubprocessStartupError(- f"LLM gateway venv not found at {venv_dir}. "- "Run `bin/start-llm-gateway` once or `cd services/llm-gateway && uv venv .venv && uv pip install -e .`"+def ai_gateway_repo() -> Path:+ """The local ``PostHog/ai-gateway`` clone, at the path ``bin/start-ai-gateway`` uses."""+ override = os.environ.get("AI_GATEWAY_REPO")+ return (Path(override).expanduser() if override else Path.home() / "Development" / "ai-gateway").resolve()+++def _ai_gateway_binary() -> Path:+ return ai_gateway_repo() / ".run" / "eval-gateway"+++def ensure_ai_gateway_binary() -> None:+ """Build the Go ai-gateway from the local clone, during preflight.++ The sandboxed agent's model calls go through the Go gateway, as they do in+ production for PostHog AI runs. The build is incremental, so only the first+ one after a pull is slow.+ """+ repo = ai_gateway_repo()+ if not (repo / "cmd" / "gateway").is_dir():+ raise PreflightError(+ f"No PostHog/ai-gateway clone at {repo}. The sandboxed agent's model calls go through it. "+ f"Run `git clone [email protected]:PostHog/ai-gateway.git {repo}`, "+ "or set AI_GATEWAY_REPO to the path of your clone.")- # The LLM gateway uses pydantic BaseSettings with env_prefix="LLM_GATEWAY_".- # DATABASE_URL and LLM_GATEWAY_ANTHROPIC_API_KEY come from the parent env- # (e.g. .env.local / op run). We need to point it at the test database.- conn = connections["default"]- # The test DB setup rewrites settings_dict["NAME"] to the test DB name in- # place, so don't re-prefix it.- db_name = conn.settings_dict["NAME"]- db_user = conn.settings_dict.get("USER", "posthog")- db_password = conn.settings_dict.get("PASSWORD", "posthog")- db_host = conn.settings_dict.get("HOST", "localhost")- db_port = conn.settings_dict.get("PORT", "5432")- test_db_url = f"postgres://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}"-- # No team rate-limit multiplier: the gateway keys multipliers by team id, but- # each eval case runs as its own freshly minted team, so a mapping like {"1": 10}- # never matches. Rate limits are per team, so every case gets the full base- # budget (hundreds of requests/minute), which is already ample for eval traffic.- env = {- **os.environ,- "UV_PROJECT_ENVIRONMENT": str(venv_dir),- "LLM_GATEWAY_DATABASE_URL": test_db_url,- "LLM_GATEWAY_DEBUG": "true",- "LLM_GATEWAY_POSTHOG_HOST": live_server_url,- }+ if shutil.which("go") is None:+ raise PreflightError("`go` not found on PATH. The harness builds the ai-gateway from your clone.")++ logger.info("Building the ai-gateway from %s", repo)+ result = subprocess.run(+ ["go", "build", "-o", str(_ai_gateway_binary()), "./cmd/gateway"],+ cwd=repo,+ capture_output=True,+ text=True,+ timeout=10 * 60,+ )+ if result.returncode != 0:+ tail = "\n".join(((result.stdout or "") + (result.stderr or "")).splitlines()[-30:])+ raise PreflightError(f"Failed to build the ai-gateway (go exit {result.returncode}):\n{tail}")+++def _ai_gateway_compose(*args: str) -> subprocess.CompletedProcess[str]:+ # PostHog's dev env exports COMPOSE_PROJECT_NAME=posthog, which outranks the+ # gateway compose file's own project name.+ return subprocess.run(+ ["docker", "compose", *args],+ cwd=ai_gateway_repo(),+ env={**os.environ, "COMPOSE_PROJECT_NAME": "ai-gateway"},+ check=True,+ capture_output=True,+ text=True,+ timeout=5 * 60,+ )+++def start_ai_gateway() -> Callable[[], None]:+ """Start the Go ai-gateway on a non-default port, in resolver mode.++ It shares the Postgres and Valkey containers of the gateway repo's compose file+ with a dev-stack gateway, but uses its own database and Valkey DB index, so eval+ ledger rows and team policy blobs never mix with dev ones. The containers stay up+ after the run, as they do for ``bin/start-ai-gateway``.+ """+ _ai_gateway_compose("up", "-d", "--wait", "postgres", "valkey")+ exists = _ai_gateway_compose(+ "exec",+ "-T",+ "postgres",+ "psql",+ "-U",+ "postgres",+ "-tAc",+ f"SELECT 1 FROM pg_database WHERE datname = '{AI_GATEWAY_EVAL_DATABASE}'",+ ).stdout.strip()+ if exists != "1":+ _ai_gateway_compose("exec", "-T", "postgres", "createdb", "-U", "postgres", AI_GATEWAY_EVAL_DATABASE)++ # A dev .env can carry AI_GATEWAY_* settings meant for Django or for a dev gateway.+ env = {key: value for key, value in os.environ.items() if not key.startswith("AI_GATEWAY_")}+ env.update(+ {+ "AI_GATEWAY_ENV": "dev",+ "AI_GATEWAY_AUTH_MODE": "resolver",+ "AI_GATEWAY_LISTEN_ADDR": f":{AI_GATEWAY_PORT}",+ "AI_GATEWAY_POSTGRES_DSN": f"postgres://postgres:postgres@localhost:5435/{AI_GATEWAY_EVAL_DATABASE}",+ "AI_GATEWAY_REDIS_URL": AI_GATEWAY_REDIS_URL,+ "AI_GATEWAY_INTERNAL_TOKEN": _AI_GATEWAY_INTERNAL_TOKEN,+ "AI_GATEWAY_ANTHROPIC_API_KEY": os.environ["LLM_GATEWAY_ANTHROPIC_API_KEY"],+ # The gateway refuses to boot without an OpenAI key, which only codex runs use.+ "AI_GATEWAY_OPENAI_API_KEY": os.environ.get("LLM_GATEWAY_OPENAI_API_KEY") or "unused",+ "AI_GATEWAY_LOG_FORMAT": "text",+ "AI_GATEWAY_LOG_LEVEL": "info",+ }+ )+ binary = _ai_gateway_binary()+ migrate = subprocess.run([str(binary), "migrate"], env=env, capture_output=True, text=True, timeout=120)+ if migrate.returncode != 0:+ raise SubprocessStartupError(f"ai-gateway ledger migration failed:\n{migrate.stdout}{migrate.stderr}")- logger.info("Starting LLM gateway on port %d", LLM_GATEWAY_PORT)+ logger.info("Starting ai-gateway on port %d", AI_GATEWAY_PORT)_, stop = LONG_LIVED_SUBPROCESSES.start(- name="LLM gateway",- port=LLM_GATEWAY_PORT,- cmd=[- str(uvicorn_bin),- "llm_gateway.main:app",- "--host",- "0.0.0.0",- "--port",- str(LLM_GATEWAY_PORT),- ],- cwd=gateway_dir,+ name="ai-gateway",+ port=AI_GATEWAY_PORT,+ cmd=[str(binary)],+ cwd=ai_gateway_repo(),env=env,- log_prefix="llm-gateway",+ log_prefix="ai-gateway",)-- logger.info("LLM gateway ready on port %d", LLM_GATEWAY_PORT)+ try:+ _wait_for_ai_gateway_ready()+ except Exception:+ stop()+ raise+ logger.info("ai-gateway ready on port %d", AI_GATEWAY_PORT)return stop+def _wait_for_ai_gateway_ready(timeout_seconds: float = 120) -> None:+ # The gateway listens before its model catalog is built. Until a provider listing+ # succeeds, /readyz answers 503 and /v1/models is empty. A bad provider key never gets past this.+ deadline = time.monotonic() + timeout_seconds+ body = ""+ while time.monotonic() < deadline:+ try:+ response = requests.get(f"http://localhost:{AI_GATEWAY_PORT}/readyz", timeout=5)+ except requests.RequestException as e:+ body = str(e)+ else:+ if response.status_code == 200:+ return+ body = response.text[:500]+ time.sleep(1)+ raise SubprocessStartupError(+ f"ai-gateway did not become ready in {timeout_seconds:.0f}s. Check the provider API keys. Last /readyz: {body}"+ )+++def refresh_ai_gateway_mint_key(team_id: int, mint_key: str) -> None:+ try:+ call_command("setup_local_gateway_credential", "--phs", mint_key, "--team", str(team_id), stdout=io.StringIO())+ call_command("llm_gateway_team", "refresh", str(team_id), stdout=io.StringIO())+ finally:+ close_old_connections()+++async def maintain_ai_gateway_mint_key(team_id: int, mint_key: str) -> AsyncIterator[None]:+ async def refresh() -> None:+ expires_at = time.monotonic() + GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL+ while True:+ await asyncio.sleep(AI_GATEWAY_CREDENTIAL_REFRESH_SECONDS)+ retry_delay = 1.0+ while True:+ pending = asyncio.create_task(asyncio.to_thread(refresh_ai_gateway_mint_key, team_id, mint_key))+ try:+ await asyncio.shield(pending)+ except asyncio.CancelledError:+ await pending+ raise+ except (InterfaceError, OperationalError, RedisConnectionError, RedisTimeoutError):+ remaining = expires_at - time.monotonic()+ if remaining <= 0:+ raise+ logger.warning("Gateway credential refresh failed; retrying", exc_info=True)+ await asyncio.sleep(min(retry_delay, remaining))+ retry_delay = min(retry_delay * 2, 60)+ else:+ expires_at = time.monotonic() + GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL+ break++ async with asyncio.TaskGroup() as group:+ task = group.create_task(refresh())+ try:+ yield+ finally:+ task.cancel()+++def provision_ai_gateway_mint_key(team_id: int, *, agent_model: str) -> str:+ """Give the master demo team a ``phs_`` key the worker mints per-run tokens with.++ Each run's ``phe_`` token bills the minting team, so only the master team needs+ admission and spend headroom, not the per-case teams. Scoped tokens cannot+ use an overspend allowance, so the isolated eval ledger needs test credits.+ A model listing and one mint check the catalog and the credential, so a broken+ setup fails here and not as a model error in every case.+ """+ mint_key = f"phs_{secrets.token_hex(24)}"+ refresh_ai_gateway_mint_key(team_id, mint_key)++ base_url = f"http://localhost:{AI_GATEWAY_PORT}"+ credits = requests.post(+ f"{base_url}/internal/teams/{team_id}/credits",+ json={"amount_usd": AI_GATEWAY_EVAL_CREDITS_USD, "reason": "Local eval test credits"},+ headers={+ "Authorization": f"Bearer {_AI_GATEWAY_INTERNAL_TOKEN}",+ "Idempotency-Key": f"eval-{team_id}-{_AI_GATEWAY_CREDIT_GRANT_ID}",+ "X-Internal-Actor": "eval-harness",+ },+ timeout=10,+ )+ if not 200 <= credits.status_code < 300:+ raise SubprocessStartupError(f"ai-gateway refused eval test credits: HTTP {credits.status_code}")+ headers = {"Authorization": f"Bearer {mint_key}"}+ models = requests.get(f"{base_url}/v1/models", headers=headers, timeout=10)+ models.raise_for_status()+ model_ids = {entry.get("id") for entry in models.json().get("data", [])}+ if agent_model not in model_ids:+ raise SubprocessStartupError(f"The ai-gateway model catalog does not serve {agent_model!r}")+ mint = requests.post(+ f"{base_url}/v1/tokens",+ json={"cap_usd": "0.01", "ttl_seconds": 60, "product": EVAL_AI_PRODUCT, "obo": str(team_id)},+ headers=headers,+ timeout=10,+ )+ if not 200 <= mint.status_code < 300:+ raise SubprocessStartupError(+ f"ai-gateway refused to mint a run token: HTTP {mint.status_code} {mint.text[:300]}"+ )+ logger.info("ai-gateway mint key provisioned for team_id=%d", team_id)+ return mint_key++def start_mcp_server(live_server_url: str,skill_archive_url: str | None,diff --git a/products/posthog_ai/eval_harness/harness/tunnels.py b/products/posthog_ai/eval_harness/harness/tunnels.pyindex 54d31190808f..c624989e209b 100644--- a/products/posthog_ai/eval_harness/harness/tunnels.py+++ b/products/posthog_ai/eval_harness/harness/tunnels.py@@ -19,7 +19,7 @@ SETUP_GUIDE = "docs/internal/sandboxes-setup-guide.md"FUNNEL_PUBLIC_PORTS: tuple[int, ...] = (443, 8443, 10000)"""The only public ports Tailscale Funnel will serve — the daemon rejects any-other. Three is exactly enough for the harness's Django, LLM gateway, and MCP+other. Three is exactly enough for the harness's Django, AI gateway, and MCPcallbacks; each requested service is assigned one in the order the caller liststhem."""@@ -77,7 +77,7 @@ def funnel_preflight_error() -> str | None:if shutil.which("tailscale") is None:return ("`tailscale` not found on PATH. Modal sandboxes run outside this host, so the "- "Django API, LLM gateway, and MCP server must be publicly reachable via Tailscale Funnel. "+ "Django API, AI gateway, and MCP server must be publicly reachable via Tailscale Funnel. "f"See {SETUP_GUIDE}.")status = tailscale_status()@@ -167,7 +167,7 @@ def _relay_answers(ip: str, host: str, public_port: int) -> bool:class TailscaleFunnel:"""Modal-only Tailscale Funnel lifecycle: publicly exposes the host services a- remote Modal sandbox must reach (Django API, LLM gateway, MCP server).+ remote Modal sandbox must reach (Django API, AI gateway, MCP server).Each service is served over HTTPS on one of Tailscale's three permitted publicports and is reachable by anyone on the internet who learns the URL, so the@@ -326,7 +326,7 @@ class TailscaleFunnel:def _failure_message(self, detail: str) -> str:return (f"{detail}\n"- "Tailscale Funnel could not serve the Django, LLM gateway, and MCP services. "+ "Tailscale Funnel could not serve the Django, AI gateway, and MCP services. ""Make sure tailscaled is running and this node is logged in (`tailscale up`), that Funnel ""is enabled for the tailnet and this node in the admin console's ACLs, and that HTTPS "f"certificates are enabled for the tailnet. See {SETUP_GUIDE}."diff --git a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py b/products/posthog_ai/eval_harness/test/test_one_shot_runner.pyindex fb60ac4eade3..7553667c1193 100644--- a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py+++ b/products/posthog_ai/eval_harness/test/test_one_shot_runner.py@@ -12,6 +12,7 @@ import pytestfrom unittest.mock import patchfrom django.conf import settings+from django.test import override_settingsfrom posthoganalytics import Posthog@@ -278,7 +279,7 @@ def test_run_routes_through_the_engine(),):harness = SandboxedEvalHarness(parse_args(["--agent-model", "claude-test"]))- with harness._stack:+ with override_settings(AI_GATEWAY_REDIS_URL="redis://localhost:6381/5"), harness._stack:harness._bootstrap(frozenset(), ProgressReporter(total_suites=0))ctx = harness._build_context(frozenset(), reporter) # type: ignore[arg-type]runs = [_build_run(tmp_path, monkeypatch, ctx, task) for _ in range(2)]@@ -290,6 +291,7 @@ def test_run_routes_through_the_engine(run.case_trace_meta["c1"] = {"prompt": "the prompt", "duration": 1.0, "first_timestamp": ""}asyncio.run(run_suite(run))assert settings.TEST+ assert settings.AI_GATEWAY_REDIS_URL is Noneassert ctx.posthog_client is not Noneassert ctx.posthog_client.disabledctx.posthog_client.capture(event="ordinary_test_event", distinct_id="test")diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness.pyindex 9d769aca6342..e556c786fc3a 100644--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py@@ -7,7 +7,7 @@ import subprocessfrom collections.abc import Awaitable, Callablefrom pathlib import Pathfrom types import SimpleNamespace-from typing import Literal, cast+from typing import Any, Literal, castfrom uuid import UUIDimport pytest@@ -25,7 +25,11 @@ from products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.discovery import EvalSuitefrom products.posthog_ai.eval_harness.harness.lifecycle import SandboxedEvalHarnessfrom products.posthog_ai.eval_harness.harness.live_server import EvalLiveServer-from products.posthog_ai.eval_harness.harness.providers import ModalProviderStrategy, SandboxProviderStrategy+from products.posthog_ai.eval_harness.harness.providers import (+ DockerProviderStrategy,+ ModalProviderStrategy,+ SandboxProviderStrategy,+)from products.posthog_ai.eval_harness.harness.reporting import ProgressReporterfrom products.tasks.backend.facade.agents import MultiTurnSession, TurnPollResult, TurnPollTimeoutfrom products.tasks.backend.facade.contracts import AgentTaskRunDTO@@ -443,6 +447,31 @@ def test_modal_cleanup_case_terminates_only_the_task_sandboxes(monkeypatch: pytesandbox.terminate.assert_called_once_with()+def _docker_provider() -> SandboxProviderStrategy:+ return DockerProviderStrategy(keep_containers=False)+++def _modal_provider() -> SandboxProviderStrategy:+ provider = ModalProviderStrategy()+ provider._tunnels = cast(Any, SimpleNamespace(url_for=lambda name: f"https://{name}.funnel.example.com"))+ return provider+++ [+ ("docker", _docker_provider, "http://localhost:18080", "http://host.docker.internal:18080"),+ ("modal", _modal_provider, "https://gateway.funnel.example.com", "https://gateway.funnel.example.com"),+ ]+)+def test_provider_routes_model_calls_to_a_gateway_the_host_worker_can_mint_through(+ _name: str, build: Callable[[], SandboxProviderStrategy], gateway_url: str, fallback_url: str+) -> None:+ overrides = build().settings_overrides()++ assert overrides["SANDBOX_AI_GATEWAY_URL"] == gateway_url+ assert overrides["SANDBOX_LLM_GATEWAY_URL"] == fallback_url++class TestAgentRunFailureDetection:# An agent-server failure reaches the log as a session error, and the workflow still finishes# cleanly afterwards. Miss it and a run where no agent ever spoke reports exit_code_zero=1 withdiff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_services.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_services.pyindex 7a1fa6db923c..4bf762a8ce0a 100644--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_services.py+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_services.py@@ -2,16 +2,34 @@ from __future__ import annotationsimport osimport json+import asyncioimport zipfileimport itertools+import subprocessfrom functools import partialfrom pathlib import Path+from types import SimpleNamespaceimport pytestfrom unittest.mock import MagicMock, patch+from django.db import InterfaceError, OperationalError++import requests_mock+from redis.exceptions import (+ ConnectionError as RedisConnectionError,+ TimeoutError as RedisTimeoutError,+)+from products.posthog_ai.eval_harness import long_lived_subprocess-from products.posthog_ai.eval_harness.harness.services import package_local_skills_archive, start_mcp_server+from products.posthog_ai.eval_harness.harness import services+from products.posthog_ai.eval_harness.harness.services import (+ ensure_ai_gateway_binary,+ maintain_ai_gateway_mint_key,+ package_local_skills_archive,+ provision_ai_gateway_mint_key,+ start_mcp_server,+)from products.posthog_ai.eval_harness.long_lived_subprocess import (LongLivedSubprocessManager,SubprocessReadinessTimeout,@@ -153,3 +171,110 @@ def test_service_startup_does_not_retry_permanent_errors_or_interrupts(with pytest.raises(type(error)):manager.start(name="test service", port=12345, cmd=["fake-service"], cwd=tmp_path, env={}, log_prefix="test")assert start.call_count == 1++[email protected]("credit_status", [200, 503])+def test_gateway_provisioning_requires_test_credits(requests_mock: requests_mock.Mocker, credit_status: int) -> None:+ base_url = "http://localhost:18080"+ credits = requests_mock.post(f"{base_url}/internal/teams/42/credits", status_code=credit_status, json={})+ requests_mock.get(f"{base_url}/v1/models", json={"data": [{"id": "test-model"}]})+ requests_mock.post(f"{base_url}/v1/tokens", json={"token": "phe_test"})++ with patch("products.posthog_ai.eval_harness.harness.services.call_command"):+ if credit_status == 200:+ assert provision_ai_gateway_mint_key(42, agent_model="test-model").startswith("phs_")+ else:+ with pytest.raises(SubprocessStartupError, match="refused eval test credits: HTTP 503"):+ provision_ai_gateway_mint_key(42, agent_model="test-model")++ assert credits.last_request is not None+ assert credits.last_request.json() == {"amount_usd": "10000", "reason": "Local eval test credits"}+ assert credits.last_request.headers["Idempotency-Key"] != credits.last_request.headers["Authorization"]++[email protected]("relative_path", ["gateway", "./gateway"])+def test_gateway_build_from_relative_clone_creates_the_binary_used_at_startup(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, relative_path: str+) -> None:+ repo = tmp_path / "gateway"+ (repo / "cmd" / "gateway").mkdir(parents=True)+ monkeypatch.chdir(tmp_path)+ monkeypatch.setenv("AI_GATEWAY_REPO", relative_path)+ monkeypatch.setattr(services.shutil, "which", lambda _name: "/go")++ def build(cmd: list[str], *, cwd: Path, **_kwargs: object) -> subprocess.CompletedProcess[str]:+ output = cwd / cmd[cmd.index("-o") + 1]+ output.parent.mkdir(parents=True, exist_ok=True)+ output.write_text("binary")+ return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")++ monkeypatch.setattr(services.subprocess, "run", build)+ ensure_ai_gateway_binary()++ assert (repo / ".run" / "eval-gateway").read_text() == "binary"+++ "error_type", [None, InterfaceError, OperationalError, RedisConnectionError, RedisTimeoutError]+)+async def test_gateway_mint_credential_is_renewed_without_rotating_the_key(+ monkeypatch: pytest.MonkeyPatch, error_type: type[Exception] | None+) -> None:+ loop = asyncio.get_running_loop()+ refreshed = asyncio.Event()+ parked = asyncio.Event()+ intervals: list[float] = []+ keys: list[str] = []+ attempts = 0++ async def tick(seconds: float) -> None:+ intervals.append(seconds)+ if len(intervals) > (2 if error_type else 1):+ await parked.wait()++ def command(name: str, *args: str, **_kwargs: object) -> None:+ nonlocal attempts+ if name == "setup_local_gateway_credential":+ attempts += 1+ if attempts == 1 and error_type:+ raise error_type("temporary connection failure")+ keys.append(args[args.index("--phs") + 1])+ loop.call_soon_threadsafe(refreshed.set)++ monkeypatch.setattr(services.asyncio, "sleep", tick)+ monkeypatch.setattr(services, "call_command", command)+ async with maintain_ai_gateway_mint_key(42, "phs_test"):+ await asyncio.wait_for(refreshed.wait(), timeout=5)++ assert keys == ["phs_test"]+ assert intervals[0] < services.GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL+ if error_type:+ assert intervals[1] == 1++[email protected]("error_type", [OperationalError, RedisConnectionError, ValueError])+async def test_gateway_mint_credential_refresh_stops_at_expiry_or_permanent_failure(+ monkeypatch: pytest.MonkeyPatch, error_type: type[Exception]+) -> None:+ now = 0.0+ intervals: list[float] = []++ async def tick(seconds: float) -> None:+ nonlocal now+ intervals.append(seconds)+ now += seconds++ monkeypatch.setattr(services, "time", SimpleNamespace(monotonic=lambda: now))+ monkeypatch.setattr(services.asyncio, "sleep", tick)+ monkeypatch.setattr(services, "GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL", 10)+ monkeypatch.setattr(services, "AI_GATEWAY_CREDENTIAL_REFRESH_SECONDS", 3)+ monkeypatch.setattr(services, "refresh_ai_gateway_mint_key", MagicMock(side_effect=error_type("refresh failed")))++ with pytest.raises(ExceptionGroup) as failure:+ async with maintain_ai_gateway_mint_key(42, "phs_test"):+ await asyncio.wait_for(asyncio.Event().wait(), timeout=5)++ assert isinstance(failure.value.exceptions[0], error_type)+ assert intervals == ([3] if error_type is ValueError else [3, 1, 2, 4])