Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(evals): route sandboxed eval agents through the go ai-gateway (#118445)

Daniel Parkerb92cad71d864
diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.md
index 755d2b98c1ca..4527f4517ec1 100644
--- a/products/posthog_ai/eval_harness/README.md
+++ b/products/posthog_ai/eval_harness/README.md
@@ -7,11 +7,11 @@ The harness builds product skills from the current checkout and uses native bund
`--skill-delivery exec` instead packages those skills for MCP distribution, enables the exec skill prompt, and removes native skills from every sandbox so the two paths cannot satisfy the same case.
This is the eval harness for PostHog AI 2.0. It does **not** run under pytest.
-It runs on a standalone harness that boots the shared infrastructure once (test database, Django live server, LLM gateway, MCP server, Temporal) and then runs every selected suite concurrently.
+It runs on a standalone harness that boots the shared infrastructure once (test database, Django live server, AI gateway, MCP server, Temporal) and then runs every selected suite concurrently.
See [`harness/README.md`](harness/README.md) for how that works internally.
The harness also hosts non-sandboxed suite kinds — see [Suite kinds](#suite-kinds).
-A run boots only the infrastructure its selected suites need, so a one-shot-only run starts no sandbox provider, Temporal, live server, LLM gateway, or MCP server.
+A run boots only the infrastructure its selected suites need, so a one-shot-only run starts no sandbox provider, Temporal, live server, AI gateway, or MCP server.
Two other docs sit next to this one:
@@ -37,7 +37,7 @@ python -m products.posthog_ai.eval_harness.harness [SELECTOR ...] [flags]
No manual env sourcing is needed on either path.
The harness loads the repo-root `.env` itself (shell values win), and `hogli evals` additionally layers in `.env.local` / `.env.development` / `.env.services` through hogli's standard env loading — including 1Password resolution when `.env.local` holds `op://` references.
Before any infrastructure boots, a preflight validates that the required variables are set and fails with a one-line fix per missing variable.
-Which variables are required depends on the eval engine and the selected suites' kinds: the braintrust engine (the default, and only one today) requires `BRAINTRUST_API_KEY` on every run — it is the engine's own `required_env()`, not a core harness variable; sandboxed suites add `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`; one-shot suites add `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).
+Which variables are required depends on the eval engine and the selected suites' kinds: the braintrust engine (the default, and only one today) requires `BRAINTRUST_API_KEY` on every run — it is the engine's own `required_env()`, not a core harness variable; sandboxed suites add `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`, and need a [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway) clone plus `go` on `PATH` (see [AI gateway](#ai-gateway)); one-shot suites add `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).
Selectors are substrings matched against a suite id of the form `<domain>/<module>::<fn>`, for example `experiments`, `sql`, or `eval_lifecycle_skills`.
Omit them to run every suite.
@@ -117,6 +117,21 @@ Dependency warnings, SDK progress bars, and direct setup prints stay in the tran
`--verbose` also shows Python warnings, SDK output, and detailed logs in the console.
Native subprocess output always goes to the transcript.
+### AI gateway
+
+Sandboxed agents call their model through the Go [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway), as PostHog AI runs do in production.
+The harness builds and starts it from a local clone at `~/Development/ai-gateway`, the same path `bin/start-ai-gateway` uses. Set `AI_GATEWAY_REPO` to use a clone at a different path.
+
+```bash
+git clone [email protected]:PostHog/ai-gateway.git ~/Development/ai-gateway
+```
+
+The gateway runs on `:18080` with its own Postgres database and Valkey DB index in the clone's compose containers, so it does not touch a dev-stack gateway's ledger or blobs.
+It uses `LLM_GATEWAY_ANTHROPIC_API_KEY` and `LLM_GATEWAY_OPENAI_API_KEY` as its provider keys.
+The harness adds test credits to the master demo team in this separate ledger. Scoped tokens require a positive balance and cannot use an overspend allowance.
+Each run gets a capped `phe_` token minted with a key on that team, so spend is per run and never reaches a real wallet. The harness refreshes the mint key during long runs. It retries temporary database or Redis errors until the credential expires. Retry delays increase to at most one minute.
+If token minting fails, the fallback URL stays on the eval gateway, which rejects requests without a run token.
+
### Interactive view
The view is `phrocs evals`, the same Bubble Tea program that renders `hogli start`.
@@ -170,7 +185,7 @@ The experiment metadata records `skill_delivery`; compare the shared `expected_s
### Codex runtime
`--agent-runtime codex` runs the same agent-server with OpenAI's Codex harness instead of Claude, defaulting the model to `gpt-6.1-sol`.
-It requires `LLM_GATEWAY_OPENAI_API_KEY` in the environment (checked by preflight), which the harness's LLM gateway uses to proxy the agent's OpenAI calls.
+It requires `LLM_GATEWAY_OPENAI_API_KEY` in the environment (checked by preflight), which the harness's AI gateway uses to proxy the agent's OpenAI calls.
Experiment names don't change with the runtime or skill delivery. Each Braintrust experiment and every PostHog event records `agent_model`, `trials`, `git_sha` and `git_dirty`. Sandboxed runs add `agent_runtime`, `skill_delivery` and `reasoning_effort`. Use these fields to separate runs. A comparison across runtimes measures the model and runtime together.
### Claude and Codex smoke comparison
@@ -226,7 +241,7 @@ The first modal run pays a one-time remote image build; later runs reuse the cac
Modal prerequisites, all checked by preflight before anything boots:
-- `tailscale` on `PATH`, with tailscaled running and this node logged in (`tailscale up`). Funnel must be enabled for the tailnet and this node in the admin console's ACLs, with HTTPS certificates enabled. The harness serves Django, the LLM gateway, and MCP on Funnel's three public ports (443, 8443, 10000).
+- `tailscale` on `PATH`, with tailscaled running and this node logged in (`tailscale up`). Funnel must be enabled for the tailnet and this node in the admin console's ACLs, with HTTPS certificates enabled. The harness serves Django, the AI gateway, and MCP on Funnel's three public ports (443, 8443, 10000).
- Modal credentials: `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`, or a `~/.modal.toml` from `modal token new`.
- `SANDBOX_JWT_PRIVATE_KEY` in the environment. The dev key ships in `.env.example`; the harness auto-loads it from `.env`.
diff --git a/products/posthog_ai/eval_harness/harness/AGENTS.md b/products/posthog_ai/eval_harness/harness/AGENTS.md
index 61d61e0ee416..d0ef3b0bb86c 100644
--- a/products/posthog_ai/eval_harness/harness/AGENTS.md
+++ b/products/posthog_ai/eval_harness/harness/AGENTS.md
@@ -90,7 +90,7 @@ Braintrust's `EvalAsync` always dispatches through `eval_async`, and the base `S
Bootstrap registers teardown on an `ExitStack` as each resource comes up, so a failure halfway through still unwinds what already started.
`atexit` hooks plus the subprocess manager's signal handlers cover Ctrl-C, where neither stack unwinds.
-After any change, a Ctrl-C mid-run must leave no listeners on 18000 / 13308 / 18787 / 18788 / 15051 / 15052, no `task-sandbox-*` containers, no Temporal dev server, and no Tailscale Funnel mappings left enabled on 443 / 8443 / 10000.
+After any change, a Ctrl-C mid-run must leave no listeners on 18000 / 18080 / 18787 / 18788 / 15051 / 15052, no `task-sandbox-*` containers, no Temporal dev server, and no Tailscale Funnel mappings left enabled on 443 / 8443 / 10000.
A case can also own a sandbox the harness never created: a notebook python or duckdb cell provisions the notebook kernel through the notebook Temporal workflow.
Nothing upstream reclaims it — the docker backend ignores `SandboxConfig.ttl_seconds`, and the sweeps below match agent containers by task id, which a kernel container's name never carries.
diff --git a/products/posthog_ai/eval_harness/harness/README.md b/products/posthog_ai/eval_harness/harness/README.md
index 340822493973..40aa514da76a 100644
--- a/products/posthog_ai/eval_harness/harness/README.md
+++ b/products/posthog_ai/eval_harness/harness/README.md
@@ -7,7 +7,7 @@ This file explains how it fits together.
## Why it exists
-The sandboxed evals need heavy shared infrastructure: a committed test database, an in-process Django server the sandbox can call back into, an LLM gateway, an MCP server, and a Temporal server plus worker.
+The sandboxed evals need heavy shared infrastructure: a committed test database, an in-process Django server the sandbox can call back into, the Go AI gateway, an MCP server, and a Temporal server plus worker.
Under pytest that infrastructure lived in session fixtures, and suites ran strictly one at a time, with Braintrust parallelizing only _within_ a suite.
Wall-clock was dominated by suites queueing behind each other while sandbox capacity sat idle.
@@ -16,32 +16,32 @@ Braintrust remains the eval engine and reporting backend; it just no longer cont
The `EvalAsync` call itself lives behind an `EvalEngine` seam (`../engines/`): `_BaseEvalRun.run()` hands a neutral `ExperimentSpec` to the engine resolved by `engines/registry.resolve_engine()` (shared via `EvalContext.engine`) and gets back a neutral `ExperimentResult` (`engines/types.py`), so a future PostHog-native engine can slot in behind the same interface — implementing the `EvalEngine` protocol and passing the conformance suite — without touching the run base or any suite.
Not every suite is sandboxed anymore: each suite declares a `SUITE_KIND` (`requirements.py`), the kind maps to a set of `Infra` requirements, and the harness boots only the union of what the selected suites need.
-A run of one-shot suites never pays for — or fails preflight on — the sandbox provider, Temporal, the live server, the LLM gateway, or the MCP server.
+A run of one-shot suites never pays for — or fails preflight on — the sandbox provider, Temporal, the live server, the AI gateway, or the MCP server.
## Modules
-| Module | Role |
-| ------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
-| `__main__.py` | Entry point. Parses args, starts the transcript, configures Django, then hands off to `lifecycle`. |
-| `cli.py` | `HarnessOptions` and the argparse builder. Resolves per-provider defaults. |
-| `env_preflight.py` | Loads the repo-root `.env` (stdlib parser, shell wins) and validates per-kind env vars via pydantic models. |
-| `ports.py` | The seven port constants. Deliberately free of Django imports. |
-| `providers.py` | `SandboxProviderStrategy` and its docker/modal implementations: preflight, settings overrides, sandbox TTL, cleanup. |
-| `tunnels.py` | `TailscaleFunnel`. Modal only: enables Tailscale Funnel on the host services, waits for the public relays, and resolves the URLs. |
-| `requirements.py` | `SuiteKind`, `Infra`, and the kind → infrastructure mapping (with implication closure). |
-| `django_env.py` | `setup_django()`, the `NullDbBlocker` shim, and `EvalDatabase` (test database lifecycle). |
-| `live_server.py` | `EvalLiveServer`, a session-lifetime Uvicorn server for PostHog's full ASGI application. |
-| `services.py` | Starts the LLM gateway, MCP server, skill archive server, and personhog subprocesses; builds local skills. |
-| `temporal_env.py` | Local Temporal dev server, stale-workflow cleanup, and the worker thread. |
-| `demo_data.py` | `SandboxedDemoData`: seeds the master Hedgebox team once, then mints an isolated team per case. |
-| `discovery.py` | Walks the tree for `eval_*.py` and collects `eval_*` coroutines into `EvalSuite` objects. |
-| `context.py` | `EvalContext`, the single object every suite receives. |
-| `reporting.py` | `ProgressReporter` and the final summary table. The quiet Braintrust reporter lives with the engine (`../engines/braintrust.py`). |
-| `transcript.py` | Mirrors stdout and stderr to one run log, then publishes its absolute path as the final line. |
-| `dashboard.py` | Starts the `phrocs evals` view process and streams run progress to it over a socket pair. |
-| `progress.py` | Immutable trial progress shared by the reporter and interactive view. |
-| `terminal.py` | Opens terminal streams without sharing the descriptors used by transcript capture. |
-| `lifecycle.py` | `SandboxedEvalHarness`: orchestrates bootstrap, the run, and teardown. |
+| Module | Role |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
+| `__main__.py` | Entry point. Parses args, starts the transcript, configures Django, then hands off to `lifecycle`. |
+| `cli.py` | `HarnessOptions` and the argparse builder. Resolves per-provider defaults. |
+| `env_preflight.py` | Loads the repo-root `.env` (stdlib parser, shell wins) and validates per-kind env vars via pydantic models. |
+| `ports.py` | The seven port constants. Deliberately free of Django imports. |
+| `providers.py` | `SandboxProviderStrategy` and its docker/modal implementations: preflight, settings overrides, sandbox TTL, cleanup. |
+| `tunnels.py` | `TailscaleFunnel`. Modal only: enables Tailscale Funnel on the host services, waits for the public relays, and resolves the URLs. |
+| `requirements.py` | `SuiteKind`, `Infra`, and the kind → infrastructure mapping (with implication closure). |
+| `django_env.py` | `setup_django()`, the `NullDbBlocker` shim, and `EvalDatabase` (test database lifecycle). |
+| `live_server.py` | `EvalLiveServer`, a session-lifetime Uvicorn server for PostHog's full ASGI application. |
+| `services.py` | Starts the AI gateway, MCP server, skill archive server, and personhog subprocesses; provisions the gateway mint key; builds local skills. |
+| `temporal_env.py` | Local Temporal dev server, stale-workflow cleanup, and the worker thread. |
+| `demo_data.py` | `SandboxedDemoData`: seeds the master Hedgebox team once, then mints an isolated team per case. |
+| `discovery.py` | Walks the tree for `eval_*.py` and collects `eval_*` coroutines into `EvalSuite` objects. |
+| `context.py` | `EvalContext`, the single object every suite receives. |
+| `reporting.py` | `ProgressReporter` and the final summary table. The quiet Braintrust reporter lives with the engine (`../engines/braintrust.py`). |
+| `transcript.py` | Mirrors stdout and stderr to one run log, then publishes its absolute path as the final line. |
+| `dashboard.py` | Starts the `phrocs evals` view process and streams run progress to it over a socket pair. |
+| `progress.py` | Immutable trial progress shared by the reporter and interactive view. |
+| `terminal.py` | Opens terminal streams without sharing the descriptors used by transcript capture. |
+| `lifecycle.py` | `SandboxedEvalHarness`: orchestrates bootstrap, the run, and teardown. |
## Boot sequence
@@ -52,14 +52,16 @@ The sequence as written is the sandboxed (full) path; a one-shot-only run perfor
The bootstrap is deliberately synchronous and runs before any event loop exists, because it is ORM-heavy and Django's async-safety guard rejects sync ORM calls from an async context:
1. `__main__` creates the run transcript, then loads the repo-root `.env` (never overriding what the shell — or hogli's own env loading, under `hogli evals` — already exported), then `setup_django()` sets `DEBUG` / `TEST` / `IN_EVAL_TESTING`, forces `SELF_CAPTURE=0`, and runs `django.setup()` and `setup_test_environment()`. Eval telemetry still uses the explicit regional client configured by the harness.
- The env preflight (required variables, one-line fix per missing one) and provider preflight then run, followed by the personhog binary build (`cargo build`, incremental after the first run) — a missing key or toolchain fails here, before any database work.
+ The env preflight (required variables, one-line fix per missing one) and provider preflight then run, followed by the personhog binary build (`cargo build`, incremental after the first run) and the AI gateway build (`go build` in the `PostHog/ai-gateway` clone) — a missing key or toolchain fails here, before any database work.
2. `EvalDatabase.setup()` creates the `default` test database and drives PostHog's own eval database setup (persons database, ClickHouse).
3. `personhog-replica` (`:15051`) and `personhog-router` (`:15052`) start against the test persons database — before anything can query, so a dead router never poisons the negative group-types cache.
4. `EvalLiveServer` serves PostHog's ASGI application on `0.0.0.0:18000`, including the sandbox event-ingest route.
-5. The LLM gateway (`:13308`) starts and local skills are built. In bundled mode, the MCP server (`:18787`) starts with exec skill distribution forced off. In exec mode, the skills are packaged, the local archive server (`:18788`) starts, and MCP receives its content-addressed URL with distribution forced on.
+5. The Go AI gateway (`:18080`) starts in resolver mode, on the gateway compose Postgres (database `ai_gateway_evals`) and Valkey (DB 5), and local skills are built. In bundled mode, the MCP server (`:18787`) starts with exec skill distribution forced off. In exec mode, the skills are packaged, the local archive server (`:18788`) starts, and MCP receives its content-addressed URL with distribution forced on.
6. Docker only: the `posthog-sandbox-base` image freshness check runs (rebuilding when the Dockerfile or its agent pin changed). Modal only: Tailscale Funnel comes up, exposing the callback services publicly. Setup then waits until every public Funnel relay serves every service, because the relays start serving at different times and can take minutes.
7. Bundled mode bind-mounts the built skills on Docker and bakes them into the Modal image. Exec mode suppresses those skills and clears native skill directories before every case; `disable_bundled_skills=True` remains available as a per-case override in other modes.
-8. The master Hedgebox team is seeded.
+8. The master Hedgebox team is seeded. It then gets the AI gateway mint key: a `phs_` key, gateway admission, and test credits in the separate eval ledger. One model listing and one token mint check that path before any case starts. The harness refreshes the same mint credential before its cache TTL expires during the run.
+
+Gateway cache writes stay disabled until the gateway starts. A one-shot run leaves them disabled.
The async phase then starts the Temporal dev server on the main loop, applies the provider's settings overrides, terminates stale workflows, starts the Temporal worker on its own thread and loop, and fans the suites out with `asyncio.gather`.
diff --git a/products/posthog_ai/eval_harness/harness/__init__.py b/products/posthog_ai/eval_harness/harness/__init__.py
index 85e1721612a9..f478439f869d 100644
--- a/products/posthog_ai/eval_harness/harness/__init__.py
+++ b/products/posthog_ai/eval_harness/harness/__init__.py
@@ -1,7 +1,7 @@
"""Standalone runner for the sandboxed agent evals.
Replaces pytest as the driver: boots the shared session infrastructure once
-(test DB, Django live server, LLM gateway, MCP server, Temporal), then runs
+(test DB, Django live server, AI gateway, MCP server, Temporal), then runs
every selected eval suite concurrently on one event loop with a single global
sandbox semaphore bounding total load. Braintrust stays as the eval engine.
diff --git a/products/posthog_ai/eval_harness/harness/__main__.py b/products/posthog_ai/eval_harness/harness/__main__.py
index de613775f5ce..a951b5d80b93 100644
--- a/products/posthog_ai/eval_harness/harness/__main__.py
+++ b/products/posthog_ai/eval_harness/harness/__main__.py
@@ -9,7 +9,7 @@ from typing import TYPE_CHECKING
from .cli import HarnessOptions, parse_args
from .django_env import setup_django
from .env_preflight import load_env_file
-from .ports import PERSONHOG_ROUTER_PORT
+from .ports import AI_GATEWAY_REDIS_URL, PERSONHOG_ROUTER_PORT
from .providers import SANDBOX_PROVIDER_SETTING, PreflightError
from .terminal import open_terminal
from .transcript import RunTranscript, write_console
@@ -47,6 +47,10 @@ def _run(
# would otherwise silently point eval reads at the dev persons DB.
os.environ["PERSONHOG_ADDR"] = f"127.0.0.1:{PERSONHOG_ROUTER_PORT}"
+ # HyperCache selects its cache alias at import time. Bootstrap enables
+ # gateway projections only after the selected suites start the gateway Redis.
+ os.environ["AI_GATEWAY_REDIS_URL"] = AI_GATEWAY_REDIS_URL
+
setup_django(keep_logging=transcript is not None)
if transcript is not None:
_configure_logging(transcript, verbose=options.verbose)
diff --git a/products/posthog_ai/eval_harness/harness/env_preflight.py b/products/posthog_ai/eval_harness/harness/env_preflight.py
index cc7a81148a1e..f8ba39603da1 100644
--- a/products/posthog_ai/eval_harness/harness/env_preflight.py
+++ b/products/posthog_ai/eval_harness/harness/env_preflight.py
@@ -34,7 +34,7 @@ class SandboxEvalEnv(BaseModel):
)
LLM_GATEWAY_ANTHROPIC_API_KEY: str = Field(
min_length=1,
- description="Anthropic API key the LLM gateway proxies the agent's model calls with",
+ description="Anthropic API key the AI gateway proxies the agent's model calls with",
)
@@ -56,7 +56,7 @@ class CodexEvalEnv(BaseModel):
LLM_GATEWAY_OPENAI_API_KEY: str = Field(
min_length=1,
- description="OpenAI API key the LLM gateway proxies the codex agent's model calls with",
+ description="OpenAI API key the AI gateway proxies the codex agent's model calls with",
)
diff --git a/products/posthog_ai/eval_harness/harness/lifecycle.py b/products/posthog_ai/eval_harness/harness/lifecycle.py
index ebac910a0768..f13387707326 100644
--- a/products/posthog_ai/eval_harness/harness/lifecycle.py
+++ b/products/posthog_ai/eval_harness/harness/lifecycle.py
@@ -42,7 +42,7 @@ from .env_preflight import validate_eval_env
from .kernel_sandboxes import reclaim_kernels
from .live_server import EvalLiveServer
from .offline_delivery import OfflineEvalDelivery
-from .ports import DJANGO_LIVE_PORT
+from .ports import AI_GATEWAY_REDIS_URL, DJANGO_LIVE_PORT
from .providers import PreflightError, SandboxProviderStrategy, build_provider
from .reporting import ProgressReporter, SuiteRunResult
from .requirements import Infra, infra_union
@@ -50,11 +50,15 @@ from .requirements import Infra, infra_union
if TYPE_CHECKING:
from .dashboard import EvalDashboard
from .services import (
+ EVAL_AI_PRODUCT,
LONG_LIVED_SUBPROCESSES,
build_local_skills,
+ ensure_ai_gateway_binary,
ensure_personhog_binaries,
+ maintain_ai_gateway_mint_key,
package_local_skills_archive,
- start_llm_gateway,
+ provision_ai_gateway_mint_key,
+ start_ai_gateway,
start_mcp_server,
start_personhog,
start_skill_archive_server,
@@ -121,6 +125,7 @@ class SandboxedEvalHarness:
self._posthog_client: Posthog | None = None
self._demo_data: SandboxedDemoData | None = None
self._offline_delivery: OfflineEvalDelivery | None = None
+ self._ai_gateway_mint_key: str | None = None
# Capture the revision before any eval starts so every result describes the same run.
self._git_sha = _git("rev-parse", "HEAD")
self._git_dirty = _worktree_dirty()
@@ -189,6 +194,9 @@ class SandboxedEvalHarness:
if Infra.PERSONHOG in required:
reporter.setup_started("build personhog")
ensure_personhog_binaries()
+ if Infra.AI_GATEWAY in required:
+ reporter.setup_started("build ai-gateway")
+ ensure_ai_gateway_binary()
reporter.setup_started("start shared services and prepare demo data")
self._bootstrap(required, reporter)
if self._dashboard:
@@ -262,6 +270,7 @@ class SandboxedEvalHarness:
def _bootstrap(self, required: frozenset[Infra], reporter: ProgressReporter) -> None:
"""Stand up everything the selected suites share. Teardown is registered in
reverse on the stack, so what wasn't started is never torn down."""
+ self._stack.enter_context(override_settings(AI_GATEWAY_REDIS_URL=None))
LONG_LIVED_SUBPROCESSES.startup_retries = self.options.infrastructure_retries
# Belt and braces for the SIGINT / SIGTERM paths, where the ExitStack never unwinds.
atexit.register(stop_all_subprocesses)
@@ -289,10 +298,10 @@ class SandboxedEvalHarness:
self._stack.callback(live_server.stop)
self._live_server = live_server
- if Infra.LLM_GATEWAY in required:
- reporter.setup_started("start LLM gateway")
- assert self._live_server is not None
- self._stack.callback(start_llm_gateway(self._live_server.url))
+ if Infra.AI_GATEWAY in required:
+ reporter.setup_started("start AI gateway")
+ self._stack.callback(start_ai_gateway())
+ self._stack.enter_context(override_settings(AI_GATEWAY_REDIS_URL=AI_GATEWAY_REDIS_URL))
if Infra.MCP_SERVER in required:
reporter.setup_started("build skills and start MCP server")
assert self._live_server is not None
@@ -341,10 +350,22 @@ class SandboxedEvalHarness:
),
)
+ if Infra.AI_GATEWAY in required:
+ assert self._demo_data is not None and self._database is not None
+ with self._database.blocker.unblock():
+ self._ai_gateway_mint_key = provision_ai_gateway_mint_key(
+ self._demo_data.master_team_id, agent_model=self.options.agent_model
+ )
+
async def _run_suites(
self, suites: Sequence[EvalSuite], required: frozenset[Infra], reporter: ProgressReporter
) -> list[SuiteRunResult]:
async with AsyncExitStack() as stack:
+ if Infra.AI_GATEWAY in required:
+ assert self._demo_data is not None and self._ai_gateway_mint_key is not None
+ await stack.enter_async_context(
+ maintain_ai_gateway_mint_key(self._demo_data.master_team_id, self._ai_gateway_mint_key)
+ )
overrides: dict[str, object] = {}
if Infra.LIVE_SERVER in required:
overrides.update(
@@ -375,6 +396,10 @@ class SandboxedEvalHarness:
# The harness never starts run_task_workflow_dispatcher, so an outbox dispatch would
# leave every case parked in QUEUED until the case's poll budget runs out.
TASKS_WORKFLOW_DISPATCH_OUTBOX_ENABLED=False,
+ # Only the eval product routes to the Go gateway. The worker mints each
+ # run a capped scoped token with this key, as it does in production.
+ SANDBOX_AI_GATEWAY_PRODUCTS=EVAL_AI_PRODUCT,
+ SANDBOX_AI_GATEWAY_MINT_KEY=self._ai_gateway_mint_key,
**self.provider.settings_overrides(),
)
diff --git a/products/posthog_ai/eval_harness/harness/ports.py b/products/posthog_ai/eval_harness/harness/ports.py
index 59ac1ebc7833..9b1c70b702e1 100644
--- a/products/posthog_ai/eval_harness/harness/ports.py
+++ b/products/posthog_ai/eval_harness/harness/ports.py
@@ -9,8 +9,15 @@ SKILL_ARCHIVE_PORT = 18788
DJANGO_LIVE_PORT = 18000
"""Non-default port for the in-process Django server."""
-LLM_GATEWAY_PORT = 13308
-"""Non-default port to avoid conflicts with a running dev LLM gateway."""
+AI_GATEWAY_PORT = 18080
+"""Non-default port so a dev-stack Go ai-gateway on 8080 never collides."""
+
+AI_GATEWAY_REDIS_URL = "redis://localhost:6381/5"
+"""The ai-gateway compose valkey, on its own DB index.
+
+Django publishes the credential and team policy blobs here, and the eval gateway
+reads them. Policy blobs are keyed by team id, and eval team ids overlap with dev
+team ids, so DB 0, which a dev-stack gateway reads, must stay untouched."""
PERSONHOG_REPLICA_PORT = 15051
"""Non-default port so a dev-stack personhog-replica on 50051 never collides."""
diff --git a/products/posthog_ai/eval_harness/harness/providers.py b/products/posthog_ai/eval_harness/harness/providers.py
index 8dae5236b06c..eeae2a6a4006 100644
--- a/products/posthog_ai/eval_harness/harness/providers.py
+++ b/products/posthog_ai/eval_harness/harness/providers.py
@@ -9,7 +9,7 @@ from contextlib import ExitStack
from pathlib import Path
from typing import Any, ClassVar, Literal
-from .ports import DJANGO_LIVE_PORT, LLM_GATEWAY_PORT, MCP_PORT
+from .ports import AI_GATEWAY_PORT, DJANGO_LIVE_PORT, MCP_PORT
from .tunnels import TailscaleFunnel, funnel_preflight_error
logger = logging.getLogger(__name__)
@@ -219,11 +219,14 @@ class DockerProviderStrategy(SandboxProviderStrategy):
ensure_template_image(SandboxTemplate.NOTEBOOK_BASE)
def settings_overrides(self) -> dict[str, Any]:
- # Docker containers reach the host via host.docker.internal.
+ # Docker containers reach the host via host.docker.internal. The gateway URL is the
+ # exception: the worker on the host mints run tokens through it, and DockerSandbox
+ # rewrites localhost to host.docker.internal in the container's env.
return {
"SANDBOX_PROVIDER": SANDBOX_PROVIDER_SETTING[self.name],
"SANDBOX_API_URL": f"http://host.docker.internal:{DJANGO_LIVE_PORT}",
- "SANDBOX_LLM_GATEWAY_URL": f"http://host.docker.internal:{LLM_GATEWAY_PORT}",
+ "SANDBOX_AI_GATEWAY_URL": f"http://localhost:{AI_GATEWAY_PORT}",
+ "SANDBOX_LLM_GATEWAY_URL": f"http://host.docker.internal:{AI_GATEWAY_PORT}",
"SANDBOX_MCP_URL": f"http://host.docker.internal:{MCP_PORT}/mcp",
}
@@ -293,7 +296,7 @@ class ModalProviderStrategy(SandboxProviderStrategy):
tunnels = TailscaleFunnel(
{
"django": DJANGO_LIVE_PORT,
- "gateway": LLM_GATEWAY_PORT,
+ "gateway": AI_GATEWAY_PORT,
"mcp": MCP_PORT,
}
)
@@ -307,6 +310,7 @@ class ModalProviderStrategy(SandboxProviderStrategy):
return {
"SANDBOX_PROVIDER": SANDBOX_PROVIDER_SETTING[self.name],
"SANDBOX_API_URL": self._tunnels.url_for("django"),
+ "SANDBOX_AI_GATEWAY_URL": self._tunnels.url_for("gateway"),
"SANDBOX_LLM_GATEWAY_URL": self._tunnels.url_for("gateway"),
"SANDBOX_MCP_URL": f"{self._tunnels.url_for('mcp')}/mcp",
}
diff --git a/products/posthog_ai/eval_harness/harness/requirements.py b/products/posthog_ai/eval_harness/harness/requirements.py
index b9143f4ffa53..543c8da6c348 100644
--- a/products/posthog_ai/eval_harness/harness/requirements.py
+++ b/products/posthog_ai/eval_harness/harness/requirements.py
@@ -22,8 +22,8 @@ class Infra(enum.Enum):
LIVE_SERVER = "live_server"
"""The Django live server sandboxes and the MCP server call back into."""
- LLM_GATEWAY = "llm_gateway"
- """The gateway proxying the sandboxed agent's model calls."""
+ AI_GATEWAY = "ai_gateway"
+ """The Go ai-gateway proxying the sandboxed agent's model calls."""
MCP_SERVER = "mcp_server"
"""The MCP server the sandboxed agent's tools talk to."""
@@ -47,13 +47,13 @@ class SuiteKind(enum.Enum):
_IMPLIES: dict[Infra, frozenset[Infra]] = {
Infra.LIVE_SERVER: frozenset({Infra.DATABASE}),
- # Both services take the live server URL as their upstream.
- Infra.LLM_GATEWAY: frozenset({Infra.LIVE_SERVER}),
+ # The gateway's mint credential belongs to the master demo team.
+ Infra.AI_GATEWAY: frozenset({Infra.DEMO_DATA}),
Infra.MCP_SERVER: frozenset({Infra.LIVE_SERVER}),
# Demo seeding's person/group reads go through the personhog router; a dead
# router would poison its 30s negative group-types cache for the whole run.
Infra.DEMO_DATA: frozenset({Infra.DATABASE, Infra.PERSONHOG}),
- Infra.SANDBOX: frozenset({Infra.LLM_GATEWAY, Infra.MCP_SERVER, Infra.DEMO_DATA}),
+ Infra.SANDBOX: frozenset({Infra.AI_GATEWAY, Infra.MCP_SERVER, Infra.DEMO_DATA}),
}
INFRA_BY_KIND: dict[SuiteKind, frozenset[Infra]] = {
diff --git a/products/posthog_ai/eval_harness/harness/services.py b/products/posthog_ai/eval_harness/harness/services.py
index 93cde2a0f4be..50cea1bf1264 100644
--- a/products/posthog_ai/eval_harness/harness/services.py
+++ b/products/posthog_ai/eval_harness/harness/services.py
@@ -1,20 +1,33 @@
from __future__ import annotations
+import io
import os
import sys
import json
+import time
import shutil
import signal
+import asyncio
import hashlib
import logging
+import secrets
import zipfile
import contextlib
import subprocess
-from collections.abc import Callable
+from collections.abc import AsyncIterator, Callable
from pathlib import Path
from django.conf import settings
-from django.db import connections
+from django.core.management import call_command
+from django.db import InterfaceError, OperationalError, close_old_connections
+
+import requests
+from redis.exceptions import (
+ ConnectionError as RedisConnectionError,
+ TimeoutError as RedisTimeoutError,
+)
+
+from posthog.storage.gateway_credential_cache import GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL
from products.posthog_ai.eval_harness.long_lived_subprocess import LongLivedSubprocessManager, SubprocessStartupError
from products.tasks.backend.facade.agents import (
@@ -23,7 +36,14 @@ from products.tasks.backend.facade.agents import (
LocalSkillsCache,
)
-from .ports import LLM_GATEWAY_PORT, MCP_PORT, PERSONHOG_REPLICA_PORT, PERSONHOG_ROUTER_PORT, SKILL_ARCHIVE_PORT
+from .ports import (
+ AI_GATEWAY_PORT,
+ AI_GATEWAY_REDIS_URL,
+ MCP_PORT,
+ PERSONHOG_REPLICA_PORT,
+ PERSONHOG_ROUTER_PORT,
+ SKILL_ARCHIVE_PORT,
+)
from .providers import PreflightError
logger = logging.getLogger(__name__)
@@ -31,69 +51,238 @@ logger = logging.getLogger(__name__)
LONG_LIVED_SUBPROCESSES = LongLivedSubprocessManager()
_ZIP_FIXED_TIME = (2020, 1, 1, 0, 0, 0)
+EVAL_AI_PRODUCT = "posthog_ai"
+"""The gateway product eval tasks resolve to through their ``posthog_ai`` origin."""
-def start_llm_gateway(live_server_url: str) -> Callable[[], None]:
- """Start the LLM gateway as a subprocess.
+AI_GATEWAY_EVAL_DATABASE = "ai_gateway_evals"
+AI_GATEWAY_EVAL_CREDITS_USD = "10000"
+_AI_GATEWAY_INTERNAL_TOKEN = secrets.token_urlsafe(32)
+_AI_GATEWAY_CREDIT_GRANT_ID = secrets.token_hex(16)
+AI_GATEWAY_CREDENTIAL_REFRESH_SECONDS = max(1, min(3600, GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL // 3))
- Mirrors ``bin/start-llm-gateway``: runs uvicorn on a non-default port.
- The sandbox's agent-server uses this to proxy LLM calls to Anthropic, through the
- ``posthog_ai`` product that eval tasks select with their origin.
- """
- gateway_dir = Path(settings.BASE_DIR) / "services" / "llm-gateway"
- venv_dir = gateway_dir / ".venv"
- uvicorn_bin = venv_dir / "bin" / "uvicorn"
- if not uvicorn_bin.exists():
- raise SubprocessStartupError(
- f"LLM gateway venv not found at {venv_dir}. "
- "Run `bin/start-llm-gateway` once or `cd services/llm-gateway && uv venv .venv && uv pip install -e .`"
+def ai_gateway_repo() -> Path:
+ """The local ``PostHog/ai-gateway`` clone, at the path ``bin/start-ai-gateway`` uses."""
+ override = os.environ.get("AI_GATEWAY_REPO")
+ return (Path(override).expanduser() if override else Path.home() / "Development" / "ai-gateway").resolve()
+
+
+def _ai_gateway_binary() -> Path:
+ return ai_gateway_repo() / ".run" / "eval-gateway"
+
+
+def ensure_ai_gateway_binary() -> None:
+ """Build the Go ai-gateway from the local clone, during preflight.
+
+ The sandboxed agent's model calls go through the Go gateway, as they do in
+ production for PostHog AI runs. The build is incremental, so only the first
+ one after a pull is slow.
+ """
+ repo = ai_gateway_repo()
+ if not (repo / "cmd" / "gateway").is_dir():
+ raise PreflightError(
+ f"No PostHog/ai-gateway clone at {repo}. The sandboxed agent's model calls go through it. "
+ f"Run `git clone [email protected]:PostHog/ai-gateway.git {repo}`, "
+ "or set AI_GATEWAY_REPO to the path of your clone."
)
- # The LLM gateway uses pydantic BaseSettings with env_prefix="LLM_GATEWAY_".
- # DATABASE_URL and LLM_GATEWAY_ANTHROPIC_API_KEY come from the parent env
- # (e.g. .env.local / op run). We need to point it at the test database.
- conn = connections["default"]
- # The test DB setup rewrites settings_dict["NAME"] to the test DB name in
- # place, so don't re-prefix it.
- db_name = conn.settings_dict["NAME"]
- db_user = conn.settings_dict.get("USER", "posthog")
- db_password = conn.settings_dict.get("PASSWORD", "posthog")
- db_host = conn.settings_dict.get("HOST", "localhost")
- db_port = conn.settings_dict.get("PORT", "5432")
- test_db_url = f"postgres://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}"
-
- # No team rate-limit multiplier: the gateway keys multipliers by team id, but
- # each eval case runs as its own freshly minted team, so a mapping like {"1": 10}
- # never matches. Rate limits are per team, so every case gets the full base
- # budget (hundreds of requests/minute), which is already ample for eval traffic.
- env = {
- **os.environ,
- "UV_PROJECT_ENVIRONMENT": str(venv_dir),
- "LLM_GATEWAY_DATABASE_URL": test_db_url,
- "LLM_GATEWAY_DEBUG": "true",
- "LLM_GATEWAY_POSTHOG_HOST": live_server_url,
- }
+ if shutil.which("go") is None:
+ raise PreflightError("`go` not found on PATH. The harness builds the ai-gateway from your clone.")
+
+ logger.info("Building the ai-gateway from %s", repo)
+ result = subprocess.run(
+ ["go", "build", "-o", str(_ai_gateway_binary()), "./cmd/gateway"],
+ cwd=repo,
+ capture_output=True,
+ text=True,
+ timeout=10 * 60,
+ )
+ if result.returncode != 0:
+ tail = "\n".join(((result.stdout or "") + (result.stderr or "")).splitlines()[-30:])
+ raise PreflightError(f"Failed to build the ai-gateway (go exit {result.returncode}):\n{tail}")
+
+
+def _ai_gateway_compose(*args: str) -> subprocess.CompletedProcess[str]:
+ # PostHog's dev env exports COMPOSE_PROJECT_NAME=posthog, which outranks the
+ # gateway compose file's own project name.
+ return subprocess.run(
+ ["docker", "compose", *args],
+ cwd=ai_gateway_repo(),
+ env={**os.environ, "COMPOSE_PROJECT_NAME": "ai-gateway"},
+ check=True,
+ capture_output=True,
+ text=True,
+ timeout=5 * 60,
+ )
+
+
+def start_ai_gateway() -> Callable[[], None]:
+ """Start the Go ai-gateway on a non-default port, in resolver mode.
+
+ It shares the Postgres and Valkey containers of the gateway repo's compose file
+ with a dev-stack gateway, but uses its own database and Valkey DB index, so eval
+ ledger rows and team policy blobs never mix with dev ones. The containers stay up
+ after the run, as they do for ``bin/start-ai-gateway``.
+ """
+ _ai_gateway_compose("up", "-d", "--wait", "postgres", "valkey")
+ exists = _ai_gateway_compose(
+ "exec",
+ "-T",
+ "postgres",
+ "psql",
+ "-U",
+ "postgres",
+ "-tAc",
+ f"SELECT 1 FROM pg_database WHERE datname = '{AI_GATEWAY_EVAL_DATABASE}'",
+ ).stdout.strip()
+ if exists != "1":
+ _ai_gateway_compose("exec", "-T", "postgres", "createdb", "-U", "postgres", AI_GATEWAY_EVAL_DATABASE)
+
+ # A dev .env can carry AI_GATEWAY_* settings meant for Django or for a dev gateway.
+ env = {key: value for key, value in os.environ.items() if not key.startswith("AI_GATEWAY_")}
+ env.update(
+ {
+ "AI_GATEWAY_ENV": "dev",
+ "AI_GATEWAY_AUTH_MODE": "resolver",
+ "AI_GATEWAY_LISTEN_ADDR": f":{AI_GATEWAY_PORT}",
+ "AI_GATEWAY_POSTGRES_DSN": f"postgres://postgres:postgres@localhost:5435/{AI_GATEWAY_EVAL_DATABASE}",
+ "AI_GATEWAY_REDIS_URL": AI_GATEWAY_REDIS_URL,
+ "AI_GATEWAY_INTERNAL_TOKEN": _AI_GATEWAY_INTERNAL_TOKEN,
+ "AI_GATEWAY_ANTHROPIC_API_KEY": os.environ["LLM_GATEWAY_ANTHROPIC_API_KEY"],
+ # The gateway refuses to boot without an OpenAI key, which only codex runs use.
+ "AI_GATEWAY_OPENAI_API_KEY": os.environ.get("LLM_GATEWAY_OPENAI_API_KEY") or "unused",
+ "AI_GATEWAY_LOG_FORMAT": "text",
+ "AI_GATEWAY_LOG_LEVEL": "info",
+ }
+ )
+ binary = _ai_gateway_binary()
+ migrate = subprocess.run([str(binary), "migrate"], env=env, capture_output=True, text=True, timeout=120)
+ if migrate.returncode != 0:
+ raise SubprocessStartupError(f"ai-gateway ledger migration failed:\n{migrate.stdout}{migrate.stderr}")
- logger.info("Starting LLM gateway on port %d", LLM_GATEWAY_PORT)
+ logger.info("Starting ai-gateway on port %d", AI_GATEWAY_PORT)
_, stop = LONG_LIVED_SUBPROCESSES.start(
- name="LLM gateway",
- port=LLM_GATEWAY_PORT,
- cmd=[
- str(uvicorn_bin),
- "llm_gateway.main:app",
- "--host",
- "0.0.0.0",
- "--port",
- str(LLM_GATEWAY_PORT),
- ],
- cwd=gateway_dir,
+ name="ai-gateway",
+ port=AI_GATEWAY_PORT,
+ cmd=[str(binary)],
+ cwd=ai_gateway_repo(),
env=env,
- log_prefix="llm-gateway",
+ log_prefix="ai-gateway",
)
-
- logger.info("LLM gateway ready on port %d", LLM_GATEWAY_PORT)
+ try:
+ _wait_for_ai_gateway_ready()
+ except Exception:
+ stop()
+ raise
+ logger.info("ai-gateway ready on port %d", AI_GATEWAY_PORT)
return stop
+def _wait_for_ai_gateway_ready(timeout_seconds: float = 120) -> None:
+ # The gateway listens before its model catalog is built. Until a provider listing
+ # succeeds, /readyz answers 503 and /v1/models is empty. A bad provider key never gets past this.
+ deadline = time.monotonic() + timeout_seconds
+ body = ""
+ while time.monotonic() < deadline:
+ try:
+ response = requests.get(f"http://localhost:{AI_GATEWAY_PORT}/readyz", timeout=5)
+ except requests.RequestException as e:
+ body = str(e)
+ else:
+ if response.status_code == 200:
+ return
+ body = response.text[:500]
+ time.sleep(1)
+ raise SubprocessStartupError(
+ f"ai-gateway did not become ready in {timeout_seconds:.0f}s. Check the provider API keys. Last /readyz: {body}"
+ )
+
+
+def refresh_ai_gateway_mint_key(team_id: int, mint_key: str) -> None:
+ try:
+ call_command("setup_local_gateway_credential", "--phs", mint_key, "--team", str(team_id), stdout=io.StringIO())
+ call_command("llm_gateway_team", "refresh", str(team_id), stdout=io.StringIO())
+ finally:
+ close_old_connections()
+
+
+async def maintain_ai_gateway_mint_key(team_id: int, mint_key: str) -> AsyncIterator[None]:
+ async def refresh() -> None:
+ expires_at = time.monotonic() + GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL
+ while True:
+ await asyncio.sleep(AI_GATEWAY_CREDENTIAL_REFRESH_SECONDS)
+ retry_delay = 1.0
+ while True:
+ pending = asyncio.create_task(asyncio.to_thread(refresh_ai_gateway_mint_key, team_id, mint_key))
+ try:
+ await asyncio.shield(pending)
+ except asyncio.CancelledError:
+ await pending
+ raise
+ except (InterfaceError, OperationalError, RedisConnectionError, RedisTimeoutError):
+ remaining = expires_at - time.monotonic()
+ if remaining <= 0:
+ raise
+ logger.warning("Gateway credential refresh failed; retrying", exc_info=True)
+ await asyncio.sleep(min(retry_delay, remaining))
+ retry_delay = min(retry_delay * 2, 60)
+ else:
+ expires_at = time.monotonic() + GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL
+ break
+
+ async with asyncio.TaskGroup() as group:
+ task = group.create_task(refresh())
+ try:
+ yield
+ finally:
+ task.cancel()
+
+
+def provision_ai_gateway_mint_key(team_id: int, *, agent_model: str) -> str:
+ """Give the master demo team a ``phs_`` key the worker mints per-run tokens with.
+
+ Each run's ``phe_`` token bills the minting team, so only the master team needs
+ admission and spend headroom, not the per-case teams. Scoped tokens cannot
+ use an overspend allowance, so the isolated eval ledger needs test credits.
+ A model listing and one mint check the catalog and the credential, so a broken
+ setup fails here and not as a model error in every case.
+ """
+ mint_key = f"phs_{secrets.token_hex(24)}"
+ refresh_ai_gateway_mint_key(team_id, mint_key)
+
+ base_url = f"http://localhost:{AI_GATEWAY_PORT}"
+ credits = requests.post(
+ f"{base_url}/internal/teams/{team_id}/credits",
+ json={"amount_usd": AI_GATEWAY_EVAL_CREDITS_USD, "reason": "Local eval test credits"},
+ headers={
+ "Authorization": f"Bearer {_AI_GATEWAY_INTERNAL_TOKEN}",
+ "Idempotency-Key": f"eval-{team_id}-{_AI_GATEWAY_CREDIT_GRANT_ID}",
+ "X-Internal-Actor": "eval-harness",
+ },
+ timeout=10,
+ )
+ if not 200 <= credits.status_code < 300:
+ raise SubprocessStartupError(f"ai-gateway refused eval test credits: HTTP {credits.status_code}")
+ headers = {"Authorization": f"Bearer {mint_key}"}
+ models = requests.get(f"{base_url}/v1/models", headers=headers, timeout=10)
+ models.raise_for_status()
+ model_ids = {entry.get("id") for entry in models.json().get("data", [])}
+ if agent_model not in model_ids:
+ raise SubprocessStartupError(f"The ai-gateway model catalog does not serve {agent_model!r}")
+ mint = requests.post(
+ f"{base_url}/v1/tokens",
+ json={"cap_usd": "0.01", "ttl_seconds": 60, "product": EVAL_AI_PRODUCT, "obo": str(team_id)},
+ headers=headers,
+ timeout=10,
+ )
+ if not 200 <= mint.status_code < 300:
+ raise SubprocessStartupError(
+ f"ai-gateway refused to mint a run token: HTTP {mint.status_code} {mint.text[:300]}"
+ )
+ logger.info("ai-gateway mint key provisioned for team_id=%d", team_id)
+ return mint_key
+
+
def start_mcp_server(
live_server_url: str,
skill_archive_url: str | None,
diff --git a/products/posthog_ai/eval_harness/harness/tunnels.py b/products/posthog_ai/eval_harness/harness/tunnels.py
index 54d31190808f..c624989e209b 100644
--- a/products/posthog_ai/eval_harness/harness/tunnels.py
+++ b/products/posthog_ai/eval_harness/harness/tunnels.py
@@ -19,7 +19,7 @@ SETUP_GUIDE = "docs/internal/sandboxes-setup-guide.md"
FUNNEL_PUBLIC_PORTS: tuple[int, ...] = (443, 8443, 10000)
"""The only public ports Tailscale Funnel will serve — the daemon rejects any
-other. Three is exactly enough for the harness's Django, LLM gateway, and MCP
+other. Three is exactly enough for the harness's Django, AI gateway, and MCP
callbacks; each requested service is assigned one in the order the caller lists
them."""
@@ -77,7 +77,7 @@ def funnel_preflight_error() -> str | None:
if shutil.which("tailscale") is None:
return (
"`tailscale` not found on PATH. Modal sandboxes run outside this host, so the "
- "Django API, LLM gateway, and MCP server must be publicly reachable via Tailscale Funnel. "
+ "Django API, AI gateway, and MCP server must be publicly reachable via Tailscale Funnel. "
f"See {SETUP_GUIDE}."
)
status = tailscale_status()
@@ -167,7 +167,7 @@ def _relay_answers(ip: str, host: str, public_port: int) -> bool:
class TailscaleFunnel:
"""Modal-only Tailscale Funnel lifecycle: publicly exposes the host services a
- remote Modal sandbox must reach (Django API, LLM gateway, MCP server).
+ remote Modal sandbox must reach (Django API, AI gateway, MCP server).
Each service is served over HTTPS on one of Tailscale's three permitted public
ports and is reachable by anyone on the internet who learns the URL, so the
@@ -326,7 +326,7 @@ class TailscaleFunnel:
def _failure_message(self, detail: str) -> str:
return (
f"{detail}\n"
- "Tailscale Funnel could not serve the Django, LLM gateway, and MCP services. "
+ "Tailscale Funnel could not serve the Django, AI gateway, and MCP services. "
"Make sure tailscaled is running and this node is logged in (`tailscale up`), that Funnel "
"is enabled for the tailnet and this node in the admin console's ACLs, and that HTTPS "
f"certificates are enabled for the tailnet. See {SETUP_GUIDE}."
diff --git a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py b/products/posthog_ai/eval_harness/test/test_one_shot_runner.py
index fb60ac4eade3..7553667c1193 100644
--- a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py
+++ b/products/posthog_ai/eval_harness/test/test_one_shot_runner.py
@@ -12,6 +12,7 @@ import pytest
from unittest.mock import patch
from django.conf import settings
+from django.test import override_settings
from posthoganalytics import Posthog
@@ -278,7 +279,7 @@ def test_run_routes_through_the_engine(
),
):
harness = SandboxedEvalHarness(parse_args(["--agent-model", "claude-test"]))
- with harness._stack:
+ with override_settings(AI_GATEWAY_REDIS_URL="redis://localhost:6381/5"), harness._stack:
harness._bootstrap(frozenset(), ProgressReporter(total_suites=0))
ctx = harness._build_context(frozenset(), reporter) # type: ignore[arg-type]
runs = [_build_run(tmp_path, monkeypatch, ctx, task) for _ in range(2)]
@@ -290,6 +291,7 @@ def test_run_routes_through_the_engine(
run.case_trace_meta["c1"] = {"prompt": "the prompt", "duration": 1.0, "first_timestamp": ""}
asyncio.run(run_suite(run))
assert settings.TEST
+ assert settings.AI_GATEWAY_REDIS_URL is None
assert ctx.posthog_client is not None
assert ctx.posthog_client.disabled
ctx.posthog_client.capture(event="ordinary_test_event", distinct_id="test")
diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py
index 9d769aca6342..e556c786fc3a 100644
--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py
+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py
@@ -7,7 +7,7 @@ import subprocess
from collections.abc import Awaitable, Callable
from pathlib import Path
from types import SimpleNamespace
-from typing import Literal, cast
+from typing import Any, Literal, cast
from uuid import UUID
import pytest
@@ -25,7 +25,11 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.harness.discovery import EvalSuite
from products.posthog_ai.eval_harness.harness.lifecycle import SandboxedEvalHarness
from products.posthog_ai.eval_harness.harness.live_server import EvalLiveServer
-from products.posthog_ai.eval_harness.harness.providers import ModalProviderStrategy, SandboxProviderStrategy
+from products.posthog_ai.eval_harness.harness.providers import (
+ DockerProviderStrategy,
+ ModalProviderStrategy,
+ SandboxProviderStrategy,
+)
from products.posthog_ai.eval_harness.harness.reporting import ProgressReporter
from products.tasks.backend.facade.agents import MultiTurnSession, TurnPollResult, TurnPollTimeout
from products.tasks.backend.facade.contracts import AgentTaskRunDTO
@@ -443,6 +447,31 @@ def test_modal_cleanup_case_terminates_only_the_task_sandboxes(monkeypatch: pyte
sandbox.terminate.assert_called_once_with()
+def _docker_provider() -> SandboxProviderStrategy:
+ return DockerProviderStrategy(keep_containers=False)
+
+
+def _modal_provider() -> SandboxProviderStrategy:
+ provider = ModalProviderStrategy()
+ provider._tunnels = cast(Any, SimpleNamespace(url_for=lambda name: f"https://{name}.funnel.example.com"))
+ return provider
+
+
+ [
+ ("docker", _docker_provider, "http://localhost:18080", "http://host.docker.internal:18080"),
+ ("modal", _modal_provider, "https://gateway.funnel.example.com", "https://gateway.funnel.example.com"),
+ ]
+)
+def test_provider_routes_model_calls_to_a_gateway_the_host_worker_can_mint_through(
+ _name: str, build: Callable[[], SandboxProviderStrategy], gateway_url: str, fallback_url: str
+) -> None:
+ overrides = build().settings_overrides()
+
+ assert overrides["SANDBOX_AI_GATEWAY_URL"] == gateway_url
+ assert overrides["SANDBOX_LLM_GATEWAY_URL"] == fallback_url
+
+
class TestAgentRunFailureDetection:
# An agent-server failure reaches the log as a session error, and the workflow still finishes
# cleanly afterwards. Miss it and a run where no agent ever spoke reports exit_code_zero=1 with
diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_services.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_services.py
index 7a1fa6db923c..4bf762a8ce0a 100644
--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_services.py
+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_services.py
@@ -2,16 +2,34 @@ from __future__ import annotations
import os
import json
+import asyncio
import zipfile
import itertools
+import subprocess
from functools import partial
from pathlib import Path
+from types import SimpleNamespace
import pytest
from unittest.mock import MagicMock, patch
+from django.db import InterfaceError, OperationalError
+
+import requests_mock
+from redis.exceptions import (
+ ConnectionError as RedisConnectionError,
+ TimeoutError as RedisTimeoutError,
+)
+
from products.posthog_ai.eval_harness import long_lived_subprocess
-from products.posthog_ai.eval_harness.harness.services import package_local_skills_archive, start_mcp_server
+from products.posthog_ai.eval_harness.harness import services
+from products.posthog_ai.eval_harness.harness.services import (
+ ensure_ai_gateway_binary,
+ maintain_ai_gateway_mint_key,
+ package_local_skills_archive,
+ provision_ai_gateway_mint_key,
+ start_mcp_server,
+)
from products.posthog_ai.eval_harness.long_lived_subprocess import (
LongLivedSubprocessManager,
SubprocessReadinessTimeout,
@@ -153,3 +171,110 @@ def test_service_startup_does_not_retry_permanent_errors_or_interrupts(
with pytest.raises(type(error)):
manager.start(name="test service", port=12345, cmd=["fake-service"], cwd=tmp_path, env={}, log_prefix="test")
assert start.call_count == 1
+
+
[email protected]("credit_status", [200, 503])
+def test_gateway_provisioning_requires_test_credits(requests_mock: requests_mock.Mocker, credit_status: int) -> None:
+ base_url = "http://localhost:18080"
+ credits = requests_mock.post(f"{base_url}/internal/teams/42/credits", status_code=credit_status, json={})
+ requests_mock.get(f"{base_url}/v1/models", json={"data": [{"id": "test-model"}]})
+ requests_mock.post(f"{base_url}/v1/tokens", json={"token": "phe_test"})
+
+ with patch("products.posthog_ai.eval_harness.harness.services.call_command"):
+ if credit_status == 200:
+ assert provision_ai_gateway_mint_key(42, agent_model="test-model").startswith("phs_")
+ else:
+ with pytest.raises(SubprocessStartupError, match="refused eval test credits: HTTP 503"):
+ provision_ai_gateway_mint_key(42, agent_model="test-model")
+
+ assert credits.last_request is not None
+ assert credits.last_request.json() == {"amount_usd": "10000", "reason": "Local eval test credits"}
+ assert credits.last_request.headers["Idempotency-Key"] != credits.last_request.headers["Authorization"]
+
+
[email protected]("relative_path", ["gateway", "./gateway"])
+def test_gateway_build_from_relative_clone_creates_the_binary_used_at_startup(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, relative_path: str
+) -> None:
+ repo = tmp_path / "gateway"
+ (repo / "cmd" / "gateway").mkdir(parents=True)
+ monkeypatch.chdir(tmp_path)
+ monkeypatch.setenv("AI_GATEWAY_REPO", relative_path)
+ monkeypatch.setattr(services.shutil, "which", lambda _name: "/go")
+
+ def build(cmd: list[str], *, cwd: Path, **_kwargs: object) -> subprocess.CompletedProcess[str]:
+ output = cwd / cmd[cmd.index("-o") + 1]
+ output.parent.mkdir(parents=True, exist_ok=True)
+ output.write_text("binary")
+ return subprocess.CompletedProcess(cmd, 0, stdout="", stderr="")
+
+ monkeypatch.setattr(services.subprocess, "run", build)
+ ensure_ai_gateway_binary()
+
+ assert (repo / ".run" / "eval-gateway").read_text() == "binary"
+
+
+ "error_type", [None, InterfaceError, OperationalError, RedisConnectionError, RedisTimeoutError]
+)
+async def test_gateway_mint_credential_is_renewed_without_rotating_the_key(
+ monkeypatch: pytest.MonkeyPatch, error_type: type[Exception] | None
+) -> None:
+ loop = asyncio.get_running_loop()
+ refreshed = asyncio.Event()
+ parked = asyncio.Event()
+ intervals: list[float] = []
+ keys: list[str] = []
+ attempts = 0
+
+ async def tick(seconds: float) -> None:
+ intervals.append(seconds)
+ if len(intervals) > (2 if error_type else 1):
+ await parked.wait()
+
+ def command(name: str, *args: str, **_kwargs: object) -> None:
+ nonlocal attempts
+ if name == "setup_local_gateway_credential":
+ attempts += 1
+ if attempts == 1 and error_type:
+ raise error_type("temporary connection failure")
+ keys.append(args[args.index("--phs") + 1])
+ loop.call_soon_threadsafe(refreshed.set)
+
+ monkeypatch.setattr(services.asyncio, "sleep", tick)
+ monkeypatch.setattr(services, "call_command", command)
+ async with maintain_ai_gateway_mint_key(42, "phs_test"):
+ await asyncio.wait_for(refreshed.wait(), timeout=5)
+
+ assert keys == ["phs_test"]
+ assert intervals[0] < services.GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL
+ if error_type:
+ assert intervals[1] == 1
+
+
[email protected]("error_type", [OperationalError, RedisConnectionError, ValueError])
+async def test_gateway_mint_credential_refresh_stops_at_expiry_or_permanent_failure(
+ monkeypatch: pytest.MonkeyPatch, error_type: type[Exception]
+) -> None:
+ now = 0.0
+ intervals: list[float] = []
+
+ async def tick(seconds: float) -> None:
+ nonlocal now
+ intervals.append(seconds)
+ now += seconds
+
+ monkeypatch.setattr(services, "time", SimpleNamespace(monotonic=lambda: now))
+ monkeypatch.setattr(services.asyncio, "sleep", tick)
+ monkeypatch.setattr(services, "GATEWAY_CREDENTIAL_SECRET_KEY_CACHE_TTL", 10)
+ monkeypatch.setattr(services, "AI_GATEWAY_CREDENTIAL_REFRESH_SECONDS", 3)
+ monkeypatch.setattr(services, "refresh_ai_gateway_mint_key", MagicMock(side_effect=error_type("refresh failed")))
+
+ with pytest.raises(ExceptionGroup) as failure:
+ async with maintain_ai_gateway_mint_key(42, "phs_test"):
+ await asyncio.wait_for(asyncio.Event().wait(), timeout=5)
+
+ assert isinstance(failure.value.exceptions[0], error_type)
+ assert intervals == ([3] if error_type is ValueError else [3, 1, 2, 4])