Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(signals): run evals through the Go ai-gateway (#124792)

Jonathan Mieloob51d6b0dd1a1
diff --git a/products/signals/eval/AGENTS.md b/products/signals/eval/AGENTS.md
index aad18c702141..c0a3e23ef6b1 100644
--- a/products/signals/eval/AGENTS.md
+++ b/products/signals/eval/AGENTS.md
@@ -18,7 +18,7 @@ interleaved across groups to simulate real-world arrival patterns.
### Pipeline stages per signal
-1. **Pre-emit** — summarize long descriptions, check actionability (Claude Haiku via the internal LLM gateway).
+1. **Pre-emit** — summarize long descriptions, check actionability (Claude Haiku via the Go ai-gateway).
Signals that fail actionability are dropped.
2. **Match** — generate search queries (LLM), embed queries (OpenAI),
cosine search against stored signals, LLM match to existing report or create new, verify specificity of match (LLM).
@@ -41,18 +41,19 @@ pytest products/signals/eval/eval_grouping_e2e.py -xvs --limit 10 --no-capture
### Required environment variables
-Set in `.env` at the repo root (loaded automatically):
-
-| Variable | Purpose |
-| ------------------------------ | ---------------------------------------------------------------------------------------------------------- |
-| `OPENAI_API_KEY` | Embeddings (text-embedding-3-small) |
-| `LLM_GATEWAY_URL` | Internal LLM gateway base URL (all LLM calls: matching, specificity, summarization, actionability, safety) |
-| `LLM_GATEWAY_PERSONAL_API_KEY` | Bearer token for the LLM gateway (PostHog personal API key) |
-| `SIGNALS_EVAL_TEAM_ID` | Team id used for LLM cost attribution headers (defaults to `1`) |
-| `POSTHOG_OFFLINE_EVAL_API_KEY` | Offline result uploads; requires `offline_evaluation_ingestion:write` |
-| `POSTHOG_OFFLINE_EVAL_HOST` | Offline API origin (defaults to `https://us.posthog.com`; project `2`) |
-| `POSTHOG_PROJECT_API_KEY` | Optional LLM trace capture |
-| `POSTHOG_HOST` | Trace destination (defaults to `http://localhost:8010`) |
+Set in `.env` at the repo root (loaded automatically).
+The gateway token expires, so pass it per run instead: `AI_GATEWAY_API_KEY=$(hogli gateway:token) pytest ...`.
+
+| Variable | Purpose |
+| ------------------------------ | ---------------------------------------------------------------------------------------------- |
+| `OPENAI_API_KEY` | Embeddings (text-embedding-3-small) |
+| `AI_GATEWAY_URL` | Go ai-gateway base URL with `/v1`, e.g. `https://ai-gateway.us.posthog.com/v1` (all LLM calls) |
+| `AI_GATEWAY_API_KEY` | Developer-mode `phe_` token from `hogli gateway:token` (8h TTL, $25 per token, $50 per day) |
+| `SIGNALS_EVAL_TEAM_ID` | Team id used for LLM cost attribution headers (defaults to `1`) |
+| `POSTHOG_OFFLINE_EVAL_API_KEY` | Offline result uploads; requires `offline_evaluation_ingestion:write` |
+| `POSTHOG_OFFLINE_EVAL_HOST` | Offline API origin (defaults to `https://us.posthog.com`; project `2`) |
+| `POSTHOG_PROJECT_API_KEY` | Optional LLM trace capture |
+| `POSTHOG_HOST` | Trace destination (defaults to `http://localhost:8010`) |
### CLI options
diff --git a/products/signals/eval/conftest.py b/products/signals/eval/conftest.py
index 4472cd369417..e03cab7c020c 100644
--- a/products/signals/eval/conftest.py
+++ b/products/signals/eval/conftest.py
@@ -12,7 +12,7 @@ from dotenv import load_dotenv
from posthoganalytics import Posthog
from posthoganalytics.ai.openai import AsyncOpenAI
-from posthog.llm.gateway_client import get_async_anthropic_gateway_client
+from posthog.llm.gateway_client import build_async_anthropic_client
# This eval runs via bare pytest (no hogli), so the conftest owns loading repo-root .env.
load_dotenv(Path(__file__).resolve().parents[3] / ".env")
@@ -29,12 +29,10 @@ posthoganalytics.default_client = Posthog(
# Override settings that need to come from .env.
if not getattr(settings, "OPENAI_API_KEY", ""):
settings.OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY", "")
-if not getattr(settings, "LLM_GATEWAY_URL", ""):
- settings.LLM_GATEWAY_URL = os.environ.get("LLM_GATEWAY_URL", "")
-if not getattr(settings, "LLM_GATEWAY_API_KEY", ""):
- settings.LLM_GATEWAY_API_KEY = os.environ.get("LLM_GATEWAY_PERSONAL_API_KEY", "") or os.environ.get(
- "LLM_GATEWAY_API_KEY", ""
- )
+if not getattr(settings, "AI_GATEWAY_URL", ""):
+ settings.AI_GATEWAY_URL = os.environ.get("AI_GATEWAY_URL", "")
+if not getattr(settings, "AI_GATEWAY_API_KEY", ""):
+ settings.AI_GATEWAY_API_KEY = os.environ.get("AI_GATEWAY_API_KEY", "")
# Team id used by the eval harness when attributing LLM cost via the gateway.
EVAL_TEAM_ID = int(os.environ.get("SIGNALS_EVAL_TEAM_ID", "1"))
@@ -100,12 +98,12 @@ async def openai_client(posthog_client):
@pytest.fixture
def gateway_client():
- """Async Anthropic client pointed at the internal LLM gateway's native Messages endpoint.
+ """Async Anthropic client pointed at the Go ai-gateway's native Messages endpoint.
Used by eval_grouping_e2e to drive the production signals pre-emit pipeline
(`check_actionability` etc.) through the gateway, attributing cost to EVAL_TEAM_ID.
"""
- return get_async_anthropic_gateway_client(product="signals", team_id=EVAL_TEAM_ID)
+ return build_async_anthropic_client(product="signals", ai_product="signals_eval", team_id=EVAL_TEAM_ID)
@pytest.fixture
diff --git a/products/signals/eval/eval_scout_safety.py b/products/signals/eval/eval_scout_safety.py
index a3a171f2e607..c58f103391b1 100644
--- a/products/signals/eval/eval_scout_safety.py
+++ b/products/signals/eval/eval_scout_safety.py
@@ -7,7 +7,7 @@ references, and priority framing — the dominant emit→assign drop. `safety_fi
trust-aware prompt for `source_product="signals_scout"`. This eval feeds it real false-positives
(expected SAFE) and synthetic payloads (expected UNSAFE) and measures both directions.
-Run (needs the LLM gateway env, same as eval_grouping_e2e):
+Run (needs the AI gateway env, same as eval_grouping_e2e):
pytest products/signals/eval/eval_scout_safety.py -xvs
pytest products/signals/eval/eval_scout_safety.py -xvs --no-capture
"""