:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(signals): run evals through the Go ai-gateway (#124792)
Jonathan Mieloob51d6b0dd1a1
diff --git a/products/signals/eval/AGENTS.md b/products/signals/eval/AGENTS.mdindex aad18c702141..c0a3e23ef6b1 100644--- a/products/signals/eval/AGENTS.md+++ b/products/signals/eval/AGENTS.md@@ -18,7 +18,7 @@ interleaved across groups to simulate real-world arrival patterns.### Pipeline stages per signal-1. **Pre-emit** — summarize long descriptions, check actionability (Claude Haiku via the internal LLM gateway).+1. **Pre-emit** — summarize long descriptions, check actionability (Claude Haiku via the Go ai-gateway).Signals that fail actionability are dropped.2. **Match** — generate search queries (LLM), embed queries (OpenAI),cosine search against stored signals, LLM match to existing report or create new, verify specificity of match (LLM).@@ -41,18 +41,19 @@ pytest products/signals/eval/eval_grouping_e2e.py -xvs --limit 10 --no-capture### Required environment variables-Set in `.env` at the repo root (loaded automatically):--| Variable | Purpose |-| ------------------------------ | ---------------------------------------------------------------------------------------------------------- |-| `OPENAI_API_KEY` | Embeddings (text-embedding-3-small) |-| `LLM_GATEWAY_URL` | Internal LLM gateway base URL (all LLM calls: matching, specificity, summarization, actionability, safety) |-| `LLM_GATEWAY_PERSONAL_API_KEY` | Bearer token for the LLM gateway (PostHog personal API key) |-| `SIGNALS_EVAL_TEAM_ID` | Team id used for LLM cost attribution headers (defaults to `1`) |-| `POSTHOG_OFFLINE_EVAL_API_KEY` | Offline result uploads; requires `offline_evaluation_ingestion:write` |-| `POSTHOG_OFFLINE_EVAL_HOST` | Offline API origin (defaults to `https://us.posthog.com`; project `2`) |-| `POSTHOG_PROJECT_API_KEY` | Optional LLM trace capture |-| `POSTHOG_HOST` | Trace destination (defaults to `http://localhost:8010`) |+Set in `.env` at the repo root (loaded automatically).+The gateway token expires, so pass it per run instead: `AI_GATEWAY_API_KEY=$(hogli gateway:token) pytest ...`.++| Variable | Purpose |+| ------------------------------ | ---------------------------------------------------------------------------------------------- |+| `OPENAI_API_KEY` | Embeddings (text-embedding-3-small) |+| `AI_GATEWAY_URL` | Go ai-gateway base URL with `/v1`, e.g. `https://ai-gateway.us.posthog.com/v1` (all LLM calls) |+| `AI_GATEWAY_API_KEY` | Developer-mode `phe_` token from `hogli gateway:token` (8h TTL, $25 per token, $50 per day) |+| `SIGNALS_EVAL_TEAM_ID` | Team id used for LLM cost attribution headers (defaults to `1`) |+| `POSTHOG_OFFLINE_EVAL_API_KEY` | Offline result uploads; requires `offline_evaluation_ingestion:write` |+| `POSTHOG_OFFLINE_EVAL_HOST` | Offline API origin (defaults to `https://us.posthog.com`; project `2`) |+| `POSTHOG_PROJECT_API_KEY` | Optional LLM trace capture |+| `POSTHOG_HOST` | Trace destination (defaults to `http://localhost:8010`) |### CLI optionsdiff --git a/products/signals/eval/conftest.py b/products/signals/eval/conftest.pyindex 4472cd369417..e03cab7c020c 100644--- a/products/signals/eval/conftest.py+++ b/products/signals/eval/conftest.py@@ -12,7 +12,7 @@ from dotenv import load_dotenvfrom posthoganalytics import Posthogfrom posthoganalytics.ai.openai import AsyncOpenAI-from posthog.llm.gateway_client import get_async_anthropic_gateway_client+from posthog.llm.gateway_client import build_async_anthropic_client# This eval runs via bare pytest (no hogli), so the conftest owns loading repo-root .env.load_dotenv(Path(__file__).resolve().parents[3] / ".env")@@ -29,12 +29,10 @@ posthoganalytics.default_client = Posthog(# Override settings that need to come from .env.if not getattr(settings, "OPENAI_API_KEY", ""):settings.OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY", "")-if not getattr(settings, "LLM_GATEWAY_URL", ""):- settings.LLM_GATEWAY_URL = os.environ.get("LLM_GATEWAY_URL", "")-if not getattr(settings, "LLM_GATEWAY_API_KEY", ""):- settings.LLM_GATEWAY_API_KEY = os.environ.get("LLM_GATEWAY_PERSONAL_API_KEY", "") or os.environ.get(- "LLM_GATEWAY_API_KEY", ""- )+if not getattr(settings, "AI_GATEWAY_URL", ""):+ settings.AI_GATEWAY_URL = os.environ.get("AI_GATEWAY_URL", "")+if not getattr(settings, "AI_GATEWAY_API_KEY", ""):+ settings.AI_GATEWAY_API_KEY = os.environ.get("AI_GATEWAY_API_KEY", "")# Team id used by the eval harness when attributing LLM cost via the gateway.EVAL_TEAM_ID = int(os.environ.get("SIGNALS_EVAL_TEAM_ID", "1"))@@ -100,12 +98,12 @@ async def openai_client(posthog_client):@pytest.fixturedef gateway_client():- """Async Anthropic client pointed at the internal LLM gateway's native Messages endpoint.+ """Async Anthropic client pointed at the Go ai-gateway's native Messages endpoint.Used by eval_grouping_e2e to drive the production signals pre-emit pipeline(`check_actionability` etc.) through the gateway, attributing cost to EVAL_TEAM_ID."""- return get_async_anthropic_gateway_client(product="signals", team_id=EVAL_TEAM_ID)+ return build_async_anthropic_client(product="signals", ai_product="signals_eval", team_id=EVAL_TEAM_ID)@pytest.fixturediff --git a/products/signals/eval/eval_scout_safety.py b/products/signals/eval/eval_scout_safety.pyindex a3a171f2e607..c58f103391b1 100644--- a/products/signals/eval/eval_scout_safety.py+++ b/products/signals/eval/eval_scout_safety.py@@ -7,7 +7,7 @@ references, and priority framing — the dominant emit→assign drop. `safety_fitrust-aware prompt for `source_product="signals_scout"`. This eval feeds it real false-positives(expected SAFE) and synthetic payloads (expected UNSAFE) and measures both directions.-Run (needs the LLM gateway env, same as eval_grouping_e2e):+Run (needs the AI gateway env, same as eval_grouping_e2e):pytest products/signals/eval/eval_scout_safety.py -xvspytest products/signals/eval/eval_scout_safety.py -xvs --no-capture"""