Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(evals): seed demo data before building skills, fail on a stale build (#120945)

Thomas Obermüllerb786f531fc4e
diff --git a/products/posthog_ai/eval_harness/harness/README.md b/products/posthog_ai/eval_harness/harness/README.md
index 41eb4e84f7fb..4f1015d3f50c 100644
--- a/products/posthog_ai/eval_harness/harness/README.md
+++ b/products/posthog_ai/eval_harness/harness/README.md
@@ -47,7 +47,7 @@ A run of one-shot suites never pays for — or fails preflight on — the sandbo
Discovery happens first, before anything is provisioned, so a typo'd selector costs a module import rather than a database build.
Every step below is gated on the selected suites' infrastructure union — a step whose `Infra` member no run requires is skipped, and skipped steps register no teardown.
-The sequence as written is the sandboxed (full) path; a one-shot-only run performs just steps 1–3 and 8.
+The sequence as written is the sandboxed (full) path; a one-shot-only run performs just steps 1–3 and 6.
The bootstrap is deliberately synchronous and runs before any event loop exists, because it is ORM-heavy and Django's async-safety guard rejects sync ORM calls from an async context:
@@ -56,10 +56,12 @@ The bootstrap is deliberately synchronous and runs before any event loop exists,
2. `EvalDatabase.setup()` creates the `default` test database and drives PostHog's own eval database setup (persons database, ClickHouse).
3. `personhog-replica` (`:15051`) and `personhog-router` (`:15052`) start against the test persons database — before anything can query, so a dead router never poisons the negative group-types cache.
4. `EvalLiveServer` serves PostHog's ASGI application on `0.0.0.0:18000`, including the sandbox event-ingest route.
-5. The Go AI gateway (`:18080`) starts in resolver mode, on the gateway compose Postgres (database `ai_gateway_evals`) and Valkey (DB 5), and local skills are built. In bundled mode, the MCP server (`:18787`) starts with exec skill distribution forced off. In exec mode, the skills are packaged, the local archive server (`:18788`) starts, and MCP receives its content-addressed URL with distribution forced on.
-6. Docker only: the `posthog-sandbox-base` image freshness check runs (rebuilding when the Dockerfile or its agent pin changed). Modal only: Tailscale Funnel comes up, exposing the callback services publicly. Setup then waits until every public Funnel relay serves every service, because the relays start serving at different times and can take minutes.
-7. Bundled mode bind-mounts the built skills on Docker and bakes them into the Modal image. Exec mode suppresses those skills and clears native skill directories before every case; `disable_bundled_skills=True` remains available as a per-case override in other modes.
-8. The master Hedgebox team is seeded. It then gets the AI gateway mint key: a `phs_` key, gateway admission, and test credits in the separate eval ledger. One model listing and one token mint check that path before any case starts. The harness refreshes the same mint credential before its cache TTL expires during the run.
+5. The Go AI gateway (`:18080`) starts in resolver mode, on the gateway compose Postgres (database `ai_gateway_evals`) and Valkey (DB 5).
+6. The master Hedgebox team is seeded. This happens before the skill build, because HogQL examples in skill templates render against a Team, and a fresh test database has none until this step.
+7. Local skills are built. A failed build stops the run instead of reusing an older `dist/skills`. In bundled mode, the MCP server (`:18787`) starts with exec skill distribution forced off. In exec mode, the skills are packaged, the local archive server (`:18788`) starts, and MCP receives its content-addressed URL with distribution forced on.
+8. Docker only: the `posthog-sandbox-base` image freshness check runs (rebuilding when the Dockerfile or its agent pin changed). Modal only: Tailscale Funnel comes up, exposing the callback services publicly. Setup then waits until every public Funnel relay serves every service, because the relays start serving at different times and can take minutes.
+9. Bundled mode bind-mounts the built skills on Docker and bakes them into the Modal image. Exec mode suppresses those skills and clears native skill directories before every case; `disable_bundled_skills=True` remains available as a per-case override in other modes.
+10. The master team gets the AI gateway mint key: a `phs_` key, gateway admission, and test credits in the separate eval ledger. One model listing and one token mint check that path before any case starts. The harness refreshes the same mint credential before its cache TTL expires during the run.
Gateway cache writes stay disabled until the gateway starts. A one-shot run leaves them disabled.
diff --git a/products/posthog_ai/eval_harness/harness/lifecycle.py b/products/posthog_ai/eval_harness/harness/lifecycle.py
index 52f25315bf32..91e58fb96614 100644
--- a/products/posthog_ai/eval_harness/harness/lifecycle.py
+++ b/products/posthog_ai/eval_harness/harness/lifecycle.py
@@ -328,6 +328,23 @@ class SandboxedEvalHarness:
reporter.setup_started("start AI gateway")
self._stack.callback(start_ai_gateway())
self._stack.enter_context(override_settings(AI_GATEWAY_REDIS_URL=AI_GATEWAY_REDIS_URL))
+ # Seed before building skills: HogQL examples in skill templates render against a Team,
+ # and on a fresh test database the master team is the first one that exists.
+ if Infra.DEMO_DATA in required:
+ reporter.setup_started("prepare demo data")
+ assert self._database is not None
+ self._demo_data = ensure_demo_ready(
+ blocker=self._database.blocker,
+ agent_model=self.options.agent_model,
+ agent_runtime=self.options.agent_runtime,
+ reasoning_effort=self.options.reasoning_effort,
+ sandbox_timeout_seconds=(
+ self.provider.sandbox_timeout_seconds(self.options.per_case_timeout_seconds)
+ if self.provider is not None
+ else None
+ ),
+ )
+
if Infra.MCP_SERVER in required:
reporter.setup_started("build skills and start MCP server")
assert self._live_server is not None
@@ -361,21 +378,6 @@ class SandboxedEvalHarness:
if self._posthog_client is not None:
self._stack.callback(self._posthog_client.shutdown)
- if Infra.DEMO_DATA in required:
- reporter.setup_started("prepare demo data")
- assert self._database is not None
- self._demo_data = ensure_demo_ready(
- blocker=self._database.blocker,
- agent_model=self.options.agent_model,
- agent_runtime=self.options.agent_runtime,
- reasoning_effort=self.options.reasoning_effort,
- sandbox_timeout_seconds=(
- self.provider.sandbox_timeout_seconds(self.options.per_case_timeout_seconds)
- if self.provider is not None
- else None
- ),
- )
-
if Infra.AI_GATEWAY in required:
assert self._demo_data is not None and self._database is not None
with self._database.blocker.unblock():
diff --git a/products/posthog_ai/eval_harness/harness/requirements.py b/products/posthog_ai/eval_harness/harness/requirements.py
index 543c8da6c348..07dac8cd0fcc 100644
--- a/products/posthog_ai/eval_harness/harness/requirements.py
+++ b/products/posthog_ai/eval_harness/harness/requirements.py
@@ -49,7 +49,8 @@ _IMPLIES: dict[Infra, frozenset[Infra]] = {
Infra.LIVE_SERVER: frozenset({Infra.DATABASE}),
# The gateway's mint credential belongs to the master demo team.
Infra.AI_GATEWAY: frozenset({Infra.DEMO_DATA}),
- Infra.MCP_SERVER: frozenset({Infra.LIVE_SERVER}),
+ # The MCP step also builds the skills, whose HogQL examples render against the seeded master team.
+ Infra.MCP_SERVER: frozenset({Infra.LIVE_SERVER, Infra.DEMO_DATA}),
# Demo seeding's person/group reads go through the personhog router; a dead
# router would poison its 30s negative group-types cache for the whole run.
Infra.DEMO_DATA: frozenset({Infra.DATABASE, Infra.PERSONHOG}),
diff --git a/products/posthog_ai/eval_harness/harness/services.py b/products/posthog_ai/eval_harness/harness/services.py
index 50cea1bf1264..ecfd72ad0746 100644
--- a/products/posthog_ai/eval_harness/harness/services.py
+++ b/products/posthog_ai/eval_harness/harness/services.py
@@ -494,9 +494,13 @@ def build_local_skills(*, set_bind_mount_env: bool) -> Path:
template helpers (e.g. HogQL example rendering) guard on that setting. The
harness sets ``DEBUG=1`` in the environment before ``django.setup()``, so
that ordering holds by the time this runs.
+
+ A failed build raises instead of reusing an older ``dist/skills``: the cache
+ would pin the new source hash onto the old output, and every later run would
+ silently give the agent skills from another branch or edit.
"""
cache = LocalSkillsCache()
- dist_dir = cache.ensure_built()
+ dist_dir = cache.ensure_built(allow_stale=False)
if set_bind_mount_env:
# Only ``DockerSandbox.create`` reads this to bind-mount the built skills.
# Under the modal provider the locally built skills are instead baked into
diff --git a/products/posthog_ai/eval_harness/test/test_suite_requirements.py b/products/posthog_ai/eval_harness/test/test_suite_requirements.py
index e9fd0aea6c15..008f9a7dd236 100644
--- a/products/posthog_ai/eval_harness/test/test_suite_requirements.py
+++ b/products/posthog_ai/eval_harness/test/test_suite_requirements.py
@@ -26,8 +26,8 @@ ALL_ENV_VARS = (
pytest.param({Infra.DATABASE}, {Infra.DATABASE}, id="database_implies_nothing"),
pytest.param(
{Infra.MCP_SERVER},
- {Infra.MCP_SERVER, Infra.LIVE_SERVER, Infra.DATABASE},
- id="mcp_server_pulls_live_server_and_database",
+ {Infra.MCP_SERVER, Infra.LIVE_SERVER, Infra.DATABASE, Infra.DEMO_DATA, Infra.PERSONHOG},
+ id="mcp_server_pulls_live_server_and_demo_data",
),
pytest.param(
{Infra.DEMO_DATA},