Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(posthog-ai): regenerate the eval master demo team after a day (#127435)

Daniel Parkera1b2e5f90f6a
diff --git a/products/posthog_ai/eval_harness/data_setup.py b/products/posthog_ai/eval_harness/data_setup.py
index a11b1e267243..b709647249b1 100644
--- a/products/posthog_ai/eval_harness/data_setup.py
+++ b/products/posthog_ai/eval_harness/data_setup.py
@@ -10,6 +10,7 @@ from typing import TYPE_CHECKING
from django.conf import settings
from django.db import transaction
from django.test import override_settings
+from django.utils import timezone
from posthog.clickhouse.client import sync_execute
from posthog.models import Organization, OrganizationMembership, Team, User
@@ -32,6 +33,9 @@ EVAL_USER_FULL_NAME = "Karen Smith"
MASTER_ORG_NAME = "Hedgebox Master Seed"
MASTER_USER_EMAIL = "[email protected]"
+# The simulation ends its event history at generation time, so an older master leaves cases about
+# "yesterday" or "the last 7 days" with no events.
+MASTER_DEMO_MAX_AGE = datetime.timedelta(days=1)
def _build_hedgebox_matrix() -> HedgeboxMatrix:
@@ -123,8 +127,8 @@ def create_demo_org_team_user(django_db_blocker) -> tuple[Organization, Team, Us
def ensure_master_demo_team(django_db_blocker) -> int:
"""Ensure a master Hedgebox demo team exists with events in both PSQL and CH.
- Reuses the master team across sessions when it has CH events. Rebuilds from
- scratch when either side is empty. Returns the master team id (autoincrement,
+ Reuses the master team across sessions when it has CH events and is less than
+ ``MASTER_DEMO_MAX_AGE`` old. Rebuilds from scratch when it is older or either side is empty. Returns the master team id (autoincrement,
not ``MatrixManager.MASTER_TEAM_ID`` — we own this team's lifecycle).
"""
@@ -138,7 +142,14 @@ def ensure_master_demo_team(django_db_blocker) -> int:
"SELECT count() FROM events WHERE team_id = %(team_id)s",
{"team_id": team.id},
)[0][0]
- if ch_event_count > 0:
+ if team.created_at < timezone.now() - MASTER_DEMO_MAX_AGE:
+ logger.warning(
+ "Master demo team id=%d was generated at %s, more than %s ago; regenerating",
+ team.id,
+ team.created_at.isoformat(),
+ MASTER_DEMO_MAX_AGE,
+ )
+ elif ch_event_count > 0:
if not settings.OBJECT_STORAGE_ENABLED or _master_warehouse_ready(team):
logger.info("Reusing master demo team id=%d (events=%d)", team.id, ch_event_count)
return team.id
diff --git a/products/posthog_ai/evals/AGENTS.md b/products/posthog_ai/evals/AGENTS.md
index 66e28fa9df3e..068e4967280b 100644
--- a/products/posthog_ai/evals/AGENTS.md
+++ b/products/posthog_ai/evals/AGENTS.md
@@ -36,7 +36,7 @@ HedgeboxMatrix(
```
- The seed is fixed, so the dataset is **byte-for-byte reproducible** — assertions on relative shapes (e.g. "signups trend over -8w") are stable, but **do not hard-code absolute counts**; they can drift if the simulation code changes.
-- **Sandboxed evals** (`SandboxedDemoData` in `harness/demo_data.py`): a master Hedgebox team is generated once via `ensure_master_demo_team`, then each eval case gets its own org/team via `copy_demo_data_to_new_team` (ClickHouse `INSERT ... SELECT` copy + `set_project_up` re-run + taxonomy re-inference). When object storage is enabled, the master must also have all five warehouse CSVs and metadata rows. Case teams share those immutable master CSVs but receive isolated credentials, table metadata, annotations, and the persons join. Each case is isolated; the seeded user is **"Karen Smith"**.
+- **Sandboxed evals** (`SandboxedDemoData` in `harness/demo_data.py`): a master Hedgebox team is generated via `ensure_master_demo_team` and reused for up to a day, because its event history ends at generation time; then each eval case gets its own org/team via `copy_demo_data_to_new_team` (ClickHouse `INSERT ... SELECT` copy + `set_project_up` re-run + taxonomy re-inference). When object storage is enabled, the master must also have all five warehouse CSVs and metadata rows. Case teams share those immutable master CSVs but receive isolated credentials, table metadata, annotations, and the persons join. Each case is isolated; the seeded user is **"Karen Smith"**.
- Events span both past (`days_past`) and future (`days_future` — `paid_bill` events are scheduled forward). When choosing date ranges in expected queries, prefer relative ranges like `-30d`, `-8w`, `-6m`.
- Most insights/dashboards are built with `filterTestAccounts=True`. The team has `test_account_filters` configured, and a "Signed-up users" cohort exists.