:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(aio): report remaining offline eval suites to posthog (#120545)
Radu Raicea676169268523
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.mdindex 3bfc4f287e40..068a31b1c7fa 100644--- a/docs/internal/ai-offline-evaluation-reporting.md+++ b/docs/internal/ai-offline-evaluation-reporting.md@@ -1,7 +1,7 @@# Offline evaluation result reportingThe evaluation harness in `products/posthog_ai/eval_harness/` reports public-suite scorer results to Braintrust.-Public suites under `products/posthog_ai/evals/` and `products/slack_app/evals/` also publish those results to PostHog's offline experiment API when configured.+Suites under `products/posthog_ai/evals/` and the product-owned `products/*/evals/` trees also publish those results to PostHog's offline experiment API when a publishing credential is configured.The Signals pytest suites reuse the same publisher for their completed results.Agent and judge trace capture remains available.Reporting does not run the agent or scorers again.@@ -15,6 +15,12 @@ Use `hogli evals` to run PostHog AI 2.0 suites. See the [eval harness guide](../Set `send_to_posthog=True` alongside `offline_suite` to publish results to PostHog while keeping Braintrust uploads disabled.Set `send_to_posthog=False` to disable PostHog result uploads independently. The same options apply to one-shot suites.+Suites that disable Braintrust uploads declare `NO_SEND_LOGS = True` at module level and pass `no_send_logs=NO_SEND_LOGS` to their runner.+Discovery uses this declaration to validate credentials before starting infrastructure; the enrollment test checks that it matches the runner.+`BRAINTRUST_API_KEY` is required for reporting only when at least one selected suite enables Braintrust uploads.+PostHog-only suites still use the Braintrust library locally, without authenticating to its experiment service.+Scorer model credentials remain separate: using Braintrust's model proxy still needs its credential, while other providers use their own keys.+Offline API uploads run through the Python SDK independently of `TEST` and `OPT_OUT_CAPTURE`.The shared trace client retains its existing `TEST` and `OPT_OUT_CAPTURE` guards for `$ai_generation`, `$ai_span`, and `$ai_trace` events.@@ -41,14 +47,16 @@ A scorer failure marks the case as an error. Statistics still include scores fro## Configured offline experiment suites-Configured public PostHog AI and Slack app suites opt into the offline experiment publisher, including both sandboxed and one-shot suites.-The 11 experiment-skill suites retain `no_send_logs=True` and explicitly set `send_to_posthog=True`, publishing results only to PostHog.+Shared-harness suites are enrolled in the offline experiment publisher, including both sandboxed and one-shot suites, except for `logs/eval_agent_observability::eval_agent_observability`, which currently reports only to Braintrust.+The experiment-skill, Feature Flags, Dashboards, turn-suggestion and Business Knowledge retrieval suites retain `no_send_logs=True` and explicitly set `send_to_posthog=True`, publishing results only to PostHog.+Public suites retain their Braintrust uploads during enrollment. Braintrust also remains the execution engine for every shared-harness suite, including those whose results publish only to PostHog.After scoring, the harness prepares the completed results in memory and queues delivery through the Python SDK's `AsyncOfflineEvaluations` client. It does not execute another agent run or judge call.Without an explicit `send_to_posthog` override, `no_send_logs=True` suppresses this upload too.-The [PostHog AI catalog](../../products/posthog_ai/evals/offline_config.py) and [Slack app catalog](../../products/slack_app/evals/offline_config.py) pin destination project `2`, scorer kinds, immutable version UUIDs, and component tags.+The [PostHog AI catalog](../../products/posthog_ai/evals/offline_config.py) and each product's `evals/offline_config.py` pin destination project `2`, scorer kinds, immutable version UUIDs, and component tags.Suites select their own metrics from the catalog; sandboxed suites also include the harness-provided `exit_code_zero` metric.The SQL suite retains its seven existing scorer versions.+The discovery-wide enrollment test checks every enrolled suite's publishing flag, stable key, scorer coverage, pinned UUIDs and ownership tags without executing agents or judges. It also checks that the Logs agent-observability suite retains its Braintrust-only reporting configuration.Provide the API key through the environment before running the suite:| Variable | Purpose |@@ -97,6 +105,34 @@ Invalid configuration produces a warning and skips this destination; BraintrustAn upload failure also warns without changing the evaluation's score gate.The final summary shows a PostHog results link only after a successful offline upload; otherwise it shows the upload status.+### Testing uploads against local PostHog++Set `POSTHOG_OFFLINE_EVAL_HOST=http://localhost:8010` and use a personal API key created in that local instance.+HTTP is accepted only for `localhost`, `127.0.0.1`, and `::1`; other hosts require HTTPS.+The eval harness's temporary test database is separate from the development instance receiving uploads.++The suite catalogs contain production project and scorer-version IDs. To use local scorers without editing those catalogs, set+`POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG=.env.local.offline-evals.json` in `.env.local`. The JSON file selects a local project and maps+each catalog scorer-version UUID to a version created in that local project:++```json+{+ "project_id": 2,+ "scorer_versions": {+ "00000000-0000-4000-8000-000000000002": "00000000-0000-4000-8000-000000000003"+ }+}+```++The UUIDs above are placeholders. Create the needed scorers under **Evaluations → Scorers**, matching each catalog metric's kind,+and use their immutable version IDs. Include every scorer used by the selected suites; a single file can cover multiple suites.+The file is gitignored by the `.env*` rule. Local configuration is accepted only with a loopback upload host.+An unreadable, invalid, or incomplete file disables the upload with an error; it never falls back to the production catalog.++Run through `hogli evals` to load `.env.local`; existing shell variables take precedence.+For PostHog-only testing, select suites declaring `NO_SEND_LOGS = True`, such as Feature Flags or Dashboards.+Set `OPT_OUT_CAPTURE=1` to suppress ordinary SDK/trace capture while retaining explicit offline-result uploads.+### Scorer and experiment tagsScore definitions and experiment runs use the same team and component tags:@@ -105,6 +141,12 @@ Score definitions and experiment runs use the same team and component tags:| -------------------- | ------------------------------------------------------------------------------------------- || PostHog AI | `team:surfaces`, `component:posthog-ai/<domain>` (for example `sql` or `product-analytics`) || Slack app | `team:surfaces`, `component:slack-app/<area>` (for example `followup-classifier`) |+| Conversations | `team:conversations`, `component:conversations` |+| Business Knowledge | `team:conversations`, `component:business-knowledge` |+| Data Catalog | `team:data-modeling`, `component:data-catalog` |+| Notebooks | `team:data-tools`, `component:notebooks` |+| Feature Flags | `team:feature-flags`, `component:feature-flags` |+| Dashboards | `team:product-analytics`, `component:dashboards` || Signals grouping | `team:self-driving`, `component:signals/grouping` || Signals scout safety | `team:self-driving`, `component:signals/scout-safety` |@@ -114,6 +156,34 @@ To upload **and tag** experiments, `POSTHOG_OFFLINE_EVAL_API_KEY` must be a persEach prepared experiment's tag update is attempted once per publisher, including when delivery pauses and resumes. A failed or interrupted replacement is not repeated automatically, because it may already have succeeded or someone may have edited the tags afterward. Inspect the experiment's tags before manually retrying a replacement.+### Additional one-shot suites++Run these through the same harness and publishing lifecycle:++```bash+hogli evals turn_suggestions/eval_turn_suggestions::eval_turn_suggestions+hogli evals business_knowledge/eval_retrieval::eval_retrieval+hogli evals conversations/eval_support_reply::eval_support_reply+```++Turn suggestions reuse the labeled benchmark cases and the configured System One judge, once per case.+`TURN_SUGGESTIONS_EVAL_THRESHOLD` overrides the production show threshold for the run; it must be between 0 and 1.+Case metadata records the threshold and requested judge model, and outputs retain the selected offer, probability, outcome and duration.+An accepted optional offer counts as correct; an unavailable judgment reports an error.+The existing `turn_suggestions_benchmark` command remains available for multi-endpoint comparisons and threshold sweeps; those modes are not part of this suite.++Business Knowledge retrieval reuses the labeled `retrieval_eval.yaml` corpus and queries.+Each case seeds an isolated team, searches the corpus and cleans up the team.+Its boolean score checks the existing found/not-found expectation, including queries expected to return no results.+It does not score source relevance or replace the ordinary retrieval unit tests.++Support reply retains its existing behavior: mocked by default, or live when `SUPPORT_REPLY_EVAL_LIVE=1`.+Both case metadata and output identify the mode so runs can be compared within the same mode.++These suites use the existing one-shot infrastructure and credential prerequisites described in the harness guide.+The turn-suggestion judge additionally needs its existing System One configuration; selecting the suite does not configure that service.+MCP saved-report grading, semantic-layer canary commands and ReviewHog campaigns retain their separate workflows and are not enrolled by these wrappers.+### CI configurationCI jobs running enrolled suites need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment.@@ -160,7 +230,7 @@ Existing local logs, enabled case exports, and Braintrust results remain availabBoth `SandboxedEval` and `OneShotEval` accept `offline_suite=OfflineEvalSuite(...)`.Each suite declares its destination `project_id`, selects at least one metric in `scorer_kinds`, and pins the corresponding immutable version UUIDs in `scorer_versions`.These fields belong to `OfflineEvalSuite` in code. Scorer version UUIDs must be distinct for the suite's enrolled metrics.-The AI and Slack catalog helpers add `exit_code_zero` for sandboxed suites. One-shot suites pass `suite_kind=SUITE_KIND` to enroll only their declared metrics.+Catalog helpers add `exit_code_zero` for sandboxed suites. One-shot suites pass `suite_kind=SUITE_KIND` to enroll only their declared metrics.Numeric scores can use the domain of their configured scorer version. The API validates bounds and steps; the harness does not restrict every numeric metric to `[0, 1]`.`OfflineEvalSuite.experiment_options` accepts the SDK's `ExperimentOptions` for external dataset identifiers, `prompt_version`, model/application versions, and `run_source`.diff --git a/products/business_knowledge/evals/eval_retrieval.py b/products/business_knowledge/evals/eval_retrieval.pynew file mode 100644index 000000000000..7ead451fa2d4--- /dev/null+++ b/products/business_knowledge/evals/eval_retrieval.py@@ -0,0 +1,101 @@+from __future__ import annotations++import asyncio+from pathlib import Path+from typing import TYPE_CHECKING, TypedDict++import yaml++from products.business_knowledge.backend.logic import create_text_source, search_knowledge+from products.business_knowledge.backend.models import KnowledgeDocument, SafetyVerdict+from products.business_knowledge.evals.offline_config import offline_suite+from products.conversations.evals.seeders import provision_eval_team, teardown_eval_team+from products.posthog_ai.eval_harness.config import BaseEvalCase+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.one_shot import OneShotPrivateEval+from products.posthog_ai.eval_harness.scorers.contract import Score, Scorer++if TYPE_CHECKING:+ from products.posthog_ai.eval_harness.harness.context import EvalContext++NO_SEND_LOGS = True++SUITE_KIND = SuiteKind.ONE_SHOT+++class RetrievalQuery(TypedDict):+ query: str+ expect_found: bool+++class RetrievalSource(TypedDict):+ source_name: str+ text: str+ queries: list[RetrievalQuery]+++CORPUS: list[RetrievalSource] = yaml.safe_load(+ (Path(__file__).parents[1] / "backend/tests/fixtures/retrieval_eval.yaml").read_text()+)+CASES = [+ BaseEvalCase(+ name=f"{entry['source_name']}|{query['query']}",+ prompt=query["query"],+ expected={"retrieval_match": {"found": query["expect_found"]}},+ metadata={"source_name": entry["source_name"]},+ )+ for entry in CORPUS+ for query in entry["queries"]+]+++class RetrievalMatch(Scorer):+ def _name(self) -> str:+ return "retrieval_match"++ def _run_eval_sync(+ self, output: dict[str, object] | None, expected: dict[str, object] | None = None, **kwargs: object+ ) -> Score:+ expectation = (expected or {}).get(self._name())+ if output is None or not isinstance(expectation, dict):+ return Score(name=self._name(), score=None, metadata={"reason": "Retrieval did not complete"})+ return Score(+ name=self._name(),+ score=float(output["found"] == expectation["found"]),+ metadata={"source_names": output["source_names"]},+ )+++def search_case(case: BaseEvalCase) -> dict[str, object]:+ eval_team = provision_eval_team(label="knowledge-retrieval")+ try:+ for entry in CORPUS:+ create_text_source(+ team_id=eval_team.team.id,+ created_by_id=eval_team.user.id,+ name=entry["source_name"],+ text=entry["text"],+ )+ KnowledgeDocument.objects.for_team(eval_team.team.id).update(safety_verdict=SafetyVerdict.SAFE)+ results = search_knowledge(eval_team.team.id, case.prompt)+ return {"found": bool(results), "source_names": sorted({result.source_name for result in results})}+ finally:+ teardown_eval_team(eval_team=eval_team)+++async def eval_retrieval(ctx: EvalContext) -> None:+ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict[str, object]:+ return await asyncio.to_thread(search_case, case)++ await OneShotPrivateEval(+ no_send_logs=NO_SEND_LOGS,+ experiment_name="business-knowledge-retrieval",+ cases=CASES,+ task=task,+ scorers=[RetrievalMatch()],+ offline_suite=offline_suite(+ "business_knowledge/eval_retrieval::eval_retrieval", metrics=["retrieval_match"], suite_kind=SUITE_KIND+ ),+ send_to_posthog=True,+ ctx=ctx,+ )diff --git a/products/business_knowledge/evals/offline_config.py b/products/business_knowledge/evals/offline_config.pynew file mode 100644index 000000000000..39882e35c8bf--- /dev/null+++ b/products/business_knowledge/evals/offline_config.py@@ -0,0 +1,19 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, OfflineScorer] = {+ "retrieval_match": OfflineScorer(kind="boolean", version=UUID("01a15a29-f71a-7bdd-9a8c-c018881f9702")),+}+++def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},+ tags=("team:conversations", "component:business-knowledge"),+ )diff --git a/products/conversations/evals/eval_support_reply.py b/products/conversations/evals/eval_support_reply.pyindex 316ae6eb808b..f615787ce198 100644--- a/products/conversations/evals/eval_support_reply.py+++ b/products/conversations/evals/eval_support_reply.py@@ -20,6 +20,7 @@ import asynciofrom products.conversations.evals.constants import LIVE_EVAL_ENV_VARfrom products.conversations.evals.fixtures import FIXTURES, FIXTURES_BY_NAME, expected_for+from products.conversations.evals.offline_config import offline_suitefrom products.conversations.evals.runner import live_eval_enabled, run_fixturefrom products.conversations.evals.scorers import ALL_SCORERSfrom products.conversations.evals.seeders import provision_eval_team, seed_case, teardown_eval_team@@ -48,9 +49,10 @@ CASES = [async def eval_support_reply(ctx: EvalContext) -> None:+ live = live_eval_enabled()+async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict:fixture = FIXTURES_BY_NAME[case.name]- live = live_eval_enabled()eval_team = await asyncio.to_thread(lambda: provision_eval_team(label=fixture.name))try:seed = await asyncio.to_thread(lambda: seed_case(eval_team=eval_team, fixture=fixture))@@ -60,7 +62,22 @@ async def eval_support_reply(ctx: EvalContext) -> None:await OneShotPublicEval(experiment_name="support-reply-pipeline",- cases=CASES,+ offline_suite=offline_suite(+ "conversations/eval_support_reply::eval_support_reply",+ metrics=[+ "outcome_match",+ "citation_precision",+ "forbidden_claims",+ "cost",+ "grounding",+ "clarifying_question",+ ],+ suite_kind=SUITE_KIND,+ ),+ cases=[+ case.model_copy(update={"metadata": {**case.metadata, "mode": "live" if live else "mocked"}})+ for case in CASES+ ],scorers=list(ALL_SCORERS),task=task,ctx=ctx,diff --git a/products/conversations/evals/offline_config.py b/products/conversations/evals/offline_config.pynew file mode 100644index 000000000000..09638c9bcd7d--- /dev/null+++ b/products/conversations/evals/offline_config.py@@ -0,0 +1,24 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, OfflineScorer] = {+ "citation_precision": OfflineScorer(kind="numeric", version=UUID("01a15a27-9528-7df7-89dc-717fdb43d339")),+ "clarifying_question": OfflineScorer(kind="boolean", version=UUID("01a15a27-ba72-7483-8369-8a46861c680f")),+ "cost": OfflineScorer(kind="boolean", version=UUID("01a15a27-a827-7494-a247-daf97f55b7fd")),+ "forbidden_claims": OfflineScorer(kind="boolean", version=UUID("01a15a27-9d10-7260-ace1-335661c5bfdc")),+ "grounding": OfflineScorer(kind="boolean", version=UUID("01a15a27-b117-7e00-aefd-30eb5f5b7e4a")),+ "outcome_match": OfflineScorer(kind="boolean", version=UUID("01a15a27-8bab-7c0e-8a33-d9ac6327874e")),+}+++def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},+ tags=("team:conversations", "component:conversations"),+ )diff --git a/products/dashboards/evals/eval_dashboard_contents.py b/products/dashboards/evals/eval_dashboard_contents.pyindex 954bd4bd45bb..2c3d34e87c5d 100644--- a/products/dashboards/evals/eval_dashboard_contents.py+++ b/products/dashboards/evals/eval_dashboard_contents.py@@ -6,6 +6,7 @@ from posthog.utils import generate_short_idfrom products.dashboards.backend.models.dashboard import Dashboardfrom products.dashboards.backend.models.dashboard_tile import DashboardTile+from products.dashboards.evals.offline_config import offline_suitefrom products.dashboards.evals.scorers import SavedDashboardContentsfrom products.posthog_ai.eval_harness.base import SandboxedPrivateEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCase@@ -14,6 +15,8 @@ from products.product_analytics.backend.facade.api import get_or_create_saved_infrom products.product_analytics.backend.facade.models import Insightfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext+NO_SEND_LOGS = True+DASHBOARD_NAME = "File activity overview"SIGNUPS_NAME = "Saved daily signups"UPLOADS_NAME = "Saved daily file uploads"@@ -67,7 +70,13 @@ def seed_existing_dashboard(context: CustomPromptSandboxContext) -> dict[str, Anasync def eval_dashboard_contents(ctx: EvalContext) -> None:await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-dashboards-saved-contents-cli",+ offline_suite=offline_suite(+ "dashboards/eval_dashboard_contents::eval_dashboard_contents",+ metrics=["saved_dashboard_contents"],+ ),+ send_to_posthog=True,cases=[SandboxedEvalCase(name="create_from_saved_insights",diff --git a/products/dashboards/evals/eval_dashboard_sql_date_range.py b/products/dashboards/evals/eval_dashboard_sql_date_range.pyindex bfc961ebb346..0e7d661c87b2 100644--- a/products/dashboards/evals/eval_dashboard_sql_date_range.py+++ b/products/dashboards/evals/eval_dashboard_sql_date_range.py@@ -3,12 +3,15 @@ from __future__ import annotationsfrom typing import Anyfrom products.dashboards.backend.models.dashboard import Dashboard+from products.dashboards.evals.offline_config import offline_suitefrom products.dashboards.evals.scorers import SQLTilesFollowDashboardDateRangefrom products.posthog_ai.eval_harness.base import SandboxedPrivateEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext+NO_SEND_LOGS = True+SIGNUPS_DASHBOARD_NAME = "Signups and uploads (SQL)"BILLING_DASHBOARD_NAME = "Billing and downloads (SQL)"@@ -22,7 +25,13 @@ def seed_team(context: CustomPromptSandboxContext) -> dict[str, Any]:async def eval_dashboard_sql_date_range(ctx: EvalContext) -> None:await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-dashboards-sql-date-range-cli",+ offline_suite=offline_suite(+ "dashboards/eval_dashboard_sql_date_range::eval_dashboard_sql_date_range",+ metrics=["sql_tiles_follow_dashboard_date_range"],+ ),+ send_to_posthog=True,cases=[SandboxedEvalCase(name="sql_tiles_without_date_hint",diff --git a/products/dashboards/evals/offline_config.py b/products/dashboards/evals/offline_config.pynew file mode 100644index 000000000000..a6b132274628--- /dev/null+++ b/products/dashboards/evals/offline_config.py@@ -0,0 +1,23 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, OfflineScorer] = {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15a27-ce66-7f1a-babd-0e1423882e9b")),+ "saved_dashboard_contents": OfflineScorer(kind="boolean", version=UUID("01a15a27-c451-7557-9bca-c968f6358a91")),+ "sql_tiles_follow_dashboard_date_range": OfflineScorer(+ kind="numeric", version=UUID("01a15a27-d794-74b3-8822-063a74a674fd")+ ),+}+++def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},+ tags=("team:product-analytics", "component:dashboards"),+ )diff --git a/products/data_catalog/evals/eval_governed_metrics.py b/products/data_catalog/evals/eval_governed_metrics.pyindex 7f8e53442430..64d69540c222 100644--- a/products/data_catalog/evals/eval_governed_metrics.py+++ b/products/data_catalog/evals/eval_governed_metrics.py@@ -69,6 +69,7 @@ from products.data_catalog.evals.constants import (WEBSITE_404_HITS_DAILY_METRIC_NAME,YOY_MRR_GROWTH_METRIC_NAME,)+from products.data_catalog.evals.offline_config import offline_suitefrom products.data_catalog.evals.scorers import (CanonicalMetricRun,GovernedBehaviorCorrectness,@@ -691,6 +692,20 @@ async def eval_governed_metrics(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-governed-metrics-cli",+ offline_suite=offline_suite(+ "data_catalog/eval_governed_metrics::eval_governed_metrics",+ metrics=[+ "metrics_catalog_queried",+ "metrics_catalog_before_answer",+ "metrics_catalog_before_data_discovery",+ "canonical_metric_run",+ "metrics_catalog_not_queried",+ "governed_behavior_correctness",+ "trust_badge",+ "proposed_badge",+ "no_metric_write",+ ],+ ),cases=cases,scorers=[MetricsCatalogQueried(),diff --git a/products/data_catalog/evals/eval_metric_authoring.py b/products/data_catalog/evals/eval_metric_authoring.pyindex 83d27f4e434e..31692facb282 100644--- a/products/data_catalog/evals/eval_metric_authoring.py+++ b/products/data_catalog/evals/eval_metric_authoring.py@@ -16,6 +16,7 @@ To run a single case::from __future__ import annotationsfrom products.data_catalog.evals.constants import METRIC_CREATE_TOOL+from products.data_catalog.evals.offline_config import offline_suitefrom products.data_catalog.evals.scorers import MetricDescriptionConcise, MetricDescriptionQualityfrom products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCase@@ -54,6 +55,10 @@ async def eval_metric_authoring(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-metric-authoring-cli",+ offline_suite=offline_suite(+ "data_catalog/eval_metric_authoring::eval_metric_authoring",+ metrics=["metric_create_called", "metric_description_concise", "metric_description_quality"],+ ),cases=cases,scorers=[RequiredToolCall([METRIC_CREATE_TOOL], name="metric_create_called"),diff --git a/products/data_catalog/evals/eval_semantic_discovery.py b/products/data_catalog/evals/eval_semantic_discovery.pyindex bf5b0d6d273b..65ef00f6e6ae 100644--- a/products/data_catalog/evals/eval_semantic_discovery.py+++ b/products/data_catalog/evals/eval_semantic_discovery.py@@ -21,6 +21,7 @@ from products.data_catalog.evals.constants import (RELATIONSHIP_SOURCE_NAME,RELATIONSHIP_TARGET_KEY,)+from products.data_catalog.evals.offline_config import offline_suitefrom products.data_catalog.evals.scorers import (DeprecationProposed,SemanticMetadataQueried,@@ -118,6 +119,10 @@ async def eval_semantic_discovery(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-semantic-discovery-cli",+ offline_suite=offline_suite(+ "data_catalog/eval_semantic_discovery::eval_semantic_discovery",+ metrics=["semantic_metadata_queried", "semantic_trust_decision_correctness", "deprecation_proposed"],+ ),cases=cases,scorers=[SemanticMetadataQueried(), SemanticTrustDecisionCorrectness(), DeprecationProposed()],ctx=ctx,diff --git a/products/data_catalog/evals/offline_config.py b/products/data_catalog/evals/offline_config.pynew file mode 100644index 000000000000..901d03618bb8--- /dev/null+++ b/products/data_catalog/evals/offline_config.py@@ -0,0 +1,42 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, OfflineScorer] = {+ "canonical_metric_run": OfflineScorer(kind="boolean", version=UUID("01a15a27-fa96-71e0-b904-13f14079440e")),+ "deprecation_proposed": OfflineScorer(kind="boolean", version=UUID("01a15a28-7a51-7732-838a-34de17d02bbf")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15a28-3e1d-7d93-86bb-de7f4e4a5c63")),+ "governed_behavior_correctness": OfflineScorer(+ kind="numeric", version=UUID("01a15a28-0f80-7d6e-b0bb-0573fcb583fa")+ ),+ "metric_create_called": OfflineScorer(kind="boolean", version=UUID("01a15a28-47d2-7eea-ba8a-55266556d782")),+ "metric_description_concise": OfflineScorer(kind="boolean", version=UUID("01a15a28-5017-75a6-b5e5-5ddfe270c02a")),+ "metric_description_quality": OfflineScorer(kind="boolean", version=UUID("01a15a28-5aa6-746d-8d30-bc84b857f15a")),+ "metrics_catalog_before_answer": OfflineScorer(+ kind="boolean", version=UUID("01a15a27-e92c-792a-ab48-24162ed407e9")+ ),+ "metrics_catalog_before_data_discovery": OfflineScorer(+ kind="boolean", version=UUID("01a15a27-f18f-76e9-a67e-0259133b733c")+ ),+ "metrics_catalog_not_queried": OfflineScorer(kind="boolean", version=UUID("01a15a28-0586-7594-9cd5-3f1ff8ca3c06")),+ "metrics_catalog_queried": OfflineScorer(kind="boolean", version=UUID("01a15a27-e0c4-74f7-ae0c-acda47b99ab3")),+ "no_metric_write": OfflineScorer(kind="boolean", version=UUID("01a15a28-30cb-7118-b1c8-9084eba7378e")),+ "proposed_badge": OfflineScorer(kind="boolean", version=UUID("01a15a28-206d-79ea-b231-4bf05e131c80")),+ "semantic_metadata_queried": OfflineScorer(kind="boolean", version=UUID("01a15a28-64d0-787c-af9e-94c4730b8a1c")),+ "semantic_trust_decision_correctness": OfflineScorer(+ kind="numeric", version=UUID("01a15a28-6ded-74ed-a08c-4bfc378b0b5b")+ ),+ "trust_badge": OfflineScorer(kind="boolean", version=UUID("01a15a28-17ed-7638-82e4-d39bd349ce9e")),+}+++def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},+ tags=("team:data-modeling", "component:data-catalog"),+ )diff --git a/products/feature_flags/evals/eval_cleanup_stale_flags.py b/products/feature_flags/evals/eval_cleanup_stale_flags.pyindex 567ac5c15e55..19e7c8ae7cc1 100644--- a/products/feature_flags/evals/eval_cleanup_stale_flags.py+++ b/products/feature_flags/evals/eval_cleanup_stale_flags.py@@ -60,6 +60,7 @@ To run:from __future__ import annotations+from products.feature_flags.evals.offline_config import offline_suitefrom products.feature_flags.evals.scorers import (ASSESSMENT_ONLY_NO_EDIT_CLAIM_QUESTION,DEPENDENTS_READ_TOOLS,@@ -88,6 +89,8 @@ from products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCallfrom products.posthog_ai.evals.retrieval.scorers import SkillTriggered+NO_SEND_LOGS = True+SKILL_NAME = "cleaning-up-stale-feature-flags"TRIGGER_SCORER_NAME = "cleanup_skill_triggered"EDIT_SCORER_NAME = "code_edit_direction"@@ -226,7 +229,24 @@ async def eval_cleanup_stale_flags(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-feature-flags-cleanup-stale-cli",+ offline_suite=offline_suite(+ "feature_flags/eval_cleanup_stale_flags::eval_cleanup_stale_flags",+ metrics=[+ "cleanup_skill_triggered",+ "no_flag_mutation",+ "flag_state_unchanged",+ "code_edit_direction",+ "flag_lookup_direction",+ "dependents_check_direction",+ "schedule_check_direction",+ "recency_refusal_without_override",+ "tour_unknown_waits",+ "assessment_only_no_edit_claim",+ ],+ ),+ send_to_posthog=True,cases=cases,scorers=[SkillTriggered(SKILL_NAME, name=TRIGGER_SCORER_NAME),diff --git a/products/feature_flags/evals/eval_flag_answers.py b/products/feature_flags/evals/eval_flag_answers.pyindex 4d958c995a12..f0ad24f6518d 100644--- a/products/feature_flags/evals/eval_flag_answers.py+++ b/products/feature_flags/evals/eval_flag_answers.py@@ -23,6 +23,7 @@ from __future__ import annotationsimport jsonfrom pathlib import Path+from products.feature_flags.evals.offline_config import offline_suitefrom products.feature_flags.evals.scorers import (EXPLAINED_TAG_REQUIREMENT_QUESTION,REFUSED_WITHOUT_BLAMING_QUESTION,@@ -47,6 +48,8 @@ from products.posthog_ai.eval_harness.base import SandboxedPrivateEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContext+NO_SEND_LOGS = True+_TOOL_DEFINITIONS = Path(__file__).parents[3] / "services/mcp/schema/generated-tool-definitions.json"FLAG_TOOL_CATEGORY = "Feature flags"@@ -111,7 +114,22 @@ async def eval_flag_answers(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-feature-flags-answers-cli",+ offline_suite=offline_suite(+ "feature_flags/eval_flag_answers::eval_flag_answers",+ metrics=[+ "called_expected_tool",+ "attempted_tool",+ "avoided_tool",+ "created_flag_with_tags",+ "final_message_names",+ "stale_is_not_safe_to_remove",+ "refused_without_blaming",+ "explained_tag_requirement",+ ],+ ),+ send_to_posthog=True,cases=cases,scorers=[CalledExpectedTool(),diff --git a/products/feature_flags/evals/eval_flag_creation.py b/products/feature_flags/evals/eval_flag_creation.pyindex 3c00dc70659e..cd0095cf6d81 100644--- a/products/feature_flags/evals/eval_flag_creation.py+++ b/products/feature_flags/evals/eval_flag_creation.py@@ -4,11 +4,14 @@ from typing import Anyfrom products.feature_flags.backend.models.feature_flag import FeatureFlagfrom products.feature_flags.evals.creation_scorers import CreatedFlagConfiguration+from products.feature_flags.evals.offline_config import offline_suitefrom products.posthog_ai.eval_harness.base import SandboxedPrivateEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext+NO_SEND_LOGS = True+FLAG_KEY = "bulk-file-export-preview"@@ -20,7 +23,13 @@ def seed_flag_creation(context: CustomPromptSandboxContext) -> dict[str, Any]:async def eval_flag_creation(ctx: EvalContext) -> None:await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-feature-flags-creation-cli",+ offline_suite=offline_suite(+ "feature_flags/eval_flag_creation::eval_flag_creation",+ metrics=["created_flag_configuration"],+ ),+ send_to_posthog=True,cases=[SandboxedEvalCase(name="boolean_flag_with_targeted_rollout",diff --git a/products/feature_flags/evals/eval_flag_tool_selection.py b/products/feature_flags/evals/eval_flag_tool_selection.pyindex 205c60aa07e9..098cc7f985b8 100644--- a/products/feature_flags/evals/eval_flag_tool_selection.py+++ b/products/feature_flags/evals/eval_flag_tool_selection.py@@ -28,6 +28,7 @@ To run:from __future__ import annotations+from products.feature_flags.evals.offline_config import offline_suitefrom products.feature_flags.evals.scorers import (EXPLAINED_KEY_REUSE_QUESTION,AvoidedTool,@@ -58,6 +59,8 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall+NO_SEND_LOGS = True+# The rename case asks for these two values, so the prompt and the scorer read the same# constants: a prompt that changed without the expectation would grade the old request.METADATA_RENAMED_KEY = "file-preview-tiles"@@ -140,7 +143,22 @@ async def eval_flag_tool_selection(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-feature-flags-tool-selection-cli",+ offline_suite=offline_suite(+ "feature_flags/eval_flag_tool_selection::eval_flag_tool_selection",+ metrics=[+ "resolved_flag_by_key",+ "called_expected_tool",+ "avoided_tool",+ "generic_update_omits_fields",+ "generic_update_sets_fields",+ "preserved_unrelated_config",+ "updated_rollout_to",+ "explained_key_reuse",+ ],+ ),+ send_to_posthog=True,cases=cases,scorers=[# Every prompt names its flag by key, and the write tools take a numeric id.diff --git a/products/feature_flags/evals/eval_instrument_flags.py b/products/feature_flags/evals/eval_instrument_flags.pyindex af2c91c35e06..97478918dd43 100644--- a/products/feature_flags/evals/eval_instrument_flags.py+++ b/products/feature_flags/evals/eval_instrument_flags.py@@ -53,6 +53,7 @@ from typing import Anyfrom django.conf import settings+from products.feature_flags.evals.offline_config import offline_suitefrom products.feature_flags.evals.scorers import FILE_EDIT_TOOLSfrom products.posthog_ai.eval_harness.base import SandboxedPrivateEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCase@@ -61,6 +62,8 @@ from products.posthog_ai.eval_harness.scorers import RequiredToolCallfrom products.posthog_ai.evals.retrieval.scorers import SkillTriggeredfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext+NO_SEND_LOGS = True+SKILL_NAME = "instrument-feature-flags"SCORER_NAME = "instrument_flags_skill_triggered"@@ -144,7 +147,13 @@ async def eval_instrument_flags(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-feature-flags-instrument-trigger-cli",+ offline_suite=offline_suite(+ "feature_flags/eval_instrument_flags::eval_instrument_flags",+ metrics=["instrument_flags_skill_triggered", "edited_a_file"],+ ),+ send_to_posthog=True,cases=cases,scorers=[SkillTriggered(SKILL_NAME, name=SCORER_NAME),diff --git a/products/feature_flags/evals/offline_config.py b/products/feature_flags/evals/offline_config.pynew file mode 100644index 000000000000..4d1e679cdd0d--- /dev/null+++ b/products/feature_flags/evals/offline_config.py@@ -0,0 +1,52 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, OfflineScorer] = {+ "assessment_only_no_edit_claim": OfflineScorer(+ kind="boolean", version=UUID("01a15a28-f08d-773f-ab1a-88323b444706")+ ),+ "attempted_tool": OfflineScorer(kind="boolean", version=UUID("01a15a29-15e7-7a4d-bc3e-cbc2a6d7fde5")),+ "avoided_tool": OfflineScorer(kind="boolean", version=UUID("01a15a29-229f-7a8c-96ad-e2e4070f634e")),+ "called_expected_tool": OfflineScorer(kind="boolean", version=UUID("01a15a29-0a13-76c8-9fa1-620c98110555")),+ "cleanup_skill_triggered": OfflineScorer(kind="boolean", version=UUID("01a15a28-8366-71a4-9790-39b96dec786e")),+ "code_edit_direction": OfflineScorer(kind="boolean", version=UUID("01a15a28-a1c4-71a9-8d49-14fecb1fcf6e")),+ "created_flag_configuration": OfflineScorer(kind="boolean", version=UUID("01a15a29-7bfc-7ed2-9b70-3efd6f845ef2")),+ "created_flag_with_tags": OfflineScorer(kind="boolean", version=UUID("01a15a29-2de7-7a7d-b0a2-b7cf40ca00df")),+ "dependents_check_direction": OfflineScorer(kind="boolean", version=UUID("01a15a28-b815-7ce3-b591-524d66fd33b4")),+ "edited_a_file": OfflineScorer(kind="boolean", version=UUID("01a15a29-e150-7307-aaa9-e9228706f89e")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15a28-fcc8-71df-b886-752d5b82a419")),+ "explained_key_reuse": OfflineScorer(kind="boolean", version=UUID("01a15a29-c36a-7bef-95ad-b5151cf2bfb9")),+ "explained_tag_requirement": OfflineScorer(kind="boolean", version=UUID("01a15a29-70ca-7bb0-86bf-6430216b3ac7")),+ "final_message_names": OfflineScorer(kind="boolean", version=UUID("01a15a29-38b0-71c0-bfd9-419cd39d6847")),+ "flag_lookup_direction": OfflineScorer(kind="boolean", version=UUID("01a15a28-ae91-7185-9bce-80cd868d7c50")),+ "flag_state_unchanged": OfflineScorer(kind="boolean", version=UUID("01a15a28-9755-7c4a-8139-56ed6d942103")),+ "generic_update_omits_fields": OfflineScorer(kind="boolean", version=UUID("01a15a29-9766-741f-8775-fff77c2c81ff")),+ "generic_update_sets_fields": OfflineScorer(kind="boolean", version=UUID("01a15a29-a338-7a79-8b29-acd7a2aec492")),+ "instrument_flags_skill_triggered": OfflineScorer(+ kind="boolean", version=UUID("01a15a29-cf2e-74a8-a6af-a66281cd8637")+ ),+ "no_flag_mutation": OfflineScorer(kind="boolean", version=UUID("01a15a28-8cb1-7485-8766-e631517451fd")),+ "preserved_unrelated_config": OfflineScorer(kind="boolean", version=UUID("01a15a29-af51-7ae8-94e3-115000b1e2bb")),+ "recency_refusal_without_override": OfflineScorer(+ kind="boolean", version=UUID("01a15a28-d41d-7793-818b-ca478ebf5ba3")+ ),+ "refused_without_blaming": OfflineScorer(kind="boolean", version=UUID("01a15a29-583d-7c75-80d5-cff77fd3047b")),+ "resolved_flag_by_key": OfflineScorer(kind="boolean", version=UUID("01a15a29-8600-7983-b104-6c30aa8ab6bd")),+ "schedule_check_direction": OfflineScorer(kind="boolean", version=UUID("01a15a28-c70e-7c31-a7ff-2ad25d23d410")),+ "stale_is_not_safe_to_remove": OfflineScorer(kind="boolean", version=UUID("01a15a29-47b0-7e3d-b7f4-d1e6775c9ff2")),+ "tour_unknown_waits": OfflineScorer(kind="boolean", version=UUID("01a15a28-e252-76a5-8945-0321df22e8fa")),+ "updated_rollout_to": OfflineScorer(kind="boolean", version=UUID("01a15a29-b8a7-7873-aada-8d3d0f396eb4")),+}+++def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},+ tags=("team:feature-flags", "component:feature-flags"),+ )diff --git a/products/notebooks/evals/eval_notebook_analysis.py b/products/notebooks/evals/eval_notebook_analysis.pyindex 826020797bb8..19d274892c58 100644--- a/products/notebooks/evals/eval_notebook_analysis.py+++ b/products/notebooks/evals/eval_notebook_analysis.py@@ -26,6 +26,7 @@ Because these cases need a completed Python run, run them on docker:from __future__ import annotations+from products.notebooks.evals.offline_config import offline_suitefrom products.notebooks.evals.scorers import (CellRunsCompleted,ChurnCohortSurfaced,@@ -125,6 +126,16 @@ async def eval_notebook_analysis(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-notebook-analysis-cli",+ offline_suite=offline_suite(+ "notebooks/eval_notebook_analysis::eval_notebook_analysis",+ metrics=[+ "required_tool_call",+ "notebook_created",+ "cell_runs_completed",+ "churn_cohort_surfaced",+ "notebook_approach_quality",+ ],+ ),cases=cases,scorers=[RequiredToolCall({"notebooks-add-cell"}),diff --git a/products/notebooks/evals/eval_notebook_cells.py b/products/notebooks/evals/eval_notebook_cells.pyindex 0b4bbf43b202..fec2cd291701 100644--- a/products/notebooks/evals/eval_notebook_cells.py+++ b/products/notebooks/evals/eval_notebook_cells.py@@ -10,6 +10,7 @@ To run one case:from __future__ import annotations+from products.notebooks.evals.offline_config import offline_suitefrom products.notebooks.evals.scorers import CellRunsCompleted, NotebookCreated, NotebookRunCompletedfrom products.notebooks.evals.seeders import seed_case_teamfrom products.posthog_ai.eval_harness.base import SandboxedPublicEval@@ -86,6 +87,10 @@ async def eval_notebook_cells(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-notebook-cells-cli",+ offline_suite=offline_suite(+ "notebooks/eval_notebook_cells::eval_notebook_cells",+ metrics=["required_tool_call", "notebook_created", "cell_runs_completed", "notebook_run_completed"],+ ),cases=cases,scorers=[RequiredToolCall({"notebooks-add-cell"}),diff --git a/products/notebooks/evals/offline_config.py b/products/notebooks/evals/offline_config.pynew file mode 100644index 000000000000..1d1f3d28c60c--- /dev/null+++ b/products/notebooks/evals/offline_config.py@@ -0,0 +1,25 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, OfflineScorer] = {+ "cell_runs_completed": OfflineScorer(kind="boolean", version=UUID("01a143b7-7d19-7d4a-8fc8-3d06a32da5ef")),+ "churn_cohort_surfaced": OfflineScorer(kind="numeric", version=UUID("01a143b7-830b-77b4-82f5-89f8d83f4ab7")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a143b7-3a45-7c94-9e46-4cf75ab3ee12")),+ "notebook_approach_quality": OfflineScorer(kind="numeric", version=UUID("01a143b7-850d-72c8-85d9-0d74005d8f37")),+ "notebook_created": OfflineScorer(kind="boolean", version=UUID("01a143b7-7b0e-7834-ab39-e6d37d772ff7")),+ "notebook_run_completed": OfflineScorer(kind="boolean", version=UUID("01a143b7-7ec4-77b5-9a32-7e9984aa2755")),+ "required_tool_call": OfflineScorer(kind="boolean", version=UUID("01a143b7-78c5-78c4-9f0d-60e0f5965b1a")),+}+++def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},+ tags=("team:data-tools", "component:notebooks"),+ )diff --git a/products/posthog_ai/backend/turn_suggestions/benchmark.py b/products/posthog_ai/backend/turn_suggestions/benchmark.pyindex 163124ba1e70..e1879ff903a0 100644--- a/products/posthog_ai/backend/turn_suggestions/benchmark.py+++ b/products/posthog_ai/backend/turn_suggestions/benchmark.py@@ -277,7 +277,7 @@ def _judge_at_endpoint(case: BenchmarkCase, endpoint: SystemOneEndpoint) -> Turnreturn read_judgment(parse_system_one_response(response.json(), questions), case.transcript, case.available)-def _judge(case: BenchmarkCase, endpoint: SystemOneEndpoint | None) -> CaseResult:+def judge_case(case: BenchmarkCase, endpoint: SystemOneEndpoint | None = None) -> CaseResult:started = time.monotonic()judgment: TurnJudgment | None = Nonefailure: str | None = None@@ -310,7 +310,7 @@ def run_cases(the order of ``cases``."""results = []with ThreadPoolExecutor(max_workers=workers) as pool:- for future in as_completed([pool.submit(_judge, case, endpoint) for case in cases]):+ for future in as_completed([pool.submit(judge_case, case, endpoint) for case in cases]):result = future.result()on_result(result)results.append(result)diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.mdindex d6b0806f7e5d..5fd455f63bff 100644--- a/products/posthog_ai/eval_harness/README.md+++ b/products/posthog_ai/eval_harness/README.md@@ -37,7 +37,8 @@ python -m products.posthog_ai.eval_harness.harness [SELECTOR ...] [flags]No manual env sourcing is needed on either path.The harness loads the repo-root `.env` itself (shell values win), and `hogli evals` additionally layers in `.env.local` / `.env.development` / `.env.services` through hogli's standard env loading — including 1Password resolution when `.env.local` holds `op://` references.Before any infrastructure boots, a preflight validates that the required variables are set and fails with a one-line fix per missing variable.-Which variables are required depends on the eval engine and the selected suites' kinds: the braintrust engine (the default, and only one today) requires `BRAINTRUST_API_KEY` on every run — it is the engine's own `required_env()`, not a core harness variable; sandboxed suites add `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`, and need a [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway) clone plus `go` on `PATH` (see [AI gateway](#ai-gateway)); one-shot suites add `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).+Which variables are required depends on the selected suites' reporting settings and kinds: the Braintrust engine requires `BRAINTRUST_API_KEY` only if at least one selected suite sends logs to Braintrust. A run consisting entirely of suites declaring `NO_SEND_LOGS = True` does not require that reporting credential. Sandboxed suites require `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`, and need a [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway) clone plus `go` on `PATH` (see [AI gateway](#ai-gateway)); one-shot suites require `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).+Scorers still need credentials for their model provider. A scorer configured to use Braintrust's model proxy needs its proxy credential independently of result reporting.Selectors are substrings matched against a suite id of the form `<domain>/<module>::<fn>`, for example `experiments`, `sql`, or `eval_lifecycle_skills`.Omit them to run every suite.@@ -452,6 +453,9 @@ Agent and judge trace capture remains available. `no_send_logs=True` suppressesSet `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal key with `offline_evaluation_ingestion:write`, `evaluation:write`, and access to project `2` to upload and tag results. An ingestion-only personal or project secret key can upload results, but tagging will warn and leave the run untagged.The [SQL suite definition](../evals/sql/eval_sql.py) pins that project ID and all seven immutable scorer version UUIDs in code.`POSTHOG_OFFLINE_EVAL_HOST` defaults to `https://us.posthog.com`.+For development, use `http://localhost:8010` with a local API key and set `POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG`+to a gitignored JSON file containing the local project ID and scorer-version mappings.+See [local upload setup](../../../docs/internal/ai-offline-evaluation-reporting.md#testing-uploads-against-local-posthog).The [suite configuration](../../../docs/internal/ai-offline-evaluation-reporting.md#configured-offline-experiment-suites) lists the metrics and explains local and CI key configuration.Delivery runs in a bounded background queue after local results are saved. Prepared uploads stay in memory, with up to two uploads running concurrently, at most 32 queued or active uploads, and a 60-second deadline per active upload.A completed upload records its results link before tagging begins. Tagging has a separate five-second deadline; failures, timeouts, and shutdown interruptions only warn and preserve the successful upload status.diff --git a/products/posthog_ai/eval_harness/engines/base.py b/products/posthog_ai/eval_harness/engines/base.pyindex ddd66620ebcb..b52aba1f554c 100644--- a/products/posthog_ai/eval_harness/engines/base.py+++ b/products/posthog_ai/eval_harness/engines/base.py@@ -41,7 +41,7 @@ class EvalEngine(Protocol):"""Whether the engine has a notion of a public experiment (``is_public``)."""@classmethod- def required_env(cls) -> tuple[EnvVarSpec, ...]:+ def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:"""The environment variables this engine needs, validated in preflight."""...diff --git a/products/posthog_ai/eval_harness/engines/braintrust.py b/products/posthog_ai/eval_harness/engines/braintrust.pyindex a6ba71180f5d..f714c7fc9a7e 100644--- a/products/posthog_ai/eval_harness/engines/braintrust.py+++ b/products/posthog_ai/eval_harness/engines/braintrust.py@@ -89,7 +89,9 @@ class BraintrustEngine:supports_public_experiments: ClassVar[bool] = True@classmethod- def required_env(cls) -> tuple[EnvVarSpec, ...]:+ def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:+ if no_send_logs:+ return ()return (EnvVarSpec("BRAINTRUST_API_KEY", "records experiments and scores to Braintrust"),)@staticmethoddiff --git a/products/posthog_ai/eval_harness/harness/discovery.py b/products/posthog_ai/eval_harness/harness/discovery.pyindex 91252a007a8f..2d77c1b44d18 100644--- a/products/posthog_ai/eval_harness/harness/discovery.py+++ b/products/posthog_ai/eval_harness/harness/discovery.py@@ -4,11 +4,12 @@ import inspectimport loggingimport importlibfrom collections.abc import Callable, Coroutine, Iterator, Sequence-from dataclasses import dataclassfrom pathlib import Pathfrom types import ModuleTypefrom typing import TYPE_CHECKING, Any+from posthog.dataclasses import frozen+from .requirements import SuiteKindif TYPE_CHECKING:@@ -37,7 +38,7 @@ cover a whole conversation. ``_collect_suites`` copies the marker onto everysuite function of the module so lifecycle can read it off the suite."""-@dataclass(frozen=True)+@frozenclass EvalSuite:domain: str"""Directory the suite lives in, e.g. ``experiments`` for a sandboxed suite or@@ -50,6 +51,9 @@ class EvalSuite:kind: SuiteKind = SuiteKind.SANDBOXED"""How the suite's cases execute, from the module-level ``SUITE_KIND``."""+ no_send_logs: bool = False+ """Module-level ``NO_SEND_LOGS`` lets preflight check credentials before suite setup."""+@propertydef id(self) -> str:return f"{self.domain}/{self.module_name}::{self.fn_name}"@@ -111,6 +115,9 @@ def _collect_suites(module: ModuleType, domain: str, module_name: str, dotted: skind = getattr(module, "SUITE_KIND", SuiteKind.SANDBOXED)if not isinstance(kind, SuiteKind):raise SuiteDiscoveryError(f"{dotted}.SUITE_KIND must be a SuiteKind member, got {kind!r}")+ no_send_logs = getattr(module, "NO_SEND_LOGS", False)+ if not isinstance(no_send_logs, bool):+ raise SuiteDiscoveryError(f"{dotted}.NO_SEND_LOGS must be a bool, got {no_send_logs!r}")multi_turn = bool(getattr(module, MULTI_TURN_MODULE_MARKER, False))collected: list[EvalSuite] = []for fn_name, fn in vars(module).items():@@ -123,7 +130,11 @@ def _collect_suites(module: ModuleType, domain: str, module_name: str, dotted: sif multi_turn:# The dataclass is frozen, so carry the marker on the function.setattr(fn, MULTI_TURN_MODULE_MARKER, True)- collected.append(EvalSuite(domain=domain, module_name=module_name, fn_name=fn_name, fn=fn, kind=kind))+ collected.append(+ EvalSuite(+ domain=domain, module_name=module_name, fn_name=fn_name, fn=fn, kind=kind, no_send_logs=no_send_logs+ )+ )return collecteddiff --git a/products/posthog_ai/eval_harness/harness/env_preflight.py b/products/posthog_ai/eval_harness/harness/env_preflight.pyindex f8ba39603da1..14c8fc25c1aa 100644--- a/products/posthog_ai/eval_harness/harness/env_preflight.py+++ b/products/posthog_ai/eval_harness/harness/env_preflight.py@@ -143,8 +143,8 @@ def validate_eval_env() -> None:"""Fail fast, before any infrastructure boots, if a required variable is unset.- ``engine_env`` is the selected engine's ``required_env()`` (e.g. the Braintrust- engine's ``BRAINTRUST_API_KEY``). Only the env models for the selected suites'+ ``engine_env`` is the selected engine's ``required_env(no_send_logs=...)``+ for the selected suites' reporting settings. Only the env models for the selected suites'kinds are checked, so a run without sandboxed suites doesn't demand sandboxcredentials. Without this a missing key surfaces minutes into a run as anopaque mid-case failure (gateway 401s, engine login errors) instead of adiff --git a/products/posthog_ai/eval_harness/harness/lifecycle.py b/products/posthog_ai/eval_harness/harness/lifecycle.pyindex 1e7e9d5c33c3..52f25315bf32 100644--- a/products/posthog_ai/eval_harness/harness/lifecycle.py+++ b/products/posthog_ai/eval_harness/harness/lifecycle.py@@ -189,7 +189,11 @@ class SandboxedEvalHarness:with asyncio.Runner() as runner:try:reporter.setup_started("check configuration")- validate_eval_env(self.options.agent_runtime, kinds=kinds, engine_env=self._engine.required_env())+ validate_eval_env(+ self.options.agent_runtime,+ kinds=kinds,+ engine_env=self._engine.required_env(no_send_logs=all(suite.no_send_logs for suite in suites)),+ )if Infra.SANDBOX in required:self.provider = build_provider(self.options.provider,diff --git a/products/posthog_ai/eval_harness/offline_results.py b/products/posthog_ai/eval_harness/offline_results.pyindex 02d4bc5014f9..3a505069fa51 100644--- a/products/posthog_ai/eval_harness/offline_results.py+++ b/products/posthog_ai/eval_harness/offline_results.py@@ -10,6 +10,7 @@ from collections.abc import Mappingfrom contextlib import suppressfrom dataclasses import fieldfrom datetime import datetime+from pathlib import Pathfrom typing import castfrom urllib.parse import urlsplitfrom uuid import UUID, uuid5@@ -33,6 +34,7 @@ from .engines.types import CaseResult, ExperimentResult, ScorerResultlogger = logging.getLogger(__name__)TAG_TIMEOUT_SECONDS = 5.0+LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"})@frozen@@ -109,6 +111,30 @@ class OfflineUploadError(Exception):super().__init__(message)+class LocalOfflineEvalConfig(BaseModel):+ model_config = ConfigDict(frozen=True, extra="forbid", hide_input_in_errors=True)++ project_id: int = Field(gt=0)+ scorer_versions: dict[UUID, UUID]++ @classmethod+ def load(cls, path: str) -> LocalOfflineEvalConfig:+ try:+ return cls.model_validate_json(Path(path).read_text())+ except (OSError, UnicodeError, ValidationError):+ raise OfflineUploadError(+ "POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG must point to a valid local destination JSON"+ ) from None++ def remap_scorers(self, suite: OfflineEvalSuite) -> dict[str, UUID]:+ try:+ return {metric: self.scorer_versions[suite.scorer_versions[metric]] for metric in suite.scorer_kinds}+ except KeyError:+ raise OfflineUploadError(+ f"Local offline evaluation config must map every enrolled scorer version for {suite.key}"+ ) from None++class OfflineEvalSettings(BaseModel):model_config = ConfigDict(frozen=True, hide_input_in_errors=True)@@ -130,13 +156,14 @@ class OfflineEvalSettings(BaseModel):@field_validator("host")@classmethoddef normalize_host(cls, host: str) -> str:+ error = "POSTHOG_OFFLINE_EVAL_HOST must be an HTTPS origin or HTTP loopback origin"try:parsed = urlsplit(host.strip().rstrip("/"))port = parsed.portexcept ValueError:- raise OfflineUploadError("POSTHOG_OFFLINE_EVAL_HOST must be an HTTPS origin") from None+ raise OfflineUploadError(error) from Noneif (- parsed.scheme != "https"+ not (parsed.scheme == "https" or (parsed.scheme == "http" and parsed.hostname in LOOPBACK_HOSTS))or not parsed.hostnameor parsed.username is not Noneor parsed.password is not None@@ -144,15 +171,15 @@ class OfflineEvalSettings(BaseModel):or parsed.fragmentor parsed.path):- raise OfflineUploadError("POSTHOG_OFFLINE_EVAL_HOST must be an HTTPS origin")+ raise OfflineUploadError(error)origin = parsed.netloc.lower()- if port in (None, 443):+ if parsed.scheme == "https" and port in (None, 443):origin = {"app.posthog.com": "us.posthog.com","us.i.posthog.com": "us.posthog.com","eu.i.posthog.com": "eu.posthog.com",}.get(parsed.hostname, origin.removesuffix(":443"))- return f"https://{origin}"+ return f"{parsed.scheme}://{origin}"@classmethoddef from_env(cls, suite: OfflineEvalSuite) -> OfflineEvalSettings | None:@@ -160,12 +187,20 @@ class OfflineEvalSettings(BaseModel):api_key = os.getenv(prefix + "API_KEY")if not api_key:return None+ host = cls.normalize_host(os.getenv(prefix + "HOST", "https://us.posthog.com"))+ local_config = None+ if local_config_path := os.getenv(prefix + "LOCAL_CONFIG"):+ if urlsplit(host).hostname not in LOOPBACK_HOSTS:+ raise OfflineUploadError("POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG requires a loopback upload host")+ local_config = LocalOfflineEvalConfig.load(local_config_path)try:settings = cls(- host=os.getenv(prefix + "HOST", "https://us.posthog.com"),- project_id=suite.project_id,+ host=host,+ project_id=local_config.project_id if local_config is not None else suite.project_id,api_key=SecretStr(api_key),- scorer_versions=dict(suite.scorer_versions),+ scorer_versions=local_config.remap_scorers(suite)+ if local_config is not None+ else dict(suite.scorer_versions),)except ValidationError:raise OfflineUploadError(diff --git a/products/posthog_ai/eval_harness/test/test_benchmark_scorers.py b/products/posthog_ai/eval_harness/test/test_benchmark_scorers.pynew file mode 100644index 000000000000..c02859a0222d--- /dev/null+++ b/products/posthog_ai/eval_harness/test/test_benchmark_scorers.py@@ -0,0 +1,37 @@+import pytest++from products.business_knowledge.evals.eval_retrieval import RetrievalMatch+from products.posthog_ai.backend.turn_suggestions.benchmark import Outcome+from products.posthog_ai.evals.turn_suggestions.eval_turn_suggestions import OfferCorrectness+++ "outcome,score",+ [+ (Outcome.CORRECT, 1.0),+ (Outcome.EITHER, 1.0),+ (Outcome.FALSE_OFFER, 0.0),+ (Outcome.MISSED, 0.0),+ (Outcome.WRONG_KIND, 0.0),+ (Outcome.FAILED, None),+ ],+)+def test_offer_correctness_preserves_benchmark_outcomes(outcome: Outcome, score: float | None) -> None:+ result = OfferCorrectness().eval({"outcome": outcome, "error": "judge unavailable"})+ assert result.score == score+ if outcome == Outcome.FAILED:+ assert result.result is not None+ assert result.result.status == "error"+ assert result.result.error_code == "judge_error"+ assert result.result.error_message == "judge unavailable"+ else:+ assert result.result is None++[email protected]("found,expected,score", [(True, True, 1.0), (False, False, 1.0), (False, True, 0.0)])+def test_retrieval_scores_absence_as_a_valid_outcome(found: bool, expected: bool, score: float) -> None:+ result = RetrievalMatch().eval(+ {"found": found, "source_names": ["Refund Policy"] if found else []},+ {"retrieval_match": {"found": expected}},+ )+ assert result.score == scorediff --git a/products/posthog_ai/eval_harness/test/test_discovery.py b/products/posthog_ai/eval_harness/test/test_discovery.pyindex afcf74e55e1f..d98737f45366 100644--- a/products/posthog_ai/eval_harness/test/test_discovery.py+++ b/products/posthog_ai/eval_harness/test/test_discovery.py@@ -4,13 +4,14 @@ from pathlib import Pathimport pytest-from products.posthog_ai.eval_harness.harness.discovery import SuiteDiscoveryError, discover_suites+from products.posthog_ai.eval_harness.harness.discovery import EvalSuite, SuiteDiscoveryError, discover_suitesfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind_ONE_SHOT_MODULE = """from products.posthog_ai.eval_harness.harness.requirements import SuiteKindSUITE_KIND = SuiteKind.ONE_SHOT+NO_SEND_LOGS = Trueasync def eval_alpha(ctx) -> None:@@ -53,10 +54,10 @@ def product_roots(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> tuple[Pathreturn products_root, empty_builtin_root-def _discover(product_roots: tuple[Path, Path]) -> dict[str, SuiteKind]:+def _discover(product_roots: tuple[Path, Path]) -> dict[str, EvalSuite]:products_root, builtin_root = product_rootssuites = discover_suites(builtin_root=builtin_root, products_root=products_root)- return {suite.id: suite.kind for suite in suites}+ return {suite.id: suite for suite in suites}def test_product_suites_discovered_with_product_scoped_ids(product_roots: tuple[Path, Path]) -> None:@@ -70,16 +71,18 @@ def test_singular_eval_dir_is_ignored(product_roots: tuple[Path, Path]) -> None:@pytest.mark.parametrize(- "suite_id, expected_kind",+ "suite_id, expected_kind, no_send_logs",[- pytest.param("analytics/eval_one::eval_alpha", SuiteKind.ONE_SHOT, id="marker_respected"),- pytest.param("core/eval_two::eval_beta", SuiteKind.SANDBOXED, id="absent_marker_defaults_sandboxed"),+ pytest.param("analytics/eval_one::eval_alpha", SuiteKind.ONE_SHOT, True, id="markers_respected"),+ pytest.param("core/eval_two::eval_beta", SuiteKind.SANDBOXED, False, id="absent_markers_keep_defaults"),],)-def test_suite_kind_resolved_from_product_module(- product_roots: tuple[Path, Path], suite_id: str, expected_kind: SuiteKind+def test_suite_configuration_resolved_from_product_module(+ product_roots: tuple[Path, Path], suite_id: str, expected_kind: SuiteKind, no_send_logs: bool) -> None:- assert _discover(product_roots)[suite_id] == expected_kind+ suite = _discover(product_roots)[suite_id]+ assert suite.kind == expected_kind+ assert suite.no_send_logs == no_send_logsdef test_selector_matches_product_suite_ids(product_roots: tuple[Path, Path]) -> None:diff --git a/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py b/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.pyindex bf8e7958d855..b7d7360b0cac 100644--- a/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py+++ b/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py@@ -35,7 +35,7 @@ def test_harness_tears_down_infrastructure_before_draining_uploads(interrupted: bool,teardown_phase: Literal["async", "sync"],) -> None:- monkeypatch.setenv("BRAINTRUST_API_KEY", "test-only-key")+ monkeypatch.delenv("BRAINTRUST_API_KEY", raising=False)monkeypatch.setenv("LLM_GATEWAY_ANTHROPIC_API_KEY", "test-only-key")monkeypatch.delenv("EXPORT_EVAL_RESULTS", raising=False)started = asyncio.Event()@@ -83,7 +83,14 @@ def test_harness_tears_down_infrastructure_before_draining_uploads(lifecycle,"discover_suites",lambda _: [- EvalSuite(domain="test", module_name="eval_test", fn_name="eval_test", fn=suite, kind=SuiteKind.ONE_SHOT)+ EvalSuite(+ domain="test",+ module_name="eval_test",+ fn_name="eval_test",+ fn=suite,+ kind=SuiteKind.ONE_SHOT,+ no_send_logs=True,+ )],)monkeypatch.setattr(lifecycle, "ensure_personhog_binaries", lambda: None)diff --git a/products/posthog_ai/eval_harness/test/test_offline_enrollment.py b/products/posthog_ai/eval_harness/test/test_offline_enrollment.pynew file mode 100644index 000000000000..259f38d36a87--- /dev/null+++ b/products/posthog_ai/eval_harness/test/test_offline_enrollment.py@@ -0,0 +1,62 @@+from pathlib import Path+from uuid import UUID++import pytest+from unittest.mock import Mock, patch++from products.posthog_ai.eval_harness.base import _BaseEvalRun+from products.posthog_ai.eval_harness.harness.context import EvalContext+from products.posthog_ai.eval_harness.harness.discovery import discover_suites+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+++async def test_discovered_suites_match_reporting_enrollment(tmp_path: Path) -> None:+ ctx = Mock(spec=EvalContext, posthog_client=None, case_filter=None, skill_delivery="bundled")+ ctx.engine = Mock()+ ctx.provider_strategy = Mock()+ ctx.demo_data = Mock()+ ctx.sandbox_slots = Mock()+ runs: list[_BaseEvalRun] = []++ async def capture(run: _BaseEvalRun) -> None:+ runs.append(run)++ suites = discover_suites()+ assert suites+ keys: set[str] = set()+ with (+ patch.object(_BaseEvalRun, "run", capture),+ patch("products.posthog_ai.eval_harness.base.build_case_dir", return_value=tmp_path),+ patch("products.posthog_ai.evals.taxonomic_filter.eval_event_match.system_one_configured", return_value=True),+ patch("products.posthog_ai.evals.taxonomic_filter.eval_event_match.EVENT_MATCH_MODEL"),+ patch("products.posthog_ai.evals.taxonomic_filter.eval_search_intent.system_one_configured", return_value=True),+ patch("products.posthog_ai.evals.taxonomic_filter.eval_search_intent.fetch_search_intent_prompt"),+ ):+ for suite in suites:+ runs.clear()+ await suite.fn(ctx)+ assert len(runs) == 1, suite.id+ run = runs[0]+ assert run.no_send_logs == suite.no_send_logs, suite.id+ config = run.offline_suite+ if suite.id == "logs/eval_agent_observability::eval_agent_observability":+ assert config is None+ assert not run.no_send_logs+ continue+ assert config is not None, suite.id+ assert run.send_to_posthog, suite.id+ assert config.key == suite.id+ assert config.key not in keys+ keys.add(config.key)+ assert config.project_id == 2+ metrics = {scorer._name() for scorer in run.active_scorers}+ assert set(config.scorer_kinds) == set(config.scorer_versions) == metrics, suite.id+ assert len(set(config.scorer_versions.values())) == len(metrics), suite.id+ assert all(isinstance(version, UUID) and version.int for version in config.scorer_versions.values())+ assert ("exit_code_zero" in metrics) == (suite.kind == SuiteKind.SANDBOXED), suite.id+ assert any(tag.startswith("team:") for tag in config.tags), suite.id+ assert any(tag.startswith("component:") for tag in config.tags), suite.id++ if suite.domain in {"feature_flags", "dashboards", "business_knowledge", "turn_suggestions"}:+ assert run.no_send_logs, suite.iddiff --git a/products/posthog_ai/eval_harness/test/test_offline_results.py b/products/posthog_ai/eval_harness/test/test_offline_results.pyindex 4e3ae913157c..f7e10b51c954 100644--- a/products/posthog_ai/eval_harness/test/test_offline_results.py+++ b/products/posthog_ai/eval_harness/test/test_offline_results.py@@ -5,6 +5,7 @@ import asynciofrom dataclasses import replacefrom datetime import UTC, datetimefrom functools import partial+from pathlib import Pathfrom uuid import UUIDimport pytest@@ -594,6 +595,9 @@ def test_publish_refuses_a_different_destination(_name: str, host: str, project_("custom", "https://posthog.example.com:8443", "https://posthog.example.com:8443"),("custom_cloud_port", "https://us.i.posthog.com:8443", "https://us.i.posthog.com:8443"),("ipv6", "https://[::1]:8443", "https://[::1]:8443"),+ ("localhost", "http://localhost:8010", "http://localhost:8010"),+ ("loopback", "http://127.0.0.1:8010/", "http://127.0.0.1:8010"),+ ("loopback_ipv6", "http://[::1]:8010", "http://[::1]:8010"),])def test_publish_uses_the_same_origin_for_requests_and_result_links(_name: str, host: str, origin: str) -> None:@@ -616,6 +620,96 @@ def test_publish_uses_the_same_origin_for_requests_and_result_links(_name: str,assert url == f"{origin}/project/123/ai-evals/evaluations/offline/experiments/{EXPERIMENT_ID}"+def test_local_config_routes_uploads_to_local_project_and_scorer_versions(tmp_path: Path) -> None:+ local_versions = {+ str(SCORER_VERSIONS["boolean_metric"]): "00000000-0000-4000-8000-000000000004",+ str(SCORER_VERSIONS["numeric_metric"]): "00000000-0000-4000-8000-000000000005",+ }+ config = tmp_path / "local.json"+ config.write_text(json.dumps({"project_id": 456, "scorer_versions": local_versions}))+ with patch.dict(+ "os.environ",+ {+ "POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token",+ "POSTHOG_OFFLINE_EVAL_HOST": "http://localhost:8010",+ "POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG": str(config),+ },+ clear=True,+ ):+ settings = OfflineEvalSettings.from_env(SUITE)+ assert settings is not None+ publisher = OfflineEvalPublisher(settings, SUITE)+ upload = _prepare(publisher, [_case()])+ server = OfflineServer()++ with patch("httpx.AsyncClient", new=partial(httpx.AsyncClient, transport=httpx.MockTransport(server.respond))):+ url = asyncio.run(publisher.publish(upload))++ assert all(str(request.url).startswith("http://localhost:8010/api/projects/456/") for request in server.requests)+ assert [str(result.scorer_version_id) for result in upload.results] == list(local_versions.values())+ assert url == f"http://localhost:8010/project/456/ai-evals/evaluations/offline/experiments/{EXPERIMENT_ID}"+ assert SUITE.project_id == 123+ assert SUITE.scorer_versions == SCORER_VERSIONS+++ "contents,host",+ [+ (None, "http://localhost:8010"),+ ("invalid json", "http://localhost:8010"),+ ('{"project_id": 0, "scorer_versions": {}}', "http://localhost:8010"),+ ('{"project_id": 456, "scorer_versions": {}}', "http://localhost:8010"),+ ('{"project_id": 456, "scorer_versions": {"invalid": "invalid"}}', "http://localhost:8010"),+ (+ json.dumps(+ {+ "project_id": 456,+ "scorer_versions": {str(version): str(version) for version in SCORER_VERSIONS.values()},+ }+ ),+ "https://us.posthog.com",+ ),+ (+ json.dumps(+ {+ "project_id": 456,+ "scorer_versions": dict.fromkeys(+ map(str, SCORER_VERSIONS.values()), "00000000-0000-4000-8000-000000000004"+ ),+ }+ ),+ "http://localhost:8010",+ ),+ ],+ ids=[+ "missing_file",+ "invalid_json",+ "invalid_project",+ "missing_versions",+ "invalid_uuid",+ "cloud_host",+ "duplicate_versions",+ ],+)+def test_invalid_local_config_never_falls_back_to_catalog(tmp_path: Path, contents: str | None, host: str) -> None:+ config = tmp_path / "local.json"+ if contents is not None:+ config.write_text(contents)+ with (+ patch.dict(+ "os.environ",+ {+ "POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token",+ "POSTHOG_OFFLINE_EVAL_HOST": host,+ "POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG": str(config),+ },+ clear=True,+ ),+ pytest.raises(OfflineUploadError),+ ):+ OfflineEvalSettings.from_env(SUITE)++def test_settings_use_each_suite_destination_with_only_an_api_key() -> None:environment = {"POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token"}with patch.dict("os.environ", environment, clear=True):@@ -661,6 +755,8 @@ def test_settings_reject_invalid_suite_configuration(_name: str, suite: OfflineE("key_internal_space", {"POSTHOG_OFFLINE_EVAL_API_KEY": "test-only token"}),("key_leading_tab", {"POSTHOG_OFFLINE_EVAL_API_KEY": "\ttest-only-token"}),("insecure_host", {"POSTHOG_OFFLINE_EVAL_HOST": "http://posthog.example.com"}),+ ("loopback_suffix", {"POSTHOG_OFFLINE_EVAL_HOST": "http://localhost.example.com:8010"}),+ ("loopback_credentials", {"POSTHOG_OFFLINE_EVAL_HOST": "http://[email protected]:8010"}),("malformed_host", {"POSTHOG_OFFLINE_EVAL_HOST": "https://["}),("invalid_port", {"POSTHOG_OFFLINE_EVAL_HOST": "https://posthog.example.com:invalid"}),("host_credentials", {"POSTHOG_OFFLINE_EVAL_HOST": "https://[email protected]"}),diff --git a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py b/products/posthog_ai/eval_harness/test/test_one_shot_runner.pyindex 93d72fc36f53..57bf2da45036 100644--- a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py+++ b/products/posthog_ai/eval_harness/test/test_one_shot_runner.py@@ -93,7 +93,7 @@ class _StubEngine:self.calls: list[ExperimentSpec] = []@classmethod- def required_env(cls) -> tuple[EnvVarSpec, ...]:+ def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:return ()async def run_experiment(self, spec: ExperimentSpec) -> ExperimentResult:diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.pyindex bf1d6a95cd7a..e900c5d7dc35 100644--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py@@ -413,7 +413,7 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repohttp_status: int,delivery_state: Literal["ready", "unavailable", "closed"],) -> None:- for key in ("API_KEY", "HOST"):+ for key in ("API_KEY", "HOST", "LOCAL_CONFIG"):monkeypatch.delenv(f"POSTHOG_OFFLINE_EVAL_{key}", raising=False)monkeypatch.setenv("EXPORT_EVAL_RESULTS", "1")monkeypatch.chdir(tmp_path)@@ -570,10 +570,9 @@ async def test_suite_enrolls_every_scorer_in_offline_publishing(else:monkeypatch.setattr(module, "fetch_search_intent_prompt", MagicMock())run = AsyncMock()- runner = "SandboxedPrivateEval" if domain == "experiments" else "SandboxedPublicEval"- if suite_kind == SuiteKind.ONE_SHOT:- runner = "OneShotPublicEval"- monkeypatch.setattr(module, runner, run)+ runner_kind = "OneShot" if suite_kind == SuiteKind.ONE_SHOT else "Sandboxed"+ runner_visibility = "Private" if discovered.no_send_logs else "Public"+ monkeypatch.setattr(module, f"{runner_kind}{runner_visibility}Eval", run)await discovered.fn(MagicMock(skill_delivery="bundled"))diff --git a/products/posthog_ai/eval_harness/test/test_suite_requirements.py b/products/posthog_ai/eval_harness/test/test_suite_requirements.pyindex eed3b1ea5348..e9fd0aea6c15 100644--- a/products/posthog_ai/eval_harness/test/test_suite_requirements.py+++ b/products/posthog_ai/eval_harness/test/test_suite_requirements.py@@ -1,8 +1,13 @@from __future__ import annotationsimport pytest+from unittest.mock import Mockfrom products.posthog_ai.eval_harness.engines.registry import resolve_engine+from products.posthog_ai.eval_harness.harness import lifecycle+from products.posthog_ai.eval_harness.harness.cli import parse_args+from products.posthog_ai.eval_harness.harness.context import EvalContext+from products.posthog_ai.eval_harness.harness.discovery import EvalSuitefrom products.posthog_ai.eval_harness.harness.env_preflight import validate_eval_envfrom products.posthog_ai.eval_harness.harness.providers import PreflightErrorfrom products.posthog_ai.eval_harness.harness.requirements import INFRA_BY_KIND, Infra, SuiteKind, expand, infra_union@@ -130,14 +135,69 @@ def test_validate_eval_env_by_kind(assert name not in message-def test_validate_eval_env_reports_missing_engine_env(monkeypatch: pytest.MonkeyPatch) -> None:[email protected]("no_send_logs", [False, True])+def test_validate_eval_env_reports_missing_engine_env(monkeypatch: pytest.MonkeyPatch, no_send_logs: bool) -> None:for name in ALL_ENV_VARS:monkeypatch.delenv(name, raising=False)monkeypatch.setenv("LLM_GATEWAY_ANTHROPIC_API_KEY", "x")- with pytest.raises(PreflightError) as exc_info:- validate_eval_env("claude", kinds={SuiteKind.ONE_SHOT}, engine_env=resolve_engine().required_env())+ engine_env = resolve_engine().required_env(no_send_logs=no_send_logs)+ if no_send_logs:+ validate_eval_env("claude", kinds={SuiteKind.ONE_SHOT}, engine_env=engine_env)+ return- # The Braintrust engine's requirement is reported with the same line the core- # env model used to emit, so a missing key reads identically to before.+ with pytest.raises(PreflightError) as exc_info:+ validate_eval_env("claude", kinds={SuiteKind.ONE_SHOT}, engine_env=engine_env)assert " - BRAINTRUST_API_KEY: records experiments and scores to Braintrust" in str(exc_info.value)+++ "no_send_logs, has_key",+ [+ pytest.param([True], False, id="posthog_only_without_braintrust_key"),+ pytest.param([False], False, id="braintrust_reporting_without_key"),+ pytest.param([True, False], False, id="mixed_reporting_without_key"),+ pytest.param([True, False], True, id="mixed_reporting_with_key"),+ ],+)+def test_harness_checks_selected_reporting_before_starting_infrastructure(+ monkeypatch: pytest.MonkeyPatch, no_send_logs: list[bool], has_key: bool+) -> None:+ monkeypatch.delenv("BRAINTRUST_API_KEY", raising=False)+ monkeypatch.delenv("EXPORT_EVAL_RESULTS", raising=False)+ monkeypatch.setenv("LLM_GATEWAY_ANTHROPIC_API_KEY", "test-only-key")+ if has_key:+ monkeypatch.setenv("BRAINTRUST_API_KEY", "test-only-key")+ completed: list[bool] = []++ async def run_suite(ctx: EvalContext) -> None:+ completed.append(True)++ suites = [+ EvalSuite(+ domain="test",+ module_name="eval_test",+ fn_name=f"eval_test_{index}",+ fn=run_suite,+ kind=SuiteKind.ONE_SHOT,+ no_send_logs=disabled,+ )+ for index, disabled in enumerate(no_send_logs)+ ]+ monkeypatch.setattr(lifecycle, "discover_suites", lambda _: suites)+ build = Mock()+ monkeypatch.setattr(lifecycle, "ensure_personhog_binaries", build)+ harness = lifecycle.SandboxedEvalHarness(parse_args([]))++ def bootstrap(required: frozenset[Infra], reporter: lifecycle.ProgressReporter) -> None:+ harness._demo_data = Mock()++ monkeypatch.setattr(harness, "_bootstrap", bootstrap)+ if has_key or all(no_send_logs):+ assert harness.run() == 0+ assert len(completed) == len(suites)+ else:+ with pytest.raises(PreflightError, match="BRAINTRUST_API_KEY"):+ harness.run()+ build.assert_not_called()+ assert not completeddiff --git a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.pyindex 9e7b72f954a1..867db0ffcd27 100644--- a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py+++ b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py@@ -41,6 +41,8 @@ from products.posthog_ai.evals.experiments.seeders import ()from products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+async def eval_bias_uneven_split(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -107,6 +109,7 @@ async def eval_bias_uneven_split(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-diagnose-bias-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_create_control_variant.py b/products/posthog_ai/evals/experiments/eval_create_control_variant.pyindex b7a6e5e93a0e..e3993092b539 100644--- a/products/posthog_ai/evals/experiments/eval_create_control_variant.py+++ b/products/posthog_ai/evals/experiments/eval_create_control_variant.py@@ -36,6 +36,8 @@ from products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCallfrom products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+async def eval_create_control_variant(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -56,6 +58,7 @@ async def eval_create_control_variant(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-create-control-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_empty_experiment.py b/products/posthog_ai/evals/experiments/eval_empty_experiment.pyindex 9e962c140071..474dd56e0acc 100644--- a/products/posthog_ai/evals/experiments/eval_empty_experiment.py+++ b/products/posthog_ai/evals/experiments/eval_empty_experiment.py@@ -54,6 +54,8 @@ from products.posthog_ai.evals.experiments.seeders import ()from products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+async def eval_empty_experiment(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -139,6 +141,7 @@ async def eval_empty_experiment(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-diagnose-empty-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py b/products/posthog_ai/evals/experiments/eval_interpretation_traps.pyindex 262879511506..37204d92d4a1 100644--- a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py+++ b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py@@ -52,6 +52,8 @@ from products.posthog_ai.evals.experiments.seeders import ()from products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+async def eval_interpretation_traps(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -161,6 +163,7 @@ async def eval_interpretation_traps(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-diagnose-interpretation-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.pyindex e085458bcad5..e1001365b69d 100644--- a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py+++ b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py@@ -42,6 +42,8 @@ from products.posthog_ai.evals.experiments.scorers import (from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+# Keep the cases in one experiment so their scores can be compared together.async def eval_lifecycle_skills(ctx: EvalContext) -> None:@@ -84,6 +86,7 @@ async def eval_lifecycle_skills(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-lifecycle-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.pyindex 5b4cac746250..b54d76c1bb31 100644--- a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py+++ b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py@@ -27,6 +27,8 @@ from products.posthog_ai.evals.experiments.scorers import (from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+async def eval_metric_schema_discovery(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -53,6 +55,7 @@ async def eval_metric_schema_discovery(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-metric-schema-discovery-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_midrun_changes.py b/products/posthog_ai/evals/experiments/eval_midrun_changes.pyindex 7db3ca178df9..9062c588fac6 100644--- a/products/posthog_ai/evals/experiments/eval_midrun_changes.py+++ b/products/posthog_ai/evals/experiments/eval_midrun_changes.py@@ -63,6 +63,8 @@ from products.posthog_ai.evals.experiments.seeders import ()from products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+async def eval_midrun_changes(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -152,6 +154,7 @@ async def eval_midrun_changes(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-diagnose-midrun-changes-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.pyindex 4604a3d058e9..db3f2b2fbd23 100644--- a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py+++ b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py@@ -45,6 +45,8 @@ from products.posthog_ai.evals.experiments.scorers import CitesDiagnosticGroupfrom products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+async def eval_numbers_vs_sql(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -154,6 +156,7 @@ async def eval_numbers_vs_sql(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-diagnose-numbers-vs-sql-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_rollout_skill.py b/products/posthog_ai/evals/experiments/eval_rollout_skill.pyindex 59170914ec04..021026a3d1ee 100644--- a/products/posthog_ai/evals/experiments/eval_rollout_skill.py+++ b/products/posthog_ai/evals/experiments/eval_rollout_skill.py@@ -25,6 +25,8 @@ from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAMfrom products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.retrieval.scorers import SkillLoaded+NO_SEND_LOGS = True+SKILL_NAME = "configuring-experiment-rollout"@@ -40,6 +42,7 @@ async def eval_rollout_skill(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-rollout-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.pyindex 178b7f8bc1d8..3673915aaf4b 100644--- a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py+++ b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py@@ -15,6 +15,8 @@ from products.posthog_ai.evals.experiments.scorers import SharedMetricValidationfrom products.posthog_ai.evals.experiments.seeders import SHARED_METRIC_NAME, seed_shared_metric_purchase_countfrom products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+async def eval_shared_metric_validation(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -50,6 +52,7 @@ async def eval_shared_metric_validation(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-shared-metric-validation-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.pyindex 9d58348b98a5..d748fd4858ee 100644--- a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py+++ b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py@@ -42,6 +42,8 @@ from products.posthog_ai.evals.experiments.scorers import AskedForConfirmationfrom products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suite+NO_SEND_LOGS = True+# Event name chosen to be obviously fictional so the agent can't plausibly# justify it as a typo of a real PostHog event in the seeded team.UNKNOWN_EVENT_NAME = "totally_made_up_event_qzx"@@ -66,6 +68,7 @@ async def eval_unknown_events_confirmation(ctx: EvalContext) -> None:]await SandboxedPrivateEval(+ no_send_logs=NO_SEND_LOGS,experiment_name="sandboxed-experiments-unknown-events-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/offline_config.py b/products/posthog_ai/evals/offline_config.pyindex 4479074fb0ec..0f6fd1df4bba 100644--- a/products/posthog_ai/evals/offline_config.py+++ b/products/posthog_ai/evals/offline_config.py@@ -127,6 +127,16 @@ _SCORERS: dict[str, dict[str, OfflineScorer]] = {"surfaces_all_findings": OfflineScorer(kind="boolean", version=UUID("01a1558f-ad52-7b81-8a93-d7e602bf0125")),},"product_analytics": {+ "answer_tool_call_not_forbidden": OfflineScorer(+ kind="boolean", version=UUID("01a15a27-7d06-768b-940c-f47b5f3ee78c")+ ),+ "did_not_cite_raw_sql_after_typed_query": OfflineScorer(+ kind="boolean", version=UUID("01a15a27-735b-700b-9f5b-51533df620b3")+ ),+ "no_markdown_data_table": OfflineScorer(kind="boolean", version=UUID("01a15a27-69b4-7bc0-b6f6-b366acc272b2")),+ "chart_from_typed_query": OfflineScorer(kind="boolean", version=UUID("01a15a27-5e27-7511-9f3e-58a57e74556d")),+ "chart_shown": OfflineScorer(kind="boolean", version=UUID("01a15a27-553d-72d2-aeb2-903e7ed54473")),+ "answer_from_typed_query": OfflineScorer(kind="boolean", version=UUID("01a15a27-4bce-7e6c-b3ef-34d292489b30")),"answer_tool_not_execute_sql": OfflineScorer(kind="boolean", version=UUID("01a15581-3926-7c14-a907-e994586bd216")),@@ -201,6 +211,9 @@ _SCORERS: dict[str, dict[str, OfflineScorer]] = {kind="boolean", version=UUID("01a15582-7cea-7b35-9632-1fd1c3e22185")),},+ "turn_suggestions": {+ "offer_correctness": OfflineScorer(kind="boolean", version=UUID("01a15a29-ea9e-743e-9a21-cec75e5e417e")),+ },"taxonomic_filter": {"event_match_found": OfflineScorer(kind="boolean", version=UUID("01a15582-8f96-76a8-a6b0-9ad9c932f12a")),"exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15582-85bb-71aa-b4c8-dc008015ab30")),diff --git a/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py b/products/posthog_ai/evals/product_analytics/eval_chart_not_table.pyindex de7efdda29ec..be88effa6500 100644--- a/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py+++ b/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py@@ -34,6 +34,7 @@ from products.posthog_ai.evals.cli_mcp.scorers import (DidNotCiteRawSqlAfterTypedQuery,NoMarkdownDataTable,)+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS, AnswerFromTypedQuery@@ -93,6 +94,17 @@ async def eval_chart_not_table(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-chart-not-table-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_chart_not_table::eval_chart_not_table",+ metrics=[+ "no_persistent_insight_save",+ "answer_from_typed_query",+ "chart_shown",+ "chart_from_typed_query",+ "no_markdown_data_table",+ "did_not_cite_raw_sql_after_typed_query",+ ],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),diff --git a/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py b/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.pyindex 8b52c9c9580e..0222167831f2 100644--- a/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py+++ b/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py@@ -35,6 +35,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCallfrom products.posthog_ai.evals.cli_mcp.scorers import FirstRelevantTool+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import (INSIGHT_WRITE_TOOLS,TYPED_QUERY_TOOLS,@@ -164,6 +165,15 @@ async def eval_typed_runner_misses(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-typed-runner-misses-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_typed_runner_misses::eval_typed_runner_misses",+ metrics=[+ "no_persistent_insight_save",+ "answer_from_typed_query",+ "answer_tool_call_not_forbidden",+ "insight_shape",+ ],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),@@ -197,6 +207,10 @@ async def eval_justified_sql_controls(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-justified-sql-controls-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_typed_runner_misses::eval_justified_sql_controls",+ metrics=["no_persistent_insight_save", "first_relevant_tool"],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),diff --git a/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.py b/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.pynew file mode 100644index 000000000000..29693b85e68c--- /dev/null+++ b/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.py@@ -0,0 +1,92 @@+from __future__ import annotations++import os+import json+import asyncio+from typing import TYPE_CHECKING++from products.posthog_ai.backend.turn_suggestions.benchmark import Outcome, judge_case, load_cases+from products.posthog_ai.backend.turn_suggestions.classifier import SHOW_THRESHOLD+from products.posthog_ai.backend.turn_suggestions.judgment import build_judge_state, judge_model+from products.posthog_ai.eval_harness.config import BaseEvalCase+from products.posthog_ai.eval_harness.engines.types import ScorerResult+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.one_shot import OneShotPrivateEval+from products.posthog_ai.eval_harness.scorers.contract import Score, Scorer+from products.posthog_ai.evals.offline_config import offline_suite++if TYPE_CHECKING:+ from products.posthog_ai.eval_harness.harness.context import EvalContext++NO_SEND_LOGS = True++SUITE_KIND = SuiteKind.ONE_SHOT+BENCHMARK_CASES = {case.name: case for case in load_cases()}+CASES = [+ BaseEvalCase(+ name=case.name,+ prompt=json.dumps(build_judge_state(case.transcript)),+ expected={"acceptable": sorted(case.acceptable)},+ metadata={"category": case.category, "available": sorted(case.available)},+ )+ for case in BENCHMARK_CASES.values()+]+++class OfferCorrectness(Scorer):+ def _name(self) -> str:+ return "offer_correctness"++ def _run_eval_sync(+ self, output: dict[str, object] | None, expected: dict[str, object] | None = None, **kwargs: object+ ) -> Score:+ outcome = (output or {}).get("outcome")+ if outcome is None or outcome == Outcome.FAILED:+ reason = (output or {}).get("error") or "Turn judge returned no judgment"+ return Score(+ name=self._name(),+ score=None,+ result=ScorerResult(status="error", error_code="judge_error", error_message=str(reason)),+ )+ return Score(+ name=self._name(),+ score=float(outcome in (Outcome.CORRECT, Outcome.EITHER)),+ metadata={"reason": outcome},+ )+++async def eval_turn_suggestions(ctx: EvalContext) -> None:+ threshold = float(os.environ.get("TURN_SUGGESTIONS_EVAL_THRESHOLD", str(SHOW_THRESHOLD)))+ if not 0 <= threshold <= 1:+ raise ValueError("TURN_SUGGESTIONS_EVAL_THRESHOLD must be between 0 and 1")+ model = judge_model()++ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict[str, object]:+ result = await asyncio.to_thread(judge_case, BENCHMARK_CASES[case.name])+ return {+ "outcome": result.outcome(threshold),+ "picked": result.picked(threshold),+ "show_probability": result.judgment.show_probability if result.judgment else None,+ "threshold": threshold,+ "judge_model": model,+ "seconds": result.seconds,+ "error": result.error,+ }++ await OneShotPrivateEval(+ no_send_logs=NO_SEND_LOGS,+ experiment_name="turn-suggestions",+ cases=[+ case.model_copy(update={"metadata": {**case.metadata, "threshold": threshold, "judge_model": model}})+ for case in CASES+ ],+ task=task,+ scorers=[OfferCorrectness()],+ offline_suite=offline_suite(+ "turn_suggestions/eval_turn_suggestions::eval_turn_suggestions",+ metrics=["offer_correctness"],+ suite_kind=SUITE_KIND,+ ),+ send_to_posthog=True,+ ctx=ctx,+ )diff --git a/products/slack_app/evals/eval_untagged_question.py b/products/slack_app/evals/eval_untagged_question.pyindex 66f077c862da..39052dbf17ab 100644--- a/products/slack_app/evals/eval_untagged_question.py+++ b/products/slack_app/evals/eval_untagged_question.py@@ -21,6 +21,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKindfrom products.posthog_ai.eval_harness.one_shot import OneShotPublicEvalfrom products.slack_app.backend.temporal.activities import untagged_question+from products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import UNTAGGED_QUESTION_KEY, NoUnaskedAnswer, UntaggedQuestionMatchSUITE_KIND = SuiteKind.ONE_SHOT@@ -256,6 +257,11 @@ async def eval_untagged_question(ctx: EvalContext) -> None:await OneShotPublicEval(experiment_name="slack-app-untagged-question",+ offline_suite=offline_suite(+ "slack_app/eval_untagged_question::eval_untagged_question",+ metrics=[UNTAGGED_QUESTION_KEY, "no_unasked_answer"],+ suite_kind=SUITE_KIND,+ ),cases=[*ANSWERABLE_CASES, *LEAVE_ALONE_CASES],scorers=[UntaggedQuestionMatch(), NoUnaskedAnswer()],task=task,diff --git a/products/slack_app/evals/offline_config.py b/products/slack_app/evals/offline_config.pyindex b86cc92224c9..ae6c5fa1261b 100644--- a/products/slack_app/evals/offline_config.py+++ b/products/slack_app/evals/offline_config.py@@ -36,6 +36,10 @@ _SCORERS: dict[str, dict[str, OfflineScorer]] = {"reply_offer": OfflineScorer(kind="boolean", version=UUID("01a1557d-5266-7853-a431-6213cfba1b5a")),"suggested_replies_fit": OfflineScorer(kind="boolean", version=UUID("01a1557d-bcc0-7a96-8d22-4d6e91998d5d")),},+ "untagged_question": {+ "untagged_question": OfflineScorer(kind="boolean", version=UUID("01a15a43-79f8-76bd-98ad-aaacc69b7059")),+ "no_unasked_answer": OfflineScorer(kind="boolean", version=UUID("01a15a43-8605-78ef-acf6-cc940d3876ab")),+ },}