Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(aio): report remaining offline eval suites to posthog (#120545)

Radu Raicea676169268523
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md
index 3bfc4f287e40..068a31b1c7fa 100644
--- a/docs/internal/ai-offline-evaluation-reporting.md
+++ b/docs/internal/ai-offline-evaluation-reporting.md
@@ -1,7 +1,7 @@
# Offline evaluation result reporting
The evaluation harness in `products/posthog_ai/eval_harness/` reports public-suite scorer results to Braintrust.
-Public suites under `products/posthog_ai/evals/` and `products/slack_app/evals/` also publish those results to PostHog's offline experiment API when configured.
+Suites under `products/posthog_ai/evals/` and the product-owned `products/*/evals/` trees also publish those results to PostHog's offline experiment API when a publishing credential is configured.
The Signals pytest suites reuse the same publisher for their completed results.
Agent and judge trace capture remains available.
Reporting does not run the agent or scorers again.
@@ -15,6 +15,12 @@ Use `hogli evals` to run PostHog AI 2.0 suites. See the [eval harness guide](../
Set `send_to_posthog=True` alongside `offline_suite` to publish results to PostHog while keeping Braintrust uploads disabled.
Set `send_to_posthog=False` to disable PostHog result uploads independently. The same options apply to one-shot suites.
+Suites that disable Braintrust uploads declare `NO_SEND_LOGS = True` at module level and pass `no_send_logs=NO_SEND_LOGS` to their runner.
+Discovery uses this declaration to validate credentials before starting infrastructure; the enrollment test checks that it matches the runner.
+`BRAINTRUST_API_KEY` is required for reporting only when at least one selected suite enables Braintrust uploads.
+PostHog-only suites still use the Braintrust library locally, without authenticating to its experiment service.
+Scorer model credentials remain separate: using Braintrust's model proxy still needs its credential, while other providers use their own keys.
+
Offline API uploads run through the Python SDK independently of `TEST` and `OPT_OUT_CAPTURE`.
The shared trace client retains its existing `TEST` and `OPT_OUT_CAPTURE` guards for `$ai_generation`, `$ai_span`, and `$ai_trace` events.
@@ -41,14 +47,16 @@ A scorer failure marks the case as an error. Statistics still include scores fro
## Configured offline experiment suites
-Configured public PostHog AI and Slack app suites opt into the offline experiment publisher, including both sandboxed and one-shot suites.
-The 11 experiment-skill suites retain `no_send_logs=True` and explicitly set `send_to_posthog=True`, publishing results only to PostHog.
+Shared-harness suites are enrolled in the offline experiment publisher, including both sandboxed and one-shot suites, except for `logs/eval_agent_observability::eval_agent_observability`, which currently reports only to Braintrust.
+The experiment-skill, Feature Flags, Dashboards, turn-suggestion and Business Knowledge retrieval suites retain `no_send_logs=True` and explicitly set `send_to_posthog=True`, publishing results only to PostHog.
+Public suites retain their Braintrust uploads during enrollment. Braintrust also remains the execution engine for every shared-harness suite, including those whose results publish only to PostHog.
After scoring, the harness prepares the completed results in memory and queues delivery through the Python SDK's `AsyncOfflineEvaluations` client. It does not execute another agent run or judge call.
Without an explicit `send_to_posthog` override, `no_send_logs=True` suppresses this upload too.
-The [PostHog AI catalog](../../products/posthog_ai/evals/offline_config.py) and [Slack app catalog](../../products/slack_app/evals/offline_config.py) pin destination project `2`, scorer kinds, immutable version UUIDs, and component tags.
+The [PostHog AI catalog](../../products/posthog_ai/evals/offline_config.py) and each product's `evals/offline_config.py` pin destination project `2`, scorer kinds, immutable version UUIDs, and component tags.
Suites select their own metrics from the catalog; sandboxed suites also include the harness-provided `exit_code_zero` metric.
The SQL suite retains its seven existing scorer versions.
+The discovery-wide enrollment test checks every enrolled suite's publishing flag, stable key, scorer coverage, pinned UUIDs and ownership tags without executing agents or judges. It also checks that the Logs agent-observability suite retains its Braintrust-only reporting configuration.
Provide the API key through the environment before running the suite:
| Variable | Purpose |
@@ -97,6 +105,34 @@ Invalid configuration produces a warning and skips this destination; Braintrust
An upload failure also warns without changing the evaluation's score gate.
The final summary shows a PostHog results link only after a successful offline upload; otherwise it shows the upload status.
+### Testing uploads against local PostHog
+
+Set `POSTHOG_OFFLINE_EVAL_HOST=http://localhost:8010` and use a personal API key created in that local instance.
+HTTP is accepted only for `localhost`, `127.0.0.1`, and `::1`; other hosts require HTTPS.
+The eval harness's temporary test database is separate from the development instance receiving uploads.
+
+The suite catalogs contain production project and scorer-version IDs. To use local scorers without editing those catalogs, set
+`POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG=.env.local.offline-evals.json` in `.env.local`. The JSON file selects a local project and maps
+each catalog scorer-version UUID to a version created in that local project:
+
+```json
+{
+ "project_id": 2,
+ "scorer_versions": {
+ "00000000-0000-4000-8000-000000000002": "00000000-0000-4000-8000-000000000003"
+ }
+}
+```
+
+The UUIDs above are placeholders. Create the needed scorers under **Evaluations → Scorers**, matching each catalog metric's kind,
+and use their immutable version IDs. Include every scorer used by the selected suites; a single file can cover multiple suites.
+The file is gitignored by the `.env*` rule. Local configuration is accepted only with a loopback upload host.
+An unreadable, invalid, or incomplete file disables the upload with an error; it never falls back to the production catalog.
+
+Run through `hogli evals` to load `.env.local`; existing shell variables take precedence.
+For PostHog-only testing, select suites declaring `NO_SEND_LOGS = True`, such as Feature Flags or Dashboards.
+Set `OPT_OUT_CAPTURE=1` to suppress ordinary SDK/trace capture while retaining explicit offline-result uploads.
+
### Scorer and experiment tags
Score definitions and experiment runs use the same team and component tags:
@@ -105,6 +141,12 @@ Score definitions and experiment runs use the same team and component tags:
| -------------------- | ------------------------------------------------------------------------------------------- |
| PostHog AI | `team:surfaces`, `component:posthog-ai/<domain>` (for example `sql` or `product-analytics`) |
| Slack app | `team:surfaces`, `component:slack-app/<area>` (for example `followup-classifier`) |
+| Conversations | `team:conversations`, `component:conversations` |
+| Business Knowledge | `team:conversations`, `component:business-knowledge` |
+| Data Catalog | `team:data-modeling`, `component:data-catalog` |
+| Notebooks | `team:data-tools`, `component:notebooks` |
+| Feature Flags | `team:feature-flags`, `component:feature-flags` |
+| Dashboards | `team:product-analytics`, `component:dashboards` |
| Signals grouping | `team:self-driving`, `component:signals/grouping` |
| Signals scout safety | `team:self-driving`, `component:signals/scout-safety` |
@@ -114,6 +156,34 @@ To upload **and tag** experiments, `POSTHOG_OFFLINE_EVAL_API_KEY` must be a pers
Each prepared experiment's tag update is attempted once per publisher, including when delivery pauses and resumes. A failed or interrupted replacement is not repeated automatically, because it may already have succeeded or someone may have edited the tags afterward. Inspect the experiment's tags before manually retrying a replacement.
+### Additional one-shot suites
+
+Run these through the same harness and publishing lifecycle:
+
+```bash
+hogli evals turn_suggestions/eval_turn_suggestions::eval_turn_suggestions
+hogli evals business_knowledge/eval_retrieval::eval_retrieval
+hogli evals conversations/eval_support_reply::eval_support_reply
+```
+
+Turn suggestions reuse the labeled benchmark cases and the configured System One judge, once per case.
+`TURN_SUGGESTIONS_EVAL_THRESHOLD` overrides the production show threshold for the run; it must be between 0 and 1.
+Case metadata records the threshold and requested judge model, and outputs retain the selected offer, probability, outcome and duration.
+An accepted optional offer counts as correct; an unavailable judgment reports an error.
+The existing `turn_suggestions_benchmark` command remains available for multi-endpoint comparisons and threshold sweeps; those modes are not part of this suite.
+
+Business Knowledge retrieval reuses the labeled `retrieval_eval.yaml` corpus and queries.
+Each case seeds an isolated team, searches the corpus and cleans up the team.
+Its boolean score checks the existing found/not-found expectation, including queries expected to return no results.
+It does not score source relevance or replace the ordinary retrieval unit tests.
+
+Support reply retains its existing behavior: mocked by default, or live when `SUPPORT_REPLY_EVAL_LIVE=1`.
+Both case metadata and output identify the mode so runs can be compared within the same mode.
+
+These suites use the existing one-shot infrastructure and credential prerequisites described in the harness guide.
+The turn-suggestion judge additionally needs its existing System One configuration; selecting the suite does not configure that service.
+MCP saved-report grading, semantic-layer canary commands and ReviewHog campaigns retain their separate workflows and are not enrolled by these wrappers.
+
### CI configuration
CI jobs running enrolled suites need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment.
@@ -160,7 +230,7 @@ Existing local logs, enabled case exports, and Braintrust results remain availab
Both `SandboxedEval` and `OneShotEval` accept `offline_suite=OfflineEvalSuite(...)`.
Each suite declares its destination `project_id`, selects at least one metric in `scorer_kinds`, and pins the corresponding immutable version UUIDs in `scorer_versions`.
These fields belong to `OfflineEvalSuite` in code. Scorer version UUIDs must be distinct for the suite's enrolled metrics.
-The AI and Slack catalog helpers add `exit_code_zero` for sandboxed suites. One-shot suites pass `suite_kind=SUITE_KIND` to enroll only their declared metrics.
+Catalog helpers add `exit_code_zero` for sandboxed suites. One-shot suites pass `suite_kind=SUITE_KIND` to enroll only their declared metrics.
Numeric scores can use the domain of their configured scorer version. The API validates bounds and steps; the harness does not restrict every numeric metric to `[0, 1]`.
`OfflineEvalSuite.experiment_options` accepts the SDK's `ExperimentOptions` for external dataset identifiers, `prompt_version`, model/application versions, and `run_source`.
diff --git a/products/business_knowledge/evals/eval_retrieval.py b/products/business_knowledge/evals/eval_retrieval.py
new file mode 100644
index 000000000000..7ead451fa2d4
--- /dev/null
+++ b/products/business_knowledge/evals/eval_retrieval.py
@@ -0,0 +1,101 @@
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+from typing import TYPE_CHECKING, TypedDict
+
+import yaml
+
+from products.business_knowledge.backend.logic import create_text_source, search_knowledge
+from products.business_knowledge.backend.models import KnowledgeDocument, SafetyVerdict
+from products.business_knowledge.evals.offline_config import offline_suite
+from products.conversations.evals.seeders import provision_eval_team, teardown_eval_team
+from products.posthog_ai.eval_harness.config import BaseEvalCase
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.one_shot import OneShotPrivateEval
+from products.posthog_ai.eval_harness.scorers.contract import Score, Scorer
+
+if TYPE_CHECKING:
+ from products.posthog_ai.eval_harness.harness.context import EvalContext
+
+NO_SEND_LOGS = True
+
+SUITE_KIND = SuiteKind.ONE_SHOT
+
+
+class RetrievalQuery(TypedDict):
+ query: str
+ expect_found: bool
+
+
+class RetrievalSource(TypedDict):
+ source_name: str
+ text: str
+ queries: list[RetrievalQuery]
+
+
+CORPUS: list[RetrievalSource] = yaml.safe_load(
+ (Path(__file__).parents[1] / "backend/tests/fixtures/retrieval_eval.yaml").read_text()
+)
+CASES = [
+ BaseEvalCase(
+ name=f"{entry['source_name']}|{query['query']}",
+ prompt=query["query"],
+ expected={"retrieval_match": {"found": query["expect_found"]}},
+ metadata={"source_name": entry["source_name"]},
+ )
+ for entry in CORPUS
+ for query in entry["queries"]
+]
+
+
+class RetrievalMatch(Scorer):
+ def _name(self) -> str:
+ return "retrieval_match"
+
+ def _run_eval_sync(
+ self, output: dict[str, object] | None, expected: dict[str, object] | None = None, **kwargs: object
+ ) -> Score:
+ expectation = (expected or {}).get(self._name())
+ if output is None or not isinstance(expectation, dict):
+ return Score(name=self._name(), score=None, metadata={"reason": "Retrieval did not complete"})
+ return Score(
+ name=self._name(),
+ score=float(output["found"] == expectation["found"]),
+ metadata={"source_names": output["source_names"]},
+ )
+
+
+def search_case(case: BaseEvalCase) -> dict[str, object]:
+ eval_team = provision_eval_team(label="knowledge-retrieval")
+ try:
+ for entry in CORPUS:
+ create_text_source(
+ team_id=eval_team.team.id,
+ created_by_id=eval_team.user.id,
+ name=entry["source_name"],
+ text=entry["text"],
+ )
+ KnowledgeDocument.objects.for_team(eval_team.team.id).update(safety_verdict=SafetyVerdict.SAFE)
+ results = search_knowledge(eval_team.team.id, case.prompt)
+ return {"found": bool(results), "source_names": sorted({result.source_name for result in results})}
+ finally:
+ teardown_eval_team(eval_team=eval_team)
+
+
+async def eval_retrieval(ctx: EvalContext) -> None:
+ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict[str, object]:
+ return await asyncio.to_thread(search_case, case)
+
+ await OneShotPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
+ experiment_name="business-knowledge-retrieval",
+ cases=CASES,
+ task=task,
+ scorers=[RetrievalMatch()],
+ offline_suite=offline_suite(
+ "business_knowledge/eval_retrieval::eval_retrieval", metrics=["retrieval_match"], suite_kind=SUITE_KIND
+ ),
+ send_to_posthog=True,
+ ctx=ctx,
+ )
diff --git a/products/business_knowledge/evals/offline_config.py b/products/business_knowledge/evals/offline_config.py
new file mode 100644
index 000000000000..39882e35c8bf
--- /dev/null
+++ b/products/business_knowledge/evals/offline_config.py
@@ -0,0 +1,19 @@
+from collections.abc import Sequence
+from uuid import UUID
+
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer
+
+_SCORERS: dict[str, OfflineScorer] = {
+ "retrieval_match": OfflineScorer(kind="boolean", version=UUID("01a15a29-f71a-7bdd-9a8c-c018881f9702")),
+}
+
+
+def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:
+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics
+ return OfflineEvalSuite.from_scorers(
+ key=key,
+ project_id=2,
+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},
+ tags=("team:conversations", "component:business-knowledge"),
+ )
diff --git a/products/conversations/evals/eval_support_reply.py b/products/conversations/evals/eval_support_reply.py
index 316ae6eb808b..f615787ce198 100644
--- a/products/conversations/evals/eval_support_reply.py
+++ b/products/conversations/evals/eval_support_reply.py
@@ -20,6 +20,7 @@ import asyncio
from products.conversations.evals.constants import LIVE_EVAL_ENV_VAR
from products.conversations.evals.fixtures import FIXTURES, FIXTURES_BY_NAME, expected_for
+from products.conversations.evals.offline_config import offline_suite
from products.conversations.evals.runner import live_eval_enabled, run_fixture
from products.conversations.evals.scorers import ALL_SCORERS
from products.conversations.evals.seeders import provision_eval_team, seed_case, teardown_eval_team
@@ -48,9 +49,10 @@ CASES = [
async def eval_support_reply(ctx: EvalContext) -> None:
+ live = live_eval_enabled()
+
async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict:
fixture = FIXTURES_BY_NAME[case.name]
- live = live_eval_enabled()
eval_team = await asyncio.to_thread(lambda: provision_eval_team(label=fixture.name))
try:
seed = await asyncio.to_thread(lambda: seed_case(eval_team=eval_team, fixture=fixture))
@@ -60,7 +62,22 @@ async def eval_support_reply(ctx: EvalContext) -> None:
await OneShotPublicEval(
experiment_name="support-reply-pipeline",
- cases=CASES,
+ offline_suite=offline_suite(
+ "conversations/eval_support_reply::eval_support_reply",
+ metrics=[
+ "outcome_match",
+ "citation_precision",
+ "forbidden_claims",
+ "cost",
+ "grounding",
+ "clarifying_question",
+ ],
+ suite_kind=SUITE_KIND,
+ ),
+ cases=[
+ case.model_copy(update={"metadata": {**case.metadata, "mode": "live" if live else "mocked"}})
+ for case in CASES
+ ],
scorers=list(ALL_SCORERS),
task=task,
ctx=ctx,
diff --git a/products/conversations/evals/offline_config.py b/products/conversations/evals/offline_config.py
new file mode 100644
index 000000000000..09638c9bcd7d
--- /dev/null
+++ b/products/conversations/evals/offline_config.py
@@ -0,0 +1,24 @@
+from collections.abc import Sequence
+from uuid import UUID
+
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer
+
+_SCORERS: dict[str, OfflineScorer] = {
+ "citation_precision": OfflineScorer(kind="numeric", version=UUID("01a15a27-9528-7df7-89dc-717fdb43d339")),
+ "clarifying_question": OfflineScorer(kind="boolean", version=UUID("01a15a27-ba72-7483-8369-8a46861c680f")),
+ "cost": OfflineScorer(kind="boolean", version=UUID("01a15a27-a827-7494-a247-daf97f55b7fd")),
+ "forbidden_claims": OfflineScorer(kind="boolean", version=UUID("01a15a27-9d10-7260-ace1-335661c5bfdc")),
+ "grounding": OfflineScorer(kind="boolean", version=UUID("01a15a27-b117-7e00-aefd-30eb5f5b7e4a")),
+ "outcome_match": OfflineScorer(kind="boolean", version=UUID("01a15a27-8bab-7c0e-8a33-d9ac6327874e")),
+}
+
+
+def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:
+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics
+ return OfflineEvalSuite.from_scorers(
+ key=key,
+ project_id=2,
+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},
+ tags=("team:conversations", "component:conversations"),
+ )
diff --git a/products/dashboards/evals/eval_dashboard_contents.py b/products/dashboards/evals/eval_dashboard_contents.py
index 954bd4bd45bb..2c3d34e87c5d 100644
--- a/products/dashboards/evals/eval_dashboard_contents.py
+++ b/products/dashboards/evals/eval_dashboard_contents.py
@@ -6,6 +6,7 @@ from posthog.utils import generate_short_id
from products.dashboards.backend.models.dashboard import Dashboard
from products.dashboards.backend.models.dashboard_tile import DashboardTile
+from products.dashboards.evals.offline_config import offline_suite
from products.dashboards.evals.scorers import SavedDashboardContents
from products.posthog_ai.eval_harness.base import SandboxedPrivateEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
@@ -14,6 +15,8 @@ from products.product_analytics.backend.facade.api import get_or_create_saved_in
from products.product_analytics.backend.facade.models import Insight
from products.tasks.backend.facade.agents import CustomPromptSandboxContext
+NO_SEND_LOGS = True
+
DASHBOARD_NAME = "File activity overview"
SIGNUPS_NAME = "Saved daily signups"
UPLOADS_NAME = "Saved daily file uploads"
@@ -67,7 +70,13 @@ def seed_existing_dashboard(context: CustomPromptSandboxContext) -> dict[str, An
async def eval_dashboard_contents(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-dashboards-saved-contents-cli",
+ offline_suite=offline_suite(
+ "dashboards/eval_dashboard_contents::eval_dashboard_contents",
+ metrics=["saved_dashboard_contents"],
+ ),
+ send_to_posthog=True,
cases=[
SandboxedEvalCase(
name="create_from_saved_insights",
diff --git a/products/dashboards/evals/eval_dashboard_sql_date_range.py b/products/dashboards/evals/eval_dashboard_sql_date_range.py
index bfc961ebb346..0e7d661c87b2 100644
--- a/products/dashboards/evals/eval_dashboard_sql_date_range.py
+++ b/products/dashboards/evals/eval_dashboard_sql_date_range.py
@@ -3,12 +3,15 @@ from __future__ import annotations
from typing import Any
from products.dashboards.backend.models.dashboard import Dashboard
+from products.dashboards.evals.offline_config import offline_suite
from products.dashboards.evals.scorers import SQLTilesFollowDashboardDateRange
from products.posthog_ai.eval_harness.base import SandboxedPrivateEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.tasks.backend.facade.agents import CustomPromptSandboxContext
+NO_SEND_LOGS = True
+
SIGNUPS_DASHBOARD_NAME = "Signups and uploads (SQL)"
BILLING_DASHBOARD_NAME = "Billing and downloads (SQL)"
@@ -22,7 +25,13 @@ def seed_team(context: CustomPromptSandboxContext) -> dict[str, Any]:
async def eval_dashboard_sql_date_range(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-dashboards-sql-date-range-cli",
+ offline_suite=offline_suite(
+ "dashboards/eval_dashboard_sql_date_range::eval_dashboard_sql_date_range",
+ metrics=["sql_tiles_follow_dashboard_date_range"],
+ ),
+ send_to_posthog=True,
cases=[
SandboxedEvalCase(
name="sql_tiles_without_date_hint",
diff --git a/products/dashboards/evals/offline_config.py b/products/dashboards/evals/offline_config.py
new file mode 100644
index 000000000000..a6b132274628
--- /dev/null
+++ b/products/dashboards/evals/offline_config.py
@@ -0,0 +1,23 @@
+from collections.abc import Sequence
+from uuid import UUID
+
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer
+
+_SCORERS: dict[str, OfflineScorer] = {
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15a27-ce66-7f1a-babd-0e1423882e9b")),
+ "saved_dashboard_contents": OfflineScorer(kind="boolean", version=UUID("01a15a27-c451-7557-9bca-c968f6358a91")),
+ "sql_tiles_follow_dashboard_date_range": OfflineScorer(
+ kind="numeric", version=UUID("01a15a27-d794-74b3-8822-063a74a674fd")
+ ),
+}
+
+
+def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:
+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics
+ return OfflineEvalSuite.from_scorers(
+ key=key,
+ project_id=2,
+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},
+ tags=("team:product-analytics", "component:dashboards"),
+ )
diff --git a/products/data_catalog/evals/eval_governed_metrics.py b/products/data_catalog/evals/eval_governed_metrics.py
index 7f8e53442430..64d69540c222 100644
--- a/products/data_catalog/evals/eval_governed_metrics.py
+++ b/products/data_catalog/evals/eval_governed_metrics.py
@@ -69,6 +69,7 @@ from products.data_catalog.evals.constants import (
WEBSITE_404_HITS_DAILY_METRIC_NAME,
YOY_MRR_GROWTH_METRIC_NAME,
)
+from products.data_catalog.evals.offline_config import offline_suite
from products.data_catalog.evals.scorers import (
CanonicalMetricRun,
GovernedBehaviorCorrectness,
@@ -691,6 +692,20 @@ async def eval_governed_metrics(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-governed-metrics-cli",
+ offline_suite=offline_suite(
+ "data_catalog/eval_governed_metrics::eval_governed_metrics",
+ metrics=[
+ "metrics_catalog_queried",
+ "metrics_catalog_before_answer",
+ "metrics_catalog_before_data_discovery",
+ "canonical_metric_run",
+ "metrics_catalog_not_queried",
+ "governed_behavior_correctness",
+ "trust_badge",
+ "proposed_badge",
+ "no_metric_write",
+ ],
+ ),
cases=cases,
scorers=[
MetricsCatalogQueried(),
diff --git a/products/data_catalog/evals/eval_metric_authoring.py b/products/data_catalog/evals/eval_metric_authoring.py
index 83d27f4e434e..31692facb282 100644
--- a/products/data_catalog/evals/eval_metric_authoring.py
+++ b/products/data_catalog/evals/eval_metric_authoring.py
@@ -16,6 +16,7 @@ To run a single case::
from __future__ import annotations
from products.data_catalog.evals.constants import METRIC_CREATE_TOOL
+from products.data_catalog.evals.offline_config import offline_suite
from products.data_catalog.evals.scorers import MetricDescriptionConcise, MetricDescriptionQuality
from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
@@ -54,6 +55,10 @@ async def eval_metric_authoring(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-metric-authoring-cli",
+ offline_suite=offline_suite(
+ "data_catalog/eval_metric_authoring::eval_metric_authoring",
+ metrics=["metric_create_called", "metric_description_concise", "metric_description_quality"],
+ ),
cases=cases,
scorers=[
RequiredToolCall([METRIC_CREATE_TOOL], name="metric_create_called"),
diff --git a/products/data_catalog/evals/eval_semantic_discovery.py b/products/data_catalog/evals/eval_semantic_discovery.py
index bf5b0d6d273b..65ef00f6e6ae 100644
--- a/products/data_catalog/evals/eval_semantic_discovery.py
+++ b/products/data_catalog/evals/eval_semantic_discovery.py
@@ -21,6 +21,7 @@ from products.data_catalog.evals.constants import (
RELATIONSHIP_SOURCE_NAME,
RELATIONSHIP_TARGET_KEY,
)
+from products.data_catalog.evals.offline_config import offline_suite
from products.data_catalog.evals.scorers import (
DeprecationProposed,
SemanticMetadataQueried,
@@ -118,6 +119,10 @@ async def eval_semantic_discovery(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-semantic-discovery-cli",
+ offline_suite=offline_suite(
+ "data_catalog/eval_semantic_discovery::eval_semantic_discovery",
+ metrics=["semantic_metadata_queried", "semantic_trust_decision_correctness", "deprecation_proposed"],
+ ),
cases=cases,
scorers=[SemanticMetadataQueried(), SemanticTrustDecisionCorrectness(), DeprecationProposed()],
ctx=ctx,
diff --git a/products/data_catalog/evals/offline_config.py b/products/data_catalog/evals/offline_config.py
new file mode 100644
index 000000000000..901d03618bb8
--- /dev/null
+++ b/products/data_catalog/evals/offline_config.py
@@ -0,0 +1,42 @@
+from collections.abc import Sequence
+from uuid import UUID
+
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer
+
+_SCORERS: dict[str, OfflineScorer] = {
+ "canonical_metric_run": OfflineScorer(kind="boolean", version=UUID("01a15a27-fa96-71e0-b904-13f14079440e")),
+ "deprecation_proposed": OfflineScorer(kind="boolean", version=UUID("01a15a28-7a51-7732-838a-34de17d02bbf")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15a28-3e1d-7d93-86bb-de7f4e4a5c63")),
+ "governed_behavior_correctness": OfflineScorer(
+ kind="numeric", version=UUID("01a15a28-0f80-7d6e-b0bb-0573fcb583fa")
+ ),
+ "metric_create_called": OfflineScorer(kind="boolean", version=UUID("01a15a28-47d2-7eea-ba8a-55266556d782")),
+ "metric_description_concise": OfflineScorer(kind="boolean", version=UUID("01a15a28-5017-75a6-b5e5-5ddfe270c02a")),
+ "metric_description_quality": OfflineScorer(kind="boolean", version=UUID("01a15a28-5aa6-746d-8d30-bc84b857f15a")),
+ "metrics_catalog_before_answer": OfflineScorer(
+ kind="boolean", version=UUID("01a15a27-e92c-792a-ab48-24162ed407e9")
+ ),
+ "metrics_catalog_before_data_discovery": OfflineScorer(
+ kind="boolean", version=UUID("01a15a27-f18f-76e9-a67e-0259133b733c")
+ ),
+ "metrics_catalog_not_queried": OfflineScorer(kind="boolean", version=UUID("01a15a28-0586-7594-9cd5-3f1ff8ca3c06")),
+ "metrics_catalog_queried": OfflineScorer(kind="boolean", version=UUID("01a15a27-e0c4-74f7-ae0c-acda47b99ab3")),
+ "no_metric_write": OfflineScorer(kind="boolean", version=UUID("01a15a28-30cb-7118-b1c8-9084eba7378e")),
+ "proposed_badge": OfflineScorer(kind="boolean", version=UUID("01a15a28-206d-79ea-b231-4bf05e131c80")),
+ "semantic_metadata_queried": OfflineScorer(kind="boolean", version=UUID("01a15a28-64d0-787c-af9e-94c4730b8a1c")),
+ "semantic_trust_decision_correctness": OfflineScorer(
+ kind="numeric", version=UUID("01a15a28-6ded-74ed-a08c-4bfc378b0b5b")
+ ),
+ "trust_badge": OfflineScorer(kind="boolean", version=UUID("01a15a28-17ed-7638-82e4-d39bd349ce9e")),
+}
+
+
+def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:
+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics
+ return OfflineEvalSuite.from_scorers(
+ key=key,
+ project_id=2,
+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},
+ tags=("team:data-modeling", "component:data-catalog"),
+ )
diff --git a/products/feature_flags/evals/eval_cleanup_stale_flags.py b/products/feature_flags/evals/eval_cleanup_stale_flags.py
index 567ac5c15e55..19e7c8ae7cc1 100644
--- a/products/feature_flags/evals/eval_cleanup_stale_flags.py
+++ b/products/feature_flags/evals/eval_cleanup_stale_flags.py
@@ -60,6 +60,7 @@ To run:
from __future__ import annotations
+from products.feature_flags.evals.offline_config import offline_suite
from products.feature_flags.evals.scorers import (
ASSESSMENT_ONLY_NO_EDIT_CLAIM_QUESTION,
DEPENDENTS_READ_TOOLS,
@@ -88,6 +89,8 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import NoToolCall
from products.posthog_ai.evals.retrieval.scorers import SkillTriggered
+NO_SEND_LOGS = True
+
SKILL_NAME = "cleaning-up-stale-feature-flags"
TRIGGER_SCORER_NAME = "cleanup_skill_triggered"
EDIT_SCORER_NAME = "code_edit_direction"
@@ -226,7 +229,24 @@ async def eval_cleanup_stale_flags(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-feature-flags-cleanup-stale-cli",
+ offline_suite=offline_suite(
+ "feature_flags/eval_cleanup_stale_flags::eval_cleanup_stale_flags",
+ metrics=[
+ "cleanup_skill_triggered",
+ "no_flag_mutation",
+ "flag_state_unchanged",
+ "code_edit_direction",
+ "flag_lookup_direction",
+ "dependents_check_direction",
+ "schedule_check_direction",
+ "recency_refusal_without_override",
+ "tour_unknown_waits",
+ "assessment_only_no_edit_claim",
+ ],
+ ),
+ send_to_posthog=True,
cases=cases,
scorers=[
SkillTriggered(SKILL_NAME, name=TRIGGER_SCORER_NAME),
diff --git a/products/feature_flags/evals/eval_flag_answers.py b/products/feature_flags/evals/eval_flag_answers.py
index 4d958c995a12..f0ad24f6518d 100644
--- a/products/feature_flags/evals/eval_flag_answers.py
+++ b/products/feature_flags/evals/eval_flag_answers.py
@@ -23,6 +23,7 @@ from __future__ import annotations
import json
from pathlib import Path
+from products.feature_flags.evals.offline_config import offline_suite
from products.feature_flags.evals.scorers import (
EXPLAINED_TAG_REQUIREMENT_QUESTION,
REFUSED_WITHOUT_BLAMING_QUESTION,
@@ -47,6 +48,8 @@ from products.posthog_ai.eval_harness.base import SandboxedPrivateEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
+NO_SEND_LOGS = True
+
_TOOL_DEFINITIONS = Path(__file__).parents[3] / "services/mcp/schema/generated-tool-definitions.json"
FLAG_TOOL_CATEGORY = "Feature flags"
@@ -111,7 +114,22 @@ async def eval_flag_answers(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-feature-flags-answers-cli",
+ offline_suite=offline_suite(
+ "feature_flags/eval_flag_answers::eval_flag_answers",
+ metrics=[
+ "called_expected_tool",
+ "attempted_tool",
+ "avoided_tool",
+ "created_flag_with_tags",
+ "final_message_names",
+ "stale_is_not_safe_to_remove",
+ "refused_without_blaming",
+ "explained_tag_requirement",
+ ],
+ ),
+ send_to_posthog=True,
cases=cases,
scorers=[
CalledExpectedTool(),
diff --git a/products/feature_flags/evals/eval_flag_creation.py b/products/feature_flags/evals/eval_flag_creation.py
index 3c00dc70659e..cd0095cf6d81 100644
--- a/products/feature_flags/evals/eval_flag_creation.py
+++ b/products/feature_flags/evals/eval_flag_creation.py
@@ -4,11 +4,14 @@ from typing import Any
from products.feature_flags.backend.models.feature_flag import FeatureFlag
from products.feature_flags.evals.creation_scorers import CreatedFlagConfiguration
+from products.feature_flags.evals.offline_config import offline_suite
from products.posthog_ai.eval_harness.base import SandboxedPrivateEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.tasks.backend.facade.agents import CustomPromptSandboxContext
+NO_SEND_LOGS = True
+
FLAG_KEY = "bulk-file-export-preview"
@@ -20,7 +23,13 @@ def seed_flag_creation(context: CustomPromptSandboxContext) -> dict[str, Any]:
async def eval_flag_creation(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-feature-flags-creation-cli",
+ offline_suite=offline_suite(
+ "feature_flags/eval_flag_creation::eval_flag_creation",
+ metrics=["created_flag_configuration"],
+ ),
+ send_to_posthog=True,
cases=[
SandboxedEvalCase(
name="boolean_flag_with_targeted_rollout",
diff --git a/products/feature_flags/evals/eval_flag_tool_selection.py b/products/feature_flags/evals/eval_flag_tool_selection.py
index 205c60aa07e9..098cc7f985b8 100644
--- a/products/feature_flags/evals/eval_flag_tool_selection.py
+++ b/products/feature_flags/evals/eval_flag_tool_selection.py
@@ -28,6 +28,7 @@ To run:
from __future__ import annotations
+from products.feature_flags.evals.offline_config import offline_suite
from products.feature_flags.evals.scorers import (
EXPLAINED_KEY_REUSE_QUESTION,
AvoidedTool,
@@ -58,6 +59,8 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import RequiredToolCall
+NO_SEND_LOGS = True
+
# The rename case asks for these two values, so the prompt and the scorer read the same
# constants: a prompt that changed without the expectation would grade the old request.
METADATA_RENAMED_KEY = "file-preview-tiles"
@@ -140,7 +143,22 @@ async def eval_flag_tool_selection(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-feature-flags-tool-selection-cli",
+ offline_suite=offline_suite(
+ "feature_flags/eval_flag_tool_selection::eval_flag_tool_selection",
+ metrics=[
+ "resolved_flag_by_key",
+ "called_expected_tool",
+ "avoided_tool",
+ "generic_update_omits_fields",
+ "generic_update_sets_fields",
+ "preserved_unrelated_config",
+ "updated_rollout_to",
+ "explained_key_reuse",
+ ],
+ ),
+ send_to_posthog=True,
cases=cases,
scorers=[
# Every prompt names its flag by key, and the write tools take a numeric id.
diff --git a/products/feature_flags/evals/eval_instrument_flags.py b/products/feature_flags/evals/eval_instrument_flags.py
index af2c91c35e06..97478918dd43 100644
--- a/products/feature_flags/evals/eval_instrument_flags.py
+++ b/products/feature_flags/evals/eval_instrument_flags.py
@@ -53,6 +53,7 @@ from typing import Any
from django.conf import settings
+from products.feature_flags.evals.offline_config import offline_suite
from products.feature_flags.evals.scorers import FILE_EDIT_TOOLS
from products.posthog_ai.eval_harness.base import SandboxedPrivateEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
@@ -61,6 +62,8 @@ from products.posthog_ai.eval_harness.scorers import RequiredToolCall
from products.posthog_ai.evals.retrieval.scorers import SkillTriggered
from products.tasks.backend.facade.agents import CustomPromptSandboxContext
+NO_SEND_LOGS = True
+
SKILL_NAME = "instrument-feature-flags"
SCORER_NAME = "instrument_flags_skill_triggered"
@@ -144,7 +147,13 @@ async def eval_instrument_flags(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-feature-flags-instrument-trigger-cli",
+ offline_suite=offline_suite(
+ "feature_flags/eval_instrument_flags::eval_instrument_flags",
+ metrics=["instrument_flags_skill_triggered", "edited_a_file"],
+ ),
+ send_to_posthog=True,
cases=cases,
scorers=[
SkillTriggered(SKILL_NAME, name=SCORER_NAME),
diff --git a/products/feature_flags/evals/offline_config.py b/products/feature_flags/evals/offline_config.py
new file mode 100644
index 000000000000..4d1e679cdd0d
--- /dev/null
+++ b/products/feature_flags/evals/offline_config.py
@@ -0,0 +1,52 @@
+from collections.abc import Sequence
+from uuid import UUID
+
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer
+
+_SCORERS: dict[str, OfflineScorer] = {
+ "assessment_only_no_edit_claim": OfflineScorer(
+ kind="boolean", version=UUID("01a15a28-f08d-773f-ab1a-88323b444706")
+ ),
+ "attempted_tool": OfflineScorer(kind="boolean", version=UUID("01a15a29-15e7-7a4d-bc3e-cbc2a6d7fde5")),
+ "avoided_tool": OfflineScorer(kind="boolean", version=UUID("01a15a29-229f-7a8c-96ad-e2e4070f634e")),
+ "called_expected_tool": OfflineScorer(kind="boolean", version=UUID("01a15a29-0a13-76c8-9fa1-620c98110555")),
+ "cleanup_skill_triggered": OfflineScorer(kind="boolean", version=UUID("01a15a28-8366-71a4-9790-39b96dec786e")),
+ "code_edit_direction": OfflineScorer(kind="boolean", version=UUID("01a15a28-a1c4-71a9-8d49-14fecb1fcf6e")),
+ "created_flag_configuration": OfflineScorer(kind="boolean", version=UUID("01a15a29-7bfc-7ed2-9b70-3efd6f845ef2")),
+ "created_flag_with_tags": OfflineScorer(kind="boolean", version=UUID("01a15a29-2de7-7a7d-b0a2-b7cf40ca00df")),
+ "dependents_check_direction": OfflineScorer(kind="boolean", version=UUID("01a15a28-b815-7ce3-b591-524d66fd33b4")),
+ "edited_a_file": OfflineScorer(kind="boolean", version=UUID("01a15a29-e150-7307-aaa9-e9228706f89e")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15a28-fcc8-71df-b886-752d5b82a419")),
+ "explained_key_reuse": OfflineScorer(kind="boolean", version=UUID("01a15a29-c36a-7bef-95ad-b5151cf2bfb9")),
+ "explained_tag_requirement": OfflineScorer(kind="boolean", version=UUID("01a15a29-70ca-7bb0-86bf-6430216b3ac7")),
+ "final_message_names": OfflineScorer(kind="boolean", version=UUID("01a15a29-38b0-71c0-bfd9-419cd39d6847")),
+ "flag_lookup_direction": OfflineScorer(kind="boolean", version=UUID("01a15a28-ae91-7185-9bce-80cd868d7c50")),
+ "flag_state_unchanged": OfflineScorer(kind="boolean", version=UUID("01a15a28-9755-7c4a-8139-56ed6d942103")),
+ "generic_update_omits_fields": OfflineScorer(kind="boolean", version=UUID("01a15a29-9766-741f-8775-fff77c2c81ff")),
+ "generic_update_sets_fields": OfflineScorer(kind="boolean", version=UUID("01a15a29-a338-7a79-8b29-acd7a2aec492")),
+ "instrument_flags_skill_triggered": OfflineScorer(
+ kind="boolean", version=UUID("01a15a29-cf2e-74a8-a6af-a66281cd8637")
+ ),
+ "no_flag_mutation": OfflineScorer(kind="boolean", version=UUID("01a15a28-8cb1-7485-8766-e631517451fd")),
+ "preserved_unrelated_config": OfflineScorer(kind="boolean", version=UUID("01a15a29-af51-7ae8-94e3-115000b1e2bb")),
+ "recency_refusal_without_override": OfflineScorer(
+ kind="boolean", version=UUID("01a15a28-d41d-7793-818b-ca478ebf5ba3")
+ ),
+ "refused_without_blaming": OfflineScorer(kind="boolean", version=UUID("01a15a29-583d-7c75-80d5-cff77fd3047b")),
+ "resolved_flag_by_key": OfflineScorer(kind="boolean", version=UUID("01a15a29-8600-7983-b104-6c30aa8ab6bd")),
+ "schedule_check_direction": OfflineScorer(kind="boolean", version=UUID("01a15a28-c70e-7c31-a7ff-2ad25d23d410")),
+ "stale_is_not_safe_to_remove": OfflineScorer(kind="boolean", version=UUID("01a15a29-47b0-7e3d-b7f4-d1e6775c9ff2")),
+ "tour_unknown_waits": OfflineScorer(kind="boolean", version=UUID("01a15a28-e252-76a5-8945-0321df22e8fa")),
+ "updated_rollout_to": OfflineScorer(kind="boolean", version=UUID("01a15a29-b8a7-7873-aada-8d3d0f396eb4")),
+}
+
+
+def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:
+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics
+ return OfflineEvalSuite.from_scorers(
+ key=key,
+ project_id=2,
+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},
+ tags=("team:feature-flags", "component:feature-flags"),
+ )
diff --git a/products/notebooks/evals/eval_notebook_analysis.py b/products/notebooks/evals/eval_notebook_analysis.py
index 826020797bb8..19d274892c58 100644
--- a/products/notebooks/evals/eval_notebook_analysis.py
+++ b/products/notebooks/evals/eval_notebook_analysis.py
@@ -26,6 +26,7 @@ Because these cases need a completed Python run, run them on docker:
from __future__ import annotations
+from products.notebooks.evals.offline_config import offline_suite
from products.notebooks.evals.scorers import (
CellRunsCompleted,
ChurnCohortSurfaced,
@@ -125,6 +126,16 @@ async def eval_notebook_analysis(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-notebook-analysis-cli",
+ offline_suite=offline_suite(
+ "notebooks/eval_notebook_analysis::eval_notebook_analysis",
+ metrics=[
+ "required_tool_call",
+ "notebook_created",
+ "cell_runs_completed",
+ "churn_cohort_surfaced",
+ "notebook_approach_quality",
+ ],
+ ),
cases=cases,
scorers=[
RequiredToolCall({"notebooks-add-cell"}),
diff --git a/products/notebooks/evals/eval_notebook_cells.py b/products/notebooks/evals/eval_notebook_cells.py
index 0b4bbf43b202..fec2cd291701 100644
--- a/products/notebooks/evals/eval_notebook_cells.py
+++ b/products/notebooks/evals/eval_notebook_cells.py
@@ -10,6 +10,7 @@ To run one case:
from __future__ import annotations
+from products.notebooks.evals.offline_config import offline_suite
from products.notebooks.evals.scorers import CellRunsCompleted, NotebookCreated, NotebookRunCompleted
from products.notebooks.evals.seeders import seed_case_team
from products.posthog_ai.eval_harness.base import SandboxedPublicEval
@@ -86,6 +87,10 @@ async def eval_notebook_cells(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-notebook-cells-cli",
+ offline_suite=offline_suite(
+ "notebooks/eval_notebook_cells::eval_notebook_cells",
+ metrics=["required_tool_call", "notebook_created", "cell_runs_completed", "notebook_run_completed"],
+ ),
cases=cases,
scorers=[
RequiredToolCall({"notebooks-add-cell"}),
diff --git a/products/notebooks/evals/offline_config.py b/products/notebooks/evals/offline_config.py
new file mode 100644
index 000000000000..1d1f3d28c60c
--- /dev/null
+++ b/products/notebooks/evals/offline_config.py
@@ -0,0 +1,25 @@
+from collections.abc import Sequence
+from uuid import UUID
+
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer
+
+_SCORERS: dict[str, OfflineScorer] = {
+ "cell_runs_completed": OfflineScorer(kind="boolean", version=UUID("01a143b7-7d19-7d4a-8fc8-3d06a32da5ef")),
+ "churn_cohort_surfaced": OfflineScorer(kind="numeric", version=UUID("01a143b7-830b-77b4-82f5-89f8d83f4ab7")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a143b7-3a45-7c94-9e46-4cf75ab3ee12")),
+ "notebook_approach_quality": OfflineScorer(kind="numeric", version=UUID("01a143b7-850d-72c8-85d9-0d74005d8f37")),
+ "notebook_created": OfflineScorer(kind="boolean", version=UUID("01a143b7-7b0e-7834-ab39-e6d37d772ff7")),
+ "notebook_run_completed": OfflineScorer(kind="boolean", version=UUID("01a143b7-7ec4-77b5-9a32-7e9984aa2755")),
+ "required_tool_call": OfflineScorer(kind="boolean", version=UUID("01a143b7-78c5-78c4-9f0d-60e0f5965b1a")),
+}
+
+
+def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:
+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics
+ return OfflineEvalSuite.from_scorers(
+ key=key,
+ project_id=2,
+ scorers={metric: _SCORERS[metric] for metric in enrolled_metrics},
+ tags=("team:data-tools", "component:notebooks"),
+ )
diff --git a/products/posthog_ai/backend/turn_suggestions/benchmark.py b/products/posthog_ai/backend/turn_suggestions/benchmark.py
index 163124ba1e70..e1879ff903a0 100644
--- a/products/posthog_ai/backend/turn_suggestions/benchmark.py
+++ b/products/posthog_ai/backend/turn_suggestions/benchmark.py
@@ -277,7 +277,7 @@ def _judge_at_endpoint(case: BenchmarkCase, endpoint: SystemOneEndpoint) -> Turn
return read_judgment(parse_system_one_response(response.json(), questions), case.transcript, case.available)
-def _judge(case: BenchmarkCase, endpoint: SystemOneEndpoint | None) -> CaseResult:
+def judge_case(case: BenchmarkCase, endpoint: SystemOneEndpoint | None = None) -> CaseResult:
started = time.monotonic()
judgment: TurnJudgment | None = None
failure: str | None = None
@@ -310,7 +310,7 @@ def run_cases(
the order of ``cases``."""
results = []
with ThreadPoolExecutor(max_workers=workers) as pool:
- for future in as_completed([pool.submit(_judge, case, endpoint) for case in cases]):
+ for future in as_completed([pool.submit(judge_case, case, endpoint) for case in cases]):
result = future.result()
on_result(result)
results.append(result)
diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.md
index d6b0806f7e5d..5fd455f63bff 100644
--- a/products/posthog_ai/eval_harness/README.md
+++ b/products/posthog_ai/eval_harness/README.md
@@ -37,7 +37,8 @@ python -m products.posthog_ai.eval_harness.harness [SELECTOR ...] [flags]
No manual env sourcing is needed on either path.
The harness loads the repo-root `.env` itself (shell values win), and `hogli evals` additionally layers in `.env.local` / `.env.development` / `.env.services` through hogli's standard env loading — including 1Password resolution when `.env.local` holds `op://` references.
Before any infrastructure boots, a preflight validates that the required variables are set and fails with a one-line fix per missing variable.
-Which variables are required depends on the eval engine and the selected suites' kinds: the braintrust engine (the default, and only one today) requires `BRAINTRUST_API_KEY` on every run — it is the engine's own `required_env()`, not a core harness variable; sandboxed suites add `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`, and need a [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway) clone plus `go` on `PATH` (see [AI gateway](#ai-gateway)); one-shot suites add `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).
+Which variables are required depends on the selected suites' reporting settings and kinds: the Braintrust engine requires `BRAINTRUST_API_KEY` only if at least one selected suite sends logs to Braintrust. A run consisting entirely of suites declaring `NO_SEND_LOGS = True` does not require that reporting credential. Sandboxed suites require `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`, and need a [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway) clone plus `go` on `PATH` (see [AI gateway](#ai-gateway)); one-shot suites require `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).
+Scorers still need credentials for their model provider. A scorer configured to use Braintrust's model proxy needs its proxy credential independently of result reporting.
Selectors are substrings matched against a suite id of the form `<domain>/<module>::<fn>`, for example `experiments`, `sql`, or `eval_lifecycle_skills`.
Omit them to run every suite.
@@ -452,6 +453,9 @@ Agent and judge trace capture remains available. `no_send_logs=True` suppresses
Set `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal key with `offline_evaluation_ingestion:write`, `evaluation:write`, and access to project `2` to upload and tag results. An ingestion-only personal or project secret key can upload results, but tagging will warn and leave the run untagged.
The [SQL suite definition](../evals/sql/eval_sql.py) pins that project ID and all seven immutable scorer version UUIDs in code.
`POSTHOG_OFFLINE_EVAL_HOST` defaults to `https://us.posthog.com`.
+For development, use `http://localhost:8010` with a local API key and set `POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG`
+to a gitignored JSON file containing the local project ID and scorer-version mappings.
+See [local upload setup](../../../docs/internal/ai-offline-evaluation-reporting.md#testing-uploads-against-local-posthog).
The [suite configuration](../../../docs/internal/ai-offline-evaluation-reporting.md#configured-offline-experiment-suites) lists the metrics and explains local and CI key configuration.
Delivery runs in a bounded background queue after local results are saved. Prepared uploads stay in memory, with up to two uploads running concurrently, at most 32 queued or active uploads, and a 60-second deadline per active upload.
A completed upload records its results link before tagging begins. Tagging has a separate five-second deadline; failures, timeouts, and shutdown interruptions only warn and preserve the successful upload status.
diff --git a/products/posthog_ai/eval_harness/engines/base.py b/products/posthog_ai/eval_harness/engines/base.py
index ddd66620ebcb..b52aba1f554c 100644
--- a/products/posthog_ai/eval_harness/engines/base.py
+++ b/products/posthog_ai/eval_harness/engines/base.py
@@ -41,7 +41,7 @@ class EvalEngine(Protocol):
"""Whether the engine has a notion of a public experiment (``is_public``)."""
@classmethod
- def required_env(cls) -> tuple[EnvVarSpec, ...]:
+ def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:
"""The environment variables this engine needs, validated in preflight."""
...
diff --git a/products/posthog_ai/eval_harness/engines/braintrust.py b/products/posthog_ai/eval_harness/engines/braintrust.py
index a6ba71180f5d..f714c7fc9a7e 100644
--- a/products/posthog_ai/eval_harness/engines/braintrust.py
+++ b/products/posthog_ai/eval_harness/engines/braintrust.py
@@ -89,7 +89,9 @@ class BraintrustEngine:
supports_public_experiments: ClassVar[bool] = True
@classmethod
- def required_env(cls) -> tuple[EnvVarSpec, ...]:
+ def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:
+ if no_send_logs:
+ return ()
return (EnvVarSpec("BRAINTRUST_API_KEY", "records experiments and scores to Braintrust"),)
@staticmethod
diff --git a/products/posthog_ai/eval_harness/harness/discovery.py b/products/posthog_ai/eval_harness/harness/discovery.py
index 91252a007a8f..2d77c1b44d18 100644
--- a/products/posthog_ai/eval_harness/harness/discovery.py
+++ b/products/posthog_ai/eval_harness/harness/discovery.py
@@ -4,11 +4,12 @@ import inspect
import logging
import importlib
from collections.abc import Callable, Coroutine, Iterator, Sequence
-from dataclasses import dataclass
from pathlib import Path
from types import ModuleType
from typing import TYPE_CHECKING, Any
+from posthog.dataclasses import frozen
+
from .requirements import SuiteKind
if TYPE_CHECKING:
@@ -37,7 +38,7 @@ cover a whole conversation. ``_collect_suites`` copies the marker onto every
suite function of the module so lifecycle can read it off the suite."""
-@dataclass(frozen=True)
+@frozen
class EvalSuite:
domain: str
"""Directory the suite lives in, e.g. ``experiments`` for a sandboxed suite or
@@ -50,6 +51,9 @@ class EvalSuite:
kind: SuiteKind = SuiteKind.SANDBOXED
"""How the suite's cases execute, from the module-level ``SUITE_KIND``."""
+ no_send_logs: bool = False
+ """Module-level ``NO_SEND_LOGS`` lets preflight check credentials before suite setup."""
+
@property
def id(self) -> str:
return f"{self.domain}/{self.module_name}::{self.fn_name}"
@@ -111,6 +115,9 @@ def _collect_suites(module: ModuleType, domain: str, module_name: str, dotted: s
kind = getattr(module, "SUITE_KIND", SuiteKind.SANDBOXED)
if not isinstance(kind, SuiteKind):
raise SuiteDiscoveryError(f"{dotted}.SUITE_KIND must be a SuiteKind member, got {kind!r}")
+ no_send_logs = getattr(module, "NO_SEND_LOGS", False)
+ if not isinstance(no_send_logs, bool):
+ raise SuiteDiscoveryError(f"{dotted}.NO_SEND_LOGS must be a bool, got {no_send_logs!r}")
multi_turn = bool(getattr(module, MULTI_TURN_MODULE_MARKER, False))
collected: list[EvalSuite] = []
for fn_name, fn in vars(module).items():
@@ -123,7 +130,11 @@ def _collect_suites(module: ModuleType, domain: str, module_name: str, dotted: s
if multi_turn:
# The dataclass is frozen, so carry the marker on the function.
setattr(fn, MULTI_TURN_MODULE_MARKER, True)
- collected.append(EvalSuite(domain=domain, module_name=module_name, fn_name=fn_name, fn=fn, kind=kind))
+ collected.append(
+ EvalSuite(
+ domain=domain, module_name=module_name, fn_name=fn_name, fn=fn, kind=kind, no_send_logs=no_send_logs
+ )
+ )
return collected
diff --git a/products/posthog_ai/eval_harness/harness/env_preflight.py b/products/posthog_ai/eval_harness/harness/env_preflight.py
index f8ba39603da1..14c8fc25c1aa 100644
--- a/products/posthog_ai/eval_harness/harness/env_preflight.py
+++ b/products/posthog_ai/eval_harness/harness/env_preflight.py
@@ -143,8 +143,8 @@ def validate_eval_env(
) -> None:
"""Fail fast, before any infrastructure boots, if a required variable is unset.
- ``engine_env`` is the selected engine's ``required_env()`` (e.g. the Braintrust
- engine's ``BRAINTRUST_API_KEY``). Only the env models for the selected suites'
+ ``engine_env`` is the selected engine's ``required_env(no_send_logs=...)``
+ for the selected suites' reporting settings. Only the env models for the selected suites'
kinds are checked, so a run without sandboxed suites doesn't demand sandbox
credentials. Without this a missing key surfaces minutes into a run as an
opaque mid-case failure (gateway 401s, engine login errors) instead of a
diff --git a/products/posthog_ai/eval_harness/harness/lifecycle.py b/products/posthog_ai/eval_harness/harness/lifecycle.py
index 1e7e9d5c33c3..52f25315bf32 100644
--- a/products/posthog_ai/eval_harness/harness/lifecycle.py
+++ b/products/posthog_ai/eval_harness/harness/lifecycle.py
@@ -189,7 +189,11 @@ class SandboxedEvalHarness:
with asyncio.Runner() as runner:
try:
reporter.setup_started("check configuration")
- validate_eval_env(self.options.agent_runtime, kinds=kinds, engine_env=self._engine.required_env())
+ validate_eval_env(
+ self.options.agent_runtime,
+ kinds=kinds,
+ engine_env=self._engine.required_env(no_send_logs=all(suite.no_send_logs for suite in suites)),
+ )
if Infra.SANDBOX in required:
self.provider = build_provider(
self.options.provider,
diff --git a/products/posthog_ai/eval_harness/offline_results.py b/products/posthog_ai/eval_harness/offline_results.py
index 02d4bc5014f9..3a505069fa51 100644
--- a/products/posthog_ai/eval_harness/offline_results.py
+++ b/products/posthog_ai/eval_harness/offline_results.py
@@ -10,6 +10,7 @@ from collections.abc import Mapping
from contextlib import suppress
from dataclasses import field
from datetime import datetime
+from pathlib import Path
from typing import cast
from urllib.parse import urlsplit
from uuid import UUID, uuid5
@@ -33,6 +34,7 @@ from .engines.types import CaseResult, ExperimentResult, ScorerResult
logger = logging.getLogger(__name__)
TAG_TIMEOUT_SECONDS = 5.0
+LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"})
@frozen
@@ -109,6 +111,30 @@ class OfflineUploadError(Exception):
super().__init__(message)
+class LocalOfflineEvalConfig(BaseModel):
+ model_config = ConfigDict(frozen=True, extra="forbid", hide_input_in_errors=True)
+
+ project_id: int = Field(gt=0)
+ scorer_versions: dict[UUID, UUID]
+
+ @classmethod
+ def load(cls, path: str) -> LocalOfflineEvalConfig:
+ try:
+ return cls.model_validate_json(Path(path).read_text())
+ except (OSError, UnicodeError, ValidationError):
+ raise OfflineUploadError(
+ "POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG must point to a valid local destination JSON"
+ ) from None
+
+ def remap_scorers(self, suite: OfflineEvalSuite) -> dict[str, UUID]:
+ try:
+ return {metric: self.scorer_versions[suite.scorer_versions[metric]] for metric in suite.scorer_kinds}
+ except KeyError:
+ raise OfflineUploadError(
+ f"Local offline evaluation config must map every enrolled scorer version for {suite.key}"
+ ) from None
+
+
class OfflineEvalSettings(BaseModel):
model_config = ConfigDict(frozen=True, hide_input_in_errors=True)
@@ -130,13 +156,14 @@ class OfflineEvalSettings(BaseModel):
@field_validator("host")
@classmethod
def normalize_host(cls, host: str) -> str:
+ error = "POSTHOG_OFFLINE_EVAL_HOST must be an HTTPS origin or HTTP loopback origin"
try:
parsed = urlsplit(host.strip().rstrip("/"))
port = parsed.port
except ValueError:
- raise OfflineUploadError("POSTHOG_OFFLINE_EVAL_HOST must be an HTTPS origin") from None
+ raise OfflineUploadError(error) from None
if (
- parsed.scheme != "https"
+ not (parsed.scheme == "https" or (parsed.scheme == "http" and parsed.hostname in LOOPBACK_HOSTS))
or not parsed.hostname
or parsed.username is not None
or parsed.password is not None
@@ -144,15 +171,15 @@ class OfflineEvalSettings(BaseModel):
or parsed.fragment
or parsed.path
):
- raise OfflineUploadError("POSTHOG_OFFLINE_EVAL_HOST must be an HTTPS origin")
+ raise OfflineUploadError(error)
origin = parsed.netloc.lower()
- if port in (None, 443):
+ if parsed.scheme == "https" and port in (None, 443):
origin = {
"app.posthog.com": "us.posthog.com",
"us.i.posthog.com": "us.posthog.com",
"eu.i.posthog.com": "eu.posthog.com",
}.get(parsed.hostname, origin.removesuffix(":443"))
- return f"https://{origin}"
+ return f"{parsed.scheme}://{origin}"
@classmethod
def from_env(cls, suite: OfflineEvalSuite) -> OfflineEvalSettings | None:
@@ -160,12 +187,20 @@ class OfflineEvalSettings(BaseModel):
api_key = os.getenv(prefix + "API_KEY")
if not api_key:
return None
+ host = cls.normalize_host(os.getenv(prefix + "HOST", "https://us.posthog.com"))
+ local_config = None
+ if local_config_path := os.getenv(prefix + "LOCAL_CONFIG"):
+ if urlsplit(host).hostname not in LOOPBACK_HOSTS:
+ raise OfflineUploadError("POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG requires a loopback upload host")
+ local_config = LocalOfflineEvalConfig.load(local_config_path)
try:
settings = cls(
- host=os.getenv(prefix + "HOST", "https://us.posthog.com"),
- project_id=suite.project_id,
+ host=host,
+ project_id=local_config.project_id if local_config is not None else suite.project_id,
api_key=SecretStr(api_key),
- scorer_versions=dict(suite.scorer_versions),
+ scorer_versions=local_config.remap_scorers(suite)
+ if local_config is not None
+ else dict(suite.scorer_versions),
)
except ValidationError:
raise OfflineUploadError(
diff --git a/products/posthog_ai/eval_harness/test/test_benchmark_scorers.py b/products/posthog_ai/eval_harness/test/test_benchmark_scorers.py
new file mode 100644
index 000000000000..c02859a0222d
--- /dev/null
+++ b/products/posthog_ai/eval_harness/test/test_benchmark_scorers.py
@@ -0,0 +1,37 @@
+import pytest
+
+from products.business_knowledge.evals.eval_retrieval import RetrievalMatch
+from products.posthog_ai.backend.turn_suggestions.benchmark import Outcome
+from products.posthog_ai.evals.turn_suggestions.eval_turn_suggestions import OfferCorrectness
+
+
+ "outcome,score",
+ [
+ (Outcome.CORRECT, 1.0),
+ (Outcome.EITHER, 1.0),
+ (Outcome.FALSE_OFFER, 0.0),
+ (Outcome.MISSED, 0.0),
+ (Outcome.WRONG_KIND, 0.0),
+ (Outcome.FAILED, None),
+ ],
+)
+def test_offer_correctness_preserves_benchmark_outcomes(outcome: Outcome, score: float | None) -> None:
+ result = OfferCorrectness().eval({"outcome": outcome, "error": "judge unavailable"})
+ assert result.score == score
+ if outcome == Outcome.FAILED:
+ assert result.result is not None
+ assert result.result.status == "error"
+ assert result.result.error_code == "judge_error"
+ assert result.result.error_message == "judge unavailable"
+ else:
+ assert result.result is None
+
+
[email protected]("found,expected,score", [(True, True, 1.0), (False, False, 1.0), (False, True, 0.0)])
+def test_retrieval_scores_absence_as_a_valid_outcome(found: bool, expected: bool, score: float) -> None:
+ result = RetrievalMatch().eval(
+ {"found": found, "source_names": ["Refund Policy"] if found else []},
+ {"retrieval_match": {"found": expected}},
+ )
+ assert result.score == score
diff --git a/products/posthog_ai/eval_harness/test/test_discovery.py b/products/posthog_ai/eval_harness/test/test_discovery.py
index afcf74e55e1f..d98737f45366 100644
--- a/products/posthog_ai/eval_harness/test/test_discovery.py
+++ b/products/posthog_ai/eval_harness/test/test_discovery.py
@@ -4,13 +4,14 @@ from pathlib import Path
import pytest
-from products.posthog_ai.eval_harness.harness.discovery import SuiteDiscoveryError, discover_suites
+from products.posthog_ai.eval_harness.harness.discovery import EvalSuite, SuiteDiscoveryError, discover_suites
from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
_ONE_SHOT_MODULE = """
from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
SUITE_KIND = SuiteKind.ONE_SHOT
+NO_SEND_LOGS = True
async def eval_alpha(ctx) -> None:
@@ -53,10 +54,10 @@ def product_roots(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> tuple[Path
return products_root, empty_builtin_root
-def _discover(product_roots: tuple[Path, Path]) -> dict[str, SuiteKind]:
+def _discover(product_roots: tuple[Path, Path]) -> dict[str, EvalSuite]:
products_root, builtin_root = product_roots
suites = discover_suites(builtin_root=builtin_root, products_root=products_root)
- return {suite.id: suite.kind for suite in suites}
+ return {suite.id: suite for suite in suites}
def test_product_suites_discovered_with_product_scoped_ids(product_roots: tuple[Path, Path]) -> None:
@@ -70,16 +71,18 @@ def test_singular_eval_dir_is_ignored(product_roots: tuple[Path, Path]) -> None:
@pytest.mark.parametrize(
- "suite_id, expected_kind",
+ "suite_id, expected_kind, no_send_logs",
[
- pytest.param("analytics/eval_one::eval_alpha", SuiteKind.ONE_SHOT, id="marker_respected"),
- pytest.param("core/eval_two::eval_beta", SuiteKind.SANDBOXED, id="absent_marker_defaults_sandboxed"),
+ pytest.param("analytics/eval_one::eval_alpha", SuiteKind.ONE_SHOT, True, id="markers_respected"),
+ pytest.param("core/eval_two::eval_beta", SuiteKind.SANDBOXED, False, id="absent_markers_keep_defaults"),
],
)
-def test_suite_kind_resolved_from_product_module(
- product_roots: tuple[Path, Path], suite_id: str, expected_kind: SuiteKind
+def test_suite_configuration_resolved_from_product_module(
+ product_roots: tuple[Path, Path], suite_id: str, expected_kind: SuiteKind, no_send_logs: bool
) -> None:
- assert _discover(product_roots)[suite_id] == expected_kind
+ suite = _discover(product_roots)[suite_id]
+ assert suite.kind == expected_kind
+ assert suite.no_send_logs == no_send_logs
def test_selector_matches_product_suite_ids(product_roots: tuple[Path, Path]) -> None:
diff --git a/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py b/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py
index bf8e7958d855..b7d7360b0cac 100644
--- a/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py
+++ b/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py
@@ -35,7 +35,7 @@ def test_harness_tears_down_infrastructure_before_draining_uploads(
interrupted: bool,
teardown_phase: Literal["async", "sync"],
) -> None:
- monkeypatch.setenv("BRAINTRUST_API_KEY", "test-only-key")
+ monkeypatch.delenv("BRAINTRUST_API_KEY", raising=False)
monkeypatch.setenv("LLM_GATEWAY_ANTHROPIC_API_KEY", "test-only-key")
monkeypatch.delenv("EXPORT_EVAL_RESULTS", raising=False)
started = asyncio.Event()
@@ -83,7 +83,14 @@ def test_harness_tears_down_infrastructure_before_draining_uploads(
lifecycle,
"discover_suites",
lambda _: [
- EvalSuite(domain="test", module_name="eval_test", fn_name="eval_test", fn=suite, kind=SuiteKind.ONE_SHOT)
+ EvalSuite(
+ domain="test",
+ module_name="eval_test",
+ fn_name="eval_test",
+ fn=suite,
+ kind=SuiteKind.ONE_SHOT,
+ no_send_logs=True,
+ )
],
)
monkeypatch.setattr(lifecycle, "ensure_personhog_binaries", lambda: None)
diff --git a/products/posthog_ai/eval_harness/test/test_offline_enrollment.py b/products/posthog_ai/eval_harness/test/test_offline_enrollment.py
new file mode 100644
index 000000000000..259f38d36a87
--- /dev/null
+++ b/products/posthog_ai/eval_harness/test/test_offline_enrollment.py
@@ -0,0 +1,62 @@
+from pathlib import Path
+from uuid import UUID
+
+import pytest
+from unittest.mock import Mock, patch
+
+from products.posthog_ai.eval_harness.base import _BaseEvalRun
+from products.posthog_ai.eval_harness.harness.context import EvalContext
+from products.posthog_ai.eval_harness.harness.discovery import discover_suites
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+
+
+async def test_discovered_suites_match_reporting_enrollment(tmp_path: Path) -> None:
+ ctx = Mock(spec=EvalContext, posthog_client=None, case_filter=None, skill_delivery="bundled")
+ ctx.engine = Mock()
+ ctx.provider_strategy = Mock()
+ ctx.demo_data = Mock()
+ ctx.sandbox_slots = Mock()
+ runs: list[_BaseEvalRun] = []
+
+ async def capture(run: _BaseEvalRun) -> None:
+ runs.append(run)
+
+ suites = discover_suites()
+ assert suites
+ keys: set[str] = set()
+ with (
+ patch.object(_BaseEvalRun, "run", capture),
+ patch("products.posthog_ai.eval_harness.base.build_case_dir", return_value=tmp_path),
+ patch("products.posthog_ai.evals.taxonomic_filter.eval_event_match.system_one_configured", return_value=True),
+ patch("products.posthog_ai.evals.taxonomic_filter.eval_event_match.EVENT_MATCH_MODEL"),
+ patch("products.posthog_ai.evals.taxonomic_filter.eval_search_intent.system_one_configured", return_value=True),
+ patch("products.posthog_ai.evals.taxonomic_filter.eval_search_intent.fetch_search_intent_prompt"),
+ ):
+ for suite in suites:
+ runs.clear()
+ await suite.fn(ctx)
+ assert len(runs) == 1, suite.id
+ run = runs[0]
+ assert run.no_send_logs == suite.no_send_logs, suite.id
+ config = run.offline_suite
+ if suite.id == "logs/eval_agent_observability::eval_agent_observability":
+ assert config is None
+ assert not run.no_send_logs
+ continue
+ assert config is not None, suite.id
+ assert run.send_to_posthog, suite.id
+ assert config.key == suite.id
+ assert config.key not in keys
+ keys.add(config.key)
+ assert config.project_id == 2
+ metrics = {scorer._name() for scorer in run.active_scorers}
+ assert set(config.scorer_kinds) == set(config.scorer_versions) == metrics, suite.id
+ assert len(set(config.scorer_versions.values())) == len(metrics), suite.id
+ assert all(isinstance(version, UUID) and version.int for version in config.scorer_versions.values())
+ assert ("exit_code_zero" in metrics) == (suite.kind == SuiteKind.SANDBOXED), suite.id
+ assert any(tag.startswith("team:") for tag in config.tags), suite.id
+ assert any(tag.startswith("component:") for tag in config.tags), suite.id
+
+ if suite.domain in {"feature_flags", "dashboards", "business_knowledge", "turn_suggestions"}:
+ assert run.no_send_logs, suite.id
diff --git a/products/posthog_ai/eval_harness/test/test_offline_results.py b/products/posthog_ai/eval_harness/test/test_offline_results.py
index 4e3ae913157c..f7e10b51c954 100644
--- a/products/posthog_ai/eval_harness/test/test_offline_results.py
+++ b/products/posthog_ai/eval_harness/test/test_offline_results.py
@@ -5,6 +5,7 @@ import asyncio
from dataclasses import replace
from datetime import UTC, datetime
from functools import partial
+from pathlib import Path
from uuid import UUID
import pytest
@@ -594,6 +595,9 @@ def test_publish_refuses_a_different_destination(_name: str, host: str, project_
("custom", "https://posthog.example.com:8443", "https://posthog.example.com:8443"),
("custom_cloud_port", "https://us.i.posthog.com:8443", "https://us.i.posthog.com:8443"),
("ipv6", "https://[::1]:8443", "https://[::1]:8443"),
+ ("localhost", "http://localhost:8010", "http://localhost:8010"),
+ ("loopback", "http://127.0.0.1:8010/", "http://127.0.0.1:8010"),
+ ("loopback_ipv6", "http://[::1]:8010", "http://[::1]:8010"),
]
)
def test_publish_uses_the_same_origin_for_requests_and_result_links(_name: str, host: str, origin: str) -> None:
@@ -616,6 +620,96 @@ def test_publish_uses_the_same_origin_for_requests_and_result_links(_name: str,
assert url == f"{origin}/project/123/ai-evals/evaluations/offline/experiments/{EXPERIMENT_ID}"
+def test_local_config_routes_uploads_to_local_project_and_scorer_versions(tmp_path: Path) -> None:
+ local_versions = {
+ str(SCORER_VERSIONS["boolean_metric"]): "00000000-0000-4000-8000-000000000004",
+ str(SCORER_VERSIONS["numeric_metric"]): "00000000-0000-4000-8000-000000000005",
+ }
+ config = tmp_path / "local.json"
+ config.write_text(json.dumps({"project_id": 456, "scorer_versions": local_versions}))
+ with patch.dict(
+ "os.environ",
+ {
+ "POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token",
+ "POSTHOG_OFFLINE_EVAL_HOST": "http://localhost:8010",
+ "POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG": str(config),
+ },
+ clear=True,
+ ):
+ settings = OfflineEvalSettings.from_env(SUITE)
+ assert settings is not None
+ publisher = OfflineEvalPublisher(settings, SUITE)
+ upload = _prepare(publisher, [_case()])
+ server = OfflineServer()
+
+ with patch("httpx.AsyncClient", new=partial(httpx.AsyncClient, transport=httpx.MockTransport(server.respond))):
+ url = asyncio.run(publisher.publish(upload))
+
+ assert all(str(request.url).startswith("http://localhost:8010/api/projects/456/") for request in server.requests)
+ assert [str(result.scorer_version_id) for result in upload.results] == list(local_versions.values())
+ assert url == f"http://localhost:8010/project/456/ai-evals/evaluations/offline/experiments/{EXPERIMENT_ID}"
+ assert SUITE.project_id == 123
+ assert SUITE.scorer_versions == SCORER_VERSIONS
+
+
+ "contents,host",
+ [
+ (None, "http://localhost:8010"),
+ ("invalid json", "http://localhost:8010"),
+ ('{"project_id": 0, "scorer_versions": {}}', "http://localhost:8010"),
+ ('{"project_id": 456, "scorer_versions": {}}', "http://localhost:8010"),
+ ('{"project_id": 456, "scorer_versions": {"invalid": "invalid"}}', "http://localhost:8010"),
+ (
+ json.dumps(
+ {
+ "project_id": 456,
+ "scorer_versions": {str(version): str(version) for version in SCORER_VERSIONS.values()},
+ }
+ ),
+ "https://us.posthog.com",
+ ),
+ (
+ json.dumps(
+ {
+ "project_id": 456,
+ "scorer_versions": dict.fromkeys(
+ map(str, SCORER_VERSIONS.values()), "00000000-0000-4000-8000-000000000004"
+ ),
+ }
+ ),
+ "http://localhost:8010",
+ ),
+ ],
+ ids=[
+ "missing_file",
+ "invalid_json",
+ "invalid_project",
+ "missing_versions",
+ "invalid_uuid",
+ "cloud_host",
+ "duplicate_versions",
+ ],
+)
+def test_invalid_local_config_never_falls_back_to_catalog(tmp_path: Path, contents: str | None, host: str) -> None:
+ config = tmp_path / "local.json"
+ if contents is not None:
+ config.write_text(contents)
+ with (
+ patch.dict(
+ "os.environ",
+ {
+ "POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token",
+ "POSTHOG_OFFLINE_EVAL_HOST": host,
+ "POSTHOG_OFFLINE_EVAL_LOCAL_CONFIG": str(config),
+ },
+ clear=True,
+ ),
+ pytest.raises(OfflineUploadError),
+ ):
+ OfflineEvalSettings.from_env(SUITE)
+
+
def test_settings_use_each_suite_destination_with_only_an_api_key() -> None:
environment = {"POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token"}
with patch.dict("os.environ", environment, clear=True):
@@ -661,6 +755,8 @@ def test_settings_reject_invalid_suite_configuration(_name: str, suite: OfflineE
("key_internal_space", {"POSTHOG_OFFLINE_EVAL_API_KEY": "test-only token"}),
("key_leading_tab", {"POSTHOG_OFFLINE_EVAL_API_KEY": "\ttest-only-token"}),
("insecure_host", {"POSTHOG_OFFLINE_EVAL_HOST": "http://posthog.example.com"}),
+ ("loopback_suffix", {"POSTHOG_OFFLINE_EVAL_HOST": "http://localhost.example.com:8010"}),
+ ("loopback_credentials", {"POSTHOG_OFFLINE_EVAL_HOST": "http://[email protected]:8010"}),
("malformed_host", {"POSTHOG_OFFLINE_EVAL_HOST": "https://["}),
("invalid_port", {"POSTHOG_OFFLINE_EVAL_HOST": "https://posthog.example.com:invalid"}),
("host_credentials", {"POSTHOG_OFFLINE_EVAL_HOST": "https://[email protected]"}),
diff --git a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py b/products/posthog_ai/eval_harness/test/test_one_shot_runner.py
index 93d72fc36f53..57bf2da45036 100644
--- a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py
+++ b/products/posthog_ai/eval_harness/test/test_one_shot_runner.py
@@ -93,7 +93,7 @@ class _StubEngine:
self.calls: list[ExperimentSpec] = []
@classmethod
- def required_env(cls) -> tuple[EnvVarSpec, ...]:
+ def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:
return ()
async def run_experiment(self, spec: ExperimentSpec) -> ExperimentResult:
diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py
index bf1d6a95cd7a..e900c5d7dc35 100644
--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py
+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py
@@ -413,7 +413,7 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repo
http_status: int,
delivery_state: Literal["ready", "unavailable", "closed"],
) -> None:
- for key in ("API_KEY", "HOST"):
+ for key in ("API_KEY", "HOST", "LOCAL_CONFIG"):
monkeypatch.delenv(f"POSTHOG_OFFLINE_EVAL_{key}", raising=False)
monkeypatch.setenv("EXPORT_EVAL_RESULTS", "1")
monkeypatch.chdir(tmp_path)
@@ -570,10 +570,9 @@ async def test_suite_enrolls_every_scorer_in_offline_publishing(
else:
monkeypatch.setattr(module, "fetch_search_intent_prompt", MagicMock())
run = AsyncMock()
- runner = "SandboxedPrivateEval" if domain == "experiments" else "SandboxedPublicEval"
- if suite_kind == SuiteKind.ONE_SHOT:
- runner = "OneShotPublicEval"
- monkeypatch.setattr(module, runner, run)
+ runner_kind = "OneShot" if suite_kind == SuiteKind.ONE_SHOT else "Sandboxed"
+ runner_visibility = "Private" if discovered.no_send_logs else "Public"
+ monkeypatch.setattr(module, f"{runner_kind}{runner_visibility}Eval", run)
await discovered.fn(MagicMock(skill_delivery="bundled"))
diff --git a/products/posthog_ai/eval_harness/test/test_suite_requirements.py b/products/posthog_ai/eval_harness/test/test_suite_requirements.py
index eed3b1ea5348..e9fd0aea6c15 100644
--- a/products/posthog_ai/eval_harness/test/test_suite_requirements.py
+++ b/products/posthog_ai/eval_harness/test/test_suite_requirements.py
@@ -1,8 +1,13 @@
from __future__ import annotations
import pytest
+from unittest.mock import Mock
from products.posthog_ai.eval_harness.engines.registry import resolve_engine
+from products.posthog_ai.eval_harness.harness import lifecycle
+from products.posthog_ai.eval_harness.harness.cli import parse_args
+from products.posthog_ai.eval_harness.harness.context import EvalContext
+from products.posthog_ai.eval_harness.harness.discovery import EvalSuite
from products.posthog_ai.eval_harness.harness.env_preflight import validate_eval_env
from products.posthog_ai.eval_harness.harness.providers import PreflightError
from products.posthog_ai.eval_harness.harness.requirements import INFRA_BY_KIND, Infra, SuiteKind, expand, infra_union
@@ -130,14 +135,69 @@ def test_validate_eval_env_by_kind(
assert name not in message
-def test_validate_eval_env_reports_missing_engine_env(monkeypatch: pytest.MonkeyPatch) -> None:
[email protected]("no_send_logs", [False, True])
+def test_validate_eval_env_reports_missing_engine_env(monkeypatch: pytest.MonkeyPatch, no_send_logs: bool) -> None:
for name in ALL_ENV_VARS:
monkeypatch.delenv(name, raising=False)
monkeypatch.setenv("LLM_GATEWAY_ANTHROPIC_API_KEY", "x")
- with pytest.raises(PreflightError) as exc_info:
- validate_eval_env("claude", kinds={SuiteKind.ONE_SHOT}, engine_env=resolve_engine().required_env())
+ engine_env = resolve_engine().required_env(no_send_logs=no_send_logs)
+ if no_send_logs:
+ validate_eval_env("claude", kinds={SuiteKind.ONE_SHOT}, engine_env=engine_env)
+ return
- # The Braintrust engine's requirement is reported with the same line the core
- # env model used to emit, so a missing key reads identically to before.
+ with pytest.raises(PreflightError) as exc_info:
+ validate_eval_env("claude", kinds={SuiteKind.ONE_SHOT}, engine_env=engine_env)
assert " - BRAINTRUST_API_KEY: records experiments and scores to Braintrust" in str(exc_info.value)
+
+
+ "no_send_logs, has_key",
+ [
+ pytest.param([True], False, id="posthog_only_without_braintrust_key"),
+ pytest.param([False], False, id="braintrust_reporting_without_key"),
+ pytest.param([True, False], False, id="mixed_reporting_without_key"),
+ pytest.param([True, False], True, id="mixed_reporting_with_key"),
+ ],
+)
+def test_harness_checks_selected_reporting_before_starting_infrastructure(
+ monkeypatch: pytest.MonkeyPatch, no_send_logs: list[bool], has_key: bool
+) -> None:
+ monkeypatch.delenv("BRAINTRUST_API_KEY", raising=False)
+ monkeypatch.delenv("EXPORT_EVAL_RESULTS", raising=False)
+ monkeypatch.setenv("LLM_GATEWAY_ANTHROPIC_API_KEY", "test-only-key")
+ if has_key:
+ monkeypatch.setenv("BRAINTRUST_API_KEY", "test-only-key")
+ completed: list[bool] = []
+
+ async def run_suite(ctx: EvalContext) -> None:
+ completed.append(True)
+
+ suites = [
+ EvalSuite(
+ domain="test",
+ module_name="eval_test",
+ fn_name=f"eval_test_{index}",
+ fn=run_suite,
+ kind=SuiteKind.ONE_SHOT,
+ no_send_logs=disabled,
+ )
+ for index, disabled in enumerate(no_send_logs)
+ ]
+ monkeypatch.setattr(lifecycle, "discover_suites", lambda _: suites)
+ build = Mock()
+ monkeypatch.setattr(lifecycle, "ensure_personhog_binaries", build)
+ harness = lifecycle.SandboxedEvalHarness(parse_args([]))
+
+ def bootstrap(required: frozenset[Infra], reporter: lifecycle.ProgressReporter) -> None:
+ harness._demo_data = Mock()
+
+ monkeypatch.setattr(harness, "_bootstrap", bootstrap)
+ if has_key or all(no_send_logs):
+ assert harness.run() == 0
+ assert len(completed) == len(suites)
+ else:
+ with pytest.raises(PreflightError, match="BRAINTRUST_API_KEY"):
+ harness.run()
+ build.assert_not_called()
+ assert not completed
diff --git a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py
index 9e7b72f954a1..867db0ffcd27 100644
--- a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py
+++ b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py
@@ -41,6 +41,8 @@ from products.posthog_ai.evals.experiments.seeders import (
)
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
async def eval_bias_uneven_split(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
@@ -107,6 +109,7 @@ async def eval_bias_uneven_split(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-diagnose-bias-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_create_control_variant.py b/products/posthog_ai/evals/experiments/eval_create_control_variant.py
index b7a6e5e93a0e..e3993092b539 100644
--- a/products/posthog_ai/evals/experiments/eval_create_control_variant.py
+++ b/products/posthog_ai/evals/experiments/eval_create_control_variant.py
@@ -36,6 +36,8 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import RequiredToolCall
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
async def eval_create_control_variant(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
@@ -56,6 +58,7 @@ async def eval_create_control_variant(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-create-control-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_empty_experiment.py b/products/posthog_ai/evals/experiments/eval_empty_experiment.py
index 9e962c140071..474dd56e0acc 100644
--- a/products/posthog_ai/evals/experiments/eval_empty_experiment.py
+++ b/products/posthog_ai/evals/experiments/eval_empty_experiment.py
@@ -54,6 +54,8 @@ from products.posthog_ai.evals.experiments.seeders import (
)
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
async def eval_empty_experiment(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
@@ -139,6 +141,7 @@ async def eval_empty_experiment(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-diagnose-empty-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py
index 262879511506..37204d92d4a1 100644
--- a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py
+++ b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py
@@ -52,6 +52,8 @@ from products.posthog_ai.evals.experiments.seeders import (
)
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
async def eval_interpretation_traps(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
@@ -161,6 +163,7 @@ async def eval_interpretation_traps(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-diagnose-interpretation-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py
index e085458bcad5..e1001365b69d 100644
--- a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py
+++ b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py
@@ -42,6 +42,8 @@ from products.posthog_ai.evals.experiments.scorers import (
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
# Keep the cases in one experiment so their scores can be compared together.
async def eval_lifecycle_skills(ctx: EvalContext) -> None:
@@ -84,6 +86,7 @@ async def eval_lifecycle_skills(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-lifecycle-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py
index 5b4cac746250..b54d76c1bb31 100644
--- a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py
+++ b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py
@@ -27,6 +27,8 @@ from products.posthog_ai.evals.experiments.scorers import (
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
async def eval_metric_schema_discovery(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
@@ -53,6 +55,7 @@ async def eval_metric_schema_discovery(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-metric-schema-discovery-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_midrun_changes.py b/products/posthog_ai/evals/experiments/eval_midrun_changes.py
index 7db3ca178df9..9062c588fac6 100644
--- a/products/posthog_ai/evals/experiments/eval_midrun_changes.py
+++ b/products/posthog_ai/evals/experiments/eval_midrun_changes.py
@@ -63,6 +63,8 @@ from products.posthog_ai.evals.experiments.seeders import (
)
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
async def eval_midrun_changes(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
@@ -152,6 +154,7 @@ async def eval_midrun_changes(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-diagnose-midrun-changes-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py
index 4604a3d058e9..db3f2b2fbd23 100644
--- a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py
+++ b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py
@@ -45,6 +45,8 @@ from products.posthog_ai.evals.experiments.scorers import CitesDiagnosticGroup
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
async def eval_numbers_vs_sql(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
@@ -154,6 +156,7 @@ async def eval_numbers_vs_sql(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-diagnose-numbers-vs-sql-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_rollout_skill.py b/products/posthog_ai/evals/experiments/eval_rollout_skill.py
index 59170914ec04..021026a3d1ee 100644
--- a/products/posthog_ai/evals/experiments/eval_rollout_skill.py
+++ b/products/posthog_ai/evals/experiments/eval_rollout_skill.py
@@ -25,6 +25,8 @@ from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAM
from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.retrieval.scorers import SkillLoaded
+NO_SEND_LOGS = True
+
SKILL_NAME = "configuring-experiment-rollout"
@@ -40,6 +42,7 @@ async def eval_rollout_skill(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-rollout-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py
index 178b7f8bc1d8..3673915aaf4b 100644
--- a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py
+++ b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py
@@ -15,6 +15,8 @@ from products.posthog_ai.evals.experiments.scorers import SharedMetricValidation
from products.posthog_ai.evals.experiments.seeders import SHARED_METRIC_NAME, seed_shared_metric_purchase_count
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
async def eval_shared_metric_validation(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
@@ -50,6 +52,7 @@ async def eval_shared_metric_validation(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-shared-metric-validation-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py
index 9d58348b98a5..d748fd4858ee 100644
--- a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py
+++ b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py
@@ -42,6 +42,8 @@ from products.posthog_ai.evals.experiments.scorers import AskedForConfirmation
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
from products.posthog_ai.evals.offline_config import offline_suite
+NO_SEND_LOGS = True
+
# Event name chosen to be obviously fictional so the agent can't plausibly
# justify it as a typo of a real PostHog event in the seeded team.
UNKNOWN_EVENT_NAME = "totally_made_up_event_qzx"
@@ -66,6 +68,7 @@ async def eval_unknown_events_confirmation(ctx: EvalContext) -> None:
]
await SandboxedPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
experiment_name="sandboxed-experiments-unknown-events-cli",
send_to_posthog=True,
offline_suite=offline_suite(
diff --git a/products/posthog_ai/evals/offline_config.py b/products/posthog_ai/evals/offline_config.py
index 4479074fb0ec..0f6fd1df4bba 100644
--- a/products/posthog_ai/evals/offline_config.py
+++ b/products/posthog_ai/evals/offline_config.py
@@ -127,6 +127,16 @@ _SCORERS: dict[str, dict[str, OfflineScorer]] = {
"surfaces_all_findings": OfflineScorer(kind="boolean", version=UUID("01a1558f-ad52-7b81-8a93-d7e602bf0125")),
},
"product_analytics": {
+ "answer_tool_call_not_forbidden": OfflineScorer(
+ kind="boolean", version=UUID("01a15a27-7d06-768b-940c-f47b5f3ee78c")
+ ),
+ "did_not_cite_raw_sql_after_typed_query": OfflineScorer(
+ kind="boolean", version=UUID("01a15a27-735b-700b-9f5b-51533df620b3")
+ ),
+ "no_markdown_data_table": OfflineScorer(kind="boolean", version=UUID("01a15a27-69b4-7bc0-b6f6-b366acc272b2")),
+ "chart_from_typed_query": OfflineScorer(kind="boolean", version=UUID("01a15a27-5e27-7511-9f3e-58a57e74556d")),
+ "chart_shown": OfflineScorer(kind="boolean", version=UUID("01a15a27-553d-72d2-aeb2-903e7ed54473")),
+ "answer_from_typed_query": OfflineScorer(kind="boolean", version=UUID("01a15a27-4bce-7e6c-b3ef-34d292489b30")),
"answer_tool_not_execute_sql": OfflineScorer(
kind="boolean", version=UUID("01a15581-3926-7c14-a907-e994586bd216")
),
@@ -201,6 +211,9 @@ _SCORERS: dict[str, dict[str, OfflineScorer]] = {
kind="boolean", version=UUID("01a15582-7cea-7b35-9632-1fd1c3e22185")
),
},
+ "turn_suggestions": {
+ "offer_correctness": OfflineScorer(kind="boolean", version=UUID("01a15a29-ea9e-743e-9a21-cec75e5e417e")),
+ },
"taxonomic_filter": {
"event_match_found": OfflineScorer(kind="boolean", version=UUID("01a15582-8f96-76a8-a6b0-9ad9c932f12a")),
"exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15582-85bb-71aa-b4c8-dc008015ab30")),
diff --git a/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py b/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py
index de7efdda29ec..be88effa6500 100644
--- a/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py
+++ b/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py
@@ -34,6 +34,7 @@ from products.posthog_ai.evals.cli_mcp.scorers import (
DidNotCiteRawSqlAfterTypedQuery,
NoMarkdownDataTable,
)
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS, AnswerFromTypedQuery
@@ -93,6 +94,17 @@ async def eval_chart_not_table(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-chart-not-table-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_chart_not_table::eval_chart_not_table",
+ metrics=[
+ "no_persistent_insight_save",
+ "answer_from_typed_query",
+ "chart_shown",
+ "chart_from_typed_query",
+ "no_markdown_data_table",
+ "did_not_cite_raw_sql_after_typed_query",
+ ],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
diff --git a/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py b/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py
index 8b52c9c9580e..0222167831f2 100644
--- a/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py
+++ b/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py
@@ -35,6 +35,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall
from products.posthog_ai.evals.cli_mcp.scorers import FirstRelevantTool
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import (
INSIGHT_WRITE_TOOLS,
TYPED_QUERY_TOOLS,
@@ -164,6 +165,15 @@ async def eval_typed_runner_misses(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-typed-runner-misses-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_typed_runner_misses::eval_typed_runner_misses",
+ metrics=[
+ "no_persistent_insight_save",
+ "answer_from_typed_query",
+ "answer_tool_call_not_forbidden",
+ "insight_shape",
+ ],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
@@ -197,6 +207,10 @@ async def eval_justified_sql_controls(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-justified-sql-controls-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_typed_runner_misses::eval_justified_sql_controls",
+ metrics=["no_persistent_insight_save", "first_relevant_tool"],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
diff --git a/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.py b/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.py
new file mode 100644
index 000000000000..29693b85e68c
--- /dev/null
+++ b/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.py
@@ -0,0 +1,92 @@
+from __future__ import annotations
+
+import os
+import json
+import asyncio
+from typing import TYPE_CHECKING
+
+from products.posthog_ai.backend.turn_suggestions.benchmark import Outcome, judge_case, load_cases
+from products.posthog_ai.backend.turn_suggestions.classifier import SHOW_THRESHOLD
+from products.posthog_ai.backend.turn_suggestions.judgment import build_judge_state, judge_model
+from products.posthog_ai.eval_harness.config import BaseEvalCase
+from products.posthog_ai.eval_harness.engines.types import ScorerResult
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.one_shot import OneShotPrivateEval
+from products.posthog_ai.eval_harness.scorers.contract import Score, Scorer
+from products.posthog_ai.evals.offline_config import offline_suite
+
+if TYPE_CHECKING:
+ from products.posthog_ai.eval_harness.harness.context import EvalContext
+
+NO_SEND_LOGS = True
+
+SUITE_KIND = SuiteKind.ONE_SHOT
+BENCHMARK_CASES = {case.name: case for case in load_cases()}
+CASES = [
+ BaseEvalCase(
+ name=case.name,
+ prompt=json.dumps(build_judge_state(case.transcript)),
+ expected={"acceptable": sorted(case.acceptable)},
+ metadata={"category": case.category, "available": sorted(case.available)},
+ )
+ for case in BENCHMARK_CASES.values()
+]
+
+
+class OfferCorrectness(Scorer):
+ def _name(self) -> str:
+ return "offer_correctness"
+
+ def _run_eval_sync(
+ self, output: dict[str, object] | None, expected: dict[str, object] | None = None, **kwargs: object
+ ) -> Score:
+ outcome = (output or {}).get("outcome")
+ if outcome is None or outcome == Outcome.FAILED:
+ reason = (output or {}).get("error") or "Turn judge returned no judgment"
+ return Score(
+ name=self._name(),
+ score=None,
+ result=ScorerResult(status="error", error_code="judge_error", error_message=str(reason)),
+ )
+ return Score(
+ name=self._name(),
+ score=float(outcome in (Outcome.CORRECT, Outcome.EITHER)),
+ metadata={"reason": outcome},
+ )
+
+
+async def eval_turn_suggestions(ctx: EvalContext) -> None:
+ threshold = float(os.environ.get("TURN_SUGGESTIONS_EVAL_THRESHOLD", str(SHOW_THRESHOLD)))
+ if not 0 <= threshold <= 1:
+ raise ValueError("TURN_SUGGESTIONS_EVAL_THRESHOLD must be between 0 and 1")
+ model = judge_model()
+
+ async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict[str, object]:
+ result = await asyncio.to_thread(judge_case, BENCHMARK_CASES[case.name])
+ return {
+ "outcome": result.outcome(threshold),
+ "picked": result.picked(threshold),
+ "show_probability": result.judgment.show_probability if result.judgment else None,
+ "threshold": threshold,
+ "judge_model": model,
+ "seconds": result.seconds,
+ "error": result.error,
+ }
+
+ await OneShotPrivateEval(
+ no_send_logs=NO_SEND_LOGS,
+ experiment_name="turn-suggestions",
+ cases=[
+ case.model_copy(update={"metadata": {**case.metadata, "threshold": threshold, "judge_model": model}})
+ for case in CASES
+ ],
+ task=task,
+ scorers=[OfferCorrectness()],
+ offline_suite=offline_suite(
+ "turn_suggestions/eval_turn_suggestions::eval_turn_suggestions",
+ metrics=["offer_correctness"],
+ suite_kind=SUITE_KIND,
+ ),
+ send_to_posthog=True,
+ ctx=ctx,
+ )
diff --git a/products/slack_app/evals/eval_untagged_question.py b/products/slack_app/evals/eval_untagged_question.py
index 66f077c862da..39052dbf17ab 100644
--- a/products/slack_app/evals/eval_untagged_question.py
+++ b/products/slack_app/evals/eval_untagged_question.py
@@ -21,6 +21,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval
from products.slack_app.backend.temporal.activities import untagged_question
+from products.slack_app.evals.offline_config import offline_suite
from products.slack_app.evals.scorers import UNTAGGED_QUESTION_KEY, NoUnaskedAnswer, UntaggedQuestionMatch
SUITE_KIND = SuiteKind.ONE_SHOT
@@ -256,6 +257,11 @@ async def eval_untagged_question(ctx: EvalContext) -> None:
await OneShotPublicEval(
experiment_name="slack-app-untagged-question",
+ offline_suite=offline_suite(
+ "slack_app/eval_untagged_question::eval_untagged_question",
+ metrics=[UNTAGGED_QUESTION_KEY, "no_unasked_answer"],
+ suite_kind=SUITE_KIND,
+ ),
cases=[*ANSWERABLE_CASES, *LEAVE_ALONE_CASES],
scorers=[UntaggedQuestionMatch(), NoUnaskedAnswer()],
task=task,
diff --git a/products/slack_app/evals/offline_config.py b/products/slack_app/evals/offline_config.py
index b86cc92224c9..ae6c5fa1261b 100644
--- a/products/slack_app/evals/offline_config.py
+++ b/products/slack_app/evals/offline_config.py
@@ -36,6 +36,10 @@ _SCORERS: dict[str, dict[str, OfflineScorer]] = {
"reply_offer": OfflineScorer(kind="boolean", version=UUID("01a1557d-5266-7853-a431-6213cfba1b5a")),
"suggested_replies_fit": OfflineScorer(kind="boolean", version=UUID("01a1557d-bcc0-7a96-8d22-4d6e91998d5d")),
},
+ "untagged_question": {
+ "untagged_question": OfflineScorer(kind="boolean", version=UUID("01a15a43-79f8-76bd-98ad-aaacc69b7059")),
+ "no_unasked_answer": OfflineScorer(kind="boolean", version=UUID("01a15a43-8605-78ef-acf6-cc940d3876ab")),
+ },
}