:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(aio): publish tagged offline eval results to posthog (#119849)
Radu Raiceae341b726273a
diff --git a/.agents/skills/writing-evals/SKILL.md b/.agents/skills/writing-evals/SKILL.mdindex 06cfd02ec133..f27747bb113f 100644--- a/.agents/skills/writing-evals/SKILL.md+++ b/.agents/skills/writing-evals/SKILL.md@@ -48,7 +48,9 @@ async def eval_my_thing(ctx: EvalContext) -> None:- One suite = one Braintrust experiment. Bundle related cases into one suite; split only when the scorecard would be heterogeneous (self-skipping scorers stretch a shared scorer list across mildly different cases, but a scorecard where half the scorers skip half the cases is a sign to split).- `experiment_name` is the Braintrust history key — renaming it resets cross-run comparison. Existing sandboxed suites end in `-cli` because the MCP server serves the `cli` surface.-- `SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust and PostHog, including available input, output, and expected values. `SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service; local logs are still written. PostHog result uploads follow `no_send_logs` independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md).+- `SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust, plus PostHog when `offline_suite` is configured. `SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service by default; local logs are still written. Set `send_to_posthog=True` with `offline_suite` to publish only to PostHog, or `send_to_posthog=False` to disable that destination independently. These options also apply to one-shot suites. Ordinary SDK and trace clients still honor `OPT_OUT_CAPTURE`. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md).+- To enroll a suite in PostHog reporting, register its scorer kinds and immutable version UUIDs in the owning `offline_config.py` catalog, then pass `offline_suite=offline_suite("<domain>/<module>::<function>", metrics=[...])` to the runner. Use the PostHog AI catalog for built-in suites and the Slack app catalog for Slack suites. List every explicit scorer's `_name()` in `metrics`; the catalog adds `exit_code_zero` only for sandboxed suites. For one-shot suites, pass `suite_kind=SUITE_KIND`. The enrollment test discovers catalog-using modules and checks every suite function against its scorer list.+- Uploading and tagging require `POSTHOG_OFFLINE_EVAL_API_KEY` to be a personal key with `offline_evaluation_ingestion:write` and `evaluation:write`. Ingestion-only personal or project secret keys can upload without tags. Upload success is recorded before the separate, bounded tag update, so a tagging failure or interruption keeps the results link.## One-shot suites@@ -135,7 +137,7 @@ class MyCheck(Scorer):Rules that apply to both:-- The harness adds `ExitCodeZero` to every experiment. Do not include it in a suite's `scorers`; an explicit duplicate is rejected.+- The sandboxed harness adds `ExitCodeZero` to every experiment. Do not include it in a suite's `scorers`; an explicit duplicate is rejected. One-shot suites do not receive it.- Reuse the other generic scorers first: `NoToolCall`, `RequiredToolCall`, and `LastToolCallNot` (all in `products.posthog_ai.eval_harness.scorers`).- Always parse logs via `LogParser.cached(...)` so a case's log is parsed once, not once per scorer.- Count only successful tool calls (`is_error=False`) — the agent is free to attempt and fail.diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.mdindex 4f13ce74d02c..450f46879499 100644--- a/docs/internal/ai-offline-evaluation-reporting.md+++ b/docs/internal/ai-offline-evaluation-reporting.md@@ -1,7 +1,7 @@# Offline evaluation result reportingThe evaluation harness in `products/posthog_ai/eval_harness/` reports public-suite scorer results to Braintrust.-The SQL suite also publishes those results to PostHog's offline experiment API when configured.+Public suites under `products/posthog_ai/evals/` and `products/slack_app/evals/` also publish those results to PostHog's offline experiment API when configured.The Signals pytest suites reuse the same publisher for their completed results.Agent and judge trace capture remains available.Reporting does not run the agent or scorers again.@@ -9,9 +9,11 @@ Use `hogli evals` to run PostHog AI 2.0 suites. See the [eval harness guide](../## Capture settings-Evaluation result uploads to Braintrust and PostHog share the `no_send_logs` setting.+`no_send_logs` controls Braintrust uploads and, by default, PostHog result uploads.`SandboxedPublicEval` sets `no_send_logs=False` and uploads to Braintrust, plus PostHog when the suite has offline publishing configured.-`SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service; local logs are still written.+`SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service by default; local logs are still written.+Set `send_to_posthog=True` alongside `offline_suite` to publish results to PostHog while keeping Braintrust uploads disabled.+Set `send_to_posthog=False` to disable PostHog result uploads independently. The same options apply to one-shot suites.Offline API uploads run through the Python SDK independently of `TEST` and `OPT_OUT_CAPTURE`.The shared trace client retains its existing `TEST` and `OPT_OUT_CAPTURE` guards for `$ai_generation`, `$ai_span`, and `$ai_trace` events.@@ -33,25 +35,27 @@ The scoring phase includes time waiting for the experiment results, rather thanA failed final metadata upload produces a warning and keeps the completed results available locally.A scorer failure marks the case as an error. Statistics still include scores from other successful scorers.-## SQL offline experiment pilot+## Configured offline experiment suites-The `sql/eval_sql::eval_sql` suite (`sandboxed-sql-cli`) opts into the offline experiment publisher.-Other public suites continue reporting scores to Braintrust until they opt into offline publishing.+Configured public PostHog AI and Slack app suites opt into the offline experiment publisher, including both sandboxed and one-shot suites.+The 11 experiment-skill suites retain `no_send_logs=True` and explicitly set `send_to_posthog=True`, publishing results only to PostHog.After scoring, the harness prepares the completed results in memory and queues delivery through the Python SDK's `AsyncOfflineEvaluations` client. It does not execute another agent run or judge call.-`no_send_logs=True` suppresses this upload too.+Without an explicit `send_to_posthog` override, `no_send_logs=True` suppresses this upload too.-The [SQL suite definition](../../products/posthog_ai/evals/sql/eval_sql.py) pins destination project `2` and the seven immutable scorer version UUIDs.+The [PostHog AI catalog](../../products/posthog_ai/evals/offline_config.py) and [Slack app catalog](../../products/slack_app/evals/offline_config.py) pin destination project `2`, scorer kinds, immutable version UUIDs, and component tags.+Suites select their own metrics from the catalog; sandboxed suites also include the harness-provided `exit_code_zero` metric.+The SQL suite retains its seven existing scorer versions.Provide the API key through the environment before running the suite:-| Variable | Purpose |-| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |-| `POSTHOG_OFFLINE_EVAL_API_KEY` | Personal or project secret API key with `offline_evaluation_ingestion:write`. A public project token cannot authenticate uploads. |-| `POSTHOG_OFFLINE_EVAL_HOST` | App origin; defaults to `https://us.posthog.com`. |+| Variable | Purpose |+| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `POSTHOG_OFFLINE_EVAL_API_KEY` | Personal API key with `offline_evaluation_ingestion:write` and `evaluation:write` to upload and tag results. Ingestion-only personal or project secret keys can upload without tags. A public project token cannot authenticate uploads. |+| `POSTHOG_OFFLINE_EVAL_HOST` | App origin; defaults to `https://us.posthog.com`. |The publisher normalizes the cloud ingestion hosts (`us.i.posthog.com`, `eu.i.posthog.com`) and the legacy `app.posthog.com` host to their app origins.Uploads, result links, and trace-destination checks use that same origin.-The suite uses these existing scorers:+The SQL suite uses these existing scorers:| Metric name | Scorer kind || ------------------------------------ | -------------------------------- |@@ -72,15 +76,32 @@ export POSTHOG_OFFLINE_EVAL_API_KEY='your-secret-key'hogli evals sql/eval_sql::eval_sql```-Without an API key, the SQL run reports that offline uploads are disabled.+Without an API key, each enrolled suite reports that offline uploads are disabled.Keys containing whitespace are rejected before evaluation starts; remove any spaces, tabs, or pasted trailing newlines.Invalid configuration produces a warning and skips this destination; Braintrust and local reporting continue.An upload failure also warns without changing the evaluation's score gate.The final summary shows a PostHog results link only after a successful offline upload; otherwise it shows the upload status.+### Scorer and experiment tags++Score definitions and experiment runs use the same team and component tags:++| Suites | Tags |+| -------------------- | ------------------------------------------------------------------------------------------- |+| PostHog AI | `team:surfaces`, `component:posthog-ai/<domain>` (for example `sql` or `product-analytics`) |+| Slack app | `team:surfaces`, `component:slack-app/<area>` (for example `followup-classifier`) |+| Signals grouping | `team:self-driving`, `component:signals/grouping` |+| Signals scout safety | `team:self-driving`, `component:signals/scout-safety` |++`OfflineEvalSuite.tags` supplies the experiment tags. Delivery records the successful upload and results link once the experiment has completed, then calls the publisher's `tag()` method to update tags through the Python SDK's `experiment.set_tags(...)`. Tagging has its own five-second deadline outside the upload deadline. Tags are mutable metadata, separate from the immutable creation declaration and scorer versions.++To upload **and tag** experiments, `POSTHOG_OFFLINE_EVAL_API_KEY` must be a personal API key with both `offline_evaluation_ingestion:write` and `evaluation:write`. An ingestion-only personal or project secret key can still upload results, but the tag update will warn. Tagging failures, timeouts, and shutdown interruptions retain the successful upload status and results link and do not change the evaluation score gate.++Each prepared experiment's tag update is attempted once per publisher, including when delivery pauses and resumes. A failed or interrupted replacement is not repeated automatically, because it may already have succeeded or someone may have edited the tags afterward. Inspect the experiment's tags before manually retrying a replacement.+### CI configuration-CI jobs running this suite need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment.+CI jobs running enrolled suites need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment.Map the dedicated `ANTHROPIC_OFFLINE_EVALS_API_KEY` and `OPENAI_OFFLINE_EVALS_API_KEY` Actions secrets to `LLM_GATEWAY_ANTHROPIC_API_KEY` and `OPENAI_API_KEY` for the harness and judges.Publishing failures remain warnings, so check the final harness upload status even when a CI job is green.The harness sets the SDK's `run_source` to `ci` when `CI` is set, including manual Actions runs, and to `local` otherwise.@@ -124,6 +145,7 @@ Existing local logs, enabled case exports, and Braintrust results remain availabBoth `SandboxedEval` and `OneShotEval` accept `offline_suite=OfflineEvalSuite(...)`.Each suite declares its destination `project_id`, selects at least one metric in `scorer_kinds`, and pins the corresponding immutable version UUIDs in `scorer_versions`.These fields belong to `OfflineEvalSuite` in code. Scorer version UUIDs must be distinct for the suite's enrolled metrics.+The AI and Slack catalog helpers add `exit_code_zero` for sandboxed suites. One-shot suites pass `suite_kind=SUITE_KIND` to enroll only their declared metrics.Numeric scores can use the domain of their configured scorer version. The API validates bounds and steps; the harness does not restrict every numeric metric to `[0, 1]`.`OfflineEvalSuite.experiment_options` accepts the SDK's `ExperimentOptions` for external dataset identifiers, `prompt_version`, model/application versions, and `run_source`.@@ -185,7 +207,7 @@ Runs appear in the offline experiments UI.## Postgres experiment ingestionThe project API accepts offline experiment results behind the `ai-observability-offline-evaluations` feature flag.-The configured SQL pilot calls this API to publish completed scorer results.+The configured public suites call this API to publish completed scorer results.Experiments and their items, results, and payloads belong to the exact project/environment in the request path.Scorers and hosted datasets must belong to that same environment.diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.mdindex 87a88875a9c1..755d2b98c1ca 100644--- a/products/posthog_ai/eval_harness/README.md+++ b/products/posthog_ai/eval_harness/README.md@@ -387,22 +387,25 @@ The summary includes the trial ID, the agent trace ID when available, phase duraCase names and `--eval` filters stay the same.`SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust, plus PostHog when the suite has offline publishing configured.-`SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service; local logs are still written.-PostHog result uploads follow `no_send_logs` independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients.+`SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service by default; local logs are still written.+Both sandboxed and one-shot suites can set `send_to_posthog=True` with `offline_suite` to publish only to PostHog while Braintrust uploads remain disabled, or `send_to_posthog=False` to disable PostHog uploads independently.+The 11 experiment-skill suites use this PostHog-only mode.+PostHog result uploads default to following `no_send_logs`, independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients.See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md) for capture settings and scope.### SQL offline experiment pilotThe `sql/eval_sql::eval_sql` suite can also send its seven scorer results to PostHog's offline experiment API.It publishes the same completed case results through the Python SDK and keeps sending to Braintrust.-Other public suites report scores to Braintrust until they opt into offline publishing.-Agent and judge trace capture remains available, and `no_send_logs=True` suppresses offline result uploads.+PostHog AI and Slack app public suites also opt into offline publishing with their own tagged scorer versions.+Agent and judge trace capture remains available. `no_send_logs=True` suppresses offline result uploads unless `send_to_posthog=True` is explicitly set.-Set `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal or project secret key with `offline_evaluation_ingestion:write` and access to project `2`.+Set `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal key with `offline_evaluation_ingestion:write`, `evaluation:write`, and access to project `2` to upload and tag results. An ingestion-only personal or project secret key can upload results, but tagging will warn and leave the run untagged.The [SQL suite definition](../evals/sql/eval_sql.py) pins that project ID and all seven immutable scorer version UUIDs in code.`POSTHOG_OFFLINE_EVAL_HOST` defaults to `https://us.posthog.com`.-The [pilot configuration](../../../docs/internal/ai-offline-evaluation-reporting.md#sql-offline-experiment-pilot) lists the metrics and explains local and CI key configuration.+The [suite configuration](../../../docs/internal/ai-offline-evaluation-reporting.md#configured-offline-experiment-suites) lists the metrics and explains local and CI key configuration.Delivery runs in a bounded background queue after local results are saved. Prepared uploads stay in memory, with up to two uploads running concurrently, at most 32 queued or active uploads, and a 60-second deadline per active upload.+A completed upload records its results link before tagging begins. Tagging has a separate five-second deadline; failures, timeouts, and shutdown interruptions only warn and preserve the successful upload status.Unfinished uploads pause before infrastructure teardown. After teardown, the harness retries the same prepared objects and identities with fresh deadlines, waiting up to the full 60-second upload deadline for delivery to finish.Requests use a 5-second timeout and up to two retries within that overall deadline. Terminal API failures trigger a best-effort attempt to mark the experiment failed, limited to 5 seconds; pauses and interruptions leave it resumable.Both sandboxed and one-shot suites can opt in through `offline_suite`. Rich scorer recording runs only when offline publishing is configured and uploads are allowed.diff --git a/products/posthog_ai/eval_harness/base.py b/products/posthog_ai/eval_harness/base.pyindex 663c38261065..6f9d5047f92a 100644--- a/products/posthog_ai/eval_harness/base.py+++ b/products/posthog_ai/eval_harness/base.py@@ -166,12 +166,14 @@ class _BaseEvalRun:no_send_logs: bool,engine: EvalEngine | None = None,offline_suite: OfflineEvalSuite | None = None,+ send_to_posthog: bool | None = None,) -> None:self.experiment_name = experiment_nameself.cases = casesself.ctx = ctxself.is_public = is_publicself.no_send_logs = no_send_logs+ self.send_to_posthog = not no_send_logs if send_to_posthog is None else send_to_posthogself.offline_suite = offline_suiteself.offline_publisher: OfflineEvalPublisher | None = Noneself.started_at = datetime.now(UTC)@@ -394,7 +396,7 @@ class _BaseEvalRun:await self._enqueue_offline_upload(result)async def _enqueue_offline_upload(self, result: ExperimentResult) -> None:- if self.offline_publisher is None or self.no_send_logs:+ if self.offline_publisher is None or not self.send_to_posthog:returntry:upload = await asyncio.to_thread(@@ -471,7 +473,7 @@ class _BaseEvalRun:async def run(self) -> ExperimentResult:eval_cases = self._build_eval_cases()- if self.offline_suite is not None and not self.no_send_logs:+ if self.offline_suite is not None and self.send_to_posthog:try:settings = OfflineEvalSettings.from_env(self.offline_suite)if settings is not None:@@ -607,6 +609,7 @@ class _SandboxedEvalRun(_BaseEvalRun):is_public: bool,no_send_logs: bool,offline_suite: OfflineEvalSuite | None = None,+ send_to_posthog: bool | None = None,) -> None:if any(isinstance(scorer, ExitCodeZero) for scorer in scorers):raise ValueError("ExitCodeZero is added by the sandboxed eval harness; remove it from scorers")@@ -618,6 +621,7 @@ class _SandboxedEvalRun(_BaseEvalRun):is_public=is_public,no_send_logs=no_send_logs,offline_suite=offline_suite,+ send_to_posthog=send_to_posthog,)# Narrow the infra-backed optionals once: they are None only when the# harness didn't boot sandbox infrastructure for this run.@@ -864,6 +868,7 @@ async def SandboxedEval(is_public: bool = False,no_send_logs: bool = True,offline_suite: OfflineEvalSuite | None = None,+ send_to_posthog: bool | None = None,) -> ExperimentResult:"""Run a sandboxed agent evaluation suite via Braintrust.@@ -888,6 +893,7 @@ async def SandboxedEval(is_public=is_public,no_send_logs=no_send_logs,offline_suite=offline_suite,+ send_to_posthog=send_to_posthog,)return await run.run()diff --git a/products/posthog_ai/eval_harness/harness/offline_delivery.py b/products/posthog_ai/eval_harness/harness/offline_delivery.pyindex ab8eb84d9ab9..232d31a8c0c7 100644--- a/products/posthog_ai/eval_harness/harness/offline_delivery.py+++ b/products/posthog_ai/eval_harness/harness/offline_delivery.py@@ -66,6 +66,7 @@ class OfflineEvalDelivery:await self._report_failed(job, reason)else:await self._reporter.record_posthog_offline_result(job.upload.experiment_name, "uploaded", url=url)+ await job.publisher.tag(job.upload)def _finished(self, task: asyncio.Task[None]) -> None:self._tasks.pop(task, None)diff --git a/products/posthog_ai/eval_harness/offline_results.py b/products/posthog_ai/eval_harness/offline_results.pyindex 5cfe9e1dcad9..02d4bc5014f9 100644--- a/products/posthog_ai/eval_harness/offline_results.py+++ b/products/posthog_ai/eval_harness/offline_results.py@@ -4,6 +4,7 @@ import osimport reimport jsonimport asyncio+import loggingfrom collections import Counterfrom collections.abc import Mappingfrom contextlib import suppress@@ -30,6 +31,15 @@ from posthog.dataclasses import frozenfrom .engines.types import CaseResult, ExperimentResult, ScorerResult+logger = logging.getLogger(__name__)+TAG_TIMEOUT_SECONDS = 5.0+++@frozen+class OfflineScorer:+ kind: ScorerKind+ version: UUID+@frozenclass OfflineEvalSuite:@@ -37,8 +47,21 @@ class OfflineEvalSuite:project_id: intscorer_kinds: Mapping[str, ScorerKind]scorer_versions: Mapping[str, UUID]+ tags: tuple[str, ...] = ()experiment_options: ExperimentOptions = field(default_factory=lambda: ExperimentOptions())+ @classmethod+ def from_scorers(+ cls, *, key: str, project_id: int, scorers: Mapping[str, OfflineScorer], tags: tuple[str, ...]+ ) -> OfflineEvalSuite:+ return cls(+ key=key,+ project_id=project_id,+ scorer_kinds={metric: scorer.kind for metric, scorer in scorers.items()},+ scorer_versions={metric: scorer.version for metric, scorer in scorers.items()},+ tags=tags,+ )+class OfflineUploadError(Exception):@staticmethod@@ -163,6 +186,7 @@ class OfflineEvalUpload:experiment_name: strexperiment: ExperimentOptionsresults: tuple[EvaluationResult, ...]+ tags: tuple[str, ...] = ()class OfflineEvalPublisher:@@ -176,6 +200,7 @@ class OfflineEvalPublisher:self.settings = settingsself.suite = suiteself.application_trace_ids_enabled = application_trace_ids_enabled+ self._attempted_tag_updates: set[str] = set()def _item_payload(self,@@ -377,19 +402,54 @@ class OfflineEvalPublisher:experiment_name=experiment_name,experiment=experiment,results=tuple(scores),+ tags=self.suite.tags,)+ def _client(self) -> AsyncOfflineEvaluations:+ return AsyncOfflineEvaluations(+ host=self.settings.host,+ project_id=self.settings.project_id,+ secret_key=self.settings.api_key.get_secret_value(),+ timeout=self.settings.request_timeout,+ max_retries=self.settings.max_retries,+ )++ async def tag(self, upload: OfflineEvalUpload) -> None:+ if upload.host != self.settings.host or upload.project_id != self.settings.project_id:+ raise OfflineUploadError("Upload destination differs from the configured host/project")+ experiment_id = str(upload.experiment["id"])+ if not upload.tags or experiment_id in self._attempted_tag_updates:+ return+ self._attempted_tag_updates.add(experiment_id)+ try:+ async with asyncio.timeout(TAG_TIMEOUT_SECONDS), self._client() as client:+ await client.resume_experiment(experiment_id).set_tags(upload.tags)+ except EvaluationAPIError as error:+ logger.warning(+ "PostHog offline results uploaded, but tagging failed for experiment %s (status=%s, code=%s). "+ "Tagging requires a personal key with evaluation:write. Tag updates are not retried.",+ experiment_id,+ error.status,+ OfflineUploadError._safe_token(error.code, path=False),+ )+ except TimeoutError:+ logger.warning(+ "PostHog offline results uploaded, but tagging timed out for experiment %s. "+ "Tag updates are not retried.",+ experiment_id,+ )+ except asyncio.CancelledError:+ logger.warning(+ "PostHog offline tag update interrupted for experiment %s; inspect its tags before retrying.",+ experiment_id,+ )+ raise+async def publish(self, upload: OfflineEvalUpload) -> str:if upload.host != self.settings.host or upload.project_id != self.settings.project_id:raise OfflineUploadError("Upload destination differs from the configured host/project")try:- async with AsyncOfflineEvaluations(- host=self.settings.host,- project_id=self.settings.project_id,- secret_key=self.settings.api_key.get_secret_value(),- timeout=self.settings.request_timeout,- max_retries=self.settings.max_retries,- ) as client:+ async with self._client() as client:experiment = await client.create_experiment(name=upload.experiment_name, **upload.experiment)try:await experiment.upload_results(upload.results)diff --git a/products/posthog_ai/eval_harness/one_shot.py b/products/posthog_ai/eval_harness/one_shot.pyindex d7b79a8dfa6f..ec3219a59ad3 100644--- a/products/posthog_ai/eval_harness/one_shot.py+++ b/products/posthog_ai/eval_harness/one_shot.py@@ -52,6 +52,7 @@ class _OneShotEvalRun(_BaseEvalRun):no_send_logs: bool,task_fn: OneShotTaskFn,offline_suite: OfflineEvalSuite | None = None,+ send_to_posthog: bool | None = None,) -> None:super().__init__(experiment_name=experiment_name,@@ -61,6 +62,7 @@ class _OneShotEvalRun(_BaseEvalRun):is_public=is_public,no_send_logs=no_send_logs,offline_suite=offline_suite,+ send_to_posthog=send_to_posthog,)self._task_fn = task_fn@@ -114,6 +116,7 @@ async def OneShotEval(is_public: bool = False,no_send_logs: bool = True,offline_suite: OfflineEvalSuite | None = None,+ send_to_posthog: bool | None = None,) -> ExperimentResult:"""Run a one-shot evaluation suite via Braintrust.@@ -132,6 +135,7 @@ async def OneShotEval(no_send_logs=no_send_logs,task_fn=task,offline_suite=offline_suite,+ send_to_posthog=send_to_posthog,)return await run.run()@@ -140,4 +144,4 @@ OneShotPublicEval = partial(OneShotEval, is_public=True, no_send_logs=False)"""One-shot evaluation whose Braintrust experiment is publicly accessible."""OneShotPrivateEval = partial(OneShotEval, is_public=False, no_send_logs=True)-"""One-shot evaluation that is not accessible publicly; local logs are the record."""+"""One-shot evaluation with Braintrust uploads disabled; PostHog publishing is opt-in."""diff --git a/products/posthog_ai/eval_harness/test/offline_api.py b/products/posthog_ai/eval_harness/test/offline_api.pyindex f1e75ada44bf..df16412d148e 100644--- a/products/posthog_ai/eval_harness/test/offline_api.py+++ b/products/posthog_ai/eval_harness/test/offline_api.py@@ -13,10 +13,14 @@ class OfflineServer:self.results: dict[tuple[str, str], object] = {}self.experiment: dict[str, object] = {}self.status = "uploading"+ self.tags: list[str] = []def respond(self, request: httpx.Request) -> httpx.Response:self.requests.append(request)body = json.loads(request.content)+ if request.method == "PATCH":+ self.tags = body["tags"]+ return httpx.Response(200, json={"id": self.experiment["id"], "tags": self.tags})now = str(self.experiment.get("started_at", body.get("started_at")))if request.url.path.endswith("/upload/"):item_receipts = []diff --git a/products/posthog_ai/eval_harness/test/test_offline_delivery.py b/products/posthog_ai/eval_harness/test/test_offline_delivery.pyindex db7746051d82..a0f07f641c16 100644--- a/products/posthog_ai/eval_harness/test/test_offline_delivery.py+++ b/products/posthog_ai/eval_harness/test/test_offline_delivery.py@@ -15,6 +15,7 @@ import httpxfrom posthoganalytics.ai.evaluations import EvaluationItem, EvaluationResultfrom pydantic import SecretStr+from products.posthog_ai.eval_harness import offline_resultsfrom products.posthog_ai.eval_harness.engines.types import EvalSummaryfrom products.posthog_ai.eval_harness.harness import reportingfrom products.posthog_ai.eval_harness.harness.offline_delivery import OfflineEvalDelivery@@ -198,6 +199,61 @@ async def test_finished_uploads_are_not_retried_or_marked_interrupted_while_repoassert "403" in output[email protected]("shutdown", ["pause", "deadline", "interrupt", "tag_timeout"])+async def test_completed_upload_stays_successful_when_tagging_is_interrupted(+ monkeypatch: pytest.MonkeyPatch,+ capsys: pytest.CaptureFixture[str],+ caplog: pytest.LogCaptureFixture,+ shutdown: str,+) -> None:+ monkeypatch.setattr(offline_results.logger, "disabled", False)+ caplog.set_level("WARNING", logger=offline_results.__name__)+ tag_started = asyncio.Event()+ release_tag = asyncio.Event()+ requests: list[httpx.Request] = []+ server = OfflineServer()++ async def respond(request: httpx.Request) -> httpx.Response:+ requests.append(request)+ if request.method == "PATCH":+ tag_started.set()+ await release_tag.wait()+ return server.respond(request)++ publisher = _publisher(monkeypatch, respond)+ reporter = ProgressReporter(total_suites=1)+ await reporter.record_summary(+ "experiment-1", EvalSummary(engine_name="braintrust", experiment_name="experiment-1", scores={})+ )+ delivery = OfflineEvalDelivery(reporter)+ if shutdown == "tag_timeout":+ monkeypatch.setattr(offline_results, "TAG_TIMEOUT_SECONDS", 0)+ assert delivery.submit(publisher, replace(_upload(1), tags=("team:example",)))+ if shutdown != "tag_timeout":+ await asyncio.wait_for(tag_started.wait(), timeout=5)+ if shutdown == "pause":+ await delivery.pause()+ release_tag.set()+ elif shutdown == "deadline":+ await delivery.drain(timeout=0)+ else:+ await delivery.cancel()+ await delivery.drain()++ reporter.print_final_summary([], exit_code=0, fail_under=None, duration_seconds=0)+ output = capsys.readouterr().out+ assert server.status == "completed"+ assert sum(request.url.path.endswith("/complete/") for request in requests) == 1+ assert output.count("POSTHOG OFFLINE") == 1+ assert "PostHog: https://posthog.example.com/project/123/" in output+ assert "upload failed" not in output+ if shutdown == "tag_timeout":+ assert "tagging timed out" in caplog.text+ elif shutdown != "pause":+ assert "tag update interrupted" in caplog.text++@pytest.mark.asyncio@pytest.mark.parametrize("shutdown", ["deadline", "interrupt", "before_start", "paused"])async def test_shutdown_cancels_active_and_waiting_uploads_and_reports_unsaved_results(diff --git a/products/posthog_ai/eval_harness/test/test_offline_results.py b/products/posthog_ai/eval_harness/test/test_offline_results.pyindex 83436608c9e5..4e3ae913157c 100644--- a/products/posthog_ai/eval_harness/test/test_offline_results.py+++ b/products/posthog_ai/eval_harness/test/test_offline_results.py@@ -39,10 +39,10 @@ SUITE = OfflineEvalSuite(STARTED_AT = datetime(2026, 1, 1, tzinfo=UTC)-def _publisher(*, link_traces: bool = False) -> OfflineEvalPublisher:+def _publisher(*, link_traces: bool = False, suite: OfflineEvalSuite = SUITE) -> OfflineEvalPublisher:return OfflineEvalPublisher(OfflineEvalSettings(project_id=123, api_key=SecretStr("test-only-token"), scorer_versions=SCORER_VERSIONS),- SUITE,+ suite,application_trace_ids_enabled=link_traces,)@@ -368,6 +368,58 @@ def test_publish_retries_identical_batches_and_completes_after_all_acknowledgmenassert url.endswith(f"/ai-evals/evaluations/offline/experiments/{EXPERIMENT_ID}")+ [+ ("success", None),+ ("forbidden", 403),+ ("server_error", 503),+ ("lost_response", "lost_response"),+ ("interrupted", "interrupted"),+ ]+)+def test_publish_tags_completed_results_without_repeating_mutable_updates(+ _name: str, tag_error: int | str | None+) -> None:+ publisher = _publisher(suite=replace(SUITE, tags=("team:example", "component:example")))+ upload = _prepare(publisher, [_case()])+ server = OfflineServer()+ tag_requests: list[httpx.Request] = []++ def respond(request: httpx.Request) -> httpx.Response:+ if request.method == "PATCH":+ tag_requests.append(request)+ assert server.status == "completed"+ if isinstance(tag_error, int):+ return httpx.Response(tag_error)+ response = server.respond(request)+ if tag_error == "lost_response":+ raise httpx.ReadError("Lost tag acknowledgment", request=request)+ if tag_error == "interrupted":+ raise asyncio.CancelledError+ return response+ return server.respond(request)++ with patch("httpx.AsyncClient", new=partial(httpx.AsyncClient, transport=httpx.MockTransport(respond))):+ assert asyncio.run(publisher.publish(upload)).endswith(EXPERIMENT_ID)+ if tag_error == "interrupted":+ with pytest.raises(asyncio.CancelledError):+ asyncio.run(publisher.tag(upload))+ else:+ asyncio.run(publisher.tag(upload))+ assert server.status == "completed"+ assert len(server.results) == 2+ assert "tags" not in server.experiment+ assert server.tags == ([] if isinstance(tag_error, int) else list(upload.tags))++ server.tags = ["edited-after-upload"]+ asyncio.run(publisher.publish(upload))+ asyncio.run(publisher.tag(upload))++ assert len(tag_requests) == 1+ assert json.loads(tag_requests[0].content) == {"tags": list(upload.tags)}+ assert server.tags == ["edited-after-upload"]++@parameterized.expand([("bad_request", 400),diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.pyindex 19d29c050f55..bf1d6a95cd7a 100644--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py@@ -7,6 +7,7 @@ import jsonimport asyncioimport loggingimport warnings+import importlibimport subprocessfrom collections.abc import Iteratorfrom contextlib import contextmanager@@ -41,15 +42,16 @@ from products.posthog_ai.eval_harness.engines.types import ()from products.posthog_ai.eval_harness.harness import reportingfrom products.posthog_ai.eval_harness.harness.cli import SkillDelivery+from products.posthog_ai.eval_harness.harness.discovery import EvalSuite, discover_suitesfrom products.posthog_ai.eval_harness.harness.offline_delivery import OfflineEvalDeliveryfrom products.posthog_ai.eval_harness.harness.reporting import ProgressReporter, SuiteRunResult+from products.posthog_ai.eval_harness.harness.requirements import SuiteKindfrom products.posthog_ai.eval_harness.harness.transcript import RunTranscriptfrom products.posthog_ai.eval_harness.harness.trial_stats import ScorerTrialStatsfrom products.posthog_ai.eval_harness.offline_results import OfflineEvalSuitefrom products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.posthog_ai.eval_harness.scorers import ExitCodeZerofrom products.posthog_ai.eval_harness.test.offline_api import OfflineServer-from products.posthog_ai.evals.sql import eval_sql as sql_evalfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext@@ -379,17 +381,22 @@ def test_sandboxed_eval_run_adds_exit_code_scorer(monkeypatch: pytest.MonkeyPatc@pytest.mark.parametrize(- "suite_kind,enrolled,no_send_logs,configuration,http_status,delivery_state",+ "suite_kind,enrolled,no_send_logs,send_to_posthog,posthog_enabled,configuration,http_status,delivery_state",[- ("sandboxed", True, False, "valid", 200, "ready"),- ("sandboxed", True, False, "valid", 403, "ready"),- ("sandboxed", True, False, "invalid", 200, "ready"),- ("sandboxed", True, False, "absent", 200, "ready"),- ("sandboxed", True, True, "valid", 200, "ready"),- ("sandboxed", False, False, "valid", 200, "ready"),- ("one-shot", True, False, "valid", 200, "ready"),- ("sandboxed", True, False, "valid", 200, "unavailable"),- ("sandboxed", True, False, "valid", 200, "closed"),+ ("sandboxed", True, False, None, True, "valid", 200, "ready"),+ ("sandboxed", True, False, None, True, "valid", 403, "ready"),+ ("sandboxed", True, False, None, True, "invalid", 200, "ready"),+ ("sandboxed", True, False, None, True, "absent", 200, "ready"),+ ("sandboxed", True, True, None, False, "valid", 200, "ready"),+ ("sandboxed", False, False, None, True, "valid", 200, "ready"),+ ("one-shot", True, False, None, True, "valid", 200, "ready"),+ ("sandboxed", True, False, None, True, "valid", 200, "unavailable"),+ ("sandboxed", True, False, None, True, "valid", 200, "closed"),+ ("sandboxed", True, True, True, True, "valid", 200, "ready"),+ ("one-shot", True, True, True, True, "valid", 200, "ready"),+ ("sandboxed", True, False, False, False, "valid", 200, "ready"),+ ("sandboxed", True, True, True, True, "absent", 200, "ready"),+ ("sandboxed", False, True, True, True, "valid", 200, "ready"),],)@pytest.mark.asyncio@@ -400,6 +407,8 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_reposuite_kind: Literal["sandboxed", "one-shot"],enrolled: bool,no_send_logs: bool,+ send_to_posthog: bool | None,+ posthog_enabled: bool,configuration: Literal["valid", "invalid", "absent"],http_status: int,delivery_state: Literal["ready", "unavailable", "closed"],@@ -473,6 +482,7 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repoctx=ctx,is_public=not no_send_logs,no_send_logs=no_send_logs,+ send_to_posthog=send_to_posthog,offline_suite=OfflineEvalSuite(key="example/suite",project_id=123,@@ -494,14 +504,14 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repoengine.run_experiment.assert_awaited_once()engine_spec = engine.run_experiment.call_args.args[0]assert engine_spec.no_send_logs is no_send_logs- assert engine_spec.record_scorer_results is (enrolled and not no_send_logs and configuration == "valid")+ assert engine_spec.record_scorer_results is (enrolled and posthog_enabled and configuration == "valid")assert reporter.mean_score() == 1.0reporter.print_final_summary([], exit_code=0, fail_under=None, duration_seconds=0)output = capsys.readouterr().outassert "Braintrust: https://braintrust.example.com/experiment/test" in outputassert "exit_code_zero: 100.0%" in outputassert not (tmp_path / "offline-pilot" / "latest" / "posthog-offline-upload.json").exists()- if enrolled and not no_send_logs and configuration == "valid" and delivery_state == "ready":+ if enrolled and posthog_enabled and configuration == "valid" and delivery_state == "ready":declaration = json.loads(requests[0].content)assert declaration["name"] == "offline-pilot"assert declaration["expected_item_count"] == 1@@ -530,34 +540,69 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repo)assert reason in outputassert "Results were not saved for retry." in output- elif enrolled and not no_send_logs:+ elif enrolled and posthog_enabled:assert "PostHog offline: disabled:" in outputassert "POSTHOG_OFFLINE_EVAL_" in outputelse:assert "POSTHOG OFFLINE" not in output+ "discovered",+ [+ suite+ for suite in discover_suites()+ if suite.fn.__module__.startswith(("products.posthog_ai.evals.", "products.slack_app.evals."))+ and hasattr(importlib.import_module(suite.fn.__module__), "offline_suite")+ ],+ ids=lambda suite: suite.id,+)@pytest.mark.asyncio-async def test_sql_suite_enrolls_every_scorer_in_offline_publishing(monkeypatch: pytest.MonkeyPatch) -> None:+async def test_suite_enrolls_every_scorer_in_offline_publishing(+ monkeypatch: pytest.MonkeyPatch, discovered: EvalSuite+) -> None:+ domain, module_name, suite_kind = discovered.domain, discovered.module_name, discovered.kind+ module = importlib.import_module(discovered.fn.__module__)+ if domain == "taxonomic_filter":+ monkeypatch.setattr(module, "system_one_configured", lambda: True)+ if module_name == "eval_event_match":+ monkeypatch.setattr(module, "EVENT_MATCH_MODEL", MagicMock())+ else:+ monkeypatch.setattr(module, "fetch_search_intent_prompt", MagicMock())run = AsyncMock()- monkeypatch.setattr(sql_eval, "SandboxedPublicEval", run)+ runner = "SandboxedPrivateEval" if domain == "experiments" else "SandboxedPublicEval"+ if suite_kind == SuiteKind.ONE_SHOT:+ runner = "OneShotPublicEval"+ monkeypatch.setattr(module, runner, run)- await sql_eval.eval_sql(MagicMock())+ await discovered.fn(MagicMock(skill_delivery="bundled"))run.assert_awaited_once()suite = run.call_args.kwargs["offline_suite"]- assert suite.key == "sql/eval_sql::eval_sql"- assert suite.scorer_kinds == {- "exit_code_zero": "boolean",- "no_persistent_insight_save": "boolean",- "execute_sql_called": "boolean",- "answer_tool_not_typed_query": "boolean",- "querying_posthog_data_skill_loaded": "boolean",- "sql_schema_alignment": "numeric",- "sql_result_message_alignment": "numeric",- }+ assert suite.key == discovered.id+ component = (+ f"slack-app/{module_name.removeprefix('eval_').replace('_', '-')}"+ if domain == "slack_app"+ else f"posthog-ai/{domain.replace('_', '-')}"+ )+ assert suite.tags == ("team:surfaces", f"component:{component}")+ if domain == "experiments":+ assert run.call_args.kwargs["send_to_posthog"] is True+ assert set(suite.scorer_kinds.values()) == {"boolean"}+ elif domain == "sql":+ assert suite.scorer_kinds == {+ "exit_code_zero": "boolean",+ "no_persistent_insight_save": "boolean",+ "execute_sql_called": "boolean",+ "answer_tool_not_typed_query": "boolean",+ "querying_posthog_data_skill_loaded": "boolean",+ "sql_schema_alignment": "numeric",+ "sql_result_message_alignment": "numeric",+ }configured_names = {scorer._name() for scorer in run.call_args.kwargs["scorers"]}- assert set(suite.scorer_kinds) == {"exit_code_zero", *configured_names}+ if suite_kind == SuiteKind.SANDBOXED:+ configured_names.add("exit_code_zero")+ assert set(suite.scorer_kinds) == set(suite.scorer_versions) == configured_names@pytest.mark.parametrize(diff --git a/products/posthog_ai/evals/ci/eval_basic.py b/products/posthog_ai/evals/ci/eval_basic.pyindex fad97bc84cb5..f7edf05de098 100644--- a/products/posthog_ai/evals/ci/eval_basic.py+++ b/products/posthog_ai/evals/ci/eval_basic.py@@ -15,6 +15,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.cli_mcp.scorers import CalledTargetTool, SurfacedGeneratedAppUrl+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_bugfix(ctx: EvalContext) -> None:@@ -37,6 +38,10 @@ async def eval_bugfix(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-bugfix-cli",+ offline_suite=offline_suite(+ "ci/eval_basic::eval_bugfix",+ metrics=[],+ ),cases=cases,scorers=[],ctx=ctx,@@ -86,6 +91,10 @@ async def eval_app_link_generation(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-app-link-generation-cli",+ offline_suite=offline_suite(+ "ci/eval_basic::eval_app_link_generation",+ metrics=["called_target_tool", "surfaced_generated_app_url"],+ ),cases=cases,scorers=[CalledTargetTool(), SurfacedGeneratedAppUrl()],ctx=ctx,diff --git a/products/posthog_ai/evals/cli_mcp/eval_render_ui.py b/products/posthog_ai/evals/cli_mcp/eval_render_ui.pyindex 492d8f3e323d..d58fb71145fc 100644--- a/products/posthog_ai/evals/cli_mcp/eval_render_ui.py+++ b/products/posthog_ai/evals/cli_mcp/eval_render_ui.py@@ -53,6 +53,7 @@ from products.posthog_ai.evals.cli_mcp.scorers import (ExecBeforeRender,RenderedEntityUi,)+from products.posthog_ai.evals.offline_config import offline_suite# Renderable experiment tools (detail + results). The agent may render the detail# view (``experiment-get``) or a results view for a status/results question. Two@@ -96,6 +97,10 @@ async def eval_renders_entity_ui(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-renders-entity-ui-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_render_ui::eval_renders_entity_ui",+ metrics=["rendered_entity_ui", "exec_before_render"],+ ),cases=cases,scorers=[RenderedEntityUi(), ExecBeforeRender()],ctx=ctx,@@ -127,6 +132,10 @@ async def eval_no_render_for_query_results(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-no-render-for-query-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_render_ui::eval_no_render_for_query_results",+ metrics=["called_target_tool", "did_not_render_ui", "did_not_cite_raw_sql_after_typed_query"],+ ),cases=cases,scorers=[CalledTargetTool(), DidNotRenderUi(), DidNotCiteRawSqlAfterTypedQuery()],ctx=ctx,diff --git a/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py b/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.pyindex 116b50f68d69..7126dda1e16f 100644--- a/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py+++ b/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py@@ -18,6 +18,7 @@ from products.posthog_ai.evals.cli_mcp.skill_distribution_scorers import (skill_distribution_expectations,)from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues+from products.posthog_ai.evals.offline_config import offline_suitefrom products.tasks.backend.facade.agents import EVAL_INTERACTION_ORIGIN, CustomPromptSandboxContext@@ -173,6 +174,18 @@ async def eval_skill_distribution(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-skill-distribution-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_skill_distribution::eval_skill_distribution",+ metrics=[+ "skill_search_first",+ "expected_skill_discovered",+ "expected_skill_loaded",+ "excluded_skill_not_loaded",+ "skill_loaded_before_tool",+ "no_bundled_skill_bypass",+ "no_exec_skill_bypass",+ ],+ ),cases=cases,scorers=[SkillSearchFirst(),diff --git a/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py b/products/posthog_ai/evals/cli_mcp/eval_skill_usage.pyindex 8031426b0a00..db1a4eb61e0f 100644--- a/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py+++ b/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py@@ -27,6 +27,7 @@ from products.posthog_ai.evals.cli_mcp.skill_usage_seeder import (PROJECT_SKILL_REFERENCE_PATH,seed_project_skill,)+from products.posthog_ai.evals.offline_config import offline_suitefrom products.tasks.backend.facade.agents import EVAL_INTERACTION_ORIGIN@@ -138,6 +139,20 @@ async def eval_skill_usage(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-skill-usage-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_skill_usage::eval_skill_usage",+ metrics=[+ "skill_search_first",+ "expected_skill_discovered",+ "expected_skill_loaded",+ "skill_loaded_before_tool",+ "no_bundled_skill_bypass",+ "no_exec_skill_bypass",+ "expected_reference_pulled",+ "search_recovery_after_zero_hit",+ "skill_answer_correctness",+ ],+ ),cases=cases,scorers=[SkillSearchFirst(),diff --git a/products/posthog_ai/evals/cli_mcp/eval_workflow.py b/products/posthog_ai/evals/cli_mcp/eval_workflow.pyindex 8f02a6ff7a34..688be90b6858 100644--- a/products/posthog_ai/evals/cli_mcp/eval_workflow.py+++ b/products/posthog_ai/evals/cli_mcp/eval_workflow.py@@ -39,6 +39,7 @@ from products.posthog_ai.evals.cli_mcp.scorers import (UsedJsonOutputFormat,VerifiedEventBeforeQuery,)+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_typo_recovery(ctx: EvalContext) -> None:@@ -69,6 +70,10 @@ async def eval_typo_recovery(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-typo-recovery-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_workflow::eval_typo_recovery",+ metrics=["recovered_to_correct_tool"],+ ),cases=cases,scorers=[RecoveredToCorrectTool()],ctx=ctx,@@ -102,6 +107,10 @@ async def eval_schema_drilldown(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-schema-drilldown-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_workflow::eval_schema_drilldown",+ metrics=["called_target_tool", "drilled_into_schema"],+ ),cases=cases,scorers=[CalledTargetTool(), DrilledIntoSchema()],ctx=ctx,@@ -124,6 +133,10 @@ async def eval_search_first(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-search-first-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_workflow::eval_search_first",+ metrics=["called_target_tool", "preferred_search_over_tools"],+ ),cases=cases,scorers=[CalledTargetTool(), PreferredSearchOverTools()],ctx=ctx,@@ -154,6 +167,10 @@ async def eval_info_before_call(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-info-before-call-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_workflow::eval_info_before_call",+ metrics=["called_target_tool", "info_before_call"],+ ),cases=cases,scorers=[CalledTargetTool(), InfoBeforeCall()],ctx=ctx,@@ -190,6 +207,10 @@ async def eval_json_for_post_processing(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-json-postprocess-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_workflow::eval_json_for_post_processing",+ metrics=["called_target_tool", "used_json_output_format", "ran_python_post_processing"],+ ),cases=cases,scorers=[CalledTargetTool(),@@ -216,6 +237,10 @@ async def eval_verify_event_before_query(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cli-mcp-verify-event-cli",+ offline_suite=offline_suite(+ "cli_mcp/eval_workflow::eval_verify_event_before_query",+ metrics=["called_target_tool", "verified_event_before_query"],+ ),cases=cases,scorers=[CalledTargetTool(), VerifiedEventBeforeQuery()],ctx=ctx,diff --git a/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py b/products/posthog_ai/evals/cohorts/eval_cohort_from_query.pyindex a525ff96d299..7b434dd3a7cf 100644--- a/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py+++ b/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py@@ -27,6 +27,7 @@ from products.posthog_ai.evals.cohorts.scorers import (CohortFromQueryUsed,QueryTargetsActorColumn,)+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_cohort_from_query(ctx: EvalContext) -> None:@@ -59,6 +60,10 @@ async def eval_cohort_from_query(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-cohorts-from-query-cli",+ offline_suite=offline_suite(+ "cohorts/eval_cohort_from_query::eval_cohort_from_query",+ metrics=["cohort_from_query_used", "query_targets_actor_column", "no_uuid_batching"],+ ),cases=cases,scorers=[CohortFromQueryUsed(),diff --git a/products/posthog_ai/evals/data_warehouse/eval_information_schema.py b/products/posthog_ai/evals/data_warehouse/eval_information_schema.pyindex 262a3f131759..136827813293 100644--- a/products/posthog_ai/evals/data_warehouse/eval_information_schema.py+++ b/products/posthog_ai/evals/data_warehouse/eval_information_schema.py@@ -58,6 +58,7 @@ from products.posthog_ai.evals.data_warehouse.synthesizer import (TYPE_NEEDLE_TABLE,VIEW_NEEDLE_NAME,)+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_dw_discovery(ctx: EvalContext) -> None:@@ -110,6 +111,10 @@ async def eval_dw_discovery(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-warehouse-discovery-cli",+ offline_suite=offline_suite(+ "data_warehouse/eval_information_schema::eval_dw_discovery",+ metrics=["information_schema_queried", "agentic_search_used", "needle_table_identified"],+ ),cases=cases,scorers=[InformationSchemaQueried(),@@ -172,6 +177,16 @@ async def eval_dw_value_retrieval(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-warehouse-retrieval-cli",+ offline_suite=offline_suite(+ "data_warehouse/eval_information_schema::eval_dw_value_retrieval",+ metrics=[+ "information_schema_queried",+ "information_schema_before_answer",+ "needle_value_retrieved",+ "answer_query_ran",+ "warehouse_answer_correctness",+ ],+ ),cases=cases,scorers=[InformationSchemaQueried(),@@ -228,6 +243,15 @@ async def eval_dw_relationships(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-warehouse-relationships-cli",+ offline_suite=offline_suite(+ "data_warehouse/eval_information_schema::eval_dw_relationships",+ metrics=[+ "information_schema_queried",+ "relationship_discovery",+ "join_path_traversed",+ "warehouse_answer_correctness",+ ],+ ),cases=cases,scorers=[InformationSchemaQueried(),@@ -271,6 +295,16 @@ async def eval_dw_table_relevancy(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-warehouse-relevancy-cli",+ offline_suite=offline_suite(+ "data_warehouse/eval_information_schema::eval_dw_table_relevancy",+ metrics=[+ "information_schema_queried",+ "agentic_search_used",+ "needle_table_identified",+ "stale_table_avoided",+ "warehouse_answer_correctness",+ ],+ ),cases=cases,scorers=[InformationSchemaQueried(),diff --git a/products/posthog_ai/evals/error_tracking/eval_events_sampling.py b/products/posthog_ai/evals/error_tracking/eval_events_sampling.pyindex e5f7fc38c973..70965ed042db 100644--- a/products/posthog_ai/evals/error_tracking/eval_events_sampling.py+++ b/products/posthog_ai/evals/error_tracking/eval_events_sampling.py@@ -33,6 +33,7 @@ from products.posthog_ai.evals.error_tracking.scorers import (IssueIdMatchesTarget,)from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues+from products.posthog_ai.evals.offline_config import offline_suitedef _events_case(@@ -107,6 +108,10 @@ async def eval_events_sampling(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-error-tracking-events-sampling-cli",+ offline_suite=offline_suite(+ "error_tracking/eval_events_sampling::eval_events_sampling",+ metrics=["no_error_tracking_write", "events_tool_used", "events_args_alignment", "issue_id_matches_target"],+ ),cases=cases,scorers=[NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"),diff --git a/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py b/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.pyindex a9dceb789cbe..3e82533e9c17 100644--- a/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py+++ b/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py@@ -40,6 +40,7 @@ from products.posthog_ai.evals.error_tracking.scorers import (IssueInputAlignment,)from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues+from products.posthog_ai.evals.offline_config import offline_suitedef _drilldown_case(@@ -121,6 +122,15 @@ async def eval_issue_drilldown(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-error-tracking-issue-drilldown-cli",+ offline_suite=offline_suite(+ "error_tracking/eval_issue_drilldown::eval_issue_drilldown",+ metrics=[+ "no_error_tracking_write",+ "issue_drilldown_order",+ "issue_id_matches_target",+ "issue_input_alignment",+ ],+ ),cases=cases,scorers=[NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"),diff --git a/products/posthog_ai/evals/error_tracking/eval_issues_list.py b/products/posthog_ai/evals/error_tracking/eval_issues_list.pyindex b133a2e24c32..a016b2a02cd5 100644--- a/products/posthog_ai/evals/error_tracking/eval_issues_list.py+++ b/products/posthog_ai/evals/error_tracking/eval_issues_list.py@@ -39,6 +39,7 @@ from products.posthog_ai.evals.error_tracking.scorers import (IssuesListToolUsed,)from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues+from products.posthog_ai.evals.offline_config import offline_suitedef _list_case(@@ -86,6 +87,10 @@ async def eval_issues_list(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-error-tracking-issues-list-cli",+ offline_suite=offline_suite(+ "error_tracking/eval_issues_list::eval_issues_list",+ metrics=["no_error_tracking_write", "issues_list_tool_used", "issues_list_input_alignment"],+ ),cases=cases,scorers=[NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"),diff --git a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.pyindex 450164d638f1..9e7b72f954a1 100644--- a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py+++ b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py@@ -39,6 +39,7 @@ from products.posthog_ai.evals.experiments.seeders import (seed_running_experiment,seed_uneven_split_experiment,)+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_bias_uneven_split(ctx: EvalContext) -> None:@@ -107,6 +108,11 @@ async def eval_bias_uneven_split(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-diagnose-bias-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_bias_uneven_split::eval_bias_uneven_split",+ metrics=["diagnosis_group", "surfaces_all_findings"],+ ),cases=cases,scorers=[CitesDiagnosticGroup(),diff --git a/products/posthog_ai/evals/experiments/eval_create_control_variant.py b/products/posthog_ai/evals/experiments/eval_create_control_variant.pyindex 493d53c8e9e6..b7a6e5e93a0e 100644--- a/products/posthog_ai/evals/experiments/eval_create_control_variant.py+++ b/products/posthog_ai/evals/experiments/eval_create_control_variant.py@@ -34,6 +34,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPrivateEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_create_control_variant(ctx: EvalContext) -> None:@@ -56,6 +57,11 @@ async def eval_create_control_variant(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-create-control-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_create_control_variant::eval_create_control_variant",+ metrics=["experiment_created_successfully"],+ ),cases=cases,scorers=[# The single outcome scorer: did experiment-create return without error?diff --git a/products/posthog_ai/evals/experiments/eval_empty_experiment.py b/products/posthog_ai/evals/experiments/eval_empty_experiment.pyindex a6374b4ebca3..9e962c140071 100644--- a/products/posthog_ai/evals/experiments/eval_empty_experiment.py+++ b/products/posthog_ai/evals/experiments/eval_empty_experiment.py@@ -52,6 +52,7 @@ from products.posthog_ai.evals.experiments.seeders import (seed_inactive_flag_experiment,seed_running_experiment,)+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_empty_experiment(ctx: EvalContext) -> None:@@ -139,6 +140,11 @@ async def eval_empty_experiment(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-diagnose-empty-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_empty_experiment::eval_empty_experiment",+ metrics=["diagnosis_group"],+ ),cases=cases,scorers=[CitesDiagnosticGroup(),diff --git a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py b/products/posthog_ai/evals/experiments/eval_interpretation_traps.pyindex 98b786ab6e29..262879511506 100644--- a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py+++ b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py@@ -50,6 +50,7 @@ from products.posthog_ai.evals.experiments.seeders import (seed_day_old_experiment,seed_running_experiment,)+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_interpretation_traps(ctx: EvalContext) -> None:@@ -161,6 +162,11 @@ async def eval_interpretation_traps(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-diagnose-interpretation-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_interpretation_traps::eval_interpretation_traps",+ metrics=["diagnosis_group", "advises_against_shipping"],+ ),cases=cases,scorers=[CitesDiagnosticGroup(),diff --git a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.pyindex f1d0f956b849..e085458bcad5 100644--- a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py+++ b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py@@ -40,16 +40,10 @@ from products.posthog_ai.evals.experiments.scorers import (RecommendsShipVariant,)from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment+from products.posthog_ai.evals.offline_config import offline_suite-# Cases are bundled into one suite function rather than one function per case-# because the harness wraps a list of cases in a single SandboxedPrivateEval-# call: one Braintrust experiment, with concurrency bounded by the harness's-# global sandbox semaphore and per-case filtering via --eval. Splitting into-# one function per case would create N separate Braintrust experiments and break-# cross-case comparison. Every existing sandboxed eval (eval_funnel,-# eval_retention, eval_insight_retrieval, eval_rollout_skill) follows this same-# single-function-multiple-cases shape.+# Keep the cases in one experiment so their scores can be compared together.async def eval_lifecycle_skills(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [# Case 1: Ship variant must require confirmation.@@ -91,6 +85,17 @@ async def eval_lifecycle_skills(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-lifecycle-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_lifecycle_skills::eval_lifecycle_skills",+ metrics=[+ "resolved_experiment_by_name",+ "no_unconfirmed_ship",+ "asked_for_confirmation",+ "duplicate_unique_flag_key",+ "recommends_ship_variant",+ ],+ ),cases=cases,scorers=[# Agent must look up the named experiment.diff --git a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.pyindex 5a3d26b5003b..5b4cac746250 100644--- a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py+++ b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py@@ -25,6 +25,7 @@ from products.posthog_ai.evals.experiments.scorers import (validate_retention_metric,)from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_metric_schema_discovery(ctx: EvalContext) -> None:@@ -53,6 +54,11 @@ async def eval_metric_schema_discovery(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-metric-schema-discovery-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_metric_schema_discovery::eval_metric_schema_discovery",+ metrics=["called_experiment_update", "first_update_metric_shape"],+ ),cases=cases,scorers=[RequiredToolCall(required={"experiment-update"}, name="called_experiment_update"),diff --git a/products/posthog_ai/evals/experiments/eval_midrun_changes.py b/products/posthog_ai/evals/experiments/eval_midrun_changes.pyindex 36baf69749ef..7db3ca178df9 100644--- a/products/posthog_ai/evals/experiments/eval_midrun_changes.py+++ b/products/posthog_ai/evals/experiments/eval_midrun_changes.py@@ -61,6 +61,7 @@ from products.posthog_ai.evals.experiments.seeders import (seed_paused_and_resumed_experiment,seed_running_experiment,)+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_midrun_changes(ctx: EvalContext) -> None:@@ -152,6 +153,11 @@ async def eval_midrun_changes(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-diagnose-midrun-changes-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_midrun_changes::eval_midrun_changes",+ metrics=["diagnosis_group", "does_not_recommend_edit", "advises_against_shipping"],+ ),cases=cases,scorers=[CitesDiagnosticGroup(),diff --git a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.pyindex 852cfa4fdf1c..4604a3d058e9 100644--- a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py+++ b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py@@ -43,6 +43,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.experiments.scorers import CitesDiagnosticGroupfrom products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_numbers_vs_sql(ctx: EvalContext) -> None:@@ -154,6 +155,11 @@ async def eval_numbers_vs_sql(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-diagnose-numbers-vs-sql-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_numbers_vs_sql::eval_numbers_vs_sql",+ metrics=["diagnosis_group"],+ ),cases=cases,scorers=[CitesDiagnosticGroup(),diff --git a/products/posthog_ai/evals/experiments/eval_rollout_skill.py b/products/posthog_ai/evals/experiments/eval_rollout_skill.pyindex fc646a7b4c51..59170914ec04 100644--- a/products/posthog_ai/evals/experiments/eval_rollout_skill.py+++ b/products/posthog_ai/evals/experiments/eval_rollout_skill.py@@ -8,8 +8,9 @@ cheap to reproduce while iterating on the skill's description.Maps to stress-test finding #7 (skills rarely auto-load), specificallythe variant-split rows (2.5, 5.3, 5.4, 5.5) in STEP_1_HARNESS.md. Uses-``SandboxedPrivateEval`` so it runs without a Braintrust API key —-local logs and PostHog ``$ai_trace`` events still emit.+``SandboxedPrivateEval`` with ``send_to_posthog=True`` to publish results+only to PostHog. The harness still requires a Braintrust API key for its+evaluation engine, but Braintrust result uploads remain disabled.To run:flox activate -- bash -c "set -a; source .env; set +a; python -m products.posthog_ai.eval_harness.harness eval_rollout_skill"@@ -21,6 +22,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPrivateEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.retrieval.scorers import SkillLoadedSKILL_NAME = "configuring-experiment-rollout"@@ -39,6 +41,11 @@ async def eval_rollout_skill(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-rollout-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_rollout_skill::eval_rollout_skill",+ metrics=["skill_loaded"],+ ),cases=cases,scorers=[SkillLoaded(skill_name=SKILL_NAME),diff --git a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.pyindex b1a8cc250afc..178b7f8bc1d8 100644--- a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py+++ b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py@@ -13,6 +13,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCallfrom products.posthog_ai.evals.experiments.scorers import SharedMetricValidationVerdictfrom products.posthog_ai.evals.experiments.seeders import SHARED_METRIC_NAME, seed_shared_metric_purchase_count+from products.posthog_ai.evals.offline_config import offline_suiteasync def eval_shared_metric_validation(ctx: EvalContext) -> None:@@ -50,6 +51,11 @@ async def eval_shared_metric_validation(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-shared-metric-validation-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_shared_metric_validation::eval_shared_metric_validation",+ metrics=["loaded_shared_metric", "shared_metric_validation_verdict"],+ ),cases=cases,scorers=[# Agent must actually read the seeded metric — either via listdiff --git a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.pyindex d34379afeb26..9d58348b98a5 100644--- a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py+++ b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py@@ -40,6 +40,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCallfrom products.posthog_ai.evals.experiments.scorers import AskedForConfirmationfrom products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment+from products.posthog_ai.evals.offline_config import offline_suite# Event name chosen to be obviously fictional so the agent can't plausibly# justify it as a typo of a real PostHog event in the seeded team.@@ -66,6 +67,11 @@ async def eval_unknown_events_confirmation(ctx: EvalContext) -> None:await SandboxedPrivateEval(experiment_name="sandboxed-experiments-unknown-events-cli",+ send_to_posthog=True,+ offline_suite=offline_suite(+ "experiments/eval_unknown_events_confirmation::eval_unknown_events_confirmation",+ metrics=["resolved_experiment_by_name", "no_silent_update_with_unknown_event", "asked_for_confirmation"],+ ),cases=cases,scorers=[# Agent must look up the named experiment first.diff --git a/products/posthog_ai/evals/offline_config.py b/products/posthog_ai/evals/offline_config.pynew file mode 100644index 000000000000..4479074fb0ec--- /dev/null+++ b/products/posthog_ai/evals/offline_config.py@@ -0,0 +1,225 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, dict[str, OfflineScorer]] = {+ "ci": {+ "called_target_tool": OfflineScorer(kind="boolean", version=UUID("01a1557d-d3f2-7807-88ff-12d4c4b2679c")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-ca2a-75c8-b3a4-5b9a0a405a64")),+ "surfaced_generated_app_url": OfflineScorer(+ kind="boolean", version=UUID("01a1557d-dda3-782a-a2cb-dac316f6e7d6")+ ),+ },+ "cli_mcp": {+ "called_target_tool": OfflineScorer(kind="boolean", version=UUID("01a1557e-090d-700e-8bc7-584c54a089fc")),+ "did_not_cite_raw_sql_after_typed_query": OfflineScorer(+ kind="boolean", version=UUID("01a1557e-1c0f-7302-96f5-ccdb2eed2b27")+ ),+ "did_not_render_ui": OfflineScorer(kind="boolean", version=UUID("01a1557e-1330-7ce0-a05a-914503a66cb0")),+ "drilled_into_schema": OfflineScorer(kind="boolean", version=UUID("01a1557e-fd96-783d-8de2-dc4ddb5d6468")),+ "excluded_skill_not_loaded": OfflineScorer(+ kind="boolean", version=UUID("01a1557e-aca4-7826-8388-c66db0633772")+ ),+ "exec_before_render": OfflineScorer(kind="boolean", version=UUID("01a1557d-fb44-769a-b5cf-6d96a971182d")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-e737-7f9a-bb69-3dae3e58df7c")),+ "expected_reference_pulled": OfflineScorer(+ kind="boolean", version=UUID("01a1557e-d5f9-7e74-8369-2e4627c02a4a")+ ),+ "expected_skill_discovered": OfflineScorer(+ kind="boolean", version=UUID("01a1557e-354f-726f-8e3e-c04ad776e8a7")+ ),+ "expected_skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a1557e-a3f3-72c2-899f-3d5d9c903b39")),+ "info_before_call": OfflineScorer(kind="boolean", version=UUID("01a1557f-1350-7eb4-8821-1b38bb2d8325")),+ "no_bundled_skill_bypass": OfflineScorer(kind="boolean", version=UUID("01a1557e-c286-7545-a3ba-14a8e354cafe")),+ "no_exec_skill_bypass": OfflineScorer(kind="boolean", version=UUID("01a1557e-cb02-7a20-a775-8dd932e4d07c")),+ "preferred_search_over_tools": OfflineScorer(+ kind="boolean", version=UUID("01a1557f-0679-7a2e-9cc6-a0559e513dcd")+ ),+ "ran_python_post_processing": OfflineScorer(+ kind="boolean", version=UUID("01a1557f-85a2-7380-94ff-6d5de79a68a1")+ ),+ "recovered_to_correct_tool": OfflineScorer(+ kind="boolean", version=UUID("01a1557e-f33b-7e99-9408-f3548316c76f")+ ),+ "rendered_entity_ui": OfflineScorer(kind="boolean", version=UUID("01a1557d-f13a-7eda-8447-e54d0ea0e7b1")),+ "search_recovery_after_zero_hit": OfflineScorer(+ kind="boolean", version=UUID("01a1557e-dedb-70fc-a4a4-c8be5f7c4939")+ ),+ "skill_answer_correctness": OfflineScorer(kind="boolean", version=UUID("01a1557e-e9b1-79b4-82c4-3639a73a9d66")),+ "skill_loaded_before_tool": OfflineScorer(kind="boolean", version=UUID("01a1557e-b9d5-7cb3-9136-2a8176f7a8f3")),+ "skill_search_first": OfflineScorer(kind="boolean", version=UUID("01a1557e-2b07-7fd6-95ed-e1f5cc459aff")),+ "used_json_output_format": OfflineScorer(kind="boolean", version=UUID("01a1557f-7ce7-7ee9-8a62-0d219935a7cf")),+ "verified_event_before_query": OfflineScorer(+ kind="boolean", version=UUID("01a1557f-8e04-7c10-a084-3bcd95fd5fe8")+ ),+ },+ "cohorts": {+ "cohort_from_query_used": OfflineScorer(kind="boolean", version=UUID("01a1557f-a2a9-7cd5-a647-24fbc34fdaca")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557f-974a-79e8-a589-a41e4ff8bf2d")),+ "no_uuid_batching": OfflineScorer(kind="boolean", version=UUID("01a1557f-bbad-7dcc-b140-d5c1fe69a9d2")),+ "query_targets_actor_column": OfflineScorer(+ kind="boolean", version=UUID("01a1557f-b048-7ebe-8db4-2d89aa79787e")+ ),+ },+ "data_warehouse": {+ "agentic_search_used": OfflineScorer(kind="boolean", version=UUID("01a1557f-db0e-76f5-ade3-847fb199d132")),+ "answer_query_ran": OfflineScorer(kind="boolean", version=UUID("01a15580-3330-7033-a26b-bd7241292083")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557f-c598-76ff-8312-e8261f7357c2")),+ "information_schema_before_answer": OfflineScorer(+ kind="boolean", version=UUID("01a1557f-efac-7e78-b26d-3c45cab69449")+ ),+ "information_schema_queried": OfflineScorer(+ kind="boolean", version=UUID("01a1557f-d0c7-7e57-9845-8c4b1f4d3004")+ ),+ "join_path_traversed": OfflineScorer(kind="boolean", version=UUID("01a15580-55a9-77a1-9815-f898ba72b96a")),+ "needle_table_identified": OfflineScorer(kind="boolean", version=UUID("01a1557f-e4dc-75a1-9606-24647018c5c5")),+ "needle_value_retrieved": OfflineScorer(kind="boolean", version=UUID("01a15580-29c9-7d6e-afa2-c650a1958662")),+ "relationship_discovery": OfflineScorer(kind="boolean", version=UUID("01a15580-452b-7ccd-9a91-d87dace78770")),+ "stale_table_avoided": OfflineScorer(kind="boolean", version=UUID("01a15580-5f0c-7e20-baca-eb544aba6950")),+ "warehouse_answer_correctness": OfflineScorer(+ kind="numeric", version=UUID("01a15580-3c17-77bc-af79-dfb04a075815")+ ),+ },+ "error_tracking": {+ "events_args_alignment": OfflineScorer(kind="boolean", version=UUID("01a15580-83eb-7edb-88ed-e3a278c1bad6")),+ "events_tool_used": OfflineScorer(kind="boolean", version=UUID("01a15580-7acc-7e5b-bb9c-6e670b54e66d")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15580-67d1-7a2c-bdfe-d4de0553020c")),+ "issue_drilldown_order": OfflineScorer(kind="boolean", version=UUID("01a15580-9762-7200-b518-2da4b1d854ce")),+ "issue_id_matches_target": OfflineScorer(kind="boolean", version=UUID("01a15580-8e80-70df-b21a-15945ed85bdb")),+ "issue_input_alignment": OfflineScorer(kind="boolean", version=UUID("01a15581-0aec-75b5-b0e7-e371e5853164")),+ "issues_list_input_alignment": OfflineScorer(+ kind="boolean", version=UUID("01a15581-1e2c-7ae1-9522-0f54a9c6e4a9")+ ),+ "issues_list_tool_used": OfflineScorer(kind="boolean", version=UUID("01a15581-148b-74d7-93d7-8c3110ab2a95")),+ "no_error_tracking_write": OfflineScorer(kind="boolean", version=UUID("01a15580-7158-7c13-856a-ff7669b5e0f2")),+ },+ "experiments": {+ "advises_against_shipping": OfflineScorer(kind="boolean", version=UUID("01a1558e-f848-7def-b3d8-970df097d0e5")),+ "asked_for_confirmation": OfflineScorer(kind="boolean", version=UUID("01a1558f-1274-77da-9d79-2e410a63cbbc")),+ "called_experiment_update": OfflineScorer(kind="boolean", version=UUID("01a1558f-1ac0-7176-8fbe-8730bbd9074a")),+ "diagnosis_group": OfflineScorer(kind="boolean", version=UUID("01a1558f-2481-7ab2-95a5-a9f4e7e087eb")),+ "does_not_recommend_edit": OfflineScorer(kind="boolean", version=UUID("01a1558f-314c-7d2f-bf87-43b94f4917ff")),+ "duplicate_unique_flag_key": OfflineScorer(+ kind="boolean", version=UUID("01a1558f-3b7c-7347-8fd2-a2eba94f5723")+ ),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1558f-4447-7a87-b21c-4418a4152cc0")),+ "experiment_created_successfully": OfflineScorer(+ kind="boolean", version=UUID("01a1558f-55a3-7584-8fae-39efa8c6b11e")+ ),+ "first_update_metric_shape": OfflineScorer(+ kind="boolean", version=UUID("01a1558f-5ee1-77eb-b837-843569f351b3")+ ),+ "loaded_shared_metric": OfflineScorer(kind="boolean", version=UUID("01a1558f-69dc-7efe-adfe-1debc1acbfa0")),+ "no_silent_update_with_unknown_event": OfflineScorer(+ kind="boolean", version=UUID("01a1558f-71f0-74a5-aec8-a03922367ecc")+ ),+ "no_unconfirmed_ship": OfflineScorer(kind="boolean", version=UUID("01a1558f-7d13-7a20-9035-2aa9cdddb4ac")),+ "recommends_ship_variant": OfflineScorer(kind="boolean", version=UUID("01a1558f-8617-7a83-87a2-27f3adf7d54a")),+ "resolved_experiment_by_name": OfflineScorer(+ kind="boolean", version=UUID("01a1558f-8e45-7480-b669-ae4c5d655366")+ ),+ "shared_metric_validation_verdict": OfflineScorer(+ kind="boolean", version=UUID("01a1558f-9a54-7743-905c-bb10f59c3264")+ ),+ "skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a1558f-a299-7e75-938e-07c2ff9c747f")),+ "surfaces_all_findings": OfflineScorer(kind="boolean", version=UUID("01a1558f-ad52-7b81-8a93-d7e602bf0125")),+ },+ "product_analytics": {+ "answer_tool_not_execute_sql": OfflineScorer(+ kind="boolean", version=UUID("01a15581-3926-7c14-a907-e994586bd216")+ ),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15581-2721-7343-9ee8-d4cb29adbe1a")),+ "first_relevant_tool": OfflineScorer(kind="boolean", version=UUID("01a15581-9357-7433-89c5-b66294e872d5")),+ "first_relevant_tool_t1": OfflineScorer(kind="boolean", version=UUID("01a15581-bca9-7af2-a9e1-fd29a55c7291")),+ "first_relevant_tool_t2": OfflineScorer(kind="boolean", version=UUID("01a15581-c7a2-7706-9d55-f2bd6640b817")),+ "first_relevant_tool_t3": OfflineScorer(kind="boolean", version=UUID("01a15581-d0cf-7b8a-ae3f-558eb875af9c")),+ "first_relevant_tool_t4": OfflineScorer(kind="boolean", version=UUID("01a15581-db3d-7eb7-8184-ef777d8900c1")),+ "funnel_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a15581-4c67-7321-bad6-608fbbe18564")),+ "funnel_time_range_relevancy": OfflineScorer(+ kind="boolean", version=UUID("01a15581-5652-79d8-a3b0-48be756495eb")+ ),+ "insight_shape": OfflineScorer(kind="boolean", version=UUID("01a15581-e3df-7304-b06e-8ae4ad58ccf5")),+ "no_persistent_insight_save": OfflineScorer(+ kind="boolean", version=UUID("01a15581-2fc3-7c68-b349-d44a670b232d")+ ),+ "providing_insights_skill_loaded": OfflineScorer(+ kind="boolean", version=UUID("01a15581-9c84-7f36-837a-e20bbb99ac17")+ ),+ "retention_schema_alignment": OfflineScorer(+ kind="numeric", version=UUID("01a15581-5faa-7855-b46a-90ff034dae82")+ ),+ "retention_time_range_relevancy": OfflineScorer(+ kind="boolean", version=UUID("01a15581-6891-7f9b-bbb4-199b8b2edec0")+ ),+ "saved_insight": OfflineScorer(kind="boolean", version=UUID("01a15581-ee89-7754-b0dd-6012efeaa1c4")),+ "schema_discovery_order": OfflineScorer(kind="boolean", version=UUID("01a15581-7429-7fff-85a7-9c223feddee6")),+ "trends_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a15581-a78c-7d03-bb2d-de43ab247f81")),+ "trends_time_range_relevancy": OfflineScorer(+ kind="boolean", version=UUID("01a15581-b135-7a70-ae09-b3d0e93bc894")+ ),+ "verified_event_exists": OfflineScorer(kind="boolean", version=UUID("01a15581-41af-7c8a-bf72-5e7c4ef63b0b")),+ },+ "retrieval": {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15581-f842-7336-8237-f5a8bf96d317")),+ "info_before_execute_sql": OfflineScorer(kind="boolean", version=UUID("01a15582-4169-7666-8583-809f548f5ee7")),+ "information_schema_before_sql": OfflineScorer(+ kind="boolean", version=UUID("01a15582-3921-7c7f-9066-b2e9a767cd33")+ ),+ "lookup_id_in_output": OfflineScorer(kind="boolean", version=UUID("01a15582-2caf-725e-b7f0-ac9a02218b09")),+ "skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a15582-0134-7aaa-b171-52e5fb222bfa")),+ },+ "sql": {+ "answer_tool_not_typed_query": OfflineScorer(+ kind="boolean", version=UUID("01a11725-3002-7a3f-b9b5-3426bdec5f00")+ ),+ "execute_sql_called": OfflineScorer(kind="boolean", version=UUID("01a11725-20aa-78c3-af96-1aae34bca429")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a11724-d52d-704a-af08-2ceaf5c65a6d")),+ "no_persistent_insight_save": OfflineScorer(+ kind="boolean", version=UUID("01a11725-17f6-7673-93e9-4d525fa121ab")+ ),+ "querying_posthog_data_skill_loaded": OfflineScorer(+ kind="boolean", version=UUID("01a11725-54c1-71e0-b606-132d5f4cb339")+ ),+ "sql_result_message_alignment": OfflineScorer(+ kind="numeric", version=UUID("01a11725-6940-7f49-b979-11b3d6521393")+ ),+ "sql_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a11725-5da7-7dae-ab09-1c0beaa77224")),+ },+ "surveys": {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15582-4a50-75e1-a17a-3d483bb95586")),+ "no_forbidden_survey_side_effects": OfflineScorer(+ kind="boolean", version=UUID("01a15582-5371-7f81-8356-53532c73fac2")+ ),+ "survey_create_alignment": OfflineScorer(kind="numeric", version=UUID("01a15582-711a-7b7a-bd93-061a83ea909d")),+ "survey_create_outcome": OfflineScorer(kind="boolean", version=UUID("01a15582-5d5a-786a-965f-ee2075db057d")),+ "survey_create_returned_id": OfflineScorer(+ kind="boolean", version=UUID("01a15582-6656-7fb2-b67d-6b918a8363e0")+ ),+ "survey_id_in_final_message": OfflineScorer(+ kind="boolean", version=UUID("01a15582-7cea-7b35-9632-1fd1c3e22185")+ ),+ },+ "taxonomic_filter": {+ "event_match_found": OfflineScorer(kind="boolean", version=UUID("01a15582-8f96-76a8-a6b0-9ad9c932f12a")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15582-85bb-71aa-b4c8-dc008015ab30")),+ "no_wrong_event_match": OfflineScorer(kind="boolean", version=UUID("01a15582-9ce5-711e-b612-3b0be41c2669")),+ "no_wrong_switch": OfflineScorer(kind="boolean", version=UUID("01a15582-b862-766e-b1ee-6664b70a9a93")),+ "search_intent_match": OfflineScorer(kind="boolean", version=UUID("01a15582-a5dd-7f46-8681-5f0dd77b22d4")),+ "switch_when_needed": OfflineScorer(kind="boolean", version=UUID("01a15582-aef0-748b-8282-c92f901762b5")),+ },+}+++def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:+ domain = key.split("/", 1)[0]+ scorers = _SCORERS[domain]+ component = domain.replace("_", "-")+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: scorers[metric] for metric in enrolled_metrics},+ tags=("team:surfaces", f"component:posthog-ai/{component}"),+ )diff --git a/products/posthog_ai/evals/product_analytics/eval_funnel.py b/products/posthog_ai/evals/product_analytics/eval_funnel.pyindex 2bf6c71adc29..7c21408e64c8 100644--- a/products/posthog_ai/evals/product_analytics/eval_funnel.py+++ b/products/posthog_ai/evals/product_analytics/eval_funnel.py@@ -15,6 +15,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall, RequiredToolCall+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import (INSIGHT_WRITE_TOOLS,FunnelSchemaAlignment,@@ -587,6 +588,16 @@ async def eval_funnel(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-funnels",+ offline_suite=offline_suite(+ "product_analytics/eval_funnel::eval_funnel",+ metrics=[+ "no_persistent_insight_save",+ "answer_tool_not_execute_sql",+ "verified_event_exists",+ "funnel_schema_alignment",+ "funnel_time_range_relevancy",+ ],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),diff --git a/products/posthog_ai/evals/product_analytics/eval_retention.py b/products/posthog_ai/evals/product_analytics/eval_retention.pyindex ee77fff5b70f..9cfb1d1f797a 100644--- a/products/posthog_ai/evals/product_analytics/eval_retention.py+++ b/products/posthog_ai/evals/product_analytics/eval_retention.py@@ -8,6 +8,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import (INSIGHT_WRITE_TOOLS,RetentionSchemaAlignment,@@ -116,6 +117,15 @@ async def eval_retention(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-retention-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_retention::eval_retention",+ metrics=[+ "no_persistent_insight_save",+ "answer_tool_not_execute_sql",+ "retention_schema_alignment",+ "retention_time_range_relevancy",+ ],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),diff --git a/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py b/products/posthog_ai/evals/product_analytics/eval_schema_discovery.pyindex 3c4f9b5bb735..c0d1c1522856 100644--- a/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py+++ b/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py@@ -22,6 +22,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS, SchemaDiscoveryOrder@@ -84,6 +85,10 @@ async def eval_schema_discovery(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-schema-discovery-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_schema_discovery::eval_schema_discovery",+ metrics=["no_persistent_insight_save", "answer_tool_not_execute_sql", "schema_discovery_order"],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),diff --git a/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py b/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.pyindex ecf2747c183f..c024a7fd52c5 100644--- a/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py+++ b/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py@@ -17,6 +17,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCallfrom products.posthog_ai.evals.cli_mcp.scorers import FirstRelevantTool+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLSANALYSIS_QUERY_TOOLS = frozenset(@@ -119,6 +120,10 @@ async def eval_sql_shaped_typed_query_routing(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-sql-shaped-typed-query-routing-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_sql_shaped_routing::eval_sql_shaped_typed_query_routing",+ metrics=["no_persistent_insight_save", "first_relevant_tool"],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),@@ -150,6 +155,10 @@ async def eval_sql_shaped_sql_controls(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-sql-shaped-sql-controls-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_sql_shaped_routing::eval_sql_shaped_sql_controls",+ metrics=["no_persistent_insight_save", "first_relevant_tool"],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),diff --git a/products/posthog_ai/evals/product_analytics/eval_trends.py b/products/posthog_ai/evals/product_analytics/eval_trends.pyindex 89d9d73e2289..1fa92c1989f4 100644--- a/products/posthog_ai/evals/product_analytics/eval_trends.py+++ b/products/posthog_ai/evals/product_analytics/eval_trends.py@@ -18,6 +18,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import (INSIGHT_WRITE_TOOLS,TrendsSchemaAlignment,@@ -301,6 +302,16 @@ async def eval_trends(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-trends",+ offline_suite=offline_suite(+ "product_analytics/eval_trends::eval_trends",+ metrics=[+ "no_persistent_insight_save",+ "answer_tool_not_execute_sql",+ "providing_insights_skill_loaded",+ "trends_schema_alignment",+ "trends_time_range_relevancy",+ ],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),diff --git a/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py b/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.pyindex d02003e698e6..d4c51ef0e614 100644--- a/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py+++ b/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py@@ -33,6 +33,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCallfrom products.posthog_ai.evals.cli_mcp.scorers import FirstRelevantTool+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLSfrom .eval_sql_shaped_routing import ANALYSIS_QUERY_TOOLS@@ -106,6 +107,16 @@ async def eval_typed_runner_no_lock_in_control(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-typed-runner-no-lock-in-control-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_typed_runner_lock_in::eval_typed_runner_no_lock_in_control",+ metrics=[+ "no_persistent_insight_save",+ "first_relevant_tool_t1",+ "first_relevant_tool_t2",+ "first_relevant_tool_t3",+ "first_relevant_tool_t4",+ ],+ ),cases=cases,scorers=_lock_in_scorers(turn2_expected="query-trends"),ctx=ctx,@@ -140,6 +151,16 @@ async def eval_typed_runner_lock_in(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-typed-runner-lock-in-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_typed_runner_lock_in::eval_typed_runner_lock_in",+ metrics=[+ "no_persistent_insight_save",+ "first_relevant_tool_t1",+ "first_relevant_tool_t2",+ "first_relevant_tool_t3",+ "first_relevant_tool_t4",+ ],+ ),cases=cases,scorers=_lock_in_scorers(turn2_expected="execute-sql"),ctx=ctx,diff --git a/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py b/products/posthog_ai/evals/product_analytics/eval_visualization_choice.pyindex b9b670559824..2073fc7b27bf 100644--- a/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py+++ b/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py@@ -20,6 +20,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCall+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS, InsightShapeTIME_SERIES_DISPLAYS = ["ActionsLineGraph", "ActionsBar", "ActionsAreaGraph", "ActionsUnstackedBar"]@@ -104,6 +105,10 @@ async def eval_visualization_choice(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-visualization-choice-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_visualization_choice::eval_visualization_choice",+ metrics=["no_persistent_insight_save", "insight_shape"],+ ),cases=cases,scorers=[NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),@@ -151,6 +156,10 @@ async def eval_saved_insight_display(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-saved-insight-display-cli",+ offline_suite=offline_suite(+ "product_analytics/eval_visualization_choice::eval_saved_insight_display",+ metrics=["saved_insight", "insight_shape"],+ ),cases=cases,scorers=[RequiredToolCall(["insight-create"], name="saved_insight"), InsightShape()],ctx=ctx,diff --git a/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py b/products/posthog_ai/evals/retrieval/eval_insight_retrieval.pyindex ef9a878e98c1..4c4ca5bfa3f3 100644--- a/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py+++ b/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py@@ -23,6 +23,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.seeders.insight import seed_insight_noise+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.retrieval.scorers import LookupIdInOutput, SkillLoadedSKILL_NAME = "querying-posthog-data"@@ -72,6 +73,10 @@ async def eval_insight_retrieval(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-retrieval-cli",+ offline_suite=offline_suite(+ "retrieval/eval_insight_retrieval::eval_insight_retrieval",+ metrics=["skill_loaded", "lookup_id_in_output"],+ ),cases=cases,scorers=[SkillLoaded(skill_name=SKILL_NAME),diff --git a/products/posthog_ai/evals/retrieval/eval_system_table_search.py b/products/posthog_ai/evals/retrieval/eval_system_table_search.pyindex fcb2dc80d646..adde6d5e19bd 100644--- a/products/posthog_ai/evals/retrieval/eval_system_table_search.py+++ b/products/posthog_ai/evals/retrieval/eval_system_table_search.py@@ -34,6 +34,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.seeders.insight import seed_insight_noise+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.retrieval.scorers import InfoCalledBeforeTool, InformationSchemaBeforeSql@@ -73,6 +74,10 @@ async def eval_system_table_search(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-system-table-search-cli",+ offline_suite=offline_suite(+ "retrieval/eval_system_table_search::eval_system_table_search",+ metrics=["information_schema_before_sql", "info_before_execute_sql"],+ ),cases=cases,scorers=[InformationSchemaBeforeSql(),diff --git a/products/posthog_ai/evals/sql/eval_sql.py b/products/posthog_ai/evals/sql/eval_sql.pyindex 12d78f18dd8b..cea50c41769a 100644--- a/products/posthog_ai/evals/sql/eval_sql.py+++ b/products/posthog_ai/evals/sql/eval_sql.py@@ -10,13 +10,11 @@ To run:from __future__ import annotations-from uuid import UUID-from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContext-from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuitefrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLSfrom products.posthog_ai.evals.retrieval.scorers import SkillLoadedfrom products.posthog_ai.evals.sql.scorers import AnswerQueryRan, SQLResultMessageAlignment, SQLSchemaAlignment@@ -375,27 +373,16 @@ WHERE properties.interests IS NOT NULLawait SandboxedPublicEval(experiment_name="sandboxed-sql-cli",- offline_suite=OfflineEvalSuite(- key="sql/eval_sql::eval_sql",- project_id=2,- scorer_kinds={- "exit_code_zero": "boolean",- "no_persistent_insight_save": "boolean",- "execute_sql_called": "boolean",- "answer_tool_not_typed_query": "boolean",- "querying_posthog_data_skill_loaded": "boolean",- "sql_schema_alignment": "numeric",- "sql_result_message_alignment": "numeric",- },- scorer_versions={- "exit_code_zero": UUID("01a11724-d52d-704a-af08-2ceaf5c65a6d"),- "no_persistent_insight_save": UUID("01a11725-17f6-7673-93e9-4d525fa121ab"),- "execute_sql_called": UUID("01a11725-20aa-78c3-af96-1aae34bca429"),- "answer_tool_not_typed_query": UUID("01a11725-3002-7a3f-b9b5-3426bdec5f00"),- "querying_posthog_data_skill_loaded": UUID("01a11725-54c1-71e0-b606-132d5f4cb339"),- "sql_schema_alignment": UUID("01a11725-5da7-7dae-ab09-1c0beaa77224"),- "sql_result_message_alignment": UUID("01a11725-6940-7f49-b979-11b3d6521393"),- },+ offline_suite=offline_suite(+ "sql/eval_sql::eval_sql",+ metrics=[+ "no_persistent_insight_save",+ "execute_sql_called",+ "answer_tool_not_typed_query",+ "querying_posthog_data_skill_loaded",+ "sql_schema_alignment",+ "sql_result_message_alignment",+ ],),cases=cases,scorers=[diff --git a/products/posthog_ai/evals/surveys/eval_surveys.py b/products/posthog_ai/evals/surveys/eval_surveys.pyindex 187a36a4eb5f..7214ac6a247c 100644--- a/products/posthog_ai/evals/surveys/eval_surveys.py+++ b/products/posthog_ai/evals/surveys/eval_surveys.py@@ -10,6 +10,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCallfrom products.posthog_ai.eval_harness.seeders.survey import seed_survey_feature_flags+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLSfrom products.posthog_ai.evals.surveys.scorers import (SURVEY_FORBIDDEN_WRITE_TOOLS,@@ -270,6 +271,16 @@ async def eval_surveys(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-surveys-cli",+ offline_suite=offline_suite(+ "surveys/eval_surveys::eval_surveys",+ metrics=[+ "no_forbidden_survey_side_effects",+ "survey_create_outcome",+ "survey_create_returned_id",+ "survey_create_alignment",+ "survey_id_in_final_message",+ ],+ ),cases=cases,scorers=[NoToolCall(diff --git a/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py b/products/posthog_ai/evals/taxonomic_filter/eval_event_match.pyindex 493605b721a5..721285f7f3a4 100644--- a/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py+++ b/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py@@ -36,6 +36,7 @@ from products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKindfrom products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.taxonomic_filter.scorers import EVENT_MATCH_KEY, EventMatchFound, NoWrongEventMatchSUITE_KIND = SuiteKind.ONE_SHOT@@ -103,6 +104,11 @@ async def eval_event_match(ctx: EvalContext) -> None:await OneShotPublicEval(experiment_name="taxonomic-filter-event-match",+ offline_suite=offline_suite(+ "taxonomic_filter/eval_event_match::eval_event_match",+ metrics=["event_match_found", "no_wrong_event_match"],+ suite_kind=SUITE_KIND,+ ),cases=[*DESCRIBED_CASES, *NO_MATCH_CASES],scorers=[EventMatchFound(), NoWrongEventMatch()],task=task,diff --git a/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py b/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.pyindex 9f78585527d6..cae15518e0c1 100644--- a/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py+++ b/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py@@ -46,6 +46,7 @@ from products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKindfrom products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.taxonomic_filter.scorers import (SEARCH_INTENT_KEY,NoWrongSwitch,@@ -357,6 +358,11 @@ async def eval_search_intent(ctx: EvalContext) -> None:await OneShotPublicEval(experiment_name="ml-inference-search-intent",+ offline_suite=offline_suite(+ "taxonomic_filter/eval_search_intent::eval_search_intent",+ metrics=["search_intent_match", "switch_when_needed", "no_wrong_switch"],+ suite_kind=SUITE_KIND,+ ),cases=[*WRONG_TAB_CASES, *RIGHT_TAB_CASES, *PARTIAL_CASES, *CONTEXT_CASES],scorers=[SearchIntentMatch(), SwitchWhenNeeded(), NoWrongSwitch()],task=task,diff --git a/products/signals/backend/test/test_eval_reporting.py b/products/signals/backend/test/test_eval_reporting.pyindex 8469247a7885..98631c355c22 100644--- a/products/signals/backend/test/test_eval_reporting.py+++ b/products/signals/backend/test/test_eval_reporting.py@@ -127,12 +127,22 @@ def test_disabled_reporting_never_attempts_network(_name: str, disabled: bool, aassert asyncio.run(recorder.publish()) is None-def test_upload_failure_is_reported_without_hiding_completed_scores(capsys: pytest.CaptureFixture[str]) -> None:[email protected]("failed_action", ["upload", "tag"])+def test_upload_failure_is_reported_without_hiding_completed_scores(+ capsys: pytest.CaptureFixture[str], failed_action: str+) -> None:+ server = OfflineServer()++ def respond(request: httpx.Request) -> httpx.Response:+ if failed_action == "upload" or request.method == "PATCH":+ return httpx.Response(403)+ return server.respond(request)+with (patch.dict(os.environ, {"POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token"}),patch("httpx.AsyncClient",- new=partial(httpx.AsyncClient, transport=httpx.MockTransport(lambda request: httpx.Response(403))),+ new=partial(httpx.AsyncClient, transport=httpx.MockTransport(respond)),),):recorder = SignalsEvalRecorder(SCOUT_SAFETY_SUITE)@@ -141,6 +151,14 @@ def test_upload_failure_is_reported_without_hiding_completed_scores(capsys: pyteitem_name="case",metrics=[EvalMetric(name="correct_classification", score=0)],)- assert asyncio.run(recorder.publish()) is None- assert "upload failed" in capsys.readouterr().err+ url = asyncio.run(recorder.publish())+ output = capsys.readouterr().err+ if failed_action == "upload":+ assert url is None+ assert "upload failed" in output+ else:+ assert server.status == "completed"+ assert url is not None and recorder.experiment_id in url+ assert f"uploaded: {url}" in output+ assert "upload failed" not in outputassert recorder.cases[0].scores["scout_safety_correct_classification"] == 0diff --git a/products/signals/eval/offline.py b/products/signals/eval/offline.pyindex 43c9532f5e0d..f6054ab6e271 100644--- a/products/signals/eval/offline.py+++ b/products/signals/eval/offline.py@@ -135,4 +135,5 @@ class SignalsEvalRecorder:self._report(f"upload failed: {reason}. Results were not saved for retry.")return Noneself._report(f"uploaded: {url}")+ await self.publisher.tag(upload)return urldiff --git a/products/signals/eval/offline_config.py b/products/signals/eval/offline_config.pyindex f829a71a0b00..19a1d5d390eb 100644--- a/products/signals/eval/offline_config.py+++ b/products/signals/eval/offline_config.py@@ -4,6 +4,7 @@ from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuiteGROUPING_SUITE = OfflineEvalSuite(key="signals/grouping-e2e",+ tags=("team:self-driving", "component:signals/grouping"),project_id=2,scorer_kinds={"signal_safety_correct_classification": "boolean",@@ -45,6 +46,7 @@ GROUPING_SUITE = OfflineEvalSuite(SCOUT_SAFETY_SUITE = OfflineEvalSuite(key="signals/scout-safety",+ tags=("team:self-driving", "component:signals/scout-safety"),project_id=2,scorer_kinds={"scout_safety_correct_classification": "boolean",diff --git a/products/slack_app/evals/eval_followup_classifier.py b/products/slack_app/evals/eval_followup_classifier.pyindex 7fdcdb78e84d..4526e841d3e4 100644--- a/products/slack_app/evals/eval_followup_classifier.py+++ b/products/slack_app/evals/eval_followup_classifier.py@@ -26,6 +26,7 @@ from products.posthog_ai.eval_harness.harness.requirements import SuiteKindfrom products.posthog_ai.eval_harness.one_shot import OneShotPublicEvalfrom products.slack_app.backend.services.slack_messages import SlackThreadMessagefrom products.slack_app.backend.temporal.activities import classifiers+from products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import FOLLOWUP_KEY, FollowupRoutingMatch, NoUnaskedWakeSUITE_KIND = SuiteKind.ONE_SHOT@@ -239,6 +240,11 @@ async def eval_followup_classifier(ctx: EvalContext) -> None:await OneShotPublicEval(experiment_name="slack-app-followup-classifier",+ offline_suite=offline_suite(+ "slack_app/eval_followup_classifier::eval_followup_classifier",+ metrics=["followup_routing", "no_unasked_wake"],+ suite_kind=SUITE_KIND,+ ),cases=[*DIRECTED_CASES, *CHATTER_CASES],scorers=[FollowupRoutingMatch(), NoUnaskedWake()],task=task,diff --git a/products/slack_app/evals/eval_model_classifier.py b/products/slack_app/evals/eval_model_classifier.pyindex 49b128f92ed3..d56f185160a6 100644--- a/products/slack_app/evals/eval_model_classifier.py+++ b/products/slack_app/evals/eval_model_classifier.py@@ -30,6 +30,7 @@ from products.posthog_ai.eval_harness.harness.requirements import SuiteKindfrom products.posthog_ai.eval_harness.one_shot import OneShotPublicEvalfrom products.slack_app.backend.services.model_catalogue import ModelChoicefrom products.slack_app.backend.temporal.activities import classifiers+from products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import MODEL_OVERRIDE_KEY, ModelOverrideMatch, NoUnaskedOverrideSUITE_KIND = SuiteKind.ONE_SHOT@@ -275,6 +276,11 @@ async def eval_model_classifier(ctx: EvalContext) -> None:await OneShotPublicEval(experiment_name="slack-app-model-classifier",+ offline_suite=offline_suite(+ "slack_app/eval_model_classifier::eval_model_classifier",+ metrics=["model_override_match", "no_unasked_override"],+ suite_kind=SUITE_KIND,+ ),cases=[*INSTRUCTION_CASES, *UNDERSPECIFIED_CASES, *SUBJECT_MATTER_CASES],scorers=[ModelOverrideMatch(), NoUnaskedOverride()],task=task,diff --git a/products/slack_app/evals/eval_posthog_first.py b/products/slack_app/evals/eval_posthog_first.pyindex 87a7216df952..2d9ea513269c 100644--- a/products/slack_app/evals/eval_posthog_first.py+++ b/products/slack_app/evals/eval_posthog_first.py@@ -28,6 +28,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall+from products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import (DOCS_SEARCH_TOOL_NAME,AnsweredAboutPostHogProduct,@@ -62,6 +63,10 @@ async def eval_posthog_first(ctx: EvalContext) -> None:await SandboxedPublicEval(experiment_name="sandboxed-slack-posthog-first-cli",+ offline_suite=offline_suite(+ "slack_app/eval_posthog_first::eval_posthog_first",+ metrics=["docs_searched", "posthog_product_answer", "no_third_party_recommendation"],+ ),cases=cases,scorers=[RequiredToolCall({DOCS_SEARCH_TOOL_NAME}, name="docs_searched"),diff --git a/products/slack_app/evals/eval_project_classifier.py b/products/slack_app/evals/eval_project_classifier.pyindex cfc917583ea0..c806af1dce15 100644--- a/products/slack_app/evals/eval_project_classifier.py+++ b/products/slack_app/evals/eval_project_classifier.py@@ -47,6 +47,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKindfrom products.posthog_ai.eval_harness.one_shot import OneShotPublicEvalfrom products.slack_app.backend.temporal.activities import classifiers+from products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import PROJECT_ROUTE_KEY, NoUnaskedProjectSwitch, ProjectRouteMatchSUITE_KIND = SuiteKind.ONE_SHOT@@ -531,6 +532,11 @@ async def eval_project_classifier(ctx: EvalContext) -> None:await OneShotPublicEval(experiment_name="slack-app-project-classifier",+ offline_suite=offline_suite(+ "slack_app/eval_project_classifier::eval_project_classifier",+ metrics=["project_route_match", "no_unasked_project_switch"],+ suite_kind=SUITE_KIND,+ ),cases=[*ROUTING_CASES,*CROWDED_ROUTING_CASES,diff --git a/products/slack_app/evals/eval_reply_suggestions.py b/products/slack_app/evals/eval_reply_suggestions.pyindex eb80eca083e9..4c0ee6837a62 100644--- a/products/slack_app/evals/eval_reply_suggestions.py+++ b/products/slack_app/evals/eval_reply_suggestions.py@@ -17,6 +17,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKindfrom products.posthog_ai.eval_harness.one_shot import OneShotPublicEvalfrom products.slack_app.backend.logic import reply_suggestions+from products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import REPLY_OFFER_KEY, NoUnaskedButtons, ReplyOfferMatch, SuggestedRepliesFitSUITE_KIND = SuiteKind.ONE_SHOT@@ -206,6 +207,11 @@ async def eval_reply_suggestions(ctx: EvalContext) -> None:await OneShotPublicEval(experiment_name="slack-app-reply-suggestions",+ offline_suite=offline_suite(+ "slack_app/eval_reply_suggestions::eval_reply_suggestions",+ metrics=["reply_offer", "no_unasked_buttons", "suggested_replies_fit"],+ suite_kind=SUITE_KIND,+ ),cases=[*WAITING_CASES, *BARE_CASES],scorers=[ReplyOfferMatch(), NoUnaskedButtons(), SuggestedRepliesFit()],task=task,diff --git a/products/slack_app/evals/offline_config.py b/products/slack_app/evals/offline_config.pynew file mode 100644index 000000000000..b86cc92224c9--- /dev/null+++ b/products/slack_app/evals/offline_config.py@@ -0,0 +1,52 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, dict[str, OfflineScorer]] = {+ "followup_classifier": {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557c-9b63-776a-91f3-113ab57c025a")),+ "followup_routing": OfflineScorer(kind="boolean", version=UUID("01a1557c-d1ca-7c68-96eb-5406fd8dde02")),+ "no_unasked_wake": OfflineScorer(kind="boolean", version=UUID("01a1557c-dd52-78d0-a1f1-1854076c09ac")),+ },+ "model_classifier": {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557c-e620-710d-82d2-de63990000f1")),+ "model_override_match": OfflineScorer(kind="boolean", version=UUID("01a1557c-ee78-7412-9b90-2ecf80591aa0")),+ "no_unasked_override": OfflineScorer(kind="boolean", version=UUID("01a1557c-fd45-7a93-8852-b64bcf13eeb8")),+ },+ "posthog_first": {+ "docs_searched": OfflineScorer(kind="boolean", version=UUID("01a1557d-0f40-7999-9f55-d969442fe434")),+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-0682-75c3-bb88-5a620ba6ce5d")),+ "no_third_party_recommendation": OfflineScorer(+ kind="boolean", version=UUID("01a1557d-2542-7805-a791-6c3ac5f875b2")+ ),+ "posthog_product_answer": OfflineScorer(kind="boolean", version=UUID("01a1557d-1aa6-7fb4-ae3b-2fab4be4c44e")),+ },+ "project_classifier": {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-2ef4-7ee8-9f53-ae73633d593d")),+ "no_unasked_project_switch": OfflineScorer(+ kind="boolean", version=UUID("01a1557d-415e-78ff-8bcb-1e6cc67b77ac")+ ),+ "project_route_match": OfflineScorer(kind="boolean", version=UUID("01a1557d-37df-756c-a5fb-73edde0b1561")),+ },+ "reply_suggestions": {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-49a7-7417-aa9f-4dbe00ec7b47")),+ "no_unasked_buttons": OfflineScorer(kind="boolean", version=UUID("01a1557d-5b16-755e-ab88-b824e5103a97")),+ "reply_offer": OfflineScorer(kind="boolean", version=UUID("01a1557d-5266-7853-a431-6213cfba1b5a")),+ "suggested_replies_fit": OfflineScorer(kind="boolean", version=UUID("01a1557d-bcc0-7a96-8d22-4d6e91998d5d")),+ },+}+++def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:+ domain = key.split("/", 1)[1].split("::", 1)[0].removeprefix("eval_")+ scorers = _SCORERS[domain]+ component = domain.replace("_", "-")+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: scorers[metric] for metric in enrolled_metrics},+ tags=("team:surfaces", f"component:slack-app/{component}"),+ )diff --git a/pyproject.toml b/pyproject.tomlindex 80340aa70da3..65749b80b297 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -105,7 +105,7 @@ dependencies = ["paramiko~=3.5.0","pillow==12.3.0","protobuf~=6.31.1",- "posthoganalytics==7.67.1",+ "posthoganalytics==7.73.0","polars==1.37.1","psycopg2-binary==2.9.11","psycopg[binary]==3.2.10",diff --git a/uv.lock b/uv.lockindex a1acc6bad859..390c4454d298 100644--- a/uv.lock+++ b/uv.lock@@ -6252,7 +6252,7 @@ requires-dist = [{ name = "playwright", specifier = "~=1.60.0" },{ name = "polars", specifier = "==1.37.1" },{ name = "posthog-hogland", specifier = "==0.4.1" },- { name = "posthoganalytics", specifier = "==7.67.1" },+ { name = "posthoganalytics", specifier = "==7.73.0" },{ name = "protobuf", specifier = "~=6.31.1" },{ name = "psycopg", extras = ["binary"], specifier = "==3.2.10" },{ name = "psycopg2-binary", specifier = "==2.9.11" },@@ -6419,7 +6419,7 @@ wheels = [[[package]]name = "posthoganalytics"-version = "7.67.1"+version = "7.73.0"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "backoff" },@@ -6427,9 +6427,9 @@ dependencies = [{ name = "requests" },{ name = "typing-extensions" },]-sdist = { url = "https://files.pythonhosted.org/packages/76/ff/1b18c85d904340607c70284c9d867df588c22c18d615e2ef67741126f64c/posthoganalytics-7.67.1.tar.gz", hash = "sha256:c861483738eb6da05eeaac3540f2d9f6622849e7a8bd3d05a4a0221a32abc120", size = 752005, upload-time = "2026-10-12T14:17:21.915Z" }+sdist = { url = "https://files.pythonhosted.org/packages/f4/8e/30cf3b4ce18f2224799e7fe38d147e4d8b6d3a54ef9998234100253180e6/posthoganalytics-7.73.0.tar.gz", hash = "sha256:6ea816666f116f132a4ae4e5fb3e36240e4f7de7c2a8afed12d2a258c227912a", size = 764745, upload-time = "2026-10-19T19:21:34.253Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/3f/d4/48706a5d0c2304f299a4564b86ce220eae3de6ab087a2bca9669644b8e3a/posthoganalytics-7.67.1-py3-none-any.whl", hash = "sha256:29a721c2e9ca383ed35deb736b390744e68ad9d2bf69b4bf08ebbc4487db97a4", size = 884481, upload-time = "2026-10-12T14:17:19.659Z" },+ { url = "https://files.pythonhosted.org/packages/66/01/d336d90c2b615b3db08864f004841d60bee2ebd4930cd1c2d761b30bc32a/posthoganalytics-7.73.0-py3-none-any.whl", hash = "sha256:a1ab79a6abf9d8d655d9e7e37fe8fe3a10781415037cf199c516a6013cc9a905", size = 899662, upload-time = "2026-10-19T19:21:32.037Z" },][[package]]