Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(aio): publish tagged offline eval results to posthog (#119849)

Radu Raiceae341b726273a
diff --git a/.agents/skills/writing-evals/SKILL.md b/.agents/skills/writing-evals/SKILL.md
index 06cfd02ec133..f27747bb113f 100644
--- a/.agents/skills/writing-evals/SKILL.md
+++ b/.agents/skills/writing-evals/SKILL.md
@@ -48,7 +48,9 @@ async def eval_my_thing(ctx: EvalContext) -> None:
- One suite = one Braintrust experiment. Bundle related cases into one suite; split only when the scorecard would be heterogeneous (self-skipping scorers stretch a shared scorer list across mildly different cases, but a scorecard where half the scorers skip half the cases is a sign to split).
- `experiment_name` is the Braintrust history key — renaming it resets cross-run comparison. Existing sandboxed suites end in `-cli` because the MCP server serves the `cli` surface.
-- `SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust and PostHog, including available input, output, and expected values. `SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service; local logs are still written. PostHog result uploads follow `no_send_logs` independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md).
+- `SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust, plus PostHog when `offline_suite` is configured. `SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service by default; local logs are still written. Set `send_to_posthog=True` with `offline_suite` to publish only to PostHog, or `send_to_posthog=False` to disable that destination independently. These options also apply to one-shot suites. Ordinary SDK and trace clients still honor `OPT_OUT_CAPTURE`. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md).
+- To enroll a suite in PostHog reporting, register its scorer kinds and immutable version UUIDs in the owning `offline_config.py` catalog, then pass `offline_suite=offline_suite("<domain>/<module>::<function>", metrics=[...])` to the runner. Use the PostHog AI catalog for built-in suites and the Slack app catalog for Slack suites. List every explicit scorer's `_name()` in `metrics`; the catalog adds `exit_code_zero` only for sandboxed suites. For one-shot suites, pass `suite_kind=SUITE_KIND`. The enrollment test discovers catalog-using modules and checks every suite function against its scorer list.
+- Uploading and tagging require `POSTHOG_OFFLINE_EVAL_API_KEY` to be a personal key with `offline_evaluation_ingestion:write` and `evaluation:write`. Ingestion-only personal or project secret keys can upload without tags. Upload success is recorded before the separate, bounded tag update, so a tagging failure or interruption keeps the results link.
## One-shot suites
@@ -135,7 +137,7 @@ class MyCheck(Scorer):
Rules that apply to both:
-- The harness adds `ExitCodeZero` to every experiment. Do not include it in a suite's `scorers`; an explicit duplicate is rejected.
+- The sandboxed harness adds `ExitCodeZero` to every experiment. Do not include it in a suite's `scorers`; an explicit duplicate is rejected. One-shot suites do not receive it.
- Reuse the other generic scorers first: `NoToolCall`, `RequiredToolCall`, and `LastToolCallNot` (all in `products.posthog_ai.eval_harness.scorers`).
- Always parse logs via `LogParser.cached(...)` so a case's log is parsed once, not once per scorer.
- Count only successful tool calls (`is_error=False`) — the agent is free to attempt and fail.
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.md
index 4f13ce74d02c..450f46879499 100644
--- a/docs/internal/ai-offline-evaluation-reporting.md
+++ b/docs/internal/ai-offline-evaluation-reporting.md
@@ -1,7 +1,7 @@
# Offline evaluation result reporting
The evaluation harness in `products/posthog_ai/eval_harness/` reports public-suite scorer results to Braintrust.
-The SQL suite also publishes those results to PostHog's offline experiment API when configured.
+Public suites under `products/posthog_ai/evals/` and `products/slack_app/evals/` also publish those results to PostHog's offline experiment API when configured.
The Signals pytest suites reuse the same publisher for their completed results.
Agent and judge trace capture remains available.
Reporting does not run the agent or scorers again.
@@ -9,9 +9,11 @@ Use `hogli evals` to run PostHog AI 2.0 suites. See the [eval harness guide](../
## Capture settings
-Evaluation result uploads to Braintrust and PostHog share the `no_send_logs` setting.
+`no_send_logs` controls Braintrust uploads and, by default, PostHog result uploads.
`SandboxedPublicEval` sets `no_send_logs=False` and uploads to Braintrust, plus PostHog when the suite has offline publishing configured.
-`SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service; local logs are still written.
+`SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service by default; local logs are still written.
+Set `send_to_posthog=True` alongside `offline_suite` to publish results to PostHog while keeping Braintrust uploads disabled.
+Set `send_to_posthog=False` to disable PostHog result uploads independently. The same options apply to one-shot suites.
Offline API uploads run through the Python SDK independently of `TEST` and `OPT_OUT_CAPTURE`.
The shared trace client retains its existing `TEST` and `OPT_OUT_CAPTURE` guards for `$ai_generation`, `$ai_span`, and `$ai_trace` events.
@@ -33,25 +35,27 @@ The scoring phase includes time waiting for the experiment results, rather than
A failed final metadata upload produces a warning and keeps the completed results available locally.
A scorer failure marks the case as an error. Statistics still include scores from other successful scorers.
-## SQL offline experiment pilot
+## Configured offline experiment suites
-The `sql/eval_sql::eval_sql` suite (`sandboxed-sql-cli`) opts into the offline experiment publisher.
-Other public suites continue reporting scores to Braintrust until they opt into offline publishing.
+Configured public PostHog AI and Slack app suites opt into the offline experiment publisher, including both sandboxed and one-shot suites.
+The 11 experiment-skill suites retain `no_send_logs=True` and explicitly set `send_to_posthog=True`, publishing results only to PostHog.
After scoring, the harness prepares the completed results in memory and queues delivery through the Python SDK's `AsyncOfflineEvaluations` client. It does not execute another agent run or judge call.
-`no_send_logs=True` suppresses this upload too.
+Without an explicit `send_to_posthog` override, `no_send_logs=True` suppresses this upload too.
-The [SQL suite definition](../../products/posthog_ai/evals/sql/eval_sql.py) pins destination project `2` and the seven immutable scorer version UUIDs.
+The [PostHog AI catalog](../../products/posthog_ai/evals/offline_config.py) and [Slack app catalog](../../products/slack_app/evals/offline_config.py) pin destination project `2`, scorer kinds, immutable version UUIDs, and component tags.
+Suites select their own metrics from the catalog; sandboxed suites also include the harness-provided `exit_code_zero` metric.
+The SQL suite retains its seven existing scorer versions.
Provide the API key through the environment before running the suite:
-| Variable | Purpose |
-| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
-| `POSTHOG_OFFLINE_EVAL_API_KEY` | Personal or project secret API key with `offline_evaluation_ingestion:write`. A public project token cannot authenticate uploads. |
-| `POSTHOG_OFFLINE_EVAL_HOST` | App origin; defaults to `https://us.posthog.com`. |
+| Variable | Purpose |
+| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `POSTHOG_OFFLINE_EVAL_API_KEY` | Personal API key with `offline_evaluation_ingestion:write` and `evaluation:write` to upload and tag results. Ingestion-only personal or project secret keys can upload without tags. A public project token cannot authenticate uploads. |
+| `POSTHOG_OFFLINE_EVAL_HOST` | App origin; defaults to `https://us.posthog.com`. |
The publisher normalizes the cloud ingestion hosts (`us.i.posthog.com`, `eu.i.posthog.com`) and the legacy `app.posthog.com` host to their app origins.
Uploads, result links, and trace-destination checks use that same origin.
-The suite uses these existing scorers:
+The SQL suite uses these existing scorers:
| Metric name | Scorer kind |
| ------------------------------------ | -------------------------------- |
@@ -72,15 +76,32 @@ export POSTHOG_OFFLINE_EVAL_API_KEY='your-secret-key'
hogli evals sql/eval_sql::eval_sql
```
-Without an API key, the SQL run reports that offline uploads are disabled.
+Without an API key, each enrolled suite reports that offline uploads are disabled.
Keys containing whitespace are rejected before evaluation starts; remove any spaces, tabs, or pasted trailing newlines.
Invalid configuration produces a warning and skips this destination; Braintrust and local reporting continue.
An upload failure also warns without changing the evaluation's score gate.
The final summary shows a PostHog results link only after a successful offline upload; otherwise it shows the upload status.
+### Scorer and experiment tags
+
+Score definitions and experiment runs use the same team and component tags:
+
+| Suites | Tags |
+| -------------------- | ------------------------------------------------------------------------------------------- |
+| PostHog AI | `team:surfaces`, `component:posthog-ai/<domain>` (for example `sql` or `product-analytics`) |
+| Slack app | `team:surfaces`, `component:slack-app/<area>` (for example `followup-classifier`) |
+| Signals grouping | `team:self-driving`, `component:signals/grouping` |
+| Signals scout safety | `team:self-driving`, `component:signals/scout-safety` |
+
+`OfflineEvalSuite.tags` supplies the experiment tags. Delivery records the successful upload and results link once the experiment has completed, then calls the publisher's `tag()` method to update tags through the Python SDK's `experiment.set_tags(...)`. Tagging has its own five-second deadline outside the upload deadline. Tags are mutable metadata, separate from the immutable creation declaration and scorer versions.
+
+To upload **and tag** experiments, `POSTHOG_OFFLINE_EVAL_API_KEY` must be a personal API key with both `offline_evaluation_ingestion:write` and `evaluation:write`. An ingestion-only personal or project secret key can still upload results, but the tag update will warn. Tagging failures, timeouts, and shutdown interruptions retain the successful upload status and results link and do not change the evaluation score gate.
+
+Each prepared experiment's tag update is attempted once per publisher, including when delivery pauses and resumes. A failed or interrupted replacement is not repeated automatically, because it may already have succeeded or someone may have edited the tags afterward. Inspect the experiment's tags before manually retrying a replacement.
+
### CI configuration
-CI jobs running this suite need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment.
+CI jobs running enrolled suites need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment.
Map the dedicated `ANTHROPIC_OFFLINE_EVALS_API_KEY` and `OPENAI_OFFLINE_EVALS_API_KEY` Actions secrets to `LLM_GATEWAY_ANTHROPIC_API_KEY` and `OPENAI_API_KEY` for the harness and judges.
Publishing failures remain warnings, so check the final harness upload status even when a CI job is green.
The harness sets the SDK's `run_source` to `ci` when `CI` is set, including manual Actions runs, and to `local` otherwise.
@@ -124,6 +145,7 @@ Existing local logs, enabled case exports, and Braintrust results remain availab
Both `SandboxedEval` and `OneShotEval` accept `offline_suite=OfflineEvalSuite(...)`.
Each suite declares its destination `project_id`, selects at least one metric in `scorer_kinds`, and pins the corresponding immutable version UUIDs in `scorer_versions`.
These fields belong to `OfflineEvalSuite` in code. Scorer version UUIDs must be distinct for the suite's enrolled metrics.
+The AI and Slack catalog helpers add `exit_code_zero` for sandboxed suites. One-shot suites pass `suite_kind=SUITE_KIND` to enroll only their declared metrics.
Numeric scores can use the domain of their configured scorer version. The API validates bounds and steps; the harness does not restrict every numeric metric to `[0, 1]`.
`OfflineEvalSuite.experiment_options` accepts the SDK's `ExperimentOptions` for external dataset identifiers, `prompt_version`, model/application versions, and `run_source`.
@@ -185,7 +207,7 @@ Runs appear in the offline experiments UI.
## Postgres experiment ingestion
The project API accepts offline experiment results behind the `ai-observability-offline-evaluations` feature flag.
-The configured SQL pilot calls this API to publish completed scorer results.
+The configured public suites call this API to publish completed scorer results.
Experiments and their items, results, and payloads belong to the exact project/environment in the request path.
Scorers and hosted datasets must belong to that same environment.
diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.md
index 87a88875a9c1..755d2b98c1ca 100644
--- a/products/posthog_ai/eval_harness/README.md
+++ b/products/posthog_ai/eval_harness/README.md
@@ -387,22 +387,25 @@ The summary includes the trial ID, the agent trace ID when available, phase dura
Case names and `--eval` filters stay the same.
`SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust, plus PostHog when the suite has offline publishing configured.
-`SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service; local logs are still written.
-PostHog result uploads follow `no_send_logs` independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients.
+`SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service by default; local logs are still written.
+Both sandboxed and one-shot suites can set `send_to_posthog=True` with `offline_suite` to publish only to PostHog while Braintrust uploads remain disabled, or `send_to_posthog=False` to disable PostHog uploads independently.
+The 11 experiment-skill suites use this PostHog-only mode.
+PostHog result uploads default to following `no_send_logs`, independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients.
See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md) for capture settings and scope.
### SQL offline experiment pilot
The `sql/eval_sql::eval_sql` suite can also send its seven scorer results to PostHog's offline experiment API.
It publishes the same completed case results through the Python SDK and keeps sending to Braintrust.
-Other public suites report scores to Braintrust until they opt into offline publishing.
-Agent and judge trace capture remains available, and `no_send_logs=True` suppresses offline result uploads.
+PostHog AI and Slack app public suites also opt into offline publishing with their own tagged scorer versions.
+Agent and judge trace capture remains available. `no_send_logs=True` suppresses offline result uploads unless `send_to_posthog=True` is explicitly set.
-Set `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal or project secret key with `offline_evaluation_ingestion:write` and access to project `2`.
+Set `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal key with `offline_evaluation_ingestion:write`, `evaluation:write`, and access to project `2` to upload and tag results. An ingestion-only personal or project secret key can upload results, but tagging will warn and leave the run untagged.
The [SQL suite definition](../evals/sql/eval_sql.py) pins that project ID and all seven immutable scorer version UUIDs in code.
`POSTHOG_OFFLINE_EVAL_HOST` defaults to `https://us.posthog.com`.
-The [pilot configuration](../../../docs/internal/ai-offline-evaluation-reporting.md#sql-offline-experiment-pilot) lists the metrics and explains local and CI key configuration.
+The [suite configuration](../../../docs/internal/ai-offline-evaluation-reporting.md#configured-offline-experiment-suites) lists the metrics and explains local and CI key configuration.
Delivery runs in a bounded background queue after local results are saved. Prepared uploads stay in memory, with up to two uploads running concurrently, at most 32 queued or active uploads, and a 60-second deadline per active upload.
+A completed upload records its results link before tagging begins. Tagging has a separate five-second deadline; failures, timeouts, and shutdown interruptions only warn and preserve the successful upload status.
Unfinished uploads pause before infrastructure teardown. After teardown, the harness retries the same prepared objects and identities with fresh deadlines, waiting up to the full 60-second upload deadline for delivery to finish.
Requests use a 5-second timeout and up to two retries within that overall deadline. Terminal API failures trigger a best-effort attempt to mark the experiment failed, limited to 5 seconds; pauses and interruptions leave it resumable.
Both sandboxed and one-shot suites can opt in through `offline_suite`. Rich scorer recording runs only when offline publishing is configured and uploads are allowed.
diff --git a/products/posthog_ai/eval_harness/base.py b/products/posthog_ai/eval_harness/base.py
index 663c38261065..6f9d5047f92a 100644
--- a/products/posthog_ai/eval_harness/base.py
+++ b/products/posthog_ai/eval_harness/base.py
@@ -166,12 +166,14 @@ class _BaseEvalRun:
no_send_logs: bool,
engine: EvalEngine | None = None,
offline_suite: OfflineEvalSuite | None = None,
+ send_to_posthog: bool | None = None,
) -> None:
self.experiment_name = experiment_name
self.cases = cases
self.ctx = ctx
self.is_public = is_public
self.no_send_logs = no_send_logs
+ self.send_to_posthog = not no_send_logs if send_to_posthog is None else send_to_posthog
self.offline_suite = offline_suite
self.offline_publisher: OfflineEvalPublisher | None = None
self.started_at = datetime.now(UTC)
@@ -394,7 +396,7 @@ class _BaseEvalRun:
await self._enqueue_offline_upload(result)
async def _enqueue_offline_upload(self, result: ExperimentResult) -> None:
- if self.offline_publisher is None or self.no_send_logs:
+ if self.offline_publisher is None or not self.send_to_posthog:
return
try:
upload = await asyncio.to_thread(
@@ -471,7 +473,7 @@ class _BaseEvalRun:
async def run(self) -> ExperimentResult:
eval_cases = self._build_eval_cases()
- if self.offline_suite is not None and not self.no_send_logs:
+ if self.offline_suite is not None and self.send_to_posthog:
try:
settings = OfflineEvalSettings.from_env(self.offline_suite)
if settings is not None:
@@ -607,6 +609,7 @@ class _SandboxedEvalRun(_BaseEvalRun):
is_public: bool,
no_send_logs: bool,
offline_suite: OfflineEvalSuite | None = None,
+ send_to_posthog: bool | None = None,
) -> None:
if any(isinstance(scorer, ExitCodeZero) for scorer in scorers):
raise ValueError("ExitCodeZero is added by the sandboxed eval harness; remove it from scorers")
@@ -618,6 +621,7 @@ class _SandboxedEvalRun(_BaseEvalRun):
is_public=is_public,
no_send_logs=no_send_logs,
offline_suite=offline_suite,
+ send_to_posthog=send_to_posthog,
)
# Narrow the infra-backed optionals once: they are None only when the
# harness didn't boot sandbox infrastructure for this run.
@@ -864,6 +868,7 @@ async def SandboxedEval(
is_public: bool = False,
no_send_logs: bool = True,
offline_suite: OfflineEvalSuite | None = None,
+ send_to_posthog: bool | None = None,
) -> ExperimentResult:
"""Run a sandboxed agent evaluation suite via Braintrust.
@@ -888,6 +893,7 @@ async def SandboxedEval(
is_public=is_public,
no_send_logs=no_send_logs,
offline_suite=offline_suite,
+ send_to_posthog=send_to_posthog,
)
return await run.run()
diff --git a/products/posthog_ai/eval_harness/harness/offline_delivery.py b/products/posthog_ai/eval_harness/harness/offline_delivery.py
index ab8eb84d9ab9..232d31a8c0c7 100644
--- a/products/posthog_ai/eval_harness/harness/offline_delivery.py
+++ b/products/posthog_ai/eval_harness/harness/offline_delivery.py
@@ -66,6 +66,7 @@ class OfflineEvalDelivery:
await self._report_failed(job, reason)
else:
await self._reporter.record_posthog_offline_result(job.upload.experiment_name, "uploaded", url=url)
+ await job.publisher.tag(job.upload)
def _finished(self, task: asyncio.Task[None]) -> None:
self._tasks.pop(task, None)
diff --git a/products/posthog_ai/eval_harness/offline_results.py b/products/posthog_ai/eval_harness/offline_results.py
index 5cfe9e1dcad9..02d4bc5014f9 100644
--- a/products/posthog_ai/eval_harness/offline_results.py
+++ b/products/posthog_ai/eval_harness/offline_results.py
@@ -4,6 +4,7 @@ import os
import re
import json
import asyncio
+import logging
from collections import Counter
from collections.abc import Mapping
from contextlib import suppress
@@ -30,6 +31,15 @@ from posthog.dataclasses import frozen
from .engines.types import CaseResult, ExperimentResult, ScorerResult
+logger = logging.getLogger(__name__)
+TAG_TIMEOUT_SECONDS = 5.0
+
+
+@frozen
+class OfflineScorer:
+ kind: ScorerKind
+ version: UUID
+
@frozen
class OfflineEvalSuite:
@@ -37,8 +47,21 @@ class OfflineEvalSuite:
project_id: int
scorer_kinds: Mapping[str, ScorerKind]
scorer_versions: Mapping[str, UUID]
+ tags: tuple[str, ...] = ()
experiment_options: ExperimentOptions = field(default_factory=lambda: ExperimentOptions())
+ @classmethod
+ def from_scorers(
+ cls, *, key: str, project_id: int, scorers: Mapping[str, OfflineScorer], tags: tuple[str, ...]
+ ) -> OfflineEvalSuite:
+ return cls(
+ key=key,
+ project_id=project_id,
+ scorer_kinds={metric: scorer.kind for metric, scorer in scorers.items()},
+ scorer_versions={metric: scorer.version for metric, scorer in scorers.items()},
+ tags=tags,
+ )
+
class OfflineUploadError(Exception):
@staticmethod
@@ -163,6 +186,7 @@ class OfflineEvalUpload:
experiment_name: str
experiment: ExperimentOptions
results: tuple[EvaluationResult, ...]
+ tags: tuple[str, ...] = ()
class OfflineEvalPublisher:
@@ -176,6 +200,7 @@ class OfflineEvalPublisher:
self.settings = settings
self.suite = suite
self.application_trace_ids_enabled = application_trace_ids_enabled
+ self._attempted_tag_updates: set[str] = set()
def _item_payload(
self,
@@ -377,19 +402,54 @@ class OfflineEvalPublisher:
experiment_name=experiment_name,
experiment=experiment,
results=tuple(scores),
+ tags=self.suite.tags,
)
+ def _client(self) -> AsyncOfflineEvaluations:
+ return AsyncOfflineEvaluations(
+ host=self.settings.host,
+ project_id=self.settings.project_id,
+ secret_key=self.settings.api_key.get_secret_value(),
+ timeout=self.settings.request_timeout,
+ max_retries=self.settings.max_retries,
+ )
+
+ async def tag(self, upload: OfflineEvalUpload) -> None:
+ if upload.host != self.settings.host or upload.project_id != self.settings.project_id:
+ raise OfflineUploadError("Upload destination differs from the configured host/project")
+ experiment_id = str(upload.experiment["id"])
+ if not upload.tags or experiment_id in self._attempted_tag_updates:
+ return
+ self._attempted_tag_updates.add(experiment_id)
+ try:
+ async with asyncio.timeout(TAG_TIMEOUT_SECONDS), self._client() as client:
+ await client.resume_experiment(experiment_id).set_tags(upload.tags)
+ except EvaluationAPIError as error:
+ logger.warning(
+ "PostHog offline results uploaded, but tagging failed for experiment %s (status=%s, code=%s). "
+ "Tagging requires a personal key with evaluation:write. Tag updates are not retried.",
+ experiment_id,
+ error.status,
+ OfflineUploadError._safe_token(error.code, path=False),
+ )
+ except TimeoutError:
+ logger.warning(
+ "PostHog offline results uploaded, but tagging timed out for experiment %s. "
+ "Tag updates are not retried.",
+ experiment_id,
+ )
+ except asyncio.CancelledError:
+ logger.warning(
+ "PostHog offline tag update interrupted for experiment %s; inspect its tags before retrying.",
+ experiment_id,
+ )
+ raise
+
async def publish(self, upload: OfflineEvalUpload) -> str:
if upload.host != self.settings.host or upload.project_id != self.settings.project_id:
raise OfflineUploadError("Upload destination differs from the configured host/project")
try:
- async with AsyncOfflineEvaluations(
- host=self.settings.host,
- project_id=self.settings.project_id,
- secret_key=self.settings.api_key.get_secret_value(),
- timeout=self.settings.request_timeout,
- max_retries=self.settings.max_retries,
- ) as client:
+ async with self._client() as client:
experiment = await client.create_experiment(name=upload.experiment_name, **upload.experiment)
try:
await experiment.upload_results(upload.results)
diff --git a/products/posthog_ai/eval_harness/one_shot.py b/products/posthog_ai/eval_harness/one_shot.py
index d7b79a8dfa6f..ec3219a59ad3 100644
--- a/products/posthog_ai/eval_harness/one_shot.py
+++ b/products/posthog_ai/eval_harness/one_shot.py
@@ -52,6 +52,7 @@ class _OneShotEvalRun(_BaseEvalRun):
no_send_logs: bool,
task_fn: OneShotTaskFn,
offline_suite: OfflineEvalSuite | None = None,
+ send_to_posthog: bool | None = None,
) -> None:
super().__init__(
experiment_name=experiment_name,
@@ -61,6 +62,7 @@ class _OneShotEvalRun(_BaseEvalRun):
is_public=is_public,
no_send_logs=no_send_logs,
offline_suite=offline_suite,
+ send_to_posthog=send_to_posthog,
)
self._task_fn = task_fn
@@ -114,6 +116,7 @@ async def OneShotEval(
is_public: bool = False,
no_send_logs: bool = True,
offline_suite: OfflineEvalSuite | None = None,
+ send_to_posthog: bool | None = None,
) -> ExperimentResult:
"""Run a one-shot evaluation suite via Braintrust.
@@ -132,6 +135,7 @@ async def OneShotEval(
no_send_logs=no_send_logs,
task_fn=task,
offline_suite=offline_suite,
+ send_to_posthog=send_to_posthog,
)
return await run.run()
@@ -140,4 +144,4 @@ OneShotPublicEval = partial(OneShotEval, is_public=True, no_send_logs=False)
"""One-shot evaluation whose Braintrust experiment is publicly accessible."""
OneShotPrivateEval = partial(OneShotEval, is_public=False, no_send_logs=True)
-"""One-shot evaluation that is not accessible publicly; local logs are the record."""
+"""One-shot evaluation with Braintrust uploads disabled; PostHog publishing is opt-in."""
diff --git a/products/posthog_ai/eval_harness/test/offline_api.py b/products/posthog_ai/eval_harness/test/offline_api.py
index f1e75ada44bf..df16412d148e 100644
--- a/products/posthog_ai/eval_harness/test/offline_api.py
+++ b/products/posthog_ai/eval_harness/test/offline_api.py
@@ -13,10 +13,14 @@ class OfflineServer:
self.results: dict[tuple[str, str], object] = {}
self.experiment: dict[str, object] = {}
self.status = "uploading"
+ self.tags: list[str] = []
def respond(self, request: httpx.Request) -> httpx.Response:
self.requests.append(request)
body = json.loads(request.content)
+ if request.method == "PATCH":
+ self.tags = body["tags"]
+ return httpx.Response(200, json={"id": self.experiment["id"], "tags": self.tags})
now = str(self.experiment.get("started_at", body.get("started_at")))
if request.url.path.endswith("/upload/"):
item_receipts = []
diff --git a/products/posthog_ai/eval_harness/test/test_offline_delivery.py b/products/posthog_ai/eval_harness/test/test_offline_delivery.py
index db7746051d82..a0f07f641c16 100644
--- a/products/posthog_ai/eval_harness/test/test_offline_delivery.py
+++ b/products/posthog_ai/eval_harness/test/test_offline_delivery.py
@@ -15,6 +15,7 @@ import httpx
from posthoganalytics.ai.evaluations import EvaluationItem, EvaluationResult
from pydantic import SecretStr
+from products.posthog_ai.eval_harness import offline_results
from products.posthog_ai.eval_harness.engines.types import EvalSummary
from products.posthog_ai.eval_harness.harness import reporting
from products.posthog_ai.eval_harness.harness.offline_delivery import OfflineEvalDelivery
@@ -198,6 +199,61 @@ async def test_finished_uploads_are_not_retried_or_marked_interrupted_while_repo
assert "403" in output
[email protected]("shutdown", ["pause", "deadline", "interrupt", "tag_timeout"])
+async def test_completed_upload_stays_successful_when_tagging_is_interrupted(
+ monkeypatch: pytest.MonkeyPatch,
+ capsys: pytest.CaptureFixture[str],
+ caplog: pytest.LogCaptureFixture,
+ shutdown: str,
+) -> None:
+ monkeypatch.setattr(offline_results.logger, "disabled", False)
+ caplog.set_level("WARNING", logger=offline_results.__name__)
+ tag_started = asyncio.Event()
+ release_tag = asyncio.Event()
+ requests: list[httpx.Request] = []
+ server = OfflineServer()
+
+ async def respond(request: httpx.Request) -> httpx.Response:
+ requests.append(request)
+ if request.method == "PATCH":
+ tag_started.set()
+ await release_tag.wait()
+ return server.respond(request)
+
+ publisher = _publisher(monkeypatch, respond)
+ reporter = ProgressReporter(total_suites=1)
+ await reporter.record_summary(
+ "experiment-1", EvalSummary(engine_name="braintrust", experiment_name="experiment-1", scores={})
+ )
+ delivery = OfflineEvalDelivery(reporter)
+ if shutdown == "tag_timeout":
+ monkeypatch.setattr(offline_results, "TAG_TIMEOUT_SECONDS", 0)
+ assert delivery.submit(publisher, replace(_upload(1), tags=("team:example",)))
+ if shutdown != "tag_timeout":
+ await asyncio.wait_for(tag_started.wait(), timeout=5)
+ if shutdown == "pause":
+ await delivery.pause()
+ release_tag.set()
+ elif shutdown == "deadline":
+ await delivery.drain(timeout=0)
+ else:
+ await delivery.cancel()
+ await delivery.drain()
+
+ reporter.print_final_summary([], exit_code=0, fail_under=None, duration_seconds=0)
+ output = capsys.readouterr().out
+ assert server.status == "completed"
+ assert sum(request.url.path.endswith("/complete/") for request in requests) == 1
+ assert output.count("POSTHOG OFFLINE") == 1
+ assert "PostHog: https://posthog.example.com/project/123/" in output
+ assert "upload failed" not in output
+ if shutdown == "tag_timeout":
+ assert "tagging timed out" in caplog.text
+ elif shutdown != "pause":
+ assert "tag update interrupted" in caplog.text
+
+
@pytest.mark.asyncio
@pytest.mark.parametrize("shutdown", ["deadline", "interrupt", "before_start", "paused"])
async def test_shutdown_cancels_active_and_waiting_uploads_and_reports_unsaved_results(
diff --git a/products/posthog_ai/eval_harness/test/test_offline_results.py b/products/posthog_ai/eval_harness/test/test_offline_results.py
index 83436608c9e5..4e3ae913157c 100644
--- a/products/posthog_ai/eval_harness/test/test_offline_results.py
+++ b/products/posthog_ai/eval_harness/test/test_offline_results.py
@@ -39,10 +39,10 @@ SUITE = OfflineEvalSuite(
STARTED_AT = datetime(2026, 1, 1, tzinfo=UTC)
-def _publisher(*, link_traces: bool = False) -> OfflineEvalPublisher:
+def _publisher(*, link_traces: bool = False, suite: OfflineEvalSuite = SUITE) -> OfflineEvalPublisher:
return OfflineEvalPublisher(
OfflineEvalSettings(project_id=123, api_key=SecretStr("test-only-token"), scorer_versions=SCORER_VERSIONS),
- SUITE,
+ suite,
application_trace_ids_enabled=link_traces,
)
@@ -368,6 +368,58 @@ def test_publish_retries_identical_batches_and_completes_after_all_acknowledgmen
assert url.endswith(f"/ai-evals/evaluations/offline/experiments/{EXPERIMENT_ID}")
+ [
+ ("success", None),
+ ("forbidden", 403),
+ ("server_error", 503),
+ ("lost_response", "lost_response"),
+ ("interrupted", "interrupted"),
+ ]
+)
+def test_publish_tags_completed_results_without_repeating_mutable_updates(
+ _name: str, tag_error: int | str | None
+) -> None:
+ publisher = _publisher(suite=replace(SUITE, tags=("team:example", "component:example")))
+ upload = _prepare(publisher, [_case()])
+ server = OfflineServer()
+ tag_requests: list[httpx.Request] = []
+
+ def respond(request: httpx.Request) -> httpx.Response:
+ if request.method == "PATCH":
+ tag_requests.append(request)
+ assert server.status == "completed"
+ if isinstance(tag_error, int):
+ return httpx.Response(tag_error)
+ response = server.respond(request)
+ if tag_error == "lost_response":
+ raise httpx.ReadError("Lost tag acknowledgment", request=request)
+ if tag_error == "interrupted":
+ raise asyncio.CancelledError
+ return response
+ return server.respond(request)
+
+ with patch("httpx.AsyncClient", new=partial(httpx.AsyncClient, transport=httpx.MockTransport(respond))):
+ assert asyncio.run(publisher.publish(upload)).endswith(EXPERIMENT_ID)
+ if tag_error == "interrupted":
+ with pytest.raises(asyncio.CancelledError):
+ asyncio.run(publisher.tag(upload))
+ else:
+ asyncio.run(publisher.tag(upload))
+ assert server.status == "completed"
+ assert len(server.results) == 2
+ assert "tags" not in server.experiment
+ assert server.tags == ([] if isinstance(tag_error, int) else list(upload.tags))
+
+ server.tags = ["edited-after-upload"]
+ asyncio.run(publisher.publish(upload))
+ asyncio.run(publisher.tag(upload))
+
+ assert len(tag_requests) == 1
+ assert json.loads(tag_requests[0].content) == {"tags": list(upload.tags)}
+ assert server.tags == ["edited-after-upload"]
+
+
@parameterized.expand(
[
("bad_request", 400),
diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py
index 19d29c050f55..bf1d6a95cd7a 100644
--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py
+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py
@@ -7,6 +7,7 @@ import json
import asyncio
import logging
import warnings
+import importlib
import subprocess
from collections.abc import Iterator
from contextlib import contextmanager
@@ -41,15 +42,16 @@ from products.posthog_ai.eval_harness.engines.types import (
)
from products.posthog_ai.eval_harness.harness import reporting
from products.posthog_ai.eval_harness.harness.cli import SkillDelivery
+from products.posthog_ai.eval_harness.harness.discovery import EvalSuite, discover_suites
from products.posthog_ai.eval_harness.harness.offline_delivery import OfflineEvalDelivery
from products.posthog_ai.eval_harness.harness.reporting import ProgressReporter, SuiteRunResult
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
from products.posthog_ai.eval_harness.harness.transcript import RunTranscript
from products.posthog_ai.eval_harness.harness.trial_stats import ScorerTrialStats
from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite
from products.posthog_ai.eval_harness.one_shot import OneShotEval
from products.posthog_ai.eval_harness.scorers import ExitCodeZero
from products.posthog_ai.eval_harness.test.offline_api import OfflineServer
-from products.posthog_ai.evals.sql import eval_sql as sql_eval
from products.tasks.backend.facade.agents import CustomPromptSandboxContext
@@ -379,17 +381,22 @@ def test_sandboxed_eval_run_adds_exit_code_scorer(monkeypatch: pytest.MonkeyPatc
@pytest.mark.parametrize(
- "suite_kind,enrolled,no_send_logs,configuration,http_status,delivery_state",
+ "suite_kind,enrolled,no_send_logs,send_to_posthog,posthog_enabled,configuration,http_status,delivery_state",
[
- ("sandboxed", True, False, "valid", 200, "ready"),
- ("sandboxed", True, False, "valid", 403, "ready"),
- ("sandboxed", True, False, "invalid", 200, "ready"),
- ("sandboxed", True, False, "absent", 200, "ready"),
- ("sandboxed", True, True, "valid", 200, "ready"),
- ("sandboxed", False, False, "valid", 200, "ready"),
- ("one-shot", True, False, "valid", 200, "ready"),
- ("sandboxed", True, False, "valid", 200, "unavailable"),
- ("sandboxed", True, False, "valid", 200, "closed"),
+ ("sandboxed", True, False, None, True, "valid", 200, "ready"),
+ ("sandboxed", True, False, None, True, "valid", 403, "ready"),
+ ("sandboxed", True, False, None, True, "invalid", 200, "ready"),
+ ("sandboxed", True, False, None, True, "absent", 200, "ready"),
+ ("sandboxed", True, True, None, False, "valid", 200, "ready"),
+ ("sandboxed", False, False, None, True, "valid", 200, "ready"),
+ ("one-shot", True, False, None, True, "valid", 200, "ready"),
+ ("sandboxed", True, False, None, True, "valid", 200, "unavailable"),
+ ("sandboxed", True, False, None, True, "valid", 200, "closed"),
+ ("sandboxed", True, True, True, True, "valid", 200, "ready"),
+ ("one-shot", True, True, True, True, "valid", 200, "ready"),
+ ("sandboxed", True, False, False, False, "valid", 200, "ready"),
+ ("sandboxed", True, True, True, True, "absent", 200, "ready"),
+ ("sandboxed", False, True, True, True, "valid", 200, "ready"),
],
)
@pytest.mark.asyncio
@@ -400,6 +407,8 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repo
suite_kind: Literal["sandboxed", "one-shot"],
enrolled: bool,
no_send_logs: bool,
+ send_to_posthog: bool | None,
+ posthog_enabled: bool,
configuration: Literal["valid", "invalid", "absent"],
http_status: int,
delivery_state: Literal["ready", "unavailable", "closed"],
@@ -473,6 +482,7 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repo
ctx=ctx,
is_public=not no_send_logs,
no_send_logs=no_send_logs,
+ send_to_posthog=send_to_posthog,
offline_suite=OfflineEvalSuite(
key="example/suite",
project_id=123,
@@ -494,14 +504,14 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repo
engine.run_experiment.assert_awaited_once()
engine_spec = engine.run_experiment.call_args.args[0]
assert engine_spec.no_send_logs is no_send_logs
- assert engine_spec.record_scorer_results is (enrolled and not no_send_logs and configuration == "valid")
+ assert engine_spec.record_scorer_results is (enrolled and posthog_enabled and configuration == "valid")
assert reporter.mean_score() == 1.0
reporter.print_final_summary([], exit_code=0, fail_under=None, duration_seconds=0)
output = capsys.readouterr().out
assert "Braintrust: https://braintrust.example.com/experiment/test" in output
assert "exit_code_zero: 100.0%" in output
assert not (tmp_path / "offline-pilot" / "latest" / "posthog-offline-upload.json").exists()
- if enrolled and not no_send_logs and configuration == "valid" and delivery_state == "ready":
+ if enrolled and posthog_enabled and configuration == "valid" and delivery_state == "ready":
declaration = json.loads(requests[0].content)
assert declaration["name"] == "offline-pilot"
assert declaration["expected_item_count"] == 1
@@ -530,34 +540,69 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repo
)
assert reason in output
assert "Results were not saved for retry." in output
- elif enrolled and not no_send_logs:
+ elif enrolled and posthog_enabled:
assert "PostHog offline: disabled:" in output
assert "POSTHOG_OFFLINE_EVAL_" in output
else:
assert "POSTHOG OFFLINE" not in output
+ "discovered",
+ [
+ suite
+ for suite in discover_suites()
+ if suite.fn.__module__.startswith(("products.posthog_ai.evals.", "products.slack_app.evals."))
+ and hasattr(importlib.import_module(suite.fn.__module__), "offline_suite")
+ ],
+ ids=lambda suite: suite.id,
+)
@pytest.mark.asyncio
-async def test_sql_suite_enrolls_every_scorer_in_offline_publishing(monkeypatch: pytest.MonkeyPatch) -> None:
+async def test_suite_enrolls_every_scorer_in_offline_publishing(
+ monkeypatch: pytest.MonkeyPatch, discovered: EvalSuite
+) -> None:
+ domain, module_name, suite_kind = discovered.domain, discovered.module_name, discovered.kind
+ module = importlib.import_module(discovered.fn.__module__)
+ if domain == "taxonomic_filter":
+ monkeypatch.setattr(module, "system_one_configured", lambda: True)
+ if module_name == "eval_event_match":
+ monkeypatch.setattr(module, "EVENT_MATCH_MODEL", MagicMock())
+ else:
+ monkeypatch.setattr(module, "fetch_search_intent_prompt", MagicMock())
run = AsyncMock()
- monkeypatch.setattr(sql_eval, "SandboxedPublicEval", run)
+ runner = "SandboxedPrivateEval" if domain == "experiments" else "SandboxedPublicEval"
+ if suite_kind == SuiteKind.ONE_SHOT:
+ runner = "OneShotPublicEval"
+ monkeypatch.setattr(module, runner, run)
- await sql_eval.eval_sql(MagicMock())
+ await discovered.fn(MagicMock(skill_delivery="bundled"))
run.assert_awaited_once()
suite = run.call_args.kwargs["offline_suite"]
- assert suite.key == "sql/eval_sql::eval_sql"
- assert suite.scorer_kinds == {
- "exit_code_zero": "boolean",
- "no_persistent_insight_save": "boolean",
- "execute_sql_called": "boolean",
- "answer_tool_not_typed_query": "boolean",
- "querying_posthog_data_skill_loaded": "boolean",
- "sql_schema_alignment": "numeric",
- "sql_result_message_alignment": "numeric",
- }
+ assert suite.key == discovered.id
+ component = (
+ f"slack-app/{module_name.removeprefix('eval_').replace('_', '-')}"
+ if domain == "slack_app"
+ else f"posthog-ai/{domain.replace('_', '-')}"
+ )
+ assert suite.tags == ("team:surfaces", f"component:{component}")
+ if domain == "experiments":
+ assert run.call_args.kwargs["send_to_posthog"] is True
+ assert set(suite.scorer_kinds.values()) == {"boolean"}
+ elif domain == "sql":
+ assert suite.scorer_kinds == {
+ "exit_code_zero": "boolean",
+ "no_persistent_insight_save": "boolean",
+ "execute_sql_called": "boolean",
+ "answer_tool_not_typed_query": "boolean",
+ "querying_posthog_data_skill_loaded": "boolean",
+ "sql_schema_alignment": "numeric",
+ "sql_result_message_alignment": "numeric",
+ }
configured_names = {scorer._name() for scorer in run.call_args.kwargs["scorers"]}
- assert set(suite.scorer_kinds) == {"exit_code_zero", *configured_names}
+ if suite_kind == SuiteKind.SANDBOXED:
+ configured_names.add("exit_code_zero")
+ assert set(suite.scorer_kinds) == set(suite.scorer_versions) == configured_names
@pytest.mark.parametrize(
diff --git a/products/posthog_ai/evals/ci/eval_basic.py b/products/posthog_ai/evals/ci/eval_basic.py
index fad97bc84cb5..f7edf05de098 100644
--- a/products/posthog_ai/evals/ci/eval_basic.py
+++ b/products/posthog_ai/evals/ci/eval_basic.py
@@ -15,6 +15,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.evals.cli_mcp.scorers import CalledTargetTool, SurfacedGeneratedAppUrl
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_bugfix(ctx: EvalContext) -> None:
@@ -37,6 +38,10 @@ async def eval_bugfix(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-bugfix-cli",
+ offline_suite=offline_suite(
+ "ci/eval_basic::eval_bugfix",
+ metrics=[],
+ ),
cases=cases,
scorers=[],
ctx=ctx,
@@ -86,6 +91,10 @@ async def eval_app_link_generation(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-app-link-generation-cli",
+ offline_suite=offline_suite(
+ "ci/eval_basic::eval_app_link_generation",
+ metrics=["called_target_tool", "surfaced_generated_app_url"],
+ ),
cases=cases,
scorers=[CalledTargetTool(), SurfacedGeneratedAppUrl()],
ctx=ctx,
diff --git a/products/posthog_ai/evals/cli_mcp/eval_render_ui.py b/products/posthog_ai/evals/cli_mcp/eval_render_ui.py
index 492d8f3e323d..d58fb71145fc 100644
--- a/products/posthog_ai/evals/cli_mcp/eval_render_ui.py
+++ b/products/posthog_ai/evals/cli_mcp/eval_render_ui.py
@@ -53,6 +53,7 @@ from products.posthog_ai.evals.cli_mcp.scorers import (
ExecBeforeRender,
RenderedEntityUi,
)
+from products.posthog_ai.evals.offline_config import offline_suite
# Renderable experiment tools (detail + results). The agent may render the detail
# view (``experiment-get``) or a results view for a status/results question. Two
@@ -96,6 +97,10 @@ async def eval_renders_entity_ui(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-renders-entity-ui-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_render_ui::eval_renders_entity_ui",
+ metrics=["rendered_entity_ui", "exec_before_render"],
+ ),
cases=cases,
scorers=[RenderedEntityUi(), ExecBeforeRender()],
ctx=ctx,
@@ -127,6 +132,10 @@ async def eval_no_render_for_query_results(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-no-render-for-query-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_render_ui::eval_no_render_for_query_results",
+ metrics=["called_target_tool", "did_not_render_ui", "did_not_cite_raw_sql_after_typed_query"],
+ ),
cases=cases,
scorers=[CalledTargetTool(), DidNotRenderUi(), DidNotCiteRawSqlAfterTypedQuery()],
ctx=ctx,
diff --git a/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py b/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py
index 116b50f68d69..7126dda1e16f 100644
--- a/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py
+++ b/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py
@@ -18,6 +18,7 @@ from products.posthog_ai.evals.cli_mcp.skill_distribution_scorers import (
skill_distribution_expectations,
)
from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues
+from products.posthog_ai.evals.offline_config import offline_suite
from products.tasks.backend.facade.agents import EVAL_INTERACTION_ORIGIN, CustomPromptSandboxContext
@@ -173,6 +174,18 @@ async def eval_skill_distribution(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-skill-distribution-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_skill_distribution::eval_skill_distribution",
+ metrics=[
+ "skill_search_first",
+ "expected_skill_discovered",
+ "expected_skill_loaded",
+ "excluded_skill_not_loaded",
+ "skill_loaded_before_tool",
+ "no_bundled_skill_bypass",
+ "no_exec_skill_bypass",
+ ],
+ ),
cases=cases,
scorers=[
SkillSearchFirst(),
diff --git a/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py b/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py
index 8031426b0a00..db1a4eb61e0f 100644
--- a/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py
+++ b/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py
@@ -27,6 +27,7 @@ from products.posthog_ai.evals.cli_mcp.skill_usage_seeder import (
PROJECT_SKILL_REFERENCE_PATH,
seed_project_skill,
)
+from products.posthog_ai.evals.offline_config import offline_suite
from products.tasks.backend.facade.agents import EVAL_INTERACTION_ORIGIN
@@ -138,6 +139,20 @@ async def eval_skill_usage(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-skill-usage-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_skill_usage::eval_skill_usage",
+ metrics=[
+ "skill_search_first",
+ "expected_skill_discovered",
+ "expected_skill_loaded",
+ "skill_loaded_before_tool",
+ "no_bundled_skill_bypass",
+ "no_exec_skill_bypass",
+ "expected_reference_pulled",
+ "search_recovery_after_zero_hit",
+ "skill_answer_correctness",
+ ],
+ ),
cases=cases,
scorers=[
SkillSearchFirst(),
diff --git a/products/posthog_ai/evals/cli_mcp/eval_workflow.py b/products/posthog_ai/evals/cli_mcp/eval_workflow.py
index 8f02a6ff7a34..688be90b6858 100644
--- a/products/posthog_ai/evals/cli_mcp/eval_workflow.py
+++ b/products/posthog_ai/evals/cli_mcp/eval_workflow.py
@@ -39,6 +39,7 @@ from products.posthog_ai.evals.cli_mcp.scorers import (
UsedJsonOutputFormat,
VerifiedEventBeforeQuery,
)
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_typo_recovery(ctx: EvalContext) -> None:
@@ -69,6 +70,10 @@ async def eval_typo_recovery(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-typo-recovery-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_workflow::eval_typo_recovery",
+ metrics=["recovered_to_correct_tool"],
+ ),
cases=cases,
scorers=[RecoveredToCorrectTool()],
ctx=ctx,
@@ -102,6 +107,10 @@ async def eval_schema_drilldown(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-schema-drilldown-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_workflow::eval_schema_drilldown",
+ metrics=["called_target_tool", "drilled_into_schema"],
+ ),
cases=cases,
scorers=[CalledTargetTool(), DrilledIntoSchema()],
ctx=ctx,
@@ -124,6 +133,10 @@ async def eval_search_first(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-search-first-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_workflow::eval_search_first",
+ metrics=["called_target_tool", "preferred_search_over_tools"],
+ ),
cases=cases,
scorers=[CalledTargetTool(), PreferredSearchOverTools()],
ctx=ctx,
@@ -154,6 +167,10 @@ async def eval_info_before_call(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-info-before-call-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_workflow::eval_info_before_call",
+ metrics=["called_target_tool", "info_before_call"],
+ ),
cases=cases,
scorers=[CalledTargetTool(), InfoBeforeCall()],
ctx=ctx,
@@ -190,6 +207,10 @@ async def eval_json_for_post_processing(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-json-postprocess-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_workflow::eval_json_for_post_processing",
+ metrics=["called_target_tool", "used_json_output_format", "ran_python_post_processing"],
+ ),
cases=cases,
scorers=[
CalledTargetTool(),
@@ -216,6 +237,10 @@ async def eval_verify_event_before_query(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cli-mcp-verify-event-cli",
+ offline_suite=offline_suite(
+ "cli_mcp/eval_workflow::eval_verify_event_before_query",
+ metrics=["called_target_tool", "verified_event_before_query"],
+ ),
cases=cases,
scorers=[CalledTargetTool(), VerifiedEventBeforeQuery()],
ctx=ctx,
diff --git a/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py b/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py
index a525ff96d299..7b434dd3a7cf 100644
--- a/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py
+++ b/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py
@@ -27,6 +27,7 @@ from products.posthog_ai.evals.cohorts.scorers import (
CohortFromQueryUsed,
QueryTargetsActorColumn,
)
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_cohort_from_query(ctx: EvalContext) -> None:
@@ -59,6 +60,10 @@ async def eval_cohort_from_query(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-cohorts-from-query-cli",
+ offline_suite=offline_suite(
+ "cohorts/eval_cohort_from_query::eval_cohort_from_query",
+ metrics=["cohort_from_query_used", "query_targets_actor_column", "no_uuid_batching"],
+ ),
cases=cases,
scorers=[
CohortFromQueryUsed(),
diff --git a/products/posthog_ai/evals/data_warehouse/eval_information_schema.py b/products/posthog_ai/evals/data_warehouse/eval_information_schema.py
index 262a3f131759..136827813293 100644
--- a/products/posthog_ai/evals/data_warehouse/eval_information_schema.py
+++ b/products/posthog_ai/evals/data_warehouse/eval_information_schema.py
@@ -58,6 +58,7 @@ from products.posthog_ai.evals.data_warehouse.synthesizer import (
TYPE_NEEDLE_TABLE,
VIEW_NEEDLE_NAME,
)
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_dw_discovery(ctx: EvalContext) -> None:
@@ -110,6 +111,10 @@ async def eval_dw_discovery(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-warehouse-discovery-cli",
+ offline_suite=offline_suite(
+ "data_warehouse/eval_information_schema::eval_dw_discovery",
+ metrics=["information_schema_queried", "agentic_search_used", "needle_table_identified"],
+ ),
cases=cases,
scorers=[
InformationSchemaQueried(),
@@ -172,6 +177,16 @@ async def eval_dw_value_retrieval(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-warehouse-retrieval-cli",
+ offline_suite=offline_suite(
+ "data_warehouse/eval_information_schema::eval_dw_value_retrieval",
+ metrics=[
+ "information_schema_queried",
+ "information_schema_before_answer",
+ "needle_value_retrieved",
+ "answer_query_ran",
+ "warehouse_answer_correctness",
+ ],
+ ),
cases=cases,
scorers=[
InformationSchemaQueried(),
@@ -228,6 +243,15 @@ async def eval_dw_relationships(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-warehouse-relationships-cli",
+ offline_suite=offline_suite(
+ "data_warehouse/eval_information_schema::eval_dw_relationships",
+ metrics=[
+ "information_schema_queried",
+ "relationship_discovery",
+ "join_path_traversed",
+ "warehouse_answer_correctness",
+ ],
+ ),
cases=cases,
scorers=[
InformationSchemaQueried(),
@@ -271,6 +295,16 @@ async def eval_dw_table_relevancy(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-warehouse-relevancy-cli",
+ offline_suite=offline_suite(
+ "data_warehouse/eval_information_schema::eval_dw_table_relevancy",
+ metrics=[
+ "information_schema_queried",
+ "agentic_search_used",
+ "needle_table_identified",
+ "stale_table_avoided",
+ "warehouse_answer_correctness",
+ ],
+ ),
cases=cases,
scorers=[
InformationSchemaQueried(),
diff --git a/products/posthog_ai/evals/error_tracking/eval_events_sampling.py b/products/posthog_ai/evals/error_tracking/eval_events_sampling.py
index e5f7fc38c973..70965ed042db 100644
--- a/products/posthog_ai/evals/error_tracking/eval_events_sampling.py
+++ b/products/posthog_ai/evals/error_tracking/eval_events_sampling.py
@@ -33,6 +33,7 @@ from products.posthog_ai.evals.error_tracking.scorers import (
IssueIdMatchesTarget,
)
from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues
+from products.posthog_ai.evals.offline_config import offline_suite
def _events_case(
@@ -107,6 +108,10 @@ async def eval_events_sampling(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-error-tracking-events-sampling-cli",
+ offline_suite=offline_suite(
+ "error_tracking/eval_events_sampling::eval_events_sampling",
+ metrics=["no_error_tracking_write", "events_tool_used", "events_args_alignment", "issue_id_matches_target"],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"),
diff --git a/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py b/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py
index a9dceb789cbe..3e82533e9c17 100644
--- a/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py
+++ b/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py
@@ -40,6 +40,7 @@ from products.posthog_ai.evals.error_tracking.scorers import (
IssueInputAlignment,
)
from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues
+from products.posthog_ai.evals.offline_config import offline_suite
def _drilldown_case(
@@ -121,6 +122,15 @@ async def eval_issue_drilldown(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-error-tracking-issue-drilldown-cli",
+ offline_suite=offline_suite(
+ "error_tracking/eval_issue_drilldown::eval_issue_drilldown",
+ metrics=[
+ "no_error_tracking_write",
+ "issue_drilldown_order",
+ "issue_id_matches_target",
+ "issue_input_alignment",
+ ],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"),
diff --git a/products/posthog_ai/evals/error_tracking/eval_issues_list.py b/products/posthog_ai/evals/error_tracking/eval_issues_list.py
index b133a2e24c32..a016b2a02cd5 100644
--- a/products/posthog_ai/evals/error_tracking/eval_issues_list.py
+++ b/products/posthog_ai/evals/error_tracking/eval_issues_list.py
@@ -39,6 +39,7 @@ from products.posthog_ai.evals.error_tracking.scorers import (
IssuesListToolUsed,
)
from products.posthog_ai.evals.error_tracking.seeders import seed_error_tracking_issues
+from products.posthog_ai.evals.offline_config import offline_suite
def _list_case(
@@ -86,6 +87,10 @@ async def eval_issues_list(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-error-tracking-issues-list-cli",
+ offline_suite=offline_suite(
+ "error_tracking/eval_issues_list::eval_issues_list",
+ metrics=["no_error_tracking_write", "issues_list_tool_used", "issues_list_input_alignment"],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=ERROR_TRACKING_WRITE_TOOLS, name="no_error_tracking_write"),
diff --git a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py
index 450164d638f1..9e7b72f954a1 100644
--- a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py
+++ b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py
@@ -39,6 +39,7 @@ from products.posthog_ai.evals.experiments.seeders import (
seed_running_experiment,
seed_uneven_split_experiment,
)
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_bias_uneven_split(ctx: EvalContext) -> None:
@@ -107,6 +108,11 @@ async def eval_bias_uneven_split(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-diagnose-bias-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_bias_uneven_split::eval_bias_uneven_split",
+ metrics=["diagnosis_group", "surfaces_all_findings"],
+ ),
cases=cases,
scorers=[
CitesDiagnosticGroup(),
diff --git a/products/posthog_ai/evals/experiments/eval_create_control_variant.py b/products/posthog_ai/evals/experiments/eval_create_control_variant.py
index 493d53c8e9e6..b7a6e5e93a0e 100644
--- a/products/posthog_ai/evals/experiments/eval_create_control_variant.py
+++ b/products/posthog_ai/evals/experiments/eval_create_control_variant.py
@@ -34,6 +34,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPrivateEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import RequiredToolCall
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_create_control_variant(ctx: EvalContext) -> None:
@@ -56,6 +57,11 @@ async def eval_create_control_variant(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-create-control-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_create_control_variant::eval_create_control_variant",
+ metrics=["experiment_created_successfully"],
+ ),
cases=cases,
scorers=[
# The single outcome scorer: did experiment-create return without error?
diff --git a/products/posthog_ai/evals/experiments/eval_empty_experiment.py b/products/posthog_ai/evals/experiments/eval_empty_experiment.py
index a6374b4ebca3..9e962c140071 100644
--- a/products/posthog_ai/evals/experiments/eval_empty_experiment.py
+++ b/products/posthog_ai/evals/experiments/eval_empty_experiment.py
@@ -52,6 +52,7 @@ from products.posthog_ai.evals.experiments.seeders import (
seed_inactive_flag_experiment,
seed_running_experiment,
)
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_empty_experiment(ctx: EvalContext) -> None:
@@ -139,6 +140,11 @@ async def eval_empty_experiment(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-diagnose-empty-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_empty_experiment::eval_empty_experiment",
+ metrics=["diagnosis_group"],
+ ),
cases=cases,
scorers=[
CitesDiagnosticGroup(),
diff --git a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py
index 98b786ab6e29..262879511506 100644
--- a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py
+++ b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py
@@ -50,6 +50,7 @@ from products.posthog_ai.evals.experiments.seeders import (
seed_day_old_experiment,
seed_running_experiment,
)
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_interpretation_traps(ctx: EvalContext) -> None:
@@ -161,6 +162,11 @@ async def eval_interpretation_traps(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-diagnose-interpretation-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_interpretation_traps::eval_interpretation_traps",
+ metrics=["diagnosis_group", "advises_against_shipping"],
+ ),
cases=cases,
scorers=[
CitesDiagnosticGroup(),
diff --git a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py
index f1d0f956b849..e085458bcad5 100644
--- a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py
+++ b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py
@@ -40,16 +40,10 @@ from products.posthog_ai.evals.experiments.scorers import (
RecommendsShipVariant,
)
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
+from products.posthog_ai.evals.offline_config import offline_suite
-# Cases are bundled into one suite function rather than one function per case
-# because the harness wraps a list of cases in a single SandboxedPrivateEval
-# call: one Braintrust experiment, with concurrency bounded by the harness's
-# global sandbox semaphore and per-case filtering via --eval. Splitting into
-# one function per case would create N separate Braintrust experiments and break
-# cross-case comparison. Every existing sandboxed eval (eval_funnel,
-# eval_retention, eval_insight_retrieval, eval_rollout_skill) follows this same
-# single-function-multiple-cases shape.
+# Keep the cases in one experiment so their scores can be compared together.
async def eval_lifecycle_skills(ctx: EvalContext) -> None:
cases: list[SandboxedEvalCase] = [
# Case 1: Ship variant must require confirmation.
@@ -91,6 +85,17 @@ async def eval_lifecycle_skills(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-lifecycle-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_lifecycle_skills::eval_lifecycle_skills",
+ metrics=[
+ "resolved_experiment_by_name",
+ "no_unconfirmed_ship",
+ "asked_for_confirmation",
+ "duplicate_unique_flag_key",
+ "recommends_ship_variant",
+ ],
+ ),
cases=cases,
scorers=[
# Agent must look up the named experiment.
diff --git a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py
index 5a3d26b5003b..5b4cac746250 100644
--- a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py
+++ b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py
@@ -25,6 +25,7 @@ from products.posthog_ai.evals.experiments.scorers import (
validate_retention_metric,
)
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_metric_schema_discovery(ctx: EvalContext) -> None:
@@ -53,6 +54,11 @@ async def eval_metric_schema_discovery(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-metric-schema-discovery-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_metric_schema_discovery::eval_metric_schema_discovery",
+ metrics=["called_experiment_update", "first_update_metric_shape"],
+ ),
cases=cases,
scorers=[
RequiredToolCall(required={"experiment-update"}, name="called_experiment_update"),
diff --git a/products/posthog_ai/evals/experiments/eval_midrun_changes.py b/products/posthog_ai/evals/experiments/eval_midrun_changes.py
index 36baf69749ef..7db3ca178df9 100644
--- a/products/posthog_ai/evals/experiments/eval_midrun_changes.py
+++ b/products/posthog_ai/evals/experiments/eval_midrun_changes.py
@@ -61,6 +61,7 @@ from products.posthog_ai.evals.experiments.seeders import (
seed_paused_and_resumed_experiment,
seed_running_experiment,
)
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_midrun_changes(ctx: EvalContext) -> None:
@@ -152,6 +153,11 @@ async def eval_midrun_changes(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-diagnose-midrun-changes-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_midrun_changes::eval_midrun_changes",
+ metrics=["diagnosis_group", "does_not_recommend_edit", "advises_against_shipping"],
+ ),
cases=cases,
scorers=[
CitesDiagnosticGroup(),
diff --git a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py
index 852cfa4fdf1c..4604a3d058e9 100644
--- a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py
+++ b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py
@@ -43,6 +43,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.evals.experiments.scorers import CitesDiagnosticGroup
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_numbers_vs_sql(ctx: EvalContext) -> None:
@@ -154,6 +155,11 @@ async def eval_numbers_vs_sql(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-diagnose-numbers-vs-sql-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_numbers_vs_sql::eval_numbers_vs_sql",
+ metrics=["diagnosis_group"],
+ ),
cases=cases,
scorers=[
CitesDiagnosticGroup(),
diff --git a/products/posthog_ai/evals/experiments/eval_rollout_skill.py b/products/posthog_ai/evals/experiments/eval_rollout_skill.py
index fc646a7b4c51..59170914ec04 100644
--- a/products/posthog_ai/evals/experiments/eval_rollout_skill.py
+++ b/products/posthog_ai/evals/experiments/eval_rollout_skill.py
@@ -8,8 +8,9 @@ cheap to reproduce while iterating on the skill's description.
Maps to stress-test finding #7 (skills rarely auto-load), specifically
the variant-split rows (2.5, 5.3, 5.4, 5.5) in STEP_1_HARNESS.md. Uses
-``SandboxedPrivateEval`` so it runs without a Braintrust API key —
-local logs and PostHog ``$ai_trace`` events still emit.
+``SandboxedPrivateEval`` with ``send_to_posthog=True`` to publish results
+only to PostHog. The harness still requires a Braintrust API key for its
+evaluation engine, but Braintrust result uploads remain disabled.
To run:
flox activate -- bash -c "set -a; source .env; set +a; python -m products.posthog_ai.eval_harness.harness eval_rollout_skill"
@@ -21,6 +22,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPrivateEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.retrieval.scorers import SkillLoaded
SKILL_NAME = "configuring-experiment-rollout"
@@ -39,6 +41,11 @@ async def eval_rollout_skill(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-rollout-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_rollout_skill::eval_rollout_skill",
+ metrics=["skill_loaded"],
+ ),
cases=cases,
scorers=[
SkillLoaded(skill_name=SKILL_NAME),
diff --git a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py
index b1a8cc250afc..178b7f8bc1d8 100644
--- a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py
+++ b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py
@@ -13,6 +13,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import RequiredToolCall
from products.posthog_ai.evals.experiments.scorers import SharedMetricValidationVerdict
from products.posthog_ai.evals.experiments.seeders import SHARED_METRIC_NAME, seed_shared_metric_purchase_count
+from products.posthog_ai.evals.offline_config import offline_suite
async def eval_shared_metric_validation(ctx: EvalContext) -> None:
@@ -50,6 +51,11 @@ async def eval_shared_metric_validation(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-shared-metric-validation-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_shared_metric_validation::eval_shared_metric_validation",
+ metrics=["loaded_shared_metric", "shared_metric_validation_verdict"],
+ ),
cases=cases,
scorers=[
# Agent must actually read the seeded metric — either via list
diff --git a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py
index d34379afeb26..9d58348b98a5 100644
--- a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py
+++ b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py
@@ -40,6 +40,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCall
from products.posthog_ai.evals.experiments.scorers import AskedForConfirmation
from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experiment
+from products.posthog_ai.evals.offline_config import offline_suite
# Event name chosen to be obviously fictional so the agent can't plausibly
# justify it as a typo of a real PostHog event in the seeded team.
@@ -66,6 +67,11 @@ async def eval_unknown_events_confirmation(ctx: EvalContext) -> None:
await SandboxedPrivateEval(
experiment_name="sandboxed-experiments-unknown-events-cli",
+ send_to_posthog=True,
+ offline_suite=offline_suite(
+ "experiments/eval_unknown_events_confirmation::eval_unknown_events_confirmation",
+ metrics=["resolved_experiment_by_name", "no_silent_update_with_unknown_event", "asked_for_confirmation"],
+ ),
cases=cases,
scorers=[
# Agent must look up the named experiment first.
diff --git a/products/posthog_ai/evals/offline_config.py b/products/posthog_ai/evals/offline_config.py
new file mode 100644
index 000000000000..4479074fb0ec
--- /dev/null
+++ b/products/posthog_ai/evals/offline_config.py
@@ -0,0 +1,225 @@
+from collections.abc import Sequence
+from uuid import UUID
+
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer
+
+_SCORERS: dict[str, dict[str, OfflineScorer]] = {
+ "ci": {
+ "called_target_tool": OfflineScorer(kind="boolean", version=UUID("01a1557d-d3f2-7807-88ff-12d4c4b2679c")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-ca2a-75c8-b3a4-5b9a0a405a64")),
+ "surfaced_generated_app_url": OfflineScorer(
+ kind="boolean", version=UUID("01a1557d-dda3-782a-a2cb-dac316f6e7d6")
+ ),
+ },
+ "cli_mcp": {
+ "called_target_tool": OfflineScorer(kind="boolean", version=UUID("01a1557e-090d-700e-8bc7-584c54a089fc")),
+ "did_not_cite_raw_sql_after_typed_query": OfflineScorer(
+ kind="boolean", version=UUID("01a1557e-1c0f-7302-96f5-ccdb2eed2b27")
+ ),
+ "did_not_render_ui": OfflineScorer(kind="boolean", version=UUID("01a1557e-1330-7ce0-a05a-914503a66cb0")),
+ "drilled_into_schema": OfflineScorer(kind="boolean", version=UUID("01a1557e-fd96-783d-8de2-dc4ddb5d6468")),
+ "excluded_skill_not_loaded": OfflineScorer(
+ kind="boolean", version=UUID("01a1557e-aca4-7826-8388-c66db0633772")
+ ),
+ "exec_before_render": OfflineScorer(kind="boolean", version=UUID("01a1557d-fb44-769a-b5cf-6d96a971182d")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-e737-7f9a-bb69-3dae3e58df7c")),
+ "expected_reference_pulled": OfflineScorer(
+ kind="boolean", version=UUID("01a1557e-d5f9-7e74-8369-2e4627c02a4a")
+ ),
+ "expected_skill_discovered": OfflineScorer(
+ kind="boolean", version=UUID("01a1557e-354f-726f-8e3e-c04ad776e8a7")
+ ),
+ "expected_skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a1557e-a3f3-72c2-899f-3d5d9c903b39")),
+ "info_before_call": OfflineScorer(kind="boolean", version=UUID("01a1557f-1350-7eb4-8821-1b38bb2d8325")),
+ "no_bundled_skill_bypass": OfflineScorer(kind="boolean", version=UUID("01a1557e-c286-7545-a3ba-14a8e354cafe")),
+ "no_exec_skill_bypass": OfflineScorer(kind="boolean", version=UUID("01a1557e-cb02-7a20-a775-8dd932e4d07c")),
+ "preferred_search_over_tools": OfflineScorer(
+ kind="boolean", version=UUID("01a1557f-0679-7a2e-9cc6-a0559e513dcd")
+ ),
+ "ran_python_post_processing": OfflineScorer(
+ kind="boolean", version=UUID("01a1557f-85a2-7380-94ff-6d5de79a68a1")
+ ),
+ "recovered_to_correct_tool": OfflineScorer(
+ kind="boolean", version=UUID("01a1557e-f33b-7e99-9408-f3548316c76f")
+ ),
+ "rendered_entity_ui": OfflineScorer(kind="boolean", version=UUID("01a1557d-f13a-7eda-8447-e54d0ea0e7b1")),
+ "search_recovery_after_zero_hit": OfflineScorer(
+ kind="boolean", version=UUID("01a1557e-dedb-70fc-a4a4-c8be5f7c4939")
+ ),
+ "skill_answer_correctness": OfflineScorer(kind="boolean", version=UUID("01a1557e-e9b1-79b4-82c4-3639a73a9d66")),
+ "skill_loaded_before_tool": OfflineScorer(kind="boolean", version=UUID("01a1557e-b9d5-7cb3-9136-2a8176f7a8f3")),
+ "skill_search_first": OfflineScorer(kind="boolean", version=UUID("01a1557e-2b07-7fd6-95ed-e1f5cc459aff")),
+ "used_json_output_format": OfflineScorer(kind="boolean", version=UUID("01a1557f-7ce7-7ee9-8a62-0d219935a7cf")),
+ "verified_event_before_query": OfflineScorer(
+ kind="boolean", version=UUID("01a1557f-8e04-7c10-a084-3bcd95fd5fe8")
+ ),
+ },
+ "cohorts": {
+ "cohort_from_query_used": OfflineScorer(kind="boolean", version=UUID("01a1557f-a2a9-7cd5-a647-24fbc34fdaca")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557f-974a-79e8-a589-a41e4ff8bf2d")),
+ "no_uuid_batching": OfflineScorer(kind="boolean", version=UUID("01a1557f-bbad-7dcc-b140-d5c1fe69a9d2")),
+ "query_targets_actor_column": OfflineScorer(
+ kind="boolean", version=UUID("01a1557f-b048-7ebe-8db4-2d89aa79787e")
+ ),
+ },
+ "data_warehouse": {
+ "agentic_search_used": OfflineScorer(kind="boolean", version=UUID("01a1557f-db0e-76f5-ade3-847fb199d132")),
+ "answer_query_ran": OfflineScorer(kind="boolean", version=UUID("01a15580-3330-7033-a26b-bd7241292083")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557f-c598-76ff-8312-e8261f7357c2")),
+ "information_schema_before_answer": OfflineScorer(
+ kind="boolean", version=UUID("01a1557f-efac-7e78-b26d-3c45cab69449")
+ ),
+ "information_schema_queried": OfflineScorer(
+ kind="boolean", version=UUID("01a1557f-d0c7-7e57-9845-8c4b1f4d3004")
+ ),
+ "join_path_traversed": OfflineScorer(kind="boolean", version=UUID("01a15580-55a9-77a1-9815-f898ba72b96a")),
+ "needle_table_identified": OfflineScorer(kind="boolean", version=UUID("01a1557f-e4dc-75a1-9606-24647018c5c5")),
+ "needle_value_retrieved": OfflineScorer(kind="boolean", version=UUID("01a15580-29c9-7d6e-afa2-c650a1958662")),
+ "relationship_discovery": OfflineScorer(kind="boolean", version=UUID("01a15580-452b-7ccd-9a91-d87dace78770")),
+ "stale_table_avoided": OfflineScorer(kind="boolean", version=UUID("01a15580-5f0c-7e20-baca-eb544aba6950")),
+ "warehouse_answer_correctness": OfflineScorer(
+ kind="numeric", version=UUID("01a15580-3c17-77bc-af79-dfb04a075815")
+ ),
+ },
+ "error_tracking": {
+ "events_args_alignment": OfflineScorer(kind="boolean", version=UUID("01a15580-83eb-7edb-88ed-e3a278c1bad6")),
+ "events_tool_used": OfflineScorer(kind="boolean", version=UUID("01a15580-7acc-7e5b-bb9c-6e670b54e66d")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15580-67d1-7a2c-bdfe-d4de0553020c")),
+ "issue_drilldown_order": OfflineScorer(kind="boolean", version=UUID("01a15580-9762-7200-b518-2da4b1d854ce")),
+ "issue_id_matches_target": OfflineScorer(kind="boolean", version=UUID("01a15580-8e80-70df-b21a-15945ed85bdb")),
+ "issue_input_alignment": OfflineScorer(kind="boolean", version=UUID("01a15581-0aec-75b5-b0e7-e371e5853164")),
+ "issues_list_input_alignment": OfflineScorer(
+ kind="boolean", version=UUID("01a15581-1e2c-7ae1-9522-0f54a9c6e4a9")
+ ),
+ "issues_list_tool_used": OfflineScorer(kind="boolean", version=UUID("01a15581-148b-74d7-93d7-8c3110ab2a95")),
+ "no_error_tracking_write": OfflineScorer(kind="boolean", version=UUID("01a15580-7158-7c13-856a-ff7669b5e0f2")),
+ },
+ "experiments": {
+ "advises_against_shipping": OfflineScorer(kind="boolean", version=UUID("01a1558e-f848-7def-b3d8-970df097d0e5")),
+ "asked_for_confirmation": OfflineScorer(kind="boolean", version=UUID("01a1558f-1274-77da-9d79-2e410a63cbbc")),
+ "called_experiment_update": OfflineScorer(kind="boolean", version=UUID("01a1558f-1ac0-7176-8fbe-8730bbd9074a")),
+ "diagnosis_group": OfflineScorer(kind="boolean", version=UUID("01a1558f-2481-7ab2-95a5-a9f4e7e087eb")),
+ "does_not_recommend_edit": OfflineScorer(kind="boolean", version=UUID("01a1558f-314c-7d2f-bf87-43b94f4917ff")),
+ "duplicate_unique_flag_key": OfflineScorer(
+ kind="boolean", version=UUID("01a1558f-3b7c-7347-8fd2-a2eba94f5723")
+ ),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1558f-4447-7a87-b21c-4418a4152cc0")),
+ "experiment_created_successfully": OfflineScorer(
+ kind="boolean", version=UUID("01a1558f-55a3-7584-8fae-39efa8c6b11e")
+ ),
+ "first_update_metric_shape": OfflineScorer(
+ kind="boolean", version=UUID("01a1558f-5ee1-77eb-b837-843569f351b3")
+ ),
+ "loaded_shared_metric": OfflineScorer(kind="boolean", version=UUID("01a1558f-69dc-7efe-adfe-1debc1acbfa0")),
+ "no_silent_update_with_unknown_event": OfflineScorer(
+ kind="boolean", version=UUID("01a1558f-71f0-74a5-aec8-a03922367ecc")
+ ),
+ "no_unconfirmed_ship": OfflineScorer(kind="boolean", version=UUID("01a1558f-7d13-7a20-9035-2aa9cdddb4ac")),
+ "recommends_ship_variant": OfflineScorer(kind="boolean", version=UUID("01a1558f-8617-7a83-87a2-27f3adf7d54a")),
+ "resolved_experiment_by_name": OfflineScorer(
+ kind="boolean", version=UUID("01a1558f-8e45-7480-b669-ae4c5d655366")
+ ),
+ "shared_metric_validation_verdict": OfflineScorer(
+ kind="boolean", version=UUID("01a1558f-9a54-7743-905c-bb10f59c3264")
+ ),
+ "skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a1558f-a299-7e75-938e-07c2ff9c747f")),
+ "surfaces_all_findings": OfflineScorer(kind="boolean", version=UUID("01a1558f-ad52-7b81-8a93-d7e602bf0125")),
+ },
+ "product_analytics": {
+ "answer_tool_not_execute_sql": OfflineScorer(
+ kind="boolean", version=UUID("01a15581-3926-7c14-a907-e994586bd216")
+ ),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15581-2721-7343-9ee8-d4cb29adbe1a")),
+ "first_relevant_tool": OfflineScorer(kind="boolean", version=UUID("01a15581-9357-7433-89c5-b66294e872d5")),
+ "first_relevant_tool_t1": OfflineScorer(kind="boolean", version=UUID("01a15581-bca9-7af2-a9e1-fd29a55c7291")),
+ "first_relevant_tool_t2": OfflineScorer(kind="boolean", version=UUID("01a15581-c7a2-7706-9d55-f2bd6640b817")),
+ "first_relevant_tool_t3": OfflineScorer(kind="boolean", version=UUID("01a15581-d0cf-7b8a-ae3f-558eb875af9c")),
+ "first_relevant_tool_t4": OfflineScorer(kind="boolean", version=UUID("01a15581-db3d-7eb7-8184-ef777d8900c1")),
+ "funnel_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a15581-4c67-7321-bad6-608fbbe18564")),
+ "funnel_time_range_relevancy": OfflineScorer(
+ kind="boolean", version=UUID("01a15581-5652-79d8-a3b0-48be756495eb")
+ ),
+ "insight_shape": OfflineScorer(kind="boolean", version=UUID("01a15581-e3df-7304-b06e-8ae4ad58ccf5")),
+ "no_persistent_insight_save": OfflineScorer(
+ kind="boolean", version=UUID("01a15581-2fc3-7c68-b349-d44a670b232d")
+ ),
+ "providing_insights_skill_loaded": OfflineScorer(
+ kind="boolean", version=UUID("01a15581-9c84-7f36-837a-e20bbb99ac17")
+ ),
+ "retention_schema_alignment": OfflineScorer(
+ kind="numeric", version=UUID("01a15581-5faa-7855-b46a-90ff034dae82")
+ ),
+ "retention_time_range_relevancy": OfflineScorer(
+ kind="boolean", version=UUID("01a15581-6891-7f9b-bbb4-199b8b2edec0")
+ ),
+ "saved_insight": OfflineScorer(kind="boolean", version=UUID("01a15581-ee89-7754-b0dd-6012efeaa1c4")),
+ "schema_discovery_order": OfflineScorer(kind="boolean", version=UUID("01a15581-7429-7fff-85a7-9c223feddee6")),
+ "trends_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a15581-a78c-7d03-bb2d-de43ab247f81")),
+ "trends_time_range_relevancy": OfflineScorer(
+ kind="boolean", version=UUID("01a15581-b135-7a70-ae09-b3d0e93bc894")
+ ),
+ "verified_event_exists": OfflineScorer(kind="boolean", version=UUID("01a15581-41af-7c8a-bf72-5e7c4ef63b0b")),
+ },
+ "retrieval": {
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15581-f842-7336-8237-f5a8bf96d317")),
+ "info_before_execute_sql": OfflineScorer(kind="boolean", version=UUID("01a15582-4169-7666-8583-809f548f5ee7")),
+ "information_schema_before_sql": OfflineScorer(
+ kind="boolean", version=UUID("01a15582-3921-7c7f-9066-b2e9a767cd33")
+ ),
+ "lookup_id_in_output": OfflineScorer(kind="boolean", version=UUID("01a15582-2caf-725e-b7f0-ac9a02218b09")),
+ "skill_loaded": OfflineScorer(kind="boolean", version=UUID("01a15582-0134-7aaa-b171-52e5fb222bfa")),
+ },
+ "sql": {
+ "answer_tool_not_typed_query": OfflineScorer(
+ kind="boolean", version=UUID("01a11725-3002-7a3f-b9b5-3426bdec5f00")
+ ),
+ "execute_sql_called": OfflineScorer(kind="boolean", version=UUID("01a11725-20aa-78c3-af96-1aae34bca429")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a11724-d52d-704a-af08-2ceaf5c65a6d")),
+ "no_persistent_insight_save": OfflineScorer(
+ kind="boolean", version=UUID("01a11725-17f6-7673-93e9-4d525fa121ab")
+ ),
+ "querying_posthog_data_skill_loaded": OfflineScorer(
+ kind="boolean", version=UUID("01a11725-54c1-71e0-b606-132d5f4cb339")
+ ),
+ "sql_result_message_alignment": OfflineScorer(
+ kind="numeric", version=UUID("01a11725-6940-7f49-b979-11b3d6521393")
+ ),
+ "sql_schema_alignment": OfflineScorer(kind="numeric", version=UUID("01a11725-5da7-7dae-ab09-1c0beaa77224")),
+ },
+ "surveys": {
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15582-4a50-75e1-a17a-3d483bb95586")),
+ "no_forbidden_survey_side_effects": OfflineScorer(
+ kind="boolean", version=UUID("01a15582-5371-7f81-8356-53532c73fac2")
+ ),
+ "survey_create_alignment": OfflineScorer(kind="numeric", version=UUID("01a15582-711a-7b7a-bd93-061a83ea909d")),
+ "survey_create_outcome": OfflineScorer(kind="boolean", version=UUID("01a15582-5d5a-786a-965f-ee2075db057d")),
+ "survey_create_returned_id": OfflineScorer(
+ kind="boolean", version=UUID("01a15582-6656-7fb2-b67d-6b918a8363e0")
+ ),
+ "survey_id_in_final_message": OfflineScorer(
+ kind="boolean", version=UUID("01a15582-7cea-7b35-9632-1fd1c3e22185")
+ ),
+ },
+ "taxonomic_filter": {
+ "event_match_found": OfflineScorer(kind="boolean", version=UUID("01a15582-8f96-76a8-a6b0-9ad9c932f12a")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a15582-85bb-71aa-b4c8-dc008015ab30")),
+ "no_wrong_event_match": OfflineScorer(kind="boolean", version=UUID("01a15582-9ce5-711e-b612-3b0be41c2669")),
+ "no_wrong_switch": OfflineScorer(kind="boolean", version=UUID("01a15582-b862-766e-b1ee-6664b70a9a93")),
+ "search_intent_match": OfflineScorer(kind="boolean", version=UUID("01a15582-a5dd-7f46-8681-5f0dd77b22d4")),
+ "switch_when_needed": OfflineScorer(kind="boolean", version=UUID("01a15582-aef0-748b-8282-c92f901762b5")),
+ },
+}
+
+
+def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:
+ domain = key.split("/", 1)[0]
+ scorers = _SCORERS[domain]
+ component = domain.replace("_", "-")
+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics
+ return OfflineEvalSuite.from_scorers(
+ key=key,
+ project_id=2,
+ scorers={metric: scorers[metric] for metric in enrolled_metrics},
+ tags=("team:surfaces", f"component:posthog-ai/{component}"),
+ )
diff --git a/products/posthog_ai/evals/product_analytics/eval_funnel.py b/products/posthog_ai/evals/product_analytics/eval_funnel.py
index 2bf6c71adc29..7c21408e64c8 100644
--- a/products/posthog_ai/evals/product_analytics/eval_funnel.py
+++ b/products/posthog_ai/evals/product_analytics/eval_funnel.py
@@ -15,6 +15,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall, RequiredToolCall
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import (
INSIGHT_WRITE_TOOLS,
FunnelSchemaAlignment,
@@ -587,6 +588,16 @@ async def eval_funnel(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-funnels",
+ offline_suite=offline_suite(
+ "product_analytics/eval_funnel::eval_funnel",
+ metrics=[
+ "no_persistent_insight_save",
+ "answer_tool_not_execute_sql",
+ "verified_event_exists",
+ "funnel_schema_alignment",
+ "funnel_time_range_relevancy",
+ ],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
diff --git a/products/posthog_ai/evals/product_analytics/eval_retention.py b/products/posthog_ai/evals/product_analytics/eval_retention.py
index ee77fff5b70f..9cfb1d1f797a 100644
--- a/products/posthog_ai/evals/product_analytics/eval_retention.py
+++ b/products/posthog_ai/evals/product_analytics/eval_retention.py
@@ -8,6 +8,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import (
INSIGHT_WRITE_TOOLS,
RetentionSchemaAlignment,
@@ -116,6 +117,15 @@ async def eval_retention(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-retention-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_retention::eval_retention",
+ metrics=[
+ "no_persistent_insight_save",
+ "answer_tool_not_execute_sql",
+ "retention_schema_alignment",
+ "retention_time_range_relevancy",
+ ],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
diff --git a/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py b/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py
index 3c4f9b5bb735..c0d1c1522856 100644
--- a/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py
+++ b/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py
@@ -22,6 +22,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS, SchemaDiscoveryOrder
@@ -84,6 +85,10 @@ async def eval_schema_discovery(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-schema-discovery-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_schema_discovery::eval_schema_discovery",
+ metrics=["no_persistent_insight_save", "answer_tool_not_execute_sql", "schema_discovery_order"],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
diff --git a/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py b/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py
index ecf2747c183f..c024a7fd52c5 100644
--- a/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py
+++ b/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py
@@ -17,6 +17,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import NoToolCall
from products.posthog_ai.evals.cli_mcp.scorers import FirstRelevantTool
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS
ANALYSIS_QUERY_TOOLS = frozenset(
@@ -119,6 +120,10 @@ async def eval_sql_shaped_typed_query_routing(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-sql-shaped-typed-query-routing-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_sql_shaped_routing::eval_sql_shaped_typed_query_routing",
+ metrics=["no_persistent_insight_save", "first_relevant_tool"],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
@@ -150,6 +155,10 @@ async def eval_sql_shaped_sql_controls(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-sql-shaped-sql-controls-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_sql_shaped_routing::eval_sql_shaped_sql_controls",
+ metrics=["no_persistent_insight_save", "first_relevant_tool"],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
diff --git a/products/posthog_ai/evals/product_analytics/eval_trends.py b/products/posthog_ai/evals/product_analytics/eval_trends.py
index 89d9d73e2289..1fa92c1989f4 100644
--- a/products/posthog_ai/evals/product_analytics/eval_trends.py
+++ b/products/posthog_ai/evals/product_analytics/eval_trends.py
@@ -18,6 +18,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import (
INSIGHT_WRITE_TOOLS,
TrendsSchemaAlignment,
@@ -301,6 +302,16 @@ async def eval_trends(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-trends",
+ offline_suite=offline_suite(
+ "product_analytics/eval_trends::eval_trends",
+ metrics=[
+ "no_persistent_insight_save",
+ "answer_tool_not_execute_sql",
+ "providing_insights_skill_loaded",
+ "trends_schema_alignment",
+ "trends_time_range_relevancy",
+ ],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
diff --git a/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py b/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py
index d02003e698e6..d4c51ef0e614 100644
--- a/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py
+++ b/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py
@@ -33,6 +33,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import NoToolCall
from products.posthog_ai.evals.cli_mcp.scorers import FirstRelevantTool
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS
from .eval_sql_shaped_routing import ANALYSIS_QUERY_TOOLS
@@ -106,6 +107,16 @@ async def eval_typed_runner_no_lock_in_control(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-typed-runner-no-lock-in-control-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_typed_runner_lock_in::eval_typed_runner_no_lock_in_control",
+ metrics=[
+ "no_persistent_insight_save",
+ "first_relevant_tool_t1",
+ "first_relevant_tool_t2",
+ "first_relevant_tool_t3",
+ "first_relevant_tool_t4",
+ ],
+ ),
cases=cases,
scorers=_lock_in_scorers(turn2_expected="query-trends"),
ctx=ctx,
@@ -140,6 +151,16 @@ async def eval_typed_runner_lock_in(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-typed-runner-lock-in-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_typed_runner_lock_in::eval_typed_runner_lock_in",
+ metrics=[
+ "no_persistent_insight_save",
+ "first_relevant_tool_t1",
+ "first_relevant_tool_t2",
+ "first_relevant_tool_t3",
+ "first_relevant_tool_t4",
+ ],
+ ),
cases=cases,
scorers=_lock_in_scorers(turn2_expected="execute-sql"),
ctx=ctx,
diff --git a/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py b/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py
index b9b670559824..2073fc7b27bf 100644
--- a/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py
+++ b/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py
@@ -20,6 +20,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCall
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS, InsightShape
TIME_SERIES_DISPLAYS = ["ActionsLineGraph", "ActionsBar", "ActionsAreaGraph", "ActionsUnstackedBar"]
@@ -104,6 +105,10 @@ async def eval_visualization_choice(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-visualization-choice-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_visualization_choice::eval_visualization_choice",
+ metrics=["no_persistent_insight_save", "insight_shape"],
+ ),
cases=cases,
scorers=[
NoToolCall(forbidden=INSIGHT_WRITE_TOOLS, name="no_persistent_insight_save"),
@@ -151,6 +156,10 @@ async def eval_saved_insight_display(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-saved-insight-display-cli",
+ offline_suite=offline_suite(
+ "product_analytics/eval_visualization_choice::eval_saved_insight_display",
+ metrics=["saved_insight", "insight_shape"],
+ ),
cases=cases,
scorers=[RequiredToolCall(["insight-create"], name="saved_insight"), InsightShape()],
ctx=ctx,
diff --git a/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py b/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py
index ef9a878e98c1..4c4ca5bfa3f3 100644
--- a/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py
+++ b/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py
@@ -23,6 +23,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.seeders.insight import seed_insight_noise
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.retrieval.scorers import LookupIdInOutput, SkillLoaded
SKILL_NAME = "querying-posthog-data"
@@ -72,6 +73,10 @@ async def eval_insight_retrieval(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-retrieval-cli",
+ offline_suite=offline_suite(
+ "retrieval/eval_insight_retrieval::eval_insight_retrieval",
+ metrics=["skill_loaded", "lookup_id_in_output"],
+ ),
cases=cases,
scorers=[
SkillLoaded(skill_name=SKILL_NAME),
diff --git a/products/posthog_ai/evals/retrieval/eval_system_table_search.py b/products/posthog_ai/evals/retrieval/eval_system_table_search.py
index fcb2dc80d646..adde6d5e19bd 100644
--- a/products/posthog_ai/evals/retrieval/eval_system_table_search.py
+++ b/products/posthog_ai/evals/retrieval/eval_system_table_search.py
@@ -34,6 +34,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.seeders.insight import seed_insight_noise
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.retrieval.scorers import InfoCalledBeforeTool, InformationSchemaBeforeSql
@@ -73,6 +74,10 @@ async def eval_system_table_search(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-system-table-search-cli",
+ offline_suite=offline_suite(
+ "retrieval/eval_system_table_search::eval_system_table_search",
+ metrics=["information_schema_before_sql", "info_before_execute_sql"],
+ ),
cases=cases,
scorers=[
InformationSchemaBeforeSql(),
diff --git a/products/posthog_ai/evals/sql/eval_sql.py b/products/posthog_ai/evals/sql/eval_sql.py
index 12d78f18dd8b..cea50c41769a 100644
--- a/products/posthog_ai/evals/sql/eval_sql.py
+++ b/products/posthog_ai/evals/sql/eval_sql.py
@@ -10,13 +10,11 @@ To run:
from __future__ import annotations
-from uuid import UUID
-
from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
-from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite
from products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS
from products.posthog_ai.evals.retrieval.scorers import SkillLoaded
from products.posthog_ai.evals.sql.scorers import AnswerQueryRan, SQLResultMessageAlignment, SQLSchemaAlignment
@@ -375,27 +373,16 @@ WHERE properties.interests IS NOT NULL
await SandboxedPublicEval(
experiment_name="sandboxed-sql-cli",
- offline_suite=OfflineEvalSuite(
- key="sql/eval_sql::eval_sql",
- project_id=2,
- scorer_kinds={
- "exit_code_zero": "boolean",
- "no_persistent_insight_save": "boolean",
- "execute_sql_called": "boolean",
- "answer_tool_not_typed_query": "boolean",
- "querying_posthog_data_skill_loaded": "boolean",
- "sql_schema_alignment": "numeric",
- "sql_result_message_alignment": "numeric",
- },
- scorer_versions={
- "exit_code_zero": UUID("01a11724-d52d-704a-af08-2ceaf5c65a6d"),
- "no_persistent_insight_save": UUID("01a11725-17f6-7673-93e9-4d525fa121ab"),
- "execute_sql_called": UUID("01a11725-20aa-78c3-af96-1aae34bca429"),
- "answer_tool_not_typed_query": UUID("01a11725-3002-7a3f-b9b5-3426bdec5f00"),
- "querying_posthog_data_skill_loaded": UUID("01a11725-54c1-71e0-b606-132d5f4cb339"),
- "sql_schema_alignment": UUID("01a11725-5da7-7dae-ab09-1c0beaa77224"),
- "sql_result_message_alignment": UUID("01a11725-6940-7f49-b979-11b3d6521393"),
- },
+ offline_suite=offline_suite(
+ "sql/eval_sql::eval_sql",
+ metrics=[
+ "no_persistent_insight_save",
+ "execute_sql_called",
+ "answer_tool_not_typed_query",
+ "querying_posthog_data_skill_loaded",
+ "sql_schema_alignment",
+ "sql_result_message_alignment",
+ ],
),
cases=cases,
scorers=[
diff --git a/products/posthog_ai/evals/surveys/eval_surveys.py b/products/posthog_ai/evals/surveys/eval_surveys.py
index 187a36a4eb5f..7214ac6a247c 100644
--- a/products/posthog_ai/evals/surveys/eval_surveys.py
+++ b/products/posthog_ai/evals/surveys/eval_surveys.py
@@ -10,6 +10,7 @@ from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import NoToolCall
from products.posthog_ai.eval_harness.seeders.survey import seed_survey_feature_flags
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.product_analytics.scorers import INSIGHT_WRITE_TOOLS
from products.posthog_ai.evals.surveys.scorers import (
SURVEY_FORBIDDEN_WRITE_TOOLS,
@@ -270,6 +271,16 @@ async def eval_surveys(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-surveys-cli",
+ offline_suite=offline_suite(
+ "surveys/eval_surveys::eval_surveys",
+ metrics=[
+ "no_forbidden_survey_side_effects",
+ "survey_create_outcome",
+ "survey_create_returned_id",
+ "survey_create_alignment",
+ "survey_id_in_final_message",
+ ],
+ ),
cases=cases,
scorers=[
NoToolCall(
diff --git a/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py b/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py
index 493605b721a5..721285f7f3a4 100644
--- a/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py
+++ b/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py
@@ -36,6 +36,7 @@ from products.posthog_ai.eval_harness.config import BaseEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.taxonomic_filter.scorers import EVENT_MATCH_KEY, EventMatchFound, NoWrongEventMatch
SUITE_KIND = SuiteKind.ONE_SHOT
@@ -103,6 +104,11 @@ async def eval_event_match(ctx: EvalContext) -> None:
await OneShotPublicEval(
experiment_name="taxonomic-filter-event-match",
+ offline_suite=offline_suite(
+ "taxonomic_filter/eval_event_match::eval_event_match",
+ metrics=["event_match_found", "no_wrong_event_match"],
+ suite_kind=SUITE_KIND,
+ ),
cases=[*DESCRIBED_CASES, *NO_MATCH_CASES],
scorers=[EventMatchFound(), NoWrongEventMatch()],
task=task,
diff --git a/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py b/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py
index 9f78585527d6..cae15518e0c1 100644
--- a/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py
+++ b/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py
@@ -46,6 +46,7 @@ from products.posthog_ai.eval_harness.config import BaseEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval
+from products.posthog_ai.evals.offline_config import offline_suite
from products.posthog_ai.evals.taxonomic_filter.scorers import (
SEARCH_INTENT_KEY,
NoWrongSwitch,
@@ -357,6 +358,11 @@ async def eval_search_intent(ctx: EvalContext) -> None:
await OneShotPublicEval(
experiment_name="ml-inference-search-intent",
+ offline_suite=offline_suite(
+ "taxonomic_filter/eval_search_intent::eval_search_intent",
+ metrics=["search_intent_match", "switch_when_needed", "no_wrong_switch"],
+ suite_kind=SUITE_KIND,
+ ),
cases=[*WRONG_TAB_CASES, *RIGHT_TAB_CASES, *PARTIAL_CASES, *CONTEXT_CASES],
scorers=[SearchIntentMatch(), SwitchWhenNeeded(), NoWrongSwitch()],
task=task,
diff --git a/products/signals/backend/test/test_eval_reporting.py b/products/signals/backend/test/test_eval_reporting.py
index 8469247a7885..98631c355c22 100644
--- a/products/signals/backend/test/test_eval_reporting.py
+++ b/products/signals/backend/test/test_eval_reporting.py
@@ -127,12 +127,22 @@ def test_disabled_reporting_never_attempts_network(_name: str, disabled: bool, a
assert asyncio.run(recorder.publish()) is None
-def test_upload_failure_is_reported_without_hiding_completed_scores(capsys: pytest.CaptureFixture[str]) -> None:
[email protected]("failed_action", ["upload", "tag"])
+def test_upload_failure_is_reported_without_hiding_completed_scores(
+ capsys: pytest.CaptureFixture[str], failed_action: str
+) -> None:
+ server = OfflineServer()
+
+ def respond(request: httpx.Request) -> httpx.Response:
+ if failed_action == "upload" or request.method == "PATCH":
+ return httpx.Response(403)
+ return server.respond(request)
+
with (
patch.dict(os.environ, {"POSTHOG_OFFLINE_EVAL_API_KEY": "test-only-token"}),
patch(
"httpx.AsyncClient",
- new=partial(httpx.AsyncClient, transport=httpx.MockTransport(lambda request: httpx.Response(403))),
+ new=partial(httpx.AsyncClient, transport=httpx.MockTransport(respond)),
),
):
recorder = SignalsEvalRecorder(SCOUT_SAFETY_SUITE)
@@ -141,6 +151,14 @@ def test_upload_failure_is_reported_without_hiding_completed_scores(capsys: pyte
item_name="case",
metrics=[EvalMetric(name="correct_classification", score=0)],
)
- assert asyncio.run(recorder.publish()) is None
- assert "upload failed" in capsys.readouterr().err
+ url = asyncio.run(recorder.publish())
+ output = capsys.readouterr().err
+ if failed_action == "upload":
+ assert url is None
+ assert "upload failed" in output
+ else:
+ assert server.status == "completed"
+ assert url is not None and recorder.experiment_id in url
+ assert f"uploaded: {url}" in output
+ assert "upload failed" not in output
assert recorder.cases[0].scores["scout_safety_correct_classification"] == 0
diff --git a/products/signals/eval/offline.py b/products/signals/eval/offline.py
index 43c9532f5e0d..f6054ab6e271 100644
--- a/products/signals/eval/offline.py
+++ b/products/signals/eval/offline.py
@@ -135,4 +135,5 @@ class SignalsEvalRecorder:
self._report(f"upload failed: {reason}. Results were not saved for retry.")
return None
self._report(f"uploaded: {url}")
+ await self.publisher.tag(upload)
return url
diff --git a/products/signals/eval/offline_config.py b/products/signals/eval/offline_config.py
index f829a71a0b00..19a1d5d390eb 100644
--- a/products/signals/eval/offline_config.py
+++ b/products/signals/eval/offline_config.py
@@ -4,6 +4,7 @@ from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite
GROUPING_SUITE = OfflineEvalSuite(
key="signals/grouping-e2e",
+ tags=("team:self-driving", "component:signals/grouping"),
project_id=2,
scorer_kinds={
"signal_safety_correct_classification": "boolean",
@@ -45,6 +46,7 @@ GROUPING_SUITE = OfflineEvalSuite(
SCOUT_SAFETY_SUITE = OfflineEvalSuite(
key="signals/scout-safety",
+ tags=("team:self-driving", "component:signals/scout-safety"),
project_id=2,
scorer_kinds={
"scout_safety_correct_classification": "boolean",
diff --git a/products/slack_app/evals/eval_followup_classifier.py b/products/slack_app/evals/eval_followup_classifier.py
index 7fdcdb78e84d..4526e841d3e4 100644
--- a/products/slack_app/evals/eval_followup_classifier.py
+++ b/products/slack_app/evals/eval_followup_classifier.py
@@ -26,6 +26,7 @@ from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval
from products.slack_app.backend.services.slack_messages import SlackThreadMessage
from products.slack_app.backend.temporal.activities import classifiers
+from products.slack_app.evals.offline_config import offline_suite
from products.slack_app.evals.scorers import FOLLOWUP_KEY, FollowupRoutingMatch, NoUnaskedWake
SUITE_KIND = SuiteKind.ONE_SHOT
@@ -239,6 +240,11 @@ async def eval_followup_classifier(ctx: EvalContext) -> None:
await OneShotPublicEval(
experiment_name="slack-app-followup-classifier",
+ offline_suite=offline_suite(
+ "slack_app/eval_followup_classifier::eval_followup_classifier",
+ metrics=["followup_routing", "no_unasked_wake"],
+ suite_kind=SUITE_KIND,
+ ),
cases=[*DIRECTED_CASES, *CHATTER_CASES],
scorers=[FollowupRoutingMatch(), NoUnaskedWake()],
task=task,
diff --git a/products/slack_app/evals/eval_model_classifier.py b/products/slack_app/evals/eval_model_classifier.py
index 49b128f92ed3..d56f185160a6 100644
--- a/products/slack_app/evals/eval_model_classifier.py
+++ b/products/slack_app/evals/eval_model_classifier.py
@@ -30,6 +30,7 @@ from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval
from products.slack_app.backend.services.model_catalogue import ModelChoice
from products.slack_app.backend.temporal.activities import classifiers
+from products.slack_app.evals.offline_config import offline_suite
from products.slack_app.evals.scorers import MODEL_OVERRIDE_KEY, ModelOverrideMatch, NoUnaskedOverride
SUITE_KIND = SuiteKind.ONE_SHOT
@@ -275,6 +276,11 @@ async def eval_model_classifier(ctx: EvalContext) -> None:
await OneShotPublicEval(
experiment_name="slack-app-model-classifier",
+ offline_suite=offline_suite(
+ "slack_app/eval_model_classifier::eval_model_classifier",
+ metrics=["model_override_match", "no_unasked_override"],
+ suite_kind=SUITE_KIND,
+ ),
cases=[*INSTRUCTION_CASES, *UNDERSPECIFIED_CASES, *SUBJECT_MATTER_CASES],
scorers=[ModelOverrideMatch(), NoUnaskedOverride()],
task=task,
diff --git a/products/slack_app/evals/eval_posthog_first.py b/products/slack_app/evals/eval_posthog_first.py
index 87a7216df952..2d9ea513269c 100644
--- a/products/slack_app/evals/eval_posthog_first.py
+++ b/products/slack_app/evals/eval_posthog_first.py
@@ -28,6 +28,7 @@ from products.posthog_ai.eval_harness.base import SandboxedPublicEval
from products.posthog_ai.eval_harness.config import SandboxedEvalCase
from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.scorers import RequiredToolCall
+from products.slack_app.evals.offline_config import offline_suite
from products.slack_app.evals.scorers import (
DOCS_SEARCH_TOOL_NAME,
AnsweredAboutPostHogProduct,
@@ -62,6 +63,10 @@ async def eval_posthog_first(ctx: EvalContext) -> None:
await SandboxedPublicEval(
experiment_name="sandboxed-slack-posthog-first-cli",
+ offline_suite=offline_suite(
+ "slack_app/eval_posthog_first::eval_posthog_first",
+ metrics=["docs_searched", "posthog_product_answer", "no_third_party_recommendation"],
+ ),
cases=cases,
scorers=[
RequiredToolCall({DOCS_SEARCH_TOOL_NAME}, name="docs_searched"),
diff --git a/products/slack_app/evals/eval_project_classifier.py b/products/slack_app/evals/eval_project_classifier.py
index cfc917583ea0..c806af1dce15 100644
--- a/products/slack_app/evals/eval_project_classifier.py
+++ b/products/slack_app/evals/eval_project_classifier.py
@@ -47,6 +47,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval
from products.slack_app.backend.temporal.activities import classifiers
+from products.slack_app.evals.offline_config import offline_suite
from products.slack_app.evals.scorers import PROJECT_ROUTE_KEY, NoUnaskedProjectSwitch, ProjectRouteMatch
SUITE_KIND = SuiteKind.ONE_SHOT
@@ -531,6 +532,11 @@ async def eval_project_classifier(ctx: EvalContext) -> None:
await OneShotPublicEval(
experiment_name="slack-app-project-classifier",
+ offline_suite=offline_suite(
+ "slack_app/eval_project_classifier::eval_project_classifier",
+ metrics=["project_route_match", "no_unasked_project_switch"],
+ suite_kind=SUITE_KIND,
+ ),
cases=[
*ROUTING_CASES,
*CROWDED_ROUTING_CASES,
diff --git a/products/slack_app/evals/eval_reply_suggestions.py b/products/slack_app/evals/eval_reply_suggestions.py
index eb80eca083e9..4c0ee6837a62 100644
--- a/products/slack_app/evals/eval_reply_suggestions.py
+++ b/products/slack_app/evals/eval_reply_suggestions.py
@@ -17,6 +17,7 @@ from products.posthog_ai.eval_harness.harness.context import EvalContext
from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval
from products.slack_app.backend.logic import reply_suggestions
+from products.slack_app.evals.offline_config import offline_suite
from products.slack_app.evals.scorers import REPLY_OFFER_KEY, NoUnaskedButtons, ReplyOfferMatch, SuggestedRepliesFit
SUITE_KIND = SuiteKind.ONE_SHOT
@@ -206,6 +207,11 @@ async def eval_reply_suggestions(ctx: EvalContext) -> None:
await OneShotPublicEval(
experiment_name="slack-app-reply-suggestions",
+ offline_suite=offline_suite(
+ "slack_app/eval_reply_suggestions::eval_reply_suggestions",
+ metrics=["reply_offer", "no_unasked_buttons", "suggested_replies_fit"],
+ suite_kind=SUITE_KIND,
+ ),
cases=[*WAITING_CASES, *BARE_CASES],
scorers=[ReplyOfferMatch(), NoUnaskedButtons(), SuggestedRepliesFit()],
task=task,
diff --git a/products/slack_app/evals/offline_config.py b/products/slack_app/evals/offline_config.py
new file mode 100644
index 000000000000..b86cc92224c9
--- /dev/null
+++ b/products/slack_app/evals/offline_config.py
@@ -0,0 +1,52 @@
+from collections.abc import Sequence
+from uuid import UUID
+
+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind
+from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer
+
+_SCORERS: dict[str, dict[str, OfflineScorer]] = {
+ "followup_classifier": {
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557c-9b63-776a-91f3-113ab57c025a")),
+ "followup_routing": OfflineScorer(kind="boolean", version=UUID("01a1557c-d1ca-7c68-96eb-5406fd8dde02")),
+ "no_unasked_wake": OfflineScorer(kind="boolean", version=UUID("01a1557c-dd52-78d0-a1f1-1854076c09ac")),
+ },
+ "model_classifier": {
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557c-e620-710d-82d2-de63990000f1")),
+ "model_override_match": OfflineScorer(kind="boolean", version=UUID("01a1557c-ee78-7412-9b90-2ecf80591aa0")),
+ "no_unasked_override": OfflineScorer(kind="boolean", version=UUID("01a1557c-fd45-7a93-8852-b64bcf13eeb8")),
+ },
+ "posthog_first": {
+ "docs_searched": OfflineScorer(kind="boolean", version=UUID("01a1557d-0f40-7999-9f55-d969442fe434")),
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-0682-75c3-bb88-5a620ba6ce5d")),
+ "no_third_party_recommendation": OfflineScorer(
+ kind="boolean", version=UUID("01a1557d-2542-7805-a791-6c3ac5f875b2")
+ ),
+ "posthog_product_answer": OfflineScorer(kind="boolean", version=UUID("01a1557d-1aa6-7fb4-ae3b-2fab4be4c44e")),
+ },
+ "project_classifier": {
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-2ef4-7ee8-9f53-ae73633d593d")),
+ "no_unasked_project_switch": OfflineScorer(
+ kind="boolean", version=UUID("01a1557d-415e-78ff-8bcb-1e6cc67b77ac")
+ ),
+ "project_route_match": OfflineScorer(kind="boolean", version=UUID("01a1557d-37df-756c-a5fb-73edde0b1561")),
+ },
+ "reply_suggestions": {
+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-49a7-7417-aa9f-4dbe00ec7b47")),
+ "no_unasked_buttons": OfflineScorer(kind="boolean", version=UUID("01a1557d-5b16-755e-ab88-b824e5103a97")),
+ "reply_offer": OfflineScorer(kind="boolean", version=UUID("01a1557d-5266-7853-a431-6213cfba1b5a")),
+ "suggested_replies_fit": OfflineScorer(kind="boolean", version=UUID("01a1557d-bcc0-7a96-8d22-4d6e91998d5d")),
+ },
+}
+
+
+def offline_suite(key: str, *, metrics: Sequence[str], suite_kind: SuiteKind = SuiteKind.SANDBOXED) -> OfflineEvalSuite:
+ domain = key.split("/", 1)[1].split("::", 1)[0].removeprefix("eval_")
+ scorers = _SCORERS[domain]
+ component = domain.replace("_", "-")
+ enrolled_metrics = ("exit_code_zero", *metrics) if suite_kind == SuiteKind.SANDBOXED else metrics
+ return OfflineEvalSuite.from_scorers(
+ key=key,
+ project_id=2,
+ scorers={metric: scorers[metric] for metric in enrolled_metrics},
+ tags=("team:surfaces", f"component:slack-app/{component}"),
+ )
diff --git a/pyproject.toml b/pyproject.toml
index 80340aa70da3..65749b80b297 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -105,7 +105,7 @@ dependencies = [
"paramiko~=3.5.0",
"pillow==12.3.0",
"protobuf~=6.31.1",
- "posthoganalytics==7.67.1",
+ "posthoganalytics==7.73.0",
"polars==1.37.1",
"psycopg2-binary==2.9.11",
"psycopg[binary]==3.2.10",
diff --git a/uv.lock b/uv.lock
index a1acc6bad859..390c4454d298 100644
--- a/uv.lock
+++ b/uv.lock
@@ -6252,7 +6252,7 @@ requires-dist = [
{ name = "playwright", specifier = "~=1.60.0" },
{ name = "polars", specifier = "==1.37.1" },
{ name = "posthog-hogland", specifier = "==0.4.1" },
- { name = "posthoganalytics", specifier = "==7.67.1" },
+ { name = "posthoganalytics", specifier = "==7.73.0" },
{ name = "protobuf", specifier = "~=6.31.1" },
{ name = "psycopg", extras = ["binary"], specifier = "==3.2.10" },
{ name = "psycopg2-binary", specifier = "==2.9.11" },
@@ -6419,7 +6419,7 @@ wheels = [
[[package]]
name = "posthoganalytics"
-version = "7.67.1"
+version = "7.73.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "backoff" },
@@ -6427,9 +6427,9 @@ dependencies = [
{ name = "requests" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/76/ff/1b18c85d904340607c70284c9d867df588c22c18d615e2ef67741126f64c/posthoganalytics-7.67.1.tar.gz", hash = "sha256:c861483738eb6da05eeaac3540f2d9f6622849e7a8bd3d05a4a0221a32abc120", size = 752005, upload-time = "2026-10-12T14:17:21.915Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/f4/8e/30cf3b4ce18f2224799e7fe38d147e4d8b6d3a54ef9998234100253180e6/posthoganalytics-7.73.0.tar.gz", hash = "sha256:6ea816666f116f132a4ae4e5fb3e36240e4f7de7c2a8afed12d2a258c227912a", size = 764745, upload-time = "2026-10-19T19:21:34.253Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/3f/d4/48706a5d0c2304f299a4564b86ce220eae3de6ab087a2bca9669644b8e3a/posthoganalytics-7.67.1-py3-none-any.whl", hash = "sha256:29a721c2e9ca383ed35deb736b390744e68ad9d2bf69b4bf08ebbc4487db97a4", size = 884481, upload-time = "2026-10-12T14:17:19.659Z" },
+ { url = "https://files.pythonhosted.org/packages/66/01/d336d90c2b615b3db08864f004841d60bee2ebd4930cd1c2d761b30bc32a/posthoganalytics-7.73.0-py3-none-any.whl", hash = "sha256:a1ab79a6abf9d8d655d9e7e37fe8fe3a10781415037cf199c516a6013cc9a905", size = 899662, upload-time = "2026-10-19T19:21:32.037Z" },
]
[[package]]