:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(aio): stop reporting offline evals to braintrust (#123522)
Radu Raicea8d3a666ac55d
diff --git a/.agents/skills/writing-evals/SKILL.md b/.agents/skills/writing-evals/SKILL.mdindex f27747bb113f..bb0640f3caa3 100644--- a/.agents/skills/writing-evals/SKILL.md+++ b/.agents/skills/writing-evals/SKILL.md@@ -32,13 +32,13 @@ New product-owned evals belong under `products/<product>/evals/`. Both root setsNo `evals/` tree needs a pytest-collection exclusion: pytest's default `python_files` matches `test_*.py`, so `eval_*.py` files there are never collected as tests. Do not add a `conftest.py` under an `evals/` tree.```python-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextasync def eval_my_thing(ctx: EvalContext) -> None:- await SandboxedPrivateEval(+ await SandboxedEval(experiment_name="sandboxed-my-thing-cli",cases=[SandboxedEvalCase(name="my_case", prompt="...")],scorers=[],@@ -46,22 +46,22 @@ async def eval_my_thing(ctx: EvalContext) -> None:)```-- One suite = one Braintrust experiment. Bundle related cases into one suite; split only when the scorecard would be heterogeneous (self-skipping scorers stretch a shared scorer list across mildly different cases, but a scorecard where half the scorers skip half the cases is a sign to split).-- `experiment_name` is the Braintrust history key — renaming it resets cross-run comparison. Existing sandboxed suites end in `-cli` because the MCP server serves the `cli` surface.-- `SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust, plus PostHog when `offline_suite` is configured. `SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service by default; local logs are still written. Set `send_to_posthog=True` with `offline_suite` to publish only to PostHog, or `send_to_posthog=False` to disable that destination independently. These options also apply to one-shot suites. Ordinary SDK and trace clients still honor `OPT_OUT_CAPTURE`. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md).+- One suite = one experiment. Bundle related cases into one suite; split only when the scorecard would be heterogeneous (self-skipping scorers stretch a shared scorer list across mildly different cases, but a scorecard where half the scorers skip half the cases is a sign to split).+- `experiment_name` identifies local and PostHog runs — keep it stable for cross-run comparison. Existing sandboxed suites end in `-cli` because the MCP server serves the `cli` surface.+- `SandboxedEval` and `OneShotEval` always run Braintrust locally without uploading to it. With `offline_suite` and a publishing credential, results publish to PostHog by default. Set `send_to_posthog=False` for local-only results; local logs are always written. Ordinary SDK and trace clients still honor `OPT_OUT_CAPTURE`. See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md).- To enroll a suite in PostHog reporting, register its scorer kinds and immutable version UUIDs in the owning `offline_config.py` catalog, then pass `offline_suite=offline_suite("<domain>/<module>::<function>", metrics=[...])` to the runner. Use the PostHog AI catalog for built-in suites and the Slack app catalog for Slack suites. List every explicit scorer's `_name()` in `metrics`; the catalog adds `exit_code_zero` only for sandboxed suites. For one-shot suites, pass `suite_kind=SUITE_KIND`. The enrollment test discovers catalog-using modules and checks every suite function against its scorer list.- Uploading and tagging require `POSTHOG_OFFLINE_EVAL_API_KEY` to be a personal key with `offline_evaluation_ingestion:write` and `evaluation:write`. Ingestion-only personal or project secret keys can upload without tags. Upload success is recorded before the separate, bounded tag update, so a tagging failure or interruption keeps the results link.## One-shot suitesA module-level `SUITE_KIND` declares how a module's suites execute; absent means sandboxed, so existing suites need nothing.-`SuiteKind.ONE_SHOT` suites skip the sandbox entirely: the harness boots only the test DB, personhog, and demo data for them, and preflight requires only `BRAINTRUST_API_KEY` + `LLM_GATEWAY_ANTHROPIC_API_KEY`.+`SuiteKind.ONE_SHOT` suites skip the sandbox entirely: the harness boots only the test DB, personhog, and demo data for them, and preflight requires only `LLM_GATEWAY_ANTHROPIC_API_KEY`. Autoevals judges use `OPENAI_API_KEY` through a direct OpenAI client; they do not use Braintrust's model proxy.```pythonfrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPrivateEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalSUITE_KIND = SuiteKind.ONE_SHOT@@ -70,7 +70,7 @@ async def eval_my_generation(ctx: EvalContext) -> None:async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict:return {"answer": ...} # one model invocation, JSON-serializable- await OneShotPrivateEval(+ await OneShotEval(experiment_name="my-generation",cases=[BaseEvalCase(name="my_case", prompt="...")],scorers=[...],@@ -165,7 +165,7 @@ hogli evals eval_skill_distribution --skill-delivery exec # exec-distributed sk- Team cloning and case seeding use one setup slot on ordinary local machines and four when `CODER` or `CI` is set.- `--trials N` repeats every case for variance on stochastic behavior; `--fail-under <fraction>` gates the run's mean score.- `--skill-delivery bundled` is the default native-skill path. `--skill-delivery exec` enables the MCP skill prompt and clears all native skill directories; compare the modes in separate runs because MCP configuration is process-wide.-- `--agent-runtime codex` runs the OpenAI Codex harness (default model `gpt-6.1-sol`) instead of Claude; it requires `LLM_GATEWAY_OPENAI_API_KEY`. The runtime/model land in the Braintrust experiment metadata, so compare scores within one runtime.+- `--agent-runtime codex` runs the OpenAI Codex harness (default model `gpt-6.1-sol`) instead of Claude; it requires `LLM_GATEWAY_OPENAI_API_KEY`. The runtime/model land in the local and PostHog experiment metadata, so compare scores within one runtime.- Every real eval invocation mirrors its complete stdout and stderr to `products/posthog_ai/eval_harness/logs/harness/<timestamp>_<id>.log`. The ending label identifies it as the full run transcript, the last terminal line is its absolute path, and `logs/harness/latest.log` points to the newest transcript. `--list` and argument errors do not create one.- The plain-text summary is stable for people and agents: only the overall run says `PASS` or `FAIL`; completed suites, experiments, and cases say `DONE`; experiment blocks label scores, URLs, and agent-log directories.- `EXPORT_EVAL_RESULTS=1` additionally appends one structured Braintrust summary per experiment to `eval_results.jsonl`. It does not replace the always-on transcript.diff --git a/.agents/skills/writing-evals/references/running-evals.md b/.agents/skills/writing-evals/references/running-evals.mdindex e337614ed4ad..a511dba35d1e 100644--- a/.agents/skills/writing-evals/references/running-evals.md+++ b/.agents/skills/writing-evals/references/running-evals.md@@ -26,10 +26,12 @@ The required variables depend on the selected suite and runtime:| Selection | Required variables || ------------------- | --------------------------------------------------------------- |-| Every suite | `BRAINTRUST_API_KEY`, `LLM_GATEWAY_ANTHROPIC_API_KEY` |+| Every suite | `LLM_GATEWAY_ANTHROPIC_API_KEY` || Any sandboxed suite | `SANDBOX_JWT_PRIVATE_KEY` in addition to the variables above || Codex runtime | `LLM_GATEWAY_OPENAI_API_KEY` in addition to the variables above |+Autoevals judges additionally need `OPENAI_API_KEY`. They use a direct OpenAI client, honoring `OPENAI_BASE_URL` when explicitly configured. Braintrust is used locally and needs no API key.+The local-development sandbox JWT key ships in `.env.example` and is normally copied into `.env` as part of repository setup.Use that key only for local development, never for production.Preflight reports every missing variable before it starts the database or any services.diff --git a/.codex/with-flox b/.codex/with-floxindex b7064e92bbfc..afb97703c2ed 100755--- a/.codex/with-flox+++ b/.codex/with-flox@@ -257,7 +257,6 @@ if [[ "$prepare" -eq 0 ]]; then# Read runtime credentials at invocation time, never from the toolchain cache.for name in \ANTHROPIC_API_KEY \- BRAINTRUST_API_KEY \GH_TOKEN \LLM_GATEWAY_ANTHROPIC_API_KEY \LLM_GATEWAY_OPENAI_API_KEY \diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.mdindex 7f0fb52eeea6..9f808559ebb1 100644--- a/docs/internal/ai-offline-evaluation-reporting.md+++ b/docs/internal/ai-offline-evaluation-reporting.md@@ -1,6 +1,6 @@# Offline evaluation result reporting-The evaluation harness in `products/posthog_ai/eval_harness/` reports public-suite scorer results to Braintrust.+The evaluation harness in `products/posthog_ai/eval_harness/` uses Braintrust locally to execute and score cases. It never uploads results or traces to Braintrust.Suites under `products/posthog_ai/evals/` and the product-owned `products/*/evals/` trees also publish those results to PostHog's offline experiment API when a publishing credential is configured.The Signals pytest suites reuse the same publisher for their completed results.Agent and judge trace capture remains available.@@ -9,17 +9,12 @@ Use `hogli evals` to run PostHog AI 2.0 suites. See the [eval harness guide](../## Capture settings-`no_send_logs` controls Braintrust uploads and, by default, PostHog result uploads.-`SandboxedPublicEval` sets `no_send_logs=False` and uploads to Braintrust, plus PostHog when the suite has offline publishing configured.-`SandboxedPrivateEval` sets `no_send_logs=True` and uploads to neither service by default; local logs are still written.-Set `send_to_posthog=True` alongside `offline_suite` to publish results to PostHog while keeping Braintrust uploads disabled.-Set `send_to_posthog=False` to disable PostHog result uploads independently. The same options apply to one-shot suites.--Suites that disable Braintrust uploads declare `NO_SEND_LOGS = True` at module level and pass `no_send_logs=NO_SEND_LOGS` to their runner.-Discovery uses this declaration to validate credentials before starting infrastructure; the enrollment test checks that it matches the runner.-`BRAINTRUST_API_KEY` is required for reporting only when at least one selected suite enables Braintrust uploads.-PostHog-only suites still use the Braintrust library locally, without authenticating to its experiment service.-Scorer model credentials remain separate: using Braintrust's model proxy still needs its credential, while other providers use their own keys.+`SandboxedEval` and `OneShotEval` publish completed results to PostHog when `offline_suite` and a publishing credential are configured.+`send_to_posthog` defaults to `True`; set it to `False` for local-only results. Local logs are always written.+The Braintrust engine always runs with `no_send_logs=True` internally. Suites cannot enable Braintrust uploads, and no Braintrust credential is required.+Autoevals judges use an explicit OpenAI client, including nested judges and runs without PostHog trace capture.+They require `OPENAI_API_KEY` and honor `OPENAI_BASE_URL` when set; they do not fall back to Braintrust's model proxy or API key.+Scorer models and other provider credentials are unchanged.Offline API uploads run through the Python SDK independently of `TEST` and `OPT_OUT_CAPTURE`.The shared trace client retains its existing `TEST` and `OPT_OUT_CAPTURE` guards for `$ai_generation`, `$ai_span`, and `$ai_trace` events.@@ -42,23 +37,22 @@ before running agents, retains each model's results, and reports case or scorerThis local comparison does not change the suite's upload or capture settings.Trial metadata and local summaries also record phase durations in seconds.The scoring phase includes time waiting for the experiment results, rather than scorer execution alone.-A failed final metadata upload produces a warning and keeps the completed results available locally.A scorer failure marks the case as an error. Statistics still include scores from other successful scorers.## Configured offline experiment suites-Shared-harness suites are enrolled in the offline experiment publisher, including both sandboxed and one-shot suites, except for `logs/eval_agent_observability::eval_agent_observability`, which currently reports only to Braintrust.-The experiment-skill, Feature Flags, Dashboards, turn-suggestion, Business Knowledge retrieval and Signals first-report suites retain `no_send_logs=True` and explicitly set `send_to_posthog=True`, publishing results only to PostHog.+Shared-harness suites are enrolled in the offline experiment publisher, including sandboxed and one-shot suites and `logs/eval_agent_observability::eval_agent_observability`.+The Logs catalog pins numeric `answer_mentions` and `call_budget` scorers and the boolean `exit_code_zero` scorer.+Braintrust remains the local execution engine for every shared-harness suite.The [first-report scout suite](../../products/posthog_ai/eval_harness/README.md#scout-evals) records the persisted report and structured scout outcome alongside context, progress, and first-visible-report timings. Its seeded regression, matched control, and sparse-data cases separate useful report delivery from unsupported findings and justified abstention. Compare quality, the five-minute qualified-report rate, and planted-issue recall separately; sparse cases are excluded from report-quality and report-latency scores because they should not produce a report.-Public suites retain their Braintrust uploads during enrollment. Braintrust also remains the execution engine for every shared-harness suite, including those whose results publish only to PostHog.After scoring, the harness prepares the completed results in memory and queues delivery through the Python SDK's `AsyncOfflineEvaluations` client. It does not execute another agent run or judge call.-Without an explicit `send_to_posthog` override, `no_send_logs=True` suppresses this upload too.+Set `send_to_posthog=False` to suppress this upload.The [PostHog AI catalog](../../products/posthog_ai/evals/offline_config.py) and each product's `evals/offline_config.py` pin destination project `2`, scorer kinds, immutable version UUIDs, and component tags.Suites select their own metrics from the catalog; sandboxed suites also include the harness-provided `exit_code_zero` metric.The SQL suite retains its seven existing scorer versions.-The discovery-wide enrollment test checks every enrolled suite's publishing flag, stable key, scorer coverage, pinned UUIDs and ownership tags without executing agents or judges. It also checks that the Logs agent-observability suite retains its Braintrust-only reporting configuration.+The discovery-wide enrollment test checks every enrolled suite's publishing flag, stable key, scorer coverage, pinned UUIDs and ownership tags without executing agents or judges.Provide the API key through the environment before running the suite:| Variable | Purpose |@@ -120,7 +114,7 @@ hogli evals sql/eval_sql::eval_sqlWithout an API key, each enrolled suite reports that offline uploads are disabled.Keys containing whitespace are rejected before evaluation starts; remove any spaces, tabs, or pasted trailing newlines.-Invalid configuration produces a warning and skips this destination; Braintrust and local reporting continue.+Invalid configuration produces a warning and skips this destination; local reporting continues.An upload failure also warns without changing the evaluation's score gate.The final summary shows a PostHog results link only after a successful offline upload; otherwise it shows the upload status.@@ -149,7 +143,7 @@ The file is gitignored by the `.env*` rule. Local configuration is accepted onlyAn unreadable, invalid, or incomplete file disables the upload with an error; it never falls back to the production catalog.Run through `hogli evals` to load `.env.local`; existing shell variables take precedence.-For PostHog-only testing, select suites declaring `NO_SEND_LOGS = True`, such as Feature Flags or Dashboards.+Every suite runs without Braintrust reporting credentials. PostHog publishing depends only on its own configuration.Set `OPT_OUT_CAPTURE=1` to suppress ordinary SDK/trace capture while retaining explicit offline-result uploads.### Scorer and experiment tags@@ -205,7 +199,7 @@ MCP saved-report grading, semantic-layer canary commands and ReviewHog campaigns### CI configuration-CI jobs running enrolled suites need `POSTHOG_OFFLINE_EVAL_API_KEY` and `BRAINTRUST_API_KEY` in the environment.+CI jobs running enrolled suites need `POSTHOG_OFFLINE_EVAL_API_KEY` in the environment.Map the dedicated `ANTHROPIC_OFFLINE_EVALS_API_KEY` and `OPENAI_OFFLINE_EVALS_API_KEY` Actions secrets to `LLM_GATEWAY_ANTHROPIC_API_KEY` and `OPENAI_API_KEY` for the harness and judges.Publishing failures remain warnings, so check the final harness upload status even when a CI job is green.The harness sets the SDK's `run_source` to `ci` when `CI` is set, including manual Actions runs, and to `local` otherwise.@@ -216,7 +210,7 @@ Each case and trial becomes a separate item, with one result per configured scorBoolean scores convert `0` and `1` to `false` and `true`; numeric scores retain their values.Both `false` and numeric zero have status `ok`: this status means scoring produced a value, not that the agent passed.A `None` score becomes `skipped`; a task error or missing scorer result becomes `error`, with no score value.-LLM judge exceptions publish an `error` outcome with code `judge_error`; the existing zero fallback remains in Braintrust and local aggregates.+LLM judge exceptions publish an `error` outcome with code `judge_error`; the existing zero fallback remains in local aggregates.A failed scorer does not replace successful results from other scorers on the same case.Scorer metadata and `reason` or `rationale` explanations are retained for each trial, including skip reasons and judge errors.The scorer recorder observes return values and exceptions only when a suite has valid offline publishing configuration and allows uploads. Other runs pass scorers directly to Braintrust.@@ -225,7 +219,7 @@ Application and evaluator trace links are included when capture is enabled and tApplication and dataset revisions use the commit captured at run start.Items include available input, output, expected output, and case metadata.-The publisher excludes `raw_log` from the new API payload; the existing local logs and Braintrust output retain it.+The publisher excludes `raw_log` from the new API payload; the existing local logs retain it.The SDK validates optional fields and the combined payload as fields are added. An invalid or oversized field is omitted as a whole, with its name recorded in `metadata.omitted_fields`.Item payloads prioritize input, output, expected output, then metadata. Output fields are not pruned individually, and accepted fields are not removed to make room for omission diagnostics.Invalid optional case keys, dataset references, or trace identifiers are omitted individually, preserving the other cases and scores in the prepared upload.@@ -242,7 +236,7 @@ After an API error following successful experiment creation, the publisher makesPausing or cancelling delivery does not mark the experiment failed, so teardown pauses can resume the same experiment. Timeouts, interruptions, and failed cleanup can leave an experiment in the uploading state.Timeout, cancellation, a full queue, or a rejected request produces a warning without changing the score gate.Failed or interrupted uploads are not saved for later retry. To try publishing again, correct the configuration or connectivity problem and rerun the suite.-Existing local logs, enabled case exports, and Braintrust results remain available.+Existing local logs and enabled case exports remain available.### Enrolling other suites and richer results@@ -270,7 +264,7 @@ result = Score()```-The numeric `score` still controls existing Braintrust and local aggregates; `result` describes the offline outcome. Configure this example's scorer kind as `categorical`.+The numeric `score` still controls existing local aggregates; `result` describes the offline outcome. Configure this example's scorer kind as `categorical`.Scorer provisioning stays separate from uploading results, so publishing requires only the ingestion scope and does not create or modify scorer definitions.## Signals pytest suitesdiff --git a/ee/settings.py b/ee/settings.pyindex 23f9f12f198e..29d49dba60df 100644--- a/ee/settings.py+++ b/ee/settings.py@@ -109,7 +109,6 @@ REPLAY_VISION_GEMINI_API_KEY = get_from_env("REPLAY_VISION_GEMINI_API_KEY", "")PPLX_API_KEY = get_from_env("PPLX_API_KEY", "")AZURE_INFERENCE_ENDPOINT = get_from_env("AZURE_INFERENCE_ENDPOINT", "", type_cast=str)AZURE_INFERENCE_CREDENTIAL = get_from_env("AZURE_INFERENCE_CREDENTIAL", "", type_cast=str)-BRAINTRUST_API_KEY = get_from_env("BRAINTRUST_API_KEY", "")LLMA_EVAL_OPENAI_API_KEY = get_from_env("LLMA_EVAL_OPENAI_API_KEY", "")LLMA_EVAL_OPENAI_BASE_URL = get_from_env("LLMA_EVAL_OPENAI_BASE_URL", "")diff --git a/products/business_knowledge/evals/eval_retrieval.py b/products/business_knowledge/evals/eval_retrieval.pyindex 7ead451fa2d4..982c55d09574 100644--- a/products/business_knowledge/evals/eval_retrieval.py+++ b/products/business_knowledge/evals/eval_retrieval.py@@ -12,14 +12,12 @@ from products.business_knowledge.evals.offline_config import offline_suitefrom products.conversations.evals.seeders import provision_eval_team, teardown_eval_teamfrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPrivateEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.posthog_ai.eval_harness.scorers.contract import Score, Scorerif TYPE_CHECKING:from products.posthog_ai.eval_harness.harness.context import EvalContext-NO_SEND_LOGS = True-SUITE_KIND = SuiteKind.ONE_SHOT@@ -87,8 +85,7 @@ async def eval_retrieval(ctx: EvalContext) -> None:async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict[str, object]:return await asyncio.to_thread(search_case, case)- await OneShotPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await OneShotEval(experiment_name="business-knowledge-retrieval",cases=CASES,task=task,diff --git a/products/conversations/evals/eval_support_reply.py b/products/conversations/evals/eval_support_reply.pyindex f615787ce198..11052a5e8cbd 100644--- a/products/conversations/evals/eval_support_reply.py+++ b/products/conversations/evals/eval_support_reply.py@@ -27,7 +27,7 @@ from products.conversations.evals.seeders import provision_eval_team, seed_case,from products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalSUITE_KIND = SuiteKind.ONE_SHOT@@ -60,7 +60,7 @@ async def eval_support_reply(ctx: EvalContext) -> None:finally:await asyncio.to_thread(lambda: teardown_eval_team(eval_team=eval_team))- await OneShotPublicEval(+ await OneShotEval(experiment_name="support-reply-pipeline",offline_suite=offline_suite("conversations/eval_support_reply::eval_support_reply",diff --git a/products/dashboards/evals/eval_dashboard_contents.py b/products/dashboards/evals/eval_dashboard_contents.pyindex 2c3d34e87c5d..d51c1b0283ca 100644--- a/products/dashboards/evals/eval_dashboard_contents.py+++ b/products/dashboards/evals/eval_dashboard_contents.py@@ -8,15 +8,13 @@ from products.dashboards.backend.models.dashboard import Dashboardfrom products.dashboards.backend.models.dashboard_tile import DashboardTilefrom products.dashboards.evals.offline_config import offline_suitefrom products.dashboards.evals.scorers import SavedDashboardContents-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.product_analytics.backend.facade.api import get_or_create_saved_insightfrom products.product_analytics.backend.facade.models import Insightfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext-NO_SEND_LOGS = True-DASHBOARD_NAME = "File activity overview"SIGNUPS_NAME = "Saved daily signups"UPLOADS_NAME = "Saved daily file uploads"@@ -69,8 +67,7 @@ def seed_existing_dashboard(context: CustomPromptSandboxContext) -> dict[str, Anasync def eval_dashboard_contents(ctx: EvalContext) -> None:- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-dashboards-saved-contents-cli",offline_suite=offline_suite("dashboards/eval_dashboard_contents::eval_dashboard_contents",diff --git a/products/dashboards/evals/eval_dashboard_sql_date_range.py b/products/dashboards/evals/eval_dashboard_sql_date_range.pyindex 0e7d661c87b2..7ad1b9a470c6 100644--- a/products/dashboards/evals/eval_dashboard_sql_date_range.py+++ b/products/dashboards/evals/eval_dashboard_sql_date_range.py@@ -5,13 +5,11 @@ from typing import Anyfrom products.dashboards.backend.models.dashboard import Dashboardfrom products.dashboards.evals.offline_config import offline_suitefrom products.dashboards.evals.scorers import SQLTilesFollowDashboardDateRange-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext-NO_SEND_LOGS = True-SIGNUPS_DASHBOARD_NAME = "Signups and uploads (SQL)"BILLING_DASHBOARD_NAME = "Billing and downloads (SQL)"@@ -24,8 +22,7 @@ def seed_team(context: CustomPromptSandboxContext) -> dict[str, Any]:async def eval_dashboard_sql_date_range(ctx: EvalContext) -> None:- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-dashboards-sql-date-range-cli",offline_suite=offline_suite("dashboards/eval_dashboard_sql_date_range::eval_dashboard_sql_date_range",diff --git a/products/dashboards/evals/eval_dashboard_widgets.py b/products/dashboards/evals/eval_dashboard_widgets.pyindex 45cf2a69aa55..82bea9af96ca 100644--- a/products/dashboards/evals/eval_dashboard_widgets.py+++ b/products/dashboards/evals/eval_dashboard_widgets.py@@ -8,13 +8,11 @@ from products.dashboards.backend.models.dashboard_widget import DashboardWidgetfrom products.dashboards.backend.widget_specs.registry import validate_widget_configfrom products.dashboards.evals.offline_config import offline_suitefrom products.dashboards.evals.scorers import SavedWidgetState, WidgetFirstWriteValid-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext-NO_SEND_LOGS = True-DASHBOARD_NAME = "Release health"ERROR_TRACKING_WIDGET = "error_tracking_list"LOGS_WIDGET = "logs_list"@@ -41,8 +39,7 @@ def seed_error_tracking_widget(context: CustomPromptSandboxContext) -> dict[str,async def eval_dashboard_widgets(ctx: EvalContext) -> None:- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-dashboards-widgets-cli",offline_suite=offline_suite("dashboards/eval_dashboard_widgets::eval_dashboard_widgets",diff --git a/products/data_catalog/evals/eval_governed_metrics.py b/products/data_catalog/evals/eval_governed_metrics.pyindex 64d69540c222..93522fd04c78 100644--- a/products/data_catalog/evals/eval_governed_metrics.py+++ b/products/data_catalog/evals/eval_governed_metrics.py@@ -100,7 +100,7 @@ from products.data_catalog.evals.seeders import (seed_web_sessions_daily_metric,seed_website_404_hits_daily_metric,)-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -690,7 +690,7 @@ async def eval_governed_metrics(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-governed-metrics-cli",offline_suite=offline_suite("data_catalog/eval_governed_metrics::eval_governed_metrics",diff --git a/products/data_catalog/evals/eval_metric_authoring.py b/products/data_catalog/evals/eval_metric_authoring.pyindex 31692facb282..9dcf8b84b39e 100644--- a/products/data_catalog/evals/eval_metric_authoring.py+++ b/products/data_catalog/evals/eval_metric_authoring.py@@ -18,7 +18,7 @@ from __future__ import annotationsfrom products.data_catalog.evals.constants import METRIC_CREATE_TOOLfrom products.data_catalog.evals.offline_config import offline_suitefrom products.data_catalog.evals.scorers import MetricDescriptionConcise, MetricDescriptionQuality-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall@@ -53,7 +53,7 @@ async def eval_metric_authoring(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-metric-authoring-cli",offline_suite=offline_suite("data_catalog/eval_metric_authoring::eval_metric_authoring",diff --git a/products/data_catalog/evals/eval_semantic_discovery.py b/products/data_catalog/evals/eval_semantic_discovery.pyindex 65ef00f6e6ae..5543ca930126 100644--- a/products/data_catalog/evals/eval_semantic_discovery.py+++ b/products/data_catalog/evals/eval_semantic_discovery.py@@ -33,7 +33,7 @@ from products.data_catalog.evals.seeders import (seed_deprecation_candidate_sources,seed_instruction_like_relationship_context,)-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContext@@ -117,7 +117,7 @@ async def eval_semantic_discovery(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-semantic-discovery-cli",offline_suite=offline_suite("data_catalog/eval_semantic_discovery::eval_semantic_discovery",diff --git a/products/feature_flags/evals/eval_cleanup_stale_flags.py b/products/feature_flags/evals/eval_cleanup_stale_flags.pyindex 19e7c8ae7cc1..c6aa69f4ebb1 100644--- a/products/feature_flags/evals/eval_cleanup_stale_flags.py+++ b/products/feature_flags/evals/eval_cleanup_stale_flags.py@@ -47,7 +47,7 @@ pull request that adds the first fixture case registers it.Most scorers are deterministic; the three wording cases use one LLM judge each(``FinalMessageJudge``), which costs a model call per case but is what the "does the-message avoid X" question needs. ``SandboxedPrivateEval`` runs without a Braintrust+message avoid X" question needs. ``SandboxedEval`` runs without a Braintrustkey.**Claude runtime only.** The edit-direction scorer matches Claude's named file tools,@@ -83,14 +83,12 @@ from products.feature_flags.evals.seeders import (seed_stale_full_rollout_flag,seed_stale_partial_rollout_flag,)-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCallfrom products.posthog_ai.evals.retrieval.scorers import SkillTriggered-NO_SEND_LOGS = True-SKILL_NAME = "cleaning-up-stale-feature-flags"TRIGGER_SCORER_NAME = "cleanup_skill_triggered"EDIT_SCORER_NAME = "code_edit_direction"@@ -228,8 +226,7 @@ async def eval_cleanup_stale_flags(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-feature-flags-cleanup-stale-cli",offline_suite=offline_suite("feature_flags/eval_cleanup_stale_flags::eval_cleanup_stale_flags",diff --git a/products/feature_flags/evals/eval_flag_answers.py b/products/feature_flags/evals/eval_flag_answers.pyindex f0ad24f6518d..db86fbf3a3a2 100644--- a/products/feature_flags/evals/eval_flag_answers.py+++ b/products/feature_flags/evals/eval_flag_answers.py@@ -44,12 +44,10 @@ from products.feature_flags.evals.seeders import (seed_require_flag_tags,seed_stale_flag,)-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContext-NO_SEND_LOGS = True-_TOOL_DEFINITIONS = Path(__file__).parents[3] / "services/mcp/schema/generated-tool-definitions.json"FLAG_TOOL_CATEGORY = "Feature flags"@@ -113,8 +111,7 @@ async def eval_flag_answers(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-feature-flags-answers-cli",offline_suite=offline_suite("feature_flags/eval_flag_answers::eval_flag_answers",diff --git a/products/feature_flags/evals/eval_flag_creation.py b/products/feature_flags/evals/eval_flag_creation.pyindex cd0095cf6d81..c0d0b8208a85 100644--- a/products/feature_flags/evals/eval_flag_creation.py+++ b/products/feature_flags/evals/eval_flag_creation.py@@ -5,13 +5,11 @@ from typing import Anyfrom products.feature_flags.backend.models.feature_flag import FeatureFlagfrom products.feature_flags.evals.creation_scorers import CreatedFlagConfigurationfrom products.feature_flags.evals.offline_config import offline_suite-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext-NO_SEND_LOGS = True-FLAG_KEY = "bulk-file-export-preview"@@ -22,8 +20,7 @@ def seed_flag_creation(context: CustomPromptSandboxContext) -> dict[str, Any]:async def eval_flag_creation(ctx: EvalContext) -> None:- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-feature-flags-creation-cli",offline_suite=offline_suite("feature_flags/eval_flag_creation::eval_flag_creation",diff --git a/products/feature_flags/evals/eval_flag_tool_selection.py b/products/feature_flags/evals/eval_flag_tool_selection.pyindex 098cc7f985b8..702c7f5fa137 100644--- a/products/feature_flags/evals/eval_flag_tool_selection.py+++ b/products/feature_flags/evals/eval_flag_tool_selection.py@@ -54,13 +54,11 @@ from products.feature_flags.evals.seeders import (seed_metadata_flag,seed_rollout_flag,)-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall-NO_SEND_LOGS = True-# The rename case asks for these two values, so the prompt and the scorer read the same# constants: a prompt that changed without the expectation would grade the old request.METADATA_RENAMED_KEY = "file-preview-tiles"@@ -142,8 +140,7 @@ async def eval_flag_tool_selection(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-feature-flags-tool-selection-cli",offline_suite=offline_suite("feature_flags/eval_flag_tool_selection::eval_flag_tool_selection",diff --git a/products/feature_flags/evals/eval_instrument_flags.py b/products/feature_flags/evals/eval_instrument_flags.pyindex 97478918dd43..2c7df4017277 100644--- a/products/feature_flags/evals/eval_instrument_flags.py+++ b/products/feature_flags/evals/eval_instrument_flags.py@@ -34,7 +34,7 @@ score as a trigger regression here. The trigger wording is what is graded.The pair is a baseline, not a quality bar: it measures whether the descriptionfires on the right request, not whether the workflow the skill teaches is good.-Both cases are ``SandboxedPrivateEval`` so they run without a Braintrust key.+Both cases are ``SandboxedEval`` so they run without a Braintrust key.**Claude runtime only.** Under ``--agent-runtime codex`` the guard refuses thecases rather than scoring them. See ``_CODEX_UNSUPPORTED`` for the two reasons and@@ -55,15 +55,13 @@ from django.conf import settingsfrom products.feature_flags.evals.offline_config import offline_suitefrom products.feature_flags.evals.scorers import FILE_EDIT_TOOLS-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCallfrom products.posthog_ai.evals.retrieval.scorers import SkillTriggeredfrom products.tasks.backend.facade.agents import CustomPromptSandboxContext-NO_SEND_LOGS = True-SKILL_NAME = "instrument-feature-flags"SCORER_NAME = "instrument_flags_skill_triggered"@@ -146,8 +144,7 @@ async def eval_instrument_flags(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-feature-flags-instrument-trigger-cli",offline_suite=offline_suite("feature_flags/eval_instrument_flags::eval_instrument_flags",diff --git a/products/logs/evals/eval_agent_observability.py b/products/logs/evals/eval_agent_observability.pyindex 7d35f5624e9f..d85e88011d0d 100644--- a/products/logs/evals/eval_agent_observability.py+++ b/products/logs/evals/eval_agent_observability.py@@ -8,7 +8,8 @@ from typing import Anyfrom posthog.clickhouse.client import sync_execute-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.logs.evals.offline_config import offline_suite+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.log_parser import LogParser@@ -172,8 +173,11 @@ INCIDENT_CASES = [async def eval_agent_observability(ctx: EvalContext) -> None:- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-logs-agent-observability-cli",+ offline_suite=offline_suite(+ "logs/eval_agent_observability::eval_agent_observability", metrics=["answer_mentions", "call_budget"]+ ),cases=[SandboxedEvalCase(name=name,diff --git a/products/logs/evals/offline_config.py b/products/logs/evals/offline_config.pynew file mode 100644index 000000000000..7b60e23b1ab3--- /dev/null+++ b/products/logs/evals/offline_config.py@@ -0,0 +1,19 @@+from collections.abc import Sequence+from uuid import UUID++from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++_SCORERS: dict[str, OfflineScorer] = {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a178ac-ded9-7c35-a805-7fa5d89181c3")),+ "answer_mentions": OfflineScorer(kind="numeric", version=UUID("01a178ac-a3e7-767b-845b-e0cd688962b7")),+ "call_budget": OfflineScorer(kind="numeric", version=UUID("01a178ac-d4d6-7f42-b31b-efa4546672ff")),+}+++def offline_suite(key: str, *, metrics: Sequence[str]) -> OfflineEvalSuite:+ return OfflineEvalSuite.from_scorers(+ key=key,+ project_id=2,+ scorers={metric: _SCORERS[metric] for metric in ("exit_code_zero", *metrics)},+ tags=("team:apm", "component:logs"),+ )diff --git a/products/notebooks/evals/eval_notebook_analysis.py b/products/notebooks/evals/eval_notebook_analysis.pyindex 19d274892c58..09c928baefe8 100644--- a/products/notebooks/evals/eval_notebook_analysis.py+++ b/products/notebooks/evals/eval_notebook_analysis.py@@ -34,7 +34,7 @@ from products.notebooks.evals.scorers import (NotebookCreated,)from products.notebooks.evals.seeders import seed_churn_signal-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall@@ -124,7 +124,7 @@ async def eval_notebook_analysis(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-notebook-analysis-cli",offline_suite=offline_suite("notebooks/eval_notebook_analysis::eval_notebook_analysis",diff --git a/products/notebooks/evals/eval_notebook_cells.py b/products/notebooks/evals/eval_notebook_cells.pyindex fec2cd291701..bb75d37444cd 100644--- a/products/notebooks/evals/eval_notebook_cells.py+++ b/products/notebooks/evals/eval_notebook_cells.py@@ -13,7 +13,7 @@ from __future__ import annotationsfrom products.notebooks.evals.offline_config import offline_suitefrom products.notebooks.evals.scorers import CellRunsCompleted, NotebookCreated, NotebookRunCompletedfrom products.notebooks.evals.seeders import seed_case_team-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall@@ -85,7 +85,7 @@ async def eval_notebook_cells(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-notebook-cells-cli",offline_suite=offline_suite("notebooks/eval_notebook_cells::eval_notebook_cells",diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.mdindex 6e54a16273be..92b2b8c5aedb 100644--- a/products/posthog_ai/eval_harness/README.md+++ b/products/posthog_ai/eval_harness/README.md@@ -49,8 +49,8 @@ python -m products.posthog_ai.eval_harness.harness [SELECTOR ...] [flags]No manual env sourcing is needed on either path.The harness loads the repo-root `.env` itself (shell values win), and `hogli evals` additionally layers in `.env.local` / `.env.development` / `.env.services` through hogli's standard env loading — including 1Password resolution when `.env.local` holds `op://` references.Before any infrastructure boots, a preflight validates that the required variables are set and fails with a one-line fix per missing variable.-Which variables are required depends on the selected suites' reporting settings and kinds: the Braintrust engine requires `BRAINTRUST_API_KEY` only if at least one selected suite sends logs to Braintrust. A run consisting entirely of suites declaring `NO_SEND_LOGS = True` does not require that reporting credential. Sandboxed suites require `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`, and need a [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway) clone plus `go` on `PATH` (see [AI gateway](#ai-gateway)); one-shot suites require `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).-Scorers still need credentials for their model provider. A scorer configured to use Braintrust's model proxy needs its proxy credential independently of result reporting.+Required variables depend on the selected suite kinds. Braintrust runs locally and needs no reporting credential. Sandboxed suites require `SANDBOX_JWT_PRIVATE_KEY` and `LLM_GATEWAY_ANTHROPIC_API_KEY`, and need a [`PostHog/ai-gateway`](https://github.com/PostHog/ai-gateway) clone plus `go` on `PATH` (see [AI gateway](#ai-gateway)); one-shot suites require `LLM_GATEWAY_ANTHROPIC_API_KEY` (used directly, no gateway).+Autoevals judges use a direct OpenAI client with `OPENAI_API_KEY`, including when PostHog trace capture is unavailable. `OPENAI_BASE_URL` can select an explicit provider endpoint. No judge falls back to Braintrust's model proxy or API key.Selectors are substrings matched against a suite id of the form `<domain>/<module>::<fn>`, for example `experiments`, `sql`, or `eval_lifecycle_skills`.Omit them to run every suite.@@ -193,14 +193,14 @@ hogli evals eval_skill_distribution --skill-delivery bundled --agent-runtime clahogli evals eval_skill_distribution --skill-delivery exec --agent-runtime claude --agent-model claude-opus-4-8 --trials 3```-Both runs use the same prompts and Braintrust history key.+Both runs use the same prompts and experiment name.The experiment metadata records `skill_delivery`; compare the shared `expected_skill_loaded` and `skill_loaded_before_tool` scores rather than the aggregate mean, because the exec arm also reports search and discovery diagnostics.### Codex runtime`--agent-runtime codex` runs the same agent-server with OpenAI's Codex harness instead of Claude, defaulting the model to `gpt-6.1-sol`.It requires `LLM_GATEWAY_OPENAI_API_KEY` in the environment (checked by preflight), which the harness's AI gateway uses to proxy the agent's OpenAI calls.-Experiment names don't change with the runtime or skill delivery. Each Braintrust experiment and every PostHog event records `agent_model`, `trials`, `git_sha` and `git_dirty`. Sandboxed runs add `agent_runtime`, `skill_delivery` and `reasoning_effort`. Use these fields to separate runs. A comparison across runtimes measures the model and runtime together.+Experiment names don't change with the runtime or skill delivery. Each local evaluation and every PostHog event records `agent_model`, `trials`, `git_sha` and `git_dirty`. Sandboxed runs add `agent_runtime`, `skill_delivery` and `reasoning_effort`. Use these fields to separate runs. A comparison across runtimes measures the model and runtime together.### Claude and Codex smoke comparison@@ -331,7 +331,7 @@ When object storage is enabled, setup also validates the master warehouse and clWhen a setup completes, its slot becomes available to the next case while the prepared case runs its agent.A case holds a sandbox slot only for the window it actually needs one: team setup and the agent run.-Log parsing, Braintrust span building, trace emission, and scoring all happen after the slot is released.+Log parsing, trace emission, and scoring all happen after the slot is released.The per-case timeout starts after team setup, so neither sandbox nor setup queueing consumes the agent's budget.## Suite kinds@@ -359,16 +359,16 @@ New product-owned evals belong under `products/<product>/evals/`; that directory1. Create `products/posthog_ai/evals/<domain>/eval_<name>.py` (or `products/<product>/evals/eval_<name>.py`). The directory becomes the suite's domain.2. Write one or more coroutines named `eval_*` taking a single `ctx: EvalContext`.-3. Build a list of `SandboxedEvalCase` and hand it to `SandboxedPrivateEval` or `SandboxedPublicEval` along with your scorers and `ctx=ctx`.+3. Build a list of `SandboxedEvalCase` and hand it to `SandboxedEval` along with your scorers and `ctx=ctx`.```python-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextasync def eval_my_thing(ctx: EvalContext) -> None:- await SandboxedPrivateEval(+ await SandboxedEval(experiment_name="sandboxed-my-thing-cli",cases=[SandboxedEvalCase(name="my_case", prompt="...")],scorers=[],@@ -379,13 +379,13 @@ async def eval_my_thing(ctx: EvalContext) -> None:The harness automatically adds the `ExitCodeZero` scorer to every sandboxed experiment.Do not add it to a suite's `scorers` list; the harness rejects duplicates.-For a one-shot suite, declare the kind, build `BaseEvalCase`s, and pass a task function to `OneShotPrivateEval` / `OneShotPublicEval` — the task runs once per case under the global one-shot limiter and returns the scorer `output` dict directly:+For a one-shot suite, declare the kind, build `BaseEvalCase`s, and pass a task function to `OneShotEval` — the task runs once per case under the global one-shot limiter and returns the scorer `output` dict directly:```pythonfrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPrivateEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalSUITE_KIND = SuiteKind.ONE_SHOT@@ -394,7 +394,7 @@ async def eval_my_generation(ctx: EvalContext) -> None:async def task(case: BaseEvalCase, task_ctx: EvalContext) -> dict:return {"answer": ...} # one model invocation; JSON-serializable- await OneShotPrivateEval(+ await OneShotEval(experiment_name="my-generation",cases=[BaseEvalCase(name="my_case", prompt="...")],scorers=[...],@@ -404,9 +404,9 @@ async def eval_my_generation(ctx: EvalContext) -> None:```Bundle related cases into one suite function rather than splitting them across many.-One suite is one Braintrust experiment, which is what makes cross-case comparison and `--eval` filtering useful.+One suite is one experiment, which is what makes cross-case comparison and `--eval` filtering useful.-`experiment_name` is the Braintrust experiment key, so changing it starts a fresh history.+`experiment_name` identifies the run in local output and PostHog; keep it stable across runs.Existing suites end in `-cli` because the MCP server serves the `cli` surface; that suffix is kept so history lines up with earlier runs.Confirm discovery picked your suite up:@@ -428,7 +428,7 @@ Use `--verbose` to show detailed service and agent logs in the console. The tranWarnings and errors use short console messages, including the exception type and message when available.Open the transcript for full tracebacks.-The final summary gives labeled suite and case totals, the score gate, total duration, and one block per experiment with scorer averages, PostHog and Braintrust URLs, and the agent-log directory.+The final summary gives labeled suite and case totals, the score gate, total duration, and one block per experiment with scorer averages, PostHog URLs, and the agent-log directory.Each scorer line shows its score with a 95% confidence interval clustered by case. With `--trials N` a second line adds pass^N (every trial passed), pass@N (any trial passed) and the number of flaky cases, counting only cases scored on all N trials, so a noisy suite is distinguishable from a real score change.A crashed suite is labeled `CRASH` and makes the run exit nonzero without taking down the other suites.Its full traceback stays in the transcript. `--verbose` also shows the traceback in the final summary.@@ -449,19 +449,18 @@ The harness shortens long case names in filenames to stay within 255 bytes. It kThe summary includes the trial ID, the agent trace ID when available, phase durations in seconds, and the scores for that trial.Case names and `--eval` filters stay the same.-`SandboxedPublicEval` sets `no_send_logs=False` and reports results to Braintrust, plus PostHog when the suite has offline publishing configured.-`SandboxedPrivateEval` sets `no_send_logs=True` and uploads results to neither service by default; local logs are still written.-Both sandboxed and one-shot suites can set `send_to_posthog=True` with `offline_suite` to publish only to PostHog while Braintrust uploads remain disabled, or `send_to_posthog=False` to disable PostHog uploads independently.-The 11 experiment-skill suites use this PostHog-only mode.-PostHog result uploads default to following `no_send_logs`, independently of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients.+`SandboxedEval` and `OneShotEval` always execute locally through Braintrust without reporting to its service.+Configure `offline_suite` and `POSTHOG_OFFLINE_EVAL_API_KEY` to publish completed results to PostHog.+`send_to_posthog` defaults to `True`; set it to `False` for local-only results.+PostHog result uploads are independent of `OPT_OUT_CAPTURE`, which still applies to ordinary SDK and trace clients.See [evaluation result reporting](../../../docs/internal/ai-offline-evaluation-reporting.md) for capture settings and scope.### SQL offline experiment pilotThe `sql/eval_sql::eval_sql` suite can also send its seven scorer results to PostHog's offline experiment API.-It publishes the same completed case results through the Python SDK and keeps sending to Braintrust.-PostHog AI and Slack app public suites also opt into offline publishing with their own tagged scorer versions.-Agent and judge trace capture remains available. `no_send_logs=True` suppresses offline result uploads unless `send_to_posthog=True` is explicitly set.+It publishes the same completed case results through the Python SDK.+PostHog AI and product-owned suites also opt into offline publishing with their own tagged scorer versions.+Agent and judge trace capture remains available. `send_to_posthog=False` suppresses offline result uploads.Set `POSTHOG_OFFLINE_EVAL_API_KEY` to a personal key with `offline_evaluation_ingestion:write`, `evaluation:write`, and access to project `2` to upload and tag results. An ingestion-only personal or project secret key can upload results, but tagging will warn and leave the run untagged.The [SQL suite definition](../evals/sql/eval_sql.py) pins that project ID and all seven immutable scorer version UUIDs in code.@@ -482,10 +481,10 @@ hogli evals sql/eval_sql::eval_sql```With no API key, the SQL run reports that offline uploads are disabled.-Invalid configuration and upload failures produce warnings while Braintrust and local reporting continue.+Invalid configuration and upload failures produce warnings while local reporting continues.Successful uploads use separate items for each case and trial; skipped scores and execution errors remain distinct from scores of zero.-Raw agent logs stay in the existing local and Braintrust outputs. The SDK validates payloads; invalid or oversized optional fields are omitted as a whole, with their names recorded in metadata.+Raw agent logs stay in the existing local outputs. The SDK validates payloads; invalid or oversized optional fields are omitted as a whole, with their names recorded in metadata.Failed or interrupted uploads and uploads rejected by a full queue are not saved for later retry.To try publishing again, correct the configuration or connectivity problem and rerun the suite.-Existing local logs, enabled case exports, and Braintrust results remain available; upload failures do not change the score gate.+Existing local logs and enabled case exports remain available; upload failures do not change the score gate.diff --git a/products/posthog_ai/eval_harness/base.py b/products/posthog_ai/eval_harness/base.pyindex e7d77f2bbb23..e7f99a74a66a 100644--- a/products/posthog_ai/eval_harness/base.py+++ b/products/posthog_ai/eval_harness/base.py@@ -34,12 +34,11 @@ from products.tasks.backend.facade.agents import EVAL_INTERACTION_ORIGIN, Customfrom .acp_log import ParsedLog, parse_logfrom .config import AgentArtifacts, BaseEvalCase, SandboxedEvalCasefrom .engines.base import EvalEngine-from .engines.types import CaseHooks, CaseSpec, ExperimentResult, ExperimentSpec, SpanKind+from .engines.types import CaseHooks, CaseSpec, ExperimentResult, ExperimentSpecfrom .harness.kernel_sandboxes import reclaim_kernelsfrom .harness.reporting import CasePhasefrom .harness.retries import infrastructure_retry_delayfrom .harness.trial_stats import trial_stats-from .log_parser import describe_tool_usefrom .log_sink import append_case_scores, build_case_dir, write_case_logsfrom .offline_results import OfflineEvalPublisher, OfflineEvalSettings, OfflineEvalSuite, OfflineUploadErrorfrom .runner import AgentNeverRanError, EvalCaseResult, SandboxedCaseRunner, agent_never_ran, run_eval_case@@ -64,86 +63,6 @@ def _get_last_assistant_text(parsed: ParsedLog) -> str:return ""-def _log_conversation_spans(hooks: CaseHooks, parsed: ParsedLog) -> None:- """Log each conversation message as a child span so Braintrust renders a trace tree.-- Uses the same Anthropic-format messages as PostHog trace capture.- """- for msg in parsed.messages:- role = msg.get("role", "system")- content = msg.get("content", "")-- # Anthropic format: content can be a string or list of content blocks- tool_calls: list[dict[str, Any]] = []- if isinstance(content, list):- # Render content blocks for display- parts: list[str] = []- for block in content:- if not isinstance(block, dict):- continue- block_type = block.get("type", "")- if block_type == "text":- parts.append(block.get("text", ""))- elif block_type == "tool_use":- block_input = block.get("input")- tool, tool_input = describe_tool_use(- block.get("name"), block_input if isinstance(block_input, dict) else {}- )- tool_calls.append({"tool": tool, "input": tool_input})- parts.append(f"[tool_use: {tool or '?'}]")- elif block_type == "tool_result":- result_text = str(block.get("content", ""))[:500]- is_error = block.get("is_error", False)- prefix = "[tool_result error]" if is_error else "[tool_result]"- parts.append(f"{prefix} {result_text}")- display_content = "\n".join(parts)- else:- display_content = str(content)-- if role == "assistant" and tool_calls:- for tool_call in tool_calls:- with hooks.start_span(f"tool_call: {tool_call['tool']}", "function") as span:- span.log(input=[tool_call], output=display_content)- continue-- span_type: SpanKind- if role == "assistant":- span_type = "function" if tool_calls else "llm"- # Naming the span after the resolved tool keeps the trace tree scannable;- # every single-exec call would otherwise read as an undifferentiated "exec".- if len(tool_calls) == 1:- name = f"tool_call: {tool_calls[0]['tool']}"- else:- name = "tool_call" if tool_calls else "agent"- elif role == "user":- has_tool_result = isinstance(content, list) and any(- isinstance(b, dict) and b.get("type") == "tool_result" for b in content- )- span_type = "function"- name = "tool_result" if has_tool_result else "user"- else:- span_type = "function"- name = role-- with hooks.start_span(name, span_type) as span:- if role == "user":- span.log(input=display_content)- elif role == "assistant":- # Logged untruncated: the arguments are the only record of what the- # agent asked for, and a query payload is easy to cut short.- if tool_calls:- span.log(input=tool_calls, output=display_content)- else:- span.log(output=display_content)- else:- span.log(metadata={"message": display_content})-- # Also log non-AI spans (console, errors)- for span_desc in parsed.spans:- with hooks.start_span(span_desc.span_name, "function") as span:- span.log(metadata={"message": span_desc.content})--class _BaseEvalRun:"""One evaluation suite run — the per-experiment state and the genericBraintrust orchestration, independent of how a case's task executes.@@ -164,18 +83,14 @@ class _BaseEvalRun:cases: Sequence[BaseEvalCase],scorers: Sequence[Any],ctx: EvalContext,- is_public: bool,- no_send_logs: bool,engine: EvalEngine | None = None,offline_suite: OfflineEvalSuite | None = None,- send_to_posthog: bool | None = None,+ send_to_posthog: bool = True,) -> None:self.experiment_name = experiment_nameself.cases = casesself.ctx = ctx- self.is_public = is_public- self.no_send_logs = no_send_logs- self.send_to_posthog = not no_send_logs if send_to_posthog is None else send_to_posthog+ self.send_to_posthog = send_to_posthogself.offline_suite = offline_suiteself.offline_publisher: OfflineEvalPublisher | None = Noneself.started_at = datetime.now(UTC)@@ -520,8 +435,6 @@ class _BaseEvalRun:task=self._task,scorers=self.active_scorers,trial_count=self.ctx.trials,- is_public=self.is_public,- no_send_logs=self.no_send_logs,# Experiment names stay runtime/model-agnostic so history lines up across# runs; the metadata is what lets an engine filter or compare by them.metadata=self._experiment_metadata(),@@ -619,10 +532,8 @@ class _SandboxedEvalRun(_BaseEvalRun):cases: Sequence[SandboxedEvalCase],scorers: Sequence[Any],ctx: EvalContext,- is_public: bool,- no_send_logs: bool,offline_suite: OfflineEvalSuite | None = None,- send_to_posthog: bool | None = None,+ send_to_posthog: bool = True,case_runner: SandboxedCaseRunner | None = None,) -> None:if any(isinstance(scorer, ExitCodeZero) for scorer in scorers):@@ -632,8 +543,6 @@ class _SandboxedEvalRun(_BaseEvalRun):cases=cases,scorers=[ExitCodeZero(), *scorers],ctx=ctx,- is_public=is_public,- no_send_logs=no_send_logs,offline_suite=offline_suite,send_to_posthog=send_to_posthog,)@@ -785,7 +694,7 @@ class _SandboxedEvalRun(_BaseEvalRun):result: EvalCaseResult,seed_result: dict[str, Any],) -> dict[str, Any]:- """Build Braintrust spans, write local logs, emit trace events, and shape+ """Write local logs, emit trace events, and shapethe scorer input dict — all off the sandbox slot."""# Store trace_id in metadata so evaluation events can link to the tracetrial_id = hooks.metadata.get("trial_id", eval_case.name)@@ -794,12 +703,11 @@ class _SandboxedEvalRun(_BaseEvalRun):self.agent_trace_id_lookup[trial_id] = result.trace_idhooks.metadata["artifacts"] = result.artifacts.model_dump()- # Parse the log once, use for both Braintrust spans and PostHog trace capture+ # Parse the log once for local output and PostHog trace capturelast_message = ""messages: list[dict[str, Any]] = []if result.raw_log:parsed = parse_log(result.raw_log, initial_prompt=eval_case.prompt)- _log_conversation_spans(hooks, parsed)last_message = _get_last_assistant_text(parsed)messages = parsed.messages@@ -878,9 +786,6 @@ class _SandboxedEvalRun(_BaseEvalRun):stderr=f"case timeout after {self.ctx.per_case_timeout_seconds}s",).model_dump()- def _project_name(self) -> str:- return f"sandboxed-agent-{self.experiment_name}" if self.is_public else self.experiment_name-def _experiment_metadata(self) -> RunMetadata:return {**super()._experiment_metadata(),@@ -895,10 +800,8 @@ async def SandboxedEval(cases: Sequence[SandboxedEvalCase],scorers: Sequence[Any],ctx: EvalContext,- is_public: bool = False,- no_send_logs: bool = True,offline_suite: OfflineEvalSuite | None = None,- send_to_posthog: bool | None = None,+ send_to_posthog: bool = True,case_runner: SandboxedCaseRunner | None = None,) -> ExperimentResult:"""Run a sandboxed agent evaluation suite via Braintrust.@@ -921,17 +824,8 @@ async def SandboxedEval(cases=cases,scorers=scorers,ctx=ctx,- is_public=is_public,- no_send_logs=no_send_logs,offline_suite=offline_suite,send_to_posthog=send_to_posthog,case_runner=case_runner,)return await run.run()---SandboxedPublicEval = partial(SandboxedEval, is_public=True, no_send_logs=False)-"""Sandboxed evaluation case that is publicly accessible."""--SandboxedPrivateEval = partial(SandboxedEval, is_public=False, no_send_logs=True)-"""Sandboxed evaluation case that is not accessible publicly."""diff --git a/products/posthog_ai/eval_harness/engines/base.py b/products/posthog_ai/eval_harness/engines/base.pyindex b52aba1f554c..fcc7a5ee3e20 100644--- a/products/posthog_ai/eval_harness/engines/base.py+++ b/products/posthog_ai/eval_harness/engines/base.py@@ -37,11 +37,8 @@ class EvalEngine(Protocol):name: ClassVar[str]"""Stable engine identifier; ``reporting`` renders ``name.title()`` as the label."""- supports_public_experiments: ClassVar[bool]- """Whether the engine has a notion of a public experiment (``is_public``)."""-@classmethod- def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:+ def required_env(cls) -> tuple[EnvVarSpec, ...]:"""The environment variables this engine needs, validated in preflight."""...diff --git a/products/posthog_ai/eval_harness/engines/braintrust.py b/products/posthog_ai/eval_harness/engines/braintrust.pyindex f714c7fc9a7e..61ae612fdb85 100644--- a/products/posthog_ai/eval_harness/engines/braintrust.py+++ b/products/posthog_ai/eval_harness/engines/braintrust.py@@ -2,23 +2,16 @@from __future__ import annotations-import asyncio-import logging-from collections.abc import Iterator-from contextlib import contextmanagerfrom typing import Any, ClassVarfrom braintrust import EvalAsync, EvalCase, EvalHooksfrom braintrust.framework import EvalResultWithSummary, Evaluator, ReporterDef-from braintrust.logger import Span, SpanImplfrom ..scorers.recording import ScorerRecorder# Imported for its side effect: braintrust's offline summary crashes on a skipped score.from . import braintrust_patches # noqa: F401-from .types import AggregateScore, CaseResult, EnvVarSpec, EvalSummary, ExperimentResult, ExperimentSpec, SpanKind--logger = logging.getLogger(__name__)+from .types import AggregateScore, CaseResult, EnvVarSpec, EvalSummary, ExperimentResult, ExperimentSpecdef _quiet_report_eval(evaluator: Evaluator, result: EvalResultWithSummary, verbose: bool, jsonl: bool) -> bool:@@ -43,9 +36,7 @@ class _BraintrustCaseHooks:"""Adapts braintrust's ``EvalHooks`` to the neutral ``CaseHooks``.``metadata`` is a write-through onto the braintrust hooks (so mutations land- on the eventual ``CaseResult.metadata``); ``start_span`` maps the neutral- ``(name, kind)`` onto braintrust's ``start_span(name=, span_attributes=)`` and- yields the braintrust ``Span``, whose ``.log`` already matches ``SpanHandle``.+ on the eventual ``CaseResult.metadata``)."""def __init__(self, hooks: EvalHooks) -> None:@@ -55,14 +46,9 @@ class _BraintrustCaseHooks:def metadata(self) -> dict[str, Any]:return self._hooks.metadata- @contextmanager- def start_span(self, name: str, kind: SpanKind) -> Iterator[Any]:- with self._hooks.span.start_span(name=name, span_attributes={"type": kind}) as span:- yield span-class BraintrustEngine:- """Runs experiments on Braintrust and reports through it.+ """Runs evaluations locally with Braintrust; results are never uploaded to it.These knobs are load-bearing invariants (they used to live in``harness/AGENTS.md``); breaking one produces a hang or a silently wrong@@ -79,44 +65,24 @@ class BraintrustEngine:each experiment from dumping its own score table into the interleavedstream. Its callbacks are called synchronously by ``EvalAsync`` and mustnot be coroutines.- - **``update=True``.** Experiment names stay runtime/model-agnostic so history- lines up across runs; updating keeps that history rather than forking it.- **``braintrust_patches``.** Importing that module replaces braintrust'soffline summary, which crashes on a skipped score. See its docstring."""name: ClassVar[str] = "braintrust"- supports_public_experiments: ClassVar[bool] = True@classmethod- def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:- if no_send_logs:- return ()- return (EnvVarSpec("BRAINTRUST_API_KEY", "records experiments and scores to Braintrust"),)-- @staticmethod- def _persist_trial_metadata(result: ExperimentResult, trial_spans: dict[str, Span]) -> None:- for case in result.results:- if span := trial_spans.get(case.metadata.get("trial_id", "")):- span.log(metadata=case.metadata)- next(iter(trial_spans.values())).flush()+ def required_env(cls) -> tuple[EnvVarSpec, ...]:+ return ()async def run_experiment(self, spec: ExperimentSpec) -> ExperimentResult:recorder = ScorerRecorder() if spec.record_scorer_results else None- trial_spans: dict[str, Span] = {}async def bridged_task(input: dict[str, Any], hooks: EvalHooks) -> dict[str, Any] | None:if recorder is not None:recorder.register(hooks.metadata, hooks.trial_index)# Wrap braintrust's hooks so the suite task only ever sees neutral CaseHooks.- try:- return await spec.task(input, _BraintrustCaseHooks(hooks))- finally:- if not spec.no_send_logs and isinstance(hooks.span, SpanImpl):- # The pinned SDK exposes only the task span. Its context token retains the trial root span.- parent = hooks.span._context_token.old_value- if isinstance(parent, Span) and (trial_id := hooks.metadata.get("trial_id")):- trial_spans[trial_id] = parent+ return await spec.task(input, _BraintrustCaseHooks(hooks))cases = [EvalCase(input=case.input, expected=case.expected, metadata=case.metadata) for case in spec.cases]result = await EvalAsync(@@ -139,30 +105,21 @@ class BraintrustEngine:# Suites share one stdout; the quiet reporter stops each experiment# from dumping its own score table into the interleaved stream.reporter=QUIET_REPORTER,- update=True,- is_public=spec.is_public,- no_send_logs=spec.no_send_logs,+ no_send_logs=True,# Experiment names stay runtime/model-agnostic so history lines up- # across runs; the metadata is what lets Braintrust filter or compare.+ # across runs; metadata is retained with the local results.metadata=spec.metadata,)translated = self._translate(result, recorder)if spec.on_results is not None:await spec.on_results(translated)- if trial_spans:- try:- await asyncio.to_thread(self._persist_trial_metadata, translated, trial_spans)- except Exception:- logger.warning(- "Could not upload final trial metadata; completed eval results remain available", exc_info=True- )return translateddef _translate(self, result: EvalResultWithSummary, recorder: ScorerRecorder | None = None) -> ExperimentResult:"""Map braintrust's ``EvalResultWithSummary`` onto the neutral model.- ``score=None`` is preserved per-case (the aggregate drops it, online- through braintrust and offline through ``braintrust_patches``); a task+ ``score=None`` is preserved per-case (the local aggregate drops it+ through ``braintrust_patches``); a taskexception becomes ``CaseResult.error`` as a string socallers never re-handle a live ``Exception``; ``summary.raw`` is thebraintrust summary's ``as_dict()`` so the jsonl export round-tripsdiff --git a/products/posthog_ai/eval_harness/engines/braintrust_patches.py b/products/posthog_ai/eval_harness/engines/braintrust_patches.pyindex eab4d4f21289..6962dc12aea7 100644--- a/products/posthog_ai/eval_harness/engines/braintrust_patches.py+++ b/products/posthog_ai/eval_harness/engines/braintrust_patches.py@@ -19,7 +19,7 @@ def none_safe_local_summary(evaluator: Evaluator[Any, Any], results: Sequence[AnThe stock ``build_local_summary`` guards on the accumulator rather than thescore, so it reaches ``0 + None`` and raises ``TypeError`` on the first skippedscore. Every ``no_send_logs`` experiment takes that path, because- ``no_send_logs`` is what leaves the braintrust experiment unset, so a private+ ``no_send_logs`` is what leaves the braintrust experiment unset, so asuite whose scorers skip a check cannot summarize without this.Braintrust ships the same guard from 0.4.2, and this repo pins 0.2.4 indiff --git a/products/posthog_ai/eval_harness/engines/types.py b/products/posthog_ai/eval_harness/engines/types.pyindex 447b5df95ea1..28a1bfd944d0 100644--- a/products/posthog_ai/eval_harness/engines/types.py+++ b/products/posthog_ai/eval_harness/engines/types.py@@ -12,8 +12,7 @@ base, reporting, and trace emission never see an engine-specific type.from __future__ import annotationsimport json-from collections.abc import Awaitable, Callable, Iterator, Sequence-from contextlib import AbstractContextManager, contextmanager+from collections.abc import Awaitable, Callable, Sequencefrom dataclasses import dataclass, fieldfrom datetime import datetimefrom typing import Any, Literal, Protocol@@ -30,44 +29,24 @@ class CaseSpec:metadata: dict[str, Any] = field(default_factory=dict)-SpanKind = Literal["llm", "function", "task", "score"]---class SpanHandle(Protocol):- """A single span the task logs onto; the engine renders it into its own trace."""-- def log(self, *, input: Any = None, output: Any = None, metadata: dict[str, Any] | None = None) -> None: ...--class CaseHooks(Protocol):"""The per-case handle the engine passes to the task.``metadata`` is a mutable dict the engine must persist onto- ``CaseResult.metadata``; ``start_span`` opens a child span the task logs onto.+ ``CaseResult.metadata``."""@propertydef metadata(self) -> dict[str, Any]: ...- def start_span(self, name: str, kind: SpanKind) -> AbstractContextManager[SpanHandle]: ...-EvalTaskFn = Callable[[dict[str, Any], CaseHooks], Awaitable[dict[str, Any] | None]]"""The per-case task the engine drives: it takes the JSON-safe case input and theneutral ``CaseHooks`` and returns the scorer ``output`` dict (or ``None``)."""-class _NullSpanHandle:- def log(self, *, input: Any = None, output: Any = None, metadata: dict[str, Any] | None = None) -> None:- return None--class NullCaseHooks:- """A concrete no-op ``CaseHooks`` for tests and span-less engines.-- Replaces the old ``hooks=None`` sentinel: ``metadata`` is a real (discarded)- dict and ``start_span`` yields a span that swallows every ``log``.- """+ """A metadata-only ``CaseHooks`` for tests and local engines."""def __init__(self) -> None:self._metadata: dict[str, Any] = {}@@ -76,10 +55,6 @@ class NullCaseHooks:def metadata(self) -> dict[str, Any]:return self._metadata- @contextmanager- def start_span(self, name: str, kind: SpanKind) -> Iterator[SpanHandle]:- yield _NullSpanHandle()-@dataclass(frozen=True)class ExperimentSpec:@@ -91,8 +66,6 @@ class ExperimentSpec:task: EvalTaskFnscorers: Sequence[Any] # duck-typed: eval_async(output, expected, **kwargs) + _name()trial_count: int- is_public: bool- no_send_logs: boolmetadata: dict[str, Any]on_results: Callable[[ExperimentResult], Awaitable[None]] | None = Nonerecord_scorer_results: bool = Falsediff --git a/products/posthog_ai/eval_harness/harness/AGENTS.md b/products/posthog_ai/eval_harness/harness/AGENTS.mdindex d0ef3b0bb86c..f3357ea5ae63 100644--- a/products/posthog_ai/eval_harness/harness/AGENTS.md+++ b/products/posthog_ai/eval_harness/harness/AGENTS.md@@ -15,7 +15,7 @@ If you add an import to one of those modules, check that `--list` still runs.## Concurrency-The Braintrust-specific knobs below (`timeout`, `max_concurrency`, `QUIET_REPORTER`, `update`) are applied in one place — `BraintrustEngine.run_experiment` in [`../engines/braintrust.py`](../engines/braintrust.py), the sole `EvalEngine` implementation — and documented as obligations in its docstring. `_BaseEvalRun.run()` only hands the engine the cases, task, scorers, and metadata. A future engine (e.g. a PostHog-native one) owns its own equivalents.+The Braintrust-specific knobs below (`timeout`, `max_concurrency`, `QUIET_REPORTER`, `no_send_logs=True`) are applied in one place — `BraintrustEngine.run_experiment` in [`../engines/braintrust.py`](../engines/braintrust.py), the sole `EvalEngine` implementation — and documented as obligations in its docstring. `_BaseEvalRun.run()` only hands the engine the cases, task, scorers, and metadata. A future engine (e.g. a PostHog-native one) owns its own equivalents.- **Never give Braintrust a `timeout`.** `EvalAsync`'s timeout wraps the whole task invocation, including time queued on our concurrency semaphores, so it would kill cases that never started. The budget belongs in the `asyncio.wait_for` inside the slot.- **Never let Braintrust's `max_concurrency` bind.** It is set to the case count so that the harness's own semaphores are the only limiters.@@ -29,6 +29,7 @@ The Braintrust-specific knobs below (`timeout`, `max_concurrency`, `QUIET_REPORT`_BaseEvalRun.run()` builds an `ExperimentSpec` (cases, task, scorers, flags, metadata) and hands it to `ctx.engine.run_experiment`, which returns an engine-neutral `ExperimentResult` ([`../engines/types.py`](../engines/types.py)).The engine is resolved once per run by `engines/registry.resolve_engine()` and shared via `EvalContext.engine`; there is no `--engine` flag until a second engine exists.+`BraintrustEngine` always runs locally with `no_send_logs=True`, without authentication or uploads to Braintrust.`BraintrustEngine` ([`../engines/braintrust.py`](../engines/braintrust.py)) is the only implementation and the sole runtime module that knows braintrust exists, alongside the deferred autoevals judge substrate in `../scorers/`.Every engine owes the run base these obligations, pinned by the conformance suite ([`../test/test_engine_conformance.py`](../test/test_engine_conformance.py)):@@ -37,9 +38,9 @@ Every engine owes the run base these obligations, pinned by the conformance suit- Persist `hooks.metadata` mutations onto `CaseResult.metadata`.- Turn a task exception into `CaseResult.error` (a string) and exclude that case from aggregates.- Preserve `score=None` per-case and exclude it from the means — `None` is "skipped", `0.0` is "failed".-- Print nothing to the shared stdout (braintrust's `QUIET_REPORTER` is how it stays quiet), and accumulate history under a stable `project_name`.+- Print nothing to the shared stdout (braintrust's `QUIET_REPORTER` is how it stays quiet), and use a stable `project_name` for local summaries.-A new engine implements the `EvalEngine` protocol ([`../engines/base.py`](../engines/base.py)) — `name`, `supports_public_experiments`, `required_env()`, `run_experiment(spec)` — registers in `engines/registry.py`, joins the conformance fixture list, and must pass it unchanged.+A new engine implements the `EvalEngine` protocol ([`../engines/base.py`](../engines/base.py)) — `name`, `required_env()`, `run_experiment(spec)` — registers in `engines/registry.py`, joins the conformance fixture list, and must pass it unchanged.## Terminal output@@ -131,7 +132,7 @@ Env loading follows the same before-`django.setup()` rule: `__main__` loads theThe explicit `SANDBOX_PROVIDER` / `PERSONHOG_ADDR` assignments come after the load, so they trump any env file on every path.`setup_django()` also forces `SELF_CAPTURE=0` before `django.setup()`. The ASGI app must not re-enable the global SDK with the local development personal API key during evals; eval trace emission uses the harness's explicit regional client instead.-Required variables are then validated by `env_preflight.validate_eval_env()` at the top of `lifecycle.run()`, before any infrastructure boots — add new hard env requirements to the per-kind env models there (`SandboxEvalEnv`, `OneShotEvalEnv`), or, for engine-specific keys, to the engine's `required_env()` (`BRAINTRUST_API_KEY` lives on `BraintrustEngine`), which `lifecycle` threads in via the `engine_env` argument — not as ad-hoc checks scattered through bootstrap.+Required variables are then validated by `env_preflight.validate_eval_env()` at the top of `lifecycle.run()`, before any infrastructure boots — add new hard env requirements to the per-kind env models there (`SandboxEvalEnv`, `OneShotEvalEnv`), or, for engine-specific keys, to the engine's `required_env()`, which `lifecycle` threads in via the `engine_env` argument — not as ad-hoc checks scattered through bootstrap.## pytestdiff --git a/products/posthog_ai/eval_harness/harness/README.md b/products/posthog_ai/eval_harness/harness/README.mdindex 16c82abd2d93..cca017c85025 100644--- a/products/posthog_ai/eval_harness/harness/README.md+++ b/products/posthog_ai/eval_harness/harness/README.md@@ -12,7 +12,7 @@ Under pytest that infrastructure lived in session fixtures, and suites ran stricWall-clock was dominated by suites queueing behind each other while sandbox capacity sat idle.The harness boots the same infrastructure once, then runs every suite concurrently on a single event loop and bounds sandbox and team-setup load with shared semaphores.-Braintrust remains the eval engine and reporting backend; it just no longer controls scheduling.+Braintrust remains the local eval engine. PostHog owns remote result reporting, and the harness controls scheduling.The `EvalAsync` call itself lives behind an `EvalEngine` seam (`../engines/`): `_BaseEvalRun.run()` hands a neutral `ExperimentSpec` to the engine resolved by `engines/registry.resolve_engine()` (shared via `EvalContext.engine`) and gets back a neutral `ExperimentResult` (`engines/types.py`), so a future PostHog-native engine can slot in behind the same interface — implementing the `EvalEngine` protocol and passing the conformance suite — without touching the run base or any suite.Not every suite is sandboxed anymore: each suite declares a `SUITE_KIND` (`requirements.py`), the kind maps to a set of `Infra` requirements, and the harness boots only the union of what the selected suites need.@@ -89,7 +89,7 @@ Only the final overall run status uses `PASS` or `FAIL`.`CASE START` and `CASE PHASE` identify each trial. `CASE READY` marks completed execution with scoring still pending.The reporter prints phase counts and the oldest active trial every 30 seconds while suites run.The `scoring` duration includes waiting for the experiment results. It does not measure scorer execution alone.-Public runs update Braintrust trial metadata after scoring so it contains the final phase durations.+Local results and PostHog uploads include final phase durations after scoring.Scorer exceptions produce case errors in the console and final summary.The reporter writes output in worker threads so slow output does not stop the event loop.Terminal writes do not hold the transcript file lock, so progress output cannot block logging through that lock.diff --git a/products/posthog_ai/eval_harness/harness/discovery.py b/products/posthog_ai/eval_harness/harness/discovery.pyindex 2d77c1b44d18..0b56382a4222 100644--- a/products/posthog_ai/eval_harness/harness/discovery.py+++ b/products/posthog_ai/eval_harness/harness/discovery.py@@ -51,9 +51,6 @@ class EvalSuite:kind: SuiteKind = SuiteKind.SANDBOXED"""How the suite's cases execute, from the module-level ``SUITE_KIND``."""- no_send_logs: bool = False- """Module-level ``NO_SEND_LOGS`` lets preflight check credentials before suite setup."""-@propertydef id(self) -> str:return f"{self.domain}/{self.module_name}::{self.fn_name}"@@ -115,9 +112,6 @@ def _collect_suites(module: ModuleType, domain: str, module_name: str, dotted: skind = getattr(module, "SUITE_KIND", SuiteKind.SANDBOXED)if not isinstance(kind, SuiteKind):raise SuiteDiscoveryError(f"{dotted}.SUITE_KIND must be a SuiteKind member, got {kind!r}")- no_send_logs = getattr(module, "NO_SEND_LOGS", False)- if not isinstance(no_send_logs, bool):- raise SuiteDiscoveryError(f"{dotted}.NO_SEND_LOGS must be a bool, got {no_send_logs!r}")multi_turn = bool(getattr(module, MULTI_TURN_MODULE_MARKER, False))collected: list[EvalSuite] = []for fn_name, fn in vars(module).items():@@ -130,11 +124,7 @@ def _collect_suites(module: ModuleType, domain: str, module_name: str, dotted: sif multi_turn:# The dataclass is frozen, so carry the marker on the function.setattr(fn, MULTI_TURN_MODULE_MARKER, True)- collected.append(- EvalSuite(- domain=domain, module_name=module_name, fn_name=fn_name, fn=fn, kind=kind, no_send_logs=no_send_logs- )- )+ collected.append(EvalSuite(domain=domain, module_name=module_name, fn_name=fn_name, fn=fn, kind=kind))return collecteddiff --git a/products/posthog_ai/eval_harness/harness/env_preflight.py b/products/posthog_ai/eval_harness/harness/env_preflight.pyindex 14c8fc25c1aa..c52a89ffd74e 100644--- a/products/posthog_ai/eval_harness/harness/env_preflight.py+++ b/products/posthog_ai/eval_harness/harness/env_preflight.py@@ -143,8 +143,7 @@ def validate_eval_env() -> None:"""Fail fast, before any infrastructure boots, if a required variable is unset.- ``engine_env`` is the selected engine's ``required_env(no_send_logs=...)``- for the selected suites' reporting settings. Only the env models for the selected suites'+ ``engine_env`` is the selected engine's ``required_env()``. Only the env models for the selected suites'kinds are checked, so a run without sandboxed suites doesn't demand sandboxcredentials. Without this a missing key surfaces minutes into a run as anopaque mid-case failure (gateway 401s, engine login errors) instead of a@@ -159,8 +158,6 @@ def validate_eval_env(lines = []seen: set[str] = set()- # Engine env first, so its line (e.g. BRAINTRUST_API_KEY) leads the report as- # the core CoreEvalEnv field used to.for spec in engine_env:if spec.name in seen or os.environ.get(spec.name):continuediff --git a/products/posthog_ai/eval_harness/harness/lifecycle.py b/products/posthog_ai/eval_harness/harness/lifecycle.pyindex bc5c898798f5..4df26dee95ca 100644--- a/products/posthog_ai/eval_harness/harness/lifecycle.py+++ b/products/posthog_ai/eval_harness/harness/lifecycle.py@@ -209,7 +209,7 @@ class SandboxedEvalHarness:validate_eval_env(self.options.agent_runtime,kinds=kinds,- engine_env=self._engine.required_env(no_send_logs=all(suite.no_send_logs for suite in suites)),+ engine_env=self._engine.required_env(),)if Infra.SANDBOX in required:self.provider = build_provider(diff --git a/products/posthog_ai/eval_harness/harness/reporting.py b/products/posthog_ai/eval_harness/harness/reporting.pyindex 71b821804d2d..e26ff5dcd559 100644--- a/products/posthog_ai/eval_harness/harness/reporting.py+++ b/products/posthog_ai/eval_harness/harness/reporting.py@@ -530,7 +530,6 @@ class ProgressReporter:lines.append(f" PostHog: {posthog_url}")if (offline_status := self._posthog_offline_statuses.get(experiment_name)) and offline_status != "uploaded":lines.append(f" PostHog offline: {offline_status}")- lines.append(f" {summary.engine_name.title()}: {summary.experiment_url or '(local run, not uploaded)'}")log_dir = self._log_dirs.get(experiment_name)if log_dir:lines.append(f" Agent logs: {log_dir}")diff --git a/products/posthog_ai/eval_harness/one_shot.py b/products/posthog_ai/eval_harness/one_shot.pyindex ec3219a59ad3..4e85cf3b26f1 100644--- a/products/posthog_ai/eval_harness/one_shot.py+++ b/products/posthog_ai/eval_harness/one_shot.py@@ -13,7 +13,6 @@ import timeimport asyncioimport loggingfrom collections.abc import Awaitable, Callable, Sequence-from functools import partialfrom typing import TYPE_CHECKING, Anyfrom .base import _BaseEvalRun@@ -48,19 +47,15 @@ class _OneShotEvalRun(_BaseEvalRun):cases: Sequence[BaseEvalCase],scorers: Sequence[Any],ctx: EvalContext,- is_public: bool,- no_send_logs: bool,task_fn: OneShotTaskFn,offline_suite: OfflineEvalSuite | None = None,- send_to_posthog: bool | None = None,+ send_to_posthog: bool = True,) -> None:super().__init__(experiment_name=experiment_name,cases=cases,scorers=scorers,ctx=ctx,- is_public=is_public,- no_send_logs=no_send_logs,offline_suite=offline_suite,send_to_posthog=send_to_posthog,)@@ -113,10 +108,8 @@ async def OneShotEval(scorers: Sequence[Any],task: OneShotTaskFn,ctx: EvalContext,- is_public: bool = False,- no_send_logs: bool = True,offline_suite: OfflineEvalSuite | None = None,- send_to_posthog: bool | None = None,+ send_to_posthog: bool = True,) -> ExperimentResult:"""Run a one-shot evaluation suite via Braintrust.@@ -131,17 +124,8 @@ async def OneShotEval(cases=cases,scorers=scorers,ctx=ctx,- is_public=is_public,- no_send_logs=no_send_logs,task_fn=task,offline_suite=offline_suite,send_to_posthog=send_to_posthog,)return await run.run()---OneShotPublicEval = partial(OneShotEval, is_public=True, no_send_logs=False)-"""One-shot evaluation whose Braintrust experiment is publicly accessible."""--OneShotPrivateEval = partial(OneShotEval, is_public=False, no_send_logs=True)-"""One-shot evaluation with Braintrust uploads disabled; PostHog publishing is opt-in."""diff --git a/products/posthog_ai/eval_harness/scorers/judged.py b/products/posthog_ai/eval_harness/scorers/judged.pyindex 521e3fe205b8..7c74dd27f0ac 100644--- a/products/posthog_ai/eval_harness/scorers/judged.py+++ b/products/posthog_ai/eval_harness/scorers/judged.py@@ -8,10 +8,13 @@ instead of a silently divergent code path.from __future__ import annotations+from copy import copyfrom typing import Anyfrom autoevals.llm import LLMClassifier+from autoevals.oai import LLMClientfrom braintrust import Score as BraintrustScore+from openai import AsyncOpenAI, RateLimitErrorfrom ..engines.types import ScorerResultfrom .contract import AsyncOnlyScorerMixin, Score@@ -39,7 +42,26 @@ GRADED_ALIGNMENT_CHOICE_SCORES = {JUDGE_MODEL = "gpt-5.4"-class JudgedScorer(AsyncOnlyScorerMixin, LLMClassifier):+class LocalLLMClassifier(AsyncOnlyScorerMixin, LLMClassifier):+ """Use the configured OpenAI provider without Autoevals' Braintrust proxy fallback."""++ async def _run_eval_async(self, output: Any, expected: Any = None, **kwargs: Any) -> BraintrustScore:+ if self.client is not None:+ return await super()._run_eval_async(output, expected, **kwargs)+ async with AsyncOpenAI() as client:+ scorer = copy(self)+ scorer.client = LLMClient(+ openai=client,+ complete=client.chat.completions.create,+ embed=client.embeddings.create,+ moderation=client.moderations.create,+ is_async=True,+ RateLimitError=RateLimitError,+ )+ return await LLMClassifier._run_eval_async(scorer, output, expected, **kwargs)+++class JudgedScorer(LocalLLMClassifier):"""Base class for sandboxed LLM judges.Subclasses implement ``_prepare(output, expected)`` returning either adiff --git a/products/posthog_ai/eval_harness/test/test_braintrust_engine.py b/products/posthog_ai/eval_harness/test/test_braintrust_engine.pyindex 219ef04cee40..f8c5c4ab213d 100644--- a/products/posthog_ai/eval_harness/test/test_braintrust_engine.py+++ b/products/posthog_ai/eval_harness/test/test_braintrust_engine.py@@ -5,17 +5,11 @@ from typing import Anyimport pytestfrom unittest.mock import MagicMock-from braintrust import (- framework as bt_framework,- logger as bt_logger,-)+from braintrust import framework as bt_frameworkfrom braintrust.framework import EvalResult, EvalResultWithSummary-from braintrust.logger import ExperimentSummary, ScoreSummary, SpanImpl-from braintrust.span_identifier_v3 import SpanObjectTypeV3-from braintrust.util import LazyValue+from braintrust.logger import ExperimentSummary, ScoreSummary-from products.posthog_ai.eval_harness.engines import braintrust as engine_module-from products.posthog_ai.eval_harness.engines.braintrust import BraintrustEngine, _BraintrustCaseHooks+from products.posthog_ai.eval_harness.engines.braintrust import BraintrustEnginefrom products.posthog_ai.eval_harness.engines.types import CaseHooks, CaseSpec, ExperimentResult, ExperimentSpecfrom products.posthog_ai.eval_harness.harness.trial_stats import trial_stats@@ -69,42 +63,16 @@ def test_translate_preserves_none_scores_and_stringifies_errors() -> None:@pytest.mark.asyncio[email protected]("has_key", [False, True])@pytest.mark.parametrize("scorer_fails", [False, True])[email protected]("metadata_fails", [False, True])-async def test_public_trial_metadata_keeps_final_scoring_durations(- monkeypatch: pytest.MonkeyPatch, scorer_fails: bool, metadata_fails: bool+async def test_local_results_keep_final_scoring_durations_without_uploading(+ monkeypatch: pytest.MonkeyPatch, has_key: bool, scorer_fails: bool) -> None:- logger = MagicMock()- monkeypatch.setattr(engine_module, "logger", logger)- events: list[dict[str, Any]] = []- flushed: list[list[dict[str, Any]]] = []- background = MagicMock()- background.log.side_effect = lambda event: events.append(event.get())- updating_metadata = False-- def flush() -> None:- if updating_metadata and metadata_fails:- raise ConnectionError("metadata upload failed")- flushed.append(list(events))-- background.flush.side_effect = flush- monkeypatch.setattr(bt_logger._state, "global_bg_logger", lambda: background)-- def start_span(name: str, **kwargs: Any) -> SpanImpl:- return SpanImpl(- parent_object_type=SpanObjectTypeV3.EXPERIMENT,- parent_object_id=LazyValue(lambda: "example-experiment", use_mutex=False),- parent_compute_object_metadata_args=None,- parent_span_ids=None,- name=name,- event=kwargs,- )-- experiment = MagicMock(dataset=None)- experiment.start_span.side_effect = start_span- experiment.summarize.return_value = _summary()- experiment.flush.side_effect = background.flush- monkeypatch.setattr(bt_framework, "init_experiment", lambda **_: experiment)+ monkeypatch.delenv("BRAINTRUST_API_KEY", raising=False)+ if has_key:+ monkeypatch.setenv("BRAINTRUST_API_KEY", "test-only-key")+ remote = MagicMock(side_effect=AssertionError("Braintrust reporting must never initialize"))+ monkeypatch.setattr(bt_framework, "init_experiment", remote)trials = 0async def task(input: dict[str, Any], hooks: CaseHooks) -> dict[str, Any]:@@ -120,9 +88,6 @@ async def test_public_trial_metadata_keeps_final_scoring_durations(return 1.0async def on_results(result: ExperimentResult) -> None:- nonlocal updating_metadata- assert flushed- updating_metadata = Truefor case in result.results:assert (case.error is not None) is scorer_failscase.metadata["phase_durations"]["scoring"] = 3.0@@ -134,28 +99,15 @@ async def test_public_trial_metadata_keeps_final_scoring_durations(task=task,scorers=[score],trial_count=2,- is_public=True,- no_send_logs=False,metadata={},on_results=on_results,))-- rows: dict[str, dict[str, Any]] = {}- for event in flushed[-1]:- if event.get("span_attributes", {}).get("name") == "eval":- rows[event["id"]] = event- elif event["id"] in rows:- rows[event["id"]].update(event)- assert len(rows) == 2- assert {row["metadata"]["trial_id"] for row in rows.values()} == {"trial-0", "trial-1"}- if metadata_fails:- logger.warning.assert_called_once_with(- "Could not upload final trial metadata; completed eval results remain available", exc_info=True- )- else:- assert all(row["metadata"]["phase_durations"] == {"agent": 2.0, "scoring": 3.0} for row in rows.values())- assert all(case.metadata["phase_durations"]["scoring"] == 3.0 for case in result.results)+ remote.assert_not_called()+ assert result.summary.experiment_url is None+ assert len(result.results) == 2+ assert {case.metadata["trial_id"] for case in result.results} == {"trial-0", "trial-1"}+ assert all(case.metadata["phase_durations"] == {"agent": 2.0, "scoring": 3.0} for case in result.results)def test_translate_summary_raw_round_trips_as_json() -> None:@@ -164,49 +116,3 @@ def test_translate_summary_raw_round_trips_as_json() -> None:# The jsonl export must stay byte-identical to what braintrust would write.assert translated.summary.as_json() == summary.as_json()---class _FakeSpan:- def __init__(self) -> None:- self.logs: list[dict[str, Any]] = []-- def log(self, **kwargs: Any) -> None:- self.logs.append(kwargs)-- def __enter__(self) -> _FakeSpan:- return self-- def __exit__(self, *exc: Any) -> None:- return None---class _FakeSpanFactory:- def __init__(self) -> None:- self.started: list[tuple[str, dict[str, Any], _FakeSpan]] = []-- def start_span(self, *, name: str, span_attributes: dict[str, Any]) -> _FakeSpan:- span = _FakeSpan()- self.started.append((name, span_attributes, span))- return span---class _FakeEvalHooks:- def __init__(self) -> None:- self.metadata: dict[str, Any] = {}- self.span = _FakeSpanFactory()---def test_case_hooks_adapter_writes_through_metadata_and_translates_spans() -> None:- hooks = _FakeEvalHooks()- adapter = _BraintrustCaseHooks(hooks) # type: ignore[arg-type]-- adapter.metadata["trace_id"] = "t1"- # Write-through: mutations must land on the underlying braintrust hooks.- assert hooks.metadata == {"trace_id": "t1"}-- with adapter.start_span("agent", "llm") as span:- span.log(output="hi")-- name, attrs, started_span = hooks.span.started[0]- assert (name, attrs) == ("agent", {"type": "llm"})- assert started_span.logs == [{"output": "hi"}]diff --git a/products/posthog_ai/eval_harness/test/test_discovery.py b/products/posthog_ai/eval_harness/test/test_discovery.pyindex d98737f45366..7236ae83a6cf 100644--- a/products/posthog_ai/eval_harness/test/test_discovery.py+++ b/products/posthog_ai/eval_harness/test/test_discovery.py@@ -11,7 +11,6 @@ _ONE_SHOT_MODULE = """from products.posthog_ai.eval_harness.harness.requirements import SuiteKindSUITE_KIND = SuiteKind.ONE_SHOT-NO_SEND_LOGS = Trueasync def eval_alpha(ctx) -> None:@@ -71,18 +70,17 @@ def test_singular_eval_dir_is_ignored(product_roots: tuple[Path, Path]) -> None:@pytest.mark.parametrize(- "suite_id, expected_kind, no_send_logs",+ "suite_id, expected_kind",[- pytest.param("analytics/eval_one::eval_alpha", SuiteKind.ONE_SHOT, True, id="markers_respected"),- pytest.param("core/eval_two::eval_beta", SuiteKind.SANDBOXED, False, id="absent_markers_keep_defaults"),+ pytest.param("analytics/eval_one::eval_alpha", SuiteKind.ONE_SHOT, id="markers_respected"),+ pytest.param("core/eval_two::eval_beta", SuiteKind.SANDBOXED, id="absent_markers_keep_defaults"),],)def test_suite_configuration_resolved_from_product_module(- product_roots: tuple[Path, Path], suite_id: str, expected_kind: SuiteKind, no_send_logs: bool+ product_roots: tuple[Path, Path], suite_id: str, expected_kind: SuiteKind) -> None:suite = _discover(product_roots)[suite_id]assert suite.kind == expected_kind- assert suite.no_send_logs == no_send_logsdef test_selector_matches_product_suite_ids(product_roots: tuple[Path, Path]) -> None:diff --git a/products/posthog_ai/eval_harness/test/test_engine_conformance.py b/products/posthog_ai/eval_harness/test/test_engine_conformance.pyindex 013323898f2a..7b4c981bb3d8 100644--- a/products/posthog_ai/eval_harness/test/test_engine_conformance.py+++ b/products/posthog_ai/eval_harness/test/test_engine_conformance.py@@ -69,8 +69,6 @@ def _spec(task=task,scorers=scorers if scorers is not None else [_MaybeNoneScorer()],trial_count=trial_count,- is_public=False,- no_send_logs=True,metadata={},record_scorer_results=record_scorer_results,)@@ -219,6 +217,8 @@ def test_scorer_details_and_errors_stay_with_concurrent_trials(engine: Braintrusdef test_judge_failure_preserves_zero_aggregate_and_error_upload(engine: BraintrustEngine, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("OPENAI_API_KEY", "test-only-key")+class _Judge(JudgedScorer):def _prepare(self, output: Any, expected: Any) -> dict[str, Any]:return {"output": output}diff --git a/products/posthog_ai/eval_harness/test/test_judged_clients.py b/products/posthog_ai/eval_harness/test/test_judged_clients.pynew file mode 100644index 000000000000..bce25e92c6b5--- /dev/null+++ b/products/posthog_ai/eval_harness/test/test_judged_clients.py@@ -0,0 +1,82 @@+from typing import Any++import pytest++import httpx+from openai import AsyncOpenAI, OpenAIError++from products.posthog_ai.eval_harness.scorers import judged++[email protected]("provider_key", [None, "test-openai-key"])[email protected]("scorer_type", ["nested", "judged"])+async def test_judges_use_direct_provider_without_braintrust_fallback(+ monkeypatch: pytest.MonkeyPatch, provider_key: str | None, scorer_type: str+) -> None:+ monkeypatch.setenv("BRAINTRUST_API_KEY", "test-braintrust-key")+ monkeypatch.delenv("OPENAI_BASE_URL", raising=False)+ monkeypatch.delenv("OPENAI_API_KEY", raising=False)+ if provider_key:+ monkeypatch.setenv("OPENAI_API_KEY", provider_key)+ requests: list[httpx.Request] = []++ def respond(request: httpx.Request) -> httpx.Response:+ requests.append(request)+ return httpx.Response(+ 200,+ json={+ "id": "test-completion",+ "object": "chat.completion",+ "created": 0,+ "model": "test-model",+ "choices": [+ {+ "index": 0,+ "finish_reason": "tool_calls",+ "message": {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "test-call",+ "type": "function",+ "function": {"name": "select_choice", "arguments": '{"choice":"yes"}'},+ }+ ],+ },+ }+ ],+ },+ )++ clients: list[AsyncOpenAI] = []++ def create_client() -> AsyncOpenAI:+ client = AsyncOpenAI(http_client=httpx.AsyncClient(transport=httpx.MockTransport(respond)))+ clients.append(client)+ return client++ monkeypatch.setattr(judged, "AsyncOpenAI", create_client)++ class PreparedJudge(judged.JudgedScorer):+ def _prepare(self, output: Any, expected: Any) -> dict[str, Any]:+ return {"output": output}++ classifier = judged.LocalLLMClassifier if scorer_type == "nested" else PreparedJudge+ scorer = classifier(name="judge", prompt_template="{{output}}", choice_scores={"yes": 1.0}, model="test-model")+ if provider_key is None and scorer_type == "nested":+ with pytest.raises(OpenAIError, match="OPENAI_API_KEY"):+ await scorer.eval_async("Synthetic answer")+ else:+ score = await scorer.eval_async("Synthetic answer")+ assert score.score == (1.0 if provider_key else 0.0)+ if provider_key is None:+ assert "OPENAI_API_KEY" in score.metadata["reason"]++ assert len(requests) == (1 if provider_key else 0)+ if requests:+ assert requests[0].url == "https://api.openai.com/v1/chat/completions"+ assert requests[0].headers["authorization"] == "Bearer test-openai-key"+ assert all(client.is_closed() for client in clients)+ assert scorer.client is Nonediff --git a/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py b/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.pyindex b7d7360b0cac..5e6b94adf685 100644--- a/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py+++ b/products/posthog_ai/eval_harness/test/test_offline_delivery_lifecycle.py@@ -89,7 +89,6 @@ def test_harness_tears_down_infrastructure_before_draining_uploads(fn_name="eval_test",fn=suite,kind=SuiteKind.ONE_SHOT,- no_send_logs=True,)],)diff --git a/products/posthog_ai/eval_harness/test/test_offline_enrollment.py b/products/posthog_ai/eval_harness/test/test_offline_enrollment.pyindex 259f38d36a87..c3b6aec3cca6 100644--- a/products/posthog_ai/eval_harness/test/test_offline_enrollment.py+++ b/products/posthog_ai/eval_harness/test/test_offline_enrollment.py@@ -38,12 +38,7 @@ async def test_discovered_suites_match_reporting_enrollment(tmp_path: Path) -> Nawait suite.fn(ctx)assert len(runs) == 1, suite.idrun = runs[0]- assert run.no_send_logs == suite.no_send_logs, suite.idconfig = run.offline_suite- if suite.id == "logs/eval_agent_observability::eval_agent_observability":- assert config is None- assert not run.no_send_logs- continueassert config is not None, suite.idassert run.send_to_posthog, suite.idassert config.key == suite.id@@ -57,6 +52,3 @@ async def test_discovered_suites_match_reporting_enrollment(tmp_path: Path) -> Nassert ("exit_code_zero" in metrics) == (suite.kind == SuiteKind.SANDBOXED), suite.idassert any(tag.startswith("team:") for tag in config.tags), suite.idassert any(tag.startswith("component:") for tag in config.tags), suite.id-- if suite.domain in {"feature_flags", "dashboards", "business_knowledge", "turn_suggestions"}:- assert run.no_send_logs, suite.iddiff --git a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py b/products/posthog_ai/eval_harness/test/test_one_shot_runner.pyindex 57bf2da45036..50952878e010 100644--- a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py+++ b/products/posthog_ai/eval_harness/test/test_one_shot_runner.py@@ -86,14 +86,13 @@ class _StubReporter:class _StubEngine:name: ClassVar[str] = "stub"- supports_public_experiments: ClassVar[bool] = Falsedef __init__(self, result: ExperimentResult) -> None:self.result = resultself.calls: list[ExperimentSpec] = []@classmethod- def required_env(cls, *, no_send_logs: bool = False) -> tuple[EnvVarSpec, ...]:+ def required_env(cls) -> tuple[EnvVarSpec, ...]:return ()async def run_experiment(self, spec: ExperimentSpec) -> ExperimentResult:@@ -133,8 +132,6 @@ def _build_run(cases=[BaseEvalCase(name=case_name, prompt="the prompt"), BaseEvalCase(name="c2", prompt="other")],scorers=[],ctx=ctx,- is_public=False,- no_send_logs=True,task_fn=task_fn,)@@ -238,10 +235,7 @@ def test_case_filter_narrows_eval_cases(tmp_path: Path, monkeypatch: pytest.Monk@pytest.mark.parametrize("opt_out_capture", ["", "1"])[email protected]("no_send_logs", [False, True])-def test_run_routes_through_the_engine(- tmp_path: Path, monkeypatch: pytest.MonkeyPatch, opt_out_capture: str, no_send_logs: bool-) -> None:+def test_run_routes_through_the_engine(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, opt_out_capture: str) -> None:async def task(case: BaseEvalCase, ctx: EvalContext) -> dict[str, Any]:raise AssertionError("the engine is stubbed, so the task must not run")@@ -292,8 +286,6 @@ def test_run_routes_through_the_engine(runs = [_build_run(tmp_path, monkeypatch, ctx, task) for _ in range(2)]for run in runs:run.engine = engine- run.no_send_logs = no_send_logs- run.is_public = not no_send_logsrun.agent_trace_id_lookup["c1"] = "trace-1"run.case_trace_meta["c1"] = {"prompt": "the prompt", "duration": 1.0, "first_timestamp": ""}asyncio.run(run_suite(run))@@ -317,7 +309,6 @@ def test_run_routes_through_the_engine("git_sha": "abc123","git_dirty": False,}- assert experiment.no_send_logs == no_send_logsassert reporter.started == [("one-shot-test", 2)] * 2assert reporter.summaries == [("one-shot-test", canned.summary, 0)] * 2diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.pyindex 93fcbb5ffc78..1712b77cce73 100644--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_output.py@@ -10,11 +10,10 @@ import warningsimport importlibimport subprocessfrom collections.abc import Iterator-from contextlib import contextmanagerfrom functools import partialfrom pathlib import Pathfrom threading import Event, Thread, get_ident-from typing import Any, Literal+from typing import Literalfrom uuid import UUIDimport pytest@@ -31,7 +30,6 @@ from psycopg.errors import AdminShutdown, InvalidPasswordfrom posthog.exceptions import ClickHouseAtCapacity, ClickHouseQueryMemoryLimitExceeded, ClickHouseQueryTimeOutfrom products.posthog_ai.eval_harness import base, log_sink-from products.posthog_ai.eval_harness.acp_log import GenerationDescriptor, ParsedLogfrom products.posthog_ai.eval_harness.config import AgentArtifacts, SandboxedEvalCasefrom products.posthog_ai.eval_harness.engines.types import (AggregateScore,@@ -192,7 +190,7 @@ async def test_reporter_output_is_labeled_and_reserves_pass_for_the_run(assert "exit_code_zero: 100.0%" in outputassert "called_target_tool: 0.0%" in outputassert "PostHog: https://us.posthog.com/" in output- assert "Braintrust: https://experiments.example/e" in output+ assert "Braintrust:" not in outputassert f"Agent logs: {tmp_path}" in outputassert output.count("PASS") == 1@@ -233,7 +231,7 @@ async def test_reporter_output_is_labeled_and_reserves_pass_for_the_run(pass_any=None,flaky_cases=0,),- " s: 72.0% (95% CI 61.0-83.0%)\n Braintrust",+ " s: 72.0% (95% CI 61.0-83.0%)\n",id="single_trial",),pytest.param(@@ -269,89 +267,6 @@ async def test_reporter_shows_the_engine_score_with_trial_stats(assert expected in capsys.readouterr().out-def test_tool_call_spans_carry_the_resolved_tool_and_its_arguments() -> None:- trends_query = {"kind": "TrendsQuery", "series": [{"kind": "EventsNode", "event": "$pageview"}]}- parsed = ParsedLog(- generations=[- GenerationDescriptor(- output_content=[- {- "type": "tool_use",- "id": "1",- "name": "mcp__posthog__exec",- "input": {"command": f"call query-trends {json.dumps(trends_query)}"},- }- ],- )- ]- )-- spans = _collect_spans(parsed)-- assert spans == [("tool_call: query-trends", [{"tool": "query-trends", "input": trends_query}])]---def test_exec_commands_wrapping_no_inner_tool_stay_under_the_raw_name() -> None:- parsed = ParsedLog(- generations=[- GenerationDescriptor(- output_content=[- {- "type": "tool_use",- "id": "1",- "name": "mcp__posthog__exec",- "input": {"command": "schema query-trends series"},- }- ],- )- ]- )-- spans = _collect_spans(parsed)-- assert spans == [("tool_call: exec", [{"tool": "exec", "input": {"command": "schema query-trends series"}}])]---def test_tool_call_spans_are_split_when_one_agent_message_has_multiple_calls() -> None:- parsed = ParsedLog(- generations=[- GenerationDescriptor(- output_content=[- {"type": "tool_use", "id": "1", "name": "query-retention", "input": {}},- {"type": "tool_use", "id": "2", "name": "execute-sql", "input": {}},- ],- )- ]- )-- spans = _collect_spans(parsed)-- assert spans == [- ("tool_call: query-retention", [{"tool": "query-retention", "input": {}}]),- ("tool_call: execute-sql", [{"tool": "execute-sql", "input": {}}]),- ]---def _collect_spans(parsed: ParsedLog) -> list[tuple[str, Any]]:- collected: list[tuple[str, Any]] = []-- class _Span:- def __init__(self, name: str) -> None:- self.name = name-- def log(self, *, input: Any = None, output: Any = None, metadata: Any = None) -> None:- if input is not None:- collected.append((self.name, input))-- class _Hooks(NullCaseHooks):- @contextmanager- def start_span(self, name: str, kind: Any) -> Iterator[_Span]:- yield _Span(name)-- base._log_conversation_spans(_Hooks(), parsed)- return collected--def test_sandboxed_eval_run_adds_exit_code_scorer(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:monkeypatch.setattr(base, "build_case_dir", MagicMock(return_value=tmp_path))ctx = MagicMock(posthog_client=None, case_filter=None)@@ -362,8 +277,6 @@ def test_sandboxed_eval_run_adds_exit_code_scorer(monkeypatch: pytest.MonkeyPatccases=[],scorers=[custom_scorer],ctx=ctx,- is_public=False,- no_send_logs=True,)assert isinstance(run.active_scorers[0], ExitCodeZero)@@ -375,28 +288,22 @@ def test_sandboxed_eval_run_adds_exit_code_scorer(monkeypatch: pytest.MonkeyPatccases=[],scorers=[ExitCodeZero()],ctx=ctx,- is_public=False,- no_send_logs=True,)@pytest.mark.parametrize(- "suite_kind,enrolled,no_send_logs,send_to_posthog,posthog_enabled,configuration,http_status,delivery_state",+ "suite_kind,enrolled,send_to_posthog,configuration,http_status,delivery_state",[- ("sandboxed", True, False, None, True, "valid", 200, "ready"),- ("sandboxed", True, False, None, True, "valid", 403, "ready"),- ("sandboxed", True, False, None, True, "invalid", 200, "ready"),- ("sandboxed", True, False, None, True, "absent", 200, "ready"),- ("sandboxed", True, True, None, False, "valid", 200, "ready"),- ("sandboxed", False, False, None, True, "valid", 200, "ready"),- ("one-shot", True, False, None, True, "valid", 200, "ready"),- ("sandboxed", True, False, None, True, "valid", 200, "unavailable"),- ("sandboxed", True, False, None, True, "valid", 200, "closed"),- ("sandboxed", True, True, True, True, "valid", 200, "ready"),- ("one-shot", True, True, True, True, "valid", 200, "ready"),- ("sandboxed", True, False, False, False, "valid", 200, "ready"),- ("sandboxed", True, True, True, True, "absent", 200, "ready"),- ("sandboxed", False, True, True, True, "valid", 200, "ready"),+ ("sandboxed", True, True, "valid", 200, "ready"),+ ("sandboxed", True, True, "valid", 403, "ready"),+ ("sandboxed", True, True, "invalid", 200, "ready"),+ ("sandboxed", True, True, "absent", 200, "ready"),+ ("sandboxed", True, False, "valid", 200, "ready"),+ ("sandboxed", False, True, "valid", 200, "ready"),+ ("one-shot", True, True, "valid", 200, "ready"),+ ("one-shot", True, False, "valid", 200, "ready"),+ ("sandboxed", True, True, "valid", 200, "unavailable"),+ ("sandboxed", True, True, "valid", 200, "closed"),],)@pytest.mark.asyncio@@ -406,9 +313,7 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repocapsys: pytest.CaptureFixture[str],suite_kind: Literal["sandboxed", "one-shot"],enrolled: bool,- no_send_logs: bool,- send_to_posthog: bool | None,- posthog_enabled: bool,+ send_to_posthog: bool,configuration: Literal["valid", "invalid", "absent"],http_status: int,delivery_state: Literal["ready", "unavailable", "closed"],@@ -480,8 +385,6 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repocases=[SandboxedEvalCase(name="example-case", prompt="Run the task")],scorers=[],ctx=ctx,- is_public=not no_send_logs,- no_send_logs=no_send_logs,send_to_posthog=send_to_posthog,offline_suite=OfflineEvalSuite(key="example/suite",@@ -503,15 +406,14 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repoawait delivery.drain()engine.run_experiment.assert_awaited_once()engine_spec = engine.run_experiment.call_args.args[0]- assert engine_spec.no_send_logs is no_send_logs- assert engine_spec.record_scorer_results is (enrolled and posthog_enabled and configuration == "valid")+ assert engine_spec.record_scorer_results is (enrolled and send_to_posthog and configuration == "valid")assert reporter.mean_score() == 1.0reporter.print_final_summary([], exit_code=0, fail_under=None, duration_seconds=0)output = capsys.readouterr().out- assert "Braintrust: https://braintrust.example.com/experiment/test" in output+ assert "Braintrust:" not in outputassert "exit_code_zero: 100.0%" in outputassert not (tmp_path / "offline-pilot" / "latest" / "posthog-offline-upload.json").exists()- if enrolled and posthog_enabled and configuration == "valid" and delivery_state == "ready":+ if enrolled and send_to_posthog and configuration == "valid" and delivery_state == "ready":declaration = json.loads(requests[0].content)assert declaration["name"] == "offline-pilot"assert declaration["expected_item_count"] == 1@@ -540,7 +442,7 @@ async def test_sandboxed_eval_publishes_existing_results_without_disrupting_repo)assert reason in outputassert "Results were not saved for retry." in output- elif enrolled and posthog_enabled:+ elif enrolled and send_to_posthog:assert "PostHog offline: disabled:" in outputassert "POSTHOG_OFFLINE_EVAL_" in outputelse:@@ -571,8 +473,7 @@ async def test_suite_enrolls_every_scorer_in_offline_publishing(monkeypatch.setattr(module, "fetch_search_intent_prompt", MagicMock())run = AsyncMock()runner_kind = "OneShot" if suite_kind == SuiteKind.ONE_SHOT else "Sandboxed"- runner_visibility = "Private" if discovered.no_send_logs else "Public"- monkeypatch.setattr(module, f"{runner_kind}{runner_visibility}Eval", run)+ monkeypatch.setattr(module, f"{runner_kind}Eval", run)await discovered.fn(MagicMock(skill_delivery="bundled"))@@ -648,8 +549,6 @@ async def test_sandboxed_eval_skill_delivery_sets_the_agent_origin(cases=[case] if has_original else [],scorers=[],ctx=ctx,- is_public=False,- no_send_logs=True,)launch = AsyncMock(return_value=base.EvalCaseResult(artifacts=AgentArtifacts(exit_code=0)))monkeypatch.setattr(base, "run_eval_case", launch)@@ -830,8 +729,6 @@ async def test_case_setup_retries_use_fresh_data_and_report_one_outcome(cases=[case],scorers=[],ctx=ctx,- is_public=False,- no_send_logs=True,)launch = AsyncMock(side_effect=failure if stage == "agent" else None,@@ -888,9 +785,7 @@ async def test_kernel_cleanup_preserves_case_outcome_when_progress_fails(ctx.provider_strategy.keeps_sandboxes.return_value = Falsecase = SandboxedEvalCase(name="case", prompt="Run the task")launch = AsyncMock()- run = base._SandboxedEvalRun(- "cleanup", [case], [], ctx, is_public=False, no_send_logs=True, case_runner=launch if custom_runner else None- )+ run = base._SandboxedEvalRun("cleanup", [case], [], ctx, case_runner=launch if custom_runner else None)context = CustomPromptSandboxContext(team_id=1, user_id=2)monkeypatch.setattr(run, "_prepare_case", AsyncMock(return_value=base._SandboxedCaseSetup(context=context, seed={}))@@ -935,7 +830,7 @@ async def test_sandboxed_trials_keep_separate_logs_and_trace_roots(monkeypatch.setattr(base, "build_case_dir", lambda *_: tmp_path)ctx = MagicMock(posthog_client=None, reporter=ProgressReporter(total_suites=1))ctx.trials = 3- run = base._SandboxedEvalRun("trials", [], [], ctx, is_public=False, no_send_logs=True)+ run = base._SandboxedEvalRun("trials", [], [], ctx)run.posthog_client = MagicMock()results = []for index in range(3):diff --git a/products/posthog_ai/eval_harness/test/test_suite_requirements.py b/products/posthog_ai/eval_harness/test/test_suite_requirements.pyindex 008f9a7dd236..d27db4c64577 100644--- a/products/posthog_ai/eval_harness/test/test_suite_requirements.py+++ b/products/posthog_ai/eval_harness/test/test_suite_requirements.py@@ -3,7 +3,6 @@ from __future__ import annotationsimport pytestfrom unittest.mock import Mock-from products.posthog_ai.eval_harness.engines.registry import resolve_enginefrom products.posthog_ai.eval_harness.harness import lifecyclefrom products.posthog_ai.eval_harness.harness.cli import parse_argsfrom products.posthog_ai.eval_harness.harness.context import EvalContext@@ -69,42 +68,42 @@ def test_every_kind_declares_a_closed_infra_set() -> None:pytest.param({SuiteKind.ONE_SHOT},"claude",- {"BRAINTRUST_API_KEY": "x", "LLM_GATEWAY_ANTHROPIC_API_KEY": "x"},+ {"LLM_GATEWAY_ANTHROPIC_API_KEY": "x"},(),id="one_shot_run_needs_no_sandbox_credentials",),pytest.param({SuiteKind.ONE_SHOT},"claude",- {"BRAINTRUST_API_KEY": "x"},+ {},("LLM_GATEWAY_ANTHROPIC_API_KEY",),id="one_shot_run_still_needs_the_generation_key",),pytest.param({SuiteKind.SANDBOXED},"claude",- {"BRAINTRUST_API_KEY": "x", "LLM_GATEWAY_ANTHROPIC_API_KEY": "x"},+ {"LLM_GATEWAY_ANTHROPIC_API_KEY": "x"},("SANDBOX_JWT_PRIVATE_KEY",),id="sandboxed_run_needs_the_jwt_key",),pytest.param({SuiteKind.SANDBOXED, SuiteKind.ONE_SHOT},"claude",- {"BRAINTRUST_API_KEY": "x"},+ {},("SANDBOX_JWT_PRIVATE_KEY", "LLM_GATEWAY_ANTHROPIC_API_KEY"),id="mixed_run_reports_each_missing_variable_once",),pytest.param({SuiteKind.ONE_SHOT},"codex",- {"BRAINTRUST_API_KEY": "x", "LLM_GATEWAY_ANTHROPIC_API_KEY": "x"},+ {"LLM_GATEWAY_ANTHROPIC_API_KEY": "x"},(),id="codex_requirement_only_applies_to_sandboxed_runs",),pytest.param({SuiteKind.SANDBOXED},"codex",- {"BRAINTRUST_API_KEY": "x", "LLM_GATEWAY_ANTHROPIC_API_KEY": "x", "SANDBOX_JWT_PRIVATE_KEY": "x"},+ {"LLM_GATEWAY_ANTHROPIC_API_KEY": "x", "SANDBOX_JWT_PRIVATE_KEY": "x"},("LLM_GATEWAY_OPENAI_API_KEY",),id="sandboxed_codex_run_needs_the_openai_key",),@@ -135,34 +134,8 @@ def test_validate_eval_env_by_kind(assert name not in message[email protected]("no_send_logs", [False, True])-def test_validate_eval_env_reports_missing_engine_env(monkeypatch: pytest.MonkeyPatch, no_send_logs: bool) -> None:- for name in ALL_ENV_VARS:- monkeypatch.delenv(name, raising=False)- monkeypatch.setenv("LLM_GATEWAY_ANTHROPIC_API_KEY", "x")-- engine_env = resolve_engine().required_env(no_send_logs=no_send_logs)- if no_send_logs:- validate_eval_env("claude", kinds={SuiteKind.ONE_SHOT}, engine_env=engine_env)- return-- with pytest.raises(PreflightError) as exc_info:- validate_eval_env("claude", kinds={SuiteKind.ONE_SHOT}, engine_env=engine_env)- assert " - BRAINTRUST_API_KEY: records experiments and scores to Braintrust" in str(exc_info.value)--- "no_send_logs, has_key",- [- pytest.param([True], False, id="posthog_only_without_braintrust_key"),- pytest.param([False], False, id="braintrust_reporting_without_key"),- pytest.param([True, False], False, id="mixed_reporting_without_key"),- pytest.param([True, False], True, id="mixed_reporting_with_key"),- ],-)-def test_harness_checks_selected_reporting_before_starting_infrastructure(- monkeypatch: pytest.MonkeyPatch, no_send_logs: list[bool], has_key: bool-) -> None:[email protected]("has_key", [False, True])+def test_harness_runs_without_braintrust_credentials(monkeypatch: pytest.MonkeyPatch, has_key: bool) -> None:monkeypatch.delenv("BRAINTRUST_API_KEY", raising=False)monkeypatch.delenv("EXPORT_EVAL_RESULTS", raising=False)monkeypatch.setenv("LLM_GATEWAY_ANTHROPIC_API_KEY", "test-only-key")@@ -180,9 +153,8 @@ def test_harness_checks_selected_reporting_before_starting_infrastructure(fn_name=f"eval_test_{index}",fn=run_suite,kind=SuiteKind.ONE_SHOT,- no_send_logs=disabled,)- for index, disabled in enumerate(no_send_logs)+ for index in range(2)]monkeypatch.setattr(lifecycle, "discover_suites", lambda _: suites)build = Mock()@@ -193,11 +165,5 @@ def test_harness_checks_selected_reporting_before_starting_infrastructure(harness._demo_data = Mock()monkeypatch.setattr(harness, "_bootstrap", bootstrap)- if has_key or all(no_send_logs):- assert harness.run() == 0- assert len(completed) == len(suites)- else:- with pytest.raises(PreflightError, match="BRAINTRUST_API_KEY"):- harness.run()- build.assert_not_called()- assert not completed+ assert harness.run() == 0+ assert len(completed) == len(suites)diff --git a/products/posthog_ai/evals/ci/eval_basic.py b/products/posthog_ai/evals/ci/eval_basic.pyindex f7edf05de098..18246c3c7339 100644--- a/products/posthog_ai/evals/ci/eval_basic.py+++ b/products/posthog_ai/evals/ci/eval_basic.py@@ -6,12 +6,12 @@ To run:Example eval case pattern:1. Define a SandboxedEvalCase with prompt + expected outcomes- 2. Call SandboxedPublicEval (or SandboxedPrivateEval) with scorers+ 2. Call SandboxedEval (or SandboxedEval) with scorers"""from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.cli_mcp.scorers import CalledTargetTool, SurfacedGeneratedAppUrl@@ -36,7 +36,7 @@ async def eval_bugfix(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-bugfix-cli",offline_suite=offline_suite("ci/eval_basic::eval_bugfix",@@ -89,7 +89,7 @@ async def eval_app_link_generation(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-app-link-generation-cli",offline_suite=offline_suite("ci/eval_basic::eval_app_link_generation",diff --git a/products/posthog_ai/evals/cli_mcp/eval_render_ui.py b/products/posthog_ai/evals/cli_mcp/eval_render_ui.pyindex d58fb71145fc..90c712bd448c 100644--- a/products/posthog_ai/evals/cli_mcp/eval_render_ui.py+++ b/products/posthog_ai/evals/cli_mcp/eval_render_ui.py@@ -43,7 +43,7 @@ To run a single eval:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.cli_mcp.scorers import (@@ -95,7 +95,7 @@ async def eval_renders_entity_ui(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-renders-entity-ui-cli",offline_suite=offline_suite("cli_mcp/eval_render_ui::eval_renders_entity_ui",@@ -130,7 +130,7 @@ async def eval_no_render_for_query_results(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-no-render-for-query-cli",offline_suite=offline_suite("cli_mcp/eval_render_ui::eval_no_render_for_query_results",diff --git a/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py b/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.pyindex 7126dda1e16f..bdf46cfe4e74 100644--- a/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py+++ b/products/posthog_ai/evals/cli_mcp/eval_skill_distribution.py@@ -3,7 +3,7 @@ from __future__ import annotationsfrom collections.abc import Callablefrom typing import Any-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.cli import SkillDeliveryfrom products.posthog_ai.eval_harness.harness.context import EvalContext@@ -172,7 +172,7 @@ async def eval_skill_distribution(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-skill-distribution-cli",offline_suite=offline_suite("cli_mcp/eval_skill_distribution::eval_skill_distribution",diff --git a/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py b/products/posthog_ai/evals/cli_mcp/eval_skill_usage.pyindex db1a4eb61e0f..c5f8876c033d 100644--- a/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py+++ b/products/posthog_ai/evals/cli_mcp/eval_skill_usage.py@@ -3,7 +3,7 @@ from __future__ import annotationsfrom collections.abc import Callablefrom typing import Any-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.cli import SkillDeliveryfrom products.posthog_ai.eval_harness.harness.context import EvalContext@@ -137,7 +137,7 @@ async def eval_skill_usage(ctx: EvalContext) -> None:))- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-skill-usage-cli",offline_suite=offline_suite("cli_mcp/eval_skill_usage::eval_skill_usage",diff --git a/products/posthog_ai/evals/cli_mcp/eval_workflow.py b/products/posthog_ai/evals/cli_mcp/eval_workflow.pyindex 688be90b6858..fb4433ef4acf 100644--- a/products/posthog_ai/evals/cli_mcp/eval_workflow.py+++ b/products/posthog_ai/evals/cli_mcp/eval_workflow.py@@ -26,7 +26,7 @@ To run a single eval:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.cli_mcp.scorers import (@@ -68,7 +68,7 @@ async def eval_typo_recovery(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-typo-recovery-cli",offline_suite=offline_suite("cli_mcp/eval_workflow::eval_typo_recovery",@@ -105,7 +105,7 @@ async def eval_schema_drilldown(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-schema-drilldown-cli",offline_suite=offline_suite("cli_mcp/eval_workflow::eval_schema_drilldown",@@ -131,7 +131,7 @@ async def eval_search_first(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-search-first-cli",offline_suite=offline_suite("cli_mcp/eval_workflow::eval_search_first",@@ -165,7 +165,7 @@ async def eval_info_before_call(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-info-before-call-cli",offline_suite=offline_suite("cli_mcp/eval_workflow::eval_info_before_call",@@ -205,7 +205,7 @@ async def eval_json_for_post_processing(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-json-postprocess-cli",offline_suite=offline_suite("cli_mcp/eval_workflow::eval_json_for_post_processing",@@ -235,7 +235,7 @@ async def eval_verify_event_before_query(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cli-mcp-verify-event-cli",offline_suite=offline_suite("cli_mcp/eval_workflow::eval_verify_event_before_query",diff --git a/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py b/products/posthog_ai/evals/cohorts/eval_cohort_from_query.pyindex 7b434dd3a7cf..23dd0cbb4fe8 100644--- a/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py+++ b/products/posthog_ai/evals/cohorts/eval_cohort_from_query.py@@ -18,7 +18,7 @@ To run::from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -58,7 +58,7 @@ async def eval_cohort_from_query(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-cohorts-from-query-cli",offline_suite=offline_suite("cohorts/eval_cohort_from_query::eval_cohort_from_query",diff --git a/products/posthog_ai/evals/data_warehouse/eval_information_schema.py b/products/posthog_ai/evals/data_warehouse/eval_information_schema.pyindex 136827813293..5104a02d8a22 100644--- a/products/posthog_ai/evals/data_warehouse/eval_information_schema.py+++ b/products/posthog_ai/evals/data_warehouse/eval_information_schema.py@@ -29,7 +29,7 @@ To run a single eval::from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.data_warehouse.scorers import (@@ -109,7 +109,7 @@ async def eval_dw_discovery(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-warehouse-discovery-cli",offline_suite=offline_suite("data_warehouse/eval_information_schema::eval_dw_discovery",@@ -175,7 +175,7 @@ async def eval_dw_value_retrieval(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-warehouse-retrieval-cli",offline_suite=offline_suite("data_warehouse/eval_information_schema::eval_dw_value_retrieval",@@ -241,7 +241,7 @@ async def eval_dw_relationships(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-warehouse-relationships-cli",offline_suite=offline_suite("data_warehouse/eval_information_schema::eval_dw_relationships",@@ -293,7 +293,7 @@ async def eval_dw_table_relevancy(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-warehouse-relevancy-cli",offline_suite=offline_suite("data_warehouse/eval_information_schema::eval_dw_table_relevancy",diff --git a/products/posthog_ai/evals/error_tracking/eval_events_sampling.py b/products/posthog_ai/evals/error_tracking/eval_events_sampling.pyindex 70965ed042db..f33e001a87b5 100644--- a/products/posthog_ai/evals/error_tracking/eval_events_sampling.py+++ b/products/posthog_ai/evals/error_tracking/eval_events_sampling.py@@ -22,7 +22,7 @@ from __future__ import annotationsfrom typing import Any-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -106,7 +106,7 @@ async def eval_events_sampling(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-error-tracking-events-sampling-cli",offline_suite=offline_suite("error_tracking/eval_events_sampling::eval_events_sampling",diff --git a/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py b/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.pyindex 3e82533e9c17..cfa023d9f8f9 100644--- a/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py+++ b/products/posthog_ai/evals/error_tracking/eval_issue_drilldown.py@@ -29,7 +29,7 @@ from __future__ import annotationsfrom typing import Any-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -120,7 +120,7 @@ async def eval_issue_drilldown(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-error-tracking-issue-drilldown-cli",offline_suite=offline_suite("error_tracking/eval_issue_drilldown::eval_issue_drilldown",diff --git a/products/posthog_ai/evals/error_tracking/eval_issues_list.py b/products/posthog_ai/evals/error_tracking/eval_issues_list.pyindex a016b2a02cd5..b965336a54c2 100644--- a/products/posthog_ai/evals/error_tracking/eval_issues_list.py+++ b/products/posthog_ai/evals/error_tracking/eval_issues_list.py@@ -29,7 +29,7 @@ from __future__ import annotationsfrom typing import Any-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -85,7 +85,7 @@ async def eval_issues_list(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-error-tracking-issues-list-cli",offline_suite=offline_suite("error_tracking/eval_issues_list::eval_issues_list",diff --git a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.pyindex 867db0ffcd27..e2a35db61226 100644--- a/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py+++ b/products/posthog_ai/evals/experiments/eval_bias_uneven_split.py@@ -29,7 +29,7 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.experiments.scorers import CitesDiagnosticGroup, SurfacesAllFindings@@ -41,8 +41,6 @@ from products.posthog_ai.evals.experiments.seeders import ()from products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-async def eval_bias_uneven_split(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -108,8 +106,7 @@ async def eval_bias_uneven_split(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-diagnose-bias-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_create_control_variant.py b/products/posthog_ai/evals/experiments/eval_create_control_variant.pyindex e3993092b539..c0bbb91cbd08 100644--- a/products/posthog_ai/evals/experiments/eval_create_control_variant.py+++ b/products/posthog_ai/evals/experiments/eval_create_control_variant.py@@ -30,14 +30,12 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCallfrom products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-async def eval_create_control_variant(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -57,8 +55,7 @@ async def eval_create_control_variant(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-create-control-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_empty_experiment.py b/products/posthog_ai/evals/experiments/eval_empty_experiment.pyindex 474dd56e0acc..4d2e98241a4d 100644--- a/products/posthog_ai/evals/experiments/eval_empty_experiment.py+++ b/products/posthog_ai/evals/experiments/eval_empty_experiment.py@@ -42,7 +42,7 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.experiments.scorers import CitesDiagnosticGroup@@ -54,8 +54,6 @@ from products.posthog_ai.evals.experiments.seeders import ()from products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-async def eval_empty_experiment(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -140,8 +138,7 @@ async def eval_empty_experiment(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-diagnose-empty-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py b/products/posthog_ai/evals/experiments/eval_interpretation_traps.pyindex 37204d92d4a1..24ffe59769d7 100644--- a/products/posthog_ai/evals/experiments/eval_interpretation_traps.py+++ b/products/posthog_ai/evals/experiments/eval_interpretation_traps.py@@ -41,7 +41,7 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.experiments.scorers import AdvisesAgainstShipping, CitesDiagnosticGroup@@ -52,8 +52,6 @@ from products.posthog_ai.evals.experiments.seeders import ()from products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-async def eval_interpretation_traps(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -162,8 +160,7 @@ async def eval_interpretation_traps(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-diagnose-interpretation-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.pyindex e1001365b69d..f63940a77678 100644--- a/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py+++ b/products/posthog_ai/evals/experiments/eval_lifecycle_skills.py@@ -30,7 +30,7 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCall@@ -42,8 +42,6 @@ from products.posthog_ai.evals.experiments.scorers import (from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-# Keep the cases in one experiment so their scores can be compared together.async def eval_lifecycle_skills(ctx: EvalContext) -> None:@@ -85,8 +83,7 @@ async def eval_lifecycle_skills(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-lifecycle-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.pyindex b54d76c1bb31..524d61024c78 100644--- a/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py+++ b/products/posthog_ai/evals/experiments/eval_metric_schema_discovery.py@@ -15,7 +15,7 @@ recovering on a later call does not flip the score.from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall@@ -27,8 +27,6 @@ from products.posthog_ai.evals.experiments.scorers import (from products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-async def eval_metric_schema_discovery(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -54,8 +52,7 @@ async def eval_metric_schema_discovery(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-metric-schema-discovery-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_midrun_changes.py b/products/posthog_ai/evals/experiments/eval_midrun_changes.pyindex 9062c588fac6..92cffbb25146 100644--- a/products/posthog_ai/evals/experiments/eval_midrun_changes.py+++ b/products/posthog_ai/evals/experiments/eval_midrun_changes.py@@ -46,7 +46,7 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.experiments.scorers import (@@ -63,8 +63,6 @@ from products.posthog_ai.evals.experiments.seeders import ()from products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-async def eval_midrun_changes(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -153,8 +151,7 @@ async def eval_midrun_changes(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-diagnose-midrun-changes-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.pyindex db3f2b2fbd23..868e50b9eaf9 100644--- a/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py+++ b/products/posthog_ai/evals/experiments/eval_numbers_vs_sql.py@@ -38,15 +38,13 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.experiments.scorers import CitesDiagnosticGroupfrom products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-async def eval_numbers_vs_sql(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -155,8 +153,7 @@ async def eval_numbers_vs_sql(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-diagnose-numbers-vs-sql-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_rollout_skill.py b/products/posthog_ai/evals/experiments/eval_rollout_skill.pyindex 021026a3d1ee..4d2ffd7f8517 100644--- a/products/posthog_ai/evals/experiments/eval_rollout_skill.py+++ b/products/posthog_ai/evals/experiments/eval_rollout_skill.py@@ -8,7 +8,7 @@ cheap to reproduce while iterating on the skill's description.Maps to stress-test finding #7 (skills rarely auto-load), specificallythe variant-split rows (2.5, 5.3, 5.4, 5.5) in STEP_1_HARNESS.md. Uses-``SandboxedPrivateEval`` with ``send_to_posthog=True`` to publish results+``SandboxedEval`` with ``send_to_posthog=True`` to publish resultsonly to PostHog. The harness still requires a Braintrust API key for itsevaluation engine, but Braintrust result uploads remain disabled.@@ -18,15 +18,13 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.retrieval.scorers import SkillLoaded-NO_SEND_LOGS = True-SKILL_NAME = "configuring-experiment-rollout"@@ -41,8 +39,7 @@ async def eval_rollout_skill(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-rollout-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.pyindex 3673915aaf4b..79ef03b29c55 100644--- a/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py+++ b/products/posthog_ai/evals/experiments/eval_shared_metric_validation.py@@ -7,7 +7,7 @@ and (2) reach the correct verdict citing grounded discrepancies.from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall@@ -15,8 +15,6 @@ from products.posthog_ai.evals.experiments.scorers import SharedMetricValidationfrom products.posthog_ai.evals.experiments.seeders import SHARED_METRIC_NAME, seed_shared_metric_purchase_countfrom products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-async def eval_shared_metric_validation(ctx: EvalContext) -> None:cases: list[SandboxedEvalCase] = [@@ -51,8 +49,7 @@ async def eval_shared_metric_validation(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-shared-metric-validation-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.pyindex d748fd4858ee..5201c856f90c 100644--- a/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py+++ b/products/posthog_ai/evals/experiments/eval_unknown_events_confirmation.py@@ -34,7 +34,7 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCall@@ -42,8 +42,6 @@ from products.posthog_ai.evals.experiments.scorers import AskedForConfirmationfrom products.posthog_ai.evals.experiments.seeders import ROLLOUT_EXPERIMENT_NAME, seed_running_experimentfrom products.posthog_ai.evals.offline_config import offline_suite-NO_SEND_LOGS = True-# Event name chosen to be obviously fictional so the agent can't plausibly# justify it as a typo of a real PostHog event in the seeded team.UNKNOWN_EVENT_NAME = "totally_made_up_event_qzx"@@ -67,8 +65,7 @@ async def eval_unknown_events_confirmation(ctx: EvalContext) -> None:),]- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(experiment_name="sandboxed-experiments-unknown-events-cli",send_to_posthog=True,offline_suite=offline_suite(diff --git a/products/posthog_ai/evals/experiments/scorers.py b/products/posthog_ai/evals/experiments/scorers.pyindex c369dec2ac6a..d0c1f90192aa 100644--- a/products/posthog_ai/evals/experiments/scorers.py+++ b/products/posthog_ai/evals/experiments/scorers.py@@ -57,13 +57,12 @@ from __future__ import annotationsfrom typing import Any-from autoevals.llm import LLMClassifier-from products.posthog_ai.eval_harness.scorers import BINARY_CHOICE_SCORES, AsyncOnlyScorerMixin, JudgedScorer, LogParserfrom products.posthog_ai.eval_harness.scorers.contract import Score, Scorer+from products.posthog_ai.eval_harness.scorers.judged import LocalLLMClassifier# Not the shared JUDGE_MODEL: these judges were baselined on gpt-4.1, and-# swapping the model resets their Braintrust history.+# swapping the model changes the scoring baseline._JUDGE_MODEL = "gpt-4.1"@@ -123,7 +122,7 @@ class DuplicateUniqueFlagKey(AsyncOnlyScorerMixin, Scorer):self._label = name# LLM judge for the no-call branch. Instantiated lazily to avoid# paying its setup cost on cases that take the deterministic path.- self._refusal_judge = LLMClassifier(+ self._refusal_judge = LocalLLMClassifier(name=f"_internal_{name}_refusal",prompt_template="""You are judging whether an agent correctly refused to duplicate an experiment because it lacked a required unique feature_flag_key.diff --git a/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py b/products/posthog_ai/evals/product_analytics/eval_chart_not_table.pyindex be88effa6500..19ca89093648 100644--- a/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py+++ b/products/posthog_ai/evals/product_analytics/eval_chart_not_table.py@@ -24,7 +24,7 @@ Run on the latest Claude Opus and GPT Sol models, one runtime per run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -92,7 +92,7 @@ async def eval_chart_not_table(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-chart-not-table-cli",offline_suite=offline_suite("product_analytics/eval_chart_not_table::eval_chart_not_table",diff --git a/products/posthog_ai/evals/product_analytics/eval_funnel.py b/products/posthog_ai/evals/product_analytics/eval_funnel.pyindex 7c21408e64c8..70a0cca3725e 100644--- a/products/posthog_ai/evals/product_analytics/eval_funnel.py+++ b/products/posthog_ai/evals/product_analytics/eval_funnel.py@@ -11,7 +11,7 @@ from posthog.schema import (AssistantFunnelsQuery,)-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall, RequiredToolCall@@ -586,7 +586,7 @@ async def eval_funnel(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-funnels",offline_suite=offline_suite("product_analytics/eval_funnel::eval_funnel",diff --git a/products/posthog_ai/evals/product_analytics/eval_retention.py b/products/posthog_ai/evals/product_analytics/eval_retention.pyindex 9cfb1d1f797a..37f876791ff0 100644--- a/products/posthog_ai/evals/product_analytics/eval_retention.py+++ b/products/posthog_ai/evals/product_analytics/eval_retention.py@@ -4,7 +4,7 @@ from __future__ import annotationsfrom posthog.schema import AssistantRetentionEventsNode, AssistantRetentionFilter, AssistantRetentionQuery-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall@@ -115,7 +115,7 @@ async def eval_retention(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-retention-cli",offline_suite=offline_suite("product_analytics/eval_retention::eval_retention",diff --git a/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py b/products/posthog_ai/evals/product_analytics/eval_schema_discovery.pyindex c0d1c1522856..c088a7b09e70 100644--- a/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py+++ b/products/posthog_ai/evals/product_analytics/eval_schema_discovery.py@@ -18,7 +18,7 @@ from __future__ import annotationsfrom typing import Any-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall@@ -83,7 +83,7 @@ async def eval_schema_discovery(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-schema-discovery-cli",offline_suite=offline_suite("product_analytics/eval_schema_discovery::eval_schema_discovery",diff --git a/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py b/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.pyindex c024a7fd52c5..4a79ddea91bd 100644--- a/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py+++ b/products/posthog_ai/evals/product_analytics/eval_sql_shaped_routing.py@@ -12,7 +12,7 @@ see ``retrieval/eval_system_table_search.py`` — and isn't duplicated here.from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -118,7 +118,7 @@ async def eval_sql_shaped_typed_query_routing(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-sql-shaped-typed-query-routing-cli",offline_suite=offline_suite("product_analytics/eval_sql_shaped_routing::eval_sql_shaped_typed_query_routing",@@ -153,7 +153,7 @@ async def eval_sql_shaped_sql_controls(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-sql-shaped-sql-controls-cli",offline_suite=offline_suite("product_analytics/eval_sql_shaped_routing::eval_sql_shaped_sql_controls",diff --git a/products/posthog_ai/evals/product_analytics/eval_trends.py b/products/posthog_ai/evals/product_analytics/eval_trends.pyindex 1fa92c1989f4..66c0ff515912 100644--- a/products/posthog_ai/evals/product_analytics/eval_trends.py+++ b/products/posthog_ai/evals/product_analytics/eval_trends.py@@ -14,7 +14,7 @@ from posthog.schema import (TrendsFormulaNode,)-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall@@ -300,7 +300,7 @@ async def eval_trends(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-trends",offline_suite=offline_suite("product_analytics/eval_trends::eval_trends",diff --git a/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py b/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.pyindex d4c51ef0e614..80d2ab4e609c 100644--- a/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py+++ b/products/posthog_ai/evals/product_analytics/eval_typed_runner_lock_in.py@@ -28,7 +28,7 @@ score confounds routing with "I answered this already".from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -105,7 +105,7 @@ async def eval_typed_runner_no_lock_in_control(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-typed-runner-no-lock-in-control-cli",offline_suite=offline_suite("product_analytics/eval_typed_runner_lock_in::eval_typed_runner_no_lock_in_control",@@ -149,7 +149,7 @@ async def eval_typed_runner_lock_in(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-typed-runner-lock-in-cli",offline_suite=offline_suite("product_analytics/eval_typed_runner_lock_in::eval_typed_runner_lock_in",diff --git a/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py b/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.pyindex 0222167831f2..5c38e6727407 100644--- a/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py+++ b/products/posthog_ai/evals/product_analytics/eval_typed_runner_misses.py@@ -30,7 +30,7 @@ from __future__ import annotationsfrom typing import Unpack-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall@@ -163,7 +163,7 @@ async def eval_typed_runner_misses(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-typed-runner-misses-cli",offline_suite=offline_suite("product_analytics/eval_typed_runner_misses::eval_typed_runner_misses",@@ -205,7 +205,7 @@ async def eval_justified_sql_controls(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-justified-sql-controls-cli",offline_suite=offline_suite("product_analytics/eval_typed_runner_misses::eval_justified_sql_controls",diff --git a/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py b/products/posthog_ai/evals/product_analytics/eval_visualization_choice.pyindex 2073fc7b27bf..970178238fbd 100644--- a/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py+++ b/products/posthog_ai/evals/product_analytics/eval_visualization_choice.py@@ -16,7 +16,7 @@ from __future__ import annotationsfrom typing import Any-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall, RequiredToolCall@@ -103,7 +103,7 @@ async def eval_visualization_choice(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-visualization-choice-cli",offline_suite=offline_suite("product_analytics/eval_visualization_choice::eval_visualization_choice",@@ -154,7 +154,7 @@ async def eval_saved_insight_display(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-saved-insight-display-cli",offline_suite=offline_suite("product_analytics/eval_visualization_choice::eval_saved_insight_display",diff --git a/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py b/products/posthog_ai/evals/retrieval/eval_insight_retrieval.pyindex 4c4ca5bfa3f3..b732f18298b6 100644--- a/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py+++ b/products/posthog_ai/evals/retrieval/eval_insight_retrieval.py@@ -19,7 +19,7 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.seeders.insight import seed_insight_noise@@ -71,7 +71,7 @@ async def eval_insight_retrieval(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-retrieval-cli",offline_suite=offline_suite("retrieval/eval_insight_retrieval::eval_insight_retrieval",diff --git a/products/posthog_ai/evals/retrieval/eval_system_table_search.py b/products/posthog_ai/evals/retrieval/eval_system_table_search.pyindex adde6d5e19bd..2c04c75398f4 100644--- a/products/posthog_ai/evals/retrieval/eval_system_table_search.py+++ b/products/posthog_ai/evals/retrieval/eval_system_table_search.py@@ -30,7 +30,7 @@ To run::from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.seeders.insight import seed_insight_noise@@ -72,7 +72,7 @@ async def eval_system_table_search(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-system-table-search-cli",offline_suite=offline_suite("retrieval/eval_system_table_search::eval_system_table_search",diff --git a/products/posthog_ai/evals/sql/eval_sql.py b/products/posthog_ai/evals/sql/eval_sql.pyindex cea50c41769a..a0b995b5f0dd 100644--- a/products/posthog_ai/evals/sql/eval_sql.py+++ b/products/posthog_ai/evals/sql/eval_sql.py@@ -10,7 +10,7 @@ To run:from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import AnswerToolCallNot, NoToolCall@@ -371,7 +371,7 @@ WHERE properties.interests IS NOT NULL),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-sql-cli",offline_suite=offline_suite("sql/eval_sql::eval_sql",diff --git a/products/posthog_ai/evals/surveys/eval_surveys.py b/products/posthog_ai/evals/surveys/eval_surveys.pyindex 7214ac6a247c..10162061de76 100644--- a/products/posthog_ai/evals/surveys/eval_surveys.py+++ b/products/posthog_ai/evals/surveys/eval_surveys.py@@ -5,7 +5,7 @@ from __future__ import annotationsfrom collections.abc import Callablefrom typing import Any-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import NoToolCall@@ -269,7 +269,7 @@ async def eval_surveys(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-surveys-cli",offline_suite=offline_suite("surveys/eval_surveys::eval_surveys",diff --git a/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py b/products/posthog_ai/evals/taxonomic_filter/eval_event_match.pyindex 721285f7f3a4..c58a6c96a24f 100644--- a/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py+++ b/products/posthog_ai/evals/taxonomic_filter/eval_event_match.py@@ -17,7 +17,7 @@ The suite scores the model's likely events before the ingested-event check, so tdo not change the result. It shows at most `MAX_MATCHES` events, like the picker.No CI job runs this suite. Run it by hand, with AI_GATEWAY_URL and AI_GATEWAY_API_KEY set for the decision-model, and with the harness's own BRAINTRUST_API_KEY and LLM_GATEWAY_ANTHROPIC_API_KEY:+model, and with the harness's own LLM_GATEWAY_ANTHROPIC_API_KEY:hogli evals eval_event_matchSet EVENT_MATCH_PROMPT_VERSION to score an unpublished prompt version before moving its production label.Fetching a version needs POSTHOG_PERSONAL_API_KEY with read access to the PostHog project, and the suite fails without it.@@ -35,7 +35,7 @@ from posthog.taxonomic_search_intent.event_match import EVENT_MATCH_MODEL, MAX_Mfrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.taxonomic_filter.scorers import EVENT_MATCH_KEY, EventMatchFound, NoWrongEventMatch@@ -102,7 +102,7 @@ async def eval_event_match(ctx: EvalContext) -> None:"last_message": f"{case.prompt!r}: {suggested}",}- await OneShotPublicEval(+ await OneShotEval(experiment_name="taxonomic-filter-event-match",offline_suite=offline_suite("taxonomic_filter/eval_event_match::eval_event_match",diff --git a/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py b/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.pyindex cae15518e0c1..d338ed7c6e2b 100644--- a/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py+++ b/products/posthog_ai/evals/taxonomic_filter/eval_search_intent.py@@ -21,7 +21,7 @@ label moves to it, and expect the baseline to step when the prompt or the modelPublic rather than private: the value is the comparison across runs, and every case is synthetic.No CI job runs this suite. Run it by hand, with AI_GATEWAY_URL and AI_GATEWAY_API_KEY set for the-decision model, and with the harness's own BRAINTRUST_API_KEY and LLM_GATEWAY_ANTHROPIC_API_KEY:+decision model, and with the harness's own LLM_GATEWAY_ANTHROPIC_API_KEY:hogli evals eval_search_intenthogli evals eval_search_intent --eval email_in_events_tab@@ -45,7 +45,7 @@ from posthog.taxonomic_search_intent.prompt import fetch_search_intent_promptfrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.posthog_ai.evals.offline_config import offline_suitefrom products.posthog_ai.evals.taxonomic_filter.scorers import (SEARCH_INTENT_KEY,@@ -356,7 +356,7 @@ async def eval_search_intent(ctx: EvalContext) -> None:"last_message": f"{case.prompt!r} in {request.active_group_type}: {answer}",}- await OneShotPublicEval(+ await OneShotEval(experiment_name="ml-inference-search-intent",offline_suite=offline_suite("taxonomic_filter/eval_search_intent::eval_search_intent",diff --git a/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.py b/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.pyindex 29693b85e68c..62f9db0de23b 100644--- a/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.py+++ b/products/posthog_ai/evals/turn_suggestions/eval_turn_suggestions.py@@ -11,15 +11,13 @@ from products.posthog_ai.backend.turn_suggestions.judgment import build_judge_stfrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.engines.types import ScorerResultfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPrivateEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.posthog_ai.eval_harness.scorers.contract import Score, Scorerfrom products.posthog_ai.evals.offline_config import offline_suiteif TYPE_CHECKING:from products.posthog_ai.eval_harness.harness.context import EvalContext-NO_SEND_LOGS = True-SUITE_KIND = SuiteKind.ONE_SHOTBENCHMARK_CASES = {case.name: case for case in load_cases()}CASES = [@@ -73,8 +71,7 @@ async def eval_turn_suggestions(ctx: EvalContext) -> None:"error": result.error,}- await OneShotPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await OneShotEval(experiment_name="turn-suggestions",cases=[case.model_copy(update={"metadata": {**case.metadata, "threshold": threshold, "judge_model": model}})diff --git a/products/signals/backend/test/test_first_report_evals.py b/products/signals/backend/test/test_first_report_evals.pyindex 41919d476e93..2d8a6d616cd9 100644--- a/products/signals/backend/test/test_first_report_evals.py+++ b/products/signals/backend/test/test_first_report_evals.py@@ -123,7 +123,7 @@ class TestFirstReportScorers(TestCase):),}with TemporaryDirectory() as directory, patch.object(base, "build_case_dir", return_value=Path(directory)):- run = base._SandboxedEvalRun("scout", [], [], MagicMock(posthog_client=None), False, True)+ run = base._SandboxedEvalRun("scout", [], [], MagicMock(posthog_client=None))output = async_to_sync(run._post_process)(SandboxedEvalCase(name="sparse", prompt="Investigate"),NullCaseHooks(),diff --git a/products/signals/evals/eval_first_report.py b/products/signals/evals/eval_first_report.pyindex 43743415c3f6..cdc58807e1fb 100644--- a/products/signals/evals/eval_first_report.py+++ b/products/signals/evals/eval_first_report.py@@ -1,6 +1,6 @@from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPrivateEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContext@@ -9,14 +9,11 @@ from .first_report_seeders import seed_sparse_project, seed_upload_control, seedfrom .offline_config import FIRST_REPORT_SUITEfrom .scout_runner import run_first_report_scout-NO_SEND_LOGS = True-async def eval_first_report(ctx: EvalContext) -> None:if ctx.skill_delivery != "bundled":raise ValueError("Scout evals use the production bundled skill configuration")- await SandboxedPrivateEval(- no_send_logs=NO_SEND_LOGS,+ await SandboxedEval(offline_suite=FIRST_REPORT_SUITE,send_to_posthog=True,experiment_name="first-report-scout",diff --git a/products/slack_app/evals/eval_feature_tip_picker.py b/products/slack_app/evals/eval_feature_tip_picker.pyindex 0576368857ed..26a8a8a8e5fb 100644--- a/products/slack_app/evals/eval_feature_tip_picker.py+++ b/products/slack_app/evals/eval_feature_tip_picker.py@@ -15,7 +15,7 @@ import asynciofrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.slack_app.backend.logic.feature_tips import pickerfrom products.slack_app.backend.logic.feature_tips.catalog import CATALOGfrom products.slack_app.backend.logic.feature_tips.features import SlackFeature@@ -154,7 +154,7 @@ async def eval_feature_tip_picker(ctx: EvalContext) -> None:"last_message": f"{model}: feature={picked}",}- await OneShotPublicEval(+ await OneShotEval(experiment_name="slack-app-feature-tip-picker",offline_suite=offline_suite("slack_app/eval_feature_tip_picker::eval_feature_tip_picker",diff --git a/products/slack_app/evals/eval_followup_classifier.py b/products/slack_app/evals/eval_followup_classifier.pyindex 4526e841d3e4..889d61197904 100644--- a/products/slack_app/evals/eval_followup_classifier.py+++ b/products/slack_app/evals/eval_followup_classifier.py@@ -23,7 +23,7 @@ import asynciofrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.slack_app.backend.services.slack_messages import SlackThreadMessagefrom products.slack_app.backend.temporal.activities import classifiersfrom products.slack_app.evals.offline_config import offline_suite@@ -238,7 +238,7 @@ async def eval_followup_classifier(ctx: EvalContext) -> None:"last_message": f"{classifier_model}: agent_directed={agent_directed}",}- await OneShotPublicEval(+ await OneShotEval(experiment_name="slack-app-followup-classifier",offline_suite=offline_suite("slack_app/eval_followup_classifier::eval_followup_classifier",diff --git a/products/slack_app/evals/eval_model_classifier.py b/products/slack_app/evals/eval_model_classifier.pyindex d56f185160a6..dac51b7d296c 100644--- a/products/slack_app/evals/eval_model_classifier.py+++ b/products/slack_app/evals/eval_model_classifier.py@@ -27,7 +27,7 @@ import asynciofrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.slack_app.backend.services.model_catalogue import ModelChoicefrom products.slack_app.backend.temporal.activities import classifiersfrom products.slack_app.evals.offline_config import offline_suite@@ -274,7 +274,7 @@ async def eval_model_classifier(ctx: EvalContext) -> None:"last_message": f"{classifier_model}: {override.model_dump() if override else None}",}- await OneShotPublicEval(+ await OneShotEval(experiment_name="slack-app-model-classifier",offline_suite=offline_suite("slack_app/eval_model_classifier::eval_model_classifier",diff --git a/products/slack_app/evals/eval_posthog_first.py b/products/slack_app/evals/eval_posthog_first.pyindex 2d9ea513269c..58ccc034d7b7 100644--- a/products/slack_app/evals/eval_posthog_first.py+++ b/products/slack_app/evals/eval_posthog_first.py@@ -24,7 +24,7 @@ To run::from __future__ import annotations-from products.posthog_ai.eval_harness.base import SandboxedPublicEval+from products.posthog_ai.eval_harness.base import SandboxedEvalfrom products.posthog_ai.eval_harness.config import SandboxedEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.scorers import RequiredToolCall@@ -61,7 +61,7 @@ async def eval_posthog_first(ctx: EvalContext) -> None:),]- await SandboxedPublicEval(+ await SandboxedEval(experiment_name="sandboxed-slack-posthog-first-cli",offline_suite=offline_suite("slack_app/eval_posthog_first::eval_posthog_first",diff --git a/products/slack_app/evals/eval_project_classifier.py b/products/slack_app/evals/eval_project_classifier.pyindex c806af1dce15..f39c50b679db 100644--- a/products/slack_app/evals/eval_project_classifier.py+++ b/products/slack_app/evals/eval_project_classifier.py@@ -45,7 +45,7 @@ from posthog.models.team.team import Teamfrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.slack_app.backend.temporal.activities import classifiersfrom products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import PROJECT_ROUTE_KEY, NoUnaskedProjectSwitch, ProjectRouteMatch@@ -530,7 +530,7 @@ async def eval_project_classifier(ctx: EvalContext) -> None:"last_message": f"{classifier_model}: {chosen.team.name if chosen else None}",}- await OneShotPublicEval(+ await OneShotEval(experiment_name="slack-app-project-classifier",offline_suite=offline_suite("slack_app/eval_project_classifier::eval_project_classifier",diff --git a/products/slack_app/evals/eval_reply_suggestions.py b/products/slack_app/evals/eval_reply_suggestions.pyindex 4c0ee6837a62..551328d70bbe 100644--- a/products/slack_app/evals/eval_reply_suggestions.py+++ b/products/slack_app/evals/eval_reply_suggestions.py@@ -15,7 +15,7 @@ import asynciofrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.slack_app.backend.logic import reply_suggestionsfrom products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import REPLY_OFFER_KEY, NoUnaskedButtons, ReplyOfferMatch, SuggestedRepliesFit@@ -205,7 +205,7 @@ async def eval_reply_suggestions(ctx: EvalContext) -> None:"last_message": f"{model}: {offered}",}- await OneShotPublicEval(+ await OneShotEval(experiment_name="slack-app-reply-suggestions",offline_suite=offline_suite("slack_app/eval_reply_suggestions::eval_reply_suggestions",diff --git a/products/slack_app/evals/eval_untagged_question.py b/products/slack_app/evals/eval_untagged_question.pyindex 39052dbf17ab..784047579254 100644--- a/products/slack_app/evals/eval_untagged_question.py+++ b/products/slack_app/evals/eval_untagged_question.py@@ -19,7 +19,7 @@ import asynciofrom products.posthog_ai.eval_harness.config import BaseEvalCasefrom products.posthog_ai.eval_harness.harness.context import EvalContextfrom products.posthog_ai.eval_harness.harness.requirements import SuiteKind-from products.posthog_ai.eval_harness.one_shot import OneShotPublicEval+from products.posthog_ai.eval_harness.one_shot import OneShotEvalfrom products.slack_app.backend.temporal.activities import untagged_questionfrom products.slack_app.evals.offline_config import offline_suitefrom products.slack_app.evals.scorers import UNTAGGED_QUESTION_KEY, NoUnaskedAnswer, UntaggedQuestionMatch@@ -255,7 +255,7 @@ async def eval_untagged_question(ctx: EvalContext) -> None:"last_message": f"{model}: answerable={answerable}",}- await OneShotPublicEval(+ await OneShotEval(experiment_name="slack-app-untagged-question",offline_suite=offline_suite("slack_app/eval_untagged_question::eval_untagged_question",