:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(evals): automate claude and codex model comparisons (#119406)
Daniel Parker2f9f81255286
diff --git a/docs/internal/ai-offline-evaluation-reporting.md b/docs/internal/ai-offline-evaluation-reporting.mdindex 450f46879499..639ed24726d2 100644--- a/docs/internal/ai-offline-evaluation-reporting.md+++ b/docs/internal/ai-offline-evaluation-reporting.md@@ -30,6 +30,10 @@ Case names stay the same across trials.For a repeatable smoke comparison, use the [Claude and Codex command pair](../../products/posthog_ai/eval_harness/README.md#claude-and-codex-smoke-comparison).Both commands select the same MCP case and record the model, runtime, delivery mode, and checkout.Compare completed coverage and infrastructure errors alongside named scores and duration.+Add `--compare` to one selected sandboxed run to execute both default models and save a combined local comparison.+Repeated `--compare-model RUNTIME:MODEL` options select explicit models. The comparison checks both configurations+before running agents, retains each model's results, and reports case or scorer selection differences.+This local comparison does not change the suite's upload or capture settings.Trial metadata and local summaries also record phase durations in seconds.The scoring phase includes time waiting for the experiment results, rather than scorer execution alone.A failed final metadata upload produces a warning and keeps the completed results available locally.diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.mdindex 4527f4517ec1..d6b0806f7e5d 100644--- a/products/posthog_ai/eval_harness/README.md+++ b/products/posthog_ai/eval_harness/README.md@@ -190,6 +190,40 @@ Experiment names don't change with the runtime or skill delivery. Each Braintrus### Claude and Codex smoke comparison+Run both default models and print one comparison with `--compare`:++```bash+hogli evals cli_mcp/eval_skill_distribution::eval_skill_distribution \+ --eval historical_traffic_share_by_type \+ --provider modal --skill-delivery exec \+ --trials 3 --max-sandboxes 2 --compare+```++Both configurations are checked before either agent run starts. The models run in order, in separate processes,+with the same suite selectors, case filter, provider, skill delivery, trial count, timeout, and concurrency.+Comparisons use plain text output so each model can finish without waiting for the interactive view to close.+Ctrl-C stops the comparison after the current child finishes cleanup.++Use repeated `--compare-model claude:MODEL` and `--compare-model codex:MODEL` options to replace the default pair.+At least two distinct runtime/model pairs and one suite selector are required. Only sandboxed suites are supported;+one-shot suites select their models internally. Do not combine comparison options with `--agent-model` or `--agent-runtime`.++The command saves `comparison.json`, `comparison.txt`, and each child's run report under `logs/comparisons/<run>/`.+Use `--comparison-dir PATH` to choose a new directory. Existing directories are rejected to protect previous results.+Each model retains its own full transcript, and the comparison has a combined transcript whose path prints last.+Reports retain completed models if a later model fails or the comparison is interrupted.++The table shows named scorer means with scored-trial counts, completed execution coverage, errors, timeouts, and wall time.+Missing and skipped scores show `n/a`; a real zero shows `0.0%`. Costs show `unavailable` because these results do not contain cost data.+Case-set, scorer-set, or checkout differences produce a warning and a nonzero exit status.+Errors, timeouts, incomplete results, and failed score gates also make the comparison exit nonzero.+The comparison measures model plus runtime. A small smoke case is not a model ranking.++Use `--check` without comparison options to validate a single run without starting services or agents.+Use `--results-file PATH` on a single run to save its structured scores, coverage, settings, and log locations.++The manual command pair remains available:+Start with one fixed MCP case. It checks skill discovery and tool use for a historical traffic query.Both commands below use the same case, provider, skill delivery, concurrency, and scorers.Run them in order from the same checkout. Separate harness processes share service ports, so do not run them together.diff --git a/products/posthog_ai/eval_harness/base.py b/products/posthog_ai/eval_harness/base.pyindex 6f9d5047f92a..922aa5e489f4 100644--- a/products/posthog_ai/eval_harness/base.py+++ b/products/posthog_ai/eval_harness/base.py@@ -388,6 +388,7 @@ class _BaseEvalRun:result.summary,error_count=error_count,trial_stats=trial_stats(result.results, trials=self.ctx.trials),+ case_results=result.results,)if os.getenv("EXPORT_EVAL_RESULTS"):@@ -497,7 +498,12 @@ class _BaseEvalRun:# Register the case total (post-filter, times trials) so the reporter can# append a per-experiment progress counter to each case line.planned_cases = len(eval_cases) * self.ctx.trials- await self.ctx.reporter.experiment_started(self.experiment_name, planned_cases, self.run_log_dir)+ await self.ctx.reporter.experiment_started(+ self.experiment_name,+ planned_cases,+ self.run_log_dir,+ case_names=[str(case.input["name"]) for case in eval_cases],+ )try:result = await self.engine.run_experiment(diff --git a/products/posthog_ai/eval_harness/harness/README.md b/products/posthog_ai/eval_harness/harness/README.mdindex 40aa514da76a..41eb4e84f7fb 100644--- a/products/posthog_ai/eval_harness/harness/README.md+++ b/products/posthog_ai/eval_harness/harness/README.md@@ -71,6 +71,12 @@ The final summary renders after teardown, logging is flushed, and the transcript## Output model+`--compare` runs the selected sandboxed suites through Claude and Codex in separate, sequential child processes.+The comparison checks every configuration first with `--check`, then forwards child output into its combined transcript.+`--results-file` writes a validated run report using the reporter's existing scores, status counts, and case selection.+The comparison retains these reports after each model and checks that case and scorer selections match.+No prompts, model answers, or credentials are included in the comparison report.+The reporter emits plain-text records with stable labels so terminals and agents can scan the same output.`START` marks work entering flight, `DONE` marks successful completion, and `TIMEOUT`, `ERROR`, or `CRASH` name exceptional outcomes.Only the final overall run status uses `PASS` or `FAIL`.diff --git a/products/posthog_ai/eval_harness/harness/__main__.py b/products/posthog_ai/eval_harness/harness/__main__.pyindex a951b5d80b93..e2b7d5cb2a9f 100644--- a/products/posthog_ai/eval_harness/harness/__main__.py+++ b/products/posthog_ai/eval_harness/harness/__main__.py@@ -128,6 +128,22 @@ def _run_captured(options: HarnessOptions, transcript: RunTranscript, dashboard:def main(argv: list[str] | None = None) -> int:options = parse_args(argv)+ if options.comparison_models:+ from .comparison import ModelComparison # noqa: PLC0415 — keep comparison orchestration off normal startup++ transcript = RunTranscript.create()+ try:+ with transcript.capture(verbose=options.verbose):+ _configure_logging(transcript, verbose=options.verbose)+ try:+ return ModelComparison(options).run()+ except OSError as error:+ write_console(f"error: Model comparison failed: {error}\n", stream=sys.stderr)+ return USAGE_ERROR_EXIT_CODE+ finally:+ logging.shutdown()+ finally:+ transcript.finish()if options.list_only:_configure_logging()return _run(options)diff --git a/products/posthog_ai/eval_harness/harness/cli.py b/products/posthog_ai/eval_harness/harness/cli.pyindex 22122902bdfd..4348a3e65e3b 100644--- a/products/posthog_ai/eval_harness/harness/cli.py+++ b/products/posthog_ai/eval_harness/harness/cli.py@@ -3,8 +3,11 @@ from __future__ import annotationsimport osimport argparsefrom dataclasses import dataclass+from pathlib import Pathfrom typing import Literal, cast, get_args+from posthog.dataclasses import frozen+from .providers import DockerProviderStrategy, ModalProviderStrategy, SandboxProviderfrom .retries import DEFAULT_INFRA_RETRIES@@ -19,6 +22,14 @@ DEFAULT_CODEX_AGENT_MODEL = "gpt-6.1-sol"# Django-free (see harness/AGENTS.md), so it cannot import the enum.AGENT_RUNTIMES = ("claude", "codex")DEFAULT_AGENT_MODEL_BY_RUNTIME = {"claude": DEFAULT_AGENT_MODEL, "codex": DEFAULT_CODEX_AGENT_MODEL}+++@frozen+class ModelTarget:+ runtime: str+ model: str++SkillDelivery = Literal["bundled", "exec"]DEFAULT_SKILL_DELIVERY: SkillDelivery = "bundled"DEFAULT_CASE_TIMEOUT_SECONDS = 60 * 15@@ -64,6 +75,10 @@ class HarnessOptions:verbose: bool = Falseinfrastructure_retries: int = DEFAULT_INFRA_RETRIESinteractive: bool | None = None+ comparison_models: tuple[ModelTarget, ...] = ()+ comparison_dir: Path | None = None+ results_file: Path | None = None+ check_only: bool = Falsedef _default_case_timeout() -> int:@@ -190,6 +205,22 @@ def build_parser() -> argparse.ArgumentParser:default=None,help="Exit nonzero when the mean score across all experiments falls below this fraction (0-1).",)+ parser.add_argument("--compare", action="store_true", help="Compare the default Claude and Codex models in order.")+ parser.add_argument(+ "--compare-model",+ action="append",+ default=[],+ metavar="RUNTIME:MODEL",+ help="Compare explicit runtime/model pairs. Repeat at least twice; replaces --compare defaults.",+ )+ parser.add_argument("--comparison-dir", type=Path, help="New directory for comparison reports.")+ parser.add_argument("--results-file", type=Path, help="Save a machine-readable run report to this path.")+ parser.add_argument(+ "--check",+ dest="check_only",+ action="store_true",+ help="Check configuration without starting services or agents.",+ )return parser@@ -197,6 +228,35 @@ def parse_args(argv: list[str] | None = None) -> HarnessOptions:parser = build_parser()args = parser.parse_args(argv)+ comparison_models: list[ModelTarget] = []+ for value in args.compare_model:+ runtime, separator, model = value.partition(":")+ if not separator or runtime not in AGENT_RUNTIMES or not model.strip():+ parser.error("--compare-model must be claude:MODEL or codex:MODEL")+ comparison_models.append(ModelTarget(runtime=runtime, model=model))+ if args.compare and not comparison_models:+ comparison_models = [+ ModelTarget(runtime=runtime, model=model) for runtime, model in DEFAULT_AGENT_MODEL_BY_RUNTIME.items()+ ]+ if comparison_models:+ if not args.selectors:+ parser.error("Model comparisons require at least one suite selector")+ if len(comparison_models) < 2 or len(set(comparison_models)) != len(comparison_models):+ parser.error("Compare at least two distinct runtime/model pairs")+ if (+ args.agent_runtime+ or args.agent_model+ or args.list_only+ or args.results_file+ or args.check_only+ or args.interactive+ ):+ parser.error(+ "Model comparisons cannot use --agent-runtime, --agent-model, --list, --results-file, --check, or --tui"+ )+ elif args.comparison_dir:+ parser.error("--comparison-dir requires --compare or --compare-model")+# Sandbox-only flags keep None/False defaults so an explicit use is# detectable: a run without sandboxed suites rejects them in preflight# instead of silently ignoring them.@@ -268,4 +328,8 @@ def parse_args(argv: list[str] | None = None) -> HarnessOptions:infrastructure_retries=args.infra_retries,verbose=args.verbose,interactive=args.interactive,+ comparison_models=tuple(comparison_models),+ comparison_dir=args.comparison_dir,+ results_file=args.results_file,+ check_only=args.check_only,)diff --git a/products/posthog_ai/eval_harness/harness/comparison.py b/products/posthog_ai/eval_harness/harness/comparison.pynew file mode 100644index 000000000000..d23fe6747ed0--- /dev/null+++ b/products/posthog_ai/eval_harness/harness/comparison.py@@ -0,0 +1,300 @@+from __future__ import annotations++import sys+import json+import time+import signal+import subprocess+from datetime import UTC, datetime+from pathlib import Path+from types import FrameType+from uuid import uuid4++from pydantic import BaseModel, ConfigDict, ValidationError++from ..log_sink import LOGS_ROOT+from .cli import HarnessOptions, ModelTarget+from .run_report import RunReport+from .transcript import write_console+++class ModelRun(BaseModel):+ model_config = ConfigDict(frozen=True, extra="forbid")++ runtime: str+ model: str+ exit_code: int | None = None+ wall_time_seconds: float = 0+ report: RunReport | None = None+ issue: str | None = None+++class ModelComparison:+ def __init__(self, options: HarnessOptions) -> None:+ self.options = options+ self.directory = (+ options.comparison_dir or LOGS_ROOT / "comparisons" / f"{datetime.now(UTC):%Y%m%d-%H%M%S}_{uuid4().hex[:8]}"+ ).resolve()+ self.runs = [ModelRun(runtime=target.runtime, model=target.model) for target in options.comparison_models]+ self.issues: list[str] = []++ def _command(self, target: ModelTarget, path: Path, *, check: bool) -> list[str]:+ options = self.options+ command = [+ sys.executable,+ "-m",+ "products.posthog_ai.eval_harness.harness",+ *options.selectors,+ "--agent-runtime",+ target.runtime,+ "--agent-model",+ target.model,+ "--provider",+ options.provider,+ "--skill-delivery",+ options.skill_delivery,+ "--trials",+ str(options.trials),+ "--max-sandboxes",+ str(options.max_sandboxes),+ "--case-timeout",+ str(options.per_case_timeout_seconds),+ "--infra-retries",+ str(options.infrastructure_retries),+ "--results-file",+ str(path),+ "--no-tui",+ ]+ for flag, value in (+ ("--eval", options.case_filter),+ ("--reasoning-effort", options.reasoning_effort),+ ("--fail-under", options.fail_under),+ ):+ if value is not None:+ command.extend([flag, str(value)])+ for flag, enabled in (+ ("--check", check),+ ("--verbose", options.verbose),+ ("--create-db", options.create_db),+ ("--keep-sandbox-containers", options.keep_sandbox_containers),+ ("--rebuild-sandbox-image", options.rebuild_sandbox_image),+ ):+ if enabled:+ command.append(flag)+ return command++ @staticmethod+ def _interrupt_on_sigterm(_signum: int, _frame: FrameType | None) -> None:+ raise KeyboardInterrupt++ @staticmethod+ def _execute(command: list[str]) -> int:+ # The child runs in its own session, so a SIGTERM sent to this process never reaches it.+ # Treat SIGTERM like Ctrl-C so the child gets SIGINT and finishes its own cleanup.+ previous_handler = signal.signal(signal.SIGTERM, ModelComparison._interrupt_on_sigterm)+ try:+ return ModelComparison._stream(command)+ finally:+ signal.signal(signal.SIGTERM, previous_handler)++ @staticmethod+ def _stream(command: list[str]) -> int:+ with subprocess.Popen(+ command,+ start_new_session=True,+ stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT,+ encoding="utf-8",+ errors="replace",+ bufsize=1,+ ) as child:+ assert child.stdout is not None+ try:+ for line in child.stdout:+ write_console(line)+ return child.wait()+ except KeyboardInterrupt:+ child.send_signal(signal.SIGINT)+ for line in child.stdout:+ write_console(line)+ child.wait()+ raise++ @staticmethod+ def _read_report(path: Path) -> RunReport | None:+ try:+ return RunReport.model_validate_json(path.read_text(encoding="utf-8"))+ except (OSError, ValidationError):+ return None++ def _validate_run(self, run: ModelRun) -> str | None:+ report = run.report+ if report is None:+ return "No valid run report; inspect the child transcript."+ if report.runtime != run.runtime or report.model != run.model or report.exit_code != run.exit_code:+ return "Run report does not match the requested model or exit status."+ if not report.experiments:+ return "No experiment results."+ if not any(+ score.mean is not None and score.scored_trials+ for experiment in report.experiments.values()+ for score in experiment.scores.values()+ ):+ return "No scored trials are available."+ for experiment in report.experiments.values():+ if not experiment.cases and not experiment.planned_trials and not experiment.completed_trials:+ continue+ if not experiment.cases or sum(experiment.cases.values()) * report.trials != experiment.planned_trials:+ return "Missing or inconsistent case selection."+ if experiment.completed_trials != experiment.planned_trials or experiment.errors or experiment.timeouts:+ return "Incomplete trial coverage, errors, or timeouts; compare scores with care."+ return None++ def _check_selection(self) -> None:+ reports = [run.report for run in self.runs if run.report and run.report.experiments]+ if len(reports) < 2:+ return+ selections = [+ (+ report.suite_ids,+ report.revision,+ report.worktree_dirty,+ {+ name: (experiment.cases, sorted(experiment.scores))+ for name, experiment in report.experiments.items()+ },+ )+ for report in reports+ ]+ if any(selection != selections[0] for selection in selections[1:]):+ self.issues.append("Checkout, case, or scorer selections differ. These scores are not directly comparable.")++ def _text(self) -> str:+ rows = [+ ["Metric", *(f"{run.runtime}/{run.model}" for run in self.runs)],+ ["Exit status", *(str(run.exit_code) if run.exit_code is not None else "not run" for run in self.runs)],+ [+ "Wall time",+ *(f"{run.wall_time_seconds:.1f}s" if run.exit_code is not None else "n/a" for run in self.runs),+ ],+ ]+ for metric in ("completed_trials", "errors", "timeouts"):+ values = []+ for run in self.runs:+ if run.report is None or not run.report.experiments:+ values.append("n/a")+ continue+ actual = sum(+ {+ "completed_trials": experiment.completed_trials,+ "errors": experiment.errors,+ "timeouts": experiment.timeouts,+ }[metric]+ for experiment in run.report.experiments.values()+ )+ planned = sum(experiment.planned_trials for experiment in run.report.experiments.values())+ values.append(f"{actual}/{planned}" if metric == "completed_trials" else str(actual))+ rows.append([metric.replace("_", " ").capitalize(), *values])+ rows.append(["Cost", *("unavailable" for _run in self.runs)])+ score_keys = sorted(+ {+ (name, scorer)+ for run in self.runs+ if run.report is not None+ for name, experiment in run.report.experiments.items()+ for scorer in experiment.scores+ }+ )+ for name, scorer in score_keys:+ values = []+ for run in self.runs:+ experiment = run.report.experiments.get(name) if run.report else None+ score = experiment.scores.get(scorer) if experiment else None+ values.append(+ f"{score.mean:.1%} ({score.scored_trials}/{experiment.planned_trials} scored)"+ if score is not None and score.mean is not None and experiment is not None+ else "n/a"+ )+ rows.append([f"{name} / {scorer}", *values])+ widths = [max(len(row[index]) for row in rows) for index in range(len(self.runs) + 1)]+ lines = [+ "Model comparison (model + runtime)",+ *(" | ".join(value.ljust(widths[index]) for index, value in enumerate(row)).rstrip() for row in rows),+ ]+ lines.extend(f"NOTE: {issue}" for issue in self.issues)+ lines.extend(f"NOTE: {run.runtime}/{run.model}: {run.issue}" for run in self.runs if run.issue)+ return "\n".join(lines) + "\n"++ def _save(self) -> None:+ report = {+ "schema_version": 1,+ "settings": {+ "selectors": self.options.selectors,+ "case_filter": self.options.case_filter,+ "provider": self.options.provider,+ "trials": self.options.trials,+ "skill_delivery": self.options.skill_delivery,+ "reasoning_effort": self.options.reasoning_effort,+ "max_sandboxes": self.options.max_sandboxes,+ "case_timeout": self.options.per_case_timeout_seconds,+ "infrastructure_retries": self.options.infrastructure_retries,+ "fail_under": self.options.fail_under,+ },+ "issues": self.issues,+ "runs": [run.model_dump(mode="json") for run in self.runs],+ }+ temporary = self.directory / "comparison.tmp"+ temporary.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")+ temporary.replace(self.directory / "comparison.json")+ (self.directory / "comparison.txt").write_text(self._text(), encoding="utf-8")++ def run(self) -> int:+ self.directory.mkdir(parents=True, exist_ok=False)+ self._save()+ try:+ checked: list[RunReport] = []+ for index, target in enumerate(self.options.comparison_models):+ write_console(f"MODEL CHECK {index + 1}/{len(self.runs)} {target.runtime}/{target.model}\n")+ path = self.directory / f"{index + 1}-check.json"+ code = self._execute(self._command(target, path, check=True))+ report = self._read_report(path)+ if code or report is None or report.suite_kinds != ["sandboxed"]:+ self.issues.append(+ f"{target.runtime}/{target.model}: configuration check failed or selected suites are not all sandboxed. "+ "No model runs started. Inspect the child transcript above."+ )+ return 1+ checked.append(report)+ if any(report.suite_ids != checked[0].suite_ids for report in checked[1:]):+ self.issues.append("Suite selections differ. No model runs started.")+ return 1+ for index, target in enumerate(self.options.comparison_models):+ write_console(f"MODEL START {index + 1}/{len(self.runs)} {target.runtime}/{target.model}\n")+ started = time.monotonic()+ path = self.directory / f"{index + 1}-results.json"+ try:+ code = self._execute(self._command(target, path, check=False))+ except KeyboardInterrupt:+ code = 130+ run = ModelRun(+ runtime=target.runtime,+ model=target.model,+ exit_code=code,+ wall_time_seconds=time.monotonic() - started,+ report=self._read_report(path),+ )+ self.runs[index] = run.model_copy(update={"issue": self._validate_run(run)})+ self._save()+ if code == 130:+ return 130+ self._check_selection()+ return int(bool(self.issues) or any(run.exit_code or run.issue for run in self.runs))+ except KeyboardInterrupt:+ self.issues.append("Comparison interrupted during configuration checks.")+ return 130+ except OSError as error:+ self.issues.append(f"Comparison could not run: {error}")+ return 1+ finally:+ self._save()+ write_console("\n" + self._text() + f"Comparison reports: {self.directory}\n")diff --git a/products/posthog_ai/eval_harness/harness/lifecycle.py b/products/posthog_ai/eval_harness/harness/lifecycle.pyindex f13387707326..1e7e9d5c33c3 100644--- a/products/posthog_ai/eval_harness/harness/lifecycle.py+++ b/products/posthog_ai/eval_harness/harness/lifecycle.py@@ -165,7 +165,14 @@ class SandboxedEvalHarness:flags = ", ".join(self.options.sandbox_flags_set)raise PreflightError(f"{flags}: no selected suite is sandboxed, so sandbox flags have no effect")- reporter = ProgressReporter(total_suites=len(suites), verbose=self.options.verbose, dashboard=self._dashboard)+ reporter = ProgressReporter(+ total_suites=len(suites),+ verbose=self.options.verbose,+ dashboard=self._dashboard,+ revision=self._git_sha,+ worktree_dirty=self._git_dirty,+ )+ reporter.write_run_report(self.options, suites, exit_code=1, duration_seconds=0)reporter.print_run_header(provider=self.options.provider,agent_runtime=self.options.agent_runtime,@@ -191,6 +198,11 @@ class SandboxedEvalHarness:)self._validate_agent_options()self.provider.preflight()+ if self.options.check_only:+ reporter.write_run_report(+ self.options, suites, exit_code=0, duration_seconds=time.monotonic() - started+ )+ return 0if Infra.PERSONHOG in required:reporter.setup_started("build personhog")ensure_personhog_binaries()@@ -228,15 +240,25 @@ class SandboxedEvalHarness:except KeyboardInterrupt:interrupted = Truerunner.run(self._offline_delivery.cancel())+ finally:+ if not self.options.check_only:+ reporter.write_run_report(+ self.options,+ suites,+ exit_code=130 if interrupted else 1,+ duration_seconds=time.monotonic() - started,+ )duration_seconds = time.monotonic() - startedif interrupted:reporter.print_incomplete_summary(status="INTERRUPTED", duration_seconds=duration_seconds)+ reporter.write_run_report(self.options, suites, exit_code=130, duration_seconds=duration_seconds)return 130if results is None:raise RuntimeError("Sandboxed eval run finished without suite results")exit_code = self._exit_code(results, reporter)+ reporter.write_run_report(self.options, suites, exit_code=exit_code, duration_seconds=duration_seconds)reporter.print_final_summary(results,exit_code=exit_code,diff --git a/products/posthog_ai/eval_harness/harness/reporting.py b/products/posthog_ai/eval_harness/harness/reporting.pyindex 271ec323743d..cf8b48a9d84a 100644--- a/products/posthog_ai/eval_harness/harness/reporting.py+++ b/products/posthog_ai/eval_harness/harness/reporting.py@@ -15,11 +15,15 @@ from posthog.dataclasses import frozenfrom ..engines.types import EvalSummaryfrom .progress import TrialPhase, TrialProgress+from .run_report import ExperimentReport, RunReport, ScoreReportfrom .transcript import write_consolefrom .trial_stats import ScorerTrialStatsif TYPE_CHECKING:+ from ..engines.types import CaseResult+ from .cli import HarnessOptionsfrom .dashboard import EvalDashboard+ from .discovery import EvalSuiteEVAL_RESULTS_JSONL = "eval_results.jsonl""""Machine-readable per-experiment summary export, opt-in via ``EXPORT_EVAL_RESULTS``."""@@ -68,7 +72,17 @@ class _CaseProgress:class ProgressReporter:"""Serializes harness output and renders the stable final summary."""- def __init__(self, total_suites: int, *, verbose: bool = False, dashboard: EvalDashboard | None = None) -> None:+ def __init__(+ self,+ total_suites: int,+ *,+ verbose: bool = False,+ dashboard: EvalDashboard | None = None,+ revision: str | None = None,+ worktree_dirty: bool | None = None,+ ) -> None:+ self._revision = revision+ self._worktree_dirty = worktree_dirtyself._total_suites = total_suitesself._verbose = verboseself._lock = asyncio.Lock()@@ -81,6 +95,9 @@ class ProgressReporter:self._summary_error_counts: dict[str, int] = defaultdict(int)self._trial_stats: dict[str, Sequence[ScorerTrialStats]] = {}self._experiment_totals: dict[str, int] = {}+ self._case_selections: dict[str, dict[str, int]] = {}+ self._scored_trial_counts: dict[str, Counter[str]] = {}+ self._scorer_names: dict[str, frozenset[str]] = {}self._posthog_urls: dict[str, str] = {}self._posthog_offline_statuses: dict[str, str] = {}self._log_dirs: dict[str, Path] = {}@@ -230,9 +247,12 @@ class ProgressReporter:async with self._lock:await asyncio.to_thread(_emit, f"SUITE START {suite_id}")- async def experiment_started(self, experiment_name: str, planned_cases: int, log_dir: Path) -> None:+ async def experiment_started(+ self, experiment_name: str, planned_cases: int, log_dir: Path, *, case_names: Sequence[str] = ()+ ) -> None:async with self._lock:self._experiment_totals[experiment_name] = planned_cases+ self._case_selections[experiment_name] = dict(Counter(case_names))self._log_dirs[experiment_name] = log_dirif self._dashboard:self._dashboard.experiment_started(experiment_name, planned_cases)@@ -327,17 +347,73 @@ class ProgressReporter:*,error_count: int = 0,trial_stats: Sequence[ScorerTrialStats] = (),+ case_results: Sequence[CaseResult] = (),) -> None:async with self._lock:self._summaries[experiment_name] = summaryself._trial_stats[experiment_name] = trial_statsself._summary_error_counts[experiment_name] = error_count+ # The offline summary drops a scorer that skipped every trial, so keep the names the trials reported.+ self._scorer_names[experiment_name] = frozenset(summary.scores).union(+ *(result.scores for result in case_results)+ )+ self._scored_trial_counts[experiment_name] = Counter(+ name+ for result in case_results+ if result.error is None or result.error_kind == "scorer"+ for name, score in result.scores.items()+ if score is not None+ )if os.getenv("EXPORT_EVAL_RESULTS"):await asyncio.to_thread(self._append_summary, summary)total = self._experiment_totals.get(experiment_name)progress = f"[{self._case_counts[experiment_name]}/{total} cases]" if total is not None else ""await asyncio.to_thread(_emit, f"EXPERIMENT DONE {experiment_name} {progress}")+ def write_run_report(+ self, options: HarnessOptions, suites: Sequence[EvalSuite], *, exit_code: int, duration_seconds: float+ ) -> None:+ if options.results_file is None:+ return+ experiments: dict[str, ExperimentReport] = {}+ for name, planned in self._experiment_totals.items():+ summary = self._summaries.get(name)+ experiments[name] = ExperimentReport(+ cases=self._case_selections[name],+ planned_trials=planned,+ completed_trials=self._case_counts[name],+ errors=max(self._case_statuses[name]["error"], self._summary_error_counts[name]),+ timeouts=self._case_statuses[name]["timeout"],+ scores={+ key: ScoreReport(+ mean=summary.scores[key].score if key in summary.scores else None,+ scored_trials=self._scored_trial_counts[name][key],+ )+ for key in sorted(self._scorer_names[name])+ }+ if summary is not None+ else {},+ log_dir=str(self._log_dirs[name].resolve()),+ )+ report = RunReport(+ runtime=options.agent_runtime,+ model=options.agent_model,+ reasoning_effort=options.reasoning_effort,+ provider=options.provider,+ trials=options.trials,+ suite_ids=[suite.id for suite in suites],+ suite_kinds=sorted({suite.kind.value for suite in suites}),+ exit_code=exit_code,+ duration_seconds=duration_seconds,+ experiments=experiments,+ revision=self._revision,+ worktree_dirty=self._worktree_dirty,+ )+ options.results_file.parent.mkdir(parents=True, exist_ok=True)+ temporary = options.results_file.with_suffix(".tmp")+ temporary.write_text(report.model_dump_json(indent=2) + "\n", encoding="utf-8")+ temporary.replace(options.results_file)+async def record_posthog_offline_result(self, experiment_name: str, status: str, *, url: str | None = None) -> None:async with self._lock:self._posthog_offline_statuses[experiment_name] = statusdiff --git a/products/posthog_ai/eval_harness/harness/run_report.py b/products/posthog_ai/eval_harness/harness/run_report.pynew file mode 100644index 000000000000..9ffe98df9ceb--- /dev/null+++ b/products/posthog_ai/eval_harness/harness/run_report.py@@ -0,0 +1,42 @@+from __future__ import annotations++from typing import Literal++from pydantic import BaseModel, ConfigDict, Field+++class ScoreReport(BaseModel):+ model_config = ConfigDict(frozen=True, extra="forbid", allow_inf_nan=False)++ mean: float | None+ scored_trials: int+++class ExperimentReport(BaseModel):+ model_config = ConfigDict(frozen=True, extra="forbid")++ cases: dict[str, int]+ planned_trials: int+ completed_trials: int+ errors: int+ timeouts: int+ scores: dict[str, ScoreReport]+ log_dir: str+++class RunReport(BaseModel):+ model_config = ConfigDict(frozen=True, extra="forbid", allow_inf_nan=False)++ schema_version: Literal[1] = 1+ runtime: str+ model: str+ reasoning_effort: str | None+ provider: str+ trials: int+ suite_ids: list[str]+ suite_kinds: list[str]+ exit_code: int+ duration_seconds: float+ revision: str | None = None+ worktree_dirty: bool | None = None+ experiments: dict[str, ExperimentReport] = Field(default_factory=dict)diff --git a/products/posthog_ai/eval_harness/test/test_model_comparison.py b/products/posthog_ai/eval_harness/test/test_model_comparison.pynew file mode 100644index 000000000000..c780b81e8077--- /dev/null+++ b/products/posthog_ai/eval_harness/test/test_model_comparison.py@@ -0,0 +1,264 @@+from __future__ import annotations++import os+import sys+import json+import signal+from pathlib import Path++import pytest++from products.posthog_ai.eval_harness.engines.types import AggregateScore, CaseResult, EvalSummary+from products.posthog_ai.eval_harness.harness.cli import parse_args+from products.posthog_ai.eval_harness.harness.comparison import ModelComparison+from products.posthog_ai.eval_harness.harness.discovery import EvalSuite+from products.posthog_ai.eval_harness.harness.reporting import ProgressReporter+from products.posthog_ai.eval_harness.harness.requirements import SuiteKind+from products.posthog_ai.eval_harness.harness.run_report import ExperimentReport, RunReport, ScoreReport+++ "scenario,expected_exit,expected_calls",+ [+ ("success", 0, 4),+ ("skipped", 1, 4),+ ("filtered_out_experiment", 0, 4),+ ("launch_error", 1, 1),+ ("error", 1, 4),+ ("timeout", 1, 4),+ ("different_cases", 1, 4),+ ("different_scorers", 1, 4),+ ("different_checkout", 1, 4),+ ("missing_report", 1, 4),+ ("failed_check", 1, 1),+ ("one_shot", 1, 1),+ ("interrupted", 130, 3),+ ],+)+def test_model_comparison_preserves_settings_and_partial_results(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+ capsys: pytest.CaptureFixture[str],+ scenario: str,+ expected_exit: int,+ expected_calls: int,+) -> None:+ directory = tmp_path / "comparison"+ options = parse_args(+ [+ "example/suite",+ "--compare",+ "--comparison-dir",+ str(directory),+ "--eval",+ "selected",+ "--provider",+ "modal",+ "--skill-delivery",+ "exec",+ "--trials",+ "3",+ "--max-sandboxes",+ "2",+ "--case-timeout",+ "123",+ "--infra-retries",+ "1",+ "--reasoning-effort",+ "medium",+ ]+ )+ calls: list[bool] = []++ def execute(command: list[str]) -> int:+ child = parse_args(command[3:])+ calls.append(child.check_only)+ assert child.selectors == options.selectors+ assert child.case_filter == options.case_filter+ assert child.provider == options.provider+ assert child.skill_delivery == options.skill_delivery+ assert child.trials == options.trials+ assert child.max_sandboxes == options.max_sandboxes+ assert child.per_case_timeout_seconds == options.per_case_timeout_seconds+ assert child.infrastructure_retries == options.infrastructure_retries+ assert child.reasoning_effort == options.reasoning_effort+ assert child.interactive is False+ assert not child.comparison_models+ assert child.results_file is not None+ if child.check_only and scenario == "failed_check":+ return 2+ if scenario == "launch_error":+ raise OSError("Example child launch failure")+ if not child.check_only and scenario == "interrupted":+ raise KeyboardInterrupt+ if not child.check_only and child.agent_runtime == "claude" and scenario == "missing_report":+ return 1+ second = child.agent_runtime == "codex"+ error = scenario == "error" and not second and not child.check_only+ timeout = scenario == "timeout" and not second+ report = RunReport(+ runtime=child.agent_runtime,+ model=child.agent_model,+ reasoning_effort=child.reasoning_effort,+ provider=child.provider,+ trials=child.trials,+ suite_ids=["example/suite::eval_suite"],+ suite_kinds=["one-shot" if scenario == "one_shot" else "sandboxed"],+ exit_code=int(error),+ duration_seconds=2,+ revision="changed" if scenario == "different_checkout" and second else "example-revision",+ experiments={}+ if child.check_only+ else {+ "example": ExperimentReport(+ cases={"other" if scenario == "different_cases" and second else "selected": 1},+ planned_trials=3,+ completed_trials=3,+ errors=int(error),+ timeouts=int(timeout),+ scores={+ "other_score" if scenario == "different_scorers" and second else "score": ScoreReport(+ mean=None if scenario == "skipped" else (1 if second else 0),+ scored_trials=0 if scenario == "skipped" else (2 if error else 3),+ )+ },+ log_dir=str(tmp_path),+ ),+ **(+ {+ "filtered_out": ExperimentReport(+ cases={},+ planned_trials=0,+ completed_trials=0,+ errors=0,+ timeouts=0,+ scores={},+ log_dir=str(tmp_path),+ )+ }+ if scenario == "filtered_out_experiment"+ else {}+ ),+ },+ )+ child.results_file.write_text(report.model_dump_json())+ return report.exit_code++ monkeypatch.setattr(ModelComparison, "_execute", staticmethod(execute))+ assert ModelComparison(options).run() == expected_exit+ assert len(calls) == expected_calls+ assert calls[:2] == [True, True][: min(expected_calls, 2)]+ assert not any(calls[2:])+ saved = json.loads((directory / "comparison.json").read_text())+ text = (directory / "comparison.txt").read_text()+ text = "\n".join(" ".join(line.split()) for line in text.splitlines())+ assert "Cost | unavailable | unavailable" in text+ assert "model + runtime" in text+ assert str(directory) in capsys.readouterr().out+ assert len(saved["runs"]) == 2+ assert saved["settings"]["skill_delivery"] == "exec"+ if scenario == "success":+ assert "0.0% (3/3 scored) | 100.0% (3/3 scored)" in text+ if scenario == "skipped":+ assert "example / score | n/a | n/a" in text+ if scenario == "launch_error":+ assert "Example child launch failure" in text+ if scenario.startswith("different_"):+ assert "not directly comparable" in text+ if scenario in {"failed_check", "one_shot", "interrupted"}:+ assert saved["runs"][1]["exit_code"] is None+ if scenario == "missing_report":+ assert saved["runs"][0]["report"] is None+ assert saved["runs"][1]["report"] is not None++[email protected]("interrupt", [None, "keyboard", "sigterm"])+def test_comparison_streams_child_output_and_waits_for_interrupt_cleanup(+ interrupt: str | None,+ monkeypatch: pytest.MonkeyPatch,+ capsys: pytest.CaptureFixture[str],+) -> None:+ from products.posthog_ai.eval_harness.harness.transcript import write_console++ program = (+ "import signal, threading; ready = threading.Event(); "+ "signal.signal(signal.SIGINT, lambda *args: ready.set()); "+ "print('Child ready', flush=True); "+ + (+ "ready.wait(); print('Cleanup complete', flush=True); raise SystemExit(130)"+ if interrupt+ else "raise SystemExit(3)"+ )+ )++ def output(text: str) -> None:+ write_console(text)+ if interrupt == "keyboard" and "Child ready" in text:+ raise KeyboardInterrupt+ if interrupt == "sigterm" and "Child ready" in text:+ os.kill(os.getpid(), signal.SIGTERM)++ monkeypatch.setattr("products.posthog_ai.eval_harness.harness.comparison.write_console", output)+ if interrupt:+ with pytest.raises(KeyboardInterrupt):+ ModelComparison._execute([sys.executable, "-c", program])+ assert "Cleanup complete" in capsys.readouterr().out+ else:+ assert ModelComparison._execute([sys.executable, "-c", program]) == 3+ assert "Child ready" in capsys.readouterr().out+++async def test_run_report_preserves_zero_skipped_scores_and_scorer_errors(+ tmp_path: Path,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.delenv("EXPORT_EVAL_RESULTS", raising=False)+ options = parse_args(["example", "--trials", "3", "--results-file", str(tmp_path / "results.json")])+ reporter = ProgressReporter(1, revision="example-revision", worktree_dirty=True)+ await reporter.experiment_started("example", 3, tmp_path, case_names=["selected"])+ for status in ("ok", "ok", "timeout"):+ await reporter.case_done("example", "selected", duration_seconds=1, status=status)+ await reporter.record_summary(+ "example",+ EvalSummary(+ engine_name="example",+ experiment_name="example",+ scores={+ "score": AggregateScore("score", 0),+ },+ ),+ error_count=1,+ case_results=[+ CaseResult(input={"name": "selected"}, output=None, scores={"score": 0, "skipped": None}),+ CaseResult(+ input={"name": "selected"},+ output=None,+ scores={"score": 0},+ error="Example scorer failure",+ error_kind="scorer",+ ),+ CaseResult(input={"name": "selected"}, output=None, scores={"score": 1}, error="Example task failure"),+ ],+ )++ async def suite(_ctx: object) -> None:+ pass++ suites = [+ EvalSuite(domain="example", module_name="example", fn_name="eval_suite", fn=suite, kind=SuiteKind.SANDBOXED)+ ]+ reporter.write_run_report(options, suites, exit_code=1, duration_seconds=3)+ assert options.results_file is not None+ report = RunReport.model_validate_json(options.results_file.read_text())+ experiment = report.experiments["example"]+ assert experiment.cases == {"selected": 1}+ assert experiment.errors == 1+ assert experiment.timeouts == 1+ assert experiment.scores["score"].mean == 0+ assert experiment.scores["score"].scored_trials == 2+ assert experiment.scores["skipped"].mean is None+ assert experiment.scores["skipped"].scored_trials == 0+ assert report.revision == "example-revision"+ assert report.worktree_dirty is True+ assert not list(tmp_path.glob("*.tmp"))diff --git a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py b/products/posthog_ai/eval_harness/test/test_one_shot_runner.pyindex 7553667c1193..93d72fc36f53 100644--- a/products/posthog_ai/eval_harness/test/test_one_shot_runner.py+++ b/products/posthog_ai/eval_harness/test/test_one_shot_runner.py@@ -68,11 +68,18 @@ class _StubReporter:) -> None:self.done.append((case_name, status))- async def experiment_started(self, experiment_name: str, planned_cases: int, log_dir: Path) -> None:+ async def experiment_started(+ self, experiment_name: str, planned_cases: int, log_dir: Path, *, case_names: Sequence[str] = ()+ ) -> None:self.started.append((experiment_name, planned_cases))async def record_summary(- self, experiment_name: str, summary: Any, error_count: int, trial_stats: Sequence[ScorerTrialStats] = ()+ self,+ experiment_name: str,+ summary: Any,+ error_count: int,+ trial_stats: Sequence[ScorerTrialStats] = (),+ case_results: Sequence[Any] = (),) -> None:self.summaries.append((experiment_name, summary, error_count))diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness.pyindex e556c786fc3a..c0bf097541b0 100644--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness.py@@ -173,12 +173,13 @@ def test_parse_args_resolves_team_setup_concurrency(pytest.param("ok", 0.5, None, 1, id="no_scores_with_gate"),pytest.param("timeout", None, 0.0, 0, id="timeout_without_gate"),pytest.param("timeout", 0.5, 0.0, 1, id="timeout_below_gate"),+ pytest.param("check", None, None, 0, id="configuration_check_only"),],)def test_eval_run_exit_status(monkeypatch: pytest.MonkeyPatch,capsys: pytest.CaptureFixture[str],- outcome: Literal["ok", "case_error", "engine_error", "both_errors", "timeout", "crash"],+ outcome: Literal["ok", "case_error", "engine_error", "both_errors", "timeout", "crash", "check"],fail_under: float | None,score: float | None,expected_exit_code: int,@@ -186,6 +187,7 @@ def test_eval_run_exit_status(monkeypatch.delenv("EXPORT_EVAL_RESULTS", raising=False)async def suite(ctx: EvalContext) -> None:+ assert outcome != "check", "Configuration checks must not execute cases"if outcome == "crash":raise RuntimeError("Synthetic suite failure")await ctx.reporter.case_done("experiment", "completed-case", duration_seconds=1.0)@@ -213,12 +215,19 @@ def test_eval_run_exit_status()monkeypatch.setattr(lifecycle, "infra_union", lambda _kinds: frozenset())monkeypatch.setattr(lifecycle, "validate_eval_env", lambda *_args, **_kwargs: None)- monkeypatch.setattr(SandboxedEvalHarness, "_bootstrap", lambda *_args: None)+ bootstrap = MagicMock()+ monkeypatch.setattr(SandboxedEvalHarness, "_bootstrap", bootstrap)args = [] if fail_under is None else ["--fail-under", str(fail_under)]+ if outcome == "check":+ args.append("--check")assert SandboxedEvalHarness(parse_args(args)).run() == expected_exit_codeoutput = capsys.readouterr().out- assert f"Status: {'PASS' if expected_exit_code == 0 else 'FAIL'}" in output+ if outcome == "check":+ bootstrap.assert_not_called()+ assert "SUITE START" not in output+ else:+ assert f"Status: {'PASS' if expected_exit_code == 0 else 'FAIL'}" in output@pytest.mark.asynciodiff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_cli.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_cli.pyindex 9eb6ddffabf8..9debffb953d9 100644--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_cli.py+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_cli.py@@ -4,7 +4,34 @@ import pytestfrom unittest.mock import MagicMockfrom products.posthog_ai.eval_harness.harness.__main__ import use_dashboard-from products.posthog_ai.eval_harness.harness.cli import SkillDelivery, parse_args+from products.posthog_ai.eval_harness.harness.cli import DEFAULT_AGENT_MODEL_BY_RUNTIME, SkillDelivery, parse_args+++ "arguments,expected",+ [+ (["suite", "--compare"], list(DEFAULT_AGENT_MODEL_BY_RUNTIME.items())),+ (+ ["suite", "--compare-model", "codex:gpt-example", "--compare-model", "codex:gpt-other"],+ [("codex", "gpt-example"), ("codex", "gpt-other")],+ ),+ (["suite", "--compare-model", "codex:gpt-example"], None),+ (["suite", "--compare-model", "unknown:model"], None),+ (["suite", "--compare-model", "claude:"], None),+ (["suite", "--compare-model", "claude:model", "--compare-model", "claude:model"], None),+ (["--compare"], None),+ (["suite", "--compare", "--agent-runtime", "codex"], None),+ (["suite", "--compare", "--tui"], None),+ (["--comparison-dir", "example"], None),+ ],+)+def test_model_comparison_arguments(arguments: list[str], expected: list[tuple[str, str]] | None) -> None:+ if expected is None:+ with pytest.raises(SystemExit) as error:+ parse_args(arguments)+ assert error.value.code == 2+ else:+ assert [(target.runtime, target.model) for target in parse_args(arguments).comparison_models] == expected@pytest.mark.parametrize(