:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(evals): migrate repo-selection evals to the shared harness (#128308)
Georgiy Tarasov5c3329b025ec
diff --git a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt b/posthog/test/repo_invariants/dataclass_frozen_baseline.txtindex a738e44a6021..fd60bbb8e484 100644--- a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt+++ b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt@@ -537,7 +537,6 @@3 posthog/session/risk.py3 posthog/slo/types.py3 posthog/tasks/ai_observability_usage_report.py-3 products/slack_app/backend/temporal/eval_slack_repo_selection.py3 posthog/temporal/ai_observability/evaluation_clustering/coordinator.py3 posthog/temporal/ai_observability/run_aggregate_evaluation.py3 posthog/temporal/ai_observability/run_tagger.pydiff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.mdindex 92b2b8c5aedb..118bbc65af88 100644--- a/products/posthog_ai/eval_harness/README.md+++ b/products/posthog_ai/eval_harness/README.md@@ -30,6 +30,34 @@ Product suites can supply `case_runner=` to `SandboxedEval` with the `SandboxedCTest cases should keep the default `SandboxedEvalCase.inherit_demo_data=True` to inherit Hedgebox data. Setting it to `False` is an explicit opt-in only for cases that intentionally need an empty isolated project, such as testing abstention with insufficient data. The setup hook must seed any data the case needs. The sparse scout case uses this option to add just four signups: inheriting the populated Hedgebox dataset would invalidate that test. Regression and control cases keep the default.+## Repository selection evals++Two suites measure repository selection on a synthetic `fernhill` catalog that each case installs in its own isolated project: repository names, descriptions, READMEs, tree paths, visibility, and routing rules come from [`products/tasks/evals/repo_selection/fixtures.py`](../../tasks/evals/repo_selection/fixtures.py).+No case depends on a real team's connected repositories, and expected answers stay out of the agent's input.++- `slack_app/eval_repo_selection_routing` runs a Slack mention through the production cascade, the needs-repo classifier, and the discovery agent. It scores the terminal stage and outcome, and the routed repository.+- `tasks/eval_repo_selection` runs only the production `select_repository` on the Self-Driving path, so classifier behavior cannot hide a difference between selector models. It scores accepted-set accuracy, correct abstention on cases with no plausible candidate, and completion.++Selector errors, timeouts, and picks outside the candidates score zero.+A trial that fails before the agent's first tool call, such as a gateway refusal at startup, is an infrastructure error, so the harness summary leaves it out of score averages.+The report below counts every planned trial, and scores those trials as wrong.+Compare selector models with per-arm settings, and set `EXPORT_EVAL_RESULTS=1` so each arm writes the per-trial results the report reads:++```bash+EXPORT_EVAL_RESULTS=1 hogli evals tasks/eval_repo_selection --trials 3 --max-sandboxes 4 \+ --compare-model claude:claude-haiku-5-5 \+ --compare-model codex:gpt-6-luna,reasoning_effort=medium \+ --compare-model codex:gpt-6-luna,reasoning_effort=medium,service_tier=flex+python -m products.tasks.evals.repo_selection.report <comparison-dir>+```++The report joins each trial with the eval gateway's ledger and request log. It lists accuracy, abstention, errors, latency, tokens, gateway-accounted cost, and the tier that served each call, so a Flex request that fell back to Standard is counted as Standard.+The eval gateway declares a Baseten host when `AI_GATEWAY_BASETEN_API_KEY` is set, which open-weight arms such as DeepSeek need.++The eval gateway posts every `$ai_generation` event to a local capture endpoint, and the run writes them to `logs/ai_gateway/<run>.generations.jsonl`, without prompt and completion bodies. The path prints as `AI GATEWAY GENERATIONS` and is saved in the run report. These events carry the requested and served service tier, Flex attempts, cache and reasoning tokens, and gateway cost, keyed by `task_run_id`.++Repository-selection runs attribute their model calls to their production products: `signals_repo_selection` for the selector suite and `slack_app` for the routing suite. The gateway applies each product's model restrictions, so a model reserved for other products fails at agent startup and the trial counts as an error. The `slack_app` model pin covers only models in `products/tasks/backend/model_catalog.py`, so run the routing suite on a catalog model such as `claude-sonnet-5-5`.+## Running```bash@@ -85,6 +113,7 @@ python -m products.posthog_ai.eval_harness.harness --list| `--agent-runtime {claude,codex}` | Agent runtime serving the model. Default `claude`. || `--skill-delivery {bundled,exec}` | Skill delivery path. Default `bundled`; `exec` removes native sandbox skills. || `--reasoning-effort <effort>` | Agent reasoning effort; valid values depend on runtime+model. |+| `--service-tier <tier>` | OpenAI service tier for codex runs (`default`, `flex`, `priority`). The claude runtime ignores it. || `--keep-sandbox-containers` | Skip the end-of-run Docker sweep, to inspect a leftover container. Docker only. || `--rebuild-sandbox-image` | Force a rebuild of the `posthog-sandbox-base` image before the run. Docker only. || `--create-db` | Rebuild the eval test database instead of reusing it. |@@ -97,7 +126,7 @@ python -m products.posthog_ai.eval_harness.harness --list| `--fail-under <fraction>` | Exit nonzero when the mean score across all experiments falls below this (0-1). || `--list` | Print the discovered suite ids (with their kinds) and exit. |-Sandbox-only flags (`--provider`, `--max-sandboxes`, `--agent-runtime`, `--skill-delivery`, `--reasoning-effort`, `--keep-sandbox-containers`, `--rebuild-sandbox-image`) are rejected in preflight when no selected suite is sandboxed, instead of being silently ignored.+Sandbox-only flags (`--provider`, `--max-sandboxes`, `--agent-runtime`, `--skill-delivery`, `--reasoning-effort`, `--service-tier`, `--keep-sandbox-containers`, `--rebuild-sandbox-image`) are rejected in preflight when no selected suite is sandboxed, instead of being silently ignored.The run exits with status 1 if a suite crashes or a case has an infrastructure error, even without `--fail-under`.Infrastructure errors remain separate from score averages.@@ -219,6 +248,7 @@ Comparisons use plain text output so each model can finish without waiting for tCtrl-C stops the comparison after the current child finishes cleanup.Use repeated `--compare-model claude:MODEL` and `--compare-model codex:MODEL` options to replace the default pair.+An arm can add its own settings after commas, such as `--compare-model codex:gpt-6-luna,reasoning_effort=medium,service_tier=flex`. Per-arm settings override `--reasoning-effort` and `--service-tier`, so one comparison can hold the same model on Standard and Flex as separate arms. The arm label, run report, and experiment metadata record both settings.At least two distinct runtime/model pairs and one suite selector are required. Only sandboxed suites are supported;one-shot suites select their models internally. Do not combine comparison options with `--agent-model` or `--agent-runtime`.diff --git a/products/posthog_ai/eval_harness/base.py b/products/posthog_ai/eval_harness/base.pyindex e7f99a74a66a..6ab31896f797 100644--- a/products/posthog_ai/eval_harness/base.py+++ b/products/posthog_ai/eval_harness/base.py@@ -792,6 +792,7 @@ class _SandboxedEvalRun(_BaseEvalRun):"agent_runtime": self.ctx.agent_runtime,"skill_delivery": self.ctx.skill_delivery,"reasoning_effort": self.ctx.reasoning_effort,+ "service_tier": self.ctx.service_tier,}diff --git a/products/posthog_ai/eval_harness/harness/cli.py b/products/posthog_ai/eval_harness/harness/cli.pyindex 423dc0c29c27..1602c444a052 100644--- a/products/posthog_ai/eval_harness/harness/cli.py+++ b/products/posthog_ai/eval_harness/harness/cli.py@@ -24,10 +24,25 @@ AGENT_RUNTIMES = ("claude", "codex")DEFAULT_AGENT_MODEL_BY_RUNTIME = {"claude": DEFAULT_AGENT_MODEL, "codex": DEFAULT_CODEX_AGENT_MODEL}+# Mirror of the agent server's `ServiceTier` values, kept literal for the same Django-free reason.+SERVICE_TIERS = ("default", "flex", "priority")++@frozenclass ModelTarget:runtime: strmodel: str+ service_tier: str | None = None+ reasoning_effort: str | None = None++ @property+ def label(self) -> str:+ settings = [+ f"{key}={value}"+ for key, value in (("service_tier", self.service_tier), ("reasoning_effort", self.reasoning_effort))+ if value is not None+ ]+ return f"{self.runtime}/{self.model}" + (f" ({', '.join(settings)})" if settings else "")SkillDelivery = Literal["bundled", "exec"]@@ -81,6 +96,8 @@ class HarnessOptions:check_only: bool = Falseskill_snapshot_report: Path | None = Noneskill_snapshot_output_dir: Path | None = None+ service_tier: str | None = None+ """OpenAI request tier for codex runs; ``None`` sends the request untiered."""def _default_case_timeout() -> int:@@ -152,6 +169,12 @@ def build_parser() -> argparse.ArgumentParser:default=None,help="Agent reasoning effort (e.g. 'low'…'xhigh'); valid values depend on runtime+model.",)+ parser.add_argument(+ "--service-tier",+ choices=SERVICE_TIERS,+ default=None,+ help="OpenAI service tier for codex runs. The claude runtime ignores it.",+ )parser.add_argument("--max-sandboxes",type=int,@@ -213,8 +236,11 @@ def build_parser() -> argparse.ArgumentParser:"--compare-model",action="append",default=[],- metavar="RUNTIME:MODEL",- help="Compare explicit runtime/model pairs. Repeat at least twice; replaces --compare defaults.",+ metavar="RUNTIME:MODEL[,service_tier=TIER][,reasoning_effort=EFFORT]",+ help=(+ "Compare explicit runtime/model arms. Repeat at least twice; replaces --compare defaults. "+ "Per-arm settings override --service-tier and --reasoning-effort."+ ),)parser.add_argument("--comparison-dir", type=Path, help="New directory for comparison reports.")parser.add_argument("--results-file", type=Path, help="Save a machine-readable run report to this path.")@@ -232,16 +258,40 @@ def build_parser() -> argparse.ArgumentParser:return parser+def _parse_model_target(value: str) -> ModelTarget | None:+ head, *settings = value.split(",")+ runtime, separator, model = head.partition(":")+ if not separator or runtime not in AGENT_RUNTIMES or not model.strip():+ return None+ options: dict[str, str] = {}+ for setting in settings:+ key, separator, setting_value = setting.partition("=")+ if not separator or key not in ("service_tier", "reasoning_effort") or not setting_value or key in options:+ return None+ options[key] = setting_value+ if options.get("service_tier", SERVICE_TIERS[0]) not in SERVICE_TIERS:+ return None+ return ModelTarget(+ runtime=runtime,+ model=model.strip(),+ service_tier=options.get("service_tier"),+ reasoning_effort=options.get("reasoning_effort"),+ )++def parse_args(argv: list[str] | None = None) -> HarnessOptions:parser = build_parser()args = parser.parse_args(argv)comparison_models: list[ModelTarget] = []for value in args.compare_model:- runtime, separator, model = value.partition(":")- if not separator or runtime not in AGENT_RUNTIMES or not model.strip():- parser.error("--compare-model must be claude:MODEL or codex:MODEL")- comparison_models.append(ModelTarget(runtime=runtime, model=model))+ target = _parse_model_target(value)+ if target is None:+ parser.error(+ "--compare-model must be claude:MODEL or codex:MODEL, optionally followed by "+ f",service_tier=<{'|'.join(SERVICE_TIERS)}> and ,reasoning_effort=<effort>"+ )+ comparison_models.append(target)if args.compare and not comparison_models:comparison_models = [ModelTarget(runtime=runtime, model=model) for runtime, model in DEFAULT_AGENT_MODEL_BY_RUNTIME.items()@@ -250,7 +300,7 @@ def parse_args(argv: list[str] | None = None) -> HarnessOptions:if not args.selectors:parser.error("Model comparisons require at least one suite selector")if len(comparison_models) < 2 or len(set(comparison_models)) != len(comparison_models):- parser.error("Compare at least two distinct runtime/model pairs")+ parser.error("Compare at least two distinct runtime/model arms")if (args.agent_runtimeor args.agent_model@@ -275,6 +325,7 @@ def parse_args(argv: list[str] | None = None) -> HarnessOptions:("--agent-runtime", args.agent_runtime is not None),("--skill-delivery", args.skill_delivery is not None),("--reasoning-effort", args.reasoning_effort is not None),+ ("--service-tier", args.service_tier is not None),("--max-sandboxes", args.max_sandboxes is not None),("--keep-sandbox-containers", args.keep_sandbox_containers),("--rebuild-sandbox-image", args.rebuild_sandbox_image),@@ -343,4 +394,5 @@ def parse_args(argv: list[str] | None = None) -> HarnessOptions:check_only=args.check_only,skill_snapshot_report=args.skill_snapshot_report,skill_snapshot_output_dir=args.skill_snapshot_output_dir,+ service_tier=args.service_tier,)diff --git a/products/posthog_ai/eval_harness/harness/comparison.py b/products/posthog_ai/eval_harness/harness/comparison.pyindex e1c003cdec4a..c0074bacbf2e 100644--- a/products/posthog_ai/eval_harness/harness/comparison.py+++ b/products/posthog_ai/eval_harness/harness/comparison.py@@ -23,6 +23,8 @@ class ModelRun(BaseModel):runtime: strmodel: str+ service_tier: str | None = None+ reasoning_effort: str | None = Noneexit_code: int | None = Nonewall_time_seconds: float = 0report: RunReport | None = None@@ -35,10 +37,24 @@ class ModelComparison:self.directory = (options.comparison_dir or LOGS_ROOT / "comparisons" / f"{datetime.now(UTC):%Y%m%d-%H%M%S}_{uuid4().hex[:8]}").resolve()- self.runs = [ModelRun(runtime=target.runtime, model=target.model) for target in options.comparison_models]+ self.runs = [self._model_run(target) for target in options.comparison_models]self.issues: list[str] = []self._snapshot_report = options.skill_snapshot_report+ def _model_run(self, target: ModelTarget) -> ModelRun:+ return ModelRun(+ runtime=target.runtime,+ model=target.model,+ service_tier=target.service_tier or self.options.service_tier,+ reasoning_effort=target.reasoning_effort or self.options.reasoning_effort,+ )++ @staticmethod+ def _label(run: ModelRun) -> str:+ return ModelTarget(+ runtime=run.runtime, model=run.model, service_tier=run.service_tier, reasoning_effort=run.reasoning_effort+ ).label+def _command(self, target: ModelTarget, path: Path, *, check: bool) -> list[str]:options = self.optionscommand = [@@ -66,9 +82,11 @@ class ModelComparison:str(path),"--no-tui",]+ arm = self._model_run(target)for flag, value in (("--eval", options.case_filter),- ("--reasoning-effort", options.reasoning_effort),+ ("--reasoning-effort", arm.reasoning_effort),+ ("--service-tier", arm.service_tier),("--fail-under", options.fail_under),("--skill-snapshot-report", self._snapshot_report if not check else None),("--skill-snapshot-output-dir", self.directory if not check else None),@@ -134,8 +152,14 @@ class ModelComparison:report = run.reportif report is None:return "No valid run report; inspect the child transcript."- if report.runtime != run.runtime or report.model != run.model or report.exit_code != run.exit_code:- return "Run report does not match the requested model or exit status."+ if (+ report.runtime != run.runtime+ or report.model != run.model+ or report.service_tier != run.service_tier+ or report.reasoning_effort != run.reasoning_effort+ or report.exit_code != run.exit_code+ ):+ return "Run report does not match the requested arm or exit status."if not report.experiments:return "No experiment results."if not report.skills_sha256 or not report.skills_source_sha256 or not report.skills_archive_path:@@ -179,7 +203,7 @@ class ModelComparison:def _text(self) -> str:rows = [- ["Metric", *(f"{run.runtime}/{run.model}" for run in self.runs)],+ ["Metric", *(self._label(run) for run in self.runs)],["Exit status", *(str(run.exit_code) if run.exit_code is not None else "not run" for run in self.runs)],["Skill snapshot",@@ -234,7 +258,7 @@ class ModelComparison:*(" | ".join(value.ljust(widths[index]) for index, value in enumerate(row)).rstrip() for row in rows),]lines.extend(f"NOTE: {issue}" for issue in self.issues)- lines.extend(f"NOTE: {run.runtime}/{run.model}: {run.issue}" for run in self.runs if run.issue)+ lines.extend(f"NOTE: {self._label(run)}: {run.issue}" for run in self.runs if run.issue)return "\n".join(lines) + "\n"def _save(self) -> None:@@ -247,6 +271,7 @@ class ModelComparison:"trials": self.options.trials,"skill_delivery": self.options.skill_delivery,"reasoning_effort": self.options.reasoning_effort,+ "service_tier": self.options.service_tier,"max_sandboxes": self.options.max_sandboxes,"case_timeout": self.options.per_case_timeout_seconds,"infrastructure_retries": self.options.infrastructure_retries,@@ -266,13 +291,13 @@ class ModelComparison:try:checked: list[RunReport] = []for index, target in enumerate(self.options.comparison_models):- write_console(f"MODEL CHECK {index + 1}/{len(self.runs)} {target.runtime}/{target.model}\n")+ write_console(f"MODEL CHECK {index + 1}/{len(self.runs)} {target.label}\n")path = self.directory / f"{index + 1}-check.json"code = self._execute(self._command(target, path, check=True))report = self._read_report(path)if code or report is None or report.suite_kinds != ["sandboxed"]:self.issues.append(- f"{target.runtime}/{target.model}: configuration check failed or selected suites are not all sandboxed. "+ f"{target.label}: configuration check failed or selected suites are not all sandboxed. ""No model runs started. Inspect the child transcript above.")return 1@@ -281,19 +306,19 @@ class ModelComparison:self.issues.append("Suite selections differ. No model runs started.")return 1for index, target in enumerate(self.options.comparison_models):- write_console(f"MODEL START {index + 1}/{len(self.runs)} {target.runtime}/{target.model}\n")+ write_console(f"MODEL START {index + 1}/{len(self.runs)} {target.label}\n")started = time.monotonic()path = self.directory / f"{index + 1}-results.json"try:code = self._execute(self._command(target, path, check=False))except KeyboardInterrupt:code = 130- run = ModelRun(- runtime=target.runtime,- model=target.model,- exit_code=code,- wall_time_seconds=time.monotonic() - started,- report=self._read_report(path),+ run = self._model_run(target).model_copy(+ update={+ "exit_code": code,+ "wall_time_seconds": time.monotonic() - started,+ "report": self._read_report(path),+ })self.runs[index] = run.model_copy(update={"issue": self._validate_run(run)})if (diff --git a/products/posthog_ai/eval_harness/harness/context.py b/products/posthog_ai/eval_harness/harness/context.pyindex b90f799996e5..9fcb5550ffe3 100644--- a/products/posthog_ai/eval_harness/harness/context.py+++ b/products/posthog_ai/eval_harness/harness/context.py@@ -100,3 +100,6 @@ class EvalContext:skills_sha256: str | None = Noneskills_source_sha256: str | None = None++ service_tier: str | None = None+ """OpenAI request tier for the sandboxed agent; ``None`` sends the request untiered."""diff --git a/products/posthog_ai/eval_harness/harness/demo_data.py b/products/posthog_ai/eval_harness/harness/demo_data.pyindex d2c937c42dfa..ce74c1c90cee 100644--- a/products/posthog_ai/eval_harness/harness/demo_data.py+++ b/products/posthog_ai/eval_harness/harness/demo_data.py@@ -45,12 +45,14 @@ class SandboxedDemoData:agent_runtime: str | None = None,reasoning_effort: str | None = None,sandbox_timeout_seconds: int | None = None,+ service_tier: str | None = None,):self.master_team_id = master_team_idself._django_db_blocker = django_db_blockerself.agent_model = agent_modelself.agent_runtime = agent_runtimeself.reasoning_effort = reasoning_effort+ self.service_tier = service_tierself.sandbox_timeout_seconds = sandbox_timeout_secondsdef make_context(@@ -80,6 +82,7 @@ class SandboxedDemoData:model=self.agent_model,runtime_adapter=self.agent_runtime,reasoning_effort=self.reasoning_effort,+ service_tier=self.service_tier,# Under TEST=1, SANDBOX_TTL_SECONDS is 15 minutes — equal to the default# per-case timeout — so a slow Modal case would have its sandbox reaped# exactly as it was finishing. The modal provider passes a larger TTL here.@@ -121,6 +124,7 @@ def ensure_demo_ready(agent_runtime: str,reasoning_effort: str | None,sandbox_timeout_seconds: int | None,+ service_tier: str | None = None,) -> SandboxedDemoData:"""Seed the master Hedgebox team (once) and expose a per-case context factory."""master_team_id = ensure_master_demo_team(blocker)@@ -144,4 +148,5 @@ def ensure_demo_ready(agent_runtime=agent_runtime,reasoning_effort=reasoning_effort,sandbox_timeout_seconds=sandbox_timeout_seconds,+ service_tier=service_tier,)diff --git a/products/posthog_ai/eval_harness/harness/generation_capture.py b/products/posthog_ai/eval_harness/harness/generation_capture.pynew file mode 100644index 000000000000..84f3de6375c0--- /dev/null+++ b/products/posthog_ai/eval_harness/harness/generation_capture.py@@ -0,0 +1,94 @@+"""A local capture endpoint for the eval AI gateway's `$ai_generation` events.++Without a capture URL the gateway only logs a summary of each generation, which leaves out the+served service tier, cache and reasoning tokens, and per-part costs. Pointing it here keeps the+full property set for each eval run in one JSONL file. Prompt and completion bodies are dropped:+the agent log already holds the conversation.+"""++from __future__ import annotations++import json+import logging+import threading+from email.parser import BytesParser+from email.policy import HTTP+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from pathlib import Path+from typing import Any++logger = logging.getLogger(__name__)++CAPTURE_AI_PATH = "/i/v0/ai"++_DROPPED_PROPERTIES = frozenset({"$ai_input", "$ai_output", "$ai_output_choices", "$ai_tools"})+++def parse_generation(content_type: str, body: bytes) -> dict[str, Any] | None:+ """Read the gateway's multipart `event` and `event.properties` parts into one record."""+ message = BytesParser(policy=HTTP).parsebytes(f"Content-Type: {content_type}\r\n\r\n".encode() + body)+ if not message.is_multipart():+ return None+ parts: dict[str, Any] = {}+ for part in message.iter_parts():+ name = part.get_param("name", header="content-disposition")+ payload = part.get_payload(decode=True)+ if isinstance(name, str) and isinstance(payload, bytes):+ parts[name] = json.loads(payload)+ event = parts.get("event")+ properties = parts.get("event.properties")+ if not isinstance(event, dict) or not isinstance(properties, dict):+ return None+ return {+ **event,+ "properties": {key: value for key, value in properties.items() if key not in _DROPPED_PROPERTIES},+ }+++class GenerationCapture:+ """Accepts capture POSTs on localhost and appends each generation to `path`."""++ def __init__(self, path: Path) -> None:+ self.path = path+ self._lock = threading.Lock()+ capture = self++ class Handler(BaseHTTPRequestHandler):+ def do_POST(self) -> None:+ body = self.rfile.read(int(self.headers.get("Content-Length") or 0))+ record = None+ if self.path.split("?", 1)[0] == CAPTURE_AI_PATH:+ try:+ record = parse_generation(self.headers.get("Content-Type", ""), body)+ except (ValueError, UnicodeDecodeError):+ logger.warning("Could not parse a gateway generation event", exc_info=True)+ if record is not None:+ capture.append(record)+ self.send_response(200 if record is not None else 400)+ self.end_headers()++ def log_message(self, format: str, *args: Any) -> None:+ return++ self._server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)+ self._thread = threading.Thread(target=self._server.serve_forever, name="ai-gateway-capture", daemon=True)++ @property+ def url(self) -> str:+ host, port = self._server.server_address[:2]+ return f"http://{host!s}:{port}"++ def append(self, record: dict[str, Any]) -> None:+ line = json.dumps(record, separators=(",", ":"), default=str)+ with self._lock, self.path.open("a", encoding="utf-8") as file:+ file.write(line + "\n")++ def start(self) -> GenerationCapture:+ self.path.parent.mkdir(parents=True, exist_ok=True)+ self.path.touch()+ self._thread.start()+ return self++ def stop(self) -> None:+ self._server.shutdown()+ self._server.server_close()diff --git a/products/posthog_ai/eval_harness/harness/lifecycle.py b/products/posthog_ai/eval_harness/harness/lifecycle.pyindex 4df26dee95ca..5101d4379a61 100644--- a/products/posthog_ai/eval_harness/harness/lifecycle.py+++ b/products/posthog_ai/eval_harness/harness/lifecycle.py@@ -11,9 +11,11 @@ import subprocessfrom collections.abc import Sequencefrom contextlib import AsyncExitStack, ExitStackfrom dataclasses import replace+from datetime import UTC, datetimefrom functools import partialfrom pathlib import Pathfrom typing import TYPE_CHECKING+from uuid import uuid4from unittest.mock import patch@@ -35,6 +37,7 @@ from products.tasks.backend.temporal.process_task.utils import get_reasoning_efffrom ..engines.base import EvalEnginefrom ..engines.registry import resolve_engine+from ..log_sink import LOGS_ROOTfrom .cli import (DEFAULT_ONE_SHOT_CONCURRENCY,DEFAULT_SKILL_DELIVERY,@@ -47,6 +50,7 @@ from .demo_data import SandboxedDemoData, ensure_demo_readyfrom .discovery import MULTI_TURN_MODULE_MARKER, REPO_ROOT, EvalSuite, discover_suitesfrom .django_env import EvalDatabasefrom .env_preflight import validate_eval_env+from .generation_capture import GenerationCapturefrom .kernel_sandboxes import reclaim_kernelsfrom .live_server import EvalLiveServerfrom .offline_delivery import OfflineEvalDelivery@@ -343,7 +347,12 @@ class SandboxedEvalHarness:if Infra.AI_GATEWAY in required:reporter.setup_started("start AI gateway")- self._stack.callback(start_ai_gateway())+ capture = GenerationCapture(+ LOGS_ROOT / "ai_gateway" / f"{datetime.now(UTC):%Y%m%d-%H%M%S}_{uuid4().hex[:8]}.generations.jsonl"+ ).start()+ self._stack.callback(capture.stop)+ reporter.set_ai_gateway_generations(capture.path)+ self._stack.callback(start_ai_gateway(capture_url=capture.url))self._stack.enter_context(override_settings(AI_GATEWAY_REDIS_URL=AI_GATEWAY_REDIS_URL))# Seed before building skills: HogQL examples in skill templates render against a Team,# and on a fresh test database the master team is the first one that exists.@@ -355,6 +364,7 @@ class SandboxedEvalHarness:agent_model=self.options.agent_model,agent_runtime=self.options.agent_runtime,reasoning_effort=self.options.reasoning_effort,+ service_tier=self.options.service_tier,sandbox_timeout_seconds=(self.provider.sandbox_timeout_seconds(self.options.per_case_timeout_seconds)if self.provider is not None@@ -497,9 +507,9 @@ class SandboxedEvalHarness:# The harness never starts run_task_workflow_dispatcher, so an outbox dispatch would# leave every case parked in QUEUED until the case's poll budget runs out.TASKS_WORKFLOW_DISPATCH_OUTBOX_ENABLED=False,- # Chat and scout evals retain their production product attribution. The worker mints each- # run a capped scoped token with this key, as it does in production.- SANDBOX_AI_GATEWAY_PRODUCTS=f"{EVAL_AI_PRODUCT},signals_scout",+ # Chat, scout, and repository-selection evals retain their production product attribution.+ # The worker mints each run a capped scoped token with this key, as it does in production.+ SANDBOX_AI_GATEWAY_PRODUCTS=f"{EVAL_AI_PRODUCT},signals_scout,signals_repo_selection,slack_app",SANDBOX_AI_GATEWAY_MINT_KEY=self._ai_gateway_mint_key,# Tool-side judges share the eval ledger with the sandboxed agent.AI_GATEWAY_URL=f"http://localhost:{AI_GATEWAY_PORT}/v1",@@ -587,6 +597,7 @@ class SandboxedEvalHarness:posthog_trace_host="https://us.posthog.com",posthog_trace_project_id=settings.LLM_ANALYTICS_INTERNAL_TEAM_ID,infrastructure_retries=self.options.infrastructure_retries,+ service_tier=self.options.service_tier,)async def _run_suite(self, suite: EvalSuite, ctx: EvalContext) -> SuiteRunResult:diff --git a/products/posthog_ai/eval_harness/harness/reporting.py b/products/posthog_ai/eval_harness/harness/reporting.pyindex e26ff5dcd559..69676d46b351 100644--- a/products/posthog_ai/eval_harness/harness/reporting.py+++ b/products/posthog_ai/eval_harness/harness/reporting.py@@ -85,6 +85,7 @@ class ProgressReporter:self._revision = revisionself._worktree_dirty = worktree_dirtyself._skills_snapshot: SkillSnapshot | None = None+ self._ai_gateway_generations_path: Path | None = Noneself._total_suites = total_suitesself._verbose = verboseself._lock = asyncio.Lock()@@ -372,6 +373,10 @@ class ProgressReporter:progress = f"[{self._case_counts[experiment_name]}/{total} cases]" if total is not None else ""await asyncio.to_thread(_emit, f"EXPERIMENT DONE {experiment_name} {progress}")+ def set_ai_gateway_generations(self, path: Path) -> None:+ self._ai_gateway_generations_path = path+ _emit(f"AI GATEWAY GENERATIONS {path}")+def set_skill_snapshot(self, snapshot: SkillSnapshot) -> None:self._skills_snapshot = snapshot_emit(f"SKILLS SNAPSHOT sha256={snapshot.sha256} manifest={snapshot.manifest_path}")@@ -405,6 +410,7 @@ class ProgressReporter:runtime=options.agent_runtime,model=options.agent_model,reasoning_effort=options.reasoning_effort,+ service_tier=options.service_tier,provider=options.provider,trials=options.trials,suite_ids=[suite.id for suite in suites],@@ -420,6 +426,9 @@ class ProgressReporter:else None,skills_source_sha256=self._skills_snapshot.source_sha256 if self._skills_snapshot is not None else None,skills_archive_path=str(self._skills_snapshot.archive_path) if self._skills_snapshot is not None else None,+ ai_gateway_generations_path=str(self._ai_gateway_generations_path)+ if self._ai_gateway_generations_path is not None+ else None,)options.results_file.parent.mkdir(parents=True, exist_ok=True)temporary = options.results_file.with_suffix(".tmp")diff --git a/products/posthog_ai/eval_harness/harness/run_report.py b/products/posthog_ai/eval_harness/harness/run_report.pyindex 0b54b55f7a99..49c5c36562fc 100644--- a/products/posthog_ai/eval_harness/harness/run_report.py+++ b/products/posthog_ai/eval_harness/harness/run_report.py@@ -31,6 +31,7 @@ class RunReport(BaseModel):runtime: strmodel: strreasoning_effort: str | None+ service_tier: str | None = Noneprovider: strtrials: intsuite_ids: list[str]@@ -43,4 +44,5 @@ class RunReport(BaseModel):skills_manifest_path: str | None = Noneskills_source_sha256: str | None = Noneskills_archive_path: str | None = None+ ai_gateway_generations_path: str | None = Noneexperiments: dict[str, ExperimentReport] = Field(default_factory=dict)diff --git a/products/posthog_ai/eval_harness/harness/services.py b/products/posthog_ai/eval_harness/harness/services.pyindex 211133d9ca33..c35784b3f5cb 100644--- a/products/posthog_ai/eval_harness/harness/services.py+++ b/products/posthog_ai/eval_harness/harness/services.py@@ -115,7 +115,7 @@ def _ai_gateway_compose(*args: str) -> subprocess.CompletedProcess[str]:)-def start_ai_gateway() -> Callable[[], None]:+def start_ai_gateway(capture_url: str | None = None) -> Callable[[], None]:"""Start the Go ai-gateway on a non-default port, in resolver mode.It shares the Postgres and Valkey containers of the gateway repo's compose file@@ -154,6 +154,17 @@ def start_ai_gateway() -> Callable[[], None]:"AI_GATEWAY_LOG_LEVEL": "info",})+ if capture_url is not None:+ env["AI_GATEWAY_CAPTURE_URL"] = capture_url+ # The gateway emits to capture only signed events, so the local sink needs a secret too.+ env["AI_GATEWAY_SIGNING_SECRET"] = secrets.token_urlsafe(32)+ # Open-weight models such as DeepSeek run on Baseten. The host is declared only with a key,+ # because the gateway refuses to boot a declared host that has no credential.+ if baseten_api_key := os.environ.get("AI_GATEWAY_BASETEN_API_KEY"):+ env["AI_GATEWAY_BASETEN_API_KEY"] = baseten_api_key+ env["AI_GATEWAY_SERVED_HOSTS"] = json.dumps(+ [{"id": "baseten", "kind": "baseten", "base_url": "https://inference.baseten.co/v1"}]+ )binary = _ai_gateway_binary()migrate = subprocess.run([str(binary), "migrate"], env=env, capture_output=True, text=True, timeout=120)if migrate.returncode != 0:diff --git a/products/posthog_ai/eval_harness/test/test_generation_capture.py b/products/posthog_ai/eval_harness/test/test_generation_capture.pynew file mode 100644index 000000000000..c0d1bdda6959--- /dev/null+++ b/products/posthog_ai/eval_harness/test/test_generation_capture.py@@ -0,0 +1,34 @@+import json++from products.posthog_ai.eval_harness.harness.generation_capture import parse_generation+++def test_parse_generation_keeps_tier_and_usage_and_drops_bodies():+ boundary = "b0undary"+ event = {"uuid": "u-1", "event": "$ai_generation", "distinct_id": "user-1", "timestamp": "2026-01-01T00:00:00Z"}+ properties = {+ "$ai_service_tier": "flex",+ "$ai_service_tier_requested": "flex",+ "$ai_cache_read_input_tokens": 1200,+ "$ai_input": [{"role": "user", "content": "x" * 50}],+ "$ai_output_choices": [{"role": "assistant", "content": "y"}],+ }+ body = (+ "".join(+ f'--{boundary}\r\nContent-Disposition: form-data; name="{name}"\r\nContent-Type: application/json\r\n\r\n'+ f"{json.dumps(part)}\r\n"+ for name, part in (("event", event), ("event.properties", properties))+ )+ + f"--{boundary}--\r\n"+ )++ record = parse_generation(f"multipart/form-data; boundary={boundary}", body.encode())++ assert record == {+ **event,+ "properties": {+ "$ai_service_tier": "flex",+ "$ai_service_tier_requested": "flex",+ "$ai_cache_read_input_tokens": 1200,+ },+ }diff --git a/products/posthog_ai/eval_harness/test/test_model_comparison.py b/products/posthog_ai/eval_harness/test/test_model_comparison.pyindex c0508090cf3a..63329a5ca2df 100644--- a/products/posthog_ai/eval_harness/test/test_model_comparison.py+++ b/products/posthog_ai/eval_harness/test/test_model_comparison.py@@ -202,6 +202,53 @@ def test_model_comparison_preserves_settings_and_partial_results(assert saved["runs"][1]["report"] is not None+def test_comparison_arms_forward_their_own_tier_and_effort(tmp_path: Path) -> None:+ options = parse_args(+ [+ "example/suite",+ "--comparison-dir",+ str(tmp_path / "comparison"),+ "--reasoning-effort",+ "medium",+ "--compare-model",+ "claude:claude-example",+ "--compare-model",+ "codex:gpt-example",+ "--compare-model",+ "codex:gpt-example,service_tier=flex,reasoning_effort=low",+ ]+ )+ comparison = ModelComparison(options)+ children = [+ parse_args(comparison._command(target, tmp_path / "report.json", check=False)[3:])+ for target in options.comparison_models+ ]++ assert [(child.agent_model, child.service_tier, child.reasoning_effort) for child in children] == [+ ("claude-example", None, "medium"),+ ("gpt-example", None, "medium"),+ ("gpt-example", "flex", "low"),+ ]+ flex_run = comparison.runs[2].model_copy(+ update={+ "exit_code": 0,+ "report": RunReport(+ runtime="codex",+ model="gpt-example",+ reasoning_effort="low",+ service_tier=None,+ provider="docker",+ trials=1,+ suite_ids=[],+ suite_kinds=["sandboxed"],+ exit_code=0,+ duration_seconds=1,+ ),+ }+ )+ assert comparison._validate_run(flex_run) == "Run report does not match the requested arm or exit status."++@pytest.mark.parametrize("interrupt", [None, "keyboard", "sigterm"])def test_comparison_streams_child_output_and_waits_for_interrupt_cleanup(interrupt: str | None,diff --git a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_cli.py b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_cli.pyindex 9debffb953d9..f44b40d17b5f 100644--- a/products/posthog_ai/eval_harness/test/test_sandboxed_harness_cli.py+++ b/products/posthog_ai/eval_harness/test/test_sandboxed_harness_cli.py@@ -34,6 +34,26 @@ def test_model_comparison_arguments(arguments: list[str], expected: list[tuple[sassert [(target.runtime, target.model) for target in parse_args(arguments).comparison_models] == expected+ "arm,expected",+ [+ ("codex:gpt-example,service_tier=flex", ("gpt-example", "flex", None)),+ ("codex:gpt-example,reasoning_effort=low,service_tier=default", ("gpt-example", "default", "low")),+ ("codex:gpt-example,service_tier=batch", None),+ ("codex:gpt-example,service_tier=flex,service_tier=default", None),+ ("codex:gpt-example,temperature=0", None),+ ],+)+def test_comparison_arm_settings(arm: str, expected: tuple[str, str | None, str | None] | None) -> None:+ arguments = ["suite", "--compare-model", "codex:gpt-example", "--compare-model", arm]+ if expected is None:+ with pytest.raises(SystemExit):+ parse_args(arguments)+ return+ target = parse_args(arguments).comparison_models[1]+ assert (target.model, target.service_tier, target.reasoning_effort) == expected++@pytest.mark.parametrize("argv,expected_delivery",[diff --git a/products/slack_app/backend/temporal/README.md b/products/slack_app/backend/temporal/README.mdindex f077719ca945..107b77084538 100644--- a/products/slack_app/backend/temporal/README.md+++ b/products/slack_app/backend/temporal/README.md@@ -4,9 +4,9 @@ The Slack app workflows and activities run on `settings.TASKS_TASK_QUEUE`.The worker registers them through `products/slack_app/backend/facade/temporal.py`.For general Temporal conventions, read [`posthog/temporal/AGENTS.md`](../../../../posthog/temporal/AGENTS.md).-## Local eval scripts+## Evals-- [`eval_slack_repo_selection.py`](./eval_slack_repo_selection.py) — exercises the Slack `@PostHog` repo-selection cascade (cascade → Haiku gate → discovery agent) against a real team with a connected GitHub integration. Pass/fail summary, no Slack needed. Run as a file (`python products/slack_app/backend/temporal/eval_slack_repo_selection.py --list-cases`), not via `python -m` — the latter would force `backend/temporal/__init__.py` to load activities before `django.setup()`.+- Repository selection for `@PostHog` mentions (cascade → needs-repo classifier → discovery agent) is evaluated by the harness suite [`slack_app/eval_repo_selection_routing`](../../evals/eval_repo_selection_routing.py) on a synthetic repository catalog, with no real team or Slack traffic. Run it with `hogli evals slack_app/eval_repo_selection_routing`; see the [eval harness guide](../../../posthog_ai/eval_harness/README.md#repository-selection-evals).## Checking the Slack repo discovery agentdiff --git a/products/slack_app/backend/temporal/eval_slack_repo_selection.py b/products/slack_app/backend/temporal/eval_slack_repo_selection.pydeleted file mode 100644index db767f149b08..000000000000--- a/products/slack_app/backend/temporal/eval_slack_repo_selection.py+++ /dev/null@@ -1,694 +0,0 @@-"""Local eval for the Slack repo-selection flow — the canonical "how to test locally" tool.--Replicates the production cascade → Haiku gate → discovery agent path against a real-team with a real GitHub integration, without needing a Slack Integration row or any-actual Slack traffic. Each case declares its expected terminal stage/outcome;-the eval prints a pass/fail summary so you can re-run after touching repo selection-and spot regressions immediately.--Lives next to the Slack workflow it exercises (`slack_app_mention_processing.py`) rather-than in `posthog/management/commands/` because (a) it's co-located with the thing it-tests and (b) `products/slack_app/` can't tach-depend on `products/tasks/`, which the-eval needs to import from.--# Prerequisites--- `DEBUG=1` (refuses to run otherwise — burns sandbox + LLM credits)-- A team with a connected GitHub integration that has >=2 cached repos-- Dev stack up (sandbox manager + temporal + LLM creds): `./bin/start` / `hogli start`--# Usage-- # Full eval (~3-6 min — most cost is the agent runs)- python posthog/temporal/ai/eval_slack_repo_selection.py --team-id 1 --user-id 1-- # Iterate fast on cascade-only changes (no LLM at all)- python posthog/temporal/ai/eval_slack_repo_selection.py --team-id 1 --user-id 1 --skip-llm-- # Iterate on Haiku changes without paying for agent runs- python posthog/temporal/ai/eval_slack_repo_selection.py --team-id 1 --user-id 1 --skip-agent-- # Single case- python posthog/temporal/ai/eval_slack_repo_selection.py --team-id 1 --user-id 1 --case vague_code_bug-- # See what the Slack picker message renders on each failure mode- python posthog/temporal/ai/eval_slack_repo_selection.py --team-id 1 --user-id 1 --case vague_code_bug \\- --show-picker-guidance-- # See the case catalogue without running anything- python posthog/temporal/ai/eval_slack_repo_selection.py --list-cases--# Reading the output--Each case prints what each stage decides, then the summary tallies PASS / FAIL / SKIP.-For `agent → found` cases, the agent's `reason` text is the quality signal — it should-cite cache evidence (tree_paths matches, README excerpts), not vibes.--To force the failure modes the picker fallback handles:-- `RepoSelectionUnavailableError`: temporarily set `archived=True` on all rows in- `IntegrationRepositoryCacheEntry` for the team, run any agent case.-- `RepoSelectionRejectedError`: hard to force reliably; use `--show-picker-guidance`- to preview what the user would see if it did fire.--# Routing rules--Cases with `routing_rules` create temporary `RepoRoutingRule` rows for the team before-running and delete them afterwards. Rules the team already has stay in place and reach-both the needs-repo gate and the agent prompt on every case, so a heavily-rule-configured team can shift the expected outcomes of unrelated cases — including-haiku-stage cases, because any configured rule disables the product-term heuristic.-"""--# ruff: noqa: T201, E402--from __future__ import annotations--import os-import sys-import asyncio-import argparse-import textwrap-from dataclasses import dataclass, field, replace-from pathlib import Path-from typing import Literal--# Must run as `python products/slack_app/backend/temporal/eval_slack_repo_selection.py` (not `python -m ...`):-# `python -m` would import `products/slack_app/backend/temporal/__init__.py` first, which loads activities that-# reference Django models before django.setup() has a chance to fire.-_repo_root = Path(__file__).resolve().parents[4]-if str(_repo_root) not in sys.path:- sys.path.insert(0, str(_repo_root))--os.environ.setdefault("DJANGO_SETTINGS_MODULE", "posthog.settings")--import django--django.setup()--from django.conf import settings--from posthog.egress.limiter.policies import Priority-from posthog.models import Team-from posthog.models.repo_routing_rule import RepoRoutingRule--from products.slack_app.backend.api import _extract_explicit_repo, _extract_explicit_repo_from_thread-from products.slack_app.backend.services.slack_messages import SlackThreadMessage-from products.slack_app.backend.temporal import POSTHOG_CODE_SLACK_MENTION_PICKER_GUIDANCE-from products.slack_app.backend.temporal.activities.classifiers import classify_task_needs_repo, team_routing_rule_lines-from products.tasks.backend.facade import api as tasks_facade-from products.tasks.backend.facade.repo_selection import (- RepoSelectionRejectedError,- RepoSelectionResult,- RepoSelectionUnavailableError,- resolve_team_github_integration,- select_repository,-)--Stage = Literal["cascade", "haiku", "agent", "skipped"]-Outcome = Literal[- "auto", # cascade picked a repo (explicit mention / single repo)- "no_repo", # cascade or Haiku decided no repo is needed- "found", # agent picked a repo- "no_match", # agent returned None (no plausible candidate)- "rejected", # agent returned a repo not in the candidate list- "unavailable", # agent raised RepoSelectionUnavailableError- "crashed", # agent raised unexpected exception- "skipped", # case wasn't run due to --skip-* flag-]---@dataclass(frozen=True)-class Case:- name: str- description: str- # `{first_repo}`/`{second_repo}` are substituted with the team's connected repos at runtime.- text_template: str- thread_messages: list[SlackThreadMessage]- expected_stage: Stage- expected_outcome: Outcome- # Optional human note explaining current behavior quirks.- note: str = ""- # `(rule_text, repository_template)` pairs created as temporary RepoRoutingRule rows for- # this case and deleted afterwards.- routing_rules: tuple[tuple[str, str], ...] = ()- # When set, the case only passes if the picked repo equals this (template-substituted).- expected_repo_template: str = ""---@dataclass(frozen=True)-class CaseResult:- case: Case- actual_stage: Stage- actual_outcome: Outcome- detail: str = "" # repo name, error type, etc.- expected_repo: str = "" # resolved from `expected_repo_template`; empty = don't check the repo-- @property- def status(self) -> Literal["PASS", "FAIL", "SKIP"]:- if self.actual_stage == "skipped":- return "SKIP"- if (self.actual_stage, self.actual_outcome) != (self.case.expected_stage, self.case.expected_outcome):- return "FAIL"- if self.expected_repo and self.detail.lower() != self.expected_repo.lower():- return "FAIL"- return "PASS"---# Cases are parameterized so they work across teams; `{first_repo}` is substituted at runtime.-# Expected outcomes match observed behavior on master + this PR — the eval is a regression-# baseline. Cases with `note` fields call out known quirks.-CASES: list[Case] = [- # --- Cascade short-circuit -------------------------------------------------- Case(- name="explicit_mention",- description="Cascade picks the repo directly when the text contains a connected org/repo.",- text_template="@PostHog can you look at {first_repo} and fix the readme typo",- thread_messages=[- SlackThreadMessage(user="tester", text="@PostHog can you look at {first_repo} and fix the readme typo")- ],- expected_stage="cascade",- expected_outcome="auto",- ),- Case(- name="ci_run_link",- description="Cascade reads the repo out of a workflow-run link, so a CI ask never reaches the agent.",- text_template="@PostHog is this flaky? https://github.com/{first_repo}/actions/runs/30560492835",- thread_messages=[- SlackThreadMessage(- user="tester", text="@PostHog is this flaky? https://github.com/{first_repo}/actions/runs/30560492835"- )- ],- expected_stage="cascade",- expected_outcome="auto",- ),- Case(- name="ci_run_link_earlier_in_the_thread",- description="Cascade reads the repo from a link someone posted before the mention, the usual shape of a CI ask.",- text_template="@PostHog is this one flaky?",- thread_messages=[- SlackThreadMessage(- user="tester", text="https://github.com/{first_repo}/actions/runs/30560492835 went red again"- ),- SlackThreadMessage(user="tester", text="@PostHog is this one flaky?"),- ],- expected_stage="cascade",- expected_outcome="auto",- ),- # --- Haiku gate short-circuits (heuristic + LLM) ---------------------------- Case(- name="billing_question",- description="Haiku LLM should classify as no-repo (billing/account question).",- text_template="@PostHog how do I update the credit card on our subscription?",- thread_messages=[- SlackThreadMessage(user="tester", text="@PostHog how do I update the credit card on our subscription?")- ],- expected_stage="haiku",- expected_outcome="no_repo",- ),- Case(- name="dashboard_config",- description="Haiku heuristic short-circuits on 'dashboard' with no explicit code pattern.",- text_template="@PostHog the dashboard tile filters are not persisting across refreshes",- thread_messages=[- SlackThreadMessage(- user="tester", text="@PostHog the dashboard tile filters are not persisting across refreshes"- )- ],- expected_stage="haiku",- expected_outcome="no_repo",- ),- Case(- name="marketing_site",- description="Perf complaint about a site the team likely owns code for — agent should route to docs/marketing repo.",- text_template="@PostHog the docs site loads really slowly on mobile",- thread_messages=[- SlackThreadMessage(user="tester", text="@PostHog the docs site loads really slowly on mobile"),- SlackThreadMessage(user="other", text="yeah I noticed the same on /docs/getting-started"),- ],- expected_stage="agent",- expected_outcome="found",- ),- Case(- name="sdk_specific_trace_trip",- description="App/SDK crash with stack trace — agent should route to the relevant SDK repo (e.g. posthog-ios).",- text_template="@PostHog the iOS SDK is crashing on app launch after upgrade to 3.19",- thread_messages=[- SlackThreadMessage(- user="tester", text="@PostHog the iOS SDK is crashing on app launch after upgrade to 3.19"- ),- SlackThreadMessage(user="other", text="stack trace shows PostHogReplay.start() failing"),- ],- expected_stage="agent",- expected_outcome="found",- ),- Case(- name="posthog_product_hang",- description="Complaint about a PostHog product page hanging — looks like 'broken page' but it's the SaaS, not the team's code.",- text_template="@PostHog the trends page hangs forever when I add 10+ series",- thread_messages=[- SlackThreadMessage(user="tester", text="@PostHog the trends page hangs forever when I add 10+ series"),- SlackThreadMessage(user="other", text="tried Firefox and Chrome, just a spinner"),- ],- expected_stage="haiku",- expected_outcome="no_repo",- ),- # --- Agent path (the new logic this PR introduces) -------------------------- Case(- name="sdk_instrumentation_on_own_site",- description="PostHog SDK behaving oddly on the team's own site — the issue is in their code, not in PostHog.",- text_template="@PostHog autocapture isn't picking up clicks on our checkout button, other buttons work",- thread_messages=[- SlackThreadMessage(- user="tester",- text="@PostHog autocapture isn't picking up clicks on our checkout button, other buttons work",- ),- SlackThreadMessage(user="other", text="we just shipped a redesign yesterday"),- ],- expected_stage="agent",- expected_outcome="found",- ),- Case(- name="wrong_data_tracking_bug",- description="'Wrong data in PostHog' — almost always a tracking bug in the team's own code, not PostHog.",- text_template="@PostHog we're not seeing any signup_completed events even though we tested signups today",- thread_messages=[- SlackThreadMessage(- user="tester",- text="@PostHog we're not seeing any signup_completed events even though we tested signups today",- ),- SlackThreadMessage(user="other", text="the funnel shows 0 conversions but our team manually completed 5"),- ],- expected_stage="agent",- expected_outcome="found",- ),- Case(- name="vague_code_bug",- description="Vague but code-flavored; agent should disambiguate from cache.",- text_template="@PostHog there's a bug in how we render user signup, can you fix it",- thread_messages=[- SlackThreadMessage(- user="tester", text="@PostHog there's a bug in how we render user signup, can you fix it"- )- ],- expected_stage="agent",- expected_outcome="found",- ),- Case(- name="api_viewset",- description="Explicit code pattern ('viewset') bypasses heuristic; agent picks the API repo.",- text_template="@PostHog the /api/projects/ viewset crashes on large payloads, can you fix it",- thread_messages=[- SlackThreadMessage(- user="tester", text="@PostHog the /api/projects/ viewset crashes on large payloads, can you fix it"- )- ],- expected_stage="agent",- expected_outcome="found",- ),- Case(- name="explicit_code_file",- description="Explicit '.tsx' file extension bypasses heuristic; agent picks the frontend repo.",- text_template="@PostHog add a Cancel button to the signup form in the .tsx component",- thread_messages=[- SlackThreadMessage(- user="tester", text="@PostHog add a Cancel button to the signup form in the .tsx component"- )- ],- expected_stage="agent",- expected_outcome="found",- ),- Case(- name="refactor_request",- description="No debug terms, code-flavored verb; Haiku LLM should allow, agent picks.",- text_template="@PostHog please refactor the user permission check into a single helper",- thread_messages=[- SlackThreadMessage(- user="tester", text="@PostHog please refactor the user permission check into a single helper"- )- ],- expected_stage="agent",- expected_outcome="found",- ),- # --- Routing rules (steer the agent when the ask names no repo) -------------- Case(- name="routing_rule_steers_vague_ask",- description="A configured routing rule steers an ambiguous ask to its repository.",- text_template="@PostHog the internal support desk search endpoint is throwing 500s, can you fix it",- thread_messages=[- SlackThreadMessage(- user="tester",- text="@PostHog the internal support desk search endpoint is throwing 500s, can you fix it",- )- ],- expected_stage="agent",- expected_outcome="found",- routing_rules=(("Anything about the internal support desk", "{second_repo}"),),- expected_repo_template="{second_repo}",- note="The rule is the only signal linking 'support desk' to that repo — nothing in the "- "repo caches mentions it — so a pass means the rule reached and steered the agent.",- ),- Case(- name="routing_rule_overrides_no_repo_gate",- description="A routing rule keeps a product-term ask ('dashboard') from stopping at the no-repo gate.",- text_template="@PostHog the internal metrics dashboard shows a blank page, can you fix it",- thread_messages=[- SlackThreadMessage(- user="tester", text="@PostHog the internal metrics dashboard shows a blank page, can you fix it"- )- ],- expected_stage="agent",- expected_outcome="found",- routing_rules=(("The internal metrics dashboard", "{second_repo}"),),- expected_repo_template="{second_repo}",- note="'dashboard' trips the classifier's product-term heuristic when the team has no rules, "- "so a pass means configured rules reached the needs-repo gate and carried the ask through.",- ),- Case(- name="routing_rule_loses_to_explicit_mention",- description="An explicit org/repo in the text wins in the cascade; the rule never reaches the agent.",- text_template="@PostHog can you look at {first_repo} and fix the readme typo",- thread_messages=[- SlackThreadMessage(user="tester", text="@PostHog can you look at {first_repo} and fix the readme typo")- ],- expected_stage="cascade",- expected_outcome="auto",- routing_rules=(("Route every request to the second repo", "{second_repo}"),),- expected_repo_template="{first_repo}",- ),-]---@dataclass-class RunFlags:- skip_llm: bool = False- skip_agent: bool = False- show_picker: bool = False---@dataclass(frozen=True)-class TeamContext:- team: Team- team_id: int- user_id: int- all_repos: list[str]- first_repo: str- second_repo: str- results: list[CaseResult] = field(default_factory=list)-- @property- def repo_substitutions(self) -> dict[str, str]:- return {"first_repo": self.first_repo, "second_repo": self.second_repo}---class CommandError(Exception):- """Raised on user-visible misconfiguration — translated to exit 1 in main()."""---class _Style:- """Minimal stand-in for Django BaseCommand.style — just enough for our colored output."""-- @staticmethod- def _wrap(code: int, s: str) -> str:- return f"\033[{code}m{s}\033[0m"-- def SUCCESS(self, s: str) -> str:- return self._wrap(32, s)-- def ERROR(self, s: str) -> str:- return self._wrap(31, s)-- def WARNING(self, s: str) -> str:- return self._wrap(33, s)-- def MIGRATE_HEADING(self, s: str) -> str:- return self._wrap(36, s)-- def HTTP_INFO(self, s: str) -> str:- return self._wrap(35, s)---class _Stdout:- def write(self, s: str = "") -> None:- print(s)---class Command:- """Eval runner — keeps the BaseCommand-shaped API so the body reads identically to a manage.py command."""-- def __init__(self) -> None:- self.stdout = _Stdout()- self.style = _Style()-- def add_arguments(self, parser: argparse.ArgumentParser) -> None:- parser.add_argument("--team-id", type=int, required=False, help="Team ID with a GitHub integration.")- parser.add_argument("--user-id", type=int, required=False, help="User ID to attribute sandbox runs to.")- parser.add_argument("--case", type=str, default=None, help="Run a single case by name.")- parser.add_argument("--list-cases", action="store_true", help="Print available cases and exit.")- parser.add_argument(- "--skip-llm",- action="store_true",- help="Stop after cascade. No LLM calls. Iterate fast on _extract_explicit_repo changes.",- )- parser.add_argument(- "--skip-agent",- action="store_true",- help="Stop after Haiku. No sandbox runs. Iterate on classify_task_needs_repo changes.",- )- parser.add_argument(- "--show-picker-guidance",- action="store_true",- help="After each agent run, preview the picker guidance string for each failure mode.",- )-- def handle(self, *args, **options) -> None:- if options["list_cases"]:- for case in CASES:- self.stdout.write(f" {case.name:24s} → {case.expected_stage}/{case.expected_outcome}")- self.stdout.write(f" {'':24s} {case.description}")- return-- if not settings.DEBUG:- raise CommandError("Refusing to run outside DEBUG mode — this consumes sandbox + LLM credits.")- if options["team_id"] is None or options["user_id"] is None:- raise CommandError("--team-id and --user-id are required (unless --list-cases).")-- flags = RunFlags(- skip_llm=options["skip_llm"],- skip_agent=options["skip_agent"],- show_picker=options["show_picker_guidance"],- )- ctx = self._build_context(team_id=options["team_id"], user_id=options["user_id"])-- cases_to_run = [c for c in CASES if c.name == options["case"]] if options["case"] else CASES- if options["case"] and not cases_to_run:- raise CommandError(f"Unknown case '{options['case']}'. Use --list-cases to see options.")-- for case in cases_to_run:- result = self._run_case(case, ctx=ctx, flags=flags)- ctx.results.append(result)- if flags.show_picker and result.actual_stage == "agent":- self._print_picker_previews()-- self._print_summary(ctx.results)-- # --- Setup ------------------------------------------------------------------ def _build_context(self, *, team_id: int, user_id: int) -> TeamContext:- team = Team.objects.filter(id=team_id).first()- if team is None:- raise CommandError(f"Team {team_id} not found.")-- github = resolve_team_github_integration(team_id)- if github is None:- raise CommandError(- f"Team {team_id} has no GitHub integration. Connect one (with >=2 repos) before running."- )- all_repos = sorted(- {- entry["full_name"]- for entry in github.list_all_cached_repositories(max_repos=1000)- if entry.get("full_name")- }- )- if len(all_repos) < 2:- raise CommandError(f"Team {team_id} has {len(all_repos)} connected repo(s). Agent path requires >=2.")-- self.stdout.write("")- self.stdout.write(self.style.MIGRATE_HEADING(f"Team: {team.name} (id={team.id})"))- self.stdout.write(f"Connected repos ({len(all_repos)}):")- for repo in all_repos[:20]:- self.stdout.write(f" - {repo}")- if len(all_repos) > 20:- self.stdout.write(f" ... and {len(all_repos) - 20} more")- self.stdout.write("")-- return TeamContext(- team=team,- team_id=team_id,- user_id=user_id,- all_repos=all_repos,- first_repo=all_repos[0],- second_repo=all_repos[1],- )-- # --- Case execution --------------------------------------------------------- def _run_case(self, case: Case, *, ctx: TeamContext, flags: RunFlags) -> CaseResult:- substitutions = ctx.repo_substitutions- text = case.text_template.format(**substitutions)- thread_messages = [replace(msg, text=msg.text.format(**substitutions)) for msg in case.thread_messages]-- self.stdout.write(self.style.MIGRATE_HEADING(f"── {case.name} ──"))- self.stdout.write(f" text: {text}")- self.stdout.write(f" expected: {case.expected_stage}/{case.expected_outcome}")-- rule_ids = []- for priority, (rule_text, repo_template) in enumerate(case.routing_rules):- rule = RepoRoutingRule.objects.create(- team_id=ctx.team_id,- rule_text=rule_text,- repository=repo_template.format(**substitutions),- priority=priority,- )- rule_ids.append(rule.id)- self.stdout.write(f" rule: {rule.rule_text} → {rule.repository} (temporary)")-- try:- result = self._run_stages(case, text, thread_messages, ctx=ctx, flags=flags)- finally:- if rule_ids:- RepoRoutingRule.objects.filter(id__in=rule_ids).delete()-- if case.expected_repo_template:- result = replace(result, expected_repo=case.expected_repo_template.format(**substitutions))- return result-- def _run_stages(- self, case: Case, text: str, thread_messages: list[SlackThreadMessage], *, ctx: TeamContext, flags: RunFlags- ) -> CaseResult:- # Stage 1: cascade (synchronous, no LLM). Mirrors `cascade_posthog_code_repository_activity`,- # because reading only the mention here would pass cases that production sends to the agent.- explicit = _extract_explicit_repo(text, ctx.all_repos) or _extract_explicit_repo_from_thread(- thread_messages, ctx.all_repos- )- if explicit:- self.stdout.write(self.style.SUCCESS(f" cascade → auto: {explicit}"))- return CaseResult(case=case, actual_stage="cascade", actual_outcome="auto", detail=explicit)- self.stdout.write(" cascade → needs_agent (no explicit mention, >=2 repos)")-- if flags.skip_llm:- self.stdout.write(self.style.WARNING(" skipped (--skip-llm)"))- return CaseResult(case=case, actual_stage="skipped", actual_outcome="skipped")-- # Stage 2: Haiku gate (heuristic + LLM), with the team's routing rules like the activity.- needs_repo = classify_task_needs_repo(- text,- thread_messages,- routing_rules=team_routing_rule_lines(- ctx.team_id, candidate_repos={repo.lower() for repo in ctx.all_repos}- ),- )- if not needs_repo:- self.stdout.write(self.style.SUCCESS(" haiku → no_repo (task doesn't need code)"))- return CaseResult(case=case, actual_stage="haiku", actual_outcome="no_repo")- self.stdout.write(" haiku → needs_repo")-- if flags.skip_agent:- self.stdout.write(self.style.WARNING(" skipped (--skip-agent)"))- return CaseResult(case=case, actual_stage="skipped", actual_outcome="skipped")-- # Stage 3: discovery agent (full sandbox)- self.stdout.write(" agent → running (30-60s)...")- context = "\n".join(f"{msg.user}: {msg.text}" for msg in thread_messages)- try:- result: RepoSelectionResult = asyncio.run(- select_repository(- team_id=ctx.team_id,- user_id=ctx.user_id,- context=context,- origin_product=tasks_facade.TaskOriginProduct.SLACK,- priority=Priority.BATCH,- )- )- if result.repository is None:- self.stdout.write(self.style.WARNING(f" agent → no_match: {result.reason}"))- return CaseResult(case=case, actual_stage="agent", actual_outcome="no_match", detail=result.reason)- self.stdout.write(self.style.SUCCESS(f" agent → found: {result.repository}"))- self.stdout.write(f" reason: {self._wrap(result.reason)}")- return CaseResult(case=case, actual_stage="agent", actual_outcome="found", detail=result.repository)- except RepoSelectionRejectedError as exc:- self.stdout.write(self.style.ERROR(f" agent → rejected (hallucinated '{exc.returned_repository}')"))- return CaseResult(- case=case, actual_stage="agent", actual_outcome="rejected", detail=exc.returned_repository- )- except RepoSelectionUnavailableError as exc:- self.stdout.write(self.style.ERROR(f" agent → unavailable: {exc.reason}"))- return CaseResult(case=case, actual_stage="agent", actual_outcome="unavailable", detail=exc.reason)- except Exception as exc:- self.stdout.write(self.style.ERROR(f" agent → crashed ({type(exc).__name__}): {exc}"))- return CaseResult(case=case, actual_stage="agent", actual_outcome="crashed", detail=type(exc).__name__)-- # --- Output helpers --------------------------------------------------------- def _print_picker_previews(self) -> None:- self.stdout.write(" picker preview (what the user sees on each agent failure):")- for label, reason in [- ("rejected", "Agent returned an invalid repository: posthog/foo"),- (- "unavailable",- "Repo selection unavailable: No connected GitHub repositories are eligible (archived or missing cache data).",- ),- ("crashed", "Agent failed: TimeoutError"),- ]:- picker_guidance = f"_{reason}_\n\n{POSTHOG_CODE_SLACK_MENTION_PICKER_GUIDANCE}"- self.stdout.write(self.style.HTTP_INFO(f" [{label}]"))- for line in picker_guidance.splitlines():- self.stdout.write(f" {line}")-- def _print_summary(self, results: list[CaseResult]) -> None:- self.stdout.write("")- self.stdout.write(self.style.MIGRATE_HEADING("── summary ──"))-- passed = sum(1 for r in results if r.status == "PASS")- failed = sum(1 for r in results if r.status == "FAIL")- skipped = sum(1 for r in results if r.status == "SKIP")-- for r in results:- badge = {"PASS": self.style.SUCCESS, "FAIL": self.style.ERROR, "SKIP": self.style.WARNING}[r.status]- actual = f"{r.actual_stage}/{r.actual_outcome}"- detail = f" ({r.detail})" if r.detail and r.status != "SKIP" else ""- self.stdout.write(f" {badge(r.status):>14s} {r.case.name:24s} → {actual}{detail}")- if r.status == "FAIL":- expected_repo = f" ({r.expected_repo})" if r.expected_repo else ""- self.stdout.write(f" expected {r.case.expected_stage}/{r.case.expected_outcome}{expected_repo}")- if r.case.note and r.status != "SKIP":- self.stdout.write(- textwrap.fill(r.case.note, width=100, initial_indent=" note: ", subsequent_indent=" " * 14)- )-- self.stdout.write("")- total = len(results)- self.stdout.write(f" {passed} passed, {failed} failed, {skipped} skipped ({total} total)")-- @staticmethod- def _wrap(text: str, width: int = 90, indent: str = " ") -> str:- return textwrap.fill(text, width=width, subsequent_indent=indent)---def main() -> None:- parser = argparse.ArgumentParser(- prog="python posthog/temporal/ai/eval_slack_repo_selection.py",- description="Eval the Slack repo selection flow (cascade → Haiku gate → agent) on a real team.",- )- cmd = Command()- cmd.add_arguments(parser)- options = vars(parser.parse_args())- try:- cmd.handle(**options)- except CommandError as exc:- print(f"error: {exc}", file=sys.stderr)- sys.exit(1)---if __name__ == "__main__":- main()diff --git a/products/slack_app/evals/eval_repo_selection_routing.py b/products/slack_app/evals/eval_repo_selection_routing.pynew file mode 100644index 000000000000..f22ffe40a6f9--- /dev/null+++ b/products/slack_app/evals/eval_repo_selection_routing.py@@ -0,0 +1,141 @@+"""End-to-end routing for a Slack `@PostHog` mention: does the cascade, then the needs-repo+classifier, then the discovery agent stop at the right stage with the right outcome?++Each case gets an isolated team with the synthetic `fernhill` catalog and the case's routing+rules, so no case depends on a real team's connected repositories. The classifier runs its+production model through the eval AI gateway. The selector runs on the harness's+`--agent-runtime`/`--agent-model`; compare selector models with `tasks/eval_repo_selection`,+which holds the classifier out of the measurement.++ hogli evals slack_app/eval_repo_selection_routing --agent-model claude-sonnet-5-5+ hogli evals slack_app/eval_repo_selection_routing --eval explicit_mention+"""++from __future__ import annotations++import time+import asyncio+from collections.abc import Awaitable, Callable++from posthog.egress.limiter.policies import Priority++from products.posthog_ai.eval_harness.base import SandboxedEval+from products.posthog_ai.eval_harness.config import SandboxedEvalCase+from products.posthog_ai.eval_harness.harness.context import EvalContext+from products.posthog_ai.eval_harness.harness.providers import SandboxProviderStrategy+from products.posthog_ai.eval_harness.runner import EvalCaseResult+from products.slack_app.backend.services.slack_messages import SlackThreadMessage+from products.slack_app.backend.temporal.activities.classifiers import (+ CLASSIFIER_MODEL,+ classify_task_needs_repo,+ team_routing_rule_lines,+)+from products.slack_app.backend.temporal.activities.repo_selection import _resolve_explicit_repo+from products.slack_app.evals.offline_config import offline_suite+from products.tasks.backend.facade import api as tasks_facade+from products.tasks.backend.facade.agents import CustomPromptSandboxContext, TurnPollTimeout+from products.tasks.evals.repo_selection.fixtures import (+ FIXTURE_VERSION,+ REPOSITORY_NAMES,+ ROUTING_CASES,+ RepoSelectionScenario,+ fixture_fingerprint,+)+from products.tasks.evals.repo_selection.runner import RepoSelectionArtifacts, run_selector+from products.tasks.evals.repo_selection.scorers import ROUTING_SCORERS, routing_expected+from products.tasks.evals.repo_selection.seeders import scenario_seeder++EXPERIMENT_NAME = "slack-repo-selection-routing"++_SCENARIOS: dict[str, RepoSelectionScenario] = {case.scenario.name: case.scenario for case in ROUTING_CASES}+++def _thread(scenario: RepoSelectionScenario) -> list[SlackThreadMessage]:+ return [SlackThreadMessage(user=message.user, text=message.text) for message in scenario.thread]+++def _classify(scenario: RepoSelectionScenario, team_id: int) -> bool:+ candidates = {repo.lower() for repo in REPOSITORY_NAMES}+ return classify_task_needs_repo(+ scenario.mention_text,+ _thread(scenario),+ routing_rules=team_routing_rule_lines(team_id, candidate_repos=candidates),+ team_id=team_id,+ )+++async def run_routing_case(+ case: SandboxedEvalCase,+ context: CustomPromptSandboxContext,+ *,+ provider: SandboxProviderStrategy,+ on_cleanup: Callable[[], Awaitable[None]] | None = None,+ timeout_seconds: float | None = None,+ on_poll_timeout: Callable[[TurnPollTimeout], None] | None = None,+) -> EvalCaseResult:+ scenario = _SCENARIOS[case.name]+ started = time.monotonic()+ artifacts = RepoSelectionArtifacts(exit_code=0, stage="cascade")++ # The production cascade's explicit-mention tiers, with the mention and the thread around it.+ cascade = _resolve_explicit_repo(+ scenario.mention_text, _thread(scenario), [repo.lower() for repo in REPOSITORY_NAMES]+ )+ artifacts.cascade_reason = cascade.reason+ if cascade.mode == "auto":+ artifacts.outcome, artifacts.selected_repository = "auto", cascade.repository+ artifacts.duration_seconds = time.monotonic() - started+ return EvalCaseResult(artifacts=artifacts)++ artifacts.stage = "classifier"+ needs_repo = await asyncio.to_thread(_classify, scenario, context.team_id)+ artifacts.classifier_needs_repo = needs_repo+ if not needs_repo:+ artifacts.outcome = "no_repo"+ artifacts.duration_seconds = time.monotonic() - started+ return EvalCaseResult(artifacts=artifacts)++ artifacts.stage = "selector"+ remaining = None if timeout_seconds is None else timeout_seconds - (time.monotonic() - started)+ return await run_selector(+ scenario,+ context,+ origin_product=tasks_facade.TaskOriginProduct.SLACK,+ # A person waits in the Slack thread for the answer.+ priority=Priority.NORMAL,+ provider=provider,+ on_cleanup=on_cleanup,+ timeout_seconds=remaining,+ artifacts=artifacts,+ )+++async def eval_repo_selection_routing(ctx: EvalContext) -> None:+ await SandboxedEval(+ offline_suite=offline_suite(+ "slack_app/eval_repo_selection_routing::eval_repo_selection_routing",+ metrics=[scorer()._name() for scorer in ROUTING_SCORERS],+ ),+ send_to_posthog=True,+ experiment_name=EXPERIMENT_NAME,+ cases=[+ SandboxedEvalCase(+ name=case.scenario.name,+ prompt=case.scenario.context_block,+ setup=scenario_seeder(case.scenario),+ expected=routing_expected(case.expected),+ metadata={+ "expected_stage": case.expected.stage,+ "classifier_model": CLASSIFIER_MODEL,+ "fixture_version": FIXTURE_VERSION,+ "fixture_fingerprint": fixture_fingerprint(),+ "note": case.note,+ },+ inherit_demo_data=False,+ )+ for case in ROUTING_CASES+ ],+ scorers=[scorer() for scorer in ROUTING_SCORERS],+ case_runner=run_routing_case,+ ctx=ctx,+ )diff --git a/products/slack_app/evals/offline_config.py b/products/slack_app/evals/offline_config.pyindex f8e0c7c0d4c8..29c1c6d289ed 100644--- a/products/slack_app/evals/offline_config.py+++ b/products/slack_app/evals/offline_config.py@@ -38,6 +38,13 @@ _SCORERS: dict[str, dict[str, OfflineScorer]] = {),"project_route_match": OfflineScorer(kind="boolean", version=UUID("01a1557d-37df-756c-a5fb-73edde0b1561")),},+ "repo_selection_routing": {+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1a29b-30cf-7fef-907b-69efc56159bb")),+ "routing_stage_correct": OfflineScorer(kind="boolean", version=UUID("01a1a29b-32a3-7dfc-b5b7-33931ddc73ac")),+ "routing_repository_correct": OfflineScorer(+ kind="boolean", version=UUID("01a1a29b-3447-794a-afcb-83778635c273")+ ),+ },"reply_suggestions": {"exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1557d-49a7-7417-aa9f-4dbe00ec7b47")),"no_unasked_buttons": OfflineScorer(kind="boolean", version=UUID("01a1557d-5b16-755e-ab88-b824e5103a97")),diff --git a/products/tasks/backend/tests/test_repo_selection_eval_fixtures.py b/products/tasks/backend/tests/test_repo_selection_eval_fixtures.pynew file mode 100644index 000000000000..d0aa8a6bca2d--- /dev/null+++ b/products/tasks/backend/tests/test_repo_selection_eval_fixtures.py@@ -0,0 +1,76 @@+import re++import pytest++from products.tasks.evals.repo_selection.fixtures import (+ METRICS_DASHBOARD_RULE,+ REPOSITORIES,+ REPOSITORY_NAMES,+ ROUTING_CASES,+ SELECTOR_CASES,+ SUPPORT_DESK_RULE,+ SelectorExpectation,+)+from products.tasks.evals.repo_selection.scorers import judge_selection++[email protected]("case", SELECTOR_CASES, ids=lambda case: case.scenario.name)+def test_selector_input_does_not_name_the_expected_repository(case):+ visible = case.scenario.context_block.lower()+ for repository in case.expected.accepted:+ assert repository.lower() not in visible+ assert not re.search(rf"\b{re.escape(repository.split('/', 1)[1])}\b", visible)+++ "rule,topic_words",+ [(SUPPORT_DESK_RULE, ("support", "desk")), (METRICS_DASHBOARD_RULE, ("metric", "dashboard"))],+)+def test_routing_rule_is_the_only_link_to_its_repository(rule, topic_words):+ repo = next(repo for repo in REPOSITORIES if repo.full_name == rule.repository)+ cache_text = " ".join((repo.description, repo.readme, *repo.topics, *repo.tree_paths)).lower()+ assert not [word for word in topic_words if word in cache_text]+++def test_cases_cover_the_preserved_scenarios_once():+ routing_names = [case.scenario.name for case in ROUTING_CASES]+ selector_names = [case.scenario.name for case in SELECTOR_CASES]+ assert len(routing_names) == len(set(routing_names)) == 17+ assert len(selector_names) == len(set(selector_names))+ assert all(+ case.expected.selector or case.expected.repository or case.expected.stage == "classifier"+ for case in ROUTING_CASES+ )+ assert {repo for case in SELECTOR_CASES for repo in case.expected.accepted} <= set(REPOSITORY_NAMES)+++def test_expectation_names_a_repository_or_an_abstention():+ with pytest.raises(ValueError):+ SelectorExpectation()+ with pytest.raises(ValueError):+ SelectorExpectation(accepted=("fernhill/storefront",), abstain=True)+ with pytest.raises(ValueError):+ SelectorExpectation(accepted=("fernhill/not-in-catalog",))+++ACCEPTED = {"accepted": ["fernhill/fernhill-ios", "fernhill/fernhill-android"], "abstain": False}+ABSTAIN = {"accepted": [], "abstain": True}+++ "output,spec,expected",+ [+ ({"outcome": "found", "selected_repository": "Fernhill/Fernhill-Android"}, ACCEPTED, 1.0),+ ({"outcome": "found", "selected_repository": "fernhill/storefront"}, ACCEPTED, 0.0),+ ({"outcome": "no_match", "selected_repository": None}, ACCEPTED, 0.0),+ ({"outcome": "no_match", "selected_repository": None}, ABSTAIN, 1.0),+ ({"outcome": "found", "selected_repository": "fernhill/storefront"}, ABSTAIN, 0.0),+ ({"outcome": "rejected", "selected_repository": "fernhill/fernhill-ios-v2"}, ACCEPTED, 0.0),+ ({"outcome": "timeout"}, ABSTAIN, 0.0),+ ({"outcome": "error"}, ACCEPTED, 0.0),+ ({"exit_code": 1, "stderr": "case timeout after 600s"}, ABSTAIN, 0.0),+ (None, ABSTAIN, 0.0),+ ],+)+def test_judge_selection(output, spec, expected):+ assert judge_selection(output, spec)[0] == expecteddiff --git a/products/tasks/backend/tests/test_repo_selection_eval_runner.py b/products/tasks/backend/tests/test_repo_selection_eval_runner.pynew file mode 100644index 000000000000..e9aaee366b5f--- /dev/null+++ b/products/tasks/backend/tests/test_repo_selection_eval_runner.py@@ -0,0 +1,76 @@+import asyncio+from types import SimpleNamespace++import pytest+from unittest.mock import MagicMock, patch++from posthog.egress.limiter.policies import Priority++from products.tasks.backend.facade.agents import CustomPromptSandboxContext+from products.tasks.backend.facade.repo_selection import RepoSelectionUsageLimitError+from products.tasks.evals.repo_selection import runner+from products.tasks.evals.repo_selection.fixtures import SELECTOR_CASES+from products.tasks.evals.repo_selection.runner import RepoSelectionArtifacts, run_selector++_SCENARIO = SELECTOR_CASES[0].scenario+_CONTEXT = CustomPromptSandboxContext(team_id=1, user_id=2, repository="posthog/hedgebox")+_TASK_RUN = SimpleNamespace(id="run-1", task_id="task-1", workflow_id="workflow-1")+++async def _run(provider: MagicMock, *, artifacts: RepoSelectionArtifacts | None = None):+ return await run_selector(+ _SCENARIO,+ _CONTEXT,+ origin_product="signal_report", # type: ignore[arg-type]+ priority=Priority.BATCH,+ provider=provider,+ on_cleanup=None,+ timeout_seconds=600,+ artifacts=artifacts,+ )+++ "error,outcome",+ [+ (RepoSelectionUsageLimitError("API Error: 402"), "usage_limited"),+ (RuntimeError("agent failed during startup"), "error"),+ ],+)+async def test_selector_failure_is_not_scored_as_a_clean_exit(error: Exception, outcome: str) -> None:+ cascade_artifacts = RepoSelectionArtifacts(exit_code=0, stage="selector")+ with (+ patch.object(runner, "select_repository", side_effect=error),+ patch.object(runner, "_distinct_id", return_value=None),+ patch.object(runner, "_latest_task_run", return_value=None),+ ):+ result = await _run(MagicMock(), artifacts=cascade_artifacts)++ assert (result.artifacts.exit_code, result.artifacts.outcome) == (1, outcome)+++async def test_cancelled_selection_still_reclaims_its_sandbox() -> None:+ started = asyncio.Event()++ async def hang(**_kwargs: object) -> None:+ started.set()+ await asyncio.Event().wait()++ provider = MagicMock()+ with (+ patch.object(runner, "select_repository", side_effect=hang),+ patch.object(runner, "_distinct_id", return_value=None),+ patch.object(runner, "_latest_task_run", return_value=_TASK_RUN),+ patch.object(runner.tasks_api, "read_task_run_conversation_logs", return_value=""),+ patch.object(runner, "async_connect", side_effect=RuntimeError("no temporal")),+ ):+ case = asyncio.create_task(_run(provider))+ await started.wait()+ case.cancel()+ with pytest.raises(asyncio.CancelledError):+ await case++ provider.register_task.assert_called_with("task-1")+ provider.cleanup_case.assert_called_once_with("task-1")diff --git a/products/tasks/evals/repo_selection/eval_repo_selection.py b/products/tasks/evals/repo_selection/eval_repo_selection.pynew file mode 100644index 000000000000..0480177e17bd--- /dev/null+++ b/products/tasks/evals/repo_selection/eval_repo_selection.py@@ -0,0 +1,66 @@+"""Selector quality: given one request and the same candidate evidence, does the discovery agent+pick the right repository or abstain?++Every case runs the production `select_repository` against the synthetic catalog in+`fixtures`. The needs-repo classifier and the cascade do not run here, so+classifier behavior cannot hide a difference between selector models; the end-to-end path is+`slack_app/eval_repo_selection_routing`.++Compare models with one harness comparison, for example:++ hogli evals tasks/eval_repo_selection --trials 3 --max-sandboxes 4 \\+ --compare-model claude:claude-haiku-5-5 \\+ --compare-model codex:gpt-6-luna,reasoning_effort=medium \\+ --compare-model codex:gpt-6-luna,reasoning_effort=medium,service_tier=flex+"""++from __future__ import annotations++from posthog.egress.limiter.policies import Priority++from products.posthog_ai.eval_harness.base import SandboxedEval+from products.posthog_ai.eval_harness.config import SandboxedEvalCase+from products.posthog_ai.eval_harness.harness.context import EvalContext+from products.tasks.backend.facade import api as tasks_facade++from .fixtures import FIXTURE_VERSION, SELECTOR_CASES, fixture_fingerprint+from .offline_config import REPO_SELECTION_SUITE+from .runner import selector_case_runner+from .scorers import SELECTOR_SCORERS, selector_expected+from .seeders import scenario_seeder++EXPERIMENT_NAME = "repo-selection-selector"+++async def eval_repo_selection(ctx: EvalContext) -> None:+ await SandboxedEval(+ offline_suite=REPO_SELECTION_SUITE,+ send_to_posthog=True,+ experiment_name=EXPERIMENT_NAME,+ cases=[+ SandboxedEvalCase(+ name=case.scenario.name,+ # Shown in local logs only; the selector receives the rendered thread from the runner.+ prompt=case.scenario.context_block,+ setup=scenario_seeder(case.scenario),+ expected=selector_expected(case.expected),+ metadata={+ "category": case.category,+ "fixture_version": FIXTURE_VERSION,+ "fixture_fingerprint": fixture_fingerprint(),+ },+ # The selector reads only the seeded repository cache, never project data.+ inherit_demo_data=False,+ )+ for case in SELECTOR_CASES+ ],+ scorers=[scorer() for scorer in SELECTOR_SCORERS],+ # Self-Driving's selection path: the Signals origin attributes the run to+ # `signals_repo_selection`, and report generation runs at batch priority.+ case_runner=selector_case_runner(+ {case.scenario.name: case.scenario for case in SELECTOR_CASES},+ origin_product=tasks_facade.TaskOriginProduct.SIGNAL_REPORT,+ priority=Priority.BATCH,+ ),+ ctx=ctx,+ )diff --git a/products/tasks/evals/repo_selection/fixtures.py b/products/tasks/evals/repo_selection/fixtures.pynew file mode 100644index 000000000000..5938400e2fd0--- /dev/null+++ b/products/tasks/evals/repo_selection/fixtures.py@@ -0,0 +1,652 @@+"""Synthetic repository catalog and cases for the repository-selection evals.++Pure and Django-free, so the catalog is unit-testable and byte-for-byte reproducible.+The seeder installs it into each case's isolated team; prompts, scorers, and seeders all+import the names below so they cannot drift apart.++Everything here is invented. The `fernhill` organization, its repositories, and every path+and README line exist only in this file.++Model-visible input is a case's Slack thread and the seeded catalog. Expected answers live+in `SelectorExpectation` and `RoutingExpectation`, which only scorers read.+"""++from __future__ import annotations++import hashlib+from collections.abc import Sequence+from typing import Literal++from posthog.dataclasses import frozen++FIXTURE_VERSION = "2026-11-03.1"++ORG = "fernhill"+BOT_MENTION = "@PostHog"++STOREFRONT = f"{ORG}/storefront"+CORE_API = f"{ORG}/core-api"+DOCS_SITE = f"{ORG}/docs-site"+MARKETING_SITE = f"{ORG}/marketing-site"+IOS_SDK = f"{ORG}/fernhill-ios"+ANDROID_SDK = f"{ORG}/fernhill-android"+PAYMENTS_UI = f"{ORG}/payments-ui"+HERMES = f"{ORG}/hermes"+ATLAS = f"{ORG}/atlas"+++@frozen+class FixtureRepository:+ full_name: str+ description: str+ primary_language: str+ topics: tuple[str, ...]+ private: bool+ readme: str+ tree_paths: tuple[str, ...]+ archived: bool = False++ @property+ def name(self) -> str:+ return self.full_name.split("/", 1)[1]++ @property+ def default_branch_sha(self) -> str:+ # Stable per content, so a reseeded catalog reads as the same commit.+ digest = hashlib.sha256(f"{self.full_name}\n{self.readme}\n{chr(10).join(self.tree_paths)}".encode())+ # A git commit SHA is 40 hex characters.+ return digest.hexdigest()[:40]+++REPOSITORIES: tuple[FixtureRepository, ...] = (+ FixtureRepository(+ full_name=STOREFRONT,+ description="Customer-facing web shop: catalogue, cart, checkout, and account signup.",+ primary_language="TypeScript",+ topics=("nextjs", "react", "ecommerce"),+ private=True,+ readme=(+ "# Storefront\n\n"+ "The Next.js web shop customers use at shop.fernhill.example. It renders the product catalogue, "+ "the cart, the checkout flow, and account signup.\n\n"+ "## Analytics\n\n"+ "Product analytics is initialised in `src/analytics/posthog-init.ts` with autocapture enabled. "+ "Custom events, including `signup_completed` and `checkout_started`, are captured from "+ "`src/analytics/events.ts`.\n\n"+ "## Development\n\n`pnpm install && pnpm dev`\n"+ ),+ tree_paths=(+ "package.json",+ "next.config.mjs",+ "src/analytics/posthog-init.ts",+ "src/analytics/events.ts",+ "src/analytics/track-signup-completed.ts",+ "src/pages/index.tsx",+ "src/pages/cart.tsx",+ "src/pages/checkout/index.tsx",+ "src/pages/checkout/confirmation.tsx",+ "src/pages/signup/index.tsx",+ "src/pages/signup/SignupForm.tsx",+ "src/pages/signup/SignupForm.module.css",+ "src/components/checkout/PayButton.tsx",+ "src/components/checkout/AddressFields.tsx",+ "src/components/checkout/OrderSummary.tsx",+ "src/components/catalogue/ProductCard.tsx",+ "src/components/layout/Header.tsx",+ "src/lib/api-client.ts",+ "src/lib/money.ts",+ "tests/checkout/pay-button.spec.ts",+ "tests/signup/signup-form.spec.ts",+ ),+ ),+ FixtureRepository(+ full_name=CORE_API,+ description="Django REST backend for accounts, projects, orders, and billing.",+ primary_language="Python",+ topics=("django", "rest-api", "backend"),+ private=True,+ readme=(+ "# Core API\n\n"+ "The Django REST Framework service behind the storefront and the mobile apps. It owns accounts, "+ "projects, orders, and billing webhooks.\n\n"+ "Access checks live in `accounts/permissions.py`; each viewset declares its own "+ "`permission_classes`.\n\n"+ "## Running\n\n`uv sync && ./manage.py runserver`\n"+ ),+ tree_paths=(+ "pyproject.toml",+ "manage.py",+ "core_api/settings.py",+ "core_api/urls.py",+ "accounts/models.py",+ "accounts/permissions.py",+ "accounts/views/signup.py",+ "accounts/views/session.py",+ "accounts/tests/test_permissions.py",+ "api/projects/viewsets.py",+ "api/projects/serializers.py",+ "api/projects/pagination.py",+ "api/projects/tests/test_viewsets.py",+ "api/orders/viewsets.py",+ "api/orders/serializers.py",+ "billing/stripe_webhooks.py",+ "billing/invoices.py",+ "billing/tests/test_webhooks.py",+ ),+ ),+ FixtureRepository(+ full_name=DOCS_SITE,+ description="Source of docs.fernhill.example, the product documentation website.",+ primary_language="TypeScript",+ topics=("astro", "documentation", "website"),+ private=False,+ readme=(+ "# Docs site\n\n"+ "The Astro site served at docs.fernhill.example. Pages live in `src/content/docs/`; the "+ "getting-started guide is `src/content/docs/getting-started.mdx`.\n\n"+ "Large screenshots go through `src/components/ResponsiveImage.astro`.\n"+ ),+ tree_paths=(+ "package.json",+ "astro.config.mjs",+ "src/content/docs/getting-started.mdx",+ "src/content/docs/sdks/ios.mdx",+ "src/content/docs/sdks/android.mdx",+ "src/content/docs/api/authentication.mdx",+ "src/components/ResponsiveImage.astro",+ "src/components/Search.astro",+ "src/components/Sidebar.astro",+ "src/layouts/DocsLayout.astro",+ "public/images/getting-started/dashboard-overview.png",+ "public/images/getting-started/install-snippet.png",+ ),+ ),+ FixtureRepository(+ full_name=MARKETING_SITE,+ description="The marketing homepage, pricing page, and blog at fernhill.example.",+ primary_language="TypeScript",+ topics=("website", "marketing", "blog"),+ private=False,+ readme=(+ "# Marketing site\n\n"+ "Homepage, pricing, customer stories, and the blog at fernhill.example. Product documentation is "+ "not here.\n"+ ),+ tree_paths=(+ "package.json",+ "src/pages/index.tsx",+ "src/pages/pricing.tsx",+ "src/pages/customers/[slug].tsx",+ "src/pages/blog/[slug].tsx",+ "src/components/Hero.tsx",+ "src/components/PricingTable.tsx",+ "content/blog/2026-launch-week.md",+ ),+ ),+ FixtureRepository(+ full_name=IOS_SDK,+ description="Fernhill iOS SDK: event capture and session replay for Swift apps.",+ primary_language="Swift",+ topics=("ios", "sdk", "swift", "session-replay"),+ private=False,+ readme=(+ "# Fernhill iOS SDK\n\n"+ "Swift package for capturing events and recording sessions in iOS apps. Session replay starts "+ "with `FernhillReplay.start()`.\n\n"+ "## Changelog\n\nSee `CHANGELOG.md`. Version 3.19.0 moved replay start into app launch.\n"+ ),+ tree_paths=(+ "Package.swift",+ "CHANGELOG.md",+ "Sources/Fernhill/Fernhill.swift",+ "Sources/Fernhill/Capture/EventQueue.swift",+ "Sources/Fernhill/Replay/FernhillReplay.swift",+ "Sources/Fernhill/Replay/ReplayRecorder.swift",+ "Sources/Fernhill/Replay/ReplayConfig.swift",+ "Sources/Fernhill/Lifecycle/AppLaunchObserver.swift",+ "Tests/FernhillTests/ReplayTests.swift",+ "Tests/FernhillTests/AppLaunchTests.swift",+ ),+ ),+ FixtureRepository(+ full_name=ANDROID_SDK,+ description="Fernhill Android SDK: event capture and session replay for Kotlin apps.",+ primary_language="Kotlin",+ topics=("android", "sdk", "kotlin", "session-replay"),+ private=False,+ readme=(+ "# Fernhill Android SDK\n\n"+ "Kotlin library for capturing events and recording sessions in Android apps. Version 3.19.0 "+ "added frame batching to session replay.\n"+ ),+ tree_paths=(+ "build.gradle.kts",+ "CHANGELOG.md",+ "fernhill/src/main/java/com/fernhill/Fernhill.kt",+ "fernhill/src/main/java/com/fernhill/capture/EventQueue.kt",+ "fernhill/src/main/java/com/fernhill/replay/ReplayRecorder.kt",+ "fernhill/src/main/java/com/fernhill/replay/FrameBatcher.kt",+ "fernhill/src/main/java/com/fernhill/replay/ReplayConfig.kt",+ "fernhill/src/test/java/com/fernhill/replay/FrameBatcherTest.kt",+ ),+ ),+ # Misleading name: the checkout code lives in the storefront. This repo holds old prototypes.+ FixtureRepository(+ full_name=PAYMENTS_UI,+ description="Design prototypes for an early payments page. Not deployed.",+ primary_language="JavaScript",+ topics=("prototype", "design"),+ private=True,+ readme=(+ "# payments-ui\n\n"+ "Static design prototypes from the 2024 payments redesign. Nothing here is built or deployed; "+ "the production checkout was rebuilt elsewhere.\n"+ ),+ tree_paths=(+ "README.md",+ "prototypes/pay-page-v1.html",+ "prototypes/pay-page-v2.html",+ "prototypes/assets/pay-page.css",+ ),+ ),+ # Routing-rule targets. Nothing in their caches ties them to the rules' topics, so only the+ # rule can steer a request here.+ FixtureRepository(+ full_name=HERMES,+ description="Internal service.",+ primary_language="Python",+ topics=("internal",),+ private=True,+ readme="# Hermes\n\nInternal service. Deploy with `make deploy`.\n",+ tree_paths=(+ "pyproject.toml",+ "Makefile",+ "hermes/app.py",+ "hermes/handlers/query.py",+ "hermes/handlers/index.py",+ "hermes/store.py",+ "tests/test_query.py",+ ),+ ),+ FixtureRepository(+ full_name=ATLAS,+ description="Internal service.",+ primary_language="TypeScript",+ topics=("internal",),+ private=True,+ readme="# Atlas\n\nInternal service. `pnpm start` runs it locally.\n",+ tree_paths=(+ "package.json",+ "src/server.ts",+ "src/routes/home.tsx",+ "src/routes/tiles.tsx",+ "src/lib/fetcher.ts",+ "tests/home.spec.ts",+ ),+ ),+)++REPOSITORY_NAMES: tuple[str, ...] = tuple(repo.full_name for repo in REPOSITORIES)+++@frozen+class RoutingRuleFixture:+ rule_text: str+ repository: str+++@frozen+class ThreadMessage:+ user: str+ text: str+++Stage = Literal["cascade", "classifier", "selector"]+Outcome = Literal["auto", "no_repo", "found", "no_match"]+++@frozen+class SelectorExpectation:+ """What the discovery agent should return. Exactly one of `accepted` or `abstain` is set."""++ accepted: tuple[str, ...] = ()+ abstain: bool = False++ def __post_init__(self) -> None:+ if bool(self.accepted) == self.abstain:+ raise ValueError("A selector expectation names accepted repositories or an abstention, not both")+ unknown = set(self.accepted) - set(REPOSITORY_NAMES)+ if unknown:+ raise ValueError(f"Expected repositories are not in the catalog: {sorted(unknown)}")+++@frozen+class RoutingExpectation:+ stage: Stage+ outcome: Outcome+ # Set when the terminal stage selects a repository; scored against the selected repo.+ selector: SelectorExpectation | None = None+ # Set for cascade selections, which are deterministic.+ repository: str | None = None+++@frozen+class RepoSelectionScenario:+ name: str+ description: str+ thread: tuple[ThreadMessage, ...]+ routing_rules: tuple[RoutingRuleFixture, ...] = ()++ @property+ def mention_text(self) -> str:+ # The message that addresses the bot. Later replies are thread context, as in Slack.+ return next(message.text for message in reversed(self.thread) if BOT_MENTION in message.text)++ @property+ def context_block(self) -> str:+ # The same rendering the Slack discovery activity hands the selector.+ return "\n".join(f"{message.user}: {message.text}" for message in self.thread)+++@frozen+class RoutingCase:+ scenario: RepoSelectionScenario+ expected: RoutingExpectation+ note: str = ""+++@frozen+class SelectorCase:+ scenario: RepoSelectionScenario+ expected: SelectorExpectation+ # Names the property this case isolates, for per-category reporting.+ category: str+++def _thread(*messages: tuple[str, str]) -> tuple[ThreadMessage, ...]:+ return tuple(ThreadMessage(user=user, text=text) for user, text in messages)+++def _ask(text: str, *earlier: tuple[str, str]) -> tuple[ThreadMessage, ...]:+ return _thread(*earlier, ("tester", text))+++SUPPORT_DESK_RULE = RoutingRuleFixture(rule_text="Anything about the internal support desk", repository=HERMES)+METRICS_DASHBOARD_RULE = RoutingRuleFixture(rule_text="The internal metrics dashboard", repository=ATLAS)+CATCH_ALL_RULE = RoutingRuleFixture(rule_text="Route every request to the internal service", repository=HERMES)+++def _scenario(+ name: str,+ description: str,+ thread: tuple[ThreadMessage, ...],+ routing_rules: Sequence[RoutingRuleFixture] = (),+) -> RepoSelectionScenario:+ return RepoSelectionScenario(name=name, description=description, thread=thread, routing_rules=tuple(routing_rules))+++# Scenarios that reach the discovery agent, shared by both suites so a selector change and an+# end-to-end change are measured on the same request text.+DOCS_SITE_SLOW = _scenario(+ "marketing_site",+ "Performance complaint about the docs site; the marketing site is a same-domain distractor.",+ _thread(+ ("tester", "@PostHog the docs site loads really slowly on mobile"),+ ("other", "yeah I noticed the same on /docs/getting-started"),+ ),+)+IOS_CRASH = _scenario(+ "sdk_specific_trace_trip",+ "iOS SDK crash with a stack frame; the Android SDK also has replay code.",+ _thread(+ ("tester", "@PostHog the iOS SDK is crashing on app launch after upgrade to 3.19"),+ ("other", "stack trace shows FernhillReplay.start() failing"),+ ),+)+AUTOCAPTURE_CHECKOUT = _scenario(+ "sdk_instrumentation_on_own_site",+ "Analytics SDK misbehaving on the team's own site; the fix is in the site's code.",+ _thread(+ ("tester", "@PostHog autocapture isn't picking up clicks on our checkout button, other buttons work"),+ ("other", "we just shipped a redesign yesterday"),+ ),+)+MISSING_SIGNUP_EVENTS = _scenario(+ "wrong_data_tracking_bug",+ "Missing analytics data is almost always a tracking bug in the team's code.",+ _thread(+ ("tester", "@PostHog we're not seeing any signup_completed events even though we tested signups today"),+ ("other", "the funnel shows 0 conversions but our team manually completed 5"),+ ),+)+VAGUE_SIGNUP_RENDER = _scenario(+ "vague_code_bug",+ "Vague but code-flavored; the cache disambiguates the signup UI.",+ _ask("@PostHog there's a bug in how we render user signup, can you fix it"),+)+PROJECTS_VIEWSET = _scenario(+ "api_viewset",+ "An explicit code pattern (viewset) names backend code.",+ _ask("@PostHog the /api/projects/ viewset crashes on large payloads, can you fix it"),+)+SIGNUP_TSX = _scenario(+ "explicit_code_file",+ "An explicit .tsx file extension names frontend code.",+ _ask("@PostHog add a Cancel button to the signup form in the .tsx component"),+)+PERMISSION_REFACTOR = _scenario(+ "refactor_request",+ "A code-flavored refactor with no debug terms.",+ _ask("@PostHog please refactor the user permission check into a single helper"),+)+SUPPORT_DESK = _scenario(+ "routing_rule_steers_vague_ask",+ "A configured routing rule steers an ask no cache content links to a repository.",+ _ask("@PostHog the internal support desk search endpoint is throwing 500s, can you fix it"),+ routing_rules=(SUPPORT_DESK_RULE,),+)+METRICS_DASHBOARD = _scenario(+ "routing_rule_overrides_no_repo_gate",+ "A routing rule keeps a product-term ask (dashboard) from stopping at the no-repo gate.",+ _ask("@PostHog the internal metrics dashboard shows a blank page, can you fix it"),+ routing_rules=(METRICS_DASHBOARD_RULE,),+)++ROUTING_CASES: tuple[RoutingCase, ...] = (+ RoutingCase(+ scenario=_scenario(+ "explicit_mention",+ "The cascade picks the repo directly when the text names a connected org/repo.",+ _ask(f"@PostHog can you look at {CORE_API} and fix the readme typo"),+ ),+ expected=RoutingExpectation(stage="cascade", outcome="auto", repository=CORE_API),+ ),+ RoutingCase(+ scenario=_scenario(+ "ci_run_link",+ "The cascade reads the repo out of a workflow-run link.",+ _ask(f"@PostHog is this flaky? https://github.com/{STOREFRONT}/actions/runs/30560492835"),+ ),+ expected=RoutingExpectation(stage="cascade", outcome="auto", repository=STOREFRONT),+ ),+ RoutingCase(+ scenario=_scenario(+ "ci_run_link_earlier_in_the_thread",+ "The cascade reads the repo from a link posted before the mention.",+ _ask(+ "@PostHog is this one flaky?",+ ("tester", f"https://github.com/{STOREFRONT}/actions/runs/30560492835 went red again"),+ ),+ ),+ expected=RoutingExpectation(stage="cascade", outcome="auto", repository=STOREFRONT),+ ),+ RoutingCase(+ scenario=_scenario(+ "billing_question",+ "A billing question needs no repository.",+ _ask("@PostHog how do I update the credit card on our subscription?"),+ ),+ expected=RoutingExpectation(stage="classifier", outcome="no_repo"),+ ),+ RoutingCase(+ scenario=_scenario(+ "dashboard_config",+ "The product-term heuristic stops a dashboard question with no code pattern.",+ _ask("@PostHog the dashboard tile filters are not persisting across refreshes"),+ ),+ expected=RoutingExpectation(stage="classifier", outcome="no_repo"),+ ),+ RoutingCase(+ scenario=DOCS_SITE_SLOW,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(DOCS_SITE,))+ ),+ ),+ RoutingCase(+ scenario=IOS_CRASH,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(IOS_SDK,))+ ),+ ),+ RoutingCase(+ scenario=_scenario(+ "posthog_product_hang",+ "A complaint about an analytics product page is a product question, not the team's code.",+ _thread(+ ("tester", "@PostHog the trends page hangs forever when I add 10+ series"),+ ("other", "tried Firefox and Chrome, just a spinner"),+ ),+ ),+ expected=RoutingExpectation(stage="classifier", outcome="no_repo"),+ ),+ RoutingCase(+ scenario=AUTOCAPTURE_CHECKOUT,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(STOREFRONT,))+ ),+ ),+ RoutingCase(+ scenario=MISSING_SIGNUP_EVENTS,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(STOREFRONT,))+ ),+ note="The thread says 'events' and 'funnel', which the product-term heuristic treats as an analytics ask.",+ ),+ RoutingCase(+ scenario=VAGUE_SIGNUP_RENDER,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(STOREFRONT,))+ ),+ ),+ RoutingCase(+ scenario=PROJECTS_VIEWSET,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(CORE_API,))+ ),+ ),+ RoutingCase(+ scenario=SIGNUP_TSX,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(STOREFRONT,))+ ),+ ),+ RoutingCase(+ scenario=PERMISSION_REFACTOR,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(CORE_API,))+ ),+ ),+ RoutingCase(+ scenario=SUPPORT_DESK,+ expected=RoutingExpectation(+ stage="selector", outcome="found", selector=SelectorExpectation(accepted=(HERMES,))+ ),+ note="The rule is the only signal linking 'support desk' to hermes, so a pass means the rule reached the agent.",+ ),+ RoutingCase(+ scenario=METRICS_DASHBOARD,+ expected=RoutingExpectation(stage="selector", outcome="found", selector=SelectorExpectation(accepted=(ATLAS,))),+ note="Without a rule, 'dashboard' stops at the heuristic; a pass means rules reached the needs-repo gate.",+ ),+ RoutingCase(+ scenario=_scenario(+ "routing_rule_loses_to_explicit_mention",+ "An explicit org/repo wins in the cascade; the rule never reaches the agent.",+ _ask(f"@PostHog can you look at {CORE_API} and fix the readme typo"),+ routing_rules=(CATCH_ALL_RULE,),+ ),+ expected=RoutingExpectation(stage="cascade", outcome="auto", repository=CORE_API),+ ),+)++SELECTOR_CASES: tuple[SelectorCase, ...] = (+ SelectorCase(scenario=DOCS_SITE_SLOW, expected=SelectorExpectation(accepted=(DOCS_SITE,)), category="domain"),+ SelectorCase(scenario=IOS_CRASH, expected=SelectorExpectation(accepted=(IOS_SDK,)), category="identifier"),+ SelectorCase(+ scenario=AUTOCAPTURE_CHECKOUT, expected=SelectorExpectation(accepted=(STOREFRONT,)), category="domain"+ ),+ SelectorCase(+ scenario=MISSING_SIGNUP_EVENTS, expected=SelectorExpectation(accepted=(STOREFRONT,)), category="identifier"+ ),+ SelectorCase(scenario=VAGUE_SIGNUP_RENDER, expected=SelectorExpectation(accepted=(STOREFRONT,)), category="vague"),+ SelectorCase(scenario=PROJECTS_VIEWSET, expected=SelectorExpectation(accepted=(CORE_API,)), category="identifier"),+ SelectorCase(scenario=SIGNUP_TSX, expected=SelectorExpectation(accepted=(STOREFRONT,)), category="identifier"),+ SelectorCase(scenario=PERMISSION_REFACTOR, expected=SelectorExpectation(accepted=(CORE_API,)), category="vague"),+ SelectorCase(scenario=SUPPORT_DESK, expected=SelectorExpectation(accepted=(HERMES,)), category="routing_rule"),+ SelectorCase(scenario=METRICS_DASHBOARD, expected=SelectorExpectation(accepted=(ATLAS,)), category="routing_rule"),+ SelectorCase(+ scenario=_scenario(+ "misleading_repo_name",+ "The repo named for payments holds undeployed prototypes; the checkout code is in the storefront.",+ _ask("@PostHog the pay button on the checkout page does nothing on Safari, can you fix it"),+ ),+ expected=SelectorExpectation(accepted=(STOREFRONT,)),+ category="misleading_name",+ ),+ SelectorCase(+ scenario=_scenario(+ "replay_regression_both_sdks",+ "Both mobile SDKs changed replay in 3.19, so either is a defensible subject.",+ _ask("@PostHog session replay drops frames on mobile after the 3.19 upgrade"),+ ),+ expected=SelectorExpectation(accepted=(IOS_SDK, ANDROID_SDK)),+ category="ambiguous",+ ),+ SelectorCase(+ scenario=_scenario(+ "no_plausible_match_desktop_installer",+ "A code request about a desktop installer that no candidate contains.",+ _ask("@PostHog the Windows desktop installer (.msi) fails with error 1603, can you fix the build script"),+ ),+ expected=SelectorExpectation(abstain=True),+ category="no_match",+ ),+ SelectorCase(+ scenario=_scenario(+ "no_plausible_match_billing",+ "A billing question that reaches the selector directly; no candidate owns it.",+ _ask("@PostHog how do I update the credit card on our subscription?"),+ ),+ expected=SelectorExpectation(abstain=True),+ category="no_match",+ ),+ SelectorCase(+ scenario=_scenario(+ "no_plausible_match_hr_tooling",+ "An internal operations request with a misleadingly technical word.",+ _ask("@PostHog please update the API key for our payroll provider in the HR portal"),+ ),+ expected=SelectorExpectation(abstain=True),+ category="no_match",+ ),+)+++def fixture_fingerprint() -> str:+ """Hash of everything a model can see, so a result is tied to the exact catalog it ran on."""+ digest = hashlib.sha256(FIXTURE_VERSION.encode())+ for repo in REPOSITORIES:+ digest.update(repr(repo).encode())+ for case in (*ROUTING_CASES, *SELECTOR_CASES):+ digest.update(repr(case).encode())+ return digest.hexdigest()[:16]diff --git a/products/tasks/evals/repo_selection/offline_config.py b/products/tasks/evals/repo_selection/offline_config.pynew file mode 100644index 000000000000..83f9b296f4c1--- /dev/null+++ b/products/tasks/evals/repo_selection/offline_config.py@@ -0,0 +1,15 @@+from uuid import UUID++from products.posthog_ai.eval_harness.offline_results import OfflineEvalSuite, OfflineScorer++REPO_SELECTION_SUITE = OfflineEvalSuite.from_scorers(+ key="tasks/eval_repo_selection::eval_repo_selection",+ project_id=2,+ tags=("team:self-driving", "component:tasks/repo-selection"),+ scorers={+ "exit_code_zero": OfflineScorer(kind="boolean", version=UUID("01a1a29b-2872-7fa1-9657-362160748643")),+ "repo_selection_correct": OfflineScorer(kind="boolean", version=UUID("01a1a29b-2b7c-780a-844c-0d6554836396")),+ "correct_abstention": OfflineScorer(kind="boolean", version=UUID("01a1a29b-2d22-7be2-969a-cd8d145c6287")),+ "selector_completed": OfflineScorer(kind="boolean", version=UUID("01a1a29b-2ee1-788f-ae8b-0081c4a997b0")),+ },+)diff --git a/products/tasks/evals/repo_selection/report.py b/products/tasks/evals/repo_selection/report.pynew file mode 100644index 000000000000..2182406c3578--- /dev/null+++ b/products/tasks/evals/repo_selection/report.py@@ -0,0 +1,333 @@+"""Turn a repository-selection model comparison into a quality, latency, and cost table.++ python -m products.tasks.evals.repo_selection.report <comparison-dir> [--json out.json]++Reads each arm's run report, per-trial artifacts, and the eval gateway's captured+`$ai_generation` events. Every planned trial stays in the denominators: a trial with an+error, timeout, or no result counts as incorrect. Gateway cost is what the gateway charged;+the list estimate reprices the same tokens from provider list rates, and halves a call only+when the gateway reports that Flex served it.+"""++from __future__ import annotations++import sys+import json+import argparse+import statistics+from collections import Counter, defaultdict+from pathlib import Path+from typing import Any++from products.posthog_ai.eval_harness.log_sink import _case_log_slug++from .fixtures import FIXTURE_VERSION, SELECTOR_CASES, fixture_fingerprint+from .scorers import judge_selection, selector_expected++EXPERIMENT = "repo-selection-selector"+REPO_ROOT = Path(__file__).resolve().parents[4]+LLM_COSTS = REPO_ROOT / "nodejs/src/ingestion/pipelines/ai/costs/providers/llm-costs.json"++# The provider row of `llm-costs.json` each arm's gateway route bills at.+LIST_PRICE_ROWS: dict[str, tuple[str, str]] = {+ "claude-haiku-5-5": ("anthropic/claude-haiku-5.5", "anthropic"),+ "claude-haiku-4-5": ("anthropic/claude-haiku-4.5", "anthropic"),+ "gpt-6-luna": ("openai/gpt-6-luna", "openai"),+ "gpt-5.6-luna": ("openai/gpt-5.6-luna", "openai"),+ "gpt-6.1-terra": ("openai/gpt-6.1-terra", "openai"),+ # No list row names the fast build; the gateway prices it at Baseten's rate (see served.go).+ "deepseek-ai/deepseek-v4.1-flash-1017-fast": ("", "baseten"),+}+BASETEN_DEEPSEEK_V4_1_FAST = {"prompt_token": 0.60e-6, "completion_token": 2.40e-6, "cache_read_token": 0.014e-6}+FLEX_DISCOUNT = 0.5+++def _list_prices(model: str) -> dict[str, float] | None:+ row_name, provider = LIST_PRICE_ROWS.get(model, ("", ""))+ if model == "deepseek-ai/deepseek-v4.1-flash-1017-fast":+ return BASETEN_DEEPSEEK_V4_1_FAST+ if not row_name or not LLM_COSTS.exists():+ return None+ rows = json.loads(LLM_COSTS.read_text())+ for row in rows if isinstance(rows, list) else []:+ if row.get("model") == row_name:+ return row["cost"].get(provider) or row["cost"].get("default")+ return None+++def _number(value: Any) -> float:+ try:+ return float(value or 0)+ except (TypeError, ValueError):+ return 0.0+++def _list_cost(properties: dict[str, Any], prices: dict[str, float] | None) -> float | None:+ if prices is None:+ return None+ # The gateway reports disjoint buckets for every provider: uncached input, cache reads,+ # cache writes, and output (which includes reasoning tokens).+ cache_read = _number(properties.get("$ai_cache_read_input_tokens"))+ cache_write = _number(properties.get("$ai_cache_creation_input_tokens"))+ input_tokens = _number(properties.get("$ai_input_tokens"))+ cost = (+ input_tokens * prices.get("prompt_token", 0)+ + _number(properties.get("$ai_output_tokens")) * prices.get("completion_token", 0)+ + cache_read * prices.get("cache_read_token", prices.get("prompt_token", 0))+ + cache_write * prices.get("cache_write_token", prices.get("prompt_token", 0))+ )+ return cost * FLEX_DISCOUNT if properties.get("$ai_service_tier") == "flex" else cost+++def _generations_by_run(path: str | None) -> dict[str, list[dict[str, Any]]]:+ grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)+ if not path or not Path(path).exists():+ return grouped+ for line in Path(path).read_text().splitlines():+ if line.strip():+ record = json.loads(line)+ grouped[str(record["properties"].get("task_run_id"))].append(record["properties"])+ return grouped+++def _percentile(values: list[float], fraction: float) -> float | None:+ if not values:+ return None+ ordered = sorted(values)+ return ordered[min(len(ordered) - 1, round(fraction * (len(ordered) - 1)))]+++def summarize_arm(run: dict[str, Any]) -> dict[str, Any]:+ report = run.get("report") or {}+ label = ", ".join(+ part+ for part in (+ f"{run['runtime']}/{run['model']}",+ f"tier={run['service_tier']}" if run.get("service_tier") else "",+ f"effort={run['reasoning_effort']}" if run.get("reasoning_effort") else "",+ )+ if part+ )+ experiment = (report.get("experiments") or {}).get(EXPERIMENT)+ summary: dict[str, Any] = {+ "arm": label,+ "run": {k: run.get(k) for k in ("runtime", "model", "service_tier", "reasoning_effort", "exit_code", "issue")},+ }+ if not experiment:+ return summary | {"status": "not run" if run.get("exit_code") is None else "no results"}++ log_dir = Path(experiment["log_dir"])+ case_results = log_dir / "case_results.jsonl"+ if not case_results.exists():+ raise SystemExit(f"{case_results} is missing. Rerun the comparison with EXPORT_EVAL_RESULTS=1.")+ rows = [json.loads(line) for line in case_results.read_text().splitlines() if line.strip()]+ generations = _generations_by_run(report.get("ai_gateway_generations_path"))+ prices = _list_prices(run["model"])+ expected = {case.scenario.name: (case, selector_expected(case.expected)) for case in SELECTOR_CASES}++ outcomes: Counter[str] = Counter()+ tiers: Counter[str] = Counter()+ trials: list[dict[str, Any]] = []+ for row in rows:+ case, spec = expected[row["case_name"]]+ artifacts_path = log_dir / f"{_case_log_slug(row['case_name'], row.get('trial_id'))}.artifacts.json"+ artifacts = json.loads(artifacts_path.read_text()) if artifacts_path.exists() else None+ if row.get("error"):+ outcome = "infra_error"+ correct = 0.0+ else:+ correct, _ = judge_selection(artifacts, spec["repo_selection_correct"])+ outcome = (artifacts or {}).get("outcome") or "timeout"+ outcomes[outcome] += 1+ calls = generations.get(str((artifacts or {}).get("task_run_id")), [])+ for call in calls:+ requested = call.get("$ai_service_tier_requested") or "none"+ served = call.get("$ai_service_tier") or "unreported"+ fallback = "+fallback" if call.get("$ai_service_tier_fallback") else ""+ tiers[f"{requested}->{served}{fallback}"] += 1+ list_costs = [_list_cost(call, prices) for call in calls]+ trials.append(+ {+ "case": row["case_name"],+ "category": case.category,+ "abstain_expected": case.expected.abstain,+ "trial_id": row.get("trial_id"),+ "outcome": outcome,+ "selected": (artifacts or {}).get("selected_repository"),+ "correct": correct,+ "latency_seconds": (artifacts or {}).get("selector_seconds"),+ "calls": len(calls),+ "model_seconds": sum(_number(c.get("$ai_latency")) for c in calls),+ "input_tokens": sum(_number(c.get("$ai_input_tokens")) for c in calls),+ "output_tokens": sum(_number(c.get("$ai_output_tokens")) for c in calls),+ "cache_read_tokens": sum(_number(c.get("$ai_cache_read_input_tokens")) for c in calls),+ "cache_write_tokens": sum(_number(c.get("$ai_cache_creation_input_tokens")) for c in calls),+ "reasoning_tokens": sum(_number(c.get("$ai_reasoning_tokens")) for c in calls),+ "gateway_cost_usd": sum(_number(c.get("$ai_total_cost_usd")) for c in calls),+ "list_cost_usd": None if any(cost is None for cost in list_costs) else sum(list_costs),+ "models_served": sorted({str(c.get("$ai_model")) for c in calls}),+ "providers": sorted({str(c.get("$ai_provider")) for c in calls}),+ "efforts": sorted({str(c.get("$ai_effort")) for c in calls}),+ "products": sorted({str(c.get("$ai_product")) for c in calls}),+ "failed_calls": sum(1 for c in calls if c.get("$ai_gateway_outcome") not in (None, "ok")),+ }+ )++ planned = experiment.get("planned_trials") or len(trials)+ match_trials = [t for t in trials if not t["abstain_expected"]]+ no_match_trials = [t for t in trials if t["abstain_expected"]]+ selected_cases = experiment.get("cases") or {}+ trials_per_case = report.get("trials", 1)+ planned_match = sum(+ count * trials_per_case for name, count in selected_cases.items() if not expected[name][0].expected.abstain+ )+ planned_no_match = sum(+ count * trials_per_case for name, count in selected_cases.items() if expected[name][0].expected.abstain+ )+ correct = sum(t["correct"] for t in trials)+ latencies = [+ t["latency_seconds"]+ for t in trials+ if t["latency_seconds"] is not None and t["outcome"] in {"found", "no_match", "rejected"}+ ]+ model_times = [+ t["model_seconds"] for t in trials if t["calls"] and t["outcome"] in {"found", "no_match", "rejected"}+ ]+ gateway_cost = sum(t["gateway_cost_usd"] for t in trials)+ list_costs = [t["list_cost_usd"] for t in trials]+ return summary | {+ "status": "ok",+ "planned_trials": planned,+ "recorded_trials": len(trials),+ "missing_trials": planned - len(trials),+ "accuracy": correct / planned if planned else None,+ "selection_accuracy": sum(t["correct"] for t in match_trials) / planned_match if planned_match else None,+ "correct_abstention": sum(t["correct"] for t in no_match_trials) / planned_no_match+ if planned_no_match+ else None,+ "false_selection_on_no_match": sum(1 for t in no_match_trials if t["outcome"] in {"found", "rejected"})+ / planned_no_match+ if planned_no_match+ else None,+ "abstained_on_answerable": sum(1 for t in match_trials if t["outcome"] == "no_match"),+ "out_of_candidate": outcomes["rejected"],+ "errors": outcomes["error"] + outcomes["unavailable"] + outcomes["usage_limited"] + outcomes["infra_error"],+ "timeouts": outcomes["timeout"],+ "outcomes": dict(outcomes),+ "latency_p50_seconds": statistics.median(latencies) if latencies else None,+ "latency_p90_seconds": _percentile(latencies, 0.9),+ # Selector latency is quantized by the session's ~10s poll; model time is the gateway's per-call sum.+ "model_seconds_p50": statistics.median(model_times) if model_times else None,+ "model_seconds_p90": _percentile(model_times, 0.9),+ "gateway_cost_usd": gateway_cost,+ "gateway_cost_per_case_usd": gateway_cost / planned if planned else None,+ "gateway_cost_per_correct_usd": gateway_cost / correct if correct else None,+ "list_cost_usd": None if any(c is None for c in list_costs) else sum(c for c in list_costs if c is not None),+ "tokens": {+ key: sum(t[key] for t in trials)+ for key in ("input_tokens", "output_tokens", "cache_read_tokens", "cache_write_tokens", "reasoning_tokens")+ },+ "model_calls": sum(t["calls"] for t in trials),+ "service_tiers": dict(tiers),+ "models_served": sorted({m for t in trials for m in t["models_served"]}),+ "providers": sorted({p for t in trials for p in t["providers"]}),+ "efforts_observed": sorted({e for t in trials for e in t["efforts"]}),+ "products": sorted({p for t in trials for p in t["products"]}),+ "failed_model_calls": sum(t["failed_calls"] for t in trials),+ "per_case": {+ name: {+ "correct": sum(t["correct"] for t in trials if t["case"] == name),+ "trials": sum(1 for t in trials if t["case"] == name),+ "selected": [t["selected"] or t["outcome"] for t in trials if t["case"] == name],+ }+ for name in sorted({t["case"] for t in trials})+ },+ "trials": trials,+ }+++def _fmt(value: Any, kind: str = "") -> str:+ if value is None:+ return "n/a"+ if kind == "%":+ return f"{value:.0%}"+ if kind == "$":+ return f"${value:.4f}"+ if kind == "s":+ return f"{value:.0f}s"+ return str(value)+++def render(arms: list[dict[str, Any]]) -> str:+ header = [+ "Arm",+ "Accuracy",+ "Selection",+ "Abstain",+ "False sel.",+ "Out-of-cand.",+ "Errors",+ "Timeouts",+ "p50",+ "p90",+ "Model p50",+ "$/case",+ "$/correct",+ "List $/case",+ "Tiers",+ ]+ lines = ["| " + " | ".join(header) + " |", "|" + "---|" * len(header)]+ for arm in arms:+ if arm.get("status") != "ok":+ lines.append(f"| {arm['arm']} | {arm.get('status')} |" + " |" * (len(header) - 2))+ continue+ lines.append(+ "| "+ + " | ".join(+ [+ arm["arm"],+ f"{_fmt(arm['accuracy'], '%')} ({sum(t['correct'] for t in arm['trials']):.0f}/{arm['planned_trials']})",+ _fmt(arm["selection_accuracy"], "%"),+ _fmt(arm["correct_abstention"], "%"),+ _fmt(arm["false_selection_on_no_match"], "%"),+ str(arm["out_of_candidate"]),+ str(arm["errors"] + arm["missing_trials"]),+ str(arm["timeouts"]),+ _fmt(arm["latency_p50_seconds"], "s"),+ _fmt(arm["latency_p90_seconds"], "s"),+ _fmt(arm["model_seconds_p50"], "s"),+ _fmt(arm["gateway_cost_per_case_usd"], "$"),+ _fmt(arm["gateway_cost_per_correct_usd"], "$"),+ _fmt(+ arm["list_cost_usd"] / arm["planned_trials"] if arm["list_cost_usd"] is not None else None, "$"+ ),+ ", ".join(f"{k}:{v}" for k, v in sorted(arm["service_tiers"].items())),+ ]+ )+ + " |"+ )+ return "\n".join(lines) + "\n"+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])+ parser.add_argument("comparison_dir", type=Path)+ parser.add_argument("--json", type=Path, help="Write the full per-arm and per-trial summary here.")+ args = parser.parse_args(argv)+ comparison = json.loads((args.comparison_dir / "comparison.json").read_text())+ arms = [summarize_arm(run) for run in comparison["runs"]]+ output = {+ "fixture_version": FIXTURE_VERSION,+ "fixture_fingerprint": fixture_fingerprint(),+ "settings": comparison["settings"],+ "issues": comparison.get("issues", []),+ "arms": arms,+ }+ if args.json:+ args.json.write_text(json.dumps(output, indent=2, default=str))+ sys.stdout.write(render(arms))+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/products/tasks/evals/repo_selection/runner.py b/products/tasks/evals/repo_selection/runner.pynew file mode 100644index 000000000000..da3f5c02b53e--- /dev/null+++ b/products/tasks/evals/repo_selection/runner.py@@ -0,0 +1,286 @@+"""Run the production repository selector for one eval case.++Calls `select_repository` itself, so the prompt, candidate filtering, cache sync, sandbox+runtime, and result validation are the code production runs. The arm's model, runtime adapter,+reasoning effort, and service tier come from the harness context.+"""++from __future__ import annotations++import time+import asyncio+import logging+from collections.abc import Awaitable, Callable+from typing import Literal++from pydantic import Field++from posthog.egress.limiter.policies import Priority+from posthog.temporal.common.client import async_connect++from products.posthog_ai.eval_harness.config import AgentArtifacts, SandboxedEvalCase+from products.posthog_ai.eval_harness.harness.providers import SandboxProviderStrategy+from products.posthog_ai.eval_harness.runner import (+ EvalCaseResult,+ SandboxedCaseRunner,+ _finish_workflow,+ _parse_artifacts_from_log,+)+from products.tasks.backend.facade import api as tasks_api+from products.tasks.backend.facade.agents import CustomPromptSandboxContext, TurnPollTimeout+from products.tasks.backend.facade.contracts import TaskRunDTO+from products.tasks.backend.facade.repo_selection import (+ RepoSelectionRejectedError,+ RepoSelectionUnavailableError,+ RepoSelectionUsageLimitError,+ select_repository,+)+from products.tasks.backend.models import TaskRun++from .fixtures import RepoSelectionScenario++logger = logging.getLogger(__name__)++SelectorOutcome = Literal[+ "auto", # the cascade picked a repository without the agent+ "no_repo", # the needs-repo classifier decided no repository is needed+ "found", # the agent picked a candidate+ "no_match", # the agent returned null+ "rejected", # the agent returned a repository outside the candidate list+ "unavailable", # selection could not run (no eligible cached candidates)+ "usage_limited", # the gateway refused the model calls with a 402+ "timeout", # the selector outran its deadline+ "error", # any other failure+ "not_run", # an earlier stage ended the request+]++# Leaves room inside the harness case budget to read logs and tear the sandbox down, so a slow+# arm records a selector timeout instead of an infrastructure timeout.+SELECTOR_CLEANUP_RESERVE_SECONDS = 90+TASK_DISCOVERY_INTERVAL_SECONDS = 1.0+++class RepoSelectionArtifacts(AgentArtifacts):+ stage: str = "selector"+ outcome: SelectorOutcome = "not_run"+ selected_repository: str | None = None+ reason: str = ""+ error_type: str | None = None+ selector_seconds: float | None = None+ task_id: str | None = None+ task_run_id: str | None = None+ distinct_id: str | None = None+ model: str | None = None+ runtime_adapter: str | None = None+ reasoning_effort: str | None = None+ requested_service_tier: str | None = None+ classifier_needs_repo: bool | None = None+ cascade_reason: str | None = None+ notes: list[str] = Field(default_factory=list)+++def _latest_task_run(team_id: int) -> TaskRunDTO | None:+ run_id = TaskRun.objects.filter(team_id=team_id).order_by("-created_at").values_list("id", flat=True).first()+ return tasks_api.get_task_run(run_id, team_id=team_id) if run_id is not None else None+++def _distinct_id(user_id: int) -> str | None:+ from posthog.models import User # noqa: PLC0415 — keeps Django models off the harness import path++ return User.objects.filter(id=user_id).values_list("distinct_id", flat=True).first()+++async def _finish(+ task_run: TaskRunDTO,+ *,+ successful: bool,+ provider: SandboxProviderStrategy,+ on_cleanup: Callable[[], Awaitable[None]] | None,+) -> None:+ if on_cleanup is not None:+ try:+ await on_cleanup()+ except Exception:+ logger.warning("Could not report cleanup progress", exc_info=True)+ try:+ client = await async_connect()+ await _finish_workflow(+ client.get_workflow_handle(task_run.workflow_id),+ status="completed" if successful else "failed",+ reason=None if successful else "Repository selection eval failed",+ )+ except Exception:+ logger.warning("Could not confirm repository selection workflow cleanup", exc_info=True)+ finally:+ await asyncio.to_thread(provider.cleanup_case, str(task_run.task_id))+++async def _select(+ scenario: RepoSelectionScenario,+ context: CustomPromptSandboxContext,+ *,+ origin_product: tasks_api.TaskOriginProduct,+ priority: Priority,+ deadline: float | None,+ artifacts: RepoSelectionArtifacts,+) -> None:+ """Run the production selector and record its decision or failure on `artifacts`."""+ try:+ result = await asyncio.wait_for(+ select_repository(+ team_id=context.team_id,+ user_id=context.user_id,+ context=scenario.context_block,+ origin_product=origin_product,+ priority=priority,+ sandbox_environment_id=context.sandbox_environment_id,+ model=context.model,+ runtime_adapter=context.runtime_adapter,+ reasoning_effort=context.reasoning_effort,+ service_tier=context.service_tier,+ ),+ timeout=deadline,+ )+ artifacts.outcome = "found" if result.repository else "no_match"+ artifacts.selected_repository = result.repository+ artifacts.reason = result.reason+ artifacts.exit_code = 0+ except RepoSelectionRejectedError as error:+ artifacts.outcome = "rejected"+ artifacts.selected_repository = error.returned_repository+ artifacts.reason = error.reason+ # A pick outside the candidates is a model answer, so it is scored rather than excluded.+ artifacts.exit_code = 0+ except RepoSelectionUnavailableError as error:+ artifacts.outcome, artifacts.error_type, artifacts.stderr = "unavailable", type(error).__name__, error.reason+ except RepoSelectionUsageLimitError as error:+ artifacts.outcome, artifacts.error_type, artifacts.stderr = "usage_limited", type(error).__name__, error.message+ except TimeoutError:+ artifacts.outcome, artifacts.error_type = "timeout", "TimeoutError"+ artifacts.stderr = f"selector deadline of {deadline or 0:.0f}s passed"+ except Exception as error:+ logger.warning("Repository selection eval failed", exc_info=True)+ artifacts.outcome, artifacts.error_type, artifacts.stderr = "error", type(error).__name__, str(error)[:2000]+++async def _collect_and_finish(+ context: CustomPromptSandboxContext,+ task_run: TaskRunDTO | None,+ artifacts: RepoSelectionArtifacts,+ *,+ provider: SandboxProviderStrategy,+ on_cleanup: Callable[[], Awaitable[None]] | None,+) -> str:+ """Read the selector's agent log, then end its workflow and reclaim its sandbox."""+ task_run = task_run or await asyncio.to_thread(_latest_task_run, context.team_id)+ if task_run is None:+ return ""+ provider.register_task(str(task_run.task_id))+ artifacts.task_id, artifacts.task_run_id = str(task_run.task_id), str(task_run.id)+ raw_log = ""+ try:+ conversation = await asyncio.to_thread(+ tasks_api.read_task_run_conversation_logs, task_run.id, task_run.task_id, context.team_id+ )+ raw_log = conversation or ""+ except Exception:+ logger.warning("Could not read the selector's agent log", exc_info=True)+ artifacts.notes.append("agent log unavailable")+ await _finish(task_run, successful=artifacts.exit_code == 0, provider=provider, on_cleanup=on_cleanup)+ return raw_log+++async def run_selector(+ scenario: RepoSelectionScenario,+ context: CustomPromptSandboxContext,+ *,+ origin_product: tasks_api.TaskOriginProduct,+ priority: Priority,+ provider: SandboxProviderStrategy,+ on_cleanup: Callable[[], Awaitable[None]] | None,+ timeout_seconds: float | None,+ artifacts: RepoSelectionArtifacts | None = None,+) -> EvalCaseResult:+ """Run the selector on the scenario's thread and record what it decided and what it cost."""+ artifacts = artifacts or RepoSelectionArtifacts(exit_code=1)+ # A routing case passes the cascade's artifacts, which carry its success status.+ artifacts.exit_code = 1+ artifacts.model = context.model+ artifacts.runtime_adapter = context.runtime_adapter+ artifacts.reasoning_effort = context.reasoning_effort+ artifacts.requested_service_tier = context.service_tier+ artifacts.distinct_id = await asyncio.to_thread(_distinct_id, context.user_id)+ deadline = None if timeout_seconds is None else max(30.0, timeout_seconds - SELECTOR_CLEANUP_RESERVE_SECONDS)++ started = time.monotonic()+ selection = asyncio.create_task(+ _select(+ scenario,+ context,+ origin_product=origin_product,+ priority=priority,+ deadline=deadline,+ artifacts=artifacts,+ )+ )+ task_run: TaskRunDTO | None = None+ raw_log = ""+ try:+ # Register the sandbox as soon as its task exists, so the run-end sweep reclaims it even+ # when the case is cancelled before the selector returns.+ while not selection.done():+ if task_run is None:+ task_run = await asyncio.to_thread(_latest_task_run, context.team_id)+ if task_run is not None:+ provider.register_task(str(task_run.task_id))+ await asyncio.wait({selection}, timeout=TASK_DISCOVERY_INTERVAL_SECONDS)+ await selection+ finally:+ if not selection.done():+ selection.cancel()+ await asyncio.gather(selection, return_exceptions=True)+ artifacts.selector_seconds = time.monotonic() - started+ collection = asyncio.create_task(+ _collect_and_finish(context, task_run, artifacts, provider=provider, on_cleanup=on_cleanup)+ )+ try:+ raw_log = await asyncio.shield(collection)+ except asyncio.CancelledError:+ try:+ await asyncio.shield(collection)+ except Exception:+ logger.warning("Could not finish cancelled repository selection cleanup", exc_info=True)+ raise+ parsed = _parse_artifacts_from_log(raw_log, artifacts.selector_seconds, agent_finished=artifacts.exit_code == 0)+ merged = artifacts.model_copy(+ update={+ **{key: value for key, value in parsed.model_dump().items() if key not in {"exit_code", "stderr"}},+ "stderr": artifacts.stderr or parsed.stderr,+ }+ )+ return EvalCaseResult(artifacts=merged, raw_log=raw_log, trace_id=artifacts.task_run_id or "")+++def selector_case_runner(+ scenarios: dict[str, RepoSelectionScenario], *, origin_product: tasks_api.TaskOriginProduct, priority: Priority+) -> SandboxedCaseRunner:+ async def run(+ case: SandboxedEvalCase,+ context: CustomPromptSandboxContext,+ *,+ provider: SandboxProviderStrategy,+ on_cleanup: Callable[[], Awaitable[None]] | None = None,+ timeout_seconds: float | None = None,+ on_poll_timeout: Callable[[TurnPollTimeout], None] | None = None,+ ) -> EvalCaseResult:+ return await run_selector(+ scenarios[case.name],+ context,+ origin_product=origin_product,+ priority=priority,+ provider=provider,+ on_cleanup=on_cleanup,+ timeout_seconds=timeout_seconds,+ )++ return rundiff --git a/products/tasks/evals/repo_selection/scorers.py b/products/tasks/evals/repo_selection/scorers.pynew file mode 100644index 000000000000..c87943901664--- /dev/null+++ b/products/tasks/evals/repo_selection/scorers.py@@ -0,0 +1,177 @@+"""Deterministic scorers for the repository-selection evals.++Each scorer reads only its own `expected` key and skips when the case does not declare it.+Errors, timeouts, and picks outside the candidates score 0, so a failed trial stays in the+denominator instead of vanishing from it. Rationale quality is diagnostic and unscored.+"""++from __future__ import annotations++from typing import Any++from products.posthog_ai.eval_harness.scorers.contract import Score, Scorer++from .fixtures import RoutingExpectation, SelectorExpectation++SCORER_VERSION = "1"++SELECTION_KEY = "repo_selection_correct"+ABSTENTION_KEY = "correct_abstention"+COMPLETED_KEY = "selector_completed"+ROUTING_STAGE_KEY = "routing_stage_correct"+ROUTING_REPOSITORY_KEY = "routing_repository_correct"++_SELECTOR_FAILURES = frozenset({"unavailable", "usage_limited", "timeout", "error"})+++def selector_expected(expectation: SelectorExpectation) -> dict[str, Any]:+ spec = {"accepted": list(expectation.accepted), "abstain": expectation.abstain}+ expected: dict[str, Any] = {SELECTION_KEY: spec, COMPLETED_KEY: {}}+ if expectation.abstain:+ expected[ABSTENTION_KEY] = spec+ return expected+++def routing_expected(expectation: RoutingExpectation) -> dict[str, Any]:+ expected: dict[str, Any] = {ROUTING_STAGE_KEY: {"stage": expectation.stage, "outcome": expectation.outcome}}+ if expectation.repository is not None:+ expected[ROUTING_REPOSITORY_KEY] = {"accepted": [expectation.repository], "abstain": False}+ elif expectation.selector is not None:+ expected[ROUTING_REPOSITORY_KEY] = {+ "accepted": list(expectation.selector.accepted),+ "abstain": expectation.selector.abstain,+ }+ return expected+++def _decision(output: dict[str, Any] | None) -> tuple[str, str | None]:+ if not output:+ return "error", None+ outcome = output.get("outcome")+ if not isinstance(outcome, str):+ # The harness's timeout output carries no outcome.+ return "timeout" if "timeout" in str(output.get("stderr", "")) else "error", None+ repository = output.get("selected_repository")+ return outcome, repository.lower() if isinstance(repository, str) else None+++def judge_selection(output: dict[str, Any] | None, spec: dict[str, Any]) -> tuple[float, str]:+ """Score one selector decision against an accepted set or an expected abstention."""+ outcome, repository = _decision(output)+ accepted = {repo.lower() for repo in spec.get("accepted", [])}+ if outcome in _SELECTOR_FAILURES:+ return 0.0, f"selector {outcome}"+ if outcome == "rejected":+ return 0.0, f"picked {repository!r}, which is not a candidate"+ if spec.get("abstain"):+ if outcome == "no_match":+ return 1.0, "abstained as expected"+ return 0.0, f"picked {repository!r} where no candidate fits"+ if outcome == "no_match":+ return 0.0, "abstained where a candidate fits"+ if repository in accepted:+ return 1.0, f"picked {repository!r}"+ return 0.0, f"picked {repository!r}, expected one of {sorted(accepted)}"+++class _ExpectedKeyScorer(Scorer):+ key: str++ def _name(self) -> str:+ return self.key++ def _spec(self, expected: Any) -> dict[str, Any] | None:+ spec = expected.get(self.key) if isinstance(expected, dict) else None+ return spec if isinstance(spec, dict) else None++ def _score(self, value: float | None, reason: str, output: dict[str, Any] | None) -> Score:+ outcome, repository = _decision(output)+ return Score(+ name=self._name(),+ score=value,+ metadata={+ "reason": reason,+ "outcome": outcome,+ "selected_repository": repository,+ "scorer_version": SCORER_VERSION,+ },+ )+++class RepoSelectionCorrect(_ExpectedKeyScorer):+ """1 when the pick is in the accepted set, or the selector abstains where no candidate fits."""++ key = SELECTION_KEY++ def _run_eval_sync(self, output: dict[str, Any] | None, expected: Any = None, **kwargs: Any) -> Score:+ spec = self._spec(expected)+ if spec is None:+ return self._score(None, "case declares no expected selection", output)+ value, reason = judge_selection(output, spec)+ return self._score(value, reason, output)+++class CorrectAbstention(_ExpectedKeyScorer):+ """On cases with no plausible candidate only: 1 for an abstention, 0 for any pick or failure."""++ key = ABSTENTION_KEY++ def _run_eval_sync(self, output: dict[str, Any] | None, expected: Any = None, **kwargs: Any) -> Score:+ spec = self._spec(expected)+ if spec is None:+ return self._score(None, "case has a plausible candidate", output)+ value, reason = judge_selection(output, spec)+ return self._score(value, reason, output)+++class SelectorCompleted(_ExpectedKeyScorer):+ """1 when the selector returned a decision, 0 for an error, timeout, or refusal."""++ key = COMPLETED_KEY++ def _run_eval_sync(self, output: dict[str, Any] | None, expected: Any = None, **kwargs: Any) -> Score:+ if self._spec(expected) is None:+ return self._score(None, "not a selector case", output)+ outcome, _ = _decision(output)+ if outcome in _SELECTOR_FAILURES:+ return self._score(0.0, f"selector {outcome}", output)+ return self._score(1.0, f"selector returned {outcome}", output)+++class RoutingStageCorrect(_ExpectedKeyScorer):+ """1 when the request stops at the expected stage with the expected outcome."""++ key = ROUTING_STAGE_KEY++ def _run_eval_sync(self, output: dict[str, Any] | None, expected: Any = None, **kwargs: Any) -> Score:+ spec = self._spec(expected)+ if spec is None:+ return self._score(None, "case declares no routing expectation", output)+ stage = (output or {}).get("stage")+ outcome, _ = _decision(output)+ actual = f"{stage}/{outcome}"+ wanted = f"{spec['stage']}/{spec['outcome']}"+ return self._score(1.0 if actual == wanted else 0.0, f"reached {actual}, expected {wanted}", output)+++class RoutingRepositoryCorrect(_ExpectedKeyScorer):+ """1 when the routed repository is in the accepted set; skipped for no-repo expectations."""++ key = ROUTING_REPOSITORY_KEY++ def _run_eval_sync(self, output: dict[str, Any] | None, expected: Any = None, **kwargs: Any) -> Score:+ spec = self._spec(expected)+ if spec is None:+ return self._score(None, "case expects no repository", output)+ outcome, _ = _decision(output)+ if outcome == "auto":+ # A cascade pick is a selection for scoring purposes.+ output = {**(output or {}), "outcome": "found"}+ elif outcome == "not_run" or outcome == "no_repo":+ return self._score(0.0, f"request stopped before selection ({outcome})", output)+ value, reason = judge_selection(output, spec)+ return self._score(value, reason, output)+++SELECTOR_SCORERS: tuple[type[_ExpectedKeyScorer], ...] = (RepoSelectionCorrect, CorrectAbstention, SelectorCompleted)+ROUTING_SCORERS: tuple[type[_ExpectedKeyScorer], ...] = (RoutingStageCorrect, RoutingRepositoryCorrect)diff --git a/products/tasks/evals/repo_selection/seeders.py b/products/tasks/evals/repo_selection/seeders.pynew file mode 100644index 000000000000..4fef2b8763dc--- /dev/null+++ b/products/tasks/evals/repo_selection/seeders.py@@ -0,0 +1,101 @@+"""Install the synthetic repository catalog into one case's isolated team.++The seeded team-level GitHub integration carries a placeholder token and a fresh light and heavy+cache, so the production selector reads the fixture instead of GitHub: `sync_full_cache` takes its+TTL fast path, and the agent's `execute-sql` queries see only these rows.+"""++from __future__ import annotations++from collections.abc import Callable+from typing import Any++from django.utils import timezone++from posthog.egress.limiter.policies import Priority+from posthog.integrations import IntegrationManager+from posthog.models.integration import GitHubIntegration+from posthog.models.integration_repository_cache import IntegrationRepositoryCacheEntry+from posthog.models.repo_routing_rule import RepoRoutingRule++from products.tasks.backend.facade.agents import CustomPromptSandboxContext++from .fixtures import FIXTURE_VERSION, ORG, REPOSITORIES, RepoSelectionScenario, fixture_fingerprint++# Not a credential. GitHub rejects it, so a `gh api` call from the sandbox cannot reach real repositories.+PLACEHOLDER_GITHUB_TOKEN = "fixture-token-not-a-github-credential"+++def install_repository_catalog(team_id: int, user_id: int, scenario: RepoSelectionScenario) -> dict[str, Any]:+ now = timezone.now()+ record = IntegrationManager.github.oauth.create(+ team_id,+ external_id=f"eval-fixture-{team_id}",+ config={"installation_id": f"eval-fixture-{team_id}", "account": {"type": "Organization", "name": ORG}},+ sensitive_config={"access_token": PLACEHOLDER_GITHUB_TOKEN},+ created_by_id=user_id,+ )+ # The SDK has no repository cache write: production fills it from GitHub. The fixture writes+ # the synthetic listing directly so the selector never calls GitHub.+ github_integration = GitHubIntegration(record, source="repo_selection_eval", priority=Priority.BATCH).integration+ github_integration.repository_cache = [+ {+ "id": index,+ "name": repo.name,+ "full_name": repo.full_name,+ "private": repo.private,+ "archived": repo.archived,+ "language": repo.primary_language,+ "default_branch": "main",+ }+ for index, repo in enumerate(REPOSITORIES, start=1)+ ]+ github_integration.repository_cache_updated_at = now+ github_integration.save( # nosemgrep: integration-model-through-sdk -- synthetic eval fixture; the SDK has no repository cache write+ update_fields=["repository_cache", "repository_cache_updated_at"]+ )+ IntegrationRepositoryCacheEntry.objects.bulk_create(+ [+ IntegrationRepositoryCacheEntry(+ integration_id=record.id,+ team_id=team_id,+ full_name=repo.full_name.lower(),+ description=repo.description,+ topics=list(repo.topics),+ archived=repo.archived,+ primary_language=repo.primary_language,+ default_branch="main",+ default_branch_sha=repo.default_branch_sha,+ readme=repo.readme,+ tree_paths="\n".join(repo.tree_paths),+ tree_truncated=False,+ )+ for repo in REPOSITORIES+ ]+ )+ RepoRoutingRule.objects.bulk_create(+ [+ RepoRoutingRule(+ team_id=team_id,+ rule_text=rule.rule_text,+ repository=rule.repository,+ priority=priority,+ created_by_id=user_id,+ )+ for priority, rule in enumerate(scenario.routing_rules)+ ]+ )+ return {+ "fixture_version": FIXTURE_VERSION,+ "fixture_fingerprint": fixture_fingerprint(),+ "candidate_repositories": sorted(repo.full_name.lower() for repo in REPOSITORIES if not repo.archived),+ "routing_rules": [f"{rule.rule_text} → {rule.repository}" for rule in scenario.routing_rules],+ }+++def scenario_seeder(scenario: RepoSelectionScenario) -> Callable[[CustomPromptSandboxContext], dict[str, Any]]:+ def seed(context: CustomPromptSandboxContext) -> dict[str, Any]:+ return install_repository_catalog(context.team_id, context.user_id, scenario)++ seed.__name__ = f"seed_{scenario.name}"+ return seed