Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(evals): default sandbox eval models to opus 5.5 and gpt-6.1 sol (#120089)

Daniel Parker019fd6e9f627
diff --git a/.agents/skills/writing-evals/SKILL.md b/.agents/skills/writing-evals/SKILL.md
index b1617f1f415b..06cfd02ec133 100644
--- a/.agents/skills/writing-evals/SKILL.md
+++ b/.agents/skills/writing-evals/SKILL.md
@@ -163,7 +163,7 @@ hogli evals eval_skill_distribution --skill-delivery exec # exec-distributed sk
- Team cloning and case seeding use one setup slot on ordinary local machines and four when `CODER` or `CI` is set.
- `--trials N` repeats every case for variance on stochastic behavior; `--fail-under <fraction>` gates the run's mean score.
- `--skill-delivery bundled` is the default native-skill path. `--skill-delivery exec` enables the MCP skill prompt and clears all native skill directories; compare the modes in separate runs because MCP configuration is process-wide.
-- `--agent-runtime codex` runs the OpenAI Codex harness (default model `gpt-5.5`) instead of Claude; it requires `LLM_GATEWAY_OPENAI_API_KEY`. The runtime/model land in the Braintrust experiment metadata, so compare scores within one runtime.
+- `--agent-runtime codex` runs the OpenAI Codex harness (default model `gpt-6.1-sol`) instead of Claude; it requires `LLM_GATEWAY_OPENAI_API_KEY`. The runtime/model land in the Braintrust experiment metadata, so compare scores within one runtime.
- Every real eval invocation mirrors its complete stdout and stderr to `products/posthog_ai/eval_harness/logs/harness/<timestamp>_<id>.log`. The ending label identifies it as the full run transcript, the last terminal line is its absolute path, and `logs/harness/latest.log` points to the newest transcript. `--list` and argument errors do not create one.
- The plain-text summary is stable for people and agents: only the overall run says `PASS` or `FAIL`; completed suites, experiments, and cases say `DONE`; experiment blocks label scores, URLs, and agent-log directories.
- `EXPORT_EVAL_RESULTS=1` additionally appends one structured Braintrust summary per experiment to `eval_results.jsonl`. It does not replace the always-on transcript.
diff --git a/products/posthog_ai/eval_harness/README.md b/products/posthog_ai/eval_harness/README.md
index 2ce0cfe9aeee..87a88875a9c1 100644
--- a/products/posthog_ai/eval_harness/README.md
+++ b/products/posthog_ai/eval_harness/README.md
@@ -169,7 +169,7 @@ The experiment metadata records `skill_delivery`; compare the shared `expected_s
### Codex runtime
-`--agent-runtime codex` runs the same agent-server with OpenAI's Codex harness instead of Claude, defaulting the model to `gpt-5.5`.
+`--agent-runtime codex` runs the same agent-server with OpenAI's Codex harness instead of Claude, defaulting the model to `gpt-6.1-sol`.
It requires `LLM_GATEWAY_OPENAI_API_KEY` in the environment (checked by preflight), which the harness's LLM gateway uses to proxy the agent's OpenAI calls.
Experiment names don't change with the runtime or skill delivery. Each Braintrust experiment and every PostHog event records `agent_model`, `trials`, `git_sha` and `git_dirty`. Sandboxed runs add `agent_runtime`, `skill_delivery` and `reasoning_effort`. Use these fields to separate runs. A comparison across runtimes measures the model and runtime together.
@@ -191,8 +191,8 @@ eval_comparison_args=(
--skill-delivery exec
--trials 1
)
-EXPORT_EVAL_RESULTS=1 hogli evals "${eval_comparison_args[@]}" --agent-runtime claude --agent-model claude-opus-5
-EXPORT_EVAL_RESULTS=1 hogli evals "${eval_comparison_args[@]}" --agent-runtime codex --agent-model gpt-5.5
+EXPORT_EVAL_RESULTS=1 hogli evals "${eval_comparison_args[@]}" --agent-runtime claude --agent-model claude-opus-5-5
+EXPORT_EVAL_RESULTS=1 hogli evals "${eval_comparison_args[@]}" --agent-runtime codex --agent-model gpt-6.1-sol
```
Compare the named scorer results, completed cases, timeouts, infrastructure errors, and total duration in both final summaries.
diff --git a/products/posthog_ai/eval_harness/harness/cli.py b/products/posthog_ai/eval_harness/harness/cli.py
index a7f9b68a1d51..22122902bdfd 100644
--- a/products/posthog_ai/eval_harness/harness/cli.py
+++ b/products/posthog_ai/eval_harness/harness/cli.py
@@ -12,8 +12,8 @@ from .retries import DEFAULT_INFRA_RETRIES
# bare-id allowlist with startswith, so the prefixed form is rejected with a 403 and the agent
# finishes without doing anything. A run that dies at the gate fails as an error rather than
# scoring zero on every outcome scorer.
-DEFAULT_AGENT_MODEL = "claude-opus-5"
-DEFAULT_CODEX_AGENT_MODEL = "gpt-5.5"
+DEFAULT_AGENT_MODEL = "claude-opus-5-5"
+DEFAULT_CODEX_AGENT_MODEL = "gpt-6.1-sol"
# Literal mirror of products.tasks' RuntimeAdapter values: this module must stay
# Django-free (see harness/AGENTS.md), so it cannot import the enum.