The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
Minor skill update + regraded benchmarks
narrow-adventureaf7e3639613e
diff --git a/cli/pkg/mcpserver/knowledge/debug-flow.md b/cli/pkg/mcpserver/knowledge/debug-flow.mdindex 019a4f59..7ed30a09 100644--- a/cli/pkg/mcpserver/knowledge/debug-flow.md+++ b/cli/pkg/mcpserver/knowledge/debug-flow.md@@ -23,10 +23,10 @@ traceway exceptions list --since 7d --search "checkout" --output json \### 2. Drill into the issue```bash-traceway exceptions show <hash>+traceway exceptions show <hash> --page-size 5```-This is the high-value call: full stack trace, occurrence list with `recordedAt`, `attributes` (user IDs, app versions, request context), and optional `traceId` / `spanId` / `sessionId` per occurrence. `traceId` is the OpenTelemetry trace id (32 hex characters), the same id the request's logs and spans carry. The response also has `relatedEntity`: the endpoint, task or AI trace the newest occurrence happened in, with the `id` and `recordedAt` that `endpoints show` / `tasks show` / `ai-traces show` take. `firstSeen` correlates with deploys: a group that first appeared right after a release points at that release's diff. A bogus hash exits 5 with `not_found`; fall back to search.+This is the high-value call: full stack trace, occurrence list with `recordedAt`, `attributes` (user IDs, app versions, request context), and optional `traceId` / `spanId` / `sessionId` per occurrence. Occurrences come newest first, and each carries its own full stack trace, so the default page of 50 runs to well over 100k characters and gets cut off by any tool output limit; `--page-size 5` keeps the group header plus the five newest occurrences, which is what a first read needs. Keep the default page only when you pipe the JSON through `jq` to reduce it, as below. `traceId` is the OpenTelemetry trace id (32 hex characters), the same id the request's logs and spans carry. The response also has `relatedEntity`: the endpoint, task or AI trace the newest occurrence happened in, with the `id` and `recordedAt` that `endpoints show` / `tasks show` / `ai-traces show` take. `firstSeen` correlates with deploys: a group that first appeared right after a release points at that release's diff. A bogus hash exits 5 with `not_found`; fall back to search.**When the user gave an issue URL (or hash), fix the LAST occurrence — not "the group".** A single hash can bundle *several distinct errors*: the hash is computed from a normalized stack trace with the message stripped, so two unrelated failures that share their top frames (e.g. both captured at the same middleware/recovery frame) collapse into one group. The group's representative stack trace and `firstSeen` may belong to a different, now-dormant error than the one the user is looking at. Anchor on the most recent occurrence and fix that specific failure path:diff --git a/cli/test/evals/README.md b/cli/test/evals/README.mdindex f9661cdf..09c65fcb 100644--- a/cli/test/evals/README.md+++ b/cli/test/evals/README.md@@ -110,7 +110,14 @@ oracle on every arm it lists and writes `results.json` and `summary.md` to`--out` (default `.cache/oracle/<timestamp>/`).`twevals report <dir>... --out DIR` merges shard results (`--out` isrequired for more than one directory, and a run that appears twice is an-error). `twevals compare <dir>`+error unless `--replace` says the later directory is a rerun that replaces+it). `twevals regrade <dir>` runs the offline graders (answer, trajectory,+efficiency, grounding, report) again over a run directory's stored records+with the cases and graders as they are now, keeps the verdicts that needed+the live instance (state, hidden tests, judge, setup verify), recomputes each+status and rewrites `results.json`, `summary.md` and every `result.json`+(`--out DIR` writes the first two elsewhere instead). It is how a grader fix+is applied to runs already paid for. `twevals compare <dir>`applies the regression rule. `twevals baseline update <dir>` rewrites`baseline.json` for you to commit.diff --git a/cli/test/evals/cmd/twevals/commands_ops.go b/cli/test/evals/cmd/twevals/commands_ops.goindex 31c944ee..49d3b2a4 100644--- a/cli/test/evals/cmd/twevals/commands_ops.go+++ b/cli/test/evals/cmd/twevals/commands_ops.go@@ -116,6 +116,7 @@ func cmdEnvUp(ctx context.Context, args []string) error {func cmdReport(args []string) error {fs := flag.NewFlagSet("report", flag.ExitOnError)merge := fs.Bool("merge", false, "merge several run directories (shards) into one")+ replace := fs.Bool("replace", false, "a run that a later directory repeats replaces the earlier one (a rerun) instead of being an error")out := fs.String("out", "", "write results.json and summary.md here")pos := parseInterleaved(fs, args)if len(pos) == 0 {@@ -143,7 +144,7 @@ func cmdReport(args []string) error {if *out == "" {return &exitError{2, "merging several run directories needs --out (writing into the first one would overwrite that shard's results)"}}- merged, notes, err := report.Merge(parts)+ merged, notes, err := report.Merge(parts, *replace)if err != nil {return &exitError{2, err.Error()}}diff --git a/cli/test/evals/cmd/twevals/main.go b/cli/test/evals/cmd/twevals/main.goindex eec95d66..041bed0d 100644--- a/cli/test/evals/cmd/twevals/main.go+++ b/cli/test/evals/cmd/twevals/main.go@@ -22,7 +22,8 @@ Free:twevals plan [selection] --models opus,astra [--runs N] --max-cost-usd X [--shards] [--out plan.json] [--summary FILE]twevals oracle [--all | <case-id>... | --suites S] [--arms A] [-j N] [--keep] [--out DIR]twevals env up <case-id>- twevals report <run-dir>... [--merge] [--out DIR]+ twevals report <run-dir>... [--merge] [--replace] [--out DIR]+ twevals regrade <run-dir> [--out DIR]twevals compare <run-dir> [baseline.json]twevals baseline update <run-dir> [--out baseline.json]@@ -32,7 +33,9 @@ Paid (OpenRouter; guarded, see README):Selection flags: --preset smoke|core|full, --cases (comma-separated ids or globs; overrides the preset), --suites, --tags, --arms.list defaults to --preset full, plan and run to --preset smoke. oracle writes results.json and summary.md to --out-(default .cache/oracle/<timestamp>).+(default .cache/oracle/<timestamp>). regrade runs the offline graders (answer, trajectory, grounding, report) again+over a run directory's stored records and keeps the state, hidden-test and judge verdicts; it rewrites the directory's+results.json and summary.md unless --out names another place.Exit codes: 0 ok, 1 a regression, a failed oracle or an error, 2 bad usage or partial (cost ceiling, or infra_error above 10%).`@@ -68,6 +71,8 @@ func main() {err = cmdEnvUp(ctx, args[1:])case "report":err = cmdReport(args)+ case "regrade":+ err = cmdRegrade(args)case "compare":err = cmdCompare(args)case "baseline":diff --git a/cli/test/evals/cmd/twevals/regrade.go b/cli/test/evals/cmd/twevals/regrade.gonew file mode 100644index 00000000..36a4f57c--- /dev/null+++ b/cli/test/evals/cmd/twevals/regrade.go@@ -0,0 +1,258 @@+package main++import (+ "encoding/json"+ "errors"+ "flag"+ "fmt"+ "os"+ "path/filepath"+ "strings"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/grade"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/report"+)++// cmdRegrade runs the offline graders again over a run directory's stored+// records (run.json and seed.json next to each transcript) with the cases+// as they are now, keeps the verdicts that needed the live instance (state,+// hidden tests, judge, setup verify), and recomputes each run's status. It+// is how a grader fix is applied to runs already paid for.+func cmdRegrade(args []string) error {+ fs := flag.NewFlagSet("regrade", flag.ExitOnError)+ out := fs.String("out", "", "write results.json and summary.md here (default: the run directory itself)")+ pos := parseInterleaved(fs, args)+ if len(pos) != 1 {+ return errors.New("regrade needs exactly one run directory")+ }+ ws, err := load()+ if err != nil {+ return err+ }+ dir := pos[0]+ res, err := report.Load(dir)+ if err != nil {+ return err+ }+ byID := map[string]*catalog.Case{}+ for _, c := range ws.cases {+ byID[c.ID] = c+ }+ type change struct {+ key, from, to string+ was, now []string+ }+ var changes []change+ skipped := 0+ for _, r := range res.Runs {+ switch r.Status {+ case "infra_error", "cost_ceiling":+ continue+ }+ c := byID[r.Case]+ if c == nil || r.Transcript == "" {+ skipped+++ continue+ }+ runDir := filepath.Dir(filepath.Join(dir, filepath.FromSlash(r.Transcript)))+ run, seed, err := loadRunRecord(runDir)+ if err != nil {+ fmt.Fprintf(os.Stderr, "note: %s: %v (kept as recorded)\n", r.Key(), err)+ skipped+++ continue+ }+ resolved, err := c.Resolve(map[string]any{"seed": seed["exposed"], "env": seed["env"]})+ if err != nil {+ return fmt.Errorf("%s: %w", r.Key(), err)+ }+ in := &grade.Input{Case: resolved, Arm: r.Arm, Run: run, EvalsRoot: ws.root}+ // A failed run keeps its workspace; the culprit-line mapping reads+ // the edited file from it rather than replaying the edits.+ if fi, err := os.Stat(filepath.Join(runDir, "work")); err == nil && fi.IsDir() {+ in.Workspace = filepath.Join(runDir, "work")+ }+ if m, ok := seed["exposed"].(map[string]any); ok {+ in.Seed = m+ }+ before := r.Status+ was := gatingFailures(r.Graders)+ r.Graders = mergeVerdicts(r.Graders, grade.Offline(in))+ if before != "budget_exhausted" {+ r.Status = "fail"+ if grade.Passed(r.Graders) {+ r.Status = "pass"+ }+ }+ if now := gatingFailures(r.Graders); before != r.Status || strings.Join(was, "|") != strings.Join(now, "|") {+ changes = append(changes, change{r.Key(), before, r.Status, was, now})+ }+ }+ res.Header.Regraded = time.Now().UTC().Format(time.RFC3339)+ target := *out+ if target == "" {+ target = dir+ }+ if err := os.MkdirAll(target, 0o755); err != nil {+ return err+ }+ if err := report.Save(target, res); err != nil {+ return err+ }+ if target == dir {+ for _, r := range res.Runs {+ if r.Transcript == "" {+ continue+ }+ raw, _ := json.MarshalIndent(r, "", " ")+ _ = os.WriteFile(filepath.Join(filepath.Dir(filepath.Join(dir, filepath.FromSlash(r.Transcript))), "result.json"), raw, 0o644)+ }+ }+ flipped := 0+ for _, ch := range changes {+ if ch.from != ch.to {+ flipped+++ }+ }+ fmt.Printf("regraded %d runs (%d without a record, kept as recorded): %d changed a verdict, %d changed status\n", len(res.Runs), skipped, len(changes), flipped)+ for _, ch := range changes {+ fmt.Printf(" %s: %s -> %s\n", ch.key, ch.from, ch.to)+ if len(ch.was) > 0 {+ fmt.Printf(" was: %s\n", strings.Join(ch.was, "; "))+ }+ if len(ch.now) > 0 {+ fmt.Printf(" now: %s\n", strings.Join(ch.now, "; "))+ }+ }+ return nil+}++// loadRunRecord reads a run's record and seed. A run directory without a+// run.json (the process died before it was written) is rebuilt from its+// transcript, which carries the same calls, prompt and submission.+func loadRunRecord(runDir string) (*record.Run, map[string]any, error) {+ var run record.Run+ raw, err := os.ReadFile(filepath.Join(runDir, "run.json"))+ switch {+ case err == nil:+ if err := json.Unmarshal(raw, &run); err != nil {+ return nil, nil, fmt.Errorf("run.json: %w", err)+ }+ case errors.Is(err, os.ErrNotExist):+ r, terr := runFromTranscript(filepath.Join(runDir, "transcript.jsonl"))+ if terr != nil {+ return nil, nil, fmt.Errorf("%w; transcript: %v", err, terr)+ }+ run = *r+ default:+ return nil, nil, err+ }+ raw, err = os.ReadFile(filepath.Join(runDir, "seed.json"))+ if err != nil {+ return nil, nil, err+ }+ var seed map[string]any+ if err := json.Unmarshal(raw, &seed); err != nil {+ return nil, nil, fmt.Errorf("seed.json: %w", err)+ }+ return &run, seed, nil+}++// runFromTranscript rebuilds a run record from transcript.jsonl: the start+// event's prompt, every tool event's call, the accepted submit call's+// arguments, the last response text and the stop event.+func runFromTranscript(path string) (*record.Run, error) {+ raw, err := os.ReadFile(path)+ if err != nil {+ return nil, err+ }+ run := &record.Run{}+ for _, line := range strings.Split(string(raw), "\n") {+ if strings.TrimSpace(line) == "" {+ continue+ }+ var ev struct {+ Type string `json:"type"`+ Prompt string `json:"prompt"`+ Call *record.Call `json:"call"`+ Content json.RawMessage `json:"content"`+ Reason string `json:"reason"`+ Detail string `json:"detail"`+ Turns int `json:"turns"`+ Usage *record.Usage `json:"usage"`+ }+ if err := json.Unmarshal([]byte(line), &ev); err != nil {+ return nil, err+ }+ switch ev.Type {+ case "start":+ run.Prompt = ev.Prompt+ case "tool":+ if ev.Call == nil {+ continue+ }+ run.Calls = append(run.Calls, ev.Call)+ if strings.HasPrefix(ev.Call.Tool, "submit") && ev.Call.OK && strings.HasPrefix(ev.Call.Output, "Submission received") {+ var sub map[string]any+ if json.Unmarshal(ev.Call.Args, &sub) == nil {+ run.Submission, run.SubmitTool, run.SubmitCall = sub, ev.Call.Tool, ev.Call.ID+ }+ }+ case "response":+ var text string+ if json.Unmarshal(ev.Content, &text) == nil && text != "" {+ run.FinalText = text+ }+ case "stop":+ run.StopReason, run.StopDetail, run.Turns = ev.Reason, ev.Detail, ev.Turns+ if ev.Usage != nil {+ run.Usage = *ev.Usage+ }+ }+ }+ if run.StopReason == "" {+ return nil, errors.New("no stop event")+ }+ return run, nil+}++// mergeVerdicts replaces the recorded verdicts the fresh ones cover, in+// place, and keeps the rest (state, hidden tests, judge, setup verify).+func mergeVerdicts(recorded, fresh []*grade.Verdict) []*grade.Verdict {+ out := make([]*grade.Verdict, 0, len(recorded)+len(fresh))+ used := map[string]bool{}+ for _, v := range recorded {+ replaced := false+ for _, f := range fresh {+ if f.Grader == v.Grader {+ out = append(out, f)+ used[f.Grader] = true+ replaced = true+ break+ }+ }+ if !replaced {+ out = append(out, v)+ }+ }+ for _, f := range fresh {+ if !used[f.Grader] {+ out = append(out, f)+ }+ }+ return out+}++func gatingFailures(vs []*grade.Verdict) []string {+ var out []string+ for _, v := range vs {+ if v.Gating && !v.Skipped && !v.Pass {+ for _, f := range v.Failures {+ out = append(out, v.Grader+": "+f)+ }+ }+ }+ return out+}diff --git a/cli/test/evals/internal/grade/answer.go b/cli/test/evals/internal/grade/answer.goindex 91d02dc4..89ca45cb 100644--- a/cli/test/evals/internal/grade/answer.go+++ b/cli/test/evals/internal/grade/answer.go@@ -55,7 +55,7 @@ func Answer(in *Input) *Verdict {gradeOnset(v, sub["onset"], exp.Onset)}case "debug":- answerDebug(v, exp, sub)+ answerDebug(v, in, exp, sub)}if len(exp.MentionsAll) > 0 {raw, _ := json.Marshal(sub)diff --git a/cli/test/evals/internal/grade/answer_debug.go b/cli/test/evals/internal/grade/answer_debug.goindex 561ff8a6..f70894ca 100644--- a/cli/test/evals/internal/grade/answer_debug.go+++ b/cli/test/evals/internal/grade/answer_debug.go@@ -1,6 +1,7 @@package gradeimport (+ "encoding/json""path/filepath""regexp""strings"@@ -9,13 +10,29 @@ import ()// answerDebug grades a submit_debug_diagnosis payload.-func answerDebug(v *Verdict, exp *catalog.Answer, sub map[string]any) {+//+// issue_hash names the group the diagnosis is about. An empty field is+// accepted when the expected hash is in the prompt or anywhere in the+// answer: the model worked on that group and left a bookkeeping field+// blank. A different hash is a different group, and fails.+//+// culprit.line is read against the file before the agent's edit. A line+// that is off by more than the tolerance is mapped back through the edit+// (lineBeforeEdit), so a model that inserted a guard above the faulty+// statement and reported where it now sits is not marked wrong.+func answerDebug(v *Verdict, in *Input, exp *catalog.Answer, sub map[string]any) {str := func(m map[string]any, k string) string { s, _ := m[k].(string); return strings.TrimSpace(s) }if exp.Category != "" && str(sub, "category") != exp.Category {v.fail("category %q, want %q", str(sub, "category"), exp.Category)}- if exp.IssueHash != "" && !strings.EqualFold(str(sub, "issue_hash"), exp.IssueHash) {- v.fail("issue_hash %q, want %q", str(sub, "issue_hash"), exp.IssueHash)+ if exp.IssueHash != "" {+ switch got := str(sub, "issue_hash"); {+ case strings.EqualFold(got, exp.IssueHash):+ case got == "" && hashNamed(in, exp.IssueHash):+ v.note("issue_hash left empty; %s is named in the prompt or the answer", exp.IssueHash)+ default:+ v.fail("issue_hash %q, want %q", got, exp.IssueHash)+ }}if exp.OccurrenceID != "" && !strings.EqualFold(str(sub, "occurrence_id"), exp.OccurrenceID) {v.fail("occurrence_id %q, want %q", str(sub, "occurrence_id"), exp.OccurrenceID)@@ -37,8 +54,13 @@ func answerDebug(v *Verdict, exp *catalog.Answer, sub map[string]any) {tol = 3}line, _ := toFloat(c["line"])- if d := int(line) - exp.Culprit.Line; d < -tol || d > tol {- v.fail("culprit.line %d, want %d±%d", int(line), exp.Culprit.Line, tol)+ got := int(line)+ if d := got - exp.Culprit.Line; d < -tol || d > tol {+ if before, ok := lineBeforeEdit(in, exp.Culprit.File, got); ok && before-exp.Culprit.Line >= -tol && before-exp.Culprit.Line <= tol {+ v.note("culprit.line %d is in the edited file; before the edit it was line %d", got, before)+ } else {+ v.fail("culprit.line %d, want %d±%d", got, exp.Culprit.Line, tol)+ }}}}@@ -50,6 +72,17 @@ func answerDebug(v *Verdict, exp *catalog.Answer, sub map[string]any) {}}+// hashNamed reports whether the expected hash appears in the prompt, the+// submission or the final text.+func hashNamed(in *Input, hash string) bool {+ if in == nil || in.Run == nil {+ return false+ }+ raw, _ := json.Marshal(in.Run.Submission)+ text := strings.ToLower(in.Run.Prompt + "\n" + string(raw) + "\n" + in.Run.FinalText)+ return strings.Contains(text, strings.ToLower(hash))+}+// samePath compares a cited file with the expected one, accepting a path// prefix, ./, backslashes, backticks and a trailing :line(:col).func samePath(got, want string) bool {diff --git a/cli/test/evals/internal/grade/grade.go b/cli/test/evals/internal/grade/grade.goindex 5f321c65..20d7cea5 100644--- a/cli/test/evals/internal/grade/grade.go+++ b/cli/test/evals/internal/grade/grade.go@@ -106,6 +106,27 @@ func All(ctx context.Context, in *Input) []*Verdict {return out}+// Offline runs the graders that need only the run record: answer,+// trajectory, efficiency, grounding and report. The state, hidden-test and+// judge verdicts need the run's instance, workspace or a model, so a regrade+// keeps the recorded ones.+func Offline(in *Input) []*Verdict {+ var out []*Verdict+ if in.Case.Expect.Answer != nil {+ out = append(out, Answer(in))+ }+ if in.Case.Expect.Calls != nil {+ out = append(out, Trajectory(in.Run, in.Case.Expect.Calls), Efficiency(in.Run, in.Case.Expect.Calls))+ }+ if in.Case.Suite == "perf" || in.Case.Suite == "debug" || len(in.Case.Expect.Grounding) > 0 {+ out = append(out, Grounding(in))+ }+ if in.Case.Expect.Report != nil {+ out = append(out, Report(in))+ }+ return out+}+// Passed reports whether every gating grader passed.func Passed(vs []*Verdict) bool {for _, v := range vs {diff --git a/cli/test/evals/internal/grade/grade_test.go b/cli/test/evals/internal/grade/grade_test.goindex 38a9e11f..e5e32711 100644--- a/cli/test/evals/internal/grade/grade_test.go+++ b/cli/test/evals/internal/grade/grade_test.go@@ -238,14 +238,26 @@ func TestGrounding(t *testing.T) {if v := Grounding(perfInput(sub("c01", ""), jqFailed("c01", out+"\njq: error: Cannot iterate over null", out))); !v.Pass {t.Fatalf("a trailing jq error failed a cited call: %s", failures(v))}+ // A call cut at the output cap is evidence for the part the model saw.+ cut := call("c01", true, out+"\n[output truncated: 90000 more characters.]", "traceway exceptions show abc")+ cut.Truncated = true+ if v := Grounding(perfInput(sub("c01", ""), cut)); !v.Pass {+ t.Fatalf("a truncated call with the key in view failed: %s", failures(v))+ } else if len(v.Notes) == 0 {+ t.Fatal("a truncated citation should be noted")+ }+ // ... and for nothing past the cut, even when the raw output had it.+ hidden := call("c01", true, "{\"group\":{}}\n[output truncated: 90000 more characters.]", "traceway exceptions show abc")+ hidden.Truncated = true+ hidden.Ops[0].Output = outtests := map[string]struct {in *Inputwant string}{- "unknown call": {perfInput(sub("c07", ""), c1), "not a call"},- "failed call": {perfInput(sub("c01", ""), call("c01", false, out, "traceway endpoints show x")), "which failed"},- "truncated call": {perfInput(sub("c01", ""), &record.Call{ID: "c01", OK: true, Truncated: true, Output: out}), "truncated"},- "invented id": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1), "no tool returned"},+ "unknown call": {perfInput(sub("c07", ""), c1), "not a call"},+ "failed call": {perfInput(sub("c01", ""), call("c01", false, out, "traceway endpoints show x")), "which failed"},+ "truncated call, key past the cut": {perfInput(sub("c01", ""), hidden), "contains"},+ "invented id": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1), "no tool returned"},// The submit call itself is recorded with the submission as its// arguments; that must not vouch for the ids in it."invented id, submit recorded": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1,diff --git a/cli/test/evals/internal/grade/grounding.go b/cli/test/evals/internal/grade/grounding.goindex e49df94b..da71084a 100644--- a/cli/test/evals/internal/grade/grounding.go+++ b/cli/test/evals/internal/grade/grounding.go@@ -13,11 +13,13 @@ import ()// Grounding checks that the evidence is real:-// - every evidence.call_id names a call that exists, succeeded and was-// not truncated (a shell call succeeded when one of its operations did,-// even if a later command in it, such as jq, failed);+// - every evidence.call_id names a call that exists and succeeded (a shell+// call succeeded when one of its operations did, even if a later command+// in it, such as jq, failed);// - every grounding key appears in the output of at least one cited call-// (what the model saw, or the raw traceway output inside that call);+// (what the model saw, or the raw traceway output inside that call). A+// call whose output was cut at the cap is evidence for the part the+// model saw, and for nothing past the cut;// - every UUID, 32-hex trace id or 16-hex hash in the answer appears in// some tool output or the prompt: no invented ids.func Grounding(in *Input) *Verdict {@@ -29,6 +31,7 @@ func Grounding(in *Input) *Verdict {return v}var cited []*record.Call+ visibleOnly := map[*record.Call]bool{}if ev, ok := sub["evidence"].([]any); ok {for i, e := range ev {m, _ := e.(map[string]any)@@ -42,7 +45,9 @@ func Grounding(in *Input) *Verdict {case !anyOpOK(c) || (len(c.Ops) == 0 && !c.OK && !grepFoundNothing(c)):v.fail("evidence[%d] cites %s, which failed", i, id)case c.Truncated:- v.fail("evidence[%d] cites %s, whose output was truncated", i, id)+ v.note("evidence[%d] cites %s, whose output was truncated; only the part the model saw counts", i, id)+ visibleOnly[c] = true+ cited = append(cited, c)default:cited = append(cited, c)}@@ -53,7 +58,7 @@ func Grounding(in *Input) *Verdict {for _, key := range in.Case.Expect.Grounding {found := falsefor _, c := range cited {- if callContains(c, key) {+ if visibleOnly[c] && strings.Contains(unescapeJSON(c.Output), key) || !visibleOnly[c] && callContains(c, key) {found = truebreak}diff --git a/cli/test/evals/internal/grade/linemap.go b/cli/test/evals/internal/grade/linemap.gonew file mode 100644index 00000000..d654a152--- /dev/null+++ b/cli/test/evals/internal/grade/linemap.go@@ -0,0 +1,185 @@+package grade++import (+ "encoding/json"+ "os"+ "path/filepath"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// lineBeforeEdit maps a line of the agent's edited copy of rel back to the+// line of the fixture file it came from. The fixture is the case's repo+// under testdata/repos; the edited copy is read from the workspace when+// there is one, else rebuilt from the recorded edit_file and write_file+// calls. It reports false when either side is unavailable, when a shell+// redirection wrote the file (its content was not recorded) or when the+// file was not edited at all.+func lineBeforeEdit(in *Input, rel string, line int) (int, bool) {+ if in == nil || in.Case == nil || in.EvalsRoot == "" || in.Case.Workspace.Repo == "" || line < 1 {+ return 0, false+ }+ raw, err := os.ReadFile(filepath.Join(in.EvalsRoot, "testdata", "repos", in.Case.Workspace.Repo, filepath.FromSlash(rel)))+ if err != nil {+ return 0, false+ }+ before := string(raw)+ var after string+ if in.Workspace != "" {+ now, err := os.ReadFile(filepath.Join(in.Workspace, filepath.FromSlash(rel)))+ if err != nil {+ return 0, false+ }+ after = string(now)+ } else {+ var ok bool+ if after, ok = replayEdits(in.Run, rel, before); !ok {+ return 0, false+ }+ }+ if after == before {+ return 0, false+ }+ return mapLineBack(before, after, line)+}++// replayEdits applies the run's successful edit_file and write_file calls+// on rel to base, in order, the way the tools did. A write through the shell+// (a redirection) has no recorded content, so the result is unknown.+func replayEdits(run *record.Run, rel, base string) (string, bool) {+ if run == nil {+ return "", false+ }+ text := base+ for _, c := range run.Calls {+ if c.Rejected {+ continue+ }+ for _, op := range c.Ops {+ if op.Op.Op != ops.OpEditFile && op.Op.Op != ops.OpWriteFile {+ continue+ }+ p, _ := op.Args["path"].(string)+ if !samePath(p, rel) {+ continue+ }+ if !op.OK {+ continue+ }+ switch c.Tool {+ case "edit_file":+ var in struct {+ Old string `json:"old_string"`+ New string `json:"new_string"`+ ReplaceAll bool `json:"replace_all"`+ }+ if json.Unmarshal(c.Args, &in) != nil || in.Old == "" || !strings.Contains(text, in.Old) {+ return "", false+ }+ if in.ReplaceAll {+ text = strings.ReplaceAll(text, in.Old, in.New)+ } else {+ text = strings.Replace(text, in.Old, in.New, 1)+ }+ case "write_file":+ var in struct {+ Content string `json:"content"`+ }+ if json.Unmarshal(c.Args, &in) != nil {+ return "", false+ }+ text = in.Content+ default:+ return "", false+ }+ }+ }+ return text, true+}++// mapLineBack maps a 1-based line of after to the line of before it+// corresponds to, through a longest-common-subsequence alignment of the two+// files' lines. A line that exists in both maps to its match. A line the+// edit introduced or changed maps to where it sits between the nearest+// matched lines around it, clamped to that gap, so an inserted guard above+// a statement maps onto the statement's old neighbourhood.+func mapLineBack(before, after string, line int) (int, bool) {+ a := strings.Split(before, "\n")+ b := strings.Split(after, "\n")+ if line < 1 || line > len(b) {+ return 0, false+ }+ const maxLines = 6000+ if len(a) > maxLines || len(b) > maxLines {+ return 0, false+ }+ // dp[i][j] is the LCS length of a[i:] and b[j:].+ dp := make([][]int32, len(a)+1)+ for i := range dp {+ dp[i] = make([]int32, len(b)+1)+ }+ for i := len(a) - 1; i >= 0; i-- {+ for j := len(b) - 1; j >= 0; j-- {+ if a[i] == b[j] {+ dp[i][j] = dp[i+1][j+1] + 1+ } else if dp[i+1][j] >= dp[i][j+1] {+ dp[i][j] = dp[i+1][j]+ } else {+ dp[i][j] = dp[i][j+1]+ }+ }+ }+ // toBefore[j] is the 1-based before line matched to after line j+1, or 0.+ toBefore := make([]int, len(b))+ for i, j := 0, 0; i < len(a) && j < len(b); {+ switch {+ case a[i] == b[j]:+ toBefore[j] = i + 1+ i+++ j+++ case dp[i+1][j] >= dp[i][j+1]:+ i+++ default:+ j+++ }+ }+ if m := toBefore[line-1]; m > 0 {+ return m, true+ }+ prevJ, prevI := 0, 0+ for j := line - 2; j >= 0; j-- {+ if toBefore[j] > 0 {+ prevJ, prevI = j+1, toBefore[j]+ break+ }+ }+ nextJ, nextI := len(b)+1, len(a)+1+ for j := line; j < len(b); j++ {+ if toBefore[j] > 0 {+ nextJ, nextI = j+1, toBefore[j]+ break+ }+ }+ // The unmatched after lines between prevJ and nextJ replace the before+ // lines between prevI and nextI; place the line proportionally in that+ // gap, and at least one past the previous match.+ gapB := nextJ - prevJ+ gapA := nextI - prevI+ off := (line - prevJ) * gapA / gapB+ if off < 1 {+ off = 1+ }+ if off > gapA-1 && gapA > 1 {+ off = gapA - 1+ }+ got := prevI + off+ if got < 1 {+ got = 1+ }+ if got > len(a) {+ got = len(a)+ }+ return got, true+}diff --git a/cli/test/evals/internal/grade/linemap_test.go b/cli/test/evals/internal/grade/linemap_test.gonew file mode 100644index 00000000..1a4a668a--- /dev/null+++ b/cli/test/evals/internal/grade/linemap_test.go@@ -0,0 +1,112 @@+package grade++import (+ "encoding/json"+ "os"+ "path/filepath"+ "strings"+ "testing"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++const renderBefore = `package invoices++func formatAddress(a *Address) string {+ s := a.Line1+ if a.Line2 != "" {+ s += "\n" + a.Line2+ }+ return s+}+`++func TestMapLineBack(t *testing.T) {+ guard := strings.Replace(renderBefore, "\ts := a.Line1", "\tif a == nil {\n\t\treturn \"\"\n\t}\n\ts := a.Line1", 1)+ comment := strings.Replace(renderBefore, "func formatAddress", "// formatAddress renders a billing address.\n// A nil address renders as \"\".\nfunc formatAddress", 1)+ changed := strings.Replace(renderBefore, "\ts := a.Line1", "\ts := safeLine1(a)", 1)+ deleted := strings.Replace(renderBefore, "package invoices\n\n", "", 1)+ for name, tc := range map[string]struct {+ after string+ line int+ want int+ }{+ "statement below an inserted guard": {guard, 7, 4},+ "the inserted guard itself": {guard, 4, 4},+ "statement below an inserted comment": {comment, 6, 4},+ "the changed statement": {changed, 4, 4},+ "statement above the edit": {guard, 3, 3},+ "statement below a deletion": {deleted, 2, 4},+ "end of file": {guard, 12, 9},+ } {+ got, ok := mapLineBack(renderBefore, tc.after, tc.line)+ if !ok || got != tc.want {+ t.Errorf("%s: mapLineBack(line %d) = %d, %v, want %d", name, tc.line, got, ok, tc.want)+ }+ }+ if _, ok := mapLineBack(renderBefore, guard, 40); ok {+ t.Error("a line past the end of the edited file mapped")+ }+}++func TestCulpritLineThroughTheEdit(t *testing.T) {+ root := t.TempDir()+ rel := "internal/invoices/render.go"+ path := filepath.Join(root, "testdata", "repos", "billing-api-go", rel)+ if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {+ t.Fatal(err)+ }+ if err := os.WriteFile(path, []byte(renderBefore), 0o644); err != nil {+ t.Fatal(err)+ }+ // The guard is four lines, so the statement moves past the ±3 tolerance.+ editArgs, _ := json.Marshal(map[string]any{"path": rel, "old_string": "\ts := a.Line1", "new_string": "\tif a == nil {\n\t\t// Digital-only customers never enter one.\n\t\treturn \"\"\n\t}\n\ts := a.Line1"})+ edit := &record.Call{ID: "c03", Tool: "edit_file", OK: true, Args: editArgs,+ Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpEditFile, Args: map[string]any{"path": rel}, Write: true}, OK: true}}}+ exp := &catalog.Answer{IssueHash: "e4c2373afd52ef13", Category: "null_reference", Culprit: &catalog.Culprit{File: rel, Symbol: "formatAddress", Line: 4}}+ input := func(calls []*record.Call, sub map[string]any, prompt string) *Input {+ return &Input{EvalsRoot: root, Case: &catalog.Case{ID: "debug-x", Suite: "debug", Workspace: catalog.Workspace{Repo: "billing-api-go"}, Expect: catalog.Expect{Answer: exp}},+ Run: &record.Run{Prompt: prompt, Submission: sub, Calls: calls}}+ }+ sub := func(line int, hash string) map[string]any {+ return map[string]any{"issue_hash": hash, "category": "null_reference", "culprit": map[string]any{"file": rel, "symbol": "formatAddress", "line": line},+ "evidence": []any{map[string]any{"call_id": "c01", "claim": "nil pointer in formatAddress"}}, "report": "formatAddress dereferences a nil address"}+ }+ // Line 8 is the statement's line after the guard went in above it.+ if v := Answer(input([]*record.Call{edit}, sub(8, "e4c2373afd52ef13"), "/traceway debug e4c2373afd52ef13")); !v.Pass {+ t.Fatalf("the line in the edited file failed: %s", failures(v))+ } else if len(v.Notes) == 0 {+ t.Fatal("mapping through the edit should be noted")+ }+ // The same line with no recorded edit is just wrong.+ if v := Answer(input(nil, sub(8, "e4c2373afd52ef13"), "")); v.Pass {+ t.Fatal("line 8 passed without an edit to explain it")+ }+ // A shell redirection wrote the file: its content is unknown, so the line is wrong.+ shellWrite := &record.Call{ID: "c03", Tool: "shell", OK: true, Args: json.RawMessage(`{"command":"sed -i x render.go"}`),+ Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpWriteFile, Args: map[string]any{"path": rel}}, OK: true}}}+ if v := Answer(input([]*record.Call{shellWrite}, sub(8, "e4c2373afd52ef13"), "")); v.Pass {+ t.Fatal("line 8 passed after an unrecorded shell write")+ }+ // A line far from both files' statement stays wrong.+ if v := Answer(input([]*record.Call{edit}, sub(30, "e4c2373afd52ef13"), "")); v.Pass {+ t.Fatal("line 30 passed")+ }+ // issue_hash: empty is fine when the prompt or the answer names the hash.+ if v := Answer(input([]*record.Call{edit}, sub(4, ""), "/traceway debug e4c2373afd52ef13")); !v.Pass {+ t.Fatalf("an empty issue_hash with the hash in the prompt failed: %s", failures(v))+ }+ named := sub(4, "")+ named["report"] = "Issue e4c2373afd52ef13: formatAddress dereferences a nil address"+ if v := Answer(input([]*record.Call{edit}, named, "the invoice page is broken")); !v.Pass {+ t.Fatalf("an empty issue_hash with the hash in the report failed: %s", failures(v))+ }+ if v := Answer(input([]*record.Call{edit}, sub(4, ""), "the invoice page is broken")); v.Pass {+ t.Fatal("an empty issue_hash passed with the hash nowhere")+ }+ if v := Answer(input([]*record.Call{edit}, sub(4, "aac163b08f71ab3f"), "/traceway debug e4c2373afd52ef13")); v.Pass {+ t.Fatal("another group's hash passed")+ }+}diff --git a/cli/test/evals/internal/report/results.go b/cli/test/evals/internal/report/results.goindex 3d372be4..4840a1e3 100644--- a/cli/test/evals/internal/report/results.go+++ b/cli/test/evals/internal/report/results.go@@ -41,6 +41,9 @@ type Header struct {ReplacedSpendUSD float64 `json:"replacedSpendUsd,omitempty"`Oracle bool `json:"oracle,omitempty"`CI map[string]string `json:"ci,omitempty"`+ // Regraded is set when the offline graders were run again over the+ // stored run records (twevals regrade).+ Regraded string `json:"regraded,omitempty"`}// Results is results.json.@@ -155,9 +158,11 @@ func Load(dir string) (*Results, error) {// Merge combines shard results under one header. A run that appears in// two parts (a merged directory passed with its own shards, or the same shard-// twice) is an error, and parts from different commits, models or binaries-// are reported in the returned notes.-func Merge(parts []*Results) (*Results, []string, error) {+// twice) is an error, unless replace is set: then the later part's run+// replaces the earlier one (a rerun of the same case, arm and model), and+// each replacement is listed in the returned notes, along with parts from+// different commits, models or binaries.+func Merge(parts []*Results, replace bool) (*Results, []string, error) {out := &Results{}total, replaced := 0.0, 0.0from := map[string]int{}@@ -180,11 +185,20 @@ func Merge(parts []*Results) (*Results, []string, error) {}for _, r := range p.Runs {if j, ok := from[r.Key()]; ok {- return nil, nil, fmt.Errorf("run %s is in part %d and part %d: pass each shard once, and not a merged directory together with its shards", r.Key(), j+1, i+1)+ if !replace {+ return nil, nil, fmt.Errorf("run %s is in part %d and part %d: pass each shard once, and not a merged directory together with its shards (or pass --replace for a rerun)", r.Key(), j+1, i+1)+ }+ notes = append(notes, fmt.Sprintf("run %s from part %d (%s) replaces the one in part %d", r.Key(), i+1, p.Header.RunID, j+1))+ for k, prev := range out.Runs {+ if prev.Key() == r.Key() {+ out.Runs = append(out.Runs[:k], out.Runs[k+1:]...)+ break+ }+ }}from[r.Key()] = i+ out.Runs = append(out.Runs, r)}- out.Runs = append(out.Runs, p.Runs...)}out.Header.MaxCostUSD = totalout.Header.ReplacedSpendUSD = replaceddiff --git a/cli/test/evals/internal/tools/submit.go b/cli/test/evals/internal/tools/submit.goindex 39092d20..7f8ffead 100644--- a/cli/test/evals/internal/tools/submit.go+++ b/cli/test/evals/internal/tools/submit.go@@ -72,13 +72,13 @@ func (t *Submit) Spec() Spec {case SubmitDebug:return Spec{Name: SubmitDebug, Description: "Submit your final debugging diagnosis. This ends the task. Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",Parameters: obj(map[string]any{- "issue_hash": strProp("The exception group hash (16 hex), if the bug has one"),+ "issue_hash": strProp("The exception group hash (16 hex) of the issue you diagnosed. Set it whenever the issue has one: it is in the task text, in the /issues/<hash> URL, or in the exceptions list and show output. Leave it empty only when there is no exception group."),"occurrence_id": strProp("The occurrence you anchored on, if any"),"category": map[string]any{"type": "string", "enum": DebugCategories},"culprit": obj(map[string]any{"file": strProp("Workspace-relative file"),"symbol": strProp("Function or method"),- "line": intProp("Line number"),+ "line": intProp("Line number of the faulty statement, in the file as you found it; a line in the edited file is accepted too"),}, "file"),"evidence": evidenceSchema(),"fix_summary": strProp("What you changed or propose to change"),diff --git a/docs/pages/learn/agent-benchmarks.mdx b/docs/pages/learn/agent-benchmarks.mdxindex 3b6f694f..7911600d 100644--- a/docs/pages/learn/agent-benchmarks.mdx+++ b/docs/pages/learn/agent-benchmarks.mdx@@ -169,8 +169,8 @@ Each case runs on several setups, so the skill and the two surfaces can be compa| Grader | Gates | Checks ||---|---|---|-| `answer` | yes | The diagnosis: the right checklist row and not the distractor, the right endpoint or service, the onset within tolerance. For debug cases, the category, culprit file, symbol and line |-| `grounding` | yes | Every cited call exists, succeeded, was not truncated, and its output contains the deciding signal. An id the model invented fails the run |+| `answer` | yes | The diagnosis: the right checklist row and not the distractor, the right endpoint or service, the onset within tolerance. For debug cases, the category, culprit file, symbol and line. The line is read against the file before the fix, and a line reported in the edited file is mapped back through the edit. An empty issue hash is accepted when the prompt or the answer names the group; a different hash is a different group and fails |+| `grounding` | yes | Every cited call exists and succeeded, and the output of a cited call contains the deciding signal. A call whose output was cut at the tool's size cap is evidence for the part the model saw and for nothing past the cut. An id the model invented fails the run || `trajectory` | yes | Required, forbidden and ordered operations, such as opening one slow request, never writing on a read-only task, or asking before an ambiguous bulk write. CLI and MCP calls map to the same operations, and many cases accept more than one route to the same evidence || `report` | yes | Statements the report must make, such as naming the figure that decides the case || `state` | yes | After a write or setup run, the backend and the workspace are exactly as expected: the right two issues archived, not three |@@ -194,7 +194,7 @@ The benchmarks page also shows **right answer and evidence**: runs whose answer,- **Every case is passable.** Each case has an oracle: a reference trajectory played by a fake model. It must pass on every setup the case lists, for $0, before the case is added, and `just eval-harness` replays every oracle after a harness change.- **A sandbox, not a host shell.** The agent's shell is an interpreter with the setup's `traceway` binary, `jq`, `awk`, read-only text utilities confined to the workspace and read-only `git`. Tool processes get a scrubbed environment, so no key or secret reaches them.- **The same conditions for every model.** `models.yaml` pins exact OpenRouter model ids, never a floating alias, with the same reasoning effort (medium), the same per-turn token limit (16k), the same prompts, tools and budgets. Requests go to the model's own vendor first, and every run records which provider served it.-- **Compare like with like.** Results are only compared between runs on the same commit and the same graders. When a grader changes, saved transcripts are re-scored instead of comparing old numbers with new ones.+- **Compare like with like.** Results are only compared between runs on the same commit and the same graders. When a grader changes, saved transcripts are re-scored instead of comparing old numbers with new ones: `twevals regrade <run-dir>` runs the answer, trajectory, grounding and report graders again over the stored records, keeps the hidden-test, state and judge verdicts, and recomputes each run's status. A change to a skill cannot be re-scored, because it changes what the model does; the affected setups are run again and replace the earlier runs (`twevals report --merge --replace`), and the page names the batch they came from.## Spending controlsdiff --git a/skills/traceway/SKILL.md b/skills/traceway/SKILL.mdindex af945d2d..7cc0c6bb 100644--- a/skills/traceway/SKILL.md+++ b/skills/traceway/SKILL.md@@ -213,10 +213,10 @@ traceway exceptions list --since 7d --search "checkout" --output json \### 2. Drill into the issue```bash-traceway exceptions show <hash>+traceway exceptions show <hash> --page-size 5```-This is the high-value call: full stack trace, occurrence list with `recordedAt`, `attributes` (user IDs, app versions, request context), and optional `traceId` / `spanId` / `sessionId` per occurrence. `traceId` is the OpenTelemetry trace id (32 hex characters), the same id the request's logs and spans carry. The response also has `relatedEntity`: the endpoint, task or AI trace the newest occurrence happened in, with the `id` and `recordedAt` that `endpoints show` / `tasks show` / `ai-traces show` take. `firstSeen` correlates with deploys: a group that first appeared right after a release points at that release's diff. A bogus hash exits 5 with `not_found`; fall back to search.+This is the high-value call: full stack trace, occurrence list with `recordedAt`, `attributes` (user IDs, app versions, request context), and optional `traceId` / `spanId` / `sessionId` per occurrence. Occurrences come newest first, and each carries its own full stack trace, so the default page of 50 runs to well over 100k characters and gets cut off by any tool output limit; `--page-size 5` keeps the group header plus the five newest occurrences, which is what a first read needs. Keep the default page only when you pipe the JSON through `jq` to reduce it, as below. `traceId` is the OpenTelemetry trace id (32 hex characters), the same id the request's logs and spans carry. The response also has `relatedEntity`: the endpoint, task or AI trace the newest occurrence happened in, with the `id` and `recordedAt` that `endpoints show` / `tasks show` / `ai-traces show` take. `firstSeen` correlates with deploys: a group that first appeared right after a release points at that release's diff. A bogus hash exits 5 with `not_found`; fall back to search.**When the user gave an issue URL (or hash), fix the LAST occurrence — not "the group".** A single hash can bundle *several distinct errors*: the hash is computed from a normalized stack trace with the message stripped, so two unrelated failures that share their top frames (e.g. both captured at the same middleware/recovery frame) collapse into one group. The group's representative stack trace and `firstSeen` may belong to a different, now-dormant error than the one the user is looking at. Anchor on the most recent occurrence and fix that specific failure path:diff --git a/website/app/benchmarks/page.tsx b/website/app/benchmarks/page.tsxindex dcd8a251..1f489805 100644--- a/website/app/benchmarks/page.tsx+++ b/website/app/benchmarks/page.tsx@@ -41,6 +41,17 @@ const setupLabels = Object.fromEntries(setups.map((s) => [s.arm, s.label]));const suiteLabel = Object.fromEntries(suites.map((s) => [s.id, s.label]));const groupById = Object.fromEntries(groups.map((g) => [g.id, g]));const capitalize = (s: string) => s[0].toUpperCase() + s.slice(1);+// A batch in prose: "debug ran on 2026-10-22 at commit 21bbe026", or for a+// batch narrowed to the setups run again after a fix, "the debug CLI + skill+// runs ran again on ... at commit ...".+const partText = (p: (typeof run.parts)[number], withCommit: boolean) => {+ if (!p.arms) return `${joinList(p.suites)} ${withCommit ? "ran " : ""}on ${p.date}${withCommit ? ` at commit ${p.commit}` : ""}`;+ const labels = p.arms.map((a) => setupLabels[a] ?? a);+ return `the ${joinList(p.suites)} ${joinList(labels)} runs ${withCommit ? "ran again" : "again"} on ${p.date}${withCommit ? `, with the skill fix on top of commit ${p.commit}` : ""}`;+};+const regradedText = run.regraded+ ? ` Every run was scored by the graders as they stood on ${run.regraded}: after a grader fix the saved transcripts are re-scored, so no number here compares two graders.`+ : "";const joinList = (items: string[]) =>items.length < 2? items.join("")@@ -835,7 +846,7 @@ export default function BenchmarksPage() {Runs went three at a time on one machine{run.parts.length > 1? `, in ${numberWord(run.parts.length)} batches (${run.parts- .map((p) => `${joinList(p.suites)} on ${p.date}`)+ .map((p) => partText(p, false)).join("; ")})`: ""}, so wall time includes some contention.{" "}@@ -881,14 +892,9 @@ export default function BenchmarksPage() {}description={`Each task is a fresh Traceway backend seeded with weeks of telemetry from a system broken in one way, plus a planted red herring. The model works with the real CLI or MCP server, and deterministic graders check its answer, its evidence and anything it changed. ${run.parts.length > 1- ? run.parts- .map(- (p) =>- `${capitalize(joinList(p.suites))} ran on ${p.date} at commit ${p.commit}`,- )- .join("; ")+ ? run.parts.map((p) => capitalize(partText(p, true))).join("; "): `All runs on ${run.date}, at commit ${run.commit}`- }, ${run.runsPerCase} run per case.`}+ }, ${run.runsPerCase} run per case.${regradedText}`}cta={<div className="flex flex-wrap items-center gap-x-6 gap-y-3"><Linkdiff --git a/website/content/benchmarks/2026-10-21.md b/website/content/benchmarks/2026-10-21.mdindex f9def456..9ffaedb3 100644--- a/website/content/benchmarks/2026-10-21.md+++ b/website/content/benchmarks/2026-10-21.md@@ -9,6 +9,11 @@ parts:- suites: "debug"commit: "21bbe026"date: "2026-10-22"+ - suites: "debug"+ arms: "cli+skill"+ commit: "21bbe026"+ date: "2026-10-22"+regraded: "2026-10-22"cliVersion: "cli/v2.0.5"backendVersion: "backend/v2.0.5"harnessUrl: "https://github.com/tracewayapp/traceway/tree/main/cli/test/evals"@@ -69,7 +74,7 @@ models:# Traceway agent benchmarks: 2026-10-21-Three models, Claude Haiku 5.5, DeepSeek V4.1 Flash and Claude Opus 5.5, each ran the same 172 eval runs in two batches: perf and write on 2026-10-21 at commit `3b00b083`, debug on 2026-10-22 at commit `21bbe026`. Every run gets a fresh Traceway backend seeded with the telemetry of a system broken in one specific way, and deterministic code grades the result. [How we measure](https://docs.tracewayapp.com/learn/agent-benchmarks) · [the harness and the cases](https://github.com/tracewayapp/traceway/tree/main/cli/test/evals).+Three models, Claude Haiku 5.5, DeepSeek V4.1 Flash and Claude Opus 5.5, each ran the same 172 eval runs in three batches: perf and write on 2026-10-21 at commit `3b00b083`, debug on 2026-10-22 at commit `21bbe026`, the debug CLI + skill runs again on 2026-10-22, with the skill fix on top of commit `21bbe026`. Every run gets a fresh Traceway backend seeded with the telemetry of a system broken in one specific way, and deterministic code grades the result. Every run was scored by the graders as they stood on 2026-10-22: after a grader fix the saved transcripts are re-scored, so no number here compares two graders. [How we measure](https://docs.tracewayapp.com/learn/agent-benchmarks) · [the harness and the cases](https://github.com/tracewayapp/traceway/tree/main/cli/test/evals).This file is generated by `website/scripts/build-benchmarks-data.mjs` from the eval results and is the data behind https://tracewayapp.com/benchmarks. Change the generator, not this file.@@ -79,36 +84,36 @@ A run passes when its answer is right, its evidence is grounded and its writes a| Model | Runs | Passed | Pass rate | Right answer and evidence | Not graded | Agent cost | Cost per run ||---|---:|---:|---:|---:|---:|---:|---:|-| Claude Haiku 5.5 | 172 | 106 | 62% | 112 | 0 | $1.90 | $0.01103 |-| DeepSeek V4.1 Flash | 172 | 123 | 72% | 127 | 0 | $2.30 | $0.01339 |-| Claude Opus 5.5 | 172 | 146 | 85% | 152 | 0 | $51.34 | $0.29851 |+| Claude Haiku 5.5 | 172 | 117 | 68% | 126 | 0 | $1.90 | $0.01104 |+| DeepSeek V4.1 Flash | 172 | 144 | 84% | 149 | 0 | $2.17 | $0.01264 |+| Claude Opus 5.5 | 172 | 156 | 91% | 161 | 0 | $51.81 | $0.30121 |## Pass rate by suite and setup| Suite | Setup | Label | Description | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 ||---|---|---|---|---|---|---|| perf | cli+skill | CLI + skill | The traceway CLI plus the Traceway agent skill. | 28/35 (80%) | 27/35 (77%) | 33/35 (94%) |-| perf | cli-noskill | CLI, no skill | The same CLI with no skill: the control group. | 19/35 (54%) | 24/35 (69%) | 28/35 (80%) |-| perf | mcp | MCP server | The Traceway MCP server: the same knowledge, plus tools. | 24/35 (69%) | 30/35 (86%) | 31/35 (89%) |-| debug | cli+skill | CLI + skill | The traceway CLI plus the Traceway agent skill. | 3/13 (23%) | 5/13 (38%) | 12/13 (92%) |-| debug | cli-noskill | CLI, no skill | The same CLI with no skill: the control group. | 1/13 (8%) | 3/13 (23%) | 4/13 (31%) |-| debug | mcp | MCP server | The Traceway MCP server: the same knowledge, plus tools. | 9/13 (69%) | 9/13 (69%) | 11/13 (85%) |+| perf | cli-noskill | CLI, no skill | The same CLI with no skill: the control group. | 20/35 (57%) | 27/35 (77%) | 30/35 (86%) |+| perf | mcp | MCP server | The Traceway MCP server: the same knowledge, plus tools. | 27/35 (77%) | 31/35 (89%) | 32/35 (91%) |+| debug | cli+skill | CLI + skill | The traceway CLI plus the Traceway agent skill. | 7/13 (54%) | 11/13 (85%) | 11/13 (85%) |+| debug | cli-noskill | CLI, no skill | The same CLI with no skill: the control group. | 4/13 (31%) | 11/13 (85%) | 12/13 (92%) |+| debug | mcp | MCP server | The Traceway MCP server: the same knowledge, plus tools. | 9/13 (69%) | 12/13 (92%) | 11/13 (85%) || debug | cli-write+skill | Write CLI + skill | The traceway-write CLI plus the write skill. | 0/1 (0%) | 0/1 (0%) | 1/1 (100%) || debug | mcp-write | Write MCP server | The write MCP server. | 0/1 (0%) | 0/1 (0%) | 0/1 (0%) || write | cli-write+skill | Write CLI + skill | The traceway-write CLI plus the write skill. | 10/13 (77%) | 12/13 (92%) | 13/13 (100%) || write | mcp-write | Write MCP server | The write MCP server. | 12/13 (92%) | 13/13 (100%) | 13/13 (100%) |-| all | | All runs | | 106/172 (62%) | 123/172 (72%) | 146/172 (85%) |+| all | | All runs | | 117/172 (68%) | 144/172 (84%) | 156/172 (91%) |### Right answer and evidence by suite and setup| Suite | Setup | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 ||---|---|---|---|---|| perf | cli+skill | 29/35 | 27/35 | 34/35 |-| perf | cli-noskill | 22/35 | 26/35 | 30/35 |-| perf | mcp | 24/35 | 31/35 | 33/35 |-| debug | cli+skill | 3/13 | 5/13 | 13/13 |-| debug | cli-noskill | 2/13 | 3/13 | 4/13 |-| debug | mcp | 9/13 | 9/13 | 11/13 |+| perf | cli-noskill | 23/35 | 29/35 | 32/35 |+| perf | mcp | 27/35 | 32/35 | 34/35 |+| debug | cli+skill | 9/13 | 11/13 | 11/13 |+| debug | cli-noskill | 6/13 | 12/13 | 12/13 |+| debug | mcp | 9/13 | 12/13 | 11/13 || debug | cli-write+skill | 0/1 | 1/1 | 1/1 || debug | mcp-write | 0/1 | 0/1 | 0/1 || write | cli-write+skill | 11/13 | 12/13 | 13/13 |@@ -120,10 +125,10 @@ Setup A minus setup B, in percentage points, computed before rounding.| Comparison | Suite | Setup A | Setup B | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 ||---|---|---|---|---|---|---|-| Skill uplift | perf | cli+skill | cli-noskill | +26pp (80% vs 54%) | +9pp (77% vs 69%) | +14pp (94% vs 80%) |-| Skill uplift | debug | cli+skill | cli-noskill | +15pp (23% vs 8%) | +15pp (38% vs 23%) | +62pp (92% vs 31%) |-| CLI vs MCP | perf | cli+skill | mcp | +11pp (80% vs 69%) | -9pp (77% vs 86%) | +6pp (94% vs 89%) |-| CLI vs MCP | debug | cli+skill | mcp | -46pp (23% vs 69%) | -31pp (38% vs 69%) | +8pp (92% vs 85%) |+| Skill uplift | perf | cli+skill | cli-noskill | +23pp (80% vs 57%) | +0pp (77% vs 77%) | +9pp (94% vs 86%) |+| Skill uplift | debug | cli+skill | cli-noskill | +23pp (54% vs 31%) | +0pp (85% vs 85%) | -8pp (85% vs 92%) |+| CLI vs MCP | perf | cli+skill | mcp | +3pp (80% vs 77%) | -11pp (77% vs 89%) | +3pp (94% vs 91%) |+| CLI vs MCP | debug | cli+skill | mcp | -15pp (54% vs 69%) | -8pp (85% vs 92%) | +0pp (85% vs 85%) || CLI vs MCP | debug | cli-write+skill | mcp-write | +0pp (0% vs 0%) | +0pp (0% vs 0%) | +100pp (100% vs 0%) || CLI vs MCP | write | cli-write+skill | mcp-write | -15pp (77% vs 92%) | -8pp (92% vs 100%) | +0pp (100% vs 100%) |@@ -133,16 +138,16 @@ Passes over every setup a case ran on.| Group | Label | Code | Description | Cases | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 ||---|---|---|---|---|---|---|---|-| perf-app | App code | perf-app-* | Slow code in the request path: serialization, quadratic loops, large allocations, debug logging, work that belongs in the background. | 5 | 11/15 (73%) | 13/15 (87%) | 13/15 (87%) |-| perf-cache | Caching | perf-cache-* | Cold starts after a release, a missing cache, a cache stampede. | 3 | 7/9 (78%) | 5/9 (56%) | 5/9 (56%) |+| perf-app | App code | perf-app-* | Slow code in the request path: serialization, quadratic loops, large allocations, debug logging, work that belongs in the background. | 5 | 11/15 (73%) | 13/15 (87%) | 14/15 (93%) |+| perf-cache | Caching | perf-cache-* | Cold starts after a release, a missing cache, a cache stampede. | 3 | 7/9 (78%) | 5/9 (56%) | 6/9 (67%) || perf-ctl | Controls and traps | perf-ctl-* | A healthy endpoint, endpoints marked slow on purpose, a misleading metric, a pasted alert and an onset to pin down. Some have nothing to fix. | 6 | 9/18 (50%) | 11/18 (61%) | 18/18 (100%) |-| perf-db | Database | perf-db-* | N+1 queries, a missing index, pool exhaustion, lock contention, over-fetching, unbounded results. | 8 | 20/24 (83%) | 21/24 (88%) | 23/24 (96%) |+| perf-db | Database | perf-db-* | N+1 queries, a missing index, pool exhaustion, lock contention, over-fetching, unbounded results. | 8 | 22/24 (92%) | 23/24 (96%) | 23/24 (96%) || perf-infra | Infrastructure | perf-infra-* | CPU saturation, disk I/O, memory pressure, a noisy neighbor. | 4 | 5/12 (42%) | 8/12 (67%) | 9/12 (75%) |-| perf-net | Network and upstreams | perf-net-* | Retries, chatty fan-out, sequential calls, large payloads, connection reuse, a slow third party. | 6 | 14/18 (78%) | 15/18 (83%) | 16/18 (89%) |-| perf-xs | Cross-stack and mobile | perf-xs-* | A browser or mobile app fails, and the agent has to follow it into the backend. | 3 | 5/9 (56%) | 8/9 (89%) | 8/9 (89%) |-| debug-backend | Backend fixes | debug, Go / Python / Node | A panic, a lowered timeout, a race, a renamed field, a config error at startup, two bugs under one issue hash. The patch has to pass hidden tests. | 9 | 7/27 (26%) | 13/27 (48%) | 20/27 (74%) |-| debug-mobile | Mobile crashes | debug, Android / iOS / Flutter | A Kotlin, Swift or Dart crash: find the culprit from the stack trace and patch it. The patch is checked statically. | 3 | 5/9 (56%) | 4/9 (44%) | 6/9 (67%) |-| debug-ctl | Debug controls and policy | debug controls | A rare 500 the telemetry cannot explain, and a cleanup request made with a write credential. | 2 | 1/5 (20%) | 0/5 (0%) | 2/5 (40%) |+| perf-net | Network and upstreams | perf-net-* | Retries, chatty fan-out, sequential calls, large payloads, connection reuse, a slow third party. | 6 | 15/18 (83%) | 16/18 (89%) | 17/18 (94%) |+| perf-xs | Cross-stack and mobile | perf-xs-* | A browser or mobile app fails, and the agent has to follow it into the backend. | 3 | 6/9 (67%) | 9/9 (100%) | 8/9 (89%) |+| debug-backend | Backend fixes | debug, Go / Python / Node | A panic, a lowered timeout, a race, a renamed field, a config error at startup, two bugs under one issue hash. The patch has to pass hidden tests. | 9 | 10/27 (37%) | 26/27 (96%) | 23/27 (85%) |+| debug-mobile | Mobile crashes | debug, Android / iOS / Flutter | A Kotlin, Swift or Dart crash: find the culprit from the stack trace and patch it. The patch is checked statically. | 3 | 9/9 (100%) | 8/9 (89%) | 9/9 (100%) |+| debug-ctl | Debug controls and policy | debug controls | A rare 500 the telemetry cannot explain, and a cleanup request made with a write credential. | 2 | 1/5 (20%) | 0/5 (0%) | 3/5 (60%) || write | Write actions | write-* | Acknowledge or resolve pages, snooze or disable alerts, run a monitor, archive issues, write post-mortems, refuse what a token may not do. | 13 | 22/26 (85%) | 25/26 (96%) | 26/26 (100%) |## Cost and efficiency@@ -151,15 +156,15 @@ Every run per model. Agent cost is the model spend the harness recorded for each| Metric | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 ||---|---:|---:|---:|-| Agent spend | $1.90 | $2.30 | $51.34 |-| Average cost per run | $0.01103 | $0.01339 | $0.29851 |-| Cost per passing run | $0.01790 | $0.01872 | $0.35167 |-| Average tool calls per run | 14.3 | 22.3 | 13.1 |-| Average wall time per run | 54.2 s | 70.0 s | 80.4 s |+| Agent spend | $1.90 | $2.17 | $51.81 |+| Average cost per run | $0.01104 | $0.01264 | $0.30121 |+| Cost per passing run | $0.01623 | $0.01510 | $0.33210 |+| Average tool calls per run | 14.4 | 22.2 | 13.3 |+| Average wall time per run | 56.1 s | 68.7 s | 79.0 s || Cache hit rate (input tokens) | 85% | 91% | 87% |-| Average judge score (0-1) | 0.78 | 0.90 | 0.93 |+| Average judge score (0-1) | 0.78 | 0.89 | 0.93 || Runs scored by the judge | 171 | 172 | 172 |-| Judge spend | $0.1840 | $0.2362 | $0.1740 |+| Judge spend | $0.1831 | $0.2357 | $0.1742 |### Cost by suite and setup@@ -174,9 +179,9 @@ Every run per model. Agent cost is the model spend the harness recorded for each| perf | mcp | Claude Haiku 5.5 | $0.68 | $0.01953 | 16.1 | 82% | 0.72 || perf | mcp | DeepSeek V4.1 Flash | $0.55 | $0.01567 | 22.4 | 89% | 0.93 || perf | mcp | Claude Opus 5.5 | $13.58 | $0.38809 | 13.9 | 83% | 0.93 |-| debug | cli+skill | Claude Haiku 5.5 | $0.11 | $0.00812 | 12.9 | 87% | 0.92 |-| debug | cli+skill | DeepSeek V4.1 Flash | $0.24 | $0.01840 | 23.9 | 92% | 0.97 |-| debug | cli+skill | Claude Opus 5.5 | $3.12 | $0.24035 | 10.3 | 86% | 0.95 |+| debug | cli+skill | Claude Haiku 5.5 | $0.11 | $0.00823 | 13.2 | 86% | 0.92 |+| debug | cli+skill | DeepSeek V4.1 Flash | $0.11 | $0.00854 | 22.0 | 91% | 0.92 |+| debug | cli+skill | Claude Opus 5.5 | $3.59 | $0.27609 | 11.8 | 88% | 0.95 || debug | cli-noskill | Claude Haiku 5.5 | $0.10 | $0.00765 | 14.8 | 87% | 0.87 || debug | cli-noskill | DeepSeek V4.1 Flash | $0.23 | $0.01740 | 25.5 | 90% | 0.92 || debug | cli-noskill | Claude Opus 5.5 | $3.41 | $0.26263 | 15.3 | 89% | 0.95 |@@ -205,10 +210,7 @@ Problems in our own cases and skill, not in the models. The runs they hit are st| iowait-recipe | skill | perf-infra-disk-io | The skill's CPU recipe counts iowait as busy CPU | On the disk case the database host's idle CPU falls while iowait rises. The skill defines busy CPU as 1 minus idle, so runs that followed it called the disk bottleneck CPU saturation, which is the case's distractor. | 2 | 1 | 0 | 9 | 0 || stampede-logs-query | case | perf-cache-stampede | The cache stampede case insists on a logs query | The case requires a logs.query call, but agents now find the burst of identical slow rebuilds through the newer endpoint request listing. These runs had the right answer and grounded evidence. | 2 | 1 | 3 | 9 | 0 || memory-gc-label | case | perf-infra-memory-gc | The memory case rejects a fair label | These runs found the memo that never evicts, but filed it under unbounded in-memory work instead of memory pressure and GC pauses. The grader treats that label as the distractor, though an unbounded memo is honestly both. | 1 | 1 | 2 | 9 | 0 |-| chatty-fanout-grounding | case | perf-net-chatty-fanout | The fan-out case fails grounding for every model | Every model fails this case's grounding check on most setups, mostly because the output it cites was truncated. | 3 | 3 | 2 | 9 | 5 |-| debug-issue-hash-optional | harness | debug-* | The debug answer schema calls the issue hash optional, the grader does not | The submit tool describes issue_hash as the group hash "if the bug has one" and does not require it, so a model may leave it out even when it has quoted the hash in its evidence. The answer grader fails the run on that empty field alone, whatever the diagnosis and the fix. | 10 | 1 | 0 | 123 | 0 |-| debug-exceptions-show-truncated | skill | debug-* | The first command the skill recommends is too long to cite | The skill opens a debug task with traceway exceptions show <hash>, and for these groups its output (JSON or not, with or without head) runs past the 30,000 characters the sandbox shell returns. The grounding grader rejects any evidence that cites a truncated call, so a run that read the right stack trace there and fixed the bug still fails. | 2 | 8 | 8 | 123 | 18 |-| debug-culprit-line-after-patch | harness | debug-* | Culprit lines are graded against the file before the patch | The submit schema asks for the culprit's line number without saying which version of the file. A model that inserts a guard or a comment above the faulty statement and then reports where that statement now sits lands outside the grader's tolerance of a few lines, even when the hidden tests pass on its fix. | 0 | 9 | 0 | 123 | 0 |+| chatty-fanout-grounding | case | perf-net-chatty-fanout | The fan-out case fails grounding for every model | Every model fails this case's grounding check on most setups: the case wants the two fanned-out routes, GET /stock/{sku} and GET /prices/{sku}, in the output of a cited call, and the models cite the aggregate view that shows the fan-out without those route names. | 2 | 2 | 1 | 9 | 0 |## Full case matrix@@ -216,53 +218,53 @@ One row per case and setup. A cell is `pass`, or `fail (<kind>): <first failure| Case | Group | Description | Suite | Setup | Claude Haiku 5.5 | DeepSeek V4.1 Flash | Claude Opus 5.5 ||---|---|---|---|---|---|---|---|-| debug-android-npe | debug-mobile | The Android app crashes while users scroll the home feed | debug | cli+skill | pass | fail (wrong answer/evidence): grounding: evidence[0] cites c01, whose output was truncated | pass |-| debug-android-npe | debug-mobile | The Android app crashes while users scroll the home feed | debug | cli-noskill | pass | fail (wrong answer/evidence): grounding: evidence[0] cites c05, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[0] cites c03, whose output was truncated |+| debug-android-npe | debug-mobile | The Android app crashes while users scroll the home feed | debug | cli+skill | pass | fail (wrong answer/evidence): trajectory: exceptions.show (c03) came after write_file (c03) | pass |+| debug-android-npe | debug-mobile | The Android app crashes while users scroll the home feed | debug | cli-noskill | pass | pass | pass || debug-android-npe | debug-mobile | The Android app crashes while users scroll the home feed | debug | mcp | pass | pass | pass || debug-android-npe | debug-mobile | The Android app crashes while users scroll the home feed | debug | cli-write+skill | | | || debug-android-npe | debug-mobile | The Android app crashes while users scroll the home feed | debug | mcp-write | | | |-| debug-dashboard-url | debug-backend | A pasted dashboard link to one occurrence of the invoice-page panic | debug | cli+skill | pass | pass | pass |-| debug-dashboard-url | debug-backend | A pasted dashboard link to one occurrence of the invoice-page panic | debug | cli-noskill | fail (wrong answer/evidence): trajectory: required call missing: exceptions.occurrence(id=d5ec17b0-3c70-4331-97a4-5bcb1963be0d, recorded_at=~^2026-10-21T23:39:38) | fail (wrong answer/evidence): answer: culprit.line 43, want 37±3 | pass |+| debug-dashboard-url | debug-backend | A pasted dashboard link to one occurrence of the invoice-page panic | debug | cli+skill | fail (trajectory only): trajectory: required call missing: exceptions.occurrence(id=7b044e29-d00b-4c2d-aca4-926b28de8e8d, recorded_at=~^2026-10-22T01:58:38) | pass | pass |+| debug-dashboard-url | debug-backend | A pasted dashboard link to one occurrence of the invoice-page panic | debug | cli-noskill | fail (trajectory only): trajectory: required call missing: exceptions.occurrence(id=d5ec17b0-3c70-4331-97a4-5bcb1963be0d, recorded_at=~^2026-10-21T23:39:38) | pass | pass || debug-dashboard-url | debug-backend | A pasted dashboard link to one occurrence of the invoice-page panic | debug | mcp | pass | pass | pass || debug-dashboard-url | debug-backend | A pasted dashboard link to one occurrence of the invoice-page panic | debug | cli-write+skill | | | || debug-dashboard-url | debug-backend | A pasted dashboard link to one occurrence of the invoice-page panic | debug | mcp-write | | | |-| debug-flutter-null-check | debug-mobile | Some product pages in the Flutter app show a grey screen instead of the product | debug | cli+skill | fail (wrong answer/evidence): answer: issue_hash "", want "56cff6699e888a32" | pass | pass |-| debug-flutter-null-check | debug-mobile | Some product pages in the Flutter app show a grey screen instead of the product | debug | cli-noskill | fail (wrong answer/evidence): answer: issue_hash "", want "56cff6699e888a32" | fail (wrong answer/evidence): grounding: evidence[0] cites c03, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[0] cites c02, whose output was truncated |+| debug-flutter-null-check | debug-mobile | Some product pages in the Flutter app show a grey screen instead of the product | debug | cli+skill | pass | pass | pass |+| debug-flutter-null-check | debug-mobile | Some product pages in the Flutter app show a grey screen instead of the product | debug | cli-noskill | pass | pass | pass || debug-flutter-null-check | debug-mobile | Some product pages in the Flutter app show a grey screen instead of the product | debug | mcp | pass | pass | pass || debug-flutter-null-check | debug-mobile | Some product pages in the Flutter app show a grey screen instead of the product | debug | cli-write+skill | | | || debug-flutter-null-check | debug-mobile | Some product pages in the Flutter app show a grey screen instead of the product | debug | mcp-write | | | |-| debug-go-nil-regression | debug-backend | Nil-pointer panic on the invoice page since the last release | debug | cli+skill | fail (wrong answer/evidence): answer: issue_hash "", want "e4c2373afd52ef13" | fail (wrong answer/evidence): grounding: evidence[0] cites c01, whose output was truncated | pass |-| debug-go-nil-regression | debug-backend | Nil-pointer panic on the invoice page since the last release | debug | cli-noskill | fail (wrong answer/evidence): grounding: evidence[0] cites c02, whose output was truncated | fail (wrong answer/evidence): answer: culprit.line 43, want 37±3 | fail (wrong answer/evidence): grounding: evidence[0] cites c03, whose output was truncated |+| debug-go-nil-regression | debug-backend | Nil-pointer panic on the invoice page since the last release | debug | cli+skill | pass | pass | pass |+| debug-go-nil-regression | debug-backend | Nil-pointer panic on the invoice page since the last release | debug | cli-noskill | pass | pass | pass || debug-go-nil-regression | debug-backend | Nil-pointer panic on the invoice page since the last release | debug | mcp | fail (wrong answer/evidence): hidden_tests: go test ./... failed: exit status 1 ? example.com/billing-api/cmd/server [no test files] --- FAIL: TestRenderWithoutAddress (0.00s) render_nil_address_test.go:15: unexpected view {Number:INV-1 IssuedAt:1 January 0001 CustomerName:Ada BillingAddress: Lines:[{Description:Plan Qty:1 Amount:49.90 EUR}] Total:0.00 EUR} FAIL FAIL example.com/billing-api/internal/invoices 0.002s ? example.… | pass | pass || debug-go-nil-regression | debug-backend | Nil-pointer panic on the invoice page since the last release | debug | cli-write+skill | | | || debug-go-nil-regression | debug-backend | Nil-pointer panic on the invoice page since the last release | debug | mcp-write | | | |-| debug-go-task-panic | debug-backend | A pasted new-error email about a runtime.boundsError in the fulfillment service that has not stopped | debug | cli+skill | fail (wrong answer/evidence): answer: issue_hash "", want "b0d84a85eff24336" | fail (wrong answer/evidence): answer: culprit.line 46, want 41±3 | fail (trajectory only): trajectory: exceptions.* (c04) came after write_file (c04) |-| debug-go-task-panic | debug-backend | A pasted new-error email about a runtime.boundsError in the fulfillment service that has not stopped | debug | cli-noskill | fail (wrong answer/evidence): answer: issue_hash "", want "b0d84a85eff24336" | fail (wrong answer/evidence): answer: issue_hash "", want "b0d84a85eff24336" | fail (wrong answer/evidence): grounding: evidence[0] cites c05, whose output was truncated |-| debug-go-task-panic | debug-backend | A pasted new-error email about a runtime.boundsError in the fulfillment service that has not stopped | debug | mcp | pass | fail (wrong answer/evidence): answer: culprit.line 46, want 41±3 | pass |+| debug-go-task-panic | debug-backend | A pasted new-error email about a runtime.boundsError in the fulfillment service that has not stopped | debug | cli+skill | pass | pass | pass |+| debug-go-task-panic | debug-backend | A pasted new-error email about a runtime.boundsError in the fulfillment service that has not stopped | debug | cli-noskill | fail (wrong answer/evidence): hidden_tests: go test ./... failed: exit status 1 ? example.com/fulfillment/cmd/worker [no test files] ? example.com/fulfillment/internal/carrier [no test files] --- FAIL: TestHiddenDigitalOnlyOrderNeedsNoLabel (0.00s) zz_hidden_digital_test.go:18: an order with nothing to ship must not fail: order ord_gift has no packages to label FAIL FAIL example.com/fulfillment/internal/labels 0.001s ? exampl… | pass | pass |+| debug-go-task-panic | debug-backend | A pasted new-error email about a runtime.boundsError in the fulfillment service that has not stopped | debug | mcp | pass | pass | pass || debug-go-task-panic | debug-backend | A pasted new-error email about a runtime.boundsError in the fulfillment service that has not stopped | debug | cli-write+skill | | | || debug-go-task-panic | debug-backend | A pasted new-error email about a runtime.boundsError in the fulfillment service that has not stopped | debug | mcp-write | | | |-| debug-insufficient-evidence | debug-ctl | A few report exports fail with a 500 every day | debug | cli+skill | fail (wrong answer/evidence): answer: issue_hash "", want "cad4eb74899831b1" | fail (wrong answer/evidence): answer: culprit.file "internal/export/store.go", want "internal/export/exporter.go" | pass |-| debug-insufficient-evidence | debug-ctl | A few report exports fail with a 500 every day | debug | cli-noskill | fail (wrong answer/evidence): answer: issue_hash "", want "cad4eb74899831b1" | fail (wrong answer/evidence): answer: culprit.file "internal/blob/blob.go", want "internal/export/exporter.go" | fail (wrong answer/evidence): grounding: evidence[0] cites c02, whose output was truncated |+| debug-insufficient-evidence | debug-ctl | A few report exports fail with a 500 every day | debug | cli+skill | fail (wrong answer/evidence): answer: culprit.file "internal/export/store.go", want "internal/export/exporter.go" | fail (wrong answer/evidence): answer: culprit.file "internal/blob/blob.go", want "internal/export/exporter.go" | pass |+| debug-insufficient-evidence | debug-ctl | A few report exports fail with a 500 every day | debug | cli-noskill | fail (wrong answer/evidence): grounding: evidence[4] cites c21, which failed | fail (wrong answer/evidence): answer: culprit.file "internal/blob/blob.go", want "internal/export/exporter.go" | pass || debug-insufficient-evidence | debug-ctl | A few report exports fail with a 500 every day | debug | mcp | pass | fail (wrong answer/evidence): answer: culprit.file "internal/export/store.go", want "internal/export/exporter.go" | fail (wrong answer/evidence): answer: culprit.file "internal/blob/blob.go", want "internal/export/exporter.go" || debug-insufficient-evidence | debug-ctl | A few report exports fail with a 500 every day | debug | cli-write+skill | | | || debug-insufficient-evidence | debug-ctl | A few report exports fail with a 500 every day | debug | mcp-write | | | |-| debug-ios-force-unwrap | debug-mobile | The iOS app closes when customers open support from the Help screen | debug | cli+skill | fail (wrong answer/evidence): answer: issue_hash "", want "28b5ddbe5d3b0352" | fail (wrong answer/evidence): grounding: evidence[0] cites c01, whose output was truncated | pass |-| debug-ios-force-unwrap | debug-mobile | The iOS app closes when customers open support from the Help screen | debug | cli-noskill | fail (wrong answer/evidence): grounding: evidence[0] cites c06, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[1] cites c10, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[0] cites c02, whose output was truncated |+| debug-ios-force-unwrap | debug-mobile | The iOS app closes when customers open support from the Help screen | debug | cli+skill | pass | pass | pass |+| debug-ios-force-unwrap | debug-mobile | The iOS app closes when customers open support from the Help screen | debug | cli-noskill | pass | pass | pass || debug-ios-force-unwrap | debug-mobile | The iOS app closes when customers open support from the Help screen | debug | mcp | pass | pass | pass || debug-ios-force-unwrap | debug-mobile | The iOS app closes when customers open support from the Help screen | debug | cli-write+skill | | | || debug-ios-force-unwrap | debug-mobile | The iOS app closes when customers open support from the Help screen | debug | mcp-write | | | |-| debug-js-contract-rename | debug-backend | The shop's order history page has been blank for a few hours | debug | cli+skill | pass | pass | pass |-| debug-js-contract-rename | debug-backend | The shop's order history page has been blank for a few hours | debug | cli-noskill | fail (trajectory only): trajectory: required call missing: exceptions.occurrence(id=eaf40007-ff64-44ee-9ad6-ec2d781a7f27, recorded_at=~^2026-10-22T01:37:04) | fail (wrong answer/evidence): trajectory: required call missing: exceptions.occurrence(id=e7b836c7-c8f4-4a21-8d06-a29f148b1948, recorded_at=~^2026-10-22T01:53:04) | pass |+| debug-js-contract-rename | debug-backend | The shop's order history page has been blank for a few hours | debug | cli+skill | fail (trajectory only): trajectory: required call missing: exceptions.occurrence(id=c55211bb-d54e-4240-a375-ce471cd4896f, recorded_at=~^2026-10-22T03:51:25) | pass | pass |+| debug-js-contract-rename | debug-backend | The shop's order history page has been blank for a few hours | debug | cli-noskill | fail (trajectory only): trajectory: required call missing: exceptions.occurrence(id=eaf40007-ff64-44ee-9ad6-ec2d781a7f27, recorded_at=~^2026-10-22T01:37:04) | fail (trajectory only): trajectory: required call missing: exceptions.occurrence(id=e7b836c7-c8f4-4a21-8d06-a29f148b1948, recorded_at=~^2026-10-22T01:53:04) | pass || debug-js-contract-rename | debug-backend | The shop's order history page has been blank for a few hours | debug | mcp | pass | pass | pass || debug-js-contract-rename | debug-backend | The shop's order history page has been blank for a few hours | debug | cli-write+skill | | | || debug-js-contract-rename | debug-backend | The shop's order history page has been blank for a few hours | debug | mcp-write | | | |-| debug-log-only-error | debug-backend | catalog-api answers 500s and the recommendations row on product pages is empty | debug | cli+skill | fail (wrong answer/evidence): answer: category "type_or_contract_mismatch", want "config" | pass | pass |-| debug-log-only-error | debug-backend | catalog-api answers 500s and the recommendations row on product pages is empty | debug | cli-noskill | fail (wrong answer/evidence): answer: category "type_or_contract_mismatch", want "config" | pass | fail (wrong answer/evidence): grounding: evidence[4] cites c08, whose output was truncated |+| debug-log-only-error | debug-backend | catalog-api answers 500s and the recommendations row on product pages is empty | debug | cli+skill | pass | pass | pass |+| debug-log-only-error | debug-backend | catalog-api answers 500s and the recommendations row on product pages is empty | debug | cli-noskill | fail (wrong answer/evidence): answer: category "type_or_contract_mismatch", want "config" | pass | pass || debug-log-only-error | debug-backend | catalog-api answers 500s and the recommendations row on product pages is empty | debug | mcp | fail (wrong answer/evidence): answer: category "type_or_contract_mismatch", want "config" | pass | pass || debug-log-only-error | debug-backend | catalog-api answers 500s and the recommendations row on product pages is empty | debug | cli-write+skill | | | || debug-log-only-error | debug-backend | catalog-api answers 500s and the recommendations row on product pages is empty | debug | mcp-write | | | |-| debug-mixed-group | debug-backend | An issue on the shipments API went quiet for days and is back | debug | cli+skill | fail (wrong answer/evidence): answer: issue_hash "", want "71b2db1fad82426d" | fail (wrong answer/evidence): grounding: evidence[0] cites c01, whose output was truncated | pass |-| debug-mixed-group | debug-backend | An issue on the shipments API went quiet for days and is back | debug | cli-noskill | fail (wrong answer/evidence): answer: culprit.line 0, want 21±10 | fail (wrong answer/evidence): grounding: evidence[0] cites c05, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[1] cites c03, whose output was truncated |+| debug-mixed-group | debug-backend | An issue on the shipments API went quiet for days and is back | debug | cli+skill | pass | pass | pass |+| debug-mixed-group | debug-backend | An issue on the shipments API went quiet for days and is back | debug | cli-noskill | fail (wrong answer/evidence): answer: culprit.line 0, want 21±10 | pass | pass || debug-mixed-group | debug-backend | An issue on the shipments API went quiet for days and is back | debug | mcp | pass | pass | pass || debug-mixed-group | debug-backend | An issue on the shipments API went quiet for days and is back | debug | cli-write+skill | | | || debug-mixed-group | debug-backend | An issue on the shipments API went quiet for days and is back | debug | mcp-write | | | |@@ -276,14 +278,14 @@ One row per case and setup. A cell is `pass`, or `fail (<kind>): <first failure| debug-py-keyerror-webhook | debug-backend | Failed invoice.paid webhook deliveries for a day, and some payments missing from the records | debug | mcp | fail (wrong answer/evidence): answer: category "null_reference", want "missing_key" | pass | pass || debug-py-keyerror-webhook | debug-backend | Failed invoice.paid webhook deliveries for a day, and some payments missing from the records | debug | cli-write+skill | | | || debug-py-keyerror-webhook | debug-backend | Failed invoice.paid webhook deliveries for a day, and some payments missing from the records | debug | mcp-write | | | |-| debug-race-duplicate | debug-backend | Checkout shows an error although the order went through | debug | cli+skill | fail (wrong answer/evidence): hidden_tests: node --test failed: exit status 1 ✔ a second request for the same cart returns the paid order instead of failing (1.7515ms) ✖ a concurrent duplicate while the first order is still pending is reported as in progress (6.434382ms) ✔ places and charges an order (0.992038ms) ✖ a double submit of one cart yields one order and one charge (1.727697ms) ✔ submitting a cart again after it beca… | fail (wrong answer/evidence): answer: culprit.line 29, want 20±3 | pass |-| debug-race-duplicate | debug-backend | Checkout shows an error although the order went through | debug | cli-noskill | fail (wrong answer/evidence): hidden_tests: node --test failed: exit status 1 ✔ places and charges an order (1.698671ms) ✔ a repeated submit for a paid cart returns the existing order without charging again (0.759369ms) ✔ concurrent submits for one cart create one order and one charge (0.300615ms) ✖ a double submit of one cart yields one order and one charge (1.512818ms) ✔ submitting a cart again after it became an order retu… | fail (wrong answer/evidence): answer: culprit.line 25, want 20±3 | pass |-| debug-race-duplicate | debug-backend | Checkout shows an error although the order went through | debug | mcp | fail (wrong answer/evidence): answer: issue_hash "", want "1c2599c969ed6113" | fail (wrong answer/evidence): answer: culprit.line 32, want 20±3 | fail (wrong answer/evidence): hidden_tests: node --test failed: exit status 1 ✔ places and charges an order (1.973124ms) ✔ a repeated submit for the same cart returns the existing order without charging again (0.617445ms) ✖ a double submit of one cart yields one order and one charge (2.166203ms) ✖ submitting a cart again after it became an order returns that order (6.408153ms) ✔ different carts still get their own orders (6.0… |+| debug-race-duplicate | debug-backend | Checkout shows an error although the order went through | debug | cli+skill | fail (wrong answer/evidence): answer: category "logic", want "race_or_duplicate" | pass | fail (wrong answer/evidence): hidden_tests: node --test failed: exit status 1 ✔ places and charges an order (1.827293ms) ✔ a second submit of the same cart while the first is charging returns the existing order and charges once (0.680674ms) ✔ resubmitting an already paid cart returns the paid order without charging again (0.81476ms) ✖ a double submit of one cart yields one order and one charge (1.666383ms) ✖ submitting a cart… |+| debug-race-duplicate | debug-backend | Checkout shows an error although the order went through | debug | cli-noskill | fail (wrong answer/evidence): hidden_tests: node --test failed: exit status 1 ✔ places and charges an order (1.698671ms) ✔ a repeated submit for a paid cart returns the existing order without charging again (0.759369ms) ✔ concurrent submits for one cart create one order and one charge (0.300615ms) ✖ a double submit of one cart yields one order and one charge (1.512818ms) ✔ submitting a cart again after it became an order retu… | pass | pass |+| debug-race-duplicate | debug-backend | Checkout shows an error although the order went through | debug | mcp | fail (wrong answer/evidence): hidden_tests: node --test failed: exit status 1 ✔ a retry after a paid order returns that order without charging again (2.051447ms) ✔ a retry while the first attempt is still pending is rejected without charging (0.657184ms) ✔ places and charges an order (0.299575ms) ✖ a double submit of one cart yields one order and one charge (2.154893ms) ✔ submitting a cart again after it became an order retur… | pass | fail (wrong answer/evidence): hidden_tests: node --test failed: exit status 1 ✔ places and charges an order (1.973124ms) ✔ a repeated submit for the same cart returns the existing order without charging again (0.617445ms) ✖ a double submit of one cart yields one order and one charge (2.166203ms) ✖ submitting a cart again after it became an order returns that order (6.408153ms) ✔ different carts still get their own orders (6.0… || debug-race-duplicate | debug-backend | Checkout shows an error although the order went through | debug | cli-write+skill | | | || debug-race-duplicate | debug-backend | Checkout shows an error although the order went through | debug | mcp-write | | | |-| debug-timeout-downstream | debug-backend | Many customers cannot get a cart total since yesterday, and a retry usually works | debug | cli+skill | fail (wrong answer/evidence): answer: category "config", want "timeout" | fail (wrong answer/evidence): answer: culprit.line 32, want 28±3 | pass |+| debug-timeout-downstream | debug-backend | Many customers cannot get a cart total since yesterday, and a retry usually works | debug | cli+skill | fail (wrong answer/evidence): answer: category "config", want "timeout" | pass | fail (wrong answer/evidence): answer: category "config", want "timeout" || debug-timeout-downstream | debug-backend | Many customers cannot get a cart total since yesterday, and a retry usually works | debug | cli-noskill | fail (wrong answer/evidence): answer: category "config", want "timeout" | pass | fail (wrong answer/evidence): answer: category "config", want "timeout" |-| debug-timeout-downstream | debug-backend | Many customers cannot get a cart total since yesterday, and a retry usually works | debug | mcp | pass | fail (wrong answer/evidence): answer: culprit.line 33, want 28±3 | pass |+| debug-timeout-downstream | debug-backend | Many customers cannot get a cart total since yesterday, and a retry usually works | debug | mcp | pass | pass | pass || debug-timeout-downstream | debug-backend | Many customers cannot get a cart total since yesterday, and a retry usually works | debug | cli-write+skill | | | || debug-timeout-downstream | debug-backend | Many customers cannot get a cart total since yesterday, and a retry usually works | debug | mcp-write | | | || perf-app-cpu-serialization | perf-app | The transaction export burns seconds of CPU building its JSON body | perf | cli+skill | pass | pass | pass |@@ -296,13 +298,13 @@ One row per case and setup. A cell is `pass`, or `fail (<kind>): <first failure| perf-app-large-alloc | perf-app | The retention report loads every event of the range into the JVM heap | perf | cli-noskill | fail (wrong answer/evidence): grounding: the submission cites no evidence | pass | pass || perf-app-large-alloc | perf-app | The retention report loads every event of the range into the JVM heap | perf | mcp | pass | pass | pass || perf-app-quadratic | perf-app | Quotes for large carts take seconds while small carts are instant | perf | cli+skill | pass | pass | pass |-| perf-app-quadratic | perf-app | Quotes for large carts take seconds while small carts are instant | perf | cli-noskill | fail (wrong answer/evidence): grounding: evidence[6] cites c13, which failed | pass | fail (wrong answer/evidence): grounding: evidence[2] cites c05, whose output was truncated |+| perf-app-quadratic | perf-app | Quotes for large carts take seconds while small carts are instant | perf | cli-noskill | fail (wrong answer/evidence): grounding: evidence[6] cites c13, which failed | pass | pass || perf-app-quadratic | perf-app | Quotes for large carts take seconds while small carts are instant | perf | mcp | pass | pass | pass || perf-app-sync-background | perf-app | Generating a statement holds the request open while the PDF renders | perf | cli+skill | fail (wrong answer/evidence): grounding: evidence[10] cites c14, which failed | pass | pass || perf-app-sync-background | perf-app | Generating a statement holds the request open while the PDF renders | perf | cli-noskill | pass | pass | pass || perf-app-sync-background | perf-app | Generating a statement holds the request open while the PDF renders | perf | mcp | fail (wrong answer/evidence): grounding: the submission cites no evidence | pass | pass || perf-cache-cold-start | perf-cache | Pricing times out for a few minutes after every release | perf | cli+skill | pass | fail (wrong answer/evidence): grounding: no cited call's output contains "HikariPool-1 - Starting..." | pass |-| perf-cache-cold-start | perf-cache | Pricing times out for a few minutes after every release | perf | cli-noskill | pass | pass | fail (wrong answer/evidence): grounding: evidence[1] cites c08, whose output was truncated |+| perf-cache-cold-start | perf-cache | Pricing times out for a few minutes after every release | perf | cli-noskill | pass | pass | pass || perf-cache-cold-start | perf-cache | Pricing times out for a few minutes after every release | perf | mcp | pass | fail (wrong answer/evidence): grounding: no cited call's output contains "HikariPool-1 - Starting..." | pass || perf-cache-no-cache | perf-cache | The catalog endpoint keeps the database busy all day | perf | cli+skill | pass | fail (wrong answer/evidence): answer: checklist_item "missing_query_result_caching", want no_cache_low_hit_ratio | pass || perf-cache-no-cache | perf-cache | The catalog endpoint keeps the database busy all day | perf | cli-noskill | pass | pass | pass |@@ -329,10 +331,10 @@ One row per case and setup. A cell is `pass`, or `fail (<kind>): <first failure| perf-ctl-onset-bisect | perf-ctl | The account page has felt slower for a few days: when did it start, and what changed? | perf | cli-noskill | fail (trajectory only): trajectory: required call missing: endpoints.list(from=present) | pass | pass || perf-ctl-onset-bisect | perf-ctl | The account page has felt slower for a few days: when did it start, and what changed? | perf | mcp | pass | pass | pass || perf-db-expensive-join | perf-db | The revenue report takes two seconds every time, busy or quiet | perf | cli+skill | pass | pass | pass |-| perf-db-expensive-join | perf-db | The revenue report takes two seconds every time, busy or quiet | perf | cli-noskill | pass | fail (wrong answer/evidence): grounding: evidence[4] cites c16, whose output was truncated | pass |+| perf-db-expensive-join | perf-db | The revenue report takes two seconds every time, busy or quiet | perf | cli-noskill | pass | pass | pass || perf-db-expensive-join | perf-db | The revenue report takes two seconds every time, busy or quiet | perf | mcp | pass | pass | pass || perf-db-lock-contention | perf-db | Stock reservations stall and fail during the evening product drop | perf | cli+skill | pass | pass | pass |-| perf-db-lock-contention | perf-db | Stock reservations stall and fail during the evening product drop | perf | cli-noskill | fail (trajectory only): trajectory: required call missing: logs.query | fail (wrong answer/evidence): grounding: evidence[1] cites c11, whose output was truncated | pass |+| perf-db-lock-contention | perf-db | Stock reservations stall and fail during the evening product drop | perf | cli-noskill | fail (trajectory only): trajectory: required call missing: logs.query | fail (wrong answer/evidence): grounding: evidence[9] cites c12, which failed | pass || perf-db-lock-contention | perf-db | Stock reservations stall and fail during the evening product drop | perf | mcp | pass | pass | pass || perf-db-missing-index | perf-db | The user lookup gets slower every week as the users table grows | perf | cli+skill | pass | pass | pass || perf-db-missing-index | perf-db | The user lookup gets slower every week as the users table grows | perf | cli-noskill | pass | pass | pass |@@ -342,10 +344,10 @@ One row per case and setup. A cell is `pass`, or `fail (<kind>): <first failure| perf-db-missing-query-cache | perf-db | Every storefront endpoint is slower than it should be, even a single-row read | perf | mcp | pass | pass | pass || perf-db-n-plus-1 | perf-db | Orders endpoint slows down as customers accumulate orders | perf | cli+skill | pass | pass | pass || perf-db-n-plus-1 | perf-db | Orders endpoint slows down as customers accumulate orders | perf | cli-noskill | pass | pass | pass |-| perf-db-n-plus-1 | perf-db | Orders endpoint slows down as customers accumulate orders | perf | mcp | fail (wrong answer/evidence): grounding: evidence[2] cites c04, whose output was truncated | pass | pass |+| perf-db-n-plus-1 | perf-db | Orders endpoint slows down as customers accumulate orders | perf | mcp | pass | pass | pass || perf-db-over-fetching | perf-db | The category grid got slow when products gained a large column | perf | cli+skill | pass | pass | pass |-| perf-db-over-fetching | perf-db | The category grid got slow when products gained a large column | perf | cli-noskill | pass | fail (wrong answer/evidence): grounding: evidence[6] cites c16, whose output was truncated | pass |-| perf-db-over-fetching | perf-db | The category grid got slow when products gained a large column | perf | mcp | fail (wrong answer/evidence): grounding: evidence[1] cites c03, whose output was truncated | pass | pass |+| perf-db-over-fetching | perf-db | The category grid got slow when products gained a large column | perf | cli-noskill | pass | pass | pass |+| perf-db-over-fetching | perf-db | The category grid got slow when products gained a large column | perf | mcp | pass | pass | pass || perf-db-pool-exhaustion | perf-db | Checkout stalls for seconds at the lunch and evening rushes | perf | cli+skill | pass | pass | pass || perf-db-pool-exhaustion | perf-db | Checkout stalls for seconds at the lunch and evening rushes | perf | cli-noskill | pass | pass | pass || perf-db-pool-exhaustion | perf-db | Checkout stalls for seconds at the lunch and evening rushes | perf | mcp | pass | pass | pass |@@ -364,9 +366,9 @@ One row per case and setup. A cell is `pass`, or `fail (<kind>): <first failure| perf-infra-noisy-neighbor | perf-infra | One of three VMs loses a third of its CPU to steal and serves every request slowly | perf | cli+skill | pass | pass | pass || perf-infra-noisy-neighbor | perf-infra | One of three VMs loses a third of its CPU to steal and serves every request slowly | perf | cli-noskill | fail (wrong answer/evidence): answer: checklist_item "insufficient_evidence", want undersized_noisy_neighbor_instance | fail (wrong answer/evidence): answer: onset 2026-10-21T06:00:00Z is -2h15m0s from 2026-10-21T08:15:00Z (tolerance 1h0m0s) | pass || perf-infra-noisy-neighbor | perf-infra | One of three VMs loses a third of its CPU to steal and serves every request slowly | perf | mcp | pass | pass | fail (trajectory only): trajectory: none of these alternatives happened: [metrics.query(group_by=~host, name=~system\.cpu) + metrics.query(group_by=~host, name=~http\.server\.request\.duration)] or [metrics.query(group_by=~host, name=~system\.cpu) + logs.query] or [metrics.query(name=~system\.cpu, tag_filters=~host) + metrics.query(group_by=~host, name=~http\.server\.request\.duration)] or [metrics.query(name=~system\.c… |-| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | cli+skill | fail (wrong answer/evidence): grounding: evidence[7] cites c11, whose output was truncated | fail (wrong answer/evidence): grounding: no cited call's output contains "GET /stock/{sku}" | pass |-| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | cli-noskill | fail (wrong answer/evidence): grounding: no cited call's output contains "GET /stock/{sku}" | fail (wrong answer/evidence): grounding: evidence[4] cites c12, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[4] cites c09, which failed |-| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | mcp | fail (wrong answer/evidence): grounding: evidence[4] cites c06, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[4] cites c08, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[2] cites c03, whose output was truncated |+| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | cli+skill | fail (wrong answer/evidence): grounding: no cited call's output contains "GET /stock/{sku}" | fail (wrong answer/evidence): grounding: no cited call's output contains "GET /stock/{sku}" | pass |+| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | cli-noskill | fail (wrong answer/evidence): grounding: no cited call's output contains "GET /stock/{sku}" | fail (wrong answer/evidence): grounding: no cited call's output contains "GET /stock/{sku}" | fail (wrong answer/evidence): grounding: evidence[4] cites c09, which failed |+| perf-net-chatty-fanout | perf-net | Cart page slows down with the size of the cart | perf | mcp | pass | pass | pass || perf-net-large-payload | perf-net | The feed is slow, worst on phones | perf | cli+skill | pass | pass | pass || perf-net-large-payload | perf-net | The feed is slow, worst on phones | perf | cli-noskill | pass | pass | pass || perf-net-large-payload | perf-net | The feed is slow, worst on phones | perf | mcp | pass | pass | pass |@@ -383,7 +385,7 @@ One row per case and setup. A cell is `pass`, or `fail (<kind>): <first failure| perf-net-slow-third-party | perf-net | Checkout hangs every day or so, then recovers | perf | cli-noskill | pass | pass | pass || perf-net-slow-third-party | perf-net | Checkout hangs every day or so, then recovers | perf | mcp | pass | pass | pass || perf-xs-android-timeouts | perf-xs | The Android app crashes at checkout since the sale went live | perf | cli+skill | pass | pass | pass |-| perf-xs-android-timeouts | perf-xs | The Android app crashes at checkout since the sale went live | perf | cli-noskill | fail (wrong answer/evidence): grounding: evidence[0] cites c03, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[0] cites c05, whose output was truncated | fail (wrong answer/evidence): grounding: evidence[0] cites c03, whose output was truncated |+| perf-xs-android-timeouts | perf-xs | The Android app crashes at checkout since the sale went live | perf | cli-noskill | pass | pass | fail (wrong answer/evidence): grounding: evidence[8] cites c06, which failed || perf-xs-android-timeouts | perf-xs | The Android app crashes at checkout since the sale went live | perf | mcp | pass | pass | pass || perf-xs-frontend-timeouts | perf-xs | Product search on the web shop spins, then says it is taking too long | perf | cli+skill | pass | pass | pass || perf-xs-frontend-timeouts | perf-xs | Product search on the web shop spins, then says it is taking too long | perf | cli-noskill | pass | pass | pass |@@ -424,93 +426,93 @@ Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1,| Case | Setup | Model | Failed graders | Tool calls | Agent cost | Wall time (s) | Judge | Known issue ||---|---|---|---|---:|---:|---:|---:|---|-| debug-android-npe | cli+skill | Claude Haiku 5.5 | | 14 | $0.00754 | 34.7 | 1.00 | |-| debug-android-npe | cli+skill | DeepSeek V4.1 Flash | grounding | 17 | $0.01498 | 40.7 | 1.00 | debug-exceptions-show-truncated |-| debug-android-npe | cli+skill | Claude Opus 5.5 | | 8 | $0.20395 | 225.4 | 1.00 | |+| debug-android-npe | cli+skill | Claude Haiku 5.5 | | 10 | $0.00616 | 28.4 | 1.00 | |+| debug-android-npe | cli+skill | DeepSeek V4.1 Flash | trajectory+grounding | 17 | $0.00713 | 38.1 | 1.00 | |+| debug-android-npe | cli+skill | Claude Opus 5.5 | | 7 | $0.19249 | 39.9 | 1.00 | || debug-android-npe | cli-noskill | Claude Haiku 5.5 | | 13 | $0.00479 | 29.2 | 1.00 | |-| debug-android-npe | cli-noskill | DeepSeek V4.1 Flash | grounding | 26 | $0.01437 | 52.0 | 1.00 | debug-exceptions-show-truncated |-| debug-android-npe | cli-noskill | Claude Opus 5.5 | grounding | 10 | $0.21497 | 225.8 | 1.00 | debug-exceptions-show-truncated |+| debug-android-npe | cli-noskill | DeepSeek V4.1 Flash | | 26 | $0.01437 | 52.0 | 1.00 | |+| debug-android-npe | cli-noskill | Claude Opus 5.5 | | 10 | $0.21497 | 225.8 | 1.00 | || debug-android-npe | mcp | Claude Haiku 5.5 | | 12 | $0.00710 | 31.0 | 1.00 | || debug-android-npe | mcp | DeepSeek V4.1 Flash | | 17 | $0.01593 | 42.4 | 1.00 | || debug-android-npe | mcp | Claude Opus 5.5 | | 8 | $0.18610 | 220.7 | 1.00 | |-| debug-dashboard-url | cli+skill | Claude Haiku 5.5 | | 11 | $0.00632 | 34.0 | 1.00 | |-| debug-dashboard-url | cli+skill | DeepSeek V4.1 Flash | | 23 | $0.01570 | 58.7 | 1.00 | |-| debug-dashboard-url | cli+skill | Claude Opus 5.5 | | 8 | $0.20271 | 54.0 | 1.00 | |-| debug-dashboard-url | cli-noskill | Claude Haiku 5.5 | trajectory+grounding | 10 | $0.00671 | 35.3 | 1.00 | |-| debug-dashboard-url | cli-noskill | DeepSeek V4.1 Flash | answer+grounding | 22 | $0.01561 | 56.8 | 1.00 | debug-culprit-line-after-patch |+| debug-dashboard-url | cli+skill | Claude Haiku 5.5 | trajectory | 13 | $0.00784 | 38.2 | 1.00 | |+| debug-dashboard-url | cli+skill | DeepSeek V4.1 Flash | | 15 | $0.00675 | 42.2 | 1.00 | |+| debug-dashboard-url | cli+skill | Claude Opus 5.5 | | 8 | $0.20910 | 46.9 | 0.67 | |+| debug-dashboard-url | cli-noskill | Claude Haiku 5.5 | trajectory | 10 | $0.00671 | 35.3 | 1.00 | |+| debug-dashboard-url | cli-noskill | DeepSeek V4.1 Flash | | 22 | $0.01561 | 56.8 | 1.00 | || debug-dashboard-url | cli-noskill | Claude Opus 5.5 | | 11 | $0.14578 | 52.4 | 1.00 | || debug-dashboard-url | mcp | Claude Haiku 5.5 | | 13 | $0.00862 | 36.9 | 1.00 | || debug-dashboard-url | mcp | DeepSeek V4.1 Flash | | 20 | $0.01669 | 46.6 | 1.00 | || debug-dashboard-url | mcp | Claude Opus 5.5 | | 9 | $0.30706 | 52.7 | 1.00 | |-| debug-flutter-null-check | cli+skill | Claude Haiku 5.5 | answer | 10 | $0.00702 | 35.9 | 1.00 | debug-issue-hash-optional |-| debug-flutter-null-check | cli+skill | DeepSeek V4.1 Flash | | 18 | $0.01318 | 43.6 | 1.00 | |-| debug-flutter-null-check | cli+skill | Claude Opus 5.5 | | 8 | $0.22968 | 50.0 | 1.00 | |-| debug-flutter-null-check | cli-noskill | Claude Haiku 5.5 | answer | 11 | $0.00413 | 27.0 | 0.67 | debug-issue-hash-optional |-| debug-flutter-null-check | cli-noskill | DeepSeek V4.1 Flash | grounding | 26 | $0.01496 | 53.6 | 1.00 | debug-exceptions-show-truncated |-| debug-flutter-null-check | cli-noskill | Claude Opus 5.5 | grounding | 10 | $0.21631 | 48.9 | 1.00 | debug-exceptions-show-truncated |+| debug-flutter-null-check | cli+skill | Claude Haiku 5.5 | | 10 | $0.00723 | 30.0 | 1.00 | |+| debug-flutter-null-check | cli+skill | DeepSeek V4.1 Flash | | 19 | $0.00740 | 43.6 | 1.00 | |+| debug-flutter-null-check | cli+skill | Claude Opus 5.5 | | 8 | $0.21772 | 46.2 | 1.00 | |+| debug-flutter-null-check | cli-noskill | Claude Haiku 5.5 | | 11 | $0.00413 | 27.0 | 0.67 | |+| debug-flutter-null-check | cli-noskill | DeepSeek V4.1 Flash | | 26 | $0.01496 | 53.6 | 1.00 | |+| debug-flutter-null-check | cli-noskill | Claude Opus 5.5 | | 10 | $0.21631 | 48.9 | 1.00 | || debug-flutter-null-check | mcp | Claude Haiku 5.5 | | 15 | $0.00894 | 36.1 | 1.00 | || debug-flutter-null-check | mcp | DeepSeek V4.1 Flash | | 18 | $0.01920 | 48.5 | 1.00 | || debug-flutter-null-check | mcp | Claude Opus 5.5 | | 8 | $0.25589 | 38.4 | 1.00 | |-| debug-go-nil-regression | cli+skill | Claude Haiku 5.5 | answer | 14 | $0.00748 | 42.2 | 1.00 | debug-issue-hash-optional |-| debug-go-nil-regression | cli+skill | DeepSeek V4.1 Flash | grounding | 24 | $0.01791 | 52.5 | 1.00 | debug-exceptions-show-truncated |-| debug-go-nil-regression | cli+skill | Claude Opus 5.5 | | 8 | $0.21833 | 47.4 | 1.00 | |-| debug-go-nil-regression | cli-noskill | Claude Haiku 5.5 | grounding | 9 | $0.00622 | 33.7 | 1.00 | debug-exceptions-show-truncated |-| debug-go-nil-regression | cli-noskill | DeepSeek V4.1 Flash | answer+grounding | 16 | $0.01227 | 36.3 | 1.00 | debug-culprit-line-after-patch |-| debug-go-nil-regression | cli-noskill | Claude Opus 5.5 | grounding | 10 | $0.23729 | 48.1 | 1.00 | debug-exceptions-show-truncated |+| debug-go-nil-regression | cli+skill | Claude Haiku 5.5 | | 11 | $0.00729 | 46.5 | 1.00 | |+| debug-go-nil-regression | cli+skill | DeepSeek V4.1 Flash | | 22 | $0.00775 | 43.9 | 1.00 | |+| debug-go-nil-regression | cli+skill | Claude Opus 5.5 | | 8 | $0.22580 | 50.0 | 1.00 | |+| debug-go-nil-regression | cli-noskill | Claude Haiku 5.5 | | 9 | $0.00622 | 33.7 | 1.00 | |+| debug-go-nil-regression | cli-noskill | DeepSeek V4.1 Flash | | 16 | $0.01227 | 36.3 | 1.00 | |+| debug-go-nil-regression | cli-noskill | Claude Opus 5.5 | | 10 | $0.23729 | 48.1 | 1.00 | || debug-go-nil-regression | mcp | Claude Haiku 5.5 | hidden_tests | 11 | $0.00828 | 38.3 | 1.00 | || debug-go-nil-regression | mcp | DeepSeek V4.1 Flash | | 16 | $0.01500 | 36.5 | 1.00 | || debug-go-nil-regression | mcp | Claude Opus 5.5 | | 9 | $0.30672 | 40.2 | 1.00 | |-| debug-go-task-panic | cli+skill | Claude Haiku 5.5 | answer | 9 | $0.00576 | 32.4 | 0.33 | debug-issue-hash-optional |-| debug-go-task-panic | cli+skill | DeepSeek V4.1 Flash | answer | 19 | $0.01505 | 47.4 | 0.67 | debug-culprit-line-after-patch |-| debug-go-task-panic | cli+skill | Claude Opus 5.5 | trajectory | 12 | $0.24956 | 255.5 | 0.67 | |-| debug-go-task-panic | cli-noskill | Claude Haiku 5.5 | answer+hidden_tests | 10 | $0.00651 | 42.8 | 0.67 | debug-issue-hash-optional |-| debug-go-task-panic | cli-noskill | DeepSeek V4.1 Flash | answer | 20 | $0.01473 | 97.1 | 1.00 | debug-issue-hash-optional |-| debug-go-task-panic | cli-noskill | Claude Opus 5.5 | grounding | 14 | $0.24714 | 53.0 | 0.67 | debug-exceptions-show-truncated |+| debug-go-task-panic | cli+skill | Claude Haiku 5.5 | | 10 | $0.00663 | 37.6 | 0.67 | |+| debug-go-task-panic | cli+skill | DeepSeek V4.1 Flash | | 22 | $0.00697 | 47.1 | 0.67 | |+| debug-go-task-panic | cli+skill | Claude Opus 5.5 | | 11 | $0.20621 | 53.0 | 0.67 | |+| debug-go-task-panic | cli-noskill | Claude Haiku 5.5 | hidden_tests | 10 | $0.00651 | 42.8 | 0.67 | |+| debug-go-task-panic | cli-noskill | DeepSeek V4.1 Flash | | 20 | $0.01473 | 97.1 | 1.00 | |+| debug-go-task-panic | cli-noskill | Claude Opus 5.5 | | 14 | $0.24714 | 53.0 | 0.67 | || debug-go-task-panic | mcp | Claude Haiku 5.5 | | 12 | $0.00831 | 43.4 | 0.67 | |-| debug-go-task-panic | mcp | DeepSeek V4.1 Flash | answer | 19 | $0.01467 | 46.8 | 0.67 | debug-culprit-line-after-patch |+| debug-go-task-panic | mcp | DeepSeek V4.1 Flash | | 19 | $0.01467 | 46.8 | 0.67 | || debug-go-task-panic | mcp | Claude Opus 5.5 | | 10 | $0.25975 | 51.5 | 1.00 | |-| debug-insufficient-evidence | cli+skill | Claude Haiku 5.5 | answer | 16 | $0.01343 | 83.8 | 0.67 | debug-issue-hash-optional |-| debug-insufficient-evidence | cli+skill | DeepSeek V4.1 Flash | answer+grounding | 30 | $0.03675 | 116.2 | 1.00 | |-| debug-insufficient-evidence | cli+skill | Claude Opus 5.5 | | 20 | $0.43302 | 283.1 | 1.00 | |-| debug-insufficient-evidence | cli-noskill | Claude Haiku 5.5 | answer+grounding | 29 | $0.02027 | 107.6 | 0.67 | debug-issue-hash-optional |+| debug-insufficient-evidence | cli+skill | Claude Haiku 5.5 | answer | 21 | $0.01521 | 104.6 | 0.67 | |+| debug-insufficient-evidence | cli+skill | DeepSeek V4.1 Flash | answer+hidden_tests | 31 | $0.01488 | 102.8 | 0.33 | |+| debug-insufficient-evidence | cli+skill | Claude Opus 5.5 | | 20 | $0.63316 | 209.8 | 1.00 | |+| debug-insufficient-evidence | cli-noskill | Claude Haiku 5.5 | grounding | 29 | $0.02027 | 107.6 | 0.67 | || debug-insufficient-evidence | cli-noskill | DeepSeek V4.1 Flash | answer+grounding+hidden_tests | 31 | $0.04165 | 150.7 | 0.33 | |-| debug-insufficient-evidence | cli-noskill | Claude Opus 5.5 | grounding | 29 | $0.59420 | 328.6 | 0.67 | debug-exceptions-show-truncated |+| debug-insufficient-evidence | cli-noskill | Claude Opus 5.5 | | 29 | $0.59420 | 328.6 | 0.67 | || debug-insufficient-evidence | mcp | Claude Haiku 5.5 | | 26 | $0.02115 | 96.2 | 1.00 | || debug-insufficient-evidence | mcp | DeepSeek V4.1 Flash | answer | 30 | $0.04048 | 95.2 | 0.00 | || debug-insufficient-evidence | mcp | Claude Opus 5.5 | answer+hidden_tests | 24 | $0.78355 | 150.3 | 0.00 | |-| debug-ios-force-unwrap | cli+skill | Claude Haiku 5.5 | answer | 13 | $0.00645 | 31.2 | 1.00 | debug-issue-hash-optional |-| debug-ios-force-unwrap | cli+skill | DeepSeek V4.1 Flash | grounding | 26 | $0.01899 | 50.7 | 1.00 | debug-exceptions-show-truncated |-| debug-ios-force-unwrap | cli+skill | Claude Opus 5.5 | | 8 | $0.18000 | 35.5 | 1.00 | |-| debug-ios-force-unwrap | cli-noskill | Claude Haiku 5.5 | grounding | 12 | $0.00577 | 22.1 | 1.00 | debug-exceptions-show-truncated |-| debug-ios-force-unwrap | cli-noskill | DeepSeek V4.1 Flash | grounding | 27 | $0.01485 | 220.2 | 1.00 | debug-exceptions-show-truncated |-| debug-ios-force-unwrap | cli-noskill | Claude Opus 5.5 | grounding | 11 | $0.23372 | 46.5 | 1.00 | debug-exceptions-show-truncated |+| debug-ios-force-unwrap | cli+skill | Claude Haiku 5.5 | | 14 | $0.00627 | 25.9 | 1.00 | |+| debug-ios-force-unwrap | cli+skill | DeepSeek V4.1 Flash | | 24 | $0.00971 | 58.0 | 1.00 | |+| debug-ios-force-unwrap | cli+skill | Claude Opus 5.5 | | 15 | $0.23090 | 53.5 | 1.00 | |+| debug-ios-force-unwrap | cli-noskill | Claude Haiku 5.5 | | 12 | $0.00577 | 22.1 | 1.00 | |+| debug-ios-force-unwrap | cli-noskill | DeepSeek V4.1 Flash | | 27 | $0.01485 | 220.2 | 1.00 | |+| debug-ios-force-unwrap | cli-noskill | Claude Opus 5.5 | | 11 | $0.23372 | 46.5 | 1.00 | || debug-ios-force-unwrap | mcp | Claude Haiku 5.5 | | 13 | $0.00675 | 25.7 | 1.00 | || debug-ios-force-unwrap | mcp | DeepSeek V4.1 Flash | | 19 | $0.01360 | 209.8 | 1.00 | || debug-ios-force-unwrap | mcp | Claude Opus 5.5 | | 9 | $0.19575 | 55.3 | 1.00 | |-| debug-js-contract-rename | cli+skill | Claude Haiku 5.5 | | 12 | $0.00696 | 35.6 | 1.00 | |-| debug-js-contract-rename | cli+skill | DeepSeek V4.1 Flash | | 20 | $0.01274 | 212.0 | 1.00 | |-| debug-js-contract-rename | cli+skill | Claude Opus 5.5 | | 9 | $0.19321 | 42.7 | 1.00 | |+| debug-js-contract-rename | cli+skill | Claude Haiku 5.5 | trajectory | 13 | $0.00712 | 40.6 | 0.67 | |+| debug-js-contract-rename | cli+skill | DeepSeek V4.1 Flash | | 26 | $0.00611 | 40.9 | 1.00 | |+| debug-js-contract-rename | cli+skill | Claude Opus 5.5 | | 10 | $0.18978 | 40.9 | 1.00 | || debug-js-contract-rename | cli-noskill | Claude Haiku 5.5 | trajectory | 11 | $0.00432 | 35.6 | 0.67 | |-| debug-js-contract-rename | cli-noskill | DeepSeek V4.1 Flash | trajectory+grounding | 25 | $0.01175 | 39.1 | 0.67 | |+| debug-js-contract-rename | cli-noskill | DeepSeek V4.1 Flash | trajectory | 25 | $0.01175 | 39.1 | 0.67 | || debug-js-contract-rename | cli-noskill | Claude Opus 5.5 | | 11 | $0.12316 | 42.8 | 1.00 | || debug-js-contract-rename | mcp | Claude Haiku 5.5 | | 15 | $0.00733 | 34.2 | 1.00 | || debug-js-contract-rename | mcp | DeepSeek V4.1 Flash | | 19 | $0.01344 | 42.3 | 1.00 | || debug-js-contract-rename | mcp | Claude Opus 5.5 | | 10 | $0.20837 | 44.2 | 1.00 | |-| debug-log-only-error | cli+skill | Claude Haiku 5.5 | answer | 10 | $0.00607 | 31.8 | 1.00 | |-| debug-log-only-error | cli+skill | DeepSeek V4.1 Flash | | 21 | $0.01215 | 40.2 | 1.00 | |-| debug-log-only-error | cli+skill | Claude Opus 5.5 | | 9 | $0.19606 | 33.4 | 1.00 | |+| debug-log-only-error | cli+skill | Claude Haiku 5.5 | | 12 | $0.00588 | 29.2 | 1.00 | |+| debug-log-only-error | cli+skill | DeepSeek V4.1 Flash | | 17 | $0.00493 | 30.8 | 1.00 | |+| debug-log-only-error | cli+skill | Claude Opus 5.5 | | 11 | $0.23060 | 48.0 | 1.00 | || debug-log-only-error | cli-noskill | Claude Haiku 5.5 | answer | 16 | $0.00567 | 37.1 | 1.00 | || debug-log-only-error | cli-noskill | DeepSeek V4.1 Flash | | 23 | $0.01181 | 41.9 | 1.00 | |-| debug-log-only-error | cli-noskill | Claude Opus 5.5 | grounding | 19 | $0.27564 | 72.9 | 1.00 | debug-exceptions-show-truncated |+| debug-log-only-error | cli-noskill | Claude Opus 5.5 | | 19 | $0.27564 | 72.9 | 1.00 | || debug-log-only-error | mcp | Claude Haiku 5.5 | answer | 17 | $0.00919 | 31.0 | 1.00 | || debug-log-only-error | mcp | DeepSeek V4.1 Flash | | 16 | $0.01311 | 36.8 | 1.00 | || debug-log-only-error | mcp | Claude Opus 5.5 | | 15 | $0.21080 | 37.7 | 1.00 | |-| debug-mixed-group | cli+skill | Claude Haiku 5.5 | answer | 11 | $0.01055 | 65.1 | 1.00 | debug-issue-hash-optional |-| debug-mixed-group | cli+skill | DeepSeek V4.1 Flash | grounding | 30 | $0.02886 | 85.5 | 1.00 | debug-exceptions-show-truncated |-| debug-mixed-group | cli+skill | Claude Opus 5.5 | | 7 | $0.22502 | 55.6 | 1.00 | |+| debug-mixed-group | cli+skill | Claude Haiku 5.5 | | 9 | $0.00860 | 47.9 | 1.00 | |+| debug-mixed-group | cli+skill | DeepSeek V4.1 Flash | | 18 | $0.00945 | 61.7 | 1.00 | |+| debug-mixed-group | cli+skill | Claude Opus 5.5 | | 11 | $0.26573 | 124.3 | 1.00 | || debug-mixed-group | cli-noskill | Claude Haiku 5.5 | answer | 14 | $0.01086 | 54.8 | 1.00 | |-| debug-mixed-group | cli-noskill | DeepSeek V4.1 Flash | grounding | 30 | $0.02163 | 92.7 | 1.00 | debug-exceptions-show-truncated |-| debug-mixed-group | cli-noskill | Claude Opus 5.5 | grounding | 18 | $0.35749 | 284.8 | 1.00 | debug-exceptions-show-truncated |+| debug-mixed-group | cli-noskill | DeepSeek V4.1 Flash | | 30 | $0.02163 | 92.7 | 1.00 | |+| debug-mixed-group | cli-noskill | Claude Opus 5.5 | | 18 | $0.35749 | 284.8 | 1.00 | || debug-mixed-group | mcp | Claude Haiku 5.5 | | 10 | $0.00968 | 43.5 | 1.00 | || debug-mixed-group | mcp | DeepSeek V4.1 Flash | | 24 | $0.04007 | 110.5 | 1.00 | || debug-mixed-group | mcp | Claude Opus 5.5 | | 9 | $0.32637 | 62.5 | 1.00 | |@@ -520,32 +522,32 @@ Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1,| debug-no-writes | mcp-write | Claude Haiku 5.5 | answer | 15 | $0.01384 | 44.3 | 1.00 | || debug-no-writes | mcp-write | DeepSeek V4.1 Flash | answer | 20 | $0.01998 | 51.2 | 0.67 | || debug-no-writes | mcp-write | Claude Opus 5.5 | answer | 12 | $0.30456 | 226.7 | 0.67 | |-| debug-py-keyerror-webhook | cli+skill | Claude Haiku 5.5 | answer | 12 | $0.00778 | 37.6 | 1.00 | |-| debug-py-keyerror-webhook | cli+skill | DeepSeek V4.1 Flash | | 26 | $0.01407 | 45.7 | 1.00 | |-| debug-py-keyerror-webhook | cli+skill | Claude Opus 5.5 | | 10 | $0.21655 | 50.7 | 0.67 | |+| debug-py-keyerror-webhook | cli+skill | Claude Haiku 5.5 | answer | 19 | $0.01067 | 50.9 | 1.00 | |+| debug-py-keyerror-webhook | cli+skill | DeepSeek V4.1 Flash | | 23 | $0.00757 | 46.9 | 1.00 | |+| debug-py-keyerror-webhook | cli+skill | Claude Opus 5.5 | | 14 | $0.28732 | 117.6 | 1.00 | || debug-py-keyerror-webhook | cli-noskill | Claude Haiku 5.5 | answer | 20 | $0.00822 | 48.3 | 1.00 | || debug-py-keyerror-webhook | cli-noskill | DeepSeek V4.1 Flash | | 29 | $0.01610 | 53.2 | 1.00 | || debug-py-keyerror-webhook | cli-noskill | Claude Opus 5.5 | | 18 | $0.27188 | 77.7 | 1.00 | || debug-py-keyerror-webhook | mcp | Claude Haiku 5.5 | answer | 17 | $0.01339 | 35.9 | 0.67 | || debug-py-keyerror-webhook | mcp | DeepSeek V4.1 Flash | | 24 | $0.01872 | 47.9 | 0.67 | || debug-py-keyerror-webhook | mcp | Claude Opus 5.5 | | 14 | $0.37378 | 58.1 | 1.00 | |-| debug-race-duplicate | cli+skill | Claude Haiku 5.5 | hidden_tests | 19 | $0.01101 | 68.9 | 1.00 | |-| debug-race-duplicate | cli+skill | DeepSeek V4.1 Flash | answer+grounding | 29 | $0.02163 | 71.1 | 1.00 | debug-culprit-line-after-patch |-| debug-race-duplicate | cli+skill | Claude Opus 5.5 | | 17 | $0.34428 | 82.9 | 1.00 | |+| debug-race-duplicate | cli+skill | Claude Haiku 5.5 | answer | 15 | $0.01060 | 228.5 | 1.00 | |+| debug-race-duplicate | cli+skill | DeepSeek V4.1 Flash | | 28 | $0.01082 | 70.8 | 1.00 | |+| debug-race-duplicate | cli+skill | Claude Opus 5.5 | hidden_tests | 21 | $0.46764 | 144.7 | 1.00 | || debug-race-duplicate | cli-noskill | Claude Haiku 5.5 | hidden_tests | 21 | $0.00998 | 64.5 | 1.00 | |-| debug-race-duplicate | cli-noskill | DeepSeek V4.1 Flash | answer | 29 | $0.01850 | 59.6 | 1.00 | debug-culprit-line-after-patch |+| debug-race-duplicate | cli-noskill | DeepSeek V4.1 Flash | | 29 | $0.01850 | 59.6 | 1.00 | || debug-race-duplicate | cli-noskill | Claude Opus 5.5 | | 22 | $0.32067 | 98.4 | 1.00 | |-| debug-race-duplicate | mcp | Claude Haiku 5.5 | answer+hidden_tests | 20 | $0.01019 | 49.7 | 1.00 | debug-issue-hash-optional |-| debug-race-duplicate | mcp | DeepSeek V4.1 Flash | answer | 29 | $0.01950 | 62.4 | 1.00 | debug-culprit-line-after-patch |+| debug-race-duplicate | mcp | Claude Haiku 5.5 | hidden_tests | 20 | $0.01019 | 49.7 | 1.00 | |+| debug-race-duplicate | mcp | DeepSeek V4.1 Flash | | 29 | $0.01950 | 62.4 | 1.00 | || debug-race-duplicate | mcp | Claude Opus 5.5 | hidden_tests | 16 | $0.29429 | 64.5 | 1.00 | |-| debug-timeout-downstream | cli+skill | Claude Haiku 5.5 | answer | 17 | $0.00921 | 70.8 | 1.00 | |-| debug-timeout-downstream | cli+skill | DeepSeek V4.1 Flash | answer | 28 | $0.01718 | 57.4 | 1.00 | debug-culprit-line-after-patch |-| debug-timeout-downstream | cli+skill | Claude Opus 5.5 | | 10 | $0.23222 | 61.9 | 1.00 | |+| debug-timeout-downstream | cli+skill | Claude Haiku 5.5 | answer | 15 | $0.00747 | 218.9 | 1.00 | |+| debug-timeout-downstream | cli+skill | DeepSeek V4.1 Flash | | 24 | $0.01158 | 70.0 | 1.00 | |+| debug-timeout-downstream | cli+skill | Claude Opus 5.5 | answer | 9 | $0.23267 | 53.6 | 1.00 | || debug-timeout-downstream | cli-noskill | Claude Haiku 5.5 | answer | 17 | $0.00601 | 53.1 | 0.67 | || debug-timeout-downstream | cli-noskill | DeepSeek V4.1 Flash | | 27 | $0.01795 | 57.4 | 1.00 | || debug-timeout-downstream | cli-noskill | Claude Opus 5.5 | answer | 16 | $0.17592 | 57.9 | 1.00 | || debug-timeout-downstream | mcp | Claude Haiku 5.5 | | 12 | $0.01247 | 50.8 | 1.00 | |-| debug-timeout-downstream | mcp | DeepSeek V4.1 Flash | answer | 28 | $0.03610 | 67.1 | 1.00 | debug-culprit-line-after-patch |+| debug-timeout-downstream | mcp | DeepSeek V4.1 Flash | | 28 | $0.03610 | 67.1 | 1.00 | || debug-timeout-downstream | mcp | Claude Opus 5.5 | | 16 | $0.41916 | 65.3 | 1.00 | || perf-app-cpu-serialization | cli+skill | Claude Haiku 5.5 | | 15 | $0.01503 | 85.1 | 0.33 | || perf-app-cpu-serialization | cli+skill | DeepSeek V4.1 Flash | | 31 | $0.01231 | 81.9 | 0.67 | |@@ -579,7 +581,7 @@ Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1,| perf-app-quadratic | cli+skill | Claude Opus 5.5 | | 14 | $0.33868 | 68.2 | 0.67 | || perf-app-quadratic | cli-noskill | Claude Haiku 5.5 | grounding | 20 | $0.00812 | 53.8 | 0.67 | || perf-app-quadratic | cli-noskill | DeepSeek V4.1 Flash | | 20 | $0.00668 | 48.9 | 0.67 | |-| perf-app-quadratic | cli-noskill | Claude Opus 5.5 | grounding | 9 | $0.21189 | 46.9 | 0.33 | |+| perf-app-quadratic | cli-noskill | Claude Opus 5.5 | | 9 | $0.21189 | 46.9 | 0.33 | || perf-app-quadratic | mcp | Claude Haiku 5.5 | | 14 | $0.00983 | 44.2 | 0.67 | || perf-app-quadratic | mcp | DeepSeek V4.1 Flash | | 21 | $0.01315 | 72.8 | 0.67 | || perf-app-quadratic | mcp | Claude Opus 5.5 | | 12 | $0.29217 | 46.7 | 0.67 | |@@ -597,7 +599,7 @@ Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1,| perf-cache-cold-start | cli+skill | Claude Opus 5.5 | | 15 | $0.34653 | 82.4 | 1.00 | || perf-cache-cold-start | cli-noskill | Claude Haiku 5.5 | | 23 | $0.01016 | 68.3 | 0.33 | || perf-cache-cold-start | cli-noskill | DeepSeek V4.1 Flash | | 28 | $0.01200 | 66.1 | 1.00 | |-| perf-cache-cold-start | cli-noskill | Claude Opus 5.5 | grounding | 18 | $0.38983 | 95.9 | 1.00 | |+| perf-cache-cold-start | cli-noskill | Claude Opus 5.5 | | 18 | $0.38983 | 95.9 | 1.00 | || perf-cache-cold-start | mcp | Claude Haiku 5.5 | | 19 | $0.01683 | 70.4 | 0.33 | || perf-cache-cold-start | mcp | DeepSeek V4.1 Flash | grounding | 25 | $0.02134 | 92.8 | 0.67 | || perf-cache-cold-start | mcp | Claude Opus 5.5 | | 18 | $0.51514 | 84.1 | 1.00 | |@@ -677,7 +679,7 @@ Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1,| perf-db-expensive-join | cli+skill | DeepSeek V4.1 Flash | | 19 | $0.01063 | 58.8 | 1.00 | || perf-db-expensive-join | cli+skill | Claude Opus 5.5 | | 9 | $0.23874 | 54.4 | 1.00 | || perf-db-expensive-join | cli-noskill | Claude Haiku 5.5 | | 16 | $0.00635 | 44.1 | 1.00 | |-| perf-db-expensive-join | cli-noskill | DeepSeek V4.1 Flash | grounding | 24 | $0.01149 | 68.5 | 1.00 | |+| perf-db-expensive-join | cli-noskill | DeepSeek V4.1 Flash | | 24 | $0.01149 | 68.5 | 1.00 | || perf-db-expensive-join | cli-noskill | Claude Opus 5.5 | | 12 | $0.24249 | 66.0 | 1.00 | || perf-db-expensive-join | mcp | Claude Haiku 5.5 | | 11 | $0.00870 | 38.6 | 1.00 | || perf-db-expensive-join | mcp | DeepSeek V4.1 Flash | | 19 | $0.01299 | 69.2 | 1.00 | |@@ -715,16 +717,16 @@ Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1,| perf-db-n-plus-1 | cli-noskill | Claude Haiku 5.5 | | 10 | $0.00588 | 35.4 | 0.67 | || perf-db-n-plus-1 | cli-noskill | DeepSeek V4.1 Flash | | 26 | $0.00943 | 72.4 | 0.67 | || perf-db-n-plus-1 | cli-noskill | Claude Opus 5.5 | | 13 | $0.20943 | 61.7 | 0.67 | |-| perf-db-n-plus-1 | mcp | Claude Haiku 5.5 | grounding | 14 | $0.06958 | 50.3 | 0.67 | |+| perf-db-n-plus-1 | mcp | Claude Haiku 5.5 | | 14 | $0.06958 | 50.3 | 0.67 | || perf-db-n-plus-1 | mcp | DeepSeek V4.1 Flash | | 15 | $0.01259 | 45.7 | 1.00 | || perf-db-n-plus-1 | mcp | Claude Opus 5.5 | | 10 | $0.38496 | 47.5 | 0.67 | || perf-db-over-fetching | cli+skill | Claude Haiku 5.5 | | 9 | $0.00857 | 40.3 | 0.67 | || perf-db-over-fetching | cli+skill | DeepSeek V4.1 Flash | | 26 | $0.01092 | 64.6 | 1.00 | || perf-db-over-fetching | cli+skill | Claude Opus 5.5 | | 15 | $0.31148 | 79.2 | 0.67 | || perf-db-over-fetching | cli-noskill | Claude Haiku 5.5 | | 17 | $0.00755 | 52.0 | 0.67 | |-| perf-db-over-fetching | cli-noskill | DeepSeek V4.1 Flash | grounding | 30 | $0.00940 | 51.3 | 1.00 | |+| perf-db-over-fetching | cli-noskill | DeepSeek V4.1 Flash | | 30 | $0.00940 | 51.3 | 1.00 | || perf-db-over-fetching | cli-noskill | Claude Opus 5.5 | | 15 | $0.21028 | 80.7 | 1.00 | |-| perf-db-over-fetching | mcp | Claude Haiku 5.5 | grounding | 12 | $0.01724 | 48.1 | 0.67 | |+| perf-db-over-fetching | mcp | Claude Haiku 5.5 | | 12 | $0.01724 | 48.1 | 0.67 | || perf-db-over-fetching | mcp | DeepSeek V4.1 Flash | | 24 | $0.01334 | 63.2 | 1.00 | || perf-db-over-fetching | mcp | Claude Opus 5.5 | | 15 | $0.43795 | 67.7 | 1.00 | || perf-db-pool-exhaustion | cli+skill | Claude Haiku 5.5 | | 9 | $0.01103 | 58.2 | 1.00 | |@@ -787,9 +789,9 @@ Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1,| perf-net-chatty-fanout | cli-noskill | Claude Haiku 5.5 | grounding | 12 | $0.00711 | 53.7 | 1.00 | chatty-fanout-grounding || perf-net-chatty-fanout | cli-noskill | DeepSeek V4.1 Flash | grounding | 24 | $0.01154 | 63.9 | 1.00 | chatty-fanout-grounding || perf-net-chatty-fanout | cli-noskill | Claude Opus 5.5 | grounding | 16 | $0.22570 | 83.0 | 1.00 | chatty-fanout-grounding |-| perf-net-chatty-fanout | mcp | Claude Haiku 5.5 | grounding | 15 | $0.06257 | 65.1 | 1.00 | chatty-fanout-grounding |-| perf-net-chatty-fanout | mcp | DeepSeek V4.1 Flash | grounding | 18 | $0.01899 | 68.8 | 1.00 | chatty-fanout-grounding |-| perf-net-chatty-fanout | mcp | Claude Opus 5.5 | grounding | 10 | $0.62898 | 57.2 | 1.00 | chatty-fanout-grounding |+| perf-net-chatty-fanout | mcp | Claude Haiku 5.5 | | 15 | $0.06257 | 65.1 | 1.00 | |+| perf-net-chatty-fanout | mcp | DeepSeek V4.1 Flash | | 18 | $0.01899 | 68.8 | 1.00 | |+| perf-net-chatty-fanout | mcp | Claude Opus 5.5 | | 10 | $0.62898 | 57.2 | 1.00 | || perf-net-large-payload | cli+skill | Claude Haiku 5.5 | | 13 | $0.00900 | 50.7 | 0.67 | || perf-net-large-payload | cli+skill | DeepSeek V4.1 Flash | | 22 | $0.01193 | 68.5 | 1.00 | || perf-net-large-payload | cli+skill | Claude Opus 5.5 | | 14 | $0.32918 | 83.4 | 1.00 | |@@ -838,8 +840,8 @@ Every run's failed graders, tool calls, agent cost, wall time, judge score (0-1,| perf-xs-android-timeouts | cli+skill | Claude Haiku 5.5 | | 21 | $0.01327 | 75.8 | 1.00 | || perf-xs-android-timeouts | cli+skill | DeepSeek V4.1 Flash | | 28 | $0.01149 | 68.8 | 1.00 | || perf-xs-android-timeouts | cli+skill | Claude Opus 5.5 | | 17 | $0.45062 | 108.6 | 1.00 | |-| perf-xs-android-timeouts | cli-noskill | Claude Haiku 5.5 | grounding | 29 | $0.02029 | 102.0 | 0.67 | |-| perf-xs-android-timeouts | cli-noskill | DeepSeek V4.1 Flash | grounding | 31 | $0.00985 | 62.3 | 0.67 | |+| perf-xs-android-timeouts | cli-noskill | Claude Haiku 5.5 | | 29 | $0.02029 | 102.0 | 0.67 | |+| perf-xs-android-timeouts | cli-noskill | DeepSeek V4.1 Flash | | 31 | $0.00985 | 62.3 | 0.67 | || perf-xs-android-timeouts | cli-noskill | Claude Opus 5.5 | grounding | 17 | $0.46912 | 98.6 | 1.00 | || perf-xs-android-timeouts | mcp | Claude Haiku 5.5 | | 23 | $0.01540 | 63.6 | 1.00 | || perf-xs-android-timeouts | mcp | DeepSeek V4.1 Flash | | 25 | $0.01500 | 61.0 | 1.00 | |diff --git a/website/lib/benchmarks.ts b/website/lib/benchmarks.tsindex 092ef055..0698882c 100644--- a/website/lib/benchmarks.ts+++ b/website/lib/benchmarks.ts@@ -92,7 +92,7 @@ export type Group = {// One batch of runs: which suites ran on which day at which commit. A page// built from one run has one part.-export type RunPart = { suites: string[]; commit: string; date: string };+export type RunPart = { suites: string[]; arms?: string[]; commit: string; date: string };export type BenchmarkData = {run: {@@ -100,6 +100,9 @@ export type BenchmarkData = {date: string;commit: string;parts: RunPart[];+ // The date the stored transcripts were last re-scored, when the page's+ // runs were graded after a grader fix rather than at their own run time.+ regraded?: string;cliVersion: string;backendVersion: string;harnessUrl: string;@@ -400,9 +403,14 @@ export function parseBenchmark(source: string): BenchmarkData {}));const totalRuns = models.reduce((n, m) => n + m.totals.runs, 0);- const fmParts = (fm.parts ?? []) as { suites: string; commit: string; date: string }[];+ const fmParts = (fm.parts ?? []) as { suites: string; arms?: string; commit: string; date: string }[];const parts: RunPart[] = fmParts.length- ? fmParts.map((p) => ({ suites: String(p.suites).split(","), commit: String(p.commit), date: String(p.date) }))+ ? fmParts.map((p) => ({+ suites: String(p.suites).split(","),+ ...(p.arms ? { arms: String(p.arms).split(",") } : {}),+ commit: String(p.commit),+ date: String(p.date),+ })): [{ suites: (fm.suites as { id: string }[]).map((s) => s.id), commit: String(fm.commit), date: String(fm.date) }];return {run: {@@ -410,6 +418,7 @@ export function parseBenchmark(source: string): BenchmarkData {date: String(fm.date),commit: String(fm.commit),parts,+ ...(fm.regraded ? { regraded: String(fm.regraded) } : {}),cliVersion: String(fm.cliVersion),backendVersion: String(fm.backendVersion),harnessUrl: String(fm.harnessUrl),diff --git a/website/scripts/build-benchmarks-data.mjs b/website/scripts/build-benchmarks-data.mjsindex 0c005ee5..e58b9413 100644--- a/website/scripts/build-benchmarks-data.mjs+++ b/website/scripts/build-benchmarks-data.mjs@@ -282,37 +282,9 @@ const KNOWN_ISSUES = [case: "perf-net-chatty-fanout",title: "The fan-out case fails grounding for every model",detail:- "Every model fails this case's grounding check on most setups, mostly because the output it cites was truncated.",+ "Every model fails this case's grounding check on most setups: the case wants the two fanned-out routes, GET /stock/{sku} and GET /prices/{sku}, in the output of a cited call, and the models cite the aggregate view that shows the fan-out without those route names.",matches: (r) => r.graders.includes("grounding"),},- {- id: "debug-issue-hash-optional",- kind: "harness",- case: "debug-*",- title: "The debug answer schema calls the issue hash optional, the grader does not",- detail:- "The submit tool describes issue_hash as the group hash \"if the bug has one\" and does not require it, so a model may leave it out even when it has quoted the hash in its evidence. The answer grader fails the run on that empty field alone, whatever the diagnosis and the fix.",- matches: (r) => r.firstFailure.includes('issue_hash ""'),- },- {- id: "debug-exceptions-show-truncated",- kind: "skill",- case: "debug-*",- title: "The first command the skill recommends is too long to cite",- detail:- "The skill opens a debug task with traceway exceptions show <hash>, and for these groups its output (JSON or not, with or without head) runs past the 30,000 characters the sandbox shell returns. The grounding grader rejects any evidence that cites a truncated call, so a run that read the right stack trace there and fixed the bug still fails.",- matches: (r) =>- r.graders.includes("grounding") && r.firstFailure.includes("whose output was truncated"),- },- {- id: "debug-culprit-line-after-patch",- kind: "harness",- case: "debug-*",- title: "Culprit lines are graded against the file before the patch",- detail:- "The submit schema asks for the culprit's line number without saying which version of the file. A model that inserts a guard or a comment above the faulty statement and then reports where that statement now sits lands outside the grader's tolerance of a few lines, even when the hidden tests pass on its fix.",- matches: (r) => /\bculprit\.line [1-9]\d*, want \d+±\d+$/.test(r.firstFailure),- },];const KIND_BY_FAILURE = {@@ -340,24 +312,45 @@ function parseArgs(argv) {// A part is one batch of runs: `--part perf,write:3b00b083:2026-10-21`. A// merged results.json keeps only its first part's commit and date, so the-// batches are named on the command line and written into the page.-function parseParts(values, suites, header) {+// batches are named on the command line and written into the page. A part+// may be narrowed to the setups that were run again after a fix:+// `--part debug@cli+skill:1a2b3c4d:2026-10-22` says the debug suite's+// cli+skill runs come from that later batch, and the suite's other setups+// from the part that names the suite alone.+function parseParts(values, suites, header, runs) {if (!values) return [{ suites: suites.map((s) => s.id), commit: header.gitSha.slice(0, 8), date: header.started.slice(0, 10) }];const parts = values.map((v) => {- const m = v.match(/^([\w,-]+):([0-9a-f]{7,40}):(\d{4}-\d{2}-\d{2})$/);- if (!m) throw new Error(`--part ${v}: want <suites>:<commit>:<YYYY-MM-DD>`);- return { suites: m[1].split(","), commit: m[2].slice(0, 8), date: m[3] };+ const m = v.match(/^([\w,-]+)(?:@([\w+,-]+))?:([0-9a-f]{7,40}):(\d{4}-\d{2}-\d{2})$/);+ if (!m) throw new Error(`--part ${v}: want <suites>[@<arms>]:<commit>:<YYYY-MM-DD>`);+ const part = { suites: m[1].split(",") };+ if (m[2]) part.arms = m[2].split(",");+ part.commit = m[3].slice(0, 8);+ part.date = m[4];+ return part;});for (const s of suites) {- const owners = parts.filter((p) => p.suites.includes(s.id));- if (owners.length !== 1) throw new Error(`suite ${s.id} must be in exactly one --part, found ${owners.length}`);+ const owners = parts.filter((p) => !p.arms && p.suites.includes(s.id));+ if (owners.length !== 1) throw new Error(`suite ${s.id} must be in exactly one --part without arms, found ${owners.length}`);}for (const p of parts)- for (const id of p.suites)+ for (const id of p.suites) {if (!suites.some((s) => s.id === id)) throw new Error(`--part names suite ${id}, which has no runs`);+ for (const arm of p.arms ?? [])+ if (!runs.some((r) => r.suite === id && r.arm === arm)) throw new Error(`--part names ${id}@${arm}, which has no runs`);+ }return parts;}+// partText names a part in prose: "debug on 2026-10-22 at commit 21bbe026",+// or for a part narrowed to the setups run again after a skill fix, "the+// debug CLI + skill runs again on 2026-10-22, with the skill fix on top of+// commit 21bbe026" (the fix is what the rerun measures, so it is named).+function partText(p, setups) {+ if (!p.arms) return `${joinWords(p.suites)} on ${p.date} at commit \`${p.commit}\``;+ const labels = p.arms.map((a) => setups.find((x) => x.suite === p.suites[0] && x.arm === a)?.label ?? a);+ return `the ${joinWords(p.suites)} ${joinWords(labels)} runs again on ${p.date}, with the skill fix on top of commit \`${p.commit}\``;+}+function parseCsv(text) {const rows = [];let row = [];@@ -542,7 +535,10 @@ function main() {const header = results.header;const runDate = header.started.slice(0, 10);const commit = header.gitSha.slice(0, 8);- const parts = parseParts(args.part, activeSuites, header);+ const parts = parseParts(args.part, activeSuites, header, runs);+ // Set when the stored transcripts were re-scored after a grader change+ // (twevals regrade); the page says so instead of comparing old and new graders.+ const regraded = header.regraded ? header.regraded.slice(0, 10) : null;const providersByModel = {};const modelIdByKey = {};@@ -725,6 +721,7 @@ function main() {date: runDate,commit,parts,+ regraded,cliVersion: header.cliVersion,backendVersion: header.backendVersion,runsPerCase: Math.max(...results.runs.map((r) => r.runIdx)),@@ -891,7 +888,8 @@ function renderMarkdown(data) {title: run.title,date: run.date,commit: run.commit,- parts: run.parts.map((p) => ({ suites: p.suites.join(","), commit: p.commit, date: p.date })),+ parts: run.parts.map((p) => ({ suites: p.suites.join(","), ...(p.arms ? { arms: p.arms.join(",") } : {}), commit: p.commit, date: p.date })),+ ...(run.regraded ? { regraded: run.regraded } : {}),cliVersion: run.cliVersion,backendVersion: run.backendVersion,harnessUrl: run.harnessUrl,@@ -938,10 +936,11 @@ function renderMarkdown(data) {const partsText =run.parts.length > 1- ? ` in ${words[run.parts.length]} batches: ${run.parts- .map((p) => `${joinWords(p.suites)} on ${p.date} at commit \`${p.commit}\``)- .join(", ")}`+ ? ` in ${words[run.parts.length]} batches: ${run.parts.map((p) => partText(p, setups)).join(", ")}`: ` on ${run.date}, at commit \`${run.commit}\``;+ const regradedText = run.regraded+ ? ` Every run was scored by the graders as they stood on ${run.regraded}: after a grader fix the saved transcripts are re-scored, so no number here compares two graders.`+ : "";const metricRows = [[COST_METRICS.spend, ...models.map((m) => usd(m.totals.costUsd, 2))],@@ -1000,7 +999,7 @@ ${yamlBlock(front)}# ${run.title}-${words[models.length][0].toUpperCase()}${words[models.length].slice(1)} models, ${modelList}, each ran the same ${run.runsPerModel} eval runs${partsText}. Every run gets a fresh Traceway backend seeded with the telemetry of a system broken in one specific way, and deterministic code grades the result. [How we measure](${run.methodologyUrl}) · [the harness and the cases](${run.harnessUrl}).+${words[models.length][0].toUpperCase()}${words[models.length].slice(1)} models, ${modelList}, each ran the same ${run.runsPerModel} eval runs${partsText}. Every run gets a fresh Traceway backend seeded with the telemetry of a system broken in one specific way, and deterministic code grades the result.${regradedText} [How we measure](${run.methodologyUrl}) · [the harness and the cases](${run.harnessUrl}).This file is generated by \`website/scripts/build-benchmarks-data.mjs\` from the eval results and is the data behind ${PAGE_URL}. Change the generator, not this file.@@ -1102,6 +1101,7 @@ function parseBack(data, markdown, summaryMd) {expect("total runs", parsed.run.totalRuns, data.run.totalRuns);expect("commit", parsed.run.commit, data.run.commit);expect("parts", JSON.stringify(parsed.run.parts), JSON.stringify(data.run.parts));+ expect("regraded", parsed.run.regraded ?? null, data.run.regraded);expect("all graded", parsed.run.allGraded, (data.run.statuses.pass ?? 0) + (data.run.statuses.fail ?? 0) === data.run.totalRuns);for (const m of data.models) {const p = parsed.models.find((x) => x.key === m.key);