The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
Fixes
narrow-adventure866f0f295138
diff --git a/cli/test/evals/README.md b/cli/test/evals/README.mdindex 356b459c..e1ae5e4d 100644--- a/cli/test/evals/README.md+++ b/cli/test/evals/README.md@@ -32,12 +32,13 @@ commands:- `twevals list [--preset P] [--cases GLOB] [--arms A]` prints the case × armcombinations a selection picks.-- `twevals oracle [--all | <case-id>...] [--arms A] [-j N] [--keep]` plays- the oracles and writes `results.json` and `summary.md`. It exits `1` when- an oracle fails.+- `twevals oracle [--all | <case-id>... | --suites S] [--arms A] [-j N]+ [--keep] [--out DIR]` plays the oracles and writes `results.json` and+ `summary.md` to `--out` (default `.cache/oracle/<timestamp>/`). It exits `1`+ when an oracle fails.-Presets (`presets.yaml`): `smoke` is the starred cases on the primary arm,-`core` every case on the primary arm, `full` every arm a case lists.+Presets (`presets.yaml`): `smoke` is the cases tagged `smoke` on the primary+arm, `core` every case on the primary arm, `full` every arm a case lists.Requirements: Go (the `go`/`toolchain` lines of `go.mod`) and git. Theharness builds the backend and both CLI editions from this checkout into@@ -48,7 +49,8 @@ harness builds the backend and both CLI editions from this checkout intoA run is `pass`, `fail`, `budget_exhausted` (counted as a fail) or`infra_error` (excluded from pass rates).-Each run directory, `runs/<case>/<arm>/<model>/r<N>/`, holds:+Each run directory, `runs/<case>/<arm>/<model>/r<N>/` (with `+` in the arm+written as `_`, so `cli_skill`), holds:- `transcript.jsonl`: every model response, every tool call with its id, op,argv, exit code and output;@@ -146,9 +148,12 @@ func seedNPlusOne(s *seed.Scenario) error {Backend facts that bite (each one cost a case author an afternoon):-- Health-check routes (`/health`, `/healthz`, `/ping`, `/ready`, ...) are- dropped at ingest, so an endpoint seeded on them never appears and `Verify`- fails with "endpoints never became visible".+- Successful `GET`/`HEAD` requests to health-check routes (`/health`,+ `/healthz`, `/ping`, `/ready`, `/status`, `/up`, ...; the list is in+ `backend/app/services/healthcheck.service.go`) are dropped at ingest, which+ is the projects' default. An endpoint seeded on one never appears and+ `Verify` fails with "endpoints never became visible"; a failing one (status+ 400 or more) is kept.- `endpoints list --order-by p95|lastSeen` does not sort: the CLI sends`p95`/`lastSeen`, the SQLite/DuckDB repositories only know`p95_duration`/`last_seen`, and the list falls back to impact order. Keep@@ -176,7 +181,8 @@ Backend facts that bite (each one cost a case author an afternoon):See `perf-db-n-plus-1.yaml` for the full shape. The keys:- `arms` and `invoke`: `slash` preloads SKILL.md the way a `/traceway`- command does, and `natural` makes the agent load the skill itself.+ command does, and `natural` makes the agent load the skill itself. A slash+ case's prompt starts with `/traceway`, and only a slash case's does.- `workspace.repo`: a fixture under `testdata/repos`. Fixtures can carryhistory: a `history.yaml` whose commits are overlays under `.history/`(only the files that differ from the fixture, plus an `absent:` list of@@ -229,6 +235,8 @@ reached` or Postgres's `deadlock detected`, and neutral names.```models.yaml presets.yamlcmd/twevals/ list | oracle+internal/catalog/ the case schema, presets, models.yaml+internal/checklist/ the bottleneck rows of performance.md, the perf answer enuminternal/env/ backend subprocess, registration, PATs, CLI logins, the stubinternal/seed/ the DSL, OTLP and /api/report writers, read-back verificationinternal/scenarios/ one seed function per case@@ -236,15 +244,19 @@ internal/agent/ the Chat Completions client (loopback only), the loop, printernal/tools/ the sandboxed shell (mvdan.cc/sh, in-process coreutils, gojq, goawk), files, skills, ask_user, submitinternal/ops/ CLI → operation mapping, cross-checked against tool-map.mdinternal/grade/ answer, trajectory, grounding, report, leakage+internal/record/ the run record: calls, operations, usageinternal/runner/ one run end to endinternal/report/ results.json and summary.md+internal/suite/ catalog-wide testscases/ testdata/ cases, fixture repos, oracle files```The agent's shell is an interpreter, never the host shell. It has the`traceway` binary (logged in with a read PAT whose config lives outside theworkspace), `jq`, `awk`, read-only text utilities confined to the workspace,-and read-only `git`. Tool subprocesses get a scrubbed environment (`HOME`,+and read-only `git`. No tool writes under a `.git` directory, git always+uses the workspace's repository, and git refuses arguments that name a file+outside the workspace or options that write files or run programs. Tool subprocesses get a scrubbed environment (`HOME`,`XDG_*`, `PATH`, `TZ`, `LANG`), so secrets in the harness's environmentnever reach them.@@ -252,8 +264,9 @@ never reach them.The first performance scenarios were contributed by Liyu in PR #405("feat(skills): add performance skill evals"). They were ported to this-harness as `perf-db-n-plus-1`, `perf-db-missing-index`, `perf-net-sequential`,+module as `perf-db-n-plus-1`, `perf-db-missing-index`, `perf-net-sequential`,`perf-db-pool-exhaustion`, `perf-app-sync-background`, `perf-net-large-payload`,`perf-cache-no-cache`, `perf-infra-cpu-saturation`, `perf-cache-stampede`,`perf-xs-ios-watchdog`, `perf-xs-android-timeouts`, `debug-android-npe` and-`debug-flutter-null-check`.+`debug-flutter-null-check`. The two debug cases sit in `cases/debug/` and are+not loaded yet: the harness runs the perf suite only.diff --git a/cli/test/evals/cmd/twevals/commands.go b/cli/test/evals/cmd/twevals/commands.goindex a8897e35..44772507 100644--- a/cli/test/evals/cmd/twevals/commands.go+++ b/cli/test/evals/cmd/twevals/commands.go@@ -84,13 +84,6 @@ func (s *selFlags) bind(fs *flag.FlagSet) {fs.StringVar(&s.sel.Arms, "arms", "", "comma-separated arms")}-func (s *selFlags) selection() catalog.Selection {- if s.sel.Preset == "" && s.sel.Cases == "" && s.sel.Suites == "" && s.sel.Tags == "" {- s.sel.Preset = "smoke"- }- return s.sel-}-func cmdList(args []string) error {fs := flag.NewFlagSet("list", flag.ExitOnError)var sf selFlags@@ -158,6 +151,7 @@ func cmdOracle(ctx context.Context, args []string) error {Runs: 1, Jobs: *jobs, Keep: *keep, Log: os.Stderr}header := report.NewHeader(ws.repo, filepath.Base(outDir), ws.models, bins.Hash, os.Args)header.Oracle = true+ header.Models = map[string]string{oracleModel.Key: oracleModel.ID}results := runner.Execute(ctx, opt, runner.Jobs(units, opt.Models, 1))header.Finished = time.Now().UTC()if err := report.Save(outDir, &report.Results{Header: header, Runs: results}); err != nil {diff --git a/cli/test/evals/cmd/twevals/main.go b/cli/test/evals/cmd/twevals/main.goindex 71e01b66..e0f4eca5 100644--- a/cli/test/evals/cmd/twevals/main.go+++ b/cli/test/evals/cmd/twevals/main.go@@ -14,10 +14,12 @@ import (const usage = `twevals: Traceway skill and MCP evals (oracle runs only, free)twevals list [--preset P] [--suites S] [--tags T] [--cases GLOB] [--arms A]- twevals oracle [--all | <case-id>...] [--arms A] [-j N] [--keep]+ twevals oracle [--all | <case-id>... | --suites S] [--arms A] [-j N] [--keep] [--out DIR]Selection flags: --preset smoke|core|full, --cases (comma-separated ids or globs; overrides the preset), --suites, --tags, --arms.-Exit codes: 0 ok, 1 failed oracle.+list defaults to --preset full. oracle writes results.json and summary.md to --out+(default .cache/oracle/<timestamp>).+Exit codes: 0 ok, 1 a failed oracle or an error, 2 bad usage.`func main() {@@ -38,7 +40,7 @@ func main() {fmt.Print(usage)returndefault:- err = fmt.Errorf("unknown command %q", os.Args[1])+ err = &exitError{2, fmt.Sprintf("unknown command %q (twevals help lists them)", os.Args[1])}}if err != nil {var ex *exitErrordiff --git a/cli/test/evals/internal/agent/agent_test.go b/cli/test/evals/internal/agent/agent_test.goindex 6e0ff134..76ed7d53 100644--- a/cli/test/evals/internal/agent/agent_test.go+++ b/cli/test/evals/internal/agent/agent_test.go@@ -62,18 +62,23 @@ func (echoTool) Call(context.Context, json.RawMessage) tools.Result {return tools.Result{Output: "echoed", OK: true}}+// submission is a valid submit_perf_diagnosis payload whose report is r.+func submission(r string) string {+ return `{"checklist_item":"none_found","locus":{},"evidence":[{"call_id":"c01","claim":"x"}],"fix":"none","confidence":"low","report":"` + r + `"}`+}+func cfg(s *scripted) Config {return Config{- Model: &catalog.Model{Key: "m", ID: "x/y", Caching: "anthropic"}, Chat: s, System: "sys", Prompt: "do it", SubmitTool: tools.Finish,- Tools: []tools.Tool{echoTool{}, &tools.Submit{Kind: tools.Finish, MaxRepairs: 2}},+ Model: &catalog.Model{Key: "m", ID: "x/y", Caching: "anthropic"}, Chat: s, System: "sys", Prompt: "do it", SubmitTool: tools.SubmitPerf,+ Tools: []tools.Tool{echoTool{}, &tools.Submit{Kind: tools.SubmitPerf, MaxRepairs: 2}},Budget: catalog.Budget{MaxTurns: 5, MaxToolCalls: 3},}}func TestLoopSubmits(t *testing.T) {- s := &scripted{resps: []*Response{toolResp("echo", `{}`, 0.01), toolResp("finish", `{"summary":"done","changes":[]}`, 0.01)}}+ s := &scripted{resps: []*Response{toolResp("echo", `{}`, 0.01), toolResp(tools.SubmitPerf, submission("done"), 0.01)}}run := Run(context.Background(), cfg(s))- if run.StopReason != StopSubmitted || run.Submission["summary"] != "done" || len(run.Calls) != 2 || run.Calls[0].ID != "c01" {+ if run.StopReason != StopSubmitted || run.Submission["report"] != "done" || len(run.Calls) != 2 || run.Calls[0].ID != "c01" {t.Fatalf("run = %+v", run)}if run.Usage.CostUSD != 0.02 {@@ -104,7 +109,7 @@ func TestLoopNudgesOnceThenStops(t *testing.T) {t.Fatalf("stop %s after %d requests", run.StopReason, len(s.reqs))}last := s.reqs[1].Messages- if !strings.Contains(textOf(last[len(last)-1]), "call finish") {+ if !strings.Contains(textOf(last[len(last)-1]), "call "+tools.SubmitPerf) {t.Fatal("the nudge must name the submit tool")}}@@ -140,7 +145,7 @@ func TestLoopBudgets(t *testing.T) {}func TestUnknownToolIsAnError(t *testing.T) {- s := &scripted{resps: []*Response{toolResp("nope", `{}`, 0), toolResp("finish", `{"summary":"x","changes":[]}`, 0)}}+ s := &scripted{resps: []*Response{toolResp("nope", `{}`, 0), toolResp(tools.SubmitPerf, submission("x"), 0)}}run := Run(context.Background(), cfg(s))if run.Calls[0].OK || !strings.Contains(run.Calls[0].Output, "no tool named") {t.Fatalf("%+v", run.Calls[0])@@ -193,8 +198,31 @@ func TestPromptHelpers(t *testing.T) {if SlashSkill("/traceway debug x") != "traceway" || SlashSkill("why is it slow") != "" {t.Fatal("SlashSkill")}- p := SystemPrompt(PromptInput{SubmitTool: "finish", Skills: []tools.Skill{{Name: "traceway", Description: "d"}}})- if !strings.Contains(p, "- traceway: d") || !strings.Contains(p, "call finish") {+ p := SystemPrompt(PromptInput{SubmitTool: tools.SubmitPerf, Skills: []tools.Skill{{Name: "traceway", Description: "d"}}})+ if !strings.Contains(p, "- traceway: d") || !strings.Contains(p, "call "+tools.SubmitPerf) {t.Fatal(p)}}++// TestFakeRuntimeTemplates: numbers looked up from a tool result keep the+// text they had there, at any size.+func TestFakeRuntimeTemplates(t *testing.T) {+ msgs := []Message{{Role: "tool", Content: "[call c01]\n{\"d\":45200000,\"t\":1760000000123456789,\"f\":0.25,\"o\":{\"a\":[1,2]}}\n[budget: 3 tool calls and 4 turns left]"}}+ got, err := resolveRuntime("--min {{last.d}} --at {{calls.c01.t}} --r {{last.f}} {{last.o}}", msgs)+ if err != nil || got != `--min 45200000 --at 1760000000123456789 --r 0.25 {"a":[1,2]}` {+ t.Fatalf("got %q, %v", got, err)+ }+ args, err := resolveArgs(map[string]any{"n": "{{last.d}}"}, msgs)+ if raw, _ := json.Marshal(args); err != nil || string(raw) != `{"n":45200000}` {+ t.Fatalf("whole-string template: %s, %v", raw, err)+ }+}++func TestFakeSubmitTool(t *testing.T) {+ if got := submitTool(map[string]bool{"shell": true, tools.SubmitPerf: true, "read_file": true}); got != tools.SubmitPerf {+ t.Fatalf("submitTool = %q", got)+ }+ if got := submitTool(map[string]bool{"shell": true}); got != "" {+ t.Fatalf("submitTool without one = %q", got)+ }+}diff --git a/cli/test/evals/internal/agent/fake.go b/cli/test/evals/internal/agent/fake.goindex 85556549..41bd0e6c 100644--- a/cli/test/evals/internal/agent/fake.go+++ b/cli/test/evals/internal/agent/fake.go@@ -3,6 +3,7 @@ package agentimport ("encoding/json""fmt"+ "io""net/http""net/http/httptest""os"@@ -13,6 +14,7 @@ import ("sync""github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools")// Fake is an httptest server speaking Chat Completions that plays a case's@@ -138,17 +140,32 @@ func (f *Fake) render(s catalog.Step, tools map[string]bool, msgs []Message) (st}return need(tools, "ask_user", map[string]any{"question": q})case s.Submit != nil:- for name := range tools {- if strings.HasPrefix(name, "submit_") {- args, err := resolveArgs(s.Submit, msgs)- return name, args, "", err- }+ name := submitTool(tools)+ if name == "" {+ return "", nil, "", fmt.Errorf("no submit tool on this arm")}- return "", nil, "", fmt.Errorf("no submit tool on this arm")+ args, err := resolveArgs(s.Submit, msgs)+ return name, args, "", err}return "", nil, "", fmt.Errorf("empty oracle step")}+// submitTool names the arm's final-answer tool, the first by name so the+// choice never depends on map order.+func submitTool(offered map[string]bool) string {+ var names []string+ for n := range offered {+ if tools.IsSubmit(n) {+ names = append(names, n)+ }+ }+ sort.Strings(names)+ if len(names) == 0 {+ return ""+ }+ return names[0]+}+func need(tools map[string]bool, name string, args map[string]any) (string, map[string]any, string, error) {if !tools[name] {return "", nil, "", fmt.Errorf("this arm has no %s tool", name)@@ -168,11 +185,27 @@ func resolveRuntime(s string, msgs []Message) (string, error) {if err != nil && firstErr == nil {firstErr = err}- return fmt.Sprint(v)+ return runtimeText(v)})return out, firstErr}+// runtimeText renders a looked-up value inside a string as it reads in the+// tool output: numbers as their JSON text (never 4.52e+07), objects as JSON.+func runtimeText(v any) string {+ switch t := v.(type) {+ case string:+ return t+ case json.Number:+ return t.String()+ case map[string]any, []any:+ raw, _ := json.Marshal(t)+ return string(raw)+ default:+ return fmt.Sprint(t)+ }+}+func resolveArgs(in map[string]any, msgs []Message) (map[string]any, error) {out := map[string]any{}var firstErr error@@ -256,8 +289,15 @@ func lookupRuntime(expr string, msgs []Message) (any, error) {if path == "" {return body, nil}+ // UseNumber keeps numbers as written: a nanosecond timestamp does not+ // survive a float64.var v any- if err := json.Unmarshal([]byte(firstJSON(body)), &v); err != nil {+ dec := json.NewDecoder(strings.NewReader(firstJSON(body)))+ dec.UseNumber()+ if err := dec.Decode(&v); err != nil {+ return nil, fmt.Errorf("template %s: result is not JSON", expr)+ }+ if _, err := dec.Token(); err != io.EOF {return nil, fmt.Errorf("template %s: result is not JSON", expr)}for _, seg := range splitPath(path) {diff --git a/cli/test/evals/internal/agent/loop.go b/cli/test/evals/internal/agent/loop.goindex 9cff574c..5e3deae3 100644--- a/cli/test/evals/internal/agent/loop.go+++ b/cli/test/evals/internal/agent/loop.go@@ -150,7 +150,7 @@ func Run(ctx context.Context, cfg Config) *record.Run {case final:res = tools.Result{Output: "not executed: the task was already submitted"}c.Rejected = true- case callN > budget.MaxToolCalls && !isSubmit(tc.Function.Name):+ case callN > budget.MaxToolCalls && !tools.IsSubmit(tc.Function.Name):res = tools.Result{Output: fmt.Sprintf("not executed: the tool-call budget (%d) is used up. Call %s now.", budget.MaxToolCalls, cfg.SubmitTool)}c.Rejected = truedefault:@@ -190,10 +190,6 @@ func Run(ctx context.Context, cfg Config) *record.Run {}}-func isSubmit(name string) bool {- return name == tools.SubmitPerf || name == tools.SubmitDebug || name == tools.Finish-}-func nonEmpty(s string) string {if strings.TrimSpace(s) == "" {return "{}"diff --git a/cli/test/evals/internal/catalog/case.go b/cli/test/evals/internal/catalog/case.goindex d0ee8b24..b74d7a49 100644--- a/cli/test/evals/internal/catalog/case.go+++ b/cli/test/evals/internal/catalog/case.go@@ -133,7 +133,7 @@ type Onset struct {ToleranceMinutes int `yaml:"tolerance_minutes"`}-// Trajectory is the tool-call DSL (plan §6.4).+// Trajectory is the tool-call DSL.type Trajectory struct {Required []CallMatcher `yaml:"required"`AnyOf [][]CallMatcher `yaml:"any_of"`@@ -241,6 +241,11 @@ func (c *Case) Validate() error {default:add("invoke must be slash or natural, got %q", c.Invoke)}+ // The runner preloads the skill for slash cases, and the prompt is what+ // the agent reads, so the two must agree.+ if slash := strings.HasPrefix(strings.TrimSpace(c.Prompt), "/"); slash != (c.Invoke == "slash") {+ add("invoke: slash goes with a prompt that starts with /<skill>, and only with one")+ }if strings.TrimSpace(c.Prompt) == "" {add("prompt is required")}diff --git a/cli/test/evals/internal/catalog/case_test.go b/cli/test/evals/internal/catalog/case_test.gonew file mode 100644index 00000000..25ebe46d--- /dev/null+++ b/cli/test/evals/internal/catalog/case_test.go@@ -0,0 +1,28 @@+package catalog++import (+ "strings"+ "testing"+)++// TestInvokeMatchesPrompt: the runner preloads the skill for invoke: slash,+// so a case's invoke and its prompt must agree.+func TestInvokeMatchesPrompt(t *testing.T) {+ for _, c := range []struct {+ invoke, prompt string+ ok bool+ }{+ {"slash", "/traceway perf GET /api/orders", true},+ {"natural", "Checkout got slow this week.", true},+ {"", "Checkout got slow this week.", true},+ {"slash", "Checkout got slow this week.", false},+ {"natural", "/traceway perf GET /api/orders", false},+ } {+ cs := &Case{ID: "perf-x", Suite: "perf", Seed: "perf/x", Arms: []string{ArmCLISkill}, Invoke: c.invoke, Prompt: c.prompt,+ Oracle: []Step{{Shell: "true"}}, Expect: Expect{Answer: &Answer{ChecklistItem: "none_found"}}, Distractor: "x"}+ err := cs.Validate()+ if got := err == nil || !strings.Contains(err.Error(), "invoke: slash"); got != c.ok {+ t.Errorf("invoke %q, prompt %q: %v", c.invoke, c.prompt, err)+ }+ }+}diff --git a/cli/test/evals/internal/catalog/models.go b/cli/test/evals/internal/catalog/models.goindex a4f825a1..da98cd61 100644--- a/cli/test/evals/internal/catalog/models.go+++ b/cli/test/evals/internal/catalog/models.go@@ -30,11 +30,10 @@ type Model struct {// MaxTokensPerTurn caps one completion.MaxTokensPerTurn int `yaml:"max_tokens_per_turn"`Temperature *float64 `yaml:"temperature"`- // Estimate is the per-run token envelope used by the preflight until- // baseline.json carries measured averages.+ // Estimate is the per-run token envelope for estimating a paid run's+ // cost. The oracle harness does not read it.Estimate Estimate `yaml:"estimate"`- // Fallback prices (USD per million tokens) used when the live price list- // cannot be fetched. The preflight says when it fell back.+ // FallbackPrice is USD per million tokens, for the same estimate.FallbackPrice Price `yaml:"fallback_price"`}diff --git a/cli/test/evals/internal/catalog/select.go b/cli/test/evals/internal/catalog/select.goindex aa414f83..25fba0a9 100644--- a/cli/test/evals/internal/catalog/select.go+++ b/cli/test/evals/internal/catalog/select.go@@ -59,7 +59,6 @@ type Selection struct {Suites string // comma-separatedTags string // comma-separated, all must matchArms string // comma-separated- Shard string // one suite}// Unit is one (case, arm) pair; the runner multiplies it by models and runs.@@ -102,9 +101,6 @@ func Select(cases []*Case, presets *Presets, sel Selection) ([]Unit, error) {return nil, fmt.Errorf("unknown suite %q", s)}}- if sel.Shard != "" && !slices.Contains(Suites, sel.Shard) {- return nil, fmt.Errorf("unknown shard %q (shards are suites)", sel.Shard)- }var out []UnitmatchedGlob := map[string]bool{}@@ -127,9 +123,6 @@ func Select(cases []*Case, presets *Presets, sel Selection) ([]Unit, error) {if len(suites) > 0 && !slices.Contains(suites, c.Suite) {continue}- if sel.Shard != "" && c.Suite != sel.Shard {- continue- }if !hasAll(c, tags) {continue}diff --git a/cli/test/evals/internal/checklist/checklist.go b/cli/test/evals/internal/checklist/checklist.goindex 7b40c83a..1e1558e6 100644--- a/cli/test/evals/internal/checklist/checklist.go+++ b/cli/test/evals/internal/checklist/checklist.go@@ -29,7 +29,7 @@ var (checklist []Row)-// Checklist parses the canonical performance.md once. The enum therefore+// Rows parses the canonical performance.md once. The enum therefore// tracks the skill: renaming a row renames its id, and the case test fails.func Rows() []Row {checklistOnce.Do(func() { checklist = ParseChecklist(knowledge.MustRead("performance.md")) })diff --git a/cli/test/evals/internal/grade/answer.go b/cli/test/evals/internal/grade/answer.goindex e2e9e5c5..0542a1d2 100644--- a/cli/test/evals/internal/grade/answer.go+++ b/cli/test/evals/internal/grade/answer.go@@ -114,8 +114,11 @@ func sameEndpoint(got, want string) bool {if g == w {return true}+ // The answer may leave the method out, or put a word before the+ // endpoint ("endpoint POST /api/checkout"); it may not name another+ // method._, wr, _ := strings.Cut(w, " ")- return g == wr || strings.HasSuffix(g, " "+wr)+ return g == wr || strings.HasSuffix(g, " "+w)}func containsFold(s, sub string) bool {diff --git a/cli/test/evals/internal/grade/grade.go b/cli/test/evals/internal/grade/grade.goindex 8886dac2..4ba9b027 100644--- a/cli/test/evals/internal/grade/grade.go+++ b/cli/test/evals/internal/grade/grade.go@@ -20,7 +20,6 @@ type Verdict struct {Score *float64 `json:"score,omitempty"`Failures []string `json:"failures,omitempty"`Notes []string `json:"notes,omitempty"`- CostUSD float64 `json:"costUsd,omitempty"`}func (v *Verdict) fail(format string, a ...any) {@@ -51,7 +50,7 @@ func All(ctx context.Context, in *Input) []*Verdict {if in.Case.Expect.Calls != nil {out = append(out, Trajectory(in.Run, in.Case.Expect.Calls))}- if in.Case.Suite == "perf" || in.Case.Suite == "debug" || len(in.Case.Expect.Grounding) > 0 {+ if in.Case.Suite == "perf" || len(in.Case.Expect.Grounding) > 0 {out = append(out, Grounding(in))}if in.Case.Expect.Report != nil {diff --git a/cli/test/evals/internal/grade/grade_test.go b/cli/test/evals/internal/grade/grade_test.goindex 4f51529f..1603d64d 100644--- a/cli/test/evals/internal/grade/grade_test.go+++ b/cli/test/evals/internal/grade/grade_test.go@@ -2,6 +2,7 @@ package gradeimport ("encoding/json"+ "fmt""strings""testing"@@ -62,12 +63,37 @@ func TestTrajectory(t *testing.T) {s.MaxIdenticalRepeats = 1r.Calls = append(r.Calls, call("c09", true, "{}", "traceway endpoints list --since 24h --output json"))}, "same call ran 2 times"},+ "thrash, read in chunks of the same window": {func(r *record.Run, s *catalog.Trajectory) {+ s.MaxIdenticalRepeats = 1+ read := func(id string) *record.Call {+ return &record.Call{ID: id, Tool: "read_file", OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: map[string]any{"path": "big.go", "offset": 2001}}, OK: true}}}+ }+ r.Calls = append(r.Calls, read("c11"), read("c12"))+ }, "same call ran 2 times"},"too many failures": {func(r *record.Run, s *catalog.Trajectory) {s.MaxFailedCalls = 0s.MaxFailedCalls = 1r.Calls = append(r.Calls, call("c10", false, "x", "traceway logs query"))}, "failed tool calls"},}+ // Reading a file in chunks and changing one file several times are not+ // thrashing.+ chunked := &record.Run{Calls: append([]*record.Call{}, run.Calls...)}+ for i, off := range []int{0, 2001, 4001} {+ args := map[string]any{"path": "big.go"}+ if off > 0 {+ args["offset"] = off+ }+ chunked.Calls = append(chunked.Calls,+ &record.Call{ID: fmt.Sprintf("r%d", i), Tool: "read_file", OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: args}, OK: true}}},+ &record.Call{ID: fmt.Sprintf("e%d", i), Tool: "edit_file", OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpEditFile, Args: map[string]any{"path": "big.go"}}, OK: true}}})+ }+ repeats := *spec+ repeats.MaxIdenticalRepeats = 1+ if v := Trajectory(chunked, &repeats); !v.Pass {+ t.Fatalf("chunked reads and repeated edits counted as thrashing:\n%s", failures(v))+ }+for name, c := range cases {t.Run(name, func(t *testing.T) {r := &record.Run{Calls: append([]*record.Call{}, run.Calls...)}@@ -100,6 +126,7 @@ func TestAnswer(t *testing.T) {"wrong row": {"checklist_item": "missing_or_unused_index", "locus": good["locus"]},"distractor": {"checklist_item": "over_fetching", "locus": good["locus"]},"wrong locus": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "GET /api/orders/:id", "span": "order_items"}},+ "wrong method": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "POST /api/orders", "span": "order_items"}},"no span": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "GET /api/orders"}},"no submission": nil,} {@@ -109,6 +136,25 @@ func TestAnswer(t *testing.T) {}}+func TestSameEndpoint(t *testing.T) {+ for _, c := range []struct {+ got, want string+ same bool+ }{+ {"POST /api/checkout", "POST /api/checkout", true},+ {"post /api/checkout/", "POST /api/checkout", true},+ {"/api/checkout", "POST /api/checkout", true},+ {"endpoint POST /api/checkout", "POST /api/checkout", true},+ {"GET /api/checkout", "/api/checkout", true},+ {"GET /api/checkout", "POST /api/checkout", false},+ {"POST /api/checkout/items", "POST /api/checkout", false},+ } {+ if got := sameEndpoint(c.got, c.want); got != c.same {+ t.Errorf("sameEndpoint(%q, %q) = %v", c.got, c.want, got)+ }+ }+}+func TestGrounding(t *testing.T) {out := `{"spans":[{"attributes":{"db.query.text":"SELECT id FROM order_items WHERE order_id = $1"}}],"id":"6f9619ff-8b86-d011-b42d-00c04fc964ff"}`c1 := call("c01", true, out, "traceway endpoints show 6f9619ff-8b86-d011-b42d-00c04fc964ff --recorded-at 2026-10-17T10:00:00Z")@@ -126,7 +172,11 @@ func TestGrounding(t *testing.T) {"failed call": {perfInput(sub("c01", ""), call("c01", false, out, "traceway endpoints show x")), "which failed"},"truncated call": {perfInput(sub("c01", ""), &record.Call{ID: "c01", OK: true, Truncated: true, Output: out}), "truncated"},"invented id": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1), "no tool returned"},- "missing key": {perfInput(sub("c02", ""), c1, call("c02", true, `{"data":[]}`, "traceway endpoints list")), "contains"},+ // The submit call itself is recorded with the submission as its+ // arguments; that must not vouch for the ids in it.+ "invented id, submit recorded": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1,+ &record.Call{ID: "c02", Tool: "submit_perf_diagnosis", OK: true, Args: json.RawMessage(`{"report":"trace 4bf92f3577b34da6a3ce929d0e0e4736"}`)}), "no tool returned"},+ "missing key": {perfInput(sub("c02", ""), c1, call("c02", true, `{"data":[]}`, "traceway endpoints list")), "contains"},}for name, tc := range tests {t.Run(name, func(t *testing.T) {diff --git a/cli/test/evals/internal/grade/grounding.go b/cli/test/evals/internal/grade/grounding.goindex 98a05ea2..374982ce 100644--- a/cli/test/evals/internal/grade/grounding.go+++ b/cli/test/evals/internal/grade/grounding.go@@ -7,6 +7,7 @@ import ("strings""github.com/tracewayapp/traceway/cli/test/evals/internal/record"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools")// Grounding checks that the evidence is real:@@ -43,7 +44,7 @@ func Grounding(in *Input) *Verdict {cited = append(cited, c)}}- } else if in.Case.Suite == "perf" || in.Case.Suite == "debug" {+ } else if in.Case.Suite == "perf" {v.fail("the submission cites no evidence")}for _, key := range in.Case.Expect.Grounding {@@ -107,13 +108,17 @@ func unescapeJSON(s string) string {var idRe = regexp.MustCompile(`(?i)\b(?:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32}|[0-9a-f]{16})\b`)// InventedIDs lists ids in the submission (and final text) that appear in no-// tool output and not in the prompt.+// tool output and not in the prompt. The submit calls are left out of the+// corpus: their arguments are the submission itself.func InventedIDs(run *record.Run) []string {raw, _ := json.Marshal(run.Submission)text := string(raw) + "\n" + run.FinalTextvar corpus strings.Buildercorpus.WriteString(strings.ToLower(run.Prompt))for _, c := range run.Calls {+ if tools.IsSubmit(c.Tool) {+ continue+ }corpus.WriteString(strings.ToLower(c.Output))corpus.WriteString(strings.ToLower(string(c.Args)))for _, op := range c.Ops {diff --git a/cli/test/evals/internal/grade/leakage.go b/cli/test/evals/internal/grade/leakage.goindex 67aba577..29ad964f 100644--- a/cli/test/evals/internal/grade/leakage.go+++ b/cli/test/evals/internal/grade/leakage.go@@ -11,7 +11,7 @@ import ("github.com/tracewayapp/traceway/cli/test/evals/internal/checklist")-// Leakage lint (plan §4.8): the seeded data and the prompt must not name the+// Leakage lint: the seeded data and the prompt must not name the// answer. Seeded strings (span names, attribute values, log bodies, stack// traces, object names) are checked against a denylist of checklist terms// and against the expected checklist row's title; prompts get the checklistdiff --git a/cli/test/evals/internal/grade/trajectory.go b/cli/test/evals/internal/grade/trajectory.goindex 1f9623d1..c913bab1 100644--- a/cli/test/evals/internal/grade/trajectory.go+++ b/cli/test/evals/internal/grade/trajectory.go@@ -10,7 +10,9 @@ import ("strings""github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops""github.com/tracewayapp/traceway/cli/test/evals/internal/record"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools")// Trajectory checks the tool calls against the case's call DSL:@@ -81,7 +83,7 @@ func Trajectory(run *record.Run, t *catalog.Trajectory) *Verdict {executed := 0failed := 0for _, c := range run.Calls {- if c.Rejected || isSubmitTool(c.Tool) {+ if c.Rejected || tools.IsSubmit(c.Tool) {continue}executed++@@ -96,8 +98,13 @@ func Trajectory(run *record.Run, t *catalog.Trajectory) *Verdict {v.fail("%d failed tool calls, more than max_failed_calls %d", failed, t.MaxFailedCalls)}if t.MaxIdenticalRepeats > 0 {+ // Thrashing is asking the same question again. Changing a local file+ // (several edits, or appending to one file) is not.counts := map[string]int{}for _, op := range all {+ if op.Op.Op == ops.OpWriteFile || op.Op.Op == ops.OpEditFile {+ continue+ }counts[opKey(op.OpRecord)]++}keys := make([]string, 0, len(counts))@@ -115,10 +122,6 @@ func Trajectory(run *record.Run, t *catalog.Trajectory) *Verdict {return v}-func isSubmitTool(name string) bool {- return name == "submit_perf_diagnosis" || name == "submit_debug_diagnosis" || name == "finish"-}-func firstMatch(all []record.CallOp, m catalog.CallMatcher) int {for i, op := range all {if matches(op.OpRecord, m) {diff --git a/cli/test/evals/internal/ops/ops_test.go b/cli/test/evals/internal/ops/ops_test.goindex e4f6a7a9..aeea53f1 100644--- a/cli/test/evals/internal/ops/ops_test.go+++ b/cli/test/evals/internal/ops/ops_test.go@@ -30,15 +30,33 @@ func TestOpsMatchToolMap(t *testing.T) {}// TestEveryCLICommandMaps walks both built editions' help: every leaf-// command must map to an op, and write-edition commands must be writes.+// command must map to an op, and exactly the commands only traceway-write+// has are marked WriteEdition, as writes.func TestEveryCLICommandMaps(t *testing.T) {bins := env.BinariesForTest(t)- for _, bin := range []string{bins.CLI, bins.CLIWrite} {- for _, path := range leafCommands(t, bin, nil) {- op := ParseCLI(path)- if op.Op == OpUnknownCLI {- t.Errorf("%s %v maps to no op", bin, path)- }+ inRead := map[string]bool{}+ for _, path := range leafCommands(t, bins.CLI, nil) {+ inRead[strings.Join(path, " ")] = true+ op := ParseCLI(path)+ if op.Op == OpUnknownCLI {+ t.Errorf("traceway %v maps to no op", path)+ continue+ }+ if spec, ok := SpecFor(op.Op); ok && spec.WriteEdition {+ t.Errorf("traceway %v exists in the read edition, but %s is marked WriteEdition", path, op.Op)+ }+ }+ for _, path := range leafCommands(t, bins.CLIWrite, nil) {+ op := ParseCLI(path)+ if op.Op == OpUnknownCLI {+ t.Errorf("traceway-write %v maps to no op", path)+ continue+ }+ if inRead[strings.Join(path, " ")] {+ continue+ }+ if spec, ok := SpecFor(op.Op); !ok || !spec.WriteEdition || !op.Write {+ t.Errorf("traceway-write %v exists only in the write edition, so %s must be WriteEdition and Write", path, op.Op)}}}diff --git a/cli/test/evals/internal/report/aggregate.go b/cli/test/evals/internal/report/aggregate.goindex 1653f308..12c95092 100644--- a/cli/test/evals/internal/report/aggregate.go+++ b/cli/test/evals/internal/report/aggregate.go@@ -57,7 +57,6 @@ type Aggregates struct {Models []Delta `json:"models"`Cost map[string]*Costs `json:"cost"` // by modelSpendUSD float64 `json:"spendUsd"`- JudgeUSD float64 `json:"judgeUsd"`Statuses map[string]int `json:"statuses"`TopFailing []GraderCount `json:"topFailingGraders"`}@@ -81,8 +80,7 @@ func Aggregate(runs []*runner.Result) Aggregates {cacheIn, cacheHit := map[string]int{}, map[string]int{}for _, r := range runs {a.Statuses[r.Status]++- a.SpendUSD += r.Metrics.CostUSD + r.Metrics.JudgeCostUSD- a.JudgeUSD += r.Metrics.JudgeCostUSD+ a.SpendUSD += r.Metrics.CostUSDc := a.Cost[r.Model]if c == nil {c = &Costs{}diff --git a/cli/test/evals/internal/report/markdown.go b/cli/test/evals/internal/report/markdown.goindex 4cd3d881..740647dc 100644--- a/cli/test/evals/internal/report/markdown.go+++ b/cli/test/evals/internal/report/markdown.go@@ -35,7 +35,11 @@ func Markdown(r *Results) string {}sort.Strings(models)fmt.Fprintf(&b, "- **Models:** %s\n", strings.Join(models, ", "))- fmt.Fprintf(&b, "- **Spend:** $%.2f (judge $%.2f) of a $%.2f cap\n", a.SpendUSD, a.JudgeUSD, h.MaxCostUSD)+ // An oracle run costs nothing, so its summary leaves money out.+ paid := !h.Oracle+ if paid {+ fmt.Fprintf(&b, "- **Spend:** $%.2f\n", a.SpendUSD)+ }var st []stringfor k, v := range a.Statuses {st = append(st, fmt.Sprintf("%s %d", k, v))@@ -62,16 +66,18 @@ func Markdown(r *Results) string {fmt.Fprintf(&b, "| %s | %s | %s | %.0f%% | %.0f%% | %+.0fpp |\n", d.Suite, d.A, d.B, d.RateA*100, d.RateB*100, d.Delta)}}- b.WriteString("\n## Cost (averages over passing runs)\n\n| Model | Passing runs | Avg cost | Avg tool calls | Avg input tokens | Avg output tokens | Cache hit rate | Total spend |\n|---|---|---|---|---|---|---|---|\n")- for _, m := range sortedKeys(a.Cost) {- c := a.Cost[m]- fmt.Fprintf(&b, "| %s | %d | $%.3f | %.1f | %.0f | %.0f | %.0f%% | $%.2f |\n", m, c.PassingRuns, c.AvgCostUSD, c.AvgToolCalls, c.AvgInputTokens, c.AvgOutputTokens, c.CacheHitRate*100, c.TotalCostUSD)+ if paid {+ b.WriteString("\n## Cost (averages over passing runs)\n\n| Model | Passing runs | Avg cost | Avg tool calls | Avg input tokens | Avg output tokens | Cache hit rate | Total spend |\n|---|---|---|---|---|---|---|---|\n")+ for _, m := range sortedKeys(a.Cost) {+ c := a.Cost[m]+ fmt.Fprintf(&b, "| %s | %d | $%.3f | %.1f | %.0f | %.0f | %.0f%% | $%.2f |\n", m, c.PassingRuns, c.AvgCostUSD, c.AvgToolCalls, c.AvgInputTokens, c.AvgOutputTokens, c.CacheHitRate*100, c.TotalCostUSD)+ }}bySuiteCost := map[string]float64{}for _, run := range r.Runs {bySuiteCost[run.Suite+" / "+run.Model] += run.Metrics.CostUSD}- if len(bySuiteCost) > 0 {+ if paid && len(bySuiteCost) > 0 {b.WriteString("\n| Suite / model | Spend |\n|---|---|\n")for _, k := range sortedKeys(bySuiteCost) {fmt.Fprintf(&b, "| %s | $%.2f |\n", k, bySuiteCost[k])@@ -92,11 +98,22 @@ func Markdown(r *Results) string {fmt.Fprintf(&b, "| %s | %s | %s | %d/%d |\n", cs, m, arm, v.Pass, v.Total)}}- b.WriteString("\n## Runs\n\n| Case | Arm | Model | Run | Status | Calls | Cost | First failure |\n|---|---|---|---|---|---|---|---|\n")+ b.WriteString("\n## Runs\n\n| Case | Arm | Model | Run | Status | Calls |")+ if paid {+ b.WriteString(" Cost |")+ }+ b.WriteString(" First failure | Transcript |\n|---|---|---|---|---|---|")+ if paid {+ b.WriteString("---|")+ }+ b.WriteString("---|---|\n")for _, run := range r.Runs {- fmt.Fprintf(&b, "| %s | %s | %s | %d | %s | %d | $%.3f | %s |\n", run.Case, run.Arm, run.Model, run.RunIdx, run.Status, run.Metrics.ToolCalls, run.Metrics.CostUSD, mdEscape(firstFailure(run)))+ fmt.Fprintf(&b, "| %s | %s | %s | %d | %s | %d |", run.Case, run.Arm, run.Model, run.RunIdx, run.Status, run.Metrics.ToolCalls)+ if paid {+ fmt.Fprintf(&b, " $%.3f |", run.Metrics.CostUSD)+ }+ fmt.Fprintf(&b, " %s | %s |\n", mdEscape(firstFailure(run)), orDash(run.Transcript))}- b.WriteString("\nTranscripts are in the artifact under `runs/<case>/<arm>/<model>/r<N>/transcript.jsonl`.\n")return b.String()}diff --git a/cli/test/evals/internal/report/report_test.go b/cli/test/evals/internal/report/report_test.goindex 24bdbd5b..bf286f87 100644--- a/cli/test/evals/internal/report/report_test.go+++ b/cli/test/evals/internal/report/report_test.go@@ -2,6 +2,7 @@ package reportimport ("math"+ "strings""testing""github.com/tracewayapp/traceway/cli/test/evals/internal/runner"@@ -34,3 +35,23 @@ func TestAggregate(t *testing.T) {t.Fatalf("cost %+v", c)}}++// TestOracleSummary: an oracle run costs nothing, so its summary names no+// spend, and each run's transcript is listed at the path it was written to.+func TestOracleSummary(t *testing.T) {+ r := res("perf-a", "cli+skill", "oracle", 1, "pass", 0)+ r.Transcript = "runs/perf-a/cli_skill/oracle/r1/transcript.jsonl"+ out := Markdown(&Results{Header: Header{Oracle: true}, Runs: []*runner.Result{r}, Aggregates: Aggregate([]*runner.Result{r})})+ for _, s := range []string{"Spend", "Cost", "cap"} {+ if strings.Contains(out, s) {+ t.Errorf("oracle summary mentions %q:\n%s", s, out)+ }+ }+ if !strings.Contains(out, r.Transcript) {+ t.Errorf("summary lacks the transcript path:\n%s", out)+ }+ paid := Markdown(&Results{Runs: []*runner.Result{r}, Aggregates: Aggregate([]*runner.Result{r})})+ if !strings.Contains(paid, "Spend") || !strings.Contains(paid, "| Cost |") {+ t.Errorf("a paid run's summary lost its cost:\n%s", paid)+ }+}diff --git a/cli/test/evals/internal/report/results.go b/cli/test/evals/internal/report/results.goindex 16807d6c..f877560d 100644--- a/cli/test/evals/internal/report/results.go+++ b/cli/test/evals/internal/report/results.go@@ -34,7 +34,6 @@ type Header struct {Command string `json:"command"`Started time.Time `json:"started"`Finished time.Time `json:"finished"`- MaxCostUSD float64 `json:"maxCostUsd"`Oracle bool `json:"oracle,omitempty"`CI map[string]string `json:"ci,omitempty"`}@@ -148,22 +147,3 @@ func Load(dir string) (*Results, error) {}return r, nil}--// Merge combines shard results under one header.-func Merge(parts []*Results) *Results {- out := &Results{}- total := 0.0- for i, p := range parts {- if i == 0 {- out.Header = p.Header- }- total += p.Header.MaxCostUSD- if p.Header.Finished.After(out.Header.Finished) {- out.Header.Finished = p.Header.Finished- }- out.Runs = append(out.Runs, p.Runs...)- }- out.Header.MaxCostUSD = total- sort.SliceStable(out.Runs, func(i, j int) bool { return out.Runs[i].Key() < out.Runs[j].Key() })- return out-}diff --git a/cli/test/evals/internal/runner/runner.go b/cli/test/evals/internal/runner/runner.goindex ff663e86..59bf3af5 100644--- a/cli/test/evals/internal/runner/runner.go+++ b/cli/test/evals/internal/runner/runner.go@@ -72,7 +72,6 @@ type Metrics struct {CacheWriteTokens int `json:"cacheWriteTokens"`OutputTokens int `json:"outputTokens"`CostUSD float64 `json:"costUsd"`- JudgeCostUSD float64 `json:"judgeCostUsd"`WallMs int64 `json:"wallMs"`SkillLoaded bool `json:"skillLoaded"`Skills []string `json:"skills,omitempty"`@@ -136,17 +135,6 @@ type surface struct {submit string}-func submitFor(suite string) string {- switch suite {- case "perf":- return tools.SubmitPerf- case "debug":- return tools.SubmitDebug- default:- return tools.Finish- }-}-func runOne(ctx context.Context, opt Options, j Job) (res *Result) {c := j.Unit.Casearm := j.Unit.Arm@@ -258,7 +246,8 @@ func runOne(ctx context.Context, opt Options, j Job) (res *Result) {prompt := strings.TrimSpace(resolved.Prompt)var preloaded *tools.Skill- if name := agent.SlashSkill(prompt); name != "" {+ if resolved.Invoke == "slash" {+ name := agent.SlashSkill(prompt)for i := range skillDefs {if skillDefs[i].Name == name {preloaded = &skillDefs[i]@@ -308,9 +297,6 @@ func runOne(ctx context.Context, opt Options, j Job) (res *Result) {in := &grade.Input{Case: resolved, Arm: arm, Run: run, Seed: jsonify(exposed)}res.Graders = append(res.Graders, grade.All(ctx, in)...)- for _, v := range res.Graders {- res.Metrics.JudgeCostUSD += v.CostUSD- }switch {case run.StopReason == agent.StopBudget:res.Status = "budget_exhausted"@@ -321,7 +307,7 @@ func runOne(ctx context.Context, opt Options, j Job) (res *Result) {default:res.Status = "fail"}- logf(opt, " %s %s (%d calls, $%.4f)", RunKey(c.ID, arm, res.Model, j.Idx), res.Status, res.Metrics.ToolCalls, res.Metrics.CostUSD)+ logf(opt, " %s %s (%d calls)", RunKey(c.ID, arm, res.Model, j.Idx), res.Status, res.Metrics.ToolCalls)return res}diff --git a/cli/test/evals/internal/runner/surface.go b/cli/test/evals/internal/runner/surface.goindex 76fde75d..56606c4d 100644--- a/cli/test/evals/internal/runner/surface.go+++ b/cli/test/evals/internal/runner/surface.go@@ -6,12 +6,13 @@ import ("github.com/tracewayapp/traceway/cli/test/evals/internal/catalog""github.com/tracewayapp/traceway/cli/test/evals/internal/env"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools")// prepareSurface mints the arm's credential and connects its surface: the// CLI logged in with a read PAT (XDG dirs outside the workspace).func prepareSurface(ctx context.Context, opt Options, c *catalog.Case, arm string, inst *env.Instance, primary env.Project, dir string) (surface, error) {- s := surface{submit: submitFor(c.Suite)}+ s := surface{submit: tools.SubmitPerf}if arm == catalog.ArmCLISkill {s.skills = []string{"traceway"}}diff --git a/cli/test/evals/internal/seed/metrics.go b/cli/test/evals/internal/seed/metrics.goindex beeeb582..11df53e4 100644--- a/cli/test/evals/internal/seed/metrics.go+++ b/cli/test/evals/internal/seed/metrics.go@@ -125,9 +125,11 @@ type metricSeries struct {window time.Durationscope string// cumTotal and cumPrev carry a counter's running total across the- // chunks metricRequests renders it in.+ // chunks metricRequests renders it in; histPrev carries the end of a+ // histogram's last delta window the same way.cumTotal float64cumPrev time.Time+ histPrev time.Time}// MetricStep controls sampling: points every Recent inside the last day,@@ -341,19 +343,30 @@ func (m *metricSeries) render(times []time.Time, origin time.Time) *metricspb.Mem.cumTotal, m.cumPrev = total, prevout.Data = &metricspb.Metric_Sum{Sum: sum}case histogramKind:+ // Each delta point covers the time since the previous one, which is+ // longer where points are sparser (MetricStep.Old), and its count+ // covers the same window.h := &metricspb.Histogram{AggregationTemporality: metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_DELTA}- prev := times[0].Add(-time.Minute)+ prev := m.histPrev+ if prev.IsZero() {+ step := time.Minute+ if len(times) > 1 {+ step = times[1].Sub(times[0])+ }+ prev = times[0].Add(-step)+ }for _, t := range times {per := m.count(t) * t.Sub(prev).Minutes()- prev = tcnt := uint64(math.Max(0, math.Round(per)))mean := m.value(t)s := mean * float64(cnt)h.DataPoints = append(h.DataPoints, &metricspb.HistogramDataPoint{- Attributes: attrs, StartTimeUnixNano: nanos(t.Add(-time.Minute)), TimeUnixNano: nanos(t),+ Attributes: attrs, StartTimeUnixNano: nanos(prev), TimeUnixNano: nanos(t),Count: cnt, Sum: &s,})+ prev = t}+ m.histPrev = prevout.Data = &metricspb.Metric_Histogram{Histogram: h}}return outdiff --git a/cli/test/evals/internal/seed/metrics_test.go b/cli/test/evals/internal/seed/metrics_test.gonew file mode 100644index 00000000..67994483--- /dev/null+++ b/cli/test/evals/internal/seed/metrics_test.go@@ -0,0 +1,41 @@+package seed++import (+ "testing"+ "time"+)++// TestHistogramDeltaWindows: each delta point starts where the previous one+// ended, across the chunks a series is rendered in, and counts the+// observations of exactly that window.+func TestHistogramDeltaWindows(t *testing.T) {+ m := &metricSeries{name: "h", kind: histogramKind, value: func(time.Time) float64 { return 5 }, count: func(time.Time) float64 { return 10 }}+ base := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)+ var times []time.Time+ for i := range 6 {+ times = append(times, base.Add(time.Duration(i)*MetricStep.Old))+ }+ var points []struct {+ start, end uint64+ count uint64+ }+ for _, chunk := range [][]time.Time{times[:3], times[3:]} {+ for _, dp := range m.render(chunk, base).GetHistogram().DataPoints {+ points = append(points, struct {+ start, end uint64+ count uint64+ }{dp.StartTimeUnixNano, dp.TimeUnixNano, dp.Count})+ }+ }+ for i, p := range points {+ if p.count != 100 {+ t.Errorf("point %d counts %d observations, want 100 (10/min over 10 min)", i, p.count)+ }+ if i > 0 && p.start != points[i-1].end {+ t.Errorf("point %d starts at %d, want the previous point's end %d", i, p.start, points[i-1].end)+ }+ if got := time.Duration(p.end - p.start); got != MetricStep.Old {+ t.Errorf("point %d covers %s, want %s", i, got, MetricStep.Old)+ }+ }+}diff --git a/cli/test/evals/internal/tools/coreutils.go b/cli/test/evals/internal/tools/coreutils.goindex c85b58d9..08b09d57 100644--- a/cli/test/evals/internal/tools/coreutils.go+++ b/cli/test/evals/internal/tools/coreutils.go@@ -99,14 +99,16 @@ func coreutilNames() []string {}// flags is a tiny getopt: single-dash letters (bundled), --long, and-// options that take a value.+// options that take a value. set keeps an option's last value, vals every+// value of a repeatable one (grep -e A -e B).type flags struct {set map[string]string+ vals map[string][]stringargs []string}func parseFlags(args []string, withValue string) flags {- f := flags{set: map[string]string{}}+ f := flags{set: map[string]string{}, vals: map[string][]string{}}for i := 0; i < len(args); i++ {a := args[i]if a == "--" {@@ -131,6 +133,7 @@ func parseFlags(args []string, withValue string) flags {v = args[i]}f.set[c] = v+ f.vals[c] = append(f.vals[c], v)break}f.set[c] = "true"@@ -281,10 +284,7 @@ func breToERE(p string) string {func uGrep(u *uctx, args []string) int {f := parseFlags(args, "eABCm")- var patterns []string- if v, ok := f.set["e"]; ok {- patterns = append(patterns, v)- }+ patterns := f.vals["e"]rest := f.argsif len(patterns) == 0 {if len(rest) == 0 {@@ -383,6 +383,7 @@ func uGrep(u *uctx, args []string) int {lines := splitLines(data)count := 0lastPrinted := -1+ afterLeft := 0prefix := func(i int, sep string) string {p := ""if multi {@@ -393,10 +394,26 @@ func uGrep(u *uctx, args []string) int {}return p}+ // printLine prints line i once, with grep's "--" between context+ // groups that are not adjacent.+ printLine := func(i int, sep string) {+ if (before > 0 || after > 0) && lastPrinted >= 0 && i > lastPrinted+1 {+ fmt.Fprintln(w, "--")+ }+ fmt.Fprintln(w, prefix(i, sep)+lines[i])+ lastPrinted = i+ }for i, l := range lines {if !match(l) {+ if afterLeft > 0 && !f.has("c") && !f.has("o") {+ printLine(i, "-")+ afterLeft--+ }continue}+ if maxCount >= 0 && count >= maxCount {+ break+ }found = truecount++if f.has("q") {@@ -407,9 +424,6 @@ func uGrep(u *uctx, args []string) int {break}if f.has("c") {- if maxCount >= 0 && count >= maxCount {- break- }continue}if f.has("o") && !f.has("v") {@@ -418,22 +432,13 @@ func uGrep(u *uctx, args []string) int {fmt.Fprintln(w, prefix(i, ":")+m)}}- } else {- for b := max(lastPrinted+1, i-before); b < i; b++ {- fmt.Fprintln(w, prefix(b, "-")+lines[b])- }- fmt.Fprintln(w, prefix(i, ":")+l)- lastPrinted = i- for a := i + 1; a <= i+after && a < len(lines); a++ {- if !match(lines[a]) {- fmt.Fprintln(w, prefix(a, "-")+lines[a])- lastPrinted = a- }- }+ continue}- if maxCount >= 0 && count >= maxCount {- break+ for b := max(lastPrinted+1, i-before); b < i; b++ {+ printLine(b, "-")}+ printLine(i, ":")+ afterLeft = after}if f.has("c") {if multi {diff --git a/cli/test/evals/internal/tools/files.go b/cli/test/evals/internal/tools/files.goindex 58e85d88..cf8d6750 100644--- a/cli/test/evals/internal/tools/files.go+++ b/cli/test/evals/internal/tools/files.go@@ -74,7 +74,16 @@ func (t *ReadFile) Call(_ context.Context, raw json.RawMessage) Result {fmt.Fprintf(&b, "[%d more lines; continue with offset %d]\n", len(lines)-(start+limit)+1, start+limit)}out, trunc := Cap(b.String())- return Result{Output: out, OK: true, Truncated: trunc, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: map[string]any{"path": rel}}, OK: true, Output: text}}}+ // The window is part of the operation, so reading a long file in chunks+ // is not the same call repeated.+ args := map[string]any{"path": rel}+ if start > 1 {+ args["offset"] = start+ }+ if in.Limit > 0 {+ args["limit"] = in.Limit+ }+ return Result{Output: out, OK: true, Truncated: trunc, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: args}, OK: true, Output: text}}}}// ListFiles lists files under a directory.diff --git a/cli/test/evals/internal/tools/jqawk.go b/cli/test/evals/internal/tools/jqawk.goindex 0b443a86..5d9fb64f 100644--- a/cli/test/evals/internal/tools/jqawk.go+++ b/cli/test/evals/internal/tools/jqawk.go@@ -24,9 +24,19 @@ func runJQ(hc interp.HandlerContext, r *shellRun, args []string) int {var files []stringvar varNames []stringvar varValues []any+ // Options may come before or after the filter, as in jq; "--" ends them.+ optionsDone := falsefor i := 0; i < len(args); i++ {a := args[i]switch {+ case optionsDone || a == "-" || !strings.HasPrefix(a, "-"):+ if !haveFilter {+ filter, haveFilter = a, true+ } else {+ files = append(files, a)+ }+ case a == "--":+ optionsDone = truecase a == "--arg" && i+2 < len(args):varNames = append(varNames, "$"+args[i+1])varValues = append(varValues, args[i+2])@@ -68,7 +78,7 @@ func runJQ(hc interp.HandlerContext, r *shellRun, args []string) int {u.errf("jq: unsupported option %s", a)return 2}- case strings.HasPrefix(a, "-") && len(a) > 1 && !haveFilter:+ default:for _, c := range a[1:] {switch c {case 'r':@@ -93,12 +103,6 @@ func runJQ(hc interp.HandlerContext, r *shellRun, args []string) int {return 2}}- default:- if !haveFilter {- filter, haveFilter = a, true- } else {- files = append(files, a)- }}}if !haveFilter {diff --git a/cli/test/evals/internal/tools/shell.go b/cli/test/evals/internal/tools/shell.goindex 867b1886..8f8ed220 100644--- a/cli/test/evals/internal/tools/shell.go+++ b/cli/test/evals/internal/tools/shell.go@@ -56,7 +56,7 @@ func (s *Shell) Spec() Spec {allowed := append(names, "jq", "awk")allowed = append(allowed, coreutilNames()...)if s.Git {- allowed = append(allowed, "git (log, show, diff, status, blame, ls-files, rev-parse)")+ allowed = append(allowed, "git ("+strings.Join(gitAllowed, ", ")+")")}return Spec{Name: "shell",@@ -246,25 +246,45 @@ func (r *shellRun) execTraceway(ctx context.Context, hc interp.HandlerContext, bvar gitAllowed = []string{"log", "show", "diff", "status", "blame", "ls-files", "rev-parse", "shortlog"}+// gitDenied are options of the allowed subcommands that write a file or run+// a program.+var gitDenied = []string{"--output", "--ext-diff"}++// execGit runs a read-only git subcommand on the workspace repository. GIT_DIR+// is pinned, so git never discovers a repository (and its config) written+// elsewhere in the workspace; the workspace's .git is read-only to every tool.func (r *shellRun) execGit(ctx context.Context, hc interp.HandlerContext, args []string) error {+ refuse := func(format string, a ...any) error {+ fmt.Fprintf(hc.Stderr, "git: "+format+"\n", a...)+ return interp.NewExitStatus(2)+ }+ if _, err := r.s.Sandbox.Resolve("", hc.Dir); err != nil {+ return refuse("only the workspace repository is available")+ }+ rest := args[1:]+ for len(rest) > 0 && (rest[0] == "--no-pager" || rest[0] == "-P") {+ rest = rest[1:]+ }+ if len(rest) > 0 && strings.HasPrefix(rest[0], "-") {+ return refuse("global options are not allowed in this sandbox")+ }sub := ""- for _, a := range args[1:] {- if !strings.HasPrefix(a, "-") {- sub = a- break- }- if a == "-C" || strings.HasPrefix(a, "--git-dir") || strings.HasPrefix(a, "--work-tree") || strings.HasPrefix(a, "-c") {- fmt.Fprintln(hc.Stderr, "git: global options are not allowed in this sandbox")- return interp.NewExitStatus(2)- }+ if len(rest) > 0 {+ sub = rest[0]}if !slices.Contains(gitAllowed, sub) {- fmt.Fprintf(hc.Stderr, "git %s: only read-only git (%s) is available\n", sub, strings.Join(gitAllowed, ", "))- return interp.NewExitStatus(2)+ return refuse("%s: only read-only git (%s) is available", sub, strings.Join(gitAllowed, ", "))}- cmd := exec.CommandContext(ctx, "git", append([]string{"--no-pager"}, args[1:]...)...)+ for _, a := range rest[1:] {+ if err := r.gitArg(hc.Dir, a); err != nil {+ return refuse("%v", err)+ }+ }+ root := r.s.Sandbox.Root+ cmd := exec.CommandContext(ctx, "git", append([]string{"--no-pager"}, rest...)...)cmd.Dir = hc.Dir- cmd.Env = []string{"HOME=" + r.s.Sandbox.Root, "PATH=/usr/bin:/bin", "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_PAGER=cat", "PAGER=cat", "TZ=UTC", "LANG=C.UTF-8", "GIT_OPTIONAL_LOCKS=0"}+ cmd.Env = []string{"HOME=" + root, "PATH=/usr/bin:/bin", "GIT_DIR=" + filepath.Join(root, ".git"), "GIT_WORK_TREE=" + root,+ "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_PAGER=cat", "PAGER=cat", "TZ=UTC", "LANG=C.UTF-8", "GIT_OPTIONAL_LOCKS=0"}so := &limitedBuffer{n: 100000}cmd.Stdout = io.MultiWriter(hc.Stdout, so)cmd.Stderr = hc.Stderr@@ -282,13 +302,55 @@ func (r *shellRun) execGit(ctx context.Context, hc interp.HandlerContext, args [return exitStatus(exit)}+// gitArg refuses a denied option, and any argument or option value that+// names an existing file outside the workspace: git diff reads two such paths+// even without --no-index, and blame --contents, -O and the like read one.+// Values that name nothing on disk (a pickaxe string such as -S/api/orders)+// pass.+func (r *shellRun) gitArg(dir, a string) error {+ var values []string+ switch {+ case a == "--":+ case strings.HasPrefix(a, "--"):+ name, value, _ := strings.Cut(a, "=")+ if slices.Contains(gitDenied, name) {+ return fmt.Errorf("%s is not available in this sandbox", name)+ }+ values = []string{value}+ case strings.HasPrefix(a, "-"):+ // A short option's value is attached after its letter (-O../x), and+ // letters can be bundled before it (-pO../x).+ for i := 2; i < len(a); i++ {+ values = append(values, a[i:])+ }+ default:+ values = []string{a}+ }+ for _, v := range values {+ if v == "" {+ continue+ }+ if _, err := r.s.Sandbox.Resolve(dir, v); err == nil {+ continue+ }+ p := v+ if !filepath.IsAbs(p) {+ p = filepath.Join(dir, p)+ }+ if _, err := os.Lstat(p); err == nil {+ return fmt.Errorf("%s: %w", v, ErrOutside)+ }+ }+ return nil+}+func (r *shellRun) open(ctx context.Context, path string, flag int, perm os.FileMode) (io.ReadWriteCloser, error) {hc := interp.HandlerCtx(ctx)abs, err := r.s.Sandbox.Resolve(hc.Dir, path)if err != nil {return nil, err}- if flag&(os.O_WRONLY|os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC) != 0 {+ if flag&(os.O_WRONLY|os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC) != 0 && abs != "/dev/null" {if err := r.s.Sandbox.Writable(abs); err != nil {return nil, err}diff --git a/cli/test/evals/internal/tools/shell_test.go b/cli/test/evals/internal/tools/shell_test.goindex eb8f13cf..29939db5 100644--- a/cli/test/evals/internal/tools/shell_test.go+++ b/cli/test/evals/internal/tools/shell_test.go@@ -4,6 +4,7 @@ import ("context""encoding/json""os"+ "os/exec""path/filepath""strings""testing"@@ -45,6 +46,12 @@ func TestShellUtilities(t *testing.T) {{`grep -c beta notes.txt`, "2"},{`wc -l < notes.txt`, "4"},{`X=$(echo nested); echo "got $X"`, "got nested"},+ {`printf 'M\nx\nM\ny\nz\n' | grep -A3 M | tr '\n' ,`, "M,x,M,y,z,"},+ {`printf 'M\na\nb\nc\nM\n' | grep -A1 M | tr '\n' ,`, "M,a,--,M,"},+ {`printf 'a\nb\nM\nc\n' | grep -B1 -n M | tr '\n' ,`, "2-b,3:M,"},+ {`grep -e alpha -e gamma notes.txt | wc -l`, "2"},+ {`echo '{"a":"x"}' | jq .a -r`, "x"},+ {`echo '{"a":{"b":1}}' | jq -c .a`, `{"b":1}`},}for _, c := range cases {r := run(t, s, c.cmd)@@ -77,6 +84,15 @@ func TestShellConfinement(t *testing.T) {}}+// TestShellDevNullIsNotAWrite: discarding stderr is not a file operation,+// so it never counts toward the trajectory grader's repeat limit.+func TestShellDevNullIsNotAWrite(t *testing.T) {+ s, _ := newShell(t)+ if r := run(t, s, `grep beta notes.txt 2>/dev/null >/dev/null`); !r.OK || len(r.Ops) != 0 {+ t.Fatalf("ok=%v ops=%+v", r.OK, r.Ops)+ }+}+func TestShellRecordsTracewayInPipelines(t *testing.T) {s, _ := newShell(t)r := run(t, s, `traceway endpoints list --since 24h --order-by p95 --output json | jq -r '.data[0].endpoint'`)@@ -95,7 +111,7 @@ func TestShellRecordsTracewayInPipelines(t *testing.T) {}}-// TestToolEnvIsScrubbed asserts plan §3.5: secrets in the harness's own+// TestToolEnvIsScrubbed asserts that secrets in the harness's own// environment never reach tool subprocesses.func TestToolEnvIsScrubbed(t *testing.T) {t.Setenv("OPENROUTER_API_KEY", "sk-or-secret")@@ -142,3 +158,76 @@ func TestAskUserRules(t *testing.T) {t.Fatal(got)}}++// TestShellGitIsReadOnly holds git to the workspace: no config written into+// the workspace can make git run a command on the host, and no argument reads+// or writes a file outside it (the run directory around it holds seed.json).+func TestShellGitIsReadOnly(t *testing.T) {+ s, work := newShell(t)+ s.Git = true+ outside := filepath.Dir(work)+ os.WriteFile(filepath.Join(outside, "seed.json"), []byte("answer-key"), 0o644)+ for _, args := range [][]string{{"init", "-q"}, {"add", "notes.txt"}, {"-c", "user.email=x@y", "-c", "user.name=x", "commit", "-qm", "init"}} {+ cmd := exec.Command("git", args...)+ cmd.Dir = work+ if out, err := cmd.CombinedOutput(); err != nil {+ t.Skipf("git unavailable: %v %s", err, out)+ }+ }+ const hook = `[core]\n\tfsmonitor = \"echo x > ` + "`" + `pwd` + "`" + `/../PWNED; false\"\n[diff]\n\texternal = \"echo x > ../PWNED; false\"\n`+ write := &WriteFile{Sandbox: s.Sandbox}+ for _, path := range []string{".git/config", ".GIT/config", "sub/.git/config", "bare/config"} {+ res := write.Call(context.Background(), json.RawMessage(`{"path":"`+path+`","content":"`+hook+`"}`))+ if strings.HasSuffix(path, ".git/config") || strings.HasSuffix(path, ".GIT/config") {+ if res.OK {+ t.Errorf("write_file wrote %s", path)+ }+ }+ }+ if r := run(t, s, `printf '[core]\n\tfsmonitor = "echo x > ../PWNED; false"\n' >> .git/config`); r.OK {+ t.Errorf("the shell wrote .git/config: %q", r.Output)+ }+ // A bare repository's layout in a subdirectory, which git would otherwise+ // discover and take its config from.+ for _, f := range []string{"HEAD", "objects/x", "refs/x"} {+ write.Call(context.Background(), json.RawMessage(`{"path":"bare/`+f+`","content":"ref: refs/heads/main\n"}`))+ }+ if r := run(t, s, `cd bare && git rev-parse --absolute-git-dir`); strings.TrimSpace(r.Output) != filepath.Join(work, ".git") {+ t.Errorf("git in bare/ used %q, want the workspace repository", r.Output)+ }+ for _, cmd := range []string{`git status`, `git diff`, `git log -p -1`, `git show HEAD`, `cd sub && git status`} {+ run(t, s, cmd)+ }+ if _, err := os.Stat(filepath.Join(outside, "PWNED")); err == nil {+ t.Error("a git command ran a command written into the workspace")+ }+ for _, cmd := range []string{+ `git diff --no-index ../seed.json /dev/null`,+ `git diff --no-index -- ../seed.json /dev/null`,+ `git diff ../seed.json /dev/null`,+ `git diff -- ` + filepath.Join(outside, "seed.json") + ` /dev/null`,+ `git blame --contents ../seed.json notes.txt`,+ `git blame --contents=../seed.json notes.txt`,+ `git log -pO../seed.json -1`,+ `git log --output=../out.txt -1`,+ `git diff --output ../out.txt HEAD`,+ `git log --ext-diff -p -1`,+ `git --exec-path=.. status`,+ `git --config-env=core.fsmonitor=HOME status`,+ `git -c core.pager=cat log`,+ `cd .. && git diff seed.json /dev/null`,+ } {+ r := run(t, s, cmd)+ if r.OK || strings.Contains(r.Output, "answer-key") {+ t.Errorf("%s: ok=%v %q", cmd, r.OK, r.Output)+ }+ }+ if _, err := os.Stat(filepath.Join(outside, "out.txt")); err == nil {+ t.Error("git wrote a file outside the workspace")+ }+ for _, cmd := range []string{`git log --oneline | wc -l`, `git --no-pager log --oneline -S/api/orders | wc -l`, `git log --format=%s -- notes.txt`, `cd .skills && git status --short`} {+ if r := run(t, s, cmd); !r.OK {+ t.Errorf("%s: %q", cmd, r.Output)+ }+ }+}diff --git a/cli/test/evals/internal/tools/submit.go b/cli/test/evals/internal/tools/submit.goindex 44c080e9..2e9f7c70 100644--- a/cli/test/evals/internal/tools/submit.go+++ b/cli/test/evals/internal/tools/submit.go@@ -13,22 +13,18 @@ import ("github.com/tracewayapp/traceway/cli/test/evals/internal/record")-// Submit names.-const (- SubmitPerf = "submit_perf_diagnosis"- SubmitDebug = "submit_debug_diagnosis"- Finish = "finish"-)+// SubmitPerf is the perf suite's final-answer tool.+const SubmitPerf = "submit_perf_diagnosis"-// DebugCategories are submit_debug_diagnosis's category values.-var DebugCategories = []string{"null_reference", "missing_key", "type_or_contract_mismatch", "race_or_duplicate", "timeout", "config", "resource_exhaustion", "logic", "unknown"}+// IsSubmit reports whether a tool is a final-answer tool.+func IsSubmit(name string) bool { return name == SubmitPerf }// Submit is a final-answer tool. It validates the payload against its// schema and the call ids it cites; a failure goes back to the model for// repair, up to MaxRepairs times, after which the last payload is accepted// as-is for grading.type Submit struct {- Kind string // SubmitPerf | SubmitDebug | Finish+ Kind string // the tool's name: SubmitPerfCallExists func(id string) (ok, exists bool)MaxRepairs int@@ -37,63 +33,31 @@ type Submit struct {}func (t *Submit) Spec() Spec {- switch t.Kind {- case SubmitPerf:- var rows []string- for _, r := range checklist.Rows() {- rows = append(rows, fmt.Sprintf("%s (%s)", r.ID, r.Title))- }- ids := checklist.IDs()- item := map[string]any{"type": "string", "enum": ids}- return Spec{Name: SubmitPerf, Description: "Submit your final performance diagnosis. This ends the task. " +- "checklist_item is the bottleneck checklist row that explains the latency, or none_found when the latency is within expectations, or insufficient_evidence when the telemetry cannot tell. Rows: " + strings.Join(rows, "; ") + ". " +- "Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",- Parameters: obj(map[string]any{- "checklist_item": item,- "category": strProp("database, application, network, caching, infrastructure, or none"),- "locus": obj(map[string]any{- "endpoint": strProp("Endpoint name, e.g. GET /api/orders"),- "span": strProp("The dominating span or query"),- "service": strProp("Service name"),- "host": strProp("Host or instance, when the cause is host-specific"),- }),- "onset": map[string]any{"type": []string{"string", "null"}, "description": "RFC 3339 time the slowness started, or null"},- "evidence": evidenceSchema(),- "ruled_out": map[string]any{"type": "array", "items": obj(map[string]any{"checklist_item": item, "call_id": strProp("Call id"), "reason": strProp("Why it is ruled out")}, "checklist_item", "reason")},- "fix": strProp("The concrete fix"),- "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},- "report": strProp("Your report to the user, in Markdown"),- }, "checklist_item", "locus", "evidence", "fix", "confidence", "report")}- case SubmitDebug:- return Spec{Name: SubmitDebug, Description: "Submit your final debugging diagnosis. This ends the task. Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",- Parameters: obj(map[string]any{- "issue_hash": strProp("The exception group hash (16 hex), if the bug has one"),- "occurrence_id": strProp("The occurrence you anchored on, if any"),- "category": map[string]any{"type": "string", "enum": DebugCategories},- "culprit": obj(map[string]any{- "file": strProp("Workspace-relative file"),- "symbol": strProp("Function or method"),- "line": intProp("Line number"),- }, "file"),- "evidence": evidenceSchema(),- "fix_summary": strProp("What you changed or propose to change"),- "files_changed": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},- "needs_instrumentation": map[string]any{"type": "boolean", "description": "True when the telemetry cannot explain the failure and you added instrumentation instead of a fix"},- "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},- "report": strProp("Your report to the user, in Markdown"),- }, "category", "culprit", "evidence", "fix_summary", "needs_instrumentation", "confidence", "report")}- default:- return Spec{Name: Finish, Description: "Finish the task and report what you did. This ends the task.",- Parameters: obj(map[string]any{- "summary": strProp("Your report to the user, in Markdown"),- "changes": map[string]any{"type": "array", "items": obj(map[string]any{- "action": strProp("What you changed, e.g. snooze alert rule"),- "target": strProp("The record, e.g. rule 12"),- "new_value": strProp("The new state"),- "undo": strProp("How to undo it"),- }, "action", "target")},- }, "summary", "changes")}+ var rows []string+ for _, r := range checklist.Rows() {+ rows = append(rows, fmt.Sprintf("%s (%s)", r.ID, r.Title))}+ ids := checklist.IDs()+ item := map[string]any{"type": "string", "enum": ids}+ return Spec{Name: SubmitPerf, Description: "Submit your final performance diagnosis. This ends the task. " ++ "checklist_item is the bottleneck checklist row that explains the latency, or none_found when the latency is within expectations, or insufficient_evidence when the telemetry cannot tell. Rows: " + strings.Join(rows, "; ") + ". " ++ "Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",+ Parameters: obj(map[string]any{+ "checklist_item": item,+ "category": strProp("database, application, network, caching, infrastructure, or none"),+ "locus": obj(map[string]any{+ "endpoint": strProp("Endpoint name, e.g. GET /api/orders"),+ "span": strProp("The dominating span or query"),+ "service": strProp("Service name"),+ "host": strProp("Host or instance, when the cause is host-specific"),+ }),+ "onset": map[string]any{"type": []string{"string", "null"}, "description": "RFC 3339 time the slowness started, or null"},+ "evidence": evidenceSchema(),+ "ruled_out": map[string]any{"type": "array", "items": obj(map[string]any{"checklist_item": item, "call_id": strProp("Call id"), "reason": strProp("Why it is ruled out")}, "checklist_item", "reason")},+ "fix": strProp("The concrete fix"),+ "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},+ "report": strProp("Your report to the user, in Markdown"),+ }, "checklist_item", "locus", "evidence", "fix", "confidence", "report")}}func evidenceSchema() map[string]any {@@ -165,35 +129,19 @@ func (t *Submit) validate(p map[string]any) []string {}}}- switch t.Kind {- case SubmitPerf:- need("checklist_item", "locus", "evidence", "fix", "confidence", "report")- if v := str("checklist_item"); v != "" && !checklist.IsID(v) {- probs = append(probs, fmt.Sprintf("checklist_item %q is not one of the allowed values", v))- }- if rs, ok := p["ruled_out"].([]any); ok {- for i, r := range rs {- m, _ := r.(map[string]any)- if v, _ := m["checklist_item"].(string); v != "" && !checklist.IsID(v) {- probs = append(probs, fmt.Sprintf("ruled_out[%d].checklist_item %q is not an allowed value", i, v))- }- }- }- checkEvidence()- case SubmitDebug:- need("category", "culprit", "evidence", "fix_summary", "needs_instrumentation", "confidence", "report")- if v := str("category"); v != "" && !slices.Contains(DebugCategories, v) {- probs = append(probs, fmt.Sprintf("category %q is not one of %v", v, DebugCategories))- }- if c, ok := p["culprit"].(map[string]any); ok {- if f, _ := c["file"].(string); f == "" {- probs = append(probs, "culprit.file is required")+ need("checklist_item", "locus", "evidence", "fix", "confidence", "report")+ if v := str("checklist_item"); v != "" && !checklist.IsID(v) {+ probs = append(probs, fmt.Sprintf("checklist_item %q is not one of the allowed values", v))+ }+ if rs, ok := p["ruled_out"].([]any); ok {+ for i, r := range rs {+ m, _ := r.(map[string]any)+ if v, _ := m["checklist_item"].(string); v != "" && !checklist.IsID(v) {+ probs = append(probs, fmt.Sprintf("ruled_out[%d].checklist_item %q is not an allowed value", i, v))}}- checkEvidence()- default:- need("summary", "changes")}+ checkEvidence()if c := str("confidence"); c != "" && !slices.Contains([]string{"low", "medium", "high"}, c) {probs = append(probs, "confidence must be low, medium or high")}diff --git a/cli/test/evals/internal/tools/tools.go b/cli/test/evals/internal/tools/tools.goindex 781be5e8..33de377c 100644--- a/cli/test/evals/internal/tools/tools.go+++ b/cli/test/evals/internal/tools/tools.go@@ -80,33 +80,16 @@ func (s *Sandbox) Resolve(base, p string) (string, error) {} else {abs = filepath.Clean(filepath.Join(base, p))}- if !within(s.Root, abs) {+ if !within(s.Root, abs) || !within(realPath(s.Root), realPath(abs)) {return "", fmt.Errorf("%s: %w", p, ErrOutside)}- // Resolve the longest existing prefix to catch symlink escapes.- probe := abs- for {- real, err := filepath.EvalSymlinks(probe)- if err == nil {- rootReal, _ := filepath.EvalSymlinks(s.Root)- if rootReal == "" {- rootReal = s.Root- }- if !within(rootReal, real) {- return "", fmt.Errorf("%s: %w", p, ErrOutside)- }- break- }- parent := filepath.Dir(probe)- if parent == probe || !within(s.Root, parent) {- break- }- probe = parent- }return abs, nil}-// Writable reports whether abs may be written.+// Writable reports whether abs may be written. Anything under a .git+// directory is read-only wherever it sits: git reads its config, so a write+// there could make a git command run a program on the host. Both the path and+// what a symlink makes of it are checked, without regard to case.func (s *Sandbox) Writable(abs string) error {if abs == "/dev/null" {return nil@@ -115,14 +98,48 @@ func (s *Sandbox) Writable(abs string) error {if err != nil {return err}+ if err := s.readOnly(rel); err != nil {+ return err+ }+ if real, err := filepath.Rel(realPath(s.Root), realPath(abs)); err == nil {+ return s.readOnly(real)+ }+ return nil+}++func (s *Sandbox) readOnly(rel string) error {+ rel = filepath.ToSlash(rel)+ lower := strings.ToLower(rel)+ for _, part := range strings.Split(lower, "/") {+ if part == ".git" {+ return fmt.Errorf("%s is read-only", rel)+ }+ }for _, ro := range s.ReadOnly {- if rel == ro || strings.HasPrefix(rel, ro+string(filepath.Separator)) {- return fmt.Errorf("%s is read-only", filepath.ToSlash(rel))+ ro = strings.ToLower(ro)+ if lower == ro || strings.HasPrefix(lower, ro+"/") {+ return fmt.Errorf("%s is read-only", rel)}}return nil}+// realPath resolves the symlinks in the longest existing prefix of p.+func realPath(p string) string {+ rest := ""+ for probe := p; ; {+ if real, err := filepath.EvalSymlinks(probe); err == nil {+ return filepath.Join(real, rest)+ }+ parent := filepath.Dir(probe)+ if parent == probe {+ return p+ }+ rest = filepath.Join(filepath.Base(probe), rest)+ probe = parent+ }+}+// Rel returns the workspace-relative form of abs.func (s *Sandbox) Rel(abs string) string {rel, err := filepath.Rel(s.Root, abs)diff --git a/cli/test/evals/models.yaml b/cli/test/evals/models.yamlindex d7ffb190..d6ff02e3 100644--- a/cli/test/evals/models.yaml+++ b/cli/test/evals/models.yaml@@ -1,14 +1,15 @@-# Pinned OpenRouter models for the eval agents and the judge. Every call goes-# through OpenRouter Chat Completions with one key (OPENROUTER_API_KEY).+# Pinned OpenRouter models for the eval agents and the judge, for paid runs+# through OpenRouter Chat Completions. This harness makes no paid calls: it+# plays oracles with the fake model and only records these ids in the+# results header.## Never use a floating alias (~provider/model-latest): results must stay# comparable over time, so a model change is a reviewed edit to this file.# Prices and supported_parameters were checked against-# GET https://openrouter.ai/api/v1/models on 2026-10-17. The preflight reads live-# prices; fallback_price is used only when that list cannot be fetched.+# GET https://openrouter.ai/api/v1/models on 2026-10-17.#-# estimate is the per-run token envelope the preflight multiplies by live-# prices until baseline.json carries measured averages (plan §10).+# estimate (a per-run token envelope) and fallback_price (USD per million+# tokens) are for estimating a paid run's cost before it starts.agents:opus:diff --git a/cli/test/evals/presets.yaml b/cli/test/evals/presets.yamlindex 4352c574..f0b4c98b 100644--- a/cli/test/evals/presets.yaml+++ b/cli/test/evals/presets.yaml@@ -1,4 +1,4 @@-# Case and arm selections for `twevals plan|run --preset`. A --cases glob+# Case and arm selections for `twevals list --preset`. A --cases glob# overrides the preset's case filter; --arms overrides its arm policy.presets:smoke: