Skip to content

The only tool you need to know what is happening and how to fix it.

Fixes

narrow-adventure866f0f295138
diff --git a/cli/test/evals/README.md b/cli/test/evals/README.md
index 356b459c..e1ae5e4d 100644
--- a/cli/test/evals/README.md
+++ b/cli/test/evals/README.md
@@ -32,12 +32,13 @@ commands:
- `twevals list [--preset P] [--cases GLOB] [--arms A]` prints the case × arm
combinations a selection picks.
-- `twevals oracle [--all | <case-id>...] [--arms A] [-j N] [--keep]` plays
- the oracles and writes `results.json` and `summary.md`. It exits `1` when
- an oracle fails.
+- `twevals oracle [--all | <case-id>... | --suites S] [--arms A] [-j N]
+ [--keep] [--out DIR]` plays the oracles and writes `results.json` and
+ `summary.md` to `--out` (default `.cache/oracle/<timestamp>/`). It exits `1`
+ when an oracle fails.
-Presets (`presets.yaml`): `smoke` is the starred cases on the primary arm,
-`core` every case on the primary arm, `full` every arm a case lists.
+Presets (`presets.yaml`): `smoke` is the cases tagged `smoke` on the primary
+arm, `core` every case on the primary arm, `full` every arm a case lists.
Requirements: Go (the `go`/`toolchain` lines of `go.mod`) and git. The
harness builds the backend and both CLI editions from this checkout into
@@ -48,7 +49,8 @@ harness builds the backend and both CLI editions from this checkout into
A run is `pass`, `fail`, `budget_exhausted` (counted as a fail) or
`infra_error` (excluded from pass rates).
-Each run directory, `runs/<case>/<arm>/<model>/r<N>/`, holds:
+Each run directory, `runs/<case>/<arm>/<model>/r<N>/` (with `+` in the arm
+written as `_`, so `cli_skill`), holds:
- `transcript.jsonl`: every model response, every tool call with its id, op,
argv, exit code and output;
@@ -146,9 +148,12 @@ func seedNPlusOne(s *seed.Scenario) error {
Backend facts that bite (each one cost a case author an afternoon):
-- Health-check routes (`/health`, `/healthz`, `/ping`, `/ready`, ...) are
- dropped at ingest, so an endpoint seeded on them never appears and `Verify`
- fails with "endpoints never became visible".
+- Successful `GET`/`HEAD` requests to health-check routes (`/health`,
+ `/healthz`, `/ping`, `/ready`, `/status`, `/up`, ...; the list is in
+ `backend/app/services/healthcheck.service.go`) are dropped at ingest, which
+ is the projects' default. An endpoint seeded on one never appears and
+ `Verify` fails with "endpoints never became visible"; a failing one (status
+ 400 or more) is kept.
- `endpoints list --order-by p95|lastSeen` does not sort: the CLI sends
`p95`/`lastSeen`, the SQLite/DuckDB repositories only know
`p95_duration`/`last_seen`, and the list falls back to impact order. Keep
@@ -176,7 +181,8 @@ Backend facts that bite (each one cost a case author an afternoon):
See `perf-db-n-plus-1.yaml` for the full shape. The keys:
- `arms` and `invoke`: `slash` preloads SKILL.md the way a `/traceway`
- command does, and `natural` makes the agent load the skill itself.
+ command does, and `natural` makes the agent load the skill itself. A slash
+ case's prompt starts with `/traceway`, and only a slash case's does.
- `workspace.repo`: a fixture under `testdata/repos`. Fixtures can carry
history: a `history.yaml` whose commits are overlays under `.history/`
(only the files that differ from the fixture, plus an `absent:` list of
@@ -229,6 +235,8 @@ reached` or Postgres's `deadlock detected`, and neutral names.
```
models.yaml presets.yaml
cmd/twevals/ list | oracle
+internal/catalog/ the case schema, presets, models.yaml
+internal/checklist/ the bottleneck rows of performance.md, the perf answer enum
internal/env/ backend subprocess, registration, PATs, CLI logins, the stub
internal/seed/ the DSL, OTLP and /api/report writers, read-back verification
internal/scenarios/ one seed function per case
@@ -236,15 +244,19 @@ internal/agent/ the Chat Completions client (loopback only), the loop, pr
internal/tools/ the sandboxed shell (mvdan.cc/sh, in-process coreutils, gojq, goawk), files, skills, ask_user, submit
internal/ops/ CLI → operation mapping, cross-checked against tool-map.md
internal/grade/ answer, trajectory, grounding, report, leakage
+internal/record/ the run record: calls, operations, usage
internal/runner/ one run end to end
internal/report/ results.json and summary.md
+internal/suite/ catalog-wide tests
cases/ testdata/ cases, fixture repos, oracle files
```
The agent's shell is an interpreter, never the host shell. It has the
`traceway` binary (logged in with a read PAT whose config lives outside the
workspace), `jq`, `awk`, read-only text utilities confined to the workspace,
-and read-only `git`. Tool subprocesses get a scrubbed environment (`HOME`,
+and read-only `git`. No tool writes under a `.git` directory, git always
+uses the workspace's repository, and git refuses arguments that name a file
+outside the workspace or options that write files or run programs. Tool subprocesses get a scrubbed environment (`HOME`,
`XDG_*`, `PATH`, `TZ`, `LANG`), so secrets in the harness's environment
never reach them.
@@ -252,8 +264,9 @@ never reach them.
The first performance scenarios were contributed by Liyu in PR #405
("feat(skills): add performance skill evals"). They were ported to this
-harness as `perf-db-n-plus-1`, `perf-db-missing-index`, `perf-net-sequential`,
+module as `perf-db-n-plus-1`, `perf-db-missing-index`, `perf-net-sequential`,
`perf-db-pool-exhaustion`, `perf-app-sync-background`, `perf-net-large-payload`,
`perf-cache-no-cache`, `perf-infra-cpu-saturation`, `perf-cache-stampede`,
`perf-xs-ios-watchdog`, `perf-xs-android-timeouts`, `debug-android-npe` and
-`debug-flutter-null-check`.
+`debug-flutter-null-check`. The two debug cases sit in `cases/debug/` and are
+not loaded yet: the harness runs the perf suite only.
diff --git a/cli/test/evals/cmd/twevals/commands.go b/cli/test/evals/cmd/twevals/commands.go
index a8897e35..44772507 100644
--- a/cli/test/evals/cmd/twevals/commands.go
+++ b/cli/test/evals/cmd/twevals/commands.go
@@ -84,13 +84,6 @@ func (s *selFlags) bind(fs *flag.FlagSet) {
fs.StringVar(&s.sel.Arms, "arms", "", "comma-separated arms")
}
-func (s *selFlags) selection() catalog.Selection {
- if s.sel.Preset == "" && s.sel.Cases == "" && s.sel.Suites == "" && s.sel.Tags == "" {
- s.sel.Preset = "smoke"
- }
- return s.sel
-}
-
func cmdList(args []string) error {
fs := flag.NewFlagSet("list", flag.ExitOnError)
var sf selFlags
@@ -158,6 +151,7 @@ func cmdOracle(ctx context.Context, args []string) error {
Runs: 1, Jobs: *jobs, Keep: *keep, Log: os.Stderr}
header := report.NewHeader(ws.repo, filepath.Base(outDir), ws.models, bins.Hash, os.Args)
header.Oracle = true
+ header.Models = map[string]string{oracleModel.Key: oracleModel.ID}
results := runner.Execute(ctx, opt, runner.Jobs(units, opt.Models, 1))
header.Finished = time.Now().UTC()
if err := report.Save(outDir, &report.Results{Header: header, Runs: results}); err != nil {
diff --git a/cli/test/evals/cmd/twevals/main.go b/cli/test/evals/cmd/twevals/main.go
index 71e01b66..e0f4eca5 100644
--- a/cli/test/evals/cmd/twevals/main.go
+++ b/cli/test/evals/cmd/twevals/main.go
@@ -14,10 +14,12 @@ import (
const usage = `twevals: Traceway skill and MCP evals (oracle runs only, free)
twevals list [--preset P] [--suites S] [--tags T] [--cases GLOB] [--arms A]
- twevals oracle [--all | <case-id>...] [--arms A] [-j N] [--keep]
+ twevals oracle [--all | <case-id>... | --suites S] [--arms A] [-j N] [--keep] [--out DIR]
Selection flags: --preset smoke|core|full, --cases (comma-separated ids or globs; overrides the preset), --suites, --tags, --arms.
-Exit codes: 0 ok, 1 failed oracle.
+list defaults to --preset full. oracle writes results.json and summary.md to --out
+(default .cache/oracle/<timestamp>).
+Exit codes: 0 ok, 1 a failed oracle or an error, 2 bad usage.
`
func main() {
@@ -38,7 +40,7 @@ func main() {
fmt.Print(usage)
return
default:
- err = fmt.Errorf("unknown command %q", os.Args[1])
+ err = &exitError{2, fmt.Sprintf("unknown command %q (twevals help lists them)", os.Args[1])}
}
if err != nil {
var ex *exitError
diff --git a/cli/test/evals/internal/agent/agent_test.go b/cli/test/evals/internal/agent/agent_test.go
index 6e0ff134..76ed7d53 100644
--- a/cli/test/evals/internal/agent/agent_test.go
+++ b/cli/test/evals/internal/agent/agent_test.go
@@ -62,18 +62,23 @@ func (echoTool) Call(context.Context, json.RawMessage) tools.Result {
return tools.Result{Output: "echoed", OK: true}
}
+// submission is a valid submit_perf_diagnosis payload whose report is r.
+func submission(r string) string {
+ return `{"checklist_item":"none_found","locus":{},"evidence":[{"call_id":"c01","claim":"x"}],"fix":"none","confidence":"low","report":"` + r + `"}`
+}
+
func cfg(s *scripted) Config {
return Config{
- Model: &catalog.Model{Key: "m", ID: "x/y", Caching: "anthropic"}, Chat: s, System: "sys", Prompt: "do it", SubmitTool: tools.Finish,
- Tools: []tools.Tool{echoTool{}, &tools.Submit{Kind: tools.Finish, MaxRepairs: 2}},
+ Model: &catalog.Model{Key: "m", ID: "x/y", Caching: "anthropic"}, Chat: s, System: "sys", Prompt: "do it", SubmitTool: tools.SubmitPerf,
+ Tools: []tools.Tool{echoTool{}, &tools.Submit{Kind: tools.SubmitPerf, MaxRepairs: 2}},
Budget: catalog.Budget{MaxTurns: 5, MaxToolCalls: 3},
}
}
func TestLoopSubmits(t *testing.T) {
- s := &scripted{resps: []*Response{toolResp("echo", `{}`, 0.01), toolResp("finish", `{"summary":"done","changes":[]}`, 0.01)}}
+ s := &scripted{resps: []*Response{toolResp("echo", `{}`, 0.01), toolResp(tools.SubmitPerf, submission("done"), 0.01)}}
run := Run(context.Background(), cfg(s))
- if run.StopReason != StopSubmitted || run.Submission["summary"] != "done" || len(run.Calls) != 2 || run.Calls[0].ID != "c01" {
+ if run.StopReason != StopSubmitted || run.Submission["report"] != "done" || len(run.Calls) != 2 || run.Calls[0].ID != "c01" {
t.Fatalf("run = %+v", run)
}
if run.Usage.CostUSD != 0.02 {
@@ -104,7 +109,7 @@ func TestLoopNudgesOnceThenStops(t *testing.T) {
t.Fatalf("stop %s after %d requests", run.StopReason, len(s.reqs))
}
last := s.reqs[1].Messages
- if !strings.Contains(textOf(last[len(last)-1]), "call finish") {
+ if !strings.Contains(textOf(last[len(last)-1]), "call "+tools.SubmitPerf) {
t.Fatal("the nudge must name the submit tool")
}
}
@@ -140,7 +145,7 @@ func TestLoopBudgets(t *testing.T) {
}
func TestUnknownToolIsAnError(t *testing.T) {
- s := &scripted{resps: []*Response{toolResp("nope", `{}`, 0), toolResp("finish", `{"summary":"x","changes":[]}`, 0)}}
+ s := &scripted{resps: []*Response{toolResp("nope", `{}`, 0), toolResp(tools.SubmitPerf, submission("x"), 0)}}
run := Run(context.Background(), cfg(s))
if run.Calls[0].OK || !strings.Contains(run.Calls[0].Output, "no tool named") {
t.Fatalf("%+v", run.Calls[0])
@@ -193,8 +198,31 @@ func TestPromptHelpers(t *testing.T) {
if SlashSkill("/traceway debug x") != "traceway" || SlashSkill("why is it slow") != "" {
t.Fatal("SlashSkill")
}
- p := SystemPrompt(PromptInput{SubmitTool: "finish", Skills: []tools.Skill{{Name: "traceway", Description: "d"}}})
- if !strings.Contains(p, "- traceway: d") || !strings.Contains(p, "call finish") {
+ p := SystemPrompt(PromptInput{SubmitTool: tools.SubmitPerf, Skills: []tools.Skill{{Name: "traceway", Description: "d"}}})
+ if !strings.Contains(p, "- traceway: d") || !strings.Contains(p, "call "+tools.SubmitPerf) {
t.Fatal(p)
}
}
+
+// TestFakeRuntimeTemplates: numbers looked up from a tool result keep the
+// text they had there, at any size.
+func TestFakeRuntimeTemplates(t *testing.T) {
+ msgs := []Message{{Role: "tool", Content: "[call c01]\n{\"d\":45200000,\"t\":1760000000123456789,\"f\":0.25,\"o\":{\"a\":[1,2]}}\n[budget: 3 tool calls and 4 turns left]"}}
+ got, err := resolveRuntime("--min {{last.d}} --at {{calls.c01.t}} --r {{last.f}} {{last.o}}", msgs)
+ if err != nil || got != `--min 45200000 --at 1760000000123456789 --r 0.25 {"a":[1,2]}` {
+ t.Fatalf("got %q, %v", got, err)
+ }
+ args, err := resolveArgs(map[string]any{"n": "{{last.d}}"}, msgs)
+ if raw, _ := json.Marshal(args); err != nil || string(raw) != `{"n":45200000}` {
+ t.Fatalf("whole-string template: %s, %v", raw, err)
+ }
+}
+
+func TestFakeSubmitTool(t *testing.T) {
+ if got := submitTool(map[string]bool{"shell": true, tools.SubmitPerf: true, "read_file": true}); got != tools.SubmitPerf {
+ t.Fatalf("submitTool = %q", got)
+ }
+ if got := submitTool(map[string]bool{"shell": true}); got != "" {
+ t.Fatalf("submitTool without one = %q", got)
+ }
+}
diff --git a/cli/test/evals/internal/agent/fake.go b/cli/test/evals/internal/agent/fake.go
index 85556549..41bd0e6c 100644
--- a/cli/test/evals/internal/agent/fake.go
+++ b/cli/test/evals/internal/agent/fake.go
@@ -3,6 +3,7 @@ package agent
import (
"encoding/json"
"fmt"
+ "io"
"net/http"
"net/http/httptest"
"os"
@@ -13,6 +14,7 @@ import (
"sync"
"github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"
)
// Fake is an httptest server speaking Chat Completions that plays a case's
@@ -138,17 +140,32 @@ func (f *Fake) render(s catalog.Step, tools map[string]bool, msgs []Message) (st
}
return need(tools, "ask_user", map[string]any{"question": q})
case s.Submit != nil:
- for name := range tools {
- if strings.HasPrefix(name, "submit_") {
- args, err := resolveArgs(s.Submit, msgs)
- return name, args, "", err
- }
+ name := submitTool(tools)
+ if name == "" {
+ return "", nil, "", fmt.Errorf("no submit tool on this arm")
}
- return "", nil, "", fmt.Errorf("no submit tool on this arm")
+ args, err := resolveArgs(s.Submit, msgs)
+ return name, args, "", err
}
return "", nil, "", fmt.Errorf("empty oracle step")
}
+// submitTool names the arm's final-answer tool, the first by name so the
+// choice never depends on map order.
+func submitTool(offered map[string]bool) string {
+ var names []string
+ for n := range offered {
+ if tools.IsSubmit(n) {
+ names = append(names, n)
+ }
+ }
+ sort.Strings(names)
+ if len(names) == 0 {
+ return ""
+ }
+ return names[0]
+}
+
func need(tools map[string]bool, name string, args map[string]any) (string, map[string]any, string, error) {
if !tools[name] {
return "", nil, "", fmt.Errorf("this arm has no %s tool", name)
@@ -168,11 +185,27 @@ func resolveRuntime(s string, msgs []Message) (string, error) {
if err != nil && firstErr == nil {
firstErr = err
}
- return fmt.Sprint(v)
+ return runtimeText(v)
})
return out, firstErr
}
+// runtimeText renders a looked-up value inside a string as it reads in the
+// tool output: numbers as their JSON text (never 4.52e+07), objects as JSON.
+func runtimeText(v any) string {
+ switch t := v.(type) {
+ case string:
+ return t
+ case json.Number:
+ return t.String()
+ case map[string]any, []any:
+ raw, _ := json.Marshal(t)
+ return string(raw)
+ default:
+ return fmt.Sprint(t)
+ }
+}
+
func resolveArgs(in map[string]any, msgs []Message) (map[string]any, error) {
out := map[string]any{}
var firstErr error
@@ -256,8 +289,15 @@ func lookupRuntime(expr string, msgs []Message) (any, error) {
if path == "" {
return body, nil
}
+ // UseNumber keeps numbers as written: a nanosecond timestamp does not
+ // survive a float64.
var v any
- if err := json.Unmarshal([]byte(firstJSON(body)), &v); err != nil {
+ dec := json.NewDecoder(strings.NewReader(firstJSON(body)))
+ dec.UseNumber()
+ if err := dec.Decode(&v); err != nil {
+ return nil, fmt.Errorf("template %s: result is not JSON", expr)
+ }
+ if _, err := dec.Token(); err != io.EOF {
return nil, fmt.Errorf("template %s: result is not JSON", expr)
}
for _, seg := range splitPath(path) {
diff --git a/cli/test/evals/internal/agent/loop.go b/cli/test/evals/internal/agent/loop.go
index 9cff574c..5e3deae3 100644
--- a/cli/test/evals/internal/agent/loop.go
+++ b/cli/test/evals/internal/agent/loop.go
@@ -150,7 +150,7 @@ func Run(ctx context.Context, cfg Config) *record.Run {
case final:
res = tools.Result{Output: "not executed: the task was already submitted"}
c.Rejected = true
- case callN > budget.MaxToolCalls && !isSubmit(tc.Function.Name):
+ case callN > budget.MaxToolCalls && !tools.IsSubmit(tc.Function.Name):
res = tools.Result{Output: fmt.Sprintf("not executed: the tool-call budget (%d) is used up. Call %s now.", budget.MaxToolCalls, cfg.SubmitTool)}
c.Rejected = true
default:
@@ -190,10 +190,6 @@ func Run(ctx context.Context, cfg Config) *record.Run {
}
}
-func isSubmit(name string) bool {
- return name == tools.SubmitPerf || name == tools.SubmitDebug || name == tools.Finish
-}
-
func nonEmpty(s string) string {
if strings.TrimSpace(s) == "" {
return "{}"
diff --git a/cli/test/evals/internal/catalog/case.go b/cli/test/evals/internal/catalog/case.go
index d0ee8b24..b74d7a49 100644
--- a/cli/test/evals/internal/catalog/case.go
+++ b/cli/test/evals/internal/catalog/case.go
@@ -133,7 +133,7 @@ type Onset struct {
ToleranceMinutes int `yaml:"tolerance_minutes"`
}
-// Trajectory is the tool-call DSL (plan §6.4).
+// Trajectory is the tool-call DSL.
type Trajectory struct {
Required []CallMatcher `yaml:"required"`
AnyOf [][]CallMatcher `yaml:"any_of"`
@@ -241,6 +241,11 @@ func (c *Case) Validate() error {
default:
add("invoke must be slash or natural, got %q", c.Invoke)
}
+ // The runner preloads the skill for slash cases, and the prompt is what
+ // the agent reads, so the two must agree.
+ if slash := strings.HasPrefix(strings.TrimSpace(c.Prompt), "/"); slash != (c.Invoke == "slash") {
+ add("invoke: slash goes with a prompt that starts with /<skill>, and only with one")
+ }
if strings.TrimSpace(c.Prompt) == "" {
add("prompt is required")
}
diff --git a/cli/test/evals/internal/catalog/case_test.go b/cli/test/evals/internal/catalog/case_test.go
new file mode 100644
index 00000000..25ebe46d
--- /dev/null
+++ b/cli/test/evals/internal/catalog/case_test.go
@@ -0,0 +1,28 @@
+package catalog
+
+import (
+ "strings"
+ "testing"
+)
+
+// TestInvokeMatchesPrompt: the runner preloads the skill for invoke: slash,
+// so a case's invoke and its prompt must agree.
+func TestInvokeMatchesPrompt(t *testing.T) {
+ for _, c := range []struct {
+ invoke, prompt string
+ ok bool
+ }{
+ {"slash", "/traceway perf GET /api/orders", true},
+ {"natural", "Checkout got slow this week.", true},
+ {"", "Checkout got slow this week.", true},
+ {"slash", "Checkout got slow this week.", false},
+ {"natural", "/traceway perf GET /api/orders", false},
+ } {
+ cs := &Case{ID: "perf-x", Suite: "perf", Seed: "perf/x", Arms: []string{ArmCLISkill}, Invoke: c.invoke, Prompt: c.prompt,
+ Oracle: []Step{{Shell: "true"}}, Expect: Expect{Answer: &Answer{ChecklistItem: "none_found"}}, Distractor: "x"}
+ err := cs.Validate()
+ if got := err == nil || !strings.Contains(err.Error(), "invoke: slash"); got != c.ok {
+ t.Errorf("invoke %q, prompt %q: %v", c.invoke, c.prompt, err)
+ }
+ }
+}
diff --git a/cli/test/evals/internal/catalog/models.go b/cli/test/evals/internal/catalog/models.go
index a4f825a1..da98cd61 100644
--- a/cli/test/evals/internal/catalog/models.go
+++ b/cli/test/evals/internal/catalog/models.go
@@ -30,11 +30,10 @@ type Model struct {
// MaxTokensPerTurn caps one completion.
MaxTokensPerTurn int `yaml:"max_tokens_per_turn"`
Temperature *float64 `yaml:"temperature"`
- // Estimate is the per-run token envelope used by the preflight until
- // baseline.json carries measured averages.
+ // Estimate is the per-run token envelope for estimating a paid run's
+ // cost. The oracle harness does not read it.
Estimate Estimate `yaml:"estimate"`
- // Fallback prices (USD per million tokens) used when the live price list
- // cannot be fetched. The preflight says when it fell back.
+ // FallbackPrice is USD per million tokens, for the same estimate.
FallbackPrice Price `yaml:"fallback_price"`
}
diff --git a/cli/test/evals/internal/catalog/select.go b/cli/test/evals/internal/catalog/select.go
index aa414f83..25fba0a9 100644
--- a/cli/test/evals/internal/catalog/select.go
+++ b/cli/test/evals/internal/catalog/select.go
@@ -59,7 +59,6 @@ type Selection struct {
Suites string // comma-separated
Tags string // comma-separated, all must match
Arms string // comma-separated
- Shard string // one suite
}
// Unit is one (case, arm) pair; the runner multiplies it by models and runs.
@@ -102,9 +101,6 @@ func Select(cases []*Case, presets *Presets, sel Selection) ([]Unit, error) {
return nil, fmt.Errorf("unknown suite %q", s)
}
}
- if sel.Shard != "" && !slices.Contains(Suites, sel.Shard) {
- return nil, fmt.Errorf("unknown shard %q (shards are suites)", sel.Shard)
- }
var out []Unit
matchedGlob := map[string]bool{}
@@ -127,9 +123,6 @@ func Select(cases []*Case, presets *Presets, sel Selection) ([]Unit, error) {
if len(suites) > 0 && !slices.Contains(suites, c.Suite) {
continue
}
- if sel.Shard != "" && c.Suite != sel.Shard {
- continue
- }
if !hasAll(c, tags) {
continue
}
diff --git a/cli/test/evals/internal/checklist/checklist.go b/cli/test/evals/internal/checklist/checklist.go
index 7b40c83a..1e1558e6 100644
--- a/cli/test/evals/internal/checklist/checklist.go
+++ b/cli/test/evals/internal/checklist/checklist.go
@@ -29,7 +29,7 @@ var (
checklist []Row
)
-// Checklist parses the canonical performance.md once. The enum therefore
+// Rows parses the canonical performance.md once. The enum therefore
// tracks the skill: renaming a row renames its id, and the case test fails.
func Rows() []Row {
checklistOnce.Do(func() { checklist = ParseChecklist(knowledge.MustRead("performance.md")) })
diff --git a/cli/test/evals/internal/grade/answer.go b/cli/test/evals/internal/grade/answer.go
index e2e9e5c5..0542a1d2 100644
--- a/cli/test/evals/internal/grade/answer.go
+++ b/cli/test/evals/internal/grade/answer.go
@@ -114,8 +114,11 @@ func sameEndpoint(got, want string) bool {
if g == w {
return true
}
+ // The answer may leave the method out, or put a word before the
+ // endpoint ("endpoint POST /api/checkout"); it may not name another
+ // method.
_, wr, _ := strings.Cut(w, " ")
- return g == wr || strings.HasSuffix(g, " "+wr)
+ return g == wr || strings.HasSuffix(g, " "+w)
}
func containsFold(s, sub string) bool {
diff --git a/cli/test/evals/internal/grade/grade.go b/cli/test/evals/internal/grade/grade.go
index 8886dac2..4ba9b027 100644
--- a/cli/test/evals/internal/grade/grade.go
+++ b/cli/test/evals/internal/grade/grade.go
@@ -20,7 +20,6 @@ type Verdict struct {
Score *float64 `json:"score,omitempty"`
Failures []string `json:"failures,omitempty"`
Notes []string `json:"notes,omitempty"`
- CostUSD float64 `json:"costUsd,omitempty"`
}
func (v *Verdict) fail(format string, a ...any) {
@@ -51,7 +50,7 @@ func All(ctx context.Context, in *Input) []*Verdict {
if in.Case.Expect.Calls != nil {
out = append(out, Trajectory(in.Run, in.Case.Expect.Calls))
}
- if in.Case.Suite == "perf" || in.Case.Suite == "debug" || len(in.Case.Expect.Grounding) > 0 {
+ if in.Case.Suite == "perf" || len(in.Case.Expect.Grounding) > 0 {
out = append(out, Grounding(in))
}
if in.Case.Expect.Report != nil {
diff --git a/cli/test/evals/internal/grade/grade_test.go b/cli/test/evals/internal/grade/grade_test.go
index 4f51529f..1603d64d 100644
--- a/cli/test/evals/internal/grade/grade_test.go
+++ b/cli/test/evals/internal/grade/grade_test.go
@@ -2,6 +2,7 @@ package grade
import (
"encoding/json"
+ "fmt"
"strings"
"testing"
@@ -62,12 +63,37 @@ func TestTrajectory(t *testing.T) {
s.MaxIdenticalRepeats = 1
r.Calls = append(r.Calls, call("c09", true, "{}", "traceway endpoints list --since 24h --output json"))
}, "same call ran 2 times"},
+ "thrash, read in chunks of the same window": {func(r *record.Run, s *catalog.Trajectory) {
+ s.MaxIdenticalRepeats = 1
+ read := func(id string) *record.Call {
+ return &record.Call{ID: id, Tool: "read_file", OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: map[string]any{"path": "big.go", "offset": 2001}}, OK: true}}}
+ }
+ r.Calls = append(r.Calls, read("c11"), read("c12"))
+ }, "same call ran 2 times"},
"too many failures": {func(r *record.Run, s *catalog.Trajectory) {
s.MaxFailedCalls = 0
s.MaxFailedCalls = 1
r.Calls = append(r.Calls, call("c10", false, "x", "traceway logs query"))
}, "failed tool calls"},
}
+ // Reading a file in chunks and changing one file several times are not
+ // thrashing.
+ chunked := &record.Run{Calls: append([]*record.Call{}, run.Calls...)}
+ for i, off := range []int{0, 2001, 4001} {
+ args := map[string]any{"path": "big.go"}
+ if off > 0 {
+ args["offset"] = off
+ }
+ chunked.Calls = append(chunked.Calls,
+ &record.Call{ID: fmt.Sprintf("r%d", i), Tool: "read_file", OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: args}, OK: true}}},
+ &record.Call{ID: fmt.Sprintf("e%d", i), Tool: "edit_file", OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpEditFile, Args: map[string]any{"path": "big.go"}}, OK: true}}})
+ }
+ repeats := *spec
+ repeats.MaxIdenticalRepeats = 1
+ if v := Trajectory(chunked, &repeats); !v.Pass {
+ t.Fatalf("chunked reads and repeated edits counted as thrashing:\n%s", failures(v))
+ }
+
for name, c := range cases {
t.Run(name, func(t *testing.T) {
r := &record.Run{Calls: append([]*record.Call{}, run.Calls...)}
@@ -100,6 +126,7 @@ func TestAnswer(t *testing.T) {
"wrong row": {"checklist_item": "missing_or_unused_index", "locus": good["locus"]},
"distractor": {"checklist_item": "over_fetching", "locus": good["locus"]},
"wrong locus": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "GET /api/orders/:id", "span": "order_items"}},
+ "wrong method": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "POST /api/orders", "span": "order_items"}},
"no span": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "GET /api/orders"}},
"no submission": nil,
} {
@@ -109,6 +136,25 @@ func TestAnswer(t *testing.T) {
}
}
+func TestSameEndpoint(t *testing.T) {
+ for _, c := range []struct {
+ got, want string
+ same bool
+ }{
+ {"POST /api/checkout", "POST /api/checkout", true},
+ {"post /api/checkout/", "POST /api/checkout", true},
+ {"/api/checkout", "POST /api/checkout", true},
+ {"endpoint POST /api/checkout", "POST /api/checkout", true},
+ {"GET /api/checkout", "/api/checkout", true},
+ {"GET /api/checkout", "POST /api/checkout", false},
+ {"POST /api/checkout/items", "POST /api/checkout", false},
+ } {
+ if got := sameEndpoint(c.got, c.want); got != c.same {
+ t.Errorf("sameEndpoint(%q, %q) = %v", c.got, c.want, got)
+ }
+ }
+}
+
func TestGrounding(t *testing.T) {
out := `{"spans":[{"attributes":{"db.query.text":"SELECT id FROM order_items WHERE order_id = $1"}}],"id":"6f9619ff-8b86-d011-b42d-00c04fc964ff"}`
c1 := call("c01", true, out, "traceway endpoints show 6f9619ff-8b86-d011-b42d-00c04fc964ff --recorded-at 2026-10-17T10:00:00Z")
@@ -126,7 +172,11 @@ func TestGrounding(t *testing.T) {
"failed call": {perfInput(sub("c01", ""), call("c01", false, out, "traceway endpoints show x")), "which failed"},
"truncated call": {perfInput(sub("c01", ""), &record.Call{ID: "c01", OK: true, Truncated: true, Output: out}), "truncated"},
"invented id": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1), "no tool returned"},
- "missing key": {perfInput(sub("c02", ""), c1, call("c02", true, `{"data":[]}`, "traceway endpoints list")), "contains"},
+ // The submit call itself is recorded with the submission as its
+ // arguments; that must not vouch for the ids in it.
+ "invented id, submit recorded": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1,
+ &record.Call{ID: "c02", Tool: "submit_perf_diagnosis", OK: true, Args: json.RawMessage(`{"report":"trace 4bf92f3577b34da6a3ce929d0e0e4736"}`)}), "no tool returned"},
+ "missing key": {perfInput(sub("c02", ""), c1, call("c02", true, `{"data":[]}`, "traceway endpoints list")), "contains"},
}
for name, tc := range tests {
t.Run(name, func(t *testing.T) {
diff --git a/cli/test/evals/internal/grade/grounding.go b/cli/test/evals/internal/grade/grounding.go
index 98a05ea2..374982ce 100644
--- a/cli/test/evals/internal/grade/grounding.go
+++ b/cli/test/evals/internal/grade/grounding.go
@@ -7,6 +7,7 @@ import (
"strings"
"github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"
)
// Grounding checks that the evidence is real:
@@ -43,7 +44,7 @@ func Grounding(in *Input) *Verdict {
cited = append(cited, c)
}
}
- } else if in.Case.Suite == "perf" || in.Case.Suite == "debug" {
+ } else if in.Case.Suite == "perf" {
v.fail("the submission cites no evidence")
}
for _, key := range in.Case.Expect.Grounding {
@@ -107,13 +108,17 @@ func unescapeJSON(s string) string {
var idRe = regexp.MustCompile(`(?i)\b(?:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32}|[0-9a-f]{16})\b`)
// InventedIDs lists ids in the submission (and final text) that appear in no
-// tool output and not in the prompt.
+// tool output and not in the prompt. The submit calls are left out of the
+// corpus: their arguments are the submission itself.
func InventedIDs(run *record.Run) []string {
raw, _ := json.Marshal(run.Submission)
text := string(raw) + "\n" + run.FinalText
var corpus strings.Builder
corpus.WriteString(strings.ToLower(run.Prompt))
for _, c := range run.Calls {
+ if tools.IsSubmit(c.Tool) {
+ continue
+ }
corpus.WriteString(strings.ToLower(c.Output))
corpus.WriteString(strings.ToLower(string(c.Args)))
for _, op := range c.Ops {
diff --git a/cli/test/evals/internal/grade/leakage.go b/cli/test/evals/internal/grade/leakage.go
index 67aba577..29ad964f 100644
--- a/cli/test/evals/internal/grade/leakage.go
+++ b/cli/test/evals/internal/grade/leakage.go
@@ -11,7 +11,7 @@ import (
"github.com/tracewayapp/traceway/cli/test/evals/internal/checklist"
)
-// Leakage lint (plan §4.8): the seeded data and the prompt must not name the
+// Leakage lint: the seeded data and the prompt must not name the
// answer. Seeded strings (span names, attribute values, log bodies, stack
// traces, object names) are checked against a denylist of checklist terms
// and against the expected checklist row's title; prompts get the checklist
diff --git a/cli/test/evals/internal/grade/trajectory.go b/cli/test/evals/internal/grade/trajectory.go
index 1f9623d1..c913bab1 100644
--- a/cli/test/evals/internal/grade/trajectory.go
+++ b/cli/test/evals/internal/grade/trajectory.go
@@ -10,7 +10,9 @@ import (
"strings"
"github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"
"github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"
)
// Trajectory checks the tool calls against the case's call DSL:
@@ -81,7 +83,7 @@ func Trajectory(run *record.Run, t *catalog.Trajectory) *Verdict {
executed := 0
failed := 0
for _, c := range run.Calls {
- if c.Rejected || isSubmitTool(c.Tool) {
+ if c.Rejected || tools.IsSubmit(c.Tool) {
continue
}
executed++
@@ -96,8 +98,13 @@ func Trajectory(run *record.Run, t *catalog.Trajectory) *Verdict {
v.fail("%d failed tool calls, more than max_failed_calls %d", failed, t.MaxFailedCalls)
}
if t.MaxIdenticalRepeats > 0 {
+ // Thrashing is asking the same question again. Changing a local file
+ // (several edits, or appending to one file) is not.
counts := map[string]int{}
for _, op := range all {
+ if op.Op.Op == ops.OpWriteFile || op.Op.Op == ops.OpEditFile {
+ continue
+ }
counts[opKey(op.OpRecord)]++
}
keys := make([]string, 0, len(counts))
@@ -115,10 +122,6 @@ func Trajectory(run *record.Run, t *catalog.Trajectory) *Verdict {
return v
}
-func isSubmitTool(name string) bool {
- return name == "submit_perf_diagnosis" || name == "submit_debug_diagnosis" || name == "finish"
-}
-
func firstMatch(all []record.CallOp, m catalog.CallMatcher) int {
for i, op := range all {
if matches(op.OpRecord, m) {
diff --git a/cli/test/evals/internal/ops/ops_test.go b/cli/test/evals/internal/ops/ops_test.go
index e4f6a7a9..aeea53f1 100644
--- a/cli/test/evals/internal/ops/ops_test.go
+++ b/cli/test/evals/internal/ops/ops_test.go
@@ -30,15 +30,33 @@ func TestOpsMatchToolMap(t *testing.T) {
}
// TestEveryCLICommandMaps walks both built editions' help: every leaf
-// command must map to an op, and write-edition commands must be writes.
+// command must map to an op, and exactly the commands only traceway-write
+// has are marked WriteEdition, as writes.
func TestEveryCLICommandMaps(t *testing.T) {
bins := env.BinariesForTest(t)
- for _, bin := range []string{bins.CLI, bins.CLIWrite} {
- for _, path := range leafCommands(t, bin, nil) {
- op := ParseCLI(path)
- if op.Op == OpUnknownCLI {
- t.Errorf("%s %v maps to no op", bin, path)
- }
+ inRead := map[string]bool{}
+ for _, path := range leafCommands(t, bins.CLI, nil) {
+ inRead[strings.Join(path, " ")] = true
+ op := ParseCLI(path)
+ if op.Op == OpUnknownCLI {
+ t.Errorf("traceway %v maps to no op", path)
+ continue
+ }
+ if spec, ok := SpecFor(op.Op); ok && spec.WriteEdition {
+ t.Errorf("traceway %v exists in the read edition, but %s is marked WriteEdition", path, op.Op)
+ }
+ }
+ for _, path := range leafCommands(t, bins.CLIWrite, nil) {
+ op := ParseCLI(path)
+ if op.Op == OpUnknownCLI {
+ t.Errorf("traceway-write %v maps to no op", path)
+ continue
+ }
+ if inRead[strings.Join(path, " ")] {
+ continue
+ }
+ if spec, ok := SpecFor(op.Op); !ok || !spec.WriteEdition || !op.Write {
+ t.Errorf("traceway-write %v exists only in the write edition, so %s must be WriteEdition and Write", path, op.Op)
}
}
}
diff --git a/cli/test/evals/internal/report/aggregate.go b/cli/test/evals/internal/report/aggregate.go
index 1653f308..12c95092 100644
--- a/cli/test/evals/internal/report/aggregate.go
+++ b/cli/test/evals/internal/report/aggregate.go
@@ -57,7 +57,6 @@ type Aggregates struct {
Models []Delta `json:"models"`
Cost map[string]*Costs `json:"cost"` // by model
SpendUSD float64 `json:"spendUsd"`
- JudgeUSD float64 `json:"judgeUsd"`
Statuses map[string]int `json:"statuses"`
TopFailing []GraderCount `json:"topFailingGraders"`
}
@@ -81,8 +80,7 @@ func Aggregate(runs []*runner.Result) Aggregates {
cacheIn, cacheHit := map[string]int{}, map[string]int{}
for _, r := range runs {
a.Statuses[r.Status]++
- a.SpendUSD += r.Metrics.CostUSD + r.Metrics.JudgeCostUSD
- a.JudgeUSD += r.Metrics.JudgeCostUSD
+ a.SpendUSD += r.Metrics.CostUSD
c := a.Cost[r.Model]
if c == nil {
c = &Costs{}
diff --git a/cli/test/evals/internal/report/markdown.go b/cli/test/evals/internal/report/markdown.go
index 4cd3d881..740647dc 100644
--- a/cli/test/evals/internal/report/markdown.go
+++ b/cli/test/evals/internal/report/markdown.go
@@ -35,7 +35,11 @@ func Markdown(r *Results) string {
}
sort.Strings(models)
fmt.Fprintf(&b, "- **Models:** %s\n", strings.Join(models, ", "))
- fmt.Fprintf(&b, "- **Spend:** $%.2f (judge $%.2f) of a $%.2f cap\n", a.SpendUSD, a.JudgeUSD, h.MaxCostUSD)
+ // An oracle run costs nothing, so its summary leaves money out.
+ paid := !h.Oracle
+ if paid {
+ fmt.Fprintf(&b, "- **Spend:** $%.2f\n", a.SpendUSD)
+ }
var st []string
for k, v := range a.Statuses {
st = append(st, fmt.Sprintf("%s %d", k, v))
@@ -62,16 +66,18 @@ func Markdown(r *Results) string {
fmt.Fprintf(&b, "| %s | %s | %s | %.0f%% | %.0f%% | %+.0fpp |\n", d.Suite, d.A, d.B, d.RateA*100, d.RateB*100, d.Delta)
}
}
- b.WriteString("\n## Cost (averages over passing runs)\n\n| Model | Passing runs | Avg cost | Avg tool calls | Avg input tokens | Avg output tokens | Cache hit rate | Total spend |\n|---|---|---|---|---|---|---|---|\n")
- for _, m := range sortedKeys(a.Cost) {
- c := a.Cost[m]
- fmt.Fprintf(&b, "| %s | %d | $%.3f | %.1f | %.0f | %.0f | %.0f%% | $%.2f |\n", m, c.PassingRuns, c.AvgCostUSD, c.AvgToolCalls, c.AvgInputTokens, c.AvgOutputTokens, c.CacheHitRate*100, c.TotalCostUSD)
+ if paid {
+ b.WriteString("\n## Cost (averages over passing runs)\n\n| Model | Passing runs | Avg cost | Avg tool calls | Avg input tokens | Avg output tokens | Cache hit rate | Total spend |\n|---|---|---|---|---|---|---|---|\n")
+ for _, m := range sortedKeys(a.Cost) {
+ c := a.Cost[m]
+ fmt.Fprintf(&b, "| %s | %d | $%.3f | %.1f | %.0f | %.0f | %.0f%% | $%.2f |\n", m, c.PassingRuns, c.AvgCostUSD, c.AvgToolCalls, c.AvgInputTokens, c.AvgOutputTokens, c.CacheHitRate*100, c.TotalCostUSD)
+ }
}
bySuiteCost := map[string]float64{}
for _, run := range r.Runs {
bySuiteCost[run.Suite+" / "+run.Model] += run.Metrics.CostUSD
}
- if len(bySuiteCost) > 0 {
+ if paid && len(bySuiteCost) > 0 {
b.WriteString("\n| Suite / model | Spend |\n|---|---|\n")
for _, k := range sortedKeys(bySuiteCost) {
fmt.Fprintf(&b, "| %s | $%.2f |\n", k, bySuiteCost[k])
@@ -92,11 +98,22 @@ func Markdown(r *Results) string {
fmt.Fprintf(&b, "| %s | %s | %s | %d/%d |\n", cs, m, arm, v.Pass, v.Total)
}
}
- b.WriteString("\n## Runs\n\n| Case | Arm | Model | Run | Status | Calls | Cost | First failure |\n|---|---|---|---|---|---|---|---|\n")
+ b.WriteString("\n## Runs\n\n| Case | Arm | Model | Run | Status | Calls |")
+ if paid {
+ b.WriteString(" Cost |")
+ }
+ b.WriteString(" First failure | Transcript |\n|---|---|---|---|---|---|")
+ if paid {
+ b.WriteString("---|")
+ }
+ b.WriteString("---|---|\n")
for _, run := range r.Runs {
- fmt.Fprintf(&b, "| %s | %s | %s | %d | %s | %d | $%.3f | %s |\n", run.Case, run.Arm, run.Model, run.RunIdx, run.Status, run.Metrics.ToolCalls, run.Metrics.CostUSD, mdEscape(firstFailure(run)))
+ fmt.Fprintf(&b, "| %s | %s | %s | %d | %s | %d |", run.Case, run.Arm, run.Model, run.RunIdx, run.Status, run.Metrics.ToolCalls)
+ if paid {
+ fmt.Fprintf(&b, " $%.3f |", run.Metrics.CostUSD)
+ }
+ fmt.Fprintf(&b, " %s | %s |\n", mdEscape(firstFailure(run)), orDash(run.Transcript))
}
- b.WriteString("\nTranscripts are in the artifact under `runs/<case>/<arm>/<model>/r<N>/transcript.jsonl`.\n")
return b.String()
}
diff --git a/cli/test/evals/internal/report/report_test.go b/cli/test/evals/internal/report/report_test.go
index 24bdbd5b..bf286f87 100644
--- a/cli/test/evals/internal/report/report_test.go
+++ b/cli/test/evals/internal/report/report_test.go
@@ -2,6 +2,7 @@ package report
import (
"math"
+ "strings"
"testing"
"github.com/tracewayapp/traceway/cli/test/evals/internal/runner"
@@ -34,3 +35,23 @@ func TestAggregate(t *testing.T) {
t.Fatalf("cost %+v", c)
}
}
+
+// TestOracleSummary: an oracle run costs nothing, so its summary names no
+// spend, and each run's transcript is listed at the path it was written to.
+func TestOracleSummary(t *testing.T) {
+ r := res("perf-a", "cli+skill", "oracle", 1, "pass", 0)
+ r.Transcript = "runs/perf-a/cli_skill/oracle/r1/transcript.jsonl"
+ out := Markdown(&Results{Header: Header{Oracle: true}, Runs: []*runner.Result{r}, Aggregates: Aggregate([]*runner.Result{r})})
+ for _, s := range []string{"Spend", "Cost", "cap"} {
+ if strings.Contains(out, s) {
+ t.Errorf("oracle summary mentions %q:\n%s", s, out)
+ }
+ }
+ if !strings.Contains(out, r.Transcript) {
+ t.Errorf("summary lacks the transcript path:\n%s", out)
+ }
+ paid := Markdown(&Results{Runs: []*runner.Result{r}, Aggregates: Aggregate([]*runner.Result{r})})
+ if !strings.Contains(paid, "Spend") || !strings.Contains(paid, "| Cost |") {
+ t.Errorf("a paid run's summary lost its cost:\n%s", paid)
+ }
+}
diff --git a/cli/test/evals/internal/report/results.go b/cli/test/evals/internal/report/results.go
index 16807d6c..f877560d 100644
--- a/cli/test/evals/internal/report/results.go
+++ b/cli/test/evals/internal/report/results.go
@@ -34,7 +34,6 @@ type Header struct {
Command string `json:"command"`
Started time.Time `json:"started"`
Finished time.Time `json:"finished"`
- MaxCostUSD float64 `json:"maxCostUsd"`
Oracle bool `json:"oracle,omitempty"`
CI map[string]string `json:"ci,omitempty"`
}
@@ -148,22 +147,3 @@ func Load(dir string) (*Results, error) {
}
return r, nil
}
-
-// Merge combines shard results under one header.
-func Merge(parts []*Results) *Results {
- out := &Results{}
- total := 0.0
- for i, p := range parts {
- if i == 0 {
- out.Header = p.Header
- }
- total += p.Header.MaxCostUSD
- if p.Header.Finished.After(out.Header.Finished) {
- out.Header.Finished = p.Header.Finished
- }
- out.Runs = append(out.Runs, p.Runs...)
- }
- out.Header.MaxCostUSD = total
- sort.SliceStable(out.Runs, func(i, j int) bool { return out.Runs[i].Key() < out.Runs[j].Key() })
- return out
-}
diff --git a/cli/test/evals/internal/runner/runner.go b/cli/test/evals/internal/runner/runner.go
index ff663e86..59bf3af5 100644
--- a/cli/test/evals/internal/runner/runner.go
+++ b/cli/test/evals/internal/runner/runner.go
@@ -72,7 +72,6 @@ type Metrics struct {
CacheWriteTokens int `json:"cacheWriteTokens"`
OutputTokens int `json:"outputTokens"`
CostUSD float64 `json:"costUsd"`
- JudgeCostUSD float64 `json:"judgeCostUsd"`
WallMs int64 `json:"wallMs"`
SkillLoaded bool `json:"skillLoaded"`
Skills []string `json:"skills,omitempty"`
@@ -136,17 +135,6 @@ type surface struct {
submit string
}
-func submitFor(suite string) string {
- switch suite {
- case "perf":
- return tools.SubmitPerf
- case "debug":
- return tools.SubmitDebug
- default:
- return tools.Finish
- }
-}
-
func runOne(ctx context.Context, opt Options, j Job) (res *Result) {
c := j.Unit.Case
arm := j.Unit.Arm
@@ -258,7 +246,8 @@ func runOne(ctx context.Context, opt Options, j Job) (res *Result) {
prompt := strings.TrimSpace(resolved.Prompt)
var preloaded *tools.Skill
- if name := agent.SlashSkill(prompt); name != "" {
+ if resolved.Invoke == "slash" {
+ name := agent.SlashSkill(prompt)
for i := range skillDefs {
if skillDefs[i].Name == name {
preloaded = &skillDefs[i]
@@ -308,9 +297,6 @@ func runOne(ctx context.Context, opt Options, j Job) (res *Result) {
in := &grade.Input{Case: resolved, Arm: arm, Run: run, Seed: jsonify(exposed)}
res.Graders = append(res.Graders, grade.All(ctx, in)...)
- for _, v := range res.Graders {
- res.Metrics.JudgeCostUSD += v.CostUSD
- }
switch {
case run.StopReason == agent.StopBudget:
res.Status = "budget_exhausted"
@@ -321,7 +307,7 @@ func runOne(ctx context.Context, opt Options, j Job) (res *Result) {
default:
res.Status = "fail"
}
- logf(opt, " %s %s (%d calls, $%.4f)", RunKey(c.ID, arm, res.Model, j.Idx), res.Status, res.Metrics.ToolCalls, res.Metrics.CostUSD)
+ logf(opt, " %s %s (%d calls)", RunKey(c.ID, arm, res.Model, j.Idx), res.Status, res.Metrics.ToolCalls)
return res
}
diff --git a/cli/test/evals/internal/runner/surface.go b/cli/test/evals/internal/runner/surface.go
index 76fde75d..56606c4d 100644
--- a/cli/test/evals/internal/runner/surface.go
+++ b/cli/test/evals/internal/runner/surface.go
@@ -6,12 +6,13 @@ import (
"github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
"github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"
)
// prepareSurface mints the arm's credential and connects its surface: the
// CLI logged in with a read PAT (XDG dirs outside the workspace).
func prepareSurface(ctx context.Context, opt Options, c *catalog.Case, arm string, inst *env.Instance, primary env.Project, dir string) (surface, error) {
- s := surface{submit: submitFor(c.Suite)}
+ s := surface{submit: tools.SubmitPerf}
if arm == catalog.ArmCLISkill {
s.skills = []string{"traceway"}
}
diff --git a/cli/test/evals/internal/seed/metrics.go b/cli/test/evals/internal/seed/metrics.go
index beeeb582..11df53e4 100644
--- a/cli/test/evals/internal/seed/metrics.go
+++ b/cli/test/evals/internal/seed/metrics.go
@@ -125,9 +125,11 @@ type metricSeries struct {
window time.Duration
scope string
// cumTotal and cumPrev carry a counter's running total across the
- // chunks metricRequests renders it in.
+ // chunks metricRequests renders it in; histPrev carries the end of a
+ // histogram's last delta window the same way.
cumTotal float64
cumPrev time.Time
+ histPrev time.Time
}
// MetricStep controls sampling: points every Recent inside the last day,
@@ -341,19 +343,30 @@ func (m *metricSeries) render(times []time.Time, origin time.Time) *metricspb.Me
m.cumTotal, m.cumPrev = total, prev
out.Data = &metricspb.Metric_Sum{Sum: sum}
case histogramKind:
+ // Each delta point covers the time since the previous one, which is
+ // longer where points are sparser (MetricStep.Old), and its count
+ // covers the same window.
h := &metricspb.Histogram{AggregationTemporality: metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_DELTA}
- prev := times[0].Add(-time.Minute)
+ prev := m.histPrev
+ if prev.IsZero() {
+ step := time.Minute
+ if len(times) > 1 {
+ step = times[1].Sub(times[0])
+ }
+ prev = times[0].Add(-step)
+ }
for _, t := range times {
per := m.count(t) * t.Sub(prev).Minutes()
- prev = t
cnt := uint64(math.Max(0, math.Round(per)))
mean := m.value(t)
s := mean * float64(cnt)
h.DataPoints = append(h.DataPoints, &metricspb.HistogramDataPoint{
- Attributes: attrs, StartTimeUnixNano: nanos(t.Add(-time.Minute)), TimeUnixNano: nanos(t),
+ Attributes: attrs, StartTimeUnixNano: nanos(prev), TimeUnixNano: nanos(t),
Count: cnt, Sum: &s,
})
+ prev = t
}
+ m.histPrev = prev
out.Data = &metricspb.Metric_Histogram{Histogram: h}
}
return out
diff --git a/cli/test/evals/internal/seed/metrics_test.go b/cli/test/evals/internal/seed/metrics_test.go
new file mode 100644
index 00000000..67994483
--- /dev/null
+++ b/cli/test/evals/internal/seed/metrics_test.go
@@ -0,0 +1,41 @@
+package seed
+
+import (
+ "testing"
+ "time"
+)
+
+// TestHistogramDeltaWindows: each delta point starts where the previous one
+// ended, across the chunks a series is rendered in, and counts the
+// observations of exactly that window.
+func TestHistogramDeltaWindows(t *testing.T) {
+ m := &metricSeries{name: "h", kind: histogramKind, value: func(time.Time) float64 { return 5 }, count: func(time.Time) float64 { return 10 }}
+ base := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
+ var times []time.Time
+ for i := range 6 {
+ times = append(times, base.Add(time.Duration(i)*MetricStep.Old))
+ }
+ var points []struct {
+ start, end uint64
+ count uint64
+ }
+ for _, chunk := range [][]time.Time{times[:3], times[3:]} {
+ for _, dp := range m.render(chunk, base).GetHistogram().DataPoints {
+ points = append(points, struct {
+ start, end uint64
+ count uint64
+ }{dp.StartTimeUnixNano, dp.TimeUnixNano, dp.Count})
+ }
+ }
+ for i, p := range points {
+ if p.count != 100 {
+ t.Errorf("point %d counts %d observations, want 100 (10/min over 10 min)", i, p.count)
+ }
+ if i > 0 && p.start != points[i-1].end {
+ t.Errorf("point %d starts at %d, want the previous point's end %d", i, p.start, points[i-1].end)
+ }
+ if got := time.Duration(p.end - p.start); got != MetricStep.Old {
+ t.Errorf("point %d covers %s, want %s", i, got, MetricStep.Old)
+ }
+ }
+}
diff --git a/cli/test/evals/internal/tools/coreutils.go b/cli/test/evals/internal/tools/coreutils.go
index c85b58d9..08b09d57 100644
--- a/cli/test/evals/internal/tools/coreutils.go
+++ b/cli/test/evals/internal/tools/coreutils.go
@@ -99,14 +99,16 @@ func coreutilNames() []string {
}
// flags is a tiny getopt: single-dash letters (bundled), --long, and
-// options that take a value.
+// options that take a value. set keeps an option's last value, vals every
+// value of a repeatable one (grep -e A -e B).
type flags struct {
set map[string]string
+ vals map[string][]string
args []string
}
func parseFlags(args []string, withValue string) flags {
- f := flags{set: map[string]string{}}
+ f := flags{set: map[string]string{}, vals: map[string][]string{}}
for i := 0; i < len(args); i++ {
a := args[i]
if a == "--" {
@@ -131,6 +133,7 @@ func parseFlags(args []string, withValue string) flags {
v = args[i]
}
f.set[c] = v
+ f.vals[c] = append(f.vals[c], v)
break
}
f.set[c] = "true"
@@ -281,10 +284,7 @@ func breToERE(p string) string {
func uGrep(u *uctx, args []string) int {
f := parseFlags(args, "eABCm")
- var patterns []string
- if v, ok := f.set["e"]; ok {
- patterns = append(patterns, v)
- }
+ patterns := f.vals["e"]
rest := f.args
if len(patterns) == 0 {
if len(rest) == 0 {
@@ -383,6 +383,7 @@ func uGrep(u *uctx, args []string) int {
lines := splitLines(data)
count := 0
lastPrinted := -1
+ afterLeft := 0
prefix := func(i int, sep string) string {
p := ""
if multi {
@@ -393,10 +394,26 @@ func uGrep(u *uctx, args []string) int {
}
return p
}
+ // printLine prints line i once, with grep's "--" between context
+ // groups that are not adjacent.
+ printLine := func(i int, sep string) {
+ if (before > 0 || after > 0) && lastPrinted >= 0 && i > lastPrinted+1 {
+ fmt.Fprintln(w, "--")
+ }
+ fmt.Fprintln(w, prefix(i, sep)+lines[i])
+ lastPrinted = i
+ }
for i, l := range lines {
if !match(l) {
+ if afterLeft > 0 && !f.has("c") && !f.has("o") {
+ printLine(i, "-")
+ afterLeft--
+ }
continue
}
+ if maxCount >= 0 && count >= maxCount {
+ break
+ }
found = true
count++
if f.has("q") {
@@ -407,9 +424,6 @@ func uGrep(u *uctx, args []string) int {
break
}
if f.has("c") {
- if maxCount >= 0 && count >= maxCount {
- break
- }
continue
}
if f.has("o") && !f.has("v") {
@@ -418,22 +432,13 @@ func uGrep(u *uctx, args []string) int {
fmt.Fprintln(w, prefix(i, ":")+m)
}
}
- } else {
- for b := max(lastPrinted+1, i-before); b < i; b++ {
- fmt.Fprintln(w, prefix(b, "-")+lines[b])
- }
- fmt.Fprintln(w, prefix(i, ":")+l)
- lastPrinted = i
- for a := i + 1; a <= i+after && a < len(lines); a++ {
- if !match(lines[a]) {
- fmt.Fprintln(w, prefix(a, "-")+lines[a])
- lastPrinted = a
- }
- }
+ continue
}
- if maxCount >= 0 && count >= maxCount {
- break
+ for b := max(lastPrinted+1, i-before); b < i; b++ {
+ printLine(b, "-")
}
+ printLine(i, ":")
+ afterLeft = after
}
if f.has("c") {
if multi {
diff --git a/cli/test/evals/internal/tools/files.go b/cli/test/evals/internal/tools/files.go
index 58e85d88..cf8d6750 100644
--- a/cli/test/evals/internal/tools/files.go
+++ b/cli/test/evals/internal/tools/files.go
@@ -74,7 +74,16 @@ func (t *ReadFile) Call(_ context.Context, raw json.RawMessage) Result {
fmt.Fprintf(&b, "[%d more lines; continue with offset %d]\n", len(lines)-(start+limit)+1, start+limit)
}
out, trunc := Cap(b.String())
- return Result{Output: out, OK: true, Truncated: trunc, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: map[string]any{"path": rel}}, OK: true, Output: text}}}
+ // The window is part of the operation, so reading a long file in chunks
+ // is not the same call repeated.
+ args := map[string]any{"path": rel}
+ if start > 1 {
+ args["offset"] = start
+ }
+ if in.Limit > 0 {
+ args["limit"] = in.Limit
+ }
+ return Result{Output: out, OK: true, Truncated: trunc, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: args}, OK: true, Output: text}}}
}
// ListFiles lists files under a directory.
diff --git a/cli/test/evals/internal/tools/jqawk.go b/cli/test/evals/internal/tools/jqawk.go
index 0b443a86..5d9fb64f 100644
--- a/cli/test/evals/internal/tools/jqawk.go
+++ b/cli/test/evals/internal/tools/jqawk.go
@@ -24,9 +24,19 @@ func runJQ(hc interp.HandlerContext, r *shellRun, args []string) int {
var files []string
var varNames []string
var varValues []any
+ // Options may come before or after the filter, as in jq; "--" ends them.
+ optionsDone := false
for i := 0; i < len(args); i++ {
a := args[i]
switch {
+ case optionsDone || a == "-" || !strings.HasPrefix(a, "-"):
+ if !haveFilter {
+ filter, haveFilter = a, true
+ } else {
+ files = append(files, a)
+ }
+ case a == "--":
+ optionsDone = true
case a == "--arg" && i+2 < len(args):
varNames = append(varNames, "$"+args[i+1])
varValues = append(varValues, args[i+2])
@@ -68,7 +78,7 @@ func runJQ(hc interp.HandlerContext, r *shellRun, args []string) int {
u.errf("jq: unsupported option %s", a)
return 2
}
- case strings.HasPrefix(a, "-") && len(a) > 1 && !haveFilter:
+ default:
for _, c := range a[1:] {
switch c {
case 'r':
@@ -93,12 +103,6 @@ func runJQ(hc interp.HandlerContext, r *shellRun, args []string) int {
return 2
}
}
- default:
- if !haveFilter {
- filter, haveFilter = a, true
- } else {
- files = append(files, a)
- }
}
}
if !haveFilter {
diff --git a/cli/test/evals/internal/tools/shell.go b/cli/test/evals/internal/tools/shell.go
index 867b1886..8f8ed220 100644
--- a/cli/test/evals/internal/tools/shell.go
+++ b/cli/test/evals/internal/tools/shell.go
@@ -56,7 +56,7 @@ func (s *Shell) Spec() Spec {
allowed := append(names, "jq", "awk")
allowed = append(allowed, coreutilNames()...)
if s.Git {
- allowed = append(allowed, "git (log, show, diff, status, blame, ls-files, rev-parse)")
+ allowed = append(allowed, "git ("+strings.Join(gitAllowed, ", ")+")")
}
return Spec{
Name: "shell",
@@ -246,25 +246,45 @@ func (r *shellRun) execTraceway(ctx context.Context, hc interp.HandlerContext, b
var gitAllowed = []string{"log", "show", "diff", "status", "blame", "ls-files", "rev-parse", "shortlog"}
+// gitDenied are options of the allowed subcommands that write a file or run
+// a program.
+var gitDenied = []string{"--output", "--ext-diff"}
+
+// execGit runs a read-only git subcommand on the workspace repository. GIT_DIR
+// is pinned, so git never discovers a repository (and its config) written
+// elsewhere in the workspace; the workspace's .git is read-only to every tool.
func (r *shellRun) execGit(ctx context.Context, hc interp.HandlerContext, args []string) error {
+ refuse := func(format string, a ...any) error {
+ fmt.Fprintf(hc.Stderr, "git: "+format+"\n", a...)
+ return interp.NewExitStatus(2)
+ }
+ if _, err := r.s.Sandbox.Resolve("", hc.Dir); err != nil {
+ return refuse("only the workspace repository is available")
+ }
+ rest := args[1:]
+ for len(rest) > 0 && (rest[0] == "--no-pager" || rest[0] == "-P") {
+ rest = rest[1:]
+ }
+ if len(rest) > 0 && strings.HasPrefix(rest[0], "-") {
+ return refuse("global options are not allowed in this sandbox")
+ }
sub := ""
- for _, a := range args[1:] {
- if !strings.HasPrefix(a, "-") {
- sub = a
- break
- }
- if a == "-C" || strings.HasPrefix(a, "--git-dir") || strings.HasPrefix(a, "--work-tree") || strings.HasPrefix(a, "-c") {
- fmt.Fprintln(hc.Stderr, "git: global options are not allowed in this sandbox")
- return interp.NewExitStatus(2)
- }
+ if len(rest) > 0 {
+ sub = rest[0]
}
if !slices.Contains(gitAllowed, sub) {
- fmt.Fprintf(hc.Stderr, "git %s: only read-only git (%s) is available\n", sub, strings.Join(gitAllowed, ", "))
- return interp.NewExitStatus(2)
+ return refuse("%s: only read-only git (%s) is available", sub, strings.Join(gitAllowed, ", "))
}
- cmd := exec.CommandContext(ctx, "git", append([]string{"--no-pager"}, args[1:]...)...)
+ for _, a := range rest[1:] {
+ if err := r.gitArg(hc.Dir, a); err != nil {
+ return refuse("%v", err)
+ }
+ }
+ root := r.s.Sandbox.Root
+ cmd := exec.CommandContext(ctx, "git", append([]string{"--no-pager"}, rest...)...)
cmd.Dir = hc.Dir
- cmd.Env = []string{"HOME=" + r.s.Sandbox.Root, "PATH=/usr/bin:/bin", "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_PAGER=cat", "PAGER=cat", "TZ=UTC", "LANG=C.UTF-8", "GIT_OPTIONAL_LOCKS=0"}
+ cmd.Env = []string{"HOME=" + root, "PATH=/usr/bin:/bin", "GIT_DIR=" + filepath.Join(root, ".git"), "GIT_WORK_TREE=" + root,
+ "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_PAGER=cat", "PAGER=cat", "TZ=UTC", "LANG=C.UTF-8", "GIT_OPTIONAL_LOCKS=0"}
so := &limitedBuffer{n: 100000}
cmd.Stdout = io.MultiWriter(hc.Stdout, so)
cmd.Stderr = hc.Stderr
@@ -282,13 +302,55 @@ func (r *shellRun) execGit(ctx context.Context, hc interp.HandlerContext, args [
return exitStatus(exit)
}
+// gitArg refuses a denied option, and any argument or option value that
+// names an existing file outside the workspace: git diff reads two such paths
+// even without --no-index, and blame --contents, -O and the like read one.
+// Values that name nothing on disk (a pickaxe string such as -S/api/orders)
+// pass.
+func (r *shellRun) gitArg(dir, a string) error {
+ var values []string
+ switch {
+ case a == "--":
+ case strings.HasPrefix(a, "--"):
+ name, value, _ := strings.Cut(a, "=")
+ if slices.Contains(gitDenied, name) {
+ return fmt.Errorf("%s is not available in this sandbox", name)
+ }
+ values = []string{value}
+ case strings.HasPrefix(a, "-"):
+ // A short option's value is attached after its letter (-O../x), and
+ // letters can be bundled before it (-pO../x).
+ for i := 2; i < len(a); i++ {
+ values = append(values, a[i:])
+ }
+ default:
+ values = []string{a}
+ }
+ for _, v := range values {
+ if v == "" {
+ continue
+ }
+ if _, err := r.s.Sandbox.Resolve(dir, v); err == nil {
+ continue
+ }
+ p := v
+ if !filepath.IsAbs(p) {
+ p = filepath.Join(dir, p)
+ }
+ if _, err := os.Lstat(p); err == nil {
+ return fmt.Errorf("%s: %w", v, ErrOutside)
+ }
+ }
+ return nil
+}
+
func (r *shellRun) open(ctx context.Context, path string, flag int, perm os.FileMode) (io.ReadWriteCloser, error) {
hc := interp.HandlerCtx(ctx)
abs, err := r.s.Sandbox.Resolve(hc.Dir, path)
if err != nil {
return nil, err
}
- if flag&(os.O_WRONLY|os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC) != 0 {
+ if flag&(os.O_WRONLY|os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC) != 0 && abs != "/dev/null" {
if err := r.s.Sandbox.Writable(abs); err != nil {
return nil, err
}
diff --git a/cli/test/evals/internal/tools/shell_test.go b/cli/test/evals/internal/tools/shell_test.go
index eb8f13cf..29939db5 100644
--- a/cli/test/evals/internal/tools/shell_test.go
+++ b/cli/test/evals/internal/tools/shell_test.go
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"os"
+ "os/exec"
"path/filepath"
"strings"
"testing"
@@ -45,6 +46,12 @@ func TestShellUtilities(t *testing.T) {
{`grep -c beta notes.txt`, "2"},
{`wc -l < notes.txt`, "4"},
{`X=$(echo nested); echo "got $X"`, "got nested"},
+ {`printf 'M\nx\nM\ny\nz\n' | grep -A3 M | tr '\n' ,`, "M,x,M,y,z,"},
+ {`printf 'M\na\nb\nc\nM\n' | grep -A1 M | tr '\n' ,`, "M,a,--,M,"},
+ {`printf 'a\nb\nM\nc\n' | grep -B1 -n M | tr '\n' ,`, "2-b,3:M,"},
+ {`grep -e alpha -e gamma notes.txt | wc -l`, "2"},
+ {`echo '{"a":"x"}' | jq .a -r`, "x"},
+ {`echo '{"a":{"b":1}}' | jq -c .a`, `{"b":1}`},
}
for _, c := range cases {
r := run(t, s, c.cmd)
@@ -77,6 +84,15 @@ func TestShellConfinement(t *testing.T) {
}
}
+// TestShellDevNullIsNotAWrite: discarding stderr is not a file operation,
+// so it never counts toward the trajectory grader's repeat limit.
+func TestShellDevNullIsNotAWrite(t *testing.T) {
+ s, _ := newShell(t)
+ if r := run(t, s, `grep beta notes.txt 2>/dev/null >/dev/null`); !r.OK || len(r.Ops) != 0 {
+ t.Fatalf("ok=%v ops=%+v", r.OK, r.Ops)
+ }
+}
+
func TestShellRecordsTracewayInPipelines(t *testing.T) {
s, _ := newShell(t)
r := run(t, s, `traceway endpoints list --since 24h --order-by p95 --output json | jq -r '.data[0].endpoint'`)
@@ -95,7 +111,7 @@ func TestShellRecordsTracewayInPipelines(t *testing.T) {
}
}
-// TestToolEnvIsScrubbed asserts plan §3.5: secrets in the harness's own
+// TestToolEnvIsScrubbed asserts that secrets in the harness's own
// environment never reach tool subprocesses.
func TestToolEnvIsScrubbed(t *testing.T) {
t.Setenv("OPENROUTER_API_KEY", "sk-or-secret")
@@ -142,3 +158,76 @@ func TestAskUserRules(t *testing.T) {
t.Fatal(got)
}
}
+
+// TestShellGitIsReadOnly holds git to the workspace: no config written into
+// the workspace can make git run a command on the host, and no argument reads
+// or writes a file outside it (the run directory around it holds seed.json).
+func TestShellGitIsReadOnly(t *testing.T) {
+ s, work := newShell(t)
+ s.Git = true
+ outside := filepath.Dir(work)
+ os.WriteFile(filepath.Join(outside, "seed.json"), []byte("answer-key"), 0o644)
+ for _, args := range [][]string{{"init", "-q"}, {"add", "notes.txt"}, {"-c", "user.email=x@y", "-c", "user.name=x", "commit", "-qm", "init"}} {
+ cmd := exec.Command("git", args...)
+ cmd.Dir = work
+ if out, err := cmd.CombinedOutput(); err != nil {
+ t.Skipf("git unavailable: %v %s", err, out)
+ }
+ }
+ const hook = `[core]\n\tfsmonitor = \"echo x > ` + "`" + `pwd` + "`" + `/../PWNED; false\"\n[diff]\n\texternal = \"echo x > ../PWNED; false\"\n`
+ write := &WriteFile{Sandbox: s.Sandbox}
+ for _, path := range []string{".git/config", ".GIT/config", "sub/.git/config", "bare/config"} {
+ res := write.Call(context.Background(), json.RawMessage(`{"path":"`+path+`","content":"`+hook+`"}`))
+ if strings.HasSuffix(path, ".git/config") || strings.HasSuffix(path, ".GIT/config") {
+ if res.OK {
+ t.Errorf("write_file wrote %s", path)
+ }
+ }
+ }
+ if r := run(t, s, `printf '[core]\n\tfsmonitor = "echo x > ../PWNED; false"\n' >> .git/config`); r.OK {
+ t.Errorf("the shell wrote .git/config: %q", r.Output)
+ }
+ // A bare repository's layout in a subdirectory, which git would otherwise
+ // discover and take its config from.
+ for _, f := range []string{"HEAD", "objects/x", "refs/x"} {
+ write.Call(context.Background(), json.RawMessage(`{"path":"bare/`+f+`","content":"ref: refs/heads/main\n"}`))
+ }
+ if r := run(t, s, `cd bare && git rev-parse --absolute-git-dir`); strings.TrimSpace(r.Output) != filepath.Join(work, ".git") {
+ t.Errorf("git in bare/ used %q, want the workspace repository", r.Output)
+ }
+ for _, cmd := range []string{`git status`, `git diff`, `git log -p -1`, `git show HEAD`, `cd sub && git status`} {
+ run(t, s, cmd)
+ }
+ if _, err := os.Stat(filepath.Join(outside, "PWNED")); err == nil {
+ t.Error("a git command ran a command written into the workspace")
+ }
+ for _, cmd := range []string{
+ `git diff --no-index ../seed.json /dev/null`,
+ `git diff --no-index -- ../seed.json /dev/null`,
+ `git diff ../seed.json /dev/null`,
+ `git diff -- ` + filepath.Join(outside, "seed.json") + ` /dev/null`,
+ `git blame --contents ../seed.json notes.txt`,
+ `git blame --contents=../seed.json notes.txt`,
+ `git log -pO../seed.json -1`,
+ `git log --output=../out.txt -1`,
+ `git diff --output ../out.txt HEAD`,
+ `git log --ext-diff -p -1`,
+ `git --exec-path=.. status`,
+ `git --config-env=core.fsmonitor=HOME status`,
+ `git -c core.pager=cat log`,
+ `cd .. && git diff seed.json /dev/null`,
+ } {
+ r := run(t, s, cmd)
+ if r.OK || strings.Contains(r.Output, "answer-key") {
+ t.Errorf("%s: ok=%v %q", cmd, r.OK, r.Output)
+ }
+ }
+ if _, err := os.Stat(filepath.Join(outside, "out.txt")); err == nil {
+ t.Error("git wrote a file outside the workspace")
+ }
+ for _, cmd := range []string{`git log --oneline | wc -l`, `git --no-pager log --oneline -S/api/orders | wc -l`, `git log --format=%s -- notes.txt`, `cd .skills && git status --short`} {
+ if r := run(t, s, cmd); !r.OK {
+ t.Errorf("%s: %q", cmd, r.Output)
+ }
+ }
+}
diff --git a/cli/test/evals/internal/tools/submit.go b/cli/test/evals/internal/tools/submit.go
index 44c080e9..2e9f7c70 100644
--- a/cli/test/evals/internal/tools/submit.go
+++ b/cli/test/evals/internal/tools/submit.go
@@ -13,22 +13,18 @@ import (
"github.com/tracewayapp/traceway/cli/test/evals/internal/record"
)
-// Submit names.
-const (
- SubmitPerf = "submit_perf_diagnosis"
- SubmitDebug = "submit_debug_diagnosis"
- Finish = "finish"
-)
+// SubmitPerf is the perf suite's final-answer tool.
+const SubmitPerf = "submit_perf_diagnosis"
-// DebugCategories are submit_debug_diagnosis's category values.
-var DebugCategories = []string{"null_reference", "missing_key", "type_or_contract_mismatch", "race_or_duplicate", "timeout", "config", "resource_exhaustion", "logic", "unknown"}
+// IsSubmit reports whether a tool is a final-answer tool.
+func IsSubmit(name string) bool { return name == SubmitPerf }
// Submit is a final-answer tool. It validates the payload against its
// schema and the call ids it cites; a failure goes back to the model for
// repair, up to MaxRepairs times, after which the last payload is accepted
// as-is for grading.
type Submit struct {
- Kind string // SubmitPerf | SubmitDebug | Finish
+ Kind string // the tool's name: SubmitPerf
CallExists func(id string) (ok, exists bool)
MaxRepairs int
@@ -37,63 +33,31 @@ type Submit struct {
}
func (t *Submit) Spec() Spec {
- switch t.Kind {
- case SubmitPerf:
- var rows []string
- for _, r := range checklist.Rows() {
- rows = append(rows, fmt.Sprintf("%s (%s)", r.ID, r.Title))
- }
- ids := checklist.IDs()
- item := map[string]any{"type": "string", "enum": ids}
- return Spec{Name: SubmitPerf, Description: "Submit your final performance diagnosis. This ends the task. " +
- "checklist_item is the bottleneck checklist row that explains the latency, or none_found when the latency is within expectations, or insufficient_evidence when the telemetry cannot tell. Rows: " + strings.Join(rows, "; ") + ". " +
- "Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",
- Parameters: obj(map[string]any{
- "checklist_item": item,
- "category": strProp("database, application, network, caching, infrastructure, or none"),
- "locus": obj(map[string]any{
- "endpoint": strProp("Endpoint name, e.g. GET /api/orders"),
- "span": strProp("The dominating span or query"),
- "service": strProp("Service name"),
- "host": strProp("Host or instance, when the cause is host-specific"),
- }),
- "onset": map[string]any{"type": []string{"string", "null"}, "description": "RFC 3339 time the slowness started, or null"},
- "evidence": evidenceSchema(),
- "ruled_out": map[string]any{"type": "array", "items": obj(map[string]any{"checklist_item": item, "call_id": strProp("Call id"), "reason": strProp("Why it is ruled out")}, "checklist_item", "reason")},
- "fix": strProp("The concrete fix"),
- "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},
- "report": strProp("Your report to the user, in Markdown"),
- }, "checklist_item", "locus", "evidence", "fix", "confidence", "report")}
- case SubmitDebug:
- return Spec{Name: SubmitDebug, Description: "Submit your final debugging diagnosis. This ends the task. Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",
- Parameters: obj(map[string]any{
- "issue_hash": strProp("The exception group hash (16 hex), if the bug has one"),
- "occurrence_id": strProp("The occurrence you anchored on, if any"),
- "category": map[string]any{"type": "string", "enum": DebugCategories},
- "culprit": obj(map[string]any{
- "file": strProp("Workspace-relative file"),
- "symbol": strProp("Function or method"),
- "line": intProp("Line number"),
- }, "file"),
- "evidence": evidenceSchema(),
- "fix_summary": strProp("What you changed or propose to change"),
- "files_changed": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
- "needs_instrumentation": map[string]any{"type": "boolean", "description": "True when the telemetry cannot explain the failure and you added instrumentation instead of a fix"},
- "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},
- "report": strProp("Your report to the user, in Markdown"),
- }, "category", "culprit", "evidence", "fix_summary", "needs_instrumentation", "confidence", "report")}
- default:
- return Spec{Name: Finish, Description: "Finish the task and report what you did. This ends the task.",
- Parameters: obj(map[string]any{
- "summary": strProp("Your report to the user, in Markdown"),
- "changes": map[string]any{"type": "array", "items": obj(map[string]any{
- "action": strProp("What you changed, e.g. snooze alert rule"),
- "target": strProp("The record, e.g. rule 12"),
- "new_value": strProp("The new state"),
- "undo": strProp("How to undo it"),
- }, "action", "target")},
- }, "summary", "changes")}
+ var rows []string
+ for _, r := range checklist.Rows() {
+ rows = append(rows, fmt.Sprintf("%s (%s)", r.ID, r.Title))
}
+ ids := checklist.IDs()
+ item := map[string]any{"type": "string", "enum": ids}
+ return Spec{Name: SubmitPerf, Description: "Submit your final performance diagnosis. This ends the task. " +
+ "checklist_item is the bottleneck checklist row that explains the latency, or none_found when the latency is within expectations, or insufficient_evidence when the telemetry cannot tell. Rows: " + strings.Join(rows, "; ") + ". " +
+ "Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",
+ Parameters: obj(map[string]any{
+ "checklist_item": item,
+ "category": strProp("database, application, network, caching, infrastructure, or none"),
+ "locus": obj(map[string]any{
+ "endpoint": strProp("Endpoint name, e.g. GET /api/orders"),
+ "span": strProp("The dominating span or query"),
+ "service": strProp("Service name"),
+ "host": strProp("Host or instance, when the cause is host-specific"),
+ }),
+ "onset": map[string]any{"type": []string{"string", "null"}, "description": "RFC 3339 time the slowness started, or null"},
+ "evidence": evidenceSchema(),
+ "ruled_out": map[string]any{"type": "array", "items": obj(map[string]any{"checklist_item": item, "call_id": strProp("Call id"), "reason": strProp("Why it is ruled out")}, "checklist_item", "reason")},
+ "fix": strProp("The concrete fix"),
+ "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},
+ "report": strProp("Your report to the user, in Markdown"),
+ }, "checklist_item", "locus", "evidence", "fix", "confidence", "report")}
}
func evidenceSchema() map[string]any {
@@ -165,35 +129,19 @@ func (t *Submit) validate(p map[string]any) []string {
}
}
}
- switch t.Kind {
- case SubmitPerf:
- need("checklist_item", "locus", "evidence", "fix", "confidence", "report")
- if v := str("checklist_item"); v != "" && !checklist.IsID(v) {
- probs = append(probs, fmt.Sprintf("checklist_item %q is not one of the allowed values", v))
- }
- if rs, ok := p["ruled_out"].([]any); ok {
- for i, r := range rs {
- m, _ := r.(map[string]any)
- if v, _ := m["checklist_item"].(string); v != "" && !checklist.IsID(v) {
- probs = append(probs, fmt.Sprintf("ruled_out[%d].checklist_item %q is not an allowed value", i, v))
- }
- }
- }
- checkEvidence()
- case SubmitDebug:
- need("category", "culprit", "evidence", "fix_summary", "needs_instrumentation", "confidence", "report")
- if v := str("category"); v != "" && !slices.Contains(DebugCategories, v) {
- probs = append(probs, fmt.Sprintf("category %q is not one of %v", v, DebugCategories))
- }
- if c, ok := p["culprit"].(map[string]any); ok {
- if f, _ := c["file"].(string); f == "" {
- probs = append(probs, "culprit.file is required")
+ need("checklist_item", "locus", "evidence", "fix", "confidence", "report")
+ if v := str("checklist_item"); v != "" && !checklist.IsID(v) {
+ probs = append(probs, fmt.Sprintf("checklist_item %q is not one of the allowed values", v))
+ }
+ if rs, ok := p["ruled_out"].([]any); ok {
+ for i, r := range rs {
+ m, _ := r.(map[string]any)
+ if v, _ := m["checklist_item"].(string); v != "" && !checklist.IsID(v) {
+ probs = append(probs, fmt.Sprintf("ruled_out[%d].checklist_item %q is not an allowed value", i, v))
}
}
- checkEvidence()
- default:
- need("summary", "changes")
}
+ checkEvidence()
if c := str("confidence"); c != "" && !slices.Contains([]string{"low", "medium", "high"}, c) {
probs = append(probs, "confidence must be low, medium or high")
}
diff --git a/cli/test/evals/internal/tools/tools.go b/cli/test/evals/internal/tools/tools.go
index 781be5e8..33de377c 100644
--- a/cli/test/evals/internal/tools/tools.go
+++ b/cli/test/evals/internal/tools/tools.go
@@ -80,33 +80,16 @@ func (s *Sandbox) Resolve(base, p string) (string, error) {
} else {
abs = filepath.Clean(filepath.Join(base, p))
}
- if !within(s.Root, abs) {
+ if !within(s.Root, abs) || !within(realPath(s.Root), realPath(abs)) {
return "", fmt.Errorf("%s: %w", p, ErrOutside)
}
- // Resolve the longest existing prefix to catch symlink escapes.
- probe := abs
- for {
- real, err := filepath.EvalSymlinks(probe)
- if err == nil {
- rootReal, _ := filepath.EvalSymlinks(s.Root)
- if rootReal == "" {
- rootReal = s.Root
- }
- if !within(rootReal, real) {
- return "", fmt.Errorf("%s: %w", p, ErrOutside)
- }
- break
- }
- parent := filepath.Dir(probe)
- if parent == probe || !within(s.Root, parent) {
- break
- }
- probe = parent
- }
return abs, nil
}
-// Writable reports whether abs may be written.
+// Writable reports whether abs may be written. Anything under a .git
+// directory is read-only wherever it sits: git reads its config, so a write
+// there could make a git command run a program on the host. Both the path and
+// what a symlink makes of it are checked, without regard to case.
func (s *Sandbox) Writable(abs string) error {
if abs == "/dev/null" {
return nil
@@ -115,14 +98,48 @@ func (s *Sandbox) Writable(abs string) error {
if err != nil {
return err
}
+ if err := s.readOnly(rel); err != nil {
+ return err
+ }
+ if real, err := filepath.Rel(realPath(s.Root), realPath(abs)); err == nil {
+ return s.readOnly(real)
+ }
+ return nil
+}
+
+func (s *Sandbox) readOnly(rel string) error {
+ rel = filepath.ToSlash(rel)
+ lower := strings.ToLower(rel)
+ for _, part := range strings.Split(lower, "/") {
+ if part == ".git" {
+ return fmt.Errorf("%s is read-only", rel)
+ }
+ }
for _, ro := range s.ReadOnly {
- if rel == ro || strings.HasPrefix(rel, ro+string(filepath.Separator)) {
- return fmt.Errorf("%s is read-only", filepath.ToSlash(rel))
+ ro = strings.ToLower(ro)
+ if lower == ro || strings.HasPrefix(lower, ro+"/") {
+ return fmt.Errorf("%s is read-only", rel)
}
}
return nil
}
+// realPath resolves the symlinks in the longest existing prefix of p.
+func realPath(p string) string {
+ rest := ""
+ for probe := p; ; {
+ if real, err := filepath.EvalSymlinks(probe); err == nil {
+ return filepath.Join(real, rest)
+ }
+ parent := filepath.Dir(probe)
+ if parent == probe {
+ return p
+ }
+ rest = filepath.Join(filepath.Base(probe), rest)
+ probe = parent
+ }
+}
+
// Rel returns the workspace-relative form of abs.
func (s *Sandbox) Rel(abs string) string {
rel, err := filepath.Rel(s.Root, abs)
diff --git a/cli/test/evals/models.yaml b/cli/test/evals/models.yaml
index d7ffb190..d6ff02e3 100644
--- a/cli/test/evals/models.yaml
+++ b/cli/test/evals/models.yaml
@@ -1,14 +1,15 @@
-# Pinned OpenRouter models for the eval agents and the judge. Every call goes
-# through OpenRouter Chat Completions with one key (OPENROUTER_API_KEY).
+# Pinned OpenRouter models for the eval agents and the judge, for paid runs
+# through OpenRouter Chat Completions. This harness makes no paid calls: it
+# plays oracles with the fake model and only records these ids in the
+# results header.
#
# Never use a floating alias (~provider/model-latest): results must stay
# comparable over time, so a model change is a reviewed edit to this file.
# Prices and supported_parameters were checked against
-# GET https://openrouter.ai/api/v1/models on 2026-10-17. The preflight reads live
-# prices; fallback_price is used only when that list cannot be fetched.
+# GET https://openrouter.ai/api/v1/models on 2026-10-17.
#
-# estimate is the per-run token envelope the preflight multiplies by live
-# prices until baseline.json carries measured averages (plan §10).
+# estimate (a per-run token envelope) and fallback_price (USD per million
+# tokens) are for estimating a paid run's cost before it starts.
agents:
opus:
diff --git a/cli/test/evals/presets.yaml b/cli/test/evals/presets.yaml
index 4352c574..f0b4c98b 100644
--- a/cli/test/evals/presets.yaml
+++ b/cli/test/evals/presets.yaml
@@ -1,4 +1,4 @@
-# Case and arm selections for `twevals plan|run --preset`. A --cases glob
+# Case and arm selections for `twevals list --preset`. A --cases glob
# overrides the preset's case filter; --arms overrides its arm policy.
presets:
smoke: