Skip to content

The only tool you need to know what is happening and how to fix it.

Harness started

narrow-adventure1372f5f021c7
diff --git a/.gitignore b/.gitignore
index c7636775..57b592a8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -64,6 +64,11 @@ backend/static/dist/
/benchmarks/_artifact/
benchmarks/loadgen/bench-loadgen
+# Eval harness (cli/test/evals): built binaries, oracle caches and run
+# results. Results are never committed.
+/cli/test/evals/.cache/
+/cli/test/evals/results/
+
# Rust build artifacts (oxc symbolicator shim)
**/oxc-shim/target/
symbolicator-bench.txt
diff --git a/CLAUDE.md b/CLAUDE.md
index dc955920..b7055cf6 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -34,6 +34,7 @@ Traceway is an error tracking and monitoring platform consisting of:
| CLI | `cd cli && just test` | Runs unit tests |
| CLI | `cd cli && just check` | Lint + test + vulncheck + skill drift + contract tests (pre-commit gate; CI enforces it for `cli/`) |
| CLI | `cd cli && just smoke-test` | Live E2E (needs `TRACEWAY_SMOKE_*` env vars) |
+| CLI | `cd cli && just eval-harness` | Eval harness tests plus every case's oracle trajectory, fake model only ($0) |
| Any | `nix develop` (or direnv via `.envrc`) | Dev shell with the Go and Node versions read from `backend/go.mod` and `.nvmrc`, plus `just`, `golangci-lint`, `govulncheck` (`flake.nix`; shells `default`, `backend`, `frontend`, `oxc`). Put `JWT_SECRET` in the gitignored `.envrc.local` |
| Any | `./scripts/check-node-pins.sh` | Asserts every Dockerfile's `ARG NODE_VERSION` default matches `.nvmrc`; run by `release-traceway.yml` before it builds |
diff --git a/cli/README.md b/cli/README.md
index 7654d7d7..3a4062a4 100644
--- a/cli/README.md
+++ b/cli/README.md
@@ -232,6 +232,14 @@ just smoke-test
If any of those vars is missing, the smoke tests skip cleanly rather than fail.
+## Skill and MCP evals
+
+`test/evals` measures how well AI agents use Traceway through the skills,
+against a seeded hermetic backend. It is **manual only** and free: nothing
+triggers it, and it plays every case with a fake model. Run
+`just eval-harness` (unit tests plus every oracle trajectory). See
+[`test/evals/README.md`](test/evals/README.md).
+
## Contributing
```bash
diff --git a/cli/justfile b/cli/justfile
index 3b2e656d..d2d92e39 100644
--- a/cli/justfile
+++ b/cli/justfile
@@ -23,6 +23,13 @@ lint:
vulncheck:
govulncheck ./...
+# Skill and MCP evals (test/evals). Manual only: none of these recipes is part
+# of check, test or default. See test/evals/README.md.
+
+# Free: harness tests plus every case's oracle trajectory, with the fake model.
+eval-harness:
+ cd test/evals && go test ./... && go run ./cmd/twevals oracle --all
+
# Regenerate the published skill (skills/traceway/) from the canonical
# knowledge chunks in pkg/mcpserver/knowledge.
gen-skills:
diff --git a/cli/test/evals/README.md b/cli/test/evals/README.md
index 7c4241f3..356b459c 100644
--- a/cli/test/evals/README.md
+++ b/cli/test/evals/README.md
@@ -1,57 +1,259 @@
# Traceway skill and MCP evals
-**Manual only.** Nothing here runs automatically, and nothing here calls a
-model yet: this directory holds eval cases and their fixtures. The harness
-that seeds a backend, drives an agent and grades the result is added next,
-in its own Go module under this directory.
+**Manual only, and free.** Nothing here runs automatically: no workflow, no
+`just check`/`test`/`default` recipe. This harness plays every case with a
+fake model on loopback and makes no paid model calls; its Chat Completions
+client refuses any non-loopback URL.
-The cases started as the performance scenarios of PR #405
-(`skills/traceway/evals/*.md`). They moved out of `skills/` because anything
-in a skill directory ships to users and to the agent under test, which would
-hand it the expected answers.
+This module measures how well AI agents use Traceway through its skills. Every
+run gets its own hermetic Traceway instance, the standalone backend in SQLite
+mode, seeded through the real ingest endpoints. The agent works through real
+`traceway` commands in a sandboxed shell, and deterministic graders score the
+result.
-## What changed in the port
+| Suite | Measures | Arms |
+|---|---|---|
+| perf | Diagnosing latency against the bottleneck checklist in `performance.md` | `cli+skill`, `cli-noskill` |
+
+`cli+skill` installs the `traceway` skill; `cli-noskill` is the same CLI with
+no skill, the baseline that skill uplift is measured against.
+
+## Running it
+
+From `cli/`:
+
+```
+just eval-harness
+```
+
+runs `go test ./...` and then every case's oracle on every arm it lists, with
+the fake model. From `cli/test/evals`, `go run ./cmd/twevals help` lists the
+commands:
+
+- `twevals list [--preset P] [--cases GLOB] [--arms A]` prints the case × arm
+ combinations a selection picks.
+- `twevals oracle [--all | <case-id>...] [--arms A] [-j N] [--keep]` plays
+ the oracles and writes `results.json` and `summary.md`. It exits `1` when
+ an oracle fails.
+
+Presets (`presets.yaml`): `smoke` is the starred cases on the primary arm,
+`core` every case on the primary arm, `full` every arm a case lists.
+
+Requirements: Go (the `go`/`toolchain` lines of `go.mod`) and git. The
+harness builds the backend and both CLI editions from this checkout into
+`.cache/` (gitignored) and reuses them until the sources change.
-Each Markdown scenario became a structured case with one correct answer, one
-named distractor, and telemetry that the real SDKs and the real ingest path
-can produce:
+## Reading the results
-| PR #405 file | Case | What was fixed |
+A run is `pass`, `fail`, `budget_exhausted` (counted as a fail) or
+`infra_error` (excluded from pass rates).
+
+Each run directory, `runs/<case>/<arm>/<model>/r<N>/`, holds:
+
+- `transcript.jsonl`: every model response, every tool call with its id, op,
+ argv, exit code and output;
+- `run.json` and `result.json`;
+- `seed.json`: the ids the seed exposed. The agent never sees it;
+- the workspace, when the run failed or `--keep` was set.
+
+Graders:
+
+| Grader | Gates | Checks |
|---|---|---|
-| `backend-go-select-n-plus-1` | `perf-db-n-plus-1` | Consistent counts: N item queries for N orders |
-| `backend-missing-index` | `perf-db-missing-index` | No `EXPLAIN` handed to the agent; a 30-day window that matches the claim |
-| `backend-node-sequential-calls` | `perf-net-sequential` | No unexplained 700ms per hop |
-| `backend-python-pool-exhaustion` | `perf-db-pool-exhaustion` | Pool-wait gaps that add up to the request time |
-| `backend-sync-background-task` | `perf-app-sync-background` | Fix advice that does not depend on the retired `CaptureTask` |
-| `frontend-js-large-payloads` | `perf-net-large-payload` | A real `http.response.body.size` attribute instead of a fictional `transfer` span |
-| `frontend-js-no-cache` | `perf-cache-no-cache` | A browser project is present, and only one checklist row is correct |
-| `infra-cpu-saturation` | `perf-infra-cpu-saturation` | Saturation driven by traffic, not by a deploy |
-| `infra-cache-stampede` | `perf-cache-stampede` | Realistic request rates |
-| `mobile-ios-cold-start` | `perf-xs-ios-watchdog` | Built only from telemetry the iOS SDK sends: a launch watchdog crash whose cause is a slow backend endpoint |
-| `mobile-android-gc-pauses` | `perf-xs-android-timeouts`, `debug-android-npe` | The Android SDK sends crashes, actions and logs, not frame or GC metrics |
-| `mobile-flutter-shader-jank` | `debug-flutter-null-check` | The Flutter SDK sends errors, not frame timings |
+| `answer` | yes | The structured answer: checklist row (and not the distractor), locus, onset |
+| `trajectory` | yes | Required, alternative, forbidden and ordered operations, and thrash limits, on surface-neutral operations (`internal/ops`) |
+| `grounding` | yes | Cited calls exist, succeeded and were not truncated. Their output contains the deciding signal. No id in the answer was invented |
+| `report` | yes | Regexps the final report must or must not match, for cases that set `expect.report` |
+
+## Adding a case
+
+A case is three things:
+
+- a seed function in `internal/scenarios/<suite>/`, registered by name;
+- a YAML file `cases/<suite>/<id>.yaml`;
+- fixtures under `testdata/`.
+
+`internal/suite` tests the catalog as a whole: references exist, checklist ids
+are valid, and the leakage lint is clean. Done means:
+
+1. The seed builds data with exactly one correct answer and one named
+ distractor (`distractor:` in the YAML).
+2. `twevals oracle <id>` passes on every arm the case lists.
+3. `go test ./internal/suite/` passes.
+
+`perf-db-n-plus-1` is the exemplar.
+
+### The seed DSL (`internal/seed`)
+
+```go
+func init() { scenarios.Register("perf/db-n-plus-1", seedNPlusOne) }
+
+func seedNPlusOne(s *seed.Scenario) error {
+ p := s.Project("storefront", seed.OTel) // a real project, created through the API
+ api := p.Service("orders-api", seed.Go("nethttp", "sql")).Versions("2.14.3")
+ orders := api.Endpoint("GET", "/api/orders")
+ orders.Traffic(seed.Daily(160), seed.Diurnal(), seed.Over(29*24*time.Hour))
+ orders.Handler(func(r *seed.Req) {
+ n := int(r.Lerp(6, 72) * (0.35 + 0.95*r.Float())) // grows over the window
+ r.DB("SELECT * FROM orders WHERE user_id = $1 ...", r.Ms(3, 6), seed.Rows(n))
+ for range n {
+ r.DB("SELECT id, sku, qty, unit_price FROM order_items WHERE order_id = $1", r.Ms(6, 14))
+ }
+ r.Compute("encode response", r.Ms(2, 6))
+ })
+ s.Background(api, seed.TypicalTraffic(10)) // healthy endpoints, a benign 400, host metrics
+ api.Host().CPU(seed.Flat(0.22, 0.31))
+ s.Expose("slowRequest", orders.Slowest(24*time.Hour)) // for the oracle and graders, never the agent
+ return nil
+}
+```
+
+- **Timing is consistent by construction.** Work is laid out on a cursor.
+ `r.DB`, `r.HTTP`, `r.Cache`, `r.Compute` and `r.Call(otherEndpoint, rtt)`
+ each add a child span. `r.Gap(d)` is waiting time with no span. `r.Span`
+ nests, and `r.Parallel` runs branches side by side. A parent's duration is
+ where its cursor stops.
+- **Releases** are `Versions(initial, seed.Deploy{At, Version})`, which sets
+ the `service.version` resource attribute. `r.Version()` lets the handler
+ change behaviour after a deploy.
+- **Multiple instances** are `Instances("node-a", ...)`, with `r.Host()` and
+ `HostNamed(h).CPU(...)`.
+- **Traffic shapes** are `Daily`, `Diurnal`, `GrowRate`, `Burst`, `RateFunc`
+ and `Over`. Use `Statuses` or `r.Status` to fail requests.
+- **Logs and errors:**
+ - `r.Log(sev, body, attrs...)` writes a trace-correlated OTLP log.
+ - `svc.Log(at, ...)` writes a log outside any request.
+ - `r.Error(seed.Exc{Type, Message, Stack})` records an exception event in
+ the stack format of the service's language.
+ - Every detailed request also gets an access-log line,
+ `GET /path 200 1243ms`, with its trace id. That is how an agent finds a
+ slow request's id, because `endpoints list` has none.
+- **Metrics** come from `Host().CPU/Memory/Disk/Filesystem` (OTel
+ hostmetrics names, `system.cpu.utilization` split by `state`) and from
+ `svc.Gauge/Counter/Histogram(name, unit, attrs, series)`. Histograms arrive
+ as `<name>.avg` and `<name>.count`. Series helpers: `Flat`, `Step`, `Ramp`,
+ `Periodic`, `Sawtooth`, `Plus`, `Scale`, `Clamp`.
+- **Browser and mobile SDK data** goes through `p.Crash(seed.Crash{...})`, with
+ network and navigation `Actions` and console `Logs`, and `p.Session(...)`.
+ It travels the native `/api/report` protocol, and covers only what those
+ SDKs really send: no frame timings, no mobile metrics. `p.GroupMatching`
+ reads an exception group's hash back after ingest; put work that needs the
+ ingested data in `s.AfterFlush`.
+- **Volume.** Requests outside the detail sample are sent as their root span
+ (and any downstream server spans), so percentiles and counts stay complete
+ while a scenario stays under about 20k spans. Requests with logs, errors or
+ an `Expose` always keep their full tree.
+
+Backend facts that bite (each one cost a case author an afternoon):
+
+- Health-check routes (`/health`, `/healthz`, `/ping`, `/ready`, ...) are
+ dropped at ingest, so an endpoint seeded on them never appears and `Verify`
+ fails with "endpoints never became visible".
+- `endpoints list --order-by p95|lastSeen` does not sort: the CLI sends
+ `p95`/`lastSeen`, the SQLite/DuckDB repositories only know
+ `p95_duration`/`last_seen`, and the list falls back to impact order. Keep
+ a case's endpoint count at ten or fewer so the slow one is on the first page
+ whatever the order. This is a product bug, not a harness one; the evals
+ should catch the skill's reliance on it.
+- A log body search over more than 24 hours answers 422 unless another filter
+ (service, severity, trace) narrows it.
+- Rare events confined to a short window can come out empty at some hours of
+ the day (the arrival process follows the wall-clock hour). Use `Burst`, or
+ seed the case at every hour of the day into `seed.NewDrySink()` and check
+ the event is there before relying on it.
+
+- Numeric OTel attributes must be integers, which the DSL does for you.
+- Projects with a browser framework never get endpoints from OTLP.
+- An exception's message does not split groups: vary the type or the frames
+ to get distinct issues.
+- Session recordings are stamped with the ingest time, so keep crashes that
+ carry actions within the last ~12 hours.
+- The unscoped `metrics query --name system.cpu.utilization --aggregation max`
+ returns the idle state. That is a real trap the skill should steer around.
+
+### The case YAML
+
+See `perf-db-n-plus-1.yaml` for the full shape. The keys:
+
+- `arms` and `invoke`: `slash` preloads SKILL.md the way a `/traceway`
+ command does, and `natural` makes the agent load the skill itself.
+- `workspace.repo`: a fixture under `testdata/repos`. Fixtures can carry
+ history: a `history.yaml` whose commits are overlays under `.history/`
+ (only the files that differ from the fixture, plus an `absent:` list of
+ files the commit lacks), committed with ages relative to the run so
+ `git log` lines up with the seeded deploys.
+- `prompt`, `budget`, `user`: scripted `ask_user` answers, first match wins.
+- `expect.answer`, `expect.grounding`, `expect.calls`, `expect.report`.
+- `judge.rubric` and `judge.expected`, the rubric for report quality. The
+ harness does not score it.
+
+Templates are resolved after seeding:
+
+| Template | Value |
+|---|---|
+| `{{seed.<key>}}` | What the seed exposed |
+| `{{env.url}}` | The instance URL |
+| `{{env.project}}` | The primary project's id |
+| `{{env.projects.<name>.id}}` | Another project's id |
+| `{{env.stub}}` | The switchable stub's URL |
+| `{{env.now}}` | The scenario's "now" |
+
+A scalar that is exactly one template keeps its type.
+
+The `oracle` is the reference trajectory the fake model plays:
+
+- **One step is one tool call**, so steps are cited as `c01`, `c02`, ... in
+ the submitted evidence.
+- **`shell:` steps** run CLI commands. Use `jq` to keep cited outputs under
+ the 30k-character cap: a truncated citation fails grounding.
+- **`{{last.<json.path>}}` and `{{calls.c03.<path>}}`** read earlier tool
+ results at play time.
+- **`load_skill:` steps** play as a harmless `list_files .` on arms without
+ skills (`cli-noskill`), so call ids stay the same on every arm.
+- **The oracle ends** with `submit:`.
+
+While developing a case, `TWEVALS_ONLY=<ids>` limits loading to those cases,
+so an unfinished YAML file elsewhere cannot break your run.
+
+### Leakage
+
+The seeded strings (span names, attribute values, log bodies, stack traces,
+object names) and the prompt must not name the answer. `internal/grade/leakage.go`
+holds the denylist, which covers phrases like `n+1`, `stampede`,
+`missing index`, `pool exhaust`, `gc pause` and `noisy neighbor`. Use real
+library messages, such as SQLAlchemy's `QueuePool limit of size 5 overflow 10
+reached` or Postgres's `deadlock detected`, and neutral names.
## Layout
```
-cases/perf/<id>.yaml 11 performance cases
-cases/debug/<id>.yaml 2 mobile debugging cases
-testdata/repos/<repo>/ the fixture repository each case's agent works in
-testdata/oracle/<case>/ files the reference trajectory writes (debug fixes)
+models.yaml presets.yaml
+cmd/twevals/ list | oracle
+internal/env/ backend subprocess, registration, PATs, CLI logins, the stub
+internal/seed/ the DSL, OTLP and /api/report writers, read-back verification
+internal/scenarios/ one seed function per case
+internal/agent/ the Chat Completions client (loopback only), the loop, prompt caching, transcripts, the fake model
+internal/tools/ the sandboxed shell (mvdan.cc/sh, in-process coreutils, gojq, goawk), files, skills, ask_user, submit
+internal/ops/ CLI → operation mapping, cross-checked against tool-map.md
+internal/grade/ answer, trajectory, grounding, report, leakage
+internal/runner/ one run end to end
+internal/report/ results.json and summary.md
+cases/ testdata/ cases, fixture repos, oracle files
```
-A case names its seed (the Go function that will generate its telemetry),
-its arms (`cli+skill`, `cli-noskill`), the prompt, budgets, the expected
-answer, the tool calls that must, may and must not happen, a judge rubric,
-and an `oracle`: the reference trajectory, one tool call per step, ending in
-the structured answer.
-
-A fixture repo can carry git history: `history.yaml` lists commits, each an
-overlay under `.history/` (only the files that differ from the fixture, plus
-an `absent:` list of files the commit lacks), with ages relative to the run so
-`git log` lines up with the seeded deploys.
+The agent's shell is an interpreter, never the host shell. It has the
+`traceway` binary (logged in with a read PAT whose config lives outside the
+workspace), `jq`, `awk`, read-only text utilities confined to the workspace,
+and read-only `git`. Tool subprocesses get a scrubbed environment (`HOME`,
+`XDG_*`, `PATH`, `TZ`, `LANG`), so secrets in the harness's environment
+never reach them.
## Credits
The first performance scenarios were contributed by Liyu in PR #405
-("feat(skills): add performance skill evals").
+("feat(skills): add performance skill evals"). They were ported to this
+harness as `perf-db-n-plus-1`, `perf-db-missing-index`, `perf-net-sequential`,
+`perf-db-pool-exhaustion`, `perf-app-sync-background`, `perf-net-large-payload`,
+`perf-cache-no-cache`, `perf-infra-cpu-saturation`, `perf-cache-stampede`,
+`perf-xs-ios-watchdog`, `perf-xs-android-timeouts`, `debug-android-npe` and
+`debug-flutter-null-check`.
diff --git a/cli/test/evals/cmd/twevals/commands.go b/cli/test/evals/cmd/twevals/commands.go
new file mode 100644
index 00000000..a8897e35
--- /dev/null
+++ b/cli/test/evals/cmd/twevals/commands.go
@@ -0,0 +1,190 @@
+package main
+
+import (
+ "context"
+ "errors"
+ "flag"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+ "text/tabwriter"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/report"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"
+ _ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios/all"
+)
+
+type workspace struct {
+ root string // cli/test/evals
+ repo string
+ cases []*catalog.Case
+ presets *catalog.Presets
+ models *catalog.Models
+}
+
+func load() (*workspace, error) {
+ root := os.Getenv("TWEVALS_ROOT")
+ if root == "" {
+ wd, _ := os.Getwd()
+ for d := wd; ; d = filepath.Dir(d) {
+ if _, err := os.Stat(filepath.Join(d, "models.yaml")); err == nil {
+ if _, err := os.Stat(filepath.Join(d, "presets.yaml")); err == nil {
+ root = d
+ break
+ }
+ }
+ if filepath.Dir(d) == d {
+ return nil, errors.New("run twevals from cli/test/evals (or set TWEVALS_ROOT)")
+ }
+ }
+ }
+ repo, err := env.RepoRoot(root)
+ if err != nil {
+ return nil, err
+ }
+ cases, err := catalog.LoadCases(root)
+ if err != nil {
+ return nil, err
+ }
+ presets, err := catalog.LoadPresets(root)
+ if err != nil {
+ return nil, err
+ }
+ models, err := catalog.LoadModels(root)
+ if err != nil {
+ return nil, err
+ }
+ return &workspace{root: root, repo: repo, cases: cases, presets: presets, models: models}, nil
+}
+
+// parseInterleaved parses flags that may come after positional arguments.
+func parseInterleaved(fs *flag.FlagSet, args []string) []string {
+ var pos []string
+ for {
+ _ = fs.Parse(args)
+ if fs.NArg() == 0 {
+ return pos
+ }
+ pos = append(pos, fs.Arg(0))
+ args = fs.Args()[1:]
+ }
+}
+
+type selFlags struct{ sel catalog.Selection }
+
+func (s *selFlags) bind(fs *flag.FlagSet) {
+ fs.StringVar(&s.sel.Preset, "preset", "", "smoke | core | full")
+ fs.StringVar(&s.sel.Cases, "cases", "", "comma-separated case ids or globs (overrides the preset)")
+ fs.StringVar(&s.sel.Suites, "suites", "", "comma-separated suites")
+ fs.StringVar(&s.sel.Tags, "tags", "", "comma-separated tags, all required")
+ fs.StringVar(&s.sel.Arms, "arms", "", "comma-separated arms")
+}
+
+func (s *selFlags) selection() catalog.Selection {
+ if s.sel.Preset == "" && s.sel.Cases == "" && s.sel.Suites == "" && s.sel.Tags == "" {
+ s.sel.Preset = "smoke"
+ }
+ return s.sel
+}
+
+func cmdList(args []string) error {
+ fs := flag.NewFlagSet("list", flag.ExitOnError)
+ var sf selFlags
+ sf.bind(fs)
+ _ = fs.Parse(args)
+ ws, err := load()
+ if err != nil {
+ return err
+ }
+ sel := sf.sel
+ if sel.Preset == "" && sel.Cases == "" {
+ sel.Preset = "full"
+ }
+ units, err := catalog.Select(ws.cases, ws.presets, sel)
+ if err != nil {
+ return err
+ }
+ tw := tabwriter.NewWriter(os.Stdout, 2, 4, 2, ' ', 0)
+ fmt.Fprintln(tw, "CASE\tARM\tTAGS\tTITLE")
+ for _, u := range units {
+ fmt.Fprintf(tw, "%s\t%s\t%s\t%s\n", u.Case.ID, u.Arm, strings.Join(u.Case.Tags, ","), u.Case.Title)
+ }
+ tw.Flush()
+ fmt.Printf("\n%d runs per model\n", len(units))
+ return nil
+}
+
+func cmdOracle(ctx context.Context, args []string) error {
+ fs := flag.NewFlagSet("oracle", flag.ExitOnError)
+ all := fs.Bool("all", false, "every case")
+ arms := fs.String("arms", "", "comma-separated arms (default: every arm the case lists)")
+ jobs := fs.Int("j", 3, "parallel runs")
+ keep := fs.Bool("keep", false, "keep run directories")
+ suites := fs.String("suites", "", "comma-separated suites")
+ out := fs.String("out", "", "results directory")
+ ids := parseInterleaved(fs, args)
+ ws, err := load()
+ if err != nil {
+ return err
+ }
+ sel := catalog.Selection{Preset: "full", Arms: *arms, Suites: *suites}
+ if !*all {
+ if len(ids) == 0 && *suites == "" {
+ return errors.New("name case ids, or pass --all")
+ }
+ sel.Cases = strings.Join(ids, ",")
+ }
+ units, err := catalog.Select(ws.cases, ws.presets, sel)
+ if err != nil {
+ return err
+ }
+ bins, err := env.BuildBinaries(ws.repo, filepath.Join(ws.root, ".cache"))
+ if err != nil {
+ return err
+ }
+ outDir := *out
+ if outDir == "" {
+ outDir = filepath.Join(ws.root, ".cache", "oracle", time.Now().UTC().Format("20060102-150405"))
+ }
+ if err := os.MkdirAll(outDir, 0o755); err != nil {
+ return err
+ }
+ oracleModel := &catalog.Model{Key: "oracle", ID: "fake/oracle", Caching: "none"}
+ opt := runner.Options{EvalsRoot: ws.root, RepoRoot: ws.repo, OutDir: outDir, Bins: bins, Models: []*catalog.Model{oracleModel},
+ Runs: 1, Jobs: *jobs, Keep: *keep, Log: os.Stderr}
+ header := report.NewHeader(ws.repo, filepath.Base(outDir), ws.models, bins.Hash, os.Args)
+ header.Oracle = true
+ results := runner.Execute(ctx, opt, runner.Jobs(units, opt.Models, 1))
+ header.Finished = time.Now().UTC()
+ if err := report.Save(outDir, &report.Results{Header: header, Runs: results}); err != nil {
+ return err
+ }
+ failed := 0
+ for _, r := range results {
+ if r.Status != "pass" {
+ failed++
+ fmt.Fprintf(os.Stderr, "FAIL %s: %s\n", r.Key(), oneLine(r))
+ }
+ }
+ fmt.Fprintf(os.Stderr, "\noracle: %d/%d passed (%s)\n", len(results)-failed, len(results), outDir)
+ if failed > 0 {
+ return &exitError{1, ""}
+ }
+ return nil
+}
+
+func oneLine(r *runner.Result) string {
+ if r.Error != "" {
+ return r.Status + ": " + r.Error
+ }
+ for _, g := range r.Graders {
+ if g.Gating && !g.Skipped && !g.Pass {
+ return r.Status + ": " + g.Grader + ": " + strings.Join(g.Failures, "; ")
+ }
+ }
+ return r.Status
+}
diff --git a/cli/test/evals/cmd/twevals/main.go b/cli/test/evals/cmd/twevals/main.go
new file mode 100644
index 00000000..71e01b66
--- /dev/null
+++ b/cli/test/evals/cmd/twevals/main.go
@@ -0,0 +1,61 @@
+// Command twevals runs the Traceway skill and MCP evals with the fake model
+// that plays each case's oracle. It makes no paid model calls.
+package main
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "os"
+ "os/signal"
+ "syscall"
+)
+
+const usage = `twevals: Traceway skill and MCP evals (oracle runs only, free)
+
+ twevals list [--preset P] [--suites S] [--tags T] [--cases GLOB] [--arms A]
+ twevals oracle [--all | <case-id>...] [--arms A] [-j N] [--keep]
+
+Selection flags: --preset smoke|core|full, --cases (comma-separated ids or globs; overrides the preset), --suites, --tags, --arms.
+Exit codes: 0 ok, 1 failed oracle.
+`
+
+func main() {
+ ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
+ defer stop()
+ if len(os.Args) < 2 {
+ fmt.Fprint(os.Stderr, usage)
+ os.Exit(2)
+ }
+ var err error
+ args := os.Args[2:]
+ switch os.Args[1] {
+ case "list":
+ err = cmdList(args)
+ case "oracle":
+ err = cmdOracle(ctx, args)
+ case "-h", "--help", "help":
+ fmt.Print(usage)
+ return
+ default:
+ err = fmt.Errorf("unknown command %q", os.Args[1])
+ }
+ if err != nil {
+ var ex *exitError
+ if errors.As(err, &ex) {
+ if ex.msg != "" {
+ fmt.Fprintln(os.Stderr, "twevals:", ex.msg)
+ }
+ os.Exit(ex.code)
+ }
+ fmt.Fprintln(os.Stderr, "twevals:", err)
+ os.Exit(1)
+ }
+}
+
+type exitError struct {
+ code int
+ msg string
+}
+
+func (e *exitError) Error() string { return e.msg }
diff --git a/cli/test/evals/go.mod b/cli/test/evals/go.mod
new file mode 100644
index 00000000..27dd8c25
--- /dev/null
+++ b/cli/test/evals/go.mod
@@ -0,0 +1,30 @@
+module github.com/tracewayapp/traceway/cli/test/evals
+
+go 1.26.2
+
+toolchain go1.26.9
+
+require (
+ github.com/benhoyt/goawk v1.32.0
+ github.com/google/uuid v1.6.0
+ github.com/itchyny/gojq v0.12.19
+ github.com/tracewayapp/traceway/cli v1.9.15
+ go.opentelemetry.io/proto/otlp v1.10.0
+ google.golang.org/protobuf v1.36.11
+ gopkg.in/yaml.v3 v3.0.1
+ mvdan.cc/sh/v3 v3.14.1
+)
+
+require (
+ github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
+ github.com/itchyny/timefmt-go v0.1.8 // indirect
+ golang.org/x/net v0.50.0 // indirect
+ golang.org/x/sys v0.47.0 // indirect
+ golang.org/x/term v0.45.0 // indirect
+ golang.org/x/text v0.34.0 // indirect
+ google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 // indirect
+ google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57 // indirect
+ google.golang.org/grpc v1.79.2 // indirect
+)
+
+replace github.com/tracewayapp/traceway/cli => ../..
diff --git a/cli/test/evals/go.sum b/cli/test/evals/go.sum
new file mode 100644
index 00000000..628b8809
--- /dev/null
+++ b/cli/test/evals/go.sum
@@ -0,0 +1,69 @@
+github.com/benhoyt/goawk v1.32.0 h1:CK/mnBxGsNqX8hdghPIctl2aGIcwfnhT6sv+9M/bY7w=
+github.com/benhoyt/goawk v1.32.0/go.mod h1:jXTQxBxtQ0VsjFqc8dw7tIJj3SDzQN8kcdMq7r83/ZA=
+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
+github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
+github.com/go-quicktest/qt v1.102.0 h1:HSQxCeh5YZH3EL3W39ixjtyaEhcWSXQHtHnMBzSs474=
+github.com/go-quicktest/qt v1.102.0/go.mod h1:p4lGIVX+8Wa6ZPNDvqcxq36XpUDLh42FLetFU7odllI=
+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0/go.mod h1:JfhWUomR1baixubs02l85lZYYOm7LV6om4ceouMv45c=
+github.com/itchyny/gojq v0.12.19 h1:ttXA0XCLEMoaLOz5lSeFOZ6u6Q3QxmG46vfgI4O0DEs=
+github.com/itchyny/gojq v0.12.19/go.mod h1:5galtVPDywX8SPSOrqjGxkBeDhSxEW1gSxoy7tn1iZY=
+github.com/itchyny/timefmt-go v0.1.8 h1:1YEo1JvfXeAHKdjelbYr/uCuhkybaHCeTkH8Bo791OI=
+github.com/itchyny/timefmt-go v0.1.8/go.mod h1:5E46Q+zj7vbTgWY8o5YkMeYb4I6GeWLFnetPy5oBrAI=
+github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
+github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=
+github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
+go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
+go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
+go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=
+go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=
+go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=
+go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=
+go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=
+go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=
+go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=
+go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=
+go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=
+go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=
+go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g=
+go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk=
+golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=
+golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=
+golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
+golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
+golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
+golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
+golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
+gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
+gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
+google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 h1:JLQynH/LBHfCTSbDWl+py8C+Rg/k1OVH3xfcaiANuF0=
+google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57/go.mod h1:kSJwQxqmFXeo79zOmbrALdflXQeAYcUbgS7PbpMknCY=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57 h1:mWPCjDEyshlQYzBpMNHaEof6UX1PmHcaUODUywQ0uac=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ=
+google.golang.org/grpc v1.79.2 h1:fRMD94s2tITpyJGtBBn7MkMseNpOZU8ZxgC3MMBaXRU=
+google.golang.org/grpc v1.79.2/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
+google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
+google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+mvdan.cc/sh/v3 v3.14.1 h1:bXkhQWNHCs0KZEChF8hYS6FC+T2N9mUZLbQv9blditI=
+mvdan.cc/sh/v3 v3.14.1/go.mod h1:syYCoFET8w9tvevxiXUtY8/ICrU+l26jHmhJDra3Vwo=
diff --git a/cli/test/evals/internal/agent/agent_test.go b/cli/test/evals/internal/agent/agent_test.go
new file mode 100644
index 00000000..6e0ff134
--- /dev/null
+++ b/cli/test/evals/internal/agent/agent_test.go
@@ -0,0 +1,200 @@
+package agent
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync"
+ "testing"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"
+)
+
+// scripted is a Chatter that returns canned responses and records requests.
+type scripted struct {
+ mu sync.Mutex
+ resps []*Response
+ reqs []*Request
+}
+
+func (s *scripted) Chat(_ context.Context, req *Request) (*Response, error) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ s.reqs = append(s.reqs, req)
+ if len(s.resps) == 0 {
+ return nil, errors.New("script exhausted")
+ }
+ r := s.resps[0]
+ s.resps = s.resps[1:]
+ return r, nil
+}
+
+func toolResp(name, args string, cost float64) *Response {
+ r := &Response{Provider: "test"}
+ r.Choices = append(r.Choices, struct {
+ Message Message `json:"message"`
+ FinishReason string `json:"finish_reason"`
+ }{Message: Message{Role: "assistant", ToolCalls: []ToolCall{{ID: "t1", Type: "function", Function: FunctionCall{Name: name, Arguments: args}}}}})
+ r.Usage.Cost = cost
+ r.Usage.PromptTokens = 1000
+ return r
+}
+
+func textResp(text string) *Response {
+ r := &Response{}
+ r.Choices = append(r.Choices, struct {
+ Message Message `json:"message"`
+ FinishReason string `json:"finish_reason"`
+ }{Message: Message{Role: "assistant", Content: text}})
+ return r
+}
+
+type echoTool struct{}
+
+func (echoTool) Spec() tools.Spec {
+ return tools.Spec{Name: "echo", Parameters: map[string]any{"type": "object"}}
+}
+func (echoTool) Call(context.Context, json.RawMessage) tools.Result {
+ return tools.Result{Output: "echoed", OK: true}
+}
+
+func cfg(s *scripted) Config {
+ return Config{
+ Model: &catalog.Model{Key: "m", ID: "x/y", Caching: "anthropic"}, Chat: s, System: "sys", Prompt: "do it", SubmitTool: tools.Finish,
+ Tools: []tools.Tool{echoTool{}, &tools.Submit{Kind: tools.Finish, MaxRepairs: 2}},
+ Budget: catalog.Budget{MaxTurns: 5, MaxToolCalls: 3},
+ }
+}
+
+func TestLoopSubmits(t *testing.T) {
+ s := &scripted{resps: []*Response{toolResp("echo", `{}`, 0.01), toolResp("finish", `{"summary":"done","changes":[]}`, 0.01)}}
+ run := Run(context.Background(), cfg(s))
+ if run.StopReason != StopSubmitted || run.Submission["summary"] != "done" || len(run.Calls) != 2 || run.Calls[0].ID != "c01" {
+ t.Fatalf("run = %+v", run)
+ }
+ if run.Usage.CostUSD != 0.02 {
+ t.Fatalf("cost %v", run.Usage.CostUSD)
+ }
+ // The second request carries the assistant tool call before its result,
+ // and the result starts with the call id the model cites.
+ msgs := s.reqs[1].Messages
+ if msgs[2].Role != "assistant" || len(msgs[2].ToolCalls) != 1 || msgs[3].Role != "tool" {
+ t.Fatalf("history order: %+v", msgs)
+ }
+ parts, ok := msgs[3].Content.([]Part)
+ if !ok || !strings.HasPrefix(parts[0].Text, "[call c01]") || parts[0].CacheControl == nil {
+ t.Fatalf("the newest tool result must carry the call id and a cache breakpoint: %#v", msgs[3].Content)
+ }
+ if sys, ok := msgs[0].Content.([]Part); !ok || sys[0].CacheControl == nil {
+ t.Fatal("the system prompt must carry a cache breakpoint for anthropic models")
+ }
+ if _, ok := msgs[1].Content.(string); !ok {
+ t.Fatal("only the system prompt and the newest message get breakpoints")
+ }
+}
+
+func TestLoopNudgesOnceThenStops(t *testing.T) {
+ s := &scripted{resps: []*Response{textResp("I think it is fine."), textResp("Really.")}}
+ run := Run(context.Background(), cfg(s))
+ if run.StopReason != StopNoSubmit || len(s.reqs) != 2 {
+ t.Fatalf("stop %s after %d requests", run.StopReason, len(s.reqs))
+ }
+ last := s.reqs[1].Messages
+ if !strings.Contains(textOf(last[len(last)-1]), "call finish") {
+ t.Fatal("the nudge must name the submit tool")
+ }
+}
+
+func textOf(m Message) string {
+ if s, ok := m.Content.(string); ok {
+ return s
+ }
+ if ps, ok := m.Content.([]Part); ok && len(ps) > 0 {
+ return ps[0].Text
+ }
+ return ""
+}
+
+func TestLoopBudgets(t *testing.T) {
+ var resps []*Response
+ for i := 0; i < 10; i++ {
+ resps = append(resps, toolResp("echo", `{}`, 0))
+ }
+ run := Run(context.Background(), cfg(&scripted{resps: resps}))
+ if run.StopReason != StopBudget {
+ t.Fatalf("stop %s", run.StopReason)
+ }
+ rejected := 0
+ for _, c := range run.Calls {
+ if c.Rejected {
+ rejected++
+ }
+ }
+ if rejected == 0 {
+ t.Fatal("calls past max_tool_calls must be refused")
+ }
+}
+
+func TestUnknownToolIsAnError(t *testing.T) {
+ s := &scripted{resps: []*Response{toolResp("nope", `{}`, 0), toolResp("finish", `{"summary":"x","changes":[]}`, 0)}}
+ run := Run(context.Background(), cfg(s))
+ if run.Calls[0].OK || !strings.Contains(run.Calls[0].Output, "no tool named") {
+ t.Fatalf("%+v", run.Calls[0])
+ }
+}
+
+func TestClientRefusesRemote(t *testing.T) {
+ c := &Client{BaseURL: "https://openrouter.ai/api/v1", Key: "sk-or-test"}
+ if _, err := c.Chat(context.Background(), &Request{Model: "x"}); !errors.Is(err, ErrNotAuthorized) {
+ t.Fatalf("a remote call under go test = %v", err)
+ }
+}
+
+func TestClientWireFormat(t *testing.T) {
+ var got map[string]any
+ var hdr http.Header
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ hdr = r.Header
+ _ = json.NewDecoder(r.Body).Decode(&got)
+ w.Write([]byte(`{"id":"g1","provider":"Anthropic","choices":[{"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":2,"cost":0.001,"prompt_tokens_details":{"cached_tokens":4}}}`))
+ }))
+ defer srv.Close()
+ c := &Client{BaseURL: srv.URL, Key: "k"}
+ allow := true
+ resp, err := c.Chat(context.Background(), &Request{Model: "anthropic/claude-opus-5.5", Messages: applyCaching("anthropic", []Message{{Role: "system", Content: "s"}, {Role: "user", Content: "u"}}),
+ Provider: &ProviderPrefs{Order: []string{"anthropic"}, AllowFallbacks: &allow, RequireParameters: true}, Usage: map[string]any{"include": true}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if resp.Provider != "Anthropic" || resp.Usage.PromptTokensDetails.CachedTokens != 4 {
+ t.Fatalf("%+v", resp)
+ }
+ if hdr.Get("Authorization") != "Bearer k" || hdr.Get("X-Title") != "traceway-evals" || hdr.Get("HTTP-Referer") == "" {
+ t.Fatalf("headers %v", hdr)
+ }
+ prov := got["provider"].(map[string]any)
+ if prov["require_parameters"] != true || prov["order"].([]any)[0] != "anthropic" {
+ t.Fatalf("provider %v", prov)
+ }
+ sys := got["messages"].([]any)[0].(map[string]any)["content"].([]any)[0].(map[string]any)
+ if sys["cache_control"].(map[string]any)["type"] != "ephemeral" {
+ t.Fatalf("system %v", sys)
+ }
+}
+
+func TestPromptHelpers(t *testing.T) {
+ if got := StripSlash("/traceway perf GET /x\nslow"); got != "perf GET /x\nslow" {
+ t.Fatal(got)
+ }
+ if SlashSkill("/traceway debug x") != "traceway" || SlashSkill("why is it slow") != "" {
+ t.Fatal("SlashSkill")
+ }
+ p := SystemPrompt(PromptInput{SubmitTool: "finish", Skills: []tools.Skill{{Name: "traceway", Description: "d"}}})
+ if !strings.Contains(p, "- traceway: d") || !strings.Contains(p, "call finish") {
+ t.Fatal(p)
+ }
+}
diff --git a/cli/test/evals/internal/agent/caching.go b/cli/test/evals/internal/agent/caching.go
new file mode 100644
index 00000000..f997293a
--- /dev/null
+++ b/cli/test/evals/internal/agent/caching.go
@@ -0,0 +1,39 @@
+package agent
+
+// Prompt caching. Agent loops resend the whole history every turn, so caching
+// is most of the cost saving.
+//
+// - Anthropic models (caching: anthropic) get explicit cache_control
+// breakpoints: one on the system message (the generic prompt plus any
+// preloaded skill; tools sit before it in the cached prefix) and one on
+// the most recent message, so each turn reads all earlier history from
+// cache. Anthropic checks earlier block boundaries for hits on its own, so
+// a single moving breakpoint is enough.
+// - OpenAI models (caching: automatic) cache any stable prefix. The system
+// prompt therefore carries no timestamps and tools keep a fixed order.
+
+// applyCaching returns the messages to send for a caching mode, without
+// mutating the conversation itself.
+func applyCaching(mode string, msgs []Message) []Message {
+ if mode != "anthropic" || len(msgs) == 0 {
+ return msgs
+ }
+ out := make([]Message, len(msgs))
+ copy(out, msgs)
+ mark := func(i int) {
+ m := out[i]
+ text, ok := m.Content.(string)
+ if !ok || text == "" {
+ return
+ }
+ m.Content = []Part{{Type: "text", Text: text, CacheControl: &CacheControl{Type: "ephemeral"}}}
+ out[i] = m
+ }
+ if out[0].Role == "system" {
+ mark(0)
+ }
+ if last := len(out) - 1; last > 0 && (out[last].Role == "tool" || out[last].Role == "user") {
+ mark(last)
+ }
+ return out
+}
diff --git a/cli/test/evals/internal/agent/fake.go b/cli/test/evals/internal/agent/fake.go
new file mode 100644
index 00000000..85556549
--- /dev/null
+++ b/cli/test/evals/internal/agent/fake.go
@@ -0,0 +1,313 @@
+package agent
+
+import (
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "regexp"
+ "sort"
+ "strconv"
+ "strings"
+ "sync"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+)
+
+// Fake is an httptest server speaking Chat Completions that plays a case's
+// oracle steps, one tool call per turn. It is the only model `go test` can
+// reach.
+type Fake struct {
+ Steps []catalog.Step
+ // FileRoot resolves write_file `from:` paths.
+ FileRoot string
+
+ srv *httptest.Server
+ mu sync.Mutex
+ errs []string
+}
+
+// NewFake starts the fake model.
+func NewFake(steps []catalog.Step, fileRoot string) *Fake {
+ f := &Fake{Steps: steps, FileRoot: fileRoot}
+ f.srv = httptest.NewServer(http.HandlerFunc(f.serve))
+ return f
+}
+
+// Client returns a Chat Completions client pointed at the fake.
+func (f *Fake) Client() *Client { return &Client{BaseURL: f.srv.URL, Retries: 1} }
+
+// Close stops the server.
+func (f *Fake) Close() { f.srv.Close() }
+
+// Errors returns oracle rendering problems (a step naming a tool the arm
+// does not have, a template that resolved to nothing, ...).
+func (f *Fake) Errors() []string {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return append([]string{}, f.errs...)
+}
+
+func (f *Fake) fail(format string, a ...any) {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.errs = append(f.errs, fmt.Sprintf(format, a...))
+}
+
+func (f *Fake) serve(w http.ResponseWriter, r *http.Request) {
+ var req Request
+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+ http.Error(w, err.Error(), 400)
+ return
+ }
+ step := 0
+ for _, m := range req.Messages {
+ if m.Role == "assistant" {
+ step++
+ }
+ }
+ tools := map[string]bool{}
+ for _, t := range req.Tools {
+ tools[t.Function.Name] = true
+ }
+ msg := Message{Role: "assistant"}
+ if step >= len(f.Steps) {
+ msg.Content = "Done."
+ } else {
+ name, args, text, err := f.render(f.Steps[step], tools, req.Messages)
+ if err != nil {
+ f.fail("step %d: %v", step+1, err)
+ msg.Content = "oracle error: " + err.Error()
+ } else if name == "" {
+ msg.Content = text
+ } else {
+ raw, _ := json.Marshal(args)
+ msg.ToolCalls = []ToolCall{{ID: fmt.Sprintf("fake_%d", step+1), Type: "function", Function: FunctionCall{Name: name, Arguments: string(raw)}}}
+ }
+ }
+ resp := map[string]any{
+ "id": fmt.Sprintf("fake-%d", step+1), "model": req.Model, "provider": "fake",
+ "choices": []any{map[string]any{"message": msg, "finish_reason": "tool_calls"}},
+ "usage": map[string]any{"prompt_tokens": 0, "completion_tokens": 0, "cost": 0},
+ }
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(resp)
+}
+
+func (f *Fake) render(s catalog.Step, tools map[string]bool, msgs []Message) (string, map[string]any, string, error) {
+ tmpl := func(v string) (string, error) { return resolveRuntime(v, msgs) }
+ switch {
+ case s.Say != "":
+ return "", nil, s.Say, nil
+ case s.Shell != "":
+ cmd, err := tmpl(s.Shell)
+ if err != nil {
+ return "", nil, "", err
+ }
+ return "shell", map[string]any{"command": cmd}, "", nil
+ case s.ReadFile != "":
+ return need(tools, "read_file", map[string]any{"path": s.ReadFile})
+ case s.ListFiles != "":
+ return need(tools, "list_files", map[string]any{"path": s.ListFiles})
+ case s.Search != nil:
+ return need(tools, "search", s.Search)
+ case s.WriteFile != nil:
+ content := s.WriteFile.Content
+ if s.WriteFile.From != "" {
+ raw, err := os.ReadFile(f.FileRoot + "/" + s.WriteFile.From)
+ if err != nil {
+ return "", nil, "", err
+ }
+ content = string(raw)
+ }
+ return need(tools, "write_file", map[string]any{"path": s.WriteFile.Path, "content": content})
+ case s.EditFile != nil:
+ return need(tools, "edit_file", map[string]any{"path": s.EditFile.Path, "old_string": s.EditFile.Old, "new_string": s.EditFile.New})
+ case s.LoadSkill != "":
+ if !tools["load_skill"] {
+ // Arms without skills play the step as a harmless look at the
+ // repository, so later call ids stay the same on every arm.
+ return need(tools, "list_files", map[string]any{"path": "."})
+ }
+ return need(tools, "load_skill", map[string]any{"name": s.LoadSkill})
+ case s.AskUser != "":
+ q, err := tmpl(s.AskUser)
+ if err != nil {
+ return "", nil, "", err
+ }
+ return need(tools, "ask_user", map[string]any{"question": q})
+ case s.Submit != nil:
+ for name := range tools {
+ if strings.HasPrefix(name, "submit_") {
+ args, err := resolveArgs(s.Submit, msgs)
+ return name, args, "", err
+ }
+ }
+ return "", nil, "", fmt.Errorf("no submit tool on this arm")
+ }
+ return "", nil, "", fmt.Errorf("empty oracle step")
+}
+
+func need(tools map[string]bool, name string, args map[string]any) (string, map[string]any, string, error) {
+ if !tools[name] {
+ return "", nil, "", fmt.Errorf("this arm has no %s tool", name)
+ }
+ return name, args, "", nil
+}
+
+var runtimeRe = regexp.MustCompile(`\{\{\s*((?:last|calls\.c\d+)(?:\.[^}\s]+)?)\s*\}\}`)
+
+// resolveRuntime fills {{last.x}} and {{calls.c03.x}} from the tool results
+// already in the conversation.
+func resolveRuntime(s string, msgs []Message) (string, error) {
+ var firstErr error
+ out := runtimeRe.ReplaceAllStringFunc(s, func(m string) string {
+ expr := runtimeRe.FindStringSubmatch(m)[1]
+ v, err := lookupRuntime(expr, msgs)
+ if err != nil && firstErr == nil {
+ firstErr = err
+ }
+ return fmt.Sprint(v)
+ })
+ return out, firstErr
+}
+
+func resolveArgs(in map[string]any, msgs []Message) (map[string]any, error) {
+ out := map[string]any{}
+ var firstErr error
+ var walk func(v any) any
+ walk = func(v any) any {
+ switch t := v.(type) {
+ case string:
+ // A whole-string template keeps the looked-up value's type.
+ if m := runtimeRe.FindStringSubmatch(t); m != nil && m[0] == strings.TrimSpace(t) {
+ val, err := lookupRuntime(m[1], msgs)
+ if err != nil && firstErr == nil {
+ firstErr = err
+ }
+ return val
+ }
+ s, err := resolveRuntime(t, msgs)
+ if err != nil && firstErr == nil {
+ firstErr = err
+ }
+ return s
+ case map[string]any:
+ m := map[string]any{}
+ for k, x := range t {
+ m[k] = walk(x)
+ }
+ return m
+ case []any:
+ l := make([]any, len(t))
+ for i, x := range t {
+ l[i] = walk(x)
+ }
+ return l
+ default:
+ return v
+ }
+ }
+ keys := make([]string, 0, len(in))
+ for k := range in {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ for _, k := range keys {
+ out[k] = walk(in[k])
+ }
+ return out, firstErr
+}
+
+func lookupRuntime(expr string, msgs []Message) (any, error) {
+ parts := strings.SplitN(expr, ".", 3)
+ var content string
+ var path string
+ switch parts[0] {
+ case "last":
+ for i := len(msgs) - 1; i >= 0; i-- {
+ if msgs[i].Role == "tool" {
+ content, _ = msgs[i].Content.(string)
+ break
+ }
+ }
+ if len(parts) > 1 {
+ path = strings.Join(parts[1:], ".")
+ }
+ case "calls":
+ if len(parts) < 2 {
+ return nil, fmt.Errorf("bad template %s", expr)
+ }
+ prefix := "[call " + parts[1] + "]"
+ for _, m := range msgs {
+ if s, ok := m.Content.(string); ok && m.Role == "tool" && strings.HasPrefix(s, prefix) {
+ content = s
+ }
+ }
+ if len(parts) > 2 {
+ path = parts[2]
+ }
+ }
+ if content == "" {
+ return nil, fmt.Errorf("template %s: no such tool result", expr)
+ }
+ body := stripToolFrame(content)
+ if path == "" {
+ return body, nil
+ }
+ var v any
+ if err := json.Unmarshal([]byte(firstJSON(body)), &v); err != nil {
+ return nil, fmt.Errorf("template %s: result is not JSON", expr)
+ }
+ for _, seg := range splitPath(path) {
+ switch t := v.(type) {
+ case map[string]any:
+ v = t[seg]
+ case []any:
+ i, err := strconv.Atoi(seg)
+ if err != nil || i < 0 || i >= len(t) {
+ return nil, fmt.Errorf("template %s: bad index %s", expr, seg)
+ }
+ v = t[i]
+ default:
+ return nil, fmt.Errorf("template %s: cannot index %T with %s", expr, v, seg)
+ }
+ }
+ if v == nil {
+ return nil, fmt.Errorf("template %s resolved to nothing", expr)
+ }
+ return v, nil
+}
+
+func splitPath(p string) []string {
+ p = strings.NewReplacer("[", ".", "]", "").Replace(p)
+ var out []string
+ for _, s := range strings.Split(p, ".") {
+ if s != "" {
+ out = append(out, s)
+ }
+ }
+ return out
+}
+
+func stripToolFrame(s string) string {
+ if i := strings.Index(s, "\n"); i >= 0 && strings.HasPrefix(s, "[call ") {
+ s = s[i+1:]
+ }
+ if i := strings.LastIndex(s, "\n[budget:"); i >= 0 {
+ s = s[:i]
+ }
+ return s
+}
+
+func firstJSON(s string) string {
+ s = strings.TrimSpace(s)
+ if i := strings.IndexAny(s, "{["); i > 0 {
+ s = s[i:]
+ }
+ if i := strings.Index(s, "\n[exit status"); i >= 0 {
+ s = s[:i]
+ }
+ return s
+}
diff --git a/cli/test/evals/internal/agent/loop.go b/cli/test/evals/internal/agent/loop.go
new file mode 100644
index 00000000..9cff574c
--- /dev/null
+++ b/cli/test/evals/internal/agent/loop.go
@@ -0,0 +1,213 @@
+package agent
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "sort"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"
+)
+
+// Config is one agent run.
+type Config struct {
+ Model *catalog.Model
+ Chat Chatter
+ Tools []tools.Tool
+ System string
+ Prompt string
+ SubmitTool string
+ Budget catalog.Budget
+ Transcript *Transcript
+ // AfterCall runs harness hooks after each executed call.
+ AfterCall func(ctx context.Context, c *record.Call)
+ // Questions returns the ask_user exchanges at the end.
+ Questions func() []record.Question
+ // SkillLoaded returns the skills loaded during the run.
+ SkillLoaded func() []string
+}
+
+// Stop reasons.
+const (
+ StopSubmitted = "submitted"
+ StopBudget = "budget_exhausted"
+ StopNoSubmit = "no_submit"
+ StopInfra = "infra_error"
+)
+
+// Run drives the model until it submits, a budget runs out, or it stops
+// without submitting (after one nudge).
+func Run(ctx context.Context, cfg Config) *record.Run {
+ budget := cfg.Budget.Defaults()
+ ctx, cancel := context.WithTimeout(ctx, budget.Timeout.D())
+ defer cancel()
+
+ run := &record.Run{Prompt: cfg.Prompt, Started: time.Now().UTC()}
+ defer func() { run.Elapsed = time.Since(run.Started) }()
+
+ byName := map[string]tools.Tool{}
+ defs := make([]ToolDef, 0, len(cfg.Tools))
+ for _, t := range cfg.Tools {
+ s := t.Spec()
+ byName[s.Name] = t
+ defs = append(defs, ToolDef{Type: "function", Function: FunctionDef{Name: s.Name, Description: s.Description, Parameters: s.Parameters}})
+ }
+ sort.SliceStable(defs, func(i, j int) bool { return defs[i].Function.Name < defs[j].Function.Name })
+
+ msgs := []Message{{Role: "system", Content: cfg.System}, {Role: "user", Content: cfg.Prompt}}
+ cfg.Transcript.Event("start", map[string]any{"model": cfg.Model.ID, "system": cfg.System, "prompt": cfg.Prompt, "tools": defs})
+
+ nudged := false
+ callN := 0
+ outputTokens := 0
+ providers := map[string]bool{}
+ finish := func(reason, detail string) *record.Run {
+ run.StopReason, run.StopDetail = reason, detail
+ if cfg.Questions != nil {
+ run.Questions = cfg.Questions()
+ }
+ if cfg.SkillLoaded != nil {
+ run.SkillLoaded = cfg.SkillLoaded()
+ }
+ for p := range providers {
+ run.Providers = append(run.Providers, p)
+ }
+ sort.Strings(run.Providers)
+ cfg.Transcript.Event("stop", map[string]any{"reason": reason, "detail": detail, "usage": run.Usage, "turns": run.Turns})
+ return run
+ }
+
+ for {
+ if run.Turns >= budget.MaxTurns {
+ return finish(StopBudget, fmt.Sprintf("max_turns %d reached", budget.MaxTurns))
+ }
+ run.Turns++
+ req := &Request{
+ Model: cfg.Model.ID,
+ Messages: applyCaching(cfg.Model.Caching, msgs),
+ Tools: defs,
+ ToolChoice: "auto",
+ MaxTokens: cfg.Model.MaxTokensPerTurn,
+ Usage: map[string]any{"include": true},
+ }
+ if cfg.Model.Temperature != nil {
+ req.Temperature = cfg.Model.Temperature
+ }
+ if len(cfg.Model.Provider.Order) > 0 || cfg.Model.Provider.AllowFallbacks != nil {
+ req.Provider = &ProviderPrefs{Order: cfg.Model.Provider.Order, AllowFallbacks: cfg.Model.Provider.AllowFallbacks, RequireParameters: true}
+ }
+ if cfg.Model.ReasoningEffort != "" {
+ req.Reasoning = map[string]any{"effort": cfg.Model.ReasoningEffort}
+ }
+ resp, err := cfg.Chat.Chat(ctx, req)
+ if err != nil {
+ if errors.Is(err, context.DeadlineExceeded) {
+ return finish(StopBudget, "wall-clock timeout")
+ }
+ return finish(StopInfra, err.Error())
+ }
+ u := usageOf(resp)
+ run.Usage.Add(u)
+ outputTokens += u.OutputTokens - u.ReasoningTokens
+ if resp.Provider != "" {
+ providers[resp.Provider] = true
+ }
+ msg := resp.Choices[0].Message
+ msg.Role = "assistant"
+ cfg.Transcript.Event("response", map[string]any{"turn": run.Turns, "id": resp.ID, "provider": resp.Provider, "model": resp.Model,
+ "finish_reason": resp.Choices[0].FinishReason, "content": msg.Content, "tool_calls": msg.ToolCalls, "usage": u})
+ if text, ok := msg.Content.(string); ok && strings.TrimSpace(text) != "" {
+ run.FinalText = text
+ }
+ // The assistant message, tool calls included, must precede the
+ // tool results or Anthropic models reject the history.
+ msgs = append(msgs, msg)
+
+ if len(msg.ToolCalls) == 0 {
+ if nudged {
+ return finish(StopNoSubmit, "stopped without calling "+cfg.SubmitTool)
+ }
+ nudged = true
+ msgs = append(msgs, Message{Role: "user", Content: fmt.Sprintf("You have not finished: call %s with your final answer now.", cfg.SubmitTool)})
+ continue
+ }
+ if outputTokens > budget.MaxOutputTokens {
+ return finish(StopBudget, fmt.Sprintf("max_output_tokens %d exceeded", budget.MaxOutputTokens))
+ }
+
+ final := false
+ for _, tc := range msg.ToolCalls {
+ callN++
+ id := fmt.Sprintf("c%02d", callN)
+ c := &record.Call{ID: id, Turn: run.Turns, Tool: tc.Function.Name, Args: json.RawMessage(nonEmpty(tc.Function.Arguments))}
+ var res tools.Result
+ switch {
+ case final:
+ res = tools.Result{Output: "not executed: the task was already submitted"}
+ c.Rejected = true
+ case callN > budget.MaxToolCalls && !isSubmit(tc.Function.Name):
+ res = tools.Result{Output: fmt.Sprintf("not executed: the tool-call budget (%d) is used up. Call %s now.", budget.MaxToolCalls, cfg.SubmitTool)}
+ c.Rejected = true
+ default:
+ t, ok := byName[tc.Function.Name]
+ if !ok {
+ res = tools.Result{Output: fmt.Sprintf("error: there is no tool named %q", tc.Function.Name)}
+ } else {
+ started := time.Now()
+ res = t.Call(ctx, c.Args)
+ c.DurationMs = time.Since(started).Milliseconds()
+ }
+ }
+ c.Output, c.OK, c.Truncated, c.Ops = res.Output, res.OK, res.Truncated, res.Ops
+ run.Calls = append(run.Calls, c)
+ cfg.Transcript.Event("tool", map[string]any{"call": c})
+ if !c.Rejected && cfg.AfterCall != nil {
+ cfg.AfterCall(ctx, c)
+ }
+ if res.Final {
+ final = true
+ run.Submission = res.Submission
+ run.SubmitTool = tc.Function.Name
+ run.SubmitCall = id
+ }
+ content := fmt.Sprintf("[call %s]\n%s", id, res.Output)
+ if !res.Final {
+ content += fmt.Sprintf("\n[budget: %d tool calls and %d turns left]", max(0, budget.MaxToolCalls-callN), max(0, budget.MaxTurns-run.Turns))
+ }
+ msgs = append(msgs, Message{Role: "tool", ToolCallID: tc.ID, Content: content})
+ }
+ if final {
+ return finish(StopSubmitted, "")
+ }
+ if ctx.Err() != nil {
+ return finish(StopBudget, "wall-clock timeout")
+ }
+ }
+}
+
+func isSubmit(name string) bool {
+ return name == tools.SubmitPerf || name == tools.SubmitDebug || name == tools.Finish
+}
+
+func nonEmpty(s string) string {
+ if strings.TrimSpace(s) == "" {
+ return "{}"
+ }
+ return s
+}
+
+func usageOf(r *Response) record.Usage {
+ return record.Usage{
+ InputTokens: r.Usage.PromptTokens,
+ CachedTokens: r.Usage.PromptTokensDetails.CachedTokens,
+ CacheWriteTokens: r.Usage.PromptTokensDetails.CacheWriteTokens,
+ OutputTokens: r.Usage.CompletionTokens,
+ ReasoningTokens: r.Usage.CompletionTokensDetails.ReasoningTokens,
+ CostUSD: r.Usage.Cost,
+ }
+}
diff --git a/cli/test/evals/internal/agent/openrouter.go b/cli/test/evals/internal/agent/openrouter.go
new file mode 100644
index 00000000..042d375e
--- /dev/null
+++ b/cli/test/evals/internal/agent/openrouter.go
@@ -0,0 +1,247 @@
+// Package agent drives a model through the eval: a Chat Completions client,
+// the tool-calling loop with budgets, prompt caching, transcripts, and the
+// fake model that plays oracle trajectories.
+package agent
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "net/url"
+ "strings"
+ "time"
+)
+
+// Message is one Chat Completions message.
+type Message struct {
+ Role string `json:"role"`
+ Content any `json:"content,omitempty"` // string or []Part
+ ToolCalls []ToolCall `json:"tool_calls,omitempty"`
+ ToolCallID string `json:"tool_call_id,omitempty"`
+ // ReasoningDetails must be sent back unchanged on the assistant message
+ // so reasoning models keep their chain across tool calls.
+ ReasoningDetails json.RawMessage `json:"reasoning_details,omitempty"`
+}
+
+// Part is one content part.
+type Part struct {
+ Type string `json:"type"`
+ Text string `json:"text"`
+ CacheControl *CacheControl `json:"cache_control,omitempty"`
+}
+
+// CacheControl marks an Anthropic prompt-cache breakpoint.
+type CacheControl struct {
+ Type string `json:"type"`
+}
+
+// ToolCall is an assistant tool call.
+type ToolCall struct {
+ ID string `json:"id"`
+ Type string `json:"type"`
+ Function FunctionCall `json:"function"`
+}
+
+// FunctionCall is the called function and its JSON arguments.
+type FunctionCall struct {
+ Name string `json:"name"`
+ Arguments string `json:"arguments"`
+}
+
+// ToolDef is a function tool definition.
+type ToolDef struct {
+ Type string `json:"type"`
+ Function FunctionDef `json:"function"`
+}
+
+// FunctionDef describes a function.
+type FunctionDef struct {
+ Name string `json:"name"`
+ Description string `json:"description,omitempty"`
+ Parameters map[string]any `json:"parameters"`
+}
+
+// ProviderPrefs is OpenRouter's provider routing.
+type ProviderPrefs struct {
+ Order []string `json:"order,omitempty"`
+ AllowFallbacks *bool `json:"allow_fallbacks,omitempty"`
+ RequireParameters bool `json:"require_parameters,omitempty"`
+}
+
+// Request is a Chat Completions request.
+type Request struct {
+ Model string `json:"model"`
+ Messages []Message `json:"messages"`
+ Tools []ToolDef `json:"tools,omitempty"`
+ ToolChoice any `json:"tool_choice,omitempty"`
+ MaxTokens int `json:"max_tokens,omitempty"`
+ Temperature *float64 `json:"temperature,omitempty"`
+ Provider *ProviderPrefs `json:"provider,omitempty"`
+ Reasoning map[string]any `json:"reasoning,omitempty"`
+ ResponseFormat map[string]any `json:"response_format,omitempty"`
+ Usage map[string]any `json:"usage,omitempty"`
+ ParallelTools *bool `json:"parallel_tool_calls,omitempty"`
+}
+
+// Response is a Chat Completions response.
+type Response struct {
+ ID string `json:"id"`
+ Model string `json:"model"`
+ Provider string `json:"provider"`
+ Choices []struct {
+ Message Message `json:"message"`
+ FinishReason string `json:"finish_reason"`
+ } `json:"choices"`
+ Usage struct {
+ PromptTokens int `json:"prompt_tokens"`
+ CompletionTokens int `json:"completion_tokens"`
+ Cost float64 `json:"cost"`
+ PromptTokensDetails struct {
+ CachedTokens int `json:"cached_tokens"`
+ CacheWriteTokens int `json:"cache_write_tokens"`
+ } `json:"prompt_tokens_details"`
+ CompletionTokensDetails struct {
+ ReasoningTokens int `json:"reasoning_tokens"`
+ } `json:"completion_tokens_details"`
+ } `json:"usage"`
+ Error *struct {
+ Message string `json:"message"`
+ Code any `json:"code"`
+ } `json:"error,omitempty"`
+}
+
+// Chatter sends one Chat Completions request.
+type Chatter interface {
+ Chat(ctx context.Context, req *Request) (*Response, error)
+}
+
+// Client talks Chat Completions to the fake model on loopback. A
+// non-loopback base URL is always refused: this harness makes no paid calls.
+type Client struct {
+ BaseURL string
+ Key string
+ HTTP *http.Client
+ // Retries on 429/5xx/timeouts (default 5).
+ Retries int
+}
+
+// ErrNotAuthorized is returned for a request to a remote endpoint.
+var ErrNotAuthorized = errors.New("remote model calls are not supported: the harness only talks to the fake model on loopback")
+
+func isLoopback(base string) bool {
+ u, err := url.Parse(base)
+ if err != nil {
+ return false
+ }
+ host := u.Hostname()
+ if host == "localhost" {
+ return true
+ }
+ ip := net.ParseIP(host)
+ return ip != nil && ip.IsLoopback()
+}
+
+// RetryableError marks an error worth retrying.
+type RetryableError struct{ Err error }
+
+func (e *RetryableError) Error() string { return e.Err.Error() }
+func (e *RetryableError) Unwrap() error { return e.Err }
+
+// Chat sends a request with retries and exponential backoff.
+func (c *Client) Chat(ctx context.Context, req *Request) (*Response, error) {
+ if !isLoopback(c.BaseURL) {
+ return nil, ErrNotAuthorized
+ }
+ retries := c.Retries
+ if retries == 0 {
+ retries = 5
+ }
+ var last error
+ for attempt := 0; attempt <= retries; attempt++ {
+ if attempt > 0 {
+ d := time.Duration(1<<uint(attempt-1)) * 2 * time.Second
+ select {
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ case <-time.After(d):
+ }
+ }
+ resp, err := c.once(ctx, req)
+ if err == nil {
+ return resp, nil
+ }
+ last = err
+ var re *RetryableError
+ if !errors.As(err, &re) {
+ return nil, err
+ }
+ }
+ return nil, fmt.Errorf("after %d retries: %w", retries, last)
+}
+
+func (c *Client) once(ctx context.Context, req *Request) (*Response, error) {
+ body, err := json.Marshal(req)
+ if err != nil {
+ return nil, err
+ }
+ hreq, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimSuffix(c.BaseURL, "/")+"/chat/completions", bytes.NewReader(body))
+ if err != nil {
+ return nil, err
+ }
+ hreq.Header.Set("Content-Type", "application/json")
+ if c.Key != "" {
+ hreq.Header.Set("Authorization", "Bearer "+c.Key)
+ }
+ hreq.Header.Set("HTTP-Referer", "https://github.com/tracewayapp/traceway")
+ hreq.Header.Set("X-Title", "traceway-evals")
+ hc := c.HTTP
+ if hc == nil {
+ hc = &http.Client{Timeout: 5 * time.Minute}
+ }
+ resp, err := hc.Do(hreq)
+ if err != nil {
+ if ctx.Err() != nil {
+ return nil, ctx.Err()
+ }
+ return nil, &RetryableError{err}
+ }
+ defer resp.Body.Close()
+ raw, err := io.ReadAll(resp.Body)
+ if err != nil {
+ return nil, &RetryableError{err}
+ }
+ if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 {
+ return nil, &RetryableError{fmt.Errorf("openrouter HTTP %d: %s", resp.StatusCode, clip(raw))}
+ }
+ if resp.StatusCode != http.StatusOK {
+ return nil, fmt.Errorf("openrouter HTTP %d: %s", resp.StatusCode, clip(raw))
+ }
+ var out Response
+ if err := json.Unmarshal(raw, &out); err != nil {
+ return nil, &RetryableError{fmt.Errorf("decode response: %w (%s)", err, clip(raw))}
+ }
+ if out.Error != nil {
+ code := fmt.Sprint(out.Error.Code)
+ if code == "429" || strings.HasPrefix(code, "5") {
+ return nil, &RetryableError{fmt.Errorf("openrouter error %s: %s", code, out.Error.Message)}
+ }
+ return nil, fmt.Errorf("openrouter error %s: %s", code, out.Error.Message)
+ }
+ if len(out.Choices) == 0 {
+ return nil, &RetryableError{fmt.Errorf("openrouter returned no choices: %s", clip(raw))}
+ }
+ return &out, nil
+}
+
+func clip(b []byte) string {
+ s := strings.TrimSpace(string(b))
+ if len(s) > 500 {
+ return s[:500] + "..."
+ }
+ return s
+}
diff --git a/cli/test/evals/internal/agent/prompt.go b/cli/test/evals/internal/agent/prompt.go
new file mode 100644
index 00000000..9618c320
--- /dev/null
+++ b/cli/test/evals/internal/agent/prompt.go
@@ -0,0 +1,76 @@
+package agent
+
+import (
+ "fmt"
+ "strings"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"
+)
+
+// PromptInput is what an arm contributes to the system prompt. Everything
+// else is identical across models and arms.
+type PromptInput struct {
+ SubmitTool string
+ // Skills lists installed skills (name: description), as agent harnesses
+ // advertise them.
+ Skills []tools.Skill
+ // Preloaded is the SKILL.md body of a slash-invoked skill.
+ Preloaded *tools.Skill
+ // Environment describes the machine (the CLI installed and logged in).
+ Environment string
+}
+
+// SystemPrompt builds the neutral coding-agent system prompt. It carries no
+// timestamps so OpenAI's automatic prefix cache keeps hitting.
+func SystemPrompt(in PromptInput) string {
+ var b strings.Builder
+ b.WriteString(`You are a software engineering agent working for a developer. You work in their repository, which is the current directory, and you act only through the tools provided. You cannot see their screen.
+
+Work like a careful senior engineer: gather evidence before you conclude, quantify what you claim, and do not change anything the user did not ask you to change. When you need a decision only the user can make, ask with ask_user.
+
+Every tool result starts with a line like [call c07]. When you cite evidence, cite those ids.
+`)
+ fmt.Fprintf(&b, "\nWhen you are done, call %s exactly once with your final answer. Do not stop without calling it.\n", in.SubmitTool)
+ if in.Environment != "" {
+ b.WriteString("\n# Environment\n\n" + in.Environment + "\n")
+ }
+ if len(in.Skills) > 0 {
+ b.WriteString("\n# Skills\n\nThese skills are installed. When a task matches a skill's description, load it with load_skill before you start and follow it. A skill's companion files are under .skills/<name>/ and can be read with read_file.\n\n")
+ for _, s := range in.Skills {
+ fmt.Fprintf(&b, "- %s: %s\n", s.Name, s.Description)
+ }
+ }
+ if in.Preloaded != nil {
+ fmt.Fprintf(&b, "\n# The user invoked /%s\n\nThe %s skill is loaded; its instructions follow. Its companion files are under .skills/%s/.\n\n", in.Preloaded.Name, in.Preloaded.Name, in.Preloaded.Name)
+ b.WriteString(in.Preloaded.Body)
+ b.WriteString("\n")
+ }
+ return b.String()
+}
+
+// StripSlash turns "/traceway perf X ..." into "perf X ..." for arms that do
+// not have the invoked skill, so every arm gets the same request.
+func StripSlash(prompt string) string {
+ p := strings.TrimLeft(prompt, " \n")
+ if !strings.HasPrefix(p, "/") {
+ return prompt
+ }
+ sp := strings.IndexAny(p, " \n")
+ if sp < 0 {
+ return ""
+ }
+ return strings.TrimLeft(p[sp:], " ")
+}
+
+// SlashSkill returns the skill name a prompt invokes ("/traceway ..."), or "".
+func SlashSkill(prompt string) string {
+ p := strings.TrimLeft(prompt, " \n")
+ if !strings.HasPrefix(p, "/") {
+ return ""
+ }
+ end := strings.IndexAny(p, " \n")
+ if end < 0 {
+ end = len(p)
+ }
+ return p[1:end]
+}
diff --git a/cli/test/evals/internal/agent/transcript.go b/cli/test/evals/internal/agent/transcript.go
new file mode 100644
index 00000000..a2093f99
--- /dev/null
+++ b/cli/test/evals/internal/agent/transcript.go
@@ -0,0 +1,51 @@
+package agent
+
+import (
+ "encoding/json"
+ "os"
+ "sync"
+ "time"
+)
+
+// Transcript appends one JSON event per line: the opening messages, every
+// model response (with usage, provider and generation id), every tool call
+// (id, op, argv/args, exit, ok, truncated, output) and the stop reason.
+type Transcript struct {
+ mu sync.Mutex
+ f *os.File
+}
+
+// OpenTranscript creates path.
+func OpenTranscript(path string) (*Transcript, error) {
+ f, err := os.Create(path)
+ if err != nil {
+ return nil, err
+ }
+ return &Transcript{f: f}, nil
+}
+
+// Event writes one event.
+func (t *Transcript) Event(kind string, fields map[string]any) {
+ if t == nil {
+ return
+ }
+ t.mu.Lock()
+ defer t.mu.Unlock()
+ fields["type"] = kind
+ fields["at"] = time.Now().UTC().Format(time.RFC3339Nano)
+ raw, err := json.Marshal(fields)
+ if err != nil {
+ return
+ }
+ t.f.Write(append(raw, '\n'))
+}
+
+// Close flushes and closes the file.
+func (t *Transcript) Close() error {
+ if t == nil {
+ return nil
+ }
+ t.mu.Lock()
+ defer t.mu.Unlock()
+ return t.f.Close()
+}
diff --git a/cli/test/evals/internal/catalog/case.go b/cli/test/evals/internal/catalog/case.go
new file mode 100644
index 00000000..d0ee8b24
--- /dev/null
+++ b/cli/test/evals/internal/catalog/case.go
@@ -0,0 +1,337 @@
+// Package catalog loads the eval cases (cases/<suite>/<id>.yaml), the presets
+// that select them, and the model configuration. It owns the case schema; every
+// other package reads cases through these types.
+package catalog
+
+import (
+ "fmt"
+ "os"
+ "path/filepath"
+ "regexp"
+ "slices"
+ "sort"
+ "strings"
+ "time"
+
+ "gopkg.in/yaml.v3"
+)
+
+// Suites, in report order.
+var Suites = []string{"perf"}
+
+// Arms. An arm fixes the surface the agent works through and what it is given.
+const (
+ ArmCLISkill = "cli+skill" // traceway CLI in the shell, traceway skill available
+ ArmCLINoSkill = "cli-noskill" // traceway CLI in the shell, no skills (baseline)
+)
+
+// AllArms lists every arm the runner knows.
+var AllArms = []string{ArmCLISkill, ArmCLINoSkill}
+
+// Case is one eval scenario. Strings in Prompt, Expect, Oracle and Hooks may
+// contain {{seed.x}} / {{env.x}} placeholders, resolved after seeding.
+type Case struct {
+ ID string `yaml:"id"`
+ Suite string `yaml:"suite"`
+ Title string `yaml:"title"`
+ Tags []string `yaml:"tags"`
+ Seed string `yaml:"seed"`
+ Arms []string `yaml:"arms"`
+ Invoke string `yaml:"invoke"` // slash | natural
+ Distractor string `yaml:"distractor"`
+
+ Workspace Workspace `yaml:"workspace"`
+ Prompt string `yaml:"prompt"`
+ Budget Budget `yaml:"budget"`
+ User []UserRule `yaml:"user"`
+ Expect Expect `yaml:"expect"`
+ Judge JudgeSpec `yaml:"judge"`
+ Oracle []Step `yaml:"oracle"`
+ OracleByArm map[string][]Step `yaml:"oracle_by_arm"`
+
+ // Path is the YAML file the case was loaded from.
+ Path string `yaml:"-"`
+}
+
+// Workspace names the fixture repo copied into the agent's work dir.
+type Workspace struct {
+ Repo string `yaml:"repo"`
+}
+
+// Budget bounds one run. Zero values take the defaults in Defaults().
+type Budget struct {
+ MaxTurns int `yaml:"max_turns"`
+ MaxToolCalls int `yaml:"max_tool_calls"`
+ MaxOutputTokens int `yaml:"max_output_tokens"`
+ Timeout Duration `yaml:"timeout"`
+}
+
+// Defaults fills unset budget fields.
+func (b Budget) Defaults() Budget {
+ if b.MaxTurns == 0 {
+ b.MaxTurns = 40
+ }
+ if b.MaxToolCalls == 0 {
+ b.MaxToolCalls = 30
+ }
+ if b.MaxOutputTokens == 0 {
+ b.MaxOutputTokens = 12000
+ }
+ if b.Timeout == 0 {
+ b.Timeout = Duration(10 * time.Minute)
+ }
+ return b
+}
+
+// UserRule scripts one ask_user answer. The first rule whose Match regexp
+// matches the question wins; a rule with Default set matches anything.
+type UserRule struct {
+ Match string `yaml:"match"`
+ Reply string `yaml:"reply"`
+ Default string `yaml:"default"`
+ // Once makes the rule apply to its first match only.
+ Once bool `yaml:"once"`
+}
+
+// Expect is everything the deterministic graders check.
+type Expect struct {
+ Answer *Answer `yaml:"answer"`
+ Grounding []string `yaml:"grounding"`
+ Calls *Trajectory `yaml:"calls"`
+ // Report holds regexps the final report text must (or must not) match.
+ Report *ReportSpec `yaml:"report"`
+}
+
+// ReportSpec checks the final free-text report.
+type ReportSpec struct {
+ Contains []string `yaml:"contains"`
+ NotContains []string `yaml:"not_contains"`
+}
+
+// Answer is the expected structured answer to submit_perf_diagnosis.
+type Answer struct {
+ ChecklistItem string `yaml:"checklist_item"`
+ AnyOf []string `yaml:"any_of"`
+ Not []string `yaml:"not"`
+ Locus *Locus `yaml:"locus"`
+ Onset *Onset `yaml:"onset"`
+ // MentionsAll lists regexps that must all match somewhere in the answer.
+ MentionsAll []string `yaml:"mentions_all"`
+}
+
+// Locus pins where the bottleneck is. Every set field must match.
+type Locus struct {
+ Endpoint string `yaml:"endpoint"`
+ SpanContains string `yaml:"span_contains"`
+ Service string `yaml:"service"`
+ Host string `yaml:"host"`
+}
+
+// Onset is the expected start of a regression.
+type Onset struct {
+ At string `yaml:"at"`
+ ToleranceMinutes int `yaml:"tolerance_minutes"`
+}
+
+// Trajectory is the tool-call DSL (plan §6.4).
+type Trajectory struct {
+ Required []CallMatcher `yaml:"required"`
+ AnyOf [][]CallMatcher `yaml:"any_of"`
+ Forbidden []CallMatcher `yaml:"forbidden"`
+ Order [][]string `yaml:"order"`
+ MaxCalls int `yaml:"max_calls"`
+ MaxFailedCalls int `yaml:"max_failed_calls"`
+ MaxIdenticalRepeats int `yaml:"max_identical_repeats"`
+}
+
+// CallMatcher matches one recorded operation.
+type CallMatcher struct {
+ Op string `yaml:"op"`
+ Args map[string]any `yaml:"args"`
+ Write *bool `yaml:"write"`
+ OK *bool `yaml:"ok"`
+}
+
+// JudgeSpec is the LLM judge's rubric. Expected is shown to the judge only.
+type JudgeSpec struct {
+ Rubric []string `yaml:"rubric"`
+ Expected string `yaml:"expected"`
+}
+
+// Step is one oracle action, played by the fake model.
+type Step struct {
+ Shell string `yaml:"shell"`
+ Stdin string `yaml:"stdin"`
+ ReadFile string `yaml:"read_file"`
+ ListFiles string `yaml:"list_files"`
+ Search map[string]any `yaml:"search"`
+ WriteFile *FileStep `yaml:"write_file"`
+ EditFile *EditStep `yaml:"edit_file"`
+ LoadSkill string `yaml:"load_skill"`
+ AskUser string `yaml:"ask_user"`
+ Submit map[string]any `yaml:"submit"`
+ Say string `yaml:"say"`
+}
+
+// FileStep writes a whole file.
+type FileStep struct {
+ Path string `yaml:"path"`
+ Content string `yaml:"content"`
+ // From copies the content of a testdata file instead.
+ From string `yaml:"from"`
+}
+
+// EditStep replaces one string in a file.
+type EditStep struct {
+ Path string `yaml:"path"`
+ Old string `yaml:"old"`
+ New string `yaml:"new"`
+}
+
+// Duration is a time.Duration that unmarshals from "10m".
+type Duration time.Duration
+
+// UnmarshalYAML parses Go duration strings.
+func (d *Duration) UnmarshalYAML(n *yaml.Node) error {
+ var s string
+ if err := n.Decode(&s); err != nil {
+ return err
+ }
+ v, err := time.ParseDuration(s)
+ if err != nil {
+ return fmt.Errorf("duration %q: %w", s, err)
+ }
+ *d = Duration(v)
+ return nil
+}
+
+// D returns the time.Duration.
+func (d Duration) D() time.Duration { return time.Duration(d) }
+
+var idRe = regexp.MustCompile(`^[a-z0-9]+(-[a-z0-9]+)+$`)
+
+// Validate checks a case for structural errors that YAML decoding cannot.
+func (c *Case) Validate() error {
+ var errs []string
+ add := func(f string, a ...any) { errs = append(errs, fmt.Sprintf(f, a...)) }
+ if !idRe.MatchString(c.ID) {
+ add("id %q must be kebab-case", c.ID)
+ }
+ if !slices.Contains(Suites, c.Suite) {
+ add("suite %q must be one of %v", c.Suite, Suites)
+ } else if !strings.HasPrefix(c.ID, c.Suite+"-") {
+ add("id %q must start with %q", c.ID, c.Suite+"-")
+ }
+ if c.Path != "" && filepath.Base(c.Path) != c.ID+".yaml" {
+ add("file %s must be named %s.yaml", c.Path, c.ID)
+ }
+ if c.Seed == "" {
+ add("seed is required")
+ }
+ if len(c.Arms) == 0 {
+ add("arms is required")
+ }
+ for _, a := range c.Arms {
+ if !slices.Contains(AllArms, a) {
+ add("unknown arm %q", a)
+ }
+ }
+ switch c.Invoke {
+ case "", "slash", "natural":
+ default:
+ add("invoke must be slash or natural, got %q", c.Invoke)
+ }
+ if strings.TrimSpace(c.Prompt) == "" {
+ add("prompt is required")
+ }
+ if len(c.Oracle) == 0 {
+ add("oracle is required")
+ }
+ for arm := range c.OracleByArm {
+ if !slices.Contains(c.Arms, arm) {
+ add("oracle_by_arm names arm %q that the case does not run", arm)
+ }
+ }
+ if c.Expect.Answer == nil {
+ add("%s cases need expect.answer", c.Suite)
+ }
+ if c.Distractor == "" && !slices.Contains(c.Tags, "control") {
+ add("distractor is required (or tag the case control)")
+ }
+ if len(errs) > 0 {
+ return fmt.Errorf("%s: %s", c.ID, strings.Join(errs, "; "))
+ }
+ return nil
+}
+
+// HasTag reports whether the case carries tag.
+func (c *Case) HasTag(tag string) bool { return slices.Contains(c.Tags, tag) }
+
+// OracleFor returns the oracle steps for an arm.
+func (c *Case) OracleFor(arm string) []Step {
+ if s, ok := c.OracleByArm[arm]; ok {
+ return s
+ }
+ return c.Oracle
+}
+
+// LoadCases reads every cases/<suite>/*.yaml under root. TWEVALS_ONLY (a
+// comma-separated list of ids) limits loading to those cases, so work in
+// progress on other cases cannot break a run.
+func LoadCases(root string) ([]*Case, error) {
+ var out []*Case
+ seen := map[string]string{}
+ only := map[string]bool{}
+ for _, id := range strings.Split(os.Getenv("TWEVALS_ONLY"), ",") {
+ if id = strings.TrimSpace(id); id != "" {
+ only[id] = true
+ }
+ }
+ for _, suite := range Suites {
+ files, err := filepath.Glob(filepath.Join(root, "cases", suite, "*.yaml"))
+ if err != nil {
+ return nil, err
+ }
+ sort.Strings(files)
+ for _, f := range files {
+ if len(only) > 0 && !only[strings.TrimSuffix(filepath.Base(f), ".yaml")] {
+ continue
+ }
+ c, err := LoadCase(f)
+ if err != nil {
+ return nil, err
+ }
+ if c.Suite != suite {
+ return nil, fmt.Errorf("%s: suite %q does not match directory %q", f, c.Suite, suite)
+ }
+ if prev, dup := seen[c.ID]; dup {
+ return nil, fmt.Errorf("duplicate case id %s in %s and %s", c.ID, prev, f)
+ }
+ seen[c.ID] = f
+ out = append(out, c)
+ }
+ }
+ return out, nil
+}
+
+// LoadCase reads and validates one case file. Unknown keys are errors.
+func LoadCase(path string) (*Case, error) {
+ raw, err := os.ReadFile(path)
+ if err != nil {
+ return nil, err
+ }
+ dec := yaml.NewDecoder(strings.NewReader(string(raw)))
+ dec.KnownFields(true)
+ var c Case
+ if err := dec.Decode(&c); err != nil {
+ return nil, fmt.Errorf("%s: %w", path, err)
+ }
+ c.Path = path
+ if c.Invoke == "" {
+ c.Invoke = "natural"
+ }
+ if err := c.Validate(); err != nil {
+ return nil, err
+ }
+ return &c, nil
+}
diff --git a/cli/test/evals/internal/catalog/models.go b/cli/test/evals/internal/catalog/models.go
new file mode 100644
index 00000000..a4f825a1
--- /dev/null
+++ b/cli/test/evals/internal/catalog/models.go
@@ -0,0 +1,159 @@
+package catalog
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "fmt"
+ "os"
+ "path/filepath"
+ "slices"
+ "sort"
+ "strings"
+
+ "gopkg.in/yaml.v3"
+)
+
+// Model is one OpenRouter model as configured in models.yaml.
+type Model struct {
+ // Key is the short name used on the command line (opus, astra, judge).
+ Key string `yaml:"-"`
+ // ID is the pinned OpenRouter model id. Never a floating alias.
+ ID string `yaml:"id"`
+ Label string `yaml:"label"`
+ Provider Provider `yaml:"provider"`
+ // Caching is "anthropic" (explicit cache_control breakpoints) or
+ // "automatic" (provider-side prefix caching, keep the prefix stable).
+ Caching string `yaml:"caching"`
+ // ReasoningEffort is sent as reasoning.effort when the model lists
+ // "reasoning" in its supported_parameters. Empty disables it.
+ ReasoningEffort string `yaml:"reasoning_effort"`
+ // MaxTokensPerTurn caps one completion.
+ MaxTokensPerTurn int `yaml:"max_tokens_per_turn"`
+ Temperature *float64 `yaml:"temperature"`
+ // Estimate is the per-run token envelope used by the preflight until
+ // baseline.json carries measured averages.
+ Estimate Estimate `yaml:"estimate"`
+ // Fallback prices (USD per million tokens) used when the live price list
+ // cannot be fetched. The preflight says when it fell back.
+ FallbackPrice Price `yaml:"fallback_price"`
+}
+
+// Provider is OpenRouter's provider routing block.
+type Provider struct {
+ Order []string `yaml:"order" json:"order,omitempty"`
+ AllowFallbacks *bool `yaml:"allow_fallbacks" json:"allow_fallbacks,omitempty"`
+}
+
+// Estimate is a per-run token envelope.
+type Estimate struct {
+ InputTokens [2]int `yaml:"input_tokens"`
+ CachedFraction float64 `yaml:"cached_fraction"`
+ OutputTokens [2]int `yaml:"output_tokens"`
+}
+
+// Price is USD per million tokens.
+type Price struct {
+ Prompt float64 `yaml:"prompt" json:"prompt"`
+ Completion float64 `yaml:"completion" json:"completion"`
+ CacheRead float64 `yaml:"cache_read" json:"cacheRead"`
+ CacheWrite float64 `yaml:"cache_write" json:"cacheWrite"`
+}
+
+// Models is models.yaml.
+type Models struct {
+ Agents map[string]*Model `yaml:"agents"`
+ Judge *Model `yaml:"judge"`
+ // Digest is the SHA-256 of the file, recorded in every results header.
+ Digest string `yaml:"-"`
+}
+
+// LoadModels reads models.yaml under root.
+func LoadModels(root string) (*Models, error) {
+ path := filepath.Join(root, "models.yaml")
+ raw, err := os.ReadFile(path)
+ if err != nil {
+ return nil, err
+ }
+ var m Models
+ dec := yaml.NewDecoder(strings.NewReader(string(raw)))
+ dec.KnownFields(true)
+ if err := dec.Decode(&m); err != nil {
+ return nil, fmt.Errorf("%s: %w", path, err)
+ }
+ sum := sha256.Sum256(raw)
+ m.Digest = hex.EncodeToString(sum[:])
+ for k, a := range m.Agents {
+ a.Key = k
+ }
+ if m.Judge != nil {
+ m.Judge.Key = "judge"
+ }
+ return &m, m.validate()
+}
+
+func (m *Models) validate() error {
+ all := []*Model{}
+ for _, a := range m.Agents {
+ all = append(all, a)
+ }
+ if m.Judge == nil {
+ return fmt.Errorf("models.yaml: judge is required")
+ }
+ all = append(all, m.Judge)
+ for _, a := range all {
+ if a.ID == "" || !strings.Contains(a.ID, "/") {
+ return fmt.Errorf("models.yaml: %s: id must be a provider/model OpenRouter id", a.Key)
+ }
+ if strings.HasPrefix(a.ID, "~") || strings.HasSuffix(a.ID, "latest") || strings.Contains(a.ID, ":") {
+ return fmt.Errorf("models.yaml: %s: %q is a floating alias or variant; pin an exact id", a.Key, a.ID)
+ }
+ switch a.Caching {
+ case "anthropic", "automatic", "none":
+ default:
+ return fmt.Errorf("models.yaml: %s: caching must be anthropic, automatic or none", a.Key)
+ }
+ }
+ return nil
+}
+
+// Agent returns the agent model by key.
+func (m *Models) Agent(key string) (*Model, error) {
+ a, ok := m.Agents[key]
+ if !ok {
+ return nil, fmt.Errorf("unknown model %q (models.yaml has %s)", key, strings.Join(m.AgentKeys(), ", "))
+ }
+ return a, nil
+}
+
+// AgentKeys returns the configured agent keys, sorted.
+func (m *Models) AgentKeys() []string {
+ keys := make([]string, 0, len(m.Agents))
+ for k := range m.Agents {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ return keys
+}
+
+// ParseModelList splits "opus,astra" and checks every key exists.
+func (m *Models) ParseModelList(s string) ([]*Model, error) {
+ var out []*Model
+ for _, k := range strings.Split(s, ",") {
+ k = strings.TrimSpace(k)
+ if k == "" {
+ continue
+ }
+ a, err := m.Agent(k)
+ if err != nil {
+ return nil, err
+ }
+ if slices.ContainsFunc(out, func(x *Model) bool { return x.Key == k }) {
+ continue
+ }
+ out = append(out, a)
+ }
+ if len(out) == 0 {
+ return nil, fmt.Errorf("no models selected")
+ }
+ return out, nil
+}
diff --git a/cli/test/evals/internal/catalog/select.go b/cli/test/evals/internal/catalog/select.go
new file mode 100644
index 00000000..aa414f83
--- /dev/null
+++ b/cli/test/evals/internal/catalog/select.go
@@ -0,0 +1,178 @@
+package catalog
+
+import (
+ "fmt"
+ "os"
+ "path"
+ "path/filepath"
+ "slices"
+ "strings"
+
+ "gopkg.in/yaml.v3"
+)
+
+// Preset selects cases and arms (presets.yaml).
+type Preset struct {
+ Description string `yaml:"description"`
+ Tags []string `yaml:"tags"`
+ // Arms is "primary" (each suite's primary arm) or "all" (every arm a
+ // case lists).
+ Arms string `yaml:"arms"`
+}
+
+// Presets is presets.yaml.
+type Presets struct {
+ Presets map[string]Preset `yaml:"presets"`
+ PrimaryArms map[string][]string `yaml:"primary_arms"`
+}
+
+// LoadPresets reads presets.yaml under root.
+func LoadPresets(root string) (*Presets, error) {
+ raw, err := os.ReadFile(filepath.Join(root, "presets.yaml"))
+ if err != nil {
+ return nil, err
+ }
+ var p Presets
+ dec := yaml.NewDecoder(strings.NewReader(string(raw)))
+ dec.KnownFields(true)
+ if err := dec.Decode(&p); err != nil {
+ return nil, fmt.Errorf("presets.yaml: %w", err)
+ }
+ for name, pr := range p.Presets {
+ if pr.Arms != "primary" && pr.Arms != "all" {
+ return nil, fmt.Errorf("presets.yaml: %s: arms must be primary or all", name)
+ }
+ }
+ for _, s := range Suites {
+ if len(p.PrimaryArms[s]) == 0 {
+ return nil, fmt.Errorf("presets.yaml: primary_arms.%s is missing", s)
+ }
+ }
+ return &p, nil
+}
+
+// Selection is the user's case/arm filter. Cases (when set) overrides Preset
+// for which cases run; Arms (when set) overrides the preset's arm policy.
+type Selection struct {
+ Preset string
+ Cases string // comma-separated ids or globs
+ Suites string // comma-separated
+ Tags string // comma-separated, all must match
+ Arms string // comma-separated
+ Shard string // one suite
+}
+
+// Unit is one (case, arm) pair; the runner multiplies it by models and runs.
+type Unit struct {
+ Case *Case
+ Arm string
+}
+
+func splitList(s string) []string {
+ var out []string
+ for _, p := range strings.Split(s, ",") {
+ if p = strings.TrimSpace(p); p != "" {
+ out = append(out, p)
+ }
+ }
+ return out
+}
+
+// Select applies a selection to the loaded cases.
+func Select(cases []*Case, presets *Presets, sel Selection) ([]Unit, error) {
+ var preset *Preset
+ if sel.Preset != "" {
+ p, ok := presets.Presets[sel.Preset]
+ if !ok {
+ return nil, fmt.Errorf("unknown preset %q", sel.Preset)
+ }
+ preset = &p
+ }
+ globs := splitList(sel.Cases)
+ suites := splitList(sel.Suites)
+ tags := splitList(sel.Tags)
+ arms := splitList(sel.Arms)
+ for _, a := range arms {
+ if !slices.Contains(AllArms, a) {
+ return nil, fmt.Errorf("unknown arm %q", a)
+ }
+ }
+ for _, s := range suites {
+ if !slices.Contains(Suites, s) {
+ return nil, fmt.Errorf("unknown suite %q", s)
+ }
+ }
+ if sel.Shard != "" && !slices.Contains(Suites, sel.Shard) {
+ return nil, fmt.Errorf("unknown shard %q (shards are suites)", sel.Shard)
+ }
+
+ var out []Unit
+ matchedGlob := map[string]bool{}
+ for _, c := range cases {
+ if len(globs) > 0 {
+ hit := false
+ for _, g := range globs {
+ if ok, _ := path.Match(g, c.ID); ok {
+ hit, matchedGlob[g] = true, true
+ }
+ }
+ if !hit {
+ continue
+ }
+ } else if preset != nil {
+ if !hasAll(c, preset.Tags) {
+ continue
+ }
+ }
+ if len(suites) > 0 && !slices.Contains(suites, c.Suite) {
+ continue
+ }
+ if sel.Shard != "" && c.Suite != sel.Shard {
+ continue
+ }
+ if !hasAll(c, tags) {
+ continue
+ }
+ var caseArms []string
+ switch {
+ case len(arms) > 0:
+ for _, a := range arms {
+ if slices.Contains(c.Arms, a) {
+ caseArms = append(caseArms, a)
+ }
+ }
+ case preset != nil && preset.Arms == "all":
+ caseArms = c.Arms
+ default:
+ for _, a := range presets.PrimaryArms[c.Suite] {
+ if slices.Contains(c.Arms, a) {
+ caseArms = append(caseArms, a)
+ }
+ }
+ if len(caseArms) == 0 {
+ caseArms = c.Arms[:1]
+ }
+ }
+ for _, a := range caseArms {
+ out = append(out, Unit{Case: c, Arm: a})
+ }
+ }
+ for _, g := range globs {
+ if !matchedGlob[g] {
+ return nil, fmt.Errorf("case %q matches nothing", g)
+ }
+ }
+ if len(out) == 0 {
+ return nil, fmt.Errorf("the selection is empty")
+ }
+ return out, nil
+}
+
+func hasAll(c *Case, tags []string) bool {
+ for _, t := range tags {
+ if !c.HasTag(t) {
+ return false
+ }
+ }
+ return true
+}
diff --git a/cli/test/evals/internal/catalog/template.go b/cli/test/evals/internal/catalog/template.go
new file mode 100644
index 00000000..645b500b
--- /dev/null
+++ b/cli/test/evals/internal/catalog/template.go
@@ -0,0 +1,143 @@
+package catalog
+
+import (
+ "encoding/json"
+ "fmt"
+ "os"
+ "regexp"
+ "strconv"
+ "strings"
+
+ "gopkg.in/yaml.v3"
+)
+
+// templateRe matches {{seed.x.y}}, {{env.url}}, {{now}}; {{last.x}} and
+// {{calls.cNN.x}} are runtime templates the fake model resolves, so they are
+// left in place here.
+var templateRe = regexp.MustCompile(`\{\{\s*([A-Za-z0-9_.\-\[\]]+)\s*\}\}`)
+
+func isRuntime(expr string) bool {
+ return expr == "last" || strings.HasPrefix(expr, "last.") || strings.HasPrefix(expr, "calls.")
+}
+
+// Lookup resolves a dotted path in vars ("seed.slowRequest.id").
+func Lookup(vars map[string]any, expr string) (any, bool) {
+ var cur any = vars
+ for _, seg := range strings.Split(strings.NewReplacer("[", ".", "]", "").Replace(expr), ".") {
+ if seg == "" {
+ continue
+ }
+ switch t := cur.(type) {
+ case map[string]any:
+ v, ok := t[seg]
+ if !ok {
+ return nil, false
+ }
+ cur = v
+ case []any:
+ i, err := strconv.Atoi(seg)
+ if err != nil || i < 0 || i >= len(t) {
+ return nil, false
+ }
+ cur = t[i]
+ default:
+ return nil, false
+ }
+ }
+ return cur, cur != nil
+}
+
+// ExpandString replaces templates in s. Unknown templates are an error;
+// runtime templates are kept.
+func ExpandString(s string, vars map[string]any) (string, error) {
+ var firstErr error
+ out := templateRe.ReplaceAllStringFunc(s, func(m string) string {
+ expr := templateRe.FindStringSubmatch(m)[1]
+ if isRuntime(expr) {
+ return m
+ }
+ v, ok := Lookup(vars, expr)
+ if !ok {
+ if firstErr == nil {
+ firstErr = fmt.Errorf("template {{%s}} has no value", expr)
+ }
+ return m
+ }
+ return scalarString(v)
+ })
+ return out, firstErr
+}
+
+func scalarString(v any) string {
+ switch t := v.(type) {
+ case string:
+ return t
+ case float64:
+ return strconv.FormatFloat(t, 'f', -1, 64)
+ case map[string]any, []any:
+ raw, _ := json.Marshal(t)
+ return string(raw)
+ default:
+ return fmt.Sprint(t)
+ }
+}
+
+// Resolve returns a copy of the case with every template filled from vars.
+// It re-reads the YAML and expands scalar nodes, so a scalar that is exactly
+// one template keeps the value's type (a number stays a number).
+func (c *Case) Resolve(vars map[string]any) (*Case, error) {
+ raw, err := os.ReadFile(c.Path)
+ if err != nil {
+ return nil, err
+ }
+ var doc yaml.Node
+ if err := yaml.Unmarshal(raw, &doc); err != nil {
+ return nil, err
+ }
+ var firstErr error
+ var walk func(n *yaml.Node)
+ walk = func(n *yaml.Node) {
+ if n.Kind == yaml.ScalarNode && strings.Contains(n.Value, "{{") {
+ trimmed := strings.TrimSpace(n.Value)
+ if m := templateRe.FindStringSubmatch(trimmed); m != nil && m[0] == trimmed && !isRuntime(m[1]) {
+ if v, ok := Lookup(vars, m[1]); ok {
+ switch t := v.(type) {
+ case float64:
+ n.Value, n.Tag, n.Style = strconv.FormatFloat(t, 'f', -1, 64), "!!float", 0
+ case int:
+ n.Value, n.Tag, n.Style = strconv.Itoa(t), "!!int", 0
+ case bool:
+ n.Value, n.Tag, n.Style = strconv.FormatBool(t), "!!bool", 0
+ default:
+ n.Value, n.Tag = scalarString(v), "!!str"
+ }
+ return
+ }
+ }
+ s, err := ExpandString(n.Value, vars)
+ if err != nil && firstErr == nil {
+ firstErr = fmt.Errorf("%s: %w", c.ID, err)
+ }
+ n.Value = s
+ if n.Tag != "!!str" && strings.Contains(s, "{{") {
+ n.Tag = "!!str"
+ }
+ }
+ for _, ch := range n.Content {
+ walk(ch)
+ }
+ }
+ walk(&doc)
+ if firstErr != nil {
+ return nil, firstErr
+ }
+ var out Case
+ if err := doc.Decode(&out); err != nil {
+ return nil, fmt.Errorf("%s: after templating: %w", c.ID, err)
+ }
+ out.Path = c.Path
+ if out.Invoke == "" {
+ out.Invoke = "natural"
+ }
+ return &out, nil
+}
diff --git a/cli/test/evals/internal/checklist/checklist.go b/cli/test/evals/internal/checklist/checklist.go
new file mode 100644
index 00000000..7b40c83a
--- /dev/null
+++ b/cli/test/evals/internal/checklist/checklist.go
@@ -0,0 +1,104 @@
+// Package checklist turns the bottleneck checklist in performance.md into the
+// enum submit_perf_diagnosis takes.
+package checklist
+
+import (
+ "regexp"
+ "strings"
+ "sync"
+
+ "github.com/tracewayapp/traceway/cli/pkg/mcpserver/knowledge"
+)
+
+// Row is one row of the bottleneck checklist in performance.md.
+type Row struct {
+ ID string // slug, the enum value submit_perf_diagnosis takes
+ Title string // row title as written, markdown stripped
+ Category string // the ### heading above the table
+ Confirm string // the "How to confirm in Traceway" cell
+}
+
+// Extra answer values beyond the checklist rows.
+const (
+ NoneFound = "none_found"
+ InsufficientEvidence = "insufficient_evidence"
+)
+
+var (
+ checklistOnce sync.Once
+ checklist []Row
+)
+
+// Checklist parses the canonical performance.md once. The enum therefore
+// tracks the skill: renaming a row renames its id, and the case test fails.
+func Rows() []Row {
+ checklistOnce.Do(func() { checklist = ParseChecklist(knowledge.MustRead("performance.md")) })
+ return checklist
+}
+
+var (
+ rowRe = regexp.MustCompile(`^\|\s*\*\*(.+?)\*\*\s*\|\s*(.+?)\s*\|\s*$`)
+ parenRe = regexp.MustCompile(`\s*\([^)]*\s[^)]*\)`)
+ nonAlnumRe = regexp.MustCompile(`[^a-z0-9]+`)
+ backtickRe = regexp.MustCompile("`")
+ headingRe = regexp.MustCompile(`^###\s+(.+?)\s*$`)
+ checklistTop = "## Bottleneck checklist"
+)
+
+// ParseChecklist extracts the checklist rows from a performance.md body.
+func ParseChecklist(md string) []Row {
+ var rows []Row
+ in := false
+ category := ""
+ for _, line := range strings.Split(md, "\n") {
+ if strings.HasPrefix(line, "## ") {
+ in = strings.HasPrefix(line, checklistTop)
+ continue
+ }
+ if !in {
+ continue
+ }
+ if m := headingRe.FindStringSubmatch(line); m != nil {
+ category = m[1]
+ continue
+ }
+ m := rowRe.FindStringSubmatch(line)
+ if m == nil {
+ continue
+ }
+ title := backtickRe.ReplaceAllString(m[1], "")
+ rows = append(rows, Row{ID: Slug(title), Title: title, Category: category, Confirm: m[2]})
+ }
+ return rows
+}
+
+// Slug turns a checklist title into its id: explanatory parentheticals (ones
+// with a space inside) dropped, "+" spelled out, "^" dropped, everything else
+// non-alphanumeric collapsed to "_".
+func Slug(title string) string {
+ s := parenRe.ReplaceAllString(title, "")
+ s = strings.ToLower(s)
+ s = strings.ReplaceAll(s, "+", " plus ")
+ s = strings.ReplaceAll(s, "^", "")
+ s = nonAlnumRe.ReplaceAllString(s, "_")
+ return strings.Trim(s, "_")
+}
+
+// ChecklistIDs returns every valid checklist_item value, rows first.
+func IDs() []string {
+ var ids []string
+ for _, r := range Rows() {
+ ids = append(ids, r.ID)
+ }
+ return append(ids, NoneFound, InsufficientEvidence)
+}
+
+// IsChecklistID reports whether id is a valid checklist_item.
+func IsID(id string) bool {
+ for _, v := range IDs() {
+ if v == id {
+ return true
+ }
+ }
+ return false
+}
diff --git a/cli/test/evals/internal/env/api.go b/cli/test/evals/internal/env/api.go
new file mode 100644
index 00000000..6cdd24e6
--- /dev/null
+++ b/cli/test/evals/internal/env/api.go
@@ -0,0 +1,125 @@
+// Package env runs one hermetic Traceway instance per eval run: the
+// standalone backend binary in SQLite mode in a temp dir, plus the users,
+// projects, tokens and CLI logins a case needs. It talks HTTP only and never
+// imports the backend.
+package env
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "net/url"
+ "strings"
+ "time"
+)
+
+// API is a small JSON client for the dashboard API.
+type API struct {
+ Base string // http://127.0.0.1:<port>, no /api suffix
+ Bearer string
+ HTTP *http.Client
+}
+
+// Response is a decoded API reply.
+type Response struct {
+ Status int
+ Body []byte
+}
+
+// JSON decodes the body into v.
+func (r *Response) JSON(v any) error {
+ if err := json.Unmarshal(r.Body, v); err != nil {
+ return fmt.Errorf("decode %d response: %w (body: %s)", r.Status, err, clip(r.Body))
+ }
+ return nil
+}
+
+// Map decodes the body as an object.
+func (r *Response) Map() map[string]any {
+ var m map[string]any
+ _ = json.Unmarshal(r.Body, &m)
+ return m
+}
+
+// OK reports a 2xx status.
+func (r *Response) OK() bool { return r.Status >= 200 && r.Status < 300 }
+
+// Err returns an error describing a non-2xx reply, or nil.
+func (r *Response) Err(what string) error {
+ if r.OK() {
+ return nil
+ }
+ return fmt.Errorf("%s: HTTP %d: %s", what, r.Status, clip(r.Body))
+}
+
+// WithBearer returns a copy authenticated with another token.
+func (a *API) WithBearer(token string) *API {
+ c := *a
+ c.Bearer = token
+ return &c
+}
+
+// Do sends a request. path starts with /api/...; query may be nil.
+func (a *API) Do(ctx context.Context, method, path string, query url.Values, body any) (*Response, error) {
+ u := a.Base + path
+ if len(query) > 0 {
+ u += "?" + query.Encode()
+ }
+ var rd io.Reader
+ if body != nil {
+ raw, err := json.Marshal(body)
+ if err != nil {
+ return nil, err
+ }
+ rd = bytes.NewReader(raw)
+ }
+ req, err := http.NewRequestWithContext(ctx, method, u, rd)
+ if err != nil {
+ return nil, err
+ }
+ if body != nil {
+ req.Header.Set("Content-Type", "application/json")
+ }
+ if a.Bearer != "" {
+ req.Header.Set("Authorization", "Bearer "+a.Bearer)
+ }
+ hc := a.HTTP
+ if hc == nil {
+ hc = &http.Client{Timeout: 60 * time.Second}
+ }
+ resp, err := hc.Do(req)
+ if err != nil {
+ return nil, err
+ }
+ defer resp.Body.Close()
+ raw, err := io.ReadAll(resp.Body)
+ if err != nil {
+ return nil, err
+ }
+ return &Response{Status: resp.StatusCode, Body: raw}, nil
+}
+
+// MustOK sends a request and fails on a non-2xx status.
+func (a *API) MustOK(ctx context.Context, method, path string, query url.Values, body any) (*Response, error) {
+ r, err := a.Do(ctx, method, path, query, body)
+ if err != nil {
+ return nil, fmt.Errorf("%s %s: %w", method, path, err)
+ }
+ return r, r.Err(method + " " + path)
+}
+
+// ProjectQuery returns ?projectId=<id>.
+func ProjectQuery(projectID string) url.Values {
+ return url.Values{"projectId": []string{projectID}}
+}
+
+func clip(b []byte) string {
+ s := strings.TrimSpace(string(b))
+ if len(s) > 400 {
+ return s[:400] + "..."
+ }
+ return s
+}
diff --git a/cli/test/evals/internal/env/backend.go b/cli/test/evals/internal/env/backend.go
new file mode 100644
index 00000000..bd2039f6
--- /dev/null
+++ b/cli/test/evals/internal/env/backend.go
@@ -0,0 +1,245 @@
+package env
+
+import (
+ "bytes"
+ "context"
+ "crypto/rand"
+ "encoding/hex"
+ "errors"
+ "fmt"
+ "net"
+ "net/http"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "sync"
+ "syscall"
+ "time"
+)
+
+// Backend is one running backend subprocess.
+type Backend struct {
+ URL string
+ Port int
+ Dir string // run/backend: cwd, SQLite files, ./storage
+ DBPath string
+ HealthToken string
+
+ cmd *exec.Cmd
+ log *syncBuffer
+ done chan error
+}
+
+// BootOptions tunes the backend.
+type BootOptions struct {
+ // Dir is the run's backend directory (created if missing).
+ Dir string
+ // Env adds or overrides variables.
+ Env map[string]string
+ // Ready bounds how long to wait for /health.
+ Ready time.Duration
+}
+
+// errPortTaken means another server answered on the port FreePort picked
+// before this backend could bind it (a parallel run won the race).
+var errPortTaken = errors.New("another server answered on the backend's port")
+
+// Boot starts the backend binary in SQLite mode with fast background workers
+// and waits until it answers /health/deep with its own token. A port lost to
+// a parallel run is retried on a fresh one.
+func Boot(ctx context.Context, bin string, opt BootOptions) (*Backend, error) {
+ for attempt := 1; ; attempt++ {
+ b, err := boot(ctx, bin, opt)
+ if errors.Is(err, errPortTaken) && attempt < 3 {
+ continue
+ }
+ return b, err
+ }
+}
+
+func boot(ctx context.Context, bin string, opt BootOptions) (*Backend, error) {
+ if err := os.MkdirAll(opt.Dir, 0o755); err != nil {
+ return nil, err
+ }
+ port, err := FreePort()
+ if err != nil {
+ return nil, err
+ }
+ b := &Backend{
+ URL: fmt.Sprintf("http://127.0.0.1:%d", port),
+ Port: port,
+ Dir: opt.Dir,
+ DBPath: filepath.Join(opt.Dir, "traceway.db"),
+ HealthToken: randomHex(16),
+ log: &syncBuffer{},
+ done: make(chan error, 1),
+ }
+ vars := map[string]string{
+ "PATH": os.Getenv("PATH"),
+ "HOME": opt.Dir,
+ "TMPDIR": os.TempDir(),
+ "TZ": "UTC",
+ "JWT_SECRET": randomHex(24), // 48 characters
+ "DB_TYPE": "sqlite",
+ "SQLITE_PATH": b.DBPath,
+ "STORAGE_TYPE": "local",
+ "STORAGE_PATH": filepath.Join(opt.Dir, "storage"),
+ "PORTS": strconv.Itoa(port),
+ "APP_BASE_URL": b.URL,
+ "SQLITE_RETENTION_DAYS": "0",
+ "NOTIFICATION_POLL_SECONDS": "5",
+ "ONCALL_POLL_SECONDS": "5",
+ "OUTBOX_POLL_SECONDS": "5",
+ "SYNTHETICS_POLL_SECONDS": "5",
+ "SYNTHETICS_ALLOW_PRIVATE_TARGETS": "true",
+ "HEALTH_DEEP_TOKEN": b.HealthToken,
+ "SQLITE_CACHE_SIZE_MB": "64",
+ "INGEST_ADMISSION_WAIT_SECONDS": "30",
+ "REPORT_MAX_BODY_MB": "64",
+ }
+ for k, v := range opt.Env {
+ vars[k] = v
+ }
+ cmd := exec.Command(bin)
+ cmd.Dir = opt.Dir
+ cmd.Env = make([]string, 0, len(vars))
+ for k, v := range vars {
+ cmd.Env = append(cmd.Env, k+"="+v)
+ }
+ cmd.Stdout = b.log
+ cmd.Stderr = b.log
+ cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
+ if err := cmd.Start(); err != nil {
+ return nil, fmt.Errorf("start backend: %w", err)
+ }
+ b.cmd = cmd
+ go func() { b.done <- cmd.Wait() }()
+
+ ready := opt.Ready
+ if ready == 0 {
+ ready = 60 * time.Second
+ }
+ if err := b.waitReady(ctx, ready); err != nil {
+ b.Stop()
+ if errors.Is(err, errPortTaken) {
+ return nil, err
+ }
+ return nil, fmt.Errorf("%w\n--- backend log ---\n%s", err, b.LogTail(4000))
+ }
+ return b, nil
+}
+
+func (b *Backend) waitReady(ctx context.Context, timeout time.Duration) error {
+ deadline := time.Now().Add(timeout)
+ hc := &http.Client{Timeout: 2 * time.Second}
+ for time.Now().Before(deadline) {
+ select {
+ case err := <-b.done:
+ b.done <- err
+ return fmt.Errorf("backend exited before it was ready: %v", err)
+ case <-ctx.Done():
+ return ctx.Err()
+ default:
+ }
+ // /health/deep needs this backend's own token, so a server that
+ // took the port first answers 401 instead of passing for ours.
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, b.URL+"/api/health/deep", nil)
+ if err != nil {
+ return err
+ }
+ req.Header.Set("Authorization", "Bearer "+b.HealthToken)
+ if resp, err := hc.Do(req); err == nil {
+ _ = resp.Body.Close()
+ switch resp.StatusCode {
+ case http.StatusOK:
+ return nil
+ case http.StatusUnauthorized:
+ return errPortTaken
+ }
+ }
+ time.Sleep(100 * time.Millisecond)
+ }
+ return errors.New("backend never answered /health")
+}
+
+// Stop terminates the backend and its process group.
+func (b *Backend) Stop() {
+ if b.cmd == nil || b.cmd.Process == nil {
+ return
+ }
+ _ = syscall.Kill(-b.cmd.Process.Pid, syscall.SIGTERM)
+ select {
+ case <-b.done:
+ case <-time.After(10 * time.Second):
+ _ = syscall.Kill(-b.cmd.Process.Pid, syscall.SIGKILL)
+ <-b.done
+ }
+ b.cmd = nil
+}
+
+// Log returns the backend's combined output so far.
+func (b *Backend) Log() string { return b.log.String() }
+
+// LogTail returns the last n bytes of the log.
+func (b *Backend) LogTail(n int) string {
+ s := b.log.String()
+ if len(s) > n {
+ return s[len(s)-n:]
+ }
+ return s
+}
+
+// WriteLog saves the log next to the run.
+func (b *Backend) WriteLog(path string) error {
+ return os.WriteFile(path, []byte(b.log.String()), 0o644)
+}
+
+var inviteTokenRe = regexp.MustCompile(`accept-invitation\?token=([0-9a-f-]{36})`)
+
+// InvitationTokens returns every invitation accept token the backend has
+// logged so far. With SMTP off the email is written to the log, link included.
+func (b *Backend) InvitationTokens() []string {
+ var out []string
+ for _, m := range inviteTokenRe.FindAllStringSubmatch(b.log.String(), -1) {
+ out = append(out, m[1])
+ }
+ return out
+}
+
+// API returns an unauthenticated API client for this backend.
+func (b *Backend) API() *API { return &API{Base: b.URL} }
+
+// FreePort asks the OS for an unused TCP port.
+func FreePort() (int, error) {
+ l, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ return 0, err
+ }
+ defer l.Close()
+ return l.Addr().(*net.TCPAddr).Port, nil
+}
+
+func randomHex(n int) string {
+ b := make([]byte, n)
+ _, _ = rand.Read(b)
+ return hex.EncodeToString(b)
+}
+
+type syncBuffer struct {
+ mu sync.Mutex
+ buf bytes.Buffer
+}
+
+func (s *syncBuffer) Write(p []byte) (int, error) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return s.buf.Write(p)
+}
+
+func (s *syncBuffer) String() string {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return s.buf.String()
+}
diff --git a/cli/test/evals/internal/env/build.go b/cli/test/evals/internal/env/build.go
new file mode 100644
index 00000000..e503e4ff
--- /dev/null
+++ b/cli/test/evals/internal/env/build.go
@@ -0,0 +1,147 @@
+package env
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "sync"
+)
+
+// Binaries are the executables a run needs.
+type Binaries struct {
+ Backend string // backend/cmd/traceway
+ CLI string // cli/cmd/traceway (read edition)
+ CLIWrite string // cli/cmd/traceway -tags write
+ Hash string // tree hash they were built from
+}
+
+var (
+ buildMu sync.Mutex
+ buildCache = map[string]*Binaries{}
+)
+
+// BuildBinaries builds the backend and both CLI editions from repoRoot, cached
+// under cacheDir/bin/<tree-hash>/ so an unchanged tree never rebuilds.
+func BuildBinaries(repoRoot, cacheDir string) (*Binaries, error) {
+ buildMu.Lock()
+ defer buildMu.Unlock()
+ hash, err := TreeHash(repoRoot)
+ if err != nil {
+ return nil, err
+ }
+ if b, ok := buildCache[hash]; ok {
+ return b, nil
+ }
+ dir := filepath.Join(cacheDir, "bin", hash)
+ b := &Binaries{
+ Backend: filepath.Join(dir, "traceway-backend"),
+ CLI: filepath.Join(dir, "traceway"),
+ CLIWrite: filepath.Join(dir, "traceway-write"),
+ Hash: hash,
+ }
+ if err := os.MkdirAll(dir, 0o755); err != nil {
+ return nil, err
+ }
+ steps := []struct {
+ out, dir string
+ args []string
+ }{
+ {b.Backend, filepath.Join(repoRoot, "backend"), []string{"build", "-o", b.Backend, "./cmd/traceway"}},
+ {b.CLI, filepath.Join(repoRoot, "cli"), []string{"build", "-o", b.CLI, "./cmd/traceway"}},
+ {b.CLIWrite, filepath.Join(repoRoot, "cli"), []string{"build", "-tags", "write", "-o", b.CLIWrite, "./cmd/traceway"}},
+ }
+ for _, s := range steps {
+ if _, err := os.Stat(s.out); err == nil {
+ continue
+ }
+ // Build to a private path and rename, so concurrent harness
+ // processes never see (or execute) a half-written binary.
+ tmp := fmt.Sprintf("%s.tmp-%d", s.out, os.Getpid())
+ args := append([]string{}, s.args...)
+ for i := range args {
+ if args[i] == s.out {
+ args[i] = tmp
+ }
+ }
+ cmd := exec.Command("go", args...)
+ cmd.Dir = s.dir
+ cmd.Env = append(os.Environ(), "CGO_ENABLED=0")
+ if out, err := cmd.CombinedOutput(); err != nil {
+ return nil, fmt.Errorf("go %s in %s: %w\n%s", strings.Join(args, " "), s.dir, err, out)
+ }
+ if err := os.Rename(tmp, s.out); err != nil {
+ return nil, err
+ }
+ }
+ buildCache[hash] = b
+ return b, nil
+}
+
+// TreeHash fingerprints the backend and CLI sources: the committed trees,
+// the uncommitted diff, and untracked files. The evals module itself is
+// excluded so editing a case never rebuilds the binaries.
+func TreeHash(repoRoot string) (string, error) {
+ git := func(args ...string) ([]byte, error) {
+ cmd := exec.Command("git", args...)
+ cmd.Dir = repoRoot
+ out, err := cmd.Output()
+ if err != nil {
+ return nil, fmt.Errorf("git %s: %w", strings.Join(args, " "), err)
+ }
+ return out, nil
+ }
+ h := sha256.New()
+ paths := []string{"backend", "cli", ":(exclude)cli/test"}
+ head, err := git("ls-tree", "-r", "HEAD", "--", "backend", "cli")
+ if err != nil {
+ return "", err
+ }
+ for _, line := range strings.Split(string(head), "\n") {
+ if !strings.Contains(line, "\tcli/test/") {
+ h.Write([]byte(line))
+ }
+ }
+ diff, err := git(append([]string{"diff", "HEAD", "--binary", "--"}, paths...)...)
+ if err != nil {
+ return "", err
+ }
+ h.Write(diff)
+ untracked, err := git(append([]string{"ls-files", "-o", "--exclude-standard", "--"}, paths...)...)
+ if err != nil {
+ return "", err
+ }
+ for _, f := range strings.Split(strings.TrimSpace(string(untracked)), "\n") {
+ if f == "" {
+ continue
+ }
+ raw, err := os.ReadFile(filepath.Join(repoRoot, f))
+ if err != nil {
+ continue
+ }
+ h.Write([]byte(f))
+ h.Write(raw)
+ }
+ return hex.EncodeToString(h.Sum(nil))[:16], nil
+}
+
+// RepoRoot walks up from dir to the directory holding .git.
+func RepoRoot(dir string) (string, error) {
+ d, err := filepath.Abs(dir)
+ if err != nil {
+ return "", err
+ }
+ for {
+ if _, err := os.Stat(filepath.Join(d, ".git")); err == nil {
+ return d, nil
+ }
+ parent := filepath.Dir(d)
+ if parent == d {
+ return "", fmt.Errorf("no git repository above %s", dir)
+ }
+ d = parent
+ }
+}
diff --git a/cli/test/evals/internal/env/clis.go b/cli/test/evals/internal/env/clis.go
new file mode 100644
index 00000000..35d6ce80
--- /dev/null
+++ b/cli/test/evals/internal/env/clis.go
@@ -0,0 +1,74 @@
+package env
+
+import (
+ "bytes"
+ "context"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+)
+
+// CLIHome is the isolated home of one CLI edition: its XDG dirs live outside
+// the agent's workspace so the agent cannot read the stored token.
+type CLIHome struct {
+ Bin string
+ Home string
+ XDGCf string
+ XDGSt string
+}
+
+// NewCLIHome prepares dirs under root for a binary.
+func NewCLIHome(root, bin string) (*CLIHome, error) {
+ h := &CLIHome{Bin: bin, Home: filepath.Join(root, "home"), XDGCf: filepath.Join(root, "xdg", "config"), XDGSt: filepath.Join(root, "xdg", "state")}
+ for _, d := range []string{h.Home, h.XDGCf, h.XDGSt} {
+ if err := os.MkdirAll(d, 0o700); err != nil {
+ return nil, err
+ }
+ }
+ return h, nil
+}
+
+// Env is the minimal environment every tool subprocess gets. Nothing from the
+// harness's own environment (API keys, GitHub tokens, JWT secrets) is passed.
+func (h *CLIHome) Env(path string) []string {
+ return []string{
+ "HOME=" + h.Home,
+ "XDG_CONFIG_HOME=" + h.XDGCf,
+ "XDG_STATE_HOME=" + h.XDGSt,
+ "XDG_CACHE_HOME=" + filepath.Join(h.Home, ".cache"),
+ "PATH=" + path,
+ "TZ=UTC",
+ "LANG=C.UTF-8",
+ }
+}
+
+// Run executes the CLI with args and optional stdin.
+func (h *CLIHome) Run(ctx context.Context, stdin string, args ...string) (string, string, error) {
+ cmd := exec.CommandContext(ctx, h.Bin, args...)
+ cmd.Env = h.Env(filepath.Dir(h.Bin))
+ cmd.Dir = h.Home
+ if stdin != "" {
+ cmd.Stdin = strings.NewReader(stdin)
+ }
+ var so, se bytes.Buffer
+ cmd.Stdout, cmd.Stderr = &so, &se
+ err := cmd.Run()
+ return so.String(), se.String(), err
+}
+
+// Login stores a token for the instance and selects a project, the way a
+// user would: `login --url U --token-stdin` then `projects use <id>`.
+func (h *CLIHome) Login(ctx context.Context, url, token, projectID string) error {
+ if so, se, err := h.Run(ctx, token+"\n", "login", "--url", url, "--token-stdin"); err != nil {
+ return fmt.Errorf("%s login: %w\n%s%s", filepath.Base(h.Bin), err, so, se)
+ }
+ if projectID == "" {
+ return nil
+ }
+ if so, se, err := h.Run(ctx, "", "projects", "use", projectID); err != nil {
+ return fmt.Errorf("%s projects use: %w\n%s%s", filepath.Base(h.Bin), err, so, se)
+ }
+ return nil
+}
diff --git a/cli/test/evals/internal/env/env_test.go b/cli/test/evals/internal/env/env_test.go
new file mode 100644
index 00000000..05b5cec2
--- /dev/null
+++ b/cli/test/evals/internal/env/env_test.go
@@ -0,0 +1,97 @@
+package env
+
+import (
+ "context"
+ "net/http"
+ "path/filepath"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+)
+
+// TestTwoInstances boots two backends in parallel, registers each, and logs
+// the CLI in with a read token.
+func TestTwoInstances(t *testing.T) {
+ bins := BinariesForTest(t)
+ ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
+ defer cancel()
+
+ var wg sync.WaitGroup
+ for i := 0; i < 2; i++ {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ checkInstance(t, ctx, bins)
+ }()
+ }
+ wg.Wait()
+}
+
+func checkInstance(t *testing.T, ctx context.Context, bins *Binaries) {
+ dir := t.TempDir()
+ b, err := Boot(ctx, bins.Backend, BootOptions{Dir: filepath.Join(dir, "backend")})
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ defer b.Stop()
+
+ admin, err := Register(ctx, b, "[email protected]", "eval-password-1", "Eval Org")
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ projects, err := CreateProjects(ctx, b, admin, []Project{{Name: "api", Framework: "opentelemetry"}})
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ pid := projects[0].ID
+
+ readPAT, err := CreatePAT(ctx, b, admin.JWT, "eval-read", "read", 0)
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ if !strings.HasPrefix(readPAT, "twp_") {
+ t.Errorf("unexpected read token prefix %q", readPAT[:4])
+ }
+
+ cli, err := NewCLIHome(filepath.Join(dir, "cli"), bins.CLI)
+ if err != nil {
+ t.Error(err)
+ return
+ }
+ if err := cli.Login(ctx, b.URL, readPAT, pid); err != nil {
+ t.Error(err)
+ return
+ }
+ out, _, err := cli.Run(ctx, "", "projects", "list", "--output", "json")
+ if err != nil || !strings.Contains(out, pid) {
+ t.Errorf("projects list after login: %v %s", err, out)
+ }
+
+}
+
+func TestStubSwitches(t *testing.T) {
+ s, err := StartStub(false)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer s.Close()
+ res, err := http.Get(s.URL + "/health")
+ if err != nil {
+ t.Fatal(err)
+ }
+ res.Body.Close()
+ if res.StatusCode != 503 {
+ t.Fatalf("unhealthy stub answered %d", res.StatusCode)
+ }
+ s.SetHealthy(true)
+ res, _ = http.Get(s.URL + "/health")
+ res.Body.Close()
+ if res.StatusCode != 200 || s.Hits() != 2 {
+ t.Fatalf("healthy stub answered %d after %d hits", res.StatusCode, s.Hits())
+ }
+}
diff --git a/cli/test/evals/internal/env/instance.go b/cli/test/evals/internal/env/instance.go
new file mode 100644
index 00000000..b5d16ca3
--- /dev/null
+++ b/cli/test/evals/internal/env/instance.go
@@ -0,0 +1,51 @@
+package env
+
+import (
+ "context"
+ "fmt"
+ "path/filepath"
+)
+
+// Instance is a booted, registered backend with its monitor stub.
+type Instance struct {
+ Backend *Backend
+ Admin *Admin
+ Stub *Stub
+ Dir string
+}
+
+// AdminEmail and AdminPassword are the owner credentials of every instance.
+const (
+ AdminEmail = "[email protected]"
+ AdminPassword = "eval-owner-password"
+)
+
+// NewInstance boots a backend under dir/backend, registers the owner and
+// organization, and starts a healthy stub.
+func NewInstance(ctx context.Context, bins *Binaries, dir string) (*Instance, error) {
+ b, err := Boot(ctx, bins.Backend, BootOptions{Dir: filepath.Join(dir, "backend")})
+ if err != nil {
+ return nil, err
+ }
+ a, err := Register(ctx, b, AdminEmail, AdminPassword, "Acme")
+ if err != nil {
+ b.Stop()
+ return nil, fmt.Errorf("register: %w", err)
+ }
+ stub, err := StartStub(true)
+ if err != nil {
+ b.Stop()
+ return nil, err
+ }
+ return &Instance{Backend: b, Admin: a, Stub: stub, Dir: dir}, nil
+}
+
+// Close stops everything.
+func (i *Instance) Close() {
+ if i.Stub != nil {
+ i.Stub.Close()
+ }
+ if i.Backend != nil {
+ i.Backend.Stop()
+ }
+}
diff --git a/cli/test/evals/internal/env/register.go b/cli/test/evals/internal/env/register.go
new file mode 100644
index 00000000..ecaf5c54
--- /dev/null
+++ b/cli/test/evals/internal/env/register.go
@@ -0,0 +1,113 @@
+package env
+
+import (
+ "context"
+ "fmt"
+ "net/http"
+)
+
+// Project is one Traceway project of the instance.
+type Project struct {
+ ID string `json:"id"`
+ Name string `json:"name"`
+ Framework string `json:"framework"`
+ Token string `json:"-"` // ingest token; never written to seed.json
+}
+
+// Admin is the registered owner of the instance's one organization.
+type Admin struct {
+ Email string
+ Password string
+ UserID int
+ OrgID int
+ JWT string
+}
+
+// Register creates the instance's owner and organization. A self-hosted
+// instance allows exactly one registration, which is why every run boots its
+// own backend.
+func Register(ctx context.Context, b *Backend, email, password, org string) (*Admin, error) {
+ res, err := b.API().MustOK(ctx, http.MethodPost, "/api/register", nil, map[string]any{
+ "email": email, "name": "Eval Admin", "password": password,
+ "organizationName": org, "timezone": "UTC",
+ })
+ if err != nil {
+ return nil, err
+ }
+ var out struct {
+ Token string `json:"token"`
+ User struct {
+ ID int `json:"id"`
+ } `json:"user"`
+ Organizations []struct {
+ ID int `json:"id"`
+ } `json:"organizations"`
+ }
+ if err := res.JSON(&out); err != nil {
+ return nil, err
+ }
+ if out.Token == "" || len(out.Organizations) == 0 {
+ return nil, fmt.Errorf("register returned no token or organization: %s", clip(res.Body))
+ }
+ return &Admin{Email: email, Password: password, UserID: out.User.ID, OrgID: out.Organizations[0].ID, JWT: out.Token}, nil
+}
+
+// CreateProjects creates projects in the admin's organization in one batch
+// and returns them with their ingest tokens.
+func CreateProjects(ctx context.Context, b *Backend, a *Admin, specs []Project) ([]Project, error) {
+ if len(specs) == 0 {
+ return nil, nil
+ }
+ body := map[string]any{"organizationId": a.OrgID}
+ var list []map[string]string
+ for _, p := range specs {
+ list = append(list, map[string]string{"name": p.Name, "framework": p.Framework})
+ }
+ body["projects"] = list
+ res, err := b.API().WithBearer(a.JWT).MustOK(ctx, http.MethodPost, "/api/projects/batch", nil, body)
+ if err != nil {
+ return nil, err
+ }
+ var out struct {
+ Projects []struct {
+ ID string `json:"id"`
+ Name string `json:"name"`
+ Framework string `json:"framework"`
+ Token string `json:"token"`
+ } `json:"projects"`
+ }
+ if err := res.JSON(&out); err != nil {
+ return nil, err
+ }
+ if len(out.Projects) != len(specs) {
+ return nil, fmt.Errorf("batch create returned %d projects, want %d", len(out.Projects), len(specs))
+ }
+ projects := make([]Project, 0, len(out.Projects))
+ for _, p := range out.Projects {
+ projects = append(projects, Project{ID: p.ID, Name: p.Name, Framework: p.Framework, Token: p.Token})
+ }
+ return projects, nil
+}
+
+// CreatePAT mints a personal access token for the bearer's user. Write
+// tokens must expire; expiresInDays 0 means "never" for read tokens.
+func CreatePAT(ctx context.Context, b *Backend, jwt, name, scope string, expiresInDays int) (string, error) {
+ body := map[string]any{"name": name, "scope": scope}
+ if expiresInDays > 0 {
+ body["expiresInDays"] = expiresInDays
+ }
+ res, err := b.API().WithBearer(jwt).MustOK(ctx, http.MethodPost, "/api/personal-access-tokens", nil, body)
+ if err != nil {
+ return "", err
+ }
+ var out struct {
+ Token string `json:"token"`
+ }
+ if err := res.JSON(&out); err != nil {
+ return "", err
+ }
+ if out.Token == "" {
+ return "", fmt.Errorf("PAT create returned no token")
+ }
+ return out.Token, nil
+}
diff --git a/cli/test/evals/internal/env/stub.go b/cli/test/evals/internal/env/stub.go
new file mode 100644
index 00000000..42a51b14
--- /dev/null
+++ b/cli/test/evals/internal/env/stub.go
@@ -0,0 +1,53 @@
+package env
+
+import (
+ "fmt"
+ "net"
+ "net/http"
+ "sync/atomic"
+)
+
+// Stub is a local HTTP target whose health the harness switches, so monitors
+// can go down and recover on cue.
+type Stub struct {
+ URL string
+ healthy atomic.Bool
+ hits atomic.Int64
+ srv *http.Server
+ ln net.Listener
+}
+
+// StartStub starts a stub on a free loopback port.
+func StartStub(healthy bool) (*Stub, error) {
+ ln, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ return nil, err
+ }
+ s := &Stub{URL: "http://" + ln.Addr().String(), ln: ln}
+ s.healthy.Store(healthy)
+ mux := http.NewServeMux()
+ mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+ s.hits.Add(1)
+ if s.healthy.Load() {
+ w.Header().Set("Content-Type", "application/json")
+ fmt.Fprint(w, `{"status":"ok"}`)
+ return
+ }
+ http.Error(w, `{"status":"unavailable"}`, http.StatusServiceUnavailable)
+ })
+ s.srv = &http.Server{Handler: mux}
+ go func() { _ = s.srv.Serve(ln) }()
+ return s, nil
+}
+
+// SetHealthy switches the stub's answers.
+func (s *Stub) SetHealthy(v bool) { s.healthy.Store(v) }
+
+// Healthy reports the current state.
+func (s *Stub) Healthy() bool { return s.healthy.Load() }
+
+// Hits counts requests served.
+func (s *Stub) Hits() int64 { return s.hits.Load() }
+
+// Close stops the stub.
+func (s *Stub) Close() { _ = s.srv.Close() }
diff --git a/cli/test/evals/internal/env/testhelp.go b/cli/test/evals/internal/env/testhelp.go
new file mode 100644
index 00000000..dcf70833
--- /dev/null
+++ b/cli/test/evals/internal/env/testhelp.go
@@ -0,0 +1,36 @@
+package env
+
+import (
+ "os/exec"
+ "path/filepath"
+ "runtime"
+ "testing"
+)
+
+// EvalsRoot is the cli/test/evals module directory.
+func EvalsRoot() string {
+ _, file, _, _ := runtime.Caller(0)
+ return filepath.Clean(filepath.Join(filepath.Dir(file), "..", ".."))
+}
+
+// BinariesForTest builds (or reuses) the binaries for a test, skipping it in
+// -short mode or when no Go toolchain is on PATH.
+func BinariesForTest(t testing.TB) *Binaries {
+ t.Helper()
+ if testing.Short() {
+ t.Skip("boots a backend")
+ }
+ if _, err := exec.LookPath("go"); err != nil {
+ t.Skip("go toolchain not on PATH")
+ }
+ root := EvalsRoot()
+ repo, err := RepoRoot(root)
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, err := BuildBinaries(repo, filepath.Join(root, ".cache"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return b
+}
diff --git a/cli/test/evals/internal/grade/answer.go b/cli/test/evals/internal/grade/answer.go
new file mode 100644
index 00000000..e2e9e5c5
--- /dev/null
+++ b/cli/test/evals/internal/grade/answer.go
@@ -0,0 +1,143 @@
+package grade
+
+import (
+ "encoding/json"
+ "regexp"
+ "slices"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+)
+
+// Answer grades the structured submission against expect.answer.
+func Answer(in *Input) *Verdict {
+ v := newVerdict("answer", true)
+ exp := in.Case.Expect.Answer
+ sub := in.Run.Submission
+ if sub == nil {
+ v.fail("no submission (%s)", in.Run.StopReason)
+ return v
+ }
+ str := func(m map[string]any, k string) string { s, _ := m[k].(string); return strings.TrimSpace(s) }
+ switch in.Case.Suite {
+ case "perf":
+ got := str(sub, "checklist_item")
+ want := append([]string{}, exp.AnyOf...)
+ if exp.ChecklistItem != "" {
+ want = append(want, exp.ChecklistItem)
+ }
+ if len(want) > 0 && !slices.Contains(want, got) {
+ v.fail("checklist_item %q, want %s", got, strings.Join(want, " or "))
+ }
+ if slices.Contains(exp.Not, got) {
+ v.fail("checklist_item %q is the distractor", got)
+ }
+ if exp.Locus != nil {
+ locus, _ := sub["locus"].(map[string]any)
+ if locus == nil {
+ locus = map[string]any{}
+ }
+ if exp.Locus.Endpoint != "" && !sameEndpoint(str(locus, "endpoint"), exp.Locus.Endpoint) {
+ v.fail("locus.endpoint %q, want %q", str(locus, "endpoint"), exp.Locus.Endpoint)
+ }
+ if exp.Locus.SpanContains != "" && !containsFold(str(locus, "span"), exp.Locus.SpanContains) {
+ v.fail("locus.span %q does not contain %q", str(locus, "span"), exp.Locus.SpanContains)
+ }
+ if exp.Locus.Service != "" && !containsFold(str(locus, "service"), exp.Locus.Service) {
+ v.fail("locus.service %q, want %q", str(locus, "service"), exp.Locus.Service)
+ }
+ if exp.Locus.Host != "" && !containsFold(str(locus, "host"), exp.Locus.Host) {
+ v.fail("locus.host %q, want %q", str(locus, "host"), exp.Locus.Host)
+ }
+ }
+ if exp.Onset != nil && exp.Onset.At != "" {
+ gradeOnset(v, sub["onset"], exp.Onset)
+ }
+ }
+ if len(exp.MentionsAll) > 0 {
+ raw, _ := json.Marshal(sub)
+ text := string(raw) + "\n" + in.Run.FinalText
+ for _, pat := range exp.MentionsAll {
+ re, err := regexp.Compile("(?is)" + pat)
+ if err != nil {
+ v.fail("bad mentions_all pattern %q: %v", pat, err)
+ continue
+ }
+ if !re.MatchString(text) {
+ v.fail("the answer never mentions /%s/", pat)
+ }
+ }
+ }
+ if p, ok := sub["_validation_problems"].(string); ok {
+ v.note("accepted after failed repairs: %s", p)
+ }
+ return v
+}
+
+func gradeOnset(v *Verdict, got any, exp *catalog.Onset) {
+ want, err := time.Parse(time.RFC3339Nano, exp.At)
+ if err != nil {
+ v.fail("case onset %q is not RFC 3339", exp.At)
+ return
+ }
+ s, _ := got.(string)
+ at, err := time.Parse(time.RFC3339Nano, s)
+ if err != nil {
+ v.fail("onset %v is not an RFC 3339 time", got)
+ return
+ }
+ tol := time.Duration(exp.ToleranceMinutes) * time.Minute
+ if tol == 0 {
+ tol = time.Hour
+ }
+ if d := at.Sub(want); d < -tol || d > tol {
+ v.fail("onset %s is %s from %s (tolerance %s)", at.Format(time.RFC3339), d.Round(time.Minute), want.Format(time.RFC3339), tol)
+ }
+}
+
+var spaceRe = regexp.MustCompile(`\s+`)
+
+// sameEndpoint compares "GET /api/orders" forms, tolerating case, spacing,
+// a trailing slash and a missing method when the route matches.
+func sameEndpoint(got, want string) bool {
+ norm := func(s string) string {
+ s = strings.TrimSpace(spaceRe.ReplaceAllString(s, " "))
+ s = strings.Trim(s, "`\"'")
+ method, route, ok := strings.Cut(s, " ")
+ if !ok {
+ return strings.TrimSuffix(s, "/")
+ }
+ return strings.ToUpper(method) + " " + strings.TrimSuffix(route, "/")
+ }
+ g, w := norm(got), norm(want)
+ if g == w {
+ return true
+ }
+ _, wr, _ := strings.Cut(w, " ")
+ return g == wr || strings.HasSuffix(g, " "+wr)
+}
+
+func containsFold(s, sub string) bool {
+ return strings.Contains(strings.ToLower(s), strings.ToLower(sub))
+}
+
+// Report checks free-text requirements on the final report.
+func Report(in *Input) *Verdict {
+ v := newVerdict("report", true)
+ text := FinalText(in.Run)
+ for _, pat := range in.Case.Expect.Report.Contains {
+ if re, err := regexp.Compile("(?is)" + pat); err != nil || !re.MatchString(text) {
+ v.fail("report does not match /%s/", pat)
+ }
+ }
+ for _, pat := range in.Case.Expect.Report.NotContains {
+ if re, err := regexp.Compile("(?is)" + pat); err == nil && re.MatchString(text) {
+ v.fail("report must not match /%s/", pat)
+ }
+ }
+ if text == "" {
+ v.fail("no report")
+ }
+ return v
+}
diff --git a/cli/test/evals/internal/grade/grade.go b/cli/test/evals/internal/grade/grade.go
new file mode 100644
index 00000000..8886dac2
--- /dev/null
+++ b/cli/test/evals/internal/grade/grade.go
@@ -0,0 +1,84 @@
+// Package grade scores one run. Deterministic graders gate the result:
+// answer, trajectory, grounding and the report checks.
+package grade
+
+import (
+ "context"
+ "fmt"
+ "strings"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// Verdict is one grader's result.
+type Verdict struct {
+ Grader string `json:"grader"`
+ Pass bool `json:"pass"`
+ Gating bool `json:"gating"`
+ Skipped bool `json:"skipped,omitempty"`
+ Score *float64 `json:"score,omitempty"`
+ Failures []string `json:"failures,omitempty"`
+ Notes []string `json:"notes,omitempty"`
+ CostUSD float64 `json:"costUsd,omitempty"`
+}
+
+func (v *Verdict) fail(format string, a ...any) {
+ v.Pass = false
+ v.Failures = append(v.Failures, fmt.Sprintf(format, a...))
+}
+
+func (v *Verdict) note(format string, a ...any) { v.Notes = append(v.Notes, fmt.Sprintf(format, a...)) }
+
+func newVerdict(name string, gating bool) *Verdict {
+ return &Verdict{Grader: name, Pass: true, Gating: gating}
+}
+
+// Input is everything the graders see.
+type Input struct {
+ Case *catalog.Case // templates resolved
+ Arm string
+ Run *record.Run
+ Seed map[string]any
+}
+
+// All runs every applicable grader in a fixed order.
+func All(ctx context.Context, in *Input) []*Verdict {
+ var out []*Verdict
+ if in.Case.Expect.Answer != nil {
+ out = append(out, Answer(in))
+ }
+ if in.Case.Expect.Calls != nil {
+ out = append(out, Trajectory(in.Run, in.Case.Expect.Calls))
+ }
+ if in.Case.Suite == "perf" || in.Case.Suite == "debug" || len(in.Case.Expect.Grounding) > 0 {
+ out = append(out, Grounding(in))
+ }
+ if in.Case.Expect.Report != nil {
+ out = append(out, Report(in))
+ }
+ return out
+}
+
+// Passed reports whether every gating grader passed.
+func Passed(vs []*Verdict) bool {
+ for _, v := range vs {
+ if v.Gating && !v.Skipped && !v.Pass {
+ return false
+ }
+ }
+ return true
+}
+
+// FinalText is the agent's report: the submission's report/summary field,
+// else its last message.
+func FinalText(r *record.Run) string {
+ if r.Submission != nil {
+ for _, k := range []string{"report", "summary"} {
+ if s, ok := r.Submission[k].(string); ok && strings.TrimSpace(s) != "" {
+ return s
+ }
+ }
+ }
+ return r.FinalText
+}
diff --git a/cli/test/evals/internal/grade/grade_test.go b/cli/test/evals/internal/grade/grade_test.go
new file mode 100644
index 00000000..4f51529f
--- /dev/null
+++ b/cli/test/evals/internal/grade/grade_test.go
@@ -0,0 +1,166 @@
+package grade
+
+import (
+ "encoding/json"
+ "strings"
+ "testing"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+func bp(b bool) *bool { return &b }
+
+// call builds a recorded call from CLI argv strings.
+func call(id string, ok bool, output string, argvs ...string) *record.Call {
+ c := &record.Call{ID: id, Tool: "shell", OK: ok, Output: output, Args: json.RawMessage(`{}`)}
+ for _, a := range argvs {
+ argv := strings.Fields(a)
+ c.Ops = append(c.Ops, record.OpRecord{Op: ops.ParseCLI(argv[1:]), Argv: argv, OK: ok, Output: output})
+ }
+ return c
+}
+
+func failures(v *Verdict) string { return strings.Join(v.Failures, "\n") }
+
+func TestTrajectory(t *testing.T) {
+ run := &record.Run{Calls: []*record.Call{
+ call("c01", true, "{}", "traceway endpoints list --since 24h"),
+ call("c02", true, "{}", "traceway endpoints show abc --recorded-at 2026-10-17T10:00:00Z"),
+ call("c03", false, "error", "traceway metrics query --name system.cpu.utilization"),
+ {ID: "c04", Tool: "read_file", OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: map[string]any{"path": "internal/handlers/orders.go"}}, OK: true}}},
+ }}
+ spec := &catalog.Trajectory{
+ Required: []catalog.CallMatcher{{Op: "endpoints.list"}, {Op: "endpoints.show", Args: map[string]any{"recorded_at": "present"}}, {Op: "read_file:internal/handlers/*.go"}},
+ AnyOf: [][]catalog.CallMatcher{{{Op: "metrics.query", Args: map[string]any{"name": "~system\\.cpu"}}}},
+ Forbidden: []catalog.CallMatcher{{Op: "*", Write: bp(true)}},
+ Order: [][]string{{"endpoints.list", "endpoints.show"}},
+ }
+ if v := Trajectory(run, spec); !v.Pass {
+ t.Fatalf("a good trajectory failed:\n%s", failures(v))
+ }
+
+ cases := map[string]struct {
+ mutate func(r *record.Run, s *catalog.Trajectory)
+ want string
+ }{
+ "forbidden write": {func(r *record.Run, s *catalog.Trajectory) {
+ r.Calls = append(r.Calls, call("c05", true, "{}", "traceway-write exceptions archive abcd1234abcd1234 --yes"))
+ }, "forbidden call"},
+ "wrong order": {func(r *record.Run, s *catalog.Trajectory) {
+ r.Calls[0], r.Calls[1] = r.Calls[1], r.Calls[0]
+ }, "came after"},
+ "missing timestamp": {func(r *record.Run, s *catalog.Trajectory) {
+ r.Calls[1] = call("c02", true, "{}", "traceway endpoints show abc")
+ }, "required call missing: endpoints.show"},
+ "no alternative": {func(r *record.Run, s *catalog.Trajectory) {
+ r.Calls = r.Calls[:2]
+ r.Calls = append(r.Calls, call("c03", true, "{}", "traceway metrics query --name mem.used"))
+ }, "none of these alternatives"},
+ "thrash": {func(r *record.Run, s *catalog.Trajectory) {
+ s.MaxIdenticalRepeats = 1
+ r.Calls = append(r.Calls, call("c09", true, "{}", "traceway endpoints list --since 24h --output json"))
+ }, "same call ran 2 times"},
+ "too many failures": {func(r *record.Run, s *catalog.Trajectory) {
+ s.MaxFailedCalls = 0
+ s.MaxFailedCalls = 1
+ r.Calls = append(r.Calls, call("c10", false, "x", "traceway logs query"))
+ }, "failed tool calls"},
+ }
+ for name, c := range cases {
+ t.Run(name, func(t *testing.T) {
+ r := &record.Run{Calls: append([]*record.Call{}, run.Calls...)}
+ s := *spec
+ c.mutate(r, &s)
+ v := Trajectory(r, &s)
+ if v.Pass || !strings.Contains(failures(v), c.want) {
+ t.Fatalf("want failure %q, got pass=%v:\n%s", c.want, v.Pass, failures(v))
+ }
+ })
+ }
+}
+
+func perfInput(sub map[string]any, calls ...*record.Call) *Input {
+ return &Input{
+ Case: &catalog.Case{ID: "perf-x", Suite: "perf", Expect: catalog.Expect{
+ Answer: &catalog.Answer{ChecklistItem: "select_n_plus_1", Not: []string{"over_fetching"}, Locus: &catalog.Locus{Endpoint: "GET /api/orders", SpanContains: "order_items"}},
+ Grounding: []string{"FROM order_items WHERE order_id"},
+ }},
+ Run: &record.Run{Prompt: "why is GET /api/orders slow", Submission: sub, Calls: calls},
+ }
+}
+
+func TestAnswer(t *testing.T) {
+ good := map[string]any{"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "get /api/orders/", "span": "SELECT ... FROM order_items WHERE order_id = $1"}}
+ if v := Answer(perfInput(good)); !v.Pass {
+ t.Fatalf("good answer failed: %s", failures(v))
+ }
+ for name, sub := range map[string]map[string]any{
+ "wrong row": {"checklist_item": "missing_or_unused_index", "locus": good["locus"]},
+ "distractor": {"checklist_item": "over_fetching", "locus": good["locus"]},
+ "wrong locus": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "GET /api/orders/:id", "span": "order_items"}},
+ "no span": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "GET /api/orders"}},
+ "no submission": nil,
+ } {
+ if v := Answer(perfInput(sub)); v.Pass {
+ t.Errorf("%s: should fail", name)
+ }
+ }
+}
+
+func TestGrounding(t *testing.T) {
+ out := `{"spans":[{"attributes":{"db.query.text":"SELECT id FROM order_items WHERE order_id = $1"}}],"id":"6f9619ff-8b86-d011-b42d-00c04fc964ff"}`
+ c1 := call("c01", true, out, "traceway endpoints show 6f9619ff-8b86-d011-b42d-00c04fc964ff --recorded-at 2026-10-17T10:00:00Z")
+ sub := func(id string, extra string) map[string]any {
+ return map[string]any{"evidence": []any{map[string]any{"call_id": id, "claim": "x"}}, "report": "request 6f9619ff-8b86-d011-b42d-00c04fc964ff " + extra}
+ }
+ if v := Grounding(perfInput(sub("c01", ""), c1)); !v.Pass {
+ t.Fatalf("grounded answer failed: %s", failures(v))
+ }
+ tests := map[string]struct {
+ in *Input
+ want string
+ }{
+ "unknown call": {perfInput(sub("c07", ""), c1), "not a call"},
+ "failed call": {perfInput(sub("c01", ""), call("c01", false, out, "traceway endpoints show x")), "which failed"},
+ "truncated call": {perfInput(sub("c01", ""), &record.Call{ID: "c01", OK: true, Truncated: true, Output: out}), "truncated"},
+ "invented id": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1), "no tool returned"},
+ "missing key": {perfInput(sub("c02", ""), c1, call("c02", true, `{"data":[]}`, "traceway endpoints list")), "contains"},
+ }
+ for name, tc := range tests {
+ t.Run(name, func(t *testing.T) {
+ v := Grounding(tc.in)
+ if v.Pass || !strings.Contains(failures(v), tc.want) {
+ t.Fatalf("want %q, got pass=%v: %s", tc.want, v.Pass, failures(v))
+ }
+ })
+ }
+}
+
+func TestLeakageLint(t *testing.T) {
+ c := &catalog.Case{Prompt: "Why is checkout slow?", Expect: catalog.Expect{Answer: &catalog.Answer{ChecklistItem: "cache_stampede_thundering_herd"}}}
+ if got := Lint(c, []string{"GET catalog:v3", "SELECT id FROM products WHERE id = $1"}); len(got) != 0 {
+ t.Fatalf("clean data flagged: %v", got)
+ }
+ got := Lint(c, []string{"recompute after cache stampede", "N+1 queries detected"})
+ if len(got) < 2 {
+ t.Fatalf("leaks not flagged: %v", got)
+ }
+ c.Prompt = "Is this a cache stampede?"
+ if got := Lint(c, nil); len(got) == 0 {
+ t.Fatal("a prompt naming the answer must be flagged")
+ }
+}
+
+func TestReport(t *testing.T) {
+ in := &Input{Case: &catalog.Case{Expect: catalog.Expect{Report: &catalog.ReportSpec{Contains: []string{"snooz"}, NotContains: []string{"disabled"}}}},
+ Run: &record.Run{Submission: map[string]any{"summary": "Snoozed rule 4 for 60 minutes."}}}
+ if v := Report(in); !v.Pass {
+ t.Fatal(failures(v))
+ }
+ in.Run.Submission["summary"] = "I disabled the rule."
+ if v := Report(in); v.Pass {
+ t.Fatal("must fail")
+ }
+}
diff --git a/cli/test/evals/internal/grade/grounding.go b/cli/test/evals/internal/grade/grounding.go
new file mode 100644
index 00000000..98a05ea2
--- /dev/null
+++ b/cli/test/evals/internal/grade/grounding.go
@@ -0,0 +1,147 @@
+package grade
+
+import (
+ "encoding/json"
+ "regexp"
+ "sort"
+ "strings"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// Grounding checks that the evidence is real:
+// - every evidence.call_id names a call that exists, succeeded and was
+// not truncated;
+// - every grounding key appears in the output of at least one cited call
+// (what the model saw, or the raw traceway output inside that call);
+// - every UUID, 32-hex trace id or 16-hex hash in the answer appears in
+// some tool output or the prompt: no invented ids.
+func Grounding(in *Input) *Verdict {
+ v := newVerdict("grounding", true)
+ run := in.Run
+ sub := run.Submission
+ if sub == nil {
+ v.fail("no submission to ground")
+ return v
+ }
+ var cited []*record.Call
+ if ev, ok := sub["evidence"].([]any); ok {
+ for i, e := range ev {
+ m, _ := e.(map[string]any)
+ id, _ := m["call_id"].(string)
+ c := run.CallByID(strings.TrimSpace(id))
+ switch {
+ case c == nil:
+ v.fail("evidence[%d] cites %q, which is not a call", i, id)
+ case c.Rejected:
+ v.fail("evidence[%d] cites %s, which was never executed", i, id)
+ case !c.OK || !anyOpOK(c):
+ v.fail("evidence[%d] cites %s, which failed", i, id)
+ case c.Truncated:
+ v.fail("evidence[%d] cites %s, whose output was truncated", i, id)
+ default:
+ cited = append(cited, c)
+ }
+ }
+ } else if in.Case.Suite == "perf" || in.Case.Suite == "debug" {
+ v.fail("the submission cites no evidence")
+ }
+ for _, key := range in.Case.Expect.Grounding {
+ found := false
+ for _, c := range cited {
+ if callContains(c, key) {
+ found = true
+ break
+ }
+ }
+ if !found {
+ v.fail("no cited call's output contains %q", key)
+ }
+ }
+ if invented := InventedIDs(run); len(invented) > 0 {
+ v.fail("ids in the answer that no tool returned: %s", strings.Join(invented, ", "))
+ }
+ return v
+}
+
+// callContains looks for a grounding key in what the model saw and in the
+// raw output of the call's successful operations. MCP results are JSON, which
+// escapes &, < and > (\u0026, \u003c, \u003e), so those are compared
+// unescaped.
+func callContains(c *record.Call, key string) bool {
+ if anyOpOK(c) && strings.Contains(unescapeJSON(c.Output), key) {
+ return true
+ }
+ for _, op := range c.Ops {
+ if op.OK && strings.Contains(unescapeJSON(op.Output), key) {
+ return true
+ }
+ }
+ return false
+}
+
+// anyOpOK reports whether a call has no operations (a plain shell command)
+// or at least one that succeeded. A traceway command that failed inside a
+// pipeline does not make the call evidence just because jq exited 0.
+func anyOpOK(c *record.Call) bool {
+ if len(c.Ops) == 0 {
+ return true
+ }
+ for _, op := range c.Ops {
+ if op.OK {
+ return true
+ }
+ }
+ return false
+}
+
+var jsonEscapes = strings.NewReplacer(`\u0026`, "&", `\u003c`, "<", `\u003e`, ">", `\u0027`, "'")
+
+func unescapeJSON(s string) string {
+ if !strings.Contains(s, `\u00`) {
+ return s
+ }
+ return jsonEscapes.Replace(s)
+}
+
+var idRe = regexp.MustCompile(`(?i)\b(?:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32}|[0-9a-f]{16})\b`)
+
+// InventedIDs lists ids in the submission (and final text) that appear in no
+// tool output and not in the prompt.
+func InventedIDs(run *record.Run) []string {
+ raw, _ := json.Marshal(run.Submission)
+ text := string(raw) + "\n" + run.FinalText
+ var corpus strings.Builder
+ corpus.WriteString(strings.ToLower(run.Prompt))
+ for _, c := range run.Calls {
+ corpus.WriteString(strings.ToLower(c.Output))
+ corpus.WriteString(strings.ToLower(string(c.Args)))
+ for _, op := range c.Ops {
+ corpus.WriteString(strings.ToLower(op.Output))
+ }
+ }
+ all := corpus.String()
+ seen := map[string]bool{}
+ var out []string
+ for _, id := range idRe.FindAllString(text, -1) {
+ l := strings.ToLower(id)
+ if seen[l] || isAllDigits(l) {
+ continue
+ }
+ seen[l] = true
+ if !strings.Contains(all, l) && !strings.Contains(all, strings.ReplaceAll(l, "-", "")) {
+ out = append(out, id)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
+
+func isAllDigits(s string) bool {
+ for _, c := range s {
+ if c < '0' || c > '9' {
+ return false
+ }
+ }
+ return true
+}
diff --git a/cli/test/evals/internal/grade/leakage.go b/cli/test/evals/internal/grade/leakage.go
new file mode 100644
index 00000000..67aba577
--- /dev/null
+++ b/cli/test/evals/internal/grade/leakage.go
@@ -0,0 +1,110 @@
+package grade
+
+import (
+ "fmt"
+ "regexp"
+ "sort"
+ "strings"
+ "sync"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/checklist"
+)
+
+// Leakage lint (plan §4.8): the seeded data and the prompt must not name the
+// answer. Seeded strings (span names, attribute values, log bodies, stack
+// traces, object names) are checked against a denylist of checklist terms
+// and against the expected checklist row's title; prompts get the checklist
+// terms only, since a user naturally says "slow" or "bug".
+
+// seedDenylist are phrases that give a diagnosis away when they appear in
+// telemetry. Real library messages never contain them.
+var seedDenylist = []string{
+ "n+1", "n + 1", "nplus1", "n_plus_1", "select n", "stampede", "thundering herd", "missing index", "unused index",
+ "no index", "pool exhaust", "exhausted pool", "exhaustion", "gc pause", "noisy neighbo", "cold start", "warm-up",
+ "warmup", "lock contention", "contention", "over-fetch", "overfetch", "no pagination", "unbounded", "fan-out",
+ "fanout", "chatty", "staircase", "parallelize", "run in parallel", "retry storm", "aggressive retr", "connection reuse",
+ "large payload", "payload too large", "low hit ratio", "cpu saturat", "saturation", "memory pressure",
+ "disk bottleneck", "io bottleneck", "quadratic", "o(n^2)", "o(n2)", "o(n²)", "background task", "should be async",
+ "excessive logging", "log spam", "root cause", "bottleneck", "slow query", "culprit", "the cause", "deliberately",
+ "bug here", "intentional", "eval", "seeded", "distractor",
+}
+
+// promptDenylist are checklist phrases a prompt must not contain.
+var promptDenylist = []string{
+ "n+1", "n + 1", "stampede", "thundering herd", "missing index", "pool exhaust", "exhaustion", "gc pause",
+ "garbage collect", "noisy neighbo", "cold start", "lock contention", "over-fetch", "pagination", "fan-out",
+ "staircase", "in parallel", "retries", "connection reuse", "hit ratio", "saturat", "memory pressure",
+ "quadratic", "o(n", "background task", "excessive logging", "culprit", "distractor",
+}
+
+// Finding is one leak.
+type Finding struct {
+ Where string
+ Term string
+ Text string
+}
+
+func (f Finding) String() string {
+ return fmt.Sprintf("%s contains %q: %s", f.Where, f.Term, clipS(f.Text, 160))
+}
+
+// Lint checks a case's seeded strings and prompt.
+func Lint(c *catalog.Case, seeded []string) []Finding {
+ var out []Finding
+ seen := map[string]bool{}
+ terms := append([]string{}, seedDenylist...)
+ if c.Expect.Answer != nil && c.Expect.Answer.ChecklistItem != "" {
+ for _, r := range checklist.Rows() {
+ if r.ID == c.Expect.Answer.ChecklistItem {
+ terms = append(terms, strings.ToLower(r.Title))
+ }
+ }
+ }
+ for _, s := range seeded {
+ low := strings.ToLower(s)
+ for _, t := range terms {
+ if containsTerm(low, t) {
+ k := t + "|" + s
+ if !seen[k] {
+ seen[k] = true
+ out = append(out, Finding{Where: "seed", Term: t, Text: s})
+ }
+ }
+ }
+ }
+ low := strings.ToLower(c.Prompt)
+ for _, t := range promptDenylist {
+ if containsTerm(low, t) {
+ out = append(out, Finding{Where: "prompt", Term: t, Text: c.Prompt})
+ }
+ }
+ sort.Slice(out, func(i, j int) bool { return out[i].String() < out[j].String() })
+ return out
+}
+
+var wordCache sync.Map // term -> *regexp.Regexp
+
+// containsTerm matches a term on word boundaries when it starts and ends
+// with a letter, so "eval" does not match "evaluate" but does match "eval".
+func containsTerm(s, term string) bool {
+ if !strings.Contains(s, term) {
+ return false
+ }
+ isWord := func(b byte) bool { return b >= 'a' && b <= 'z' }
+ if !isWord(term[0]) || !isWord(term[len(term)-1]) {
+ return true
+ }
+ re, ok := wordCache.Load(term)
+ if !ok {
+ re, _ = wordCache.LoadOrStore(term, regexp.MustCompile(`\b`+regexp.QuoteMeta(term)+`\b`))
+ }
+ return re.(*regexp.Regexp).MatchString(s)
+}
+
+func clipS(s string, n int) string {
+ if len(s) > n {
+ return s[:n] + "..."
+ }
+ return s
+}
diff --git a/cli/test/evals/internal/grade/trajectory.go b/cli/test/evals/internal/grade/trajectory.go
new file mode 100644
index 00000000..1f9623d1
--- /dev/null
+++ b/cli/test/evals/internal/grade/trajectory.go
@@ -0,0 +1,292 @@
+package grade
+
+import (
+ "encoding/json"
+ "fmt"
+ "path"
+ "regexp"
+ "sort"
+ "strconv"
+ "strings"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// Trajectory checks the tool calls against the case's call DSL:
+//
+// required each matcher matches at least one operation
+// any_of at least one group is fully satisfied
+// forbidden no operation matches
+// order the first match of A precedes the first match of B
+// max_calls, max_failed_calls, max_identical_repeats thrash detection
+//
+// An op pattern is an exact op, "*", a prefix ("endpoints.*"), or a file op
+// with a path glob ("edit_file:handlers/*.go").
+func Trajectory(run *record.Run, t *catalog.Trajectory) *Verdict {
+ v := newVerdict("trajectory", true)
+ all := run.AllOps()
+ for _, m := range t.Required {
+ if firstMatch(all, m) < 0 {
+ v.fail("required call missing: %s", describe(m))
+ }
+ }
+ if len(t.AnyOf) > 0 {
+ ok := false
+ for _, group := range t.AnyOf {
+ hit := true
+ for _, m := range group {
+ if firstMatch(all, m) < 0 {
+ hit = false
+ break
+ }
+ }
+ if hit {
+ ok = true
+ break
+ }
+ }
+ if !ok {
+ var groups []string
+ for _, g := range t.AnyOf {
+ var parts []string
+ for _, m := range g {
+ parts = append(parts, describe(m))
+ }
+ groups = append(groups, "["+strings.Join(parts, " + ")+"]")
+ }
+ v.fail("none of these alternatives happened: %s", strings.Join(groups, " or "))
+ }
+ }
+ for _, m := range t.Forbidden {
+ if i := firstMatch(all, m); i >= 0 {
+ v.fail("forbidden call %s happened in %s: %s", describe(m), all[i].Call.ID, opString(all[i].OpRecord))
+ }
+ }
+ for _, pair := range t.Order {
+ if len(pair) != 2 {
+ v.fail("order entries need exactly two ops, got %v", pair)
+ continue
+ }
+ a := firstMatch(all, catalog.CallMatcher{Op: pair[0]})
+ b := firstMatch(all, catalog.CallMatcher{Op: pair[1]})
+ switch {
+ case b < 0:
+ case a < 0:
+ v.fail("%s happened without %s before it", pair[1], pair[0])
+ case a > b:
+ v.fail("%s (%s) came after %s (%s)", pair[0], all[a].Call.ID, pair[1], all[b].Call.ID)
+ }
+ }
+ executed := 0
+ failed := 0
+ for _, c := range run.Calls {
+ if c.Rejected || isSubmitTool(c.Tool) {
+ continue
+ }
+ executed++
+ if !c.OK || !anyOpOK(c) {
+ failed++
+ }
+ }
+ if t.MaxCalls > 0 && executed > t.MaxCalls {
+ v.fail("%d tool calls, more than max_calls %d", executed, t.MaxCalls)
+ }
+ if t.MaxFailedCalls > 0 && failed > t.MaxFailedCalls {
+ v.fail("%d failed tool calls, more than max_failed_calls %d", failed, t.MaxFailedCalls)
+ }
+ if t.MaxIdenticalRepeats > 0 {
+ counts := map[string]int{}
+ for _, op := range all {
+ counts[opKey(op.OpRecord)]++
+ }
+ keys := make([]string, 0, len(counts))
+ for k := range counts {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ for _, k := range keys {
+ if counts[k] > t.MaxIdenticalRepeats {
+ v.fail("the same call ran %d times (max_identical_repeats %d): %s", counts[k], t.MaxIdenticalRepeats, k)
+ }
+ }
+ }
+ v.note("%d operations in %d calls (%d failed)", len(all), executed, failed)
+ return v
+}
+
+func isSubmitTool(name string) bool {
+ return name == "submit_perf_diagnosis" || name == "submit_debug_diagnosis" || name == "finish"
+}
+
+func firstMatch(all []record.CallOp, m catalog.CallMatcher) int {
+ for i, op := range all {
+ if matches(op.OpRecord, m) {
+ return i
+ }
+ }
+ return -1
+}
+
+// Matches reports whether an operation satisfies a matcher.
+func Matches(op *record.OpRecord, m catalog.CallMatcher) bool { return matches(op, m) }
+
+func matches(op *record.OpRecord, m catalog.CallMatcher) bool {
+ pattern, glob, _ := strings.Cut(m.Op, ":")
+ if !opPattern(pattern, op.Op.Op) {
+ return false
+ }
+ if glob != "" {
+ var target string
+ if p, ok := op.Args["path"].(string); ok {
+ target = p
+ } else if n, ok := op.Args["name"].(string); ok {
+ target = n
+ }
+ if ok, _ := path.Match(glob, target); !ok && !strings.HasSuffix(target, strings.TrimPrefix(glob, "*")) {
+ return false
+ }
+ }
+ if m.Write != nil && op.Write != *m.Write {
+ return false
+ }
+ if m.OK != nil && op.OK != *m.OK {
+ return false
+ }
+ for k, want := range m.Args {
+ got, present := op.Args[k]
+ if !argMatches(got, present, want) {
+ return false
+ }
+ }
+ return true
+}
+
+func opPattern(p, op string) bool {
+ switch {
+ case p == "*" || p == "":
+ return true
+ case strings.HasSuffix(p, ".*"):
+ return strings.HasPrefix(op, strings.TrimSuffix(p, "*"))
+ default:
+ return p == op
+ }
+}
+
+// argMatches implements the arg matchers: present, absent, ~regex, exact
+// value, and {min, max}.
+func argMatches(got any, present bool, want any) bool {
+ switch w := want.(type) {
+ case string:
+ switch {
+ case w == "present":
+ return present && fmt.Sprint(got) != ""
+ case w == "absent":
+ return !present
+ case strings.HasPrefix(w, "~"):
+ re, err := regexp.Compile(w[1:])
+ return err == nil && present && re.MatchString(argString(got))
+ default:
+ return present && argString(got) == w
+ }
+ case map[string]any:
+ n, ok := toFloat(got)
+ if !ok {
+ return false
+ }
+ if lo, ok := toFloat(w["min"]); ok && n < lo {
+ return false
+ }
+ if hi, ok := toFloat(w["max"]); ok && n > hi {
+ return false
+ }
+ return present
+ case bool:
+ b, _ := strconv.ParseBool(argString(got))
+ if gb, ok := got.(bool); ok {
+ b = gb
+ }
+ return present && b == w
+ default:
+ return present && argString(got) == argString(w)
+ }
+}
+
+func argString(v any) string {
+ switch t := v.(type) {
+ case nil:
+ return ""
+ case string:
+ return t
+ case float64:
+ return strconv.FormatFloat(t, 'f', -1, 64)
+ case []any:
+ parts := make([]string, len(t))
+ for i, x := range t {
+ parts[i] = argString(x)
+ }
+ return strings.Join(parts, ",")
+ case map[string]any:
+ raw, _ := json.Marshal(t)
+ return string(raw)
+ default:
+ return fmt.Sprint(t)
+ }
+}
+
+func toFloat(v any) (float64, bool) {
+ switch t := v.(type) {
+ case float64:
+ return t, true
+ case int:
+ return float64(t), true
+ case string:
+ f, err := strconv.ParseFloat(t, 64)
+ return f, err == nil
+ }
+ return 0, false
+}
+
+func describe(m catalog.CallMatcher) string {
+ s := m.Op
+ if s == "" {
+ s = "*"
+ }
+ var parts []string
+ keys := make([]string, 0, len(m.Args))
+ for k := range m.Args {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ for _, k := range keys {
+ parts = append(parts, fmt.Sprintf("%s=%v", k, m.Args[k]))
+ }
+ if m.Write != nil {
+ parts = append(parts, fmt.Sprintf("write=%v", *m.Write))
+ }
+ if len(parts) > 0 {
+ s += "(" + strings.Join(parts, ", ") + ")"
+ }
+ return s
+}
+
+func opString(op *record.OpRecord) string {
+ if len(op.Argv) > 0 {
+ return strings.Join(op.Argv, " ")
+ }
+ raw, _ := json.Marshal(op.Args)
+ return op.Op.Op + " " + string(raw)
+}
+
+// opKey identifies a call for repeat detection, ignoring output-format flags.
+func opKey(op *record.OpRecord) string {
+ args := map[string]any{}
+ for k, v := range op.Args {
+ if k == "output" || k == "fields" {
+ continue
+ }
+ args[k] = v
+ }
+ raw, _ := json.Marshal(args)
+ return op.Op.Op + " " + string(raw)
+}
diff --git a/cli/test/evals/internal/ops/ops.go b/cli/test/evals/internal/ops/ops.go
new file mode 100644
index 00000000..9be178f5
--- /dev/null
+++ b/cli/test/evals/internal/ops/ops.go
@@ -0,0 +1,399 @@
+// Package ops maps CLI invocations and MCP tool calls onto surface-neutral
+// operations (endpoints.show, pages.ack, ...) so one case spec grades both.
+package ops
+
+import (
+ "fmt"
+ "regexp"
+ "sort"
+ "strings"
+ "sync"
+
+ "github.com/tracewayapp/traceway/cli/pkg/mcpserver/knowledge"
+)
+
+// An operation is the surface-neutral name of what a tool call did
+// (endpoints.show, pages.ack, read_file, ...), so one case spec grades CLI and
+// MCP runs alike. Args are keyed by the MCP parameter name, which is the CLI
+// flag in snake_case except where Spec.Rename says otherwise.
+
+// Spec describes one Traceway operation on both surfaces.
+type Spec struct {
+ Op string // endpoints.show
+ CLI []string // command path after the binary: [endpoints show]
+ MCP string // MCP tool name; empty for CLI-only commands
+ Write bool
+ // Positional names the CLI positional args in order; a trailing "..."
+ // collects the rest into a list (exceptions archive <hash>...).
+ Positional []string
+ // Rename maps a snake_case CLI flag to its MCP parameter.
+ Rename map[string]string
+ // WriteEdition marks commands that exist only in traceway-write.
+ WriteEdition bool
+}
+
+// specs is the static half of the operations map: positional names, renames
+// and write flags. The CLI<->MCP pairing itself is parsed from tool-map.md and
+// cross-checked against this table by TestOpsMatchToolMap.
+var specs = []Spec{
+ {Op: "projects.list", CLI: []string{"projects", "list"}, MCP: "list_projects"},
+ {Op: "projects.use", CLI: []string{"projects", "use"}, Positional: []string{"project_id"}},
+ {Op: "exceptions.list", CLI: []string{"exceptions", "list"}, MCP: "list_exceptions"},
+ {Op: "exceptions.show", CLI: []string{"exceptions", "show"}, MCP: "get_exception", Positional: []string{"hash"}},
+ {Op: "exceptions.occurrence", CLI: []string{"exceptions", "occurrence"}, MCP: "get_exception_occurrence", Positional: []string{"id"}},
+ {Op: "logs.query", CLI: []string{"logs", "query"}, MCP: "query_logs", Rename: map[string]string{"service": "service_name"}},
+ {Op: "endpoints.list", CLI: []string{"endpoints", "list"}, MCP: "list_endpoints"},
+ {Op: "endpoints.show", CLI: []string{"endpoints", "show"}, MCP: "get_endpoint_request", Positional: []string{"id"}},
+ {Op: "endpoints.chart", CLI: []string{"endpoints", "chart"}, MCP: "endpoints_chart"},
+ {Op: "endpoints.slow", CLI: []string{"endpoints", "slow"}, MCP: "get_slow_endpoint_config", Positional: []string{"endpoint"}},
+ {Op: "tasks.show", CLI: []string{"tasks", "show"}, MCP: "get_task", Positional: []string{"id"}},
+ {Op: "ai_traces.show", CLI: []string{"ai-traces", "show"}, MCP: "get_ai_trace", Positional: []string{"id"}},
+ {Op: "sessions.show", CLI: []string{"sessions", "show"}, MCP: "get_session", Positional: []string{"id"}},
+ {Op: "traces.show", CLI: []string{"traces", "show"}, MCP: "get_trace", Positional: []string{"id"}},
+ {Op: "metrics.query", CLI: []string{"metrics", "query"}, MCP: "query_metrics", Rename: map[string]string{"tag": "tag_filters"}},
+ {Op: "pages.list", CLI: []string{"pages", "list"}, MCP: "list_pages"},
+ {Op: "pages.show", CLI: []string{"pages", "show"}, MCP: "get_page", Positional: []string{"id"}},
+ {Op: "alerts.list", CLI: []string{"alerts", "list"}, MCP: "list_notification_rules"},
+ {Op: "monitors.list", CLI: []string{"monitors", "list"}, MCP: "list_checks"},
+ {Op: "monitors.show", CLI: []string{"monitors", "show"}, MCP: "get_check", Positional: []string{"id"}},
+ {Op: "post_mortems.list", CLI: []string{"post-mortems", "list"}, MCP: "list_post_mortems", Rename: map[string]string{"tag": "tags"}},
+ {Op: "post_mortems.show", CLI: []string{"post-mortems", "show"}, MCP: "get_post_mortem", Positional: []string{"id"}},
+
+ {Op: "exceptions.archive", CLI: []string{"exceptions", "archive"}, MCP: "archive_exceptions", Write: true, WriteEdition: true, Positional: []string{"hashes..."}},
+ {Op: "exceptions.unarchive", CLI: []string{"exceptions", "unarchive"}, MCP: "unarchive_exceptions", Write: true, WriteEdition: true, Positional: []string{"hashes..."}},
+ {Op: "pages.ack", CLI: []string{"pages", "ack"}, MCP: "acknowledge_page", Write: true, WriteEdition: true, Positional: []string{"id"}},
+ {Op: "pages.resolve", CLI: []string{"pages", "resolve"}, MCP: "resolve_page", Write: true, WriteEdition: true, Positional: []string{"id"}},
+ {Op: "alerts.enable", CLI: []string{"alerts", "enable"}, MCP: "set_notification_rule_enabled", Write: true, WriteEdition: true, Positional: []string{"id"}},
+ {Op: "alerts.disable", CLI: []string{"alerts", "disable"}, MCP: "set_notification_rule_enabled", Write: true, WriteEdition: true, Positional: []string{"id"}},
+ {Op: "alerts.snooze", CLI: []string{"alerts", "snooze"}, MCP: "snooze_notification_rule", Write: true, WriteEdition: true, Positional: []string{"id"}, Rename: map[string]string{"minutes": "duration_minutes"}},
+ {Op: "monitors.run", CLI: []string{"monitors", "run"}, MCP: "run_check", Write: true, WriteEdition: true, Positional: []string{"id"}},
+ {Op: "post_mortems.create", CLI: []string{"post-mortems", "create"}, MCP: "create_post_mortem", Write: true, WriteEdition: true, Rename: map[string]string{"tag": "tags", "incident": "incident_id", "content_file": "content_md"}},
+ {Op: "post_mortems.update", CLI: []string{"post-mortems", "update"}, MCP: "update_post_mortem", Write: true, WriteEdition: true, Positional: []string{"id"}, Rename: map[string]string{"tag": "tags", "incident": "incident_id", "content_file": "content_md"}},
+ {Op: "endpoints.mark_slow", CLI: []string{"endpoints", "mark-slow"}, MCP: "set_slow_endpoint", Write: true, WriteEdition: true, Positional: []string{"endpoint"}},
+
+ // CLI-only commands. setup apply proposes a plan draft; nothing is
+ // created until a person approves it, so it is not an agent write.
+ {Op: "setup.apply", CLI: []string{"setup", "apply"}},
+ {Op: "login", CLI: []string{"login"}},
+ {Op: "logout", CLI: []string{"logout"}},
+ {Op: "profiles.list", CLI: []string{"profiles", "list"}},
+ {Op: "profiles.use", CLI: []string{"profiles", "use"}, Positional: []string{"profile"}},
+ {Op: "version", CLI: []string{"version"}},
+ {Op: "mcp.serve", CLI: []string{"mcp"}},
+}
+
+// Harness tool operations that are not Traceway commands.
+const (
+ OpReadFile = "read_file"
+ OpListFiles = "list_files"
+ OpSearch = "search"
+ OpEditFile = "edit_file"
+ OpWriteFile = "write_file"
+ OpLoadSkill = "load_skill"
+ OpAskUser = "ask_user"
+ OpReadResource = "read_resource"
+ OpListRes = "list_resources"
+ OpShell = "shell"
+ OpGit = "git"
+ OpSubmit = "submit"
+ OpUnknownCLI = "traceway.unknown"
+)
+
+// boolFlags lists CLI flags that take no value.
+var boolFlags = map[string]bool{
+ "yes": true, "include_archived": true, "archive_issue": true, "unlink_incident": true,
+ "password": true, "password_stdin": true, "token_stdin": true, "no_browser": true,
+ "help": true, "version": true,
+}
+
+var shortFlags = map[string]string{"o": "output", "h": "help", "v": "version"}
+
+var (
+ opsOnce sync.Once
+ byOp map[string]*Spec
+ byMCP map[string][]*Spec
+ cliSorted []*Spec
+)
+
+func initOps() {
+ opsOnce.Do(func() {
+ byOp = map[string]*Spec{}
+ byMCP = map[string][]*Spec{}
+ for i := range specs {
+ s := &specs[i]
+ byOp[s.Op] = s
+ if s.MCP != "" {
+ byMCP[s.MCP] = append(byMCP[s.MCP], s)
+ }
+ cliSorted = append(cliSorted, s)
+ }
+ sort.SliceStable(cliSorted, func(i, j int) bool { return len(cliSorted[i].CLI) > len(cliSorted[j].CLI) })
+ })
+}
+
+// Specs returns the operations table.
+func Specs() []Spec { return specs }
+
+// SpecFor returns an operation's spec.
+func SpecFor(op string) (*Spec, bool) {
+ initOps()
+ s, ok := byOp[op]
+ return s, ok
+}
+
+// IsWriteOp reports whether an op mutates Traceway state.
+func IsWriteOp(op string) bool {
+ s, ok := SpecFor(op)
+ return ok && s.Write
+}
+
+// Op is one recorded operation.
+type Op struct {
+ Op string `json:"op"`
+ Args map[string]any `json:"args,omitempty"`
+ Write bool `json:"write,omitempty"`
+}
+
+// ParseCLI maps a traceway[-write] argv (without the binary) to an op.
+func ParseCLI(argv []string) Op {
+ initOps()
+ var words []string
+ flags := map[string]any{}
+ for i := 0; i < len(argv); i++ {
+ a := argv[i]
+ if a == "--" {
+ words = append(words, argv[i+1:]...)
+ break
+ }
+ if strings.HasPrefix(a, "-") && len(a) > 1 {
+ name, val, hasVal := strings.Cut(strings.TrimLeft(a, "-"), "=")
+ if !strings.HasPrefix(a, "--") {
+ if long, ok := shortFlags[name]; ok {
+ name = long
+ }
+ }
+ key := strings.ReplaceAll(name, "-", "_")
+ if !hasVal && !boolFlags[key] && i+1 < len(argv) {
+ val = argv[i+1]
+ i++
+ }
+ if !hasVal && boolFlags[key] {
+ addArg(flags, key, true)
+ continue
+ }
+ addArg(flags, key, val)
+ continue
+ }
+ words = append(words, a)
+ }
+ var spec *Spec
+ for _, s := range cliSorted {
+ if len(words) >= len(s.CLI) && equalWords(words[:len(s.CLI)], s.CLI) {
+ spec = s
+ break
+ }
+ }
+ if spec == nil {
+ return Op{Op: OpUnknownCLI, Args: map[string]any{"argv": strings.Join(argv, " ")}}
+ }
+ args := map[string]any{}
+ for k, v := range flags {
+ if r, ok := spec.Rename[k]; ok {
+ k = r
+ }
+ if k == "project" {
+ k = "project_id"
+ }
+ args[k] = v
+ }
+ pos := words[len(spec.CLI):]
+ for i, name := range spec.Positional {
+ if strings.HasSuffix(name, "...") {
+ if i < len(pos) {
+ list := make([]any, 0, len(pos)-i)
+ for _, p := range pos[i:] {
+ list = append(list, p)
+ }
+ args[strings.TrimSuffix(name, "...")] = list
+ }
+ break
+ }
+ if i < len(pos) {
+ args[name] = pos[i]
+ }
+ }
+ return Op{Op: spec.Op, Args: args, Write: spec.Write}
+}
+
+func addArg(m map[string]any, k string, v any) {
+ prev, ok := m[k]
+ if !ok {
+ m[k] = v
+ return
+ }
+ switch p := prev.(type) {
+ case []any:
+ m[k] = append(p, v)
+ default:
+ m[k] = []any{p, v}
+ }
+}
+
+func equalWords(a, b []string) bool {
+ for i := range a {
+ if a[i] != b[i] {
+ return false
+ }
+ }
+ return true
+}
+
+// ToolMapPair is one row of tool-map.md.
+type ToolMapPair struct {
+ CLI []string
+ MCP string
+}
+
+var (
+ toolMapRowRe = regexp.MustCompile("^\\|\\s*(`[^|]+`(?:\\s*/\\s*`[^|]+`)*)\\s*\\|\\s*(`[^|]+`(?:\\s*/\\s*`[^|]+`)*)\\s*\\|")
+ codeRe = regexp.MustCompile("`([^`]+)`")
+)
+
+// ParseToolMap reads the CLI->MCP pairs from tool-map.md. A row like
+// "`pages list` / `pages show <id>` | `list_pages` / `get_page`" yields two
+// pairs. Commands that elide the binary and noun ("`unarchive`") inherit them
+// from the previous command in the cell.
+func ParseToolMap() []ToolMapPair {
+ var out []ToolMapPair
+ for _, line := range strings.Split(knowledge.MustRead("tool-map.md"), "\n") {
+ m := toolMapRowRe.FindStringSubmatch(line)
+ if m == nil {
+ continue
+ }
+ clis := codeRe.FindAllStringSubmatch(m[1], -1)
+ tools := codeRe.FindAllStringSubmatch(m[2], -1)
+ var prev []string
+ var cmds [][]string
+ for _, c := range clis {
+ words := commandWords(c[1])
+ if len(words) == 0 {
+ continue
+ }
+ if len(words) == 1 && len(prev) >= 1 {
+ words = append(append([]string{}, prev[:len(prev)-1]...), words[0])
+ } else if len(words) >= 1 && words[0] != prev0(prev) && len(prev) > 0 && !isNoun(words[0]) {
+ words = append([]string{prev[0]}, words...)
+ }
+ prev = words
+ cmds = append(cmds, words)
+ }
+ for i, c := range cmds {
+ t := tools[len(tools)-1][1]
+ if i < len(tools) {
+ t = tools[i][1]
+ }
+ out = append(out, ToolMapPair{CLI: c, MCP: t})
+ }
+ }
+ return out
+}
+
+func prev0(p []string) string {
+ if len(p) == 0 {
+ return ""
+ }
+ return p[0]
+}
+
+var nouns = map[string]bool{
+ "projects": true, "exceptions": true, "logs": true, "endpoints": true, "tasks": true, "ai-traces": true,
+ "sessions": true, "traces": true, "metrics": true, "pages": true, "alerts": true, "monitors": true, "post-mortems": true,
+}
+
+func isNoun(w string) bool { return nouns[w] }
+
+// commandWords strips the binary name, flags and placeholders from a code span.
+func commandWords(s string) []string {
+ var out []string
+ for _, w := range strings.Fields(s) {
+ if w == "traceway" || w == "traceway-write" {
+ continue
+ }
+ if strings.HasPrefix(w, "-") || strings.HasPrefix(w, "<") || w == "N" {
+ break
+ }
+ out = append(out, w)
+ }
+ return out
+}
+
+// RenderCLI renders an op back into a traceway argv (used by the fake model
+// to play surface-neutral oracle steps on CLI arms).
+func RenderCLI(op string, args map[string]any) ([]string, error) {
+ spec, ok := SpecFor(op)
+ if !ok {
+ return nil, fmt.Errorf("unknown op %q", op)
+ }
+ argv := append([]string{}, spec.CLI...)
+ used := map[string]bool{}
+ for _, name := range spec.Positional {
+ key := strings.TrimSuffix(name, "...")
+ v, ok := args[key]
+ if !ok {
+ continue
+ }
+ used[key] = true
+ if list, ok := v.([]any); ok {
+ for _, x := range list {
+ argv = append(argv, fmt.Sprint(x))
+ }
+ } else {
+ argv = append(argv, fmt.Sprint(v))
+ }
+ }
+ reverse := map[string]string{}
+ for cli, mcp := range spec.Rename {
+ reverse[mcp] = cli
+ }
+ keys := make([]string, 0, len(args))
+ for k := range args {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ for _, k := range keys {
+ if used[k] {
+ continue
+ }
+ v := args[k]
+ flag := k
+ if c, ok := reverse[k]; ok {
+ flag = c
+ }
+ if k == "project_id" {
+ flag = "project"
+ }
+ flag = "--" + strings.ReplaceAll(flag, "_", "-")
+ switch t := v.(type) {
+ case bool:
+ if t {
+ argv = append(argv, flag)
+ }
+ case []any:
+ for _, x := range t {
+ argv = append(argv, flag, fmt.Sprint(x))
+ }
+ case map[string]any:
+ mk := make([]string, 0, len(t))
+ for kk := range t {
+ mk = append(mk, kk)
+ }
+ sort.Strings(mk)
+ for _, kk := range mk {
+ argv = append(argv, flag, kk+"="+fmt.Sprint(t[kk]))
+ }
+ default:
+ argv = append(argv, flag, fmt.Sprint(v))
+ }
+ }
+ return argv, nil
+}
diff --git a/cli/test/evals/internal/ops/ops_test.go b/cli/test/evals/internal/ops/ops_test.go
new file mode 100644
index 00000000..e4f6a7a9
--- /dev/null
+++ b/cli/test/evals/internal/ops/ops_test.go
@@ -0,0 +1,117 @@
+package ops
+
+import (
+ "os/exec"
+ "reflect"
+ "strings"
+ "testing"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+)
+
+// TestOpsMatchToolMap cross-checks the static table against tool-map.md,
+// the published CLI-to-MCP translation table.
+func TestOpsMatchToolMap(t *testing.T) {
+ pairs := ParseToolMap()
+ if len(pairs) < 25 {
+ t.Fatalf("parsed only %d rows from tool-map.md", len(pairs))
+ }
+ for _, p := range pairs {
+ op := ParseCLI(p.CLI)
+ spec, ok := SpecFor(op.Op)
+ if !ok {
+ t.Errorf("tool-map.md row %v -> %s maps to no op (%s)", p.CLI, p.MCP, op.Op)
+ continue
+ }
+ if spec.MCP != p.MCP {
+ t.Errorf("%v: tool-map.md says %s, the ops table says %s", p.CLI, p.MCP, spec.MCP)
+ }
+ }
+}
+
+// TestEveryCLICommandMaps walks both built editions' help: every leaf
+// command must map to an op, and write-edition commands must be writes.
+func TestEveryCLICommandMaps(t *testing.T) {
+ bins := env.BinariesForTest(t)
+ for _, bin := range []string{bins.CLI, bins.CLIWrite} {
+ for _, path := range leafCommands(t, bin, nil) {
+ op := ParseCLI(path)
+ if op.Op == OpUnknownCLI {
+ t.Errorf("%s %v maps to no op", bin, path)
+ }
+ }
+ }
+}
+
+func leafCommands(t *testing.T, bin string, prefix []string) [][]string {
+ out, _ := exec.Command(bin, append(append([]string{}, prefix...), "--help")...).CombinedOutput()
+ var subs []string
+ in := false
+ for _, line := range strings.Split(string(out), "\n") {
+ if strings.HasPrefix(line, "Available Commands:") {
+ in = true
+ continue
+ }
+ if in {
+ if strings.TrimSpace(line) == "" {
+ break
+ }
+ name := strings.Fields(line)[0]
+ if name != "help" && name != "completion" {
+ subs = append(subs, name)
+ }
+ }
+ }
+ if len(subs) == 0 {
+ if len(prefix) == 0 {
+ t.Fatalf("%s --help listed no commands", bin)
+ }
+ return [][]string{prefix}
+ }
+ var leaves [][]string
+ for _, s := range subs {
+ leaves = append(leaves, leafCommands(t, bin, append(append([]string{}, prefix...), s))...)
+ }
+ return leaves
+}
+
+func TestParseCLI(t *testing.T) {
+ cases := []struct {
+ argv string
+ want Op
+ }{
+ {"endpoints show 7e3 --recorded-at 2026-10-17T10:00:00Z -o json", Op{Op: "endpoints.show", Args: map[string]any{"id": "7e3", "recorded_at": "2026-10-17T10:00:00Z", "output": "json"}}},
+ {"exceptions archive aa bb --yes", Op{Op: "exceptions.archive", Args: map[string]any{"hashes": []any{"aa", "bb"}, "yes": true}, Write: true}},
+ {"logs query --service api --min-severity=17", Op{Op: "logs.query", Args: map[string]any{"service_name": "api", "min_severity": "17"}}},
+ {"metrics query --name x --tag a=b --tag c=d", Op{Op: "metrics.query", Args: map[string]any{"name": "x", "tag_filters": []any{"a=b", "c=d"}}}},
+ {"endpoints mark-slow 'GET /x' --offset-ms 200", Op{Op: "endpoints.mark_slow", Args: map[string]any{"endpoint": "'GET", "offset_ms": "200"}, Write: true}},
+ {"post-mortems update 4 --content-file - --expected-updated-at T", Op{Op: "post_mortems.update", Args: map[string]any{"id": "4", "content_md": "-", "expected_updated_at": "T"}, Write: true}},
+ }
+ for _, c := range cases {
+ got := ParseCLI(strings.Fields(c.argv))
+ if !reflect.DeepEqual(got, c.want) {
+ t.Errorf("%s:\n got %+v\nwant %+v", c.argv, got, c.want)
+ }
+ }
+}
+
+func TestRenderCLI(t *testing.T) {
+ argv, err := RenderCLI("alerts.snooze", map[string]any{"id": 12, "duration_minutes": 60})
+ if err != nil || strings.Join(argv, " ") != "alerts snooze 12 --minutes 60" {
+ t.Fatalf("RenderCLI = %v, %v", argv, err)
+ }
+}
+
+func TestInvocationsUnquote(t *testing.T) {
+ inv, err := Invocations(`traceway logs query --search "/api/orders " --since 6h | jq -r '.data[] | .body'; echo a\ b`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(inv) != 3 || inv[0][4] != "/api/orders " || inv[1][2] != ".data[] | .body" || inv[2][1] != "a b" {
+ t.Fatalf("%q", inv)
+ }
+ op := ParseCLI(inv[0][1:])
+ if op.Args["search"] != "/api/orders " {
+ t.Fatalf("%+v", op)
+ }
+}
diff --git a/cli/test/evals/internal/ops/words.go b/cli/test/evals/internal/ops/words.go
new file mode 100644
index 00000000..36af9c0a
--- /dev/null
+++ b/cli/test/evals/internal/ops/words.go
@@ -0,0 +1,81 @@
+package ops
+
+import (
+ "bytes"
+ "strings"
+
+ "mvdan.cc/sh/v3/syntax"
+)
+
+// Invocations parses a shell command line and returns the words of every
+// simple command in it, with quoting removed ("a b" and 'a b' become a b).
+// Words that need expansion ($VAR, $(...)) keep their source text.
+func Invocations(cmdline string) ([][]string, error) {
+ file, err := syntax.NewParser().Parse(strings.NewReader(cmdline), "")
+ if err != nil {
+ return nil, err
+ }
+ var out [][]string
+ syntax.Walk(file, func(n syntax.Node) bool {
+ ce, ok := n.(*syntax.CallExpr)
+ if !ok || len(ce.Args) == 0 {
+ return true
+ }
+ words := make([]string, 0, len(ce.Args))
+ for _, w := range ce.Args {
+ words = append(words, WordText(w))
+ }
+ out = append(out, words)
+ return true
+ })
+ return out, nil
+}
+
+// WordText is a word's value with quotes and backslash escapes removed.
+func WordText(w *syntax.Word) string {
+ var b strings.Builder
+ for _, p := range w.Parts {
+ b.WriteString(partText(p, false))
+ }
+ return b.String()
+}
+
+func partText(p syntax.WordPart, inDouble bool) string {
+ switch t := p.(type) {
+ case *syntax.Lit:
+ return unescape(t.Value, inDouble)
+ case *syntax.SglQuoted:
+ return t.Value
+ case *syntax.DblQuoted:
+ var b strings.Builder
+ for _, q := range t.Parts {
+ b.WriteString(partText(q, true))
+ }
+ return b.String()
+ default:
+ var buf bytes.Buffer
+ _ = syntax.NewPrinter().Print(&buf, p)
+ return buf.String()
+ }
+}
+
+func unescape(s string, inDouble bool) string {
+ if !strings.Contains(s, `\`) {
+ return s
+ }
+ var b strings.Builder
+ for i := 0; i < len(s); i++ {
+ if s[i] == '\\' && i+1 < len(s) {
+ next := s[i+1]
+ if !inDouble || strings.IndexByte("\"\\$`\n", next) >= 0 {
+ if next != '\n' {
+ b.WriteByte(next)
+ }
+ i++
+ continue
+ }
+ }
+ b.WriteByte(s[i])
+ }
+ return b.String()
+}
diff --git a/cli/test/evals/internal/record/record.go b/cli/test/evals/internal/record/record.go
new file mode 100644
index 00000000..97d2e9da
--- /dev/null
+++ b/cli/test/evals/internal/record/record.go
@@ -0,0 +1,113 @@
+// Package record holds what one eval run produced: every tool call with the
+// operations it performed, the final answer, and usage. The agent loop writes
+// it, the graders read it, and transcript.jsonl is its serialized form.
+package record
+
+import (
+ "encoding/json"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"
+)
+
+// OpRecord is one operation inside a tool call: a traceway invocation in a
+// shell pipeline, an MCP tool call, a file read, ...
+type OpRecord struct {
+ ops.Op
+ Argv []string `json:"argv,omitempty"`
+ Exit int `json:"exit"`
+ OK bool `json:"ok"`
+ Output string `json:"output,omitempty"` // raw output before any pipe
+ Truncated bool `json:"truncated,omitempty"`
+}
+
+// Call is one tool call the model made. Its ID ("c07") is what the model
+// cites as evidence.
+type Call struct {
+ ID string `json:"id"`
+ Turn int `json:"turn"`
+ Tool string `json:"tool"`
+ Args json.RawMessage `json:"args"`
+ Output string `json:"output"` // what the model saw
+ OK bool `json:"ok"`
+ Truncated bool `json:"truncated,omitempty"`
+ Ops []OpRecord `json:"ops,omitempty"`
+ DurationMs int64 `json:"durationMs"`
+ Rejected bool `json:"rejected,omitempty"` // schema/budget refusal, not executed
+}
+
+// Usage is token and cost accounting for one model response.
+type Usage struct {
+ InputTokens int `json:"inputTokens"`
+ CachedTokens int `json:"cachedTokens"`
+ CacheWriteTokens int `json:"cacheWriteTokens"`
+ OutputTokens int `json:"outputTokens"`
+ ReasoningTokens int `json:"reasoningTokens"`
+ CostUSD float64 `json:"costUsd"`
+}
+
+// Add accumulates u2 into u.
+func (u *Usage) Add(u2 Usage) {
+ u.InputTokens += u2.InputTokens
+ u.CachedTokens += u2.CachedTokens
+ u.CacheWriteTokens += u2.CacheWriteTokens
+ u.OutputTokens += u2.OutputTokens
+ u.ReasoningTokens += u2.ReasoningTokens
+ u.CostUSD += u2.CostUSD
+}
+
+// Question is one ask_user exchange.
+type Question struct {
+ CallID string `json:"callId"`
+ Question string `json:"question"`
+ Answer string `json:"answer"`
+}
+
+// Run is the complete record of one agent run.
+type Run struct {
+ Calls []*Call `json:"calls"`
+ Submission map[string]any `json:"submission,omitempty"`
+ SubmitTool string `json:"submitTool,omitempty"`
+ SubmitCall string `json:"submitCall,omitempty"`
+ FinalText string `json:"finalText,omitempty"`
+ Questions []Question `json:"questions,omitempty"`
+ Usage Usage `json:"usage"`
+ Turns int `json:"turns"`
+ StopReason string `json:"stopReason"` // submitted | budget_exhausted | no_submit | infra_error
+ StopDetail string `json:"stopDetail,omitempty"`
+ SkillLoaded []string `json:"skillLoaded,omitempty"`
+ Providers []string `json:"providers,omitempty"`
+ Prompt string `json:"prompt"`
+ Started time.Time `json:"started"`
+ Elapsed time.Duration `json:"elapsed"`
+}
+
+// AllOps flattens every operation in call order, each with its call id.
+func (r *Run) AllOps() []CallOp {
+ var out []CallOp
+ for _, c := range r.Calls {
+ if c.Rejected {
+ continue
+ }
+ for i := range c.Ops {
+ out = append(out, CallOp{Call: c, OpRecord: &c.Ops[i]})
+ }
+ }
+ return out
+}
+
+// CallOp pairs an operation with the call that performed it.
+type CallOp struct {
+ Call *Call
+ *OpRecord
+}
+
+// Call returns the call with id, or nil.
+func (r *Run) CallByID(id string) *Call {
+ for _, c := range r.Calls {
+ if c.ID == id {
+ return c
+ }
+ }
+ return nil
+}
diff --git a/cli/test/evals/internal/report/aggregate.go b/cli/test/evals/internal/report/aggregate.go
new file mode 100644
index 00000000..1653f308
--- /dev/null
+++ b/cli/test/evals/internal/report/aggregate.go
@@ -0,0 +1,228 @@
+package report
+
+import (
+ "sort"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"
+)
+
+// Rate is a pass count.
+type Rate struct {
+ Pass int `json:"pass"`
+ Total int `json:"total"`
+ Rate float64 `json:"rate"`
+}
+
+func (r *Rate) add(pass bool) {
+ r.Total++
+ if pass {
+ r.Pass++
+ }
+ r.Rate = float64(r.Pass) / float64(r.Total)
+}
+
+// Delta compares two arms or models.
+type Delta struct {
+ Suite string `json:"suite"`
+ Model string `json:"model,omitempty"`
+ A string `json:"a"`
+ B string `json:"b"`
+ RateA float64 `json:"rateA"`
+ RateB float64 `json:"rateB"`
+ Delta float64 `json:"delta"` // A - B in percentage points
+ N int `json:"n"`
+}
+
+// Costs aggregates spend on passing runs.
+type Costs struct {
+ PassingRuns int `json:"passingRuns"`
+ AvgCostUSD float64 `json:"avgCostUsd"`
+ AvgToolCalls float64 `json:"avgToolCalls"`
+ AvgInputTokens float64 `json:"avgInputTokens"`
+ AvgOutputTokens float64 `json:"avgOutputTokens"`
+ TotalCostUSD float64 `json:"totalCostUsd"`
+ CacheHitRate float64 `json:"cacheHitRate"`
+ AllRunsAvgCostUSD float64 `json:"allRunsAvgCostUsd"`
+ AllRuns int `json:"allRuns"`
+}
+
+// Aggregates is the summary block of results.json.
+type Aggregates struct {
+ // Cases is keyed "case|model|arm".
+ Cases map[string]*Rate `json:"cases"`
+ // Suites is keyed "suite|model|arm".
+ Suites map[string]*Rate `json:"suites"`
+ SkillUplift []Delta `json:"skillUplift"`
+ Models []Delta `json:"models"`
+ Cost map[string]*Costs `json:"cost"` // by model
+ SpendUSD float64 `json:"spendUsd"`
+ JudgeUSD float64 `json:"judgeUsd"`
+ Statuses map[string]int `json:"statuses"`
+ TopFailing []GraderCount `json:"topFailingGraders"`
+}
+
+// GraderCount counts failing graders.
+type GraderCount struct {
+ Grader string `json:"grader"`
+ Count int `json:"count"`
+}
+
+// Counted reports whether a run counts toward pass rates: infra errors are
+// excluded, budget exhaustion is a failure.
+func Counted(r *runner.Result) bool {
+ return r.Status != "infra_error"
+}
+
+// Aggregate computes pass rates and deltas.
+func Aggregate(runs []*runner.Result) Aggregates {
+ a := Aggregates{Cases: map[string]*Rate{}, Suites: map[string]*Rate{}, Cost: map[string]*Costs{}, Statuses: map[string]int{}}
+ graderFails := map[string]int{}
+ cacheIn, cacheHit := map[string]int{}, map[string]int{}
+ for _, r := range runs {
+ a.Statuses[r.Status]++
+ a.SpendUSD += r.Metrics.CostUSD + r.Metrics.JudgeCostUSD
+ a.JudgeUSD += r.Metrics.JudgeCostUSD
+ c := a.Cost[r.Model]
+ if c == nil {
+ c = &Costs{}
+ a.Cost[r.Model] = c
+ }
+ c.AllRuns++
+ c.AllRunsAvgCostUSD += r.Metrics.CostUSD
+ c.TotalCostUSD += r.Metrics.CostUSD
+ cacheIn[r.Model] += r.Metrics.InputTokens
+ cacheHit[r.Model] += r.Metrics.CachedTokens
+ if !Counted(r) {
+ continue
+ }
+ pass := r.Status == "pass"
+ key := r.Case + "|" + r.Model + "|" + r.Arm
+ if a.Cases[key] == nil {
+ a.Cases[key] = &Rate{}
+ }
+ a.Cases[key].add(pass)
+ skey := r.Suite + "|" + r.Model + "|" + r.Arm
+ if a.Suites[skey] == nil {
+ a.Suites[skey] = &Rate{}
+ }
+ a.Suites[skey].add(pass)
+ if pass {
+ c.PassingRuns++
+ c.AvgCostUSD += r.Metrics.CostUSD
+ c.AvgToolCalls += float64(r.Metrics.ToolCalls)
+ c.AvgInputTokens += float64(r.Metrics.InputTokens)
+ c.AvgOutputTokens += float64(r.Metrics.OutputTokens)
+ }
+ for _, g := range r.Graders {
+ if g.Gating && !g.Skipped && !g.Pass {
+ graderFails[g.Grader]++
+ }
+ }
+ }
+ for m, c := range a.Cost {
+ if c.PassingRuns > 0 {
+ n := float64(c.PassingRuns)
+ c.AvgCostUSD /= n
+ c.AvgToolCalls /= n
+ c.AvgInputTokens /= n
+ c.AvgOutputTokens /= n
+ }
+ if c.AllRuns > 0 {
+ c.AllRunsAvgCostUSD /= float64(c.AllRuns)
+ }
+ if cacheIn[m] > 0 {
+ c.CacheHitRate = float64(cacheHit[m]) / float64(cacheIn[m])
+ }
+ }
+ for g, n := range graderFails {
+ a.TopFailing = append(a.TopFailing, GraderCount{g, n})
+ }
+ sort.Slice(a.TopFailing, func(i, j int) bool { return a.TopFailing[i].Count > a.TopFailing[j].Count })
+
+ // Deltas are computed on cases run on both sides, so a partial
+ // selection does not compare different case sets.
+ pair := func(suite, model, armA, armB string) (Delta, bool) {
+ var ra, rb Rate
+ for key, r := range a.Cases {
+ cs, m, arm := split3(key)
+ if m != model || suiteOf(cs) != suite || arm != armA {
+ continue
+ }
+ other := a.Cases[cs+"|"+model+"|"+armB]
+ if other == nil {
+ continue
+ }
+ ra.Pass, ra.Total = ra.Pass+r.Pass, ra.Total+r.Total
+ rb.Pass, rb.Total = rb.Pass+other.Pass, rb.Total+other.Total
+ }
+ if ra.Total == 0 || rb.Total == 0 {
+ return Delta{}, false
+ }
+ pa, pb := float64(ra.Pass)/float64(ra.Total), float64(rb.Pass)/float64(rb.Total)
+ return Delta{Suite: suite, Model: model, A: armA, B: armB, RateA: pa, RateB: pb, Delta: (pa - pb) * 100, N: ra.Total}, true
+ }
+ models := map[string]bool{}
+ for _, r := range runs {
+ models[r.Model] = true
+ }
+ var modelList []string
+ for m := range models {
+ modelList = append(modelList, m)
+ }
+ sort.Strings(modelList)
+ for _, s := range catalog.Suites {
+ for _, m := range modelList {
+ if d, ok := pair(s, m, catalog.ArmCLISkill, catalog.ArmCLINoSkill); ok {
+ a.SkillUplift = append(a.SkillUplift, d)
+ }
+ }
+ // Model vs model on each arm, over the cases both ran.
+ for i := 0; i < len(modelList); i++ {
+ for j := i + 1; j < len(modelList); j++ {
+ for _, arm := range catalog.AllArms {
+ var ra, rb Rate
+ for key, r := range a.Cases {
+ cs, m, ar := split3(key)
+ if m != modelList[i] || ar != arm || suiteOf(cs) != s {
+ continue
+ }
+ if o := a.Cases[cs+"|"+modelList[j]+"|"+arm]; o != nil {
+ ra.Pass, ra.Total = ra.Pass+r.Pass, ra.Total+r.Total
+ rb.Pass, rb.Total = rb.Pass+o.Pass, rb.Total+o.Total
+ }
+ }
+ if ra.Total > 0 && rb.Total > 0 {
+ pa, pb := float64(ra.Pass)/float64(ra.Total), float64(rb.Pass)/float64(rb.Total)
+ a.Models = append(a.Models, Delta{Suite: s, A: modelList[i] + " " + arm, B: modelList[j] + " " + arm, RateA: pa, RateB: pb, Delta: (pa - pb) * 100, N: ra.Total})
+ }
+ }
+ }
+ }
+ }
+ return a
+}
+
+func split3(k string) (string, string, string) {
+ var parts [3]string
+ i := 0
+ start := 0
+ for p := 0; p < len(k) && i < 2; p++ {
+ if k[p] == '|' {
+ parts[i] = k[start:p]
+ i++
+ start = p + 1
+ }
+ }
+ parts[2] = k[start:]
+ return parts[0], parts[1], parts[2]
+}
+
+func suiteOf(caseID string) string {
+ for _, s := range catalog.Suites {
+ if len(caseID) > len(s) && caseID[:len(s)+1] == s+"-" {
+ return s
+ }
+ }
+ return ""
+}
diff --git a/cli/test/evals/internal/report/markdown.go b/cli/test/evals/internal/report/markdown.go
new file mode 100644
index 00000000..4cd3d881
--- /dev/null
+++ b/cli/test/evals/internal/report/markdown.go
@@ -0,0 +1,176 @@
+package report
+
+import (
+ "fmt"
+ "sort"
+ "strings"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"
+)
+
+// Markdown renders summary.md.
+func Markdown(r *Results) string {
+ a := r.Aggregates
+ var b strings.Builder
+ h := r.Header
+ title := "Traceway evals"
+ if h.Oracle {
+ title += " (oracle, fake model, $0)"
+ }
+ fmt.Fprintf(&b, "# %s\n\n", title)
+ fmt.Fprintf(&b, "- **Run:** `%s`, started by %s\n", h.RunID, orDash(h.StartedBy))
+ sha := h.GitSHA
+ if len(sha) > 12 {
+ sha = sha[:12]
+ }
+ dirty := ""
+ if h.GitDirty {
+ dirty = " (uncommitted changes)"
+ }
+ fmt.Fprintf(&b, "- **Commit:** `%s`%s, skill `traceway` %s, knowledge %s\n", orDash(sha), dirty, short(h.SkillHashes["traceway"]), short(h.KnowledgeHash))
+ var models []string
+ for k, v := range h.Models {
+ models = append(models, fmt.Sprintf("%s=`%s`", k, v))
+ }
+ sort.Strings(models)
+ fmt.Fprintf(&b, "- **Models:** %s\n", strings.Join(models, ", "))
+ fmt.Fprintf(&b, "- **Spend:** $%.2f (judge $%.2f) of a $%.2f cap\n", a.SpendUSD, a.JudgeUSD, h.MaxCostUSD)
+ var st []string
+ for k, v := range a.Statuses {
+ st = append(st, fmt.Sprintf("%s %d", k, v))
+ }
+ sort.Strings(st)
+ fmt.Fprintf(&b, "- **Runs:** %d (%s)\n\n", len(r.Runs), strings.Join(st, ", "))
+
+ b.WriteString("## Pass rate by suite\n\n| Suite | Model | Arm | Pass | Rate |\n|---|---|---|---|---|\n")
+ keys := sortedKeys(a.Suites)
+ for _, k := range keys {
+ s, m, arm := split3(k)
+ v := a.Suites[k]
+ fmt.Fprintf(&b, "| %s | %s | %s | %d/%d | %.0f%% |\n", s, m, arm, v.Pass, v.Total, v.Rate*100)
+ }
+ if len(a.SkillUplift) > 0 {
+ b.WriteString("\n## Skill uplift (cli+skill minus cli-noskill)\n\n| Suite | Model | With skill | Without | Uplift |\n|---|---|---|---|---|\n")
+ for _, d := range a.SkillUplift {
+ fmt.Fprintf(&b, "| %s | %s | %.0f%% | %.0f%% | %+.0fpp |\n", d.Suite, d.Model, d.RateA*100, d.RateB*100, d.Delta)
+ }
+ }
+ if len(a.Models) > 0 {
+ b.WriteString("\n## Model vs model (same cases)\n\n| Suite | A | B | A rate | B rate | Delta |\n|---|---|---|---|---|---|\n")
+ for _, d := range a.Models {
+ fmt.Fprintf(&b, "| %s | %s | %s | %.0f%% | %.0f%% | %+.0fpp |\n", d.Suite, d.A, d.B, d.RateA*100, d.RateB*100, d.Delta)
+ }
+ }
+ b.WriteString("\n## Cost (averages over passing runs)\n\n| Model | Passing runs | Avg cost | Avg tool calls | Avg input tokens | Avg output tokens | Cache hit rate | Total spend |\n|---|---|---|---|---|---|---|---|\n")
+ for _, m := range sortedKeys(a.Cost) {
+ c := a.Cost[m]
+ fmt.Fprintf(&b, "| %s | %d | $%.3f | %.1f | %.0f | %.0f | %.0f%% | $%.2f |\n", m, c.PassingRuns, c.AvgCostUSD, c.AvgToolCalls, c.AvgInputTokens, c.AvgOutputTokens, c.CacheHitRate*100, c.TotalCostUSD)
+ }
+ bySuiteCost := map[string]float64{}
+ for _, run := range r.Runs {
+ bySuiteCost[run.Suite+" / "+run.Model] += run.Metrics.CostUSD
+ }
+ if len(bySuiteCost) > 0 {
+ b.WriteString("\n| Suite / model | Spend |\n|---|---|\n")
+ for _, k := range sortedKeys(bySuiteCost) {
+ fmt.Fprintf(&b, "| %s | $%.2f |\n", k, bySuiteCost[k])
+ }
+ }
+ if len(a.TopFailing) > 0 {
+ b.WriteString("\n## Top failing graders\n\n")
+ for _, g := range a.TopFailing {
+ fmt.Fprintf(&b, "- %s: %d\n", g.Grader, g.Count)
+ }
+ }
+ worst := worstCases(a)
+ if len(worst) > 0 {
+ b.WriteString("\n## Worst cases\n\n| Case | Model | Arm | Pass |\n|---|---|---|---|\n")
+ for _, k := range worst {
+ cs, m, arm := split3(k)
+ v := a.Cases[k]
+ fmt.Fprintf(&b, "| %s | %s | %s | %d/%d |\n", cs, m, arm, v.Pass, v.Total)
+ }
+ }
+ b.WriteString("\n## Runs\n\n| Case | Arm | Model | Run | Status | Calls | Cost | First failure |\n|---|---|---|---|---|---|---|---|\n")
+ for _, run := range r.Runs {
+ fmt.Fprintf(&b, "| %s | %s | %s | %d | %s | %d | $%.3f | %s |\n", run.Case, run.Arm, run.Model, run.RunIdx, run.Status, run.Metrics.ToolCalls, run.Metrics.CostUSD, mdEscape(firstFailure(run)))
+ }
+ b.WriteString("\nTranscripts are in the artifact under `runs/<case>/<arm>/<model>/r<N>/transcript.jsonl`.\n")
+ return b.String()
+}
+
+func firstFailure(r *runner.Result) string {
+ if r.Error != "" {
+ return clip(r.Error, 140)
+ }
+ for _, g := range r.Graders {
+ if g.Gating && !g.Skipped && !g.Pass && len(g.Failures) > 0 {
+ return g.Grader + ": " + clip(g.Failures[0], 140)
+ }
+ }
+ return ""
+}
+
+func worstCases(a Aggregates) []string {
+ var keys []string
+ for k, v := range a.Cases {
+ if v.Rate < 1 {
+ keys = append(keys, k)
+ }
+ }
+ sort.Slice(keys, func(i, j int) bool {
+ if a.Cases[keys[i]].Rate != a.Cases[keys[j]].Rate {
+ return a.Cases[keys[i]].Rate < a.Cases[keys[j]].Rate
+ }
+ return keys[i] < keys[j]
+ })
+ if len(keys) > 15 {
+ keys = keys[:15]
+ }
+ return keys
+}
+
+func sortedKeys[V any](m map[string]V) []string {
+ keys := make([]string, 0, len(m))
+ for k := range m {
+ keys = append(keys, k)
+ }
+ sort.Slice(keys, func(i, j int) bool {
+ return suiteRank(keys[i]) < suiteRank(keys[j]) || suiteRank(keys[i]) == suiteRank(keys[j]) && keys[i] < keys[j]
+ })
+ return keys
+}
+
+func suiteRank(k string) int {
+ for i, s := range catalog.Suites {
+ if strings.HasPrefix(k, s) {
+ return i
+ }
+ }
+ return len(catalog.Suites)
+}
+
+func mdEscape(s string) string { return strings.NewReplacer("|", "\\|", "\n", " ").Replace(s) }
+
+func orDash(s string) string {
+ if s == "" {
+ return "-"
+ }
+ return s
+}
+
+func short(h string) string {
+ if len(h) > 10 {
+ return "`" + h[:10] + "`"
+ }
+ return "`" + h + "`"
+}
+
+func clip(s string, n int) string {
+ s = strings.TrimSpace(s)
+ if len(s) > n {
+ return s[:n] + "..."
+ }
+ return s
+}
diff --git a/cli/test/evals/internal/report/report_test.go b/cli/test/evals/internal/report/report_test.go
new file mode 100644
index 00000000..24bdbd5b
--- /dev/null
+++ b/cli/test/evals/internal/report/report_test.go
@@ -0,0 +1,36 @@
+package report
+
+import (
+ "math"
+ "testing"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"
+)
+
+func res(c, arm, model string, idx int, status string, cost float64) *runner.Result {
+ suite := c[:len("perf")]
+ if c[:5] == "debug" {
+ suite = "debug"
+ }
+ return &runner.Result{Case: c, Suite: suite, Arm: arm, Model: model, RunIdx: idx, Status: status, Metrics: runner.Metrics{CostUSD: cost, ToolCalls: 10}}
+}
+
+func TestAggregate(t *testing.T) {
+ runs := []*runner.Result{
+ res("perf-a", "cli+skill", "opus", 1, "pass", 0.5),
+ res("perf-a", "cli-noskill", "opus", 1, "fail", 0.4),
+ res("perf-b", "cli+skill", "opus", 1, "pass", 0.7),
+ res("perf-b", "cli-noskill", "opus", 1, "pass", 0.6),
+ res("perf-c", "cli+skill", "opus", 1, "infra_error", 0),
+ }
+ a := Aggregate(runs)
+ if r := a.Suites["perf|opus|cli+skill"]; r.Pass != 2 || r.Total != 2 {
+ t.Fatalf("infra errors must not count: %+v", r)
+ }
+ if len(a.SkillUplift) != 1 || math.Abs(a.SkillUplift[0].Delta-50) > 1e-9 {
+ t.Fatalf("uplift %+v", a.SkillUplift)
+ }
+ if c := a.Cost["opus"]; c.PassingRuns != 3 || math.Abs(c.AvgCostUSD-0.6) > 1e-9 {
+ t.Fatalf("cost %+v", c)
+ }
+}
diff --git a/cli/test/evals/internal/report/results.go b/cli/test/evals/internal/report/results.go
new file mode 100644
index 00000000..16807d6c
--- /dev/null
+++ b/cli/test/evals/internal/report/results.go
@@ -0,0 +1,169 @@
+// Package report turns run results into results.json and summary.md.
+package report
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+ "io/fs"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "sort"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"
+)
+
+// Header identifies what was evaluated and who started it.
+type Header struct {
+ RunID string `json:"runId"`
+ GitSHA string `json:"gitSha"`
+ GitDirty bool `json:"gitDirty"`
+ SkillHashes map[string]string `json:"skillHashes"`
+ KnowledgeHash string `json:"knowledgeHash"`
+ BinariesHash string `json:"binariesHash"`
+ CLIVersion string `json:"cliVersion"`
+ BackendVersion string `json:"backendVersion"`
+ ModelsDigest string `json:"modelsDigest"`
+ Models map[string]string `json:"models"`
+ StartedBy string `json:"startedBy"`
+ Command string `json:"command"`
+ Started time.Time `json:"started"`
+ Finished time.Time `json:"finished"`
+ MaxCostUSD float64 `json:"maxCostUsd"`
+ Oracle bool `json:"oracle,omitempty"`
+ CI map[string]string `json:"ci,omitempty"`
+}
+
+// Results is results.json.
+type Results struct {
+ Header Header `json:"header"`
+ Runs []*runner.Result `json:"runs"`
+ Aggregates Aggregates `json:"aggregates"`
+}
+
+// NewHeader fills the provenance fields.
+func NewHeader(repoRoot, runID string, models *catalog.Models, binHash string, cmdline []string) Header {
+ h := Header{RunID: runID, SkillHashes: map[string]string{}, Models: map[string]string{}, Started: time.Now().UTC(), BinariesHash: binHash}
+ git := func(args ...string) string {
+ cmd := exec.Command("git", args...)
+ cmd.Dir = repoRoot
+ out, _ := cmd.Output()
+ return strings.TrimSpace(string(out))
+ }
+ h.GitSHA = git("rev-parse", "HEAD")
+ h.GitDirty = git("status", "--porcelain", "--", "backend", "cli", "skills", "plugins") != ""
+ h.StartedBy = git("config", "user.email")
+ if actor := os.Getenv("GITHUB_ACTOR"); actor != "" {
+ h.StartedBy = actor + " (GitHub Actions)"
+ h.CI = map[string]string{"event": os.Getenv("GITHUB_EVENT_NAME"), "workflow": os.Getenv("GITHUB_WORKFLOW_REF"),
+ "runId": os.Getenv("GITHUB_RUN_ID"), "ref": os.Getenv("GITHUB_REF")}
+ }
+ for name, dir := range map[string]string{
+ "traceway": "skills/traceway", "traceway-write": "plugins/tw-write/skills/traceway-write", "traceway-setup": "skills/traceway-setup",
+ } {
+ h.SkillHashes[name] = HashDir(filepath.Join(repoRoot, dir))
+ }
+ h.KnowledgeHash = HashDir(filepath.Join(repoRoot, "cli/pkg/mcpserver/knowledge"))
+ h.CLIVersion = git("describe", "--tags", "--match", "cli/v*", "--abbrev=0")
+ h.BackendVersion = git("describe", "--tags", "--match", "backend/v*", "--abbrev=0")
+ if models != nil {
+ h.ModelsDigest = models.Digest
+ for k, m := range models.Agents {
+ h.Models[k] = m.ID
+ }
+ if models.Judge != nil {
+ h.Models["judge"] = models.Judge.ID
+ }
+ }
+ h.Command = strings.Join(cmdline, " ")
+ return h
+}
+
+// HashDir is the SHA-256 of a directory's files (paths and contents).
+func HashDir(dir string) string {
+ h := sha256.New()
+ var files []string
+ _ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+ if err == nil && !d.IsDir() {
+ files = append(files, p)
+ }
+ return nil
+ })
+ sort.Strings(files)
+ for _, f := range files {
+ rel, _ := filepath.Rel(dir, f)
+ raw, _ := os.ReadFile(f)
+ h.Write([]byte(rel))
+ h.Write(raw)
+ }
+ return hex.EncodeToString(h.Sum(nil))
+}
+
+// Save writes results.json and summary.md into dir.
+func Save(dir string, r *Results) error {
+ r.Aggregates = Aggregate(r.Runs)
+ raw, err := json.MarshalIndent(r, "", " ")
+ if err != nil {
+ return err
+ }
+ if err := os.WriteFile(filepath.Join(dir, "results.json"), raw, 0o644); err != nil {
+ return err
+ }
+ return os.WriteFile(filepath.Join(dir, "summary.md"), []byte(Markdown(r)), 0o644)
+}
+
+// Load reads a results.json, or rebuilds one from the result.json files
+// under a run directory (a shard that died before writing its summary).
+func Load(dir string) (*Results, error) {
+ raw, err := os.ReadFile(filepath.Join(dir, "results.json"))
+ if err == nil {
+ var r Results
+ if err := json.Unmarshal(raw, &r); err != nil {
+ return nil, fmt.Errorf("%s: %w", dir, err)
+ }
+ return &r, nil
+ }
+ r := &Results{}
+ err = filepath.WalkDir(filepath.Join(dir, "runs"), func(p string, d fs.DirEntry, err error) error {
+ if err != nil || d.IsDir() || d.Name() != "result.json" {
+ return nil
+ }
+ b, err := os.ReadFile(p)
+ if err != nil {
+ return nil
+ }
+ var one runner.Result
+ if json.Unmarshal(b, &one) == nil {
+ r.Runs = append(r.Runs, &one)
+ }
+ return nil
+ })
+ if err != nil || len(r.Runs) == 0 {
+ return nil, fmt.Errorf("%s: no results.json and no run results", dir)
+ }
+ return r, nil
+}
+
+// Merge combines shard results under one header.
+func Merge(parts []*Results) *Results {
+ out := &Results{}
+ total := 0.0
+ for i, p := range parts {
+ if i == 0 {
+ out.Header = p.Header
+ }
+ total += p.Header.MaxCostUSD
+ if p.Header.Finished.After(out.Header.Finished) {
+ out.Header.Finished = p.Header.Finished
+ }
+ out.Runs = append(out.Runs, p.Runs...)
+ }
+ out.Header.MaxCostUSD = total
+ sort.SliceStable(out.Runs, func(i, j int) bool { return out.Runs[i].Key() < out.Runs[j].Key() })
+ return out
+}
diff --git a/cli/test/evals/internal/runner/runner.go b/cli/test/evals/internal/runner/runner.go
new file mode 100644
index 00000000..ff663e86
--- /dev/null
+++ b/cli/test/evals/internal/runner/runner.go
@@ -0,0 +1,402 @@
+// Package runner executes eval runs: for each (case, arm, model, run) it boots
+// a hermetic backend, seeds it, prepares the workspace and credentials for
+// the arm, drives the agent, grades the result and tears everything down.
+package runner
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io"
+ "os"
+ "path/filepath"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/agent"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/grade"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"
+)
+
+// Options configures a batch of runs.
+type Options struct {
+ EvalsRoot string
+ RepoRoot string
+ OutDir string
+ Bins *env.Binaries
+ Models []*catalog.Model
+ Runs int
+ Jobs int
+ Keep bool
+ Log io.Writer
+}
+
+// Result is one run's outcome (results.json "runs" entries).
+type Result struct {
+ Case string `json:"case"`
+ Suite string `json:"suite"`
+ Arm string `json:"arm"`
+ Model string `json:"model"`
+ ModelID string `json:"modelId"`
+ Providers []string `json:"provider,omitempty"`
+ RunIdx int `json:"runIdx"`
+ Status string `json:"status"` // pass | fail | budget_exhausted | infra_error
+ Graders []*grade.Verdict `json:"graders"`
+ Metrics Metrics `json:"metrics"`
+ Transcript string `json:"transcript,omitempty"`
+ Error string `json:"error,omitempty"`
+ Seed *seed.Stats `json:"seed,omitempty"`
+}
+
+// Key identifies the run.
+func (r *Result) Key() string { return RunKey(r.Case, r.Arm, r.Model, r.RunIdx) }
+
+// RunKey builds a run key.
+func RunKey(c, arm, model string, idx int) string {
+ return fmt.Sprintf("%s/%s/%s/r%d", c, arm, model, idx)
+}
+
+// Metrics are a run's measurements.
+type Metrics struct {
+ Turns int `json:"turns"`
+ ToolCalls int `json:"toolCalls"`
+ FailedCalls int `json:"failedCalls"`
+ InputTokens int `json:"inputTokens"`
+ CachedTokens int `json:"cachedTokens"`
+ CacheWriteTokens int `json:"cacheWriteTokens"`
+ OutputTokens int `json:"outputTokens"`
+ CostUSD float64 `json:"costUsd"`
+ JudgeCostUSD float64 `json:"judgeCostUsd"`
+ WallMs int64 `json:"wallMs"`
+ SkillLoaded bool `json:"skillLoaded"`
+ Skills []string `json:"skills,omitempty"`
+}
+
+// Job is one scheduled run.
+type Job struct {
+ Unit catalog.Unit
+ Model *catalog.Model
+ Idx int
+}
+
+// Jobs expands units × models × runs.
+func Jobs(units []catalog.Unit, models []*catalog.Model, runs int) []Job {
+ var out []Job
+ for _, u := range units {
+ for _, m := range models {
+ for i := 1; i <= runs; i++ {
+ out = append(out, Job{Unit: u, Model: m, Idx: i})
+ }
+ }
+ }
+ return out
+}
+
+// Execute runs every job with up to Jobs in parallel. Every run plays its
+// case's oracle with the fake model.
+func Execute(ctx context.Context, opt Options, jobs []Job) []*Result {
+ par := opt.Jobs
+ if par <= 0 {
+ par = 1
+ }
+ results := make([]*Result, len(jobs))
+ sem := make(chan struct{}, par)
+ var wg sync.WaitGroup
+ for i, j := range jobs {
+ sem <- struct{}{}
+ wg.Add(1)
+ go func(i int, j Job) {
+ defer wg.Done()
+ defer func() { <-sem }()
+ results[i] = runOne(ctx, opt, j)
+ }(i, j)
+ }
+ wg.Wait()
+ return results
+}
+
+func logf(opt Options, format string, a ...any) {
+ if opt.Log != nil {
+ fmt.Fprintf(opt.Log, format+"\n", a...)
+ }
+}
+
+// surface is what the arm hands the agent.
+type surface struct {
+ cliBin string // executable path of the traceway CLI
+ cliName string // traceway
+ cliHome *env.CLIHome
+ skills []string
+ submit string
+}
+
+func submitFor(suite string) string {
+ switch suite {
+ case "perf":
+ return tools.SubmitPerf
+ case "debug":
+ return tools.SubmitDebug
+ default:
+ return tools.Finish
+ }
+}
+
+func runOne(ctx context.Context, opt Options, j Job) (res *Result) {
+ c := j.Unit.Case
+ arm := j.Unit.Arm
+ res = &Result{Case: c.ID, Suite: c.Suite, Arm: arm, Model: j.Model.Key, ModelID: j.Model.ID, RunIdx: j.Idx}
+ started := time.Now()
+ dir := filepath.Join(opt.OutDir, "runs", c.ID, sanitize(arm), res.Model, fmt.Sprintf("r%d", j.Idx))
+ defer func() {
+ res.Metrics.WallMs = time.Since(started).Milliseconds()
+ raw, _ := json.MarshalIndent(res, "", " ")
+ _ = os.WriteFile(filepath.Join(dir, "result.json"), raw, 0o644)
+ if !opt.Keep && res.Status == "pass" {
+ // A passing run keeps its record (result, run, seed, transcript,
+ // backend log); the database, CLI state and workspace go.
+ for _, d := range []string{"backend", "cli", "work", ".twevals-bin", ".twevals-home"} {
+ _ = os.RemoveAll(filepath.Join(dir, d))
+ }
+ }
+ }()
+ infra := func(err error) *Result {
+ res.Status, res.Error = agent.StopInfra, err.Error()
+ logf(opt, " %s infra_error: %v", RunKey(c.ID, arm, res.Model, j.Idx), err)
+ return res
+ }
+ // A rerun into the same --out starts clean: a database left by an
+ // earlier failed run already has its owner registered.
+ if err := os.RemoveAll(dir); err != nil {
+ return infra(err)
+ }
+ if err := os.MkdirAll(dir, 0o755); err != nil {
+ return infra(err)
+ }
+ logf(opt, "▶ %s", RunKey(c.ID, arm, res.Model, j.Idx))
+
+ inst, err := env.NewInstance(ctx, opt.Bins, dir)
+ if err != nil {
+ return infra(fmt.Errorf("boot: %w", err))
+ }
+ defer func() {
+ _ = inst.Backend.WriteLog(filepath.Join(dir, "backend.log"))
+ inst.Close()
+ }()
+
+ sink := seed.NewLiveSink(ctx, inst.Backend, inst.Admin)
+ sink.StubSrv = inst.Stub
+ sc := seed.NewScenario(c.ID, time.Now(), sink)
+ if err := scenarios.Run(c.Seed, sc); err != nil {
+ return infra(err)
+ }
+ if err := sc.Verify(60 * time.Second); err != nil {
+ return infra(err)
+ }
+ stats := sc.Stats()
+ res.Seed = &stats
+ projects := sink.Projects()
+ if len(projects) == 0 {
+ return infra(fmt.Errorf("seed %s created no project", c.Seed))
+ }
+ exposed := sc.Exposed()
+ primary := primaryProject(exposed, projects)
+
+ vars := map[string]any{"seed": jsonify(exposed), "env": envVars(inst, projects, primary, sc.Now)}
+ resolved, err := c.Resolve(vars)
+ if err != nil {
+ return infra(err)
+ }
+ writeJSON(filepath.Join(dir, "seed.json"), map[string]any{"exposed": jsonify(exposed), "env": vars["env"], "stats": stats})
+
+ work := filepath.Join(dir, "work")
+ if err := buildWorkspace(opt.EvalsRoot, resolved.Workspace.Repo, work, sc.Now); err != nil {
+ return infra(err)
+ }
+ surf, err := prepareSurface(ctx, opt, resolved, arm, inst, primary, dir)
+ if err != nil {
+ return infra(err)
+ }
+ if err := installSkills(opt.RepoRoot, work, surf.skills); err != nil {
+ return infra(err)
+ }
+
+ sb := &tools.Sandbox{Root: work, ReadOnly: []string{".skills"}}
+ ask := &tools.AskUser{Rules: resolved.User}
+ var loaded []string
+ var submit *tools.Submit
+ var toolset []tools.Tool
+ var calls []*record.Call
+ var callsMu sync.Mutex
+ callExists := func(id string) (bool, bool) {
+ callsMu.Lock()
+ defer callsMu.Unlock()
+ for _, cl := range calls {
+ if cl.ID == id {
+ return cl.OK, true
+ }
+ }
+ return false, false
+ }
+ sh := &tools.Shell{Sandbox: sb, Git: true, Denied: deniedFor()}
+ sh.Bins = map[string]string{surf.cliName: surf.cliBin}
+ sh.BinEnv = surf.cliHome.Env(filepath.Dir(surf.cliBin))
+ toolset = append(toolset, sh)
+ toolset = append(toolset, &tools.ReadFile{Sandbox: sb}, &tools.ListFiles{Sandbox: sb}, &tools.Search{Sandbox: sb},
+ &tools.EditFile{Sandbox: sb}, &tools.WriteFile{Sandbox: sb}, ask)
+ skillDefs, _ := tools.LoadSkills(filepath.Join(work, ".skills"))
+ if len(skillDefs) > 0 {
+ toolset = append(toolset, &tools.LoadSkill{Skills: skillDefs, Rel: sb.Rel, Loaded: &loaded})
+ }
+ submit = &tools.Submit{Kind: surf.submit, CallExists: callExists, MaxRepairs: 2}
+ toolset = append(toolset, submit)
+
+ prompt := strings.TrimSpace(resolved.Prompt)
+ var preloaded *tools.Skill
+ if name := agent.SlashSkill(prompt); name != "" {
+ for i := range skillDefs {
+ if skillDefs[i].Name == name {
+ preloaded = &skillDefs[i]
+ loaded = append(loaded, name)
+ }
+ }
+ if preloaded == nil {
+ prompt = agent.StripSlash(prompt)
+ }
+ }
+ system := agent.SystemPrompt(agent.PromptInput{
+ SubmitTool: surf.submit, Skills: skillDefs, Preloaded: preloaded,
+ Environment: environmentNote(surf, inst, primary),
+ })
+
+ fake := agent.NewFake(resolved.OracleFor(arm), filepath.Join(opt.EvalsRoot, "testdata"))
+ defer fake.Close()
+ tr, err := agent.OpenTranscript(filepath.Join(dir, "transcript.jsonl"))
+ if err != nil {
+ return infra(err)
+ }
+ defer tr.Close()
+ res.Transcript, _ = filepath.Rel(opt.OutDir, filepath.Join(dir, "transcript.jsonl"))
+
+ run := agent.Run(ctx, agent.Config{
+ Model: j.Model, Chat: fake.Client(), Tools: toolset, System: system, Prompt: prompt, SubmitTool: surf.submit,
+ Budget: resolved.Budget, Transcript: tr,
+ AfterCall: func(ctx context.Context, cl *record.Call) {
+ callsMu.Lock()
+ calls = append(calls, cl)
+ callsMu.Unlock()
+ },
+ Questions: ask.Questions,
+ SkillLoaded: func() []string { return loaded },
+ })
+ writeJSON(filepath.Join(dir, "run.json"), run)
+
+ res.Providers = run.Providers
+ res.Metrics = metricsOf(run)
+ if errs := fake.Errors(); len(errs) > 0 {
+ res.Error = "oracle: " + strings.Join(errs, "; ")
+ }
+ if run.StopReason == agent.StopInfra {
+ res.Status, res.Error = agent.StopInfra, run.StopDetail
+ return res
+ }
+
+ in := &grade.Input{Case: resolved, Arm: arm, Run: run, Seed: jsonify(exposed)}
+ res.Graders = append(res.Graders, grade.All(ctx, in)...)
+ for _, v := range res.Graders {
+ res.Metrics.JudgeCostUSD += v.CostUSD
+ }
+ switch {
+ case run.StopReason == agent.StopBudget:
+ res.Status = "budget_exhausted"
+ case res.Error != "":
+ res.Status = "fail"
+ case grade.Passed(res.Graders):
+ res.Status = "pass"
+ default:
+ res.Status = "fail"
+ }
+ logf(opt, " %s %s (%d calls, $%.4f)", RunKey(c.ID, arm, res.Model, j.Idx), res.Status, res.Metrics.ToolCalls, res.Metrics.CostUSD)
+ return res
+}
+
+func primaryProject(exposed map[string]any, projects []env.Project) env.Project {
+ if name, ok := exposed["primaryProject"].(string); ok {
+ for _, p := range projects {
+ if p.Name == name {
+ return p
+ }
+ }
+ }
+ if len(projects) == 0 {
+ return env.Project{}
+ }
+ return projects[0]
+}
+
+func envVars(inst *env.Instance, projects []env.Project, primary env.Project, now time.Time) map[string]any {
+ ps := map[string]any{}
+ for _, p := range projects {
+ ps[p.Name] = map[string]any{"id": p.ID, "name": p.Name, "framework": p.Framework}
+ }
+ return map[string]any{
+ "url": inst.Backend.URL, "projects": ps, "project": primary.ID, "projectName": primary.Name,
+ "org": float64(inst.Admin.OrgID), "stub": inst.Stub.URL, "now": now.UTC().Format(time.RFC3339),
+ }
+}
+
+// jsonify round-trips a value through JSON so templates and graders see
+// plain maps, strings and float64s.
+func jsonify(v map[string]any) map[string]any {
+ raw, _ := json.Marshal(v)
+ var out map[string]any
+ _ = json.Unmarshal(raw, &out)
+ if out == nil {
+ out = map[string]any{}
+ }
+ return out
+}
+
+func writeJSON(path string, v any) {
+ raw, err := json.MarshalIndent(v, "", " ")
+ if err == nil {
+ _ = os.WriteFile(path, raw, 0o644)
+ }
+}
+
+func sanitize(s string) string { return strings.NewReplacer("+", "_", "/", "_").Replace(s) }
+
+func metricsOf(run *record.Run) Metrics {
+ m := Metrics{Turns: run.Turns, InputTokens: run.Usage.InputTokens, CachedTokens: run.Usage.CachedTokens,
+ CacheWriteTokens: run.Usage.CacheWriteTokens, OutputTokens: run.Usage.OutputTokens, CostUSD: run.Usage.CostUSD,
+ Skills: run.SkillLoaded, SkillLoaded: len(run.SkillLoaded) > 0}
+ for _, c := range run.Calls {
+ if c.Rejected {
+ continue
+ }
+ m.ToolCalls++
+ if !c.OK {
+ m.FailedCalls++
+ }
+ }
+ return m
+}
+
+func deniedFor() map[string]string {
+ d := map[string]string{
+ "curl": "no network tools in this sandbox", "wget": "no network tools in this sandbox",
+ "python": "no interpreters in this sandbox", "python3": "no interpreters in this sandbox", "node": "no interpreters in this sandbox",
+ "bash": "this shell already is the interpreter; run the command directly", "sh": "this shell already is the interpreter; run the command directly",
+ }
+ d["traceway-write"] = "traceway-write is not installed on this machine"
+ return d
+}
+
+func environmentNote(s surface, inst *env.Instance, primary env.Project) string {
+ return fmt.Sprintf("The %s CLI is installed and logged in to the team's Traceway instance at %s, with the %q project selected.", s.cliName, inst.Backend.URL, primary.Name)
+}
diff --git a/cli/test/evals/internal/runner/runner_test.go b/cli/test/evals/internal/runner/runner_test.go
new file mode 100644
index 00000000..6c8dfe6b
--- /dev/null
+++ b/cli/test/evals/internal/runner/runner_test.go
@@ -0,0 +1,41 @@
+package runner
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+ _ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios/all"
+)
+
+// TestOracleNPlusOne is the Phase 3 acceptance check: the N+1 case runs end
+// to end with the fake model (backend, seed, CLI, shell, graders), with no
+// network call apart from the local backend.
+func TestOracleNPlusOne(t *testing.T) {
+ bins := env.BinariesForTest(t)
+ root := env.EvalsRoot()
+ repo, err := env.RepoRoot(root)
+ if err != nil {
+ t.Fatal(err)
+ }
+ c, err := catalog.LoadCase(root + "/cases/perf/perf-db-n-plus-1.yaml")
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
+ defer cancel()
+ model := &catalog.Model{Key: "oracle", ID: "fake/oracle", Caching: "none"}
+ opt := Options{EvalsRoot: root, RepoRoot: repo, OutDir: t.TempDir(), Bins: bins, Models: []*catalog.Model{model}, Runs: 1, Jobs: 1}
+ res := Execute(ctx, opt, Jobs([]catalog.Unit{{Case: c, Arm: catalog.ArmCLISkill}}, opt.Models, 1))
+ if len(res) != 1 || res[0].Status != "pass" {
+ for _, r := range res {
+ t.Logf("%s %s %s", r.Key(), r.Status, r.Error)
+ for _, g := range r.Graders {
+ t.Logf(" %s pass=%v %v", g.Grader, g.Pass, g.Failures)
+ }
+ }
+ t.Fatal("the N+1 oracle must pass")
+ }
+}
diff --git a/cli/test/evals/internal/runner/surface.go b/cli/test/evals/internal/runner/surface.go
new file mode 100644
index 00000000..76fde75d
--- /dev/null
+++ b/cli/test/evals/internal/runner/surface.go
@@ -0,0 +1,32 @@
+package runner
+
+import (
+ "context"
+ "path/filepath"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+)
+
+// prepareSurface mints the arm's credential and connects its surface: the
+// CLI logged in with a read PAT (XDG dirs outside the workspace).
+func prepareSurface(ctx context.Context, opt Options, c *catalog.Case, arm string, inst *env.Instance, primary env.Project, dir string) (surface, error) {
+ s := surface{submit: submitFor(c.Suite)}
+ if arm == catalog.ArmCLISkill {
+ s.skills = []string{"traceway"}
+ }
+ pat, err := env.CreatePAT(ctx, inst.Backend, inst.Admin.JWT, "eval agent", "read", 0)
+ if err != nil {
+ return s, err
+ }
+ s.cliName, s.cliBin = "traceway", opt.Bins.CLI
+ home, err := env.NewCLIHome(filepath.Join(dir, "cli"), s.cliBin)
+ if err != nil {
+ return s, err
+ }
+ if err := home.Login(ctx, inst.Backend.URL, pat, primary.ID); err != nil {
+ return s, err
+ }
+ s.cliHome = home
+ return s, nil
+}
diff --git a/cli/test/evals/internal/runner/workspace.go b/cli/test/evals/internal/runner/workspace.go
new file mode 100644
index 00000000..28c51e18
--- /dev/null
+++ b/cli/test/evals/internal/runner/workspace.go
@@ -0,0 +1,210 @@
+package runner
+
+import (
+ "fmt"
+ "io/fs"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "time"
+
+ "gopkg.in/yaml.v3"
+)
+
+// history.yaml in a fixture describes the commits leading to its final tree,
+// with ages relative to the run so `git log` lines up with the seeded
+// deploys. Each commit is the final tree with an overlay applied: the files
+// under its .history/ directory (only those that differ from the final
+// tree) replace or add files, and `absent` lists files that did not exist
+// yet. The final tree (the fixture minus .history and history.yaml) is
+// committed last as HEAD.
+type fixtureHistory struct {
+ Commits []struct {
+ Dir string `yaml:"dir"`
+ Absent []string `yaml:"absent"`
+ Message string `yaml:"message"`
+ Age string `yaml:"age"`
+ Author string `yaml:"author"`
+ } `yaml:"commits"`
+ Head struct {
+ Message string `yaml:"message"`
+ Age string `yaml:"age"`
+ Author string `yaml:"author"`
+ } `yaml:"head"`
+}
+
+// buildWorkspace copies a fixture repo into work and commits its history.
+func buildWorkspace(evalsRoot, repo, work string, now time.Time) error {
+ if err := os.MkdirAll(work, 0o755); err != nil {
+ return err
+ }
+ git := func(env []string, args ...string) error {
+ cmd := exec.Command("git", args...)
+ cmd.Dir = work
+ cmd.Env = append([]string{"HOME=" + work, "PATH=" + os.Getenv("PATH"), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null"}, env...)
+ if out, err := cmd.CombinedOutput(); err != nil {
+ return fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, out)
+ }
+ return nil
+ }
+ if err := git(nil, "init", "-q", "-b", "main"); err != nil {
+ return err
+ }
+ _ = os.WriteFile(filepath.Join(work, ".git", "info", "exclude"), []byte(".skills/\n.twevals*\n"), 0o644)
+ commit := func(msg, age, author string) error {
+ if author == "" {
+ author = "Dana Developer <[email protected]>"
+ }
+ name, email, _ := strings.Cut(strings.TrimSuffix(author, ">"), " <")
+ when := now.Add(-time.Hour)
+ if age != "" {
+ d, err := time.ParseDuration(age)
+ if err != nil {
+ return fmt.Errorf("history age %q: %w", age, err)
+ }
+ when = now.Add(-d)
+ }
+ date := when.UTC().Format(time.RFC3339)
+ env := []string{"GIT_AUTHOR_NAME=" + name, "GIT_AUTHOR_EMAIL=" + email, "GIT_COMMITTER_NAME=" + name, "GIT_COMMITTER_EMAIL=" + email,
+ "GIT_AUTHOR_DATE=" + date, "GIT_COMMITTER_DATE=" + date}
+ if err := git(env, "add", "-A"); err != nil {
+ return err
+ }
+ return git(env, "commit", "-q", "--allow-empty", "-m", msg)
+ }
+ if repo == "" {
+ if err := os.WriteFile(filepath.Join(work, "README.md"), []byte("# service\n"), 0o644); err != nil {
+ return err
+ }
+ return commit("Initial commit", "720h", "")
+ }
+ src := filepath.Join(evalsRoot, "testdata", "repos", repo)
+ if _, err := os.Stat(src); err != nil {
+ return fmt.Errorf("fixture repo %s: not in testdata/repos", repo)
+ }
+ var hist fixtureHistory
+ if raw, err := os.ReadFile(filepath.Join(src, "history.yaml")); err == nil {
+ if err := yaml.Unmarshal(raw, &hist); err != nil {
+ return fmt.Errorf("%s/history.yaml: %w", repo, err)
+ }
+ }
+ for _, c := range hist.Commits {
+ if err := clearTree(work); err != nil {
+ return err
+ }
+ if err := copyFixture(src, work); err != nil {
+ return err
+ }
+ if c.Dir != "" {
+ if err := copyFixture(filepath.Join(src, c.Dir), work); err != nil {
+ return err
+ }
+ }
+ for _, a := range c.Absent {
+ if err := os.RemoveAll(filepath.Join(work, filepath.FromSlash(a))); err != nil {
+ return err
+ }
+ }
+ if err := commit(c.Message, c.Age, c.Author); err != nil {
+ return err
+ }
+ }
+ if err := clearTree(work); err != nil {
+ return err
+ }
+ if err := copyFixture(src, work); err != nil {
+ return err
+ }
+ msg := hist.Head.Message
+ if msg == "" {
+ msg = "Initial commit"
+ }
+ age := hist.Head.Age
+ if age == "" && len(hist.Commits) == 0 {
+ age = "720h"
+ }
+ return commit(msg, age, hist.Head.Author)
+}
+
+// clearTree removes everything in work except .git and .skills.
+func clearTree(work string) error {
+ entries, err := os.ReadDir(work)
+ if err != nil {
+ return err
+ }
+ for _, e := range entries {
+ if e.Name() == ".git" || e.Name() == ".skills" {
+ continue
+ }
+ if err := os.RemoveAll(filepath.Join(work, e.Name())); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// copyFixture copies a fixture tree, skipping its history metadata. Files
+// named *.tmpl are copied without the suffix (so Go tooling never compiles
+// fixture go.mod files inside the evals module).
+func copyFixture(src, dst string) error {
+ return filepath.WalkDir(src, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ rel, _ := filepath.Rel(src, p)
+ if rel == "." {
+ return nil
+ }
+ if rel == "history.yaml" || rel == ".history" || strings.HasPrefix(rel, ".history"+string(filepath.Separator)) {
+ if d.IsDir() {
+ return filepath.SkipDir
+ }
+ return nil
+ }
+ target := filepath.Join(dst, strings.TrimSuffix(rel, ".tmpl"))
+ if d.IsDir() {
+ return os.MkdirAll(target, 0o755)
+ }
+ raw, err := os.ReadFile(p)
+ if err != nil {
+ return err
+ }
+ info, _ := d.Info()
+ mode := os.FileMode(0o644)
+ if info != nil && info.Mode()&0o111 != 0 {
+ mode = 0o755
+ }
+ return os.WriteFile(target, raw, mode)
+ })
+}
+
+// installSkills copies skill directories into work/.skills/<name>/.
+func installSkills(repoRoot, work string, names []string) error {
+ for _, n := range names {
+ src, ok := skillSources[n]
+ if !ok {
+ return fmt.Errorf("unknown skill %s", n)
+ }
+ dst := filepath.Join(work, ".skills", n)
+ if err := os.MkdirAll(dst, 0o755); err != nil {
+ return err
+ }
+ if err := copyFixture(filepath.Join(repoRoot, src), dst); err != nil {
+ return err
+ }
+ // The agent may read but never edit its skills.
+ _ = filepath.WalkDir(dst, func(p string, d fs.DirEntry, err error) error {
+ if err == nil && !d.IsDir() {
+ _ = os.Chmod(p, 0o444)
+ }
+ return nil
+ })
+ }
+ return nil
+}
+
+// skillSources maps a skill name to its published directory in the repo.
+var skillSources = map[string]string{
+ "traceway": "skills/traceway",
+}
diff --git a/cli/test/evals/internal/scenarios/all/all.go b/cli/test/evals/internal/scenarios/all/all.go
new file mode 100644
index 00000000..0983d25c
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/all/all.go
@@ -0,0 +1,6 @@
+// Package all imports every scenario suite so their seed functions register.
+package all
+
+import (
+ _ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios/perf"
+)
diff --git a/cli/test/evals/internal/scenarios/perf/app_sync_background.go b/cli/test/evals/internal/scenarios/perf/app_sync_background.go
new file mode 100644
index 00000000..83e95d90
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/app_sync_background.go
@@ -0,0 +1,95 @@
+package perf
+
+import (
+ "fmt"
+ "sort"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/app-sync-background", papSeedSyncBackground) }
+
+// papSeedSyncBackground: POST /api/statements renders the statement PDF in
+// the request (6-11s) and uploads it before answering, although the
+// dashboard only polls GET /api/statements/:id for the result. Distractor:
+// host CPU jumps to 65-95% while a render runs, which looks like saturation,
+// but the average is ~20%, the endpoint is just as slow when the host is
+// otherwise idle, and every other endpoint is fast.
+func papSeedSyncBackground(s *seed.Scenario) error {
+ p := s.Project("billing", seed.OTel)
+ api := p.Service("statements-api", seed.Go("nethttp", "pgx")).Versions("1.12.4")
+ window := 7 * 24 * time.Hour
+
+ var renders []papInterval
+ var sorted bool
+ create := api.Endpoint("POST", "/api/statements")
+ create.Traffic(seed.Daily(420), seed.Diurnal(), seed.Over(window))
+ create.Detail(seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.2, Max: 40})
+ create.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.6, 1.4))
+ r.DB("SELECT id, legal_name, currency, billing_email FROM accounts WHERE id = $1", r.Ms(1.5, 3.5))
+ invoices := 40 + r.IntN(400)
+ r.DB("SELECT id, number, issued_at, due_at, total_cents, status FROM invoices WHERE account_id = $1 AND issued_at >= $2 AND issued_at < $3 ORDER BY issued_at",
+ r.Ms(8, 24), seed.Rows(invoices))
+ r.DB("INSERT INTO statements (id, account_id, period_start, period_end, status, created_at) VALUES ($1, $2, $3, $4, 'rendering', now())", r.Ms(2, 5))
+ r.Gap(r.Ms(0.2, 0.6))
+ pages := 2 + invoices/25
+ render := time.Duration((6200 + float64(pages)*250) * (0.9 + 0.2*r.Float()) * float64(time.Millisecond))
+ at := r.At()
+ r.Compute("render statement pdf", render, seed.A("statement.pages", pages), seed.A("statement.invoices", invoices))
+ renders = append(renders, papInterval{at, at.Add(render)})
+ sorted = false
+ key := fmt.Sprintf("statements/%d/%06d.pdf", 2026, 100000+r.IntN(900000))
+ r.HTTP("PUT", "https://acme-billing-statements.s3.eu-west-1.amazonaws.com/"+key, r.Ms(160, 420), 200,
+ seed.A("http.request.body.size", 90000+pages*38000))
+ r.DB("UPDATE statements SET status = 'ready', object_key = $2, rendered_at = now() WHERE id = $1", r.Ms(2, 4))
+ r.Gap(r.Ms(0.3, 0.9))
+ r.Status(201)
+ })
+
+ poll := api.Endpoint("GET", "/api/statements/:id")
+ poll.Traffic(seed.Daily(470), seed.Diurnal(), seed.Over(window))
+ poll.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.5, 1.2))
+ r.DB("SELECT id, account_id, status, object_key, rendered_at FROM statements WHERE id = $1", r.Ms(1.5, 4))
+ r.Gap(r.Ms(0.2, 0.6))
+ })
+
+ download := api.Endpoint("GET", "/api/statements/:id/download")
+ download.Traffic(seed.Daily(260), seed.Diurnal(), seed.Over(window))
+ download.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.5, 1.2))
+ r.DB("SELECT object_key FROM statements WHERE id = $1 AND status = 'ready'", r.Ms(1.5, 3.5))
+ r.Compute("presign object url", r.Ms(0.3, 0.9))
+ r.Status(302)
+ })
+
+ invoices := api.Endpoint("GET", "/api/invoices")
+ invoices.Traffic(seed.Daily(500), seed.Diurnal(), seed.Over(window))
+ invoices.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.5, 1.2))
+ r.DB("SELECT id, number, issued_at, total_cents, status FROM invoices WHERE account_id = $1 ORDER BY issued_at DESC LIMIT 50", r.Ms(4, 12), seed.Rows(50))
+ r.Gap(r.Ms(0.5, 2))
+ })
+
+ s.Background(api, seed.TypicalTraffic(8))
+
+ // The host has two vCPUs and a render keeps one of them busy, so a
+ // sample taken while a render runs reads ~80%.
+ renderLoad := func(t time.Time) float64 {
+ if !sorted {
+ sort.Slice(renders, func(i, j int) bool { return renders[i].from.Before(renders[j].from) })
+ sorted = true
+ }
+ return 0.6 * float64(papActive(renders, t, 15*time.Second))
+ }
+ api.Host().CPU(seed.Plus(seed.Flat(0.14, 0.22), renderLoad))
+ api.Host().Memory(seed.Flat(0.41, 0.47))
+
+ s.Expose("slowRequest", create.Slowest(24*time.Hour))
+ s.Expose("p50Ms24h", func() any { return create.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })
+ s.Expose("p95Ms24h", func() any { return create.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/cache_no_cache.go b/cli/test/evals/internal/scenarios/perf/cache_no_cache.go
new file mode 100644
index 00000000..88b75279
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/cache_no_cache.go
@@ -0,0 +1,165 @@
+package perf
+
+import (
+ "fmt"
+ "strings"
+ "time"
+
+ "github.com/google/uuid"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/cache-no-cache", pncSeedNoCache) }
+
+const pncWebOrigin = "https://shop.example.com"
+
+var pncWebPages = []string{"/c/sneakers", "/c/boots", "/c/sandals", "/c/running", "/c/outdoor", "/c/kids", "/c/sale", "/c/new-in"}
+
+// pncSeedNoCache: the shop's web app fetches the whole catalog (~410KB,
+// 1200 products) again on every route change, with a cache-busting `_=`
+// timestamp, and catalog-api keys its Redis cache by the full URL. Every
+// request misses (cache.hit_ratio for the catalog cache ~3%), runs the
+// catalog query and serializes 1200 products, so GET /api/catalog is by far
+// the busiest endpoint for a catalog that changes twice a day. The browser
+// project's occurrences carry the captured navigations and fetches that
+// show the repeat. Distractor: the catalog query is the long pole of each
+// request (~45ms), which reads like a slow query, but it is steady and the
+// cost is volume times repetition.
+func pncSeedNoCache(s *seed.Scenario) error {
+ window := 24 * time.Hour
+ api := s.Project("shop", seed.OTel)
+ web := s.Project("shop-web", seed.Browser, "react")
+ s.Expose("primaryProject", "shop")
+ svc := api.Service("catalog-api", seed.Node("express", "pg")).Versions("6.2.1")
+
+ catalog := svc.Endpoint("GET", "/api/catalog")
+ catalog.Traffic(seed.Daily(9000), seed.Diurnal(), seed.Over(window))
+ catalog.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.002, Max: 8})
+ catalog.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.3, 0.8))
+ url := "/api/catalog?locale=en-US"
+ hit := r.Chance(0.03) // the prerender job calls without the buster
+ if !hit {
+ url += fmt.Sprintf("&_=%d", r.Start().UnixMilli()-int64(r.IntN(400)))
+ }
+ r.RootAttr("url.query", url[len("/api/catalog?"):])
+ if hit {
+ r.Cache("GET", "catalog:"+url, r.Ms(0.6, 1.6))
+ r.RootAttr("http.response.body.size", 412366)
+ r.Gap(r.Ms(1.5, 3))
+ return
+ }
+ r.Cache("GET", "catalog:"+url, r.Ms(0.4, 1.2))
+ r.DB("SELECT p.id, p.sku, p.title, p.brand, p.image_url, c.slug AS category, pr.amount_minor, pr.currency FROM products p JOIN categories c ON c.id = p.category_id JOIN prices pr ON pr.product_id = p.id AND pr.price_list = $1 WHERE p.active ORDER BY c.position, p.rank",
+ r.LogNormal(44*time.Millisecond, 95*time.Millisecond), seed.Rows(1180+r.IntN(40)))
+ ser := r.Compute("serialize catalog", r.Ms(6, 12))
+ ser.Scope = "catalog-api"
+ r.Cache("SET", "catalog:"+url+" [3 other arguments]", r.Ms(0.4, 1.1))
+ r.RootAttr("http.response.body.size", 412366)
+ r.Gap(r.Ms(0.3, 0.8))
+ })
+
+ reviews := svc.Endpoint("GET", "/api/products/:sku/reviews")
+ reviews.Traffic(seed.Daily(1300), seed.Diurnal(), seed.Over(window))
+ reviews.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.3, 0.8))
+ r.Cache("GET", "sess:"+fmt.Sprintf("%x", r.IntN(1<<30)), r.Ms(0.3, 0.9))
+ r.DB("SELECT id, rating, title, body, created_at FROM reviews WHERE product_sku = $1 ORDER BY created_at DESC LIMIT 10 OFFSET $2", r.LogNormal(4*time.Millisecond, 10*time.Millisecond), seed.Rows(10))
+ r.Compute("serialize", r.Ms(0.3, 1))
+ })
+
+ s.Background(svc, seed.TrafficOpts{Endpoints: 3, PerDay: 150})
+ svc.Host().CPU(seed.Flat(0.34, 0.46))
+ svc.Host().Memory(seed.Flat(0.52, 0.6))
+ svc.Gauge("cache.hit_ratio", "1", map[string]any{"cache": "catalog"}, seed.Flat(0.022, 0.045))
+ svc.Gauge("cache.hit_ratio", "1", map[string]any{"cache": "session"}, seed.Flat(0.955, 0.985))
+
+ // The web app: sessions, and the AbortError a quick route change raises
+ // when it aborts the in-flight catalog fetch. Each occurrence carries the
+ // navigations and fetches the SDK captured before it.
+ const bundle = pncWebOrigin + "/assets/index-DqK3x8aP.js"
+ abortStack := "AbortError: signal is aborted without reason\n at " + bundle + ":48:20113\n at " + bundle + ":48:19871"
+ uas := []string{
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
+ "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Mobile/15E148 Safari/604.1",
+ }
+ rng := s.RNG
+ catalogURL := func(at time.Time) string {
+ return fmt.Sprintf("%s/api/catalog?locale=en-US&_=%d", pncWebOrigin, at.UnixMilli())
+ }
+ sessionAttrs := func(ua, path string) map[string]string {
+ platform, viewport, screen := "Win32", "1440x812", "1920x1080"
+ switch {
+ case strings.Contains(ua, "iPhone"):
+ platform, viewport, screen = "iPhone", "390x664", "390x844"
+ case strings.Contains(ua, "Macintosh"):
+ platform, viewport, screen = "MacIntel", "1512x823", "1512x982"
+ }
+ return map[string]string{"url": pncWebOrigin + path, "path": path, "referrer": "", "userAgent": ua, "language": "en-US",
+ "platform": platform, "viewport": viewport, "screen": screen, "timezone": "Europe/Berlin"}
+ }
+ for i := 0; i < 380; i++ {
+ start := s.Ago(time.Duration(30+rng.IntN(23*60)) * time.Minute).Add(time.Duration(rng.IntN(60)) * time.Second)
+ ua := uas[rng.IntN(len(uas))]
+ web.Session(uuid.NewString(), start, start.Add(time.Duration(3+rng.IntN(25))*time.Minute), sessionAttrs(ua, "/"), "2026.10.3")
+ }
+ var firstAt time.Time
+ for i := 0; i < 9; i++ {
+ at := s.Ago(time.Duration(25+i*61+rng.IntN(20)) * time.Minute).Add(time.Duration(rng.IntN(60)) * time.Second)
+ if i == 0 {
+ firstAt = at
+ }
+ sid := uuid.NewString()
+ ua := uas[rng.IntN(len(uas))]
+ started := at.Add(-time.Duration(4+rng.IntN(10)) * time.Minute)
+ web.Session(sid, started, at.Add(time.Duration(2+rng.IntN(6))*time.Minute), sessionAttrs(ua, "/"), "2026.10.3")
+
+ var acts []seed.Action
+ from := "/"
+ t := at.Add(-time.Duration(40+rng.IntN(15)) * time.Second)
+ for step := 0; step < 4; step++ {
+ to := pncWebPages[rng.IntN(len(pncWebPages))]
+ if step == 1 {
+ to = fmt.Sprintf("/p/%s", fmt.Sprintf("SH-%05d", 1000+rng.IntN(80000)))
+ }
+ acts = append(acts, seed.Action{Type: "navigation", At: t, From: from, To: to})
+ acts = append(acts, seed.Action{Type: "network", At: t.Add(30 * time.Millisecond), Method: "GET", URL: catalogURL(t.Add(30 * time.Millisecond)),
+ DurationMs: float64(62 + rng.IntN(60)), Status: 200, RespBytes: 412366})
+ if step == 1 {
+ acts = append(acts, seed.Action{Type: "network", At: t.Add(45 * time.Millisecond), Method: "GET",
+ URL: pncWebOrigin + "/api/products" + to[2:] + "/reviews?page=1", DurationMs: float64(25 + rng.IntN(25)), Status: 200, RespBytes: 9000 + rng.IntN(3000)})
+ }
+ from = to
+ t = t.Add(time.Duration(7+rng.IntN(9)) * time.Second)
+ }
+ // A quick second click aborts the fetch the previous route started.
+ last := pncWebPages[rng.IntN(len(pncWebPages))]
+ acts = append(acts, seed.Action{Type: "navigation", At: at.Add(-900 * time.Millisecond), From: from, To: last})
+ acts = append(acts, seed.Action{Type: "network", At: at.Add(-870 * time.Millisecond), Method: "GET", URL: catalogURL(at.Add(-870 * time.Millisecond)),
+ DurationMs: 860, Error: "AbortError: signal is aborted without reason"})
+ next := fmt.Sprintf("/p/BT-%05d", 1000+rng.IntN(80000))
+ acts = append(acts, seed.Action{Type: "navigation", At: at.Add(-10 * time.Millisecond), From: last, To: next})
+ acts = append(acts, seed.Action{Type: "network", At: at, Method: "GET", URL: catalogURL(at), DurationMs: float64(60 + rng.IntN(50)), Status: 200, RespBytes: 412366})
+ web.Crash(seed.Crash{At: at, Stack: abortStack, AppVersion: "2026.10.3", SessionID: sid,
+ Attributes: map[string]string{"url": pncWebOrigin + next, "userAgent": ua}, Actions: acts})
+ }
+
+ s.Expose("slowRequest", catalog.Pick(func(r *seed.Request) bool {
+ q, _ := r.Root.Attrs["url.query"].(string)
+ return r.Start.After(s.Ago(3*time.Hour)) && len(q) > len("locale=en-US")
+ }))
+ s.Expose("crashAt", firstAt.UTC().Format(time.RFC3339Nano))
+ s.Expose("catalogCount24h", func() any { return catalog.Count(24 * time.Hour) })
+ s.AfterFlush(func() error {
+ hash, err := web.GroupMatching(24*time.Hour, "signal is aborted")
+ if err != nil {
+ return err
+ }
+ s.Expose("webHash", hash)
+ return nil
+ })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/cache_stampede.go b/cli/test/evals/internal/scenarios/perf/cache_stampede.go
new file mode 100644
index 00000000..a8e83888
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/cache_stampede.go
@@ -0,0 +1,149 @@
+package perf
+
+import (
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/cache-stampede", pncSeedStampede) }
+
+// pncHomeSections are the queries that rebuild the home feed, with their
+// share of a rebuild's DB time.
+var pncHomeSections = []struct {
+ q string
+ w float64
+}{
+ {"SELECT id, image_url, target_url, starts_at, ends_at FROM banners WHERE placement = 'home' AND now() BETWEEN starts_at AND ends_at ORDER BY priority", 0.03},
+ {"SELECT c.id, c.title, c.slug, c.hero_image FROM collections c WHERE c.featured ORDER BY c.position LIMIT 6", 0.04},
+ {"SELECT ol.sku, SUM(ol.qty) AS sold FROM order_lines ol JOIN orders o ON o.id = ol.order_id WHERE o.placed_at > now() - interval '7 days' GROUP BY ol.sku ORDER BY sold DESC LIMIT 24", 0.31},
+ {"SELECT id, sku, title, price_minor, image_url FROM products WHERE published_at > now() - interval '14 days' ORDER BY published_at DESC LIMIT 24", 0.07},
+ {"SELECT p.id, p.sku, p.title, p.price_minor, p.image_url FROM editorial_picks e JOIN products p ON p.id = e.product_id WHERE e.week = date_trunc('week', now()) ORDER BY e.position", 0.04},
+ {"SELECT term, count(*) AS n FROM search_events WHERE created_at > now() - interval '1 day' GROUP BY term ORDER BY n DESC LIMIT 10", 0.18},
+ {"SELECT id, slug, title, tile_image FROM categories WHERE parent_id IS NULL ORDER BY position", 0.02},
+ {"SELECT d.id, d.product_id, d.discount_pct, d.ends_at FROM deals d WHERE d.active AND d.ends_at > now() ORDER BY d.discount_pct DESC LIMIT 12", 0.05},
+ {"SELECT b.id, b.name, b.logo_url FROM brands b WHERE b.spotlight_until > now() LIMIT 4", 0.02},
+ {"SELECT r.product_id, avg(r.rating) AS rating, count(*) AS n FROM reviews r WHERE r.created_at > now() - interval '3 days' GROUP BY r.product_id ORDER BY n DESC LIMIT 12", 0.14},
+ {"SELECT id, title, image_url FROM seasonal_stories WHERE season = $1 ORDER BY position LIMIT 3", 0.03},
+ {"SELECT sku FROM product_scores WHERE segment = 'default' ORDER BY score DESC LIMIT 24", 0.07},
+}
+
+// pncSeedStampede: GET /api/home serves the home feed from Redis
+// (home:feed:v3, 10-minute TTL). When the key expires, every request that
+// arrives before the first rebuild has written it back misses and rebuilds
+// the feed itself: ~2.4 req/s times a ~7s rebuild under that contention is
+// 15-25 identical rebuild traces per expiry, every ~10 minutes. Between
+// expiries the hit ratio is ~99%. Distractor: the misses run 12 queries
+// against the DB on the request path, which reads like a missing cache, but
+// the cache is there and hits ~99% of the time.
+func pncSeedStampede(s *seed.Scenario) error {
+ window := 2 * time.Hour
+ p := s.Project("storefront-home", seed.OTel)
+ api := p.Service("home-api", seed.Go("chi", "pgx")).Versions("3.4.1")
+
+ // Expiry schedule: each window [at, at+fill) is when the key is gone
+ // and requests miss; the first rebuild's SET closes it and the next
+ // expiry is ttl later. Built over 24h so the hit-ratio gauge shows it
+ // all day.
+ type expiry struct {
+ at time.Time
+ fill time.Duration
+ }
+ var expiries []expiry
+ for t := s.Ago(24*time.Hour + 7*time.Minute + 23*time.Second); t.Before(s.Now); {
+ fill := time.Duration(7000+s.RNG.IntN(3000)) * time.Millisecond
+ expiries = append(expiries, expiry{t, fill})
+ t = t.Add(fill + 10*time.Minute)
+ }
+ missWindow := func(t time.Time) (expiry, bool) {
+ for _, e := range expiries {
+ if !t.Before(e.at) && t.Before(e.at.Add(e.fill)) {
+ return e, true
+ }
+ }
+ return expiry{}, false
+ }
+ var recent []time.Time // the last two expiries inside the traffic window
+ for i := len(expiries) - 1; i >= 0 && len(recent) < 2; i-- {
+ if expiries[i].at.Add(expiries[i].fill).Before(s.Now.Add(-time.Minute)) {
+ recent = append(recent, expiries[i].at)
+ }
+ }
+
+ misses := map[*seed.Request]bool{}
+ home := api.Endpoint("GET", "/api/home")
+ home.Traffic(seed.Daily(207360), seed.Over(window)) // 2.4 req/s
+ home.Detail(seed.DetailPolicy{Recent: window, Rate: 0.0015, Max: 12})
+ home.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.2, 0.6))
+ e, miss := missWindow(r.Start())
+ if !miss {
+ r.Cache("get", "home:feed:v3", r.LogNormal(700*time.Microsecond, 2*time.Millisecond))
+ r.Compute("decode feed", r.Ms(0.8, 2.2))
+ r.Gap(r.Ms(0.3, 1.2))
+ return
+ }
+ misses[r.Request()] = true
+ for _, at := range recent {
+ if at.Equal(e.at) {
+ r.KeepDetail()
+ }
+ }
+ r.Cache("get", "home:feed:v3", r.LogNormal(700*time.Microsecond, 2*time.Millisecond))
+ total := float64(e.fill) * (0.88 + 0.24*r.Float())
+ r.Span("buildHomeFeed", func(r *seed.Req) {
+ for _, sec := range pncHomeSections {
+ r.DB(sec.q, time.Duration(total*sec.w*(0.9+0.2*r.Float())))
+ }
+ r.Compute("encode feed", r.Ms(6, 14))
+ }).Scope = "home-api"
+ r.Cache("set", "home:feed:v3", r.Ms(0.6, 1.8), seed.A("db.query.text", "set home:feed:v3 ? ex 600"))
+ r.Gap(r.Ms(0.3, 1.2))
+ })
+
+ product := api.Endpoint("GET", "/api/home/stories/:id")
+ product.Traffic(seed.Daily(6000), seed.Over(window))
+ product.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.2, 0.6))
+ r.DB("SELECT id, title, body, image_url, published_at FROM seasonal_stories WHERE id = $1", r.LogNormal(2500*time.Microsecond, 7*time.Millisecond))
+ r.Compute("encode response", r.Ms(0.3, 1))
+ })
+
+ s.Background(api, seed.TrafficOpts{Endpoints: 3, PerDay: 900, Window: window})
+ api.Host().CPU(seed.Flat(0.21, 0.3))
+ api.Host().Memory(seed.Flat(0.4, 0.47))
+ api.Gauge("cache.hit_ratio", "1", map[string]any{"cache": "home-feed"}, func(t time.Time) float64 {
+ for _, e := range expiries {
+ if !t.Before(e.at) && t.Before(e.at.Add(time.Minute)) {
+ return 0.89 - 0.03*float64(e.fill)/float64(10*time.Second)
+ }
+ }
+ return 0.984
+ })
+
+ s.Expose("missRequest", home.Pick(func(r *seed.Request) bool { return misses[r] && len(recent) > 0 && r.Start.After(recent[0]) }))
+ if len(recent) > 0 {
+ s.Expose("lastExpiry", recent[0].UTC().Format(time.RFC3339))
+ s.Expose("spikeFrom", recent[0].Truncate(5*time.Minute).UTC().Format(time.RFC3339))
+ s.Expose("spikeTo", recent[0].Add(time.Minute).Truncate(5*time.Minute).Add(5*time.Minute).UTC().Format(time.RFC3339))
+ }
+ s.Expose("missesPerExpiry", func() any {
+ counts := map[time.Time]int{}
+ for r := range misses {
+ if e, ok := missWindow(r.Start); ok {
+ counts[e.at]++
+ }
+ }
+ out := []int{}
+ for _, e := range expiries {
+ if n, ok := counts[e.at]; ok && e.at.After(s.Ago(window)) {
+ out = append(out, n)
+ }
+ }
+ return out
+ })
+ s.Expose("p50Ms2h", func() any { return home.Percentile(0.5, window).Seconds() * 1000 })
+ s.Expose("p99Ms2h", func() any { return home.Percentile(0.99, window).Seconds() * 1000 })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/db_n_plus_1.go b/cli/test/evals/internal/scenarios/perf/db_n_plus_1.go
new file mode 100644
index 00000000..b525554b
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/db_n_plus_1.go
@@ -0,0 +1,64 @@
+// Package perf holds the seed functions of the perf suite: one scenario per
+// bottleneck checklist row, plus controls and cross-surface cases.
+package perf
+
+import (
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/db-n-plus-1", seedNPlusOne) }
+
+// seedNPlusOne: the orders endpoint loads a customer's orders, then one
+// order_items query per order. Customers accumulate orders over the 29 days,
+// so latency climbs with no deploy and flat CPU. Distractor: the orders query
+// selects every column (over-fetching), but it is fast.
+func seedNPlusOne(s *seed.Scenario) error {
+ p := s.Project("storefront", seed.OTel)
+ api := p.Service("orders-api", seed.Go("nethttp", "sql")).Versions("2.14.3")
+ window := 29 * 24 * time.Hour
+
+ orders := api.Endpoint("GET", "/api/orders")
+ orders.Traffic(seed.Daily(160), seed.Diurnal(), seed.Over(window))
+ orders.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.25, Max: 12})
+ orders.Handler(func(r *seed.Req) {
+ // Orders per customer grow over the window; LIMIT 100 caps them.
+ n := int(r.Lerp(6, 72) * (0.35 + 0.95*r.Float()))
+ if n > 100 {
+ n = 100
+ }
+ if n < 1 {
+ n = 1
+ }
+ r.Gap(r.Ms(0.4, 1.1)) // auth middleware
+ r.DB("SELECT * FROM orders WHERE user_id = $1 ORDER BY created_at DESC LIMIT 100", r.Ms(3, 6), seed.Rows(n))
+ for range n {
+ r.DB("SELECT id, sku, qty, unit_price FROM order_items WHERE order_id = $1", r.Ms(6, 14))
+ }
+ r.Compute("encode response", r.Ms(2, 6))
+ r.Gap(r.Ms(0.2, 0.6))
+ })
+
+ order := api.Endpoint("GET", "/api/orders/:id")
+ order.Traffic(seed.Daily(120), seed.Diurnal(), seed.Over(window))
+ order.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.4, 1.1))
+ r.DB("SELECT id, user_id, status, total_cents, created_at FROM orders WHERE id = $1", r.Ms(2, 5))
+ r.DB("SELECT o.id, o.sku, o.qty, o.unit_price FROM order_items o WHERE o.order_id = $1", r.Ms(3, 7))
+ r.Compute("encode response", r.Ms(0.5, 1.5))
+ })
+
+ s.Background(api, seed.TypicalTraffic(10))
+ api.Host().CPU(seed.Flat(0.22, 0.31))
+ api.Host().Memory(seed.Flat(0.48, 0.55))
+
+ s.Expose("slowRequest", orders.Slowest(24*time.Hour))
+ s.Expose("p50Ms24h", func() any { return orders.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })
+ s.Expose("p95Ms24h", func() any { return orders.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })
+ s.Expose("p95MsFirstWeek", func() any {
+ return orders.PercentileBetween(0.95, s.Ago(window), s.Ago(window-7*24*time.Hour)).Seconds() * 1000
+ })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/db_n_plus_1_test.go b/cli/test/evals/internal/scenarios/perf/db_n_plus_1_test.go
new file mode 100644
index 00000000..7435b863
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/db_n_plus_1_test.go
@@ -0,0 +1,114 @@
+package perf
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+ "strings"
+ "testing"
+ "time"
+
+ client "github.com/tracewayapp/traceway/cli/pkg/client"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+// TestSeedNPlusOne is the Phase 2 acceptance check: the slow request's
+// waterfall has at least 40 order_items spans, every parent covers its
+// children, and the listed percentiles match what the DSL generated.
+func TestSeedNPlusOne(t *testing.T) {
+ bins := env.BinariesForTest(t)
+ ctx, cancel := context.WithTimeout(context.Background(), 4*time.Minute)
+ defer cancel()
+ inst, err := env.NewInstance(ctx, bins, t.TempDir())
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer inst.Close()
+
+ sink := seed.NewLiveSink(ctx, inst.Backend, inst.Admin)
+ s := seed.NewScenario("perf-db-n-plus-1", time.Now(), sink)
+ if err := scenarios.Run("perf/db-n-plus-1", s); err != nil {
+ t.Fatal(err)
+ }
+ if err := s.Verify(time.Minute); err != nil {
+ t.Fatal(err)
+ }
+ t.Logf("seed stats: %+v", s.Stats())
+ ex := s.Exposed()
+ slow := ex["slowRequest"].(map[string]any)
+ pid := s.Projects()[0].ID
+ api := inst.Backend.API().WithBearer(inst.Admin.JWT)
+
+ at, _ := time.Parse(time.RFC3339Nano, slow["recordedAt"].(string))
+ c := client.New(inst.Backend.URL, client.WithJWT(inst.Admin.JWT))
+ detail, err := c.GetEndpoint(ctx, pid, slow["id"].(string), at)
+ if err != nil {
+ t.Fatal(err)
+ }
+ items := 0
+ children := map[string]int64{}
+ byID := map[string]int64{}
+ for _, sp := range detail.Spans {
+ byID[sp.SpanId] = int64(sp.Duration)
+ if sp.ParentSpanId != "" {
+ children[sp.ParentSpanId] += int64(sp.Duration)
+ }
+ if sp.Name == "SELECT order_items" {
+ items++
+ }
+ }
+ if items < 40 {
+ t.Errorf("slowest request has %d order_items spans, want >= 40 (spans: %d)", items, len(detail.Spans))
+ }
+ for parent, sum := range children {
+ if d, ok := byID[parent]; ok && d < sum {
+ t.Errorf("span %s lasts %d but its children sum to %d", parent, d, sum)
+ }
+ }
+
+ from, to := s.Now.Add(-24*time.Hour), s.Now.Add(time.Minute)
+ lres, err := api.MustOK(ctx, http.MethodPost, "/api/endpoints/grouped", env.ProjectQuery(pid), map[string]any{
+ "fromDate": from.Format(time.RFC3339), "toDate": to.Format(time.RFC3339), "orderBy": "p95",
+ "search": "GET /api/orders", "pagination": map[string]int{"page": 1, "pageSize": 20},
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ var list struct {
+ Data []struct {
+ Endpoint string `json:"endpoint"`
+ Count int `json:"count"`
+ P50Duration int64 `json:"p50Duration"`
+ P95Duration int64 `json:"p95Duration"`
+ } `json:"data"`
+ }
+ if err := json.Unmarshal(lres.Body, &list); err != nil {
+ t.Fatal(err)
+ }
+ var found bool
+ for _, d := range list.Data {
+ if d.Endpoint != "GET /api/orders" {
+ continue
+ }
+ found = true
+ p95 := float64(d.P95Duration) / 1e6
+ want := ex["p95Ms24h"].(float64)
+ if p95 < want*0.85 || p95 > want*1.15 {
+ t.Errorf("listed p95 %.0fms, generated %.0fms", p95, want)
+ }
+ if p95 < 500 || p95 > 2500 {
+ t.Errorf("p95 %.0fms is outside the designed 0.5-2.5s range", p95)
+ }
+ t.Logf("GET /api/orders 24h: count=%d p50=%.0fms p95=%.0fms (first week p95 %.0fms)", d.Count, float64(d.P50Duration)/1e6, p95, ex["p95MsFirstWeek"].(float64))
+ }
+ if !found {
+ t.Fatalf("GET /api/orders not listed: %s", lres.Body)
+ }
+ for _, sp := range detail.Spans {
+ if strings.Contains(strings.ToLower(sp.Name), "n+1") {
+ t.Error("seeded data names the answer")
+ }
+ }
+}
diff --git a/cli/test/evals/internal/scenarios/perf/infra_cpu_saturation.go b/cli/test/evals/internal/scenarios/perf/infra_cpu_saturation.go
new file mode 100644
index 00000000..3d32da5b
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/infra_cpu_saturation.go
@@ -0,0 +1,167 @@
+package perf
+
+import (
+ "math"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/infra-cpu-saturation", papSeedCPUSaturation) }
+
+// papSeedCPUSaturation: 16-40h ago (14:20 UTC) a campaign tripled commerce-api's traffic
+// (ramping over 40 minutes) and it has stayed there. Both instances went
+// from ~30% to ~95% CPU and every endpoint got 2-4x slower at once: each
+// span is stretched and run-queue gaps appear, with no single long pole.
+// Nothing was deployed. Distractor: a code regression; but service.version
+// never changed, no commit landed, and the slowdown follows the request rate
+// on every endpoint alike.
+func papSeedCPUSaturation(s *seed.Scenario) error {
+ p := s.Project("commerce", seed.OTel)
+ api := p.Service("commerce-api", seed.Node("express", "pg")).Versions("4.18.2").
+ Instances("commerce-api-7f6d9c4b8-2xkqv", "commerce-api-7f6d9c4b8-9jw4t")
+ window := 7 * 24 * time.Hour
+ // The campaign went live at 14:20 UTC, 16-40 hours before the run.
+ onset := s.Now.UTC().Truncate(24 * time.Hour).Add(14*time.Hour + 20*time.Minute)
+ for onset.After(s.Ago(16 * time.Hour)) {
+ onset = onset.Add(-24 * time.Hour)
+ }
+ ramp := 40 * time.Minute
+
+ load := func(t time.Time) float64 { // traffic multiple of the usual rate
+ switch {
+ case t.Before(onset):
+ return 1
+ case t.Before(onset.Add(ramp)):
+ return 1 + 2*float64(t.Sub(onset))/float64(ramp)
+ default:
+ return 3
+ }
+ }
+ diurnal := func(t time.Time) float64 {
+ h := float64(t.Hour()) + float64(t.Minute())/60
+ return 1 + 0.25*math.Cos((h-14)/24*2*math.Pi)
+ }
+ // Utilization follows the request rate; past ~70% the run queue grows
+ // and every request is stretched.
+ util := func(t time.Time) float64 { return math.Min(0.97, 0.27*load(t)*diurnal(t)) }
+ stretch := func(t time.Time) float64 {
+ u := util(t)
+ if u <= 0.45 {
+ return 1
+ }
+ return 1 + 2.6*math.Pow((u-0.45)/0.52, 1.5)
+ }
+ traffic := func(e *seed.Endpoint, salt string, perDay int) {
+ e.Traffic(papJitter(salt, 0.06), seed.RateFunc(load), seed.RateFunc(diurnal), seed.Daily(perDay), seed.Over(window))
+ e.Detail(seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.02, Max: 12})
+ }
+ // cpu is in-process work (no span: Express is instrumented only at the
+ // edges), stretched fully; db is a query, of which only the client-side
+ // half (driver, event loop) is stretched.
+ cpu := func(r *seed.Req, d time.Duration) time.Duration {
+ return time.Duration(float64(d) * stretch(r.Start()))
+ }
+ db := func(r *seed.Req, d time.Duration) time.Duration {
+ return time.Duration(float64(d) * (1 + 0.5*(stretch(r.Start())-1)))
+ }
+ queue := func(r *seed.Req) {
+ r.Gap(cpu(r, r.Ms(0.3, 0.9)) + time.Duration((stretch(r.Start())-1)*float64(r.Ms(2, 9))))
+ }
+
+ products := api.Endpoint("GET", "/api/products")
+ traffic(products, "products", 260)
+ products.Handler(func(r *seed.Req) {
+ queue(r)
+ r.DB("SELECT id, sku, name, price_cents, thumbnail_url FROM products WHERE category_id = $1 AND visible ORDER BY rank LIMIT 48", db(r, r.Ms(6, 14)), seed.Rows(48))
+ r.Gap(cpu(r, r.Ms(4, 9))) // serialize products
+ })
+ product := api.Endpoint("GET", "/api/products/:id")
+ traffic(product, "product", 320)
+ product.Handler(func(r *seed.Req) {
+ queue(r)
+ r.DB("SELECT id, sku, name, description, price_cents, stock FROM products WHERE id = $1", db(r, r.Ms(1.5, 4)))
+ r.DB("SELECT url, alt FROM product_images WHERE product_id = $1 ORDER BY position", db(r, r.Ms(1.5, 4)))
+ r.Gap(cpu(r, r.Ms(3, 7))) // render product
+ })
+ search := api.Endpoint("GET", "/api/search")
+ traffic(search, "search", 240)
+ search.Handler(func(r *seed.Req) {
+ queue(r)
+ r.HTTP("POST", "http://search.internal:9200/products/_search", db(r, r.Ms(12, 30)), 200)
+ r.Gap(cpu(r, r.Ms(6, 14))) // hydrate results
+ })
+ cart := api.Endpoint("GET", "/api/cart")
+ traffic(cart, "cart", 220)
+ cart.Handler(func(r *seed.Req) {
+ queue(r)
+ r.Cache("GET", "cart:session", db(r, r.Ms(0.4, 1.2)))
+ r.DB("SELECT p.id, p.name, p.price_cents, ci.qty FROM cart_items ci JOIN products p ON p.id = ci.product_id WHERE ci.cart_id = $1", db(r, r.Ms(2, 5)))
+ r.Gap(cpu(r, r.Ms(2, 5))) // price cart
+ })
+ add := api.Endpoint("POST", "/api/cart/items")
+ traffic(add, "add", 160)
+ add.Handler(func(r *seed.Req) {
+ queue(r)
+ r.Gap(cpu(r, r.Ms(0.6, 1.5))) // validate body
+ r.DB("INSERT INTO cart_items (cart_id, product_id, qty) VALUES ($1, $2, $3) ON CONFLICT (cart_id, product_id) DO UPDATE SET qty = cart_items.qty + EXCLUDED.qty", db(r, r.Ms(2, 5)))
+ r.Cache("DEL", "cart:session", db(r, r.Ms(0.3, 0.9)))
+ r.Status(201)
+ })
+ checkout := api.Endpoint("POST", "/api/checkout")
+ traffic(checkout, "checkout", 70)
+ checkout.Handler(func(r *seed.Req) {
+ queue(r)
+ r.DB("SELECT p.id, p.price_cents, p.stock, ci.qty FROM cart_items ci JOIN products p ON p.id = ci.product_id WHERE ci.cart_id = $1 FOR UPDATE", db(r, r.Ms(3, 7)))
+ r.Gap(cpu(r, r.Ms(4, 9))) // price order
+ r.HTTP("POST", "https://api.payments.example/v1/payment_intents", r.LogNormal(180*time.Millisecond, 420*time.Millisecond), 200)
+ r.DB("INSERT INTO orders (id, user_id, total_cents, status, created_at) VALUES ($1, $2, $3, 'paid', now())", db(r, r.Ms(2, 5)))
+ r.Status(201)
+ })
+ recs := api.Endpoint("GET", "/api/recommendations")
+ traffic(recs, "recs", 200)
+ recs.Handler(func(r *seed.Req) {
+ queue(r)
+ r.Cache("GET", "recs:user", db(r, r.Ms(0.4, 1.2)))
+ r.DB("SELECT product_id, score FROM recommendations WHERE user_id = $1 ORDER BY score DESC LIMIT 12", db(r, r.Ms(3, 8)))
+ r.Gap(cpu(r, r.Ms(5, 12))) // rank recommendations
+ })
+ cats := api.Endpoint("GET", "/api/categories")
+ traffic(cats, "cats", 130)
+ cats.Handler(func(r *seed.Req) {
+ queue(r)
+ r.Cache("GET", "categories:tree", db(r, r.Ms(0.3, 0.9)))
+ r.Gap(cpu(r, r.Ms(1, 3))) // serialize categories
+ })
+
+ for _, h := range api.Hosts {
+ api.HostNamed(h).CPU(seed.Plus(seed.Flat(-0.02, 0.02), util))
+ api.HostNamed(h).Memory(seed.Plus(seed.Flat(0.42, 0.46), func(t time.Time) float64 { return 0.04 * (load(t) - 1) }))
+ }
+
+ rfc := func(t time.Time) string { return t.UTC().Format(time.RFC3339) }
+ s.Expose("onsetAt", rfc(onset))
+ s.Expose("beforeFrom", rfc(onset.Add(-24*time.Hour+time.Hour)))
+ s.Expose("beforeTo", rfc(onset.Add(-24*time.Hour+7*time.Hour)))
+ s.Expose("afterFrom", rfc(onset.Add(time.Hour)))
+ s.Expose("afterTo", rfc(onset.Add(7*time.Hour)))
+ var slow *seed.Request
+ for _, rq := range product.Requests() {
+ if rq.Start.After(onset.Add(time.Hour)) && rq.Start.Before(onset.Add(7*time.Hour)) && (slow == nil || rq.Duration() > slow.Duration()) {
+ slow = rq
+ }
+ }
+ slow.KeepDetail()
+ s.Expose("slowRequest", slow.Ref())
+ s.Expose("earlierRequest", product.Pick(func(rq *seed.Request) bool {
+ return rq.Start.After(onset.Add(-23*time.Hour)) && rq.Start.Before(onset.Add(-17*time.Hour)) && rq.Duration() > 10*time.Millisecond
+ }))
+ s.Expose("p95ProductBefore", func() any {
+ return product.PercentileBetween(0.95, onset.Add(-23*time.Hour), onset.Add(-17*time.Hour)).Seconds() * 1000
+ })
+ s.Expose("p95ProductAfter", func() any {
+ return product.PercentileBetween(0.95, onset.Add(time.Hour), onset.Add(7*time.Hour)).Seconds() * 1000
+ })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/net_large_payload.go b/cli/test/evals/internal/scenarios/perf/net_large_payload.go
new file mode 100644
index 00000000..6fc28072
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/net_large_payload.go
@@ -0,0 +1,94 @@
+package perf
+
+import (
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/net-large-payload", pncSeedLargePayload) }
+
+// pncMobileUA reports whether a request came from a phone (the iOS web view
+// or the Android app's okhttp).
+func pncMobileUA(r *seed.Req) bool {
+ ua, _ := r.Request().Root.Attrs["user_agent.original"].(string)
+ return strings.Contains(ua, "iPhone") || strings.HasPrefix(ua, "okhttp")
+}
+
+// pncSeedLargePayload: the feed endpoint returns 50 cards with their images
+// inlined as data URIs, ~2.3MB per response. The query and the handler's
+// own work take ~30ms; the rest of the request is the body going out, so
+// latency follows http.response.body.size and the client's bandwidth (phones
+// take seconds, desktops a few hundred ms). Distractor: the feed query runs
+// on every request with no cache in front of it, but it takes ~15ms.
+func pncSeedLargePayload(s *seed.Scenario) error {
+ window := 2 * 24 * time.Hour
+ p := s.Project("social", seed.OTel)
+ api := p.Service("feed-api", seed.Go("nethttp", "pgx")).Versions("8.4.2")
+
+ feed := api.Endpoint("GET", "/api/feed")
+ feed.Traffic(seed.Daily(2400), seed.Diurnal(), seed.Over(window))
+ feed.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.02, Max: 10})
+ feed.Handler(func(r *seed.Req) {
+ withImages := 40 + r.IntN(11)
+ size := 58000 + r.IntN(9000) + withImages*(44000+r.IntN(4000))
+ r.RootAttr("http.response.body.size", size)
+ r.Gap(r.Ms(0.3, 0.9))
+ r.DB("SELECT c.id, c.kind, c.title, c.body, c.image_data, c.published_at, a.id, a.display_name, a.avatar_data FROM feed_cards c JOIN authors a ON a.id = c.author_id WHERE c.audience = ANY($1) ORDER BY c.published_at DESC LIMIT $2",
+ r.LogNormal(14*time.Millisecond, 32*time.Millisecond), seed.Rows(50))
+ r.Compute("rank cards", r.Ms(2, 6))
+ // json.NewEncoder(w).Encode writes the body as it encodes; on a
+ // slow link the writes block until the client has read it.
+ mbps := 40 + 110*r.Float()
+ if pncMobileUA(r) {
+ mbps = 5 + 15*r.Float()
+ }
+ r.Gap(r.Ms(14, 30) + time.Duration(float64(size)*8/(mbps*1e6)*float64(time.Second)))
+ })
+
+ card := api.Endpoint("GET", "/api/feed/cards/:id")
+ card.Traffic(seed.Daily(900), seed.Diurnal(), seed.Over(window))
+ card.Handler(func(r *seed.Req) {
+ size := 2200 + r.IntN(1800)
+ if r.Chance(0.8) {
+ size += 44000 + r.IntN(4000)
+ }
+ r.RootAttr("http.response.body.size", size)
+ r.Gap(r.Ms(0.3, 0.9))
+ r.DB("SELECT c.id, c.kind, c.title, c.body, c.image_data, c.published_at, a.id, a.display_name FROM feed_cards c JOIN authors a ON a.id = c.author_id WHERE c.id = $1", r.LogNormal(2*time.Millisecond, 5*time.Millisecond))
+ mbps := 40 + 110*r.Float()
+ if pncMobileUA(r) {
+ mbps = 5 + 15*r.Float()
+ }
+ r.Gap(r.Ms(0.5, 1.5) + time.Duration(float64(size)*8/(mbps*1e6)*float64(time.Second)))
+ })
+
+ authors := api.Endpoint("GET", "/api/authors/:id")
+ authors.Traffic(seed.Daily(600), seed.Diurnal(), seed.Over(window))
+ authors.Handler(func(r *seed.Req) {
+ r.RootAttr("http.response.body.size", 900+r.IntN(700))
+ r.Gap(r.Ms(0.3, 0.9))
+ r.DB("SELECT id, display_name, bio, followers FROM authors WHERE id = $1", r.LogNormal(2*time.Millisecond, 5*time.Millisecond))
+ r.Gap(r.Ms(0.3, 1))
+ })
+
+ s.Background(api, seed.TrafficOpts{Endpoints: 3, PerDay: 150})
+ api.Host().CPU(seed.Flat(0.17, 0.26))
+ api.Host().Memory(seed.Flat(0.44, 0.52))
+
+ isMobile := func(r *seed.Request) bool {
+ ua, _ := r.Root.Attrs["user_agent.original"].(string)
+ return strings.Contains(ua, "iPhone") || strings.HasPrefix(ua, "okhttp")
+ }
+ s.Expose("mobileRequest", feed.Pick(func(r *seed.Request) bool {
+ return isMobile(r) && r.Start.After(s.Ago(3*time.Hour)) && r.Duration() > 1500*time.Millisecond
+ }))
+ s.Expose("desktopRequest", feed.Pick(func(r *seed.Request) bool {
+ return !isMobile(r) && r.Start.After(s.Ago(3*time.Hour)) && r.Duration() < 500*time.Millisecond
+ }))
+ s.Expose("p50Ms24h", func() any { return feed.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })
+ s.Expose("p95Ms24h", func() any { return feed.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/net_sequential.go b/cli/test/evals/internal/scenarios/perf/net_sequential.go
new file mode 100644
index 00000000..048026eb
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/net_sequential.go
@@ -0,0 +1,116 @@
+package perf
+
+import (
+ "fmt"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/net-sequential", pncSeedSequential) }
+
+// pncNestScope is the scope of the NestJS controller/handler spans.
+const pncNestScope = "@opentelemetry/instrumentation-nestjs-core"
+
+// pncNest wraps a NestJS route the way @opentelemetry/instrumentation-nestjs-core
+// does: a request_context span for Controller.method with the handler span
+// inside it.
+func pncNest(r *seed.Req, controller, method, route string, body func(r *seed.Req)) {
+ ctx := r.Span(controller+"."+method, func(r *seed.Req) {
+ h := r.Span(method, body, seed.A("nestjs.type", "handler"), seed.A("nestjs.callback", method))
+ h.Scope = pncNestScope
+ }, seed.A("nestjs.type", "request_context"), seed.A("nestjs.controller", controller), seed.A("nestjs.callback", method),
+ seed.A("http.route", route), seed.A("component", "@nestjs/core"), seed.A("nestjs.version", "11.1.6"))
+ ctx.Scope = pncNestScope
+}
+
+// pncCall is r.Call with the concrete path and query the caller really
+// requested, on both the CLIENT span and the downstream SERVER span.
+func pncCall(r *seed.Req, target *seed.Endpoint, rtt time.Duration, path, query string) *seed.Span {
+ c := r.Call(target, rtt)
+ full := "http://" + target.Service().Name + ".internal:8080" + path
+ if query != "" {
+ full += "?" + query
+ }
+ c.Attrs["url.full"] = full
+ if len(c.Children) > 0 {
+ c.Children[0].Attrs["url.path"] = path
+ if query != "" {
+ c.Children[0].Attrs["url.query"] = query
+ }
+ }
+ return c
+}
+
+// pncSeedSequential: the portal's dashboard endpoint (NestJS) calls four
+// internal services one after another. Each service is fast and steady on
+// its own; the dashboard pays their sum. Distractor: recommendations is the
+// longest hop, but it is an internal service in line with its own p95, and no
+// hop dominates.
+func pncSeedSequential(s *seed.Scenario) error {
+ p := s.Project("portal", seed.OTel)
+ window := 3 * 24 * time.Hour
+
+ web := p.Service("web-api", seed.Node("nestjs", "pg")).Versions("3.7.0")
+ accounts := p.Service("accounts-service", seed.Go("chi", "pgx")).Versions("1.22.4")
+ billing := p.Service("billing-service", seed.Java("spring-boot")).Versions("5.3.1")
+ notify := p.Service("notifications-service", seed.Node("express", "pg")).Versions("2.0.9")
+ reco := p.Service("recommendations-service", seed.Python("fastapi", "psycopg")).Versions("0.41.2")
+
+ profile := accounts.Endpoint("GET", "/internal/accounts/:id/profile")
+ profile.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.2, 0.6))
+ r.DB("SELECT a.id, a.name, a.plan, p.seats FROM accounts a JOIN plans p ON p.id = a.plan_id WHERE a.id = $1", r.LogNormal(20*time.Millisecond, 45*time.Millisecond))
+ r.DB("SELECT user_id, role FROM account_members WHERE account_id = $1", r.LogNormal(13*time.Millisecond, 32*time.Millisecond), seed.Rows(4+r.IntN(20)))
+ r.DB("SELECT flag_key, enabled FROM feature_flags WHERE account_id = $1", r.LogNormal(9*time.Millisecond, 22*time.Millisecond), seed.Rows(18))
+ r.Compute("encode response", r.Ms(1, 3))
+ })
+ summary := billing.Endpoint("GET", "/internal/billing/:accountId/summary")
+ summary.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.5, 1.5))
+ r.DB("SELECT id, number, status, total_cents, due_at FROM invoices WHERE account_id = ? ORDER BY due_at DESC LIMIT 3", r.LogNormal(18*time.Millisecond, 42*time.Millisecond), seed.Rows(3))
+ r.DB("SELECT SUM(amount_cents) FROM ledger_entries WHERE account_id = ? AND posted_at >= ?", r.LogNormal(52*time.Millisecond, 110*time.Millisecond))
+ r.Compute("BillingSummaryMapper.toDto", r.Ms(1, 3))
+ })
+ unread := notify.Endpoint("GET", "/internal/notifications/unread")
+ unread.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.3, 0.8))
+ r.DB("SELECT count(*) FROM notifications WHERE account_id = $1 AND read_at IS NULL", r.LogNormal(22*time.Millisecond, 50*time.Millisecond))
+ r.DB("SELECT id, kind, title, created_at FROM notifications WHERE account_id = $1 AND read_at IS NULL ORDER BY created_at DESC LIMIT 5", r.LogNormal(16*time.Millisecond, 38*time.Millisecond), seed.Rows(5))
+ r.Compute("serialize", r.Ms(0.5, 2))
+ })
+ suggest := reco.Endpoint("GET", "/internal/recommendations")
+ suggest.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.4, 1.2))
+ r.Cache("GET", "reco:features", r.Ms(0.6, 1.8))
+ r.DB("SELECT item_id, category, popularity FROM catalog_items WHERE category = ANY(%s) AND active LIMIT 400", r.LogNormal(24*time.Millisecond, 55*time.Millisecond), seed.Rows(400))
+ r.Compute("score_candidates", r.LogNormal(72*time.Millisecond, 130*time.Millisecond))
+ })
+
+ dash := web.Endpoint("GET", "/api/dashboard")
+ dash.Traffic(seed.Daily(800), seed.Diurnal(), seed.Over(window))
+ dash.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.05, Max: 12})
+ dash.Handler(func(r *seed.Req) {
+ acct := fmt.Sprint(20000 + r.IntN(60000))
+ r.Gap(r.Ms(0.6, 1.6)) // auth guard
+ pncNest(r, "DashboardController", "getDashboard", "/api/dashboard", func(r *seed.Req) {
+ pncCall(r, profile, r.Ms(1.2, 3), "/internal/accounts/"+acct+"/profile", "")
+ pncCall(r, summary, r.Ms(1.2, 3), "/internal/billing/"+acct+"/summary", "")
+ pncCall(r, unread, r.Ms(1.2, 3), "/internal/notifications/unread", "accountId="+acct)
+ pncCall(r, suggest, r.Ms(1.2, 3), "/internal/recommendations", "accountId="+acct+"&limit=8")
+ c := r.Compute("DashboardPresenter.compose", r.Ms(2, 6))
+ c.Scope = "web-api"
+ })
+ r.Gap(r.Ms(0.3, 0.8))
+ })
+
+ s.Background(web, seed.TrafficOpts{Endpoints: 4, PerDay: 150})
+ web.Host().CPU(seed.Flat(0.16, 0.27))
+ web.Host().Memory(seed.Flat(0.44, 0.52))
+
+ s.Expose("slowRequest", dash.Slowest(24*time.Hour))
+ s.Expose("p50Ms24h", func() any { return dash.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })
+ s.Expose("p95Ms24h", func() any { return dash.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/pap_helpers.go b/cli/test/evals/internal/scenarios/perf/pap_helpers.go
new file mode 100644
index 00000000..aeeb6c0f
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/pap_helpers.go
@@ -0,0 +1,38 @@
+package perf
+
+import (
+ "fmt"
+ "hash/fnv"
+ "sort"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+// Helpers shared by the application and infrastructure perf seeds.
+
+// papInterval is a stretch of wall time during which some work ran.
+type papInterval struct{ from, to time.Time }
+
+// papJitter varies an endpoint's rate by a few percent per hour, so the same
+// hours of two days never carry identical request counts.
+func papJitter(salt string, depth float64) seed.Shape {
+ return seed.RateFunc(func(t time.Time) float64 {
+ h := fnv.New64a()
+ fmt.Fprintf(h, "%s|%s", salt, t.UTC().Format("2006-01-02T15"))
+ u := float64(h.Sum64()>>11) / float64(1<<53)
+ return 1 - depth + 2*depth*u
+ })
+}
+
+// papActive counts the intervals that cover t. ivs must be sorted by from.
+func papActive(ivs []papInterval, t time.Time, maxLen time.Duration) int {
+ i := sort.Search(len(ivs), func(i int) bool { return !ivs[i].from.Before(t.Add(-maxLen)) })
+ n := 0
+ for ; i < len(ivs) && !ivs[i].from.After(t); i++ {
+ if t.Before(ivs[i].to) {
+ n++
+ }
+ }
+ return n
+}
diff --git a/cli/test/evals/internal/scenarios/perf/pct_helpers.go b/cli/test/evals/internal/scenarios/perf/pct_helpers.go
new file mode 100644
index 00000000..3ac973bb
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/pct_helpers.go
@@ -0,0 +1,27 @@
+package perf
+
+import (
+ "math"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+// pctMs exposes a duration as float milliseconds rounded to one decimal.
+func pctMs(d time.Duration) float64 {
+ return math.Round(float64(d)/float64(time.Millisecond)*10) / 10
+}
+
+// pctLazyMs is a lazy Expose of an endpoint percentile over [from, to).
+func pctLazyMs(e *seed.Endpoint, p float64, from, to time.Time) func() any {
+ return func() any { return pctMs(e.PercentileBetween(p, from, to)) }
+}
+
+// pctRFC3339 formats a time for templates.
+func pctRFC3339(t time.Time) string { return t.UTC().Format(time.RFC3339) }
+
+// pctDiurnal is seed.Diurnal's factor, so CPU can follow the same load.
+func pctDiurnal(t time.Time) float64 {
+ h := float64(t.Hour()) + float64(t.Minute())/60
+ return 1 + 0.6*math.Cos((h-14)/24*2*math.Pi)
+}
diff --git a/cli/test/evals/internal/scenarios/perf/pct_xs_android_timeouts.go b/cli/test/evals/internal/scenarios/perf/pct_xs_android_timeouts.go
new file mode 100644
index 00000000..349be0c3
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/pct_xs_android_timeouts.go
@@ -0,0 +1,181 @@
+package perf
+
+import (
+ "fmt"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/xs-android-timeouts", pctSeedXSAndroidTimeouts) }
+
+// pctAndroidTimeoutStack is the uncaught SocketTimeoutException the Android
+// app crashes with when OkHttp's 10s read timeout fires on POST
+// /api/checkout (deobfuscated with the uploaded mapping).
+const pctAndroidTimeoutStack = `java.net.SocketTimeoutException: timeout
+ at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:147)
+ at okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:186)
+ at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:390)
+ at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:436)
+ at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:329)
+ at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
+ at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180)
+ at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110)
+ at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)
+ at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
+ at com.tracewayapp.traceway.network.TracewayOkHttpInterceptor.intercept(TracewayOkHttpInterceptor.kt:31)
+ at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
+ at com.example.shop.net.AuthInterceptor.intercept(AuthInterceptor.kt:18)
+ at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
+ at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
+ at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
+ at retrofit2.OkHttpCall.execute(OkHttpCall.java:207)
+ at com.example.shop.checkout.CheckoutRepository$placeOrder$2.invokeSuspend(CheckoutRepository.kt:31)
+ at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
+ at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)
+ at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113)
+ at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
+ at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
+ at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
+ at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
+ at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
+ Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@4b1c2a8, Dispatchers.Main.immediate]`
+
+// pctDeadlockStack is what Spring reports when two checkouts lock the same
+// inventory rows in opposite order.
+const pctDeadlockStack = `org.springframework.dao.CannotAcquireLockException: could not execute statement [ERROR: deadlock detected
+ Detail: Process 31877 waits for ShareLock on transaction 9182736; blocked by process 31790.
+Process 31790 waits for ShareLock on transaction 9182741; blocked by process 31877.
+ Where: while updating tuple (412,7) in relation "inventory"] [update inventory set reserved=reserved+? where sku=? and available-reserved>=?]
+ at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:299)
+ at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:244)
+ at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
+ at com.example.shop.checkout.InventoryRepository.reserve(InventoryRepository.java)
+ at com.example.shop.checkout.CheckoutService.placeOrder(CheckoutService.java:39)
+ at com.example.shop.checkout.CheckoutController.checkout(CheckoutController.java:27)
+Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected
+ at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2733)
+ at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:155)
+ at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)`
+
+// pctSeedXSAndroidTimeouts: a flash sale started three hours ago. Every
+// checkout reserves stock with UPDATE inventory inside a transaction that
+// also calls the payment provider, so checkouts of the sale item queue on
+// that row's lock for seconds; some run past OkHttp's 10s read timeout and
+// the Android app, which does not catch the SocketTimeoutException, crashes.
+// Expected: lock_transaction_contention. Distractors: an app bug (the crash
+// is unhandled) and connection-pool exhaustion (there is no wait before the
+// first query; the time is inside the UPDATE).
+func pctSeedXSAndroidTimeouts(s *seed.Scenario) error {
+ app := s.Project("shop-android", seed.Mobile, "android")
+ apiProj := s.Project("commerce-api", seed.OTel)
+ saleAt := s.Ago(3*time.Hour + 10*time.Minute)
+ window := 2 * 24 * time.Hour
+ api := apiProj.Service("checkout-api", seed.Java("spring")).Versions("7.4.2")
+
+ checkout := api.Endpoint("POST", "/api/checkout")
+ checkout.Traffic(seed.Daily(600), seed.Diurnal(), seed.Burst(saleAt, s.Now, 18), seed.Over(window))
+ checkout.Detail(seed.DetailPolicy{Recent: 3 * time.Hour, Rate: 0.01, Max: 20})
+ checkout.Handler(func(r *seed.Req) {
+ sale := !r.Start().Before(saleAt)
+ hot := sale && r.Chance(0.75)
+ items := 1 + r.IntN(3)
+ r.Gap(r.Ms(0.6, 1.5))
+ r.DB("select c.id, ci.sku, ci.qty, ci.unit_price_cents from carts c join cart_items ci on ci.cart_id=c.id where c.id=?", r.LogNormal(3*time.Millisecond, 9*time.Millisecond), seed.Rows(items))
+ for i := 0; i < items; i++ {
+ wait := r.LogNormal(2*time.Millisecond, 7*time.Millisecond)
+ if hot && i == 0 {
+ wait = r.LogNormal(1800*time.Millisecond, 14*time.Second)
+ }
+ r.DB("update inventory set reserved=reserved+? where sku=? and available-reserved>=?", wait, seed.Rows(1))
+ if hot && i == 1 && r.Chance(0.012) {
+ r.Error(seed.Exc{Type: "org.springframework.dao.CannotAcquireLockException", Message: "could not execute statement [ERROR: deadlock detected]", Stack: pctDeadlockStack, Escaped: true})
+ r.Log(seed.SevWarn, "SQL Error: 0, SQLState: 40P01", seed.A("logger.name", "org.hibernate.engine.jdbc.spi.SqlExceptionHelper"))
+ r.Log(seed.SevError, "ERROR: deadlock detected\n Detail: Process 31877 waits for ShareLock on transaction 9182736; blocked by process 31790.", seed.A("logger.name", "org.hibernate.engine.jdbc.spi.SqlExceptionHelper"))
+ r.Status(500)
+ return
+ }
+ }
+ r.HTTP("POST", "https://api.payments.example/v1/charges", r.LogNormal(420*time.Millisecond, 1100*time.Millisecond), 201)
+ r.DB("insert into orders (id, cart_id, total_cents, currency, payment_ref, status, created_at) values (?, ?, ?, ?, ?, ?, ?)", r.LogNormal(3*time.Millisecond, 9*time.Millisecond))
+ r.DB("insert into order_lines (order_id, sku, qty, unit_price_cents) values (?, ?, ?, ?)", r.LogNormal(2*time.Millisecond, 8*time.Millisecond))
+ r.Gap(r.Ms(0.4, 1.2))
+ })
+ cart := api.Endpoint("GET", "/api/cart")
+ cart.Traffic(seed.Daily(900), seed.Diurnal(), seed.Burst(saleAt, s.Now, 6), seed.Over(window))
+ cart.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.5, 1.2))
+ r.DB("select c.id, ci.sku, ci.qty, ci.unit_price_cents from carts c join cart_items ci on ci.cart_id=c.id where c.id=?", r.LogNormal(3*time.Millisecond, 9*time.Millisecond), seed.Rows(1+r.IntN(4)))
+ r.Compute("render cart", r.Ms(1, 4))
+ })
+ addItem := api.Endpoint("POST", "/api/cart/items")
+ addItem.Traffic(seed.Daily(500), seed.Diurnal(), seed.Burst(saleAt, s.Now, 6), seed.Over(window))
+ addItem.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.5, 1.2))
+ r.DB("select available, reserved from inventory where sku=?", r.LogNormal(2*time.Millisecond, 7*time.Millisecond), seed.Rows(1))
+ r.DB("insert into cart_items (cart_id, sku, qty, unit_price_cents) values (?, ?, ?, ?) on conflict (cart_id, sku) do update set qty=cart_items.qty+excluded.qty", r.LogNormal(3*time.Millisecond, 9*time.Millisecond))
+ })
+ s.Background(api, seed.TypicalTraffic(4))
+ api.Host().CPU(seed.Step(saleAt, seed.Flat(0.2, 0.3), seed.Flat(0.34, 0.46)))
+ api.Host().Memory(seed.Flat(0.55, 0.63))
+
+ // The Android side: checkouts that ran past 10s, for two in three (the
+ // rest are web and iOS customers), crashed the app.
+ const readTimeout = 10 * time.Second
+ devices := []string{"Pixel 8 / Android 15", "Samsung SM-S921B / Android 14", "Pixel 7a / Android 15", "Xiaomi 23129RAA4G / Android 14", "Samsung SM-A546B / Android 14"}
+ var newest *seed.Request
+ crashes := 0
+ // Crashes are derived from the generated backend requests, so they are
+ // built lazily, once, when the scenario flushes.
+ done := false
+ build := func() {
+ if done {
+ return
+ }
+ done = true
+ for i, r := range checkout.Requests() {
+ if r.Start.Before(saleAt) || r.Duration() < readTimeout+200*time.Millisecond || r.Start.After(s.Ago(2*time.Minute)) || i%3 == 0 {
+ continue
+ }
+ r.KeepDetail()
+ newest = r
+ at := r.Start.Add(readTimeout + 15*time.Millisecond)
+ cartAt := r.Start.Add(-time.Duration(25+i%50) * time.Second)
+ app.Crash(seed.Crash{
+ At: at, Stack: pctAndroidTimeoutStack, AppVersion: "5.18.0", Device: devices[i%len(devices)],
+ Attributes: map[string]string{"screen": "CheckoutActivity"},
+ Actions: []seed.Action{
+ {Type: "navigation", At: cartAt.Add(-3 * time.Second), From: "ProductActivity", To: "CartActivity"},
+ {Type: "network", At: cartAt, Method: "GET", URL: "https://api.shop.example/api/cart", DurationMs: float64(90 + i%120), Status: 200, RespBytes: 1800 + i%900},
+ {Type: "navigation", At: cartAt.Add(4 * time.Second), From: "CartActivity", To: "CheckoutActivity"},
+ {Type: "network", At: at, Method: "POST", URL: "https://api.shop.example/api/checkout", DurationMs: float64(readTimeout.Milliseconds()) + 4},
+ },
+ Logs: []seed.ActionLog{{At: r.Start.Add(-20 * time.Millisecond), Level: "info", Message: fmt.Sprintf("CheckoutViewModel: placing order (%d items)", 1+i%3)}},
+ })
+ crashes++
+ }
+ }
+ s.Expose("androidCrashes", func() any { build(); return crashes })
+ s.Expose("slowRequest", func() any {
+ build()
+ if newest == nil {
+ return nil
+ }
+ return newest.Ref()
+ })
+ s.Expose("saleAt", pctRFC3339(saleAt))
+ s.Expose("p95MsBefore", pctLazyMs(checkout, 0.95, s.Ago(window), saleAt))
+ s.Expose("p95MsSale", pctLazyMs(checkout, 0.95, saleAt, s.Now))
+ s.Expose("p50MsSale", pctLazyMs(checkout, 0.5, saleAt, s.Now))
+ s.Expose("primaryProject", "shop-android")
+ s.AfterFlush(func() error {
+ hash, err := app.GroupMatching(24*time.Hour, "SocketTimeoutException")
+ if err != nil {
+ return err
+ }
+ s.Expose("hash", hash)
+ return nil
+ })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/pct_xs_ios_watchdog.go b/cli/test/evals/internal/scenarios/perf/pct_xs_ios_watchdog.go
new file mode 100644
index 00000000..be91e45a
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/pct_xs_ios_watchdog.go
@@ -0,0 +1,170 @@
+package perf
+
+import (
+ "math"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/xs-ios-watchdog", pctSeedXSIOSWatchdog) }
+
+// pctIOSWatchdogStack is the launch watchdog termination as the app's
+// MetricKit forwarder reports it (Traceway iOS frame format, symbolicated
+// with the uploaded dSYM; system frames stay image+offset).
+const pctIOSWatchdogStack = `Watchdog termination: EXC_CRASH (SIGKILL), FRONTBOARD 0x8badf00d scene-create watchdog transgression: exhausted real (wall clock) time allowance of 20.00 seconds
+*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
+os: ios arch: arm64
+#0 libsystem_kernel.dylib+0x1c48
+#1 libdispatch.dylib+0x4a24
+#2 libdispatch.dylib+0x49d0
+#3 ConfigClient.fetchBlocking(timeout:) (ConfigClient.swift:33:23)
+#4 RemoteConfig.loadAtLaunch() (RemoteConfig.swift:18:37)
+#5 AppDelegate.application(_:didFinishLaunchingWithOptions:) (AppDelegate.swift:17:29)
+#6 UIKitCore+0x2f1a84
+#7 UIKitCore+0x2f0e2c
+#8 UIKitCore+0x2ef9c0
+#9 FrontBoardServices+0x1f2c8
+#10 libdispatch.dylib+0x3fa0
+#11 CoreFoundation+0x77d0c
+#12 GraphicsServices+0x1980
+#13 UIKitCore+0x3c3ef8
+#14 Shop+0x6a4c
+#15 dyld+0x5e4c`
+
+// pctSeedXSIOSWatchdog: two marketing pushes (5h and 1h35m ago) each set off
+// a wave of app launches. Every launch calls GET /api/config and registers
+// its push token, and the mobile API's database pool (8 connections, shared
+// with the feed) runs dry: requests wait up to ~30s for a connection before
+// their first query. The iOS app fetches the config synchronously on the
+// main thread during launch, so launches that wait past ~20s are killed by
+// the watchdog. No app release in three weeks. Expected:
+// connection_pool_exhaustion. Distractor: a pure client bug (the blocking
+// fetch is real, but it only bites since the API started taking 20s).
+func pctSeedXSIOSWatchdog(s *seed.Scenario) error {
+ app := s.Project("shop-ios", seed.Mobile, "ios")
+ apiProj := s.Project("mobile-api", seed.OTel)
+ pushes := []time.Time{s.Ago(5*time.Hour + 10*time.Minute), s.Ago(95 * time.Minute)}
+ window := 2 * 24 * time.Hour
+ api := apiProj.Service("mobile-api", seed.Go("nethttp", "sql")).Versions("2.9.0")
+
+ // wave is how hard a push wave is hitting at t: 1 at the push, decaying
+ // over a few minutes, 0 outside the waves.
+ decay := func(t time.Time, tau float64) float64 {
+ for _, p := range pushes {
+ if d := t.Sub(p); d >= 0 && d < 40*time.Minute {
+ return math.Exp(-d.Minutes() / tau)
+ }
+ }
+ return 0
+ }
+ wave := func(t time.Time) float64 { return decay(t, 4) }
+ // The connection queue builds up faster than it drains.
+ backlog := func(t time.Time) float64 { return decay(t, 9) }
+ // A push reaches every user whatever the hour, so the wave adds the
+ // same number of launches by day or night (it is divided by the
+ // day/night factor the traffic is multiplied by).
+ launches := seed.RateFunc(func(t time.Time) float64 { return 1 + 45*wave(t)/pctDiurnal(t) })
+ poolWait := func(r *seed.Req) time.Duration {
+ w := backlog(r.Start())
+ if w < 0.05 {
+ return r.Ms(0.02, 0.3)
+ }
+ return r.LogNormal(time.Duration(12000*w)*time.Millisecond, time.Duration(36000*w)*time.Millisecond)
+ }
+
+ config := api.Endpoint("GET", "/api/config")
+ config.Traffic(seed.Daily(2400), seed.Diurnal(), launches, seed.Over(window))
+ config.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.01, Max: 20})
+ config.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.3, 0.8))
+ r.Gap(poolWait(r))
+ r.DB("SELECT key, value, rollout_pct FROM feature_flags WHERE platform = $1 AND enabled", r.LogNormal(2*time.Millisecond, 6*time.Millisecond), seed.Rows(40+r.IntN(10)))
+ r.DB("SELECT min_supported, latest FROM app_versions WHERE platform = $1", r.LogNormal(1*time.Millisecond, 4*time.Millisecond), seed.Rows(1))
+ r.Compute("encode response", r.Ms(0.3, 1))
+ })
+ devices := api.Endpoint("POST", "/api/devices")
+ devices.Traffic(seed.Daily(1200), seed.Diurnal(), launches, seed.Over(window))
+ devices.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.3, 0.8))
+ r.Gap(poolWait(r))
+ r.DB("INSERT INTO push_tokens (user_id, platform, token, updated_at) VALUES ($1, $2, $3, now()) ON CONFLICT (token) DO UPDATE SET user_id = EXCLUDED.user_id, updated_at = now()", r.LogNormal(3*time.Millisecond, 10*time.Millisecond))
+ })
+ feed := api.Endpoint("GET", "/api/feed")
+ feed.Traffic(seed.Daily(1500), seed.Diurnal(), launches, seed.Over(window))
+ feed.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.3, 0.8))
+ r.Gap(poolWait(r))
+ r.DB("SELECT p.id, p.title, p.image_url, p.price_cents, p.badge FROM promotions p WHERE p.active AND p.segment = ANY($1) ORDER BY p.priority DESC LIMIT 30", r.LogNormal(120*time.Millisecond, 380*time.Millisecond), seed.Rows(30))
+ r.DB("SELECT product_id, count(*) FROM recent_views WHERE user_id = $1 GROUP BY product_id ORDER BY count(*) DESC LIMIT 20", r.LogNormal(60*time.Millisecond, 220*time.Millisecond), seed.Rows(20))
+ r.Compute("assemble feed", r.Ms(2, 6))
+ })
+ // Everything that touches the database shares the pool, so every route
+ // waits in a wave (no unrelated background routes on this service).
+ product := api.Endpoint("GET", "/api/products/:id")
+ product.Traffic(seed.Daily(1800), seed.Diurnal(), seed.RateFunc(func(t time.Time) float64 { return 1 + 8*wave(t)/pctDiurnal(t) }), seed.Over(window))
+ product.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.3, 0.8))
+ r.Gap(poolWait(r))
+ r.DB("SELECT id, name, description, price_cents, images FROM products WHERE id = $1", r.LogNormal(2*time.Millisecond, 7*time.Millisecond), seed.Rows(1))
+ r.Compute("encode response", r.Ms(0.3, 1))
+ })
+ api.Host().CPU(seed.Flat(0.18, 0.32))
+ api.Host().Memory(seed.Flat(0.48, 0.56))
+
+ // The iOS side: a launch whose config fetch had not returned after ~19s
+ // (the rest of launch takes about a second) is killed by the watchdog;
+ // half the launches are iOS.
+ const killAfter = 19500 * time.Millisecond
+ models := []string{"iPhone16,2 / iOS 18.0.1", "iPhone15,3 / iOS 18.0", "iPhone14,5 / iOS 17.6.1", "iPhone17,1 / iOS 18.0.1", "iPhone13,2 / iOS 17.5"}
+ crashes := 0
+ var newest *seed.Request
+ // Crashes are derived from the generated backend requests, so they are
+ // built lazily, once, when the scenario flushes.
+ done := false
+ build := func() {
+ if done {
+ return
+ }
+ done = true
+ for i, r := range config.Requests() {
+ if r.Duration() < killAfter || i%2 == 0 || r.Start.After(s.Ago(2*time.Minute)) {
+ continue
+ }
+ r.KeepDetail()
+ newest = r
+ app.Crash(seed.Crash{
+ At: r.Start.Add(20*time.Second + time.Duration(200+i%600)*time.Millisecond), Stack: pctIOSWatchdogStack,
+ AppVersion: "4.12.0", Device: models[i%len(models)],
+ Attributes: map[string]string{"source": "metrickit", "phase": "launch"},
+ })
+ crashes++
+ }
+ }
+ s.Expose("watchdogCrashes", func() any { build(); return crashes })
+ s.Expose("slowRequest", func() any {
+ build()
+ if newest == nil {
+ return nil
+ }
+ return newest.Ref()
+ })
+ s.Expose("push1", pctRFC3339(pushes[0]))
+ s.Expose("push2", pctRFC3339(pushes[1]))
+ s.Expose("wave2From", pctRFC3339(pushes[1].Add(-5*time.Minute)))
+ s.Expose("wave2To", pctRFC3339(pushes[1].Add(20*time.Minute)))
+ s.Expose("p50Ms24h", pctLazyMs(config, 0.5, s.Ago(24*time.Hour), s.Now))
+ s.Expose("p99Ms24h", pctLazyMs(config, 0.99, s.Ago(24*time.Hour), s.Now))
+ s.Expose("p95MsWave2", pctLazyMs(config, 0.95, pushes[1], pushes[1].Add(10*time.Minute)))
+ s.Expose("primaryProject", "shop-ios")
+ s.AfterFlush(func() error {
+ hash, err := app.GroupMatching(24*time.Hour, "0x8badf00d")
+ if err != nil {
+ return err
+ }
+ s.Expose("hash", hash)
+ return nil
+ })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/pdb_helpers.go b/cli/test/evals/internal/scenarios/perf/pdb_helpers.go
new file mode 100644
index 00000000..dfa5e8dd
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/pdb_helpers.go
@@ -0,0 +1,76 @@
+package perf
+
+import (
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+// pdbRoute is one healthy route of a perf-db scenario: a few fast queries
+// and a small encode step, so the slow route has neighbours that look normal.
+type pdbRoute struct {
+ method, route string
+ perDay int
+ queries []string
+ // dbMedianMs is the median of each query (default 2ms).
+ dbMedianMs float64
+}
+
+// pdbHealthy adds healthy routes to a service over window.
+func pdbHealthy(api *seed.Service, window time.Duration, routes ...pdbRoute) {
+ for _, rt := range routes {
+ rt := rt
+ med := rt.dbMedianMs
+ if med == 0 {
+ med = 2
+ }
+ e := api.Endpoint(rt.method, rt.route)
+ e.Traffic(seed.Daily(rt.perDay), seed.Diurnal(), seed.Over(window))
+ e.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.02, Max: 4})
+ e.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.4, 1.2))
+ for _, q := range rt.queries {
+ m := time.Duration(med * float64(time.Millisecond))
+ r.DB(q, r.LogNormal(m, m*5))
+ }
+ r.Compute("serialize response", r.Ms(0.2, 1.4))
+ r.Gap(r.Ms(0.1, 0.5))
+ })
+ }
+}
+
+// pdbSlowest returns the slowest request of e started after since that
+// satisfies keep, forcing its detail, as a lazy seed.json reference.
+func pdbSlowest(e *seed.Endpoint, since time.Time, keep func(r *seed.Request) bool) func() any {
+ return func() any {
+ r := pdbSlowestRequest(e, since, keep)
+ if r == nil {
+ return nil
+ }
+ return r.Ref()
+ }
+}
+
+func pdbSlowestRequest(e *seed.Endpoint, since time.Time, keep func(r *seed.Request) bool) *seed.Request {
+ var best *seed.Request
+ for _, r := range e.Requests() {
+ if r.Start.Before(since) || (keep != nil && !keep(r)) {
+ continue
+ }
+ if best == nil || r.Duration() > best.Duration() {
+ best = r
+ }
+ }
+ if best != nil {
+ best.KeepDetail()
+ }
+ return best
+}
+
+// pdbMs is a lazy percentile in milliseconds over [from, to).
+func pdbMs(e *seed.Endpoint, p float64, from, to time.Time) func() any {
+ return func() any { return float64(e.PercentileBetween(p, from, to)) / float64(time.Millisecond) }
+}
+
+// pdbDur converts milliseconds to a duration.
+func pdbDur(ms float64) time.Duration { return time.Duration(ms * float64(time.Millisecond)) }
diff --git a/cli/test/evals/internal/scenarios/perf/pdb_missing_index.go b/cli/test/evals/internal/scenarios/perf/pdb_missing_index.go
new file mode 100644
index 00000000..4f76432f
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/pdb_missing_index.go
@@ -0,0 +1,97 @@
+package perf
+
+import (
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/db-missing-index", pdbSeedMissingIndex) }
+
+// pdbSeedMissingIndex: the user lookup filters users by email, a column with
+// no index, so Postgres reads the whole table on every call. The table grows
+// steadily with signups, and so does the one query span, for every request
+// (it returns one row or none). Primary-key lookups on the same table stay at
+// a millisecond or two. Distractor: release 1.13.0 two weeks ago changed the
+// invite endpoint (two extra primary-key reads and an email enqueue); the
+// lookup curve has no step there and the release did not touch it.
+func pdbSeedMissingIndex(s *seed.Scenario) error {
+ window := 29 * 24 * time.Hour
+ deployAt := s.Ago(14*24*time.Hour + 2*time.Hour) // two hours after the head commit
+ p := s.Project("teamspace", seed.OTel)
+ api := p.Service("accounts-api", seed.Python("fastapi", "sqlalchemy")).
+ Versions("1.12.4", seed.Deploy{At: deployAt, Version: "1.13.0"})
+
+ lookup := api.Endpoint("GET", "/api/users/lookup")
+ lookup.Traffic(seed.Daily(160), seed.Diurnal(), seed.Over(window))
+ lookup.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.25, Max: 12})
+ lookup.Handler(func(r *seed.Req) {
+ users := r.Lerp(1.62e6, 3.94e6) // rows in users; signups accrue steadily
+ r.Gap(r.Ms(0.6, 1.6)) // session dependency
+ found := !r.Chance(0.12)
+ rows := 0
+ if found {
+ rows = 1
+ }
+ // A sequential scan reads every row whether or not one matches.
+ scan := pdbDur(users / 7600 * (0.93 + 0.14*r.Float()))
+ r.DB("SELECT users.id, users.email, users.display_name, users.org_id, users.created_at FROM users WHERE users.email = $1::VARCHAR",
+ scan, seed.Rows(rows))
+ if !found {
+ r.Status(404)
+ r.Gap(r.Ms(0.2, 0.6))
+ return
+ }
+ r.DB("SELECT orgs.id, orgs.name, orgs.plan FROM orgs WHERE orgs.id = $1::INTEGER", r.Ms(0.7, 2.2), seed.Rows(1))
+ r.Gap(r.Ms(0.3, 0.9)) // pydantic response model
+ })
+
+ invites := api.Endpoint("POST", "/api/invites")
+ invites.Traffic(seed.Daily(50), seed.Diurnal(), seed.Over(window))
+ invites.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.1, Max: 4})
+ invites.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.6, 1.6))
+ r.DB("SELECT org_members.role FROM org_members WHERE org_members.org_id = $1::INTEGER AND org_members.user_id = $2::INTEGER", r.Ms(0.7, 2.0), seed.Rows(1))
+ r.DB("INSERT INTO invites (org_id, email, role, token, invited_by, created_at) VALUES ($1::INTEGER, $2::VARCHAR, $3::VARCHAR, $4::VARCHAR, $5::INTEGER, $6::TIMESTAMP WITHOUT TIME ZONE) RETURNING invites.id",
+ r.Ms(1.4, 3.8))
+ if r.Version() == "1.13.0" {
+ // 1.13.0: the invite email names the inviter and the org.
+ r.DB("SELECT orgs.id, orgs.name, orgs.plan FROM orgs WHERE orgs.id = $1::INTEGER", r.Ms(0.7, 2.0), seed.Rows(1))
+ r.DB("SELECT users.id, users.email, users.display_name, users.org_id, users.created_at FROM users WHERE users.id = $1::INTEGER", r.Ms(0.8, 2.4), seed.Rows(1))
+ r.Cache("RPUSH", "queue:emails", r.Ms(0.3, 0.9))
+ }
+ r.Gap(r.Ms(0.3, 0.9))
+ })
+
+ pdbHealthy(api, window,
+ pdbRoute{method: "GET", route: "/api/me", perDay: 45, queries: []string{
+ "SELECT users.id, users.email, users.display_name, users.org_id, users.created_at FROM users WHERE users.id = $1::INTEGER",
+ "SELECT orgs.id, orgs.name, orgs.plan FROM orgs WHERE orgs.id = $1::INTEGER",
+ }, dbMedianMs: 1.2},
+ pdbRoute{method: "GET", route: "/api/orgs/{org_id}/members", perDay: 30, queries: []string{
+ "SELECT users.id, users.email, users.display_name, org_members.role FROM org_members JOIN users ON users.id = org_members.user_id WHERE org_members.org_id = $1::INTEGER ORDER BY users.display_name LIMIT $2::INTEGER",
+ }, dbMedianMs: 3},
+ pdbRoute{method: "GET", route: "/api/invites", perDay: 20, queries: []string{
+ "SELECT invites.id, invites.email, invites.role, invites.created_at FROM invites WHERE invites.org_id = $1::INTEGER AND invites.accepted_at IS NULL ORDER BY invites.created_at DESC",
+ }},
+ pdbRoute{method: "POST", route: "/api/invites/{token}/accept", perDay: 15, queries: []string{
+ "SELECT invites.id, invites.org_id, invites.email, invites.role FROM invites WHERE invites.token = $1::VARCHAR",
+ "INSERT INTO org_members (org_id, user_id, role) VALUES ($1::INTEGER, $2::INTEGER, $3::VARCHAR)",
+ }},
+ pdbRoute{method: "PATCH", route: "/api/me", perDay: 12, queries: []string{
+ "UPDATE users SET display_name=$1::VARCHAR WHERE users.id = $2::INTEGER",
+ }, dbMedianMs: 1.5},
+ )
+ api.Host().CPU(seed.Flat(0.17, 0.26))
+ api.Host().Memory(seed.Flat(0.41, 0.47))
+
+ s.Expose("slowRequest", pdbSlowest(lookup, s.Ago(24*time.Hour), nil))
+ s.Expose("deployAt", deployAt.Format(time.RFC3339))
+ s.Expose("p50Ms24h", pdbMs(lookup, 0.5, s.Ago(24*time.Hour), s.Now))
+ s.Expose("p95Ms24h", pdbMs(lookup, 0.95, s.Ago(24*time.Hour), s.Now))
+ s.Expose("p50MsFirstWeek", pdbMs(lookup, 0.5, s.Ago(window), s.Ago(window-7*24*time.Hour)))
+ s.Expose("invitesP50MsBeforeDeploy", pdbMs(invites, 0.5, s.Ago(window), deployAt))
+ s.Expose("invitesP50MsAfterDeploy", pdbMs(invites, 0.5, deployAt, s.Now))
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/perf/pdb_pool_exhaustion.go b/cli/test/evals/internal/scenarios/perf/pdb_pool_exhaustion.go
new file mode 100644
index 00000000..bfb2b1e7
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/perf/pdb_pool_exhaustion.go
@@ -0,0 +1,180 @@
+package perf
+
+import (
+ "fmt"
+ "math"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func init() { scenarios.Register("perf/db-pool-exhaustion", pdbSeedPoolExhaustion) }
+
+// pdbPoolTimeout is the SQLAlchemy QueuePool message for pool_size=10,
+// max_overflow=5, pool_timeout=30.
+const pdbPoolTimeout = "QueuePool limit of size 10 overflow 5 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)"
+
+// pdbSeedPoolExhaustion: checkout holds its pooled connection for the whole
+// request, including the payment provider call. At the lunch and evening
+// rushes the 15 connections (pool_size 10 + max_overflow 5) are all checked
+// out, and requests wait for one before their first query: a gap with no
+// span before the first DB span, seconds long, only at the peaks. A few wait
+// the full pool_timeout and fail with sqlalchemy.exc.TimeoutError. The cart
+// endpoint shares the pool and waits the same way although its only query
+// takes 3ms; the shipping options, which need no connection, never do. The
+// pool's own metric shows 15 used and 0 idle at exactly those hours.
+// Distractors: a slow database (the inventory query takes 40-60ms at every
+// hour) and the payment provider (150-260ms at every hour, peaks included).
+func pdbSeedPoolExhaustion(s *seed.Scenario) error {
+ window := 3 * 24 * time.Hour
+ p := s.Project("marketplace", seed.OTel)
+ api := p.Service("checkout-api", seed.Python("fastapi", "sqlalchemy")).Versions("5.3.1")
+
+ // bump is 0 off-peak and 1 on the plateau of the lunch (12:00-13:00)
+ // and evening (19:00-20:30) rushes, with 25-minute ramps.
+ type rush struct{ from, to float64 }
+ rushes := []rush{{12, 13}, {19, 20.5}}
+ const ramp = 25.0 / 60
+ bump := func(t time.Time) float64 {
+ h := float64(t.Hour()) + float64(t.Minute())/60 + float64(t.Second())/3600
+ best := 0.0
+ for _, r := range rushes {
+ var v float64
+ switch {
+ case h >= r.from && h <= r.to:
+ v = 1
+ case h > r.from-ramp && h < r.from:
+ v = (h - (r.from - ramp)) / ramp
+ case h > r.to && h < r.to+ramp:
+ v = (r.to + ramp - h) / ramp
+ }
+ best = math.Max(best, v)
+ }
+ return best
+ }
+ load := func(t time.Time) float64 { return 1 + 2.4*bump(t) }
+ // over is how far demand for connections exceeds the 15 the pool has,
+ // 0..1; it is non-zero only near the plateau of a rush.
+ over := func(t time.Time) float64 { return math.Max(0, (load(t)-2.7)/0.7) }
+
+ // waitFor is the time a request spends in QueuePool.get before its first
+ // query; ok is false when it waited the whole pool_timeout.
+ waitFor := func(r *seed.Req) (time.Duration, bool) {
+ o := over(r.Start())
+ if o == 0 || !r.Chance(0.25+0.65*o) {
+ return 0, true
+ }
+ w := time.Duration(o * float64(r.LogNormal(3200*time.Millisecond, 26*time.Second)))
+ if w >= 30*time.Second {
+ return 30*time.Second + r.Ms(1, 4), false
+ }
+ return w, true
+ }
+ poolTimeout := func(r *seed.Req, route, frame string) {
+ r.Error(seed.Exc{Type: "sqlalchemy.exc.TimeoutError", Message: pdbPoolTimeout, Escaped: true, Stack: "Traceback (most recent call last):\n" +
+ " File \"/usr/local/lib/python3.12/site-packages/fastapi/routing.py\", line 219, in run_endpoint_function\n return await run_in_threadpool(dependant.call, **values)\n" +
+ " File \"/usr/local/lib/python3.12/site-packages/starlette/concurrency.py\", line 38, in run_in_threadpool\n return await anyio.to_thread.run_sync(func)\n" +
+ frame +
+ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/session.py\", line 2365, in execute\n return self._execute_internal(\n" +
+ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/session.py\", line 2241, in _execute_internal\n conn = self._connection_for_bind(bind)\n" +
+ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py\", line 3297, in raw_connection\n return self.pool.connect()\n" +
+ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/pool/base.py\", line 449, in connect\n return _ConnectionFairy._checkout(self)\n" +
+ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/pool/impl.py\", line 167, in _do_get\n raise exc.TimeoutError(\n" +
+ "sqlalchemy.exc.TimeoutError: " + pdbPoolTimeout})
+ r.Log(seed.SevError, "Exception in ASGI application", seed.A("exception.type", "sqlalchemy.exc.TimeoutError"), seed.A("exception.message", pdbPoolTimeout),
+ seed.A("http.route", route))
+ r.Status(500)
+ }
+
+ checkout := api.Endpoint("POST", "/api/checkout")
+ checkout.Traffic(seed.Daily(1500), seed.RateFunc(load), seed.Over(window))
+ checkout.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.03, Max: 14})
+ checkout.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.8, 2.2)) // auth dependency, body validation
+ wait, ok := waitFor(r)
+ r.Gap(wait)
+ cart := fmt.Sprintf("%08x-%04x-4%03x-%04x-%06x%06x", r.IntN(1<<30)|1<<28, r.IntN(1<<16), r.IntN(1<<12), 0x8000+r.IntN(1<<14), r.IntN(1<<24), r.IntN(1<<24))
+ if !ok {
+ r.Log(seed.SevWarn, "checkout failed for cart "+cart+": "+pdbPoolTimeout, seed.A("cart_id", cart))
+ poolTimeout(r, "/api/checkout", " File \"/app/app/checkout.py\", line 32, in create_checkout\n return _checkout(body, x_user_id)\n"+
+ " File \"/app/app/checkout.py\", line 40, in _checkout\n items = session.execute(select(CartItem).where(CartItem.cart_id == body.cart_id)).scalars().all()\n")
+ return
+ }
+ lines := 1 + r.IntN(5)
+ r.DB("SELECT cart_items.sku, cart_items.qty, cart_items.unit_price_cents FROM cart_items WHERE cart_items.cart_id = %(cart_id_1)s::UUID", r.Ms(2, 5), seed.Rows(lines))
+ r.DB("SELECT inventory.sku, inventory.available FROM inventory WHERE inventory.sku = ANY (%(sku_1)s::VARCHAR[])", r.Ms(40, 60), seed.Rows(lines))
+ r.HTTP("POST", "https://api.payments.example/v1/charges", r.Ms(150, 260), 201)
+ r.DB("INSERT INTO orders (id, cart_id, user_id, total_cents, status, created_at) VALUES (%(id)s::UUID, %(cart_id)s::UUID, %(user_id)s::INTEGER, %(total_cents)s::INTEGER, %(status)s::VARCHAR, now())", r.Ms(4, 8))
+ r.DB("UPDATE inventory SET available=(inventory.available - %(param_1)s::INTEGER) WHERE inventory.sku = %(sku_1)s::VARCHAR", r.Ms(3, 6))
+ r.DB("DELETE FROM cart_items WHERE cart_items.cart_id = %(cart_id_1)s::UUID", r.Ms(2, 4))
+ r.Gap(r.Ms(0.5, 1.5))
+ r.Status(201)
+ })
+
+ cartEp := api.Endpoint("GET", "/api/cart")
+ cartEp.Traffic(seed.Daily(1100), seed.RateFunc(load), seed.Over(window))
+ cartEp.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.02, Max: 8})
+ cartEp.Handler(func(r *seed.Req) {
+ r.Gap(r.Ms(0.6, 1.6))
+ wait, ok := waitFor(r)
+ r.Gap(wait)
+ if !ok {
+ poolTimeout(r, "/api/cart", " File \"/app/app/cart.py\", line 18, in get_cart\n rows = session.execute(select(CartItem).where(CartItem.cart_id == cart_id)).scalars().all()\n")
+ return
+ }
+ r.DB("SELECT cart_items.sku, cart_items.qty, cart_items.unit_price_cents FROM cart_items WHERE cart_items.cart_id = %(cart_id_1)s::UUID", r.Ms(2, 5), seed.Rows(1+r.IntN(5)))
+ r.Gap(r.Ms(0.4, 1.0))
+ })
+
+ // Shipping options come from config; no connection needed.
+ shipping := api.Endpoint("GET", "/api/shipping/options")
+ shipping.Traffic(seed.Daily(500), seed.RateFunc(load), seed.Over(window))
+ shipping.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.01, Max: 3})
+ shipping.Handler(func(r *seed.Req) { r.Gap(r.Ms(0.6, 1.8)) })
+
+ // opentelemetry-instrumentation-sqlalchemy's pool usage metric.
+ pool := map[string]any{"db.client.connection.pool.name": "checkout-db.internal:5432/checkout"}
+ used := func(t time.Time) float64 {
+ if over(t) > 0 {
+ return 15
+ }
+ busy := 1 + 13*math.Pow((load(t)-1)/1.7, 1.4)*(0.9+0.2*math.Sin(float64(t.Unix()/600)))
+ return math.Min(14, math.Round(busy))
+ }
+ withState := func(state string) map[string]any {
+ m := map[string]any{"db.client.connection.state": state}
+ for k, v := range pool {
+ m[k] = v
+ }
+ return m
+ }
+ api.Gauge("db.client.connection.count", "{connection}", withState("used"), used)
+ api.Gauge("db.client.connection.count", "{connection}", withState("idle"), func(t time.Time) float64 { return math.Max(0, 10-used(t)) })
+
+ api.Host().CPU(seed.Plus(seed.Flat(0.27, 0.34), func(t time.Time) float64 { return 0.05 * bump(t) }))
+ api.Host().Memory(seed.Flat(0.49, 0.55))
+
+ day := s.Ago(24 * time.Hour)
+ // A slow one a customer would notice (10+ s), not a request that waited
+ // out the whole pool_timeout.
+ s.Expose("slowRequest", pdbSlowest(checkout, day, func(r *seed.Request) bool {
+ return r.Status == 201 && r.Duration() < 15*time.Second
+ }))
+ s.Expose("p50Ms24h", pdbMs(checkout, 0.5, day, s.Now))
+ s.Expose("p95Ms24h", pdbMs(checkout, 0.95, day, s.Now))
+ s.Expose("p99Ms24h", pdbMs(checkout, 0.99, day, s.Now))
+ s.Expose("cartP99Ms24h", pdbMs(cartEp, 0.99, day, s.Now))
+ s.Expose("timeouts", func() any {
+ n := 0
+ for _, e := range []*seed.Endpoint{checkout, cartEp} {
+ for _, r := range e.Requests() {
+ if r.Status == 500 {
+ n++
+ }
+ }
+ }
+ return n
+ })
+ return nil
+}
diff --git a/cli/test/evals/internal/scenarios/registry.go b/cli/test/evals/internal/scenarios/registry.go
new file mode 100644
index 00000000..95ad7ffe
--- /dev/null
+++ b/cli/test/evals/internal/scenarios/registry.go
@@ -0,0 +1,75 @@
+// Package scenarios holds one Go seed function per case, registered by the
+// name a case's `seed:` field uses (perf/db-n-plus-1, ...). The per-suite
+// subpackages register themselves; import scenarios/all to load every one.
+package scenarios
+
+import (
+ "fmt"
+ "sort"
+ "sync"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+// Func seeds one scenario.
+type Func func(s *seed.Scenario) error
+
+var (
+ mu sync.RWMutex
+ registry = map[string]Func{}
+)
+
+// Register adds a seed function. Duplicate names panic at init.
+func Register(name string, f Func) {
+ mu.Lock()
+ defer mu.Unlock()
+ if _, dup := registry[name]; dup {
+ panic("scenarios: duplicate seed " + name)
+ }
+ registry[name] = f
+}
+
+// Get returns a registered seed function.
+func Get(name string) (Func, error) {
+ mu.RLock()
+ defer mu.RUnlock()
+ f, ok := registry[name]
+ if !ok {
+ return nil, fmt.Errorf("no seed function registered as %q", name)
+ }
+ return f, nil
+}
+
+// Names lists every registered seed.
+func Names() []string {
+ mu.RLock()
+ defer mu.RUnlock()
+ out := make([]string, 0, len(registry))
+ for k := range registry {
+ out = append(out, k)
+ }
+ sort.Strings(out)
+ return out
+}
+
+// Run seeds s with the named function, converting panics from the DSL (which
+// panics on API errors to keep scenario code flat) into errors.
+func Run(name string, s *seed.Scenario) (err error) {
+ f, err := Get(name)
+ if err != nil {
+ return err
+ }
+ defer func() {
+ if r := recover(); r != nil {
+ if e, ok := r.(error); ok {
+ err = fmt.Errorf("seed %s: %w", name, e)
+ return
+ }
+ err = fmt.Errorf("seed %s: %v", name, r)
+ }
+ }()
+ if err := f(s); err != nil {
+ return fmt.Errorf("seed %s: %w", name, err)
+ }
+ return s.Flush()
+}
diff --git a/cli/test/evals/internal/seed/dsl.go b/cli/test/evals/internal/seed/dsl.go
new file mode 100644
index 00000000..c260e1fd
--- /dev/null
+++ b/cli/test/evals/internal/seed/dsl.go
@@ -0,0 +1,743 @@
+// Package seed builds realistic Traceway telemetry for one eval scenario and
+// sends it through the real ingest endpoints (OTLP/HTTP and native
+// /api/report). The DSL owns consistency: a request's duration is the sum of
+// its children plus explicit gaps, percentiles come from the generated
+// requests, and deploys are service.version changes on the resource.
+package seed
+
+import (
+ "crypto/sha256"
+ "encoding/binary"
+ "fmt"
+ "math"
+ "math/rand/v2"
+ "sort"
+ "time"
+)
+
+// ProjectKind selects how a project's telemetry arrives.
+type ProjectKind int
+
+const (
+ // OTel is a backend project fed over OTLP (framework opentelemetry).
+ OTel ProjectKind = iota
+ // Browser is a frontend project fed by a Traceway browser SDK.
+ Browser
+ // Mobile is a mobile project fed by a Traceway mobile SDK.
+ Mobile
+)
+
+// Scenario is the state of one seed run.
+type Scenario struct {
+ CaseID string
+ Now time.Time
+ RNG *rand.Rand
+ Sink Sink
+
+ projects []*Project
+ services []*Service
+ exposed map[string]any
+ lazy map[string]func() any
+ strings []string
+ flushed bool
+ stats Stats
+ // SpanBudget caps the spans sent with full detail (default 20000).
+ SpanBudget int
+ afterFlush []func() error
+}
+
+// Stats summarizes what a seed sent.
+type Stats struct {
+ Requests int `json:"requests"`
+ Spans int `json:"spans"`
+ DetailedReqs int `json:"detailedRequests"`
+ Logs int `json:"logs"`
+ MetricPoints int `json:"metricPoints"`
+ Exceptions int `json:"exceptions"`
+ NativeFrames int `json:"nativeFrames"`
+ IngestElapsed time.Duration `json:"ingestElapsed"`
+}
+
+// NewScenario starts a scenario anchored at now (truncated to the minute)
+// with an RNG seeded from the case id, so a case always draws the same data.
+func NewScenario(caseID string, now time.Time, sink Sink) *Scenario {
+ sum := sha256.Sum256([]byte(caseID))
+ seed1 := binary.BigEndian.Uint64(sum[:8])
+ seed2 := binary.BigEndian.Uint64(sum[8:16])
+ return &Scenario{
+ CaseID: caseID,
+ Now: now.UTC().Truncate(time.Minute),
+ RNG: rand.New(rand.NewPCG(seed1, seed2)),
+ Sink: sink,
+ exposed: map[string]any{},
+ lazy: map[string]func() any{},
+ SpanBudget: 20000,
+ }
+}
+
+// Stats returns what was sent.
+func (s *Scenario) Stats() Stats { return s.stats }
+
+// Strings returns every human-readable string the seed emitted (span names,
+// attribute values, log bodies, stack traces, object names), for the leakage
+// lint.
+func (s *Scenario) Strings() []string { return s.strings }
+
+func (s *Scenario) record(v ...string) {
+ for _, x := range v {
+ if x != "" {
+ s.strings = append(s.strings, x)
+ }
+ }
+}
+
+// Project declares a Traceway project. framework defaults per kind.
+func (s *Scenario) Project(name string, kind ProjectKind, framework ...string) *Project {
+ fw := "opentelemetry"
+ switch kind {
+ case Browser:
+ fw = "react"
+ case Mobile:
+ fw = "android"
+ }
+ if len(framework) > 0 {
+ fw = framework[0]
+ }
+ p := &Project{s: s, Name: name, Kind: kind, Framework: fw}
+ id, token, err := s.Sink.CreateProject(name, fw)
+ if err != nil {
+ panic(fmt.Errorf("create project %s: %w", name, err))
+ }
+ p.ID, p.token = id, token
+ s.projects = append(s.projects, p)
+ s.record(name)
+ return p
+}
+
+// Projects returns the declared projects in order.
+func (s *Scenario) Projects() []*Project { return s.projects }
+
+// Expose records a value for seed.json (graders and the oracle read it; the
+// agent never sees it). A func() any value is resolved at Flush.
+func (s *Scenario) Expose(key string, v any) {
+ if f, ok := v.(func() any); ok {
+ s.lazy[key] = f
+ return
+ }
+ s.exposed[key] = v
+}
+
+// Exposed returns the resolved exposed values.
+func (s *Scenario) Exposed() map[string]any {
+ out := map[string]any{}
+ for k, v := range s.exposed {
+ out[k] = v
+ }
+ return out
+}
+
+// AfterFlush schedules work that needs the telemetry in place (for example
+// objects that reference an ingested exception).
+func (s *Scenario) AfterFlush(f func() error) { s.afterFlush = append(s.afterFlush, f) }
+
+// Ago returns Now minus d.
+func (s *Scenario) Ago(d time.Duration) time.Time { return s.Now.Add(-d) }
+
+// Project is one Traceway project.
+type Project struct {
+ s *Scenario
+ Name string
+ Kind ProjectKind
+ Framework string
+ ID string
+ token string
+
+ native []nativeFrame
+}
+
+// Service declares an OTel-instrumented service reporting to the project.
+func (p *Project) Service(name string, stack Stack) *Service {
+ svc := &Service{s: p.s, p: p, Name: name, Stack: stack, Env: "production", Hosts: []string{name + "-" + hostSuffix(p.s, name)}}
+ p.s.services = append(p.s.services, svc)
+ p.s.record(name)
+ return svc
+}
+
+func hostSuffix(s *Scenario, name string) string {
+ h := sha256.Sum256([]byte(s.CaseID + name))
+ return fmt.Sprintf("%x-%x", h[0:3], h[3:5])
+}
+
+// Service is one deployable component.
+type Service struct {
+ s *Scenario
+ p *Project
+ Name string
+ Stack Stack
+ Env string
+ Hosts []string
+ deploys []Deploy
+ // HostWeight biases request placement per host (default uniform).
+ HostWeight map[string]float64
+
+ endpoints []*Endpoint
+ tasks []*Endpoint
+ metrics []*metricSeries
+ logs []logRecord
+ defaultHost bool
+}
+
+// Deploy is a release: service.version changes at At.
+type Deploy struct {
+ At time.Time
+ Version string
+}
+
+// Project returns the owning project.
+func (svc *Service) Project() *Project { return svc.p }
+
+// Instances replaces the host list (multi-instance service).
+func (svc *Service) Instances(hosts ...string) *Service {
+ svc.Hosts = hosts
+ svc.s.record(hosts...)
+ return svc
+}
+
+// Versions sets the release history: the first version applies from the
+// start, each later one from its deploy time.
+func (svc *Service) Versions(initial string, deploys ...Deploy) *Service {
+ svc.deploys = append([]Deploy{{At: time.Time{}, Version: initial}}, deploys...)
+ sort.SliceStable(svc.deploys, func(i, j int) bool { return svc.deploys[i].At.Before(svc.deploys[j].At) })
+ return svc
+}
+
+// VersionAt returns the service.version live at t.
+func (svc *Service) VersionAt(t time.Time) string {
+ if len(svc.deploys) == 0 {
+ return "1.0.0"
+ }
+ v := svc.deploys[0].Version
+ for _, d := range svc.deploys[1:] {
+ if !t.Before(d.At) {
+ v = d.Version
+ }
+ }
+ return v
+}
+
+// Endpoint declares an HTTP route served by the service.
+func (svc *Service) Endpoint(method, route string) *Endpoint {
+ e := &Endpoint{svc: svc, Method: method, Route: route, policy: DetailPolicy{Recent: 6 * time.Hour, Rate: 0.03, Max: 60}}
+ svc.endpoints = append(svc.endpoints, e)
+ svc.s.record(method + " " + route)
+ return e
+}
+
+// Task declares a background task (a CONSUMER span root).
+func (svc *Service) Task(name string) *Endpoint {
+ e := &Endpoint{svc: svc, Task: true, Route: name, policy: DetailPolicy{Recent: 6 * time.Hour, Rate: 0.05, Max: 40}}
+ svc.tasks = append(svc.tasks, e)
+ svc.s.record(name)
+ return e
+}
+
+// Endpoint is an HTTP route (or a task, when Task is set).
+type Endpoint struct {
+ svc *Service
+ Method string
+ Route string
+ Task bool
+ shapes []Shape
+ window time.Duration
+ handler func(r *Req)
+ policy DetailPolicy
+ reqs []*Request
+ gen bool
+ // Status picks the response status; nil means 200.
+ status func(r *Req) int
+ // SkipAccessLog disables the per-request access log line.
+ SkipAccessLog bool
+}
+
+// Name returns "METHOD /route" (or the task name).
+func (e *Endpoint) Name() string {
+ if e.Task {
+ return e.Route
+ }
+ return e.Method + " " + e.Route
+}
+
+// Service returns the serving service.
+func (e *Endpoint) Service() *Service { return e.svc }
+
+// Traffic sets the arrival process: rate shapes multiplied together over a
+// window (Over, default 24h).
+func (e *Endpoint) Traffic(shapes ...Shape) *Endpoint {
+ for _, sh := range shapes {
+ if o, ok := sh.(over); ok {
+ e.window = time.Duration(o)
+ continue
+ }
+ e.shapes = append(e.shapes, sh)
+ }
+ return e
+}
+
+// Handler sets the per-request span builder.
+func (e *Endpoint) Handler(f func(r *Req)) *Endpoint { e.handler = f; return e }
+
+// Detail tunes which requests are sent with their full span tree. Requests
+// that carry logs or exceptions, or are exposed, always are.
+func (e *Endpoint) Detail(p DetailPolicy) *Endpoint { e.policy = p; return e }
+
+// DetailPolicy selects requests to send with full span trees: every request
+// in the last Recent window with probability Rate, at most Max of them.
+type DetailPolicy struct {
+ Recent time.Duration
+ Rate float64
+ Max int
+}
+
+// Shape is a factor on the arrival rate.
+type Shape interface {
+ factor(s *Scenario, t time.Time, start, end time.Time) float64
+}
+
+type daily float64
+
+func (d daily) factor(*Scenario, time.Time, time.Time, time.Time) float64 { return float64(d) }
+
+// Daily sets the base rate in requests per day.
+func Daily(n int) Shape { return daily(n) }
+
+type diurnal struct{}
+
+func (diurnal) factor(_ *Scenario, t, _, _ time.Time) float64 {
+ h := float64(t.Hour()) + float64(t.Minute())/60
+ // Peak around 14:00 UTC, trough around 02:00; mean ~1.
+ return 1 + 0.6*math.Cos((h-14)/24*2*math.Pi)
+}
+
+// Diurnal applies a day/night cycle (peak mid-afternoon UTC).
+func Diurnal() Shape { return diurnal{} }
+
+type growth struct{ from, to float64 }
+
+func (g growth) factor(_ *Scenario, t, start, end time.Time) float64 {
+ return g.from + (g.to-g.from)*progress(t, start, end)
+}
+
+// GrowRate multiplies the rate from `from` at the window start to `to` at its end.
+func GrowRate(from, to float64) Shape { return growth{from, to} }
+
+type window struct {
+ from, to time.Time
+ mult float64
+}
+
+func (w window) factor(_ *Scenario, t, _, _ time.Time) float64 {
+ if !t.Before(w.from) && t.Before(w.to) {
+ return w.mult
+ }
+ return 1
+}
+
+// Burst multiplies the rate inside [from, to).
+func Burst(from, to time.Time, mult float64) Shape { return window{from, to, mult} }
+
+type funcShape func(t time.Time) float64
+
+func (f funcShape) factor(_ *Scenario, t, _, _ time.Time) float64 { return f(t) }
+
+// RateFunc is an arbitrary rate factor.
+func RateFunc(f func(t time.Time) float64) Shape { return funcShape(f) }
+
+type over time.Duration
+
+func (over) factor(*Scenario, time.Time, time.Time, time.Time) float64 { return 1 }
+
+// Over sets the traffic window ending now.
+func Over(d time.Duration) Shape { return over(d) }
+
+func progress(t, start, end time.Time) float64 {
+ if !end.After(start) {
+ return 1
+ }
+ p := float64(t.Sub(start)) / float64(end.Sub(start))
+ return math.Max(0, math.Min(1, p))
+}
+
+// Request is one generated request (or task run).
+type Request struct {
+ e *Endpoint
+ Index int
+ Start time.Time
+ Host string
+ Version string
+ TraceID [16]byte
+ SpanID [8]byte
+ Root *Span
+ Status int
+ Detail bool
+ forced bool
+ Progress float64
+ ParentRef *Span // for downstream SERVER spans: the CLIENT span calling it
+}
+
+// Duration is the request's total duration.
+func (r *Request) Duration() time.Duration { return r.Root.Dur }
+
+// TraceHex is the 32-hex trace id.
+func (r *Request) TraceHex() string { return fmt.Sprintf("%x", r.TraceID) }
+
+// SpanHex is the 16-hex root span id.
+func (r *Request) SpanHex() string { return fmt.Sprintf("%x", r.SpanID) }
+
+// Endpoint returns the request's endpoint.
+func (r *Request) Endpoint() *Endpoint { return r.e }
+
+// KeepDetail forces the full span tree to be sent.
+func (r *Request) KeepDetail() { r.forced = true; r.Detail = true }
+
+// Ref is what seed.json carries for a request.
+func (r *Request) Ref() map[string]any {
+ return map[string]any{
+ "id": EntityID(r.e.svc.p.ID, r.TraceID, r.SpanID),
+ "recordedAt": r.Start.UTC().Format(time.RFC3339Nano),
+ "traceId": r.TraceHex(),
+ "spanId": r.SpanHex(),
+ "durationMs": float64(r.Duration()) / float64(time.Millisecond),
+ "endpoint": r.e.Name(),
+ "service": r.e.svc.Name,
+ "host": r.Host,
+ "version": r.Version,
+ "project": r.e.svc.p.Name,
+ }
+}
+
+// Span is one span in a request tree. Start is the offset from the
+// request's start.
+type Span struct {
+ Name string
+ Kind int
+ Start time.Duration
+ Dur time.Duration
+ Attrs map[string]any
+ Status int
+ StatusMsg string
+ Events []Event
+ Children []*Span
+ Scope string
+ SpanID [8]byte
+ // Service is set when the span belongs to another service (a
+ // downstream SERVER span and its subtree).
+ Service *Service
+ Host string
+ logs []logRecord
+}
+
+// Event is a span event.
+type Event struct {
+ Name string
+ At time.Duration
+ Attrs map[string]any
+}
+
+// Requests generates (once) and returns the endpoint's requests in time order.
+func (e *Endpoint) Requests() []*Request {
+ if e.gen {
+ return e.reqs
+ }
+ e.gen = true
+ if e.handler == nil {
+ panic(fmt.Errorf("endpoint %s has no handler", e.Name()))
+ }
+ s := e.svc.s
+ win := e.window
+ if win == 0 {
+ win = 24 * time.Hour
+ }
+ end := s.Now.Add(-time.Minute)
+ start := end.Add(-win)
+ base := 0.0
+ var mods []Shape
+ for _, sh := range e.shapes {
+ if d, ok := sh.(daily); ok {
+ base += float64(d)
+ continue
+ }
+ mods = append(mods, sh)
+ }
+ if base == 0 {
+ panic(fmt.Errorf("endpoint %s: traffic needs a Daily(n) base rate", e.Name()))
+ }
+ // Walk the window minute by minute; expected arrivals per minute is the
+ // base rate times every shape factor. Fractional arrivals carry over so
+ // low rates still produce requests (deterministic dithering).
+ carry := 0.0
+ idx := 0
+ for t := start; t.Before(end); t = t.Add(time.Minute) {
+ lambda := base / 1440
+ for _, sh := range mods {
+ lambda *= sh.factor(s, t, start, end)
+ }
+ carry += lambda
+ n := int(carry)
+ carry -= float64(n)
+ for k := 0; k < n; k++ {
+ at := t.Add(time.Duration(s.RNG.Int64N(int64(time.Minute))))
+ r := e.newRequest(idx, at, progress(at, start, end))
+ idx++
+ e.reqs = append(e.reqs, r)
+ }
+ }
+ sort.SliceStable(e.reqs, func(i, j int) bool { return e.reqs[i].Start.Before(e.reqs[j].Start) })
+ e.selectDetail()
+ return e.reqs
+}
+
+func (e *Endpoint) newRequest(idx int, at time.Time, prog float64) *Request {
+ s := e.svc.s
+ r := &Request{e: e, Index: idx, Start: at, Progress: prog}
+ r.Host = e.svc.pickHost()
+ r.Version = e.svc.VersionAt(at)
+ fillRandom(s, r.TraceID[:])
+ fillRandom(s, r.SpanID[:])
+ kind := KindServer
+ if e.Task {
+ kind = KindConsumer
+ }
+ root := &Span{Name: e.Name(), Kind: kind, Scope: e.svc.Stack.ServerScope, SpanID: r.SpanID, Attrs: map[string]any{}, Host: r.Host}
+ r.Root = root
+ b := &Req{s: s, req: r, cur: root}
+ if !e.Task {
+ root.Attrs["http.request.method"] = e.Method
+ root.Attrs["http.route"] = e.Route
+ root.Attrs["url.path"] = concretePath(s, e.Route)
+ root.Attrs["url.scheme"] = "https"
+ root.Attrs["network.protocol.version"] = "1.1"
+ root.Attrs["user_agent.original"] = pick(s, userAgents)
+ root.Attrs["client.address"] = fmt.Sprintf("10.%d.%d.%d", 1+s.RNG.IntN(30), s.RNG.IntN(255), 1+s.RNG.IntN(250))
+ } else {
+ root.Attrs["messaging.operation.type"] = "process"
+ }
+ e.handler(b)
+ b.finish()
+ r.Status = 200
+ if e.status != nil {
+ r.Status = e.status(b)
+ }
+ if b.status != 0 {
+ r.Status = b.status
+ }
+ if !e.Task {
+ root.Attrs["http.response.status_code"] = r.Status
+ if r.Status >= 500 {
+ root.Status = StatusError
+ }
+ }
+ if b.forceDetail {
+ r.forced, r.Detail = true, true
+ }
+ return r
+}
+
+// Statuses sets a per-request status picker.
+func (e *Endpoint) Statuses(f func(r *Req) int) *Endpoint { e.status = f; return e }
+
+func (e *Endpoint) selectDetail() {
+ s := e.svc.s
+ cut := s.Now.Add(-e.policy.Recent)
+ // Forced detail (logs, errors) never uses up the sample's cap.
+ n := 0
+ // Prefer the newest requests so the detail sample sits where the agent
+ // looks first, but keep it random within the window.
+ for i := len(e.reqs) - 1; i >= 0 && n < e.policy.Max; i-- {
+ r := e.reqs[i]
+ if r.Detail || r.Start.Before(cut) {
+ continue
+ }
+ if s.RNG.Float64() < e.policy.Rate {
+ r.Detail = true
+ n++
+ }
+ }
+}
+
+func (svc *Service) pickHost() string {
+ if len(svc.Hosts) == 1 {
+ return svc.Hosts[0]
+ }
+ total := 0.0
+ for _, h := range svc.Hosts {
+ total += svc.weight(h)
+ }
+ x := svc.s.RNG.Float64() * total
+ for _, h := range svc.Hosts {
+ x -= svc.weight(h)
+ if x <= 0 {
+ return h
+ }
+ }
+ return svc.Hosts[len(svc.Hosts)-1]
+}
+
+func (svc *Service) weight(h string) float64 {
+ if w, ok := svc.HostWeight[h]; ok {
+ return w
+ }
+ return 1
+}
+
+func fillRandom(s *Scenario, b []byte) {
+ for i := 0; i < len(b); i += 8 {
+ v := s.RNG.Uint64()
+ for j := 0; j < 8 && i+j < len(b); j++ {
+ b[i+j] = byte(v >> (8 * j))
+ }
+ }
+ if allZero(b) {
+ b[0] = 1
+ }
+}
+
+func allZero(b []byte) bool {
+ for _, x := range b {
+ if x != 0 {
+ return false
+ }
+ }
+ return true
+}
+
+// Slowest returns a lazy reference to the slowest request in the trailing
+// window, for Expose.
+func (e *Endpoint) Slowest(within time.Duration) func() any {
+ return func() any {
+ r := e.SlowestRequest(within)
+ if r == nil {
+ return nil
+ }
+ return r.Ref()
+ }
+}
+
+// SlowestRequest returns the slowest request in the trailing window and
+// forces it to be sent in detail.
+func (e *Endpoint) SlowestRequest(within time.Duration) *Request {
+ var best *Request
+ cut := e.svc.s.Now.Add(-within)
+ for _, r := range e.Requests() {
+ if r.Start.Before(cut) {
+ continue
+ }
+ if best == nil || r.Duration() > best.Duration() {
+ best = r
+ }
+ }
+ if best != nil {
+ best.KeepDetail()
+ }
+ return best
+}
+
+// Percentile returns the p-quantile (0..1) of durations in the trailing window.
+func (e *Endpoint) Percentile(p float64, within time.Duration) time.Duration {
+ var ds []time.Duration
+ cut := e.svc.s.Now.Add(-within)
+ for _, r := range e.Requests() {
+ if !r.Start.Before(cut) {
+ ds = append(ds, r.Duration())
+ }
+ }
+ return quantile(ds, p)
+}
+
+// PercentileBetween returns the p-quantile of durations started in [from, to).
+func (e *Endpoint) PercentileBetween(p float64, from, to time.Time) time.Duration {
+ var ds []time.Duration
+ for _, r := range e.Requests() {
+ if !r.Start.Before(from) && r.Start.Before(to) {
+ ds = append(ds, r.Duration())
+ }
+ }
+ return quantile(ds, p)
+}
+
+func quantile(ds []time.Duration, p float64) time.Duration {
+ if len(ds) == 0 {
+ return 0
+ }
+ sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })
+ pos := p * float64(len(ds)-1)
+ lo := int(math.Floor(pos))
+ hi := int(math.Ceil(pos))
+ frac := pos - float64(lo)
+ return time.Duration(float64(ds[lo])*(1-frac) + float64(ds[hi])*frac)
+}
+
+// Pick returns a lazy reference to the first request matching f (newest
+// first), forcing it to detail.
+func (e *Endpoint) Pick(f func(r *Request) bool) func() any {
+ return func() any {
+ r := e.PickRequest(f)
+ if r == nil {
+ return nil
+ }
+ return r.Ref()
+ }
+}
+
+// PickRequest returns the newest request matching f and forces its detail.
+func (e *Endpoint) PickRequest(f func(r *Request) bool) *Request {
+ reqs := e.Requests()
+ for i := len(reqs) - 1; i >= 0; i-- {
+ if f(reqs[i]) {
+ reqs[i].KeepDetail()
+ return reqs[i]
+ }
+ }
+ return nil
+}
+
+// Count returns how many requests started in the trailing window.
+func (e *Endpoint) Count(within time.Duration) int {
+ cut := e.svc.s.Now.Add(-within)
+ n := 0
+ for _, r := range e.Requests() {
+ if !r.Start.Before(cut) {
+ n++
+ }
+ }
+ return n
+}
+
+var userAgents = []string{
+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15",
+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
+ "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148",
+ "okhttp/4.12.0",
+}
+
+func pick[T any](s *Scenario, xs []T) T { return xs[s.RNG.IntN(len(xs))] }
+
+// concretePath turns a route template into a plausible concrete path.
+func concretePath(s *Scenario, route string) string {
+ out := []byte{}
+ for i := 0; i < len(route); i++ {
+ if route[i] == ':' || route[i] == '{' {
+ j := i + 1
+ for j < len(route) && route[j] != '/' && route[j] != '}' {
+ j++
+ }
+ if j < len(route) && route[j] == '}' {
+ j++
+ }
+ out = append(out, []byte(fmt.Sprint(1000+s.RNG.IntN(90000)))...)
+ i = j - 1
+ continue
+ }
+ out = append(out, route[i])
+ }
+ return string(out)
+}
diff --git a/cli/test/evals/internal/seed/flush.go b/cli/test/evals/internal/seed/flush.go
new file mode 100644
index 00000000..3f2affc8
--- /dev/null
+++ b/cli/test/evals/internal/seed/flush.go
@@ -0,0 +1,223 @@
+package seed
+
+import (
+ "fmt"
+ "sort"
+ "time"
+
+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ "google.golang.org/protobuf/proto"
+)
+
+const (
+ pathTraces = "/api/otel/v1/traces"
+ pathMetrics = "/api/otel/v1/metrics"
+ pathLogs = "/api/otel/v1/logs"
+ pathReport = "/api/report"
+ protoType = "application/x-protobuf"
+ // spansPerRequest keeps each OTLP body far below the 10MB ingest cap.
+ spansPerRequest = 1500
+)
+
+// Flush generates every request, resolves lazy exposes, enforces the span
+// budget and sends everything: traces, logs, metrics, native SDK frames.
+func (s *Scenario) Flush() error {
+ if s.flushed {
+ return fmt.Errorf("scenario flushed twice")
+ }
+ s.flushed = true
+ started := time.Now()
+
+ for _, svc := range s.services {
+ applyDefaults(svc)
+ }
+ var all []*Request
+ for _, svc := range s.services {
+ for _, e := range append(append([]*Endpoint{}, svc.endpoints...), svc.tasks...) {
+ if len(e.shapes) == 0 {
+ continue // only reached through Call
+ }
+ all = append(all, e.Requests()...)
+ }
+ }
+ keys := make([]string, 0, len(s.lazy))
+ for k := range s.lazy {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ for _, k := range keys {
+ s.exposed[k] = s.lazy[k]()
+ }
+ s.enforceBudget(all)
+ sort.SliceStable(all, func(i, j int) bool { return all[i].Start.Before(all[j].Start) })
+
+ batches := map[*Project]*spanBatch{}
+ logs := map[*Service][]logOut{}
+ for _, r := range all {
+ s.stats.Requests++
+ if r.Detail {
+ s.stats.DetailedReqs++
+ }
+ s.emit(r, batches, logs)
+ for p, b := range batches {
+ if b.count >= spansPerRequest {
+ if err := s.sendTraces(p, b); err != nil {
+ return err
+ }
+ delete(batches, p)
+ }
+ }
+ }
+ projects := make([]*Project, 0, len(batches))
+ for p := range batches {
+ projects = append(projects, p)
+ }
+ sort.Slice(projects, func(i, j int) bool { return projects[i].Name < projects[j].Name })
+ for _, p := range projects {
+ if err := s.sendTraces(p, batches[p]); err != nil {
+ return err
+ }
+ }
+
+ for _, svc := range s.services {
+ outs := logs[svc]
+ for _, l := range svc.logs {
+ outs = append(outs, logOut{rec: l, at: l.abs, host: l.host, version: svc.VersionAt(l.abs), noTrace: true})
+ }
+ if len(outs) == 0 {
+ continue
+ }
+ sort.SliceStable(outs, func(i, j int) bool { return outs[i].at.Before(outs[j].at) })
+ for i := 0; i < len(outs); i += 2000 {
+ j := min(i+2000, len(outs))
+ body, err := proto.Marshal(logsRequest(svc, outs[i:j]))
+ if err != nil {
+ return err
+ }
+ if _, err := s.Sink.Ingest(svc.p.token, pathLogs, protoType, body, true); err != nil {
+ return fmt.Errorf("logs for %s: %w", svc.Name, err)
+ }
+ s.stats.Logs += j - i
+ }
+ }
+
+ for _, svc := range s.services {
+ for _, req := range s.metricRequests(svc) {
+ body, err := proto.Marshal(req)
+ if err != nil {
+ return err
+ }
+ if _, err := s.Sink.Ingest(svc.p.token, pathMetrics, protoType, body, true); err != nil {
+ return fmt.Errorf("metrics for %s: %w", svc.Name, err)
+ }
+ }
+ }
+
+ recordings := 0
+ for _, p := range s.projects {
+ for _, f := range p.native {
+ recordings += len(f.recordings)
+ }
+ bodies, err := p.nativeBodies()
+ if err != nil {
+ return err
+ }
+ for _, body := range bodies {
+ if _, err := s.Sink.Ingest(p.token, pathReport, "application/json", body, true); err != nil {
+ return fmt.Errorf("native report for %s: %w", p.Name, err)
+ }
+ s.stats.NativeFrames++
+ }
+ }
+
+ if recordings > 0 && !s.Sink.Dry() {
+ // Session recordings are uploaded by a worker pool and inserted in
+ // batches every ~2s; wait so occurrences show their actions.
+ time.Sleep(3 * time.Second)
+ }
+ for _, f := range s.afterFlush {
+ if err := f(); err != nil {
+ return err
+ }
+ }
+ s.stats.IngestElapsed = time.Since(started)
+ return nil
+}
+
+func (s *Scenario) sendTraces(p *Project, b *spanBatch) error {
+ body, err := proto.Marshal(b.request())
+ if err != nil {
+ return err
+ }
+ resp, err := s.Sink.Ingest(p.token, pathTraces, protoType, body, true)
+ if err != nil {
+ return fmt.Errorf("traces for %s: %w", p.Name, err)
+ }
+ if len(resp) > 0 {
+ var out coltracepb.ExportTraceServiceResponse
+ if proto.Unmarshal(resp, &out) == nil && out.GetPartialSuccess().GetRejectedSpans() > 0 {
+ return fmt.Errorf("traces for %s: backend rejected %d spans: %s", p.Name, out.GetPartialSuccess().GetRejectedSpans(), out.GetPartialSuccess().GetErrorMessage())
+ }
+ }
+ return nil
+}
+
+func spanCount(sp *Span) int {
+ n := 1
+ for _, c := range sp.Children {
+ n += spanCount(c)
+ }
+ return n
+}
+
+func serverSpans(sp *Span) int {
+ n := 0
+ if sp.Kind == KindServer {
+ n = 1
+ }
+ for _, c := range sp.Children {
+ n += serverSpans(c)
+ }
+ return n
+}
+
+// enforceBudget keeps the detailed span volume under SpanBudget by dropping
+// sampled (not forced) detail, oldest first.
+func (s *Scenario) enforceBudget(all []*Request) {
+ total := 0
+ for _, r := range all {
+ total += s.sentSpans(r)
+ }
+ if total <= s.SpanBudget {
+ return
+ }
+ sampled := make([]*Request, 0)
+ for _, r := range all {
+ if r.Detail && !r.forced {
+ sampled = append(sampled, r)
+ }
+ }
+ sort.SliceStable(sampled, func(i, j int) bool { return sampled[i].Start.Before(sampled[j].Start) })
+ for _, r := range sampled {
+ if total <= s.SpanBudget {
+ return
+ }
+ before := s.sentSpans(r)
+ r.Detail = false
+ total -= before - s.sentSpans(r)
+ }
+}
+
+func (s *Scenario) sentSpans(r *Request) int {
+ if r.Detail {
+ return spanCount(r.Root)
+ }
+ return 1 + serverSpans(r.Root) - boolInt(r.Root.Kind == KindServer)
+}
+
+func boolInt(b bool) int {
+ if b {
+ return 1
+ }
+ return 0
+}
diff --git a/cli/test/evals/internal/seed/groups.go b/cli/test/evals/internal/seed/groups.go
new file mode 100644
index 00000000..e5c20551
--- /dev/null
+++ b/cli/test/evals/internal/seed/groups.go
@@ -0,0 +1,88 @@
+package seed
+
+import (
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+)
+
+// Group is an exception group read back from the API.
+type Group struct {
+ Hash string `json:"exceptionHash"`
+ StackTrace string `json:"stackTrace"`
+ FirstSeen time.Time `json:"firstSeen"`
+ LastSeen time.Time `json:"lastSeen"`
+ Count int `json:"count"`
+}
+
+// Groups lists the project's exception groups over the last `within`.
+func (p *Project) Groups(within time.Duration, includeArchived bool) ([]Group, error) {
+ if p.s.Sink.Dry() {
+ return nil, nil
+ }
+ res, err := p.s.Sink.Admin(http.MethodPost, "/api/exception-stack-traces", env.ProjectQuery(p.ID), map[string]any{
+ "fromDate": p.s.Now.Add(-within).Format(time.RFC3339), "toDate": p.s.Now.Add(time.Hour).Format(time.RFC3339),
+ "orderBy": "count", "pagination": map[string]int{"page": 1, "pageSize": 100}, "includeArchived": includeArchived,
+ })
+ if err != nil {
+ return nil, err
+ }
+ if err := res.Err("list exceptions"); err != nil {
+ return nil, err
+ }
+ var out struct {
+ Data []Group `json:"data"`
+ }
+ if err := json.Unmarshal(res.Body, &out); err != nil {
+ return nil, err
+ }
+ return out.Data, nil
+}
+
+// GroupMatching returns the hash of the group whose stack contains needle.
+func (p *Project) GroupMatching(within time.Duration, needle string) (string, error) {
+ if p.s.Sink.Dry() {
+ return fmt.Sprintf("%016x", len(needle)), nil
+ }
+ groups, err := p.Groups(within, true)
+ if err != nil {
+ return "", err
+ }
+ for _, g := range groups {
+ if containsFold(g.StackTrace, needle) {
+ return g.Hash, nil
+ }
+ }
+ return "", fmt.Errorf("no exception group in %s contains %q", p.Name, needle)
+}
+
+func containsFold(s, sub string) bool {
+ return len(sub) == 0 || indexFold(s, sub) >= 0
+}
+
+func indexFold(s, sub string) int {
+ n := len(sub)
+ for i := 0; i+n <= len(s); i++ {
+ match := true
+ for j := 0; j < n; j++ {
+ a, b := s[i+j], sub[j]
+ if 'A' <= a && a <= 'Z' {
+ a += 'a' - 'A'
+ }
+ if 'A' <= b && b <= 'Z' {
+ b += 'a' - 'A'
+ }
+ if a != b {
+ match = false
+ break
+ }
+ }
+ if match {
+ return i
+ }
+ }
+ return -1
+}
diff --git a/cli/test/evals/internal/seed/logs.go b/cli/test/evals/internal/seed/logs.go
new file mode 100644
index 00000000..f1231b57
--- /dev/null
+++ b/cli/test/evals/internal/seed/logs.go
@@ -0,0 +1,110 @@
+package seed
+
+import (
+ "sort"
+ "time"
+
+ collogspb "go.opentelemetry.io/proto/otlp/collector/logs/v1"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ logspb "go.opentelemetry.io/proto/otlp/logs/v1"
+)
+
+type logRecord struct {
+ at time.Duration // offset within the owning span
+ abs time.Time // absolute time, for logs outside a request
+ host string
+ severity int
+ body string
+ attrs map[string]any
+ scope string
+}
+
+type logOut struct {
+ rec logRecord
+ at time.Time
+ traceID [16]byte
+ spanID [8]byte
+ host string
+ version string
+ noTrace bool
+}
+
+// Log emits a log line from the service that is not tied to a request (a
+// startup message, a scheduler line, a pool warning without a trace).
+func (svc *Service) Log(at time.Time, severity int, body string, attrs ...KV) {
+ svc.LogOn(svc.Hosts[0], at, severity, body, attrs...)
+}
+
+// LogOn is Log from a specific instance.
+func (svc *Service) LogOn(host string, at time.Time, severity int, body string, attrs ...KV) {
+ svc.logs = append(svc.logs, logRecord{abs: at, host: host, severity: severity, body: body, attrs: kvMap(attrs)})
+ svc.s.record(body)
+ for _, kv := range attrs {
+ if s, ok := kv.V.(string); ok {
+ svc.s.record(s)
+ }
+ }
+}
+
+func logScope(svc *Service) string {
+ switch svc.Stack.Language {
+ case "python":
+ return "opentelemetry.sdk._logs"
+ case "nodejs":
+ return "@opentelemetry/winston-transport"
+ case "java":
+ return "io.opentelemetry.logback-appender-1.0"
+ case "dotnet":
+ return "Microsoft.Extensions.Logging"
+ case "php":
+ return "io.opentelemetry.contrib.php.monolog"
+ default:
+ return "go.opentelemetry.io/contrib/bridges/otelslog"
+ }
+}
+
+// logsRequest renders a service's logs as one OTLP request per resource.
+func logsRequest(svc *Service, outs []logOut) *collogspb.ExportLogsServiceRequest {
+ type key struct{ version, host string }
+ groups := map[key][]*logspb.LogRecord{}
+ var keys []key
+ for _, o := range outs {
+ k := key{o.version, o.host}
+ if _, ok := groups[k]; !ok {
+ keys = append(keys, k)
+ }
+ lr := &logspb.LogRecord{
+ TimeUnixNano: nanos(o.at),
+ ObservedTimeUnixNano: nanos(o.at.Add(3 * time.Millisecond)),
+ SeverityNumber: logspb.SeverityNumber(o.rec.severity),
+ SeverityText: SeverityText(o.rec.severity),
+ Body: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: o.rec.body}},
+ Attributes: attrList(o.rec.attrs),
+ }
+ if !o.noTrace {
+ lr.TraceId = append([]byte{}, o.traceID[:]...)
+ lr.SpanId = append([]byte{}, o.spanID[:]...)
+ lr.Flags = 1
+ }
+ groups[k] = append(groups[k], lr)
+ }
+ sort.Slice(keys, func(i, j int) bool {
+ if keys[i].version != keys[j].version {
+ return keys[i].version < keys[j].version
+ }
+ return keys[i].host < keys[j].host
+ })
+ req := &collogspb.ExportLogsServiceRequest{}
+ for _, k := range keys {
+ recs := groups[k]
+ sort.SliceStable(recs, func(i, j int) bool { return recs[i].TimeUnixNano < recs[j].TimeUnixNano })
+ req.ResourceLogs = append(req.ResourceLogs, &logspb.ResourceLogs{
+ Resource: svc.resource(k.version, k.host),
+ ScopeLogs: []*logspb.ScopeLogs{{
+ Scope: &commonpb.InstrumentationScope{Name: logScope(svc)},
+ LogRecords: recs,
+ }},
+ })
+ }
+ return req
+}
diff --git a/cli/test/evals/internal/seed/metrics.go b/cli/test/evals/internal/seed/metrics.go
new file mode 100644
index 00000000..beeeb582
--- /dev/null
+++ b/cli/test/evals/internal/seed/metrics.go
@@ -0,0 +1,360 @@
+package seed
+
+import (
+ "crypto/sha256"
+ "encoding/binary"
+ "fmt"
+ "math"
+ "sort"
+ "time"
+
+ colmetricspb "go.opentelemetry.io/proto/otlp/collector/metrics/v1"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ metricspb "go.opentelemetry.io/proto/otlp/metrics/v1"
+)
+
+// Series is a value over time. Noise inside a Series is a deterministic
+// function of (series, t), so the same scenario always draws the same curve.
+type Series func(t time.Time) float64
+
+// noise returns a deterministic [0,1) value for a label and a time.
+func noise(label string, t time.Time) float64 {
+ var b [8]byte
+ binary.BigEndian.PutUint64(b[:], uint64(t.Unix()))
+ sum := sha256.Sum256(append([]byte(label), b[:]...))
+ return float64(binary.BigEndian.Uint64(sum[:8])>>11) / float64(1<<53)
+}
+
+// Flat is a value wandering between lo and hi.
+func Flat(lo, hi float64) Series {
+ return func(t time.Time) float64 {
+ // Smooth-ish: blend the current and previous 5-minute noise.
+ a := noise("flat", t.Truncate(5*time.Minute))
+ b := noise("flat", t.Truncate(5*time.Minute).Add(5*time.Minute))
+ f := float64(t.Sub(t.Truncate(5*time.Minute))) / float64(5*time.Minute)
+ return lo + (hi-lo)*(a*(1-f)+b*f)
+ }
+}
+
+// Const is a constant value.
+func Const(v float64) Series { return func(time.Time) float64 { return v } }
+
+// Step switches from before to after at at.
+func Step(at time.Time, before, after Series) Series {
+ return func(t time.Time) float64 {
+ if t.Before(at) {
+ return before(t)
+ }
+ return after(t)
+ }
+}
+
+// Ramp interpolates linearly from `from` at start to `to` at end, plus a
+// little noise (±3% of the span).
+func Ramp(from, to float64, start, end time.Time) Series {
+ return func(t time.Time) float64 {
+ p := progress(t, start, end)
+ return from + (to-from)*p + (to-from)*0.03*(noise("ramp", t.Truncate(time.Minute))-0.5)
+ }
+}
+
+// Scale multiplies a series by f(t).
+func Scale(s Series, f func(t time.Time) float64) Series {
+ return func(t time.Time) float64 { return s(t) * f(t) }
+}
+
+// Plus adds series.
+func Plus(ss ...Series) Series {
+ return func(t time.Time) float64 {
+ v := 0.0
+ for _, s := range ss {
+ v += s(t)
+ }
+ return v
+ }
+}
+
+// Clamp bounds a series to [lo, hi].
+func Clamp(s Series, lo, hi float64) Series {
+ return func(t time.Time) float64 { return math.Max(lo, math.Min(hi, s(t))) }
+}
+
+// Periodic is base, raised to peak for width at every multiple of period.
+func Periodic(period, width time.Duration, base, peak Series) Series {
+ return func(t time.Time) float64 {
+ off := t.Sub(t.Truncate(period))
+ if off < width {
+ return peak(t)
+ }
+ return base(t)
+ }
+}
+
+// Sawtooth climbs from lo to hi over period, then drops back (a leak that
+// is cleared by a restart).
+func Sawtooth(period time.Duration, lo, hi float64) Series {
+ return func(t time.Time) float64 {
+ f := float64(t.Sub(t.Truncate(period))) / float64(period)
+ return lo + (hi-lo)*f + (hi-lo)*0.02*(noise("saw", t.Truncate(time.Minute))-0.5)
+ }
+}
+
+// DiurnalSeries modulates a series with the same day/night cycle as Diurnal.
+func DiurnalSeries(s Series, depth float64) Series {
+ return func(t time.Time) float64 {
+ h := float64(t.Hour()) + float64(t.Minute())/60
+ return s(t) * (1 + depth*math.Cos((h-14)/24*2*math.Pi))
+ }
+}
+
+type metricKind int
+
+const (
+ gaugeKind metricKind = iota
+ counterKind
+ histogramKind
+)
+
+type metricSeries struct {
+ name, unit, desc string
+ kind metricKind
+ attrs map[string]any
+ host string
+ value Series // gauge value; counter rate per second; histogram mean
+ count Series // histogram observations per minute
+ window time.Duration
+ scope string
+ // cumTotal and cumPrev carry a counter's running total across the
+ // chunks metricRequests renders it in.
+ cumTotal float64
+ cumPrev time.Time
+}
+
+// MetricStep controls sampling: points every Recent inside the last day,
+// every Old before that.
+var MetricStep = struct{ Recent, Old time.Duration }{time.Minute, 10 * time.Minute}
+
+// Gauge adds a gauge series on the service's first host.
+func (svc *Service) Gauge(name, unit string, attrs map[string]any, v Series) {
+ svc.GaugeOn(svc.Hosts[0], name, unit, attrs, v)
+}
+
+// GaugeOn adds a gauge series for one instance.
+func (svc *Service) GaugeOn(host, name, unit string, attrs map[string]any, v Series) {
+ label := fmt.Sprint(host, name, attrs)
+ jittered := func(t time.Time) float64 { return v(t) * (1 + 0.03*(noise(label, t)-0.5)) }
+ svc.metrics = append(svc.metrics, &metricSeries{name: name, unit: unit, kind: gaugeKind, attrs: attrs, host: host, value: jittered, scope: metricScope(name)})
+ svc.s.record(name)
+}
+
+// Counter adds a monotonic cumulative sum whose rate per second is v.
+func (svc *Service) Counter(name, unit string, attrs map[string]any, ratePerSec Series) {
+ svc.metrics = append(svc.metrics, &metricSeries{name: name, unit: unit, kind: counterKind, attrs: attrs, host: svc.Hosts[0], value: ratePerSec, scope: metricScope(name)})
+ svc.s.record(name)
+}
+
+// Histogram adds a delta histogram: mean value and observations per minute.
+// The backend stores it as <name>.avg and <name>.count.
+func (svc *Service) Histogram(name, unit string, attrs map[string]any, mean, perMinute Series) {
+ svc.HistogramOn(svc.Hosts[0], name, unit, attrs, mean, perMinute)
+}
+
+// HistogramOn is Histogram for one instance.
+func (svc *Service) HistogramOn(host, name, unit string, attrs map[string]any, mean, perMinute Series) {
+ svc.metrics = append(svc.metrics, &metricSeries{name: name, unit: unit, kind: histogramKind, attrs: attrs, host: host, value: mean, count: perMinute, scope: metricScope(name)})
+ svc.s.record(name)
+}
+
+// MetricsWindow sets how far back the service's metric series go (default:
+// the scenario's longest traffic window, at least 24h).
+func (svc *Service) metricsWindow() time.Duration {
+ w := 24 * time.Hour
+ for _, e := range append(append([]*Endpoint{}, svc.endpoints...), svc.tasks...) {
+ if e.window > w {
+ w = e.window
+ }
+ }
+ return w
+}
+
+func metricScope(name string) string {
+ switch {
+ case len(name) >= 7 && name[:7] == "system.":
+ return "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver"
+ case len(name) > 4 && name[:4] == "jvm.":
+ return "io.opentelemetry.runtime-telemetry-java17"
+ case len(name) > 3 && name[:3] == "go.":
+ return "go.opentelemetry.io/contrib/instrumentation/runtime"
+ case len(name) > 10 && name[:10] == "db.client.":
+ return "io.opentelemetry.hikaricp-3.0"
+ case len(name) > 12 && name[:12] == "http.server.":
+ return "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
+ default:
+ return "app"
+ }
+}
+
+// Host is a convenience over one instance's host metrics.
+type Host struct {
+ svc *Service
+ Name string
+}
+
+// Host returns the first instance's host metrics.
+func (svc *Service) Host() *Host { return &Host{svc: svc, Name: svc.Hosts[0]} }
+
+// HostNamed returns one instance's host metrics.
+func (svc *Service) HostNamed(name string) *Host { return &Host{svc: svc, Name: name} }
+
+// CPU emits system.cpu.utilization (OTel hostmetrics) split into user,
+// system, wait, steal and idle, with busy = user+system. Optional wait and
+// steal series default to small flat values.
+func (h *Host) CPU(busy Series, extra ...CPUExtra) {
+ wait, steal := Flat(0.002, 0.01), Flat(0, 0.004)
+ for _, e := range extra {
+ if e.Wait != nil {
+ wait = e.Wait
+ }
+ if e.Steal != nil {
+ steal = e.Steal
+ }
+ }
+ clampFrac := func(s Series) Series { return Clamp(s, 0, 1) }
+ busy, wait, steal = clampFrac(busy), clampFrac(wait), clampFrac(steal)
+ user := Scale(busy, func(time.Time) float64 { return 0.78 })
+ sys := Scale(busy, func(time.Time) float64 { return 0.22 })
+ idle := func(t time.Time) float64 { return math.Max(0, 1-busy(t)-wait(t)-steal(t)) }
+ for state, s := range map[string]Series{"user": user, "system": sys, "wait": wait, "steal": steal, "idle": idle} {
+ h.svc.GaugeOn(h.Name, "system.cpu.utilization", "1", map[string]any{"state": state, "cpu": "cpu0"}, s)
+ }
+}
+
+// CPUExtra adds iowait or steal to CPU.
+type CPUExtra struct{ Wait, Steal Series }
+
+// Memory emits system.memory.utilization{state=used|free}.
+func (h *Host) Memory(used Series) {
+ used = Clamp(used, 0, 1)
+ h.svc.GaugeOn(h.Name, "system.memory.utilization", "1", map[string]any{"state": "used"}, used)
+ h.svc.GaugeOn(h.Name, "system.memory.utilization", "1", map[string]any{"state": "free"}, func(t time.Time) float64 { return 1 - used(t) })
+}
+
+// Disk emits the hostmetrics disk series for one device: system.disk.io
+// (cumulative bytes, by direction) and system.disk.pending_operations (the
+// queue depth gauge).
+func (h *Host) Disk(device string, readBps, writeBps, pending Series) {
+ h.svc.metrics = append(h.svc.metrics,
+ &metricSeries{name: "system.disk.io", unit: "By", kind: counterKind, attrs: map[string]any{"device": device, "direction": "read"}, host: h.Name, value: readBps, scope: metricScope("system.")},
+ &metricSeries{name: "system.disk.io", unit: "By", kind: counterKind, attrs: map[string]any{"device": device, "direction": "write"}, host: h.Name, value: writeBps, scope: metricScope("system.")},
+ )
+ h.svc.GaugeOn(h.Name, "system.disk.pending_operations", "{operations}", map[string]any{"device": device}, pending)
+}
+
+// Filesystem emits system.filesystem.utilization for a mountpoint.
+func (h *Host) Filesystem(mount string, used Series) {
+ h.svc.GaugeOn(h.Name, "system.filesystem.utilization", "1", map[string]any{"mountpoint": mount, "state": "used"}, Clamp(used, 0, 1))
+}
+
+func (s *Scenario) metricRequests(svc *Service) []*colmetricspb.ExportMetricsServiceRequest {
+ if len(svc.metrics) == 0 {
+ return nil
+ }
+ end := s.Now.Add(-time.Minute)
+ win := svc.metricsWindow()
+ start := end.Add(-win)
+ recentCut := end.Add(-24 * time.Hour)
+ var times []time.Time
+ for t := start.Truncate(MetricStep.Old); t.Before(recentCut); t = t.Add(MetricStep.Old) {
+ if !t.Before(start) {
+ times = append(times, t)
+ }
+ }
+ for t := recentCut.Truncate(MetricStep.Recent); !t.After(end); t = t.Add(MetricStep.Recent) {
+ times = append(times, t)
+ }
+ byHost := map[string][]*metricSeries{}
+ var hosts []string
+ for _, m := range svc.metrics {
+ if _, ok := byHost[m.host]; !ok {
+ hosts = append(hosts, m.host)
+ }
+ byHost[m.host] = append(byHost[m.host], m)
+ }
+ sort.Strings(hosts)
+ var out []*colmetricspb.ExportMetricsServiceRequest
+ const chunk = 400 // timestamps per request keeps each body small
+ for i := 0; i < len(times); i += chunk {
+ j := min(i+chunk, len(times))
+ req := &colmetricspb.ExportMetricsServiceRequest{}
+ for _, host := range hosts {
+ rm := &metricspb.ResourceMetrics{Resource: svc.resource(svc.VersionAt(times[j-1]), host)}
+ scopes := map[string]*metricspb.ScopeMetrics{}
+ var scopeNames []string
+ for _, m := range byHost[host] {
+ sm := scopes[m.scope]
+ if sm == nil {
+ sm = &metricspb.ScopeMetrics{Scope: &commonpb.InstrumentationScope{Name: m.scope}}
+ scopes[m.scope] = sm
+ scopeNames = append(scopeNames, m.scope)
+ }
+ sm.Metrics = append(sm.Metrics, m.render(times[i:j], start))
+ s.stats.MetricPoints += j - i
+ }
+ sort.Strings(scopeNames)
+ for _, n := range scopeNames {
+ rm.ScopeMetrics = append(rm.ScopeMetrics, scopes[n])
+ }
+ req.ResourceMetrics = append(req.ResourceMetrics, rm)
+ }
+ out = append(out, req)
+ }
+ return out
+}
+
+func (m *metricSeries) render(times []time.Time, origin time.Time) *metricspb.Metric {
+ attrs := attrList(m.attrs)
+ out := &metricspb.Metric{Name: m.name, Unit: m.unit}
+ switch m.kind {
+ case gaugeKind:
+ g := &metricspb.Gauge{}
+ for _, t := range times {
+ g.DataPoints = append(g.DataPoints, &metricspb.NumberDataPoint{
+ Attributes: attrs, TimeUnixNano: nanos(t),
+ Value: &metricspb.NumberDataPoint_AsDouble{AsDouble: m.value(t)},
+ })
+ }
+ out.Data = &metricspb.Metric_Gauge{Gauge: g}
+ case counterKind:
+ sum := &metricspb.Sum{IsMonotonic: true, AggregationTemporality: metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_CUMULATIVE}
+ if m.cumPrev.IsZero() {
+ m.cumPrev = origin
+ }
+ total, prev := m.cumTotal, m.cumPrev
+ for _, t := range times {
+ total += m.value(t) * t.Sub(prev).Seconds()
+ prev = t
+ sum.DataPoints = append(sum.DataPoints, &metricspb.NumberDataPoint{
+ Attributes: attrs, StartTimeUnixNano: nanos(origin), TimeUnixNano: nanos(t),
+ Value: &metricspb.NumberDataPoint_AsDouble{AsDouble: total},
+ })
+ }
+ m.cumTotal, m.cumPrev = total, prev
+ out.Data = &metricspb.Metric_Sum{Sum: sum}
+ case histogramKind:
+ h := &metricspb.Histogram{AggregationTemporality: metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_DELTA}
+ prev := times[0].Add(-time.Minute)
+ for _, t := range times {
+ per := m.count(t) * t.Sub(prev).Minutes()
+ prev = t
+ cnt := uint64(math.Max(0, math.Round(per)))
+ mean := m.value(t)
+ s := mean * float64(cnt)
+ h.DataPoints = append(h.DataPoints, &metricspb.HistogramDataPoint{
+ Attributes: attrs, StartTimeUnixNano: nanos(t.Add(-time.Minute)), TimeUnixNano: nanos(t),
+ Count: cnt, Sum: &s,
+ })
+ }
+ out.Data = &metricspb.Metric_Histogram{Histogram: h}
+ }
+ return out
+}
diff --git a/cli/test/evals/internal/seed/native_live_test.go b/cli/test/evals/internal/seed/native_live_test.go
new file mode 100644
index 00000000..6793f494
--- /dev/null
+++ b/cli/test/evals/internal/seed/native_live_test.go
@@ -0,0 +1,59 @@
+package seed_test
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ client "github.com/tracewayapp/traceway/cli/pkg/client"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+// TestNativeCrashWithActions sends a mobile crash with captured network
+// actions over /api/report and reads it back as an agent would.
+func TestNativeCrashWithActions(t *testing.T) {
+ bins := env.BinariesForTest(t)
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+ defer cancel()
+ inst, err := env.NewInstance(ctx, bins, t.TempDir())
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer inst.Close()
+ s := seed.NewScenario("native-test", time.Now(), seed.NewLiveSink(ctx, inst.Backend, inst.Admin))
+ p := s.Project("shop-android", seed.Mobile, "android")
+ at := s.Ago(2 * time.Hour)
+ p.Crash(seed.Crash{At: at, AppVersion: "4.12.0", Device: "Pixel 8",
+ Stack: "java.net.SocketTimeoutException: timeout\n\tat okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:675)\n\tat com.example.shop.checkout.CheckoutRepository.placeOrder(CheckoutRepository.kt:88)",
+ Attributes: map[string]string{"device.model": "Pixel 8", "os.version": "15"},
+ Actions: []seed.Action{{Type: "network", At: at.Add(-10 * time.Second), Method: "POST", URL: "https://api.shop.example/api/checkout", DurationMs: 10012, Error: "timeout"}},
+ })
+ if err := s.Flush(); err != nil {
+ t.Fatal(err)
+ }
+ c := client.New(inst.Backend.URL, client.WithJWT(inst.Admin.JWT))
+ list, err := c.ListExceptions(ctx, p.ID, client.ListExceptionsRequest{TimeRange: client.TimeRangeFromExplicit(s.Ago(24*time.Hour), s.Now.Add(time.Hour)), Pagination: client.PaginationParams{Page: 1, PageSize: 10}})
+ if err != nil || len(list.Data) != 1 {
+ t.Fatalf("list: %v %+v", err, list)
+ }
+ g, err := c.GetException(ctx, p.ID, list.Data[0].ExceptionHash, client.PaginationParams{Page: 1, PageSize: 5})
+ if err != nil || len(g.Occurrences) != 1 {
+ t.Fatalf("show: %v", err)
+ }
+ var occ *client.ExceptionByIdResponse
+ for i := 0; i < 40; i++ {
+ occ, err = c.GetExceptionById(ctx, p.ID, g.Occurrences[0].Id.String(), g.Occurrences[0].RecordedAt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(occ.SessionRecording) > 0 && string(occ.SessionRecording) != "null" {
+ break
+ }
+ time.Sleep(250 * time.Millisecond)
+ }
+ if len(occ.SessionRecording) == 0 || string(occ.SessionRecording) == "null" {
+ t.Fatalf("the occurrence carries no recording with the captured actions: %+v", occ)
+ }
+ t.Logf("recording: %.300s", string(occ.SessionRecording))
+}
diff --git a/cli/test/evals/internal/seed/noise.go b/cli/test/evals/internal/seed/noise.go
new file mode 100644
index 00000000..bbbc1be8
--- /dev/null
+++ b/cli/test/evals/internal/seed/noise.go
@@ -0,0 +1,150 @@
+package seed
+
+import (
+ "fmt"
+ "math"
+ "time"
+)
+
+// TrafficOpts configures background noise.
+type TrafficOpts struct {
+ Endpoints int
+ // PerDay is the request rate per background endpoint (capped so the
+ // background stays under ~5000 requests per service).
+ PerDay int
+ // Window defaults to the service's longest scenario window.
+ Window time.Duration
+ // NoErrors suppresses the low-rate unrelated client error.
+ NoErrors bool
+ // Hosts sets default host metrics (CPU and memory) when the scenario
+ // did not set its own.
+ NoHostMetrics bool
+}
+
+// TypicalTraffic is n healthy endpoints, a low-rate unrelated error and flat
+// host metrics.
+func TypicalTraffic(n int) TrafficOpts { return TrafficOpts{Endpoints: n, PerDay: 150} }
+
+// backgroundRoutes is a neutral pool of routes for healthy traffic.
+var backgroundRoutes = [][2]string{
+ {"GET", "/api/me"}, {"GET", "/api/products"}, {"GET", "/api/products/:id"}, {"POST", "/api/cart/items"},
+ {"GET", "/api/cart"}, {"GET", "/api/categories"}, {"POST", "/api/auth/session"}, {"GET", "/api/notifications"},
+ {"PUT", "/api/users/:id/preferences"}, {"POST", "/api/events"}, {"GET", "/api/addresses"}, {"DELETE", "/api/cart/items/:id"},
+ {"GET", "/api/inventory/:sku"}, {"GET", "/api/recommendations"}, {"POST", "/api/checkout/quote"}, {"GET", "/api/stores/:id/hours"},
+ {"GET", "/api/promotions"}, {"POST", "/api/newsletter/subscriptions"}, {"GET", "/api/shipping/rates"}, {"GET", "/api/help/articles"},
+}
+
+var backgroundTables = []string{"users", "products", "categories", "carts", "cart_items", "notifications", "addresses", "inventory", "stores", "promotions"}
+
+// Background adds healthy traffic around the scenario so the signal has to
+// be found, not handed over.
+func (s *Scenario) Background(svc *Service, o TrafficOpts) {
+ taken := map[string]bool{}
+ for _, e := range svc.endpoints {
+ taken[e.Name()] = true
+ }
+ win := o.Window
+ if win == 0 {
+ win = svc.metricsWindow()
+ }
+ days := math.Max(1, win.Hours()/24)
+ rate := o.PerDay
+ if o.Endpoints > 0 && float64(rate)*float64(o.Endpoints)*days > 5000 {
+ rate = int(5000 / (float64(o.Endpoints) * days))
+ if rate < 8 {
+ rate = 8
+ }
+ }
+ added := 0
+ for i := 0; added < o.Endpoints && i < len(backgroundRoutes)*2; i++ {
+ rt := backgroundRoutes[(i*7+len(svc.Name))%len(backgroundRoutes)]
+ name := rt[0] + " " + rt[1]
+ if taken[name] {
+ continue
+ }
+ taken[name] = true
+ added++
+ tables := []string{backgroundTables[(i*3)%len(backgroundTables)], backgroundTables[(i*5+1)%len(backgroundTables)]}
+ method := rt[0]
+ medianDB := time.Duration(2+(i%5)) * time.Millisecond
+ withError := !o.NoErrors && method == "POST" && added <= 6
+ e := svc.Endpoint(rt[0], rt[1])
+ e.Traffic(Daily(rate), Diurnal(), Over(win))
+ e.Detail(DetailPolicy{Recent: 6 * time.Hour, Rate: 0.01, Max: 8})
+ e.Handler(func(r *Req) {
+ r.Gap(r.Ms(0.3, 1.2))
+ if withError && r.Chance(0.004) {
+ r.Compute("decode request", r.Ms(0.1, 0.4))
+ r.Error(clientDecodeError(svc.Stack))
+ r.Status(400)
+ r.Gap(r.Ms(0.2, 0.6))
+ return
+ }
+ r.DB(selectFor(method, tables[0]), r.LogNormal(medianDB, medianDB*6))
+ if r.Chance(0.6) {
+ r.DB(selectFor("GET", tables[1]), r.LogNormal(medianDB, medianDB*5))
+ }
+ r.Compute("encode response", r.Ms(0.4, 2.5))
+ r.Gap(r.Ms(0.2, 0.8))
+ })
+ }
+ if !o.NoHostMetrics {
+ svc.defaultHost = true
+ }
+}
+
+func selectFor(method, table string) string {
+ switch method {
+ case "POST":
+ return fmt.Sprintf("INSERT INTO %s (id, payload, created_at) VALUES ($1, $2, now())", table)
+ case "PUT":
+ return fmt.Sprintf("UPDATE %s SET payload = $2, updated_at = now() WHERE id = $1", table)
+ case "DELETE":
+ return fmt.Sprintf("DELETE FROM %s WHERE id = $1", table)
+ default:
+ return fmt.Sprintf("SELECT id, name, updated_at FROM %s WHERE id = $1", table)
+ }
+}
+
+// clientDecodeError is a benign 400: a client sent malformed JSON.
+func clientDecodeError(st Stack) Exc {
+ switch st.StackFormat {
+ case "python":
+ return Exc{Type: "json.decoder.JSONDecodeError", Message: "Expecting property name enclosed in double quotes: line 1 column 2 (char 1)",
+ Stack: "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/starlette/requests.py\", line 252, in json\n self._json = json.loads(body)\n File \"/usr/local/lib/python3.12/json/__init__.py\", line 346, in loads\n return _default_decoder.decode(s)\njson.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"}
+ case "node":
+ return Exc{Type: "SyntaxError", Message: "Expected property name or '}' in JSON at position 1",
+ Stack: "SyntaxError: Expected property name or '}' in JSON at position 1\n at JSON.parse (<anonymous>)\n at parse (/app/node_modules/body-parser/lib/types/json.js:92:19)\n at /app/node_modules/body-parser/lib/read.js:128:18"}
+ case "java":
+ return Exc{Type: "com.fasterxml.jackson.core.JsonParseException", Message: "Unexpected character ('}' (code 125)): was expecting double-quote to start field name",
+ Stack: "com.fasterxml.jackson.core.JsonParseException: Unexpected character ('}' (code 125)): was expecting double-quote to start field name\n\tat com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2481)\n\tat org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406)"}
+ case "php":
+ return Exc{Type: "JsonException", Message: "Syntax error", Stack: "JsonException: Syntax error in /var/www/vendor/laravel/framework/src/Illuminate/Http/Request.php:671\nStack trace:\n#0 /var/www/vendor/laravel/framework/src/Illuminate/Http/Request.php(671): json_decode()"}
+ case "dotnet":
+ return Exc{Type: "System.Text.Json.JsonException", Message: "'}' is an invalid start of a property name. Path: $ | LineNumber: 0 | BytePositionInLine: 1.",
+ Stack: "System.Text.Json.JsonException: '}' is an invalid start of a property name.\n at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)\n at Microsoft.AspNetCore.Http.Json.HttpRequestJsonExtensions.ReadFromJsonAsync[TValue](HttpRequest request)"}
+ default:
+ return Exc{Type: "*json.SyntaxError", Message: "invalid character '}' looking for beginning of object key string",
+ Stack: "goroutine 4113 [running]:\nencoding/json.(*decodeState).object(0xc0004b2000)\n\t/usr/local/go/src/encoding/json/decode.go:651 +0x1ef\ngithub.com/gin-gonic/gin.(*Context).ShouldBindJSON(0xc000521100)\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:780 +0x45"}
+ }
+}
+
+// applyDefaults fills host metrics a scenario did not set itself.
+func applyDefaults(svc *Service) {
+ if !svc.defaultHost {
+ return
+ }
+ has := map[string]bool{}
+ for _, m := range svc.metrics {
+ has[m.name+"@"+m.host] = true
+ }
+ for _, h := range svc.Hosts {
+ host := svc.HostNamed(h)
+ if !has["system.cpu.utilization@"+h] {
+ host.CPU(Flat(0.18, 0.32))
+ }
+ if !has["system.memory.utilization@"+h] {
+ host.Memory(Flat(0.52, 0.6))
+ }
+ }
+}
diff --git a/cli/test/evals/internal/seed/report.go b/cli/test/evals/internal/seed/report.go
new file mode 100644
index 00000000..a20fed36
--- /dev/null
+++ b/cli/test/evals/internal/seed/report.go
@@ -0,0 +1,204 @@
+package seed
+
+import (
+ "encoding/json"
+ "fmt"
+ "time"
+
+ "github.com/google/uuid"
+)
+
+// The native /api/report protocol is what the browser and mobile SDKs speak:
+// exceptions with their own stack format, sessions, and session recordings
+// whose actions carry the network requests and navigations the SDK captured.
+// Only what those SDKs really send is modelled here.
+
+type nativeFrame struct {
+ appVersion string
+ serverName string
+ exceptions []nativeException
+ sessions []nativeSession
+ recordings []nativeRecording
+}
+
+type nativeException struct {
+ Stack string `json:"stackTrace"`
+ RecordedAt string `json:"recordedAt"`
+ Attributes map[string]string `json:"attributes"`
+ IsMessage bool `json:"isMessage"`
+ IsTask bool `json:"isTask"`
+ SessionID string `json:"sessionId,omitempty"`
+ RecordingID string `json:"sessionRecordingId,omitempty"`
+}
+
+type nativeSession struct {
+ ID string `json:"id"`
+ StartedAt string `json:"startedAt"`
+ EndedAt string `json:"endedAt,omitempty"`
+ ClientIP string `json:"clientIP"`
+ Attributes map[string]string `json:"attributes"`
+}
+
+type nativeRecording struct {
+ ExceptionID string `json:"exceptionId"`
+ SessionID string `json:"sessionId,omitempty"`
+ Events []any `json:"events"`
+ Logs []map[string]any `json:"logs,omitempty"`
+ Actions []map[string]any `json:"actions,omitempty"`
+ StartedAt string `json:"startedAt,omitempty"`
+ EndedAt string `json:"endedAt,omitempty"`
+}
+
+// Crash is one exception reported by a browser or mobile SDK.
+type Crash struct {
+ At time.Time
+ Stack string
+ Attributes map[string]string
+ IsMessage bool
+ AppVersion string
+ // Device is the SDK's serverName (the device or browser identity).
+ Device string
+ // SessionID links the crash to a browser session.
+ SessionID string
+ // Actions are what the SDK captured before the crash (network,
+ // navigation, custom), shown with the occurrence.
+ Actions []Action
+ // Logs are console/logcat lines captured before the crash.
+ Logs []ActionLog
+}
+
+// Action is one captured client action.
+type Action struct {
+ Type string // network | navigation | custom
+ At time.Time
+ Method string
+ URL string
+ DurationMs float64
+ Status int
+ Error string
+ From, To string
+ RespBytes int
+}
+
+// ActionLog is one captured client log line.
+type ActionLog struct {
+ At time.Time
+ Level string
+ Message string
+}
+
+func (a Action) wire() map[string]any {
+ m := map[string]any{"type": a.Type, "timestamp": a.At.UTC().Format(time.RFC3339Nano)}
+ switch a.Type {
+ case "network":
+ m["method"], m["url"], m["durationMs"] = a.Method, a.URL, a.DurationMs
+ if a.Status != 0 {
+ m["statusCode"] = a.Status
+ }
+ if a.Error != "" {
+ m["error"] = a.Error
+ }
+ if a.RespBytes != 0 {
+ m["responseBytes"] = a.RespBytes
+ }
+ case "navigation":
+ m["action"] = "push"
+ m["from"], m["to"] = a.From, a.To
+ default:
+ m["category"], m["name"] = "app", a.URL
+ }
+ return m
+}
+
+// Crash queues a native SDK exception for the project.
+func (p *Project) Crash(c Crash) {
+ f := nativeFrame{appVersion: c.AppVersion, serverName: c.Device}
+ ex := nativeException{
+ Stack: c.Stack, RecordedAt: c.At.UTC().Format(time.RFC3339Nano),
+ Attributes: c.Attributes, IsMessage: c.IsMessage, SessionID: c.SessionID,
+ }
+ if ex.Attributes == nil {
+ ex.Attributes = map[string]string{}
+ }
+ if len(c.Actions) > 0 || len(c.Logs) > 0 {
+ ex.RecordingID = uuid.NewString()
+ rec := nativeRecording{ExceptionID: ex.RecordingID, SessionID: c.SessionID, Events: []any{}}
+ first, last := c.At, c.At
+ for _, a := range c.Actions {
+ rec.Actions = append(rec.Actions, a.wire())
+ if a.At.Before(first) {
+ first = a.At
+ }
+ p.s.record(a.URL, a.Error)
+ }
+ for _, l := range c.Logs {
+ rec.Logs = append(rec.Logs, map[string]any{"timestamp": l.At.UTC().Format(time.RFC3339Nano), "level": l.Level, "message": l.Message})
+ p.s.record(l.Message)
+ }
+ rec.StartedAt, rec.EndedAt = first.UTC().Format(time.RFC3339Nano), last.UTC().Format(time.RFC3339Nano)
+ f.recordings = append(f.recordings, rec)
+ }
+ f.exceptions = append(f.exceptions, ex)
+ p.native = append(p.native, f)
+ p.s.record(c.Stack, c.Device)
+ for _, v := range c.Attributes {
+ p.s.record(v)
+ }
+}
+
+// Session queues a browser session (only the browser SDK reports sessions).
+func (p *Project) Session(id string, started, ended time.Time, attrs map[string]string, appVersion string) {
+ if attrs == nil {
+ attrs = map[string]string{}
+ }
+ p.native = append(p.native, nativeFrame{appVersion: appVersion, sessions: []nativeSession{{
+ ID: id, StartedAt: started.UTC().Format(time.RFC3339Nano), EndedAt: ended.UTC().Format(time.RFC3339Nano),
+ ClientIP: "203.0.113.24", Attributes: attrs,
+ }}})
+}
+
+// nativeBodies groups queued frames by (appVersion, serverName) into
+// /api/report bodies of at most 200 exceptions each.
+func (p *Project) nativeBodies() ([][]byte, error) {
+ type key struct{ v, d string }
+ groups := map[key]*nativeFrame{}
+ var order []key
+ for _, f := range p.native {
+ k := key{f.appVersion, f.serverName}
+ g := groups[k]
+ if g == nil {
+ g = &nativeFrame{appVersion: f.appVersion, serverName: f.serverName}
+ groups[k] = g
+ order = append(order, k)
+ }
+ g.exceptions = append(g.exceptions, f.exceptions...)
+ g.sessions = append(g.sessions, f.sessions...)
+ g.recordings = append(g.recordings, f.recordings...)
+ }
+ var out [][]byte
+ for _, k := range order {
+ g := groups[k]
+ frame := map[string]any{
+ "traces": []any{}, "metrics": []any{},
+ "stackTraces": g.exceptions, "sessions": g.sessions, "sessionRecordings": g.recordings,
+ }
+ if g.exceptions == nil {
+ frame["stackTraces"] = []any{}
+ }
+ if g.sessions == nil {
+ frame["sessions"] = []any{}
+ }
+ if g.recordings == nil {
+ frame["sessionRecordings"] = []any{}
+ }
+ body, err := json.Marshal(map[string]any{
+ "appVersion": g.appVersion, "serverName": g.serverName,
+ "collectionFrames": []any{frame},
+ })
+ if err != nil {
+ return nil, fmt.Errorf("native frame: %w", err)
+ }
+ out = append(out, body)
+ }
+ return out, nil
+}
diff --git a/cli/test/evals/internal/seed/req.go b/cli/test/evals/internal/seed/req.go
new file mode 100644
index 00000000..a84fb1c0
--- /dev/null
+++ b/cli/test/evals/internal/seed/req.go
@@ -0,0 +1,367 @@
+package seed
+
+import (
+ "fmt"
+ "math"
+ "strings"
+ "time"
+)
+
+// Req builds one request's span tree. Work is laid out on a cursor: every
+// child starts where the previous one ended (plus any Gap), and a span's
+// duration is where its cursor stopped, so a parent always covers its
+// children exactly.
+type Req struct {
+ s *Scenario
+ req *Request
+ cur *Span
+ cursor time.Duration
+ status int
+ forceDetail bool
+ svc *Service // service owning cur (nil = the request's)
+ base time.Duration // offset of cur's start from the request's start
+}
+
+// Request returns the request being built.
+func (r *Req) Request() *Request { return r.req }
+
+// At is the absolute time of the cursor.
+func (r *Req) At() time.Time { return r.req.Start.Add(r.base + r.cursor) }
+
+// Start is the request's start time.
+func (r *Req) Start() time.Time { return r.req.Start }
+
+// Progress is how far through the traffic window the request falls (0..1).
+func (r *Req) Progress() float64 { return r.req.Progress }
+
+// Host is the instance serving the request.
+func (r *Req) Host() string { return r.req.Host }
+
+// Version is the service.version live when the request started.
+func (r *Req) Version() string { return r.req.Version }
+
+// Float returns a uniform [0,1) draw.
+func (r *Req) Float() float64 { return r.s.RNG.Float64() }
+
+// Chance reports true with probability p.
+func (r *Req) Chance(p float64) bool { return r.s.RNG.Float64() < p }
+
+// IntN returns a uniform int in [0,n).
+func (r *Req) IntN(n int) int { return r.s.RNG.IntN(n) }
+
+// Ms returns a uniform duration between lo and hi milliseconds.
+func (r *Req) Ms(lo, hi float64) time.Duration {
+ return time.Duration((lo + (hi-lo)*r.s.RNG.Float64()) * float64(time.Millisecond))
+}
+
+// LogNormal draws a duration with the given median and p99.
+func (r *Req) LogNormal(median, p99 time.Duration) time.Duration {
+ mu := math.Log(float64(median))
+ sigma := (math.Log(float64(p99)) - mu) / 2.326
+ return time.Duration(math.Exp(mu + sigma*r.s.RNG.NormFloat64()))
+}
+
+// Growth interpolates from lo at the window start to hi at its end, with
+// ±10% noise, as an integer.
+func (r *Req) Growth(lo, hi float64) int {
+ v := lo + (hi-lo)*r.req.Progress
+ v *= 0.9 + 0.2*r.s.RNG.Float64()
+ return int(math.Round(math.Max(0, v)))
+}
+
+// Lerp interpolates from lo to hi by window progress, without noise.
+func (r *Req) Lerp(lo, hi float64) float64 { return lo + (hi-lo)*r.req.Progress }
+
+// Status sets the response status.
+func (r *Req) Status(code int) { r.status = code }
+
+// KeepDetail forces this request's full span tree to be sent.
+func (r *Req) KeepDetail() { r.forceDetail = true }
+
+// Attr sets an attribute on the current span.
+func (r *Req) Attr(k string, v any) {
+ r.cur.Attrs[k] = v
+ if s, ok := v.(string); ok {
+ r.s.record(s)
+ }
+}
+
+// RootAttr sets an attribute on the request's root span.
+func (r *Req) RootAttr(k string, v any) {
+ r.req.Root.Attrs[k] = v
+ if s, ok := v.(string); ok {
+ r.s.record(s)
+ }
+}
+
+// Gap advances the cursor without a span: time the request spent waiting.
+func (r *Req) Gap(d time.Duration) { r.cursor += d }
+
+func (r *Req) owner() *Service {
+ if r.svc != nil {
+ return r.svc
+ }
+ return r.req.e.svc
+}
+
+func (r *Req) child(name string, kind int, scope string, d time.Duration, attrs map[string]any) *Span {
+ sp := &Span{Name: name, Kind: kind, Start: r.cursor, Dur: d, Attrs: attrs, Scope: scope}
+ fillRandom(r.s, sp.SpanID[:])
+ r.cur.Children = append(r.cur.Children, sp)
+ r.cursor += d
+ r.s.record(name)
+ for _, v := range attrs {
+ if s, ok := v.(string); ok {
+ r.s.record(s)
+ }
+ }
+ return sp
+}
+
+// Compute records in-process work as an INTERNAL span.
+func (r *Req) Compute(name string, d time.Duration, attrs ...KV) *Span {
+ return r.child(name, KindInternal, r.owner().Stack.ServerScope, d, kvMap(attrs))
+}
+
+// Span records an INTERNAL span whose children are built by f; its duration
+// is whatever f lays out.
+func (r *Req) Span(name string, f func(r *Req), attrs ...KV) *Span {
+ sp := &Span{Name: name, Kind: KindInternal, Start: r.cursor, Attrs: kvMap(attrs), Scope: r.owner().Stack.ServerScope}
+ fillRandom(r.s, sp.SpanID[:])
+ r.cur.Children = append(r.cur.Children, sp)
+ r.s.record(name)
+ sub := &Req{s: r.s, req: r.req, cur: sp, cursor: 0, svc: r.svc, base: r.base + sp.Start}
+ f(sub)
+ sp.Dur = sub.cursor
+ r.cursor += sp.Dur
+ r.forceDetail = r.forceDetail || sub.forceDetail
+ if sub.status != 0 {
+ r.status = sub.status
+ }
+ return sp
+}
+
+// Parallel runs branches concurrently from the current cursor; the cursor
+// then moves to the end of the longest branch.
+func (r *Req) Parallel(branches ...func(r *Req)) {
+ start := r.cursor
+ longest := start
+ for _, b := range branches {
+ sub := &Req{s: r.s, req: r.req, cur: r.cur, cursor: start, svc: r.svc, base: r.base}
+ b(sub)
+ if sub.cursor > longest {
+ longest = sub.cursor
+ }
+ r.forceDetail = r.forceDetail || sub.forceDetail
+ if sub.status != 0 {
+ r.status = sub.status
+ }
+ }
+ r.cursor = longest
+}
+
+// KV is an attribute pair.
+type KV struct {
+ K string
+ V any
+}
+
+// A is shorthand for a KV.
+func A(k string, v any) KV { return KV{k, v} }
+
+func kvMap(kvs []KV) map[string]any {
+ m := map[string]any{}
+ for _, kv := range kvs {
+ m[kv.K] = kv.V
+ }
+ return m
+}
+
+// DB records a database CLIENT span for query. The span name follows the
+// semantic conventions ("{operation} {collection}").
+func (r *Req) DB(query string, d time.Duration, attrs ...KV) *Span {
+ st := r.owner().Stack
+ op, coll := sqlParts(query)
+ name := op
+ if coll != "" {
+ name = op + " " + coll
+ }
+ m := map[string]any{
+ "db.system.name": st.DBSystem,
+ "db.query.text": query,
+ "db.namespace": r.owner().dbName(),
+ "db.operation.name": op,
+ "server.address": r.owner().dbHost(),
+ "server.port": 5432,
+ }
+ if st.DBSystem == "mysql" {
+ m["server.port"] = 3306
+ }
+ if coll != "" {
+ m["db.collection.name"] = coll
+ }
+ for _, kv := range attrs {
+ m[kv.K] = kv.V
+ }
+ return r.child(name, KindClient, st.DBScope, d, m)
+}
+
+// Rows is the db.response.returned_rows attribute.
+func Rows(n int) KV { return KV{"db.response.returned_rows", n} }
+
+func sqlParts(q string) (op, coll string) {
+ fields := strings.Fields(q)
+ if len(fields) == 0 {
+ return "QUERY", ""
+ }
+ op = strings.ToUpper(fields[0])
+ for i, f := range fields {
+ u := strings.ToUpper(f)
+ if (u == "FROM" || u == "INTO" || (u == "UPDATE" && i == 0)) && i+1 < len(fields) {
+ coll = strings.Trim(fields[i+1], "(),;")
+ break
+ }
+ }
+ return op, strings.ToLower(coll)
+}
+
+func (svc *Service) dbName() string {
+ return strings.ReplaceAll(strings.TrimSuffix(strings.TrimSuffix(svc.Name, "-api"), "-service"), "-", "_")
+}
+
+func (svc *Service) dbHost() string { return svc.dbName() + "-db.internal" }
+
+// Cache records a cache CLIENT span (redis).
+func (r *Req) Cache(cmd, key string, d time.Duration, attrs ...KV) *Span {
+ m := map[string]any{
+ "db.system.name": "redis",
+ "db.operation.name": cmd,
+ "db.query.text": cmd + " " + key,
+ "server.address": "cache.internal",
+ "server.port": 6379,
+ }
+ for _, kv := range attrs {
+ m[kv.K] = kv.V
+ }
+ return r.child(cmd, KindClient, r.owner().Stack.CacheScope, d, m)
+}
+
+// HTTP records an outgoing HTTP CLIENT span. The span name is the method,
+// as the semantic conventions specify for client spans.
+func (r *Req) HTTP(method, url string, d time.Duration, status int, attrs ...KV) *Span {
+ host, port := hostOf(url)
+ m := map[string]any{
+ "http.request.method": method,
+ "url.full": url,
+ "server.address": host,
+ "server.port": port,
+ "http.response.status_code": status,
+ }
+ for _, kv := range attrs {
+ m[kv.K] = kv.V
+ }
+ sp := r.child(method, KindClient, r.owner().Stack.HTTPClient, d, m)
+ if status >= 500 {
+ sp.Status = StatusError
+ }
+ return sp
+}
+
+func hostOf(u string) (string, int) {
+ rest := u
+ port := 443
+ if strings.HasPrefix(rest, "http://") {
+ rest, port = strings.TrimPrefix(rest, "http://"), 80
+ }
+ rest = strings.TrimPrefix(rest, "https://")
+ if i := strings.IndexAny(rest, "/?"); i >= 0 {
+ rest = rest[:i]
+ }
+ if h, p, ok := strings.Cut(rest, ":"); ok {
+ var n int
+ fmt.Sscanf(p, "%d", &n)
+ return h, n
+ }
+ return rest, port
+}
+
+// Call makes an HTTP call to another service's endpoint: a CLIENT span here
+// and, under it, the downstream SERVER span built by the target's handler.
+// rtt is the network round trip added around the server time.
+func (r *Req) Call(target *Endpoint, rtt time.Duration) *Span {
+ tsvc := target.svc
+ url := fmt.Sprintf("http://%s.internal:8080%s", tsvc.Name, concretePath(r.s, target.Route))
+ client := &Span{Name: target.Method, Kind: KindClient, Start: r.cursor, Scope: r.owner().Stack.HTTPClient,
+ Attrs: map[string]any{"http.request.method": target.Method, "url.full": url, "server.address": tsvc.Name + ".internal", "server.port": 8080}}
+ fillRandom(r.s, client.SpanID[:])
+ r.cur.Children = append(r.cur.Children, client)
+
+ server := &Span{Name: target.Name(), Kind: KindServer, Start: rtt / 2, Scope: tsvc.Stack.ServerScope, Service: tsvc, Host: tsvc.pickHost(),
+ Attrs: map[string]any{"http.request.method": target.Method, "http.route": target.Route, "url.path": concretePath(r.s, target.Route), "url.scheme": "http"}}
+ fillRandom(r.s, server.SpanID[:])
+ client.Children = append(client.Children, server)
+ sub := &Req{s: r.s, req: r.req, cur: server, svc: tsvc, base: r.base + client.Start + server.Start}
+ if target.handler != nil {
+ target.handler(sub)
+ }
+ server.Dur = sub.cursor
+ status := 200
+ if sub.status != 0 {
+ status = sub.status
+ }
+ server.Attrs["http.response.status_code"] = status
+ client.Attrs["http.response.status_code"] = status
+ if status >= 500 {
+ server.Status, client.Status = StatusError, StatusError
+ }
+ client.Dur = server.Dur + rtt
+ r.cursor += client.Dur
+ r.forceDetail = r.forceDetail || sub.forceDetail
+ return client
+}
+
+// Log emits a log record correlated with the current span at the cursor.
+// Logged requests are always sent in detail so the trace id resolves.
+func (r *Req) Log(severity int, body string, attrs ...KV) {
+ r.cur.logs = append(r.cur.logs, logRecord{at: r.cursor, severity: severity, body: body, attrs: kvMap(attrs)})
+ r.forceDetail = true
+ r.s.record(body)
+ for _, kv := range attrs {
+ if s, ok := kv.V.(string); ok {
+ r.s.record(s)
+ }
+ }
+}
+
+// Exc is an exception to record on a span.
+type Exc struct {
+ Type string
+ Message string
+ Stack string
+ // Escaped marks exception.escaped.
+ Escaped bool
+}
+
+// Error records an exception event on the current span at the cursor and
+// marks the span (and, for escaped errors, the request) as failed.
+func (r *Req) Error(e Exc) {
+ attrs := map[string]any{"exception.type": e.Type, "exception.message": e.Message}
+ if e.Stack != "" {
+ attrs["exception.stacktrace"] = e.Stack
+ }
+ if e.Escaped {
+ attrs["exception.escaped"] = true
+ }
+ r.cur.Events = append(r.cur.Events, Event{Name: "exception", At: r.cursor, Attrs: attrs})
+ r.cur.Status = StatusError
+ r.cur.StatusMsg = e.Message
+ r.forceDetail = true
+ r.s.record(e.Type, e.Message, e.Stack)
+}
+
+func (r *Req) finish() {
+ r.req.Root.Dur = r.cursor
+ if r.req.Root.Dur <= 0 {
+ r.req.Root.Dur = time.Millisecond
+ }
+}
diff --git a/cli/test/evals/internal/seed/semconv.go b/cli/test/evals/internal/seed/semconv.go
new file mode 100644
index 00000000..30a3e026
--- /dev/null
+++ b/cli/test/evals/internal/seed/semconv.go
@@ -0,0 +1,149 @@
+package seed
+
+// Stack fixes what one service's telemetry looks like: the instrumentation
+// scope names its libraries really report, the SDK language, the database
+// system, and which runtime metrics it emits. Names follow the OpenTelemetry
+// semantic conventions as of 2026 (http.request.method, http.route,
+// http.response.status_code, db.system.name, db.query.text, ...).
+type Stack struct {
+ Language string // telemetry.sdk.language
+ Framework string // label only
+ ServerScope string // scope of the HTTP SERVER span
+ DBScope string // scope of database CLIENT spans
+ DBSystem string // db.system.name
+ HTTPClient string // scope of outgoing HTTP CLIENT spans
+ CacheScope string // scope of cache (redis) spans
+ Runtime string // go | jvm | python | node | dotnet | php
+ // StackFormat selects how exception.stacktrace is rendered.
+ StackFormat string
+}
+
+// Go returns a Go service. web: gin | nethttp | chi. db: pgx | sql.
+func Go(web, db string) Stack {
+ s := Stack{Language: "go", Framework: "go/" + web, Runtime: "go", StackFormat: "go",
+ HTTPClient: "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",
+ CacheScope: "github.com/redis/go-redis/extra/redisotel"}
+ switch web {
+ case "gin":
+ s.ServerScope = "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
+ case "chi":
+ s.ServerScope = "github.com/riandyrn/otelchi"
+ default:
+ s.ServerScope = "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
+ }
+ switch db {
+ case "pgx":
+ s.DBScope, s.DBSystem = "github.com/exaring/otelpgx", "postgresql"
+ case "mysql":
+ s.DBScope, s.DBSystem = "github.com/XSAM/otelsql", "mysql"
+ default:
+ s.DBScope, s.DBSystem = "github.com/XSAM/otelsql", "postgresql"
+ }
+ return s
+}
+
+// Python returns a Python service. web: fastapi | django | flask.
+func Python(web, db string) Stack {
+ s := Stack{Language: "python", Framework: "python/" + web, Runtime: "python", StackFormat: "python",
+ HTTPClient: "opentelemetry.instrumentation.httpx", CacheScope: "opentelemetry.instrumentation.redis"}
+ switch web {
+ case "django":
+ s.ServerScope = "opentelemetry.instrumentation.django"
+ case "flask":
+ s.ServerScope = "opentelemetry.instrumentation.flask"
+ default:
+ s.ServerScope = "opentelemetry.instrumentation.fastapi"
+ }
+ switch db {
+ case "psycopg":
+ s.DBScope, s.DBSystem = "opentelemetry.instrumentation.psycopg", "postgresql"
+ default:
+ s.DBScope, s.DBSystem = "opentelemetry.instrumentation.sqlalchemy", "postgresql"
+ }
+ return s
+}
+
+// Node returns a Node.js service. web: express | nestjs | fastify.
+func Node(web, db string) Stack {
+ s := Stack{Language: "nodejs", Framework: "node/" + web, Runtime: "node", StackFormat: "node",
+ HTTPClient: "@opentelemetry/instrumentation-undici", CacheScope: "@opentelemetry/instrumentation-ioredis"}
+ switch web {
+ case "nestjs":
+ s.ServerScope = "@opentelemetry/instrumentation-http"
+ case "fastify":
+ s.ServerScope = "@opentelemetry/instrumentation-fastify"
+ default:
+ s.ServerScope = "@opentelemetry/instrumentation-http"
+ }
+ switch db {
+ case "mysql":
+ s.DBScope, s.DBSystem = "@opentelemetry/instrumentation-mysql2", "mysql"
+ default:
+ s.DBScope, s.DBSystem = "@opentelemetry/instrumentation-pg", "postgresql"
+ }
+ return s
+}
+
+// Java returns a JVM service instrumented by the OpenTelemetry Java agent.
+func Java(web string) Stack {
+ return Stack{Language: "java", Framework: "java/" + web, Runtime: "jvm", StackFormat: "java",
+ ServerScope: "io.opentelemetry.tomcat-10.0", DBScope: "io.opentelemetry.jdbc", DBSystem: "postgresql",
+ HTTPClient: "io.opentelemetry.java-http-client", CacheScope: "io.opentelemetry.lettuce-5.1"}
+}
+
+// PHP returns a PHP service (Laravel auto-instrumentation).
+func PHP(web string) Stack {
+ return Stack{Language: "php", Framework: "php/" + web, Runtime: "php", StackFormat: "php",
+ ServerScope: "io.opentelemetry.contrib.php.laravel", DBScope: "io.opentelemetry.contrib.php.pdo", DBSystem: "mysql",
+ HTTPClient: "io.opentelemetry.contrib.php.guzzle", CacheScope: "io.opentelemetry.contrib.php.laravel"}
+}
+
+// DotNet returns an ASP.NET Core service.
+func DotNet() Stack {
+ return Stack{Language: "dotnet", Framework: "dotnet/aspnetcore", Runtime: "dotnet", StackFormat: "dotnet",
+ ServerScope: "Microsoft.AspNetCore", DBScope: "Npgsql", DBSystem: "postgresql",
+ HTTPClient: "System.Net.Http", CacheScope: "OpenTelemetry.Instrumentation.StackExchangeRedis"}
+}
+
+// Span kinds (OTLP enum values).
+const (
+ KindInternal = 1
+ KindServer = 2
+ KindClient = 3
+ KindProducer = 4
+ KindConsumer = 5
+)
+
+// Status codes (OTLP enum values).
+const (
+ StatusUnset = 0
+ StatusOK = 1
+ StatusError = 2
+)
+
+// OTel log severity numbers.
+const (
+ SevDebug = 5
+ SevInfo = 9
+ SevWarn = 13
+ SevError = 17
+ SevFatal = 21
+)
+
+// SeverityText maps a severity number to its OTel short name.
+func SeverityText(n int) string {
+ switch {
+ case n >= 21:
+ return "FATAL"
+ case n >= 17:
+ return "ERROR"
+ case n >= 13:
+ return "WARN"
+ case n >= 9:
+ return "INFO"
+ case n >= 5:
+ return "DEBUG"
+ default:
+ return "TRACE"
+ }
+}
diff --git a/cli/test/evals/internal/seed/sink.go b/cli/test/evals/internal/seed/sink.go
new file mode 100644
index 00000000..238c1bdf
--- /dev/null
+++ b/cli/test/evals/internal/seed/sink.go
@@ -0,0 +1,195 @@
+package seed
+
+import (
+ "bytes"
+ "compress/gzip"
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/url"
+ "sync"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+)
+
+// Sink is where a scenario's telemetry and objects go: a live backend, or a
+// dry recorder used by the leakage lint and unit tests.
+type Sink interface {
+ CreateProject(name, framework string) (id, token string, err error)
+ // Ingest posts a body to an ingest path with a project token and
+ // returns the response body.
+ Ingest(token, path, contentType string, body []byte, gz bool) ([]byte, error)
+ // Admin calls the dashboard API as the instance owner.
+ Admin(method, path string, query url.Values, body any) (*env.Response, error)
+ // Dry reports a recorder that has no backend behind it.
+ Dry() bool
+ // Org is the instance's organization and owner.
+ Org() (orgID, adminUserID int)
+ // Stub is the switchable HTTP target for monitors (nil when absent).
+ Stub() StubControl
+}
+
+// StubControl is the part of env.Stub the seed uses.
+type StubControl interface {
+ SetHealthy(bool)
+ Healthy() bool
+}
+
+// LiveSink sends to a running backend.
+type LiveSink struct {
+ Ctx context.Context
+ Backend *env.Backend
+ Admin_ *env.Admin
+ HTTP *http.Client
+
+ StubSrv *env.Stub
+
+ mu sync.Mutex
+ projects []env.Project
+}
+
+// NewLiveSink returns a sink for a booted, registered instance.
+func NewLiveSink(ctx context.Context, b *env.Backend, a *env.Admin) *LiveSink {
+ return &LiveSink{Ctx: ctx, Backend: b, Admin_: a, HTTP: &http.Client{Timeout: 2 * time.Minute}}
+}
+
+// Projects returns the projects created so far, tokens included.
+func (l *LiveSink) Projects() []env.Project {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return append([]env.Project{}, l.projects...)
+}
+
+// CreateProject creates a project in the admin's organization.
+func (l *LiveSink) CreateProject(name, framework string) (string, string, error) {
+ ps, err := env.CreateProjects(l.Ctx, l.Backend, l.Admin_, []env.Project{{Name: name, Framework: framework}})
+ if err != nil {
+ return "", "", err
+ }
+ l.mu.Lock()
+ l.projects = append(l.projects, ps[0])
+ l.mu.Unlock()
+ return ps[0].ID, ps[0].Token, nil
+}
+
+// Ingest posts with retries on the admission gate's 503.
+func (l *LiveSink) Ingest(token, path, contentType string, body []byte, gz bool) ([]byte, error) {
+ payload := body
+ if gz {
+ var buf bytes.Buffer
+ w := gzip.NewWriter(&buf)
+ _, _ = w.Write(body)
+ _ = w.Close()
+ payload = buf.Bytes()
+ }
+ var last error
+ for attempt := 0; attempt < 6; attempt++ {
+ req, err := http.NewRequestWithContext(l.Ctx, http.MethodPost, l.Backend.URL+path, bytes.NewReader(payload))
+ if err != nil {
+ return nil, err
+ }
+ req.Header.Set("Authorization", "Bearer "+token)
+ req.Header.Set("Content-Type", contentType)
+ if gz {
+ req.Header.Set("Content-Encoding", "gzip")
+ }
+ resp, err := l.HTTP.Do(req)
+ if err != nil {
+ last = err
+ } else {
+ var msg bytes.Buffer
+ _, _ = msg.ReadFrom(resp.Body)
+ resp.Body.Close()
+ if resp.StatusCode == http.StatusOK {
+ return msg.Bytes(), nil
+ }
+ last = fmt.Errorf("POST %s: HTTP %d: %s", path, resp.StatusCode, msg.String())
+ if resp.StatusCode != http.StatusServiceUnavailable && resp.StatusCode != http.StatusTooManyRequests {
+ return nil, last
+ }
+ }
+ time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond)
+ }
+ return nil, last
+}
+
+// Admin calls the dashboard API with the owner's session.
+func (l *LiveSink) Admin(method, path string, query url.Values, body any) (*env.Response, error) {
+ return l.Backend.API().WithBearer(l.Admin_.JWT).Do(l.Ctx, method, path, query, body)
+}
+
+// Dry reports false.
+func (l *LiveSink) Dry() bool { return false }
+
+// Org returns the organization and owner ids.
+func (l *LiveSink) Org() (int, int) { return l.Admin_.OrgID, l.Admin_.UserID }
+
+// Stub returns the monitor target.
+func (l *LiveSink) Stub() StubControl {
+ if l.StubSrv == nil {
+ return nil
+ }
+ return l.StubSrv
+}
+
+// StubURL returns the stub's base URL.
+func (l *LiveSink) StubURL() string {
+ if l.StubSrv == nil {
+ return ""
+ }
+ return l.StubSrv.URL
+}
+
+// DrySink records what would be sent.
+type DrySink struct {
+ mu sync.Mutex
+ nextID int
+ Bodies map[string]int // path -> count
+ Calls []string
+ project int
+}
+
+// NewDrySink returns an empty recorder.
+func NewDrySink() *DrySink { return &DrySink{Bodies: map[string]int{}} }
+
+// CreateProject returns a synthetic id and token.
+func (d *DrySink) CreateProject(name, framework string) (string, string, error) {
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ d.project++
+ return fmt.Sprintf("00000000-0000-4000-8000-%012d", d.project), fmt.Sprintf("dry-token-%d", d.project), nil
+}
+
+// Ingest counts the body.
+func (d *DrySink) Ingest(_, path, _ string, _ []byte, _ bool) ([]byte, error) {
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ d.Bodies[path]++
+ return nil, nil
+}
+
+// Admin answers creates with a synthetic id and everything else with {}.
+func (d *DrySink) Admin(method, path string, _ url.Values, _ any) (*env.Response, error) {
+ d.mu.Lock()
+ defer d.mu.Unlock()
+ d.Calls = append(d.Calls, method+" "+path)
+ d.nextID++
+ body, _ := json.Marshal(map[string]any{"id": d.nextID, "updatedAt": time.Unix(0, 0).UTC().Format(time.RFC3339Nano)})
+ return &env.Response{Status: http.StatusOK, Body: body}, nil
+}
+
+// Dry reports true.
+func (d *DrySink) Dry() bool { return true }
+
+// Org returns fixed ids.
+func (d *DrySink) Org() (int, int) { return 1, 1 }
+
+// Stub returns a stub that only remembers its state.
+func (d *DrySink) Stub() StubControl { return &dryStub{} }
+
+type dryStub struct{ healthy bool }
+
+func (s *dryStub) SetHealthy(v bool) { s.healthy = v }
+func (s *dryStub) Healthy() bool { return s.healthy }
diff --git a/cli/test/evals/internal/seed/traces.go b/cli/test/evals/internal/seed/traces.go
new file mode 100644
index 00000000..38aa5ace
--- /dev/null
+++ b/cli/test/evals/internal/seed/traces.go
@@ -0,0 +1,247 @@
+package seed
+
+import (
+ "fmt"
+ "sort"
+ "time"
+
+ "github.com/google/uuid"
+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"
+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"
+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
+)
+
+// EntityID is the id the backend gives the endpoint/task/exception row
+// promoted from a span: UUIDv5(projectId, traceId || spanId).
+func EntityID(projectID string, traceID [16]byte, spanID [8]byte) string {
+ pid, err := uuid.Parse(projectID)
+ if err != nil {
+ return ""
+ }
+ var identity [24]byte
+ copy(identity[:16], traceID[:])
+ copy(identity[16:], spanID[:])
+ return uuid.NewSHA1(pid, identity[:]).String()
+}
+
+// resourceKey groups spans that share one OTLP Resource.
+type resourceKey struct {
+ svc *Service
+ version string
+ host string
+}
+
+// spanBatch accumulates the spans of one project, grouped by resource.
+type spanBatch struct {
+ project *Project
+ groups map[resourceKey]map[string][]*tracepb.Span // scope -> spans
+ count int
+}
+
+func newSpanBatch(p *Project) *spanBatch {
+ return &spanBatch{project: p, groups: map[resourceKey]map[string][]*tracepb.Span{}}
+}
+
+func (b *spanBatch) add(k resourceKey, scope string, sp *tracepb.Span) {
+ g := b.groups[k]
+ if g == nil {
+ g = map[string][]*tracepb.Span{}
+ b.groups[k] = g
+ }
+ g[scope] = append(g[scope], sp)
+ b.count++
+}
+
+func (b *spanBatch) request() *coltracepb.ExportTraceServiceRequest {
+ keys := make([]resourceKey, 0, len(b.groups))
+ for k := range b.groups {
+ keys = append(keys, k)
+ }
+ sort.Slice(keys, func(i, j int) bool {
+ if keys[i].svc.Name != keys[j].svc.Name {
+ return keys[i].svc.Name < keys[j].svc.Name
+ }
+ if keys[i].version != keys[j].version {
+ return keys[i].version < keys[j].version
+ }
+ return keys[i].host < keys[j].host
+ })
+ req := &coltracepb.ExportTraceServiceRequest{}
+ for _, k := range keys {
+ rs := &tracepb.ResourceSpans{Resource: k.svc.resource(k.version, k.host)}
+ scopes := make([]string, 0, len(b.groups[k]))
+ for sc := range b.groups[k] {
+ scopes = append(scopes, sc)
+ }
+ sort.Strings(scopes)
+ for _, sc := range scopes {
+ rs.ScopeSpans = append(rs.ScopeSpans, &tracepb.ScopeSpans{
+ Scope: &commonpb.InstrumentationScope{Name: sc, Version: scopeVersion(sc)},
+ Spans: b.groups[k][sc],
+ })
+ }
+ req.ResourceSpans = append(req.ResourceSpans, rs)
+ }
+ return req
+}
+
+func scopeVersion(scope string) string {
+ switch {
+ case len(scope) > 0 && scope[0] == '@':
+ return "0.57.0"
+ case len(scope) > 3 && scope[:3] == "io.":
+ return "2.12.0"
+ default:
+ return "0.63.0"
+ }
+}
+
+// resource builds the OTLP Resource of a service instance.
+func (svc *Service) resource(version, host string) *resourcepb.Resource {
+ attrs := []*commonpb.KeyValue{
+ str("service.name", svc.Name),
+ str("service.version", version),
+ str("deployment.environment.name", svc.Env),
+ str("host.name", host),
+ str("telemetry.sdk.language", svc.Stack.Language),
+ str("telemetry.sdk.name", "opentelemetry"),
+ str("telemetry.sdk.version", sdkVersion(svc.Stack.Language)),
+ }
+ return &resourcepb.Resource{Attributes: attrs}
+}
+
+func sdkVersion(lang string) string {
+ switch lang {
+ case "go":
+ return "1.38.0"
+ case "python":
+ return "1.37.0"
+ case "nodejs":
+ return "2.1.0"
+ case "java":
+ return "1.55.0"
+ case "dotnet":
+ return "1.13.1"
+ default:
+ return "1.0.0"
+ }
+}
+
+func str(k, v string) *commonpb.KeyValue {
+ return &commonpb.KeyValue{Key: k, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: v}}}
+}
+
+// attrList converts a map to OTLP attributes. Integers become intValue, as the
+// backend reads numeric semantic-convention attributes only from intValue.
+func attrList(m map[string]any) []*commonpb.KeyValue {
+ keys := make([]string, 0, len(m))
+ for k := range m {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ out := make([]*commonpb.KeyValue, 0, len(keys))
+ for _, k := range keys {
+ out = append(out, &commonpb.KeyValue{Key: k, Value: anyValue(m[k])})
+ }
+ return out
+}
+
+func anyValue(v any) *commonpb.AnyValue {
+ switch t := v.(type) {
+ case string:
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: t}}
+ case int:
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: int64(t)}}
+ case int64:
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: t}}
+ case float64:
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_DoubleValue{DoubleValue: t}}
+ case bool:
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_BoolValue{BoolValue: t}}
+ case []string:
+ vals := make([]*commonpb.AnyValue, 0, len(t))
+ for _, s := range t {
+ vals = append(vals, anyValue(s))
+ }
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: vals}}}
+ default:
+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: fmt.Sprint(t)}}
+ }
+}
+
+func nanos(t time.Time) uint64 { return uint64(t.UnixNano()) }
+
+// emit walks a request tree and adds its spans to per-project batches. A
+// non-detailed request sends only its root and every downstream SERVER span,
+// so endpoint rows and percentiles stay complete while the span volume stays
+// bounded.
+func (s *Scenario) emit(r *Request, batches map[*Project]*spanBatch, logs map[*Service][]logOut) {
+ rootSvc := r.e.svc
+ var walk func(sp *Span, parent *Span, svc *Service, host string, abs time.Time, depthDetail bool)
+ walk = func(sp *Span, parent *Span, svc *Service, host string, abs time.Time, detail bool) {
+ if sp.Service != nil {
+ svc = sp.Service
+ }
+ if sp.Host != "" {
+ host = sp.Host
+ }
+ start := abs.Add(sp.Start)
+ send := detail || parent == nil || sp.Kind == KindServer
+ if send {
+ pb := &tracepb.Span{
+ TraceId: r.TraceID[:],
+ SpanId: sp.SpanID[:],
+ Name: sp.Name,
+ Kind: tracepb.Span_SpanKind(sp.Kind),
+ StartTimeUnixNano: nanos(start),
+ EndTimeUnixNano: nanos(start.Add(sp.Dur)),
+ Attributes: attrList(sp.Attrs),
+ Status: &tracepb.Status{Code: tracepb.Status_StatusCode(sp.Status), Message: sp.StatusMsg},
+ }
+ if parent != nil {
+ pb.ParentSpanId = parent.SpanID[:]
+ }
+ for _, ev := range sp.Events {
+ pb.Events = append(pb.Events, &tracepb.Span_Event{
+ TimeUnixNano: nanos(start.Add(ev.At)),
+ Name: ev.Name,
+ Attributes: attrList(ev.Attrs),
+ })
+ }
+ b := batches[svc.p]
+ if b == nil {
+ b = newSpanBatch(svc.p)
+ batches[svc.p] = b
+ }
+ b.add(resourceKey{svc: svc, version: svc.VersionAt(r.Start), host: host}, sp.Scope, pb)
+ s.stats.Spans++
+ if len(sp.Events) > 0 {
+ s.stats.Exceptions += len(sp.Events)
+ }
+ }
+ if detail {
+ for _, l := range sp.logs {
+ logs[svc] = append(logs[svc], logOut{rec: l, at: start.Add(l.at), traceID: r.TraceID, spanID: sp.SpanID, host: host, version: svc.VersionAt(r.Start)})
+ }
+ }
+ for _, c := range sp.Children {
+ walk(c, sp, svc, host, start, detail)
+ }
+ }
+ walk(r.Root, nil, rootSvc, r.Host, r.Start, r.Detail)
+ if r.Detail && !r.e.Task && !r.e.SkipAccessLog {
+ // The access log line a request logger writes on completion,
+ // correlated with the trace like any OTel log bridge does.
+ end := r.Start.Add(r.Root.Dur)
+ path, _ := r.Root.Attrs["url.path"].(string)
+ body := fmt.Sprintf("%s %s %d %dms", r.e.Method, path, r.Status, r.Root.Dur.Milliseconds())
+ logs[rootSvc] = append(logs[rootSvc], logOut{
+ rec: logRecord{severity: SevInfo, body: body, attrs: map[string]any{
+ "http.request.method": r.e.Method, "url.path": path, "http.route": r.e.Route,
+ "http.response.status_code": r.Status, "duration_ms": r.Root.Dur.Milliseconds(),
+ }},
+ at: end, traceID: r.TraceID, spanID: r.SpanID, host: r.Host, version: r.Version,
+ })
+ }
+}
diff --git a/cli/test/evals/internal/seed/verify.go b/cli/test/evals/internal/seed/verify.go
new file mode 100644
index 00000000..b35a81a2
--- /dev/null
+++ b/cli/test/evals/internal/seed/verify.go
@@ -0,0 +1,109 @@
+package seed
+
+import (
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "strings"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+)
+
+// Verify reads the seeded data back through the dashboard API until every
+// declared endpoint is listed, so a run never starts on half-ingested data.
+// After timeout the run is an infra_error.
+func (s *Scenario) Verify(timeout time.Duration) error {
+ if s.Sink.Dry() {
+ return nil
+ }
+ type want struct {
+ p *Project
+ names []string
+ win time.Duration
+ }
+ var wants []want
+ for _, p := range s.projects {
+ if p.Kind != OTel {
+ continue
+ }
+ w := want{p: p}
+ for _, svc := range s.services {
+ if svc.p != p {
+ continue
+ }
+ for _, e := range svc.endpoints {
+ if len(e.shapes) == 0 || len(e.Requests()) == 0 {
+ continue
+ }
+ w.names = append(w.names, e.Name())
+ if e.window > w.win {
+ w.win = e.window
+ }
+ }
+ }
+ if len(w.names) > 0 {
+ if w.win == 0 {
+ w.win = 24 * time.Hour
+ }
+ wants = append(wants, w)
+ }
+ }
+ deadline := time.Now().Add(timeout)
+ for _, w := range wants {
+ for {
+ missing, err := s.missingEndpoints(w.p, w.names, w.win)
+ if err == nil && len(missing) == 0 {
+ break
+ }
+ if time.Now().After(deadline) {
+ if err != nil {
+ return fmt.Errorf("verify %s: %w", w.p.Name, err)
+ }
+ return fmt.Errorf("verify %s: endpoints never became visible: %s", w.p.Name, strings.Join(missing, ", "))
+ }
+ time.Sleep(250 * time.Millisecond)
+ }
+ }
+ return nil
+}
+
+func (s *Scenario) missingEndpoints(p *Project, names []string, win time.Duration) ([]string, error) {
+ seen := map[string]bool{}
+ for page := 1; page <= 5; page++ {
+ res, err := s.Sink.Admin(http.MethodPost, "/api/endpoints/grouped", env.ProjectQuery(p.ID), map[string]any{
+ "fromDate": s.Now.Add(-win - time.Hour).Format(time.RFC3339), "toDate": s.Now.Add(time.Hour).Format(time.RFC3339),
+ "orderBy": "count", "pagination": map[string]int{"page": page, "pageSize": 100},
+ })
+ if err != nil {
+ return nil, err
+ }
+ if err := res.Err("list endpoints"); err != nil {
+ return nil, err
+ }
+ var out struct {
+ Data []struct {
+ Endpoint string `json:"endpoint"`
+ } `json:"data"`
+ Pagination struct {
+ TotalPages int `json:"totalPages"`
+ } `json:"pagination"`
+ }
+ if err := json.Unmarshal(res.Body, &out); err != nil {
+ return nil, err
+ }
+ for _, d := range out.Data {
+ seen[d.Endpoint] = true
+ }
+ if page >= out.Pagination.TotalPages {
+ break
+ }
+ }
+ var missing []string
+ for _, n := range names {
+ if !seen[n] {
+ missing = append(missing, n)
+ }
+ }
+ return missing, nil
+}
diff --git a/cli/test/evals/internal/suite/fixtures_test.go b/cli/test/evals/internal/suite/fixtures_test.go
new file mode 100644
index 00000000..20857cd8
--- /dev/null
+++ b/cli/test/evals/internal/suite/fixtures_test.go
@@ -0,0 +1,63 @@
+package suite_test
+
+import (
+ "io/fs"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "gopkg.in/yaml.v3"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+)
+
+// TestFixtureHistory checks every fixture's history.yaml: each overlay
+// directory holds at least one file (git does not keep empty directories, so
+// an empty overlay exists only in the checkout that made it), and every
+// absent path is a file of the fixture.
+func TestFixtureHistory(t *testing.T) {
+ files, err := filepath.Glob(filepath.Join(env.EvalsRoot(), "testdata", "repos", "*", "history.yaml"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, f := range files {
+ repo := filepath.Dir(f)
+ t.Run(filepath.Base(repo), func(t *testing.T) {
+ raw, err := os.ReadFile(f)
+ if err != nil {
+ t.Fatal(err)
+ }
+ var hist struct {
+ Commits []struct {
+ Dir string `yaml:"dir"`
+ Absent []string `yaml:"absent"`
+ } `yaml:"commits"`
+ }
+ if err := yaml.Unmarshal(raw, &hist); err != nil {
+ t.Fatal(err)
+ }
+ for i, c := range hist.Commits {
+ if c.Dir != "" && !hasFile(filepath.Join(repo, c.Dir)) {
+ t.Errorf("commit %d: overlay %s has no files; drop its dir: key", i+1, c.Dir)
+ }
+ for _, a := range c.Absent {
+ if !exists(filepath.Join(repo, filepath.FromSlash(a))) {
+ t.Errorf("commit %d: absent path %s is not in the fixture", i+1, a)
+ }
+ }
+ }
+ })
+ }
+}
+
+func hasFile(dir string) bool {
+ found := false
+ _ = filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {
+ if err == nil && !d.IsDir() {
+ found = true
+ return fs.SkipAll
+ }
+ return nil
+ })
+ return found
+}
diff --git a/cli/test/evals/internal/suite/suite_test.go b/cli/test/evals/internal/suite/suite_test.go
new file mode 100644
index 00000000..905e0295
--- /dev/null
+++ b/cli/test/evals/internal/suite/suite_test.go
@@ -0,0 +1,134 @@
+// Package suite_test checks the case catalog as a whole: every case loads,
+// references things that exist, uses valid checklist ids, and seeds data
+// that does not give the answer away.
+package suite_test
+
+import (
+ "fmt"
+ "os"
+ "path/filepath"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/checklist"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/grade"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"
+ _ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios/all"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"
+)
+
+func loadAll(t *testing.T) []*catalog.Case {
+ t.Helper()
+ cases, err := catalog.LoadCases(env.EvalsRoot())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(cases) == 0 {
+ t.Fatal("no cases")
+ }
+ return cases
+}
+
+func exists(p string) bool { _, err := os.Stat(p); return err == nil }
+
+func TestCaseReferences(t *testing.T) {
+ root := env.EvalsRoot()
+ seen := map[string]bool{}
+ for _, c := range loadAll(t) {
+ t.Run(c.ID, func(t *testing.T) {
+ if _, err := scenarios.Get(c.Seed); err != nil {
+ t.Error(err)
+ }
+ seen[c.Seed] = true
+ if r := c.Workspace.Repo; r != "" && !exists(filepath.Join(root, "testdata", "repos", r)) {
+ t.Errorf("fixture %s does not exist", r)
+ }
+ checkIDs := func(where string, ids ...string) {
+ for _, id := range ids {
+ if id != "" && !strings.Contains(id, "{{") && !checklist.IsID(id) {
+ t.Errorf("%s: %q is not a checklist id (valid: %s)", where, id, strings.Join(checklist.IDs(), ", "))
+ }
+ }
+ }
+ if a := c.Expect.Answer; a != nil {
+ checkIDs("expect.answer.checklist_item", a.ChecklistItem)
+ checkIDs("expect.answer.any_of", a.AnyOf...)
+ checkIDs("expect.answer.not", a.Not...)
+ if c.Suite == "perf" && a.ChecklistItem == "" && len(a.AnyOf) == 0 {
+ t.Error("perf answers need checklist_item or any_of")
+ }
+ }
+ for arm, steps := range map[string][]catalog.Step{"oracle": c.Oracle} {
+ for i, s := range steps {
+ if s.Submit != nil {
+ item, _ := s.Submit["checklist_item"].(string)
+ checkIDs(fmt.Sprintf("%s[%d].submit", arm, i), item)
+ }
+ if s.WriteFile != nil && s.WriteFile.From != "" && !exists(filepath.Join(root, "testdata", s.WriteFile.From)) {
+ t.Errorf("%s[%d] writes from %s, which does not exist", arm, i, s.WriteFile.From)
+ }
+ }
+ }
+ last := c.Oracle[len(c.Oracle)-1]
+ if last.Submit == nil {
+ t.Error("the oracle must end with submit")
+ }
+ if len(c.Judge.Rubric) == 0 {
+ t.Error("perf cases need a judge rubric")
+ }
+ })
+ }
+}
+
+// TestLeakage seeds every case into a dry sink and lints the strings it
+// would send, plus the prompt, for terms that name the answer.
+func TestLeakage(t *testing.T) {
+ for _, c := range loadAll(t) {
+ t.Run(c.ID, func(t *testing.T) {
+ s := seed.NewScenario(c.ID, time.Date(2026, 10, 17, 12, 0, 0, 0, time.UTC), seed.NewDrySink())
+ if err := scenarios.Run(c.Seed, s); err != nil {
+ t.Fatal(err)
+ }
+ for _, f := range grade.Lint(c, s.Strings()) {
+ t.Error(f)
+ }
+ if st := s.Stats(); st.Spans > 25000 {
+ t.Errorf("seed sends %d spans, over the ~20k budget", st.Spans)
+ }
+ })
+ }
+}
+
+func TestPresets(t *testing.T) {
+ cases := loadAll(t)
+ presets, err := catalog.LoadPresets(env.EvalsRoot())
+ if err != nil {
+ t.Fatal(err)
+ }
+ for name := range presets.Presets {
+ units, err := catalog.Select(cases, presets, catalog.Selection{Preset: name})
+ if err != nil {
+ t.Errorf("preset %s: %v", name, err)
+ continue
+ }
+ for _, u := range units {
+ if name != "full" && !slices.Contains(presets.PrimaryArms[u.Case.Suite], u.Arm) && u.Arm != u.Case.Arms[0] {
+ t.Errorf("preset %s picked non-primary arm %s for %s", name, u.Arm, u.Case.ID)
+ }
+ }
+ }
+ if _, err := catalog.LoadModels(env.EvalsRoot()); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// TestChecklistIsComplete: the enum has every performance.md row.
+func TestChecklistIsComplete(t *testing.T) {
+ if n := len(checklist.Rows()); n != 26 {
+ t.Fatalf("performance.md has %d checklist rows; update the perf catalog when rows change", n)
+ }
+}
diff --git a/cli/test/evals/internal/tools/askuser.go b/cli/test/evals/internal/tools/askuser.go
new file mode 100644
index 00000000..853a9464
--- /dev/null
+++ b/cli/test/evals/internal/tools/askuser.go
@@ -0,0 +1,76 @@
+package tools
+
+import (
+ "context"
+ "encoding/json"
+ "regexp"
+ "sync"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// AskUser answers questions from the case's scripted user rules: the first
+// rule whose regexp matches the question wins, a default rule matches
+// anything, and with no rule the user says they have nothing to add.
+type AskUser struct {
+ Rules []catalog.UserRule
+
+ mu sync.Mutex
+ used map[int]bool
+ Log []record.Question
+}
+
+func (t *AskUser) Spec() Spec {
+ return Spec{Name: "ask_user", Description: "Ask the user a question and wait for the answer. Use it for decisions only the user can make (confirming a change, choosing between candidates, approving a draft).",
+ Parameters: obj(map[string]any{"question": strProp("The question, including any draft or options the user must see.")}, "question")}
+}
+
+func (t *AskUser) Call(_ context.Context, raw json.RawMessage) Result {
+ var in struct {
+ Question string `json:"question"`
+ }
+ if err := decodeArgs(raw, &in); err != nil {
+ return errResult("%v", err)
+ }
+ answer := t.Answer(in.Question)
+ t.mu.Lock()
+ t.Log = append(t.Log, record.Question{Question: in.Question, Answer: answer})
+ t.mu.Unlock()
+ return Result{Output: answer, OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpAskUser, Args: map[string]any{"question": in.Question}}, OK: true, Output: answer}}}
+}
+
+// Answer picks the scripted reply for a question.
+func (t *AskUser) Answer(q string) string {
+ t.mu.Lock()
+ defer t.mu.Unlock()
+ if t.used == nil {
+ t.used = map[int]bool{}
+ }
+ for i, r := range t.Rules {
+ if r.Once && t.used[i] {
+ continue
+ }
+ if r.Match != "" {
+ re, err := regexp.Compile("(?is)" + r.Match)
+ if err != nil || !re.MatchString(q) {
+ continue
+ }
+ t.used[i] = true
+ return r.Reply
+ }
+ if r.Default != "" {
+ t.used[i] = true
+ return r.Default
+ }
+ }
+ return "I don't have anything to add. Please continue."
+}
+
+// Questions returns the exchanges so far.
+func (t *AskUser) Questions() []record.Question {
+ t.mu.Lock()
+ defer t.mu.Unlock()
+ return append([]record.Question{}, t.Log...)
+}
diff --git a/cli/test/evals/internal/tools/coreutils.go b/cli/test/evals/internal/tools/coreutils.go
new file mode 100644
index 00000000..c85b58d9
--- /dev/null
+++ b/cli/test/evals/internal/tools/coreutils.go
@@ -0,0 +1,792 @@
+package tools
+
+import (
+ "bufio"
+ "bytes"
+ "fmt"
+ "io"
+ "io/fs"
+ "os"
+ "path"
+ "path/filepath"
+ "regexp"
+ "sort"
+ "strconv"
+ "strings"
+
+ "mvdan.cc/sh/v3/interp"
+)
+
+// In-process text utilities. They implement the commonly used flags of their
+// GNU namesakes and read files only through the sandbox, so `cat /etc/passwd`
+// fails like any other path outside the workspace.
+
+type uctx struct {
+ hc interp.HandlerContext
+ run *shellRun
+}
+
+func (u *uctx) errf(format string, a ...any) { fmt.Fprintf(u.hc.Stderr, format+"\n", a...) }
+
+func (u *uctx) open(p string) (io.ReadCloser, error) {
+ if p == "-" {
+ if in := stdinOf(u.hc); in != nil {
+ return io.NopCloser(in), nil
+ }
+ return io.NopCloser(strings.NewReader("")), nil
+ }
+ abs, err := u.run.s.Sandbox.Resolve(u.hc.Dir, p)
+ if err != nil {
+ return nil, err
+ }
+ return os.Open(abs)
+}
+
+// inputs concatenates the named files, or stdin when none are given.
+func (u *uctx) inputs(files []string) ([]byte, int) {
+ if len(files) == 0 {
+ files = []string{"-"}
+ }
+ var buf bytes.Buffer
+ code := 0
+ for _, f := range files {
+ rc, err := u.open(f)
+ if err != nil {
+ u.errf("%s: %v", f, err)
+ code = 1
+ continue
+ }
+ _, _ = io.Copy(&buf, rc)
+ rc.Close()
+ }
+ return buf.Bytes(), code
+}
+
+func splitLines(b []byte) []string {
+ s := string(b)
+ if s == "" {
+ return nil
+ }
+ s = strings.TrimSuffix(s, "\n")
+ return strings.Split(s, "\n")
+}
+
+func (u *uctx) writeLines(lines []string) {
+ w := bufio.NewWriter(u.hc.Stdout)
+ for _, l := range lines {
+ w.WriteString(l)
+ w.WriteByte('\n')
+ }
+ w.Flush()
+}
+
+// coreutils are the in-process utilities the sandboxed shell offers;
+// coreutils_more.go adds the rest at init.
+var coreutils = map[string]func(u *uctx, args []string) int{
+ "cat": uCat, "head": uHead, "tail": uTail, "wc": uWc, "grep": uGrep, "egrep": func(u *uctx, a []string) int { return uGrep(u, append([]string{"-E"}, a...)) },
+ "sort": uSort, "uniq": uUniq, "cut": uCut, "tr": uTr, "ls": uLs, "echo": uEcho,
+}
+
+func coreutilNames() []string {
+ var out []string
+ for k := range coreutils {
+ if k != "egrep" && k != "echo" {
+ out = append(out, k)
+ }
+ }
+ sort.Strings(out)
+ return append(out, "xargs", "echo", "printf", "test")
+}
+
+// flags is a tiny getopt: single-dash letters (bundled), --long, and
+// options that take a value.
+type flags struct {
+ set map[string]string
+ args []string
+}
+
+func parseFlags(args []string, withValue string) flags {
+ f := flags{set: map[string]string{}}
+ for i := 0; i < len(args); i++ {
+ a := args[i]
+ if a == "--" {
+ f.args = append(f.args, args[i+1:]...)
+ break
+ }
+ if strings.HasPrefix(a, "--") && len(a) > 2 {
+ k, v, ok := strings.Cut(a[2:], "=")
+ if !ok {
+ v = "true"
+ }
+ f.set["-"+k] = v
+ continue
+ }
+ if strings.HasPrefix(a, "-") && len(a) > 1 && !isNumber(a) {
+ for j := 1; j < len(a); j++ {
+ c := string(a[j])
+ if strings.Contains(withValue, c) {
+ v := a[j+1:]
+ if v == "" && i+1 < len(args) {
+ i++
+ v = args[i]
+ }
+ f.set[c] = v
+ break
+ }
+ f.set[c] = "true"
+ }
+ continue
+ }
+ f.args = append(f.args, a)
+ }
+ return f
+}
+
+func (f flags) has(k string) bool { _, ok := f.set[k]; return ok }
+
+func (f flags) int(k string, def int) int {
+ if v, ok := f.set[k]; ok {
+ n, err := strconv.Atoi(strings.TrimPrefix(v, "+"))
+ if err == nil {
+ return n
+ }
+ }
+ return def
+}
+
+func isNumber(s string) bool {
+ _, err := strconv.Atoi(s)
+ return err == nil
+}
+
+func uCat(u *uctx, args []string) int {
+ f := parseFlags(args, "")
+ data, code := u.inputs(f.args)
+ if f.has("n") {
+ lines := splitLines(data)
+ for i, l := range lines {
+ lines[i] = fmt.Sprintf("%6d\t%s", i+1, l)
+ }
+ u.writeLines(lines)
+ return code
+ }
+ u.hc.Stdout.Write(data)
+ return code
+}
+
+// numericShorthand turns `head -20` into `head -n 20`.
+func numericShorthand(args []string) []string {
+ out := make([]string, 0, len(args))
+ for _, a := range args {
+ if len(a) > 1 && a[0] == '-' && isNumber(a[1:]) {
+ out = append(out, "-n", a[1:])
+ continue
+ }
+ out = append(out, a)
+ }
+ return out
+}
+
+func uHead(u *uctx, args []string) int {
+ f := parseFlags(numericShorthand(args), "nc")
+ data, code := u.inputs(f.args)
+ if f.has("c") {
+ n := f.int("c", 0)
+ if n < len(data) {
+ data = data[:n]
+ }
+ u.hc.Stdout.Write(data)
+ return code
+ }
+ n := f.int("n", 10)
+ lines := splitLines(data)
+ if n < 0 {
+ n = max(0, len(lines)+n)
+ }
+ if n < len(lines) {
+ lines = lines[:n]
+ }
+ u.writeLines(lines)
+ return code
+}
+
+func uTail(u *uctx, args []string) int {
+ f := parseFlags(numericShorthand(args), "nc")
+ data, code := u.inputs(f.args)
+ lines := splitLines(data)
+ v := f.set["n"]
+ if strings.HasPrefix(v, "+") {
+ from := f.int("n", 1)
+ if from-1 < len(lines) {
+ lines = lines[max(0, from-1):]
+ } else {
+ lines = nil
+ }
+ } else {
+ n := f.int("n", 10)
+ if n < len(lines) {
+ lines = lines[len(lines)-n:]
+ }
+ }
+ u.writeLines(lines)
+ return code
+}
+
+func uWc(u *uctx, args []string) int {
+ f := parseFlags(args, "")
+ data, code := u.inputs(f.args)
+ lines := bytes.Count(data, []byte("\n"))
+ words := len(strings.Fields(string(data)))
+ chars := len(data)
+ var parts []string
+ all := !f.has("l") && !f.has("w") && !f.has("c") && !f.has("m")
+ if all || f.has("l") {
+ parts = append(parts, strconv.Itoa(lines))
+ }
+ if all || f.has("w") {
+ parts = append(parts, strconv.Itoa(words))
+ }
+ if all || f.has("c") || f.has("m") {
+ parts = append(parts, strconv.Itoa(chars))
+ }
+ fmt.Fprintln(u.hc.Stdout, strings.Join(parts, " "))
+ return code
+}
+
+// breToERE converts a POSIX basic regexp to Go syntax.
+func breToERE(p string) string {
+ var b strings.Builder
+ for i := 0; i < len(p); i++ {
+ c := p[i]
+ if c == '\\' && i+1 < len(p) {
+ n := p[i+1]
+ switch n {
+ case '(', ')', '|', '+', '?', '{', '}':
+ b.WriteByte(n)
+ default:
+ b.WriteByte('\\')
+ b.WriteByte(n)
+ }
+ i++
+ continue
+ }
+ switch c {
+ case '(', ')', '|', '+', '?', '{', '}':
+ b.WriteByte('\\')
+ }
+ b.WriteByte(c)
+ }
+ return b.String()
+}
+
+func uGrep(u *uctx, args []string) int {
+ f := parseFlags(args, "eABCm")
+ var patterns []string
+ if v, ok := f.set["e"]; ok {
+ patterns = append(patterns, v)
+ }
+ rest := f.args
+ if len(patterns) == 0 {
+ if len(rest) == 0 {
+ u.errf("grep: no pattern")
+ return 2
+ }
+ patterns, rest = []string{rest[0]}, rest[1:]
+ }
+ var res []*regexp.Regexp
+ for _, p := range patterns {
+ switch {
+ case f.has("F"):
+ p = regexp.QuoteMeta(p)
+ case f.has("E") || f.has("P"):
+ default:
+ p = breToERE(p)
+ }
+ if f.has("w") {
+ p = `\b(?:` + p + `)\b`
+ }
+ if f.has("x") {
+ p = `^(?:` + p + `)$`
+ }
+ if f.has("i") {
+ p = "(?i)" + p
+ }
+ re, err := regexp.Compile(p)
+ if err != nil {
+ u.errf("grep: %v", err)
+ return 2
+ }
+ res = append(res, re)
+ }
+ match := func(l string) bool {
+ for _, re := range res {
+ if re.MatchString(l) {
+ return !f.has("v")
+ }
+ }
+ return f.has("v")
+ }
+ files := rest
+ if f.has("r") || f.has("R") {
+ if len(files) == 0 {
+ files = []string{"."}
+ }
+ var expanded []string
+ for _, root := range files {
+ abs, err := u.run.s.Sandbox.Resolve(u.hc.Dir, root)
+ if err != nil {
+ u.errf("grep: %v", err)
+ continue
+ }
+ _ = filepath.WalkDir(abs, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return nil
+ }
+ if d.IsDir() && (d.Name() == ".git" || d.Name() == "node_modules") {
+ return filepath.SkipDir
+ }
+ if !d.IsDir() {
+ rel, _ := filepath.Rel(u.hc.Dir, p)
+ expanded = append(expanded, rel)
+ }
+ return nil
+ })
+ }
+ files = expanded
+ }
+ multi := len(files) > 1 || f.has("r") || f.has("R")
+ if f.has("h") {
+ multi = false
+ }
+ if f.has("H") {
+ multi = true
+ }
+ if len(files) == 0 {
+ files = []string{"-"}
+ }
+ after, before := f.int("A", 0), f.int("B", 0)
+ if c := f.int("C", 0); c > 0 {
+ after, before = c, c
+ }
+ maxCount := f.int("m", -1)
+ found := false
+ w := bufio.NewWriter(u.hc.Stdout)
+ defer w.Flush()
+ for _, file := range files {
+ rc, err := u.open(file)
+ if err != nil {
+ u.errf("grep: %s: %v", file, err)
+ continue
+ }
+ data, _ := io.ReadAll(rc)
+ rc.Close()
+ lines := splitLines(data)
+ count := 0
+ lastPrinted := -1
+ prefix := func(i int, sep string) string {
+ p := ""
+ if multi {
+ p = file + sep
+ }
+ if f.has("n") {
+ p += strconv.Itoa(i+1) + sep
+ }
+ return p
+ }
+ for i, l := range lines {
+ if !match(l) {
+ continue
+ }
+ found = true
+ count++
+ if f.has("q") {
+ return 0
+ }
+ if f.has("l") {
+ fmt.Fprintln(w, file)
+ break
+ }
+ if f.has("c") {
+ if maxCount >= 0 && count >= maxCount {
+ break
+ }
+ continue
+ }
+ if f.has("o") && !f.has("v") {
+ for _, re := range res {
+ for _, m := range re.FindAllString(l, -1) {
+ fmt.Fprintln(w, prefix(i, ":")+m)
+ }
+ }
+ } else {
+ for b := max(lastPrinted+1, i-before); b < i; b++ {
+ fmt.Fprintln(w, prefix(b, "-")+lines[b])
+ }
+ fmt.Fprintln(w, prefix(i, ":")+l)
+ lastPrinted = i
+ for a := i + 1; a <= i+after && a < len(lines); a++ {
+ if !match(lines[a]) {
+ fmt.Fprintln(w, prefix(a, "-")+lines[a])
+ lastPrinted = a
+ }
+ }
+ }
+ if maxCount >= 0 && count >= maxCount {
+ break
+ }
+ }
+ if f.has("c") {
+ if multi {
+ fmt.Fprintf(w, "%s:%d\n", file, count)
+ } else {
+ fmt.Fprintln(w, count)
+ }
+ }
+ }
+ if !found {
+ return 1
+ }
+ return 0
+}
+
+func uSort(u *uctx, args []string) int {
+ f := parseFlags(args, "kt")
+ data, code := u.inputs(f.args)
+ lines := splitLines(data)
+ sep := f.set["t"]
+ keyStart, keyEnd := 0, 0
+ if k, ok := f.set["k"]; ok {
+ a, b, _ := strings.Cut(k, ",")
+ keyStart, _ = strconv.Atoi(strings.TrimRight(a, "nrh"))
+ keyEnd, _ = strconv.Atoi(strings.TrimRight(b, "nrh"))
+ if strings.ContainsAny(a+b, "n") {
+ f.set["n"] = "true"
+ }
+ if strings.ContainsAny(a+b, "r") {
+ f.set["r"] = "true"
+ }
+ }
+ key := func(l string) string {
+ if keyStart == 0 {
+ return l
+ }
+ var fields []string
+ if sep != "" {
+ fields = strings.Split(l, sep)
+ } else {
+ fields = strings.Fields(l)
+ }
+ if keyStart-1 >= len(fields) {
+ return ""
+ }
+ end := len(fields)
+ if keyEnd > 0 && keyEnd < end {
+ end = keyEnd
+ }
+ j := " "
+ if sep != "" {
+ j = sep
+ }
+ return strings.Join(fields[keyStart-1:end], j)
+ }
+ numeric := f.has("n") || f.has("g") || f.has("h")
+ less := func(a, b string) bool {
+ ka, kb := key(a), key(b)
+ if numeric {
+ na, _ := strconv.ParseFloat(strings.TrimSpace(firstNumber(ka)), 64)
+ nb, _ := strconv.ParseFloat(strings.TrimSpace(firstNumber(kb)), 64)
+ if na != nb {
+ return na < nb
+ }
+ return ka < kb
+ }
+ if f.has("f") {
+ return strings.ToLower(ka) < strings.ToLower(kb)
+ }
+ return ka < kb
+ }
+ sort.SliceStable(lines, func(i, j int) bool {
+ if f.has("r") {
+ return less(lines[j], lines[i])
+ }
+ return less(lines[i], lines[j])
+ })
+ if f.has("u") {
+ var out []string
+ for i, l := range lines {
+ if i == 0 || key(l) != key(lines[i-1]) {
+ out = append(out, l)
+ }
+ }
+ lines = out
+ }
+ u.writeLines(lines)
+ return code
+}
+
+var numRe = regexp.MustCompile(`-?[0-9]+(\.[0-9]+)?([eE][-+]?[0-9]+)?`)
+
+func firstNumber(s string) string {
+ if m := numRe.FindString(s); m != "" {
+ return m
+ }
+ return "0"
+}
+
+func uUniq(u *uctx, args []string) int {
+ f := parseFlags(args, "")
+ data, code := u.inputs(f.args)
+ lines := splitLines(data)
+ type run struct {
+ line string
+ n int
+ }
+ var runs []run
+ for _, l := range lines {
+ cmp := l
+ if f.has("i") {
+ cmp = strings.ToLower(l)
+ }
+ if len(runs) > 0 {
+ last := runs[len(runs)-1].line
+ if f.has("i") {
+ last = strings.ToLower(last)
+ }
+ if last == cmp {
+ runs[len(runs)-1].n++
+ continue
+ }
+ }
+ runs = append(runs, run{l, 1})
+ }
+ var out []string
+ for _, r := range runs {
+ if f.has("d") && r.n < 2 || f.has("u") && r.n > 1 {
+ continue
+ }
+ if f.has("c") {
+ out = append(out, fmt.Sprintf("%7d %s", r.n, r.line))
+ } else {
+ out = append(out, r.line)
+ }
+ }
+ u.writeLines(out)
+ return code
+}
+
+func parseRanges(spec string) [][2]int {
+ var out [][2]int
+ for _, part := range strings.Split(spec, ",") {
+ a, b, isRange := strings.Cut(part, "-")
+ lo, _ := strconv.Atoi(a)
+ hi := lo
+ if isRange {
+ if b == "" {
+ hi = 1 << 30
+ } else {
+ hi, _ = strconv.Atoi(b)
+ }
+ if a == "" {
+ lo = 1
+ }
+ }
+ out = append(out, [2]int{lo, hi})
+ }
+ return out
+}
+
+func inRanges(i int, rs [][2]int) bool {
+ for _, r := range rs {
+ if i >= r[0] && i <= r[1] {
+ return true
+ }
+ }
+ return false
+}
+
+func uCut(u *uctx, args []string) int {
+ f := parseFlags(args, "dfc")
+ data, code := u.inputs(f.args)
+ var out []string
+ for _, l := range splitLines(data) {
+ if c, ok := f.set["c"]; ok {
+ rs := parseRanges(c)
+ var b strings.Builder
+ for i, r := range []rune(l) {
+ if inRanges(i+1, rs) {
+ b.WriteRune(r)
+ }
+ }
+ out = append(out, b.String())
+ continue
+ }
+ d := "\t"
+ if v, ok := f.set["d"]; ok {
+ d = v
+ }
+ fields := strings.Split(l, d)
+ rs := parseRanges(f.set["f"])
+ var keep []string
+ for i, x := range fields {
+ if inRanges(i+1, rs) {
+ keep = append(keep, x)
+ }
+ }
+ out = append(out, strings.Join(keep, d))
+ }
+ u.writeLines(out)
+ return code
+}
+
+func expandSet(s string) []rune {
+ s = strings.NewReplacer(`\n`, "\n", `\t`, "\t", `\\`, `\`,
+ "[:upper:]", "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "[:lower:]", "abcdefghijklmnopqrstuvwxyz",
+ "[:digit:]", "0123456789", "[:space:]", " \t\n\r\f\v", "[:punct:]", "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~").Replace(s)
+ rs := []rune(s)
+ var out []rune
+ for i := 0; i < len(rs); i++ {
+ if i+2 < len(rs) && rs[i+1] == '-' {
+ for c := rs[i]; c <= rs[i+2]; c++ {
+ out = append(out, c)
+ }
+ i += 2
+ continue
+ }
+ out = append(out, rs[i])
+ }
+ return out
+}
+
+func uTr(u *uctx, args []string) int {
+ f := parseFlags(args, "")
+ data, _ := u.inputs(nil)
+ in := []rune(string(data))
+ if len(f.args) == 0 {
+ u.errf("tr: missing operand")
+ return 1
+ }
+ set1 := expandSet(f.args[0])
+ contains := func(set []rune, r rune) int {
+ for i, x := range set {
+ if x == r {
+ return i
+ }
+ }
+ return -1
+ }
+ var out []rune
+ switch {
+ case f.has("d"):
+ for _, r := range in {
+ if contains(set1, r) < 0 {
+ out = append(out, r)
+ }
+ }
+ case len(f.args) >= 2:
+ set2 := expandSet(f.args[1])
+ for _, r := range in {
+ if i := contains(set1, r); i >= 0 && len(set2) > 0 {
+ out = append(out, set2[min(i, len(set2)-1)])
+ } else {
+ out = append(out, r)
+ }
+ }
+ default:
+ out = in
+ }
+ if f.has("s") {
+ sq := set1
+ if len(f.args) >= 2 && !f.has("d") {
+ sq = expandSet(f.args[1])
+ }
+ var squeezed []rune
+ for i, r := range out {
+ if i > 0 && r == out[i-1] && contains(sq, r) >= 0 {
+ continue
+ }
+ squeezed = append(squeezed, r)
+ }
+ out = squeezed
+ }
+ io.WriteString(u.hc.Stdout, string(out))
+ return 0
+}
+
+func uLs(u *uctx, args []string) int {
+ f := parseFlags(args, "")
+ targets := f.args
+ if len(targets) == 0 {
+ targets = []string{"."}
+ }
+ code := 0
+ for _, t := range targets {
+ abs, err := u.run.s.Sandbox.Resolve(u.hc.Dir, t)
+ if err != nil {
+ u.errf("ls: %v", err)
+ code = 2
+ continue
+ }
+ info, err := os.Stat(abs)
+ if err != nil {
+ u.errf("ls: %s: no such file or directory", t)
+ code = 2
+ continue
+ }
+ if !info.IsDir() {
+ fmt.Fprintln(u.hc.Stdout, t)
+ continue
+ }
+ var walk func(dir, label string)
+ walk = func(dir, label string) {
+ entries, _ := os.ReadDir(dir)
+ if f.has("R") || len(targets) > 1 {
+ fmt.Fprintf(u.hc.Stdout, "%s:\n", label)
+ }
+ var subdirs []string
+ for _, e := range entries {
+ if !f.has("a") && !f.has("A") && strings.HasPrefix(e.Name(), ".") {
+ continue
+ }
+ name := e.Name()
+ if e.IsDir() {
+ subdirs = append(subdirs, name)
+ }
+ if f.has("l") {
+ fi, _ := e.Info()
+ mode, size := "-rw-r--r--", int64(0)
+ if fi != nil {
+ mode, size = fi.Mode().String(), fi.Size()
+ }
+ fmt.Fprintf(u.hc.Stdout, "%s %8d %s\n", mode, size, name)
+ } else {
+ if e.IsDir() && f.has("p") {
+ name += "/"
+ }
+ fmt.Fprintln(u.hc.Stdout, name)
+ }
+ }
+ if f.has("R") {
+ for _, d := range subdirs {
+ fmt.Fprintln(u.hc.Stdout)
+ walk(filepath.Join(dir, d), path.Join(label, d))
+ }
+ }
+ }
+ walk(abs, t)
+ }
+ return code
+}
+
+// uEcho is only reached through xargs; the interpreter's builtin echo
+// handles every direct use.
+func uEcho(u *uctx, args []string) int {
+ nl := true
+ if len(args) > 0 && args[0] == "-n" {
+ nl, args = false, args[1:]
+ }
+ io.WriteString(u.hc.Stdout, strings.Join(args, " "))
+ if nl {
+ io.WriteString(u.hc.Stdout, "\n")
+ }
+ return 0
+}
diff --git a/cli/test/evals/internal/tools/files.go b/cli/test/evals/internal/tools/files.go
new file mode 100644
index 00000000..58e85d88
--- /dev/null
+++ b/cli/test/evals/internal/tools/files.go
@@ -0,0 +1,304 @@
+package tools
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "io/fs"
+ "os"
+ "path"
+ "path/filepath"
+ "regexp"
+ "sort"
+ "strings"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// File tools, confined to the workspace. .skills/ is readable but not
+// writable.
+
+func fileOp(name, rel string, ok bool) []record.OpRecord {
+ return []record.OpRecord{{Op: ops.Op{Op: name, Args: map[string]any{"path": rel}}, OK: ok}}
+}
+
+// ReadFile reads a file with line numbers.
+type ReadFile struct{ Sandbox *Sandbox }
+
+func (t *ReadFile) Spec() Spec {
+ return Spec{Name: "read_file", Description: "Read a text file from the workspace. Lines are numbered. Use offset/limit (1-based lines) for long files.",
+ Parameters: obj(map[string]any{
+ "path": strProp("Workspace-relative path."),
+ "offset": intProp("First line to return (1-based)."),
+ "limit": intProp("Maximum number of lines (default 2000)."),
+ }, "path")}
+}
+
+func (t *ReadFile) Call(_ context.Context, raw json.RawMessage) Result {
+ var in struct {
+ Path string `json:"path"`
+ Offset int `json:"offset"`
+ Limit int `json:"limit"`
+ }
+ if err := decodeArgs(raw, &in); err != nil {
+ return errResult("%v", err)
+ }
+ abs, err := t.Sandbox.Resolve("", in.Path)
+ if err != nil {
+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpReadFile, in.Path, false)}
+ }
+ rel := t.Sandbox.Rel(abs)
+ info, err := os.Stat(abs)
+ if err != nil {
+ return Result{Output: fmt.Sprintf("error: %s: no such file", in.Path), Ops: fileOp(ops.OpReadFile, rel, false)}
+ }
+ if info.IsDir() {
+ return Result{Output: fmt.Sprintf("error: %s is a directory; use list_files", in.Path), Ops: fileOp(ops.OpReadFile, rel, false)}
+ }
+ text, err := readFileCapped(abs)
+ if err != nil {
+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpReadFile, rel, false)}
+ }
+ lines := strings.Split(strings.TrimSuffix(text, "\n"), "\n")
+ start := max(1, in.Offset)
+ limit := in.Limit
+ if limit <= 0 {
+ limit = 2000
+ }
+ var b strings.Builder
+ for i := start; i <= len(lines) && i < start+limit; i++ {
+ fmt.Fprintf(&b, "%6d\t%s\n", i, lines[i-1])
+ }
+ if start+limit <= len(lines) {
+ fmt.Fprintf(&b, "[%d more lines; continue with offset %d]\n", len(lines)-(start+limit)+1, start+limit)
+ }
+ out, trunc := Cap(b.String())
+ return Result{Output: out, OK: true, Truncated: trunc, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: map[string]any{"path": rel}}, OK: true, Output: text}}}
+}
+
+// ListFiles lists files under a directory.
+type ListFiles struct{ Sandbox *Sandbox }
+
+func (t *ListFiles) Spec() Spec {
+ return Spec{Name: "list_files", Description: "List files under a workspace directory (recursive, .git skipped). Optional glob filters file names, e.g. *.go.",
+ Parameters: obj(map[string]any{
+ "path": strProp("Directory, default the workspace root."),
+ "glob": strProp("Optional file-name glob."),
+ })}
+}
+
+func (t *ListFiles) Call(_ context.Context, raw json.RawMessage) Result {
+ var in struct {
+ Path string `json:"path"`
+ Glob string `json:"glob"`
+ }
+ if err := decodeArgs(raw, &in); err != nil {
+ return errResult("%v", err)
+ }
+ if in.Path == "" {
+ in.Path = "."
+ }
+ abs, err := t.Sandbox.Resolve("", in.Path)
+ if err != nil {
+ return errResult("%v", err)
+ }
+ var files []string
+ _ = filepath.WalkDir(abs, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return nil
+ }
+ if d.IsDir() && (d.Name() == ".git" || d.Name() == "node_modules" || d.Name() == "__pycache__") {
+ return filepath.SkipDir
+ }
+ if d.IsDir() {
+ return nil
+ }
+ if in.Glob != "" {
+ if ok, _ := path.Match(in.Glob, d.Name()); !ok {
+ return nil
+ }
+ }
+ files = append(files, t.Sandbox.Rel(p))
+ return nil
+ })
+ sort.Strings(files)
+ out, trunc := Cap(strings.Join(files, "\n"))
+ return Result{Output: out, OK: true, Truncated: trunc, Ops: fileOp(ops.OpListFiles, t.Sandbox.Rel(abs), true)}
+}
+
+// Search greps the workspace with a regular expression.
+type Search struct{ Sandbox *Sandbox }
+
+func (t *Search) Spec() Spec {
+ return Spec{Name: "search", Description: "Search workspace files for a regular expression (RE2 syntax). Returns path:line: text.",
+ Parameters: obj(map[string]any{
+ "pattern": strProp("Regular expression."),
+ "path": strProp("Directory or file to search, default the workspace root."),
+ "glob": strProp("Optional file-name glob, e.g. *.py."),
+ }, "pattern")}
+}
+
+func (t *Search) Call(_ context.Context, raw json.RawMessage) Result {
+ var in struct {
+ Pattern string `json:"pattern"`
+ Path string `json:"path"`
+ Glob string `json:"glob"`
+ }
+ if err := decodeArgs(raw, &in); err != nil {
+ return errResult("%v", err)
+ }
+ re, err := regexp.Compile(in.Pattern)
+ if err != nil {
+ return errResult("invalid pattern: %v", err)
+ }
+ if in.Path == "" {
+ in.Path = "."
+ }
+ abs, err := t.Sandbox.Resolve("", in.Path)
+ if err != nil {
+ return errResult("%v", err)
+ }
+ var b strings.Builder
+ hits := 0
+ _ = filepath.WalkDir(abs, func(p string, d fs.DirEntry, err error) error {
+ if err != nil || hits > 500 {
+ return nil
+ }
+ if d.IsDir() && (d.Name() == ".git" || d.Name() == "node_modules") {
+ return filepath.SkipDir
+ }
+ if d.IsDir() {
+ return nil
+ }
+ if in.Glob != "" {
+ if ok, _ := path.Match(in.Glob, d.Name()); !ok {
+ return nil
+ }
+ }
+ data, err := os.ReadFile(p)
+ if err != nil || isBinary(data) {
+ return nil
+ }
+ for i, l := range strings.Split(string(data), "\n") {
+ if re.MatchString(l) {
+ fmt.Fprintf(&b, "%s:%d: %s\n", t.Sandbox.Rel(p), i+1, l)
+ hits++
+ }
+ }
+ return nil
+ })
+ if hits == 0 {
+ b.WriteString("no matches")
+ }
+ out, trunc := Cap(b.String())
+ return Result{Output: out, OK: true, Truncated: trunc, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpSearch, Args: map[string]any{"pattern": in.Pattern, "path": t.Sandbox.Rel(abs)}}, OK: true}}}
+}
+
+func isBinary(b []byte) bool {
+ n := min(len(b), 8000)
+ for _, c := range b[:n] {
+ if c == 0 {
+ return true
+ }
+ }
+ return false
+}
+
+// EditFile replaces an exact string in a file.
+type EditFile struct{ Sandbox *Sandbox }
+
+func (t *EditFile) Spec() Spec {
+ return Spec{Name: "edit_file", Description: "Replace an exact string in a workspace file. old_string must match exactly once unless replace_all is true.",
+ Parameters: obj(map[string]any{
+ "path": strProp("Workspace-relative path."),
+ "old_string": strProp("Exact text to replace."),
+ "new_string": strProp("Replacement text."),
+ "replace_all": map[string]any{"type": "boolean", "description": "Replace every occurrence."},
+ }, "path", "old_string", "new_string")}
+}
+
+func (t *EditFile) Call(_ context.Context, raw json.RawMessage) Result {
+ var in struct {
+ Path string `json:"path"`
+ Old string `json:"old_string"`
+ New string `json:"new_string"`
+ ReplaceAll bool `json:"replace_all"`
+ }
+ if err := decodeArgs(raw, &in); err != nil {
+ return errResult("%v", err)
+ }
+ abs, err := t.Sandbox.Resolve("", in.Path)
+ if err == nil {
+ err = t.Sandbox.Writable(abs)
+ }
+ if err != nil {
+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpEditFile, in.Path, false)}
+ }
+ rel := t.Sandbox.Rel(abs)
+ data, err := os.ReadFile(abs)
+ if err != nil {
+ return Result{Output: fmt.Sprintf("error: %s: no such file", in.Path), Ops: fileOp(ops.OpEditFile, rel, false)}
+ }
+ text := string(data)
+ n := strings.Count(text, in.Old)
+ switch {
+ case in.Old == "":
+ return Result{Output: "error: old_string is empty", Ops: fileOp(ops.OpEditFile, rel, false)}
+ case n == 0:
+ return Result{Output: "error: old_string not found in " + rel, Ops: fileOp(ops.OpEditFile, rel, false)}
+ case n > 1 && !in.ReplaceAll:
+ return Result{Output: fmt.Sprintf("error: old_string matches %d times in %s; add context or set replace_all", n, rel), Ops: fileOp(ops.OpEditFile, rel, false)}
+ }
+ if in.ReplaceAll {
+ text = strings.ReplaceAll(text, in.Old, in.New)
+ } else {
+ text = strings.Replace(text, in.Old, in.New, 1)
+ }
+ if err := os.WriteFile(abs, []byte(text), 0o644); err != nil {
+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpEditFile, rel, false)}
+ }
+ return Result{Output: fmt.Sprintf("edited %s (%d replacement(s))", rel, max(1, n*boolToInt(in.ReplaceAll))), OK: true, Ops: fileOp(ops.OpEditFile, rel, true)}
+}
+
+func boolToInt(b bool) int {
+ if b {
+ return 1
+ }
+ return 0
+}
+
+// WriteFile writes a whole file.
+type WriteFile struct{ Sandbox *Sandbox }
+
+func (t *WriteFile) Spec() Spec {
+ return Spec{Name: "write_file", Description: "Create or overwrite a workspace file with the given content.",
+ Parameters: obj(map[string]any{
+ "path": strProp("Workspace-relative path."),
+ "content": strProp("Full file content."),
+ }, "path", "content")}
+}
+
+func (t *WriteFile) Call(_ context.Context, raw json.RawMessage) Result {
+ var in struct {
+ Path string `json:"path"`
+ Content string `json:"content"`
+ }
+ if err := decodeArgs(raw, &in); err != nil {
+ return errResult("%v", err)
+ }
+ abs, err := t.Sandbox.Resolve("", in.Path)
+ if err == nil {
+ err = t.Sandbox.Writable(abs)
+ }
+ if err != nil {
+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpWriteFile, in.Path, false)}
+ }
+ if err := os.MkdirAll(filepath.Dir(abs), 0o755); err != nil {
+ return errResult("%v", err)
+ }
+ if err := os.WriteFile(abs, []byte(in.Content), 0o644); err != nil {
+ return errResult("%v", err)
+ }
+ return Result{Output: fmt.Sprintf("wrote %s (%d bytes)", t.Sandbox.Rel(abs), len(in.Content)), OK: true, Ops: fileOp(ops.OpWriteFile, t.Sandbox.Rel(abs), true)}
+}
diff --git a/cli/test/evals/internal/tools/jqawk.go b/cli/test/evals/internal/tools/jqawk.go
new file mode 100644
index 00000000..0b443a86
--- /dev/null
+++ b/cli/test/evals/internal/tools/jqawk.go
@@ -0,0 +1,303 @@
+package tools
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "strings"
+
+ awkinterp "github.com/benhoyt/goawk/interp"
+ awkparser "github.com/benhoyt/goawk/parser"
+ "github.com/itchyny/gojq"
+ "mvdan.cc/sh/v3/interp"
+)
+
+// runJQ implements jq with gojq, in-process.
+func runJQ(hc interp.HandlerContext, r *shellRun, args []string) int {
+ u := &uctx{hc: hc, run: r}
+ raw, compact, slurp, nullIn, exitStatus, sortKeys, rawIn, join := false, false, false, false, false, false, false, false
+ indent := 2
+ var filter string
+ haveFilter := false
+ var files []string
+ var varNames []string
+ var varValues []any
+ for i := 0; i < len(args); i++ {
+ a := args[i]
+ switch {
+ case a == "--arg" && i+2 < len(args):
+ varNames = append(varNames, "$"+args[i+1])
+ varValues = append(varValues, args[i+2])
+ i += 2
+ case a == "--argjson" && i+2 < len(args):
+ var v any
+ if err := json.Unmarshal([]byte(args[i+2]), &v); err != nil {
+ u.errf("jq: --argjson %s: invalid JSON", args[i+1])
+ return 2
+ }
+ varNames = append(varNames, "$"+args[i+1])
+ varValues = append(varValues, v)
+ i += 2
+ case a == "--tab":
+ indent = -1
+ case a == "--indent" && i+1 < len(args):
+ fmt.Sscanf(args[i+1], "%d", &indent)
+ i++
+ case strings.HasPrefix(a, "--"):
+ switch a {
+ case "--raw-output":
+ raw = true
+ case "--compact-output":
+ compact = true
+ case "--slurp":
+ slurp = true
+ case "--null-input":
+ nullIn = true
+ case "--exit-status":
+ exitStatus = true
+ case "--sort-keys":
+ sortKeys = true
+ case "--raw-input":
+ rawIn = true
+ case "--join-output":
+ raw, join = true, true
+ case "--monochrome-output", "--color-output":
+ default:
+ u.errf("jq: unsupported option %s", a)
+ return 2
+ }
+ case strings.HasPrefix(a, "-") && len(a) > 1 && !haveFilter:
+ for _, c := range a[1:] {
+ switch c {
+ case 'r':
+ raw = true
+ case 'j':
+ raw, join = true, true
+ case 'c':
+ compact = true
+ case 's':
+ slurp = true
+ case 'n':
+ nullIn = true
+ case 'e':
+ exitStatus = true
+ case 'S':
+ sortKeys = true
+ case 'R':
+ rawIn = true
+ case 'M', 'C', 'a':
+ default:
+ u.errf("jq: unsupported option -%c", c)
+ return 2
+ }
+ }
+ default:
+ if !haveFilter {
+ filter, haveFilter = a, true
+ } else {
+ files = append(files, a)
+ }
+ }
+ }
+ if !haveFilter {
+ filter = "."
+ }
+ _ = sortKeys
+ query, err := gojq.Parse(filter)
+ if err != nil {
+ u.errf("jq: error: %v", err)
+ return 3
+ }
+ code, err := gojq.Compile(query, gojq.WithVariables(varNames), gojq.WithEnvironLoader(func() []string { return nil }))
+ if err != nil {
+ u.errf("jq: error: %v", err)
+ return 3
+ }
+ data, rc := u.inputs(files)
+ if rc != 0 {
+ return 2
+ }
+ var inputs []any
+ switch {
+ case nullIn:
+ inputs = []any{nil}
+ case rawIn:
+ lines := splitLines(data)
+ if slurp {
+ inputs = []any{string(data)}
+ } else {
+ for _, l := range lines {
+ inputs = append(inputs, l)
+ }
+ }
+ default:
+ dec := json.NewDecoder(bytes.NewReader(data))
+ var all []any
+ for {
+ var v any
+ if err := dec.Decode(&v); err != nil {
+ if errors.Is(err, io.EOF) {
+ break
+ }
+ u.errf("jq: error (at <stdin>): %v", err)
+ return 2
+ }
+ all = append(all, normalizeJSON(v))
+ }
+ if slurp {
+ if all == nil {
+ all = []any{}
+ }
+ inputs = []any{all}
+ } else {
+ inputs = all
+ }
+ }
+ var out bytes.Buffer
+ last := any(nil)
+ produced := false
+ status := 0
+ for _, in := range inputs {
+ iter := code.Run(in, varValues...)
+ for {
+ v, ok := iter.Next()
+ if !ok {
+ break
+ }
+ if err, ok := v.(error); ok {
+ var halt *gojq.HaltError
+ if errors.As(err, &halt) {
+ if halt.Value() != nil {
+ u.errf("jq: error: %v", halt.Value())
+ }
+ hc.Stdout.Write(out.Bytes())
+ return halt.ExitCode()
+ }
+ u.errf("jq: error: %v", err)
+ status = 5
+ break
+ }
+ produced, last = true, v
+ if s, ok := v.(string); ok && raw {
+ out.WriteString(s)
+ } else {
+ out.Write(encodeJSON(v, compact, indent))
+ }
+ if !join {
+ out.WriteByte('\n')
+ }
+ }
+ }
+ hc.Stdout.Write(out.Bytes())
+ if status != 0 {
+ return status
+ }
+ if exitStatus && (!produced || last == nil || last == false) {
+ return 1
+ }
+ return 0
+}
+
+// normalizeJSON converts encoding/json values into the types gojq expects.
+func normalizeJSON(v any) any {
+ switch t := v.(type) {
+ case map[string]any:
+ for k, x := range t {
+ t[k] = normalizeJSON(x)
+ }
+ return t
+ case []any:
+ for i, x := range t {
+ t[i] = normalizeJSON(x)
+ }
+ return t
+ case float64:
+ if t == float64(int64(t)) && t < 1<<53 && t > -(1<<53) {
+ return int(t)
+ }
+ return t
+ default:
+ return v
+ }
+}
+
+func encodeJSON(v any, compact bool, indent int) []byte {
+ var buf bytes.Buffer
+ enc := json.NewEncoder(&buf)
+ enc.SetEscapeHTML(false)
+ if !compact {
+ if indent < 0 {
+ enc.SetIndent("", "\t")
+ } else if indent > 0 {
+ enc.SetIndent("", strings.Repeat(" ", indent))
+ }
+ }
+ if err := enc.Encode(v); err != nil {
+ return []byte(fmt.Sprint(v))
+ }
+ return bytes.TrimSuffix(buf.Bytes(), []byte("\n"))
+}
+
+// runAWK implements awk with goawk: no exec, no file writes; file operands
+// are read through the sandbox.
+func runAWK(hc interp.HandlerContext, r *shellRun, args []string) int {
+ u := &uctx{hc: hc, run: r}
+ var vars []string
+ var src string
+ haveSrc := false
+ var files []string
+ for i := 0; i < len(args); i++ {
+ a := args[i]
+ switch {
+ case a == "-F" && i+1 < len(args):
+ vars = append(vars, "FS", unescapeAwk(args[i+1]))
+ i++
+ case strings.HasPrefix(a, "-F") && len(a) > 2:
+ vars = append(vars, "FS", unescapeAwk(a[2:]))
+ case a == "-v" && i+1 < len(args):
+ k, v, _ := strings.Cut(args[i+1], "=")
+ vars = append(vars, k, v)
+ i++
+ case a == "-f":
+ u.errf("awk: -f is not supported; pass the program inline")
+ return 2
+ default:
+ if !haveSrc {
+ src, haveSrc = a, true
+ } else {
+ files = append(files, a)
+ }
+ }
+ }
+ if !haveSrc {
+ u.errf("awk: no program")
+ return 2
+ }
+ prog, err := awkparser.ParseProgram([]byte(src), nil)
+ if err != nil {
+ u.errf("awk: %v", err)
+ return 2
+ }
+ data, rc := u.inputs(files)
+ if rc != 0 {
+ return 2
+ }
+ status, err := awkinterp.ExecProgram(prog, &awkinterp.Config{
+ Stdin: bytes.NewReader(data), Output: hc.Stdout, Error: hc.Stderr,
+ Vars: vars, NoExec: true, NoFileWrites: true, NoFileReads: true, Environ: []string{},
+ })
+ if err != nil {
+ u.errf("awk: %v", err)
+ return 2
+ }
+ return status
+}
+
+func unescapeAwk(s string) string {
+ if s == "\\t" || s == "t" {
+ return "\t"
+ }
+ return s
+}
diff --git a/cli/test/evals/internal/tools/shell.go b/cli/test/evals/internal/tools/shell.go
new file mode 100644
index 00000000..867b1886
--- /dev/null
+++ b/cli/test/evals/internal/tools/shell.go
@@ -0,0 +1,420 @@
+package tools
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "io/fs"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "slices"
+ "sort"
+ "strings"
+ "sync"
+ "time"
+
+ "mvdan.cc/sh/v3/expand"
+ "mvdan.cc/sh/v3/interp"
+ "mvdan.cc/sh/v3/syntax"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// Shell runs bash-syntax command lines in an interpreter (mvdan.cc/sh), not
+// the host shell. Only allowlisted commands exist: the arm's traceway binary,
+// jq and awk (in-process), read-only text utilities (in-process, confined to
+// the workspace), and read-only git. Each traceway invocation inside a
+// pipeline is recorded as its own operation.
+type Shell struct {
+ Sandbox *Sandbox
+ // Bins maps an allowed command name to the executable it runs
+ // (traceway or traceway-write).
+ Bins map[string]string
+ // BinEnv is the environment of those executables (scrubbed: HOME, XDG,
+ // PATH, TZ, LANG only).
+ BinEnv []string
+ // Git enables read-only git subcommands in the workspace.
+ Git bool
+ // Timeout bounds one command line.
+ Timeout time.Duration
+ // Denied names commands that exist on other arms, to explain refusals.
+ Denied map[string]string
+}
+
+// Spec describes the tool.
+func (s *Shell) Spec() Spec {
+ var names []string
+ for n := range s.Bins {
+ names = append(names, n)
+ }
+ sort.Strings(names)
+ allowed := append(names, "jq", "awk")
+ allowed = append(allowed, coreutilNames()...)
+ if s.Git {
+ allowed = append(allowed, "git (log, show, diff, status, blame, ls-files, rev-parse)")
+ }
+ return Spec{
+ Name: "shell",
+ Description: "Run a bash command line in the repository workspace (pipes, redirects, $(...), && and here-docs work). " +
+ "Available commands: " + strings.Join(allowed, ", ") + ". " +
+ "There is no network access other than through the commands above, and no interpreters or package managers. " +
+ "Output is capped at 30000 characters.",
+ Parameters: obj(map[string]any{
+ "command": strProp("The command line to run."),
+ }, "command"),
+ }
+}
+
+type shellRun struct {
+ s *Shell
+ mu sync.Mutex
+ ops []record.OpRecord
+ ctx context.Context
+ runner *interp.Runner
+}
+
+// Call runs one command line.
+func (s *Shell) Call(ctx context.Context, raw json.RawMessage) Result {
+ var in struct {
+ Command string `json:"command"`
+ }
+ if err := decodeArgs(raw, &in); err != nil {
+ return errResult("%v", err)
+ }
+ if strings.TrimSpace(in.Command) == "" {
+ return errResult("command is empty")
+ }
+ file, err := syntax.NewParser(syntax.Variant(syntax.LangBash)).Parse(strings.NewReader(in.Command), "")
+ if err != nil {
+ return errResult("syntax error: %v", err)
+ }
+ timeout := s.Timeout
+ if timeout == 0 {
+ timeout = 2 * time.Minute
+ }
+ ctx, cancel := context.WithTimeout(ctx, timeout)
+ defer cancel()
+
+ var out lockedBuffer
+ run := &shellRun{s: s, ctx: ctx}
+ env := []string{"HOME=" + s.Sandbox.Root, "PATH=/usr/bin:/bin", "TZ=UTC", "LANG=C.UTF-8", "PWD=" + s.Sandbox.Root}
+ runner, err := interp.New(
+ interp.StdIO(nil, &out, &out),
+ interp.Dir(s.Sandbox.Root),
+ interp.Env(expand.ListEnviron(env...)),
+ interp.ExecHandlers(func(next interp.ExecHandlerFunc) interp.ExecHandlerFunc { return run.exec }),
+ interp.OpenHandler(run.open),
+ interp.ReadDirHandler2(run.readDir),
+ interp.StatHandler(run.stat),
+ )
+ if err != nil {
+ return errResult("interpreter: %v", err)
+ }
+ run.runner = runner
+ runErr := runner.Run(ctx, file)
+ exit := 0
+ if runErr != nil {
+ if st, ok := interp.IsExitStatus(runErr); ok {
+ exit = int(st)
+ } else if errors.Is(ctx.Err(), context.DeadlineExceeded) {
+ out.WriteString(fmt.Sprintf("\n[command timed out after %s]", timeout))
+ exit = 124
+ } else {
+ out.WriteString("\n" + runErr.Error())
+ exit = 1
+ }
+ }
+ text := out.String()
+ if exit != 0 {
+ text += fmt.Sprintf("\n[exit status %d]", exit)
+ }
+ capped, trunc := Cap(text)
+ return Result{Output: capped, OK: exit == 0, Truncated: trunc, Ops: run.ops}
+}
+
+func (r *shellRun) record(op record.OpRecord) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ r.ops = append(r.ops, op)
+}
+
+func (r *shellRun) exec(ctx context.Context, args []string) error {
+ hc := interp.HandlerCtx(ctx)
+ name := args[0]
+ if strings.Contains(name, "/") {
+ name = filepath.Base(name)
+ }
+ if bin, ok := r.s.Bins[name]; ok {
+ return r.execTraceway(ctx, hc, bin, args)
+ }
+ if name == "git" && r.s.Git {
+ return r.execGit(ctx, hc, args)
+ }
+ if name == "jq" {
+ return exitStatus(runJQ(hc, r, args[1:]))
+ }
+ if name == "awk" {
+ return exitStatus(runAWK(hc, r, args[1:]))
+ }
+ if name == "xargs" {
+ return exitStatus(r.xargs(ctx, hc, args[1:]))
+ }
+ if f, ok := coreutils[name]; ok {
+ return exitStatus(f(&uctx{hc: hc, run: r}, args[1:]))
+ }
+ msg := fmt.Sprintf("%s: command not found in this sandbox", name)
+ if why, ok := r.s.Denied[name]; ok {
+ msg = fmt.Sprintf("%s: %s", name, why)
+ }
+ fmt.Fprintln(hc.Stderr, msg)
+ return interp.NewExitStatus(127)
+}
+
+func exitStatus(code int) error {
+ if code == 0 {
+ return nil
+ }
+ return interp.NewExitStatus(uint8(code))
+}
+
+// limitedBuffer keeps the first n bytes it is given.
+type limitedBuffer struct {
+ buf bytes.Buffer
+ n int
+ trunc bool
+}
+
+func (l *limitedBuffer) Write(p []byte) (int, error) {
+ if room := l.n - l.buf.Len(); room > 0 {
+ if len(p) > room {
+ l.buf.Write(p[:room])
+ l.trunc = true
+ } else {
+ l.buf.Write(p)
+ }
+ } else if len(p) > 0 {
+ l.trunc = true
+ }
+ return len(p), nil
+}
+
+func stdinOf(hc interp.HandlerContext) io.Reader {
+ if hc.Stdin == nil {
+ return nil
+ }
+ if f, ok := hc.Stdin.(*os.File); ok && f == nil {
+ return nil
+ }
+ return hc.Stdin
+}
+
+func (r *shellRun) execTraceway(ctx context.Context, hc interp.HandlerContext, bin string, args []string) error {
+ cmd := exec.CommandContext(ctx, bin, args[1:]...)
+ cmd.Dir = hc.Dir
+ cmd.Env = r.s.BinEnv
+ if in := stdinOf(hc); in != nil {
+ cmd.Stdin = in
+ }
+ so := &limitedBuffer{n: 200000}
+ se := &limitedBuffer{n: 20000}
+ cmd.Stdout = io.MultiWriter(hc.Stdout, so)
+ cmd.Stderr = io.MultiWriter(hc.Stderr, se)
+ err := cmd.Run()
+ exit := 0
+ if err != nil {
+ var ee *exec.ExitError
+ if errors.As(err, &ee) {
+ exit = ee.ExitCode()
+ } else {
+ fmt.Fprintln(hc.Stderr, err)
+ exit = 126
+ }
+ }
+ output := so.buf.String()
+ if se.buf.Len() > 0 {
+ output += "\n" + se.buf.String()
+ }
+ op := ops.ParseCLI(args[1:])
+ r.record(record.OpRecord{Op: op, Argv: append([]string{filepath.Base(bin)}, args[1:]...), Exit: exit, OK: exit == 0, Output: output, Truncated: so.trunc})
+ return exitStatus(exit)
+}
+
+var gitAllowed = []string{"log", "show", "diff", "status", "blame", "ls-files", "rev-parse", "shortlog"}
+
+func (r *shellRun) execGit(ctx context.Context, hc interp.HandlerContext, args []string) error {
+ sub := ""
+ for _, a := range args[1:] {
+ if !strings.HasPrefix(a, "-") {
+ sub = a
+ break
+ }
+ if a == "-C" || strings.HasPrefix(a, "--git-dir") || strings.HasPrefix(a, "--work-tree") || strings.HasPrefix(a, "-c") {
+ fmt.Fprintln(hc.Stderr, "git: global options are not allowed in this sandbox")
+ return interp.NewExitStatus(2)
+ }
+ }
+ if !slices.Contains(gitAllowed, sub) {
+ fmt.Fprintf(hc.Stderr, "git %s: only read-only git (%s) is available\n", sub, strings.Join(gitAllowed, ", "))
+ return interp.NewExitStatus(2)
+ }
+ cmd := exec.CommandContext(ctx, "git", append([]string{"--no-pager"}, args[1:]...)...)
+ cmd.Dir = hc.Dir
+ cmd.Env = []string{"HOME=" + r.s.Sandbox.Root, "PATH=/usr/bin:/bin", "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_PAGER=cat", "PAGER=cat", "TZ=UTC", "LANG=C.UTF-8", "GIT_OPTIONAL_LOCKS=0"}
+ so := &limitedBuffer{n: 100000}
+ cmd.Stdout = io.MultiWriter(hc.Stdout, so)
+ cmd.Stderr = hc.Stderr
+ err := cmd.Run()
+ exit := 0
+ if err != nil {
+ var ee *exec.ExitError
+ if errors.As(err, &ee) {
+ exit = ee.ExitCode()
+ } else {
+ exit = 126
+ }
+ }
+ r.record(record.OpRecord{Op: ops.Op{Op: ops.OpGit, Args: map[string]any{"subcommand": sub}}, Argv: args, Exit: exit, OK: exit == 0, Output: so.buf.String()})
+ return exitStatus(exit)
+}
+
+func (r *shellRun) open(ctx context.Context, path string, flag int, perm os.FileMode) (io.ReadWriteCloser, error) {
+ hc := interp.HandlerCtx(ctx)
+ abs, err := r.s.Sandbox.Resolve(hc.Dir, path)
+ if err != nil {
+ return nil, err
+ }
+ if flag&(os.O_WRONLY|os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC) != 0 {
+ if err := r.s.Sandbox.Writable(abs); err != nil {
+ return nil, err
+ }
+ r.record(record.OpRecord{Op: ops.Op{Op: ops.OpWriteFile, Args: map[string]any{"path": r.s.Sandbox.Rel(abs)}, Write: false}, OK: true})
+ }
+ return os.OpenFile(abs, flag, perm)
+}
+
+func (r *shellRun) readDir(ctx context.Context, path string) ([]fs.DirEntry, error) {
+ hc := interp.HandlerCtx(ctx)
+ abs, err := r.s.Sandbox.Resolve(hc.Dir, path)
+ if err != nil {
+ return nil, err
+ }
+ return os.ReadDir(abs)
+}
+
+func (r *shellRun) stat(ctx context.Context, name string, follow bool) (fs.FileInfo, error) {
+ hc := interp.HandlerCtx(ctx)
+ abs, err := r.s.Sandbox.Resolve(hc.Dir, name)
+ if err != nil {
+ return nil, os.ErrNotExist
+ }
+ if follow {
+ return os.Stat(abs)
+ }
+ return os.Lstat(abs)
+}
+
+// xargs builds command lines from stdin and runs them through the same
+// allowlisted handler.
+func (r *shellRun) xargs(ctx context.Context, hc interp.HandlerContext, args []string) int {
+ replace := ""
+ perCall := 0
+ i := 0
+ for ; i < len(args); i++ {
+ a := args[i]
+ switch {
+ case a == "-I" && i+1 < len(args):
+ replace = args[i+1]
+ i++
+ case strings.HasPrefix(a, "-I") && len(a) > 2:
+ replace = a[2:]
+ case a == "-n" && i+1 < len(args):
+ fmt.Sscanf(args[i+1], "%d", &perCall)
+ i++
+ case a == "-r" || a == "--no-run-if-empty" || a == "-0":
+ case strings.HasPrefix(a, "-"):
+ fmt.Fprintf(hc.Stderr, "xargs: unsupported option %s\n", a)
+ return 2
+ default:
+ goto done
+ }
+ }
+done:
+ cmdArgs := args[i:]
+ if len(cmdArgs) == 0 {
+ cmdArgs = []string{"echo"}
+ }
+ in := stdinOf(hc)
+ if in == nil {
+ return 0
+ }
+ data, _ := io.ReadAll(in)
+ var items []string
+ if replace != "" {
+ for _, line := range strings.Split(string(data), "\n") {
+ if strings.TrimSpace(line) != "" {
+ items = append(items, strings.TrimSpace(line))
+ }
+ }
+ } else {
+ items = strings.Fields(string(data))
+ }
+ code := 0
+ runOne := func(argv []string) {
+ if err := r.exec(ctx, argv); err != nil {
+ if st, ok := interp.IsExitStatus(err); ok {
+ code = 123
+ _ = st
+ } else {
+ code = 1
+ }
+ }
+ }
+ switch {
+ case replace != "":
+ for _, it := range items {
+ argv := make([]string, len(cmdArgs))
+ for k, a := range cmdArgs {
+ argv[k] = strings.ReplaceAll(a, replace, it)
+ }
+ runOne(argv)
+ }
+ case perCall > 0:
+ for k := 0; k < len(items); k += perCall {
+ end := min(k+perCall, len(items))
+ runOne(append(append([]string{}, cmdArgs...), items[k:end]...))
+ }
+ default:
+ if len(items) > 0 {
+ runOne(append(append([]string{}, cmdArgs...), items...))
+ }
+ }
+ return code
+}
+
+type lockedBuffer struct {
+ mu sync.Mutex
+ buf bytes.Buffer
+}
+
+func (l *lockedBuffer) Write(p []byte) (int, error) {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return l.buf.Write(p)
+}
+
+func (l *lockedBuffer) WriteString(s string) {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ l.buf.WriteString(s)
+}
+
+func (l *lockedBuffer) String() string {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return l.buf.String()
+}
diff --git a/cli/test/evals/internal/tools/shell_test.go b/cli/test/evals/internal/tools/shell_test.go
new file mode 100644
index 00000000..eb8f13cf
--- /dev/null
+++ b/cli/test/evals/internal/tools/shell_test.go
@@ -0,0 +1,144 @@
+package tools
+
+import (
+ "context"
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+func newShell(t *testing.T) (*Shell, string) {
+ t.Helper()
+ root := t.TempDir()
+ if err := os.MkdirAll(filepath.Join(root, "work", ".skills", "traceway"), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ work := filepath.Join(root, "work")
+ os.WriteFile(filepath.Join(work, ".skills", "traceway", "SKILL.md"), []byte("---\nname: traceway\ndescription: 'x'\n---\nbody"), 0o644)
+ os.WriteFile(filepath.Join(work, "notes.txt"), []byte("alpha 3\nbeta 1\ngamma 2\nbeta 1\n"), 0o644)
+ // A stand-in for the traceway binary: prints its args and environment.
+ bin := filepath.Join(root, "bin", "traceway")
+ os.MkdirAll(filepath.Dir(bin), 0o755)
+ os.WriteFile(bin, []byte("#!/bin/sh\nif [ \"$1\" = env ]; then env; exit 0; fi\nif [ \"$1\" = fail ]; then echo '{\"error\":\"not_found\"}' >&2; exit 5; fi\necho '{\"data\":[{\"endpoint\":\"GET /a\",\"p95Duration\":1500000000},{\"endpoint\":\"GET /b\",\"p95Duration\":20000000}]}'\n"), 0o755)
+ sb := &Sandbox{Root: work, ReadOnly: []string{".skills"}}
+ return &Shell{Sandbox: sb, Bins: map[string]string{"traceway": bin}, BinEnv: []string{"HOME=" + root, "PATH=/usr/bin:/bin", "TZ=UTC"}}, work
+}
+
+func run(t *testing.T, s *Shell, cmd string) Result {
+ t.Helper()
+ raw, _ := json.Marshal(map[string]string{"command": cmd})
+ return s.Call(context.Background(), raw)
+}
+
+func TestShellUtilities(t *testing.T) {
+ s, _ := newShell(t)
+ cases := []struct{ cmd, want string }{
+ {`echo hi | grep h`, "hi"},
+ {`sort -k2 -n notes.txt | head -1`, "beta 1"},
+ {`sort notes.txt | uniq -c | sort -rn | head -1 | tr -s ' '`, "2 beta 1"},
+ {`cut -d' ' -f1 notes.txt | tail -n 1`, "beta"},
+ {`awk '{s+=$2} END {print s}' notes.txt`, "7"},
+ {`echo '{"a":[1,2,3]}' | jq '.a | length'`, "3"},
+ {`echo '{"a":"x"}' | jq -r .a`, "x"},
+ {`grep -c beta notes.txt`, "2"},
+ {`wc -l < notes.txt`, "4"},
+ {`X=$(echo nested); echo "got $X"`, "got nested"},
+ }
+ for _, c := range cases {
+ r := run(t, s, c.cmd)
+ if !r.OK || strings.TrimSpace(r.Output) != c.want {
+ t.Errorf("%s: got ok=%v %q, want %q", c.cmd, r.OK, r.Output, c.want)
+ }
+ }
+}
+
+func TestShellConfinement(t *testing.T) {
+ s, work := newShell(t)
+ for _, cmd := range []string{`cat /etc/passwd`, `cat ../../etc/passwd`, `ls /`, `grep root /etc/passwd`, `cd / && cat etc/hostname`} {
+ if r := run(t, s, cmd); r.OK || !strings.Contains(r.Output, "outside the workspace") && !strings.Contains(r.Output, "exit status") {
+ t.Errorf("%s escaped the sandbox: %q", cmd, r.Output)
+ }
+ }
+ for _, cmd := range []string{`curl https://example.com`, `python3 -c 1`, `bash -c id`, `sh -c id`, `node -e 1`, `go version`} {
+ if r := run(t, s, cmd); r.OK || !strings.Contains(r.Output, "not found") {
+ t.Errorf("%s should be unavailable: %q", cmd, r.Output)
+ }
+ }
+ if r := run(t, s, `echo hacked > .skills/traceway/SKILL.md`); r.OK {
+ t.Errorf("wrote into .skills: %q", r.Output)
+ }
+ if r := run(t, s, `echo kept > out.txt && cat out.txt`); !r.OK || strings.TrimSpace(r.Output) != "kept" {
+ t.Errorf("workspace write failed: %q", r.Output)
+ }
+ if _, err := os.Stat(filepath.Join(work, "out.txt")); err != nil {
+ t.Error(err)
+ }
+}
+
+func TestShellRecordsTracewayInPipelines(t *testing.T) {
+ s, _ := newShell(t)
+ r := run(t, s, `traceway endpoints list --since 24h --order-by p95 --output json | jq -r '.data[0].endpoint'`)
+ if !r.OK || strings.TrimSpace(r.Output) != "GET /a" {
+ t.Fatalf("pipeline output %q", r.Output)
+ }
+ if len(r.Ops) != 1 || r.Ops[0].Op.Op != "endpoints.list" || r.Ops[0].Args["since"] != "24h" || r.Ops[0].Args["order_by"] != "p95" {
+ t.Fatalf("ops %+v", r.Ops)
+ }
+ if !strings.Contains(r.Ops[0].Output, "p95Duration") {
+ t.Error("the raw output before the pipe must be kept for grounding")
+ }
+ r = run(t, s, `traceway fail x; traceway exceptions show abc`)
+ if len(r.Ops) != 2 || r.Ops[0].OK || r.Ops[0].Exit != 5 || !r.Ops[1].OK || r.Ops[1].Args["hash"] != "abc" {
+ t.Fatalf("ops %+v", r.Ops)
+ }
+}
+
+// TestToolEnvIsScrubbed asserts plan §3.5: secrets in the harness's own
+// environment never reach tool subprocesses.
+func TestToolEnvIsScrubbed(t *testing.T) {
+ t.Setenv("OPENROUTER_API_KEY", "sk-or-secret")
+ t.Setenv("GITHUB_TOKEN", "ghp_secret")
+ t.Setenv("JWT_SECRET", "jwt-secret-value-that-is-long-enough")
+ s, _ := newShell(t)
+ r := run(t, s, `traceway env`)
+ for _, leak := range []string{"sk-or-secret", "ghp_secret", "jwt-secret-value", "OPENROUTER", "GITHUB_TOKEN", "JWT_SECRET"} {
+ if strings.Contains(r.Output, leak) {
+ t.Errorf("tool environment leaked %s:\n%s", leak, r.Output)
+ }
+ }
+ r = run(t, s, `echo "$OPENROUTER_API_KEY$GITHUB_TOKEN$JWT_SECRET"`)
+ if strings.TrimSpace(r.Output) != "" {
+ t.Errorf("interpreter environment leaked: %q", r.Output)
+ }
+}
+
+func TestEditAndReadFile(t *testing.T) {
+ s, _ := newShell(t)
+ ed := &EditFile{Sandbox: s.Sandbox}
+ res := ed.Call(context.Background(), json.RawMessage(`{"path":"notes.txt","old_string":"gamma 2","new_string":"gamma 9"}`))
+ if !res.OK {
+ t.Fatal(res.Output)
+ }
+ rd := &ReadFile{Sandbox: s.Sandbox}
+ res = rd.Call(context.Background(), json.RawMessage(`{"path":"notes.txt"}`))
+ if !strings.Contains(res.Output, "gamma 9") {
+ t.Fatal(res.Output)
+ }
+ res = ed.Call(context.Background(), json.RawMessage(`{"path":".skills/traceway/SKILL.md","old_string":"body","new_string":"x"}`))
+ if res.OK || !strings.Contains(res.Output, "read-only") {
+ t.Fatalf("edited .skills: %q", res.Output)
+ }
+ res = rd.Call(context.Background(), json.RawMessage(`{"path":"/etc/passwd"}`))
+ if res.OK {
+ t.Fatal("read outside the workspace")
+ }
+}
+
+func TestAskUserRules(t *testing.T) {
+ a := &AskUser{}
+ if got := a.Answer("anything"); !strings.Contains(got, "anything to add") {
+ t.Fatal(got)
+ }
+}
diff --git a/cli/test/evals/internal/tools/skills.go b/cli/test/evals/internal/tools/skills.go
new file mode 100644
index 00000000..7d989529
--- /dev/null
+++ b/cli/test/evals/internal/tools/skills.go
@@ -0,0 +1,118 @@
+package tools
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "os"
+ "path/filepath"
+ "sort"
+ "strings"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// Skill is one installed skill: a directory with SKILL.md and companions.
+type Skill struct {
+ Name string
+ Description string
+ Dir string // absolute
+ Body string // SKILL.md without frontmatter
+}
+
+// LoadSkills reads every <dir>/<name>/SKILL.md.
+func LoadSkills(dir string) ([]Skill, error) {
+ entries, err := os.ReadDir(dir)
+ if err != nil {
+ if os.IsNotExist(err) {
+ return nil, nil
+ }
+ return nil, err
+ }
+ var out []Skill
+ for _, e := range entries {
+ if !e.IsDir() {
+ continue
+ }
+ raw, err := os.ReadFile(filepath.Join(dir, e.Name(), "SKILL.md"))
+ if err != nil {
+ continue
+ }
+ name, desc, body := ParseFrontmatter(string(raw))
+ if name == "" {
+ name = e.Name()
+ }
+ out = append(out, Skill{Name: name, Description: desc, Dir: filepath.Join(dir, e.Name()), Body: body})
+ }
+ sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
+ return out, nil
+}
+
+// ParseFrontmatter splits a SKILL.md into name, description and body.
+func ParseFrontmatter(s string) (name, desc, body string) {
+ if !strings.HasPrefix(s, "---\n") {
+ return "", "", s
+ }
+ end := strings.Index(s[4:], "\n---")
+ if end < 0 {
+ return "", "", s
+ }
+ front := s[4 : 4+end]
+ body = strings.TrimLeft(s[4+end+4:], "\n")
+ for _, line := range strings.Split(front, "\n") {
+ k, v, ok := strings.Cut(line, ":")
+ if !ok {
+ continue
+ }
+ v = strings.TrimSpace(v)
+ if len(v) >= 2 && (v[0] == '\'' && v[len(v)-1] == '\'' || v[0] == '"' && v[len(v)-1] == '"') {
+ v = v[1 : len(v)-1]
+ v = strings.ReplaceAll(v, "''", "'")
+ }
+ switch strings.TrimSpace(k) {
+ case "name":
+ name = v
+ case "description":
+ desc = v
+ }
+ }
+ return name, desc, body
+}
+
+// LoadSkill returns a skill's SKILL.md, the way agent harnesses load a
+// skill on demand. Companion files are read with read_file.
+type LoadSkill struct {
+ Skills []Skill
+ // Rel maps a skill's dir to its workspace-relative path (.skills/<name>).
+ Rel func(abs string) string
+ Loaded *[]string
+}
+
+func (t *LoadSkill) Spec() Spec {
+ var names []string
+ for _, s := range t.Skills {
+ names = append(names, s.Name)
+ }
+ return Spec{Name: "load_skill", Description: "Load an installed skill's instructions (its SKILL.md). Installed skills: " + strings.Join(names, ", ") + ". Files the skill mentions live next to it and can be read with read_file.",
+ Parameters: obj(map[string]any{"name": map[string]any{"type": "string", "enum": names, "description": "Skill name."}}, "name")}
+}
+
+func (t *LoadSkill) Call(_ context.Context, raw json.RawMessage) Result {
+ var in struct {
+ Name string `json:"name"`
+ }
+ if err := decodeArgs(raw, &in); err != nil {
+ return errResult("%v", err)
+ }
+ for _, s := range t.Skills {
+ if s.Name == in.Name {
+ if t.Loaded != nil {
+ *t.Loaded = append(*t.Loaded, s.Name)
+ }
+ out := fmt.Sprintf("Skill %s (files in %s/):\n\n%s", s.Name, t.Rel(s.Dir), s.Body)
+ return Result{Output: out, OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpLoadSkill, Args: map[string]any{"name": s.Name}}, OK: true}}}
+ }
+ }
+ return Result{Output: "error: no skill named " + in.Name, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpLoadSkill, Args: map[string]any{"name": in.Name}}}}}
+}
diff --git a/cli/test/evals/internal/tools/submit.go b/cli/test/evals/internal/tools/submit.go
new file mode 100644
index 00000000..44c080e9
--- /dev/null
+++ b/cli/test/evals/internal/tools/submit.go
@@ -0,0 +1,201 @@
+package tools
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "slices"
+ "strings"
+ "sync"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/checklist"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// Submit names.
+const (
+ SubmitPerf = "submit_perf_diagnosis"
+ SubmitDebug = "submit_debug_diagnosis"
+ Finish = "finish"
+)
+
+// DebugCategories are submit_debug_diagnosis's category values.
+var DebugCategories = []string{"null_reference", "missing_key", "type_or_contract_mismatch", "race_or_duplicate", "timeout", "config", "resource_exhaustion", "logic", "unknown"}
+
+// Submit is a final-answer tool. It validates the payload against its
+// schema and the call ids it cites; a failure goes back to the model for
+// repair, up to MaxRepairs times, after which the last payload is accepted
+// as-is for grading.
+type Submit struct {
+ Kind string // SubmitPerf | SubmitDebug | Finish
+ CallExists func(id string) (ok, exists bool)
+ MaxRepairs int
+
+ mu sync.Mutex
+ repairs int
+}
+
+func (t *Submit) Spec() Spec {
+ switch t.Kind {
+ case SubmitPerf:
+ var rows []string
+ for _, r := range checklist.Rows() {
+ rows = append(rows, fmt.Sprintf("%s (%s)", r.ID, r.Title))
+ }
+ ids := checklist.IDs()
+ item := map[string]any{"type": "string", "enum": ids}
+ return Spec{Name: SubmitPerf, Description: "Submit your final performance diagnosis. This ends the task. " +
+ "checklist_item is the bottleneck checklist row that explains the latency, or none_found when the latency is within expectations, or insufficient_evidence when the telemetry cannot tell. Rows: " + strings.Join(rows, "; ") + ". " +
+ "Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",
+ Parameters: obj(map[string]any{
+ "checklist_item": item,
+ "category": strProp("database, application, network, caching, infrastructure, or none"),
+ "locus": obj(map[string]any{
+ "endpoint": strProp("Endpoint name, e.g. GET /api/orders"),
+ "span": strProp("The dominating span or query"),
+ "service": strProp("Service name"),
+ "host": strProp("Host or instance, when the cause is host-specific"),
+ }),
+ "onset": map[string]any{"type": []string{"string", "null"}, "description": "RFC 3339 time the slowness started, or null"},
+ "evidence": evidenceSchema(),
+ "ruled_out": map[string]any{"type": "array", "items": obj(map[string]any{"checklist_item": item, "call_id": strProp("Call id"), "reason": strProp("Why it is ruled out")}, "checklist_item", "reason")},
+ "fix": strProp("The concrete fix"),
+ "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},
+ "report": strProp("Your report to the user, in Markdown"),
+ }, "checklist_item", "locus", "evidence", "fix", "confidence", "report")}
+ case SubmitDebug:
+ return Spec{Name: SubmitDebug, Description: "Submit your final debugging diagnosis. This ends the task. Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",
+ Parameters: obj(map[string]any{
+ "issue_hash": strProp("The exception group hash (16 hex), if the bug has one"),
+ "occurrence_id": strProp("The occurrence you anchored on, if any"),
+ "category": map[string]any{"type": "string", "enum": DebugCategories},
+ "culprit": obj(map[string]any{
+ "file": strProp("Workspace-relative file"),
+ "symbol": strProp("Function or method"),
+ "line": intProp("Line number"),
+ }, "file"),
+ "evidence": evidenceSchema(),
+ "fix_summary": strProp("What you changed or propose to change"),
+ "files_changed": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
+ "needs_instrumentation": map[string]any{"type": "boolean", "description": "True when the telemetry cannot explain the failure and you added instrumentation instead of a fix"},
+ "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},
+ "report": strProp("Your report to the user, in Markdown"),
+ }, "category", "culprit", "evidence", "fix_summary", "needs_instrumentation", "confidence", "report")}
+ default:
+ return Spec{Name: Finish, Description: "Finish the task and report what you did. This ends the task.",
+ Parameters: obj(map[string]any{
+ "summary": strProp("Your report to the user, in Markdown"),
+ "changes": map[string]any{"type": "array", "items": obj(map[string]any{
+ "action": strProp("What you changed, e.g. snooze alert rule"),
+ "target": strProp("The record, e.g. rule 12"),
+ "new_value": strProp("The new state"),
+ "undo": strProp("How to undo it"),
+ }, "action", "target")},
+ }, "summary", "changes")}
+ }
+}
+
+func evidenceSchema() map[string]any {
+ return map[string]any{"type": "array", "minItems": 1, "items": obj(map[string]any{
+ "call_id": strProp("Id of the tool call, e.g. c04"),
+ "claim": strProp("What that call showed"),
+ }, "call_id", "claim")}
+}
+
+func (t *Submit) Call(_ context.Context, raw json.RawMessage) Result {
+ var payload map[string]any
+ if err := decodeArgs(raw, &payload); err != nil {
+ return t.repair(nil, []string{err.Error()})
+ }
+ problems := t.validate(payload)
+ if len(problems) > 0 {
+ return t.repair(payload, problems)
+ }
+ return t.accept(payload, "")
+}
+
+func (t *Submit) repair(payload map[string]any, problems []string) Result {
+ t.mu.Lock()
+ defer t.mu.Unlock()
+ t.repairs++
+ if t.repairs > t.MaxRepairs && payload != nil {
+ return t.accept(payload, strings.Join(problems, "; "))
+ }
+ return Result{Output: "Your submission was not accepted. Fix these problems and call " + t.Kind + " again:\n- " + strings.Join(problems, "\n- "), OK: false,
+ Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpSubmit, Args: map[string]any{"tool": t.Kind, "accepted": false}}}}}
+}
+
+func (t *Submit) accept(payload map[string]any, problems string) Result {
+ msg := "Submission received."
+ if problems != "" {
+ payload["_validation_problems"] = problems
+ }
+ return Result{Output: msg, OK: true, Final: true, Submission: payload,
+ Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpSubmit, Args: map[string]any{"tool": t.Kind, "accepted": true}}, OK: true}}}
+}
+
+func (t *Submit) validate(p map[string]any) []string {
+ var probs []string
+ str := func(k string) string { s, _ := p[k].(string); return s }
+ need := func(keys ...string) {
+ for _, k := range keys {
+ if _, ok := p[k]; !ok {
+ probs = append(probs, k+" is required")
+ }
+ }
+ }
+ checkEvidence := func() {
+ ev, ok := p["evidence"].([]any)
+ if !ok || len(ev) == 0 {
+ probs = append(probs, "evidence must cite at least one tool call")
+ return
+ }
+ for i, e := range ev {
+ m, _ := e.(map[string]any)
+ id, _ := m["call_id"].(string)
+ if id == "" {
+ probs = append(probs, fmt.Sprintf("evidence[%d].call_id is required", i))
+ continue
+ }
+ if t.CallExists != nil {
+ if _, exists := t.CallExists(id); !exists {
+ probs = append(probs, fmt.Sprintf("evidence[%d].call_id %q is not a call in this session (ids look like c04)", i, id))
+ }
+ }
+ }
+ }
+ switch t.Kind {
+ case SubmitPerf:
+ need("checklist_item", "locus", "evidence", "fix", "confidence", "report")
+ if v := str("checklist_item"); v != "" && !checklist.IsID(v) {
+ probs = append(probs, fmt.Sprintf("checklist_item %q is not one of the allowed values", v))
+ }
+ if rs, ok := p["ruled_out"].([]any); ok {
+ for i, r := range rs {
+ m, _ := r.(map[string]any)
+ if v, _ := m["checklist_item"].(string); v != "" && !checklist.IsID(v) {
+ probs = append(probs, fmt.Sprintf("ruled_out[%d].checklist_item %q is not an allowed value", i, v))
+ }
+ }
+ }
+ checkEvidence()
+ case SubmitDebug:
+ need("category", "culprit", "evidence", "fix_summary", "needs_instrumentation", "confidence", "report")
+ if v := str("category"); v != "" && !slices.Contains(DebugCategories, v) {
+ probs = append(probs, fmt.Sprintf("category %q is not one of %v", v, DebugCategories))
+ }
+ if c, ok := p["culprit"].(map[string]any); ok {
+ if f, _ := c["file"].(string); f == "" {
+ probs = append(probs, "culprit.file is required")
+ }
+ }
+ checkEvidence()
+ default:
+ need("summary", "changes")
+ }
+ if c := str("confidence"); c != "" && !slices.Contains([]string{"low", "medium", "high"}, c) {
+ probs = append(probs, "confidence must be low, medium or high")
+ }
+ return probs
+}
diff --git a/cli/test/evals/internal/tools/tools.go b/cli/test/evals/internal/tools/tools.go
new file mode 100644
index 00000000..781be5e8
--- /dev/null
+++ b/cli/test/evals/internal/tools/tools.go
@@ -0,0 +1,181 @@
+// Package tools implements what the agent under test can call: a sandboxed
+// shell (an interpreter with an allowlist, never the host shell), confined
+// file tools, skills, ask_user, and the structured submit tools. Every call
+// returns the operations it performed so the graders can check the
+// trajectory.
+package tools
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+
+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"
+)
+
+// Spec is a tool's function definition as the model sees it.
+type Spec struct {
+ Name string
+ Description string
+ Parameters map[string]any
+}
+
+// Result is what one call produced.
+type Result struct {
+ Output string
+ OK bool
+ Truncated bool
+ Ops []record.OpRecord
+ // Final is set when a submit/finish payload validated: the run ends.
+ Final bool
+ Submission map[string]any
+}
+
+// Tool is one callable tool.
+type Tool interface {
+ Spec() Spec
+ Call(ctx context.Context, args json.RawMessage) Result
+}
+
+// MaxOutput caps what one shell or file call returns to the model.
+const MaxOutput = 30000
+
+// Cap truncates s to MaxOutput characters with a marker.
+func Cap(s string) (string, bool) { return CapAt(s, MaxOutput) }
+
+// CapAt truncates s to n characters with a marker.
+func CapAt(s string, n int) (string, bool) {
+ if len(s) <= n {
+ return s, false
+ }
+ more := len(s) - n
+ return s[:n] + fmt.Sprintf("\n[output truncated: %d more characters. Narrow the query (smaller window, page size, field selection) or filter the output.]", more), true
+}
+
+// Sandbox confines file access to the run's workspace.
+type Sandbox struct {
+ Root string // absolute workspace path
+ ReadOnly []string // workspace-relative prefixes that cannot be written (.skills)
+}
+
+// ErrOutside is returned for any path that leaves the workspace.
+var ErrOutside = errors.New("path is outside the workspace")
+
+// Resolve maps p (absolute, or relative to base) to an absolute path inside
+// the workspace. Symlinks pointing out of the workspace are refused.
+func (s *Sandbox) Resolve(base, p string) (string, error) {
+ if p == "/dev/null" {
+ return p, nil
+ }
+ if base == "" {
+ base = s.Root
+ }
+ var abs string
+ if filepath.IsAbs(p) {
+ abs = filepath.Clean(p)
+ } else {
+ abs = filepath.Clean(filepath.Join(base, p))
+ }
+ if !within(s.Root, abs) {
+ return "", fmt.Errorf("%s: %w", p, ErrOutside)
+ }
+ // Resolve the longest existing prefix to catch symlink escapes.
+ probe := abs
+ for {
+ real, err := filepath.EvalSymlinks(probe)
+ if err == nil {
+ rootReal, _ := filepath.EvalSymlinks(s.Root)
+ if rootReal == "" {
+ rootReal = s.Root
+ }
+ if !within(rootReal, real) {
+ return "", fmt.Errorf("%s: %w", p, ErrOutside)
+ }
+ break
+ }
+ parent := filepath.Dir(probe)
+ if parent == probe || !within(s.Root, parent) {
+ break
+ }
+ probe = parent
+ }
+ return abs, nil
+}
+
+// Writable reports whether abs may be written.
+func (s *Sandbox) Writable(abs string) error {
+ if abs == "/dev/null" {
+ return nil
+ }
+ rel, err := filepath.Rel(s.Root, abs)
+ if err != nil {
+ return err
+ }
+ for _, ro := range s.ReadOnly {
+ if rel == ro || strings.HasPrefix(rel, ro+string(filepath.Separator)) {
+ return fmt.Errorf("%s is read-only", filepath.ToSlash(rel))
+ }
+ }
+ return nil
+}
+
+// Rel returns the workspace-relative form of abs.
+func (s *Sandbox) Rel(abs string) string {
+ rel, err := filepath.Rel(s.Root, abs)
+ if err != nil {
+ return abs
+ }
+ return filepath.ToSlash(rel)
+}
+
+func within(root, p string) bool {
+ if p == root {
+ return true
+ }
+ return strings.HasPrefix(p, strings.TrimSuffix(root, string(filepath.Separator))+string(filepath.Separator))
+}
+
+// decodeArgs unmarshals tool arguments, tolerating an empty payload.
+func decodeArgs(raw json.RawMessage, v any) error {
+ if len(raw) == 0 || string(raw) == "null" {
+ raw = []byte("{}")
+ }
+ if err := json.Unmarshal(raw, v); err != nil {
+ return fmt.Errorf("invalid arguments: %v", err)
+ }
+ return nil
+}
+
+func errResult(format string, a ...any) Result {
+ return Result{Output: "error: " + fmt.Sprintf(format, a...), OK: false}
+}
+
+// readFileCapped reads a workspace file for a tool.
+func readFileCapped(path string) (string, error) {
+ b, err := os.ReadFile(path)
+ if err != nil {
+ return "", err
+ }
+ return string(b), nil
+}
+
+// obj builds a JSON-schema object.
+func obj(props map[string]any, required ...string) map[string]any {
+ m := map[string]any{"type": "object", "properties": props, "additionalProperties": false}
+ if len(required) > 0 {
+ m["required"] = required
+ }
+ return m
+}
+
+func strProp(desc string) map[string]any {
+ return map[string]any{"type": "string", "description": desc}
+}
+
+func intProp(desc string) map[string]any {
+ return map[string]any{"type": "integer", "description": desc}
+}
diff --git a/cli/test/evals/models.yaml b/cli/test/evals/models.yaml
new file mode 100644
index 00000000..d7ffb190
--- /dev/null
+++ b/cli/test/evals/models.yaml
@@ -0,0 +1,60 @@
+# Pinned OpenRouter models for the eval agents and the judge. Every call goes
+# through OpenRouter Chat Completions with one key (OPENROUTER_API_KEY).
+#
+# Never use a floating alias (~provider/model-latest): results must stay
+# comparable over time, so a model change is a reviewed edit to this file.
+# Prices and supported_parameters were checked against
+# GET https://openrouter.ai/api/v1/models on 2026-10-17. The preflight reads live
+# prices; fallback_price is used only when that list cannot be fetched.
+#
+# estimate is the per-run token envelope the preflight multiplies by live
+# prices until baseline.json carries measured averages (plan §10).
+
+agents:
+ opus:
+ id: anthropic/claude-opus-5.5
+ label: Claude Opus 5.5
+ provider:
+ # Anthropic first so the explicit prompt cache keeps hitting the same
+ # upstream; fallbacks stay allowed so one provider outage is not a
+ # failed run (the serving provider is recorded per run).
+ order: [anthropic]
+ allow_fallbacks: true
+ caching: anthropic
+ reasoning_effort: medium
+ max_tokens_per_turn: 16000
+ estimate:
+ input_tokens: [100000, 400000]
+ cached_fraction: 0.7
+ output_tokens: [5000, 15000]
+ fallback_price: {prompt: 4.00, completion: 20.00, cache_read: 0.20, cache_write: 5.00}
+
+ astra:
+ id: openai/gpt-6-astra
+ label: GPT-6 Astra
+ provider:
+ order: [openai]
+ allow_fallbacks: true
+ caching: automatic
+ reasoning_effort: medium
+ max_tokens_per_turn: 16000
+ estimate:
+ input_tokens: [100000, 400000]
+ cached_fraction: 0.7
+ output_tokens: [5000, 15000]
+ fallback_price: {prompt: 10.00, completion: 50.00, cache_read: 1.00, cache_write: 12.50}
+
+judge:
+ id: anthropic/claude-haiku-5.5
+ label: Claude Haiku 5.5 (judge)
+ provider:
+ order: [anthropic]
+ allow_fallbacks: true
+ caching: none
+ max_tokens_per_turn: 2000
+ temperature: 0
+ estimate:
+ input_tokens: [6000, 15000]
+ cached_fraction: 0
+ output_tokens: [300, 1200]
+ fallback_price: {prompt: 0.10, completion: 0.50, cache_read: 0.01, cache_write: 0.125}
diff --git a/cli/test/evals/presets.yaml b/cli/test/evals/presets.yaml
new file mode 100644
index 00000000..4352c574
--- /dev/null
+++ b/cli/test/evals/presets.yaml
@@ -0,0 +1,16 @@
+# Case and arm selections for `twevals plan|run --preset`. A --cases glob
+# overrides the preset's case filter; --arms overrides its arm policy.
+presets:
+ smoke:
+ description: The starred cases, each suite's primary arm only
+ tags: [smoke]
+ arms: primary
+ core:
+ description: Every case, each suite's primary arm only
+ arms: primary
+ full:
+ description: Every case on every arm it lists (adds cli-noskill)
+ arms: all
+
+primary_arms:
+ perf: [cli+skill]