The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
Harness started
narrow-adventure1372f5f021c7
diff --git a/.gitignore b/.gitignoreindex c7636775..57b592a8 100644--- a/.gitignore+++ b/.gitignore@@ -64,6 +64,11 @@ backend/static/dist//benchmarks/_artifact/benchmarks/loadgen/bench-loadgen+# Eval harness (cli/test/evals): built binaries, oracle caches and run+# results. Results are never committed.+/cli/test/evals/.cache/+/cli/test/evals/results/+# Rust build artifacts (oxc symbolicator shim)**/oxc-shim/target/symbolicator-bench.txtdiff --git a/CLAUDE.md b/CLAUDE.mdindex dc955920..b7055cf6 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -34,6 +34,7 @@ Traceway is an error tracking and monitoring platform consisting of:| CLI | `cd cli && just test` | Runs unit tests || CLI | `cd cli && just check` | Lint + test + vulncheck + skill drift + contract tests (pre-commit gate; CI enforces it for `cli/`) || CLI | `cd cli && just smoke-test` | Live E2E (needs `TRACEWAY_SMOKE_*` env vars) |+| CLI | `cd cli && just eval-harness` | Eval harness tests plus every case's oracle trajectory, fake model only ($0) || Any | `nix develop` (or direnv via `.envrc`) | Dev shell with the Go and Node versions read from `backend/go.mod` and `.nvmrc`, plus `just`, `golangci-lint`, `govulncheck` (`flake.nix`; shells `default`, `backend`, `frontend`, `oxc`). Put `JWT_SECRET` in the gitignored `.envrc.local` || Any | `./scripts/check-node-pins.sh` | Asserts every Dockerfile's `ARG NODE_VERSION` default matches `.nvmrc`; run by `release-traceway.yml` before it builds |diff --git a/cli/README.md b/cli/README.mdindex 7654d7d7..3a4062a4 100644--- a/cli/README.md+++ b/cli/README.md@@ -232,6 +232,14 @@ just smoke-testIf any of those vars is missing, the smoke tests skip cleanly rather than fail.+## Skill and MCP evals++`test/evals` measures how well AI agents use Traceway through the skills,+against a seeded hermetic backend. It is **manual only** and free: nothing+triggers it, and it plays every case with a fake model. Run+`just eval-harness` (unit tests plus every oracle trajectory). See+[`test/evals/README.md`](test/evals/README.md).+## Contributing```bashdiff --git a/cli/justfile b/cli/justfileindex 3b2e656d..d2d92e39 100644--- a/cli/justfile+++ b/cli/justfile@@ -23,6 +23,13 @@ lint:vulncheck:govulncheck ./...+# Skill and MCP evals (test/evals). Manual only: none of these recipes is part+# of check, test or default. See test/evals/README.md.++# Free: harness tests plus every case's oracle trajectory, with the fake model.+eval-harness:+ cd test/evals && go test ./... && go run ./cmd/twevals oracle --all+# Regenerate the published skill (skills/traceway/) from the canonical# knowledge chunks in pkg/mcpserver/knowledge.gen-skills:diff --git a/cli/test/evals/README.md b/cli/test/evals/README.mdindex 7c4241f3..356b459c 100644--- a/cli/test/evals/README.md+++ b/cli/test/evals/README.md@@ -1,57 +1,259 @@# Traceway skill and MCP evals-**Manual only.** Nothing here runs automatically, and nothing here calls a-model yet: this directory holds eval cases and their fixtures. The harness-that seeds a backend, drives an agent and grades the result is added next,-in its own Go module under this directory.+**Manual only, and free.** Nothing here runs automatically: no workflow, no+`just check`/`test`/`default` recipe. This harness plays every case with a+fake model on loopback and makes no paid model calls; its Chat Completions+client refuses any non-loopback URL.-The cases started as the performance scenarios of PR #405-(`skills/traceway/evals/*.md`). They moved out of `skills/` because anything-in a skill directory ships to users and to the agent under test, which would-hand it the expected answers.+This module measures how well AI agents use Traceway through its skills. Every+run gets its own hermetic Traceway instance, the standalone backend in SQLite+mode, seeded through the real ingest endpoints. The agent works through real+`traceway` commands in a sandboxed shell, and deterministic graders score the+result.-## What changed in the port+| Suite | Measures | Arms |+|---|---|---|+| perf | Diagnosing latency against the bottleneck checklist in `performance.md` | `cli+skill`, `cli-noskill` |++`cli+skill` installs the `traceway` skill; `cli-noskill` is the same CLI with+no skill, the baseline that skill uplift is measured against.++## Running it++From `cli/`:++```+just eval-harness+```++runs `go test ./...` and then every case's oracle on every arm it lists, with+the fake model. From `cli/test/evals`, `go run ./cmd/twevals help` lists the+commands:++- `twevals list [--preset P] [--cases GLOB] [--arms A]` prints the case × arm+ combinations a selection picks.+- `twevals oracle [--all | <case-id>...] [--arms A] [-j N] [--keep]` plays+ the oracles and writes `results.json` and `summary.md`. It exits `1` when+ an oracle fails.++Presets (`presets.yaml`): `smoke` is the starred cases on the primary arm,+`core` every case on the primary arm, `full` every arm a case lists.++Requirements: Go (the `go`/`toolchain` lines of `go.mod`) and git. The+harness builds the backend and both CLI editions from this checkout into+`.cache/` (gitignored) and reuses them until the sources change.-Each Markdown scenario became a structured case with one correct answer, one-named distractor, and telemetry that the real SDKs and the real ingest path-can produce:+## Reading the results-| PR #405 file | Case | What was fixed |+A run is `pass`, `fail`, `budget_exhausted` (counted as a fail) or+`infra_error` (excluded from pass rates).++Each run directory, `runs/<case>/<arm>/<model>/r<N>/`, holds:++- `transcript.jsonl`: every model response, every tool call with its id, op,+ argv, exit code and output;+- `run.json` and `result.json`;+- `seed.json`: the ids the seed exposed. The agent never sees it;+- the workspace, when the run failed or `--keep` was set.++Graders:++| Grader | Gates | Checks ||---|---|---|-| `backend-go-select-n-plus-1` | `perf-db-n-plus-1` | Consistent counts: N item queries for N orders |-| `backend-missing-index` | `perf-db-missing-index` | No `EXPLAIN` handed to the agent; a 30-day window that matches the claim |-| `backend-node-sequential-calls` | `perf-net-sequential` | No unexplained 700ms per hop |-| `backend-python-pool-exhaustion` | `perf-db-pool-exhaustion` | Pool-wait gaps that add up to the request time |-| `backend-sync-background-task` | `perf-app-sync-background` | Fix advice that does not depend on the retired `CaptureTask` |-| `frontend-js-large-payloads` | `perf-net-large-payload` | A real `http.response.body.size` attribute instead of a fictional `transfer` span |-| `frontend-js-no-cache` | `perf-cache-no-cache` | A browser project is present, and only one checklist row is correct |-| `infra-cpu-saturation` | `perf-infra-cpu-saturation` | Saturation driven by traffic, not by a deploy |-| `infra-cache-stampede` | `perf-cache-stampede` | Realistic request rates |-| `mobile-ios-cold-start` | `perf-xs-ios-watchdog` | Built only from telemetry the iOS SDK sends: a launch watchdog crash whose cause is a slow backend endpoint |-| `mobile-android-gc-pauses` | `perf-xs-android-timeouts`, `debug-android-npe` | The Android SDK sends crashes, actions and logs, not frame or GC metrics |-| `mobile-flutter-shader-jank` | `debug-flutter-null-check` | The Flutter SDK sends errors, not frame timings |+| `answer` | yes | The structured answer: checklist row (and not the distractor), locus, onset |+| `trajectory` | yes | Required, alternative, forbidden and ordered operations, and thrash limits, on surface-neutral operations (`internal/ops`) |+| `grounding` | yes | Cited calls exist, succeeded and were not truncated. Their output contains the deciding signal. No id in the answer was invented |+| `report` | yes | Regexps the final report must or must not match, for cases that set `expect.report` |++## Adding a case++A case is three things:++- a seed function in `internal/scenarios/<suite>/`, registered by name;+- a YAML file `cases/<suite>/<id>.yaml`;+- fixtures under `testdata/`.++`internal/suite` tests the catalog as a whole: references exist, checklist ids+are valid, and the leakage lint is clean. Done means:++1. The seed builds data with exactly one correct answer and one named+ distractor (`distractor:` in the YAML).+2. `twevals oracle <id>` passes on every arm the case lists.+3. `go test ./internal/suite/` passes.++`perf-db-n-plus-1` is the exemplar.++### The seed DSL (`internal/seed`)++```go+func init() { scenarios.Register("perf/db-n-plus-1", seedNPlusOne) }++func seedNPlusOne(s *seed.Scenario) error {+ p := s.Project("storefront", seed.OTel) // a real project, created through the API+ api := p.Service("orders-api", seed.Go("nethttp", "sql")).Versions("2.14.3")+ orders := api.Endpoint("GET", "/api/orders")+ orders.Traffic(seed.Daily(160), seed.Diurnal(), seed.Over(29*24*time.Hour))+ orders.Handler(func(r *seed.Req) {+ n := int(r.Lerp(6, 72) * (0.35 + 0.95*r.Float())) // grows over the window+ r.DB("SELECT * FROM orders WHERE user_id = $1 ...", r.Ms(3, 6), seed.Rows(n))+ for range n {+ r.DB("SELECT id, sku, qty, unit_price FROM order_items WHERE order_id = $1", r.Ms(6, 14))+ }+ r.Compute("encode response", r.Ms(2, 6))+ })+ s.Background(api, seed.TypicalTraffic(10)) // healthy endpoints, a benign 400, host metrics+ api.Host().CPU(seed.Flat(0.22, 0.31))+ s.Expose("slowRequest", orders.Slowest(24*time.Hour)) // for the oracle and graders, never the agent+ return nil+}+```++- **Timing is consistent by construction.** Work is laid out on a cursor.+ `r.DB`, `r.HTTP`, `r.Cache`, `r.Compute` and `r.Call(otherEndpoint, rtt)`+ each add a child span. `r.Gap(d)` is waiting time with no span. `r.Span`+ nests, and `r.Parallel` runs branches side by side. A parent's duration is+ where its cursor stops.+- **Releases** are `Versions(initial, seed.Deploy{At, Version})`, which sets+ the `service.version` resource attribute. `r.Version()` lets the handler+ change behaviour after a deploy.+- **Multiple instances** are `Instances("node-a", ...)`, with `r.Host()` and+ `HostNamed(h).CPU(...)`.+- **Traffic shapes** are `Daily`, `Diurnal`, `GrowRate`, `Burst`, `RateFunc`+ and `Over`. Use `Statuses` or `r.Status` to fail requests.+- **Logs and errors:**+ - `r.Log(sev, body, attrs...)` writes a trace-correlated OTLP log.+ - `svc.Log(at, ...)` writes a log outside any request.+ - `r.Error(seed.Exc{Type, Message, Stack})` records an exception event in+ the stack format of the service's language.+ - Every detailed request also gets an access-log line,+ `GET /path 200 1243ms`, with its trace id. That is how an agent finds a+ slow request's id, because `endpoints list` has none.+- **Metrics** come from `Host().CPU/Memory/Disk/Filesystem` (OTel+ hostmetrics names, `system.cpu.utilization` split by `state`) and from+ `svc.Gauge/Counter/Histogram(name, unit, attrs, series)`. Histograms arrive+ as `<name>.avg` and `<name>.count`. Series helpers: `Flat`, `Step`, `Ramp`,+ `Periodic`, `Sawtooth`, `Plus`, `Scale`, `Clamp`.+- **Browser and mobile SDK data** goes through `p.Crash(seed.Crash{...})`, with+ network and navigation `Actions` and console `Logs`, and `p.Session(...)`.+ It travels the native `/api/report` protocol, and covers only what those+ SDKs really send: no frame timings, no mobile metrics. `p.GroupMatching`+ reads an exception group's hash back after ingest; put work that needs the+ ingested data in `s.AfterFlush`.+- **Volume.** Requests outside the detail sample are sent as their root span+ (and any downstream server spans), so percentiles and counts stay complete+ while a scenario stays under about 20k spans. Requests with logs, errors or+ an `Expose` always keep their full tree.++Backend facts that bite (each one cost a case author an afternoon):++- Health-check routes (`/health`, `/healthz`, `/ping`, `/ready`, ...) are+ dropped at ingest, so an endpoint seeded on them never appears and `Verify`+ fails with "endpoints never became visible".+- `endpoints list --order-by p95|lastSeen` does not sort: the CLI sends+ `p95`/`lastSeen`, the SQLite/DuckDB repositories only know+ `p95_duration`/`last_seen`, and the list falls back to impact order. Keep+ a case's endpoint count at ten or fewer so the slow one is on the first page+ whatever the order. This is a product bug, not a harness one; the evals+ should catch the skill's reliance on it.+- A log body search over more than 24 hours answers 422 unless another filter+ (service, severity, trace) narrows it.+- Rare events confined to a short window can come out empty at some hours of+ the day (the arrival process follows the wall-clock hour). Use `Burst`, or+ seed the case at every hour of the day into `seed.NewDrySink()` and check+ the event is there before relying on it.++- Numeric OTel attributes must be integers, which the DSL does for you.+- Projects with a browser framework never get endpoints from OTLP.+- An exception's message does not split groups: vary the type or the frames+ to get distinct issues.+- Session recordings are stamped with the ingest time, so keep crashes that+ carry actions within the last ~12 hours.+- The unscoped `metrics query --name system.cpu.utilization --aggregation max`+ returns the idle state. That is a real trap the skill should steer around.++### The case YAML++See `perf-db-n-plus-1.yaml` for the full shape. The keys:++- `arms` and `invoke`: `slash` preloads SKILL.md the way a `/traceway`+ command does, and `natural` makes the agent load the skill itself.+- `workspace.repo`: a fixture under `testdata/repos`. Fixtures can carry+ history: a `history.yaml` whose commits are overlays under `.history/`+ (only the files that differ from the fixture, plus an `absent:` list of+ files the commit lacks), committed with ages relative to the run so+ `git log` lines up with the seeded deploys.+- `prompt`, `budget`, `user`: scripted `ask_user` answers, first match wins.+- `expect.answer`, `expect.grounding`, `expect.calls`, `expect.report`.+- `judge.rubric` and `judge.expected`, the rubric for report quality. The+ harness does not score it.++Templates are resolved after seeding:++| Template | Value |+|---|---|+| `{{seed.<key>}}` | What the seed exposed |+| `{{env.url}}` | The instance URL |+| `{{env.project}}` | The primary project's id |+| `{{env.projects.<name>.id}}` | Another project's id |+| `{{env.stub}}` | The switchable stub's URL |+| `{{env.now}}` | The scenario's "now" |++A scalar that is exactly one template keeps its type.++The `oracle` is the reference trajectory the fake model plays:++- **One step is one tool call**, so steps are cited as `c01`, `c02`, ... in+ the submitted evidence.+- **`shell:` steps** run CLI commands. Use `jq` to keep cited outputs under+ the 30k-character cap: a truncated citation fails grounding.+- **`{{last.<json.path>}}` and `{{calls.c03.<path>}}`** read earlier tool+ results at play time.+- **`load_skill:` steps** play as a harmless `list_files .` on arms without+ skills (`cli-noskill`), so call ids stay the same on every arm.+- **The oracle ends** with `submit:`.++While developing a case, `TWEVALS_ONLY=<ids>` limits loading to those cases,+so an unfinished YAML file elsewhere cannot break your run.++### Leakage++The seeded strings (span names, attribute values, log bodies, stack traces,+object names) and the prompt must not name the answer. `internal/grade/leakage.go`+holds the denylist, which covers phrases like `n+1`, `stampede`,+`missing index`, `pool exhaust`, `gc pause` and `noisy neighbor`. Use real+library messages, such as SQLAlchemy's `QueuePool limit of size 5 overflow 10+reached` or Postgres's `deadlock detected`, and neutral names.## Layout```-cases/perf/<id>.yaml 11 performance cases-cases/debug/<id>.yaml 2 mobile debugging cases-testdata/repos/<repo>/ the fixture repository each case's agent works in-testdata/oracle/<case>/ files the reference trajectory writes (debug fixes)+models.yaml presets.yaml+cmd/twevals/ list | oracle+internal/env/ backend subprocess, registration, PATs, CLI logins, the stub+internal/seed/ the DSL, OTLP and /api/report writers, read-back verification+internal/scenarios/ one seed function per case+internal/agent/ the Chat Completions client (loopback only), the loop, prompt caching, transcripts, the fake model+internal/tools/ the sandboxed shell (mvdan.cc/sh, in-process coreutils, gojq, goawk), files, skills, ask_user, submit+internal/ops/ CLI → operation mapping, cross-checked against tool-map.md+internal/grade/ answer, trajectory, grounding, report, leakage+internal/runner/ one run end to end+internal/report/ results.json and summary.md+cases/ testdata/ cases, fixture repos, oracle files```-A case names its seed (the Go function that will generate its telemetry),-its arms (`cli+skill`, `cli-noskill`), the prompt, budgets, the expected-answer, the tool calls that must, may and must not happen, a judge rubric,-and an `oracle`: the reference trajectory, one tool call per step, ending in-the structured answer.--A fixture repo can carry git history: `history.yaml` lists commits, each an-overlay under `.history/` (only the files that differ from the fixture, plus-an `absent:` list of files the commit lacks), with ages relative to the run so-`git log` lines up with the seeded deploys.+The agent's shell is an interpreter, never the host shell. It has the+`traceway` binary (logged in with a read PAT whose config lives outside the+workspace), `jq`, `awk`, read-only text utilities confined to the workspace,+and read-only `git`. Tool subprocesses get a scrubbed environment (`HOME`,+`XDG_*`, `PATH`, `TZ`, `LANG`), so secrets in the harness's environment+never reach them.## CreditsThe first performance scenarios were contributed by Liyu in PR #405-("feat(skills): add performance skill evals").+("feat(skills): add performance skill evals"). They were ported to this+harness as `perf-db-n-plus-1`, `perf-db-missing-index`, `perf-net-sequential`,+`perf-db-pool-exhaustion`, `perf-app-sync-background`, `perf-net-large-payload`,+`perf-cache-no-cache`, `perf-infra-cpu-saturation`, `perf-cache-stampede`,+`perf-xs-ios-watchdog`, `perf-xs-android-timeouts`, `debug-android-npe` and+`debug-flutter-null-check`.diff --git a/cli/test/evals/cmd/twevals/commands.go b/cli/test/evals/cmd/twevals/commands.gonew file mode 100644index 00000000..a8897e35--- /dev/null+++ b/cli/test/evals/cmd/twevals/commands.go@@ -0,0 +1,190 @@+package main++import (+ "context"+ "errors"+ "flag"+ "fmt"+ "os"+ "path/filepath"+ "strings"+ "text/tabwriter"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/report"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"+ _ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios/all"+)++type workspace struct {+ root string // cli/test/evals+ repo string+ cases []*catalog.Case+ presets *catalog.Presets+ models *catalog.Models+}++func load() (*workspace, error) {+ root := os.Getenv("TWEVALS_ROOT")+ if root == "" {+ wd, _ := os.Getwd()+ for d := wd; ; d = filepath.Dir(d) {+ if _, err := os.Stat(filepath.Join(d, "models.yaml")); err == nil {+ if _, err := os.Stat(filepath.Join(d, "presets.yaml")); err == nil {+ root = d+ break+ }+ }+ if filepath.Dir(d) == d {+ return nil, errors.New("run twevals from cli/test/evals (or set TWEVALS_ROOT)")+ }+ }+ }+ repo, err := env.RepoRoot(root)+ if err != nil {+ return nil, err+ }+ cases, err := catalog.LoadCases(root)+ if err != nil {+ return nil, err+ }+ presets, err := catalog.LoadPresets(root)+ if err != nil {+ return nil, err+ }+ models, err := catalog.LoadModels(root)+ if err != nil {+ return nil, err+ }+ return &workspace{root: root, repo: repo, cases: cases, presets: presets, models: models}, nil+}++// parseInterleaved parses flags that may come after positional arguments.+func parseInterleaved(fs *flag.FlagSet, args []string) []string {+ var pos []string+ for {+ _ = fs.Parse(args)+ if fs.NArg() == 0 {+ return pos+ }+ pos = append(pos, fs.Arg(0))+ args = fs.Args()[1:]+ }+}++type selFlags struct{ sel catalog.Selection }++func (s *selFlags) bind(fs *flag.FlagSet) {+ fs.StringVar(&s.sel.Preset, "preset", "", "smoke | core | full")+ fs.StringVar(&s.sel.Cases, "cases", "", "comma-separated case ids or globs (overrides the preset)")+ fs.StringVar(&s.sel.Suites, "suites", "", "comma-separated suites")+ fs.StringVar(&s.sel.Tags, "tags", "", "comma-separated tags, all required")+ fs.StringVar(&s.sel.Arms, "arms", "", "comma-separated arms")+}++func (s *selFlags) selection() catalog.Selection {+ if s.sel.Preset == "" && s.sel.Cases == "" && s.sel.Suites == "" && s.sel.Tags == "" {+ s.sel.Preset = "smoke"+ }+ return s.sel+}++func cmdList(args []string) error {+ fs := flag.NewFlagSet("list", flag.ExitOnError)+ var sf selFlags+ sf.bind(fs)+ _ = fs.Parse(args)+ ws, err := load()+ if err != nil {+ return err+ }+ sel := sf.sel+ if sel.Preset == "" && sel.Cases == "" {+ sel.Preset = "full"+ }+ units, err := catalog.Select(ws.cases, ws.presets, sel)+ if err != nil {+ return err+ }+ tw := tabwriter.NewWriter(os.Stdout, 2, 4, 2, ' ', 0)+ fmt.Fprintln(tw, "CASE\tARM\tTAGS\tTITLE")+ for _, u := range units {+ fmt.Fprintf(tw, "%s\t%s\t%s\t%s\n", u.Case.ID, u.Arm, strings.Join(u.Case.Tags, ","), u.Case.Title)+ }+ tw.Flush()+ fmt.Printf("\n%d runs per model\n", len(units))+ return nil+}++func cmdOracle(ctx context.Context, args []string) error {+ fs := flag.NewFlagSet("oracle", flag.ExitOnError)+ all := fs.Bool("all", false, "every case")+ arms := fs.String("arms", "", "comma-separated arms (default: every arm the case lists)")+ jobs := fs.Int("j", 3, "parallel runs")+ keep := fs.Bool("keep", false, "keep run directories")+ suites := fs.String("suites", "", "comma-separated suites")+ out := fs.String("out", "", "results directory")+ ids := parseInterleaved(fs, args)+ ws, err := load()+ if err != nil {+ return err+ }+ sel := catalog.Selection{Preset: "full", Arms: *arms, Suites: *suites}+ if !*all {+ if len(ids) == 0 && *suites == "" {+ return errors.New("name case ids, or pass --all")+ }+ sel.Cases = strings.Join(ids, ",")+ }+ units, err := catalog.Select(ws.cases, ws.presets, sel)+ if err != nil {+ return err+ }+ bins, err := env.BuildBinaries(ws.repo, filepath.Join(ws.root, ".cache"))+ if err != nil {+ return err+ }+ outDir := *out+ if outDir == "" {+ outDir = filepath.Join(ws.root, ".cache", "oracle", time.Now().UTC().Format("20060102-150405"))+ }+ if err := os.MkdirAll(outDir, 0o755); err != nil {+ return err+ }+ oracleModel := &catalog.Model{Key: "oracle", ID: "fake/oracle", Caching: "none"}+ opt := runner.Options{EvalsRoot: ws.root, RepoRoot: ws.repo, OutDir: outDir, Bins: bins, Models: []*catalog.Model{oracleModel},+ Runs: 1, Jobs: *jobs, Keep: *keep, Log: os.Stderr}+ header := report.NewHeader(ws.repo, filepath.Base(outDir), ws.models, bins.Hash, os.Args)+ header.Oracle = true+ results := runner.Execute(ctx, opt, runner.Jobs(units, opt.Models, 1))+ header.Finished = time.Now().UTC()+ if err := report.Save(outDir, &report.Results{Header: header, Runs: results}); err != nil {+ return err+ }+ failed := 0+ for _, r := range results {+ if r.Status != "pass" {+ failed+++ fmt.Fprintf(os.Stderr, "FAIL %s: %s\n", r.Key(), oneLine(r))+ }+ }+ fmt.Fprintf(os.Stderr, "\noracle: %d/%d passed (%s)\n", len(results)-failed, len(results), outDir)+ if failed > 0 {+ return &exitError{1, ""}+ }+ return nil+}++func oneLine(r *runner.Result) string {+ if r.Error != "" {+ return r.Status + ": " + r.Error+ }+ for _, g := range r.Graders {+ if g.Gating && !g.Skipped && !g.Pass {+ return r.Status + ": " + g.Grader + ": " + strings.Join(g.Failures, "; ")+ }+ }+ return r.Status+}diff --git a/cli/test/evals/cmd/twevals/main.go b/cli/test/evals/cmd/twevals/main.gonew file mode 100644index 00000000..71e01b66--- /dev/null+++ b/cli/test/evals/cmd/twevals/main.go@@ -0,0 +1,61 @@+// Command twevals runs the Traceway skill and MCP evals with the fake model+// that plays each case's oracle. It makes no paid model calls.+package main++import (+ "context"+ "errors"+ "fmt"+ "os"+ "os/signal"+ "syscall"+)++const usage = `twevals: Traceway skill and MCP evals (oracle runs only, free)++ twevals list [--preset P] [--suites S] [--tags T] [--cases GLOB] [--arms A]+ twevals oracle [--all | <case-id>...] [--arms A] [-j N] [--keep]++Selection flags: --preset smoke|core|full, --cases (comma-separated ids or globs; overrides the preset), --suites, --tags, --arms.+Exit codes: 0 ok, 1 failed oracle.+`++func main() {+ ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)+ defer stop()+ if len(os.Args) < 2 {+ fmt.Fprint(os.Stderr, usage)+ os.Exit(2)+ }+ var err error+ args := os.Args[2:]+ switch os.Args[1] {+ case "list":+ err = cmdList(args)+ case "oracle":+ err = cmdOracle(ctx, args)+ case "-h", "--help", "help":+ fmt.Print(usage)+ return+ default:+ err = fmt.Errorf("unknown command %q", os.Args[1])+ }+ if err != nil {+ var ex *exitError+ if errors.As(err, &ex) {+ if ex.msg != "" {+ fmt.Fprintln(os.Stderr, "twevals:", ex.msg)+ }+ os.Exit(ex.code)+ }+ fmt.Fprintln(os.Stderr, "twevals:", err)+ os.Exit(1)+ }+}++type exitError struct {+ code int+ msg string+}++func (e *exitError) Error() string { return e.msg }diff --git a/cli/test/evals/go.mod b/cli/test/evals/go.modnew file mode 100644index 00000000..27dd8c25--- /dev/null+++ b/cli/test/evals/go.mod@@ -0,0 +1,30 @@+module github.com/tracewayapp/traceway/cli/test/evals++go 1.26.2++toolchain go1.26.9++require (+ github.com/benhoyt/goawk v1.32.0+ github.com/google/uuid v1.6.0+ github.com/itchyny/gojq v0.12.19+ github.com/tracewayapp/traceway/cli v1.9.15+ go.opentelemetry.io/proto/otlp v1.10.0+ google.golang.org/protobuf v1.36.11+ gopkg.in/yaml.v3 v3.0.1+ mvdan.cc/sh/v3 v3.14.1+)++require (+ github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect+ github.com/itchyny/timefmt-go v0.1.8 // indirect+ golang.org/x/net v0.50.0 // indirect+ golang.org/x/sys v0.47.0 // indirect+ golang.org/x/term v0.45.0 // indirect+ golang.org/x/text v0.34.0 // indirect+ google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 // indirect+ google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57 // indirect+ google.golang.org/grpc v1.79.2 // indirect+)++replace github.com/tracewayapp/traceway/cli => ../..diff --git a/cli/test/evals/go.sum b/cli/test/evals/go.sumnew file mode 100644index 00000000..628b8809--- /dev/null+++ b/cli/test/evals/go.sum@@ -0,0 +1,69 @@+github.com/benhoyt/goawk v1.32.0 h1:CK/mnBxGsNqX8hdghPIctl2aGIcwfnhT6sv+9M/bY7w=+github.com/benhoyt/goawk v1.32.0/go.mod h1:jXTQxBxtQ0VsjFqc8dw7tIJj3SDzQN8kcdMq7r83/ZA=+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=+github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=+github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=+github.com/go-quicktest/qt v1.102.0 h1:HSQxCeh5YZH3EL3W39ixjtyaEhcWSXQHtHnMBzSs474=+github.com/go-quicktest/qt v1.102.0/go.mod h1:p4lGIVX+8Wa6ZPNDvqcxq36XpUDLh42FLetFU7odllI=+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=+github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs=+github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0/go.mod h1:JfhWUomR1baixubs02l85lZYYOm7LV6om4ceouMv45c=+github.com/itchyny/gojq v0.12.19 h1:ttXA0XCLEMoaLOz5lSeFOZ6u6Q3QxmG46vfgI4O0DEs=+github.com/itchyny/gojq v0.12.19/go.mod h1:5galtVPDywX8SPSOrqjGxkBeDhSxEW1gSxoy7tn1iZY=+github.com/itchyny/timefmt-go v0.1.8 h1:1YEo1JvfXeAHKdjelbYr/uCuhkybaHCeTkH8Bo791OI=+github.com/itchyny/timefmt-go v0.1.8/go.mod h1:5E46Q+zj7vbTgWY8o5YkMeYb4I6GeWLFnetPy5oBrAI=+github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=+github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=+github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=+github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=+go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=+go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=+go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48=+go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8=+go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0=+go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs=+go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18=+go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE=+go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8=+go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew=+go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI=+go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA=+go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g=+go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk=+golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=+golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=+golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=+golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=+golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=+golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=+golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=+golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=+gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=+gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=+google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 h1:JLQynH/LBHfCTSbDWl+py8C+Rg/k1OVH3xfcaiANuF0=+google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57/go.mod h1:kSJwQxqmFXeo79zOmbrALdflXQeAYcUbgS7PbpMknCY=+google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57 h1:mWPCjDEyshlQYzBpMNHaEof6UX1PmHcaUODUywQ0uac=+google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ=+google.golang.org/grpc v1.79.2 h1:fRMD94s2tITpyJGtBBn7MkMseNpOZU8ZxgC3MMBaXRU=+google.golang.org/grpc v1.79.2/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=+google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=+google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=+mvdan.cc/sh/v3 v3.14.1 h1:bXkhQWNHCs0KZEChF8hYS6FC+T2N9mUZLbQv9blditI=+mvdan.cc/sh/v3 v3.14.1/go.mod h1:syYCoFET8w9tvevxiXUtY8/ICrU+l26jHmhJDra3Vwo=diff --git a/cli/test/evals/internal/agent/agent_test.go b/cli/test/evals/internal/agent/agent_test.gonew file mode 100644index 00000000..6e0ff134--- /dev/null+++ b/cli/test/evals/internal/agent/agent_test.go@@ -0,0 +1,200 @@+package agent++import (+ "context"+ "encoding/json"+ "errors"+ "net/http"+ "net/http/httptest"+ "strings"+ "sync"+ "testing"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"+)++// scripted is a Chatter that returns canned responses and records requests.+type scripted struct {+ mu sync.Mutex+ resps []*Response+ reqs []*Request+}++func (s *scripted) Chat(_ context.Context, req *Request) (*Response, error) {+ s.mu.Lock()+ defer s.mu.Unlock()+ s.reqs = append(s.reqs, req)+ if len(s.resps) == 0 {+ return nil, errors.New("script exhausted")+ }+ r := s.resps[0]+ s.resps = s.resps[1:]+ return r, nil+}++func toolResp(name, args string, cost float64) *Response {+ r := &Response{Provider: "test"}+ r.Choices = append(r.Choices, struct {+ Message Message `json:"message"`+ FinishReason string `json:"finish_reason"`+ }{Message: Message{Role: "assistant", ToolCalls: []ToolCall{{ID: "t1", Type: "function", Function: FunctionCall{Name: name, Arguments: args}}}}})+ r.Usage.Cost = cost+ r.Usage.PromptTokens = 1000+ return r+}++func textResp(text string) *Response {+ r := &Response{}+ r.Choices = append(r.Choices, struct {+ Message Message `json:"message"`+ FinishReason string `json:"finish_reason"`+ }{Message: Message{Role: "assistant", Content: text}})+ return r+}++type echoTool struct{}++func (echoTool) Spec() tools.Spec {+ return tools.Spec{Name: "echo", Parameters: map[string]any{"type": "object"}}+}+func (echoTool) Call(context.Context, json.RawMessage) tools.Result {+ return tools.Result{Output: "echoed", OK: true}+}++func cfg(s *scripted) Config {+ return Config{+ Model: &catalog.Model{Key: "m", ID: "x/y", Caching: "anthropic"}, Chat: s, System: "sys", Prompt: "do it", SubmitTool: tools.Finish,+ Tools: []tools.Tool{echoTool{}, &tools.Submit{Kind: tools.Finish, MaxRepairs: 2}},+ Budget: catalog.Budget{MaxTurns: 5, MaxToolCalls: 3},+ }+}++func TestLoopSubmits(t *testing.T) {+ s := &scripted{resps: []*Response{toolResp("echo", `{}`, 0.01), toolResp("finish", `{"summary":"done","changes":[]}`, 0.01)}}+ run := Run(context.Background(), cfg(s))+ if run.StopReason != StopSubmitted || run.Submission["summary"] != "done" || len(run.Calls) != 2 || run.Calls[0].ID != "c01" {+ t.Fatalf("run = %+v", run)+ }+ if run.Usage.CostUSD != 0.02 {+ t.Fatalf("cost %v", run.Usage.CostUSD)+ }+ // The second request carries the assistant tool call before its result,+ // and the result starts with the call id the model cites.+ msgs := s.reqs[1].Messages+ if msgs[2].Role != "assistant" || len(msgs[2].ToolCalls) != 1 || msgs[3].Role != "tool" {+ t.Fatalf("history order: %+v", msgs)+ }+ parts, ok := msgs[3].Content.([]Part)+ if !ok || !strings.HasPrefix(parts[0].Text, "[call c01]") || parts[0].CacheControl == nil {+ t.Fatalf("the newest tool result must carry the call id and a cache breakpoint: %#v", msgs[3].Content)+ }+ if sys, ok := msgs[0].Content.([]Part); !ok || sys[0].CacheControl == nil {+ t.Fatal("the system prompt must carry a cache breakpoint for anthropic models")+ }+ if _, ok := msgs[1].Content.(string); !ok {+ t.Fatal("only the system prompt and the newest message get breakpoints")+ }+}++func TestLoopNudgesOnceThenStops(t *testing.T) {+ s := &scripted{resps: []*Response{textResp("I think it is fine."), textResp("Really.")}}+ run := Run(context.Background(), cfg(s))+ if run.StopReason != StopNoSubmit || len(s.reqs) != 2 {+ t.Fatalf("stop %s after %d requests", run.StopReason, len(s.reqs))+ }+ last := s.reqs[1].Messages+ if !strings.Contains(textOf(last[len(last)-1]), "call finish") {+ t.Fatal("the nudge must name the submit tool")+ }+}++func textOf(m Message) string {+ if s, ok := m.Content.(string); ok {+ return s+ }+ if ps, ok := m.Content.([]Part); ok && len(ps) > 0 {+ return ps[0].Text+ }+ return ""+}++func TestLoopBudgets(t *testing.T) {+ var resps []*Response+ for i := 0; i < 10; i++ {+ resps = append(resps, toolResp("echo", `{}`, 0))+ }+ run := Run(context.Background(), cfg(&scripted{resps: resps}))+ if run.StopReason != StopBudget {+ t.Fatalf("stop %s", run.StopReason)+ }+ rejected := 0+ for _, c := range run.Calls {+ if c.Rejected {+ rejected+++ }+ }+ if rejected == 0 {+ t.Fatal("calls past max_tool_calls must be refused")+ }+}++func TestUnknownToolIsAnError(t *testing.T) {+ s := &scripted{resps: []*Response{toolResp("nope", `{}`, 0), toolResp("finish", `{"summary":"x","changes":[]}`, 0)}}+ run := Run(context.Background(), cfg(s))+ if run.Calls[0].OK || !strings.Contains(run.Calls[0].Output, "no tool named") {+ t.Fatalf("%+v", run.Calls[0])+ }+}++func TestClientRefusesRemote(t *testing.T) {+ c := &Client{BaseURL: "https://openrouter.ai/api/v1", Key: "sk-or-test"}+ if _, err := c.Chat(context.Background(), &Request{Model: "x"}); !errors.Is(err, ErrNotAuthorized) {+ t.Fatalf("a remote call under go test = %v", err)+ }+}++func TestClientWireFormat(t *testing.T) {+ var got map[string]any+ var hdr http.Header+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ hdr = r.Header+ _ = json.NewDecoder(r.Body).Decode(&got)+ w.Write([]byte(`{"id":"g1","provider":"Anthropic","choices":[{"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":10,"completion_tokens":2,"cost":0.001,"prompt_tokens_details":{"cached_tokens":4}}}`))+ }))+ defer srv.Close()+ c := &Client{BaseURL: srv.URL, Key: "k"}+ allow := true+ resp, err := c.Chat(context.Background(), &Request{Model: "anthropic/claude-opus-5.5", Messages: applyCaching("anthropic", []Message{{Role: "system", Content: "s"}, {Role: "user", Content: "u"}}),+ Provider: &ProviderPrefs{Order: []string{"anthropic"}, AllowFallbacks: &allow, RequireParameters: true}, Usage: map[string]any{"include": true}})+ if err != nil {+ t.Fatal(err)+ }+ if resp.Provider != "Anthropic" || resp.Usage.PromptTokensDetails.CachedTokens != 4 {+ t.Fatalf("%+v", resp)+ }+ if hdr.Get("Authorization") != "Bearer k" || hdr.Get("X-Title") != "traceway-evals" || hdr.Get("HTTP-Referer") == "" {+ t.Fatalf("headers %v", hdr)+ }+ prov := got["provider"].(map[string]any)+ if prov["require_parameters"] != true || prov["order"].([]any)[0] != "anthropic" {+ t.Fatalf("provider %v", prov)+ }+ sys := got["messages"].([]any)[0].(map[string]any)["content"].([]any)[0].(map[string]any)+ if sys["cache_control"].(map[string]any)["type"] != "ephemeral" {+ t.Fatalf("system %v", sys)+ }+}++func TestPromptHelpers(t *testing.T) {+ if got := StripSlash("/traceway perf GET /x\nslow"); got != "perf GET /x\nslow" {+ t.Fatal(got)+ }+ if SlashSkill("/traceway debug x") != "traceway" || SlashSkill("why is it slow") != "" {+ t.Fatal("SlashSkill")+ }+ p := SystemPrompt(PromptInput{SubmitTool: "finish", Skills: []tools.Skill{{Name: "traceway", Description: "d"}}})+ if !strings.Contains(p, "- traceway: d") || !strings.Contains(p, "call finish") {+ t.Fatal(p)+ }+}diff --git a/cli/test/evals/internal/agent/caching.go b/cli/test/evals/internal/agent/caching.gonew file mode 100644index 00000000..f997293a--- /dev/null+++ b/cli/test/evals/internal/agent/caching.go@@ -0,0 +1,39 @@+package agent++// Prompt caching. Agent loops resend the whole history every turn, so caching+// is most of the cost saving.+//+// - Anthropic models (caching: anthropic) get explicit cache_control+// breakpoints: one on the system message (the generic prompt plus any+// preloaded skill; tools sit before it in the cached prefix) and one on+// the most recent message, so each turn reads all earlier history from+// cache. Anthropic checks earlier block boundaries for hits on its own, so+// a single moving breakpoint is enough.+// - OpenAI models (caching: automatic) cache any stable prefix. The system+// prompt therefore carries no timestamps and tools keep a fixed order.++// applyCaching returns the messages to send for a caching mode, without+// mutating the conversation itself.+func applyCaching(mode string, msgs []Message) []Message {+ if mode != "anthropic" || len(msgs) == 0 {+ return msgs+ }+ out := make([]Message, len(msgs))+ copy(out, msgs)+ mark := func(i int) {+ m := out[i]+ text, ok := m.Content.(string)+ if !ok || text == "" {+ return+ }+ m.Content = []Part{{Type: "text", Text: text, CacheControl: &CacheControl{Type: "ephemeral"}}}+ out[i] = m+ }+ if out[0].Role == "system" {+ mark(0)+ }+ if last := len(out) - 1; last > 0 && (out[last].Role == "tool" || out[last].Role == "user") {+ mark(last)+ }+ return out+}diff --git a/cli/test/evals/internal/agent/fake.go b/cli/test/evals/internal/agent/fake.gonew file mode 100644index 00000000..85556549--- /dev/null+++ b/cli/test/evals/internal/agent/fake.go@@ -0,0 +1,313 @@+package agent++import (+ "encoding/json"+ "fmt"+ "net/http"+ "net/http/httptest"+ "os"+ "regexp"+ "sort"+ "strconv"+ "strings"+ "sync"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+)++// Fake is an httptest server speaking Chat Completions that plays a case's+// oracle steps, one tool call per turn. It is the only model `go test` can+// reach.+type Fake struct {+ Steps []catalog.Step+ // FileRoot resolves write_file `from:` paths.+ FileRoot string++ srv *httptest.Server+ mu sync.Mutex+ errs []string+}++// NewFake starts the fake model.+func NewFake(steps []catalog.Step, fileRoot string) *Fake {+ f := &Fake{Steps: steps, FileRoot: fileRoot}+ f.srv = httptest.NewServer(http.HandlerFunc(f.serve))+ return f+}++// Client returns a Chat Completions client pointed at the fake.+func (f *Fake) Client() *Client { return &Client{BaseURL: f.srv.URL, Retries: 1} }++// Close stops the server.+func (f *Fake) Close() { f.srv.Close() }++// Errors returns oracle rendering problems (a step naming a tool the arm+// does not have, a template that resolved to nothing, ...).+func (f *Fake) Errors() []string {+ f.mu.Lock()+ defer f.mu.Unlock()+ return append([]string{}, f.errs...)+}++func (f *Fake) fail(format string, a ...any) {+ f.mu.Lock()+ defer f.mu.Unlock()+ f.errs = append(f.errs, fmt.Sprintf(format, a...))+}++func (f *Fake) serve(w http.ResponseWriter, r *http.Request) {+ var req Request+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil {+ http.Error(w, err.Error(), 400)+ return+ }+ step := 0+ for _, m := range req.Messages {+ if m.Role == "assistant" {+ step+++ }+ }+ tools := map[string]bool{}+ for _, t := range req.Tools {+ tools[t.Function.Name] = true+ }+ msg := Message{Role: "assistant"}+ if step >= len(f.Steps) {+ msg.Content = "Done."+ } else {+ name, args, text, err := f.render(f.Steps[step], tools, req.Messages)+ if err != nil {+ f.fail("step %d: %v", step+1, err)+ msg.Content = "oracle error: " + err.Error()+ } else if name == "" {+ msg.Content = text+ } else {+ raw, _ := json.Marshal(args)+ msg.ToolCalls = []ToolCall{{ID: fmt.Sprintf("fake_%d", step+1), Type: "function", Function: FunctionCall{Name: name, Arguments: string(raw)}}}+ }+ }+ resp := map[string]any{+ "id": fmt.Sprintf("fake-%d", step+1), "model": req.Model, "provider": "fake",+ "choices": []any{map[string]any{"message": msg, "finish_reason": "tool_calls"}},+ "usage": map[string]any{"prompt_tokens": 0, "completion_tokens": 0, "cost": 0},+ }+ w.Header().Set("Content-Type", "application/json")+ _ = json.NewEncoder(w).Encode(resp)+}++func (f *Fake) render(s catalog.Step, tools map[string]bool, msgs []Message) (string, map[string]any, string, error) {+ tmpl := func(v string) (string, error) { return resolveRuntime(v, msgs) }+ switch {+ case s.Say != "":+ return "", nil, s.Say, nil+ case s.Shell != "":+ cmd, err := tmpl(s.Shell)+ if err != nil {+ return "", nil, "", err+ }+ return "shell", map[string]any{"command": cmd}, "", nil+ case s.ReadFile != "":+ return need(tools, "read_file", map[string]any{"path": s.ReadFile})+ case s.ListFiles != "":+ return need(tools, "list_files", map[string]any{"path": s.ListFiles})+ case s.Search != nil:+ return need(tools, "search", s.Search)+ case s.WriteFile != nil:+ content := s.WriteFile.Content+ if s.WriteFile.From != "" {+ raw, err := os.ReadFile(f.FileRoot + "/" + s.WriteFile.From)+ if err != nil {+ return "", nil, "", err+ }+ content = string(raw)+ }+ return need(tools, "write_file", map[string]any{"path": s.WriteFile.Path, "content": content})+ case s.EditFile != nil:+ return need(tools, "edit_file", map[string]any{"path": s.EditFile.Path, "old_string": s.EditFile.Old, "new_string": s.EditFile.New})+ case s.LoadSkill != "":+ if !tools["load_skill"] {+ // Arms without skills play the step as a harmless look at the+ // repository, so later call ids stay the same on every arm.+ return need(tools, "list_files", map[string]any{"path": "."})+ }+ return need(tools, "load_skill", map[string]any{"name": s.LoadSkill})+ case s.AskUser != "":+ q, err := tmpl(s.AskUser)+ if err != nil {+ return "", nil, "", err+ }+ return need(tools, "ask_user", map[string]any{"question": q})+ case s.Submit != nil:+ for name := range tools {+ if strings.HasPrefix(name, "submit_") {+ args, err := resolveArgs(s.Submit, msgs)+ return name, args, "", err+ }+ }+ return "", nil, "", fmt.Errorf("no submit tool on this arm")+ }+ return "", nil, "", fmt.Errorf("empty oracle step")+}++func need(tools map[string]bool, name string, args map[string]any) (string, map[string]any, string, error) {+ if !tools[name] {+ return "", nil, "", fmt.Errorf("this arm has no %s tool", name)+ }+ return name, args, "", nil+}++var runtimeRe = regexp.MustCompile(`\{\{\s*((?:last|calls\.c\d+)(?:\.[^}\s]+)?)\s*\}\}`)++// resolveRuntime fills {{last.x}} and {{calls.c03.x}} from the tool results+// already in the conversation.+func resolveRuntime(s string, msgs []Message) (string, error) {+ var firstErr error+ out := runtimeRe.ReplaceAllStringFunc(s, func(m string) string {+ expr := runtimeRe.FindStringSubmatch(m)[1]+ v, err := lookupRuntime(expr, msgs)+ if err != nil && firstErr == nil {+ firstErr = err+ }+ return fmt.Sprint(v)+ })+ return out, firstErr+}++func resolveArgs(in map[string]any, msgs []Message) (map[string]any, error) {+ out := map[string]any{}+ var firstErr error+ var walk func(v any) any+ walk = func(v any) any {+ switch t := v.(type) {+ case string:+ // A whole-string template keeps the looked-up value's type.+ if m := runtimeRe.FindStringSubmatch(t); m != nil && m[0] == strings.TrimSpace(t) {+ val, err := lookupRuntime(m[1], msgs)+ if err != nil && firstErr == nil {+ firstErr = err+ }+ return val+ }+ s, err := resolveRuntime(t, msgs)+ if err != nil && firstErr == nil {+ firstErr = err+ }+ return s+ case map[string]any:+ m := map[string]any{}+ for k, x := range t {+ m[k] = walk(x)+ }+ return m+ case []any:+ l := make([]any, len(t))+ for i, x := range t {+ l[i] = walk(x)+ }+ return l+ default:+ return v+ }+ }+ keys := make([]string, 0, len(in))+ for k := range in {+ keys = append(keys, k)+ }+ sort.Strings(keys)+ for _, k := range keys {+ out[k] = walk(in[k])+ }+ return out, firstErr+}++func lookupRuntime(expr string, msgs []Message) (any, error) {+ parts := strings.SplitN(expr, ".", 3)+ var content string+ var path string+ switch parts[0] {+ case "last":+ for i := len(msgs) - 1; i >= 0; i-- {+ if msgs[i].Role == "tool" {+ content, _ = msgs[i].Content.(string)+ break+ }+ }+ if len(parts) > 1 {+ path = strings.Join(parts[1:], ".")+ }+ case "calls":+ if len(parts) < 2 {+ return nil, fmt.Errorf("bad template %s", expr)+ }+ prefix := "[call " + parts[1] + "]"+ for _, m := range msgs {+ if s, ok := m.Content.(string); ok && m.Role == "tool" && strings.HasPrefix(s, prefix) {+ content = s+ }+ }+ if len(parts) > 2 {+ path = parts[2]+ }+ }+ if content == "" {+ return nil, fmt.Errorf("template %s: no such tool result", expr)+ }+ body := stripToolFrame(content)+ if path == "" {+ return body, nil+ }+ var v any+ if err := json.Unmarshal([]byte(firstJSON(body)), &v); err != nil {+ return nil, fmt.Errorf("template %s: result is not JSON", expr)+ }+ for _, seg := range splitPath(path) {+ switch t := v.(type) {+ case map[string]any:+ v = t[seg]+ case []any:+ i, err := strconv.Atoi(seg)+ if err != nil || i < 0 || i >= len(t) {+ return nil, fmt.Errorf("template %s: bad index %s", expr, seg)+ }+ v = t[i]+ default:+ return nil, fmt.Errorf("template %s: cannot index %T with %s", expr, v, seg)+ }+ }+ if v == nil {+ return nil, fmt.Errorf("template %s resolved to nothing", expr)+ }+ return v, nil+}++func splitPath(p string) []string {+ p = strings.NewReplacer("[", ".", "]", "").Replace(p)+ var out []string+ for _, s := range strings.Split(p, ".") {+ if s != "" {+ out = append(out, s)+ }+ }+ return out+}++func stripToolFrame(s string) string {+ if i := strings.Index(s, "\n"); i >= 0 && strings.HasPrefix(s, "[call ") {+ s = s[i+1:]+ }+ if i := strings.LastIndex(s, "\n[budget:"); i >= 0 {+ s = s[:i]+ }+ return s+}++func firstJSON(s string) string {+ s = strings.TrimSpace(s)+ if i := strings.IndexAny(s, "{["); i > 0 {+ s = s[i:]+ }+ if i := strings.Index(s, "\n[exit status"); i >= 0 {+ s = s[:i]+ }+ return s+}diff --git a/cli/test/evals/internal/agent/loop.go b/cli/test/evals/internal/agent/loop.gonew file mode 100644index 00000000..9cff574c--- /dev/null+++ b/cli/test/evals/internal/agent/loop.go@@ -0,0 +1,213 @@+package agent++import (+ "context"+ "encoding/json"+ "errors"+ "fmt"+ "sort"+ "strings"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"+)++// Config is one agent run.+type Config struct {+ Model *catalog.Model+ Chat Chatter+ Tools []tools.Tool+ System string+ Prompt string+ SubmitTool string+ Budget catalog.Budget+ Transcript *Transcript+ // AfterCall runs harness hooks after each executed call.+ AfterCall func(ctx context.Context, c *record.Call)+ // Questions returns the ask_user exchanges at the end.+ Questions func() []record.Question+ // SkillLoaded returns the skills loaded during the run.+ SkillLoaded func() []string+}++// Stop reasons.+const (+ StopSubmitted = "submitted"+ StopBudget = "budget_exhausted"+ StopNoSubmit = "no_submit"+ StopInfra = "infra_error"+)++// Run drives the model until it submits, a budget runs out, or it stops+// without submitting (after one nudge).+func Run(ctx context.Context, cfg Config) *record.Run {+ budget := cfg.Budget.Defaults()+ ctx, cancel := context.WithTimeout(ctx, budget.Timeout.D())+ defer cancel()++ run := &record.Run{Prompt: cfg.Prompt, Started: time.Now().UTC()}+ defer func() { run.Elapsed = time.Since(run.Started) }()++ byName := map[string]tools.Tool{}+ defs := make([]ToolDef, 0, len(cfg.Tools))+ for _, t := range cfg.Tools {+ s := t.Spec()+ byName[s.Name] = t+ defs = append(defs, ToolDef{Type: "function", Function: FunctionDef{Name: s.Name, Description: s.Description, Parameters: s.Parameters}})+ }+ sort.SliceStable(defs, func(i, j int) bool { return defs[i].Function.Name < defs[j].Function.Name })++ msgs := []Message{{Role: "system", Content: cfg.System}, {Role: "user", Content: cfg.Prompt}}+ cfg.Transcript.Event("start", map[string]any{"model": cfg.Model.ID, "system": cfg.System, "prompt": cfg.Prompt, "tools": defs})++ nudged := false+ callN := 0+ outputTokens := 0+ providers := map[string]bool{}+ finish := func(reason, detail string) *record.Run {+ run.StopReason, run.StopDetail = reason, detail+ if cfg.Questions != nil {+ run.Questions = cfg.Questions()+ }+ if cfg.SkillLoaded != nil {+ run.SkillLoaded = cfg.SkillLoaded()+ }+ for p := range providers {+ run.Providers = append(run.Providers, p)+ }+ sort.Strings(run.Providers)+ cfg.Transcript.Event("stop", map[string]any{"reason": reason, "detail": detail, "usage": run.Usage, "turns": run.Turns})+ return run+ }++ for {+ if run.Turns >= budget.MaxTurns {+ return finish(StopBudget, fmt.Sprintf("max_turns %d reached", budget.MaxTurns))+ }+ run.Turns+++ req := &Request{+ Model: cfg.Model.ID,+ Messages: applyCaching(cfg.Model.Caching, msgs),+ Tools: defs,+ ToolChoice: "auto",+ MaxTokens: cfg.Model.MaxTokensPerTurn,+ Usage: map[string]any{"include": true},+ }+ if cfg.Model.Temperature != nil {+ req.Temperature = cfg.Model.Temperature+ }+ if len(cfg.Model.Provider.Order) > 0 || cfg.Model.Provider.AllowFallbacks != nil {+ req.Provider = &ProviderPrefs{Order: cfg.Model.Provider.Order, AllowFallbacks: cfg.Model.Provider.AllowFallbacks, RequireParameters: true}+ }+ if cfg.Model.ReasoningEffort != "" {+ req.Reasoning = map[string]any{"effort": cfg.Model.ReasoningEffort}+ }+ resp, err := cfg.Chat.Chat(ctx, req)+ if err != nil {+ if errors.Is(err, context.DeadlineExceeded) {+ return finish(StopBudget, "wall-clock timeout")+ }+ return finish(StopInfra, err.Error())+ }+ u := usageOf(resp)+ run.Usage.Add(u)+ outputTokens += u.OutputTokens - u.ReasoningTokens+ if resp.Provider != "" {+ providers[resp.Provider] = true+ }+ msg := resp.Choices[0].Message+ msg.Role = "assistant"+ cfg.Transcript.Event("response", map[string]any{"turn": run.Turns, "id": resp.ID, "provider": resp.Provider, "model": resp.Model,+ "finish_reason": resp.Choices[0].FinishReason, "content": msg.Content, "tool_calls": msg.ToolCalls, "usage": u})+ if text, ok := msg.Content.(string); ok && strings.TrimSpace(text) != "" {+ run.FinalText = text+ }+ // The assistant message, tool calls included, must precede the+ // tool results or Anthropic models reject the history.+ msgs = append(msgs, msg)++ if len(msg.ToolCalls) == 0 {+ if nudged {+ return finish(StopNoSubmit, "stopped without calling "+cfg.SubmitTool)+ }+ nudged = true+ msgs = append(msgs, Message{Role: "user", Content: fmt.Sprintf("You have not finished: call %s with your final answer now.", cfg.SubmitTool)})+ continue+ }+ if outputTokens > budget.MaxOutputTokens {+ return finish(StopBudget, fmt.Sprintf("max_output_tokens %d exceeded", budget.MaxOutputTokens))+ }++ final := false+ for _, tc := range msg.ToolCalls {+ callN+++ id := fmt.Sprintf("c%02d", callN)+ c := &record.Call{ID: id, Turn: run.Turns, Tool: tc.Function.Name, Args: json.RawMessage(nonEmpty(tc.Function.Arguments))}+ var res tools.Result+ switch {+ case final:+ res = tools.Result{Output: "not executed: the task was already submitted"}+ c.Rejected = true+ case callN > budget.MaxToolCalls && !isSubmit(tc.Function.Name):+ res = tools.Result{Output: fmt.Sprintf("not executed: the tool-call budget (%d) is used up. Call %s now.", budget.MaxToolCalls, cfg.SubmitTool)}+ c.Rejected = true+ default:+ t, ok := byName[tc.Function.Name]+ if !ok {+ res = tools.Result{Output: fmt.Sprintf("error: there is no tool named %q", tc.Function.Name)}+ } else {+ started := time.Now()+ res = t.Call(ctx, c.Args)+ c.DurationMs = time.Since(started).Milliseconds()+ }+ }+ c.Output, c.OK, c.Truncated, c.Ops = res.Output, res.OK, res.Truncated, res.Ops+ run.Calls = append(run.Calls, c)+ cfg.Transcript.Event("tool", map[string]any{"call": c})+ if !c.Rejected && cfg.AfterCall != nil {+ cfg.AfterCall(ctx, c)+ }+ if res.Final {+ final = true+ run.Submission = res.Submission+ run.SubmitTool = tc.Function.Name+ run.SubmitCall = id+ }+ content := fmt.Sprintf("[call %s]\n%s", id, res.Output)+ if !res.Final {+ content += fmt.Sprintf("\n[budget: %d tool calls and %d turns left]", max(0, budget.MaxToolCalls-callN), max(0, budget.MaxTurns-run.Turns))+ }+ msgs = append(msgs, Message{Role: "tool", ToolCallID: tc.ID, Content: content})+ }+ if final {+ return finish(StopSubmitted, "")+ }+ if ctx.Err() != nil {+ return finish(StopBudget, "wall-clock timeout")+ }+ }+}++func isSubmit(name string) bool {+ return name == tools.SubmitPerf || name == tools.SubmitDebug || name == tools.Finish+}++func nonEmpty(s string) string {+ if strings.TrimSpace(s) == "" {+ return "{}"+ }+ return s+}++func usageOf(r *Response) record.Usage {+ return record.Usage{+ InputTokens: r.Usage.PromptTokens,+ CachedTokens: r.Usage.PromptTokensDetails.CachedTokens,+ CacheWriteTokens: r.Usage.PromptTokensDetails.CacheWriteTokens,+ OutputTokens: r.Usage.CompletionTokens,+ ReasoningTokens: r.Usage.CompletionTokensDetails.ReasoningTokens,+ CostUSD: r.Usage.Cost,+ }+}diff --git a/cli/test/evals/internal/agent/openrouter.go b/cli/test/evals/internal/agent/openrouter.gonew file mode 100644index 00000000..042d375e--- /dev/null+++ b/cli/test/evals/internal/agent/openrouter.go@@ -0,0 +1,247 @@+// Package agent drives a model through the eval: a Chat Completions client,+// the tool-calling loop with budgets, prompt caching, transcripts, and the+// fake model that plays oracle trajectories.+package agent++import (+ "bytes"+ "context"+ "encoding/json"+ "errors"+ "fmt"+ "io"+ "net"+ "net/http"+ "net/url"+ "strings"+ "time"+)++// Message is one Chat Completions message.+type Message struct {+ Role string `json:"role"`+ Content any `json:"content,omitempty"` // string or []Part+ ToolCalls []ToolCall `json:"tool_calls,omitempty"`+ ToolCallID string `json:"tool_call_id,omitempty"`+ // ReasoningDetails must be sent back unchanged on the assistant message+ // so reasoning models keep their chain across tool calls.+ ReasoningDetails json.RawMessage `json:"reasoning_details,omitempty"`+}++// Part is one content part.+type Part struct {+ Type string `json:"type"`+ Text string `json:"text"`+ CacheControl *CacheControl `json:"cache_control,omitempty"`+}++// CacheControl marks an Anthropic prompt-cache breakpoint.+type CacheControl struct {+ Type string `json:"type"`+}++// ToolCall is an assistant tool call.+type ToolCall struct {+ ID string `json:"id"`+ Type string `json:"type"`+ Function FunctionCall `json:"function"`+}++// FunctionCall is the called function and its JSON arguments.+type FunctionCall struct {+ Name string `json:"name"`+ Arguments string `json:"arguments"`+}++// ToolDef is a function tool definition.+type ToolDef struct {+ Type string `json:"type"`+ Function FunctionDef `json:"function"`+}++// FunctionDef describes a function.+type FunctionDef struct {+ Name string `json:"name"`+ Description string `json:"description,omitempty"`+ Parameters map[string]any `json:"parameters"`+}++// ProviderPrefs is OpenRouter's provider routing.+type ProviderPrefs struct {+ Order []string `json:"order,omitempty"`+ AllowFallbacks *bool `json:"allow_fallbacks,omitempty"`+ RequireParameters bool `json:"require_parameters,omitempty"`+}++// Request is a Chat Completions request.+type Request struct {+ Model string `json:"model"`+ Messages []Message `json:"messages"`+ Tools []ToolDef `json:"tools,omitempty"`+ ToolChoice any `json:"tool_choice,omitempty"`+ MaxTokens int `json:"max_tokens,omitempty"`+ Temperature *float64 `json:"temperature,omitempty"`+ Provider *ProviderPrefs `json:"provider,omitempty"`+ Reasoning map[string]any `json:"reasoning,omitempty"`+ ResponseFormat map[string]any `json:"response_format,omitempty"`+ Usage map[string]any `json:"usage,omitempty"`+ ParallelTools *bool `json:"parallel_tool_calls,omitempty"`+}++// Response is a Chat Completions response.+type Response struct {+ ID string `json:"id"`+ Model string `json:"model"`+ Provider string `json:"provider"`+ Choices []struct {+ Message Message `json:"message"`+ FinishReason string `json:"finish_reason"`+ } `json:"choices"`+ Usage struct {+ PromptTokens int `json:"prompt_tokens"`+ CompletionTokens int `json:"completion_tokens"`+ Cost float64 `json:"cost"`+ PromptTokensDetails struct {+ CachedTokens int `json:"cached_tokens"`+ CacheWriteTokens int `json:"cache_write_tokens"`+ } `json:"prompt_tokens_details"`+ CompletionTokensDetails struct {+ ReasoningTokens int `json:"reasoning_tokens"`+ } `json:"completion_tokens_details"`+ } `json:"usage"`+ Error *struct {+ Message string `json:"message"`+ Code any `json:"code"`+ } `json:"error,omitempty"`+}++// Chatter sends one Chat Completions request.+type Chatter interface {+ Chat(ctx context.Context, req *Request) (*Response, error)+}++// Client talks Chat Completions to the fake model on loopback. A+// non-loopback base URL is always refused: this harness makes no paid calls.+type Client struct {+ BaseURL string+ Key string+ HTTP *http.Client+ // Retries on 429/5xx/timeouts (default 5).+ Retries int+}++// ErrNotAuthorized is returned for a request to a remote endpoint.+var ErrNotAuthorized = errors.New("remote model calls are not supported: the harness only talks to the fake model on loopback")++func isLoopback(base string) bool {+ u, err := url.Parse(base)+ if err != nil {+ return false+ }+ host := u.Hostname()+ if host == "localhost" {+ return true+ }+ ip := net.ParseIP(host)+ return ip != nil && ip.IsLoopback()+}++// RetryableError marks an error worth retrying.+type RetryableError struct{ Err error }++func (e *RetryableError) Error() string { return e.Err.Error() }+func (e *RetryableError) Unwrap() error { return e.Err }++// Chat sends a request with retries and exponential backoff.+func (c *Client) Chat(ctx context.Context, req *Request) (*Response, error) {+ if !isLoopback(c.BaseURL) {+ return nil, ErrNotAuthorized+ }+ retries := c.Retries+ if retries == 0 {+ retries = 5+ }+ var last error+ for attempt := 0; attempt <= retries; attempt++ {+ if attempt > 0 {+ d := time.Duration(1<<uint(attempt-1)) * 2 * time.Second+ select {+ case <-ctx.Done():+ return nil, ctx.Err()+ case <-time.After(d):+ }+ }+ resp, err := c.once(ctx, req)+ if err == nil {+ return resp, nil+ }+ last = err+ var re *RetryableError+ if !errors.As(err, &re) {+ return nil, err+ }+ }+ return nil, fmt.Errorf("after %d retries: %w", retries, last)+}++func (c *Client) once(ctx context.Context, req *Request) (*Response, error) {+ body, err := json.Marshal(req)+ if err != nil {+ return nil, err+ }+ hreq, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimSuffix(c.BaseURL, "/")+"/chat/completions", bytes.NewReader(body))+ if err != nil {+ return nil, err+ }+ hreq.Header.Set("Content-Type", "application/json")+ if c.Key != "" {+ hreq.Header.Set("Authorization", "Bearer "+c.Key)+ }+ hreq.Header.Set("HTTP-Referer", "https://github.com/tracewayapp/traceway")+ hreq.Header.Set("X-Title", "traceway-evals")+ hc := c.HTTP+ if hc == nil {+ hc = &http.Client{Timeout: 5 * time.Minute}+ }+ resp, err := hc.Do(hreq)+ if err != nil {+ if ctx.Err() != nil {+ return nil, ctx.Err()+ }+ return nil, &RetryableError{err}+ }+ defer resp.Body.Close()+ raw, err := io.ReadAll(resp.Body)+ if err != nil {+ return nil, &RetryableError{err}+ }+ if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 {+ return nil, &RetryableError{fmt.Errorf("openrouter HTTP %d: %s", resp.StatusCode, clip(raw))}+ }+ if resp.StatusCode != http.StatusOK {+ return nil, fmt.Errorf("openrouter HTTP %d: %s", resp.StatusCode, clip(raw))+ }+ var out Response+ if err := json.Unmarshal(raw, &out); err != nil {+ return nil, &RetryableError{fmt.Errorf("decode response: %w (%s)", err, clip(raw))}+ }+ if out.Error != nil {+ code := fmt.Sprint(out.Error.Code)+ if code == "429" || strings.HasPrefix(code, "5") {+ return nil, &RetryableError{fmt.Errorf("openrouter error %s: %s", code, out.Error.Message)}+ }+ return nil, fmt.Errorf("openrouter error %s: %s", code, out.Error.Message)+ }+ if len(out.Choices) == 0 {+ return nil, &RetryableError{fmt.Errorf("openrouter returned no choices: %s", clip(raw))}+ }+ return &out, nil+}++func clip(b []byte) string {+ s := strings.TrimSpace(string(b))+ if len(s) > 500 {+ return s[:500] + "..."+ }+ return s+}diff --git a/cli/test/evals/internal/agent/prompt.go b/cli/test/evals/internal/agent/prompt.gonew file mode 100644index 00000000..9618c320--- /dev/null+++ b/cli/test/evals/internal/agent/prompt.go@@ -0,0 +1,76 @@+package agent++import (+ "fmt"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"+)++// PromptInput is what an arm contributes to the system prompt. Everything+// else is identical across models and arms.+type PromptInput struct {+ SubmitTool string+ // Skills lists installed skills (name: description), as agent harnesses+ // advertise them.+ Skills []tools.Skill+ // Preloaded is the SKILL.md body of a slash-invoked skill.+ Preloaded *tools.Skill+ // Environment describes the machine (the CLI installed and logged in).+ Environment string+}++// SystemPrompt builds the neutral coding-agent system prompt. It carries no+// timestamps so OpenAI's automatic prefix cache keeps hitting.+func SystemPrompt(in PromptInput) string {+ var b strings.Builder+ b.WriteString(`You are a software engineering agent working for a developer. You work in their repository, which is the current directory, and you act only through the tools provided. You cannot see their screen.++Work like a careful senior engineer: gather evidence before you conclude, quantify what you claim, and do not change anything the user did not ask you to change. When you need a decision only the user can make, ask with ask_user.++Every tool result starts with a line like [call c07]. When you cite evidence, cite those ids.+`)+ fmt.Fprintf(&b, "\nWhen you are done, call %s exactly once with your final answer. Do not stop without calling it.\n", in.SubmitTool)+ if in.Environment != "" {+ b.WriteString("\n# Environment\n\n" + in.Environment + "\n")+ }+ if len(in.Skills) > 0 {+ b.WriteString("\n# Skills\n\nThese skills are installed. When a task matches a skill's description, load it with load_skill before you start and follow it. A skill's companion files are under .skills/<name>/ and can be read with read_file.\n\n")+ for _, s := range in.Skills {+ fmt.Fprintf(&b, "- %s: %s\n", s.Name, s.Description)+ }+ }+ if in.Preloaded != nil {+ fmt.Fprintf(&b, "\n# The user invoked /%s\n\nThe %s skill is loaded; its instructions follow. Its companion files are under .skills/%s/.\n\n", in.Preloaded.Name, in.Preloaded.Name, in.Preloaded.Name)+ b.WriteString(in.Preloaded.Body)+ b.WriteString("\n")+ }+ return b.String()+}++// StripSlash turns "/traceway perf X ..." into "perf X ..." for arms that do+// not have the invoked skill, so every arm gets the same request.+func StripSlash(prompt string) string {+ p := strings.TrimLeft(prompt, " \n")+ if !strings.HasPrefix(p, "/") {+ return prompt+ }+ sp := strings.IndexAny(p, " \n")+ if sp < 0 {+ return ""+ }+ return strings.TrimLeft(p[sp:], " ")+}++// SlashSkill returns the skill name a prompt invokes ("/traceway ..."), or "".+func SlashSkill(prompt string) string {+ p := strings.TrimLeft(prompt, " \n")+ if !strings.HasPrefix(p, "/") {+ return ""+ }+ end := strings.IndexAny(p, " \n")+ if end < 0 {+ end = len(p)+ }+ return p[1:end]+}diff --git a/cli/test/evals/internal/agent/transcript.go b/cli/test/evals/internal/agent/transcript.gonew file mode 100644index 00000000..a2093f99--- /dev/null+++ b/cli/test/evals/internal/agent/transcript.go@@ -0,0 +1,51 @@+package agent++import (+ "encoding/json"+ "os"+ "sync"+ "time"+)++// Transcript appends one JSON event per line: the opening messages, every+// model response (with usage, provider and generation id), every tool call+// (id, op, argv/args, exit, ok, truncated, output) and the stop reason.+type Transcript struct {+ mu sync.Mutex+ f *os.File+}++// OpenTranscript creates path.+func OpenTranscript(path string) (*Transcript, error) {+ f, err := os.Create(path)+ if err != nil {+ return nil, err+ }+ return &Transcript{f: f}, nil+}++// Event writes one event.+func (t *Transcript) Event(kind string, fields map[string]any) {+ if t == nil {+ return+ }+ t.mu.Lock()+ defer t.mu.Unlock()+ fields["type"] = kind+ fields["at"] = time.Now().UTC().Format(time.RFC3339Nano)+ raw, err := json.Marshal(fields)+ if err != nil {+ return+ }+ t.f.Write(append(raw, '\n'))+}++// Close flushes and closes the file.+func (t *Transcript) Close() error {+ if t == nil {+ return nil+ }+ t.mu.Lock()+ defer t.mu.Unlock()+ return t.f.Close()+}diff --git a/cli/test/evals/internal/catalog/case.go b/cli/test/evals/internal/catalog/case.gonew file mode 100644index 00000000..d0ee8b24--- /dev/null+++ b/cli/test/evals/internal/catalog/case.go@@ -0,0 +1,337 @@+// Package catalog loads the eval cases (cases/<suite>/<id>.yaml), the presets+// that select them, and the model configuration. It owns the case schema; every+// other package reads cases through these types.+package catalog++import (+ "fmt"+ "os"+ "path/filepath"+ "regexp"+ "slices"+ "sort"+ "strings"+ "time"++ "gopkg.in/yaml.v3"+)++// Suites, in report order.+var Suites = []string{"perf"}++// Arms. An arm fixes the surface the agent works through and what it is given.+const (+ ArmCLISkill = "cli+skill" // traceway CLI in the shell, traceway skill available+ ArmCLINoSkill = "cli-noskill" // traceway CLI in the shell, no skills (baseline)+)++// AllArms lists every arm the runner knows.+var AllArms = []string{ArmCLISkill, ArmCLINoSkill}++// Case is one eval scenario. Strings in Prompt, Expect, Oracle and Hooks may+// contain {{seed.x}} / {{env.x}} placeholders, resolved after seeding.+type Case struct {+ ID string `yaml:"id"`+ Suite string `yaml:"suite"`+ Title string `yaml:"title"`+ Tags []string `yaml:"tags"`+ Seed string `yaml:"seed"`+ Arms []string `yaml:"arms"`+ Invoke string `yaml:"invoke"` // slash | natural+ Distractor string `yaml:"distractor"`++ Workspace Workspace `yaml:"workspace"`+ Prompt string `yaml:"prompt"`+ Budget Budget `yaml:"budget"`+ User []UserRule `yaml:"user"`+ Expect Expect `yaml:"expect"`+ Judge JudgeSpec `yaml:"judge"`+ Oracle []Step `yaml:"oracle"`+ OracleByArm map[string][]Step `yaml:"oracle_by_arm"`++ // Path is the YAML file the case was loaded from.+ Path string `yaml:"-"`+}++// Workspace names the fixture repo copied into the agent's work dir.+type Workspace struct {+ Repo string `yaml:"repo"`+}++// Budget bounds one run. Zero values take the defaults in Defaults().+type Budget struct {+ MaxTurns int `yaml:"max_turns"`+ MaxToolCalls int `yaml:"max_tool_calls"`+ MaxOutputTokens int `yaml:"max_output_tokens"`+ Timeout Duration `yaml:"timeout"`+}++// Defaults fills unset budget fields.+func (b Budget) Defaults() Budget {+ if b.MaxTurns == 0 {+ b.MaxTurns = 40+ }+ if b.MaxToolCalls == 0 {+ b.MaxToolCalls = 30+ }+ if b.MaxOutputTokens == 0 {+ b.MaxOutputTokens = 12000+ }+ if b.Timeout == 0 {+ b.Timeout = Duration(10 * time.Minute)+ }+ return b+}++// UserRule scripts one ask_user answer. The first rule whose Match regexp+// matches the question wins; a rule with Default set matches anything.+type UserRule struct {+ Match string `yaml:"match"`+ Reply string `yaml:"reply"`+ Default string `yaml:"default"`+ // Once makes the rule apply to its first match only.+ Once bool `yaml:"once"`+}++// Expect is everything the deterministic graders check.+type Expect struct {+ Answer *Answer `yaml:"answer"`+ Grounding []string `yaml:"grounding"`+ Calls *Trajectory `yaml:"calls"`+ // Report holds regexps the final report text must (or must not) match.+ Report *ReportSpec `yaml:"report"`+}++// ReportSpec checks the final free-text report.+type ReportSpec struct {+ Contains []string `yaml:"contains"`+ NotContains []string `yaml:"not_contains"`+}++// Answer is the expected structured answer to submit_perf_diagnosis.+type Answer struct {+ ChecklistItem string `yaml:"checklist_item"`+ AnyOf []string `yaml:"any_of"`+ Not []string `yaml:"not"`+ Locus *Locus `yaml:"locus"`+ Onset *Onset `yaml:"onset"`+ // MentionsAll lists regexps that must all match somewhere in the answer.+ MentionsAll []string `yaml:"mentions_all"`+}++// Locus pins where the bottleneck is. Every set field must match.+type Locus struct {+ Endpoint string `yaml:"endpoint"`+ SpanContains string `yaml:"span_contains"`+ Service string `yaml:"service"`+ Host string `yaml:"host"`+}++// Onset is the expected start of a regression.+type Onset struct {+ At string `yaml:"at"`+ ToleranceMinutes int `yaml:"tolerance_minutes"`+}++// Trajectory is the tool-call DSL (plan §6.4).+type Trajectory struct {+ Required []CallMatcher `yaml:"required"`+ AnyOf [][]CallMatcher `yaml:"any_of"`+ Forbidden []CallMatcher `yaml:"forbidden"`+ Order [][]string `yaml:"order"`+ MaxCalls int `yaml:"max_calls"`+ MaxFailedCalls int `yaml:"max_failed_calls"`+ MaxIdenticalRepeats int `yaml:"max_identical_repeats"`+}++// CallMatcher matches one recorded operation.+type CallMatcher struct {+ Op string `yaml:"op"`+ Args map[string]any `yaml:"args"`+ Write *bool `yaml:"write"`+ OK *bool `yaml:"ok"`+}++// JudgeSpec is the LLM judge's rubric. Expected is shown to the judge only.+type JudgeSpec struct {+ Rubric []string `yaml:"rubric"`+ Expected string `yaml:"expected"`+}++// Step is one oracle action, played by the fake model.+type Step struct {+ Shell string `yaml:"shell"`+ Stdin string `yaml:"stdin"`+ ReadFile string `yaml:"read_file"`+ ListFiles string `yaml:"list_files"`+ Search map[string]any `yaml:"search"`+ WriteFile *FileStep `yaml:"write_file"`+ EditFile *EditStep `yaml:"edit_file"`+ LoadSkill string `yaml:"load_skill"`+ AskUser string `yaml:"ask_user"`+ Submit map[string]any `yaml:"submit"`+ Say string `yaml:"say"`+}++// FileStep writes a whole file.+type FileStep struct {+ Path string `yaml:"path"`+ Content string `yaml:"content"`+ // From copies the content of a testdata file instead.+ From string `yaml:"from"`+}++// EditStep replaces one string in a file.+type EditStep struct {+ Path string `yaml:"path"`+ Old string `yaml:"old"`+ New string `yaml:"new"`+}++// Duration is a time.Duration that unmarshals from "10m".+type Duration time.Duration++// UnmarshalYAML parses Go duration strings.+func (d *Duration) UnmarshalYAML(n *yaml.Node) error {+ var s string+ if err := n.Decode(&s); err != nil {+ return err+ }+ v, err := time.ParseDuration(s)+ if err != nil {+ return fmt.Errorf("duration %q: %w", s, err)+ }+ *d = Duration(v)+ return nil+}++// D returns the time.Duration.+func (d Duration) D() time.Duration { return time.Duration(d) }++var idRe = regexp.MustCompile(`^[a-z0-9]+(-[a-z0-9]+)+$`)++// Validate checks a case for structural errors that YAML decoding cannot.+func (c *Case) Validate() error {+ var errs []string+ add := func(f string, a ...any) { errs = append(errs, fmt.Sprintf(f, a...)) }+ if !idRe.MatchString(c.ID) {+ add("id %q must be kebab-case", c.ID)+ }+ if !slices.Contains(Suites, c.Suite) {+ add("suite %q must be one of %v", c.Suite, Suites)+ } else if !strings.HasPrefix(c.ID, c.Suite+"-") {+ add("id %q must start with %q", c.ID, c.Suite+"-")+ }+ if c.Path != "" && filepath.Base(c.Path) != c.ID+".yaml" {+ add("file %s must be named %s.yaml", c.Path, c.ID)+ }+ if c.Seed == "" {+ add("seed is required")+ }+ if len(c.Arms) == 0 {+ add("arms is required")+ }+ for _, a := range c.Arms {+ if !slices.Contains(AllArms, a) {+ add("unknown arm %q", a)+ }+ }+ switch c.Invoke {+ case "", "slash", "natural":+ default:+ add("invoke must be slash or natural, got %q", c.Invoke)+ }+ if strings.TrimSpace(c.Prompt) == "" {+ add("prompt is required")+ }+ if len(c.Oracle) == 0 {+ add("oracle is required")+ }+ for arm := range c.OracleByArm {+ if !slices.Contains(c.Arms, arm) {+ add("oracle_by_arm names arm %q that the case does not run", arm)+ }+ }+ if c.Expect.Answer == nil {+ add("%s cases need expect.answer", c.Suite)+ }+ if c.Distractor == "" && !slices.Contains(c.Tags, "control") {+ add("distractor is required (or tag the case control)")+ }+ if len(errs) > 0 {+ return fmt.Errorf("%s: %s", c.ID, strings.Join(errs, "; "))+ }+ return nil+}++// HasTag reports whether the case carries tag.+func (c *Case) HasTag(tag string) bool { return slices.Contains(c.Tags, tag) }++// OracleFor returns the oracle steps for an arm.+func (c *Case) OracleFor(arm string) []Step {+ if s, ok := c.OracleByArm[arm]; ok {+ return s+ }+ return c.Oracle+}++// LoadCases reads every cases/<suite>/*.yaml under root. TWEVALS_ONLY (a+// comma-separated list of ids) limits loading to those cases, so work in+// progress on other cases cannot break a run.+func LoadCases(root string) ([]*Case, error) {+ var out []*Case+ seen := map[string]string{}+ only := map[string]bool{}+ for _, id := range strings.Split(os.Getenv("TWEVALS_ONLY"), ",") {+ if id = strings.TrimSpace(id); id != "" {+ only[id] = true+ }+ }+ for _, suite := range Suites {+ files, err := filepath.Glob(filepath.Join(root, "cases", suite, "*.yaml"))+ if err != nil {+ return nil, err+ }+ sort.Strings(files)+ for _, f := range files {+ if len(only) > 0 && !only[strings.TrimSuffix(filepath.Base(f), ".yaml")] {+ continue+ }+ c, err := LoadCase(f)+ if err != nil {+ return nil, err+ }+ if c.Suite != suite {+ return nil, fmt.Errorf("%s: suite %q does not match directory %q", f, c.Suite, suite)+ }+ if prev, dup := seen[c.ID]; dup {+ return nil, fmt.Errorf("duplicate case id %s in %s and %s", c.ID, prev, f)+ }+ seen[c.ID] = f+ out = append(out, c)+ }+ }+ return out, nil+}++// LoadCase reads and validates one case file. Unknown keys are errors.+func LoadCase(path string) (*Case, error) {+ raw, err := os.ReadFile(path)+ if err != nil {+ return nil, err+ }+ dec := yaml.NewDecoder(strings.NewReader(string(raw)))+ dec.KnownFields(true)+ var c Case+ if err := dec.Decode(&c); err != nil {+ return nil, fmt.Errorf("%s: %w", path, err)+ }+ c.Path = path+ if c.Invoke == "" {+ c.Invoke = "natural"+ }+ if err := c.Validate(); err != nil {+ return nil, err+ }+ return &c, nil+}diff --git a/cli/test/evals/internal/catalog/models.go b/cli/test/evals/internal/catalog/models.gonew file mode 100644index 00000000..a4f825a1--- /dev/null+++ b/cli/test/evals/internal/catalog/models.go@@ -0,0 +1,159 @@+package catalog++import (+ "crypto/sha256"+ "encoding/hex"+ "fmt"+ "os"+ "path/filepath"+ "slices"+ "sort"+ "strings"++ "gopkg.in/yaml.v3"+)++// Model is one OpenRouter model as configured in models.yaml.+type Model struct {+ // Key is the short name used on the command line (opus, astra, judge).+ Key string `yaml:"-"`+ // ID is the pinned OpenRouter model id. Never a floating alias.+ ID string `yaml:"id"`+ Label string `yaml:"label"`+ Provider Provider `yaml:"provider"`+ // Caching is "anthropic" (explicit cache_control breakpoints) or+ // "automatic" (provider-side prefix caching, keep the prefix stable).+ Caching string `yaml:"caching"`+ // ReasoningEffort is sent as reasoning.effort when the model lists+ // "reasoning" in its supported_parameters. Empty disables it.+ ReasoningEffort string `yaml:"reasoning_effort"`+ // MaxTokensPerTurn caps one completion.+ MaxTokensPerTurn int `yaml:"max_tokens_per_turn"`+ Temperature *float64 `yaml:"temperature"`+ // Estimate is the per-run token envelope used by the preflight until+ // baseline.json carries measured averages.+ Estimate Estimate `yaml:"estimate"`+ // Fallback prices (USD per million tokens) used when the live price list+ // cannot be fetched. The preflight says when it fell back.+ FallbackPrice Price `yaml:"fallback_price"`+}++// Provider is OpenRouter's provider routing block.+type Provider struct {+ Order []string `yaml:"order" json:"order,omitempty"`+ AllowFallbacks *bool `yaml:"allow_fallbacks" json:"allow_fallbacks,omitempty"`+}++// Estimate is a per-run token envelope.+type Estimate struct {+ InputTokens [2]int `yaml:"input_tokens"`+ CachedFraction float64 `yaml:"cached_fraction"`+ OutputTokens [2]int `yaml:"output_tokens"`+}++// Price is USD per million tokens.+type Price struct {+ Prompt float64 `yaml:"prompt" json:"prompt"`+ Completion float64 `yaml:"completion" json:"completion"`+ CacheRead float64 `yaml:"cache_read" json:"cacheRead"`+ CacheWrite float64 `yaml:"cache_write" json:"cacheWrite"`+}++// Models is models.yaml.+type Models struct {+ Agents map[string]*Model `yaml:"agents"`+ Judge *Model `yaml:"judge"`+ // Digest is the SHA-256 of the file, recorded in every results header.+ Digest string `yaml:"-"`+}++// LoadModels reads models.yaml under root.+func LoadModels(root string) (*Models, error) {+ path := filepath.Join(root, "models.yaml")+ raw, err := os.ReadFile(path)+ if err != nil {+ return nil, err+ }+ var m Models+ dec := yaml.NewDecoder(strings.NewReader(string(raw)))+ dec.KnownFields(true)+ if err := dec.Decode(&m); err != nil {+ return nil, fmt.Errorf("%s: %w", path, err)+ }+ sum := sha256.Sum256(raw)+ m.Digest = hex.EncodeToString(sum[:])+ for k, a := range m.Agents {+ a.Key = k+ }+ if m.Judge != nil {+ m.Judge.Key = "judge"+ }+ return &m, m.validate()+}++func (m *Models) validate() error {+ all := []*Model{}+ for _, a := range m.Agents {+ all = append(all, a)+ }+ if m.Judge == nil {+ return fmt.Errorf("models.yaml: judge is required")+ }+ all = append(all, m.Judge)+ for _, a := range all {+ if a.ID == "" || !strings.Contains(a.ID, "/") {+ return fmt.Errorf("models.yaml: %s: id must be a provider/model OpenRouter id", a.Key)+ }+ if strings.HasPrefix(a.ID, "~") || strings.HasSuffix(a.ID, "latest") || strings.Contains(a.ID, ":") {+ return fmt.Errorf("models.yaml: %s: %q is a floating alias or variant; pin an exact id", a.Key, a.ID)+ }+ switch a.Caching {+ case "anthropic", "automatic", "none":+ default:+ return fmt.Errorf("models.yaml: %s: caching must be anthropic, automatic or none", a.Key)+ }+ }+ return nil+}++// Agent returns the agent model by key.+func (m *Models) Agent(key string) (*Model, error) {+ a, ok := m.Agents[key]+ if !ok {+ return nil, fmt.Errorf("unknown model %q (models.yaml has %s)", key, strings.Join(m.AgentKeys(), ", "))+ }+ return a, nil+}++// AgentKeys returns the configured agent keys, sorted.+func (m *Models) AgentKeys() []string {+ keys := make([]string, 0, len(m.Agents))+ for k := range m.Agents {+ keys = append(keys, k)+ }+ sort.Strings(keys)+ return keys+}++// ParseModelList splits "opus,astra" and checks every key exists.+func (m *Models) ParseModelList(s string) ([]*Model, error) {+ var out []*Model+ for _, k := range strings.Split(s, ",") {+ k = strings.TrimSpace(k)+ if k == "" {+ continue+ }+ a, err := m.Agent(k)+ if err != nil {+ return nil, err+ }+ if slices.ContainsFunc(out, func(x *Model) bool { return x.Key == k }) {+ continue+ }+ out = append(out, a)+ }+ if len(out) == 0 {+ return nil, fmt.Errorf("no models selected")+ }+ return out, nil+}diff --git a/cli/test/evals/internal/catalog/select.go b/cli/test/evals/internal/catalog/select.gonew file mode 100644index 00000000..aa414f83--- /dev/null+++ b/cli/test/evals/internal/catalog/select.go@@ -0,0 +1,178 @@+package catalog++import (+ "fmt"+ "os"+ "path"+ "path/filepath"+ "slices"+ "strings"++ "gopkg.in/yaml.v3"+)++// Preset selects cases and arms (presets.yaml).+type Preset struct {+ Description string `yaml:"description"`+ Tags []string `yaml:"tags"`+ // Arms is "primary" (each suite's primary arm) or "all" (every arm a+ // case lists).+ Arms string `yaml:"arms"`+}++// Presets is presets.yaml.+type Presets struct {+ Presets map[string]Preset `yaml:"presets"`+ PrimaryArms map[string][]string `yaml:"primary_arms"`+}++// LoadPresets reads presets.yaml under root.+func LoadPresets(root string) (*Presets, error) {+ raw, err := os.ReadFile(filepath.Join(root, "presets.yaml"))+ if err != nil {+ return nil, err+ }+ var p Presets+ dec := yaml.NewDecoder(strings.NewReader(string(raw)))+ dec.KnownFields(true)+ if err := dec.Decode(&p); err != nil {+ return nil, fmt.Errorf("presets.yaml: %w", err)+ }+ for name, pr := range p.Presets {+ if pr.Arms != "primary" && pr.Arms != "all" {+ return nil, fmt.Errorf("presets.yaml: %s: arms must be primary or all", name)+ }+ }+ for _, s := range Suites {+ if len(p.PrimaryArms[s]) == 0 {+ return nil, fmt.Errorf("presets.yaml: primary_arms.%s is missing", s)+ }+ }+ return &p, nil+}++// Selection is the user's case/arm filter. Cases (when set) overrides Preset+// for which cases run; Arms (when set) overrides the preset's arm policy.+type Selection struct {+ Preset string+ Cases string // comma-separated ids or globs+ Suites string // comma-separated+ Tags string // comma-separated, all must match+ Arms string // comma-separated+ Shard string // one suite+}++// Unit is one (case, arm) pair; the runner multiplies it by models and runs.+type Unit struct {+ Case *Case+ Arm string+}++func splitList(s string) []string {+ var out []string+ for _, p := range strings.Split(s, ",") {+ if p = strings.TrimSpace(p); p != "" {+ out = append(out, p)+ }+ }+ return out+}++// Select applies a selection to the loaded cases.+func Select(cases []*Case, presets *Presets, sel Selection) ([]Unit, error) {+ var preset *Preset+ if sel.Preset != "" {+ p, ok := presets.Presets[sel.Preset]+ if !ok {+ return nil, fmt.Errorf("unknown preset %q", sel.Preset)+ }+ preset = &p+ }+ globs := splitList(sel.Cases)+ suites := splitList(sel.Suites)+ tags := splitList(sel.Tags)+ arms := splitList(sel.Arms)+ for _, a := range arms {+ if !slices.Contains(AllArms, a) {+ return nil, fmt.Errorf("unknown arm %q", a)+ }+ }+ for _, s := range suites {+ if !slices.Contains(Suites, s) {+ return nil, fmt.Errorf("unknown suite %q", s)+ }+ }+ if sel.Shard != "" && !slices.Contains(Suites, sel.Shard) {+ return nil, fmt.Errorf("unknown shard %q (shards are suites)", sel.Shard)+ }++ var out []Unit+ matchedGlob := map[string]bool{}+ for _, c := range cases {+ if len(globs) > 0 {+ hit := false+ for _, g := range globs {+ if ok, _ := path.Match(g, c.ID); ok {+ hit, matchedGlob[g] = true, true+ }+ }+ if !hit {+ continue+ }+ } else if preset != nil {+ if !hasAll(c, preset.Tags) {+ continue+ }+ }+ if len(suites) > 0 && !slices.Contains(suites, c.Suite) {+ continue+ }+ if sel.Shard != "" && c.Suite != sel.Shard {+ continue+ }+ if !hasAll(c, tags) {+ continue+ }+ var caseArms []string+ switch {+ case len(arms) > 0:+ for _, a := range arms {+ if slices.Contains(c.Arms, a) {+ caseArms = append(caseArms, a)+ }+ }+ case preset != nil && preset.Arms == "all":+ caseArms = c.Arms+ default:+ for _, a := range presets.PrimaryArms[c.Suite] {+ if slices.Contains(c.Arms, a) {+ caseArms = append(caseArms, a)+ }+ }+ if len(caseArms) == 0 {+ caseArms = c.Arms[:1]+ }+ }+ for _, a := range caseArms {+ out = append(out, Unit{Case: c, Arm: a})+ }+ }+ for _, g := range globs {+ if !matchedGlob[g] {+ return nil, fmt.Errorf("case %q matches nothing", g)+ }+ }+ if len(out) == 0 {+ return nil, fmt.Errorf("the selection is empty")+ }+ return out, nil+}++func hasAll(c *Case, tags []string) bool {+ for _, t := range tags {+ if !c.HasTag(t) {+ return false+ }+ }+ return true+}diff --git a/cli/test/evals/internal/catalog/template.go b/cli/test/evals/internal/catalog/template.gonew file mode 100644index 00000000..645b500b--- /dev/null+++ b/cli/test/evals/internal/catalog/template.go@@ -0,0 +1,143 @@+package catalog++import (+ "encoding/json"+ "fmt"+ "os"+ "regexp"+ "strconv"+ "strings"++ "gopkg.in/yaml.v3"+)++// templateRe matches {{seed.x.y}}, {{env.url}}, {{now}}; {{last.x}} and+// {{calls.cNN.x}} are runtime templates the fake model resolves, so they are+// left in place here.+var templateRe = regexp.MustCompile(`\{\{\s*([A-Za-z0-9_.\-\[\]]+)\s*\}\}`)++func isRuntime(expr string) bool {+ return expr == "last" || strings.HasPrefix(expr, "last.") || strings.HasPrefix(expr, "calls.")+}++// Lookup resolves a dotted path in vars ("seed.slowRequest.id").+func Lookup(vars map[string]any, expr string) (any, bool) {+ var cur any = vars+ for _, seg := range strings.Split(strings.NewReplacer("[", ".", "]", "").Replace(expr), ".") {+ if seg == "" {+ continue+ }+ switch t := cur.(type) {+ case map[string]any:+ v, ok := t[seg]+ if !ok {+ return nil, false+ }+ cur = v+ case []any:+ i, err := strconv.Atoi(seg)+ if err != nil || i < 0 || i >= len(t) {+ return nil, false+ }+ cur = t[i]+ default:+ return nil, false+ }+ }+ return cur, cur != nil+}++// ExpandString replaces templates in s. Unknown templates are an error;+// runtime templates are kept.+func ExpandString(s string, vars map[string]any) (string, error) {+ var firstErr error+ out := templateRe.ReplaceAllStringFunc(s, func(m string) string {+ expr := templateRe.FindStringSubmatch(m)[1]+ if isRuntime(expr) {+ return m+ }+ v, ok := Lookup(vars, expr)+ if !ok {+ if firstErr == nil {+ firstErr = fmt.Errorf("template {{%s}} has no value", expr)+ }+ return m+ }+ return scalarString(v)+ })+ return out, firstErr+}++func scalarString(v any) string {+ switch t := v.(type) {+ case string:+ return t+ case float64:+ return strconv.FormatFloat(t, 'f', -1, 64)+ case map[string]any, []any:+ raw, _ := json.Marshal(t)+ return string(raw)+ default:+ return fmt.Sprint(t)+ }+}++// Resolve returns a copy of the case with every template filled from vars.+// It re-reads the YAML and expands scalar nodes, so a scalar that is exactly+// one template keeps the value's type (a number stays a number).+func (c *Case) Resolve(vars map[string]any) (*Case, error) {+ raw, err := os.ReadFile(c.Path)+ if err != nil {+ return nil, err+ }+ var doc yaml.Node+ if err := yaml.Unmarshal(raw, &doc); err != nil {+ return nil, err+ }+ var firstErr error+ var walk func(n *yaml.Node)+ walk = func(n *yaml.Node) {+ if n.Kind == yaml.ScalarNode && strings.Contains(n.Value, "{{") {+ trimmed := strings.TrimSpace(n.Value)+ if m := templateRe.FindStringSubmatch(trimmed); m != nil && m[0] == trimmed && !isRuntime(m[1]) {+ if v, ok := Lookup(vars, m[1]); ok {+ switch t := v.(type) {+ case float64:+ n.Value, n.Tag, n.Style = strconv.FormatFloat(t, 'f', -1, 64), "!!float", 0+ case int:+ n.Value, n.Tag, n.Style = strconv.Itoa(t), "!!int", 0+ case bool:+ n.Value, n.Tag, n.Style = strconv.FormatBool(t), "!!bool", 0+ default:+ n.Value, n.Tag = scalarString(v), "!!str"+ }+ return+ }+ }+ s, err := ExpandString(n.Value, vars)+ if err != nil && firstErr == nil {+ firstErr = fmt.Errorf("%s: %w", c.ID, err)+ }+ n.Value = s+ if n.Tag != "!!str" && strings.Contains(s, "{{") {+ n.Tag = "!!str"+ }+ }+ for _, ch := range n.Content {+ walk(ch)+ }+ }+ walk(&doc)+ if firstErr != nil {+ return nil, firstErr+ }+ var out Case+ if err := doc.Decode(&out); err != nil {+ return nil, fmt.Errorf("%s: after templating: %w", c.ID, err)+ }+ out.Path = c.Path+ if out.Invoke == "" {+ out.Invoke = "natural"+ }+ return &out, nil+}diff --git a/cli/test/evals/internal/checklist/checklist.go b/cli/test/evals/internal/checklist/checklist.gonew file mode 100644index 00000000..7b40c83a--- /dev/null+++ b/cli/test/evals/internal/checklist/checklist.go@@ -0,0 +1,104 @@+// Package checklist turns the bottleneck checklist in performance.md into the+// enum submit_perf_diagnosis takes.+package checklist++import (+ "regexp"+ "strings"+ "sync"++ "github.com/tracewayapp/traceway/cli/pkg/mcpserver/knowledge"+)++// Row is one row of the bottleneck checklist in performance.md.+type Row struct {+ ID string // slug, the enum value submit_perf_diagnosis takes+ Title string // row title as written, markdown stripped+ Category string // the ### heading above the table+ Confirm string // the "How to confirm in Traceway" cell+}++// Extra answer values beyond the checklist rows.+const (+ NoneFound = "none_found"+ InsufficientEvidence = "insufficient_evidence"+)++var (+ checklistOnce sync.Once+ checklist []Row+)++// Checklist parses the canonical performance.md once. The enum therefore+// tracks the skill: renaming a row renames its id, and the case test fails.+func Rows() []Row {+ checklistOnce.Do(func() { checklist = ParseChecklist(knowledge.MustRead("performance.md")) })+ return checklist+}++var (+ rowRe = regexp.MustCompile(`^\|\s*\*\*(.+?)\*\*\s*\|\s*(.+?)\s*\|\s*$`)+ parenRe = regexp.MustCompile(`\s*\([^)]*\s[^)]*\)`)+ nonAlnumRe = regexp.MustCompile(`[^a-z0-9]+`)+ backtickRe = regexp.MustCompile("`")+ headingRe = regexp.MustCompile(`^###\s+(.+?)\s*$`)+ checklistTop = "## Bottleneck checklist"+)++// ParseChecklist extracts the checklist rows from a performance.md body.+func ParseChecklist(md string) []Row {+ var rows []Row+ in := false+ category := ""+ for _, line := range strings.Split(md, "\n") {+ if strings.HasPrefix(line, "## ") {+ in = strings.HasPrefix(line, checklistTop)+ continue+ }+ if !in {+ continue+ }+ if m := headingRe.FindStringSubmatch(line); m != nil {+ category = m[1]+ continue+ }+ m := rowRe.FindStringSubmatch(line)+ if m == nil {+ continue+ }+ title := backtickRe.ReplaceAllString(m[1], "")+ rows = append(rows, Row{ID: Slug(title), Title: title, Category: category, Confirm: m[2]})+ }+ return rows+}++// Slug turns a checklist title into its id: explanatory parentheticals (ones+// with a space inside) dropped, "+" spelled out, "^" dropped, everything else+// non-alphanumeric collapsed to "_".+func Slug(title string) string {+ s := parenRe.ReplaceAllString(title, "")+ s = strings.ToLower(s)+ s = strings.ReplaceAll(s, "+", " plus ")+ s = strings.ReplaceAll(s, "^", "")+ s = nonAlnumRe.ReplaceAllString(s, "_")+ return strings.Trim(s, "_")+}++// ChecklistIDs returns every valid checklist_item value, rows first.+func IDs() []string {+ var ids []string+ for _, r := range Rows() {+ ids = append(ids, r.ID)+ }+ return append(ids, NoneFound, InsufficientEvidence)+}++// IsChecklistID reports whether id is a valid checklist_item.+func IsID(id string) bool {+ for _, v := range IDs() {+ if v == id {+ return true+ }+ }+ return false+}diff --git a/cli/test/evals/internal/env/api.go b/cli/test/evals/internal/env/api.gonew file mode 100644index 00000000..6cdd24e6--- /dev/null+++ b/cli/test/evals/internal/env/api.go@@ -0,0 +1,125 @@+// Package env runs one hermetic Traceway instance per eval run: the+// standalone backend binary in SQLite mode in a temp dir, plus the users,+// projects, tokens and CLI logins a case needs. It talks HTTP only and never+// imports the backend.+package env++import (+ "bytes"+ "context"+ "encoding/json"+ "fmt"+ "io"+ "net/http"+ "net/url"+ "strings"+ "time"+)++// API is a small JSON client for the dashboard API.+type API struct {+ Base string // http://127.0.0.1:<port>, no /api suffix+ Bearer string+ HTTP *http.Client+}++// Response is a decoded API reply.+type Response struct {+ Status int+ Body []byte+}++// JSON decodes the body into v.+func (r *Response) JSON(v any) error {+ if err := json.Unmarshal(r.Body, v); err != nil {+ return fmt.Errorf("decode %d response: %w (body: %s)", r.Status, err, clip(r.Body))+ }+ return nil+}++// Map decodes the body as an object.+func (r *Response) Map() map[string]any {+ var m map[string]any+ _ = json.Unmarshal(r.Body, &m)+ return m+}++// OK reports a 2xx status.+func (r *Response) OK() bool { return r.Status >= 200 && r.Status < 300 }++// Err returns an error describing a non-2xx reply, or nil.+func (r *Response) Err(what string) error {+ if r.OK() {+ return nil+ }+ return fmt.Errorf("%s: HTTP %d: %s", what, r.Status, clip(r.Body))+}++// WithBearer returns a copy authenticated with another token.+func (a *API) WithBearer(token string) *API {+ c := *a+ c.Bearer = token+ return &c+}++// Do sends a request. path starts with /api/...; query may be nil.+func (a *API) Do(ctx context.Context, method, path string, query url.Values, body any) (*Response, error) {+ u := a.Base + path+ if len(query) > 0 {+ u += "?" + query.Encode()+ }+ var rd io.Reader+ if body != nil {+ raw, err := json.Marshal(body)+ if err != nil {+ return nil, err+ }+ rd = bytes.NewReader(raw)+ }+ req, err := http.NewRequestWithContext(ctx, method, u, rd)+ if err != nil {+ return nil, err+ }+ if body != nil {+ req.Header.Set("Content-Type", "application/json")+ }+ if a.Bearer != "" {+ req.Header.Set("Authorization", "Bearer "+a.Bearer)+ }+ hc := a.HTTP+ if hc == nil {+ hc = &http.Client{Timeout: 60 * time.Second}+ }+ resp, err := hc.Do(req)+ if err != nil {+ return nil, err+ }+ defer resp.Body.Close()+ raw, err := io.ReadAll(resp.Body)+ if err != nil {+ return nil, err+ }+ return &Response{Status: resp.StatusCode, Body: raw}, nil+}++// MustOK sends a request and fails on a non-2xx status.+func (a *API) MustOK(ctx context.Context, method, path string, query url.Values, body any) (*Response, error) {+ r, err := a.Do(ctx, method, path, query, body)+ if err != nil {+ return nil, fmt.Errorf("%s %s: %w", method, path, err)+ }+ return r, r.Err(method + " " + path)+}++// ProjectQuery returns ?projectId=<id>.+func ProjectQuery(projectID string) url.Values {+ return url.Values{"projectId": []string{projectID}}+}++func clip(b []byte) string {+ s := strings.TrimSpace(string(b))+ if len(s) > 400 {+ return s[:400] + "..."+ }+ return s+}diff --git a/cli/test/evals/internal/env/backend.go b/cli/test/evals/internal/env/backend.gonew file mode 100644index 00000000..bd2039f6--- /dev/null+++ b/cli/test/evals/internal/env/backend.go@@ -0,0 +1,245 @@+package env++import (+ "bytes"+ "context"+ "crypto/rand"+ "encoding/hex"+ "errors"+ "fmt"+ "net"+ "net/http"+ "os"+ "os/exec"+ "path/filepath"+ "regexp"+ "strconv"+ "sync"+ "syscall"+ "time"+)++// Backend is one running backend subprocess.+type Backend struct {+ URL string+ Port int+ Dir string // run/backend: cwd, SQLite files, ./storage+ DBPath string+ HealthToken string++ cmd *exec.Cmd+ log *syncBuffer+ done chan error+}++// BootOptions tunes the backend.+type BootOptions struct {+ // Dir is the run's backend directory (created if missing).+ Dir string+ // Env adds or overrides variables.+ Env map[string]string+ // Ready bounds how long to wait for /health.+ Ready time.Duration+}++// errPortTaken means another server answered on the port FreePort picked+// before this backend could bind it (a parallel run won the race).+var errPortTaken = errors.New("another server answered on the backend's port")++// Boot starts the backend binary in SQLite mode with fast background workers+// and waits until it answers /health/deep with its own token. A port lost to+// a parallel run is retried on a fresh one.+func Boot(ctx context.Context, bin string, opt BootOptions) (*Backend, error) {+ for attempt := 1; ; attempt++ {+ b, err := boot(ctx, bin, opt)+ if errors.Is(err, errPortTaken) && attempt < 3 {+ continue+ }+ return b, err+ }+}++func boot(ctx context.Context, bin string, opt BootOptions) (*Backend, error) {+ if err := os.MkdirAll(opt.Dir, 0o755); err != nil {+ return nil, err+ }+ port, err := FreePort()+ if err != nil {+ return nil, err+ }+ b := &Backend{+ URL: fmt.Sprintf("http://127.0.0.1:%d", port),+ Port: port,+ Dir: opt.Dir,+ DBPath: filepath.Join(opt.Dir, "traceway.db"),+ HealthToken: randomHex(16),+ log: &syncBuffer{},+ done: make(chan error, 1),+ }+ vars := map[string]string{+ "PATH": os.Getenv("PATH"),+ "HOME": opt.Dir,+ "TMPDIR": os.TempDir(),+ "TZ": "UTC",+ "JWT_SECRET": randomHex(24), // 48 characters+ "DB_TYPE": "sqlite",+ "SQLITE_PATH": b.DBPath,+ "STORAGE_TYPE": "local",+ "STORAGE_PATH": filepath.Join(opt.Dir, "storage"),+ "PORTS": strconv.Itoa(port),+ "APP_BASE_URL": b.URL,+ "SQLITE_RETENTION_DAYS": "0",+ "NOTIFICATION_POLL_SECONDS": "5",+ "ONCALL_POLL_SECONDS": "5",+ "OUTBOX_POLL_SECONDS": "5",+ "SYNTHETICS_POLL_SECONDS": "5",+ "SYNTHETICS_ALLOW_PRIVATE_TARGETS": "true",+ "HEALTH_DEEP_TOKEN": b.HealthToken,+ "SQLITE_CACHE_SIZE_MB": "64",+ "INGEST_ADMISSION_WAIT_SECONDS": "30",+ "REPORT_MAX_BODY_MB": "64",+ }+ for k, v := range opt.Env {+ vars[k] = v+ }+ cmd := exec.Command(bin)+ cmd.Dir = opt.Dir+ cmd.Env = make([]string, 0, len(vars))+ for k, v := range vars {+ cmd.Env = append(cmd.Env, k+"="+v)+ }+ cmd.Stdout = b.log+ cmd.Stderr = b.log+ cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}+ if err := cmd.Start(); err != nil {+ return nil, fmt.Errorf("start backend: %w", err)+ }+ b.cmd = cmd+ go func() { b.done <- cmd.Wait() }()++ ready := opt.Ready+ if ready == 0 {+ ready = 60 * time.Second+ }+ if err := b.waitReady(ctx, ready); err != nil {+ b.Stop()+ if errors.Is(err, errPortTaken) {+ return nil, err+ }+ return nil, fmt.Errorf("%w\n--- backend log ---\n%s", err, b.LogTail(4000))+ }+ return b, nil+}++func (b *Backend) waitReady(ctx context.Context, timeout time.Duration) error {+ deadline := time.Now().Add(timeout)+ hc := &http.Client{Timeout: 2 * time.Second}+ for time.Now().Before(deadline) {+ select {+ case err := <-b.done:+ b.done <- err+ return fmt.Errorf("backend exited before it was ready: %v", err)+ case <-ctx.Done():+ return ctx.Err()+ default:+ }+ // /health/deep needs this backend's own token, so a server that+ // took the port first answers 401 instead of passing for ours.+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, b.URL+"/api/health/deep", nil)+ if err != nil {+ return err+ }+ req.Header.Set("Authorization", "Bearer "+b.HealthToken)+ if resp, err := hc.Do(req); err == nil {+ _ = resp.Body.Close()+ switch resp.StatusCode {+ case http.StatusOK:+ return nil+ case http.StatusUnauthorized:+ return errPortTaken+ }+ }+ time.Sleep(100 * time.Millisecond)+ }+ return errors.New("backend never answered /health")+}++// Stop terminates the backend and its process group.+func (b *Backend) Stop() {+ if b.cmd == nil || b.cmd.Process == nil {+ return+ }+ _ = syscall.Kill(-b.cmd.Process.Pid, syscall.SIGTERM)+ select {+ case <-b.done:+ case <-time.After(10 * time.Second):+ _ = syscall.Kill(-b.cmd.Process.Pid, syscall.SIGKILL)+ <-b.done+ }+ b.cmd = nil+}++// Log returns the backend's combined output so far.+func (b *Backend) Log() string { return b.log.String() }++// LogTail returns the last n bytes of the log.+func (b *Backend) LogTail(n int) string {+ s := b.log.String()+ if len(s) > n {+ return s[len(s)-n:]+ }+ return s+}++// WriteLog saves the log next to the run.+func (b *Backend) WriteLog(path string) error {+ return os.WriteFile(path, []byte(b.log.String()), 0o644)+}++var inviteTokenRe = regexp.MustCompile(`accept-invitation\?token=([0-9a-f-]{36})`)++// InvitationTokens returns every invitation accept token the backend has+// logged so far. With SMTP off the email is written to the log, link included.+func (b *Backend) InvitationTokens() []string {+ var out []string+ for _, m := range inviteTokenRe.FindAllStringSubmatch(b.log.String(), -1) {+ out = append(out, m[1])+ }+ return out+}++// API returns an unauthenticated API client for this backend.+func (b *Backend) API() *API { return &API{Base: b.URL} }++// FreePort asks the OS for an unused TCP port.+func FreePort() (int, error) {+ l, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ return 0, err+ }+ defer l.Close()+ return l.Addr().(*net.TCPAddr).Port, nil+}++func randomHex(n int) string {+ b := make([]byte, n)+ _, _ = rand.Read(b)+ return hex.EncodeToString(b)+}++type syncBuffer struct {+ mu sync.Mutex+ buf bytes.Buffer+}++func (s *syncBuffer) Write(p []byte) (int, error) {+ s.mu.Lock()+ defer s.mu.Unlock()+ return s.buf.Write(p)+}++func (s *syncBuffer) String() string {+ s.mu.Lock()+ defer s.mu.Unlock()+ return s.buf.String()+}diff --git a/cli/test/evals/internal/env/build.go b/cli/test/evals/internal/env/build.gonew file mode 100644index 00000000..e503e4ff--- /dev/null+++ b/cli/test/evals/internal/env/build.go@@ -0,0 +1,147 @@+package env++import (+ "crypto/sha256"+ "encoding/hex"+ "fmt"+ "os"+ "os/exec"+ "path/filepath"+ "strings"+ "sync"+)++// Binaries are the executables a run needs.+type Binaries struct {+ Backend string // backend/cmd/traceway+ CLI string // cli/cmd/traceway (read edition)+ CLIWrite string // cli/cmd/traceway -tags write+ Hash string // tree hash they were built from+}++var (+ buildMu sync.Mutex+ buildCache = map[string]*Binaries{}+)++// BuildBinaries builds the backend and both CLI editions from repoRoot, cached+// under cacheDir/bin/<tree-hash>/ so an unchanged tree never rebuilds.+func BuildBinaries(repoRoot, cacheDir string) (*Binaries, error) {+ buildMu.Lock()+ defer buildMu.Unlock()+ hash, err := TreeHash(repoRoot)+ if err != nil {+ return nil, err+ }+ if b, ok := buildCache[hash]; ok {+ return b, nil+ }+ dir := filepath.Join(cacheDir, "bin", hash)+ b := &Binaries{+ Backend: filepath.Join(dir, "traceway-backend"),+ CLI: filepath.Join(dir, "traceway"),+ CLIWrite: filepath.Join(dir, "traceway-write"),+ Hash: hash,+ }+ if err := os.MkdirAll(dir, 0o755); err != nil {+ return nil, err+ }+ steps := []struct {+ out, dir string+ args []string+ }{+ {b.Backend, filepath.Join(repoRoot, "backend"), []string{"build", "-o", b.Backend, "./cmd/traceway"}},+ {b.CLI, filepath.Join(repoRoot, "cli"), []string{"build", "-o", b.CLI, "./cmd/traceway"}},+ {b.CLIWrite, filepath.Join(repoRoot, "cli"), []string{"build", "-tags", "write", "-o", b.CLIWrite, "./cmd/traceway"}},+ }+ for _, s := range steps {+ if _, err := os.Stat(s.out); err == nil {+ continue+ }+ // Build to a private path and rename, so concurrent harness+ // processes never see (or execute) a half-written binary.+ tmp := fmt.Sprintf("%s.tmp-%d", s.out, os.Getpid())+ args := append([]string{}, s.args...)+ for i := range args {+ if args[i] == s.out {+ args[i] = tmp+ }+ }+ cmd := exec.Command("go", args...)+ cmd.Dir = s.dir+ cmd.Env = append(os.Environ(), "CGO_ENABLED=0")+ if out, err := cmd.CombinedOutput(); err != nil {+ return nil, fmt.Errorf("go %s in %s: %w\n%s", strings.Join(args, " "), s.dir, err, out)+ }+ if err := os.Rename(tmp, s.out); err != nil {+ return nil, err+ }+ }+ buildCache[hash] = b+ return b, nil+}++// TreeHash fingerprints the backend and CLI sources: the committed trees,+// the uncommitted diff, and untracked files. The evals module itself is+// excluded so editing a case never rebuilds the binaries.+func TreeHash(repoRoot string) (string, error) {+ git := func(args ...string) ([]byte, error) {+ cmd := exec.Command("git", args...)+ cmd.Dir = repoRoot+ out, err := cmd.Output()+ if err != nil {+ return nil, fmt.Errorf("git %s: %w", strings.Join(args, " "), err)+ }+ return out, nil+ }+ h := sha256.New()+ paths := []string{"backend", "cli", ":(exclude)cli/test"}+ head, err := git("ls-tree", "-r", "HEAD", "--", "backend", "cli")+ if err != nil {+ return "", err+ }+ for _, line := range strings.Split(string(head), "\n") {+ if !strings.Contains(line, "\tcli/test/") {+ h.Write([]byte(line))+ }+ }+ diff, err := git(append([]string{"diff", "HEAD", "--binary", "--"}, paths...)...)+ if err != nil {+ return "", err+ }+ h.Write(diff)+ untracked, err := git(append([]string{"ls-files", "-o", "--exclude-standard", "--"}, paths...)...)+ if err != nil {+ return "", err+ }+ for _, f := range strings.Split(strings.TrimSpace(string(untracked)), "\n") {+ if f == "" {+ continue+ }+ raw, err := os.ReadFile(filepath.Join(repoRoot, f))+ if err != nil {+ continue+ }+ h.Write([]byte(f))+ h.Write(raw)+ }+ return hex.EncodeToString(h.Sum(nil))[:16], nil+}++// RepoRoot walks up from dir to the directory holding .git.+func RepoRoot(dir string) (string, error) {+ d, err := filepath.Abs(dir)+ if err != nil {+ return "", err+ }+ for {+ if _, err := os.Stat(filepath.Join(d, ".git")); err == nil {+ return d, nil+ }+ parent := filepath.Dir(d)+ if parent == d {+ return "", fmt.Errorf("no git repository above %s", dir)+ }+ d = parent+ }+}diff --git a/cli/test/evals/internal/env/clis.go b/cli/test/evals/internal/env/clis.gonew file mode 100644index 00000000..35d6ce80--- /dev/null+++ b/cli/test/evals/internal/env/clis.go@@ -0,0 +1,74 @@+package env++import (+ "bytes"+ "context"+ "fmt"+ "os"+ "os/exec"+ "path/filepath"+ "strings"+)++// CLIHome is the isolated home of one CLI edition: its XDG dirs live outside+// the agent's workspace so the agent cannot read the stored token.+type CLIHome struct {+ Bin string+ Home string+ XDGCf string+ XDGSt string+}++// NewCLIHome prepares dirs under root for a binary.+func NewCLIHome(root, bin string) (*CLIHome, error) {+ h := &CLIHome{Bin: bin, Home: filepath.Join(root, "home"), XDGCf: filepath.Join(root, "xdg", "config"), XDGSt: filepath.Join(root, "xdg", "state")}+ for _, d := range []string{h.Home, h.XDGCf, h.XDGSt} {+ if err := os.MkdirAll(d, 0o700); err != nil {+ return nil, err+ }+ }+ return h, nil+}++// Env is the minimal environment every tool subprocess gets. Nothing from the+// harness's own environment (API keys, GitHub tokens, JWT secrets) is passed.+func (h *CLIHome) Env(path string) []string {+ return []string{+ "HOME=" + h.Home,+ "XDG_CONFIG_HOME=" + h.XDGCf,+ "XDG_STATE_HOME=" + h.XDGSt,+ "XDG_CACHE_HOME=" + filepath.Join(h.Home, ".cache"),+ "PATH=" + path,+ "TZ=UTC",+ "LANG=C.UTF-8",+ }+}++// Run executes the CLI with args and optional stdin.+func (h *CLIHome) Run(ctx context.Context, stdin string, args ...string) (string, string, error) {+ cmd := exec.CommandContext(ctx, h.Bin, args...)+ cmd.Env = h.Env(filepath.Dir(h.Bin))+ cmd.Dir = h.Home+ if stdin != "" {+ cmd.Stdin = strings.NewReader(stdin)+ }+ var so, se bytes.Buffer+ cmd.Stdout, cmd.Stderr = &so, &se+ err := cmd.Run()+ return so.String(), se.String(), err+}++// Login stores a token for the instance and selects a project, the way a+// user would: `login --url U --token-stdin` then `projects use <id>`.+func (h *CLIHome) Login(ctx context.Context, url, token, projectID string) error {+ if so, se, err := h.Run(ctx, token+"\n", "login", "--url", url, "--token-stdin"); err != nil {+ return fmt.Errorf("%s login: %w\n%s%s", filepath.Base(h.Bin), err, so, se)+ }+ if projectID == "" {+ return nil+ }+ if so, se, err := h.Run(ctx, "", "projects", "use", projectID); err != nil {+ return fmt.Errorf("%s projects use: %w\n%s%s", filepath.Base(h.Bin), err, so, se)+ }+ return nil+}diff --git a/cli/test/evals/internal/env/env_test.go b/cli/test/evals/internal/env/env_test.gonew file mode 100644index 00000000..05b5cec2--- /dev/null+++ b/cli/test/evals/internal/env/env_test.go@@ -0,0 +1,97 @@+package env++import (+ "context"+ "net/http"+ "path/filepath"+ "strings"+ "sync"+ "testing"+ "time"+)++// TestTwoInstances boots two backends in parallel, registers each, and logs+// the CLI in with a read token.+func TestTwoInstances(t *testing.T) {+ bins := BinariesForTest(t)+ ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)+ defer cancel()++ var wg sync.WaitGroup+ for i := 0; i < 2; i++ {+ wg.Add(1)+ go func() {+ defer wg.Done()+ checkInstance(t, ctx, bins)+ }()+ }+ wg.Wait()+}++func checkInstance(t *testing.T, ctx context.Context, bins *Binaries) {+ dir := t.TempDir()+ b, err := Boot(ctx, bins.Backend, BootOptions{Dir: filepath.Join(dir, "backend")})+ if err != nil {+ t.Error(err)+ return+ }+ defer b.Stop()++ admin, err := Register(ctx, b, "[email protected]", "eval-password-1", "Eval Org")+ if err != nil {+ t.Error(err)+ return+ }+ projects, err := CreateProjects(ctx, b, admin, []Project{{Name: "api", Framework: "opentelemetry"}})+ if err != nil {+ t.Error(err)+ return+ }+ pid := projects[0].ID++ readPAT, err := CreatePAT(ctx, b, admin.JWT, "eval-read", "read", 0)+ if err != nil {+ t.Error(err)+ return+ }+ if !strings.HasPrefix(readPAT, "twp_") {+ t.Errorf("unexpected read token prefix %q", readPAT[:4])+ }++ cli, err := NewCLIHome(filepath.Join(dir, "cli"), bins.CLI)+ if err != nil {+ t.Error(err)+ return+ }+ if err := cli.Login(ctx, b.URL, readPAT, pid); err != nil {+ t.Error(err)+ return+ }+ out, _, err := cli.Run(ctx, "", "projects", "list", "--output", "json")+ if err != nil || !strings.Contains(out, pid) {+ t.Errorf("projects list after login: %v %s", err, out)+ }++}++func TestStubSwitches(t *testing.T) {+ s, err := StartStub(false)+ if err != nil {+ t.Fatal(err)+ }+ defer s.Close()+ res, err := http.Get(s.URL + "/health")+ if err != nil {+ t.Fatal(err)+ }+ res.Body.Close()+ if res.StatusCode != 503 {+ t.Fatalf("unhealthy stub answered %d", res.StatusCode)+ }+ s.SetHealthy(true)+ res, _ = http.Get(s.URL + "/health")+ res.Body.Close()+ if res.StatusCode != 200 || s.Hits() != 2 {+ t.Fatalf("healthy stub answered %d after %d hits", res.StatusCode, s.Hits())+ }+}diff --git a/cli/test/evals/internal/env/instance.go b/cli/test/evals/internal/env/instance.gonew file mode 100644index 00000000..b5d16ca3--- /dev/null+++ b/cli/test/evals/internal/env/instance.go@@ -0,0 +1,51 @@+package env++import (+ "context"+ "fmt"+ "path/filepath"+)++// Instance is a booted, registered backend with its monitor stub.+type Instance struct {+ Backend *Backend+ Admin *Admin+ Stub *Stub+ Dir string+}++// AdminEmail and AdminPassword are the owner credentials of every instance.+const (+ AdminEmail = "[email protected]"+ AdminPassword = "eval-owner-password"+)++// NewInstance boots a backend under dir/backend, registers the owner and+// organization, and starts a healthy stub.+func NewInstance(ctx context.Context, bins *Binaries, dir string) (*Instance, error) {+ b, err := Boot(ctx, bins.Backend, BootOptions{Dir: filepath.Join(dir, "backend")})+ if err != nil {+ return nil, err+ }+ a, err := Register(ctx, b, AdminEmail, AdminPassword, "Acme")+ if err != nil {+ b.Stop()+ return nil, fmt.Errorf("register: %w", err)+ }+ stub, err := StartStub(true)+ if err != nil {+ b.Stop()+ return nil, err+ }+ return &Instance{Backend: b, Admin: a, Stub: stub, Dir: dir}, nil+}++// Close stops everything.+func (i *Instance) Close() {+ if i.Stub != nil {+ i.Stub.Close()+ }+ if i.Backend != nil {+ i.Backend.Stop()+ }+}diff --git a/cli/test/evals/internal/env/register.go b/cli/test/evals/internal/env/register.gonew file mode 100644index 00000000..ecaf5c54--- /dev/null+++ b/cli/test/evals/internal/env/register.go@@ -0,0 +1,113 @@+package env++import (+ "context"+ "fmt"+ "net/http"+)++// Project is one Traceway project of the instance.+type Project struct {+ ID string `json:"id"`+ Name string `json:"name"`+ Framework string `json:"framework"`+ Token string `json:"-"` // ingest token; never written to seed.json+}++// Admin is the registered owner of the instance's one organization.+type Admin struct {+ Email string+ Password string+ UserID int+ OrgID int+ JWT string+}++// Register creates the instance's owner and organization. A self-hosted+// instance allows exactly one registration, which is why every run boots its+// own backend.+func Register(ctx context.Context, b *Backend, email, password, org string) (*Admin, error) {+ res, err := b.API().MustOK(ctx, http.MethodPost, "/api/register", nil, map[string]any{+ "email": email, "name": "Eval Admin", "password": password,+ "organizationName": org, "timezone": "UTC",+ })+ if err != nil {+ return nil, err+ }+ var out struct {+ Token string `json:"token"`+ User struct {+ ID int `json:"id"`+ } `json:"user"`+ Organizations []struct {+ ID int `json:"id"`+ } `json:"organizations"`+ }+ if err := res.JSON(&out); err != nil {+ return nil, err+ }+ if out.Token == "" || len(out.Organizations) == 0 {+ return nil, fmt.Errorf("register returned no token or organization: %s", clip(res.Body))+ }+ return &Admin{Email: email, Password: password, UserID: out.User.ID, OrgID: out.Organizations[0].ID, JWT: out.Token}, nil+}++// CreateProjects creates projects in the admin's organization in one batch+// and returns them with their ingest tokens.+func CreateProjects(ctx context.Context, b *Backend, a *Admin, specs []Project) ([]Project, error) {+ if len(specs) == 0 {+ return nil, nil+ }+ body := map[string]any{"organizationId": a.OrgID}+ var list []map[string]string+ for _, p := range specs {+ list = append(list, map[string]string{"name": p.Name, "framework": p.Framework})+ }+ body["projects"] = list+ res, err := b.API().WithBearer(a.JWT).MustOK(ctx, http.MethodPost, "/api/projects/batch", nil, body)+ if err != nil {+ return nil, err+ }+ var out struct {+ Projects []struct {+ ID string `json:"id"`+ Name string `json:"name"`+ Framework string `json:"framework"`+ Token string `json:"token"`+ } `json:"projects"`+ }+ if err := res.JSON(&out); err != nil {+ return nil, err+ }+ if len(out.Projects) != len(specs) {+ return nil, fmt.Errorf("batch create returned %d projects, want %d", len(out.Projects), len(specs))+ }+ projects := make([]Project, 0, len(out.Projects))+ for _, p := range out.Projects {+ projects = append(projects, Project{ID: p.ID, Name: p.Name, Framework: p.Framework, Token: p.Token})+ }+ return projects, nil+}++// CreatePAT mints a personal access token for the bearer's user. Write+// tokens must expire; expiresInDays 0 means "never" for read tokens.+func CreatePAT(ctx context.Context, b *Backend, jwt, name, scope string, expiresInDays int) (string, error) {+ body := map[string]any{"name": name, "scope": scope}+ if expiresInDays > 0 {+ body["expiresInDays"] = expiresInDays+ }+ res, err := b.API().WithBearer(jwt).MustOK(ctx, http.MethodPost, "/api/personal-access-tokens", nil, body)+ if err != nil {+ return "", err+ }+ var out struct {+ Token string `json:"token"`+ }+ if err := res.JSON(&out); err != nil {+ return "", err+ }+ if out.Token == "" {+ return "", fmt.Errorf("PAT create returned no token")+ }+ return out.Token, nil+}diff --git a/cli/test/evals/internal/env/stub.go b/cli/test/evals/internal/env/stub.gonew file mode 100644index 00000000..42a51b14--- /dev/null+++ b/cli/test/evals/internal/env/stub.go@@ -0,0 +1,53 @@+package env++import (+ "fmt"+ "net"+ "net/http"+ "sync/atomic"+)++// Stub is a local HTTP target whose health the harness switches, so monitors+// can go down and recover on cue.+type Stub struct {+ URL string+ healthy atomic.Bool+ hits atomic.Int64+ srv *http.Server+ ln net.Listener+}++// StartStub starts a stub on a free loopback port.+func StartStub(healthy bool) (*Stub, error) {+ ln, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ return nil, err+ }+ s := &Stub{URL: "http://" + ln.Addr().String(), ln: ln}+ s.healthy.Store(healthy)+ mux := http.NewServeMux()+ mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {+ s.hits.Add(1)+ if s.healthy.Load() {+ w.Header().Set("Content-Type", "application/json")+ fmt.Fprint(w, `{"status":"ok"}`)+ return+ }+ http.Error(w, `{"status":"unavailable"}`, http.StatusServiceUnavailable)+ })+ s.srv = &http.Server{Handler: mux}+ go func() { _ = s.srv.Serve(ln) }()+ return s, nil+}++// SetHealthy switches the stub's answers.+func (s *Stub) SetHealthy(v bool) { s.healthy.Store(v) }++// Healthy reports the current state.+func (s *Stub) Healthy() bool { return s.healthy.Load() }++// Hits counts requests served.+func (s *Stub) Hits() int64 { return s.hits.Load() }++// Close stops the stub.+func (s *Stub) Close() { _ = s.srv.Close() }diff --git a/cli/test/evals/internal/env/testhelp.go b/cli/test/evals/internal/env/testhelp.gonew file mode 100644index 00000000..dcf70833--- /dev/null+++ b/cli/test/evals/internal/env/testhelp.go@@ -0,0 +1,36 @@+package env++import (+ "os/exec"+ "path/filepath"+ "runtime"+ "testing"+)++// EvalsRoot is the cli/test/evals module directory.+func EvalsRoot() string {+ _, file, _, _ := runtime.Caller(0)+ return filepath.Clean(filepath.Join(filepath.Dir(file), "..", ".."))+}++// BinariesForTest builds (or reuses) the binaries for a test, skipping it in+// -short mode or when no Go toolchain is on PATH.+func BinariesForTest(t testing.TB) *Binaries {+ t.Helper()+ if testing.Short() {+ t.Skip("boots a backend")+ }+ if _, err := exec.LookPath("go"); err != nil {+ t.Skip("go toolchain not on PATH")+ }+ root := EvalsRoot()+ repo, err := RepoRoot(root)+ if err != nil {+ t.Fatal(err)+ }+ b, err := BuildBinaries(repo, filepath.Join(root, ".cache"))+ if err != nil {+ t.Fatal(err)+ }+ return b+}diff --git a/cli/test/evals/internal/grade/answer.go b/cli/test/evals/internal/grade/answer.gonew file mode 100644index 00000000..e2e9e5c5--- /dev/null+++ b/cli/test/evals/internal/grade/answer.go@@ -0,0 +1,143 @@+package grade++import (+ "encoding/json"+ "regexp"+ "slices"+ "strings"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+)++// Answer grades the structured submission against expect.answer.+func Answer(in *Input) *Verdict {+ v := newVerdict("answer", true)+ exp := in.Case.Expect.Answer+ sub := in.Run.Submission+ if sub == nil {+ v.fail("no submission (%s)", in.Run.StopReason)+ return v+ }+ str := func(m map[string]any, k string) string { s, _ := m[k].(string); return strings.TrimSpace(s) }+ switch in.Case.Suite {+ case "perf":+ got := str(sub, "checklist_item")+ want := append([]string{}, exp.AnyOf...)+ if exp.ChecklistItem != "" {+ want = append(want, exp.ChecklistItem)+ }+ if len(want) > 0 && !slices.Contains(want, got) {+ v.fail("checklist_item %q, want %s", got, strings.Join(want, " or "))+ }+ if slices.Contains(exp.Not, got) {+ v.fail("checklist_item %q is the distractor", got)+ }+ if exp.Locus != nil {+ locus, _ := sub["locus"].(map[string]any)+ if locus == nil {+ locus = map[string]any{}+ }+ if exp.Locus.Endpoint != "" && !sameEndpoint(str(locus, "endpoint"), exp.Locus.Endpoint) {+ v.fail("locus.endpoint %q, want %q", str(locus, "endpoint"), exp.Locus.Endpoint)+ }+ if exp.Locus.SpanContains != "" && !containsFold(str(locus, "span"), exp.Locus.SpanContains) {+ v.fail("locus.span %q does not contain %q", str(locus, "span"), exp.Locus.SpanContains)+ }+ if exp.Locus.Service != "" && !containsFold(str(locus, "service"), exp.Locus.Service) {+ v.fail("locus.service %q, want %q", str(locus, "service"), exp.Locus.Service)+ }+ if exp.Locus.Host != "" && !containsFold(str(locus, "host"), exp.Locus.Host) {+ v.fail("locus.host %q, want %q", str(locus, "host"), exp.Locus.Host)+ }+ }+ if exp.Onset != nil && exp.Onset.At != "" {+ gradeOnset(v, sub["onset"], exp.Onset)+ }+ }+ if len(exp.MentionsAll) > 0 {+ raw, _ := json.Marshal(sub)+ text := string(raw) + "\n" + in.Run.FinalText+ for _, pat := range exp.MentionsAll {+ re, err := regexp.Compile("(?is)" + pat)+ if err != nil {+ v.fail("bad mentions_all pattern %q: %v", pat, err)+ continue+ }+ if !re.MatchString(text) {+ v.fail("the answer never mentions /%s/", pat)+ }+ }+ }+ if p, ok := sub["_validation_problems"].(string); ok {+ v.note("accepted after failed repairs: %s", p)+ }+ return v+}++func gradeOnset(v *Verdict, got any, exp *catalog.Onset) {+ want, err := time.Parse(time.RFC3339Nano, exp.At)+ if err != nil {+ v.fail("case onset %q is not RFC 3339", exp.At)+ return+ }+ s, _ := got.(string)+ at, err := time.Parse(time.RFC3339Nano, s)+ if err != nil {+ v.fail("onset %v is not an RFC 3339 time", got)+ return+ }+ tol := time.Duration(exp.ToleranceMinutes) * time.Minute+ if tol == 0 {+ tol = time.Hour+ }+ if d := at.Sub(want); d < -tol || d > tol {+ v.fail("onset %s is %s from %s (tolerance %s)", at.Format(time.RFC3339), d.Round(time.Minute), want.Format(time.RFC3339), tol)+ }+}++var spaceRe = regexp.MustCompile(`\s+`)++// sameEndpoint compares "GET /api/orders" forms, tolerating case, spacing,+// a trailing slash and a missing method when the route matches.+func sameEndpoint(got, want string) bool {+ norm := func(s string) string {+ s = strings.TrimSpace(spaceRe.ReplaceAllString(s, " "))+ s = strings.Trim(s, "`\"'")+ method, route, ok := strings.Cut(s, " ")+ if !ok {+ return strings.TrimSuffix(s, "/")+ }+ return strings.ToUpper(method) + " " + strings.TrimSuffix(route, "/")+ }+ g, w := norm(got), norm(want)+ if g == w {+ return true+ }+ _, wr, _ := strings.Cut(w, " ")+ return g == wr || strings.HasSuffix(g, " "+wr)+}++func containsFold(s, sub string) bool {+ return strings.Contains(strings.ToLower(s), strings.ToLower(sub))+}++// Report checks free-text requirements on the final report.+func Report(in *Input) *Verdict {+ v := newVerdict("report", true)+ text := FinalText(in.Run)+ for _, pat := range in.Case.Expect.Report.Contains {+ if re, err := regexp.Compile("(?is)" + pat); err != nil || !re.MatchString(text) {+ v.fail("report does not match /%s/", pat)+ }+ }+ for _, pat := range in.Case.Expect.Report.NotContains {+ if re, err := regexp.Compile("(?is)" + pat); err == nil && re.MatchString(text) {+ v.fail("report must not match /%s/", pat)+ }+ }+ if text == "" {+ v.fail("no report")+ }+ return v+}diff --git a/cli/test/evals/internal/grade/grade.go b/cli/test/evals/internal/grade/grade.gonew file mode 100644index 00000000..8886dac2--- /dev/null+++ b/cli/test/evals/internal/grade/grade.go@@ -0,0 +1,84 @@+// Package grade scores one run. Deterministic graders gate the result:+// answer, trajectory, grounding and the report checks.+package grade++import (+ "context"+ "fmt"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// Verdict is one grader's result.+type Verdict struct {+ Grader string `json:"grader"`+ Pass bool `json:"pass"`+ Gating bool `json:"gating"`+ Skipped bool `json:"skipped,omitempty"`+ Score *float64 `json:"score,omitempty"`+ Failures []string `json:"failures,omitempty"`+ Notes []string `json:"notes,omitempty"`+ CostUSD float64 `json:"costUsd,omitempty"`+}++func (v *Verdict) fail(format string, a ...any) {+ v.Pass = false+ v.Failures = append(v.Failures, fmt.Sprintf(format, a...))+}++func (v *Verdict) note(format string, a ...any) { v.Notes = append(v.Notes, fmt.Sprintf(format, a...)) }++func newVerdict(name string, gating bool) *Verdict {+ return &Verdict{Grader: name, Pass: true, Gating: gating}+}++// Input is everything the graders see.+type Input struct {+ Case *catalog.Case // templates resolved+ Arm string+ Run *record.Run+ Seed map[string]any+}++// All runs every applicable grader in a fixed order.+func All(ctx context.Context, in *Input) []*Verdict {+ var out []*Verdict+ if in.Case.Expect.Answer != nil {+ out = append(out, Answer(in))+ }+ if in.Case.Expect.Calls != nil {+ out = append(out, Trajectory(in.Run, in.Case.Expect.Calls))+ }+ if in.Case.Suite == "perf" || in.Case.Suite == "debug" || len(in.Case.Expect.Grounding) > 0 {+ out = append(out, Grounding(in))+ }+ if in.Case.Expect.Report != nil {+ out = append(out, Report(in))+ }+ return out+}++// Passed reports whether every gating grader passed.+func Passed(vs []*Verdict) bool {+ for _, v := range vs {+ if v.Gating && !v.Skipped && !v.Pass {+ return false+ }+ }+ return true+}++// FinalText is the agent's report: the submission's report/summary field,+// else its last message.+func FinalText(r *record.Run) string {+ if r.Submission != nil {+ for _, k := range []string{"report", "summary"} {+ if s, ok := r.Submission[k].(string); ok && strings.TrimSpace(s) != "" {+ return s+ }+ }+ }+ return r.FinalText+}diff --git a/cli/test/evals/internal/grade/grade_test.go b/cli/test/evals/internal/grade/grade_test.gonew file mode 100644index 00000000..4f51529f--- /dev/null+++ b/cli/test/evals/internal/grade/grade_test.go@@ -0,0 +1,166 @@+package grade++import (+ "encoding/json"+ "strings"+ "testing"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++func bp(b bool) *bool { return &b }++// call builds a recorded call from CLI argv strings.+func call(id string, ok bool, output string, argvs ...string) *record.Call {+ c := &record.Call{ID: id, Tool: "shell", OK: ok, Output: output, Args: json.RawMessage(`{}`)}+ for _, a := range argvs {+ argv := strings.Fields(a)+ c.Ops = append(c.Ops, record.OpRecord{Op: ops.ParseCLI(argv[1:]), Argv: argv, OK: ok, Output: output})+ }+ return c+}++func failures(v *Verdict) string { return strings.Join(v.Failures, "\n") }++func TestTrajectory(t *testing.T) {+ run := &record.Run{Calls: []*record.Call{+ call("c01", true, "{}", "traceway endpoints list --since 24h"),+ call("c02", true, "{}", "traceway endpoints show abc --recorded-at 2026-10-17T10:00:00Z"),+ call("c03", false, "error", "traceway metrics query --name system.cpu.utilization"),+ {ID: "c04", Tool: "read_file", OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: map[string]any{"path": "internal/handlers/orders.go"}}, OK: true}}},+ }}+ spec := &catalog.Trajectory{+ Required: []catalog.CallMatcher{{Op: "endpoints.list"}, {Op: "endpoints.show", Args: map[string]any{"recorded_at": "present"}}, {Op: "read_file:internal/handlers/*.go"}},+ AnyOf: [][]catalog.CallMatcher{{{Op: "metrics.query", Args: map[string]any{"name": "~system\\.cpu"}}}},+ Forbidden: []catalog.CallMatcher{{Op: "*", Write: bp(true)}},+ Order: [][]string{{"endpoints.list", "endpoints.show"}},+ }+ if v := Trajectory(run, spec); !v.Pass {+ t.Fatalf("a good trajectory failed:\n%s", failures(v))+ }++ cases := map[string]struct {+ mutate func(r *record.Run, s *catalog.Trajectory)+ want string+ }{+ "forbidden write": {func(r *record.Run, s *catalog.Trajectory) {+ r.Calls = append(r.Calls, call("c05", true, "{}", "traceway-write exceptions archive abcd1234abcd1234 --yes"))+ }, "forbidden call"},+ "wrong order": {func(r *record.Run, s *catalog.Trajectory) {+ r.Calls[0], r.Calls[1] = r.Calls[1], r.Calls[0]+ }, "came after"},+ "missing timestamp": {func(r *record.Run, s *catalog.Trajectory) {+ r.Calls[1] = call("c02", true, "{}", "traceway endpoints show abc")+ }, "required call missing: endpoints.show"},+ "no alternative": {func(r *record.Run, s *catalog.Trajectory) {+ r.Calls = r.Calls[:2]+ r.Calls = append(r.Calls, call("c03", true, "{}", "traceway metrics query --name mem.used"))+ }, "none of these alternatives"},+ "thrash": {func(r *record.Run, s *catalog.Trajectory) {+ s.MaxIdenticalRepeats = 1+ r.Calls = append(r.Calls, call("c09", true, "{}", "traceway endpoints list --since 24h --output json"))+ }, "same call ran 2 times"},+ "too many failures": {func(r *record.Run, s *catalog.Trajectory) {+ s.MaxFailedCalls = 0+ s.MaxFailedCalls = 1+ r.Calls = append(r.Calls, call("c10", false, "x", "traceway logs query"))+ }, "failed tool calls"},+ }+ for name, c := range cases {+ t.Run(name, func(t *testing.T) {+ r := &record.Run{Calls: append([]*record.Call{}, run.Calls...)}+ s := *spec+ c.mutate(r, &s)+ v := Trajectory(r, &s)+ if v.Pass || !strings.Contains(failures(v), c.want) {+ t.Fatalf("want failure %q, got pass=%v:\n%s", c.want, v.Pass, failures(v))+ }+ })+ }+}++func perfInput(sub map[string]any, calls ...*record.Call) *Input {+ return &Input{+ Case: &catalog.Case{ID: "perf-x", Suite: "perf", Expect: catalog.Expect{+ Answer: &catalog.Answer{ChecklistItem: "select_n_plus_1", Not: []string{"over_fetching"}, Locus: &catalog.Locus{Endpoint: "GET /api/orders", SpanContains: "order_items"}},+ Grounding: []string{"FROM order_items WHERE order_id"},+ }},+ Run: &record.Run{Prompt: "why is GET /api/orders slow", Submission: sub, Calls: calls},+ }+}++func TestAnswer(t *testing.T) {+ good := map[string]any{"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "get /api/orders/", "span": "SELECT ... FROM order_items WHERE order_id = $1"}}+ if v := Answer(perfInput(good)); !v.Pass {+ t.Fatalf("good answer failed: %s", failures(v))+ }+ for name, sub := range map[string]map[string]any{+ "wrong row": {"checklist_item": "missing_or_unused_index", "locus": good["locus"]},+ "distractor": {"checklist_item": "over_fetching", "locus": good["locus"]},+ "wrong locus": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "GET /api/orders/:id", "span": "order_items"}},+ "no span": {"checklist_item": "select_n_plus_1", "locus": map[string]any{"endpoint": "GET /api/orders"}},+ "no submission": nil,+ } {+ if v := Answer(perfInput(sub)); v.Pass {+ t.Errorf("%s: should fail", name)+ }+ }+}++func TestGrounding(t *testing.T) {+ out := `{"spans":[{"attributes":{"db.query.text":"SELECT id FROM order_items WHERE order_id = $1"}}],"id":"6f9619ff-8b86-d011-b42d-00c04fc964ff"}`+ c1 := call("c01", true, out, "traceway endpoints show 6f9619ff-8b86-d011-b42d-00c04fc964ff --recorded-at 2026-10-17T10:00:00Z")+ sub := func(id string, extra string) map[string]any {+ return map[string]any{"evidence": []any{map[string]any{"call_id": id, "claim": "x"}}, "report": "request 6f9619ff-8b86-d011-b42d-00c04fc964ff " + extra}+ }+ if v := Grounding(perfInput(sub("c01", ""), c1)); !v.Pass {+ t.Fatalf("grounded answer failed: %s", failures(v))+ }+ tests := map[string]struct {+ in *Input+ want string+ }{+ "unknown call": {perfInput(sub("c07", ""), c1), "not a call"},+ "failed call": {perfInput(sub("c01", ""), call("c01", false, out, "traceway endpoints show x")), "which failed"},+ "truncated call": {perfInput(sub("c01", ""), &record.Call{ID: "c01", OK: true, Truncated: true, Output: out}), "truncated"},+ "invented id": {perfInput(sub("c01", "and trace 4bf92f3577b34da6a3ce929d0e0e4736"), c1), "no tool returned"},+ "missing key": {perfInput(sub("c02", ""), c1, call("c02", true, `{"data":[]}`, "traceway endpoints list")), "contains"},+ }+ for name, tc := range tests {+ t.Run(name, func(t *testing.T) {+ v := Grounding(tc.in)+ if v.Pass || !strings.Contains(failures(v), tc.want) {+ t.Fatalf("want %q, got pass=%v: %s", tc.want, v.Pass, failures(v))+ }+ })+ }+}++func TestLeakageLint(t *testing.T) {+ c := &catalog.Case{Prompt: "Why is checkout slow?", Expect: catalog.Expect{Answer: &catalog.Answer{ChecklistItem: "cache_stampede_thundering_herd"}}}+ if got := Lint(c, []string{"GET catalog:v3", "SELECT id FROM products WHERE id = $1"}); len(got) != 0 {+ t.Fatalf("clean data flagged: %v", got)+ }+ got := Lint(c, []string{"recompute after cache stampede", "N+1 queries detected"})+ if len(got) < 2 {+ t.Fatalf("leaks not flagged: %v", got)+ }+ c.Prompt = "Is this a cache stampede?"+ if got := Lint(c, nil); len(got) == 0 {+ t.Fatal("a prompt naming the answer must be flagged")+ }+}++func TestReport(t *testing.T) {+ in := &Input{Case: &catalog.Case{Expect: catalog.Expect{Report: &catalog.ReportSpec{Contains: []string{"snooz"}, NotContains: []string{"disabled"}}}},+ Run: &record.Run{Submission: map[string]any{"summary": "Snoozed rule 4 for 60 minutes."}}}+ if v := Report(in); !v.Pass {+ t.Fatal(failures(v))+ }+ in.Run.Submission["summary"] = "I disabled the rule."+ if v := Report(in); v.Pass {+ t.Fatal("must fail")+ }+}diff --git a/cli/test/evals/internal/grade/grounding.go b/cli/test/evals/internal/grade/grounding.gonew file mode 100644index 00000000..98a05ea2--- /dev/null+++ b/cli/test/evals/internal/grade/grounding.go@@ -0,0 +1,147 @@+package grade++import (+ "encoding/json"+ "regexp"+ "sort"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// Grounding checks that the evidence is real:+// - every evidence.call_id names a call that exists, succeeded and was+// not truncated;+// - every grounding key appears in the output of at least one cited call+// (what the model saw, or the raw traceway output inside that call);+// - every UUID, 32-hex trace id or 16-hex hash in the answer appears in+// some tool output or the prompt: no invented ids.+func Grounding(in *Input) *Verdict {+ v := newVerdict("grounding", true)+ run := in.Run+ sub := run.Submission+ if sub == nil {+ v.fail("no submission to ground")+ return v+ }+ var cited []*record.Call+ if ev, ok := sub["evidence"].([]any); ok {+ for i, e := range ev {+ m, _ := e.(map[string]any)+ id, _ := m["call_id"].(string)+ c := run.CallByID(strings.TrimSpace(id))+ switch {+ case c == nil:+ v.fail("evidence[%d] cites %q, which is not a call", i, id)+ case c.Rejected:+ v.fail("evidence[%d] cites %s, which was never executed", i, id)+ case !c.OK || !anyOpOK(c):+ v.fail("evidence[%d] cites %s, which failed", i, id)+ case c.Truncated:+ v.fail("evidence[%d] cites %s, whose output was truncated", i, id)+ default:+ cited = append(cited, c)+ }+ }+ } else if in.Case.Suite == "perf" || in.Case.Suite == "debug" {+ v.fail("the submission cites no evidence")+ }+ for _, key := range in.Case.Expect.Grounding {+ found := false+ for _, c := range cited {+ if callContains(c, key) {+ found = true+ break+ }+ }+ if !found {+ v.fail("no cited call's output contains %q", key)+ }+ }+ if invented := InventedIDs(run); len(invented) > 0 {+ v.fail("ids in the answer that no tool returned: %s", strings.Join(invented, ", "))+ }+ return v+}++// callContains looks for a grounding key in what the model saw and in the+// raw output of the call's successful operations. MCP results are JSON, which+// escapes &, < and > (\u0026, \u003c, \u003e), so those are compared+// unescaped.+func callContains(c *record.Call, key string) bool {+ if anyOpOK(c) && strings.Contains(unescapeJSON(c.Output), key) {+ return true+ }+ for _, op := range c.Ops {+ if op.OK && strings.Contains(unescapeJSON(op.Output), key) {+ return true+ }+ }+ return false+}++// anyOpOK reports whether a call has no operations (a plain shell command)+// or at least one that succeeded. A traceway command that failed inside a+// pipeline does not make the call evidence just because jq exited 0.+func anyOpOK(c *record.Call) bool {+ if len(c.Ops) == 0 {+ return true+ }+ for _, op := range c.Ops {+ if op.OK {+ return true+ }+ }+ return false+}++var jsonEscapes = strings.NewReplacer(`\u0026`, "&", `\u003c`, "<", `\u003e`, ">", `\u0027`, "'")++func unescapeJSON(s string) string {+ if !strings.Contains(s, `\u00`) {+ return s+ }+ return jsonEscapes.Replace(s)+}++var idRe = regexp.MustCompile(`(?i)\b(?:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}|[0-9a-f]{32}|[0-9a-f]{16})\b`)++// InventedIDs lists ids in the submission (and final text) that appear in no+// tool output and not in the prompt.+func InventedIDs(run *record.Run) []string {+ raw, _ := json.Marshal(run.Submission)+ text := string(raw) + "\n" + run.FinalText+ var corpus strings.Builder+ corpus.WriteString(strings.ToLower(run.Prompt))+ for _, c := range run.Calls {+ corpus.WriteString(strings.ToLower(c.Output))+ corpus.WriteString(strings.ToLower(string(c.Args)))+ for _, op := range c.Ops {+ corpus.WriteString(strings.ToLower(op.Output))+ }+ }+ all := corpus.String()+ seen := map[string]bool{}+ var out []string+ for _, id := range idRe.FindAllString(text, -1) {+ l := strings.ToLower(id)+ if seen[l] || isAllDigits(l) {+ continue+ }+ seen[l] = true+ if !strings.Contains(all, l) && !strings.Contains(all, strings.ReplaceAll(l, "-", "")) {+ out = append(out, id)+ }+ }+ sort.Strings(out)+ return out+}++func isAllDigits(s string) bool {+ for _, c := range s {+ if c < '0' || c > '9' {+ return false+ }+ }+ return true+}diff --git a/cli/test/evals/internal/grade/leakage.go b/cli/test/evals/internal/grade/leakage.gonew file mode 100644index 00000000..67aba577--- /dev/null+++ b/cli/test/evals/internal/grade/leakage.go@@ -0,0 +1,110 @@+package grade++import (+ "fmt"+ "regexp"+ "sort"+ "strings"+ "sync"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/checklist"+)++// Leakage lint (plan §4.8): the seeded data and the prompt must not name the+// answer. Seeded strings (span names, attribute values, log bodies, stack+// traces, object names) are checked against a denylist of checklist terms+// and against the expected checklist row's title; prompts get the checklist+// terms only, since a user naturally says "slow" or "bug".++// seedDenylist are phrases that give a diagnosis away when they appear in+// telemetry. Real library messages never contain them.+var seedDenylist = []string{+ "n+1", "n + 1", "nplus1", "n_plus_1", "select n", "stampede", "thundering herd", "missing index", "unused index",+ "no index", "pool exhaust", "exhausted pool", "exhaustion", "gc pause", "noisy neighbo", "cold start", "warm-up",+ "warmup", "lock contention", "contention", "over-fetch", "overfetch", "no pagination", "unbounded", "fan-out",+ "fanout", "chatty", "staircase", "parallelize", "run in parallel", "retry storm", "aggressive retr", "connection reuse",+ "large payload", "payload too large", "low hit ratio", "cpu saturat", "saturation", "memory pressure",+ "disk bottleneck", "io bottleneck", "quadratic", "o(n^2)", "o(n2)", "o(n²)", "background task", "should be async",+ "excessive logging", "log spam", "root cause", "bottleneck", "slow query", "culprit", "the cause", "deliberately",+ "bug here", "intentional", "eval", "seeded", "distractor",+}++// promptDenylist are checklist phrases a prompt must not contain.+var promptDenylist = []string{+ "n+1", "n + 1", "stampede", "thundering herd", "missing index", "pool exhaust", "exhaustion", "gc pause",+ "garbage collect", "noisy neighbo", "cold start", "lock contention", "over-fetch", "pagination", "fan-out",+ "staircase", "in parallel", "retries", "connection reuse", "hit ratio", "saturat", "memory pressure",+ "quadratic", "o(n", "background task", "excessive logging", "culprit", "distractor",+}++// Finding is one leak.+type Finding struct {+ Where string+ Term string+ Text string+}++func (f Finding) String() string {+ return fmt.Sprintf("%s contains %q: %s", f.Where, f.Term, clipS(f.Text, 160))+}++// Lint checks a case's seeded strings and prompt.+func Lint(c *catalog.Case, seeded []string) []Finding {+ var out []Finding+ seen := map[string]bool{}+ terms := append([]string{}, seedDenylist...)+ if c.Expect.Answer != nil && c.Expect.Answer.ChecklistItem != "" {+ for _, r := range checklist.Rows() {+ if r.ID == c.Expect.Answer.ChecklistItem {+ terms = append(terms, strings.ToLower(r.Title))+ }+ }+ }+ for _, s := range seeded {+ low := strings.ToLower(s)+ for _, t := range terms {+ if containsTerm(low, t) {+ k := t + "|" + s+ if !seen[k] {+ seen[k] = true+ out = append(out, Finding{Where: "seed", Term: t, Text: s})+ }+ }+ }+ }+ low := strings.ToLower(c.Prompt)+ for _, t := range promptDenylist {+ if containsTerm(low, t) {+ out = append(out, Finding{Where: "prompt", Term: t, Text: c.Prompt})+ }+ }+ sort.Slice(out, func(i, j int) bool { return out[i].String() < out[j].String() })+ return out+}++var wordCache sync.Map // term -> *regexp.Regexp++// containsTerm matches a term on word boundaries when it starts and ends+// with a letter, so "eval" does not match "evaluate" but does match "eval".+func containsTerm(s, term string) bool {+ if !strings.Contains(s, term) {+ return false+ }+ isWord := func(b byte) bool { return b >= 'a' && b <= 'z' }+ if !isWord(term[0]) || !isWord(term[len(term)-1]) {+ return true+ }+ re, ok := wordCache.Load(term)+ if !ok {+ re, _ = wordCache.LoadOrStore(term, regexp.MustCompile(`\b`+regexp.QuoteMeta(term)+`\b`))+ }+ return re.(*regexp.Regexp).MatchString(s)+}++func clipS(s string, n int) string {+ if len(s) > n {+ return s[:n] + "..."+ }+ return s+}diff --git a/cli/test/evals/internal/grade/trajectory.go b/cli/test/evals/internal/grade/trajectory.gonew file mode 100644index 00000000..1f9623d1--- /dev/null+++ b/cli/test/evals/internal/grade/trajectory.go@@ -0,0 +1,292 @@+package grade++import (+ "encoding/json"+ "fmt"+ "path"+ "regexp"+ "sort"+ "strconv"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// Trajectory checks the tool calls against the case's call DSL:+//+// required each matcher matches at least one operation+// any_of at least one group is fully satisfied+// forbidden no operation matches+// order the first match of A precedes the first match of B+// max_calls, max_failed_calls, max_identical_repeats thrash detection+//+// An op pattern is an exact op, "*", a prefix ("endpoints.*"), or a file op+// with a path glob ("edit_file:handlers/*.go").+func Trajectory(run *record.Run, t *catalog.Trajectory) *Verdict {+ v := newVerdict("trajectory", true)+ all := run.AllOps()+ for _, m := range t.Required {+ if firstMatch(all, m) < 0 {+ v.fail("required call missing: %s", describe(m))+ }+ }+ if len(t.AnyOf) > 0 {+ ok := false+ for _, group := range t.AnyOf {+ hit := true+ for _, m := range group {+ if firstMatch(all, m) < 0 {+ hit = false+ break+ }+ }+ if hit {+ ok = true+ break+ }+ }+ if !ok {+ var groups []string+ for _, g := range t.AnyOf {+ var parts []string+ for _, m := range g {+ parts = append(parts, describe(m))+ }+ groups = append(groups, "["+strings.Join(parts, " + ")+"]")+ }+ v.fail("none of these alternatives happened: %s", strings.Join(groups, " or "))+ }+ }+ for _, m := range t.Forbidden {+ if i := firstMatch(all, m); i >= 0 {+ v.fail("forbidden call %s happened in %s: %s", describe(m), all[i].Call.ID, opString(all[i].OpRecord))+ }+ }+ for _, pair := range t.Order {+ if len(pair) != 2 {+ v.fail("order entries need exactly two ops, got %v", pair)+ continue+ }+ a := firstMatch(all, catalog.CallMatcher{Op: pair[0]})+ b := firstMatch(all, catalog.CallMatcher{Op: pair[1]})+ switch {+ case b < 0:+ case a < 0:+ v.fail("%s happened without %s before it", pair[1], pair[0])+ case a > b:+ v.fail("%s (%s) came after %s (%s)", pair[0], all[a].Call.ID, pair[1], all[b].Call.ID)+ }+ }+ executed := 0+ failed := 0+ for _, c := range run.Calls {+ if c.Rejected || isSubmitTool(c.Tool) {+ continue+ }+ executed+++ if !c.OK || !anyOpOK(c) {+ failed+++ }+ }+ if t.MaxCalls > 0 && executed > t.MaxCalls {+ v.fail("%d tool calls, more than max_calls %d", executed, t.MaxCalls)+ }+ if t.MaxFailedCalls > 0 && failed > t.MaxFailedCalls {+ v.fail("%d failed tool calls, more than max_failed_calls %d", failed, t.MaxFailedCalls)+ }+ if t.MaxIdenticalRepeats > 0 {+ counts := map[string]int{}+ for _, op := range all {+ counts[opKey(op.OpRecord)]+++ }+ keys := make([]string, 0, len(counts))+ for k := range counts {+ keys = append(keys, k)+ }+ sort.Strings(keys)+ for _, k := range keys {+ if counts[k] > t.MaxIdenticalRepeats {+ v.fail("the same call ran %d times (max_identical_repeats %d): %s", counts[k], t.MaxIdenticalRepeats, k)+ }+ }+ }+ v.note("%d operations in %d calls (%d failed)", len(all), executed, failed)+ return v+}++func isSubmitTool(name string) bool {+ return name == "submit_perf_diagnosis" || name == "submit_debug_diagnosis" || name == "finish"+}++func firstMatch(all []record.CallOp, m catalog.CallMatcher) int {+ for i, op := range all {+ if matches(op.OpRecord, m) {+ return i+ }+ }+ return -1+}++// Matches reports whether an operation satisfies a matcher.+func Matches(op *record.OpRecord, m catalog.CallMatcher) bool { return matches(op, m) }++func matches(op *record.OpRecord, m catalog.CallMatcher) bool {+ pattern, glob, _ := strings.Cut(m.Op, ":")+ if !opPattern(pattern, op.Op.Op) {+ return false+ }+ if glob != "" {+ var target string+ if p, ok := op.Args["path"].(string); ok {+ target = p+ } else if n, ok := op.Args["name"].(string); ok {+ target = n+ }+ if ok, _ := path.Match(glob, target); !ok && !strings.HasSuffix(target, strings.TrimPrefix(glob, "*")) {+ return false+ }+ }+ if m.Write != nil && op.Write != *m.Write {+ return false+ }+ if m.OK != nil && op.OK != *m.OK {+ return false+ }+ for k, want := range m.Args {+ got, present := op.Args[k]+ if !argMatches(got, present, want) {+ return false+ }+ }+ return true+}++func opPattern(p, op string) bool {+ switch {+ case p == "*" || p == "":+ return true+ case strings.HasSuffix(p, ".*"):+ return strings.HasPrefix(op, strings.TrimSuffix(p, "*"))+ default:+ return p == op+ }+}++// argMatches implements the arg matchers: present, absent, ~regex, exact+// value, and {min, max}.+func argMatches(got any, present bool, want any) bool {+ switch w := want.(type) {+ case string:+ switch {+ case w == "present":+ return present && fmt.Sprint(got) != ""+ case w == "absent":+ return !present+ case strings.HasPrefix(w, "~"):+ re, err := regexp.Compile(w[1:])+ return err == nil && present && re.MatchString(argString(got))+ default:+ return present && argString(got) == w+ }+ case map[string]any:+ n, ok := toFloat(got)+ if !ok {+ return false+ }+ if lo, ok := toFloat(w["min"]); ok && n < lo {+ return false+ }+ if hi, ok := toFloat(w["max"]); ok && n > hi {+ return false+ }+ return present+ case bool:+ b, _ := strconv.ParseBool(argString(got))+ if gb, ok := got.(bool); ok {+ b = gb+ }+ return present && b == w+ default:+ return present && argString(got) == argString(w)+ }+}++func argString(v any) string {+ switch t := v.(type) {+ case nil:+ return ""+ case string:+ return t+ case float64:+ return strconv.FormatFloat(t, 'f', -1, 64)+ case []any:+ parts := make([]string, len(t))+ for i, x := range t {+ parts[i] = argString(x)+ }+ return strings.Join(parts, ",")+ case map[string]any:+ raw, _ := json.Marshal(t)+ return string(raw)+ default:+ return fmt.Sprint(t)+ }+}++func toFloat(v any) (float64, bool) {+ switch t := v.(type) {+ case float64:+ return t, true+ case int:+ return float64(t), true+ case string:+ f, err := strconv.ParseFloat(t, 64)+ return f, err == nil+ }+ return 0, false+}++func describe(m catalog.CallMatcher) string {+ s := m.Op+ if s == "" {+ s = "*"+ }+ var parts []string+ keys := make([]string, 0, len(m.Args))+ for k := range m.Args {+ keys = append(keys, k)+ }+ sort.Strings(keys)+ for _, k := range keys {+ parts = append(parts, fmt.Sprintf("%s=%v", k, m.Args[k]))+ }+ if m.Write != nil {+ parts = append(parts, fmt.Sprintf("write=%v", *m.Write))+ }+ if len(parts) > 0 {+ s += "(" + strings.Join(parts, ", ") + ")"+ }+ return s+}++func opString(op *record.OpRecord) string {+ if len(op.Argv) > 0 {+ return strings.Join(op.Argv, " ")+ }+ raw, _ := json.Marshal(op.Args)+ return op.Op.Op + " " + string(raw)+}++// opKey identifies a call for repeat detection, ignoring output-format flags.+func opKey(op *record.OpRecord) string {+ args := map[string]any{}+ for k, v := range op.Args {+ if k == "output" || k == "fields" {+ continue+ }+ args[k] = v+ }+ raw, _ := json.Marshal(args)+ return op.Op.Op + " " + string(raw)+}diff --git a/cli/test/evals/internal/ops/ops.go b/cli/test/evals/internal/ops/ops.gonew file mode 100644index 00000000..9be178f5--- /dev/null+++ b/cli/test/evals/internal/ops/ops.go@@ -0,0 +1,399 @@+// Package ops maps CLI invocations and MCP tool calls onto surface-neutral+// operations (endpoints.show, pages.ack, ...) so one case spec grades both.+package ops++import (+ "fmt"+ "regexp"+ "sort"+ "strings"+ "sync"++ "github.com/tracewayapp/traceway/cli/pkg/mcpserver/knowledge"+)++// An operation is the surface-neutral name of what a tool call did+// (endpoints.show, pages.ack, read_file, ...), so one case spec grades CLI and+// MCP runs alike. Args are keyed by the MCP parameter name, which is the CLI+// flag in snake_case except where Spec.Rename says otherwise.++// Spec describes one Traceway operation on both surfaces.+type Spec struct {+ Op string // endpoints.show+ CLI []string // command path after the binary: [endpoints show]+ MCP string // MCP tool name; empty for CLI-only commands+ Write bool+ // Positional names the CLI positional args in order; a trailing "..."+ // collects the rest into a list (exceptions archive <hash>...).+ Positional []string+ // Rename maps a snake_case CLI flag to its MCP parameter.+ Rename map[string]string+ // WriteEdition marks commands that exist only in traceway-write.+ WriteEdition bool+}++// specs is the static half of the operations map: positional names, renames+// and write flags. The CLI<->MCP pairing itself is parsed from tool-map.md and+// cross-checked against this table by TestOpsMatchToolMap.+var specs = []Spec{+ {Op: "projects.list", CLI: []string{"projects", "list"}, MCP: "list_projects"},+ {Op: "projects.use", CLI: []string{"projects", "use"}, Positional: []string{"project_id"}},+ {Op: "exceptions.list", CLI: []string{"exceptions", "list"}, MCP: "list_exceptions"},+ {Op: "exceptions.show", CLI: []string{"exceptions", "show"}, MCP: "get_exception", Positional: []string{"hash"}},+ {Op: "exceptions.occurrence", CLI: []string{"exceptions", "occurrence"}, MCP: "get_exception_occurrence", Positional: []string{"id"}},+ {Op: "logs.query", CLI: []string{"logs", "query"}, MCP: "query_logs", Rename: map[string]string{"service": "service_name"}},+ {Op: "endpoints.list", CLI: []string{"endpoints", "list"}, MCP: "list_endpoints"},+ {Op: "endpoints.show", CLI: []string{"endpoints", "show"}, MCP: "get_endpoint_request", Positional: []string{"id"}},+ {Op: "endpoints.chart", CLI: []string{"endpoints", "chart"}, MCP: "endpoints_chart"},+ {Op: "endpoints.slow", CLI: []string{"endpoints", "slow"}, MCP: "get_slow_endpoint_config", Positional: []string{"endpoint"}},+ {Op: "tasks.show", CLI: []string{"tasks", "show"}, MCP: "get_task", Positional: []string{"id"}},+ {Op: "ai_traces.show", CLI: []string{"ai-traces", "show"}, MCP: "get_ai_trace", Positional: []string{"id"}},+ {Op: "sessions.show", CLI: []string{"sessions", "show"}, MCP: "get_session", Positional: []string{"id"}},+ {Op: "traces.show", CLI: []string{"traces", "show"}, MCP: "get_trace", Positional: []string{"id"}},+ {Op: "metrics.query", CLI: []string{"metrics", "query"}, MCP: "query_metrics", Rename: map[string]string{"tag": "tag_filters"}},+ {Op: "pages.list", CLI: []string{"pages", "list"}, MCP: "list_pages"},+ {Op: "pages.show", CLI: []string{"pages", "show"}, MCP: "get_page", Positional: []string{"id"}},+ {Op: "alerts.list", CLI: []string{"alerts", "list"}, MCP: "list_notification_rules"},+ {Op: "monitors.list", CLI: []string{"monitors", "list"}, MCP: "list_checks"},+ {Op: "monitors.show", CLI: []string{"monitors", "show"}, MCP: "get_check", Positional: []string{"id"}},+ {Op: "post_mortems.list", CLI: []string{"post-mortems", "list"}, MCP: "list_post_mortems", Rename: map[string]string{"tag": "tags"}},+ {Op: "post_mortems.show", CLI: []string{"post-mortems", "show"}, MCP: "get_post_mortem", Positional: []string{"id"}},++ {Op: "exceptions.archive", CLI: []string{"exceptions", "archive"}, MCP: "archive_exceptions", Write: true, WriteEdition: true, Positional: []string{"hashes..."}},+ {Op: "exceptions.unarchive", CLI: []string{"exceptions", "unarchive"}, MCP: "unarchive_exceptions", Write: true, WriteEdition: true, Positional: []string{"hashes..."}},+ {Op: "pages.ack", CLI: []string{"pages", "ack"}, MCP: "acknowledge_page", Write: true, WriteEdition: true, Positional: []string{"id"}},+ {Op: "pages.resolve", CLI: []string{"pages", "resolve"}, MCP: "resolve_page", Write: true, WriteEdition: true, Positional: []string{"id"}},+ {Op: "alerts.enable", CLI: []string{"alerts", "enable"}, MCP: "set_notification_rule_enabled", Write: true, WriteEdition: true, Positional: []string{"id"}},+ {Op: "alerts.disable", CLI: []string{"alerts", "disable"}, MCP: "set_notification_rule_enabled", Write: true, WriteEdition: true, Positional: []string{"id"}},+ {Op: "alerts.snooze", CLI: []string{"alerts", "snooze"}, MCP: "snooze_notification_rule", Write: true, WriteEdition: true, Positional: []string{"id"}, Rename: map[string]string{"minutes": "duration_minutes"}},+ {Op: "monitors.run", CLI: []string{"monitors", "run"}, MCP: "run_check", Write: true, WriteEdition: true, Positional: []string{"id"}},+ {Op: "post_mortems.create", CLI: []string{"post-mortems", "create"}, MCP: "create_post_mortem", Write: true, WriteEdition: true, Rename: map[string]string{"tag": "tags", "incident": "incident_id", "content_file": "content_md"}},+ {Op: "post_mortems.update", CLI: []string{"post-mortems", "update"}, MCP: "update_post_mortem", Write: true, WriteEdition: true, Positional: []string{"id"}, Rename: map[string]string{"tag": "tags", "incident": "incident_id", "content_file": "content_md"}},+ {Op: "endpoints.mark_slow", CLI: []string{"endpoints", "mark-slow"}, MCP: "set_slow_endpoint", Write: true, WriteEdition: true, Positional: []string{"endpoint"}},++ // CLI-only commands. setup apply proposes a plan draft; nothing is+ // created until a person approves it, so it is not an agent write.+ {Op: "setup.apply", CLI: []string{"setup", "apply"}},+ {Op: "login", CLI: []string{"login"}},+ {Op: "logout", CLI: []string{"logout"}},+ {Op: "profiles.list", CLI: []string{"profiles", "list"}},+ {Op: "profiles.use", CLI: []string{"profiles", "use"}, Positional: []string{"profile"}},+ {Op: "version", CLI: []string{"version"}},+ {Op: "mcp.serve", CLI: []string{"mcp"}},+}++// Harness tool operations that are not Traceway commands.+const (+ OpReadFile = "read_file"+ OpListFiles = "list_files"+ OpSearch = "search"+ OpEditFile = "edit_file"+ OpWriteFile = "write_file"+ OpLoadSkill = "load_skill"+ OpAskUser = "ask_user"+ OpReadResource = "read_resource"+ OpListRes = "list_resources"+ OpShell = "shell"+ OpGit = "git"+ OpSubmit = "submit"+ OpUnknownCLI = "traceway.unknown"+)++// boolFlags lists CLI flags that take no value.+var boolFlags = map[string]bool{+ "yes": true, "include_archived": true, "archive_issue": true, "unlink_incident": true,+ "password": true, "password_stdin": true, "token_stdin": true, "no_browser": true,+ "help": true, "version": true,+}++var shortFlags = map[string]string{"o": "output", "h": "help", "v": "version"}++var (+ opsOnce sync.Once+ byOp map[string]*Spec+ byMCP map[string][]*Spec+ cliSorted []*Spec+)++func initOps() {+ opsOnce.Do(func() {+ byOp = map[string]*Spec{}+ byMCP = map[string][]*Spec{}+ for i := range specs {+ s := &specs[i]+ byOp[s.Op] = s+ if s.MCP != "" {+ byMCP[s.MCP] = append(byMCP[s.MCP], s)+ }+ cliSorted = append(cliSorted, s)+ }+ sort.SliceStable(cliSorted, func(i, j int) bool { return len(cliSorted[i].CLI) > len(cliSorted[j].CLI) })+ })+}++// Specs returns the operations table.+func Specs() []Spec { return specs }++// SpecFor returns an operation's spec.+func SpecFor(op string) (*Spec, bool) {+ initOps()+ s, ok := byOp[op]+ return s, ok+}++// IsWriteOp reports whether an op mutates Traceway state.+func IsWriteOp(op string) bool {+ s, ok := SpecFor(op)+ return ok && s.Write+}++// Op is one recorded operation.+type Op struct {+ Op string `json:"op"`+ Args map[string]any `json:"args,omitempty"`+ Write bool `json:"write,omitempty"`+}++// ParseCLI maps a traceway[-write] argv (without the binary) to an op.+func ParseCLI(argv []string) Op {+ initOps()+ var words []string+ flags := map[string]any{}+ for i := 0; i < len(argv); i++ {+ a := argv[i]+ if a == "--" {+ words = append(words, argv[i+1:]...)+ break+ }+ if strings.HasPrefix(a, "-") && len(a) > 1 {+ name, val, hasVal := strings.Cut(strings.TrimLeft(a, "-"), "=")+ if !strings.HasPrefix(a, "--") {+ if long, ok := shortFlags[name]; ok {+ name = long+ }+ }+ key := strings.ReplaceAll(name, "-", "_")+ if !hasVal && !boolFlags[key] && i+1 < len(argv) {+ val = argv[i+1]+ i+++ }+ if !hasVal && boolFlags[key] {+ addArg(flags, key, true)+ continue+ }+ addArg(flags, key, val)+ continue+ }+ words = append(words, a)+ }+ var spec *Spec+ for _, s := range cliSorted {+ if len(words) >= len(s.CLI) && equalWords(words[:len(s.CLI)], s.CLI) {+ spec = s+ break+ }+ }+ if spec == nil {+ return Op{Op: OpUnknownCLI, Args: map[string]any{"argv": strings.Join(argv, " ")}}+ }+ args := map[string]any{}+ for k, v := range flags {+ if r, ok := spec.Rename[k]; ok {+ k = r+ }+ if k == "project" {+ k = "project_id"+ }+ args[k] = v+ }+ pos := words[len(spec.CLI):]+ for i, name := range spec.Positional {+ if strings.HasSuffix(name, "...") {+ if i < len(pos) {+ list := make([]any, 0, len(pos)-i)+ for _, p := range pos[i:] {+ list = append(list, p)+ }+ args[strings.TrimSuffix(name, "...")] = list+ }+ break+ }+ if i < len(pos) {+ args[name] = pos[i]+ }+ }+ return Op{Op: spec.Op, Args: args, Write: spec.Write}+}++func addArg(m map[string]any, k string, v any) {+ prev, ok := m[k]+ if !ok {+ m[k] = v+ return+ }+ switch p := prev.(type) {+ case []any:+ m[k] = append(p, v)+ default:+ m[k] = []any{p, v}+ }+}++func equalWords(a, b []string) bool {+ for i := range a {+ if a[i] != b[i] {+ return false+ }+ }+ return true+}++// ToolMapPair is one row of tool-map.md.+type ToolMapPair struct {+ CLI []string+ MCP string+}++var (+ toolMapRowRe = regexp.MustCompile("^\\|\\s*(`[^|]+`(?:\\s*/\\s*`[^|]+`)*)\\s*\\|\\s*(`[^|]+`(?:\\s*/\\s*`[^|]+`)*)\\s*\\|")+ codeRe = regexp.MustCompile("`([^`]+)`")+)++// ParseToolMap reads the CLI->MCP pairs from tool-map.md. A row like+// "`pages list` / `pages show <id>` | `list_pages` / `get_page`" yields two+// pairs. Commands that elide the binary and noun ("`unarchive`") inherit them+// from the previous command in the cell.+func ParseToolMap() []ToolMapPair {+ var out []ToolMapPair+ for _, line := range strings.Split(knowledge.MustRead("tool-map.md"), "\n") {+ m := toolMapRowRe.FindStringSubmatch(line)+ if m == nil {+ continue+ }+ clis := codeRe.FindAllStringSubmatch(m[1], -1)+ tools := codeRe.FindAllStringSubmatch(m[2], -1)+ var prev []string+ var cmds [][]string+ for _, c := range clis {+ words := commandWords(c[1])+ if len(words) == 0 {+ continue+ }+ if len(words) == 1 && len(prev) >= 1 {+ words = append(append([]string{}, prev[:len(prev)-1]...), words[0])+ } else if len(words) >= 1 && words[0] != prev0(prev) && len(prev) > 0 && !isNoun(words[0]) {+ words = append([]string{prev[0]}, words...)+ }+ prev = words+ cmds = append(cmds, words)+ }+ for i, c := range cmds {+ t := tools[len(tools)-1][1]+ if i < len(tools) {+ t = tools[i][1]+ }+ out = append(out, ToolMapPair{CLI: c, MCP: t})+ }+ }+ return out+}++func prev0(p []string) string {+ if len(p) == 0 {+ return ""+ }+ return p[0]+}++var nouns = map[string]bool{+ "projects": true, "exceptions": true, "logs": true, "endpoints": true, "tasks": true, "ai-traces": true,+ "sessions": true, "traces": true, "metrics": true, "pages": true, "alerts": true, "monitors": true, "post-mortems": true,+}++func isNoun(w string) bool { return nouns[w] }++// commandWords strips the binary name, flags and placeholders from a code span.+func commandWords(s string) []string {+ var out []string+ for _, w := range strings.Fields(s) {+ if w == "traceway" || w == "traceway-write" {+ continue+ }+ if strings.HasPrefix(w, "-") || strings.HasPrefix(w, "<") || w == "N" {+ break+ }+ out = append(out, w)+ }+ return out+}++// RenderCLI renders an op back into a traceway argv (used by the fake model+// to play surface-neutral oracle steps on CLI arms).+func RenderCLI(op string, args map[string]any) ([]string, error) {+ spec, ok := SpecFor(op)+ if !ok {+ return nil, fmt.Errorf("unknown op %q", op)+ }+ argv := append([]string{}, spec.CLI...)+ used := map[string]bool{}+ for _, name := range spec.Positional {+ key := strings.TrimSuffix(name, "...")+ v, ok := args[key]+ if !ok {+ continue+ }+ used[key] = true+ if list, ok := v.([]any); ok {+ for _, x := range list {+ argv = append(argv, fmt.Sprint(x))+ }+ } else {+ argv = append(argv, fmt.Sprint(v))+ }+ }+ reverse := map[string]string{}+ for cli, mcp := range spec.Rename {+ reverse[mcp] = cli+ }+ keys := make([]string, 0, len(args))+ for k := range args {+ keys = append(keys, k)+ }+ sort.Strings(keys)+ for _, k := range keys {+ if used[k] {+ continue+ }+ v := args[k]+ flag := k+ if c, ok := reverse[k]; ok {+ flag = c+ }+ if k == "project_id" {+ flag = "project"+ }+ flag = "--" + strings.ReplaceAll(flag, "_", "-")+ switch t := v.(type) {+ case bool:+ if t {+ argv = append(argv, flag)+ }+ case []any:+ for _, x := range t {+ argv = append(argv, flag, fmt.Sprint(x))+ }+ case map[string]any:+ mk := make([]string, 0, len(t))+ for kk := range t {+ mk = append(mk, kk)+ }+ sort.Strings(mk)+ for _, kk := range mk {+ argv = append(argv, flag, kk+"="+fmt.Sprint(t[kk]))+ }+ default:+ argv = append(argv, flag, fmt.Sprint(v))+ }+ }+ return argv, nil+}diff --git a/cli/test/evals/internal/ops/ops_test.go b/cli/test/evals/internal/ops/ops_test.gonew file mode 100644index 00000000..e4f6a7a9--- /dev/null+++ b/cli/test/evals/internal/ops/ops_test.go@@ -0,0 +1,117 @@+package ops++import (+ "os/exec"+ "reflect"+ "strings"+ "testing"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+)++// TestOpsMatchToolMap cross-checks the static table against tool-map.md,+// the published CLI-to-MCP translation table.+func TestOpsMatchToolMap(t *testing.T) {+ pairs := ParseToolMap()+ if len(pairs) < 25 {+ t.Fatalf("parsed only %d rows from tool-map.md", len(pairs))+ }+ for _, p := range pairs {+ op := ParseCLI(p.CLI)+ spec, ok := SpecFor(op.Op)+ if !ok {+ t.Errorf("tool-map.md row %v -> %s maps to no op (%s)", p.CLI, p.MCP, op.Op)+ continue+ }+ if spec.MCP != p.MCP {+ t.Errorf("%v: tool-map.md says %s, the ops table says %s", p.CLI, p.MCP, spec.MCP)+ }+ }+}++// TestEveryCLICommandMaps walks both built editions' help: every leaf+// command must map to an op, and write-edition commands must be writes.+func TestEveryCLICommandMaps(t *testing.T) {+ bins := env.BinariesForTest(t)+ for _, bin := range []string{bins.CLI, bins.CLIWrite} {+ for _, path := range leafCommands(t, bin, nil) {+ op := ParseCLI(path)+ if op.Op == OpUnknownCLI {+ t.Errorf("%s %v maps to no op", bin, path)+ }+ }+ }+}++func leafCommands(t *testing.T, bin string, prefix []string) [][]string {+ out, _ := exec.Command(bin, append(append([]string{}, prefix...), "--help")...).CombinedOutput()+ var subs []string+ in := false+ for _, line := range strings.Split(string(out), "\n") {+ if strings.HasPrefix(line, "Available Commands:") {+ in = true+ continue+ }+ if in {+ if strings.TrimSpace(line) == "" {+ break+ }+ name := strings.Fields(line)[0]+ if name != "help" && name != "completion" {+ subs = append(subs, name)+ }+ }+ }+ if len(subs) == 0 {+ if len(prefix) == 0 {+ t.Fatalf("%s --help listed no commands", bin)+ }+ return [][]string{prefix}+ }+ var leaves [][]string+ for _, s := range subs {+ leaves = append(leaves, leafCommands(t, bin, append(append([]string{}, prefix...), s))...)+ }+ return leaves+}++func TestParseCLI(t *testing.T) {+ cases := []struct {+ argv string+ want Op+ }{+ {"endpoints show 7e3 --recorded-at 2026-10-17T10:00:00Z -o json", Op{Op: "endpoints.show", Args: map[string]any{"id": "7e3", "recorded_at": "2026-10-17T10:00:00Z", "output": "json"}}},+ {"exceptions archive aa bb --yes", Op{Op: "exceptions.archive", Args: map[string]any{"hashes": []any{"aa", "bb"}, "yes": true}, Write: true}},+ {"logs query --service api --min-severity=17", Op{Op: "logs.query", Args: map[string]any{"service_name": "api", "min_severity": "17"}}},+ {"metrics query --name x --tag a=b --tag c=d", Op{Op: "metrics.query", Args: map[string]any{"name": "x", "tag_filters": []any{"a=b", "c=d"}}}},+ {"endpoints mark-slow 'GET /x' --offset-ms 200", Op{Op: "endpoints.mark_slow", Args: map[string]any{"endpoint": "'GET", "offset_ms": "200"}, Write: true}},+ {"post-mortems update 4 --content-file - --expected-updated-at T", Op{Op: "post_mortems.update", Args: map[string]any{"id": "4", "content_md": "-", "expected_updated_at": "T"}, Write: true}},+ }+ for _, c := range cases {+ got := ParseCLI(strings.Fields(c.argv))+ if !reflect.DeepEqual(got, c.want) {+ t.Errorf("%s:\n got %+v\nwant %+v", c.argv, got, c.want)+ }+ }+}++func TestRenderCLI(t *testing.T) {+ argv, err := RenderCLI("alerts.snooze", map[string]any{"id": 12, "duration_minutes": 60})+ if err != nil || strings.Join(argv, " ") != "alerts snooze 12 --minutes 60" {+ t.Fatalf("RenderCLI = %v, %v", argv, err)+ }+}++func TestInvocationsUnquote(t *testing.T) {+ inv, err := Invocations(`traceway logs query --search "/api/orders " --since 6h | jq -r '.data[] | .body'; echo a\ b`)+ if err != nil {+ t.Fatal(err)+ }+ if len(inv) != 3 || inv[0][4] != "/api/orders " || inv[1][2] != ".data[] | .body" || inv[2][1] != "a b" {+ t.Fatalf("%q", inv)+ }+ op := ParseCLI(inv[0][1:])+ if op.Args["search"] != "/api/orders " {+ t.Fatalf("%+v", op)+ }+}diff --git a/cli/test/evals/internal/ops/words.go b/cli/test/evals/internal/ops/words.gonew file mode 100644index 00000000..36af9c0a--- /dev/null+++ b/cli/test/evals/internal/ops/words.go@@ -0,0 +1,81 @@+package ops++import (+ "bytes"+ "strings"++ "mvdan.cc/sh/v3/syntax"+)++// Invocations parses a shell command line and returns the words of every+// simple command in it, with quoting removed ("a b" and 'a b' become a b).+// Words that need expansion ($VAR, $(...)) keep their source text.+func Invocations(cmdline string) ([][]string, error) {+ file, err := syntax.NewParser().Parse(strings.NewReader(cmdline), "")+ if err != nil {+ return nil, err+ }+ var out [][]string+ syntax.Walk(file, func(n syntax.Node) bool {+ ce, ok := n.(*syntax.CallExpr)+ if !ok || len(ce.Args) == 0 {+ return true+ }+ words := make([]string, 0, len(ce.Args))+ for _, w := range ce.Args {+ words = append(words, WordText(w))+ }+ out = append(out, words)+ return true+ })+ return out, nil+}++// WordText is a word's value with quotes and backslash escapes removed.+func WordText(w *syntax.Word) string {+ var b strings.Builder+ for _, p := range w.Parts {+ b.WriteString(partText(p, false))+ }+ return b.String()+}++func partText(p syntax.WordPart, inDouble bool) string {+ switch t := p.(type) {+ case *syntax.Lit:+ return unescape(t.Value, inDouble)+ case *syntax.SglQuoted:+ return t.Value+ case *syntax.DblQuoted:+ var b strings.Builder+ for _, q := range t.Parts {+ b.WriteString(partText(q, true))+ }+ return b.String()+ default:+ var buf bytes.Buffer+ _ = syntax.NewPrinter().Print(&buf, p)+ return buf.String()+ }+}++func unescape(s string, inDouble bool) string {+ if !strings.Contains(s, `\`) {+ return s+ }+ var b strings.Builder+ for i := 0; i < len(s); i++ {+ if s[i] == '\\' && i+1 < len(s) {+ next := s[i+1]+ if !inDouble || strings.IndexByte("\"\\$`\n", next) >= 0 {+ if next != '\n' {+ b.WriteByte(next)+ }+ i+++ continue+ }+ }+ b.WriteByte(s[i])+ }+ return b.String()+}diff --git a/cli/test/evals/internal/record/record.go b/cli/test/evals/internal/record/record.gonew file mode 100644index 00000000..97d2e9da--- /dev/null+++ b/cli/test/evals/internal/record/record.go@@ -0,0 +1,113 @@+// Package record holds what one eval run produced: every tool call with the+// operations it performed, the final answer, and usage. The agent loop writes+// it, the graders read it, and transcript.jsonl is its serialized form.+package record++import (+ "encoding/json"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+)++// OpRecord is one operation inside a tool call: a traceway invocation in a+// shell pipeline, an MCP tool call, a file read, ...+type OpRecord struct {+ ops.Op+ Argv []string `json:"argv,omitempty"`+ Exit int `json:"exit"`+ OK bool `json:"ok"`+ Output string `json:"output,omitempty"` // raw output before any pipe+ Truncated bool `json:"truncated,omitempty"`+}++// Call is one tool call the model made. Its ID ("c07") is what the model+// cites as evidence.+type Call struct {+ ID string `json:"id"`+ Turn int `json:"turn"`+ Tool string `json:"tool"`+ Args json.RawMessage `json:"args"`+ Output string `json:"output"` // what the model saw+ OK bool `json:"ok"`+ Truncated bool `json:"truncated,omitempty"`+ Ops []OpRecord `json:"ops,omitempty"`+ DurationMs int64 `json:"durationMs"`+ Rejected bool `json:"rejected,omitempty"` // schema/budget refusal, not executed+}++// Usage is token and cost accounting for one model response.+type Usage struct {+ InputTokens int `json:"inputTokens"`+ CachedTokens int `json:"cachedTokens"`+ CacheWriteTokens int `json:"cacheWriteTokens"`+ OutputTokens int `json:"outputTokens"`+ ReasoningTokens int `json:"reasoningTokens"`+ CostUSD float64 `json:"costUsd"`+}++// Add accumulates u2 into u.+func (u *Usage) Add(u2 Usage) {+ u.InputTokens += u2.InputTokens+ u.CachedTokens += u2.CachedTokens+ u.CacheWriteTokens += u2.CacheWriteTokens+ u.OutputTokens += u2.OutputTokens+ u.ReasoningTokens += u2.ReasoningTokens+ u.CostUSD += u2.CostUSD+}++// Question is one ask_user exchange.+type Question struct {+ CallID string `json:"callId"`+ Question string `json:"question"`+ Answer string `json:"answer"`+}++// Run is the complete record of one agent run.+type Run struct {+ Calls []*Call `json:"calls"`+ Submission map[string]any `json:"submission,omitempty"`+ SubmitTool string `json:"submitTool,omitempty"`+ SubmitCall string `json:"submitCall,omitempty"`+ FinalText string `json:"finalText,omitempty"`+ Questions []Question `json:"questions,omitempty"`+ Usage Usage `json:"usage"`+ Turns int `json:"turns"`+ StopReason string `json:"stopReason"` // submitted | budget_exhausted | no_submit | infra_error+ StopDetail string `json:"stopDetail,omitempty"`+ SkillLoaded []string `json:"skillLoaded,omitempty"`+ Providers []string `json:"providers,omitempty"`+ Prompt string `json:"prompt"`+ Started time.Time `json:"started"`+ Elapsed time.Duration `json:"elapsed"`+}++// AllOps flattens every operation in call order, each with its call id.+func (r *Run) AllOps() []CallOp {+ var out []CallOp+ for _, c := range r.Calls {+ if c.Rejected {+ continue+ }+ for i := range c.Ops {+ out = append(out, CallOp{Call: c, OpRecord: &c.Ops[i]})+ }+ }+ return out+}++// CallOp pairs an operation with the call that performed it.+type CallOp struct {+ Call *Call+ *OpRecord+}++// Call returns the call with id, or nil.+func (r *Run) CallByID(id string) *Call {+ for _, c := range r.Calls {+ if c.ID == id {+ return c+ }+ }+ return nil+}diff --git a/cli/test/evals/internal/report/aggregate.go b/cli/test/evals/internal/report/aggregate.gonew file mode 100644index 00000000..1653f308--- /dev/null+++ b/cli/test/evals/internal/report/aggregate.go@@ -0,0 +1,228 @@+package report++import (+ "sort"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"+)++// Rate is a pass count.+type Rate struct {+ Pass int `json:"pass"`+ Total int `json:"total"`+ Rate float64 `json:"rate"`+}++func (r *Rate) add(pass bool) {+ r.Total+++ if pass {+ r.Pass+++ }+ r.Rate = float64(r.Pass) / float64(r.Total)+}++// Delta compares two arms or models.+type Delta struct {+ Suite string `json:"suite"`+ Model string `json:"model,omitempty"`+ A string `json:"a"`+ B string `json:"b"`+ RateA float64 `json:"rateA"`+ RateB float64 `json:"rateB"`+ Delta float64 `json:"delta"` // A - B in percentage points+ N int `json:"n"`+}++// Costs aggregates spend on passing runs.+type Costs struct {+ PassingRuns int `json:"passingRuns"`+ AvgCostUSD float64 `json:"avgCostUsd"`+ AvgToolCalls float64 `json:"avgToolCalls"`+ AvgInputTokens float64 `json:"avgInputTokens"`+ AvgOutputTokens float64 `json:"avgOutputTokens"`+ TotalCostUSD float64 `json:"totalCostUsd"`+ CacheHitRate float64 `json:"cacheHitRate"`+ AllRunsAvgCostUSD float64 `json:"allRunsAvgCostUsd"`+ AllRuns int `json:"allRuns"`+}++// Aggregates is the summary block of results.json.+type Aggregates struct {+ // Cases is keyed "case|model|arm".+ Cases map[string]*Rate `json:"cases"`+ // Suites is keyed "suite|model|arm".+ Suites map[string]*Rate `json:"suites"`+ SkillUplift []Delta `json:"skillUplift"`+ Models []Delta `json:"models"`+ Cost map[string]*Costs `json:"cost"` // by model+ SpendUSD float64 `json:"spendUsd"`+ JudgeUSD float64 `json:"judgeUsd"`+ Statuses map[string]int `json:"statuses"`+ TopFailing []GraderCount `json:"topFailingGraders"`+}++// GraderCount counts failing graders.+type GraderCount struct {+ Grader string `json:"grader"`+ Count int `json:"count"`+}++// Counted reports whether a run counts toward pass rates: infra errors are+// excluded, budget exhaustion is a failure.+func Counted(r *runner.Result) bool {+ return r.Status != "infra_error"+}++// Aggregate computes pass rates and deltas.+func Aggregate(runs []*runner.Result) Aggregates {+ a := Aggregates{Cases: map[string]*Rate{}, Suites: map[string]*Rate{}, Cost: map[string]*Costs{}, Statuses: map[string]int{}}+ graderFails := map[string]int{}+ cacheIn, cacheHit := map[string]int{}, map[string]int{}+ for _, r := range runs {+ a.Statuses[r.Status]+++ a.SpendUSD += r.Metrics.CostUSD + r.Metrics.JudgeCostUSD+ a.JudgeUSD += r.Metrics.JudgeCostUSD+ c := a.Cost[r.Model]+ if c == nil {+ c = &Costs{}+ a.Cost[r.Model] = c+ }+ c.AllRuns+++ c.AllRunsAvgCostUSD += r.Metrics.CostUSD+ c.TotalCostUSD += r.Metrics.CostUSD+ cacheIn[r.Model] += r.Metrics.InputTokens+ cacheHit[r.Model] += r.Metrics.CachedTokens+ if !Counted(r) {+ continue+ }+ pass := r.Status == "pass"+ key := r.Case + "|" + r.Model + "|" + r.Arm+ if a.Cases[key] == nil {+ a.Cases[key] = &Rate{}+ }+ a.Cases[key].add(pass)+ skey := r.Suite + "|" + r.Model + "|" + r.Arm+ if a.Suites[skey] == nil {+ a.Suites[skey] = &Rate{}+ }+ a.Suites[skey].add(pass)+ if pass {+ c.PassingRuns+++ c.AvgCostUSD += r.Metrics.CostUSD+ c.AvgToolCalls += float64(r.Metrics.ToolCalls)+ c.AvgInputTokens += float64(r.Metrics.InputTokens)+ c.AvgOutputTokens += float64(r.Metrics.OutputTokens)+ }+ for _, g := range r.Graders {+ if g.Gating && !g.Skipped && !g.Pass {+ graderFails[g.Grader]+++ }+ }+ }+ for m, c := range a.Cost {+ if c.PassingRuns > 0 {+ n := float64(c.PassingRuns)+ c.AvgCostUSD /= n+ c.AvgToolCalls /= n+ c.AvgInputTokens /= n+ c.AvgOutputTokens /= n+ }+ if c.AllRuns > 0 {+ c.AllRunsAvgCostUSD /= float64(c.AllRuns)+ }+ if cacheIn[m] > 0 {+ c.CacheHitRate = float64(cacheHit[m]) / float64(cacheIn[m])+ }+ }+ for g, n := range graderFails {+ a.TopFailing = append(a.TopFailing, GraderCount{g, n})+ }+ sort.Slice(a.TopFailing, func(i, j int) bool { return a.TopFailing[i].Count > a.TopFailing[j].Count })++ // Deltas are computed on cases run on both sides, so a partial+ // selection does not compare different case sets.+ pair := func(suite, model, armA, armB string) (Delta, bool) {+ var ra, rb Rate+ for key, r := range a.Cases {+ cs, m, arm := split3(key)+ if m != model || suiteOf(cs) != suite || arm != armA {+ continue+ }+ other := a.Cases[cs+"|"+model+"|"+armB]+ if other == nil {+ continue+ }+ ra.Pass, ra.Total = ra.Pass+r.Pass, ra.Total+r.Total+ rb.Pass, rb.Total = rb.Pass+other.Pass, rb.Total+other.Total+ }+ if ra.Total == 0 || rb.Total == 0 {+ return Delta{}, false+ }+ pa, pb := float64(ra.Pass)/float64(ra.Total), float64(rb.Pass)/float64(rb.Total)+ return Delta{Suite: suite, Model: model, A: armA, B: armB, RateA: pa, RateB: pb, Delta: (pa - pb) * 100, N: ra.Total}, true+ }+ models := map[string]bool{}+ for _, r := range runs {+ models[r.Model] = true+ }+ var modelList []string+ for m := range models {+ modelList = append(modelList, m)+ }+ sort.Strings(modelList)+ for _, s := range catalog.Suites {+ for _, m := range modelList {+ if d, ok := pair(s, m, catalog.ArmCLISkill, catalog.ArmCLINoSkill); ok {+ a.SkillUplift = append(a.SkillUplift, d)+ }+ }+ // Model vs model on each arm, over the cases both ran.+ for i := 0; i < len(modelList); i++ {+ for j := i + 1; j < len(modelList); j++ {+ for _, arm := range catalog.AllArms {+ var ra, rb Rate+ for key, r := range a.Cases {+ cs, m, ar := split3(key)+ if m != modelList[i] || ar != arm || suiteOf(cs) != s {+ continue+ }+ if o := a.Cases[cs+"|"+modelList[j]+"|"+arm]; o != nil {+ ra.Pass, ra.Total = ra.Pass+r.Pass, ra.Total+r.Total+ rb.Pass, rb.Total = rb.Pass+o.Pass, rb.Total+o.Total+ }+ }+ if ra.Total > 0 && rb.Total > 0 {+ pa, pb := float64(ra.Pass)/float64(ra.Total), float64(rb.Pass)/float64(rb.Total)+ a.Models = append(a.Models, Delta{Suite: s, A: modelList[i] + " " + arm, B: modelList[j] + " " + arm, RateA: pa, RateB: pb, Delta: (pa - pb) * 100, N: ra.Total})+ }+ }+ }+ }+ }+ return a+}++func split3(k string) (string, string, string) {+ var parts [3]string+ i := 0+ start := 0+ for p := 0; p < len(k) && i < 2; p++ {+ if k[p] == '|' {+ parts[i] = k[start:p]+ i+++ start = p + 1+ }+ }+ parts[2] = k[start:]+ return parts[0], parts[1], parts[2]+}++func suiteOf(caseID string) string {+ for _, s := range catalog.Suites {+ if len(caseID) > len(s) && caseID[:len(s)+1] == s+"-" {+ return s+ }+ }+ return ""+}diff --git a/cli/test/evals/internal/report/markdown.go b/cli/test/evals/internal/report/markdown.gonew file mode 100644index 00000000..4cd3d881--- /dev/null+++ b/cli/test/evals/internal/report/markdown.go@@ -0,0 +1,176 @@+package report++import (+ "fmt"+ "sort"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"+)++// Markdown renders summary.md.+func Markdown(r *Results) string {+ a := r.Aggregates+ var b strings.Builder+ h := r.Header+ title := "Traceway evals"+ if h.Oracle {+ title += " (oracle, fake model, $0)"+ }+ fmt.Fprintf(&b, "# %s\n\n", title)+ fmt.Fprintf(&b, "- **Run:** `%s`, started by %s\n", h.RunID, orDash(h.StartedBy))+ sha := h.GitSHA+ if len(sha) > 12 {+ sha = sha[:12]+ }+ dirty := ""+ if h.GitDirty {+ dirty = " (uncommitted changes)"+ }+ fmt.Fprintf(&b, "- **Commit:** `%s`%s, skill `traceway` %s, knowledge %s\n", orDash(sha), dirty, short(h.SkillHashes["traceway"]), short(h.KnowledgeHash))+ var models []string+ for k, v := range h.Models {+ models = append(models, fmt.Sprintf("%s=`%s`", k, v))+ }+ sort.Strings(models)+ fmt.Fprintf(&b, "- **Models:** %s\n", strings.Join(models, ", "))+ fmt.Fprintf(&b, "- **Spend:** $%.2f (judge $%.2f) of a $%.2f cap\n", a.SpendUSD, a.JudgeUSD, h.MaxCostUSD)+ var st []string+ for k, v := range a.Statuses {+ st = append(st, fmt.Sprintf("%s %d", k, v))+ }+ sort.Strings(st)+ fmt.Fprintf(&b, "- **Runs:** %d (%s)\n\n", len(r.Runs), strings.Join(st, ", "))++ b.WriteString("## Pass rate by suite\n\n| Suite | Model | Arm | Pass | Rate |\n|---|---|---|---|---|\n")+ keys := sortedKeys(a.Suites)+ for _, k := range keys {+ s, m, arm := split3(k)+ v := a.Suites[k]+ fmt.Fprintf(&b, "| %s | %s | %s | %d/%d | %.0f%% |\n", s, m, arm, v.Pass, v.Total, v.Rate*100)+ }+ if len(a.SkillUplift) > 0 {+ b.WriteString("\n## Skill uplift (cli+skill minus cli-noskill)\n\n| Suite | Model | With skill | Without | Uplift |\n|---|---|---|---|---|\n")+ for _, d := range a.SkillUplift {+ fmt.Fprintf(&b, "| %s | %s | %.0f%% | %.0f%% | %+.0fpp |\n", d.Suite, d.Model, d.RateA*100, d.RateB*100, d.Delta)+ }+ }+ if len(a.Models) > 0 {+ b.WriteString("\n## Model vs model (same cases)\n\n| Suite | A | B | A rate | B rate | Delta |\n|---|---|---|---|---|---|\n")+ for _, d := range a.Models {+ fmt.Fprintf(&b, "| %s | %s | %s | %.0f%% | %.0f%% | %+.0fpp |\n", d.Suite, d.A, d.B, d.RateA*100, d.RateB*100, d.Delta)+ }+ }+ b.WriteString("\n## Cost (averages over passing runs)\n\n| Model | Passing runs | Avg cost | Avg tool calls | Avg input tokens | Avg output tokens | Cache hit rate | Total spend |\n|---|---|---|---|---|---|---|---|\n")+ for _, m := range sortedKeys(a.Cost) {+ c := a.Cost[m]+ fmt.Fprintf(&b, "| %s | %d | $%.3f | %.1f | %.0f | %.0f | %.0f%% | $%.2f |\n", m, c.PassingRuns, c.AvgCostUSD, c.AvgToolCalls, c.AvgInputTokens, c.AvgOutputTokens, c.CacheHitRate*100, c.TotalCostUSD)+ }+ bySuiteCost := map[string]float64{}+ for _, run := range r.Runs {+ bySuiteCost[run.Suite+" / "+run.Model] += run.Metrics.CostUSD+ }+ if len(bySuiteCost) > 0 {+ b.WriteString("\n| Suite / model | Spend |\n|---|---|\n")+ for _, k := range sortedKeys(bySuiteCost) {+ fmt.Fprintf(&b, "| %s | $%.2f |\n", k, bySuiteCost[k])+ }+ }+ if len(a.TopFailing) > 0 {+ b.WriteString("\n## Top failing graders\n\n")+ for _, g := range a.TopFailing {+ fmt.Fprintf(&b, "- %s: %d\n", g.Grader, g.Count)+ }+ }+ worst := worstCases(a)+ if len(worst) > 0 {+ b.WriteString("\n## Worst cases\n\n| Case | Model | Arm | Pass |\n|---|---|---|---|\n")+ for _, k := range worst {+ cs, m, arm := split3(k)+ v := a.Cases[k]+ fmt.Fprintf(&b, "| %s | %s | %s | %d/%d |\n", cs, m, arm, v.Pass, v.Total)+ }+ }+ b.WriteString("\n## Runs\n\n| Case | Arm | Model | Run | Status | Calls | Cost | First failure |\n|---|---|---|---|---|---|---|---|\n")+ for _, run := range r.Runs {+ fmt.Fprintf(&b, "| %s | %s | %s | %d | %s | %d | $%.3f | %s |\n", run.Case, run.Arm, run.Model, run.RunIdx, run.Status, run.Metrics.ToolCalls, run.Metrics.CostUSD, mdEscape(firstFailure(run)))+ }+ b.WriteString("\nTranscripts are in the artifact under `runs/<case>/<arm>/<model>/r<N>/transcript.jsonl`.\n")+ return b.String()+}++func firstFailure(r *runner.Result) string {+ if r.Error != "" {+ return clip(r.Error, 140)+ }+ for _, g := range r.Graders {+ if g.Gating && !g.Skipped && !g.Pass && len(g.Failures) > 0 {+ return g.Grader + ": " + clip(g.Failures[0], 140)+ }+ }+ return ""+}++func worstCases(a Aggregates) []string {+ var keys []string+ for k, v := range a.Cases {+ if v.Rate < 1 {+ keys = append(keys, k)+ }+ }+ sort.Slice(keys, func(i, j int) bool {+ if a.Cases[keys[i]].Rate != a.Cases[keys[j]].Rate {+ return a.Cases[keys[i]].Rate < a.Cases[keys[j]].Rate+ }+ return keys[i] < keys[j]+ })+ if len(keys) > 15 {+ keys = keys[:15]+ }+ return keys+}++func sortedKeys[V any](m map[string]V) []string {+ keys := make([]string, 0, len(m))+ for k := range m {+ keys = append(keys, k)+ }+ sort.Slice(keys, func(i, j int) bool {+ return suiteRank(keys[i]) < suiteRank(keys[j]) || suiteRank(keys[i]) == suiteRank(keys[j]) && keys[i] < keys[j]+ })+ return keys+}++func suiteRank(k string) int {+ for i, s := range catalog.Suites {+ if strings.HasPrefix(k, s) {+ return i+ }+ }+ return len(catalog.Suites)+}++func mdEscape(s string) string { return strings.NewReplacer("|", "\\|", "\n", " ").Replace(s) }++func orDash(s string) string {+ if s == "" {+ return "-"+ }+ return s+}++func short(h string) string {+ if len(h) > 10 {+ return "`" + h[:10] + "`"+ }+ return "`" + h + "`"+}++func clip(s string, n int) string {+ s = strings.TrimSpace(s)+ if len(s) > n {+ return s[:n] + "..."+ }+ return s+}diff --git a/cli/test/evals/internal/report/report_test.go b/cli/test/evals/internal/report/report_test.gonew file mode 100644index 00000000..24bdbd5b--- /dev/null+++ b/cli/test/evals/internal/report/report_test.go@@ -0,0 +1,36 @@+package report++import (+ "math"+ "testing"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"+)++func res(c, arm, model string, idx int, status string, cost float64) *runner.Result {+ suite := c[:len("perf")]+ if c[:5] == "debug" {+ suite = "debug"+ }+ return &runner.Result{Case: c, Suite: suite, Arm: arm, Model: model, RunIdx: idx, Status: status, Metrics: runner.Metrics{CostUSD: cost, ToolCalls: 10}}+}++func TestAggregate(t *testing.T) {+ runs := []*runner.Result{+ res("perf-a", "cli+skill", "opus", 1, "pass", 0.5),+ res("perf-a", "cli-noskill", "opus", 1, "fail", 0.4),+ res("perf-b", "cli+skill", "opus", 1, "pass", 0.7),+ res("perf-b", "cli-noskill", "opus", 1, "pass", 0.6),+ res("perf-c", "cli+skill", "opus", 1, "infra_error", 0),+ }+ a := Aggregate(runs)+ if r := a.Suites["perf|opus|cli+skill"]; r.Pass != 2 || r.Total != 2 {+ t.Fatalf("infra errors must not count: %+v", r)+ }+ if len(a.SkillUplift) != 1 || math.Abs(a.SkillUplift[0].Delta-50) > 1e-9 {+ t.Fatalf("uplift %+v", a.SkillUplift)+ }+ if c := a.Cost["opus"]; c.PassingRuns != 3 || math.Abs(c.AvgCostUSD-0.6) > 1e-9 {+ t.Fatalf("cost %+v", c)+ }+}diff --git a/cli/test/evals/internal/report/results.go b/cli/test/evals/internal/report/results.gonew file mode 100644index 00000000..16807d6c--- /dev/null+++ b/cli/test/evals/internal/report/results.go@@ -0,0 +1,169 @@+// Package report turns run results into results.json and summary.md.+package report++import (+ "crypto/sha256"+ "encoding/hex"+ "encoding/json"+ "fmt"+ "io/fs"+ "os"+ "os/exec"+ "path/filepath"+ "sort"+ "strings"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/runner"+)++// Header identifies what was evaluated and who started it.+type Header struct {+ RunID string `json:"runId"`+ GitSHA string `json:"gitSha"`+ GitDirty bool `json:"gitDirty"`+ SkillHashes map[string]string `json:"skillHashes"`+ KnowledgeHash string `json:"knowledgeHash"`+ BinariesHash string `json:"binariesHash"`+ CLIVersion string `json:"cliVersion"`+ BackendVersion string `json:"backendVersion"`+ ModelsDigest string `json:"modelsDigest"`+ Models map[string]string `json:"models"`+ StartedBy string `json:"startedBy"`+ Command string `json:"command"`+ Started time.Time `json:"started"`+ Finished time.Time `json:"finished"`+ MaxCostUSD float64 `json:"maxCostUsd"`+ Oracle bool `json:"oracle,omitempty"`+ CI map[string]string `json:"ci,omitempty"`+}++// Results is results.json.+type Results struct {+ Header Header `json:"header"`+ Runs []*runner.Result `json:"runs"`+ Aggregates Aggregates `json:"aggregates"`+}++// NewHeader fills the provenance fields.+func NewHeader(repoRoot, runID string, models *catalog.Models, binHash string, cmdline []string) Header {+ h := Header{RunID: runID, SkillHashes: map[string]string{}, Models: map[string]string{}, Started: time.Now().UTC(), BinariesHash: binHash}+ git := func(args ...string) string {+ cmd := exec.Command("git", args...)+ cmd.Dir = repoRoot+ out, _ := cmd.Output()+ return strings.TrimSpace(string(out))+ }+ h.GitSHA = git("rev-parse", "HEAD")+ h.GitDirty = git("status", "--porcelain", "--", "backend", "cli", "skills", "plugins") != ""+ h.StartedBy = git("config", "user.email")+ if actor := os.Getenv("GITHUB_ACTOR"); actor != "" {+ h.StartedBy = actor + " (GitHub Actions)"+ h.CI = map[string]string{"event": os.Getenv("GITHUB_EVENT_NAME"), "workflow": os.Getenv("GITHUB_WORKFLOW_REF"),+ "runId": os.Getenv("GITHUB_RUN_ID"), "ref": os.Getenv("GITHUB_REF")}+ }+ for name, dir := range map[string]string{+ "traceway": "skills/traceway", "traceway-write": "plugins/tw-write/skills/traceway-write", "traceway-setup": "skills/traceway-setup",+ } {+ h.SkillHashes[name] = HashDir(filepath.Join(repoRoot, dir))+ }+ h.KnowledgeHash = HashDir(filepath.Join(repoRoot, "cli/pkg/mcpserver/knowledge"))+ h.CLIVersion = git("describe", "--tags", "--match", "cli/v*", "--abbrev=0")+ h.BackendVersion = git("describe", "--tags", "--match", "backend/v*", "--abbrev=0")+ if models != nil {+ h.ModelsDigest = models.Digest+ for k, m := range models.Agents {+ h.Models[k] = m.ID+ }+ if models.Judge != nil {+ h.Models["judge"] = models.Judge.ID+ }+ }+ h.Command = strings.Join(cmdline, " ")+ return h+}++// HashDir is the SHA-256 of a directory's files (paths and contents).+func HashDir(dir string) string {+ h := sha256.New()+ var files []string+ _ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {+ if err == nil && !d.IsDir() {+ files = append(files, p)+ }+ return nil+ })+ sort.Strings(files)+ for _, f := range files {+ rel, _ := filepath.Rel(dir, f)+ raw, _ := os.ReadFile(f)+ h.Write([]byte(rel))+ h.Write(raw)+ }+ return hex.EncodeToString(h.Sum(nil))+}++// Save writes results.json and summary.md into dir.+func Save(dir string, r *Results) error {+ r.Aggregates = Aggregate(r.Runs)+ raw, err := json.MarshalIndent(r, "", " ")+ if err != nil {+ return err+ }+ if err := os.WriteFile(filepath.Join(dir, "results.json"), raw, 0o644); err != nil {+ return err+ }+ return os.WriteFile(filepath.Join(dir, "summary.md"), []byte(Markdown(r)), 0o644)+}++// Load reads a results.json, or rebuilds one from the result.json files+// under a run directory (a shard that died before writing its summary).+func Load(dir string) (*Results, error) {+ raw, err := os.ReadFile(filepath.Join(dir, "results.json"))+ if err == nil {+ var r Results+ if err := json.Unmarshal(raw, &r); err != nil {+ return nil, fmt.Errorf("%s: %w", dir, err)+ }+ return &r, nil+ }+ r := &Results{}+ err = filepath.WalkDir(filepath.Join(dir, "runs"), func(p string, d fs.DirEntry, err error) error {+ if err != nil || d.IsDir() || d.Name() != "result.json" {+ return nil+ }+ b, err := os.ReadFile(p)+ if err != nil {+ return nil+ }+ var one runner.Result+ if json.Unmarshal(b, &one) == nil {+ r.Runs = append(r.Runs, &one)+ }+ return nil+ })+ if err != nil || len(r.Runs) == 0 {+ return nil, fmt.Errorf("%s: no results.json and no run results", dir)+ }+ return r, nil+}++// Merge combines shard results under one header.+func Merge(parts []*Results) *Results {+ out := &Results{}+ total := 0.0+ for i, p := range parts {+ if i == 0 {+ out.Header = p.Header+ }+ total += p.Header.MaxCostUSD+ if p.Header.Finished.After(out.Header.Finished) {+ out.Header.Finished = p.Header.Finished+ }+ out.Runs = append(out.Runs, p.Runs...)+ }+ out.Header.MaxCostUSD = total+ sort.SliceStable(out.Runs, func(i, j int) bool { return out.Runs[i].Key() < out.Runs[j].Key() })+ return out+}diff --git a/cli/test/evals/internal/runner/runner.go b/cli/test/evals/internal/runner/runner.gonew file mode 100644index 00000000..ff663e86--- /dev/null+++ b/cli/test/evals/internal/runner/runner.go@@ -0,0 +1,402 @@+// Package runner executes eval runs: for each (case, arm, model, run) it boots+// a hermetic backend, seeds it, prepares the workspace and credentials for+// the arm, drives the agent, grades the result and tears everything down.+package runner++import (+ "context"+ "encoding/json"+ "fmt"+ "io"+ "os"+ "path/filepath"+ "strings"+ "sync"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/agent"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/grade"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/tools"+)++// Options configures a batch of runs.+type Options struct {+ EvalsRoot string+ RepoRoot string+ OutDir string+ Bins *env.Binaries+ Models []*catalog.Model+ Runs int+ Jobs int+ Keep bool+ Log io.Writer+}++// Result is one run's outcome (results.json "runs" entries).+type Result struct {+ Case string `json:"case"`+ Suite string `json:"suite"`+ Arm string `json:"arm"`+ Model string `json:"model"`+ ModelID string `json:"modelId"`+ Providers []string `json:"provider,omitempty"`+ RunIdx int `json:"runIdx"`+ Status string `json:"status"` // pass | fail | budget_exhausted | infra_error+ Graders []*grade.Verdict `json:"graders"`+ Metrics Metrics `json:"metrics"`+ Transcript string `json:"transcript,omitempty"`+ Error string `json:"error,omitempty"`+ Seed *seed.Stats `json:"seed,omitempty"`+}++// Key identifies the run.+func (r *Result) Key() string { return RunKey(r.Case, r.Arm, r.Model, r.RunIdx) }++// RunKey builds a run key.+func RunKey(c, arm, model string, idx int) string {+ return fmt.Sprintf("%s/%s/%s/r%d", c, arm, model, idx)+}++// Metrics are a run's measurements.+type Metrics struct {+ Turns int `json:"turns"`+ ToolCalls int `json:"toolCalls"`+ FailedCalls int `json:"failedCalls"`+ InputTokens int `json:"inputTokens"`+ CachedTokens int `json:"cachedTokens"`+ CacheWriteTokens int `json:"cacheWriteTokens"`+ OutputTokens int `json:"outputTokens"`+ CostUSD float64 `json:"costUsd"`+ JudgeCostUSD float64 `json:"judgeCostUsd"`+ WallMs int64 `json:"wallMs"`+ SkillLoaded bool `json:"skillLoaded"`+ Skills []string `json:"skills,omitempty"`+}++// Job is one scheduled run.+type Job struct {+ Unit catalog.Unit+ Model *catalog.Model+ Idx int+}++// Jobs expands units × models × runs.+func Jobs(units []catalog.Unit, models []*catalog.Model, runs int) []Job {+ var out []Job+ for _, u := range units {+ for _, m := range models {+ for i := 1; i <= runs; i++ {+ out = append(out, Job{Unit: u, Model: m, Idx: i})+ }+ }+ }+ return out+}++// Execute runs every job with up to Jobs in parallel. Every run plays its+// case's oracle with the fake model.+func Execute(ctx context.Context, opt Options, jobs []Job) []*Result {+ par := opt.Jobs+ if par <= 0 {+ par = 1+ }+ results := make([]*Result, len(jobs))+ sem := make(chan struct{}, par)+ var wg sync.WaitGroup+ for i, j := range jobs {+ sem <- struct{}{}+ wg.Add(1)+ go func(i int, j Job) {+ defer wg.Done()+ defer func() { <-sem }()+ results[i] = runOne(ctx, opt, j)+ }(i, j)+ }+ wg.Wait()+ return results+}++func logf(opt Options, format string, a ...any) {+ if opt.Log != nil {+ fmt.Fprintf(opt.Log, format+"\n", a...)+ }+}++// surface is what the arm hands the agent.+type surface struct {+ cliBin string // executable path of the traceway CLI+ cliName string // traceway+ cliHome *env.CLIHome+ skills []string+ submit string+}++func submitFor(suite string) string {+ switch suite {+ case "perf":+ return tools.SubmitPerf+ case "debug":+ return tools.SubmitDebug+ default:+ return tools.Finish+ }+}++func runOne(ctx context.Context, opt Options, j Job) (res *Result) {+ c := j.Unit.Case+ arm := j.Unit.Arm+ res = &Result{Case: c.ID, Suite: c.Suite, Arm: arm, Model: j.Model.Key, ModelID: j.Model.ID, RunIdx: j.Idx}+ started := time.Now()+ dir := filepath.Join(opt.OutDir, "runs", c.ID, sanitize(arm), res.Model, fmt.Sprintf("r%d", j.Idx))+ defer func() {+ res.Metrics.WallMs = time.Since(started).Milliseconds()+ raw, _ := json.MarshalIndent(res, "", " ")+ _ = os.WriteFile(filepath.Join(dir, "result.json"), raw, 0o644)+ if !opt.Keep && res.Status == "pass" {+ // A passing run keeps its record (result, run, seed, transcript,+ // backend log); the database, CLI state and workspace go.+ for _, d := range []string{"backend", "cli", "work", ".twevals-bin", ".twevals-home"} {+ _ = os.RemoveAll(filepath.Join(dir, d))+ }+ }+ }()+ infra := func(err error) *Result {+ res.Status, res.Error = agent.StopInfra, err.Error()+ logf(opt, " %s infra_error: %v", RunKey(c.ID, arm, res.Model, j.Idx), err)+ return res+ }+ // A rerun into the same --out starts clean: a database left by an+ // earlier failed run already has its owner registered.+ if err := os.RemoveAll(dir); err != nil {+ return infra(err)+ }+ if err := os.MkdirAll(dir, 0o755); err != nil {+ return infra(err)+ }+ logf(opt, "▶ %s", RunKey(c.ID, arm, res.Model, j.Idx))++ inst, err := env.NewInstance(ctx, opt.Bins, dir)+ if err != nil {+ return infra(fmt.Errorf("boot: %w", err))+ }+ defer func() {+ _ = inst.Backend.WriteLog(filepath.Join(dir, "backend.log"))+ inst.Close()+ }()++ sink := seed.NewLiveSink(ctx, inst.Backend, inst.Admin)+ sink.StubSrv = inst.Stub+ sc := seed.NewScenario(c.ID, time.Now(), sink)+ if err := scenarios.Run(c.Seed, sc); err != nil {+ return infra(err)+ }+ if err := sc.Verify(60 * time.Second); err != nil {+ return infra(err)+ }+ stats := sc.Stats()+ res.Seed = &stats+ projects := sink.Projects()+ if len(projects) == 0 {+ return infra(fmt.Errorf("seed %s created no project", c.Seed))+ }+ exposed := sc.Exposed()+ primary := primaryProject(exposed, projects)++ vars := map[string]any{"seed": jsonify(exposed), "env": envVars(inst, projects, primary, sc.Now)}+ resolved, err := c.Resolve(vars)+ if err != nil {+ return infra(err)+ }+ writeJSON(filepath.Join(dir, "seed.json"), map[string]any{"exposed": jsonify(exposed), "env": vars["env"], "stats": stats})++ work := filepath.Join(dir, "work")+ if err := buildWorkspace(opt.EvalsRoot, resolved.Workspace.Repo, work, sc.Now); err != nil {+ return infra(err)+ }+ surf, err := prepareSurface(ctx, opt, resolved, arm, inst, primary, dir)+ if err != nil {+ return infra(err)+ }+ if err := installSkills(opt.RepoRoot, work, surf.skills); err != nil {+ return infra(err)+ }++ sb := &tools.Sandbox{Root: work, ReadOnly: []string{".skills"}}+ ask := &tools.AskUser{Rules: resolved.User}+ var loaded []string+ var submit *tools.Submit+ var toolset []tools.Tool+ var calls []*record.Call+ var callsMu sync.Mutex+ callExists := func(id string) (bool, bool) {+ callsMu.Lock()+ defer callsMu.Unlock()+ for _, cl := range calls {+ if cl.ID == id {+ return cl.OK, true+ }+ }+ return false, false+ }+ sh := &tools.Shell{Sandbox: sb, Git: true, Denied: deniedFor()}+ sh.Bins = map[string]string{surf.cliName: surf.cliBin}+ sh.BinEnv = surf.cliHome.Env(filepath.Dir(surf.cliBin))+ toolset = append(toolset, sh)+ toolset = append(toolset, &tools.ReadFile{Sandbox: sb}, &tools.ListFiles{Sandbox: sb}, &tools.Search{Sandbox: sb},+ &tools.EditFile{Sandbox: sb}, &tools.WriteFile{Sandbox: sb}, ask)+ skillDefs, _ := tools.LoadSkills(filepath.Join(work, ".skills"))+ if len(skillDefs) > 0 {+ toolset = append(toolset, &tools.LoadSkill{Skills: skillDefs, Rel: sb.Rel, Loaded: &loaded})+ }+ submit = &tools.Submit{Kind: surf.submit, CallExists: callExists, MaxRepairs: 2}+ toolset = append(toolset, submit)++ prompt := strings.TrimSpace(resolved.Prompt)+ var preloaded *tools.Skill+ if name := agent.SlashSkill(prompt); name != "" {+ for i := range skillDefs {+ if skillDefs[i].Name == name {+ preloaded = &skillDefs[i]+ loaded = append(loaded, name)+ }+ }+ if preloaded == nil {+ prompt = agent.StripSlash(prompt)+ }+ }+ system := agent.SystemPrompt(agent.PromptInput{+ SubmitTool: surf.submit, Skills: skillDefs, Preloaded: preloaded,+ Environment: environmentNote(surf, inst, primary),+ })++ fake := agent.NewFake(resolved.OracleFor(arm), filepath.Join(opt.EvalsRoot, "testdata"))+ defer fake.Close()+ tr, err := agent.OpenTranscript(filepath.Join(dir, "transcript.jsonl"))+ if err != nil {+ return infra(err)+ }+ defer tr.Close()+ res.Transcript, _ = filepath.Rel(opt.OutDir, filepath.Join(dir, "transcript.jsonl"))++ run := agent.Run(ctx, agent.Config{+ Model: j.Model, Chat: fake.Client(), Tools: toolset, System: system, Prompt: prompt, SubmitTool: surf.submit,+ Budget: resolved.Budget, Transcript: tr,+ AfterCall: func(ctx context.Context, cl *record.Call) {+ callsMu.Lock()+ calls = append(calls, cl)+ callsMu.Unlock()+ },+ Questions: ask.Questions,+ SkillLoaded: func() []string { return loaded },+ })+ writeJSON(filepath.Join(dir, "run.json"), run)++ res.Providers = run.Providers+ res.Metrics = metricsOf(run)+ if errs := fake.Errors(); len(errs) > 0 {+ res.Error = "oracle: " + strings.Join(errs, "; ")+ }+ if run.StopReason == agent.StopInfra {+ res.Status, res.Error = agent.StopInfra, run.StopDetail+ return res+ }++ in := &grade.Input{Case: resolved, Arm: arm, Run: run, Seed: jsonify(exposed)}+ res.Graders = append(res.Graders, grade.All(ctx, in)...)+ for _, v := range res.Graders {+ res.Metrics.JudgeCostUSD += v.CostUSD+ }+ switch {+ case run.StopReason == agent.StopBudget:+ res.Status = "budget_exhausted"+ case res.Error != "":+ res.Status = "fail"+ case grade.Passed(res.Graders):+ res.Status = "pass"+ default:+ res.Status = "fail"+ }+ logf(opt, " %s %s (%d calls, $%.4f)", RunKey(c.ID, arm, res.Model, j.Idx), res.Status, res.Metrics.ToolCalls, res.Metrics.CostUSD)+ return res+}++func primaryProject(exposed map[string]any, projects []env.Project) env.Project {+ if name, ok := exposed["primaryProject"].(string); ok {+ for _, p := range projects {+ if p.Name == name {+ return p+ }+ }+ }+ if len(projects) == 0 {+ return env.Project{}+ }+ return projects[0]+}++func envVars(inst *env.Instance, projects []env.Project, primary env.Project, now time.Time) map[string]any {+ ps := map[string]any{}+ for _, p := range projects {+ ps[p.Name] = map[string]any{"id": p.ID, "name": p.Name, "framework": p.Framework}+ }+ return map[string]any{+ "url": inst.Backend.URL, "projects": ps, "project": primary.ID, "projectName": primary.Name,+ "org": float64(inst.Admin.OrgID), "stub": inst.Stub.URL, "now": now.UTC().Format(time.RFC3339),+ }+}++// jsonify round-trips a value through JSON so templates and graders see+// plain maps, strings and float64s.+func jsonify(v map[string]any) map[string]any {+ raw, _ := json.Marshal(v)+ var out map[string]any+ _ = json.Unmarshal(raw, &out)+ if out == nil {+ out = map[string]any{}+ }+ return out+}++func writeJSON(path string, v any) {+ raw, err := json.MarshalIndent(v, "", " ")+ if err == nil {+ _ = os.WriteFile(path, raw, 0o644)+ }+}++func sanitize(s string) string { return strings.NewReplacer("+", "_", "/", "_").Replace(s) }++func metricsOf(run *record.Run) Metrics {+ m := Metrics{Turns: run.Turns, InputTokens: run.Usage.InputTokens, CachedTokens: run.Usage.CachedTokens,+ CacheWriteTokens: run.Usage.CacheWriteTokens, OutputTokens: run.Usage.OutputTokens, CostUSD: run.Usage.CostUSD,+ Skills: run.SkillLoaded, SkillLoaded: len(run.SkillLoaded) > 0}+ for _, c := range run.Calls {+ if c.Rejected {+ continue+ }+ m.ToolCalls+++ if !c.OK {+ m.FailedCalls+++ }+ }+ return m+}++func deniedFor() map[string]string {+ d := map[string]string{+ "curl": "no network tools in this sandbox", "wget": "no network tools in this sandbox",+ "python": "no interpreters in this sandbox", "python3": "no interpreters in this sandbox", "node": "no interpreters in this sandbox",+ "bash": "this shell already is the interpreter; run the command directly", "sh": "this shell already is the interpreter; run the command directly",+ }+ d["traceway-write"] = "traceway-write is not installed on this machine"+ return d+}++func environmentNote(s surface, inst *env.Instance, primary env.Project) string {+ return fmt.Sprintf("The %s CLI is installed and logged in to the team's Traceway instance at %s, with the %q project selected.", s.cliName, inst.Backend.URL, primary.Name)+}diff --git a/cli/test/evals/internal/runner/runner_test.go b/cli/test/evals/internal/runner/runner_test.gonew file mode 100644index 00000000..6c8dfe6b--- /dev/null+++ b/cli/test/evals/internal/runner/runner_test.go@@ -0,0 +1,41 @@+package runner++import (+ "context"+ "testing"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+ _ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios/all"+)++// TestOracleNPlusOne is the Phase 3 acceptance check: the N+1 case runs end+// to end with the fake model (backend, seed, CLI, shell, graders), with no+// network call apart from the local backend.+func TestOracleNPlusOne(t *testing.T) {+ bins := env.BinariesForTest(t)+ root := env.EvalsRoot()+ repo, err := env.RepoRoot(root)+ if err != nil {+ t.Fatal(err)+ }+ c, err := catalog.LoadCase(root + "/cases/perf/perf-db-n-plus-1.yaml")+ if err != nil {+ t.Fatal(err)+ }+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)+ defer cancel()+ model := &catalog.Model{Key: "oracle", ID: "fake/oracle", Caching: "none"}+ opt := Options{EvalsRoot: root, RepoRoot: repo, OutDir: t.TempDir(), Bins: bins, Models: []*catalog.Model{model}, Runs: 1, Jobs: 1}+ res := Execute(ctx, opt, Jobs([]catalog.Unit{{Case: c, Arm: catalog.ArmCLISkill}}, opt.Models, 1))+ if len(res) != 1 || res[0].Status != "pass" {+ for _, r := range res {+ t.Logf("%s %s %s", r.Key(), r.Status, r.Error)+ for _, g := range r.Graders {+ t.Logf(" %s pass=%v %v", g.Grader, g.Pass, g.Failures)+ }+ }+ t.Fatal("the N+1 oracle must pass")+ }+}diff --git a/cli/test/evals/internal/runner/surface.go b/cli/test/evals/internal/runner/surface.gonew file mode 100644index 00000000..76fde75d--- /dev/null+++ b/cli/test/evals/internal/runner/surface.go@@ -0,0 +1,32 @@+package runner++import (+ "context"+ "path/filepath"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+)++// prepareSurface mints the arm's credential and connects its surface: the+// CLI logged in with a read PAT (XDG dirs outside the workspace).+func prepareSurface(ctx context.Context, opt Options, c *catalog.Case, arm string, inst *env.Instance, primary env.Project, dir string) (surface, error) {+ s := surface{submit: submitFor(c.Suite)}+ if arm == catalog.ArmCLISkill {+ s.skills = []string{"traceway"}+ }+ pat, err := env.CreatePAT(ctx, inst.Backend, inst.Admin.JWT, "eval agent", "read", 0)+ if err != nil {+ return s, err+ }+ s.cliName, s.cliBin = "traceway", opt.Bins.CLI+ home, err := env.NewCLIHome(filepath.Join(dir, "cli"), s.cliBin)+ if err != nil {+ return s, err+ }+ if err := home.Login(ctx, inst.Backend.URL, pat, primary.ID); err != nil {+ return s, err+ }+ s.cliHome = home+ return s, nil+}diff --git a/cli/test/evals/internal/runner/workspace.go b/cli/test/evals/internal/runner/workspace.gonew file mode 100644index 00000000..28c51e18--- /dev/null+++ b/cli/test/evals/internal/runner/workspace.go@@ -0,0 +1,210 @@+package runner++import (+ "fmt"+ "io/fs"+ "os"+ "os/exec"+ "path/filepath"+ "strings"+ "time"++ "gopkg.in/yaml.v3"+)++// history.yaml in a fixture describes the commits leading to its final tree,+// with ages relative to the run so `git log` lines up with the seeded+// deploys. Each commit is the final tree with an overlay applied: the files+// under its .history/ directory (only those that differ from the final+// tree) replace or add files, and `absent` lists files that did not exist+// yet. The final tree (the fixture minus .history and history.yaml) is+// committed last as HEAD.+type fixtureHistory struct {+ Commits []struct {+ Dir string `yaml:"dir"`+ Absent []string `yaml:"absent"`+ Message string `yaml:"message"`+ Age string `yaml:"age"`+ Author string `yaml:"author"`+ } `yaml:"commits"`+ Head struct {+ Message string `yaml:"message"`+ Age string `yaml:"age"`+ Author string `yaml:"author"`+ } `yaml:"head"`+}++// buildWorkspace copies a fixture repo into work and commits its history.+func buildWorkspace(evalsRoot, repo, work string, now time.Time) error {+ if err := os.MkdirAll(work, 0o755); err != nil {+ return err+ }+ git := func(env []string, args ...string) error {+ cmd := exec.Command("git", args...)+ cmd.Dir = work+ cmd.Env = append([]string{"HOME=" + work, "PATH=" + os.Getenv("PATH"), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null"}, env...)+ if out, err := cmd.CombinedOutput(); err != nil {+ return fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, out)+ }+ return nil+ }+ if err := git(nil, "init", "-q", "-b", "main"); err != nil {+ return err+ }+ _ = os.WriteFile(filepath.Join(work, ".git", "info", "exclude"), []byte(".skills/\n.twevals*\n"), 0o644)+ commit := func(msg, age, author string) error {+ if author == "" {+ author = "Dana Developer <[email protected]>"+ }+ name, email, _ := strings.Cut(strings.TrimSuffix(author, ">"), " <")+ when := now.Add(-time.Hour)+ if age != "" {+ d, err := time.ParseDuration(age)+ if err != nil {+ return fmt.Errorf("history age %q: %w", age, err)+ }+ when = now.Add(-d)+ }+ date := when.UTC().Format(time.RFC3339)+ env := []string{"GIT_AUTHOR_NAME=" + name, "GIT_AUTHOR_EMAIL=" + email, "GIT_COMMITTER_NAME=" + name, "GIT_COMMITTER_EMAIL=" + email,+ "GIT_AUTHOR_DATE=" + date, "GIT_COMMITTER_DATE=" + date}+ if err := git(env, "add", "-A"); err != nil {+ return err+ }+ return git(env, "commit", "-q", "--allow-empty", "-m", msg)+ }+ if repo == "" {+ if err := os.WriteFile(filepath.Join(work, "README.md"), []byte("# service\n"), 0o644); err != nil {+ return err+ }+ return commit("Initial commit", "720h", "")+ }+ src := filepath.Join(evalsRoot, "testdata", "repos", repo)+ if _, err := os.Stat(src); err != nil {+ return fmt.Errorf("fixture repo %s: not in testdata/repos", repo)+ }+ var hist fixtureHistory+ if raw, err := os.ReadFile(filepath.Join(src, "history.yaml")); err == nil {+ if err := yaml.Unmarshal(raw, &hist); err != nil {+ return fmt.Errorf("%s/history.yaml: %w", repo, err)+ }+ }+ for _, c := range hist.Commits {+ if err := clearTree(work); err != nil {+ return err+ }+ if err := copyFixture(src, work); err != nil {+ return err+ }+ if c.Dir != "" {+ if err := copyFixture(filepath.Join(src, c.Dir), work); err != nil {+ return err+ }+ }+ for _, a := range c.Absent {+ if err := os.RemoveAll(filepath.Join(work, filepath.FromSlash(a))); err != nil {+ return err+ }+ }+ if err := commit(c.Message, c.Age, c.Author); err != nil {+ return err+ }+ }+ if err := clearTree(work); err != nil {+ return err+ }+ if err := copyFixture(src, work); err != nil {+ return err+ }+ msg := hist.Head.Message+ if msg == "" {+ msg = "Initial commit"+ }+ age := hist.Head.Age+ if age == "" && len(hist.Commits) == 0 {+ age = "720h"+ }+ return commit(msg, age, hist.Head.Author)+}++// clearTree removes everything in work except .git and .skills.+func clearTree(work string) error {+ entries, err := os.ReadDir(work)+ if err != nil {+ return err+ }+ for _, e := range entries {+ if e.Name() == ".git" || e.Name() == ".skills" {+ continue+ }+ if err := os.RemoveAll(filepath.Join(work, e.Name())); err != nil {+ return err+ }+ }+ return nil+}++// copyFixture copies a fixture tree, skipping its history metadata. Files+// named *.tmpl are copied without the suffix (so Go tooling never compiles+// fixture go.mod files inside the evals module).+func copyFixture(src, dst string) error {+ return filepath.WalkDir(src, func(p string, d fs.DirEntry, err error) error {+ if err != nil {+ return err+ }+ rel, _ := filepath.Rel(src, p)+ if rel == "." {+ return nil+ }+ if rel == "history.yaml" || rel == ".history" || strings.HasPrefix(rel, ".history"+string(filepath.Separator)) {+ if d.IsDir() {+ return filepath.SkipDir+ }+ return nil+ }+ target := filepath.Join(dst, strings.TrimSuffix(rel, ".tmpl"))+ if d.IsDir() {+ return os.MkdirAll(target, 0o755)+ }+ raw, err := os.ReadFile(p)+ if err != nil {+ return err+ }+ info, _ := d.Info()+ mode := os.FileMode(0o644)+ if info != nil && info.Mode()&0o111 != 0 {+ mode = 0o755+ }+ return os.WriteFile(target, raw, mode)+ })+}++// installSkills copies skill directories into work/.skills/<name>/.+func installSkills(repoRoot, work string, names []string) error {+ for _, n := range names {+ src, ok := skillSources[n]+ if !ok {+ return fmt.Errorf("unknown skill %s", n)+ }+ dst := filepath.Join(work, ".skills", n)+ if err := os.MkdirAll(dst, 0o755); err != nil {+ return err+ }+ if err := copyFixture(filepath.Join(repoRoot, src), dst); err != nil {+ return err+ }+ // The agent may read but never edit its skills.+ _ = filepath.WalkDir(dst, func(p string, d fs.DirEntry, err error) error {+ if err == nil && !d.IsDir() {+ _ = os.Chmod(p, 0o444)+ }+ return nil+ })+ }+ return nil+}++// skillSources maps a skill name to its published directory in the repo.+var skillSources = map[string]string{+ "traceway": "skills/traceway",+}diff --git a/cli/test/evals/internal/scenarios/all/all.go b/cli/test/evals/internal/scenarios/all/all.gonew file mode 100644index 00000000..0983d25c--- /dev/null+++ b/cli/test/evals/internal/scenarios/all/all.go@@ -0,0 +1,6 @@+// Package all imports every scenario suite so their seed functions register.+package all++import (+ _ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios/perf"+)diff --git a/cli/test/evals/internal/scenarios/perf/app_sync_background.go b/cli/test/evals/internal/scenarios/perf/app_sync_background.gonew file mode 100644index 00000000..83e95d90--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/app_sync_background.go@@ -0,0 +1,95 @@+package perf++import (+ "fmt"+ "sort"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/app-sync-background", papSeedSyncBackground) }++// papSeedSyncBackground: POST /api/statements renders the statement PDF in+// the request (6-11s) and uploads it before answering, although the+// dashboard only polls GET /api/statements/:id for the result. Distractor:+// host CPU jumps to 65-95% while a render runs, which looks like saturation,+// but the average is ~20%, the endpoint is just as slow when the host is+// otherwise idle, and every other endpoint is fast.+func papSeedSyncBackground(s *seed.Scenario) error {+ p := s.Project("billing", seed.OTel)+ api := p.Service("statements-api", seed.Go("nethttp", "pgx")).Versions("1.12.4")+ window := 7 * 24 * time.Hour++ var renders []papInterval+ var sorted bool+ create := api.Endpoint("POST", "/api/statements")+ create.Traffic(seed.Daily(420), seed.Diurnal(), seed.Over(window))+ create.Detail(seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.2, Max: 40})+ create.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.6, 1.4))+ r.DB("SELECT id, legal_name, currency, billing_email FROM accounts WHERE id = $1", r.Ms(1.5, 3.5))+ invoices := 40 + r.IntN(400)+ r.DB("SELECT id, number, issued_at, due_at, total_cents, status FROM invoices WHERE account_id = $1 AND issued_at >= $2 AND issued_at < $3 ORDER BY issued_at",+ r.Ms(8, 24), seed.Rows(invoices))+ r.DB("INSERT INTO statements (id, account_id, period_start, period_end, status, created_at) VALUES ($1, $2, $3, $4, 'rendering', now())", r.Ms(2, 5))+ r.Gap(r.Ms(0.2, 0.6))+ pages := 2 + invoices/25+ render := time.Duration((6200 + float64(pages)*250) * (0.9 + 0.2*r.Float()) * float64(time.Millisecond))+ at := r.At()+ r.Compute("render statement pdf", render, seed.A("statement.pages", pages), seed.A("statement.invoices", invoices))+ renders = append(renders, papInterval{at, at.Add(render)})+ sorted = false+ key := fmt.Sprintf("statements/%d/%06d.pdf", 2026, 100000+r.IntN(900000))+ r.HTTP("PUT", "https://acme-billing-statements.s3.eu-west-1.amazonaws.com/"+key, r.Ms(160, 420), 200,+ seed.A("http.request.body.size", 90000+pages*38000))+ r.DB("UPDATE statements SET status = 'ready', object_key = $2, rendered_at = now() WHERE id = $1", r.Ms(2, 4))+ r.Gap(r.Ms(0.3, 0.9))+ r.Status(201)+ })++ poll := api.Endpoint("GET", "/api/statements/:id")+ poll.Traffic(seed.Daily(470), seed.Diurnal(), seed.Over(window))+ poll.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.2))+ r.DB("SELECT id, account_id, status, object_key, rendered_at FROM statements WHERE id = $1", r.Ms(1.5, 4))+ r.Gap(r.Ms(0.2, 0.6))+ })++ download := api.Endpoint("GET", "/api/statements/:id/download")+ download.Traffic(seed.Daily(260), seed.Diurnal(), seed.Over(window))+ download.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.2))+ r.DB("SELECT object_key FROM statements WHERE id = $1 AND status = 'ready'", r.Ms(1.5, 3.5))+ r.Compute("presign object url", r.Ms(0.3, 0.9))+ r.Status(302)+ })++ invoices := api.Endpoint("GET", "/api/invoices")+ invoices.Traffic(seed.Daily(500), seed.Diurnal(), seed.Over(window))+ invoices.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.2))+ r.DB("SELECT id, number, issued_at, total_cents, status FROM invoices WHERE account_id = $1 ORDER BY issued_at DESC LIMIT 50", r.Ms(4, 12), seed.Rows(50))+ r.Gap(r.Ms(0.5, 2))+ })++ s.Background(api, seed.TypicalTraffic(8))++ // The host has two vCPUs and a render keeps one of them busy, so a+ // sample taken while a render runs reads ~80%.+ renderLoad := func(t time.Time) float64 {+ if !sorted {+ sort.Slice(renders, func(i, j int) bool { return renders[i].from.Before(renders[j].from) })+ sorted = true+ }+ return 0.6 * float64(papActive(renders, t, 15*time.Second))+ }+ api.Host().CPU(seed.Plus(seed.Flat(0.14, 0.22), renderLoad))+ api.Host().Memory(seed.Flat(0.41, 0.47))++ s.Expose("slowRequest", create.Slowest(24*time.Hour))+ s.Expose("p50Ms24h", func() any { return create.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return create.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/cache_no_cache.go b/cli/test/evals/internal/scenarios/perf/cache_no_cache.gonew file mode 100644index 00000000..88b75279--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/cache_no_cache.go@@ -0,0 +1,165 @@+package perf++import (+ "fmt"+ "strings"+ "time"++ "github.com/google/uuid"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/cache-no-cache", pncSeedNoCache) }++const pncWebOrigin = "https://shop.example.com"++var pncWebPages = []string{"/c/sneakers", "/c/boots", "/c/sandals", "/c/running", "/c/outdoor", "/c/kids", "/c/sale", "/c/new-in"}++// pncSeedNoCache: the shop's web app fetches the whole catalog (~410KB,+// 1200 products) again on every route change, with a cache-busting `_=`+// timestamp, and catalog-api keys its Redis cache by the full URL. Every+// request misses (cache.hit_ratio for the catalog cache ~3%), runs the+// catalog query and serializes 1200 products, so GET /api/catalog is by far+// the busiest endpoint for a catalog that changes twice a day. The browser+// project's occurrences carry the captured navigations and fetches that+// show the repeat. Distractor: the catalog query is the long pole of each+// request (~45ms), which reads like a slow query, but it is steady and the+// cost is volume times repetition.+func pncSeedNoCache(s *seed.Scenario) error {+ window := 24 * time.Hour+ api := s.Project("shop", seed.OTel)+ web := s.Project("shop-web", seed.Browser, "react")+ s.Expose("primaryProject", "shop")+ svc := api.Service("catalog-api", seed.Node("express", "pg")).Versions("6.2.1")++ catalog := svc.Endpoint("GET", "/api/catalog")+ catalog.Traffic(seed.Daily(9000), seed.Diurnal(), seed.Over(window))+ catalog.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.002, Max: 8})+ catalog.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ url := "/api/catalog?locale=en-US"+ hit := r.Chance(0.03) // the prerender job calls without the buster+ if !hit {+ url += fmt.Sprintf("&_=%d", r.Start().UnixMilli()-int64(r.IntN(400)))+ }+ r.RootAttr("url.query", url[len("/api/catalog?"):])+ if hit {+ r.Cache("GET", "catalog:"+url, r.Ms(0.6, 1.6))+ r.RootAttr("http.response.body.size", 412366)+ r.Gap(r.Ms(1.5, 3))+ return+ }+ r.Cache("GET", "catalog:"+url, r.Ms(0.4, 1.2))+ r.DB("SELECT p.id, p.sku, p.title, p.brand, p.image_url, c.slug AS category, pr.amount_minor, pr.currency FROM products p JOIN categories c ON c.id = p.category_id JOIN prices pr ON pr.product_id = p.id AND pr.price_list = $1 WHERE p.active ORDER BY c.position, p.rank",+ r.LogNormal(44*time.Millisecond, 95*time.Millisecond), seed.Rows(1180+r.IntN(40)))+ ser := r.Compute("serialize catalog", r.Ms(6, 12))+ ser.Scope = "catalog-api"+ r.Cache("SET", "catalog:"+url+" [3 other arguments]", r.Ms(0.4, 1.1))+ r.RootAttr("http.response.body.size", 412366)+ r.Gap(r.Ms(0.3, 0.8))+ })++ reviews := svc.Endpoint("GET", "/api/products/:sku/reviews")+ reviews.Traffic(seed.Daily(1300), seed.Diurnal(), seed.Over(window))+ reviews.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ r.Cache("GET", "sess:"+fmt.Sprintf("%x", r.IntN(1<<30)), r.Ms(0.3, 0.9))+ r.DB("SELECT id, rating, title, body, created_at FROM reviews WHERE product_sku = $1 ORDER BY created_at DESC LIMIT 10 OFFSET $2", r.LogNormal(4*time.Millisecond, 10*time.Millisecond), seed.Rows(10))+ r.Compute("serialize", r.Ms(0.3, 1))+ })++ s.Background(svc, seed.TrafficOpts{Endpoints: 3, PerDay: 150})+ svc.Host().CPU(seed.Flat(0.34, 0.46))+ svc.Host().Memory(seed.Flat(0.52, 0.6))+ svc.Gauge("cache.hit_ratio", "1", map[string]any{"cache": "catalog"}, seed.Flat(0.022, 0.045))+ svc.Gauge("cache.hit_ratio", "1", map[string]any{"cache": "session"}, seed.Flat(0.955, 0.985))++ // The web app: sessions, and the AbortError a quick route change raises+ // when it aborts the in-flight catalog fetch. Each occurrence carries the+ // navigations and fetches the SDK captured before it.+ const bundle = pncWebOrigin + "/assets/index-DqK3x8aP.js"+ abortStack := "AbortError: signal is aborted without reason\n at " + bundle + ":48:20113\n at " + bundle + ":48:19871"+ uas := []string{+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",+ "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Mobile/15E148 Safari/604.1",+ }+ rng := s.RNG+ catalogURL := func(at time.Time) string {+ return fmt.Sprintf("%s/api/catalog?locale=en-US&_=%d", pncWebOrigin, at.UnixMilli())+ }+ sessionAttrs := func(ua, path string) map[string]string {+ platform, viewport, screen := "Win32", "1440x812", "1920x1080"+ switch {+ case strings.Contains(ua, "iPhone"):+ platform, viewport, screen = "iPhone", "390x664", "390x844"+ case strings.Contains(ua, "Macintosh"):+ platform, viewport, screen = "MacIntel", "1512x823", "1512x982"+ }+ return map[string]string{"url": pncWebOrigin + path, "path": path, "referrer": "", "userAgent": ua, "language": "en-US",+ "platform": platform, "viewport": viewport, "screen": screen, "timezone": "Europe/Berlin"}+ }+ for i := 0; i < 380; i++ {+ start := s.Ago(time.Duration(30+rng.IntN(23*60)) * time.Minute).Add(time.Duration(rng.IntN(60)) * time.Second)+ ua := uas[rng.IntN(len(uas))]+ web.Session(uuid.NewString(), start, start.Add(time.Duration(3+rng.IntN(25))*time.Minute), sessionAttrs(ua, "/"), "2026.10.3")+ }+ var firstAt time.Time+ for i := 0; i < 9; i++ {+ at := s.Ago(time.Duration(25+i*61+rng.IntN(20)) * time.Minute).Add(time.Duration(rng.IntN(60)) * time.Second)+ if i == 0 {+ firstAt = at+ }+ sid := uuid.NewString()+ ua := uas[rng.IntN(len(uas))]+ started := at.Add(-time.Duration(4+rng.IntN(10)) * time.Minute)+ web.Session(sid, started, at.Add(time.Duration(2+rng.IntN(6))*time.Minute), sessionAttrs(ua, "/"), "2026.10.3")++ var acts []seed.Action+ from := "/"+ t := at.Add(-time.Duration(40+rng.IntN(15)) * time.Second)+ for step := 0; step < 4; step++ {+ to := pncWebPages[rng.IntN(len(pncWebPages))]+ if step == 1 {+ to = fmt.Sprintf("/p/%s", fmt.Sprintf("SH-%05d", 1000+rng.IntN(80000)))+ }+ acts = append(acts, seed.Action{Type: "navigation", At: t, From: from, To: to})+ acts = append(acts, seed.Action{Type: "network", At: t.Add(30 * time.Millisecond), Method: "GET", URL: catalogURL(t.Add(30 * time.Millisecond)),+ DurationMs: float64(62 + rng.IntN(60)), Status: 200, RespBytes: 412366})+ if step == 1 {+ acts = append(acts, seed.Action{Type: "network", At: t.Add(45 * time.Millisecond), Method: "GET",+ URL: pncWebOrigin + "/api/products" + to[2:] + "/reviews?page=1", DurationMs: float64(25 + rng.IntN(25)), Status: 200, RespBytes: 9000 + rng.IntN(3000)})+ }+ from = to+ t = t.Add(time.Duration(7+rng.IntN(9)) * time.Second)+ }+ // A quick second click aborts the fetch the previous route started.+ last := pncWebPages[rng.IntN(len(pncWebPages))]+ acts = append(acts, seed.Action{Type: "navigation", At: at.Add(-900 * time.Millisecond), From: from, To: last})+ acts = append(acts, seed.Action{Type: "network", At: at.Add(-870 * time.Millisecond), Method: "GET", URL: catalogURL(at.Add(-870 * time.Millisecond)),+ DurationMs: 860, Error: "AbortError: signal is aborted without reason"})+ next := fmt.Sprintf("/p/BT-%05d", 1000+rng.IntN(80000))+ acts = append(acts, seed.Action{Type: "navigation", At: at.Add(-10 * time.Millisecond), From: last, To: next})+ acts = append(acts, seed.Action{Type: "network", At: at, Method: "GET", URL: catalogURL(at), DurationMs: float64(60 + rng.IntN(50)), Status: 200, RespBytes: 412366})+ web.Crash(seed.Crash{At: at, Stack: abortStack, AppVersion: "2026.10.3", SessionID: sid,+ Attributes: map[string]string{"url": pncWebOrigin + next, "userAgent": ua}, Actions: acts})+ }++ s.Expose("slowRequest", catalog.Pick(func(r *seed.Request) bool {+ q, _ := r.Root.Attrs["url.query"].(string)+ return r.Start.After(s.Ago(3*time.Hour)) && len(q) > len("locale=en-US")+ }))+ s.Expose("crashAt", firstAt.UTC().Format(time.RFC3339Nano))+ s.Expose("catalogCount24h", func() any { return catalog.Count(24 * time.Hour) })+ s.AfterFlush(func() error {+ hash, err := web.GroupMatching(24*time.Hour, "signal is aborted")+ if err != nil {+ return err+ }+ s.Expose("webHash", hash)+ return nil+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/cache_stampede.go b/cli/test/evals/internal/scenarios/perf/cache_stampede.gonew file mode 100644index 00000000..a8e83888--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/cache_stampede.go@@ -0,0 +1,149 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/cache-stampede", pncSeedStampede) }++// pncHomeSections are the queries that rebuild the home feed, with their+// share of a rebuild's DB time.+var pncHomeSections = []struct {+ q string+ w float64+}{+ {"SELECT id, image_url, target_url, starts_at, ends_at FROM banners WHERE placement = 'home' AND now() BETWEEN starts_at AND ends_at ORDER BY priority", 0.03},+ {"SELECT c.id, c.title, c.slug, c.hero_image FROM collections c WHERE c.featured ORDER BY c.position LIMIT 6", 0.04},+ {"SELECT ol.sku, SUM(ol.qty) AS sold FROM order_lines ol JOIN orders o ON o.id = ol.order_id WHERE o.placed_at > now() - interval '7 days' GROUP BY ol.sku ORDER BY sold DESC LIMIT 24", 0.31},+ {"SELECT id, sku, title, price_minor, image_url FROM products WHERE published_at > now() - interval '14 days' ORDER BY published_at DESC LIMIT 24", 0.07},+ {"SELECT p.id, p.sku, p.title, p.price_minor, p.image_url FROM editorial_picks e JOIN products p ON p.id = e.product_id WHERE e.week = date_trunc('week', now()) ORDER BY e.position", 0.04},+ {"SELECT term, count(*) AS n FROM search_events WHERE created_at > now() - interval '1 day' GROUP BY term ORDER BY n DESC LIMIT 10", 0.18},+ {"SELECT id, slug, title, tile_image FROM categories WHERE parent_id IS NULL ORDER BY position", 0.02},+ {"SELECT d.id, d.product_id, d.discount_pct, d.ends_at FROM deals d WHERE d.active AND d.ends_at > now() ORDER BY d.discount_pct DESC LIMIT 12", 0.05},+ {"SELECT b.id, b.name, b.logo_url FROM brands b WHERE b.spotlight_until > now() LIMIT 4", 0.02},+ {"SELECT r.product_id, avg(r.rating) AS rating, count(*) AS n FROM reviews r WHERE r.created_at > now() - interval '3 days' GROUP BY r.product_id ORDER BY n DESC LIMIT 12", 0.14},+ {"SELECT id, title, image_url FROM seasonal_stories WHERE season = $1 ORDER BY position LIMIT 3", 0.03},+ {"SELECT sku FROM product_scores WHERE segment = 'default' ORDER BY score DESC LIMIT 24", 0.07},+}++// pncSeedStampede: GET /api/home serves the home feed from Redis+// (home:feed:v3, 10-minute TTL). When the key expires, every request that+// arrives before the first rebuild has written it back misses and rebuilds+// the feed itself: ~2.4 req/s times a ~7s rebuild under that contention is+// 15-25 identical rebuild traces per expiry, every ~10 minutes. Between+// expiries the hit ratio is ~99%. Distractor: the misses run 12 queries+// against the DB on the request path, which reads like a missing cache, but+// the cache is there and hits ~99% of the time.+func pncSeedStampede(s *seed.Scenario) error {+ window := 2 * time.Hour+ p := s.Project("storefront-home", seed.OTel)+ api := p.Service("home-api", seed.Go("chi", "pgx")).Versions("3.4.1")++ // Expiry schedule: each window [at, at+fill) is when the key is gone+ // and requests miss; the first rebuild's SET closes it and the next+ // expiry is ttl later. Built over 24h so the hit-ratio gauge shows it+ // all day.+ type expiry struct {+ at time.Time+ fill time.Duration+ }+ var expiries []expiry+ for t := s.Ago(24*time.Hour + 7*time.Minute + 23*time.Second); t.Before(s.Now); {+ fill := time.Duration(7000+s.RNG.IntN(3000)) * time.Millisecond+ expiries = append(expiries, expiry{t, fill})+ t = t.Add(fill + 10*time.Minute)+ }+ missWindow := func(t time.Time) (expiry, bool) {+ for _, e := range expiries {+ if !t.Before(e.at) && t.Before(e.at.Add(e.fill)) {+ return e, true+ }+ }+ return expiry{}, false+ }+ var recent []time.Time // the last two expiries inside the traffic window+ for i := len(expiries) - 1; i >= 0 && len(recent) < 2; i-- {+ if expiries[i].at.Add(expiries[i].fill).Before(s.Now.Add(-time.Minute)) {+ recent = append(recent, expiries[i].at)+ }+ }++ misses := map[*seed.Request]bool{}+ home := api.Endpoint("GET", "/api/home")+ home.Traffic(seed.Daily(207360), seed.Over(window)) // 2.4 req/s+ home.Detail(seed.DetailPolicy{Recent: window, Rate: 0.0015, Max: 12})+ home.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.2, 0.6))+ e, miss := missWindow(r.Start())+ if !miss {+ r.Cache("get", "home:feed:v3", r.LogNormal(700*time.Microsecond, 2*time.Millisecond))+ r.Compute("decode feed", r.Ms(0.8, 2.2))+ r.Gap(r.Ms(0.3, 1.2))+ return+ }+ misses[r.Request()] = true+ for _, at := range recent {+ if at.Equal(e.at) {+ r.KeepDetail()+ }+ }+ r.Cache("get", "home:feed:v3", r.LogNormal(700*time.Microsecond, 2*time.Millisecond))+ total := float64(e.fill) * (0.88 + 0.24*r.Float())+ r.Span("buildHomeFeed", func(r *seed.Req) {+ for _, sec := range pncHomeSections {+ r.DB(sec.q, time.Duration(total*sec.w*(0.9+0.2*r.Float())))+ }+ r.Compute("encode feed", r.Ms(6, 14))+ }).Scope = "home-api"+ r.Cache("set", "home:feed:v3", r.Ms(0.6, 1.8), seed.A("db.query.text", "set home:feed:v3 ? ex 600"))+ r.Gap(r.Ms(0.3, 1.2))+ })++ product := api.Endpoint("GET", "/api/home/stories/:id")+ product.Traffic(seed.Daily(6000), seed.Over(window))+ product.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.2, 0.6))+ r.DB("SELECT id, title, body, image_url, published_at FROM seasonal_stories WHERE id = $1", r.LogNormal(2500*time.Microsecond, 7*time.Millisecond))+ r.Compute("encode response", r.Ms(0.3, 1))+ })++ s.Background(api, seed.TrafficOpts{Endpoints: 3, PerDay: 900, Window: window})+ api.Host().CPU(seed.Flat(0.21, 0.3))+ api.Host().Memory(seed.Flat(0.4, 0.47))+ api.Gauge("cache.hit_ratio", "1", map[string]any{"cache": "home-feed"}, func(t time.Time) float64 {+ for _, e := range expiries {+ if !t.Before(e.at) && t.Before(e.at.Add(time.Minute)) {+ return 0.89 - 0.03*float64(e.fill)/float64(10*time.Second)+ }+ }+ return 0.984+ })++ s.Expose("missRequest", home.Pick(func(r *seed.Request) bool { return misses[r] && len(recent) > 0 && r.Start.After(recent[0]) }))+ if len(recent) > 0 {+ s.Expose("lastExpiry", recent[0].UTC().Format(time.RFC3339))+ s.Expose("spikeFrom", recent[0].Truncate(5*time.Minute).UTC().Format(time.RFC3339))+ s.Expose("spikeTo", recent[0].Add(time.Minute).Truncate(5*time.Minute).Add(5*time.Minute).UTC().Format(time.RFC3339))+ }+ s.Expose("missesPerExpiry", func() any {+ counts := map[time.Time]int{}+ for r := range misses {+ if e, ok := missWindow(r.Start); ok {+ counts[e.at]+++ }+ }+ out := []int{}+ for _, e := range expiries {+ if n, ok := counts[e.at]; ok && e.at.After(s.Ago(window)) {+ out = append(out, n)+ }+ }+ return out+ })+ s.Expose("p50Ms2h", func() any { return home.Percentile(0.5, window).Seconds() * 1000 })+ s.Expose("p99Ms2h", func() any { return home.Percentile(0.99, window).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/db_n_plus_1.go b/cli/test/evals/internal/scenarios/perf/db_n_plus_1.gonew file mode 100644index 00000000..b525554b--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/db_n_plus_1.go@@ -0,0 +1,64 @@+// Package perf holds the seed functions of the perf suite: one scenario per+// bottleneck checklist row, plus controls and cross-surface cases.+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/db-n-plus-1", seedNPlusOne) }++// seedNPlusOne: the orders endpoint loads a customer's orders, then one+// order_items query per order. Customers accumulate orders over the 29 days,+// so latency climbs with no deploy and flat CPU. Distractor: the orders query+// selects every column (over-fetching), but it is fast.+func seedNPlusOne(s *seed.Scenario) error {+ p := s.Project("storefront", seed.OTel)+ api := p.Service("orders-api", seed.Go("nethttp", "sql")).Versions("2.14.3")+ window := 29 * 24 * time.Hour++ orders := api.Endpoint("GET", "/api/orders")+ orders.Traffic(seed.Daily(160), seed.Diurnal(), seed.Over(window))+ orders.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.25, Max: 12})+ orders.Handler(func(r *seed.Req) {+ // Orders per customer grow over the window; LIMIT 100 caps them.+ n := int(r.Lerp(6, 72) * (0.35 + 0.95*r.Float()))+ if n > 100 {+ n = 100+ }+ if n < 1 {+ n = 1+ }+ r.Gap(r.Ms(0.4, 1.1)) // auth middleware+ r.DB("SELECT * FROM orders WHERE user_id = $1 ORDER BY created_at DESC LIMIT 100", r.Ms(3, 6), seed.Rows(n))+ for range n {+ r.DB("SELECT id, sku, qty, unit_price FROM order_items WHERE order_id = $1", r.Ms(6, 14))+ }+ r.Compute("encode response", r.Ms(2, 6))+ r.Gap(r.Ms(0.2, 0.6))+ })++ order := api.Endpoint("GET", "/api/orders/:id")+ order.Traffic(seed.Daily(120), seed.Diurnal(), seed.Over(window))+ order.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.1))+ r.DB("SELECT id, user_id, status, total_cents, created_at FROM orders WHERE id = $1", r.Ms(2, 5))+ r.DB("SELECT o.id, o.sku, o.qty, o.unit_price FROM order_items o WHERE o.order_id = $1", r.Ms(3, 7))+ r.Compute("encode response", r.Ms(0.5, 1.5))+ })++ s.Background(api, seed.TypicalTraffic(10))+ api.Host().CPU(seed.Flat(0.22, 0.31))+ api.Host().Memory(seed.Flat(0.48, 0.55))++ s.Expose("slowRequest", orders.Slowest(24*time.Hour))+ s.Expose("p50Ms24h", func() any { return orders.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return orders.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95MsFirstWeek", func() any {+ return orders.PercentileBetween(0.95, s.Ago(window), s.Ago(window-7*24*time.Hour)).Seconds() * 1000+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/db_n_plus_1_test.go b/cli/test/evals/internal/scenarios/perf/db_n_plus_1_test.gonew file mode 100644index 00000000..7435b863--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/db_n_plus_1_test.go@@ -0,0 +1,114 @@+package perf++import (+ "context"+ "encoding/json"+ "net/http"+ "strings"+ "testing"+ "time"++ client "github.com/tracewayapp/traceway/cli/pkg/client"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++// TestSeedNPlusOne is the Phase 2 acceptance check: the slow request's+// waterfall has at least 40 order_items spans, every parent covers its+// children, and the listed percentiles match what the DSL generated.+func TestSeedNPlusOne(t *testing.T) {+ bins := env.BinariesForTest(t)+ ctx, cancel := context.WithTimeout(context.Background(), 4*time.Minute)+ defer cancel()+ inst, err := env.NewInstance(ctx, bins, t.TempDir())+ if err != nil {+ t.Fatal(err)+ }+ defer inst.Close()++ sink := seed.NewLiveSink(ctx, inst.Backend, inst.Admin)+ s := seed.NewScenario("perf-db-n-plus-1", time.Now(), sink)+ if err := scenarios.Run("perf/db-n-plus-1", s); err != nil {+ t.Fatal(err)+ }+ if err := s.Verify(time.Minute); err != nil {+ t.Fatal(err)+ }+ t.Logf("seed stats: %+v", s.Stats())+ ex := s.Exposed()+ slow := ex["slowRequest"].(map[string]any)+ pid := s.Projects()[0].ID+ api := inst.Backend.API().WithBearer(inst.Admin.JWT)++ at, _ := time.Parse(time.RFC3339Nano, slow["recordedAt"].(string))+ c := client.New(inst.Backend.URL, client.WithJWT(inst.Admin.JWT))+ detail, err := c.GetEndpoint(ctx, pid, slow["id"].(string), at)+ if err != nil {+ t.Fatal(err)+ }+ items := 0+ children := map[string]int64{}+ byID := map[string]int64{}+ for _, sp := range detail.Spans {+ byID[sp.SpanId] = int64(sp.Duration)+ if sp.ParentSpanId != "" {+ children[sp.ParentSpanId] += int64(sp.Duration)+ }+ if sp.Name == "SELECT order_items" {+ items+++ }+ }+ if items < 40 {+ t.Errorf("slowest request has %d order_items spans, want >= 40 (spans: %d)", items, len(detail.Spans))+ }+ for parent, sum := range children {+ if d, ok := byID[parent]; ok && d < sum {+ t.Errorf("span %s lasts %d but its children sum to %d", parent, d, sum)+ }+ }++ from, to := s.Now.Add(-24*time.Hour), s.Now.Add(time.Minute)+ lres, err := api.MustOK(ctx, http.MethodPost, "/api/endpoints/grouped", env.ProjectQuery(pid), map[string]any{+ "fromDate": from.Format(time.RFC3339), "toDate": to.Format(time.RFC3339), "orderBy": "p95",+ "search": "GET /api/orders", "pagination": map[string]int{"page": 1, "pageSize": 20},+ })+ if err != nil {+ t.Fatal(err)+ }+ var list struct {+ Data []struct {+ Endpoint string `json:"endpoint"`+ Count int `json:"count"`+ P50Duration int64 `json:"p50Duration"`+ P95Duration int64 `json:"p95Duration"`+ } `json:"data"`+ }+ if err := json.Unmarshal(lres.Body, &list); err != nil {+ t.Fatal(err)+ }+ var found bool+ for _, d := range list.Data {+ if d.Endpoint != "GET /api/orders" {+ continue+ }+ found = true+ p95 := float64(d.P95Duration) / 1e6+ want := ex["p95Ms24h"].(float64)+ if p95 < want*0.85 || p95 > want*1.15 {+ t.Errorf("listed p95 %.0fms, generated %.0fms", p95, want)+ }+ if p95 < 500 || p95 > 2500 {+ t.Errorf("p95 %.0fms is outside the designed 0.5-2.5s range", p95)+ }+ t.Logf("GET /api/orders 24h: count=%d p50=%.0fms p95=%.0fms (first week p95 %.0fms)", d.Count, float64(d.P50Duration)/1e6, p95, ex["p95MsFirstWeek"].(float64))+ }+ if !found {+ t.Fatalf("GET /api/orders not listed: %s", lres.Body)+ }+ for _, sp := range detail.Spans {+ if strings.Contains(strings.ToLower(sp.Name), "n+1") {+ t.Error("seeded data names the answer")+ }+ }+}diff --git a/cli/test/evals/internal/scenarios/perf/infra_cpu_saturation.go b/cli/test/evals/internal/scenarios/perf/infra_cpu_saturation.gonew file mode 100644index 00000000..3d32da5b--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/infra_cpu_saturation.go@@ -0,0 +1,167 @@+package perf++import (+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/infra-cpu-saturation", papSeedCPUSaturation) }++// papSeedCPUSaturation: 16-40h ago (14:20 UTC) a campaign tripled commerce-api's traffic+// (ramping over 40 minutes) and it has stayed there. Both instances went+// from ~30% to ~95% CPU and every endpoint got 2-4x slower at once: each+// span is stretched and run-queue gaps appear, with no single long pole.+// Nothing was deployed. Distractor: a code regression; but service.version+// never changed, no commit landed, and the slowdown follows the request rate+// on every endpoint alike.+func papSeedCPUSaturation(s *seed.Scenario) error {+ p := s.Project("commerce", seed.OTel)+ api := p.Service("commerce-api", seed.Node("express", "pg")).Versions("4.18.2").+ Instances("commerce-api-7f6d9c4b8-2xkqv", "commerce-api-7f6d9c4b8-9jw4t")+ window := 7 * 24 * time.Hour+ // The campaign went live at 14:20 UTC, 16-40 hours before the run.+ onset := s.Now.UTC().Truncate(24 * time.Hour).Add(14*time.Hour + 20*time.Minute)+ for onset.After(s.Ago(16 * time.Hour)) {+ onset = onset.Add(-24 * time.Hour)+ }+ ramp := 40 * time.Minute++ load := func(t time.Time) float64 { // traffic multiple of the usual rate+ switch {+ case t.Before(onset):+ return 1+ case t.Before(onset.Add(ramp)):+ return 1 + 2*float64(t.Sub(onset))/float64(ramp)+ default:+ return 3+ }+ }+ diurnal := func(t time.Time) float64 {+ h := float64(t.Hour()) + float64(t.Minute())/60+ return 1 + 0.25*math.Cos((h-14)/24*2*math.Pi)+ }+ // Utilization follows the request rate; past ~70% the run queue grows+ // and every request is stretched.+ util := func(t time.Time) float64 { return math.Min(0.97, 0.27*load(t)*diurnal(t)) }+ stretch := func(t time.Time) float64 {+ u := util(t)+ if u <= 0.45 {+ return 1+ }+ return 1 + 2.6*math.Pow((u-0.45)/0.52, 1.5)+ }+ traffic := func(e *seed.Endpoint, salt string, perDay int) {+ e.Traffic(papJitter(salt, 0.06), seed.RateFunc(load), seed.RateFunc(diurnal), seed.Daily(perDay), seed.Over(window))+ e.Detail(seed.DetailPolicy{Recent: 24 * time.Hour, Rate: 0.02, Max: 12})+ }+ // cpu is in-process work (no span: Express is instrumented only at the+ // edges), stretched fully; db is a query, of which only the client-side+ // half (driver, event loop) is stretched.+ cpu := func(r *seed.Req, d time.Duration) time.Duration {+ return time.Duration(float64(d) * stretch(r.Start()))+ }+ db := func(r *seed.Req, d time.Duration) time.Duration {+ return time.Duration(float64(d) * (1 + 0.5*(stretch(r.Start())-1)))+ }+ queue := func(r *seed.Req) {+ r.Gap(cpu(r, r.Ms(0.3, 0.9)) + time.Duration((stretch(r.Start())-1)*float64(r.Ms(2, 9))))+ }++ products := api.Endpoint("GET", "/api/products")+ traffic(products, "products", 260)+ products.Handler(func(r *seed.Req) {+ queue(r)+ r.DB("SELECT id, sku, name, price_cents, thumbnail_url FROM products WHERE category_id = $1 AND visible ORDER BY rank LIMIT 48", db(r, r.Ms(6, 14)), seed.Rows(48))+ r.Gap(cpu(r, r.Ms(4, 9))) // serialize products+ })+ product := api.Endpoint("GET", "/api/products/:id")+ traffic(product, "product", 320)+ product.Handler(func(r *seed.Req) {+ queue(r)+ r.DB("SELECT id, sku, name, description, price_cents, stock FROM products WHERE id = $1", db(r, r.Ms(1.5, 4)))+ r.DB("SELECT url, alt FROM product_images WHERE product_id = $1 ORDER BY position", db(r, r.Ms(1.5, 4)))+ r.Gap(cpu(r, r.Ms(3, 7))) // render product+ })+ search := api.Endpoint("GET", "/api/search")+ traffic(search, "search", 240)+ search.Handler(func(r *seed.Req) {+ queue(r)+ r.HTTP("POST", "http://search.internal:9200/products/_search", db(r, r.Ms(12, 30)), 200)+ r.Gap(cpu(r, r.Ms(6, 14))) // hydrate results+ })+ cart := api.Endpoint("GET", "/api/cart")+ traffic(cart, "cart", 220)+ cart.Handler(func(r *seed.Req) {+ queue(r)+ r.Cache("GET", "cart:session", db(r, r.Ms(0.4, 1.2)))+ r.DB("SELECT p.id, p.name, p.price_cents, ci.qty FROM cart_items ci JOIN products p ON p.id = ci.product_id WHERE ci.cart_id = $1", db(r, r.Ms(2, 5)))+ r.Gap(cpu(r, r.Ms(2, 5))) // price cart+ })+ add := api.Endpoint("POST", "/api/cart/items")+ traffic(add, "add", 160)+ add.Handler(func(r *seed.Req) {+ queue(r)+ r.Gap(cpu(r, r.Ms(0.6, 1.5))) // validate body+ r.DB("INSERT INTO cart_items (cart_id, product_id, qty) VALUES ($1, $2, $3) ON CONFLICT (cart_id, product_id) DO UPDATE SET qty = cart_items.qty + EXCLUDED.qty", db(r, r.Ms(2, 5)))+ r.Cache("DEL", "cart:session", db(r, r.Ms(0.3, 0.9)))+ r.Status(201)+ })+ checkout := api.Endpoint("POST", "/api/checkout")+ traffic(checkout, "checkout", 70)+ checkout.Handler(func(r *seed.Req) {+ queue(r)+ r.DB("SELECT p.id, p.price_cents, p.stock, ci.qty FROM cart_items ci JOIN products p ON p.id = ci.product_id WHERE ci.cart_id = $1 FOR UPDATE", db(r, r.Ms(3, 7)))+ r.Gap(cpu(r, r.Ms(4, 9))) // price order+ r.HTTP("POST", "https://api.payments.example/v1/payment_intents", r.LogNormal(180*time.Millisecond, 420*time.Millisecond), 200)+ r.DB("INSERT INTO orders (id, user_id, total_cents, status, created_at) VALUES ($1, $2, $3, 'paid', now())", db(r, r.Ms(2, 5)))+ r.Status(201)+ })+ recs := api.Endpoint("GET", "/api/recommendations")+ traffic(recs, "recs", 200)+ recs.Handler(func(r *seed.Req) {+ queue(r)+ r.Cache("GET", "recs:user", db(r, r.Ms(0.4, 1.2)))+ r.DB("SELECT product_id, score FROM recommendations WHERE user_id = $1 ORDER BY score DESC LIMIT 12", db(r, r.Ms(3, 8)))+ r.Gap(cpu(r, r.Ms(5, 12))) // rank recommendations+ })+ cats := api.Endpoint("GET", "/api/categories")+ traffic(cats, "cats", 130)+ cats.Handler(func(r *seed.Req) {+ queue(r)+ r.Cache("GET", "categories:tree", db(r, r.Ms(0.3, 0.9)))+ r.Gap(cpu(r, r.Ms(1, 3))) // serialize categories+ })++ for _, h := range api.Hosts {+ api.HostNamed(h).CPU(seed.Plus(seed.Flat(-0.02, 0.02), util))+ api.HostNamed(h).Memory(seed.Plus(seed.Flat(0.42, 0.46), func(t time.Time) float64 { return 0.04 * (load(t) - 1) }))+ }++ rfc := func(t time.Time) string { return t.UTC().Format(time.RFC3339) }+ s.Expose("onsetAt", rfc(onset))+ s.Expose("beforeFrom", rfc(onset.Add(-24*time.Hour+time.Hour)))+ s.Expose("beforeTo", rfc(onset.Add(-24*time.Hour+7*time.Hour)))+ s.Expose("afterFrom", rfc(onset.Add(time.Hour)))+ s.Expose("afterTo", rfc(onset.Add(7*time.Hour)))+ var slow *seed.Request+ for _, rq := range product.Requests() {+ if rq.Start.After(onset.Add(time.Hour)) && rq.Start.Before(onset.Add(7*time.Hour)) && (slow == nil || rq.Duration() > slow.Duration()) {+ slow = rq+ }+ }+ slow.KeepDetail()+ s.Expose("slowRequest", slow.Ref())+ s.Expose("earlierRequest", product.Pick(func(rq *seed.Request) bool {+ return rq.Start.After(onset.Add(-23*time.Hour)) && rq.Start.Before(onset.Add(-17*time.Hour)) && rq.Duration() > 10*time.Millisecond+ }))+ s.Expose("p95ProductBefore", func() any {+ return product.PercentileBetween(0.95, onset.Add(-23*time.Hour), onset.Add(-17*time.Hour)).Seconds() * 1000+ })+ s.Expose("p95ProductAfter", func() any {+ return product.PercentileBetween(0.95, onset.Add(time.Hour), onset.Add(7*time.Hour)).Seconds() * 1000+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/net_large_payload.go b/cli/test/evals/internal/scenarios/perf/net_large_payload.gonew file mode 100644index 00000000..6fc28072--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/net_large_payload.go@@ -0,0 +1,94 @@+package perf++import (+ "strings"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/net-large-payload", pncSeedLargePayload) }++// pncMobileUA reports whether a request came from a phone (the iOS web view+// or the Android app's okhttp).+func pncMobileUA(r *seed.Req) bool {+ ua, _ := r.Request().Root.Attrs["user_agent.original"].(string)+ return strings.Contains(ua, "iPhone") || strings.HasPrefix(ua, "okhttp")+}++// pncSeedLargePayload: the feed endpoint returns 50 cards with their images+// inlined as data URIs, ~2.3MB per response. The query and the handler's+// own work take ~30ms; the rest of the request is the body going out, so+// latency follows http.response.body.size and the client's bandwidth (phones+// take seconds, desktops a few hundred ms). Distractor: the feed query runs+// on every request with no cache in front of it, but it takes ~15ms.+func pncSeedLargePayload(s *seed.Scenario) error {+ window := 2 * 24 * time.Hour+ p := s.Project("social", seed.OTel)+ api := p.Service("feed-api", seed.Go("nethttp", "pgx")).Versions("8.4.2")++ feed := api.Endpoint("GET", "/api/feed")+ feed.Traffic(seed.Daily(2400), seed.Diurnal(), seed.Over(window))+ feed.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.02, Max: 10})+ feed.Handler(func(r *seed.Req) {+ withImages := 40 + r.IntN(11)+ size := 58000 + r.IntN(9000) + withImages*(44000+r.IntN(4000))+ r.RootAttr("http.response.body.size", size)+ r.Gap(r.Ms(0.3, 0.9))+ r.DB("SELECT c.id, c.kind, c.title, c.body, c.image_data, c.published_at, a.id, a.display_name, a.avatar_data FROM feed_cards c JOIN authors a ON a.id = c.author_id WHERE c.audience = ANY($1) ORDER BY c.published_at DESC LIMIT $2",+ r.LogNormal(14*time.Millisecond, 32*time.Millisecond), seed.Rows(50))+ r.Compute("rank cards", r.Ms(2, 6))+ // json.NewEncoder(w).Encode writes the body as it encodes; on a+ // slow link the writes block until the client has read it.+ mbps := 40 + 110*r.Float()+ if pncMobileUA(r) {+ mbps = 5 + 15*r.Float()+ }+ r.Gap(r.Ms(14, 30) + time.Duration(float64(size)*8/(mbps*1e6)*float64(time.Second)))+ })++ card := api.Endpoint("GET", "/api/feed/cards/:id")+ card.Traffic(seed.Daily(900), seed.Diurnal(), seed.Over(window))+ card.Handler(func(r *seed.Req) {+ size := 2200 + r.IntN(1800)+ if r.Chance(0.8) {+ size += 44000 + r.IntN(4000)+ }+ r.RootAttr("http.response.body.size", size)+ r.Gap(r.Ms(0.3, 0.9))+ r.DB("SELECT c.id, c.kind, c.title, c.body, c.image_data, c.published_at, a.id, a.display_name FROM feed_cards c JOIN authors a ON a.id = c.author_id WHERE c.id = $1", r.LogNormal(2*time.Millisecond, 5*time.Millisecond))+ mbps := 40 + 110*r.Float()+ if pncMobileUA(r) {+ mbps = 5 + 15*r.Float()+ }+ r.Gap(r.Ms(0.5, 1.5) + time.Duration(float64(size)*8/(mbps*1e6)*float64(time.Second)))+ })++ authors := api.Endpoint("GET", "/api/authors/:id")+ authors.Traffic(seed.Daily(600), seed.Diurnal(), seed.Over(window))+ authors.Handler(func(r *seed.Req) {+ r.RootAttr("http.response.body.size", 900+r.IntN(700))+ r.Gap(r.Ms(0.3, 0.9))+ r.DB("SELECT id, display_name, bio, followers FROM authors WHERE id = $1", r.LogNormal(2*time.Millisecond, 5*time.Millisecond))+ r.Gap(r.Ms(0.3, 1))+ })++ s.Background(api, seed.TrafficOpts{Endpoints: 3, PerDay: 150})+ api.Host().CPU(seed.Flat(0.17, 0.26))+ api.Host().Memory(seed.Flat(0.44, 0.52))++ isMobile := func(r *seed.Request) bool {+ ua, _ := r.Root.Attrs["user_agent.original"].(string)+ return strings.Contains(ua, "iPhone") || strings.HasPrefix(ua, "okhttp")+ }+ s.Expose("mobileRequest", feed.Pick(func(r *seed.Request) bool {+ return isMobile(r) && r.Start.After(s.Ago(3*time.Hour)) && r.Duration() > 1500*time.Millisecond+ }))+ s.Expose("desktopRequest", feed.Pick(func(r *seed.Request) bool {+ return !isMobile(r) && r.Start.After(s.Ago(3*time.Hour)) && r.Duration() < 500*time.Millisecond+ }))+ s.Expose("p50Ms24h", func() any { return feed.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return feed.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/net_sequential.go b/cli/test/evals/internal/scenarios/perf/net_sequential.gonew file mode 100644index 00000000..048026eb--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/net_sequential.go@@ -0,0 +1,116 @@+package perf++import (+ "fmt"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/net-sequential", pncSeedSequential) }++// pncNestScope is the scope of the NestJS controller/handler spans.+const pncNestScope = "@opentelemetry/instrumentation-nestjs-core"++// pncNest wraps a NestJS route the way @opentelemetry/instrumentation-nestjs-core+// does: a request_context span for Controller.method with the handler span+// inside it.+func pncNest(r *seed.Req, controller, method, route string, body func(r *seed.Req)) {+ ctx := r.Span(controller+"."+method, func(r *seed.Req) {+ h := r.Span(method, body, seed.A("nestjs.type", "handler"), seed.A("nestjs.callback", method))+ h.Scope = pncNestScope+ }, seed.A("nestjs.type", "request_context"), seed.A("nestjs.controller", controller), seed.A("nestjs.callback", method),+ seed.A("http.route", route), seed.A("component", "@nestjs/core"), seed.A("nestjs.version", "11.1.6"))+ ctx.Scope = pncNestScope+}++// pncCall is r.Call with the concrete path and query the caller really+// requested, on both the CLIENT span and the downstream SERVER span.+func pncCall(r *seed.Req, target *seed.Endpoint, rtt time.Duration, path, query string) *seed.Span {+ c := r.Call(target, rtt)+ full := "http://" + target.Service().Name + ".internal:8080" + path+ if query != "" {+ full += "?" + query+ }+ c.Attrs["url.full"] = full+ if len(c.Children) > 0 {+ c.Children[0].Attrs["url.path"] = path+ if query != "" {+ c.Children[0].Attrs["url.query"] = query+ }+ }+ return c+}++// pncSeedSequential: the portal's dashboard endpoint (NestJS) calls four+// internal services one after another. Each service is fast and steady on+// its own; the dashboard pays their sum. Distractor: recommendations is the+// longest hop, but it is an internal service in line with its own p95, and no+// hop dominates.+func pncSeedSequential(s *seed.Scenario) error {+ p := s.Project("portal", seed.OTel)+ window := 3 * 24 * time.Hour++ web := p.Service("web-api", seed.Node("nestjs", "pg")).Versions("3.7.0")+ accounts := p.Service("accounts-service", seed.Go("chi", "pgx")).Versions("1.22.4")+ billing := p.Service("billing-service", seed.Java("spring-boot")).Versions("5.3.1")+ notify := p.Service("notifications-service", seed.Node("express", "pg")).Versions("2.0.9")+ reco := p.Service("recommendations-service", seed.Python("fastapi", "psycopg")).Versions("0.41.2")++ profile := accounts.Endpoint("GET", "/internal/accounts/:id/profile")+ profile.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.2, 0.6))+ r.DB("SELECT a.id, a.name, a.plan, p.seats FROM accounts a JOIN plans p ON p.id = a.plan_id WHERE a.id = $1", r.LogNormal(20*time.Millisecond, 45*time.Millisecond))+ r.DB("SELECT user_id, role FROM account_members WHERE account_id = $1", r.LogNormal(13*time.Millisecond, 32*time.Millisecond), seed.Rows(4+r.IntN(20)))+ r.DB("SELECT flag_key, enabled FROM feature_flags WHERE account_id = $1", r.LogNormal(9*time.Millisecond, 22*time.Millisecond), seed.Rows(18))+ r.Compute("encode response", r.Ms(1, 3))+ })+ summary := billing.Endpoint("GET", "/internal/billing/:accountId/summary")+ summary.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.5))+ r.DB("SELECT id, number, status, total_cents, due_at FROM invoices WHERE account_id = ? ORDER BY due_at DESC LIMIT 3", r.LogNormal(18*time.Millisecond, 42*time.Millisecond), seed.Rows(3))+ r.DB("SELECT SUM(amount_cents) FROM ledger_entries WHERE account_id = ? AND posted_at >= ?", r.LogNormal(52*time.Millisecond, 110*time.Millisecond))+ r.Compute("BillingSummaryMapper.toDto", r.Ms(1, 3))+ })+ unread := notify.Endpoint("GET", "/internal/notifications/unread")+ unread.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ r.DB("SELECT count(*) FROM notifications WHERE account_id = $1 AND read_at IS NULL", r.LogNormal(22*time.Millisecond, 50*time.Millisecond))+ r.DB("SELECT id, kind, title, created_at FROM notifications WHERE account_id = $1 AND read_at IS NULL ORDER BY created_at DESC LIMIT 5", r.LogNormal(16*time.Millisecond, 38*time.Millisecond), seed.Rows(5))+ r.Compute("serialize", r.Ms(0.5, 2))+ })+ suggest := reco.Endpoint("GET", "/internal/recommendations")+ suggest.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.2))+ r.Cache("GET", "reco:features", r.Ms(0.6, 1.8))+ r.DB("SELECT item_id, category, popularity FROM catalog_items WHERE category = ANY(%s) AND active LIMIT 400", r.LogNormal(24*time.Millisecond, 55*time.Millisecond), seed.Rows(400))+ r.Compute("score_candidates", r.LogNormal(72*time.Millisecond, 130*time.Millisecond))+ })++ dash := web.Endpoint("GET", "/api/dashboard")+ dash.Traffic(seed.Daily(800), seed.Diurnal(), seed.Over(window))+ dash.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.05, Max: 12})+ dash.Handler(func(r *seed.Req) {+ acct := fmt.Sprint(20000 + r.IntN(60000))+ r.Gap(r.Ms(0.6, 1.6)) // auth guard+ pncNest(r, "DashboardController", "getDashboard", "/api/dashboard", func(r *seed.Req) {+ pncCall(r, profile, r.Ms(1.2, 3), "/internal/accounts/"+acct+"/profile", "")+ pncCall(r, summary, r.Ms(1.2, 3), "/internal/billing/"+acct+"/summary", "")+ pncCall(r, unread, r.Ms(1.2, 3), "/internal/notifications/unread", "accountId="+acct)+ pncCall(r, suggest, r.Ms(1.2, 3), "/internal/recommendations", "accountId="+acct+"&limit=8")+ c := r.Compute("DashboardPresenter.compose", r.Ms(2, 6))+ c.Scope = "web-api"+ })+ r.Gap(r.Ms(0.3, 0.8))+ })++ s.Background(web, seed.TrafficOpts{Endpoints: 4, PerDay: 150})+ web.Host().CPU(seed.Flat(0.16, 0.27))+ web.Host().Memory(seed.Flat(0.44, 0.52))++ s.Expose("slowRequest", dash.Slowest(24*time.Hour))+ s.Expose("p50Ms24h", func() any { return dash.Percentile(0.5, 24*time.Hour).Seconds() * 1000 })+ s.Expose("p95Ms24h", func() any { return dash.Percentile(0.95, 24*time.Hour).Seconds() * 1000 })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pap_helpers.go b/cli/test/evals/internal/scenarios/perf/pap_helpers.gonew file mode 100644index 00000000..aeeb6c0f--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pap_helpers.go@@ -0,0 +1,38 @@+package perf++import (+ "fmt"+ "hash/fnv"+ "sort"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++// Helpers shared by the application and infrastructure perf seeds.++// papInterval is a stretch of wall time during which some work ran.+type papInterval struct{ from, to time.Time }++// papJitter varies an endpoint's rate by a few percent per hour, so the same+// hours of two days never carry identical request counts.+func papJitter(salt string, depth float64) seed.Shape {+ return seed.RateFunc(func(t time.Time) float64 {+ h := fnv.New64a()+ fmt.Fprintf(h, "%s|%s", salt, t.UTC().Format("2006-01-02T15"))+ u := float64(h.Sum64()>>11) / float64(1<<53)+ return 1 - depth + 2*depth*u+ })+}++// papActive counts the intervals that cover t. ivs must be sorted by from.+func papActive(ivs []papInterval, t time.Time, maxLen time.Duration) int {+ i := sort.Search(len(ivs), func(i int) bool { return !ivs[i].from.Before(t.Add(-maxLen)) })+ n := 0+ for ; i < len(ivs) && !ivs[i].from.After(t); i++ {+ if t.Before(ivs[i].to) {+ n+++ }+ }+ return n+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_helpers.go b/cli/test/evals/internal/scenarios/perf/pct_helpers.gonew file mode 100644index 00000000..3ac973bb--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_helpers.go@@ -0,0 +1,27 @@+package perf++import (+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++// pctMs exposes a duration as float milliseconds rounded to one decimal.+func pctMs(d time.Duration) float64 {+ return math.Round(float64(d)/float64(time.Millisecond)*10) / 10+}++// pctLazyMs is a lazy Expose of an endpoint percentile over [from, to).+func pctLazyMs(e *seed.Endpoint, p float64, from, to time.Time) func() any {+ return func() any { return pctMs(e.PercentileBetween(p, from, to)) }+}++// pctRFC3339 formats a time for templates.+func pctRFC3339(t time.Time) string { return t.UTC().Format(time.RFC3339) }++// pctDiurnal is seed.Diurnal's factor, so CPU can follow the same load.+func pctDiurnal(t time.Time) float64 {+ h := float64(t.Hour()) + float64(t.Minute())/60+ return 1 + 0.6*math.Cos((h-14)/24*2*math.Pi)+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_xs_android_timeouts.go b/cli/test/evals/internal/scenarios/perf/pct_xs_android_timeouts.gonew file mode 100644index 00000000..349be0c3--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_xs_android_timeouts.go@@ -0,0 +1,181 @@+package perf++import (+ "fmt"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/xs-android-timeouts", pctSeedXSAndroidTimeouts) }++// pctAndroidTimeoutStack is the uncaught SocketTimeoutException the Android+// app crashes with when OkHttp's 10s read timeout fires on POST+// /api/checkout (deobfuscated with the uploaded mapping).+const pctAndroidTimeoutStack = `java.net.SocketTimeoutException: timeout+ at okio.SocketAsyncTimeout.newTimeoutException(JvmOkio.kt:147)+ at okio.AsyncTimeout.access$newTimeoutException(AsyncTimeout.kt:186)+ at okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:390)+ at okio.RealBufferedSource.indexOf(RealBufferedSource.kt:436)+ at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:329)+ at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)+ at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180)+ at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110)+ at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)+ at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)+ at com.tracewayapp.traceway.network.TracewayOkHttpInterceptor.intercept(TracewayOkHttpInterceptor.kt:31)+ at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)+ at com.example.shop.net.AuthInterceptor.intercept(AuthInterceptor.kt:18)+ at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)+ at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)+ at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)+ at retrofit2.OkHttpCall.execute(OkHttpCall.java:207)+ at com.example.shop.checkout.CheckoutRepository$placeOrder$2.invokeSuspend(CheckoutRepository.kt:31)+ at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)+ at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:101)+ at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:113)+ at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)+ at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)+ at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)+ at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)+ at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)+ Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@4b1c2a8, Dispatchers.Main.immediate]`++// pctDeadlockStack is what Spring reports when two checkouts lock the same+// inventory rows in opposite order.+const pctDeadlockStack = `org.springframework.dao.CannotAcquireLockException: could not execute statement [ERROR: deadlock detected+ Detail: Process 31877 waits for ShareLock on transaction 9182736; blocked by process 31790.+Process 31790 waits for ShareLock on transaction 9182741; blocked by process 31877.+ Where: while updating tuple (412,7) in relation "inventory"] [update inventory set reserved=reserved+? where sku=? and available-reserved>=?]+ at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:299)+ at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:244)+ at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)+ at com.example.shop.checkout.InventoryRepository.reserve(InventoryRepository.java)+ at com.example.shop.checkout.CheckoutService.placeOrder(CheckoutService.java:39)+ at com.example.shop.checkout.CheckoutController.checkout(CheckoutController.java:27)+Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected+ at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2733)+ at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:155)+ at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)`++// pctSeedXSAndroidTimeouts: a flash sale started three hours ago. Every+// checkout reserves stock with UPDATE inventory inside a transaction that+// also calls the payment provider, so checkouts of the sale item queue on+// that row's lock for seconds; some run past OkHttp's 10s read timeout and+// the Android app, which does not catch the SocketTimeoutException, crashes.+// Expected: lock_transaction_contention. Distractors: an app bug (the crash+// is unhandled) and connection-pool exhaustion (there is no wait before the+// first query; the time is inside the UPDATE).+func pctSeedXSAndroidTimeouts(s *seed.Scenario) error {+ app := s.Project("shop-android", seed.Mobile, "android")+ apiProj := s.Project("commerce-api", seed.OTel)+ saleAt := s.Ago(3*time.Hour + 10*time.Minute)+ window := 2 * 24 * time.Hour+ api := apiProj.Service("checkout-api", seed.Java("spring")).Versions("7.4.2")++ checkout := api.Endpoint("POST", "/api/checkout")+ checkout.Traffic(seed.Daily(600), seed.Diurnal(), seed.Burst(saleAt, s.Now, 18), seed.Over(window))+ checkout.Detail(seed.DetailPolicy{Recent: 3 * time.Hour, Rate: 0.01, Max: 20})+ checkout.Handler(func(r *seed.Req) {+ sale := !r.Start().Before(saleAt)+ hot := sale && r.Chance(0.75)+ items := 1 + r.IntN(3)+ r.Gap(r.Ms(0.6, 1.5))+ r.DB("select c.id, ci.sku, ci.qty, ci.unit_price_cents from carts c join cart_items ci on ci.cart_id=c.id where c.id=?", r.LogNormal(3*time.Millisecond, 9*time.Millisecond), seed.Rows(items))+ for i := 0; i < items; i++ {+ wait := r.LogNormal(2*time.Millisecond, 7*time.Millisecond)+ if hot && i == 0 {+ wait = r.LogNormal(1800*time.Millisecond, 14*time.Second)+ }+ r.DB("update inventory set reserved=reserved+? where sku=? and available-reserved>=?", wait, seed.Rows(1))+ if hot && i == 1 && r.Chance(0.012) {+ r.Error(seed.Exc{Type: "org.springframework.dao.CannotAcquireLockException", Message: "could not execute statement [ERROR: deadlock detected]", Stack: pctDeadlockStack, Escaped: true})+ r.Log(seed.SevWarn, "SQL Error: 0, SQLState: 40P01", seed.A("logger.name", "org.hibernate.engine.jdbc.spi.SqlExceptionHelper"))+ r.Log(seed.SevError, "ERROR: deadlock detected\n Detail: Process 31877 waits for ShareLock on transaction 9182736; blocked by process 31790.", seed.A("logger.name", "org.hibernate.engine.jdbc.spi.SqlExceptionHelper"))+ r.Status(500)+ return+ }+ }+ r.HTTP("POST", "https://api.payments.example/v1/charges", r.LogNormal(420*time.Millisecond, 1100*time.Millisecond), 201)+ r.DB("insert into orders (id, cart_id, total_cents, currency, payment_ref, status, created_at) values (?, ?, ?, ?, ?, ?, ?)", r.LogNormal(3*time.Millisecond, 9*time.Millisecond))+ r.DB("insert into order_lines (order_id, sku, qty, unit_price_cents) values (?, ?, ?, ?)", r.LogNormal(2*time.Millisecond, 8*time.Millisecond))+ r.Gap(r.Ms(0.4, 1.2))+ })+ cart := api.Endpoint("GET", "/api/cart")+ cart.Traffic(seed.Daily(900), seed.Diurnal(), seed.Burst(saleAt, s.Now, 6), seed.Over(window))+ cart.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.2))+ r.DB("select c.id, ci.sku, ci.qty, ci.unit_price_cents from carts c join cart_items ci on ci.cart_id=c.id where c.id=?", r.LogNormal(3*time.Millisecond, 9*time.Millisecond), seed.Rows(1+r.IntN(4)))+ r.Compute("render cart", r.Ms(1, 4))+ })+ addItem := api.Endpoint("POST", "/api/cart/items")+ addItem.Traffic(seed.Daily(500), seed.Diurnal(), seed.Burst(saleAt, s.Now, 6), seed.Over(window))+ addItem.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.5, 1.2))+ r.DB("select available, reserved from inventory where sku=?", r.LogNormal(2*time.Millisecond, 7*time.Millisecond), seed.Rows(1))+ r.DB("insert into cart_items (cart_id, sku, qty, unit_price_cents) values (?, ?, ?, ?) on conflict (cart_id, sku) do update set qty=cart_items.qty+excluded.qty", r.LogNormal(3*time.Millisecond, 9*time.Millisecond))+ })+ s.Background(api, seed.TypicalTraffic(4))+ api.Host().CPU(seed.Step(saleAt, seed.Flat(0.2, 0.3), seed.Flat(0.34, 0.46)))+ api.Host().Memory(seed.Flat(0.55, 0.63))++ // The Android side: checkouts that ran past 10s, for two in three (the+ // rest are web and iOS customers), crashed the app.+ const readTimeout = 10 * time.Second+ devices := []string{"Pixel 8 / Android 15", "Samsung SM-S921B / Android 14", "Pixel 7a / Android 15", "Xiaomi 23129RAA4G / Android 14", "Samsung SM-A546B / Android 14"}+ var newest *seed.Request+ crashes := 0+ // Crashes are derived from the generated backend requests, so they are+ // built lazily, once, when the scenario flushes.+ done := false+ build := func() {+ if done {+ return+ }+ done = true+ for i, r := range checkout.Requests() {+ if r.Start.Before(saleAt) || r.Duration() < readTimeout+200*time.Millisecond || r.Start.After(s.Ago(2*time.Minute)) || i%3 == 0 {+ continue+ }+ r.KeepDetail()+ newest = r+ at := r.Start.Add(readTimeout + 15*time.Millisecond)+ cartAt := r.Start.Add(-time.Duration(25+i%50) * time.Second)+ app.Crash(seed.Crash{+ At: at, Stack: pctAndroidTimeoutStack, AppVersion: "5.18.0", Device: devices[i%len(devices)],+ Attributes: map[string]string{"screen": "CheckoutActivity"},+ Actions: []seed.Action{+ {Type: "navigation", At: cartAt.Add(-3 * time.Second), From: "ProductActivity", To: "CartActivity"},+ {Type: "network", At: cartAt, Method: "GET", URL: "https://api.shop.example/api/cart", DurationMs: float64(90 + i%120), Status: 200, RespBytes: 1800 + i%900},+ {Type: "navigation", At: cartAt.Add(4 * time.Second), From: "CartActivity", To: "CheckoutActivity"},+ {Type: "network", At: at, Method: "POST", URL: "https://api.shop.example/api/checkout", DurationMs: float64(readTimeout.Milliseconds()) + 4},+ },+ Logs: []seed.ActionLog{{At: r.Start.Add(-20 * time.Millisecond), Level: "info", Message: fmt.Sprintf("CheckoutViewModel: placing order (%d items)", 1+i%3)}},+ })+ crashes+++ }+ }+ s.Expose("androidCrashes", func() any { build(); return crashes })+ s.Expose("slowRequest", func() any {+ build()+ if newest == nil {+ return nil+ }+ return newest.Ref()+ })+ s.Expose("saleAt", pctRFC3339(saleAt))+ s.Expose("p95MsBefore", pctLazyMs(checkout, 0.95, s.Ago(window), saleAt))+ s.Expose("p95MsSale", pctLazyMs(checkout, 0.95, saleAt, s.Now))+ s.Expose("p50MsSale", pctLazyMs(checkout, 0.5, saleAt, s.Now))+ s.Expose("primaryProject", "shop-android")+ s.AfterFlush(func() error {+ hash, err := app.GroupMatching(24*time.Hour, "SocketTimeoutException")+ if err != nil {+ return err+ }+ s.Expose("hash", hash)+ return nil+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pct_xs_ios_watchdog.go b/cli/test/evals/internal/scenarios/perf/pct_xs_ios_watchdog.gonew file mode 100644index 00000000..be91e45a--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pct_xs_ios_watchdog.go@@ -0,0 +1,170 @@+package perf++import (+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/xs-ios-watchdog", pctSeedXSIOSWatchdog) }++// pctIOSWatchdogStack is the launch watchdog termination as the app's+// MetricKit forwarder reports it (Traceway iOS frame format, symbolicated+// with the uploaded dSYM; system frames stay image+offset).+const pctIOSWatchdogStack = `Watchdog termination: EXC_CRASH (SIGKILL), FRONTBOARD 0x8badf00d scene-create watchdog transgression: exhausted real (wall clock) time allowance of 20.00 seconds+*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***+os: ios arch: arm64+#0 libsystem_kernel.dylib+0x1c48+#1 libdispatch.dylib+0x4a24+#2 libdispatch.dylib+0x49d0+#3 ConfigClient.fetchBlocking(timeout:) (ConfigClient.swift:33:23)+#4 RemoteConfig.loadAtLaunch() (RemoteConfig.swift:18:37)+#5 AppDelegate.application(_:didFinishLaunchingWithOptions:) (AppDelegate.swift:17:29)+#6 UIKitCore+0x2f1a84+#7 UIKitCore+0x2f0e2c+#8 UIKitCore+0x2ef9c0+#9 FrontBoardServices+0x1f2c8+#10 libdispatch.dylib+0x3fa0+#11 CoreFoundation+0x77d0c+#12 GraphicsServices+0x1980+#13 UIKitCore+0x3c3ef8+#14 Shop+0x6a4c+#15 dyld+0x5e4c`++// pctSeedXSIOSWatchdog: two marketing pushes (5h and 1h35m ago) each set off+// a wave of app launches. Every launch calls GET /api/config and registers+// its push token, and the mobile API's database pool (8 connections, shared+// with the feed) runs dry: requests wait up to ~30s for a connection before+// their first query. The iOS app fetches the config synchronously on the+// main thread during launch, so launches that wait past ~20s are killed by+// the watchdog. No app release in three weeks. Expected:+// connection_pool_exhaustion. Distractor: a pure client bug (the blocking+// fetch is real, but it only bites since the API started taking 20s).+func pctSeedXSIOSWatchdog(s *seed.Scenario) error {+ app := s.Project("shop-ios", seed.Mobile, "ios")+ apiProj := s.Project("mobile-api", seed.OTel)+ pushes := []time.Time{s.Ago(5*time.Hour + 10*time.Minute), s.Ago(95 * time.Minute)}+ window := 2 * 24 * time.Hour+ api := apiProj.Service("mobile-api", seed.Go("nethttp", "sql")).Versions("2.9.0")++ // wave is how hard a push wave is hitting at t: 1 at the push, decaying+ // over a few minutes, 0 outside the waves.+ decay := func(t time.Time, tau float64) float64 {+ for _, p := range pushes {+ if d := t.Sub(p); d >= 0 && d < 40*time.Minute {+ return math.Exp(-d.Minutes() / tau)+ }+ }+ return 0+ }+ wave := func(t time.Time) float64 { return decay(t, 4) }+ // The connection queue builds up faster than it drains.+ backlog := func(t time.Time) float64 { return decay(t, 9) }+ // A push reaches every user whatever the hour, so the wave adds the+ // same number of launches by day or night (it is divided by the+ // day/night factor the traffic is multiplied by).+ launches := seed.RateFunc(func(t time.Time) float64 { return 1 + 45*wave(t)/pctDiurnal(t) })+ poolWait := func(r *seed.Req) time.Duration {+ w := backlog(r.Start())+ if w < 0.05 {+ return r.Ms(0.02, 0.3)+ }+ return r.LogNormal(time.Duration(12000*w)*time.Millisecond, time.Duration(36000*w)*time.Millisecond)+ }++ config := api.Endpoint("GET", "/api/config")+ config.Traffic(seed.Daily(2400), seed.Diurnal(), launches, seed.Over(window))+ config.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.01, Max: 20})+ config.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ r.Gap(poolWait(r))+ r.DB("SELECT key, value, rollout_pct FROM feature_flags WHERE platform = $1 AND enabled", r.LogNormal(2*time.Millisecond, 6*time.Millisecond), seed.Rows(40+r.IntN(10)))+ r.DB("SELECT min_supported, latest FROM app_versions WHERE platform = $1", r.LogNormal(1*time.Millisecond, 4*time.Millisecond), seed.Rows(1))+ r.Compute("encode response", r.Ms(0.3, 1))+ })+ devices := api.Endpoint("POST", "/api/devices")+ devices.Traffic(seed.Daily(1200), seed.Diurnal(), launches, seed.Over(window))+ devices.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ r.Gap(poolWait(r))+ r.DB("INSERT INTO push_tokens (user_id, platform, token, updated_at) VALUES ($1, $2, $3, now()) ON CONFLICT (token) DO UPDATE SET user_id = EXCLUDED.user_id, updated_at = now()", r.LogNormal(3*time.Millisecond, 10*time.Millisecond))+ })+ feed := api.Endpoint("GET", "/api/feed")+ feed.Traffic(seed.Daily(1500), seed.Diurnal(), launches, seed.Over(window))+ feed.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ r.Gap(poolWait(r))+ r.DB("SELECT p.id, p.title, p.image_url, p.price_cents, p.badge FROM promotions p WHERE p.active AND p.segment = ANY($1) ORDER BY p.priority DESC LIMIT 30", r.LogNormal(120*time.Millisecond, 380*time.Millisecond), seed.Rows(30))+ r.DB("SELECT product_id, count(*) FROM recent_views WHERE user_id = $1 GROUP BY product_id ORDER BY count(*) DESC LIMIT 20", r.LogNormal(60*time.Millisecond, 220*time.Millisecond), seed.Rows(20))+ r.Compute("assemble feed", r.Ms(2, 6))+ })+ // Everything that touches the database shares the pool, so every route+ // waits in a wave (no unrelated background routes on this service).+ product := api.Endpoint("GET", "/api/products/:id")+ product.Traffic(seed.Daily(1800), seed.Diurnal(), seed.RateFunc(func(t time.Time) float64 { return 1 + 8*wave(t)/pctDiurnal(t) }), seed.Over(window))+ product.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.3, 0.8))+ r.Gap(poolWait(r))+ r.DB("SELECT id, name, description, price_cents, images FROM products WHERE id = $1", r.LogNormal(2*time.Millisecond, 7*time.Millisecond), seed.Rows(1))+ r.Compute("encode response", r.Ms(0.3, 1))+ })+ api.Host().CPU(seed.Flat(0.18, 0.32))+ api.Host().Memory(seed.Flat(0.48, 0.56))++ // The iOS side: a launch whose config fetch had not returned after ~19s+ // (the rest of launch takes about a second) is killed by the watchdog;+ // half the launches are iOS.+ const killAfter = 19500 * time.Millisecond+ models := []string{"iPhone16,2 / iOS 18.0.1", "iPhone15,3 / iOS 18.0", "iPhone14,5 / iOS 17.6.1", "iPhone17,1 / iOS 18.0.1", "iPhone13,2 / iOS 17.5"}+ crashes := 0+ var newest *seed.Request+ // Crashes are derived from the generated backend requests, so they are+ // built lazily, once, when the scenario flushes.+ done := false+ build := func() {+ if done {+ return+ }+ done = true+ for i, r := range config.Requests() {+ if r.Duration() < killAfter || i%2 == 0 || r.Start.After(s.Ago(2*time.Minute)) {+ continue+ }+ r.KeepDetail()+ newest = r+ app.Crash(seed.Crash{+ At: r.Start.Add(20*time.Second + time.Duration(200+i%600)*time.Millisecond), Stack: pctIOSWatchdogStack,+ AppVersion: "4.12.0", Device: models[i%len(models)],+ Attributes: map[string]string{"source": "metrickit", "phase": "launch"},+ })+ crashes+++ }+ }+ s.Expose("watchdogCrashes", func() any { build(); return crashes })+ s.Expose("slowRequest", func() any {+ build()+ if newest == nil {+ return nil+ }+ return newest.Ref()+ })+ s.Expose("push1", pctRFC3339(pushes[0]))+ s.Expose("push2", pctRFC3339(pushes[1]))+ s.Expose("wave2From", pctRFC3339(pushes[1].Add(-5*time.Minute)))+ s.Expose("wave2To", pctRFC3339(pushes[1].Add(20*time.Minute)))+ s.Expose("p50Ms24h", pctLazyMs(config, 0.5, s.Ago(24*time.Hour), s.Now))+ s.Expose("p99Ms24h", pctLazyMs(config, 0.99, s.Ago(24*time.Hour), s.Now))+ s.Expose("p95MsWave2", pctLazyMs(config, 0.95, pushes[1], pushes[1].Add(10*time.Minute)))+ s.Expose("primaryProject", "shop-ios")+ s.AfterFlush(func() error {+ hash, err := app.GroupMatching(24*time.Hour, "0x8badf00d")+ if err != nil {+ return err+ }+ s.Expose("hash", hash)+ return nil+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pdb_helpers.go b/cli/test/evals/internal/scenarios/perf/pdb_helpers.gonew file mode 100644index 00000000..dfa5e8dd--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pdb_helpers.go@@ -0,0 +1,76 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++// pdbRoute is one healthy route of a perf-db scenario: a few fast queries+// and a small encode step, so the slow route has neighbours that look normal.+type pdbRoute struct {+ method, route string+ perDay int+ queries []string+ // dbMedianMs is the median of each query (default 2ms).+ dbMedianMs float64+}++// pdbHealthy adds healthy routes to a service over window.+func pdbHealthy(api *seed.Service, window time.Duration, routes ...pdbRoute) {+ for _, rt := range routes {+ rt := rt+ med := rt.dbMedianMs+ if med == 0 {+ med = 2+ }+ e := api.Endpoint(rt.method, rt.route)+ e.Traffic(seed.Daily(rt.perDay), seed.Diurnal(), seed.Over(window))+ e.Detail(seed.DetailPolicy{Recent: 6 * time.Hour, Rate: 0.02, Max: 4})+ e.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.4, 1.2))+ for _, q := range rt.queries {+ m := time.Duration(med * float64(time.Millisecond))+ r.DB(q, r.LogNormal(m, m*5))+ }+ r.Compute("serialize response", r.Ms(0.2, 1.4))+ r.Gap(r.Ms(0.1, 0.5))+ })+ }+}++// pdbSlowest returns the slowest request of e started after since that+// satisfies keep, forcing its detail, as a lazy seed.json reference.+func pdbSlowest(e *seed.Endpoint, since time.Time, keep func(r *seed.Request) bool) func() any {+ return func() any {+ r := pdbSlowestRequest(e, since, keep)+ if r == nil {+ return nil+ }+ return r.Ref()+ }+}++func pdbSlowestRequest(e *seed.Endpoint, since time.Time, keep func(r *seed.Request) bool) *seed.Request {+ var best *seed.Request+ for _, r := range e.Requests() {+ if r.Start.Before(since) || (keep != nil && !keep(r)) {+ continue+ }+ if best == nil || r.Duration() > best.Duration() {+ best = r+ }+ }+ if best != nil {+ best.KeepDetail()+ }+ return best+}++// pdbMs is a lazy percentile in milliseconds over [from, to).+func pdbMs(e *seed.Endpoint, p float64, from, to time.Time) func() any {+ return func() any { return float64(e.PercentileBetween(p, from, to)) / float64(time.Millisecond) }+}++// pdbDur converts milliseconds to a duration.+func pdbDur(ms float64) time.Duration { return time.Duration(ms * float64(time.Millisecond)) }diff --git a/cli/test/evals/internal/scenarios/perf/pdb_missing_index.go b/cli/test/evals/internal/scenarios/perf/pdb_missing_index.gonew file mode 100644index 00000000..4f76432f--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pdb_missing_index.go@@ -0,0 +1,97 @@+package perf++import (+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/db-missing-index", pdbSeedMissingIndex) }++// pdbSeedMissingIndex: the user lookup filters users by email, a column with+// no index, so Postgres reads the whole table on every call. The table grows+// steadily with signups, and so does the one query span, for every request+// (it returns one row or none). Primary-key lookups on the same table stay at+// a millisecond or two. Distractor: release 1.13.0 two weeks ago changed the+// invite endpoint (two extra primary-key reads and an email enqueue); the+// lookup curve has no step there and the release did not touch it.+func pdbSeedMissingIndex(s *seed.Scenario) error {+ window := 29 * 24 * time.Hour+ deployAt := s.Ago(14*24*time.Hour + 2*time.Hour) // two hours after the head commit+ p := s.Project("teamspace", seed.OTel)+ api := p.Service("accounts-api", seed.Python("fastapi", "sqlalchemy")).+ Versions("1.12.4", seed.Deploy{At: deployAt, Version: "1.13.0"})++ lookup := api.Endpoint("GET", "/api/users/lookup")+ lookup.Traffic(seed.Daily(160), seed.Diurnal(), seed.Over(window))+ lookup.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.25, Max: 12})+ lookup.Handler(func(r *seed.Req) {+ users := r.Lerp(1.62e6, 3.94e6) // rows in users; signups accrue steadily+ r.Gap(r.Ms(0.6, 1.6)) // session dependency+ found := !r.Chance(0.12)+ rows := 0+ if found {+ rows = 1+ }+ // A sequential scan reads every row whether or not one matches.+ scan := pdbDur(users / 7600 * (0.93 + 0.14*r.Float()))+ r.DB("SELECT users.id, users.email, users.display_name, users.org_id, users.created_at FROM users WHERE users.email = $1::VARCHAR",+ scan, seed.Rows(rows))+ if !found {+ r.Status(404)+ r.Gap(r.Ms(0.2, 0.6))+ return+ }+ r.DB("SELECT orgs.id, orgs.name, orgs.plan FROM orgs WHERE orgs.id = $1::INTEGER", r.Ms(0.7, 2.2), seed.Rows(1))+ r.Gap(r.Ms(0.3, 0.9)) // pydantic response model+ })++ invites := api.Endpoint("POST", "/api/invites")+ invites.Traffic(seed.Daily(50), seed.Diurnal(), seed.Over(window))+ invites.Detail(seed.DetailPolicy{Recent: 12 * time.Hour, Rate: 0.1, Max: 4})+ invites.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.6, 1.6))+ r.DB("SELECT org_members.role FROM org_members WHERE org_members.org_id = $1::INTEGER AND org_members.user_id = $2::INTEGER", r.Ms(0.7, 2.0), seed.Rows(1))+ r.DB("INSERT INTO invites (org_id, email, role, token, invited_by, created_at) VALUES ($1::INTEGER, $2::VARCHAR, $3::VARCHAR, $4::VARCHAR, $5::INTEGER, $6::TIMESTAMP WITHOUT TIME ZONE) RETURNING invites.id",+ r.Ms(1.4, 3.8))+ if r.Version() == "1.13.0" {+ // 1.13.0: the invite email names the inviter and the org.+ r.DB("SELECT orgs.id, orgs.name, orgs.plan FROM orgs WHERE orgs.id = $1::INTEGER", r.Ms(0.7, 2.0), seed.Rows(1))+ r.DB("SELECT users.id, users.email, users.display_name, users.org_id, users.created_at FROM users WHERE users.id = $1::INTEGER", r.Ms(0.8, 2.4), seed.Rows(1))+ r.Cache("RPUSH", "queue:emails", r.Ms(0.3, 0.9))+ }+ r.Gap(r.Ms(0.3, 0.9))+ })++ pdbHealthy(api, window,+ pdbRoute{method: "GET", route: "/api/me", perDay: 45, queries: []string{+ "SELECT users.id, users.email, users.display_name, users.org_id, users.created_at FROM users WHERE users.id = $1::INTEGER",+ "SELECT orgs.id, orgs.name, orgs.plan FROM orgs WHERE orgs.id = $1::INTEGER",+ }, dbMedianMs: 1.2},+ pdbRoute{method: "GET", route: "/api/orgs/{org_id}/members", perDay: 30, queries: []string{+ "SELECT users.id, users.email, users.display_name, org_members.role FROM org_members JOIN users ON users.id = org_members.user_id WHERE org_members.org_id = $1::INTEGER ORDER BY users.display_name LIMIT $2::INTEGER",+ }, dbMedianMs: 3},+ pdbRoute{method: "GET", route: "/api/invites", perDay: 20, queries: []string{+ "SELECT invites.id, invites.email, invites.role, invites.created_at FROM invites WHERE invites.org_id = $1::INTEGER AND invites.accepted_at IS NULL ORDER BY invites.created_at DESC",+ }},+ pdbRoute{method: "POST", route: "/api/invites/{token}/accept", perDay: 15, queries: []string{+ "SELECT invites.id, invites.org_id, invites.email, invites.role FROM invites WHERE invites.token = $1::VARCHAR",+ "INSERT INTO org_members (org_id, user_id, role) VALUES ($1::INTEGER, $2::INTEGER, $3::VARCHAR)",+ }},+ pdbRoute{method: "PATCH", route: "/api/me", perDay: 12, queries: []string{+ "UPDATE users SET display_name=$1::VARCHAR WHERE users.id = $2::INTEGER",+ }, dbMedianMs: 1.5},+ )+ api.Host().CPU(seed.Flat(0.17, 0.26))+ api.Host().Memory(seed.Flat(0.41, 0.47))++ s.Expose("slowRequest", pdbSlowest(lookup, s.Ago(24*time.Hour), nil))+ s.Expose("deployAt", deployAt.Format(time.RFC3339))+ s.Expose("p50Ms24h", pdbMs(lookup, 0.5, s.Ago(24*time.Hour), s.Now))+ s.Expose("p95Ms24h", pdbMs(lookup, 0.95, s.Ago(24*time.Hour), s.Now))+ s.Expose("p50MsFirstWeek", pdbMs(lookup, 0.5, s.Ago(window), s.Ago(window-7*24*time.Hour)))+ s.Expose("invitesP50MsBeforeDeploy", pdbMs(invites, 0.5, s.Ago(window), deployAt))+ s.Expose("invitesP50MsAfterDeploy", pdbMs(invites, 0.5, deployAt, s.Now))+ return nil+}diff --git a/cli/test/evals/internal/scenarios/perf/pdb_pool_exhaustion.go b/cli/test/evals/internal/scenarios/perf/pdb_pool_exhaustion.gonew file mode 100644index 00000000..bfb2b1e7--- /dev/null+++ b/cli/test/evals/internal/scenarios/perf/pdb_pool_exhaustion.go@@ -0,0 +1,180 @@+package perf++import (+ "fmt"+ "math"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func init() { scenarios.Register("perf/db-pool-exhaustion", pdbSeedPoolExhaustion) }++// pdbPoolTimeout is the SQLAlchemy QueuePool message for pool_size=10,+// max_overflow=5, pool_timeout=30.+const pdbPoolTimeout = "QueuePool limit of size 10 overflow 5 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)"++// pdbSeedPoolExhaustion: checkout holds its pooled connection for the whole+// request, including the payment provider call. At the lunch and evening+// rushes the 15 connections (pool_size 10 + max_overflow 5) are all checked+// out, and requests wait for one before their first query: a gap with no+// span before the first DB span, seconds long, only at the peaks. A few wait+// the full pool_timeout and fail with sqlalchemy.exc.TimeoutError. The cart+// endpoint shares the pool and waits the same way although its only query+// takes 3ms; the shipping options, which need no connection, never do. The+// pool's own metric shows 15 used and 0 idle at exactly those hours.+// Distractors: a slow database (the inventory query takes 40-60ms at every+// hour) and the payment provider (150-260ms at every hour, peaks included).+func pdbSeedPoolExhaustion(s *seed.Scenario) error {+ window := 3 * 24 * time.Hour+ p := s.Project("marketplace", seed.OTel)+ api := p.Service("checkout-api", seed.Python("fastapi", "sqlalchemy")).Versions("5.3.1")++ // bump is 0 off-peak and 1 on the plateau of the lunch (12:00-13:00)+ // and evening (19:00-20:30) rushes, with 25-minute ramps.+ type rush struct{ from, to float64 }+ rushes := []rush{{12, 13}, {19, 20.5}}+ const ramp = 25.0 / 60+ bump := func(t time.Time) float64 {+ h := float64(t.Hour()) + float64(t.Minute())/60 + float64(t.Second())/3600+ best := 0.0+ for _, r := range rushes {+ var v float64+ switch {+ case h >= r.from && h <= r.to:+ v = 1+ case h > r.from-ramp && h < r.from:+ v = (h - (r.from - ramp)) / ramp+ case h > r.to && h < r.to+ramp:+ v = (r.to + ramp - h) / ramp+ }+ best = math.Max(best, v)+ }+ return best+ }+ load := func(t time.Time) float64 { return 1 + 2.4*bump(t) }+ // over is how far demand for connections exceeds the 15 the pool has,+ // 0..1; it is non-zero only near the plateau of a rush.+ over := func(t time.Time) float64 { return math.Max(0, (load(t)-2.7)/0.7) }++ // waitFor is the time a request spends in QueuePool.get before its first+ // query; ok is false when it waited the whole pool_timeout.+ waitFor := func(r *seed.Req) (time.Duration, bool) {+ o := over(r.Start())+ if o == 0 || !r.Chance(0.25+0.65*o) {+ return 0, true+ }+ w := time.Duration(o * float64(r.LogNormal(3200*time.Millisecond, 26*time.Second)))+ if w >= 30*time.Second {+ return 30*time.Second + r.Ms(1, 4), false+ }+ return w, true+ }+ poolTimeout := func(r *seed.Req, route, frame string) {+ r.Error(seed.Exc{Type: "sqlalchemy.exc.TimeoutError", Message: pdbPoolTimeout, Escaped: true, Stack: "Traceback (most recent call last):\n" ++ " File \"/usr/local/lib/python3.12/site-packages/fastapi/routing.py\", line 219, in run_endpoint_function\n return await run_in_threadpool(dependant.call, **values)\n" ++ " File \"/usr/local/lib/python3.12/site-packages/starlette/concurrency.py\", line 38, in run_in_threadpool\n return await anyio.to_thread.run_sync(func)\n" ++ frame ++ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/session.py\", line 2365, in execute\n return self._execute_internal(\n" ++ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/session.py\", line 2241, in _execute_internal\n conn = self._connection_for_bind(bind)\n" ++ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/base.py\", line 3297, in raw_connection\n return self.pool.connect()\n" ++ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/pool/base.py\", line 449, in connect\n return _ConnectionFairy._checkout(self)\n" ++ " File \"/usr/local/lib/python3.12/site-packages/sqlalchemy/pool/impl.py\", line 167, in _do_get\n raise exc.TimeoutError(\n" ++ "sqlalchemy.exc.TimeoutError: " + pdbPoolTimeout})+ r.Log(seed.SevError, "Exception in ASGI application", seed.A("exception.type", "sqlalchemy.exc.TimeoutError"), seed.A("exception.message", pdbPoolTimeout),+ seed.A("http.route", route))+ r.Status(500)+ }++ checkout := api.Endpoint("POST", "/api/checkout")+ checkout.Traffic(seed.Daily(1500), seed.RateFunc(load), seed.Over(window))+ checkout.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.03, Max: 14})+ checkout.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.8, 2.2)) // auth dependency, body validation+ wait, ok := waitFor(r)+ r.Gap(wait)+ cart := fmt.Sprintf("%08x-%04x-4%03x-%04x-%06x%06x", r.IntN(1<<30)|1<<28, r.IntN(1<<16), r.IntN(1<<12), 0x8000+r.IntN(1<<14), r.IntN(1<<24), r.IntN(1<<24))+ if !ok {+ r.Log(seed.SevWarn, "checkout failed for cart "+cart+": "+pdbPoolTimeout, seed.A("cart_id", cart))+ poolTimeout(r, "/api/checkout", " File \"/app/app/checkout.py\", line 32, in create_checkout\n return _checkout(body, x_user_id)\n"++ " File \"/app/app/checkout.py\", line 40, in _checkout\n items = session.execute(select(CartItem).where(CartItem.cart_id == body.cart_id)).scalars().all()\n")+ return+ }+ lines := 1 + r.IntN(5)+ r.DB("SELECT cart_items.sku, cart_items.qty, cart_items.unit_price_cents FROM cart_items WHERE cart_items.cart_id = %(cart_id_1)s::UUID", r.Ms(2, 5), seed.Rows(lines))+ r.DB("SELECT inventory.sku, inventory.available FROM inventory WHERE inventory.sku = ANY (%(sku_1)s::VARCHAR[])", r.Ms(40, 60), seed.Rows(lines))+ r.HTTP("POST", "https://api.payments.example/v1/charges", r.Ms(150, 260), 201)+ r.DB("INSERT INTO orders (id, cart_id, user_id, total_cents, status, created_at) VALUES (%(id)s::UUID, %(cart_id)s::UUID, %(user_id)s::INTEGER, %(total_cents)s::INTEGER, %(status)s::VARCHAR, now())", r.Ms(4, 8))+ r.DB("UPDATE inventory SET available=(inventory.available - %(param_1)s::INTEGER) WHERE inventory.sku = %(sku_1)s::VARCHAR", r.Ms(3, 6))+ r.DB("DELETE FROM cart_items WHERE cart_items.cart_id = %(cart_id_1)s::UUID", r.Ms(2, 4))+ r.Gap(r.Ms(0.5, 1.5))+ r.Status(201)+ })++ cartEp := api.Endpoint("GET", "/api/cart")+ cartEp.Traffic(seed.Daily(1100), seed.RateFunc(load), seed.Over(window))+ cartEp.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.02, Max: 8})+ cartEp.Handler(func(r *seed.Req) {+ r.Gap(r.Ms(0.6, 1.6))+ wait, ok := waitFor(r)+ r.Gap(wait)+ if !ok {+ poolTimeout(r, "/api/cart", " File \"/app/app/cart.py\", line 18, in get_cart\n rows = session.execute(select(CartItem).where(CartItem.cart_id == cart_id)).scalars().all()\n")+ return+ }+ r.DB("SELECT cart_items.sku, cart_items.qty, cart_items.unit_price_cents FROM cart_items WHERE cart_items.cart_id = %(cart_id_1)s::UUID", r.Ms(2, 5), seed.Rows(1+r.IntN(5)))+ r.Gap(r.Ms(0.4, 1.0))+ })++ // Shipping options come from config; no connection needed.+ shipping := api.Endpoint("GET", "/api/shipping/options")+ shipping.Traffic(seed.Daily(500), seed.RateFunc(load), seed.Over(window))+ shipping.Detail(seed.DetailPolicy{Recent: 10 * time.Hour, Rate: 0.01, Max: 3})+ shipping.Handler(func(r *seed.Req) { r.Gap(r.Ms(0.6, 1.8)) })++ // opentelemetry-instrumentation-sqlalchemy's pool usage metric.+ pool := map[string]any{"db.client.connection.pool.name": "checkout-db.internal:5432/checkout"}+ used := func(t time.Time) float64 {+ if over(t) > 0 {+ return 15+ }+ busy := 1 + 13*math.Pow((load(t)-1)/1.7, 1.4)*(0.9+0.2*math.Sin(float64(t.Unix()/600)))+ return math.Min(14, math.Round(busy))+ }+ withState := func(state string) map[string]any {+ m := map[string]any{"db.client.connection.state": state}+ for k, v := range pool {+ m[k] = v+ }+ return m+ }+ api.Gauge("db.client.connection.count", "{connection}", withState("used"), used)+ api.Gauge("db.client.connection.count", "{connection}", withState("idle"), func(t time.Time) float64 { return math.Max(0, 10-used(t)) })++ api.Host().CPU(seed.Plus(seed.Flat(0.27, 0.34), func(t time.Time) float64 { return 0.05 * bump(t) }))+ api.Host().Memory(seed.Flat(0.49, 0.55))++ day := s.Ago(24 * time.Hour)+ // A slow one a customer would notice (10+ s), not a request that waited+ // out the whole pool_timeout.+ s.Expose("slowRequest", pdbSlowest(checkout, day, func(r *seed.Request) bool {+ return r.Status == 201 && r.Duration() < 15*time.Second+ }))+ s.Expose("p50Ms24h", pdbMs(checkout, 0.5, day, s.Now))+ s.Expose("p95Ms24h", pdbMs(checkout, 0.95, day, s.Now))+ s.Expose("p99Ms24h", pdbMs(checkout, 0.99, day, s.Now))+ s.Expose("cartP99Ms24h", pdbMs(cartEp, 0.99, day, s.Now))+ s.Expose("timeouts", func() any {+ n := 0+ for _, e := range []*seed.Endpoint{checkout, cartEp} {+ for _, r := range e.Requests() {+ if r.Status == 500 {+ n+++ }+ }+ }+ return n+ })+ return nil+}diff --git a/cli/test/evals/internal/scenarios/registry.go b/cli/test/evals/internal/scenarios/registry.gonew file mode 100644index 00000000..95ad7ffe--- /dev/null+++ b/cli/test/evals/internal/scenarios/registry.go@@ -0,0 +1,75 @@+// Package scenarios holds one Go seed function per case, registered by the+// name a case's `seed:` field uses (perf/db-n-plus-1, ...). The per-suite+// subpackages register themselves; import scenarios/all to load every one.+package scenarios++import (+ "fmt"+ "sort"+ "sync"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++// Func seeds one scenario.+type Func func(s *seed.Scenario) error++var (+ mu sync.RWMutex+ registry = map[string]Func{}+)++// Register adds a seed function. Duplicate names panic at init.+func Register(name string, f Func) {+ mu.Lock()+ defer mu.Unlock()+ if _, dup := registry[name]; dup {+ panic("scenarios: duplicate seed " + name)+ }+ registry[name] = f+}++// Get returns a registered seed function.+func Get(name string) (Func, error) {+ mu.RLock()+ defer mu.RUnlock()+ f, ok := registry[name]+ if !ok {+ return nil, fmt.Errorf("no seed function registered as %q", name)+ }+ return f, nil+}++// Names lists every registered seed.+func Names() []string {+ mu.RLock()+ defer mu.RUnlock()+ out := make([]string, 0, len(registry))+ for k := range registry {+ out = append(out, k)+ }+ sort.Strings(out)+ return out+}++// Run seeds s with the named function, converting panics from the DSL (which+// panics on API errors to keep scenario code flat) into errors.+func Run(name string, s *seed.Scenario) (err error) {+ f, err := Get(name)+ if err != nil {+ return err+ }+ defer func() {+ if r := recover(); r != nil {+ if e, ok := r.(error); ok {+ err = fmt.Errorf("seed %s: %w", name, e)+ return+ }+ err = fmt.Errorf("seed %s: %v", name, r)+ }+ }()+ if err := f(s); err != nil {+ return fmt.Errorf("seed %s: %w", name, err)+ }+ return s.Flush()+}diff --git a/cli/test/evals/internal/seed/dsl.go b/cli/test/evals/internal/seed/dsl.gonew file mode 100644index 00000000..c260e1fd--- /dev/null+++ b/cli/test/evals/internal/seed/dsl.go@@ -0,0 +1,743 @@+// Package seed builds realistic Traceway telemetry for one eval scenario and+// sends it through the real ingest endpoints (OTLP/HTTP and native+// /api/report). The DSL owns consistency: a request's duration is the sum of+// its children plus explicit gaps, percentiles come from the generated+// requests, and deploys are service.version changes on the resource.+package seed++import (+ "crypto/sha256"+ "encoding/binary"+ "fmt"+ "math"+ "math/rand/v2"+ "sort"+ "time"+)++// ProjectKind selects how a project's telemetry arrives.+type ProjectKind int++const (+ // OTel is a backend project fed over OTLP (framework opentelemetry).+ OTel ProjectKind = iota+ // Browser is a frontend project fed by a Traceway browser SDK.+ Browser+ // Mobile is a mobile project fed by a Traceway mobile SDK.+ Mobile+)++// Scenario is the state of one seed run.+type Scenario struct {+ CaseID string+ Now time.Time+ RNG *rand.Rand+ Sink Sink++ projects []*Project+ services []*Service+ exposed map[string]any+ lazy map[string]func() any+ strings []string+ flushed bool+ stats Stats+ // SpanBudget caps the spans sent with full detail (default 20000).+ SpanBudget int+ afterFlush []func() error+}++// Stats summarizes what a seed sent.+type Stats struct {+ Requests int `json:"requests"`+ Spans int `json:"spans"`+ DetailedReqs int `json:"detailedRequests"`+ Logs int `json:"logs"`+ MetricPoints int `json:"metricPoints"`+ Exceptions int `json:"exceptions"`+ NativeFrames int `json:"nativeFrames"`+ IngestElapsed time.Duration `json:"ingestElapsed"`+}++// NewScenario starts a scenario anchored at now (truncated to the minute)+// with an RNG seeded from the case id, so a case always draws the same data.+func NewScenario(caseID string, now time.Time, sink Sink) *Scenario {+ sum := sha256.Sum256([]byte(caseID))+ seed1 := binary.BigEndian.Uint64(sum[:8])+ seed2 := binary.BigEndian.Uint64(sum[8:16])+ return &Scenario{+ CaseID: caseID,+ Now: now.UTC().Truncate(time.Minute),+ RNG: rand.New(rand.NewPCG(seed1, seed2)),+ Sink: sink,+ exposed: map[string]any{},+ lazy: map[string]func() any{},+ SpanBudget: 20000,+ }+}++// Stats returns what was sent.+func (s *Scenario) Stats() Stats { return s.stats }++// Strings returns every human-readable string the seed emitted (span names,+// attribute values, log bodies, stack traces, object names), for the leakage+// lint.+func (s *Scenario) Strings() []string { return s.strings }++func (s *Scenario) record(v ...string) {+ for _, x := range v {+ if x != "" {+ s.strings = append(s.strings, x)+ }+ }+}++// Project declares a Traceway project. framework defaults per kind.+func (s *Scenario) Project(name string, kind ProjectKind, framework ...string) *Project {+ fw := "opentelemetry"+ switch kind {+ case Browser:+ fw = "react"+ case Mobile:+ fw = "android"+ }+ if len(framework) > 0 {+ fw = framework[0]+ }+ p := &Project{s: s, Name: name, Kind: kind, Framework: fw}+ id, token, err := s.Sink.CreateProject(name, fw)+ if err != nil {+ panic(fmt.Errorf("create project %s: %w", name, err))+ }+ p.ID, p.token = id, token+ s.projects = append(s.projects, p)+ s.record(name)+ return p+}++// Projects returns the declared projects in order.+func (s *Scenario) Projects() []*Project { return s.projects }++// Expose records a value for seed.json (graders and the oracle read it; the+// agent never sees it). A func() any value is resolved at Flush.+func (s *Scenario) Expose(key string, v any) {+ if f, ok := v.(func() any); ok {+ s.lazy[key] = f+ return+ }+ s.exposed[key] = v+}++// Exposed returns the resolved exposed values.+func (s *Scenario) Exposed() map[string]any {+ out := map[string]any{}+ for k, v := range s.exposed {+ out[k] = v+ }+ return out+}++// AfterFlush schedules work that needs the telemetry in place (for example+// objects that reference an ingested exception).+func (s *Scenario) AfterFlush(f func() error) { s.afterFlush = append(s.afterFlush, f) }++// Ago returns Now minus d.+func (s *Scenario) Ago(d time.Duration) time.Time { return s.Now.Add(-d) }++// Project is one Traceway project.+type Project struct {+ s *Scenario+ Name string+ Kind ProjectKind+ Framework string+ ID string+ token string++ native []nativeFrame+}++// Service declares an OTel-instrumented service reporting to the project.+func (p *Project) Service(name string, stack Stack) *Service {+ svc := &Service{s: p.s, p: p, Name: name, Stack: stack, Env: "production", Hosts: []string{name + "-" + hostSuffix(p.s, name)}}+ p.s.services = append(p.s.services, svc)+ p.s.record(name)+ return svc+}++func hostSuffix(s *Scenario, name string) string {+ h := sha256.Sum256([]byte(s.CaseID + name))+ return fmt.Sprintf("%x-%x", h[0:3], h[3:5])+}++// Service is one deployable component.+type Service struct {+ s *Scenario+ p *Project+ Name string+ Stack Stack+ Env string+ Hosts []string+ deploys []Deploy+ // HostWeight biases request placement per host (default uniform).+ HostWeight map[string]float64++ endpoints []*Endpoint+ tasks []*Endpoint+ metrics []*metricSeries+ logs []logRecord+ defaultHost bool+}++// Deploy is a release: service.version changes at At.+type Deploy struct {+ At time.Time+ Version string+}++// Project returns the owning project.+func (svc *Service) Project() *Project { return svc.p }++// Instances replaces the host list (multi-instance service).+func (svc *Service) Instances(hosts ...string) *Service {+ svc.Hosts = hosts+ svc.s.record(hosts...)+ return svc+}++// Versions sets the release history: the first version applies from the+// start, each later one from its deploy time.+func (svc *Service) Versions(initial string, deploys ...Deploy) *Service {+ svc.deploys = append([]Deploy{{At: time.Time{}, Version: initial}}, deploys...)+ sort.SliceStable(svc.deploys, func(i, j int) bool { return svc.deploys[i].At.Before(svc.deploys[j].At) })+ return svc+}++// VersionAt returns the service.version live at t.+func (svc *Service) VersionAt(t time.Time) string {+ if len(svc.deploys) == 0 {+ return "1.0.0"+ }+ v := svc.deploys[0].Version+ for _, d := range svc.deploys[1:] {+ if !t.Before(d.At) {+ v = d.Version+ }+ }+ return v+}++// Endpoint declares an HTTP route served by the service.+func (svc *Service) Endpoint(method, route string) *Endpoint {+ e := &Endpoint{svc: svc, Method: method, Route: route, policy: DetailPolicy{Recent: 6 * time.Hour, Rate: 0.03, Max: 60}}+ svc.endpoints = append(svc.endpoints, e)+ svc.s.record(method + " " + route)+ return e+}++// Task declares a background task (a CONSUMER span root).+func (svc *Service) Task(name string) *Endpoint {+ e := &Endpoint{svc: svc, Task: true, Route: name, policy: DetailPolicy{Recent: 6 * time.Hour, Rate: 0.05, Max: 40}}+ svc.tasks = append(svc.tasks, e)+ svc.s.record(name)+ return e+}++// Endpoint is an HTTP route (or a task, when Task is set).+type Endpoint struct {+ svc *Service+ Method string+ Route string+ Task bool+ shapes []Shape+ window time.Duration+ handler func(r *Req)+ policy DetailPolicy+ reqs []*Request+ gen bool+ // Status picks the response status; nil means 200.+ status func(r *Req) int+ // SkipAccessLog disables the per-request access log line.+ SkipAccessLog bool+}++// Name returns "METHOD /route" (or the task name).+func (e *Endpoint) Name() string {+ if e.Task {+ return e.Route+ }+ return e.Method + " " + e.Route+}++// Service returns the serving service.+func (e *Endpoint) Service() *Service { return e.svc }++// Traffic sets the arrival process: rate shapes multiplied together over a+// window (Over, default 24h).+func (e *Endpoint) Traffic(shapes ...Shape) *Endpoint {+ for _, sh := range shapes {+ if o, ok := sh.(over); ok {+ e.window = time.Duration(o)+ continue+ }+ e.shapes = append(e.shapes, sh)+ }+ return e+}++// Handler sets the per-request span builder.+func (e *Endpoint) Handler(f func(r *Req)) *Endpoint { e.handler = f; return e }++// Detail tunes which requests are sent with their full span tree. Requests+// that carry logs or exceptions, or are exposed, always are.+func (e *Endpoint) Detail(p DetailPolicy) *Endpoint { e.policy = p; return e }++// DetailPolicy selects requests to send with full span trees: every request+// in the last Recent window with probability Rate, at most Max of them.+type DetailPolicy struct {+ Recent time.Duration+ Rate float64+ Max int+}++// Shape is a factor on the arrival rate.+type Shape interface {+ factor(s *Scenario, t time.Time, start, end time.Time) float64+}++type daily float64++func (d daily) factor(*Scenario, time.Time, time.Time, time.Time) float64 { return float64(d) }++// Daily sets the base rate in requests per day.+func Daily(n int) Shape { return daily(n) }++type diurnal struct{}++func (diurnal) factor(_ *Scenario, t, _, _ time.Time) float64 {+ h := float64(t.Hour()) + float64(t.Minute())/60+ // Peak around 14:00 UTC, trough around 02:00; mean ~1.+ return 1 + 0.6*math.Cos((h-14)/24*2*math.Pi)+}++// Diurnal applies a day/night cycle (peak mid-afternoon UTC).+func Diurnal() Shape { return diurnal{} }++type growth struct{ from, to float64 }++func (g growth) factor(_ *Scenario, t, start, end time.Time) float64 {+ return g.from + (g.to-g.from)*progress(t, start, end)+}++// GrowRate multiplies the rate from `from` at the window start to `to` at its end.+func GrowRate(from, to float64) Shape { return growth{from, to} }++type window struct {+ from, to time.Time+ mult float64+}++func (w window) factor(_ *Scenario, t, _, _ time.Time) float64 {+ if !t.Before(w.from) && t.Before(w.to) {+ return w.mult+ }+ return 1+}++// Burst multiplies the rate inside [from, to).+func Burst(from, to time.Time, mult float64) Shape { return window{from, to, mult} }++type funcShape func(t time.Time) float64++func (f funcShape) factor(_ *Scenario, t, _, _ time.Time) float64 { return f(t) }++// RateFunc is an arbitrary rate factor.+func RateFunc(f func(t time.Time) float64) Shape { return funcShape(f) }++type over time.Duration++func (over) factor(*Scenario, time.Time, time.Time, time.Time) float64 { return 1 }++// Over sets the traffic window ending now.+func Over(d time.Duration) Shape { return over(d) }++func progress(t, start, end time.Time) float64 {+ if !end.After(start) {+ return 1+ }+ p := float64(t.Sub(start)) / float64(end.Sub(start))+ return math.Max(0, math.Min(1, p))+}++// Request is one generated request (or task run).+type Request struct {+ e *Endpoint+ Index int+ Start time.Time+ Host string+ Version string+ TraceID [16]byte+ SpanID [8]byte+ Root *Span+ Status int+ Detail bool+ forced bool+ Progress float64+ ParentRef *Span // for downstream SERVER spans: the CLIENT span calling it+}++// Duration is the request's total duration.+func (r *Request) Duration() time.Duration { return r.Root.Dur }++// TraceHex is the 32-hex trace id.+func (r *Request) TraceHex() string { return fmt.Sprintf("%x", r.TraceID) }++// SpanHex is the 16-hex root span id.+func (r *Request) SpanHex() string { return fmt.Sprintf("%x", r.SpanID) }++// Endpoint returns the request's endpoint.+func (r *Request) Endpoint() *Endpoint { return r.e }++// KeepDetail forces the full span tree to be sent.+func (r *Request) KeepDetail() { r.forced = true; r.Detail = true }++// Ref is what seed.json carries for a request.+func (r *Request) Ref() map[string]any {+ return map[string]any{+ "id": EntityID(r.e.svc.p.ID, r.TraceID, r.SpanID),+ "recordedAt": r.Start.UTC().Format(time.RFC3339Nano),+ "traceId": r.TraceHex(),+ "spanId": r.SpanHex(),+ "durationMs": float64(r.Duration()) / float64(time.Millisecond),+ "endpoint": r.e.Name(),+ "service": r.e.svc.Name,+ "host": r.Host,+ "version": r.Version,+ "project": r.e.svc.p.Name,+ }+}++// Span is one span in a request tree. Start is the offset from the+// request's start.+type Span struct {+ Name string+ Kind int+ Start time.Duration+ Dur time.Duration+ Attrs map[string]any+ Status int+ StatusMsg string+ Events []Event+ Children []*Span+ Scope string+ SpanID [8]byte+ // Service is set when the span belongs to another service (a+ // downstream SERVER span and its subtree).+ Service *Service+ Host string+ logs []logRecord+}++// Event is a span event.+type Event struct {+ Name string+ At time.Duration+ Attrs map[string]any+}++// Requests generates (once) and returns the endpoint's requests in time order.+func (e *Endpoint) Requests() []*Request {+ if e.gen {+ return e.reqs+ }+ e.gen = true+ if e.handler == nil {+ panic(fmt.Errorf("endpoint %s has no handler", e.Name()))+ }+ s := e.svc.s+ win := e.window+ if win == 0 {+ win = 24 * time.Hour+ }+ end := s.Now.Add(-time.Minute)+ start := end.Add(-win)+ base := 0.0+ var mods []Shape+ for _, sh := range e.shapes {+ if d, ok := sh.(daily); ok {+ base += float64(d)+ continue+ }+ mods = append(mods, sh)+ }+ if base == 0 {+ panic(fmt.Errorf("endpoint %s: traffic needs a Daily(n) base rate", e.Name()))+ }+ // Walk the window minute by minute; expected arrivals per minute is the+ // base rate times every shape factor. Fractional arrivals carry over so+ // low rates still produce requests (deterministic dithering).+ carry := 0.0+ idx := 0+ for t := start; t.Before(end); t = t.Add(time.Minute) {+ lambda := base / 1440+ for _, sh := range mods {+ lambda *= sh.factor(s, t, start, end)+ }+ carry += lambda+ n := int(carry)+ carry -= float64(n)+ for k := 0; k < n; k++ {+ at := t.Add(time.Duration(s.RNG.Int64N(int64(time.Minute))))+ r := e.newRequest(idx, at, progress(at, start, end))+ idx+++ e.reqs = append(e.reqs, r)+ }+ }+ sort.SliceStable(e.reqs, func(i, j int) bool { return e.reqs[i].Start.Before(e.reqs[j].Start) })+ e.selectDetail()+ return e.reqs+}++func (e *Endpoint) newRequest(idx int, at time.Time, prog float64) *Request {+ s := e.svc.s+ r := &Request{e: e, Index: idx, Start: at, Progress: prog}+ r.Host = e.svc.pickHost()+ r.Version = e.svc.VersionAt(at)+ fillRandom(s, r.TraceID[:])+ fillRandom(s, r.SpanID[:])+ kind := KindServer+ if e.Task {+ kind = KindConsumer+ }+ root := &Span{Name: e.Name(), Kind: kind, Scope: e.svc.Stack.ServerScope, SpanID: r.SpanID, Attrs: map[string]any{}, Host: r.Host}+ r.Root = root+ b := &Req{s: s, req: r, cur: root}+ if !e.Task {+ root.Attrs["http.request.method"] = e.Method+ root.Attrs["http.route"] = e.Route+ root.Attrs["url.path"] = concretePath(s, e.Route)+ root.Attrs["url.scheme"] = "https"+ root.Attrs["network.protocol.version"] = "1.1"+ root.Attrs["user_agent.original"] = pick(s, userAgents)+ root.Attrs["client.address"] = fmt.Sprintf("10.%d.%d.%d", 1+s.RNG.IntN(30), s.RNG.IntN(255), 1+s.RNG.IntN(250))+ } else {+ root.Attrs["messaging.operation.type"] = "process"+ }+ e.handler(b)+ b.finish()+ r.Status = 200+ if e.status != nil {+ r.Status = e.status(b)+ }+ if b.status != 0 {+ r.Status = b.status+ }+ if !e.Task {+ root.Attrs["http.response.status_code"] = r.Status+ if r.Status >= 500 {+ root.Status = StatusError+ }+ }+ if b.forceDetail {+ r.forced, r.Detail = true, true+ }+ return r+}++// Statuses sets a per-request status picker.+func (e *Endpoint) Statuses(f func(r *Req) int) *Endpoint { e.status = f; return e }++func (e *Endpoint) selectDetail() {+ s := e.svc.s+ cut := s.Now.Add(-e.policy.Recent)+ // Forced detail (logs, errors) never uses up the sample's cap.+ n := 0+ // Prefer the newest requests so the detail sample sits where the agent+ // looks first, but keep it random within the window.+ for i := len(e.reqs) - 1; i >= 0 && n < e.policy.Max; i-- {+ r := e.reqs[i]+ if r.Detail || r.Start.Before(cut) {+ continue+ }+ if s.RNG.Float64() < e.policy.Rate {+ r.Detail = true+ n+++ }+ }+}++func (svc *Service) pickHost() string {+ if len(svc.Hosts) == 1 {+ return svc.Hosts[0]+ }+ total := 0.0+ for _, h := range svc.Hosts {+ total += svc.weight(h)+ }+ x := svc.s.RNG.Float64() * total+ for _, h := range svc.Hosts {+ x -= svc.weight(h)+ if x <= 0 {+ return h+ }+ }+ return svc.Hosts[len(svc.Hosts)-1]+}++func (svc *Service) weight(h string) float64 {+ if w, ok := svc.HostWeight[h]; ok {+ return w+ }+ return 1+}++func fillRandom(s *Scenario, b []byte) {+ for i := 0; i < len(b); i += 8 {+ v := s.RNG.Uint64()+ for j := 0; j < 8 && i+j < len(b); j++ {+ b[i+j] = byte(v >> (8 * j))+ }+ }+ if allZero(b) {+ b[0] = 1+ }+}++func allZero(b []byte) bool {+ for _, x := range b {+ if x != 0 {+ return false+ }+ }+ return true+}++// Slowest returns a lazy reference to the slowest request in the trailing+// window, for Expose.+func (e *Endpoint) Slowest(within time.Duration) func() any {+ return func() any {+ r := e.SlowestRequest(within)+ if r == nil {+ return nil+ }+ return r.Ref()+ }+}++// SlowestRequest returns the slowest request in the trailing window and+// forces it to be sent in detail.+func (e *Endpoint) SlowestRequest(within time.Duration) *Request {+ var best *Request+ cut := e.svc.s.Now.Add(-within)+ for _, r := range e.Requests() {+ if r.Start.Before(cut) {+ continue+ }+ if best == nil || r.Duration() > best.Duration() {+ best = r+ }+ }+ if best != nil {+ best.KeepDetail()+ }+ return best+}++// Percentile returns the p-quantile (0..1) of durations in the trailing window.+func (e *Endpoint) Percentile(p float64, within time.Duration) time.Duration {+ var ds []time.Duration+ cut := e.svc.s.Now.Add(-within)+ for _, r := range e.Requests() {+ if !r.Start.Before(cut) {+ ds = append(ds, r.Duration())+ }+ }+ return quantile(ds, p)+}++// PercentileBetween returns the p-quantile of durations started in [from, to).+func (e *Endpoint) PercentileBetween(p float64, from, to time.Time) time.Duration {+ var ds []time.Duration+ for _, r := range e.Requests() {+ if !r.Start.Before(from) && r.Start.Before(to) {+ ds = append(ds, r.Duration())+ }+ }+ return quantile(ds, p)+}++func quantile(ds []time.Duration, p float64) time.Duration {+ if len(ds) == 0 {+ return 0+ }+ sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] })+ pos := p * float64(len(ds)-1)+ lo := int(math.Floor(pos))+ hi := int(math.Ceil(pos))+ frac := pos - float64(lo)+ return time.Duration(float64(ds[lo])*(1-frac) + float64(ds[hi])*frac)+}++// Pick returns a lazy reference to the first request matching f (newest+// first), forcing it to detail.+func (e *Endpoint) Pick(f func(r *Request) bool) func() any {+ return func() any {+ r := e.PickRequest(f)+ if r == nil {+ return nil+ }+ return r.Ref()+ }+}++// PickRequest returns the newest request matching f and forces its detail.+func (e *Endpoint) PickRequest(f func(r *Request) bool) *Request {+ reqs := e.Requests()+ for i := len(reqs) - 1; i >= 0; i-- {+ if f(reqs[i]) {+ reqs[i].KeepDetail()+ return reqs[i]+ }+ }+ return nil+}++// Count returns how many requests started in the trailing window.+func (e *Endpoint) Count(within time.Duration) int {+ cut := e.svc.s.Now.Add(-within)+ n := 0+ for _, r := range e.Requests() {+ if !r.Start.Before(cut) {+ n+++ }+ }+ return n+}++var userAgents = []string{+ "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15",+ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",+ "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148",+ "okhttp/4.12.0",+}++func pick[T any](s *Scenario, xs []T) T { return xs[s.RNG.IntN(len(xs))] }++// concretePath turns a route template into a plausible concrete path.+func concretePath(s *Scenario, route string) string {+ out := []byte{}+ for i := 0; i < len(route); i++ {+ if route[i] == ':' || route[i] == '{' {+ j := i + 1+ for j < len(route) && route[j] != '/' && route[j] != '}' {+ j+++ }+ if j < len(route) && route[j] == '}' {+ j+++ }+ out = append(out, []byte(fmt.Sprint(1000+s.RNG.IntN(90000)))...)+ i = j - 1+ continue+ }+ out = append(out, route[i])+ }+ return string(out)+}diff --git a/cli/test/evals/internal/seed/flush.go b/cli/test/evals/internal/seed/flush.gonew file mode 100644index 00000000..3f2affc8--- /dev/null+++ b/cli/test/evals/internal/seed/flush.go@@ -0,0 +1,223 @@+package seed++import (+ "fmt"+ "sort"+ "time"++ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"+ "google.golang.org/protobuf/proto"+)++const (+ pathTraces = "/api/otel/v1/traces"+ pathMetrics = "/api/otel/v1/metrics"+ pathLogs = "/api/otel/v1/logs"+ pathReport = "/api/report"+ protoType = "application/x-protobuf"+ // spansPerRequest keeps each OTLP body far below the 10MB ingest cap.+ spansPerRequest = 1500+)++// Flush generates every request, resolves lazy exposes, enforces the span+// budget and sends everything: traces, logs, metrics, native SDK frames.+func (s *Scenario) Flush() error {+ if s.flushed {+ return fmt.Errorf("scenario flushed twice")+ }+ s.flushed = true+ started := time.Now()++ for _, svc := range s.services {+ applyDefaults(svc)+ }+ var all []*Request+ for _, svc := range s.services {+ for _, e := range append(append([]*Endpoint{}, svc.endpoints...), svc.tasks...) {+ if len(e.shapes) == 0 {+ continue // only reached through Call+ }+ all = append(all, e.Requests()...)+ }+ }+ keys := make([]string, 0, len(s.lazy))+ for k := range s.lazy {+ keys = append(keys, k)+ }+ sort.Strings(keys)+ for _, k := range keys {+ s.exposed[k] = s.lazy[k]()+ }+ s.enforceBudget(all)+ sort.SliceStable(all, func(i, j int) bool { return all[i].Start.Before(all[j].Start) })++ batches := map[*Project]*spanBatch{}+ logs := map[*Service][]logOut{}+ for _, r := range all {+ s.stats.Requests+++ if r.Detail {+ s.stats.DetailedReqs+++ }+ s.emit(r, batches, logs)+ for p, b := range batches {+ if b.count >= spansPerRequest {+ if err := s.sendTraces(p, b); err != nil {+ return err+ }+ delete(batches, p)+ }+ }+ }+ projects := make([]*Project, 0, len(batches))+ for p := range batches {+ projects = append(projects, p)+ }+ sort.Slice(projects, func(i, j int) bool { return projects[i].Name < projects[j].Name })+ for _, p := range projects {+ if err := s.sendTraces(p, batches[p]); err != nil {+ return err+ }+ }++ for _, svc := range s.services {+ outs := logs[svc]+ for _, l := range svc.logs {+ outs = append(outs, logOut{rec: l, at: l.abs, host: l.host, version: svc.VersionAt(l.abs), noTrace: true})+ }+ if len(outs) == 0 {+ continue+ }+ sort.SliceStable(outs, func(i, j int) bool { return outs[i].at.Before(outs[j].at) })+ for i := 0; i < len(outs); i += 2000 {+ j := min(i+2000, len(outs))+ body, err := proto.Marshal(logsRequest(svc, outs[i:j]))+ if err != nil {+ return err+ }+ if _, err := s.Sink.Ingest(svc.p.token, pathLogs, protoType, body, true); err != nil {+ return fmt.Errorf("logs for %s: %w", svc.Name, err)+ }+ s.stats.Logs += j - i+ }+ }++ for _, svc := range s.services {+ for _, req := range s.metricRequests(svc) {+ body, err := proto.Marshal(req)+ if err != nil {+ return err+ }+ if _, err := s.Sink.Ingest(svc.p.token, pathMetrics, protoType, body, true); err != nil {+ return fmt.Errorf("metrics for %s: %w", svc.Name, err)+ }+ }+ }++ recordings := 0+ for _, p := range s.projects {+ for _, f := range p.native {+ recordings += len(f.recordings)+ }+ bodies, err := p.nativeBodies()+ if err != nil {+ return err+ }+ for _, body := range bodies {+ if _, err := s.Sink.Ingest(p.token, pathReport, "application/json", body, true); err != nil {+ return fmt.Errorf("native report for %s: %w", p.Name, err)+ }+ s.stats.NativeFrames+++ }+ }++ if recordings > 0 && !s.Sink.Dry() {+ // Session recordings are uploaded by a worker pool and inserted in+ // batches every ~2s; wait so occurrences show their actions.+ time.Sleep(3 * time.Second)+ }+ for _, f := range s.afterFlush {+ if err := f(); err != nil {+ return err+ }+ }+ s.stats.IngestElapsed = time.Since(started)+ return nil+}++func (s *Scenario) sendTraces(p *Project, b *spanBatch) error {+ body, err := proto.Marshal(b.request())+ if err != nil {+ return err+ }+ resp, err := s.Sink.Ingest(p.token, pathTraces, protoType, body, true)+ if err != nil {+ return fmt.Errorf("traces for %s: %w", p.Name, err)+ }+ if len(resp) > 0 {+ var out coltracepb.ExportTraceServiceResponse+ if proto.Unmarshal(resp, &out) == nil && out.GetPartialSuccess().GetRejectedSpans() > 0 {+ return fmt.Errorf("traces for %s: backend rejected %d spans: %s", p.Name, out.GetPartialSuccess().GetRejectedSpans(), out.GetPartialSuccess().GetErrorMessage())+ }+ }+ return nil+}++func spanCount(sp *Span) int {+ n := 1+ for _, c := range sp.Children {+ n += spanCount(c)+ }+ return n+}++func serverSpans(sp *Span) int {+ n := 0+ if sp.Kind == KindServer {+ n = 1+ }+ for _, c := range sp.Children {+ n += serverSpans(c)+ }+ return n+}++// enforceBudget keeps the detailed span volume under SpanBudget by dropping+// sampled (not forced) detail, oldest first.+func (s *Scenario) enforceBudget(all []*Request) {+ total := 0+ for _, r := range all {+ total += s.sentSpans(r)+ }+ if total <= s.SpanBudget {+ return+ }+ sampled := make([]*Request, 0)+ for _, r := range all {+ if r.Detail && !r.forced {+ sampled = append(sampled, r)+ }+ }+ sort.SliceStable(sampled, func(i, j int) bool { return sampled[i].Start.Before(sampled[j].Start) })+ for _, r := range sampled {+ if total <= s.SpanBudget {+ return+ }+ before := s.sentSpans(r)+ r.Detail = false+ total -= before - s.sentSpans(r)+ }+}++func (s *Scenario) sentSpans(r *Request) int {+ if r.Detail {+ return spanCount(r.Root)+ }+ return 1 + serverSpans(r.Root) - boolInt(r.Root.Kind == KindServer)+}++func boolInt(b bool) int {+ if b {+ return 1+ }+ return 0+}diff --git a/cli/test/evals/internal/seed/groups.go b/cli/test/evals/internal/seed/groups.gonew file mode 100644index 00000000..e5c20551--- /dev/null+++ b/cli/test/evals/internal/seed/groups.go@@ -0,0 +1,88 @@+package seed++import (+ "encoding/json"+ "fmt"+ "net/http"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+)++// Group is an exception group read back from the API.+type Group struct {+ Hash string `json:"exceptionHash"`+ StackTrace string `json:"stackTrace"`+ FirstSeen time.Time `json:"firstSeen"`+ LastSeen time.Time `json:"lastSeen"`+ Count int `json:"count"`+}++// Groups lists the project's exception groups over the last `within`.+func (p *Project) Groups(within time.Duration, includeArchived bool) ([]Group, error) {+ if p.s.Sink.Dry() {+ return nil, nil+ }+ res, err := p.s.Sink.Admin(http.MethodPost, "/api/exception-stack-traces", env.ProjectQuery(p.ID), map[string]any{+ "fromDate": p.s.Now.Add(-within).Format(time.RFC3339), "toDate": p.s.Now.Add(time.Hour).Format(time.RFC3339),+ "orderBy": "count", "pagination": map[string]int{"page": 1, "pageSize": 100}, "includeArchived": includeArchived,+ })+ if err != nil {+ return nil, err+ }+ if err := res.Err("list exceptions"); err != nil {+ return nil, err+ }+ var out struct {+ Data []Group `json:"data"`+ }+ if err := json.Unmarshal(res.Body, &out); err != nil {+ return nil, err+ }+ return out.Data, nil+}++// GroupMatching returns the hash of the group whose stack contains needle.+func (p *Project) GroupMatching(within time.Duration, needle string) (string, error) {+ if p.s.Sink.Dry() {+ return fmt.Sprintf("%016x", len(needle)), nil+ }+ groups, err := p.Groups(within, true)+ if err != nil {+ return "", err+ }+ for _, g := range groups {+ if containsFold(g.StackTrace, needle) {+ return g.Hash, nil+ }+ }+ return "", fmt.Errorf("no exception group in %s contains %q", p.Name, needle)+}++func containsFold(s, sub string) bool {+ return len(sub) == 0 || indexFold(s, sub) >= 0+}++func indexFold(s, sub string) int {+ n := len(sub)+ for i := 0; i+n <= len(s); i++ {+ match := true+ for j := 0; j < n; j++ {+ a, b := s[i+j], sub[j]+ if 'A' <= a && a <= 'Z' {+ a += 'a' - 'A'+ }+ if 'A' <= b && b <= 'Z' {+ b += 'a' - 'A'+ }+ if a != b {+ match = false+ break+ }+ }+ if match {+ return i+ }+ }+ return -1+}diff --git a/cli/test/evals/internal/seed/logs.go b/cli/test/evals/internal/seed/logs.gonew file mode 100644index 00000000..f1231b57--- /dev/null+++ b/cli/test/evals/internal/seed/logs.go@@ -0,0 +1,110 @@+package seed++import (+ "sort"+ "time"++ collogspb "go.opentelemetry.io/proto/otlp/collector/logs/v1"+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"+ logspb "go.opentelemetry.io/proto/otlp/logs/v1"+)++type logRecord struct {+ at time.Duration // offset within the owning span+ abs time.Time // absolute time, for logs outside a request+ host string+ severity int+ body string+ attrs map[string]any+ scope string+}++type logOut struct {+ rec logRecord+ at time.Time+ traceID [16]byte+ spanID [8]byte+ host string+ version string+ noTrace bool+}++// Log emits a log line from the service that is not tied to a request (a+// startup message, a scheduler line, a pool warning without a trace).+func (svc *Service) Log(at time.Time, severity int, body string, attrs ...KV) {+ svc.LogOn(svc.Hosts[0], at, severity, body, attrs...)+}++// LogOn is Log from a specific instance.+func (svc *Service) LogOn(host string, at time.Time, severity int, body string, attrs ...KV) {+ svc.logs = append(svc.logs, logRecord{abs: at, host: host, severity: severity, body: body, attrs: kvMap(attrs)})+ svc.s.record(body)+ for _, kv := range attrs {+ if s, ok := kv.V.(string); ok {+ svc.s.record(s)+ }+ }+}++func logScope(svc *Service) string {+ switch svc.Stack.Language {+ case "python":+ return "opentelemetry.sdk._logs"+ case "nodejs":+ return "@opentelemetry/winston-transport"+ case "java":+ return "io.opentelemetry.logback-appender-1.0"+ case "dotnet":+ return "Microsoft.Extensions.Logging"+ case "php":+ return "io.opentelemetry.contrib.php.monolog"+ default:+ return "go.opentelemetry.io/contrib/bridges/otelslog"+ }+}++// logsRequest renders a service's logs as one OTLP request per resource.+func logsRequest(svc *Service, outs []logOut) *collogspb.ExportLogsServiceRequest {+ type key struct{ version, host string }+ groups := map[key][]*logspb.LogRecord{}+ var keys []key+ for _, o := range outs {+ k := key{o.version, o.host}+ if _, ok := groups[k]; !ok {+ keys = append(keys, k)+ }+ lr := &logspb.LogRecord{+ TimeUnixNano: nanos(o.at),+ ObservedTimeUnixNano: nanos(o.at.Add(3 * time.Millisecond)),+ SeverityNumber: logspb.SeverityNumber(o.rec.severity),+ SeverityText: SeverityText(o.rec.severity),+ Body: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: o.rec.body}},+ Attributes: attrList(o.rec.attrs),+ }+ if !o.noTrace {+ lr.TraceId = append([]byte{}, o.traceID[:]...)+ lr.SpanId = append([]byte{}, o.spanID[:]...)+ lr.Flags = 1+ }+ groups[k] = append(groups[k], lr)+ }+ sort.Slice(keys, func(i, j int) bool {+ if keys[i].version != keys[j].version {+ return keys[i].version < keys[j].version+ }+ return keys[i].host < keys[j].host+ })+ req := &collogspb.ExportLogsServiceRequest{}+ for _, k := range keys {+ recs := groups[k]+ sort.SliceStable(recs, func(i, j int) bool { return recs[i].TimeUnixNano < recs[j].TimeUnixNano })+ req.ResourceLogs = append(req.ResourceLogs, &logspb.ResourceLogs{+ Resource: svc.resource(k.version, k.host),+ ScopeLogs: []*logspb.ScopeLogs{{+ Scope: &commonpb.InstrumentationScope{Name: logScope(svc)},+ LogRecords: recs,+ }},+ })+ }+ return req+}diff --git a/cli/test/evals/internal/seed/metrics.go b/cli/test/evals/internal/seed/metrics.gonew file mode 100644index 00000000..beeeb582--- /dev/null+++ b/cli/test/evals/internal/seed/metrics.go@@ -0,0 +1,360 @@+package seed++import (+ "crypto/sha256"+ "encoding/binary"+ "fmt"+ "math"+ "sort"+ "time"++ colmetricspb "go.opentelemetry.io/proto/otlp/collector/metrics/v1"+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"+ metricspb "go.opentelemetry.io/proto/otlp/metrics/v1"+)++// Series is a value over time. Noise inside a Series is a deterministic+// function of (series, t), so the same scenario always draws the same curve.+type Series func(t time.Time) float64++// noise returns a deterministic [0,1) value for a label and a time.+func noise(label string, t time.Time) float64 {+ var b [8]byte+ binary.BigEndian.PutUint64(b[:], uint64(t.Unix()))+ sum := sha256.Sum256(append([]byte(label), b[:]...))+ return float64(binary.BigEndian.Uint64(sum[:8])>>11) / float64(1<<53)+}++// Flat is a value wandering between lo and hi.+func Flat(lo, hi float64) Series {+ return func(t time.Time) float64 {+ // Smooth-ish: blend the current and previous 5-minute noise.+ a := noise("flat", t.Truncate(5*time.Minute))+ b := noise("flat", t.Truncate(5*time.Minute).Add(5*time.Minute))+ f := float64(t.Sub(t.Truncate(5*time.Minute))) / float64(5*time.Minute)+ return lo + (hi-lo)*(a*(1-f)+b*f)+ }+}++// Const is a constant value.+func Const(v float64) Series { return func(time.Time) float64 { return v } }++// Step switches from before to after at at.+func Step(at time.Time, before, after Series) Series {+ return func(t time.Time) float64 {+ if t.Before(at) {+ return before(t)+ }+ return after(t)+ }+}++// Ramp interpolates linearly from `from` at start to `to` at end, plus a+// little noise (±3% of the span).+func Ramp(from, to float64, start, end time.Time) Series {+ return func(t time.Time) float64 {+ p := progress(t, start, end)+ return from + (to-from)*p + (to-from)*0.03*(noise("ramp", t.Truncate(time.Minute))-0.5)+ }+}++// Scale multiplies a series by f(t).+func Scale(s Series, f func(t time.Time) float64) Series {+ return func(t time.Time) float64 { return s(t) * f(t) }+}++// Plus adds series.+func Plus(ss ...Series) Series {+ return func(t time.Time) float64 {+ v := 0.0+ for _, s := range ss {+ v += s(t)+ }+ return v+ }+}++// Clamp bounds a series to [lo, hi].+func Clamp(s Series, lo, hi float64) Series {+ return func(t time.Time) float64 { return math.Max(lo, math.Min(hi, s(t))) }+}++// Periodic is base, raised to peak for width at every multiple of period.+func Periodic(period, width time.Duration, base, peak Series) Series {+ return func(t time.Time) float64 {+ off := t.Sub(t.Truncate(period))+ if off < width {+ return peak(t)+ }+ return base(t)+ }+}++// Sawtooth climbs from lo to hi over period, then drops back (a leak that+// is cleared by a restart).+func Sawtooth(period time.Duration, lo, hi float64) Series {+ return func(t time.Time) float64 {+ f := float64(t.Sub(t.Truncate(period))) / float64(period)+ return lo + (hi-lo)*f + (hi-lo)*0.02*(noise("saw", t.Truncate(time.Minute))-0.5)+ }+}++// DiurnalSeries modulates a series with the same day/night cycle as Diurnal.+func DiurnalSeries(s Series, depth float64) Series {+ return func(t time.Time) float64 {+ h := float64(t.Hour()) + float64(t.Minute())/60+ return s(t) * (1 + depth*math.Cos((h-14)/24*2*math.Pi))+ }+}++type metricKind int++const (+ gaugeKind metricKind = iota+ counterKind+ histogramKind+)++type metricSeries struct {+ name, unit, desc string+ kind metricKind+ attrs map[string]any+ host string+ value Series // gauge value; counter rate per second; histogram mean+ count Series // histogram observations per minute+ window time.Duration+ scope string+ // cumTotal and cumPrev carry a counter's running total across the+ // chunks metricRequests renders it in.+ cumTotal float64+ cumPrev time.Time+}++// MetricStep controls sampling: points every Recent inside the last day,+// every Old before that.+var MetricStep = struct{ Recent, Old time.Duration }{time.Minute, 10 * time.Minute}++// Gauge adds a gauge series on the service's first host.+func (svc *Service) Gauge(name, unit string, attrs map[string]any, v Series) {+ svc.GaugeOn(svc.Hosts[0], name, unit, attrs, v)+}++// GaugeOn adds a gauge series for one instance.+func (svc *Service) GaugeOn(host, name, unit string, attrs map[string]any, v Series) {+ label := fmt.Sprint(host, name, attrs)+ jittered := func(t time.Time) float64 { return v(t) * (1 + 0.03*(noise(label, t)-0.5)) }+ svc.metrics = append(svc.metrics, &metricSeries{name: name, unit: unit, kind: gaugeKind, attrs: attrs, host: host, value: jittered, scope: metricScope(name)})+ svc.s.record(name)+}++// Counter adds a monotonic cumulative sum whose rate per second is v.+func (svc *Service) Counter(name, unit string, attrs map[string]any, ratePerSec Series) {+ svc.metrics = append(svc.metrics, &metricSeries{name: name, unit: unit, kind: counterKind, attrs: attrs, host: svc.Hosts[0], value: ratePerSec, scope: metricScope(name)})+ svc.s.record(name)+}++// Histogram adds a delta histogram: mean value and observations per minute.+// The backend stores it as <name>.avg and <name>.count.+func (svc *Service) Histogram(name, unit string, attrs map[string]any, mean, perMinute Series) {+ svc.HistogramOn(svc.Hosts[0], name, unit, attrs, mean, perMinute)+}++// HistogramOn is Histogram for one instance.+func (svc *Service) HistogramOn(host, name, unit string, attrs map[string]any, mean, perMinute Series) {+ svc.metrics = append(svc.metrics, &metricSeries{name: name, unit: unit, kind: histogramKind, attrs: attrs, host: host, value: mean, count: perMinute, scope: metricScope(name)})+ svc.s.record(name)+}++// MetricsWindow sets how far back the service's metric series go (default:+// the scenario's longest traffic window, at least 24h).+func (svc *Service) metricsWindow() time.Duration {+ w := 24 * time.Hour+ for _, e := range append(append([]*Endpoint{}, svc.endpoints...), svc.tasks...) {+ if e.window > w {+ w = e.window+ }+ }+ return w+}++func metricScope(name string) string {+ switch {+ case len(name) >= 7 && name[:7] == "system.":+ return "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver"+ case len(name) > 4 && name[:4] == "jvm.":+ return "io.opentelemetry.runtime-telemetry-java17"+ case len(name) > 3 && name[:3] == "go.":+ return "go.opentelemetry.io/contrib/instrumentation/runtime"+ case len(name) > 10 && name[:10] == "db.client.":+ return "io.opentelemetry.hikaricp-3.0"+ case len(name) > 12 && name[:12] == "http.server.":+ return "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"+ default:+ return "app"+ }+}++// Host is a convenience over one instance's host metrics.+type Host struct {+ svc *Service+ Name string+}++// Host returns the first instance's host metrics.+func (svc *Service) Host() *Host { return &Host{svc: svc, Name: svc.Hosts[0]} }++// HostNamed returns one instance's host metrics.+func (svc *Service) HostNamed(name string) *Host { return &Host{svc: svc, Name: name} }++// CPU emits system.cpu.utilization (OTel hostmetrics) split into user,+// system, wait, steal and idle, with busy = user+system. Optional wait and+// steal series default to small flat values.+func (h *Host) CPU(busy Series, extra ...CPUExtra) {+ wait, steal := Flat(0.002, 0.01), Flat(0, 0.004)+ for _, e := range extra {+ if e.Wait != nil {+ wait = e.Wait+ }+ if e.Steal != nil {+ steal = e.Steal+ }+ }+ clampFrac := func(s Series) Series { return Clamp(s, 0, 1) }+ busy, wait, steal = clampFrac(busy), clampFrac(wait), clampFrac(steal)+ user := Scale(busy, func(time.Time) float64 { return 0.78 })+ sys := Scale(busy, func(time.Time) float64 { return 0.22 })+ idle := func(t time.Time) float64 { return math.Max(0, 1-busy(t)-wait(t)-steal(t)) }+ for state, s := range map[string]Series{"user": user, "system": sys, "wait": wait, "steal": steal, "idle": idle} {+ h.svc.GaugeOn(h.Name, "system.cpu.utilization", "1", map[string]any{"state": state, "cpu": "cpu0"}, s)+ }+}++// CPUExtra adds iowait or steal to CPU.+type CPUExtra struct{ Wait, Steal Series }++// Memory emits system.memory.utilization{state=used|free}.+func (h *Host) Memory(used Series) {+ used = Clamp(used, 0, 1)+ h.svc.GaugeOn(h.Name, "system.memory.utilization", "1", map[string]any{"state": "used"}, used)+ h.svc.GaugeOn(h.Name, "system.memory.utilization", "1", map[string]any{"state": "free"}, func(t time.Time) float64 { return 1 - used(t) })+}++// Disk emits the hostmetrics disk series for one device: system.disk.io+// (cumulative bytes, by direction) and system.disk.pending_operations (the+// queue depth gauge).+func (h *Host) Disk(device string, readBps, writeBps, pending Series) {+ h.svc.metrics = append(h.svc.metrics,+ &metricSeries{name: "system.disk.io", unit: "By", kind: counterKind, attrs: map[string]any{"device": device, "direction": "read"}, host: h.Name, value: readBps, scope: metricScope("system.")},+ &metricSeries{name: "system.disk.io", unit: "By", kind: counterKind, attrs: map[string]any{"device": device, "direction": "write"}, host: h.Name, value: writeBps, scope: metricScope("system.")},+ )+ h.svc.GaugeOn(h.Name, "system.disk.pending_operations", "{operations}", map[string]any{"device": device}, pending)+}++// Filesystem emits system.filesystem.utilization for a mountpoint.+func (h *Host) Filesystem(mount string, used Series) {+ h.svc.GaugeOn(h.Name, "system.filesystem.utilization", "1", map[string]any{"mountpoint": mount, "state": "used"}, Clamp(used, 0, 1))+}++func (s *Scenario) metricRequests(svc *Service) []*colmetricspb.ExportMetricsServiceRequest {+ if len(svc.metrics) == 0 {+ return nil+ }+ end := s.Now.Add(-time.Minute)+ win := svc.metricsWindow()+ start := end.Add(-win)+ recentCut := end.Add(-24 * time.Hour)+ var times []time.Time+ for t := start.Truncate(MetricStep.Old); t.Before(recentCut); t = t.Add(MetricStep.Old) {+ if !t.Before(start) {+ times = append(times, t)+ }+ }+ for t := recentCut.Truncate(MetricStep.Recent); !t.After(end); t = t.Add(MetricStep.Recent) {+ times = append(times, t)+ }+ byHost := map[string][]*metricSeries{}+ var hosts []string+ for _, m := range svc.metrics {+ if _, ok := byHost[m.host]; !ok {+ hosts = append(hosts, m.host)+ }+ byHost[m.host] = append(byHost[m.host], m)+ }+ sort.Strings(hosts)+ var out []*colmetricspb.ExportMetricsServiceRequest+ const chunk = 400 // timestamps per request keeps each body small+ for i := 0; i < len(times); i += chunk {+ j := min(i+chunk, len(times))+ req := &colmetricspb.ExportMetricsServiceRequest{}+ for _, host := range hosts {+ rm := &metricspb.ResourceMetrics{Resource: svc.resource(svc.VersionAt(times[j-1]), host)}+ scopes := map[string]*metricspb.ScopeMetrics{}+ var scopeNames []string+ for _, m := range byHost[host] {+ sm := scopes[m.scope]+ if sm == nil {+ sm = &metricspb.ScopeMetrics{Scope: &commonpb.InstrumentationScope{Name: m.scope}}+ scopes[m.scope] = sm+ scopeNames = append(scopeNames, m.scope)+ }+ sm.Metrics = append(sm.Metrics, m.render(times[i:j], start))+ s.stats.MetricPoints += j - i+ }+ sort.Strings(scopeNames)+ for _, n := range scopeNames {+ rm.ScopeMetrics = append(rm.ScopeMetrics, scopes[n])+ }+ req.ResourceMetrics = append(req.ResourceMetrics, rm)+ }+ out = append(out, req)+ }+ return out+}++func (m *metricSeries) render(times []time.Time, origin time.Time) *metricspb.Metric {+ attrs := attrList(m.attrs)+ out := &metricspb.Metric{Name: m.name, Unit: m.unit}+ switch m.kind {+ case gaugeKind:+ g := &metricspb.Gauge{}+ for _, t := range times {+ g.DataPoints = append(g.DataPoints, &metricspb.NumberDataPoint{+ Attributes: attrs, TimeUnixNano: nanos(t),+ Value: &metricspb.NumberDataPoint_AsDouble{AsDouble: m.value(t)},+ })+ }+ out.Data = &metricspb.Metric_Gauge{Gauge: g}+ case counterKind:+ sum := &metricspb.Sum{IsMonotonic: true, AggregationTemporality: metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_CUMULATIVE}+ if m.cumPrev.IsZero() {+ m.cumPrev = origin+ }+ total, prev := m.cumTotal, m.cumPrev+ for _, t := range times {+ total += m.value(t) * t.Sub(prev).Seconds()+ prev = t+ sum.DataPoints = append(sum.DataPoints, &metricspb.NumberDataPoint{+ Attributes: attrs, StartTimeUnixNano: nanos(origin), TimeUnixNano: nanos(t),+ Value: &metricspb.NumberDataPoint_AsDouble{AsDouble: total},+ })+ }+ m.cumTotal, m.cumPrev = total, prev+ out.Data = &metricspb.Metric_Sum{Sum: sum}+ case histogramKind:+ h := &metricspb.Histogram{AggregationTemporality: metricspb.AggregationTemporality_AGGREGATION_TEMPORALITY_DELTA}+ prev := times[0].Add(-time.Minute)+ for _, t := range times {+ per := m.count(t) * t.Sub(prev).Minutes()+ prev = t+ cnt := uint64(math.Max(0, math.Round(per)))+ mean := m.value(t)+ s := mean * float64(cnt)+ h.DataPoints = append(h.DataPoints, &metricspb.HistogramDataPoint{+ Attributes: attrs, StartTimeUnixNano: nanos(t.Add(-time.Minute)), TimeUnixNano: nanos(t),+ Count: cnt, Sum: &s,+ })+ }+ out.Data = &metricspb.Metric_Histogram{Histogram: h}+ }+ return out+}diff --git a/cli/test/evals/internal/seed/native_live_test.go b/cli/test/evals/internal/seed/native_live_test.gonew file mode 100644index 00000000..6793f494--- /dev/null+++ b/cli/test/evals/internal/seed/native_live_test.go@@ -0,0 +1,59 @@+package seed_test++import (+ "context"+ "testing"+ "time"++ client "github.com/tracewayapp/traceway/cli/pkg/client"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++// TestNativeCrashWithActions sends a mobile crash with captured network+// actions over /api/report and reads it back as an agent would.+func TestNativeCrashWithActions(t *testing.T) {+ bins := env.BinariesForTest(t)+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)+ defer cancel()+ inst, err := env.NewInstance(ctx, bins, t.TempDir())+ if err != nil {+ t.Fatal(err)+ }+ defer inst.Close()+ s := seed.NewScenario("native-test", time.Now(), seed.NewLiveSink(ctx, inst.Backend, inst.Admin))+ p := s.Project("shop-android", seed.Mobile, "android")+ at := s.Ago(2 * time.Hour)+ p.Crash(seed.Crash{At: at, AppVersion: "4.12.0", Device: "Pixel 8",+ Stack: "java.net.SocketTimeoutException: timeout\n\tat okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.kt:675)\n\tat com.example.shop.checkout.CheckoutRepository.placeOrder(CheckoutRepository.kt:88)",+ Attributes: map[string]string{"device.model": "Pixel 8", "os.version": "15"},+ Actions: []seed.Action{{Type: "network", At: at.Add(-10 * time.Second), Method: "POST", URL: "https://api.shop.example/api/checkout", DurationMs: 10012, Error: "timeout"}},+ })+ if err := s.Flush(); err != nil {+ t.Fatal(err)+ }+ c := client.New(inst.Backend.URL, client.WithJWT(inst.Admin.JWT))+ list, err := c.ListExceptions(ctx, p.ID, client.ListExceptionsRequest{TimeRange: client.TimeRangeFromExplicit(s.Ago(24*time.Hour), s.Now.Add(time.Hour)), Pagination: client.PaginationParams{Page: 1, PageSize: 10}})+ if err != nil || len(list.Data) != 1 {+ t.Fatalf("list: %v %+v", err, list)+ }+ g, err := c.GetException(ctx, p.ID, list.Data[0].ExceptionHash, client.PaginationParams{Page: 1, PageSize: 5})+ if err != nil || len(g.Occurrences) != 1 {+ t.Fatalf("show: %v", err)+ }+ var occ *client.ExceptionByIdResponse+ for i := 0; i < 40; i++ {+ occ, err = c.GetExceptionById(ctx, p.ID, g.Occurrences[0].Id.String(), g.Occurrences[0].RecordedAt)+ if err != nil {+ t.Fatal(err)+ }+ if len(occ.SessionRecording) > 0 && string(occ.SessionRecording) != "null" {+ break+ }+ time.Sleep(250 * time.Millisecond)+ }+ if len(occ.SessionRecording) == 0 || string(occ.SessionRecording) == "null" {+ t.Fatalf("the occurrence carries no recording with the captured actions: %+v", occ)+ }+ t.Logf("recording: %.300s", string(occ.SessionRecording))+}diff --git a/cli/test/evals/internal/seed/noise.go b/cli/test/evals/internal/seed/noise.gonew file mode 100644index 00000000..bbbc1be8--- /dev/null+++ b/cli/test/evals/internal/seed/noise.go@@ -0,0 +1,150 @@+package seed++import (+ "fmt"+ "math"+ "time"+)++// TrafficOpts configures background noise.+type TrafficOpts struct {+ Endpoints int+ // PerDay is the request rate per background endpoint (capped so the+ // background stays under ~5000 requests per service).+ PerDay int+ // Window defaults to the service's longest scenario window.+ Window time.Duration+ // NoErrors suppresses the low-rate unrelated client error.+ NoErrors bool+ // Hosts sets default host metrics (CPU and memory) when the scenario+ // did not set its own.+ NoHostMetrics bool+}++// TypicalTraffic is n healthy endpoints, a low-rate unrelated error and flat+// host metrics.+func TypicalTraffic(n int) TrafficOpts { return TrafficOpts{Endpoints: n, PerDay: 150} }++// backgroundRoutes is a neutral pool of routes for healthy traffic.+var backgroundRoutes = [][2]string{+ {"GET", "/api/me"}, {"GET", "/api/products"}, {"GET", "/api/products/:id"}, {"POST", "/api/cart/items"},+ {"GET", "/api/cart"}, {"GET", "/api/categories"}, {"POST", "/api/auth/session"}, {"GET", "/api/notifications"},+ {"PUT", "/api/users/:id/preferences"}, {"POST", "/api/events"}, {"GET", "/api/addresses"}, {"DELETE", "/api/cart/items/:id"},+ {"GET", "/api/inventory/:sku"}, {"GET", "/api/recommendations"}, {"POST", "/api/checkout/quote"}, {"GET", "/api/stores/:id/hours"},+ {"GET", "/api/promotions"}, {"POST", "/api/newsletter/subscriptions"}, {"GET", "/api/shipping/rates"}, {"GET", "/api/help/articles"},+}++var backgroundTables = []string{"users", "products", "categories", "carts", "cart_items", "notifications", "addresses", "inventory", "stores", "promotions"}++// Background adds healthy traffic around the scenario so the signal has to+// be found, not handed over.+func (s *Scenario) Background(svc *Service, o TrafficOpts) {+ taken := map[string]bool{}+ for _, e := range svc.endpoints {+ taken[e.Name()] = true+ }+ win := o.Window+ if win == 0 {+ win = svc.metricsWindow()+ }+ days := math.Max(1, win.Hours()/24)+ rate := o.PerDay+ if o.Endpoints > 0 && float64(rate)*float64(o.Endpoints)*days > 5000 {+ rate = int(5000 / (float64(o.Endpoints) * days))+ if rate < 8 {+ rate = 8+ }+ }+ added := 0+ for i := 0; added < o.Endpoints && i < len(backgroundRoutes)*2; i++ {+ rt := backgroundRoutes[(i*7+len(svc.Name))%len(backgroundRoutes)]+ name := rt[0] + " " + rt[1]+ if taken[name] {+ continue+ }+ taken[name] = true+ added+++ tables := []string{backgroundTables[(i*3)%len(backgroundTables)], backgroundTables[(i*5+1)%len(backgroundTables)]}+ method := rt[0]+ medianDB := time.Duration(2+(i%5)) * time.Millisecond+ withError := !o.NoErrors && method == "POST" && added <= 6+ e := svc.Endpoint(rt[0], rt[1])+ e.Traffic(Daily(rate), Diurnal(), Over(win))+ e.Detail(DetailPolicy{Recent: 6 * time.Hour, Rate: 0.01, Max: 8})+ e.Handler(func(r *Req) {+ r.Gap(r.Ms(0.3, 1.2))+ if withError && r.Chance(0.004) {+ r.Compute("decode request", r.Ms(0.1, 0.4))+ r.Error(clientDecodeError(svc.Stack))+ r.Status(400)+ r.Gap(r.Ms(0.2, 0.6))+ return+ }+ r.DB(selectFor(method, tables[0]), r.LogNormal(medianDB, medianDB*6))+ if r.Chance(0.6) {+ r.DB(selectFor("GET", tables[1]), r.LogNormal(medianDB, medianDB*5))+ }+ r.Compute("encode response", r.Ms(0.4, 2.5))+ r.Gap(r.Ms(0.2, 0.8))+ })+ }+ if !o.NoHostMetrics {+ svc.defaultHost = true+ }+}++func selectFor(method, table string) string {+ switch method {+ case "POST":+ return fmt.Sprintf("INSERT INTO %s (id, payload, created_at) VALUES ($1, $2, now())", table)+ case "PUT":+ return fmt.Sprintf("UPDATE %s SET payload = $2, updated_at = now() WHERE id = $1", table)+ case "DELETE":+ return fmt.Sprintf("DELETE FROM %s WHERE id = $1", table)+ default:+ return fmt.Sprintf("SELECT id, name, updated_at FROM %s WHERE id = $1", table)+ }+}++// clientDecodeError is a benign 400: a client sent malformed JSON.+func clientDecodeError(st Stack) Exc {+ switch st.StackFormat {+ case "python":+ return Exc{Type: "json.decoder.JSONDecodeError", Message: "Expecting property name enclosed in double quotes: line 1 column 2 (char 1)",+ Stack: "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/site-packages/starlette/requests.py\", line 252, in json\n self._json = json.loads(body)\n File \"/usr/local/lib/python3.12/json/__init__.py\", line 346, in loads\n return _default_decoder.decode(s)\njson.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"}+ case "node":+ return Exc{Type: "SyntaxError", Message: "Expected property name or '}' in JSON at position 1",+ Stack: "SyntaxError: Expected property name or '}' in JSON at position 1\n at JSON.parse (<anonymous>)\n at parse (/app/node_modules/body-parser/lib/types/json.js:92:19)\n at /app/node_modules/body-parser/lib/read.js:128:18"}+ case "java":+ return Exc{Type: "com.fasterxml.jackson.core.JsonParseException", Message: "Unexpected character ('}' (code 125)): was expecting double-quote to start field name",+ Stack: "com.fasterxml.jackson.core.JsonParseException: Unexpected character ('}' (code 125)): was expecting double-quote to start field name\n\tat com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2481)\n\tat org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406)"}+ case "php":+ return Exc{Type: "JsonException", Message: "Syntax error", Stack: "JsonException: Syntax error in /var/www/vendor/laravel/framework/src/Illuminate/Http/Request.php:671\nStack trace:\n#0 /var/www/vendor/laravel/framework/src/Illuminate/Http/Request.php(671): json_decode()"}+ case "dotnet":+ return Exc{Type: "System.Text.Json.JsonException", Message: "'}' is an invalid start of a property name. Path: $ | LineNumber: 0 | BytePositionInLine: 1.",+ Stack: "System.Text.Json.JsonException: '}' is an invalid start of a property name.\n at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex)\n at Microsoft.AspNetCore.Http.Json.HttpRequestJsonExtensions.ReadFromJsonAsync[TValue](HttpRequest request)"}+ default:+ return Exc{Type: "*json.SyntaxError", Message: "invalid character '}' looking for beginning of object key string",+ Stack: "goroutine 4113 [running]:\nencoding/json.(*decodeState).object(0xc0004b2000)\n\t/usr/local/go/src/encoding/json/decode.go:651 +0x1ef\ngithub.com/gin-gonic/gin.(*Context).ShouldBindJSON(0xc000521100)\n\t/go/pkg/mod/github.com/gin-gonic/[email protected]/context.go:780 +0x45"}+ }+}++// applyDefaults fills host metrics a scenario did not set itself.+func applyDefaults(svc *Service) {+ if !svc.defaultHost {+ return+ }+ has := map[string]bool{}+ for _, m := range svc.metrics {+ has[m.name+"@"+m.host] = true+ }+ for _, h := range svc.Hosts {+ host := svc.HostNamed(h)+ if !has["system.cpu.utilization@"+h] {+ host.CPU(Flat(0.18, 0.32))+ }+ if !has["system.memory.utilization@"+h] {+ host.Memory(Flat(0.52, 0.6))+ }+ }+}diff --git a/cli/test/evals/internal/seed/report.go b/cli/test/evals/internal/seed/report.gonew file mode 100644index 00000000..a20fed36--- /dev/null+++ b/cli/test/evals/internal/seed/report.go@@ -0,0 +1,204 @@+package seed++import (+ "encoding/json"+ "fmt"+ "time"++ "github.com/google/uuid"+)++// The native /api/report protocol is what the browser and mobile SDKs speak:+// exceptions with their own stack format, sessions, and session recordings+// whose actions carry the network requests and navigations the SDK captured.+// Only what those SDKs really send is modelled here.++type nativeFrame struct {+ appVersion string+ serverName string+ exceptions []nativeException+ sessions []nativeSession+ recordings []nativeRecording+}++type nativeException struct {+ Stack string `json:"stackTrace"`+ RecordedAt string `json:"recordedAt"`+ Attributes map[string]string `json:"attributes"`+ IsMessage bool `json:"isMessage"`+ IsTask bool `json:"isTask"`+ SessionID string `json:"sessionId,omitempty"`+ RecordingID string `json:"sessionRecordingId,omitempty"`+}++type nativeSession struct {+ ID string `json:"id"`+ StartedAt string `json:"startedAt"`+ EndedAt string `json:"endedAt,omitempty"`+ ClientIP string `json:"clientIP"`+ Attributes map[string]string `json:"attributes"`+}++type nativeRecording struct {+ ExceptionID string `json:"exceptionId"`+ SessionID string `json:"sessionId,omitempty"`+ Events []any `json:"events"`+ Logs []map[string]any `json:"logs,omitempty"`+ Actions []map[string]any `json:"actions,omitempty"`+ StartedAt string `json:"startedAt,omitempty"`+ EndedAt string `json:"endedAt,omitempty"`+}++// Crash is one exception reported by a browser or mobile SDK.+type Crash struct {+ At time.Time+ Stack string+ Attributes map[string]string+ IsMessage bool+ AppVersion string+ // Device is the SDK's serverName (the device or browser identity).+ Device string+ // SessionID links the crash to a browser session.+ SessionID string+ // Actions are what the SDK captured before the crash (network,+ // navigation, custom), shown with the occurrence.+ Actions []Action+ // Logs are console/logcat lines captured before the crash.+ Logs []ActionLog+}++// Action is one captured client action.+type Action struct {+ Type string // network | navigation | custom+ At time.Time+ Method string+ URL string+ DurationMs float64+ Status int+ Error string+ From, To string+ RespBytes int+}++// ActionLog is one captured client log line.+type ActionLog struct {+ At time.Time+ Level string+ Message string+}++func (a Action) wire() map[string]any {+ m := map[string]any{"type": a.Type, "timestamp": a.At.UTC().Format(time.RFC3339Nano)}+ switch a.Type {+ case "network":+ m["method"], m["url"], m["durationMs"] = a.Method, a.URL, a.DurationMs+ if a.Status != 0 {+ m["statusCode"] = a.Status+ }+ if a.Error != "" {+ m["error"] = a.Error+ }+ if a.RespBytes != 0 {+ m["responseBytes"] = a.RespBytes+ }+ case "navigation":+ m["action"] = "push"+ m["from"], m["to"] = a.From, a.To+ default:+ m["category"], m["name"] = "app", a.URL+ }+ return m+}++// Crash queues a native SDK exception for the project.+func (p *Project) Crash(c Crash) {+ f := nativeFrame{appVersion: c.AppVersion, serverName: c.Device}+ ex := nativeException{+ Stack: c.Stack, RecordedAt: c.At.UTC().Format(time.RFC3339Nano),+ Attributes: c.Attributes, IsMessage: c.IsMessage, SessionID: c.SessionID,+ }+ if ex.Attributes == nil {+ ex.Attributes = map[string]string{}+ }+ if len(c.Actions) > 0 || len(c.Logs) > 0 {+ ex.RecordingID = uuid.NewString()+ rec := nativeRecording{ExceptionID: ex.RecordingID, SessionID: c.SessionID, Events: []any{}}+ first, last := c.At, c.At+ for _, a := range c.Actions {+ rec.Actions = append(rec.Actions, a.wire())+ if a.At.Before(first) {+ first = a.At+ }+ p.s.record(a.URL, a.Error)+ }+ for _, l := range c.Logs {+ rec.Logs = append(rec.Logs, map[string]any{"timestamp": l.At.UTC().Format(time.RFC3339Nano), "level": l.Level, "message": l.Message})+ p.s.record(l.Message)+ }+ rec.StartedAt, rec.EndedAt = first.UTC().Format(time.RFC3339Nano), last.UTC().Format(time.RFC3339Nano)+ f.recordings = append(f.recordings, rec)+ }+ f.exceptions = append(f.exceptions, ex)+ p.native = append(p.native, f)+ p.s.record(c.Stack, c.Device)+ for _, v := range c.Attributes {+ p.s.record(v)+ }+}++// Session queues a browser session (only the browser SDK reports sessions).+func (p *Project) Session(id string, started, ended time.Time, attrs map[string]string, appVersion string) {+ if attrs == nil {+ attrs = map[string]string{}+ }+ p.native = append(p.native, nativeFrame{appVersion: appVersion, sessions: []nativeSession{{+ ID: id, StartedAt: started.UTC().Format(time.RFC3339Nano), EndedAt: ended.UTC().Format(time.RFC3339Nano),+ ClientIP: "203.0.113.24", Attributes: attrs,+ }}})+}++// nativeBodies groups queued frames by (appVersion, serverName) into+// /api/report bodies of at most 200 exceptions each.+func (p *Project) nativeBodies() ([][]byte, error) {+ type key struct{ v, d string }+ groups := map[key]*nativeFrame{}+ var order []key+ for _, f := range p.native {+ k := key{f.appVersion, f.serverName}+ g := groups[k]+ if g == nil {+ g = &nativeFrame{appVersion: f.appVersion, serverName: f.serverName}+ groups[k] = g+ order = append(order, k)+ }+ g.exceptions = append(g.exceptions, f.exceptions...)+ g.sessions = append(g.sessions, f.sessions...)+ g.recordings = append(g.recordings, f.recordings...)+ }+ var out [][]byte+ for _, k := range order {+ g := groups[k]+ frame := map[string]any{+ "traces": []any{}, "metrics": []any{},+ "stackTraces": g.exceptions, "sessions": g.sessions, "sessionRecordings": g.recordings,+ }+ if g.exceptions == nil {+ frame["stackTraces"] = []any{}+ }+ if g.sessions == nil {+ frame["sessions"] = []any{}+ }+ if g.recordings == nil {+ frame["sessionRecordings"] = []any{}+ }+ body, err := json.Marshal(map[string]any{+ "appVersion": g.appVersion, "serverName": g.serverName,+ "collectionFrames": []any{frame},+ })+ if err != nil {+ return nil, fmt.Errorf("native frame: %w", err)+ }+ out = append(out, body)+ }+ return out, nil+}diff --git a/cli/test/evals/internal/seed/req.go b/cli/test/evals/internal/seed/req.gonew file mode 100644index 00000000..a84fb1c0--- /dev/null+++ b/cli/test/evals/internal/seed/req.go@@ -0,0 +1,367 @@+package seed++import (+ "fmt"+ "math"+ "strings"+ "time"+)++// Req builds one request's span tree. Work is laid out on a cursor: every+// child starts where the previous one ended (plus any Gap), and a span's+// duration is where its cursor stopped, so a parent always covers its+// children exactly.+type Req struct {+ s *Scenario+ req *Request+ cur *Span+ cursor time.Duration+ status int+ forceDetail bool+ svc *Service // service owning cur (nil = the request's)+ base time.Duration // offset of cur's start from the request's start+}++// Request returns the request being built.+func (r *Req) Request() *Request { return r.req }++// At is the absolute time of the cursor.+func (r *Req) At() time.Time { return r.req.Start.Add(r.base + r.cursor) }++// Start is the request's start time.+func (r *Req) Start() time.Time { return r.req.Start }++// Progress is how far through the traffic window the request falls (0..1).+func (r *Req) Progress() float64 { return r.req.Progress }++// Host is the instance serving the request.+func (r *Req) Host() string { return r.req.Host }++// Version is the service.version live when the request started.+func (r *Req) Version() string { return r.req.Version }++// Float returns a uniform [0,1) draw.+func (r *Req) Float() float64 { return r.s.RNG.Float64() }++// Chance reports true with probability p.+func (r *Req) Chance(p float64) bool { return r.s.RNG.Float64() < p }++// IntN returns a uniform int in [0,n).+func (r *Req) IntN(n int) int { return r.s.RNG.IntN(n) }++// Ms returns a uniform duration between lo and hi milliseconds.+func (r *Req) Ms(lo, hi float64) time.Duration {+ return time.Duration((lo + (hi-lo)*r.s.RNG.Float64()) * float64(time.Millisecond))+}++// LogNormal draws a duration with the given median and p99.+func (r *Req) LogNormal(median, p99 time.Duration) time.Duration {+ mu := math.Log(float64(median))+ sigma := (math.Log(float64(p99)) - mu) / 2.326+ return time.Duration(math.Exp(mu + sigma*r.s.RNG.NormFloat64()))+}++// Growth interpolates from lo at the window start to hi at its end, with+// ±10% noise, as an integer.+func (r *Req) Growth(lo, hi float64) int {+ v := lo + (hi-lo)*r.req.Progress+ v *= 0.9 + 0.2*r.s.RNG.Float64()+ return int(math.Round(math.Max(0, v)))+}++// Lerp interpolates from lo to hi by window progress, without noise.+func (r *Req) Lerp(lo, hi float64) float64 { return lo + (hi-lo)*r.req.Progress }++// Status sets the response status.+func (r *Req) Status(code int) { r.status = code }++// KeepDetail forces this request's full span tree to be sent.+func (r *Req) KeepDetail() { r.forceDetail = true }++// Attr sets an attribute on the current span.+func (r *Req) Attr(k string, v any) {+ r.cur.Attrs[k] = v+ if s, ok := v.(string); ok {+ r.s.record(s)+ }+}++// RootAttr sets an attribute on the request's root span.+func (r *Req) RootAttr(k string, v any) {+ r.req.Root.Attrs[k] = v+ if s, ok := v.(string); ok {+ r.s.record(s)+ }+}++// Gap advances the cursor without a span: time the request spent waiting.+func (r *Req) Gap(d time.Duration) { r.cursor += d }++func (r *Req) owner() *Service {+ if r.svc != nil {+ return r.svc+ }+ return r.req.e.svc+}++func (r *Req) child(name string, kind int, scope string, d time.Duration, attrs map[string]any) *Span {+ sp := &Span{Name: name, Kind: kind, Start: r.cursor, Dur: d, Attrs: attrs, Scope: scope}+ fillRandom(r.s, sp.SpanID[:])+ r.cur.Children = append(r.cur.Children, sp)+ r.cursor += d+ r.s.record(name)+ for _, v := range attrs {+ if s, ok := v.(string); ok {+ r.s.record(s)+ }+ }+ return sp+}++// Compute records in-process work as an INTERNAL span.+func (r *Req) Compute(name string, d time.Duration, attrs ...KV) *Span {+ return r.child(name, KindInternal, r.owner().Stack.ServerScope, d, kvMap(attrs))+}++// Span records an INTERNAL span whose children are built by f; its duration+// is whatever f lays out.+func (r *Req) Span(name string, f func(r *Req), attrs ...KV) *Span {+ sp := &Span{Name: name, Kind: KindInternal, Start: r.cursor, Attrs: kvMap(attrs), Scope: r.owner().Stack.ServerScope}+ fillRandom(r.s, sp.SpanID[:])+ r.cur.Children = append(r.cur.Children, sp)+ r.s.record(name)+ sub := &Req{s: r.s, req: r.req, cur: sp, cursor: 0, svc: r.svc, base: r.base + sp.Start}+ f(sub)+ sp.Dur = sub.cursor+ r.cursor += sp.Dur+ r.forceDetail = r.forceDetail || sub.forceDetail+ if sub.status != 0 {+ r.status = sub.status+ }+ return sp+}++// Parallel runs branches concurrently from the current cursor; the cursor+// then moves to the end of the longest branch.+func (r *Req) Parallel(branches ...func(r *Req)) {+ start := r.cursor+ longest := start+ for _, b := range branches {+ sub := &Req{s: r.s, req: r.req, cur: r.cur, cursor: start, svc: r.svc, base: r.base}+ b(sub)+ if sub.cursor > longest {+ longest = sub.cursor+ }+ r.forceDetail = r.forceDetail || sub.forceDetail+ if sub.status != 0 {+ r.status = sub.status+ }+ }+ r.cursor = longest+}++// KV is an attribute pair.+type KV struct {+ K string+ V any+}++// A is shorthand for a KV.+func A(k string, v any) KV { return KV{k, v} }++func kvMap(kvs []KV) map[string]any {+ m := map[string]any{}+ for _, kv := range kvs {+ m[kv.K] = kv.V+ }+ return m+}++// DB records a database CLIENT span for query. The span name follows the+// semantic conventions ("{operation} {collection}").+func (r *Req) DB(query string, d time.Duration, attrs ...KV) *Span {+ st := r.owner().Stack+ op, coll := sqlParts(query)+ name := op+ if coll != "" {+ name = op + " " + coll+ }+ m := map[string]any{+ "db.system.name": st.DBSystem,+ "db.query.text": query,+ "db.namespace": r.owner().dbName(),+ "db.operation.name": op,+ "server.address": r.owner().dbHost(),+ "server.port": 5432,+ }+ if st.DBSystem == "mysql" {+ m["server.port"] = 3306+ }+ if coll != "" {+ m["db.collection.name"] = coll+ }+ for _, kv := range attrs {+ m[kv.K] = kv.V+ }+ return r.child(name, KindClient, st.DBScope, d, m)+}++// Rows is the db.response.returned_rows attribute.+func Rows(n int) KV { return KV{"db.response.returned_rows", n} }++func sqlParts(q string) (op, coll string) {+ fields := strings.Fields(q)+ if len(fields) == 0 {+ return "QUERY", ""+ }+ op = strings.ToUpper(fields[0])+ for i, f := range fields {+ u := strings.ToUpper(f)+ if (u == "FROM" || u == "INTO" || (u == "UPDATE" && i == 0)) && i+1 < len(fields) {+ coll = strings.Trim(fields[i+1], "(),;")+ break+ }+ }+ return op, strings.ToLower(coll)+}++func (svc *Service) dbName() string {+ return strings.ReplaceAll(strings.TrimSuffix(strings.TrimSuffix(svc.Name, "-api"), "-service"), "-", "_")+}++func (svc *Service) dbHost() string { return svc.dbName() + "-db.internal" }++// Cache records a cache CLIENT span (redis).+func (r *Req) Cache(cmd, key string, d time.Duration, attrs ...KV) *Span {+ m := map[string]any{+ "db.system.name": "redis",+ "db.operation.name": cmd,+ "db.query.text": cmd + " " + key,+ "server.address": "cache.internal",+ "server.port": 6379,+ }+ for _, kv := range attrs {+ m[kv.K] = kv.V+ }+ return r.child(cmd, KindClient, r.owner().Stack.CacheScope, d, m)+}++// HTTP records an outgoing HTTP CLIENT span. The span name is the method,+// as the semantic conventions specify for client spans.+func (r *Req) HTTP(method, url string, d time.Duration, status int, attrs ...KV) *Span {+ host, port := hostOf(url)+ m := map[string]any{+ "http.request.method": method,+ "url.full": url,+ "server.address": host,+ "server.port": port,+ "http.response.status_code": status,+ }+ for _, kv := range attrs {+ m[kv.K] = kv.V+ }+ sp := r.child(method, KindClient, r.owner().Stack.HTTPClient, d, m)+ if status >= 500 {+ sp.Status = StatusError+ }+ return sp+}++func hostOf(u string) (string, int) {+ rest := u+ port := 443+ if strings.HasPrefix(rest, "http://") {+ rest, port = strings.TrimPrefix(rest, "http://"), 80+ }+ rest = strings.TrimPrefix(rest, "https://")+ if i := strings.IndexAny(rest, "/?"); i >= 0 {+ rest = rest[:i]+ }+ if h, p, ok := strings.Cut(rest, ":"); ok {+ var n int+ fmt.Sscanf(p, "%d", &n)+ return h, n+ }+ return rest, port+}++// Call makes an HTTP call to another service's endpoint: a CLIENT span here+// and, under it, the downstream SERVER span built by the target's handler.+// rtt is the network round trip added around the server time.+func (r *Req) Call(target *Endpoint, rtt time.Duration) *Span {+ tsvc := target.svc+ url := fmt.Sprintf("http://%s.internal:8080%s", tsvc.Name, concretePath(r.s, target.Route))+ client := &Span{Name: target.Method, Kind: KindClient, Start: r.cursor, Scope: r.owner().Stack.HTTPClient,+ Attrs: map[string]any{"http.request.method": target.Method, "url.full": url, "server.address": tsvc.Name + ".internal", "server.port": 8080}}+ fillRandom(r.s, client.SpanID[:])+ r.cur.Children = append(r.cur.Children, client)++ server := &Span{Name: target.Name(), Kind: KindServer, Start: rtt / 2, Scope: tsvc.Stack.ServerScope, Service: tsvc, Host: tsvc.pickHost(),+ Attrs: map[string]any{"http.request.method": target.Method, "http.route": target.Route, "url.path": concretePath(r.s, target.Route), "url.scheme": "http"}}+ fillRandom(r.s, server.SpanID[:])+ client.Children = append(client.Children, server)+ sub := &Req{s: r.s, req: r.req, cur: server, svc: tsvc, base: r.base + client.Start + server.Start}+ if target.handler != nil {+ target.handler(sub)+ }+ server.Dur = sub.cursor+ status := 200+ if sub.status != 0 {+ status = sub.status+ }+ server.Attrs["http.response.status_code"] = status+ client.Attrs["http.response.status_code"] = status+ if status >= 500 {+ server.Status, client.Status = StatusError, StatusError+ }+ client.Dur = server.Dur + rtt+ r.cursor += client.Dur+ r.forceDetail = r.forceDetail || sub.forceDetail+ return client+}++// Log emits a log record correlated with the current span at the cursor.+// Logged requests are always sent in detail so the trace id resolves.+func (r *Req) Log(severity int, body string, attrs ...KV) {+ r.cur.logs = append(r.cur.logs, logRecord{at: r.cursor, severity: severity, body: body, attrs: kvMap(attrs)})+ r.forceDetail = true+ r.s.record(body)+ for _, kv := range attrs {+ if s, ok := kv.V.(string); ok {+ r.s.record(s)+ }+ }+}++// Exc is an exception to record on a span.+type Exc struct {+ Type string+ Message string+ Stack string+ // Escaped marks exception.escaped.+ Escaped bool+}++// Error records an exception event on the current span at the cursor and+// marks the span (and, for escaped errors, the request) as failed.+func (r *Req) Error(e Exc) {+ attrs := map[string]any{"exception.type": e.Type, "exception.message": e.Message}+ if e.Stack != "" {+ attrs["exception.stacktrace"] = e.Stack+ }+ if e.Escaped {+ attrs["exception.escaped"] = true+ }+ r.cur.Events = append(r.cur.Events, Event{Name: "exception", At: r.cursor, Attrs: attrs})+ r.cur.Status = StatusError+ r.cur.StatusMsg = e.Message+ r.forceDetail = true+ r.s.record(e.Type, e.Message, e.Stack)+}++func (r *Req) finish() {+ r.req.Root.Dur = r.cursor+ if r.req.Root.Dur <= 0 {+ r.req.Root.Dur = time.Millisecond+ }+}diff --git a/cli/test/evals/internal/seed/semconv.go b/cli/test/evals/internal/seed/semconv.gonew file mode 100644index 00000000..30a3e026--- /dev/null+++ b/cli/test/evals/internal/seed/semconv.go@@ -0,0 +1,149 @@+package seed++// Stack fixes what one service's telemetry looks like: the instrumentation+// scope names its libraries really report, the SDK language, the database+// system, and which runtime metrics it emits. Names follow the OpenTelemetry+// semantic conventions as of 2026 (http.request.method, http.route,+// http.response.status_code, db.system.name, db.query.text, ...).+type Stack struct {+ Language string // telemetry.sdk.language+ Framework string // label only+ ServerScope string // scope of the HTTP SERVER span+ DBScope string // scope of database CLIENT spans+ DBSystem string // db.system.name+ HTTPClient string // scope of outgoing HTTP CLIENT spans+ CacheScope string // scope of cache (redis) spans+ Runtime string // go | jvm | python | node | dotnet | php+ // StackFormat selects how exception.stacktrace is rendered.+ StackFormat string+}++// Go returns a Go service. web: gin | nethttp | chi. db: pgx | sql.+func Go(web, db string) Stack {+ s := Stack{Language: "go", Framework: "go/" + web, Runtime: "go", StackFormat: "go",+ HTTPClient: "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",+ CacheScope: "github.com/redis/go-redis/extra/redisotel"}+ switch web {+ case "gin":+ s.ServerScope = "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"+ case "chi":+ s.ServerScope = "github.com/riandyrn/otelchi"+ default:+ s.ServerScope = "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"+ }+ switch db {+ case "pgx":+ s.DBScope, s.DBSystem = "github.com/exaring/otelpgx", "postgresql"+ case "mysql":+ s.DBScope, s.DBSystem = "github.com/XSAM/otelsql", "mysql"+ default:+ s.DBScope, s.DBSystem = "github.com/XSAM/otelsql", "postgresql"+ }+ return s+}++// Python returns a Python service. web: fastapi | django | flask.+func Python(web, db string) Stack {+ s := Stack{Language: "python", Framework: "python/" + web, Runtime: "python", StackFormat: "python",+ HTTPClient: "opentelemetry.instrumentation.httpx", CacheScope: "opentelemetry.instrumentation.redis"}+ switch web {+ case "django":+ s.ServerScope = "opentelemetry.instrumentation.django"+ case "flask":+ s.ServerScope = "opentelemetry.instrumentation.flask"+ default:+ s.ServerScope = "opentelemetry.instrumentation.fastapi"+ }+ switch db {+ case "psycopg":+ s.DBScope, s.DBSystem = "opentelemetry.instrumentation.psycopg", "postgresql"+ default:+ s.DBScope, s.DBSystem = "opentelemetry.instrumentation.sqlalchemy", "postgresql"+ }+ return s+}++// Node returns a Node.js service. web: express | nestjs | fastify.+func Node(web, db string) Stack {+ s := Stack{Language: "nodejs", Framework: "node/" + web, Runtime: "node", StackFormat: "node",+ HTTPClient: "@opentelemetry/instrumentation-undici", CacheScope: "@opentelemetry/instrumentation-ioredis"}+ switch web {+ case "nestjs":+ s.ServerScope = "@opentelemetry/instrumentation-http"+ case "fastify":+ s.ServerScope = "@opentelemetry/instrumentation-fastify"+ default:+ s.ServerScope = "@opentelemetry/instrumentation-http"+ }+ switch db {+ case "mysql":+ s.DBScope, s.DBSystem = "@opentelemetry/instrumentation-mysql2", "mysql"+ default:+ s.DBScope, s.DBSystem = "@opentelemetry/instrumentation-pg", "postgresql"+ }+ return s+}++// Java returns a JVM service instrumented by the OpenTelemetry Java agent.+func Java(web string) Stack {+ return Stack{Language: "java", Framework: "java/" + web, Runtime: "jvm", StackFormat: "java",+ ServerScope: "io.opentelemetry.tomcat-10.0", DBScope: "io.opentelemetry.jdbc", DBSystem: "postgresql",+ HTTPClient: "io.opentelemetry.java-http-client", CacheScope: "io.opentelemetry.lettuce-5.1"}+}++// PHP returns a PHP service (Laravel auto-instrumentation).+func PHP(web string) Stack {+ return Stack{Language: "php", Framework: "php/" + web, Runtime: "php", StackFormat: "php",+ ServerScope: "io.opentelemetry.contrib.php.laravel", DBScope: "io.opentelemetry.contrib.php.pdo", DBSystem: "mysql",+ HTTPClient: "io.opentelemetry.contrib.php.guzzle", CacheScope: "io.opentelemetry.contrib.php.laravel"}+}++// DotNet returns an ASP.NET Core service.+func DotNet() Stack {+ return Stack{Language: "dotnet", Framework: "dotnet/aspnetcore", Runtime: "dotnet", StackFormat: "dotnet",+ ServerScope: "Microsoft.AspNetCore", DBScope: "Npgsql", DBSystem: "postgresql",+ HTTPClient: "System.Net.Http", CacheScope: "OpenTelemetry.Instrumentation.StackExchangeRedis"}+}++// Span kinds (OTLP enum values).+const (+ KindInternal = 1+ KindServer = 2+ KindClient = 3+ KindProducer = 4+ KindConsumer = 5+)++// Status codes (OTLP enum values).+const (+ StatusUnset = 0+ StatusOK = 1+ StatusError = 2+)++// OTel log severity numbers.+const (+ SevDebug = 5+ SevInfo = 9+ SevWarn = 13+ SevError = 17+ SevFatal = 21+)++// SeverityText maps a severity number to its OTel short name.+func SeverityText(n int) string {+ switch {+ case n >= 21:+ return "FATAL"+ case n >= 17:+ return "ERROR"+ case n >= 13:+ return "WARN"+ case n >= 9:+ return "INFO"+ case n >= 5:+ return "DEBUG"+ default:+ return "TRACE"+ }+}diff --git a/cli/test/evals/internal/seed/sink.go b/cli/test/evals/internal/seed/sink.gonew file mode 100644index 00000000..238c1bdf--- /dev/null+++ b/cli/test/evals/internal/seed/sink.go@@ -0,0 +1,195 @@+package seed++import (+ "bytes"+ "compress/gzip"+ "context"+ "encoding/json"+ "fmt"+ "net/http"+ "net/url"+ "sync"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+)++// Sink is where a scenario's telemetry and objects go: a live backend, or a+// dry recorder used by the leakage lint and unit tests.+type Sink interface {+ CreateProject(name, framework string) (id, token string, err error)+ // Ingest posts a body to an ingest path with a project token and+ // returns the response body.+ Ingest(token, path, contentType string, body []byte, gz bool) ([]byte, error)+ // Admin calls the dashboard API as the instance owner.+ Admin(method, path string, query url.Values, body any) (*env.Response, error)+ // Dry reports a recorder that has no backend behind it.+ Dry() bool+ // Org is the instance's organization and owner.+ Org() (orgID, adminUserID int)+ // Stub is the switchable HTTP target for monitors (nil when absent).+ Stub() StubControl+}++// StubControl is the part of env.Stub the seed uses.+type StubControl interface {+ SetHealthy(bool)+ Healthy() bool+}++// LiveSink sends to a running backend.+type LiveSink struct {+ Ctx context.Context+ Backend *env.Backend+ Admin_ *env.Admin+ HTTP *http.Client++ StubSrv *env.Stub++ mu sync.Mutex+ projects []env.Project+}++// NewLiveSink returns a sink for a booted, registered instance.+func NewLiveSink(ctx context.Context, b *env.Backend, a *env.Admin) *LiveSink {+ return &LiveSink{Ctx: ctx, Backend: b, Admin_: a, HTTP: &http.Client{Timeout: 2 * time.Minute}}+}++// Projects returns the projects created so far, tokens included.+func (l *LiveSink) Projects() []env.Project {+ l.mu.Lock()+ defer l.mu.Unlock()+ return append([]env.Project{}, l.projects...)+}++// CreateProject creates a project in the admin's organization.+func (l *LiveSink) CreateProject(name, framework string) (string, string, error) {+ ps, err := env.CreateProjects(l.Ctx, l.Backend, l.Admin_, []env.Project{{Name: name, Framework: framework}})+ if err != nil {+ return "", "", err+ }+ l.mu.Lock()+ l.projects = append(l.projects, ps[0])+ l.mu.Unlock()+ return ps[0].ID, ps[0].Token, nil+}++// Ingest posts with retries on the admission gate's 503.+func (l *LiveSink) Ingest(token, path, contentType string, body []byte, gz bool) ([]byte, error) {+ payload := body+ if gz {+ var buf bytes.Buffer+ w := gzip.NewWriter(&buf)+ _, _ = w.Write(body)+ _ = w.Close()+ payload = buf.Bytes()+ }+ var last error+ for attempt := 0; attempt < 6; attempt++ {+ req, err := http.NewRequestWithContext(l.Ctx, http.MethodPost, l.Backend.URL+path, bytes.NewReader(payload))+ if err != nil {+ return nil, err+ }+ req.Header.Set("Authorization", "Bearer "+token)+ req.Header.Set("Content-Type", contentType)+ if gz {+ req.Header.Set("Content-Encoding", "gzip")+ }+ resp, err := l.HTTP.Do(req)+ if err != nil {+ last = err+ } else {+ var msg bytes.Buffer+ _, _ = msg.ReadFrom(resp.Body)+ resp.Body.Close()+ if resp.StatusCode == http.StatusOK {+ return msg.Bytes(), nil+ }+ last = fmt.Errorf("POST %s: HTTP %d: %s", path, resp.StatusCode, msg.String())+ if resp.StatusCode != http.StatusServiceUnavailable && resp.StatusCode != http.StatusTooManyRequests {+ return nil, last+ }+ }+ time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond)+ }+ return nil, last+}++// Admin calls the dashboard API with the owner's session.+func (l *LiveSink) Admin(method, path string, query url.Values, body any) (*env.Response, error) {+ return l.Backend.API().WithBearer(l.Admin_.JWT).Do(l.Ctx, method, path, query, body)+}++// Dry reports false.+func (l *LiveSink) Dry() bool { return false }++// Org returns the organization and owner ids.+func (l *LiveSink) Org() (int, int) { return l.Admin_.OrgID, l.Admin_.UserID }++// Stub returns the monitor target.+func (l *LiveSink) Stub() StubControl {+ if l.StubSrv == nil {+ return nil+ }+ return l.StubSrv+}++// StubURL returns the stub's base URL.+func (l *LiveSink) StubURL() string {+ if l.StubSrv == nil {+ return ""+ }+ return l.StubSrv.URL+}++// DrySink records what would be sent.+type DrySink struct {+ mu sync.Mutex+ nextID int+ Bodies map[string]int // path -> count+ Calls []string+ project int+}++// NewDrySink returns an empty recorder.+func NewDrySink() *DrySink { return &DrySink{Bodies: map[string]int{}} }++// CreateProject returns a synthetic id and token.+func (d *DrySink) CreateProject(name, framework string) (string, string, error) {+ d.mu.Lock()+ defer d.mu.Unlock()+ d.project+++ return fmt.Sprintf("00000000-0000-4000-8000-%012d", d.project), fmt.Sprintf("dry-token-%d", d.project), nil+}++// Ingest counts the body.+func (d *DrySink) Ingest(_, path, _ string, _ []byte, _ bool) ([]byte, error) {+ d.mu.Lock()+ defer d.mu.Unlock()+ d.Bodies[path]+++ return nil, nil+}++// Admin answers creates with a synthetic id and everything else with {}.+func (d *DrySink) Admin(method, path string, _ url.Values, _ any) (*env.Response, error) {+ d.mu.Lock()+ defer d.mu.Unlock()+ d.Calls = append(d.Calls, method+" "+path)+ d.nextID+++ body, _ := json.Marshal(map[string]any{"id": d.nextID, "updatedAt": time.Unix(0, 0).UTC().Format(time.RFC3339Nano)})+ return &env.Response{Status: http.StatusOK, Body: body}, nil+}++// Dry reports true.+func (d *DrySink) Dry() bool { return true }++// Org returns fixed ids.+func (d *DrySink) Org() (int, int) { return 1, 1 }++// Stub returns a stub that only remembers its state.+func (d *DrySink) Stub() StubControl { return &dryStub{} }++type dryStub struct{ healthy bool }++func (s *dryStub) SetHealthy(v bool) { s.healthy = v }+func (s *dryStub) Healthy() bool { return s.healthy }diff --git a/cli/test/evals/internal/seed/traces.go b/cli/test/evals/internal/seed/traces.gonew file mode 100644index 00000000..38aa5ace--- /dev/null+++ b/cli/test/evals/internal/seed/traces.go@@ -0,0 +1,247 @@+package seed++import (+ "fmt"+ "sort"+ "time"++ "github.com/google/uuid"+ coltracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"+ commonpb "go.opentelemetry.io/proto/otlp/common/v1"+ resourcepb "go.opentelemetry.io/proto/otlp/resource/v1"+ tracepb "go.opentelemetry.io/proto/otlp/trace/v1"+)++// EntityID is the id the backend gives the endpoint/task/exception row+// promoted from a span: UUIDv5(projectId, traceId || spanId).+func EntityID(projectID string, traceID [16]byte, spanID [8]byte) string {+ pid, err := uuid.Parse(projectID)+ if err != nil {+ return ""+ }+ var identity [24]byte+ copy(identity[:16], traceID[:])+ copy(identity[16:], spanID[:])+ return uuid.NewSHA1(pid, identity[:]).String()+}++// resourceKey groups spans that share one OTLP Resource.+type resourceKey struct {+ svc *Service+ version string+ host string+}++// spanBatch accumulates the spans of one project, grouped by resource.+type spanBatch struct {+ project *Project+ groups map[resourceKey]map[string][]*tracepb.Span // scope -> spans+ count int+}++func newSpanBatch(p *Project) *spanBatch {+ return &spanBatch{project: p, groups: map[resourceKey]map[string][]*tracepb.Span{}}+}++func (b *spanBatch) add(k resourceKey, scope string, sp *tracepb.Span) {+ g := b.groups[k]+ if g == nil {+ g = map[string][]*tracepb.Span{}+ b.groups[k] = g+ }+ g[scope] = append(g[scope], sp)+ b.count+++}++func (b *spanBatch) request() *coltracepb.ExportTraceServiceRequest {+ keys := make([]resourceKey, 0, len(b.groups))+ for k := range b.groups {+ keys = append(keys, k)+ }+ sort.Slice(keys, func(i, j int) bool {+ if keys[i].svc.Name != keys[j].svc.Name {+ return keys[i].svc.Name < keys[j].svc.Name+ }+ if keys[i].version != keys[j].version {+ return keys[i].version < keys[j].version+ }+ return keys[i].host < keys[j].host+ })+ req := &coltracepb.ExportTraceServiceRequest{}+ for _, k := range keys {+ rs := &tracepb.ResourceSpans{Resource: k.svc.resource(k.version, k.host)}+ scopes := make([]string, 0, len(b.groups[k]))+ for sc := range b.groups[k] {+ scopes = append(scopes, sc)+ }+ sort.Strings(scopes)+ for _, sc := range scopes {+ rs.ScopeSpans = append(rs.ScopeSpans, &tracepb.ScopeSpans{+ Scope: &commonpb.InstrumentationScope{Name: sc, Version: scopeVersion(sc)},+ Spans: b.groups[k][sc],+ })+ }+ req.ResourceSpans = append(req.ResourceSpans, rs)+ }+ return req+}++func scopeVersion(scope string) string {+ switch {+ case len(scope) > 0 && scope[0] == '@':+ return "0.57.0"+ case len(scope) > 3 && scope[:3] == "io.":+ return "2.12.0"+ default:+ return "0.63.0"+ }+}++// resource builds the OTLP Resource of a service instance.+func (svc *Service) resource(version, host string) *resourcepb.Resource {+ attrs := []*commonpb.KeyValue{+ str("service.name", svc.Name),+ str("service.version", version),+ str("deployment.environment.name", svc.Env),+ str("host.name", host),+ str("telemetry.sdk.language", svc.Stack.Language),+ str("telemetry.sdk.name", "opentelemetry"),+ str("telemetry.sdk.version", sdkVersion(svc.Stack.Language)),+ }+ return &resourcepb.Resource{Attributes: attrs}+}++func sdkVersion(lang string) string {+ switch lang {+ case "go":+ return "1.38.0"+ case "python":+ return "1.37.0"+ case "nodejs":+ return "2.1.0"+ case "java":+ return "1.55.0"+ case "dotnet":+ return "1.13.1"+ default:+ return "1.0.0"+ }+}++func str(k, v string) *commonpb.KeyValue {+ return &commonpb.KeyValue{Key: k, Value: &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: v}}}+}++// attrList converts a map to OTLP attributes. Integers become intValue, as the+// backend reads numeric semantic-convention attributes only from intValue.+func attrList(m map[string]any) []*commonpb.KeyValue {+ keys := make([]string, 0, len(m))+ for k := range m {+ keys = append(keys, k)+ }+ sort.Strings(keys)+ out := make([]*commonpb.KeyValue, 0, len(keys))+ for _, k := range keys {+ out = append(out, &commonpb.KeyValue{Key: k, Value: anyValue(m[k])})+ }+ return out+}++func anyValue(v any) *commonpb.AnyValue {+ switch t := v.(type) {+ case string:+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: t}}+ case int:+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: int64(t)}}+ case int64:+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_IntValue{IntValue: t}}+ case float64:+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_DoubleValue{DoubleValue: t}}+ case bool:+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_BoolValue{BoolValue: t}}+ case []string:+ vals := make([]*commonpb.AnyValue, 0, len(t))+ for _, s := range t {+ vals = append(vals, anyValue(s))+ }+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_ArrayValue{ArrayValue: &commonpb.ArrayValue{Values: vals}}}+ default:+ return &commonpb.AnyValue{Value: &commonpb.AnyValue_StringValue{StringValue: fmt.Sprint(t)}}+ }+}++func nanos(t time.Time) uint64 { return uint64(t.UnixNano()) }++// emit walks a request tree and adds its spans to per-project batches. A+// non-detailed request sends only its root and every downstream SERVER span,+// so endpoint rows and percentiles stay complete while the span volume stays+// bounded.+func (s *Scenario) emit(r *Request, batches map[*Project]*spanBatch, logs map[*Service][]logOut) {+ rootSvc := r.e.svc+ var walk func(sp *Span, parent *Span, svc *Service, host string, abs time.Time, depthDetail bool)+ walk = func(sp *Span, parent *Span, svc *Service, host string, abs time.Time, detail bool) {+ if sp.Service != nil {+ svc = sp.Service+ }+ if sp.Host != "" {+ host = sp.Host+ }+ start := abs.Add(sp.Start)+ send := detail || parent == nil || sp.Kind == KindServer+ if send {+ pb := &tracepb.Span{+ TraceId: r.TraceID[:],+ SpanId: sp.SpanID[:],+ Name: sp.Name,+ Kind: tracepb.Span_SpanKind(sp.Kind),+ StartTimeUnixNano: nanos(start),+ EndTimeUnixNano: nanos(start.Add(sp.Dur)),+ Attributes: attrList(sp.Attrs),+ Status: &tracepb.Status{Code: tracepb.Status_StatusCode(sp.Status), Message: sp.StatusMsg},+ }+ if parent != nil {+ pb.ParentSpanId = parent.SpanID[:]+ }+ for _, ev := range sp.Events {+ pb.Events = append(pb.Events, &tracepb.Span_Event{+ TimeUnixNano: nanos(start.Add(ev.At)),+ Name: ev.Name,+ Attributes: attrList(ev.Attrs),+ })+ }+ b := batches[svc.p]+ if b == nil {+ b = newSpanBatch(svc.p)+ batches[svc.p] = b+ }+ b.add(resourceKey{svc: svc, version: svc.VersionAt(r.Start), host: host}, sp.Scope, pb)+ s.stats.Spans+++ if len(sp.Events) > 0 {+ s.stats.Exceptions += len(sp.Events)+ }+ }+ if detail {+ for _, l := range sp.logs {+ logs[svc] = append(logs[svc], logOut{rec: l, at: start.Add(l.at), traceID: r.TraceID, spanID: sp.SpanID, host: host, version: svc.VersionAt(r.Start)})+ }+ }+ for _, c := range sp.Children {+ walk(c, sp, svc, host, start, detail)+ }+ }+ walk(r.Root, nil, rootSvc, r.Host, r.Start, r.Detail)+ if r.Detail && !r.e.Task && !r.e.SkipAccessLog {+ // The access log line a request logger writes on completion,+ // correlated with the trace like any OTel log bridge does.+ end := r.Start.Add(r.Root.Dur)+ path, _ := r.Root.Attrs["url.path"].(string)+ body := fmt.Sprintf("%s %s %d %dms", r.e.Method, path, r.Status, r.Root.Dur.Milliseconds())+ logs[rootSvc] = append(logs[rootSvc], logOut{+ rec: logRecord{severity: SevInfo, body: body, attrs: map[string]any{+ "http.request.method": r.e.Method, "url.path": path, "http.route": r.e.Route,+ "http.response.status_code": r.Status, "duration_ms": r.Root.Dur.Milliseconds(),+ }},+ at: end, traceID: r.TraceID, spanID: r.SpanID, host: r.Host, version: r.Version,+ })+ }+}diff --git a/cli/test/evals/internal/seed/verify.go b/cli/test/evals/internal/seed/verify.gonew file mode 100644index 00000000..b35a81a2--- /dev/null+++ b/cli/test/evals/internal/seed/verify.go@@ -0,0 +1,109 @@+package seed++import (+ "encoding/json"+ "fmt"+ "net/http"+ "strings"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+)++// Verify reads the seeded data back through the dashboard API until every+// declared endpoint is listed, so a run never starts on half-ingested data.+// After timeout the run is an infra_error.+func (s *Scenario) Verify(timeout time.Duration) error {+ if s.Sink.Dry() {+ return nil+ }+ type want struct {+ p *Project+ names []string+ win time.Duration+ }+ var wants []want+ for _, p := range s.projects {+ if p.Kind != OTel {+ continue+ }+ w := want{p: p}+ for _, svc := range s.services {+ if svc.p != p {+ continue+ }+ for _, e := range svc.endpoints {+ if len(e.shapes) == 0 || len(e.Requests()) == 0 {+ continue+ }+ w.names = append(w.names, e.Name())+ if e.window > w.win {+ w.win = e.window+ }+ }+ }+ if len(w.names) > 0 {+ if w.win == 0 {+ w.win = 24 * time.Hour+ }+ wants = append(wants, w)+ }+ }+ deadline := time.Now().Add(timeout)+ for _, w := range wants {+ for {+ missing, err := s.missingEndpoints(w.p, w.names, w.win)+ if err == nil && len(missing) == 0 {+ break+ }+ if time.Now().After(deadline) {+ if err != nil {+ return fmt.Errorf("verify %s: %w", w.p.Name, err)+ }+ return fmt.Errorf("verify %s: endpoints never became visible: %s", w.p.Name, strings.Join(missing, ", "))+ }+ time.Sleep(250 * time.Millisecond)+ }+ }+ return nil+}++func (s *Scenario) missingEndpoints(p *Project, names []string, win time.Duration) ([]string, error) {+ seen := map[string]bool{}+ for page := 1; page <= 5; page++ {+ res, err := s.Sink.Admin(http.MethodPost, "/api/endpoints/grouped", env.ProjectQuery(p.ID), map[string]any{+ "fromDate": s.Now.Add(-win - time.Hour).Format(time.RFC3339), "toDate": s.Now.Add(time.Hour).Format(time.RFC3339),+ "orderBy": "count", "pagination": map[string]int{"page": page, "pageSize": 100},+ })+ if err != nil {+ return nil, err+ }+ if err := res.Err("list endpoints"); err != nil {+ return nil, err+ }+ var out struct {+ Data []struct {+ Endpoint string `json:"endpoint"`+ } `json:"data"`+ Pagination struct {+ TotalPages int `json:"totalPages"`+ } `json:"pagination"`+ }+ if err := json.Unmarshal(res.Body, &out); err != nil {+ return nil, err+ }+ for _, d := range out.Data {+ seen[d.Endpoint] = true+ }+ if page >= out.Pagination.TotalPages {+ break+ }+ }+ var missing []string+ for _, n := range names {+ if !seen[n] {+ missing = append(missing, n)+ }+ }+ return missing, nil+}diff --git a/cli/test/evals/internal/suite/fixtures_test.go b/cli/test/evals/internal/suite/fixtures_test.gonew file mode 100644index 00000000..20857cd8--- /dev/null+++ b/cli/test/evals/internal/suite/fixtures_test.go@@ -0,0 +1,63 @@+package suite_test++import (+ "io/fs"+ "os"+ "path/filepath"+ "testing"++ "gopkg.in/yaml.v3"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+)++// TestFixtureHistory checks every fixture's history.yaml: each overlay+// directory holds at least one file (git does not keep empty directories, so+// an empty overlay exists only in the checkout that made it), and every+// absent path is a file of the fixture.+func TestFixtureHistory(t *testing.T) {+ files, err := filepath.Glob(filepath.Join(env.EvalsRoot(), "testdata", "repos", "*", "history.yaml"))+ if err != nil {+ t.Fatal(err)+ }+ for _, f := range files {+ repo := filepath.Dir(f)+ t.Run(filepath.Base(repo), func(t *testing.T) {+ raw, err := os.ReadFile(f)+ if err != nil {+ t.Fatal(err)+ }+ var hist struct {+ Commits []struct {+ Dir string `yaml:"dir"`+ Absent []string `yaml:"absent"`+ } `yaml:"commits"`+ }+ if err := yaml.Unmarshal(raw, &hist); err != nil {+ t.Fatal(err)+ }+ for i, c := range hist.Commits {+ if c.Dir != "" && !hasFile(filepath.Join(repo, c.Dir)) {+ t.Errorf("commit %d: overlay %s has no files; drop its dir: key", i+1, c.Dir)+ }+ for _, a := range c.Absent {+ if !exists(filepath.Join(repo, filepath.FromSlash(a))) {+ t.Errorf("commit %d: absent path %s is not in the fixture", i+1, a)+ }+ }+ }+ })+ }+}++func hasFile(dir string) bool {+ found := false+ _ = filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {+ if err == nil && !d.IsDir() {+ found = true+ return fs.SkipAll+ }+ return nil+ })+ return found+}diff --git a/cli/test/evals/internal/suite/suite_test.go b/cli/test/evals/internal/suite/suite_test.gonew file mode 100644index 00000000..905e0295--- /dev/null+++ b/cli/test/evals/internal/suite/suite_test.go@@ -0,0 +1,134 @@+// Package suite_test checks the case catalog as a whole: every case loads,+// references things that exist, uses valid checklist ids, and seeds data+// that does not give the answer away.+package suite_test++import (+ "fmt"+ "os"+ "path/filepath"+ "slices"+ "strings"+ "testing"+ "time"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/checklist"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/env"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/grade"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios"+ _ "github.com/tracewayapp/traceway/cli/test/evals/internal/scenarios/all"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/seed"+)++func loadAll(t *testing.T) []*catalog.Case {+ t.Helper()+ cases, err := catalog.LoadCases(env.EvalsRoot())+ if err != nil {+ t.Fatal(err)+ }+ if len(cases) == 0 {+ t.Fatal("no cases")+ }+ return cases+}++func exists(p string) bool { _, err := os.Stat(p); return err == nil }++func TestCaseReferences(t *testing.T) {+ root := env.EvalsRoot()+ seen := map[string]bool{}+ for _, c := range loadAll(t) {+ t.Run(c.ID, func(t *testing.T) {+ if _, err := scenarios.Get(c.Seed); err != nil {+ t.Error(err)+ }+ seen[c.Seed] = true+ if r := c.Workspace.Repo; r != "" && !exists(filepath.Join(root, "testdata", "repos", r)) {+ t.Errorf("fixture %s does not exist", r)+ }+ checkIDs := func(where string, ids ...string) {+ for _, id := range ids {+ if id != "" && !strings.Contains(id, "{{") && !checklist.IsID(id) {+ t.Errorf("%s: %q is not a checklist id (valid: %s)", where, id, strings.Join(checklist.IDs(), ", "))+ }+ }+ }+ if a := c.Expect.Answer; a != nil {+ checkIDs("expect.answer.checklist_item", a.ChecklistItem)+ checkIDs("expect.answer.any_of", a.AnyOf...)+ checkIDs("expect.answer.not", a.Not...)+ if c.Suite == "perf" && a.ChecklistItem == "" && len(a.AnyOf) == 0 {+ t.Error("perf answers need checklist_item or any_of")+ }+ }+ for arm, steps := range map[string][]catalog.Step{"oracle": c.Oracle} {+ for i, s := range steps {+ if s.Submit != nil {+ item, _ := s.Submit["checklist_item"].(string)+ checkIDs(fmt.Sprintf("%s[%d].submit", arm, i), item)+ }+ if s.WriteFile != nil && s.WriteFile.From != "" && !exists(filepath.Join(root, "testdata", s.WriteFile.From)) {+ t.Errorf("%s[%d] writes from %s, which does not exist", arm, i, s.WriteFile.From)+ }+ }+ }+ last := c.Oracle[len(c.Oracle)-1]+ if last.Submit == nil {+ t.Error("the oracle must end with submit")+ }+ if len(c.Judge.Rubric) == 0 {+ t.Error("perf cases need a judge rubric")+ }+ })+ }+}++// TestLeakage seeds every case into a dry sink and lints the strings it+// would send, plus the prompt, for terms that name the answer.+func TestLeakage(t *testing.T) {+ for _, c := range loadAll(t) {+ t.Run(c.ID, func(t *testing.T) {+ s := seed.NewScenario(c.ID, time.Date(2026, 10, 17, 12, 0, 0, 0, time.UTC), seed.NewDrySink())+ if err := scenarios.Run(c.Seed, s); err != nil {+ t.Fatal(err)+ }+ for _, f := range grade.Lint(c, s.Strings()) {+ t.Error(f)+ }+ if st := s.Stats(); st.Spans > 25000 {+ t.Errorf("seed sends %d spans, over the ~20k budget", st.Spans)+ }+ })+ }+}++func TestPresets(t *testing.T) {+ cases := loadAll(t)+ presets, err := catalog.LoadPresets(env.EvalsRoot())+ if err != nil {+ t.Fatal(err)+ }+ for name := range presets.Presets {+ units, err := catalog.Select(cases, presets, catalog.Selection{Preset: name})+ if err != nil {+ t.Errorf("preset %s: %v", name, err)+ continue+ }+ for _, u := range units {+ if name != "full" && !slices.Contains(presets.PrimaryArms[u.Case.Suite], u.Arm) && u.Arm != u.Case.Arms[0] {+ t.Errorf("preset %s picked non-primary arm %s for %s", name, u.Arm, u.Case.ID)+ }+ }+ }+ if _, err := catalog.LoadModels(env.EvalsRoot()); err != nil {+ t.Fatal(err)+ }+}++// TestChecklistIsComplete: the enum has every performance.md row.+func TestChecklistIsComplete(t *testing.T) {+ if n := len(checklist.Rows()); n != 26 {+ t.Fatalf("performance.md has %d checklist rows; update the perf catalog when rows change", n)+ }+}diff --git a/cli/test/evals/internal/tools/askuser.go b/cli/test/evals/internal/tools/askuser.gonew file mode 100644index 00000000..853a9464--- /dev/null+++ b/cli/test/evals/internal/tools/askuser.go@@ -0,0 +1,76 @@+package tools++import (+ "context"+ "encoding/json"+ "regexp"+ "sync"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/catalog"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// AskUser answers questions from the case's scripted user rules: the first+// rule whose regexp matches the question wins, a default rule matches+// anything, and with no rule the user says they have nothing to add.+type AskUser struct {+ Rules []catalog.UserRule++ mu sync.Mutex+ used map[int]bool+ Log []record.Question+}++func (t *AskUser) Spec() Spec {+ return Spec{Name: "ask_user", Description: "Ask the user a question and wait for the answer. Use it for decisions only the user can make (confirming a change, choosing between candidates, approving a draft).",+ Parameters: obj(map[string]any{"question": strProp("The question, including any draft or options the user must see.")}, "question")}+}++func (t *AskUser) Call(_ context.Context, raw json.RawMessage) Result {+ var in struct {+ Question string `json:"question"`+ }+ if err := decodeArgs(raw, &in); err != nil {+ return errResult("%v", err)+ }+ answer := t.Answer(in.Question)+ t.mu.Lock()+ t.Log = append(t.Log, record.Question{Question: in.Question, Answer: answer})+ t.mu.Unlock()+ return Result{Output: answer, OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpAskUser, Args: map[string]any{"question": in.Question}}, OK: true, Output: answer}}}+}++// Answer picks the scripted reply for a question.+func (t *AskUser) Answer(q string) string {+ t.mu.Lock()+ defer t.mu.Unlock()+ if t.used == nil {+ t.used = map[int]bool{}+ }+ for i, r := range t.Rules {+ if r.Once && t.used[i] {+ continue+ }+ if r.Match != "" {+ re, err := regexp.Compile("(?is)" + r.Match)+ if err != nil || !re.MatchString(q) {+ continue+ }+ t.used[i] = true+ return r.Reply+ }+ if r.Default != "" {+ t.used[i] = true+ return r.Default+ }+ }+ return "I don't have anything to add. Please continue."+}++// Questions returns the exchanges so far.+func (t *AskUser) Questions() []record.Question {+ t.mu.Lock()+ defer t.mu.Unlock()+ return append([]record.Question{}, t.Log...)+}diff --git a/cli/test/evals/internal/tools/coreutils.go b/cli/test/evals/internal/tools/coreutils.gonew file mode 100644index 00000000..c85b58d9--- /dev/null+++ b/cli/test/evals/internal/tools/coreutils.go@@ -0,0 +1,792 @@+package tools++import (+ "bufio"+ "bytes"+ "fmt"+ "io"+ "io/fs"+ "os"+ "path"+ "path/filepath"+ "regexp"+ "sort"+ "strconv"+ "strings"++ "mvdan.cc/sh/v3/interp"+)++// In-process text utilities. They implement the commonly used flags of their+// GNU namesakes and read files only through the sandbox, so `cat /etc/passwd`+// fails like any other path outside the workspace.++type uctx struct {+ hc interp.HandlerContext+ run *shellRun+}++func (u *uctx) errf(format string, a ...any) { fmt.Fprintf(u.hc.Stderr, format+"\n", a...) }++func (u *uctx) open(p string) (io.ReadCloser, error) {+ if p == "-" {+ if in := stdinOf(u.hc); in != nil {+ return io.NopCloser(in), nil+ }+ return io.NopCloser(strings.NewReader("")), nil+ }+ abs, err := u.run.s.Sandbox.Resolve(u.hc.Dir, p)+ if err != nil {+ return nil, err+ }+ return os.Open(abs)+}++// inputs concatenates the named files, or stdin when none are given.+func (u *uctx) inputs(files []string) ([]byte, int) {+ if len(files) == 0 {+ files = []string{"-"}+ }+ var buf bytes.Buffer+ code := 0+ for _, f := range files {+ rc, err := u.open(f)+ if err != nil {+ u.errf("%s: %v", f, err)+ code = 1+ continue+ }+ _, _ = io.Copy(&buf, rc)+ rc.Close()+ }+ return buf.Bytes(), code+}++func splitLines(b []byte) []string {+ s := string(b)+ if s == "" {+ return nil+ }+ s = strings.TrimSuffix(s, "\n")+ return strings.Split(s, "\n")+}++func (u *uctx) writeLines(lines []string) {+ w := bufio.NewWriter(u.hc.Stdout)+ for _, l := range lines {+ w.WriteString(l)+ w.WriteByte('\n')+ }+ w.Flush()+}++// coreutils are the in-process utilities the sandboxed shell offers;+// coreutils_more.go adds the rest at init.+var coreutils = map[string]func(u *uctx, args []string) int{+ "cat": uCat, "head": uHead, "tail": uTail, "wc": uWc, "grep": uGrep, "egrep": func(u *uctx, a []string) int { return uGrep(u, append([]string{"-E"}, a...)) },+ "sort": uSort, "uniq": uUniq, "cut": uCut, "tr": uTr, "ls": uLs, "echo": uEcho,+}++func coreutilNames() []string {+ var out []string+ for k := range coreutils {+ if k != "egrep" && k != "echo" {+ out = append(out, k)+ }+ }+ sort.Strings(out)+ return append(out, "xargs", "echo", "printf", "test")+}++// flags is a tiny getopt: single-dash letters (bundled), --long, and+// options that take a value.+type flags struct {+ set map[string]string+ args []string+}++func parseFlags(args []string, withValue string) flags {+ f := flags{set: map[string]string{}}+ for i := 0; i < len(args); i++ {+ a := args[i]+ if a == "--" {+ f.args = append(f.args, args[i+1:]...)+ break+ }+ if strings.HasPrefix(a, "--") && len(a) > 2 {+ k, v, ok := strings.Cut(a[2:], "=")+ if !ok {+ v = "true"+ }+ f.set["-"+k] = v+ continue+ }+ if strings.HasPrefix(a, "-") && len(a) > 1 && !isNumber(a) {+ for j := 1; j < len(a); j++ {+ c := string(a[j])+ if strings.Contains(withValue, c) {+ v := a[j+1:]+ if v == "" && i+1 < len(args) {+ i+++ v = args[i]+ }+ f.set[c] = v+ break+ }+ f.set[c] = "true"+ }+ continue+ }+ f.args = append(f.args, a)+ }+ return f+}++func (f flags) has(k string) bool { _, ok := f.set[k]; return ok }++func (f flags) int(k string, def int) int {+ if v, ok := f.set[k]; ok {+ n, err := strconv.Atoi(strings.TrimPrefix(v, "+"))+ if err == nil {+ return n+ }+ }+ return def+}++func isNumber(s string) bool {+ _, err := strconv.Atoi(s)+ return err == nil+}++func uCat(u *uctx, args []string) int {+ f := parseFlags(args, "")+ data, code := u.inputs(f.args)+ if f.has("n") {+ lines := splitLines(data)+ for i, l := range lines {+ lines[i] = fmt.Sprintf("%6d\t%s", i+1, l)+ }+ u.writeLines(lines)+ return code+ }+ u.hc.Stdout.Write(data)+ return code+}++// numericShorthand turns `head -20` into `head -n 20`.+func numericShorthand(args []string) []string {+ out := make([]string, 0, len(args))+ for _, a := range args {+ if len(a) > 1 && a[0] == '-' && isNumber(a[1:]) {+ out = append(out, "-n", a[1:])+ continue+ }+ out = append(out, a)+ }+ return out+}++func uHead(u *uctx, args []string) int {+ f := parseFlags(numericShorthand(args), "nc")+ data, code := u.inputs(f.args)+ if f.has("c") {+ n := f.int("c", 0)+ if n < len(data) {+ data = data[:n]+ }+ u.hc.Stdout.Write(data)+ return code+ }+ n := f.int("n", 10)+ lines := splitLines(data)+ if n < 0 {+ n = max(0, len(lines)+n)+ }+ if n < len(lines) {+ lines = lines[:n]+ }+ u.writeLines(lines)+ return code+}++func uTail(u *uctx, args []string) int {+ f := parseFlags(numericShorthand(args), "nc")+ data, code := u.inputs(f.args)+ lines := splitLines(data)+ v := f.set["n"]+ if strings.HasPrefix(v, "+") {+ from := f.int("n", 1)+ if from-1 < len(lines) {+ lines = lines[max(0, from-1):]+ } else {+ lines = nil+ }+ } else {+ n := f.int("n", 10)+ if n < len(lines) {+ lines = lines[len(lines)-n:]+ }+ }+ u.writeLines(lines)+ return code+}++func uWc(u *uctx, args []string) int {+ f := parseFlags(args, "")+ data, code := u.inputs(f.args)+ lines := bytes.Count(data, []byte("\n"))+ words := len(strings.Fields(string(data)))+ chars := len(data)+ var parts []string+ all := !f.has("l") && !f.has("w") && !f.has("c") && !f.has("m")+ if all || f.has("l") {+ parts = append(parts, strconv.Itoa(lines))+ }+ if all || f.has("w") {+ parts = append(parts, strconv.Itoa(words))+ }+ if all || f.has("c") || f.has("m") {+ parts = append(parts, strconv.Itoa(chars))+ }+ fmt.Fprintln(u.hc.Stdout, strings.Join(parts, " "))+ return code+}++// breToERE converts a POSIX basic regexp to Go syntax.+func breToERE(p string) string {+ var b strings.Builder+ for i := 0; i < len(p); i++ {+ c := p[i]+ if c == '\\' && i+1 < len(p) {+ n := p[i+1]+ switch n {+ case '(', ')', '|', '+', '?', '{', '}':+ b.WriteByte(n)+ default:+ b.WriteByte('\\')+ b.WriteByte(n)+ }+ i+++ continue+ }+ switch c {+ case '(', ')', '|', '+', '?', '{', '}':+ b.WriteByte('\\')+ }+ b.WriteByte(c)+ }+ return b.String()+}++func uGrep(u *uctx, args []string) int {+ f := parseFlags(args, "eABCm")+ var patterns []string+ if v, ok := f.set["e"]; ok {+ patterns = append(patterns, v)+ }+ rest := f.args+ if len(patterns) == 0 {+ if len(rest) == 0 {+ u.errf("grep: no pattern")+ return 2+ }+ patterns, rest = []string{rest[0]}, rest[1:]+ }+ var res []*regexp.Regexp+ for _, p := range patterns {+ switch {+ case f.has("F"):+ p = regexp.QuoteMeta(p)+ case f.has("E") || f.has("P"):+ default:+ p = breToERE(p)+ }+ if f.has("w") {+ p = `\b(?:` + p + `)\b`+ }+ if f.has("x") {+ p = `^(?:` + p + `)$`+ }+ if f.has("i") {+ p = "(?i)" + p+ }+ re, err := regexp.Compile(p)+ if err != nil {+ u.errf("grep: %v", err)+ return 2+ }+ res = append(res, re)+ }+ match := func(l string) bool {+ for _, re := range res {+ if re.MatchString(l) {+ return !f.has("v")+ }+ }+ return f.has("v")+ }+ files := rest+ if f.has("r") || f.has("R") {+ if len(files) == 0 {+ files = []string{"."}+ }+ var expanded []string+ for _, root := range files {+ abs, err := u.run.s.Sandbox.Resolve(u.hc.Dir, root)+ if err != nil {+ u.errf("grep: %v", err)+ continue+ }+ _ = filepath.WalkDir(abs, func(p string, d fs.DirEntry, err error) error {+ if err != nil {+ return nil+ }+ if d.IsDir() && (d.Name() == ".git" || d.Name() == "node_modules") {+ return filepath.SkipDir+ }+ if !d.IsDir() {+ rel, _ := filepath.Rel(u.hc.Dir, p)+ expanded = append(expanded, rel)+ }+ return nil+ })+ }+ files = expanded+ }+ multi := len(files) > 1 || f.has("r") || f.has("R")+ if f.has("h") {+ multi = false+ }+ if f.has("H") {+ multi = true+ }+ if len(files) == 0 {+ files = []string{"-"}+ }+ after, before := f.int("A", 0), f.int("B", 0)+ if c := f.int("C", 0); c > 0 {+ after, before = c, c+ }+ maxCount := f.int("m", -1)+ found := false+ w := bufio.NewWriter(u.hc.Stdout)+ defer w.Flush()+ for _, file := range files {+ rc, err := u.open(file)+ if err != nil {+ u.errf("grep: %s: %v", file, err)+ continue+ }+ data, _ := io.ReadAll(rc)+ rc.Close()+ lines := splitLines(data)+ count := 0+ lastPrinted := -1+ prefix := func(i int, sep string) string {+ p := ""+ if multi {+ p = file + sep+ }+ if f.has("n") {+ p += strconv.Itoa(i+1) + sep+ }+ return p+ }+ for i, l := range lines {+ if !match(l) {+ continue+ }+ found = true+ count+++ if f.has("q") {+ return 0+ }+ if f.has("l") {+ fmt.Fprintln(w, file)+ break+ }+ if f.has("c") {+ if maxCount >= 0 && count >= maxCount {+ break+ }+ continue+ }+ if f.has("o") && !f.has("v") {+ for _, re := range res {+ for _, m := range re.FindAllString(l, -1) {+ fmt.Fprintln(w, prefix(i, ":")+m)+ }+ }+ } else {+ for b := max(lastPrinted+1, i-before); b < i; b++ {+ fmt.Fprintln(w, prefix(b, "-")+lines[b])+ }+ fmt.Fprintln(w, prefix(i, ":")+l)+ lastPrinted = i+ for a := i + 1; a <= i+after && a < len(lines); a++ {+ if !match(lines[a]) {+ fmt.Fprintln(w, prefix(a, "-")+lines[a])+ lastPrinted = a+ }+ }+ }+ if maxCount >= 0 && count >= maxCount {+ break+ }+ }+ if f.has("c") {+ if multi {+ fmt.Fprintf(w, "%s:%d\n", file, count)+ } else {+ fmt.Fprintln(w, count)+ }+ }+ }+ if !found {+ return 1+ }+ return 0+}++func uSort(u *uctx, args []string) int {+ f := parseFlags(args, "kt")+ data, code := u.inputs(f.args)+ lines := splitLines(data)+ sep := f.set["t"]+ keyStart, keyEnd := 0, 0+ if k, ok := f.set["k"]; ok {+ a, b, _ := strings.Cut(k, ",")+ keyStart, _ = strconv.Atoi(strings.TrimRight(a, "nrh"))+ keyEnd, _ = strconv.Atoi(strings.TrimRight(b, "nrh"))+ if strings.ContainsAny(a+b, "n") {+ f.set["n"] = "true"+ }+ if strings.ContainsAny(a+b, "r") {+ f.set["r"] = "true"+ }+ }+ key := func(l string) string {+ if keyStart == 0 {+ return l+ }+ var fields []string+ if sep != "" {+ fields = strings.Split(l, sep)+ } else {+ fields = strings.Fields(l)+ }+ if keyStart-1 >= len(fields) {+ return ""+ }+ end := len(fields)+ if keyEnd > 0 && keyEnd < end {+ end = keyEnd+ }+ j := " "+ if sep != "" {+ j = sep+ }+ return strings.Join(fields[keyStart-1:end], j)+ }+ numeric := f.has("n") || f.has("g") || f.has("h")+ less := func(a, b string) bool {+ ka, kb := key(a), key(b)+ if numeric {+ na, _ := strconv.ParseFloat(strings.TrimSpace(firstNumber(ka)), 64)+ nb, _ := strconv.ParseFloat(strings.TrimSpace(firstNumber(kb)), 64)+ if na != nb {+ return na < nb+ }+ return ka < kb+ }+ if f.has("f") {+ return strings.ToLower(ka) < strings.ToLower(kb)+ }+ return ka < kb+ }+ sort.SliceStable(lines, func(i, j int) bool {+ if f.has("r") {+ return less(lines[j], lines[i])+ }+ return less(lines[i], lines[j])+ })+ if f.has("u") {+ var out []string+ for i, l := range lines {+ if i == 0 || key(l) != key(lines[i-1]) {+ out = append(out, l)+ }+ }+ lines = out+ }+ u.writeLines(lines)+ return code+}++var numRe = regexp.MustCompile(`-?[0-9]+(\.[0-9]+)?([eE][-+]?[0-9]+)?`)++func firstNumber(s string) string {+ if m := numRe.FindString(s); m != "" {+ return m+ }+ return "0"+}++func uUniq(u *uctx, args []string) int {+ f := parseFlags(args, "")+ data, code := u.inputs(f.args)+ lines := splitLines(data)+ type run struct {+ line string+ n int+ }+ var runs []run+ for _, l := range lines {+ cmp := l+ if f.has("i") {+ cmp = strings.ToLower(l)+ }+ if len(runs) > 0 {+ last := runs[len(runs)-1].line+ if f.has("i") {+ last = strings.ToLower(last)+ }+ if last == cmp {+ runs[len(runs)-1].n+++ continue+ }+ }+ runs = append(runs, run{l, 1})+ }+ var out []string+ for _, r := range runs {+ if f.has("d") && r.n < 2 || f.has("u") && r.n > 1 {+ continue+ }+ if f.has("c") {+ out = append(out, fmt.Sprintf("%7d %s", r.n, r.line))+ } else {+ out = append(out, r.line)+ }+ }+ u.writeLines(out)+ return code+}++func parseRanges(spec string) [][2]int {+ var out [][2]int+ for _, part := range strings.Split(spec, ",") {+ a, b, isRange := strings.Cut(part, "-")+ lo, _ := strconv.Atoi(a)+ hi := lo+ if isRange {+ if b == "" {+ hi = 1 << 30+ } else {+ hi, _ = strconv.Atoi(b)+ }+ if a == "" {+ lo = 1+ }+ }+ out = append(out, [2]int{lo, hi})+ }+ return out+}++func inRanges(i int, rs [][2]int) bool {+ for _, r := range rs {+ if i >= r[0] && i <= r[1] {+ return true+ }+ }+ return false+}++func uCut(u *uctx, args []string) int {+ f := parseFlags(args, "dfc")+ data, code := u.inputs(f.args)+ var out []string+ for _, l := range splitLines(data) {+ if c, ok := f.set["c"]; ok {+ rs := parseRanges(c)+ var b strings.Builder+ for i, r := range []rune(l) {+ if inRanges(i+1, rs) {+ b.WriteRune(r)+ }+ }+ out = append(out, b.String())+ continue+ }+ d := "\t"+ if v, ok := f.set["d"]; ok {+ d = v+ }+ fields := strings.Split(l, d)+ rs := parseRanges(f.set["f"])+ var keep []string+ for i, x := range fields {+ if inRanges(i+1, rs) {+ keep = append(keep, x)+ }+ }+ out = append(out, strings.Join(keep, d))+ }+ u.writeLines(out)+ return code+}++func expandSet(s string) []rune {+ s = strings.NewReplacer(`\n`, "\n", `\t`, "\t", `\\`, `\`,+ "[:upper:]", "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "[:lower:]", "abcdefghijklmnopqrstuvwxyz",+ "[:digit:]", "0123456789", "[:space:]", " \t\n\r\f\v", "[:punct:]", "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~").Replace(s)+ rs := []rune(s)+ var out []rune+ for i := 0; i < len(rs); i++ {+ if i+2 < len(rs) && rs[i+1] == '-' {+ for c := rs[i]; c <= rs[i+2]; c++ {+ out = append(out, c)+ }+ i += 2+ continue+ }+ out = append(out, rs[i])+ }+ return out+}++func uTr(u *uctx, args []string) int {+ f := parseFlags(args, "")+ data, _ := u.inputs(nil)+ in := []rune(string(data))+ if len(f.args) == 0 {+ u.errf("tr: missing operand")+ return 1+ }+ set1 := expandSet(f.args[0])+ contains := func(set []rune, r rune) int {+ for i, x := range set {+ if x == r {+ return i+ }+ }+ return -1+ }+ var out []rune+ switch {+ case f.has("d"):+ for _, r := range in {+ if contains(set1, r) < 0 {+ out = append(out, r)+ }+ }+ case len(f.args) >= 2:+ set2 := expandSet(f.args[1])+ for _, r := range in {+ if i := contains(set1, r); i >= 0 && len(set2) > 0 {+ out = append(out, set2[min(i, len(set2)-1)])+ } else {+ out = append(out, r)+ }+ }+ default:+ out = in+ }+ if f.has("s") {+ sq := set1+ if len(f.args) >= 2 && !f.has("d") {+ sq = expandSet(f.args[1])+ }+ var squeezed []rune+ for i, r := range out {+ if i > 0 && r == out[i-1] && contains(sq, r) >= 0 {+ continue+ }+ squeezed = append(squeezed, r)+ }+ out = squeezed+ }+ io.WriteString(u.hc.Stdout, string(out))+ return 0+}++func uLs(u *uctx, args []string) int {+ f := parseFlags(args, "")+ targets := f.args+ if len(targets) == 0 {+ targets = []string{"."}+ }+ code := 0+ for _, t := range targets {+ abs, err := u.run.s.Sandbox.Resolve(u.hc.Dir, t)+ if err != nil {+ u.errf("ls: %v", err)+ code = 2+ continue+ }+ info, err := os.Stat(abs)+ if err != nil {+ u.errf("ls: %s: no such file or directory", t)+ code = 2+ continue+ }+ if !info.IsDir() {+ fmt.Fprintln(u.hc.Stdout, t)+ continue+ }+ var walk func(dir, label string)+ walk = func(dir, label string) {+ entries, _ := os.ReadDir(dir)+ if f.has("R") || len(targets) > 1 {+ fmt.Fprintf(u.hc.Stdout, "%s:\n", label)+ }+ var subdirs []string+ for _, e := range entries {+ if !f.has("a") && !f.has("A") && strings.HasPrefix(e.Name(), ".") {+ continue+ }+ name := e.Name()+ if e.IsDir() {+ subdirs = append(subdirs, name)+ }+ if f.has("l") {+ fi, _ := e.Info()+ mode, size := "-rw-r--r--", int64(0)+ if fi != nil {+ mode, size = fi.Mode().String(), fi.Size()+ }+ fmt.Fprintf(u.hc.Stdout, "%s %8d %s\n", mode, size, name)+ } else {+ if e.IsDir() && f.has("p") {+ name += "/"+ }+ fmt.Fprintln(u.hc.Stdout, name)+ }+ }+ if f.has("R") {+ for _, d := range subdirs {+ fmt.Fprintln(u.hc.Stdout)+ walk(filepath.Join(dir, d), path.Join(label, d))+ }+ }+ }+ walk(abs, t)+ }+ return code+}++// uEcho is only reached through xargs; the interpreter's builtin echo+// handles every direct use.+func uEcho(u *uctx, args []string) int {+ nl := true+ if len(args) > 0 && args[0] == "-n" {+ nl, args = false, args[1:]+ }+ io.WriteString(u.hc.Stdout, strings.Join(args, " "))+ if nl {+ io.WriteString(u.hc.Stdout, "\n")+ }+ return 0+}diff --git a/cli/test/evals/internal/tools/files.go b/cli/test/evals/internal/tools/files.gonew file mode 100644index 00000000..58e85d88--- /dev/null+++ b/cli/test/evals/internal/tools/files.go@@ -0,0 +1,304 @@+package tools++import (+ "context"+ "encoding/json"+ "fmt"+ "io/fs"+ "os"+ "path"+ "path/filepath"+ "regexp"+ "sort"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// File tools, confined to the workspace. .skills/ is readable but not+// writable.++func fileOp(name, rel string, ok bool) []record.OpRecord {+ return []record.OpRecord{{Op: ops.Op{Op: name, Args: map[string]any{"path": rel}}, OK: ok}}+}++// ReadFile reads a file with line numbers.+type ReadFile struct{ Sandbox *Sandbox }++func (t *ReadFile) Spec() Spec {+ return Spec{Name: "read_file", Description: "Read a text file from the workspace. Lines are numbered. Use offset/limit (1-based lines) for long files.",+ Parameters: obj(map[string]any{+ "path": strProp("Workspace-relative path."),+ "offset": intProp("First line to return (1-based)."),+ "limit": intProp("Maximum number of lines (default 2000)."),+ }, "path")}+}++func (t *ReadFile) Call(_ context.Context, raw json.RawMessage) Result {+ var in struct {+ Path string `json:"path"`+ Offset int `json:"offset"`+ Limit int `json:"limit"`+ }+ if err := decodeArgs(raw, &in); err != nil {+ return errResult("%v", err)+ }+ abs, err := t.Sandbox.Resolve("", in.Path)+ if err != nil {+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpReadFile, in.Path, false)}+ }+ rel := t.Sandbox.Rel(abs)+ info, err := os.Stat(abs)+ if err != nil {+ return Result{Output: fmt.Sprintf("error: %s: no such file", in.Path), Ops: fileOp(ops.OpReadFile, rel, false)}+ }+ if info.IsDir() {+ return Result{Output: fmt.Sprintf("error: %s is a directory; use list_files", in.Path), Ops: fileOp(ops.OpReadFile, rel, false)}+ }+ text, err := readFileCapped(abs)+ if err != nil {+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpReadFile, rel, false)}+ }+ lines := strings.Split(strings.TrimSuffix(text, "\n"), "\n")+ start := max(1, in.Offset)+ limit := in.Limit+ if limit <= 0 {+ limit = 2000+ }+ var b strings.Builder+ for i := start; i <= len(lines) && i < start+limit; i++ {+ fmt.Fprintf(&b, "%6d\t%s\n", i, lines[i-1])+ }+ if start+limit <= len(lines) {+ fmt.Fprintf(&b, "[%d more lines; continue with offset %d]\n", len(lines)-(start+limit)+1, start+limit)+ }+ out, trunc := Cap(b.String())+ return Result{Output: out, OK: true, Truncated: trunc, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpReadFile, Args: map[string]any{"path": rel}}, OK: true, Output: text}}}+}++// ListFiles lists files under a directory.+type ListFiles struct{ Sandbox *Sandbox }++func (t *ListFiles) Spec() Spec {+ return Spec{Name: "list_files", Description: "List files under a workspace directory (recursive, .git skipped). Optional glob filters file names, e.g. *.go.",+ Parameters: obj(map[string]any{+ "path": strProp("Directory, default the workspace root."),+ "glob": strProp("Optional file-name glob."),+ })}+}++func (t *ListFiles) Call(_ context.Context, raw json.RawMessage) Result {+ var in struct {+ Path string `json:"path"`+ Glob string `json:"glob"`+ }+ if err := decodeArgs(raw, &in); err != nil {+ return errResult("%v", err)+ }+ if in.Path == "" {+ in.Path = "."+ }+ abs, err := t.Sandbox.Resolve("", in.Path)+ if err != nil {+ return errResult("%v", err)+ }+ var files []string+ _ = filepath.WalkDir(abs, func(p string, d fs.DirEntry, err error) error {+ if err != nil {+ return nil+ }+ if d.IsDir() && (d.Name() == ".git" || d.Name() == "node_modules" || d.Name() == "__pycache__") {+ return filepath.SkipDir+ }+ if d.IsDir() {+ return nil+ }+ if in.Glob != "" {+ if ok, _ := path.Match(in.Glob, d.Name()); !ok {+ return nil+ }+ }+ files = append(files, t.Sandbox.Rel(p))+ return nil+ })+ sort.Strings(files)+ out, trunc := Cap(strings.Join(files, "\n"))+ return Result{Output: out, OK: true, Truncated: trunc, Ops: fileOp(ops.OpListFiles, t.Sandbox.Rel(abs), true)}+}++// Search greps the workspace with a regular expression.+type Search struct{ Sandbox *Sandbox }++func (t *Search) Spec() Spec {+ return Spec{Name: "search", Description: "Search workspace files for a regular expression (RE2 syntax). Returns path:line: text.",+ Parameters: obj(map[string]any{+ "pattern": strProp("Regular expression."),+ "path": strProp("Directory or file to search, default the workspace root."),+ "glob": strProp("Optional file-name glob, e.g. *.py."),+ }, "pattern")}+}++func (t *Search) Call(_ context.Context, raw json.RawMessage) Result {+ var in struct {+ Pattern string `json:"pattern"`+ Path string `json:"path"`+ Glob string `json:"glob"`+ }+ if err := decodeArgs(raw, &in); err != nil {+ return errResult("%v", err)+ }+ re, err := regexp.Compile(in.Pattern)+ if err != nil {+ return errResult("invalid pattern: %v", err)+ }+ if in.Path == "" {+ in.Path = "."+ }+ abs, err := t.Sandbox.Resolve("", in.Path)+ if err != nil {+ return errResult("%v", err)+ }+ var b strings.Builder+ hits := 0+ _ = filepath.WalkDir(abs, func(p string, d fs.DirEntry, err error) error {+ if err != nil || hits > 500 {+ return nil+ }+ if d.IsDir() && (d.Name() == ".git" || d.Name() == "node_modules") {+ return filepath.SkipDir+ }+ if d.IsDir() {+ return nil+ }+ if in.Glob != "" {+ if ok, _ := path.Match(in.Glob, d.Name()); !ok {+ return nil+ }+ }+ data, err := os.ReadFile(p)+ if err != nil || isBinary(data) {+ return nil+ }+ for i, l := range strings.Split(string(data), "\n") {+ if re.MatchString(l) {+ fmt.Fprintf(&b, "%s:%d: %s\n", t.Sandbox.Rel(p), i+1, l)+ hits+++ }+ }+ return nil+ })+ if hits == 0 {+ b.WriteString("no matches")+ }+ out, trunc := Cap(b.String())+ return Result{Output: out, OK: true, Truncated: trunc, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpSearch, Args: map[string]any{"pattern": in.Pattern, "path": t.Sandbox.Rel(abs)}}, OK: true}}}+}++func isBinary(b []byte) bool {+ n := min(len(b), 8000)+ for _, c := range b[:n] {+ if c == 0 {+ return true+ }+ }+ return false+}++// EditFile replaces an exact string in a file.+type EditFile struct{ Sandbox *Sandbox }++func (t *EditFile) Spec() Spec {+ return Spec{Name: "edit_file", Description: "Replace an exact string in a workspace file. old_string must match exactly once unless replace_all is true.",+ Parameters: obj(map[string]any{+ "path": strProp("Workspace-relative path."),+ "old_string": strProp("Exact text to replace."),+ "new_string": strProp("Replacement text."),+ "replace_all": map[string]any{"type": "boolean", "description": "Replace every occurrence."},+ }, "path", "old_string", "new_string")}+}++func (t *EditFile) Call(_ context.Context, raw json.RawMessage) Result {+ var in struct {+ Path string `json:"path"`+ Old string `json:"old_string"`+ New string `json:"new_string"`+ ReplaceAll bool `json:"replace_all"`+ }+ if err := decodeArgs(raw, &in); err != nil {+ return errResult("%v", err)+ }+ abs, err := t.Sandbox.Resolve("", in.Path)+ if err == nil {+ err = t.Sandbox.Writable(abs)+ }+ if err != nil {+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpEditFile, in.Path, false)}+ }+ rel := t.Sandbox.Rel(abs)+ data, err := os.ReadFile(abs)+ if err != nil {+ return Result{Output: fmt.Sprintf("error: %s: no such file", in.Path), Ops: fileOp(ops.OpEditFile, rel, false)}+ }+ text := string(data)+ n := strings.Count(text, in.Old)+ switch {+ case in.Old == "":+ return Result{Output: "error: old_string is empty", Ops: fileOp(ops.OpEditFile, rel, false)}+ case n == 0:+ return Result{Output: "error: old_string not found in " + rel, Ops: fileOp(ops.OpEditFile, rel, false)}+ case n > 1 && !in.ReplaceAll:+ return Result{Output: fmt.Sprintf("error: old_string matches %d times in %s; add context or set replace_all", n, rel), Ops: fileOp(ops.OpEditFile, rel, false)}+ }+ if in.ReplaceAll {+ text = strings.ReplaceAll(text, in.Old, in.New)+ } else {+ text = strings.Replace(text, in.Old, in.New, 1)+ }+ if err := os.WriteFile(abs, []byte(text), 0o644); err != nil {+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpEditFile, rel, false)}+ }+ return Result{Output: fmt.Sprintf("edited %s (%d replacement(s))", rel, max(1, n*boolToInt(in.ReplaceAll))), OK: true, Ops: fileOp(ops.OpEditFile, rel, true)}+}++func boolToInt(b bool) int {+ if b {+ return 1+ }+ return 0+}++// WriteFile writes a whole file.+type WriteFile struct{ Sandbox *Sandbox }++func (t *WriteFile) Spec() Spec {+ return Spec{Name: "write_file", Description: "Create or overwrite a workspace file with the given content.",+ Parameters: obj(map[string]any{+ "path": strProp("Workspace-relative path."),+ "content": strProp("Full file content."),+ }, "path", "content")}+}++func (t *WriteFile) Call(_ context.Context, raw json.RawMessage) Result {+ var in struct {+ Path string `json:"path"`+ Content string `json:"content"`+ }+ if err := decodeArgs(raw, &in); err != nil {+ return errResult("%v", err)+ }+ abs, err := t.Sandbox.Resolve("", in.Path)+ if err == nil {+ err = t.Sandbox.Writable(abs)+ }+ if err != nil {+ return Result{Output: "error: " + err.Error(), Ops: fileOp(ops.OpWriteFile, in.Path, false)}+ }+ if err := os.MkdirAll(filepath.Dir(abs), 0o755); err != nil {+ return errResult("%v", err)+ }+ if err := os.WriteFile(abs, []byte(in.Content), 0o644); err != nil {+ return errResult("%v", err)+ }+ return Result{Output: fmt.Sprintf("wrote %s (%d bytes)", t.Sandbox.Rel(abs), len(in.Content)), OK: true, Ops: fileOp(ops.OpWriteFile, t.Sandbox.Rel(abs), true)}+}diff --git a/cli/test/evals/internal/tools/jqawk.go b/cli/test/evals/internal/tools/jqawk.gonew file mode 100644index 00000000..0b443a86--- /dev/null+++ b/cli/test/evals/internal/tools/jqawk.go@@ -0,0 +1,303 @@+package tools++import (+ "bytes"+ "encoding/json"+ "errors"+ "fmt"+ "io"+ "strings"++ awkinterp "github.com/benhoyt/goawk/interp"+ awkparser "github.com/benhoyt/goawk/parser"+ "github.com/itchyny/gojq"+ "mvdan.cc/sh/v3/interp"+)++// runJQ implements jq with gojq, in-process.+func runJQ(hc interp.HandlerContext, r *shellRun, args []string) int {+ u := &uctx{hc: hc, run: r}+ raw, compact, slurp, nullIn, exitStatus, sortKeys, rawIn, join := false, false, false, false, false, false, false, false+ indent := 2+ var filter string+ haveFilter := false+ var files []string+ var varNames []string+ var varValues []any+ for i := 0; i < len(args); i++ {+ a := args[i]+ switch {+ case a == "--arg" && i+2 < len(args):+ varNames = append(varNames, "$"+args[i+1])+ varValues = append(varValues, args[i+2])+ i += 2+ case a == "--argjson" && i+2 < len(args):+ var v any+ if err := json.Unmarshal([]byte(args[i+2]), &v); err != nil {+ u.errf("jq: --argjson %s: invalid JSON", args[i+1])+ return 2+ }+ varNames = append(varNames, "$"+args[i+1])+ varValues = append(varValues, v)+ i += 2+ case a == "--tab":+ indent = -1+ case a == "--indent" && i+1 < len(args):+ fmt.Sscanf(args[i+1], "%d", &indent)+ i+++ case strings.HasPrefix(a, "--"):+ switch a {+ case "--raw-output":+ raw = true+ case "--compact-output":+ compact = true+ case "--slurp":+ slurp = true+ case "--null-input":+ nullIn = true+ case "--exit-status":+ exitStatus = true+ case "--sort-keys":+ sortKeys = true+ case "--raw-input":+ rawIn = true+ case "--join-output":+ raw, join = true, true+ case "--monochrome-output", "--color-output":+ default:+ u.errf("jq: unsupported option %s", a)+ return 2+ }+ case strings.HasPrefix(a, "-") && len(a) > 1 && !haveFilter:+ for _, c := range a[1:] {+ switch c {+ case 'r':+ raw = true+ case 'j':+ raw, join = true, true+ case 'c':+ compact = true+ case 's':+ slurp = true+ case 'n':+ nullIn = true+ case 'e':+ exitStatus = true+ case 'S':+ sortKeys = true+ case 'R':+ rawIn = true+ case 'M', 'C', 'a':+ default:+ u.errf("jq: unsupported option -%c", c)+ return 2+ }+ }+ default:+ if !haveFilter {+ filter, haveFilter = a, true+ } else {+ files = append(files, a)+ }+ }+ }+ if !haveFilter {+ filter = "."+ }+ _ = sortKeys+ query, err := gojq.Parse(filter)+ if err != nil {+ u.errf("jq: error: %v", err)+ return 3+ }+ code, err := gojq.Compile(query, gojq.WithVariables(varNames), gojq.WithEnvironLoader(func() []string { return nil }))+ if err != nil {+ u.errf("jq: error: %v", err)+ return 3+ }+ data, rc := u.inputs(files)+ if rc != 0 {+ return 2+ }+ var inputs []any+ switch {+ case nullIn:+ inputs = []any{nil}+ case rawIn:+ lines := splitLines(data)+ if slurp {+ inputs = []any{string(data)}+ } else {+ for _, l := range lines {+ inputs = append(inputs, l)+ }+ }+ default:+ dec := json.NewDecoder(bytes.NewReader(data))+ var all []any+ for {+ var v any+ if err := dec.Decode(&v); err != nil {+ if errors.Is(err, io.EOF) {+ break+ }+ u.errf("jq: error (at <stdin>): %v", err)+ return 2+ }+ all = append(all, normalizeJSON(v))+ }+ if slurp {+ if all == nil {+ all = []any{}+ }+ inputs = []any{all}+ } else {+ inputs = all+ }+ }+ var out bytes.Buffer+ last := any(nil)+ produced := false+ status := 0+ for _, in := range inputs {+ iter := code.Run(in, varValues...)+ for {+ v, ok := iter.Next()+ if !ok {+ break+ }+ if err, ok := v.(error); ok {+ var halt *gojq.HaltError+ if errors.As(err, &halt) {+ if halt.Value() != nil {+ u.errf("jq: error: %v", halt.Value())+ }+ hc.Stdout.Write(out.Bytes())+ return halt.ExitCode()+ }+ u.errf("jq: error: %v", err)+ status = 5+ break+ }+ produced, last = true, v+ if s, ok := v.(string); ok && raw {+ out.WriteString(s)+ } else {+ out.Write(encodeJSON(v, compact, indent))+ }+ if !join {+ out.WriteByte('\n')+ }+ }+ }+ hc.Stdout.Write(out.Bytes())+ if status != 0 {+ return status+ }+ if exitStatus && (!produced || last == nil || last == false) {+ return 1+ }+ return 0+}++// normalizeJSON converts encoding/json values into the types gojq expects.+func normalizeJSON(v any) any {+ switch t := v.(type) {+ case map[string]any:+ for k, x := range t {+ t[k] = normalizeJSON(x)+ }+ return t+ case []any:+ for i, x := range t {+ t[i] = normalizeJSON(x)+ }+ return t+ case float64:+ if t == float64(int64(t)) && t < 1<<53 && t > -(1<<53) {+ return int(t)+ }+ return t+ default:+ return v+ }+}++func encodeJSON(v any, compact bool, indent int) []byte {+ var buf bytes.Buffer+ enc := json.NewEncoder(&buf)+ enc.SetEscapeHTML(false)+ if !compact {+ if indent < 0 {+ enc.SetIndent("", "\t")+ } else if indent > 0 {+ enc.SetIndent("", strings.Repeat(" ", indent))+ }+ }+ if err := enc.Encode(v); err != nil {+ return []byte(fmt.Sprint(v))+ }+ return bytes.TrimSuffix(buf.Bytes(), []byte("\n"))+}++// runAWK implements awk with goawk: no exec, no file writes; file operands+// are read through the sandbox.+func runAWK(hc interp.HandlerContext, r *shellRun, args []string) int {+ u := &uctx{hc: hc, run: r}+ var vars []string+ var src string+ haveSrc := false+ var files []string+ for i := 0; i < len(args); i++ {+ a := args[i]+ switch {+ case a == "-F" && i+1 < len(args):+ vars = append(vars, "FS", unescapeAwk(args[i+1]))+ i+++ case strings.HasPrefix(a, "-F") && len(a) > 2:+ vars = append(vars, "FS", unescapeAwk(a[2:]))+ case a == "-v" && i+1 < len(args):+ k, v, _ := strings.Cut(args[i+1], "=")+ vars = append(vars, k, v)+ i+++ case a == "-f":+ u.errf("awk: -f is not supported; pass the program inline")+ return 2+ default:+ if !haveSrc {+ src, haveSrc = a, true+ } else {+ files = append(files, a)+ }+ }+ }+ if !haveSrc {+ u.errf("awk: no program")+ return 2+ }+ prog, err := awkparser.ParseProgram([]byte(src), nil)+ if err != nil {+ u.errf("awk: %v", err)+ return 2+ }+ data, rc := u.inputs(files)+ if rc != 0 {+ return 2+ }+ status, err := awkinterp.ExecProgram(prog, &awkinterp.Config{+ Stdin: bytes.NewReader(data), Output: hc.Stdout, Error: hc.Stderr,+ Vars: vars, NoExec: true, NoFileWrites: true, NoFileReads: true, Environ: []string{},+ })+ if err != nil {+ u.errf("awk: %v", err)+ return 2+ }+ return status+}++func unescapeAwk(s string) string {+ if s == "\\t" || s == "t" {+ return "\t"+ }+ return s+}diff --git a/cli/test/evals/internal/tools/shell.go b/cli/test/evals/internal/tools/shell.gonew file mode 100644index 00000000..867b1886--- /dev/null+++ b/cli/test/evals/internal/tools/shell.go@@ -0,0 +1,420 @@+package tools++import (+ "bytes"+ "context"+ "encoding/json"+ "errors"+ "fmt"+ "io"+ "io/fs"+ "os"+ "os/exec"+ "path/filepath"+ "slices"+ "sort"+ "strings"+ "sync"+ "time"++ "mvdan.cc/sh/v3/expand"+ "mvdan.cc/sh/v3/interp"+ "mvdan.cc/sh/v3/syntax"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// Shell runs bash-syntax command lines in an interpreter (mvdan.cc/sh), not+// the host shell. Only allowlisted commands exist: the arm's traceway binary,+// jq and awk (in-process), read-only text utilities (in-process, confined to+// the workspace), and read-only git. Each traceway invocation inside a+// pipeline is recorded as its own operation.+type Shell struct {+ Sandbox *Sandbox+ // Bins maps an allowed command name to the executable it runs+ // (traceway or traceway-write).+ Bins map[string]string+ // BinEnv is the environment of those executables (scrubbed: HOME, XDG,+ // PATH, TZ, LANG only).+ BinEnv []string+ // Git enables read-only git subcommands in the workspace.+ Git bool+ // Timeout bounds one command line.+ Timeout time.Duration+ // Denied names commands that exist on other arms, to explain refusals.+ Denied map[string]string+}++// Spec describes the tool.+func (s *Shell) Spec() Spec {+ var names []string+ for n := range s.Bins {+ names = append(names, n)+ }+ sort.Strings(names)+ allowed := append(names, "jq", "awk")+ allowed = append(allowed, coreutilNames()...)+ if s.Git {+ allowed = append(allowed, "git (log, show, diff, status, blame, ls-files, rev-parse)")+ }+ return Spec{+ Name: "shell",+ Description: "Run a bash command line in the repository workspace (pipes, redirects, $(...), && and here-docs work). " ++ "Available commands: " + strings.Join(allowed, ", ") + ". " ++ "There is no network access other than through the commands above, and no interpreters or package managers. " ++ "Output is capped at 30000 characters.",+ Parameters: obj(map[string]any{+ "command": strProp("The command line to run."),+ }, "command"),+ }+}++type shellRun struct {+ s *Shell+ mu sync.Mutex+ ops []record.OpRecord+ ctx context.Context+ runner *interp.Runner+}++// Call runs one command line.+func (s *Shell) Call(ctx context.Context, raw json.RawMessage) Result {+ var in struct {+ Command string `json:"command"`+ }+ if err := decodeArgs(raw, &in); err != nil {+ return errResult("%v", err)+ }+ if strings.TrimSpace(in.Command) == "" {+ return errResult("command is empty")+ }+ file, err := syntax.NewParser(syntax.Variant(syntax.LangBash)).Parse(strings.NewReader(in.Command), "")+ if err != nil {+ return errResult("syntax error: %v", err)+ }+ timeout := s.Timeout+ if timeout == 0 {+ timeout = 2 * time.Minute+ }+ ctx, cancel := context.WithTimeout(ctx, timeout)+ defer cancel()++ var out lockedBuffer+ run := &shellRun{s: s, ctx: ctx}+ env := []string{"HOME=" + s.Sandbox.Root, "PATH=/usr/bin:/bin", "TZ=UTC", "LANG=C.UTF-8", "PWD=" + s.Sandbox.Root}+ runner, err := interp.New(+ interp.StdIO(nil, &out, &out),+ interp.Dir(s.Sandbox.Root),+ interp.Env(expand.ListEnviron(env...)),+ interp.ExecHandlers(func(next interp.ExecHandlerFunc) interp.ExecHandlerFunc { return run.exec }),+ interp.OpenHandler(run.open),+ interp.ReadDirHandler2(run.readDir),+ interp.StatHandler(run.stat),+ )+ if err != nil {+ return errResult("interpreter: %v", err)+ }+ run.runner = runner+ runErr := runner.Run(ctx, file)+ exit := 0+ if runErr != nil {+ if st, ok := interp.IsExitStatus(runErr); ok {+ exit = int(st)+ } else if errors.Is(ctx.Err(), context.DeadlineExceeded) {+ out.WriteString(fmt.Sprintf("\n[command timed out after %s]", timeout))+ exit = 124+ } else {+ out.WriteString("\n" + runErr.Error())+ exit = 1+ }+ }+ text := out.String()+ if exit != 0 {+ text += fmt.Sprintf("\n[exit status %d]", exit)+ }+ capped, trunc := Cap(text)+ return Result{Output: capped, OK: exit == 0, Truncated: trunc, Ops: run.ops}+}++func (r *shellRun) record(op record.OpRecord) {+ r.mu.Lock()+ defer r.mu.Unlock()+ r.ops = append(r.ops, op)+}++func (r *shellRun) exec(ctx context.Context, args []string) error {+ hc := interp.HandlerCtx(ctx)+ name := args[0]+ if strings.Contains(name, "/") {+ name = filepath.Base(name)+ }+ if bin, ok := r.s.Bins[name]; ok {+ return r.execTraceway(ctx, hc, bin, args)+ }+ if name == "git" && r.s.Git {+ return r.execGit(ctx, hc, args)+ }+ if name == "jq" {+ return exitStatus(runJQ(hc, r, args[1:]))+ }+ if name == "awk" {+ return exitStatus(runAWK(hc, r, args[1:]))+ }+ if name == "xargs" {+ return exitStatus(r.xargs(ctx, hc, args[1:]))+ }+ if f, ok := coreutils[name]; ok {+ return exitStatus(f(&uctx{hc: hc, run: r}, args[1:]))+ }+ msg := fmt.Sprintf("%s: command not found in this sandbox", name)+ if why, ok := r.s.Denied[name]; ok {+ msg = fmt.Sprintf("%s: %s", name, why)+ }+ fmt.Fprintln(hc.Stderr, msg)+ return interp.NewExitStatus(127)+}++func exitStatus(code int) error {+ if code == 0 {+ return nil+ }+ return interp.NewExitStatus(uint8(code))+}++// limitedBuffer keeps the first n bytes it is given.+type limitedBuffer struct {+ buf bytes.Buffer+ n int+ trunc bool+}++func (l *limitedBuffer) Write(p []byte) (int, error) {+ if room := l.n - l.buf.Len(); room > 0 {+ if len(p) > room {+ l.buf.Write(p[:room])+ l.trunc = true+ } else {+ l.buf.Write(p)+ }+ } else if len(p) > 0 {+ l.trunc = true+ }+ return len(p), nil+}++func stdinOf(hc interp.HandlerContext) io.Reader {+ if hc.Stdin == nil {+ return nil+ }+ if f, ok := hc.Stdin.(*os.File); ok && f == nil {+ return nil+ }+ return hc.Stdin+}++func (r *shellRun) execTraceway(ctx context.Context, hc interp.HandlerContext, bin string, args []string) error {+ cmd := exec.CommandContext(ctx, bin, args[1:]...)+ cmd.Dir = hc.Dir+ cmd.Env = r.s.BinEnv+ if in := stdinOf(hc); in != nil {+ cmd.Stdin = in+ }+ so := &limitedBuffer{n: 200000}+ se := &limitedBuffer{n: 20000}+ cmd.Stdout = io.MultiWriter(hc.Stdout, so)+ cmd.Stderr = io.MultiWriter(hc.Stderr, se)+ err := cmd.Run()+ exit := 0+ if err != nil {+ var ee *exec.ExitError+ if errors.As(err, &ee) {+ exit = ee.ExitCode()+ } else {+ fmt.Fprintln(hc.Stderr, err)+ exit = 126+ }+ }+ output := so.buf.String()+ if se.buf.Len() > 0 {+ output += "\n" + se.buf.String()+ }+ op := ops.ParseCLI(args[1:])+ r.record(record.OpRecord{Op: op, Argv: append([]string{filepath.Base(bin)}, args[1:]...), Exit: exit, OK: exit == 0, Output: output, Truncated: so.trunc})+ return exitStatus(exit)+}++var gitAllowed = []string{"log", "show", "diff", "status", "blame", "ls-files", "rev-parse", "shortlog"}++func (r *shellRun) execGit(ctx context.Context, hc interp.HandlerContext, args []string) error {+ sub := ""+ for _, a := range args[1:] {+ if !strings.HasPrefix(a, "-") {+ sub = a+ break+ }+ if a == "-C" || strings.HasPrefix(a, "--git-dir") || strings.HasPrefix(a, "--work-tree") || strings.HasPrefix(a, "-c") {+ fmt.Fprintln(hc.Stderr, "git: global options are not allowed in this sandbox")+ return interp.NewExitStatus(2)+ }+ }+ if !slices.Contains(gitAllowed, sub) {+ fmt.Fprintf(hc.Stderr, "git %s: only read-only git (%s) is available\n", sub, strings.Join(gitAllowed, ", "))+ return interp.NewExitStatus(2)+ }+ cmd := exec.CommandContext(ctx, "git", append([]string{"--no-pager"}, args[1:]...)...)+ cmd.Dir = hc.Dir+ cmd.Env = []string{"HOME=" + r.s.Sandbox.Root, "PATH=/usr/bin:/bin", "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_PAGER=cat", "PAGER=cat", "TZ=UTC", "LANG=C.UTF-8", "GIT_OPTIONAL_LOCKS=0"}+ so := &limitedBuffer{n: 100000}+ cmd.Stdout = io.MultiWriter(hc.Stdout, so)+ cmd.Stderr = hc.Stderr+ err := cmd.Run()+ exit := 0+ if err != nil {+ var ee *exec.ExitError+ if errors.As(err, &ee) {+ exit = ee.ExitCode()+ } else {+ exit = 126+ }+ }+ r.record(record.OpRecord{Op: ops.Op{Op: ops.OpGit, Args: map[string]any{"subcommand": sub}}, Argv: args, Exit: exit, OK: exit == 0, Output: so.buf.String()})+ return exitStatus(exit)+}++func (r *shellRun) open(ctx context.Context, path string, flag int, perm os.FileMode) (io.ReadWriteCloser, error) {+ hc := interp.HandlerCtx(ctx)+ abs, err := r.s.Sandbox.Resolve(hc.Dir, path)+ if err != nil {+ return nil, err+ }+ if flag&(os.O_WRONLY|os.O_RDWR|os.O_CREATE|os.O_APPEND|os.O_TRUNC) != 0 {+ if err := r.s.Sandbox.Writable(abs); err != nil {+ return nil, err+ }+ r.record(record.OpRecord{Op: ops.Op{Op: ops.OpWriteFile, Args: map[string]any{"path": r.s.Sandbox.Rel(abs)}, Write: false}, OK: true})+ }+ return os.OpenFile(abs, flag, perm)+}++func (r *shellRun) readDir(ctx context.Context, path string) ([]fs.DirEntry, error) {+ hc := interp.HandlerCtx(ctx)+ abs, err := r.s.Sandbox.Resolve(hc.Dir, path)+ if err != nil {+ return nil, err+ }+ return os.ReadDir(abs)+}++func (r *shellRun) stat(ctx context.Context, name string, follow bool) (fs.FileInfo, error) {+ hc := interp.HandlerCtx(ctx)+ abs, err := r.s.Sandbox.Resolve(hc.Dir, name)+ if err != nil {+ return nil, os.ErrNotExist+ }+ if follow {+ return os.Stat(abs)+ }+ return os.Lstat(abs)+}++// xargs builds command lines from stdin and runs them through the same+// allowlisted handler.+func (r *shellRun) xargs(ctx context.Context, hc interp.HandlerContext, args []string) int {+ replace := ""+ perCall := 0+ i := 0+ for ; i < len(args); i++ {+ a := args[i]+ switch {+ case a == "-I" && i+1 < len(args):+ replace = args[i+1]+ i+++ case strings.HasPrefix(a, "-I") && len(a) > 2:+ replace = a[2:]+ case a == "-n" && i+1 < len(args):+ fmt.Sscanf(args[i+1], "%d", &perCall)+ i+++ case a == "-r" || a == "--no-run-if-empty" || a == "-0":+ case strings.HasPrefix(a, "-"):+ fmt.Fprintf(hc.Stderr, "xargs: unsupported option %s\n", a)+ return 2+ default:+ goto done+ }+ }+done:+ cmdArgs := args[i:]+ if len(cmdArgs) == 0 {+ cmdArgs = []string{"echo"}+ }+ in := stdinOf(hc)+ if in == nil {+ return 0+ }+ data, _ := io.ReadAll(in)+ var items []string+ if replace != "" {+ for _, line := range strings.Split(string(data), "\n") {+ if strings.TrimSpace(line) != "" {+ items = append(items, strings.TrimSpace(line))+ }+ }+ } else {+ items = strings.Fields(string(data))+ }+ code := 0+ runOne := func(argv []string) {+ if err := r.exec(ctx, argv); err != nil {+ if st, ok := interp.IsExitStatus(err); ok {+ code = 123+ _ = st+ } else {+ code = 1+ }+ }+ }+ switch {+ case replace != "":+ for _, it := range items {+ argv := make([]string, len(cmdArgs))+ for k, a := range cmdArgs {+ argv[k] = strings.ReplaceAll(a, replace, it)+ }+ runOne(argv)+ }+ case perCall > 0:+ for k := 0; k < len(items); k += perCall {+ end := min(k+perCall, len(items))+ runOne(append(append([]string{}, cmdArgs...), items[k:end]...))+ }+ default:+ if len(items) > 0 {+ runOne(append(append([]string{}, cmdArgs...), items...))+ }+ }+ return code+}++type lockedBuffer struct {+ mu sync.Mutex+ buf bytes.Buffer+}++func (l *lockedBuffer) Write(p []byte) (int, error) {+ l.mu.Lock()+ defer l.mu.Unlock()+ return l.buf.Write(p)+}++func (l *lockedBuffer) WriteString(s string) {+ l.mu.Lock()+ defer l.mu.Unlock()+ l.buf.WriteString(s)+}++func (l *lockedBuffer) String() string {+ l.mu.Lock()+ defer l.mu.Unlock()+ return l.buf.String()+}diff --git a/cli/test/evals/internal/tools/shell_test.go b/cli/test/evals/internal/tools/shell_test.gonew file mode 100644index 00000000..eb8f13cf--- /dev/null+++ b/cli/test/evals/internal/tools/shell_test.go@@ -0,0 +1,144 @@+package tools++import (+ "context"+ "encoding/json"+ "os"+ "path/filepath"+ "strings"+ "testing"+)++func newShell(t *testing.T) (*Shell, string) {+ t.Helper()+ root := t.TempDir()+ if err := os.MkdirAll(filepath.Join(root, "work", ".skills", "traceway"), 0o755); err != nil {+ t.Fatal(err)+ }+ work := filepath.Join(root, "work")+ os.WriteFile(filepath.Join(work, ".skills", "traceway", "SKILL.md"), []byte("---\nname: traceway\ndescription: 'x'\n---\nbody"), 0o644)+ os.WriteFile(filepath.Join(work, "notes.txt"), []byte("alpha 3\nbeta 1\ngamma 2\nbeta 1\n"), 0o644)+ // A stand-in for the traceway binary: prints its args and environment.+ bin := filepath.Join(root, "bin", "traceway")+ os.MkdirAll(filepath.Dir(bin), 0o755)+ os.WriteFile(bin, []byte("#!/bin/sh\nif [ \"$1\" = env ]; then env; exit 0; fi\nif [ \"$1\" = fail ]; then echo '{\"error\":\"not_found\"}' >&2; exit 5; fi\necho '{\"data\":[{\"endpoint\":\"GET /a\",\"p95Duration\":1500000000},{\"endpoint\":\"GET /b\",\"p95Duration\":20000000}]}'\n"), 0o755)+ sb := &Sandbox{Root: work, ReadOnly: []string{".skills"}}+ return &Shell{Sandbox: sb, Bins: map[string]string{"traceway": bin}, BinEnv: []string{"HOME=" + root, "PATH=/usr/bin:/bin", "TZ=UTC"}}, work+}++func run(t *testing.T, s *Shell, cmd string) Result {+ t.Helper()+ raw, _ := json.Marshal(map[string]string{"command": cmd})+ return s.Call(context.Background(), raw)+}++func TestShellUtilities(t *testing.T) {+ s, _ := newShell(t)+ cases := []struct{ cmd, want string }{+ {`echo hi | grep h`, "hi"},+ {`sort -k2 -n notes.txt | head -1`, "beta 1"},+ {`sort notes.txt | uniq -c | sort -rn | head -1 | tr -s ' '`, "2 beta 1"},+ {`cut -d' ' -f1 notes.txt | tail -n 1`, "beta"},+ {`awk '{s+=$2} END {print s}' notes.txt`, "7"},+ {`echo '{"a":[1,2,3]}' | jq '.a | length'`, "3"},+ {`echo '{"a":"x"}' | jq -r .a`, "x"},+ {`grep -c beta notes.txt`, "2"},+ {`wc -l < notes.txt`, "4"},+ {`X=$(echo nested); echo "got $X"`, "got nested"},+ }+ for _, c := range cases {+ r := run(t, s, c.cmd)+ if !r.OK || strings.TrimSpace(r.Output) != c.want {+ t.Errorf("%s: got ok=%v %q, want %q", c.cmd, r.OK, r.Output, c.want)+ }+ }+}++func TestShellConfinement(t *testing.T) {+ s, work := newShell(t)+ for _, cmd := range []string{`cat /etc/passwd`, `cat ../../etc/passwd`, `ls /`, `grep root /etc/passwd`, `cd / && cat etc/hostname`} {+ if r := run(t, s, cmd); r.OK || !strings.Contains(r.Output, "outside the workspace") && !strings.Contains(r.Output, "exit status") {+ t.Errorf("%s escaped the sandbox: %q", cmd, r.Output)+ }+ }+ for _, cmd := range []string{`curl https://example.com`, `python3 -c 1`, `bash -c id`, `sh -c id`, `node -e 1`, `go version`} {+ if r := run(t, s, cmd); r.OK || !strings.Contains(r.Output, "not found") {+ t.Errorf("%s should be unavailable: %q", cmd, r.Output)+ }+ }+ if r := run(t, s, `echo hacked > .skills/traceway/SKILL.md`); r.OK {+ t.Errorf("wrote into .skills: %q", r.Output)+ }+ if r := run(t, s, `echo kept > out.txt && cat out.txt`); !r.OK || strings.TrimSpace(r.Output) != "kept" {+ t.Errorf("workspace write failed: %q", r.Output)+ }+ if _, err := os.Stat(filepath.Join(work, "out.txt")); err != nil {+ t.Error(err)+ }+}++func TestShellRecordsTracewayInPipelines(t *testing.T) {+ s, _ := newShell(t)+ r := run(t, s, `traceway endpoints list --since 24h --order-by p95 --output json | jq -r '.data[0].endpoint'`)+ if !r.OK || strings.TrimSpace(r.Output) != "GET /a" {+ t.Fatalf("pipeline output %q", r.Output)+ }+ if len(r.Ops) != 1 || r.Ops[0].Op.Op != "endpoints.list" || r.Ops[0].Args["since"] != "24h" || r.Ops[0].Args["order_by"] != "p95" {+ t.Fatalf("ops %+v", r.Ops)+ }+ if !strings.Contains(r.Ops[0].Output, "p95Duration") {+ t.Error("the raw output before the pipe must be kept for grounding")+ }+ r = run(t, s, `traceway fail x; traceway exceptions show abc`)+ if len(r.Ops) != 2 || r.Ops[0].OK || r.Ops[0].Exit != 5 || !r.Ops[1].OK || r.Ops[1].Args["hash"] != "abc" {+ t.Fatalf("ops %+v", r.Ops)+ }+}++// TestToolEnvIsScrubbed asserts plan §3.5: secrets in the harness's own+// environment never reach tool subprocesses.+func TestToolEnvIsScrubbed(t *testing.T) {+ t.Setenv("OPENROUTER_API_KEY", "sk-or-secret")+ t.Setenv("GITHUB_TOKEN", "ghp_secret")+ t.Setenv("JWT_SECRET", "jwt-secret-value-that-is-long-enough")+ s, _ := newShell(t)+ r := run(t, s, `traceway env`)+ for _, leak := range []string{"sk-or-secret", "ghp_secret", "jwt-secret-value", "OPENROUTER", "GITHUB_TOKEN", "JWT_SECRET"} {+ if strings.Contains(r.Output, leak) {+ t.Errorf("tool environment leaked %s:\n%s", leak, r.Output)+ }+ }+ r = run(t, s, `echo "$OPENROUTER_API_KEY$GITHUB_TOKEN$JWT_SECRET"`)+ if strings.TrimSpace(r.Output) != "" {+ t.Errorf("interpreter environment leaked: %q", r.Output)+ }+}++func TestEditAndReadFile(t *testing.T) {+ s, _ := newShell(t)+ ed := &EditFile{Sandbox: s.Sandbox}+ res := ed.Call(context.Background(), json.RawMessage(`{"path":"notes.txt","old_string":"gamma 2","new_string":"gamma 9"}`))+ if !res.OK {+ t.Fatal(res.Output)+ }+ rd := &ReadFile{Sandbox: s.Sandbox}+ res = rd.Call(context.Background(), json.RawMessage(`{"path":"notes.txt"}`))+ if !strings.Contains(res.Output, "gamma 9") {+ t.Fatal(res.Output)+ }+ res = ed.Call(context.Background(), json.RawMessage(`{"path":".skills/traceway/SKILL.md","old_string":"body","new_string":"x"}`))+ if res.OK || !strings.Contains(res.Output, "read-only") {+ t.Fatalf("edited .skills: %q", res.Output)+ }+ res = rd.Call(context.Background(), json.RawMessage(`{"path":"/etc/passwd"}`))+ if res.OK {+ t.Fatal("read outside the workspace")+ }+}++func TestAskUserRules(t *testing.T) {+ a := &AskUser{}+ if got := a.Answer("anything"); !strings.Contains(got, "anything to add") {+ t.Fatal(got)+ }+}diff --git a/cli/test/evals/internal/tools/skills.go b/cli/test/evals/internal/tools/skills.gonew file mode 100644index 00000000..7d989529--- /dev/null+++ b/cli/test/evals/internal/tools/skills.go@@ -0,0 +1,118 @@+package tools++import (+ "context"+ "encoding/json"+ "fmt"+ "os"+ "path/filepath"+ "sort"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// Skill is one installed skill: a directory with SKILL.md and companions.+type Skill struct {+ Name string+ Description string+ Dir string // absolute+ Body string // SKILL.md without frontmatter+}++// LoadSkills reads every <dir>/<name>/SKILL.md.+func LoadSkills(dir string) ([]Skill, error) {+ entries, err := os.ReadDir(dir)+ if err != nil {+ if os.IsNotExist(err) {+ return nil, nil+ }+ return nil, err+ }+ var out []Skill+ for _, e := range entries {+ if !e.IsDir() {+ continue+ }+ raw, err := os.ReadFile(filepath.Join(dir, e.Name(), "SKILL.md"))+ if err != nil {+ continue+ }+ name, desc, body := ParseFrontmatter(string(raw))+ if name == "" {+ name = e.Name()+ }+ out = append(out, Skill{Name: name, Description: desc, Dir: filepath.Join(dir, e.Name()), Body: body})+ }+ sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })+ return out, nil+}++// ParseFrontmatter splits a SKILL.md into name, description and body.+func ParseFrontmatter(s string) (name, desc, body string) {+ if !strings.HasPrefix(s, "---\n") {+ return "", "", s+ }+ end := strings.Index(s[4:], "\n---")+ if end < 0 {+ return "", "", s+ }+ front := s[4 : 4+end]+ body = strings.TrimLeft(s[4+end+4:], "\n")+ for _, line := range strings.Split(front, "\n") {+ k, v, ok := strings.Cut(line, ":")+ if !ok {+ continue+ }+ v = strings.TrimSpace(v)+ if len(v) >= 2 && (v[0] == '\'' && v[len(v)-1] == '\'' || v[0] == '"' && v[len(v)-1] == '"') {+ v = v[1 : len(v)-1]+ v = strings.ReplaceAll(v, "''", "'")+ }+ switch strings.TrimSpace(k) {+ case "name":+ name = v+ case "description":+ desc = v+ }+ }+ return name, desc, body+}++// LoadSkill returns a skill's SKILL.md, the way agent harnesses load a+// skill on demand. Companion files are read with read_file.+type LoadSkill struct {+ Skills []Skill+ // Rel maps a skill's dir to its workspace-relative path (.skills/<name>).+ Rel func(abs string) string+ Loaded *[]string+}++func (t *LoadSkill) Spec() Spec {+ var names []string+ for _, s := range t.Skills {+ names = append(names, s.Name)+ }+ return Spec{Name: "load_skill", Description: "Load an installed skill's instructions (its SKILL.md). Installed skills: " + strings.Join(names, ", ") + ". Files the skill mentions live next to it and can be read with read_file.",+ Parameters: obj(map[string]any{"name": map[string]any{"type": "string", "enum": names, "description": "Skill name."}}, "name")}+}++func (t *LoadSkill) Call(_ context.Context, raw json.RawMessage) Result {+ var in struct {+ Name string `json:"name"`+ }+ if err := decodeArgs(raw, &in); err != nil {+ return errResult("%v", err)+ }+ for _, s := range t.Skills {+ if s.Name == in.Name {+ if t.Loaded != nil {+ *t.Loaded = append(*t.Loaded, s.Name)+ }+ out := fmt.Sprintf("Skill %s (files in %s/):\n\n%s", s.Name, t.Rel(s.Dir), s.Body)+ return Result{Output: out, OK: true, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpLoadSkill, Args: map[string]any{"name": s.Name}}, OK: true}}}+ }+ }+ return Result{Output: "error: no skill named " + in.Name, Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpLoadSkill, Args: map[string]any{"name": in.Name}}}}}+}diff --git a/cli/test/evals/internal/tools/submit.go b/cli/test/evals/internal/tools/submit.gonew file mode 100644index 00000000..44c080e9--- /dev/null+++ b/cli/test/evals/internal/tools/submit.go@@ -0,0 +1,201 @@+package tools++import (+ "context"+ "encoding/json"+ "fmt"+ "slices"+ "strings"+ "sync"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/checklist"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/ops"+ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// Submit names.+const (+ SubmitPerf = "submit_perf_diagnosis"+ SubmitDebug = "submit_debug_diagnosis"+ Finish = "finish"+)++// DebugCategories are submit_debug_diagnosis's category values.+var DebugCategories = []string{"null_reference", "missing_key", "type_or_contract_mismatch", "race_or_duplicate", "timeout", "config", "resource_exhaustion", "logic", "unknown"}++// Submit is a final-answer tool. It validates the payload against its+// schema and the call ids it cites; a failure goes back to the model for+// repair, up to MaxRepairs times, after which the last payload is accepted+// as-is for grading.+type Submit struct {+ Kind string // SubmitPerf | SubmitDebug | Finish+ CallExists func(id string) (ok, exists bool)+ MaxRepairs int++ mu sync.Mutex+ repairs int+}++func (t *Submit) Spec() Spec {+ switch t.Kind {+ case SubmitPerf:+ var rows []string+ for _, r := range checklist.Rows() {+ rows = append(rows, fmt.Sprintf("%s (%s)", r.ID, r.Title))+ }+ ids := checklist.IDs()+ item := map[string]any{"type": "string", "enum": ids}+ return Spec{Name: SubmitPerf, Description: "Submit your final performance diagnosis. This ends the task. " ++ "checklist_item is the bottleneck checklist row that explains the latency, or none_found when the latency is within expectations, or insufficient_evidence when the telemetry cannot tell. Rows: " + strings.Join(rows, "; ") + ". " ++ "Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",+ Parameters: obj(map[string]any{+ "checklist_item": item,+ "category": strProp("database, application, network, caching, infrastructure, or none"),+ "locus": obj(map[string]any{+ "endpoint": strProp("Endpoint name, e.g. GET /api/orders"),+ "span": strProp("The dominating span or query"),+ "service": strProp("Service name"),+ "host": strProp("Host or instance, when the cause is host-specific"),+ }),+ "onset": map[string]any{"type": []string{"string", "null"}, "description": "RFC 3339 time the slowness started, or null"},+ "evidence": evidenceSchema(),+ "ruled_out": map[string]any{"type": "array", "items": obj(map[string]any{"checklist_item": item, "call_id": strProp("Call id"), "reason": strProp("Why it is ruled out")}, "checklist_item", "reason")},+ "fix": strProp("The concrete fix"),+ "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},+ "report": strProp("Your report to the user, in Markdown"),+ }, "checklist_item", "locus", "evidence", "fix", "confidence", "report")}+ case SubmitDebug:+ return Spec{Name: SubmitDebug, Description: "Submit your final debugging diagnosis. This ends the task. Every evidence item cites the id of an earlier tool call (the [call cNN] line at the top of its result).",+ Parameters: obj(map[string]any{+ "issue_hash": strProp("The exception group hash (16 hex), if the bug has one"),+ "occurrence_id": strProp("The occurrence you anchored on, if any"),+ "category": map[string]any{"type": "string", "enum": DebugCategories},+ "culprit": obj(map[string]any{+ "file": strProp("Workspace-relative file"),+ "symbol": strProp("Function or method"),+ "line": intProp("Line number"),+ }, "file"),+ "evidence": evidenceSchema(),+ "fix_summary": strProp("What you changed or propose to change"),+ "files_changed": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},+ "needs_instrumentation": map[string]any{"type": "boolean", "description": "True when the telemetry cannot explain the failure and you added instrumentation instead of a fix"},+ "confidence": map[string]any{"type": "string", "enum": []string{"low", "medium", "high"}},+ "report": strProp("Your report to the user, in Markdown"),+ }, "category", "culprit", "evidence", "fix_summary", "needs_instrumentation", "confidence", "report")}+ default:+ return Spec{Name: Finish, Description: "Finish the task and report what you did. This ends the task.",+ Parameters: obj(map[string]any{+ "summary": strProp("Your report to the user, in Markdown"),+ "changes": map[string]any{"type": "array", "items": obj(map[string]any{+ "action": strProp("What you changed, e.g. snooze alert rule"),+ "target": strProp("The record, e.g. rule 12"),+ "new_value": strProp("The new state"),+ "undo": strProp("How to undo it"),+ }, "action", "target")},+ }, "summary", "changes")}+ }+}++func evidenceSchema() map[string]any {+ return map[string]any{"type": "array", "minItems": 1, "items": obj(map[string]any{+ "call_id": strProp("Id of the tool call, e.g. c04"),+ "claim": strProp("What that call showed"),+ }, "call_id", "claim")}+}++func (t *Submit) Call(_ context.Context, raw json.RawMessage) Result {+ var payload map[string]any+ if err := decodeArgs(raw, &payload); err != nil {+ return t.repair(nil, []string{err.Error()})+ }+ problems := t.validate(payload)+ if len(problems) > 0 {+ return t.repair(payload, problems)+ }+ return t.accept(payload, "")+}++func (t *Submit) repair(payload map[string]any, problems []string) Result {+ t.mu.Lock()+ defer t.mu.Unlock()+ t.repairs+++ if t.repairs > t.MaxRepairs && payload != nil {+ return t.accept(payload, strings.Join(problems, "; "))+ }+ return Result{Output: "Your submission was not accepted. Fix these problems and call " + t.Kind + " again:\n- " + strings.Join(problems, "\n- "), OK: false,+ Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpSubmit, Args: map[string]any{"tool": t.Kind, "accepted": false}}}}}+}++func (t *Submit) accept(payload map[string]any, problems string) Result {+ msg := "Submission received."+ if problems != "" {+ payload["_validation_problems"] = problems+ }+ return Result{Output: msg, OK: true, Final: true, Submission: payload,+ Ops: []record.OpRecord{{Op: ops.Op{Op: ops.OpSubmit, Args: map[string]any{"tool": t.Kind, "accepted": true}}, OK: true}}}+}++func (t *Submit) validate(p map[string]any) []string {+ var probs []string+ str := func(k string) string { s, _ := p[k].(string); return s }+ need := func(keys ...string) {+ for _, k := range keys {+ if _, ok := p[k]; !ok {+ probs = append(probs, k+" is required")+ }+ }+ }+ checkEvidence := func() {+ ev, ok := p["evidence"].([]any)+ if !ok || len(ev) == 0 {+ probs = append(probs, "evidence must cite at least one tool call")+ return+ }+ for i, e := range ev {+ m, _ := e.(map[string]any)+ id, _ := m["call_id"].(string)+ if id == "" {+ probs = append(probs, fmt.Sprintf("evidence[%d].call_id is required", i))+ continue+ }+ if t.CallExists != nil {+ if _, exists := t.CallExists(id); !exists {+ probs = append(probs, fmt.Sprintf("evidence[%d].call_id %q is not a call in this session (ids look like c04)", i, id))+ }+ }+ }+ }+ switch t.Kind {+ case SubmitPerf:+ need("checklist_item", "locus", "evidence", "fix", "confidence", "report")+ if v := str("checklist_item"); v != "" && !checklist.IsID(v) {+ probs = append(probs, fmt.Sprintf("checklist_item %q is not one of the allowed values", v))+ }+ if rs, ok := p["ruled_out"].([]any); ok {+ for i, r := range rs {+ m, _ := r.(map[string]any)+ if v, _ := m["checklist_item"].(string); v != "" && !checklist.IsID(v) {+ probs = append(probs, fmt.Sprintf("ruled_out[%d].checklist_item %q is not an allowed value", i, v))+ }+ }+ }+ checkEvidence()+ case SubmitDebug:+ need("category", "culprit", "evidence", "fix_summary", "needs_instrumentation", "confidence", "report")+ if v := str("category"); v != "" && !slices.Contains(DebugCategories, v) {+ probs = append(probs, fmt.Sprintf("category %q is not one of %v", v, DebugCategories))+ }+ if c, ok := p["culprit"].(map[string]any); ok {+ if f, _ := c["file"].(string); f == "" {+ probs = append(probs, "culprit.file is required")+ }+ }+ checkEvidence()+ default:+ need("summary", "changes")+ }+ if c := str("confidence"); c != "" && !slices.Contains([]string{"low", "medium", "high"}, c) {+ probs = append(probs, "confidence must be low, medium or high")+ }+ return probs+}diff --git a/cli/test/evals/internal/tools/tools.go b/cli/test/evals/internal/tools/tools.gonew file mode 100644index 00000000..781be5e8--- /dev/null+++ b/cli/test/evals/internal/tools/tools.go@@ -0,0 +1,181 @@+// Package tools implements what the agent under test can call: a sandboxed+// shell (an interpreter with an allowlist, never the host shell), confined+// file tools, skills, ask_user, and the structured submit tools. Every call+// returns the operations it performed so the graders can check the+// trajectory.+package tools++import (+ "context"+ "encoding/json"+ "errors"+ "fmt"+ "os"+ "path/filepath"+ "strings"++ "github.com/tracewayapp/traceway/cli/test/evals/internal/record"+)++// Spec is a tool's function definition as the model sees it.+type Spec struct {+ Name string+ Description string+ Parameters map[string]any+}++// Result is what one call produced.+type Result struct {+ Output string+ OK bool+ Truncated bool+ Ops []record.OpRecord+ // Final is set when a submit/finish payload validated: the run ends.+ Final bool+ Submission map[string]any+}++// Tool is one callable tool.+type Tool interface {+ Spec() Spec+ Call(ctx context.Context, args json.RawMessage) Result+}++// MaxOutput caps what one shell or file call returns to the model.+const MaxOutput = 30000++// Cap truncates s to MaxOutput characters with a marker.+func Cap(s string) (string, bool) { return CapAt(s, MaxOutput) }++// CapAt truncates s to n characters with a marker.+func CapAt(s string, n int) (string, bool) {+ if len(s) <= n {+ return s, false+ }+ more := len(s) - n+ return s[:n] + fmt.Sprintf("\n[output truncated: %d more characters. Narrow the query (smaller window, page size, field selection) or filter the output.]", more), true+}++// Sandbox confines file access to the run's workspace.+type Sandbox struct {+ Root string // absolute workspace path+ ReadOnly []string // workspace-relative prefixes that cannot be written (.skills)+}++// ErrOutside is returned for any path that leaves the workspace.+var ErrOutside = errors.New("path is outside the workspace")++// Resolve maps p (absolute, or relative to base) to an absolute path inside+// the workspace. Symlinks pointing out of the workspace are refused.+func (s *Sandbox) Resolve(base, p string) (string, error) {+ if p == "/dev/null" {+ return p, nil+ }+ if base == "" {+ base = s.Root+ }+ var abs string+ if filepath.IsAbs(p) {+ abs = filepath.Clean(p)+ } else {+ abs = filepath.Clean(filepath.Join(base, p))+ }+ if !within(s.Root, abs) {+ return "", fmt.Errorf("%s: %w", p, ErrOutside)+ }+ // Resolve the longest existing prefix to catch symlink escapes.+ probe := abs+ for {+ real, err := filepath.EvalSymlinks(probe)+ if err == nil {+ rootReal, _ := filepath.EvalSymlinks(s.Root)+ if rootReal == "" {+ rootReal = s.Root+ }+ if !within(rootReal, real) {+ return "", fmt.Errorf("%s: %w", p, ErrOutside)+ }+ break+ }+ parent := filepath.Dir(probe)+ if parent == probe || !within(s.Root, parent) {+ break+ }+ probe = parent+ }+ return abs, nil+}++// Writable reports whether abs may be written.+func (s *Sandbox) Writable(abs string) error {+ if abs == "/dev/null" {+ return nil+ }+ rel, err := filepath.Rel(s.Root, abs)+ if err != nil {+ return err+ }+ for _, ro := range s.ReadOnly {+ if rel == ro || strings.HasPrefix(rel, ro+string(filepath.Separator)) {+ return fmt.Errorf("%s is read-only", filepath.ToSlash(rel))+ }+ }+ return nil+}++// Rel returns the workspace-relative form of abs.+func (s *Sandbox) Rel(abs string) string {+ rel, err := filepath.Rel(s.Root, abs)+ if err != nil {+ return abs+ }+ return filepath.ToSlash(rel)+}++func within(root, p string) bool {+ if p == root {+ return true+ }+ return strings.HasPrefix(p, strings.TrimSuffix(root, string(filepath.Separator))+string(filepath.Separator))+}++// decodeArgs unmarshals tool arguments, tolerating an empty payload.+func decodeArgs(raw json.RawMessage, v any) error {+ if len(raw) == 0 || string(raw) == "null" {+ raw = []byte("{}")+ }+ if err := json.Unmarshal(raw, v); err != nil {+ return fmt.Errorf("invalid arguments: %v", err)+ }+ return nil+}++func errResult(format string, a ...any) Result {+ return Result{Output: "error: " + fmt.Sprintf(format, a...), OK: false}+}++// readFileCapped reads a workspace file for a tool.+func readFileCapped(path string) (string, error) {+ b, err := os.ReadFile(path)+ if err != nil {+ return "", err+ }+ return string(b), nil+}++// obj builds a JSON-schema object.+func obj(props map[string]any, required ...string) map[string]any {+ m := map[string]any{"type": "object", "properties": props, "additionalProperties": false}+ if len(required) > 0 {+ m["required"] = required+ }+ return m+}++func strProp(desc string) map[string]any {+ return map[string]any{"type": "string", "description": desc}+}++func intProp(desc string) map[string]any {+ return map[string]any{"type": "integer", "description": desc}+}diff --git a/cli/test/evals/models.yaml b/cli/test/evals/models.yamlnew file mode 100644index 00000000..d7ffb190--- /dev/null+++ b/cli/test/evals/models.yaml@@ -0,0 +1,60 @@+# Pinned OpenRouter models for the eval agents and the judge. Every call goes+# through OpenRouter Chat Completions with one key (OPENROUTER_API_KEY).+#+# Never use a floating alias (~provider/model-latest): results must stay+# comparable over time, so a model change is a reviewed edit to this file.+# Prices and supported_parameters were checked against+# GET https://openrouter.ai/api/v1/models on 2026-10-17. The preflight reads live+# prices; fallback_price is used only when that list cannot be fetched.+#+# estimate is the per-run token envelope the preflight multiplies by live+# prices until baseline.json carries measured averages (plan §10).++agents:+ opus:+ id: anthropic/claude-opus-5.5+ label: Claude Opus 5.5+ provider:+ # Anthropic first so the explicit prompt cache keeps hitting the same+ # upstream; fallbacks stay allowed so one provider outage is not a+ # failed run (the serving provider is recorded per run).+ order: [anthropic]+ allow_fallbacks: true+ caching: anthropic+ reasoning_effort: medium+ max_tokens_per_turn: 16000+ estimate:+ input_tokens: [100000, 400000]+ cached_fraction: 0.7+ output_tokens: [5000, 15000]+ fallback_price: {prompt: 4.00, completion: 20.00, cache_read: 0.20, cache_write: 5.00}++ astra:+ id: openai/gpt-6-astra+ label: GPT-6 Astra+ provider:+ order: [openai]+ allow_fallbacks: true+ caching: automatic+ reasoning_effort: medium+ max_tokens_per_turn: 16000+ estimate:+ input_tokens: [100000, 400000]+ cached_fraction: 0.7+ output_tokens: [5000, 15000]+ fallback_price: {prompt: 10.00, completion: 50.00, cache_read: 1.00, cache_write: 12.50}++judge:+ id: anthropic/claude-haiku-5.5+ label: Claude Haiku 5.5 (judge)+ provider:+ order: [anthropic]+ allow_fallbacks: true+ caching: none+ max_tokens_per_turn: 2000+ temperature: 0+ estimate:+ input_tokens: [6000, 15000]+ cached_fraction: 0+ output_tokens: [300, 1200]+ fallback_price: {prompt: 0.10, completion: 0.50, cache_read: 0.01, cache_write: 0.125}diff --git a/cli/test/evals/presets.yaml b/cli/test/evals/presets.yamlnew file mode 100644index 00000000..4352c574--- /dev/null+++ b/cli/test/evals/presets.yaml@@ -0,0 +1,16 @@+# Case and arm selections for `twevals plan|run --preset`. A --cases glob+# overrides the preset's case filter; --arms overrides its arm policy.+presets:+ smoke:+ description: The starred cases, each suite's primary arm only+ tags: [smoke]+ arms: primary+ core:+ description: Every case, each suite's primary arm only+ arms: primary+ full:+ description: Every case on every arm it lists (adds cli-noskill)+ arms: all++primary_arms:+ perf: [cli+skill]