Skip to content

Open source CapCut alternative.

TypeScript
5.4 KB
Raw Blame

Cut chat eval

One command measures the chat assistant's behavior and speed together:

npm run eval:cut-chat -- [--runs N] [--only <case>] [--bucket chat|single-tool|multi-tool]
                         [--model <id>] [--simple-model <id>] [--complex-model <id>]
                         [--matrix] [--enforce-budgets] [--out <path>]

It replays real composer turns against the live chat model through the dev server's hosted Responses route. Start next dev on :3000 first, with DONKEY_DEV_AUTH_BYPASS=1 in its env (.env.example carries it). Auth is the dev bypass header, so runs spend no credits. The spoken fixture is synthesized with macOS say, so runs need a Mac.

Every case asserts behavior (the reply, the tools called, the gate verdict, forbidden tools) and records timing: the gate's classify time, time-to-first- token (rounds stream, as production does), per-round wall time, tool-serve time, and round count. The default single run per case is the quick regression check; --runs 3 gives real p50/p95s. Latency aggregates count passing runs only — a wrong answer's speed means nothing — and pass rate is reported alongside so a fast-but-wrong config stays visible.

Every case carries a bucket: chat turns (greetings, questions — the gated fast path), single-tool edits (one decisive call), and multi-tool edits (composed cuts).

The default config is production's: the turn judge's verdict (chat / simple / complex) withholds tools on chat turns, routes simple turns to the light chat model, and declares only the routed tool areas. --model pins both roles to one model for an unrouted comparison; each run's report records the model every turn actually ran on (roundModel). --matrix runs every row of CANDIDATES in eval-cut-chat.ts and prints a comparison table. Model flags and candidate rows resolve through the registry in src/lib/inference/gemini-models.ts; raw model ids pass through.

Per-case budget values in cases.ts are checked against passing-run p50s. They report by default; --enforce-budgets turns a breach into exit 1. The run always exits 1 when a case has zero passing runs.

The queue triage eval

npm run eval:cut-queue -- [--runs N] [--only <case>] checks where a message sent mid-turn goes. Each case in queueCases.ts is a running turn (its ask and the tools it has run so far, plus what other chats of the project are doing) and the messages the user sent while it ran, each with the place it must get: fold into the running turn, spawn a parallel thread, or queue in the tray. The cases run the production triage call against the dev server's judge route and print one line per case; a wrong verdict prints the message, what it got and what it should have got. It exits 1 when a case never passes. Add a case whenever a real message landed in the wrong place.

The routing eval

npm run eval:cut-routing -- [--only <substring>] [--no-sweep] [--filler] checks what the turn judge decides for the labelled asks in routingCases.ts: the skill it attaches, the tool areas it declares, and the gate side. Every ask is judged once and the route recomposed under a grid of thresholds, so the sweep that sets the registry defaults costs no extra calls.

The provider-failure eval

npm run eval:cut-judge -- [--only <substring>] checks what a hosted provider's error text means — the content declined, a transient failure, anything else — against errors real providers returned. The class decides whether the render ladder may drop to its text-only rung and whether a music prompt is worth rewriting, so a case is added whenever a new error reached us and was read wrong.

Nightly

.github/workflows/nightly-evals.yml runs the evals a GitHub runner can carry, grouped by what each needs: a process, a browser, a dev server. The evals that ask a live model are run by hand, because a runner has no key, no database of its own and no budget to spend.

The report

A full run writes evals/cut-chat.latest-report.json (schema cut-chat-eval/v1, types in report.ts): per config → per bucket and per case → pass rate, latency aggregates, budget breaches, and every run's full timings and tool trace. A run narrowed by --only or --bucket writes only when --out names a destination, so iterating never clobbers the baseline. The committed copy is the baseline; git diff on it shows what a change did to speed and correctness.

Improving latency (the loop)

  1. Make one change: bump a role in src/lib/inference/gemini-models.ts, edit the system prompt or a tool description in the catalog, or add a CANDIDATES row for a new model id.
  2. Run npm run eval:cut-chat -- --matrix --runs 3 (narrow with --bucket or --only while iterating).
  3. Compare against the baseline: git diff evals/cut-chat.latest-report.json.
  4. Keep the change only if pass rate holds and the p50s improve.
  5. Commit the new report with the change, so the baseline tracks what ships.

What to look at per bucket: chat turns live and die on gate p50 + TTFT; single-tool on total p50 and whether rounds stays near 2; multi-tool on round count — every extra round is a full round-trip carrying the whole conversation and all ~75 tool declarations.

Adding a case: append to cases.ts with a bucket, stub any mutating tool it should call, and use simulate to assert on arguments. Run it alone with --only <name> and --runs 5 to check it's stable before relying on it.