Open source CapCut alternative.
Cut chat eval
One command measures the chat assistant's behavior and speed together:
npm run eval:cut-chat -- [--runs N] [--only <case>] [--bucket chat|single-tool|multi-tool]
[--model <id>] [--simple-model <id>] [--complex-model <id>]
[--matrix] [--enforce-budgets] [--out <path>]
It replays real composer turns against the live chat model through the dev
server's hosted Responses route. Start next dev on :3000 first, with
DONKEY_DEV_AUTH_BYPASS=1 in its env (.env.example carries it). Auth is the
dev bypass header, so runs spend no credits. The spoken fixture is synthesized
with macOS say, so runs need a Mac.
Every case asserts behavior (the reply, the tools called, the gate verdict,
forbidden tools) and records timing: the gate's classify time, time-to-first-
token (rounds stream, as production does), per-round wall time, tool-serve
time, and round count. The default single run per case is the quick regression
check; --runs 3 gives real p50/p95s. Latency aggregates count passing runs
only — a wrong answer's speed means nothing — and pass rate is reported
alongside so a fast-but-wrong config stays visible.
Every case carries a bucket: chat turns (greetings, questions — the gated
fast path), single-tool edits (one decisive call), and multi-tool edits
(composed cuts).
The default config is production's: the turn judge's verdict (chat / simple /
complex) withholds tools on chat turns, routes simple turns to the light chat
model, and declares only the routed tool areas. --model pins both roles to one model for an unrouted
comparison; each run's report records the model every turn actually ran on
(roundModel). --matrix runs every row of CANDIDATES in
eval-cut-chat.ts and prints a comparison table. Model flags and candidate
rows resolve through the registry in src/lib/inference/gemini-models.ts; raw
model ids pass through.
Per-case budget values in cases.ts are checked against passing-run p50s.
They report by default; --enforce-budgets turns a breach into exit 1. The run
always exits 1 when a case has zero passing runs.
The queue triage eval
npm run eval:cut-queue -- [--runs N] [--only <case>]
checks where a message sent mid-turn goes. Each case in queueCases.ts is a
running turn (its ask and the tools it has run so far, plus what other chats
of the project are doing) and the messages the user sent while it ran, each
with the place it must get: fold into the running turn, spawn a parallel
thread, or queue in the tray. The cases run the production triage call
against the dev server's judge route and print one line per case; a wrong
verdict prints the message, what it got and what it should have got. It exits
1 when a case never passes. Add a case whenever a real message landed in the
wrong place.
The routing eval
npm run eval:cut-routing -- [--only <substring>] [--no-sweep] [--filler]
checks what the turn judge decides for the labelled asks in
routingCases.ts: the skill it attaches, the tool areas it declares, and the
gate side. Every ask is judged once and the route recomposed under a grid of
thresholds, so the sweep that sets the registry defaults costs no extra calls.
The provider-failure eval
npm run eval:cut-judge -- [--only <substring>]
checks what a hosted provider's error text means — the content declined, a
transient failure, anything else — against errors real providers returned. The
class decides whether the render ladder may drop to its text-only rung and
whether a music prompt is worth rewriting, so a case is added whenever a new
error reached us and was read wrong.
Nightly
.github/workflows/nightly-evals.yml runs the evals a GitHub runner can carry,
grouped by what each needs: a process, a browser, a dev server. The evals that
ask a live model are run by hand, because a runner has no key, no database of
its own and no budget to spend.
The report
A full run writes evals/cut-chat.latest-report.json (schema
cut-chat-eval/v1, types in report.ts): per config → per bucket and per
case → pass rate, latency aggregates, budget breaches, and every run's full
timings and tool trace. A run narrowed by --only or --bucket writes only
when --out names a destination, so iterating never clobbers the baseline.
The committed copy is the baseline; git diff on it shows what a change did
to speed and correctness.
Improving latency (the loop)
- Make one change: bump a role in
src/lib/inference/gemini-models.ts, edit the system prompt or a tool description in the catalog, or add aCANDIDATESrow for a new model id. - Run
npm run eval:cut-chat -- --matrix --runs 3(narrow with--bucketor--onlywhile iterating). - Compare against the baseline:
git diff evals/cut-chat.latest-report.json. - Keep the change only if pass rate holds and the p50s improve.
- Commit the new report with the change, so the baseline tracks what ships.
What to look at per bucket: chat turns live and die on gate p50 + TTFT;
single-tool on total p50 and whether rounds stays near 2; multi-tool on
round count — every extra round is a full round-trip carrying the whole
conversation and all ~75 tool declarations.
Adding a case: append to cases.ts with a bucket, stub any mutating tool it
should call, and use simulate to assert on arguments. Run it alone with
--only <name> and --runs 5 to check it's stable before relying on it.