Skip to content

Open source CapCut alternative.

TypeScript

docs: cut ai-assistant guide covers the pi loop

David Le4d7b99fea995
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.md
index 8d7f7799..bba9307b 100644
--- a/docs/guides/cut/ai-assistant.md
+++ b/docs/guides/cut/ai-assistant.md
@@ -27,7 +27,9 @@ user message + @attachments + fresh <editor_state> snapshot
On the engine path, the chat route holds one streaming response open per turn. The provider sees a single MCP server named `cut`: a small stdio proxy the provider harness spawns, which forwards every tool listing and call to the engine over HTTP. The engine writes each call into the open chat stream; the tab executes it on the editor store and posts the result back; the engine hands it to the waiting provider. A call the tab never answers times out after two minutes.
-The Gemini path skips the engine entirely — the same carve-out as AI media generation. Before the first round, one light-model call judges the newest message three ways and the harness acts on the verdict: a turn that requests nothing runs with every tool declaration withheld, one self-contained ask runs the loop on the light chat model, and a composed job — or any classifier doubt — runs on the full one. The page then posts the conversation to Donkey's hosted responses route with the user's session, executes any function calls directly against the store, and repeats until the model settles on a reply (at most 24 rounds). Gemini's thought signatures are replayed exactly with each call, and an empty round surfaces as an error rather than a blank bubble. A turn that exhausts the round budget ends with one tools-off call that summarizes what landed and what's left, so a long edit hands off cleanly ("say keep going") instead of dead-ending on a raw error.
+The Gemini path skips the engine entirely — the same carve-out as AI media generation. Before the first round, one light-model call judges the newest message three ways and the harness acts on the verdict: a turn that requests nothing runs with every tool declaration withheld, one self-contained ask runs the loop on the light chat model, and a composed job — or any classifier doubt — runs on the full one. The loop itself runs on the pi agent harness (the `@earendil-works` packages) in the browser: a custom transport speaks Donkey's hosted responses route with the user's session, tools execute directly against the store, and each thread's conversation persists as the harness's own structured history — past tool calls replay as real tool records, so no bookkeeping rides as prose the model could imitate. Gemini's quirks live in that transport: thought signatures replay exactly with each call, tool media returns as its own following turn, and an empty round retries before it surfaces as an error.
+
+Two mechanisms keep a turn honest. A mutation ledger, computed in code from the tool results, rides every model call as an ephemeral context note: what ran, what failed, and what debris the edits left (a parked transition bar, an item stranded past the video's end) — the model must fix or plainly report each entry before its reply lands, and the note is never stored or rendered. And the round budget extends itself: a build that outgrows its 24 rounds gets fresh budget automatically (up to three extensions, then a hard stop that forces a status reply), so the user never types "keep going".
Which path runs is chosen by the model id alone. The picker lives in the page so a site deploy updates it for everyone; the engine only reports which CLIs are installed and signed in (probed and cached for a minute), and the page overlays Gemini availability from its own sign-in probe.
@@ -109,12 +111,12 @@ A hidden hermetic test provider exercises the full engine bridge — context, to
## Measuring the assistant
-A live eval scores the chat brain's behavior and speed together. Each of its cases replays a real composer turn against the live model and asserts what came back — the reply, the tools called, the gate verdict, tools that must stay sheathed — while timing the whole path: the gate call, time to first token, every round, every tool serve. Cases carry a bucket (chat turns, single-tool edits, multi-tool composed edits) so the numbers aggregate by the kind of turn a user feels.
+A live eval scores the chat brain's behavior and speed together. Each of its cases replays a real composer turn against the live model — through the same loop code production runs — and asserts what came back: the reply, the tools called, the gate verdict, tools that must stay sheathed, and the reply's shape (no leaked scaffolding, no trailing upsell), with a light-model judge leaving tone notes beside the pass/fail. All of it is timed: the gate call, time to first token, every round, every tool serve. Cases carry a bucket (chat turns, single-tool edits, multi-tool composed edits) so the numbers aggregate by the kind of turn a user feels.
The committed report at `evals/cut-chat.latest-report.json` is the baseline. The improvement loop: make one change — a model role, a prompt line, a tool description — run the eval, and keep the change only when pass rates hold and the medians improve; the new report commits with the change, so the baseline always describes what ships. This is how the complexity router and the batched-round cutting flow earned their way in. Run it with the dev server up: `npm run eval:cut-chat` (add `--matrix --runs 3` to compare model configs); the README beside the harness under the site's `scripts/lib/cut-eval/` folder covers the flags, the report schema, and how to add a case.
## Where it lives
-The shared catalog (system prompt, tools, skills), the chat route with its provider runners, the browser-tool bridge, and the stdio MCP proxy live with the engine's AI code under the site's Cut folder (`server/ai/`, `server/http/ai.ts`). The page side holds the snapshot builder (`lib/aiContext.ts`), the tool implementations (`lib/aiTools.ts`), the Gemini loop (`lib/geminiChat.ts`), the model catalog (`lib/aiModels.ts`), and the chat panel with threads and transport (`components/AiPanel.tsx`).
+The shared catalog (system prompt, tools, skills), the chat route with its provider runners, the browser-tool bridge, and the stdio MCP proxy live with the engine's AI code under the site's Cut folder (`server/ai/`, `server/http/ai.ts`). The page side holds the snapshot builder (`lib/aiContext.ts`), the tool implementations (`lib/aiTools.ts`), the pi-based chat loop and its adapters (`lib/pi/`), the model catalog (`lib/aiModels.ts`), and the chat panel with threads and transport (`components/AiPanel.tsx`).
The brief-to-video pipeline is its own subsystem: the orchestrator, coverage invariant, and role interfaces in `lib/genvideo/`, its hosted-model adapters in `lib/genvideo/adapters/`, the browser controller in `lib/genScene.ts`, and the progress card in `components/SceneCard.tsx`. Its strategy — story planning, the identity ladder, where audio goes next — is the [Brief to Video](brief-to-video.md) guide.