Open source CapCut alternative.
TypeScript
docs: how a speech cut finds its place
David Lee273522338d4
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.mdindex 30c08e5d..586fca0b 100644--- a/docs/guides/cut/ai-assistant.md+++ b/docs/guides/cut/ai-assistant.md@@ -86,6 +86,7 @@ With a tab attached, tool calls execute in it against the same store the user is- **Frames become images.** A tool result's `image`/`images` data URLs leave the JSON and reach the model as real images on every path — the engine bridge emits MCP image blocks after the data text, and the Gemini loop rides the first on the function response and the rest as image parts. Watching happens in whatever model the user is chatting with; there is no side model.- **Watching leaves a map.** A watch records which moments and cut candidates a source holds, persisted on the asset so it survives the thread, and the first watch of a source starts a quiet background sweep that maps the rest. The map aims later watches; only sheets returned in the conversation count as footage the model has seen.- **Watching leaves a record.** Contact sheets live in one conversation and the media trim drops the oldest of them, so the model writes each look down: a note-source call stores what a span showed — quoted on-screen text, what happens, where the cuts land — against its source seconds, on the asset, saved with the project. Notes come back in the editor snapshot as `observed`, in full from get-state, and on every watch result beside the spans still undescribed. A close note supersedes the coarse one over the same ground, and a watch of a source that still owes a note is refused, which is what makes an hour-long source workable: it is surveyed coarsely, described, then re-watched densely only where the work is.+- **Speech cuts land on the words.** A recut edge is not left where the model's arithmetic put it. The audio around it is re-read, that recording's own room tone and voice level are measured, and the edge is placed a beat clear of where a word truly starts or finishes — tail and soft attack included, so a quiet take and a noisy one are cut the same way. Pace belongs to that step: fast, natural and relaxed set how big the beat is, and the fastest of them still lets every word finish. So "make it punchy" shortens the air rather than cutting into the speech, and where a pause is too short for both sides' air the two edges meet inside it instead of one clipping a word.- **Media rides a byte budget.** Contact sheets, audio, and attachment payloads all replay inline, so before each call the session is trimmed to a 16MB media budget — oldest parts leave first, with a note naming the tools that fetch them again. Text, tool calls, tool results, and thought signatures always survive, and the trim runs on a copy so the stored session keeps everything.- **A link comes back as whatever it holds.** The chat's URL import works down a ladder: the bundled downloader for anything with a stream, X's own endpoints for a post's photos or Article, and otherwise the page itself — a content extractor (Defuddle over a parsed DOM) lifts the article out as markdown and the pictures it kept come along as assets. Words with no media are a successful import, so a link the user pasted to explain something reads fine, and that text is quoted for the user beside whatever media came with it.- **Async when rendering.** Video generation outruns the two-minute tool window, so those tools start the job and return immediately; the clip files under the chat that asked — the owning thread is captured at call time — and lands on the timeline only when the user asked for that. Each render's card state (running, landed, failed) mirrors into the project doc as it changes, so the chat shows the same outcome on machines that never ran the job.@@ -134,6 +135,8 @@ A live eval scores the chat brain's behavior and speed together. Each of its casThe committed report at `evals/cut-chat.latest-report.json` is the baseline. The improvement loop: make one change — a model role, a prompt line, a tool description — run the eval, and keep the change only when pass rates hold and the medians improve; the new report commits with the change, so the baseline always describes what ships. This is how the complexity router and the batched-round cutting flow earned their way in. Run it with the dev server up: `npm run eval:cut-chat` (add `--matrix --runs 3` to compare model configs); the README beside the harness under the site's `scripts/lib/cut-eval/` folder covers the flags, the report schema, and how to add a case. The tab-less paths have their own end-to-end scripts, which drive a real turn against a seeded cloud project through the headless session and through the queued turn job.+Two mechanical paths are guarded without a model at all, because no ear settles them from a preview: caption timing (`npm run eval:cut-sync`) and speech-cut placement (`npm run eval:cut-edges`). Both build their own truth — a take of known words rendered to audio — run the production code over it, and enforce budgets on the result, so a regression in either shows up in seconds with no network and no credits.+## Where it livesThe shared catalog (system prompt, tools, skills), the chat route with its provider runners, the browser-tool bridge, its tab-less executor (`server/ai/headlessTools.ts`), and the stdio MCP proxy live with the engine's AI code under the site's `cut/` folder (`server/ai/`, `server/http/ai.ts`). The page side holds the snapshot builder (`lib/aiContext.ts`), the tool implementations (`lib/aiTools.ts`), the pi-based chat loop and its adapters (`lib/pi/`), the model catalog (`lib/aiModels.ts`), and the chat panel with threads and transport (`components/AiPanel.tsx`). Running the same loop with no browser is `lib/headless/`, the turns route in `server/cloud/turns.ts`, and the worker job in `worker/turnJob.ts`.