Skip to content

Open source CapCut alternative.

TypeScript
29.2 KB
Raw Blame

Donkey Cut's AI Assistant

The assistant is the chat panel inside the Donkey Cut editor. It answers questions about the open project and edits it by calling typed tools against the live editor state, while the user watches the changes land. Three providers — Claude Code, Codex, and Gemini through Donkey's hosted inference — get the same system prompt, tool catalog, and skills library; only the transport differs.

The one rule: the model makes every decision. No code inspects the user's words. A message goes to a model with the project state attached, and the harness acts only on the typed tool calls that come back. If you're tempted to write if (text.includes("subtitles")) anywhere in the chat path, that knowledge belongs in the system prompt, a tool description, or a skill.

How a turn runs

The model decides what to change; whoever holds the project does the changing — usually the browser tab, and the engine or a worker container when no tab is there. Everything between them — the engine, the MCP proxy, the hosted route — is transport.

user message + @attachments + fresh <editor_state> snapshot
  │
  ├─ Claude or GPT model picked ──▶ local engine, /api/cut/ai/chat
  │       spawns the user's own CLI (Claude Agent SDK / codex exec),
  │       resumes that provider's session, streams the reply back
  │
  │       provider ──stdio MCP proxy──▶ engine ──chat stream──▶ browser tab
  │                                       ▲   runs the tool on the editor
  │                                       └── store, POSTs the result back
  │
  ├─ Gemini picked ──▶ hosted inference route (Donkey sign-in + credits)
  │       the page itself loops: model round → run the function calls
  │       on the editor store → send results back → next round,
  │       until a round returns no calls
  │
  └─ no tab ──▶ the same loop with the doc as its editor: the engine
          executes for a chat whose tab closed mid-turn, and a queued
          turn job runs the whole turn in the worker container

The Mac engine drains each turn into a disk journal independently of its readers; reopening an unfinished turn replays it under the same message id, and Stop cancels the provider. The provider sees one MCP server, cut: a stdio proxy that forwards every tool call to the engine, which writes it into the chat stream for the tab to execute and answer within two minutes. Frames reach the provider as images and sound as its words, transcribed on the Mac.

A Claude or Codex turn can outlive its tab while the Mac engine remains running. The bridge retains the session's project after its last reader disconnects and executes subsequent document tools against that project on disk. Running threads keep their editor connection when the chat panel is hidden or another thread is selected. Tool claims let one attached tab execute each call; a disconnected call with an uncertain result returns that uncertainty to the model. Headless calls serialize access to the engine's editor store, and every call records its result in the journal. Tools that need the page's media primitives or hosted sign-in still return a typed error when detached.

Gemini's interactive turn runs in the browser. Its active turn survives panel and project navigation within that page, saves its final transcript after unmount, and waits for its original project before executing another editor tool. Opening another project waits for tools already using the store. Closing the browser tab ends this loop; the cloud worker's queued-turn API is a separate execution path.

A turn the page lost resumes itself when the thread next opens, replaying the engine journal or sending the model a continuation that finishes the ask with the failure gone from the transcript.

The Gemini path skips the engine entirely — the same carve-out as AI media generation. The loop runs on the pi agent harness (the @earendil-works packages) in the browser: a custom transport speaks Donkey's hosted responses route with the user's session, tools execute directly against the store, and each thread's conversation persists as the harness's own structured history — past tool calls replay as real tool records, so no bookkeeping rides as prose the model could imitate. Gemini's quirks live in that transport: thought signatures replay exactly with each call, tool media returns as its own following turn, and an empty round retries before it surfaces as an error.

The turn's route is one typed judgment: a System One judge scores the ask's size, the skill to open first, and the tool areas it needs, and code applies its thresholds. The first round waits a bounded moment for it, then runs the light chat model with only the routed areas' tools; a simple verdict keeps the run, chat or complex restarts it routed, and a request_tools call widens the catalog for the next round. With credits, Claude and Codex turns take the same judgment, which narrows the engine's MCP listing; Codex reads tools once per run, so a widening resumes it.

The same loop code runs with no browser at all. A headless session opens a project doc into the editor store, binds both transports at an explicit origin with auth headers, and runs the production chat loop against it — which is how a chat turn becomes a durable job. The page or an API caller posts a thread's messages to the cloud project's turns route; the row is the project's agent write lease (one turn at a time per project); the worker container claims it, runs the turn under a runner grant that spends that user's credits and writes that user's project, pushes the doc back through the versioned PUT, and appends the finished exchange to the stored thread.

A job can do everything a tab can, because the same tool code runs over swapped primitives. The library draws, decodes and measures through narrow seams — a canvas factory, a frame sink, a font installer — which the page fills with the browser's own and the container fills at startup with a native canvas, native decoders, a Web Audio graph, and the editor's font files staged in the image. So watching, silence cuts, freeze frames, sticker matting, captions and frame captures behave in a job as in the tab: there is one implementation. Three things stay page-only: on-device MediaPipe matting, which falls through to the hosted vision path; motion tracking, which a job refuses back to the tab; and Lottie stickers, which need a DOM.

Two mechanisms keep a turn honest. A mutation ledger, computed in code from the tool results, rides every model call as an ephemeral context note: what ran, what failed, and what debris the edits left (a parked transition bar, an item stranded past the video's end) — the model must fix or plainly report each entry before its reply lands, and the note is never stored or rendered. And the step budget extends itself, the same way on every provider: a build that outgrows its 24 steps gets fresh budget automatically (up to three extensions, so 96 in all), so the user never types "keep going". Gemini blocks tool calls at the ceiling and gets one round to say where it landed; a CLI run that spends the budget is resumed from its own session. However a turn ends — at the ceiling, on a provider's error, or saying nothing at all — it closes with a line the person can act on, never with the provider's word for it.

The composer stays open while a turn runs. A message sent mid-turn waits in a tray on the composer's top edge and goes out when the running turn settles — one at a time, each with a fresh editor snapshot. A typed triage places each against the work in flight: a refinement is steered into the live turn, an independent ask starts a parallel thread, the rest wait, and the placement is made again as the turn advances. The running agent can park a message there itself. The tray is the review surface — reorder, rewrite, remove, pause, resume — a stop or an error pauses it, and rows are saved with the thread.

@ in the composer writes a moment — @1:34, where the preview sits — and it holds the frame, so a waiting row names the same picture after the turn rearranges the cut.

The model id alone picks the path. The picker lives in the page, so a deploy updates it everywhere; the engine reports each CLI's sign-in and the models its Codex account runs, cached a minute, and the page hides any other model.

What the model knows

Every knowledge surface is defined once — the catalog file ships in the engine and is bundled into the page, so all providers read identical text. The tools and the inspector write the same document, and every setting a tool can put on a selected item has a control in that item's panel; a test holds the pairing, so a change made in chat is always visible, and undoable, by hand.

Surface Size today When it enters context
System prompt 13K chars (3K tokens) Claude: replaces the Agent SDK default. Codex: prepended to a fresh session's first turn. Gemini: the instructions field, every round. Written for the page's language, which replies follow.
Tool catalog (127 tools in 19 areas) 140K chars (35K tokens) whole Core plus the routed areas; the whole catalog when unjudged.
Skills library (16 docs) 97K chars (24K tokens) total The judge's pick rides the turn; list-skills and read-skill cover the rest.
Editor snapshot Grows with the project; media list and subtitle cues capped at 60 each Attached to the newest user message as <editor_state>, rebuilt fresh every turn.
Attachments Metadata JSON per asset <attached_assets> on the message that carried them; on the Gemini path the newest message also carries the actual payloads (frames, images, text contents).
Full state Uncapped The get-state tool — the model calls it when the snapshot is stale or truncated.
The rendered frame One 640px JPEG The capture-frame tool: the composite an export would write at that time — footage, transitions, effects, elements, captions — at the playhead or any time asked for.
The footage itself 36 thumbnails, or 6 whole frames The watch-video tool: the browser decodes candidates on a dense floor and on the lines the source speaks, keeping the frames that differ. A detail ladder sets how many pixels each is worth — thumbnails for coverage, the source's own frames for reading type a mosaic cell destroys. Captions and transcript fuse in as one clock; text that only repeats them is read for treatment, not words. Detect-silence reports dead air.

Folder and note mentions keep their identity in saved chats and resolve on every surface. Each Library file type declares how chat mentions it; a LUT arrives as its set_color_lut call.

The decision layer

Decisions that gate, route, or pick (the turn route, queue triage, stock ranking, filler words, a voice, a provider's failure) are typed judgments from the judge route, thresholded in code. Reasoning and generation stay on the chat model.

The judgment that routes a turn also asks whether the whole turn is one known editor action, and fills its arguments in the same request. When it settles — mute this clip, make it 16:9, slow that one down — the editor runs the tool and writes the line with no model round, landing in about a third of a second. It settles only when the action and every argument clear their floors and four guards hold: the ask names no exact figure, touches no second item, leaves no second edit undone, and wants no shot made again. Anything short runs the ordinary loop, so a miss costs nothing.

Finding the moments in a long source worth clipping is the same shape at scale: a cheap sweep scores every stretch of the transcript, the best earn three closer questions — does it stand alone, does the opening hold, does it finish — and code composes the weights and picks the winners apart from one another. One clip can be two stretches that were never next to each other.

A turn built on footage is judged once more before it closes: the judge reads what it watched, wrote down and ran against the ask, and a turn holding a slice of a source while about to describe the whole of it is handed its next pass instead of its reply. The hold has to point at something the record measures: seconds nobody looked at, or calls that did not land. A doubt with no gap behind it lets the turn close, because a finished turn sent back does its work a second time.

Deciding what the user wants is prompt text, executed by the model. The prompt orders the calls it must make each turn:

  1. Deliverable first. "Write me a caption / a script / a prompt" asks for words — the answer goes in chat and the project stays untouched until the user says "do it". A request to change the project gets acted on directly with tools.
  2. Doing beats asking. Edits are free to reverse (unlimited undo), so the model acts on reasonable interpretations. Generation is the exception: undo removes the clip but credits stay spent, so it generates only when the user asked for the media itself.
  3. Free before paid. Bundled stock is checked before spending generation credits when existing media could serve. The stock sound effects are part of that catalog: a whoosh on a scene change or a click when something pops up is a stock_search for a sound and a stock_add onto the soundtrack, never a generation.

"This" resolves to the current selection, ids come verbatim from the state, and unfamiliar areas trigger a skill read first.

Doing the work

With a tab attached, tool calls execute in it against the same store the user is editing — same state, same selection, same undo history.

  • Readable failure, then retry. Tools validate and clamp their inputs and throw human-readable errors ("No clip with that id — call get_state for current ids"), which return to the model as tool errors it can recover from. A dropped bridge is separate from a bad request: the idempotent tool listing retries on a momentary engine hiccup so the model never lands with an empty tool set, and an "unreachable" call is one the model quietly re-issues, keeping talk of reconnecting out of the reply.
  • One undo step per turn. History batches while the assistant is busy, so ⌘Z reverts the whole turn in one step. A sweep rides the same batch: select_items judges every item on a row against one description in a single fan-out, and the mutating tools take ids, so finding and changing fifty clips is one call each.
  • Small results. Tools return ids and rounded numbers, plus a short note when behavior surprised ("that spot was taken — slid right").
  • Frames become images. A tool result's image/images data URLs leave the JSON and reach the model as real images on every path. Watching happens in whatever model the user is chatting with; there is no side model.
  • Watching leaves a map. A watch records which moments and cut candidates a source holds, persisted on the asset, and the first watch starts a quiet background sweep that maps the rest. The map aims later watches; only sheets returned in the conversation count as footage the model has seen.
  • Watching leaves a record. Contact sheets live in one conversation and the media trim drops the oldest, so the model writes each look down: a note-source call stores what a span showed — quoted on-screen text, what happens, where the cuts land — against its source seconds, on the asset. Notes come back in the editor snapshot as observed and on every watch result beside the spans still undescribed. A close note supersedes the coarse one over the same ground, and a watch of a source that still owes a note is refused — which is what makes an hour-long source workable: surveyed coarsely, described, then re-watched densely only where the work is.
  • A cut can be blocked out before its footage exists. A block is a shot with nothing in it yet: a source that owns no file and draws itself, holding its slot on track 0 for the length that shot runs. Laying a run of them is the ordinary clip placement, so "make one like this" becomes a timeline carrying the reference's own cut times, with titles and sound built on top. Footage dropped on a block takes its place and its length; from chat that is the same item replace any clip uses.
  • Speech cuts land on the words. A recut edge is not left where the model's arithmetic put it. The audio around it is re-read, that recording's own room tone and voice level are measured, and the edge is placed a beat clear of where a word truly starts or finishes — tail and soft attack included, so a quiet take and a noisy one are cut the same way. Pace belongs to that step: fast, natural and relaxed set how big the beat is, and the fastest of them still lets every word finish. So "make it punchy" shortens the air rather than cutting into the speech, and where a pause is too short for both sides' air the two edges meet inside it instead of one clipping a word.
  • Media rides a byte budget. Contact sheets, audio, and attachment payloads all replay inline, so before each call the session is trimmed to a 16MB media budget — oldest parts leave first, with a note naming the tools that fetch them again. Text, tool calls, tool results, and thought signatures always survive, and the trim runs on a copy so the stored session keeps everything.
  • A Donkey Cut link is a document. A project or share link pasted into chat is read as its document, and its edit is rebuilt in the open project on the user's own footage through the Library's template rail, whole or item by item, with the media the project lacks copied across; the reference's document is the recipe, so nothing is reconstructed from pixels, and a project held only in another browser's storage is readable only from that browser.
  • A link comes back as whatever it holds. The chat's URL import works down a ladder: the bundled downloader for anything with a stream, X's own endpoints for a post's photos or Article, and otherwise the page itself — a content extractor (Defuddle over a parsed DOM) lifts the article out as markdown and the pictures it kept come along as assets. Words with no media are a successful import, so a link the user pasted to explain something reads fine, and that text is quoted for the user beside whatever media came with it. Whatever comes down is H.264 by the time it lands: the downloader sorts on codec before resolution, and a source that publishes nothing Apple can decode — the VP9 and AV1 streams Instagram and YouTube serve at their top tiers — is converted on the way in, so an import plays on the phone, in the Mac's own player, and in a browser.
  • Async when rendering. Video generation outruns the two-minute tool window, so those tools start the job and return immediately; the clip files under the chat that asked — the owning thread is captured at call time — and lands on the timeline only when the user asked for that. Each render's card state (running, landed, failed) mirrors into the project doc as it changes, so the chat shows the same outcome on machines that never ran the job.
  • Chat owns what it makes. Media created by chat tools is tagged with its thread and previews on a card in the conversation. Placing it on the timeline or filing it into Media or the Library transfers ownership; deleting the thread deletes whatever it still owns.
  • Chat media is the account's storage. It is real project media: it counts toward the cloud quota and the project's size like an import, and the thread is the only place it can be seen or deleted from. So conversations travel wherever their media travels — a project moved between the Mac and the cloud, an owner's duplicate, and a share copy with chat shared all carry the threads with the files. A copy that took the files alone would hold media nobody can find, delete, or explain, and the user would still be paying for it.
  • Two tools stay server-side. The skill list and skill reads are answered by the engine directly — no browser hop.

Generating a whole video

Most generation tools make one asset — an image, a video clip, a voiceover, or a music track (generate_music, a vocal-free bed by default and a full sung song on request, scene-aware enough to score a whole video, and able to match a track to a reference the user attaches — an audio clip to emulate or a video whose tone it should carry, since the music model reads only text a multimodal pass turns the reference into the sound description it renders from). generate_scene makes a whole cut: it writes a script, breaks it into shots, and — once the user approves — renders each shot and lays them on the timeline. The video model speaks each shot's line, so the clips carry their own narration; a music bed sits under them. It is genre-agnostic; the look comes from the brief and any references. It can also animate audio the project already has (from_audio_asset_id), tiling shots over that spine with no script written.

Because a scene renders many paid shots, the tool plans and then stops. generate_scene returns a shot list and waits; approve_scene starts the renders; regenerate_shot, recut_scene, and restyle_scene revise afterward; cancel_scene kills the active run. A plan created this turn cannot be approved in the same turn — the harness blocks that call whatever the model decides, so the money gate is structural. The plan the user approves is the plan that renders: voicing runs after the gate and only rescales the approved shots to the real voice lengths (a line longer than one clip splits its shot). The scene coordinator lives beside the panels (lib/genScene.ts) and keeps working across project navigation while its page is alive. Its persisted plan (ProjectDoc.genvideo) lets a returning page resume planning or rendering. Each operation acquires project ownership: Web Locks for browser storage, an engine lease for Mac projects, and a database lease for cloud projects. Other tabs follow the saved plan. A resumed scene clears the chat tool's stale interruption, and reaching the storyboard gate settles that call. Failed plans retain their failure until an explicit retry. Nothing runs behind the user's back: cancelling, dismissing an unrendered plan, or deleting the chat thread that asked all stop the run and clear its plan — clips already placed stay on the timeline. Progress shows on a card in the chat while the timeline fills on its own. The shots ride under one consistent AI voice with the video model's own audio off; this version has no talking-head lip-sync.

Context across turns

Threads persist per project in the browser's local storage — the newest 30, titled by their first message. A cloud project also mirrors each thread to the hosted chat routes (Postgres, keyed to the account), merging the server copy on open and periodically while visible, so its history follows the account across devices. Polls read revision metadata and fetch changed conversations. Deletion records prevent late completions and offline copies from restoring removed threads. Storage events update other tabs on the same origin, including their provider sessions. Idle editors also follow saved project revisions, preserving the playhead and waiting while playback, unsaved edits, or editor tools are active. This follows committed snapshots; it does not merge simultaneous edits to the same document. What a returning thread remembers depends on the provider:

  • Claude and Codex hold their own history. Each turn sends only the newest message plus the fresh snapshot; prior turns live in the provider's native session, whose id is saved on the thread and resumed. The engine never replays conversation history itself.
  • Gemini replays its own session. The pi harness keeps the thread's conversation as structured history — real tool calls and their results — and saves it with the thread, so a reload picks it back up. Every call is replayed from that history with only the newest snapshot kept and media trimmed to budget; a thread with no saved session falls back to a text-only replay of what the panel shows.

A thread saves one session slot per provider, so switching models mid-thread keeps each provider's own context. The asymmetry to know: Gemini can pick up any thread because it replays the transcript, while a CLI provider joining a thread it hasn't chatted in starts from only the newest message.

Limits

Limit Value
Tool steps per turn 24, extending to a 96 ceiling
Inline media per model call 16MB, trimmed oldest-first
Browser tool execution 120s
One watch-video call 600s of source, 150 decoded candidates, a 90s decode budget, and 36/16/6 kept frames by detail; the result says where coverage stopped
One blockout call 160 shots, each 1/60-600s
Snapshot caps 60 media items, 60 cues
One queued turn job 1MB of messages, one live turn per project
Saved threads per project 30
CLI availability probe cache 60s

One-shot helpers

Three narrow AI calls run beside the chat, each a single engine round-trip through the user's Claude login on a small model: the caption style rewrite, caption translation, and visual subtitles (sampled frames in, timed narration cues out). The panel buttons and the chat's subtitle tools reach them through the same store actions. The style rewrite falls back to the original lines on failure; translation fails loudly instead, because filling a track with the wrong language would be silent corruption.

A hidden hermetic test provider exercises the full engine bridge — context, tool round-trips, streaming — without spending tokens; end-to-end tests select it as their model.

Measuring the assistant

A live eval scores the chat brain's behavior and speed together. Each case replays a real composer turn against the live model, through the same loop production runs, and asserts what came back: the reply, the tools called, the gate verdict, tools that must stay sheathed, and the reply's shape, with a light-model judge leaving tone notes beside the pass/fail. All of it is timed. Cases carry a bucket — chat turns, single-tool edits, multi-tool composed edits — so the numbers aggregate by the kind of turn a user feels.

The committed report at evals/cut-chat.latest-report.json is the baseline. The improvement loop: make one change — a model role, a prompt line, a tool description — run the eval, and keep it only when pass rates hold and the medians improve; the new report commits with the change, so the baseline always describes what ships. Run it with the dev server up: npm run eval:cut-chat (--matrix --runs 3 compares model configs) and npm run eval:cut-queue for mid-turn triage; the README beside the harness under the site's scripts/lib/cut-eval/ covers the flags, the report schema, and how to add a case. check-chat-continuity.mjs verifies disconnect recovery and replay without tokens, and the tab-less paths have end-to-end scripts that drive a real turn through the headless session and the queued turn job.

Two mechanical paths are guarded without a model at all, because no ear settles them from a preview: caption timing (npm run eval:cut-sync) and speech-cut placement (npm run eval:cut-edges). Both build their own truth — a take of known words rendered to audio — run the production code over it, and enforce budgets on the result. npm run eval:cut-clip measures the clipping judgments against labelled excerpts and then over a talk whose strong and flat stretches are known, asking the questions the tool itself asks.

Where it lives

The shared catalog (system prompt, tools, skills), the chat route with its provider runners, the browser-tool bridge, its tab-less executor (server/ai/headlessTools.ts), and the stdio MCP proxy live with the engine's AI code under the site's cut/ folder (server/ai/, server/http/ai.ts). The page side holds the snapshot builder (lib/aiContext.ts), the tool implementations (lib/aiTools.ts), the pi-based chat loop and its adapters (lib/pi/), the model catalog (lib/aiModels.ts), and the chat panel with threads and transport (components/AiPanel.tsx). Running the same loop with no browser is lib/headless/, the turns route in server/cloud/turns.ts, and the worker job in worker/turnJob.ts.

The brief-to-video pipeline is its own subsystem: the orchestrator, coverage invariant, and role interfaces in lib/genvideo/, its hosted-model adapters in lib/genvideo/adapters/, the browser controller in lib/genScene.ts, and the progress card in components/SceneCard.tsx. Its strategy — story planning, the identity ladder, where audio goes next — is the Brief to Video guide.

Project commands share the editor's tool schemas and executor through PROJECT_TOOLS and runProjectCommand. The command boundary checks the active project and excludes editor-only actions. Headless callers keep the existing serialized engine session or isolated cloud worker, since the editing store belongs to that session. Deterministic editing commands call the executor directly; hosted generation keeps its credit checks. A host with its own model — the ChatGPT app — sends a batch of these commands as a worker job; the ChatGPT App guide covers that path.

render_preview captures the current cut and returns a render job with its revision and playback URL. Cloud and Mac jobs include a status URL for the host to poll. A headless turn can enqueue the render and finish, freeing its worker slot for the render. A Mac session without the media runtime returns the existing detached-media error and can run the command from the editor tab.