Skip to content

Open source CapCut alternative.

TypeScript

docs: agent guide covers the Cut surfaces

David Le7d3dc6742f6c
diff --git a/AGENTS.md b/AGENTS.md
index 42ae7c29..df038558 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -20,6 +20,17 @@ Before changing `site/` UI, routes, API handlers, or data access patterns:
- Keep Prisma table/model definitions out of `site/prisma/schema.prisma`. Put tables in logically grouped sibling `.prisma` files under `site/prisma/`; reserve `schema.prisma` for shared Prisma configuration such as generator and datasource blocks.
- Treat `/prototype`, "the prototype route", or route-shaped prototype requests as work on the Next.js route under `site/`, not as a repository-root `prototype/` directory.
+## Cut Surfaces
+
+Every Cut change has to hold on all four surfaces, and the plan for it says how:
+
+- **Both residencies.** A browser project (OPFS in the page) and a cloud project (Postgres doc + R2 media) get the same behavior. Work through the backend seam in `site/src/cut/lib/backend/`.
+- **Headed and headless.** Whatever the tab can do, the Bun engine and the worker runner can do: chat tools, rendering, media reads. Headless installs browser primitives — canvas, decoders, Web Audio, fonts — behind narrow seams (`lib/raster.ts`, the frame sink in `lib/mediaRead.ts`, the font installer in `lib/fontAssets.ts`, the kit's `surface.ts`) so one implementation serves both; reach for those seams before writing a second path.
+- Allocate canvases and decode images through the raster seam. Direct `document`, `window`, `FileReader`, `createImageBitmap`, or `FontFace` use in `site/src/cut/lib/` or `packages/effects-kit/` breaks a job.
+- When a surface genuinely cannot carry a feature, give it a fallback and say so in the summary and the guide.
+- **The AI chat drives it too.** A functional change ships with its chat surface: a tool the assistant can call, and descriptions/system-prompt lines that teach the new capability. Derive tool schemas and prompt text from the same exported constants the UI uses (style id lists, model registries) so the catalog updates itself; check the tool files beside the component (`*.tools.ts`) and `site/src/cut/server/ai/catalog.ts`.
+- The chat surface is kept true, both directions. Removing or reshaping a feature means deleting its tool and its prompt/skill mentions in the same change — grep the catalog, the `*.tools.ts` files, the skills library, and `lib/aiTools.ts` for the old names, ids, and parameters. A tool that describes behavior the code no longer has, offers an option that no longer exists, or is missing a setting the UI gained is a bug: the model calls what the catalog teaches.
+
## Working Rules
- Do not touch repository-root `prototype/` unless the user explicitly asks for that filesystem path. By default, assume requested product changes are for the Mac app or the site/landing page.
diff --git a/CLAUDE.md b/CLAUDE.md
index 42ae7c29..df038558 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -20,6 +20,17 @@ Before changing `site/` UI, routes, API handlers, or data access patterns:
- Keep Prisma table/model definitions out of `site/prisma/schema.prisma`. Put tables in logically grouped sibling `.prisma` files under `site/prisma/`; reserve `schema.prisma` for shared Prisma configuration such as generator and datasource blocks.
- Treat `/prototype`, "the prototype route", or route-shaped prototype requests as work on the Next.js route under `site/`, not as a repository-root `prototype/` directory.
+## Cut Surfaces
+
+Every Cut change has to hold on all four surfaces, and the plan for it says how:
+
+- **Both residencies.** A browser project (OPFS in the page) and a cloud project (Postgres doc + R2 media) get the same behavior. Work through the backend seam in `site/src/cut/lib/backend/`.
+- **Headed and headless.** Whatever the tab can do, the Bun engine and the worker runner can do: chat tools, rendering, media reads. Headless installs browser primitives — canvas, decoders, Web Audio, fonts — behind narrow seams (`lib/raster.ts`, the frame sink in `lib/mediaRead.ts`, the font installer in `lib/fontAssets.ts`, the kit's `surface.ts`) so one implementation serves both; reach for those seams before writing a second path.
+- Allocate canvases and decode images through the raster seam. Direct `document`, `window`, `FileReader`, `createImageBitmap`, or `FontFace` use in `site/src/cut/lib/` or `packages/effects-kit/` breaks a job.
+- When a surface genuinely cannot carry a feature, give it a fallback and say so in the summary and the guide.
+- **The AI chat drives it too.** A functional change ships with its chat surface: a tool the assistant can call, and descriptions/system-prompt lines that teach the new capability. Derive tool schemas and prompt text from the same exported constants the UI uses (style id lists, model registries) so the catalog updates itself; check the tool files beside the component (`*.tools.ts`) and `site/src/cut/server/ai/catalog.ts`.
+- The chat surface is kept true, both directions. Removing or reshaping a feature means deleting its tool and its prompt/skill mentions in the same change — grep the catalog, the `*.tools.ts` files, the skills library, and `lib/aiTools.ts` for the old names, ids, and parameters. A tool that describes behavior the code no longer has, offers an option that no longer exists, or is missing a setting the UI gained is a bug: the model calls what the catalog teaches.
+
## Working Rules
- Do not touch repository-root `prototype/` unless the user explicitly asks for that filesystem path. By default, assume requested product changes are for the Mac app or the site/landing page.
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.md
index a09d50a6..3cd0ec23 100644
--- a/docs/guides/cut/ai-assistant.md
+++ b/docs/guides/cut/ai-assistant.md
@@ -39,6 +39,8 @@ The turn's route is decided by an intent gate while the turn is already running.
The same loop code runs with no browser at all. A headless session opens a project doc into the editor store, binds both transports at an explicit origin with auth headers, and runs the production chat loop against it — which is how a chat turn becomes a durable job. The page or an API caller posts a thread's messages to the cloud project's turns route; the row is the project's agent write lease (one turn at a time per project); the worker container claims it, runs the turn under a runner grant that spends that user's credits and writes that user's project, pushes the doc back through the versioned PUT, and appends the finished exchange to the stored thread.
+A job can do everything a tab can, because the same tool code runs over swapped primitives. The library draws, decodes and measures through narrow seams — a canvas factory, a frame sink, a font installer — which the page fills with the browser's own and the container fills at startup with a native canvas, native decoders, a Web Audio graph, and the editor's font files staged in the image. So watching footage, cutting on silence, freezing a frame, matting a sticker, writing captions and capturing the composited frame all behave in a job the way they behave in the tab, and parity holds by construction: there is one implementation. Two things stay page-only — on-device MediaPipe matting, which falls through to the hosted vision path, and Lottie stickers, which need a DOM.
+
Two mechanisms keep a turn honest. A mutation ledger, computed in code from the tool results, rides every model call as an ephemeral context note: what ran, what failed, and what debris the edits left (a parked transition bar, an item stranded past the video's end) — the model must fix or plainly report each entry before its reply lands, and the note is never stored or rendered. And the round budget extends itself: a build that outgrows its 24 rounds gets fresh budget automatically (up to three extensions, so 96 rounds in all), so the user never types "keep going". At the ceiling tool calls stop and the model gets one round to say where it landed; a turn still reaching for tools ends with a line offering "keep going".
The composer stays open while a turn runs. A message sent mid-turn waits in a small tray on the composer's top edge and goes out when the running turn settles — one turn at a time, each with a fresh editor snapshot, so a queued ask always sees the edits made before it. The tray is the review surface: waiting rows can be reordered, rewritten, or removed, and the queue pauses and resumes from its header. While a row is being rewritten the list holds still — the row under edit waits, the others keep draining and cross out in place as they finish. Stopping a turn or hitting an error pauses the queue automatically, so the remaining asks hold for review. Waiting rows are saved with the thread on the machine: a reload or a trip to another project brings them back paused, ready to resume.
@@ -57,7 +59,7 @@ Every knowledge surface is defined once — the catalog file ships in the engine
| Editor snapshot | Grows with the project; media list and subtitle cues capped at 60 each | Attached to the newest user message as `<editor_state>`, rebuilt fresh every turn. |
| Attachments | Metadata JSON per asset | `<attached_assets>` on the message that carried them; on the Gemini path the newest message also carries the actual payloads (frames, images, text contents). |
| Full state | Uncapped | The get-state tool — the model calls it when the snapshot is stale or truncated. |
-| The rendered frame | One 640px JPEG | The capture-frame tool, for visual judgment at the playhead. |
+| The rendered frame | One 640px JPEG | The capture-frame tool: the composite an export would write at that time — footage, transitions, effects, elements, captions — at the playhead or any time asked for. |
| The footage itself | Up to four 3×3 contact sheets per call | The watch-video tool: the browser decodes candidates on a dense steady floor, keeps only the frames that differ, tiles them into sheets stamped with their source times, and reports refined hard-cut times. Any captions or transcript fuse in as one clock, so each kept frame sits inside the speech it belongs to. Detect-silence reports dead air the same way, numbers only. |
The system prompt carries the voice, the deliverable rule (below), id discipline, the undo-versus-credits asymmetry, and pointers into the skills. The skills carry the deep per-area documentation — editor overview, timeline editing, watching and cutting by content, transitions and fades, graphics (titles, shapes, stickers, composite builds), audio and subtitles, stock and generation, scene productions, editing taste, media and library, publish and export — so the always-on cost stays near 21K tokens and detail is pulled only when the model works in that area.