The VS Code of video editing, built for agents.
TypeScript
Update documentation to remove references to the `sync` command in asset management and clarify the new `syncTo` feature for audio alignment. Adjusted README, API specifications, and composition guidelines for consistency and accuracy.
konstantin-paulus8f8c35d67cc0
diff --git a/README.md b/README.mdindex 0099a05f..1d50a1cc 100644--- a/README.md+++ b/README.md@@ -65,7 +65,6 @@ dapi asset frame clip.mp4 -t 0 12 45 # decode frames to PNGsdapi asset visualize track.mp3 # waveform / filmstrip previewsdapi asset transcript interview.wav # timed, word-level transcriptdapi asset analyze b-roll.mp4 -p "when is the product shown?" # ask a vision model-dapi asset sync lav.wav -v camera.mp4 # align external audio to camera audiodapi node screenshot # see the canvas itself```@@ -77,7 +76,7 @@ dapi node screenshot # see the canvas itself| `dapi context` | Summary of the open project: scenes, playhead, fonts || `dapi mount` | Compile and mount a JSX composition || `dapi node …` | The scene graph: `ls`, `tree`, `grep`, `patch`, `insert`, `cp`, `rm`, `screenshot`, `export` |-| `dapi asset …` | The media library: `add`, `ls`, `probe`, `frame`, `visualize`, `analyze`, `transcript`, `sync`, `export` |+| `dapi asset …` | The media library: `add`, `ls`, `probe`, `frame`, `visualize`, `analyze`, `transcript`, `export` || `dapi project …` / `dapi folder …` / `dapi selection …` | Projects, library folders, canvas selection || `dapi models` / `dapi voices` / `dapi fonts` | Discover generation models, speech voices, local fonts |diff --git a/apps/cli/api-spec.md b/apps/cli/api-spec.mdindex 2d4c55b1..95a0dd5e 100644--- a/apps/cli/api-spec.md+++ b/apps/cli/api-spec.md@@ -294,6 +294,7 @@ Any live entity is addressable — not just nodes: paints and color stops can beAll props are optional; a patch may set any subset. An unsupported property or an out-of-range value rejects that entity; a rejected patch applies none of its props (no half-applied patches). Notes beyond the JSX table:- `fill` recolors the entity's existing solid fill(s), or creates one if the entity has none.- `src` takes a path, URL, or asset id (`generate.*` declarations are JSX-only). The source resolves asynchronously after the other props land — if resolution fails, the entity is reported rejected even though its other props were applied; patch `src` on its own to avoid ambiguity.+ - `syncTo` re-runs audio alignment against the document node carrying the given key and writes the entity's `startTime` from the measured offset (see [jsx-spec.md](./jsx-spec.md#audio-sync)). The command blocks until alignment settles; a failed alignment rejects the entity.- Element applicability is not checked: props are applied wherever `mount` would apply them (e.g. `fontSize` only makes sense on a text node).- **Output:** JSON Lines, one per input id```ts@@ -595,28 +596,6 @@ Decodes one or more frames of a video asset at the given times and writes each t----### `dapi asset sync <audioId|path>`--Measures the time offset between a standalone audio recording and the camera audio embedded in a video — for aligning a separate voice/lav recording to the in-camera sound of the same take. Cross-correlates the two audio signals and returns the offset plus a confidence score. **Read-only** — it reports the offset and leaves it to the caller to act on (position the clip with `node` commands, trim, mute the camera track). Renders locally; no credits. Alias: `align`.--- **Input:**- - `<audioId|path>` — the audio (or video) asset to align: an id or a local file to add (required). Its audio track is the signal being measured.- - `-v, --video <id|path>` — the reference video asset whose camera audio to align against: an id or a local file to add (required).-- **Output:** one JSON object — the measured alignment:- ```ts- {- audioId: string; // the aligned asset- videoId: string; // the reference video- offsetSeconds: number; // sign convention: place the audio at- // audioStart = videoStart + offsetSeconds- // positive = recording started AFTER the camera rolled- confidence: number; // 0..1, normalized cross-correlation peak (clear match ≳ 0.9)- }- ```-- **Errors:** exits non-zero if either id is unknown, either asset has no decodable audio track, or the two clips share too little common audio to align (`No reliable alignment found`).------## FoldersAll under `dapi folder` (alias `dapi fld`). Operate on the open project. Folders organize the asset library and nest arbitrarily; assets reference their containing folder via `folderId` (`null` = library root). Everywhere a folder id is optional, omitting it means the library root — there is no sentinel value for the root. Read the full hierarchy (folders and assets together) with [`asset ls`](#dapi-asset-ls); move assets between folders with [`asset mv`](#dapi-asset-mv-ids---to-folderid).diff --git a/apps/cli/jsx-spec.md b/apps/cli/jsx-spec.mdindex c35a830f..fb938369 100644--- a/apps/cli/jsx-spec.md+++ b/apps/cli/jsx-spec.md@@ -33,6 +33,10 @@ component tree renders into it.6. **Generate** — declared assets generate in dependency order, **blocking thecommand** until every one has landed. Each placeholder renders a generatingstate until its asset lands, then the node's paint is attached.+7. **Sync** — nodes declaring `syncTo` are aligned once every generated asset+ has landed: each node's audio is cross-correlated against its target's and+ its `startTime` is written from the measured offset (see+ [Audio sync](#audio-sync)). Local and blocking; captions wait for it.## Project module contract@@ -182,12 +186,14 @@ alpha is ignored — use `opacity`). Takes only [paint children](#paints); use| `src` | `string \| AssetRef` | **required** | See [Media source resolution](#media-source-resolution). || `objectFit` | `"cover" \| "contain" \| "fill"` | `"cover"` | How the source maps into the box. || `volume` | `number` | `1` | `0`–`1`; `1` = unity gain. |+| `muted` | `boolean` | `false` | Excludes the node's audio from the mix; independent of `volume`. |+| `syncTo` | `string` | — | Key of another element carrying audio; derives `startTime` by audio alignment (see [Audio sync](#audio-sync)). Mutually exclusive with `startTime`. |**`<image>`** — common props plus `src` (**required**) and `objectFit` (default`"contain"`).-**`<audio>`** — `name`, timing props, `src` (**required**), `volume`. No spatial-props; no visual output.+**`<audio>`** — `name`, timing props, `src` (**required**), `volume`, `muted`,+`syncTo` (see [Audio sync](#audio-sync)). No spatial props; no visual output.**`<text>`** — common props plus:@@ -277,6 +283,8 @@ Timing props follow Lottie-inspired semantics. All values are- `startTime` controls **which part of the source plays inside that window**. It isthe composition time where source frame 0 sits, so it may be negative to skip intothe source.+- Instead of declaring `startTime`, a media node can derive it from another node's+ audio with `syncTo` (see [Audio sync](#audio-sync)).> Example: `inPoint={0} outPoint={16} startTime="-30f"` shows the clip from> composition second 0 to 16, but because source-time-0 is placed 30 frames *before*@@ -313,6 +321,52 @@ own; it lays its children out back-to-back in document order.</sequence>```+## Audio sync++`syncTo` places a node in time by listening instead of arithmetic: the node's+audio is cross-correlated against another node's audio, and its `startTime` is+computed so the two recordings coincide on the timeline. It replaces manual+offset measurement for multi-recorder material: a lav or voice track against+camera audio, two cameras on the same take, two microphones in one room.++```tsx+<scene key="talk" width={1920} height={1080}>+ <video key="camera" src="/Movies/take-3.mp4" inPoint={0} outPoint={45} muted />+ <audio src="/Movies/lav.wav" syncTo="camera" />+</scene>+```++- `syncTo` names the `key` of another element in the same render. Both sides+ must carry an audio track; any pairing works (audio-to-video, audio-to-audio,+ video-to-video).+- The computed placement is `startTime = target.startTime + offset`, where+ `offset` is the measured source-time offset between the two recordings+ (positive when this node's recording started after the target's; possibly+ negative). `syncTo` and `startTime` are mutually exclusive.+- `inPoint`/`outPoint` keep their normal meaning and remain yours to set. When+ omitted on a synced node, the window defaults to the intersection of the+ node's natural extent with the target's window (instead of the usual+ natural-duration fit), so a lav track simply covers its take.+- Alignment reads source content: `muted` and `volume` on either side do not+ affect the measurement. Use `muted` to keep only one side audible, as on the+ camera track above.+- Chains resolve in dependency order (A may sync to B while B syncs to C).+ Unknown keys, cycles, and combining `syncTo` with `startTime` are mount+ errors; nothing is inserted.++Alignment runs at the [sync stage](#pipeline) of the pipeline: after generated+assets land (either side may be generated), before captions read the scene. It+is local, consumes no credits, and blocks the command, so exit `0` means final+placement. Each resolved node logs `{ offsetSeconds, confidence }` on stderr; a+clear match scores above ~0.9, and a correlation too weak to trust fails the+command (see [Errors](#errors)) while the node keeps its default placement.+Offsets are **cached** by the pair of source contents, so re-mounting an+unchanged project re-measures nothing.++Because `syncTo` is part of the shared property table, `dapi node patch`+accepts it too: patching `syncTo` onto an existing node re-aligns it against+the document node carrying that key.+## Captions`<captions />` inside a scene transcribes that scene's audio into a caption node — a@@ -321,9 +375,11 @@ insert one with `dapi node insert.Transcription is **asynchronous and non-blocking**: the caption node is inserted atcommit and its transcript attaches once ready. Because it reads the scene's audio,-it runs **after** any generated assets in the scene have landed — so captioning a-generated `voice`/`audio` track transcribes the finished audio. The scene must-contain an unmuted audio or video source; otherwise the caption node is left empty.+it runs **after** any generated assets in the scene have landed and after+[audio sync](#audio-sync) has resolved — so captioning a generated+`voice`/`audio` track transcribes the finished audio at its final placement. The+scene must contain an unmuted audio or video source; otherwise the caption node+is left empty.Transcripts are **cached**: every transcript asset records a fingerprint of thescene's audible mix (source content, placement, source offset, playback rate,@@ -474,8 +530,9 @@ one-shot contract above is deliberately forward-compatible with it.| Stage | Where it surfaces | Effect || ----- | ----------------- | ------ || Compile (syntax/type-stripping/bundling) | CLI stderr, exit 1 | App never contacted. |-| Evaluate / mount (thrown errors, invalid props, invalid root — e.g. nested `<scene>`, a root element without `key`, missing `src`, malformed `Time`) | CLI stderr, exit 1 | Staging root discarded; **nothing inserted**. |+| Evaluate / mount (thrown errors, invalid props, invalid root — e.g. nested `<scene>`, a root element without `key`, missing `src`, malformed `Time`, an unknown or cyclic `syncTo` key, `syncTo` combined with `startTime`) | CLI stderr, exit 1 | Staging root discarded; **nothing inserted**. || Generation (per-model constraints: `aspectRatio`, `duration`, feature flags) | CLI stderr, exit 1, after every generation settled | No rollback — the mounted tree stays committed; the affected placeholder stops showing its generating state and is left without a paint. |+| Sync (a side without a decodable audio track, or no reliable alignment) | CLI stderr, exit 1, after every alignment settled | No rollback; the tree stays committed and the node keeps its default placement. |Runtime errors are mapped back to the source via inline sourcemaps produced atcompile time.diff --git a/docs/assets.md b/docs/assets.mdindex 675076b5..9d6fd29a 100644--- a/docs/assets.md+++ b/docs/assets.md@@ -1,6 +1,6 @@# Working with media-The asset library holds a project's media. Beyond storage, the CLI ships an inspection toolchain (probe, decode, visualize, transcribe, analyze, align) so an agent can *look at* footage before cutting it. All of it runs locally except `analyze` and `transcript`, which use hosted models.+The asset library holds a project's media. Beyond storage, the CLI ships an inspection toolchain (probe, decode, visualize, transcribe, analyze) so an agent can *look at* footage before cutting it. All of it runs locally except `analyze` and `transcript`, which use hosted models.**Id or path:** every inspection command takes either an asset id or a local file path. A path is imported into the library first, then inspected, so `dapi asset probe clip.mp4` works on a file that isn't in the project yet.@@ -69,13 +69,6 @@ dapi asset analyze <id|path> [-p <prompt>] [-s <start>] [-e <end>]Puts a multimodal model in front of an image, video, or audio asset. Without a prompt, returns a general description; with one, answers it ("what's the dominant color?", "when is the product shown?"). `-s`/`-e` scope the analyzed segment; timestamps in the answer are relative to `-s`.-### `asset sync`--```sh-dapi asset sync <audioId|path> -v <videoId|path>-```--Measures the offset between a standalone recording (a lav or voice track) and the camera audio of the same take, by cross-correlation. Prints `{ offsetSeconds, confidence }` with the convention `audioStart = videoStart + offsetSeconds`; a clear match scores above ~0.9. Read-only: position the clip yourself with the reported offset.## A typical agent passdiff --git a/docs/cli.md b/docs/cli.mdindex 2e648cfd..da3f0aae 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -176,7 +176,7 @@ dapi project rm <id> # delete## Assets and folders-The library commands (`asset add/ls/mv/rm/export`, `folder ls/create/rename/mv/rm`) and the media-inspection commands (`probe`, `transcript`, `analyze`, `visualize`, `frame`, `sync`) are covered in [Working with media](assets.md).+The library commands (`asset add/ls/mv/rm/export`, `folder ls/create/rename/mv/rm`) and the media-inspection commands (`probe`, `transcript`, `analyze`, `visualize`, `frame`) are covered in [Working with media](assets.md).## Discoverydiff --git a/docs/composition.md b/docs/composition.mdindex c866a728..d2526161 100644--- a/docs/composition.md+++ b/docs/composition.md@@ -65,9 +65,9 @@ User-defined components are ordinary Solid components; only intrinsic elements p- **`<scene>`**: `key`, `width`, `height` (all required), `name`, `fill`. No timing or transform props.- **`<rect>` / `<group>`**: `fill` (any CSS color; alpha is ignored, use `opacity`).-- **`<video>`**: `src` (required), `objectFit` (`"cover"` | `"contain"` | `"fill"`, default `"cover"`), `volume` (`0`–`1`).+- **`<video>`**: `src` (required), `objectFit` (`"cover"` | `"contain"` | `"fill"`, default `"cover"`), `volume` (`0`–`1`), `muted`, `syncTo` (see [Audio sync](#audio-sync)).- **`<image>`**: `src` (required), `objectFit` (default `"contain"`).-- **`<audio>`**: `src` (required), `volume`, timing props.+- **`<audio>`**: `src` (required), `volume`, `muted`, `syncTo`, timing props.- **`<text>`**: string children (required), `fontFamily` (see `dapi fonts`), `fontSize`, `fontWeight`, `fontStyle`, `fill`, `textAlign`, `textBaseline`.These same props are what `dapi node patch` assigns on existing nodes.@@ -97,6 +97,17 @@ Composition-relative, Lottie-inspired. Values take any [time format](cli.md#time`<sequence>` removes the arithmetic for back-to-back cuts: children are placed sequentially and non-overlapping in document order.+### Audio sync++`syncTo` computes a node's `startTime` by cross-correlating its audio against another element's audio (named by `key`), so two recordings of the same take coincide on the timeline: a lav track against camera audio, two cameras, two microphones. Any pairing with audio tracks works (audio-to-video, audio-to-audio, video-to-video). `muted` silences the side that shouldn't be heard; alignment reads source content regardless of `muted` or `volume`.++```tsx+<video key="camera" src="/Movies/take-3.mp4" inPoint={0} outPoint={45} muted />+<audio src="/Movies/lav.wav" syncTo="camera" />+```++`syncTo` and `startTime` are mutually exclusive; `inPoint`/`outPoint` stay yours, and when omitted the window defaults to the overlap with the target's window. Alignment runs after generated assets land (either side may be generated), locally and cached, and blocks the mount; each aligned node logs its offset and a confidence score (a clear match is above ~0.9).+## PaintsA node's fill is a **paint child**; the `fill` prop is shorthand for a solid paint. Declaring paints as children unlocks gradients and stacking (later paints render on top, including over the media paint of a `<video>` or `<image>`):@@ -137,7 +148,7 @@ Declarations are pure: nothing generates until the mount commits, and refs never## Captions-`<captions />` inside a scene transcribes that scene's audio into a styled, timed caption node. It runs after generated assets land (so it can caption generated voice-over), attaches asynchronously, and is cached against the scene's audible mix, so re-mounting with unchanged audio doesn't re-transcribe.+`<captions />` inside a scene transcribes that scene's audio into a styled, timed caption node. It runs after generated assets land and [audio sync](#audio-sync) resolves (so it can caption generated voice-over at its final placement), attaches asynchronously, and is cached against the scene's audible mix, so re-mounting with unchanged audio doesn't re-transcribe.```tsx<captions preset="spotlight" colors={["#FF0055"]} />@@ -177,4 +188,4 @@ The CLI strips types without checking them; run `tsc --noEmit` for type safety.## Errors-Compile errors surface on stderr before the app is contacted. Evaluate/mount errors (invalid props, a root without `key`, nested `<scene>`, …) abort with **nothing inserted**. Generation errors surface after all generations settle; the mounted tree stays committed and the affected placeholder is left without media. Runtime errors map back to your source via sourcemaps.+Compile errors surface on stderr before the app is contacted. Evaluate/mount errors (invalid props, a root without `key`, nested `<scene>`, an unknown or cyclic `syncTo` key, …) abort with **nothing inserted**. Generation errors surface after all generations settle; the mounted tree stays committed and the affected placeholder is left without media. Audio-sync failures (no decodable audio, no reliable alignment) surface the same way; the node keeps its default placement. Runtime errors map back to your source via sourcemaps.diff --git a/docs/getting-started.md b/docs/getting-started.mdindex 828af947..bac7dced 100644--- a/docs/getting-started.md+++ b/docs/getting-started.md@@ -75,4 +75,4 @@ Renders the active scene locally with sensible defaults (MP4, H.264, 1080p). Cod- [CLI reference](cli.md): the full command surface- [Writing compositions](composition.md): timing, paints, generative assets, captions-- [Working with media](assets.md): probe, transcribe, analyze, and align footage+- [Working with media](assets.md): probe, transcribe, and analyze footage