The VS Code of video editing, built for agents.
TypeScript
Refactor CLI commands and documentation to replace `export` with `render` for scene rendering. Updated README, API specifications, and various documentation files to reflect this change and clarify the new `syncTo` feature for audio alignment.
konstantin-paulusaac0dad45eb5
diff --git a/README.md b/README.mdindex 1d50a1cc..68e414cd 100644--- a/README.md+++ b/README.md@@ -7,7 +7,7 @@ Diffusion Studio is a canvas-based video editor. What makes it different is `dap```shdapi open # use -b to run the editor headlessdapi mount hero.tsx # render a composition into it-dapi node export -o hero.mp4 # encode the scene to disk+dapi node render -o hero.mp4 # encode the scene to disk```## Compositions as codediff --git a/apps/cli/api-spec.md b/apps/cli/api-spec.mdindex 95a0dd5e..cc7f062b 100644--- a/apps/cli/api-spec.md+++ b/apps/cli/api-spec.md@@ -317,11 +317,11 @@ Deep-clones one or more nodes, including all descendants. Alias: `duplicate`.----### `dapi node export [id] ([config] | --json <str>)`+### `dapi node render [id] ([config] | --json <str>)`-Renders a scene to a video file. Composes the scene frame-by-frame and muxes the result, writing it to disk. Operates on one scene — scenes are nodes, so this is the node-level render. If `[id]` is omitted, exports the active scene.+Renders a scene to a video file. Composes the scene frame-by-frame and muxes the result, writing it to disk. Operates on one scene — scenes are nodes, so this is the node-level render. If `[id]` is omitted, renders the active scene.-Encode settings are passed as a single JSON config — the same object the in-app render method takes (`EncoderConfig`), minus the runtime-only fields the CLI fills in for you (`scene`, `target`, `onProgress`). The whole config is **optional**; omit it to render with the defaults below. The encode window follows the scene's timeline workarea (its `Trim`): export starts at the workarea start and runs to the scene's end, unless capped earlier with `trim.end`. Renders locally; **no credits** and no authenticated account required.+Encode settings are passed as a single JSON config — the same object the in-app render method takes (`EncoderConfig`), minus the runtime-only fields the CLI fills in for you (`scene`, `target`, `onProgress`). The whole config is **optional**; omit it to render with the defaults below. The encode window follows the scene's timeline workarea (its `Trim`): the render starts at the workarea start and runs to the scene's end, unless capped earlier with `trim.end`. Renders locally; **no credits** and no authenticated account required.Like the generators, this is **long-running** — the CLI blocks until the render finishes (or fails) before printing. While waiting, a progress spinner with elapsed time is shown on **stderr** (a single static line when stderr isn't a TTY), so stdout stays clean JSON for piping.@@ -329,7 +329,7 @@ Like the generators, this is **long-running** — the CLI blocks until the rende- `[id]` — scene node id (optional; defaults to the active scene).- `-o, --output <path>` — write the video here (optional; default a uuid-named file in the system temp directory). The container extension is derived from the config's `format`.- Encode config (optional) — at most one of:- - `[config]` — path to a `.json` file containing the config object. Node ids are integers, so a lone non-numeric positional is read as the config path (`dapi node export encode.json` works without an id).+ - `[config]` — path to a `.json` file containing the config object. Node ids are integers, so a lone non-numeric positional is read as the config path (`dapi node render encode.json` works without an id).- `--json <str>` — the object inlineConfig shape (every field optional; all defaults shown):diff --git a/apps/cli/jsx-spec.md b/apps/cli/jsx-spec.mdindex fb938369..0f37e732 100644--- a/apps/cli/jsx-spec.md+++ b/apps/cli/jsx-spec.md@@ -357,11 +357,10 @@ camera audio, two cameras on the same take, two microphones in one room.Alignment runs at the [sync stage](#pipeline) of the pipeline: after generatedassets land (either side may be generated), before captions read the scene. Itis local, consumes no credits, and blocks the command, so exit `0` means final-placement. Each resolved node logs `{ offsetSeconds, confidence }` on stderr; a-clear match scores above ~0.9, and a correlation too weak to trust fails the-command (see [Errors](#errors)) while the node keeps its default placement.-Offsets are **cached** by the pair of source contents, so re-mounting an-unchanged project re-measures nothing.+placement. A correlation too weak to trust fails the command (see+[Errors](#errors)) while the node keeps its default placement. Offsets are+**cached** by the pair of source contents, so re-mounting an unchanged project+re-measures nothing.Because `syncTo` is part of the shared property table, `dapi node patch`accepts it too: patching `syncTo` onto an existing node re-aligns it againstdiff --git a/apps/cli/src/cli-channels.ts b/apps/cli/src/cli-channels.tsindex 12763118..93eb666b 100644--- a/apps/cli/src/cli-channels.ts+++ b/apps/cli/src/cli-channels.ts@@ -33,7 +33,6 @@ export const CLI_CHANNELS = {ASSET_TRANSCRIPT: "cli:asset:transcript",ASSET_VISUALIZE: "cli:asset:visualize",ASSET_ANALYZE: "cli:asset:analyze",- ASSET_SYNC: "cli:asset:sync",SELECTION_LIST: "cli:selection:list",SELECTION_SET: "cli:selection:set",SELECTION_FOCUS: "cli:selection:focus",@@ -46,7 +45,7 @@ export const CLI_CHANNELS = {NODE_DELETE: "cli:node:delete",NODE_PATCH: "cli:node:patch",NODE_DUPLICATE: "cli:node:duplicate",- NODE_EXPORT: "cli:node:export",+ NODE_RENDER: "cli:node:render",PROJECT_ACTIVE: "cli:project:active",PROJECT_LIST: "cli:project:list",PROJECT_CREATE: "cli:project:create",@@ -102,6 +101,10 @@ export type MountRequest = {code: string;};+export type MountResult =+ | { status: "fulfilled" }+ | { status: "rejected"; error: string };+export type NodeInsertRequest = {code: string;parentId: number;@@ -137,8 +140,8 @@ export type EncoderConfigInput = {trim?: { end?: number }; // seconds; caps the encode end};-export type NodeExportRequest = { id?: number; output: string; config?: EncoderConfigInput };-export type NodeExportResult = { path: string };+export type NodeRenderRequest = { id?: number; output: string; config?: EncoderConfigInput };+export type NodeRenderResult = { path: string };export type AssetProbeRequest = { id: string };@@ -158,16 +161,6 @@ export type AssetVisualizeResult = {export type AssetAnalyzeRequest = { id: string; prompt?: string; start?: number; end?: number };export type AssetAnalyzeResult = { id: string; analysis?: string; start?: number; end?: number };-export type AssetSyncRequest = { audioId: string; videoId: string };-export type AssetSyncResult = {- audioId: string;- videoId: string;- // Place the audio at: audioStart = videoStart + offsetSeconds.- // Positive = the recording started after the camera rolled.- offsetSeconds: number;- confidence: number; // 0..1 normalized cross-correlation peak; clear match ≳ 0.9-};-export type GeneratedAsset = { id: string; name: string; type: string };export type FolderInfo = { id: string; name: string; type: "folder" };@@ -272,10 +265,6 @@ export type CliRequestMap = {request: AssetAnalyzeRequest;response: AssetAnalyzeResult;};- [CLI_CHANNELS.ASSET_SYNC]: {- request: AssetSyncRequest;- response: AssetSyncResult;- };[CLI_CHANNELS.SELECTION_LIST]: { request: void; response: NodeRef[] };[CLI_CHANNELS.SELECTION_SET]: {request: { ids: number[] };@@ -300,11 +289,11 @@ export type CliRequestMap = {};[CLI_CHANNELS.MOUNT]: {request: MountRequest;- response: void;+ response: MountResult;};[CLI_CHANNELS.NODE_INSERT]: {request: NodeInsertRequest;- response: void;+ response: MountResult;};[CLI_CHANNELS.NODE_DELETE]: {request: { ids: number[] };@@ -318,9 +307,9 @@ export type CliRequestMap = {request: { ids: number[] };response: NodeDuplicateResult[];};- [CLI_CHANNELS.NODE_EXPORT]: {- request: NodeExportRequest;- response: NodeExportResult;+ [CLI_CHANNELS.NODE_RENDER]: {+ request: NodeRenderRequest;+ response: NodeRenderResult;};[CLI_CHANNELS.PROJECT_ACTIVE]: {request: void;diff --git a/apps/cli/src/cli-client.ts b/apps/cli/src/cli-client.tsindex 2043621b..01b71184 100644--- a/apps/cli/src/cli-client.ts+++ b/apps/cli/src/cli-client.ts@@ -96,8 +96,6 @@ export const cliAPI = {cliRequest(CLI_CHANNELS.ASSET_VISUALIZE, { id, start, end, scale }),assetAnalyze: (id: string, prompt?: string, start?: number, end?: number) =>cliRequest(CLI_CHANNELS.ASSET_ANALYZE, { id, prompt, start, end }, GENERATE_TIMEOUT_MS),- assetSync: (audioId: string, videoId: string) =>- cliRequest(CLI_CHANNELS.ASSET_SYNC, { audioId, videoId }, GENERATE_TIMEOUT_MS),listSelection: () => cliRequest(CLI_CHANNELS.SELECTION_LIST, undefined),setSelection: (ids: number[]) => cliRequest(CLI_CHANNELS.SELECTION_SET, { ids }),focusSelection: () => cliRequest(CLI_CHANNELS.SELECTION_FOCUS, undefined),@@ -111,8 +109,8 @@ export const cliAPI = {deleteNodes: (ids: number[]) => cliRequest(CLI_CHANNELS.NODE_DELETE, { ids }),patchNodes: (patches: NodePatch[]) => cliRequest(CLI_CHANNELS.NODE_PATCH, { patches }),duplicateNodes: (ids: number[]) => cliRequest(CLI_CHANNELS.NODE_DUPLICATE, { ids }),- exportNode: (output: string, id?: number, config?: EncoderConfigInput) =>- cliRequest(CLI_CHANNELS.NODE_EXPORT, { id, output, config }, GENERATE_TIMEOUT_MS),+ renderNode: (output: string, id?: number, config?: EncoderConfigInput) =>+ cliRequest(CLI_CHANNELS.NODE_RENDER, { id, output, config }, GENERATE_TIMEOUT_MS),activeProject: () => cliRequest(CLI_CHANNELS.PROJECT_ACTIVE, undefined),listProjects: () => cliRequest(CLI_CHANNELS.PROJECT_LIST, undefined),createProject: (name?: string) => cliRequest(CLI_CHANNELS.PROJECT_CREATE, { name }),diff --git a/apps/cli/src/index.ts b/apps/cli/src/index.tsindex 5666076b..2d9a0e38 100644--- a/apps/cli/src/index.ts+++ b/apps/cli/src/index.ts@@ -524,22 +524,6 @@ async function assetAnalyze(ref: string, opts: AssetAnalyzeOptions): Promise<voi}}-type AssetSyncOptions = { video?: string };--async function assetSync(audioRef: string, opts: AssetSyncOptions): Promise<void> {- const audioId = await resolveAssetRef(audioRef);- const videoId = await resolveAssetRef(opts.video!);- const stop = startSpinner("Measuring audio offset");- try {- const result = await cliAPI.assetSync(audioId, videoId);- stop();- console.log(JSON.stringify(result));- } catch (e) {- stop();- handleSocketError(e);- }-}-type AssetVisualizeOptions = { start?: string; end?: string; scale?: string; output?: string };function parseTimeArg(value: string, flag: string): number {@@ -625,8 +609,12 @@ async function mountProject(path: string | undefined, opts: MountOptions): Promiconst stop = startSpinner("Mounting project");try {- await cliAPI.mount({ code });+ const result = await cliAPI.mount({ code });stop();+ if (result.status === "rejected") {+ console.error(result.error);+ process.exit(1);+ }} catch (e) {stop();handleSocketError(e);@@ -652,8 +640,12 @@ async function nodeInsert(parentId: string, path: string | undefined, opts: Nodeconst stop = startSpinner("Inserting entities");try {- await cliAPI.insertNode({ code, parentId: eid, index });+ const result = await cliAPI.insertNode({ code, parentId: eid, index });stop();+ if (result.status === "rejected") {+ console.error(result.error);+ process.exit(1);+ }} catch (e) {stop();handleSocketError(e);@@ -751,12 +743,12 @@ async function duplicateNodes(ids: string[]): Promise<void> {}}-type NodeExportOptions = JsonPayloadOptions & { output?: string };+type NodeRenderOptions = JsonPayloadOptions & { output?: string };-async function nodeExport(+async function nodeRender(idArg: string | undefined,configArg: string | undefined,- opts: NodeExportOptions,+ opts: NodeRenderOptions,): Promise<void> {// Node ids are integers, so a lone non-numeric positional is the config file.let id = idArg;@@ -780,7 +772,7 @@ async function nodeExport(const stop = startSpinner("Rendering scene");try {- const { path } = await cliAPI.exportNode(output, eid, config);+ const { path } = await cliAPI.renderNode(output, eid, config);stop();console.log(JSON.stringify({ path }));} catch (e) {@@ -1051,14 +1043,6 @@ asset.option("-e, --end <time>", `end of the segment to analyze — seconds, "45f" frames, or "MM:SS" (default: asset duration)`).action((ref: string, opts: AssetAnalyzeOptions) => assetAnalyze(ref, opts));-asset- .command("sync")- .alias("align")- .description("Measure the time offset that aligns an audio recording with a video's camera audio")- .argument("<audioId|path>", "the audio (or video) asset to align: an asset id or a local file to add")- .requiredOption("-v, --video <id|path>", "the reference video asset to align against: an asset id or a local file to add")- .action((audioRef: string, opts: AssetSyncOptions) => assetSync(audioRef, opts));-const folder = program.command("folder").alias("fld")@@ -1191,14 +1175,14 @@ node.action((path: string | undefined, opts: JsonPayloadOptions) => patchNodes(path, opts));node- .command("export")+ .command("render").description("Render a scene to a video file").argument("[id]", "scene node id (optional; defaults to the active scene)").argument("[config]", "path to a .json encode config (EncoderConfig)").option("-o, --output <path>", "write the video here (default: a temp file)").option("--json <str>", "inline JSON encode config (EncoderConfig)")- .action((id: string | undefined, config: string | undefined, opts: NodeExportOptions) =>- nodeExport(id, config, opts));+ .action((id: string | undefined, config: string | undefined, opts: NodeRenderOptions) =>+ nodeRender(id, config, opts));const project = program.command("project")diff --git a/apps/web/src/components/engine/utils/audio-sync.ts b/apps/web/src/components/engine/utils/audio-sync.tsindex 0b93c156..4cc1254e 100644--- a/apps/web/src/components/engine/utils/audio-sync.ts+++ b/apps/web/src/components/engine/utils/audio-sync.ts@@ -144,6 +144,18 @@ function prefixSums(x: Float32Array): { sum: Float64Array; sumSq: Float64Array }return { sum, sumSq };}+const offsetCache = new Map<string, Promise<AudioSyncResult>>();++export function computeAudioSyncOffsetCached(a: Asset, b: Asset): Promise<AudioSyncResult> {+ const key = `${a.hash}:${b.hash}`;+ const cached = offsetCache.get(key);+ if (cached) return cached;+ const promise = computeAudioSyncOffset(a, b);+ offsetCache.set(key, promise);+ promise.catch(() => offsetCache.delete(key));+ return promise;+}+export async function computeAudioSyncOffset(audioAsset: Asset,videoAsset: Asset,diff --git a/apps/web/src/context/electron/assets.ts b/apps/web/src/context/electron/assets.tsindex 2ccc4c59..cb68e3c4 100644--- a/apps/web/src/context/electron/assets.ts+++ b/apps/web/src/context/electron/assets.ts@@ -13,7 +13,6 @@ import {transcodeForAnalysis,startResumableSession,uploadResumableStream,- computeAudioSyncOffset,assetFolderId,moveAssetsToFolder,useFolders,@@ -21,7 +20,7 @@ import {import type { Engine } from "@/components/engine";import type { Asset } from "@/components/engine/db";-import type { AssetAnalyzeRequest, AssetAnalyzeResult, AssetExportResult, AssetMoveResult, AssetProbeRequest, AssetsExportRequest, AssetSyncRequest, AssetSyncResult, AssetTranscriptResult, AssetTreeEntry, AssetVisualizeRequest, AssetVisualizeResult } from "@diffusionstudio/cli/channels";+import type { AssetAnalyzeRequest, AssetAnalyzeResult, AssetExportResult, AssetMoveResult, AssetProbeRequest, AssetsExportRequest, AssetTranscriptResult, AssetTreeEntry, AssetVisualizeRequest, AssetVisualizeResult } from "@diffusionstudio/cli/channels";import type { Accessor } from "solid-js";export function handleAssetsAdd(engine: Accessor<Engine>) {@@ -338,27 +337,6 @@ export function handleAssetTranscript(engine: Accessor<Engine>) {};}-export function handleAssetSync(engine: Accessor<Engine>) {- return async ({ audioId, videoId }: AssetSyncRequest): Promise<AssetSyncResult> => {- const { world } = engine();- const audio = world.assets.get(audioId);- assert(audio, `Asset ${audioId} not found.`);- const video = world.assets.get(videoId);- assert(video, `Asset ${videoId} not found.`);- assert(- audio.type === "AUDIO" || audio.type === "VIDEO",- `Asset ${audioId} is not an audio or video asset.`,- );- assert(- video.type === "AUDIO" || video.type === "VIDEO",- `Asset ${videoId} is not an audio or video asset.`,- );-- const { offsetSeconds, confidence } = await computeAudioSyncOffset(audio, video);- return { audioId, videoId, offsetSeconds, confidence };- };-}-export function handleAssetVisualize(engine: Accessor<Engine>) {return async (req: AssetVisualizeRequest): Promise<AssetVisualizeResult> => {const { world } = engine();diff --git a/apps/web/src/context/electron/electron.tsx b/apps/web/src/context/electron/electron.tsxindex 7b1ea12f..0853d6ca 100644--- a/apps/web/src/context/electron/electron.tsx+++ b/apps/web/src/context/electron/electron.tsx@@ -12,10 +12,10 @@ import { CLI_CHANNELS } from '@diffusionstudio/cli/channels';import { MAIN_CHANNELS } from '@desktop/main-channels';import { assert } from "@/utils/common";import { handleContextGet } from "./context";-import { handleAssetsAdd, handleAssetsList, handleAssetsDelete, handleAssetsMove, handleAssetsExport, handleAssetProbe, handleAssetFrame, handleAssetTranscript, handleAssetVisualize, handleAssetAnalyze, handleAssetSync } from "./assets";+import { handleAssetsAdd, handleAssetsList, handleAssetsDelete, handleAssetsMove, handleAssetsExport, handleAssetProbe, handleAssetFrame, handleAssetTranscript, handleAssetVisualize, handleAssetAnalyze } from "./assets";import { handleFoldersList, handleFolderCreate, handleFolderRename, handleFoldersMove, handleFoldersDelete } from "./folders";import { handleSelectionFocus, handleSelectionList, handleSelectionSet } from "./selection";-import { handleNodeList, handleNodeTree, handleNodeGrep, handleNodeScreenshot, handleNodeDelete, handleNodePatch, handleNodeDuplicate, handleNodeExport } from "./node";+import { handleNodeList, handleNodeTree, handleNodeGrep, handleNodeScreenshot, handleNodeDelete, handleNodePatch, handleNodeDuplicate, handleNodeRender } from "./node";import { handleMount, handleNodeInsert } from "./mount";import { handleProjectActive, handleProjectList, handleProjectCreate, handleProjectDelete, handleProjectOpen } from "./project";import { handleModels } from "./models";@@ -70,7 +70,6 @@ export function ElectronProvider(props: ElectronProviderProps) {const unsubAssetTranscript = cliBridge.handle(CLI_CHANNELS.ASSET_TRANSCRIPT, handleAssetTranscript(engineRef));const unsubAssetVisualize = cliBridge.handle(CLI_CHANNELS.ASSET_VISUALIZE, handleAssetVisualize(engineRef));const unsubAssetAnalyze = cliBridge.handle(CLI_CHANNELS.ASSET_ANALYZE, requireAuth(handleAssetAnalyze(engineRef)));- const unsubAssetSync = cliBridge.handle(CLI_CHANNELS.ASSET_SYNC, handleAssetSync(engineRef));const unsubSelectionList = cliBridge.handle(CLI_CHANNELS.SELECTION_LIST, handleSelectionList(engineRef));const unsubSelectionSet = cliBridge.handle(CLI_CHANNELS.SELECTION_SET, handleSelectionSet(engineRef));const unsubSelectionFocus = cliBridge.handle(CLI_CHANNELS.SELECTION_FOCUS, handleSelectionFocus(engineRef));@@ -83,7 +82,7 @@ export function ElectronProvider(props: ElectronProviderProps) {const unsubNodeDelete = cliBridge.handle(CLI_CHANNELS.NODE_DELETE, handleNodeDelete(engineRef));const unsubNodePatch = cliBridge.handle(CLI_CHANNELS.NODE_PATCH, handleNodePatch(engineRef));const unsubNodeDuplicate = cliBridge.handle(CLI_CHANNELS.NODE_DUPLICATE, handleNodeDuplicate(engineRef));- const unsubNodeExport = cliBridge.handle(CLI_CHANNELS.NODE_EXPORT, handleNodeExport(engineRef));+ const unsubNodeRender = cliBridge.handle(CLI_CHANNELS.NODE_RENDER, handleNodeRender(engineRef));const unsubProjectActive = cliBridge.handle(CLI_CHANNELS.PROJECT_ACTIVE, handleProjectActive(engineRef));const unsubProjectList = cliBridge.handle(CLI_CHANNELS.PROJECT_LIST, handleProjectList());const unsubProjectCreate = cliBridge.handle(CLI_CHANNELS.PROJECT_CREATE, handleProjectCreate(engineRef, setParams));@@ -110,7 +109,6 @@ export function ElectronProvider(props: ElectronProviderProps) {unsubAssetTranscript();unsubAssetVisualize();unsubAssetAnalyze();- unsubAssetSync();unsubSelectionList();unsubSelectionSet();unsubSelectionFocus();@@ -123,7 +121,7 @@ export function ElectronProvider(props: ElectronProviderProps) {unsubNodeDelete();unsubNodePatch();unsubNodeDuplicate();- unsubNodeExport();+ unsubNodeRender();unsubProjectActive();unsubProjectList();unsubProjectCreate();diff --git a/apps/web/src/context/electron/mount.ts b/apps/web/src/context/electron/mount.tsindex 401b9cf8..d72c858c 100644--- a/apps/web/src/context/electron/mount.ts+++ b/apps/web/src/context/electron/mount.ts@@ -11,44 +11,50 @@ import { resolveEntityEid } from "./node";import { hasComponent } from "bitecs";import type { Accessor } from "solid-js";-import type { MountRequest, NodeInsertRequest } from "@diffusionstudio/cli/channels";+import type { MountRequest, MountResult, NodeInsertRequest } from "@diffusionstudio/cli/channels";import type { Engine } from "@/components/engine";/*** `dapi mount` — evaluates a compiled project module and renders it directly into the ECS world*/export function handleMount(engine: Accessor<Engine>) {- return async ({ code }: MountRequest): Promise<void> => {+ return async ({ code }: MountRequest): Promise<MountResult> => {const e = engine();const world = e.world;const c = world.components;- // Evaluate — top-level code (including top-level await) runs here.- const module = await importProjectModule(code);- const project = module.default as () => unknown;- assert(typeof project === "function", "The project module must default-export a Solid component");+ try {+ // Evaluate — top-level code (including top-level await) runs here.+ const module = await importProjectModule(code);+ const project = module.default as () => unknown;+ assert(typeof project === "function", "The project module must default-export a Solid component");- const document = new WorldDocument(e);+ const document = new WorldDocument(e);- try {- world.history.startTransaction("Render project");- renderProject(project, document);- world.history.commitTransaction();- } catch (error) {- world.history.rollbackTransaction();- throw error;- }+ try {+ world.history.startTransaction("Render project");+ renderProject(project, document);+ world.history.commitTransaction();+ } catch (error) {+ world.history.rollbackTransaction();+ throw error;+ }- try {- world.history.startTransaction("Commit document");- await document.commit();- } finally {- world.history.commitTransaction();- }+ try {+ world.history.startTransaction("Commit document");+ await document.commit();+ } finally {+ world.history.commitTransaction();+ }- if (document.rootEid && hasComponent(world, document.rootEid, c.Scene)) {- switchActiveScene(world, document.rootEid);+ if (document.rootEid && hasComponent(world, document.rootEid, c.Scene)) {+ switchActiveScene(world, document.rootEid);+ }+ } catch (error) {+ return { status: "rejected", error: error instanceof Error ? error.message : String(error) };}++ return { status: "fulfilled" };};}@@ -60,34 +66,40 @@ export function handleMount(engine: Accessor<Engine>) {* `<colorStop>` roots.*/export function handleNodeInsert(engine: Accessor<Engine>) {- return async ({ code, parentId, index }: NodeInsertRequest): Promise<void> => {+ return async ({ code, parentId, index }: NodeInsertRequest): Promise<MountResult> => {const e = engine();const world = e.world;- const parentEid = resolveEntityEid(world, parentId);- const module = await importProjectModule(code);- const project = module.default as () => unknown;- assert(typeof project === "function", "The project module must default-export a Solid component");+ try {+ const parentEid = resolveEntityEid(world, parentId);+ const module = await importProjectModule(code);+ const project = module.default as () => unknown;+ assert(typeof project === "function", "The project module must default-export a Solid component");- const document = new WorldDocument(e, { parentEid });+ const document = new WorldDocument(e, { parentEid });- try {- world.history.startTransaction("Insert nodes");- renderProject(project, document);- if (index !== undefined && document.rootEid !== null) {- reorderEntity(world, document.rootEid, index);+ try {+ world.history.startTransaction("Insert nodes");+ renderProject(project, document);+ if (index !== undefined && document.rootEid !== null) {+ reorderEntity(world, document.rootEid, index);+ }+ world.history.commitTransaction();+ } catch (error) {+ world.history.rollbackTransaction();+ throw error;+ }++ try {+ world.history.startTransaction("Commit document");+ await document.commit();+ } finally {+ world.history.commitTransaction();}- world.history.commitTransaction();} catch (error) {- world.history.rollbackTransaction();- throw error;+ return { status: "rejected", error: error instanceof Error ? error.message : String(error) };}- try {- world.history.startTransaction("Commit document");- await document.commit();- } finally {- world.history.commitTransaction();- }+ return { status: "fulfilled" };};}diff --git a/apps/web/src/context/electron/node.ts b/apps/web/src/context/electron/node.tsindex 52f96db4..97b71fba 100644--- a/apps/web/src/context/electron/node.ts+++ b/apps/web/src/context/electron/node.ts@@ -3,7 +3,6 @@* file, You can obtain one at http://mozilla.org/MPL/2.0/. */import { entityExists, hasComponent, query, Not, Or } from "bitecs";-import { PATCH_PROP_KEYS } from "@diffusionstudio/jsx";import { createEncoder } from "@/components/engine/encode/encoder";import { ElectronWritableFileHandle } from "@/lib/electron-file-writable";@@ -37,8 +36,8 @@ import type {NodePatchResult,NodeRef,NodeTree,- NodeExportRequest,- NodeExportResult,+ NodeRenderRequest,+ NodeRenderResult,} from "@diffusionstudio/cli/channels";import type { EncoderConfig } from "@/components/engine/encode/interfaces";import type { Accessor } from "solid-js";@@ -249,8 +248,8 @@ export function handleNodeGrep(engine: Accessor<Engine>) {id !== undefined? getEntityTree(world, resolveEntityEid(world, id)): [...query(world, [c.Geometry, Not(ChildOf("*")), Not(c.Deleted)])].flatMap((root) =>- getEntityTree(world, root),- );+ getEntityTree(world, root),+ );const results: NodeGrepResult[] = [];for (const eid of eids) {@@ -331,8 +330,8 @@ export function handleNodeScreenshot(engine: Accessor<Engine>) {};}-export function handleNodeExport(engine: Accessor<Engine>) {- return async ({ id, output, config }: NodeExportRequest): Promise<NodeExportResult> => {+export function handleNodeRender(engine: Accessor<Engine>) {+ return async ({ id, output, config }: NodeRenderRequest): Promise<NodeRenderResult> => {const e = engine();const w = e.world;@@ -428,81 +427,46 @@ export function handleNodeDuplicate(engine: Accessor<Engine>) {};}-const PATCH_KEYS: ReadonlySet<string> = new Set(PATCH_PROP_KEYS);-export function handleNodePatch(engine: Accessor<Engine>) {return async ({ patches }: { patches: NodePatch[] }): Promise<NodePatchResult[]> => {const e = engine();const world = e.world;- const doc = new WorldDocument(e);-- // `src` mounts asynchronously and can't be rolled back, so a dry run only- // performs its synchronous checks.- const applyPatch = (eid: number, patch: NodePatch, dryRun: boolean) => {- const node = doc.element(eid);- for (const [key, value] of Object.entries(patch)) {- if (key === "id") continue;- if (!PATCH_KEYS.has(key)) throw new Error(`Unsupported property: ${key}`);- if (dryRun && key === "src") {- if (typeof value !== "string" || value.trim().length === 0)- throw new Error("`src` must be a non-empty string");- continue;- }- doc.setProperty(node, key, value);- }- };- const resolved = patches.map((patch) => {+ const results: NodePatchResult[] = [];+ for (const patch of patches) {+ const doc = new WorldDocument(e);+try {const eid = resolveEntityEid(world, patch.id);- world.history.startTransaction("vetting patch");++ world.history.startTransaction("patching node");try {- applyPatch(eid, patch, true);- } finally {+ const node = doc.element(eid);+ for (const [key, value] of Object.entries(patch)) {+ if (key === "id") continue;+ doc.setProperty(node, key, value);+ }+ world.history.commitTransaction();+ } catch (error) {world.history.rollbackTransaction();+ throw error;}- return { patch, eid, promises: [] as Promise<void>[] };- } catch (err) {- return { patch, error: (err as Error).message };- }- });- const toApply = resolved.filter(- (r): r is { patch: NodePatch; eid: number; promises: Promise<void>[] } => "eid" in r,- );- if (toApply.length > 0) {- world.history.transaction("patching nodes", () => {- for (const entry of toApply) {- const before = doc.promises.length;- applyPatch(entry.eid, entry.patch, false);- entry.promises = doc.promises.slice(before);+ // `src` mounts asynchronously; a failed mount rejects this patch, but+ // its synchronous props are committed at this point regardless.+ world.history.startTransaction("Commit document");+ try {+ await doc.commit();+ } finally {+ world.history.commitTransaction();}- });- }- // Await async source mounts per patch so a failed `src` rejects the right- // node; its synchronous props are applied at this point regardless.- const srcErrors = new Map<NodePatch, string>();- if (toApply.some((entry) => entry.promises.length > 0)) {- world.history.startTransaction("Commit document");- try {- await Promise.all(toApply.map(async (entry) => {- const settled = await Promise.allSettled(entry.promises);- const failed = settled.find((s): s is PromiseRejectedResult => s.status === "rejected");- if (failed === undefined) return;- const reason = failed.reason;- srcErrors.set(entry.patch, reason instanceof Error ? reason.message : String(reason));- }));- } finally {- world.history.commitTransaction();+ results.push({ status: "fulfilled", id: patch.id });+ } catch (error) {+ results.push({ status: "rejected", id: patch.id, error: (error as Error).message });}}- return resolved.map((r) => {- if (!("eid" in r)) return { status: "rejected", id: r.patch.id, error: r.error };- const srcError = srcErrors.get(r.patch);- if (srcError !== undefined) return { status: "rejected", id: r.patch.id, error: srcError };- return { status: "fulfilled", id: r.patch.id };- });+ return results;};}diff --git a/apps/web/src/utils/common.ts b/apps/web/src/utils/common.tsindex b7bb8217..9f2cfb93 100644--- a/apps/web/src/utils/common.ts+++ b/apps/web/src/utils/common.ts@@ -14,6 +14,16 @@ export function assert(condition: any, message?: string): asserts condition {}}+/**+ * Assert that all promises in the array have been settled.+ * @param results - The array of promises.+ * @returns void+ */+export function assertAllSettled(results: PromiseSettledResult<void>[]) {+ const errors = results.filter((result) => result.status === "rejected").map((result) => result.reason);+ assert(errors.length === 0, `${errors.length} pipeline stage(s) failed:\n${errors.map((err) => err instanceof Error ? err.message : String(err)).join("\n")}`);+}+/*** Clamp a value between a minimum and maximum.* @param value - The value to clamp.diff --git a/apps/web/src/utils/jsx.ts b/apps/web/src/utils/jsx.tsindex 1ef7b0f9..39e8493f 100644--- a/apps/web/src/utils/jsx.ts+++ b/apps/web/src/utils/jsx.ts@@ -17,9 +17,13 @@ import {TextBaseline,addComponent,appendChild,+ computeAudioSyncOffsetCached,createEntity,deleteEntity,+ findAssetDuration,+ findGeometryAsset,getNextName,+ getParentEntity,getSceneAncestor,hasAudioSources,removeChild,@@ -31,7 +35,7 @@ import {import { ChildOf } from "@/components/engine/components";import { ASPECT_RATIO_DIMENSIONS, findEmptyPlacement } from "@/utils/genai";import { resolveAsset, resolveGeneratedAsset } from "@/utils/jsx-generation";-import { assert, parseColor } from "@/utils";+import { assert, assertAllSettled, parseColor } from "@/utils";import type { AssetRef, AssetSpecInput, ProjectDocument } from "@diffusionstudio/jsx";import type { Engine, EngineWorld } from "@/components/engine";@@ -98,7 +102,8 @@ const CAPTION_PRESET_MAP = {type GenaiQueueItem = {node: DocumentNode;ref?: AssetRef;- type: "asset" | "caption";+ targetKey?: string;+ type: "asset" | "caption" | "sync";};export class WorldDocument implements ProjectDocument<DocumentNode> {@@ -120,63 +125,83 @@ export class WorldDocument implements ProjectDocument<DocumentNode> {public async commit(): Promise<void> {const world = this.world;const c = world.components;- const errors: unknown[] = [];// Wait for assets to settle{- await Promise.all(this.promises);+ const results = await Promise.allSettled(this.promises);+ assertAllSettled(results);}// Handle generative queue{const memo: GenerationMemo = new Map();- await Promise.all(this.queue.map(async ({ node, ref, type }) => {+ const results = await Promise.allSettled(this.queue.map(async ({ node, ref, type }) => {if (node.kind !== "element" || hasComponent(world, node.eid, c.Deleted) || type !== "asset" || !ref) return;try {const asset = await resolveGeneratedAsset(world, ref, memo);await this.mountSource(node, asset.id);- } catch (err) {- errors.push(err instanceof Error ? err : new Error(String(err), { cause: err }));} finally {world.history.untrack(() => removeComponent(world, node.eid, c.Generating));}}));++ assertAllSettled(results);}- // Handle captioning queue+ // Handle sync queue: after generated assets have landed (either side may+ // be generated), before captions read the scene's final placement.{- const scenes = new Map<number, number[]>();+ for (const { node, targetKey, type } of this.queue) {+ if (node.kind !== "element" || type !== "sync" || !targetKey) continue;+ const targetEid = query(world, [c.Key, Not(c.Deleted)]).find(eid => c.Key[eid] === targetKey);++ assert(targetEid !== undefined, `syncTo: no other element carries key "${targetKey}"`);++ const source = findGeometryAsset(world, node.eid);+ assert(+ source !== null && (source.type === "AUDIO" || source.type === "VIDEO"),+ `syncTo: node ${node.eid} has no audio or video source to align`,+ );+ const targetSource = findGeometryAsset(world, targetEid);+ assert(+ targetSource !== null && (targetSource.type === "AUDIO" || targetSource.type === "VIDEO"),+ `syncTo: target "${targetKey}" has no audio or video source to align against`,+ );++ const { offsetSeconds } = await computeAudioSyncOffsetCached(source, targetSource);++ const offsetFrames = Math.round(offsetSeconds * world.frameRate);+ const parentEid = getParentEntity(world, node.eid);+ const parentDelay = parentEid !== null ? (c.Computed.delay[parentEid] ?? 0) : 0;+ const delay = (c.Computed.delay[targetEid] ?? 0) + offsetFrames - parentDelay;++ node.startTime = delay;+ setComponent(world, node.eid, c.Delay, delay);++ const targetStart = (c.Computed.start[targetEid] ?? 0) - parentDelay;+ const targetEnd = (c.Computed.end[targetEid] ?? 0) - parentDelay;+ const duration = findAssetDuration(world, node.eid);++ const start = node.inPoint ?? Math.max(delay, targetStart);+ const end = node.outPoint ?? Math.min(duration !== null ? delay + duration : targetEnd, targetEnd);+ assert(end > start, `syncTo: the aligned clip does not overlap the window of "${targetKey}"`);+ setComponent(world, node.eid, c.Trim, { start: start - delay, end: end - delay });+ }+ }+ // Handle captioning queue+ {for (const { node, type } of this.queue) {if (node.kind !== "element" || hasComponent(world, node.eid, c.Deleted) || type !== "caption" || c.AssetId[node.eid]) continue;const sceneEid = getSceneAncestor(world, node.eid);- if (sceneEid === null) continue;- scenes.set(sceneEid, [...(scenes.get(sceneEid) ?? []), node.eid]);- }+ if (!sceneEid || !hasAudioSources(world, sceneEid)) continue;- for (const [sceneEid, eids] of scenes) {- if (!hasAudioSources(world, sceneEid)) continue;- try {- const { asset, trim } = await transcribeScene(this.engine, sceneEid);- world.history.transaction("Attach captions", () => {- for (const eid of eids) {- if (hasComponent(world, eid, c.Deleted)) continue;- setComponent(world, eid, c.AssetId, asset.id);- setComponent(world, eid, c.Trim, trim);- }- });- } catch (error) {- console.error("Captioning failed:", error);- }+ const { asset, trim } = await transcribeScene(this.engine, sceneEid);+ setComponent(world, node.eid, c.AssetId, asset.id);+ setComponent(world, node.eid, c.Trim, trim);}}-- // Handle errors- {- const messages = errors.map((err) => err instanceof Error ? err.message : String(err));- assert(errors.length === 0, `${errors.length} asset generations failed:\n${messages.join("\n")}`);- }}private get world(): EngineWorld {@@ -395,6 +420,16 @@ export class WorldDocument implements ProjectDocument<DocumentNode> {const db = value <= 0 ? -Infinity : 20 * Math.log10(value);setComponent(world, eid, c.Volume, db);break;+ } case "muted": {+ assert(typeof value === "boolean", "`muted` must be a boolean" + `, value: ${value}`);+ if (value) addComponent(world, eid, c.Muted);+ else removeComponent(world, eid, c.Muted);+ break;+ } case "syncTo": {+ assert(typeof value === "string", "`syncTo` must be a string" + `, value: ${value}`);+ assert(value.trim().length > 0, "`syncTo` must be a non-empty string");+ this.queue.push({ node, targetKey: value.trim(), type: "sync" });+ break;} case "src": {if (isAssetRef(value)) {world.history.untrack(() => addComponent(world, eid, c.Generating));diff --git a/docs/assets.md b/docs/assets.mdindex 9d6fd29a..0663dc9a 100644--- a/docs/assets.md+++ b/docs/assets.md@@ -81,5 +81,5 @@ dapi asset transcript gbHJ # what is said, and when?dapi asset analyze gbHJ -p "best moment for a thumbnail?"dapi mount cut.tsx # compose the editdapi node screenshot # verify visually-dapi node export -o cut.mp4 # ship it+dapi node render -o cut.mp4 # ship it```diff --git a/docs/cli.md b/docs/cli.mdindex da3f0aae..3a047d97 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -133,10 +133,10 @@ Deletes nodes and their descendants: `dapi node rm <ids...>`.## Rendering-### `node export`+### `node render````sh-dapi node export [id] [-o, --output <path>] ([config.json] | --json <str>)+dapi node render [id] [-o, --output <path>] ([config.json] | --json <str>)```Renders a scene to a video file, locally. Defaults to the active scene and an MP4 (H.264, 10 Mbps, 1080p, AAC 128 kbps) in the temp directory; prints `{ path }`. Long-running: progress goes to stderr, stdout stays clean JSON. The encode window follows the scene's workarea.diff --git a/docs/composition.md b/docs/composition.mdindex d2526161..efd968e2 100644--- a/docs/composition.md+++ b/docs/composition.md@@ -106,7 +106,7 @@ Composition-relative, Lottie-inspired. Values take any [time format](cli.md#time<audio src="/Movies/lav.wav" syncTo="camera" />```-`syncTo` and `startTime` are mutually exclusive; `inPoint`/`outPoint` stay yours, and when omitted the window defaults to the overlap with the target's window. Alignment runs after generated assets land (either side may be generated), locally and cached, and blocks the mount; each aligned node logs its offset and a confidence score (a clear match is above ~0.9).+`syncTo` and `startTime` are mutually exclusive; `inPoint`/`outPoint` stay yours, and when omitted the window defaults to the overlap with the target's window. Alignment runs after generated assets land (either side may be generated), locally and cached, and blocks the mount; an alignment too weak to trust fails the mount and the node keeps its default placement.## Paintsdiff --git a/docs/getting-started.md b/docs/getting-started.mdindex bac7dced..ba98a9ba 100644--- a/docs/getting-started.md+++ b/docs/getting-started.md@@ -63,13 +63,13 @@ Check the result visually:dapi node screenshot # prints { "path": "/tmp/….png" }```-## Export+## Render```sh-dapi node export -o hello.mp4+dapi node render -o hello.mp4```-Renders the active scene locally with sensible defaults (MP4, H.264, 1080p). Codec, bitrate, resolution, and trim are configurable; see [`node export`](cli.md#node-export) in the CLI reference.+Renders the active scene locally with sensible defaults (MP4, H.264, 1080p). Codec, bitrate, resolution, and trim are configurable; see [`node render`](cli.md#node-render) in the CLI reference.## Nextdiff --git a/packages/jsx/src/types.ts b/packages/jsx/src/types.tsindex 50693c85..1c4d5153 100644--- a/packages/jsx/src/types.ts+++ b/packages/jsx/src/types.ts@@ -57,6 +57,12 @@ export type PatchProps = {outPoint?: Time;/** Composition time at which the node's source time 0 is placed. Defaults to the in point. */startTime?: Time;+ /**+ * Key of another element carrying an audio track. Derives `startTime` by+ * cross-correlating the two audio signals so the recordings coincide on the+ * timeline. Mutually exclusive with `startTime`.+ */+ syncTo?: string;/** Any CSS color, applied to the node's solid fill (created if absent); alpha is ignored — use `opacity`. */fill?: string;/** Path, URL, asset id, or a `generate.*` declaration. */@@ -65,6 +71,8 @@ export type PatchProps = {objectFit?: Fit;/** 0–1; 1 = unity gain. */volume?: number;+ /** Excludes the node's audio from the mix; independent of `volume`. */+ muted?: boolean;/** A family available on the machine (`dapi fonts`). */fontFamily?: string;/** Font size, px. */@@ -100,10 +108,12 @@ export const PATCH_PROP_KEYS = Object.keys({inPoint: true,outPoint: true,startTime: true,+ syncTo: true,fill: true,src: true,objectFit: true,volume: true,+ muted: true,fontFamily: true,fontSize: true,fontWeight: true,@@ -164,7 +174,7 @@ export type ColorStopProps = Required<Pick<PatchProps, "offset" | "color">> &export type VideoProps = CommonProps &Required<Pick<PatchProps, "src">> &- Pick<PatchProps, "objectFit" | "volume">;+ Pick<PatchProps, "objectFit" | "volume" | "muted" | "syncTo">;export type ImageProps = CommonProps &Required<Pick<PatchProps, "src">> &@@ -172,7 +182,7 @@ export type ImageProps = CommonProps &export type AudioProps = TimingProps &Required<Pick<PatchProps, "src">> &- Pick<PatchProps, "name" | "volume">;+ Pick<PatchProps, "name" | "volume" | "muted" | "syncTo">;export type TextProps = CommonProps &Pick<