The VS Code of video editing, built for agents.
TypeScript
Refactor media command: replace `analyze` with `listen` for audio analysis
konstantin-paulus121ab67a2d6a
diff --git a/README.md b/README.mdindex 7c89961f..8fd48f7e 100644--- a/README.md+++ b/README.md@@ -64,7 +64,7 @@ dapi media probe clip.mp4 # container + codec metdapi media grab clip.mp4 -t 0 12 45 # decode frames to PNGsdapi media visualize track.mp3 # waveform / filmstrip previewsdapi media transcribe interview.wav # timed, word-level transcript-dapi media analyze b-roll.mp4 -p "when is the product shown?" # ask a vision model+dapi media listen interview.mp4 -p "what is said in the intro?" # ask a multimodal modeldapi node screenshot # see the canvas itself```@@ -77,7 +77,7 @@ dapi node screenshot # see the canvas itself| `dapi mount` | Compile and mount a JSX composition || `dapi node …` | The scene graph: `ls`, `tree`, `grep`, `patch`, `insert`, `cp`, `rm`, `screenshot`, `export` || `dapi asset …` | The media library: `add`, `ls`, `tree`, `mv`, `rm`, `export` |-| `dapi media …` | Inspect a file by id or path: `probe`, `grab`, `visualize`, `transcribe`, `analyze` |+| `dapi media …` | Inspect a file by id or path: `probe`, `grab`, `visualize`, `transcribe`, `listen` || `dapi project …` / `dapi folder …` / `dapi selection …` | Projects, library folders, canvas selection || `dapi models` / `dapi voices` / `dapi fonts` | Discover generation models, speech voices, local fonts |diff --git a/apps/cli/reference/README.md b/apps/cli/reference/README.mdindex 78e646c3..1873c542 100644--- a/apps/cli/reference/README.md+++ b/apps/cli/reference/README.md@@ -21,7 +21,7 @@ How the surface is divided:- Declarative composition happens through `mount`, which renders a Solid JSX project into the canvas (see [jsx/](./jsx/README.md)); `node insert` runs the same pipeline but inserts the rendered nodes into an existing parent entity instead of mounting document roots.- Scenes are created declaratively via `mount` (`<scene key="...">`); there is no imperative scene command.- AI asset generation (image / video / speech / audio) is declared in the project module (`generate.*`, see [jsx/generate.md](./jsx/generate.md)) and produced on mount. `models` and `voices` list what those declarations can reference.-- Inspecting an existing asset (probe / transcribe / analyze / visualize / frame) lives under `asset`; writing an asset's original file back to disk is `asset export`.+- Inspecting an existing asset (probe / transcribe / listen / visualize / frame) lives under `asset`; writing an asset's original file back to disk is `asset export`.- Organizing the library lives under `folder`; moving assets between folders is `asset mv`.## Commands@@ -77,7 +77,7 @@ How the surface is divided:- [`dapi media transcribe`](./media/transcribe.md): timed speech transcript- [`dapi media grab`](./media/grab.md): decode video frames to PNGs- [`dapi media visualize`](./media/visualize.md): waveform / filmstrip / thumbnail PNG-- [`dapi media analyze`](./media/analyze.md): AI description of a media file+- [`dapi media listen`](./media/listen.md): AI description of an audio track### Foldersdiff --git a/apps/cli/reference/media/analyze.md b/apps/cli/reference/media/analyze.mddeleted file mode 100644index 5e161d65..00000000--- a/apps/cli/reference/media/analyze.md+++ /dev/null@@ -1,23 +0,0 @@-# `dapi media analyze <id|path>`--Analyzes an image, video, or audio asset with a multimodal model. With no prompt it returns a general description of the asset; with `--prompt` it answers that question about the asset (e.g. "what's the dominant color?", "summarize what happens").--## Input--- `<id|path>`: an image, audio, or video asset id, or a local file to analyze in place without adding it to the library (required).-- `-p, --prompt <str>`: question or instruction about the asset (optional; defaults to a general description).-- `-s, --start <time>`: start of the segment to analyze, a `Time` value (optional; default `0`). Timestamps in the analysis are relative to this point. Ignored for images.-- `-e, --end <time>`: end of the segment to analyze, a `Time` value (optional; default the asset's duration). Ignored for images.-- `--keep-video`: include the video stream when analyzing a video asset (optional). By default only the audio track is analyzed, which is significantly faster since the video stream is neither transcoded nor uploaded. No effect on audio or image assets.--## Output--One JSON object, the model's answer:--```ts-{ analysis: string }-```--## Errors--Exits non-zero if the id is unknown, or the asset's media type isn't supported by the chosen model.diff --git a/apps/cli/reference/media/listen.md b/apps/cli/reference/media/listen.mdnew file mode 100644index 00000000..baca5387--- /dev/null+++ b/apps/cli/reference/media/listen.md@@ -0,0 +1,22 @@+# `dapi media listen <id|path>`++Puts a multimodal model in front of an audio track and returns its answer. With no prompt it returns a general description of what is heard; with `--prompt` it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). Accepts an audio file or a video, but only the audio track is analyzed: for what is on screen in a video, use `visualize` first.++## Input++- `<id|path>`: an audio or video asset id, or a local file to analyze in place without adding it to the library (required).+- `-p, --prompt <str>`: question or instruction about the audio (optional; defaults to a general description).+- `-s, --start <time>`: start of the segment to analyze, a `Time` value (optional; default `0`). Timestamps in the analysis are relative to this point.+- `-e, --end <time>`: end of the segment to analyze, a `Time` value (optional; default the asset's duration).++## Output++One JSON object, the model's answer:++```ts+{ analysis: string }+```++## Errors++Exits non-zero if the id is unknown, or the asset isn't a video or audio asset.diff --git a/apps/cli/src/cli-channels.ts b/apps/cli/src/cli-channels.tsindex bac80a4f..11a3e9e5 100644--- a/apps/cli/src/cli-channels.ts+++ b/apps/cli/src/cli-channels.ts@@ -134,8 +134,8 @@ export type AssetVisualizeResult = {base64: string;} & Record<string, unknown>;-export type AssetAnalyzeRequest = AssetRef & { prompt?: string; start?: number; end?: number; stripVideo?: boolean };-export type AssetAnalyzeResult = { analysis?: string; start?: number; end?: number };+export type AssetListenRequest = AssetRef & { prompt?: string; start?: number; end?: number; stripVideo?: boolean };+export type AssetListenResult = { analysis?: string; start?: number; end?: number };export type GeneratedAsset = { id: string; name: string; type: string };diff --git a/apps/cli/src/index.ts b/apps/cli/src/index.tsindex 981ab50c..c54b5f52 100644--- a/apps/cli/src/index.ts+++ b/apps/cli/src/index.ts@@ -531,9 +531,9 @@ async function assetTranscribe(ref: string, opts: AssetTranscribeOptions): Promi}}-type AssetAnalyzeOptions = { prompt?: string; start?: string; end?: string; keepVideo?: boolean };+type AssetListenOptions = { prompt?: string; start?: string; end?: string };-async function assetAnalyze(ref: string, opts: AssetAnalyzeOptions): Promise<void> {+async function assetListen(ref: string, opts: AssetListenOptions): Promise<void> {const start = opts.start !== undefined ? parseTimeArg(opts.start, "--start") : undefined;const end = opts.end !== undefined ? parseTimeArg(opts.end, "--end") : undefined;if (start !== undefined && end !== undefined && start >= end) {@@ -544,8 +544,8 @@ async function assetAnalyze(ref: string, opts: AssetAnalyzeOptions): Promise<voiconst target = resolveAssetRef(ref);const stop = startSpinner("Analyzing asset");try {- const result = await editor.asset.analyze.query(- { ...target, prompt: opts.prompt, start, end, stripVideo: !opts.keepVideo },+ const result = await editor.asset.listen.query(+ { ...target, prompt: opts.prompt, start, end, stripVideo: true },GENERATE,);stop();@@ -1107,14 +1107,13 @@ media.action((ref: string, opts: AssetVisualizeOptions) => assetVisualize(ref, opts));media- .command("analyze")- .description(`Audio-first and expensive: prompt a multimodal model for a semantic analysis of a video or audio file and print its answer. Shines on the semantics of audio (the name of the music playing, who is speaking, the spoken content with second-granularity timestamps). For a video this analyzes only the audio track by default; if the footage is silent or you need what is on screen, use \`visualize\` first or pass --keep-video to upload the video stream${docs("media/analyze")}`)+ .command("listen")+ .description(`Audio-only: prompt a multimodal model for a semantic analysis of an audio track and print its answer. Shines on the semantics of audio (the name of the music playing, who is speaking, the spoken content with second-granularity timestamps). Accepts an audio file or a video, but only the audio track is analyzed${docs("media/listen")}`).argument("<id|path>", "video or audio asset id, or a local file to analyze").option("-p, --prompt <str>", "question or instruction to guide the analysis").option("-s, --start <time>", `start of the segment to analyze — seconds, "45f" frames, or "MM:SS" (default: 0); timestamps in the analysis are relative to this point`).option("-e, --end <time>", `end of the segment to analyze — seconds, "45f" frames, or "MM:SS" (default: asset duration)`)- .option("--keep-video", "include the video stream when analyzing a video; by default only the audio track is analyzed, which is significantly faster since the video stream is not uploaded")- .action((ref: string, opts: AssetAnalyzeOptions) => assetAnalyze(ref, opts));+ .action((ref: string, opts: AssetListenOptions) => assetListen(ref, opts));const folder = program.command("folder")diff --git a/apps/web/src/context/editor-api/api.tsx b/apps/web/src/context/editor-api/api.tsxindex af8f7f5a..97b7e287 100644--- a/apps/web/src/context/editor-api/api.tsx+++ b/apps/web/src/context/editor-api/api.tsx@@ -8,7 +8,7 @@ import { useEngine } from '@/context/engine';import { useAuth } from '@/context/auth';import { t, q, m, q0, m0 } from "@/lib/cli-rpc";import { handleContextGet } from "./context";-import { handleAssetsAdd, handleAssetsList, handleAssetTree, handleAssetsDelete, handleAssetsMove, handleAssetsExport, handleAssetProbe, handleAssetFrame, handleAssetTranscribe, handleAssetVisualize, handleAssetAnalyze } from "./assets";+import { handleAssetsAdd, handleAssetsList, handleAssetTree, handleAssetsDelete, handleAssetsMove, handleAssetsExport, handleAssetProbe, handleAssetFrame, handleAssetTranscribe, handleAssetVisualize, handleAssetListen } from "./assets";import { handleFoldersList, handleFolderCreate, handleFolderRename, handleFoldersMove, handleFoldersDelete } from "./folders";import { handleSelectionFocus, handleSelectionList, handleSelectionSet } from "./selection";import { handleNodeList, handleNodeTree, handleNodeGrep, handleNodeScreenshot, handleNodeDelete, handleNodePatch, handleNodeDuplicate, handleNodeRender } from "./node";@@ -108,7 +108,7 @@ function createAppRouter({ getEngine, getUser, requireAuth, setParams }: AppRoutframe: q(handleAssetFrame(getEngine)),transcribe: q(handleAssetTranscribe(getEngine)),visualize: q(handleAssetVisualize(getEngine)),- analyze: q(requireAuth(handleAssetAnalyze(getEngine))),+ listen: q(requireAuth(handleAssetListen(getEngine))),}),folder: t.router({list: q(handleFoldersList(getEngine)),diff --git a/apps/web/src/context/editor-api/assets.ts b/apps/web/src/context/editor-api/assets.tsindex c7d0c18d..cd33150e 100644--- a/apps/web/src/context/editor-api/assets.ts+++ b/apps/web/src/context/editor-api/assets.ts@@ -20,7 +20,7 @@ import {import type { Engine } from "@/components/engine";import type { Asset } from "@/components/engine/db";-import type { AssetAnalyzeRequest, AssetAnalyzeResult, AssetExportResult, AssetFrameRequest, AssetListResult, AssetMoveResult, AssetProbeRequest, AssetRecord, AssetRef, AssetsExportRequest, AssetTranscribeRequest, AssetTranscribeResult, AssetTreeEntry, AssetVisualizeRequest, AssetVisualizeResult, TranscriptSegment } from "@diffusionstudio/cli/channels";+import type { AssetListenRequest, AssetListenResult, AssetExportResult, AssetFrameRequest, AssetListResult, AssetMoveResult, AssetProbeRequest, AssetRecord, AssetRef, AssetsExportRequest, AssetTranscribeRequest, AssetTranscribeResult, AssetTreeEntry, AssetVisualizeRequest, AssetVisualizeResult, TranscriptSegment } from "@diffusionstudio/cli/channels";import type { Accessor } from "solid-js";export function handleAssetsAdd(engine: Accessor<Engine>) {@@ -450,8 +450,8 @@ export function handleAssetVisualize(engine: Accessor<Engine>) {};}-export function handleAssetAnalyze(engine: Accessor<Engine>) {- return async (req: AssetAnalyzeRequest): Promise<AssetAnalyzeResult> => {+export function handleAssetListen(engine: Accessor<Engine>) {+ return async (req: AssetListenRequest): Promise<AssetListenResult> => {const { prompt, start, end } = req;let { stripVideo } = req;const { world } = engine();diff --git a/docs/assets.md b/docs/assets.mdindex 57029109..628885a2 100644--- a/docs/assets.md+++ b/docs/assets.md@@ -1,6 +1,6 @@# Working with media-The asset library holds a project's media. Beyond storage, the CLI ships an inspection toolchain under `dapi media` (probe, grab, visualize, transcribe, analyze) so an agent can *look at* footage before cutting it. All of it runs locally except `analyze` and `transcribe`, which use hosted models.+The asset library holds a project's media. Beyond storage, the CLI ships an inspection toolchain under `dapi media` (probe, grab, visualize, transcribe, listen) so an agent can *look at* footage before cutting it. All of it runs locally except `listen` and `transcribe`, which use hosted models.**Id or path:** every inspection command takes either an asset id or a local file path. A path is read directly, so `dapi media probe clip.mp4` works on a file that isn't in the project yet.@@ -64,13 +64,13 @@ dapi media transcribe <id|path>Transcribes speech in a video or audio file. Returns segments with **word-level start/end times in seconds**: the raw material for caption timing, jump cuts, and content search.-### `media analyze`+### `media listen````sh-dapi media analyze <id|path> [-p <prompt>] [-s <start>] [-e <end>]+dapi media listen <id|path> [-p <prompt>] [-s <start>] [-e <end>]```-Puts a multimodal model in front of an image, video, or audio asset. Without a prompt, returns a general description; with one, answers it ("what's the dominant color?", "when is the product shown?"). `-s`/`-e` scope the analyzed segment; timestamps in the answer are relative to `-s`.+Puts a multimodal model in front of an audio track. Without a prompt, returns a general description of what is heard; with one, answers it ("who is speaking?", "what music is playing?"). Accepts an audio file or a video, but only the audio track is analyzed. `-s`/`-e` scope the analyzed segment; timestamps in the answer are relative to `-s`.## A typical agent pass@@ -81,7 +81,7 @@ dapi asset tree --depth 1 # what's here?dapi media probe gbHJ # what format is it?dapi media visualize gbHJ # what does it look like over time?dapi media transcribe gbHJ # what is said, and when?-dapi media analyze gbHJ -p "best moment for a thumbnail?"+dapi media listen gbHJ -p "what is said in the intro?"dapi mount cut.tsx # compose the editdapi node screenshot # verify visuallydapi node render -o cut.mp4 # ship itdiff --git a/docs/cli.md b/docs/cli.mdindex 586d896f..9b20c8e4 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -176,7 +176,7 @@ dapi project rm <id> # delete## Assets and folders-The library commands (`asset add/ls/mv/rm/export`, `folder ls/create/rename/mv/rm`) and the media-inspection commands (`probe`, `transcribe`, `analyze`, `visualize`, `frame`) are covered in [Working with media](assets.md).+The library commands (`asset add/ls/mv/rm/export`, `folder ls/create/rename/mv/rm`) and the media-inspection commands (`probe`, `transcribe`, `listen`, `visualize`, `frame`) are covered in [Working with media](assets.md).## Discoverydiff --git a/docs/getting-started.md b/docs/getting-started.mdindex f3025e6e..1dad9453 100644--- a/docs/getting-started.md+++ b/docs/getting-started.md@@ -75,4 +75,4 @@ Renders the active scene locally with sensible defaults (MP4, H.264, 1080p). Cod- [CLI reference](cli.md): the full command surface- [Writing compositions](composition.md): timing, paints, generative assets, captions-- [Working with media](assets.md): probe, transcribe, and analyze footage+- [Working with media](assets.md): probe, transcribe, and listen to footage