The VS Code of video editing, built for agents.
TypeScript
Refactor media inspection commands and enhance documentation
konstantin-paulusd52f63ec7652
diff --git a/README.md b/README.mdindex 8fd48f7e..d2a1b5d6 100644--- a/README.md+++ b/README.md@@ -62,7 +62,8 @@ Cutting footage requires looking at it. The CLI ships the inspection tools an ag```shdapi media probe clip.mp4 # container + codec metadata, like ffprobedapi media grab clip.mp4 -t 0 12 45 # decode frames to PNGs-dapi media visualize track.mp3 # waveform / filmstrip previews+dapi media filmstrip clip.mp4 # grid of video frames+dapi media waveform track.mp3 # audio waveform, silence flaggeddapi media transcribe interview.wav # timed, word-level transcriptdapi media listen interview.mp4 -p "what is said in the intro?" # ask a multimodal modeldapi node screenshot # see the canvas itself@@ -77,7 +78,7 @@ dapi node screenshot # see the canvas itself| `dapi mount` | Compile and mount a JSX composition || `dapi node …` | The scene graph: `ls`, `tree`, `grep`, `patch`, `insert`, `cp`, `rm`, `screenshot`, `export` || `dapi asset …` | The media library: `add`, `ls`, `tree`, `mv`, `rm`, `export` |-| `dapi media …` | Inspect a file by id or path: `probe`, `grab`, `visualize`, `transcribe`, `listen` |+| `dapi media …` | Inspect a file by id or path: `probe`, `grab`, `filmstrip`, `waveform`, `transcribe`, `listen` || `dapi project …` / `dapi folder …` / `dapi selection …` | Projects, library folders, canvas selection || `dapi models` / `dapi voices` / `dapi fonts` | Discover generation models, speech voices, local fonts |diff --git a/apps/cli/reference/README.md b/apps/cli/reference/README.mdindex 3676c2bf..e5927541 100644--- a/apps/cli/reference/README.md+++ b/apps/cli/reference/README.md@@ -21,7 +21,7 @@ How the surface is divided:- Declarative composition happens through `mount`, which renders a Solid JSX project into the canvas (see [jsx/](./jsx/README.md)); `node insert` runs the same pipeline but inserts the rendered nodes into an existing parent entity instead of mounting document roots.- Scenes are created declaratively via `mount` (`<scene key="...">`); there is no imperative scene command.- AI asset generation (image / video / speech / audio) is declared in the project module (`generate.*`, see [jsx/generate.md](./jsx/generate.md)) and produced on mount. `models` and `voices` list what those declarations can reference.-- Inspecting an existing asset (probe / transcribe / listen / visualize / grab) lives under `media`; writing an asset's original file back to disk is `asset export`.+- Inspecting an existing asset (probe / transcribe / listen / filmstrip / waveform / grab) lives under `media`; writing an asset's original file back to disk is `asset export`.- Organizing the library lives under `folder`; moving assets between folders is `asset mv`.## Commands@@ -76,7 +76,8 @@ How the surface is divided:- [`dapi media probe`](./media/probe.md): container and track metadata- [`dapi media transcribe`](./media/transcribe.md): timed speech transcript- [`dapi media grab`](./media/grab.md): decode video frames to PNGs-- [`dapi media visualize`](./media/visualize.md): waveform / filmstrip / thumbnail PNG+- [`dapi media filmstrip`](./media/filmstrip.md): grid of video frames as a PNG+- [`dapi media waveform`](./media/waveform.md): audio waveform PNG with silence highlighting- [`dapi media listen`](./media/listen.md): AI description of an audio track### Foldersdiff --git a/apps/cli/reference/media/filmstrip.md b/apps/cli/reference/media/filmstrip.mdnew file mode 100644index 00000000..beb35ce3--- /dev/null+++ b/apps/cli/reference/media/filmstrip.md@@ -0,0 +1,29 @@+# `dapi media filmstrip <id|path>`++Renders a **filmstrip** of a video to a PNG, written to a file in the system temp directory: a grid of frames sampled at even intervals across the window, each row stamped with a timestamp ruler. Video only; use [`dapi media waveform`](./waveform.md) to inspect the audio track. Renders locally; no credits. Alias: `film`.++Tick labels use `HH:MM:SS:FF` timecode (hours, minutes, seconds, frame within the second) at every zoom level, so labels stay comparable regardless of the window's span. Frames count against the video's frame rate.++## Input++- `<id|path>`: a video asset id, or a local video file to preview in place without adding it to the library (required).+- `-s, --start <time>`: start of the window to preview, a `Time` value in source/content time (optional; default `0`).+- `-e, --end <time>`: end of the window to preview, a `Time` value (optional; default the asset's duration).+- `-x, --scale <factor>`: scale factor for the thumbnails (optional; default `1`, clamped to `0.25`-`4`). The overall canvas size stays fixed, so smaller thumbnails fit **more rows and columns** (a denser grid sampling more moments) and larger thumbnails fit fewer but show more detail each.+- `-o, --output <path>`: write the PNG here instead of a temp file (optional).++## Output++One JSON object, the absolute path to the written PNG plus the grid geometry:++```ts+{+ path: string, // e.g. "/tmp/3f2c1a8e-....png", or the --output path+ column: { width: number, count: number, duration: number },+ row: { height: number, count: number, duration: number },+}+```++## Errors++Exits non-zero if the id is unknown, the asset isn't a video, `--start`/`--end` fall outside the asset or cross (`--start` >= `--end`), `--scale` isn't a positive number, or `--output` can't be written.diff --git a/apps/cli/reference/media/listen.md b/apps/cli/reference/media/listen.mdindex baca5387..08bb0fb6 100644--- a/apps/cli/reference/media/listen.md+++ b/apps/cli/reference/media/listen.md@@ -1,6 +1,6 @@# `dapi media listen <id|path>`-Puts a multimodal model in front of an audio track and returns its answer. With no prompt it returns a general description of what is heard; with `--prompt` it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). Accepts an audio file or a video, but only the audio track is analyzed: for what is on screen in a video, use `visualize` first.+Puts a multimodal model in front of an audio track and returns its answer. With no prompt it returns a general description of what is heard; with `--prompt` it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). Accepts an audio file or a video, but only the audio track is analyzed by default. Alias: `watch`.## Input@@ -8,15 +8,16 @@ Puts a multimodal model in front of an audio track and returns its answer. With- `-p, --prompt <str>`: question or instruction about the audio (optional; defaults to a general description).- `-s, --start <time>`: start of the segment to analyze, a `Time` value (optional; default `0`). Timestamps in the analysis are relative to this point.- `-e, --end <time>`: end of the segment to analyze, a `Time` value (optional; default the asset's duration).+- `--keep-video`: for a video asset, keep the video track instead of stripping it to audio, so the model can analyze what is on screen (optional; default off). Expensive: requires a full video upload.## Output-One JSON object, the model's answer:+One JSON object, the model's answer. `start`/`end` echo the analyzed window (in seconds) and are present only when `-s`/`-e` were given:```ts-{ analysis: string }+{ result: string, start?: number, end?: number }```## Errors-Exits non-zero if the id is unknown, or the asset isn't a video or audio asset.+Exits non-zero if the id is unknown, the asset isn't a video or audio asset, or `--start`/`--end` cross (`--start` >= `--end`).diff --git a/apps/cli/reference/media/visualize.md b/apps/cli/reference/media/visualize.mddeleted file mode 100644index fcf20a2f..00000000--- a/apps/cli/reference/media/visualize.md+++ /dev/null@@ -1,31 +0,0 @@-# `dapi media visualize <id|path>`--Renders a visual preview of an asset to a PNG, written to a file in the system temp directory. The visualization is chosen by the asset's media type, so a single command covers the whole library. Renders locally; no credits. Alias: `viz`.--## By media type--- **audio**: an amplitude waveform with a time axis, drawn from decoded audio peaks.-- **video**: a composite: a **filmstrip** of frames sampled at even intervals across the window, with the **waveform** of the video's audio track drawn beneath it, sharing one time axis. Videos with no audio track render the filmstrip alone.-- **image**: a thumbnail of the image, auto-scaled.--Tick labels use `HH:MM:SS:FF` timecode (hours, minutes, seconds, frame within the second) at every zoom level, so labels stay comparable regardless of the window's span. Frames count against the video's frame rate; the audio ruler has no video track and counts against a nominal 30 fps.--## Input--- `<id|path>`: an image, audio, or video asset id, or a local file to visualize in place without adding it to the library (required).-- `-s, --start <time>`: start of the window to visualize, a `Time` value in source/content time (optional; default `0`). Ignored for images.-- `-e, --end <time>`: end of the window to visualize, a `Time` value (optional; default the asset's duration). Ignored for images.-- `-x, --scale <factor>`: scale factor for the thumbnails (optional; default `1`, clamped to `0.25`-`4`). The overall canvas size stays fixed, so smaller thumbnails fit **more rows and columns** (a denser grid sampling more moments) and larger thumbnails fit fewer but show more detail each. For images, scales the output resolution instead (never above the source size).-- `-o, --output <path>`: write the PNG here instead of a temp file (optional).--## Output--One JSON object, the absolute path to the written PNG:--```ts-{ path: string } // e.g. "/tmp/3f2c1a8e-....png", or the --output path-```--## Errors--Exits non-zero if the id is unknown, the media type isn't visualizable (e.g. a transcript asset, or an audio asset with no decodable audio), `--start`/`--end` fall outside the asset or cross (`--start` >= `--end`), `--scale` isn't a positive number, or `--output` can't be written.diff --git a/apps/cli/reference/media/waveform.md b/apps/cli/reference/media/waveform.mdnew file mode 100644index 00000000..9ad30a50--- /dev/null+++ b/apps/cli/reference/media/waveform.md@@ -0,0 +1,29 @@+# `dapi media waveform <id|path>`++Renders the audio track of a video or audio file as an amplitude **waveform** PNG, written to a file in the system temp directory: loudness over time drawn from decoded audio peaks, with a timestamp ruler. Silent stretches are highlighted in red. Renders locally; no credits. Alias: `wave`.++Tick labels use `HH:MM:SS:FF` timecode (hours, minutes, seconds, frame within the second) at every zoom level, so labels stay comparable regardless of the window's span. For a video, frames count against the video's frame rate; for a standalone audio asset, the ruler counts against a nominal 30 fps.++## Input++- `<id|path>`: a video or audio asset id, or a local file to preview in place without adding it to the library (required).+- `-s, --start <time>`: start of the window to preview, a `Time` value in source/content time (optional; default `0`).+- `-e, --end <time>`: end of the window to preview, a `Time` value (optional; default the asset's duration).+- `-x, --scale <factor>`: scale factor for the waveform (optional; default `1`, clamped to `0.25`-`4`). The overall canvas size stays fixed, so a smaller scale fits **more rows and columns** (a denser time axis) and a larger scale fits fewer but taller rows.+- `-o, --output <path>`: write the PNG here instead of a temp file (optional).++## Output++One JSON object, the absolute path to the written PNG plus the grid geometry:++```ts+{+ path: string, // e.g. "/tmp/3f2c1a8e-....png", or the --output path+ column: { width: number, count: number, duration: number },+ row: { height: number, count: number, duration: number },+}+```++## Errors++Exits non-zero if the id is unknown, the asset has no decodable audio track, `--start`/`--end` fall outside the asset or cross (`--start` >= `--end`), `--scale` isn't a positive number, or `--output` can't be written.diff --git a/apps/cli/src/cli-channels.ts b/apps/cli/src/cli-channels.tsindex 0c4eeb9c..f1556f47 100644--- a/apps/cli/src/cli-channels.ts+++ b/apps/cli/src/cli-channels.ts@@ -129,13 +129,18 @@ export type TranscriptWord = { text: string; start: number; end: number };export type TranscriptSegment = { text: string; words: TranscriptWord[] };export type MediaTranscribeResult = { segments: TranscriptSegment[] };-export type MediaVisualizeRequest = AssetRef & { start?: number; end?: number; scale?: number };-export type MediaVisualizeResult = {+export type MediaFilmstripRequest = AssetRef & { start?: number; end?: number; scale?: number };+export type MediaFilmstripResult = {+ base64: string;+} & Record<string, unknown>;++export type MediaWaveformRequest = AssetRef & { start?: number; end?: number; scale?: number };+export type MediaWaveformResult = {base64: string;} & Record<string, unknown>;export type MediaListenRequest = AssetRef & { prompt?: string; start?: number; end?: number; stripVideo?: boolean };-export type MediaListenResult = { analysis?: string; start?: number; end?: number };+export type MediaListenResult = { result?: string; start?: number; end?: number };export type GeneratedAsset = { id: string; name: string; type: string };diff --git a/apps/cli/src/index.ts b/apps/cli/src/index.tsindex 9e9917e9..d3f51c22 100644--- a/apps/cli/src/index.ts+++ b/apps/cli/src/index.ts@@ -556,7 +556,7 @@ async function mediaListen(ref: string, opts: MediaListenOptions): Promise<void>}}-type MediaVisualizeOptions = { start?: string; end?: string; scale?: string; output?: string };+type MediaPreviewOptions = { start?: string; end?: string; scale?: string; output?: string };function parseTimeArg(value: string, flag: string): number {const seconds = parseTime(value);@@ -569,7 +569,8 @@ function parseTimeArg(value: string, flag: string): number {return seconds;}-async function mediaVisualize(ref: string, opts: MediaVisualizeOptions): Promise<void> {+// Parse the window/scale flags shared by `filmstrip` and `waveform`.+function parsePreviewWindow(opts: MediaPreviewOptions): { start?: number; end?: number; scale?: number } {const start = opts.start !== undefined ? parseTimeArg(opts.start, "--start") : undefined;const end = opts.end !== undefined ? parseTimeArg(opts.end, "--end") : undefined;if (start !== undefined && end !== undefined && start >= end) {@@ -586,11 +587,32 @@ async function mediaVisualize(ref: string, opts: MediaVisualizeOptions): Promise}}+ return { start, end, scale };+}++async function mediaFilmstrip(ref: string, opts: MediaPreviewOptions): Promise<void> {+ const { start, end, scale } = parsePreviewWindow(opts);const target = resolveAssetRef(ref);const path = opts.output ?? join(tmpdir(), `${randomUUID()}.png`);- const stop = startSpinner("Rendering visualization");+ const stop = startSpinner("Rendering filmstrip");try {- const { base64, ...rest } = await editor.media.visualize.query({ ...target, start, end, scale });+ const { base64, ...rest } = await editor.media.filmstrip.query({ ...target, start, end, scale });+ stop();+ writeFileSync(path, Buffer.from(base64, "base64"));+ console.log(JSON.stringify({ path, ...rest }));+ } catch (e) {+ stop();+ handleSocketError(e);+ }+}++async function mediaWaveform(ref: string, opts: MediaPreviewOptions): Promise<void> {+ const { start, end, scale } = parsePreviewWindow(opts);+ const target = resolveAssetRef(ref);+ const path = opts.output ?? join(tmpdir(), `${randomUUID()}.png`);+ const stop = startSpinner("Rendering waveform");+ try {+ const { base64, ...rest } = await editor.media.waveform.query({ ...target, start, end, scale });stop();writeFileSync(path, Buffer.from(base64, "base64"));console.log(JSON.stringify({ path, ...rest }));@@ -1088,6 +1110,7 @@ mediamedia.command("grab")+ .alias("sample").description(`Decode one or more frames of a video file and write them as high-res PNGs, each stamped in the top-left with its HH:MM:SS:FF timestamp. Best for inspecting individual frames in detail where a filmstrip is too coarse: seeks to the exact requested time with frame-level precision${docs("media/grab")}`).argument("<id|path>", "video asset id, or a local video file to grab frames from").option("-t, --time <time...>", `one or more timestamps to grab — seconds ("1.5"), frames ("45f"), or "MM:SS" (default: 0)`)@@ -1096,15 +1119,26 @@ media.action((ref: string, opts: MediaFrameOptions) => mediaFrame(ref, opts));media- .command("visualize")- .aliases(["viz", "filmstrip"])- .description(`Start here for any video: render a compact visual preview of a media file to a PNG — a filmstrip plus waveform for video, a waveform for audio, or a thumbnail for an image. Fast and token-efficient; narrow the window to zoom into a region of interest. The waveform shows loudness over time and marks the silent stretches${docs("media/visualize")}`)- .argument("<id|path>", "image, audio, or video asset id, or a local file to visualize")- .option("-s, --start <time>", `start of the window to visualize — seconds, "45f" frames, or "MM:SS" (default: 0)`)- .option("-e, --end <time>", `end of the window to visualize — seconds, "45f" frames, or "MM:SS" (default: asset duration)`)+ .command("filmstrip")+ .alias("film")+ .description(`Start here for any video: render a grid of thumbnails sampled across the timeline to a PNG, each row stamped with timestamps. Fast and token-efficient; narrow the window to zoom into a region of interest. Video only, no audio — use \`waveform\` to inspect the audio track${docs("media/filmstrip")}`)+ .argument("<id|path>", "video asset id, or a local video file to preview")+ .option("-s, --start <time>", `start of the window to preview — seconds, "45f" frames, or "MM:SS" (default: 0)`)+ .option("-e, --end <time>", `end of the window to preview — seconds, "45f" frames, or "MM:SS" (default: asset duration)`).option("-x, --scale <factor>", "scale factor for the thumbnails; smaller thumbnails fit more rows and columns, larger fit fewer (default: 1)").option("-o, --output <path>", "write the PNG here instead of a temp file")- .action((ref: string, opts: MediaVisualizeOptions) => mediaVisualize(ref, opts));+ .action((ref: string, opts: MediaPreviewOptions) => mediaFilmstrip(ref, opts));++media+ .command("waveform")+ .alias("wave")+ .description(`Render the audio track of a video or audio file as a waveform PNG with a timestamp ruler. Shows loudness over time and highlights the silent stretches in red. Fast and token-efficient; narrow the window to zoom into a region of interest${docs("media/waveform")}`)+ .argument("<id|path>", "video or audio asset id, or a local file to preview")+ .option("-s, --start <time>", `start of the window to preview — seconds, "45f" frames, or "MM:SS" (default: 0)`)+ .option("-e, --end <time>", `end of the window to preview — seconds, "45f" frames, or "MM:SS" (default: asset duration)`)+ .option("-x, --scale <factor>", "scale factor for the waveform; smaller fits more rows and columns, larger fits fewer (default: 1)")+ .option("-o, --output <path>", "write the PNG here instead of a temp file")+ .action((ref: string, opts: MediaPreviewOptions) => mediaWaveform(ref, opts));media.command("listen")diff --git a/apps/web/src/components/engine/utils/transcode.ts b/apps/web/src/components/engine/utils/transcode.tsindex 654cba67..4d6deeb9 100644--- a/apps/web/src/components/engine/utils/transcode.ts+++ b/apps/web/src/components/engine/utils/transcode.ts@@ -6,7 +6,7 @@ import { ALL_FORMATS, AudioSampleSink, BlobSource, BufferTarget, CanvasSink, Conimport { assert } from '@/utils';import { getAssetFile } from '../api/assets';import type { StreamTargetChunk } from 'mediabunny';-import type { Asset } from '../db';+import type { Asset, VideoAsset, AudioAsset } from '../db';export async function transcodeForTranscription(asset: Asset): Promise<File> {const blob = await getAssetFile(asset);@@ -75,12 +75,10 @@ export async function transcodeForAnalysis(asset: Asset, window?: TimeWindow & {return { readable, run };}-const MAX_IMAGE_SIZE = 560 ** 2;const PATCH_SIZE = 28;const MAX_WIDTH_IN_TOKENS = 52; // 92const MAX_HEIGHT_IN_TOKENS = 52;const THUMBNAIL_HEIGHT_IN_TOKENS = 5;-const WAVEFORM_HEIGHT_IN_TOKENS = 3;const RULER_HEIGHT_IN_TOKENS = 1;const AUDIO_WAVEFORM_HEIGHT_IN_TOKENS = 6;const AUDIO_COLUMN_WIDTH_IN_TOKENS = 6;@@ -100,13 +98,13 @@ export function formatTimestamp(seconds: number, frameRate: number): string {return [hh, mm, ss, frames].map((v) => String(v).padStart(2, '0')).join(':');}-export type AssetVisualization = {+export type RenderedPreview = {dataUrl: string;} & Record<string, unknown>;export type TimeWindow = { start?: number; end?: number };-export type VisualizeOptions = TimeWindow & { scale?: number };+export type PreviewOptions = TimeWindow & { scale?: number };const MIN_SCALE = 0.25;const MAX_SCALE = 4;@@ -126,221 +124,201 @@ function resolveWindow(duration: number, window?: TimeWindow): { start: number;return { start, end, duration: end - start };}-export async function visualizeAsset(asset: Asset, options?: VisualizeOptions): Promise<AssetVisualization> {+// Render a video as a grid of thumbnails sampled across the window, each row+// stamped with a timestamp ruler. Video only; no audio.+export async function filmstripAsset(asset: Asset, options?: PreviewOptions): Promise<RenderedPreview> {+ assert(asset.type === "VIDEO", "The filmstrip needs a video asset; use `waveform` for audio.");const blob = await getAssetFile(asset);const scale = resolveScale(options?.scale);-- if (asset.type === "IMAGE") {- const pixelCount = asset.width * asset.height;- const factor = Math.min(1, Math.sqrt(MAX_IMAGE_SIZE / pixelCount) * scale);-- const outputWidth = Math.floor(asset.width * factor);- const outputHeight = Math.floor(asset.height * factor);-- const bitmap = await createImageBitmap(blob);- const canvas = document.createElement('canvas');- canvas.width = outputWidth;- canvas.height = outputHeight;- const ctx = canvas.getContext('2d')!;- ctx.drawImage(bitmap, 0, 0, outputWidth, outputHeight);- bitmap.close();- return { dataUrl: canvas.toDataURL('image/png') };+ const input = new Input({ formats: ALL_FORMATS, source: new BlobSource(blob) });+ try {+ return await renderFilmstrip(asset, input, options, scale);+ } finally {+ input.dispose();}+}- if (asset.type === "VIDEO") {- const input = new Input({ formats: ALL_FORMATS, source: new BlobSource(blob) });- const videoTrack = await input.getPrimaryVideoTrack();- assert(videoTrack, "Video track not found");- const audioTrack = await input.getPrimaryAudioTrack();- const { start: windowStart, duration } = resolveWindow(asset.duration, options);- const aspect = asset.width / asset.height;- const frameRate = asset.frameRate;-- const fixedRowTokens = RULER_HEIGHT_IN_TOKENS + (audioTrack ? WAVEFORM_HEIGHT_IN_TOKENS : 0);- const thumbnailHeightInTokens = Math.min(- Math.max(Math.round(THUMBNAIL_HEIGHT_IN_TOKENS * scale), 1),- MAX_HEIGHT_IN_TOKENS - fixedRowTokens,- );- const rowHeightInTokens = thumbnailHeightInTokens + fixedRowTokens;-- const columnWidthInTokens = Math.min(- Math.max(Math.floor(thumbnailHeightInTokens * aspect), 1),- MAX_WIDTH_IN_TOKENS,- );+async function renderFilmstrip(asset: VideoAsset, input: Input, window: TimeWindow | undefined, scale: number): Promise<RenderedPreview> {+ const videoTrack = await input.getPrimaryVideoTrack();+ assert(videoTrack, "Video track not found");+ const { start: windowStart, duration } = resolveWindow(asset.duration, window);+ const aspect = asset.width / asset.height;+ const frameRate = asset.frameRate;++ const thumbnailHeightInTokens = Math.min(+ Math.max(Math.round(THUMBNAIL_HEIGHT_IN_TOKENS * scale), 1),+ MAX_HEIGHT_IN_TOKENS - RULER_HEIGHT_IN_TOKENS,+ );+ const rowHeightInTokens = thumbnailHeightInTokens + RULER_HEIGHT_IN_TOKENS;++ const columnWidthInTokens = Math.min(+ Math.max(Math.floor(thumbnailHeightInTokens * aspect), 1),+ MAX_WIDTH_IN_TOKENS,+ );++ const thumbnailWidthInPixels = columnWidthInTokens * PATCH_SIZE;+ const thumbnailHeightInPixels = thumbnailHeightInTokens * PATCH_SIZE;+ const rowHeightInPixels = rowHeightInTokens * PATCH_SIZE;++ const rows = Math.floor(MAX_HEIGHT_IN_TOKENS / rowHeightInTokens);+ const columns = Math.floor(MAX_WIDTH_IN_TOKENS / columnWidthInTokens);+ const cells = rows * columns;++ const secondsPerColumn = Math.max(duration / cells, 1 / frameRate);+ const secondsPerRow = secondsPerColumn * columns;++ const canvas = document.createElement('canvas');+ canvas.width = columnWidthInTokens * columns * PATCH_SIZE;+ canvas.height = rowHeightInTokens * rows * PATCH_SIZE;+ const ctx = canvas.getContext('2d')!;++ ctx.fillStyle = '#000';+ ctx.fillRect(0, 0, canvas.width, canvas.height);++ const sink = new CanvasSink(videoTrack, {+ width: thumbnailWidthInPixels,+ height: thumbnailHeightInPixels,+ fit: 'cover',+ poolSize: 1,+ });- const thumbnailWidthInPixels = columnWidthInTokens * PATCH_SIZE;- const thumbnailHeightInPixels = thumbnailHeightInTokens * PATCH_SIZE;- const rowHeightInPixels = rowHeightInTokens * PATCH_SIZE;+ const rulerHeight = RULER_HEIGHT_IN_TOKENS * PATCH_SIZE;- const rows = Math.floor(MAX_HEIGHT_IN_TOKENS / rowHeightInTokens);- const columns = Math.floor(MAX_WIDTH_IN_TOKENS / columnWidthInTokens);- const cells = rows * columns;+ const timestamps = Array.from({ length: cells }, (_, i) => windowStart + i * secondsPerColumn);+ const interator = sink.canvasesAtTimestamps(timestamps);+ for (let i = 0; i < cells; i++) {+ const wrapped = await interator.next();+ if (!wrapped.value?.canvas) continue;- const secondsPerColumn = Math.max(duration / cells, 1 / frameRate);- const secondsPerRow = secondsPerColumn * columns;+ const column = i % columns;+ const row = Math.floor(i / columns);- const canvas = document.createElement('canvas');- canvas.width = columnWidthInTokens * columns * PATCH_SIZE;- canvas.height = rowHeightInTokens * rows * PATCH_SIZE;- const ctx = canvas.getContext('2d')!;+ ctx.drawImage(+ wrapped.value.canvas,+ column * columnWidthInTokens * PATCH_SIZE,+ rulerHeight + row * rowHeightInTokens * PATCH_SIZE,+ thumbnailWidthInPixels,+ thumbnailHeightInPixels+ );+ }- ctx.fillStyle = '#000';- ctx.fillRect(0, 0, canvas.width, canvas.height);+ drawRuler(ctx, {+ cells,+ columns,+ columnWidthInPixels: columnWidthInTokens * PATCH_SIZE,+ rowHeightInPixels,+ rulerHeight,+ secondsPerColumn,+ frameRate,+ startOffset: windowStart,+ });- const sink = new CanvasSink(videoTrack, {+ return {+ dataUrl: canvas.toDataURL('image/png'),+ column: {width: thumbnailWidthInPixels,- height: thumbnailHeightInPixels,- fit: 'cover',- poolSize: 1,- });-- const rulerHeight = RULER_HEIGHT_IN_TOKENS * PATCH_SIZE;-- const timestamps = Array.from({ length: cells }, (_, i) => windowStart + i * secondsPerColumn);- const interator = sink.canvasesAtTimestamps(timestamps);- for (let i = 0; i < cells; i++) {- const wrapped = await interator.next();- if (!wrapped.value?.canvas) continue;-- const column = i % columns;- const row = Math.floor(i / columns);-- ctx.drawImage(- wrapped.value.canvas,- column * columnWidthInTokens * PATCH_SIZE,- rulerHeight + row * rowHeightInTokens * PATCH_SIZE,- thumbnailWidthInPixels,- thumbnailHeightInPixels- );- }-- // Skip the waveform when the browser's AudioDecoder can't handle the codec- if (audioTrack && await audioTrack.canDecode()) {- const peaksPerSecond = Math.round(thumbnailWidthInPixels / secondsPerColumn / WAVEFORM_SAMPLE_WIDTH);- const audioPeaks = await downsampleAudio(audioTrack, { peaksPerSecond, range: [windowStart, windowStart + duration] });-- drawWaveform(ctx, audioPeaks, {- rows,- rowWidth: columns * thumbnailWidthInPixels,- rowHeightInPixels,- waveformOffset: rulerHeight + thumbnailHeightInPixels,- waveformHeight: WAVEFORM_HEIGHT_IN_TOKENS * PATCH_SIZE,- peaksPerSecond,- align: 'bottom',- });- }-- drawRuler(ctx, {- cells,- columns,- columnWidthInPixels: columnWidthInTokens * PATCH_SIZE,- rowHeightInPixels,- rulerHeight,- secondsPerColumn,- frameRate,- startOffset: windowStart,- });+ count: columns,+ duration: secondsPerColumn,+ },+ row: {+ height: rowHeightInPixels,+ count: rows,+ duration: secondsPerRow,+ },+ };+}- return {- dataUrl: canvas.toDataURL('image/png'),- column: {- width: thumbnailWidthInPixels,- count: columns,- duration: secondsPerColumn,- },- row: {- height: rowHeightInPixels,- count: rows,- duration: secondsPerRow,- },- };+// Render the audio track of a video or audio file as a loudness-over-time+// waveform with a timestamp ruler and silent stretches highlighted.+export async function waveformAsset(asset: Asset, options?: PreviewOptions): Promise<RenderedPreview> {+ assert(asset.type === "VIDEO" || asset.type === "AUDIO", "The waveform needs a video or audio asset.");+ const blob = await getAssetFile(asset);+ const scale = resolveScale(options?.scale);+ const input = new Input({ formats: ALL_FORMATS, source: new BlobSource(blob) });+ try {+ return await renderWaveform(asset, input, options, scale);+ } finally {+ input.dispose();}+}- if (asset.type === "AUDIO") {- const input = new Input({ formats: ALL_FORMATS, source: new BlobSource(blob) });- try {- const audioTrack = await input.getPrimaryAudioTrack();- assert(audioTrack, "Audio track not found");- assert(- await audioTrack.canDecode(),- `This browser's audio decoder can't handle the "${audioTrack.getCodec() ?? "unknown"}" codec, so no waveform can be rendered.`,- );- const { start: windowStart, duration } = resolveWindow(asset.duration, options);-- const waveformHeightInTokens = Math.min(- Math.max(Math.round(AUDIO_WAVEFORM_HEIGHT_IN_TOKENS * scale), 1),- MAX_HEIGHT_IN_TOKENS - RULER_HEIGHT_IN_TOKENS,- );- const rowHeightInTokens = waveformHeightInTokens + RULER_HEIGHT_IN_TOKENS;- const columnWidthInTokens = Math.min(- Math.max(Math.round(AUDIO_COLUMN_WIDTH_IN_TOKENS * scale), 1),- MAX_WIDTH_IN_TOKENS,- );-- const rows = Math.floor(MAX_HEIGHT_IN_TOKENS / rowHeightInTokens);- const columns = Math.floor(MAX_WIDTH_IN_TOKENS / columnWidthInTokens);- const cells = rows * columns;-- const secondsPerColumn = duration / cells;- const secondsPerRow = secondsPerColumn * columns;-- const columnWidthInPixels = columnWidthInTokens * PATCH_SIZE;- const rowHeightInPixels = rowHeightInTokens * PATCH_SIZE;- const rulerHeight = RULER_HEIGHT_IN_TOKENS * PATCH_SIZE;- const waveformHeight = waveformHeightInTokens * PATCH_SIZE;- const rowWidth = columns * columnWidthInPixels;-- const canvas = document.createElement('canvas');- canvas.width = rowWidth;- canvas.height = rowHeightInPixels * rows;- const ctx = canvas.getContext('2d')!;-- ctx.fillStyle = '#000';- ctx.fillRect(0, 0, canvas.width, canvas.height);-- const peaksPerSecond = Math.round(columnWidthInPixels / secondsPerColumn / WAVEFORM_SAMPLE_WIDTH);- const audioPeaks = await downsampleAudio(audioTrack, { peaksPerSecond, range: [windowStart, windowStart + duration] });-- drawWaveform(ctx, audioPeaks, {- rows,- rowWidth,- rowHeightInPixels,- waveformOffset: rulerHeight,- waveformHeight,- peaksPerSecond,- align: 'center',- });-- drawRuler(ctx, {- cells,- columns,- columnWidthInPixels,- rowHeightInPixels,- rulerHeight,- secondsPerColumn,- frameRate: AUDIO_RULER_FRAME_RATE,- startOffset: windowStart,- });-- return {- dataUrl: canvas.toDataURL('image/png'),- column: {- width: columnWidthInPixels,- count: columns,- duration: secondsPerColumn,- },- row: {- height: rowHeightInPixels,- count: rows,- duration: secondsPerRow,- },- };- } finally {- input.dispose();- }- }+async function renderWaveform(asset: VideoAsset | AudioAsset, input: Input, window: TimeWindow | undefined, scale: number): Promise<RenderedPreview> {+ const audioTrack = await input.getPrimaryAudioTrack();+ assert(audioTrack, "No audio track found, so no waveform can be rendered.");+ assert(+ await audioTrack.canDecode(),+ `This browser's audio decoder can't handle the "${audioTrack.getCodec() ?? "unknown"}" codec, so no waveform can be rendered.`,+ );+ const { start: windowStart, duration } = resolveWindow(asset.duration, window);+ const rulerFrameRate = asset.type === "VIDEO" ? asset.frameRate : AUDIO_RULER_FRAME_RATE;++ const waveformHeightInTokens = Math.min(+ Math.max(Math.round(AUDIO_WAVEFORM_HEIGHT_IN_TOKENS * scale), 1),+ MAX_HEIGHT_IN_TOKENS - RULER_HEIGHT_IN_TOKENS,+ );+ const rowHeightInTokens = waveformHeightInTokens + RULER_HEIGHT_IN_TOKENS;+ const columnWidthInTokens = Math.min(+ Math.max(Math.round(AUDIO_COLUMN_WIDTH_IN_TOKENS * scale), 1),+ MAX_WIDTH_IN_TOKENS,+ );++ const rows = Math.floor(MAX_HEIGHT_IN_TOKENS / rowHeightInTokens);+ const columns = Math.floor(MAX_WIDTH_IN_TOKENS / columnWidthInTokens);+ const cells = rows * columns;++ const secondsPerColumn = duration / cells;+ const secondsPerRow = secondsPerColumn * columns;++ const columnWidthInPixels = columnWidthInTokens * PATCH_SIZE;+ const rowHeightInPixels = rowHeightInTokens * PATCH_SIZE;+ const rulerHeight = RULER_HEIGHT_IN_TOKENS * PATCH_SIZE;+ const waveformHeight = waveformHeightInTokens * PATCH_SIZE;+ const rowWidth = columns * columnWidthInPixels;++ const canvas = document.createElement('canvas');+ canvas.width = rowWidth;+ canvas.height = rowHeightInPixels * rows;+ const ctx = canvas.getContext('2d')!;++ ctx.fillStyle = '#000';+ ctx.fillRect(0, 0, canvas.width, canvas.height);++ const peaksPerSecond = Math.round(columnWidthInPixels / secondsPerColumn / WAVEFORM_SAMPLE_WIDTH);+ const audioPeaks = await downsampleAudio(audioTrack, { peaksPerSecond, range: [windowStart, windowStart + duration] });++ drawWaveform(ctx, audioPeaks, {+ rows,+ rowWidth,+ rowHeightInPixels,+ waveformOffset: rulerHeight,+ waveformHeight,+ peaksPerSecond,+ });++ drawRuler(ctx, {+ cells,+ columns,+ columnWidthInPixels,+ rowHeightInPixels,+ rulerHeight,+ secondsPerColumn,+ frameRate: rulerFrameRate,+ startOffset: windowStart,+ });- throw new Error(`Unsupported asset type: ${asset.type}`);+ return {+ dataUrl: canvas.toDataURL('image/png'),+ column: {+ width: columnWidthInPixels,+ count: columns,+ duration: secondsPerColumn,+ },+ row: {+ height: rowHeightInPixels,+ count: rows,+ duration: secondsPerRow,+ },+ };}type WaveformLayout = {@@ -350,13 +328,13 @@ type WaveformLayout = {waveformOffset: number; // y offset within a row where the waveform beginswaveformHeight: number;peaksPerSecond: number;- align: 'center' | 'bottom';};+// Draws a center-aligned waveform, then overlays the silent stretches in red.function drawWaveform(ctx: CanvasRenderingContext2D, audioPeaks: Uint8ClampedArray, layout: WaveformLayout) {- const { rows, rowWidth, rowHeightInPixels, waveformOffset, waveformHeight, peaksPerSecond, align } = layout;+ const { rows, rowWidth, rowHeightInPixels, waveformOffset, waveformHeight, peaksPerSecond } = layout;const peaksPerRow = Math.floor(rowWidth / WAVEFORM_SAMPLE_WIDTH);- const fullHeight = align === 'bottom' ? waveformHeight : waveformHeight / 2;+ const halfHeight = waveformHeight / 2;const waveformTopOf = (row: number) => row * rowHeightInPixels + waveformOffset;@@ -368,15 +346,10 @@ function drawWaveform(ctx: CanvasRenderingContext2D, audioPeaks: Uint8ClampedArrconst peakIndex = row * peaksPerRow + col;if (peakIndex >= audioPeaks.length) break;- const amplitude = (audioPeaks[peakIndex] / 255) * fullHeight;+ const amplitude = (audioPeaks[peakIndex] / 255) * halfHeight;const x = col * WAVEFORM_SAMPLE_WIDTH;- if (align === 'bottom') {- const barHeight = Math.max(amplitude, 1);- ctx.fillRect(x, top + waveformHeight - barHeight, WAVEFORM_SAMPLE_WIDTH, barHeight);- } else {- const centerY = top + fullHeight;- ctx.fillRect(x, centerY - amplitude, WAVEFORM_SAMPLE_WIDTH, Math.max(amplitude * 2, 1));- }+ const centerY = top + halfHeight;+ ctx.fillRect(x, centerY - amplitude, WAVEFORM_SAMPLE_WIDTH, Math.max(amplitude * 2, 1));}}diff --git a/apps/web/src/context/editor-api/api.tsx b/apps/web/src/context/editor-api/api.tsxindex 12c950f1..282c175e 100644--- a/apps/web/src/context/editor-api/api.tsx+++ b/apps/web/src/context/editor-api/api.tsx@@ -9,7 +9,7 @@ import { useAuth } from '@/context/auth';import { t, q, m, q0, m0 } from "@/lib/cli-rpc";import { handleContextGet } from "./context";import { handleAssetsAdd, handleAssetsList, handleAssetTree, handleAssetsDelete, handleAssetsMove, handleAssetsExport } from "./assets";-import { handleMediaProbe, handleMediaFrame, handleMediaTranscribe, handleMediaVisualize, handleMediaListen } from "./media";+import { handleMediaProbe, handleMediaFrame, handleMediaTranscribe, handleMediaFilmstrip, handleMediaWaveform, handleMediaListen } from "./media";import { handleFoldersList, handleFolderCreate, handleFolderRename, handleFoldersMove, handleFoldersDelete } from "./folders";import { handleSelectionFocus, handleSelectionList, handleSelectionSet } from "./selection";import { handleNodeList, handleNodeTree, handleNodeGrep, handleNodeScreenshot, handleNodeDelete, handleNodePatch, handleNodeDuplicate, handleNodeRender } from "./node";@@ -110,7 +110,8 @@ function createAppRouter({ getEngine, getUser, requireAuth, setParams }: AppRoutprobe: q(handleMediaProbe(getEngine)),frame: q(handleMediaFrame(getEngine)),transcribe: q(handleMediaTranscribe(getEngine)),- visualize: q(handleMediaVisualize(getEngine)),+ filmstrip: q(handleMediaFilmstrip(getEngine)),+ waveform: q(handleMediaWaveform(getEngine)),listen: q(requireAuth(handleMediaListen(getEngine))),}),folder: t.router({diff --git a/apps/web/src/context/editor-api/media.ts b/apps/web/src/context/editor-api/media.tsindex b5a78291..c912d6e2 100644--- a/apps/web/src/context/editor-api/media.ts+++ b/apps/web/src/context/editor-api/media.ts@@ -5,7 +5,7 @@import { ALL_FORMATS, BlobSource, CanvasSink, Input } from 'mediabunny';import { ElectronFileHandle } from '@/lib/electron-file-handle';import { trpc } from '@/lib/trpc';-import { uploadBlob, visualizeAsset, describeFileAsset, getAssetFile, formatTimestamp } from '@/components/engine';+import { uploadBlob, filmstripAsset, waveformAsset, describeFileAsset, getAssetFile, formatTimestamp } from '@/components/engine';import { assert } from '@/utils';import {transcodeForTranscription,@@ -16,7 +16,7 @@ import {import type { Engine } from "@/components/engine";import type { Asset } from "@/components/engine/db";-import type { MediaListenRequest, MediaListenResult, MediaFrameRequest, MediaProbeRequest, AssetRef, MediaTranscribeRequest, MediaTranscribeResult, MediaVisualizeRequest, MediaVisualizeResult, TranscriptSegment } from "@diffusionstudio/cli/channels";+import type { MediaListenRequest, MediaListenResult, MediaFrameRequest, MediaProbeRequest, AssetRef, MediaTranscribeRequest, MediaTranscribeResult, MediaFilmstripRequest, MediaFilmstripResult, MediaWaveformRequest, MediaWaveformResult, TranscriptSegment } from "@diffusionstudio/cli/channels";import type { Accessor } from "solid-js";/**@@ -229,11 +229,21 @@ function sliceTranscript(segments: TranscriptSegment[], start?: number, end?: nureturn sliced;}-export function handleMediaVisualize(engine: Accessor<Engine>) {- return async (req: MediaVisualizeRequest): Promise<MediaVisualizeResult> => {+export function handleMediaFilmstrip(engine: Accessor<Engine>) {+ return async (req: MediaFilmstripRequest): Promise<MediaFilmstripResult> => {const { world } = engine();const asset = await resolveAssetRef(world, req);- const { dataUrl, ...rest } = await visualizeAsset(asset, { start: req.start, end: req.end, scale: req.scale });+ const { dataUrl, ...rest } = await filmstripAsset(asset, { start: req.start, end: req.end, scale: req.scale });+ const base64 = dataUrl.slice(dataUrl.indexOf(",") + 1);+ return { base64, ...rest };+ };+}++export function handleMediaWaveform(engine: Accessor<Engine>) {+ return async (req: MediaWaveformRequest): Promise<MediaWaveformResult> => {+ const { world } = engine();+ const asset = await resolveAssetRef(world, req);+ const { dataUrl, ...rest } = await waveformAsset(asset, { start: req.start, end: req.end, scale: req.scale });const base64 = dataUrl.slice(dataUrl.indexOf(",") + 1);return { base64, ...rest };};@@ -278,7 +288,7 @@ export function handleMediaListen(engine: Accessor<Engine>) {const { analysis } = await trpc.analyze.mutate({ media: fileRef, prompt });- return { analysis, start, end };+ return { result: analysis, start, end };};}diff --git a/docs/assets.md b/docs/assets.mdindex 628885a2..80cf049b 100644--- a/docs/assets.md+++ b/docs/assets.md@@ -1,6 +1,6 @@# Working with media-The asset library holds a project's media. Beyond storage, the CLI ships an inspection toolchain under `dapi media` (probe, grab, visualize, transcribe, listen) so an agent can *look at* footage before cutting it. All of it runs locally except `listen` and `transcribe`, which use hosted models.+The asset library holds a project's media. Beyond storage, the CLI ships an inspection toolchain under `dapi media` (probe, grab, filmstrip, waveform, transcribe, listen) so an agent can *look at* footage before cutting it. All of it runs locally except `listen` and `transcribe`, which use hosted models.**Id or path:** every inspection command takes either an asset id or a local file path. A path is read directly, so `dapi media probe clip.mp4` works on a file that isn't in the project yet.@@ -48,13 +48,21 @@ dapi media grab <id|path> [-t <time...>] [-o <dir>]Decodes video frames at the given timestamps to PNGs, at the asset's full resolution. Prints `{ time, path }` per frame. (For the *composited* canvas instead of the raw asset, use `dapi node screenshot`.)-### `media visualize`+### `media filmstrip````sh-dapi media visualize <id|path> [-s <start>] [-e <end>] [-x <scale>] [-o <path>]+dapi media filmstrip <id|path> [-s <start>] [-e <end>] [-x <scale>] [-o <path>]```-Renders a one-image preview chosen by media type: audio gets an amplitude **waveform** with a time axis, video gets a **filmstrip** of evenly sampled frames with the audio waveform beneath it, and images get a thumbnail. `-s`/`-e` window the preview; `-x` trades thumbnail size against grid density. Prints `{ path }`.+Renders a video to a **filmstrip**: a grid of evenly sampled frames with a per-row timestamp ruler. Video only. `-s`/`-e` window the preview; `-x` trades thumbnail size against grid density. Prints `{ path }`.++### `media waveform`++```sh+dapi media waveform <id|path> [-s <start>] [-e <end>] [-x <scale>] [-o <path>]+```++Renders the audio track of a video or audio file as an amplitude **waveform** with a time axis, **highlighting silent stretches in red**. `-s`/`-e` window the preview; `-x` trades row height against grid density. Prints `{ path }`.### `media transcribe`@@ -79,7 +87,7 @@ Puts a multimodal model in front of an audio track. Without a prompt, returns adapi open -b ./shoot # project from footagedapi asset tree --depth 1 # what's here?dapi media probe gbHJ # what format is it?-dapi media visualize gbHJ # what does it look like over time?+dapi media filmstrip gbHJ # what does it look like over time?dapi media transcribe gbHJ # what is said, and when?dapi media listen gbHJ -p "what is said in the intro?"dapi mount cut.tsx # compose the editdiff --git a/docs/cli.md b/docs/cli.mdindex 9b20c8e4..b63fe847 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -176,7 +176,7 @@ dapi project rm <id> # delete## Assets and folders-The library commands (`asset add/ls/mv/rm/export`, `folder ls/create/rename/mv/rm`) and the media-inspection commands (`probe`, `transcribe`, `listen`, `visualize`, `frame`) are covered in [Working with media](assets.md).+The library commands (`asset add/ls/mv/rm/export`, `folder ls/create/rename/mv/rm`) and the media-inspection commands (`probe`, `transcribe`, `listen`, `filmstrip`, `waveform`, `frame`) are covered in [Working with media](assets.md).## Discovery