Skip to content

The VS Code of video editing, built for agents.

TypeScript

Refactor media inspection commands and enhance documentation

konstantin-paulusd52f63ec7652
diff --git a/README.md b/README.md
index 8fd48f7e..d2a1b5d6 100644
--- a/README.md
+++ b/README.md
@@ -62,7 +62,8 @@ Cutting footage requires looking at it. The CLI ships the inspection tools an ag
```sh
dapi media probe clip.mp4 # container + codec metadata, like ffprobe
dapi media grab clip.mp4 -t 0 12 45 # decode frames to PNGs
-dapi media visualize track.mp3 # waveform / filmstrip previews
+dapi media filmstrip clip.mp4 # grid of video frames
+dapi media waveform track.mp3 # audio waveform, silence flagged
dapi media transcribe interview.wav # timed, word-level transcript
dapi media listen interview.mp4 -p "what is said in the intro?" # ask a multimodal model
dapi node screenshot # see the canvas itself
@@ -77,7 +78,7 @@ dapi node screenshot # see the canvas itself
| `dapi mount` | Compile and mount a JSX composition |
| `dapi node …` | The scene graph: `ls`, `tree`, `grep`, `patch`, `insert`, `cp`, `rm`, `screenshot`, `export` |
| `dapi asset …` | The media library: `add`, `ls`, `tree`, `mv`, `rm`, `export` |
-| `dapi media …` | Inspect a file by id or path: `probe`, `grab`, `visualize`, `transcribe`, `listen` |
+| `dapi media …` | Inspect a file by id or path: `probe`, `grab`, `filmstrip`, `waveform`, `transcribe`, `listen` |
| `dapi project …` / `dapi folder …` / `dapi selection …` | Projects, library folders, canvas selection |
| `dapi models` / `dapi voices` / `dapi fonts` | Discover generation models, speech voices, local fonts |
diff --git a/apps/cli/reference/README.md b/apps/cli/reference/README.md
index 3676c2bf..e5927541 100644
--- a/apps/cli/reference/README.md
+++ b/apps/cli/reference/README.md
@@ -21,7 +21,7 @@ How the surface is divided:
- Declarative composition happens through `mount`, which renders a Solid JSX project into the canvas (see [jsx/](./jsx/README.md)); `node insert` runs the same pipeline but inserts the rendered nodes into an existing parent entity instead of mounting document roots.
- Scenes are created declaratively via `mount` (`<scene key="...">`); there is no imperative scene command.
- AI asset generation (image / video / speech / audio) is declared in the project module (`generate.*`, see [jsx/generate.md](./jsx/generate.md)) and produced on mount. `models` and `voices` list what those declarations can reference.
-- Inspecting an existing asset (probe / transcribe / listen / visualize / grab) lives under `media`; writing an asset's original file back to disk is `asset export`.
+- Inspecting an existing asset (probe / transcribe / listen / filmstrip / waveform / grab) lives under `media`; writing an asset's original file back to disk is `asset export`.
- Organizing the library lives under `folder`; moving assets between folders is `asset mv`.
## Commands
@@ -76,7 +76,8 @@ How the surface is divided:
- [`dapi media probe`](./media/probe.md): container and track metadata
- [`dapi media transcribe`](./media/transcribe.md): timed speech transcript
- [`dapi media grab`](./media/grab.md): decode video frames to PNGs
-- [`dapi media visualize`](./media/visualize.md): waveform / filmstrip / thumbnail PNG
+- [`dapi media filmstrip`](./media/filmstrip.md): grid of video frames as a PNG
+- [`dapi media waveform`](./media/waveform.md): audio waveform PNG with silence highlighting
- [`dapi media listen`](./media/listen.md): AI description of an audio track
### Folders
diff --git a/apps/cli/reference/media/filmstrip.md b/apps/cli/reference/media/filmstrip.md
new file mode 100644
index 00000000..beb35ce3
--- /dev/null
+++ b/apps/cli/reference/media/filmstrip.md
@@ -0,0 +1,29 @@
+# `dapi media filmstrip <id|path>`
+
+Renders a **filmstrip** of a video to a PNG, written to a file in the system temp directory: a grid of frames sampled at even intervals across the window, each row stamped with a timestamp ruler. Video only; use [`dapi media waveform`](./waveform.md) to inspect the audio track. Renders locally; no credits. Alias: `film`.
+
+Tick labels use `HH:MM:SS:FF` timecode (hours, minutes, seconds, frame within the second) at every zoom level, so labels stay comparable regardless of the window's span. Frames count against the video's frame rate.
+
+## Input
+
+- `<id|path>`: a video asset id, or a local video file to preview in place without adding it to the library (required).
+- `-s, --start <time>`: start of the window to preview, a `Time` value in source/content time (optional; default `0`).
+- `-e, --end <time>`: end of the window to preview, a `Time` value (optional; default the asset's duration).
+- `-x, --scale <factor>`: scale factor for the thumbnails (optional; default `1`, clamped to `0.25`-`4`). The overall canvas size stays fixed, so smaller thumbnails fit **more rows and columns** (a denser grid sampling more moments) and larger thumbnails fit fewer but show more detail each.
+- `-o, --output <path>`: write the PNG here instead of a temp file (optional).
+
+## Output
+
+One JSON object, the absolute path to the written PNG plus the grid geometry:
+
+```ts
+{
+ path: string, // e.g. "/tmp/3f2c1a8e-....png", or the --output path
+ column: { width: number, count: number, duration: number },
+ row: { height: number, count: number, duration: number },
+}
+```
+
+## Errors
+
+Exits non-zero if the id is unknown, the asset isn't a video, `--start`/`--end` fall outside the asset or cross (`--start` >= `--end`), `--scale` isn't a positive number, or `--output` can't be written.
diff --git a/apps/cli/reference/media/listen.md b/apps/cli/reference/media/listen.md
index baca5387..08bb0fb6 100644
--- a/apps/cli/reference/media/listen.md
+++ b/apps/cli/reference/media/listen.md
@@ -1,6 +1,6 @@
# `dapi media listen <id|path>`
-Puts a multimodal model in front of an audio track and returns its answer. With no prompt it returns a general description of what is heard; with `--prompt` it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). Accepts an audio file or a video, but only the audio track is analyzed: for what is on screen in a video, use `visualize` first.
+Puts a multimodal model in front of an audio track and returns its answer. With no prompt it returns a general description of what is heard; with `--prompt` it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). Accepts an audio file or a video, but only the audio track is analyzed by default. Alias: `watch`.
## Input
@@ -8,15 +8,16 @@ Puts a multimodal model in front of an audio track and returns its answer. With
- `-p, --prompt <str>`: question or instruction about the audio (optional; defaults to a general description).
- `-s, --start <time>`: start of the segment to analyze, a `Time` value (optional; default `0`). Timestamps in the analysis are relative to this point.
- `-e, --end <time>`: end of the segment to analyze, a `Time` value (optional; default the asset's duration).
+- `--keep-video`: for a video asset, keep the video track instead of stripping it to audio, so the model can analyze what is on screen (optional; default off). Expensive: requires a full video upload.
## Output
-One JSON object, the model's answer:
+One JSON object, the model's answer. `start`/`end` echo the analyzed window (in seconds) and are present only when `-s`/`-e` were given:
```ts
-{ analysis: string }
+{ result: string, start?: number, end?: number }
```
## Errors
-Exits non-zero if the id is unknown, or the asset isn't a video or audio asset.
+Exits non-zero if the id is unknown, the asset isn't a video or audio asset, or `--start`/`--end` cross (`--start` >= `--end`).
diff --git a/apps/cli/reference/media/visualize.md b/apps/cli/reference/media/visualize.md
deleted file mode 100644
index fcf20a2f..00000000
--- a/apps/cli/reference/media/visualize.md
+++ /dev/null
@@ -1,31 +0,0 @@
-# `dapi media visualize <id|path>`
-
-Renders a visual preview of an asset to a PNG, written to a file in the system temp directory. The visualization is chosen by the asset's media type, so a single command covers the whole library. Renders locally; no credits. Alias: `viz`.
-
-## By media type
-
-- **audio**: an amplitude waveform with a time axis, drawn from decoded audio peaks.
-- **video**: a composite: a **filmstrip** of frames sampled at even intervals across the window, with the **waveform** of the video's audio track drawn beneath it, sharing one time axis. Videos with no audio track render the filmstrip alone.
-- **image**: a thumbnail of the image, auto-scaled.
-
-Tick labels use `HH:MM:SS:FF` timecode (hours, minutes, seconds, frame within the second) at every zoom level, so labels stay comparable regardless of the window's span. Frames count against the video's frame rate; the audio ruler has no video track and counts against a nominal 30 fps.
-
-## Input
-
-- `<id|path>`: an image, audio, or video asset id, or a local file to visualize in place without adding it to the library (required).
-- `-s, --start <time>`: start of the window to visualize, a `Time` value in source/content time (optional; default `0`). Ignored for images.
-- `-e, --end <time>`: end of the window to visualize, a `Time` value (optional; default the asset's duration). Ignored for images.
-- `-x, --scale <factor>`: scale factor for the thumbnails (optional; default `1`, clamped to `0.25`-`4`). The overall canvas size stays fixed, so smaller thumbnails fit **more rows and columns** (a denser grid sampling more moments) and larger thumbnails fit fewer but show more detail each. For images, scales the output resolution instead (never above the source size).
-- `-o, --output <path>`: write the PNG here instead of a temp file (optional).
-
-## Output
-
-One JSON object, the absolute path to the written PNG:
-
-```ts
-{ path: string } // e.g. "/tmp/3f2c1a8e-....png", or the --output path
-```
-
-## Errors
-
-Exits non-zero if the id is unknown, the media type isn't visualizable (e.g. a transcript asset, or an audio asset with no decodable audio), `--start`/`--end` fall outside the asset or cross (`--start` >= `--end`), `--scale` isn't a positive number, or `--output` can't be written.
diff --git a/apps/cli/reference/media/waveform.md b/apps/cli/reference/media/waveform.md
new file mode 100644
index 00000000..9ad30a50
--- /dev/null
+++ b/apps/cli/reference/media/waveform.md
@@ -0,0 +1,29 @@
+# `dapi media waveform <id|path>`
+
+Renders the audio track of a video or audio file as an amplitude **waveform** PNG, written to a file in the system temp directory: loudness over time drawn from decoded audio peaks, with a timestamp ruler. Silent stretches are highlighted in red. Renders locally; no credits. Alias: `wave`.
+
+Tick labels use `HH:MM:SS:FF` timecode (hours, minutes, seconds, frame within the second) at every zoom level, so labels stay comparable regardless of the window's span. For a video, frames count against the video's frame rate; for a standalone audio asset, the ruler counts against a nominal 30 fps.
+
+## Input
+
+- `<id|path>`: a video or audio asset id, or a local file to preview in place without adding it to the library (required).
+- `-s, --start <time>`: start of the window to preview, a `Time` value in source/content time (optional; default `0`).
+- `-e, --end <time>`: end of the window to preview, a `Time` value (optional; default the asset's duration).
+- `-x, --scale <factor>`: scale factor for the waveform (optional; default `1`, clamped to `0.25`-`4`). The overall canvas size stays fixed, so a smaller scale fits **more rows and columns** (a denser time axis) and a larger scale fits fewer but taller rows.
+- `-o, --output <path>`: write the PNG here instead of a temp file (optional).
+
+## Output
+
+One JSON object, the absolute path to the written PNG plus the grid geometry:
+
+```ts
+{
+ path: string, // e.g. "/tmp/3f2c1a8e-....png", or the --output path
+ column: { width: number, count: number, duration: number },
+ row: { height: number, count: number, duration: number },
+}
+```
+
+## Errors
+
+Exits non-zero if the id is unknown, the asset has no decodable audio track, `--start`/`--end` fall outside the asset or cross (`--start` >= `--end`), `--scale` isn't a positive number, or `--output` can't be written.
diff --git a/apps/cli/src/cli-channels.ts b/apps/cli/src/cli-channels.ts
index 0c4eeb9c..f1556f47 100644
--- a/apps/cli/src/cli-channels.ts
+++ b/apps/cli/src/cli-channels.ts
@@ -129,13 +129,18 @@ export type TranscriptWord = { text: string; start: number; end: number };
export type TranscriptSegment = { text: string; words: TranscriptWord[] };
export type MediaTranscribeResult = { segments: TranscriptSegment[] };
-export type MediaVisualizeRequest = AssetRef & { start?: number; end?: number; scale?: number };
-export type MediaVisualizeResult = {
+export type MediaFilmstripRequest = AssetRef & { start?: number; end?: number; scale?: number };
+export type MediaFilmstripResult = {
+ base64: string;
+} & Record<string, unknown>;
+
+export type MediaWaveformRequest = AssetRef & { start?: number; end?: number; scale?: number };
+export type MediaWaveformResult = {
base64: string;
} & Record<string, unknown>;
export type MediaListenRequest = AssetRef & { prompt?: string; start?: number; end?: number; stripVideo?: boolean };
-export type MediaListenResult = { analysis?: string; start?: number; end?: number };
+export type MediaListenResult = { result?: string; start?: number; end?: number };
export type GeneratedAsset = { id: string; name: string; type: string };
diff --git a/apps/cli/src/index.ts b/apps/cli/src/index.ts
index 9e9917e9..d3f51c22 100644
--- a/apps/cli/src/index.ts
+++ b/apps/cli/src/index.ts
@@ -556,7 +556,7 @@ async function mediaListen(ref: string, opts: MediaListenOptions): Promise<void>
}
}
-type MediaVisualizeOptions = { start?: string; end?: string; scale?: string; output?: string };
+type MediaPreviewOptions = { start?: string; end?: string; scale?: string; output?: string };
function parseTimeArg(value: string, flag: string): number {
const seconds = parseTime(value);
@@ -569,7 +569,8 @@ function parseTimeArg(value: string, flag: string): number {
return seconds;
}
-async function mediaVisualize(ref: string, opts: MediaVisualizeOptions): Promise<void> {
+// Parse the window/scale flags shared by `filmstrip` and `waveform`.
+function parsePreviewWindow(opts: MediaPreviewOptions): { start?: number; end?: number; scale?: number } {
const start = opts.start !== undefined ? parseTimeArg(opts.start, "--start") : undefined;
const end = opts.end !== undefined ? parseTimeArg(opts.end, "--end") : undefined;
if (start !== undefined && end !== undefined && start >= end) {
@@ -586,11 +587,32 @@ async function mediaVisualize(ref: string, opts: MediaVisualizeOptions): Promise
}
}
+ return { start, end, scale };
+}
+
+async function mediaFilmstrip(ref: string, opts: MediaPreviewOptions): Promise<void> {
+ const { start, end, scale } = parsePreviewWindow(opts);
const target = resolveAssetRef(ref);
const path = opts.output ?? join(tmpdir(), `${randomUUID()}.png`);
- const stop = startSpinner("Rendering visualization");
+ const stop = startSpinner("Rendering filmstrip");
try {
- const { base64, ...rest } = await editor.media.visualize.query({ ...target, start, end, scale });
+ const { base64, ...rest } = await editor.media.filmstrip.query({ ...target, start, end, scale });
+ stop();
+ writeFileSync(path, Buffer.from(base64, "base64"));
+ console.log(JSON.stringify({ path, ...rest }));
+ } catch (e) {
+ stop();
+ handleSocketError(e);
+ }
+}
+
+async function mediaWaveform(ref: string, opts: MediaPreviewOptions): Promise<void> {
+ const { start, end, scale } = parsePreviewWindow(opts);
+ const target = resolveAssetRef(ref);
+ const path = opts.output ?? join(tmpdir(), `${randomUUID()}.png`);
+ const stop = startSpinner("Rendering waveform");
+ try {
+ const { base64, ...rest } = await editor.media.waveform.query({ ...target, start, end, scale });
stop();
writeFileSync(path, Buffer.from(base64, "base64"));
console.log(JSON.stringify({ path, ...rest }));
@@ -1088,6 +1110,7 @@ media
media
.command("grab")
+ .alias("sample")
.description(`Decode one or more frames of a video file and write them as high-res PNGs, each stamped in the top-left with its HH:MM:SS:FF timestamp. Best for inspecting individual frames in detail where a filmstrip is too coarse: seeks to the exact requested time with frame-level precision${docs("media/grab")}`)
.argument("<id|path>", "video asset id, or a local video file to grab frames from")
.option("-t, --time <time...>", `one or more timestamps to grab — seconds ("1.5"), frames ("45f"), or "MM:SS" (default: 0)`)
@@ -1096,15 +1119,26 @@ media
.action((ref: string, opts: MediaFrameOptions) => mediaFrame(ref, opts));
media
- .command("visualize")
- .aliases(["viz", "filmstrip"])
- .description(`Start here for any video: render a compact visual preview of a media file to a PNG — a filmstrip plus waveform for video, a waveform for audio, or a thumbnail for an image. Fast and token-efficient; narrow the window to zoom into a region of interest. The waveform shows loudness over time and marks the silent stretches${docs("media/visualize")}`)
- .argument("<id|path>", "image, audio, or video asset id, or a local file to visualize")
- .option("-s, --start <time>", `start of the window to visualize — seconds, "45f" frames, or "MM:SS" (default: 0)`)
- .option("-e, --end <time>", `end of the window to visualize — seconds, "45f" frames, or "MM:SS" (default: asset duration)`)
+ .command("filmstrip")
+ .alias("film")
+ .description(`Start here for any video: render a grid of thumbnails sampled across the timeline to a PNG, each row stamped with timestamps. Fast and token-efficient; narrow the window to zoom into a region of interest. Video only, no audio — use \`waveform\` to inspect the audio track${docs("media/filmstrip")}`)
+ .argument("<id|path>", "video asset id, or a local video file to preview")
+ .option("-s, --start <time>", `start of the window to preview — seconds, "45f" frames, or "MM:SS" (default: 0)`)
+ .option("-e, --end <time>", `end of the window to preview — seconds, "45f" frames, or "MM:SS" (default: asset duration)`)
.option("-x, --scale <factor>", "scale factor for the thumbnails; smaller thumbnails fit more rows and columns, larger fit fewer (default: 1)")
.option("-o, --output <path>", "write the PNG here instead of a temp file")
- .action((ref: string, opts: MediaVisualizeOptions) => mediaVisualize(ref, opts));
+ .action((ref: string, opts: MediaPreviewOptions) => mediaFilmstrip(ref, opts));
+
+media
+ .command("waveform")
+ .alias("wave")
+ .description(`Render the audio track of a video or audio file as a waveform PNG with a timestamp ruler. Shows loudness over time and highlights the silent stretches in red. Fast and token-efficient; narrow the window to zoom into a region of interest${docs("media/waveform")}`)
+ .argument("<id|path>", "video or audio asset id, or a local file to preview")
+ .option("-s, --start <time>", `start of the window to preview — seconds, "45f" frames, or "MM:SS" (default: 0)`)
+ .option("-e, --end <time>", `end of the window to preview — seconds, "45f" frames, or "MM:SS" (default: asset duration)`)
+ .option("-x, --scale <factor>", "scale factor for the waveform; smaller fits more rows and columns, larger fits fewer (default: 1)")
+ .option("-o, --output <path>", "write the PNG here instead of a temp file")
+ .action((ref: string, opts: MediaPreviewOptions) => mediaWaveform(ref, opts));
media
.command("listen")
diff --git a/apps/web/src/components/engine/utils/transcode.ts b/apps/web/src/components/engine/utils/transcode.ts
index 654cba67..4d6deeb9 100644
--- a/apps/web/src/components/engine/utils/transcode.ts
+++ b/apps/web/src/components/engine/utils/transcode.ts
@@ -6,7 +6,7 @@ import { ALL_FORMATS, AudioSampleSink, BlobSource, BufferTarget, CanvasSink, Con
import { assert } from '@/utils';
import { getAssetFile } from '../api/assets';
import type { StreamTargetChunk } from 'mediabunny';
-import type { Asset } from '../db';
+import type { Asset, VideoAsset, AudioAsset } from '../db';
export async function transcodeForTranscription(asset: Asset): Promise<File> {
const blob = await getAssetFile(asset);
@@ -75,12 +75,10 @@ export async function transcodeForAnalysis(asset: Asset, window?: TimeWindow & {
return { readable, run };
}
-const MAX_IMAGE_SIZE = 560 ** 2;
const PATCH_SIZE = 28;
const MAX_WIDTH_IN_TOKENS = 52; // 92
const MAX_HEIGHT_IN_TOKENS = 52;
const THUMBNAIL_HEIGHT_IN_TOKENS = 5;
-const WAVEFORM_HEIGHT_IN_TOKENS = 3;
const RULER_HEIGHT_IN_TOKENS = 1;
const AUDIO_WAVEFORM_HEIGHT_IN_TOKENS = 6;
const AUDIO_COLUMN_WIDTH_IN_TOKENS = 6;
@@ -100,13 +98,13 @@ export function formatTimestamp(seconds: number, frameRate: number): string {
return [hh, mm, ss, frames].map((v) => String(v).padStart(2, '0')).join(':');
}
-export type AssetVisualization = {
+export type RenderedPreview = {
dataUrl: string;
} & Record<string, unknown>;
export type TimeWindow = { start?: number; end?: number };
-export type VisualizeOptions = TimeWindow & { scale?: number };
+export type PreviewOptions = TimeWindow & { scale?: number };
const MIN_SCALE = 0.25;
const MAX_SCALE = 4;
@@ -126,221 +124,201 @@ function resolveWindow(duration: number, window?: TimeWindow): { start: number;
return { start, end, duration: end - start };
}
-export async function visualizeAsset(asset: Asset, options?: VisualizeOptions): Promise<AssetVisualization> {
+// Render a video as a grid of thumbnails sampled across the window, each row
+// stamped with a timestamp ruler. Video only; no audio.
+export async function filmstripAsset(asset: Asset, options?: PreviewOptions): Promise<RenderedPreview> {
+ assert(asset.type === "VIDEO", "The filmstrip needs a video asset; use `waveform` for audio.");
const blob = await getAssetFile(asset);
const scale = resolveScale(options?.scale);
-
- if (asset.type === "IMAGE") {
- const pixelCount = asset.width * asset.height;
- const factor = Math.min(1, Math.sqrt(MAX_IMAGE_SIZE / pixelCount) * scale);
-
- const outputWidth = Math.floor(asset.width * factor);
- const outputHeight = Math.floor(asset.height * factor);
-
- const bitmap = await createImageBitmap(blob);
- const canvas = document.createElement('canvas');
- canvas.width = outputWidth;
- canvas.height = outputHeight;
- const ctx = canvas.getContext('2d')!;
- ctx.drawImage(bitmap, 0, 0, outputWidth, outputHeight);
- bitmap.close();
- return { dataUrl: canvas.toDataURL('image/png') };
+ const input = new Input({ formats: ALL_FORMATS, source: new BlobSource(blob) });
+ try {
+ return await renderFilmstrip(asset, input, options, scale);
+ } finally {
+ input.dispose();
}
+}
- if (asset.type === "VIDEO") {
- const input = new Input({ formats: ALL_FORMATS, source: new BlobSource(blob) });
- const videoTrack = await input.getPrimaryVideoTrack();
- assert(videoTrack, "Video track not found");
- const audioTrack = await input.getPrimaryAudioTrack();
- const { start: windowStart, duration } = resolveWindow(asset.duration, options);
- const aspect = asset.width / asset.height;
- const frameRate = asset.frameRate;
-
- const fixedRowTokens = RULER_HEIGHT_IN_TOKENS + (audioTrack ? WAVEFORM_HEIGHT_IN_TOKENS : 0);
- const thumbnailHeightInTokens = Math.min(
- Math.max(Math.round(THUMBNAIL_HEIGHT_IN_TOKENS * scale), 1),
- MAX_HEIGHT_IN_TOKENS - fixedRowTokens,
- );
- const rowHeightInTokens = thumbnailHeightInTokens + fixedRowTokens;
-
- const columnWidthInTokens = Math.min(
- Math.max(Math.floor(thumbnailHeightInTokens * aspect), 1),
- MAX_WIDTH_IN_TOKENS,
- );
+async function renderFilmstrip(asset: VideoAsset, input: Input, window: TimeWindow | undefined, scale: number): Promise<RenderedPreview> {
+ const videoTrack = await input.getPrimaryVideoTrack();
+ assert(videoTrack, "Video track not found");
+ const { start: windowStart, duration } = resolveWindow(asset.duration, window);
+ const aspect = asset.width / asset.height;
+ const frameRate = asset.frameRate;
+
+ const thumbnailHeightInTokens = Math.min(
+ Math.max(Math.round(THUMBNAIL_HEIGHT_IN_TOKENS * scale), 1),
+ MAX_HEIGHT_IN_TOKENS - RULER_HEIGHT_IN_TOKENS,
+ );
+ const rowHeightInTokens = thumbnailHeightInTokens + RULER_HEIGHT_IN_TOKENS;
+
+ const columnWidthInTokens = Math.min(
+ Math.max(Math.floor(thumbnailHeightInTokens * aspect), 1),
+ MAX_WIDTH_IN_TOKENS,
+ );
+
+ const thumbnailWidthInPixels = columnWidthInTokens * PATCH_SIZE;
+ const thumbnailHeightInPixels = thumbnailHeightInTokens * PATCH_SIZE;
+ const rowHeightInPixels = rowHeightInTokens * PATCH_SIZE;
+
+ const rows = Math.floor(MAX_HEIGHT_IN_TOKENS / rowHeightInTokens);
+ const columns = Math.floor(MAX_WIDTH_IN_TOKENS / columnWidthInTokens);
+ const cells = rows * columns;
+
+ const secondsPerColumn = Math.max(duration / cells, 1 / frameRate);
+ const secondsPerRow = secondsPerColumn * columns;
+
+ const canvas = document.createElement('canvas');
+ canvas.width = columnWidthInTokens * columns * PATCH_SIZE;
+ canvas.height = rowHeightInTokens * rows * PATCH_SIZE;
+ const ctx = canvas.getContext('2d')!;
+
+ ctx.fillStyle = '#000';
+ ctx.fillRect(0, 0, canvas.width, canvas.height);
+
+ const sink = new CanvasSink(videoTrack, {
+ width: thumbnailWidthInPixels,
+ height: thumbnailHeightInPixels,
+ fit: 'cover',
+ poolSize: 1,
+ });
- const thumbnailWidthInPixels = columnWidthInTokens * PATCH_SIZE;
- const thumbnailHeightInPixels = thumbnailHeightInTokens * PATCH_SIZE;
- const rowHeightInPixels = rowHeightInTokens * PATCH_SIZE;
+ const rulerHeight = RULER_HEIGHT_IN_TOKENS * PATCH_SIZE;
- const rows = Math.floor(MAX_HEIGHT_IN_TOKENS / rowHeightInTokens);
- const columns = Math.floor(MAX_WIDTH_IN_TOKENS / columnWidthInTokens);
- const cells = rows * columns;
+ const timestamps = Array.from({ length: cells }, (_, i) => windowStart + i * secondsPerColumn);
+ const interator = sink.canvasesAtTimestamps(timestamps);
+ for (let i = 0; i < cells; i++) {
+ const wrapped = await interator.next();
+ if (!wrapped.value?.canvas) continue;
- const secondsPerColumn = Math.max(duration / cells, 1 / frameRate);
- const secondsPerRow = secondsPerColumn * columns;
+ const column = i % columns;
+ const row = Math.floor(i / columns);
- const canvas = document.createElement('canvas');
- canvas.width = columnWidthInTokens * columns * PATCH_SIZE;
- canvas.height = rowHeightInTokens * rows * PATCH_SIZE;
- const ctx = canvas.getContext('2d')!;
+ ctx.drawImage(
+ wrapped.value.canvas,
+ column * columnWidthInTokens * PATCH_SIZE,
+ rulerHeight + row * rowHeightInTokens * PATCH_SIZE,
+ thumbnailWidthInPixels,
+ thumbnailHeightInPixels
+ );
+ }
- ctx.fillStyle = '#000';
- ctx.fillRect(0, 0, canvas.width, canvas.height);
+ drawRuler(ctx, {
+ cells,
+ columns,
+ columnWidthInPixels: columnWidthInTokens * PATCH_SIZE,
+ rowHeightInPixels,
+ rulerHeight,
+ secondsPerColumn,
+ frameRate,
+ startOffset: windowStart,
+ });
- const sink = new CanvasSink(videoTrack, {
+ return {
+ dataUrl: canvas.toDataURL('image/png'),
+ column: {
width: thumbnailWidthInPixels,
- height: thumbnailHeightInPixels,
- fit: 'cover',
- poolSize: 1,
- });
-
- const rulerHeight = RULER_HEIGHT_IN_TOKENS * PATCH_SIZE;
-
- const timestamps = Array.from({ length: cells }, (_, i) => windowStart + i * secondsPerColumn);
- const interator = sink.canvasesAtTimestamps(timestamps);
- for (let i = 0; i < cells; i++) {
- const wrapped = await interator.next();
- if (!wrapped.value?.canvas) continue;
-
- const column = i % columns;
- const row = Math.floor(i / columns);
-
- ctx.drawImage(
- wrapped.value.canvas,
- column * columnWidthInTokens * PATCH_SIZE,
- rulerHeight + row * rowHeightInTokens * PATCH_SIZE,
- thumbnailWidthInPixels,
- thumbnailHeightInPixels
- );
- }
-
- // Skip the waveform when the browser's AudioDecoder can't handle the codec
- if (audioTrack && await audioTrack.canDecode()) {
- const peaksPerSecond = Math.round(thumbnailWidthInPixels / secondsPerColumn / WAVEFORM_SAMPLE_WIDTH);
- const audioPeaks = await downsampleAudio(audioTrack, { peaksPerSecond, range: [windowStart, windowStart + duration] });
-
- drawWaveform(ctx, audioPeaks, {
- rows,
- rowWidth: columns * thumbnailWidthInPixels,
- rowHeightInPixels,
- waveformOffset: rulerHeight + thumbnailHeightInPixels,
- waveformHeight: WAVEFORM_HEIGHT_IN_TOKENS * PATCH_SIZE,
- peaksPerSecond,
- align: 'bottom',
- });
- }
-
- drawRuler(ctx, {
- cells,
- columns,
- columnWidthInPixels: columnWidthInTokens * PATCH_SIZE,
- rowHeightInPixels,
- rulerHeight,
- secondsPerColumn,
- frameRate,
- startOffset: windowStart,
- });
+ count: columns,
+ duration: secondsPerColumn,
+ },
+ row: {
+ height: rowHeightInPixels,
+ count: rows,
+ duration: secondsPerRow,
+ },
+ };
+}
- return {
- dataUrl: canvas.toDataURL('image/png'),
- column: {
- width: thumbnailWidthInPixels,
- count: columns,
- duration: secondsPerColumn,
- },
- row: {
- height: rowHeightInPixels,
- count: rows,
- duration: secondsPerRow,
- },
- };
+// Render the audio track of a video or audio file as a loudness-over-time
+// waveform with a timestamp ruler and silent stretches highlighted.
+export async function waveformAsset(asset: Asset, options?: PreviewOptions): Promise<RenderedPreview> {
+ assert(asset.type === "VIDEO" || asset.type === "AUDIO", "The waveform needs a video or audio asset.");
+ const blob = await getAssetFile(asset);
+ const scale = resolveScale(options?.scale);
+ const input = new Input({ formats: ALL_FORMATS, source: new BlobSource(blob) });
+ try {
+ return await renderWaveform(asset, input, options, scale);
+ } finally {
+ input.dispose();
}
+}
- if (asset.type === "AUDIO") {
- const input = new Input({ formats: ALL_FORMATS, source: new BlobSource(blob) });
- try {
- const audioTrack = await input.getPrimaryAudioTrack();
- assert(audioTrack, "Audio track not found");
- assert(
- await audioTrack.canDecode(),
- `This browser's audio decoder can't handle the "${audioTrack.getCodec() ?? "unknown"}" codec, so no waveform can be rendered.`,
- );
- const { start: windowStart, duration } = resolveWindow(asset.duration, options);
-
- const waveformHeightInTokens = Math.min(
- Math.max(Math.round(AUDIO_WAVEFORM_HEIGHT_IN_TOKENS * scale), 1),
- MAX_HEIGHT_IN_TOKENS - RULER_HEIGHT_IN_TOKENS,
- );
- const rowHeightInTokens = waveformHeightInTokens + RULER_HEIGHT_IN_TOKENS;
- const columnWidthInTokens = Math.min(
- Math.max(Math.round(AUDIO_COLUMN_WIDTH_IN_TOKENS * scale), 1),
- MAX_WIDTH_IN_TOKENS,
- );
-
- const rows = Math.floor(MAX_HEIGHT_IN_TOKENS / rowHeightInTokens);
- const columns = Math.floor(MAX_WIDTH_IN_TOKENS / columnWidthInTokens);
- const cells = rows * columns;
-
- const secondsPerColumn = duration / cells;
- const secondsPerRow = secondsPerColumn * columns;
-
- const columnWidthInPixels = columnWidthInTokens * PATCH_SIZE;
- const rowHeightInPixels = rowHeightInTokens * PATCH_SIZE;
- const rulerHeight = RULER_HEIGHT_IN_TOKENS * PATCH_SIZE;
- const waveformHeight = waveformHeightInTokens * PATCH_SIZE;
- const rowWidth = columns * columnWidthInPixels;
-
- const canvas = document.createElement('canvas');
- canvas.width = rowWidth;
- canvas.height = rowHeightInPixels * rows;
- const ctx = canvas.getContext('2d')!;
-
- ctx.fillStyle = '#000';
- ctx.fillRect(0, 0, canvas.width, canvas.height);
-
- const peaksPerSecond = Math.round(columnWidthInPixels / secondsPerColumn / WAVEFORM_SAMPLE_WIDTH);
- const audioPeaks = await downsampleAudio(audioTrack, { peaksPerSecond, range: [windowStart, windowStart + duration] });
-
- drawWaveform(ctx, audioPeaks, {
- rows,
- rowWidth,
- rowHeightInPixels,
- waveformOffset: rulerHeight,
- waveformHeight,
- peaksPerSecond,
- align: 'center',
- });
-
- drawRuler(ctx, {
- cells,
- columns,
- columnWidthInPixels,
- rowHeightInPixels,
- rulerHeight,
- secondsPerColumn,
- frameRate: AUDIO_RULER_FRAME_RATE,
- startOffset: windowStart,
- });
-
- return {
- dataUrl: canvas.toDataURL('image/png'),
- column: {
- width: columnWidthInPixels,
- count: columns,
- duration: secondsPerColumn,
- },
- row: {
- height: rowHeightInPixels,
- count: rows,
- duration: secondsPerRow,
- },
- };
- } finally {
- input.dispose();
- }
- }
+async function renderWaveform(asset: VideoAsset | AudioAsset, input: Input, window: TimeWindow | undefined, scale: number): Promise<RenderedPreview> {
+ const audioTrack = await input.getPrimaryAudioTrack();
+ assert(audioTrack, "No audio track found, so no waveform can be rendered.");
+ assert(
+ await audioTrack.canDecode(),
+ `This browser's audio decoder can't handle the "${audioTrack.getCodec() ?? "unknown"}" codec, so no waveform can be rendered.`,
+ );
+ const { start: windowStart, duration } = resolveWindow(asset.duration, window);
+ const rulerFrameRate = asset.type === "VIDEO" ? asset.frameRate : AUDIO_RULER_FRAME_RATE;
+
+ const waveformHeightInTokens = Math.min(
+ Math.max(Math.round(AUDIO_WAVEFORM_HEIGHT_IN_TOKENS * scale), 1),
+ MAX_HEIGHT_IN_TOKENS - RULER_HEIGHT_IN_TOKENS,
+ );
+ const rowHeightInTokens = waveformHeightInTokens + RULER_HEIGHT_IN_TOKENS;
+ const columnWidthInTokens = Math.min(
+ Math.max(Math.round(AUDIO_COLUMN_WIDTH_IN_TOKENS * scale), 1),
+ MAX_WIDTH_IN_TOKENS,
+ );
+
+ const rows = Math.floor(MAX_HEIGHT_IN_TOKENS / rowHeightInTokens);
+ const columns = Math.floor(MAX_WIDTH_IN_TOKENS / columnWidthInTokens);
+ const cells = rows * columns;
+
+ const secondsPerColumn = duration / cells;
+ const secondsPerRow = secondsPerColumn * columns;
+
+ const columnWidthInPixels = columnWidthInTokens * PATCH_SIZE;
+ const rowHeightInPixels = rowHeightInTokens * PATCH_SIZE;
+ const rulerHeight = RULER_HEIGHT_IN_TOKENS * PATCH_SIZE;
+ const waveformHeight = waveformHeightInTokens * PATCH_SIZE;
+ const rowWidth = columns * columnWidthInPixels;
+
+ const canvas = document.createElement('canvas');
+ canvas.width = rowWidth;
+ canvas.height = rowHeightInPixels * rows;
+ const ctx = canvas.getContext('2d')!;
+
+ ctx.fillStyle = '#000';
+ ctx.fillRect(0, 0, canvas.width, canvas.height);
+
+ const peaksPerSecond = Math.round(columnWidthInPixels / secondsPerColumn / WAVEFORM_SAMPLE_WIDTH);
+ const audioPeaks = await downsampleAudio(audioTrack, { peaksPerSecond, range: [windowStart, windowStart + duration] });
+
+ drawWaveform(ctx, audioPeaks, {
+ rows,
+ rowWidth,
+ rowHeightInPixels,
+ waveformOffset: rulerHeight,
+ waveformHeight,
+ peaksPerSecond,
+ });
+
+ drawRuler(ctx, {
+ cells,
+ columns,
+ columnWidthInPixels,
+ rowHeightInPixels,
+ rulerHeight,
+ secondsPerColumn,
+ frameRate: rulerFrameRate,
+ startOffset: windowStart,
+ });
- throw new Error(`Unsupported asset type: ${asset.type}`);
+ return {
+ dataUrl: canvas.toDataURL('image/png'),
+ column: {
+ width: columnWidthInPixels,
+ count: columns,
+ duration: secondsPerColumn,
+ },
+ row: {
+ height: rowHeightInPixels,
+ count: rows,
+ duration: secondsPerRow,
+ },
+ };
}
type WaveformLayout = {
@@ -350,13 +328,13 @@ type WaveformLayout = {
waveformOffset: number; // y offset within a row where the waveform begins
waveformHeight: number;
peaksPerSecond: number;
- align: 'center' | 'bottom';
};
+// Draws a center-aligned waveform, then overlays the silent stretches in red.
function drawWaveform(ctx: CanvasRenderingContext2D, audioPeaks: Uint8ClampedArray, layout: WaveformLayout) {
- const { rows, rowWidth, rowHeightInPixels, waveformOffset, waveformHeight, peaksPerSecond, align } = layout;
+ const { rows, rowWidth, rowHeightInPixels, waveformOffset, waveformHeight, peaksPerSecond } = layout;
const peaksPerRow = Math.floor(rowWidth / WAVEFORM_SAMPLE_WIDTH);
- const fullHeight = align === 'bottom' ? waveformHeight : waveformHeight / 2;
+ const halfHeight = waveformHeight / 2;
const waveformTopOf = (row: number) => row * rowHeightInPixels + waveformOffset;
@@ -368,15 +346,10 @@ function drawWaveform(ctx: CanvasRenderingContext2D, audioPeaks: Uint8ClampedArr
const peakIndex = row * peaksPerRow + col;
if (peakIndex >= audioPeaks.length) break;
- const amplitude = (audioPeaks[peakIndex] / 255) * fullHeight;
+ const amplitude = (audioPeaks[peakIndex] / 255) * halfHeight;
const x = col * WAVEFORM_SAMPLE_WIDTH;
- if (align === 'bottom') {
- const barHeight = Math.max(amplitude, 1);
- ctx.fillRect(x, top + waveformHeight - barHeight, WAVEFORM_SAMPLE_WIDTH, barHeight);
- } else {
- const centerY = top + fullHeight;
- ctx.fillRect(x, centerY - amplitude, WAVEFORM_SAMPLE_WIDTH, Math.max(amplitude * 2, 1));
- }
+ const centerY = top + halfHeight;
+ ctx.fillRect(x, centerY - amplitude, WAVEFORM_SAMPLE_WIDTH, Math.max(amplitude * 2, 1));
}
}
diff --git a/apps/web/src/context/editor-api/api.tsx b/apps/web/src/context/editor-api/api.tsx
index 12c950f1..282c175e 100644
--- a/apps/web/src/context/editor-api/api.tsx
+++ b/apps/web/src/context/editor-api/api.tsx
@@ -9,7 +9,7 @@ import { useAuth } from '@/context/auth';
import { t, q, m, q0, m0 } from "@/lib/cli-rpc";
import { handleContextGet } from "./context";
import { handleAssetsAdd, handleAssetsList, handleAssetTree, handleAssetsDelete, handleAssetsMove, handleAssetsExport } from "./assets";
-import { handleMediaProbe, handleMediaFrame, handleMediaTranscribe, handleMediaVisualize, handleMediaListen } from "./media";
+import { handleMediaProbe, handleMediaFrame, handleMediaTranscribe, handleMediaFilmstrip, handleMediaWaveform, handleMediaListen } from "./media";
import { handleFoldersList, handleFolderCreate, handleFolderRename, handleFoldersMove, handleFoldersDelete } from "./folders";
import { handleSelectionFocus, handleSelectionList, handleSelectionSet } from "./selection";
import { handleNodeList, handleNodeTree, handleNodeGrep, handleNodeScreenshot, handleNodeDelete, handleNodePatch, handleNodeDuplicate, handleNodeRender } from "./node";
@@ -110,7 +110,8 @@ function createAppRouter({ getEngine, getUser, requireAuth, setParams }: AppRout
probe: q(handleMediaProbe(getEngine)),
frame: q(handleMediaFrame(getEngine)),
transcribe: q(handleMediaTranscribe(getEngine)),
- visualize: q(handleMediaVisualize(getEngine)),
+ filmstrip: q(handleMediaFilmstrip(getEngine)),
+ waveform: q(handleMediaWaveform(getEngine)),
listen: q(requireAuth(handleMediaListen(getEngine))),
}),
folder: t.router({
diff --git a/apps/web/src/context/editor-api/media.ts b/apps/web/src/context/editor-api/media.ts
index b5a78291..c912d6e2 100644
--- a/apps/web/src/context/editor-api/media.ts
+++ b/apps/web/src/context/editor-api/media.ts
@@ -5,7 +5,7 @@
import { ALL_FORMATS, BlobSource, CanvasSink, Input } from 'mediabunny';
import { ElectronFileHandle } from '@/lib/electron-file-handle';
import { trpc } from '@/lib/trpc';
-import { uploadBlob, visualizeAsset, describeFileAsset, getAssetFile, formatTimestamp } from '@/components/engine';
+import { uploadBlob, filmstripAsset, waveformAsset, describeFileAsset, getAssetFile, formatTimestamp } from '@/components/engine';
import { assert } from '@/utils';
import {
transcodeForTranscription,
@@ -16,7 +16,7 @@ import {
import type { Engine } from "@/components/engine";
import type { Asset } from "@/components/engine/db";
-import type { MediaListenRequest, MediaListenResult, MediaFrameRequest, MediaProbeRequest, AssetRef, MediaTranscribeRequest, MediaTranscribeResult, MediaVisualizeRequest, MediaVisualizeResult, TranscriptSegment } from "@diffusionstudio/cli/channels";
+import type { MediaListenRequest, MediaListenResult, MediaFrameRequest, MediaProbeRequest, AssetRef, MediaTranscribeRequest, MediaTranscribeResult, MediaFilmstripRequest, MediaFilmstripResult, MediaWaveformRequest, MediaWaveformResult, TranscriptSegment } from "@diffusionstudio/cli/channels";
import type { Accessor } from "solid-js";
/**
@@ -229,11 +229,21 @@ function sliceTranscript(segments: TranscriptSegment[], start?: number, end?: nu
return sliced;
}
-export function handleMediaVisualize(engine: Accessor<Engine>) {
- return async (req: MediaVisualizeRequest): Promise<MediaVisualizeResult> => {
+export function handleMediaFilmstrip(engine: Accessor<Engine>) {
+ return async (req: MediaFilmstripRequest): Promise<MediaFilmstripResult> => {
const { world } = engine();
const asset = await resolveAssetRef(world, req);
- const { dataUrl, ...rest } = await visualizeAsset(asset, { start: req.start, end: req.end, scale: req.scale });
+ const { dataUrl, ...rest } = await filmstripAsset(asset, { start: req.start, end: req.end, scale: req.scale });
+ const base64 = dataUrl.slice(dataUrl.indexOf(",") + 1);
+ return { base64, ...rest };
+ };
+}
+
+export function handleMediaWaveform(engine: Accessor<Engine>) {
+ return async (req: MediaWaveformRequest): Promise<MediaWaveformResult> => {
+ const { world } = engine();
+ const asset = await resolveAssetRef(world, req);
+ const { dataUrl, ...rest } = await waveformAsset(asset, { start: req.start, end: req.end, scale: req.scale });
const base64 = dataUrl.slice(dataUrl.indexOf(",") + 1);
return { base64, ...rest };
};
@@ -278,7 +288,7 @@ export function handleMediaListen(engine: Accessor<Engine>) {
const { analysis } = await trpc.analyze.mutate({ media: fileRef, prompt });
- return { analysis, start, end };
+ return { result: analysis, start, end };
};
}
diff --git a/docs/assets.md b/docs/assets.md
index 628885a2..80cf049b 100644
--- a/docs/assets.md
+++ b/docs/assets.md
@@ -1,6 +1,6 @@
# Working with media
-The asset library holds a project's media. Beyond storage, the CLI ships an inspection toolchain under `dapi media` (probe, grab, visualize, transcribe, listen) so an agent can *look at* footage before cutting it. All of it runs locally except `listen` and `transcribe`, which use hosted models.
+The asset library holds a project's media. Beyond storage, the CLI ships an inspection toolchain under `dapi media` (probe, grab, filmstrip, waveform, transcribe, listen) so an agent can *look at* footage before cutting it. All of it runs locally except `listen` and `transcribe`, which use hosted models.
**Id or path:** every inspection command takes either an asset id or a local file path. A path is read directly, so `dapi media probe clip.mp4` works on a file that isn't in the project yet.
@@ -48,13 +48,21 @@ dapi media grab <id|path> [-t <time...>] [-o <dir>]
Decodes video frames at the given timestamps to PNGs, at the asset's full resolution. Prints `{ time, path }` per frame. (For the *composited* canvas instead of the raw asset, use `dapi node screenshot`.)
-### `media visualize`
+### `media filmstrip`
```sh
-dapi media visualize <id|path> [-s <start>] [-e <end>] [-x <scale>] [-o <path>]
+dapi media filmstrip <id|path> [-s <start>] [-e <end>] [-x <scale>] [-o <path>]
```
-Renders a one-image preview chosen by media type: audio gets an amplitude **waveform** with a time axis, video gets a **filmstrip** of evenly sampled frames with the audio waveform beneath it, and images get a thumbnail. `-s`/`-e` window the preview; `-x` trades thumbnail size against grid density. Prints `{ path }`.
+Renders a video to a **filmstrip**: a grid of evenly sampled frames with a per-row timestamp ruler. Video only. `-s`/`-e` window the preview; `-x` trades thumbnail size against grid density. Prints `{ path }`.
+
+### `media waveform`
+
+```sh
+dapi media waveform <id|path> [-s <start>] [-e <end>] [-x <scale>] [-o <path>]
+```
+
+Renders the audio track of a video or audio file as an amplitude **waveform** with a time axis, **highlighting silent stretches in red**. `-s`/`-e` window the preview; `-x` trades row height against grid density. Prints `{ path }`.
### `media transcribe`
@@ -79,7 +87,7 @@ Puts a multimodal model in front of an audio track. Without a prompt, returns a
dapi open -b ./shoot # project from footage
dapi asset tree --depth 1 # what's here?
dapi media probe gbHJ # what format is it?
-dapi media visualize gbHJ # what does it look like over time?
+dapi media filmstrip gbHJ # what does it look like over time?
dapi media transcribe gbHJ # what is said, and when?
dapi media listen gbHJ -p "what is said in the intro?"
dapi mount cut.tsx # compose the edit
diff --git a/docs/cli.md b/docs/cli.md
index 9b20c8e4..b63fe847 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -176,7 +176,7 @@ dapi project rm <id> # delete
## Assets and folders
-The library commands (`asset add/ls/mv/rm/export`, `folder ls/create/rename/mv/rm`) and the media-inspection commands (`probe`, `transcribe`, `listen`, `visualize`, `frame`) are covered in [Working with media](assets.md).
+The library commands (`asset add/ls/mv/rm/export`, `folder ls/create/rename/mv/rm`) and the media-inspection commands (`probe`, `transcribe`, `listen`, `filmstrip`, `waveform`, `frame`) are covered in [Working with media](assets.md).
## Discovery