Skip to content

The VS Code of video editing, built for agents.

TypeScript

Refactor media tool documentation and update command references

konstantin-paulus35f4cf6c2670
diff --git a/docs/guides/prompts/media-listen.md b/docs/guides/prompts/media-listen.md
index ac35f5fa..53b520aa 100644
--- a/docs/guides/prompts/media-listen.md
+++ b/docs/guides/prompts/media-listen.md
@@ -1,6 +1,6 @@
# Writing prompts for `listen`
-[`listen`](../../reference/tools/media/listen.md) puts a multimodal model in front of an audio track and answers a prompt about it. A good prompt is specific about what you want and asks for timestamps so the answer lines up with the timeline. With no prompt it returns a general description; the examples below shape that into something more useful.
+[`listen`](../../reference/tools/listen.md) puts a multimodal model in front of an audio track and answers a prompt about it. A good prompt is specific about what you want and asks for timestamps so the answer lines up with the timeline. With no prompt it returns a general description; the examples below shape that into something more useful.
## Examples
When you refer to a specific moment inside a prompt, use `MM:SS` (e.g. `01:15` for one minute fifteen seconds).
diff --git a/docs/reference/jsx/styles.md b/docs/reference/jsx/styles.md
index ea911e60..f8b03dfc 100644
--- a/docs/reference/jsx/styles.md
+++ b/docs/reference/jsx/styles.md
@@ -68,7 +68,7 @@ A filter over the parent's **rendered pixels** — its fills, strokes and childr
## `<mask>`
-A matte limiting the `<effect>` holding it: a **picture** whose alpha says where the effect applies, fitted into the node's box the way the node fits its footage. It is what the editor's object mask tool makes — pick an object in a video clip, track it, and the frames it found are written to the library and named by `src` — and what [`generate`](../tools/generate.md)'s [`sam-2.1`](../tools/media/segment.md) makes from a tool call. Several under one effect intersect. Without a `src` a `<mask>` does nothing.
+A matte limiting the `<effect>` holding it: a **picture** whose alpha says where the effect applies, fitted into the node's box the way the node fits its footage. It is what the editor's object mask tool makes — pick an object in a video clip, track it, and the frames it found are written to the library and named by `src` — and what [`generate`](../tools/generate.md)'s [`sam-2.1`](../tools/segment.md) makes from a tool call. Several under one effect intersect. Without a `src` a `<mask>` does nothing.
Under an `"opacity"` effect the mask is the cut-out: the clip shows inside the mask and goes transparent outside, as a mask on Premiere's Opacity does. Under any other effect the effect shows through inside the mask and the picture is untouched outside: an inverted mask on a `"blur"` blurs the background behind a person.
diff --git a/docs/reference/jsx/timing.md b/docs/reference/jsx/timing.md
index 92542d0b..c0ddb67d 100644
--- a/docs/reference/jsx/timing.md
+++ b/docs/reference/jsx/timing.md
@@ -62,4 +62,4 @@ The canonical internal unit is frames at **30 fps**; all formats are converted o
| `"MM:SS"` | `"02:30"` | Minutes and seconds. |
| `"HH:MM:SS"` | `"01:02:30"` | Hours, minutes, seconds. |
-CLI flags documented as taking a `Time` value (e.g. `--times` on [`capture`](../tools/capture.md) and [`media grab`](../tools/media/grab.md)) accept the same formats. Times in command **output** are plain seconds.
+CLI flags documented as taking a `Time` value (e.g. `--times` on [`capture`](../tools/capture.md) and [`grab`](../tools/grab.md)) accept the same formats. Times in command **output** are plain seconds.
diff --git a/docs/reference/models.md b/docs/reference/models.md
index ee7dbc75..ea7feba8 100644
--- a/docs/reference/models.md
+++ b/docs/reference/models.md
@@ -101,4 +101,4 @@ These run in the app, on this machine's GPU: no account, no credits. A job is sh
| Model | | Makes | Fields |
| --- | --- | --- | --- |
-| `sam-2.1` | SAM 2.1 — segments an object in a video and tracks it through the footage; `size` tiny (default), small, base-plus or large | a mask file (`.mask`) for a [`<mask src>`](./jsx/styles.md#mask) | [segment.md](./tools/media/segment.md) |
+| `sam-2.1` | SAM 2.1 — segments an object in a video and tracks it through the footage; `size` tiny (default), small, base-plus or large | a mask file (`.mask`) for a [`<mask src>`](./jsx/styles.md#mask) | [segment.md](./tools/segment.md) |
diff --git a/docs/reference/tools/README.md b/docs/reference/tools/README.md
index 23a84079..0706c3ac 100644
--- a/docs/reference/tools/README.md
+++ b/docs/reference/tools/README.md
@@ -37,12 +37,12 @@ Every tool runs inside the app, so the app has to be running. Over MCP that is a
| [`capture`](./capture.md) | `diffusion capture` | Capture frames |
| [`check`](./check.md) | `diffusion check` | Check structure |
| [`export`](./export.md) | `diffusion export` | Export scene |
-| [`probe`](./media/probe.md) | `diffusion probe` | Probe media |
-| [`grab`](./media/grab.md) | `diffusion grab` | Grab frames |
-| [`transcribe`](./media/transcribe.md) | `diffusion transcribe` | Transcribe speech |
-| [`filmstrip`](./media/filmstrip.md) | `diffusion filmstrip` | Filmstrip preview |
-| [`waveform`](./media/waveform.md) | `diffusion waveform` | Waveform preview |
-| [`listen`](./media/listen.md) | `diffusion listen` | Listen to audio |
+| [`probe`](./probe.md) | `diffusion probe` | Probe media |
+| [`grab`](./grab.md) | `diffusion grab` | Grab frames |
+| [`transcribe`](./transcribe.md) | `diffusion transcribe` | Transcribe speech |
+| [`filmstrip`](./filmstrip.md) | `diffusion filmstrip` | Filmstrip preview |
+| [`waveform`](./waveform.md) | `diffusion waveform` | Waveform preview |
+| [`listen`](./listen.md) | `diffusion listen` | Listen to audio |
| [`generate`](./generate.md) | `diffusion generate` | Generate media |
| [`job`](./job.md) | `diffusion job` | Generation job |
| [`logs`](./logs.md) | `diffusion logs` | App logs |
@@ -54,8 +54,8 @@ Every tool runs inside the app, so the app has to be running. Over MCP that is a
How the surface is divided:
- **The project loop.** [`open`](./open.md) a folder, edit its JSX, [`context`](./context.md) for what the source cannot say, [`capture`](./capture.md) and [`check`](./check.md) to verify, [`export`](./export.md) when asked.
-- **Media inspection** ([`probe`](./media/probe.md), [`grab`](./media/grab.md), [`transcribe`](./media/transcribe.md), [`filmstrip`](./media/filmstrip.md), [`waveform`](./media/waveform.md), [`listen`](./media/listen.md)): a file by path, without adding it to the project. Absolute paths and URLs work with or without an open project; library paths (`b-roll/clip.mp4`) need one.
-- **Generation.** [`generate`](./generate.md) runs a model of the Diffusion Studio API (images, video, music, speech, and tools such as upscaling) on the account's credits, or a model on this machine for free (with `estimate`, it only prices the request) — [`sam-2.1`](./media/segment.md) segments and tracks an object in footage, making the mask file a `<mask src>` names; poll [`job`](./job.md) until it has ended, and its files are saved into the library or a directory. The models and their fields are in [models.md](../models.md).
+- **Media inspection** ([`probe`](./probe.md), [`grab`](./grab.md), [`transcribe`](./transcribe.md), [`filmstrip`](./filmstrip.md), [`waveform`](./waveform.md), [`listen`](./listen.md)): a file by path, without adding it to the project. Absolute paths and URLs work with or without an open project; library paths (`b-roll/clip.mp4`) need one.
+- **Generation.** [`generate`](./generate.md) runs a model of the Diffusion Studio API (images, video, music, speech, and tools such as upscaling) on the account's credits, or a model on this machine for free (with `estimate`, it only prices the request) — [`sam-2.1`](./segment.md) segments and tracks an object in footage, making the mask file a `<mask src>` names; poll [`job`](./job.md) until it has ended, and its files are saved into the library or a directory. The models and their fields are in [models.md](../models.md).
- **What a declaration may name.** [`fonts`](./fonts.md).
- **The app and the machine.** [`logs`](./logs.md), [`screenshot`](./screenshot.md), [`window`](./window.md), [`report`](./report.md).
diff --git a/docs/reference/tools/capture.md b/docs/reference/tools/capture.md
index c1473b4c..1d105ec9 100644
--- a/docs/reference/tools/capture.md
+++ b/docs/reference/tools/capture.md
@@ -21,7 +21,7 @@ Render single frames of a scene to PNGs — each frame is the frame an export of
Each frame is **the frame an export of that scene would encode**: the scene is re-rendered from a fresh mount at its own size, position `0` is the workarea's first frame, and the requested positions are evaluated in timeline order, forward only — the way an export advances — so a composition whose look depends on having played (an `<html>` node's own animation state, for instance) captures exactly as it exports. A position past the workarea's end is not an error: the scene keeps playing past it, and the frame is what plays there — just not one an export would include.
-Scenes only: a single element renders inside its scene, so capture the scene at the times the element plays. To grab a video asset's own pixels instead of a composited frame, use [`grab`](./media/grab.md).
+Scenes only: a single element renders inside its scene, so capture the scene at the times the element plays. To grab a video asset's own pixels instead of a composited frame, use [`grab`](./grab.md).
With `separate`, each position is rendered at 720p height and named after its timecode (e.g. `01s12f.png`). Without an `output` directory the images land in a fresh `dapi-capture-*` directory under the system temp directory, so runs never overwrite each other. Writing into the same directory twice overwrites images whose name matches; with `separate`, requested times that land on the same frame share one file.
diff --git a/docs/reference/tools/media/filmstrip.md b/docs/reference/tools/filmstrip.md
similarity index 100%
rename from docs/reference/tools/media/filmstrip.md
rename to docs/reference/tools/filmstrip.md
diff --git a/docs/reference/tools/generate.md b/docs/reference/tools/generate.md
index bb01267a..dc89d9ba 100644
--- a/docs/reference/tools/generate.md
+++ b/docs/reference/tools/generate.md
@@ -1,6 +1,6 @@
# generate
-Start a job on a generative model — images, video, music, sound effects, speech, and tools such as background removal, upscaling, transcription and analysis on the Diffusion Studio API, and object segmentation and tracking ([`sam-2.1`](./media/segment.md)) on this machine — and return the job at once, with its estimated run time and the credits it costs. Poll [`job`](./job.md) with its id until it has ended; the files it made are then saved and their paths returned. With [`estimate`](#estimating), nothing starts: the answer is only what the job would cost and how long it would run. API models need a signed-in account and spend its credits; [local models](../models.md#local) run on the GPU, free, and need no account.
+Start a job on a generative model — images, video, music, sound effects, speech, and tools such as background removal, upscaling, transcription and analysis on the Diffusion Studio API, and object segmentation and tracking ([`sam-2.1`](./segment.md)) on this machine — and return the job at once, with its estimated run time and the credits it costs. Poll [`job`](./job.md) with its id until it has ended; the files it made are then saved and their paths returned. With [`estimate`](#estimating), nothing starts: the answer is only what the job would cost and how long it would run. API models need a signed-in account and spend its credits; [local models](../models.md#local) run on the GPU, free, and need no account.
| | |
| --- | --- |
@@ -63,7 +63,7 @@ Files (`{ "path": … }`) are uploaded first, as for a job, since a tool such as
- **Several files are numbered.** A job that makes more than one (an image `count` above 1) saves `fox-1.png`, `fox-2.png`, …; a single file is saved at the path itself.
- **A library path** (`b-roll/fox.png`, needs an open project) puts the file into the project's library, under `assets/`, named by `src` like any other asset, and notes the job it came from, as the app's own generations do. A file the project made already at that path is replaced, and everything naming it shows the new one; a file linked from elsewhere on disk is never written over, and the result gets a free name next to it instead.
- **An absolute path** saves the file there, replacing a file already there.
-- **Left out**, the file goes into the open project's library at its root under the API's name (`red-fox-at-dawn.png`, numbered if taken), or where a local model files its own (`sam-2.1`: `masks/<video>/Tracking <n>.mask`); with no project open, into a fresh directory of the job's own under the system temp dir, as [`transcribe`](./media/transcribe.md) and the other inspection tools do.
+- **Left out**, the file goes into the open project's library at its root under the API's name (`red-fox-at-dawn.png`, numbered if taken), or where a local model files its own (`sam-2.1`: `masks/<video>/Tracking <n>.mask`); with no project open, into a fresh directory of the job's own under the system temp dir, as [`transcribe`](./transcribe.md) and the other inspection tools do.
The files are saved by the first [`job`](./job.md) call that sees the job succeed (a local model's library files as soon as it does); later calls return the same paths. A file an API model made that went into the library records the job in `assets.yml` (its `job` field), so what made it can be looked up later.
diff --git a/docs/reference/tools/media/grab.md b/docs/reference/tools/grab.md
similarity index 96%
rename from docs/reference/tools/media/grab.md
rename to docs/reference/tools/grab.md
index cefc5093..598d7db6 100644
--- a/docs/reference/tools/media/grab.md
+++ b/docs/reference/tools/grab.md
@@ -32,7 +32,7 @@ Three ways to say which frames, mutually exclusive:
- `count`: that many frames evenly spaced across the clip, or across the `start`/`end` window, at a fixed interval of `window / count`, starting at the window start.
- `auto`: a scan at 2 fps that keeps a frame each time the picture settles into a new visual state, dropping near-duplicates and waiting out transitions so picks stay sharp. Returns at most `count` frames (default cap 30); static footage such as a screen recording returns far fewer. Needs WebGPU.
-`start` and `end` only apply with `count` or `auto`. An `end` past the asset's duration is clamped to it; a `start` at or past the end is an error, since the window would be empty. Like [`capture`](../capture.md), but this grabs the asset's own pixels; `capture` renders the composited node. Renders locally; no credits. Past ~12 frames, [`filmstrip`](./filmstrip.md) is the cheaper way to scan a clip.
+`start` and `end` only apply with `count` or `auto`. An `end` past the asset's duration is clamped to it; a `start` at or past the end is an error, since the window would be empty. Like [`capture`](./capture.md), but this grabs the asset's own pixels; `capture` renders the composited node. Renders locally; no credits. Past ~12 frames, [`filmstrip`](./filmstrip.md) is the cheaper way to scan a clip.
`quality` is a pixel budget rather than a box: a frame is scaled down, aspect ratio kept, until it holds no more pixels than the preset's square (`small` is 384², so a 16:9 frame becomes 512x288; `medium` 768² gives 1024x576; `large` 1536² gives 2048x1152), and never enlarged, so `large` on 1080p footage is the native 1920x1080.
diff --git a/docs/reference/tools/media/listen.md b/docs/reference/tools/listen.md
similarity index 94%
rename from docs/reference/tools/media/listen.md
rename to docs/reference/tools/listen.md
index 48b8cc36..be70e632 100644
--- a/docs/reference/tools/media/listen.md
+++ b/docs/reference/tools/listen.md
@@ -16,7 +16,7 @@ Prompt a multimodal model for a semantic analysis of an audio track and return i
| `start` | `Time` | `-s, --start <time>` | start of the segment to analyze (default: 0); timestamps in the analysis are relative to this point |
| `end` | `Time` | `-e, --end <time>` | end of the segment to analyze (default: media duration) |
-With no `prompt` it returns a general description of what is heard; with one it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). See [the prompt guide](../../../guides/prompts/media-listen.md) for prompts that work.
+With no `prompt` it returns a general description of what is heard; with one it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). See [the prompt guide](../../guides/prompts/media-listen.md) for prompts that work.
## Output
diff --git a/docs/reference/tools/media/probe.md b/docs/reference/tools/probe.md
similarity index 100%
rename from docs/reference/tools/media/probe.md
rename to docs/reference/tools/probe.md
diff --git a/docs/reference/tools/media/segment.md b/docs/reference/tools/segment.md
similarity index 93%
rename from docs/reference/tools/media/segment.md
rename to docs/reference/tools/segment.md
index f4b5d3b8..3af32834 100644
--- a/docs/reference/tools/media/segment.md
+++ b/docs/reference/tools/segment.md
@@ -1,15 +1,15 @@
# sam-2.1
-SAM 2.1 segments an object in a video and tracks it through the footage, making the mask file (`.mask`) a [`<mask src>`](../../jsx/styles.md#mask) names — the file the editor's object mask tool makes. It runs in the app, on this machine's GPU: no account, no credits. Like every [local model](../../models.md#local), its job is polled with [`job`](../job.md) and lasts only as long as the app.
+SAM 2.1 segments an object in a video and tracks it through the footage, making the mask file (`.mask`) a [`<mask src>`](../jsx/styles.md#mask) names — the file the editor's object mask tool makes. It runs in the app, on this machine's GPU: no account, no credits. Like every [local model](../models.md#local), its job is polled with [`job`](./job.md) and lasts only as long as the app.
| | |
| --- | --- |
-| Model | `sam-2.1`, run with [`generate`](../generate.md) |
+| Model | `sam-2.1`, run with [`generate`](./generate.md) |
| CLI | `diffusion generate sam-2.1 '<fields>' [-o <path>]` |
## Input
-`model: "sam-2.1"` plus these fields; `output` is [`generate`](../generate.md#where-the-files-go)'s.
+`model: "sam-2.1"` plus these fields; `output` is [`generate`](./generate.md#where-the-files-go)'s.
| Field | Type | Description |
| --- | --- | --- |
diff --git a/docs/reference/tools/media/transcribe.md b/docs/reference/tools/transcribe.md
similarity index 100%
rename from docs/reference/tools/media/transcribe.md
rename to docs/reference/tools/transcribe.md
diff --git a/docs/reference/tools/media/waveform.md b/docs/reference/tools/waveform.md
similarity index 100%
rename from docs/reference/tools/media/waveform.md
rename to docs/reference/tools/waveform.md
diff --git a/docs/skills/editor.md b/docs/skills/editor.md
index ac68433b..60c35b23 100644
--- a/docs/skills/editor.md
+++ b/docs/skills/editor.md
@@ -46,7 +46,7 @@ How to confirm a change actually produced what you intended. A clean save does n
- Use the built-in tags for the media a composition is made of (audio, video, images, captions).
- Hoist the properties that define the composition's look — title copy, font family and size, accent colors, key padding — into top-level consts annotated with `@inspect`, so they become live controls in the app's inspector.
- For anything 3D, use Three.js drawn into a `<surface>` tag.
-- To cut an object out of footage, or confine an effect to it, track it with `generate`'s local `sam-2.1` model (preview the prompt first, see [sam-2.1](../reference/tools/media/segment.md)) and name the mask file in a `<mask>` under the clip's `<effect>`. The track runs as a job: keep working, and poll `job` until it has succeeded before naming the mask.
+- To cut an object out of footage, or confine an effect to it, track it with `generate`'s local `sam-2.1` model (preview the prompt first, see [sam-2.1](../reference/tools/segment.md)) and name the mask file in a `<mask>` under the clip's `<effect>`. The track runs as a job: keep working, and poll `job` until it has succeeded before naming the mask.
- For motion graphics, overlays and UI-heavy graphics, the `<html>` tag driven by a paused [anime.js](https://animejs.com) timeline
- Before animating anything, read the [easings reference](../guides/motion/easings.md) and choose easings deliberately — default or linear easing is what makes motion read as a slideshow.
- For tasks that don't need an editing UI, keep the app in the background: from a shell, `diffusion open -b <dir>` launches it that way. Over MCP the app is already running, and `open` only opens the folder.
diff --git a/packages/dapi/src/catalog.ts b/packages/dapi/src/catalog.ts
index 4b2eb96b..2fe20759 100644
--- a/packages/dapi/src/catalog.ts
+++ b/packages/dapi/src/catalog.ts
@@ -12,12 +12,12 @@ import { check } from "./tools/check";
import { exportScene } from "./tools/export";
import { logs } from "./tools/logs";
import { screenshot } from "./tools/screenshot";
-import { mediaProbe } from "./tools/media-probe";
-import { mediaGrab } from "./tools/media-grab";
-import { mediaTranscribe } from "./tools/media-transcribe";
-import { mediaFilmstrip } from "./tools/media-filmstrip";
-import { mediaWaveform } from "./tools/media-waveform";
-import { mediaListen } from "./tools/media-listen";
+import { probe } from "./tools/probe";
+import { grab } from "./tools/grab";
+import { transcribe } from "./tools/transcribe";
+import { filmstrip } from "./tools/filmstrip";
+import { waveform } from "./tools/waveform";
+import { listen } from "./tools/listen";
import { generate, job } from "./tools/generate";
import { fonts } from "./tools/fonts";
import { report } from "./tools/report";
@@ -34,12 +34,12 @@ export const catalog = [
capture,
check,
exportScene,
- mediaProbe,
- mediaGrab,
- mediaTranscribe,
- mediaFilmstrip,
- mediaWaveform,
- mediaListen,
+ probe,
+ grab,
+ transcribe,
+ filmstrip,
+ waveform,
+ listen,
generate,
job,
logs,
diff --git a/packages/dapi/src/index.ts b/packages/dapi/src/index.ts
index d82d4148..6efafe90 100644
--- a/packages/dapi/src/index.ts
+++ b/packages/dapi/src/index.ts
@@ -24,7 +24,7 @@ export { JSON_SCHEMA_DIALECT, toolJsonSchemas } from "./json-schema";
export type { ToolJsonSchemas } from "./json-schema";
export { DAPI_WIRE } from "./ipc";
export type { DapiCall, DapiCancel, DapiReply } from "./ipc";
-export { FRAME_CAP } from "./tools/media-grab";
+export { FRAME_CAP } from "./tools/grab";
export { ISSUE_LOG_TAIL } from "./tools/report";
export { LOG_TAIL, LOG_MESSAGE_MAX } from "./tools/logs";
export { FONT_LIMIT } from "./tools/fonts";
@@ -47,8 +47,8 @@ export type { LocalModelId, LocalRequest, SamDetails, SamRequest } from "./local
import type { ImageRef as ImageRefSchema, LogEntry as LogEntrySchema, LogLevel as LogLevelSchema, TimecodedImage as TimecodedImageSchema } from "./schemas";
import type { CheckIssue as CheckIssueSchema, CheckIssueCode as CheckIssueCodeSchema } from "./tools/check";
import type { ExportFormat as ExportFormatSchema, ExportSettings as ExportSettingsSchema } from "./tools/export";
-import type { FrameQuality as FrameQualitySchema } from "./tools/media-grab";
-import type { TranscriptSegment as TranscriptSegmentSchema, TranscriptWord as TranscriptWordSchema } from "./tools/media-transcribe";
+import type { FrameQuality as FrameQualitySchema } from "./tools/grab";
+import type { TranscriptSegment as TranscriptSegmentSchema, TranscriptWord as TranscriptWordSchema } from "./tools/transcribe";
import type { FontFamily as FontFamilySchema } from "./tools/fonts";
import type { ToolArgs, ToolOutput, ToolResult } from "./catalog";
@@ -77,17 +77,17 @@ export type ExportResult = ToolResult<"export">;
export type LogsRequest = ToolArgs<"logs">;
export type ScreenshotResult = ToolResult<"screenshot">;
export type ScreenshotOutput = ToolOutput<"screenshot">;
-export type MediaProbeRequest = ToolArgs<"media_probe">;
-export type MediaFrameRequest = ToolArgs<"media_grab">;
-export type MediaFrameResult = ToolResult<"media_grab">;
-export type MediaTranscribeRequest = ToolArgs<"media_transcribe">;
-export type MediaTranscribeResult = ToolResult<"media_transcribe">;
-export type MediaFilmstripRequest = ToolArgs<"media_filmstrip">;
-export type MediaFilmstripResult = ToolResult<"media_filmstrip">;
-export type MediaWaveformRequest = ToolArgs<"media_waveform">;
-export type MediaWaveformResult = ToolResult<"media_waveform">;
-export type MediaListenRequest = ToolArgs<"media_listen">;
-export type MediaListenResult = ToolResult<"media_listen">;
+export type MediaProbeRequest = ToolArgs<"probe">;
+export type MediaFrameRequest = ToolArgs<"grab">;
+export type MediaFrameResult = ToolResult<"grab">;
+export type MediaTranscribeRequest = ToolArgs<"transcribe">;
+export type MediaTranscribeResult = ToolResult<"transcribe">;
+export type MediaFilmstripRequest = ToolArgs<"filmstrip">;
+export type MediaFilmstripResult = ToolResult<"filmstrip">;
+export type MediaWaveformRequest = ToolArgs<"waveform">;
+export type MediaWaveformResult = ToolResult<"waveform">;
+export type MediaListenRequest = ToolArgs<"listen">;
+export type MediaListenResult = ToolResult<"listen">;
export type GenerateRequest = ToolArgs<"generate">;
export type GenerationResult = ToolResult<"job">;
export type GenerationJob = ToolOutput<"job">;
diff --git a/packages/dapi/src/schemas.ts b/packages/dapi/src/schemas.ts
index 0539d2a6..248e717c 100644
--- a/packages/dapi/src/schemas.ts
+++ b/packages/dapi/src/schemas.ts
@@ -25,7 +25,7 @@ export const AssetPath = z
"absolute file path or URL (works with or without an open project), or a library path like `b-roll/clip.mp4` (needs an open project)",
);
-/** Beyond this the cells get too small to be worth the tokens; use `media_filmstrip`. */
+/** Beyond this the cells get too small to be worth the tokens; use `filmstrip`. */
export const MAX_FRAMES_PER_SHEET = 12;
/**
diff --git a/packages/dapi/src/tools/capture.ts b/packages/dapi/src/tools/capture.ts
index 7b997a9e..37a37639 100644
--- a/packages/dapi/src/tools/capture.ts
+++ b/packages/dapi/src/tools/capture.ts
@@ -11,7 +11,7 @@ export const capture = defineTool({
name: "capture",
title: "Capture frames",
description:
- "Render single frames of a scene to PNGs — each frame is the frame an export of that scene would encode, drawn offscreen at the scene's own size. By default the positions are merged into contact sheets: up to 12 per image, each cell labelled with its timecode (`08s10f`, zero segments dropped) and rendered as large as fits, so a few positions arrive as one high-resolution picture instead of a directory to open one by one (separate: true writes a PNG per position, at 720p height). The tool for checking composition (\"what plays at time T\": layout, overlaps, text, timing) and for verifying frames before an export. Scenes only — a single element renders inside its scene, so capture the scene at the times it plays. For a video asset's own full-resolution pixels use media_grab.",
+ "Render single frames of a scene to PNGs — each frame is the frame an export of that scene would encode, drawn offscreen at the scene's own size. By default the positions are merged into contact sheets: up to 12 per image, each cell labelled with its timecode (`08s10f`, zero segments dropped) and rendered as large as fits, so a few positions arrive as one high-resolution picture instead of a directory to open one by one (separate: true writes a PNG per position, at 720p height). The tool for checking composition (\"what plays at time T\": layout, overlaps, text, timing) and for verifying frames before an export. Scenes only — a single element renders inside its scene, so capture the scene at the times it plays. For a video asset's own full-resolution pixels use grab.",
input: z
.object({
id: SceneId,
diff --git a/packages/dapi/src/tools/media-filmstrip.ts b/packages/dapi/src/tools/filmstrip.ts
similarity index 92%
rename from packages/dapi/src/tools/media-filmstrip.ts
rename to packages/dapi/src/tools/filmstrip.ts
index c678df35..5ba2363b 100644
--- a/packages/dapi/src/tools/media-filmstrip.ts
+++ b/packages/dapi/src/tools/filmstrip.ts
@@ -17,11 +17,11 @@ export const previewFields = {
.describe("scale factor for the thumbnails; smaller fits more rows and columns, larger fits fewer (default: 1)"),
};
-export const mediaFilmstrip = defineTool({
- name: "media_filmstrip",
+export const filmstrip = defineTool({
+ name: "filmstrip",
title: "Filmstrip preview",
description:
- "Render a grid of thumbnails sampled across the timeline to a PNG (local render, no credits), each row stamped with an HH:MM:SS:FF ruler. A fast, token-efficient video track preview; narrow the window to zoom into a region of interest. Video only (use media_waveform for audio).",
+ "Render a grid of thumbnails sampled across the timeline to a PNG (local render, no credits), each row stamped with an HH:MM:SS:FF ruler. A fast, token-efficient video track preview; narrow the window to zoom into a region of interest. Video only (use waveform for audio).",
input: z.object({ path: AssetPath, ...previewFields }).superRefine(checkWindow),
output: z.looseObject({ path: z.string().describe("absolute path of the PNG") }),
result: z.looseObject({ png: Bytes }),
diff --git a/packages/dapi/src/tools/media-grab.ts b/packages/dapi/src/tools/grab.ts
similarity index 96%
rename from packages/dapi/src/tools/media-grab.ts
rename to packages/dapi/src/tools/grab.ts
index fb164570..bd25abed 100644
--- a/packages/dapi/src/tools/media-grab.ts
+++ b/packages/dapi/src/tools/grab.ts
@@ -12,11 +12,11 @@ export const FrameQuality = z.enum(["small", "medium", "large", "fullres"]);
/** Guardrail against accidentally decoding a huge number of frames; `uncapped` lifts it. */
export const FRAME_CAP = 100;
-export const mediaGrab = defineTool({
- name: "media_grab",
+export const grab = defineTool({
+ name: "grab",
title: "Grab frames",
description:
- "Decode frames of a video file and write them as PNGs (local render, no credits). By default the frames are merged into contact sheets: up to 12 per image, each cell labelled with its timecode (`08s10f`, zero segments dropped) and drawn as large as fits, so a handful of frames arrives as one high-resolution picture instead of a directory to open one by one (separate: true writes a PNG per frame). Grabs the asset's own pixels, unlike capture which renders the composited node. The recommended tool for understanding a video at the frame level; past ~12 frames prefer media_filmstrip.",
+ "Decode frames of a video file and write them as PNGs (local render, no credits). By default the frames are merged into contact sheets: up to 12 per image, each cell labelled with its timecode (`08s10f`, zero segments dropped) and drawn as large as fits, so a handful of frames arrives as one high-resolution picture instead of a directory to open one by one (separate: true writes a PNG per frame). Grabs the asset's own pixels, unlike capture which renders the composited node. The recommended tool for understanding a video at the frame level; past ~12 frames prefer filmstrip.",
input: z
.object({
path: AssetPath,
diff --git a/packages/dapi/src/tools/media-listen.ts b/packages/dapi/src/tools/listen.ts
similarity index 95%
rename from packages/dapi/src/tools/media-listen.ts
rename to packages/dapi/src/tools/listen.ts
index c333777b..08363351 100644
--- a/packages/dapi/src/tools/media-listen.ts
+++ b/packages/dapi/src/tools/listen.ts
@@ -6,8 +6,8 @@ import { z } from "zod";
import { defineTool } from "../tool";
import { AssetPath, checkWindow, windowFields } from "../schemas";
-export const mediaListen = defineTool({
- name: "media_listen",
+export const listen = defineTool({
+ name: "listen",
title: "Listen to audio",
description:
"Prompt a multimodal model for a semantic analysis of an audio track and return its answer. Shines on audio semantics (the name of the music playing, who is speaking, the spoken content with second-granularity timestamps). Accepts an audio file or a video; of a video only the audio track is analyzed.",
diff --git a/packages/dapi/src/tools/media-probe.ts b/packages/dapi/src/tools/probe.ts
similarity index 96%
rename from packages/dapi/src/tools/media-probe.ts
rename to packages/dapi/src/tools/probe.ts
index c430f2ac..27eab78f 100644
--- a/packages/dapi/src/tools/media-probe.ts
+++ b/packages/dapi/src/tools/probe.ts
@@ -15,8 +15,8 @@ const Track = z.looseObject({
duration: z.number(),
});
-export const mediaProbe = defineTool({
- name: "media_probe",
+export const probe = defineTool({
+ name: "probe",
title: "Probe media",
description:
"Read the container and per-track technical metadata of a media file (local read, no credits): container format, duration, tags, and each track's codec params, without decoding. Commonly useful for a quick technical read, e.g. checking codec compatibility or duration before cutting. Packet stats (fps, bitrate) are estimated from a leading sample; images and transcripts report file-level info only.",
diff --git a/packages/dapi/src/tools/tools.test.ts b/packages/dapi/src/tools/tools.test.ts
index 0bd7b662..c14151c2 100644
--- a/packages/dapi/src/tools/tools.test.ts
+++ b/packages/dapi/src/tools/tools.test.ts
@@ -7,10 +7,10 @@ import { capture } from "./capture";
import { context } from "./context";
import { exportScene } from "./export";
import { logs } from "./logs";
-import { mediaFilmstrip } from "./media-filmstrip";
-import { mediaGrab } from "./media-grab";
-import { mediaListen } from "./media-listen";
-import { mediaTranscribe } from "./media-transcribe";
+import { filmstrip } from "./filmstrip";
+import { grab } from "./grab";
+import { listen } from "./listen";
+import { transcribe } from "./transcribe";
import { generate, generatedPath, job } from "./generate";
import { MODEL_IDS } from "@diffusionstudio/api-contract";
import { LOCAL_MODEL_IDS, isLocalJobId, isLocalModel, samRequest } from "../local";
@@ -22,8 +22,8 @@ function issues(result: { success: boolean; error?: { issues: Array<{ path: Prop
return Object.fromEntries((result.error?.issues ?? []).map((issue) => [issue.path.join("."), issue.message]));
}
-describe("media_grab", () => {
- const input = mediaGrab.input;
+describe("grab", () => {
+ const input = grab.input;
it("parses times in every form and applies the sheet default", () => {
const args = input.parse({ path: "/clip.mp4", times: ["45f", "1:10", -1, "-2f"] });
@@ -76,16 +76,16 @@ describe("capture", () => {
expect(capture.input.safeParse({ id: "intro", times: ["-2f"] }).success).toBe(false);
});
- it("shares the sheet rule with media_grab", () => {
+ it("shares the sheet rule with grab", () => {
expect(issues(capture.input.safeParse({ id: "intro", separate: true, perSheet: 2 }))).toHaveProperty("perSheet");
});
});
-describe("media_filmstrip and media_listen", () => {
+describe("filmstrip and listen", () => {
it("apply the window rule", () => {
- expect(issues(mediaFilmstrip.input.safeParse({ path: "/c.mp4", start: 3, end: 3 }))).toHaveProperty("end");
- expect(issues(mediaListen.input.safeParse({ path: "/c.mp4", start: "0:05", end: 4 }))).toHaveProperty("end");
- expect(mediaFilmstrip.input.safeParse({ path: "/c.mp4", scale: 0 }).success).toBe(false);
+ expect(issues(filmstrip.input.safeParse({ path: "/c.mp4", start: 3, end: 3 }))).toHaveProperty("end");
+ expect(issues(listen.input.safeParse({ path: "/c.mp4", start: "0:05", end: 4 }))).toHaveProperty("end");
+ expect(filmstrip.input.safeParse({ path: "/c.mp4", scale: 0 }).success).toBe(false);
});
});
@@ -117,12 +117,12 @@ describe("context", () => {
});
});
-describe("media_transcribe", () => {
+describe("transcribe", () => {
it("presents the transcript as a file: the result carries segments, the output a path", () => {
const segments = [{ text: "Hi", words: [{ text: "Hi", start: 0, end: 0.2 }] }];
- expect(mediaTranscribe.result!.safeParse({ segments }).success).toBe(true);
- expect(mediaTranscribe.output.safeParse({ path: "/tmp/t.json", segments: 1, words: 1 }).success).toBe(true);
- expect(mediaTranscribe.output.safeParse({ segments }).success).toBe(false);
+ expect(transcribe.result!.safeParse({ segments }).success).toBe(true);
+ expect(transcribe.output.safeParse({ path: "/tmp/t.json", segments: 1, words: 1 }).success).toBe(true);
+ expect(transcribe.output.safeParse({ segments }).success).toBe(false);
});
});
diff --git a/packages/dapi/src/tools/media-transcribe.ts b/packages/dapi/src/tools/transcribe.ts
similarity index 96%
rename from packages/dapi/src/tools/media-transcribe.ts
rename to packages/dapi/src/tools/transcribe.ts
index 1b6ec1ac..091c4b33 100644
--- a/packages/dapi/src/tools/media-transcribe.ts
+++ b/packages/dapi/src/tools/transcribe.ts
@@ -18,8 +18,8 @@ export const TranscriptSegment = z.object({
words: z.array(TranscriptWord),
});
-export const mediaTranscribe = defineTool({
- name: "media_transcribe",
+export const transcribe = defineTool({
+ name: "transcribe",
title: "Transcribe speech",
description:
"Transcribe the speech in a video or audio file and write the timed transcript to a JSON file, `{ segments: [{ text, speaker?, words: [{ text, start, end }] }] }` with word-level times in seconds and speakers labelled A, B, … when told apart; returns the file's path and its segment and word counts. Search the file for the passage you need (grep, jq) rather than reading it whole: a long recording's word timings run to tens of thousands of tokens. Commonly useful for footage with speakers (talking head, interview), where the word times let you cut on a line. A transcript marks only speech; the gaps are not necessarily silent (music, score, applause).",
diff --git a/packages/dapi/src/tools/media-waveform.ts b/packages/dapi/src/tools/waveform.ts
similarity index 89%
rename from packages/dapi/src/tools/media-waveform.ts
rename to packages/dapi/src/tools/waveform.ts
index 72faf05f..69358449 100644
--- a/packages/dapi/src/tools/media-waveform.ts
+++ b/packages/dapi/src/tools/waveform.ts
@@ -5,12 +5,12 @@
import { z } from "zod";
import { defineTool } from "../tool";
import { AssetPath, Bytes, checkWindow } from "../schemas";
-import { previewFields } from "./media-filmstrip";
+import { previewFields } from "./filmstrip";
const Silences = z.array(z.object({ start: z.number(), end: z.number() })).describe("seconds");
-export const mediaWaveform = defineTool({
- name: "media_waveform",
+export const waveform = defineTool({
+ name: "waveform",
title: "Waveform preview",
description:
"Render the audio track of a video or audio file as a waveform PNG (local render, no credits) with a timestamp ruler: loudness over time, with silent stretches highlighted in red. A fast, token-efficient audio track preview; the silent spans are also returned as second ranges.",