The VS Code of video editing, built for agents.
TypeScript
Refactor media tool documentation and update command references
konstantin-paulus35f4cf6c2670
diff --git a/docs/guides/prompts/media-listen.md b/docs/guides/prompts/media-listen.mdindex ac35f5fa..53b520aa 100644--- a/docs/guides/prompts/media-listen.md+++ b/docs/guides/prompts/media-listen.md@@ -1,6 +1,6 @@# Writing prompts for `listen`-[`listen`](../../reference/tools/media/listen.md) puts a multimodal model in front of an audio track and answers a prompt about it. A good prompt is specific about what you want and asks for timestamps so the answer lines up with the timeline. With no prompt it returns a general description; the examples below shape that into something more useful.+[`listen`](../../reference/tools/listen.md) puts a multimodal model in front of an audio track and answers a prompt about it. A good prompt is specific about what you want and asks for timestamps so the answer lines up with the timeline. With no prompt it returns a general description; the examples below shape that into something more useful.## ExamplesWhen you refer to a specific moment inside a prompt, use `MM:SS` (e.g. `01:15` for one minute fifteen seconds).diff --git a/docs/reference/jsx/styles.md b/docs/reference/jsx/styles.mdindex ea911e60..f8b03dfc 100644--- a/docs/reference/jsx/styles.md+++ b/docs/reference/jsx/styles.md@@ -68,7 +68,7 @@ A filter over the parent's **rendered pixels** — its fills, strokes and childr## `<mask>`-A matte limiting the `<effect>` holding it: a **picture** whose alpha says where the effect applies, fitted into the node's box the way the node fits its footage. It is what the editor's object mask tool makes — pick an object in a video clip, track it, and the frames it found are written to the library and named by `src` — and what [`generate`](../tools/generate.md)'s [`sam-2.1`](../tools/media/segment.md) makes from a tool call. Several under one effect intersect. Without a `src` a `<mask>` does nothing.+A matte limiting the `<effect>` holding it: a **picture** whose alpha says where the effect applies, fitted into the node's box the way the node fits its footage. It is what the editor's object mask tool makes — pick an object in a video clip, track it, and the frames it found are written to the library and named by `src` — and what [`generate`](../tools/generate.md)'s [`sam-2.1`](../tools/segment.md) makes from a tool call. Several under one effect intersect. Without a `src` a `<mask>` does nothing.Under an `"opacity"` effect the mask is the cut-out: the clip shows inside the mask and goes transparent outside, as a mask on Premiere's Opacity does. Under any other effect the effect shows through inside the mask and the picture is untouched outside: an inverted mask on a `"blur"` blurs the background behind a person.diff --git a/docs/reference/jsx/timing.md b/docs/reference/jsx/timing.mdindex 92542d0b..c0ddb67d 100644--- a/docs/reference/jsx/timing.md+++ b/docs/reference/jsx/timing.md@@ -62,4 +62,4 @@ The canonical internal unit is frames at **30 fps**; all formats are converted o| `"MM:SS"` | `"02:30"` | Minutes and seconds. || `"HH:MM:SS"` | `"01:02:30"` | Hours, minutes, seconds. |-CLI flags documented as taking a `Time` value (e.g. `--times` on [`capture`](../tools/capture.md) and [`media grab`](../tools/media/grab.md)) accept the same formats. Times in command **output** are plain seconds.+CLI flags documented as taking a `Time` value (e.g. `--times` on [`capture`](../tools/capture.md) and [`grab`](../tools/grab.md)) accept the same formats. Times in command **output** are plain seconds.diff --git a/docs/reference/models.md b/docs/reference/models.mdindex ee7dbc75..ea7feba8 100644--- a/docs/reference/models.md+++ b/docs/reference/models.md@@ -101,4 +101,4 @@ These run in the app, on this machine's GPU: no account, no credits. A job is sh| Model | | Makes | Fields || --- | --- | --- | --- |-| `sam-2.1` | SAM 2.1 — segments an object in a video and tracks it through the footage; `size` tiny (default), small, base-plus or large | a mask file (`.mask`) for a [`<mask src>`](./jsx/styles.md#mask) | [segment.md](./tools/media/segment.md) |+| `sam-2.1` | SAM 2.1 — segments an object in a video and tracks it through the footage; `size` tiny (default), small, base-plus or large | a mask file (`.mask`) for a [`<mask src>`](./jsx/styles.md#mask) | [segment.md](./tools/segment.md) |diff --git a/docs/reference/tools/README.md b/docs/reference/tools/README.mdindex 23a84079..0706c3ac 100644--- a/docs/reference/tools/README.md+++ b/docs/reference/tools/README.md@@ -37,12 +37,12 @@ Every tool runs inside the app, so the app has to be running. Over MCP that is a| [`capture`](./capture.md) | `diffusion capture` | Capture frames || [`check`](./check.md) | `diffusion check` | Check structure || [`export`](./export.md) | `diffusion export` | Export scene |-| [`probe`](./media/probe.md) | `diffusion probe` | Probe media |-| [`grab`](./media/grab.md) | `diffusion grab` | Grab frames |-| [`transcribe`](./media/transcribe.md) | `diffusion transcribe` | Transcribe speech |-| [`filmstrip`](./media/filmstrip.md) | `diffusion filmstrip` | Filmstrip preview |-| [`waveform`](./media/waveform.md) | `diffusion waveform` | Waveform preview |-| [`listen`](./media/listen.md) | `diffusion listen` | Listen to audio |+| [`probe`](./probe.md) | `diffusion probe` | Probe media |+| [`grab`](./grab.md) | `diffusion grab` | Grab frames |+| [`transcribe`](./transcribe.md) | `diffusion transcribe` | Transcribe speech |+| [`filmstrip`](./filmstrip.md) | `diffusion filmstrip` | Filmstrip preview |+| [`waveform`](./waveform.md) | `diffusion waveform` | Waveform preview |+| [`listen`](./listen.md) | `diffusion listen` | Listen to audio || [`generate`](./generate.md) | `diffusion generate` | Generate media || [`job`](./job.md) | `diffusion job` | Generation job || [`logs`](./logs.md) | `diffusion logs` | App logs |@@ -54,8 +54,8 @@ Every tool runs inside the app, so the app has to be running. Over MCP that is aHow the surface is divided:- **The project loop.** [`open`](./open.md) a folder, edit its JSX, [`context`](./context.md) for what the source cannot say, [`capture`](./capture.md) and [`check`](./check.md) to verify, [`export`](./export.md) when asked.-- **Media inspection** ([`probe`](./media/probe.md), [`grab`](./media/grab.md), [`transcribe`](./media/transcribe.md), [`filmstrip`](./media/filmstrip.md), [`waveform`](./media/waveform.md), [`listen`](./media/listen.md)): a file by path, without adding it to the project. Absolute paths and URLs work with or without an open project; library paths (`b-roll/clip.mp4`) need one.-- **Generation.** [`generate`](./generate.md) runs a model of the Diffusion Studio API (images, video, music, speech, and tools such as upscaling) on the account's credits, or a model on this machine for free (with `estimate`, it only prices the request) — [`sam-2.1`](./media/segment.md) segments and tracks an object in footage, making the mask file a `<mask src>` names; poll [`job`](./job.md) until it has ended, and its files are saved into the library or a directory. The models and their fields are in [models.md](../models.md).+- **Media inspection** ([`probe`](./probe.md), [`grab`](./grab.md), [`transcribe`](./transcribe.md), [`filmstrip`](./filmstrip.md), [`waveform`](./waveform.md), [`listen`](./listen.md)): a file by path, without adding it to the project. Absolute paths and URLs work with or without an open project; library paths (`b-roll/clip.mp4`) need one.+- **Generation.** [`generate`](./generate.md) runs a model of the Diffusion Studio API (images, video, music, speech, and tools such as upscaling) on the account's credits, or a model on this machine for free (with `estimate`, it only prices the request) — [`sam-2.1`](./segment.md) segments and tracks an object in footage, making the mask file a `<mask src>` names; poll [`job`](./job.md) until it has ended, and its files are saved into the library or a directory. The models and their fields are in [models.md](../models.md).- **What a declaration may name.** [`fonts`](./fonts.md).- **The app and the machine.** [`logs`](./logs.md), [`screenshot`](./screenshot.md), [`window`](./window.md), [`report`](./report.md).diff --git a/docs/reference/tools/capture.md b/docs/reference/tools/capture.mdindex c1473b4c..1d105ec9 100644--- a/docs/reference/tools/capture.md+++ b/docs/reference/tools/capture.md@@ -21,7 +21,7 @@ Render single frames of a scene to PNGs — each frame is the frame an export ofEach frame is **the frame an export of that scene would encode**: the scene is re-rendered from a fresh mount at its own size, position `0` is the workarea's first frame, and the requested positions are evaluated in timeline order, forward only — the way an export advances — so a composition whose look depends on having played (an `<html>` node's own animation state, for instance) captures exactly as it exports. A position past the workarea's end is not an error: the scene keeps playing past it, and the frame is what plays there — just not one an export would include.-Scenes only: a single element renders inside its scene, so capture the scene at the times the element plays. To grab a video asset's own pixels instead of a composited frame, use [`grab`](./media/grab.md).+Scenes only: a single element renders inside its scene, so capture the scene at the times the element plays. To grab a video asset's own pixels instead of a composited frame, use [`grab`](./grab.md).With `separate`, each position is rendered at 720p height and named after its timecode (e.g. `01s12f.png`). Without an `output` directory the images land in a fresh `dapi-capture-*` directory under the system temp directory, so runs never overwrite each other. Writing into the same directory twice overwrites images whose name matches; with `separate`, requested times that land on the same frame share one file.diff --git a/docs/reference/tools/media/filmstrip.md b/docs/reference/tools/filmstrip.mdsimilarity index 100%rename from docs/reference/tools/media/filmstrip.mdrename to docs/reference/tools/filmstrip.mddiff --git a/docs/reference/tools/generate.md b/docs/reference/tools/generate.mdindex bb01267a..dc89d9ba 100644--- a/docs/reference/tools/generate.md+++ b/docs/reference/tools/generate.md@@ -1,6 +1,6 @@# generate-Start a job on a generative model — images, video, music, sound effects, speech, and tools such as background removal, upscaling, transcription and analysis on the Diffusion Studio API, and object segmentation and tracking ([`sam-2.1`](./media/segment.md)) on this machine — and return the job at once, with its estimated run time and the credits it costs. Poll [`job`](./job.md) with its id until it has ended; the files it made are then saved and their paths returned. With [`estimate`](#estimating), nothing starts: the answer is only what the job would cost and how long it would run. API models need a signed-in account and spend its credits; [local models](../models.md#local) run on the GPU, free, and need no account.+Start a job on a generative model — images, video, music, sound effects, speech, and tools such as background removal, upscaling, transcription and analysis on the Diffusion Studio API, and object segmentation and tracking ([`sam-2.1`](./segment.md)) on this machine — and return the job at once, with its estimated run time and the credits it costs. Poll [`job`](./job.md) with its id until it has ended; the files it made are then saved and their paths returned. With [`estimate`](#estimating), nothing starts: the answer is only what the job would cost and how long it would run. API models need a signed-in account and spend its credits; [local models](../models.md#local) run on the GPU, free, and need no account.| | || --- | --- |@@ -63,7 +63,7 @@ Files (`{ "path": … }`) are uploaded first, as for a job, since a tool such as- **Several files are numbered.** A job that makes more than one (an image `count` above 1) saves `fox-1.png`, `fox-2.png`, …; a single file is saved at the path itself.- **A library path** (`b-roll/fox.png`, needs an open project) puts the file into the project's library, under `assets/`, named by `src` like any other asset, and notes the job it came from, as the app's own generations do. A file the project made already at that path is replaced, and everything naming it shows the new one; a file linked from elsewhere on disk is never written over, and the result gets a free name next to it instead.- **An absolute path** saves the file there, replacing a file already there.-- **Left out**, the file goes into the open project's library at its root under the API's name (`red-fox-at-dawn.png`, numbered if taken), or where a local model files its own (`sam-2.1`: `masks/<video>/Tracking <n>.mask`); with no project open, into a fresh directory of the job's own under the system temp dir, as [`transcribe`](./media/transcribe.md) and the other inspection tools do.+- **Left out**, the file goes into the open project's library at its root under the API's name (`red-fox-at-dawn.png`, numbered if taken), or where a local model files its own (`sam-2.1`: `masks/<video>/Tracking <n>.mask`); with no project open, into a fresh directory of the job's own under the system temp dir, as [`transcribe`](./transcribe.md) and the other inspection tools do.The files are saved by the first [`job`](./job.md) call that sees the job succeed (a local model's library files as soon as it does); later calls return the same paths. A file an API model made that went into the library records the job in `assets.yml` (its `job` field), so what made it can be looked up later.diff --git a/docs/reference/tools/media/grab.md b/docs/reference/tools/grab.mdsimilarity index 96%rename from docs/reference/tools/media/grab.mdrename to docs/reference/tools/grab.mdindex cefc5093..598d7db6 100644--- a/docs/reference/tools/media/grab.md+++ b/docs/reference/tools/grab.md@@ -32,7 +32,7 @@ Three ways to say which frames, mutually exclusive:- `count`: that many frames evenly spaced across the clip, or across the `start`/`end` window, at a fixed interval of `window / count`, starting at the window start.- `auto`: a scan at 2 fps that keeps a frame each time the picture settles into a new visual state, dropping near-duplicates and waiting out transitions so picks stay sharp. Returns at most `count` frames (default cap 30); static footage such as a screen recording returns far fewer. Needs WebGPU.-`start` and `end` only apply with `count` or `auto`. An `end` past the asset's duration is clamped to it; a `start` at or past the end is an error, since the window would be empty. Like [`capture`](../capture.md), but this grabs the asset's own pixels; `capture` renders the composited node. Renders locally; no credits. Past ~12 frames, [`filmstrip`](./filmstrip.md) is the cheaper way to scan a clip.+`start` and `end` only apply with `count` or `auto`. An `end` past the asset's duration is clamped to it; a `start` at or past the end is an error, since the window would be empty. Like [`capture`](./capture.md), but this grabs the asset's own pixels; `capture` renders the composited node. Renders locally; no credits. Past ~12 frames, [`filmstrip`](./filmstrip.md) is the cheaper way to scan a clip.`quality` is a pixel budget rather than a box: a frame is scaled down, aspect ratio kept, until it holds no more pixels than the preset's square (`small` is 384², so a 16:9 frame becomes 512x288; `medium` 768² gives 1024x576; `large` 1536² gives 2048x1152), and never enlarged, so `large` on 1080p footage is the native 1920x1080.diff --git a/docs/reference/tools/media/listen.md b/docs/reference/tools/listen.mdsimilarity index 94%rename from docs/reference/tools/media/listen.mdrename to docs/reference/tools/listen.mdindex 48b8cc36..be70e632 100644--- a/docs/reference/tools/media/listen.md+++ b/docs/reference/tools/listen.md@@ -16,7 +16,7 @@ Prompt a multimodal model for a semantic analysis of an audio track and return i| `start` | `Time` | `-s, --start <time>` | start of the segment to analyze (default: 0); timestamps in the analysis are relative to this point || `end` | `Time` | `-e, --end <time>` | end of the segment to analyze (default: media duration) |-With no `prompt` it returns a general description of what is heard; with one it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). See [the prompt guide](../../../guides/prompts/media-listen.md) for prompts that work.+With no `prompt` it returns a general description of what is heard; with one it answers that question about the audio (e.g. "who is speaking?", "what music is playing?", "summarize what is said"). See [the prompt guide](../../guides/prompts/media-listen.md) for prompts that work.## Outputdiff --git a/docs/reference/tools/media/probe.md b/docs/reference/tools/probe.mdsimilarity index 100%rename from docs/reference/tools/media/probe.mdrename to docs/reference/tools/probe.mddiff --git a/docs/reference/tools/media/segment.md b/docs/reference/tools/segment.mdsimilarity index 93%rename from docs/reference/tools/media/segment.mdrename to docs/reference/tools/segment.mdindex f4b5d3b8..3af32834 100644--- a/docs/reference/tools/media/segment.md+++ b/docs/reference/tools/segment.md@@ -1,15 +1,15 @@# sam-2.1-SAM 2.1 segments an object in a video and tracks it through the footage, making the mask file (`.mask`) a [`<mask src>`](../../jsx/styles.md#mask) names — the file the editor's object mask tool makes. It runs in the app, on this machine's GPU: no account, no credits. Like every [local model](../../models.md#local), its job is polled with [`job`](../job.md) and lasts only as long as the app.+SAM 2.1 segments an object in a video and tracks it through the footage, making the mask file (`.mask`) a [`<mask src>`](../jsx/styles.md#mask) names — the file the editor's object mask tool makes. It runs in the app, on this machine's GPU: no account, no credits. Like every [local model](../models.md#local), its job is polled with [`job`](./job.md) and lasts only as long as the app.| | || --- | --- |-| Model | `sam-2.1`, run with [`generate`](../generate.md) |+| Model | `sam-2.1`, run with [`generate`](./generate.md) || CLI | `diffusion generate sam-2.1 '<fields>' [-o <path>]` |## Input-`model: "sam-2.1"` plus these fields; `output` is [`generate`](../generate.md#where-the-files-go)'s.+`model: "sam-2.1"` plus these fields; `output` is [`generate`](./generate.md#where-the-files-go)'s.| Field | Type | Description || --- | --- | --- |diff --git a/docs/reference/tools/media/transcribe.md b/docs/reference/tools/transcribe.mdsimilarity index 100%rename from docs/reference/tools/media/transcribe.mdrename to docs/reference/tools/transcribe.mddiff --git a/docs/reference/tools/media/waveform.md b/docs/reference/tools/waveform.mdsimilarity index 100%rename from docs/reference/tools/media/waveform.mdrename to docs/reference/tools/waveform.mddiff --git a/docs/skills/editor.md b/docs/skills/editor.mdindex ac68433b..60c35b23 100644--- a/docs/skills/editor.md+++ b/docs/skills/editor.md@@ -46,7 +46,7 @@ How to confirm a change actually produced what you intended. A clean save does n- Use the built-in tags for the media a composition is made of (audio, video, images, captions).- Hoist the properties that define the composition's look — title copy, font family and size, accent colors, key padding — into top-level consts annotated with `@inspect`, so they become live controls in the app's inspector.- For anything 3D, use Three.js drawn into a `<surface>` tag.-- To cut an object out of footage, or confine an effect to it, track it with `generate`'s local `sam-2.1` model (preview the prompt first, see [sam-2.1](../reference/tools/media/segment.md)) and name the mask file in a `<mask>` under the clip's `<effect>`. The track runs as a job: keep working, and poll `job` until it has succeeded before naming the mask.+- To cut an object out of footage, or confine an effect to it, track it with `generate`'s local `sam-2.1` model (preview the prompt first, see [sam-2.1](../reference/tools/segment.md)) and name the mask file in a `<mask>` under the clip's `<effect>`. The track runs as a job: keep working, and poll `job` until it has succeeded before naming the mask.- For motion graphics, overlays and UI-heavy graphics, the `<html>` tag driven by a paused [anime.js](https://animejs.com) timeline- Before animating anything, read the [easings reference](../guides/motion/easings.md) and choose easings deliberately — default or linear easing is what makes motion read as a slideshow.- For tasks that don't need an editing UI, keep the app in the background: from a shell, `diffusion open -b <dir>` launches it that way. Over MCP the app is already running, and `open` only opens the folder.diff --git a/packages/dapi/src/catalog.ts b/packages/dapi/src/catalog.tsindex 4b2eb96b..2fe20759 100644--- a/packages/dapi/src/catalog.ts+++ b/packages/dapi/src/catalog.ts@@ -12,12 +12,12 @@ import { check } from "./tools/check";import { exportScene } from "./tools/export";import { logs } from "./tools/logs";import { screenshot } from "./tools/screenshot";-import { mediaProbe } from "./tools/media-probe";-import { mediaGrab } from "./tools/media-grab";-import { mediaTranscribe } from "./tools/media-transcribe";-import { mediaFilmstrip } from "./tools/media-filmstrip";-import { mediaWaveform } from "./tools/media-waveform";-import { mediaListen } from "./tools/media-listen";+import { probe } from "./tools/probe";+import { grab } from "./tools/grab";+import { transcribe } from "./tools/transcribe";+import { filmstrip } from "./tools/filmstrip";+import { waveform } from "./tools/waveform";+import { listen } from "./tools/listen";import { generate, job } from "./tools/generate";import { fonts } from "./tools/fonts";import { report } from "./tools/report";@@ -34,12 +34,12 @@ export const catalog = [capture,check,exportScene,- mediaProbe,- mediaGrab,- mediaTranscribe,- mediaFilmstrip,- mediaWaveform,- mediaListen,+ probe,+ grab,+ transcribe,+ filmstrip,+ waveform,+ listen,generate,job,logs,diff --git a/packages/dapi/src/index.ts b/packages/dapi/src/index.tsindex d82d4148..6efafe90 100644--- a/packages/dapi/src/index.ts+++ b/packages/dapi/src/index.ts@@ -24,7 +24,7 @@ export { JSON_SCHEMA_DIALECT, toolJsonSchemas } from "./json-schema";export type { ToolJsonSchemas } from "./json-schema";export { DAPI_WIRE } from "./ipc";export type { DapiCall, DapiCancel, DapiReply } from "./ipc";-export { FRAME_CAP } from "./tools/media-grab";+export { FRAME_CAP } from "./tools/grab";export { ISSUE_LOG_TAIL } from "./tools/report";export { LOG_TAIL, LOG_MESSAGE_MAX } from "./tools/logs";export { FONT_LIMIT } from "./tools/fonts";@@ -47,8 +47,8 @@ export type { LocalModelId, LocalRequest, SamDetails, SamRequest } from "./localimport type { ImageRef as ImageRefSchema, LogEntry as LogEntrySchema, LogLevel as LogLevelSchema, TimecodedImage as TimecodedImageSchema } from "./schemas";import type { CheckIssue as CheckIssueSchema, CheckIssueCode as CheckIssueCodeSchema } from "./tools/check";import type { ExportFormat as ExportFormatSchema, ExportSettings as ExportSettingsSchema } from "./tools/export";-import type { FrameQuality as FrameQualitySchema } from "./tools/media-grab";-import type { TranscriptSegment as TranscriptSegmentSchema, TranscriptWord as TranscriptWordSchema } from "./tools/media-transcribe";+import type { FrameQuality as FrameQualitySchema } from "./tools/grab";+import type { TranscriptSegment as TranscriptSegmentSchema, TranscriptWord as TranscriptWordSchema } from "./tools/transcribe";import type { FontFamily as FontFamilySchema } from "./tools/fonts";import type { ToolArgs, ToolOutput, ToolResult } from "./catalog";@@ -77,17 +77,17 @@ export type ExportResult = ToolResult<"export">;export type LogsRequest = ToolArgs<"logs">;export type ScreenshotResult = ToolResult<"screenshot">;export type ScreenshotOutput = ToolOutput<"screenshot">;-export type MediaProbeRequest = ToolArgs<"media_probe">;-export type MediaFrameRequest = ToolArgs<"media_grab">;-export type MediaFrameResult = ToolResult<"media_grab">;-export type MediaTranscribeRequest = ToolArgs<"media_transcribe">;-export type MediaTranscribeResult = ToolResult<"media_transcribe">;-export type MediaFilmstripRequest = ToolArgs<"media_filmstrip">;-export type MediaFilmstripResult = ToolResult<"media_filmstrip">;-export type MediaWaveformRequest = ToolArgs<"media_waveform">;-export type MediaWaveformResult = ToolResult<"media_waveform">;-export type MediaListenRequest = ToolArgs<"media_listen">;-export type MediaListenResult = ToolResult<"media_listen">;+export type MediaProbeRequest = ToolArgs<"probe">;+export type MediaFrameRequest = ToolArgs<"grab">;+export type MediaFrameResult = ToolResult<"grab">;+export type MediaTranscribeRequest = ToolArgs<"transcribe">;+export type MediaTranscribeResult = ToolResult<"transcribe">;+export type MediaFilmstripRequest = ToolArgs<"filmstrip">;+export type MediaFilmstripResult = ToolResult<"filmstrip">;+export type MediaWaveformRequest = ToolArgs<"waveform">;+export type MediaWaveformResult = ToolResult<"waveform">;+export type MediaListenRequest = ToolArgs<"listen">;+export type MediaListenResult = ToolResult<"listen">;export type GenerateRequest = ToolArgs<"generate">;export type GenerationResult = ToolResult<"job">;export type GenerationJob = ToolOutput<"job">;diff --git a/packages/dapi/src/schemas.ts b/packages/dapi/src/schemas.tsindex 0539d2a6..248e717c 100644--- a/packages/dapi/src/schemas.ts+++ b/packages/dapi/src/schemas.ts@@ -25,7 +25,7 @@ export const AssetPath = z"absolute file path or URL (works with or without an open project), or a library path like `b-roll/clip.mp4` (needs an open project)",);-/** Beyond this the cells get too small to be worth the tokens; use `media_filmstrip`. */+/** Beyond this the cells get too small to be worth the tokens; use `filmstrip`. */export const MAX_FRAMES_PER_SHEET = 12;/**diff --git a/packages/dapi/src/tools/capture.ts b/packages/dapi/src/tools/capture.tsindex 7b997a9e..37a37639 100644--- a/packages/dapi/src/tools/capture.ts+++ b/packages/dapi/src/tools/capture.ts@@ -11,7 +11,7 @@ export const capture = defineTool({name: "capture",title: "Capture frames",description:- "Render single frames of a scene to PNGs — each frame is the frame an export of that scene would encode, drawn offscreen at the scene's own size. By default the positions are merged into contact sheets: up to 12 per image, each cell labelled with its timecode (`08s10f`, zero segments dropped) and rendered as large as fits, so a few positions arrive as one high-resolution picture instead of a directory to open one by one (separate: true writes a PNG per position, at 720p height). The tool for checking composition (\"what plays at time T\": layout, overlaps, text, timing) and for verifying frames before an export. Scenes only — a single element renders inside its scene, so capture the scene at the times it plays. For a video asset's own full-resolution pixels use media_grab.",+ "Render single frames of a scene to PNGs — each frame is the frame an export of that scene would encode, drawn offscreen at the scene's own size. By default the positions are merged into contact sheets: up to 12 per image, each cell labelled with its timecode (`08s10f`, zero segments dropped) and rendered as large as fits, so a few positions arrive as one high-resolution picture instead of a directory to open one by one (separate: true writes a PNG per position, at 720p height). The tool for checking composition (\"what plays at time T\": layout, overlaps, text, timing) and for verifying frames before an export. Scenes only — a single element renders inside its scene, so capture the scene at the times it plays. For a video asset's own full-resolution pixels use grab.",input: z.object({id: SceneId,diff --git a/packages/dapi/src/tools/media-filmstrip.ts b/packages/dapi/src/tools/filmstrip.tssimilarity index 92%rename from packages/dapi/src/tools/media-filmstrip.tsrename to packages/dapi/src/tools/filmstrip.tsindex c678df35..5ba2363b 100644--- a/packages/dapi/src/tools/media-filmstrip.ts+++ b/packages/dapi/src/tools/filmstrip.ts@@ -17,11 +17,11 @@ export const previewFields = {.describe("scale factor for the thumbnails; smaller fits more rows and columns, larger fits fewer (default: 1)"),};-export const mediaFilmstrip = defineTool({- name: "media_filmstrip",+export const filmstrip = defineTool({+ name: "filmstrip",title: "Filmstrip preview",description:- "Render a grid of thumbnails sampled across the timeline to a PNG (local render, no credits), each row stamped with an HH:MM:SS:FF ruler. A fast, token-efficient video track preview; narrow the window to zoom into a region of interest. Video only (use media_waveform for audio).",+ "Render a grid of thumbnails sampled across the timeline to a PNG (local render, no credits), each row stamped with an HH:MM:SS:FF ruler. A fast, token-efficient video track preview; narrow the window to zoom into a region of interest. Video only (use waveform for audio).",input: z.object({ path: AssetPath, ...previewFields }).superRefine(checkWindow),output: z.looseObject({ path: z.string().describe("absolute path of the PNG") }),result: z.looseObject({ png: Bytes }),diff --git a/packages/dapi/src/tools/media-grab.ts b/packages/dapi/src/tools/grab.tssimilarity index 96%rename from packages/dapi/src/tools/media-grab.tsrename to packages/dapi/src/tools/grab.tsindex fb164570..bd25abed 100644--- a/packages/dapi/src/tools/media-grab.ts+++ b/packages/dapi/src/tools/grab.ts@@ -12,11 +12,11 @@ export const FrameQuality = z.enum(["small", "medium", "large", "fullres"]);/** Guardrail against accidentally decoding a huge number of frames; `uncapped` lifts it. */export const FRAME_CAP = 100;-export const mediaGrab = defineTool({- name: "media_grab",+export const grab = defineTool({+ name: "grab",title: "Grab frames",description:- "Decode frames of a video file and write them as PNGs (local render, no credits). By default the frames are merged into contact sheets: up to 12 per image, each cell labelled with its timecode (`08s10f`, zero segments dropped) and drawn as large as fits, so a handful of frames arrives as one high-resolution picture instead of a directory to open one by one (separate: true writes a PNG per frame). Grabs the asset's own pixels, unlike capture which renders the composited node. The recommended tool for understanding a video at the frame level; past ~12 frames prefer media_filmstrip.",+ "Decode frames of a video file and write them as PNGs (local render, no credits). By default the frames are merged into contact sheets: up to 12 per image, each cell labelled with its timecode (`08s10f`, zero segments dropped) and drawn as large as fits, so a handful of frames arrives as one high-resolution picture instead of a directory to open one by one (separate: true writes a PNG per frame). Grabs the asset's own pixels, unlike capture which renders the composited node. The recommended tool for understanding a video at the frame level; past ~12 frames prefer filmstrip.",input: z.object({path: AssetPath,diff --git a/packages/dapi/src/tools/media-listen.ts b/packages/dapi/src/tools/listen.tssimilarity index 95%rename from packages/dapi/src/tools/media-listen.tsrename to packages/dapi/src/tools/listen.tsindex c333777b..08363351 100644--- a/packages/dapi/src/tools/media-listen.ts+++ b/packages/dapi/src/tools/listen.ts@@ -6,8 +6,8 @@ import { z } from "zod";import { defineTool } from "../tool";import { AssetPath, checkWindow, windowFields } from "../schemas";-export const mediaListen = defineTool({- name: "media_listen",+export const listen = defineTool({+ name: "listen",title: "Listen to audio",description:"Prompt a multimodal model for a semantic analysis of an audio track and return its answer. Shines on audio semantics (the name of the music playing, who is speaking, the spoken content with second-granularity timestamps). Accepts an audio file or a video; of a video only the audio track is analyzed.",diff --git a/packages/dapi/src/tools/media-probe.ts b/packages/dapi/src/tools/probe.tssimilarity index 96%rename from packages/dapi/src/tools/media-probe.tsrename to packages/dapi/src/tools/probe.tsindex c430f2ac..27eab78f 100644--- a/packages/dapi/src/tools/media-probe.ts+++ b/packages/dapi/src/tools/probe.ts@@ -15,8 +15,8 @@ const Track = z.looseObject({duration: z.number(),});-export const mediaProbe = defineTool({- name: "media_probe",+export const probe = defineTool({+ name: "probe",title: "Probe media",description:"Read the container and per-track technical metadata of a media file (local read, no credits): container format, duration, tags, and each track's codec params, without decoding. Commonly useful for a quick technical read, e.g. checking codec compatibility or duration before cutting. Packet stats (fps, bitrate) are estimated from a leading sample; images and transcripts report file-level info only.",diff --git a/packages/dapi/src/tools/tools.test.ts b/packages/dapi/src/tools/tools.test.tsindex 0bd7b662..c14151c2 100644--- a/packages/dapi/src/tools/tools.test.ts+++ b/packages/dapi/src/tools/tools.test.ts@@ -7,10 +7,10 @@ import { capture } from "./capture";import { context } from "./context";import { exportScene } from "./export";import { logs } from "./logs";-import { mediaFilmstrip } from "./media-filmstrip";-import { mediaGrab } from "./media-grab";-import { mediaListen } from "./media-listen";-import { mediaTranscribe } from "./media-transcribe";+import { filmstrip } from "./filmstrip";+import { grab } from "./grab";+import { listen } from "./listen";+import { transcribe } from "./transcribe";import { generate, generatedPath, job } from "./generate";import { MODEL_IDS } from "@diffusionstudio/api-contract";import { LOCAL_MODEL_IDS, isLocalJobId, isLocalModel, samRequest } from "../local";@@ -22,8 +22,8 @@ function issues(result: { success: boolean; error?: { issues: Array<{ path: Propreturn Object.fromEntries((result.error?.issues ?? []).map((issue) => [issue.path.join("."), issue.message]));}-describe("media_grab", () => {- const input = mediaGrab.input;+describe("grab", () => {+ const input = grab.input;it("parses times in every form and applies the sheet default", () => {const args = input.parse({ path: "/clip.mp4", times: ["45f", "1:10", -1, "-2f"] });@@ -76,16 +76,16 @@ describe("capture", () => {expect(capture.input.safeParse({ id: "intro", times: ["-2f"] }).success).toBe(false);});- it("shares the sheet rule with media_grab", () => {+ it("shares the sheet rule with grab", () => {expect(issues(capture.input.safeParse({ id: "intro", separate: true, perSheet: 2 }))).toHaveProperty("perSheet");});});-describe("media_filmstrip and media_listen", () => {+describe("filmstrip and listen", () => {it("apply the window rule", () => {- expect(issues(mediaFilmstrip.input.safeParse({ path: "/c.mp4", start: 3, end: 3 }))).toHaveProperty("end");- expect(issues(mediaListen.input.safeParse({ path: "/c.mp4", start: "0:05", end: 4 }))).toHaveProperty("end");- expect(mediaFilmstrip.input.safeParse({ path: "/c.mp4", scale: 0 }).success).toBe(false);+ expect(issues(filmstrip.input.safeParse({ path: "/c.mp4", start: 3, end: 3 }))).toHaveProperty("end");+ expect(issues(listen.input.safeParse({ path: "/c.mp4", start: "0:05", end: 4 }))).toHaveProperty("end");+ expect(filmstrip.input.safeParse({ path: "/c.mp4", scale: 0 }).success).toBe(false);});});@@ -117,12 +117,12 @@ describe("context", () => {});});-describe("media_transcribe", () => {+describe("transcribe", () => {it("presents the transcript as a file: the result carries segments, the output a path", () => {const segments = [{ text: "Hi", words: [{ text: "Hi", start: 0, end: 0.2 }] }];- expect(mediaTranscribe.result!.safeParse({ segments }).success).toBe(true);- expect(mediaTranscribe.output.safeParse({ path: "/tmp/t.json", segments: 1, words: 1 }).success).toBe(true);- expect(mediaTranscribe.output.safeParse({ segments }).success).toBe(false);+ expect(transcribe.result!.safeParse({ segments }).success).toBe(true);+ expect(transcribe.output.safeParse({ path: "/tmp/t.json", segments: 1, words: 1 }).success).toBe(true);+ expect(transcribe.output.safeParse({ segments }).success).toBe(false);});});diff --git a/packages/dapi/src/tools/media-transcribe.ts b/packages/dapi/src/tools/transcribe.tssimilarity index 96%rename from packages/dapi/src/tools/media-transcribe.tsrename to packages/dapi/src/tools/transcribe.tsindex 1b6ec1ac..091c4b33 100644--- a/packages/dapi/src/tools/media-transcribe.ts+++ b/packages/dapi/src/tools/transcribe.ts@@ -18,8 +18,8 @@ export const TranscriptSegment = z.object({words: z.array(TranscriptWord),});-export const mediaTranscribe = defineTool({- name: "media_transcribe",+export const transcribe = defineTool({+ name: "transcribe",title: "Transcribe speech",description:"Transcribe the speech in a video or audio file and write the timed transcript to a JSON file, `{ segments: [{ text, speaker?, words: [{ text, start, end }] }] }` with word-level times in seconds and speakers labelled A, B, … when told apart; returns the file's path and its segment and word counts. Search the file for the passage you need (grep, jq) rather than reading it whole: a long recording's word timings run to tens of thousands of tokens. Commonly useful for footage with speakers (talking head, interview), where the word times let you cut on a line. A transcript marks only speech; the gaps are not necessarily silent (music, score, applause).",diff --git a/packages/dapi/src/tools/media-waveform.ts b/packages/dapi/src/tools/waveform.tssimilarity index 89%rename from packages/dapi/src/tools/media-waveform.tsrename to packages/dapi/src/tools/waveform.tsindex 72faf05f..69358449 100644--- a/packages/dapi/src/tools/media-waveform.ts+++ b/packages/dapi/src/tools/waveform.ts@@ -5,12 +5,12 @@import { z } from "zod";import { defineTool } from "../tool";import { AssetPath, Bytes, checkWindow } from "../schemas";-import { previewFields } from "./media-filmstrip";+import { previewFields } from "./filmstrip";const Silences = z.array(z.object({ start: z.number(), end: z.number() })).describe("seconds");-export const mediaWaveform = defineTool({- name: "media_waveform",+export const waveform = defineTool({+ name: "waveform",title: "Waveform preview",description:"Render the audio track of a video or audio file as a waveform PNG (local render, no credits) with a timestamp ruler: loudness over time, with silent stretches highlighted in red. A fast, token-efficient audio track preview; the silent spans are also returned as second ranges.",