Skip to content

Open source CapCut alternative.

TypeScript

feat(app): the watch reads on-screen type and hears the whole source [rebuild]

David Lea04771bfc412
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.md
index 0a7dacc1..fb1434ff 100644
--- a/docs/guides/cut/ai-assistant.md
+++ b/docs/guides/cut/ai-assistant.md
@@ -66,7 +66,7 @@ Every knowledge surface is defined once — the catalog file ships in the engine
| Attachments | Metadata JSON per asset | `<attached_assets>` on the message that carried them; on the Gemini path the newest message also carries the actual payloads (frames, images, text contents). |
| Full state | Uncapped | The get-state tool — the model calls it when the snapshot is stale or truncated. |
| The rendered frame | One 640px JPEG | The capture-frame tool: the composite an export would write at that time — footage, transitions, effects, elements, captions — at the playhead or any time asked for. |
-| The footage itself | Up to four 3×3 contact sheets per call | The watch-video tool: the browser decodes candidates on a dense steady floor, keeps only the frames that differ, tiles them into sheets stamped with their source times, and reports refined hard-cut times. Any captions or transcript fuse in as one clock, so each kept frame sits inside the speech it belongs to. Detect-silence reports dead air the same way, numbers only. |
+| The footage itself | 36 thumbnails, or 6 whole frames | The watch-video tool: the browser decodes candidates on a dense floor and on the lines the source speaks, keeping the frames that differ. A detail ladder sets how many pixels each is worth — thumbnails for coverage, the source's own frames for reading type a mosaic cell destroys. Captions and transcript fuse in as one clock; text that only repeats them is read for treatment, not words. Detect-silence reports dead air. |
The system prompt carries the voice, the deliverable rule (below), id discipline, the undo-versus-credits asymmetry, and pointers into the skills. The skills carry the deep per-area documentation — editor overview, timeline editing, watching and cutting by content, transitions and fades, graphics (titles, shapes, stickers, composite builds), audio and subtitles, stock and generation, scene productions, editing taste, media and library, publish and export — so the always-on cost stays near 21K tokens and detail is pulled only when the model works in that area.
@@ -74,11 +74,11 @@ The snapshot is a compact JSON picture of everything user-visible: project meta,
## The decision layer
-Decisions that gate, route, or pick (the turn route, queue triage, stock ranking, filler words, a described voice, a provider's failure class) are typed judgments from the judge route, thresholded in code. Reasoning and generation stay on the chat model.
+Decisions that gate, route, or pick (the turn route, queue triage, stock ranking, filler words, a voice, a provider's failure) are typed judgments from the judge route, thresholded in code. Reasoning and generation stay on the chat model.
-The judgment that routes a turn also asks whether the whole turn is one known editor action, and fills that action's arguments in the same request. When it settles — mute this clip, make it 16:9, slow that one down — the editor runs the tool and writes the line with no model round at all, and the edit lands in about a third of a second instead of three. It settles only when the action and every argument clear their floors and four guards hold: the ask names no exact figure, touches no second item, leaves no second edit undone, and wants no shot made again. Anything short resolves to nothing and the ordinary loop runs, which is why a miss costs nothing. Claude and Codex run the user's own CLI, so their turns always take the model round.
+The judgment that routes a turn also asks whether the whole turn is one known editor action, and fills its arguments in the same request. When it settles — mute this clip, make it 16:9, slow that one down — the editor runs the tool and writes the line with no model round, landing in about a third of a second. It settles only when the action and every argument clear their floors and four guards hold: the ask names no exact figure, touches no second item, leaves no second edit undone, and wants no shot made again. Anything short runs the ordinary loop, so a miss costs nothing. Claude and Codex run the user's own CLI, so their turns always take the model round.
-A turn built on footage is judged once more before it closes. The same judge reads what that turn watched, wrote down and ran against the ask, and a turn holding a slice of a source while about to describe the whole of it is handed its next pass — which source, from where, at what sampling floor — instead of its reply. The hold has to point at something the record measures: seconds nobody looked at, or calls that did not land. A doubt with no gap behind it lets the turn close, because a finished turn sent back does its work a second time.
+A turn built on footage is judged once more before it closes: the judge reads what it watched, wrote down and ran against the ask, and a turn holding a slice of a source while about to describe the whole of it is handed its next pass — which source, from where, at what sampling floor — instead of its reply. The hold has to point at something the record measures: seconds nobody looked at, or calls that did not land. A doubt with no gap behind it lets the turn close, because a finished turn sent back does its work a second time.
Deciding what the user wants is prompt text, executed by the model. The prompt orders the calls it must make each turn:
@@ -95,9 +95,9 @@ With a tab attached, tool calls execute in it against the same store the user is
- **Readable failure, then retry.** Tools validate and clamp their inputs and throw human-readable errors ("No clip with that id — call get_state for current ids"), which return to the model as tool errors it can recover from. A dropped bridge is separate from a bad request: the idempotent tool listing retries on a momentary engine hiccup so the model never lands with an empty tool set, and an "unreachable" call is one the model quietly re-issues, keeping talk of reconnecting out of the reply.
- **One undo step per turn.** History batches while the assistant is busy, so ⌘Z reverts the whole turn in one step. A sweep rides the same batch: `select_items` judges every item on a row against one description in a single fan-out, and the mutating tools take `ids`, so finding and changing fifty clips is one call each.
- **Small results.** Tools return ids and rounded numbers, plus a short note when behavior surprised ("that spot was taken — slid right").
-- **Frames become images.** A tool result's `image`/`images` data URLs leave the JSON and reach the model as real images on every path — the engine bridge emits MCP image blocks after the data text, and the Gemini loop rides the first on the function response and the rest as image parts. Watching happens in whatever model the user is chatting with; there is no side model.
-- **Watching leaves a map.** A watch records which moments and cut candidates a source holds, persisted on the asset so it survives the thread, and the first watch of a source starts a quiet background sweep that maps the rest. The map aims later watches; only sheets returned in the conversation count as footage the model has seen.
-- **Watching leaves a record.** Contact sheets live in one conversation and the media trim drops the oldest of them, so the model writes each look down: a note-source call stores what a span showed — quoted on-screen text, what happens, where the cuts land — against its source seconds, on the asset, saved with the project. Notes come back in the editor snapshot as `observed`, in full from get-state, and on every watch result beside the spans still undescribed. A close note supersedes the coarse one over the same ground, and a watch of a source that still owes a note is refused, which is what makes an hour-long source workable: it is surveyed coarsely, described, then re-watched densely only where the work is.
+- **Frames become images.** A tool result's `image`/`images` data URLs leave the JSON and reach the model as real images on every path. Watching happens in whatever model the user is chatting with; there is no side model.
+- **Watching leaves a map.** A watch records which moments and cut candidates a source holds, persisted on the asset, and the first watch starts a quiet background sweep that maps the rest. The map aims later watches; only sheets returned in the conversation count as footage the model has seen.
+- **Watching leaves a record.** Contact sheets live in one conversation and the media trim drops the oldest, so the model writes each look down: a note-source call stores what a span showed — quoted on-screen text, what happens, where the cuts land — against its source seconds, on the asset. Notes come back in the editor snapshot as `observed` and on every watch result beside the spans still undescribed. A close note supersedes the coarse one over the same ground, and a watch of a source that still owes a note is refused — which is what makes an hour-long source workable: surveyed coarsely, described, then re-watched densely only where the work is.
- **A cut can be blocked out before its footage exists.** A block is a shot with nothing in it yet: a source that owns no file and draws itself, holding its slot on track 0 for the length that shot runs. Laying a run of them is the ordinary clip placement, so "make one like this" becomes a timeline carrying the reference's own cut times, with titles and sound built on top. Footage dropped on a block takes its place and its length; from chat that is the same item replace any clip uses.
- **Speech cuts land on the words.** A recut edge is not left where the model's arithmetic put it. The audio around it is re-read, that recording's own room tone and voice level are measured, and the edge is placed a beat clear of where a word truly starts or finishes — tail and soft attack included, so a quiet take and a noisy one are cut the same way. Pace belongs to that step: fast, natural and relaxed set how big the beat is, and the fastest of them still lets every word finish. So "make it punchy" shortens the air rather than cutting into the speech, and where a pause is too short for both sides' air the two edges meet inside it instead of one clipping a word.
- **Media rides a byte budget.** Contact sheets, audio, and attachment payloads all replay inline, so before each call the session is trimmed to a 16MB media budget — oldest parts leave first, with a note naming the tools that fetch them again. Text, tool calls, tool results, and thought signatures always survive, and the trim runs on a copy so the stored session keeps everything.
@@ -130,7 +130,7 @@ A thread saves one session slot per provider, so switching models mid-thread kee
| Tool steps per turn | 24, extending to a 96 ceiling |
| Inline media per model call | 16MB, trimmed oldest-first |
| Browser tool execution | 120s |
-| One watch-video call | 600s of source, 4 sheets (36 frames), 150 decoded candidates, a 90s decode budget; the result says where coverage stopped |
+| One watch-video call | 600s of source, 150 decoded candidates, a 90s decode budget, and 36/16/6 kept frames by detail; the result says where coverage stopped |
| One blockout call | 40 shots, each 0.2-600s |
| Snapshot caps | 60 media items, 60 cues |
| One queued turn job | 1MB of messages, one live turn per project |
@@ -145,9 +145,9 @@ A hidden hermetic test provider exercises the full engine bridge — context, to
## Measuring the assistant
-A live eval scores the chat brain's behavior and speed together. Each of its cases replays a real composer turn against the live model — through the same loop code production runs — and asserts what came back: the reply, the tools called, the gate verdict, tools that must stay sheathed, and the reply's shape (no leaked scaffolding, no trailing upsell), with a light-model judge leaving tone notes beside the pass/fail. All of it is timed: the gate call, time to first token, every round, every tool serve. Cases carry a bucket (chat turns, single-tool edits, multi-tool composed edits) so the numbers aggregate by the kind of turn a user feels.
+A live eval scores the chat brain's behavior and speed together. Each case replays a real composer turn against the live model, through the same loop production runs, and asserts what came back: the reply, the tools called, the gate verdict, tools that must stay sheathed, and the reply's shape, with a light-model judge leaving tone notes beside the pass/fail. All of it is timed. Cases carry a bucket — chat turns, single-tool edits, multi-tool composed edits — so the numbers aggregate by the kind of turn a user feels.
-The committed report at `evals/cut-chat.latest-report.json` is the baseline. The improvement loop: make one change — a model role, a prompt line, a tool description — run the eval, and keep the change only when pass rates hold and the medians improve; the new report commits with the change, so the baseline always describes what ships. This is how the complexity router and the batched-round cutting flow earned their way in. Run it with the dev server up: `npm run eval:cut-chat` (add `--matrix --runs 3` to compare model configs) and `npm run eval:cut-queue` for the mid-turn triage; the README beside the harness under the site's `scripts/lib/cut-eval/` folder covers the flags, the report schema, and how to add a case. Run `node scripts/check-chat-continuity.mjs /path/to/compiled-engine` to verify disconnect recovery, replay, and the native HTTP media boundary without tokens. The tab-less paths also have end-to-end scripts, which drive a real turn against a seeded cloud project through the headless session and through the queued turn job.
+The committed report at `evals/cut-chat.latest-report.json` is the baseline. The improvement loop: make one change — a model role, a prompt line, a tool description — run the eval, and keep it only when pass rates hold and the medians improve; the new report commits with the change, so the baseline always describes what ships. Run it with the dev server up: `npm run eval:cut-chat` (`--matrix --runs 3` compares model configs) and `npm run eval:cut-queue` for mid-turn triage; the README beside the harness under the site's `scripts/lib/cut-eval/` covers the flags, the report schema, and how to add a case. `check-chat-continuity.mjs` verifies disconnect recovery and replay without tokens, and the tab-less paths have end-to-end scripts that drive a real turn through the headless session and the queued turn job.
Two mechanical paths are guarded without a model at all, because no ear settles them from a preview: caption timing (`npm run eval:cut-sync`) and speech-cut placement (`npm run eval:cut-edges`). Both build their own truth — a take of known words rendered to audio — run the production code over it, and enforce budgets on the result, so a regression in either shows up in seconds with no network and no credits.
diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.ts
index 2066439a..1a7c9dac 100644
--- a/site/src/cut/components/AiPanel.tools.ts
+++ b/site/src/cut/components/AiPanel.tools.ts
@@ -8,6 +8,7 @@
*/
import { bool, num, obj, str, type AiToolDef } from "@/cut/lib/aiToolDef";
+import { WATCH_DETAIL_NOTES, WATCH_DETAILS } from "@/cut/lib/types";
export const AI_PANEL_TOOLS = [
{
@@ -19,13 +20,18 @@ export const AI_PANEL_TOOLS = [
{
name: "watch_video",
description:
- "Watch a video source with your own eyes: samples candidate frames on a dense steady floor, keeps only the ones that actually differ, and tiles them into timestamped contact-sheet images — so every cell is a distinct moment and a gap between stamps means nothing changed there. Also returns sceneChanges, hard-cut times refined to about a third of a second of the true boundary (the time is where the new shot first appears) — though a shot shorter than the sampling step can fall between candidates, so where short cuts matter, re-watch narrow with a small interval_seconds. When the clip has captions or the source's own transcript exists, a fused timeline places each kept frame inside the speech it belongs to. Pass clip_id to watch a timeline clip's source (the result includes that clip's source↔timeline time math) or asset_id for any project video or image. The stamp burned into each cell is SOURCE seconds — what trim_clip's in/out use — not timeline seconds. Coverage ends where the result says it does: `coveredTo` is how far you have looked and `unwatchedSeconds` is how much of the source you have NOT — everything before coveredTo was seen, nothing after it was. Continue from coveredTo until unwatchedSeconds is 0 before you describe, summarize, or reproduce the source as a whole; a `to` you chose yourself ends a call with truncated false and the rest of the video still unseen. Watching builds the source's frame map (moment times + cut candidates), persisted on the asset; media entries list those spans as `mapped`, and your first watch starts a quiet background sweep that maps the rest (`watching: true` while it runs). The map aims your watches — a mapped span is NOT footage you have seen; only sheets returned in this conversation are, so watch any span you need to actually look at. Every pass ends with note_source: the sheets are conversation-local and drop out as the chat grows, so write what this stretch showed against its source seconds and it stays on the asset for every later turn. The result carries `recorded` (what is already written about the stretch you just watched) and `unnoted` (the spans of this source nothing describes yet); watching a source that still owes a note is refused. Read the watching-and-cutting skill before editing footage by content.",
+ "Watch a video source with your own eyes: samples candidate frames on a dense steady floor plus one just after each line the source speaks — what a video shows changes when what it says changes, so a transcribed source is read on its own clock and a caption track lands a frame per card — keeps only the ones that actually differ, and tiles them into timestamped contact-sheet images — so every cell is a distinct moment and a gap between stamps means nothing changed there. Also returns sceneChanges, hard-cut times refined to about a third of a second of the true boundary (the time is where the new shot first appears) — though a shot shorter than the sampling step can fall between candidates, so where short cuts matter, re-watch narrow with a small interval_seconds. When the clip has captions or the source's own transcript exists, a fused timeline places each kept frame inside the speech it belongs to. Pass clip_id to watch a timeline clip's source (the result includes that clip's source↔timeline time math) or asset_id for any project video or image. `detail` decides how many pixels each kept frame is worth: scan tiles many small cells for coverage, read enlarges them for on-screen text, original hands back the source's own frames for matching type, colour and edges. A contact-sheet cell is a quarter of the frame's width, so a small caption arrives too small to read — when the job is words on screen or a look to reproduce, spend the pixels instead of guessing. `textFits` comes back with the frames: the parts of the picture that stay quiet across everything just watched, inside whatever the project's guides keep clear, quietest first — where a title or caption can go without covering the subject or the source's own on-screen text. The stamp burned into each cell is SOURCE seconds — what trim_clip's in/out use — not timeline seconds. Coverage ends where the result says it does: `coveredTo` is how far you have looked and `unwatchedSeconds` is how much of the source you have NOT — everything before coveredTo was seen, nothing after it was. Reading is measured apart from seeing: `unreadSeconds` is how much of the source no pass has looked at closely enough to read its type, and `unreadFrom` is where the next close pass starts. Both ride the asset, so they carry what every earlier chat read as well as this one. Continue from coveredTo until unwatchedSeconds is 0 before you describe, summarize, or reproduce the source as a whole; a `to` you chose yourself ends a call with truncated false and the rest of the video still unseen. Watching builds the source's frame map (moment times + cut candidates), persisted on the asset; media entries list those spans as `mapped`, and your first watch starts a quiet background sweep that maps the rest (`watching: true` while it runs). The map aims your watches — a mapped span is NOT footage you have seen; only sheets returned in this conversation are, so watch any span you need to actually look at. Every pass ends with note_source: the sheets are conversation-local and drop out as the chat grows, so write what this stretch showed against its source seconds and it stays on the asset for every later turn. The result carries `recorded` (what is already written about the stretch you just watched) and `unnoted` (the spans of this source nothing describes yet); watching a source that still owes a note is refused. Read the watching-and-cutting skill before editing footage by content.",
inputSchema: obj({
clip_id: str("Video clip id, track 0 or overlay (defaults from/to to its trimmed in/out)"),
asset_id: str("Project asset id (video or image) — watch the source itself"),
from: num("Source start s (default: the clip's in, else 0)"),
to: num("Source end s (default: the clip's out, else the source's end; spans at most 600s per call)"),
interval_seconds: num("Seconds between candidate frames, 0.5–30 (default 1; near-duplicates are dropped, so a dense default costs nothing)"),
+ detail: {
+ type: "string",
+ enum: [...WATCH_DETAILS],
+ description: `How much of each frame comes back (default scan). ${WATCH_DETAILS.map((d) => `${d}: ${WATCH_DETAIL_NOTES[d]}`).join(". ")}. The moments kept are the same at every step — only the pixels spent on each change, and small on-screen type is unreadable below "read". Scan a source first, then re-watch the stretch that matters at the detail its content needs.`,
+ },
project_link: str(
"A reference project's link or id (what read_project took): clip_id / asset_id then name that project's items, so you can look at footage of the reference whose notes and transcript leave it dark before mapping roles. A reference look is for deciding — nothing is written to the reference and no note is owed."
),
@@ -104,7 +110,7 @@ export const AI_PANEL_TOOLS = [
{
name: "listen_audio",
description:
- "Listen with your own ears to a project asset's sound — an audio asset, or the audio track of a video (its speech, music, burned-in narration) — so you can answer what it says or how it sounds. For the WORDS alone, check the source's transcript first: media entries marked speech: 'transcribed' carry timed segments (get_state includes them) at no audio cost; listen when you need the sound itself — tone, music, delivery, timing by ear. The sound rides back inline (≈12MB cap). Pass clip_id for a timeline clip's source (scopes to its trim) or asset_id for the whole source; add from/to (source seconds) to hear one stretch of a long file. Audio the user attached to their message already plays in it. To WRITE a caption track, use subtitles_generate.",
+ "Listen with your own ears to a project asset's sound — an audio asset, or the audio track of a video (its speech, music, burned-in narration) — so you can answer what it says or how it sounds. For the WORDS alone, check the source's transcript first: media entries marked speech: 'transcribed' carry timed segments (get_state includes them) at no audio cost; listen when you need the sound itself — tone, music, delivery, timing by ear. The sound rides back inline under a byte cap, so the seconds one call carries depend on the source — a stretch too long comes back as its opening, and `coveredTo` says where it stopped. Coverage works like watching's: `unheardSeconds` is how much of the source nobody has played and `unheardFrom` is where the next listen starts, both persisted on the asset, so judging how a whole source sounds means listening on from `unheardFrom` until `unheardSeconds` is 0. Pass clip_id for a timeline clip's source (scopes to its trim) or asset_id for the whole source; add from/to (source seconds) to hear one stretch of a long file. Write what you heard down with note_source — the audio leaves this conversation as it grows, the note stays on the source. Audio the user attached to their message already plays in it. To WRITE a caption track, use subtitles_generate.",
inputSchema: obj({
clip_id: str("Clip id — video or soundtrack (defaults from/to to its trimmed in/out)"),
asset_id: str("Project asset id (audio or video) — listen to the whole source"),
diff --git a/site/src/cut/components/AiPanel.tsx b/site/src/cut/components/AiPanel.tsx
index cb0909ac..29e25951 100644
--- a/site/src/cut/components/AiPanel.tsx
+++ b/site/src/cut/components/AiPanel.tsx
@@ -256,8 +256,8 @@ function readThreads(projectId: string): ChatThread[] {
}
/** Persisted copies drop frame payloads (data URLs) from tool outputs — one
- * watch_video result carries ~1MB of contact sheets and localStorage holds a
- * few MB per origin. The live thread keeps its images; replayed turns only
+ * watch_video result carries a few MB of frames and localStorage holds a few
+ * MB per origin in total. The live thread keeps its images; replayed turns only
* ever reuse text parts, so nothing downstream misses them. */
function slimForStorage(list: ChatThread[]): ChatThread[] {
const bulky = (v: unknown) =>
diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.ts
index c47369e9..352ccfea 100644
--- a/site/src/cut/lib/aiTools.ts
+++ b/site/src/cut/lib/aiTools.ts
@@ -15,7 +15,8 @@ import { requestSharing } from "@/cut/lib/sharingClient";
import { copyLibraryForSharing } from "@/cut/lib/libraryShareCopy";
import { DONKEYCUT_CANONICAL } from "@/cut/lib/hosts";
-import { GUIDE_IDS, guideFits, guidePreset, isGuideId, sanitizeGuideLines, type GuideId } from "./guides";
+import { GUIDE_IDS, guideFits, guidePreset, isGuideId, safeAreaOf, sanitizeGuideLines, type GuideId } from "./guides";
+import { textSpots } from "./textPlace";
import {
ALL_EFFECT_IDS,
autoGradeFromImageData,
@@ -125,6 +126,7 @@ import {
renderAudioSpanWav,
sampleWatchFrames,
scanSourceSpeech,
+ watchGeometry,
} from "./media";
import { convertAssetToMp4 } from "./mediaConvert";
import { isLottieAsset } from "./lottieAssets";
@@ -136,7 +138,13 @@ import {
matteBakesAvailable,
useMatteBakes,
} from "./removal/bakeJobs";
-import { blobToInlineAudio, refToInlineAudio, visualRefs, type InlineImage } from "./refMedia";
+import {
+ blobToInlineAudio,
+ MAX_AUDIO_BYTES,
+ refToInlineAudio,
+ visualRefs,
+ type InlineImage,
+} from "./refMedia";
import { characterPrompt, stockAspectDims, stockTitle } from "./stock";
import { STOCK_IMAGES } from "./stockManifest";
import { STOCK_VIDEOS } from "./stockVideoManifest";
@@ -155,8 +163,15 @@ import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl } from
import { buildAiContext, describeDoc, hiddenFromChat, placedAssetIds } from "./aiContext";
import { sampleClipFrameData } from "./previewCanvas";
import { CAPTION_STYLES, laneCues, subtitleLaneCount } from "./subtitles";
-import { fuseTimeline, renderFusedTimeline } from "./watch/fuse";
-import { mergeWatch, mergeWatchNotes, uncoveredSeconds, unnotedSpans } from "./watch/merge";
+import { fuseTimeline, renderFusedTimeline, speechOnsets, speechOver } from "./watch/fuse";
+import {
+ mergeHeard,
+ mergeWatch,
+ mergeWatchNotes,
+ nextUncoveredSpan,
+ uncoveredSeconds,
+ unnotedSpans,
+} from "./watch/merge";
import { syncLines } from "./lyricSync";
import { transcriptWords as cueWords, wordTimesFor } from "./textWords";
import {
@@ -233,6 +248,9 @@ import {
type TransitionStyle,
type VideoClip,
uploadedFontId,
+ WATCH_DETAIL_NOTES,
+ WATCH_DETAILS,
+ type WatchDetail,
} from "./types";
const round2 = (n: number) => Math.round(n * 100) / 100;
@@ -877,8 +895,11 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
throw new ToolError(`"${asset.name}" is audio — listen_audio hears it, detect_silence finds its dead air.`);
// One path for both backends: the browser decodes the source (the same
// URL the preview plays) and the shared selector keeps distinct frames.
+ const detail: WatchDetail = WATCH_DETAILS.includes(input.detail as WatchDetail)
+ ? (input.detail as WatchDetail)
+ : "scan";
if (asset.type === "image") {
- const body = await makeStillFrame(asset.url).catch((e) => {
+ const body = await makeStillFrame(asset.url, detail).catch((e) => {
throw new ToolError(e instanceof Error ? e.message : "Could not read the image.");
});
return {
@@ -887,6 +908,13 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
note: "A still image — one frame, no time axis.",
};
}
+ // Where the frames go: the steady sweep, plus a candidate on each line
+ // the source speaks. What it says and what it shows move together.
+ const aimed = speechOnsets(
+ sourceSpeech(s, asset, reference ? null : clip, from, to ?? asset.duration),
+ from,
+ to ?? asset.duration,
+ );
// The sweep yields its decoders to this call for its whole duration.
// The budget keeps the call inside the tool bridge's 120s deadline: a
// slow decode salvages what it covered (truncated + coveredTo) rather
@@ -896,6 +924,8 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
from,
...(to !== undefined ? { to } : {}),
...(isNum(input.interval_seconds) ? { interval: input.interval_seconds } : {}),
+ detail,
+ at: aimed,
budgetMs: 90_000,
})
).catch((e) => {
@@ -917,6 +947,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
to: body.coveredTo,
frames: body.frames.map((f) => ({ t: f.t, via: f.via })),
sceneChanges: body.sceneChanges,
+ readClosely: detail !== "scan",
}),
});
}
@@ -929,29 +960,27 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
// source — metadata only, merged in as segments land.
queueWatchSweep(asset.id);
}
- const unwatched = uncoveredSeconds(
- reference ? asset.watch : useEditor.getState().assets.find((x) => x.id === asset.id)?.watch,
- asset.duration
- );
+ const seen = reference
+ ? asset.watch
+ : useEditor.getState().assets.find((x) => x.id === asset.id)?.watch;
+ const unwatched = uncoveredSeconds(seen, asset.duration);
+ // Seen and read are separate measures: a stretch surveyed at scan is
+ // covered, and its words are still unknown.
+ const closeRanges = { ranges: seen?.read ?? [] };
+ const unread = uncoveredSeconds(closeRanges, asset.duration);
+ const nextUnread = nextUncoveredSpan(closeRanges, asset.duration);
// Tile the kept frames 3×3 and stamp each cell's source time.
- const sheets = await composeSheets(body.frames).catch(() => null);
+ const sheets = await composeSheets(body.frames, detail).catch(() => null);
const keptTimes = body.frames.map((f) => f.t);
// Kept frames woven into the speech on one clock, so the model reads
- // precomputed frame↔speech alignment. Project captions win (cue times
- // are timeline seconds, mapped to source through the clip); with none,
- // the asset's own transcript serves — it is already source time, so it
- // fuses for asset-only watches too.
- let speech: { start: number; end: number; text: string }[] = [];
- if (clip && !reference) {
- speech = laneCues(s.subtitles, s.subtitleLane)
- .map((c) => ({
- start: retimeOf(clip).srcAt(c.start - clip.start),
- end: retimeOf(clip).srcAt(c.end - clip.start),
- text: c.text,
- }))
- .filter((c) => c.end > from && c.start < body.coveredTo);
- }
- if (speech.length === 0) speech = (liveAsset ?? asset).speech?.segments ?? [];
+ // precomputed frame↔speech alignment instead of matching by eye.
+ const speech = sourceSpeech(
+ s,
+ liveAsset ?? asset,
+ reference ? null : clip,
+ from,
+ body.coveredTo,
+ );
let timelineText: string | undefined;
if (speech.length > 0) {
timelineText = renderFusedTimeline(
@@ -961,13 +990,38 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
// Each cell's time plus why the selector kept it — "global" a hard cut,
// "action" local motion, "settled" new settled detail (text, ink, UI).
const cellInfo = body.frames.map((f) => ({ t: round2(f.t), via: f.via }));
+ // Where a card could sit over what was just watched: the quiet parts of
+ // the picture, inside whatever the project's guides keep clear. The
+ // watch already measured the busy ones to choose these frames.
+ const showing = s.guides.filter((g) => guideFits(g, s.aspect));
+ const safe = safeAreaOf(showing.length > 0 ? showing : ["margins"], s.aspect);
+ const spots = safe && !reference ? textSpots(body.edges, safe) : [];
const perSheet = sheets ? sheets.map((sh) => sh.frames.length) : [cellInfo.length];
let cellAt = 0;
+ // How to read the images, and what the rest of the ladder is for. The
+ // frames are the same moments at every detail; only the pixels differ,
+ // and small type is unreadable until they are spent on it.
+ const tiled =
+ watchGeometry(detail).grid > 1
+ ? `Cells read left→right then top→bottom; each stamp is SOURCE seconds${reference ? " of the reference's file" : ""}. `
+ : `One frame per image, in sheetFrames order — SOURCE seconds${reference ? " of the reference's file" : ""}, no burned stamp. `;
+ const ladder =
+ detail === "original"
+ ? "These are the source's own pixels: quote on-screen text exactly and name the type, weight, spacing and colours you can see. "
+ : `Detail is "${detail}" (${WATCH_DETAIL_NOTES[detail]}). If type, UI or fine treatment is too small to read here, re-watch that stretch with detail "${detail === "scan" ? "read" : "original"}" — the same moments with more pixels on each, fewer per call — rather than guessing at what it says. `;
return {
images: sheets ? sheets.map((sh) => sh.image) : body.frames.map((f) => f.image),
sheetFrames: perSheet.map((n) => cellInfo.slice(cellAt, (cellAt += n))),
distinctFrames: body.frames.length,
candidates: body.candidates,
+ // The span this pass covered and how closely it looked at it. A
+ // stretch seen only at "scan" has been surveyed, not read.
+ detail,
+ from: round2(from),
+ // Seconds no pass has read at a detail that carries type, and where
+ // the next close pass would start.
+ unreadSeconds: unread,
+ ...(nextUnread ? { unreadFrom: round2(nextUnread.from) } : {}),
sceneChanges: body.sceneChanges.map(round2),
coveredTo: round2(body.coveredTo),
truncated: body.truncated,
@@ -986,7 +1040,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
from: round2(g.from),
to: round2(g.to),
})),
- source: { assetId: asset.id, name: asset.name, duration: round2(asset.duration) },
+ source: sourceRef(asset),
...(reference
? { reference: { projectId: reference.ref.projectId, name: reference.ref.doc.name ?? "" } }
: {}),
@@ -1016,12 +1070,18 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
}
: {}),
...(timelineText ? { timeline: timelineText } : {}),
+ // Frame fractions, quietest first: where a title or caption can go
+ // over this stretch without covering the picture or the platform's UI.
+ ...(spots.length > 0 ? { textFits: spots } : {}),
note: reference
- ? "Cells read left→right then top→bottom; each stamp is SOURCE seconds of the reference's file. This is a look at a reference project for deciding which of this project's sources plays its part — nothing is written to the reference and no note is owed." +
+ ? tiled +
+ "This is a look at a reference project for deciding which of this project's sources plays its part — nothing is written to the reference and no note is owed. " +
+ ladder +
(sheets ? "" : " (Sheets unavailable — each image is one frame; sheetFrames lists the times.)")
- : "Cells read left→right then top→bottom; each stamp is SOURCE seconds. " +
- "Cells are distinct moments (near-duplicates removed), so gaps between stamps mean nothing changed there. " +
- "sheetFrames says why each cell was kept: global = hard cut, action = local motion, settled = new settled detail (text/UI)." +
+ : tiled +
+ "Frames are distinct moments (near-duplicates removed), so gaps between stamps mean nothing changed there. " +
+ "sheetFrames says why each was kept: global = hard cut, action = local motion, text = type landing or swapping, settled = other new settled detail. " +
+ ladder +
(unwatched > 0.5
? ` You have seen ${round2(from)}-${round2(body.coveredTo)}s of a ${round2(asset.duration)}s source; ` +
`${unwatched}s of it has NOT been looked at (this call's range plus anything else already watched)` +
@@ -1052,7 +1112,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
if (owed && spanIsNoted(cur?.watch, owed)) unwritten.delete(asset.id);
const unnoted = unnotedSpans(cur?.watch, asset.duration);
return {
- source: { assetId: asset.id, name: asset.name, duration: round2(asset.duration) },
+ source: sourceRef(asset),
notes: (cur?.watch?.notes ?? []).map((n) => ({
from: round2(n.from),
to: round2(n.to),
@@ -1091,7 +1151,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
? { timeline: { start: toTimeline(x.start), end: toTimeline(x.end) } }
: {}),
})),
- source: { assetId: asset.id, name: asset.name, duration: round2(asset.duration) },
+ source: sourceRef(asset),
...(clip
? {
clip: {
@@ -1242,7 +1302,7 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
},
}
: {}),
- source: { assetId: asset.id, name: asset.name, duration: round2(asset.duration) },
+ source: sourceRef(asset),
note:
grid.beats.length === 0
? "No steady pulse heard — the source reads as speech or ambience. The user can place beats by hand (⌥-click the clip bar)."
@@ -1330,25 +1390,39 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {
// range, gets its audio track pulled off by the engine first — so the
// muxed video never travels and a long source still fits the inline cap.
const wholeAudio = asset.type === "audio" && !clip && !isNum(input.from) && !isNum(input.to);
- const inline = wholeAudio
- ? await refToInlineAudio(refFromAsset(asset))
- : await listenToSource(projectId, asset, from, to);
+ const whole = wholeAudio ? await refToInlineAudio(refFromAsset(asset)) : null;
+ // A whole audio file that clears the cap falls through to the same
+ // fitting path a range takes, so length truncates instead of refusing.
+ const heard = whole
+ ? { inline: whole, to: asset.duration }
+ : await listenSpan(projectId, asset, from, to);
// Listening shows interest in the source's sound — queue the background
// sweep so its transcript (and, for video, its visual map) fills in.
queueWatchSweep(asset.id);
- if (!inline)
- throw new ToolError(
- "That stretch is too long to listen to inline (≈12MB cap) — pass a narrower from/to."
- );
+ const live = useEditor.getState().assets.find((x) => x.id === asset.id);
+ if (live && heard.to > from)
+ s.updateAsset(asset.id, { watch: mergeHeard(live.watch, { from, to: heard.to }) });
+ const listened = {
+ ranges: useEditor.getState().assets.find((x) => x.id === asset.id)?.watch?.heard ?? [],
+ };
+ const unheard = uncoveredSeconds(listened, asset.duration);
+ const nextUnheard = nextUncoveredSpan(listened, asset.duration);
// The `audio` data URL leaves the JSON in the chat transport and rides
// to the model as an input_audio part, the way attachments do.
return {
name: asset.name,
+ source: sourceRef(asset),
duration: round2(asset.duration),
...(clip ? { clipId: clip.id } : {}),
- ...(from > 0 ? { from: round2(from) } : {}),
- ...(to !== undefined ? { to: round2(to) } : {}),
- audio: `data:${inline.mimeType};base64,${inline.data}`,
+ from: round2(from),
+ // Where this play stopped, and how much of the source nobody has
+ // heard — the sound's answer to coveredTo and unwatchedSeconds. The
+ // cap is bytes, so the seconds that fit depend on the source.
+ coveredTo: round2(heard.to),
+ truncated: heard.to < (to ?? asset.duration) - 0.05,
+ unheardSeconds: unheard,
+ ...(nextUnheard ? { unheardFrom: round2(nextUnheard.from) } : {}),
+ audio: `data:${heard.inline.mimeType};base64,${heard.inline.data}`,
};
},
@@ -4865,6 +4939,54 @@ function resolveWatchRange(
return { projectId, asset, clip, speed, from, to };
}
+/** What a source can still be asked for, from what is already known. A steer
+ * that names a tool the source refuses spends a round on an error: an image
+ * has no sound, a music asset has no picture, and a video's sound is proven by
+ * its transcript or by having been played. */
+function soundOf(asset: MediaAsset): "yes" | "none" | "unknown" {
+ if (asset.type === "image") return "none";
+ if (asset.type === "audio") return "yes";
+ if (asset.watch?.heard?.length) return "yes";
+ if (asset.speech) return asset.speech.noSpeech ? "none" : "yes";
+ return "unknown";
+}
+
+/** How every source-reading tool names the source it read. `type` and `sound`
+ * say which tools it can answer at all, so nothing downstream asks a music
+ * file for its picture or a silent take for its sound. */
+function sourceRef(asset: MediaAsset) {
+ return {
+ assetId: asset.id,
+ name: asset.name,
+ duration: round2(asset.duration),
+ type: asset.type,
+ sound: soundOf(asset),
+ };
+}
+
+/** What is said over a stretch of a source, on the source's own clock.
+ * Project captions win where they cover it — their cue times are timeline
+ * seconds, mapped back through the clip's retime — and the asset's own
+ * transcript, already source time, covers the rest. The range decides: a
+ * caption lane holding cues for one clip leaves every other clip to its
+ * source's transcript. */
+function sourceSpeech(
+ s: ReturnType<typeof useEditor.getState>,
+ asset: MediaAsset,
+ clip: { id: string; start: number; in: number; out: number; speed?: number; speedCurve?: SpeedNode[]; reverse?: boolean; smoothSlow?: boolean } | null,
+ from: number,
+ to: number,
+): { start: number; end: number; text: string }[] {
+ const cues = clip
+ ? laneCues(s.subtitles, s.subtitleLane).map((c) => ({
+ start: retimeOf(clip).srcAt(c.start - clip.start),
+ end: retimeOf(clip).srcAt(c.end - clip.start),
+ text: c.text,
+ }))
+ : [];
+ return speechOver(cues, asset.speech?.segments ?? [], from, to);
+}
+
/** A reference link the model typed into a tool argument, opened. Every
* failure is the reader's own plain reason. */
async function openReferenceLink(raw: unknown): Promise<ReferenceProject> {
@@ -5000,18 +5122,16 @@ async function fetchSilences(
/** Pull a source's audio track off (video and audio alike) and inline it for
* the model; null when the range clears the inline cap. Local: the engine
* extracts mono AAC. Cloud: the browser renders the span to 16 kHz mono WAV. */
-async function listenToSource(
+async function audioSpanBlob(
projectId: string,
asset: MediaAsset,
from: number,
to: number | undefined,
-): Promise<InlineImage | null> {
- if (getBackend().kind !== "local") {
- const wav = await renderAudioSpanWav(asset.url, from, to).catch((e) => {
+): Promise<Blob> {
+ if (getBackend().kind !== "local")
+ return renderAudioSpanWav(asset.url, from, to).catch((e) => {
throw new ToolError(e instanceof Error ? e.message : "Could not read the audio.");
});
- return blobToInlineAudio(wav);
- }
const res = await apiFetch(`/api/cut/projects/${projectId}/audio`, {
method: "POST",
headers: { "Content-Type": "application/json" },
@@ -5021,7 +5141,40 @@ async function listenToSource(
const body = await apiJson<{ error?: string }>(res).catch(() => ({ error: undefined }));
throw new ToolError(body.error ?? "Could not read the audio.");
}
- return blobToInlineAudio(await res.blob());
+ return res.blob();
+}
+
+/** Bound the work per call the way a watch does; the caller listens on from
+ * coveredTo. */
+const LISTEN_MAX_S = 600;
+
+/**
+ * A span's sound, shrunk to whatever the model can take at once. The two
+ * backends encode differently — the browser renders 16kHz mono WAV, the engine
+ * hands back AAC — so the seconds that fit the inline cap differ by an order
+ * of magnitude and neither is knowable up front. The bytes that come back say
+ * what a second of THIS source costs, which is what the retry sizes itself
+ * from. A stretch too long comes back as its opening, with the span it
+ * actually covered, rather than as a refusal the caller has to guess past.
+ */
+async function listenSpan(
+ projectId: string,
+ asset: MediaAsset,
+ from: number,
+ to: number | undefined,
+): Promise<{ inline: InlineImage; to: number }> {
+ const end = Math.min(to ?? from + LISTEN_MAX_S, from + LISTEN_MAX_S);
+ let span = end - from;
+ for (let attempt = 0; attempt < 3 && span > 0.5; attempt++) {
+ const blob = await audioSpanBlob(projectId, asset, from, round2(from + span));
+ const inline = await blobToInlineAudio(blob);
+ if (inline) return { inline, to: round2(from + span) };
+ const perSecond = blob.size / span;
+ // Leave room for the base64 inflation the cap already accounts for.
+ const fits = (MAX_AUDIO_BYTES * 0.9) / perSecond;
+ span = Math.min(span * 0.9, fits);
+ }
+ throw new ToolError("Could not read a stretch of that audio short enough to listen to.");
}
/** One line per timeline item for a sweep to judge: what it is, when it
diff --git a/site/src/cut/lib/media.ts b/site/src/cut/lib/media.ts
index 7bdd6781..8069241e 100644
--- a/site/src/cut/lib/media.ts
+++ b/site/src/cut/lib/media.ts
@@ -51,7 +51,7 @@ import {
import { pickThumbTimes, type ThumbProbe } from "./filmstrip";
import { reportSwallowed } from "./report";
import { assetIdsInUse, useEditor } from "./store";
-import type { AssetBeats, AssetType, AudioClip, MediaAsset, ProjectSummary, StoredAsset, VideoClip, WatchKeepReason } from "./types";
+import type { AssetBeats, AssetType, AudioClip, MediaAsset, ProjectSummary, StoredAsset, VideoClip, WatchDetail, WatchKeepReason } from "./types";
import { contentRect, IMAGE_CLIP_SECONDS, mediaUrl } from "./types";
import { createDedupSelector, DEDUP_TUNING } from "./watch/dedupSelector";
import { diffCellPct, frameSig } from "./watch/signatures";
@@ -1315,19 +1315,87 @@ export async function renderAudioSpanWav(
// composeSheets. One sampler serves the local and cloud backends alike — the
// browser decodes the source (local media rides the engine's media URL, cloud
// media the CDN) and the shared selector keeps the frames that differ.
-const SHEET_GRID = 3; // cells per row and column
-const SHEET_CELL = 480; // cell long side, px
const SHEET_GAP = 4; // tile margin and padding, px
-const SHEET_MAX = 4; // sheets per call
-const SHEET_QUALITY = 0.8; // jpeg encode
const CANDIDATE_CAP = 150; // decodes per call — the densest sweep
+const SELECT_CELL = 480; // what the selector decodes at when the frames are bigger
const REFINE_TARGET_S = 0.35; // localize each hard cut to about this window
const REFINE_MAX_PROBES = 48; // extra seeks the refinement pass may spend per call
+const AIMED_SHARE = 0.5; // most of the cap aimed moments may take from the floor
+const MERGE_EPS = 0.15; // two candidate times this close are one decode
+
+/** The candidate list: the moments a caller knows matter, plus a steady floor
+ * across everything else. The floor is built at whatever density the room
+ * left allows, so aiming widens the sweep instead of cutting it short — every
+ * second in [from, to] still sits between two candidates. */
+function candidateTimes(
+ from: number,
+ to: number,
+ interval: number,
+ at: number[]
+): number[] {
+ const inRange = at.filter((t) => t >= from && t < to).sort((a, b) => a - b);
+ // More lines than the share allows are thinned across the whole span. Taking
+ // the first of them would aim the head of a densely spoken stretch and leave
+ // its tail sampled thinner than the floor alone would have.
+ const cap = Math.floor(CANDIDATE_CAP * AIMED_SHARE);
+ const every = Math.max(1, Math.ceil(inRange.length / cap));
+ const aimed: number[] = [];
+ for (let i = 0; i < inRange.length; i += every) {
+ const t = inRange[i];
+ if (aimed.length === 0 || t - aimed[aimed.length - 1] >= MERGE_EPS) aimed.push(round2(t));
+ }
+ const room = Math.max(1, CANDIDATE_CAP - aimed.length);
+ const step = Math.max(interval, (to - from) / room);
+ const floor: number[] = [];
+ for (let t = from; t < to && floor.length < room; t += step) floor.push(round2(t));
+ const times: number[] = [];
+ for (const t of [...aimed, ...floor].sort((a, b) => a - b))
+ if (times.length === 0 || t - times[times.length - 1] >= MERGE_EPS) times.push(t);
+ return times;
+}
+
+/** How much of each kept frame reaches the model, from the tiled thumbnails
+ * that answer "what happens here" up to the frames themselves.
+ *
+ * Every step trades frames for pixels at roughly even cost, because a sheet
+ * is billed by its area: a 9:16 source gives cells of 405×720, 608×1080 and
+ * 1080×1920, where a 46px caption lands at about 13px, 20px and full size.
+ * Small type is the reason the ladder exists — a mosaic cell throws away the
+ * weight, spacing and edge treatment that reading text or matching a look
+ * depends on, and nothing downstream can put them back.
+ *
+ * `cell` is the cell's long side in px. */
+const WATCH_DETAIL: Record<
+ WatchDetail,
+ { cell: number; grid: number; sheets: number; quality: number; rewalk: boolean }
+> = {
+ /** Coverage: what happens, where the cuts fall, how long a stretch runs. */
+ scan: { cell: 720, grid: 3, sheets: 4, quality: 0.8, rewalk: false },
+ /** Reading: on-screen text, UI, anything whose words matter. */
+ read: { cell: 1080, grid: 2, sheets: 4, quality: 0.86, rewalk: false },
+ /** Matching: type, colour, edges and grain as the source has them. The
+ * frames are big enough that decoding every candidate at this size would
+ * cost far more than the handful that are kept, so this step selects small
+ * and walks the kept times a second time. */
+ original: { cell: 1920, grid: 1, sheets: 6, quality: 0.92, rewalk: true },
+};
+
+/** Frames one call returns at this detail, and the geometry behind them.
+ * A cell the source is already smaller than hands back the source's own
+ * pixels, which is what "original" is for a phone or 1080p source; a 4K or
+ * larger one meets the ceiling, because six 4K frames are several times the
+ * whole per-call media budget on their own. */
+export const watchGeometry = (detail: WatchDetail = "scan") => {
+ const g = WATCH_DETAIL[detail] ?? WATCH_DETAIL.scan;
+ return { ...g, maxFrames: g.grid * g.grid * g.sheets };
+};
/** The watch sample: kept frames, ascending source time, each carrying why
* the selector kept it (WatchKeepReason in types.ts). */
export interface WatchFrames {
frames: { t: number; image: string; via: WatchKeepReason }[];
+ /** Per kept frame, the TEXT_GRID² edge energies the selector measured. */
+ edges: Float32Array[];
candidates: number;
sceneChanges: number[];
coveredTo: number;
@@ -1336,17 +1404,26 @@ export interface WatchFrames {
const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
-// Round the cell's short side to even, so encoded dimensions stay codec-safe.
-const cellDims = (w: number, h: number): [number, number] =>
- w >= h
- ? [SHEET_CELL, Math.max(2, 2 * Math.round((SHEET_CELL * h) / w / 2))]
- : [Math.max(2, 2 * Math.round((SHEET_CELL * w) / h / 2)), SHEET_CELL];
+// A cell the source is already smaller than passes the frame through at its
+// own size — a watch never invents detail by upscaling, and the even-rounding
+// below would resample an odd-sized frame to no purpose. A sized cell rounds
+// its short side to even, so encoded dimensions stay codec-safe.
+const cellDims = (w: number, h: number, cell: number): [number, number] => {
+ if (cell <= 0 || cell >= Math.max(w, h)) return [Math.max(1, w), Math.max(1, h)];
+ return w >= h
+ ? [cell, Math.max(2, 2 * Math.round((cell * h) / w / 2))]
+ : [Math.max(2, 2 * Math.round((cell * w) / h / 2)), cell];
+};
-/** Watch a still image: one downscaled frame, no time axis. */
-export async function makeStillFrame(sourceUrl: string): Promise<WatchFrames> {
+/** Watch a still image: one frame at the asked-for detail, no time axis. */
+export async function makeStillFrame(
+ sourceUrl: string,
+ detail: WatchDetail = "scan"
+): Promise<WatchFrames> {
const img = await decodeRasterImageUrl(sourceUrl);
if (!img || img.width === 0 || img.height === 0) throw new Error("Could not read the image.");
- const [w, h] = cellDims(img.width, img.height);
+ const g = watchGeometry(detail);
+ const [w, h] = cellDims(img.width, img.height, g.cell);
const canvas = createRasterCanvas(w, h);
const ctx = canvas.getContext("2d") as CanvasRenderingContext2D | null;
if (!ctx) throw new Error("Could not read the image.");
@@ -1354,11 +1431,12 @@ export async function makeStillFrame(sourceUrl: string): Promise<WatchFrames> {
ctx.drawImage(img.source, 0, 0, w, h);
// A picture read through an element (a host that sends no CORS header)
// taints the canvas, and the encode is where that surfaces.
- const image = await rasterCanvasToDataUrl(canvas, "image/jpeg", SHEET_QUALITY).catch(() => {
+ const image = await rasterCanvasToDataUrl(canvas, "image/jpeg", g.quality).catch(() => {
throw new Error("Could not read the image.");
});
return {
frames: [{ t: 0, image, via: "first" }],
+ edges: [],
candidates: 1,
sceneChanges: [],
coveredTo: 0,
@@ -1366,8 +1444,9 @@ export async function makeStillFrame(sourceUrl: string): Promise<WatchFrames> {
};
}
-/** Watch a video source: decode candidates on a dense steady floor, then let
- * the shared selector keep the frames that actually differ. sceneChanges are
+/** Watch a video source: decode candidates on a dense steady floor plus any
+ * moments `at` aims at, then let the shared selector keep the frames that
+ * actually differ. sceneChanges are
* hard-cut times: each "global" keep says a cut lies inside the grid step
* before it, and a short bisection pass narrows that window to about
* REFINE_TARGET_S — the reported time is where the new shot is first seen.
@@ -1393,6 +1472,12 @@ export async function sampleWatchFrames(
metadataOnly?: boolean;
shouldPause?: () => boolean;
budgetMs?: number;
+ detail?: WatchDetail;
+ /** Moments worth a decode on top of the steady floor. What is on screen
+ * changes when what is heard changes — a cue lands, a line starts, a beat
+ * hits — so a source whose audio has already been read is sampled on its
+ * own clock instead of hoping an even sweep lands on each state. */
+ at?: number[];
}
): Promise<WatchFrames> {
const input = openMedia(sourceUrl);
@@ -1406,21 +1491,25 @@ export async function sampleWatchFrames(
if (!(wanted > from)) throw new Error("from/to describe an empty range.");
const to = Math.min(wanted, from + 600); // bound the work per call; callers resume from coveredTo
const interval = clamp(opts.interval ?? 1, 0.5, 30);
- // The candidate floor: as asked, widened only when the range would blow
- // the per-call decode budget.
- const step = Math.max(interval, (to - from) / CANDIDATE_CAP);
- const times: number[] = [];
- for (let t = from; t < to && times.length < CANDIDATE_CAP; t += step)
- times.push(round2(t));
-
- const [cw, ch] = cellDims(await track.getDisplayWidth(), await track.getDisplayHeight());
- const maxFrames = SHEET_MAX * SHEET_GRID * SHEET_GRID;
+ const times = candidateTimes(from, to, interval, opts.at ?? []);
+
+ const geom = watchGeometry(opts.detail);
+ const srcW = await track.getDisplayWidth();
+ const srcH = await track.getDisplayHeight();
+ const twoPass = !opts.metadataOnly && geom.rewalk;
+ // Selection reads a 192² signature whatever the detail, so a pass that
+ // returns no pictures — or that walks the kept times again for big ones —
+ // decodes candidates at the cheap size.
+ const selectOnly = opts.metadataOnly || twoPass;
+ const [cw, ch] = cellDims(srcW, srcH, selectOnly ? SELECT_CELL : geom.cell);
+ const maxFrames = geom.maxFrames;
// Cell canvases are held only until the selector rules on them (decisions
// run one frame behind), so memory stays near the kept set, never the
// candidate sweep.
const held = new Map<number, RasterSurface>();
const candTimes: number[] = [];
const candG16: Float32Array[] = []; // per-candidate coarse grid, for cut refinement
+ const candE24: Float32Array[] = []; // per-candidate edge energy, for where a card can sit
let keptCount = 0;
const selector = createDedupSelector({
maxFrames,
@@ -1465,7 +1554,7 @@ export async function sampleWatchFrames(
}
const next = await cells.next();
if (next.done || !next.value) continue; // a moment the decoder had no frame for
- if (!opts.metadataOnly) {
+ if (!opts.metadataOnly && !twoPass) {
const copy = createRasterCanvas(cw, ch);
const cctx = copy.getContext("2d") as CanvasRenderingContext2D | null;
if (!cctx) throw new Error("Could not sample the video.");
@@ -1481,8 +1570,12 @@ export async function sampleWatchFrames(
channels: 4 as const,
data: sigCtx.getImageData(0, 0, SIGNATURE_SIZE, SIGNATURE_SIZE).data,
};
- candG16.push(frameSig(rgb).g16);
- selector.push(rgb);
+ // One signature serves both the cut-refinement pass and the
+ // selector; measuring the same frame twice is the whole cost again.
+ const candSig = frameSig(rgb);
+ candG16.push(candSig.g16);
+ candE24.push(candSig.e24);
+ selector.push(candSig);
}
} finally {
await cells.return();
@@ -1492,22 +1585,59 @@ export async function sampleWatchFrames(
// unreadable. Say so — never record an unseen span as watched.
if (candTimes.length === 0 && !paused) throw new Error("Could not sample the video.");
// finish() judges the one pending frame, which can nudge past the cap.
- const keptIdx = selection.kept.slice(0, maxFrames);
+ const keptAll = selection.kept.slice(0, maxFrames);
+ let keptIdx = keptAll;
+ // The big frames: one more ordered walk over the kept times, each encoded
+ // as it arrives so one of them is live at a time. A walk that runs out of
+ // budget delivers what it reached and the kept set shrinks to match, so
+ // coveredTo below still describes frames that exist.
+ const encoded = new Map<number, string>();
+ if (twoPass && keptAll.length > 0) {
+ const [nw, nh] = cellDims(srcW, srcH, geom.cell);
+ const shot = createRasterCanvas(nw, nh);
+ const shotCtx = shot.getContext("2d") as CanvasRenderingContext2D | null;
+ if (!shotCtx) throw new Error("Could not sample the video.");
+ const walk = frameSink(track, { width: nw, height: nh, fit: "fill" }).canvasesAtTimestamps(
+ keptAll.map((i) => candTimes[i])
+ );
+ try {
+ for (const idx of keptAll) {
+ if (outOfTime() || opts.shouldPause?.()) break;
+ const frame = await walk.next();
+ if (frame.done) break;
+ if (!frame.value) continue;
+ shotCtx.drawImage(frame.value.canvas as CanvasImageSource, 0, 0, nw, nh);
+ encoded.set(idx, await rasterCanvasToDataUrl(shot, "image/jpeg", geom.quality));
+ }
+ } finally {
+ await walk.return();
+ }
+ keptIdx = keptAll.filter((idx) => encoded.has(idx));
+ // A walk that reached nothing saw nothing. Saying so sends the caller
+ // somewhere new; an empty pass reported as a success sends it here again.
+ if (keptIdx.length === 0 && !paused) throw new Error("Could not sample the video.");
+ }
const frames: WatchFrames["frames"] = [];
for (const idx of keptIdx) {
- const canvas = held.get(idx);
- if (!opts.metadataOnly && !canvas) continue;
+ const canvas = twoPass ? null : held.get(idx);
+ if (!opts.metadataOnly && !twoPass && !canvas) continue;
const verdict = selection.decisions[idx].verdict;
frames.push({
t: candTimes[idx],
- image: canvas ? await rasterCanvasToDataUrl(canvas, "image/jpeg", SHEET_QUALITY) : "",
+ image: twoPass
+ ? encoded.get(idx)!
+ : canvas
+ ? await rasterCanvasToDataUrl(canvas, "image/jpeg", geom.quality)
+ : "",
via: verdict === "keep-global" ? "global"
: verdict === "keep-action" ? "action"
+ : verdict === "keep-text" ? "text"
: verdict === "keep-settled" ? "settled"
: "first",
});
}
held.clear();
+ encoded.clear();
// A "global" keep only says the cut lies in the grid step before it.
// Bisect that step with a few extra decodes so each reported cut time
@@ -1540,30 +1670,45 @@ export async function sampleWatchFrames(
// An interrupted or capped scan only covered what it decoded and kept;
// the caller merges that much and comes back for the rest.
const lastKeptT = keptIdx.length > 0 ? candTimes[keptIdx[keptIdx.length - 1]] : from;
- const coveredTo = paused
- ? (candTimes[candTimes.length - 1] ?? from)
- : capped
- ? lastKeptT
- : to;
+ // A walk that delivered fewer frames than the selector kept stopped early,
+ // so the range only reaches the last frame that exists.
+ const shortWalk = twoPass && keptIdx.length < keptAll.length;
+ const coveredTo = shortWalk
+ ? lastKeptT
+ : paused
+ ? (candTimes[candTimes.length - 1] ?? from)
+ : capped
+ ? lastKeptT
+ : to;
return {
frames,
candidates: selection.candidateCount,
sceneChanges,
coveredTo,
+ // What each kept frame holds where, already measured for selection: the
+ // busy cells are where the picture's detail and its own type live, so a
+ // card placed over the quiet ones covers nothing.
+ edges: keptIdx.map((i) => candE24[i]).filter((g): g is Float32Array => !!g),
// The per-call span bound is itself truncation — the caller asked for more.
- truncated: paused || capped || to < wanted,
+ truncated: paused || capped || shortWalk || to < wanted,
};
} finally {
input.dispose();
}
}
-/** Tile kept frames into 3×3 contact sheets and stamp each cell's source
- * time. The last sheet may run short; its frames list only the real cells. */
+/** Tile kept frames into contact sheets and stamp each cell's source time, at
+ * the detail's grid. The last sheet may run short; its frames list only the
+ * real cells. At "original" there is no grid and no stamp — the frames go as
+ * they are, because a burned-in label is a graphic the source does not have,
+ * and that detail exists for reading the ones it does. */
export async function composeSheets(
- cells: { t: number; image: string }[]
+ cells: { t: number; image: string }[],
+ detail: WatchDetail = "scan"
): Promise<{ image: string; frames: { t: number }[] }[]> {
if (cells.length === 0) return [];
+ const { grid, quality } = watchGeometry(detail);
+ if (grid <= 1) return cells.map((c) => ({ image: c.image, frames: [{ t: c.t }] }));
const images = await Promise.all(
cells.map(async (c) => {
const img = await decodeRasterImageUrl(c.image);
@@ -1573,9 +1718,9 @@ export async function composeSheets(
);
const cw = images[0].width;
const ch = images[0].height;
- const perSheet = SHEET_GRID * SHEET_GRID;
- const sheetW = 2 * SHEET_GAP + SHEET_GRID * cw + (SHEET_GRID - 1) * SHEET_GAP;
- const sheetH = 2 * SHEET_GAP + SHEET_GRID * ch + (SHEET_GRID - 1) * SHEET_GAP;
+ const perSheet = grid * grid;
+ const sheetW = 2 * SHEET_GAP + grid * cw + (grid - 1) * SHEET_GAP;
+ const sheetH = 2 * SHEET_GAP + grid * ch + (grid - 1) * SHEET_GAP;
const size = Math.max(13, Math.round(ch / 12));
const sheets: { image: string; frames: { t: number }[] }[] = [];
for (let s = 0; s < cells.length; s += perSheet) {
@@ -1589,8 +1734,8 @@ export async function composeSheets(
ctx.font = `bold ${size}px ui-monospace, monospace`;
ctx.textBaseline = "bottom";
batch.forEach((c, j) => {
- const x = SHEET_GAP + (j % SHEET_GRID) * (cw + SHEET_GAP);
- const y = SHEET_GAP + Math.floor(j / SHEET_GRID) * (ch + SHEET_GAP);
+ const x = SHEET_GAP + (j % grid) * (cw + SHEET_GAP);
+ const y = SHEET_GAP + Math.floor(j / grid) * (ch + SHEET_GAP);
ctx.drawImage(images[s + j].source, x, y, cw, ch);
const label = `${round2(c.t)}s`;
const w = ctx.measureText(label).width;
@@ -1600,7 +1745,7 @@ export async function composeSheets(
ctx.fillText(label, x + 9, y + ch - 6);
});
sheets.push({
- image: await rasterCanvasToDataUrl(canvas, "image/jpeg", SHEET_QUALITY),
+ image: await rasterCanvasToDataUrl(canvas, "image/jpeg", quality),
frames: batch.map((c) => ({ t: c.t })),
});
}
diff --git a/site/src/cut/lib/refMedia.ts b/site/src/cut/lib/refMedia.ts
index 5d41ca8d..b32dce1e 100644
--- a/site/src/cut/lib/refMedia.ts
+++ b/site/src/cut/lib/refMedia.ts
@@ -158,7 +158,7 @@ export async function videoSafeInline(img: InlineImage): Promise<InlineImage> {
// Keeps the inline payload well under the Gemini per-request inline-data cap
// (base64 inflates by 4/3); larger audio degrades to a name-only marker.
-const MAX_AUDIO_BYTES = 12 * 1024 * 1024;
+export const MAX_AUDIO_BYTES = 12 * 1024 * 1024;
/** A ref's bytes, naming what failed and where when they don't arrive. A bare
* fetch rejection is the string "Failed to fetch" and nothing else — no asset,
diff --git a/site/src/cut/lib/sampleWatchFrames.test.ts b/site/src/cut/lib/sampleWatchFrames.test.ts
index 59f0fb54..e1297ed2 100644
--- a/site/src/cut/lib/sampleWatchFrames.test.ts
+++ b/site/src/cut/lib/sampleWatchFrames.test.ts
@@ -1,6 +1,6 @@
import { afterEach, expect, spyOn, test } from "bun:test";
import type { Input, InputVideoTrack, WrappedCanvas } from "mediabunny";
-import { sampleWatchFrames } from "./media";
+import { sampleWatchFrames, watchGeometry } from "./media";
import * as mediaRead from "./mediaRead";
import * as raster from "./raster";
@@ -10,6 +10,7 @@ afterEach(() => { for (const restore of restores.splice(0)) restore(); });
function fixture(failPixels = false) {
let closed = 0;
let disposed = false;
+ const asked: number[] = [];
const context = {
drawImage() {},
getImageData(_x: number, _y: number, width: number, height: number) {
@@ -29,13 +30,17 @@ function fixture(failPixels = false) {
getCanvas: async () => frame,
async *canvases() { yield frame; },
async *canvasesAtTimestamps(times) {
- try { for await (const time of times) yield { ...frame, timestamp: time }; }
- finally { closed++; }
+ try {
+ for await (const time of times) {
+ asked.push(time);
+ yield { ...frame, timestamp: time };
+ }
+ } finally { closed++; }
},
}),
];
restores.push(...spies.map((spy) => () => spy.mockRestore()));
- return { closed: () => closed, disposed: () => disposed };
+ return { closed: () => closed, disposed: () => disposed, asked: () => asked };
}
test("pausing a scene scan closes its decoded-frame iterator", async () => {
@@ -56,3 +61,58 @@ test("a failed frame read closes the iterator and its input", async () => {
expect(run.closed()).toBe(1);
expect(run.disposed()).toBe(true);
});
+
+test("a source's own lines aim the candidates", async () => {
+ // What a video shows changes when what it says changes, so each cue start
+ // earns its own decode on top of the steady sweep.
+ const run = fixture();
+ const result = await sampleWatchFrames("fixture.mp4", {
+ from: 0, to: 10, interval: 2, metadataOnly: true,
+ at: [1.2, 3.4, 3.45, 7.8],
+ });
+ // The floor (every 2s) with each line woven into it in order — and 3.45,
+ // landing within a frame of 3.4, folded into the one decode.
+ expect(run.asked()).toEqual([0, 1.2, 2, 3.4, 4, 6, 7.8, 8]);
+ expect(result.candidates).toBe(8);
+});
+
+test("a densely spoken source is aimed end to end", async () => {
+ // More lines than the budget allows: they thin across the whole span. Taking
+ // the first of them would aim the head and leave the tail sampled thinner
+ // than the floor alone would have covered it.
+ const run = fixture();
+ const at = Array.from({ length: 300 }, (_, i) => i * 0.4);
+ await sampleWatchFrames("fixture.mp4", {
+ from: 0, to: 120, interval: 0.5, metadataOnly: true, at,
+ });
+ const asked = run.asked();
+ expect(asked.length).toBeLessThanOrEqual(150);
+ // The aiming reaches the last quarter, and the floor still reaches the end.
+ const aimed = asked.filter((t) => Math.abs(t / 0.4 - Math.round(t / 0.4)) < 1e-6 && t % 1 !== 0);
+ expect(Math.max(...aimed)).toBeGreaterThan(90);
+ expect(Math.max(...asked)).toBeGreaterThan(115);
+ // And every second still sits between two candidates at the floor's density.
+ const gaps = asked.slice(1).map((t, i) => t - asked[i]);
+ expect(Math.max(...gaps)).toBeLessThanOrEqual(1.7);
+});
+
+test("the detail ladder trades frames for pixels", () => {
+ // Coverage is many small cells; matching a look is a handful of big frames.
+ // Every step is the same moments with more pixels spent on each.
+ expect(watchGeometry("scan").maxFrames).toBe(36);
+ expect(watchGeometry("read").maxFrames).toBe(16);
+ expect(watchGeometry("original").maxFrames).toBe(6);
+ expect(watchGeometry("scan").cell).toBeLessThan(watchGeometry("read").cell);
+ expect(watchGeometry("read").cell).toBeLessThan(watchGeometry("original").cell);
+ expect(watchGeometry().maxFrames).toBe(watchGeometry("scan").maxFrames);
+});
+
+test("the top of the ladder is bounded, and only it walks twice", () => {
+ // Six frames of a 4K source would be several times the whole per-call media
+ // budget, so the source's own pixels stop at a ceiling; a phone or 1080p
+ // source sits under it and passes through untouched.
+ expect(watchGeometry("original").cell).toBe(1920);
+ expect(watchGeometry("original").rewalk).toBe(true);
+ expect(watchGeometry("scan").rewalk).toBe(false);
+ expect(watchGeometry("read").rewalk).toBe(false);
+});
diff --git a/site/src/cut/lib/textPlace.test.ts b/site/src/cut/lib/textPlace.test.ts
new file mode 100644
index 00000000..9abd0688
--- /dev/null
+++ b/site/src/cut/lib/textPlace.test.ts
@@ -0,0 +1,58 @@
+import { describe, expect, test } from "bun:test";
+
+import { textSpots } from "./textPlace";
+import { TEXT_GRID } from "./watch/signatures";
+
+const FULL = { x: 0, y: 0, w: 1, h: 1 };
+
+/** A frame whose edge energy sits only in the rows between `from` and `to`,
+ * as fractions of the frame's height — a subject, a sign, a caption track. */
+const busyRows = (from: number, to: number, energy = 60): Float32Array => {
+ const g = new Float32Array(TEXT_GRID * TEXT_GRID);
+ for (let row = Math.round(from * TEXT_GRID); row < Math.round(to * TEXT_GRID); row++)
+ for (let col = 0; col < TEXT_GRID; col++) g[row * TEXT_GRID + col] = energy;
+ return g;
+};
+
+describe("textSpots", () => {
+ test("a card goes where the picture is quiet", () => {
+ // Everything happens in the top third: a head and shoulders high in frame.
+ const [best] = textSpots([busyRows(0, 1 / 3)], FULL);
+ expect(best.y).toBeGreaterThanOrEqual(0.33);
+ expect(best.clear).toBeGreaterThan(0.9);
+ });
+
+ test("footage with its own captions leaves that band alone", () => {
+ // The source burns its own text across the lower third.
+ const spots = textSpots([busyRows(2 / 3, 1)], FULL, 5);
+ const lower = spots.find((s) => s.where.includes("lower third"))!;
+ const top = spots.find((s) => s.where.includes("top third"))!;
+ expect(top.clear).toBeGreaterThan(lower.clear);
+ expect(spots[0].where).not.toContain("lower third");
+ });
+
+ test("a band is only as clear as the busiest frame of the stretch", () => {
+ // Quiet at the bottom for most of the stretch, and busy there in one
+ // frame. A card sits still while the picture moves, so that one frame
+ // decides it.
+ const spots = textSpots([busyRows(0, 1 / 3), busyRows(0, 1 / 3), busyRows(2 / 3, 1)], FULL, 5);
+ const lower = spots.find((s) => s.where.includes("lower third"))!;
+ expect(lower.clear).toBeLessThan(0.5);
+ });
+
+ test("every spot stays inside the safe area", () => {
+ // A short-form frame keeps the phone's UI clear; nothing may reach into it.
+ const safe = { x: 0.06, y: 0.12, w: 0.88, h: 0.66 };
+ for (const spot of textSpots([busyRows(0, 0.2)], safe, 5)) {
+ expect(spot.x).toBeGreaterThanOrEqual(safe.x);
+ expect(spot.y).toBeGreaterThanOrEqual(safe.y - 0.001);
+ expect(spot.x + spot.w).toBeLessThanOrEqual(safe.x + safe.w + 0.001);
+ expect(spot.y + spot.h).toBeLessThanOrEqual(safe.y + safe.h + 0.001);
+ }
+ });
+
+ test("with no frames measured the safe area is the whole answer", () => {
+ const safe = { x: 0.1, y: 0.1, w: 0.8, h: 0.8 };
+ expect(textSpots([], safe)).toEqual([{ ...safe, where: "anywhere inside the safe area", clear: 1 }]);
+ });
+});
diff --git a/site/src/cut/lib/textPlace.ts b/site/src/cut/lib/textPlace.ts
new file mode 100644
index 00000000..034967f2
--- /dev/null
+++ b/site/src/cut/lib/textPlace.ts
@@ -0,0 +1,104 @@
+import { TEXT_GRID } from "./watch/signatures";
+import type { SafeArea } from "./guides";
+
+/** Where a card can sit over a stretch of footage without covering anything.
+ *
+ * A watch already measures, for every frame it keeps, how much hard edge each
+ * cell of the picture holds — that is how it tells a caption landing from a
+ * mouth moving. The same numbers answer a different question: a face, a hand,
+ * a sign and the source's own burned-in type are all edges, and the parts of
+ * the frame that stay quiet across every frame of a stretch are the parts a
+ * title can take without hiding any of it.
+ *
+ * The answer is arithmetic over numbers the watch already has, so it costs
+ * nothing extra and it holds wherever the watch runs. Placement is judged
+ * against the WORST frame of the stretch, because a card sits still while the
+ * picture moves under it.
+ */
+
+export interface TextSpot {
+ /** Where it sits in the frame, as fractions of width and height. */
+ x: number;
+ y: number;
+ w: number;
+ h: number;
+ /** Where the spot sits, in the words someone would use for it. */
+ where: string;
+ /** How quiet it stays across the whole stretch, 1 quietest. */
+ clear: number;
+}
+
+/** The bands a card is actually placed in: a full-width strip, because text
+ * runs across. Vertical thirds of the safe area, then its halves, so a frame
+ * whose subject sits high still offers the lower half whole. */
+const BANDS: { top: number; height: number; where: string }[] = [
+ { top: 0, height: 1 / 3, where: "across the top third" },
+ { top: 1 / 3, height: 1 / 3, where: "across the middle" },
+ { top: 2 / 3, height: 1 / 3, where: "across the lower third" },
+ { top: 0, height: 0.5, where: "over the top half" },
+ { top: 0.5, height: 0.5, where: "over the bottom half" },
+];
+
+const round3 = (n: number) => Math.round(n * 1000) / 1000;
+
+/** The mean edge energy inside a rect of the frame, for one frame's grid.
+ * Cells are weighted by how much of them the rect covers, so a band edge
+ * falling inside a cell does not count the whole cell. */
+function energyIn(grid: Float32Array, r: { x: number; y: number; w: number; h: number }): number {
+ const g = TEXT_GRID;
+ let sum = 0;
+ let weight = 0;
+ for (let row = 0; row < g; row++) {
+ const cy0 = row / g;
+ const overlapY = Math.min(cy0 + 1 / g, r.y + r.h) - Math.max(cy0, r.y);
+ if (overlapY <= 0) continue;
+ for (let col = 0; col < g; col++) {
+ const cx0 = col / g;
+ const overlapX = Math.min(cx0 + 1 / g, r.x + r.w) - Math.max(cx0, r.x);
+ if (overlapX <= 0) continue;
+ const w = overlapX * overlapY;
+ sum += grid[row * g + col] * w;
+ weight += w;
+ }
+ }
+ return weight > 0 ? sum / weight : 0;
+}
+
+/** The spots a card can take over this stretch, quietest first.
+ *
+ * `safe` bounds every spot, so a platform's own UI is already out of the way
+ * before the footage is consulted. With no frames measured there is nothing to
+ * judge, and the safe area itself comes back as the only answer. */
+export function textSpots(edges: Float32Array[], safe: SafeArea, limit = 3): TextSpot[] {
+ const whole = { x: safe.x, y: safe.y, w: safe.w, h: safe.h };
+ if (edges.length === 0)
+ return [{ ...whole, where: "anywhere inside the safe area", clear: 1 }];
+ const rects = BANDS.map((b) => ({
+ where: b.where,
+ x: safe.x,
+ w: safe.w,
+ y: safe.y + b.top * safe.h,
+ h: b.height * safe.h,
+ }));
+ // A card sits still while the picture moves, so a band is only as clear as
+ // its busiest frame.
+ const scored = rects.map((r) => ({
+ rect: r,
+ worst: Math.max(...edges.map((g) => energyIn(g, r))),
+ }));
+ // Clearness is relative to the busiest band of this footage: a talking head
+ // and a title card hold different amounts of edge overall, and what matters
+ // is which part of THIS picture is free.
+ const loudest = Math.max(...scored.map((s) => s.worst), 1e-6);
+ return scored
+ .map((s) => ({
+ x: round3(s.rect.x),
+ y: round3(s.rect.y),
+ w: round3(s.rect.w),
+ h: round3(s.rect.h),
+ where: s.rect.where,
+ clear: round3(1 - s.worst / loudest),
+ }))
+ .sort((a, b) => b.clear - a.clear)
+ .slice(0, Math.max(1, limit));
+}
diff --git a/site/src/cut/lib/turnQuality.test.ts b/site/src/cut/lib/turnQuality.test.ts
index 7f78cca7..6f24729e 100644
--- a/site/src/cut/lib/turnQuality.test.ts
+++ b/site/src/cut/lib/turnQuality.test.ts
@@ -16,10 +16,17 @@ const settings = SETTINGS.cutJudge.default;
const source = (over: Partial<WatchedSource> = {}): WatchedSource => ({
name: "reference.mp4",
duration: 71.6,
+ kind: "video",
+ sound: "yes",
passes: 1,
coveredTo: 20,
unwatched: 51.6,
unnoted: [{ from: 20, to: 71.6 }],
+ unread: 71.6,
+ unreadFrom: 0,
+ unheard: 0,
+ unheardFrom: null,
+ spoken: "",
observed: [{ from: 0, to: 20, text: "hook titles over a talking head" }],
cuts: 12,
...over });
@@ -37,6 +44,8 @@ function answers(over: {
finished?: number;
seen?: number;
honest?: number;
+ hears?: number;
+ captionsSpeak?: number;
closeness?: "shape" | "normal" | "exact";
}): QualityAnswers {
const closeness = over.closeness ?? "normal";
@@ -44,6 +53,8 @@ function answers(over: {
finished: { type: "noul", noul: over.finished ?? 0.9 },
seen: { type: "noul", noul: over.seen ?? 0.9 },
honest: { type: "noul", noul: over.honest ?? 0.9 },
+ hears: { type: "noul", noul: over.hears ?? 0.1 },
+ captionsSpeak: { type: "noul", noul: over.captionsSpeak ?? 0.1 },
closeness: {
type: "choice",
choice: closeness,
@@ -85,6 +96,173 @@ describe("qualityVerdict", () => {
expect(verdict?.step).toBe("watch");
});
+ test("an ask about how it sounds holds on seconds nobody played", () => {
+ const unplayed = work({
+ request: "cut this to the beat",
+ sources: [source({ coveredTo: 71.6, unwatched: 0, unnoted: [], unread: 0, unheard: 71.6, unheardFrom: 0 })] });
+ const verdict = qualityVerdict(answers({ honest: 0.1, hears: 0.9 }), unplayed, settings);
+ expect(verdict?.step).toBe("watch");
+ expect(verdict?.steer).toContain("listen_audio from=0");
+ expect(verdict?.steer).toContain("71.6s");
+ });
+
+ test("a picture ask never holds on unplayed seconds", () => {
+ const unplayed = work({
+ sources: [source({ coveredTo: 71.6, unwatched: 0, unnoted: [], unread: 0, unheard: 71.6, unheardFrom: 0 })] });
+ expect(qualityVerdict(answers({ honest: 0.1, hears: 0.1 }), unplayed, settings)).toBeNull();
+ });
+
+ test("a listen records what the source has left unheard", () => {
+ const looks = new Map<string, WatchedSource>();
+ recordLook(looks, "listen_audio", {
+ source: { assetId: "a1", name: "track.mp3", duration: 200 },
+ from: 0, coveredTo: 120, unheardSeconds: 80, unheardFrom: 120 });
+ expect(looks.get("a1")!.unheard).toBe(80);
+ expect(looks.get("a1")!.unheardFrom).toBe(120);
+ });
+
+ test("an exact ask holds on footage surveyed but never read", () => {
+ // Every second has been looked at — as contact-sheet thumbnails, where a
+ // caption is a few pixels tall. For an ask to reproduce the source, that
+ // is a measured gap, the same as seconds nobody opened.
+ const surveyed = work({ sources: [source({ coveredTo: 71.6, unwatched: 0, unnoted: [] })] });
+ const verdict = qualityVerdict(answers({ honest: 0.1, closeness: "exact" }), surveyed, settings);
+ expect(verdict?.step).toBe("watch");
+ expect(verdict?.steer).toContain("detail=original");
+ expect(verdict?.steer).toContain("71.6s");
+ // The instruction names where to start, so following it ends the hold.
+ expect(verdict?.steer).toContain("from=0");
+ });
+
+ test("a caption track that speaks is read for treatment, not for words", () => {
+ // The text on screen is the narration, so the transcript already holds
+ // every word of it. The close read that remains is about how they are
+ // set, and the steer says so instead of asking for all 71.6s back.
+ const surveyed = work({
+ sources: [source({ coveredTo: 71.6, unwatched: 0, unnoted: [], spoken: '[0s-2s] "having to"' })],
+ });
+ const verdict = qualityVerdict(
+ answers({ honest: 0.1, closeness: "exact", captionsSpeak: 0.9 }),
+ surveyed,
+ settings,
+ );
+ expect(verdict?.step).toBe("watch");
+ expect(verdict?.steer).toContain("detail=original");
+ expect(verdict?.steer).toContain("the transcript already holds the words");
+ });
+
+ test("a sample of a spoken caption track is enough to close an exact turn", () => {
+ // Treatment reads as a sample, not as coverage: once enough of it has
+ // been read closely, the rest comes off the transcript.
+ const sampled = work({
+ sources: [
+ source({
+ coveredTo: 71.6,
+ unwatched: 0,
+ unnoted: [],
+ unread: 71.6 - settings.qualityTreatmentSeconds,
+ unreadFrom: 10,
+ }),
+ ],
+ });
+ expect(
+ qualityVerdict(
+ answers({ honest: 0.1, closeness: "exact", captionsSpeak: 0.9 }),
+ sampled,
+ settings,
+ ),
+ ).toBeNull();
+ // Text of its own still owes the full read-through.
+ expect(
+ qualityVerdict(
+ answers({ honest: 0.1, closeness: "exact", captionsSpeak: 0.1 }),
+ sampled,
+ settings,
+ )?.steer,
+ ).toContain("until unreadSeconds is 0");
+ });
+
+ test("a source read closely end to end closes an exact turn", () => {
+ // The measure rides the asset, so a source read in an earlier turn is not
+ // charged for again.
+ const read = work({
+ sources: [
+ source({ coveredTo: 71.6, unwatched: 0, unnoted: [], unread: 0, unreadFrom: null }),
+ ] });
+ expect(qualityVerdict(answers({ honest: 0.1, closeness: "exact" }), read, settings)).toBeNull();
+ });
+
+ test("a normal ask never holds on detail", () => {
+ const surveyed = work({ sources: [source({ coveredTo: 71.6, unwatched: 0, unnoted: [] })] });
+ expect(qualityVerdict(answers({ honest: 0.1, closeness: "normal" }), surveyed, settings)).toBeNull();
+ });
+
+ test("a steer never names a tool the source refuses", () => {
+ // The music the turn listened to has no picture. Steering watch_video at
+ // it spends the round on "that is audio — listen_audio hears it".
+ const music = work({
+ request: "cut this to the beat",
+ sources: [
+ source({
+ name: "track.mp3",
+ kind: "audio",
+ sound: "yes",
+ duration: 200,
+ coveredTo: 0,
+ unwatched: 200,
+ unread: 200,
+ unreadFrom: 0,
+ unheard: 0,
+ unheardFrom: null,
+ unnoted: [],
+ }),
+ ],
+ });
+ expect(qualityVerdict(answers({ finished: 0.1, closeness: "exact" }), music, settings)).toBeNull();
+ });
+
+ test("silent footage is never sent to be listened to", () => {
+ // A generated b-roll clip with no audio track: listen_audio throws on it,
+ // so an ask about the sound looks elsewhere.
+ const silent = work({
+ request: "cut this to the music",
+ sources: [
+ source({
+ kind: "video",
+ sound: "none",
+ coveredTo: 71.6,
+ unwatched: 0,
+ unnoted: [],
+ unread: 0,
+ unreadFrom: null,
+ unheard: 71.6,
+ unheardFrom: 0,
+ }),
+ ],
+ });
+ expect(qualityVerdict(answers({ honest: 0.1, hears: 0.9 }), silent, settings)).toBeNull();
+ });
+
+ test("a gap too small to reach closes the turn", () => {
+ // 98.6s of a 100s track played, the rest a sliver the coverage maths does
+ // not report. With nowhere to send the next pass, a hold would replay the
+ // whole source for seconds nobody can reach.
+ const sliver = work({
+ sources: [
+ source({
+ coveredTo: 71.6,
+ unwatched: 0,
+ unnoted: [],
+ unread: 1.4,
+ unreadFrom: null,
+ unheard: 1.4,
+ unheardFrom: null,
+ }),
+ ],
+ });
+ expect(qualityVerdict(answers({ honest: 0.1, hears: 0.9, closeness: "exact" }), sliver, settings)).toBeNull();
+ });
+
test("a doubt with nothing measurable behind it lets the turn close", () => {
// The source was watched through and every call landed: the judgment has
// no gap to point at, and a turn sent back here would build twice.
@@ -139,6 +317,18 @@ describe("qualityState", () => {
});
describe("recordLook", () => {
+ test("a pass carries the source's own unread measure, not this turn's", () => {
+ const looks = new Map<string, WatchedSource>();
+ const src = { assetId: "a1", name: "reference.mp4", duration: 71.6 };
+ recordLook(looks, "watch_video", {
+ source: src, from: 0, coveredTo: 24, detail: "scan", unreadSeconds: 71.6, unreadFrom: 0 });
+ expect(looks.get("a1")!.unread).toBe(71.6);
+ recordLook(looks, "watch_video", {
+ source: src, from: 0, coveredTo: 12, detail: "original", unreadSeconds: 59.6, unreadFrom: 12 });
+ expect(looks.get("a1")!.unread).toBe(59.6);
+ expect(looks.get("a1")!.unreadFrom).toBe(12);
+ });
+
test("a watch pass records coverage, cuts and the notes already written", () => {
const looks = new Map<string, WatchedSource>();
recordLook(looks, "watch_video", {
@@ -189,6 +379,8 @@ describe("QUALITY_QUESTIONS", () => {
"finished",
"seen",
"honest",
+ "hears",
+ "captionsSpeak",
"closeness",
]);
});
diff --git a/site/src/cut/lib/turnQuality.ts b/site/src/cut/lib/turnQuality.ts
index 57ebd75b..759db55b 100644
--- a/site/src/cut/lib/turnQuality.ts
+++ b/site/src/cut/lib/turnQuality.ts
@@ -25,11 +25,34 @@ export type CutJudgeSettings = Settings["cutJudge"];
export interface WatchedSource {
name: string;
duration: number;
+ /** What the source is, and whether it has anything to listen to. A steer
+ * that names a tool the source refuses spends a round on an error, so the
+ * holds below read these before they pick one. */
+ kind: "video" | "audio" | "image";
+ sound: "yes" | "none" | "unknown";
/** Watch passes this turn. */
passes: number;
/** How far coverage reaches, and how much of the source nobody has seen. */
coveredTo: number;
unwatched: number;
+ /** Seconds no pass has read at a detail that carries type, and where the
+ * next close pass would start. Surveying a source at "scan" says where its
+ * cuts fall; it does not say what the words on screen are, and an exact ask
+ * turns on the words. Both persist on the asset, so a source read closely in
+ * an earlier turn is not charged for again. */
+ unread: number;
+ unreadFrom: number | null;
+ /** Seconds nobody has played, and where the next listen would start. A
+ * transcript holds a source's words; only listening holds its tone, its
+ * music and its timing, and an ask about how something sounds turns on
+ * those. Persisted on the asset alongside the rest. */
+ unheard: number;
+ unheardFrom: number | null;
+ /** What the source says over the stretch just watched, on its own clock —
+ * the fused frame↔speech timeline the last pass returned. It is what the
+ * on-screen text is checked against: a caption track that repeats the
+ * narration has its words in the transcript already. */
+ spoken: string;
/** Source spans nothing is written about yet. */
unnoted: { from: number; to: number }[];
/** The written record: what someone looked at and wrote down. */
@@ -56,6 +79,7 @@ export interface TurnWork {
const TEXT_CAP = 2000;
const NOTE_CAP = 600;
const NOTES_PER_SOURCE = 12;
+const SPOKEN_CAP = 1500;
/** The state the judge reads. No urls, no media, no frames — the record in
* fields, so the same judgment holds in the page and in the worker. */
@@ -68,11 +92,23 @@ export function qualityState(work: TurnWork): Entry {
sources: work.sources.map((s) => ({
name: s.name,
duration: s.duration,
+ kind: s.kind,
+ hasSound: s.sound,
passesThisTurn: s.passes,
- seenThrough: s.coveredTo,
- secondsNeverSeen: s.unwatched,
spansWithNoNote: s.unnoted.slice(0, 8),
- cutsFound: s.cuts,
+ spokenOverIt: s.spoken.slice(0, SPOKEN_CAP),
+ // Only the measures this source can answer. A music asset reporting
+ // seconds of unseen picture reads as a gap, and the judge would be
+ // right to call the turn on it.
+ ...(s.kind === "video"
+ ? {
+ seenThrough: s.coveredTo,
+ secondsNeverSeen: s.unwatched,
+ secondsSeenOnlyAsThumbnails: s.unread,
+ cutsFound: s.cuts,
+ }
+ : {}),
+ ...(s.sound === "none" ? {} : { secondsNeverHeard: s.unheard }),
observed: s.observed
.slice(0, NOTES_PER_SOURCE)
.map((n) => ({ from: n.from, to: n.to, text: n.text.slice(0, NOTE_CAP) })),
@@ -91,11 +127,11 @@ export const QUALITY_QUESTIONS = {
},
),
seen: noul(
- "Has enough of the footage been watched AND written down for `request`? `secondsNeverSeen` is source nobody has looked at; `observed` is the written record.",
+ "Has enough of the source been taken in AND written down for `request`? `secondsNeverSeen` is source nobody has looked at, `secondsNeverHeard` is source nobody has played; `observed` is the written record.",
{
- true: "The record covers what the ask depends on: an ask about one moment needs that moment, an ask about the whole source needs the whole source seen and noted.",
+ true: "The record covers what the ask depends on: an ask about one moment needs that moment, an ask about the whole source needs the whole source taken in and noted, and an ask that turns on the picture needs only the picture.",
false:
- "The ask leans on footage nobody has looked at — describing, summarizing, reproducing or rebuilding a source with seconds never seen, or with looked-at stretches nothing is written about.",
+ "The ask leans on a source nobody has taken in — describing, summarizing, reproducing or rebuilding it with seconds never seen, judging how it sounds, paces or is scored with seconds never played, or with stretches nothing is written about.",
},
),
honest: noul(
@@ -106,6 +142,22 @@ export const QUALITY_QUESTIONS = {
"The reply describes a source as a whole from a slice of it, quotes on-screen text nothing recorded, or reports work no call in `ran` performed.",
},
),
+ hears: noul(
+ "Does answering `request` turn on how the source SOUNDS — its tone, delivery, music, pacing or sound design — rather than only on what it shows or says?",
+ {
+ true: "The ask is about the sound: matching a track's energy, cutting to music, judging a read or a mix, describing the scoring, placing a beat, telling whether a take lands.",
+ false:
+ "The ask turns on the picture, the words, or the document: what is on screen, what the transcript says, where the clips sit, how a title looks.",
+ },
+ ),
+ captionsSpeak: noul(
+ "In `sources`, is the on-screen text `observed` quotes the source's own narration — the same words `spokenOverIt` carries, on screen as they are said?",
+ {
+ true: "The quoted on-screen words and the spoken words over those times are the same words: a caption track of the narration, burned-in subtitles, a karaoke line.",
+ false:
+ "The on-screen text says something the speech does not — a title, a name card, a label, a written line of its own — or there is no speech over it, or nothing on screen has been written down yet.",
+ },
+ ),
closeness: choice(
"How closely does `request` need the footage read?",
{
@@ -121,6 +173,8 @@ export type QualityAnswers = {
finished: NoulAnswer;
seen: NoulAnswer;
honest: NoulAnswer;
+ hears: NoulAnswer;
+ captionsSpeak: NoulAnswer;
closeness: ChoiceAnswer<Record<"shape" | "normal" | "exact", string>>;
};
@@ -144,6 +198,27 @@ function intervalFor(closeness: QualityAnswers["closeness"]["choice"]): number {
const round2 = (n: number) => Math.round(n * 100) / 100;
+/** A hold names the next pass, so a source only earns one when the record can
+ * say where that pass starts. A gap too small for the coverage maths to
+ * report leaves nowhere to go, and a steer built on it sends the whole source
+ * through again for a sliver nobody can reach. */
+
+/** The source an exact ask has surveyed the most of without reading it. */
+function leastRead(sources: WatchedSource[]): WatchedSource | null {
+ let worst: WatchedSource | null = null;
+ for (const s of sources)
+ if (s.kind === "video" && s.unreadFrom !== null && (!worst || s.unread > worst.unread)) worst = s;
+ return worst && worst.unread > 0.5 ? worst : null;
+}
+
+/** The source with the most of it never played. */
+function leastHeard(sources: WatchedSource[]): WatchedSource | null {
+ let worst: WatchedSource | null = null;
+ for (const s of sources)
+ if (s.sound === "yes" && s.unheardFrom !== null && (!worst || s.unheard > worst.unheard)) worst = s;
+ return worst && worst.unheard > 0.5 ? worst : null;
+}
+
/** What the turn has already put in the editor, named in the steer. A turn
* sent back to look does not start the build over: told nothing, it lays the
* same shots down again and the cut comes out as long as the number of passes
@@ -162,7 +237,8 @@ function standing(work: TurnWork): string {
/** The source with the most left to see, which is where the next pass goes. */
function thinnest(sources: WatchedSource[]): WatchedSource | null {
let worst: WatchedSource | null = null;
- for (const s of sources) if (!worst || s.unwatched > worst.unwatched) worst = s;
+ for (const s of sources)
+ if (s.kind === "video" && (!worst || s.unwatched > worst.unwatched)) worst = s;
return worst && worst.unwatched > 0.5 ? worst : null;
}
@@ -172,9 +248,10 @@ function thinnest(sources: WatchedSource[]): WatchedSource | null {
*
* The judgment opens the question and the record answers it: a doubt about
* the work, the looking or the reply only holds the turn when something
- * measured stands behind it — source seconds nobody has looked at, or calls
- * that did not land. The instruction is built from that state, so it carries
- * real numbers: which source, from where, at what floor.
+ * measured stands behind it — source seconds nobody has looked at, seconds an
+ * exact ask has surveyed but never read at a legible size, or calls that did
+ * not land. The instruction is built from that state, so it carries real
+ * numbers: which source, from where, at what floor, at what detail.
*/
export function qualityVerdict(
answers: QualityAnswers | null,
@@ -211,6 +288,55 @@ export function qualityVerdict(
standing(work),
};
}
+ // An ask that turns on how a source SOUNDS has the same measure behind it:
+ // seconds nobody played. The judge decides whether the ask leans on the
+ // sound; the record decides whether there is anything left to hear.
+ const deaf = leastHeard(work.sources);
+ if (deaf && answers.hears.noul >= settings.qualityHears)
+ return {
+ step: "watch",
+ steer:
+ `${QUALITY_STEER_PREFIX} Not yet. ${round2(deaf.unheard)}s of "${deaf.name}" has never been played. ` +
+ `This ask turns on how it sounds: listen_audio from=${round2(deaf.unheardFrom!)}, note_source what you heard — ` +
+ "the tone, the music, where the beats and the pauses fall — and keep going from each result's unheardFrom until unheardSeconds is 0." +
+ standing(work),
+ };
+
+ // An ask for the source reproduced exactly turns on what its frames SAY.
+ // A stretch surveyed at thumbnail size has been looked at and not read, so
+ // the hold stands on seconds the record measures, the same as an unwatched
+ // span does — and the instruction names the next span, so following it ends.
+ if (answers.closeness.choice === "exact") {
+ const coarse = leastRead(work.sources);
+ // Sometimes the frames say what the source says. A caption track carrying
+ // its own narration has every one of its words in the transcript already,
+ // and reading them back off the pixels second by second buys nothing the
+ // record does not hold — what is unknown is how they are SET. That reads
+ // as a sample, not as coverage, and it ends after a few seconds of it.
+ const spoken = answers.captionsSpeak.noul >= settings.qualityCaptionsSpeak;
+ const readSoFar = coarse ? Math.max(0, coarse.duration - coarse.unread) : 0;
+ if (coarse && spoken && readSoFar < settings.qualityTreatmentSeconds)
+ return {
+ step: "watch",
+ steer:
+ `${QUALITY_STEER_PREFIX} Not yet. The text on "${coarse.name}" is the source speaking, so the transcript already holds the words — ` +
+ "what it does not hold is how they are set. Read a sample of them at full size: " +
+ `watch_video from=${round2(coarse.unreadFrom!)} interval_seconds=0.5 detail=original, note_source the face, weight, size, colour, placement, ` +
+ "how many words ride one card and how a card enters and leaves. Then build the rest from the transcript on those times." +
+ standing(work),
+ };
+ if (coarse && !spoken)
+ return {
+ step: "watch",
+ steer:
+ `${QUALITY_STEER_PREFIX} Not yet. "${coarse.name}" has ${round2(coarse.unread)}s of its ${round2(coarse.duration)}s seen only as contact-sheet thumbnails, ` +
+ "where a caption is a few pixels tall and its face, weight and spacing are gone. This ask needs the frames themselves: " +
+ `watch_video from=${round2(coarse.unreadFrom!)} interval_seconds=0.5 detail=original, note_source the words exactly as they read, ` +
+ "and keep going from each result's unreadFrom until unreadSeconds is 0." +
+ standing(work),
+ };
+ }
+
if (work.failed.length > 0) {
return {
step: "repair",
@@ -259,26 +385,52 @@ export function recordLook(
tool: string,
response: unknown,
): void {
- if (tool !== "watch_video" && tool !== "note_source") return;
+ if (tool !== "watch_video" && tool !== "note_source" && tool !== "listen_audio") return;
if (!isRec(response)) return;
const src = isRec(response.source) ? response.source : null;
const id = src && typeof src.assetId === "string" ? src.assetId : null;
if (!src || !id) return;
+ const kind = src.type === "audio" || src.type === "image" ? src.type : "video";
+ const sound = src.sound === "yes" || src.sound === "none" ? src.sound : "unknown";
const cur: WatchedSource = sources.get(id) ?? {
name: typeof src.name === "string" ? src.name : "this source",
duration: numOf(src.duration),
+ kind,
+ sound,
passes: 0,
coveredTo: 0,
unwatched: numOf(src.duration),
+ unread: numOf(src.duration),
+ unreadFrom: 0,
+ unheard: numOf(src.duration),
+ unheardFrom: 0,
+ spoken: "",
unnoted: [],
observed: [],
cuts: 0,
};
+ // The source says what it is on every result, so a record opened by one
+ // sense learns what the others can ask of it.
+ cur.kind = kind;
+ cur.sound = sound;
+ if (tool === "listen_audio") {
+ cur.passes++;
+ cur.unheard = numOf(response.unheardSeconds, cur.unheard);
+ cur.unheardFrom = typeof response.unheardFrom === "number" ? response.unheardFrom : null;
+ sources.set(id, cur);
+ return;
+ }
if (tool === "watch_video") {
cur.passes++;
cur.coveredTo = Math.max(cur.coveredTo, numOf(response.coveredTo, cur.coveredTo));
cur.unwatched = numOf(response.unwatchedSeconds, cur.unwatched);
cur.cuts += Array.isArray(response.sceneChanges) ? response.sceneChanges.length : 0;
+ // Both measures come off the asset's own record, so they carry what every
+ // earlier turn read as well as this one.
+ cur.unread = numOf(response.unreadSeconds, cur.unread);
+ cur.unreadFrom = typeof response.unreadFrom === "number" ? response.unreadFrom : null;
+ if (typeof response.timeline === "string" && response.timeline)
+ cur.spoken = response.timeline;
const recorded = notesOf(response.recorded);
// `recorded` is what was already written about the span just watched;
// note_source's own result carries the whole record and wins over it.
diff --git a/site/src/cut/lib/types.ts b/site/src/cut/lib/types.ts
index 4ba16e1c..9142dcee 100644
--- a/site/src/cut/lib/types.ts
+++ b/site/src/cut/lib/types.ts
@@ -133,11 +133,26 @@ export function nearestAspect<T extends string>(aspect: Aspect, supported: reado
/** Asset fields persisted in project.json. */
/** Why watch_video kept a frame: "first" opens the range, "global" is a
* whole-frame change (a hard cut), "action" is hard local motion (a small
- * subject), "settled" is new settled detail (text, ink, UI). */
-export type WatchKeepReason = "first" | "global" | "action" | "settled";
+ * subject), "text" is type landing, swapping or leaving, "settled" is other
+ * new settled detail (ink, UI). */
+export type WatchKeepReason = "first" | "global" | "action" | "text" | "settled";
+
+/** How much of each kept frame watch_video hands back, from the tiled
+ * thumbnails that answer "what happens here" up to the frames themselves.
+ * Every step trades frames for pixels at roughly even cost: the geometry is
+ * in media.ts, the notes below are what the tool catalog and the schema both
+ * describe it with. */
+export const WATCH_DETAILS = ["scan", "read", "original"] as const;
+export type WatchDetail = (typeof WATCH_DETAILS)[number];
+export const WATCH_DETAIL_NOTES: Record<WatchDetail, string> = {
+ scan: "36 frames as 3×3 contact sheets — coverage: what happens, where the cuts fall, how long a stretch runs",
+ read: "16 frames as 2×2 sheets at half again the cell size — on-screen text, UI, anything whose words matter",
+ original: "6 frames at the source's own resolution, untiled and unstamped — type, weight, spacing, colour and edges exactly as the source has them",
+};
-/** What the assistant has seen of a source: watch_video's kept frames and
- * detected cuts, merged across the watched spans. Times are source seconds.
+/** What the assistant has taken in from a source: watch_video's kept frames
+ * and detected cuts, the spans it read closely or listened to, and what it
+ * wrote down, merged across the spans. Times are source seconds.
* Media files are immutable per fileName, so this never goes stale; it lives
* on the asset, so it saves with the project and dies with the asset. */
export interface AssetWatch {
@@ -147,6 +162,15 @@ export interface AssetWatch {
frames: { t: number; via: WatchKeepReason }[];
/** Hard-cut moments among the kept frames. */
sceneChanges: number[];
+ /** Spans looked at closely enough to read type — the detail ladder's `read`
+ * and `original`. A scan says where the cuts fall; only these say what the
+ * words on screen are, and an ask to reproduce a source turns on the words.
+ * Absent on records written before the ladder: nothing was read closely. */
+ read?: { from: number; to: number }[];
+ /** Spans played to the assistant by listen_audio. A source's transcript
+ * holds its words; this holds where anyone heard it — the tone, the music,
+ * the timing and the room, which no transcript carries. */
+ heard?: { from: number; to: number }[];
/** What the assistant read off the source, in its own words, against the
* span it read it from. Contact sheets leave the conversation as it grows;
* these stay, so a source longer than one look can be decided from the
diff --git a/site/src/cut/lib/watch/dedupSelector.test.ts b/site/src/cut/lib/watch/dedupSelector.test.ts
index b1257bac..012acb56 100644
--- a/site/src/cut/lib/watch/dedupSelector.test.ts
+++ b/site/src/cut/lib/watch/dedupSelector.test.ts
@@ -76,10 +76,10 @@ describe("action channel", () => {
});
});
-describe("settled channel", () => {
+describe("text channel", () => {
// A "caption" lands on a static scene and stays: 1px rows of ink — thin
// enough that every cell mean stays inside the global and action
- // tolerances, so only the settled channel can see it.
+ // tolerances, so only a local channel can see it at all.
const caption = (color: [number, number, number]) =>
paint(GRAY, [
{ x: 24, y: 150, w: 144, h: 1, color },
@@ -91,48 +91,118 @@ describe("settled channel", () => {
const r = select([paint(GRAY), paint(GRAY), caption(BLACK), caption(BLACK), caption(BLACK)]);
expect(r.kept.includes(2)).toBe(true);
const d = r.decisions[2];
- expect(d.verdict).toBe("keep-settled");
+ expect(d.verdict).toBe("keep-text");
expect(d.globalDist).toBeLessThanOrEqual(8);
- expect(d.settledScore).toBeGreaterThan(d.settledGate!);
+ });
+
+ test("the last candidate of a pass gets no text keep", () => {
+ // The settle test needs a frame after this one. Without it, motion caught
+ // mid-stride reads as a state that landed, which is what the channel is
+ // built to reject — so the pass ends and the settled channel judges it.
+ const r = select([paint(GRAY), paint(GRAY), caption(BLACK)]);
+ expect(r.decisions[2].verdict).not.toBe("keep-text");
+ });
+
+ // The reason this channel exists. A caption whose colour sits near its
+ // ground — white type over a cream wall, a light serif over a white dress —
+ // moves no cell mean worth the name, and a channel that tests contrast
+ // cannot see it. Edge energy reads it by its strokes, so it lands the same
+ // as ink on paper.
+ test("a low-contrast caption is kept like any other", () => {
+ const pale: [number, number, number] = [150, 150, 150]; // Δ30 on GRAY
+ const r = select([paint(GRAY), paint(GRAY), caption(pale), caption(pale), caption(pale)]);
+ expect(r.decisions[2].verdict).toBe("keep-text");
});
test("change still in flight waits for the settled state", () => {
// The stroke keeps moving every frame; only the final state (nothing
- // after it to prove motion) can take a settled keep.
+ // after it to prove motion) can take a keep.
const moving = (y: number) => paint(GRAY, [{ x: 24, y, w: 144, h: 1, color: BLACK }]);
const r = select([paint(GRAY), moving(60), moving(90), moving(120)]);
expect(r.decisions[1].verdict).toBe("drop");
expect(r.decisions[2].verdict).toBe("drop");
- expect(r.decisions[3].verdict).toBe("keep-settled");
+ expect(r.decisions[3].verdict).not.toBe("drop");
+ });
+
+ test("a mark too narrow to be a line of type is left to the settled channel", () => {
+ // A blob one cell across has no run behind it: an eye, a mouth, a cursor.
+ const blob = paint(GRAY, [{ x: 100, y: 100, w: 6, h: 6, color: BLACK }]);
+ const r = select([paint(GRAY), paint(GRAY), blob, blob, blob]);
+ expect(r.decisions[2].verdict).toBe("keep-settled");
+ });
+
+ test("a wash across the whole frame is not type", () => {
+ // A fade moves most of the grid at once. Locality is what separates a
+ // caption from the light changing, and the global channel owns the rest.
+ const wash = (v: number) =>
+ paint(GRAY, [{ x: 0, y: 0, w: S, h: S, color: [v, v, v] }]);
+ const r = select([paint(GRAY), wash(126), wash(132), wash(138), wash(144)]);
+ expect(r.decisions.slice(1).some((d) => d.verdict === "keep-text")).toBe(false);
+ });
+
+ test("every state of a caption track is kept", () => {
+ // What the channel is for: a line swaps, holds, swaps again, and each
+ // state comes back as its own frame. A track read one state short is a
+ // line of the source nobody can quote.
+ const ys = [30, 30, 60, 60, 90, 90, 120, 120, 150, 150];
+ const r = select([
+ paint(GRAY),
+ ...ys.map((y) => paint(GRAY, [{ x: 24, y, w: 144, h: 1, color: BLACK }])),
+ paint(GRAY),
+ ]);
+ expect(r.decisions.filter((d) => d.verdict === "keep-text").length).toBe(5);
+ });
+
+ test("a second state landing on the next frame waits for the bar to fall", () => {
+ // Each keep raises the tolerance, so a fainter change arriving right
+ // behind one does not take a frame of its own until the bar decays.
+ const line = (y: number, color: [number, number, number]) =>
+ paint(GRAY, [{ x: 24, y, w: 144, h: 1, color }]);
+ const r = select([
+ paint(GRAY),
+ line(30, BLACK),
+ line(30, BLACK),
+ paint(GRAY, [
+ { x: 24, y: 30, w: 144, h: 1, color: BLACK },
+ { x: 24, y: 150, w: 144, h: 1, color: [150, 150, 150] },
+ ]),
+ paint(GRAY, [
+ { x: 24, y: 30, w: 144, h: 1, color: BLACK },
+ { x: 24, y: 150, w: 144, h: 1, color: [150, 150, 150] },
+ ]),
+ paint(GRAY),
+ ]);
+ expect(r.decisions[1].verdict).toBe("keep-text");
+ expect(r.decisions[2].verdict).toBe("drop");
});
+});
+
+describe("settled channel", () => {
+ // A settled change that is not a line: thin ink, a cursor, a UI dot.
+ const mark = (color: [number, number, number], y = 100) =>
+ paint(GRAY, [{ x: 100, y, w: 6, h: 6, color }]);
test("soft-contrast drift dies at the strict tolerance", () => {
- // A faint wash (Δ90: over the soft tolerance, under the strict 105) must
+ // A faint blob (Δ90: over the soft tolerance, under the strict 105) must
// not take a frame — smoke and fades live there.
- const wash = paint(GRAY, [
- { x: 24, y: 150, w: 144, h: 1, color: [210, 210, 210] },
- { x: 24, y: 156, w: 144, h: 1, color: [210, 210, 210] },
- { x: 24, y: 162, w: 144, h: 1, color: [210, 210, 210] },
- ]);
- const r = select([paint(GRAY), paint(GRAY), wash, wash, wash]);
+ const r = select([paint(GRAY), paint(GRAY), mark([210, 210, 210]), mark([210, 210, 210])]);
expect(r.decisions[2].verdict).toBe("drop");
});
test("the cooldown delays a settled keep right after another", () => {
- const frames = [paint(GRAY), paint(GRAY), caption(BLACK), caption(BLACK)];
- // A second small settled change (a new white stroke) lands right after
- // the caption's keep.
- const second = paint(GRAY, [
- { x: 24, y: 150, w: 144, h: 1, color: BLACK },
- { x: 24, y: 156, w: 144, h: 1, color: BLACK },
- { x: 24, y: 162, w: 144, h: 1, color: BLACK },
- { x: 24, y: 30, w: 144, h: 1, color: WHITE },
+ // A second, slighter mark lands one frame behind the first. Judged against
+ // the raised gate it waits; once the gate decays it takes its frame. This
+ // is what stops sustained settling motion from keeping every candidate.
+ const first = mark(BLACK);
+ const both = paint(GRAY, [
+ { x: 100, y: 100, w: 6, h: 6, color: BLACK },
+ { x: 100, y: 30, w: 3, h: 4, color: WHITE },
]);
- const r = select([...frames, second, second, second]);
- // Judged against the raised gate, the immediate follow-up waits...
+ const r = select([paint(GRAY), paint(GRAY), first, first, both, both, both]);
+ expect(r.decisions[2].verdict).toBe("keep-settled");
expect(r.decisions[4].verdict).toBe("drop");
+ expect(r.decisions[4].settledScore!).toBeLessThan(r.decisions[4].settledGate!);
expect(r.decisions[4].settledGate!).toBeGreaterThan(6.8);
- // ...and the state is kept once the gate decays back down.
expect(r.decisions[5].verdict).toBe("keep-settled");
});
});
diff --git a/site/src/cut/lib/watch/dedupSelector.ts b/site/src/cut/lib/watch/dedupSelector.ts
index 8f85e6c5..7c62ba95 100644
--- a/site/src/cut/lib/watch/dedupSelector.ts
+++ b/site/src/cut/lib/watch/dedupSelector.ts
@@ -1,20 +1,21 @@
import {
diffCellPct,
+ edgeNewCells,
frameSig,
- type FrameSig,
hardCellCount,
settledCellScores,
} from "./signatures";
import type {
FrameDecision,
FrameSelector,
+ FrameSig,
RgbFrame,
SelectionResult,
SelectorOptions,
Verdict,
} from "./types";
-/** The default watch selector: three change detectors vote on each candidate
+/** The default watch selector: four change detectors vote on each candidate
* against a sliding window of the last kept frames. The window (rather than
* only the previous frame) keeps A-B-A alternation out — a shot the model has
* already seen stays out even when a different frame sat in between.
@@ -24,13 +25,20 @@ import type {
* - action: a few 32-grid cells changing hard mark a frame as new no matter
* the percentage — a small subject can never move 8% of the pixels, so the
* one second that matters would otherwise dedup away.
+ * - text: cells whose EDGE ENERGY holds a new settled state against every
+ * kept frame in the window. Type is edges, so this channel reads a caption
+ * by its strokes rather than by its colour — white-on-cream and
+ * black-on-white swaps land the same, and both are invisible to the
+ * channels above (the cell average barely stirs). The settle test against
+ * the next candidate is what separates a held line from a moving mouth,
+ * so it needs no static-scene gate: it runs while the speaker talks.
* - settled: on a static scene, pixels that differ strongly from every kept
* frame in the window (shift-tolerant, so grain and jitter re-match) and
- * have stopped changing toward the next frame. Sees caption swaps, thin
- * ink, and local UI updates that measure 0.0% on the global channel. A
- * cooldown raises its bar after each keep so sustained "settling" motion
- * can't take a frame every time; a stricter second tolerance must fire in
- * the same cell so soft-contrast drift (smoke, fades) stays out.
+ * have stopped changing toward the next frame. Sees thin ink and local UI
+ * updates that measure 0.0% on the global channel. A cooldown raises its
+ * bar after each keep so sustained "settling" motion can't take a frame
+ * every time; a stricter second tolerance must fire in the same cell so
+ * soft-contrast drift (smoke, fades) stays out.
*
* Memory holds the window plus a one-frame lookahead, never the whole stream. */
@@ -45,6 +53,19 @@ export const DEDUP_TUNING = {
ACTION_HARD_TOL: 45,
/** Action channel: hard cells needed to keep. */
ACTION_HARD_CELLS: 3,
+ /** Text channel: edge-energy change for a 24-grid cell to count as new. */
+ TEXT_TOL: 6,
+ /** Text channel: adjacent new cells in one row for the run to count as a
+ * line of type, and new cells needed overall to keep. */
+ TEXT_RUN: 3,
+ TEXT_CELLS: 3,
+ /** Text channel: share of the grid the new cells may cover. Type is local;
+ * a fade, a wash or a pan moves most of the frame, and that belongs to the
+ * global channel rather than to this one. */
+ TEXT_MAX_SHARE: 0.15,
+ /** Cooldown: each text keep raises the bar additively; it decays per frame. */
+ TEXT_COOLDOWN_ADD: 1.5,
+ TEXT_COOLDOWN_DECAY: 0.6,
/** Settled channel runs only when the global diff vs the previous candidate
* sits under this (the scene is otherwise static), or on the final frame —
* a state that appears at the end has nothing after it to prove it settled. */
@@ -74,9 +95,35 @@ export function createDedupSelector(
let pending: FrameSig | null = null;
let prevG16: Float32Array | null = null; // previous candidate, kept or dropped
let cooldown = 1;
+ let textCooldown = 1;
let count = 0;
let finished = false;
+ /** Cells holding a new settled edge state, in a horizontal run wide enough
+ * to be a line of type and local enough not to be the light changing. The
+ * settle test needs a frame after this one, so the channel never runs on the
+ * last candidate of a pass: there, a state that has not been proved still is
+ * indistinguishable from motion caught mid-stride, and the settled channel
+ * is what judges it. */
+ function isTypeChange(
+ sig: FrameSig,
+ kept: FrameSig[],
+ next: FrameSig,
+ tuning: typeof DEDUP_TUNING
+ ): boolean {
+ const cells = edgeNewCells(
+ sig.e24,
+ kept.map((k) => k.e24),
+ next.e24,
+ tuning.TEXT_TOL * textCooldown,
+ tuning.TEXT_RUN
+ );
+ return (
+ cells.length >= tuning.TEXT_CELLS &&
+ cells.length <= tuning.TEXT_MAX_SHARE * sig.e24.length
+ );
+ }
+
function judge(sig: FrameSig, next: FrameSig | null): void {
const index = decisions.length;
let verdict: Verdict = "drop";
@@ -100,6 +147,9 @@ export function createDedupSelector(
)
) {
verdict = "keep-action";
+ } else if (next !== null && isTypeChange(sig, window, next, T)) {
+ verdict = "keep-text";
+ textCooldown += T.TEXT_COOLDOWN_ADD;
} else {
const motion = prevG16 === null ? 100 : diffCellPct(sig.g16, prevG16, T.GLOBAL_TOL);
if (motion < T.SETTLED_STATIC_MAX || next === null) {
@@ -131,6 +181,7 @@ export function createDedupSelector(
prevG16 = sig.g16;
cooldown = Math.max(1, cooldown * T.SETTLED_COOLDOWN_DECAY);
+ textCooldown = Math.max(1, textCooldown * T.TEXT_COOLDOWN_DECAY);
if (verdict !== "drop") {
keptIdx.push(index);
window.push(sig);
@@ -142,9 +193,9 @@ export function createDedupSelector(
}
return {
- push(frame: RgbFrame) {
+ push(frame: RgbFrame | FrameSig) {
if (finished) throw new Error("Selector already finished.");
- const sig = frameSig(frame);
+ const sig = "e24" in frame ? frame : frameSig(frame);
count++;
if (pending) judge(pending, sig);
pending = sig;
diff --git a/site/src/cut/lib/watch/fuse.test.ts b/site/src/cut/lib/watch/fuse.test.ts
index 8b32b7b8..17699f9e 100644
--- a/site/src/cut/lib/watch/fuse.test.ts
+++ b/site/src/cut/lib/watch/fuse.test.ts
@@ -1,6 +1,6 @@
import { describe, expect, test } from "bun:test";
-import { fuseTimeline, renderFusedTimeline } from "./fuse";
+import { fuseTimeline, renderFusedTimeline, speechOnsets, speechOver } from "./fuse";
const cue = (start: number, end: number, text: string) => ({ start, end, text });
@@ -62,3 +62,38 @@ describe("renderFusedTimeline", () => {
expect(renderFusedTimeline(fuseTimeline([], [], { from: 0, to: 8 }))).toBe("");
});
});
+
+describe("speechOnsets", () => {
+ test("each line earns a look just after it lands", () => {
+ // Just after: the new state is up and has stopped moving.
+ expect(speechOnsets([cue(0, 2, "having to"), cue(2, 4, "say it twice")])).toEqual([0.2, 2.2]);
+ });
+
+ test("a silent cue and anything outside the span aim at nothing", () => {
+ const cues = [cue(0, 2, " "), cue(4, 6, "in"), cue(20, 22, "out")];
+ expect(speechOnsets(cues, 3, 10)).toEqual([4.2]);
+ });
+});
+
+describe("speechOver", () => {
+ const transcript = [cue(0, 2, "from the source"), cue(4, 6, "and on")];
+
+ test("the project's cues win where they reach the stretch", () => {
+ const cues = [cue(0, 2, "from the lane")];
+ expect(speechOver(cues, transcript, 0, 6)[0].text).toBe("from the lane");
+ });
+
+ test("a lane whose cues belong to another clip falls back to the transcript", () => {
+ // The caption lane holds cues for a different clip, so mapping them
+ // through this one's retime lands them all outside its source range.
+ const elsewhere = [cue(300, 302, "another clip's line")];
+ expect(speechOver(elsewhere, transcript, 0, 6).map((c) => c.text)).toEqual([
+ "from the source",
+ "and on",
+ ]);
+ });
+
+ test("a stretch nothing covers comes back empty", () => {
+ expect(speechOver([], transcript, 100, 120)).toEqual([]);
+ });
+});
diff --git a/site/src/cut/lib/watch/fuse.ts b/site/src/cut/lib/watch/fuse.ts
index 309b5955..6d801644 100644
--- a/site/src/cut/lib/watch/fuse.ts
+++ b/site/src/cut/lib/watch/fuse.ts
@@ -69,3 +69,37 @@ export function renderFusedTimeline(spans: FusedSpan[]): string {
}
return lines.join("\n");
}
+
+/** A cue lands and the picture under it changes with it: the caption swaps,
+ * the speaker starts, the graphic cuts in. These are the moments a watch aims
+ * its frames at, just after each cue starts so the new state is up and still.
+ * A source nobody has transcribed offers none and sweeps on the steady floor
+ * alone. */
+export const CUE_LEAD_S = 0.2;
+
+export function speechOnsets(
+ cues: { start: number; end: number; text: string }[],
+ from = 0,
+ to = Infinity
+): number[] {
+ return cues
+ .filter((c) => c.text.trim().length > 0)
+ .map((c) => c.start + CUE_LEAD_S)
+ .filter((t) => t >= from && t < to);
+}
+
+/** The speech that covers a stretch of a source: the project's own caption
+ * cues where they reach it, the source's transcript everywhere else. A lane
+ * holding cues for one clip leaves every other clip to its own transcript,
+ * which is what a watch aims its frames at and fuses its timeline from. */
+export function speechOver(
+ cues: { start: number; end: number; text: string }[],
+ transcript: { start: number; end: number; text: string }[],
+ from: number,
+ to: number
+): { start: number; end: number; text: string }[] {
+ const within = (list: { start: number; end: number; text: string }[]) =>
+ list.filter((c) => c.end > from && c.start < to);
+ const mapped = within(cues);
+ return mapped.length > 0 ? mapped : within(transcript);
+}
diff --git a/site/src/cut/lib/watch/merge.test.ts b/site/src/cut/lib/watch/merge.test.ts
index b332836f..9699dddb 100644
--- a/site/src/cut/lib/watch/merge.test.ts
+++ b/site/src/cut/lib/watch/merge.test.ts
@@ -1,6 +1,6 @@
import { describe, expect, test } from "bun:test";
-import { mergeSpeech, mergeWatch, mergeWatchNotes, nextUncoveredSpan, uncoveredSeconds, unnotedSpans } from "./merge";
+import { mergeHeard, mergeSpeech, mergeWatch, mergeWatchNotes, nextUncoveredSpan, uncoveredSeconds, unnotedSpans } from "./merge";
import type { AssetWatch } from "../types";
const frame = (t: number, via: AssetWatch["frames"][number]["via"] = "global") => ({ t, via });
@@ -170,3 +170,84 @@ describe("uncoveredSeconds", () => {
expect(uncoveredSeconds(undefined, 64.1)).toBe(64.1);
});
});
+
+describe("close reads", () => {
+ test("a close pass records its span; a scan leaves the record alone", () => {
+ const scan = mergeWatch(undefined, { from: 0, to: 30, frames: [], sceneChanges: [] });
+ expect(scan.read).toBeUndefined();
+ const read = mergeWatch(scan, {
+ from: 0, to: 12, frames: [], sceneChanges: [], readClosely: true });
+ expect(read.read).toEqual([{ from: 0, to: 12 }]);
+ // A record written before the ladder carries nothing read, never an error.
+ const later = mergeWatch(read, { from: 12, to: 30, frames: [], sceneChanges: [] });
+ expect(later.read).toEqual([{ from: 0, to: 12 }]);
+ });
+
+ test("close spans union the way coverage does", () => {
+ let w = mergeWatch(undefined, {
+ from: 0, to: 10, frames: [], sceneChanges: [], readClosely: true });
+ w = mergeWatch(w, { from: 10, to: 20, frames: [], sceneChanges: [], readClosely: true });
+ w = mergeWatch(w, { from: 40, to: 50, frames: [], sceneChanges: [], readClosely: true });
+ expect(w.read).toEqual([{ from: 0, to: 20 }, { from: 40, to: 50 }]);
+ });
+});
+
+describe("mergeHeard", () => {
+ test("listened spans union, and coverage reads them like any range", () => {
+ let w = mergeHeard(undefined, { from: 0, to: 120 });
+ expect(w.heard).toEqual([{ from: 0, to: 120 }]);
+ w = mergeHeard(w, { from: 120, to: 200 });
+ expect(w.heard).toEqual([{ from: 0, to: 200 }]);
+ expect(uncoveredSeconds({ ranges: w.heard ?? [] }, 300)).toBe(100);
+ expect(nextUncoveredSpan({ ranges: w.heard ?? [] }, 300)).toEqual({ from: 200, to: 300 });
+ });
+
+ test("listening leaves the frames and notes of a watch alone", () => {
+ const watched = mergeWatchNotes(
+ mergeWatch(undefined, { from: 0, to: 30, frames: [frame(1)], sceneChanges: [1] }),
+ [{ from: 0, to: 30, text: "the hook" }]
+ );
+ const both = mergeHeard(watched, { from: 0, to: 30 });
+ expect(both.frames).toEqual(watched.frames);
+ expect(both.notes).toEqual(watched.notes);
+ expect(both.heard).toEqual([{ from: 0, to: 30 }]);
+ });
+});
+
+describe("one record, three senses", () => {
+ // Every merge carries forward what the other passes measured. Rebuilding
+ // the record from the fields one pass owns erases the rest of it, and the
+ // tools that write it run in whatever order the work takes.
+ const full = mergeHeard(
+ mergeWatchNotes(
+ mergeWatch(undefined, { from: 0, to: 12, frames: [], sceneChanges: [], readClosely: true }),
+ [{ from: 0, to: 12, text: "titles over a talking head" }],
+ ),
+ { from: 0, to: 30 },
+ );
+
+ test("the close read, the notes and the listen all survive each other", () => {
+ expect(full.read).toEqual([{ from: 0, to: 12 }]);
+ expect(full.heard).toEqual([{ from: 0, to: 30 }]);
+ expect(full.notes?.length).toBe(1);
+ });
+
+ test("writing a note keeps what was read closely and what was played", () => {
+ const after = mergeWatchNotes(full, [{ from: 12, to: 20, text: "the next stretch" }]);
+ expect(after.read).toEqual([{ from: 0, to: 12 }]);
+ expect(after.heard).toEqual([{ from: 0, to: 30 }]);
+ });
+
+ test("the background sweep's coverage merge keeps them too", () => {
+ const after = mergeWatch(full, { from: 20, to: 40, frames: [], sceneChanges: [] });
+ expect(after.read).toEqual([{ from: 0, to: 12 }]);
+ expect(after.heard).toEqual([{ from: 0, to: 30 }]);
+ expect(after.notes?.length).toBe(1);
+ });
+
+ test("a listen keeps the close read and the notes", () => {
+ const after = mergeHeard(full, { from: 30, to: 50 });
+ expect(after.read).toEqual([{ from: 0, to: 12 }]);
+ expect(after.notes?.length).toBe(1);
+ });
+});
diff --git a/site/src/cut/lib/watch/merge.ts b/site/src/cut/lib/watch/merge.ts
index 27e16532..991d7bd0 100644
--- a/site/src/cut/lib/watch/merge.ts
+++ b/site/src/cut/lib/watch/merge.ts
@@ -100,6 +100,8 @@ export function mergeWatch(
to: number;
frames: { t: number; via: WatchKeepReason }[];
sceneChanges: number[];
+ /** The pass read the span at a detail that carries type. */
+ readClosely?: boolean;
}
): AssetWatch {
const from = round2(Math.min(add.from, add.to));
@@ -121,15 +123,51 @@ export function mergeWatch(
if (last && rg.from <= last.to + EPS) last.to = Math.max(last.to, rg.to);
else ranges.push({ from: rg.from, to: rg.to });
}
- // Notes survive a re-watch: they are what someone read off the source, not
- // a product of this pass.
- return { ranges, frames, sceneChanges, ...(prev?.notes?.length ? { notes: prev.notes } : {}) };
+ const read: AssetWatch["read"] = [];
+ const closeSpans = [...(prev?.read ?? [])];
+ if (add.readClosely && to > from) closeSpans.push({ from, to });
+ for (const rg of closeSpans.sort((a, b) => a.from - b.from)) {
+ const last = read[read.length - 1];
+ if (last && rg.from <= last.to + EPS) last.to = Math.max(last.to, rg.to);
+ else read.push({ from: rg.from, to: rg.to });
+ }
+ // Every merge carries the record forward and overwrites only what it
+ // measures. Notes survive a re-watch — they are what someone read off the
+ // source — and so does every other measure a different pass owns.
+ return {
+ ...prev,
+ ranges,
+ frames,
+ sceneChanges,
+ ...(read.length ? { read } : {}),
+ };
}
/** Fold written observations into an asset's watch record. A new note owns
* its span: stored notes that fall inside it drop, so re-reading a stretch
* closely supersedes the coarse pass that first described it. Everything
* else stays, sorted by time. */
+/** Fold a listened span into the record, the way coverage folds a watched
+ * one. Listening leaves no frames and no cuts, so it has its own small merge
+ * rather than a mode on mergeWatch. */
+export function mergeHeard(
+ prev: AssetWatch | undefined,
+ span: { from: number; to: number }
+): AssetWatch {
+ const base: AssetWatch = prev ?? { ranges: [], frames: [], sceneChanges: [] };
+ const from = round2(Math.min(span.from, span.to));
+ const to = round2(Math.max(span.from, span.to));
+ const heard: { from: number; to: number }[] = [];
+ const spans = [...(prev?.heard ?? [])];
+ if (to > from) spans.push({ from, to });
+ for (const rg of spans.sort((a, b) => a.from - b.from)) {
+ const last = heard[heard.length - 1];
+ if (last && rg.from <= last.to + EPS) last.to = Math.max(last.to, rg.to);
+ else heard.push({ from: rg.from, to: rg.to });
+ }
+ return { ...base, ...(heard.length ? { heard } : {}) };
+}
+
export function mergeWatchNotes(
prev: AssetWatch | undefined,
add: { from: number; to: number; text: string }[]
@@ -156,6 +194,7 @@ export function mergeWatchNotes(
.concat(incoming)
.sort((a, b) => a.from - b.from || a.to - b.to);
return {
+ ...prev,
ranges: prev?.ranges ?? [],
frames: prev?.frames ?? [],
sceneChanges: prev?.sceneChanges ?? [],
diff --git a/site/src/cut/lib/watch/signatures.ts b/site/src/cut/lib/watch/signatures.ts
index 0443f125..20670f52 100644
--- a/site/src/cut/lib/watch/signatures.ts
+++ b/site/src/cut/lib/watch/signatures.ts
@@ -1,19 +1,16 @@
-import { type RgbFrame, SIGNATURE_SIZE } from "./types";
+import { type FrameSig, type RgbFrame, SIGNATURE_SIZE } from "./types";
/** The pixel comparisons behind frame selection. All math runs on three
* fixed views of a SIGNATURE_SIZE² frame:
* - fine: the pixels themselves (packed RGB) — local detail
* - g16 / g32: per-cell channel means — whole-frame structure
+ * - e24: per-cell edge energy — where type and hard detail live
* Comparisons use the max channel difference, so hues with equal luma still
* read as different. */
const S = SIGNATURE_SIZE;
-export interface FrameSig {
- fine: Uint8Array; // S*S*3 packed RGB
- g16: Float32Array; // 16*16*3 cell means
- g32: Float32Array; // 32*32*3 cell means
-}
+export type { FrameSig };
function meanGrid(fine: Uint8Array, grid: number): Float32Array {
const cell = S / grid;
@@ -55,7 +52,12 @@ export function frameSig(f: RgbFrame): FrameSig {
fine[o + 2] = f.data[i + 2];
}
}
- return { fine, g16: meanGrid(fine, 16), g32: meanGrid(fine, 32) };
+ return {
+ fine,
+ g16: meanGrid(fine, 16),
+ g32: meanGrid(fine, 32),
+ e24: edgeGrid(fine, TEXT_GRID),
+ };
}
/** % of grid cells whose max channel differs by more than tol. */
@@ -145,3 +147,78 @@ export function settledCellScores(
for (let i = 0; i < counts.length; i++) counts[i] *= norm;
return counts;
}
+
+export const TEXT_GRID = 24;
+
+/** Per-cell mean gradient magnitude — how much hard edge lives in each cell.
+ * Type is edges: a caption that lands, swaps or leaves moves its cells' edge
+ * energy hard, and it moves whatever the polarity or the contrast of the
+ * type. Mean-colour channels cannot see white-on-cream captions at all (the
+ * cell average barely stirs); this one reads them the same as black-on-white. */
+// The luma view is scratch: one frame's worth, reused by every call. A watch
+// runs this per candidate and the frame loop allocates nothing.
+const lumaScratch = new Float32Array(S * S);
+
+export function edgeGrid(fine: Uint8Array, grid: number): Float32Array {
+ const cell = S / grid;
+ const out = new Float32Array(grid * grid);
+ // Luma first, so a coloured caption on a same-luma ground still edges.
+ const luma = lumaScratch;
+ for (let p = 0, i = 0; p < S * S; p++, i += 3)
+ luma[p] = 0.299 * fine[i] + 0.587 * fine[i + 1] + 0.114 * fine[i + 2];
+ for (let y = 1; y < S - 1; y++) {
+ const gy = Math.floor(y / cell);
+ for (let x = 1; x < S - 1; x++) {
+ const p = y * S + x;
+ const g =
+ Math.abs(luma[p + 1] - luma[p - 1]) + Math.abs(luma[p + S] - luma[p - S]);
+ out[gy * grid + Math.floor(x / cell)] += g;
+ }
+ }
+ const norm = 1 / (cell * cell);
+ for (let i = 0; i < out.length; i++) out[i] *= norm;
+ return out;
+}
+
+/** Cells whose edge energy differs from EVERY reference by more than tol, and
+ * — when a next frame exists — has stopped moving toward it. A held caption
+ * qualifies; a mouth mid-word does not, because the next sample moves it
+ * again. Only cells inside a horizontal run of at least `run` survive: a line
+ * of type is wide, a mouth or a blinking eye is one or two cells across, and
+ * that shape is what separates them once both have settled. Returns the cell
+ * indices, which also bound the crop. */
+export function edgeNewCells(
+ cur: Float32Array,
+ refs: Float32Array[],
+ next: Float32Array | null,
+ tol: number,
+ run: number,
+ grid = TEXT_GRID
+): number[] {
+ const neu = new Uint8Array(cur.length);
+ for (let i = 0; i < cur.length; i++) {
+ if (refs.length === 0) break;
+ let isNew = true;
+ for (const r of refs)
+ if (Math.abs(cur[i] - r[i]) <= tol) {
+ isNew = false;
+ break;
+ }
+ if (isNew && next !== null && Math.abs(next[i] - cur[i]) > tol) isNew = false;
+ if (isNew) neu[i] = 1;
+ }
+ const cells: number[] = [];
+ for (let row = 0; row < grid; row++) {
+ let start = -1;
+ for (let col = 0; col <= grid; col++) {
+ const on = col < grid && neu[row * grid + col] === 1;
+ if (on && start < 0) start = col;
+ if (!on && start >= 0) {
+ if (col - start >= run)
+ for (let c = start; c < col; c++) cells.push(row * grid + c);
+ start = -1;
+ }
+ }
+ }
+ return cells;
+}
diff --git a/site/src/cut/lib/watch/sweep.ts b/site/src/cut/lib/watch/sweep.ts
index a5f6d1a7..0baa0332 100644
--- a/site/src/cut/lib/watch/sweep.ts
+++ b/site/src/cut/lib/watch/sweep.ts
@@ -12,6 +12,7 @@ import { engineTranscribeSamples } from "../localStt";
import { sampleWatchFrames } from "../media";
import { runTranscription, useEditor } from "../store";
import { trackLocale } from "../subtitles";
+import { speechOnsets } from "./fuse";
import { mergeSpeech, mergeWatch, nextUncoveredSpan } from "./merge";
/** Background watch sweep: after the assistant's first look at (or listen to)
@@ -207,6 +208,9 @@ async function run(): Promise<void> {
to: span.to,
metadataOnly: true,
shouldPause: decodersBusy,
+ // The transcript is already in by now (speech runs first), so the
+ // map is drawn on the source's own lines rather than a blind grid.
+ at: speechOnsets(asset.speech?.segments ?? [], span.from, span.to),
});
if (r.coveredTo > span.from) {
const cur = useEditor.getState().assets.find((a) => a.id === id);
diff --git a/site/src/cut/lib/watch/types.ts b/site/src/cut/lib/watch/types.ts
index f587fda7..95d522e4 100644
--- a/site/src/cut/lib/watch/types.ts
+++ b/site/src/cut/lib/watch/types.ts
@@ -3,8 +3,10 @@
* piped from ffmpeg; the browser feeds it ImageData. Pure data in, pure data
* out — no DOM, no Node. */
-/** Every candidate reaches the selector at this size; signatures divide it
- * evenly (192 = 12·16 = 6·32). */
+/** Every candidate reaches the selector at this size; every signature grid
+ * divides it evenly (192 = 12·16 = 8·24 = 6·32). A size that breaks one of
+ * those leaves a grid with fractional cells, which drifts its values by cell
+ * position rather than failing. */
export const SIGNATURE_SIZE = 192;
/** One candidate frame's pixels at SIGNATURE_SIZE × SIGNATURE_SIZE.
@@ -18,11 +20,21 @@ export interface RgbFrame {
data: Uint8Array | Uint8ClampedArray;
}
+/** The three fixed views of one candidate, from signatures.ts. Declared here
+ * so the selector's contract can name it without the cycle. */
+export interface FrameSig {
+ fine: Uint8Array;
+ g16: Float32Array;
+ g32: Float32Array;
+ e24: Float32Array;
+}
+
export type Verdict =
| "first" // the opening frame — always kept
| "keep-global" // whole-frame change vs the kept window
| "keep-action" // hard local change on a small subject
- | "keep-settled" // a settled local change on a static scene (text, ink, UI)
+ | "keep-text" // a settled new edge state — type landing, swapping or leaving
+ | "keep-settled" // a settled local change on a static scene (ink, UI)
| "drop" // a near-duplicate of the kept window
| "thinned"; // kept by the channels, removed by the max-frames cap
@@ -54,9 +66,11 @@ export interface SelectorOptions {
}
/** Push candidates in time order, then finish. Decisions stream one frame
- * behind push; finish flushes the last frame and applies the cap. */
+ * behind push; finish flushes the last frame and applies the cap. A caller
+ * that already needs a frame's signature for its own work passes that in
+ * place of the pixels, so one frame is measured once. */
export interface FrameSelector {
- push(frame: RgbFrame): void;
+ push(frame: RgbFrame | FrameSig): void;
finish(): SelectionResult;
}
diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.ts
index 255e3a5a..80eaa6d2 100644
--- a/site/src/cut/server/ai/catalog.ts
+++ b/site/src/cut/server/ai/catalog.ts
@@ -287,10 +287,10 @@ Filler words: find_filler judges every transcript word in its sentence and retur
When a request depends on what the footage actually contains — "cut the dead air", "clip the best moment", "remove the boring part", "split where the scene changes" — watch it first. Never guess at content you haven't seen.
Your eyes and ears:
-- watch_video(clip_id | asset_id, from?, to?, interval_seconds?): samples the SOURCE on a dense steady floor, keeps only the frames that actually differ, and returns them as timestamped contact sheets — cells read left→right, top→bottom, each burned stamp is source seconds, and a gap between stamps means nothing changed there — plus sceneChanges, hard-cut times refined to about a third of a second of the true boundary, and — when the clip has captions or the source's transcript exists — a fused timeline placing each frame inside the speech it belongs to. Coverage ends where the result says: when truncated (span bound, frame cap, or time budget), continue from coveredTo. A shot shorter than the sampling step can fall between candidates — where short cuts matter, re-watch narrow with interval_seconds 0.5–1. Watching builds the source's frame map, persisted on the asset; each media entry's "mapped" spans list where that map exists, and the first watch or listen of a source starts a background sweep that maps the rest, transcript included ("watching": true marks it running; the spans grow as segments land). Use the map to aim your watches — a mapped span is NOT footage you have seen; only the sheets returned to you in this conversation are, so watch any span before you cut inside it.
+- watch_video(clip_id | asset_id, from?, to?, interval_seconds?, detail?): samples the SOURCE on a dense steady floor plus a frame just after each line it speaks — what a video shows changes when what it says changes, so a transcribed source is read on its own clock and a caption track lands a frame per card — keeps only the frames that actually differ, and returns them as timestamped contact sheets — cells read left→right, top→bottom, each burned stamp is source seconds, and a gap between stamps means nothing changed there — plus sceneChanges, hard-cut times refined to about a third of a second of the true boundary, and — when the clip has captions or the source's transcript exists — a fused timeline placing each frame inside the speech it belongs to. Coverage ends where the result says: when truncated (span bound, frame cap, or time budget), continue from coveredTo. A shot shorter than the sampling step can fall between candidates — where short cuts matter, re-watch narrow with interval_seconds 0.5–1. \`detail\` sets how many pixels each kept frame is worth: scan (default) tiles small cells for coverage, read enlarges them enough for on-screen text, original returns the source's own frames for matching type, colour and edges. A scan cell is a quarter of the frame's width, so small captions arrive unreadable — reading words off the screen or reproducing a look means spending the pixels, never guessing at what the type says. \`textFits\` rides back with them: frame fractions, quietest first, for where a card can sit over that stretch without covering the picture or the platform's UI — the busy parts are what the watch measured to pick these frames in the first place. Seeing and reading are counted apart: \`unwatchedSeconds\` is source nobody has looked at, \`unreadSeconds\` is source nobody has read at a legible size, and \`unreadFrom\` says where the next close pass starts. Both persist on the asset. Watching builds the source's frame map, persisted on the asset; each media entry's "mapped" spans list where that map exists, and the first watch or listen of a source starts a background sweep that maps the rest, transcript included ("watching": true marks it running; the spans grow as segments land). Use the map to aim your watches — a mapped span is NOT footage you have seen; only the sheets returned to you in this conversation are, so watch any span before you cut inside it.
- note_source(clip_id | asset_id, notes: [{from, to, text}]): writes what you saw onto the source, in source seconds. The sheets are conversation-local and the oldest media drops out as the chat grows; a note is permanent — it saves with the project, comes back as \`observed\` on the media entry in every later turn and chat, and is returned to you by get_state and by watch_video's \`recorded\`. This is how a source longer than one look becomes usable.
-**Seeing all of it.** Every result carries \`coveredTo\` and \`unwatchedSeconds\`. A \`to\` you passed yourself ends the call with \`truncated: false\` and the rest of the source unseen — that is not coverage. Any ask about the video AS A WHOLE — "what happens in this", "summarize it", "replicate/recreate this", "match this style", "how does it end" — means watching from 0 until \`unwatchedSeconds\` reaches 0, however many calls that takes; rebuilding one is the replicating-a-video skill. Watching a slice and describing the whole is how you end up reporting text that isn't there and putting the opening line at the end. Aiming a watch at one span is for a question about that span. One call reaches as far as its ceilings allow: a 600s span, 150 decodes, 36 kept frames, 90s of decoding — so at \`interval_seconds\` 0.5 a call covers at most 75s of source, and far less where the picture keeps changing. A source of a minute or two is a few calls; a source of an hour is not something to grind through frame-dense from 0. There, sweep it coarse first (\`interval_seconds\` 15–30 end to end) to learn its shape, then watch the spans that matter at a dense floor.
+**Seeing all of it.** Every result carries \`coveredTo\` and \`unwatchedSeconds\`. A \`to\` you passed yourself ends the call with \`truncated: false\` and the rest of the source unseen — that is not coverage. Any ask about the video AS A WHOLE — "what happens in this", "summarize it", "replicate/recreate this", "match this style", "how does it end" — means watching from 0 until \`unwatchedSeconds\` reaches 0, however many calls that takes; rebuilding one is the replicating-a-video skill. An ask that turns on what the frames SAY or how they are typeset — replicating a look, matching a caption treatment, quoting on-screen text — is scanned end to end for its shape and then re-watched at \`detail\` read or original over the stretches carrying the type, because a scan cell shows that a line is there and not what it reads. Watching a slice and describing the whole is how you end up reporting text that isn't there and putting the opening line at the end. Aiming a watch at one span is for a question about that span. One call reaches as far as its ceilings allow: a 600s span, 150 decodes, 90s of decoding, and the kept-frame cap its \`detail\` sets (36 at scan, 16 at read, 6 at original) — so at \`interval_seconds\` 0.5 a scan covers at most 75s of source, and far less where the picture keeps changing, while a detailed pass covers a fraction of that and belongs on the stretch that earns it. A source of a minute or two is a few calls; a source of an hour is not something to grind through frame-dense from 0. There, sweep it coarse first (\`interval_seconds\` 15–30 end to end) to learn its shape, then watch the spans that matter at a dense floor.
**Watch, write, decide.** Watching builds a record and the record is what you work from. Every pass ends with note_source: the span you just looked at and what it showed — on-screen text word for word, what happens, how it is shot, where the cuts land. Do it while the sheets are still on screen; the conversation drops its oldest media as it grows, and a watch nobody wrote down is a watch spent for nothing. Watching a source that still has an unrecorded look behind it is refused, so the loop is one pass, one note, next pass. The record comes back to you as \`observed\` on the media entry, in full from get_state, and as \`recorded\` and \`unnoted\` on every watch result. An hour of footage is decided from notes you can hold: sweep it coarse (\`interval_seconds\` 15–30) and note its shape, then watch the spans that matter at a dense floor and note those closely — a close note replaces the coarse one over the same span. Before you plan an edit, read \`observed\` and \`unnoted\`: a span already described needs no second look, and one still blank does.
@@ -318,7 +318,7 @@ Cutting speech well: pace is part of the message, and it is set in one place —
"replicating-a-video": `# Replicating a video you can watch
Someone points at a video — a file they imported, a link you brought in, a clip on their timeline — and asks for one like it. Build its skeleton out of this editor's own parts and leave the footage for them to fill in.
-**Watch the whole thing first.** Every decision below comes off the record, so the record has to be complete: watch from 0 and keep going until \`unwatchedSeconds\` is 0, writing each pass into note_source before the next one. Anything over a couple of minutes gets a coarse pass end to end first (\`interval_seconds\` 15-30) to learn its shape, then a dense pass (0.5) over the stretches that carry text or cut fast. Read every card word for word — its text, its face, where it sits in the frame, when it comes and goes — because a turn later the sheets are gone and the notes are all you have.
+**Watch the whole thing first.** Every decision below comes off the record, so the record has to be complete: watch from 0 and keep going until \`unwatchedSeconds\` is 0, writing each pass into note_source before the next one. Anything over a couple of minutes gets a coarse pass end to end first (\`interval_seconds\` 15-30) to learn its shape, then a dense pass (0.5) over the stretches that carry text or cut fast. Read every card word for word — its text, its face, where it sits in the frame, when it comes and goes — because a turn later the sheets are gone and the notes are all you have. Reading type is what \`detail\` is for: the dense pass over a stretch carrying words runs at \`detail\` read, and a face or treatment you are about to name runs at original, where the frame arrives at the source's own resolution. A scan cell is a quarter of the frame's width; a small caption in one is a smudge, and a smudge read as a word is how a card ends up saying something the video never said. When the cards ARE the narration — a caption track of what the speaker is saying — the transcript already holds every word, and the close passes are for how they are set: face, weight, size, colour, placement, how many words ride a card, how it enters and leaves. Read a few of those at original and take the rest from the transcript.
**Block out the cuts.** \`sceneChanges\` are the cut times: shot 1 runs from 0 to the first, shot n from one to the next, the last to the end. add_clip with \`blocks\` lays the whole skeleton down in one call — one block per shot on track 0, each the length that shot runs, labelled with what the person has to supply ("wide of the kitchen, hands only"). Name the source in \`reference_asset_id\`. They fill a block by dropping their own footage on it — it takes the block's place and its length — or by asking you to (replace_item), so everything you build on top keeps its timing. The reference itself stays off the timeline: it is what the cut is copied from, and the blocks hold its shots' slots until the person's own footage arrives.
@@ -585,7 +585,7 @@ Rules:
- Transcription tools write tracks the user sees — run subtitles_generate / captions_generate only when captions were asked for, never just to read the words: existing cues are in editor_state, sources you have watched or listened to carry their own transcript (media entries marked speech: "transcribed"; get_state includes the timed segments), and audio plays to you via listen_audio. Don't transcribe a video with no speech (subtitles_from_visuals narrates silent footage), and never invent a spoken transcript.
- Voiceovers duck other audio by default so they stay audible. Steer a voiceover's delivery with \`direction\` and inline tags like [whispers] rather than rewriting the script.
- generate_image / generate_video / generate_character_video / voiceover_generate / read_subtitles_aloud / generate_music make media through hosted models (spends the user's Donkey credits, needs sign-in); call them when the user asked for the media itself — a request for the prompt or script gets text in chat. generate_music makes a music track — a vocal-free background bed by default ("add music to this video"), or a full sung song (instrumental:false); it's sung music, so a spoken narration is voiceover_generate. Bundled stock media (stock_search / stock_add) is free — use it when it fits. Media the user attached is in \`media\`; pass those asset ids as generation references when they say "use this", and place project assets in the cut with add_clip when they ask for them there ("make a movie from these photos"). Generated media previews on a chat card the user can expand, drag in, or file away; add it to the timeline (add_to_timeline:true or an index) only when they asked for it in the cut. When it lands on a card, close with a statement of where it is ("The voiceover is on the card — drag it in or tell me to place it."). Write a rich, specific prompt from their shorthand. Video renders take a minute or two. "Generate a video of/about X" means one generate_video clip; only when they name a narrated multi-shot production (a story, episode, narrated short) does generate_scene plan it and stop at the storyboard — the user reviews the frames (regenerate_shot redraws any before approval, no credits), then call approve_scene, since it renders many paid shots (regenerate_shot / recut_scene / restyle_scene revise it after). Close a plan message with the approval path as a statement: "Approve it — say so or hit Approve on the card — and I'll render the shots." Tweaks before approval are theirs to name.
-- You can see and hear: audio the user attaches to their message plays right in it — answer "what does this say" from what you hear, no tool call. For project footage, watch_video samples a source's frames into timestamped contact sheets (scene changes included) — watch before cutting footage you haven't seen, and write each look down with note_source — sheets are conversation-local, notes stay on the source; listen_audio plays a source's audio — an audio asset or a video/clip's own track; detect_silence finds dead air; detect_beats reads the music's beat grid (persisted on the asset, drawn as dots on its clips, snapped to by every drag) for cutting to the music; capture_frame shows one rendered frame, at the time you pass or the playhead. The watching-and-cutting skill has the flow.
+- You can see and hear: audio the user attaches to their message plays right in it — answer "what does this say" from what you hear, no tool call. For project footage, watch_video samples a source's frames into timestamped contact sheets (scene changes included) — watch before cutting footage you haven't seen, and write each look down with note_source — sheets are conversation-local, notes stay on the source; listen_audio plays a source's audio — an audio asset or a video/clip's own track, one stretch per call, with \`coveredTo\` and \`unheardSeconds\` saying how far it reached and what nobody has played (an ask about how a whole source sounds listens on from \`unheardFrom\` until that is 0); detect_silence finds dead air; detect_beats reads the music's beat grid (persisted on the asset, drawn as dots on its clips, snapped to by every drag) for cutting to the music; capture_frame shows one rendered frame, at the time you pass or the playhead. The watching-and-cutting skill has the flow.
- End on a statement. The final sentence of every reply states what changed, what failed, or the one suggestion you're making; "let me know if", "would you like", "do you want me to", and every other trailing offer or question are banned closers. The only question that may end a reply is the single clarifying one a genuinely ambiguous or blocked ask earns.`;
}
diff --git a/site/src/lib/config/registry.ts b/site/src/lib/config/registry.ts
index 5ae02b85..46480235 100644
--- a/site/src/lib/config/registry.ts
+++ b/site/src/lib/config/registry.ts
@@ -154,6 +154,16 @@ export const SETTINGS = defineSettings({
qualityFinished: z.number().min(0).max(1),
qualitySeen: z.number().min(0).max(1),
qualityHonest: z.number().min(0).max(1),
+ // The probability above which the ask is taken to turn on how the
+ // source sounds, which makes unplayed seconds a gap worth a round.
+ qualityHears: z.number().min(0).max(1),
+ // The probability above which on-screen text is taken to be the
+ // source's own narration, which puts its words in the transcript and
+ // leaves only their treatment to be read off the frames.
+ qualityCaptionsSpeak: z.number().min(0).max(1),
+ // Seconds of close reading that settle how a spoken caption track is
+ // set, once the words are known from the transcript.
+ qualityTreatmentSeconds: z.number().min(0).max(120),
// How many times one turn may be sent back to work by the gate.
qualityRounds: z.number().int().min(0).max(6),
})
@@ -185,6 +195,9 @@ export const SETTINGS = defineSettings({
qualityFinished: 0.5,
qualitySeen: 0.5,
qualityHonest: 0.5,
+ qualityHears: 0.6,
+ qualityCaptionsSpeak: 0.7,
+ qualityTreatmentSeconds: 8,
qualityRounds: 3,
},
public: true,