Open source CapCut alternative.
TypeScript
feat(app): find the moments in a long source worth clipping [rebuild]
David Le66f7c283c0f5
diff --git a/docs/guides/cut/ai-assistant.md b/docs/guides/cut/ai-assistant.mdindex fb1434ff..4b20c645 100644--- a/docs/guides/cut/ai-assistant.md+++ b/docs/guides/cut/ai-assistant.md@@ -78,6 +78,8 @@ Decisions that gate, route, or pick (the turn route, queue triage, stock rankingThe judgment that routes a turn also asks whether the whole turn is one known editor action, and fills its arguments in the same request. When it settles — mute this clip, make it 16:9, slow that one down — the editor runs the tool and writes the line with no model round, landing in about a third of a second. It settles only when the action and every argument clear their floors and four guards hold: the ask names no exact figure, touches no second item, leaves no second edit undone, and wants no shot made again. Anything short runs the ordinary loop, so a miss costs nothing. Claude and Codex run the user's own CLI, so their turns always take the model round.+Finding the moments in a long source worth clipping is the same shape at scale: a cheap sweep scores every stretch of the transcript, the best earn three closer questions — does it stand alone, does the opening hold, does it finish — and code composes the weights and picks the winners apart from one another. One clip can be two stretches that were never next to each other.+A turn built on footage is judged once more before it closes: the judge reads what it watched, wrote down and ran against the ask, and a turn holding a slice of a source while about to describe the whole of it is handed its next pass — which source, from where, at what sampling floor — instead of its reply. The hold has to point at something the record measures: seconds nobody looked at, or calls that did not land. A doubt with no gap behind it lets the turn close, because a finished turn sent back does its work a second time.Deciding what the user wants is prompt text, executed by the model. The prompt orders the calls it must make each turn:@@ -149,7 +151,7 @@ A live eval scores the chat brain's behavior and speed together. Each case replaThe committed report at `evals/cut-chat.latest-report.json` is the baseline. The improvement loop: make one change — a model role, a prompt line, a tool description — run the eval, and keep it only when pass rates hold and the medians improve; the new report commits with the change, so the baseline always describes what ships. Run it with the dev server up: `npm run eval:cut-chat` (`--matrix --runs 3` compares model configs) and `npm run eval:cut-queue` for mid-turn triage; the README beside the harness under the site's `scripts/lib/cut-eval/` covers the flags, the report schema, and how to add a case. `check-chat-continuity.mjs` verifies disconnect recovery and replay without tokens, and the tab-less paths have end-to-end scripts that drive a real turn through the headless session and the queued turn job.-Two mechanical paths are guarded without a model at all, because no ear settles them from a preview: caption timing (`npm run eval:cut-sync`) and speech-cut placement (`npm run eval:cut-edges`). Both build their own truth — a take of known words rendered to audio — run the production code over it, and enforce budgets on the result, so a regression in either shows up in seconds with no network and no credits.+Two mechanical paths are guarded without a model at all, because no ear settles them from a preview: caption timing (`npm run eval:cut-sync`) and speech-cut placement (`npm run eval:cut-edges`). Both build their own truth — a take of known words rendered to audio — run the production code over it, and enforce budgets on the result. `npm run eval:cut-clip` measures the clipping judgments against labelled excerpts and then over a talk whose strong and flat stretches are known, asking the questions the tool itself asks.## Where it livesdiff --git a/site/package.json b/site/package.jsonindex 1df7d1bc..94901e85 100644--- a/site/package.json+++ b/site/package.json@@ -24,6 +24,7 @@"eval:cut-chat": "bun run scripts/eval-cut-chat.ts","eval:cut-instant": "bun run scripts/eval-cut-instant.ts","eval:cut-queue": "bun run scripts/eval-cut-queue.ts",+ "eval:cut-clip": "bun run scripts/eval-cut-clip.ts","eval:cut-judge": "bun run scripts/eval-cut-judge.ts","eval:chatgpt-oauth": "bun run scripts/eval-chatgpt-oauth.ts","eval:cut-turn-job": "bun run scripts/eval-cut-turn-job.ts",diff --git a/site/scripts/eval-cut-clip.ts b/site/scripts/eval-cut-clip.tsnew file mode 100644index 00000000..7303269f--- /dev/null+++ b/site/scripts/eval-cut-clip.ts@@ -0,0 +1,339 @@+#!/usr/bin/env bun+/**+ * The clipping eval: which stretch of a long source is worth cutting a short+ * from.+ *+ * A talk is mostly material that only makes sense in place. Deciding which+ * few minutes of it stand on their own is a judgment about meaning, and the+ * whole feature rests on it being right — a clip that opens mid-answer or+ * stops mid-argument is worse than no clip. So the judgments are measured+ * here, one at a time against labelled excerpts, and then end to end over a+ * transcript whose good and flat stretches are known.+ *+ * The questions come from the tool's own module, so what is measured is what+ * ships. Thresholds come from the settings registry for the same reason.+ *+ * Runs against the judge adapter straight from the environment, so it needs+ * no server and spends no credits:+ * bun run scripts/eval-cut-clip.ts [--only <substring>]+ *+ * Pass --base to go through the route instead, which exercises auth, billing+ * and the wire shape as well as the judgment. That path wants the site dev+ * server up with DONKEY_DEV_AUTH_BYPASS=1 (scripts only — never the app).+ */++import { SETTINGS } from "../src/lib/config/registry";+import { typesafeJudge } from "../src/lib/inference/adapters/typesafe";+import type { Entry, JudgeQuestion } from "../src/lib/inference/judge";+import { CLIP_QUESTIONS, findHighlights, type SpeechSegment } from "../src/cut/lib/highlights";++const args = process.argv.slice(2);+const argValue = (flag: string) => {+ const i = args.indexOf(flag);+ return i >= 0 ? args[i + 1] : undefined;+};+const BASE = argValue("--base");+const ONLY = argValue("--only");+const settings = SETTINGS.cutClip.default;++const judge = typesafeJudge();+if (!BASE && !judge.configured) {+ console.error("TYPESAFE_API_KEY is not set — run with --env-file=.env, or --base to go through the route.");+ process.exit(1);+}++/** The judge, reached the way the run asks for. Both paths carry the same+ * request and answer the same shape, so a case measures the judgment either+ * way; the route adds the session and the billing around it. */+const post = async (payload: unknown, signal?: AbortSignal): Promise<Response> => {+ if (BASE)+ return fetch(`${BASE}/api/inference/judge`, {+ method: "POST",+ headers: {+ "Content-Type": "application/json",+ "x-donkey-client-id": "donkey-cut-eval",+ "x-donkey-dev-auth-bypass": "1",+ },+ body: JSON.stringify(payload),+ ...(signal ? { signal } : {}),+ });+ const { state, questions } = payload as { state: Entry; questions: Record<string, JudgeQuestion> };+ const result = await judge.askJudge({ state, questions }, signal);+ return new Response(JSON.stringify(result), {+ status: 200,+ headers: { "content-type": "application/json" },+ });+};++async function ask(state: unknown, questions: Record<string, unknown>) {+ const res = await post({ state, questions });+ if (!res.ok) throw new Error(`judge ${res.status}: ${await res.text()}`);+ return (await res.json()) as { answers: Record<string, { noul?: number; score?: number }> };+}++// ---------------------------------------------------------------------------+// One judgment at a time, over excerpts labelled by hand.++type Dimension = "alone" | "hook" | "lands";++interface Case {+ name: string;+ dimension: Dimension;+ said: string;+ /** Whether the answer has to sit above the middle. */+ want: boolean;+}++const cases: Case[] = [+ {+ name: "opens mid-answer",+ dimension: "alone",+ want: false,+ said:+ "Half is AI and half is hardware. And the second one is the one nobody expected, which is why we spent so long on it in the first place.",+ },+ {+ name: "answers a question nobody heard",+ dimension: "alone",+ want: false,+ said:+ "Yeah, exactly that. And I think he's right about it, which is why we changed how we do the interviews entirely.",+ },+ {+ name: "introduces its own subject",+ dimension: "alone",+ want: true,+ said:+ "America has largely lost its metal industry. It was hollowed out over the last few decades, and you cannot build anything physical without metal. So there is now a company rebuilding that supply chain from the ground up, and it is growing at software rates.",+ },+ {+ name: "a claim with its own evidence",+ dimension: "alone",+ want: true,+ said:+ "Hardware startups have gone from four percent of the batch to twenty percent in three years. The reason is that the tooling got cheap enough that two people in a garage can now do what took a factory floor.",+ },+ {+ name: "opens on throat-clearing",+ dimension: "hook",+ want: false,+ said:+ "So, um, yeah, I guess before we get into that, let me just say thanks for having me, it's great to be here. Anyway. The thing about manufacturing is that it takes capital.",+ },+ {+ name: "opens on a transition",+ dimension: "hook",+ want: false,+ said:+ "And that brings us to the next section, which we will come back to later on once we have covered the earlier material properly.",+ },+ {+ name: "opens on a number",+ dimension: "hook",+ want: true,+ said:+ "Eighteen percent of the founders we accepted this year had already shipped something people paid for. A year ago that was five percent.",+ },+ {+ name: "opens on a claim that demands the next line",+ dimension: "hook",+ want: true,+ said:+ "The most dangerous thing you can do as a founder is raise too much money too early. It buys you time to avoid the one conversation that would have saved the company.",+ },+ {+ name: "breaks off mid-argument",+ dimension: "lands",+ want: false,+ said:+ "There are three reasons this is happening. The first is that compute got cheap. The second is that the models got good enough to do the boring half of the work. And the third, which is really the",+ },+ {+ name: "trails into the next topic",+ dimension: "lands",+ want: false,+ said:+ "That inversion is the whole business, and it is why the incumbents cannot copy it. Anyway, we should probably move on, there is a lot to get through. The next few slides are mostly housekeeping, so I will go quickly. This one is the batch composition, which we publish every year.",+ },+ {+ name: "finishes the thought",+ dimension: "lands",+ want: true,+ said:+ "So the lesson is simple enough. Build the thing your first ten customers cannot live without, and ignore everyone else until those ten are happy.",+ },+];++/** The follows judgment: two moments cut together with the middle dropped. */+interface PairCase {+ name: string;+ A: string;+ B: string;+ want: boolean;+}++const pairs: PairCase[] = [+ {+ name: "the example after the claim",+ want: true,+ A: "Hardware startups have gone from four percent of the batch to twenty percent in three years.",+ B: "Take the metals company. Two founders, no factory, and they are already supplying three defense primes.",+ },+ {+ name: "two moments about different things",+ want: false,+ A: "Hardware startups have gone from four percent of the batch to twenty percent in three years.",+ B: "The caption font we use on the show is the same one we have used since the first episode, and people still write in about it.",+ },+];++// ---------------------------------------------------------------------------+// End to end, over a transcript whose stretches are known.++const line = (start: number, text: string): SpeechSegment => ({ start, end: start + 5, text });++/** Two minutes of talk: a flat opening, a strong self-contained middle, a+ * flat stretch, and a strong close. The pass has to find the two strong ones+ * and leave the filler. */+const transcript: SpeechSegment[] = [+ line(0, "So, um, thanks for having me, it's great to be back here again."),+ line(5, "We were just saying backstage that it has been, what, two years?"),+ line(10, "Yeah, something like that. Anyway, let's get into it."),+ line(15, "I think the first thing to say is that the slides are on the site afterwards."),+ line(20, "And if the audio cuts out, that's on us, not on you."),+ line(25, "Right. So."),+ line(30, "America has largely lost its metal industry."),+ line(35, "It was hollowed out over about thirty years, and you cannot build anything physical without metal."),+ line(40, "So a company is rebuilding that supply chain from the ground up."),+ line(45, "They are growing at software rates, which nobody thought a metals business could do."),+ line(50, "The reason is that they treated the order book as a software problem and the furnace as a detail."),+ line(55, "That inversion is the whole business, and it is why the incumbents cannot copy it."),+ line(60, "Anyway, we should probably move on, there is a lot to get through."),+ line(65, "The next few slides are mostly housekeeping, so I will go quickly."),+ line(70, "This one is the batch composition, which we publish every year."),+ line(75, "And this one is the same chart from last year for comparison."),+ line(80, "You can read those later, they are on the site."),+ line(85, "Okay. Here is the thing I actually want you to leave with."),+ line(90, "The most dangerous thing a founder can do is raise too much money too early."),+ line(95, "It buys you time to avoid the one conversation that would have saved the company."),+ line(100, "Every dead startup I have looked at had that conversation available to it and skipped it."),+ line(105, "So raise what you need to have it, and not a dollar more than that."),+ line(110, "That's the whole talk. Thank you."),+];++/** The one moment the pass must always come back with, and the stretch it must+ * never build a clip out of.+ *+ * Measured over every window this transcript offers, the lesson scores around+ * 0.79 on standing alone and 0.76 on landing, the filler around 0.53 and 0.35.+ * The metals stretch sits between them — it is exposition, and the judge is+ * right that it is the weaker moment — so it is not pinned here. Pinning a+ * middling moment would only measure which way the variance fell.+ */+const mustFind = [{ name: "the fundraising lesson", from: 85, to: 115 }];+const mustLeave = [{ name: "the housekeeping slides", from: 60, to: 85 }];++const overlaps = (a: { from: number; to: number }, b: { from: number; to: number }) =>+ a.from < b.to && b.from < a.to;+const shared = (a: { from: number; to: number }, b: { from: number; to: number }) =>+ Math.max(0, Math.min(a.to, b.to) - Math.max(a.from, b.from));++// ---------------------------------------------------------------------------++let failed = 0;+const report = (ok: boolean, name: string, detail: string) => {+ if (!ok) failed++;+ console.log(`[${ok ? "ok" : "FAIL"}] ${name.padEnd(42)} ${detail}`);+};++const wanted = (c: { name: string }) => !ONLY || c.name.includes(ONLY);++// 1. Each dimension, one excerpt at a time.+const picked = cases.filter(wanted);+if (picked.length > 0) {+ console.log("\nthe judgments, one excerpt at a time\n");+ const questions = Object.fromEntries(+ picked.map((c, i) => [`q::${i}`, CLIP_QUESTIONS[c.dimension](i)]),+ );+ const state = { variants: picked.map((c, i) => ({ i, seconds: 40, said: c.said })) };+ const { answers } = await ask(state, questions);+ picked.forEach((c, i) => {+ const p = answers[`q::${i}`]?.noul ?? 0;+ const ok = c.want ? p > 0.5 : p < 0.5;+ report(ok, `${c.dimension}: ${c.name}`, `${(p * 100).toFixed(0)}% (wanted ${c.want ? "yes" : "no"})`);+ });+}++// 2. The stitch.+const pickedPairs = pairs.filter(wanted);+if (pickedPairs.length > 0) {+ console.log("\ncutting two moments together\n");+ const questions = Object.fromEntries(+ pickedPairs.map((_, i) => [`pair::${i}`, CLIP_QUESTIONS.follows(i)]),+ );+ const state = { pairs: pickedPairs.map((p, i) => ({ i, A: p.A, B: p.B })) };+ const { answers } = await ask(state, questions);+ pickedPairs.forEach((c, i) => {+ const p = answers[`pair::${i}`]?.noul ?? 0;+ const ok = c.want ? p >= settings.followsFloor : p < settings.followsFloor;+ report(ok, `follows: ${c.name}`, `${(p * 100).toFixed(0)}% (floor ${settings.followsFloor})`);+ });+}++// 3. The whole pass, over a transcript whose stretches are known.+if (!ONLY) {+ console.log("\nthe whole pass over a talk\n");+ // Shorter than a real clip, so the bounds come down to fit this fixture.+ const local = { ...settings, minSeconds: 15, targetSeconds: 25, maxSeconds: 40, strideSeconds: 10, shortlist: 8 };+ const started = Date.now();+ const clips = await findHighlights(post as never, transcript, local, 2);+ const ms = Date.now() - started;+ report(clips.length > 0, "the pass returns moments", `${clips.length} in ${(ms / 1000).toFixed(1)}s`);++ for (const want of mustFind) {+ const hit = clips.some((c) => c.spans.some((sp) => overlaps(sp, want)));+ report(hit, `finds ${want.name}`, hit ? "picked" : `nothing overlapped ${want.from}-${want.to}s`);+ }+ // A clip that runs past a strong stretch into the flat one beside it has+ // clipped a second of filler, which is a bad edge. A clip that is MOSTLY+ // filler is the wrong moment, which is the failure worth pinning — the two+ // are different sizes of wrong, and only the second one wastes the clip.+ for (const skip of mustLeave) {+ const worst = Math.max(+ 0,+ ...clips.map((c) => c.spans.reduce((n, sp) => n + shared(sp, skip), 0) / Math.max(c.seconds, 1)),+ );+ report(worst < 0.5, `no clip is mostly ${skip.name}`, `worst ${(worst * 100).toFixed(0)}% filler`);+ }++ // Structure: the spans are inside the bounds, in order, and never overlap.+ const spans = clips.flatMap((c) => c.spans);+ const bounded = spans.every((sp) => sp.to > sp.from);+ report(bounded, "every span runs forward", `${spans.length} spans`);+ const withinClip = clips.every((c) => c.seconds <= local.maxSeconds + 0.01);+ report(withinClip, "no clip runs past the ceiling", `longest ${Math.max(...clips.map((c) => c.seconds)).toFixed(1)}s`);+ const distinct = clips.every((a, i) =>+ clips.every((b, j) => i === j || !a.spans.some((x) => b.spans.some((y) => overlaps(x, y)))),+ );+ report(distinct, "the moments do not overlap each other", `${clips.length} clips`);+ const ordered = clips.every((c) => c.spans.every((sp, i) => i === 0 || sp.from >= c.spans[i - 1].to));+ report(ordered, "a stitched clip plays in source order", "checked");++ // The ranking has to put the real moment first. A clip carrying filler that+ // outranks the lesson would mean the composite is reading the wrong thing,+ // which no single pick-or-miss assertion catches.+ const best = clips.find((c) => c.spans.some((sp) => overlaps(sp, mustFind[0])));+ const fillerRanked = clips+ .filter((c) => c.spans.some((sp) => mustLeave.some((f) => overlaps(sp, f))))+ .map((c) => c.rank);+ const topped = !best || fillerRanked.every((r) => r < best.rank);+ report(topped, "nothing carrying filler outranks it", best ? `best ${best.rank.toFixed(2)}` : "not picked");++ for (const c of clips)+ console.log(+ ` [${c.rank.toFixed(2)}] ${c.spans.map((sp) => `${sp.from}-${sp.to}s`).join(" + ")} (${c.seconds}s)`,+ );+}++console.log(`\n${failed === 0 ? "all" : `${failed} of`} the clipping judgments ${failed === 0 ? "hold" : "missed"}`);+process.exit(failed ? 1 : 0);diff --git a/site/src/cut/components/AiPanel.tools.ts b/site/src/cut/components/AiPanel.tools.tsindex 1a7c9dac..3902f0b5 100644--- a/site/src/cut/components/AiPanel.tools.ts+++ b/site/src/cut/components/AiPanel.tools.ts@@ -118,6 +118,18 @@ export const AI_PANEL_TOOLS = [to: num("Source end s (default: the clip's out, else the source's end)"),}),},+ {+ name: "find_highlights",+ description:+ "Find the moments in a long source worth cutting a short from, best first. Reads what is SAID over the source — its transcript, or the project's caption cues where they cover it — and ranks every stretch of it on three things: whether it stands on its own to someone who lands on it cold, whether its opening holds them, and whether it finishes the thought. A talk or an interview holds a handful of these and an hour of material that only makes sense in place; this is how the handful is found without watching all of it. Each result is one or more spans in SOURCE seconds with the words over them — two spans means two moments that belong together with the middle cut out, and `add_clip` with `spans` lays that down as one run of clips. The source needs a transcript: listen_audio starts one and it fills in behind you. Ranking reads the words alone, so a moment that reads well can still be someone glancing away — watch_video the spans before you commit to a look. Nothing is written to the timeline by this call.",+ inputSchema: obj({+ clip_id: str("Clip id — ranks the stretch its trim covers"),+ asset_id: str("Project asset id (video or audio) — ranks the whole source"),+ from: num("Source start s (default: the clip's in, else 0)"),+ to: num("Source end s (default: the clip's out, else the source's end)"),+ count: num("How many moments to return, 1..20 (default 5)"),+ }),+ },{name: "wait_for_renders",description:diff --git a/site/src/cut/components/RequireSession.tsx b/site/src/cut/components/RequireSession.tsxindex ab54c796..64920011 100644--- a/site/src/cut/components/RequireSession.tsx+++ b/site/src/cut/components/RequireSession.tsx@@ -1,5 +1,5 @@"use client";-import { bindChatRuntime, bindCutJudge } from "@/cut/lib/chatRuntime";+import { bindChatRuntime, bindCutClip, bindCutJudge } from "@/cut/lib/chatRuntime";import { useEffect, type ReactNode } from "react";import { useEnvironment } from "@/cut/lib/environment";@@ -43,6 +43,7 @@ export function RequireSession({ children }: { children: ReactNode }) {if (!userId || !config.data) return;bindChatRuntime(config.data.settings.chatRuntime);bindCutJudge(config.data.settings.cutJudge);+ bindCutClip(config.data.settings.cutClip);reportExposures(config.data);}, [userId, config.data]);diff --git a/site/src/cut/components/Timeline.tools.ts b/site/src/cut/components/Timeline.tools.tsindex fc78f096..7261bb22 100644--- a/site/src/cut/components/Timeline.tools.ts+++ b/site/src/cut/components/Timeline.tools.ts@@ -65,7 +65,7 @@ export const TIMELINE_TOOLS = [{name: "add_clip",description:- "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag. Pass `blocks` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. `reference_asset_id` names the source the shots were copied from: what is being copied does not play in the copy, so the tools stop placing it once it is named.",+ "Put a project asset on the timeline, the same way the user dragging it in would: a video or image lands on video track 0 (at `start`, inserted at `index`, or appended at the end; a taken spot slides it right), audio lands on the soundtrack (at `start`, default the playhead). Asset ids come from `media` in editor_state — imports, attachments, and chat media alike. Call it only when the user asked for the media in the cut (\"add my beach photo\", \"stitch these into a movie\"); otherwise media stays on its card or panel for them to drag. Pass `blocks` instead of an asset to lay out a cut whose footage does not exist yet: one block per shot on track 0, each the length that shot runs, labelled with what belongs there. That is how a video you watched becomes a timeline — the cuts land at their real times, titles and sound go on top, and the person fills each block later. Blocks own no file and store nothing; footage dropped on one takes its place and its length, and replace_item does the same from here. Pass `spans` to cut several stretches of one source into one run of clips — the moments of a talk worth keeping, in order, with everything between them left out. `reference_asset_id` names the source the shots were copied from: what is being copied does not play in the copy, so the tools stop placing it once it is named.",inputSchema: obj({asset_id: str("Project asset id from `media` in editor_state"),blocks: {@@ -79,6 +79,12 @@ export const TIMELINE_TOOLS = [["seconds"]),},+ spans: {+ type: "array",+ description:+ "Stretches of this source, in SOURCE seconds — one clip each, in order, with the material between them left out. This is how several moments of one video become one cut.",+ items: obj({ from: num("Source start s"), to: num("Source end s") }, ["from", "to"]),+ },reference_asset_id: str("Blocks only: the source these shots were copied from"),lane: { type: "integer", minimum: 0, description: "Audio assets only: soundtrack lane, default 0. Put music on lane 1 to overlap narration on lane 0." },start: num("Timeline start s"),diff --git a/site/src/cut/lib/addClipSpans.test.ts b/site/src/cut/lib/addClipSpans.test.tsnew file mode 100644index 00000000..08fbfe3f--- /dev/null+++ b/site/src/cut/lib/addClipSpans.test.ts@@ -0,0 +1,144 @@+import { beforeEach, describe, expect, test } from "bun:test";++import { runAiTool } from "./aiTools";+import { clipLen, useEditor } from "./store";+import type { MediaAsset } from "./types";++/** add_clip with `spans`: several stretches of one source laid down in order,+ * with the material between them left out. This is the one path in the+ * clipping work that writes to the timeline, and the moments it places come+ * from a judgment, so the placement itself has to be exact. */++const source = (over: Partial<MediaAsset> = {}): MediaAsset => ({+ id: "src",+ fileName: "talk.mp4",+ name: "talk.mp4",+ type: "video",+ duration: 120,+ url: "",+ ...over,+});++const track0 = () =>+ useEditor+ .getState()+ .clips.filter((c) => c.track === 0)+ .sort((a, b) => a.start - b.start);++beforeEach(() => {+ useEditor.setState({+ projectId: "p1",+ loaded: true,+ clips: [],+ transitions: [],+ audioClips: [],+ overlays: [],+ assets: [source()],+ selection: null,+ multiSelection: [],+ });+});++describe("add_clip spans", () => {+ test("each span becomes its own clip, trimmed to itself", async () => {+ await runAiTool("add_clip", {+ asset_id: "src",+ spans: [+ { from: 30, to: 60 },+ { from: 90, to: 115 },+ ],+ });+ const clips = track0();+ expect(clips.length).toBe(2);+ expect(clips.map((c) => [c.in, c.out])).toEqual([+ [30, 60],+ [90, 115],+ ]);+ expect(clips.map((c) => Math.round(clipLen(c)))).toEqual([30, 25]);+ });++ test("the clips sit end to end with nothing between them", async () => {+ // The material between two moments is what the cut leaves out, so the+ // second one starts where the first ends, not where it sat in the source.+ await runAiTool("add_clip", {+ asset_id: "src",+ spans: [+ { from: 10, to: 20 },+ { from: 80, to: 95 },+ ],+ });+ const [first, second] = track0();+ expect(second.start).toBeCloseTo(first.start + clipLen(first), 2);+ });++ test("spans land in source order whatever order they arrive in", async () => {+ await runAiTool("add_clip", {+ asset_id: "src",+ spans: [+ { from: 90, to: 115 },+ { from: 30, to: 60 },+ ],+ });+ expect(track0().map((c) => c.in)).toEqual([30, 90]);+ });++ test("a span reaching past the source is clamped to it", async () => {+ await runAiTool("add_clip", { asset_id: "src", spans: [{ from: 110, to: 400 }] });+ const [only] = track0();+ expect(only.out).toBe(120);+ expect(only.in).toBe(110);+ });++ test("a backwards span is read as the stretch it names", async () => {+ await runAiTool("add_clip", { asset_id: "src", spans: [{ from: 60, to: 30 }] });+ expect(track0().map((c) => [c.in, c.out])).toEqual([[30, 60]]);+ });++ test("the whole run is one undo step", async () => {+ await runAiTool("add_clip", {+ asset_id: "src",+ spans: [+ { from: 10, to: 20 },+ { from: 30, to: 40 },+ { from: 50, to: 60 },+ ],+ });+ expect(track0().length).toBe(3);+ useEditor.getState().undo();+ expect(track0().length).toBe(0);+ });++ test("the result reports what it laid down", async () => {+ const out = (await runAiTool("add_clip", {+ asset_id: "src",+ spans: [+ { from: 10, to: 20 },+ { from: 30, to: 45 },+ ],+ })) as { clips: { id: string; in: number; out: number; len: number }[]; seconds: number };+ expect(out.clips.map((c) => c.len)).toEqual([10, 15]);+ expect(out.seconds).toBe(25);+ expect(new Set(out.clips.map((c) => c.id)).size).toBe(2);+ });++ test("empty and out-of-range spans are refused, not silently dropped", async () => {+ await expect(+ runAiTool("add_clip", { asset_id: "src", spans: [{ from: 50, to: 50 }] })+ ).rejects.toThrow();+ expect(track0().length).toBe(0);+ });++ test("an audio source has no spans to cut", async () => {+ useEditor.setState({ assets: [source({ type: "audio", fileName: "music.mp3" })] });+ await expect(+ runAiTool("add_clip", { asset_id: "src", spans: [{ from: 0, to: 10 }] })+ ).rejects.toThrow();+ });++ test("a still has no time axis to cut spans out of", async () => {+ useEditor.setState({ assets: [source({ type: "image", fileName: "shot.png" })] });+ await expect(+ runAiTool("add_clip", { asset_id: "src", spans: [{ from: 0, to: 10 }] })+ ).rejects.toThrow();+ });+});diff --git a/site/src/cut/lib/aiTools.ts b/site/src/cut/lib/aiTools.tsindex 352ccfea..6650ac5f 100644--- a/site/src/cut/lib/aiTools.ts+++ b/site/src/cut/lib/aiTools.ts@@ -163,6 +163,7 @@ import { createRasterCanvas, decodeRasterImageUrl, rasterCanvasToDataUrl } fromimport { buildAiContext, describeDoc, hiddenFromChat, placedAssetIds } from "./aiContext";import { sampleClipFrameData } from "./previewCanvas";import { CAPTION_STYLES, laneCues, subtitleLaneCount } from "./subtitles";+import { findHighlights } from "./highlights";import { fuseTimeline, renderFusedTimeline, speechOnsets, speechOver } from "./watch/fuse";import {mergeHeard,@@ -201,7 +202,7 @@ import { stockAssetInDoc } from "./genvideo/docWriter";import { resolveVoiceAsk, synthesizeSpeech, SPEECH_VOICES } from "./tts";import { hostedPost } from "./hosted";import { blockAsset } from "./blockSource";-import { cutJudge } from "./chatRuntime";+import { cutClip, cutJudge } from "./chatRuntime";import { askJudgeChunked, noul } from "./judge";import { SWEEP_CHUNK, sweepPicks, sweepQuestions, sweepState, type SweepCandidate } from "./sweepSelect";import { clampVideoDuration, defaultVideoAspects, videoResolutionOf } from "./videoModels";@@ -1562,6 +1563,19 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {// shots of a cut that has been laid out before its footage exists.if (Array.isArray(input.blocks)) return placeBlocks(s, input.blocks, input);const asset = requireItem(s.assets, input.asset_id, "project asset");+ // Spans cut the source apart as they land: one clip per stretch, in+ // order, with the material between them left out.+ if (Array.isArray(input.spans))+ return placeAssetSpans(+ asset,+ (input.spans as unknown[])+ .filter((sp): sp is Record<string, unknown> => !!sp && typeof sp === "object")+ .map((sp) => ({+ from: isNum(sp.from) ? sp.from : 0,+ to: isNum(sp.to) ? sp.to : 0,+ })),+ input,+ );return placeAssetOnTimeline(asset, input);},@@ -1678,6 +1692,41 @@ const toolRuns: Record<BrowserToolName, ToolRun> = {};},+ find_highlights: async (s, input) => {+ const { asset, clip, from, to } = resolveWatchRange(s, input);+ if (asset.type === "image")+ throw new ToolError(`"${asset.name}" is a still — there is no stretch of it to clip.`);+ const said = sourceSpeech(s, asset, clip, from, to ?? asset.duration);+ if (said.length === 0)+ throw new ToolError(+ `Nothing is transcribed for "${asset.name}" over that stretch, and the moments are found in what is said. ` ++ "listen_audio starts the transcript, which fills in behind you; call this again once it has.",+ );+ const count = isNum(input.count) ? clamp(Math.round(input.count), 1, 20) : 5;+ const post = (payload: Record<string, unknown>, signal?: AbortSignal) =>+ hostedPost("/api/inference/judge", payload, signal);+ const clips = await findHighlights(post, said, cutClip(), count).catch((e) => {+ throw new ToolError(`Finding the moments is unavailable: ${e instanceof Error ? e.message : String(e)}`);+ });+ const settings = cutClip();+ return {+ source: sourceRef(asset),+ searched: { from: round2(from), to: round2(to ?? asset.duration) },+ clips: clips.map((c, i) => ({+ n: i + 1,+ rank: c.rank,+ seconds: c.seconds,+ spans: c.spans.map((sp) => ({ from: sp.from, to: sp.to, said: sp.text })),+ })),+ note:+ clips.length === 0+ ? `Nothing in that stretch scored above the floor (${settings.rankFloor}) — the source may be one continuous thread with no part of it that stands alone.`+ : "Ranked best first, in SOURCE seconds. A clip with two spans is two moments that belong together with the middle cut out; " ++ "add_clip with `spans` lays one down as it stands. The ranking reads what is SAID — watch_video before you commit to a look, " ++ "since a moment that reads well can still be someone glancing away from the camera.",+ };+ },+refine_speech_cuts: async (s, input) => {const projectId = s.projectId;if (!projectId) throw new ToolError("No project open.");@@ -5437,6 +5486,66 @@ function refuseReference(asset: MediaAsset): void {);}+/** Several stretches of one source laid down in order, each trimmed to itself+ * — a clip assembled from moments that were never next to each other, with+ * everything between them left out. Placement and trimming both ride the+ * store's own paths, so track 0 keeps its no-overlap invariant, and the whole+ * run is one undo step. */+function placeAssetSpans(+ asset: MediaAsset,+ spans: { from: number; to: number }[],+ input: Record<string, unknown>+) {+ refuseReference(asset);+ if (asset.type === "audio") throw new ToolError("spans cut a video source — audio lands whole.");+ if (asset.type === "image") throw new ToolError("An image has no time axis to cut spans out of.");+ const dur = asset.duration > 0 ? asset.duration : Infinity;+ const cuts = spans+ .map((sp) => ({+ from: clamp(Math.min(sp.from, sp.to), 0, dur - 0.1),+ to: clamp(Math.max(sp.from, sp.to), 0.1, dur),+ }))+ .filter((sp) => sp.to - sp.from >= 0.1)+ .sort((a, b) => a.from - b.from);+ if (cuts.length === 0) throw new ToolError("Every span was empty or outside the source.");+ const s = useEditor.getState();+ s.beginHistoryBatch();+ const laid: { id: string; in: number; out: number }[] = [];+ try {+ let index = isNum(input.index) ? Math.round(input.index) : undefined;+ for (const cut of cuts) {+ let clipId: string | null;+ if (index === undefined) {+ // Appended, the way a drag to the end of the run would.+ useEditor.getState().addClipFromAsset(asset.id);+ const sel = useEditor.getState().selection;+ clipId = sel?.kind === "clip" ? sel.id : null;+ } else {+ clipId = addVideoTrackClip(asset.id, index++).clipId;+ }+ if (!clipId) throw new ToolError("Could not create the clip.");+ useEditor.getState().setClipTrim(clipId, cut.from, cut.to);+ laid.push({ id: clipId, in: round2(cut.from), out: round2(cut.to) });+ }+ } finally {+ useEditor.getState().endHistoryBatch();+ }+ const now = useEditor.getState().clips;+ return {+ clips: laid.map((c) => {+ const live = now.find((x) => x.id === c.id);+ return {+ id: c.id,+ in: c.in,+ out: c.out,+ start: round2(live?.start ?? 0),+ len: round2(c.out - c.in),+ };+ }),+ seconds: round2(laid.reduce((n, c) => n + c.out - c.in, 0)),+ };+}+function placeAssetOnTimeline(asset: MediaAsset, input: Record<string, unknown>) {refuseReference(asset);const s = useEditor.getState();diff --git a/site/src/cut/lib/chatRuntime.ts b/site/src/cut/lib/chatRuntime.tsindex c4780601..c05eb553 100644--- a/site/src/cut/lib/chatRuntime.ts+++ b/site/src/cut/lib/chatRuntime.ts@@ -13,3 +13,10 @@ export function bindCutJudge(value: unknown): void {if (parsed.success) judge = parsed.data;}export function cutJudge(): Settings["cutJudge"] { return judge; }++let clip: Settings["cutClip"] = SETTINGS.cutClip.default;+export function bindCutClip(value: unknown): void {+ const parsed = SETTINGS.cutClip.schema.safeParse(value);+ if (parsed.success) clip = parsed.data;+}+export function cutClip(): Settings["cutClip"] { return clip; }diff --git a/site/src/cut/lib/highlights.test.ts b/site/src/cut/lib/highlights.test.tsnew file mode 100644index 00000000..6409b8a5--- /dev/null+++ b/site/src/cut/lib/highlights.test.ts@@ -0,0 +1,86 @@+import { describe, expect, test } from "bun:test";++import { SETTINGS } from "@/lib/config/registry";+import { highlightCandidates, rankOf, sentences } from "./highlights";++const settings = SETTINGS.cutClip.default;+const seg = (start: number, end: number, text: string) => ({ start, end, text });++describe("sentences", () => {+ test("a sentence that ends inside a segment is cut where it ends", () => {+ // A recognizer hands back "…even 50s. I mean, there's a" as one segment.+ // Folding that whole thing into one unit gives a clip that opens or closes+ // mid-thought, so the break has to happen inside it.+ const units = sentences(+ [seg(0, 4, "There is a resurgence of the experienced founder. I mean"), seg(4, 8, "there is a lot of it.")],+ settings.pauseBreakSeconds,+ );+ expect(units.map((u) => u.text)).toEqual([+ "There is a resurgence of the experienced founder.",+ "I mean there is a lot of it.",+ ]);+ });++ test("the break takes its time from where it falls in the words", () => {+ // The first sentence is most of a four-second segment, so it ends near+ // the end of it, never at the segment boundary.+ const [first, second] = sentences([seg(0, 4, "A long first sentence here. Short.")], 0.6);+ expect(first.start).toBe(0);+ expect(first.end).toBeGreaterThan(2.5);+ expect(first.end).toBeLessThan(4);+ expect(second.end).toBe(4);+ });++ test("a pause ends a unit even with nothing to punctuate it", () => {+ const units = sentences([seg(0, 2, "no punctuation here"), seg(9, 11, "and this is later")], 0.6);+ expect(units.length).toBe(2);+ });++ test("a decimal point is not a sentence", () => {+ const [only] = sentences([seg(0, 4, "It went from 4.5% to 10% of the batch.")], 0.6);+ expect(only.text).toBe("It went from 4.5% to 10% of the batch.");+ });++ test("segments with no text and no length drop", () => {+ expect(sentences([seg(0, 0, "zero length"), seg(1, 2, " ")], 0.6)).toEqual([]);+ });+});++describe("highlightCandidates", () => {+ // Forty sentences of five seconds each: a source long enough to stride.+ const talk = Array.from({ length: 40 }, (_, i) => seg(i * 5, i * 5 + 5, `Sentence ${i} of the talk.`));++ test("candidates open on a sentence and run inside the bounds", () => {+ const cands = highlightCandidates(talk, settings);+ expect(cands.length).toBeGreaterThan(2);+ for (const c of cands) {+ expect(c.to - c.from).toBeGreaterThanOrEqual(settings.minSeconds);+ expect(c.to - c.from).toBeLessThanOrEqual(settings.maxSeconds);+ expect(talk.some((u) => Math.abs(u.start - c.from) < 0.05)).toBe(true);+ }+ });++ test("starts sit at least a stride apart", () => {+ const starts = highlightCandidates(talk, settings).map((c) => c.from);+ for (let i = 1; i < starts.length; i++)+ expect(starts[i] - starts[i - 1]).toBeGreaterThanOrEqual(settings.strideSeconds);+ });++ test("a source shorter than one clip offers nothing", () => {+ expect(highlightCandidates([seg(0, 6, "Too short to clip.")], settings)).toEqual([]);+ });+});++describe("rankOf", () => {+ const h = { from: 0, to: 30, text: "", standalone: 1, hook: 0, payoff: 0 };++ test("the composite rides the settings' weights", () => {+ const only = { ...settings, standaloneWeight: 1, hookWeight: 0, payoffWeight: 0 };+ expect(rankOf(h, only)).toBe(1);+ expect(rankOf({ ...h, standalone: 0, hook: 1 }, only)).toBe(0);+ });++ test("weights that sum to nothing rank nothing", () => {+ expect(rankOf(h, { ...settings, standaloneWeight: 0, hookWeight: 0, payoffWeight: 0 })).toBe(0);+ });+});diff --git a/site/src/cut/lib/highlights.ts b/site/src/cut/lib/highlights.tsnew file mode 100644index 00000000..b2cb9982--- /dev/null+++ b/site/src/cut/lib/highlights.ts@@ -0,0 +1,439 @@+import { askJudgeChunked, noul, score } from "./judge";+import type { Entry, NoulAnswer, ScoreAnswer } from "./judge";+import type { PostFn } from "./pi/donkeyStream";+import type { Settings } from "@/lib/config/registry";++export type CutClipSettings = Settings["cutClip"];++/** Finding the moments in a long source worth cutting a short from.+ *+ * A talk holds a handful of stretches that stand on their own and an hour of+ * material that only makes sense in place. Which is which is a judgment about+ * meaning, so it goes to the judge; everything around it — where a sentence+ * ends, which windows are worth asking about, how the winners are picked so+ * they do not overlap — is arithmetic and stays in code.+ *+ * The pass is a cascade, because asking about every window of an hour would+ * be thousands of questions: a wide cheap sweep scores one window per stride,+ * then the shortlist earns the expensive questions (does it stand alone, does+ * the opening hold, does it land) and a few boundary variants each. Code+ * composes the weights and picks the non-overlapping winners, so changing what+ * "best" means is a settings change, not another round of inference.+ *+ * A clip does not have to be one stretch. Two moments that belong together+ * cut together, and the judge says which pairs those are; the spans come back+ * in source seconds for the caller to place.+ */++export interface SpeechSegment {+ start: number;+ end: number;+ text: string;+}++/** A stretch of the source, with what is said over it. */+export interface HighlightSpan {+ from: number;+ to: number;+ text: string;+}++export interface ScoredHighlight extends HighlightSpan {+ /** The composite under the settings' weights, 0..1. */+ rank: number;+ /** Stands on its own with nothing before it, 0..1. */+ standalone: number;+ /** How hard the opening holds someone who just landed on it, 0..1. */+ hook: number;+ /** Lands a complete thought by the end, 0..1. */+ payoff: number;+}++/** A clip, in the order its parts play. One span is the ordinary case; more+ * than one is two moments that belong together with the middle dropped. */+export interface HighlightClip {+ spans: HighlightSpan[];+ rank: number;+ seconds: number;+}++// A sentence break: terminal punctuation, then white space, then something+// that starts a sentence. The look-ahead is what keeps "4.5%" and "U.S." in+// one piece. A pause long enough to start something new breaks a unit too.+// Both are structure, and structure is all this does — the judge does the+// meaning. What is left over when the words run out is the last unit.+const SENTENCE_BREAK = /[.!?]["')\]]?\s+(?=[A-Z0-9"'(\[])/g;+const round2 = (n: number) => Math.round(n * 100) / 100;++/** Transcript segments folded into sentences. A recognizer cuts its segments+ * where it pleases — mid-clause, mid-breath, and often with two sentences+ * inside one — and a clip cut on one of those edges opens or closes+ * mid-thought.+ *+ * A sentence that ends inside a segment takes its time from where it ends in+ * the words: within one segment, time runs with the characters. That is an+ * approximation of a word clock, and it is the one a caption track is drawn+ * on anyway. */+export function sentences(segs: SpeechSegment[], pauseBreak: number): SpeechSegment[] {+ const out: SpeechSegment[] = [];+ // The running text, with each segment's char range and the time it covers.+ let buf = "";+ let spans: { at: number; len: number; start: number; end: number }[] = [];++ const timeAt = (i: number): number => {+ for (const sp of spans) {+ if (i <= sp.at + sp.len)+ return sp.start + ((Math.max(i, sp.at) - sp.at) / Math.max(1, sp.len)) * (sp.end - sp.start);+ }+ const last = spans[spans.length - 1];+ return last ? last.end : 0;+ };+ const emit = (upTo: number) => {+ const text = buf.slice(0, upTo).trim();+ if (text) out.push({ start: timeAt(buf.length - buf.trimStart().length), end: timeAt(upTo), text });+ buf = buf.slice(upTo);+ const shift = upTo;+ spans = spans+ .map((sp) => ({ ...sp, at: sp.at - shift }))+ .filter((sp) => sp.at + sp.len > 0);+ };++ let prevEnd: number | null = null;+ for (const s of segs) {+ const text = s.text.trim();+ if (!text || !(s.end > s.start)) continue;+ if (prevEnd !== null && s.start - prevEnd >= pauseBreak) emit(buf.length);+ if (buf && !buf.endsWith(" ")) buf += " ";+ spans.push({ at: buf.length, len: text.length, start: s.start, end: s.end });+ buf += text;+ prevEnd = s.end;+ // Every finished sentence in the buffer comes out; the unfinished tail+ // stays for the next segment to continue.+ for (;;) {+ SENTENCE_BREAK.lastIndex = 0;+ const m = SENTENCE_BREAK.exec(buf);+ if (!m) break;+ emit(m.index + m[0].length);+ }+ }+ emit(buf.length);+ return out;+}++const textOf = (units: SpeechSegment[], i: number, j: number) =>+ units+ .slice(i, j + 1)+ .map((u) => u.text)+ .join(" ");++/** The window starting at unit `i` that runs closest to `target` seconds+ * without passing `max`, ending on a sentence. Null when what is left is+ * shorter than `min`. */+function windowAt(+ units: SpeechSegment[],+ i: number,+ min: number,+ target: number,+ max: number+): { i: number; j: number; from: number; to: number } | null {+ const from = units[i].start;+ let best = -1;+ for (let j = i; j < units.length; j++) {+ const len = units[j].end - from;+ if (len > max) break;+ if (len >= min) {+ best = j;+ if (len >= target) break;+ }+ }+ if (best < 0) return null;+ return { i, j: best, from, to: units[best].end };+}++/** One window per stride across the whole source, each opening and closing on+ * a sentence. The stride is what keeps an hour's worth of starts down to a+ * sweep the judge can answer in a few requests; the shortlist gets its+ * boundaries refined afterwards. */+export function highlightCandidates(+ segs: SpeechSegment[],+ settings: CutClipSettings+): HighlightSpan[] {+ const units = sentences(segs, settings.pauseBreakSeconds);+ const out: HighlightSpan[] = [];+ let lastStart = -Infinity;+ for (let i = 0; i < units.length; i++) {+ if (units[i].start - lastStart < settings.strideSeconds) continue;+ const w = windowAt(units, i, settings.minSeconds, settings.targetSeconds, settings.maxSeconds);+ if (!w) continue;+ lastStart = w.from;+ out.push({ from: round2(w.from), to: round2(w.to), text: textOf(units, w.i, w.j) });+ }+ return out;+}++// Where a clip starts decides whether anyone stays for it, and the stride+// that found the window was never aimed at that. The start reaches further+// back than forward: a stretch that opens mid-answer is rescued by the+// question in front of it, and nothing later can stand in for that.+const START_SHIFTS = [0, -1, -2, 1];+const END_SHIFTS = [0, -1, 1];++/** The variants of a shortlisted window worth asking about: the same stretch+ * opened and closed a sentence or two either side of where the sweep put it. */+function variantsOf(+ units: SpeechSegment[],+ span: HighlightSpan,+ settings: CutClipSettings+): HighlightSpan[] {+ const i = units.findIndex((u) => Math.abs(u.start - span.from) < 0.05);+ const j = units.findIndex((u) => Math.abs(u.end - span.to) < 0.05);+ if (i < 0 || j < 0) return [span];+ const out: HighlightSpan[] = [];+ const seen = new Set<string>();+ for (const di of START_SHIFTS) {+ for (const dj of END_SHIFTS) {+ const a = i + di;+ const b = j + dj;+ if (a < 0 || b >= units.length || b < a) continue;+ const len = units[b].end - units[a].start;+ if (len < settings.minSeconds || len > settings.maxSeconds) continue;+ const key = `${a}:${b}`;+ if (seen.has(key)) continue;+ seen.add(key);+ out.push({+ from: round2(units[a].start),+ to: round2(units[b].end),+ text: textOf(units, a, b),+ });+ }+ }+ return out.length > 0 ? out : [span];+}++const TEXT_CAP = 3000;+const SWEEP_CHUNK = 50;++/** The judgments this pass is built on, in one place so the eval asks exactly+ * what the tool asks. Each takes the index of the item it is about, because a+ * fan-out names every candidate in one request. */+export const CLIP_QUESTIONS = {+ standsAlone: () => STANDS_ALONE,+ alone: (i: number) =>+ noul(`Does variant ${i} in \`variants\` make sense to someone who sees nothing before or after it?`, {+ true: "It introduces whatever it talks about and resolves it inside the clip.",+ false:+ "It opens on a pronoun, a name or an idea the listener has not been given, or answers a question that was asked before it starts.",+ }),+ hook: (i: number) =>+ noul(`Would the first sentence of variant ${i} in \`variants\` hold someone who is scrolling?`, {+ true: "It opens on a claim, a number, a turn of phrase or a question that makes the next sentence worth hearing.",+ false: "It opens on throat-clearing, an aside, a transition, or a setup whose point is far away.",+ }),+ lands: (i: number) =>+ noul(`Is variant ${i} in \`variants\` over by the time it ends — is its last sentence the end of the thing it was about?`, {+ true: "It closes on the point it was making: the conclusion, the turn, the last beat of the idea it opened with.",+ false:+ "By the end the speaker has moved on to something else, or it breaks off mid-argument. Ending on a complete sentence is not the same as being over.",+ }),+ follows: (i: number) =>+ noul(+ `In \`pairs\`, does part B of pair ${i} belong with part A as the second half of one clip, with everything between them cut out?`,+ {+ true: "B carries A further — the example after the claim, the answer after the question, the turn after the setup — and the jump between them reads as an edit.",+ false:+ "B repeats A, contradicts its footing, or is about something else, so the two together read as two clips stuck end to end.",+ },+ ),+} as const;++const STANDS_ALONE = score(+ "How well does the stretch of talk in `candidate` work as a short clip on its own, for someone who lands on it with no idea what came before?",+ [+ "It cannot be understood alone: it opens on a pronoun or a reference to something said earlier, answers a question nobody heard, or is the middle of an argument.",+ "It is followable but flat: accurate, unremarkable, the kind of stretch that fills the space between the parts people quote.",+ "It holds one clear idea from beginning to end and would read as deliberate to someone who saw nothing else.",+ "It is the part people quote: one sharp idea, stated plainly, that pays off inside the clip itself.",+ ]+);++/** The levels that score runs over, so a caller can turn a position on it back+ * into 0..1 the way the sweep does. */+export const STANDS_ALONE_LEVELS = STANDS_ALONE.criteria.length;++/** The wide sweep: one question per candidate, over the whole source. */+async function sweep(+ post: PostFn,+ candidates: HighlightSpan[],+ signal?: AbortSignal+): Promise<number[]> {+ if (candidates.length === 0) return [];+ const questions = Object.fromEntries(candidates.map((_, i) => [`clip::${i}`, STANDS_ALONE]));+ const state = (keys: string[]): Entry => ({+ candidates: keys.map((k) => {+ const i = Number(k.slice("clip::".length));+ return { i, seconds: round2(candidates[i].to - candidates[i].from), said: candidates[i].text.slice(0, TEXT_CAP) };+ }),+ });+ const answers = (await askJudgeChunked(post, state, questions, SWEEP_CHUNK, signal)) as Record<+ string,+ ScoreAnswer+ >;+ // A score's levels are ordered, so its position maps straight onto 0..1.+ const top = STANDS_ALONE_LEVELS - 1;+ return candidates.map((_, i) => (answers[`clip::${i}`]?.score ?? 0) / top);+}++/** Non-overlapping winners, best first. A source gives up a handful of real+ * moments; the rest of a ranked list is the same moment shifted by a sentence. */+function pickDistinct<T extends HighlightSpan & { rank: number }>(scored: T[], count: number): T[] {+ const out: T[] = [];+ for (const s of [...scored].sort((a, b) => b.rank - a.rank)) {+ if (out.length >= count) break;+ if (out.some((k) => s.from < k.to && k.from < s.to)) continue;+ out.push(s);+ }+ return out;+}++/** The close read of one shortlisted window and its boundary variants: does it+ * stand alone, does the opening hold, does it land. */+async function judgeVariants(+ post: PostFn,+ variants: HighlightSpan[],+ signal?: AbortSignal+): Promise<ScoredHighlight[]> {+ const questions: Record<string, ReturnType<typeof noul>> = {};+ for (let i = 0; i < variants.length; i++) {+ questions[`alone::${i}`] = CLIP_QUESTIONS.alone(i);+ questions[`hook::${i}`] = CLIP_QUESTIONS.hook(i);+ questions[`lands::${i}`] = CLIP_QUESTIONS.lands(i);+ }+ // Each request carries only the variants its own questions are about. The+ // fan-out splits by question, and three of them share one variant, so a+ // state built from every variant would ship the whole shortlist's words in+ // every chunk — a shortlist of a long talk, eleven times over.+ const state = (keys: string[]): Entry => {+ const wanted = [...new Set(keys.map((k) => Number(k.split("::")[1])))].sort((a, b) => a - b);+ return {+ variants: wanted.map((i) => ({+ i,+ seconds: round2(variants[i].to - variants[i].from),+ said: variants[i].text.slice(0, TEXT_CAP),+ })),+ };+ };+ const answers = (await askJudgeChunked(post, state, questions, SWEEP_CHUNK, signal)) as Record<+ string,+ NoulAnswer+ >;+ return variants.map((v, i) => ({+ ...v,+ standalone: answers[`alone::${i}`]?.noul ?? 0,+ hook: answers[`hook::${i}`]?.noul ?? 0,+ payoff: answers[`lands::${i}`]?.noul ?? 0,+ rank: 0,+ }));+}++/** The composite. Weights live in settings, so what "best" means is tuned+ * without asking anything again. */+export function rankOf(h: Omit<ScoredHighlight, "rank">, settings: CutClipSettings): number {+ const w = settings.standaloneWeight + settings.hookWeight + settings.payoffWeight;+ if (w <= 0) return 0;+ return (+ (h.standalone * settings.standaloneWeight ++ h.hook * settings.hookWeight ++ h.payoff * settings.payoffWeight) /+ w+ );+}++/** Pairs worth cutting together: a second beat that belongs with the first,+ * with whatever sat between them dropped. */+async function judgePairs(+ post: PostFn,+ picked: ScoredHighlight[],+ pool: ScoredHighlight[],+ settings: CutClipSettings,+ signal?: AbortSignal+): Promise<Map<number, ScoredHighlight>> {+ const pairs: { a: number; b: ScoredHighlight }[] = [];+ for (let a = 0; a < picked.length; a++) {+ const others = pool+ .filter((p) => p.from >= picked[a].to || p.to <= picked[a].from)+ .filter((p) => picked[a].to - picked[a].from + (p.to - p.from) <= settings.maxSeconds)+ .sort((x, y) => y.rank - x.rank)+ .slice(0, settings.pairsPerClip);+ for (const b of others) pairs.push({ a, b });+ }+ if (pairs.length === 0) return new Map();+ const questions = Object.fromEntries(pairs.map((_, i) => [`pair::${i}`, CLIP_QUESTIONS.follows(i)]));+ const state = (keys: string[]): Entry => ({+ pairs: keys.map((k) => {+ const i = Number(k.slice("pair::".length));+ return {+ i,+ A: picked[pairs[i].a].text.slice(0, TEXT_CAP),+ B: pairs[i].b.text.slice(0, TEXT_CAP),+ };+ }),+ });+ const answers = (await askJudgeChunked(post, state, questions, SWEEP_CHUNK, signal)) as Record<+ string,+ NoulAnswer+ >;+ const best = new Map<number, { p: number; span: ScoredHighlight }>();+ pairs.forEach((pair, i) => {+ const p = answers[`pair::${i}`]?.noul ?? 0;+ if (p < settings.followsFloor) return;+ const cur = best.get(pair.a);+ if (!cur || p > cur.p) best.set(pair.a, { p, span: pair.b });+ });+ return new Map([...best].map(([a, v]) => [a, v.span]));+}++/** The whole pass: every window scored cheaply, the shortlist read closely+ * with its boundaries refined, the winners picked apart from one another, and+ * each offered a second beat. Source seconds throughout. */+export async function findHighlights(+ post: PostFn,+ segs: SpeechSegment[],+ settings: CutClipSettings,+ count: number,+ signal?: AbortSignal+): Promise<HighlightClip[]> {+ const units = sentences(segs, settings.pauseBreakSeconds);+ const candidates = highlightCandidates(segs, settings);+ if (candidates.length === 0) return [];+ const coarse = await sweep(post, candidates, signal);+ const shortlist = pickDistinct(+ candidates.map((c, i) => ({ ...c, rank: coarse[i] })),+ Math.max(count, settings.shortlist)+ );+ const variants = shortlist.flatMap((s) => variantsOf(units, s, settings));+ const judged = (await judgeVariants(post, variants, signal)).map((h) => ({+ ...h,+ rank: rankOf(h, settings),+ }));+ // Two of the three judgments are gates, not weights. A stretch that leans on+ // what came before it, or that stops rather than finishes, is not a clip —+ // and a strong opening is exactly what would otherwise carry one past the+ // composite, because an opening is what a hook measures.+ const picked = pickDistinct(+ judged.filter(+ (h) =>+ h.standalone >= settings.standaloneFloor &&+ h.payoff >= settings.payoffFloor &&+ h.rank >= settings.rankFloor+ ),+ count+ );+ const seconds = (spans: HighlightSpan[]) => round2(spans.reduce((n, s) => n + s.to - s.from, 0));+ if (picked.length === 0) return [];+ const partners = await judgePairs(post, picked, judged, settings, signal);+ return picked.map((p, i) => {+ const b = partners.get(i);+ const spans = b ? [p, b].sort((x, y) => x.from - y.from) : [p];+ return { spans, rank: round2(p.rank), seconds: seconds(spans) };+ });+}diff --git a/site/src/cut/server/ai/catalog.ts b/site/src/cut/server/ai/catalog.tsindex 80eaa6d2..81128e73 100644--- a/site/src/cut/server/ai/catalog.ts+++ b/site/src/cut/server/ai/catalog.ts@@ -288,6 +288,7 @@ When a request depends on what the footage actually contains — "cut the dead aYour eyes and ears:- watch_video(clip_id | asset_id, from?, to?, interval_seconds?, detail?): samples the SOURCE on a dense steady floor plus a frame just after each line it speaks — what a video shows changes when what it says changes, so a transcribed source is read on its own clock and a caption track lands a frame per card — keeps only the frames that actually differ, and returns them as timestamped contact sheets — cells read left→right, top→bottom, each burned stamp is source seconds, and a gap between stamps means nothing changed there — plus sceneChanges, hard-cut times refined to about a third of a second of the true boundary, and — when the clip has captions or the source's transcript exists — a fused timeline placing each frame inside the speech it belongs to. Coverage ends where the result says: when truncated (span bound, frame cap, or time budget), continue from coveredTo. A shot shorter than the sampling step can fall between candidates — where short cuts matter, re-watch narrow with interval_seconds 0.5–1. \`detail\` sets how many pixels each kept frame is worth: scan (default) tiles small cells for coverage, read enlarges them enough for on-screen text, original returns the source's own frames for matching type, colour and edges. A scan cell is a quarter of the frame's width, so small captions arrive unreadable — reading words off the screen or reproducing a look means spending the pixels, never guessing at what the type says. \`textFits\` rides back with them: frame fractions, quietest first, for where a card can sit over that stretch without covering the picture or the platform's UI — the busy parts are what the watch measured to pick these frames in the first place. Seeing and reading are counted apart: \`unwatchedSeconds\` is source nobody has looked at, \`unreadSeconds\` is source nobody has read at a legible size, and \`unreadFrom\` says where the next close pass starts. Both persist on the asset. Watching builds the source's frame map, persisted on the asset; each media entry's "mapped" spans list where that map exists, and the first watch or listen of a source starts a background sweep that maps the rest, transcript included ("watching": true marks it running; the spans grow as segments land). Use the map to aim your watches — a mapped span is NOT footage you have seen; only the sheets returned to you in this conversation are, so watch any span before you cut inside it.+- find_highlights(clip_id | asset_id, from?, to?, count?): the moments of a long source worth cutting a short from, best first. It reads what is SAID — the transcript, or the project's cues where they cover the source — and ranks every stretch on whether it stands alone cold, whether its opening holds, and whether it finishes; a source with no transcript has nothing to rank, and listen_audio starts one. Each result is one or more spans in source seconds: two spans are two moments that belong together with the middle cut out, and add_clip \`spans\` lays that down. The ranking never saw the picture, so watch_video the spans before committing to them.- note_source(clip_id | asset_id, notes: [{from, to, text}]): writes what you saw onto the source, in source seconds. The sheets are conversation-local and the oldest media drops out as the chat grows; a note is permanent — it saves with the project, comes back as \`observed\` on the media entry in every later turn and chat, and is returned to you by get_state and by watch_video's \`recorded\`. This is how a source longer than one look becomes usable.**Seeing all of it.** Every result carries \`coveredTo\` and \`unwatchedSeconds\`. A \`to\` you passed yourself ends the call with \`truncated: false\` and the rest of the source unseen — that is not coverage. Any ask about the video AS A WHOLE — "what happens in this", "summarize it", "replicate/recreate this", "match this style", "how does it end" — means watching from 0 until \`unwatchedSeconds\` reaches 0, however many calls that takes; rebuilding one is the replicating-a-video skill. An ask that turns on what the frames SAY or how they are typeset — replicating a look, matching a caption treatment, quoting on-screen text — is scanned end to end for its shape and then re-watched at \`detail\` read or original over the stretches carrying the type, because a scan cell shows that a line is there and not what it reads. Watching a slice and describing the whole is how you end up reporting text that isn't there and putting the opening line at the end. Aiming a watch at one span is for a question about that span. One call reaches as far as its ceilings allow: a 600s span, 150 decodes, 90s of decoding, and the kept-frame cap its \`detail\` sets (36 at scan, 16 at read, 6 at original) — so at \`interval_seconds\` 0.5 a scan covers at most 75s of source, and far less where the picture keeps changing, while a detailed pass covers a fraction of that and belongs on the stretch that earns it. A source of a minute or two is a few calls; a source of an hour is not something to grind through frame-dense from 0. There, sweep it coarse first (\`interval_seconds\` 15–30 end to end) to learn its shape, then watch the spans that matter at a dense floor.@@ -318,6 +319,7 @@ Cutting speech well: pace is part of the message, and it is set in one place —"replicating-a-video": `# Replicating a video you can watchSomeone points at a video — a file they imported, a link you brought in, a clip on their timeline — and asks for one like it. Build its skeleton out of this editor's own parts and leave the footage for them to fill in.+**Clipping a long source.** A talk, an interview or a stream is mostly material that only makes sense in place. find_highlights ranks its stretches from the transcript, so an hour is read for its handful of moments without watching all of it; watch_video the spans it returns before committing, since the ranking never saw the picture. add_clip \`spans\` lays a pick down — several stretches of one source as one run of clips, the material between them left out — and a two-span result is one clip whose halves were never next to each other.**Watch the whole thing first.** Every decision below comes off the record, so the record has to be complete: watch from 0 and keep going until \`unwatchedSeconds\` is 0, writing each pass into note_source before the next one. Anything over a couple of minutes gets a coarse pass end to end first (\`interval_seconds\` 15-30) to learn its shape, then a dense pass (0.5) over the stretches that carry text or cut fast. Read every card word for word — its text, its face, where it sits in the frame, when it comes and goes — because a turn later the sheets are gone and the notes are all you have. Reading type is what \`detail\` is for: the dense pass over a stretch carrying words runs at \`detail\` read, and a face or treatment you are about to name runs at original, where the frame arrives at the source's own resolution. A scan cell is a quarter of the frame's width; a small caption in one is a smudge, and a smudge read as a word is how a card ends up saying something the video never said. When the cards ARE the narration — a caption track of what the speaker is saying — the transcript already holds every word, and the close passes are for how they are set: face, weight, size, colour, placement, how many words ride a card, how it enters and leaves. Read a few of those at original and take the rest from the transcript.**Block out the cuts.** \`sceneChanges\` are the cut times: shot 1 runs from 0 to the first, shot n from one to the next, the last to the end. add_clip with \`blocks\` lays the whole skeleton down in one call — one block per shot on track 0, each the length that shot runs, labelled with what the person has to supply ("wide of the kitchen, hands only"). Name the source in \`reference_asset_id\`. They fill a block by dropping their own footage on it — it takes the block's place and its length — or by asking you to (replace_item), so everything you build on top keeps its timing. The reference itself stays off the timeline: it is what the cut is copied from, and the blocks hold its shots' slots until the person's own footage arrives.@@ -585,7 +587,7 @@ Rules:- Transcription tools write tracks the user sees — run subtitles_generate / captions_generate only when captions were asked for, never just to read the words: existing cues are in editor_state, sources you have watched or listened to carry their own transcript (media entries marked speech: "transcribed"; get_state includes the timed segments), and audio plays to you via listen_audio. Don't transcribe a video with no speech (subtitles_from_visuals narrates silent footage), and never invent a spoken transcript.- Voiceovers duck other audio by default so they stay audible. Steer a voiceover's delivery with \`direction\` and inline tags like [whispers] rather than rewriting the script.- generate_image / generate_video / generate_character_video / voiceover_generate / read_subtitles_aloud / generate_music make media through hosted models (spends the user's Donkey credits, needs sign-in); call them when the user asked for the media itself — a request for the prompt or script gets text in chat. generate_music makes a music track — a vocal-free background bed by default ("add music to this video"), or a full sung song (instrumental:false); it's sung music, so a spoken narration is voiceover_generate. Bundled stock media (stock_search / stock_add) is free — use it when it fits. Media the user attached is in \`media\`; pass those asset ids as generation references when they say "use this", and place project assets in the cut with add_clip when they ask for them there ("make a movie from these photos"). Generated media previews on a chat card the user can expand, drag in, or file away; add it to the timeline (add_to_timeline:true or an index) only when they asked for it in the cut. When it lands on a card, close with a statement of where it is ("The voiceover is on the card — drag it in or tell me to place it."). Write a rich, specific prompt from their shorthand. Video renders take a minute or two. "Generate a video of/about X" means one generate_video clip; only when they name a narrated multi-shot production (a story, episode, narrated short) does generate_scene plan it and stop at the storyboard — the user reviews the frames (regenerate_shot redraws any before approval, no credits), then call approve_scene, since it renders many paid shots (regenerate_shot / recut_scene / restyle_scene revise it after). Close a plan message with the approval path as a statement: "Approve it — say so or hit Approve on the card — and I'll render the shots." Tweaks before approval are theirs to name.-- You can see and hear: audio the user attaches to their message plays right in it — answer "what does this say" from what you hear, no tool call. For project footage, watch_video samples a source's frames into timestamped contact sheets (scene changes included) — watch before cutting footage you haven't seen, and write each look down with note_source — sheets are conversation-local, notes stay on the source; listen_audio plays a source's audio — an audio asset or a video/clip's own track, one stretch per call, with \`coveredTo\` and \`unheardSeconds\` saying how far it reached and what nobody has played (an ask about how a whole source sounds listens on from \`unheardFrom\` until that is 0); detect_silence finds dead air; detect_beats reads the music's beat grid (persisted on the asset, drawn as dots on its clips, snapped to by every drag) for cutting to the music; capture_frame shows one rendered frame, at the time you pass or the playhead. The watching-and-cutting skill has the flow.+- You can see and hear: audio the user attaches to their message plays right in it — answer "what does this say" from what you hear, no tool call. For project footage, watch_video samples a source's frames into timestamped contact sheets (scene changes included) — watch before cutting footage you haven't seen, and write each look down with note_source — sheets are conversation-local, notes stay on the source; listen_audio plays a source's audio — an audio asset or a video/clip's own track, one stretch per call, with \`coveredTo\` and \`unheardSeconds\` saying how far it reached and what nobody has played (an ask about how a whole source sounds listens on from \`unheardFrom\` until that is 0); detect_silence finds dead air; find_highlights ranks the stretches of a long source worth clipping, from what is said over them; detect_beats reads the music's beat grid (persisted on the asset, drawn as dots on its clips, snapped to by every drag) for cutting to the music; capture_frame shows one rendered frame, at the time you pass or the playhead. The watching-and-cutting skill has the flow.- End on a statement. The final sentence of every reply states what changed, what failed, or the one suggestion you're making; "let me know if", "would you like", "do you want me to", and every other trailing offer or question are banned closers. The only question that may end a reply is the single clarifying one a genuinely ambiguous or blocked ask earns.`;}diff --git a/site/src/cut/worker/turnJob.ts b/site/src/cut/worker/turnJob.tsindex 7c48f18b..27227414 100644--- a/site/src/cut/worker/turnJob.ts+++ b/site/src/cut/worker/turnJob.ts@@ -1,6 +1,6 @@import type { UIMessage } from "ai";import { getGlobalSetting } from "@/lib/config/effective";-import { bindCutJudge } from "@/cut/lib/chatRuntime";+import { bindCutClip, bindCutJudge } from "@/cut/lib/chatRuntime";import { geminiModelRoleNames } from "@/lib/inference/gemini-models";import { openCloudProject, pushCloudProject } from "../lib/headless/docSession";import { dropPiSession, streamCutChat } from "../lib/pi/cutAgent";@@ -108,6 +108,7 @@ export async function runTurnJob(// behaves the same way in a job.const judgeSettings = await getGlobalSetting("cutJudge");bindCutJudge(judgeSettings);+ bindCutClip(await getGlobalSetting("cutClip"));deps.judgeSettings = judgeSettings;const toolCalls: string[] = [];const exec = deps.execTool;diff --git a/site/src/lib/config/registry.ts b/site/src/lib/config/registry.tsindex 46480235..a6fb2a89 100644--- a/site/src/lib/config/registry.ts+++ b/site/src/lib/config/registry.ts@@ -205,6 +205,59 @@ export const SETTINGS = defineSettings({description:"Thresholds for the typed judgments that route a chat turn (skill, tool areas), run a one-action turn with no model round, hold a turn back until its work is finished and its reply true to it, re-place the messages waiting in the tray, pick the items a sweep touches, rank stock, find filler words, and resolve described voices.",},+ cutClip: {+ schema: z+ .object({+ // A pause this long ends a sentence when the punctuation does not.+ pauseBreakSeconds: z.number().min(0.1).max(5),+ // What a clip may run. The sweep aims at the target and accepts+ // anything between the bounds, ending on a sentence.+ minSeconds: z.number().min(3).max(600),+ targetSeconds: z.number().min(3).max(600),+ maxSeconds: z.number().min(3).max(600),+ // How far apart the wide sweep's candidate starts sit. Smaller reads+ // more of the source and costs more questions.+ strideSeconds: z.number().min(1).max(120),+ // How many of the sweep's best earn the close read.+ shortlist: z.number().int().min(1).max(60),+ // Standing alone and landing are requirements: a stretch that needs+ // what came before it, or that trails off into the next thing, is not+ // a clip however well it opens. Below either floor it is not offered.+ standaloneFloor: z.number().min(0).max(1),+ payoffFloor: z.number().min(0).max(1),+ // What orders the ones that qualify.+ standaloneWeight: z.number().min(0).max(1),+ hookWeight: z.number().min(0).max(1),+ payoffWeight: z.number().min(0).max(1),+ // The composite below which a moment is not offered at all.+ rankFloor: z.number().min(0).max(1),+ // Second beats considered per clip, and the probability one has to+ // reach to be cut in behind the first.+ pairsPerClip: z.number().int().min(0).max(10),+ followsFloor: z.number().min(0).max(1),+ })+ .strict(),+ default: {+ pauseBreakSeconds: 0.6,+ minSeconds: 18,+ targetSeconds: 45,+ maxSeconds: 75,+ strideSeconds: 12,+ shortlist: 14,+ standaloneFloor: 0.35,+ payoffFloor: 0.5,+ standaloneWeight: 0.4,+ hookWeight: 0.35,+ payoffWeight: 0.25,+ rankFloor: 0.45,+ pairsPerClip: 3,+ followsFloor: 0.6,+ },+ public: true,+ title: "Clipping a long source",+ description:+ "How a talk or an interview is read for the moments worth cutting a short from: what a clip may run, how widely the first sweep looks, how many moments earn the close read, and what the ranking weighs.",+ },experimentResults: {schema: z.object({