Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
fix(hypit): compare the same stretch on both sides, timed by the reference's own words
rpone1d0d532d1cdd
diff --git a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.mdindex 21593093..1c2b891a 100644--- a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md+++ b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md@@ -36,7 +36,7 @@ looked at, since opening it yourself is what the blindness rule below forbids.Everything in this file happens on the reconstruction side, which changes every time a package,Recipe or source edge is edited.-## The loop unit is one element, across every shot that shows it+## The loop unit is one element, across every stretch it is drawn overLoop over one reconstructed element at a time — the new component, one caption system, one insertedcard.@@ -63,17 +63,11 @@ speech with the Source's own `estimate:Speech`, mocks every layer a Build has nothe package's Producer. Nothing is transcribed by hand, so nothing is transcribed one line at a time— which is what made a caption system look correct while its lines collided.-The window is named in words. `--segment` and `--selection` take a Script name, never a timestamp,-and the way to find the right name for a given shot is:--1. take the shot's `start_seconds` and `end_seconds` from the reference's `state.json`;-2. read the words spoken over that stretch out of the reference's `transcript.json`, which carries- one entry per word with its own timings;-3. find those words in the Script you wrote, and name the Selection or Segment that encloses them.--The reference clock is read exactly once, at step 1, and only to index a table. Everything after it-is words. The reconstruction's own frames come from estimated take lengths, so its clock and the-reference's do not correspond and matching them would put the window in the wrong place.+The window is named in words. `--segment` and `--selection` take a Script name, never a timestamp:+name the Segment or the Selection the element is drawn over, and give the same name to the+comparison, so both sides cover the same words. A word range and a shot are different divisions of+the same video — one Segment routinely runs across five shots — and what a render and the reference+have in common is the words, which is why the name travels between them rather than a number.### Mock the layers a Build has not made@@ -102,30 +96,40 @@ A mock lives only in this render. It is never written into the Source, and no ga`playback` check in `reconstruction_check` reads the Source's Recipes, so a mock cannot be mistakenfor coverage.-### Compare the whole shot, against every shot that shows the element+### Compare the whole stretch, against every stretch the element is drawn overCompare clips, not chosen frames:```-hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> \+hypit-reference-video-tools compare_reconstruction --reference-id <id> \+ --run projects/<name>/build.svrun --segment <id>|--selection <id> \--video <rendered>.mp4 --element <id>```+The word range is the one the render was drawn over, and the reference is cut from its own analysis+video at the seconds it speaks those words, so the two sides hold the same words for the same length+of time. Each end moves onto a shot boundary when one lies inside its own end word, and the rendered+clip is trimmed by the same seconds, so the pair opens and closes where the picture changes while+still covering exactly the words asked for. An end with no boundary inside its word leaves the pair+part-way through a shot, and the prompt then says how many seconds of it to read as an incomplete+shot and to report no differences from. `--shot-id <id>` compares one cut of the picture as the whole+shot it already is.+An element that animates in, leaves, and is replaced by another within one static board does not-produce a cut, so a shot can hold several states and no single frame represents it. Choosing one is-the failure this replaces: a caption system compared against the shot with the shortest line looks+produce a cut, so a stretch can hold several states and no single frame represents it. Choosing one is+the failure this replaces: a caption system compared against the stretch with the shortest line lookscorrect, because one line has nothing to collide with.-Compare it against **every shot the reference shows the element in**, not the clearest one. That is-the coverage round, and it comes before any repair — collect the whole difference set first, then+Compare it against **every stretch the element is drawn over**, not the clearest one. That is the+coverage round, and it comes before any repair — collect the whole difference set first, thenrepair against it. Doing it the other way makes the comparison after a repair look like a third-attempt at a shot that was never examined.+attempt at a stretch that was never examined.-`--image` against `NNN-representative.jpg` is available for one case only: the shot's own `visual:`-observation states in words that the element is completely still. The judgement comes from the-reference's observation, never from looking at your own render and concluding it does not move —-deciding that from the reconstruction is how the wrong frame got chosen in the first place. Where the-observation does not say still, compare the clip.+`--image` is available for one case only: the reference's own `visual:` observation states in words+that the element is completely still. The judgement comes from the reference's observation, never+from looking at your own render and concluding it does not move — deciding that from the+reconstruction is how the wrong frame got chosen in the first place. Where the observation does not+say still, compare the clip.Pass `--element <id>` every time so the gate credits the comparison to that element.@@ -145,8 +149,8 @@ rather than a font being chosen because it resembles what you remember.**On the `agent` observer the command returns two pictures and the question instead of an answer, andwho looks at them is up to the harness.** That observer reads pictures rather than video, so a clip-comparison arrives as two frame tiles: the reference shot's own tile, and one the command builds from-your render against that shot's duration, so both grids sample at the same rate and the same layout.+comparison arrives as two frame tiles, both built against the compared stretch's own duration, so the+two grids sample at the same rate and the same layout.What is compared is the same stretch either way.Give the pair to a subagent when you can: one handed two unlabelled pictures and the question knows@@ -183,12 +187,12 @@ When the difference is structural rather than cosmetic, repair in the order giveProducer, Type and Validator agreement, Surface vocabulary and decoding, implementation behaviour,then source usage. Fixing source usage over a broken implementation hides the defect one layer down.-## Converged means the differences are wording, in every shot+## Converged means the differences are wording, in every stretchStop when the returned differences are wording-level — a describer's phrasing rather than a visible-change — across **all** the shots the element was compared in. One shot reading clean while another-still shows lines colliding is an element mid-repair, and the shot that reads clean is usually the-easiest one. Passing `pnpm check` and `hypit check` is not convergence; it is the precondition for+change — across **all** the stretches the element was compared in. One stretch reading clean while+another still shows lines colliding is an element mid-repair, and the stretch that reads clean is+usually the easiest one. Passing `pnpm check` and `hypit check` is not convergence; it is the precondition forstarting the loop.## The loop is bounded, and stopping is the end@@ -207,8 +211,8 @@ return a difference every round for ever. So the loop ends on whichever of thesecomparison returns.- **No progress ends it immediately.** If the comparisons return the same difference they returnedbefore the repair, stop. The repair is not reaching the problem, and two more rounds of the same- reasoning will not find it. Judge this over the same set of shots before and after — a repair that- clears three shots and leaves one is progress, and re-comparing a different shot than last round+ reasoning will not find it. Judge this over the same set of stretches before and after — a repair+ that clears three and leaves one is progress, and re-comparing a different stretch than last roundmeasures nothing. Rendering and comparing cost real time on every round.- **There are two loops, each with its own two-attempt ceiling.** The first loop is over the*package*: render what it draws and compare it; a difference that the package cannot express is a@@ -229,11 +233,11 @@ return a difference every round for ever. So the loop ends on whichever of theseBeyond each ceiling, guesses start to overshoot — correcting past the reference rather thantowards it. Time beats fidelity here by explicit choice.- **Each ceiling is two attempts for the element, spread over all its shots.** Comparing an element- in five shots does not buy ten attempts; it buys a fuller picture of what one attempt has to fix.- Coverage and attempts measure different things — how much was looked at, and how many times the- element was changed — and one repair aimed at a difference several shots agree on is one attempt- however many shots reported it.+ **Each ceiling is two attempts for the element, spread over all its stretches.** Comparing an+ element in five stretches does not buy ten attempts; it buys a fuller picture of what one attempt+ has to fix. Coverage and attempts measure different things — how much was looked at, and how many+ times the element was changed — and one repair aimed at a difference several stretches agree on is+ one attempt however many of them reported it.## Measure what can be measured; iterate only on what cannot@@ -241,9 +245,9 @@ Two attempts per loop is not enough to converge by guessing, and it is not meantstated as a quantity — a stroke that is too thick, a shape that is too tall, type that is too large,a margin that is too wide — is not a guessing problem. Ask for the number: a`compare_reconstruction --question "how tall is the oval relative to the frame?"` returns a-measurement in one step, from whichever observer the reference holds. Ask it over whichever shot+measurement in one step, from whichever observer the reference holds. Ask it over whichever stretchrenders the quantity most legibly — this is one reading of one number, not the coverage round, and-reading it off a shot where the thing is small buys a worse number for the same price.+reading it off one where the thing is small buys a worse number for the same price.Do that instead of spending an attempt. An attempt is for differences that have no number — atypeface's character, a texture, a rhythm — where the only route is change it and look again.diff --git a/.agents/skills/hypit/references/reconstruction/workflow.md b/.agents/skills/hypit/references/reconstruction/workflow.mdindex 3a1870cb..d64bcbca 100644--- a/.agents/skills/hypit/references/reconstruction/workflow.md+++ b/.agents/skills/hypit/references/reconstruction/workflow.md@@ -176,12 +176,18 @@ a description of their visible differences. It is never told which is which, wha`--question` carries what to look at — which region to read, which regions hold a placeholder toskip, or one visible quantity to measure — and never what to conclude. Results are not cached. `reconstruction-loop.md` governs when and how to use it.-`--video <rendered.mp4>` compares the whole shot and is the default choice, since a shot can hold-several states of an element without a cut and no single frame stands for it. `--image` is for the-one case where the shot's own `visual:` observation says the element is completely still. Which-pictures the pair is made of follows the observer: `gemini` receives the reference clip and the-rendered clip; `agent` receives the reference shot's frame tile and one built from the render against-that shot's duration, so both grids sample alike.+Which stretch of the reference the pair is cut from is named one of two ways. `--segment` and+`--selection`, with the Run as `--run`, name a word range: the reference is cut from its own analysis+video at the seconds it speaks those words, which is the stretch a render covers, and the rendered+clip is trimmed by however far each end moved onto a shot boundary so both sides show the same word+at the same offset. `--shot-id` names a cut in the picture and compares that whole shot.++`--video <rendered.mp4>` compares the whole stretch and is the default choice, since a stretch can+hold several states of an element without a cut and no single frame stands for it. `--image` is for+the one case where the reference's own `visual:` observation says the element is completely still.+Which pictures the pair is made of follows the observer: `gemini` receives the two clips; `agent`+receives two frame tiles built against the compared stretch's own duration, so both grids sample+alike.`--element` names the reconstructed element the render draws. It never reaches the observer — thecomparison stays as blind as it is without it — and is written to the reference's `comparisons.jsonl`diff --git a/packages/reference-video-tools/src/authoring.ts b/packages/reference-video-tools/src/authoring.tsindex 0f195e58..5a7c14f6 100644--- a/packages/reference-video-tools/src/authoring.ts+++ b/packages/reference-video-tools/src/authoring.ts@@ -32,7 +32,8 @@ import { loadStudioRun } from "@hypit/studio/src/run.js";import { inspectStudioRun } from "@hypit/studio/src/studio-preflight.js";import type { SvsRecipe } from "@hypit/svs";-import { assert } from "./media.js";+import { assert, round } from "./media.js";+import type { TranscriptFile } from "./types.js";/** A frame range of the stand-in program, half-open in the Script's own order. */export type AuthoringWindow = {@@ -56,10 +57,27 @@ export type StandInTake = {readonly take: StandInTakeValue;};+/**+ * What timed one Segment's stand-in, and how long that clock said it runs.+ *+ * `seconds` is the clock's own answer. `frames` is what the Segment occupies in the program, which is+ * shorter for a Segment outside the window being drawn.+ */+export type StandInTiming = {+ readonly segment: string;+ readonly basis: "reference" | "estimate";+ readonly seconds: number;+ readonly frames: number;+ readonly tokens: number;+ /** Script words the reference's transcript matched. Zero on the estimate basis. */+ readonly matched: number;+};+export type StandInTakes = {readonly takes: readonly StandInTake[];readonly selections: ReadonlyMap<string, AuthoringWindow>;readonly frameCount: number;+ readonly timing: readonly StandInTiming[];};/** Which Segment the render is looking at, named directly or through a Selection's words. */@@ -177,25 +195,32 @@ function policiesBySegment(svml: string, sheets: ReadonlyMap<string, SvsRecipe>)const SILENT_AUDIO: BlobRef = { kind: "blob", digest: `sha256:${"0".repeat(64)}` as Digest, size: 1, mediaType: "audio/wav" };/**- * Build a stand-in SemanticTake for every Segment a Source declares, from the Source alone.+ * Build a stand-in SemanticTake for every Segment a Source declares.** A Track timed against speech cannot be projected before the speech exists, and on this route it- * does not exist: the Build has not run. What the Source does hold is the words themselves and the- * estimator it already trusts to size its own generations — `estimate:Speech` with a policy Recipe.- * Running that estimator here produces the same numbers the Source used to order its takes, so this- * introduces no second clock; it reads the one already written down.+ * does not exist: the Build has not run. Two clocks can stand in for it, and each Segment is sized by+ * whichever one can speak for it.+ *+ * **The reference's own words.** For a reconstruction, the words have already been spoken once and+ * timed: the Script was transcribed from the reference video, and `prepare_reference` wrote the+ * seconds of every word of it. A Segment timed this way runs for exactly as long as the reference+ * spends on those words, and every window inside it does too, which is what a progressive reveal, a+ * typewriter, a staggered row or an enter animation is compared on.*- * What the result is good for: which elements are on screen together, where each sits, at what size- * and colour. Those follow from word ranges and Recipe values and are exact. What it is not good for- * is real timing — the delivered speech is not the estimate, and anything measured in seconds waits- * for `production-gates.md` Gate 3.+ * **`estimate:Speech` with a policy Recipe.** The estimator the Source already trusts to size its own+ * generations. Running it here produces the same numbers the Source used to order its takes, so it+ * introduces no third clock; it reads one already written down. This is what sizes a Segment the+ * reference has nothing to say about, and what sizes every Segment when no reference is given.*- * Tokens are laid across the Segment in proportion to the same syllable count the estimator uses, so- * a long word occupies more of the window than a short one and the ordering is the Script's.+ * Word ranges and Recipe values are exact on either clock, so which elements are on screen together,+ * where each sits, at what size and colour follow from the Source itself. Alignment against the+ * speech a Build synthesizes waits for `production-gates.md` Gate 3.** @param svmlPath the Author SVML this Source is written in* @param frameRate the Program's frame rate, as a whole number of frames per second- * @returns one `{ segmentId, take }` per Segment, in Script order+ * @param focus which Segment the render is looking at+ * @param reference the reference's per-word times, from `referenceWords`+ * @returns one `{ segmentId, take }` per Segment in Script order, and what timed each of them*//** The Segment a Selection is marked in, so a window named by Selection can still name the cut. */function segmentOfSelection(svml: string, selection: string | undefined): string | undefined {@@ -216,7 +241,257 @@ function segmentOfSelection(svml: string, selection: string | undefined): stringreturn found;}-export async function standInTakes(svmlPath: string, frameRate: number, focus: StandInFocus = {}): Promise<StandInTakes> {+/**+ * The Author SVML a Run declares, both as it is written and as a path.+ *+ * Studio's unit of work is the Run, so everything on this route is handed one and reads the Source+ * back out of it. `declared` is what the Run wrote, which a derived Run has to repoint.+ */+export async function authorSource(runPath: string): Promise<{ readonly path: string; readonly declared: string }> {+ const runSource = await readFile(runPath, "utf8").catch(() => undefined);+ assert(runSource !== undefined, `cannot read ${runPath}`);+ const declared = /<author\s+source="([^"]+)"/u.exec(runSource)?.[1];+ assert(declared !== undefined, `${runPath} declares no <author source="…"/>`);+ return { path: resolve(dirname(runPath), declared), declared };+}++/** One word of a reference's transcript, at the seconds the reference speaks it. */+export type ReferenceWord = {+ readonly text: string;+ readonly startSeconds: number;+ readonly endSeconds: number;+};++/** Where `prepare_reference` writes a reference's per-word times. */+export function referenceRoot(): string {+ return join(repositoryRoot(), ".hypit", "reference-video-tools");+}++/**+ * The per-word times a prepared reference was transcribed to, in the order it speaks them.+ *+ * Words the aligner left untimed are dropped: a word with no seconds against it cannot anchor+ * anything, and carrying it would shift every index after it away from the clock.+ */+export async function referenceWords(reference: string): Promise<readonly ReferenceWord[]> {+ const path = join(referenceRoot(), reference, "transcript.json");+ const text = await readFile(path, "utf8").catch(() => undefined);+ assert(text !== undefined, `reference ${reference} has no transcript at ${path}; run prepare_reference first`);+ const file = JSON.parse(text) as TranscriptFile;+ const words = file.passages.flatMap((passage) => passage.words).flatMap((word) =>+ word.start_seconds === undefined || word.end_seconds === undefined+ ? []+ : [{ text: word.text, startSeconds: word.start_seconds, endSeconds: word.end_seconds }]);+ assert(words.length > 0, `reference ${reference} has a transcript with no timed words`);+ return words;+}++/** Lowercase letters and digits only, so `AI,` and `ai` are the same word and `11 labs` is two. */+function normalizeWord(text: string): string {+ return text.toLowerCase().replace(/[^a-z0-9]+/gu, "");+}++type MatchingBlock = { readonly left: number; readonly right: number; readonly size: number };++/**+ * The longest run of words that appears in both sequences, within the given ranges.+ *+ * `lengths` carries, for each position in the right sequence, how long a run ends there against the+ * word just read from the left. Reading one left word at a time turns the whole search into two+ * passes rather than a comparison of every pair.+ */+function longestRun(+ left: readonly string[], right: readonly string[],+ leftFrom: number, leftTo: number, rightFrom: number, rightTo: number,+ positions: ReadonlyMap<string, readonly number[]>,+): MatchingBlock {+ let bestLeft = leftFrom;+ let bestRight = rightFrom;+ let bestSize = 0;+ let lengths = new Map<number, number>();+ for (let index = leftFrom; index < leftTo; index += 1) {+ const next = new Map<number, number>();+ for (const at of positions.get(left[index]!) ?? []) {+ if (at < rightFrom) continue;+ if (at >= rightTo) break;+ const run = (lengths.get(at - 1) ?? 0) + 1;+ next.set(at, run);+ if (run > bestSize) {+ bestSize = run;+ bestLeft = index - run + 1;+ bestRight = at - run + 1;+ }+ }+ lengths = next;+ }+ return { left: bestLeft, right: bestRight, size: bestSize };+}++/**+ * Which Script word is which transcript word, as runs that appear in both in the same order.+ *+ * The two sequences say the same thing and are not the same list. A transcriber writes `11 labs`+ * where the Script writes `elevenlabs`, splits a word in two, or hears one that is not there, and a+ * single insertion is enough to put a positional pairing a word out for the whole rest of the+ * reference — on this project it drops the pairing from 96% of words to 41%. So the two are aligned:+ * the longest run common to both is taken as fixed, and the stretches on either side of it are+ * aligned the same way, down to the words that do not appear on both sides. Anchoring on runs rather+ * than on single words is what keeps a common word like `the` from pairing with a `the` elsewhere in+ * the reference.+ */+function alignWords(script: readonly string[], reference: readonly string[]): ReadonlyMap<number, number> {+ const positions = new Map<string, number[]>();+ for (const [index, word] of reference.entries()) {+ const seen = positions.get(word);+ if (seen === undefined) positions.set(word, [index]);+ else seen.push(index);+ }+ const pairs = new Map<number, number>();+ const pending: (readonly [number, number, number, number])[] = [[0, script.length, 0, reference.length]];+ while (pending.length > 0) {+ const [leftFrom, leftTo, rightFrom, rightTo] = pending.pop()!;+ const block = longestRun(script, reference, leftFrom, leftTo, rightFrom, rightTo, positions);+ if (block.size === 0) continue;+ for (let step = 0; step < block.size; step += 1) pairs.set(block.left + step, block.right + step);+ if (leftFrom < block.left && rightFrom < block.right) pending.push([leftFrom, block.left, rightFrom, block.right]);+ const leftEnd = block.left + block.size;+ const rightEnd = block.right + block.size;+ if (leftEnd < leftTo && rightEnd < rightTo) pending.push([leftEnd, leftTo, rightEnd, rightTo]);+ }+ return pairs;+}++type WordSpan = { readonly start: number; readonly end: number };++/**+ * A time for every Script word, from the ones that matched.+ *+ * A matched word takes the seconds the reference speaks it at. A run of unmatched words is spread+ * evenly across the stretch between the matched words on either side of it — the reference was+ * saying something there, and what it was saying is the run. A run at either end of the Script has a+ * matched word on one side only, so it takes the reference words immediately outside that one, at a+ * word apiece.+ */+function wordSpans(count: number, pairs: ReadonlyMap<number, number>, reference: readonly ReferenceWord[]): readonly WordSpan[] {+ const anchors = [...pairs.keys()].sort((left, right) => left - right);+ const spans: WordSpan[] = new Array(count) as WordSpan[];+ for (const index of anchors) {+ const word = reference[pairs.get(index)!]!;+ spans[index] = { start: word.startSeconds, end: word.endSeconds };+ }+ for (let position = 0; position <= anchors.length; position += 1) {+ const before = position === 0 ? undefined : anchors[position - 1]!;+ const after = position === anchors.length ? undefined : anchors[position]!;+ const from = before === undefined ? 0 : before + 1;+ const to = after === undefined ? count : after;+ if (from >= to) continue;+ const run = to - from;+ let start: number;+ let end: number;+ if (before !== undefined && after !== undefined) {+ start = reference[pairs.get(before)!]!.endSeconds;+ end = reference[pairs.get(after)!]!.startSeconds;+ } else if (after !== undefined) {+ const at = pairs.get(after)!;+ start = reference[Math.max(0, at - run)]!.startSeconds;+ end = reference[at]!.startSeconds;+ } else {+ const at = pairs.get(before!)!;+ start = reference[at]!.endSeconds;+ end = reference[Math.min(reference.length - 1, at + run)]!.endSeconds;+ }+ const step = (end - start) / run;+ for (let offset = 0; offset < run; offset += 1) {+ spans[from + offset] = { start: start + step * offset, end: start + step * (offset + 1) };+ }+ }+ return spans;+}++/**+ * The stretch of a reference that speaks one Segment's or one Selection's words.+ *+ * A render covers a word range; a shot is a cut in the picture. The two are unrelated stretches of+ * the same video, and a Segment routinely runs across five of them. So a comparison against the+ * reference finds its stretch the way every other window on this route is found — the Script's own+ * tokens, aligned against the transcript — and reads the seconds off the words it lands on.+ *+ * `first` and `last` carry the end words' own spans, which is what an end can be moved inside of+ * without the range gaining or losing a whole word.+ */+export type SpokenRange = {+ readonly startSeconds: number;+ readonly endSeconds: number;+ readonly first: ReferenceWord;+ readonly last: ReferenceWord;+ readonly words: number;+ /** How many of those words the transcript carries. The rest take their seconds from `wordSpans`. */+ readonly matched: number;+};++export async function spokenRange(+ svmlPath: string,+ focus: StandInFocus,+ reference: readonly ReferenceWord[],+): Promise<SpokenRange> {+ const svml = await readFile(svmlPath, "utf8");+ const body = scriptBody(svml);+ const parsed = parseScript(svmlPath, body.text, body.offset);++ // Which Script words the range covers. A Selection is unioned over its occurrences, the same way+ // its frame window is, since one id can be marked in several places.+ let from: number | undefined;+ let to: number | undefined;+ if (focus.selection !== undefined) {+ const selection = parsed.selections.find((item) => item.id === focus.selection);+ assert(selection !== undefined, `the Script marks no Selection ${focus.selection}`);+ for (const occurrence of selection.occurrences) {+ from = Math.min(from ?? Infinity, occurrence.open.boundary.tokenIndex);+ to = Math.max(to ?? 0, occurrence.close.boundary.tokenIndex);+ }+ } else {+ assert(focus.segment !== undefined, "name a Segment or a Selection to read a word range from");+ const segment = parsed.segments.find((item) => item.id === focus.segment);+ assert(segment !== undefined, `the Script has no Segment ${focus.segment}`);+ from = segment.tokenStart;+ to = segment.tokenEndExclusive;+ }+ const start = from ?? 0;+ const end = to ?? 0;+ assert(start < end, `${focus.selection ?? focus.segment} covers no words`);++ const pairs = alignWords(+ parsed.tokens.map((token) => normalizeWord(token.text)),+ reference.map((word) => normalizeWord(word.text)));+ let matched = 0;+ for (let index = start; index < end; index += 1) if (pairs.has(index)) matched += 1;+ // Without a matched word inside the range, its seconds are an interpolation between whatever the+ // reference was saying on either side of it, which is a stretch of the video nobody asked for.+ assert(matched > 0,+ `the reference's transcript carries none of the words of ${focus.selection ?? focus.segment}, so the seconds it spoke them at are unknown`);++ const spans = wordSpans(parsed.tokens.length, pairs, reference);+ const word = (index: number): ReferenceWord => ({+ text: parsed.tokens[index]!.text,+ startSeconds: spans[index]!.start,+ endSeconds: spans[index]!.end,+ });+ return {+ startSeconds: spans[start]!.start,+ endSeconds: spans[end - 1]!.end,+ first: word(start),+ last: word(end - 1),+ words: end - start,+ matched,+ };+}++export async function standInTakes(+ svmlPath: string,+ frameRate: number,+ focus: StandInFocus = {},+ reference: readonly ReferenceWord[] = [],+): Promise<StandInTakes> {const svml = await readFile(svmlPath, "utf8");const body = scriptBody(svml);const parsed = parseScript(svmlPath, body.text, body.offset);@@ -237,29 +512,74 @@ export async function standInTakes(svmlPath: string, frameRate: number, focus: S: parsed.segments.find((segment) => token >= segment.tokenStart && token < segment.tokenEndExclusive)?.id;}+ // Which Script word the reference speaks when. The Script was transcribed from that video, so the+ // words are the same words in the same order, and a stand-in timed from them runs at the pace the+ // reconstruction is being compared against.+ const words = reference.length === 0+ ? undefined+ : (() => {+ const pairs = alignWords(parsed.tokens.map((token) => normalizeWord(token.text)), reference.map((word) => normalizeWord(word.text)));+ return pairs.size === 0 ? undefined : { pairs, spans: wordSpans(parsed.tokens.length, pairs, reference) };+ })();+const takes: StandInTake[] = [];+ const timing: StandInTiming[] = [];// Global frame span of every word, in Script order, so a Selection can be turned into a frame// range without going near a clock. This is the correspondence the route uses everywhere else:// a stretch of the reference is found by its words, and its words are where the Script says.const frameOfToken: { readonly frame: number; readonly end: number }[] = [];let frameCursor = 0;for (const segment of parsed.segments) {- const policy = policies.get(segment.id) ?? shared;- if (policy === undefined) {- throw new Error(`Segment ${segment.id} names no estimate:Speech policy, and the Source uses ${policyCount} policies, so there is no single one to fall back to`);- }const tokens = parsed.tokens.slice(segment.tokenStart, segment.tokenEndExclusive);const text = tokens.map((token) => token.text).join(" ");- const seconds = estimateSpeechDuration({ value: text }, policy);++ // A Segment is timed from the reference when the reference was heard saying some of its words.+ // The decision is made per Segment: an ordinary transcription difference costs one Segment its+ // reference clock and leaves the rest of the Script on it.+ let matched = 0;+ if (words !== undefined) {+ for (let index = segment.tokenStart; index < segment.tokenEndExclusive; index += 1) if (words.pairs.has(index)) matched += 1;+ }+ const spoken = words === undefined || matched === 0+ ? undefined+ : words.spans.slice(segment.tokenStart, segment.tokenEndExclusive);++ // How long the Segment runs. The reference's own words when it was heard saying them, the+ // estimator the Source already trusts when it was not.+ let seconds: number;+ let basis: StandInTiming["basis"];+ let weights: readonly number[] | undefined;+ if (spoken !== undefined) {+ seconds = Math.max(0, spoken.at(-1)!.end - spoken[0]!.start);+ basis = "reference";+ } else {+ const policy = policies.get(segment.id) ?? shared;+ if (policy === undefined) {+ throw new Error(`Segment ${segment.id} names no estimate:Speech policy, and the Source uses ${policyCount} policies, so there is no single one to fall back to`);+ }+ seconds = estimateSpeechDuration({ value: text }, policy);+ basis = "estimate";+ const language = resolveSpeechEstimateLanguage(text, policy.language);+ weights = tokens.map((token) => Math.max(1, countSpeechEstimateUnits(token.text, language)));+ }+const frameCount = focused !== undefined && segment.id !== focused? Math.max(1, tokens.length): Math.max(tokens.length, Math.round(seconds * frameRate));+ timing.push({ segment: segment.id, basis, seconds: round(seconds), frames: frameCount, tokens: tokens.length, matched });++ // Where each word ends. On the reference clock a word holds the screen until the next one starts+ // and the last holds until the reference stops saying it, so the small gaps the reference leaves+ // between words go to the word before them and the Take covers its Segment without holes. On the+ // estimate the share is the estimator's own unit count, so the word order and the relative widths+ // both come from the same place the duration did.+ const total = weights?.reduce((sum, weight) => sum + weight, 0) ?? 0;+ const edge = spoken !== undefined+ ? (index: number, start: number): number =>+ Math.max(start + 1, Math.round((spoken[index + 1]!.start - spoken[0]!.start) * frameRate))+ : (index: number, start: number): number =>+ start + Math.max(1, Math.round(frameCount * weights![index]! / total));- // Share the Segment's frames out by the estimator's own unit count, so the word order and the- // relative widths both come from the same place the duration did.- const language = resolveSpeechEstimateLanguage(text, policy.language);- const weights = tokens.map((token) => Math.max(1, countSpeechEstimateUnits(token.text, language)));- const total = weights.reduce((sum, weight) => sum + weight, 0);const anchors: { readonly identity: string; readonly frame: number }[] = [{ identity: `segment:${segment.id}:start`, frame: 0 },{ identity: `segment:${segment.id}:end`, frame: frameCount },@@ -268,10 +588,11 @@ export async function standInTakes(svmlPath: string, frameRate: number, focus: Slet used = 0;for (const [index, token] of tokens.entries()) {const start = used;- // The last token closes the Segment exactly, so rounding never leaves a frame unclaimed.+ // The last token closes the Segment exactly, so rounding never leaves a frame unclaimed, and+ // every earlier one is held back far enough that the ones after it still get a frame each.used = index === tokens.length - 1? frameCount- : Math.min(frameCount - (tokens.length - 1 - index), start + Math.max(1, Math.round(frameCount * weights[index]! / total)));+ : Math.min(frameCount - (tokens.length - 1 - index), edge(index, start));const id = `segment:${segment.id}:token:${index + 1}`;placed.push({tokenId: id, segmentId: segment.id, text: token.text,@@ -319,7 +640,7 @@ export async function standInTakes(svmlPath: string, frameRate: number, focus: S}if (start < end) selections.set(selection.id, { startFrame: start, endFrameExclusive: end });}- return { takes, selections, frameCount: frameCursor };+ return { takes, selections, frameCount: frameCursor, timing };}function blobRef(bytes: Uint8Array, mediaType: string): BlobRef {@@ -351,8 +672,50 @@ export type RenderElementInput = {readonly out: string;readonly segment?: string;readonly selection?: string;+ /**+ * A prepared reference to time the stand-in from. Its transcript holds the seconds each word was+ * spoken at, and the Script was transcribed from that video, so the stand-in runs at the pace the+ * render is compared against. Without one the Source's own `estimate:Speech` sizes each Segment.+ */+ readonly reference_id?: string;+};++/** What timed the stand-in behind one render, per Segment. */+export type StandInTimingReport = {+ readonly reference_id: string | null;+ /** `mixed` when some Segments were timed from the reference and others fell back to the estimate. */+ readonly basis: "reference" | "estimate" | "mixed";+ readonly segments: readonly StandInTiming[];+};++/**+ * The record `render_element` leaves beside its output.+ *+ * A comparison is made from a file, and the file alone says nothing about what timed the picture in+ * it. Writing that beside the output means `compare_reconstruction` can copy it into the comparison+ * log from the path it was handed, and the gate can report which comparisons were made at the+ * reference's pace without re-deriving anything.+ */+export type StandInSidecar = {+ readonly element: string;+ readonly window: AuthoringWindow;+ readonly timing: StandInTimingReport;};+/** Where `render_element` writes the record of what timed a render. */+export function standInSidecarPath(outPath: string): string {+ return `${outPath}.stand-in.json`;+}++function timingReport(reference: string | undefined, segments: readonly StandInTiming[]): StandInTimingReport {+ const bases = new Set(segments.map((item) => item.basis));+ return {+ reference_id: reference ?? null,+ basis: bases.size > 1 ? "mixed" : segments.some((item) => item.basis === "reference") ? "reference" : "estimate",+ segments,+ };+}+/*** Render one element of a Source the way that Source configures it, without a Build and without a* Provider.@@ -368,17 +731,21 @@ export type RenderElementInput = {** Canvas and Frames, Recipe values, bindings the Source* which elements share a window, in what order the Script's words, through their Selections- * how long each Segment runs the Source's own `estimate:Speech`+ * how long each Segment runs the reference's own words, or `estimate:Speech`* the layers a Build has not made `make-placeholder`, sized from the Canvas** The one thing missing before a Build is real speech, and `standInTakes` supplies a Segment- * skeleton from the estimator the Source already trusts — the same numbers it ordered its takes- * with, not a second clock. Those takes enter through `<value>` and `<satisfy>`, the mechanism- * `examples/all-components-preview` uses to open in Studio without spending anything, so nothing- * here is a private back door into the graph.+ * skeleton for it. Given `reference_id` it reads the seconds out of that reference's transcript, so+ * every window is as long as the reference spends on the words inside it; without one it reads the+ * estimator the Source already trusts, the same numbers it ordered its takes with. Those takes enter+ * through `<value>` and `<satisfy>`, the mechanism `examples/all-components-preview` uses to open in+ * Studio without spending anything, so nothing here is a private back door into the graph.** What this settles: which elements are on screen together, where each sits, at what size, weight and- * colour. What it does not: real timing, which waits for `playbooks/craft/production-gates.md` Gate 3.+ * colour, and — on a reference-timed stand-in — how long each of them has to arrive in. What it does+ * not: alignment against the speech a Build synthesizes, which waits for+ * `playbooks/craft/production-gates.md` Gate 3. `timing` in the result, and the sidecar written+ * beside the output, name which of the two clocks sized each Segment.*/export async function renderElement(input: RenderElementInput): Promise<Record<string, unknown>> {const element = input.element;@@ -457,10 +824,15 @@ export async function renderElement(input: RenderElementInput): Promise<Record<ssvml = repointed;}- const { takes, selections, frameCount: programFrames } = await standInTakes(sourcePath, frameRate, focus);+ // The reference's per-word times, when one was named. Read after the cut so a transcript that is+ // missing is reported before a minute of rendering rather than after it.+ const reference = input.reference_id?.trim();+ const spoken = reference === undefined || reference.length === 0 ? [] : await referenceWords(reference);++ const { takes, selections, frameCount: programFrames, timing } = await standInTakes(sourcePath, frameRate, focus, spoken);const framesBySegment = new Map(takes.map((item) => [item.segmentId, item.take.segment.endFrameExclusive]));// A Selection's window in frames, summed over the stand-in tokens it covers. This is the same word- // span the Source binds to, carried into frames by the same estimate that sized the Segment.+ // span the Source binds to, carried into frames by the same clock that sized the Segment.const framesBySelection = new Map<string, number>();for (const { segmentId, take } of takes) {for (const token of take.tokens) framesBySelection.set(token.tokenId, token.endFrameExclusive - token.startFrame);@@ -486,7 +858,15 @@ export async function renderElement(input: RenderElementInput): Promise<Record<sif (media === undefined || windows.has(media)) continue;const segment = /\bduring=\{story\.segment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];if (segment !== undefined) { windows.set(media, framesBySegment.get(segment) ?? frameRate); continue; }- windows.set(media, frameRate * 2);+ // A cutaway's window is the Selection it is bound to, which the stand-in already measured in+ // frames. A mock shorter than its window runs out inside it, and with `playback` at its default+ // the frames after that draw nothing — a gap the Source never wrote, appearing only in the+ // comparison, in exactly the shape of the defect the comparison is looking for.+ const selection = /\bduring=\{story\.selection\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];+ const window = selection === undefined ? undefined : selections.get(selection);+ windows.set(media, window === undefined+ ? frameRate+ : window.endFrameExclusive - window.startFrame);}const carries = new Map<string, { readonly frames: number; readonly picture: boolean }>();for (const match of svml.matchAll(/<pipeline:Normalize\b([^>]*?)\/?>/gsu)) {@@ -703,6 +1083,13 @@ export async function renderElement(input: RenderElementInput): Promise<Record<s], { encoding: "utf8", windowsHide: true, timeout: 600_000 });assert(encoded.status === 0, `ffmpeg refused: ${(encoded.stderr ?? "").trim().slice(-2000)}`);+ // What timed this picture, written where the picture is. `compare_reconstruction` is handed a path+ // and nothing else, so this is how the comparison log comes to say whether the stretch it looked at+ // ran at the reference's pace or at an estimate of it.+ const report = timingReport(reference, timing);+ const sidecar: StandInSidecar = { element, window, timing: report };+ await writeFile(standInSidecarPath(outPath), `${JSON.stringify(sidecar, null, 2)}\n`, "utf8");+return {element,out: outPath,@@ -711,6 +1098,8 @@ export async function renderElement(input: RenderElementInput): Promise<Record<sframe_rate: frameRate,window,program_frames: programFrames,+ timing: report,+ stand_in_ref: standInSidecarPath(outPath),mocked: { media: mockedMedia().size, images: imageIds.length },};}@@ -785,6 +1174,9 @@ export async function renderPreviews(input: RenderPreviewsInput): Promise<Record} catch (error) {throw new Error(`${directory} could not draw ${picture}: ${error instanceof Error ? error.message : String(error)}`);}+ // A catalogue picture is compared against nothing, so the record of what timed it is dropped+ // rather than committed beside the package's own pictures.+ await rm(standInSidecarPath(out), { force: true });pictures.push({ package_dir: directory, picture, drawn: true, tag, element, out });written += 1;}diff --git a/packages/reference-video-tools/src/checks.ts b/packages/reference-video-tools/src/checks.tsindex c989c90d..d3121e7c 100644--- a/packages/reference-video-tools/src/checks.ts+++ b/packages/reference-video-tools/src/checks.ts@@ -119,14 +119,26 @@ export type ReconstructionCheckInput = {readonly reference_id?: string;};+/**+ * Which clock timed the stand-ins an element's comparisons were made over.+ *+ * `reference` is every recorded comparison drawn at the pace the reference speaks those words;+ * `estimate` is every one drawn at the pace `estimate:Speech` predicts; `mixed` is both among them.+ * `not recorded` is a comparison whose picture carried no `render_element` sidecar, so what timed it+ * is unknown.+ */+type TimingBasis = "reference" | "estimate" | "mixed" | "not recorded";+/** One element that draws a Track, and the comparisons recorded against its id. */type ElementReport = {readonly element: string;readonly id: string;readonly package_name: string;readonly comparisons: number;- readonly shots: readonly string[];+ /** The stretches it was compared against: a shot, or the words a Segment or Selection marks. */+ readonly compared_against: readonly string[];readonly state: string;+ readonly timing_basis: TimingBasis;readonly note?: string;};@@ -165,6 +177,12 @@ type PlaybackReport = {* it, so the shots are reported for a reader, and a lone comparison is called out rather than assumed* meaningful.*+ * Each element also carries what timed the stand-ins it was compared over, read from the sidecar+ * `render_element` writes beside its output and carried into the log by `compare_reconstruction`.+ * That is reported rather than required: participation is the rule here, and an estimate-timed+ * comparison is a comparison. What it says is which elements have had their behaviour over their+ * window looked at and which have only had their layout looked at.+ ** A project that places no drawing element passes with nothing to require.*/export async function reconstructionCheck(@@ -320,7 +338,20 @@ export async function reconstructionCheck(}assert(prepared.includes(reference), `reference ${reference} is not prepared under ${referenceRoot}`);- type LoggedComparison = { readonly element?: string; readonly shot_id: string; readonly status: string };+ type LoggedComparison = {+ readonly element?: string;+ readonly shot_id?: string;+ readonly range?: { readonly segment?: string; readonly selection?: string };+ readonly status: string;+ readonly stand_in?: { readonly timing?: { readonly basis?: string } };+ };+ // What the comparison was made against, as a reader would name it: a shot, or the words a Segment+ // or Selection marks.+ const against = (entry: LoggedComparison): string =>+ entry.shot_id+ ?? (entry.range?.segment === undefined ? undefined : `segment ${entry.range.segment}`)+ ?? (entry.range?.selection === undefined ? undefined : `selection ${entry.range.selection}`)+ ?? "an unnamed stretch";const logPath = join(referenceRoot, reference, "comparisons.jsonl");const log = (await readFile(logPath, "utf8").catch(() => "")).split("\n").filter((line) => line.trim().length > 0)@@ -330,16 +361,30 @@ export async function reconstructionCheck(// images, which is participation too. Only `failed` means nothing was compared..filter((entry) => entry.status === "complete" || entry.status === "pending");- // Which shots each element was compared against, in order. The gate cannot judge whether a shot was- // the right one to compare against — it does not know what the element draws — so it reports them and- // lets a reader notice a component compared against a shot that never showed it.+ // Which stretches each element was compared against, in order. The gate cannot judge whether a+ // stretch was the right one to compare against — it does not know what the element draws — so it+ // reports them and lets a reader notice a component compared against one that never showed it.const rounds = new Map<string, string[]>();+ // What timed each comparison's stand-in. A picture drawn at the estimator's pace holds every+ // element in the right place for the wrong length of time, so an element compared only that way has+ // had its layout settled and its behaviour over the window left alone.+ const bases = new Map<string, TimingBasis[]>();for (const entry of log) {if (entry.element === undefined) continue;const seen = rounds.get(entry.element) ?? [];- seen.push(entry.shot_id);+ seen.push(against(entry));rounds.set(entry.element, seen);+ const recorded = entry.stand_in?.timing?.basis;+ const basis: TimingBasis = recorded === "reference" || recorded === "estimate" || recorded === "mixed" ? recorded : "not recorded";+ bases.set(entry.element, [...(bases.get(entry.element) ?? []), basis]);}+ // One answer per element, over every comparison it has. Agreement carries through; disagreement is+ // `mixed`, which is the honest answer for an element compared once each way.+ const timingBasis = (id: string): TimingBasis => {+ const recorded = new Set(bases.get(id) ?? []);+ if (recorded.size === 0) return "not recorded";+ return recorded.size === 1 ? [...recorded][0]! : "mixed";+ };const never = drawn.filter((element) => (rounds.get(element.id) ?? []).length === 0);const unlabelled = log.filter((entry) => entry.element === undefined).length;@@ -349,23 +394,28 @@ export async function reconstructionCheck(const unknown = [...new Set([...rounds.keys()].filter((id) => !known.has(id)))];const elements: ElementReport[] = drawn.map((element) => {- const shots = rounds.get(element.id) ?? [];- const state = shots.length === 0+ const stretches = rounds.get(element.id) ?? [];+ const state = stretches.length === 0? "never compared"- : `${shots.length} comparison${shots.length === 1 ? "" : "s"} against ${[...new Set(shots)].join(", ")}`;+ : `${stretches.length} comparison${stretches.length === 1 ? "" : "s"} against ${[...new Set(stretches)].join(", ")}`;return {element: `${element.alias}:${element.tag}`,id: element.id,package_name: element.specifier,- comparisons: shots.length,- shots: [...new Set(shots)],+ comparisons: stretches.length,+ compared_against: [...new Set(stretches)],state,- // The gate cannot judge whether the shot chosen actually showed the element, so a lone comparison- // is called out for a reader to confirm rather than silently accepted.- ...(shots.length === 1 ? { note: "single comparison — confirm this shot shows the element, not a look-alike" } : {}),+ timing_basis: timingBasis(element.id),+ // The gate cannot judge whether the stretch chosen actually showed the element, so a lone+ // comparison is called out for a reader to confirm rather than silently accepted.+ ...(stretches.length === 1 ? { note: "single comparison — confirm this stretch shows the element, not a look-alike" } : {}),};});+ // Elements whose comparisons say nothing about how they behave over their window: every recorded+ // comparison was drawn at the estimator's pace, or its picture carried no record of what timed it.+ const untimed = elements.filter((element) => element.comparisons > 0 && element.timing_basis !== "reference");+const summary: string[] = [];if (never.length === 0 && running.length === 0) {summary.push(`every drawing element has been compared (${elements.length}), and every timed picture fills its window.`);@@ -374,6 +424,12 @@ export async function reconstructionCheck(if (running.length > 0) {summary.push(`${running.length} window${running.length === 1 ? "" : "s"} will empty before ${running.length === 1 ? "it ends" : "they end"}.`);}+ const compared = elements.filter((element) => element.comparisons > 0).length;+ if (compared > 0) {+ summary.push(untimed.length === 0+ ? `every compared element was looked at over a reference-timed stand-in (${compared}).`+ : `${untimed.length} of ${compared} compared elements have comparisons over a stand-in that was not reference-timed.`);+ }return {run: runPath,@@ -386,10 +442,10 @@ export async function reconstructionCheck(ids: never.map((element) => element.id),next: "Read .claude/skills/hypit/references/reconstruction/reconstruction-loop.md, then for each: "+ "render the element as the Source configures it, mock the layers a Build has not made, "- + "and compare the whole shot blind — one comparison per shot the reference shows it in.",+ + "and compare the whole stretch blind — one comparison per Segment or Selection it is drawn over.",commands: never.map((element) =>- `hypit-reference-video-tools compare_reconstruction --reference-id ${reference} `- + `--shot-id <each shot that shows ${element.id}> `+ `hypit-reference-video-tools compare_reconstruction --reference-id ${reference} --run ${runPath} `+ + `--segment <each Segment ${element.id} is drawn over> `+ `--video <rendered clip>.mp4 --element ${element.id}`),},}),@@ -406,6 +462,25 @@ export async function reconstructionCheck(note: `${unlabelled} comparison${unlabelled === 1 ? " was" : "s were"} recorded without --element and cannot be credited to one.`,},}),+ ...(untimed.length === 0 ? {} : {+ timing_next: {+ ids: untimed.map((element) => element.id),+ note: "A stand-in sized by estimate:Speech puts every element where the Source puts it and gives it "+ + "the wrong length of time to be there. Anything whose appearance is a function of elapsed time "+ + "inside its window — a progressive reveal, a typewriter, staggered rows at a fixed rate, an "+ + "enter animation scaled to the window — was therefore compared at a speed the reference never "+ + "ran at, and is unverified. A comparison with no recorded basis says the same thing, because "+ + "nothing on the record says otherwise.\n\n"+ + "Re-render each of these with render_element --reference-id "+ + `${reference} and compare again: the reference's own transcript times every Segment whose words `+ + "it carries, so each window runs for as long as the reference spends on it. What is left after "+ + "that is alignment against the speech the Build synthesizes, which is measured at "+ + "playbooks/craft/production-gates.md Gate 3.",+ commands: untimed.map((element) =>+ `hypit-reference-video-tools render_element ${runPath} --element ${element.id} `+ + `--reference-id ${reference} --segment <the Segment the shot covers> --out <rendered clip>.mp4`),+ },+ }),playback: running,...(running.length === 0 ? {} : {playback_next: "A generated take is ordered in whole seconds and its window is measured from speech the "diff --git a/packages/reference-video-tools/src/cli.ts b/packages/reference-video-tools/src/cli.tsindex fb09de2d..d5fb383d 100644--- a/packages/reference-video-tools/src/cli.ts+++ b/packages/reference-video-tools/src/cli.ts@@ -14,9 +14,10 @@ function usage(): string {" hypit-reference-video-tools observe_reference --reference-id <id> --shot-id <id> [--shot-id <id> ...] --question <text>"," hypit-reference-video-tools record_observation --reference-id <id> --key <key> --text <text>|--text-file <path>"," hypit-reference-video-tools inspect_svml_vocabulary --package <name> [--package <name> ...] [--tag <tag> ...] [--without-previews]",+ " hypit-reference-video-tools compare_reconstruction --reference-id <id> --run <build.svrun> --segment <id>|--selection <id> --video <path>|--image <path> [--question <scope>] [--element <id>]"," hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> --video <path>|--image <path> [--question <scope>] [--element <id>]"," hypit-reference-video-tools make-placeholder --out <path> --width <w> --height <h> [--color light|mid|dark|white|black|#RRGGBB] [--video] [--seconds <s>]",- " hypit-reference-video-tools render_element <build.svrun> --element <id> --out <path.png|path.mp4> [--segment <id>] [--selection <id>]",+ " hypit-reference-video-tools render_element <build.svrun> --element <id> --out <path.png|path.mp4> [--segment <id>] [--selection <id>] [--reference-id <id>]"," hypit-reference-video-tools render_previews <package-dir> [...]"," hypit-reference-video-tools preview_check <build.svrun> [<hypit.runtime.json>]"," hypit-reference-video-tools reconstruction_check <build.svrun> [--reference-id <id>]",@@ -30,14 +31,25 @@ function usage(): string {"never been compared against the reference, and which timed pictures are configured to stop before","their window ends. It asks for participation rather than convergence, so an element compared once","passes. With one prepared reference it uses that one; with several it requires --reference-id.",+ "Each element also carries the basis its comparisons were made on — reference, estimate, mixed, or",+ "not recorded — which says whether anything that changes with elapsed time inside its window has",+ "been looked at, or only its layout.","","render_element draws one element of a Source the way that Source configures it, without a Build","and without a Provider: the Canvas, frame rate, Recipe values and bindings are read from the",- "Source, the Segment skeleton comes from its own estimate:Speech, and the layers a Build has not",- "made are mocked with make-placeholder at the Canvas's size. Name the stretch in words —",- "--segment or --selection — so no reference timestamp is ever read across; without either, the",- "whole program is drawn. An --out ending .mp4, .mov or .webm writes the stretch as a clip, and any",- "other extension writes one still from the middle of it.",+ "Source, and the layers a Build has not made are mocked with make-placeholder at the Canvas's size.",+ "Name the stretch in words — --segment or --selection — so no reference timestamp is ever read",+ "across; without either, the whole program is drawn. An --out ending .mp4, .mov or .webm writes the",+ "stretch as a clip, and any other extension writes one still from the middle of it.",+ "",+ "--reference-id times the Segment skeleton from that reference's transcript. The Script was",+ "transcribed from the reference video, so its words are matched against the transcript's and each",+ "Segment runs for as long as the reference spends on them; anything whose appearance is a function",+ "of elapsed time inside its window is then compared at the pace it will be seen at. A Segment whose",+ "words the transcript does not carry is sized by the Source's own estimate:Speech, which is also",+ "what sizes every Segment when no reference is named. The result's `timing` says which of the two",+ "sized each Segment, and the same object is written to <out>.stand-in.json for",+ "compare_reconstruction to carry into the comparison log.","","render_previews draws a package's catalogue pictures from the package's own preview Source in","preview/preview.svml, preview/recipes.svs and preview/build.svrun. Which element draws which file",@@ -49,11 +61,22 @@ function usage(): string {"comparison log and never sent to the observer, so the comparison stays blind while a later gate can","still tell which elements have been compared.","",- "--video compares the whole shot instead of one frame of it, which is what removes the problem of",- "choosing a characteristic frame for an element that animates in, leaves, or is replaced within one",- "shot. The `gemini` observer receives the two clips; the `agent` observer receives two frame tiles,",- "the rendered clip tiled against the reference shot's own duration so both grids sample alike. Use",- "--image only when the shot's own visual observation states the element is completely still.",+ "compare_reconstruction takes the stretch of the reference two ways. --segment and --selection name",+ "a word range: the words come from the Author SVML of the Run given as --run, they are aligned",+ "against the reference's transcript, and the reference is cut from its own analysis video at the",+ "seconds it speaks them. That is the stretch a render covers, since a render is drawn over those",+ "same words, and one Segment routinely runs across several shots. Each end moves onto a shot",+ "boundary when one lies inside its own end word, so the pair opens and closes where the picture",+ "changes while still covering exactly the words asked for, and the rendered clip is trimmed by the",+ "seconds each end moved so both sides show the same word at the same offset. An end with no boundary",+ "inside its word leaves the pair part-way through a shot, and the prompt says how many seconds of it",+ "to read as an incomplete shot. --shot-id names a cut in the picture and compares that whole shot.",+ "",+ "--video compares the whole stretch instead of one frame of it, which is what removes the problem of",+ "choosing a characteristic frame for an element that animates in, leaves, or is replaced without a",+ "cut. The `gemini` observer receives the two clips; the `agent` observer receives two frame tiles,",+ "both tiled against the compared stretch's own duration so the grids sample alike. Use --image only",+ "when the reference's own visual observation states the element is completely still.","","make-placeholder writes a correctly-sized placeholder for a media slot the Source declares as a","generation and a Build has not filled. It is deterministic and Provider-free: the comparison loop",@@ -183,14 +206,23 @@ async function main(): Promise<void> {result = await tools.record_observation(input as { reference_id: string; key: string; text: string });} else if (command === "compare_reconstruction") {const video = one(flags, "video");+ const segment = one(flags, "segment");+ const selection = one(flags, "selection");+ const range = segment !== undefined || selection !== undefined;const input = supplied ?? {reference_id: required(flags, "reference-id"),- shot_id: required(flags, "shot-id"),+ ...(range+ ? {+ run: required(flags, "run"),+ ...(segment === undefined ? {} : { segment }),+ ...(selection === undefined ? {} : { selection }),+ }+ : { shot_id: required(flags, "shot-id") }),...(video === undefined ? { image_path: required(flags, "image") } : { video_path: video }),...(one(flags, "question") === undefined ? {} : { question: one(flags, "question") }),...(one(flags, "element") === undefined ? {} : { element: one(flags, "element") }),};- result = await tools.compare_reconstruction(input as { reference_id: string; shot_id: string; image_path?: string; video_path?: string; question?: string; element?: string });+ result = await tools.compare_reconstruction(input as { reference_id: string; shot_id?: string; segment?: string; selection?: string; run?: string; image_path?: string; video_path?: string; question?: string; element?: string });} else if (command === "inspect_svml_vocabulary") {const packages = many(flags, "package");const input = supplied ?? {@@ -218,8 +250,9 @@ async function main(): Promise<void> {out: required(flags, "out"),...(one(flags, "segment") === undefined ? {} : { segment: one(flags, "segment") }),...(one(flags, "selection") === undefined ? {} : { selection: one(flags, "selection") }),+ ...(one(flags, "reference-id") === undefined ? {} : { reference_id: one(flags, "reference-id") }),};- result = await tools.render_element(input as { run: string; element: string; out: string; segment?: string; selection?: string });+ result = await tools.render_element(input as { run: string; element: string; out: string; segment?: string; selection?: string; reference_id?: string });} else if (command === "render_previews") {if (supplied === undefined && operands.length === 0) throw new Error(`a <package-dir> is required\n\n${usage()}`);const input = supplied ?? { package_dirs: operands };diff --git a/packages/reference-video-tools/src/media.ts b/packages/reference-video-tools/src/media.tsindex c213227e..69d82ce1 100644--- a/packages/reference-video-tools/src/media.ts+++ b/packages/reference-video-tools/src/media.ts@@ -33,14 +33,22 @@ export async function command(executable: string, args: readonly string[], timeo});}-export async function probe(path: string): Promise<{ duration: number; width: number; height: number; hasAudio: boolean }> {- const raw = await command("ffprobe", ["-v", "error", "-show_entries", "format=duration:stream=codec_type,width,height", "-of", "json", path]);- const parsed = JSON.parse(raw.toString("utf8")) as { format?: { duration?: string }; streams?: readonly { codec_type?: string; width?: number; height?: number }[] };+export async function probe(path: string): Promise<{ duration: number; width: number; height: number; frameRate: number; hasAudio: boolean }> {+ const raw = await command("ffprobe", ["-v", "error", "-show_entries", "format=duration:stream=codec_type,width,height,r_frame_rate", "-of", "json", path]);+ const parsed = JSON.parse(raw.toString("utf8")) as { format?: { duration?: string }; streams?: readonly { codec_type?: string; width?: number; height?: number; r_frame_rate?: string }[] };const video = parsed.streams?.find((item) => item.codec_type === "video");const duration = Number(parsed.format?.duration);assert(Number.isFinite(duration) && duration > 0, "video duration is unavailable");assert(video?.width !== undefined && video.height !== undefined, "video dimensions are unavailable");- return { duration, width: video.width, height: video.height, hasAudio: parsed.streams?.some((item) => item.codec_type === "audio") ?? false };+ // ffprobe reports the rate as a ratio, so it is divided rather than parsed as a number. A still+ // image has no rate to report and a caller that needs one asks for it from a clip.+ const ratio = (video.r_frame_rate ?? "").split("/");+ const rate = Number(ratio[0]) / Number(ratio[1] ?? 1);+ return {+ duration, width: video.width, height: video.height,+ frameRate: Number.isFinite(rate) && rate > 0 ? rate : 0,+ hasAudio: parsed.streams?.some((item) => item.codec_type === "audio") ?? false,+ };}function distance(left: Uint8Array, right: Uint8Array): number {@@ -115,6 +123,26 @@ async function extract(path: string, args: readonly string[], target: string): Pconst TILE_COLUMNS = 3;export function tileFrames(duration: number): number { return clamp(Math.round(duration * 1.5), 4, 9); }+/**+ * One stretch of a video, as a clip of its own.+ *+ * `-ss` goes after `-i` for the same reason it does when a shot is cut: seeking the output decodes+ * from the start and lands on the frame asked for, where seeking the input lands on the keyframe+ * before it. Both sides of a comparison are cut this way, so a difference between them is a+ * difference in the pictures rather than in how far each one overshot its start.+ */+export async function cutClip(source: string, start: number, seconds: number, target: string): Promise<string> {+ return await extract(source, [+ "-i", source, "-ss", String(round(start)), "-t", String(Math.max(0.1, round(seconds))),+ "-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-c:a", "aac", "-b:a", "96k", "-movflags", "+faststart",+ ], target);+}++/** One frame of a video, at the second asked for. */+export async function cutFrame(source: string, at: number, target: string): Promise<string> {+ return await extract(source, ["-i", source, "-ss", String(round(at)), "-frames:v", "1", "-q:v", "3"], target);+}+export async function shotTile(clip: string, duration: number, target: string): Promise<string> {const frames = tileFrames(duration);const rows = Math.ceil(frames / TILE_COLUMNS);diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.tsindex 69c77aac..601f3e30 100644--- a/packages/reference-video-tools/src/tools.ts+++ b/packages/reference-video-tools/src/tools.ts@@ -11,19 +11,22 @@ import { markupSurfaceHostFacetAbi } from "@hypit/markup";import type { RegisteredSurface, SurfaceVocabulary } from "@hypit/markup";import { exactModelHostAbi } from "@hypit/model-kit";-import { renderElement, renderPreviews } from "./authoring.js";-import type { RenderElementInput, RenderPreviewsInput } from "./authoring.js";+import { authorSource, invokedFrom, referenceWords, renderElement, renderPreviews, spokenRange, standInSidecarPath } from "./authoring.js";+import type { RenderElementInput, RenderPreviewsInput, SpokenRange, StandInFocus, StandInSidecar } from "./authoring.js";import { writePlaceholder } from "./placeholder.js";import { previewCheck, reconstructionCheck } from "./checks.js";import type { PreviewCheckInput, ReconstructionCheckInput } from "./checks.js";import {assert,+ cutClip,+ cutFrame,ensureDir,prepareMedia,probe,readJson,readBytes,referenceId,+ round,shotTile,writeJson,} from "./media.js";@@ -53,7 +56,18 @@ export type InspectVocabularyInput = {};export type CompareReconstructionInput = {readonly reference_id: string;- readonly shot_id: string;+ /**+ * A stretch of the reference to compare against, named one of two ways. Exactly one is given.+ *+ * `shot_id` is a cut in the picture. `segment` and `selection` are word ranges, which is what a+ * render covers: a render is drawn over the words the Script marks, so the reference beside it is+ * the stretch that speaks those same words.+ */+ readonly shot_id?: string;+ readonly segment?: string;+ readonly selection?: string;+ /** The Run whose Author SVML the words are read from. Required with `segment` or `selection`. */+ readonly run?: string;/** A rendered still. Exactly one of `image_path` and `video_path` is given. */readonly image_path?: string;/**@@ -127,10 +141,31 @@ function stateRoot(workspaceRoot: string, reference: string): string {return join(workspaceRoot, ".hypit", "reference-video-tools", reference);}+/**+ * The stretch a word-range comparison covered: the words it was asked for, and the seconds it cut.+ *+ * The two differ by however far each end was moved onto a shot boundary lying inside its own end+ * word. `head_snapped` and `tail_snapped` say whether that end landed on a cut, which is what+ * decides whether the pair opens and closes on a whole shot or part-way through one.+ */+export type ComparedRange = {+ readonly segment?: string;+ readonly selection?: string;+ readonly words: string;+ readonly words_start_seconds: number;+ readonly words_end_seconds: number;+ readonly cut_start_seconds: number;+ readonly cut_end_seconds: number;+ readonly head_snapped: boolean;+ readonly tail_snapped: boolean;+};+/** One line per comparison performed, appended so a stopped loop still leaves its trail. */export type ComparisonRecord = {readonly at: string;- readonly shot_id: string;+ /** Which stretch was compared. A comparison names its shot or its word range, never both. */+ readonly shot_id?: string;+ readonly range?: ComparedRange;readonly element?: string;readonly image_path: string;readonly image_digest: string;@@ -139,12 +174,30 @@ export type ComparisonRecord = {readonly scoped: boolean;/** Whether the whole shot was compared as a clip, or one frame of it as a still. */readonly clip?: boolean;+ /**+ * What timed the stand-in the picture was drawn over, from the sidecar `render_element` writes+ * beside its output. Present when the picture came from `render_element`.+ */+ readonly stand_in?: StandInSidecar;};async function fileDigest(path: string): Promise<string> {return `sha256:${createHash("sha256").update(await readFile(path)).digest("hex")}`;}+/**+ * What timed the stand-in behind a rendered picture, read from beside the picture itself.+ *+ * A comparison is handed a path. Whether the stretch in it ran at the reference's pace or at an+ * estimate of it is not in the pixels, and a picture from anywhere else has no sidecar at all, so a+ * missing one is an absence to record rather than a refusal.+ */+async function standInBeside(renderedPath: string): Promise<StandInSidecar | undefined> {+ const text = await readFile(standInSidecarPath(renderedPath), "utf8").catch(() => undefined);+ if (text === undefined) return undefined;+ try { return JSON.parse(text) as StandInSidecar; } catch { return undefined; }+}+async function appendComparison(root: string, record: ComparisonRecord): Promise<void> {await appendFile(join(root, "comparisons.jsonl"), `${JSON.stringify(record)}\n`, "utf8");}@@ -362,6 +415,139 @@ async function shotFromBound(root: string, index: number, bound: { start: number};}+/** One stretch of the reference cut to a word range, with the rendered clip trimmed to match it. */+type RangeCut = {+ readonly record: ComparedRange;+ readonly reference: string;+ readonly rendered: string;+ readonly referenceTile: string;+ readonly renderedTile: string;+ /** How long the cut runs, which is the duration both frame grids are sampled against. */+ readonly seconds: number;+ readonly referenceSeconds: number;+ readonly renderedSeconds: number;+ /** What the observer is told about an end that lands part-way through a shot. Empty when neither does. */+ readonly incomplete: string;+};++/**+ * Every cut in the reference, in seconds.+ *+ * A stretch longer than fifteen seconds is divided into parts so each clip stays short enough to+ * observe, and the divisions between those parts are not cuts: the picture runs straight through+ * them. Only the first part of a group begins where the picture changes.+ */+function referenceCuts(state: ReferenceState): readonly number[] {+ return [...state.shots.filter((shot) => shot.part === 1).map((shot) => shot.start_seconds), state.shots.at(-1)!.end_seconds];+}++/**+ * Cut the reference to the stretch that speaks a range of words, and trim the render to match.+ *+ * Each end is moved onto a shot boundary when one lies inside its own end word. A boundary inside+ * the first word cannot add or drop a whole word — the word is still spoken across the cut either+ * way — so the pair then opens and closes on the picture changing rather than part-way through a+ * shot, and still covers exactly the words asked for. An end with no boundary inside its word stays+ * where the word puts it.+ *+ * The render is trimmed by the seconds each end moved. Its frames come from the same word timings,+ * so a second of reference at the head is a second of render at the head, and taking it off both+ * keeps the two stretches showing the same word at the same offset.+ */+async function cutWordRange(+ state: ReferenceState,+ root: string,+ focus: StandInFocus,+ range: SpokenRange,+ renderedPath: string,+ clip: boolean,+): Promise<RangeCut> {+ const cuts = referenceCuts(state);+ const head = cuts.find((at) => at >= range.first.startSeconds && at < range.first.endSeconds);+ const tail = [...cuts].reverse().find((at) => at > range.last.startSeconds && at <= range.last.endSeconds);+ const start = head ?? range.startSeconds;+ const end = tail ?? range.endSeconds;+ const seconds = end - start;+ assert(seconds > 0, `${focus.selection ?? focus.segment} spans no time in the reference`);++ const dir = join(root, "ranges");+ await ensureDir(dir);+ const label = focus.selection === undefined ? `segment-${focus.segment}` : `selection-${focus.selection}`;+ const reference = join(dir, `${label}-reference.${clip ? "mp4" : "jpg"}`);+ // The reference's own analysis video, which is the whole reference at the size every other+ // observation reads it at. A shot clip covers a cut in the picture and would cover the wrong words.+ if (clip) await cutClip(state.analysis_video_ref, start, seconds, reference);+ else await cutFrame(state.analysis_video_ref, start + seconds / 2, reference);++ let rendered = renderedPath;+ let referenceSeconds = 0;+ let renderedSeconds = 0;+ if (clip) {+ const drawn = await probe(renderedPath);+ assert(drawn.frameRate > 0, `${renderedPath} reports no frame rate, so it cannot be trimmed to the cut`);+ // Frames are the only unit the file has, so the seconds each end moved are converted at the+ // render's own rate and taken off as whole frames.+ const headFrames = Math.round((start - range.startSeconds) * drawn.frameRate);+ const tailFrames = Math.round((range.endSeconds - end) * drawn.frameRate);+ rendered = join(dir, `${label}-rendered.mp4`);+ await cutClip(renderedPath, headFrames / drawn.frameRate, drawn.duration - (headFrames + tailFrames) / drawn.frameRate, rendered);+ referenceSeconds = (await probe(reference)).duration;+ renderedSeconds = (await probe(rendered)).duration;+ // Two stretches of different lengths cannot be read side by side: whatever is at a given offset+ // in one is at a different word in the other, which is the defect this cut exists to remove.+ // Each side lands on its own frame grid, so they agree to within a frame rather than exactly.+ assert(Math.abs(referenceSeconds - renderedSeconds) <= 2 / drawn.frameRate,+ `the cut reference runs ${round(referenceSeconds)}s and the trimmed render ${round(renderedSeconds)}s; they must cover the same stretch`);+ }++ // An end that could not be moved onto a cut opens or closes part-way through a shot. How much of+ // the stretch that is, is measurable here, and saying it is what keeps the incomplete shot from+ // being read as a difference.+ const enclosing = (at: number): { readonly start: number; readonly end: number } => {+ let index = 0;+ for (let candidate = 0; candidate + 1 < cuts.length; candidate += 1) if (cuts[candidate]! <= at) index = candidate;+ return { start: cuts[index]!, end: cuts[index + 1]! };+ };+ const opening = enclosing(start);+ const closing = enclosing(end);+ const lead = head === undefined ? Math.min(opening.end, end) - start : 0;+ const trail = tail === undefined ? end - Math.max(closing.start, start) : 0;+ const ends = !clip+ ? []+ // A range short enough to sit inside one shot has both its ends in that shot, and naming them+ // separately describes the same seconds twice.+ : head === undefined && tail === undefined && opening.start === closing.start+ ? [`All ${seconds.toFixed(2)} seconds fall inside a single shot that begins before this stretch does and ends after it.`]+ : [+ ...(lead > 0 ? [`The first ${lead.toFixed(2)} seconds fall inside a shot that begins before this stretch does.`] : []),+ ...(trail > 0 ? [`The last ${trail.toFixed(2)} seconds fall inside a shot that ends after this stretch does.`] : []),+ ];++ return {+ record: {+ ...(focus.segment === undefined ? {} : { segment: focus.segment }),+ ...(focus.selection === undefined ? {} : { selection: focus.selection }),+ words: `${range.first.text} … ${range.last.text}`,+ words_start_seconds: round(range.startSeconds),+ words_end_seconds: round(range.endSeconds),+ cut_start_seconds: round(start),+ cut_end_seconds: round(end),+ head_snapped: head !== undefined,+ tail_snapped: tail !== undefined,+ },+ reference,+ rendered,+ referenceTile: join(dir, `${label}-reference-tile.jpg`),+ renderedTile: join(dir, `${label}-rendered-tile.jpg`),+ seconds,+ referenceSeconds: round(referenceSeconds),+ renderedSeconds: round(renderedSeconds),+ incomplete: ends.length === 0+ ? ""+ : `\n\n${ends.join(" ")} Read ${ends.length === 1 ? "that stretch as an incomplete shot" : "those stretches as incomplete shots"} and report no differences from ${ends.length === 1 ? "it" : "them"}.`,+ };+}+function prepared(state: ReferenceState): boolean {return state.transcript?.status === "complete"&& state.people_and_product?.status === "complete"@@ -757,8 +943,9 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVasync compare_reconstruction(input): Promise<Record<string, unknown>> {const state = await loadState(input.reference_id);- const shot = state.shots.find((candidate) => candidate.shot_id === input.shot_id);- assert(shot !== undefined, `shot ${input.shot_id} does not exist in reference ${input.reference_id}`);+ const root = stateRoot(workspaceRoot, state.reference_id);+ const named = [input.shot_id, input.segment, input.selection].filter((value) => value !== undefined);+ assert(named.length === 1, "name exactly one of shot_id, segment and selection");const supplied = [input.image_path, input.video_path].filter((value) => value !== undefined);assert(supplied.length === 1, "supply exactly one of image_path and video_path");const clip = input.video_path !== undefined;@@ -766,23 +953,55 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst file = await stat(renderedPath).catch(() => undefined);assert(file?.isFile(), `${clip ? "video_path" : "image_path"} is not a file: ${renderedPath}`);const scope = input.question?.trim() ?? "";+ const standIn = await standInBeside(renderedPath);const observer: Observer = state.observer ?? "gemini";const { ask, pending } = await askerFor(observer, state);+ // Which stretch of the reference the render is put beside. A render covers the words a Segment+ // or a Selection marks, so that is what the reference is cut to; a shot is a cut in the picture+ // and is compared as the whole shot it already is.+ let shot: Shot | undefined;+ let cut: RangeCut | undefined;+ let referenceMedia: string;+ let renderedMedia = renderedPath;+ let stretchSeconds: number;+ if (input.shot_id !== undefined) {+ shot = state.shots.find((candidate) => candidate.shot_id === input.shot_id);+ assert(shot !== undefined, `shot ${input.shot_id} does not exist in reference ${input.reference_id}`);+ referenceMedia = clip ? shot.clip_ref : shot.representative_frame_ref;+ stretchSeconds = shot.duration_seconds;+ } else {+ assert(input.run !== undefined,+ "run is required with segment or selection: the words are read from the Run's Author SVML");+ const { path: svmlPath } = await authorSource(resolve(invokedFrom(), input.run));+ const focus: StandInFocus = {+ ...(input.segment === undefined ? {} : { segment: input.segment }),+ ...(input.selection === undefined ? {} : { selection: input.selection }),+ };+ const range = await spokenRange(svmlPath, focus, await referenceWords(state.reference_id));+ cut = await cutWordRange(state, root, focus, range, renderedPath, clip);+ referenceMedia = cut.reference;+ renderedMedia = cut.rendered;+ stretchSeconds = cut.seconds;+ }+// A clip comparison reads the whole stretch on both sides. The observer that reads video is// given the two clips; the observer that reads pictures is given two frame tiles, which is the- // same degradation `prepare_reference` already applies to a shot. Tiling the rendered clip- // against the reference shot's own duration makes `tileFrames` choose the same frame count and- // layout for both, so the two grids are read side by side rather than as different samplings.- let referenceMedia = clip ? shot.clip_ref : shot.representative_frame_ref;- let renderedMedia = renderedPath;+ // same degradation `prepare_reference` already applies to a shot. Tiling both against the same+ // duration makes `tileFrames` choose the same frame count and layout for each, so the two grids+ // are read side by side rather than as different samplings.if (clip && observer === "agent") {- assert(shot.frames_tile_ref !== null,- `shot ${shot.shot_id} has no frame tile; re-prepare the reference with --redo all --observer agent`);- const tile = join(stateRoot(workspaceRoot, state.reference_id), "shots", `${shot.shot_id}-reconstruction.jpg`);- await shotTile(renderedPath, shot.duration_seconds, tile);- referenceMedia = shot.frames_tile_ref;- renderedMedia = tile;+ if (shot !== undefined) {+ assert(shot.frames_tile_ref !== null,+ `shot ${shot.shot_id} has no frame tile; re-prepare the reference with --redo all --observer agent`);+ referenceMedia = shot.frames_tile_ref;+ renderedMedia = await shotTile(renderedPath, stretchSeconds, join(root, "shots", `${shot.shot_id}-reconstruction.jpg`));+ } else {+ // A word range is cut when it is asked for, so neither side has a prepared tile and both+ // are built here, from the two cuts.+ referenceMedia = await shotTile(cut!.reference, stretchSeconds, cut!.referenceTile);+ renderedMedia = await shotTile(cut!.rendered, stretchSeconds, cut!.renderedTile);+ }}const unit = clip && observer !== "agent" ? "video clips" : "still images";@@ -794,15 +1013,16 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst differences = await ask("comparison", {media: [referenceMedia, renderedMedia],instruction: `You compare two supplied ${unit} and describe their visible differences in natural language only. You are not told how either was made. Do not write code, markup, SVML, component names, or production advice.`,- prompt: `Two ${unit} are supplied in order: one, then two.${reading}${scope.length === 0 ? "" : `\n\nLimit the comparison to this: ${scope}`}\n\nDescribe every visible difference between them: layout and arrangement, the position and size of each element, cropping and margins, colour, typeface, weight, letter and line spacing, alignment, outline or stroke, shadow, glow, borders and corner treatment, and anything present in one and absent from the other.${clip ? " Also describe differences in what changes over the stretch: what appears, what leaves, in what order, and how anything moves." : ""} State plainly which differences are large enough to read as a different design and which are minor. If they are visually equivalent, say exactly that.\n\nDo not speculate about how either was produced, which one is a source, or which one is a copy. Return natural language only.`,+ prompt: `Two ${unit} are supplied in order: one, then two.${reading}${cut?.incomplete ?? ""}${scope.length === 0 ? "" : `\n\nLimit the comparison to this: ${scope}`}\n\nDescribe every visible difference between them: layout and arrangement, the position and size of each element, cropping and margins, colour, typeface, weight, letter and line spacing, alignment, outline or stroke, shadow, glow, borders and corner treatment, and anything present in one and absent from the other.${clip ? " Also describe differences in what changes over the stretch: what appears, what leaves, in what order, and how anything moves." : ""} State plainly which differences are large enough to read as a different design and which are minor. If they are visually equivalent, say exactly that.\n\nDo not speculate about how either was produced, which one is a source, or which one is a copy. Return natural language only.`,});// The answer is deliberately not cached — every iteration is a fresh comparison. What is// recorded is that a comparison happened, so a gate can tell an element that was looked at// from one that never was. The loop is allowed to stop with differences remaining, so this// records participation rather than convergence.- await appendComparison(stateRoot(workspaceRoot, state.reference_id), {+ await appendComparison(root, {at: new Date().toISOString(),- shot_id: shot.shot_id,+ ...(shot === undefined ? {} : { shot_id: shot.shot_id }),+ ...(cut === undefined ? {} : { range: cut.record }),...(input.element === undefined ? {} : { element: input.element.trim() }),image_path: renderedPath,image_digest: await fileDigest(renderedPath),@@ -810,14 +1030,21 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVstatus: differences.status,scoped: scope.length > 0,clip,+ ...(standIn === undefined ? {} : { stand_in: standIn }),});return {reference_id: state.reference_id,observer,- shot_id: shot.shot_id,+ ...(shot === undefined ? {} : { shot_id: shot.shot_id }),+ ...(cut === undefined ? {} : {+ range: cut.record,+ ...(clip ? { reference_seconds: cut.referenceSeconds, rendered_seconds: cut.renderedSeconds } : {}),+ ...(cut.incomplete.length === 0 ? {} : { incomplete_ends: cut.incomplete.trim() }),+ }),compared: clip ? "clip" : "still",reference_ref: referenceMedia,rendered_ref: renderedMedia,+ ...(standIn === undefined ? {} : { stand_in: standIn }),differences,unresolved: differences.status === "complete" ? [] : ["differences"],pending_observations: pending,