Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
fix: one timing source for render and comparison, and a round that runs as one call (#59)
rponeawa1241b7de2773
diff --git a/.agents/skills/hypit/references/local-author-package.md b/.agents/skills/hypit/references/local-author-package.mdindex 9d78c48e..d45d49eb 100644--- a/.agents/skills/hypit/references/local-author-package.md+++ b/.agents/skills/hypit/references/local-author-package.md@@ -56,6 +56,19 @@ A call read out of any source that is not the installed Distribution — an oldecheckout, a snippet in an issue, something you remember — is a name that may no longer exist, andwhat it costs is a compile error at the end of a package rather than at the line that borrowed it.+Four are needed by every component that draws and are stated nowhere in the guides, so they are here.+Read the rest from source; these are the ones that stop a package before it starts:++| From | Signature |+|---|---|+| `@hypit/elaborator` | `sealGraphFragment(fragment: Omit<GraphFragment, "format" \| "id">): GraphFragment` |+| `@hypit/composition` | `sealVisualTrack(value: Omit<VisualTrack, "kind">): VisualTrack` |+| `@hypit/component-kit` | `ProducerHandlerContext` — `{ command: InvokeProducerCommand; producer: ProducerRef; inputs: Readonly<Record<string, TypedRecord>> }` |++The fourth is the shape of a filled media slot, which the boundary rules below describe in words: a+blob input arrives in `inputs` as the `BlobRef` itself, not wrapped inline the way an authored value+is, so read its `mediaType` off the Artifact rather than assuming what the slot's name suggests.+## Package boundaryPlace the package at `<project>/packages/local-<slug>/`, named in the project's own scope such asdiff --git a/.agents/skills/hypit/references/playbooks/craft/captions.md b/.agents/skills/hypit/references/playbooks/craft/captions.mdindex f240e100..91caf404 100644--- a/.agents/skills/hypit/references/playbooks/craft/captions.md+++ b/.agents/skills/hypit/references/playbooks/craft/captions.md@@ -77,6 +77,12 @@ design that wants no Cue box reads as having nothing to say about them. Write thinvisible box is `background="#00000000"` with `padding` and `radius` at `0`, which is a value, notan omission.+**`padding` is a string; `radius` and `size` are numbers.** `padding: 0` is refused with *Fine Caption+Recipe padding must be a string* while `radius: 0` beside it is correct, and the same holds for+`active-box-padding`. The reason is that padding admits a pair — `padding: "8 12"` is eight vertical+and twelve horizontal — which no JSON number can carry, so it travels as text and is split on the+space. A single value is still written as one: `padding: "0"`.+`cue-min-words` and `cue-max-words` are not read. `docs/quickstart/styles.md` and`docs/quickstart/composition.md` still show them; a Recipe carrying either is refused as an unknownproperty, since the Style admits exactly the required keys plus the documented optional ones.diff --git a/.agents/skills/hypit/references/preview.md b/.agents/skills/hypit/references/preview.mdindex 2608959e..7a91929f 100644--- a/.agents/skills/hypit/references/preview.md+++ b/.agents/skills/hypit/references/preview.md@@ -24,9 +24,10 @@ The same check is also a bin of its own:hypit-preview-check /path/to/project/build.svrun```-Prefer the subcommand. It honours `--package-root`, so it reaches a project's own-`packages/local-<slug>/` from wherever you are standing, and it is a subcommand of the bin the-reconstruction route already runs. The bare bin fixes the package root to the Run file's own+Prefer the subcommand. It takes `--package-root <dir>`, which is where the packages the Source+imports are resolved from — a project's own `packages/local-<slug>/` are installed against the+project root, so name it there and the check reaches them from wherever you are standing. Without+the flag the working directory is used. The bare bin fixes the package root to the Run file's owndirectory and takes no flags, so reach for it when the Run sits at the project root and you want theprose summary rather than a JSON result.diff --git a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.mdindex 11bb4872..e81c2e14 100644--- a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md+++ b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md@@ -146,12 +146,51 @@ coverage round, and it comes before any repair — collect the whole differencerepair against it. Doing it the other way makes the comparison after a repair look like a thirdattempt at a stretch that was never examined.+### A caption Style is compared once, where it first appears++Captions are the one element where the stretches are not different questions. A caption system is one+Style applied to whatever words fall under it, so two stretches drawn in the same Style differ only in+which words they hold — the typeface, size, weight, box, padding and placement are the same+declaration in both, and reading the second one returns the answer the first already gave.++So compare **one stretch per distinct Style**, at the first occurrence of that Style. A program with+one default Style and a `caption:Use` override for one Selection is two comparisons, wherever those+two first appear, not one per Segment. This is the same reading `continuity.md` applies to every other+system that spans cuts: the system is authored once, so it is read once.++What is not covered by that is anything a Style does not decide. A Cue that overflows the frame is a+Segment nobody broke with `||` rather than a Style at the wrong width — `../playbooks/craft/captions.md`+says so — and it belongs to the stretch whose words are long, not to the Style. Where the reference+shows a caption behaving differently somewhere, that is a stretch worth its own comparison; where it+shows the same design drawn over different words, it is not.+**Render every stretch first, then send them all at once.** No comparison's question depends onanother's answer, so nothing is gained by waiting: rendering one and comparing it before rendering thenext makes the round as long as the sum of its parts. Render the whole set, then issue thecomparisons together — concurrent requests on the `gemini` observer, one subagent each on the `agent`observer, and one after another only where the harness has no subagents.+On the `gemini` observer the whole round is one call. Write the comparisons to a JSON file — the same+objects the flags produce, minus `reference_id` — and hand the file over:++```json+[+ {"run": "projects/<name>/build.svrun", "segment": "pro",+ "video_path": "projects/<name>/renders/board-pro.mp4", "element": "board"},+ {"run": "projects/<name>/build.svrun", "selection": "wait",+ "video_path": "projects/<name>/renders/marks-wait.mp4", "element": "marks"}+]+```++```+hypit-reference-video-tools compare_reconstruction --reference-id <id> --batch round-1.json+```++It paces the requests itself, keeps each comparison's derived cuts apart, and returns them under+`comparisons`. One that fails arrives under `failures` beside the input that produced it and takes+only itself down. Do not write a shell script to fan these out: the pacing, the per-comparison result+and the isolation are what the batch is for, and a hand-rolled loop has none of them.+That is a change in how long the round takes, not in what it costs. The ceilings below count repairs,and a round is one look however many stretches it covers.diff --git a/docs/guide/author-packages.md b/docs/guide/author-packages.mdindex 20e48ada..3426c429 100644--- a/docs/guide/author-packages.md+++ b/docs/guide/author-packages.md@@ -68,7 +68,6 @@ export const myComponentMarkupSurfaces = [{mode: "structured",outputs: [ /* Types this syntax may author */ ],vocabulary: { /* what this element is, and what it looks like — see step 5 */ },- implementation: { digest: "sha256:..." },}] as const;```@@ -246,3 +245,10 @@ authority.| `packages/media-track/` | Track with Item/Sequence, layer, motion and handoff behavior || `packages/typography-track/` | Typography overlay Track || `packages/film/` | Composition target that consumes peer Tracks |+| `packages/ranking/` | A Program that lays out cells and fills media slots, with `schedule.ts` and `render.ts` as its role files |+| `packages/comment-sticker/` | The same roles under different filenames — `program.ts` and `author.ts` |+| `packages/screen-overlay/` | An overlay whose geometry is computed rather than authored |++The filenames differ between packages: what `component-anatomy.md` calls the Value layer and the+lowering is `schedule.ts`/`render.ts` in `ranking`, `program.ts`/`lower.ts` in `media-track`, and+`program.ts`/`author.ts` in `comment-sticker`. Find a role by what it exports, not by its filename.diff --git a/packages/reference-video-tools/src/authoring.ts b/packages/reference-video-tools/src/authoring.tsindex 4472e06f..0dc5e391 100644--- a/packages/reference-video-tools/src/authoring.ts+++ b/packages/reference-video-tools/src/authoring.ts@@ -153,7 +153,7 @@ function nearestPackageRoot(start: string): string | undefined {/** Where a relative path on the command line is measured from. */export function invokedFrom(): string {- return process.env.INIT_CWD ?? process.cwd();+ return process.cwd();}/** The Script body, which `parseScript` takes on its own. */@@ -495,16 +495,29 @@ export async function spokenRange(`the reference's transcript carries none of the words of ${focus.selection ?? focus.segment}, so the seconds it spoke them at are unknown`);const spans = wordSpans(parsed.tokens.length, pairs, reference);- const word = (index: number): ReferenceWord => ({++ // Where the range closes. A render gives each word the screen until the next one starts, so the+ // pause the reference leaves between two words belongs to the word before it and a Take covers its+ // Segment without holes — `standInTakes` says why. A Selection closing part-way through a Segment+ // is therefore drawn up to the next word's start, and cutting the reference at the last word's end+ // instead leaves the reference short of the render by that pause. The Segment's own last word has+ // no next word inside it and closes on its own end, which is the length the take was given.+ const holder = parsed.segments.find((item) => end - 1 >= item.tokenStart && end - 1 < item.tokenEndExclusive);+ const closesMidSegment = holder !== undefined && end < holder.tokenEndExclusive;+ const endSeconds = closesMidSegment ? spans[end]!.start : spans[end - 1]!.end;++ const word = (index: number, until = spans[index]!.end): ReferenceWord => ({text: parsed.tokens[index]!.text,startSeconds: spans[index]!.start,- endSeconds: spans[index]!.end,+ endSeconds: until,});return {startSeconds: spans[start]!.start,- endSeconds: spans[end - 1]!.end,+ endSeconds,first: word(start),- last: word(end - 1),+ // The end word carries the same close, so moving the end onto a shot boundary inside it looks at+ // the seconds the render actually drew rather than a shorter word.+ last: word(end - 1, endSeconds),words: end - start,matched,};@@ -515,10 +528,28 @@ export async function standInTakes(frameRate: number,focus: StandInFocus = {},reference: readonly ReferenceWord[] = [],+ timingSource: string = svmlPath,): Promise<StandInTakes> {const svml = await readFile(svmlPath, "utf8");const body = scriptBody(svml);const parsed = parseScript(svmlPath, body.text, body.offset);++ // Which Script the reference is aligned against. A render is drawn from a fragment holding one+ // Segment, and aligning that fragment's words against the whole transcript is not the same+ // alignment the whole Script produces: a word that matched inside a longer run of context stops+ // matching, and the unmatched runs at the fragment's own two ends are interpolated from the words+ // just outside them rather than from the neighbouring Segments. The comparison reads the whole+ // Source, so a render timed from the fragment covers seconds the comparison never cuts.+ //+ // So the alignment is made once, against the whole Source, and each Segment takes its own slice out+ // of it by id. The fragment is a byte-for-byte cut, so a Segment holds the same words in both.+ const timed = timingSource === svmlPath+ ? parsed+ : await (async () => {+ const text = await readFile(timingSource, "utf8");+ const whole = scriptBody(text);+ return parseScript(timingSource, whole.text, whole.offset);+ })();const sheets = await recipeSheets(svml, svmlPath);const { policies, shared, policyCount } = policiesBySegment(svml, sheets);@@ -542,8 +573,8 @@ export async function standInTakes(const words = reference.length === 0? undefined: (() => {- const pairs = alignWords(parsed.tokens.map((token) => normalizeWord(token.text)), reference.map((word) => normalizeWord(word.text)));- return pairs.size === 0 ? undefined : { pairs, spans: wordSpans(parsed.tokens.length, pairs, reference) };+ const pairs = alignWords(timed.tokens.map((token) => normalizeWord(token.text)), reference.map((word) => normalizeWord(word.text)));+ return pairs.size === 0 ? undefined : { pairs, spans: wordSpans(timed.tokens.length, pairs, reference) };})();const takes: StandInTake[] = [];@@ -560,13 +591,14 @@ export async function standInTakes(// A Segment is timed from the reference when the reference was heard saying some of its words.// The decision is made per Segment: an ordinary transcription difference costs one Segment its// reference clock and leaves the rest of the Script on it.+ const whole = timed.segments.find((item) => item.id === segment.id);let matched = 0;- if (words !== undefined) {- for (let index = segment.tokenStart; index < segment.tokenEndExclusive; index += 1) if (words.pairs.has(index)) matched += 1;+ if (words !== undefined && whole !== undefined) {+ for (let index = whole.tokenStart; index < whole.tokenEndExclusive; index += 1) if (words.pairs.has(index)) matched += 1;}- const spoken = words === undefined || matched === 0+ const spoken = words === undefined || whole === undefined || matched === 0? undefined- : words.spans.slice(segment.tokenStart, segment.tokenEndExclusive);+ : words.spans.slice(whole.tokenStart, whole.tokenEndExclusive);// How long the Segment runs. The reference's own words when it was heard saying them, the// estimator the Source already trusts when it was not.@@ -850,7 +882,7 @@ export async function renderElement(input: RenderElementInput): Promise<Record<sconst reference = input.reference_id?.trim();const spoken = reference === undefined || reference.length === 0 ? [] : await referenceWords(reference);- const { takes, selections, frameCount: programFrames, timing } = await standInTakes(sourcePath, frameRate, focus, spoken);+ const { takes, selections, frameCount: programFrames, timing } = await standInTakes(sourcePath, frameRate, focus, spoken, svmlPath);const framesBySegment = new Map(takes.map((item) => [item.segmentId, item.take.segment.endFrameExclusive]));// A Selection's window in frames, summed over the stand-in tokens it covers. This is the same word// span the Source binds to, carried into frames by the same clock that sized the Segment.diff --git a/packages/reference-video-tools/src/checks.ts b/packages/reference-video-tools/src/checks.tsindex 8a9395c3..2d69cd54 100644--- a/packages/reference-video-tools/src/checks.ts+++ b/packages/reference-video-tools/src/checks.ts@@ -15,7 +15,7 @@ import type { StudioSession } from "@hypit/studio/src/session.js";import { inspectStudioRun } from "@hypit/studio/src/studio-preflight.js";import { invokedFrom, referenceRoot, scriptBody } from "./authoring.js";-import { assert } from "./media.js";+import { assert, round } from "./media.js";export type PreviewCheckInput = {readonly run: string;@@ -173,6 +173,185 @@ type CoverageReport = {readonly gaps: readonly CoverageGap[];};+type Edge = "left" | "top" | "right" | "bottom";++/** One edge of a Frame that lands outside the Canvas, and how far past it reaches. */+type FrameEdge = {+ readonly edge: Edge;+ /** The value as the Source writes it. */+ readonly declared: string;+ /** How far past the Canvas edge it sits, as a share of the Canvas and in Canvas pixels. */+ readonly outside_percent: number;+ readonly outside_pixels: number;+};++/** One Frame that does not sit wholly inside its Canvas, and the elements drawn into it. */+type OutOfBoundsFrame = {+ readonly frame: string;+ readonly canvas: string;+ /** Every element that names this Frame, by its own id. */+ readonly elements: readonly string[];+ readonly edges: readonly FrameEdge[];+ /** How much of the Frame's area lands off the Canvas, from 0 to 1. */+ readonly outside_fraction: number;+};++/** A rectangle in Canvas pixels. The origin is top-left and y increases downward. */+type Box = { readonly left: number; readonly top: number; readonly right: number; readonly bottom: number };++/**+ * Every Frame that reaches past the Canvas it is measured inside.+ *+ * A Frame places its four edges against a Canvas, and an edge outside 0%–100% puts that much of+ * whatever is drawn into it off the picture. This is arithmetic on the Source: no threshold, no+ * measurement of a rendered frame, no observer.+ *+ * It is the one thing on this route a comparison cannot see. An element authored mostly below the+ * bottom edge is drawn at the Canvas the Source declares, and so is the stand-in it is compared+ * against, so the render and the reference both put it in the same place off the edge and agree with+ * each other. Both are wrong the same way, which reads as correct.+ *+ * Reaching past the edge is also how a great deal of correct authoring works: an element that slides+ * in from off-screen is outside at the start of its window, and a full-bleed picture is routinely+ * declared past the edge so a `fit` crops it rather than letterboxing it. The arithmetic is the same+ * either way, so what this produces is a list of candidates for a reader to answer one at a time. It+ * never decides `passed`: the Source cannot tell an intended overhang from an unintended one, and a+ * gate that ruled on it would be ruling on something it cannot see.+ *+ * `within` names a Canvas or a parent Frame, and a length is a number followed by `px` or `%`, a+ * percentage resolving against the parent's width on the x axis and its height on the y axis. So a+ * Frame is resolved by resolving whatever it is written inside and measuring its own edges against+ * that rectangle, however deep the chain runs.+ */+function framesPastTheCanvas(svml: string): readonly OutOfBoundsFrame[] {+ const canvases = new Map<string, Box>();+ for (const match of svml.matchAll(/<space:Canvas\b([^>]*?)\/?>/gsu)) {+ const attributes = match[1] ?? "";+ const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];+ const width = Number(/\bwidth="(\d+)"/u.exec(attributes)?.[1]);+ const height = Number(/\bheight="(\d+)"/u.exec(attributes)?.[1]);+ if (id !== undefined && Number.isFinite(width) && Number.isFinite(height)) {+ canvases.set(id, { left: 0, top: 0, right: width, bottom: height });+ }+ }++ const declared = new Map<string, { readonly within: string; readonly edges: Readonly<Record<Edge, string>> }>();+ for (const match of svml.matchAll(/<space:Frame\b([^>]*?)\/?>/gsu)) {+ const attributes = match[1] ?? "";+ const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];+ const within = /\bwithin=\{([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];+ const written = (name: Edge): string | undefined => new RegExp(`\\b${name}="([^"]+)"`, "u").exec(attributes)?.[1];+ const left = written("left");+ const top = written("top");+ const right = written("right");+ const bottom = written("bottom");+ if (id === undefined || within === undefined) continue;+ if (left === undefined || top === undefined || right === undefined || bottom === undefined) continue;+ declared.set(id, { within, edges: { left, top, right, bottom } });+ }++ const measure = (value: string, span: number, origin: number): number | undefined => {+ const parsed = /^\s*(-?\d+(?:\.\d+)?)\s*(%|px)\s*$/u.exec(value);+ if (parsed === null) return undefined;+ const number = Number(parsed[1]);+ return parsed[2] === "%" ? origin + span * number / 100 : origin + number;+ };++ const resolving = new Set<string>();+ const resolved = new Map<string, Box | undefined>();+ const resolve = (id: string): Box | undefined => {+ const canvas = canvases.get(id);+ if (canvas !== undefined) return canvas;+ if (resolved.has(id)) return resolved.get(id);+ // A Frame written inside itself is refused by `hypit check`; this keeps the walk finite anyway.+ if (resolving.has(id)) return undefined;+ const frame = declared.get(id);+ if (frame === undefined) return undefined;+ resolving.add(id);+ const parent = resolve(frame.within);+ resolving.delete(id);+ let box: Box | undefined;+ if (parent !== undefined) {+ const width = parent.right - parent.left;+ const height = parent.bottom - parent.top;+ const left = measure(frame.edges.left, width, parent.left);+ const top = measure(frame.edges.top, height, parent.top);+ const right = measure(frame.edges.right, width, parent.left);+ const bottom = measure(frame.edges.bottom, height, parent.top);+ if (left !== undefined && top !== undefined && right !== undefined && bottom !== undefined) {+ box = { left, top, right, bottom };+ }+ }+ resolved.set(id, box);+ return box;+ };++ // Which Canvas each Frame is finally measured inside, found by following `within` up to one.+ const canvasOf = (id: string): string | undefined => {+ const seen = new Set<string>();+ let at: string | undefined = id;+ while (at !== undefined && !canvases.has(at)) {+ if (seen.has(at)) return undefined;+ seen.add(at);+ at = declared.get(at)?.within;+ }+ return at;+ };++ // Which elements draw into each Frame. A Track names it `frame=`, and a speech Track names the Frame+ // it draws each Take into `visual-frame=`, so any attribute whose name ends in `frame` counts.+ const bound = new Map<string, string[]>();+ for (const match of svml.matchAll(/<([a-z][a-z0-9-]*:[A-Za-z][A-Za-z0-9]*)\b([^>]*?)\/?>/gsu)) {+ const tag = match[1] ?? "";+ const attributes = match[2] ?? "";+ const id = /\bid="([^"]+)"/u.exec(attributes)?.[1] ?? tag;+ for (const reference of attributes.matchAll(/\b[a-z-]*frame=\{([A-Za-z0-9_-]+)\}/gu)) {+ const frame = reference[1] ?? "";+ const named = bound.get(frame) ?? [];+ if (!named.includes(id)) named.push(id);+ bound.set(frame, named);+ }+ }++ const report: OutOfBoundsFrame[] = [];+ for (const [id, frame] of declared) {+ const box = resolve(id);+ const canvasId = canvasOf(id);+ const canvas = canvasId === undefined ? undefined : canvases.get(canvasId);+ if (box === undefined || canvas === undefined || canvasId === undefined) continue;+ const width = canvas.right - canvas.left;+ const height = canvas.bottom - canvas.top;+ // An edge is outside when it is outside the Canvas at all, on either side. A Frame parked wholly+ // off the left has both its x edges past the left edge, and naming only the one on its own side+ // would report half of where it sits.+ const outside = (value: number, low: number, high: number): number => Math.max(low - value, value - high, 0);+ const past: readonly { readonly edge: Edge; readonly outside: number; readonly span: number }[] = [+ { edge: "left", outside: outside(box.left, canvas.left, canvas.right), span: width },+ { edge: "top", outside: outside(box.top, canvas.top, canvas.bottom), span: height },+ { edge: "right", outside: outside(box.right, canvas.left, canvas.right), span: width },+ { edge: "bottom", outside: outside(box.bottom, canvas.top, canvas.bottom), span: height },+ ];+ const edges = past.filter((item) => item.outside > 0).map((item) => ({+ edge: item.edge,+ declared: frame.edges[item.edge],+ outside_percent: round(item.outside / item.span * 100),+ outside_pixels: round(item.outside),+ }));+ if (edges.length === 0) continue;+ const area = Math.max(0, box.right - box.left) * Math.max(0, box.bottom - box.top);+ const inside = Math.max(0, Math.min(box.right, canvas.right) - Math.max(box.left, canvas.left))+ * Math.max(0, Math.min(box.bottom, canvas.bottom) - Math.max(box.top, canvas.top));+ report.push({+ frame: id,+ canvas: canvasId,+ elements: bound.get(id) ?? [],+ edges,+ outside_fraction: round(area === 0 ? 1 : 1 - inside / area),+ });+ }+ return report;+}+/*** Report what the route can still settle after the Source is written and before a Build runs: which* reconstructed elements have never been compared against the reference, which words of the Script@@ -207,6 +386,13 @@ type CoverageReport = {* comparison is a comparison. What it says is which elements have had their behaviour over their* window looked at and which have only had their layout looked at.*+ * Where each Frame sits is decided from the Source alone and is reported rather than required. A+ * Frame with an edge outside 0%–100% of its Canvas puts that much of whatever is drawn into it off+ * the picture, and a comparison cannot see it: the render and the stand-in are drawn at the same+ * Canvas, so both put the element in the same place off the edge and agree. Reaching past the edge is+ * also how an element slides in from off-screen and how a full-bleed picture is cropped by a `fit`,+ * which look identical here, so `passed` does not depend on it.+ ** Frame coverage is decided from the Source alone and is required: a word is either drawn over by* something bound to the whole picture or it is not. What covers is read from an element's own* opening tag — a full-bleed `frame=`, the `canvas=`, or a speech Track's full-frame `visual-frame=`@@ -231,8 +417,13 @@ export async function reconstructionCheck(const svml = await readFile(svmlPath, "utf8").catch(() => undefined);assert(svml !== undefined, `cannot read ${svmlPath}`);- // Every package this Source imports.- const imports = [...svml.matchAll(/<import\s+as="([^"]+)"\s+from="(@hypit\/[^"@]+)@\d+"/gu)]+ // Read from the Source alone, so every answer below carries it, including the ones that stop early.+ const overhang = framesPastTheCanvas(svml);++ // Every package this Source imports. The scope is whatever the Source wrote: a project that declares+ // a vocabulary gap and fills it publishes under its own scope, and those elements draw exactly as an+ // installed one does.+ const imports = [...svml.matchAll(/<import\s+as="([^"]+)"\s+from="(@[^"@/]+\/[^"@]+)@\d+"/gu)].map((match) => ({ alias: match[1] ?? "", specifier: match[2] ?? "" }));if (imports.length === 0) {return {@@ -242,13 +433,26 @@ export async function reconstructionCheck(elements: [],playback: [],uncovered: [],+ out_of_bounds: overhang,};}// Which of their Surfaces draw a Track. A Style Surface publishes a Style and draws nothing, so// requiring a comparison of it would be asking for a picture that does not exist.const specifiers = [...new Set(imports.map((item) => item.specifier))];- const loaded = await loadNodePackageSelection(specifiers, packageRoot).catch(() => []);+ // One unresolvable package used to take the rest with it: the whole selection loads or none of it+ // does, and a swallowed failure left `drawingTags` empty, so every element in the Source read as+ // drawing nothing and the check passed on an empty answer. Fall back to loading them one at a time so+ // the ones that resolve still count, and keep the names of the ones that did not.+ const unresolved: string[] = [];+ const loaded = await loadNodePackageSelection(specifiers, packageRoot).catch(async () => {+ const each = await Promise.all(specifiers.map(async (specifier) =>+ await loadNodePackageSelection([specifier], packageRoot).catch(() => {+ unresolved.push(specifier);+ return [];+ })));+ return each.flat();+ });const drawingTags = new Set<string>();for (const pack of loaded) {for (const facet of pack.contribution.hostFacets ?? []) {@@ -446,6 +650,7 @@ export async function reconstructionCheck(elements: [],playback: [],uncovered: [],+ out_of_bounds: overhang,};}@@ -554,17 +759,25 @@ export async function reconstructionCheck(summary.push(uncoveredWords === 0? `every word of the Script is drawn over by something that fills the frame (${coverage.words}).`: `${uncoveredWords} of ${coverage.words} words are drawn over by nothing that fills the frame.`);+ if (overhang.length > 0) {+ summary.push(`${overhang.length} Frame${overhang.length === 1 ? "" : "s"} `+ + `${overhang.length === 1 ? "reaches" : "reach"} past the Canvas; read each one against the reference.`);+ }const compared = elements.filter((element) => element.comparisons > 0).length;if (compared > 0) {summary.push(untimed.length === 0? `every compared element was looked at over a reference-timed stand-in (${compared}).`: `${untimed.length} of ${compared} compared elements have comparisons over a stand-in that was not reference-timed.`);}+ if (unresolved.length > 0) {+ summary.push(`${unresolved.length} imported package${unresolved.length === 1 ? "" : "s"} could not be resolved, `+ + "so nothing is known about what they draw.");+ }return {run: runPath,reference_id: reference,- passed: never.length === 0 && running.length === 0 && coverage.gaps.length === 0,+ passed: never.length === 0 && running.length === 0 && coverage.gaps.length === 0 && unresolved.length === 0,summary,elements,...(never.length === 0 ? {} : {@@ -579,11 +792,25 @@ export async function reconstructionCheck(+ `--video <rendered clip>.mp4 --element ${element.id}`),},}),+ ...(unresolved.length === 0 ? {} : {+ unresolved_packages: {+ names: unresolved,+ package_root: packageRoot,+ note: `${unresolved.length} package${unresolved.length === 1 ? "" : "s"} the Source imports could not be `+ + `resolved from ${packageRoot}. Whatever they draw is invisible here, so no element of theirs is asked `+ + "for a comparison and this check cannot say the reconstruction is covered.\n\n"+ + "A project that publishes its own packages resolves them from the project root, where "+ + "`packages/<name>/package.json` carries the scoped name. Run the command from that directory, or "+ + "pass --package-root <project directory>.",+ },+ }),...(unknown.length === 0 ? {} : {unknown_elements: {names: unknown,- note: `${unknown.length} logged element name${unknown.length === 1 ? " does" : "s do"} not exist in the Source. `- + "A misspelled --element credits nothing; the element it was meant for is still at zero.",+ note: `${unknown.length} logged element name${unknown.length === 1 ? " does" : "s do"} not name a drawing `+ + "element in the Source — either the id is not there at all, or it belongs to something that puts no "+ + "picture on the Canvas. Either way the comparison credits nothing, and the element it was meant for "+ + "is still at zero.",},}),...(unlabelled === 0 ? {} : {@@ -620,6 +847,24 @@ export async function reconstructionCheck(+ "something beneath the stretch instead changes which wrong picture appears and leaves the stretch "+ "unclaimed.",}),+ out_of_bounds: overhang,+ ...(overhang.length === 0 ? {} : {+ out_of_bounds_note: "Each Frame here places part of what is drawn into it off the picture, by the amount "+ + "beside each edge. `outside_fraction` is how much of the Frame's own area lands off the Canvas. This "+ + "is read from the Source's own `space:Canvas` and `space:Frame` arithmetic — no rendered frame is "+ + "measured and no observer is asked.\n\n"+ + "Answer each one: is this overhang intended? Two shapes of it are, and both are visible in the "+ + "Source. An element that travels in from off-screen is outside for part of its window, so its "+ + "Recipe carries the movement — an `enter`, a fly-from origin, an animated offset — and the Frame is "+ + "where it starts rather than where it stays. A full-bleed picture is declared past the edge on "+ + "purpose so a `fit` crops it instead of letterboxing it, so its Recipe carries that `fit`. A Frame "+ + "with neither, holding something the reference shows whole, is placed wrong: the part past the edge "+ + "is the part nobody will see.\n\n"+ + "This decides nothing on its own, because the Source cannot tell the two apart. It is here because "+ + "comparison cannot see it at all: an element authored mostly off the picture is drawn at the Canvas "+ + "the Source declares, and the stand-in it is compared against is drawn at that same Canvas, so both "+ + "put it in the same place off the edge and agree with each other.",+ }),playback: running,...(running.length === 0 ? {} : {playback_next: "A generated take is ordered in whole seconds and its window is measured from speech the "diff --git a/packages/reference-video-tools/src/cli.ts b/packages/reference-video-tools/src/cli.tsindex f9943ff3..a638af8a 100644--- a/packages/reference-video-tools/src/cli.ts+++ b/packages/reference-video-tools/src/cli.ts@@ -2,6 +2,7 @@import { readFile } from "node:fs/promises";import { createReferenceVideoTools } from "./tools.js";+import type { CompareReconstructionInput } from "./tools.js";type Flags = ReadonlyMap<string, string | readonly string[] | boolean>;@@ -16,6 +17,7 @@ function usage(): string {" hypit-reference-video-tools inspect_svml_vocabulary --package <name> [--package <name> ...] [--tag <tag> ...] [--without-previews]"," hypit-reference-video-tools compare_reconstruction --reference-id <id> --run <build.svrun> --segment <id>|--selection <id> --video <path>|--image <path> [--question <scope>] [--element <id>]"," hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> --video <path>|--image <path> [--question <scope>] [--element <id>]",+ " hypit-reference-video-tools compare_reconstruction --reference-id <id> --batch <comparisons.json>"," hypit-reference-video-tools make-placeholder --out <path> --width <w> --height <h> [--color light|mid|dark|white|black|#RRGGBB] [--video] [--seconds <s>]"," hypit-reference-video-tools render_element <build.svrun> --element <id> --out <path.png|path.mp4> [--segment <id>] [--selection <id>] [--reference-id <id>]"," hypit-reference-video-tools render_previews <package-dir> [...]",@@ -36,6 +38,12 @@ function usage(): string {"not recorded — which says whether anything that changes with elapsed time inside its window has","been looked at, or only its layout.","",+ "It also reports every Frame with an edge outside 0%-100% of its Canvas, which places part of what",+ "is drawn into it off the picture. That is reported rather than required: an overhang is how an",+ "element slides in from off-screen and how a full-bleed picture is cropped by a fit, and a mistake",+ "looks the same. A comparison cannot answer it either way, because the render and the stand-in are",+ "drawn at the same Canvas and put the element in the same place off the edge.",+ "","render_element draws one element of a Source the way that Source configures it, without a Build","and without a Provider: the Canvas, frame rate, Recipe values and bindings are read from the","Source, and the layers a Build has not made are mocked with make-placeholder at the Canvas's size.",@@ -73,6 +81,14 @@ function usage(): string {"inside its word leaves the pair part-way through a shot, and the prompt says how many seconds of it","to read as an incomplete shot. --shot-id names a cut in the picture and compares that whole shot.","",+ "--batch runs a whole round at once. The file holds a JSON array — or an object with a `comparisons`",+ "array — of the same objects a single call takes, minus reference_id, which comes from the flag:",+ "`[{\"run\": \"main.svrun\", \"segment\": \"pro\", \"video_path\": \"renders/board-pro.mp4\", \"element\": \"board\"}, …]`.",+ "Every render is finished before any comparison starts, so a round has no order to it; they are",+ "paced by HYPIT_REFERENCE_CONCURRENCY and each one's derived cuts are kept apart, so nothing in the",+ "round reads a file another is still writing. One comparison that fails takes only itself down and",+ "arrives under `failures` with the input that produced it; the rest are under `comparisons`.",+ "","--video compares the whole stretch instead of one frame of it, which is what removes the problem of","choosing a characteristic frame for an element that animates in, leaves, or is replaced without a","cut. The `gemini` observer receives the two clips; the `agent` observer receives two frame tiles,",@@ -103,6 +119,13 @@ function usage(): string {"returns each observation as a task carrying its prompt and one tiled picture per shot, which the","calling agent answers with record_observation.","",+ "--package-root <dir> is where the packages a Source imports are resolved from, and it is accepted by",+ "every command. It defaults to the working directory. A project that declares a vocabulary gap and",+ "fills it publishes those packages under its own scope and installs them against the project root, so",+ "run these commands from the project directory or name it here: resolved from anywhere else, the",+ "project's own packages are not found, and reconstruction_check reports them under",+ "`unresolved_packages` rather than asking for a comparison of what they draw.",+ "","Every command prints one JSON result to stdout. Use --input <json> instead of flags when a complete input object is easier to pass.",].join("\n");}@@ -178,10 +201,13 @@ async function main(): Promise<void> {if (flags.has("rebuild") || flags.has("refresh")) {throw new Error("--rebuild and --refresh were removed; use --redo on prepare_reference or --reobserve on observe_reference");}+ // Where the Source's imports are resolved from. A project publishing its own packages installs them+ // against the project root, so a root taken from anywhere else resolves none of them: --package-root+ // is how a command run from elsewhere reaches them. HYPIT_DISTRIBUTION_ROOT names an installed+ // Distribution and is the fallback for the installed packages a project does not carry.+ const resolveFrom = one(flags, "package-root") ?? process.env.HYPIT_DISTRIBUTION_ROOT;const tools = createReferenceVideoTools({- ...(process.env.HYPIT_DISTRIBUTION_ROOT === undefined- ? {}- : { packageRoot: process.env.HYPIT_DISTRIBUTION_ROOT }),+ ...(resolveFrom === undefined ? {} : { packageRoot: resolveFrom }),});const supplied = inputObject(flags);let result: unknown;@@ -215,6 +241,20 @@ async function main(): Promise<void> {};result = await tools.record_observation(input as { reference_id: string; key: string; text: string });} else if (command === "compare_reconstruction") {+ // A round is a list of comparisons, and a list is too long for flags. --batch names a JSON file+ // holding it, which is also how it survives being written by one step and read by another.+ const batchFile = one(flags, "batch");+ if (batchFile !== undefined) {+ const parsed: unknown = JSON.parse(await readFile(batchFile, "utf8"));+ const list = Array.isArray(parsed) ? parsed : (parsed as { comparisons?: unknown }).comparisons;+ if (!Array.isArray(list)) throw new Error(`${batchFile} must hold a JSON array of comparisons, or an object with a "comparisons" array`);+ result = await tools.compare_reconstruction({+ reference_id: required(flags, "reference-id"),+ comparisons: list as CompareReconstructionInput["comparisons"] & object,+ });+ report(`${JSON.stringify(result, null, 2)}\n`);+ return;+ }const video = one(flags, "video");const segment = one(flags, "segment");const selection = one(flags, "selection");diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.tsindex 1e35a2dd..7711c09e 100644--- a/packages/reference-video-tools/src/tools.ts+++ b/packages/reference-video-tools/src/tools.ts@@ -3,7 +3,7 @@ import { access, appendFile, mkdir, readdir, readFile, stat, writeFile } from "nimport { spawn } from "node:child_process";import { createHash } from "node:crypto";import { deflateSync } from "node:zlib";-import { dirname, join, resolve } from "node:path";+import { basename, dirname, join, resolve } from "node:path";import { createRequire } from "node:module";import { loadNodePackageSelection } from "@hypit/package-loader-node";import { markupSurfaceHostFacetAbi } from "@hypit/markup";@@ -84,6 +84,12 @@ export type CompareReconstructionInput = {* elements have been compared and which have never been looked at.*/readonly element?: string;+ /**+ * A whole round of comparisons, run together. Each entry names its own stretch, render and element+ * exactly as a single call does, and inherits `reference_id` from the outer input. The route renders+ * every element before it compares any of them, so a round is a list by the time it starts.+ */+ readonly comparisons?: readonly Omit<CompareReconstructionInput, "reference_id" | "comparisons">[];};export type MakePlaceholderInput = {@@ -479,6 +485,7 @@ async function cutWordRange(range: SpokenRange,renderedPath: string,clip: boolean,+ slot: string,): Promise<RangeCut> {const cuts = referenceCuts(state);const head = cuts.find((at) => at >= range.first.startSeconds && at < range.first.endSeconds);@@ -488,10 +495,14 @@ async function cutWordRange(const seconds = end - start;assert(seconds > 0, `${focus.selection ?? focus.segment} spans no time in the reference`);- const dir = join(root, "ranges");- await ensureDir(dir);const label = focus.selection === undefined ? `segment-${focus.segment}` : `selection-${focus.selection}`;- const reference = join(dir, `${label}-reference.${clip ? "mp4" : "jpg"}`);+ // One directory per call. Several elements are drawn over one Segment, so several comparisons name+ // the same word range, and the route runs them at once once every render is finished. Named from the+ // range alone they would write the same six files, and each would be reading the cut another was+ // still writing.+ const dir = join(root, "comparisons", label, slot);+ await ensureDir(dir);+ const reference = join(dir, `reference.${clip ? "mp4" : "jpg"}`);// The reference's own analysis video, which is the whole reference at the size every other// observation reads it at. A shot clip covers a cut in the picture and would cover the wrong words.if (clip) await cutClip(state.analysis_video_ref, start, seconds, reference);@@ -507,15 +518,22 @@ async function cutWordRange(// render's own rate and taken off as whole frames.const headFrames = Math.round((start - range.startSeconds) * drawn.frameRate);const tailFrames = Math.round((range.endSeconds - end) * drawn.frameRate);- rendered = join(dir, `${label}-rendered.mp4`);+ rendered = join(dir, "rendered.mp4");await cutClip(renderedPath, headFrames / drawn.frameRate, drawn.duration - (headFrames + tailFrames) / drawn.frameRate, rendered);referenceSeconds = (await probe(reference)).duration;renderedSeconds = (await probe(rendered)).duration;// Two stretches of different lengths cannot be read side by side: whatever is at a given offset// in one is at a different word in the other, which is the defect this cut exists to remove.- // Each side lands on its own frame grid, so they agree to within a frame rather than exactly.- assert(Math.abs(referenceSeconds - renderedSeconds) <= 2 / drawn.frameRate,- `the cut reference runs ${round(referenceSeconds)}s and the trimmed render ${round(renderedSeconds)}s; they must cover the same stretch`);+ //+ // What they can differ by is set by the containers rather than by the cut. Each side lands on its+ // own frame grid, which is two frames between them, and a file carrying AAC reports a duration+ // rounded up to a whole audio frame — 1024 samples, so 21ms at 48kHz — which neither side's frame+ // grid divides. Three frames covers both; anything larger is the cut disagreeing with the render+ // about which words the stretch holds.+ const slack = 3 / drawn.frameRate;+ assert(Math.abs(referenceSeconds - renderedSeconds) <= slack,+ `the cut reference runs ${round(referenceSeconds)}s and the trimmed render ${round(renderedSeconds)}s, `+ + `which is more than ${round(slack)}s apart; they must cover the same stretch`);}// An end that could not be moved onto a cut opens or closes part-way through a shot. How much of@@ -555,10 +573,10 @@ async function cutWordRange(},reference,rendered,- referenceTile: join(dir, `${label}-reference-tile.jpg`),- renderedTile: join(dir, `${label}-rendered-tile.jpg`),- referenceStill: join(dir, `${label}-reference-still.jpg`),- renderedStill: join(dir, `${label}-rendered-still.jpg`),+ referenceTile: join(dir, "reference-tile.jpg"),+ renderedTile: join(dir, "rendered-tile.jpg"),+ referenceStill: join(dir, "reference-still.jpg"),+ renderedStill: join(dir, "rendered-still.jpg"),seconds,referenceSeconds: round(referenceSeconds),renderedSeconds: round(renderedSeconds),@@ -643,7 +661,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV};};- return {+ const tools: ReferenceVideoTools = {// Which packages exist is the first question of every reconstruction, and until now the only// answer was to read a guide and a directory listing by hand. The Build CLI deliberately never// scans a directory; this is a development tool, so it may.@@ -960,6 +978,32 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV},async compare_reconstruction(input): Promise<Record<string, unknown>> {+ // Every render is finished before any comparison starts, so the comparisons are independent of+ // each other and there is nothing to gain by running them one at a time. Handing the whole list+ // over means the pacing, the per-comparison result and the failure of any one of them are the+ // tool's business rather than a shell script's.+ if (input.comparisons !== undefined) {+ const batch = input.comparisons;+ assert(batch.length > 0, "comparisons is empty");+ const results = await pacedMap(batch, concurrency, gapMs, async (one) => {+ const merged = { reference_id: input.reference_id, ...one };+ // One comparison that throws — an unreadable render, a Selection the Script does not mark —+ // takes only itself down. The rest of the list is what the caller came for.+ try { return { ok: true as const, value: await tools.compare_reconstruction(merged) }; }+ catch (error) { return { ok: false as const, error: error instanceof Error ? error.message : String(error), input: merged }; }+ });+ const done = results.filter((item) => item.ok).map((item) => item.value!);+ const failed = results.filter((item) => !item.ok);+ return {+ reference_id: input.reference_id,+ compared: done.length,+ failed: failed.length,+ comparisons: done,+ ...(failed.length === 0 ? {} : {+ failures: failed.map((item) => ({ error: item.error, input: item.input })),+ }),+ };+ }const state = await loadState(input.reference_id);const root = stateRoot(state.reference_id);const named = [input.shot_id, input.segment, input.selection].filter((value) => value !== undefined);@@ -971,6 +1015,11 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst file = await stat(renderedPath).catch(() => undefined);assert(file?.isFile(), `${clip ? "video_path" : "image_path"} is not a file: ${renderedPath}`);const scope = input.question?.trim() ?? "";+ // Everything this call derives is written under this name, so two comparisons of one stretch —+ // two elements drawn over the same Segment, or one element read twice — never write each other's+ // files. The render is what distinguishes them, so the render names the slot.+ const slot = `${basename(renderedPath).replace(/\.[^.]*$/u, "")}-`+ + createHash("sha256").update(renderedPath).digest("hex").slice(0, 8);const standIn = await standInBeside(renderedPath);const observer: Observer = state.observer ?? "gemini";const { ask, pending } = await askerFor(observer, state);@@ -997,7 +1046,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV...(input.selection === undefined ? {} : { selection: input.selection }),};const range = await spokenRange(svmlPath, focus, await referenceWords(state.reference_id));- cut = await cutWordRange(state, root, focus, range, renderedPath, clip);+ cut = await cutWordRange(state, root, focus, range, renderedPath, clip, slot);referenceMedia = cut.reference;renderedMedia = cut.rendered;stretchSeconds = cut.seconds;@@ -1017,8 +1066,12 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVreferenceMedia = shot === undefined? await cutFrame(cut!.reference, stretchSeconds / 2, cut!.referenceStill): shot.representative_frame_ref;- renderedMedia = await cutFrame(renderedMedia, stretchSeconds / 2,- shot === undefined ? cut!.renderedStill : join(root, "shots", `${shot.shot_id}-reconstruction-still.jpg`));+ let stillTarget = cut?.renderedStill ?? "";+ if (shot !== undefined) {+ stillTarget = join(root, "comparisons", `shot-${shot.shot_id}`, slot, "reconstruction-still.jpg");+ await ensureDir(dirname(stillTarget));+ }+ renderedMedia = await cutFrame(renderedMedia, stretchSeconds / 2, stillTarget);}}const asClip = clip && !bothStill;@@ -1033,7 +1086,9 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVassert(shot.frames_tile_ref !== null,`shot ${shot.shot_id} has no frame tile; re-prepare the reference with --redo all --observer agent`);referenceMedia = shot.frames_tile_ref;- renderedMedia = await shotTile(renderedPath, stretchSeconds, join(root, "shots", `${shot.shot_id}-reconstruction.jpg`));+ const tileTarget = join(root, "comparisons", `shot-${shot.shot_id}`, slot, "reconstruction.jpg");+ await ensureDir(dirname(tileTarget));+ renderedMedia = await shotTile(renderedPath, stretchSeconds, tileTarget);} else {// A word range is cut when it is asked for, so neither side has a prepared tile and both// are built here, from the two cuts.@@ -1146,6 +1201,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVreturn await reconstructionCheck(input, { packageRoot });},};+ return tools;}const WHOLE_REFERENCE_KEYS = ["people_and_product", "voices", "persistent_systems", "places"] as const;