Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
feat(hypit): render the comparison picture from the Source, not by hand
rpone1ef08de8f84e
diff --git a/.agents/skills/hypit/references/local-author-package.md b/.agents/skills/hypit/references/local-author-package.mdindex 4bebf6be..ac8b9896 100644--- a/.agents/skills/hypit/references/local-author-package.md+++ b/.agents/skills/hypit/references/local-author-package.md@@ -174,35 +174,20 @@ nothing.- A component that cannot render on its own cannot produce the catalogue preview its Surface owes.Treat a missing preview as evidence of this mistake rather than a step to skip. The preview provesthe component draws; what it draws is compared against the reference from a separate render, below.-- **The render harness takes its inputs as arguments.** The catalogue preview is one picture: sample- copy, a sample Recipe, every feature on, at whatever moment shows the component best.- `reconstruction-loop.md` needs a different picture — the component drawn from the Recipe values a- Source actually passes and the Script text that Source actually feeds it — for every reference shot- the component appears in. The values are the difference, and they are the whole difference: a Source- can pass copy and sizes that collide while the sample values stay perfect, so a picture drawn from- the samples cannot report it.-- Give the harness these arguments, and let the catalogue preview be one invocation of it rather than- its only purpose:-- - the path to the Recipe sheet the Source uses, and which Recipe in it;- - the text the Source feeds the component, as the Source feeds it;- - a path per media slot, and a path for the layer beneath;- - the Canvas the Source declares, so the render composes at the geometry the mocks were built to;- - `--out`, and whether to write a still or a clip covering a stretch.-- A harness with its inputs written into it produces the catalogue picture and nothing else, and the- next agent reuses that picture for a comparison it does not fit.-- **The harness does not invent media, for slots or for the layer beneath.** A slot declared as a- generation is mockable but not fillable on this route, and neither is the base take a component- draws over. Both mocks come from the route's fixed `make-placeholder` tool, not from code the- harness ships: the harness reads the paths from its arguments and passes them in. No- placeholder-drawing code inside the harness, no generated picture committed just to have something- in a slot.-- This is a rendering argument and never a Source edit. The Source keeps declaring the generation;- what the harness receives is a path that exists for one render. The gates that decide coverage read- the Source's Recipes and bindings, so a mock cannot be counted as the thing it stands in for.+- **The package owes a catalogue preview and nothing else.** That picture is drawn from sample copy+ and a sample Recipe the package chooses, to show a reader the Surface's range, and it belongs to the+ package the way its README does.++ The picture `reconstruction-loop.md` compares is a different one and the package does not produce+ it. `scripts/render-element.mjs` reads the Source — the Canvas, the Recipe values, the Script text,+ the bindings — and drives the package's own Producer to draw the component the way this video places+ it. All the package has to be is a working Producer, which the Manifest already requires.++ Do not write a per-package harness for the comparison. One that takes its values as arguments still+ gets them by hand, one line at a time, which draws one line and lets a collision between several go+ unseen; one with its values written in draws the catalogue picture and cannot report the Source at+ all. Both also decide their own canvas, and a canvas that is not the Source's puts every proportion+ in the comparison slightly wrong.## Freeze the Types before writing in paralleldiff --git a/.agents/skills/hypit/references/preview.md b/.agents/skills/hypit/references/preview.mdindex 6081c4e7..827c76f4 100644--- a/.agents/skills/hypit/references/preview.md+++ b/.agents/skills/hypit/references/preview.md@@ -52,38 +52,37 @@ Provider. The repository's own visual test `packages/hyperframes/test/browser-vithe path end to end: build the Track, compile the HyperFrames document, and render it through thelocal HyperFrames Runtime.-There is no one command: a package's `test/render-preview.ts` harness is what renders it, and-`local-author-package.md` requires that harness to take its inputs and its output path as arguments.-That requirement exists because two different pictures come out of it, and they must not collapse-into one:+Two different pictures come out of this, and they must not collapse into one:- the **catalogue preview** — one image for Studio and the package README, drawn from sample copy and- a sample Recipe the harness supplies itself, chosen to show the Surface's range;+ a sample Recipe the package supplies itself, chosen to show the Surface's range. It is the package's+ own, like its README, and `local-author-package.md` says the package owes it;- the **comparison render** — for `reconstruction-loop.md`, drawn from the Recipe values a Sourceactually passes, the Script text that Source actually feeds the element, and mocks for the layers a- Build has not made. One per reference shot the element appears in.+ Build has not made.-The inputs are what separates them, and it is the whole separation. A Source can pass values that+The values are what separates them, and it is the whole separation. A Source can pass values thatcollide — lines that overlap, type too large for the plate it sits on — while the catalogue preview,-drawn from different values, stays perfect. A harness whose inputs are written into it can only-produce the first picture, and a route that compares that picture is answering a question about the-catalogue.--Two things are mocked rather than left out. **The layers beneath**: a component that draws over a-base take renders over black without one, and black is not neutral — text legible on it can be-illegible on the picture that replaces it. **Media slots the Source declares as generations**: empty-for the whole stretch between being written and being built. Both use the route's fixed-`make-placeholder` tool — `--video --seconds <s>` for a stretch, a PNG for a slot, `--color` so the-mock stays visible against what the element draws — sized from the Source's `space:Canvas` rather-than from the reference video, and the comparison is scoped with `--question` so the observer skips-those regions. `reconstruction-loop.md` gives the sizing rule and says why a reported mock is not a-repair target.--The render can be a clip as well as a still, and a clip is the default. `compare_reconstruction---video` compares the whole shot, which removes the problem of choosing a frame to represent a shot-that holds several states. A still is for the one case `reconstruction-loop.md` names: the shot's own-visual observation says in words that the element is completely still. That judgement comes from the-reference's observation, never from watching your own render.+drawn from different values, stays perfect. A route that compares the catalogue picture is answering+a question about the catalogue.++The comparison render has one command, shared by every package:++```bash+node --import tsx .agents/skills/hypit/scripts/render-element.mjs projects/<name>/build.svrun \+ --element <id> --segment <id>|--selection <id> --out <path>.mp4+```++It reads the Canvas, the Recipes, the Script and the bindings out of the Source; stands in for the+speech the Build has yet to synthesize using the Source's own `estimate:Speech`, so the window+lengths are the ones the Source already ordered its generations with; mocks every declared generation+with `make-placeholder` at the Canvas's size; and drives each package's Producer through the same+Studio projection `preview-check` traces, with no Provider installed. The mocks and the stand-in takes+live in a derived Run under the project's `.hypit/`, never in the Source.++Name the window in words. `--segment` and `--selection` take a Script name rather than a timestamp,+because a shot of the reference is found by the words spoken over it. Write `.mp4` to `--out` for a+clip and `.png` for the middle still; `reconstruction-loop.md` says which to use when.This is what to reach for whenever one element has to be looked at rather than the whole program.Rendering the delivery to inspect a single piece is the waste it exists to prevent.diff --git a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.mdindex f2908348..5b4403d6 100644--- a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md+++ b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md@@ -51,9 +51,21 @@ every one of those parameters with values that collide while the preview stays ppreview answers a question nobody asked. `../preview.md` renders both, from the same harness, withdifferent arguments; this loop uses the Source-configured one.-The render is local and takes no paid Provider. `packages/hyperframes/test/browser-visual.test.ts`-shows the path end to end: build the Track, compile the HyperFrames document, render it through the-local HyperFrames Runtime.+One command produces it, and it is not written per package:++```+node --import tsx .agents/skills/hypit/scripts/render-element.mjs projects/<name>/build.svrun \+ --element <id> --segment <id>|--selection <id> --out <path>.mp4+```++It reads the Canvas, the Recipe values and the Script text out of the Source, stands in for the+speech with the Source's own `estimate:Speech`, mocks every layer a Build has not made, and drives+the package's Producer. Nothing is transcribed by hand, so nothing is transcribed one line at a time+— which is what made a caption system look correct while its lines collided.++The window is named in words. `--segment` and `--selection` take a Script name, not a timestamp: a+shot of the reference is found by the words spoken over it, and those words are where the Script+says they are. No reference clock is read across.### Mock the layers a Build has not made@@ -62,24 +74,17 @@ it may be media slots the Source declares as generations. Render them as mocks rthem out: a missing base is not a neutral background, it is black, and text that is legible on blackcan be illegible on the picture that will replace it.-Use the route's fixed placeholder tool for both — `hypit-reference-video-tools make-placeholder --out-base.mp4 --width <w> --height <h> --video --seconds <s>` for the base stretch, `--out slot.png` for an-image slot — with `--color` chosen so the mock is visible against what the element draws. Never-`hypit image`, which pays for a generation the video will not reuse, and never a script written by-hand.--**The sizes come from the Source's own `space:Canvas`, not from the reference video.** The mock is-composed inside the render, and the render's geometry is the Canvas. A base mock is the Canvas-exactly — `<space:Canvas width="1080" height="1920"/>` means `--width 1080 --height 1920` — and a slot-mock is that slot's `space:Frame` fractions multiplied by the Canvas, so a Frame from 16% to 84%-across and 30% to 78% down is `--width 734 --height 922`. A mock built to the wrong box is resized by-the `fit` before the observer ever sees it, which moves edges the comparison is there to read.--The reference's own pixel dimensions are a different number and are usually smaller — the prepared-reference stores them in `state.json` under `video`. They do not size the mock, but their **aspect**-has to match the Canvas's. When it does not, every comparison in this loop is putting two-differently-shaped pictures side by side, and the observer will report proportion differences that-belong to the Canvas rather than to the element. Fix the Canvas before comparing anything.+`render-element.mjs` does this itself. It finds every generation the Source declares — the takes, the+stills, the slot contents — calls `make-placeholder` for each at the Canvas's own size, and declares+them in a derived Run under the project's `.hypit/`. Never `hypit image`, which pays for a generation+the video will not reuse, and never a placeholder drawn by a script written for the occasion.++The sizes come from the Source's `space:Canvas`, not from the reference video, because the mock is+composed inside the render and the render's geometry is the Canvas. The reference's own pixel+dimensions are a different and usually smaller number, stored under `video` in its `state.json`. They+do not size anything here, but their **aspect** has to match the Canvas's: when it does not, every+comparison puts two differently-shaped pictures side by side and the observer reports proportion+differences that belong to the Canvas rather than to the element. Fix the Canvas before comparing.Then tell the observer, through `--question`, that those regions are placeholders standing in fordeclared-but-unbuilt generations, and to compare only what the element itself draws. Said plainly itdiff --git a/.agents/skills/hypit/references/reconstruction/workflow.md b/.agents/skills/hypit/references/reconstruction/workflow.mdindex 1fe8d95a..008fc360 100644--- a/.agents/skills/hypit/references/reconstruction/workflow.md+++ b/.agents/skills/hypit/references/reconstruction/workflow.md@@ -41,9 +41,10 @@ list_svml_packages→ run preview-check (final-sources.md) and repair until the graph traces —this has no attempt ceiling; a target Studio cannot trace is not done.Waiting on unrun Providers is a pass, not a failure-→ read reconstruction-loop.md, then for each authored element: render it as the- sources configure it, mocking the layers a Build has not made, and compare it- against every shot the reference shows it in before repairing anything+→ read reconstruction-loop.md, then for each authored element: render it with+ render-element.mjs, which reads the sources and mocks the layers a Build has+ not made, and compare it against every shot the reference shows it in before+ repairing anything→ repair against the differences that round returned, within the ceilings→ run reconstruction-check (index.md) and keep going until it passes; it namesevery locally-drawn element that has never been compared, and every timeddiff --git a/.agents/skills/hypit/scripts/render-element.mjs b/.agents/skills/hypit/scripts/render-element.mjsnew file mode 100644index 00000000..5c034ad8--- /dev/null+++ b/.agents/skills/hypit/scripts/render-element.mjs@@ -0,0 +1,359 @@+#!/usr/bin/env node+/**+ * Render one element of a Source the way that Source configures it, without a Build and without a+ * Provider.+ *+ * `reconstruction-loop.md` compares a reconstructed element against the reference. What it must+ * compare is the element as this video places it — the Recipe values the Source passes, the Script+ * text it feeds, the windows it binds — because a Source can fill those with values that collide+ * while a package's catalogue preview, drawn from sample values, stays perfect. Producing that+ * picture by hand means transcribing values into a per-package harness, one line at a time, which is+ * how a caption system gets compared as a single line that has nothing to collide with.+ *+ * So this reads the values instead. Everything it needs is already written down:+ *+ * Canvas and Frames, Recipe values, bindings the Source+ * which elements share a window, in what order the Script's words, through their Selections+ * how long each Segment runs the Source's own `estimate:Speech`+ * the layers a Build has not made `make-placeholder`, sized from the Canvas+ *+ * The one thing missing before a Build is real speech, and `stand-in-takes.mjs` supplies a Segment+ * skeleton from the estimator the Source already trusts — the same numbers it ordered its takes+ * with, not a second clock. Those takes enter through `<value>` and `<satisfy>`, the mechanism+ * `examples/all-components-preview` uses to open in Studio without spending anything, so nothing+ * here is a private back door into the graph.+ *+ * What this settles: which elements are on screen together, where each sits, at what size, weight and+ * colour. What it does not: real timing, which waits for `playbooks/craft/production-gates.md` Gate 3.+ *+ * Usage:+ * node --import tsx .agents/skills/hypit/scripts/render-element.mjs <build.svrun> \+ * --element <id> --out <path.png|path.mp4> [--segment <id>] [--base <mock.mp4>]+ *+ * Exit: 0 having written --out. 1 when the element cannot be projected, naming what is unresolved.+ * 2 on a usage or discovery error.+ */+import { mkdir, readFile, readdir, rm, writeFile } from "node:fs/promises";+import { spawnSync } from "node:child_process";+import { createHash } from "node:crypto";+import { createRequire } from "node:module";+import { dirname, join, resolve } from "node:path";+import { fileURLToPath } from "node:url";++const studio = (name) => new URL(`../../../../packages/studio/src/${name}`, import.meta.url).href;+const { openStudioArchive } = await import(studio("archive.js"));+const { loadStudioDomain } = await import(studio("domain.js"));+const { loadStudioRun } = await import(studio("run.js"));+const { preview } = await import(studio("programme.js"));+const { inspectStudioRun } = await import(studio("studio-preflight.js"));+const { compileHyperframesDocument, materializeHyperframesHtml } = await import(new URL("../../../../packages/hyperframes/src/document.ts", import.meta.url).href);+const { standInTakes } = await import(new URL("./stand-in-takes.mjs", import.meta.url).href);++function fail(message, code = 2) {+ console.error(`render-element: ${message}`);+ process.exit(code);+}++function blobRef(bytes, mediaType) {+ return { kind: "blob", digest: `sha256:${createHash("sha256").update(bytes).digest("hex")}`, size: bytes.byteLength, mediaType };+}++/** 48 kHz mono PCM silence, written by hand so the mock needs no encoder and lands on exact bytes. */+function silentWav(sampleFrames) {+ const data = Math.max(2, sampleFrames * 2);+ const buffer = Buffer.alloc(44 + data);+ buffer.write("RIFF", 0);+ buffer.writeUInt32LE(36 + data, 4);+ buffer.write("WAVEfmt ", 8);+ buffer.writeUInt32LE(16, 16);+ buffer.writeUInt16LE(1, 20);+ buffer.writeUInt16LE(1, 22);+ buffer.writeUInt32LE(48_000, 24);+ buffer.writeUInt32LE(48_000 * 2, 28);+ buffer.writeUInt16LE(2, 32);+ buffer.writeUInt16LE(16, 34);+ buffer.write("data", 36);+ buffer.writeUInt32LE(data, 40);+ return buffer;+}++const argv = process.argv.slice(2);+const runArgument = argv[0];+if (runArgument === undefined || runArgument.startsWith("--")) {+ fail("usage: render-element.mjs <build.svrun> --element <id> --out <path> [--segment <id>] [--base <mock.mp4>]");+}+const flag = (name) => {+ const index = argv.indexOf(`--${name}`);+ return index === -1 ? undefined : argv[index + 1];+};+const element = flag("element") ?? fail("--element is required");+const out = flag("out") ?? fail("--out is required");++const invokedFrom = process.env.INIT_CWD ?? process.cwd();+const runPath = resolve(invokedFrom, runArgument);+const packageRoot = resolve(invokedFrom);+const projectRoot = dirname(runPath);+const outPath = resolve(invokedFrom, out);++const runSource = await readFile(runPath, "utf8").catch(() => fail(`cannot read ${runPath}`));+const authorMatch = /<author\s+source="([^"]+)"/u.exec(runSource);+if (authorMatch === null) fail(`${runArgument} declares no <author source="…"/>`);+const svmlPath = resolve(projectRoot, authorMatch[1]);+const svml = await readFile(svmlPath, "utf8").catch(() => fail(`cannot read ${svmlPath}`));++// The Program's frame rate and the Canvas the render composes at, both read rather than assumed. A+// harness that hard-codes either produces a picture at a geometry the reference never had.+const clock = /<[a-z-]*:?Clock\b[^>]*?\bframe-rate="(\d+)"/su.exec(svml)?.[1]+ ?? /<time:Clock\b[^>]*?\bfps="(\d+)"/su.exec(svml)?.[1];+const frameRate = Number(clock ?? 30);+const canvasMatch = /<space:Canvas\b[^>]*?\bwidth="(\d+)"[^>]*?\bheight="(\d+)"/su.exec(svml);+if (canvasMatch === null) fail("the Source declares no <space:Canvas width= height=/>");+const canvas = { width: Number(canvasMatch[1]), height: Number(canvasMatch[2]) };++// Which SemanticTake output belongs to which Segment, so each stand-in lands on the right one.+const takeOutputs = new Map();+for (const [, attributes] of svml.matchAll(/<whisperx:SemanticTake\b([^>]*?)\/?>/gsu)) {+ const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];+ const segment = /\bsegment=\{story\.segment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];+ if (id !== undefined && segment !== undefined) takeOutputs.set(segment, id);+}+if (takeOutputs.size === 0) fail("the Source declares no whisperx:SemanticTake to stand in for");++const { takes, selections, frameCount: programFrames } = await standInTakes(svmlPath, frameRate);+const framesBySegment = new Map(takes.map((item) => [item.segmentId, item.take.segment.endFrameExclusive]));+// A Selection's window in frames, summed over the stand-in tokens it covers. This is the same word+// span the Source binds to, carried into frames by the same estimate that sized the Segment.+const framesBySelection = new Map();+for (const { segmentId, take } of takes) {+ for (const token of take.tokens) framesBySelection.set(token.tokenId, token.endFrameExclusive - token.startFrame);+ framesBySelection.set(`segment:${segmentId}`, take.segment.endFrameExclusive);+}++/**+ * Every media a Build would have produced, and how long its window is. A speech take carries no+ * picture and a base take fills a Segment; a cutaway fills whatever its `during=` binds. Each becomes+ * a placeholder of the Canvas's own size, so the composite has the geometry the Source declares.+ */+function mockedMedia() {+ const windows = new Map();+ for (const [, attributes] of svml.matchAll(/<whisperx:SemanticTake\b([^>]*?)\/?>/gsu)) {+ const media = /\bmedia=\{([A-Za-z0-9_-]+)\.media\}/u.exec(attributes)?.[1];+ const segment = /\bsegment=\{story\.segment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];+ if (media !== undefined && segment !== undefined) windows.set(media, framesBySegment.get(segment) ?? frameRate);+ }+ for (const [, attributes] of svml.matchAll(/<[a-z][a-z0-9-]*:[A-Za-z][A-Za-z0-9]*\b([^>]*?)\/?>/gsu)) {+ const media = /\bmedia=\{([A-Za-z0-9_-]+)\.media\}/u.exec(attributes)?.[1];+ if (media === undefined || windows.has(media)) continue;+ const segment = /\bduring=\{story\.segment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];+ if (segment !== undefined) { windows.set(media, framesBySegment.get(segment) ?? frameRate); continue; }+ windows.set(media, frameRate * 2);+ }+ const carries = new Map();+ for (const [, attributes] of svml.matchAll(/<pipeline:Normalize\b([^>]*?)\/?>/gsu)) {+ const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];+ const video = /\bvideo="([^"]+)"/u.exec(attributes)?.[1];+ if (id === undefined || !windows.has(id)) continue;+ carries.set(id, { frames: windows.get(id), picture: video !== "none" });+ }+ return carries;+}++// A derived Run lives beside the project so the original's relative sources still resolve, and under+// `.hypit/` so it is never mistaken for something the author wrote. It is rewritten every run.+const compareRoot = join(projectRoot, ".hypit", "compare");+await rm(compareRoot, { recursive: true, force: true });+await mkdir(compareRoot, { recursive: true });+const declarations = [];+const satisfactions = [];+for (const { segmentId, take } of takes) {+ const output = takeOutputs.get(segmentId);+ if (output === undefined) continue;+ // The take's own audio has to be servable, not a digest stub: Studio resolves every Artifact a+ // projection references, and a preview that cannot serve one stops there.+ const sampleFrames = Math.round(take.segment.endFrameExclusive / frameRate * 48_000);+ const silence = silentWav(sampleFrames);+ await writeFile(join(compareRoot, `${segmentId}-take.wav`), silence);+ const spoken = { ...take, media: { ...take.media, audio: { artifact: blobRef(silence, "audio/wav"), sampleFrames } } };+ const fixture = join(compareRoot, `${segmentId}.json`);+ await writeFile(fixture, `${JSON.stringify({ kind: "inline", value: spoken }, null, 2)}\n`, "utf8");+ declarations.push(` <file id="stand-in-${segmentId}-audio" type="@hypit/artifact@1#BlobArtifact" from="./${segmentId}-take.wav" media-type="audio/wav"/>`);+ declarations.push(` <value id="stand-in-${segmentId}" type="@hypit/speech@1#SemanticTake" from="./${segmentId}.json"/>`);+ satisfactions.push(` <satisfy output="${output}.take" candidate="stand-in-${segmentId}"/>`);+}+// The layers a Build has not made, as placeholders of the Canvas's own size. `make-placeholder` is+// the route's one tool for this: deterministic, Provider-free, and the same mock the comparison is+// told to bypass. A mock never enters the project's own Source — it is declared in the derived Run+// and nowhere else.+const placeholderCli = new URL("../../../../packages/reference-video-tools/src/cli.ts", import.meta.url);+for (const [id, { frames, picture }] of mockedMedia()) {+ const seconds = Math.max(0.1, frames / frameRate);+ const file = join(compareRoot, `${id}.mp4`);+ if (picture) {+ const made = spawnSync(process.execPath, [+ "--import", "tsx", fileURLToPath(placeholderCli), "make-placeholder",+ "--out", file, "--width", String(canvas.width), "--height", String(canvas.height),+ "--video", "--seconds", seconds.toFixed(3), "--color", "mid",+ ], { encoding: "utf8", windowsHide: true, timeout: 120_000 });+ if (made.status !== 0) fail(`make-placeholder failed for ${id}: ${made.stderr?.trim()}`);+ }+ // Every SynchronizedMedia carries audio, and it is validated as WAV, so the silence is written as+ // one rather than pointed at the video. Both are declared: the picture is what gets drawn, the+ // silence is what makes the value legal.+ const wavPath = join(compareRoot, `${id}.wav`);+ const wav = silentWav(Math.round(frames / frameRate * 48_000));+ await writeFile(wavPath, wav);+ const audioBlob = blobRef(wav, "audio/wav");+ const media = {+ timeline: { frameRate: { numerator: frameRate, denominator: 1 }, frameCount: frames },+ audio: { artifact: audioBlob },+ };+ declarations.push(` <file id="mock-${id}-audio" type="@hypit/artifact@1#BlobArtifact" from="./${id}.wav" media-type="audio/wav"/>`);+ if (picture) {+ const bytes = await readFile(file);+ media.visual = { artifact: blobRef(bytes, "video/mp4"), width: canvas.width, height: canvas.height };+ declarations.push(` <file id="mock-${id}" type="@hypit/artifact@1#BlobArtifact" from="./${id}.mp4" media-type="video/mp4"/>`);+ }+ await writeFile(join(compareRoot, `${id}.media.json`), `${JSON.stringify({ kind: "inline", value: media }, null, 2)}\n`, "utf8");+ declarations.push(` <value id="media-${id}" type="@hypit/media@1#SynchronizedMedia" from="./${id}.media.json"/>`);+ satisfactions.push(` <satisfy output="${id}.media" candidate="media-${id}"/>`);+}++// Still images the Source declares as generations. Same treatment, same tool, at the Canvas's size:+// a picture slot that is empty in the render is black, and black is not what will be there.+const imageIds = [...new Set([...svml.matchAll(/\{([a-z0-9-]+)\.image\}/gu)].map(([, id]) => id))];+for (const id of imageIds) {+ const file = join(compareRoot, `${id}.png`);+ const made = spawnSync(process.execPath, [+ "--import", "tsx", fileURLToPath(placeholderCli), "make-placeholder",+ "--out", file, "--width", String(canvas.width), "--height", String(canvas.height), "--color", "mid",+ ], { encoding: "utf8", windowsHide: true, timeout: 120_000 });+ if (made.status !== 0) fail(`make-placeholder failed for ${id}: ${made.stderr?.trim()}`);+ declarations.push(` <file id="mock-${id}" type="@hypit/artifact@1#BlobArtifact" from="./${id}.png" media-type="image/png"/>`);+ satisfactions.push(` <satisfy output="${id}.image" candidate="mock-${id}"/>`);+}++const derivedRun = join(compareRoot, "compare.svrun");+await writeFile(derivedRun, [+ `<?svml using="@hypit/run-markup@1"?>`,+ ``,+ `<svrun version="1">`,+ ` <author source="../../${authorMatch[1].replace(/^\.\//u, "")}"/>`,+ // Targeting the element's own output rather than the Film prunes the closure to what this one+ // Track needs. Asking for the whole delivery would pull in every generation the Source declares+ // and report each as unresolved, which is true and useless: none of them is what is being looked at.+ ` <target output="final.video"/>`,+ ``,+ ...declarations,+ ``,+ ...satisfactions,+ `</svrun>`,+ ``,+].join("\n"), "utf8");++const domain = await loadStudioDomain({ run: derivedRun, workspaceRoot: projectRoot, packageRoot });+const archive = await openStudioArchive(undefined, packageRoot);+let built;+try {+ const run = await loadStudioRun({ run: derivedRun, domain, ...(archive === undefined ? {} : { archive }) });+ const inspection = inspectStudioRun(run.source, run);+ // Every Track, not only the one being looked at: the element is compared where it sits, over the+ // mocked base rather than on its own, because text that is legible on black may not be on a picture.+ built = await preview({+ source: run.source,+ run,+ domain,+ outputRefs: [inspection.filmComposition, ...inspection.projections.map((item) => item.ref)],+ compositionRef: inspection.filmComposition,+ projections: inspection.projections,+ ...(archive === undefined ? {} : { archive }),+ });+} catch (error) {+ const issues = error?.issues;+ if (Array.isArray(issues)) {+ console.error("render-element: the Source does not project yet.");+ for (const issue of issues.slice(0, 12)) console.error(` ${issue}`);+ process.exit(1);+ }+ fail(error instanceof Error ? error.message : String(error), 1);+}++const placed = built.tracks.find((track) => String(track.outputRef ?? "").endsWith(`::output::${element}.track`)+ || String(track.outputRef ?? "").includes(`::output::${element}.`));+if (placed === undefined) {+ console.error(`render-element: the Source places no element named ${element}. It places:`);+ for (const track of built.tracks) console.error(` ${String(track.outputRef ?? "").split("::output::")[1] ?? track.name}`);+ process.exit(1);+}++// The window to render, named in words. A shot of the reference is found by the words spoken over it+// and those words are a Segment or a Selection here, so no reference timestamp is ever read across.+const segmentArgument = flag("segment");+const selectionArgument = flag("selection");+let window = { startFrame: 0, endFrameExclusive: programFrames };+if (selectionArgument !== undefined) {+ window = selections.get(selectionArgument)+ ?? fail(`the Script marks no Selection ${selectionArgument}`);+} else if (segmentArgument !== undefined) {+ let cursor = 0;+ const take = takes.find((item) => { const hit = item.segmentId === segmentArgument; if (!hit) cursor += item.take.segment.endFrameExclusive; return hit; });+ if (take === undefined) fail(`the Script has no Segment ${segmentArgument}`);+ window = { startFrame: cursor, endFrameExclusive: cursor + take.take.segment.endFrameExclusive };+}++// Compile once, materialize with the artifact bytes written beside the document, and let the+// HyperFrames runtime draw it. This is the path packages/hyperframes/test/browser-visual.test.ts+// takes; nothing here is a private renderer.+const document = compileHyperframesDocument(built.composition, built.space);+const stage = join(compareRoot, "stage");+await mkdir(stage, { recursive: true });+const names = new Map();+for (const [digest, file] of built.served) {+ const name = `${digest.replace(/[^a-z0-9]/giu, "")}`;+ await writeFile(join(stage, name), file.bytes);+ names.set(digest, name);+}+await writeFile(join(stage, "index.html"), materializeHyperframesHtml(document, (artifact) => {+ const name = names.get(artifact.digest);+ if (name === undefined) fail(`the projection references Artifact ${artifact.digest}, which was not served`, 1);+ return `./${name}`;+}), "utf8");++const clip = /\.(mp4|mov|webm)$/iu.test(outPath);+const frames = join(compareRoot, "frames");+await rm(frames, { recursive: true, force: true });+await mkdir(frames, { recursive: true });+const hyperframesCli = createRequire(join(packageRoot, "packages/provider-hyperframes-local/package.json"))+ .resolve("hyperframes/bin/hyperframes.mjs");+const drawn = spawnSync(process.execPath, [+ hyperframesCli, "render", stage,+ "--format", "png-sequence", "--output", frames, "--fps", String(frameRate),+ "--workers", "1", "--no-browser-gpu", "--no-best-effort", "--quiet",+], { encoding: "utf8", windowsHide: true, timeout: 600_000 });+if (drawn.status !== 0) fail(`the HyperFrames runtime refused: ${(drawn.stderr ?? "").trim().slice(-2000)}`, 1);++const written = (await readdir(frames)).filter((name) => name.endsWith(".png")).sort();+if (written.length === 0) fail("the HyperFrames runtime wrote no frames", 1);+const first = Math.min(window.startFrame, written.length - 1);+const last = Math.min(window.endFrameExclusive, written.length);+if (clip) {+ const encoded = spawnSync("ffmpeg", [+ "-hide_banner", "-loglevel", "error", "-y", "-framerate", String(frameRate),+ "-start_number", String(first), "-i", join(frames, "frame_%06d.png"),+ "-frames:v", String(Math.max(1, last - first)),+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "20", outPath,+ ], { encoding: "utf8", windowsHide: true, timeout: 600_000 });+ if (encoded.status !== 0) fail(`ffmpeg refused: ${(encoded.stderr ?? "").trim().slice(-2000)}`, 1);+} else {+ await writeFile(outPath, await readFile(join(frames, written[Math.min(first + Math.floor((last - first) / 2), written.length - 1)])));+}++console.log(JSON.stringify({+ element,+ out: outPath,+ compared: clip ? "clip" : "still",+ canvas,+ frame_rate: frameRate,+ window,+ program_frames: programFrames,+ mocked: { media: mockedMedia().size, images: imageIds.length },+}, null, 2));diff --git a/.agents/skills/hypit/scripts/stand-in-takes.mjs b/.agents/skills/hypit/scripts/stand-in-takes.mjsnew file mode 100644index 00000000..268c7dea--- /dev/null+++ b/.agents/skills/hypit/scripts/stand-in-takes.mjs@@ -0,0 +1,174 @@+/**+ * Build a stand-in SemanticTake for every Segment a Source declares, from the Source alone.+ *+ * A Track timed against speech cannot be projected before the speech exists, and on this route it+ * does not exist: the Build has not run. What the Source does hold is the words themselves and the+ * estimator it already trusts to size its own generations — `estimate:Speech` with a policy Recipe.+ * Running that estimator here produces the same numbers the Source used to order its takes, so this+ * introduces no second clock; it reads the one already written down.+ *+ * What the result is good for: which elements are on screen together, where each sits, at what size+ * and colour. Those follow from word ranges and Recipe values and are exact. What it is not good for+ * is real timing — the delivered speech is not the estimate, and anything measured in seconds waits+ * for `production-gates.md` Gate 3.+ *+ * Tokens are laid across the Segment in proportion to the same syllable count the estimator uses, so+ * a long word occupies more of the window than a short one and the ordering is the Script's.+ */+import { readFile } from "node:fs/promises";+import { dirname, resolve } from "node:path";++const from = (path) => new URL(`../../../../packages/${path}`, import.meta.url).href;+const { parseScript } = await import(from("script/src/parser.ts"));+const { estimateSpeechDuration, countSpeechEstimateUnits, resolveSpeechEstimateLanguage } =+ await import(from("estimate/src/program.ts"));+const { speechEstimatePolicyFromRecipe } = await import(from("estimate/src/surface.ts"));++/** The Script body, which `parseScript` takes on its own. */+function scriptBody(svml) {+ const open = /<script\b[^>]*>/u.exec(svml);+ if (open === null) throw new Error("the Source declares no <script>");+ const start = open.index + open[0].length;+ const end = svml.indexOf("</script>", start);+ if (end < 0) throw new Error("the Source's <script> is not closed");+ return { text: svml.slice(start, end), offset: start };+}++/** Every Recipe body in every sheet the Source imports, keyed `alias.path`. */+async function recipeSheets(svml, svmlPath) {+ const sheets = new Map();+ for (const [, alias, source] of svml.matchAll(/<import\s+as="([^"]+)"\s+source="([^"]+\.svs)"/gu)) {+ const text = await readFile(resolve(dirname(svmlPath), source), "utf8").catch(() => undefined);+ if (text === undefined) continue;+ for (const [, name, body] of text.matchAll(/([A-Za-z0-9_.-]+)\s*\{([^}]*)\}/gu)) {+ const properties = {};+ for (const [, key, raw] of body.matchAll(/([a-z][a-z0-9-]*)\s*:\s*([^;]+);/gu)) {+ const value = raw.trim();+ properties[key] = /^-?\d+(\.\d+)?$/u.test(value) ? Number(value) : value;+ }+ sheets.set(`${alias}.${name}`, { path: `${alias}.${name}`, properties });+ }+ }+ return sheets;+}++/**+ * The policy each Segment's duration was estimated with. A Source names it once per Segment through+ * `<estimate:Speech source={story.segment.NAME.speech} policy={recipes.X}/>`, and reading it back is+ * what keeps this on the Source's own basis rather than a second one.+ */+function policiesBySegment(svml, sheets) {+ const policies = new Map();+ const named = new Set();+ for (const [, attributes] of svml.matchAll(/<estimate:Speech\b([^>]*?)\/?>/gsu)) {+ const segment = /\bsource=\{story\.segment\.([A-Za-z0-9_-]+)\.speech\}/u.exec(attributes)?.[1];+ const recipe = /\bpolicy=\{([A-Za-z0-9_.-]+)\}/u.exec(attributes)?.[1];+ if (segment === undefined || recipe === undefined) continue;+ const sheet = sheets.get(recipe);+ if (sheet === undefined) throw new Error(`policy Recipe ${recipe} is not in any imported sheet`);+ named.add(recipe);+ policies.set(segment, speechEstimatePolicyFromRecipe(sheet));+ }+ // A Segment whose take is shared with another Segment carries no estimate of its own, so it has no+ // policy named against it. One policy across the Source settles those; several leaves no basis to+ // pick from, and guessing would put a Segment on a pace the Source never chose for it.+ const shared = named.size === 1 ? policies.get([...policies.keys()][0]) : undefined;+ return { policies, shared, policyCount: named.size };+}++const SILENT_AUDIO = { kind: "blob", digest: `sha256:${"0".repeat(64)}`, size: 1, mediaType: "audio/wav" };++/**+ * @param svmlPath the Author SVML this Source is written in+ * @param frameRate the Program's frame rate, as a whole number of frames per second+ * @returns one `{ segmentId, take }` per Segment, in Script order+ */+export async function standInTakes(svmlPath, frameRate) {+ const svml = await readFile(svmlPath, "utf8");+ const body = scriptBody(svml);+ const parsed = parseScript(svmlPath, body.text, body.offset);+ const sheets = await recipeSheets(svml, svmlPath);+ const { policies, shared, policyCount } = policiesBySegment(svml, sheets);++ const takes = [];+ // Global frame span of every word, in Script order, so a Selection can be turned into a frame+ // range without going near a clock. This is the correspondence the route uses everywhere else:+ // a stretch of the reference is found by its words, and its words are where the Script says.+ const frameOfToken = [];+ let frameCursor = 0;+ for (const segment of parsed.segments) {+ const policy = policies.get(segment.id) ?? shared;+ if (policy === undefined) {+ throw new Error(`Segment ${segment.id} names no estimate:Speech policy, and the Source uses ${policyCount} policies, so there is no single one to fall back to`);+ }+ const tokens = parsed.tokens.slice(segment.tokenStart, segment.tokenEndExclusive);+ const text = tokens.map((token) => token.text).join(" ");+ const seconds = estimateSpeechDuration({ value: text }, policy);+ const frameCount = Math.max(1, Math.round(seconds * frameRate));++ // Share the Segment's frames out by the estimator's own unit count, so the word order and the+ // relative widths both come from the same place the duration did.+ const language = resolveSpeechEstimateLanguage(text, policy.language);+ const weights = tokens.map((token) => Math.max(1, countSpeechEstimateUnits(token.text, language)));+ const total = weights.reduce((sum, weight) => sum + weight, 0);+ const anchors = [+ { identity: `segment:${segment.id}:start`, frame: 0 },+ { identity: `segment:${segment.id}:end`, frame: frameCount },+ ];+ const placed = [];+ let used = 0;+ for (const [index, token] of tokens.entries()) {+ const start = used;+ // The last token closes the Segment exactly, so rounding never leaves a frame unclaimed.+ used = index === tokens.length - 1+ ? frameCount+ : Math.min(frameCount - (tokens.length - 1 - index), start + Math.max(1, Math.round(frameCount * weights[index] / total)));+ const id = `segment:${segment.id}:token:${index + 1}`;+ placed.push({+ tokenId: id, segmentId: segment.id, text: token.text,+ startAnchorId: `${id}:start`, endAnchorId: `${id}:end`,+ startFrame: start, endFrameExclusive: used,+ });+ anchors.push({ identity: `${id}:start`, frame: start }, { identity: `${id}:end`, frame: used });+ }++ frameOfToken.push(...placed.map((token) => ({ frame: frameCursor + token.startFrame, end: frameCursor + token.endFrameExclusive })));+ takes.push({+ segmentId: segment.id,+ startFrame: frameCursor,+ take: {+ media: {+ timeline: { frameRate: { numerator: frameRate, denominator: 1 }, frameCount },+ audio: { artifact: SILENT_AUDIO, sampleFrames: Math.round(frameCount / frameRate * 48_000) },+ },+ segment: {+ segmentId: segment.id,+ startAnchorId: `segment:${segment.id}:start`,+ endAnchorId: `segment:${segment.id}:end`,+ startFrame: 0,+ endFrameExclusive: frameCount,+ },+ tokens: placed,+ anchors,+ },+ });+ frameCursor += frameCount;+ }++ // Every Selection the Script marks, as the frames its words occupy. Occurrences are unioned, since+ // one Selection id can be marked in several places and they nest freely.+ const selections = new Map();+ for (const selection of parsed.selections) {+ let start = Infinity;+ let end = 0;+ for (const occurrence of selection.occurrences) {+ const first = frameOfToken[occurrence.open.boundary.tokenIndex];+ const last = frameOfToken[occurrence.close.boundary.tokenIndex - 1];+ if (first === undefined || last === undefined) continue;+ start = Math.min(start, first.frame);+ end = Math.max(end, last.end);+ }+ if (start < end) selections.set(selection.id, { startFrame: start, endFrameExclusive: end });+ }+ return { takes, selections, frameCount: frameCursor };+}