Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

feat(hypit): render the comparison picture from the Source, not by hand

rpone1ef08de8f84e
diff --git a/.agents/skills/hypit/references/local-author-package.md b/.agents/skills/hypit/references/local-author-package.md
index 4bebf6be..ac8b9896 100644
--- a/.agents/skills/hypit/references/local-author-package.md
+++ b/.agents/skills/hypit/references/local-author-package.md
@@ -174,35 +174,20 @@ nothing.
- A component that cannot render on its own cannot produce the catalogue preview its Surface owes.
Treat a missing preview as evidence of this mistake rather than a step to skip. The preview proves
the component draws; what it draws is compared against the reference from a separate render, below.
-- **The render harness takes its inputs as arguments.** The catalogue preview is one picture: sample
- copy, a sample Recipe, every feature on, at whatever moment shows the component best.
- `reconstruction-loop.md` needs a different picture — the component drawn from the Recipe values a
- Source actually passes and the Script text that Source actually feeds it — for every reference shot
- the component appears in. The values are the difference, and they are the whole difference: a Source
- can pass copy and sizes that collide while the sample values stay perfect, so a picture drawn from
- the samples cannot report it.
-
- Give the harness these arguments, and let the catalogue preview be one invocation of it rather than
- its only purpose:
-
- - the path to the Recipe sheet the Source uses, and which Recipe in it;
- - the text the Source feeds the component, as the Source feeds it;
- - a path per media slot, and a path for the layer beneath;
- - the Canvas the Source declares, so the render composes at the geometry the mocks were built to;
- - `--out`, and whether to write a still or a clip covering a stretch.
-
- A harness with its inputs written into it produces the catalogue picture and nothing else, and the
- next agent reuses that picture for a comparison it does not fit.
-- **The harness does not invent media, for slots or for the layer beneath.** A slot declared as a
- generation is mockable but not fillable on this route, and neither is the base take a component
- draws over. Both mocks come from the route's fixed `make-placeholder` tool, not from code the
- harness ships: the harness reads the paths from its arguments and passes them in. No
- placeholder-drawing code inside the harness, no generated picture committed just to have something
- in a slot.
-
- This is a rendering argument and never a Source edit. The Source keeps declaring the generation;
- what the harness receives is a path that exists for one render. The gates that decide coverage read
- the Source's Recipes and bindings, so a mock cannot be counted as the thing it stands in for.
+- **The package owes a catalogue preview and nothing else.** That picture is drawn from sample copy
+ and a sample Recipe the package chooses, to show a reader the Surface's range, and it belongs to the
+ package the way its README does.
+
+ The picture `reconstruction-loop.md` compares is a different one and the package does not produce
+ it. `scripts/render-element.mjs` reads the Source — the Canvas, the Recipe values, the Script text,
+ the bindings — and drives the package's own Producer to draw the component the way this video places
+ it. All the package has to be is a working Producer, which the Manifest already requires.
+
+ Do not write a per-package harness for the comparison. One that takes its values as arguments still
+ gets them by hand, one line at a time, which draws one line and lets a collision between several go
+ unseen; one with its values written in draws the catalogue picture and cannot report the Source at
+ all. Both also decide their own canvas, and a canvas that is not the Source's puts every proportion
+ in the comparison slightly wrong.
## Freeze the Types before writing in parallel
diff --git a/.agents/skills/hypit/references/preview.md b/.agents/skills/hypit/references/preview.md
index 6081c4e7..827c76f4 100644
--- a/.agents/skills/hypit/references/preview.md
+++ b/.agents/skills/hypit/references/preview.md
@@ -52,38 +52,37 @@ Provider. The repository's own visual test `packages/hyperframes/test/browser-vi
the path end to end: build the Track, compile the HyperFrames document, and render it through the
local HyperFrames Runtime.
-There is no one command: a package's `test/render-preview.ts` harness is what renders it, and
-`local-author-package.md` requires that harness to take its inputs and its output path as arguments.
-That requirement exists because two different pictures come out of it, and they must not collapse
-into one:
+Two different pictures come out of this, and they must not collapse into one:
- the **catalogue preview** — one image for Studio and the package README, drawn from sample copy and
- a sample Recipe the harness supplies itself, chosen to show the Surface's range;
+ a sample Recipe the package supplies itself, chosen to show the Surface's range. It is the package's
+ own, like its README, and `local-author-package.md` says the package owes it;
- the **comparison render** — for `reconstruction-loop.md`, drawn from the Recipe values a Source
actually passes, the Script text that Source actually feeds the element, and mocks for the layers a
- Build has not made. One per reference shot the element appears in.
+ Build has not made.
-The inputs are what separates them, and it is the whole separation. A Source can pass values that
+The values are what separates them, and it is the whole separation. A Source can pass values that
collide — lines that overlap, type too large for the plate it sits on — while the catalogue preview,
-drawn from different values, stays perfect. A harness whose inputs are written into it can only
-produce the first picture, and a route that compares that picture is answering a question about the
-catalogue.
-
-Two things are mocked rather than left out. **The layers beneath**: a component that draws over a
-base take renders over black without one, and black is not neutral — text legible on it can be
-illegible on the picture that replaces it. **Media slots the Source declares as generations**: empty
-for the whole stretch between being written and being built. Both use the route's fixed
-`make-placeholder` tool — `--video --seconds <s>` for a stretch, a PNG for a slot, `--color` so the
-mock stays visible against what the element draws — sized from the Source's `space:Canvas` rather
-than from the reference video, and the comparison is scoped with `--question` so the observer skips
-those regions. `reconstruction-loop.md` gives the sizing rule and says why a reported mock is not a
-repair target.
-
-The render can be a clip as well as a still, and a clip is the default. `compare_reconstruction
---video` compares the whole shot, which removes the problem of choosing a frame to represent a shot
-that holds several states. A still is for the one case `reconstruction-loop.md` names: the shot's own
-visual observation says in words that the element is completely still. That judgement comes from the
-reference's observation, never from watching your own render.
+drawn from different values, stays perfect. A route that compares the catalogue picture is answering
+a question about the catalogue.
+
+The comparison render has one command, shared by every package:
+
+```bash
+node --import tsx .agents/skills/hypit/scripts/render-element.mjs projects/<name>/build.svrun \
+ --element <id> --segment <id>|--selection <id> --out <path>.mp4
+```
+
+It reads the Canvas, the Recipes, the Script and the bindings out of the Source; stands in for the
+speech the Build has yet to synthesize using the Source's own `estimate:Speech`, so the window
+lengths are the ones the Source already ordered its generations with; mocks every declared generation
+with `make-placeholder` at the Canvas's size; and drives each package's Producer through the same
+Studio projection `preview-check` traces, with no Provider installed. The mocks and the stand-in takes
+live in a derived Run under the project's `.hypit/`, never in the Source.
+
+Name the window in words. `--segment` and `--selection` take a Script name rather than a timestamp,
+because a shot of the reference is found by the words spoken over it. Write `.mp4` to `--out` for a
+clip and `.png` for the middle still; `reconstruction-loop.md` says which to use when.
This is what to reach for whenever one element has to be looked at rather than the whole program.
Rendering the delivery to inspect a single piece is the waste it exists to prevent.
diff --git a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md
index f2908348..5b4403d6 100644
--- a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md
+++ b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md
@@ -51,9 +51,21 @@ every one of those parameters with values that collide while the preview stays p
preview answers a question nobody asked. `../preview.md` renders both, from the same harness, with
different arguments; this loop uses the Source-configured one.
-The render is local and takes no paid Provider. `packages/hyperframes/test/browser-visual.test.ts`
-shows the path end to end: build the Track, compile the HyperFrames document, render it through the
-local HyperFrames Runtime.
+One command produces it, and it is not written per package:
+
+```
+node --import tsx .agents/skills/hypit/scripts/render-element.mjs projects/<name>/build.svrun \
+ --element <id> --segment <id>|--selection <id> --out <path>.mp4
+```
+
+It reads the Canvas, the Recipe values and the Script text out of the Source, stands in for the
+speech with the Source's own `estimate:Speech`, mocks every layer a Build has not made, and drives
+the package's Producer. Nothing is transcribed by hand, so nothing is transcribed one line at a time
+— which is what made a caption system look correct while its lines collided.
+
+The window is named in words. `--segment` and `--selection` take a Script name, not a timestamp: a
+shot of the reference is found by the words spoken over it, and those words are where the Script
+says they are. No reference clock is read across.
### Mock the layers a Build has not made
@@ -62,24 +74,17 @@ it may be media slots the Source declares as generations. Render them as mocks r
them out: a missing base is not a neutral background, it is black, and text that is legible on black
can be illegible on the picture that will replace it.
-Use the route's fixed placeholder tool for both — `hypit-reference-video-tools make-placeholder --out
-base.mp4 --width <w> --height <h> --video --seconds <s>` for the base stretch, `--out slot.png` for an
-image slot — with `--color` chosen so the mock is visible against what the element draws. Never
-`hypit image`, which pays for a generation the video will not reuse, and never a script written by
-hand.
-
-**The sizes come from the Source's own `space:Canvas`, not from the reference video.** The mock is
-composed inside the render, and the render's geometry is the Canvas. A base mock is the Canvas
-exactly — `<space:Canvas width="1080" height="1920"/>` means `--width 1080 --height 1920` — and a slot
-mock is that slot's `space:Frame` fractions multiplied by the Canvas, so a Frame from 16% to 84%
-across and 30% to 78% down is `--width 734 --height 922`. A mock built to the wrong box is resized by
-the `fit` before the observer ever sees it, which moves edges the comparison is there to read.
-
-The reference's own pixel dimensions are a different number and are usually smaller — the prepared
-reference stores them in `state.json` under `video`. They do not size the mock, but their **aspect**
-has to match the Canvas's. When it does not, every comparison in this loop is putting two
-differently-shaped pictures side by side, and the observer will report proportion differences that
-belong to the Canvas rather than to the element. Fix the Canvas before comparing anything.
+`render-element.mjs` does this itself. It finds every generation the Source declares — the takes, the
+stills, the slot contents — calls `make-placeholder` for each at the Canvas's own size, and declares
+them in a derived Run under the project's `.hypit/`. Never `hypit image`, which pays for a generation
+the video will not reuse, and never a placeholder drawn by a script written for the occasion.
+
+The sizes come from the Source's `space:Canvas`, not from the reference video, because the mock is
+composed inside the render and the render's geometry is the Canvas. The reference's own pixel
+dimensions are a different and usually smaller number, stored under `video` in its `state.json`. They
+do not size anything here, but their **aspect** has to match the Canvas's: when it does not, every
+comparison puts two differently-shaped pictures side by side and the observer reports proportion
+differences that belong to the Canvas rather than to the element. Fix the Canvas before comparing.
Then tell the observer, through `--question`, that those regions are placeholders standing in for
declared-but-unbuilt generations, and to compare only what the element itself draws. Said plainly it
diff --git a/.agents/skills/hypit/references/reconstruction/workflow.md b/.agents/skills/hypit/references/reconstruction/workflow.md
index 1fe8d95a..008fc360 100644
--- a/.agents/skills/hypit/references/reconstruction/workflow.md
+++ b/.agents/skills/hypit/references/reconstruction/workflow.md
@@ -41,9 +41,10 @@ list_svml_packages
→ run preview-check (final-sources.md) and repair until the graph traces —
this has no attempt ceiling; a target Studio cannot trace is not done.
Waiting on unrun Providers is a pass, not a failure
-→ read reconstruction-loop.md, then for each authored element: render it as the
- sources configure it, mocking the layers a Build has not made, and compare it
- against every shot the reference shows it in before repairing anything
+→ read reconstruction-loop.md, then for each authored element: render it with
+ render-element.mjs, which reads the sources and mocks the layers a Build has
+ not made, and compare it against every shot the reference shows it in before
+ repairing anything
→ repair against the differences that round returned, within the ceilings
→ run reconstruction-check (index.md) and keep going until it passes; it names
every locally-drawn element that has never been compared, and every timed
diff --git a/.agents/skills/hypit/scripts/render-element.mjs b/.agents/skills/hypit/scripts/render-element.mjs
new file mode 100644
index 00000000..5c034ad8
--- /dev/null
+++ b/.agents/skills/hypit/scripts/render-element.mjs
@@ -0,0 +1,359 @@
+#!/usr/bin/env node
+/**
+ * Render one element of a Source the way that Source configures it, without a Build and without a
+ * Provider.
+ *
+ * `reconstruction-loop.md` compares a reconstructed element against the reference. What it must
+ * compare is the element as this video places it — the Recipe values the Source passes, the Script
+ * text it feeds, the windows it binds — because a Source can fill those with values that collide
+ * while a package's catalogue preview, drawn from sample values, stays perfect. Producing that
+ * picture by hand means transcribing values into a per-package harness, one line at a time, which is
+ * how a caption system gets compared as a single line that has nothing to collide with.
+ *
+ * So this reads the values instead. Everything it needs is already written down:
+ *
+ * Canvas and Frames, Recipe values, bindings the Source
+ * which elements share a window, in what order the Script's words, through their Selections
+ * how long each Segment runs the Source's own `estimate:Speech`
+ * the layers a Build has not made `make-placeholder`, sized from the Canvas
+ *
+ * The one thing missing before a Build is real speech, and `stand-in-takes.mjs` supplies a Segment
+ * skeleton from the estimator the Source already trusts — the same numbers it ordered its takes
+ * with, not a second clock. Those takes enter through `<value>` and `<satisfy>`, the mechanism
+ * `examples/all-components-preview` uses to open in Studio without spending anything, so nothing
+ * here is a private back door into the graph.
+ *
+ * What this settles: which elements are on screen together, where each sits, at what size, weight and
+ * colour. What it does not: real timing, which waits for `playbooks/craft/production-gates.md` Gate 3.
+ *
+ * Usage:
+ * node --import tsx .agents/skills/hypit/scripts/render-element.mjs <build.svrun> \
+ * --element <id> --out <path.png|path.mp4> [--segment <id>] [--base <mock.mp4>]
+ *
+ * Exit: 0 having written --out. 1 when the element cannot be projected, naming what is unresolved.
+ * 2 on a usage or discovery error.
+ */
+import { mkdir, readFile, readdir, rm, writeFile } from "node:fs/promises";
+import { spawnSync } from "node:child_process";
+import { createHash } from "node:crypto";
+import { createRequire } from "node:module";
+import { dirname, join, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const studio = (name) => new URL(`../../../../packages/studio/src/${name}`, import.meta.url).href;
+const { openStudioArchive } = await import(studio("archive.js"));
+const { loadStudioDomain } = await import(studio("domain.js"));
+const { loadStudioRun } = await import(studio("run.js"));
+const { preview } = await import(studio("programme.js"));
+const { inspectStudioRun } = await import(studio("studio-preflight.js"));
+const { compileHyperframesDocument, materializeHyperframesHtml } = await import(new URL("../../../../packages/hyperframes/src/document.ts", import.meta.url).href);
+const { standInTakes } = await import(new URL("./stand-in-takes.mjs", import.meta.url).href);
+
+function fail(message, code = 2) {
+ console.error(`render-element: ${message}`);
+ process.exit(code);
+}
+
+function blobRef(bytes, mediaType) {
+ return { kind: "blob", digest: `sha256:${createHash("sha256").update(bytes).digest("hex")}`, size: bytes.byteLength, mediaType };
+}
+
+/** 48 kHz mono PCM silence, written by hand so the mock needs no encoder and lands on exact bytes. */
+function silentWav(sampleFrames) {
+ const data = Math.max(2, sampleFrames * 2);
+ const buffer = Buffer.alloc(44 + data);
+ buffer.write("RIFF", 0);
+ buffer.writeUInt32LE(36 + data, 4);
+ buffer.write("WAVEfmt ", 8);
+ buffer.writeUInt32LE(16, 16);
+ buffer.writeUInt16LE(1, 20);
+ buffer.writeUInt16LE(1, 22);
+ buffer.writeUInt32LE(48_000, 24);
+ buffer.writeUInt32LE(48_000 * 2, 28);
+ buffer.writeUInt16LE(2, 32);
+ buffer.writeUInt16LE(16, 34);
+ buffer.write("data", 36);
+ buffer.writeUInt32LE(data, 40);
+ return buffer;
+}
+
+const argv = process.argv.slice(2);
+const runArgument = argv[0];
+if (runArgument === undefined || runArgument.startsWith("--")) {
+ fail("usage: render-element.mjs <build.svrun> --element <id> --out <path> [--segment <id>] [--base <mock.mp4>]");
+}
+const flag = (name) => {
+ const index = argv.indexOf(`--${name}`);
+ return index === -1 ? undefined : argv[index + 1];
+};
+const element = flag("element") ?? fail("--element is required");
+const out = flag("out") ?? fail("--out is required");
+
+const invokedFrom = process.env.INIT_CWD ?? process.cwd();
+const runPath = resolve(invokedFrom, runArgument);
+const packageRoot = resolve(invokedFrom);
+const projectRoot = dirname(runPath);
+const outPath = resolve(invokedFrom, out);
+
+const runSource = await readFile(runPath, "utf8").catch(() => fail(`cannot read ${runPath}`));
+const authorMatch = /<author\s+source="([^"]+)"/u.exec(runSource);
+if (authorMatch === null) fail(`${runArgument} declares no <author source="…"/>`);
+const svmlPath = resolve(projectRoot, authorMatch[1]);
+const svml = await readFile(svmlPath, "utf8").catch(() => fail(`cannot read ${svmlPath}`));
+
+// The Program's frame rate and the Canvas the render composes at, both read rather than assumed. A
+// harness that hard-codes either produces a picture at a geometry the reference never had.
+const clock = /<[a-z-]*:?Clock\b[^>]*?\bframe-rate="(\d+)"/su.exec(svml)?.[1]
+ ?? /<time:Clock\b[^>]*?\bfps="(\d+)"/su.exec(svml)?.[1];
+const frameRate = Number(clock ?? 30);
+const canvasMatch = /<space:Canvas\b[^>]*?\bwidth="(\d+)"[^>]*?\bheight="(\d+)"/su.exec(svml);
+if (canvasMatch === null) fail("the Source declares no <space:Canvas width= height=/>");
+const canvas = { width: Number(canvasMatch[1]), height: Number(canvasMatch[2]) };
+
+// Which SemanticTake output belongs to which Segment, so each stand-in lands on the right one.
+const takeOutputs = new Map();
+for (const [, attributes] of svml.matchAll(/<whisperx:SemanticTake\b([^>]*?)\/?>/gsu)) {
+ const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];
+ const segment = /\bsegment=\{story\.segment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];
+ if (id !== undefined && segment !== undefined) takeOutputs.set(segment, id);
+}
+if (takeOutputs.size === 0) fail("the Source declares no whisperx:SemanticTake to stand in for");
+
+const { takes, selections, frameCount: programFrames } = await standInTakes(svmlPath, frameRate);
+const framesBySegment = new Map(takes.map((item) => [item.segmentId, item.take.segment.endFrameExclusive]));
+// A Selection's window in frames, summed over the stand-in tokens it covers. This is the same word
+// span the Source binds to, carried into frames by the same estimate that sized the Segment.
+const framesBySelection = new Map();
+for (const { segmentId, take } of takes) {
+ for (const token of take.tokens) framesBySelection.set(token.tokenId, token.endFrameExclusive - token.startFrame);
+ framesBySelection.set(`segment:${segmentId}`, take.segment.endFrameExclusive);
+}
+
+/**
+ * Every media a Build would have produced, and how long its window is. A speech take carries no
+ * picture and a base take fills a Segment; a cutaway fills whatever its `during=` binds. Each becomes
+ * a placeholder of the Canvas's own size, so the composite has the geometry the Source declares.
+ */
+function mockedMedia() {
+ const windows = new Map();
+ for (const [, attributes] of svml.matchAll(/<whisperx:SemanticTake\b([^>]*?)\/?>/gsu)) {
+ const media = /\bmedia=\{([A-Za-z0-9_-]+)\.media\}/u.exec(attributes)?.[1];
+ const segment = /\bsegment=\{story\.segment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];
+ if (media !== undefined && segment !== undefined) windows.set(media, framesBySegment.get(segment) ?? frameRate);
+ }
+ for (const [, attributes] of svml.matchAll(/<[a-z][a-z0-9-]*:[A-Za-z][A-Za-z0-9]*\b([^>]*?)\/?>/gsu)) {
+ const media = /\bmedia=\{([A-Za-z0-9_-]+)\.media\}/u.exec(attributes)?.[1];
+ if (media === undefined || windows.has(media)) continue;
+ const segment = /\bduring=\{story\.segment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];
+ if (segment !== undefined) { windows.set(media, framesBySegment.get(segment) ?? frameRate); continue; }
+ windows.set(media, frameRate * 2);
+ }
+ const carries = new Map();
+ for (const [, attributes] of svml.matchAll(/<pipeline:Normalize\b([^>]*?)\/?>/gsu)) {
+ const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];
+ const video = /\bvideo="([^"]+)"/u.exec(attributes)?.[1];
+ if (id === undefined || !windows.has(id)) continue;
+ carries.set(id, { frames: windows.get(id), picture: video !== "none" });
+ }
+ return carries;
+}
+
+// A derived Run lives beside the project so the original's relative sources still resolve, and under
+// `.hypit/` so it is never mistaken for something the author wrote. It is rewritten every run.
+const compareRoot = join(projectRoot, ".hypit", "compare");
+await rm(compareRoot, { recursive: true, force: true });
+await mkdir(compareRoot, { recursive: true });
+const declarations = [];
+const satisfactions = [];
+for (const { segmentId, take } of takes) {
+ const output = takeOutputs.get(segmentId);
+ if (output === undefined) continue;
+ // The take's own audio has to be servable, not a digest stub: Studio resolves every Artifact a
+ // projection references, and a preview that cannot serve one stops there.
+ const sampleFrames = Math.round(take.segment.endFrameExclusive / frameRate * 48_000);
+ const silence = silentWav(sampleFrames);
+ await writeFile(join(compareRoot, `${segmentId}-take.wav`), silence);
+ const spoken = { ...take, media: { ...take.media, audio: { artifact: blobRef(silence, "audio/wav"), sampleFrames } } };
+ const fixture = join(compareRoot, `${segmentId}.json`);
+ await writeFile(fixture, `${JSON.stringify({ kind: "inline", value: spoken }, null, 2)}\n`, "utf8");
+ declarations.push(` <file id="stand-in-${segmentId}-audio" type="@hypit/artifact@1#BlobArtifact" from="./${segmentId}-take.wav" media-type="audio/wav"/>`);
+ declarations.push(` <value id="stand-in-${segmentId}" type="@hypit/speech@1#SemanticTake" from="./${segmentId}.json"/>`);
+ satisfactions.push(` <satisfy output="${output}.take" candidate="stand-in-${segmentId}"/>`);
+}
+// The layers a Build has not made, as placeholders of the Canvas's own size. `make-placeholder` is
+// the route's one tool for this: deterministic, Provider-free, and the same mock the comparison is
+// told to bypass. A mock never enters the project's own Source — it is declared in the derived Run
+// and nowhere else.
+const placeholderCli = new URL("../../../../packages/reference-video-tools/src/cli.ts", import.meta.url);
+for (const [id, { frames, picture }] of mockedMedia()) {
+ const seconds = Math.max(0.1, frames / frameRate);
+ const file = join(compareRoot, `${id}.mp4`);
+ if (picture) {
+ const made = spawnSync(process.execPath, [
+ "--import", "tsx", fileURLToPath(placeholderCli), "make-placeholder",
+ "--out", file, "--width", String(canvas.width), "--height", String(canvas.height),
+ "--video", "--seconds", seconds.toFixed(3), "--color", "mid",
+ ], { encoding: "utf8", windowsHide: true, timeout: 120_000 });
+ if (made.status !== 0) fail(`make-placeholder failed for ${id}: ${made.stderr?.trim()}`);
+ }
+ // Every SynchronizedMedia carries audio, and it is validated as WAV, so the silence is written as
+ // one rather than pointed at the video. Both are declared: the picture is what gets drawn, the
+ // silence is what makes the value legal.
+ const wavPath = join(compareRoot, `${id}.wav`);
+ const wav = silentWav(Math.round(frames / frameRate * 48_000));
+ await writeFile(wavPath, wav);
+ const audioBlob = blobRef(wav, "audio/wav");
+ const media = {
+ timeline: { frameRate: { numerator: frameRate, denominator: 1 }, frameCount: frames },
+ audio: { artifact: audioBlob },
+ };
+ declarations.push(` <file id="mock-${id}-audio" type="@hypit/artifact@1#BlobArtifact" from="./${id}.wav" media-type="audio/wav"/>`);
+ if (picture) {
+ const bytes = await readFile(file);
+ media.visual = { artifact: blobRef(bytes, "video/mp4"), width: canvas.width, height: canvas.height };
+ declarations.push(` <file id="mock-${id}" type="@hypit/artifact@1#BlobArtifact" from="./${id}.mp4" media-type="video/mp4"/>`);
+ }
+ await writeFile(join(compareRoot, `${id}.media.json`), `${JSON.stringify({ kind: "inline", value: media }, null, 2)}\n`, "utf8");
+ declarations.push(` <value id="media-${id}" type="@hypit/media@1#SynchronizedMedia" from="./${id}.media.json"/>`);
+ satisfactions.push(` <satisfy output="${id}.media" candidate="media-${id}"/>`);
+}
+
+// Still images the Source declares as generations. Same treatment, same tool, at the Canvas's size:
+// a picture slot that is empty in the render is black, and black is not what will be there.
+const imageIds = [...new Set([...svml.matchAll(/\{([a-z0-9-]+)\.image\}/gu)].map(([, id]) => id))];
+for (const id of imageIds) {
+ const file = join(compareRoot, `${id}.png`);
+ const made = spawnSync(process.execPath, [
+ "--import", "tsx", fileURLToPath(placeholderCli), "make-placeholder",
+ "--out", file, "--width", String(canvas.width), "--height", String(canvas.height), "--color", "mid",
+ ], { encoding: "utf8", windowsHide: true, timeout: 120_000 });
+ if (made.status !== 0) fail(`make-placeholder failed for ${id}: ${made.stderr?.trim()}`);
+ declarations.push(` <file id="mock-${id}" type="@hypit/artifact@1#BlobArtifact" from="./${id}.png" media-type="image/png"/>`);
+ satisfactions.push(` <satisfy output="${id}.image" candidate="mock-${id}"/>`);
+}
+
+const derivedRun = join(compareRoot, "compare.svrun");
+await writeFile(derivedRun, [
+ `<?svml using="@hypit/run-markup@1"?>`,
+ ``,
+ `<svrun version="1">`,
+ ` <author source="../../${authorMatch[1].replace(/^\.\//u, "")}"/>`,
+ // Targeting the element's own output rather than the Film prunes the closure to what this one
+ // Track needs. Asking for the whole delivery would pull in every generation the Source declares
+ // and report each as unresolved, which is true and useless: none of them is what is being looked at.
+ ` <target output="final.video"/>`,
+ ``,
+ ...declarations,
+ ``,
+ ...satisfactions,
+ `</svrun>`,
+ ``,
+].join("\n"), "utf8");
+
+const domain = await loadStudioDomain({ run: derivedRun, workspaceRoot: projectRoot, packageRoot });
+const archive = await openStudioArchive(undefined, packageRoot);
+let built;
+try {
+ const run = await loadStudioRun({ run: derivedRun, domain, ...(archive === undefined ? {} : { archive }) });
+ const inspection = inspectStudioRun(run.source, run);
+ // Every Track, not only the one being looked at: the element is compared where it sits, over the
+ // mocked base rather than on its own, because text that is legible on black may not be on a picture.
+ built = await preview({
+ source: run.source,
+ run,
+ domain,
+ outputRefs: [inspection.filmComposition, ...inspection.projections.map((item) => item.ref)],
+ compositionRef: inspection.filmComposition,
+ projections: inspection.projections,
+ ...(archive === undefined ? {} : { archive }),
+ });
+} catch (error) {
+ const issues = error?.issues;
+ if (Array.isArray(issues)) {
+ console.error("render-element: the Source does not project yet.");
+ for (const issue of issues.slice(0, 12)) console.error(` ${issue}`);
+ process.exit(1);
+ }
+ fail(error instanceof Error ? error.message : String(error), 1);
+}
+
+const placed = built.tracks.find((track) => String(track.outputRef ?? "").endsWith(`::output::${element}.track`)
+ || String(track.outputRef ?? "").includes(`::output::${element}.`));
+if (placed === undefined) {
+ console.error(`render-element: the Source places no element named ${element}. It places:`);
+ for (const track of built.tracks) console.error(` ${String(track.outputRef ?? "").split("::output::")[1] ?? track.name}`);
+ process.exit(1);
+}
+
+// The window to render, named in words. A shot of the reference is found by the words spoken over it
+// and those words are a Segment or a Selection here, so no reference timestamp is ever read across.
+const segmentArgument = flag("segment");
+const selectionArgument = flag("selection");
+let window = { startFrame: 0, endFrameExclusive: programFrames };
+if (selectionArgument !== undefined) {
+ window = selections.get(selectionArgument)
+ ?? fail(`the Script marks no Selection ${selectionArgument}`);
+} else if (segmentArgument !== undefined) {
+ let cursor = 0;
+ const take = takes.find((item) => { const hit = item.segmentId === segmentArgument; if (!hit) cursor += item.take.segment.endFrameExclusive; return hit; });
+ if (take === undefined) fail(`the Script has no Segment ${segmentArgument}`);
+ window = { startFrame: cursor, endFrameExclusive: cursor + take.take.segment.endFrameExclusive };
+}
+
+// Compile once, materialize with the artifact bytes written beside the document, and let the
+// HyperFrames runtime draw it. This is the path packages/hyperframes/test/browser-visual.test.ts
+// takes; nothing here is a private renderer.
+const document = compileHyperframesDocument(built.composition, built.space);
+const stage = join(compareRoot, "stage");
+await mkdir(stage, { recursive: true });
+const names = new Map();
+for (const [digest, file] of built.served) {
+ const name = `${digest.replace(/[^a-z0-9]/giu, "")}`;
+ await writeFile(join(stage, name), file.bytes);
+ names.set(digest, name);
+}
+await writeFile(join(stage, "index.html"), materializeHyperframesHtml(document, (artifact) => {
+ const name = names.get(artifact.digest);
+ if (name === undefined) fail(`the projection references Artifact ${artifact.digest}, which was not served`, 1);
+ return `./${name}`;
+}), "utf8");
+
+const clip = /\.(mp4|mov|webm)$/iu.test(outPath);
+const frames = join(compareRoot, "frames");
+await rm(frames, { recursive: true, force: true });
+await mkdir(frames, { recursive: true });
+const hyperframesCli = createRequire(join(packageRoot, "packages/provider-hyperframes-local/package.json"))
+ .resolve("hyperframes/bin/hyperframes.mjs");
+const drawn = spawnSync(process.execPath, [
+ hyperframesCli, "render", stage,
+ "--format", "png-sequence", "--output", frames, "--fps", String(frameRate),
+ "--workers", "1", "--no-browser-gpu", "--no-best-effort", "--quiet",
+], { encoding: "utf8", windowsHide: true, timeout: 600_000 });
+if (drawn.status !== 0) fail(`the HyperFrames runtime refused: ${(drawn.stderr ?? "").trim().slice(-2000)}`, 1);
+
+const written = (await readdir(frames)).filter((name) => name.endsWith(".png")).sort();
+if (written.length === 0) fail("the HyperFrames runtime wrote no frames", 1);
+const first = Math.min(window.startFrame, written.length - 1);
+const last = Math.min(window.endFrameExclusive, written.length);
+if (clip) {
+ const encoded = spawnSync("ffmpeg", [
+ "-hide_banner", "-loglevel", "error", "-y", "-framerate", String(frameRate),
+ "-start_number", String(first), "-i", join(frames, "frame_%06d.png"),
+ "-frames:v", String(Math.max(1, last - first)),
+ "-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "20", outPath,
+ ], { encoding: "utf8", windowsHide: true, timeout: 600_000 });
+ if (encoded.status !== 0) fail(`ffmpeg refused: ${(encoded.stderr ?? "").trim().slice(-2000)}`, 1);
+} else {
+ await writeFile(outPath, await readFile(join(frames, written[Math.min(first + Math.floor((last - first) / 2), written.length - 1)])));
+}
+
+console.log(JSON.stringify({
+ element,
+ out: outPath,
+ compared: clip ? "clip" : "still",
+ canvas,
+ frame_rate: frameRate,
+ window,
+ program_frames: programFrames,
+ mocked: { media: mockedMedia().size, images: imageIds.length },
+}, null, 2));
diff --git a/.agents/skills/hypit/scripts/stand-in-takes.mjs b/.agents/skills/hypit/scripts/stand-in-takes.mjs
new file mode 100644
index 00000000..268c7dea
--- /dev/null
+++ b/.agents/skills/hypit/scripts/stand-in-takes.mjs
@@ -0,0 +1,174 @@
+/**
+ * Build a stand-in SemanticTake for every Segment a Source declares, from the Source alone.
+ *
+ * A Track timed against speech cannot be projected before the speech exists, and on this route it
+ * does not exist: the Build has not run. What the Source does hold is the words themselves and the
+ * estimator it already trusts to size its own generations — `estimate:Speech` with a policy Recipe.
+ * Running that estimator here produces the same numbers the Source used to order its takes, so this
+ * introduces no second clock; it reads the one already written down.
+ *
+ * What the result is good for: which elements are on screen together, where each sits, at what size
+ * and colour. Those follow from word ranges and Recipe values and are exact. What it is not good for
+ * is real timing — the delivered speech is not the estimate, and anything measured in seconds waits
+ * for `production-gates.md` Gate 3.
+ *
+ * Tokens are laid across the Segment in proportion to the same syllable count the estimator uses, so
+ * a long word occupies more of the window than a short one and the ordering is the Script's.
+ */
+import { readFile } from "node:fs/promises";
+import { dirname, resolve } from "node:path";
+
+const from = (path) => new URL(`../../../../packages/${path}`, import.meta.url).href;
+const { parseScript } = await import(from("script/src/parser.ts"));
+const { estimateSpeechDuration, countSpeechEstimateUnits, resolveSpeechEstimateLanguage } =
+ await import(from("estimate/src/program.ts"));
+const { speechEstimatePolicyFromRecipe } = await import(from("estimate/src/surface.ts"));
+
+/** The Script body, which `parseScript` takes on its own. */
+function scriptBody(svml) {
+ const open = /<script\b[^>]*>/u.exec(svml);
+ if (open === null) throw new Error("the Source declares no <script>");
+ const start = open.index + open[0].length;
+ const end = svml.indexOf("</script>", start);
+ if (end < 0) throw new Error("the Source's <script> is not closed");
+ return { text: svml.slice(start, end), offset: start };
+}
+
+/** Every Recipe body in every sheet the Source imports, keyed `alias.path`. */
+async function recipeSheets(svml, svmlPath) {
+ const sheets = new Map();
+ for (const [, alias, source] of svml.matchAll(/<import\s+as="([^"]+)"\s+source="([^"]+\.svs)"/gu)) {
+ const text = await readFile(resolve(dirname(svmlPath), source), "utf8").catch(() => undefined);
+ if (text === undefined) continue;
+ for (const [, name, body] of text.matchAll(/([A-Za-z0-9_.-]+)\s*\{([^}]*)\}/gu)) {
+ const properties = {};
+ for (const [, key, raw] of body.matchAll(/([a-z][a-z0-9-]*)\s*:\s*([^;]+);/gu)) {
+ const value = raw.trim();
+ properties[key] = /^-?\d+(\.\d+)?$/u.test(value) ? Number(value) : value;
+ }
+ sheets.set(`${alias}.${name}`, { path: `${alias}.${name}`, properties });
+ }
+ }
+ return sheets;
+}
+
+/**
+ * The policy each Segment's duration was estimated with. A Source names it once per Segment through
+ * `<estimate:Speech source={story.segment.NAME.speech} policy={recipes.X}/>`, and reading it back is
+ * what keeps this on the Source's own basis rather than a second one.
+ */
+function policiesBySegment(svml, sheets) {
+ const policies = new Map();
+ const named = new Set();
+ for (const [, attributes] of svml.matchAll(/<estimate:Speech\b([^>]*?)\/?>/gsu)) {
+ const segment = /\bsource=\{story\.segment\.([A-Za-z0-9_-]+)\.speech\}/u.exec(attributes)?.[1];
+ const recipe = /\bpolicy=\{([A-Za-z0-9_.-]+)\}/u.exec(attributes)?.[1];
+ if (segment === undefined || recipe === undefined) continue;
+ const sheet = sheets.get(recipe);
+ if (sheet === undefined) throw new Error(`policy Recipe ${recipe} is not in any imported sheet`);
+ named.add(recipe);
+ policies.set(segment, speechEstimatePolicyFromRecipe(sheet));
+ }
+ // A Segment whose take is shared with another Segment carries no estimate of its own, so it has no
+ // policy named against it. One policy across the Source settles those; several leaves no basis to
+ // pick from, and guessing would put a Segment on a pace the Source never chose for it.
+ const shared = named.size === 1 ? policies.get([...policies.keys()][0]) : undefined;
+ return { policies, shared, policyCount: named.size };
+}
+
+const SILENT_AUDIO = { kind: "blob", digest: `sha256:${"0".repeat(64)}`, size: 1, mediaType: "audio/wav" };
+
+/**
+ * @param svmlPath the Author SVML this Source is written in
+ * @param frameRate the Program's frame rate, as a whole number of frames per second
+ * @returns one `{ segmentId, take }` per Segment, in Script order
+ */
+export async function standInTakes(svmlPath, frameRate) {
+ const svml = await readFile(svmlPath, "utf8");
+ const body = scriptBody(svml);
+ const parsed = parseScript(svmlPath, body.text, body.offset);
+ const sheets = await recipeSheets(svml, svmlPath);
+ const { policies, shared, policyCount } = policiesBySegment(svml, sheets);
+
+ const takes = [];
+ // Global frame span of every word, in Script order, so a Selection can be turned into a frame
+ // range without going near a clock. This is the correspondence the route uses everywhere else:
+ // a stretch of the reference is found by its words, and its words are where the Script says.
+ const frameOfToken = [];
+ let frameCursor = 0;
+ for (const segment of parsed.segments) {
+ const policy = policies.get(segment.id) ?? shared;
+ if (policy === undefined) {
+ throw new Error(`Segment ${segment.id} names no estimate:Speech policy, and the Source uses ${policyCount} policies, so there is no single one to fall back to`);
+ }
+ const tokens = parsed.tokens.slice(segment.tokenStart, segment.tokenEndExclusive);
+ const text = tokens.map((token) => token.text).join(" ");
+ const seconds = estimateSpeechDuration({ value: text }, policy);
+ const frameCount = Math.max(1, Math.round(seconds * frameRate));
+
+ // Share the Segment's frames out by the estimator's own unit count, so the word order and the
+ // relative widths both come from the same place the duration did.
+ const language = resolveSpeechEstimateLanguage(text, policy.language);
+ const weights = tokens.map((token) => Math.max(1, countSpeechEstimateUnits(token.text, language)));
+ const total = weights.reduce((sum, weight) => sum + weight, 0);
+ const anchors = [
+ { identity: `segment:${segment.id}:start`, frame: 0 },
+ { identity: `segment:${segment.id}:end`, frame: frameCount },
+ ];
+ const placed = [];
+ let used = 0;
+ for (const [index, token] of tokens.entries()) {
+ const start = used;
+ // The last token closes the Segment exactly, so rounding never leaves a frame unclaimed.
+ used = index === tokens.length - 1
+ ? frameCount
+ : Math.min(frameCount - (tokens.length - 1 - index), start + Math.max(1, Math.round(frameCount * weights[index] / total)));
+ const id = `segment:${segment.id}:token:${index + 1}`;
+ placed.push({
+ tokenId: id, segmentId: segment.id, text: token.text,
+ startAnchorId: `${id}:start`, endAnchorId: `${id}:end`,
+ startFrame: start, endFrameExclusive: used,
+ });
+ anchors.push({ identity: `${id}:start`, frame: start }, { identity: `${id}:end`, frame: used });
+ }
+
+ frameOfToken.push(...placed.map((token) => ({ frame: frameCursor + token.startFrame, end: frameCursor + token.endFrameExclusive })));
+ takes.push({
+ segmentId: segment.id,
+ startFrame: frameCursor,
+ take: {
+ media: {
+ timeline: { frameRate: { numerator: frameRate, denominator: 1 }, frameCount },
+ audio: { artifact: SILENT_AUDIO, sampleFrames: Math.round(frameCount / frameRate * 48_000) },
+ },
+ segment: {
+ segmentId: segment.id,
+ startAnchorId: `segment:${segment.id}:start`,
+ endAnchorId: `segment:${segment.id}:end`,
+ startFrame: 0,
+ endFrameExclusive: frameCount,
+ },
+ tokens: placed,
+ anchors,
+ },
+ });
+ frameCursor += frameCount;
+ }
+
+ // Every Selection the Script marks, as the frames its words occupy. Occurrences are unioned, since
+ // one Selection id can be marked in several places and they nest freely.
+ const selections = new Map();
+ for (const selection of parsed.selections) {
+ let start = Infinity;
+ let end = 0;
+ for (const occurrence of selection.occurrences) {
+ const first = frameOfToken[occurrence.open.boundary.tokenIndex];
+ const last = frameOfToken[occurrence.close.boundary.tokenIndex - 1];
+ if (first === undefined || last === undefined) continue;
+ start = Math.min(start, first.frame);
+ end = Math.max(end, last.end);
+ }
+ if (start < end) selections.set(selection.id, { startFrame: start, endFrameExclusive: end });
+ }
+ return { takes, selections, frameCount: frameCursor };
+}