Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
fix(hypit): compare the element the Source configures, over the whole shot
rpone050fbf2625d6
diff --git a/.agents/skills/hypit/references/local-author-package.md b/.agents/skills/hypit/references/local-author-package.mdindex cb48e93e..4bebf6be 100644--- a/.agents/skills/hypit/references/local-author-package.md+++ b/.agents/skills/hypit/references/local-author-package.md@@ -171,20 +171,38 @@ nothing.the Producer refuses, the Track never builds, and the rows nobody was waiting for vanish with it.This is what a Source looks like for the whole stretch between being written and being built, whichis most of its life.-- A component that cannot render on its own cannot produce the preview image its Surface owes. Treat- a missing preview as evidence of this mistake rather than a step to skip.-- **The render harness takes its state as arguments.** The Surface preview is one picture: every- feature on, at whatever moment shows the component best. `reconstruction-loop.md` needs a different- picture for every reference shot — the state that shot happens to show, at a matching moment — and- a component appears across several. A harness with its state and frame written into it produces the- catalogue picture and nothing else, and the next agent reuses that picture for a comparison it does- not fit. Give it `--out`, the frame, and whichever of the component's own switches the reference- varies; the preview then becomes one invocation of it rather than its only purpose.-- **The harness does not invent media for slots a Build has not filled.** A slot declared as a- generation is mockable but not fillable on this route, and the mock is the route's fixed- `make-placeholder` tool, not code the harness ships: the harness reads a placeholder path from its- arguments and passes it into the slot. No placeholder-drawing code inside the harness, no generated- picture committed just to have something in the slot.+- A component that cannot render on its own cannot produce the catalogue preview its Surface owes.+ Treat a missing preview as evidence of this mistake rather than a step to skip. The preview proves+ the component draws; what it draws is compared against the reference from a separate render, below.+- **The render harness takes its inputs as arguments.** The catalogue preview is one picture: sample+ copy, a sample Recipe, every feature on, at whatever moment shows the component best.+ `reconstruction-loop.md` needs a different picture — the component drawn from the Recipe values a+ Source actually passes and the Script text that Source actually feeds it — for every reference shot+ the component appears in. The values are the difference, and they are the whole difference: a Source+ can pass copy and sizes that collide while the sample values stay perfect, so a picture drawn from+ the samples cannot report it.++ Give the harness these arguments, and let the catalogue preview be one invocation of it rather than+ its only purpose:++ - the path to the Recipe sheet the Source uses, and which Recipe in it;+ - the text the Source feeds the component, as the Source feeds it;+ - a path per media slot, and a path for the layer beneath;+ - the Canvas the Source declares, so the render composes at the geometry the mocks were built to;+ - `--out`, and whether to write a still or a clip covering a stretch.++ A harness with its inputs written into it produces the catalogue picture and nothing else, and the+ next agent reuses that picture for a comparison it does not fit.+- **The harness does not invent media, for slots or for the layer beneath.** A slot declared as a+ generation is mockable but not fillable on this route, and neither is the base take a component+ draws over. Both mocks come from the route's fixed `make-placeholder` tool, not from code the+ harness ships: the harness reads the paths from its arguments and passes them in. No+ placeholder-drawing code inside the harness, no generated picture committed just to have something+ in a slot.++ This is a rendering argument and never a Source edit. The Source keeps declaring the generation;+ what the harness receives is a path that exists for one render. The gates that decide coverage read+ the Source's Recipes and bindings, so a mock cannot be counted as the thing it stands in for.## Freeze the Types before writing in paralleldiff --git a/.agents/skills/hypit/references/playbooks/craft/frame-coverage.md b/.agents/skills/hypit/references/playbooks/craft/frame-coverage.mdindex 9f6b8ea7..a041dd61 100644--- a/.agents/skills/hypit/references/playbooks/craft/frame-coverage.md+++ b/.agents/skills/hypit/references/playbooks/craft/frame-coverage.md@@ -17,7 +17,11 @@ an interval inherits its edges whenever its length comes from something other th- a window derived from speech, which starts and stops with words and therefore excludes the silencearound and between them;-- a generated take, which ends where its material ends rather than where the shot should;+- a generated take, which ends where its material ends rather than where the shot should — and then+ draws nothing for the rest of its window, because `playback` defaults to `once-start`. This is the+ one edge on the list a machine can find before a Build: `reconstruction-check` reads each Recipe+ and refuses the default on timed generated material. `generated-dependencies.md` says what to set+ instead;- a blend, whose frames are edges of partial coverage — a one-frame fade is one frame on which thelayer beneath is half visible, and naming the Recipe for a cut does not make it one;- a schedule inside a component, which draws only where its own Program says to and stops between.@@ -37,6 +41,11 @@ This matters most right after a fix: closing a black gap by bedding a layer undesymptom everyone can see into one that reads as an edit, and the reconstruction looks finished whilethe same edges are still wrong.+The placeholders the comparison loop renders under an element are not beds and cannot become one.+They exist in that render and nowhere else — the Source never names them, and the checks that decide+coverage read the Source's Recipes and bindings rather than any picture. A mock cannot quiet a hole+it is structurally unable to reach.+## Measure it; the eye is the wrong instrumentAn instant is a thirtieth of a second. Watching finds a scene that is wrong and slides straight pastdiff --git a/.agents/skills/hypit/references/playbooks/craft/generated-dependencies.md b/.agents/skills/hypit/references/playbooks/craft/generated-dependencies.mdindex 2ad35fba..b930ad0d 100644--- a/.agents/skills/hypit/references/playbooks/craft/generated-dependencies.md+++ b/.agents/skills/hypit/references/playbooks/craft/generated-dependencies.md@@ -132,13 +132,24 @@ Two different holes open, and both look identical on screen:one picking up where the last left off — rather than merely landing in the right places. A linethat introduces what comes next usually belongs to the Selection it introduces.- **A take shorter than the window it fills.** An Item whose window outlasts its own material runs- out partway and leaves the rest empty. Read the model's duration ceiling before deciding: Seedance- `mini` stops at 15 seconds, so a longer stretch needs more than one Item rather than one Item asked- for a length the model refuses.--Give a silent take a literal duration at or above its window instead of a `SpeechDuration` edge. The-estimate predicts the words; the window is decided by the audio that was actually produced, and when-the estimate falls a second short that second is black.+ out partway, and what happens for the remainder is decided by the `playback` key on its appearance+ Recipe. The default is `once-start`, which draws the material once and then draws nothing at all —+ no sampling segment is emitted for those frames, so they fall through to whatever is beneath, and+ beneath a full-frame base that is the Film background. `hold-start` pins the last frame for the+ rest of the window, `loop-start` repeats, `stretch` retimes to fit.++ Read the model's duration ceiling too: Seedance `mini` stops at 15 seconds, so a longer stretch+ needs more than one Item rather than one Item asked for a length the model refuses.++Set `playback` on every timed picture whose window comes from speech, and treat the material's length+as unknown. A generation is ordered in whole seconds, the estimate that sizes the order predicts the+words rather than measuring them, and the window is decided by the audio that was actually produced —+three separate numbers that will not agree. Lengthening the material moves the mismatch without+removing it; `playback` decides what the window shows once the material is spent, which is the part+that stays true whichever way the three numbers fall.++`reconstruction-check` reads this before a Build and refuses a Recipe that leaves `playback` at its+default on generated material.A bed makes a blend visible, so check the Items' entry and exit while you are here: `enter` and `exit`default to `none`, and a Recipe named for a cut that fades for a frame is one of the inherited edgesdiff --git a/.agents/skills/hypit/references/playbooks/craft/production-gates.md b/.agents/skills/hypit/references/playbooks/craft/production-gates.mdindex d38486b1..e5c89cb9 100644--- a/.agents/skills/hypit/references/playbooks/craft/production-gates.md+++ b/.agents/skills/hypit/references/playbooks/craft/production-gates.md@@ -117,6 +117,11 @@ visual QA.and whether each Track contributes unique information.- Treat `check` and `plan` as structural proofs only. They do not prove visual quality, real speechalignment, Provider output quality, or factual correctness.+- A reconstruction route may have compared these Tracks against a reference before this gate. That+ comparison runs on renders made from the Source's values with no SemanticTrack behind them, so it+ settles how a Track looks and leaves everything this gate is about — timing against real speech,+ drift after a recompute, occlusion between Tracks that were rendered separately — still to be+ measured here. Review them against the real SemanticTrack whatever was compared earlier.## Gate 4: delivery Builddiff --git a/.agents/skills/hypit/references/preview.md b/.agents/skills/hypit/references/preview.mdindex ccc8600b..6081c4e7 100644--- a/.agents/skills/hypit/references/preview.md+++ b/.agents/skills/hypit/references/preview.md@@ -1,7 +1,7 @@# See what you authored, without payingThree ways to look at a Source before a Provider is ever reached: prove the graph traces, render one-element to a still, and open the whole Run for a person. They answer different questions, and none of+element locally, and open the whole Run for a person. They answer different questions, and none ofthem costs a generation.## The graph traces before anybody opens it@@ -45,36 +45,48 @@ What this proves is that the graph is **wired**, not that every track **draws**.refuses the media kind it is handed is not caught here, because nothing is handed to it until theBuild runs.-## Render one element to a still, locally--One element — a new component, one caption system, one inserted card — can be rendered to an image-without a paid Provider. A new package's visual Surface needs a preview image anyway. The repository's-own visual test `packages/hyperframes/test/browser-visual.test.ts` shows the path end to end: build-the Track, compile the HyperFrames document, and render it through the local HyperFrames Runtime.--There is no one command for the still: a package's `test/render-preview.ts` harness is what renders-it, and `local-author-package.md` requires that harness to take its state, frame and output path as-arguments. That requirement exists because the still has two distinct jobs that must not collapse-into one picture:--- the **Surface preview** — one catalogue image, every feature on, whatever moment shows the- component best, for Studio and the package README;-- the **comparison still** — for `reconstruction-loop.md`, one picture per reference shot, showing- the element in the state that shot happens to show at a matching moment.--A harness whose state is written into it produces the first and nothing else; the route then reuses-that catalogue picture for a comparison it does not fit. So the still render is one invocation of a-parameterised harness, not the whole of it.--One case to handle rather than avoid: an element whose appearance depends on a media slot a Build-has not filled. The slot is mocked with the route's fixed `make-placeholder` tool (a PNG, or `--video`-for a slot that only accepts video), the comparison is scoped with `--question` to bypass the slot-as an intentional placeholder, and `reconstruction-loop.md` says the empty-slot difference is not a-repair target — so the still renders what the component draws itself with the mock in the slot, and-the round is spent on the differences that can actually be repaired.--This is the image to reach for whenever one element has to be looked at rather than the whole-program. Rendering the delivery to inspect a single piece is the waste it exists to prevent.+## Render one element locally++One element — a new component, one caption system, one inserted card — can be rendered without a paid+Provider. The repository's own visual test `packages/hyperframes/test/browser-visual.test.ts` shows+the path end to end: build the Track, compile the HyperFrames document, and render it through the+local HyperFrames Runtime.++There is no one command: a package's `test/render-preview.ts` harness is what renders it, and+`local-author-package.md` requires that harness to take its inputs and its output path as arguments.+That requirement exists because two different pictures come out of it, and they must not collapse+into one:++- the **catalogue preview** — one image for Studio and the package README, drawn from sample copy and+ a sample Recipe the harness supplies itself, chosen to show the Surface's range;+- the **comparison render** — for `reconstruction-loop.md`, drawn from the Recipe values a Source+ actually passes, the Script text that Source actually feeds the element, and mocks for the layers a+ Build has not made. One per reference shot the element appears in.++The inputs are what separates them, and it is the whole separation. A Source can pass values that+collide — lines that overlap, type too large for the plate it sits on — while the catalogue preview,+drawn from different values, stays perfect. A harness whose inputs are written into it can only+produce the first picture, and a route that compares that picture is answering a question about the+catalogue.++Two things are mocked rather than left out. **The layers beneath**: a component that draws over a+base take renders over black without one, and black is not neutral — text legible on it can be+illegible on the picture that replaces it. **Media slots the Source declares as generations**: empty+for the whole stretch between being written and being built. Both use the route's fixed+`make-placeholder` tool — `--video --seconds <s>` for a stretch, a PNG for a slot, `--color` so the+mock stays visible against what the element draws — sized from the Source's `space:Canvas` rather+than from the reference video, and the comparison is scoped with `--question` so the observer skips+those regions. `reconstruction-loop.md` gives the sizing rule and says why a reported mock is not a+repair target.++The render can be a clip as well as a still, and a clip is the default. `compare_reconstruction+--video` compares the whole shot, which removes the problem of choosing a frame to represent a shot+that holds several states. A still is for the one case `reconstruction-loop.md` names: the shot's own+visual observation says in words that the element is completely still. That judgement comes from the+reference's observation, never from watching your own render.++This is what to reach for whenever one element has to be looked at rather than the whole program.+Rendering the delivery to inspect a single piece is the waste it exists to prevent.## Open the whole Run for a person@@ -119,7 +131,7 @@ Studio keeps three stages of an item's time apart, and reading one for another mA projection line connects a source to its realized window, and the projection view is read-only.`docs/guide/studio-temporal-windows.md` is authoritative for what each stage carries.-Studio is a browser preview **for a person to look at**. It is not a source of images for an-automated comparison — that is what the local still render above is for.+Studio is a browser preview **for a person to look at**. It is not a source of pictures for an+automated comparison — that is what the local render above is for.`docs/quickstart/preview.md` is authoritative.diff --git a/.agents/skills/hypit/references/reconstruction/index.md b/.agents/skills/hypit/references/reconstruction/index.mdindex 83f3989e..0d43823f 100644--- a/.agents/skills/hypit/references/reconstruction/index.md+++ b/.agents/skills/hypit/references/reconstruction/index.md@@ -52,7 +52,7 @@ it is anyway, all of it, and then read what you wrote and change it to what theyThe reason is that the reconstruction is checkable only while it claims to reproduce the video thatwas observed. The observations describe the reference and are cached, and `compare_reconstruction`-judges a rendered element against a reference frame. Author the change into the sources early and the+judges a rendered element against the reference. Author the change into the sources early and theevidence describes one video while the sources describe another, which leaves the reconstruction withnothing to be checked against while it is still being written.@@ -169,17 +169,18 @@ Before acting on the evidence, read these files completely in order:6. `vocabulary.md` — what the reference evidence adds to that: enumerating from the observations, andmeasuring an appearance value rather than choosing it.7. `../preview.md` — proving the Run traces before a Provider is reached, and rendering one element- to a still without paying for it. Both are used later, by `final-sources.md` and+ without paying for it. Both are used later, by `final-sources.md` and`reconstruction-loop.md`; read them here so neither arrives as a surprise.Two more are required, at the point where they apply rather than now. Reading them here meansreading them half an hour before they matter, with a dozen other files in between:- `final-sources.md` when the components exist and the sources are about to be written.-- `reconstruction-loop.md` **the first time any element is rendered to a still** — including the- preview a new package owes its Surface. No Build is submitted on this route, so a trigger worded- around one never fires: the picture that gets compared is a local render, and the moment one- exists the loop applies.+- `reconstruction-loop.md` **as soon as the sources place an element** — that is the first moment the+ loop has something to render, since what it compares is the element configured the way this video+ configures it. No Build is submitted on this route, so a trigger worded around one never fires; a+ trigger worded around a package's catalogue preview fires too early, before any Source exists to+ read values from.This route ends on a command rather than on a judgement that the work looks done. Every element aproject-local package draws must have been compared against the reference at least once, and that is@@ -194,6 +195,12 @@ non-zero until none are left. It asks for participation rather than convergenceis deliberately bounded and may stop with visible differences remaining — so an element compared onceand stopped at its ceiling passes, and an element nobody looked at does not.+The same command settles one thing about the picture that a Build would otherwise be the first to+show: it reads each timed picture's appearance Recipe and refuses a `playback` left at its default,+which draws generated material once and then draws nothing for the rest of the window.+`../playbooks/craft/generated-dependencies.md` says what to set and why the material's length cannot+be relied on.+The script uses a prepared reference, and how it picks which one is worth knowing before it is run:- exactly one reference is prepared → it uses that one;@@ -203,8 +210,9 @@ The script uses a prepared reference, and how it picks which one is worth knowinA comparison run without `--element` is not credited to any element, and a misspelled `--element`credits nothing either — the script reports logged element names that do not exist in the Source. Aproject that places no `@hypit/local-*` element passes with nothing to require: the Tracks that-installed vocabulary draws carry authored values this gate does not demand (none renders before a-Build), and they are listed as deferred rather than required, to be checked at the delivery gates.+installed vocabulary draws carry authored values this gate does not demand — timed against the real+SemanticTrack none of them renders before a Build — and they are listed as deferred rather than+required, to be checked at the delivery gates.Paths above are relative to this file's directory. Do not skip a file because the task looks like afamiliar video format.diff --git a/.agents/skills/hypit/references/reconstruction/observers.md b/.agents/skills/hypit/references/reconstruction/observers.mdindex 19c42891..b06abcca 100644--- a/.agents/skills/hypit/references/reconstruction/observers.md+++ b/.agents/skills/hypit/references/reconstruction/observers.md@@ -147,6 +147,17 @@ has for free:Answer the tasks in order yourself when the harness has no subagents. It is slower, and it is thefallback rather than the shape to aim for.+The comparison loop arrives here in batches. One element is compared against every shot the reference+shows it in before any repair, so a single element produces several comparisons at once, and each is+self-contained in the same way an observation task is. Dispatch one subagent per comparison and run+them together. Where there are no subagents, answer them one after another rather than folding them+into a single look: what the round is for is the set of differences across shots, and a shot answered+in the light of the previous shot's answer stops being independent evidence of anything.++A clip comparison reaches this observer as two frame tiles — the reference shot's own, and one built+from the render against that shot's duration. Read them as a pair of grids sampling the same stretch+at the same rate.+## Comparing without subagentsAnswering the comparison yourself makes the report yours, and you know what you built, so thediff --git a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.mdindex 4fc481e0..f2908348 100644--- a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md+++ b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md@@ -17,9 +17,9 @@ receives surfaces there rather than here.The difference between the two gates is what each reports. `preview-check` names the graph failure —the target that does not trace, the chain that is missing — so its repairs are not guessing. Thisloop names the appearance difference: `compare_reconstruction` sends the rendered element and the-reference frame to Gemini and returns what differs in words, and a repair aims at a difference the-comparison named. Neither gate expects the agent to guess; each one tells you what is wrong, and you-repair that.+matching stretch of the reference to an observer and returns what differs in words, and a repair aims+at a difference the comparison named. Neither gate expects the agent to guess; each one tells you what+is wrong, and you repair that.## The reference side is finished@@ -27,35 +27,94 @@ The reference video does not change, and its observations are cached. Re-observitemperature of `1.0` buys paraphrase drift and another bill, so never re-run a completed observationto "check" it. `--reobserve` exists for a shot whose media was rebuilt, not for doubt.-Cached does not mean correct. The authority on the reference side is the frame, not the prose written-about it — `workflow.md` says when to go and look, and doubt is settled there, for free, rather than-by paying for a second description.+Cached does not mean correct. The authority on the reference side is the picture, not the prose+written about it. `workflow.md` says when an observation is worth doubting and how to settle it: a+narrow `observe_reference --question` over the shot, put to the reference's own observer. That is+cheaper than re-observing the whole shot, and on the `gemini` observer it is how the picture gets+looked at, since opening it yourself is what the blindness rule below forbids.Everything in this file happens on the reconstruction side, which changes every time a package,Recipe or source edge is edited.-## The loop unit is one element+## The loop unit is one element, across every shot that shows itLoop over one reconstructed element at a time — the new component, one caption system, one inserted-card:--1. Render that element to a still image locally, without a paid Provider. A new package's visual- Surface needs a preview image anyway, as `../local-author-package.md` requires. The repository's- own visual test `packages/hyperframes/test/browser-visual.test.ts` shows the local path end to- end: build the Track, compile the HyperFrames document, and render it through the local- HyperFrames Runtime. The still must show the element in the state the shot below shows, which is- what `local-author-package.md`'s "the render harness takes its state as arguments" exists for:- the catalogue preview is one invocation of it, not the whole of it.-2. Compare it against the reference frame that shows it most clearly:- `.hypit/reference-video-tools/<reference-id>/shots/NNN-representative.jpg`. The shot is a choice,- and the wrong one is invisible to the comparison gate, which only records that a comparison- happened. Pick the shot where the element is the clearest thing on screen — a caption system- against a frame that shows the caption, a card against a frame that shows the card — and pass- `--element <id>` so the gate credits it to that element. When a shot shows the element in- different states across several frames, pick the one that matches the state you rendered; when- the reference's own observation of that shot describes the element, use that description to- confirm the choice.-3. Repair, then render again.+card.++### Render what the Source configures, not what the catalogue shows++The thing compared is the element **as this video places it**: the Recipe values the Source actually+passes, the Script text the Source actually feeds it, and the windows the Source actually binds. A+package's catalogue preview is a different picture with different inputs — the harness supplies its+own sample copy and its own Recipe so a reader can see the Surface's range — and a Source can fill+every one of those parameters with values that collide while the preview stays perfect. Comparing the+preview answers a question nobody asked. `../preview.md` renders both, from the same harness, with+different arguments; this loop uses the Source-configured one.++The render is local and takes no paid Provider. `packages/hyperframes/test/browser-visual.test.ts`+shows the path end to end: build the Track, compile the HyperFrames document, render it through the+local HyperFrames Runtime.++### Mock the layers a Build has not made++The element rarely draws the whole frame. Below it is a base take that does not exist yet, and inside+it may be media slots the Source declares as generations. Render them as mocks rather than leaving+them out: a missing base is not a neutral background, it is black, and text that is legible on black+can be illegible on the picture that will replace it.++Use the route's fixed placeholder tool for both — `hypit-reference-video-tools make-placeholder --out+base.mp4 --width <w> --height <h> --video --seconds <s>` for the base stretch, `--out slot.png` for an+image slot — with `--color` chosen so the mock is visible against what the element draws. Never+`hypit image`, which pays for a generation the video will not reuse, and never a script written by+hand.++**The sizes come from the Source's own `space:Canvas`, not from the reference video.** The mock is+composed inside the render, and the render's geometry is the Canvas. A base mock is the Canvas+exactly — `<space:Canvas width="1080" height="1920"/>` means `--width 1080 --height 1920` — and a slot+mock is that slot's `space:Frame` fractions multiplied by the Canvas, so a Frame from 16% to 84%+across and 30% to 78% down is `--width 734 --height 922`. A mock built to the wrong box is resized by+the `fit` before the observer ever sees it, which moves edges the comparison is there to read.++The reference's own pixel dimensions are a different number and are usually smaller — the prepared+reference stores them in `state.json` under `video`. They do not size the mock, but their **aspect**+has to match the Canvas's. When it does not, every comparison in this loop is putting two+differently-shaped pictures side by side, and the observer will report proportion differences that+belong to the Canvas rather than to the element. Fix the Canvas before comparing anything.++Then tell the observer, through `--question`, that those regions are placeholders standing in for+declared-but-unbuilt generations, and to compare only what the element itself draws. Said plainly it+costs one clause and saves the mock being reported as a difference every round.++A mock lives only in this render. It is never written into the Source, and no gate reads it: the+`playback` check in `reconstruction-check` reads the Source's Recipes, so a mock cannot be mistaken+for coverage.++### Compare the whole shot, against every shot that shows the element++Compare clips, not chosen frames:++```+hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> \+ --video <rendered>.mp4 --element <id>+```++An element that animates in, leaves, and is replaced by another within one static board does not+produce a cut, so a shot can hold several states and no single frame represents it. Choosing one is+the failure this replaces: a caption system compared against the shot with the shortest line looks+correct, because one line has nothing to collide with.++Compare it against **every shot the reference shows the element in**, not the clearest one. That is+the coverage round, and it comes before any repair — collect the whole difference set first, then+repair against it. Doing it the other way makes the comparison after a repair look like a third+attempt at a shot that was never examined.++`--image` against `NNN-representative.jpg` is available for one case only: the shot's own `visual:`+observation states in words that the element is completely still. The judgement comes from the+reference's observation, never from looking at your own render and concluding it does not move —+deciding that from the reconstruction is how the wrong frame got chosen in the first place. Where the+observation does not say still, compare the clip.++Pass `--element <id>` every time so the gate credits the comparison to that element.### One observer reads the reference, and it is the one the reference was prepared with@@ -63,37 +122,45 @@ card:The reference has one observer for its whole life, and the comparison goes through the same one thatwrote its observations.-**On the `gemini` observer, that observer is not you.** The command sends the reference frame and the-rendered image as an unlabelled pair and returns the differences in words. Do not open the reference-frame yourself and look at it, whatever visual ability the model running this loop has, and do not-look at the rendered reconstruction either. A second observer is a second opinion paid for with the-very bias it claims to correct: it sees the reconstruction, knows what was built, and confirms what it-expects. This is why a font that is wrong is caught — the comparison names it, and the package is-repaired, rather than a font being chosen because it resembles what you remember.--**On the `agent` observer the command returns the two images and the question instead of an answer,-and who looks at them is up to the harness.** Give them to a subagent when you can: one handed two-unlabelled images and the question knows neither which is the reference nor what was built, which is-the same blindness this section rests on. Answer it yourself only when there are no subagents, and-then the bias above is real, so `observers.md` states the discipline that stands in for blindness —-write the differences down before naming a cause, count only repairs aimed at a difference you wrote-down, and read a quantity off the frame rather than spending an attempt guessing at it.--Hypit Studio is a browser preview for a person to look at. It is not a source of the image this-loop needs: that image is the local still render in `../preview.md`, which draws one element without-a Provider.+**On the `gemini` observer, that observer is not you.** The command sends the reference clip and the+rendered clip as an unlabelled pair and returns the differences in words. Do not open the reference+yourself and look at it, whatever visual ability the model running this loop has, and do not look at+the rendered reconstruction either. A second observer is a second opinion paid for with the very bias+it claims to correct: it sees the reconstruction, knows what was built, and confirms what it expects.+This is why a font that is wrong is caught — the comparison names it, and the package is repaired,+rather than a font being chosen because it resembles what you remember.++**On the `agent` observer the command returns two pictures and the question instead of an answer, and+who looks at them is up to the harness.** That observer reads pictures rather than video, so a clip+comparison arrives as two frame tiles: the reference shot's own tile, and one the command builds from+your render against that shot's duration, so both grids sample at the same rate and the same layout.+What is compared is the same stretch either way.++Give the pair to a subagent when you can: one handed two unlabelled pictures and the question knows+neither which is the reference nor what was built, which is the same blindness this section rests on.+The coverage round means one element produces several comparisons at once, so dispatch one subagent+per comparison and let them run together. Where the harness has no subagents, run them one after+another rather than merging them into a single look, and answer them yourself under the discipline+`observers.md` states in place of blindness — write the differences down before naming a cause, count+only repairs aimed at a difference you wrote down, and read a quantity off the picture rather than+spending an attempt guessing at it.++Hypit Studio is a browser preview for a person to look at. It is not a source of the picture this+loop needs: that picture is the local render in `../preview.md`, which draws one element as the+Source configures it, without a Provider.Repairing one element never re-runs the others. Do not rebuild the whole video to inspect one piece,and do not defer every comparison to a final delivery Build.## Comparison is blind-Use `compare_reconstruction --reference-id <id> --shot-id <id> --image <rendered.png>`.--- Never tell the observer which image is the reconstruction, what was built, which component drew it,- or what you expect to be wrong. An observer told what to confirm will confirm it.-- The only permitted extra input is scope, through `--question`: which region of the picture to look- at, and nothing else.+- Never tell the observer which picture is the reconstruction, what was built, which component drew+ it, or what you expect to be wrong. An observer told what to confirm will confirm it.+- `--question` carries what to look at, never what to conclude. Three things qualify: which region to+ read, which regions to skip because they hold a placeholder standing in for a declared-but-unbuilt+ generation, and one visible quantity to measure. None of them says which picture is the+ reconstruction, what was built, or what you expect to be wrong — and a question that hints at the+ answer, rather than naming what to look at, has stopped being scope.- Results are not cached. Every iteration is a fresh comparison.## Repair in dependency order@@ -103,10 +170,12 @@ When the difference is structural rather than cosmetic, repair in the order giveProducer, Type and Validator agreement, Surface vocabulary and decoding, implementation behaviour,then source usage. Fixing source usage over a broken implementation hides the defect one layer down.-## Converged means the differences are wording+## Converged means the differences are wording, in every shotStop when the returned differences are wording-level — a describer's phrasing rather than a visible-change. Passing `pnpm check` and `hypit check` is not convergence; it is the precondition for+change — across **all** the shots the element was compared in. One shot reading clean while another+still shows lines colliding is an element mid-repair, and the shot that reads clean is usually the+easiest one. Passing `pnpm check` and `hypit check` is not convergence; it is the precondition forstarting the loop.## The loop is bounded, and stopping is the end@@ -117,24 +186,17 @@ return a difference every round for ever. So the loop ends on whichever of these- **Every attempt aims at a difference the comparison named.** Changing something the comparison didnot mention is not an attempt; it is thrashing, and it does not earn one of the attempts below.-- **A difference caused by an input that does not exist yet is not a repair target.** This route- declares its generation rather than performing it, so a component's media slots are empty for the- whole stretch between being written and being built. The comparison sees that and names it every- round — an empty card where the reference shows a screenshot. Do not aim an attempt at it, and do- not read its reappearance as the no-progress rule below firing: that rule is about the difference- the repair was aimed at, not about every line the comparison returns.-- Mock the slot instead of leaving it empty or generating a real picture. Use the route's fixed- placeholder tool — `hypit-reference-video-tools make-placeholder --out slot.png --width <w>- --height <h>` for an image slot, and `--video [--seconds <s>]` for one that only accepts video —- never `hypit image` (that pays for a real generation the video will not reuse) and never a script- written by hand. Then scope the comparison with `--question` so the observer bypasses the slot:- name the region as an intentional placeholder for a declared-but-unbuilt generation, and compare- only what the component draws itself. The empty slot then stops being reported every round, and- the loop spends its attempts on the differences that can actually be repaired.-- **No progress ends it immediately.** If a comparison returns the same difference it returned before- the repair, stop. The repair is not reaching the problem, and two more rounds of the same reasoning- will not find it. Rendering and comparing cost real time on every round.+- **A difference caused by an input that does not exist yet is not a repair target.** The mocks above+ stand in for a base take and for media slots a Build has not filled, and an observer that was told+ where they are should bypass them. One that reports a mock anyway is describing an absence, not a+ defect. Do not aim an attempt at it, and do not read its reappearance as the no-progress rule below+ firing: that rule is about the difference the repair was aimed at, not about every line the+ comparison returns.+- **No progress ends it immediately.** If the comparisons return the same difference they returned+ before the repair, stop. The repair is not reaching the problem, and two more rounds of the same+ reasoning will not find it. Judge this over the same set of shots before and after — a repair that+ clears three shots and leaves one is progress, and re-comparing a different shot than last round+ measures nothing. Rendering and comparing cost real time on every round.- **There are two loops, each with its own two-attempt ceiling.** The first loop is over the*package*: render what it draws and compare it; a difference that the package cannot express is apackage defect, and fixing it means changing the package's structure. The second loop is over the@@ -154,13 +216,21 @@ return a difference every round for ever. So the loop ends on whichever of theseBeyond each ceiling, guesses start to overshoot — correcting past the reference rather thantowards it. Time beats fidelity here by explicit choice.+ **Each ceiling is two attempts for the element, spread over all its shots.** Comparing an element+ in five shots does not buy ten attempts; it buys a fuller picture of what one attempt has to fix.+ Coverage and attempts measure different things — how much was looked at, and how many times the+ element was changed — and one repair aimed at a difference several shots agree on is one attempt+ however many shots reported it.+## Measure what can be measured; iterate only on what cannotTwo attempts per loop is not enough to converge by guessing, and it is not meant to be. A differencestated as a quantity — a stroke that is too thick, a shape that is too tall, type that is too large,a margin that is too wide — is not a guessing problem. Ask for the number: a-`compare_reconstruction --question "how tall is the oval relative to the frame?"` over the shot that-shows it most clearly returns a measurement in one step, from whichever observer the reference holds.+`compare_reconstruction --question "how tall is the oval relative to the frame?"` returns a+measurement in one step, from whichever observer the reference holds. Ask it over whichever shot+renders the quantity most legibly — this is one reading of one number, not the coverage round, and+reading it off a shot where the thing is small buys a worse number for the same price.Do that instead of spending an attempt. An attempt is for differences that have no number — atypeface's character, a texture, a rhythm — where the only route is change it and look again.@@ -192,13 +262,18 @@ waits for in `../playbooks/craft/production-gates.md`:Gate 1 and Gate 2;- **the authored values on installed vocabulary** — a caption Style's size, colour and placement, aMedia Item's frame, a Typography Track's copy, at Gate 3 against the real SemanticTrack;-- **whether the picture is continuous** — the black stretches and the frames nobody authored that- `frame-coverage.md` is about, at Gate 4, which measures them on the delivery;+- **whether the picture is continuous** — the frames nobody authored that `frame-coverage.md` is+ about, at Gate 4, which measures them on the delivery. One inherited edge is settled before then:+ `reconstruction-check` reads each Recipe's `playback` and refuses a timed picture configured to+ stop when its material runs out, which is the edge a generated take produces every time. The rest+ of them — a window that starts late, a blend whose frames are half-covered, a schedule inside a+ component that stops between activations — are measured on the delivery;- **whether the delivery says the Script's words** — measured at Gate 4 by transcribing the finishedvideo, since a take generated from a prompt that lost its dialogue passes every gate before it.`reconstruction-check` is the mechanical half of this: it names the elements that were never compared-and refuses to pass while any remain. It cannot name what a Build has not produced yet, which is why+and the timed pictures configured to empty their windows, and refuses to pass while either remains.+It cannot name what a Build has not produced yet, which is whythis list is stated rather than computed.If the author wants the result to differ from the reference — their presenter, their product, theirdiff --git a/.agents/skills/hypit/references/reconstruction/workflow.md b/.agents/skills/hypit/references/reconstruction/workflow.mdindex 0ad8272e..1fe8d95a 100644--- a/.agents/skills/hypit/references/reconstruction/workflow.md+++ b/.agents/skills/hypit/references/reconstruction/workflow.md@@ -1,13 +1,13 @@# Reference-video workflow-`@hypit/reference-video-tools` ships six CLI subcommands. Four of them are the reference evidence+`@hypit/reference-video-tools` ships seven CLI subcommands. Four of them are the reference evidenceand belong to this route:```bashhypit-reference-video-tools prepare_reference --video-path <path> --observer gemini|agenthypit-reference-video-tools observe_reference --reference-id <reference-id>hypit-reference-video-tools record_observation --reference-id <reference-id> --key <key> --text-file <path>-hypit-reference-video-tools compare_reconstruction --reference-id <reference-id> --shot-id <shot-id> --image <path> [--element <id>]+hypit-reference-video-tools compare_reconstruction --reference-id <reference-id> --shot-id <shot-id> --video <path>|--image <path> [--question <scope>] [--element <id>]````--observer` is answered once, before anything runs, and `observers.md` owns that question.@@ -15,8 +15,10 @@ hypit-reference-video-tools compare_reconstruction --reference-id <reference-id>writes its own and the command is unused. A long answer arrives whole with `--text-file <path>`;`--text <text>` suits a short one.-The other two — `list_svml_packages` and `inspect_svml_vocabulary` — read installed vocabulary and-have nothing to do with a reference video. This route runs both, at the step the sequence names;+`make-placeholder` writes the mocks the comparison render needs for a base take or a media slot a+Build has not filled; `reconstruction-loop.md` says when. The remaining two — `list_svml_packages`+and `inspect_svml_vocabulary` — read installed vocabulary and have nothing to do with a reference+video. This route runs both, at the step the sequence names;`../vocabulary.md` documents them and decides how a package is chosen. `list_svml_packages` reads`node_modules/@hypit` relative to the working directory and refuses when it is empty, so run it fromthe repository root.@@ -39,9 +41,13 @@ list_svml_packages→ run preview-check (final-sources.md) and repair until the graph traces —this has no attempt ceiling; a target Studio cannot trace is not done.Waiting on unrun Providers is a pass, not a failure-→ read reconstruction-loop.md, then render each authored element and compare it+→ read reconstruction-loop.md, then for each authored element: render it as the+ sources configure it, mocking the layers a Build has not made, and compare it+ against every shot the reference shows it in before repairing anything+→ repair against the differences that round returned, within the ceilings→ run reconstruction-check (index.md) and keep going until it passes; it names- every locally-drawn element that has never been compared. When more than one+ every locally-drawn element that has never been compared, and every timed+ picture whose Recipe leaves playback at its default. When more than onereference is prepared it needs --reference-id <id>```@@ -143,9 +149,11 @@ correction.Be most suspicious of anything an observation asserts about change over time — something appearing,vanishing, being removed, being drawn in a single frame. A describer working from one pass infers-those rather than seeing them, and infers them wrongly. A `compare_reconstruction` against a rendered-probe, or a narrow question over the stretch, is how such a claim is checked, rather than by-believing the single pass that asserted it.+those rather than seeing them, and infers them wrongly. A narrow question over the stretch is how+such a claim is checked, rather than by believing the single pass that asserted it. A+`compare_reconstruction` against a quick rendered probe works too, and is a probe rather than the+element comparison `reconstruction-loop.md` runs — that one is rendered from the Source's own values+and is credited with `--element`.## Narrow questions@@ -162,12 +170,19 @@ Ask about visible attributes. Never ask which component to use.## compare_reconstruction-`compare_reconstruction` sends the shot's reference frame and a rendered image as an unlabelled pair-and returns a description of their visible differences. It is never told which image is which, what-was built, or how; `--question` may narrow it to one region of the picture and nothing else. Results-are not cached. `reconstruction-loop.md` governs when and how to use it.+`compare_reconstruction` sends the reference and the reconstruction as an unlabelled pair and returns+a description of their visible differences. It is never told which is which, what was built, or how;+`--question` carries what to look at — which region to read, which regions hold a placeholder to+skip, or one visible quantity to measure — and never what to conclude. Results are not cached. `reconstruction-loop.md` governs when and how to use it.-`--element` names the reconstructed element the image draws. It never reaches the observer — the+`--video <rendered.mp4>` compares the whole shot and is the default choice, since a shot can hold+several states of an element without a cut and no single frame stands for it. `--image` is for the+one case where the shot's own `visual:` observation says the element is completely still. Which+pictures the pair is made of follows the observer: `gemini` receives the reference clip and the+rendered clip; `agent` receives the reference shot's frame tile and one built from the render against+that shot's duration, so both grids sample alike.++`--element` names the reconstructed element the render draws. It never reaches the observer — thecomparison stays as blind as it is without it — and is written to the reference's `comparisons.jsonl`so `reconstruction-check` can tell an element that was looked at from one that never was. Acomparison run without it is not credited to any element.diff --git a/.agents/skills/hypit/scripts/reconstruction-check.mjs b/.agents/skills/hypit/scripts/reconstruction-check.mjsindex 82d719bd..b5aeca81 100644--- a/.agents/skills/hypit/scripts/reconstruction-check.mjs+++ b/.agents/skills/hypit/scripts/reconstruction-check.mjs@@ -1,6 +1,8 @@#!/usr/bin/env node/**- * Report which reconstructed elements have never been compared against the reference.+ * Report what the route can still settle after the Source is written and before a Build runs: which+ * reconstructed elements have never been compared against the reference, and which timed pictures are+ * configured to stop before their window ends.** `preview-check` proves the graph is wired. It proves nothing about whether what the graph draws* resembles the reference video, and a Source can pass every structural check while a@@ -98,7 +100,77 @@ for (const { alias, specifier, local } of imports) {// Build without a fixture SemanticTrack, so it is reported rather than demanded.const elements = drawn.filter((item) => item.local);const deferred = drawn.filter((item) => !item.local);-if (elements.length === 0) {++// ---- Timed pictures that stop before their window ends ----+//+// A generated take is asked for a whole number of seconds, and the window it has to fill comes from+// speech the same Build synthesizes. The two lengths are arrived at separately and do not meet: the+// estimate rounds, and the voice that finally speaks is not the voice the estimate predicted. So the+// material is routinely shorter than the window it was made for, by anything from a rounding+// remainder to a couple of seconds.+//+// What happens then is decided by one Recipe key. `playback` defaults to `once-start`, which draws+// the material once and then draws nothing at all — `sampling.ts` emits no segment for the remainder,+// so those frames fall through to whatever is beneath. Under a full-frame base that is the Film+// background, which is black. `hold-start` pins the last frame for the rest of the window instead,+// `loop-start` repeats, and `stretch` retimes to fit; any of the three fills it.+//+// This is `frame-coverage.md`'s inherited edge — "a generated take, which ends where its material+// ends rather than where the shot should" — in the one form the route can settle before a Build, from+// the Source and its Recipe sheets alone.+async function playbackReport() {+ // Every Recipe body the Source can name, by the alias its sheet was imported under.+ const sheets = new Map();+ for (const [, alias, source] of svml.matchAll(/<import\s+as="([^"]+)"\s+source="([^"]+\.svs)"/gu)) {+ const text = await readFile(resolve(dirname(svmlPath), source), "utf8").catch(() => undefined);+ if (text === undefined) continue;+ const recipes = new Map();+ for (const [, name, body] of text.matchAll(/([A-Za-z0-9_.-]+)\s*\{([^}]*)\}/gu)) recipes.set(name, body);+ sheets.set(alias, recipes);+ }++ // Which Normalize ids carry a picture. A take normalized with `video="none"` is a voice and has no+ // window to fill.+ const moving = new Set();+ for (const [, attributes] of svml.matchAll(/<pipeline:Normalize\b([^>]*?)\/?>/gsu)) {+ const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];+ const video = /\bvideo="([^"]+)"/u.exec(attributes)?.[1];+ if (id !== undefined && video !== undefined && video !== "none") moving.add(id);+ }++ // Which aliases belong to a package that draws. Plenty of elements consume a normalized media+ // without placing it — `whisperx:SemanticTake` reads one to align speech against it and puts no+ // picture on the Canvas, so it has no window to fill and no occupancy to choose. Requiring the+ // alias to come from a package with a Track that outputs a VisualTrack keeps those out, and keeps+ // the Items and Sequences written inside such a Track in, without this needing to know their tags.+ const draws = new Set(imports+ .filter(({ specifier }) => [...drawingTags.keys()].some((key) => key.startsWith(`${specifier}#`)))+ .map(({ alias }) => alias));++ // Every element that places one of those pictures, and what its Recipe says to do with a window the+ // material does not fill.+ const running = [];+ for (const [, tag, attributes] of svml.matchAll(/<([a-z][a-z0-9-]*:[A-Za-z][A-Za-z0-9]*)\b([^>]*?)\/?>/gsu)) {+ if (!draws.has(tag.slice(0, tag.indexOf(":")))) continue;+ const media = /\bmedia=\{([A-Za-z0-9_-]+)\.media\}/u.exec(attributes)?.[1];+ if (media === undefined || !moving.has(media)) continue;+ const appearance = /\bappearance=\{([A-Za-z0-9_-]+)\.([A-Za-z0-9_.-]+)\}/u.exec(attributes);+ const body = appearance === null ? undefined : sheets.get(appearance[1])?.get(appearance[2]);+ const playback = body === undefined ? undefined : /\bplayback\s*:\s*([A-Za-z-]+)/u.exec(body)?.[1];+ if (playback !== undefined && playback !== "once-start" && playback !== "once-end") continue;+ running.push({+ id: /\bid="([^"]+)"/u.exec(attributes)?.[1] ?? tag,+ tag,+ recipe: appearance === null ? "no appearance Recipe" : `${appearance[1]}.${appearance[2]}`,+ playback: playback ?? "once-start, by default",+ });+ }+ return running;+}++const running = await playbackReport();++if (elements.length === 0 && running.length === 0) {console.log("reconstruction-check: no locally-drawn element is placed in the Source; nothing to require.");if (deferred.length > 0) reportDeferred();process.exit(0);@@ -177,21 +249,43 @@ if (unlabelled > 0) {}if (deferred.length > 0) reportDeferred();-if (never.length === 0) {+if (running.length > 0) {console.log("");- console.log(`reconstruction-check: every locally-drawn element has been compared (${elements.length}).`);+ console.log(` ${running.length} timed picture${running.length === 1 ? " draws" : "s draw"} nothing once the material ends:`);+ for (const item of running) console.log(` ${item.tag} id=${item.id} — ${item.recipe}, playback ${item.playback}`);+}++if (never.length === 0 && running.length === 0) {+ console.log("");+ console.log(`reconstruction-check: every locally-drawn element has been compared (${elements.length}), and every timed picture fills its window.`);process.exit(0);}console.log("");-console.log(`reconstruction-check: ${never.length} of ${elements.length} elements have never been compared.`);-console.log("");-console.log("Read .claude/skills/hypit/references/reconstruction/reconstruction-loop.md, then for each:");-console.log("render the element to a still in the state the reference shot shows, and compare it blind.");-console.log("");-for (const element of never) {- console.log(` hypit-reference-video-tools compare_reconstruction --reference-id ${reference} \\`);- console.log(` --shot-id <the shot that shows ${element.id} most clearly> \\`);- console.log(` --image <rendered still>.png --element ${element.id}`);+if (never.length > 0) {+ console.log(`reconstruction-check: ${never.length} of ${elements.length} elements have never been compared.`);+ console.log("");+ console.log("Read .claude/skills/hypit/references/reconstruction/reconstruction-loop.md, then for each:");+ console.log("render the element as the Source configures it, mock the layers a Build has not made,");+ console.log("and compare the whole shot blind — one comparison per shot the reference shows it in.");+ console.log("");+ for (const element of never) {+ console.log(` hypit-reference-video-tools compare_reconstruction --reference-id ${reference} \\`);+ console.log(` --shot-id <each shot that shows ${element.id}> \\`);+ console.log(` --video <rendered clip>.mp4 --element ${element.id}`);+ }+}+if (running.length > 0) {+ if (never.length > 0) console.log("");+ console.log(`reconstruction-check: ${running.length} window${running.length === 1 ? "" : "s"} will empty before ${running.length === 1 ? "it ends" : "they end"}.`);+ console.log("");+ console.log("A generated take is ordered in whole seconds and its window is measured from speech the");+ console.log("Build has yet to synthesize, so the material is shorter than the window it fills more");+ console.log("often than not. Under a full-frame picture the remainder is the Film background, which");+ console.log("reads as a black gap; under a cutaway it is the layer beneath blinking through.");+ console.log("");+ console.log("Give each Recipe above a playback: hold-start to pin the last frame for the rest of the");+ console.log("window, loop-start to repeat, or stretch to retime. Read playbooks/craft/frame-coverage.md");+ console.log("on inherited edges before choosing — lengthening the material instead leaves the same edge.");}process.exit(1);diff --git a/packages/reference-video-tools/src/cli.ts b/packages/reference-video-tools/src/cli.tsindex dadfce3a..606707ef 100644--- a/packages/reference-video-tools/src/cli.ts+++ b/packages/reference-video-tools/src/cli.ts@@ -14,13 +14,19 @@ function usage(): string {" hypit-reference-video-tools observe_reference --reference-id <id> --shot-id <id> [--shot-id <id> ...] --question <text>"," hypit-reference-video-tools record_observation --reference-id <id> --key <key> --text <text>|--text-file <path>"," hypit-reference-video-tools inspect_svml_vocabulary --package <name> [--package <name> ...] [--tag <tag> ...] [--without-previews]",- " hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> --image <path> [--question <scope>] [--element <id>]",+ " hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> --video <path>|--image <path> [--question <scope>] [--element <id>]"," hypit-reference-video-tools make-placeholder --out <path> --width <w> --height <h> [--color light|mid|dark|white|black|#RRGGBB] [--video] [--seconds <s>]","","--element names the reconstructed element the image draws. It is written to the reference's","comparison log and never sent to the observer, so the comparison stays blind while a later gate can","still tell which elements have been compared.","",+ "--video compares the whole shot instead of one frame of it, which is what removes the problem of",+ "choosing a characteristic frame for an element that animates in, leaves, or is replaced within one",+ "shot. The `gemini` observer receives the two clips; the `agent` observer receives two frame tiles,",+ "the rendered clip tiled against the reference shot's own duration so both grids sample alike. Use",+ "--image only when the shot's own visual observation states the element is completely still.",+ "","make-placeholder writes a correctly-sized placeholder for a media slot the Source declares as a","generation and a Build has not filled. It is deterministic and Provider-free: the comparison loop","uses its output to mock an empty slot, and the observer is told the slot is a placeholder so it is",@@ -137,14 +143,15 @@ async function main(): Promise<void> {};result = await tools.record_observation(input as { reference_id: string; key: string; text: string });} else if (command === "compare_reconstruction") {+ const video = one(flags, "video");const input = supplied ?? {reference_id: required(flags, "reference-id"),shot_id: required(flags, "shot-id"),- image_path: required(flags, "image"),+ ...(video === undefined ? { image_path: required(flags, "image") } : { video_path: video }),...(one(flags, "question") === undefined ? {} : { question: one(flags, "question") }),...(one(flags, "element") === undefined ? {} : { element: one(flags, "element") }),};- result = await tools.compare_reconstruction(input as { reference_id: string; shot_id: string; image_path: string; question?: string; element?: string });+ result = await tools.compare_reconstruction(input as { reference_id: string; shot_id: string; image_path?: string; video_path?: string; question?: string; element?: string });} else if (command === "inspect_svml_vocabulary") {const packages = many(flags, "package");const input = supplied ?? {diff --git a/packages/reference-video-tools/src/media.ts b/packages/reference-video-tools/src/media.tsindex cecca4c8..c213227e 100644--- a/packages/reference-video-tools/src/media.ts+++ b/packages/reference-video-tools/src/media.ts@@ -113,9 +113,9 @@ async function extract(path: string, args: readonly string[], target: string): P// show what moves; nine keeps a fifteen-second one legible at a cell width a reader can still resolve// detail in. The clip is decoded once, and the sampling and the tiling happen in that one pass.const TILE_COLUMNS = 3;-function tileFrames(duration: number): number { return clamp(Math.round(duration * 1.5), 4, 9); }+export function tileFrames(duration: number): number { return clamp(Math.round(duration * 1.5), 4, 9); }-async function shotTile(clip: string, duration: number, target: string): Promise<string> {+export async function shotTile(clip: string, duration: number, target: string): Promise<string> {const frames = tileFrames(duration);const rows = Math.ceil(frames / TILE_COLUMNS);const rate = round(frames / Math.max(duration, 0.1));diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.tsindex e7432faa..c44d57d1 100644--- a/packages/reference-video-tools/src/tools.ts+++ b/packages/reference-video-tools/src/tools.ts@@ -19,6 +19,7 @@ import {readJson,readBytes,referenceId,+ shotTile,writeJson,} from "./media.js";import { prepareTranscript } from "./transcript.js";@@ -48,7 +49,15 @@ export type InspectVocabularyInput = {export type CompareReconstructionInput = {readonly reference_id: string;readonly shot_id: string;- readonly image_path: string;+ /** A rendered still. Exactly one of `image_path` and `video_path` is given. */+ readonly image_path?: string;+ /**+ * A rendered clip covering the shot. Comparing the whole stretch rather than one frame is what+ * removes the frame-choosing problem: an element that animates in, leaves and is replaced within+ * one shot has no single characteristic frame, and an observation asserting when it changed is the+ * least reliable evidence there is.+ */+ readonly video_path?: string;readonly question?: string;/*** Names the reconstructed element this image draws, for the comparison log only. It is never sent@@ -116,6 +125,8 @@ export type ComparisonRecord = {readonly observer: Observer;readonly status: Observation["status"];readonly scoped: boolean;+ /** Whether the whole shot was compared as a clip, or one frame of it as a still. */+ readonly clip?: boolean;};async function fileDigest(path: string): Promise<string> {@@ -321,6 +332,10 @@ function asPictures(media: readonly string[], state: ReferenceState): readonly s// what moves nor what recurs, so the shot tiles come with it and the question sees every frame// the shot observations see.else if (path === state.analysis_video_ref) pictures.push(state.storyboard_ref, ...state.shots.flatMap((shot) => shot.frames_tile_ref === null ? [] : [shot.frames_tile_ref]));+ // This observer reads pictures. Video that is not a prepared shot has no tile to stand in for it,+ // and handing the file through would give the agent something it cannot open — a comparison clip+ // is tiled by its caller before it arrives here.+ else if (/\.(mp4|mov|webm|mkv)$/iu.test(path)) assert(false, `${path} is video; tile it before asking the agent observer to read it`);else if (!path.toLowerCase().endsWith(".wav")) pictures.push(path);}return [...new Set(pictures)];@@ -816,16 +831,42 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst state = await loadState(input.reference_id);const shot = state.shots.find((candidate) => candidate.shot_id === input.shot_id);assert(shot !== undefined, `shot ${input.shot_id} does not exist in reference ${input.reference_id}`);- const imagePath = resolve(input.image_path);- const file = await stat(imagePath).catch(() => undefined);- assert(file?.isFile(), `image_path is not a file: ${imagePath}`);+ const supplied = [input.image_path, input.video_path].filter((value) => value !== undefined);+ assert(supplied.length === 1, "supply exactly one of image_path and video_path");+ const clip = input.video_path !== undefined;+ const renderedPath = resolve(String(supplied[0]));+ const file = await stat(renderedPath).catch(() => undefined);+ assert(file?.isFile(), `${clip ? "video_path" : "image_path"} is not a file: ${renderedPath}`);const scope = input.question?.trim() ?? "";const observer: Observer = state.observer ?? "gemini";const { ask, pending } = await askerFor(observer, state);++ // A clip comparison reads the whole stretch on both sides. The observer that reads video is+ // given the two clips; the observer that reads pictures is given two frame tiles, which is the+ // same degradation `prepare_reference` already applies to a shot. Tiling the rendered clip+ // against the reference shot's own duration makes `tileFrames` choose the same frame count and+ // layout for both, so the two grids are read side by side rather than as different samplings.+ let referenceMedia = clip ? shot.clip_ref : shot.representative_frame_ref;+ let renderedMedia = renderedPath;+ if (clip && observer === "agent") {+ assert(shot.frames_tile_ref !== null,+ `shot ${shot.shot_id} has no frame tile; re-prepare the reference with --redo all --observer agent`);+ const tile = join(stateRoot(workspaceRoot, state.reference_id), "shots", `${shot.shot_id}-reconstruction.jpg`);+ await shotTile(renderedPath, shot.duration_seconds, tile);+ referenceMedia = shot.frames_tile_ref;+ renderedMedia = tile;+ }++ const unit = clip && observer !== "agent" ? "video clips" : "still images";+ // TILE_PREAMBLE already tells the agent observer how to read a grid, so this adds only what is+ // new about a pair of them: both are stretches, and they are compared as sequences.+ const reading = clip && observer === "agent"+ ? "\n\nBoth are grids, so compare them as sequences rather than as single moments."+ : "";const differences = await ask("comparison", {- media: [shot.representative_frame_ref, imagePath],- instruction: "You compare two supplied still images and describe their visible differences in natural language only. You are not told how either image was made. Do not write code, markup, SVML, component names, or production advice.",- prompt: `Two still images are supplied in order: image one, then image two.${scope.length === 0 ? "" : `\n\nLimit the comparison to this part of the picture: ${scope}`}\n\nDescribe every visible difference between them: layout and arrangement, the position and size of each element, cropping and margins, colour, typeface, weight, letter and line spacing, alignment, outline or stroke, shadow, glow, borders and corner treatment, and anything present in one image and absent from the other. State plainly which differences are large enough to read as a different design and which are minor. If they are visually equivalent, say exactly that.\n\nDo not speculate about how either image was produced, which one is a source, or which one is a copy. Return natural language only.`,+ media: [referenceMedia, renderedMedia],+ instruction: `You compare two supplied ${unit} and describe their visible differences in natural language only. You are not told how either was made. Do not write code, markup, SVML, component names, or production advice.`,+ prompt: `Two ${unit} are supplied in order: one, then two.${reading}${scope.length === 0 ? "" : `\n\nLimit the comparison to this: ${scope}`}\n\nDescribe every visible difference between them: layout and arrangement, the position and size of each element, cropping and margins, colour, typeface, weight, letter and line spacing, alignment, outline or stroke, shadow, glow, borders and corner treatment, and anything present in one and absent from the other.${clip ? " Also describe differences in what changes over the stretch: what appears, what leaves, in what order, and how anything moves." : ""} State plainly which differences are large enough to read as a different design and which are minor. If they are visually equivalent, say exactly that.\n\nDo not speculate about how either was produced, which one is a source, or which one is a copy. Return natural language only.`,});// The answer is deliberately not cached — every iteration is a fresh comparison. What is// recorded is that a comparison happened, so a gate can tell an element that was looked at@@ -835,18 +876,20 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVat: new Date().toISOString(),shot_id: shot.shot_id,...(input.element === undefined ? {} : { element: input.element.trim() }),- image_path: imagePath,- image_digest: await fileDigest(imagePath),+ image_path: renderedPath,+ image_digest: await fileDigest(renderedPath),observer,status: differences.status,scoped: scope.length > 0,+ clip,});return {reference_id: state.reference_id,observer,shot_id: shot.shot_id,- reference_frame_ref: shot.representative_frame_ref,- image_path: imagePath,+ compared: clip ? "clip" : "still",+ reference_ref: referenceMedia,+ rendered_ref: renderedMedia,differences,unresolved: differences.status === "complete" ? [] : ["differences"],pending_observations: pending,diff --git a/packages/reference-video-tools/test/tools.test.ts b/packages/reference-video-tools/test/tools.test.tsindex 2e0fc940..a120adbd 100644--- a/packages/reference-video-tools/test/tools.test.ts+++ b/packages/reference-video-tools/test/tools.test.ts@@ -178,11 +178,11 @@ test("reconstruction comparison sends an unlabelled pair and accepts rendered PNassert.equal(calls.length, 1);assert.deepEqual(calls[0]!.mimeTypes, ["image/jpeg", "image/png"]);- assert.match(calls[0]!.text, /image one, then image two/u);+ assert.match(calls[0]!.text, /supplied in order: one, then two/u);assert.match(calls[0]!.text, /only the full-screen list area/u);assert.doesNotMatch(calls[0]!.text, /reconstruction|rendered|generated|authored|component|SVML/iu,"the comparison must never say which image was built or how");- assert.match(calls[0]!.instruction, /You are not told how either image was made/u);+ assert.match(calls[0]!.instruction, /You are not told how either was made/u);assert.deepEqual(result["differences"], { status: "complete", text: "the list starts lower in one image" });await assert.rejects(