Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
feat: a reference can be read by the agent instead of Gemini
rpone8b4d343976dd
diff --git a/.agents/skills/hypit/references/credentials.md b/.agents/skills/hypit/references/credentials.mdindex a42d76ec..61c2c496 100644--- a/.agents/skills/hypit/references/credentials.md+++ b/.agents/skills/hypit/references/credentials.md@@ -65,6 +65,10 @@ curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $MIMO_API_KEYhttps://api.xiaomimimo.com/v1/models```+`GOOGLE_CLOUD_PROJECT` and `GOOGLE_APPLICATION_CREDENTIALS_JSON` are also what the reference-video+route's `gemini` observer needs. Without them that route runs its `agent` observer instead, which+reaches no Provider — `reconstruction/observers.md` says how the author chooses between them.+`hypit runtime down` stops the Worker for the whole project, so a Build running in another terminalstops with it. The Build itself is durable and survives; bring the Worker back with `hypit runtimeup`, then reattach with `hypit status <build-id> --watch`. Observing requires a running Worker: withdiff --git a/.agents/skills/hypit/references/reconstruction/index.md b/.agents/skills/hypit/references/reconstruction/index.mdindex 5f3179a4..7afbbc12 100644--- a/.agents/skills/hypit/references/reconstruction/index.md+++ b/.agents/skills/hypit/references/reconstruction/index.md@@ -1,8 +1,10 @@# Reference-video reconstruction routeThis route reconstructs a complete reference video as usable `main.svml`, `recipes.svs`, and-`build.svrun`. The main agent owns all decisions and source code; Gemini supplies natural-language-visual and audio evidence only.+`build.svrun`. The main agent owns all decisions and source code. Observation is a separate job with a+separate output: natural-language evidence about what the reference shows, written before and apart+from any decision about what to build. `observers.md` says who does that job — Gemini, or you reading+pictures — and the boundary holds either way.## What this route delivers, and what it does not@@ -70,8 +72,9 @@ discoverable:- **Where the project goes.** Its own directory in the checkout: `projects/<something>-reverse/`,named after the video — `../runtime.md` says what that directory is and what the workspace doeswith it. Use a directory the author named only if they named one.-- **Credentials.** The repository keeps them in `.env`; load it as shown below. If a variable is- missing, say which one and stop — do not ask the author to describe their setup.+- **Credentials.** The repository keeps them in `.env`; load it as shown below. A variable a Provider+ needs and this machine does not hold is named, and the route stops there — with one exception, the+ Vertex pair, which selects an observer rather than blocking one. `observers.md` owns that.Ask the author about the video and nothing else: what it is for, who is in it, what it should say.Those answers describe the result they want rather than the reconstruction; record them and apply@@ -80,8 +83,18 @@ which directory, or whether to proceed.## Before the first command-`prepare_reference` and every observation need Vertex credentials, and a project that keeps them in a-`.env` file does not load it automatically. Load it into the environment first:+Ask the author which observer reads the reference, and ask it before anything runs. `gemini` uploads+the video to Vertex and hears it; `agent` hands each observation to you as a picture to read and needs+no credentials. Read `observers.md` first: it says what each one costs, how to report what this+machine holds rather than asking the author to recall it, and what the `agent` observer can and cannot+answer. The answer is passed once, as `--observer`, and the reference keeps it.++This is the only question this route asks about how it runs, and it is asked once, before the evidence+starts.++A project that keeps its credentials in a `.env` file does not load it automatically. Load it into the+environment first — the `gemini` observer needs the Vertex pair, and either observer may still need+credentials for the generation the author starts later:```bashset -a && source .env && set +a@@ -97,24 +110,30 @@ Start them first and read while they run. Reading first and observing afterwardsseveral minutes longer for nothing.```bash-hypit-reference-video-tools prepare_reference --video-path <path> # about a minute-hypit-reference-video-tools observe_reference --reference-id <id> # run in the background+hypit-reference-video-tools prepare_reference --video-path <path> --observer <chosen> # about a minute+hypit-reference-video-tools observe_reference --reference-id <id> # run in the background```+On the `agent` observer both commands return immediately with the observations they owe, and reading+the files below is what you do between answering them.+Then read the files below while the sweep is working, and collect its result when you are done.Before acting on the evidence, read these files completely in order:-1. `workflow.md` — CLI sequence, evidence flow and responsibility boundaries.-2. `continuity.md` — mandatory shot, overlay, B-roll, speaker, product and persistent-system rules.-3. **Every craft file listed in the first item of `../playbooks/index.md`'s required load order — the+1. `observers.md` — which observer is reading, what it is handed, and what it cannot answer. It is+ first because the answer to its question was needed before any of this ran, and because every file+ below describes evidence it shapes.+2. `workflow.md` — CLI sequence, evidence flow and responsibility boundaries.+3. `continuity.md` — mandatory shot, overlay, B-roll, speaker, product and persistent-system rules.+4. **Every craft file listed in the first item of `../playbooks/index.md`'s required load order — theones it marks always-read.** That list decides *which* files, and it grows: one added there isrequired here from the moment it is added, whether or not it appears among the notes below.Restating the set in this file is what once left a required craft file reachable from one routeand invisible to the other.Its paths are written from `../playbooks/`, so a file it names as `craft/<name>.md` is- `../playbooks/craft/<name>.md` from here. Only that first item is meant: the same section goes on+ `../playbooks/craft/<name>.md` from here. Only that list's first item is meant: the same section goes onto name Seedance directing, a format file and package READMEs conditionally, and none of thoseconditions can even be evaluated before the evidence is in.@@ -133,12 +152,12 @@ Before acting on the evidence, read these files completely in order:and accepted Records are pinned for reuse.A file that list names and this one does not is read last, before `vocabulary.md`.-4. `../vocabulary.md` — how a package is chosen for any route: enumerate every system before naming+5. `../vocabulary.md` — how a package is chosen for any route: enumerate every system before namingone, what `list_svml_packages` and `inspect_svml_vocabulary` report, reuse before compose beforedeclaring a gap, and what a real gap obliges.-5. `vocabulary.md` — what the reference evidence adds to that: enumerating from the observations, and+6. `vocabulary.md` — what the reference evidence adds to that: enumerating from the observations, andmeasuring an appearance value rather than choosing it.-6. `../preview.md` — proving the Run traces before a Provider is reached, and rendering one element+7. `../preview.md` — proving the Run traces before a Provider is reached, and rendering one elementto a still without paying for it. Both are used later, by `final-sources.md` and`reconstruction-loop.md`; read them here so neither arrives as a surprise.diff --git a/.agents/skills/hypit/references/reconstruction/observers.md b/.agents/skills/hypit/references/reconstruction/observers.mdnew file mode 100644index 00000000..06075735--- /dev/null+++ b/.agents/skills/hypit/references/reconstruction/observers.md@@ -0,0 +1,101 @@+# Who reads the reference++Two observers can read a reference video, and the author picks which one before any command runs. The+choice is made once per reference and recorded in its state: every observation of one reference is+made through one observer, so that evidence gathered two different ways never sits under the same keys+with nothing on the record saying which is which.++Everything after the evidence is the same on both paths. The observation keys are the same, the+prompts are the same, the cache is the same file, and every file this route links reads the result+without knowing which observer produced it.++## The two++**`gemini`** uploads the shot clips and the whole reference to Vertex. It sees motion as motion and+hears the sound. It needs `GOOGLE_CLOUD_PROJECT` and `GOOGLE_APPLICATION_CREDENTIALS_JSON`, and each+observation is a paid request.++**`agent`** hands the observations to you. It needs no credentials and reaches no Provider: the CLI+prepares the same media, then returns each observation as a task carrying its prompt and the pictures+to answer it from. You look, you answer, you record the answer, and the result assembles exactly as+the other path's does.++## Choosing++Ask the author once, before `prepare_reference`, and say what each one costs them. Report what this+machine actually holds rather than asking them to recall it:++```text+node .agents/skills/hypit/scripts/check-credentials.mjs GOOGLE_CLOUD_PROJECT GOOGLE_APPLICATION_CREDENTIALS_JSON+```++Both variables `set` means `gemini` is available and the author chooses. Either one `missing` means+`agent` is the path that can run today, and the author decides whether to run it or to configure+Vertex first — `../credentials.md` says what those two variables are.++This is the one question this route asks about how it runs. Everything else it decides:+`index.md` still owns the working directory, the project location, the vocabulary and the generators.++## What the `agent` observer reads++Each shot arrives as one picture: its frames sampled evenly across its duration and tiled in reading+order, left to right then top to bottom. Read the grid as time passing. The whole reference arrives as+the storyboard of representative frames. A task that carries a single picture carries one moment.++The number of frames follows the shot's length, from four for a short one to nine for a long one, so+a cell stays wide enough to read detail in.++## What sound costs on the `agent` observer++You cannot hear the reference. Three things follow, and each is a condition to work inside rather than+a step to skip:++- `transcript` is unaffected. WhisperX measures every word locally on both paths, so the words and+ their timings are exact, and `final-sources.md` takes Segment durations from `transcript_ref` as it+ always does.+- A task that needs sound says so in its own prompt. Answer it from what the pictures and the+ transcript support — who is visibly speaking, what is on screen when a passage runs — and state+ plainly which parts you could not determine. An observation that says what it could not see is+ complete; one that infers a voice from an appearance is wrong, and+ `continuity.md` depends on that distinction.+- `voices` and the sound half of each `boundary` are the observations this affects. Read what they+ return as what a viewer with the sound off would know.++## The comparison in the loop is yours too++`compare_reconstruction` on the `agent` observer returns the two images and the same question rather+than an answer. You look at the reference frame and your render and describe the differences yourself.++That makes the report yours, and you know what you built, so the discipline that keeps it useful is+explicit:++- **Write the differences down before naming a cause.** List what is visibly different, in the words+ you would use if you had never seen the source. Deciding what went wrong first, then looking, finds+ what you expected.+- **A difference you did not write down is not an attempt.** `reconstruction-loop.md`'s two-attempt+ ceilings apply unchanged, and they count repairs aimed at differences the comparison named.+- **Measure rather than iterate.** A stroke that is too thick, a shape that is too tall, a margin that+ is too wide — read the number off the frame against the frame's own width and height, and change the+ value once. `reconstruction-loop.md` says why an attempt is for differences that have no number.++## Running it++```text+hypit-reference-video-tools prepare_reference --video-path <path> --observer agent+hypit-reference-video-tools observe_reference --reference-id <id>+```++Both return `pending_observations`: a list of tasks, each with its `key`, its `instruction`, its+`prompt` and the `image_refs` to read. Answer one, then record it:++```text+hypit-reference-video-tools record_observation --reference-id <id> --key visual:shot-002 \+ --text-file <path to your answer>+```++`--text-file` is how a long answer arrives whole; `--text` suits a short one. Re-run+`observe_reference` to see what is still owed. An observation you have not answered reports as+`pending`, which `unresolved` lists and which is not a failure.++`--redo all --observer <other>` is how a reference changes observer, and it re-prepares everything,+because the observations it holds were read from the other kind of evidence.diff --git a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.mdindex e9dae240..bb914d11 100644--- a/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md+++ b/.agents/skills/hypit/references/reconstruction/reconstruction-loop.md@@ -48,21 +48,25 @@ card:`.hypit/reference-video-tools/<reference-id>/shots/NNN-representative.jpg`.3. Repair, then render again.-### The reference is seen by Gemini, never by you--The reference frame is looked at exactly one way: through `compare_reconstruction`, which sends it-and the rendered image to Gemini as an unlabelled pair and returns the differences in words. You do-not open the reference frame yourself and look at it. Whatever visual ability you have — the model-running this loop may be able to read images directly — is not to be used on the reference or on the-rendered reconstruction. There is one observer of the reference, the same VLM that wrote the-observations, and it reports through `compare_reconstruction`. A second observer is a second opinion-that is paid for with the very bias it claims to correct: it sees the reconstruction, knows what was-built, and confirms what it expects.--This is why a font that is wrong is caught: `compare_reconstruction` names it, and the package is-repaired. It is not caught by looking at the frame yourself and choosing a font that happens to-resemble it — that route silently depends on the loop model having vision, and stops working the-moment it does not.+### One observer reads the reference, and it is the one the reference was prepared with++`compare_reconstruction` is how the reference frame is looked at, and `observers.md` says who looks.+The reference has one observer for its whole life, and the comparison goes through the same one that+wrote its observations.++**On the `gemini` observer, that observer is not you.** The command sends the reference frame and the+rendered image as an unlabelled pair and returns the differences in words. Do not open the reference+frame yourself and look at it, whatever visual ability the model running this loop has, and do not+look at the rendered reconstruction either. A second observer is a second opinion paid for with the+very bias it claims to correct: it sees the reconstruction, knows what was built, and confirms what it+expects. This is why a font that is wrong is caught — the comparison names it, and the package is+repaired, rather than a font being chosen because it resembles what you remember.++**On the `agent` observer, that observer is you**, and the command returns the two images and the+question instead of an answer. The bias above is then real and unavoidable, so `observers.md` states+the discipline that replaces blindness: write the differences down before naming a cause, count only+repairs aimed at a difference you wrote down, and read a quantity off the frame rather than spending+an attempt guessing at it.Hypit Studio is a browser preview for a person to look at. It is not a source of the image thisloop needs: that image is the local still render in `../preview.md`, which draws one element without@@ -128,10 +132,9 @@ return a difference every round for ever. So the loop ends on whichever of theseTwo attempts per loop is not enough to converge by guessing, and it is not meant to be. A differencestated as a quantity — a stroke that is too thick, a shape that is too tall, type that is too large,-a margin that is too wide — is not a guessing problem. Ask Gemini for the number: a+a margin that is too wide — is not a guessing problem. Ask for the number: a`compare_reconstruction --question "how tall is the oval relative to the frame?"` over the shot that-shows it most clearly returns a measurement in one step. The reference is seen by Gemini, never by-you, and this is how a quantity is measured without ever opening the frame yourself.+shows it most clearly returns a measurement in one step, from whichever observer the reference holds.Do that instead of spending an attempt. An attempt is for differences that have no number — atypeface's character, a texture, a rhythm — where the only route is change it and look again.diff --git a/.agents/skills/hypit/references/reconstruction/workflow.md b/.agents/skills/hypit/references/reconstruction/workflow.mdindex bc911122..e74ee90b 100644--- a/.agents/skills/hypit/references/reconstruction/workflow.md+++ b/.agents/skills/hypit/references/reconstruction/workflow.md@@ -1,14 +1,19 @@# Reference-video workflow-`@hypit/reference-video-tools` ships five CLI subcommands. Three of them are the reference evidence+`@hypit/reference-video-tools` ships six CLI subcommands. Four of them are the reference evidenceand belong to this route:```bash-hypit-reference-video-tools prepare_reference --video-path <path>+hypit-reference-video-tools prepare_reference --video-path <path> --observer gemini|agenthypit-reference-video-tools observe_reference --reference-id <reference-id>+hypit-reference-video-tools record_observation --reference-id <reference-id> --key <key> --text-file <path>hypit-reference-video-tools compare_reconstruction --reference-id <reference-id> --shot-id <shot-id> --image <path>```+`--observer` is answered once, before anything runs, and `observers.md` owns that question.+`record_observation` is how the `agent` observer returns an answer; on the `gemini` observer the tool+writes its own and the command is unused.+The other two — `list_svml_packages` and `inspect_svml_vocabulary` — read installed vocabulary andhave nothing to do with a reference video. This route runs both, at the step the sequence names;`../vocabulary.md` documents them and decides how a package is chosen.@@ -43,8 +48,9 @@ each one, and each position described in enough detail to draw from the words alreconstructing a location depends on, since reference frames are never fed to generation.`transcript` is the verbatim speech of the whole reference with a start and an end for every single-word, measured locally by WhisperX. It is not an observation: no model wrote it, nothing about it is-sent to Gemini, and it does not go in the observation cache. Read it whenever a decision depends on+word, measured locally by WhisperX. It is not an observation: no model wrote it, nothing about it+reaches an observer, and it does not go in the observation cache. It is identical on both observers,+and on the `agent` observer it is the only exact record of the sound. Read it whenever a decision depends onwhen a word is said — placing each on-screen text reveal against the line that triggers it, timing acaption, checking that a voice observation matches what was actually spoken, or setting how long atake runs, which `final-sources.md` measures from these words rather than estimating. Do not run WhisperX@@ -119,17 +125,18 @@ reported letters being *deleted* from the screen where the text was only typingThe frames are already on disk at`.hypit/reference-video-tools/<reference-id>/shots/NNN-representative.jpg`, and the shot clips besidethem. When evidence disagrees — one shot against the next, a shot against a whole-reference pass, or-an observation against what the video plainly is — settle it by asking Gemini to look, through a-narrow `observe_reference --question` over that shot, not by opening the frame yourself. You do not-view the reference; `reconstruction-loop.md` says this is a rule, not a preference. Do not re-observe+an observation against what the video plainly is — settle it with a narrow+`observe_reference --question` over that shot, which puts the question to the reference's own+observer. On the `gemini` observer that means you do not open the frame yourself;+`reconstruction-loop.md` says which rule binds on which path. Do not re-observethe full shot: that is paid, and at temperature `1.0` it returns a paraphrase rather than acorrection.Be most suspicious of anything an observation asserts about change over time — something appearing,vanishing, being removed, being drawn in a single frame. A describer working from one pass infersthose rather than seeing them, and infers them wrongly. A `compare_reconstruction` against a rendered-probe, or a narrow question over the stretch, is how such a claim is checked — not by extracting-frames and reading the sequence yourself.+probe, or a narrow question over the stretch, is how such a claim is checked, rather than by+believing the single pass that asserted it.## Narrow questions@@ -151,19 +158,20 @@ and returns a description of their visible differences. It is never told which iwas built, or how; `--question` may narrow it to one region of the picture and nothing else. Resultsare not cached. `reconstruction-loop.md` governs when and how to use it.-This is the only way the reference is ever seen. Do not open the reference frame yourself and look-at it, even if the model running this route can read images — the reference has one observer, the-same Gemini that wrote the observations, and it reports through this command. `reconstruction-loop.md`-states this as a rule, not a preference.+The reference has one observer for its whole life, and this command goes through the same one that+wrote its observations. On the `gemini` observer that observer is not you, and opening the reference+frame yourself is what `reconstruction-loop.md` forbids; on the `agent` observer it is you, and that+file names the discipline that takes the place of blindness.## Boundaries-Gemini returns natural language only. Never send it SVML syntax, package declarations, vocabulary,-previews or implementation code. Never ask it to write SVML, SVS, SVRun, a structured reference plan,-component names or TypeScript. Never tell it what you built or what you expect it to find. It is-evidence, not the final decision maker.+An observation is natural language and nothing else. Never send an observer SVML syntax, package+declarations, vocabulary, previews or implementation code. Never ask for SVML, SVS, SVRun, a structured+reference plan, component names or TypeScript in an observation. Never state what you built or what you+expect to be found. An observation is evidence, not the final decision maker — which is a rule about+what an observation may contain, so it binds when you are the one writing it.Inspect every failed or unresolved result. Follow up with one narrow question over one to threerelevant shots rather than repeating the entire analysis. Preserve successful cached observationsunless the selected shot or preparation stage must be refreshed. Do not pass model, concurrency,-rate-limit or temperature settings; temperature is fixed at `1.0`.+rate-limit or temperature settings; on the `gemini` observer temperature is fixed at `1.0`.diff --git a/packages/reference-video-tools/README.md b/packages/reference-video-tools/README.mdindex f2cc75c2..bfda6d5a 100644--- a/packages/reference-video-tools/README.md+++ b/packages/reference-video-tools/README.md@@ -2,12 +2,22 @@CLI tools for reconstructing a reference video with Hypit.-The package exposes five CLI subcommands: `list_svml_packages`, `prepare_reference`,-`observe_reference`, `inspect_svml_vocabulary`, and `compare_reconstruction`. Each command prints one JSON result to-stdout. Every command except `inspect_svml_vocabulary` sends narrow natural-language requests to-Gemini with a fixed temperature of `1.0`; they never receive SVML syntax and never write SVML. The-final source files are authored by the calling agent and checked with the existing-`pnpm hypit check` command.+The package exposes six CLI subcommands: `list_svml_packages`, `prepare_reference`,+`observe_reference`, `record_observation`, `inspect_svml_vocabulary`, and `compare_reconstruction`.+Each command prints one JSON result to stdout. The final source files are authored by the calling+agent and checked with the existing `pnpm hypit check` command.++`--observer` on `prepare_reference` chooses who reads the reference, once per reference:++- `gemini` uploads the shot clips and the whole video to Vertex at a fixed temperature of `1.0`. It+ needs `GOOGLE_CLOUD_PROJECT` and `GOOGLE_APPLICATION_CREDENTIALS_JSON`, and it is the default.+- `agent` reaches no Provider. `prepare_reference` and `observe_reference` return each observation as+ a task carrying its prompt and the pictures to answer it from — one tile of evenly sampled frames+ per shot, the storyboard for the whole reference — and the calling agent answers them with+ `record_observation`.++Both observers produce the same observation keys in the same cache, so everything downstream reads one+shape. Neither receives SVML syntax, and neither writes SVML.Run the CLI from the repository or an installed package:diff --git a/packages/reference-video-tools/src/cli.ts b/packages/reference-video-tools/src/cli.tsindex dc149bf9..3ec452aa 100644--- a/packages/reference-video-tools/src/cli.ts+++ b/packages/reference-video-tools/src/cli.ts@@ -1,4 +1,6 @@#!/usr/bin/env node+import { readFile } from "node:fs/promises";+import { createReferenceVideoTools } from "./tools.js";type Flags = ReadonlyMap<string, string | readonly string[] | boolean>;@@ -7,12 +9,18 @@ function usage(): string {return ["Usage:"," hypit-reference-video-tools list_svml_packages",- " hypit-reference-video-tools prepare_reference --video-path <path> [--redo media|transcript|people|voices|systems|places|all]",+ " hypit-reference-video-tools prepare_reference --video-path <path> [--observer gemini|agent] [--redo media|transcript|people|voices|systems|places|all]"," hypit-reference-video-tools observe_reference --reference-id <id> [--shot-id <id> ...] [--reobserve]"," hypit-reference-video-tools observe_reference --reference-id <id> --shot-id <id> [--shot-id <id> ...] --question <text>",+ " hypit-reference-video-tools record_observation --reference-id <id> --key <key> --text <text>|--text-file <path>"," hypit-reference-video-tools inspect_svml_vocabulary --package <name> [--package <name> ...] [--tag <tag> ...] [--without-previews]"," hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> --image <path> [--question <scope>]","",+ "--observer picks who reads the reference, once per reference. `gemini` uploads video to Vertex and",+ "needs GOOGLE_CLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS_JSON. `agent` needs no credentials: it",+ "returns each observation as a task carrying its prompt and one tiled picture per shot, which the",+ "calling agent answers with record_observation.",+ "","Every command prints one JSON result to stdout. Use --input <json> instead of flags when a complete input object is easier to pass.",].join("\n");}@@ -83,11 +91,14 @@ async function main(): Promise<void> {if (command === "list_svml_packages") {result = await tools.list_svml_packages();} else if (command === "prepare_reference") {+ const observer = one(flags, "observer");+ if (observer !== undefined && observer !== "gemini" && observer !== "agent") throw new Error("--observer must be gemini or agent");const input = supplied ?? {video_path: required(flags, "video-path"),+ ...(observer === undefined ? {} : { observer }),...(one(flags, "redo") === undefined ? {} : { redo: one(flags, "redo") }),};- result = await tools.prepare_reference(input as { video_path: string; redo?: "media" | "transcript" | "people" | "voices" | "systems" | "places" | "all" });+ result = await tools.prepare_reference(input as { video_path: string; observer?: "gemini" | "agent"; redo?: "media" | "transcript" | "people" | "voices" | "systems" | "places" | "all" });} else if (command === "observe_reference") {const input = supplied ?? {reference_id: required(flags, "reference-id"),@@ -96,6 +107,16 @@ async function main(): Promise<void> {...(flags.get("reobserve") === true ? { reobserve: true } : {}),};result = await tools.observe_reference(input as { reference_id: string; shot_ids?: readonly string[]; question?: string; reobserve?: boolean });+ } else if (command === "record_observation") {+ // An observation is paragraphs of prose. --text keeps a short one on the command line; --text-file+ // is how a long one arrives without the shell deciding where it ends.+ const textFile = one(flags, "text-file");+ const input = supplied ?? {+ reference_id: required(flags, "reference-id"),+ key: required(flags, "key"),+ text: textFile === undefined ? required(flags, "text") : await readFile(textFile, "utf8"),+ };+ result = await tools.record_observation(input as { reference_id: string; key: string; text: string });} else if (command === "compare_reconstruction") {const input = supplied ?? {reference_id: required(flags, "reference-id"),diff --git a/packages/reference-video-tools/src/media.ts b/packages/reference-video-tools/src/media.tsindex a25b5b0d..07251958 100644--- a/packages/reference-video-tools/src/media.ts+++ b/packages/reference-video-tools/src/media.ts@@ -108,6 +108,25 @@ async function extract(path: string, args: readonly string[], target: string): Preturn target;}+// An observer that reads images rather than video sees a shot as this many frames, sampled evenly+// across it and tiled into one picture in reading order. Four is enough for a one-second shot to+// show what moves; nine keeps a fifteen-second one legible at a cell width a reader can still resolve+// detail in. The clip is decoded once, and the sampling and the tiling happen in that one pass.+const TILE_COLUMNS = 3;+function tileFrames(duration: number): number { return clamp(Math.round(duration * 1.5), 4, 9); }++async function shotTile(clip: string, duration: number, target: string): Promise<string> {+ const frames = tileFrames(duration);+ const rows = Math.ceil(frames / TILE_COLUMNS);+ const rate = round(frames / Math.max(duration, 0.1));+ await command("ffmpeg", [+ "-hide_banner", "-loglevel", "error", "-y", "-i", clip,+ "-vf", `fps=${rate},scale=480:-2,tile=layout=${TILE_COLUMNS}x${rows}:padding=8:margin=8:color=black`,+ "-frames:v", "1", "-q:v", "3", target,+ ], 300_000);+ return target;+}+export async function prepareMedia(videoPath: string, root: string, duration: number): Promise<{readonly bounds: readonly Bounds[];readonly storyboard: string;@@ -127,6 +146,7 @@ export async function prepareMedia(videoPath: string, root: string, duration: nuawait extract(videoPath, ["-i", videoPath, "-ss", String(bound.start), "-t", String(Math.max(0.1, bound.end - bound.start)), "-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-c:a", "aac", "-b:a", "96k", "-movflags", "+faststart"], clip);await extract(videoPath, ["-i", videoPath, "-ss", String(bound.start + (bound.end - bound.start) / 2), "-frames:v", "1", "-vf", "scale='min(720,iw)':-2", "-q:v", "3"], join(shotDir, `${id}-representative.jpg`));await extract(videoPath, ["-sseof", "-0.1", "-i", clip, "-update", "1", "-frames:v", "1", "-vf", "scale='min(720,iw)':-2", "-q:v", "3"], join(shotDir, `${id}-tail.jpg`));+ await shotTile(clip, bound.end - bound.start, join(shotDir, `${id}-frames.jpg`));if (await hasAudio(clip)) await extract(videoPath, ["-sseof", "-3", "-i", clip, "-map", "0:a:0", "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le"], join(shotDir, `${id}-audio.wav`));}));const inputs = bounds.map((_, index) => `[${index}:v]`).join("");diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.tsindex fe5dd833..aabc362a 100644--- a/packages/reference-video-tools/src/tools.ts+++ b/packages/reference-video-tools/src/tools.ts@@ -19,11 +19,12 @@ import {writeJson,} from "./media.js";import { prepareTranscript } from "./transcript.js";-import type { Observation, PrepareResult, ReferenceState, Shot, Transcript } from "./types.js";+import type { Observation, ObservationTaskRequest, Observer, PrepareResult, ReferenceState, Shot, Transcript } from "./types.js";export type PrepareReferenceInput = {readonly video_path: string;readonly redo?: "media" | "transcript" | "people" | "voices" | "systems" | "places" | "all";+ readonly observer?: Observer;};export type ObserveReferenceInput = {readonly reference_id: string;@@ -31,6 +32,11 @@ export type ObserveReferenceInput = {readonly question?: string;readonly reobserve?: boolean;};+export type RecordObservationInput = {+ readonly reference_id: string;+ readonly key: string;+ readonly text: string;+};export type InspectVocabularyInput = {readonly package_names: readonly string[];readonly tags?: readonly string[];@@ -49,6 +55,7 @@ export type ReferenceVideoTools = {observe_reference(input: ObserveReferenceInput): Promise<Record<string, unknown>>;inspect_svml_vocabulary(input: InspectVocabularyInput): Promise<Record<string, unknown>>;compare_reconstruction(input: CompareReconstructionInput): Promise<Record<string, unknown>>;+ record_observation(input: RecordObservationInput): Promise<Record<string, unknown>>;};type ToolOptions = {@@ -61,9 +68,9 @@ type ToolOptions = {readonly generate?: GenerateText;};export type GenerateText = (input: { readonly parts: readonly Part[]; readonly instruction: string }) => Promise<string>;-type ObservationTask = { readonly key: string; readonly run: () => Promise<Observation> };+type ObservationTask = { readonly key: string; readonly request: Request };-function observation(status: "complete" | "failed", text: string): Observation { return { status, text }; }+function observation(status: Observation["status"], text: string): Observation { return { status, text }; }function unavailable(reason: string): Transcript { return { status: "unavailable", transcript_ref: null, word_count: 0, reason }; }function positiveEnv(name: string): number | undefined {@@ -162,6 +169,41 @@ async function callSafely(retryDelayMs: number, generate: GenerateText, parts: rreturn observation("failed", last instanceof Error ? last.message : String(last));}+// The agent observer reads pictures, so each piece of evidence a request would have uploaded becomes+// one it can look at: a shot's clip becomes that shot's frame tile, the whole video becomes the+// storyboard of representative frames, and audio has no picture to become. What each observation asks+// is identical on both paths; only the shape of the evidence differs.+function asPictures(media: readonly string[], state: ReferenceState): readonly string[] {+ const tiles = new Map(state.shots.map((shot) => [shot.clip_ref, shot.frames_tile_ref]));+ const pictures: string[] = [];+ for (const path of media) {+ const tile = tiles.get(path);+ if (tile !== undefined) pictures.push(tile);+ else if (path === state.analysis_video_ref) pictures.push(state.storyboard_ref);+ else if (!path.toLowerCase().endsWith(".wav")) pictures.push(path);+ }+ return [...new Set(pictures)];+}++// What the pictures are has to be said, because a tile of frames is not what the prompt was written+// for. The prompt itself is unchanged on both paths: only this preamble is added, and only here.+const TILE_PREAMBLE = "Each supplied picture that shows a grid of frames is one shot, sampled evenly"+ + " across its duration and laid out in reading order: left to right, then top to bottom. Read the"+ + " grid as time passing. A single picture that is not a grid is one moment.";+const NO_SOUND = "You are reading pictures and cannot hear this reference. Answer what the pictures and"+ + " the measured transcript support, and say plainly which parts of the question need sound you do"+ + " not have rather than inferring them from appearance.";++/** Declares one observation: what it asks, and the evidence it asks over. The observer decides how. */+type Request = {+ readonly media: readonly string[];+ readonly prompt: string;+ readonly instruction: string;+ /** True when answering needs sound. The `agent` observer has none and is told to say so. */+ readonly sound?: boolean;+};+type Asker = (key: string, request: Request) => Promise<Observation>;+async function pacedMap<T, R>(items: readonly T[], concurrency: number, gapMs: number, run: (item: T, index: number) => Promise<R>): Promise<readonly R[]> {const result = new Array<R>(items.length);let cursor = 0;@@ -194,14 +236,18 @@ async function runObservationTasks(forcedKeys: ReadonlySet<string>,concurrency: number,gapMs: number,+ ask: Asker,): Promise<ReadonlyMap<string, Observation>> {const path = join(root, "observations.json");const cache = await readJson<Record<string, Observation>>(path) ?? {};- const pending = tasks.filter((task) => forcedKeys.has(task.key) || cache[task.key]?.status !== "complete");- const completed = await pacedMap(pending, concurrency, gapMs, async (task) => ({ key: task.key, value: await task.run() }));- for (const item of completed) cache[item.key] = item.value;+ const outstanding = tasks.filter((task) => forcedKeys.has(task.key) || cache[task.key]?.status !== "complete");+ const completed = await pacedMap(outstanding, concurrency, gapMs, async (task) => ({ key: task.key, value: await ask(task.key, task.request) }));+ const answers = new Map(completed.map((item) => [item.key, item.value]));+ // A pending answer is a task handed out, not an answer received. Caching it would make the next run+ // read the placeholder as complete and never ask again.+ for (const item of completed) if (item.value.status !== "pending") cache[item.key] = item.value;await writeJson(path, cache);- return new Map(tasks.map((task) => [task.key, cache[task.key] ?? observation("failed", "not observed")]));+ return new Map(tasks.map((task) => [task.key, cache[task.key] ?? answers.get(task.key) ?? observation("failed", "not observed")]));}async function shotFromBound(root: string, index: number, bound: { start: number; end: number; group: number; part: number; parts: number }): Promise<Shot> {@@ -222,6 +268,7 @@ async function shotFromBound(root: string, index: number, bound: { start: numberclip_ref: join(dir, `${id}.mp4`),representative_frame_ref: join(dir, `${id}-representative.jpg`),tail_frame_ref: join(dir, `${id}-tail.jpg`),+ frames_tile_ref: join(dir, `${id}-frames.jpg`),audio_tail_ref,};}@@ -272,6 +319,37 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVreturn state;};+ // One reference is read by one observer throughout. Mixing them inside a single reference would+ // leave observations of two different kinds of evidence under the same keys, with nothing on the+ // record saying which is which, so the choice is made once at prepare and carried in the state.+ const askerFor = async (+ observer: Observer,+ state: ReferenceState,+ ): Promise<{ readonly ask: Asker; readonly pending: readonly ObservationTaskRequest[] }> => {+ if (observer === "agent") {+ const pending: ObservationTaskRequest[] = [];+ return {+ pending,+ ask: async (key, { media, prompt, instruction, sound }) => {+ const preamble = sound === true ? `${TILE_PREAMBLE}\n\n${NO_SOUND}` : TILE_PREAMBLE;+ pending.push({ key, instruction, prompt: `${preamble}\n\n${prompt}`, image_refs: asPictures(media, state) });+ return observation("pending", "awaiting the agent observer");+ },+ };+ }+ const generate = await generator();+ const mediaPart = mediaParts();+ return {+ pending: [],+ ask: async (_key, { media, prompt, instruction }) => {+ const parts: Part[] = [];+ for (const path of media) parts.push(await mediaPart(path));+ parts.push({ text: prompt });+ return await callSafely(retryDelayMs, generate, parts, instruction);+ },+ };+ };+return {// Which packages exist is the first question of every reconstruction, and until now the only// answer was to read a guide and a directory listing by hand. The Build CLI deliberately never@@ -331,7 +409,13 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst redoVoices = input.redo === "voices" || input.redo === "all";const redoSystems = input.redo === "systems" || input.redo === "all";const redoPlaces = input.redo === "places" || input.redo === "all";- if (input.redo === undefined && existing !== undefined && prepared(existing)) return publicPrepare(existing);+ // A reference keeps the observer it was prepared with. Asking for the other one on a reference+ // that already holds observations would read the second half of it through different evidence+ // from the first, so the mismatch is refused rather than silently mixed.+ const observer: Observer = input.observer ?? existing?.observer ?? "gemini";+ assert(input.redo === "all" || existing?.observer === undefined || existing.observer === observer,+ `reference ${reference} was prepared for the ${existing?.observer} observer; --redo all re-prepares it for the other one`);+ if (input.redo === undefined && existing !== undefined && prepared(existing)) return { ...publicPrepare(existing), observer, pending_observations: [] };await ensureDir(root);const info = existing?.video === undefined || redoMedia ? await probe(videoPath) : {duration: existing.video.duration_seconds,@@ -366,56 +450,74 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVawait writeJson(statePath, state);await writeFile(join(root, "observations.json"), "{}\n", "utf8");}- const generate = await generator();- // WhisperX is local and slow and shares nothing with the Gemini requests, so it runs beside+ // WhisperX is local and slow and shares nothing with the observation requests, so it runs beside// them rather than ahead of them.const transcribing = state.transcript?.status === "complete" && !redoTranscript? Promise.resolve(state.transcript): prepareTranscript(reference, videoPath, root, info.hasAudio, redoTranscript);- const mediaPart = mediaParts();- const globalParts: Part[] = [await mediaPart(analysisVideo)];+ const { ask, pending } = await askerFor(observer, state);+ const whole = [analysisVideo];const [people, voices, systems, places] = await Promise.all([state.people_and_product?.status === "complete" && !redoPeople? Promise.resolve(state.people_and_product)- : callSafely(retryDelayMs, generate, [...globalParts, { text: "Describe the recurring people and the promoted product in this complete reference video. Return natural language only. Identify stable visual traits and distinguish recurring speakers from incidental people in inserts. Describe the product once, comprehensively, for reuse." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),+ : ask("people_and_product", { media: whole, prompt: "Describe the recurring people and the promoted product in this complete reference video. Return natural language only. Identify stable visual traits and distinguish recurring speakers from incidental people in inserts. Describe the product once, comprehensively, for reuse.", instruction: "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names." }),state.voices?.status === "complete" && !redoVoices? Promise.resolve(state.voices)- : callSafely(retryDelayMs, generate, [...globalParts, { text: "Listen to this complete reference video and describe the distinct voices, their order, overlap, off-screen speech, and likely correspondence to visible people. Do not assign a voice merely because a person appears in a B-roll image. Return natural language only." }], "You only listen to a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),+ : ask("voices", { media: whole, sound: true, prompt: "Listen to this complete reference video and describe the distinct voices, their order, overlap, off-screen speech, and likely correspondence to visible people. Do not assign a voice merely because a person appears in a B-roll image. Return natural language only.", instruction: "You only listen to a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names." }),state.persistent_systems?.status === "complete" && !redoSystems? Promise.resolve(state.persistent_systems)- : callSafely(retryDelayMs, generate, [...globalParts, { text: "Describe the on-screen text and graphic systems that persist or recur across this complete reference video, such as subtitles, running lists, counters, progress indicators, badges, watermarks, lower thirds and repeating full-screen graphic layouts. For each one, state when it first appears and when it stops, whether it is present continuously or intermittently, whether its own appearance stays the same throughout, and describe any point where its appearance actually changes. Report only what stays consistent across the video; ignore one-off elements that appear a single time. Return natural language only." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),+ : ask("persistent_systems", { media: whole, prompt: "Describe the on-screen text and graphic systems that persist or recur across this complete reference video, such as subtitles, running lists, counters, progress indicators, badges, watermarks, lower thirds and repeating full-screen graphic layouts. For each one, state when it first appears and when it stops, whether it is present continuously or intermittently, whether its own appearance stays the same throughout, and describe any point where its appearance actually changes. Report only what stays consistent across the video; ignore one-off elements that appear a single time. Return natural language only.", instruction: "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names." }),state.places?.status === "complete" && !redoPlaces? Promise.resolve(state.places)- : callSafely(retryDelayMs, generate, [...globalParts, { text: "Describe every distinct place this reference video was shot in, and every distinct camera position within each place. State how many places there are, which parts of the video happen in each, and for each place which camera positions appear and which parts of the video use each one. Two shots are the same camera position when the camera sees the same part of the room from the same side; a reverse angle is a different position of the same place.\n\nDescribe each camera position in enough detail that someone who has never seen this video could draw it from your words alone: what is behind and beside the subject, the shape and depth of the space, where the light comes from and how hard it is, the colours and materials of the surfaces, and the objects a viewer would use to recognise it again. Say what stays identical between positions of one place and what differs.\n\nReturn natural language only." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),+ : ask("places", { media: whole, prompt: "Describe every distinct place this reference video was shot in, and every distinct camera position within each place. State how many places there are, which parts of the video happen in each, and for each place which camera positions appear and which parts of the video use each one. Two shots are the same camera position when the camera sees the same part of the room from the same side; a reverse angle is a different position of the same place.\n\nDescribe each camera position in enough detail that someone who has never seen this video could draw it from your words alone: what is behind and beside the subject, the shape and depth of the space, where the light comes from and how hard it is, the colours and materials of the surfaces, and the objects a viewer would use to recognise it again. Say what stays identical between positions of one place and what differs.\n\nReturn natural language only.", instruction: "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names." }),]);- const complete: ReferenceState = { ...state, transcript: await transcribing, people_and_product: people, voices, persistent_systems: systems, places };+ // A pending whole-reference observation is a task the agent still owes, so it is reported rather+ // than written: writing it would make the next run treat the placeholder as an answer.+ const answered = <T extends Observation>(value: T): T | undefined => value.status === "pending" ? undefined : value;+ const complete: ReferenceState = {+ ...state,+ observer,+ transcript: await transcribing,+ ...(answered(people) === undefined ? {} : { people_and_product: people }),+ ...(answered(voices) === undefined ? {} : { voices }),+ ...(answered(systems) === undefined ? {} : { persistent_systems: systems }),+ ...(answered(places) === undefined ? {} : { places }),+ };await writeJson(statePath, complete);- return publicPrepare(complete);+ return {+ ...publicPrepare(complete),+ people_and_product: people,+ voices,+ persistent_systems: systems,+ places,+ observer,+ pending_observations: pending,+ };},async observe_reference(input): Promise<Record<string, unknown>> {- const mediaPart = mediaParts();const state = await loadState(input.reference_id);const selected = input.shot_ids === undefined ? state.shots : state.shots.filter((shot) => input.shot_ids!.includes(shot.shot_id));assert(selected.length > 0, "no requested shot ids exist");- const generate = await generator();+ const observer: Observer = state.observer ?? "gemini";+ const { ask, pending } = await askerFor(observer, state);const question = input.question?.trim() ?? "";if (question.length > 0) {assert(input.shot_ids !== undefined && input.shot_ids.length > 0, "question requires at least one shot id, so that it is answered from the shots it is about");assert(selected.length <= 3, "question accepts at most three shots");- const parts: Part[] = [];- for (const shot of selected) {- parts.push(await mediaPart(shot.clip_ref));- parts.push(await mediaPart(shot.representative_frame_ref));- }- parts.push({ text: question });- const answer = await callSafely(retryDelayMs, generate, parts, "Answer only the narrow reference-video question in natural language. Do not write code, markup, SVML, or component names.");+ const answer = await ask("question", {+ media: selected.flatMap((shot) => [shot.clip_ref, shot.representative_frame_ref]),+ prompt: question,+ instruction: "Answer only the narrow reference-video question in natural language. Do not write code, markup, SVML, or component names.",+ });return {reference_id: state.reference_id,+ observer,shot_ids: selected.map((shot) => shot.shot_id),question,answer,unresolved: answer.status === "complete" ? [] : ["question"],+ pending_observations: pending,};}const selectedIds = new Set(selected.map((shot) => shot.shot_id));@@ -428,37 +530,43 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVstate.voices?.text === undefined ? "" : `Full-reference voice evidence:\n${state.voices.text}`,state.persistent_systems?.text === undefined ? "" : `Full-reference persistent on-screen system evidence:\n${state.persistent_systems.text}`,].filter((item) => item.length > 0).join("\n\n");- const visualTasks: ObservationTask[] = selected.map((shot) => ({ key: `visual:${shot.shot_id}`, run: async () => {+ const visualTasks: ObservationTask[] = selected.map((shot) => {const previous = state.shots.find((candidate) => candidate.index === shot.index - 1);- const parts: Part[] = [await mediaPart(shot.clip_ref), await mediaPart(shot.representative_frame_ref)];- if (previous !== undefined) parts.push(await mediaPart(previous.tail_frame_ref));- parts.push({ text: `${globalContext}\n\nObserve shot ${shot.index}. Describe the current base picture, any covering or non-covering visual content, and whether visible content continues from the preceding shot. A full-screen insert is still only a picture observation.\n\nAlso answer these two questions explicitly.\n\nFirst: is the whole frame a depicted scene that has its own camera space, lighting, depth and lens behaviour, or is it a flat designed field whose purpose is to carry drawn elements such as words, rows, panels or cards? Live action and animation are both depicted scenes; a paper sheet, ruled or gridded surface, flat or gradient colour, blurred wallpaper, board or slide backdrop filling the frame is a designed field. State which one it is and the visible evidence for it.\n\nThen, whichever it is, say whether any *region* of the frame is itself a flat designed field carrying drawn content — a list, a ranking, a leaderboard, a scoreboard, a chart, a panel, a slab of colour holding rows or labels — even when the rest of the frame is a depicted scene, and even when the region has no visible border, card edge or drop shadow around it. Report each such region separately from the scene behind it: roughly where it sits and how much of the frame it covers, given as fractions of the frame width and height; what its own surface is; and what is drawn on it. A designed field occupying half the frame, one side of it, or a band across it is easy to describe as "text over the picture" and is not that: the field itself is the thing to report.\n\nSecond: for every framed element inside the picture, such as a card, phone, browser window, screenshot or inset, describe the picture inside the frame and the frame itself separately. For the inside, describe what it depicts and whether it moves. For the frame, describe its border, corner radius, outline, shadow, size, position and how it enters and leaves.\n\nThird: does this picture move at all, and how? Separate three things: whether the camera moves, and how; whether anything in the picture moves, and what; and whether the picture is completely still. A held photograph, screenshot or card that only appears and disappears is still, however long it is on screen.\n\nReturn natural language evidence only.` });- return await callSafely(retryDelayMs, generate, parts, "Observe picture only. Do not choose SVML components, do not write markup, and do not decide final source syntax.");- }}));- const typeTasks: ObservationTask[] = selected.map((shot) => ({ key: `type:${shot.shot_id}`, run: async () => {- const parts: Part[] = [await mediaPart(shot.representative_frame_ref), await mediaPart(shot.clip_ref)];- parts.push({ text: `Observe the on-screen text in shot ${shot.index}. Report only text that is drawn over or composed into the picture; ignore text belonging to a photographed object such as a device screen, sign, label or document. If the shot shows no drawn text, say exactly that and stop.\n\nFor each distinct text element describe: the typeface character (serif, sans-serif, handwritten, monospaced or display), the weight, the cap height as a fraction of the frame height, letter spacing and line spacing, alignment, letter case, fill colour, any outline or stroke with its colour and thickness relative to the stroke width of the letters, any drop shadow with its direction, distance and softness, any glow or blur, any emphasis applied to individual words or characters and how it differs from the rest, the position within the frame and the distance from the nearest edges, and how the text enters, changes and leaves during the shot.\n\nReturn natural language evidence only.` });- return await callSafely(retryDelayMs, generate, parts, "Observe the appearance of on-screen text only. Do not name SVML components, do not write markup, and do not name font files or style properties from any software.");- }}));- const audioTasks: ObservationTask[] = selected.map((shot) => ({ key: `audio:${shot.shot_id}`, run: async () => {+ return { key: `visual:${shot.shot_id}`, request: {+ media: [shot.clip_ref, shot.representative_frame_ref, ...(previous === undefined ? [] : [previous.tail_frame_ref])],+ prompt: `${globalContext}\n\nObserve shot ${shot.index}. Describe the current base picture, any covering or non-covering visual content, and whether visible content continues from the preceding shot. A full-screen insert is still only a picture observation.\n\nAlso answer these two questions explicitly.\n\nFirst: is the whole frame a depicted scene that has its own camera space, lighting, depth and lens behaviour, or is it a flat designed field whose purpose is to carry drawn elements such as words, rows, panels or cards? Live action and animation are both depicted scenes; a paper sheet, ruled or gridded surface, flat or gradient colour, blurred wallpaper, board or slide backdrop filling the frame is a designed field. State which one it is and the visible evidence for it.\n\nThen, whichever it is, say whether any *region* of the frame is itself a flat designed field carrying drawn content — a list, a ranking, a leaderboard, a scoreboard, a chart, a panel, a slab of colour holding rows or labels — even when the rest of the frame is a depicted scene, and even when the region has no visible border, card edge or drop shadow around it. Report each such region separately from the scene behind it: roughly where it sits and how much of the frame it covers, given as fractions of the frame width and height; what its own surface is; and what is drawn on it. A designed field occupying half the frame, one side of it, or a band across it is easy to describe as "text over the picture" and is not that: the field itself is the thing to report.\n\nSecond: for every framed element inside the picture, such as a card, phone, browser window, screenshot or inset, describe the picture inside the frame and the frame itself separately. For the inside, describe what it depicts and whether it moves. For the frame, describe its border, corner radius, outline, shadow, size, position and how it enters and leaves.\n\nThird: does this picture move at all, and how? Separate three things: whether the camera moves, and how; whether anything in the picture moves, and what; and whether the picture is completely still. A held photograph, screenshot or card that only appears and disappears is still, however long it is on screen.\n\nReturn natural language evidence only.`,+ instruction: "Observe picture only. Do not choose SVML components, do not write markup, and do not decide final source syntax.",+ } };+ });+ const typeTasks: ObservationTask[] = selected.map((shot) => ({ key: `type:${shot.shot_id}`, request: {+ media: [shot.representative_frame_ref, shot.clip_ref],+ prompt: `Observe the on-screen text in shot ${shot.index}. Report only text that is drawn over or composed into the picture; ignore text belonging to a photographed object such as a device screen, sign, label or document. If the shot shows no drawn text, say exactly that and stop.\n\nFor each distinct text element describe: the typeface character (serif, sans-serif, handwritten, monospaced or display), the weight, the cap height as a fraction of the frame height, letter spacing and line spacing, alignment, letter case, fill colour, any outline or stroke with its colour and thickness relative to the stroke width of the letters, any drop shadow with its direction, distance and softness, any glow or blur, any emphasis applied to individual words or characters and how it differs from the rest, the position within the frame and the distance from the nearest edges, and how the text enters, changes and leaves during the shot.\n\nReturn natural language evidence only.`,+ instruction: "Observe the appearance of on-screen text only. Do not name SVML components, do not write markup, and do not name font files or style properties from any software.",+ } }));+ const audioTasks: ObservationTask[] = selected.map((shot) => {const previous = state.shots.find((candidate) => candidate.index === shot.index - 1);- const parts: Part[] = [await mediaPart(shot.clip_ref)];- if (previous?.audio_tail_ref !== null && previous?.audio_tail_ref !== undefined) parts.push(await mediaPart(previous.audio_tail_ref));- parts.push({ text: `${globalContext}\n\nListen to shot ${shot.index}. Decide who is speaking, whether sound continues from the previous shot, whether visible people actually produce the sound, and whether a silent B-roll person only appears to speak. Handle off-screen, alternating, overlapping, and no-person-visible speech. Return natural language evidence only.` });- return await callSafely(retryDelayMs, generate, parts, "Observe sound only. Do not write markup, SVML, JSON plans, or component names.");- }}));+ return { key: `audio:${shot.shot_id}`, request: {+ media: [shot.clip_ref, ...(previous?.audio_tail_ref == null ? [] : [previous.audio_tail_ref])],+ sound: true,+ prompt: `${globalContext}\n\nListen to shot ${shot.index}. Decide who is speaking, whether sound continues from the previous shot, whether visible people actually produce the sound, and whether a silent B-roll person only appears to speak. Handle off-screen, alternating, overlapping, and no-person-visible speech. Return natural language evidence only.`,+ instruction: "Observe sound only. Do not write markup, SVML, JSON plans, or component names.",+ } };+ });// Both boundary questions look at exactly the same three files, so they travel together. Asking// them separately uploaded two clips twice to learn two things about one cut.- const boundaryTasks: ObservationTask[] = boundaryRights.map((right) => ({ key: `boundary:${right.shot_id}`, run: async () => {+ const boundaryTasks: ObservationTask[] = boundaryRights.map((right) => {const left = state.shots.find((shot) => shot.index === right.index - 1)!;- const parts: Part[] = [await mediaPart(left.clip_ref), await mediaPart(right.clip_ref), await mediaPart(left.tail_frame_ref)];- parts.push({ text: `Compare shots ${left.index} and ${right.index}, which meet at one cut. Answer two questions separately, each under its own heading.\n\nContinuous camera shot: are these one continuous camera shot, or two? Explain the visual and sound continuity evidence.\n\nOverlay continuity: does the same visible overlay or inserted picture continue across the boundary, change, or end? Explain the evidence.\n\nReturn natural language evidence only.` });- return await callSafely(retryDelayMs, generate, parts, "Analyze what happens at one cut. Do not name SVML components or write code.");- }}));+ return { key: `boundary:${right.shot_id}`, request: {+ media: [left.clip_ref, right.clip_ref, left.tail_frame_ref],+ sound: true,+ prompt: `Compare shots ${left.index} and ${right.index}, which meet at one cut. Answer two questions separately, each under its own heading.\n\nContinuous camera shot: are these one continuous camera shot, or two? Explain the visual and sound continuity evidence.\n\nOverlay continuity: does the same visible overlay or inserted picture continue across the boundary, change, or end? Explain the evidence.\n\nReturn natural language evidence only.`,+ instruction: "Analyze what happens at one cut. Do not name SVML components or write code.",+ } };+ });const allTasks = [...visualTasks, ...typeTasks, ...audioTasks, ...boundaryTasks];const reobserve = input.reobserve === true;const forcedKeys = reobserve ? new Set(allTasks.map((task) => task.key)) : new Set<string>();- const observations = await runObservationTasks(state.root, allTasks, forcedKeys, concurrency, gapMs);+ const observations = await runObservationTasks(state.root, allTasks, forcedKeys, concurrency, gapMs, ask);const unresolved = [...observations].filter(([, value]) => value.status !== "complete").map(([key]) => key);const boundaryText = boundaryRights.map((right) => observations.get(`boundary:${right.shot_id}`)?.text ?? "").join("\n");const needsThreeShotReview = selected.length >= 3 && (@@ -474,10 +582,11 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV}): []).map((items) => items as unknown as readonly [Shot, Shot, Shot]);const threeShotObservations = await pacedMap(windows, concurrency, gapMs, async (items) => {- const parts: Part[] = [];- for (const shot of items) parts.push(await mediaPart(shot.clip_ref));- parts.push({ text: `Compare shots ${items[0].index}, ${items[1].index}, and ${items[2].index}. Decide whether the three clips are one continuous camera shot and whether one visual overlay or insert persists through both boundaries. Explain the evidence in natural language only.` });- const result = await callSafely(retryDelayMs, generate, parts, "Analyze a three-shot continuity window only. Do not write markup, SVML, JSON plans, or component names.");+ const result = await ask(`window:${items[0].shot_id}`, {+ media: items.map((shot) => shot.clip_ref),+ prompt: `Compare shots ${items[0].index}, ${items[1].index}, and ${items[2].index}. Decide whether the three clips are one continuous camera shot and whether one visual overlay or insert persists through both boundaries. Explain the evidence in natural language only.`,+ instruction: "Analyze a three-shot continuity window only. Do not write markup, SVML, JSON plans, or component names.",+ });return { shot_ids: items.map((shot) => shot.shot_id), combined_duration_seconds: Number((items[2].end_seconds - items[0].start_seconds).toFixed(3)), result };});for (const window of threeShotObservations) {@@ -501,10 +610,12 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV});return {reference_id: state.reference_id,+ observer,shots: shotResults,boundaries: boundaryResults,three_shot_observations: threeShotObservations,unresolved,+ pending_observations: pending,};},@@ -533,7 +644,6 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV},async compare_reconstruction(input): Promise<Record<string, unknown>> {- const mediaPart = mediaParts();const state = await loadState(input.reference_id);const shot = state.shots.find((candidate) => candidate.shot_id === input.shot_id);assert(shot !== undefined, `shot ${input.shot_id} does not exist in reference ${input.reference_id}`);@@ -541,22 +651,54 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst file = await stat(imagePath).catch(() => undefined);assert(file?.isFile(), `image_path is not a file: ${imagePath}`);const scope = input.question?.trim() ?? "";- const generate = await generator();- const parts: Part[] = [await mediaPart(shot.representative_frame_ref), await mediaPart(imagePath)];- parts.push({ text: `Two still images are supplied in order: image one, then image two.${scope.length === 0 ? "" : `\n\nLimit the comparison to this part of the picture: ${scope}`}\n\nDescribe every visible difference between them: layout and arrangement, the position and size of each element, cropping and margins, colour, typeface, weight, letter and line spacing, alignment, outline or stroke, shadow, glow, borders and corner treatment, and anything present in one image and absent from the other. State plainly which differences are large enough to read as a different design and which are minor. If they are visually equivalent, say exactly that.\n\nDo not speculate about how either image was produced, which one is a source, or which one is a copy. Return natural language only.` });- const differences = await callSafely(retryDelayMs, generate, parts, "You compare two supplied still images and describe their visible differences in natural language only. You are not told how either image was made. Do not write code, markup, SVML, component names, or production advice.");+ const observer: Observer = state.observer ?? "gemini";+ const { ask, pending } = await askerFor(observer, state);+ const differences = await ask("comparison", {+ media: [shot.representative_frame_ref, imagePath],+ instruction: "You compare two supplied still images and describe their visible differences in natural language only. You are not told how either image was made. Do not write code, markup, SVML, component names, or production advice.",+ prompt: `Two still images are supplied in order: image one, then image two.${scope.length === 0 ? "" : `\n\nLimit the comparison to this part of the picture: ${scope}`}\n\nDescribe every visible difference between them: layout and arrangement, the position and size of each element, cropping and margins, colour, typeface, weight, letter and line spacing, alignment, outline or stroke, shadow, glow, borders and corner treatment, and anything present in one image and absent from the other. State plainly which differences are large enough to read as a different design and which are minor. If they are visually equivalent, say exactly that.\n\nDo not speculate about how either image was produced, which one is a source, or which one is a copy. Return natural language only.`,+ });return {reference_id: state.reference_id,+ observer,shot_id: shot.shot_id,reference_frame_ref: shot.representative_frame_ref,image_path: imagePath,differences,unresolved: differences.status === "complete" ? [] : ["differences"],+ pending_observations: pending,};},++ // The `agent` observer answers in its own context, so the answer comes back through here rather+ // than through a return value. The four whole-reference observations live in the state and the+ // rest in the observation cache, which is where each one was already read from.+ async record_observation(input): Promise<Record<string, unknown>> {+ const state = await loadState(input.reference_id);+ assert(state.observer === "agent", `reference ${input.reference_id} is read by the ${state.observer ?? "gemini"} observer, which records its own answers`);+ const key = input.key.trim();+ assert(key.length > 0, "key is required");+ const text = input.text.trim();+ assert(text.length > 0, "text is required; an observation that saw nothing says so in words");+ const root = stateRoot(workspaceRoot, input.reference_id);+ if (WHOLE_REFERENCE_KEYS.includes(key as typeof WHOLE_REFERENCE_KEYS[number])) {+ const field = key as typeof WHOLE_REFERENCE_KEYS[number];+ await writeJson(join(root, "state.json"), { ...state, [field]: observation("complete", text) });+ return { reference_id: state.reference_id, key, stored_in: "state" };+ }+ const shotKeys = new Set(state.shots.flatMap((shot) => [`visual:${shot.shot_id}`, `type:${shot.shot_id}`, `audio:${shot.shot_id}`, `boundary:${shot.shot_id}`, `window:${shot.shot_id}`]));+ assert(shotKeys.has(key), `key ${key} is not an observation of reference ${input.reference_id}`);+ const path = join(root, "observations.json");+ const cache = await readJson<Record<string, Observation>>(path) ?? {};+ cache[key] = observation("complete", text);+ await writeJson(path, cache);+ return { reference_id: state.reference_id, key, stored_in: "observations" };+ },};}+const WHOLE_REFERENCE_KEYS = ["people_and_product", "voices", "persistent_systems", "places"] as const;+function vocabularyForResult(value: SurfaceVocabulary | undefined): unknown {if (value === undefined) return undefined;return {diff --git a/packages/reference-video-tools/src/types.ts b/packages/reference-video-tools/src/types.tsindex 676469ba..7acb16ae 100644--- a/packages/reference-video-tools/src/types.ts+++ b/packages/reference-video-tools/src/types.ts@@ -1,4 +1,8 @@-export type Observation = { readonly status: "complete" | "failed"; readonly text: string };+/**+ * `pending` is not a failure: it is an observation the `agent` observer has been handed and has not+ * answered yet. It is never cached, so the work survives the process that reported it.+ */+export type Observation = { readonly status: "complete" | "failed" | "pending"; readonly text: string };export type TranscriptWord = {readonly text: string;@@ -41,13 +45,30 @@ export type Shot = {readonly clip_ref: string;readonly representative_frame_ref: string;readonly tail_frame_ref: string;+ /** The shot's frames sampled evenly and tiled into one picture, in reading order. */+ readonly frames_tile_ref: string;readonly audio_tail_ref: string | null;};+/**+ * Who reads the reference. `gemini` uploads video to Vertex; `agent` hands the main agent one tiled+ * picture per shot and takes the answer back. The two produce the same observation keys.+ */+export type Observer = "gemini" | "agent";++/** One observation the `agent` observer still owes, carrying everything needed to answer it. */+export type ObservationTaskRequest = {+ readonly key: string;+ readonly instruction: string;+ readonly prompt: string;+ readonly image_refs: readonly string[];+};+export type ReferenceState = {readonly reference_id: string;readonly video_path: string;readonly root: string;+ readonly observer?: Observer;readonly video: { readonly duration_seconds: number; readonly width: number; readonly height: number; readonly has_audio: boolean };readonly shots: readonly Shot[];readonly storyboard_ref: string;@@ -62,6 +83,9 @@ export type ReferenceState = {export type PrepareResult = {readonly reference_id: string;readonly status: "ready" | "partial";+ readonly observer?: Observer;+ /** Observations the `agent` observer still owes. Empty on the `gemini` observer. */+ readonly pending_observations?: readonly ObservationTaskRequest[];readonly video: ReferenceState["video"];readonly shots: readonly Shot[];readonly storyboard_ref: string;diff --git a/packages/reference-video-tools/test/tools.test.ts b/packages/reference-video-tools/test/tools.test.tsindex e6b9cab0..addd8b0f 100644--- a/packages/reference-video-tools/test/tools.test.ts+++ b/packages/reference-video-tools/test/tools.test.ts@@ -23,7 +23,8 @@ async function workspace(shotCount: number): Promise<{ readonly root: string; reconst clip = join(shots, `${id}.mp4`);const representative = join(shots, `${id}-representative.jpg`);const tail = join(shots, `${id}-tail.jpg`);- for (const path of [clip, representative, tail]) await writeFile(path, `bytes-${id}`, "utf8");+ const framesTile = join(shots, `${id}-frames.jpg`);+ for (const path of [clip, representative, tail, framesTile]) await writeFile(path, `bytes-${id}`, "utf8");media.push({shot_id: `shot-${id}`,index: index + 1,@@ -36,6 +37,7 @@ async function workspace(shotCount: number): Promise<{ readonly root: string; reclip_ref: clip,representative_frame_ref: representative,tail_frame_ref: tail,+ frames_tile_ref: framesTile,audio_tail_ref: null,});}@@ -57,6 +59,17 @@ async function workspace(shotCount: number): Promise<{ readonly root: string; rereturn { root, stateRoot };}+async function agentWorkspace(shotCount: number): Promise<{ readonly root: string; readonly stateRoot: string }> {+ const made = await workspace(shotCount);+ const path = join(made.stateRoot, "state.json");+ const state = JSON.parse(await readFile(path, "utf8")) as Record<string, unknown>;+ // The four whole-reference observations are what the agent observer still owes at this point, so+ // the fixture drops them rather than starting from a reference that was already read.+ for (const key of ["people_and_product", "voices", "persistent_systems", "places"]) delete state[key];+ await writeFile(path, `${JSON.stringify({ ...state, observer: "agent" }, null, 2)}\n`, "utf8");+ return made;+}+function recorder(calls: Call[], answer = "observed"): GenerateText {return async ({ parts, instruction }) => {calls.push({@@ -230,3 +243,71 @@ test("a failed observation is reported as unresolved instead of an empty list",const result = await failing.observe_reference({ reference_id: REFERENCE });assert.deepEqual(result["unresolved"], ["type:shot-001"]);});++test("the agent observer hands every observation out as a task, with pictures, and calls no model", async () => {+ const { root, stateRoot } = await agentWorkspace(3);+ const calls: Call[] = [];+ const result = await tools(root, calls).observe_reference({ reference_id: REFERENCE });++ assert.equal(calls.length, 0, "the agent observer is the model; nothing may be generated on its behalf");+ assert.equal(result["observer"], "agent");+ const pending = result["pending_observations"] as readonly { key: string; prompt: string; image_refs: readonly string[] }[];+ assert.deepEqual(pending.map((task) => task.key).sort(), [+ "audio:shot-001", "audio:shot-002", "audio:shot-003",+ "boundary:shot-002", "boundary:shot-003",+ "type:shot-001", "type:shot-002", "type:shot-003",+ "visual:shot-001", "visual:shot-002", "visual:shot-003",+ ], "the same keys the Gemini observer answers, so everything downstream reads one shape");++ const visual = pending.find((task) => task.key === "visual:shot-002")!;+ assert.deepEqual(visual.image_refs.map((path) => path.split(/[\\/]/).at(-1)), ["002-frames.jpg", "002-representative.jpg", "001-tail.jpg"],+ "the shot's clip is replaced by the tile of its frames; the still evidence is unchanged");+ assert.equal(visual.image_refs.some((path) => path.endsWith(".mp4")), false, "an observer that reads pictures is never handed video");+ assert.match(visual.prompt, /grid of frames is one shot/u, "the tile has to be described, or a grid reads as one picture");+ assert.doesNotMatch(visual.prompt, /cannot hear/u, "a picture question carries no note about sound it never asked for");++ const audio = pending.find((task) => task.key === "audio:shot-002")!;+ assert.match(audio.prompt, /cannot hear this reference/u, "a question that needs sound says the sound is missing rather than inviting a guess");+ assert.equal(audio.image_refs.some((path) => path.endsWith(".wav")), false, "audio has no picture to become");++ const cached = JSON.parse(await readFile(join(stateRoot, "observations.json"), "utf8")) as Record<string, unknown>;+ assert.deepEqual(cached, {}, "a task handed out is not an answer received, so nothing is cached yet");+});++test("a recorded observation completes its key, and the whole-reference ones land in the state", async () => {+ const { root, stateRoot } = await agentWorkspace(3);+ const calls: Call[] = [];+ const kit = tools(root, calls);++ const shot = await kit.record_observation({ reference_id: REFERENCE, key: "visual:shot-002", text: "a flat designed field of colour columns" });+ assert.equal(shot["stored_in"], "observations");+ const whole = await kit.record_observation({ reference_id: REFERENCE, key: "places", text: "one room, one camera position" });+ assert.equal(whole["stored_in"], "state");++ const again = await kit.observe_reference({ reference_id: REFERENCE });+ const pending = (again["pending_observations"] as readonly { key: string }[]).map((task) => task.key);+ assert.equal(pending.includes("visual:shot-002"), false, "an answered observation is not handed out a second time");+ assert.equal(pending.includes("visual:shot-001"), true, "the rest are still owed");+ const observed = (again["shots"] as readonly Record<string, { status: string; text: string }>[])+ .find((entry) => (entry["shot_id"] as unknown as string) === "shot-002")!;+ assert.equal(observed["visual"]!.status, "complete");+ assert.equal(observed["visual"]!.text, "a flat designed field of colour columns");+ assert.equal(observed["audio"]!.status, "pending", "an unanswered observation is pending, which is not a failure");++ const state = JSON.parse(await readFile(join(stateRoot, "state.json"), "utf8")) as Record<string, { text: string }>;+ assert.equal(state["places"]!.text, "one room, one camera position");+ assert.equal(calls.length, 0);+});++test("a reference keeps the observer it was prepared with", async () => {+ const { root } = await agentWorkspace(2);+ const calls: Call[] = [];+ await assert.rejects(+ () => tools(root, calls).record_observation({ reference_id: REFERENCE, key: "visual:shot-001", text: "x" })+ .then(async () => {+ const gemini = await workspace(2);+ return await tools(gemini.root, calls).record_observation({ reference_id: REFERENCE, key: "visual:shot-001", text: "x" });+ }),+ /read by the gemini observer/u,+ "recording an answer against a reference Gemini is reading would mix two kinds of evidence under one key");+});