Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
feat: observe designed fields, drawn type and persistent systems
rpone3adf574ae021
diff --git a/packages/reference-video-tools/README.md b/packages/reference-video-tools/README.mdindex 8dfb220f..ff24abb0 100644--- a/packages/reference-video-tools/README.md+++ b/packages/reference-video-tools/README.md@@ -2,11 +2,12 @@CLI tools for reconstructing a reference video with Hypit.-The package exposes three CLI subcommands: `prepare_reference`, `observe_reference`, and-`inspect_svml_vocabulary`. Each command prints one JSON result to stdout. The first two send narrow-natural-language requests to Gemini with a fixed temperature of `1.0`; they never receive SVML-syntax and never write SVML. The final source files are authored by the calling agent and checked-with the existing `pnpm hypit check` command.+The package exposes four CLI subcommands: `prepare_reference`, `observe_reference`,+`inspect_svml_vocabulary`, and `compare_reconstruction`. Each command prints one JSON result to+stdout. Every command except `inspect_svml_vocabulary` sends narrow natural-language requests to+Gemini with a fixed temperature of `1.0`; they never receive SVML syntax and never write SVML. The+final source files are authored by the calling agent and checked with the existing+`pnpm hypit check` command.Run the CLI from the repository or an installed package:@@ -15,23 +16,43 @@ pnpm hypit-reference-video-tools prepare_reference --video-path ./reference.mp4pnpm hypit-reference-video-tools prepare_reference --video-path ./reference.mp4 --redo peoplepnpm hypit-reference-video-tools observe_reference --reference-id <reference-id>pnpm hypit-reference-video-tools observe_reference --reference-id <reference-id> --shot-id shot-007-pnpm hypit-reference-video-tools observe_reference --reference-id <reference-id> --shot-id shot-007 --shot-id shot-008+pnpm hypit-reference-video-tools observe_reference --reference-id <reference-id> --shot-id shot-007 --question "How thick is the outline on the caption words?"+pnpm hypit-reference-video-tools observe_reference --reference-id <reference-id> --shot-id shot-007 --reobservepnpm hypit-reference-video-tools inspect_svml_vocabulary --package @hypit/media-track --tag Track+pnpm hypit-reference-video-tools compare_reconstruction --reference-id <reference-id> --shot-id shot-007 --image ./rendered.png```For automation, every command also accepts `--input '{"...":"..."}'` with the complete JSON inputobject. This is useful when the calling agent already has a structured request.`prepare_reference` requires a local video path and stores derived media under the project's-gitignored `.hypit/reference-video-tools/` directory. Without `--redo`, completed stages are-reused. Use `--redo media` to rebuild shot media and clear derived observations, or `--redo people`,-`--redo voices`, or `--redo all` to rerun only the selected full-reference analysis stages.--`observe_reference` without shot IDs observes all unfinished work. Supplying one or more shot IDs-forces those shots and their adjacent continuity boundaries to run again; supplying three-continuous shots also requests the three-shot continuity review. The preceding tail frame, audio-tail and full-reference context are always attached automatically. `--question` adds one narrow-follow-up for the selected shots.+gitignored `.hypit/reference-video-tools/` directory. It produces three full-reference observations:+`people_and_product`, `voices`, and `persistent_systems` — the on-screen text and graphic systems+that continue or recur across the whole video, each one's lifetime, and whether its appearance ever+changes. Without `--redo`, completed stages are reused. Use `--redo media` to rebuild shot media and+clear derived observations, or `--redo people`, `--redo voices`, `--redo systems`, or `--redo all` to+rerun only the selected full-reference stages.++`observe_reference` observes every unfinished shot, or only the shots named by `--shot-id`. Each shot+produces three observations — `visual` (base picture, covering content, continuity, whether the frame+is a depicted scene or a flat designed field, and each framed element's inner picture separately from+its frame), `text_appearance` (typeface, weight, size, spacing, colour, stroke, shadow, glow,+per-word emphasis, position and entry of drawn text), and `audio` — plus per-boundary continuity+observations. The preceding tail frame, audio tail and full-reference context are attached+automatically. Completed observations are reused; `--reobserve` is the only way to run a completed+one again.++`--question` is a separate narrow path: it requires one to three `--shot-id` values, answers only+that question from those shots' clips and frames in a single request, and neither reads nor writes+the observation cache.++`compare_reconstruction` sends the named shot's reference frame and a rendered image as an+unlabelled pair and returns a description of their visible differences. It is never told which image+is which, what was built, or how. `--question` narrows it to one region of the picture and nothing+else. Results are not cached, because the rendered side changes on every iteration.++`unresolved` lists the keys of observations that failed. It is not a judgement about evidence+quality: a complete observation that says "unclear" is still complete.Gemini uses the existing Vertex environment variables `GOOGLE_CLOUD_PROJECT` and`GOOGLE_APPLICATION_CREDENTIALS_JSON`; the location defaults to `global` and the model defaults todiff --git a/packages/reference-video-tools/src/cli.ts b/packages/reference-video-tools/src/cli.tsindex c851e9dc..35286214 100644--- a/packages/reference-video-tools/src/cli.ts+++ b/packages/reference-video-tools/src/cli.ts@@ -6,9 +6,11 @@ type Flags = ReadonlyMap<string, string | readonly string[] | boolean>;function usage(): string {return ["Usage:",- " hypit-reference-video-tools prepare_reference --video-path <path> [--redo media|people|voices|all]",- " hypit-reference-video-tools observe_reference --reference-id <id> [--shot-id <id> ...] [--question <text>]",+ " hypit-reference-video-tools prepare_reference --video-path <path> [--redo media|people|voices|systems|all]",+ " hypit-reference-video-tools observe_reference --reference-id <id> [--shot-id <id> ...] [--reobserve]",+ " hypit-reference-video-tools observe_reference --reference-id <id> --shot-id <id> [--shot-id <id> ...] --question <text>"," hypit-reference-video-tools inspect_svml_vocabulary --package <name> [--package <name> ...] [--tag <tag> ...] [--without-previews]",+ " hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> --image <path> [--question <scope>]","","Every command prints one JSON result to stdout. Use --input <json> instead of flags when a complete input object is easier to pass.",].join("\n");@@ -63,10 +65,16 @@ function inputObject(flags: Flags): Record<string, unknown> | undefined {return parsed as Record<string, unknown>;}+// stdout carries exactly one JSON result. Dependencies that announce themselves on stdout would+// corrupt it, so everything they write is moved to stderr and only `report` reaches stdout.+const report = process.stdout.write.bind(process.stdout);+process.stdout.write = ((chunk: string | Uint8Array, ...rest: readonly unknown[]): boolean =>+ (process.stderr.write as (...args: readonly unknown[]) => boolean)(chunk, ...rest)) as typeof process.stdout.write;+async function main(): Promise<void> {const { command, flags } = parse(process.argv.slice(2));if (flags.has("rebuild") || flags.has("refresh")) {- throw new Error("--rebuild and --refresh were removed; use --redo or selected --shot-id values");+ throw new Error("--rebuild and --refresh were removed; use --redo on prepare_reference or --reobserve on observe_reference");}const tools = createReferenceVideoTools();const supplied = inputObject(flags);@@ -76,14 +84,23 @@ async function main(): Promise<void> {video_path: required(flags, "video-path"),...(one(flags, "redo") === undefined ? {} : { redo: one(flags, "redo") }),};- result = await tools.prepare_reference(input as { video_path: string; redo?: "media" | "people" | "voices" | "all" });+ result = await tools.prepare_reference(input as { video_path: string; redo?: "media" | "people" | "voices" | "systems" | "all" });} else if (command === "observe_reference") {const input = supplied ?? {reference_id: required(flags, "reference-id"),...(many(flags, "shot-id").length === 0 ? {} : { shot_ids: many(flags, "shot-id") }),...(one(flags, "question") === undefined ? {} : { question: one(flags, "question") }),+ ...(flags.get("reobserve") === true ? { reobserve: true } : {}),+ };+ result = await tools.observe_reference(input as { reference_id: string; shot_ids?: readonly string[]; question?: string; reobserve?: boolean });+ } else if (command === "compare_reconstruction") {+ const input = supplied ?? {+ reference_id: required(flags, "reference-id"),+ shot_id: required(flags, "shot-id"),+ image_path: required(flags, "image"),+ ...(one(flags, "question") === undefined ? {} : { question: one(flags, "question") }),};- result = await tools.observe_reference(input as { reference_id: string; shot_ids?: readonly string[]; question?: string });+ result = await tools.compare_reconstruction(input as { reference_id: string; shot_id: string; image_path: string; question?: string });} else if (command === "inspect_svml_vocabulary") {const packages = many(flags, "package");const input = supplied ?? {@@ -95,7 +112,7 @@ async function main(): Promise<void> {} else {throw new Error(`unknown command ${command}\n\n${usage()}`);}- process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);+ report(`${JSON.stringify(result, null, 2)}\n`);}main().catch((error: unknown) => {diff --git a/packages/reference-video-tools/src/media.ts b/packages/reference-video-tools/src/media.tsindex 1d1f07cf..0ad60ca8 100644--- a/packages/reference-video-tools/src/media.ts+++ b/packages/reference-video-tools/src/media.ts@@ -110,8 +110,10 @@ export async function prepareMedia(videoPath: string, root: string, duration: nu}));const inputs = bounds.map((_, index) => `[${index}:v]`).join("");const list = bounds.flatMap((_, index) => ["-i", join(shotDir, `${String(index + 1).padStart(3, "0")}-representative.jpg`)]);+ const columns = Math.min(4, bounds.length);+ const rows = Math.ceil(bounds.length / columns);const storyboard = join(root, "storyboard.jpg");- await command("ffmpeg", ["-hide_banner", "-loglevel", "error", "-y", ...list, "-filter_complex", `${inputs}tile=columns=4:padding=8:margin=8`, "-frames:v", "1", storyboard]);+ await command("ffmpeg", ["-hide_banner", "-loglevel", "error", "-y", ...list, "-filter_complex", `${inputs}concat=n=${bounds.length}:v=1:a=0,tile=layout=${columns}x${rows}:padding=8:margin=8`, "-frames:v", "1", storyboard]);return { bounds, storyboard, analysisVideo };}diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.tsindex 9773d1ff..718901f7 100644--- a/packages/reference-video-tools/src/tools.ts+++ b/packages/reference-video-tools/src/tools.ts@@ -20,23 +20,31 @@ import type { Observation, PrepareResult, ReferenceState, Shot } from "./types.jexport type PrepareReferenceInput = {readonly video_path: string;- readonly redo?: "media" | "people" | "voices" | "all";+ readonly redo?: "media" | "people" | "voices" | "systems" | "all";};export type ObserveReferenceInput = {readonly reference_id: string;readonly shot_ids?: readonly string[];readonly question?: string;+ readonly reobserve?: boolean;};export type InspectVocabularyInput = {readonly package_names: readonly string[];readonly tags?: readonly string[];readonly include_previews?: boolean;};+export type CompareReconstructionInput = {+ readonly reference_id: string;+ readonly shot_id: string;+ readonly image_path: string;+ readonly question?: string;+};export type ReferenceVideoTools = {prepare_reference(input: PrepareReferenceInput): Promise<PrepareResult>;observe_reference(input: ObserveReferenceInput): Promise<Record<string, unknown>>;inspect_svml_vocabulary(input: InspectVocabularyInput): Promise<Record<string, unknown>>;+ compare_reconstruction(input: CompareReconstructionInput): Promise<Record<string, unknown>>;};type ToolOptions = {@@ -45,9 +53,10 @@ type ToolOptions = {readonly model?: string;readonly concurrency?: number;readonly launchGapMs?: number;+ readonly retryDelayMs?: number;readonly generate?: GenerateText;};-type GenerateText = (input: { readonly parts: readonly Part[]; readonly instruction: string }) => Promise<string>;+export type GenerateText = (input: { readonly parts: readonly Part[]; readonly instruction: string }) => Promise<string>;type ObservationTask = { readonly key: string; readonly run: () => Promise<Observation> };function observation(status: "complete" | "failed", text: string): Observation { return { status, text }; }@@ -82,11 +91,20 @@ async function defaultGenerate(model: string): Promise<GenerateText> {};}+const MIME_TYPES: Readonly<Record<string, string>> = {+ jpg: "image/jpeg",+ jpeg: "image/jpeg",+ png: "image/png",+ webp: "image/webp",+ wav: "audio/wav",+ mp4: "video/mp4",+};+async function mediaPart(path: string): Promise<Part> {const bytes = await readBytes(path);- const extension = path.toLowerCase().split(".").pop();- const mimeType = extension === "jpg" || extension === "jpeg" ? "image/jpeg"- : extension === "wav" ? "audio/wav" : extension === "mp4" ? "video/mp4" : "application/octet-stream";+ const extension = path.toLowerCase().split(".").pop() ?? "";+ const mimeType = MIME_TYPES[extension];+ assert(mimeType !== undefined, `unsupported media extension for ${path}; supported: ${Object.keys(MIME_TYPES).join(", ")}`);return { inlineData: { mimeType, data: Buffer.from(bytes).toString("base64") } };}@@ -94,14 +112,14 @@ function rateLimited(error: unknown): boolean {return /\b429\b|resource[ _]exhausted|quota|rate limit/iu.test(error instanceof Error ? error.message : String(error));}-async function callSafely(generate: GenerateText, parts: readonly Part[], instruction: string): Promise<Observation> {+async function callSafely(retryDelayMs: number, generate: GenerateText, parts: readonly Part[], instruction: string): Promise<Observation> {let last: unknown;for (let attempt = 1; attempt <= 6; attempt += 1) {try { return observation("complete", await generate({ parts, instruction })); }catch (error) {last = error;if (attempt === 6) break;- const wait = rateLimited(error) ? Math.min(900_000, 45_000 * 2 ** (attempt - 1)) : 2_000 * attempt;+ const wait = rateLimited(error) ? Math.min(900_000, 45_000 * 2 ** (attempt - 1)) : retryDelayMs * attempt;await new Promise((resolveWait) => setTimeout(resolveWait, wait));}}@@ -172,15 +190,22 @@ async function shotFromBound(root: string, index: number, bound: { start: number};}+function prepared(state: ReferenceState): boolean {+ return state.people_and_product?.status === "complete"+ && state.voices?.status === "complete"+ && state.persistent_systems?.status === "complete";+}+function publicPrepare(state: ReferenceState): PrepareResult {return {reference_id: state.reference_id,- status: state.people_and_product?.status === "complete" && state.voices?.status === "complete" ? "ready" : "partial",+ status: prepared(state) ? "ready" : "partial",video: state.video,shots: state.shots,storyboard_ref: state.storyboard_ref,people_and_product: state.people_and_product ?? observation("failed", "not analyzed"),voices: state.voices ?? observation("failed", "not analyzed"),+ persistent_systems: state.persistent_systems ?? observation("failed", "not analyzed"),};}@@ -190,6 +215,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst model = options.model ?? process.env.GEMINI_MODEL?.trim() ?? "gemini-3.1-pro-preview";const concurrency = options.concurrency ?? 2;const gapMs = options.launchGapMs ?? 6_000;+ const retryDelayMs = options.retryDelayMs ?? 2_000;let generatorPromise: Promise<GenerateText> | undefined;const generator = async (): Promise<GenerateText> => generatorPromise ??= options.generate === undefined ? defaultGenerate(model) : Promise.resolve(options.generate);@@ -202,7 +228,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVreturn {async prepare_reference(input): Promise<PrepareResult> {- assert(input.redo === undefined || input.redo === "media" || input.redo === "people" || input.redo === "voices" || input.redo === "all", "redo must be one of: media, people, voices, all");+ assert(input.redo === undefined || input.redo === "media" || input.redo === "people" || input.redo === "voices" || input.redo === "systems" || input.redo === "all", "redo must be one of: media, people, voices, systems, all");const videoPath = resolve(input.video_path);const file = await stat(videoPath).catch(() => undefined);assert(file?.isFile(), `video_path is not a file: ${videoPath}`);@@ -213,9 +239,8 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst redoMedia = input.redo === "media" || input.redo === "all";const redoPeople = input.redo === "people" || input.redo === "all";const redoVoices = input.redo === "voices" || input.redo === "all";- if (input.redo === undefined) {- if (existing !== undefined && existing.people_and_product?.status === "complete" && existing.voices?.status === "complete") return publicPrepare(existing);- }+ const redoSystems = input.redo === "systems" || input.redo === "all";+ if (input.redo === undefined && existing !== undefined && prepared(existing)) return publicPrepare(existing);await ensureDir(root);const info = existing?.video === undefined || redoMedia ? await probe(videoPath) : {duration: existing.video.duration_seconds,@@ -241,6 +266,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVanalysis_video_ref: media.analysisVideo,...(existing?.people_and_product === undefined ? {} : { people_and_product: existing.people_and_product }),...(existing?.voices === undefined ? {} : { voices: existing.voices }),+ ...(existing?.persistent_systems === undefined ? {} : { persistent_systems: existing.persistent_systems }),};analysisVideo = media.analysisVideo;await writeJson(statePath, state);@@ -248,15 +274,18 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV}const generate = await generator();const globalParts: Part[] = [await mediaPart(analysisVideo)];- const [people, voices] = await Promise.all([+ const [people, voices, systems] = await Promise.all([state.people_and_product?.status === "complete" && !redoPeople? Promise.resolve(state.people_and_product)- : callSafely(generate, [...globalParts, { text: "Describe the recurring people and the promoted product in this complete reference video. Return natural language only. Identify stable visual traits and distinguish recurring speakers from incidental people in inserts. Describe the product once, comprehensively, for reuse." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),+ : callSafely(retryDelayMs, generate, [...globalParts, { text: "Describe the recurring people and the promoted product in this complete reference video. Return natural language only. Identify stable visual traits and distinguish recurring speakers from incidental people in inserts. Describe the product once, comprehensively, for reuse." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),state.voices?.status === "complete" && !redoVoices? Promise.resolve(state.voices)- : callSafely(generate, [...globalParts, { text: "Listen to this complete reference video and describe the distinct voices, their order, overlap, off-screen speech, and likely correspondence to visible people. Do not assign a voice merely because a person appears in a B-roll image. Return natural language only." }], "You only listen to a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),+ : callSafely(retryDelayMs, generate, [...globalParts, { text: "Listen to this complete reference video and describe the distinct voices, their order, overlap, off-screen speech, and likely correspondence to visible people. Do not assign a voice merely because a person appears in a B-roll image. Return natural language only." }], "You only listen to a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),+ state.persistent_systems?.status === "complete" && !redoSystems+ ? Promise.resolve(state.persistent_systems)+ : callSafely(retryDelayMs, generate, [...globalParts, { text: "Describe the on-screen text and graphic systems that persist or recur across this complete reference video, such as subtitles, running lists, counters, progress indicators, badges, watermarks, lower thirds and repeating full-screen graphic layouts. For each one, state when it first appears and when it stops, whether it is present continuously or intermittently, whether its own appearance stays the same throughout, and describe any point where its appearance actually changes. Report only what stays consistent across the video; ignore one-off elements that appear a single time. Return natural language only." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),]);- const complete: ReferenceState = { ...state, people_and_product: people, voices };+ const complete: ReferenceState = { ...state, people_and_product: people, voices, persistent_systems: systems };await writeJson(statePath, complete);return publicPrepare(complete);},@@ -265,53 +294,75 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst state = await loadState(input.reference_id);const selected = input.shot_ids === undefined ? state.shots : state.shots.filter((shot) => input.shot_ids!.includes(shot.shot_id));assert(selected.length > 0, "no requested shot ids exist");+ const generate = await generator();+ const question = input.question?.trim() ?? "";+ if (question.length > 0) {+ assert(input.shot_ids !== undefined && input.shot_ids.length > 0, "question requires at least one shot id, so that it is answered from the shots it is about");+ assert(selected.length <= 3, "question accepts at most three shots");+ const parts: Part[] = [];+ for (const shot of selected) {+ parts.push(await mediaPart(shot.clip_ref));+ parts.push(await mediaPart(shot.representative_frame_ref));+ }+ parts.push({ text: question });+ const answer = await callSafely(retryDelayMs, generate, parts, "Answer only the narrow reference-video question in natural language. Do not write code, markup, SVML, or component names.");+ return {+ reference_id: state.reference_id,+ shot_ids: selected.map((shot) => shot.shot_id),+ question,+ answer,+ unresolved: answer.status === "complete" ? [] : ["question"],+ };+ }const selectedIds = new Set(selected.map((shot) => shot.shot_id));const boundaryRights = state.shots.filter((shot) => {const left = state.shots.find((candidate) => candidate.index === shot.index - 1);return left !== undefined && (selectedIds.has(left.shot_id) || selectedIds.has(shot.shot_id));});- const generate = await generator();const globalContext = [state.people_and_product?.text === undefined ? "" : `Full-reference people and product evidence:\n${state.people_and_product.text}`,state.voices?.text === undefined ? "" : `Full-reference voice evidence:\n${state.voices.text}`,+ state.persistent_systems?.text === undefined ? "" : `Full-reference persistent on-screen system evidence:\n${state.persistent_systems.text}`,].filter((item) => item.length > 0).join("\n\n");const visualTasks: ObservationTask[] = selected.map((shot) => ({ key: `visual:${shot.shot_id}`, run: async () => {const previous = state.shots.find((candidate) => candidate.index === shot.index - 1);const parts: Part[] = [await mediaPart(shot.clip_ref), await mediaPart(shot.representative_frame_ref)];if (previous !== undefined) parts.push(await mediaPart(previous.tail_frame_ref));- parts.push({ text: `${globalContext}\n\nObserve shot ${shot.index}. Describe the current base picture, any covering or non-covering visual content, and whether visible content continues from the preceding shot. A full-screen insert is still only a picture observation. Return natural language evidence only.` });- return await callSafely(generate, parts, "Observe picture only. Do not choose SVML components, do not write markup, and do not decide final source syntax.");+ parts.push({ text: `${globalContext}\n\nObserve shot ${shot.index}. Describe the current base picture, any covering or non-covering visual content, and whether visible content continues from the preceding shot. A full-screen insert is still only a picture observation.\n\nAlso answer these two questions explicitly.\n\nFirst: is the whole frame a depicted scene that has its own camera space, lighting, depth and lens behaviour, or is it a flat designed field whose purpose is to carry drawn elements such as words, rows, panels or cards? Live action and animation are both depicted scenes; a paper sheet, ruled or gridded surface, flat or gradient colour, blurred wallpaper, board or slide backdrop filling the frame is a designed field. State which one it is and the visible evidence for it.\n\nSecond: for every framed element inside the picture, such as a card, phone, browser window, screenshot or inset, describe the picture inside the frame and the frame itself separately. For the inside, describe what it depicts and whether it moves. For the frame, describe its border, corner radius, outline, shadow, size, position and how it enters and leaves.\n\nReturn natural language evidence only.` });+ return await callSafely(retryDelayMs, generate, parts, "Observe picture only. Do not choose SVML components, do not write markup, and do not decide final source syntax.");+ }}));+ const typeTasks: ObservationTask[] = selected.map((shot) => ({ key: `type:${shot.shot_id}`, run: async () => {+ const parts: Part[] = [await mediaPart(shot.representative_frame_ref), await mediaPart(shot.clip_ref)];+ parts.push({ text: `Observe the on-screen text in shot ${shot.index}. Report only text that is drawn over or composed into the picture; ignore text belonging to a photographed object such as a device screen, sign, label or document. If the shot shows no drawn text, say exactly that and stop.\n\nFor each distinct text element describe: the typeface character (serif, sans-serif, handwritten, monospaced or display), the weight, the cap height as a fraction of the frame height, letter spacing and line spacing, alignment, letter case, fill colour, any outline or stroke with its colour and thickness relative to the stroke width of the letters, any drop shadow with its direction, distance and softness, any glow or blur, any emphasis applied to individual words or characters and how it differs from the rest, the position within the frame and the distance from the nearest edges, and how the text enters, changes and leaves during the shot.\n\nReturn natural language evidence only.` });+ return await callSafely(retryDelayMs, generate, parts, "Observe the appearance of on-screen text only. Do not name SVML components, do not write markup, and do not name font files or style properties from any software.");}}));const audioTasks: ObservationTask[] = selected.map((shot) => ({ key: `audio:${shot.shot_id}`, run: async () => {const previous = state.shots.find((candidate) => candidate.index === shot.index - 1);const parts: Part[] = [await mediaPart(shot.clip_ref)];if (previous?.audio_tail_ref !== null && previous?.audio_tail_ref !== undefined) parts.push(await mediaPart(previous.audio_tail_ref));parts.push({ text: `${globalContext}\n\nListen to shot ${shot.index}. Decide who is speaking, whether sound continues from the previous shot, whether visible people actually produce the sound, and whether a silent B-roll person only appears to speak. Handle off-screen, alternating, overlapping, and no-person-visible speech. Return natural language evidence only.` });- return await callSafely(generate, parts, "Observe sound only. Do not write markup, SVML, JSON plans, or component names.");+ return await callSafely(retryDelayMs, generate, parts, "Observe sound only. Do not write markup, SVML, JSON plans, or component names.");}}));const sameTakeTasks: ObservationTask[] = boundaryRights.map((right) => ({ key: `same-take:${right.shot_id}`, run: async () => {const left = state.shots.find((shot) => shot.index === right.index - 1)!;const parts: Part[] = [await mediaPart(left.clip_ref), await mediaPart(right.clip_ref), await mediaPart(left.tail_frame_ref)];parts.push({ text: `Compare shots ${left.index} and ${right.index}. Answer only whether they are one continuous camera shot. Explain the visual and sound continuity evidence in natural language.` });- return await callSafely(generate, parts, "Analyze camera continuity only. Do not name SVML components or write code.");+ return await callSafely(retryDelayMs, generate, parts, "Analyze camera continuity only. Do not name SVML components or write code.");}}));const overlayTasks: ObservationTask[] = boundaryRights.map((right) => ({ key: `overlay:${right.shot_id}`, run: async () => {const left = state.shots.find((shot) => shot.index === right.index - 1)!;const parts: Part[] = [await mediaPart(left.clip_ref), await mediaPart(right.clip_ref), await mediaPart(left.tail_frame_ref)];parts.push({ text: `Compare shots ${left.index} and ${right.index}. Answer only whether the same visible overlay or inserted picture continues across the boundary, changes, or ends. Explain the evidence in natural language.` });- return await callSafely(generate, parts, "Analyze overlay continuity only. Do not name SVML components or write code.");+ return await callSafely(retryDelayMs, generate, parts, "Analyze overlay continuity only. Do not name SVML components or write code.");}}));- const allTasks = [...visualTasks, ...audioTasks, ...sameTakeTasks, ...overlayTasks];- const forcedKeys = input.shot_ids === undefined- ? new Set<string>()- : new Set(allTasks- .filter((task) => selectedIds.has(task.key.split(":")[1]!) || boundaryRights.some((right) => task.key.endsWith(`:${right.shot_id}`)))- .map((task) => task.key));+ const allTasks = [...visualTasks, ...typeTasks, ...audioTasks, ...sameTakeTasks, ...overlayTasks];+ const reobserve = input.reobserve === true;+ const forcedKeys = reobserve ? new Set(allTasks.map((task) => task.key)) : new Set<string>();const observations = await runObservationTasks(state.root, allTasks, forcedKeys, concurrency, gapMs);- const unresolved: string[] = [];+ const unresolved = [...observations].filter(([, value]) => value.status !== "complete").map(([key]) => key);const boundaryText = boundaryRights.map((right) => `${observations.get(`same-take:${right.shot_id}`)?.text ?? ""}\n${observations.get(`overlay:${right.shot_id}`)?.text ?? ""}`).join("\n");const needsThreeShotReview = selected.length >= 3 && (- input.shot_ids !== undefined ||+ reobserve ||/uncertain|unknown|possibly|may be|contin(?:ue|ues)|same take|same shot/iu.test(boundaryText));const windows = (needsThreeShotReview@@ -326,10 +377,18 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst parts: Part[] = [];for (const shot of items) parts.push(await mediaPart(shot.clip_ref));parts.push({ text: `Compare shots ${items[0].index}, ${items[1].index}, and ${items[2].index}. Decide whether the three clips are one continuous camera shot and whether one visual overlay or insert persists through both boundaries. Explain the evidence in natural language only.` });- const result = await callSafely(generate, parts, "Analyze a three-shot continuity window only. Do not write markup, SVML, JSON plans, or component names.");+ const result = await callSafely(retryDelayMs, generate, parts, "Analyze a three-shot continuity window only. Do not write markup, SVML, JSON plans, or component names.");return { shot_ids: items.map((shot) => shot.shot_id), combined_duration_seconds: Number((items[2].end_seconds - items[0].start_seconds).toFixed(3)), result };});- const shotResults = selected.map((shot) => ({ shot_id: shot.shot_id, visual: observations.get(`visual:${shot.shot_id}`)!, audio: observations.get(`audio:${shot.shot_id}`)! }));+ for (const window of threeShotObservations) {+ if (window.result.status !== "complete") unresolved.push(`three-shot:${window.shot_ids.join("+")}`);+ }+ const shotResults = selected.map((shot) => ({+ shot_id: shot.shot_id,+ visual: observations.get(`visual:${shot.shot_id}`)!,+ text_appearance: observations.get(`type:${shot.shot_id}`)!,+ audio: observations.get(`audio:${shot.shot_id}`)!,+ }));const boundaryResults = boundaryRights.map((right) => {const left = state.shots.find((shot) => shot.index === right.index - 1)!;return {@@ -341,19 +400,11 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVoverlay_continuity: observations.get(`overlay:${right.shot_id}`)!,};});- if (input.question !== undefined && input.question.trim().length > 0) {- const parts: Part[] = [];- for (const shot of selected.slice(0, 3)) parts.push(await mediaPart(shot.clip_ref));- parts.push({ text: input.question });- const follow = await callSafely(generate, parts, "Answer only the user's narrow reference-video question in natural language. Do not write code, markup, SVML, or component names.");- return { reference_id: state.reference_id, shots: shotResults, boundaries: boundaryResults, three_shot_observations: threeShotObservations, follow_ups: [{ shot_ids: selected.slice(0, 3).map((shot) => shot.shot_id), question: input.question, text: follow.text }], unresolved };- }return {reference_id: state.reference_id,shots: shotResults,boundaries: boundaryResults,three_shot_observations: threeShotObservations,- follow_ups: [],unresolved,};},@@ -381,6 +432,28 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV}return { packages: input.package_names, surfaces };},++ async compare_reconstruction(input): Promise<Record<string, unknown>> {+ const state = await loadState(input.reference_id);+ const shot = state.shots.find((candidate) => candidate.shot_id === input.shot_id);+ assert(shot !== undefined, `shot ${input.shot_id} does not exist in reference ${input.reference_id}`);+ const imagePath = resolve(input.image_path);+ const file = await stat(imagePath).catch(() => undefined);+ assert(file?.isFile(), `image_path is not a file: ${imagePath}`);+ const scope = input.question?.trim() ?? "";+ const generate = await generator();+ const parts: Part[] = [await mediaPart(shot.representative_frame_ref), await mediaPart(imagePath)];+ parts.push({ text: `Two still images are supplied in order: image one, then image two.${scope.length === 0 ? "" : `\n\nLimit the comparison to this part of the picture: ${scope}`}\n\nDescribe every visible difference between them: layout and arrangement, the position and size of each element, cropping and margins, colour, typeface, weight, letter and line spacing, alignment, outline or stroke, shadow, glow, borders and corner treatment, and anything present in one image and absent from the other. State plainly which differences are large enough to read as a different design and which are minor. If they are visually equivalent, say exactly that.\n\nDo not speculate about how either image was produced, which one is a source, or which one is a copy. Return natural language only.` });+ const differences = await callSafely(retryDelayMs, generate, parts, "You compare two supplied still images and describe their visible differences in natural language only. You are not told how either image was made. Do not write code, markup, SVML, component names, or production advice.");+ return {+ reference_id: state.reference_id,+ shot_id: shot.shot_id,+ reference_frame_ref: shot.representative_frame_ref,+ image_path: imagePath,+ differences,+ unresolved: differences.status === "complete" ? [] : ["differences"],+ };+ },};}diff --git a/packages/reference-video-tools/src/types.ts b/packages/reference-video-tools/src/types.tsindex 09cd6b94..559973a6 100644--- a/packages/reference-video-tools/src/types.ts+++ b/packages/reference-video-tools/src/types.ts@@ -25,6 +25,7 @@ export type ReferenceState = {readonly analysis_video_ref: string;readonly people_and_product?: Observation;readonly voices?: Observation;+ readonly persistent_systems?: Observation;};export type PrepareResult = {@@ -35,4 +36,5 @@ export type PrepareResult = {readonly storyboard_ref: string;readonly people_and_product: Observation;readonly voices: Observation;+ readonly persistent_systems: Observation;};diff --git a/packages/reference-video-tools/test/tools.test.ts b/packages/reference-video-tools/test/tools.test.tsnew file mode 100644index 00000000..b3d03813--- /dev/null+++ b/packages/reference-video-tools/test/tools.test.ts@@ -0,0 +1,190 @@+import assert from "node:assert/strict";+import test from "node:test";+import { mkdtemp, mkdir, writeFile, readFile } from "node:fs/promises";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { createReferenceVideoTools } from "../src/tools.js";+import type { GenerateText } from "../src/tools.js";+import type { ReferenceState } from "../src/types.js";++type Call = { readonly instruction: string; readonly mimeTypes: readonly string[]; readonly text: string };++const REFERENCE = "ref-fixture";++async function workspace(shotCount: number): Promise<{ readonly root: string; readonly stateRoot: string }> {+ const root = await mkdtemp(join(tmpdir(), "reference-video-tools-"));+ const stateRoot = join(root, ".hypit", "reference-video-tools", REFERENCE);+ const shots = join(stateRoot, "shots");+ await mkdir(shots, { recursive: true });+ const media: ReferenceState["shots"][number][] = [];+ for (let index = 0; index < shotCount; index += 1) {+ const id = String(index + 1).padStart(3, "0");+ const clip = join(shots, `${id}.mp4`);+ const representative = join(shots, `${id}-representative.jpg`);+ const tail = join(shots, `${id}-tail.jpg`);+ for (const path of [clip, representative, tail]) await writeFile(path, `bytes-${id}`, "utf8");+ media.push({+ shot_id: `shot-${id}`,+ index: index + 1,+ start_seconds: index,+ end_seconds: index + 1,+ duration_seconds: 1,+ initial_group: index,+ part: 1,+ parts: 1,+ clip_ref: clip,+ representative_frame_ref: representative,+ tail_frame_ref: tail,+ audio_tail_ref: null,+ });+ }+ const state: ReferenceState = {+ reference_id: REFERENCE,+ video_path: join(root, "reference.mp4"),+ root: stateRoot,+ video: { duration_seconds: shotCount, width: 360, height: 640, has_audio: true },+ shots: media,+ storyboard_ref: join(stateRoot, "storyboard.jpg"),+ analysis_video_ref: join(stateRoot, "analysis.mp4"),+ people_and_product: { status: "complete", text: "one recurring host" },+ voices: { status: "complete", text: "one voice" },+ persistent_systems: { status: "complete", text: "one caption system for the whole video" },+ };+ await writeFile(join(stateRoot, "state.json"), `${JSON.stringify(state, null, 2)}\n`, "utf8");+ await writeFile(join(stateRoot, "observations.json"), "{}\n", "utf8");+ return { root, stateRoot };+}++function recorder(calls: Call[], answer = "observed"): GenerateText {+ return async ({ parts, instruction }) => {+ calls.push({+ instruction,+ mimeTypes: parts.flatMap((part) => part.inlineData?.mimeType === undefined ? [] : [part.inlineData.mimeType]),+ text: parts.flatMap((part) => part.text === undefined ? [] : [part.text]).join("\n"),+ });+ return answer;+ };+}++function tools(root: string, calls: Call[], answer?: string) {+ return createReferenceVideoTools({ workspaceRoot: root, concurrency: 4, launchGapMs: 0, retryDelayMs: 0, generate: recorder(calls, answer) });+}++test("observation covers picture, drawn type and sound for every shot and caches every key", async () => {+ const { root, stateRoot } = await workspace(2);+ const calls: Call[] = [];+ const result = await tools(root, calls).observe_reference({ reference_id: REFERENCE });++ const cache = JSON.parse(await readFile(join(stateRoot, "observations.json"), "utf8")) as Record<string, unknown>;+ assert.deepEqual(Object.keys(cache).sort(), [+ "audio:shot-001", "audio:shot-002",+ "overlay:shot-002",+ "same-take:shot-002",+ "type:shot-001", "type:shot-002",+ "visual:shot-001", "visual:shot-002",+ ]);++ const shots = result["shots"] as readonly Record<string, unknown>[];+ assert.equal(shots.length, 2);+ assert.deepEqual(Object.keys(shots[0]!).sort(), ["audio", "shot_id", "text_appearance", "visual"]);+ assert.deepEqual(result["unresolved"], []);++ const picture = calls.find((call) => call.instruction.startsWith("Observe picture only"))!;+ assert.match(picture.text, /depicted scene/u);+ assert.match(picture.text, /flat designed field/u);+ assert.match(picture.text, /the picture inside the frame and the frame itself separately/u);+ assert.match(picture.text, /persistent on-screen system evidence/u);++ const type = calls.find((call) => call.instruction.startsWith("Observe the appearance of on-screen text"))!;+ assert.match(type.text, /outline or stroke/u);+ assert.match(type.text, /drop shadow/u);+ assert.match(type.text, /typeface character/u);+});++test("a second observation reuses the cache and only an explicit reobserve runs the model again", async () => {+ const { root } = await workspace(2);+ const first: Call[] = [];+ await tools(root, first).observe_reference({ reference_id: REFERENCE });+ assert.equal(first.length, 8);++ const second: Call[] = [];+ await tools(root, second).observe_reference({ reference_id: REFERENCE });+ assert.deepEqual(second, []);++ const selected: Call[] = [];+ await tools(root, selected).observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-002"] });+ assert.deepEqual(selected, [], "naming a shot must not silently re-run its cached observations");++ const forced: Call[] = [];+ await tools(root, forced).observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-002"], reobserve: true });+ assert.equal(forced.length, 5, "reobserve reruns the selected shot's picture, type and sound plus its boundary pair, and nothing else");+});++test("a narrow question costs one call, answers from the named shots and never re-runs observations", async () => {+ const { root, stateRoot } = await workspace(4);+ const calls: Call[] = [];+ const result = await tools(root, calls, "the caption sits above the lower edge")+ .observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-003"], question: "How thick is the caption outline?" });++ assert.equal(calls.length, 1);+ assert.deepEqual(calls[0]!.mimeTypes, ["video/mp4", "image/jpeg"]);+ assert.equal(calls[0]!.text, "How thick is the caption outline?");+ assert.deepEqual(result["shot_ids"], ["shot-003"]);+ assert.deepEqual(result["answer"], { status: "complete", text: "the caption sits above the lower edge" });+ assert.equal(await readFile(join(stateRoot, "observations.json"), "utf8"), "{}\n");++ await assert.rejects(+ tools(root, []).observe_reference({ reference_id: REFERENCE, question: "which shot?" }),+ /question requires at least one shot id/u);+ await assert.rejects(+ tools(root, []).observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-001", "shot-002", "shot-003", "shot-004"], question: "which shot?" }),+ /at most three shots/u);+});++test("reconstruction comparison sends an unlabelled pair and accepts rendered PNG frames", async () => {+ const { root } = await workspace(1);+ const rendered = join(root, "rendered.png");+ await writeFile(rendered, "rendered-bytes", "utf8");+ const calls: Call[] = [];+ const result = await tools(root, calls, "the list starts lower in one image")+ .compare_reconstruction({ reference_id: REFERENCE, shot_id: "shot-001", image_path: rendered, question: "only the full-screen list area" });++ assert.equal(calls.length, 1);+ assert.deepEqual(calls[0]!.mimeTypes, ["image/jpeg", "image/png"]);+ assert.match(calls[0]!.text, /image one, then image two/u);+ assert.match(calls[0]!.text, /only the full-screen list area/u);+ assert.doesNotMatch(calls[0]!.text, /reconstruction|rendered|generated|authored|component|SVML/iu,+ "the comparison must never say which image was built or how");+ assert.match(calls[0]!.instruction, /You are not told how either image was made/u);+ assert.deepEqual(result["differences"], { status: "complete", text: "the list starts lower in one image" });++ await assert.rejects(+ tools(root, []).compare_reconstruction({ reference_id: REFERENCE, shot_id: "shot-009", image_path: rendered }),+ /shot shot-009 does not exist/u);+});++test("an unsupported media extension fails loudly instead of being sent as opaque bytes", async () => {+ const { root } = await workspace(1);+ const rendered = join(root, "rendered.tiff");+ await writeFile(rendered, "rendered-bytes", "utf8");+ await assert.rejects(+ tools(root, []).compare_reconstruction({ reference_id: REFERENCE, shot_id: "shot-001", image_path: rendered }),+ /unsupported media extension/u);+});++test("a failed observation is reported as unresolved instead of an empty list", async () => {+ const { root } = await workspace(1);+ const failing = createReferenceVideoTools({+ workspaceRoot: root,+ concurrency: 4,+ launchGapMs: 0,+ retryDelayMs: 0,+ generate: async ({ instruction }) => {+ if (instruction.startsWith("Observe the appearance of on-screen text")) throw new Error("model refused");+ return "observed";+ },+ });+ const result = await failing.observe_reference({ reference_id: REFERENCE });+ assert.deepEqual(result["unresolved"], ["type:shot-001"]);+});