Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
feat: observe where the video was shot, and whether the picture moves
rpone97a388aacacd
diff --git a/.agents/skills/hypit/references/reference-video/workflow.md b/.agents/skills/hypit/references/reference-video/workflow.mdindex 02767b72..df52ddc5 100644--- a/.agents/skills/hypit/references/reference-video/workflow.md+++ b/.agents/skills/hypit/references/reference-video/workflow.md@@ -34,9 +34,12 @@ is missing.## prepare_reference`prepare_reference` deterministically prepares clips, representative frames, tail frames, audio tails-and storyboard context, then obtains three whole-reference observations: `people_and_product`,-`voices`, and `persistent_systems` — the on-screen text and graphic systems that continue or recur-across the whole video, each one's lifetime, and whether its appearance ever changes.+and storyboard context, then obtains four whole-reference observations. `people_and_product` and `voices` cover who recurs.+`persistent_systems` covers the on-screen text and graphic systems that continue or recur across the+whole video, each one's lifetime, and whether its appearance ever changes. `places` covers how many+locations the video was shot in, which camera positions appear in each, which parts of the video use+each one, and each position described in enough detail to draw from the words alone — which is what+reconstructing a location depends on, since reference frames are never fed to generation.When a completed preparation stage must be rerun, use one small `--redo` value:@@ -45,7 +48,7 @@ hypit-reference-video-tools prepare_reference --video-path <path> --redo people````--redo media` rebuilds shot media and clears every derived observation. `--redo people`,-`--redo voices` and `--redo systems` rerun one whole-reference stage. `--redo all` reruns everything.+`--redo voices`, `--redo systems` and `--redo places` rerun one whole-reference stage. `--redo all` reruns everything.## observe_reference@@ -59,6 +62,9 @@ selected shots. Each shot produces three observations:shadow, glow, per-word emphasis, position and entry of text drawn over the picture;- `audio` — who is speaking, and what continues from the previous shot.+The picture observation also says whether the picture moves and how, separating camera movement from+movement inside the frame. A held still and a moving shot are reconstructed differently.+Each cut produces one `continuity` observation answering both questions under their own headings —whether the two shots are one continuous camera shot, and whether an overlay continues across theboundary — and a window of three consecutive shots triggers a three-shot continuity review. Thediff --git a/packages/reference-video-tools/README.md b/packages/reference-video-tools/README.mdindex d6290816..d3e6b0c1 100644--- a/packages/reference-video-tools/README.md+++ b/packages/reference-video-tools/README.md@@ -32,18 +32,21 @@ is deliberately unable to scan a directory, so this is how a caller finds out whinstead of working from memory.`prepare_reference` requires a local video path and stores derived media under the project's-gitignored `.hypit/reference-video-tools/` directory. It produces three full-reference observations:-`people_and_product`, `voices`, and `persistent_systems` — the on-screen text and graphic systems-that continue or recur across the whole video, each one's lifetime, and whether its appearance ever-changes. Without `--redo`, completed stages are reused. Use `--redo media` to rebuild shot media and-clear derived observations, or `--redo people`, `--redo voices`, `--redo systems`, or `--redo all` to+gitignored `.hypit/reference-video-tools/` directory. It produces four full-reference observations: `people_and_product`,+`voices`, `persistent_systems` — the on-screen text and graphic systems that continue or recur across+the whole video, each one's lifetime, and whether its appearance ever changes — and `places`: how+many locations the video was shot in, which camera positions appear in each, which parts of the video+use each one, and each position described in enough detail to draw from the words alone. Without `--redo`, completed stages are reused. Use `--redo media` to rebuild shot media and+clear derived observations, or `--redo people`, `--redo voices`, `--redo systems`, `--redo places`, or `--redo all` torerun only the selected full-reference stages.`observe_reference` observes every unfinished shot, or only the shots named by `--shot-id`. Each shotproduces three observations — `visual` (base picture, covering content, continuity, whether the frameis a depicted scene or a flat designed field, and each framed element's inner picture separately fromits frame), `text_appearance` (typeface, weight, size, spacing, colour, stroke, shadow, glow,-per-word emphasis, position and entry of drawn text), and `audio`. Each cut adds one `continuity`+per-word emphasis, position and entry of drawn text), and `audio`. The picture observation also+states whether the picture moves and how, separating camera movement from movement within it, since a+held still and a moving shot are reconstructed differently. Each cut adds one `continuity`observation covering both whether the two shots are one continuous camera shot and whether an overlaycontinues across the boundary. The preceding tail frame, audio tail and full-reference context areattached automatically. Completed observations are reused; `--reobserve` is the only way to run adiff --git a/packages/reference-video-tools/src/cli.ts b/packages/reference-video-tools/src/cli.tsindex 721218da..c28fc3c0 100644--- a/packages/reference-video-tools/src/cli.ts+++ b/packages/reference-video-tools/src/cli.ts@@ -7,7 +7,7 @@ function usage(): string {return ["Usage:"," hypit-reference-video-tools list_svml_packages",- " hypit-reference-video-tools prepare_reference --video-path <path> [--redo media|people|voices|systems|all]",+ " hypit-reference-video-tools prepare_reference --video-path <path> [--redo media|people|voices|systems|places|all]"," hypit-reference-video-tools observe_reference --reference-id <id> [--shot-id <id> ...] [--reobserve]"," hypit-reference-video-tools observe_reference --reference-id <id> --shot-id <id> [--shot-id <id> ...] --question <text>"," hypit-reference-video-tools inspect_svml_vocabulary --package <name> [--package <name> ...] [--tag <tag> ...] [--without-previews]",@@ -87,7 +87,7 @@ async function main(): Promise<void> {video_path: required(flags, "video-path"),...(one(flags, "redo") === undefined ? {} : { redo: one(flags, "redo") }),};- result = await tools.prepare_reference(input as { video_path: string; redo?: "media" | "people" | "voices" | "systems" | "all" });+ result = await tools.prepare_reference(input as { video_path: string; redo?: "media" | "people" | "voices" | "systems" | "places" | "all" });} else if (command === "observe_reference") {const input = supplied ?? {reference_id: required(flags, "reference-id"),diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.tsindex 05f10709..b847aff5 100644--- a/packages/reference-video-tools/src/tools.ts+++ b/packages/reference-video-tools/src/tools.ts@@ -22,7 +22,7 @@ import type { Observation, PrepareResult, ReferenceState, Shot } from "./types.jexport type PrepareReferenceInput = {readonly video_path: string;- readonly redo?: "media" | "people" | "voices" | "systems" | "all";+ readonly redo?: "media" | "people" | "voices" | "systems" | "places" | "all";};export type ObserveReferenceInput = {readonly reference_id: string;@@ -227,7 +227,8 @@ async function shotFromBound(root: string, index: number, bound: { start: numberfunction prepared(state: ReferenceState): boolean {return state.people_and_product?.status === "complete"&& state.voices?.status === "complete"- && state.persistent_systems?.status === "complete";+ && state.persistent_systems?.status === "complete"+ && state.places?.status === "complete";}function publicPrepare(state: ReferenceState): PrepareResult {@@ -240,6 +241,7 @@ function publicPrepare(state: ReferenceState): PrepareResult {people_and_product: state.people_and_product ?? observation("failed", "not analyzed"),voices: state.voices ?? observation("failed", "not analyzed"),persistent_systems: state.persistent_systems ?? observation("failed", "not analyzed"),+ places: state.places ?? observation("failed", "not analyzed"),};}@@ -308,7 +310,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV},async prepare_reference(input): Promise<PrepareResult> {- assert(input.redo === undefined || input.redo === "media" || input.redo === "people" || input.redo === "voices" || input.redo === "systems" || input.redo === "all", "redo must be one of: media, people, voices, systems, all");+ assert(input.redo === undefined || input.redo === "media" || input.redo === "people" || input.redo === "voices" || input.redo === "systems" || input.redo === "places" || input.redo === "all", "redo must be one of: media, people, voices, systems, places, all");const videoPath = resolve(input.video_path);const file = await stat(videoPath).catch(() => undefined);assert(file?.isFile(), `video_path is not a file: ${videoPath}`);@@ -320,6 +322,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst redoPeople = input.redo === "people" || input.redo === "all";const redoVoices = input.redo === "voices" || input.redo === "all";const redoSystems = input.redo === "systems" || input.redo === "all";+ const redoPlaces = input.redo === "places" || input.redo === "all";if (input.redo === undefined && existing !== undefined && prepared(existing)) return publicPrepare(existing);await ensureDir(root);const info = existing?.video === undefined || redoMedia ? await probe(videoPath) : {@@ -347,6 +350,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV...(existing?.people_and_product === undefined ? {} : { people_and_product: existing.people_and_product }),...(existing?.voices === undefined ? {} : { voices: existing.voices }),...(existing?.persistent_systems === undefined ? {} : { persistent_systems: existing.persistent_systems }),+ ...(existing?.places === undefined ? {} : { places: existing.places }),};analysisVideo = media.analysisVideo;await writeJson(statePath, state);@@ -355,7 +359,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst generate = await generator();const mediaPart = mediaParts();const globalParts: Part[] = [await mediaPart(analysisVideo)];- const [people, voices, systems] = await Promise.all([+ const [people, voices, systems, places] = await Promise.all([state.people_and_product?.status === "complete" && !redoPeople? Promise.resolve(state.people_and_product): callSafely(retryDelayMs, generate, [...globalParts, { text: "Describe the recurring people and the promoted product in this complete reference video. Return natural language only. Identify stable visual traits and distinguish recurring speakers from incidental people in inserts. Describe the product once, comprehensively, for reuse." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),@@ -365,8 +369,11 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVstate.persistent_systems?.status === "complete" && !redoSystems? Promise.resolve(state.persistent_systems): callSafely(retryDelayMs, generate, [...globalParts, { text: "Describe the on-screen text and graphic systems that persist or recur across this complete reference video, such as subtitles, running lists, counters, progress indicators, badges, watermarks, lower thirds and repeating full-screen graphic layouts. For each one, state when it first appears and when it stops, whether it is present continuously or intermittently, whether its own appearance stays the same throughout, and describe any point where its appearance actually changes. Report only what stays consistent across the video; ignore one-off elements that appear a single time. Return natural language only." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),+ state.places?.status === "complete" && !redoPlaces+ ? Promise.resolve(state.places)+ : callSafely(retryDelayMs, generate, [...globalParts, { text: "Describe every distinct place this reference video was shot in, and every distinct camera position within each place. State how many places there are, which parts of the video happen in each, and for each place which camera positions appear and which parts of the video use each one. Two shots are the same camera position when the camera sees the same part of the room from the same side; a reverse angle is a different position of the same place.\n\nDescribe each camera position in enough detail that someone who has never seen this video could draw it from your words alone: what is behind and beside the subject, the shape and depth of the space, where the light comes from and how hard it is, the colours and materials of the surfaces, and the objects a viewer would use to recognise it again. Say what stays identical between positions of one place and what differs.\n\nReturn natural language only." }], "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names."),]);- const complete: ReferenceState = { ...state, people_and_product: people, voices, persistent_systems: systems };+ const complete: ReferenceState = { ...state, people_and_product: people, voices, persistent_systems: systems, places };await writeJson(statePath, complete);return publicPrepare(complete);},@@ -410,7 +417,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVconst previous = state.shots.find((candidate) => candidate.index === shot.index - 1);const parts: Part[] = [await mediaPart(shot.clip_ref), await mediaPart(shot.representative_frame_ref)];if (previous !== undefined) parts.push(await mediaPart(previous.tail_frame_ref));- parts.push({ text: `${globalContext}\n\nObserve shot ${shot.index}. Describe the current base picture, any covering or non-covering visual content, and whether visible content continues from the preceding shot. A full-screen insert is still only a picture observation.\n\nAlso answer these two questions explicitly.\n\nFirst: is the whole frame a depicted scene that has its own camera space, lighting, depth and lens behaviour, or is it a flat designed field whose purpose is to carry drawn elements such as words, rows, panels or cards? Live action and animation are both depicted scenes; a paper sheet, ruled or gridded surface, flat or gradient colour, blurred wallpaper, board or slide backdrop filling the frame is a designed field. State which one it is and the visible evidence for it.\n\nSecond: for every framed element inside the picture, such as a card, phone, browser window, screenshot or inset, describe the picture inside the frame and the frame itself separately. For the inside, describe what it depicts and whether it moves. For the frame, describe its border, corner radius, outline, shadow, size, position and how it enters and leaves.\n\nReturn natural language evidence only.` });+ parts.push({ text: `${globalContext}\n\nObserve shot ${shot.index}. Describe the current base picture, any covering or non-covering visual content, and whether visible content continues from the preceding shot. A full-screen insert is still only a picture observation.\n\nAlso answer these two questions explicitly.\n\nFirst: is the whole frame a depicted scene that has its own camera space, lighting, depth and lens behaviour, or is it a flat designed field whose purpose is to carry drawn elements such as words, rows, panels or cards? Live action and animation are both depicted scenes; a paper sheet, ruled or gridded surface, flat or gradient colour, blurred wallpaper, board or slide backdrop filling the frame is a designed field. State which one it is and the visible evidence for it.\n\nSecond: for every framed element inside the picture, such as a card, phone, browser window, screenshot or inset, describe the picture inside the frame and the frame itself separately. For the inside, describe what it depicts and whether it moves. For the frame, describe its border, corner radius, outline, shadow, size, position and how it enters and leaves.\n\nThird: does this picture move at all, and how? Separate three things: whether the camera moves, and how; whether anything in the picture moves, and what; and whether the picture is completely still. A held photograph, screenshot or card that only appears and disappears is still, however long it is on screen.\n\nReturn natural language evidence only.` });return await callSafely(retryDelayMs, generate, parts, "Observe picture only. Do not choose SVML components, do not write markup, and do not decide final source syntax.");}}));const typeTasks: ObservationTask[] = selected.map((shot) => ({ key: `type:${shot.shot_id}`, run: async () => {diff --git a/packages/reference-video-tools/src/types.ts b/packages/reference-video-tools/src/types.tsindex 559973a6..236238d7 100644--- a/packages/reference-video-tools/src/types.ts+++ b/packages/reference-video-tools/src/types.ts@@ -26,6 +26,7 @@ export type ReferenceState = {readonly people_and_product?: Observation;readonly voices?: Observation;readonly persistent_systems?: Observation;+ readonly places?: Observation;};export type PrepareResult = {@@ -37,4 +38,5 @@ export type PrepareResult = {readonly people_and_product: Observation;readonly voices: Observation;readonly persistent_systems: Observation;+ readonly places: Observation;};diff --git a/packages/reference-video-tools/test/tools.test.ts b/packages/reference-video-tools/test/tools.test.tsindex c30221ad..e6b9cab0 100644--- a/packages/reference-video-tools/test/tools.test.ts+++ b/packages/reference-video-tools/test/tools.test.ts@@ -50,6 +50,7 @@ async function workspace(shotCount: number): Promise<{ readonly root: string; repeople_and_product: { status: "complete", text: "one recurring host" },voices: { status: "complete", text: "one voice" },persistent_systems: { status: "complete", text: "one caption system for the whole video" },+ places: { status: "complete", text: "one room, one camera position" },};await writeFile(join(stateRoot, "state.json"), `${JSON.stringify(state, null, 2)}\n`, "utf8");await writeFile(join(stateRoot, "observations.json"), "{}\n", "utf8");@@ -112,6 +113,8 @@ test("observation covers picture, drawn type and sound for every shot and cachesassert.deepEqual(result["unresolved"], []);const picture = calls.find((call) => call.instruction.startsWith("Observe picture only"))!;+ assert.match(picture.text, /does this picture move at all/u,+ "whether a stretch of picture moves decides whether it is generated or held as a still");assert.match(picture.text, /depicted scene/u);assert.match(picture.text, /flat designed field/u);assert.match(picture.text, /the picture inside the frame and the frame itself separately/u);