Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
fix: the agent observer attributes speech instead of deferring it
rponec44614de3dc8
diff --git a/.agents/skills/hypit/references/reconstruction/continuity.md b/.agents/skills/hypit/references/reconstruction/continuity.mdindex 21793a08..896be43d 100644--- a/.agents/skills/hypit/references/reconstruction/continuity.md+++ b/.agents/skills/hypit/references/reconstruction/continuity.md@@ -10,7 +10,9 @@ These invariants override superficial layer order and shot boundaries:move.- A shot with no visible person can carry continuing off-screen speech. Do not clear sound or base.- For alternating, overlapping or simultaneous speech, use audio evidence rather than visible-person- presence.+ presence. The `agent` observer has no audio evidence and reads these from the frames and the word+ timings instead, which `observers.md` describes: the transcript is what decides whether anyone is+ speaking at all, so it is what keeps a moving mouth in B-roll from becoming a speaker.- One overlay that continues across a cut remains one visual track spanning its full observedlifetime. Do not recreate it once per shot.- Merge two or three incorrectly split clips only when continuity evidence confirms one camera shotdiff --git a/.agents/skills/hypit/references/reconstruction/observers.md b/.agents/skills/hypit/references/reconstruction/observers.mdindex 06075735..c5f04809 100644--- a/.agents/skills/hypit/references/reconstruction/observers.md+++ b/.agents/skills/hypit/references/reconstruction/observers.md@@ -45,21 +45,26 @@ the storyboard of representative frames. A task that carries a single picture caThe number of frames follows the shot's length, from four for a short one to nine for a long one, soa cell stays wide enough to read detail in.-## What sound costs on the `agent` observer--You cannot hear the reference. Three things follow, and each is a condition to work inside rather than-a step to skip:--- `transcript` is unaffected. WhisperX measures every word locally on both paths, so the words and- their timings are exact, and `final-sources.md` takes Segment durations from `transcript_ref` as it- always does.-- A task that needs sound says so in its own prompt. Answer it from what the pictures and the- transcript support — who is visibly speaking, what is on screen when a passage runs — and state- plainly which parts you could not determine. An observation that says what it could not see is- complete; one that infers a voice from an appearance is wrong, and- `continuity.md` depends on that distinction.-- `voices` and the sound half of each `boundary` are the observations this affects. Read what they- return as what a viewer with the sound off would know.+## How sound is answered on the `agent` observer++You cannot hear the reference, and there is no second source of sound to fall back to, so the sound+observations are answered from the two things you do have. Answer them fully; do not leave them short+and do not put the gap to the author.++- **The transcript says when.** WhisperX measures every word locally on both paths, so the words and+ their timings are exact whichever observer reads the pictures. That settles whether anyone is+ speaking at a given moment, and `final-sources.md` still takes Segment durations from+ `transcript_ref` as it always does.+- **The pictures say who.** Attribute speech to the person the frames show speaking during the words+ the transcript places there. A task that needs sound says so in its own prompt and asks for exactly+ this reading.+- **The two together are the evidence.** `continuity.md` asks for audio evidence over visible-person+ presence because a mouth moving in silent B-roll misleads. On this observer the transcript is what+ keeps that from happening: a person whose mouth moves during a stretch the transcript places no+ words in is not the speaker, however much they look like one.++`voices` and the sound half of each `boundary` are the observations this shapes. They are attributions+read off pictures and word timings, and that is what a later reader should take them for.## The comparison in the loop is yours toodiff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.tsindex aabc362a..12d5ade4 100644--- a/packages/reference-video-tools/src/tools.ts+++ b/packages/reference-video-tools/src/tools.ts@@ -190,9 +190,14 @@ function asPictures(media: readonly string[], state: ReferenceState): readonly sconst TILE_PREAMBLE = "Each supplied picture that shows a grid of frames is one shot, sampled evenly"+ " across its duration and laid out in reading order: left to right, then top to bottom. Read the"+ " grid as time passing. A single picture that is not a grid is one moment.";-const NO_SOUND = "You are reading pictures and cannot hear this reference. Answer what the pictures and"- + " the measured transcript support, and say plainly which parts of the question need sound you do"- + " not have rather than inferring them from appearance.";+// There is no second source of sound to fall back to, so this says how to answer from what there is+// rather than leaving the observation short. The transcript settles when speech happens; the picture+// settles who is on screen while it does. Read together they carry the question far enough to answer.+const NO_SOUND = "You are reading pictures and cannot hear this reference. Answer from the pictures and"+ + " the measured transcript together: the transcript says exactly when words are spoken, and the"+ + " pictures say who is on screen and whose mouth is moving while they are. Attribute speech to the"+ + " person the pictures show speaking during the words the transcript places there, and answer the"+ + " question in full rather than deferring the parts that would be easier with sound.";/** Declares one observation: what it asks, and the evidence it asks over. The observer decides how. */type Request = {diff --git a/packages/reference-video-tools/test/tools.test.ts b/packages/reference-video-tools/test/tools.test.tsindex addd8b0f..1a757a93 100644--- a/packages/reference-video-tools/test/tools.test.ts+++ b/packages/reference-video-tools/test/tools.test.ts@@ -267,7 +267,9 @@ test("the agent observer hands every observation out as a task, with pictures, aassert.doesNotMatch(visual.prompt, /cannot hear/u, "a picture question carries no note about sound it never asked for");const audio = pending.find((task) => task.key === "audio:shot-002")!;- assert.match(audio.prompt, /cannot hear this reference/u, "a question that needs sound says the sound is missing rather than inviting a guess");+ assert.match(audio.prompt, /cannot hear this reference/u, "a question that needs sound says so");+ assert.match(audio.prompt, /transcript places there/u,+ "and says how to answer it anyway, since there is no second source of sound to defer to");assert.equal(audio.image_refs.some((path) => path.endsWith(".wav")), false, "audio has no picture to become");const cached = JSON.parse(await readFile(join(stateRoot, "observations.json"), "utf8")) as Record<string, unknown>;