Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

merge: align coarse-fine rebuild with origin/main

kashorin5ec6beee7a2a
diff --cc packages/provider-hypihub/test/provider.test.ts
index df232201,4633f609..c24eed85
--- a/packages/provider-hypihub/test/provider.test.ts
+++ b/packages/provider-hypihub/test/provider.test.ts
@@@ -223,12 -118,12 +223,12 @@@ test("HypiHub uploads referenced Artifa
});
test("HypiHub fulfills Gemini through the Runtime endpoint and uploads every media Artifact", async () => {
- const artifacts = new MemoryArtifactStore();
- const image = await artifacts.put(new Uint8Array([1, 2, 3]), "image/png");
- const video = await artifacts.put(new Uint8Array([4, 5, 6]), "video/mp4");
+ const resources = new MemoryResourceStore();
+ const image = await resources.put(new Uint8Array([1, 2, 3]), "image/png");
+ const video = await resources.put(new Uint8Array([4, 5, 6]), "video/mp4");
const request: Need = {
id: "need:hypihub-gemini",
- capability: geminiCapabilities["gemini-3.7-flash-openai"],
+ capability: geminiCapabilities["gemini-3.1-pro"],
returns: textTypes.text,
constraints: sealGeminiRequest({
instruction: "Answer briefly.", prompt: "Inspect both references.",
diff --cc packages/reference-video-tools/README.md
index 332d0d50,66c0f3dd..6df15e17
--- a/packages/reference-video-tools/README.md
+++ b/packages/reference-video-tools/README.md
@@@ -21,64 -56,87 +21,76 @@@ new explicit observation; it is not sup
```bash
hypit-reference-video-tools list_svml_packages
-hypit-reference-video-tools prepare_reference --video-path ./reference.mp4
-hypit-reference-video-tools prepare_reference --video-path ./reference.mp4 --redo people
-hypit-reference-video-tools prepare_reference --video-path ./reference.mp4 --redo transcript
+hypit-reference-video-tools prepare_reference --video-path ./reference.mp4 --observer agent
hypit-reference-video-tools observe_reference --reference-id <reference-id>
-hypit-reference-video-tools observe_reference --reference-id <reference-id> --shot-id shot-007
-hypit-reference-video-tools observe_reference --reference-id <reference-id> --shot-id shot-007 --question "How thick is the outline on the caption words?"
-hypit-reference-video-tools observe_reference --reference-id <reference-id> --shot-id shot-007 --reobserve
+hypit-reference-video-tools record_observation --reference-id <reference-id> --key <key> --text-file ./answer.md
hypit-reference-video-tools inspect_svml_vocabulary --package @hypit/media-track --tag Track
-hypit-reference-video-tools compare_reconstruction --reference-id <reference-id> --shot-id shot-007 --image ./rendered.png
-hypit-reference-video-tools variant_state --action discover --project-root ./projects/base
-hypit-reference-video-tools variant_state --action start --project-root ./projects/base --output-root ./projects/base-variants/<batch-id> --count 100
-hypit-reference-video-tools variant_init --project-root ./projects/base --output-root ./projects/base-variants/<batch-id> --slate ./slate.json
-hypit-reference-video-tools variant_check --run ./projects/base-variants/<batch-id>/001-example/build.svrun
+hypit-reference-video-tools validate_local_author_packages --run ./build.svrun
+hypit-reference-video-tools validate_script_cues --run ./build.svrun
+hypit-reference-video-tools preview_check ./build.svrun
+hypit-reference-video-tools layout_check --run ./build.svrun
+hypit-reference-video-tools render_element ./build.svrun --element title --segment intro --out ./title.mp4
+hypit-reference-video-tools compare_reconstruction --reference-id <reference-id> --run ./build.svrun --segment intro --video ./title.mp4 --element title
+hypit-reference-video-tools review_element --run ./build.svrun --element title --segment intro --video ./title.mp4 --intent-file ./title-intent.md
+hypit-reference-video-tools reconstruction_check ./build.svrun --reference-id <reference-id>
+hypit-reference-video-tools authoring_check ./build.svrun
```
-For automation, every command also accepts `--input '{"...":"..."}'` with the complete JSON input
-object. This is useful when the calling agent already has a structured request.
+Every command prints one JSON result. Commands also accept `--input '{...}'` when structured input is
+more convenient. Batch forms read JSON arrays from files for observations, comparisons, renders,
+reviews and layout decisions.
+
+## Reference preparation
-`list_svml_packages` reports every installed package that declares an activation, with the Surface
-tags it registers. `inspect_svml_vocabulary` reads packages you can already name, and the Build CLI
-is deliberately unable to scan a directory, so this is how a caller finds out what vocabulary exists
-instead of working from memory.
+`prepare_reference` accepts a local path or a supported video URL. It extracts bounded shot clips,
+representative frames, a storyboard, speech audio and a word-timed transcript. The returned reference
+id contains the source filename, byte length and modification time so it is inspectable without a
+lookup table.
-`prepare_reference` requires a local video path and stores derived media under the checkout's
-gitignored `.hypit/reference-video-tools/` directory, which is where every other command reads it
-from whichever directory it is run in. It produces four full-reference observations: `people_and_product`,
-`voices`, `persistent_systems` — the on-screen text and graphic systems that continue or recur across
-the whole video, each one's lifetime, and whether its appearance ever changes — and `places`: how
-many locations the video was shot in, which camera positions appear in each, which parts of the video
-use each one, and each position described in enough detail to draw from the words alone. Without `--redo`, completed stages are reused. Use `--redo media` to rebuild shot media and
-clear derived observations, `--redo transcript` to measure the words again, or `--redo people`, `--redo voices`, `--redo systems`, `--redo places`, or `--redo all` to
-rerun only the selected full-reference stages.
+`--observer gemini` uses HypiHub or Vertex according to the configured provider. `--observer agent`
+calls no visual provider; it returns observation tasks with local pictures and transcript context for
+the calling agent to answer through `record_observation`.
+ It also produces `transcript`: the verbatim speech of the whole reference with a start and an end for
+ every single word. Placing an on-screen text reveal against the line that triggers it needs the time
+ of the word, not of the sentence around it, and that question comes up in every reconstruction. The
-speech audio is extracted to `speech.wav` beside the shot media and measured by the HypiHub WhisperX
-Provider in `@hypit/provider-hypihub`, which calls the synchronous
-`/v1/audio/transcriptions` route with word and segment timestamps. The selected HypiHub OAuth
-credential must be available before preparation.
++speech audio is extracted to `speech.wav` beside the shot media and measured through the
++`@hypit/whisperx#whisperx-alignment` capability. This tool currently instantiates the local WhisperX
++Provider directly; routing it through the selected Runtime Profile is pending.
+ The result reports `status`, `transcript_ref` and `word_count`, and the words themselves live in
+ `transcript.json` as passages, each with a `words` array of `{ text, start_seconds, end_seconds, score }`.
+ The transcript is deterministic local evidence rather than an observation: it is never written to the
-observation cache and nothing about it is sent to Gemini. An unavailable HypiHub WhisperX endpoint
-reports `status: "unavailable"` with the reason and prepares everything else.
-
-`observe_reference` observes every unfinished shot, or only the shots named by `--shot-id`. Each shot
-produces three observations — `visual` (base picture, covering content, continuity, whether the frame
-is a depicted scene or a flat designed field, and each framed element's inner picture separately from
-its frame), `text_appearance` (typeface, weight, size, spacing, colour, stroke, shadow, glow,
-per-word emphasis, position and entry of drawn text), and `audio`. The picture observation also
-states whether the picture moves and how, separating camera movement from movement within it, since a
-held still and a moving shot are reconstructed differently. Each cut adds one `continuity`
-observation covering both whether the two shots are one continuous camera shot and whether an overlay
-continues across the boundary. The preceding tail frame, audio tail and full-reference context are
-attached automatically. Completed observations are reused; `--reobserve` is the only way to run a
-completed one again.
-
-Shots are never shorter than one second: a fragment below that is merged into the shot before it,
-because clips that short carry no evidence and the model rejects them outright.
-
-`HYPIT_REFERENCE_CONCURRENCY` and `HYPIT_REFERENCE_LAUNCH_GAP_MS` set how fast requests are issued,
-defaulting to four at a time with a 1.5 second gap. They describe the quota behind the credentials
-rather than anything about the video, so they are environment settings and not flags. Lower them if a
-quota objects; the failure to expect is rate limiting, which backs off on its own before giving up.
-
-`--question` is a separate narrow path: it requires exactly one `--shot-id`, answers only that
-question from that shot's clip and frames in a single request, and neither reads nor writes the
-observation cache. Cross-shot continuity uses the built-in boundary and three-shot window observations.
-
-`compare_reconstruction` sends the named shot's reference frame and a rendered image as an
-unlabelled pair and returns a description of their visible differences. It is never told which image
-is which, what was built, or how. `--question` narrows it to one region of the picture and nothing
-else. Results are not cached, because the rendered side changes on every iteration.
-
-`unresolved` lists the keys of observations that failed. It is not a judgement about evidence
-quality: a complete observation that says "unclear" is still complete.
-
-Gemini uses `HYPIT_GEMINI_PROVIDER=auto|hypihub|vertex` to select its backend. The backends are
-provided by `@hypit/provider-hypihub` and `@hypit/provider-vertex`. HypiHub uses
-`HYPIHUB_API_KEY` and optional `HYPIHUB_BASE_URL` (default `https://hypit.ai`); Vertex uses the
-existing `GOOGLE_CLOUD_PROJECT`, `GOOGLE_APPLICATION_CREDENTIALS_JSON` and optional
-`GOOGLE_CLOUD_LOCATION`. The model defaults to `gemini-3.1-pro`. A missing or insufficient
-user key should be resolved by obtaining a HypiHub key at https://hypit.ai, not by changing Source.
++observation cache and nothing about it is sent to Gemini. An unavailable WhisperX Provider reports
++`status: "unavailable"` with the reason and prepares everything else.
++
+Prepared stages are reused until the caller explicitly asks for `--redo ...` or `--reobserve`. This
+is ordinary working data for one reference, not Build history.
+
+## Inspection and review
+
+`inspect_svml_vocabulary` reports the public Surfaces of packages the caller names.
+`inspect_visual_schema` reports the actual Composition schema accepted from drawing Producers.
+`validate_local_author_packages`, `validate_script_cues`, `preview_check`, `layout_check`,
+`reconstruction_check` and `authoring_check` are independent reports. None approves or unlocks
+another command.
+
+`layout_check` always measures the current realized DOM. Its report contains advisory candidates such
+as overflow, overlap and vertical offset. `layout_accept` attaches a human reason to a structural
+finding id; it does not fingerprint the project or cache a previous measurement.
+
+`render_element` draws one element directly from the Run using local preview media. A comparison reads
+that render against the requested reference stretch. A review reads it against an explicit intent.
+Their JSONL logs contain the actual question, paths and prose answer so later inspection does not
+depend on a separate state database.
+
+## Environment
+
+Reference observation uses `HYPIT_GEMINI_PROVIDER=auto|hypihub|vertex`. HypiHub first reads the
+`hypihub.oauth` session written by `hypit auth login` from the OS credential store, then falls back to
+`HYPIHUB_API_KEY`; `HYPIHUB_BASE_URL` remains optional. Vertex reads `GOOGLE_CLOUD_PROJECT`,
+`GOOGLE_APPLICATION_CREDENTIALS_JSON` and optional `GOOGLE_CLOUD_LOCATION`. Gemini defaults to
- `gemini-3.7-flash-openai`.
++`gemini-3.1-pro`.
+
+Transcript preparation uses the selected local WhisperX service. Start services through the normal
+Runtime configuration with `hypit runtime up`. `HYPIT_REFERENCE_CONCURRENCY` and
+`HYPIT_REFERENCE_LAUNCH_GAP_MS` only control observation request pacing.
diff --cc packages/reference-video-tools/package.json
index 82c5b7ff,aece8a60..5d375070
--- a/packages/reference-video-tools/package.json
+++ b/packages/reference-video-tools/package.json
@@@ -31,7 -31,6 +31,7 @@@
"@hypit/provider-media-local": "workspace:*",
"@hypit/provider-hypihub": "workspace:*",
"@hypit/provider-vertex": "workspace:*",
++ "@hypit/provider-whisperx-local": "workspace:*",
"@hypit/script": "workspace:*",
"@hypit/speech": "workspace:*",
"@hypit/speech-evidence": "workspace:*",