Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

refactor(cli): dismantle reference-video-tools into hypit media and vocabulary

kashorin407c77ffac7f
diff --git a/docs/guide/author-packages.md b/docs/guide/author-packages.md
index 060c0232..d751124a 100644
--- a/docs/guide/author-packages.md
+++ b/docs/guide/author-packages.md
@@ -157,35 +157,22 @@ vocabulary: {
}
```
-Declare a `preview` for every Surface whose result a reader needs to see to understand it. That is
-not only Surfaces producing a `VisualTrack`: `@hypit/speech-track` declares one for its SemanticTrack
-because its shape is easier to see than to describe. Skip it for a Surface with nothing to show, such as one
-that only assembles a request.
+Declare a `preview` for every Surface a reader should recognise at a glance in Studio's timeline or
+in a catalogue: the poster. A poster is a designed picture, not a frame of output. It says what the
+component *is* in the most abstract, most recognisable form the small space allows, the way a film
+poster is not a screenshot of the film: `@hypit/ranking`'s Column poster is a column of rank badges
+beside a column of icons; its Tier board is lettered rows with icons. Draw it as an SVG committed
+beside the package, rasterise it to the PNG the manifest names under `preview/`, and keep it stable
+while the component's look evolves. Skip it for a Surface with nothing to show, such as one that only
+assembles a request.
`appearance` and `preview` answer different questions and neither replaces the other. `appearance`
-says what the element draws in every case; the preview shows one honest instance of it.
+says what the element draws in every case; the poster says what the element is for.
The fixture's manifest is the canonical small vocabulary example. For a close sibling, read that
package's own README and vocabulary first, then only the role files needed for the changed behavior.
-
-### Producing the preview image
-
-The preview is a real frame of your own component, rendered locally. Nothing generates it for you,
-and a mock-up drawn by hand is worse than no preview at all, because it claims to be output.
-
-Render the fixture's preview Source with
-`hypit-reference-video-tools render_previews <package-dir>`. For a new package, use the same
-package-owned `preview/preview.svml`, `preview/recipes.svs` and `preview/build.svrun` shape, then keep
-one representative frame under `preview/`. The command resolves the package from its explicit
-workspace/package root and the active Distribution fallback; it does not require a
-`packages/<slug>/node_modules` self-link or a manual `npm link`. It compiles the preview Run, lets the
-native preview-mock path satisfy undeclared media, and runs the real Producers. It then finds the
-target Present's longest stable interval and seeks the middle frame in a fixed local browser before
-writing the promised image named by the Surface Manifest. It does not encode a complete PNG sequence,
-and it never changes HyperFrames itself.
-
-If no stable interval has at least two frames, the middle frame of the longest Present is used. This
-keeps the picture representative while avoiding a full render solely to obtain one catalogue image.
+`hypit vocabulary <package>` prints the installed declarations; author from that output rather than
+from another package's source.
## 6. Write the activation descriptor
diff --git a/docs/zh/guide/author-packages.md b/docs/zh/guide/author-packages.md
index b8bec17e..0db2bf6f 100644
--- a/docs/zh/guide/author-packages.md
+++ b/docs/zh/guide/author-packages.md
@@ -95,7 +95,7 @@ export const decodeMyComponentSurface: StructuredSurfaceHandler = ({ element })
通用包不要从业务包源码推断写法。使用仓库内的最小完整 fixture:
`examples/minimal-author-package/packages/example-component/`。它包含完整的
Surface 返回值、Fragment literal、Manifest ports、Producer 和 activation 形状。
-只有 `inspect_svml_vocabulary` 已证明存在高度相似的结构性 sibling 时,才允许读取该
+只有 `hypit vocabulary` 已证明存在高度相似的结构性 sibling 时,才允许读取该
sibling 的 README 和实现所需的少数 role 文件,并将其复制为新的 Module;不得比较无关业务包。
## 5. 声明 Surface 词表与预览图
@@ -126,28 +126,11 @@ vocabulary: {
}
```
-凡是读者必须看一眼产出才能理解的 Surface,都要声明 `preview`。这不限于产出 `VisualTrack` 的 Surface:`@hypit/speech-track` 也为 SemanticTrack 声明了预览图,因为它的形态看一眼比读一段描述更快。没有可展示产出的 Surface 可以不声明,例如只负责装配请求的那种。
+凡是读者应该在 Studio 时间轴或目录里一眼认出的 Surface,都要声明 `preview`:它是海报。海报是设计出来的图,不是产出的一帧。它用这一小块位置所允许的最抽象、最形象的方式说明组件是什么,就像电影海报不是电影截屏:`@hypit/ranking` 的 Column 海报是一列名次徽章挨着一列图标,Tier 榜是带字母的几行加图标。把它画成 SVG 随包提交,栅格化成 manifest 在 `preview/` 下指名的 PNG,组件外观演进时海报保持稳定。没有可展示产出的 Surface 可以不声明,例如只负责装配请求的那种。
-`appearance` 和 `preview` 回答的是两个不同的问题,谁都替代不了谁:`appearance` 说的是这个元素在所有情况下都会画出什么,预览图给出的是其中一个诚实的实例。
+`appearance` 和 `preview` 回答的是两个不同的问题,谁都替代不了谁:`appearance` 说的是这个元素在所有情况下都会画出什么,海报说的是这个元素是干什么的。
-完整词表和预览声明以最小 fixture 为权威示例。生产路线应使用
-`hypit-reference-video-tools inspect_svml_vocabulary` 读取已安装包的公开声明,
-而不是打开业务包源码。
-
-### 预览图怎么产出
-
-预览图是你自己这个组件在本地渲染出来的真实一帧。没有任何工具会替你生成它,而手工画的示意图比没有预览图更糟,因为它冒充了真实产出。
-
-用 `hypit-reference-video-tools render_previews` 渲染 fixture 的 preview Source;新包沿用同一套
-package-owned preview Source,不需要阅读仓库的视觉测试源码。步骤是:
-
-1. 用你自己包的 render 函数,在一个封好的 ProgramSpace 上构造出 Track 值;
-2. `sealComposition({ id, canvas, tracks })`,再用 `@hypit/hyperframes` 的 `compileHyperframesDocument(composition, space)`;
-3. `materializeHyperframesHtml(document, resolve)` 写进一个临时目录,其中 `resolve` 把声明的每个 Artifact(字体、图片)映射到本地文件;
-4. 通过标准的 SVRun → preview-mock → Producer 路径得到真实 Composition;
-5. 找到目标 Present 最长的稳定区间,在固定版本的本地浏览器中 seek 到区间中间帧并截图,提交到 `preview/` 下。
-
-不会为了这一张图生成完整 PNG 序列,也不修改 HyperFrames。若没有至少两帧的稳定区间,就使用该包最长 Present 的中间帧。
+完整词表和预览声明以最小 fixture 为权威示例。`hypit vocabulary <package>` 打印已安装包的公开声明;从这份输出写作,而不是打开别的业务包源码。
## 6. 编写 activation 描述符
diff --git a/package.json b/package.json
index d52f9b8c..da0cfc4f 100644
--- a/package.json
+++ b/package.json
@@ -7,8 +7,7 @@
"bin": {
"hypit": "./bin/hypit.mjs",
"hypit-studio": "./bin/hypit-studio.mjs",
- "hypit-preview-check": "./bin/hypit-preview-check.mjs",
- "hypit-reference-video-tools": "./packages/reference-video-tools/bin/reference-video-tools.mjs"
+ "hypit-preview-check": "./bin/hypit-preview-check.mjs"
},
"packageManager": "[email protected]",
"files": [
diff --git a/packages/reference-video-tools/README.md b/packages/reference-video-tools/README.md
deleted file mode 100644
index 5b334f68..00000000
--- a/packages/reference-video-tools/README.md
+++ /dev/null
@@ -1,95 +0,0 @@
-# `@hypit/reference-video-tools`
-
-Direct tools for understanding a reference video and inspecting or previewing a Hypit project.
-
-The package does not own a production workflow. It has no route/revision/variant state machine, no
-checkpoint or recovery cursor, no content-addressed evidence directory, and no aggregate approval
-state. Calling a command performs that command. It does not silently advance another process.
-
-The useful outputs are ordinary content:
-
-- prepared reference media, shots and transcript under `.hypit/reference-video-tools/`;
-- reference observations and explicit comparison/review logs;
-- requested stills, clips and package previews;
-- direct JSON reports about vocabulary, package loading, Script Cues, graph tracing and realized
- browser layout.
-
-These files can be opened, named, copied or removed directly. A repeated comparison or review is a
-new explicit observation; it is not suppressed by comparing file hashes.
-
-## Commands
-
-```bash
-hypit-reference-video-tools list_svml_packages
-hypit-reference-video-tools prepare_reference --video-path ./reference.mp4 --observer agent
-hypit-reference-video-tools observe_reference --reference-id <reference-id>
-hypit-reference-video-tools record_observation --reference-id <reference-id> --key <key> --text-file ./answer.md
-hypit-reference-video-tools inspect_svml_vocabulary --package @hypit/media-track --tag Track
-hypit-reference-video-tools validate_local_author_packages --run ./build.svrun
-hypit-reference-video-tools validate_script_cues --run ./build.svrun
-hypit-reference-video-tools preview_check ./build.svrun
-hypit-reference-video-tools layout_check --run ./build.svrun
-hypit-reference-video-tools render_element ./build.svrun --element title --segment intro --out ./title.mp4
-hypit-reference-video-tools compare_reconstruction --reference-id <reference-id> --run ./build.svrun --segment intro --video ./title.mp4 --element title
-hypit-reference-video-tools review_element --run ./build.svrun --element title --segment intro --video ./title.mp4 --intent-file ./title-intent.md
-hypit-reference-video-tools reconstruction_check ./build.svrun --reference-id <reference-id>
-hypit-reference-video-tools authoring_check ./build.svrun
-```
-
-Every command prints one JSON result. Commands also accept `--input '{...}'` when structured input is
-more convenient. Batch forms read JSON arrays from files for observations, comparisons, renders,
-reviews and layout decisions.
-
-## Reference preparation
-
-`prepare_reference` accepts a local path or a supported video URL. It extracts bounded shot clips,
-representative frames, a storyboard, speech audio and a word-timed transcript. The returned reference
-id contains the source filename, byte length and modification time so it is inspectable without a
-lookup table.
-
-`--observer gemini` uses HypiHub or Vertex according to the configured provider. `--observer agent`
-calls no visual provider; it returns observation tasks with local pictures and transcript context for
-the calling agent to answer through `record_observation`.
-
-It also produces `transcript`: the verbatim speech of the whole reference with a start and an end for
-every single word. Placing an on-screen text reveal against the line that triggers it needs the time
-of the word, not of the sentence around it, and that question comes up in every reconstruction. The
-speech audio is extracted to `speech.wav` beside the shot media and measured through the
-`@hypit/whisperx#whisperx-alignment` Endpoint of the selected Runtime Profile.
-The result reports `status`, `transcript_ref` and `word_count`, and the words themselves live in
-`transcript.json` as passages, each with a `words` array of `{ text, start_seconds, end_seconds, score }`.
-The transcript is deterministic local evidence rather than an observation: it is never written to the
-observation cache and nothing about it is sent to Gemini. An unavailable WhisperX Provider reports
-`status: "unavailable"` with the reason and prepares everything else.
-
-Prepared stages are reused until the caller explicitly asks for `--redo ...` or `--reobserve`. This
-is ordinary working data for one reference, not Build history.
-
-## Inspection and review
-
-`inspect_svml_vocabulary` reports the public Surfaces of packages the caller names.
-`inspect_visual_schema` reports the actual Composition schema accepted from drawing Producers.
-`validate_local_author_packages`, `validate_script_cues`, `preview_check`, `layout_check`,
-`reconstruction_check` and `authoring_check` are independent reports. None approves or unlocks
-another command.
-
-`layout_check` always measures the current realized DOM. Its report contains advisory candidates such
-as overflow, overlap and vertical offset. `layout_accept` attaches a human reason to a structural
-finding id; it does not fingerprint the project or cache a previous measurement.
-
-`render_element` draws one element directly from the Run using local preview media. A comparison reads
-that render against the requested reference stretch. A review reads it against an explicit intent.
-Their JSONL logs contain the actual question, paths and prose answer so later inspection does not
-depend on a separate state database.
-
-## Environment
-
-Observation and transcription run through the selected Runtime Profile, the same way a Build does:
-`--runtime <profile>` names it, otherwise the project's `.hypit/runtime` selection is read. The
-Profile's Gemini Endpoint (HypiHub, Vertex or another Provider) answers `--observer gemini`, and its
-`@hypit/whisperx#whisperx-alignment` Endpoint measures the transcript. This tool never chooses a
-Provider, reads a credential store or contacts a service itself; `hypit plan --runtime <profile>` shows
-which Endpoint serves each capability and where it publishes prices. The Gemini model defaults to the
-first model `@hypit/gemini` declares; `GEMINI_MODEL` selects another declared model.
-`HYPIT_REFERENCE_CONCURRENCY` and `HYPIT_REFERENCE_LAUNCH_GAP_MS` only control observation request
-pacing.
diff --git a/packages/reference-video-tools/bin/reference-video-tools.mjs b/packages/reference-video-tools/bin/reference-video-tools.mjs
deleted file mode 100755
index c0e232cd..00000000
--- a/packages/reference-video-tools/bin/reference-video-tools.mjs
+++ /dev/null
@@ -1,16 +0,0 @@
-#!/usr/bin/env node
-import { register } from "tsx/esm/api";
-import { dirname, resolve } from "node:path";
-import { fileURLToPath } from "node:url";
-
-const distributionRoot = resolve(dirname(fileURLToPath(import.meta.url)), "../../..");
-process.env.HYPIT_DISTRIBUTION_ROOT ??= distributionRoot;
-register();
-const {
- installDistributionPackageResolution,
- installExternalPackageResolution,
-} = await import("../../package-loader-node/src/distribution-resolution.ts");
-installDistributionPackageResolution([distributionRoot]);
-const { hypitHostPackageRoot } = await import("../../runtime-host-node/src/index.ts");
-installExternalPackageResolution([hypitHostPackageRoot()]);
-await import("../src/cli.ts");
diff --git a/packages/reference-video-tools/package.json b/packages/reference-video-tools/package.json
deleted file mode 100644
index c488ca12..00000000
--- a/packages/reference-video-tools/package.json
+++ /dev/null
@@ -1,46 +0,0 @@
-{
- "name": "@hypit/reference-video-tools",
- "version": "0.0.0-dev",
- "license": "SEE LICENSE IN LICENSE",
- "private": true,
- "type": "module",
- "bin": {
- "hypit-reference-video-tools": "./bin/reference-video-tools.mjs"
- },
- "exports": {
- ".": "./src/index.ts",
- "./cli": "./src/cli.ts"
- },
- "dependencies": {
- "@hypit/composition": "workspace:*",
- "@hypit/core": "workspace:*",
- "@hypit/driver-node": "workspace:*",
- "@hypit/gemini": "workspace:*",
- "@hypit/hyperframes": "workspace:*",
- "@hypit/markup": "workspace:*",
- "@hypit/media": "workspace:*",
- "@hypit/model-kit": "workspace:*",
- "@hypit/preview-mock": "workspace:*",
- "@hypit/package-loader-node": "workspace:*",
- "@hypit/program-space": "workspace:*",
- "@hypit/protocol": "workspace:*",
- "@hypit/run-markup": "workspace:*",
- "@hypit/runtime": "workspace:*",
- "@hypit/provider-media-local": "workspace:*",
- "@hypit/script": "workspace:*",
- "@hypit/speech": "workspace:*",
- "@hypit/speech-evidence": "workspace:*",
- "@hypit/studio": "workspace:*",
- "@hypit/svs": "workspace:*",
- "@hypit/text": "workspace:*",
- "@hypit/video-cli": "workspace:*",
- "@hypit/visual-ir": "workspace:*",
- "@hypit/whisperx": "workspace:*",
- "@hypit/yt-dlp": "workspace:*",
- "puppeteer-core": "25.5.0",
- "tsx": "4.21.0"
- },
- "devDependencies": {
- "@types/node": "24.10.1"
- }
-}
diff --git a/packages/reference-video-tools/src/authoring.ts b/packages/reference-video-tools/src/authoring.ts
deleted file mode 100644
index 84e4f4e4..00000000
--- a/packages/reference-video-tools/src/authoring.ts
+++ /dev/null
@@ -1,1017 +0,0 @@
-import { mkdir, readFile, readdir, rm, writeFile } from "node:fs/promises";
-import { existsSync } from "node:fs";
-import { spawnSync } from "node:child_process";
-
-import { createRequire } from "node:module";
-import { cpus } from "node:os";
-import { basename, dirname, join, resolve } from "node:path";
-import { fileURLToPath } from "node:url";
-
-import puppeteer from "puppeteer-core";
-
-import { compileHyperframesDocument, materializeHyperframesHtml } from "@hypit/hyperframes";
-import type { ProgramSpace } from "@hypit/program-space";
-import type { CompiledGraph } from "@hypit/protocol";
-import { parseScript } from "@hypit/script";
-import { videoCliDistribution } from "@hypit/video-cli";
-import { loadStudioCompanionRegistry } from "@hypit/studio/src/companion-profile.js";
-import { openStudioBuildLibrary } from "@hypit/studio/src/build-library.js";
-import { loadStudioDomain } from "@hypit/studio/src/domain.js";
-import type { Preview } from "@hypit/studio/src/programme.js";
-import { preview } from "@hypit/studio/src/programme.js";
-import { loadStudioRun } from "@hypit/studio/src/run.js";
-import { inspectStudioRun } from "@hypit/studio/src/studio-preflight.js";
-import { realizePreviewMock } from "@hypit/preview-mock";
-import { EndpointRegistry } from "@hypit/driver-node";
-import { createLocalMediaProvider } from "@hypit/provider-media-local";
-
-import { assert, ensureDir } from "./media.js";
-import type { TranscriptFile } from "./types.js";
-
-/** A frame range in the estimate-timed preview program, half-open in Script order. */
-export type AuthoringWindow = {
- readonly startFrame: number;
- readonly endFrameExclusive: number;
-};
-
-export type StandInTiming = {
- readonly segment: string;
- readonly basis: "reference" | "estimate";
- readonly seconds: number;
- readonly frames: number;
- readonly tokens: number;
- readonly matched: number;
-};
-
-/** Which Segment the render is looking at, named directly or through a Selection's words. */
-export type StandInFocus = {
- readonly segment?: string;
- readonly selection?: string;
- /**
- * A half-open range of the Script's own speech tokens.
- *
- * The finest thing worth looking at is not always a thing the Script named. A caption Cue is a run
- * of words ended by a speaker change or an authored break, and the planner picks it out by index —
- * so the index is what both sides take. `segment` and `selection` remain as the convenient way to
- * write down a range that does have a name.
- */
- readonly tokens?: readonly [number, number];
-};
-
-/**
- * A token window, read from whatever a caller supplied.
- *
- * The type says two numbers, and nothing but this checked that. A batch entry is JSON a caller wrote
- * by hand, so `"79:86"` arrives as a string, and destructuring one yields its first two characters:
- * `from` is `"7"` and `to` is `"9"`. Every guard downstream then compares strings — `"9" > "7"` holds,
- * and so does `"9" <= 135` once it coerces — so the window passes as valid and words 7 to 9 render
- * under the name of words 79 to 86. The call reports success, and the difference only shows up as a
- * comparison that makes no sense against a picture nobody asked for.
- */
-export function tokenWindow(value: unknown, subject: string): readonly [number, number] {
- assert(Array.isArray(value) && value.length === 2 && value.every((item) => Number.isInteger(item)),
- `${subject} must be two whole numbers, as [from, to) — received ${JSON.stringify(value)}`);
- return [value[0] as number, value[1] as number];
-}
-
-function referenceTimed(
- segment: { readonly tokenStart: number; readonly tokenEndExclusive: number },
- pairs: ReadonlyMap<number, number>,
-): boolean {
- for (let index = segment.tokenStart; index < segment.tokenEndExclusive; index += 1) {
- if (pairs.has(index)) return true;
- }
- return false;
-}
-
-/**
- * The Hypit tree this package is installed into.
- *
- * Studio's domain and the HyperFrames runtime are both found from it, and neither can be found from
- * the working directory: a command run from a project directory would resolve no packages, and one
- * that assumed the root would work there and crash anywhere else. Walking up from this module reaches
- * the tree from wherever the package was installed.
- *
- * What identifies it is holding the packages this module imports. A name in `package.json` does not:
- * a checkout and an installed Distribution carry different ones, so a check against either name
- * refuses the other, and the refusal reads as "this is not a Hypit checkout" while standing inside
- * one.
- */
-const NEEDED = ["studio", "hyperframes", "script", "estimate"] as const;
-
-function isHypitTree(directory: string): boolean {
- return NEEDED.every((name) => existsSync(join(directory, "packages", name, "package.json")));
-}
-
-function nearestTree(start: string): string | undefined {
- let directory = resolve(start);
- while (true) {
- if (isHypitTree(directory)) return directory;
- const parent = dirname(directory);
- if (parent === directory) return undefined;
- directory = parent;
- }
-}
-
-export function repositoryRoot(): string {
- // An explicit override is taken as given. Re-deriving it would refuse a layout the caller can see
- // and this cannot, which leaves no way out of a wrong guess.
- const override = process.env.HYPIT_REPOSITORY?.trim();
- if (override !== undefined && override.length > 0) {
- const directory = resolve(override);
- assert(existsSync(directory), `HYPIT_REPOSITORY names no directory: ${directory}`);
- return directory;
- }
- const found = nearestTree(dirname(fileURLToPath(import.meta.url))) ?? nearestTree(process.cwd());
- assert(found !== undefined,
- `no Hypit tree above ${dirname(fileURLToPath(import.meta.url))} or ${process.cwd()} — one holding packages/${NEEDED.join(", packages/")}. Set HYPIT_REPOSITORY to name it.`);
- return found;
-}
-
-/**
- * Where installed packages are resolved from, found the way `hypit check` finds it: the nearest
- * directory at or above the project that holds a `package.json`.
- *
- * A project's own packages are installed against the project, so a root taken from
- * anywhere else resolves none of them. This is the same walk `packages/cli/src/main.ts` performs,
- * and the reason a project is given a `package.json` of its own — without one the walk passes
- * through it and lands on the tree.
- */
-export function nearestPackageRoot(start: string): string | undefined {
- let directory = resolve(start);
- while (true) {
- if (existsSync(join(directory, "package.json"))) return directory;
- const parent = dirname(directory);
- if (parent === directory) return undefined;
- directory = parent;
- }
-}
-
-/** Where a relative path on the command line is measured from. */
-export function invokedFrom(): string {
- return process.cwd();
-}
-
-/** The one active Script identity and body, preserved by preview realization. */
-export function scriptBody(svml: string): { readonly id: string; readonly text: string; readonly offset: number } {
- const open = /<script\b[^>]*>/u.exec(svml);
- if (open === null) throw new Error("the Source declares no <script>");
- const id = /\bid="([^"]+)"/u.exec(open[0])?.[1];
- if (id === undefined || id.length === 0) throw new Error("the Source's <script> declares no id");
- const start = open.index + open[0].length;
- const end = svml.indexOf("</script>", start);
- if (end < 0) throw new Error("the Source's <script> is not closed");
- return { id, text: svml.slice(start, end), offset: start };
-}
-
-/**
- * What a Source calls each package it imports.
- *
- * These tools find elements by tag — `SemanticTake`, `Normalize`, `Canvas` — and a tag is written
- * behind whatever alias the Source chose in `<import as="…" from="…"/>`. The alias is the author's,
- * not the package's: two Sources importing one package under two names are the same Source as far as
- * anything downstream is concerned.
- *
- * Assuming the conventional alias is therefore a reading that is right until somebody writes
- * `as="speech-track"`, at which point the element is invisible and whatever depended on finding it
- * reports the Source as missing something it plainly has. So the alias is read from the Source, and
- * the conventional one is the fallback for a Source that imports without naming one.
- */
-export function aliasPattern(svml: string, specifier: string, conventional: string): string {
- const found = new Set<string>();
- for (const match of svml.matchAll(/<import\s+([^>]*?)\/?>/gu)) {
- const attributes = match[1] ?? "";
- // The scope's own `@` is part of the name, so only the last one separates it from the version.
- const from = /\bfrom="(.+)@\d+"/u.exec(attributes)?.[1];
- if (from !== specifier) continue;
- found.add(/\bas="([^"]+)"/u.exec(attributes)?.[1] ?? conventional);
- }
- if (found.size === 0) found.add(conventional);
- // Longest first, so `speech-track` is not matched as `speech` followed by a stray `-track`.
- return [...found].sort((left, right) => right.length - left.length)
- .map((alias) => alias.replace(/[.*+?^${}()|[\]\\-]/gu, "\\$&")).join("|");
-}
-
-/** Every Recipe body in every sheet the Source imports, keyed `alias.path`. */
-/**
- * The Author SVML a Run declares, both as it is written and as a path.
- *
- * Studio's unit of work is the Run, so everything on this route is handed one and reads the Source
- * back out of it. `declared` is what the Run wrote, which a derived Run has to repoint.
- */
-export async function authorSource(runPath: string): Promise<{ readonly path: string; readonly declared: string }> {
- const runSource = await readFile(runPath, "utf8").catch(() => undefined);
- assert(runSource !== undefined, `cannot read ${runPath}`);
- const declared = /<author\s+source="([^"]+)"/u.exec(runSource)?.[1];
- assert(declared !== undefined, `${runPath} declares no <author source="…"/>`);
- return { path: resolve(dirname(runPath), declared), declared };
-}
-
-/** One word of a reference's transcript, at the seconds the reference speaks it. */
-export type ReferenceWord = {
- readonly text: string;
- readonly startSeconds: number;
- readonly endSeconds: number;
-};
-
-/** Where `prepare_reference` writes a reference's per-word times. */
-export function referenceRoot(): string {
- return join(repositoryRoot(), ".hypit", "reference-video-tools");
-}
-
-/**
- * The per-word times a prepared reference was transcribed to, in the order it speaks them.
- *
- * Words the aligner left untimed are dropped: a word with no seconds against it cannot anchor
- * anything, and carrying it would shift every index after it away from the clock.
- */
-export async function referenceWords(reference: string): Promise<readonly ReferenceWord[]> {
- const path = join(referenceRoot(), reference, "transcript.json");
- const text = await readFile(path, "utf8").catch(() => undefined);
- assert(text !== undefined, `reference ${reference} has no transcript at ${path}; run prepare_reference first`);
- const file = JSON.parse(text) as TranscriptFile;
- const words = file.passages.flatMap((passage) => passage.words).flatMap((word) =>
- word.start_seconds === undefined || word.end_seconds === undefined
- ? []
- : [{ text: word.text, startSeconds: word.start_seconds, endSeconds: word.end_seconds }]);
- assert(words.length > 0, `reference ${reference} has a transcript with no timed words`);
- return words;
-}
-
-/** Lowercase letters and digits only, so `AI,` and `ai` are the same word and `11 labs` is two. */
-function normalizeWord(text: string): string {
- return text.toLowerCase().replace(/[^a-z0-9]+/gu, "");
-}
-
-type MatchingBlock = { readonly left: number; readonly right: number; readonly size: number };
-
-/**
- * The longest run of words that appears in both sequences, within the given ranges.
- *
- * `lengths` carries, for each position in the right sequence, how long a run ends there against the
- * word just read from the left. Reading one left word at a time turns the whole search into two
- * passes rather than a comparison of every pair.
- */
-function longestRun(
- left: readonly string[], right: readonly string[],
- leftFrom: number, leftTo: number, rightFrom: number, rightTo: number,
- positions: ReadonlyMap<string, readonly number[]>,
-): MatchingBlock {
- let bestLeft = leftFrom;
- let bestRight = rightFrom;
- let bestSize = 0;
- let lengths = new Map<number, number>();
- for (let index = leftFrom; index < leftTo; index += 1) {
- const next = new Map<number, number>();
- for (const at of positions.get(left[index]!) ?? []) {
- if (at < rightFrom) continue;
- if (at >= rightTo) break;
- const run = (lengths.get(at - 1) ?? 0) + 1;
- next.set(at, run);
- if (run > bestSize) {
- bestSize = run;
- bestLeft = index - run + 1;
- bestRight = at - run + 1;
- }
- }
- lengths = next;
- }
- return { left: bestLeft, right: bestRight, size: bestSize };
-}
-
-/**
- * Which Script word is which transcript word, as runs that appear in both in the same order.
- *
- * The two sequences say the same thing and are not the same list. A transcriber writes `11 labs`
- * where the Script writes `elevenlabs`, splits a word in two, or hears one that is not there, and a
- * single insertion is enough to put a positional pairing a word out for the whole rest of the
- * reference — on this project it drops the pairing from 96% of words to 41%. So the two are aligned:
- * the longest run common to both is taken as fixed, and the stretches on either side of it are
- * aligned the same way, down to the words that do not appear on both sides. Anchoring on runs rather
- * than on single words is what keeps a common word like `the` from pairing with a `the` elsewhere in
- * the reference.
- */
-function alignWords(script: readonly string[], reference: readonly string[]): ReadonlyMap<number, number> {
- const positions = new Map<string, number[]>();
- for (const [index, word] of reference.entries()) {
- const seen = positions.get(word);
- if (seen === undefined) positions.set(word, [index]);
- else seen.push(index);
- }
- const pairs = new Map<number, number>();
- const pending: (readonly [number, number, number, number])[] = [[0, script.length, 0, reference.length]];
- while (pending.length > 0) {
- const [leftFrom, leftTo, rightFrom, rightTo] = pending.pop()!;
- const block = longestRun(script, reference, leftFrom, leftTo, rightFrom, rightTo, positions);
- if (block.size === 0) continue;
- for (let step = 0; step < block.size; step += 1) pairs.set(block.left + step, block.right + step);
- if (leftFrom < block.left && rightFrom < block.right) pending.push([leftFrom, block.left, rightFrom, block.right]);
- const leftEnd = block.left + block.size;
- const rightEnd = block.right + block.size;
- if (leftEnd < leftTo && rightEnd < rightTo) pending.push([leftEnd, leftTo, rightEnd, rightTo]);
- }
- return pairs;
-}
-
-type WordSpan = { readonly start: number; readonly end: number };
-
-/**
- * A time for every Script word, from the ones that matched.
- *
- * A matched word takes the seconds the reference speaks it at. A run of unmatched words is spread
- * evenly across the stretch between the matched words on either side of it — the reference was
- * saying something there, and what it was saying is the run. A run at either end of the Script has a
- * matched word on one side only, so it takes the reference words immediately outside that one, at a
- * word apiece.
- */
-function wordSpans(count: number, pairs: ReadonlyMap<number, number>, reference: readonly ReferenceWord[]): readonly WordSpan[] {
- const anchors = [...pairs.keys()].sort((left, right) => left - right);
- const spans: WordSpan[] = new Array(count) as WordSpan[];
- for (const index of anchors) {
- const word = reference[pairs.get(index)!]!;
- spans[index] = { start: word.startSeconds, end: word.endSeconds };
- }
- for (let position = 0; position <= anchors.length; position += 1) {
- const before = position === 0 ? undefined : anchors[position - 1]!;
- const after = position === anchors.length ? undefined : anchors[position]!;
- const from = before === undefined ? 0 : before + 1;
- const to = after === undefined ? count : after;
- if (from >= to) continue;
- const run = to - from;
- let start: number;
- let end: number;
- if (before !== undefined && after !== undefined) {
- start = reference[pairs.get(before)!]!.endSeconds;
- end = reference[pairs.get(after)!]!.startSeconds;
- } else if (after !== undefined) {
- const at = pairs.get(after)!;
- start = reference[Math.max(0, at - run)]!.startSeconds;
- end = reference[at]!.startSeconds;
- } else {
- const at = pairs.get(before!)!;
- start = reference[at]!.endSeconds;
- end = reference[Math.min(reference.length - 1, at + run)]!.endSeconds;
- }
- const step = (end - start) / run;
- for (let offset = 0; offset < run; offset += 1) {
- spans[from + offset] = { start: start + step * offset, end: start + step * (offset + 1) };
- }
- }
- return spans;
-}
-
-/**
- * The stretch of a reference that speaks one Segment's or one Selection's words.
- *
- * A render covers a word range; a shot is a cut in the picture. The two are unrelated stretches of
- * the same video, and a Segment routinely runs across five of them. So a comparison against the
- * reference finds its stretch the way every other window on this route is found — the Script's own
- * tokens, aligned against the transcript — and reads the seconds off the words it lands on.
- *
- * `first` and `last` carry the end words' own spans, which is what an end can be moved inside of
- * without the range gaining or losing a whole word.
- */
-export type SpokenRange = {
- readonly startSeconds: number;
- readonly endSeconds: number;
- readonly first: ReferenceWord;
- readonly last: ReferenceWord;
- readonly words: number;
- /** How many of those words the transcript carries. The rest take their seconds from `wordSpans`. */
- readonly matched: number;
-};
-
-export async function spokenRange(
- svmlPath: string,
- focus: StandInFocus,
- reference: readonly ReferenceWord[],
-): Promise<SpokenRange> {
- const svml = await readFile(svmlPath, "utf8");
- const body = scriptBody(svml);
- const parsed = parseScript(svmlPath, body.text, body.offset);
-
- // Which Script words the range covers. A Selection is marked once, so its word range is the
- // tokens between the anchors its open and close markers name.
- //
- // A range given by index skips the lookup entirely: it is already the answer the other two forms
- // are resolved into, and it is the only form that can name a stretch the Script never marked.
- let from: number | undefined;
- let to: number | undefined;
- if (focus.tokens !== undefined) {
- [from, to] = tokenWindow(focus.tokens, "the token window");
- assert(from >= 0 && to > from && to <= parsed.tokens.length,
- `token range [${from}, ${to}) is outside the Script's ${parsed.tokens.length} words`);
- } else if (focus.selection !== undefined) {
- const selection = parsed.selections.find((item) => item.id === focus.selection);
- assert(selection !== undefined, `the Script marks no Selection ${focus.selection}`);
- from = selection.open.boundary.tokenIndex;
- to = selection.close.boundary.tokenIndex;
- } else {
- assert(focus.segment !== undefined, "name a Segment, a Selection or a token range to read a word range from");
- const segment = parsed.segments.find((item) => item.id === focus.segment);
- assert(segment !== undefined, `the Script has no Segment ${focus.segment}`);
- from = segment.tokenStart;
- to = segment.tokenEndExclusive;
- }
- const start = from ?? 0;
- const end = to ?? 0;
- assert(start < end, `${focus.selection ?? focus.segment} covers no words`);
-
- const pairs = alignWords(
- parsed.tokens.map((token) => normalizeWord(token.text)),
- reference.map((word) => normalizeWord(word.text)));
- let matched = 0;
- for (let index = start; index < end; index += 1) if (pairs.has(index)) matched += 1;
- // Without a matched word inside the range, its seconds are an interpolation between whatever the
- // reference was saying on either side of it, which is a stretch of the video nobody asked for.
- assert(matched > 0,
- `the reference's transcript carries none of the words of ${focus.selection ?? focus.segment}, so the seconds it spoke them at are unknown`);
-
- const spans = wordSpans(parsed.tokens.length, pairs, reference);
-
- // Where the range closes, which has to be the second the render holds the last word until. A word
- // keeps the screen until the next one starts, so the pause the reference leaves between two words
- // belongs to the word before it; the estimate-timed SemanticTake preserves this same boundary
- // across a Segment boundary as well as inside one. Cutting the reference at the last word's own
- // end instead would leave it short of the render by that pause.
- //
- // The Script's last word has no next word to hold until, and closes on its own end. So does a word
- // whose next Segment the transcript carries none of: that Segment runs at the estimator's length,
- // the take before it was not sized to reach it, and the render closes where the words stop.
- const opener = parsed.segments.find((item) => end >= item.tokenStart && end < item.tokenEndExclusive);
- const holder = parsed.segments.find((item) => end - 1 >= item.tokenStart && end - 1 < item.tokenEndExclusive);
- const holdsOn = end < parsed.tokens.length
- && (opener === holder || (opener !== undefined && referenceTimed(opener, pairs)));
- const endSeconds = holdsOn ? spans[end]!.start : spans[end - 1]!.end;
-
- const word = (index: number, until = spans[index]!.end): ReferenceWord => ({
- text: parsed.tokens[index]!.text,
- startSeconds: spans[index]!.start,
- endSeconds: until,
- });
- return {
- startSeconds: spans[start]!.start,
- endSeconds,
- first: word(start),
- // The end word carries the same close, so moving the end onto a shot boundary inside it looks at
- // the seconds the render actually drew rather than a shorter word.
- last: word(end - 1, endSeconds),
- words: end - start,
- matched,
- };
-}
-
-/**
- * Do `work` once per key within this process, and let every later caller await that one run.
- *
- * A round asks for several windows of one program. Each entry is a whole call, so each would draw
- * the program again to throw most of it away, and each would write the mocks and the derived Run
- * again over the ones already there. Keyed by the same hash the working directory is, so entries
- * that share a directory share the one run that fills it. Between processes this holds nothing,
- * which is correct: a new process is the one case where the Source may have changed.
- */
-const runningOnce = new Map<string, Promise<void>>();
-async function once(key: string, work: () => Promise<void>): Promise<void> {
- const running = runningOnce.get(key);
- if (running !== undefined) return await running;
- const started = work();
- runningOnce.set(key, started);
- try { await started; } catch (error) { runningOnce.delete(key); throw error; }
-}
-
-export type RenderElementInput = {
- /**
- * A round of renders, run together. Each entry names its own element, stretch and output and
- * inherits `run` and `reference_id`. The full preview is realized and drawn once; entries then cut
- * their own windows from that shared frame cache in order.
- */
- readonly renders?: readonly RenderElementInput[];
- /** Required for one render; a round carries them per entry and inherits `run` from the outer input. */
- readonly run?: string;
- readonly element?: string;
- readonly out?: string;
- readonly segment?: string;
- readonly selection?: string;
- /** A half-open token range, for a stretch the Script never named. */
- readonly tokens?: readonly [number, number];
- /**
- * A reference used only for comparison evidence and reference window selection. It never changes
- * preview timing, which is always the Source's own literal durations.
- */
- readonly reference_id?: string;
- /** Optional workspace root used by package-owned preview Sources. */
- readonly package_root?: string;
-};
-
-/** Timing metadata recorded beside one render, per Segment. */
-export type StandInTimingReport = {
- readonly reference_id: string | null;
- /** Kept as a compatibility union; native preview realization always reports `estimate`. */
- readonly basis: "reference" | "estimate" | "mixed";
- readonly segments: readonly StandInTiming[];
-};
-
-/**
- * The record `render_element` leaves beside its output.
- *
- * A comparison is made from a file, and the file alone says nothing about what timed the picture in
- * it. Writing that beside the output means `compare_reconstruction` can copy it into the comparison
- * log without re-deriving preview timing.
- */
-export type StandInSidecar = {
- readonly element: string;
- readonly window: AuthoringWindow;
- readonly timing_basis: "estimate";
- readonly timing: StandInTimingReport;
-};
-
-/** Where `render_element` writes the record of what timed a render. */
-export function standInSidecarPath(outPath: string): string {
- return `${outPath}.stand-in.json`;
-}
-
-function timingReport(reference: string | undefined, segments: readonly StandInTiming[]): StandInTimingReport {
- return {
- reference_id: reference ?? null,
- basis: "estimate",
- segments,
- };
-}
-
-export type RealizedAuthoringPreview = {
- readonly runPath: string;
- readonly runRoot: string;
- readonly projectRoot: string;
- readonly packageRoot: string;
- readonly svmlPath: string;
- readonly svml: string;
- readonly built: Preview;
- readonly previewMock: Awaited<ReturnType<typeof realizePreviewMock>>;
- readonly programFrames: number;
- readonly frameOfToken: readonly { readonly frame: number; readonly end: number }[];
- readonly selections: ReadonlyMap<string, AuthoringWindow>;
- readonly segmentTokenCounts: ReadonlyMap<string, number>;
-};
-
-/**
- * Realize the Author + Run graph with deterministic preview mocks, without drawing or encoding media.
- * Layout inspection and render_element share this path so they inspect the same Producer output.
- */
-export async function realizeAuthoringPreview(input: {
- readonly run: string;
- readonly runtime?: string;
- readonly package_root?: string;
-}): Promise<RealizedAuthoringPreview> {
- const cwd = invokedFrom();
- const runPath = resolve(cwd, input.run);
- const runRoot = dirname(runPath);
- const projectRoot = input.package_root === undefined
- ? (nearestPackageRoot(runRoot) ?? repositoryRoot())
- : resolve(cwd, input.package_root);
- const packageRoot = projectRoot;
- const runSource = await readFile(runPath, "utf8").catch(() => undefined);
- assert(runSource !== undefined, `cannot read ${runPath}`);
- const author = /<author\s+source="([^"]+)"/u.exec(runSource)?.[1];
- assert(author !== undefined, `${input.run} declares no <author source="…"/>`);
- const svmlPath = resolve(runRoot, author);
- const svml = await readFile(svmlPath, "utf8").catch(() => undefined);
- assert(svml !== undefined, `cannot read ${svmlPath}`);
-
- const distributionPackageRoot = videoCliDistribution.packageRoot;
- if (distributionPackageRoot === undefined) throw new Error("active Hypit Distribution has no package root");
- const registry = await loadStudioCompanionRegistry({ workspaceRoot: projectRoot, packageRoot, distributionPackageRoot });
- const domain = await loadStudioDomain({ run: runPath, workspaceRoot: projectRoot, packageRoot });
- const runtimePath = input.runtime === undefined ? undefined : resolve(cwd, input.runtime);
- const buildLibrary = await openStudioBuildLibrary(runtimePath, packageRoot, projectRoot, distributionPackageRoot);
- try {
- const original = await loadStudioRun({ run: runPath, domain, registry,
- ...(buildLibrary === undefined ? {} : { buildLibrary }) });
- const graph: CompiledGraph = {
- format: "hypit.graph@1",
- outputs: original.source.compiled.graph.outputs,
- candidates: [...original.source.compiled.graph.candidates, ...original.run.graph.candidates],
- operations: [...original.source.compiled.graph.operations, ...original.run.graph.operations],
- };
- const aliases = new Map(original.source.exports.map((item) => [item.ref, item.name] as const));
- for (const record of original.source.compiled.program.records) {
- const local = original.source.compiled.provenance.elements.flatMap((element) => element.records)
- .find((item) => item.id === record.id)?.local ?? record.id.split("::record::")[1];
- if (local !== undefined) aliases.set(record.id, local);
- }
- const previewMock = await realizePreviewMock({
- run: runPath, graph, records: original.source.compiled.program.records,
- targets: original.targets, timing: "estimate", author, aliases,
- });
- const previewRegistry = await loadStudioCompanionRegistry({ workspaceRoot: projectRoot, packageRoot, distributionPackageRoot });
- const previewDomain = await loadStudioDomain({ run: previewMock.previewRun, workspaceRoot: projectRoot, packageRoot });
- const previewBuildLibrary = await openStudioBuildLibrary(runtimePath, packageRoot, projectRoot, distributionPackageRoot);
- try {
- const loadedPreview = await loadStudioRun({
- run: previewMock.previewRun, domain: previewDomain, registry: previewRegistry,
- ...(previewBuildLibrary === undefined ? {} : { buildLibrary: previewBuildLibrary }),
- });
- const previewRun = { ...loadedPreview, attachments: [...loadedPreview.attachments, ...previewMock.attachments] };
- const inspection = inspectStudioRun(previewRegistry, previewRun.source, previewRun, new Set([
- "@hypit/mock-media@1#render-mock-image",
- "@hypit/mock-media@1#render-mock-video",
- "@hypit/mock-media@1#render-mock-silence",
- "@hypit/media-pipeline@1#inspect-media",
- "@hypit/media-pipeline@1#normalize-media",
- ]));
- const deterministicEndpoints = new EndpointRegistry();
- await createLocalMediaProvider({}).install(deterministicEndpoints);
- const built = await preview({
- source: previewRun.source,
- run: previewRun,
- domain: previewDomain,
- outputRefs: [inspection.filmComposition, ...inspection.projections.map((item) => item.ref)],
- compositionRef: inspection.filmComposition,
- projections: inspection.projections,
- endpoints: deterministicEndpoints,
- });
- const programFrames = Math.max(1, ...built.anchors.values());
- const frameOfToken = built.tokens.map((token) => ({
- frame: built.anchors.get(token.startAnchorId) ?? 0,
- end: built.anchors.get(token.endAnchorId) ?? 0,
- }));
- const parsed = parseScript(svmlPath, scriptBody(svml).text, scriptBody(svml).offset);
- const cursor = (from: number, to: number): AuthoringWindow => ({
- startFrame: frameOfToken[from]?.frame ?? 0,
- endFrameExclusive: frameOfToken[to - 1]?.end ?? programFrames,
- });
- const selections = new Map<string, AuthoringWindow>();
- const segmentTokenCounts = new Map<string, number>();
- for (const segment of parsed.segments) {
- selections.set(`segment:${segment.id}`, cursor(segment.tokenStart, segment.tokenEndExclusive));
- segmentTokenCounts.set(segment.id, segment.tokenEndExclusive - segment.tokenStart);
- }
- for (const selection of parsed.selections) {
- selections.set(selection.id, cursor(selection.open.boundary.tokenIndex, selection.close.boundary.tokenIndex));
- }
- return {
- runPath, runRoot, projectRoot, packageRoot, svmlPath, svml, built, previewMock,
- programFrames, frameOfToken, selections, segmentTokenCounts,
- };
- } finally {
- await previewBuildLibrary?.close();
- }
- } catch (error) {
- const issues = (error as { readonly issues?: unknown } | null)?.issues;
- if (Array.isArray(issues)) {
- throw new Error(["the Source does not project yet.", ...issues.slice(0, 12).map((issue) => ` ${String(issue)}`)].join("\n"));
- }
- throw error instanceof Error ? error : new Error(String(error));
- } finally {
- await buildLibrary?.close();
- }
-}
-
-/** Materialize a realized composition and its served artifacts without opening a browser. */
-export async function stageAuthoringPreview(realized: Pick<RealizedAuthoringPreview, "built" | "previewMock">): Promise<{
- readonly stage: string;
- readonly document: ReturnType<typeof compileHyperframesDocument>;
- readonly mediaTypes: ReadonlyMap<string, string>;
-}> {
- const stage = join(realized.previewMock.root, "stage");
- const document = compileHyperframesDocument(realized.built.composition, realized.built.space as ProgramSpace);
- await mkdir(stage, { recursive: true });
- const names = new Map<string, string>();
- const mediaTypes = new Map<string, string>();
- for (const [resource, file] of realized.built.served) {
- const extension = file.mediaType === "text/css" ? ".css"
- : file.mediaType === "application/javascript" || file.mediaType === "text/javascript" ? ".js"
- : "";
- const name = `${resource.replace(/[^a-z0-9]/giu, "")}${extension}`;
- await writeFile(join(stage, name), file.bytes);
- names.set(resource, name);
- mediaTypes.set(name, file.mediaType);
- }
- await writeFile(join(stage, "index.html"), materializeHyperframesHtml(document, (artifact) => {
- const name = names.get(artifact.resource);
- assert(name !== undefined, `the projection references Artifact ${artifact.resource}, which was not served`);
- return `./${name}`;
- }), "utf8");
- return { stage, document, mediaTypes };
-}
-
-/**
- * Render one element through the native preview realization path.
- *
- * The original Run is compiled once, its Author/Run Graph is handed to `@hypit/preview-mock`, and
- * the resulting temporary Run is previewed by Studio with only the local mock-media Provider. The
- * reference transcript remains comparison evidence; it never supplies preview timing.
- */
-export async function renderElement(input: RenderElementInput): Promise<Record<string, unknown>> {
- // A round carries these per entry; one render has to name all three itself.
- assert(input.run !== undefined, "run is required");
- assert(input.element !== undefined, "element is required");
- assert(input.out !== undefined, "out is required");
- const element = input.element;
- const out = input.out;
- const run = input.run;
- const cwd = invokedFrom();
- const outPath = resolve(cwd, out);
- // The directory the caller named, made rather than required. `out` is resolved against the working
- // directory the way `run` is, so a round whose entries name `renders/<element>.mp4` writes them
- // beside wherever it was launched from — and if nothing has created that directory, ffmpeg is the
- // one that reports it, as `Error opening output …: No such file or directory` against a path the
- // caller never typed.
- await ensureDir(dirname(outPath));
-
- // Preview is realized from the compiled Author + Run Graph. No reference transcript, placeholder
- // files, or hand-written SemanticTake values participate in this path.
- const segmentArgument = input.segment;
- const selectionArgument = input.selection;
- const tokensArgument = input.tokens === undefined ? undefined : tokenWindow(input.tokens, "tokens");
- const realized = await realizeAuthoringPreview({
- run,
- ...(input.package_root === undefined ? {} : { package_root: input.package_root }),
- });
- const {
- runPath, built, previewMock, programFrames, frameOfToken, selections, segmentTokenCounts,
- } = realized;
- const renderKey = `${resolve(runPath)}:${input.reference_id ?? "default"}`;
- const compareRoot = previewMock.root;
- const canvas = built.canvas;
- const frameRate = built.frameRate.numerator / built.frameRate.denominator;
- const timing: StandInTiming[] = [...selections.entries()]
- .filter(([id]) => id.startsWith("segment:"))
- .map(([id, window]) => ({ segment: id.slice("segment:".length), basis: "estimate", seconds: (window.endFrameExclusive - window.startFrame) / frameRate, frames: window.endFrameExclusive - window.startFrame, tokens: segmentTokenCounts.get(id.slice("segment:".length)) ?? 0, matched: 0 }));
- const mockedCount = previewMock.mocked.filter((item) => item.kind === "video" || item.kind === "audio").length;
- const imageIds = previewMock.mocked.filter((item) => item.kind === "image");
- const reference = undefined;
- const placed = built.tracks.find((track) => String(track.outputRef ?? "").endsWith(`::output::${element}.track`)
- || String(track.outputRef ?? "").includes(`::output::${element}.`));
- if (placed === undefined) {
- // An output is named `<id>.<output>`, and --element takes the id on its own. Listing the outputs
- // would hand the reader a name that fails the same way the one they passed did, so the last part
- // is dropped and one element placed twice is named once.
- const ids = [...new Set(built.tracks.map((track) => {
- const output = String(track.outputRef ?? "").split("::output::")[1];
- return output === undefined ? track.name : output.replace(/\.[^.]*$/u, "");
- }))];
- throw new Error([
- `the Source places no element named ${element}. --element takes the bare id, and these are placed:`,
- ...ids.map((id) => ` ${id}`),
- ].join("\n"));
- }
-
- // The window to render, named in words. A shot of the reference is found by the words spoken over it
- // and those words are a Segment or a Selection here, so no reference timestamp is ever read across.
- let window: AuthoringWindow = { startFrame: 0, endFrameExclusive: programFrames };
- if (tokensArgument !== undefined) {
- // A range given by index is the general form: a Segment and a Selection are both resolved into
- // one, and a Cue is a range that has no other way to be named.
- const [from, to] = tokensArgument;
- assert(from >= 0 && to > from && to <= frameOfToken.length,
- `token range [${from}, ${to}) is outside the Script's ${frameOfToken.length} words`);
- window = { startFrame: frameOfToken[from]!.frame, endFrameExclusive: frameOfToken[to - 1]!.end };
- } else if (selectionArgument !== undefined) {
- const marked = selections.get(selectionArgument);
- assert(marked !== undefined, `the Script marks no Selection ${selectionArgument}`);
- window = marked;
- } else if (segmentArgument !== undefined) {
- const marked = selections.get(`segment:${segmentArgument}`);
- assert(marked !== undefined, `the Script has no Segment ${segmentArgument}`);
- window = marked;
- }
-
- const clip = /\.(mp4|mov|webm)$/iu.test(outPath);
- const stage = join(compareRoot, "stage");
- const frames = join(compareRoot, "frames");
- // Staged and drawn once per Run and clock, however many windows are asked for.
- //
- // The picture does not depend on the element or the window — the whole program is drawn and the
- // window is cut out of these frames — so a round of eight looks over one Source is one browser
- // render and eight ffmpeg cuts. Rendering per entry would draw the same program eight times.
- //
- // The staged document is written inside the same run for the same reason the derived Run is: every
- // entry compiles the identical document to the identical path, `writeFile` truncates before it
- // writes, and the runtime is reading that path. An entry staging `index.html` while the runtime
- // loads it hands the runtime a file with no timeline in it, reported as
- // `Composition has zero duration`.
- await once(`draw:${renderKey}`, async () => {
- // Compile and materialize through the same no-media staging helper used by layout_check.
- await stageAuthoringPreview({ built, previewMock });
-
- await rm(frames, { recursive: true, force: true });
- await mkdir(frames, { recursive: true });
- // The runtime ships with the tree, not with the project. Resolving it against the package root
- // finds nothing whenever those two differ, which is every project that installs packages of its own.
- const hyperframesCli = createRequire(join(repositoryRoot(), "packages/provider-hyperframes-local/package.json"))
- .resolve("hyperframes/bin/hyperframes.mjs");
- const drawn = spawnSync(process.execPath, [
- hyperframesCli, "render", stage,
- "--format", "png-sequence", "--output", frames, "--fps", String(frameRate),
- "--workers", String(Math.max(1, cpus().length - 2)),
- // Halves the time and lands on the same pixels — verified frame for frame against a run without
- // it, PSNR reporting no error at all. The flag is marked experimental upstream; that is the
- // reason to check the pixels, which is done, rather than the reason to draw twice as long.
- "--experimental-fast-capture", "--no-best-effort", "--quiet",
- ], { encoding: "utf8", windowsHide: true, timeout: 600_000 });
- assert(drawn.status === 0, `the HyperFrames runtime refused: ${(drawn.stderr ?? "").trim().slice(-2000)}`);
- });
-
- const written = (await readdir(frames)).filter((name) => name.endsWith(".png")).sort();
- assert(written.length > 0, "the HyperFrames runtime wrote no frames");
- const first = Math.min(window.startFrame, written.length - 1);
- const last = Math.min(window.endFrameExclusive, written.length);
- // The runtime draws with an alpha channel and leaves unpainted area transparent. What a viewer is
- // under is the Film's own clear colour, so it is composited in here rather than left to whatever
- // opens the file: a reference clip is opaque, and an observer handed a transparent counterpart reads
- // the difference as design when it came from the encoding.
- const clear = built.canvas?.clearColor ?? "#000000";
- const background = `color=c=${clear.replace("#", "0x")}:s=${canvas.width}x${canvas.height}:r=${frameRate}`;
- const count = Math.max(1, clip ? last - first : 1);
- const encoded = spawnSync("ffmpeg", [
- "-hide_banner", "-loglevel", "error", "-y",
- "-f", "lavfi", "-i", background,
- "-framerate", String(frameRate), "-start_number", String(clip ? first : Math.min(first + Math.floor((last - first) / 2), written.length - 1)),
- "-i", join(frames, "frame_%06d.png"),
- "-filter_complex", "[0][1]overlay=shortest=1[v]", "-map", "[v]",
- "-frames:v", String(count),
- ...(clip ? ["-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "20"] : []),
- outPath,
- ], { encoding: "utf8", windowsHide: true, timeout: 600_000 });
- assert(encoded.status === 0, `ffmpeg refused: ${(encoded.stderr ?? "").trim().slice(-2000)}`);
-
- // What timed this picture, written where the picture is. `compare_reconstruction` is handed a path
- // and nothing else, so this is how the comparison log comes to say whether the stretch it looked at
- // ran at the reference's pace or at an estimate of it.
- const report = timingReport(reference, timing);
- const sidecar: StandInSidecar = { element, window, timing_basis: "estimate", timing: report };
- await writeFile(standInSidecarPath(outPath), `${JSON.stringify(sidecar, null, 2)}\n`, "utf8");
-
- return {
- element,
- out: outPath,
- compared: clip ? "clip" : "still",
- canvas,
- frame_rate: frameRate,
- window,
- program_frames: programFrames,
- timing: report,
- stand_in_ref: standInSidecarPath(outPath),
- mocked: { media: mockedCount, images: imageIds.length },
- ...(previewMock === undefined ? {} : { preview_mock: previewMock }),
- };
-}
-
-/**
- * Capture one package preview frame through the already-realized SVRun composition.
- *
- * This deliberately does not use HyperFrames' encoder: the normal SVRun → preview-mock → Producer
- * path has already produced the staged Composition, and a fixed local browser can seek that same
- * document directly. Package previews therefore cost one browser frame rather than a complete PNG
- * sequence while retaining the exact runtime semantics.
- */
-async function capturePreviewFrame(
- realized: Awaited<ReturnType<typeof realizeAuthoringPreview>>,
- frame: number,
- outPath: string,
-): Promise<void> {
- const { stage, mediaTypes } = await stageAuthoringPreview(realized);
- const { serve, browserPath } = await import("./layout.js");
- const local = await serve(stage, mediaTypes);
- const browser = await puppeteer.launch({ executablePath: browserPath(), headless: true, args: ["--no-sandbox", "--disable-dev-shm-usage"] });
- const page = await browser.newPage();
- try {
- await page.setViewport({ width: realized.built.canvas.width, height: realized.built.canvas.height, deviceScaleFactor: 1 });
- await page.goto(local.url, { waitUntil: "load", timeout: 60_000 });
- await page.evaluate(() => {
- for (const root of Array.from(document.querySelectorAll("[data-composition-id][data-width][data-height]"))) {
- const width = Number(root.getAttribute("data-width"));
- const height = Number(root.getAttribute("data-height"));
- if (root instanceof HTMLElement && Number.isFinite(width) && Number.isFinite(height)) {
- root.style.width = `${width}px`;
- root.style.height = `${height}px`;
- }
- }
- });
- await page.evaluate(async () => { await document.fonts.ready; });
- const fps = realized.built.frameRate.numerator / realized.built.frameRate.denominator;
- await page.evaluate(async ({ frame: at, fps: rate }) => {
- const runtime = window as unknown as { __hf?: { seek?: (seconds: number) => unknown }; __player?: { renderSeek?: (seconds: number) => unknown } };
- const seconds = at / rate;
- const seek = runtime.__player?.renderSeek ?? runtime.__hf?.seek;
- if (typeof seek === "function") await seek.call(runtime.__player ?? runtime.__hf, seconds);
- else window.dispatchEvent(new CustomEvent("hf-seek", { detail: { time: seconds } }));
- for (const clip of Array.from(document.querySelectorAll(".hypit-visual-present[data-start][data-duration]"))) {
- const start = Number(clip.getAttribute("data-start"));
- const duration = Number(clip.getAttribute("data-duration"));
- (clip as HTMLElement).style.display = seconds >= start && seconds < start + duration ? "" : "none";
- }
- await new Promise<void>((accept) => requestAnimationFrame(() => requestAnimationFrame(() => accept())));
- }, { frame, fps });
- const clip = await page.$eval("[data-composition-id]", (element) => {
- const rect = element.getBoundingClientRect();
- return { x: Math.max(0, rect.left), y: Math.max(0, rect.top), width: rect.width, height: rect.height };
- });
- await page.screenshot({ path: outPath, type: "png", clip });
- } finally {
- await page.close().catch(() => undefined);
- await browser.close().catch(() => undefined);
- await local.close().catch(() => undefined);
- }
-}
-
-export type RenderPreviewsInput = {
- readonly package_dirs: readonly string[];
-};
-
-/**
- * Draw a package's catalogue preview images from the package's own preview Source.
- *
- * A Manifest hands Studio and the README a picture per Surface, read off disk with `readFile`. Those
- * pictures were made once and committed, and nothing could make them again: a Recipe could change, a
- * Surface could gain a port, and the catalogue would go on showing what the package used to draw.
- *
- * So the sample lives in the package as a Source it can be drawn from. `preview/preview.svml`,
- * `preview/recipes.svs` and `preview/build.svrun` hold the package's own copy, its own Recipe values
- * and its own Canvas — nothing about any project — and `renderElement` draws them through the
- * real authoring path. A Surface that stopped working cannot produce a preview that looks fine.
- *
- * Which element draws which file comes from the Manifest's own naming: a Surface tagged `Track`
- * declares `preview/Track.png`, so `Track.png` is drawn from whichever element in the preview Source
- * carries that tag. No third file has to agree with the other two.
- */
-export async function renderPreviews(input: RenderPreviewsInput): Promise<Record<string, unknown>> {
- const directories = input.package_dirs;
- assert(directories.length > 0, "package_dirs must name at least one package directory");
-
- const cwd = invokedFrom();
- const pictures: Record<string, unknown>[] = [];
- let written = 0;
- for (const directory of directories) {
- const packageDir = resolve(cwd, directory);
- const previewDir = join(packageDir, "preview");
- const run = join(previewDir, "build.svrun");
- const svml = await readFile(join(previewDir, "preview.svml"), "utf8").catch(() => undefined);
- assert(svml !== undefined,
- `${directory} has no preview/preview.svml. Write one: the package's own copy, its own Recipe values, its own Canvas.`);
-
- // The alias the preview Source imports this package under. A preview Source places Tracks from
- // several packages — it needs a semantic spine and a Film like any other — and `Track` is a common
- // tag, so the tag alone would draw whichever one happened to be written first.
- const manifestJson = JSON.parse(await readFile(join(packageDir, "package.json"), "utf8").catch(() => "{}")) as { readonly name?: unknown };
- const specifier = manifestJson.name;
- assert(typeof specifier === "string", `${directory} has no package.json name`);
- const imported = [...svml.matchAll(/<import\s+as="([^"]+)"\s+from="(@[^"]+)@\d+"/gu)];
- const alias = imported.find((match) => match[2] === specifier)?.[1];
- assert(alias !== undefined,
- `${directory}'s preview Source does not import ${specifier}, so nothing in it is this package's`);
-
- // The pictures the Manifest promises, and the element tag each one is named for.
- const manifest = await readFile(join(packageDir, "src", "manifest.ts"), "utf8").catch(() => "");
- const promised = [...new Set([...manifest.matchAll(/previewImage\(\s*["']([^"']+)["']/gu)].map((match) => match[1] ?? ""))];
- const existing = (await readdir(previewDir).catch(() => [] as string[]))
- .filter((name) => /\.(png|jpe?g|svg)$/iu.test(name));
- const promisedPictures = promised.length > 0 ? promised : existing;
- assert(promisedPictures.length > 0, `${directory} promises no preview picture; nothing to draw`);
- let realized: Awaited<ReturnType<typeof realizeAuthoringPreview>> | undefined;
-
- for (const picture of promisedPictures) {
- if (/\.svg$/iu.test(picture)) {
- // Drawn by hand, not from a Source.
- pictures.push({ package_dir: directory, picture, drawn: false, reason: "drawn by hand, not from a Source" });
- continue;
- }
- const tag = basename(picture).replace(/\.[^.]+$/u, "");
- const element = new RegExp(`<${alias}:${tag}\\b[^>]*?\\bid="([^"]+)"`, "su").exec(svml)?.[1];
- assert(element !== undefined,
- `${directory} promises ${picture} but its preview Source places no <${alias}:${tag}> to draw it from`);
- const out = join(previewDir, picture);
- try {
- // Resolve the package from its workspace root rather than requiring a self-link under
- // packages/<slug>/node_modules. The preview Run still lives inside the package directory;
- // only dependency discovery needs the parent workspace.
- realized ??= await realizeAuthoringPreview({ run, package_root: dirname(dirname(packageDir)) });
- const placed = realized.built.tracks.find((track) => String(track.outputRef ?? "").endsWith(`::output::${element}.track`)
- || String(track.outputRef ?? "").includes(`::output::${element}.`));
- assert(placed !== undefined, `${directory} preview places no element named ${element}`);
- const trackId = (placed.value as { readonly id?: unknown } | null)?.id;
- assert(typeof trackId === "string", `${directory} preview element ${element} has no visual Track id`);
- const { stableFrameForTrack } = await import("./layout.js");
- const compositionTrack = realized.built.composition.tracks.find((track) => track.kind === "visual" && track.id === trackId);
- assert(compositionTrack !== undefined && compositionTrack.kind === "visual" && compositionTrack.presents.length > 0,
- `${directory} preview element ${element} has no visual Present`);
- const fallback = [...compositionTrack.presents].sort((left, right) => {
- const leftLength = left.span.endFrameExclusive - left.span.startFrame;
- const rightLength = right.span.endFrameExclusive - right.span.startFrame;
- return rightLength - leftLength || left.span.startFrame - right.span.startFrame || left.id.localeCompare(right.id);
- })[0]!;
- const frame = stableFrameForTrack(realized.built.composition, trackId)
- ?? fallback.span.startFrame + Math.floor((fallback.span.endFrameExclusive - fallback.span.startFrame - 1) / 2);
- await capturePreviewFrame(realized, frame, out);
- } catch (error) {
- throw new Error(`${directory} could not draw ${picture}: ${error instanceof Error ? error.message : String(error)}`);
- }
- // A catalogue picture is compared against nothing, so the record of what timed it is dropped
- // rather than committed beside the package's own pictures.
- await rm(standInSidecarPath(out), { force: true });
- pictures.push({ package_dir: directory, picture, drawn: true, tag, element, out });
- written += 1;
- }
- }
- return { package_dirs: directories, drew: written, pictures };
-}
diff --git a/packages/reference-video-tools/src/checks.ts b/packages/reference-video-tools/src/checks.ts
deleted file mode 100644
index 5a2d4ce2..00000000
--- a/packages/reference-video-tools/src/checks.ts
+++ /dev/null
@@ -1,1281 +0,0 @@
-import { readFile, readdir } from "node:fs/promises";
-import { spawnSync } from "node:child_process";
-import { dirname, join, resolve, sep } from "node:path";
-
-import { markupSurfaceHostFacetAbi } from "@hypit/markup";
-import type { RegisteredSurface } from "@hypit/markup";
-import { loadNodePackageSelection, physicalPackageName } from "@hypit/package-loader-node";
-import { parseScript, validateCaptionCueLengths } from "@hypit/script";
-import { videoCliDistribution } from "@hypit/video-cli";
-import { loadStudioCompanionRegistry } from "@hypit/studio/src/companion-profile.js";
-import { openStudioBuildLibrary } from "@hypit/studio/src/build-library.js";
-import { loadStudioDomain } from "@hypit/studio/src/domain.js";
-import { loadStudioRun } from "@hypit/studio/src/run.js";
-import { readStudioSession } from "@hypit/studio/src/session.js";
-import type { StudioSession } from "@hypit/studio/src/session.js";
-import { inspectStudioRun } from "@hypit/studio/src/studio-preflight.js";
-
-import { aliasPattern, invokedFrom, nearestPackageRoot, referenceRoot, scriptBody } from "./authoring.js";
-import { assert, readJson, round } from "./media.js";
-import { reviewPlan } from "./plan.js";
-import type { ReferenceState } from "./types.js";
-
-export type PreviewCheckInput = {
- readonly run: string;
- readonly runtime?: string;
-};
-
-export type ScriptCueCheckInput = { readonly run: string };
-
-type ScriptSource = { readonly path: string; readonly source: string };
-
-/** Read the Run's reachable Author Source closure without relying on compiler internals. */
-async function reachableAuthorSources(runPath: string): Promise<readonly ScriptSource[]> {
- const runSource = await readFile(runPath, "utf8");
- const author = /<author\s+source="([^"]+)"/u.exec(runSource)?.[1];
- if (author === undefined) throw new Error("AUTHOR_NOT_DECLARED");
- const queue = [resolve(dirname(runPath), author)];
- const seen = new Set<string>();
- const sources: ScriptSource[] = [];
- while (queue.length > 0) {
- const path = queue.shift()!;
- if (seen.has(path)) continue;
- seen.add(path);
- const source = await readFile(path, "utf8").catch(() => undefined);
- if (source === undefined) continue;
- sources.push({ path, source });
- for (const match of source.matchAll(/<import\s+[^>]*\bsource="([^"]+)"[^>]*\/?\s*>/gu)) {
- queue.push(resolve(dirname(path), match[1]!));
- }
- }
- return sources;
-}
-
-function scriptBodies(source: string): readonly { readonly text: string; readonly offset: number }[] {
- const bodies: { text: string; offset: number }[] = [];
- for (const opening of source.matchAll(/<script\b[^>]*>/gu)) {
- const start = opening.index! + opening[0].length;
- const end = source.indexOf("</script>", start);
- if (end < 0) throw new Error("SCRIPT_NOT_CLOSED");
- bodies.push({ text: source.slice(start, end), offset: start });
- }
- return bodies;
-}
-
-/** Validate every authored caption Cue before any preview or coverage work is attempted. */
-export async function scriptCueCheck(input: ScriptCueCheckInput): Promise<Record<string, unknown>> {
- const runPath = resolve(invokedFrom(), input.run);
- const runSource = await readFile(runPath, "utf8").catch(() => undefined);
- if (runSource === undefined) return { run: runPath, passed: false, errors: [{ code: "RUN_NOT_FOUND" }] };
- try {
- const sources = await reachableAuthorSources(runPath);
- if (sources.length === 0) {
- const author = /<author\s+source="([^"]+)"/u.exec(runSource)?.[1];
- return { run: runPath, passed: false, errors: [{ code: author === undefined ? "AUTHOR_NOT_DECLARED" : "AUTHOR_NOT_FOUND", ...(author === undefined ? {} : { path: resolve(dirname(runPath), author) }) }] };
- }
- const violations: Record<string, unknown>[] = [];
- const errors: Record<string, unknown>[] = [];
- const scriptSources: string[] = [];
- for (const item of sources) {
- let bodies: readonly { readonly text: string; readonly offset: number }[];
- try { bodies = scriptBodies(item.source); }
- catch (error) {
- errors.push({ code: "SCRIPT_PARSE", path: item.path, message: error instanceof Error ? error.message : String(error) });
- continue;
- }
- for (const body of bodies) {
- scriptSources.push(item.path);
- try {
- const parsed = parseScript(item.path, body.text, body.offset);
- violations.push(...validateCaptionCueLengths(parsed));
- } catch (error) {
- errors.push({ code: "SCRIPT_PARSE", path: item.path, message: error instanceof Error ? error.message : String(error) });
- }
- }
- }
- return { run: runPath, author: sources[0]!.path, sources: [...new Set(scriptSources)], passed: violations.length === 0 && errors.length === 0, max_words: 4, violations, ...(errors.length === 0 ? {} : { errors }) };
- } catch (error) {
- const code = error instanceof Error && error.message === "AUTHOR_NOT_DECLARED" ? "AUTHOR_NOT_DECLARED" : "AUTHOR_NOT_FOUND";
- return { run: runPath, passed: false, errors: [{ code, message: error instanceof Error ? error.message : String(error) }] };
- }
-}
-
-const AWAITING = "the Studio projection closure requires unresolved capabilities:";
-
-/**
- * Every Source the Run reaches, checked before anything is traced.
- *
- * One `hypit check` of the Run covers the closure: it compiles the Author the Run names, the Author's
- * `<import>`ed Recipe sheets and kits along with it, and it typechecks the project's own author
- * packages. A break in a kit is reported as `kits/street-interview-v1.svs:5993`, with the path the
- * file has from the project — so one check reads more than a list of them would and says where in the
- * same breath.
- */
-async function checkedSources(runPath: string): Promise<{ readonly ok: boolean; readonly output: string }> {
- // The CLI as it was launched, when this process was launched by it, and the Distribution's own
- // otherwise — `hypit-reference-video-tools` does not set the launcher variable, and the Distribution
- // root is where `bin/` sits in a checkout and in an install alike.
- const launcher = process.env.HYPIT_CLI_LAUNCHER?.trim();
- const distributionRoot = videoCliDistribution.packageRoot;
- if ((launcher === undefined || launcher.length === 0) && distributionRoot === undefined) {
- throw new Error("active Hypit Distribution has no package root");
- }
- const hypit = launcher !== undefined && launcher.length > 0
- ? launcher
- : join(distributionRoot!, "bin", "hypit.mjs");
-
- // The project is the package root, the way `hypit check` finds it when it is run from there. A
- // project inside a larger tree resolves none of its own packages without this.
- const checked = spawnSync(process.execPath, [hypit, "check", runPath, "--package-root", dirname(runPath)], {
- encoding: "utf8", windowsHide: true, timeout: 600_000,
- env: { ...process.env, HYPIT_STRICT_SCRIPT_CUES: "1" },
- });
- // Node writes its own deprecation warnings to the same stream the refusal arrives on, and this
- // output is read as the reason a check failed. Two lines about `module.register()` above the file
- // and position are two lines between the reader and the thing they came for.
- const output = `${checked.stdout ?? ""}${checked.stderr ?? ""}`
- .split("\n")
- .filter((line) => !/^\(node:\d+\)/u.test(line) && !line.startsWith("(Use `node --trace-"))
- .join("\n").trim();
- return { ok: checked.status === 0, output };
-}
-
-/**
- * Preview-check a reconstruction before it is delivered.
- *
- * `pnpm hypit check` proves a Source is legal; it proves nothing about whether
- * the tracks it declares can actually be built. A track that fails here fails
- * the same way when the author opens Studio, so find out now.
- *
- * The input is the Run Source, not the Author SVML. Studio's unit of work is
- * the Run — it reads the Author SVML back out of it — so a check that took the
- * `.svml` would be checking something Studio never opens.
- *
- * Studio names what it cannot resolve and stops, so preflight throwing is itself
- * the report rather than something to inspect around. Two kinds of issue come
- * back, and they mean opposite things. An unresolved *capability* means the
- * graph traced all the way to a Film and a semantic spine, and what remains is
- * work a Provider has to do — the ordinary state of a Source that declares its
- * generation rather than performing it. Every other issue means the graph is
- * wrong and no amount of generation will fix it. Only the second kind is
- * reported as unsound. `StudioPreflightError` carries `issues` as an array so
- * the two can be told apart rather than matched out of one joined message.
- */
-export async function previewCheck(
- input: PreviewCheckInput,
- roots: { readonly packageRoot: string },
-): Promise<Record<string, unknown>> {
- const cwd = invokedFrom();
- const runPath = resolve(cwd, input.run);
-
- // The sources first, then whether the graph traces. A Source that does not check has nothing useful
- // to say about tracing, and its refusal names the file and the position it is at; the same mistake
- // met while tracing arrives as whatever the compiler happened to fail on afterwards.
- const sources = await checkedSources(runPath);
- if (!sources.ok) return { run: runPath, sound: false, sources, refused: sources.output };
- const runtimePath = input.runtime === undefined ? undefined : resolve(cwd, input.runtime);
- const packageRoot = nearestPackageRoot(dirname(runPath)) ?? roots.packageRoot;
- const workspaceRoot = dirname(runPath);
-
- const distributionPackageRoot = videoCliDistribution.packageRoot;
- if (distributionPackageRoot === undefined) throw new Error("active Hypit Distribution has no package root");
-
- const registry = await loadStudioCompanionRegistry({ workspaceRoot, packageRoot, distributionPackageRoot });
- const domain = await loadStudioDomain({ run: runPath, workspaceRoot, packageRoot });
- const buildLibrary = await openStudioBuildLibrary(runtimePath, packageRoot, workspaceRoot, distributionPackageRoot);
-
- let session: StudioSession | undefined;
- let refusal: string | undefined;
- let awaiting: readonly string[] | undefined;
- try {
- const run = await loadStudioRun({
- run: runPath,
- domain,
- registry,
- ...(buildLibrary === undefined ? {} : { buildLibrary }),
- });
- // Preflight first, so an unopenable Run is reported as the refusal it is
- // rather than as whatever the build happens to fail on afterwards.
- inspectStudioRun(registry, run.source, run);
- session = await readStudioSession({
- domain,
- registry,
- run,
- workspaceRoot,
- revision: 0,
- });
- } catch (error) {
- const issues = (error as { readonly issues?: unknown } | null)?.issues;
- if (Array.isArray(issues) && issues.every((issue) => String(issue).startsWith(AWAITING))) {
- awaiting = issues.flatMap((issue) => String(issue).slice(AWAITING.length).split(",").map((capability) => capability.trim()));
- } else {
- refusal = error instanceof Error ? error.message : String(error);
- }
- } finally {
- // Close before reporting: the read-only Runtime connection must not outlive the check.
- await buildLibrary?.close();
- }
-
- if (refusal !== undefined) return { run: runPath, sound: false, sources, refused: refusal };
-
- if (awaiting !== undefined) {
- // The graph traced all the way to a Film and a semantic spine; what is left is
- // work a Provider has to do. That is sound for this report, and the delivery
- // measurements happen on the real Build either way.
- const noun = awaiting.length === 1 ? "capability" : "capabilities";
- return {
- run: runPath,
- sound: true,
- sources,
- summary: `every source checks; the graph is sound, waiting on ${awaiting.length} ${noun}.`,
- awaiting,
- };
- }
-
- const tracks = session!.snapshot.tracks;
- const problems: Record<string, unknown>[] = [];
- for (const track of tracks) {
- const { status, errors, output } = track.provenance;
- if (errors.length > 0) problems.push({ track: track.label, errors });
- else if (status === "unresolved") problems.push({ track: track.label, unresolved: output });
- }
-
- if (problems.length === 0) {
- return {
- run: runPath,
- sound: true,
- sources,
- summary: `every source checks; every Track resolved (${tracks.length}).`,
- tracks: tracks.length,
- };
- }
- return { run: runPath, sound: false, sources, problems };
-}
-
-export type ReconstructionCheckInput = {
- readonly run: string;
- readonly reference_id?: string;
- /** Runtime Profile used to resolve accepted Build Records in a pinned Run. */
- readonly runtime?: string;
-};
-
-/**
- * Which route wrote the Source, and therefore what "this element was looked at" means.
- *
- * `reconstruction` credits a comparison against the reference video, logged under that reference.
- * `description` credits a review of the element against what the author asked for, logged under the
- * project — there is no reference to compare to, so the question is conformance rather than
- * difference. Everything else the check does is the same on both: which elements draw, which words
- * nothing covers, which Frames reach past the Canvas, and which timed pictures empty their windows
- * are all read from the Source and know nothing about where the Source came from.
- */
-export type AuthoringCheckMode = "reconstruction" | "description";
-
-export type AuthoringCheckInput = ReconstructionCheckInput & {
- readonly mode?: AuthoringCheckMode;
-};
-
-
-/**
- * Where a description-authored project's reviews are logged.
- *
- * Beside the per-render directories rather than inside them: `render_element` empties its own
- * `.hypit/compare/<key>` on the way in, and a log kept under there would be deleted by the next
- * render of the element it is the evidence for.
- *
- * Under the project rather than under a reference, because a review is about one project. That is
- * also what spares this log the cross-project filter the comparison log needs: a reference is shared
- * between every reconstruction of one video, and a project's `.hypit` is not shared with anyone.
- */
-export function reviewLogPath(runPath: string): string {
- return join(dirname(runPath), ".hypit", "reviews.jsonl");
-}
-
-/**
- * Which timing basis the preview sidecar recorded for an element comparison.
- * Native preview realization records `estimate`; older or external pictures may be `not recorded`.
- */
-type TimingBasis = "reference" | "estimate" | "mixed" | "not recorded";
-
-/** One element that draws a Track, and the comparisons recorded against its id. */
-type ElementReport = {
- readonly element: string;
- readonly id: string;
- readonly package_name: string;
- readonly comparisons: number;
- /** The stretches it was compared against: a shot, or the words a Segment or Selection marks. */
- readonly compared_against: readonly string[];
- readonly state: string;
- readonly timing_basis: TimingBasis;
- readonly note?: string;
-};
-
-/** One timed picture and the Recipe key that decides what it draws once its material ends. */
-type PlaybackReport = {
- readonly id: string;
- readonly tag: string;
- readonly recipe: string;
- readonly playback: string;
-};
-
-/** One run of consecutive Script words that nothing filling the frame is drawn over. */
-type CoverageGap = {
- readonly segment: string;
- /** The words themselves, so a reader can find the stretch in the Script. */
- readonly words: string;
- readonly word_count: number;
-};
-
-/** How many words the Script has, and which runs of them carry no full-frame picture. */
-type CoverageReport = {
- readonly words: number;
- readonly gaps: readonly CoverageGap[];
-};
-
-/** A rectangle in Canvas pixels. The origin is top-left and y increases downward. */
-type Box = { readonly left: number; readonly top: number; readonly right: number; readonly bottom: number };
-
-/**
- * Whether some rectangles, together, leave no part of a Canvas unpainted.
- *
- * One picture filling the frame is the ordinary case and the cheap one. What this exists for is the
- * case where no single picture does: a split screen, a stacked pair, a grid of panels. Those cover the
- * delivery completely while every one of them is, on its own, an insert.
- *
- * The test is exact rather than an area sum, because rectangles overlap and overlapping ones add up to
- * more area than they cover. Cutting the Canvas at every edge any rectangle contributes leaves a grid
- * of cells, each of which is wholly inside a rectangle or wholly outside all of them; the Canvas is
- * covered when no cell is outside all of them. Rectangles are grouped by the Canvas they are rooted
- * in, since two rooted in different Canvases never meet.
- */
-function fillsACanvas(
- placed: readonly { readonly canvas: string; readonly box: Box }[],
- canvases: ReadonlyMap<string, Box>,
-): boolean {
- const byCanvas = new Map<string, Box[]>();
- for (const item of placed) byCanvas.set(item.canvas, [...(byCanvas.get(item.canvas) ?? []), item.box]);
- for (const [id, boxes] of byCanvas) {
- const canvas = canvases.get(id);
- if (canvas === undefined) continue;
- // Clipped to the Canvas: a Frame declared past the edge covers only the part of the picture that
- // exists, and counting the rest would let an overhang stand in for coverage it never provides.
- const clipped = boxes
- .map((box) => ({
- left: Math.max(box.left, canvas.left), top: Math.max(box.top, canvas.top),
- right: Math.min(box.right, canvas.right), bottom: Math.min(box.bottom, canvas.bottom),
- }))
- .filter((box) => box.right > box.left && box.bottom > box.top);
- if (clipped.length === 0) continue;
- const xs = [...new Set([canvas.left, canvas.right, ...clipped.flatMap((box) => [box.left, box.right])])]
- .filter((value) => value >= canvas.left && value <= canvas.right).sort((a, b) => a - b);
- const ys = [...new Set([canvas.top, canvas.bottom, ...clipped.flatMap((box) => [box.top, box.bottom])])]
- .filter((value) => value >= canvas.top && value <= canvas.bottom).sort((a, b) => a - b);
- let whole = true;
- for (let column = 0; whole && column + 1 < xs.length; column += 1) {
- for (let row = 0; whole && row + 1 < ys.length; row += 1) {
- const x = (xs[column]! + xs[column + 1]!) / 2;
- const y = (ys[row]! + ys[row + 1]!) / 2;
- whole = clipped.some((box) => x > box.left && x < box.right && y > box.top && y < box.bottom);
- }
- }
- if (whole) return true;
- }
- return false;
-}
-
-type FrameEdgeName = "left" | "top" | "right" | "bottom";
-type FrameDeclaration ={ readonly within: string; readonly edges: Readonly<Record<FrameEdgeName, string>> };
-type FrameGeometry = {
- readonly canvases: ReadonlyMap<string, Box>;
- readonly declared: ReadonlyMap<string, FrameDeclaration>;
- /** The rectangle a Frame or Canvas finally occupies, in the pixels of the Canvas it is rooted in. */
- readonly resolve: (id: string) => Box | undefined;
- /** Which Canvas a Frame is finally measured inside, found by following `within` up to one. */
- readonly canvasOf: (id: string) => string | undefined;
-};
-
-/**
- * Where every Canvas and Frame the Source declares actually sits.
- *
- * `within` names a Canvas or a parent Frame, and a length is a number followed by `px` or `%`, a
- * percentage resolving against the parent's width on the x axis and its height on the y axis. So a
- * Frame is resolved by resolving whatever it is written inside and measuring its own edges against
- * that rectangle, however deep the chain runs.
- *
- * The remaining Source-level checks use this only to determine which declared placements cover the
- * Canvas and whether the Canvas has the reference's aspect ratio. Layout quality is measured later
- * from the realized Composition DOM.
- */
-function frameGeometry(svml: string): FrameGeometry {
- const space = aliasPattern(svml, "@hypit/spatial", "space");
- const canvases = new Map<string, Box>();
- for (const match of svml.matchAll(new RegExp(`<(?:${space}):Canvas\\b([^>]*?)/?>`, "gsu"))) {
- const attributes = match[1] ?? "";
- const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];
- const width = Number(/\bwidth="(\d+)"/u.exec(attributes)?.[1]);
- const height = Number(/\bheight="(\d+)"/u.exec(attributes)?.[1]);
- if (id !== undefined && Number.isFinite(width) && Number.isFinite(height)) {
- canvases.set(id, { left: 0, top: 0, right: width, bottom: height });
- }
- }
-
- const declared = new Map<string, FrameDeclaration>();
- for (const match of svml.matchAll(new RegExp(`<(?:${space}):Frame\\b([^>]*?)/?>`, "gsu"))) {
- const attributes = match[1] ?? "";
- const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];
- const within = /\bwithin=\{([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];
- const written = (name: FrameEdgeName): string | undefined => new RegExp(`\\b${name}="([^"]+)"`, "u").exec(attributes)?.[1];
- const left = written("left");
- const top = written("top");
- const right = written("right");
- const bottom = written("bottom");
- if (id === undefined || within === undefined) continue;
- if (left === undefined || top === undefined || right === undefined || bottom === undefined) continue;
- declared.set(id, { within, edges: { left, top, right, bottom } });
- }
-
- const measure = (value: string, span: number, origin: number): number | undefined => {
- const parsed = /^\s*(-?\d+(?:\.\d+)?)\s*(%|px)\s*$/u.exec(value);
- if (parsed === null) return undefined;
- const number = Number(parsed[1]);
- return parsed[2] === "%" ? origin + span * number / 100 : origin + number;
- };
-
- const resolving = new Set<string>();
- const resolved = new Map<string, Box | undefined>();
- const resolve = (id: string): Box | undefined => {
- const canvas = canvases.get(id);
- if (canvas !== undefined) return canvas;
- if (resolved.has(id)) return resolved.get(id);
- // A Frame written inside itself is refused by `hypit check`; this keeps the walk finite anyway.
- if (resolving.has(id)) return undefined;
- const frame = declared.get(id);
- if (frame === undefined) return undefined;
- resolving.add(id);
- const parent = resolve(frame.within);
- resolving.delete(id);
- let box: Box | undefined;
- if (parent !== undefined) {
- const width = parent.right - parent.left;
- const height = parent.bottom - parent.top;
- const left = measure(frame.edges.left, width, parent.left);
- const top = measure(frame.edges.top, height, parent.top);
- const right = measure(frame.edges.right, width, parent.left);
- const bottom = measure(frame.edges.bottom, height, parent.top);
- if (left !== undefined && top !== undefined && right !== undefined && bottom !== undefined) {
- box = { left, top, right, bottom };
- }
- }
- resolved.set(id, box);
- return box;
- };
-
- const canvasOf = (id: string): string | undefined => {
- const seen = new Set<string>();
- let at: string | undefined = id;
- while (at !== undefined && !canvases.has(at)) {
- if (seen.has(at)) return undefined;
- seen.add(at);
- at = declared.get(at)?.within;
- }
- return at;
- };
-
- return { canvases, declared, resolve, canvasOf };
-}
-
-/**
- * Report what a route can still settle after the Source is written and before a Build runs: which
- * drawing elements nobody has looked at, which words of the Script nothing draws a full-frame
- * picture over, and which timed pictures are configured to stop before their window ends.
- *
- * Both routes end here, and `mode` is the only thing that differs. A reconstruction is looked at by
- * comparing each element against the reference; a description-authored program has no reference, so
- * each element is reviewed against what the author asked for. What counts as evidence and where it
- * is logged follow from that; everything else below is read from the Source and does not care.
- *
- * `preview_check` proves the graph is wired. It proves nothing about whether what the graph draws is
- * what was wanted, and a Source can pass every structural check while a component draws something
- * nobody asked for. Closing that gap is `references/element-review.md` and the route file beside it,
- * which is prose, and prose is what a route skips silently. This is the same requirement as a command
- * that answers whether it was met.
- *
- * What it checks is **participation, not convergence**. The round is deliberately bounded and may
- * stop with visible differences remaining, so requiring convergence here would contradict it. An
- * element that was compared once and stopped at its ceiling passes; an element nobody ever looked
- * at does not.
- *
- * Every element that draws is required, whichever package draws it. A Track from installed
- * vocabulary carries authored values exactly as a project-local one does — a caption Style's size,
- * colour and placement are as unverified as a new package's — and `render_element` stands in for the
- * speech, so it renders before a Build like any other.
- *
- * A look counts once it has been answered: in band (`complete`, the gemini observer) or handed out
- * and closed by whoever answered it (`pending` until then). Only `failed` is not a look. The report
- * cannot judge whether the stretch an element was looked at over actually showed it, so the stretches
- * are reported for a reader, and a lone look is called out rather than assumed meaningful.
- *
- * Each element also carries what timed the stand-ins it was compared over, read from the sidecar
- * `render_element` writes beside its output and carried into the log by `compare_reconstruction`.
- * That is reported rather than required: participation is the rule here, and an estimate-timed
- * comparison is a comparison. What it says is which elements have had their behaviour over their
- * window looked at and which have only had their layout looked at.
- *
- * Frame coverage is decided from the Source alone and is required: a word is either drawn over by
- * something bound to the whole picture or it is not. What covers is read from an element's own
- * opening tag — a full-bleed `frame=`, the `canvas=`, or a speech Track's full-frame `visual-frame=`
- * over the Segments whose Takes carry a picture — so a Track never inherits its children's `during=`
- * bindings.
- *
- * A project that places no drawing element passes with nothing to require.
- */
-export async function authoringCheck(
- input: AuthoringCheckInput,
- roots: { readonly packageRoot: string },
-): Promise<Record<string, unknown>> {
- const mode: AuthoringCheckMode = input.mode ?? "reconstruction";
- // What looking at an element is called on this route. A reconstruction compares it against the
- // reference; a description-authored program has nothing to compare to and is read against what the
- // author asked for. The arithmetic is the same either way, so only the word differs — and the keys
- // do not, so one shape describes both reports.
- const looked = mode === "reconstruction" ? "compared" : "reviewed";
- const looking = mode === "reconstruction" ? "comparison" : "review";
- const cwd = invokedFrom();
- const runPath = resolve(cwd, input.run);
- // Where the Source's imports resolve from. A project's own packages are installed against the
- // project, so the Run names its own root: the nearest directory at or above it holding a
- // `package.json`, which is the same walk `hypit check` makes. Taking it from the working directory
- // instead meant a Run named by a path resolved its packages somewhere else entirely, and the
- // project's own elements read as packages that do not exist.
- const packageRoot = nearestPackageRoot(dirname(runPath)) ?? roots.packageRoot;
- const distributionPackageRoot = videoCliDistribution.packageRoot;
-
- const runSource = await readFile(runPath, "utf8").catch(() => undefined);
- assert(runSource !== undefined, `cannot read ${runPath}`);
- const author = /<author\s+source="([^"]+)"/u.exec(runSource)?.[1];
- assert(author !== undefined, `${input.run} declares no <author source="…"/>`);
- const svmlPath = resolve(dirname(runPath), author);
- const svml = await readFile(svmlPath, "utf8").catch(() => undefined);
- assert(svml !== undefined, `cannot read ${svmlPath}`);
-
- // Every package this Source imports. The scope is whatever the Source wrote: a project that declares
- // a vocabulary gap and fills it publishes under its own scope, and those elements draw exactly as an
- // installed one does.
- // Attribute order is not significant in this markup, and a pattern that fixed it read a Source
- // written the other way round as importing nothing at all — then passed it, because nothing to
- // compare and nothing found look alike from here.
- const imports = [...svml.matchAll(/<import\b([^>]*?)\/?>/gsu)]
- .map((match) => ({
- alias: /\bas="([^"]+)"/u.exec(match[1] ?? "")?.[1] ?? "",
- specifier: /\bfrom="(@[^"@/]+\/[^"@]+)@\d+"/u.exec(match[1] ?? "")?.[1] ?? "",
- }))
- .filter((item) => item.alias.length > 0 && item.specifier.length > 0);
- if (imports.length === 0) {
- return {
- run: runPath,
- // Not one aliased package import could be read. Whether the Source truly imports nothing or
- // this failed to read it, what follows knows nothing about what the Source draws — and a
- // reading that came back empty is not a reconstruction that is covered.
- passed: false,
- summary: [`no aliased package import could be read from ${svmlPath}, so nothing is known about what this Source draws.`],
- elements: [],
- playback: [],
- uncovered: [],
- };
- }
-
- // Which of their Surfaces draw a Track. A Style Surface publishes a Style and draws nothing, so
- // requiring a comparison of it would be asking for a picture that does not exist.
- const specifiers = [...new Set(imports.map((item) => item.specifier))];
- // Logical submodules (for example @hypit/gpt-image/clean) are aliases owned by the
- // physical package (@hypit/gpt-image). Load the physical owner, but retain the logical
- // spelling for Source diagnostics and map drawing tags back to that owner.
- const physicalByLogical = new Map(specifiers.map((specifier) => [specifier, physicalPackageName(specifier)] as const));
- const physicalSpecifiers = [...new Set(specifiers.map((specifier) => physicalByLogical.get(specifier)!))];
- // One unresolvable package used to take the rest with it: the whole selection loads or none of it
- // does, and a swallowed failure left `drawingTags` empty, so every element in the Source read as
- // drawing nothing and the check passed on an empty answer. Fall back to loading them one at a time so
- // the ones that resolve still count, and keep the names of the ones that did not.
- const unresolved: string[] = [];
- // A project installs its own packages against itself, and `@hypit/*` comes from the Distribution
- // this command was launched from. Loading against the project root alone leaves the second kind
- // with nowhere to resolve from, so a project whose Source imports the standard vocabulary reports
- // every one of those imports as a package that does not exist.
- const loading = { ...(distributionPackageRoot === undefined ? {} : { fallbackRoots: [distributionPackageRoot] }) };
- const loaded = await loadNodePackageSelection(physicalSpecifiers, packageRoot, loading).catch(async () => {
- const each = await Promise.all(physicalSpecifiers.map(async (specifier) =>
- await loadNodePackageSelection([specifier], packageRoot, loading).catch(() => {
- unresolved.push(...specifiers.filter((logical) => physicalByLogical.get(logical) === specifier));
- return [];
- })));
- return each.flat();
- });
- const drawingTags = new Set<string>();
- for (const pack of loaded) {
- for (const facet of pack.contribution.hostFacets ?? []) {
- if (facet.abi !== markupSurfaceHostFacetAbi) continue;
- const surface = facet.implementation as RegisteredSurface;
- const draws = surface.outputs.some((output) =>
- output.module.name === "@hypit/composition" && (output.name === "VisualTrack" || output.name === "AudioTrack"));
- if (draws) drawingTags.add(`${pack.specifier}#${surface.tag}`);
- }
- }
-
- // Every use of one of those tags, by the id the Source gave it.
- const drawn: { readonly id: string; readonly tag: string; readonly alias: string; readonly specifier: string }[] = [];
- for (const { alias, specifier } of imports) {
- const physical = physicalByLogical.get(specifier) ?? specifier;
- const pattern = new RegExp(`<${alias}:([A-Za-z][A-Za-z0-9]*)\\b[^>]*?\\bid="([^"]+)"`, "gu");
- for (const match of svml.matchAll(pattern)) {
- const tag = match[1] ?? "";
- const id = match[2] ?? "";
- if (!drawingTags.has(`${physical}#${tag}`)) continue;
- if (!drawn.some((item) => item.id === id)) drawn.push({ id, tag, alias, specifier });
- }
- }
-
- // Recipe attributes are part of the authored declaration just like inline attributes. Keep a
- // small source-side index so coverage does not mistake a generated video declared by
- // `recipe={recipes.media.aroll}` for a voice merely because there is no `video=` attribute.
- const recipeSheets = new Map<string, Map<string, string>>();
- for (const match of svml.matchAll(/<import\s+as="([^"]+)"\s+source="([^"]+\.svs)"/gu)) {
- const alias = match[1] ?? "";
- const source = match[2] ?? "";
- const text = await readFile(resolve(dirname(svmlPath), source), "utf8").catch(() => undefined);
- if (text === undefined) continue;
- const recipes = new Map<string, string>();
- for (const recipe of text.matchAll(/([A-Za-z0-9_.-]+)\s*\{([^}]*)\}/gu)) recipes.set(recipe[1] ?? "", recipe[2] ?? "");
- recipeSheets.set(alias, recipes);
- }
-
- // Which Normalize ids carry a picture. A take normalized with `video="none"` is a voice: it has no
- // window to fill, and nothing it feeds puts anything on the Canvas.
- const moving = new Set<string>();
- const pipeline = aliasPattern(svml, "@hypit/media-pipeline", "pipeline");
- for (const match of svml.matchAll(new RegExp(`<(?:${pipeline}):Normalize\\b([^>]*?)/?>`, "gsu"))) {
- const attributes = match[1] ?? "";
- const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];
- const video = /\bvideo="([^"]+)"/u.exec(attributes)?.[1];
- const recipe = /\brecipe=\{([A-Za-z0-9_-]+)\.([A-Za-z0-9_.-]+)\}/u.exec(attributes);
- const recipeVideo = recipe === null ? undefined
- : /\bvideo\s*:\s*([^;\s]+)/u.exec(recipeSheets.get(recipe[1] ?? "")?.get(recipe[2] ?? "") ?? "")?.[1];
- if (id !== undefined && ((video !== undefined && video !== "none") || (video === undefined && recipeVideo !== undefined && recipeVideo !== "none"))) {
- moving.add(id);
- }
- }
-
- // ---- Timed pictures that stop before their window ends ----
- //
- // A generated take is asked for a whole number of seconds, and the window it has to fill comes from
- // speech the same Build synthesizes. The two lengths are arrived at separately and do not meet: the
- // estimate rounds, and the voice that finally speaks is not the voice the estimate predicted. So the
- // material is routinely shorter than the window it was made for, by anything from a rounding
- // remainder to a couple of seconds.
- //
- // What happens then is decided by one Recipe key. `playback` defaults to `once-start`, which draws
- // the material once and then draws nothing at all — `sampling.ts` emits no segment for the remainder,
- // so those frames fall through to whatever is beneath. Under a full-frame base that is the Film
- // background, which is black. `hold-start` pins the last frame for the rest of the window instead,
- // `loop-start` repeats, and `stretch` retimes to fit; any of the three fills it.
- //
- // This is `frame-coverage.md`'s inherited edge — "a generated take, which ends where its material
- // ends rather than where the shot should" — in the one form the route can settle before a Build, from
- // the Source and its Recipe sheets alone.
- const playbackReport = async (): Promise<readonly PlaybackReport[]> => {
- // Which aliases belong to a package that draws. Plenty of elements consume a normalized media
- // without placing it — `whisperx:SemanticTake` reads one to align speech against it and puts no
- // picture on the Canvas, so it has no window to fill and no occupancy to choose. Requiring the
- // alias to come from a package with a Track that outputs a VisualTrack keeps those out, and keeps
- // the Items and Sequences written inside such a Track in, without this needing to know their tags.
- const draws = new Set(imports
- .filter(({ specifier }) => [...drawingTags].some((key) => key.startsWith(`${specifier}#`)))
- .map(({ alias }) => alias));
-
- // Every element that places one of those pictures, and what its Recipe says to do with a window the
- // material does not fill.
- const running: PlaybackReport[] = [];
- for (const match of svml.matchAll(/<([a-z][a-z0-9-]*:[A-Za-z][A-Za-z0-9]*)\b([^>]*?)\/?>/gsu)) {
- const tag = match[1] ?? "";
- const attributes = match[2] ?? "";
- if (!draws.has(tag.slice(0, tag.indexOf(":")))) continue;
- const media = /\bmedia=\{([A-Za-z0-9_-]+)\.media\}/u.exec(attributes)?.[1];
- if (media === undefined || !moving.has(media)) continue;
- const appearance = /\bappearance=\{([A-Za-z0-9_-]+)\.([A-Za-z0-9_.-]+)\}/u.exec(attributes);
- const body = appearance === null ? undefined : recipeSheets.get(appearance[1] ?? "")?.get(appearance[2] ?? "");
- const playback = body === undefined ? undefined : /\bplayback\s*:\s*([A-Za-z-]+)/u.exec(body)?.[1];
- if (playback !== undefined && playback !== "once-start" && playback !== "once-end") continue;
- running.push({
- id: /\bid="([^"]+)"/u.exec(attributes)?.[1] ?? tag,
- tag,
- recipe: appearance === null ? "no appearance Recipe" : `${appearance[1]}.${appearance[2]}`,
- playback: playback ?? "once-start, by default",
- });
- }
- return running;
- };
-
- // ---- Words nothing draws a full-frame picture over ----
- //
- // A delivery is a picture at every frame; a Source is a list of placements. A word no full-frame
- // element is drawn over is an instant the Source hands to whatever lies beneath it, and beneath the
- // last placement is the Film background, which is black. That is `frame-coverage.md`'s rule in the
- // one form the Source settles on its own, and it needs no threshold: it is a yes or no per word.
- //
- // What covers is read from each element's own opening tag and never from what is written inside it.
- // A Track holding forty Lines with `during=` bindings draws nothing itself; crediting it with its
- // children's bindings would report full coverage for a Source that places text over nothing.
- const coverageReport = (): CoverageReport => {
- const body = scriptBody(svml);
- const parsed = parseScript(svmlPath, body.text, body.offset);
-
- // Where every Frame and Canvas the Source declares actually sits, so a word can be asked whether
- // the pictures drawn over it add up to the whole one rather than whether any single one does.
- //
- // A split screen is why. Two people on a podcast, one above the other, each half the height: the
- // one speaking owns the Segment and the one listening covers the rest, and neither Frame is the
- // whole picture. Read one Frame at a time, both are inserts and every word of the program is
- // uncovered — which is the opposite of what the delivery shows.
- const geometry = frameGeometry(svml);
-
- // What each word has drawn over it, as rectangles in the pixels of the Canvas they are rooted in.
- const claims: { readonly canvas: string; readonly box: Box }[][] = Array.from(
- { length: parsed.tokens.length }, () => []);
- const claimSpan = (from: number, to: number, canvas: string, box: Box): void => {
- for (let index = Math.max(0, from); index < Math.min(to, claims.length); index += 1) {
- claims[index]!.push({ canvas, box });
- }
- };
- const claimSegment = (id: string, canvas: string, box: Box, stops = Infinity): void => {
- const segment = parsed.segments.find((item) => item.id === id);
- if (segment !== undefined) claimSpan(segment.tokenStart, Math.min(segment.tokenEndExclusive, stops), canvas, box);
- };
- const claimSelection = (id: string, canvas: string, box: Box, stops = Infinity): void => {
- const selection = parsed.selections.find((item) => item.id === id);
- if (selection !== undefined) {
- claimSpan(selection.open.boundary.tokenIndex, Math.min(selection.close.boundary.tokenIndex, stops), canvas, box);
- }
- };
- // A Frame drawn into, resolved to where it sits. Naming the Canvas directly is the whole of it.
- const placement = (frame: string | undefined, canvas: string | undefined): { readonly canvas: string; readonly box: Box } | undefined => {
- const named = frame ?? canvas;
- if (named === undefined) return undefined;
- const rooted = geometry.canvasOf(named);
- const box = geometry.resolve(named);
- if (rooted === undefined || box === undefined) return undefined;
- return { canvas: rooted, box };
- };
-
- // Every element that draws a picture, and the words its own `during=` claims. `program` is the
- // whole Script; a Segment or a Selection is the words it marks.
- for (const match of svml.matchAll(/<[a-z][a-z0-9-]*:[A-Za-z][A-Za-z0-9]*\b([^>]*?)\/?>/gsu)) {
- const attributes = match[1] ?? "";
- const where = placement(
- /\bframe=\{([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1],
- /\bcanvas=\{([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1]);
- if (where === undefined) continue;
- // A Moment in `until=` cuts the span short, so the words after it have this picture over them in
- // the Source and not in the delivery. Reading `during=` alone would let an element that leaves
- // half way through the program answer for the whole of it.
- const until = /\buntil=\{story\.moment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];
- const stops = until === undefined
- ? claims.length
- : parsed.moments.find((item) => item.id === until)?.boundary.tokenIndex ?? claims.length;
- if (/\bduring="program"/u.test(attributes)) claimSpan(0, stops, where.canvas, where.box);
- const bound = /\bduring=\{story\.(segment|selection)\.([A-Za-z0-9_-]+)\}/u.exec(attributes);
- if (bound?.[1] === "segment") claimSegment(bound[2] ?? "", where.canvas, where.box, stops);
- if (bound?.[1] === "selection") claimSelection(bound[2] ?? "", where.canvas, where.box, stops);
- }
-
- // A speech Track draws each Take it assembles into its own `visual-frame`, so that Frame is what
- // the Take's Segment has over it. A Take fed by a Normalize with `video="none"` is a voice and
- // puts nothing there, so its Segment is spoken over whatever is already on screen.
- const takes = new Map<string, { readonly segment: string; readonly picture: boolean }>();
- const whisperx = aliasPattern(svml, "@hypit/whisperx", "whisperx");
- const speech = aliasPattern(svml, "@hypit/speech-track", "speech");
- for (const match of svml.matchAll(new RegExp(`<(?:${whisperx}):SemanticTake\\b([^>]*?)/?>`, "gsu"))) {
- const attributes = match[1] ?? "";
- const id = /\bid="([^"]+)"/u.exec(attributes)?.[1];
- const segment = /\bsegment=\{story\.segment\.([A-Za-z0-9_-]+)\}/u.exec(attributes)?.[1];
- const media = /\bmedia=\{([A-Za-z0-9_-]+)\.media\}/u.exec(attributes)?.[1];
- if (id === undefined || segment === undefined) continue;
- takes.set(id, { segment, picture: media !== undefined && moving.has(media) });
- }
- for (const track of svml.matchAll(new RegExp(`<(?:${speech}):Track\\b([^>]*?)>(.*?)</(?:${speech}):Track>`, "gsu"))) {
- const where = placement(/\bvisual-frame=\{([A-Za-z0-9_-]+)\}/u.exec(track[1] ?? "")?.[1], undefined);
- if (where === undefined) continue;
- for (const take of (track[2] ?? "").matchAll(new RegExp(`<(?:${speech}):Take\\b[^>]*?\\bsource=\\{([A-Za-z0-9_-]+)\\.take\\}`, "gu"))) {
- const named = takes.get(take[1] ?? "");
- if (named?.picture === true) claimSegment(named.segment, where.canvas, where.box);
- }
- }
-
- const covered = claims.map((placed) => placed.length > 0 && fillsACanvas(placed, geometry.canvases));
-
- // The uncovered words as runs, cut at Segment boundaries so each run is named by the Segment a
- // reader would look in to find it.
- const gaps: CoverageGap[] = [];
- for (const segment of parsed.segments) {
- let run: number[] = [];
- const close = (): void => {
- if (run.length === 0) return;
- gaps.push({ segment: segment.id, words: run.map((index) => parsed.tokens[index]!.text).join(" "), word_count: run.length });
- run = [];
- };
- for (let index = segment.tokenStart; index < segment.tokenEndExclusive; index += 1) {
- if (covered[index] === true) close();
- else run.push(index);
- }
- close();
- }
- return { words: parsed.tokens.length, gaps };
- };
-
- const running = await playbackReport();
-
- if (drawn.length === 0 && running.length === 0) {
- // Nothing placed and nothing seen are the same answer from here, and only one of them is a pass.
- // A project whose vocabulary resolved and placed no drawing element has nothing to require; one
- // whose packages did not resolve, or none of whose packages publishes a Surface that draws, has
- // been read by something that could not see — and the names of what it could not see were being
- // dropped on the way out.
- const blind = unresolved.length > 0 || drawingTags.size === 0;
- return {
- run: runPath,
- passed: !blind,
- summary: [unresolved.length > 0
- ? `${unresolved.length} imported package${unresolved.length === 1 ? "" : "s"} could not be resolved from ${packageRoot}, so nothing is known about what the Source draws.`
- : drawingTags.size === 0
- ? `no package the Source imports publishes a Surface that draws, so nothing is known about what the Source draws.`
- : "no drawing element is placed in the Source; nothing to require."],
- elements: [],
- playback: [],
- uncovered: [],
- ...(unresolved.length === 0 ? {} : {
- unresolved_packages: {
- names: unresolved,
- package_root: packageRoot,
- note: "A project that publishes its own packages resolves them from the project root, where "
- + "`packages/<name>/package.json` carries the scoped name. Run the command from that directory, "
- + "or pass --package-root <project directory>.",
- },
- }),
- };
- }
-
- // What a round has to look at, decided from the Source rather than left to whoever runs it.
- // Where each stretch's picture goes, named here rather than by whoever reads this.
- //
- // A round is three commands — this one, `render_element --batch`, `compare_reconstruction --batch` —
- // and the middle one writes exactly what the last one reads. A path invented at the call site has to
- // survive being written down twice and stay identical; a path named here survives by never being
- // written down at all. `renders/` beside the Run is where a project already keeps them.
- //
- // Named for the range and not for `named`: the range is what identifies an entry, so two entries of
- // one element always differ in it, while `named` is a sentence written to be read.
- const plan = reviewPlan({ svml, svmlPath, scriptBody: scriptBody(svml), drawn })
- .map((entry) => ({
- ...entry,
- out: join(dirname(runPath), "renders", `${entry.element}-${entry.tokens[0]}-${entry.tokens[1]}.mp4`),
- }));
- // The same parse the plan was built from, so a look recorded against a Segment can be resolved to
- // the words that Segment marks and compared with a plan entry on one axis.
- const plannedBody = scriptBody(svml);
- const plannedScript = parseScript(svmlPath, plannedBody.text, plannedBody.offset);
-
- const coverage = coverageReport();
- const uncoveredWords = coverage.gaps.reduce((sum, gap) => sum + gap.word_count, 0);
-
- // The Source does not name the reference it reconstructs, so a single prepared reference is used
- // when there is exactly one and named explicitly when there is more than one.
- //
- // Only on the route that has a reference. A description-authored project has none, and resolving
- // one here is what used to stop this command before it reached the checks that have nothing to do
- // with a reference — the `playback` refusal and frame coverage are read from
- // the Source alone, and were unreachable to that route for no reason but this block.
- const preparedRoot = referenceRoot();
- let reference: string | undefined;
- if (mode === "reconstruction") {
- const prepared = (await readdir(preparedRoot, { withFileTypes: true }).catch(() => []))
- .filter((entry) => entry.isDirectory()).map((entry) => entry.name);
- reference = input.reference_id;
- if (reference === undefined) {
- assert(prepared.length > 0, `no prepared reference under ${preparedRoot}; run prepare_reference first`);
- assert(prepared.length === 1, `${prepared.length} prepared references; pass --reference-id (${prepared.join(", ")})`);
- reference = prepared[0]!;
- }
- assert(prepared.includes(reference), `reference ${reference} is not prepared under ${preparedRoot}`);
- }
-
- // The reference's shape against the Canvas's.
- //
- // The reference's own pixel dimensions size nothing on this route — a render is drawn at the
- // Canvas's size — but their proportion has to agree. A Canvas of another shape puts two
- // differently-proportioned pictures in front of the observer, and the differences it reports are
- // the ones the Canvas made rather than the ones the element made. It is the precondition that
- // fails quietly: every comparison in the round succeeds, and every one of them is about the wrong
- // thing, which is why it is a refusal here rather than a note.
- const aspect: { readonly canvas: string; readonly canvas_size: string; readonly reference_size: string }[] = [];
- if (mode === "reconstruction" && reference !== undefined) {
- const video = (await readJson<ReferenceState>(join(preparedRoot, reference, "state.json")))?.video;
- if (video !== undefined && video.width > 0 && video.height > 0) {
- const wanted = video.width / video.height;
- for (const [id, box] of frameGeometry(svml).canvases) {
- const width = box.right - box.left;
- const height = box.bottom - box.top;
- if (!(width > 0 && height > 0)) continue;
- // One percent, so a Canvas that is the reference's shape at another size agrees: 1440x2560 and
- // 1080x1920 are both 9:16, and a reference whose own dimensions are odd is within it. Nothing
- // wider — 1080x1920 against 1080x1440 is the case this exists to catch.
- if (Math.abs(width / height - wanted) <= wanted * 0.01) continue;
- aspect.push({
- canvas: id,
- canvas_size: `${width}x${height}`,
- reference_size: `${video.width}x${video.height}`,
- });
- }
- }
- }
-
- type LoggedComparison = {
- readonly run?: string;
- readonly element?: string;
- readonly shot_id?: string;
- readonly range?: { readonly segment?: string; readonly selection?: string; readonly tokens?: readonly [number, number] };
- readonly status: string;
- readonly id?: string;
- readonly image_path?: string;
- readonly stand_in?: { readonly timing?: { readonly basis?: string } };
- };
- // What the comparison was made against, as a reader would name it: a shot, or the words a Segment
- // or Selection marks.
- const against = (entry: LoggedComparison): string =>
- entry.shot_id
- ?? (entry.range?.segment === undefined ? undefined : `segment ${entry.range.segment}`)
- ?? (entry.range?.selection === undefined ? undefined : `selection ${entry.range.selection}`)
- ?? "an unnamed stretch";
- const legacyComparisonPath = mode === "reconstruction"
- ? join(preparedRoot, reference!, "comparisons.jsonl")
- : undefined;
- const logPath = mode === "reconstruction"
- ? join(dirname(runPath), ".hypit", "comparisons.jsonl")
- : reviewLogPath(runPath);
- // A reference is keyed by the video, and rightly: its observations are about that video and cost
- // real money, so two reconstructions of one file share them. Its comparisons are not about the
- // video — they are about one reconstruction — and they were being kept in the same place, so a
- // second project reconstructing the same file was credited with the first one's looking, over
- // Segments its own Script does not contain.
- //
- // What separates them is already on every row: the render it compared. A render belongs to the
- // project that produced it, so a row whose picture is not under this project is not this project's
- // evidence. Nothing has to move and no earlier log has to be migrated.
- //
- // A review log needs none of this: it is already the project's own, so every row in it is this
- // project's evidence and there is nothing to separate.
- const inThisProject = (entry: LoggedComparison): boolean => {
- if (mode === "description") return true;
- if (entry.run !== undefined) return resolve(entry.run) === runPath;
- if (entry.image_path === undefined) return false;
- const at = resolve(entry.image_path);
- return at === packageRoot || at.startsWith(`${packageRoot}${sep}`);
- };
- const logPaths = [logPath, legacyComparisonPath].filter((path, index, all): path is string => path !== undefined && all.indexOf(path) === index);
- const everything = (await Promise.all(logPaths.map(async (path) => (await readFile(path, "utf8").catch(() => ""))
- .split("\n").filter((line) => line.trim().length > 0)
- .flatMap((line) => { try { return [JSON.parse(line) as LoggedComparison]; } catch { return []; } }))))
- .flat();
- const elsewhere = everything.length - everything.filter(inThisProject).length;
- const logged = everything.filter(inThisProject);
- // A pair handed out and not yet reported on. Naming these separately is what keeps an element with
- // three open comparisons from reading as one that was never looked at.
- const awaiting = logged.filter((entry) => entry.status === "pending");
- const log = logged
- // A comparison counts when it was answered. The observer that uploads answers in band and the
- // entry is written `complete`; the observer that reads pictures is handed the pair and closes its
- // entry with `record_observation --key comparison:<id>`. An entry still `pending` is a pair nobody
- // has reported on, and crediting it credited the tool call rather than the look.
- .filter((entry) => entry.status === "complete");
-
- // Which stretches each element was compared against, in order. The report cannot judge whether a
- // stretch was the right one to compare against — it does not know what the element draws — so it
- // reports them and lets a reader notice a component compared against one that never showed it.
- const rounds = new Map<string, string[]>();
- // What timing basis each comparison recorded. Native preview realization is deterministic estimate
- // timing; a missing sidecar is the only unknown case.
- const bases = new Map<string, TimingBasis[]>();
- for (const entry of log) {
- if (entry.element === undefined) continue;
- const seen = rounds.get(entry.element) ?? [];
- seen.push(against(entry));
- rounds.set(entry.element, seen);
- const recorded = entry.stand_in?.timing?.basis;
- const basis: TimingBasis = recorded === "reference" || recorded === "estimate" || recorded === "mixed" ? recorded : "not recorded";
- bases.set(entry.element, [...(bases.get(entry.element) ?? []), basis]);
- }
- // One answer per element, over every comparison it has. Agreement carries through; disagreement is
- // `mixed` for compatibility with historical logs.
- const timingBasis = (id: string): TimingBasis => {
- const recorded = new Set(bases.get(id) ?? []);
- if (recorded.size === 0) return "not recorded";
- return recorded.size === 1 ? [...recorded][0]! : "mixed";
- };
-
- // Which of the plan's entries have been answered.
- //
- // A logged look and a planned one are compared as word ranges, because that is the one form both
- // can always be put in: a look recorded against a Segment resolves to the words that Segment marks,
- // and a look recorded against a Cue was never anything else. Comparing the names instead would let
- // a Cue in the middle of a Segment match the Segment, and the coverage this report claims would be
- // wider than the coverage it has.
- const asTokens = (entry: LoggedComparison): string | undefined => {
- if (entry.range?.tokens !== undefined) return `${entry.range.tokens[0]}-${entry.range.tokens[1]}`;
- const segment = entry.range?.segment === undefined
- ? undefined
- : plannedScript.segments.find((item) => item.id === entry.range!.segment);
- if (segment !== undefined) return `${segment.tokenStart}-${segment.tokenEndExclusive}`;
- const selection = entry.range?.selection === undefined
- ? undefined
- : plannedScript.selections.find((item) => item.id === entry.range!.selection);
- if (selection !== undefined) return `${selection.open.boundary.tokenIndex}-${selection.close.boundary.tokenIndex}`;
- // A shot names a cut in the reference's picture rather than a range of the Script's words, so it
- // answers no plan entry. It still counts as having looked, which `rounds` already records.
- return undefined;
- };
- const answered = new Set(log.flatMap((entry) => {
- const tokens = asTokens(entry);
- return entry.element === undefined || tokens === undefined ? [] : [`${entry.element} ${tokens}`];
- }));
- const owed = plan.filter((entry) => !answered.has(`${entry.element} ${entry.tokens[0]}-${entry.tokens[1]}`));
-
- const never = drawn.filter((element) => (rounds.get(element.id) ?? []).length === 0);
- const unlabelled = log.filter((entry) => entry.element === undefined).length;
- // A misspelled --element is otherwise silent: the comparison happens, the log grows, and the element
- // it was meant for stays at zero for ever.
- const known = new Set(drawn.map((item) => item.id));
- const unknown = [...new Set([...rounds.keys()].filter((id) => !known.has(id)))];
-
- const elements: ElementReport[] = drawn.map((element) => {
- const stretches = rounds.get(element.id) ?? [];
- const state = stretches.length === 0
- ? `never ${looked}`
- : `${stretches.length} ${looking}${stretches.length === 1 ? "" : "s"} against ${[...new Set(stretches)].join(", ")}`;
- return {
- element: `${element.alias}:${element.tag}`,
- id: element.id,
- package_name: element.specifier,
- comparisons: stretches.length,
- compared_against: [...new Set(stretches)],
- state,
- timing_basis: timingBasis(element.id),
- // The report cannot judge whether the stretch chosen actually showed the element, so a lone
- // comparison is called out for a reader to confirm rather than silently accepted.
- ...(stretches.length === 1 ? { note: `single ${looking} — confirm this stretch shows the element, not a look-alike` } : {}),
- };
- });
-
- // Elements whose comparisons say nothing about how they behave over their window: every recorded
- // comparison was drawn at the estimator's pace, or its picture carried no record of what timed it.
- const untimed = elements.filter((element) => element.comparisons > 0 && element.timing_basis === "not recorded");
-
- const summary: string[] = [];
- if (never.length === 0 && running.length === 0) {
- summary.push(`every drawing element has been ${looked} (${elements.length}), and every timed picture fills its window.`);
- }
- if (never.length > 0) summary.push(`${never.length} of ${elements.length} elements have never been ${looked}.`);
- if (owed.length > 0) {
- summary.push(`${owed.length} of ${plan.length} stretch${plan.length === 1 ? "" : "es"} the Source asks for `
- + `${owed.length === 1 ? "has" : "have"} not been ${looked}.`);
- }
- if (running.length > 0) {
- summary.push(`${running.length} window${running.length === 1 ? "" : "s"} will empty before ${running.length === 1 ? "it ends" : "they end"}.`);
- }
- summary.push(uncoveredWords === 0
- ? `every word of the Script is drawn over by something that fills the frame (${coverage.words}).`
- : `${uncoveredWords} of ${coverage.words} words are drawn over by nothing that fills the frame.`);
- const compared = elements.filter((element) => element.comparisons > 0).length;
- if (compared > 0 && mode === "reconstruction") {
- summary.push(untimed.length === 0
- ? `every compared element was looked at over the deterministic estimate-timed preview (${compared}).`
- : `${untimed.length} of ${compared} compared elements have no recorded preview timing basis.`);
- }
- // On the description route every stand-in is estimate-timed, because the author's literal durations
- // are the only clock there is until the Build synthesizes the speech. Reporting that as a shortfall
- // would name every element every time and ask for a re-render against a reference that does not
- // exist, so it is stated once as what it is.
- if (compared > 0 && mode === "description") {
- summary.push(`every ${looking} was made over a stand-in timed by the Source's literal durations (${compared}); how each element `
- + "sits against the speech the Build synthesizes is settled once that speech exists.");
- }
- if (unresolved.length > 0) {
- summary.push(`${unresolved.length} imported package${unresolved.length === 1 ? "" : "s"} could not be resolved, `
- + "so nothing is known about what they draw.");
- }
- if (elsewhere > 0) {
- summary.push(`${elsewhere} comparison${elsewhere === 1 ? "" : "s"} in this reference's log compared a render `
- + "outside this project and are not counted here.");
- }
- if (awaiting.length > 0) {
- summary.push(`${awaiting.length} ${looking}${awaiting.length === 1 ? " is" : "s are"} still waiting for `
- + `${mode === "reconstruction" ? "the differences" : "the findings"} to be recorded.`);
- }
-
- return {
- run: runPath,
- ...(reference === undefined ? {} : { reference_id: reference }),
- plan,
- // The round as the two commands that run it already take it. `render_element --batch` reads
- // `renders` out of the file it is given and `compare_reconstruction --batch` reads `comparisons`,
- // so this command's own output is a batch file for both and the round is three commands with
- // nothing written by hand in between. Both are the stretches still owed rather than the whole
- // plan: what has already been looked at does not want rendering again.
- ...(owed.length === 0 ? {} : {
- renders: owed.map((entry) => ({ element: entry.element, tokens: entry.tokens, out: entry.out })),
- }),
- // Only where there is a reference to compare against. The description route's second command is
- // `review_element`, whose entries carry an `intent_file` saying what the element was asked to be,
- // and that is written rather than derived.
- ...(owed.length === 0 || mode !== "reconstruction" ? {} : {
- comparisons: owed.map((entry) => ({
- run: runPath,
- tokens: entry.tokens,
- video_path: entry.out,
- element: entry.element,
- question: "Ignore flat preview-mock regions standing in for declared-but-unbuilt media; compare authored layout, typography, motion and timing only.",
- })),
- }),
- // Where this command actually read and resolved from. Said here, no document has to describe it
- // from the outside and go stale when it moves.
- roots: mode === "reconstruction"
- ? { reference: preparedRoot, packages: packageRoot }
- : { reviews: logPath, packages: packageRoot },
- passed: owed.length === 0 && never.length === 0 && running.length === 0 && coverage.gaps.length === 0
- && unresolved.length === 0 && aspect.length === 0,
- summary,
- elements,
- ...(aspect.length === 0 ? {} : {
- canvas_aspect: {
- entries: aspect,
- note: "The reference and the Canvas are different shapes. Every comparison in a round puts the "
- + "reference clip beside a render drawn at the Canvas's proportions, so the observer reads the "
- + "difference between the two shapes and reports it as the element's — a caption that is the "
- + "right size looks wrong, and a repair made against that reading moves it away from the "
- + "reference. Change the Canvas to the reference's shape before comparing anything. The "
- + "reference's own pixel dimensions are not the requirement; its proportion is.",
- },
- }),
- ...(owed.length === 0 ? {} : {
- owed: {
- entries: owed,
- note: `${owed.length} stretch${owed.length === 1 ? "" : "es"} the Source asks for ${owed.length === 1 ? "has" : "have"} `
- + "not been looked at. Each one is an element over a range of the Script's own words, and the reason it is "
- + "here is beside it — the first appearance of a distinct visual declaration nobody has seen.",
- commands: owed.map((entry) => mode === "reconstruction"
- ? `hypit-reference-video-tools compare_reconstruction --reference-id ${reference} --run ${runPath} `
- + `--tokens ${entry.tokens[0]}:${entry.tokens[1]} --video ${entry.out} --element ${entry.element}`
- : `hypit-reference-video-tools review_element --run ${runPath} `
- + `--tokens ${entry.tokens[0]}:${entry.tokens[1]} --video ${entry.out} --element ${entry.element} `
- + `--intent-file <what ${entry.element} was asked to be>`),
- },
- }),
- ...(never.length === 0 ? {} : {
- never_compared: {
- ids: never.map((element) => element.id),
- next: mode === "reconstruction"
- ? "Read <skill-root>/references/element-review.md and "
- + "<skill-root>/references/reconstruction/comparison-round.md, then for each: "
- + "render the element as the Source configures it, mock the layers a Build has not made, "
- + "and compare the whole stretch blind — one comparison per Segment or Selection it is drawn over."
- : "Read <skill-root>/references/element-review.md and "
- + "<skill-root>/references/original-authoring/conformance-round.md, then for each: "
- + "render the element as the Source configures it, mock the layers a Build has not made, "
- + "and have it read against what this element was asked to be — one review per Segment or "
- + "Selection it is drawn over.",
- // One command per stretch the plan gives this element, rather than one per element naming the
- // Segments it is drawn over. The plan already decided which stretches this element earns and
- // named a picture for each, so both of the things a reader used to fill in are here.
- commands: never.flatMap((element) => plan.filter((entry) => entry.element === element.id)
- .map((entry) => mode === "reconstruction"
- ? `hypit-reference-video-tools compare_reconstruction --reference-id ${reference} --run ${runPath} `
- + `--tokens ${entry.tokens[0]}:${entry.tokens[1]} --video ${entry.out} --element ${element.id}`
- : `hypit-reference-video-tools review_element --run ${runPath} `
- + `--tokens ${entry.tokens[0]}:${entry.tokens[1]} --video ${entry.out} --element ${element.id} `
- + `--intent-file <what ${element.id} was asked to be>`)),
- },
- }),
- ...(awaiting.length === 0 ? {} : {
- awaiting_answer: {
- ids: awaiting.map((entry) => entry.id).filter((id) => id !== undefined),
- elements: [...new Set(awaiting.map((entry) => entry.element).filter((id) => id !== undefined))],
- note: mode === "reconstruction"
- ? `${awaiting.length} comparison${awaiting.length === 1 ? " was" : "s were"} performed and handed to an `
- + "observer that answers out of band, and the differences have not come back. Until they do, the pair has "
- + "been drawn and cut but nobody has said what it shows, so it credits nothing here.\n\n"
- + "Close each one with its id:\n"
- + " hypit-reference-video-tools record_observation --reference-id "
- + `${reference} --key comparison:<id> --text-file <the differences>`
- : `${awaiting.length} review${awaiting.length === 1 ? " was" : "s were"} handed out and the findings have `
- + "not come back. Until they do, the picture has been drawn but nobody has said what it shows, so it "
- + "credits nothing here.\n\n"
- + "Close each one with its id:\n"
- + ` hypit-reference-video-tools record_review --run ${runPath} --review-id <id> --text-file <the findings>`,
- },
- }),
- ...(unresolved.length === 0 ? {} : {
- unresolved_packages: {
- names: unresolved,
- package_root: packageRoot,
- note: `${unresolved.length} package${unresolved.length === 1 ? "" : "s"} the Source imports could not be `
- + `resolved from ${packageRoot}. Whatever they draw is invisible here, so no element of theirs is asked `
- + "for a comparison and this check cannot say the reconstruction is covered.\n\n"
- + "A project that publishes its own packages resolves them from the project root, where "
- + "`packages/<name>/package.json` carries the scoped name. Run the command from that directory, or "
- + "pass --package-root <project directory>.",
- },
- }),
- ...(unknown.length === 0 ? {} : {
- unknown_elements: {
- names: unknown,
- note: `${unknown.length} logged element name${unknown.length === 1 ? " does" : "s do"} not name a drawing `
- + "element in the Source — either the id is not there at all, or it belongs to something that puts no "
- + "picture on the Canvas. Either way the comparison credits nothing, and the element it was meant for "
- + "is still at zero.",
- },
- }),
- ...(unlabelled === 0 ? {} : {
- unlabelled_comparisons: {
- count: unlabelled,
- note: `${unlabelled} comparison${unlabelled === 1 ? " was" : "s were"} recorded without --element and cannot be credited to one.`,
- },
- }),
- ...(untimed.length === 0 ? {} : {
- timing_next: {
- ids: untimed.map((element) => element.id),
- note: "This comparison has no timing sidecar. Re-render it with render_element so the deterministic "
- + "estimate basis is recorded. Reference timing is not used to construct mock SemanticTakes; final "
- + "alignment against synthesized speech is settled after the Build.",
- commands: untimed.flatMap((element) => plan.filter((entry) => entry.element === element.id)
- .map((entry) =>
- `hypit-reference-video-tools render_element ${runPath} --element ${element.id} `
- + `--reference-id ${reference} --tokens ${entry.tokens[0]}:${entry.tokens[1]} --out ${entry.out}`)),
- },
- }),
- uncovered: coverage.gaps,
- ...(coverage.gaps.length === 0 ? {} : {
- uncovered_next: "Each stretch listed here shows whatever the placements around it left on screen, and "
- + "beneath the last of them is the Film background, which is black. Read "
- + "playbooks/craft/frame-coverage.md, then claim each stretch with whatever the reference shows "
- + "there: an element bound to the full Frame or to the Canvas, drawn over those words. Placing "
- + "something beneath the stretch instead changes which wrong picture appears and leaves the stretch "
- + "unclaimed.",
- }),
- playback: running,
- ...(running.length === 0 ? {} : {
- playback_next: "Only moving/generated media with a duration-bearing timeline needs playback. Still images have no timeline and must not be given playback. For a generated take, the take is ordered in whole seconds and its window is measured from speech the "
- + "Build has yet to synthesize, so the material is shorter than the window it fills more "
- + "often than not. Under a full-frame picture the remainder is the Film background, which "
- + "reads as a black gap; under a cutaway it is the layer beneath blinking through.\n\n"
- + "Give each Recipe listed here a playback: hold-start to pin the last frame for the rest of the "
- + "window, loop-start to repeat, or stretch to retime. Read playbooks/craft/frame-coverage.md "
- + "on inherited edges before choosing — lengthening the material instead leaves the same edge.",
- }),
- };
-}
diff --git a/packages/reference-video-tools/src/cli.ts b/packages/reference-video-tools/src/cli.ts
deleted file mode 100644
index 703aeaac..00000000
--- a/packages/reference-video-tools/src/cli.ts
+++ /dev/null
@@ -1,520 +0,0 @@
-#!/usr/bin/env node
-import { readFile } from "node:fs/promises";
-
-import { createReferenceVideoTools } from "./tools.js";
-import type { CompareReconstructionInput, ObserveReferenceInput, RecordObservationInput, RenderElementInput, ReviewElementInput } from "./tools.js";
-
-/**
- * A word range written on the command line as `from:to`, half-open.
- *
- * The planner emits ranges because the finest thing worth looking at is not always a thing the
- * Script named. `--segment` and `--selection` remain for the ones that are.
- */
-function tokenRange(value: string | undefined): readonly [number, number] | undefined {
- if (value === undefined) return undefined;
- const parts = /^(\d+):(\d+)$/u.exec(value.trim());
- if (parts === null) throw new Error(`--tokens takes a half-open word range written from:to, like 118:131; got ${value}`);
- const from = Number(parts[1]);
- const to = Number(parts[2]);
- if (to <= from) throw new Error(`--tokens ${value} ends at or before it starts`);
- return [from, to];
-}
-
-type Flags = ReadonlyMap<string, string | readonly string[] | boolean>;
-
-function usage(): string {
- return [
- "Usage:",
- " hypit-reference-video-tools list_svml_packages",
- " hypit-reference-video-tools prepare_reference --video-path <path or link> [--observer gemini|agent] [--runtime <profile>] [--redo media|transcript|people|voices|systems|places|all]",
- " hypit-reference-video-tools observe_reference --reference-id <id> [--shot-id <id> ...] [--reobserve]",
- " hypit-reference-video-tools observe_reference --reference-id <id> --shot-id <id> --question <text>",
- " hypit-reference-video-tools observe_reference --reference-id <id> --batch <questions.json>",
- " hypit-reference-video-tools record_observation --reference-id <id> [--run <build.svrun>] --key <key> --text <text>|--text-file <path>",
- " hypit-reference-video-tools record_observation --reference-id <id> [--run <build.svrun>] --batch <answers.json>",
- " hypit-reference-video-tools inspect_svml_vocabulary --package <name> [--package <name> ...] [--tag <tag> ...] [--without-previews] [--run <build.svrun>]",
- " hypit-reference-video-tools validate_local_author_packages --run <build.svrun> [--runtime <hypit.runtime.json>] [--expected-package <name> ...]",
- " hypit-reference-video-tools validate_script_cues --run <build.svrun>",
- " hypit-reference-video-tools inspect_visual_schema [--shape visual-track|visual-element|box|mask|text|image|video|surface|text-flow|text-typography|text-paint|text-document|path-command] [--producers-of <package> ...]",
- " hypit-reference-video-tools paths",
- " hypit-reference-video-tools compare_reconstruction --reference-id <id> --run <build.svrun> --segment <id>|--selection <id>|--tokens <from:to> --video <path>|--image <path> [--question <scope>] [--element <id>] [--tolerance-frames <n>]",
- " hypit-reference-video-tools compare_reconstruction --reference-id <id> --shot-id <id> --video <path>|--image <path> [--question <scope>] [--element <id>] [--tolerance-frames <n>]",
- " hypit-reference-video-tools compare_reconstruction --reference-id <id> --batch <comparisons.json>",
- " hypit-reference-video-tools review_element --run <build.svrun> --element <id> --segment <id>|--selection <id>|--tokens <from:to> --video <path>|--image <path> --intent-file <path> [--question <scope>]",
- " hypit-reference-video-tools review_element --run <build.svrun> --batch <reviews.json>",
- " hypit-reference-video-tools record_review --run <build.svrun> --review-id <id> --text <text>|--text-file <path>",
- " hypit-reference-video-tools render_element <build.svrun> --element <id> --out <path.png|path.mp4> [--segment <id>] [--selection <id>] [--tokens <from:to>] [--reference-id <id>]",
- " hypit-reference-video-tools render_element <build.svrun> --batch <renders.json> [--reference-id <id>]",
- " hypit-reference-video-tools render_previews <package-dir> [...]",
- " hypit-reference-video-tools preview_check <build.svrun> [<hypit.runtime.json>]",
- " hypit-reference-video-tools layout_check --run <build.svrun> [--runtime <hypit.runtime.json>]",
- " hypit-reference-video-tools layout_accept --run <build.svrun> --finding <id> --reason <text>",
- " hypit-reference-video-tools layout_accept --run <build.svrun> --batch <acceptances.json>",
- " hypit-reference-video-tools reconstruction_check <build.svrun> [--reference-id <id>] [--runtime <hypit.runtime.json>]",
- " hypit-reference-video-tools authoring_check <build.svrun> [--runtime <hypit.runtime.json>]",
- "",
- "preview_check opens the Run the way Studio does and reports whether the graph traces. `sound` is",
- "true when every Track resolved, and also when the only thing missing is capabilities a Provider has",
- "still to serve — those are listed under `awaiting`. It takes the Run Source, not the Author SVML:",
- "Studio's unit of work is the Run, and it reads the .svml back out of it.",
- "",
- "reconstruction_check reports what the route can settle before a Build: which drawing elements have",
- "never been compared against the reference, which words of the Script nothing draws a full-frame",
- "picture over, and which timed pictures are configured to stop before their window ends. Comparison",
- "asks for participation rather than convergence, so an element compared once passes. With one",
- "prepared reference it uses that one; with several it requires --reference-id.",
- "Each element records the preview timing basis — native estimate or not recorded — so the report knows",
- "not recorded — which says whether anything that changes with elapsed time inside its window has",
- "been looked at, or only its layout.",
- "",
- "layout_check performs a realized Composition/DOM stable-state pass without writing media or calling",
- "paid Providers. Every result is candidate evidence only: the Agent decides whether it is a genuine",
- "problem, repairs it, or records an intentional exception with layout_accept and a reason.",
- "",
- "render_element draws one element of a Source the way that Source configures it, without a Build.",
- "The Canvas, frame rate, Recipe values and bindings come from the compiled Graph; missing media is",
- "realized through @hypit/preview-mock and its local mock-media Provider.",
- "Name the stretch in words — --segment or --selection — so no reference timestamp is ever read",
- "across; without either, the whole program is drawn. An --out ending .mp4, .mov or .webm writes the",
- "stretch as a clip, and any other extension writes one still from the middle of it.",
- "",
- "Preview timing is always the Source's own literal durations. `--reference-id` only selects",
- "reference evidence and comparison windows; it never drives a synthetic SemanticTake clock. The",
- "result's `timing_basis` is `estimate`, and the same object is written to <out>.stand-in.json for",
- "compare_reconstruction to carry into the comparison log.",
- "",
- "render_previews draws a package's catalogue pictures from the package's own preview Source in",
- "preview/preview.svml, preview/recipes.svs and preview/build.svrun. Which element draws which file",
- "comes from the Manifest's own naming: a Surface tagged Track declares preview/Track.png, so",
- "Track.png is drawn from whichever element in the preview Source carries that tag. An .svg the",
- "Manifest promises is drawn by hand and left alone.",
- "",
- "--element names the reconstructed element the image draws. It is written to the reference's",
- "comparison log and never sent to the observer, so the comparison stays blind while a later report can",
- "still tell which elements have been compared.",
- "",
- "compare_reconstruction takes the stretch of the reference two ways. --segment and --selection name",
- "a word range: the words come from the Author SVML of the Run given as --run, they are aligned",
- "against the reference's transcript, and the reference is cut from its own analysis video at the",
- "seconds it speaks them. That is the stretch a render covers, since a render is drawn over those",
- "same words, and one Segment routinely runs across several shots. Each end moves onto a shot",
- "boundary when one lies inside its own end word, so the pair opens and closes where the picture",
- "changes while still covering exactly the words asked for, and the rendered clip is trimmed by the",
- "seconds each end moved so both sides show the same word at the same offset. An end with no boundary",
- "inside its word leaves the pair part-way through a shot, and the prompt says how many seconds of it",
- "to read as an incomplete shot. --shot-id names a cut in the picture and compares that whole shot.",
- "",
- "--batch runs a whole round at once. The file holds a JSON array — or an object with a `comparisons`",
- "array — of the same objects a single call takes, minus reference_id, which comes from the flag:",
- "`[{\"run\": \"main.svrun\", \"segment\": \"pro\", \"video_path\": \"renders/board-pro.mp4\", \"element\": \"board\"}, …]`.",
- "Every render is finished before any comparison starts, so a round has no order to it; they are",
- "staged serially from one shared preview frame cache; observer comparisons still use their normal",
- "pacing and each one's derived cuts are kept apart, so nothing in the round reads a file another is still writing. One comparison that fails takes only itself down and",
- "arrives under `failures` with the input that produced it; the rest are under `comparisons`.",
- "",
- "--video compares the whole stretch instead of one frame of it, which is what removes the problem of",
- "choosing a characteristic frame for an element that animates in, leaves, or is replaced without a",
- "cut. The `gemini` observer receives the two clips; the `agent` observer receives two frame tiles,",
- "both tiled against the compared stretch's own duration so the grids sample alike. Use --image only",
- "when the reference's own visual observation states the element is completely still.",
- "",
- "Both sides of a --video pair are measured once they are cut to the same stretch, and a pair that is",
- "completely still on both sides is compared as one frame from each: `compared` reads `still` and",
- "`both_sides_still` says why. Both sides have to be still, because the preview-mock base is a flat",
- "fill and is therefore still whatever the reference does.",
- "",
- "Mock media is materialized by the local mock Provider and cached by @hypit/preview-mock; it never",
- "enters Author Source and is ignored by coverage reports. Timing uses the Source's literal durations, while reference",
- "WhisperX is reserved for comparison evidence and window selection.",
- "",
- "record_observation takes the keys the agent observer is handed: the four whole-reference keys, a",
- "shot key such as visual:shot-003 or boundary:shot-004, a narrow question keyed by the shots it was",
- "asked over and by the question itself, and a comparison keyed",
- "by the `comparison_id` compare_reconstruction returned (comparison:<id>). Pass the key back exactly",
- "as it was handed out. A comparison's answer is written back onto its own line of the log, which is",
- "what makes it count toward coverage; until then reconstruction_check lists it under",
- "`awaiting_answer`.",
- "",
- "--batch records a whole sweep's answers at once. The file holds a JSON array — or an object with an",
- "`answers` array — of `{key, text}`, or `{key, text_file}` where the answer was written to a file:",
- "`[{\"key\": \"visual:shot-003\", \"text_file\": \"answers/shot-003.md\"}, …]`. Paths are read against the",
- "working directory. They are applied in the order they are written, and one that fails takes only",
- "itself down and arrives under `failures`; the rest are under `records`.",
- "",
- "review_element is the counterpart on the route with no reference: it reads one rendered element",
- "against what that element was asked to be, rather than against a video to copy. --intent-file",
- "carries the author's own words and is required; --element is required too, so a review credits an",
- "element by construction. A clip becomes a grid of its own frames, and the task comes back to be",
- "answered — nobody is billed to look at a local render — with record_review closing it. Reviews are",
- "logged under the project, at <project>/.hypit/reviews.jsonl, and authoring_check reads them.",
- "",
- "authoring_check is reconstruction_check for a program authored from a description. It resolves no",
- "reference and credits an element from the review log, and it applies every check the Source alone",
- "decides — an uncovered stretch, a Frame reaching past the Canvas, a `playback` left at its default.",
- "Pick the command that matches the route; neither infers which one you meant.",
- "",
- "--video-path takes a link as readily as a path. A TikTok, YouTube, Instagram or Bilibili URL is",
- "fetched with yt-dlp before anything else runs, cached by the link so a restarted route reaches the",
- "same bytes and therefore the same reference, and reported back as `source_url`. Everything after",
- "the download reads the file without knowing it was ever a link.",
- "",
- "--observer picks who reads the reference, once per reference. `gemini` uploads video to the Gemini Endpoint",
- "the selected Runtime Profile binds (--runtime <profile>, else the project's `hypit runtime use` selection);",
- "the Profile's whisperx-alignment Endpoint measures the transcript the same way. If that Endpoint's credential",
- "is missing, sign in with `hypit auth login` or use `agent`. `agent` needs no credentials: it",
- "returns each observation as a task carrying its prompt and one tiled picture per shot, which the",
- "calling agent answers with record_observation. A task whose question needs sound also carries the",
- "words WhisperX measured — `transcript_words` for its own stretch, `transcript_ref` for the whole",
- "file — since that observer reads pictures and the words are the only record of when speech happens.",
- "",
- "inspect_visual_schema answers what a Producer that draws may return: the element kinds, the style",
- "names admitted on them, which take an enum, the only local styles allowed in keyframes",
- "(clip-path, filter, opacity and transform), and how few keyframes an animation carries. Every line is",
- "generated from the Composition schema, so it says what the seal will accept rather than what one",
- "package happened to do. inspect_svml_vocabulary answers the other half — what a Source may write.",
- "",
- "paths reports where this command reads reference state and resolves packages from, and which",
- "references are prepared. Use it when a check reports something it cannot see.",
- "",
- "--package-root <dir> is where the project packages a Source imports are resolved from, and it is",
- "accepted by every command. It defaults to the working directory and never replaces the active",
- "Hypit Distribution root, which remains an independent fallback for @hypit/* packages. A project that declares a vocabulary gap and",
- "fills it publishes those packages under its own scope and installs them against the project root, so",
- "run these commands from the project directory or name it here: resolved from anywhere else, the",
- "project's own packages are not found, and reconstruction_check reports them under",
- "`unresolved_packages` rather than asking for a comparison of what they draw.",
- "",
- "Every command prints one JSON result to stdout. Use --input <json> instead of flags when a complete input object is easier to pass.",
- ].join("\n");
-}
-
-// The authoring and check commands name what they act on positionally — a Run for `render_element`,
-// `preview_check` and `reconstruction_check`, one or more package directories for `render_previews` —
-// so a token that is not a flag is collected rather than refused. Every other command still refuses
-// one, which is why the check is made per command rather than dropped here.
-const COMMANDS_WITH_OPERANDS = new Set(["render_element", "render_previews", "preview_check", "reconstruction_check", "authoring_check"]);
-
-function parse(argv: readonly string[]): { readonly command: string; readonly operands: readonly string[]; readonly flags: Flags } {
- const command = argv[0];
- if (command === undefined || command === "--help" || command === "-h") throw new Error(usage());
- const values = new Map<string, string | string[] | boolean>();
- const operands: string[] = [];
- for (let index = 1; index < argv.length; index += 1) {
- const token = argv[index]!;
- if (!token.startsWith("--")) {
- if (!COMMANDS_WITH_OPERANDS.has(command)) throw new Error(`unexpected argument ${token}\n\n${usage()}`);
- operands.push(token);
- continue;
- }
- const name = token.slice(2);
- if (name.length === 0) throw new Error("empty option name");
- const next = argv[index + 1];
- if (next === undefined || next.startsWith("--")) {
- values.set(name, true);
- continue;
- }
- index += 1;
- const current = values.get(name);
- if (current === undefined) values.set(name, next);
- else values.set(name, [...(Array.isArray(current) ? current : [String(current)]), next]);
- }
- return { command, operands, flags: values };
-}
-
-function one(flags: Flags, name: string): string | undefined {
- const value = flags.get(name);
- if (value === undefined || typeof value === "boolean") return undefined;
- return Array.isArray(value) ? value.at(-1) : typeof value === "string" ? value : undefined;
-}
-
-function many(flags: Flags, name: string): readonly string[] {
- const value = flags.get(name);
- if (value === undefined || typeof value === "boolean") return [];
- return Array.isArray(value) ? value : typeof value === "string" ? [value] : [];
-}
-
-function required(flags: Flags, name: string): string {
- const value = one(flags, name);
- if (value === undefined || value.trim().length === 0) throw new Error(`--${name} is required`);
- return value;
-}
-
-function inputObject(flags: Flags): Record<string, unknown> | undefined {
- const raw = one(flags, "input");
- if (raw === undefined) return undefined;
- let parsed: unknown;
- try { parsed = JSON.parse(raw); } catch (error) { throw new Error(`--input is not valid JSON: ${error instanceof Error ? error.message : String(error)}`); }
- if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) throw new Error("--input must contain a JSON object");
- return parsed as Record<string, unknown>;
-}
-
-// stdout carries exactly one JSON result. Dependencies that announce themselves on stdout would
-// corrupt it, so everything they write is moved to stderr and only `report` reaches stdout.
-const report = process.stdout.write.bind(process.stdout);
-process.stdout.write = ((chunk: string | Uint8Array, ...rest: readonly unknown[]): boolean =>
- (process.stderr.write as (...args: readonly unknown[]) => boolean)(chunk, ...rest)) as typeof process.stdout.write;
-
-async function main(): Promise<void> {
- const { command, operands, flags } = parse(process.argv.slice(2));
- if (flags.has("rebuild") || flags.has("refresh")) {
- throw new Error("--rebuild and --refresh were removed; use --redo on prepare_reference or --reobserve on observe_reference");
- }
- // Where the Source's imports are resolved from. A project publishing its own packages installs them
- // against the project root, so a root taken from anywhere else resolves none of them: --package-root
- // is how a command run from elsewhere reaches them. HYPIT_DISTRIBUTION_ROOT names an installed
- // Distribution and is the fallback for the installed packages a project does not carry.
- // --package-root selects only the project/source package root. The active Distribution root is
- // discovered independently by video-cli and is always supplied as a loader fallback.
- const resolveFrom = one(flags, "package-root");
- const tools = createReferenceVideoTools({
- ...(resolveFrom === undefined ? {} : { packageRoot: resolveFrom }),
- });
- const supplied = inputObject(flags);
- let result: unknown;
- if (command === "list_svml_packages") {
- result = await tools.list_svml_packages();
- } else if (command === "prepare_reference") {
- const observer = one(flags, "observer");
- if (observer !== undefined && observer !== "gemini" && observer !== "agent") throw new Error("--observer must be gemini or agent");
- const input = supplied ?? {
- video_path: required(flags, "video-path"),
- ...(observer === undefined ? {} : { observer }),
- ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
- ...(one(flags, "redo") === undefined ? {} : { redo: one(flags, "redo") }),
- };
- result = await tools.prepare_reference(input as { video_path: string; observer?: "gemini" | "agent"; runtime?: string; redo?: "media" | "transcript" | "people" | "voices" | "systems" | "places" | "all" });
- } else if (command === "observe_reference") {
- // A round of narrow questions is a list, and a list is too long for flags.
- const askFile = one(flags, "batch");
- if (askFile !== undefined) {
- const parsed: unknown = JSON.parse(await readFile(askFile, "utf8"));
- const list = Array.isArray(parsed) ? parsed : (parsed as { questions?: unknown }).questions;
- if (!Array.isArray(list)) throw new Error(`${askFile} must hold a JSON array of {shot_ids, question}, or an object with a "questions" array`);
- const runtime = one(flags, "runtime");
- result = await tools.observe_reference({
- reference_id: required(flags, "reference-id"),
- ...(runtime === undefined ? {} : { runtime }),
- questions: list as ObserveReferenceInput["questions"] & object,
- });
- report(`${JSON.stringify(result, null, 2)}\n`);
- return;
- }
- const input = supplied ?? {
- reference_id: required(flags, "reference-id"),
- ...(many(flags, "shot-id").length === 0 ? {} : { shot_ids: many(flags, "shot-id") }),
- ...(one(flags, "question") === undefined ? {} : { question: one(flags, "question") }),
- ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
- ...(flags.get("reobserve") === true ? { reobserve: true } : {}),
- };
- result = await tools.observe_reference(input as { reference_id: string; shot_ids?: readonly string[]; question?: string; runtime?: string; reobserve?: boolean });
- } else if (command === "record_observation") {
- // A whole sweep's answers at once. Each entry carries its own text, either inline or as a file the
- // answer was written to, which is read here the way --text-file is.
- const answerFile = one(flags, "batch");
- if (answerFile !== undefined) {
- const parsed: unknown = JSON.parse(await readFile(answerFile, "utf8"));
- const list = Array.isArray(parsed) ? parsed : (parsed as { answers?: unknown }).answers;
- if (!Array.isArray(list)) throw new Error(`${answerFile} must hold a JSON array of answers, or an object with an "answers" array`);
- const answers = await Promise.all((list as { key?: unknown; text?: unknown; text_file?: unknown }[]).map(async (entry) => {
- if (typeof entry.key !== "string") throw new Error(`every answer needs a "key"; received ${JSON.stringify(entry)}`);
- const file = entry.text_file;
- // Against the working directory, the way every other batch entry's paths are.
- if (typeof file === "string") return { key: entry.key, text: await readFile(file, "utf8") };
- if (typeof entry.text !== "string") throw new Error(`answer ${entry.key} needs a "text" or a "text_file"`);
- return { key: entry.key, text: entry.text };
- }));
- result = await tools.record_observation({ reference_id: required(flags, "reference-id"), ...(one(flags, "run") === undefined ? {} : { run: one(flags, "run") }), answers } as RecordObservationInput);
- } else {
- // An observation is paragraphs of prose. --text keeps a short one on the command line; --text-file
- // is how a long one arrives without the shell deciding where it ends.
- const textFile = one(flags, "text-file");
- const input = supplied ?? {
- reference_id: required(flags, "reference-id"),
- ...(one(flags, "run") === undefined ? {} : { run: one(flags, "run") }),
- key: required(flags, "key"),
- text: textFile === undefined ? required(flags, "text") : await readFile(textFile, "utf8"),
- };
- result = await tools.record_observation(input as { reference_id: string; run?: string; key: string; text: string });
- }
- } else if (command === "review_element") {
- const reviewFile = one(flags, "batch");
- if (reviewFile !== undefined) {
- const parsed: unknown = JSON.parse(await readFile(reviewFile, "utf8"));
- const list = Array.isArray(parsed) ? parsed : (parsed as { reviews?: unknown }).reviews;
- if (!Array.isArray(list)) throw new Error(`${reviewFile} must hold a JSON array of reviews, or an object with a "reviews" array`);
- result = await tools.review_element({ run: required(flags, "run"), reviews: list } as ReviewElementInput);
- report(`${JSON.stringify(result, null, 2)}\n`);
- return;
- }
- const input = supplied ?? {
- run: required(flags, "run"),
- element: required(flags, "element"),
- ...(one(flags, "segment") === undefined ? {} : { segment: one(flags, "segment") }),
- ...(one(flags, "selection") === undefined ? {} : { selection: one(flags, "selection") }),
- ...(tokenRange(one(flags, "tokens")) === undefined ? {} : { tokens: tokenRange(one(flags, "tokens")) }),
- ...(one(flags, "image") === undefined ? {} : { image_path: one(flags, "image") }),
- ...(one(flags, "video") === undefined ? {} : { video_path: one(flags, "video") }),
- ...(one(flags, "intent-file") === undefined ? {} : { intent_file: one(flags, "intent-file") }),
- ...(one(flags, "intent") === undefined ? {} : { intent: one(flags, "intent") }),
- ...(one(flags, "question") === undefined ? {} : { question: one(flags, "question") }),
- };
- result = await tools.review_element(input as ReviewElementInput);
- } else if (command === "record_review") {
- // Findings are paragraphs of prose, so they arrive the way an observation's do.
- const textFile = one(flags, "text-file");
- const input = supplied ?? {
- run: required(flags, "run"),
- review_id: required(flags, "review-id"),
- text: textFile === undefined ? required(flags, "text") : await readFile(textFile, "utf8"),
- };
- result = await tools.record_review(input as { run: string; review_id: string; text: string });
- } else if (command === "compare_reconstruction") {
- // A round is a list of comparisons, and a list is too long for flags. --batch names a JSON file
- // holding it, which is also how it survives being written by one step and read by another.
- const batchFile = one(flags, "batch");
- if (batchFile !== undefined) {
- const parsed: unknown = JSON.parse(await readFile(batchFile, "utf8"));
- const list = Array.isArray(parsed) ? parsed : (parsed as { comparisons?: unknown }).comparisons;
- if (!Array.isArray(list)) throw new Error(`${batchFile} must hold a JSON array of comparisons, or an object with a "comparisons" array`);
- const runtime = one(flags, "runtime");
- result = await tools.compare_reconstruction({
- reference_id: required(flags, "reference-id"),
- ...(runtime === undefined ? {} : { runtime }),
- comparisons: list as CompareReconstructionInput["comparisons"] & object,
- });
- report(`${JSON.stringify(result, null, 2)}\n`);
- return;
- }
- const video = one(flags, "video");
- const segment = one(flags, "segment");
- const selection = one(flags, "selection");
- const tokens = tokenRange(one(flags, "tokens"));
- const range = segment !== undefined || selection !== undefined || tokens !== undefined;
- const input = supplied ?? {
- reference_id: required(flags, "reference-id"),
- ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
- ...(range
- ? {
- run: required(flags, "run"),
- ...(tokens === undefined ? {} : { tokens }),
- ...(segment === undefined ? {} : { segment }),
- ...(selection === undefined ? {} : { selection }),
- }
- : { shot_id: required(flags, "shot-id") }),
- ...(video === undefined ? { image_path: required(flags, "image") } : { video_path: video }),
- ...(one(flags, "question") === undefined ? {} : { question: one(flags, "question") }),
- ...(one(flags, "element") === undefined ? {} : { element: one(flags, "element") }),
- ...(one(flags, "tolerance-frames") === undefined ? {} : { tolerance_frames: Number(one(flags, "tolerance-frames")) }),
- };
- result = await tools.compare_reconstruction(input as CompareReconstructionInput);
- } else if (command === "inspect_visual_schema") {
- result = await tools.inspect_visual_schema({
- ...(one(flags, "shape") === undefined ? {} : { shape: one(flags, "shape")! }),
- ...(many(flags, "producers-of").length === 0 ? {} : { producers: many(flags, "producers-of") }),
- });
- } else if (command === "paths") {
- result = await tools.paths();
- } else if (command === "inspect_svml_vocabulary") {
- const packages = many(flags, "package");
- const input = supplied ?? {
- package_names: packages.length > 0 ? packages : [required(flags, "package-name")],
- ...(many(flags, "tag").length === 0 ? {} : { tags: many(flags, "tag") }),
- ...(flags.get("without-previews") === true ? { include_previews: false } : {}),
- ...(one(flags, "run") === undefined ? {} : { run: one(flags, "run") }),
- };
- result = await tools.inspect_svml_vocabulary(input as { package_names: readonly string[]; tags?: readonly string[]; include_previews?: boolean; run?: string });
- } else if (command === "validate_local_author_packages") {
- const runtime = one(flags, "runtime");
- result = await tools.validate_local_author_packages({
- run: required(flags, "run"),
- ...(runtime === undefined ? {} : { runtime }),
- ...(many(flags, "expected-package").length === 0 ? {} : { expected_packages: many(flags, "expected-package") }),
- });
- } else if (command === "validate_script_cues") {
- result = await tools.validate_script_cues({ run: required(flags, "run") });
- } else if (command === "render_element") {
- const renderFile = one(flags, "batch");
- if (renderFile !== undefined) {
- const parsed: unknown = JSON.parse(await readFile(renderFile, "utf8"));
- const list = Array.isArray(parsed) ? parsed : (parsed as { renders?: unknown }).renders;
- if (!Array.isArray(list)) throw new Error(`${renderFile} must hold a JSON array of renders, or an object with a "renders" array`);
- result = await tools.render_element({
- ...(operands[0] === undefined ? {} : { run: operands[0] }),
- ...(one(flags, "reference-id") === undefined ? {} : { reference_id: one(flags, "reference-id") }),
- renders: list,
- } as RenderElementInput);
- report(`${JSON.stringify(result, null, 2)}\n`);
- return;
- }
- const run = operands[0];
- if (supplied === undefined && run === undefined) throw new Error(`a <build.svrun> is required\n\n${usage()}`);
- const input = supplied ?? {
- run,
- element: required(flags, "element"),
- out: required(flags, "out"),
- ...(one(flags, "segment") === undefined ? {} : { segment: one(flags, "segment") }),
- ...(one(flags, "selection") === undefined ? {} : { selection: one(flags, "selection") }),
- ...(tokenRange(one(flags, "tokens")) === undefined ? {} : { tokens: tokenRange(one(flags, "tokens")) }),
- ...(one(flags, "reference-id") === undefined ? {} : { reference_id: one(flags, "reference-id") }),
- };
- result = await tools.render_element(input as RenderElementInput);
- } else if (command === "render_previews") {
- if (supplied === undefined && operands.length === 0) throw new Error(`a <package-dir> is required\n\n${usage()}`);
- const input = supplied ?? { package_dirs: operands };
- result = await tools.render_previews(input as { package_dirs: readonly string[] });
- } else if (command === "preview_check") {
- // The Runtime Profile is the second operand rather than a flag, the way the Run itself is: both
- // name a file this command opens, and the pair reads as one argument list.
- const run = operands[0];
- if (supplied === undefined && run === undefined) throw new Error(`a <build.svrun> is required\n\n${usage()}`);
- const input = supplied ?? {
- run,
- ...(operands[1] === undefined ? {} : { runtime: operands[1] }),
- };
- result = await tools.preview_check(input as { run: string; runtime?: string });
- } else if (command === "layout_check") {
- result = await tools.layout_check((supplied ?? {
- run: required(flags, "run"),
- ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
- }) as { run: string; runtime?: string });
- } else if (command === "layout_accept") {
- if (supplied !== undefined) result = await tools.layout_accept(supplied as never);
- else {
- const batch = one(flags, "batch");
- if (batch === undefined) {
- result = await tools.layout_accept({ run: required(flags, "run"), finding: required(flags, "finding"), reason: required(flags, "reason") });
- } else {
- const decoded = JSON.parse(await readFile(batch, "utf8")) as unknown;
- const findings = Array.isArray(decoded) ? decoded : decoded !== null && typeof decoded === "object"
- ? (decoded as { findings?: unknown }).findings
- : undefined;
- if (!Array.isArray(findings)) throw new Error("layout_accept --batch expects a JSON array or an object with a findings array");
- result = await tools.layout_accept({ run: required(flags, "run"), findings });
- }
- }
- } else if (command === "reconstruction_check") {
- const run = operands[0];
- if (supplied === undefined && run === undefined) throw new Error(`a <build.svrun> is required\n\n${usage()}`);
- const input = supplied ?? {
- run,
- ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
- ...(one(flags, "reference-id") === undefined ? {} : { reference_id: one(flags, "reference-id") }),
- };
- result = await tools.reconstruction_check(input as { run: string; reference_id?: string; runtime?: string });
- } else if (command === "authoring_check") {
- const run = operands[0];
- if (supplied === undefined && run === undefined) throw new Error(`a <build.svrun> is required\n\n${usage()}`);
- result = await tools.authoring_check((supplied ?? {
- run,
- ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
- }) as { run: string; runtime?: string });
- } else {
- throw new Error(`unknown command ${command}\n\n${usage()}`);
- }
- report(`${JSON.stringify(result, null, 2)}\n`);
-}
-
-main().catch((error: unknown) => {
- process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
- process.exitCode = 1;
-});
diff --git a/packages/reference-video-tools/src/index.ts b/packages/reference-video-tools/src/index.ts
deleted file mode 100644
index 5abf70c0..00000000
--- a/packages/reference-video-tools/src/index.ts
+++ /dev/null
@@ -1,10 +0,0 @@
-export { createReferenceVideoTools } from "./tools.js";
-export type {
- ReferenceVideoTools,
- PrepareReferenceInput,
- ObserveReferenceInput,
- RecordObservationInput,
- InspectVocabularyInput,
- ValidateLocalAuthorPackagesInput,
-} from "./tools.js";
-export type { ScriptCueCheckInput } from "./checks.js";
diff --git a/packages/reference-video-tools/src/layout.ts b/packages/reference-video-tools/src/layout.ts
deleted file mode 100644
index f7618183..00000000
--- a/packages/reference-video-tools/src/layout.ts
+++ /dev/null
@@ -1,491 +0,0 @@
-import { createReadStream } from "node:fs";
-import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
-import { createServer } from "node:http";
-import { createRequire } from "node:module";
-import { dirname, extname, join, normalize, resolve, sep } from "node:path";
-import { spawnSync } from "node:child_process";
-
-import puppeteer from "puppeteer-core";
-
-import { authorSource, invokedFrom, realizeAuthoringPreview, repositoryRoot, stageAuthoringPreview } from "./authoring.js";
-
-export type LayoutCheckInput = {
- readonly run: string;
- readonly runtime?: string;
- readonly package_root?: string;
-};
-export type LayoutAcceptance = { readonly finding: string; readonly reason: string };
-export type LayoutAcceptInput = {
- readonly run: string;
- readonly finding?: string;
- readonly reason?: string;
- readonly findings?: readonly LayoutAcceptance[];
-};
-type Bounds = { readonly left: number; readonly top: number; readonly right: number; readonly bottom: number; readonly width: number; readonly height: number };
-type RawFinding = {
- readonly kind: "vertical-offset" | "parent-overflow" | "canvas-overflow" | "peer-overlap";
- readonly layer: "realized_layout";
- readonly track?: string;
- readonly present?: string;
- readonly element: string;
- readonly related?: string;
- readonly frame?: number;
- readonly pixels: number;
- readonly ratio: number;
- readonly bounds?: Bounds;
- readonly related_bounds?: Bounds;
- readonly message: string;
-};
-
-type LayoutDecision = {
- readonly finding: string;
- readonly reason: string;
- readonly accepted_at: string;
-};
-
-type LayoutDecisions = { readonly decisions: readonly LayoutDecision[] };
-
-function projectRoot(runPath: string): string { return dirname(runPath); }
-export function layoutCheckPath(runPath: string): string { return join(projectRoot(runPath), ".hypit", "layout-check.json"); }
-export function layoutDecisionsPath(runPath: string): string { return join(projectRoot(runPath), ".hypit", "layout-decisions.json"); }
-
-function round(value: number): number { return Math.round(value * 1000) / 1000; }
-
-async function writeJson(path: string, value: unknown): Promise<void> {
- await mkdir(dirname(path), { recursive: true });
- await writeFile(path, `${JSON.stringify(value, null, 2)}\n`, "utf8");
-}
-
-async function readDecisions(path: string): Promise<LayoutDecisions> {
- const source = await readFile(path, "utf8").catch((error: unknown) => {
- if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined;
- throw error;
- });
- if (source === undefined) return { decisions: [] };
- let value: unknown;
- try { value = JSON.parse(source); }
- catch (error) { throw new Error(`layout decisions at ${path} are invalid JSON; original file was preserved: ${error instanceof Error ? error.message : String(error)}`); }
- const decisions = value as Partial<LayoutDecisions>;
- if (!Array.isArray(decisions.decisions)) throw new Error(`layout decisions at ${path} have an unsupported shape`);
- return decisions as LayoutDecisions;
-}
-
-function contentType(path: string): string {
- const extension = extname(path).toLowerCase();
- if (extension === ".html") return "text/html; charset=utf-8";
- if (extension === ".css") return "text/css; charset=utf-8";
- if (extension === ".js") return "text/javascript; charset=utf-8";
- return "application/octet-stream";
-}
-
-export async function serve(directory: string, mediaTypes: ReadonlyMap<string, string>): Promise<{ readonly url: string; close(): Promise<void> }> {
- const root = resolve(directory);
- const server = createServer(async (request, response) => {
- const pathname = decodeURIComponent(new URL(request.url ?? "/", "http://127.0.0.1").pathname);
- const relative = pathname === "/" ? "index.html" : pathname.slice(1);
- const path = resolve(root, normalize(relative));
- if (path !== root && !path.startsWith(`${root}${sep}`)) { response.writeHead(403).end(); return; }
- const info = await stat(path).catch(() => undefined);
- if (info === undefined || !info.isFile()) { response.writeHead(404).end(); return; }
- response.writeHead(200, { "content-type": mediaTypes.get(relative) ?? contentType(path), "cache-control": "no-store" });
- createReadStream(path).pipe(response);
- });
- await new Promise<void>((accept, reject) => {
- server.once("error", reject);
- server.listen(0, "127.0.0.1", () => accept());
- });
- const address = server.address();
- if (address === null || typeof address === "string") throw new Error("layout browser server did not bind a local port");
- return {
- url: `http://127.0.0.1:${address.port}/`,
- close: async () => await new Promise<void>((accept, reject) => server.close((error) => error === undefined ? accept() : reject(error))),
- };
-}
-
-export function browserPath(): string {
- const cli = createRequire(join(repositoryRoot(), "packages/provider-hyperframes-local/package.json"))
- .resolve("hyperframes/bin/hyperframes.mjs");
- const result = spawnSync(process.execPath, [cli, "browser", "path"], { encoding: "utf8", windowsHide: true, timeout: 60_000 });
- if (result.status !== 0) throw new Error(`cannot resolve the fixed HyperFrames browser: ${(result.stderr ?? "").trim()}`);
- const path = (result.stdout ?? "").trim().split("\n").at(-1)?.trim();
- if (path === undefined || path.length === 0) throw new Error("HyperFrames returned no browser path");
- return path;
-}
-
-export type StableSample = {
- readonly frame: number;
- readonly subjects: readonly { readonly track: string; readonly present: string }[];
- readonly overlap_subjects: readonly { readonly track: string; readonly present: string }[];
-};
-
-type StableInterval = { readonly track: string; readonly present: string; readonly startFrame: number; readonly endFrameExclusive: number; readonly frame: number };
-
-function stableIntervals(composition: Awaited<ReturnType<typeof realizeAuthoringPreview>>["built"]["composition"]): readonly StableInterval[] {
- const intervals: StableInterval[] = [];
- for (const track of composition.tracks) {
- if (track.kind !== "visual") continue;
- for (const present of track.presents) {
- const duration = present.span.endFrameExclusive - present.span.startFrame;
- const changing = Array.from({ length: duration }, () => false);
- const mark = (start: number, endExclusive: number): void => {
- for (let frame = Math.max(0, Math.floor(start)); frame < Math.min(duration, Math.ceil(endExclusive)); frame += 1) changing[frame] = true;
- };
- const signature = (style: readonly { readonly name: string; readonly value: unknown }[]): string => JSON.stringify(
- [...style].map((item) => [item.name, item.value]).sort(([left], [right]) => String(left).localeCompare(String(right))),
- );
- for (const element of present.elements) {
- const keyframes = element.animation?.keyframes ?? [];
- for (let index = 0; index + 1 < keyframes.length; index += 1) {
- const left = keyframes[index]!;
- const right = keyframes[index + 1]!;
- if (signature(left.style) !== signature(right.style)) mark(left.atFrame, right.atFrame);
- }
- if (element.kind === "text-flow" || element.kind === "path-text") {
- for (const sequence of element.sequences) {
- const units = Math.max(1, sequence.range.endExclusive - sequence.range.start);
- const lastStart = sequence.startFrame + (units - 1) * sequence.staggerFrames;
- const start = Math.min(sequence.startFrame, lastStart);
- const end = Math.max(sequence.startFrame, lastStart) + sequence.unitDurationFrames * sequence.cycles;
- mark(start, end);
- }
- }
- if (element.kind === "path-text" && element.marginAnimation !== undefined) {
- const keyframes = element.marginAnimation.keyframes;
- for (let index = 0; index + 1 < keyframes.length; index += 1) {
- const left = keyframes[index]!;
- const right = keyframes[index + 1]!;
- if (left.startMarginPx !== right.startMarginPx) mark(left.atFrame, right.atFrame);
- }
- }
- }
- let best: { readonly start: number; readonly endExclusive: number } | undefined;
- let start = 0;
- while (start < duration) {
- while (start < duration && changing[start]) start += 1;
- if (start >= duration) break;
- let end = start + 1;
- while (end < duration && !changing[end]) end += 1;
- if (end - start >= 2 && (best === undefined || end - start > best.endExclusive - best.start)) {
- best = { start, endExclusive: end };
- }
- start = end;
- }
- if (best !== undefined) intervals.push({
- track: track.id,
- present: present.id,
- startFrame: present.span.startFrame + best.start,
- endFrameExclusive: present.span.startFrame + best.endExclusive,
- frame: present.span.startFrame + Math.floor((best.start + best.endExclusive - 1) / 2),
- });
- }
- }
- return intervals;
-}
-
-/** The middle frame of one Track's longest stable Present interval, with earliest interval winning ties. */
-export function stableFrameForTrack(
- composition: Awaited<ReturnType<typeof realizeAuthoringPreview>>["built"]["composition"],
- trackId: string,
-): number | undefined {
- return stableIntervals(composition)
- .filter((interval) => interval.track === trackId)
- .sort((left, right) => (right.endFrameExclusive - right.startFrame) - (left.endFrameExclusive - left.startFrame)
- || left.startFrame - right.startFrame || left.present.localeCompare(right.present))[0]?.frame;
-}
-
-export function framePlan(composition: Awaited<ReturnType<typeof realizeAuthoringPreview>>["built"]["composition"]): readonly StableSample[] {
- const intervals = stableIntervals(composition);
- const byFrame = new Map<number, { readonly frame: number; readonly subjects: { track: string; present: string }[] }>();
- for (const interval of intervals) {
- const subject = { track: interval.track, present: interval.present };
- const existing = byFrame.get(interval.frame);
- if (existing === undefined) byFrame.set(interval.frame, { frame: interval.frame, subjects: [subject] });
- else existing.subjects.push(subject);
- }
- return [...byFrame.values()]
- .map((sample) => ({
- frame: sample.frame,
- subjects: sample.subjects.sort((left, right) => `${left.track}/${left.present}`.localeCompare(`${right.track}/${right.present}`)),
- overlap_subjects: intervals
- .filter((interval) => sample.frame >= interval.startFrame && sample.frame < interval.endFrameExclusive)
- .map((interval) => ({ track: interval.track, present: interval.present }))
- .sort((left, right) => `${left.track}/${left.present}`.localeCompare(`${right.track}/${right.present}`)),
- }))
- .sort((left, right) => left.frame - right.frame);
-}
-
-function packageByTrack(realized: Awaited<ReturnType<typeof realizeAuthoringPreview>>): ReadonlyMap<string, string> {
- const result = new Map<string, string>();
- for (const item of realized.built.tracks) {
- const id = (item.value as { readonly id?: unknown } | null)?.id;
- if (typeof id !== "string") continue;
- const packageName = item.trace.module?.name ?? item.typeRef.module.name;
- if (packageName !== undefined) result.set(id, packageName);
- }
- return result;
-}
-
-async function realizedFindings(
- realized: Awaited<ReturnType<typeof realizeAuthoringPreview>>,
- stage: string,
- mediaTypes: ReadonlyMap<string, string>,
- samples: readonly StableSample[],
-): Promise<RawFinding[]> {
- const local = await serve(stage, mediaTypes);
- const browser = await puppeteer.launch({ executablePath: browserPath(), headless: true, args: ["--no-sandbox", "--disable-dev-shm-usage"] });
- const page = await browser.newPage();
- const findings: RawFinding[] = [];
- try {
- await page.setViewport({ width: realized.built.canvas.width, height: realized.built.canvas.height, deviceScaleFactor: 1 });
- await page.goto(local.url, { waitUntil: "load", timeout: 60_000 });
- await page.evaluate("window.__name = window.__name || ((target) => target)");
- await page.evaluate(() => {
- for (const root of Array.from(document.querySelectorAll("[data-composition-id][data-width][data-height]"))) {
- const width = Number(root.getAttribute("data-width"));
- const height = Number(root.getAttribute("data-height"));
- if (root instanceof HTMLElement && Number.isFinite(width) && Number.isFinite(height)) {
- root.style.width = `${width}px`;
- root.style.height = `${height}px`;
- }
- }
- });
- await page.evaluate(async () => { await document.fonts.ready; });
- const fps = realized.built.frameRate.numerator / realized.built.frameRate.denominator;
- for (const sample of samples) {
- const frame = sample.frame;
- const measured = await page.evaluate(async ({ frame: at, fps: rate, subjects: subjectKeys, overlapSubjects: overlapKeys }: { frame: number; fps: number; subjects: readonly string[]; overlapSubjects: readonly string[] }) => {
- type Box = { left: number; top: number; right: number; bottom: number; width: number; height: number };
- type Finding = RawFinding;
- const runtime = window as unknown as { __hf?: { seek?: (seconds: number) => unknown }; __player?: { renderSeek?: (seconds: number) => unknown } };
- const seconds = at / rate;
- const seek = runtime.__player?.renderSeek ?? runtime.__hf?.seek;
- if (typeof seek === "function") await seek.call(runtime.__player ?? runtime.__hf, seconds);
- else window.dispatchEvent(new CustomEvent("hf-seek", { detail: { time: seconds } }));
- for (const clip of Array.from(document.querySelectorAll(".hypit-visual-present[data-start][data-duration]"))) {
- const start = Number(clip.getAttribute("data-start"));
- const duration = Number(clip.getAttribute("data-duration"));
- (clip as HTMLElement).style.display = seconds >= start && seconds < start + duration ? "" : "none";
- }
- await new Promise<void>((accept) => requestAnimationFrame(() => requestAnimationFrame(() => accept())));
- const box = (rect: DOMRect): Box => ({ left: rect.left, top: rect.top, right: rect.right, bottom: rect.bottom, width: rect.width, height: rect.height });
- const visible = (element: Element): element is HTMLElement | SVGElement => {
- const style = getComputedStyle(element);
- const rect = element.getBoundingClientRect();
- return style.display !== "none" && style.visibility !== "hidden" && Number(style.opacity || "1") > 0.001 && rect.width > 0 && rect.height > 0;
- };
- const over = (child: Box, parent: Box) => Math.max(parent.left - child.left, parent.top - child.top, child.right - parent.right, child.bottom - parent.bottom, 0);
- const contains = (outer: Box, inner: Box) => outer.left <= inner.left && outer.top <= inner.top && outer.right >= inner.right && outer.bottom >= inner.bottom;
- const intersection = (a: Box, b: Box): Box | undefined => {
- const left = Math.max(a.left, b.left), top = Math.max(a.top, b.top), right = Math.min(a.right, b.right), bottom = Math.min(a.bottom, b.bottom);
- return right > left && bottom > top ? { left, top, right, bottom, width: right - left, height: bottom - top } : undefined;
- };
- const root = document.querySelector("[data-composition-id]") ?? document.body;
- const canvas = box(root.getBoundingClientRect());
- const result: Finding[] = [];
- const subjects = new Set(subjectKeys);
- const overlapSubjects = new Set(overlapKeys);
- const activePresents = Array.from(document.querySelectorAll("[data-hypit-track-id][data-hypit-present-id]")).filter(visible);
- const keyOf = (present: Element): string => `${present.getAttribute("data-hypit-track-id") ?? "unknown-track"}/${present.getAttribute("data-hypit-present-id") ?? "unknown-present"}`;
- const presents = activePresents.filter((present) => {
- if (!visible(present)) return false;
- return subjects.has(keyOf(present));
- });
- for (const present of presents) {
- const track = present.getAttribute("data-hypit-track-id") ?? "unknown-track";
- const presentId = present.getAttribute("data-hypit-present-id") ?? "unknown-present";
- const elements = Array.from(present.querySelectorAll("[data-hypit-element-id]")).filter(visible);
- for (const element of elements) {
- const id = element.getAttribute("data-hypit-element-id") ?? "unknown-element";
- const elementBox = box(element.getBoundingClientRect());
- const parentElement = element.parentElement?.closest("[data-hypit-element-id]") ?? null;
- const parent: Element = parentElement !== null && parentElement !== element ? parentElement : present;
- const parentBox = box(parent.getBoundingClientRect());
- const overflow = over(elementBox, parentBox);
- if (overflow > 1) result.push({
- kind: "parent-overflow", layer: "realized_layout", track, present: presentId, element: id,
- related: parent.getAttribute("data-hypit-element-id") ?? "present", frame: at,
- pixels: overflow, ratio: overflow / Math.max(1, parentBox.height), bounds: elementBox, related_bounds: parentBox,
- message: "Rendered content extends beyond its direct parent. This is a measured candidate; clipping or motion may be intentional.",
- });
- const canvasOverflow = over(elementBox, canvas);
- if (canvasOverflow > 1) result.push({
- kind: "canvas-overflow", layer: "realized_layout", track, present: presentId, element: id,
- related: "canvas", frame: at, pixels: canvasOverflow, ratio: canvasOverflow / Math.max(1, canvas.height),
- bounds: elementBox, related_bounds: canvas,
- message: "Rendered content extends beyond the Canvas. This is a measured candidate; cropping or off-screen motion may be intentional.",
- });
- let contentBox: Box | undefined;
- let container: Element | undefined;
- let verticalRelated: string | undefined;
- if (element.matches("[data-hypit-text-flow], [data-hypit-text-clock]") || Array.from(element.childNodes).some((node) => node.nodeType === Node.TEXT_NODE && node.textContent?.trim())) {
- const range = document.createRange();
- range.selectNodeContents(element);
- const rects = Array.from(range.getClientRects()).filter((rect) => rect.width > 0 && rect.height > 0);
- if (rects.length > 0) {
- const left = Math.min(...rects.map((rect) => rect.left)), top = Math.min(...rects.map((rect) => rect.top));
- const right = Math.max(...rects.map((rect) => rect.right)), bottom = Math.max(...rects.map((rect) => rect.bottom));
- contentBox = { left, top, right, bottom, width: right - left, height: bottom - top };
- container = element;
- verticalRelated = `${id}:content-box`;
- }
- } else if (parent !== present) {
- contentBox = elementBox;
- container = parent;
- verticalRelated = parent.getAttribute("data-hypit-element-id") ?? "parent";
- }
- if (contentBox !== undefined && container !== undefined) {
- const containerBox = box(container.getBoundingClientRect());
- const style = getComputedStyle(container);
- const contentParent = {
- left: containerBox.left + Number.parseFloat(style.paddingLeft || "0"),
- top: containerBox.top + Number.parseFloat(style.paddingTop || "0"),
- right: containerBox.right - Number.parseFloat(style.paddingRight || "0"),
- bottom: containerBox.bottom - Number.parseFloat(style.paddingBottom || "0"),
- width: 0, height: 0,
- };
- contentParent.width = contentParent.right - contentParent.left;
- contentParent.height = contentParent.bottom - contentParent.top;
- const offset = (contentBox.top + contentBox.bottom - contentParent.top - contentParent.bottom) / 2;
- if (Math.abs(offset) > 1) result.push({
- kind: "vertical-offset", layer: "realized_layout", track, present: presentId, element: id,
- ...(verticalRelated === undefined ? {} : { related: verticalRelated }), frame: at,
- pixels: Math.abs(offset), ratio: Math.abs(offset) / Math.max(1, contentParent.height),
- bounds: contentBox, related_bounds: contentParent,
- message: "Rendered inner content is vertically offset within its measured box. This is a candidate, not proof of a layout error.",
- });
- }
- }
- }
- const roots = activePresents.filter((present) => overlapSubjects.has(keyOf(present))).flatMap((present) => {
- const rootElement = Array.from(present.querySelectorAll("[data-hypit-element-id]")).find((element) => visible(element) && element.parentElement?.closest("[data-hypit-element-id]") === null);
- return rootElement === undefined ? [] : [{
- track: present.getAttribute("data-hypit-track-id") ?? "unknown-track",
- present: present.getAttribute("data-hypit-present-id") ?? "unknown-present",
- element: rootElement.getAttribute("data-hypit-element-id") ?? "unknown-element",
- bounds: box(rootElement.getBoundingClientRect()),
- }];
- });
- for (let first = 0; first < roots.length; first += 1) for (let second = first + 1; second < roots.length; second += 1) {
- const left = roots[first]!, right = roots[second]!;
- if (left.track === right.track || contains(left.bounds, right.bounds) || contains(right.bounds, left.bounds)) continue;
- const shared = intersection(left.bounds, right.bounds);
- if (shared === undefined || shared.width <= 1 || shared.height <= 1) continue;
- const area = shared.width * shared.height;
- result.push({
- kind: "peer-overlap", layer: "realized_layout", track: left.track, present: left.present,
- element: left.element, related: `${right.track}/${right.present}/${right.element}`, frame: at,
- pixels: Math.max(shared.width, shared.height),
- ratio: area / Math.max(1, Math.min(left.bounds.width * left.bounds.height, right.bounds.width * right.bounds.height)),
- bounds: left.bounds, related_bounds: right.bounds,
- message: "Rendered top-level content from independent tracks partially overlaps. This is a candidate; intentional layering remains valid.",
- });
- }
- return result;
- }, {
- frame,
- fps,
- subjects: sample.subjects.map((subject) => `${subject.track}/${subject.present}`),
- overlapSubjects: sample.overlap_subjects.map((subject) => `${subject.track}/${subject.present}`),
- });
- findings.push(...measured);
- }
- } finally {
- await page.close().catch(() => undefined);
- await browser.close().catch(() => undefined);
- await local.close().catch(() => undefined);
- }
- return findings;
-}
-
-function aggregate(findings: readonly RawFinding[]): RawFinding[] {
- const grouped = new Map<string, RawFinding>();
- for (const finding of findings) {
- const key = JSON.stringify([finding.layer, finding.kind, finding.track ?? "", finding.present ?? "", finding.element, finding.related ?? ""]);
- const current = grouped.get(key);
- if (current === undefined || finding.pixels > current.pixels) grouped.set(key, finding);
- }
- return [...grouped.values()].map((finding) => ({ ...finding, pixels: round(finding.pixels), ratio: round(finding.ratio) }));
-}
-
-function findingId(finding: RawFinding, packageName: string | undefined): string {
- return [finding.kind, packageName, finding.track, finding.present, finding.element, finding.related]
- .filter((part): part is string => part !== undefined && part.length > 0)
- .map((part) => encodeURIComponent(part))
- .join("/");
-}
-
-function withIdentity(findings: readonly RawFinding[], packages: ReadonlyMap<string, string>, decisions: LayoutDecisions): readonly Record<string, unknown>[] {
- const accepted = new Map(decisions.decisions.map((item) => [item.finding, item] as const));
- return findings.map((finding) => {
- const packageName = finding.track === undefined ? undefined : packages.get(finding.track);
- const id = findingId(finding, packageName);
- const decision = accepted.get(id);
- return {
- id, ...finding, ...(packageName === undefined ? {} : { package: packageName }),
- disposition: decision === undefined ? "agent-review-required" : "accepted-intentional",
- ...(decision === undefined ? {} : { acceptance_reason: decision.reason, accepted_at: decision.accepted_at }),
- };
- });
-}
-
-export async function layoutCheck(input: LayoutCheckInput): Promise<Record<string, unknown>> {
- const runPath = resolve(invokedFrom(), input.run);
- const source = await authorSource(runPath);
- const realized = await realizeAuthoringPreview(input);
- const { stage, document, mediaTypes } = await stageAuthoringPreview(realized);
- const decisionsPath = layoutDecisionsPath(runPath);
- const decisions = await readDecisions(decisionsPath);
- await writeJson(decisionsPath, decisions);
- const currentPath = layoutCheckPath(runPath);
- const samples = framePlan(realized.built.composition).filter((sample) => sample.frame < document.frameCount);
- const raw = aggregate(await realizedFindings(realized, stage, mediaTypes, samples));
- const findings = withIdentity(raw, packageByTrack(realized), decisions);
- const pending = findings.filter((finding) => finding.disposition === "agent-review-required");
- const result = {
- run: runPath,
- source: source.path,
- checked: true,
- reviewed: pending.length === 0,
- semantics: "advisory-measurements",
- note: "Mechanical measurements assist the Agent; they are not findings of fault and never override design intent. Repair a genuine issue or use layout_accept with a reason for intentional geometry.",
- samples,
- raw_findings: raw,
- findings,
- pending_count: pending.length,
- accepted_count: findings.length - pending.length,
- checked_at: new Date().toISOString(),
- };
- await writeJson(currentPath, result);
- return { ...result, report: currentPath, decisions: decisionsPath };
-}
-
-export async function layoutAccept(input: LayoutAcceptInput): Promise<Record<string, unknown>> {
- const runPath = resolve(invokedFrom(), input.run);
- const acceptances = input.findings ?? (input.finding === undefined || input.reason === undefined
- ? []
- : [{ finding: input.finding, reason: input.reason }]);
- if (acceptances.length === 0) throw new Error("layout_accept requires --finding/--reason or a non-empty findings batch");
- const reportPath = layoutCheckPath(runPath);
- const report = JSON.parse(await readFile(reportPath, "utf8")) as { findings?: unknown };
- if (!Array.isArray(report.findings)) throw new Error(`run layout_check first; ${reportPath} has no usable report`);
- const path = layoutDecisionsPath(runPath);
- const existing = await readDecisions(path);
- const decisions = [...existing.decisions];
- for (const acceptance of acceptances) {
- const reason = acceptance.reason.trim();
- if (reason.length === 0) throw new Error(`layout_accept requires a non-empty reason for ${acceptance.finding}`);
- const finding = report.findings.find((item) => item !== null && typeof item === "object" && (item as { id?: unknown }).id === acceptance.finding);
- if (finding === undefined) throw new Error(`layout report contains no finding ${acceptance.finding}`);
- const decision: LayoutDecision = { finding: acceptance.finding, reason, accepted_at: new Date().toISOString() };
- const index = decisions.findIndex((item) => item.finding === decision.finding);
- if (index >= 0) decisions.splice(index, 1);
- decisions.push(decision);
- }
- await writeJson(path, { decisions });
- return {
- accepted: true,
- findings: acceptances.map((item) => item.finding),
- decisions: path,
- next_action: `rerun layout_check --run ${runPath}`,
- };
-}
diff --git a/packages/reference-video-tools/src/media.ts b/packages/reference-video-tools/src/media.ts
deleted file mode 100644
index 49828653..00000000
--- a/packages/reference-video-tools/src/media.ts
+++ /dev/null
@@ -1,289 +0,0 @@
-import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
-import { cpus } from "node:os";
-import { join, resolve } from "node:path";
-import { spawn } from "node:child_process";
-
-export function assert(condition: unknown, message: string): asserts condition {
- if (!condition) throw new Error(message);
-}
-
-export function round(value: number): number { return Number(value.toFixed(3)); }
-export function clamp(value: number, low: number, high: number): number { return Math.max(low, Math.min(high, value)); }
-
-export async function ensureDir(path: string): Promise<void> { await mkdir(path, { recursive: true }); }
-
-export async function command(executable: string, args: readonly string[], timeoutMs = 120_000): Promise<Buffer> {
- return await new Promise((resolvePromise, reject) => {
- const child = spawn(executable, [...args], { stdio: ["ignore", "pipe", "pipe"], shell: false, windowsHide: true });
- const stdout: Buffer[] = [];
- let stderr = "";
- let settled = false;
- const timer = setTimeout(() => { child.kill("SIGKILL"); finish(new Error(`${executable} timed out`)); }, timeoutMs);
- const finish = (error?: Error): void => {
- if (settled) return;
- settled = true;
- clearTimeout(timer);
- if (error !== undefined) reject(error);
- else resolvePromise(Buffer.concat(stdout));
- };
- child.stdout.on("data", (chunk: Buffer) => stdout.push(chunk));
- child.stderr.on("data", (chunk: Buffer) => { stderr = `${stderr}${chunk.toString("utf8")}`.slice(-1_000_000); });
- child.on("error", (error) => finish(error instanceof Error ? error : new Error(String(error))));
- child.on("close", (code) => code === 0 ? finish() : finish(new Error(`${executable} failed (${code}): ${stderr.trim()}`)));
- });
-}
-
-export async function probe(path: string): Promise<{ duration: number; width: number; height: number; frameRate: number; hasAudio: boolean }> {
- const raw = await command("ffprobe", ["-v", "error", "-show_entries", "format=duration:stream=codec_type,width,height,r_frame_rate", "-of", "json", path]);
- const parsed = JSON.parse(raw.toString("utf8")) as { format?: { duration?: string }; streams?: readonly { codec_type?: string; width?: number; height?: number; r_frame_rate?: string }[] };
- const video = parsed.streams?.find((item) => item.codec_type === "video");
- const duration = Number(parsed.format?.duration);
- assert(Number.isFinite(duration) && duration > 0, "video duration is unavailable");
- assert(video?.width !== undefined && video.height !== undefined, "video dimensions are unavailable");
- // ffprobe reports the rate as a ratio, so it is divided rather than parsed as a number. A still
- // image has no rate to report and a caller that needs one asks for it from a clip.
- const ratio = (video.r_frame_rate ?? "").split("/");
- const rate = Number(ratio[0]) / Number(ratio[1] ?? 1);
- return {
- duration, width: video.width, height: video.height,
- frameRate: Number.isFinite(rate) && rate > 0 ? rate : 0,
- hasAudio: parsed.streams?.some((item) => item.codec_type === "audio") ?? false,
- };
-}
-
-/**
- * Whether a clip holds one picture for the whole of its length.
- *
- * `freezedetect` reports each stretch it saw no change across as a start, and as an end where the
- * picture changes again. One stretch that opens on the first frame and never closes is the whole
- * clip; a stretch that opens later, or one that closes before the clip does, means something moved.
- * The times are measured to a frame, so an end landing on the last frame is the clip ending.
- */
-export async function completelyStill(path: string): Promise<boolean> {
- const { duration, frameRate } = await probe(path);
- // The filter announces a stretch only once it has run for `d`, and announces it at the second it
- // began rather than the second it was noticed, so `d` has only to be shorter than the clip.
- // `metadata=print` writes what it found to stdout, which is where the result is read from.
- const printed = await command("ffmpeg", [
- "-hide_banner", "-loglevel", "error", "-i", path, "-an",
- "-vf", "freezedetect=n=-60dB:d=0.1,metadata=mode=print:file=-", "-f", "null", "-",
- ], 300_000);
- const text = printed.toString("utf8");
- const times = (name: string): readonly number[] =>
- [...text.matchAll(new RegExp(String.raw`freezedetect\.freeze_${name}=([0-9.]+)`, "gu"))].map((match) => Number(match[1]));
- const tolerance = frameRate > 0 ? 1 / frameRate : 0.05;
- const starts = times("start");
- return starts.length === 1 && starts[0]! <= tolerance && times("end").every((at) => at >= duration - tolerance);
-}
-
-function distance(left: Uint8Array, right: Uint8Array): number {
- let total = 0;
- for (let index = 0; index < left.length; index += 1) total += Math.abs(left[index]! - right[index]!);
- return total / left.length / 255;
-}
-
-async function signatures(path: string, sampleRate: number): Promise<readonly Uint8Array[]> {
- const bytes = await command("ffmpeg", ["-hide_banner", "-loglevel", "error", "-i", path, "-vf", `fps=${sampleRate},scale=32:32,format=rgb24`, "-f", "rawvideo", "-"]);
- const stride = 32 * 32 * 3;
- const result: Uint8Array[] = [];
- for (let index = 0; index + stride <= bytes.byteLength; index += stride) result.push(bytes.subarray(index, index + stride));
- return result;
-}
-
-// Boundaries are only as precise as this rate, and a clip cut on an imprecise boundary opens on
-// content belonging to the shot before it. At three samples a second that overhang reached a third
-// of a second — long enough for an observer to describe a shot as the picture that precedes it, and
-// to read a list that had been typing all along as one whose letters were being deleted. The whole
-// video is decoded once whatever the rate, so sampling four times as often costs four times as many
-// 32×32 signatures and no additional decoding.
-const SAMPLE_RATE = 12;
-
-async function contentBoundaries(path: string, duration: number): Promise<readonly number[]> {
- const frames = await signatures(path, SAMPLE_RATE);
- if (frames.length < 2) return [0, duration];
- const starts = [0];
- let anchor = frames[0]!;
- for (let index = 1; index < frames.length; index += 1) {
- if (distance(frames[index]!, anchor) > 0.1) {
- starts.push(round(index / SAMPLE_RATE));
- anchor = frames[index]!;
- }
- }
- // A boundary closer than this to the one before it does not describe a shot: it describes a
- // flicker, a flash frame or a fast pan. Clips that short are also rejected outright by the model,
- // so admitting them costs a request and returns nothing.
- const shortest = 1;
- const kept = [starts[0]!];
- for (const value of starts.slice(1)) {
- if (value - kept.at(-1)! >= shortest) kept.push(value);
- }
- // The tail is a boundary too, and a final fragment belongs to the shot before it rather than
- // becoming a shot of its own.
- if (duration - kept.at(-1)! < shortest && kept.length > 1) kept.pop();
- return [...kept, duration];
-}
-
-type Bounds = { readonly start: number; readonly end: number; readonly group: number; readonly part: number; readonly parts: number };
-function splitLong(boundaries: readonly number[]): readonly Bounds[] {
- const result: Bounds[] = [];
- for (let group = 0; group + 1 < boundaries.length; group += 1) {
- const start = boundaries[group]!;
- const end = boundaries[group + 1]!;
- const length = end - start;
- const parts = Math.max(1, Math.ceil(length / 15));
- for (let part = 0; part < parts; part += 1) result.push({ start: round(start + length * part / parts), end: round(start + length * (part + 1) / parts), group, part: part + 1, parts });
- }
- return result;
-}
-
-async function extract(path: string, args: readonly string[], target: string): Promise<string> {
- await command("ffmpeg", ["-hide_banner", "-loglevel", "error", "-y", ...args, target], 300_000);
- return target;
-}
-
-/** How far before the mark to jump, leaving a short run-up to decode exactly. */
-const SEEK_RUN_UP = 2;
-
-/**
- * Arguments that open `source` sitting on the frame at `at` seconds.
- *
- * Seeking only after `-i` decodes every frame from the start of the file and discards it, so opening
- * a shot nine minutes in costs nine minutes of decoding. Seeking only before `-i` is immediate but
- * lands on the nearest keyframe at or before the mark, which is a different picture. Doing both —
- * jump to a keyframe a little early, then decode the short run-up — lands on the frame the slow way
- * lands on. Measured on a 1080p source at nine minutes in: the same JPEG to the byte, 11.6s to 0.2s.
- */
-function openAt(source: string, at: number): readonly string[] {
- const jump = Math.max(0, at - SEEK_RUN_UP);
- const runUp = at - jump;
- return [
- ...(jump > 0 ? ["-ss", String(round(jump))] : []),
- "-i", source,
- ...(runUp > 0 ? ["-ss", String(round(runUp))] : []),
- ];
-}
-
-/**
- * Run `work` over every item, with no more than `limit` of them in flight.
- *
- * A shot is cheap on its own and a long video has hundreds of them. Starting every one at once
- * leaves that many ffmpeg processes competing for the same cores, so each takes long enough to be
- * killed for hanging while all of them are in fact working.
- */
-async function inPool<T>(items: readonly T[], limit: number, work: (item: T, index: number) => Promise<void>): Promise<void> {
- let cursor = 0;
- const worker = async (): Promise<void> => {
- while (true) {
- const index = cursor++;
- if (index >= items.length) return;
- await work(items[index]!, index);
- }
- };
- await Promise.all(Array.from({ length: Math.max(1, Math.min(limit, items.length)) }, worker));
-}
-
-// An observer that reads images rather than video sees a shot as this many frames, sampled evenly
-// across it and tiled into one picture in reading order. Four is enough for a one-second shot to
-// show what moves; nine keeps a fifteen-second one legible at a cell width a reader can still resolve
-// detail in. The clip is decoded once, and the sampling and the tiling happen in that one pass.
-const TILE_COLUMNS = 3;
-/** How wide one frame is drawn in a grid, when the caller has no reference width to match. */
-const TILE_CELL_WIDTH = 480;
-export function tileFrames(duration: number): number { return clamp(Math.round(duration * 1.5), 4, 9); }
-
-/**
- * One stretch of a video, as a clip of its own.
- *
- * `-ss` goes after `-i` for the same reason it does when a shot is cut: seeking the output decodes
- * from the start and lands on the frame asked for, where seeking the input lands on the keyframe
- * before it. Both sides of a comparison are cut this way, so a difference between them is a
- * difference in the pictures rather than in how far each one overshot its start.
- */
-export async function cutClip(source: string, start: number, seconds: number, target: string): Promise<string> {
- return await extract(source, [
- "-i", source, "-ss", String(round(start)), "-t", String(Math.max(0.1, round(seconds))),
- "-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-c:a", "aac", "-b:a", "96k", "-movflags", "+faststart",
- ], target);
-}
-
-/** One frame of a video, at the second asked for. */
-export async function cutFrame(source: string, at: number, target: string): Promise<string> {
- // `-pix_fmt yuvj420p` because JPEG is a full-range format and the encoder refuses full-range YUV
- // input without being told which one it is writing. Video from the web is routinely full-range, so
- // leaving it out refuses the still with `Non full-range YUV is non-standard` and writes nothing.
- return await extract(source, ["-i", source, "-ss", String(round(at)), "-frames:v", "1", "-pix_fmt", "yuvj420p", "-q:v", "3"], target);
-}
-
-/**
- * A grid of frames sampled evenly across a clip, for an observer that reads pictures.
- *
- * `cellWidth` is the width each frame is drawn at. Both sides of a comparison are asked for the same
- * one, because a pair drawn at two widths is read as a difference in the thing being compared: a
- * reference stretched up and a render squeezed down disagree about stroke weight and letter spacing
- * before anyone looks at them. The caller picks the width the reference can actually supply, so
- * `min` here keeps a small source from being enlarged into the same number of pixels carrying less.
- */
-export async function shotTile(
- clip: string,
- duration: number,
- target: string,
- cellWidth = TILE_CELL_WIDTH,
- frameCount = tileFrames(duration),
-): Promise<string> {
- const frames = Math.max(1, Math.round(frameCount));
- const rows = Math.ceil(frames / TILE_COLUMNS);
- const rate = frames / Math.max(duration, 0.1);
- await command("ffmpeg", [
- "-hide_banner", "-loglevel", "error", "-y", "-i", clip,
- "-vf", `fps=${rate},scale='min(${Math.round(cellWidth)},iw)':-2,tile=layout=${TILE_COLUMNS}x${rows}:padding=8:margin=8:color=black`,
- "-frames:v", "1", "-pix_fmt", "yuvj420p", "-q:v", "3", target,
- ], 300_000);
- return target;
-}
-
-export async function prepareMedia(videoPath: string, root: string, duration: number, tiles: boolean): Promise<{
- readonly bounds: readonly Bounds[];
- readonly storyboard: string;
- readonly analysisVideo: string;
-}> {
- const boundaries = await contentBoundaries(videoPath, duration);
- const bounds = splitLong(boundaries);
- await ensureDir(root);
- const analysisVideo = await extract(videoPath, ["-i", videoPath, "-vf", "scale='min(720,iw)':-2", "-c:v", "libx264", "-preset", "veryfast", "-crf", "30", "-c:a", "aac", "-b:a", "64k"], join(root, "analysis.mp4"));
- const shotDir = join(root, "shots");
- await ensureDir(shotDir);
- // Bounded rather than all at once: a fifteen-second rule cuts a ten-minute reference into nearly
- // two hundred shots, and two hundred concurrent ffmpeg runs starve each other.
- await inPool(bounds, Math.max(1, cpus().length - 1), async (bound, index) => {
- const id = String(index + 1).padStart(3, "0");
- const clip = join(shotDir, `${id}.mp4`);
- await extract(videoPath, [...openAt(videoPath, bound.start), "-t", String(Math.max(0.1, bound.end - bound.start)), "-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-c:a", "aac", "-b:a", "96k", "-movflags", "+faststart"], clip);
- await extract(videoPath, [...openAt(videoPath, bound.start + (bound.end - bound.start) / 2), "-frames:v", "1", "-vf", "scale='min(720,iw)':-2", "-pix_fmt", "yuvj420p", "-q:v", "3"], join(shotDir, `${id}-representative.jpg`));
- await extract(videoPath, ["-sseof", "-0.1", "-i", clip, "-update", "1", "-frames:v", "1", "-vf", "scale='min(720,iw)':-2", "-pix_fmt", "yuvj420p", "-q:v", "3"], join(shotDir, `${id}-tail.jpg`));
- // Only the observer that reads pictures has anything to read them from, and building a tile per
- // shot is a decode per shot. The other observer is handed the clip itself.
- if (tiles) await shotTile(clip, bound.end - bound.start, join(shotDir, `${id}-frames.jpg`));
- if (await hasAudio(clip)) await extract(videoPath, ["-sseof", "-3", "-i", clip, "-map", "0:a:0", "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le"], join(shotDir, `${id}-audio.wav`));
- });
- const inputs = bounds.map((_, index) => `[${index}:v]`).join("");
- const list = bounds.flatMap((_, index) => ["-i", join(shotDir, `${String(index + 1).padStart(3, "0")}-representative.jpg`)]);
- const columns = Math.min(4, bounds.length);
- const rows = Math.ceil(bounds.length / columns);
- const storyboard = join(root, "storyboard.jpg");
- await command("ffmpeg", ["-hide_banner", "-loglevel", "error", "-y", ...list, "-filter_complex", `${inputs}concat=n=${bounds.length}:v=1:a=0,tile=layout=${columns}x${rows}:padding=8:margin=8`, "-frames:v", "1", storyboard]);
- return { bounds, storyboard, analysisVideo };
-}
-
-async function hasAudio(path: string): Promise<boolean> {
- try { await command("ffprobe", ["-v", "error", "-select_streams", "a:0", "-show_entries", "stream=index", "-of", "csv=p=0", path]); return true; } catch { return false; }
-}
-
-export async function readBytes(path: string): Promise<Uint8Array> { return await readFile(path); }
-export async function writeJson(path: string, value: unknown): Promise<void> { await writeFile(path, `${JSON.stringify(value, null, 2)}\n`, "utf8"); }
-export async function readJson<T>(path: string): Promise<T | undefined> { try { return JSON.parse(await readFile(path, "utf8")) as T; } catch { return undefined; } }
-export async function referenceId(videoPath: string): Promise<string> {
- const info = await stat(videoPath);
- const name = resolve(videoPath).split(/[\\/]/u).at(-1)?.replace(/\.[^.]+$/u, "")
- .replace(/[^0-9A-Za-z._-]+/gu, "-").replace(/^-+|-+$/gu, "") || "video";
- return `ref-${name}-${info.size}-${Math.trunc(info.mtimeMs)}`;
-}
diff --git a/packages/reference-video-tools/src/plan.ts b/packages/reference-video-tools/src/plan.ts
deleted file mode 100644
index 7a9c07dc..00000000
--- a/packages/reference-video-tools/src/plan.ts
+++ /dev/null
@@ -1,352 +0,0 @@
-/**
- * Which stretches of a program are worth looking at, decided from the Source alone.
- *
- * An element drawn over five Segments does not need five looks. It needs one per distinct visual
- * declaration. Deciding that by reading the Source is what makes a round reproducible and what lets
- * the gate ask for coverage rather than for participation.
- *
- * **A distinct declaration is a distinct picture.** Group an element's placements by what they
- * declare and look at each group once, where it first appears. This is also the floor: every placed
- * element has at least one declaration, so every element gets at least one look.
- *
- * The unit is a token range rather than a Segment name because both sides of a comparison already
- * work in words — `frameOfToken` here, `spokenRange` on the reference — so the range is what they
- * both take, and a name comes along only to be read.
- */
-import { parseScript } from "@hypit/script";
-import type { ParsedNarrative } from "@hypit/script";
-
-/** One thing to look at: an element, over a range of the Script's words, and why it earned a look. */
-export type PlanEntry = {
- readonly element: string;
- /** Half-open over the Script's speech tokens, the same index space `frameOfToken` is in. */
- readonly tokens: readonly [number, number];
- /** What that range is, for a reader. Never parsed back — the range is the identity. */
- readonly named: string;
- readonly why: readonly string[];
-};
-
-/** A drawing element as the gate enumerates it, before its placements are read. */
-export type PlannedElement = {
- readonly id: string;
- readonly tag: string;
- readonly alias: string;
- readonly specifier: string;
-};
-
-/**
- * Attributes that say *when* rather than *what*, so they never join a declaration key.
- *
- * Two placements differing only in these are the same design over different words, which is one
- * look and two stretches — the case rule 1 exists to collapse.
- */
-const WINDOW_ATTRIBUTES = new Set([
- "during", "at", "for", "start", "end", "selection", "segment", "moment", "until",
- "until-boundary", "terminal", "semantic", "document", "narrative", "program-ref",
-]);
-
-/**
- * Attributes naming material a Build has not made, so they never join a declaration key either.
- *
- * Every one of these is a flat placeholder in the render this plan is for. Two Items differing only
- * in which unbuilt picture they name draw the identical mock, so looking at both spends a subagent
- * to see the same grey rectangle twice.
- */
-const MOCKED_ATTRIBUTES = new Set(["image", "video", "media", "source", "icon", "audio", "source-audio"]);
-
-/** Never a declaration: the id is what the key is being computed *for*. */
-const IGNORED_ATTRIBUTES = new Set(["id"]);
-
-type Placement = {
- readonly element: string;
- readonly tag: string;
- readonly key: string;
- readonly stretch: Stretch;
-};
-
-type Stretch = {
- readonly tokens: readonly [number, number];
- readonly named: string;
-};
-
-/** Every attribute on an opening tag, as written. */
-function attributesOf(text: string): ReadonlyMap<string, string> {
- const found = new Map<string, string>();
- for (const match of text.matchAll(/\b([a-z][a-z0-9-]*)="([^"]*)"/gu)) found.set(match[1] ?? "", match[2] ?? "");
- // A reference is written `attr={id}` rather than quoted, and it is the form every Recipe and Frame
- // arrives in, so a key built from quoted attributes alone would miss the whole declaration.
- for (const match of text.matchAll(/\b([a-z][a-z0-9-]*)=\{([^}]*)\}/gu)) found.set(match[1] ?? "", `{${match[2] ?? ""}}`);
- return found;
-}
-
-/**
- * What this placement declares, as one comparable string.
- *
- * Everything on the tag except the three excluded sets, sorted so attribute order — which this
- * markup does not make significant — cannot split one group into two. The tag name leads, because
- * the same attributes on a different element are a different picture.
- *
- * Taking everything rather than a list of known Recipe attributes is deliberate. `screen-overlay`
- * declares its entire appearance as tag literals and names no Recipe at all, and a package added
- * tomorrow will be read correctly without this file learning its vocabulary. The cost of the wide
- * net is a split group, which is one extra look; the cost of a narrow one is a merged group, which
- * is a picture nobody sees.
- */
-function declarationKey(tag: string, attributes: ReadonlyMap<string, string>): string {
- const parts: string[] = [];
- for (const [name, value] of attributes) {
- if (WINDOW_ATTRIBUTES.has(name) || MOCKED_ATTRIBUTES.has(name) || IGNORED_ATTRIBUTES.has(name)) continue;
- parts.push(`${name}=${value}`);
- }
- return `${tag}|${parts.sort().join("|")}`;
-}
-
-/** The whole Script, for an element bound to the program rather than to a marked range. */
-function wholeProgram(parsed: ParsedNarrative): Stretch {
- return { tokens: [0, parsed.tokens.length], named: "the whole program" };
-}
-
-/**
- * Where a placement's `during=` or `at=` puts it, in words.
- *
- * Returns undefined when the binding names something the Script does not mark, which is a Source
- * defect `preview_check` reports rather than something to guess at here.
- */
-function stretchOf(attributes: ReadonlyMap<string, string>, parsed: ParsedNarrative): Stretch | undefined {
- const during = attributes.get("during");
- if (during === "program") return wholeProgram(parsed);
-
- const named = (value: string | undefined): { readonly kind: string; readonly id: string } | undefined => {
- const match = /^\{story\.(segment|selection|moment)\.([A-Za-z0-9_-]+)\}$/u.exec(value ?? "");
- return match === undefined || match === null ? undefined : { kind: match[1]!, id: match[2]! };
- };
-
- const bound = named(during) ?? named(attributes.get("at"));
- if (bound === undefined) return undefined;
-
- if (bound.kind === "segment") {
- const segment = parsed.segments.find((item) => item.id === bound.id);
- return segment === undefined
- ? undefined
- : { tokens: [segment.tokenStart, segment.tokenEndExclusive], named: `segment ${bound.id}` };
- }
- if (bound.kind === "selection") {
- const selection = parsed.selections.find((item) => item.id === bound.id);
- return selection === undefined
- ? undefined
- : {
- tokens: [selection.open.boundary.tokenIndex, selection.close.boundary.tokenIndex],
- named: `selection ${bound.id}`,
- };
- }
- // A Moment is an instant, not a range. What it marks is a change inside whichever Segment holds
- // it, so that Segment is the stretch a look would be rendered over.
- const moment = parsed.moments.find((item) => item.id === bound.id);
- if (moment === undefined) return undefined;
- const index = moment.boundary.tokenIndex;
- const holding = parsed.segments.find((item) => index >= item.tokenStart && index < item.tokenEndExclusive);
- return holding === undefined
- ? undefined
- : { tokens: [holding.tokenStart, holding.tokenEndExclusive], named: `segment ${holding.id} (at moment ${bound.id})` };
-}
-
-/**
- * The body of one element, so its children can be read.
- *
- * These Tracks do not nest inside themselves, so the first matching close tag is this element's.
- * Returns an empty string for a self-closing element, which has no children to find.
- */
-function bodyOf(svml: string, alias: string, tag: string, id: string): string {
- const opening = new RegExp(`<${alias}:${tag}\\b[^>]*?\\bid="${id}"[^>]*?(/?)>`, "su").exec(svml);
- if (opening === null || opening[1] === "/") return "";
- const after = svml.slice(opening.index + opening[0].length);
- const close = after.indexOf(`</${alias}:${tag}>`);
- return close === -1 ? after : after.slice(0, close);
-}
-
-/**
- * What an element's own children add to its declaration.
- *
- * A child that binds a window is a placement in its own right and is read as one. A child that does
- * not is part of how its parent looks: `<media-track:Sampling>` is a zoom across the whole window,
- * `<text:Motion>` holds the keyframes, `<caption-fine:Fallback>` names the font to reach for, and a
- * `<P>` or `<Span>` overrides the Style for one run of text. None of them appears on the parent tag,
- * so a key built from attributes alone reads two Items as one design when only one of them animates.
- *
- * Sorted, because the order children are written in is not what they look like.
- */
-function childDeclarations(body: string, parsed: ParsedNarrative): readonly string[] {
- const parts: string[] = [];
- for (const child of body.matchAll(/<([a-z][a-z0-9-]*:[A-Za-z][A-Za-z0-9]*|[A-Z][A-Za-z0-9]*)\b([^>]*?)\/?>/gsu)) {
- const tag = child[1] ?? "";
- const attributes = attributesOf(child[0] ?? "");
- if (stretchOf(attributes, parsed) !== undefined) continue;
- parts.push(declarationKey(tag, attributes));
- }
- return parts.sort();
-}
-
-/** The opening tag of one element, by id. */
-function openingOf(svml: string, alias: string, tag: string, id: string): string {
- return new RegExp(`<${alias}:${tag}\\b[^>]*?\\bid="${id}"[^>]*?/?>`, "su").exec(svml)?.[0] ?? "";
-}
-
-/**
- * The Styles a caption Program hands out, and the stretch each one covers.
- *
- * A caption Track is one tag with no window on it — it draws wherever the Script has words — so
- * reading its placements the way every other element is read finds exactly one, and reports one
- * design. That is wrong whenever the Program overrides a Style: `<caption:Use style=… selection=…>`
- * is how one stretch is drawn differently from the rest, and it is written inside the Program rather
- * than on the Track, so nothing on the Track says it happened.
- *
- * Each override is its own declaration over its own words. The default is a declaration too, over
- * whatever the overrides did not claim, which is where the Track's single placement belongs.
- */
-function captionStyles(svml: string, element: PlannedElement, parsed: ParsedNarrative): readonly Placement[] {
- const opening = openingOf(svml, element.alias, element.tag, element.id);
- const program = /\bprogram=\{([A-Za-z0-9_-]+)\}/u.exec(opening)?.[1];
- if (program === undefined) return [];
- const declaring = new RegExp(`<[a-z][a-z0-9-]*:Program\\b[^>]*?\\bid="${program}"[^>]*?(/?)>`, "su").exec(svml);
- if (declaring === null) return [];
-
- const body = declaring[1] === "/" ? "" : (() => {
- const after = svml.slice(declaring.index + declaring[0].length);
- const close = after.search(/<\/[a-z][a-z0-9-]*:Program>/u);
- return close === -1 ? after : after.slice(0, close);
- })();
-
- const tag = `${element.alias}:${element.tag}`;
- const found: Placement[] = [];
- for (const use of body.matchAll(/<[a-z][a-z0-9-]*:Use\b([^>]*?)\/?>/gsu)) {
- const attributes = attributesOf(use[0] ?? "");
- // A Use names its stretch with `selection=`, not the `during=` every drawn element uses, because
- // it is assigning a Style to a run of words rather than placing a picture over them.
- const marked = /^\{story\.selection\.([A-Za-z0-9_-]+)\}$/u.exec(attributes.get("selection") ?? "")?.[1];
- const selection = marked === undefined ? undefined : parsed.selections.find((item) => item.id === marked);
- const stretch = selection === undefined
- ? stretchOf(attributes, parsed)
- : {
- tokens: [selection.open.boundary.tokenIndex, selection.close.boundary.tokenIndex] as readonly [number, number],
- named: `selection ${marked}`,
- };
- const style = attributes.get("style");
- if (stretch === undefined || style === undefined) continue;
- found.push({
- element: element.id, tag: `${tag} (caption:Use)`,
- key: `${tag}|style=${style}`, stretch,
- });
- }
- return found;
-}
-
-/**
- * Every placement one drawing element makes.
- *
- * A Track holding forty Items draws nothing itself: the Items carry the windows and the Recipes, and
- * they are what a look is actually of. So the element's own tag is a placement only when it binds a
- * window itself, and each child that binds one is a placement too.
- */
-function placementsOf(svml: string, element: PlannedElement, parsed: ParsedNarrative): readonly Placement[] {
- const found: Placement[] = [];
- const opening = openingOf(svml, element.alias, element.tag, element.id);
- const own = attributesOf(opening);
- const body = bodyOf(svml, element.alias, element.tag, element.id);
- const ownKey = [declarationKey(`${element.alias}:${element.tag}`, own), ...childDeclarations(body, parsed)].join("&&");
- const ownStretch = stretchOf(own, parsed);
- if (ownStretch !== undefined) {
- found.push({
- element: element.id, tag: `${element.alias}:${element.tag}`,
- key: ownKey, stretch: ownStretch,
- });
- }
-
- for (const child of body.matchAll(/<([a-z][a-z0-9-]*:[A-Za-z][A-Za-z0-9]*)\b([^>]*?)(\/?)>/gsu)) {
- const tag = child[1] ?? "";
- const attributes = attributesOf(child[0] ?? "");
- const stretch = stretchOf(attributes, parsed);
- if (stretch === undefined) continue;
- // An Item's own children — its Sampling, its Paint, its inline runs — are part of how it looks.
- const inner = child[3] === "/" ? "" : (() => {
- const after = body.slice((child.index ?? 0) + (child[0]?.length ?? 0));
- const close = after.indexOf(`</${tag}>`);
- return close === -1 ? "" : after.slice(0, close);
- })();
- // The child's own declaration is what it draws, but it sits inside the Track's, so the Track's
- // is part of the key: the same Item under two Tracks with different Canvases is two pictures.
- found.push({
- element: element.id,
- tag,
- key: `${ownKey}>>${[declarationKey(tag, attributes), ...childDeclarations(inner, parsed)].join("&&")}`,
- stretch,
- });
- }
-
- // An element that binds no window anywhere is on screen for the whole program — a persistent
- // badge, a Film-wide treatment — and the whole program is the stretch it is looked at over.
- if (found.length === 0) {
- found.push({
- element: element.id,
- tag: `${element.alias}:${element.tag}`,
- key: ownKey,
- stretch: wholeProgram(parsed),
- });
- }
- return found;
-}
-
-/** One look per distinct declaration, at the stretch where that declaration first appears. */
-function firstOfEachDeclaration(placements: readonly Placement[]): readonly PlanEntry[] {
- const seen = new Set<string>();
- const entries: PlanEntry[] = [];
- for (const placement of placements) {
- if (seen.has(placement.key)) continue;
- seen.add(placement.key);
- entries.push({
- element: placement.element,
- tokens: placement.stretch.tokens,
- named: placement.stretch.named,
- why: [`first use of this declaration (${placement.tag})`],
- });
- }
- return entries;
-}
-
-/**
- * Merge nominations that landed on the same thing.
- *
- * Two distinct declarations can first appear over the same stretch. That is one rendered picture
- * with both reasons, not two copies of the same picture.
- */
-function merge(entries: readonly PlanEntry[]): readonly PlanEntry[] {
- const byKey = new Map<string, PlanEntry>();
- for (const entry of entries) {
- const key = `${entry.element}${entry.tokens[0]}-${entry.tokens[1]}`;
- const held = byKey.get(key);
- if (held === undefined) { byKey.set(key, entry); continue; }
- byKey.set(key, { ...held, why: [...held.why, ...entry.why.filter((reason) => !held.why.includes(reason))] });
- }
- return [...byKey.values()];
-}
-
-/** What a round has to look at, for one Source. */
-export function reviewPlan(input: {
- readonly svml: string;
- readonly svmlPath: string;
- readonly scriptBody: { readonly id: string; readonly text: string; readonly offset: number };
- readonly drawn: readonly PlannedElement[];
-}): readonly PlanEntry[] {
- const parsed = parseScript(input.svmlPath, input.scriptBody.text, input.scriptBody.offset);
- const entries: PlanEntry[] = [];
-
- for (const element of input.drawn) {
- // A caption Track's designs are declared inside its Program, not on the Track, so they are read
- // separately and joined to whatever the ordinary placement scan found.
- const placements = [
- ...placementsOf(input.svml, element, parsed),
- ...(element.specifier.includes("caption") ? captionStyles(input.svml, element, parsed) : []),
- ];
- entries.push(...firstOfEachDeclaration(placements));
- }
- return merge(entries);
-}
diff --git a/packages/reference-video-tools/src/schema.ts b/packages/reference-video-tools/src/schema.ts
deleted file mode 100644
index 8934a3e5..00000000
--- a/packages/reference-video-tools/src/schema.ts
+++ /dev/null
@@ -1,48 +0,0 @@
-import type { ValueSchema } from "@hypit/protocol";
-
-/** Render a declared value shape as prose an author can act on. */
-export function describeSchema(schema: ValueSchema, indent = ""): string[] {
- const step = `${indent} `;
- switch (schema.kind) {
- case "object": {
- const lines: string[] = [];
- for (const [name, field] of Object.entries(schema.fields)) {
- const inner = describeSchema(field.schema, step);
- const optional = field.optional === true ? " (optional)" : "";
- lines.push(`${indent}${name}${optional}: ${inner[0]!.trimStart()}`, ...inner.slice(1));
- }
- if (schema.allowUnknown === true) lines.push(`${indent}… further fields are admitted`);
- return lines;
- }
- case "oneOf": {
- const pinned = schema.variants.map((variant) => {
- if (variant.kind !== "object") return undefined;
- const literal = Object.entries(variant.fields)
- .find(([, field]) => field.schema.kind === "literal");
- return literal === undefined ? undefined : String((literal[1].schema as { value: unknown }).value);
- });
- if (pinned.every((value) => value !== undefined)) return [`${indent}one of ${pinned.join(", ")}`];
- return [`${indent}one of ${schema.variants.length} shapes`];
- }
- case "array": {
- const bound = schema.minItems === undefined ? "" : ` (at least ${schema.minItems})`;
- const inner = describeSchema(schema.items, step);
- return [`${indent}a list${bound} of:`, ...inner];
- }
- case "string": {
- if (schema.enum !== undefined) return [`${indent}one of ${schema.enum.join(", ")}`];
- const bound = schema.minLength === undefined ? "text" : `text, at least ${schema.minLength} characters`;
- return [`${indent}${bound}`];
- }
- case "number": {
- const parts = [schema.integer === true ? "a whole number" : "a number"];
- if (schema.minimum !== undefined) parts.push(`at least ${schema.minimum}`);
- if (schema.maximum !== undefined) parts.push(`at most ${schema.maximum}`);
- return [`${indent}${parts.join(", ")}`];
- }
- case "literal": return [`${indent}exactly ${JSON.stringify(schema.value)}`];
- case "boolean": return [`${indent}true or false`];
- case "null": return [`${indent}null`];
- default: return [`${indent}${(schema as { kind: string }).kind}`];
- }
-}
diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.ts
deleted file mode 100644
index 02af47ac..00000000
--- a/packages/reference-video-tools/src/tools.ts
+++ /dev/null
@@ -1,2194 +0,0 @@
-import { access, appendFile, mkdir, readdir, readFile, stat, writeFile } from "node:fs/promises";
-import { spawn } from "node:child_process";
-import { cpus } from "node:os";
-import { deflateSync } from "node:zlib";
-import { basename, dirname, join, resolve } from "node:path";
-import { loadNodePackageSelection, locateNodePackage, physicalPackageName } from "@hypit/package-loader-node";
-import { markupSurfaceHostFacetAbi } from "@hypit/markup";
-import type { RegisteredSurface, SurfaceVocabulary } from "@hypit/markup";
-import { exactModelHostAbi } from "@hypit/model-kit";
-import type { CanonicalValue, Need, ValueSchema } from "@hypit/protocol";
-import {
- visualPathCommandSchema, visualTextDocumentSchema, visualTextFlowSchema,
- visualTextPaintSchema, visualTextTypographySchema, visualTrackSchema,
- visualBoxSchema, visualMaskSchema, visualTextSchema, visualImageSchema,
- visualVideoSchema, visualSurfaceSchema, visualElementSchema,
- animatableLocalStyles,
-} from "@hypit/composition";
-import { VISUAL_STYLE_ENUM_VALUES_V1, VISUAL_STYLE_NAMES_V1 } from "@hypit/visual-ir";
-
-import { describeSchema } from "./schema.js";
-import { findRuntimeProfile, videoCliDistribution } from "@hypit/video-cli";
-import { MemoryResourceStore } from "@hypit/driver-node";
-import { geminiCapabilities, geminiModels, sealGeminiRequest } from "@hypit/gemini";
-import type { GeminiInlinePart, GeminiMediaPart, GeminiModel } from "@hypit/gemini";
-import { textTypes } from "@hypit/text";
-import { whisperXCapabilities } from "@hypit/whisperx";
-
-import { authorSource, invokedFrom, nearestPackageRoot, referenceRoot, referenceWords, renderElement, renderPreviews, spokenRange, standInSidecarPath, tokenWindow } from "./authoring.js";
-import type { RenderElementInput, RenderPreviewsInput, SpokenRange, StandInFocus, StandInSidecar } from "./authoring.js";
-import { downloadReferenceVideo, isReferenceUrl } from "@hypit/yt-dlp";
-import { loadStudioCompanionRegistry } from "@hypit/studio/src/companion-profile.js";
-import { openStudioBuildLibrary } from "@hypit/studio/src/build-library.js";
-import { loadStudioDomain } from "@hypit/studio/src/domain.js";
-import { loadStudioRun } from "@hypit/studio/src/run.js";
-import { authoringCheck, previewCheck, reviewLogPath, scriptCueCheck } from "./checks.js";
-import type { AuthoringCheckInput, PreviewCheckInput, ReconstructionCheckInput, ScriptCueCheckInput } from "./checks.js";
-import { layoutAccept, layoutCheck } from "./layout.js";
-import type { LayoutAcceptInput, LayoutCheckInput } from "./layout.js";
-import {
- assert,
- completelyStill,
- cutClip,
- cutFrame,
- ensureDir,
- prepareMedia,
- probe,
- readJson,
- readBytes,
- referenceId,
- round,
- shotTile,
- tileFrames,
- writeJson,
-} from "./media.js";
-import { prepareTranscript } from "./transcript.js";
-import type { InvokeNeed } from "./transcript.js";
-import type { Observation, ObservationTaskRequest, Observer, PrepareResult, ReferenceState, Shot, Transcript } from "./types.js";
-
-export type PrepareReferenceInput = {
- readonly video_path: string;
- readonly redo?: "media" | "transcript" | "people" | "voices" | "systems" | "places" | "all";
- readonly observer?: Observer;
- /** Runtime Profile whose Endpoints observe and transcribe; defaults to the project's selection. */
- readonly runtime?: string;
-};
-export type ObserveReferenceInput = {
- readonly reference_id: string;
- readonly shot_ids?: readonly string[];
- readonly question?: string;
- readonly reobserve?: boolean;
- /** Runtime Profile whose Gemini Endpoint answers; defaults to the project's selection. */
- readonly runtime?: string;
- /**
- * A round of narrow questions, asked together. Each entry names its own shots and its own question
- * and inherits `reference_id`; each `shot_ids` array must contain exactly one shot. No narrow
- * question's answer depends on another's, which is why the route asks them at once — and why asking
- * them one process at a time was a shell loop.
- */
- readonly questions?: readonly { readonly shot_ids: readonly string[]; readonly question: string }[];
-};
-export type RecordObservationInput = {
- readonly reference_id: string;
- /** Optional Run used to place comparison observations beside the project that requested them. */
- readonly run?: string;
- readonly key?: string;
- readonly text?: string;
- /**
- * A round of answers in one call.
- *
- * The `agent` observer answers everything by hand, and a reference of twenty shots owes an answer
- * for each of three questions about it, plus the boundaries, plus the four about the reference
- * itself. Handed out one call at a time that is seventy-odd invocations to close one sweep.
- */
- readonly answers?: readonly { readonly key: string; readonly text: string }[];
-};
-export type InspectVocabularyInput = {
- readonly package_names: readonly string[];
- readonly tags?: readonly string[];
- readonly include_previews?: boolean;
- readonly run?: string;
-};
-export type ValidateLocalAuthorPackagesInput = {
- readonly run: string;
- /** Runtime Profile used to resolve accepted Build Records in a pinned Run. */
- readonly runtime?: string;
- readonly expected_packages?: readonly string[];
-};
-export type CompareReconstructionInput = {
- readonly reference_id: string;
- /** Runtime Profile whose Gemini Endpoint compares; defaults to the project's selection. */
- readonly runtime?: string;
- /**
- * A stretch of the reference to compare against, named one of two ways. Exactly one is given.
- *
- * `shot_id` is a cut in the picture. `segment` and `selection` are word ranges, which is what a
- * render covers: a render is drawn over the words the Script marks, so the reference beside it is
- * the stretch that speaks those same words.
- */
- readonly shot_id?: string;
- readonly segment?: string;
- readonly selection?: string;
- /** A half-open token range, for a stretch the Script never named. See `StandInFocus`. */
- readonly tokens?: readonly [number, number];
- /** The Run whose Author SVML the words are read from. Required with a word range. */
- readonly run?: string;
- /** A rendered still. Exactly one of `image_path` and `video_path` is given. */
- readonly image_path?: string;
- /**
- * A rendered clip covering the shot. Comparing the whole stretch rather than one frame is what
- * removes the frame-choosing problem: an element that animates in, leaves and is replaced within
- * one shot has no single characteristic frame, and an observation asserting when it changed is the
- * least reliable evidence there is.
- */
- readonly video_path?: string;
- readonly question?: string;
- /**
- * Names the reconstructed element this image draws, for the comparison log only. It is never sent
- * to the observer: the comparison stays blind, and this is what lets a later report tell which
- * elements have been compared and which have never been looked at.
- */
- readonly element?: string;
- /** Maximum frame-domain difference tolerated when matching the two independently encoded cuts. */
- readonly tolerance_frames?: number;
- /**
- * A whole round of comparisons, run together. Each entry names its own stretch, render and element
- * exactly as a single call does, and inherits `reference_id` from the outer input. The route renders
- * every element before it compares any of them, so a round is a list by the time it starts.
- */
- readonly comparisons?: readonly Omit<CompareReconstructionInput, "reference_id" | "comparisons">[];
-};
-
-/**
- * One element of a description-authored program, read against what it was asked to be.
- *
- * The counterpart of `compare_reconstruction` on the route with no reference. There is nothing to
- * differ from, so the question is conformance: does the picture match the description, and what is
- * visibly wrong with it. That makes it one-sided, which is the only structural difference — the
- * render is prepared the same way, the log has the same shape, and the same report reads it.
- *
- * There is no observer choice here. The picture is a local render and nobody is billed to look at it,
- * so the task always comes back to be answered rather than being sent anywhere.
- */
-export type ReviewElementInput = {
- /** The Run whose Author SVML the words are read from, and the project the log belongs to. */
- readonly run: string;
- /**
- * Which element the findings are credited to. Required on a single review and asserted there — a
- * review credited to nothing is a picture nobody can say was looked at. Optional in the type only
- * because a `--batch` call carries it per entry rather than at the top.
- */
- readonly element?: string;
- /** The window the render covers, named in words. Exactly one is given. */
- readonly segment?: string;
- readonly selection?: string;
- /** A half-open token range, for a stretch the Script never named. See `StandInFocus`. */
- readonly tokens?: readonly [number, number];
- /** The render. Exactly one of `image_path` and `video_path` is given. */
- readonly image_path?: string;
- readonly video_path?: string;
- /**
- * What this element was asked to be, in the author's own words: the answers frozen when the brief
- * was taken and the appearance values written down before the Source was. Required, because a
- * conformance question with nothing to conform to is an invitation to invent a standard.
- */
- readonly intent?: string;
- readonly intent_file?: string;
- /** Which region to read, when the picture holds more than the element. Never what to conclude. */
- readonly question?: string;
- /** A whole round, run together. Each entry inherits `run` from the outer input. */
- readonly reviews?: readonly Omit<ReviewElementInput, "run" | "reviews">[];
-};
-
-export type RecordReviewInput = {
- readonly run: string;
- readonly review_id: string;
- readonly text: string;
-};
-
-export type { RenderElementInput, RenderPreviewsInput } from "./authoring.js";
-export type { PreviewCheckInput, ReconstructionCheckInput } from "./checks.js";
-
-export type ReferenceVideoTools = {
- list_svml_packages(): Promise<Record<string, unknown>>;
- prepare_reference(input: PrepareReferenceInput): Promise<PrepareResult>;
- observe_reference(input: ObserveReferenceInput): Promise<Record<string, unknown>>;
- inspect_svml_vocabulary(input: InspectVocabularyInput): Promise<Record<string, unknown>>;
- validate_local_author_packages(input: ValidateLocalAuthorPackagesInput): Promise<Record<string, unknown>>;
- validate_script_cues(input: ScriptCueCheckInput): Promise<Record<string, unknown>>;
- inspect_visual_schema(input: { readonly shape?: string; readonly producers?: readonly string[] }): Promise<Record<string, unknown>>;
- paths(): Promise<Record<string, unknown>>;
- compare_reconstruction(input: CompareReconstructionInput): Promise<Record<string, unknown>>;
- record_observation(input: RecordObservationInput): Promise<Record<string, unknown>>;
- review_element(input: ReviewElementInput): Promise<Record<string, unknown>>;
- record_review(input: RecordReviewInput): Promise<Record<string, unknown>>;
- render_element(input: RenderElementInput): Promise<Record<string, unknown>>;
- render_previews(input: RenderPreviewsInput): Promise<Record<string, unknown>>;
- preview_check(input: PreviewCheckInput): Promise<Record<string, unknown>>;
- layout_check(input: LayoutCheckInput): Promise<Record<string, unknown>>;
- layout_accept(input: LayoutAcceptInput): Promise<Record<string, unknown>>;
- reconstruction_check(input: ReconstructionCheckInput): Promise<Record<string, unknown>>;
- /** The corresponding direct report for a program authored from a description. */
- authoring_check(input: Omit<AuthoringCheckInput, "mode" | "reference_id">): Promise<Record<string, unknown>>;
-};
-
-type ToolOptions = {
- readonly packageRoot?: string;
- readonly model?: string;
- readonly concurrency?: number;
- readonly launchGapMs?: number;
- readonly retryDelayMs?: number;
- readonly generate?: GenerateText;
-};
-export type GenerateText = (input: { readonly parts: readonly GeminiInlinePart[]; readonly instruction: string }) => Promise<string>;
-type ObservationTask = { readonly key: string; readonly request: Request };
-
-let eventSequence = 0;
-
-function eventId(label: string, at = new Date()): string {
- eventSequence += 1;
- const readable = label.trim().replace(/[^0-9A-Za-z._-]+/gu, "-").replace(/^-+|-+$/gu, "") || "item";
- const time = at.toISOString().replace(/[-:.TZ]/gu, "");
- return `${readable}-${time}-${eventSequence.toString(36)}`;
-}
-
-function observation(status: Observation["status"], text: string): Observation { return { status, text }; }
-function unavailable(reason: string): Transcript { return { status: "unavailable", transcript_ref: null, word_count: 0, reason }; }
-
-function positiveEnv(name: string): number | undefined {
- const raw = process.env[name]?.trim();
- if (raw === undefined || raw.length === 0) return undefined;
- const value = Number(raw);
- assert(Number.isSafeInteger(value) && value > 0, `${name} must be a positive integer`);
- return value;
-}
-
-/**
- * Where one reference's prepared state lives.
- *
- * `referenceRoot` finds the Hypit tree from this module's own location, so the directory
- * `prepare_reference` writes to is the directory every other command reads from, whichever directory
- * the command was run in.
- */
-function stateRoot(reference: string): string {
- return join(referenceRoot(), reference);
-}
-
-/**
- * The stretch a word-range comparison covered: the words it was asked for, and the seconds it cut.
- *
- * The two differ by however far each end was moved onto a shot boundary lying inside its own end
- * word. `head_snapped` and `tail_snapped` say whether that end landed on a cut, which is what
- * decides whether the pair opens and closes on a whole shot or part-way through one.
- */
-export type ComparedRange = {
- readonly segment?: string;
- readonly selection?: string;
- /** The word range, when the caller gave one directly rather than naming a marked stretch. */
- readonly tokens?: readonly [number, number];
- readonly words: string;
- readonly words_start_seconds: number;
- readonly words_end_seconds: number;
- readonly cut_start_seconds: number;
- readonly cut_end_seconds: number;
- readonly head_snapped: boolean;
- readonly tail_snapped: boolean;
-};
-
-/** One line per comparison performed, appended so a stopped round still leaves its trail. */
-export type ComparisonRecord = {
- readonly at: string;
- /**
- * Names this comparison so an observer that answers out of band can close it. The `agent` observer
- * is handed the pair and the question and returns nothing in band, so without a name the entry
- * stayed `pending` for ever and a report had to choose between crediting an unanswered comparison and
- * crediting none of them.
- */
- readonly id: string;
- /** The project Run, when this comparison was made over a word range. */
- readonly run?: string;
- /** Which stretch was compared. A comparison names its shot or its word range, never both. */
- readonly shot_id?: string;
- readonly range?: ComparedRange;
- readonly element?: string;
- readonly image_path: string;
- readonly observer: Observer;
- readonly status: Observation["status"];
- readonly scoped: boolean;
- /** What the observer was told to limit itself to, when anything was. */
- readonly scope?: string;
- /** Whether the whole shot was compared as a clip, or one frame of it as a still. */
- readonly clip?: boolean;
- /**
- * What timed the stand-in the picture was drawn over, from the sidecar `render_element` writes
- * beside its output. Present when the picture came from `render_element`.
- */
- readonly stand_in?: StandInSidecar;
- /** The differences, once an out-of-band observer has recorded them. */
- readonly differences?: string;
-};
-
-/**
- * One line per element review on the route that has no reference to compare against.
- *
- * The fields it shares with `ComparisonRecord` are spelled the same, so reports read one shape from
- * either log without an adapter. What differs is the question: a comparison reports how two pictures
- * differ, and a review reports whether one picture is what it was asked to be — so it carries the
- * intent it was judged against, and its answer is `findings`.
- */
-export type ReviewRecord = {
- readonly at: string;
- readonly id: string;
- /** Always present. `review_element` requires `--element`, so a review credits one by construction. */
- readonly element: string;
- /**
- * The window, named the way it was asked for. Only the name: `ComparedRange` also carries where the
- * reference speaks those words and how the cut was snapped to its shot boundaries, and none of that
- * exists here. Reports read the name and nothing else from either log.
- */
- readonly range?: { readonly segment?: string; readonly selection?: string; readonly tokens?: readonly [number, number] };
- readonly image_path: string;
- readonly status: Observation["status"];
- readonly scope?: string;
- readonly clip?: boolean;
- readonly stand_in?: StandInSidecar;
- /** What this element was asked to be, in the author's own words. */
- readonly intent: string;
- /** What the reader found, once it has come back. */
- readonly findings?: string;
-};
-
-/**
- * What timed the stand-in behind a rendered picture, read from beside the picture itself.
- *
- * A comparison is handed a path. Whether the stretch in it ran at the reference's pace or at an
- * estimate of it is not in the pixels, and a picture from anywhere else has no sidecar at all, so a
- * missing one is an absence to record rather than a refusal.
- */
-async function standInBeside(renderedPath: string): Promise<StandInSidecar | undefined> {
- const text = await readFile(standInSidecarPath(renderedPath), "utf8").catch(() => undefined);
- if (text === undefined) return undefined;
- try { return JSON.parse(text) as StandInSidecar; } catch { return undefined; }
-}
-
-/**
- * One line per look performed, appended so a stopped round still leaves its trail.
- *
- * Two logs have this shape and the same three operations over it. A comparison of a reconstruction
- * against its reference is kept under the reference, and its answer is the differences; a review of a
- * description-authored element against what it was asked to be is kept under the project, and its
- * answer is the findings. Nothing else about appending, closing or recognising an answered entry
- * differs, so the file and the name of the answer are arguments rather than a second copy.
- */
-type LookLog = {
- readonly root: string;
- readonly file: string;
- /** Which field carries the answer, and therefore what tells an open entry from a closed one. */
- readonly answer: "differences" | "findings";
-};
-
-const comparisonLog = (root: string): LookLog => ({ root, file: "comparisons.jsonl", answer: "differences" });
-
-/** Comparisons are evidence for one reconstruction, not for the shared reference. */
-function comparisonLogForRun(run: string | undefined, referenceRootPath: string): LookLog {
- return run === undefined
- ? comparisonLog(referenceRootPath)
- : comparisonLog(join(dirname(resolve(invokedFrom(), run)), ".hypit"));
-}
-const reviewLog = (root: string): LookLog => ({ root, file: "reviews.jsonl", answer: "findings" });
-
-async function appendLooked(log: LookLog, record: ComparisonRecord | ReviewRecord): Promise<void> {
- await serially(log.root, async () => {
- await mkdir(log.root, { recursive: true });
- await appendFile(join(log.root, log.file), `${JSON.stringify(record)}\n`, "utf8");
- });
-}
-
-/**
- * Record the differences against the comparison that asked for them, and mark it answered.
- *
- * The log is append-only while a round runs, so closing an entry rewrites the file. That is a
- * read-modify-write over the same bytes `appendLooked` appends to, so both go through the same
- * queue — a round fired at once would otherwise drop whichever line landed between the read and the
- * write. Returns false when no entry carries the id, which is what a mistyped one looks like.
- */
-async function closeLooked(log: LookLog, id: string, text: string): Promise<boolean> {
- return await serially(log.root, async () => {
- const path = join(log.root, log.file);
- const lines = (await readFile(path, "utf8").catch(() => "")).split("\n").filter((line) => line.trim().length > 0);
- let found = false;
- const rewritten = lines.map((line) => {
- let record: ComparisonRecord;
- try { record = JSON.parse(line) as ComparisonRecord; } catch { return line; }
- if (record.id !== id) return line;
- found = true;
- return JSON.stringify({ ...record, status: "complete", [log.answer]: text });
- });
- if (found) await writeFile(path, `${rewritten.join("\n")}\n`, "utf8");
- return found;
- });
-}
-
-/**
- * One writer at a time per reference, within this process.
- *
- * `observations.json`, `state.json` and `comparisons.jsonl` are each read, changed and written back.
- * The `agent` observer answers a round of observations with one call per answer, and those calls
- * arrive together, so two of them reading the same cache before either writes is how an answer
- * disappears with no error anywhere. Between processes this holds nothing; the route runs one.
- */
-const writeQueues = new Map<string, Promise<unknown>>();
-async function serially<T>(root: string, work: () => Promise<T>): Promise<T> {
- const previous = writeQueues.get(root) ?? Promise.resolve();
- const next = previous.then(work, work);
- writeQueues.set(root, next.catch(() => undefined));
- return await next;
-}
-
-
-
-function positiveInt(value: number, label: string): number {
- assert(Number.isSafeInteger(value) && value > 0, `${label} must be a positive integer.`);
- return value;
-}
-
-type RuntimeHost = Awaited<ReturnType<typeof videoCliDistribution.openRuntimeHost>>;
-
-/**
- * The Runtime Profile a creation-time call executes through: the one the caller named, otherwise the
- * project's own selection, read the way `hypit` reads it. Observation and transcription never pick a
- * Provider, read a credential store or contact a service themselves.
- */
-async function runtimeProfileFor(runtime: string | undefined): Promise<string> {
- if (runtime !== undefined) return resolve(invokedFrom(), runtime);
- const projectRoot = nearestPackageRoot(invokedFrom()) ?? invokedFrom();
- const selected = await findRuntimeProfile(projectRoot);
- assert(selected !== undefined,
- `No Runtime Profile is selected for ${projectRoot}; run hypit runtime use <profile> there, or pass --runtime <profile>`);
- return selected.profile;
-}
-
-async function runtimeHostFor(runtime: string | undefined): Promise<RuntimeHost> {
- const profile = await runtimeProfileFor(runtime);
- const projectRoot = nearestPackageRoot(dirname(profile)) ?? dirname(profile);
- return await videoCliDistribution.openRuntimeHost(profile, {
- packageRoot: projectRoot,
- ...(videoCliDistribution.packageRoot === undefined ? {} : { distributionPackageRoot: videoCliDistribution.packageRoot }),
- });
-}
-
-function geminiModel(model: string): GeminiModel {
- assert((geminiModels as readonly string[]).includes(model),
- `Gemini model ${model} is not declared by @hypit/gemini; declared models: ${geminiModels.join(", ")}`);
- return model as GeminiModel;
-}
-
-/**
- * Observation through the selected Runtime Profile: the same Need, Endpoint, credential and Provider
- * resolution a Build uses, without a Build. The inline parts this file assembles become one Gemini
- * request whose media live in a per-call resource store; nothing reaches Build state or a Result.
- */
-function runtimeGenerate(host: RuntimeHost, model: GeminiModel): GenerateText {
- return async ({ parts, instruction }) => {
- const resources = new MemoryResourceStore();
- const prompt: string[] = [];
- const media: GeminiMediaPart[] = [];
- for (const part of parts) {
- if (part.text !== undefined) { prompt.push(part.text); continue; }
- media.push({ artifact: await resources.put(Buffer.from(part.inlineData.data, "base64"), part.inlineData.mimeType) });
- }
- const need: Need = {
- id: "need:reference-video-observation",
- capability: geminiCapabilities[model],
- returns: textTypes.text,
- constraints: sealGeminiRequest({ instruction, prompt: prompt.join("\n\n"), media }) as unknown as CanonicalValue,
- result: "record:reference-video-observation",
- };
- const fulfillment = await host.invoke(need, resources);
- assert(fulfillment.value.kind === "inline", "Gemini returned its observation by reference");
- const text = (fulfillment.value.value as { readonly value?: unknown } | null)?.value;
- assert(typeof text === "string", "Gemini observation is not Text");
- return text;
- };
-}
-
-/**
- * How a transcript is measured: the `whisperx-alignment` Endpoint of the selected Runtime Profile.
- * Preparation reports a missing selection or Endpoint beside everything else it prepared instead of
- * refusing; the transcript stays recoverable with `--redo transcript` once the Profile serves it.
- */
-async function alignmentThrough(runtime: string | undefined): Promise<{ readonly invoke: InvokeNeed; readonly note?: string }> {
- const refusing = (note: string) => ({ invoke: (async () => { throw new Error(note); }) as InvokeNeed, note });
- try {
- const host = await runtimeHostFor(runtime);
- const [provider] = await host.providers([whisperXCapabilities.alignment]);
- if (provider?.status === "resolved") return { invoke: async (need, resources) => await host.invoke(need, resources) };
- return refusing(provider?.status === "ambiguous"
- ? `several Endpoints serve @hypit/whisperx#whisperx-alignment: ${(provider.endpoints ?? []).join(", ")}; keep exactly one in the Runtime Profile`
- : "no Endpoint in the selected Runtime Profile serves @hypit/whisperx#whisperx-alignment; hypit plan --runtime <profile> shows what each capability needs");
- } catch (error) {
- return refusing(error instanceof Error ? error.message : String(error));
- }
-}
-
-const MIME_TYPES: Readonly<Record<string, string>> = {
- jpg: "image/jpeg",
- jpeg: "image/jpeg",
- png: "image/png",
- webp: "image/webp",
- wav: "audio/wav",
- mp4: "video/mp4",
-};
-
-// The Vertex path cannot upload a file and reference it later, so the bytes travel with every
-// request. One shot's clip is asked about by its own picture, type and sound observations and by the
-// boundary on each side, so reading and encoding it once per invocation is worth the memory.
-function mediaParts(): (path: string) => Promise<GeminiInlinePart> {
- const encoded = new Map<string, Promise<GeminiInlinePart>>();
- return (path) => {
- const held = encoded.get(path);
- if (held !== undefined) return held;
- const part = (async (): Promise<GeminiInlinePart> => {
- const bytes = await readBytes(path);
- const extension = path.toLowerCase().split(".").pop() ?? "";
- const mimeType = MIME_TYPES[extension];
- assert(mimeType !== undefined, `unsupported media extension for ${path}; supported: ${Object.keys(MIME_TYPES).join(", ")}`);
- return { inlineData: { mimeType, data: Buffer.from(bytes).toString("base64") } };
- })();
- encoded.set(path, part);
- return part;
- };
-}
-
-function message(error: unknown): string {
- return error instanceof Error ? error.message : String(error);
-}
-
-function rateLimited(error: unknown): boolean {
- return /\b429\b|resource[ _]exhausted|quota|rate limit/iu.test(message(error));
-}
-
-// A rejected request is rejected every time. Retrying one only delays the failure the caller has to
-// see, so these end the attempt loop immediately; everything else stays retryable.
-function permanent(error: unknown): boolean {
- return /invalid[_ ]argument|permission[_ ]denied|unauthenticated|not[_ ]found|failed[_ ]precondition/iu.test(message(error));
-}
-
-async function callSafely(retryDelayMs: number, generate: GenerateText, parts: readonly GeminiInlinePart[], instruction: string): Promise<Observation> {
- let last: unknown;
- for (let attempt = 1; attempt <= 6; attempt += 1) {
- try { return observation("complete", await generate({ parts, instruction })); }
- catch (error) {
- last = error;
- if (permanent(error)) break;
- if (attempt === 6) break;
- const wait = rateLimited(error) ? Math.min(900_000, 45_000 * 2 ** (attempt - 1)) : retryDelayMs * attempt;
- // The retry is visible, not hidden: each failed attempt and the wait before the next one is reported.
- process.stderr.write(`observation attempt ${attempt}/6 failed: ${message(error)}; retrying in ${Math.round(wait / 1000)}s\n`);
- await new Promise((resolveWait) => setTimeout(resolveWait, wait));
- }
- }
- return observation("failed", last instanceof Error ? last.message : String(last));
-}
-
-// The agent observer reads pictures, so each piece of evidence a request would have uploaded becomes
-// one it can look at: a shot's clip becomes that shot's frame tile, the whole video becomes the
-// storyboard of representative frames, and audio has no picture to become. What each observation asks
-// is identical on both paths; only the shape of the evidence differs.
-function asPictures(media: readonly string[], state: ReferenceState): readonly string[] {
- const tiles = new Map(state.shots.map((shot) => [shot.clip_ref, shot.frames_tile_ref]));
- const pictures: string[] = [];
- for (const path of media) {
- const tile = tiles.get(path);
- if (tile !== undefined) {
- // Every shot of a reference prepared for this observer has a tile. One without means the media
- // was prepared for the observer that reads video, which `prepare_reference` refuses to mix.
- assert(tile !== null, `shot media for ${path} has no frame tile; re-prepare the reference with --redo all --observer agent`);
- pictures.push(tile);
- }
- // A whole-reference question is asked over the whole reference. The storyboard puts every shot in
- // one picture and is how the video is read at a glance, but one frame per shot answers neither
- // what moves nor what recurs, so the shot tiles come with it and the question sees every frame
- // the shot observations see.
- else if (path === state.analysis_video_ref) pictures.push(state.storyboard_ref, ...state.shots.flatMap((shot) => shot.frames_tile_ref === null ? [] : [shot.frames_tile_ref]));
- // This observer reads pictures. Video that is not a prepared shot has no tile to stand in for it,
- // and handing the file through would give the agent something it cannot open — a comparison clip
- // is tiled by its caller before it arrives here.
- else if (/\.(mp4|mov|webm|mkv)$/iu.test(path)) assert(false, `${path} is video; tile it before asking the agent observer to read it`);
- // Sound has no picture to stand in for it, so it is left out rather than handed over as a file the
- // agent cannot read. Everything else has to be a picture this observer can actually open: the
- // observer that uploads refuses an unknown extension outright, and passing one through here handed
- // the agent a path and called it evidence.
- else if (!path.toLowerCase().endsWith(".wav")) {
- assert(/\.(jpg|jpeg|png|webp)$/iu.test(path),
- `${path} is not a picture the agent observer can read; supported: jpg, jpeg, png, webp`);
- pictures.push(path);
- }
- }
- return [...new Set(pictures)];
-}
-
-// What the pictures are has to be said, because a tile of frames is not what the prompt was written
-// for. The prompt itself is unchanged on both paths: only this preamble is added, and only here.
-const TILE_PREAMBLE = "Each supplied picture that shows a grid of frames is one shot, sampled evenly"
- + " across its duration and laid out in reading order: left to right, then top to bottom. Read the"
- + " grid as time passing. A single picture that is not a grid is one moment.";
-const WATERMARK_RULE = "If a visible mark is clearly a platform UI, player chrome, export-tool or editing-software watermark/overlay rather than authored video content, ignore it and do not report it as a difference or repair target. If its provenance is uncertain or it may be an authored design element, keep it as an uncertainty instead of silently ignoring it.";
-// There is no second source of sound to fall back to, so this says how to answer from what there is
-// rather than leaving the observation short. The transcript settles when speech happens; the picture
-// settles who is on screen while it does. Read together they carry the question far enough to answer.
-// The two fields are named rather than described: an instruction to use the transcript, given to a
-// reader with no way to find it, is answered without one.
-const NO_SOUND = "You are reading pictures and cannot hear this reference. Answer from the pictures and"
- + " the measured transcript together: the transcript says exactly when words are spoken, and the"
- + " pictures say who is on screen and whose mouth is moving while they are. The words this task's own"
- + " stretch contains are supplied as `transcript_words`, and the whole measured transcript is the file"
- + " at `transcript_ref`. Attribute speech to the person the pictures show speaking during the words the"
- + " transcript places there, and answer the question in full rather than deferring the parts that would"
- + " be easier with sound.";
-
-/** Declares one observation: what it asks, and the evidence it asks over. The observer decides how. */
-type Request = {
- readonly media: readonly string[];
- readonly prompt: string;
- readonly instruction: string;
- /** True when answering needs sound. The `agent` observer has none and is told to say so. */
- readonly sound?: boolean;
- /**
- * The shots this question is asked over, when it is asked over shots rather than the whole video.
- *
- * Only a sound question reads it, and only on the observer that cannot hear: it is what narrows the
- * measured transcript to the words this task's own stretch contains. A question over the whole
- * reference leaves it unset and is given the whole transcript instead.
- */
- readonly shots?: readonly Shot[];
-};
-type Asker = (key: string, request: Request) => Promise<Observation>;
-
-/**
- * Attempt every entry of a `--batch` round, and keep one entry's failure to itself.
- *
- * Four commands take a batch and all four want the same thing from it: every entry attempted, the
- * ones that threw reported beside the input that produced them, and the rest returned whole. Written
- * out per command that was the same twenty lines four times, differing only in the noun.
- *
- * What stays at the call site is the result's own shape — `compared`/`comparisons`,
- * `reviewed`/`reviews` — because that is the part a reader of the output is looking for, and a helper
- * that also invented those names would need an argument per noun to say nothing extra.
- */
-/** A round that reaches no Provider: bounded by the machine's cores rather than by anyone's quota. */
-const locallyPaced = (items: number) => ({ concurrency: Math.max(1, Math.min(cpus().length - 1, items)), gapMs: 0 });
-
-async function runBatch<T>(
- items: readonly T[],
- rate: { readonly concurrency: number; readonly gapMs: number },
- run: (item: T) => Promise<Record<string, unknown>>,
-): Promise<{
- readonly done: readonly Record<string, unknown>[];
- readonly failures: readonly { readonly error: string; readonly input: T }[];
-}> {
- const settled = await pacedMap(items, rate.concurrency, rate.gapMs, async (item) => {
- try { return { ok: true as const, value: await run(item) }; }
- catch (error) { return { ok: false as const, error: error instanceof Error ? error.message : String(error), input: item }; }
- });
- return {
- done: settled.flatMap((item) => item.ok ? [item.value] : []),
- failures: settled.flatMap((item) => item.ok ? [] : [{ error: item.error, input: item.input }]),
- };
-}
-
-async function pacedMap<T, R>(items: readonly T[], concurrency: number, gapMs: number, run: (item: T, index: number) => Promise<R>): Promise<readonly R[]> {
- const result = new Array<R>(items.length);
- let cursor = 0;
- let previousLaunch = Promise.resolve();
- let nextLaunchAt = 0;
- const launch = (): Promise<void> => {
- const current = previousLaunch.then(async () => {
- const wait = nextLaunchAt - Date.now();
- if (wait > 0) await new Promise((resolveWait) => setTimeout(resolveWait, wait));
- nextLaunchAt = Date.now() + gapMs;
- });
- previousLaunch = current;
- return current;
- };
- const worker = async (): Promise<void> => {
- while (true) {
- const index = cursor++;
- if (index >= items.length) return;
- await launch();
- result[index] = await run(items[index]!, index);
- }
- };
- await Promise.all(Array.from({ length: Math.max(1, Math.min(concurrency, items.length || 1)) }, worker));
- return result;
-}
-
-async function runObservationTasks(
- root: string,
- tasks: readonly ObservationTask[],
- forcedKeys: ReadonlySet<string>,
- concurrency: number,
- gapMs: number,
- ask: Asker,
-): Promise<ReadonlyMap<string, Observation>> {
- const path = join(root, "observations.json");
- const cache = await readJson<Record<string, Observation>>(path) ?? {};
- const outstanding = tasks.filter((task) => forcedKeys.has(task.key) || cache[task.key]?.status !== "complete");
- const completed = await pacedMap(outstanding, concurrency, gapMs, async (task) => ({ key: task.key, value: await ask(task.key, task.request) }));
- const answers = new Map(completed.map((item) => [item.key, item.value]));
- // Only an answer is cached. A pending one is a task handed out, and a failed one is the error that
- // ended the attempt; either, written, is read back by whatever quotes an observation's text.
- // Both still reach the caller through `answers`, so `unresolved` reports them.
- await serially(root, async () => {
- // Re-read inside the queue: `record_observation` may have written an answer to a different key
- // between the read above and here, and the copy taken then no longer holds it.
- const current = await readJson<Record<string, Observation>>(path) ?? {};
- for (const item of completed) if (item.value.status === "complete") current[item.key] = item.value;
- for (const [key, value] of Object.entries(current)) cache[key] = value;
- await writeJson(path, current);
- });
- return new Map(tasks.map((task) => [task.key, cache[task.key] ?? answers.get(task.key) ?? observation("failed", "not observed")]));
-}
-
-async function shotFromBound(root: string, index: number, bound: { start: number; end: number; group: number; part: number; parts: number }, tiles: boolean): Promise<Shot> {
- const id = String(index + 1).padStart(3, "0");
- const dir = join(root, "shots");
- const audioPath = join(dir, `${id}-audio.wav`);
- let audio_tail_ref: string | null = null;
- try { await access(audioPath); audio_tail_ref = audioPath; } catch { /* silent shot */ }
- return {
- shot_id: `shot-${id}`,
- index: index + 1,
- start_seconds: bound.start,
- end_seconds: bound.end,
- duration_seconds: Number((bound.end - bound.start).toFixed(3)),
- initial_group: bound.group,
- part: bound.part,
- parts: bound.parts,
- clip_ref: join(dir, `${id}.mp4`),
- representative_frame_ref: join(dir, `${id}-representative.jpg`),
- tail_frame_ref: join(dir, `${id}-tail.jpg`),
- frames_tile_ref: tiles ? join(dir, `${id}-frames.jpg`) : null,
- audio_tail_ref,
- };
-}
-
-/** One stretch of the reference cut to a word range, with the rendered clip trimmed to match it. */
-type RangeCut = {
- readonly record: ComparedRange;
- readonly reference: string;
- readonly rendered: string;
- readonly referenceTile: string;
- readonly renderedTile: string;
- /** Where each side's single frame goes, for a stretch that turns out to hold one picture. */
- readonly referenceStill: string;
- readonly renderedStill: string;
- /** How long the cut runs, which is the duration both frame grids are sampled against. */
- readonly seconds: number;
- readonly referenceSeconds: number;
- readonly renderedSeconds: number;
- /** What the observer is told about an end that lands part-way through a shot. Empty when neither does. */
- readonly incomplete: string;
-};
-
-/**
- * Every cut in the reference, in seconds.
- *
- * A stretch longer than fifteen seconds is divided into parts so each clip stays short enough to
- * observe, and the divisions between those parts are not cuts: the picture runs straight through
- * them. Only the first part of a group begins where the picture changes.
- */
-function referenceCuts(state: ReferenceState): readonly number[] {
- return [...state.shots.filter((shot) => shot.part === 1).map((shot) => shot.start_seconds), state.shots.at(-1)!.end_seconds];
-}
-
-/**
- * Cut the reference to the stretch that speaks a range of words, and trim the render to match.
- *
- * Each end is moved onto a shot boundary when one lies inside its own end word. A boundary inside
- * the first word cannot add or drop a whole word — the word is still spoken across the cut either
- * way — so the pair then opens and closes on the picture changing rather than part-way through a
- * shot, and still covers exactly the words asked for. An end with no boundary inside its word stays
- * where the word puts it.
- *
- * The render is trimmed by the seconds each end moved. Its frames come from the same word timings,
- * so a second of reference at the head is a second of render at the head, and taking it off both
- * keeps the two stretches showing the same word at the same offset.
- */
-async function cutWordRange(
- state: ReferenceState,
- root: string,
- focus: StandInFocus,
- range: SpokenRange,
- renderedPath: string,
- clip: boolean,
- slot: string,
-): Promise<RangeCut> {
- const cuts = referenceCuts(state);
- const head = cuts.find((at) => at >= range.first.startSeconds && at < range.first.endSeconds);
- const tail = [...cuts].reverse().find((at) => at > range.last.startSeconds && at <= range.last.endSeconds);
- const start = head ?? range.startSeconds;
- const end = tail ?? range.endSeconds;
- const seconds = end - start;
- assert(seconds > 0, `${focus.selection ?? focus.segment} spans no time in the reference`);
-
- const label = focus.selection === undefined ? `segment-${focus.segment}` : `selection-${focus.selection}`;
- // One directory per call. Several elements are drawn over one Segment, so several comparisons name
- // the same word range, and the route runs them at once once every render is finished. Named from the
- // range alone they would write the same six files, and each would be reading the cut another was
- // still writing.
- const dir = join(root, "comparisons", label, slot);
- await ensureDir(dir);
- const reference = join(dir, `reference.${clip ? "mp4" : "jpg"}`);
- // The reference's own analysis video, which is the whole reference at the size every other
- // observation reads it at. A shot clip covers a cut in the picture and would cover the wrong words.
- if (clip) await cutClip(state.analysis_video_ref, start, seconds, reference);
- else await cutFrame(state.analysis_video_ref, start + seconds / 2, reference);
-
- let rendered = renderedPath;
- let referenceSeconds = 0;
- let renderedSeconds = 0;
- if (clip) {
- const drawn = await probe(renderedPath);
- assert(drawn.frameRate > 0, `${renderedPath} reports no frame rate, so it cannot be trimmed to the cut`);
- // Frames are the only unit the file has, so the seconds each end moved are converted at the
- // render's own rate and taken off as whole frames.
- const headFrames = Math.round((start - range.startSeconds) * drawn.frameRate);
- const tailFrames = Math.round((range.endSeconds - end) * drawn.frameRate);
- rendered = join(dir, "rendered.mp4");
- await cutClip(renderedPath, headFrames / drawn.frameRate, drawn.duration - (headFrames + tailFrames) / drawn.frameRate, rendered);
- referenceSeconds = (await probe(reference)).duration;
- renderedSeconds = (await probe(rendered)).duration;
- }
-
- // An end that could not be moved onto a cut opens or closes part-way through a shot. How much of
- // the stretch that is, is measurable here, and saying it is what keeps the incomplete shot from
- // being read as a difference.
- const enclosing = (at: number): { readonly start: number; readonly end: number } => {
- let index = 0;
- for (let candidate = 0; candidate + 1 < cuts.length; candidate += 1) if (cuts[candidate]! <= at) index = candidate;
- return { start: cuts[index]!, end: cuts[index + 1]! };
- };
- const opening = enclosing(start);
- const closing = enclosing(end);
- const lead = head === undefined ? Math.min(opening.end, end) - start : 0;
- const trail = tail === undefined ? end - Math.max(closing.start, start) : 0;
- const ends = !clip
- ? []
- // A range short enough to sit inside one shot has both its ends in that shot, and naming them
- // separately describes the same seconds twice.
- : head === undefined && tail === undefined && opening.start === closing.start
- ? [`All ${seconds.toFixed(2)} seconds fall inside a single shot that begins before this stretch does and ends after it.`]
- : [
- ...(lead > 0 ? [`The first ${lead.toFixed(2)} seconds fall inside a shot that begins before this stretch does.`] : []),
- ...(trail > 0 ? [`The last ${trail.toFixed(2)} seconds fall inside a shot that ends after this stretch does.`] : []),
- ];
-
- return {
- record: {
- ...(focus.segment === undefined ? {} : { segment: focus.segment }),
- ...(focus.selection === undefined ? {} : { selection: focus.selection }),
- ...(focus.tokens === undefined ? {} : { tokens: focus.tokens }),
- words: `${range.first.text} … ${range.last.text}`,
- words_start_seconds: round(range.startSeconds),
- words_end_seconds: round(range.endSeconds),
- cut_start_seconds: round(start),
- cut_end_seconds: round(end),
- head_snapped: head !== undefined,
- tail_snapped: tail !== undefined,
- },
- reference,
- rendered,
- referenceTile: join(dir, "reference-tile.jpg"),
- renderedTile: join(dir, "rendered-tile.jpg"),
- referenceStill: join(dir, "reference-still.jpg"),
- renderedStill: join(dir, "rendered-still.jpg"),
- seconds,
- referenceSeconds: round(referenceSeconds),
- renderedSeconds: round(renderedSeconds),
- incomplete: ends.length === 0
- ? ""
- : `\n\n${ends.join(" ")} Read ${ends.length === 1 ? "that stretch as an incomplete shot" : "those stretches as incomplete shots"} and report no differences from ${ends.length === 1 ? "it" : "them"}.`,
- };
-}
-
-function prepared(state: ReferenceState): boolean {
- return state.transcript?.status === "complete"
- && state.people_and_product?.status === "complete"
- && state.voices?.status === "complete"
- && state.persistent_systems?.status === "complete"
- && state.places?.status === "complete";
-}
-
-function publicPrepare(state: ReferenceState): PrepareResult {
- const observations = {
- people_and_product: state.people_and_product ?? observation("failed", "not analyzed"),
- voices: state.voices ?? observation("failed", "not analyzed"),
- persistent_systems: state.persistent_systems ?? observation("failed", "not analyzed"),
- places: state.places ?? observation("failed", "not analyzed"),
- } as const;
- return {
- reference_id: state.reference_id,
- status: prepared(state) ? "ready" : "partial",
- video: state.video,
- shots: state.shots,
- storyboard_ref: state.storyboard_ref,
- transcript: state.transcript ?? unavailable("not transcribed"),
- ...observations,
- observations,
- };
-}
-
-export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceVideoTools {
- const packageRoot = resolve(options.packageRoot ?? process.cwd());
- // Project-owned packages and the active Distribution are separate roots. A project may import
- // @hypit modules without installing or linking them locally; every direct inspection path must
- // therefore carry the same Distribution fallback used by the compiler/Studio loaders.
- const packageLoading = () => videoCliDistribution.packageRoot === undefined
- ? {}
- : { fallbackRoots: [videoCliDistribution.packageRoot] };
-
- async function validateLocalAuthorPackages(input: ValidateLocalAuthorPackagesInput): Promise<Record<string, unknown>> {
- const runPath = resolve(invokedFrom(), input.run);
- const projectRoot = nearestPackageRoot(dirname(runPath)) ?? packageRoot;
- const packagesRoot = join(projectRoot, "packages");
- // Project package directories are discovered by their location and manifest, not by a
- // naming convention. A package created for one video may use any valid descriptive slug;
- // requiring a `local-` prefix would let other project-owned packages evade this validation.
- const projectPackageDirectories = (await readdir(packagesRoot, { withFileTypes: true }).catch(() => []))
- .filter((entry) => entry.isDirectory() && entry.name !== "node_modules" && !entry.name.startsWith("."))
- .map((entry) => join(packagesRoot, entry.name));
- const expected = new Set(input.expected_packages ?? []);
- const distributionPackageRoot = videoCliDistribution.packageRoot;
- const loading = distributionPackageRoot === undefined ? {} : { fallbackRoots: [distributionPackageRoot] };
- const runSource = await readFile(runPath, "utf8").catch(() => "");
- const authorSourcePath = /<author\s+source="([^"]+)"/u.exec(runSource)?.[1];
- const authorFile = authorSourcePath === undefined ? undefined : resolve(dirname(runPath), authorSourcePath);
- const imported = new Set<string>();
- const visitedSources = new Set<string>();
- const collectSourceClosure = async (file: string): Promise<void> => {
- const absolute = resolve(file);
- if (visitedSources.has(absolute)) return;
- visitedSources.add(absolute);
- const source = await readFile(absolute, "utf8").catch(() => "");
- for (const match of source.matchAll(/\bfrom="([^"]+)@\d+"/gu)) {
- const specifier = match[1]!;
- if (specifier.startsWith("@")) imported.add(specifier);
- }
- // Author/Recipe/Run child Sources use `source=`, while package imports use `from=`. Walk only
- // local files so package declarations remain data and cannot escape the project root.
- for (const match of source.matchAll(/\bsource="([^"#]+)"/gu)) {
- const child = match[1]!;
- if (child.startsWith("@") || child.startsWith("http://") || child.startsWith("https://")) continue;
- await collectSourceClosure(resolve(dirname(absolute), child));
- }
- };
- await collectSourceClosure(runPath);
- if (authorFile !== undefined) await collectSourceClosure(authorFile);
- let graphModules = new Set<string>();
- try {
- if (distributionPackageRoot !== undefined) {
- const registry = await loadStudioCompanionRegistry({ workspaceRoot: projectRoot, packageRoot: projectRoot, distributionPackageRoot });
- const domain = await loadStudioDomain({ run: runPath, workspaceRoot: projectRoot, packageRoot: projectRoot });
- const runtimePath = input.runtime === undefined ? undefined : resolve(invokedFrom(), input.runtime);
- const buildLibrary = await openStudioBuildLibrary(runtimePath, projectRoot, projectRoot, distributionPackageRoot);
- try {
- const loaded = await loadStudioRun({ run: runPath, domain, registry,
- ...(buildLibrary === undefined ? {} : { buildLibrary }) });
- graphModules = new Set([
- ...loaded.source.compiled.graph.operations.map((operation) => `${operation.producer.module.name}@${operation.producer.module.version}`),
- ...loaded.run.graph.operations.map((operation) => `${operation.producer.module.name}@${operation.producer.module.version}`),
- ]);
- } finally {
- await buildLibrary?.close();
- }
- }
- } catch {
- // The normal check reports the compiler failure; package validation still returns precise
- // structural diagnostics for every local package instead of hiding them behind one exception.
- }
- const results: Record<string, unknown>[] = [];
- for (const directory of projectPackageDirectories) {
- const manifestPath = join(directory, "package.json");
- const manifest = await readJson<{ name?: string; hypit?: { activation?: string } }>(manifestPath);
- const specifier = manifest?.name ?? basename(directory);
- const errors: string[] = [];
- if (manifest === undefined) errors.push("PACKAGE_MANIFEST_MISSING");
- else if (manifest.hypit?.activation === undefined) errors.push("PACKAGE_ACTIVATION_MISSING");
- let loaded: Awaited<ReturnType<typeof loadNodePackageSelection>> = [];
- try { loaded = await loadNodePackageSelection([specifier], projectRoot, loading); }
- catch { errors.push("PACKAGE_ACTIVATION_INVALID"); }
- const selected = loaded.find((pack) => pack.specifier === specifier || physicalPackageName(pack.specifier) === physicalPackageName(specifier));
- const contribution = selected?.contribution;
- const modules = contribution?.modules ?? [];
- const surfaces = (contribution?.hostFacets ?? []).filter((facet) => facet.abi === markupSurfaceHostFacetAbi);
- // Author packages expand Markup Surfaces into Graph Fragments; they do not need the separate
- // run-fragment host facet used by preview/mock replacement packages.
- const runFragmentFacets = (contribution?.hostFacets ?? []).filter((facet) => facet.abi === "hypit.run-fragment-host@1");
- const producers = modules.flatMap((module) => module.manifest.producers);
- const fragmentValues = runFragmentFacets.flatMap((facet) => Object.values((facet.implementation as { fragments?: Record<string, { operations?: readonly unknown[]; exports?: readonly unknown[] }> }).fragments ?? {}));
- // Author Markup Surface handlers seal Graph Fragments while decoding Source; those
- // fragments are intentionally not exposed as a package Host facet. Keep the diagnostic
- // useful for author packages by reporting their declared Surfaces as the lower bound when
- // no explicit Run Fragment facet exists.
- const fragmentCount = fragmentValues.length > 0 ? fragmentValues.length : surfaces.length;
- if (surfaces.length === 0) errors.push("PACKAGE_NO_SURFACE");
- if (producers.length === 0) errors.push("PACKAGE_NO_PRODUCER");
- if (runFragmentFacets.length > 0 && fragmentValues.every((fragment) => (fragment.operations?.length ?? 0) === 0 || (fragment.exports?.length ?? 0) === 0)) errors.push("PACKAGE_NO_FRAGMENT");
- const declaredProducers = new Set(modules.flatMap((module) => module.manifest.producers.map((producer) => `${module.manifest.name}@${module.manifest.version}#${producer.name}`)));
- for (const fragment of fragmentValues) {
- for (const operation of fragment.operations ?? []) {
- const producer = (operation as { producer?: { module?: { name?: string; version?: string }; name?: string } }).producer;
- const key = producer?.module?.name !== undefined && producer.module.version !== undefined && producer.name !== undefined
- ? `${producer.module.name}@${producer.module.version}#${producer.name}`
- : undefined;
- if (key === undefined || !declaredProducers.has(key)) errors.push("PACKAGE_FRAGMENT_PRODUCER_UNDECLARED");
- }
- }
- for (const facet of surfaces) {
- const surface = (facet.implementation as Partial<RegisteredSurface>);
- if (surface.module?.name === undefined || surface.module.version === undefined || surface.surface?.trim().length === 0
- || surface.tag?.trim().length === 0 || !Array.isArray(surface.outputs) || surface.outputs.length === 0
- || surface.outputs.some((output) => output?.module?.name === undefined || output.module.version === undefined || output.name?.trim().length === 0)) {
- errors.push("PACKAGE_SURFACE_INVALID");
- }
- }
- const sourceImported = imported.has(specifier) || [...imported].some((name) => physicalPackageName(name) === specifier);
- const graphUsed = modules.some((module) => graphModules.has(`${module.manifest.name}@${module.manifest.version}`));
- if (!sourceImported) errors.push("PACKAGE_NOT_IMPORTED");
- else if (!graphUsed) errors.push("PACKAGE_IMPORTED_BUT_UNUSED");
- results.push({ specifier, loaded_specifier: selected?.specifier, manifest: modules.length > 0, surfaces: surfaces.length, producers: producers.length, fragments: fragmentCount, run_fragment_facets: runFragmentFacets.length, source_imported: sourceImported, source_used: graphUsed, graph_used: graphUsed, status: errors.length === 0 ? "passed" : "failed", ...(errors.length === 0 ? {} : { errors }) });
- }
- for (const specifier of expected) {
- if (!results.some((item) => item.specifier === specifier)) results.push({ specifier, status: "failed", errors: ["PACKAGE_EXPECTED_BUT_MISSING"] });
- }
- const vocabularyEvidence = await stat(join(projectRoot, ".hypit", "vocabulary.json")).then(() => true, () => false);
- const passed = results.every((item) => item.status === "passed") && vocabularyEvidence;
- return {
- run: runPath,
- package_root: projectRoot,
- passed,
- vocabulary_checked: vocabularyEvidence,
- ...(vocabularyEvidence ? {} : { errors: ["VOCABULARY_NOT_CHECKED"] }),
- packages: results,
- ...(projectPackageDirectories.length === 0 && expected.size === 0 ? { note: "no project-owned packages declared" } : {}),
- };
- }
- // The model is the author's choice among those @hypit/gemini declares; which Provider answers is the
- // Runtime Profile's.
- const model = options.model ?? process.env.GEMINI_MODEL?.trim() ?? geminiModels[0];
- // Pacing is deployment policy, not author intent: it depends on the quota behind the credentials,
- // which the calling agent has no way to know. It is settable here and through the environment, and
- // deliberately not through a CLI flag.
- // Four at a time with a short gap completed a whole reference twice with no rate limiting, and the
- // binding constraint is per-request latency rather than the launch gap: nine requests at four took
- // about as long as three waves, not nine. A quota that dislikes it lowers these.
- const concurrency = options.concurrency ?? positiveEnv("HYPIT_REFERENCE_CONCURRENCY") ?? 4;
- const gapMs = options.launchGapMs ?? positiveEnv("HYPIT_REFERENCE_LAUNCH_GAP_MS") ?? 1_500;
- const retryDelayMs = options.retryDelayMs ?? 2_000;
- // One observer per Runtime Profile: an injected generator serves every call, otherwise the Profile's
- // Gemini Endpoint is opened once per Profile path and reused.
- const generators = new Map<string, Promise<GenerateText>>();
- const generator = async (runtime: string | undefined): Promise<GenerateText> => {
- if (options.generate !== undefined) return options.generate;
- const profile = await runtimeProfileFor(runtime);
- let opened = generators.get(profile);
- if (opened === undefined) {
- opened = runtimeHostFor(profile).then((host) => runtimeGenerate(host, geminiModel(model)));
- generators.set(profile, opened);
- }
- return await opened;
- };
-
- const loadState = async (reference: string): Promise<ReferenceState> => {
- const state = await readJson<ReferenceState>(join(stateRoot(reference), "state.json"));
- assert(state !== undefined, `reference ${reference} is not prepared under ${referenceRoot()}; run prepare_reference first`);
- return state;
- };
-
- /**
- * What paces a round of work.
- *
- * The launch gap and the concurrency cap are a vendor's quota, so they belong to the observer that
- * sends requests and to nothing else. The observer that answers out of band reaches no Provider:
- * building a list of tasks is local and unbounded, and cutting a comparison's frames is local but
- * costs a decode, so it is paced by the machine rather than by a gap invented for someone's API.
- */
- const rateFor = (observer: Observer, work: "listing" | "ffmpeg", items: number) =>
- observer !== "agent" ? { concurrency, gapMs }
- : work === "ffmpeg" ? { concurrency: Math.max(1, Math.min(cpus().length - 1, items)), gapMs: 0 }
- : { concurrency: Number.MAX_SAFE_INTEGER, gapMs: 0 };
-
- // One reference is read by one observer throughout. Mixing them inside a single reference would
- // leave observations of two different kinds of evidence under the same keys, with nothing on the
- // record saying which is which, so the choice is made once at prepare and carried in the state.
- const askerFor = async (
- observer: Observer,
- state: ReferenceState,
- comparisonRun?: string,
- runtime?: string,
- ): Promise<{ readonly ask: Asker; readonly pending: readonly ObservationTaskRequest[]; readonly paced?: boolean }> => {
- if (observer === "agent") {
- const pending: ObservationTaskRequest[] = [];
- // The transcript's path is fixed by the reference, so it can be named before the file exists.
- // It has to be: `prepare_reference` builds its four tasks while WhisperX is still measuring
- // beside them, and one of those four is a sound question. By the time the observer opens it,
- // that call has returned and the file is written.
- const transcriptRef = join(stateRoot(state.reference_id), "transcript.json");
- // Read once for the whole pass rather than per task. Absent during prepare, and absent for good
- // on a silent reference: a task then carries the pictures alone, which is what it had before.
- const words = await referenceWords(state.reference_id).catch(() => []);
- return {
- pending,
- // Pacing is for a quota. This asker does no I/O — it appends to an array — so holding each
- // task to the launch gap spent minutes producing a list.
- paced: false,
- ask: async (key, { media, prompt, instruction, sound, shots }) => {
- // An observation reads grids of one shot each, which is what the preamble describes. A
- // comparison holds one grid of a render and one of a stretch that may cut several times, so
- // it says how to read its own pair and the preamble would contradict it.
- const parts = key.startsWith("comparison:")
- ? [] : sound === true ? [TILE_PREAMBLE, NO_SOUND] : [TILE_PREAMBLE];
- // A word counts as inside the stretch when it overlaps it at all, so one straddling either
- // end is carried rather than dropped: it is spoken over the pictures this task holds.
- const spoken = sound !== true || shots === undefined || shots.length === 0 ? "" : words
- .filter((word) => shots.some((shot) => word.endSeconds > shot.start_seconds && word.startSeconds < shot.end_seconds))
- .map((word) => word.text).join(" ");
- pending.push({
- key, instruction, prompt: [...parts, prompt, WATERMARK_RULE].join("\n\n"), image_refs: asPictures(media, state),
- ...(sound === true && words.length > 0 ? { transcript_ref: transcriptRef } : {}),
- ...(spoken.length === 0 ? {} : { transcript_words: spoken }),
- record_with: `record_observation --reference-id ${state.reference_id} `
- + `${comparisonRun === undefined ? "" : `--run ${comparisonRun} `}`
- + `--key ${key} --text-file <the answer>`,
- });
- return observation("pending", "awaiting the agent observer");
- },
- };
- }
- const generate = await generator(runtime);
- const mediaPart = mediaParts();
- return {
- pending: [],
- ask: async (_key, { media, prompt, instruction }) => {
- const parts: GeminiInlinePart[] = [];
- for (const path of media) parts.push(await mediaPart(path));
- parts.push({ text: `${prompt}\n\n${WATERMARK_RULE}` });
- return await callSafely(retryDelayMs, generate, parts, instruction);
- },
- };
- };
-
- const tools: ReferenceVideoTools = {
- // Which packages exist is the first question of every reconstruction, and until now the only
- // answer was to read a guide and a directory listing by hand. The Build CLI deliberately never
- // scans a directory; this is a development tool, so it may.
- async list_svml_packages(): Promise<Record<string, unknown>> {
- // Every scope, and the project's own directory, rather than `@hypit` alone. A project that
- // declares a vocabulary gap and fills it publishes under its own scope, and a listing that
- // cannot see those packages answers "which packages exist" with only half of them — including
- // for the project asking.
- // The installed packages a project does not carry are in the Distribution, and the ones it does
- // are beside it. Reading only the root the project resolves from answers "which packages exist"
- // with whichever half that root happens to hold.
- const roots = [...new Set([join(packageRoot, "node_modules"),
- ...(videoCliDistribution.packageRoot === undefined ? [] : [join(videoCliDistribution.packageRoot, "node_modules")])])];
- const candidates: { readonly specifier: string; readonly directory: string }[] = [];
- for (const modules of roots) {
- const scopes = (await readdir(modules, { withFileTypes: true }).catch(() => []))
- .filter((entry) => entry.isDirectory() && entry.name.startsWith("@"))
- .map((entry) => entry.name);
- for (const scope of scopes.sort()) {
- const inside = await readdir(join(modules, scope)).catch(() => [] as string[]);
- for (const name of inside.sort()) {
- const specifier = `${scope}/${name}`;
- if (!candidates.some((item) => item.specifier === specifier)) {
- candidates.push({ specifier, directory: join(modules, scope, name) });
- }
- }
- }
- }
- // A project's own packages are its `packages/<name>/`, which is where they are authored and
- // where the loader finds them by name whether or not a package manager linked them.
- for (const name of (await readdir(join(packageRoot, "packages")).catch(() => [] as string[])).sort()) {
- const directory = join(packageRoot, "packages", name);
- const own = await readJson<{ readonly name?: string }>(join(directory, "package.json"));
- if (own?.name !== undefined && !candidates.some((item) => item.specifier === own.name)) {
- candidates.push({ specifier: own.name, directory });
- }
- }
- assert(candidates.length > 0, `no packages under ${roots.join(", ")} or ${join(packageRoot, "packages")}`);
- const packages: Record<string, unknown>[] = [];
- for (const { specifier, directory } of candidates) {
- const manifest = await readJson<{ readonly hypit?: { readonly activation?: string }; readonly description?: string }>(
- join(directory, "package.json"));
- if (manifest?.hypit?.activation === undefined) continue;
- // A package publishes several kinds of facet. Reading them all as one kind produced a null
- // for every facet that is not a Markup Surface, which is what an exact-model package mostly
- // publishes — the listing said `["TextVideo", "Value", null]` and a Provider said `[null]`.
- const tags: string[] = [];
- const models: string[] = [];
- let note: string | undefined;
- try {
- for (const pack of await loadNodePackageSelection([specifier], packageRoot, packageLoading())) {
- for (const facet of pack.contribution.hostFacets ?? []) {
- if (facet.abi === markupSurfaceHostFacetAbi) {
- tags.push((facet.implementation as RegisteredSurface).tag);
- continue;
- }
- if (facet.abi === exactModelHostAbi) {
- models.push(...(facet as { readonly offers?: readonly string[] }).offers ?? []);
- }
- }
- }
- } catch (error) { note = error instanceof Error ? error.message : String(error); }
- packages.push({
- package_name: specifier,
- ...(manifest.description === undefined ? {} : { description: manifest.description }),
- tags: [...new Set(tags)].sort(),
- ...(models.length === 0 ? {} : { models: [...new Set(models)].sort() }),
- ...(note === undefined ? {} : { unreadable: note }),
- });
- }
- return { package_root: packageRoot, packages };
- },
-
- async prepare_reference(input): Promise<PrepareResult> {
- assert(input.redo === undefined || input.redo === "media" || input.redo === "transcript" || input.redo === "people" || input.redo === "voices" || input.redo === "systems" || input.redo === "places" || input.redo === "all", "redo must be one of: media, transcript, people, voices, systems, places, all");
- // A reference given as a link becomes a file before anything else runs, and everything after
- // this reads the file without learning where it came from.
- const fetched = isReferenceUrl(input.video_path)
- ? await downloadReferenceVideo(input.video_path.trim(), join(referenceRoot(), "downloads"))
- : undefined;
- const videoPath = fetched?.path ?? resolve(input.video_path);
- const file = await stat(videoPath).catch(() => undefined);
- assert(file?.isFile(), `video_path is not a file: ${videoPath}`);
- const reference = await referenceId(videoPath);
- const root = stateRoot(reference);
- const statePath = join(root, "state.json");
- let existing = await readJson<ReferenceState>(statePath);
- const redoMedia = input.redo === "media" || input.redo === "all";
- const redoTranscript = input.redo === "transcript" || input.redo === "all";
- const redoPeople = input.redo === "people" || input.redo === "all";
- const redoVoices = input.redo === "voices" || input.redo === "all";
- const redoSystems = input.redo === "systems" || input.redo === "all";
- const redoPlaces = input.redo === "places" || input.redo === "all";
- // A reference keeps the observer it was prepared with. Asking for the other one on a reference
- // that already holds observations would read the second half of it through different evidence
- // from the first, so the mismatch is refused rather than silently mixed.
- const observer: Observer = input.observer ?? existing?.observer ?? "gemini";
- assert(input.redo === "all" || existing?.observer === undefined || existing.observer === observer,
- `reference ${reference} was prepared for the ${existing?.observer} observer; --redo all re-prepares it for the other one`);
- if (input.redo === undefined && existing !== undefined && prepared(existing)) return { ...publicPrepare(existing), observer, pending_observations: [] };
- await ensureDir(root);
- const info = existing?.video === undefined || redoMedia ? await probe(videoPath) : {
- duration: existing.video.duration_seconds,
- width: existing.video.width,
- height: existing.video.height,
- hasAudio: existing.video.has_audio,
- };
- let state: ReferenceState;
- let analysisVideo: string;
- if (existing !== undefined && !redoMedia && existing.shots.length > 0) {
- state = existing;
- analysisVideo = existing.analysis_video_ref;
- } else {
- const media = await prepareMedia(videoPath, root, info.duration, observer === "agent");
- const shots = await Promise.all(media.bounds.map((bound, index) => shotFromBound(root, index, bound, observer === "agent")));
- state = {
- reference_id: reference,
- video_path: videoPath,
- root,
- video: { duration_seconds: Number(info.duration.toFixed(3)), width: info.width, height: info.height, has_audio: info.hasAudio },
- shots,
- storyboard_ref: media.storyboard,
- analysis_video_ref: media.analysisVideo,
- // The transcript is measured from the source audio, which rebuilt shot media cannot change.
- ...(existing?.transcript === undefined ? {} : { transcript: existing.transcript }),
- ...(existing?.people_and_product === undefined ? {} : { people_and_product: existing.people_and_product }),
- ...(existing?.voices === undefined ? {} : { voices: existing.voices }),
- ...(existing?.persistent_systems === undefined ? {} : { persistent_systems: existing.persistent_systems }),
- ...(existing?.places === undefined ? {} : { places: existing.places }),
- };
- analysisVideo = media.analysisVideo;
- await writeJson(statePath, state);
- await writeFile(join(root, "observations.json"), "{}\n", "utf8");
- }
- // WhisperX is local and slow and shares nothing with the observation requests, so it runs beside
- // them rather than ahead of them.
- //
- // The service is the one precondition this command can check itself. The route used to probe it
- // by hand first; it reports rather than refuses — a machine without the service still prepares
- // everything except the transcript, which stays recoverable — and points at the repair. A
- // reference whose transcript is already complete and unchanged needs no probe.
- const needsTranscript = !(state.transcript?.status === "complete" && !redoTranscript);
- const alignment = needsTranscript ? await alignmentThrough(input.runtime) : undefined;
- const transcribing = state.transcript?.status === "complete" && !redoTranscript
- ? Promise.resolve(state.transcript)
- : prepareTranscript(reference, videoPath, root, info.hasAudio, "en", redoTranscript, alignment!.invoke);
- const { ask, pending } = await askerFor(observer, state, undefined, input.runtime);
- const whole = [analysisVideo];
- const [people, voices, systems, places] = await Promise.all([
- state.people_and_product?.status === "complete" && !redoPeople
- ? Promise.resolve(state.people_and_product)
- : ask("people_and_product", { media: whole, prompt: "Describe the recurring people and the promoted product in this complete reference video. Return natural language only. Identify stable visual traits and distinguish recurring speakers from incidental people in inserts. Describe the product once, comprehensively, for reuse.", instruction: "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names." }),
- state.voices?.status === "complete" && !redoVoices
- ? Promise.resolve(state.voices)
- : ask("voices", { media: whole, sound: true, prompt: "Listen to this complete reference video and describe the distinct voices, their order, overlap, off-screen speech, and likely correspondence to visible people. Do not assign a voice merely because a person appears in a B-roll image. Return natural language only.", instruction: "You only listen to a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names." }),
- state.persistent_systems?.status === "complete" && !redoSystems
- ? Promise.resolve(state.persistent_systems)
- : ask("persistent_systems", { media: whole, prompt: "Describe the on-screen text and graphic systems that persist or recur across this complete reference video, such as subtitles, running lists, counters, progress indicators, badges, watermarks, lower thirds and repeating full-screen graphic layouts. For each one, state when it first appears and when it stops, whether it is present continuously or intermittently, whether its own appearance stays the same throughout, and describe any point where its appearance actually changes. Report only what stays consistent across the video; ignore one-off elements that appear a single time. Return natural language only.", instruction: "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names." }),
- state.places?.status === "complete" && !redoPlaces
- ? Promise.resolve(state.places)
- : ask("places", { media: whole, prompt: "Describe every distinct place this reference video was shot in, and every distinct camera position within each place. State how many places there are, which parts of the video happen in each, and for each place which camera positions appear and which parts of the video use each one. Two shots are the same camera position when the camera sees the same part of the room from the same side; a reverse angle is a different position of the same place.\n\nDescribe each camera position in enough detail that someone who has never seen this video could draw it from your words alone: what is behind and beside the subject, the shape and depth of the space, where the light comes from and how hard it is, the colours and materials of the surfaces, and the objects a viewer would use to recognise it again. Say what stays identical between positions of one place and what differs.\n\nReturn natural language only.", instruction: "You only observe a reference video. Return natural language evidence only. Do not write code, markup, SVML, or component names." }),
- ]);
- // Only a complete observation is written. A pending one is a task the agent still owes, and a
- // failed one is the error that ended the attempt — written, either would be read back as an
- // answer by the next run and quoted into every shot prompt as full-reference evidence.
- const answered = <T extends Observation>(value: T): T | undefined => value.status === "complete" ? value : undefined;
- const complete: ReferenceState = {
- ...state,
- observer,
- transcript: await transcribing,
- ...(answered(people) === undefined ? {} : { people_and_product: people }),
- ...(answered(voices) === undefined ? {} : { voices }),
- ...(answered(systems) === undefined ? {} : { persistent_systems: systems }),
- ...(answered(places) === undefined ? {} : { places }),
- };
- await writeJson(statePath, complete);
- return {
- ...publicPrepare(complete),
- people_and_product: people,
- voices,
- persistent_systems: systems,
- places,
- observations: {
- people_and_product: people,
- voices,
- persistent_systems: systems,
- places,
- },
- observer,
- ...(fetched === undefined ? {} : { source_url: fetched.url, downloaded: !fetched.cached }),
- pending_observations: pending,
- ...(alignment?.note === undefined ? {} : { whisperx: alignment.note }),
- };
- },
-
- async observe_reference(input): Promise<Record<string, unknown>> {
- if (input.questions !== undefined) {
- const round = input.questions;
- assert(round.length > 0, "questions is empty");
- const rate = rateFor((await loadState(input.reference_id)).observer ?? "gemini", "listing", round.length);
- const { done, failures } = await runBatch(round, rate, async (asked) => await tools.observe_reference({
- reference_id: input.reference_id, shot_ids: asked.shot_ids, question: asked.question,
- }));
- return {
- reference_id: input.reference_id,
- asked: done.length,
- failed: failures.length,
- answers: done,
- ...(failures.length === 0 ? {} : { failures }),
- // The one thing gathered across the whole round: an observer that answers out of band owes
- // one task per question, and they are answered together rather than call by call.
- pending_observations: done.flatMap((item) => (item["pending_observations"] ?? []) as readonly unknown[]),
- };
- }
- const state = await loadState(input.reference_id);
- const selected = input.shot_ids === undefined ? state.shots : state.shots.filter((shot) => input.shot_ids!.includes(shot.shot_id));
- assert(selected.length > 0, "no requested shot ids exist");
- const observer: Observer = state.observer ?? "gemini";
- const { ask, pending, paced } = await askerFor(observer, state, undefined, input.runtime);
- // An observer that answers out of band produces a list rather than a request, so the launch gap
- // and the concurrency cap have nothing to pace.
- const rate = paced === false ? { concurrency: Number.MAX_SAFE_INTEGER, gapMs: 0 } : { concurrency, gapMs };
- const question = input.question?.trim() ?? "";
- if (question.length > 0) {
- assert(input.shot_ids !== undefined && input.shot_ids.length > 0, "question requires at least one shot id, so that it is answered from the shots it is about");
- assert(selected.length === 1, "question accepts exactly one shot id; use the built-in boundary/window observations for cross-shot continuity");
- // Keep the actual question in the key. It remains directly inspectable and distinguishes two
- // questions over the same shot without inventing a second identifier namespace.
- const key = `question:${selected.map((shot) => shot.shot_id).join("+")}:${encodeURIComponent(question)}`;
- const path = join(state.root, "observations.json");
- const held = (await readJson<Record<string, Observation>>(path))?.[key];
- const reused = held?.status === "complete" && input.reobserve !== true;
- // Through the same cache the sweep uses, rather than straight to the asker. Asked directly, an
- // answer recorded against this key was written to a file nothing read: the observer that
- // answers out of band could never retrieve one, and the observer that bills was asked again.
- const answers = await runObservationTasks(
- state.root,
- [{ key, request: {
- media: selected.flatMap((shot) => [shot.clip_ref, shot.representative_frame_ref]),
- // A narrow question is as likely to be about who is speaking as about what is on screen, and
- // this is the one prompt where the observer reading pictures was not told it cannot hear.
- sound: true,
- shots: selected,
- prompt: question,
- instruction: "Answer only the narrow reference-video question in natural language. Do not write code, markup, SVML, or component names.",
- } }],
- input.reobserve === true ? new Set([key]) : new Set<string>(),
- rate.concurrency, rate.gapMs, ask,
- );
- const answer = answers.get(key)!;
- return {
- reference_id: state.reference_id,
- observer,
- shot_ids: selected.map((shot) => shot.shot_id),
- question,
- observation_key: key,
- result: answer,
- // Says the answer came back without asking anyone, so a reader knows they were not billed
- // for it and that `--reobserve` is what asks again.
- ...(reused ? { reused: true } : {}),
- unresolved: answer.status === "complete" ? [] : [key],
- pending_observations: pending,
- };
- }
- // Every shot observation is asked with the four whole-reference observations quoted into it. The
- // observer that answers in band has them the moment prepare_reference returns; the one that
- // answers out of band has not written them yet, and a sweep run first would ask every shot its
- // question with that context missing and cache the answers.
- if (observer === "agent") {
- const owed = WHOLE_REFERENCE_KEYS.filter((key) => state[key]?.status !== "complete");
- assert(owed.length === 0,
- `answer the whole-reference observations first, with record_observation: ${owed.join(", ")}. `
- + "Every shot observation quotes them, so a sweep run before they exist asks each shot with less than it should have.");
- }
- const selectedIds = new Set(selected.map((shot) => shot.shot_id));
- const boundaryRights = state.shots.filter((shot) => {
- const left = state.shots.find((candidate) => candidate.index === shot.index - 1);
- return left !== undefined && (selectedIds.has(left.shot_id) || selectedIds.has(shot.shot_id));
- });
- // Only a complete observation is evidence. A failed one carries the error that ended it, and
- // read by `.text` alone that string was quoted into every shot prompt underneath a heading
- // announcing it as full-reference evidence.
- const evidence = (value: Observation | undefined, heading: string): string =>
- value?.status === "complete" && value.text.trim().length > 0 ? `${heading}:\n${value.text}` : "";
- const globalContext = [
- evidence(state.people_and_product, "Full-reference people and product evidence"),
- evidence(state.voices, "Full-reference voice evidence"),
- evidence(state.persistent_systems, "Full-reference persistent on-screen system evidence"),
- // The sweep already refuses to start until this is complete, so leaving it out of the context
- // it was waited for made the wait buy nothing.
- evidence(state.places, "Full-reference places and camera-position evidence"),
- ].filter((item) => item.length > 0).join("\n\n");
- const visualTasks: ObservationTask[] = selected.map((shot) => {
- const previous = state.shots.find((candidate) => candidate.index === shot.index - 1);
- return { key: `visual:${shot.shot_id}`, request: {
- media: [shot.clip_ref, shot.representative_frame_ref, ...(previous === undefined ? [] : [previous.tail_frame_ref])],
- prompt: `${globalContext}\n\nObserve shot ${shot.index}. Describe the current base picture, any covering or non-covering visual content, and whether visible content continues from the preceding shot. A full-screen insert is still only a picture observation. Covering content is whatever changes what reaches the eye, not only things that sit on top with an edge: a wash, darkening, gradient or semi-transparent layer laid over the whole frame or a region of it is covering content and must be reported as such, including what the base shows through it.\n\nAlso answer these two questions explicitly.\n\nFirst: is the whole frame a depicted scene that has its own camera space, lighting, depth and lens behaviour, or is it a flat designed field whose purpose is to carry drawn elements such as words, rows, panels or cards? Live action and animation are both depicted scenes; a paper sheet, ruled or gridded surface, flat or gradient colour, blurred wallpaper, board or slide backdrop filling the frame is a designed field. State which one it is and the visible evidence for it.\n\nThen, whichever it is, say whether any *region* of the frame is itself a flat designed field carrying drawn content — a list, a ranking, a leaderboard, a scoreboard, a chart, a panel, a slab of colour holding rows or labels — even when the rest of the frame is a depicted scene, and even when the region has no visible border, card edge or drop shadow around it. Report each such region separately from the scene behind it: roughly where it sits and how much of the frame it covers, given as fractions of the frame width and height; what its own surface is; and what is drawn on it. A designed field occupying half the frame, one side of it, or a band across it is easy to describe as "text over the picture" and is not that: the field itself is the thing to report.\n\nSecond: for every framed element inside the picture, such as a card, phone, browser window, screenshot or inset, describe the picture inside the frame and the frame itself separately, and say separately whether each one moves. For the inside, describe what it depicts and whether its content moves — scrolling, paging, playing, changing. For the frame, describe its border, corner radius, outline, shadow, size and position, whether that rectangle stays in the same place and at the same size across the whole shot or travels, and how it enters and leaves. These two are independent and are routinely confused: a screen recording of a page being scrolled is a frame that does not move at all while its contents travel inside it. Decide it from the frame's own edge — track the border and corners across the stretch and say whether that rectangle held still.\n\nThird: does this picture move at all, and how? Separate three things: whether the camera moves, and how; whether anything in the picture moves, and what; and whether the picture is completely still. A held photograph, screenshot or card that only appears and disappears is still, however long it is on screen. Where the only movement is inside a framed element, say so in those words rather than reporting the picture as moving.\n\nReturn natural language evidence only.`,
- instruction: "Observe picture only. Do not choose SVML components, do not write markup, and do not decide final source syntax.",
- } };
- });
- const typeTasks: ObservationTask[] = selected.map((shot) => ({ key: `type:${shot.shot_id}`, request: {
- media: [shot.representative_frame_ref, shot.clip_ref],
- prompt: `Observe the on-screen text in shot ${shot.index}. Report only text that is drawn over or composed into the picture; ignore text belonging to a photographed object such as a device screen, sign, label or document. If the shot shows no drawn text, say exactly that and stop.\n\nFor each distinct text element describe: the typeface character (serif, sans-serif, handwritten, monospaced or display), the weight, the cap height as a fraction of the frame height, letter spacing and line spacing, alignment, letter case, fill colour, any outline or stroke with its colour and thickness relative to the stroke width of the letters, any drop shadow with its direction, distance and softness, any glow or blur, any emphasis applied to individual words or characters and how it differs from the rest, the position within the frame and the distance from the nearest edges, and how the text enters, changes and leaves during the shot.\n\nReturn natural language evidence only.`,
- instruction: "Observe the appearance of on-screen text only. Do not name SVML components, do not write markup, and do not name font files or style properties from any software.",
- } }));
- const audioTasks: ObservationTask[] = selected.map((shot) => {
- const previous = state.shots.find((candidate) => candidate.index === shot.index - 1);
- return { key: `audio:${shot.shot_id}`, request: {
- media: [shot.clip_ref, ...(previous?.audio_tail_ref == null ? [] : [previous.audio_tail_ref])],
- sound: true,
- shots: [shot],
- prompt: `${globalContext}\n\nListen to shot ${shot.index}. Decide who is speaking, whether sound continues from the previous shot, whether visible people actually produce the sound, and whether a silent B-roll person only appears to speak. Handle off-screen, alternating, overlapping, and no-person-visible speech. Return natural language evidence only.`,
- instruction: "Observe sound only. Do not write markup, SVML, JSON plans, or component names.",
- } };
- });
- // Both boundary questions look at exactly the same three files, so they travel together. Asking
- // them separately uploaded two clips twice to learn two things about one cut.
- const boundaryTasks: ObservationTask[] = boundaryRights.map((right) => {
- const left = state.shots.find((shot) => shot.index === right.index - 1)!;
- return { key: `boundary:${right.shot_id}`, request: {
- media: [left.clip_ref, right.clip_ref, left.tail_frame_ref],
- sound: true,
- shots: [left, right],
- prompt: `Compare shots ${left.index} and ${right.index}, which meet at one cut. Answer two questions separately, each under its own heading.\n\nContinuous camera shot: are these one continuous camera shot, or two? Explain the visual and sound continuity evidence.\n\nOverlay continuity: does the same visible overlay or inserted picture continue across the boundary, change, or end? Explain the evidence.\n\nReturn natural language evidence only.`,
- instruction: "Analyze what happens at one cut. Do not name SVML components or write code.",
- } };
- });
- const allTasks = [...visualTasks, ...typeTasks, ...audioTasks, ...boundaryTasks];
- const reobserve = input.reobserve === true;
- const forcedKeys = reobserve ? new Set(allTasks.map((task) => task.key)) : new Set<string>();
- const observations = await runObservationTasks(state.root, allTasks, forcedKeys, rate.concurrency, rate.gapMs, ask);
- const unresolved = [...observations].filter(([, value]) => value.status !== "complete").map(([key]) => key);
- // Only an answered boundary decides this. An unanswered one carries a placeholder or an error,
- // and reading those as prose put the review on whether the failure text happened to contain one
- // of the words below.
- const answeredBoundaries = boundaryRights
- .map((right) => observations.get(`boundary:${right.shot_id}`))
- .filter((value) => value?.status === "complete");
- const boundaryText = answeredBoundaries.map((value) => value!.text).join("\n");
- // A boundary nobody answered has not said the shots are separate. Deciding the three-shot review
- // from the answered ones alone would let a sweep whose boundaries all failed report no review
- // needed, which reads the same as a reference with no continuity to check.
- const undecidedBoundaries = boundaryRights.length - answeredBoundaries.length;
- const needsThreeShotReview = selected.length >= 3 && (
- reobserve ||
- /uncertain|unknown|possibly|may be|contin(?:ue|ues)|same take|same shot/iu.test(boundaryText)
- );
- const windows = (needsThreeShotReview
- ? selected.flatMap((shot, index) => {
- const window = selected.slice(index, index + 3);
- return window.length === 3 && window[1]!.index === shot.index + 1 && window[2]!.index === shot.index + 2
- ? [window]
- : [];
- })
- : []).map((items) => items as unknown as readonly [Shot, Shot, Shot]);
- // A three-shot review is an observation like any other, so it is cached like any other. Asking
- // for it directly meant an answer recorded against its key was never read back, which left the
- // review unanswerable for an observer that answers out of band.
- const windowTasks: ObservationTask[] = windows.map((items) => ({
- key: `window:${items[0].shot_id}`,
- request: {
- media: items.map((shot) => shot.clip_ref),
- // Continuity across a cut is heard as much as seen, which is why the two-shot boundary asks
- // with sound. This asks the same question over three shots, so an observer reading pictures
- // is told the same thing about what it is missing.
- sound: true,
- shots: items,
- prompt: `Compare shots ${items[0].index}, ${items[1].index}, and ${items[2].index}. Decide whether the three clips are one continuous camera shot and whether one visual overlay or insert persists through both boundaries. Explain the evidence in natural language only.`,
- instruction: "Analyze a three-shot continuity window only. Do not write markup, SVML, JSON plans, or component names.",
- },
- }));
- const windowObservations = await runObservationTasks(
- state.root, windowTasks,
- reobserve ? new Set(windowTasks.map((task) => task.key)) : new Set<string>(),
- rate.concurrency, rate.gapMs, ask,
- );
- const threeShotObservations = windows.map((items) => ({
- shot_ids: items.map((shot) => shot.shot_id),
- combined_duration_seconds: Number((items[2].end_seconds - items[0].start_seconds).toFixed(3)),
- result: windowObservations.get(`window:${items[0].shot_id}`)!,
- }));
- for (const window of threeShotObservations) {
- if (window.result.status !== "complete") unresolved.push(`three-shot:${window.shot_ids.join("+")}`);
- }
- // The review is decided from what the boundaries said, so boundaries nobody has answered leave it
- // undecided rather than unnecessary. Producing no window task and reporting nothing is how a
- // sweep that answered its boundaries out of band and stopped after one pass ends up with a
- // continuity review that was never run and never missed.
- if (selected.length >= 3 && undecidedBoundaries > 0 && !needsThreeShotReview) {
- unresolved.push(`three-shot-review-undecided:${undecidedBoundaries} boundar`
- + `${undecidedBoundaries === 1 ? "y is" : "ies are"} unanswered, so whether a three-shot continuity `
- + "review is needed has not been decided; answer them and run observe_reference again");
- }
- const shotResults = selected.map((shot) => ({
- shot_id: shot.shot_id,
- visual: observations.get(`visual:${shot.shot_id}`)!,
- text_appearance: observations.get(`type:${shot.shot_id}`)!,
- audio: observations.get(`audio:${shot.shot_id}`)!,
- }));
- const boundaryResults = boundaryRights.map((right) => {
- const left = state.shots.find((shot) => shot.index === right.index - 1)!;
- return {
- left_shot_id: left.shot_id,
- right_shot_id: right.shot_id,
- combined_duration_seconds: Number((right.end_seconds - left.start_seconds).toFixed(3)),
- within_15_seconds: right.end_seconds - left.start_seconds <= 15,
- continuity: observations.get(`boundary:${right.shot_id}`)!,
- };
- });
- return {
- reference_id: state.reference_id,
- observer,
- shots: shotResults,
- boundaries: boundaryResults,
- three_shot_observations: threeShotObservations,
- unresolved,
- pending_observations: pending,
- };
- },
-
- async inspect_svml_vocabulary(input): Promise<Record<string, unknown>> {
- assert(input.package_names.length > 0, "package_names must not be empty");
- const loaded = await loadNodePackageSelection(input.package_names, packageRoot, packageLoading());
- const requested = input.tags === undefined ? undefined : new Set(input.tags);
- // The answer is about the packages that were named. Loading one brings its dependencies with
- // it, and every one of their Surfaces used to be returned as well — asking what `@hypit/ranking`
- // offers answered with twenty Surfaces of which six were its own, sixty-four kilobytes for a
- // question about six. The rest are reachable by naming them, which is what `list_svml_packages`
- // is for.
- const named = new Set(input.package_names);
- const surfaces: Record<string, unknown>[] = [];
- for (const pack of loaded) {
- if (!named.has(pack.specifier)) continue;
- for (const facet of pack.contribution.hostFacets ?? []) {
- const implementation = facet.implementation as RegisteredSurface;
- if (requested !== undefined && !requested.has(implementation.tag) && !requested.has(implementation.surface)) continue;
- surfaces.push({
- package_name: pack.specifier,
- module: implementation.module,
- surface: implementation.surface,
- tag: implementation.tag,
- mode: implementation.mode,
- outputs: implementation.outputs,
- vocabulary: vocabularyForResult(input.include_previews === false ? withoutPreview(implementation.vocabulary) : implementation.vocabulary),
- readme_path: readmePath(pack.specifier, packageRoot, packageLoading()),
- });
- }
- }
- const result = { packages: input.package_names, surfaces };
- return result;
- },
-
- async validate_local_author_packages(input): Promise<Record<string, unknown>> {
- return await validateLocalAuthorPackages(input);
- },
-
- async validate_script_cues(input): Promise<Record<string, unknown>> {
- return await scriptCueCheck(input);
- },
-
- /**
- * What a Producer that draws is allowed to return.
- *
- * `inspect_svml_vocabulary` answers what a Source may write; this answers what the package behind
- * it may emit — the element kinds, the style names admitted on them, which of those take an enum,
- * and how few keyframes an animation may carry. Every line is generated from `composition`'s own
- * declared schema, so it cannot drift from what the seal will accept. Without it the only way to
- * learn the shape was to open a package that already draws and copy its habits.
- */
- async inspect_visual_schema(input): Promise<Record<string, unknown>> {
- const shapes: Record<string, ValueSchema> = {
- "visual-track": visualTrackSchema,
- "visual-element": visualElementSchema,
- box: visualBoxSchema,
- mask: visualMaskSchema,
- text: visualTextSchema,
- image: visualImageSchema,
- video: visualVideoSchema,
- surface: visualSurfaceSchema,
- "text-flow": visualTextFlowSchema,
- "text-typography": visualTextTypographySchema,
- "text-paint": visualTextPaintSchema,
- "text-document": visualTextDocumentSchema,
- "path-command": visualPathCommandSchema,
- };
- const asked = input.shape?.trim();
- assert(asked === undefined || asked.length === 0 || Object.hasOwn(shapes, asked),
- `shape must be one of ${Object.keys(shapes).join(", ")}`);
- const chosen = asked === undefined || asked.length === 0 ? Object.keys(shapes) : [asked];
- // Which Producers a Fragment may call, read off the Modules the named packages carry. A
- // component's Fragment names one per operation, and the names were reachable only by opening a
- // package that already called them.
- const wanted = input.producers ?? [];
- const calls = wanted.length === 0 ? [] : (await loadNodePackageSelection(wanted, packageRoot, packageLoading()).catch(() => []))
- .flatMap((pack) => (pack.contribution.modules ?? []).flatMap((module) =>
- module.manifest.producers.map((producer) => ({
- module: `${module.manifest.name}@${module.manifest.version}`,
- producer: producer.name,
- inputs: producer.inputs.map((port) => port.name),
- outputs: producer.outputs.map((port) => port.name),
- ...(producer.needs.length === 0 ? {} : { needs: producer.needs.map((port) => port.name) }),
- }))));
- return {
- shapes: chosen.map((name) => ({ shape: name, describes: describeSchema(shapes[name]!).join("\n") })),
- animatable_local_styles: [...animatableLocalStyles],
- visual_style_names: [...VISUAL_STYLE_NAMES_V1],
- visual_style_enum_values: VISUAL_STYLE_ENUM_VALUES_V1,
- ...(calls.length === 0 ? {} : { producers: calls }),
- // Each of these is refused somewhere, or follows from how the emitted CSS is written. None is
- // a convention: a rule stated here that the code does not hold would be worse than silence.
- rules: [
- "A Present holds exactly one element with no `parent`; every other element names one, and it "
- + "must be a box or a mask. (composition/src/track.ts)",
- "`order` is unique across the whole Present, not among siblings. (composition/src/track.ts)",
- "A child's position is measured from its parent's box, not from the Canvas, so a layout "
- + "computed in Canvas pixels subtracts the parent's origin.",
- "An animation carries at least two keyframes, and every keyframe of one animation declares "
- + "the same properties: one that appears in some and not others is interpolated from the "
- + "element's own value on the frames it is missing from. `atFrame` is Present-relative; the "
- + "document normalizes the animation domain to `max(presentDurationFrames, lastKeyframe.atFrame)`.",
- "A plain text element must provide exact ordered font artifacts and cannot combine them with "
- + "raw `font`, `font-family`, `font-style`, `font-synthesis` or `font-weight` styles.",
- "A text element using glyph Paint cannot also declare raw stroke or paint-order styles.",
- "A media element's artifact media type must match image/video, and still images cannot carry sampling.",
- "A Surface with still timing cannot carry sampling; frame-based Surface sampling must match its typed timing.",
- "Mask roots must be direct children of one Present and both mask/content references must resolve to owned elements.",
- ],
- };
- },
-
- /** Where this command reads and resolves from, so nothing has to describe it from outside. */
- async paths(): Promise<Record<string, unknown>> {
- return {
- reference_state: referenceRoot(),
- package_root: packageRoot,
- working_directory: invokedFrom(),
- references: (await readdir(referenceRoot(), { withFileTypes: true }).catch(() => []))
- .filter((entry) => entry.isDirectory()).map((entry) => entry.name),
- };
- },
-
- async compare_reconstruction(input): Promise<Record<string, unknown>> {
- // Every render is finished before any comparison starts, so the comparisons are independent of
- // each other and there is nothing to gain by running them one at a time. Handing the whole list
- // over means the pacing, the per-comparison result and the failure of any one of them are the
- // tool's business rather than a shell script's.
- if (input.comparisons !== undefined) {
- const batch = input.comparisons;
- assert(batch.length > 0, "comparisons is empty");
- // Each entry cuts and tiles its own two sides, so on the observer that answers out of band
- // this round is ffmpeg work rather than a queue of requests.
- const rate = rateFor((await loadState(input.reference_id)).observer ?? "gemini", "ffmpeg", batch.length);
- const { done, failures } = await runBatch(batch, rate,
- async (one) => await tools.compare_reconstruction({ reference_id: input.reference_id, ...one }));
- return {
- reference_id: input.reference_id,
- compared: done.length,
- failed: failures.length,
- comparisons: done,
- ...(failures.length === 0 ? {} : { failures }),
- };
- }
- const state = await loadState(input.reference_id);
- const root = stateRoot(state.reference_id);
- const named = [input.shot_id, input.segment, input.selection, input.tokens].filter((value) => value !== undefined);
- assert(named.length === 1, "name exactly one of shot_id, segment, selection and tokens");
- if (input.tokens !== undefined) tokenWindow(input.tokens, "tokens");
- const supplied = [input.image_path, input.video_path].filter((value) => value !== undefined);
- assert(supplied.length === 1, "supply exactly one of image_path and video_path");
- const clip = input.video_path !== undefined;
- const renderedPath = resolve(String(supplied[0]));
- const file = await stat(renderedPath).catch(() => undefined);
- assert(file?.isFile(), `${clip ? "video_path" : "image_path"} is not a file: ${renderedPath}`);
- const scope = input.question?.trim() ?? "";
- const startedAt = new Date().toISOString();
- // Each explicit comparison gets its own readable directory. Repeating the command is a new
- // observation by request; no hidden equivalence rule silently reuses an older answer.
- const slot = eventId(basename(renderedPath).replace(/\.[^.]*$/u, ""), new Date(startedAt));
- const standIn = await standInBeside(renderedPath);
- const observer: Observer = state.observer ?? "gemini";
- const { ask, pending } = await askerFor(observer, state, input.run, input.runtime);
-
- // Which stretch of the reference the render is put beside. A render covers the words a Segment
- // or a Selection marks, so that is what the reference is cut to; a shot is a cut in the picture
- // and is compared as the whole shot it already is.
- let shot: Shot | undefined;
- let cut: RangeCut | undefined;
- let referenceMedia: string;
- let renderedMedia = renderedPath;
- let stretchSeconds: number;
- if (input.shot_id !== undefined) {
- shot = state.shots.find((candidate) => candidate.shot_id === input.shot_id);
- assert(shot !== undefined, `shot ${input.shot_id} does not exist in reference ${input.reference_id}`);
- referenceMedia = clip ? shot.clip_ref : shot.representative_frame_ref;
- stretchSeconds = shot.duration_seconds;
- } else {
- assert(input.run !== undefined,
- "run is required with a word range: the words are read from the Run's Author SVML");
- const { path: svmlPath } = await authorSource(resolve(invokedFrom(), input.run));
- const focus: StandInFocus = {
- ...(input.segment === undefined ? {} : { segment: input.segment }),
- ...(input.selection === undefined ? {} : { selection: input.selection }),
- ...(input.tokens === undefined ? {} : { tokens: input.tokens }),
- };
- const range = await spokenRange(svmlPath, focus, await referenceWords(state.reference_id));
- cut = await cutWordRange(state, root, focus, range, renderedPath, clip, slot);
- referenceMedia = cut.reference;
- renderedMedia = cut.rendered;
- stretchSeconds = cut.seconds;
- }
-
- // A stretch that holds one picture on both sides has one picture to compare, and a clip or a
- // grid of the same frame repeated says nothing the frame alone does not.
- //
- // Both sides are measured, and both have to be still. The render's base is a flat preview mock
- // and is therefore still whatever it covers, so a render measured on its own would carry every
- // pair down to a frame — including the pairs whose reference moves, where the difference
- // between a mock base and a real one would be doing the deciding.
- let bothStill = false;
- if (clip) {
- bothStill = await completelyStill(referenceMedia) && await completelyStill(renderedMedia);
- if (bothStill) {
- referenceMedia = shot === undefined
- ? await cutFrame(cut!.reference, stretchSeconds / 2, cut!.referenceStill)
- : shot.representative_frame_ref;
- let stillTarget = cut?.renderedStill ?? "";
- if (shot !== undefined) {
- stillTarget = join(root, "comparisons", `shot-${shot.shot_id}`, slot, "reconstruction-still.jpg");
- await ensureDir(dirname(stillTarget));
- }
- renderedMedia = await cutFrame(renderedMedia, stretchSeconds / 2, stillTarget);
- }
- }
- const asClip = clip && !bothStill;
-
- // Reference and rendered clips may legitimately have different durations (for example, spoken
- // delivery and generated-video timing). The observer compares each side over its own stretch;
- // duration differences are evidence to report, not a precondition that blocks the comparison.
-
- // A clip comparison reads the whole stretch on both sides. The observer that reads video is
- // given the two clips; the observer that reads pictures is given two frame tiles, which is the
- // same degradation `prepare_reference` already applies to a shot. Tiling both against the same
- // duration makes `tileFrames` choose the same frame count and layout for each, so the two grids
- // are read side by side rather than as different samplings.
- //
- // Both grids are drawn at one cell width, and it is the width the reference itself has. Left at
- // a fixed 480 the two sides were degraded in opposite directions — a 360-wide reference enlarged
- // and a 1080-wide render shrunk — and the prompt asks about stroke weight and letter spacing,
- // which is exactly what that changes.
- const cellWidth = Math.min(480, state.video.width);
- if (asClip && observer === "agent") {
- const desiredFrames = tileFrames(stretchSeconds);
- if (shot !== undefined) {
- assert(shot.frames_tile_ref !== null,
- `shot ${shot.shot_id} has no frame tile; re-prepare the reference with --redo all --observer agent`);
- referenceMedia = shot.frames_tile_ref;
- const tileTarget = join(root, "comparisons", `shot-${shot.shot_id}`, slot, "reconstruction.jpg");
- await ensureDir(dirname(tileTarget));
- const renderedDuration = (await probe(renderedPath)).duration;
- renderedMedia = await shotTile(renderedPath, renderedDuration, tileTarget, cellWidth, desiredFrames);
- } else {
- // A word range is cut when it is asked for, so neither side has a prepared tile and both
- // are built here, from the two cuts.
- referenceMedia = await shotTile(cut!.reference, cut!.referenceSeconds, cut!.referenceTile, cellWidth, desiredFrames);
- renderedMedia = await shotTile(cut!.rendered, cut!.renderedSeconds, cut!.renderedTile, cellWidth, desiredFrames);
- }
- }
-
- // What the observer is actually holding. A grid is neither a clip nor a single moment, and
- // calling it a still image while asking what changes over the stretch described two different
- // things to the same reader.
- const grids = asClip && observer === "agent";
- const unit = grids ? "frame grids" : asClip ? "video clips" : "still images";
- const reading = grids
- ? "\n\nEach is a grid of frames sampled evenly across the same stretch, laid out in reading"
- + " order: left to right, then top to bottom. Read each grid as time passing, and compare the"
- + " two as sequences rather than as single moments. Neither grid is one continuous camera"
- + " shot; a stretch may contain cuts."
- : "";
- const comparisonId = slot;
- const comparisonEvidence = comparisonLogForRun(input.run, root);
- const differences = await ask(`comparison:${comparisonId}`, {
- media: [referenceMedia, renderedMedia],
- instruction: `You compare two supplied ${unit} and describe their visible differences in natural language only. You are not told how either was made. Do not write code, markup, SVML, component names, or production advice.`,
- prompt: `Two ${unit} are supplied in order: one, then two. Call them one and two throughout your answer, and say which of the two each difference is in.${reading}${asClip ? "\n\nThe two clips may have different total durations or playback speeds. Compare corresponding visual stages and events rather than matching the same elapsed second, and do not treat a duration difference alone as a visual defect." : ""}${asClip ? cut?.incomplete ?? "" : ""}${scope.length === 0 ? "" : `\n\nLimit the comparison to this: ${scope}`}\n\nStart with a geometry pass before style: identify the Canvas, the outer frame or background box, and the inner text/element. For every region intended to be vertically centred, compare inner and outer centres on the Y axis and report whether it is too high or too low. Do not treat deliberate left/right bias as a defect. Check containment separately (inner content inside its outer frame; outer frame inside the Canvas), naming the overflowing edge and approximate amount. Check that the outer frame is wide and tall enough for the longest line or mark including padding, stroke, shadow and corner treatment. Treat overlap candidates as prompts for inspection only: the supplied reference or settled brief and intent decide whether an overlap or offset is correct; never recommend removing an intentional overlap merely because it exists. Distinguish intentional bleed, crop, or enter/exit motion from accidental overflow. Then describe every other visible difference between them: layout and arrangement, the position and size of each element, cropping and margins, colour, typeface, weight, letter and line spacing, alignment, outline or stroke, shadow, glow, borders and corner treatment, and anything present in one and absent from the other.${asClip ? " Also describe differences in what changes over the stretch: what appears, what leaves, in what order, and how anything moves." : ""} State plainly which differences are large enough to read as a different design and which are minor. If they are visually equivalent, say exactly that.\n\nDo not speculate about how either was produced, which one is a source, or which one is a copy. Return natural language only.`,
- });
- // What was compared, and what was seen. The record is useful content, not a workflow cursor.
- await appendLooked(comparisonEvidence, {
- at: startedAt,
- id: comparisonId,
- ...(input.run === undefined ? {} : { run: resolve(invokedFrom(), input.run) }),
- ...(shot === undefined ? {} : { shot_id: shot.shot_id }),
- ...(cut === undefined ? {} : { range: cut.record }),
- ...(input.element === undefined ? {} : { element: input.element.trim() }),
- image_path: renderedPath,
- observer,
- status: differences.status,
- scoped: scope.length > 0,
- ...(scope.length === 0 ? {} : { scope }),
- clip: asClip,
- ...(standIn === undefined ? {} : { stand_in: standIn }),
- ...(differences.status === "complete" ? { differences: differences.text } : {}),
- });
- return {
- reference_id: state.reference_id,
- observer,
- ...(shot === undefined ? {} : { shot_id: shot.shot_id }),
- ...(cut === undefined ? {} : {
- range: cut.record,
- ...(clip ? { reference_seconds: cut.referenceSeconds, rendered_seconds: cut.renderedSeconds } : {}),
- ...(asClip && cut.incomplete.length > 0 ? { incomplete_ends: cut.incomplete.trim() } : {}),
- }),
- comparison_id: comparisonId,
- compared: asClip ? "clip" : "still",
- ...(bothStill ? { both_sides_still: true } : {}),
- reference_ref: referenceMedia,
- rendered_ref: renderedMedia,
- ...(standIn === undefined ? {} : { stand_in: standIn }),
- differences,
- unresolved: differences.status === "complete" ? [] : [`comparison:${comparisonId}`],
- ...(differences.status === "pending" ? {
- record_with: `record_observation --reference-id ${state.reference_id} `
- + `--key comparison:${comparisonId} --text-file <the differences>`,
- } : {}),
- pending_observations: pending,
- };
- },
-
- // The `agent` observer answers in its own context, so the answer comes back through here rather
- // than through a return value. The four whole-reference observations live in the state and the
- // rest in the observation cache, which is where each one was already read from.
- async record_observation(input): Promise<Record<string, unknown>> {
- if (input.answers !== undefined) {
- const round = input.answers;
- assert(round.length > 0, "answers is empty");
- // Read once, before the round, so a reference that records its own answers refuses the call
- // rather than every entry in it.
- const reading = await loadState(input.reference_id);
- assert(reading.observer === "agent", `reference ${input.reference_id} is read by the ${reading.observer ?? "gemini"} observer, which records its own answers`);
- // One at a time. Every answer is a read-modify-write of one file, and `serially` below already
- // holds them apart, so a round could run at once and be correct — it would only be a queue at
- // a lock, for writes that take no time. Running them in the order they were written keeps the
- // failures in that order too.
- const { done, failures } = await runBatch(round, { concurrency: 1, gapMs: 0 }, async (one) =>
- await tools.record_observation({ reference_id: input.reference_id, ...(input.run === undefined ? {} : { run: input.run }), key: one.key, text: one.text }));
- return {
- recorded: done.length,
- failed: failures.length,
- records: done,
- ...(failures.length === 0 ? {} : { failures }),
- };
- }
- const state = await loadState(input.reference_id);
- assert(state.observer === "agent", `reference ${input.reference_id} is read by the ${state.observer ?? "gemini"} observer, which records its own answers`);
- assert(input.key !== undefined, "key is required");
- assert(input.text !== undefined, "text is required");
- const key = input.key.trim();
- assert(key.length > 0, "key is required");
- const text = input.text.trim();
- assert(text.length > 0, "text is required; an observation that saw nothing says so in words");
- const root = stateRoot(input.reference_id);
- if (WHOLE_REFERENCE_KEYS.includes(key as typeof WHOLE_REFERENCE_KEYS[number])) {
- const field = key as typeof WHOLE_REFERENCE_KEYS[number];
- await serially(root, async () => {
- const current = await readJson<ReferenceState>(join(root, "state.json")) ?? state;
- await writeJson(join(root, "state.json"), { ...current, [field]: observation("complete", text) });
- });
- return { reference_id: state.reference_id, key, stored_in: "state" };
- }
- // A comparison is answered against the entry it opened in the log rather than against the
- // observation cache: the log is what a report reads to tell an element that was looked at from one
- // that never was, and until this existed an agent-observer comparison could never leave `pending`.
- if (key.startsWith("comparison:")) {
- const comparisonId = key.slice("comparison:".length);
- const comparisonEvidence = comparisonLogForRun(input.run, root);
- const closed = await closeLooked(comparisonEvidence, comparisonId, text);
- assert(closed, `${key} names no open comparison of reference ${input.reference_id}`);
- return { reference_id: state.reference_id, key, stored_in: "comparisons" };
- }
- const shotKeys = new Set(state.shots.flatMap((shot) => [`visual:${shot.shot_id}`, `type:${shot.shot_id}`, `audio:${shot.shot_id}`, `boundary:${shot.shot_id}`, `window:${shot.shot_id}`]));
- // Narrow questions carry the encoded question itself after the shot id. That is longer than an
- // opaque token, but it is inspectable and requires no side table.
- const questionKey = /^question:([0-9a-z-]+(?:\+[0-9a-z-]+)*):(.+)$/u.exec(key);
- const knownShots = new Set(state.shots.map((shot) => shot.shot_id));
- const askedOver = questionKey?.[1]!.split("+") ?? [];
- let validEncodedQuestion = false;
- if (questionKey !== null) {
- try { validEncodedQuestion = decodeURIComponent(questionKey[2]!).trim().length > 0; }
- catch { validEncodedQuestion = false; }
- }
- const isQuestion = questionKey !== null && validEncodedQuestion && askedOver.length === 1 && askedOver.every((id) => knownShots.has(id));
- assert(shotKeys.has(key) || isQuestion, `key ${key} is not an observation of reference ${input.reference_id}`);
- // A round of answers arrives as a round of calls, and each one is a read-modify-write of this
- // file. Two of them reading before either writes is how an answer disappears with no error.
- await serially(root, async () => {
- const path = join(root, "observations.json");
- const cache = await readJson<Record<string, Observation>>(path) ?? {};
- cache[key] = observation("complete", text);
- await writeJson(path, cache);
- });
- return { reference_id: state.reference_id, key, stored_in: "observations" };
- },
-
- async review_element(input): Promise<Record<string, unknown>> {
- const runPath = resolve(invokedFrom(), input.run);
- const root = dirname(reviewLogPath(runPath));
- await ensureDir(root);
-
- if (input.reviews !== undefined) {
- const batch = input.reviews;
- assert(batch.length > 0, "reviews is empty");
- // Local work throughout — a decode per entry and no request anywhere — so it is paced by the
- // machine, the way `render_element` is.
- const { done, failures } = await runBatch(batch, locallyPaced(batch.length),
- async (one) => await tools.review_element({ ...one, run: input.run }));
- return {
- run: runPath,
- reviewed: done.length,
- failed: failures.length,
- reviews: done,
- ...(failures.length === 0 ? {} : { failures }),
- };
- }
-
- const element = input.element?.trim() ?? "";
- assert(element.length > 0, "--element is required; a review with no element is credited to nothing");
- const named = [input.segment, input.selection, input.tokens].filter((value) => value !== undefined);
- assert(named.length === 1, "name exactly one of --segment, --selection or --tokens: the window the render covers");
- if (input.tokens !== undefined) tokenWindow(input.tokens, "tokens");
- const supplied = [input.image_path, input.video_path].filter((value) => value !== undefined);
- assert(supplied.length === 1, "supply exactly one of --image or --video");
-
- const intent = (input.intent ?? (input.intent_file === undefined
- ? undefined
- : await readFile(resolve(invokedFrom(), input.intent_file), "utf8")))?.trim() ?? "";
- assert(intent.length > 0,
- "--intent-file is required: what this element was asked to be, in the words the brief was taken in. "
- + "Without it the question is whether the picture looks acceptable, which it always does to whoever drew it.");
-
- const renderedPath = resolve(invokedFrom(), input.video_path ?? input.image_path!);
- await stat(renderedPath).catch(() => { throw new Error(`cannot read ${renderedPath}`); });
- const slot = eventId(basename(renderedPath).replace(/\.[^.]+$/u, ""));
- const standIn = await standInBeside(renderedPath);
- const scope = input.question?.trim() ?? "";
- const startedAt = new Date().toISOString();
-
- // The reader looks at pictures, so a clip becomes a grid of its own frames. With no reference
- // beside it there is no second side to match, so the render's own duration and width decide the
- // sampling — and its own stillness decides whether there is anything to sample at all.
- let picture = renderedPath;
- let asClip = false;
- if (input.video_path !== undefined) {
- const drawn = await probe(renderedPath);
- asClip = !(await completelyStill(renderedPath));
- const target = join(root, "reviews", slot, asClip ? "element.jpg" : "element-still.jpg");
- await ensureDir(dirname(target));
- picture = asClip
- ? await shotTile(renderedPath, drawn.duration, target, Math.min(480, drawn.width))
- : await cutFrame(renderedPath, drawn.duration / 2, target);
- }
-
- const range = {
- ...(input.segment === undefined ? {} : { segment: input.segment }),
- ...(input.selection === undefined ? {} : { selection: input.selection }),
- ...(input.tokens === undefined ? {} : { tokens: input.tokens }),
- };
- const reviewId = slot;
- const unit = asClip ? "grid of frames sampled evenly across the stretch, laid out in reading order: left to right, then top to bottom" : "still picture";
- const record: ReviewRecord = {
- at: startedAt, id: reviewId, element,
- range,
- image_path: renderedPath, status: "pending",
- ...(scope.length === 0 ? {} : { scope }), clip: asClip,
- ...(standIn === undefined ? {} : { stand_in: standIn }),
- intent,
- };
- await appendLooked(reviewLog(root), record);
-
- return {
- run: runPath, element,
- ...(input.segment === undefined ? {} : { segment: input.segment }),
- ...(input.selection === undefined ? {} : { selection: input.selection }),
- review_id: reviewId,
- reviewed: asClip ? "clip" : "still",
- ...(standIn === undefined ? {} : { stand_in: standIn }),
- pending_reviews: [{
- key: `review:${reviewId}`,
- instruction: "You are given one picture of a rendered video element and a written description of what it"
- + " was asked to be. Report whether the picture matches the description, and every visible defect."
- + " You are not told how it was produced. Do not write code, markup, SVML, component names, or"
- + " production advice.",
- prompt: `The picture is one ${unit}.\n\n`
- + `What this element was asked to be:\n\n${intent}\n\n`
- + (scope.length === 0 ? "" : `Read only this part of the picture: ${scope}\n\n`)
- // Said by the tool rather than left to be retyped each round: on this route nobody is
- // billed to look and the mocks are not the author's business, so there is no reason to
- // make it something a caller can forget.
- + "Flat-filled regions stand in for pictures that have not been generated yet. Ignore what they"
- + " contain and read only what is drawn over them.\n\n"
- + "Answer two things, each under its own heading.\n\n"
- + "Matches the description: state which parts of the description the picture satisfies and which"
- + " it does not. Name anything the description asks for that is not there, and anything present"
- + " that the description does not mention.\n\n"
- + "Visible defects: begin with a geometry pass. Identify Canvas, outer frame/background, and"
- + " inner text/element. For centred regions report the inner-to-outer centre offset and direction."
- + " Check inner-content containment inside the outer frame, outer-frame containment inside the"
- + " Canvas, and whether the frame has enough width/height for the longest content plus padding,"
- + " stroke, shadow and corner treatment. Report each overflowing edge and approximate amount;"
- + " distinguish intentional bleed/crop or enter/exit motion from accidental overflow. Then report"
- + " each other defect you can see, and say it is absent when you cannot: anything cut off by the"
- + " edge of the frame or reaching past it; anything overlapping or covering something else that"
- + " is meant to be read; anything whose position or alignment against the edges of the frame"
- + " looks unintended; and text or marks too low in contrast against what is behind them to read."
- + (asClip ? " Anything that appears, moves or leaves in a way the description does not account for." : "")
- + "\n\nMeasure what you can against the frame's own width and height rather than in pixels."
- + " Return natural language only.",
- image_refs: [picture],
- }],
- record_with: `record_review --run ${runPath} --review-id ${reviewId} --text-file <the findings>`,
- unresolved: [`review:${reviewId}`],
- };
- },
-
- // The findings come back out of band, the way an observation's do: whoever looked at the picture
- // did it somewhere else and returns text. This is where that text lands, and until it does the
- // review has been drawn and handed over with nobody having said what it shows.
- async record_review(input): Promise<Record<string, unknown>> {
- const runPath = resolve(invokedFrom(), input.run);
- const root = dirname(reviewLogPath(runPath));
- const id = input.review_id.trim().replace(/^review:/u, "");
- assert(id.length > 0, "review_id is required");
- const text = input.text.trim();
- assert(text.length > 0, "text is required; a review that found nothing wrong says so in words");
- const closed = await closeLooked(reviewLog(root), id, text);
- assert(closed, `${id} names no open review of ${runPath}`);
- return { run: runPath, review_id: id, stored_in: reviewLogPath(runPath) };
- },
-
- // A media slot declared as a generation is realized through the compiled preview-mock Graph path;
- // no Build or hand-written media fixture is used here.
- async render_element(input): Promise<Record<string, unknown>> {
- if (input.renders !== undefined) {
- const round = input.renders;
- assert(round.length > 0, "renders is empty");
- // Local work rather than a quota, so it is paced by the machine and not by the launch gap.
- // Every entry shares the same preview-mock stage and frame cache. Keep the realization
- // serialized so one worker cannot read a half-written preview.svrun while another is staging it;
- // `once` still makes the actual full-program draw happen only once for the round.
- const { done, failures } = await runBatch(round, { concurrency: 1, gapMs: 0 }, async (one) => await renderElement({
- ...one, run: one.run ?? input.run,
- ...(one.reference_id ?? input.reference_id === undefined ? {} : { reference_id: input.reference_id }),
- } as RenderElementInput));
- return {
- rendered: done.length,
- failed: failures.length,
- renders: done,
- ...(failures.length === 0 ? {} : { failures }),
- };
- }
- return await renderElement(input);
- },
-
- async render_previews(input): Promise<Record<string, unknown>> {
- return await renderPreviews(input);
- },
-
- async preview_check(input): Promise<Record<string, unknown>> {
- return await previewCheck(input, { packageRoot });
- },
-
- async layout_check(input): Promise<Record<string, unknown>> {
- return await layoutCheck(input);
- },
-
- async layout_accept(input): Promise<Record<string, unknown>> {
- return await layoutAccept(input);
- },
-
- async reconstruction_check(input): Promise<Record<string, unknown>> {
- return await authoringCheck({ ...input, mode: "reconstruction" }, { packageRoot });
- },
-
- async authoring_check(input): Promise<Record<string, unknown>> {
- return await authoringCheck({ ...input, mode: "description" }, { packageRoot });
- },
-
- };
- return tools;
-}
-
-const WHOLE_REFERENCE_KEYS = ["people_and_product", "voices", "persistent_systems", "places"] as const;
-
-function vocabularyForResult(value: SurfaceVocabulary | undefined): unknown {
- if (value === undefined) return undefined;
- return {
- ...value,
- ...(value.preview === undefined ? {} : {
- preview: { mediaType: value.preview.mediaType, path: value.preview.path },
- }),
- };
-}
-
-function withoutPreview(value: SurfaceVocabulary | undefined): SurfaceVocabulary | undefined {
- if (value === undefined) return undefined;
- const { preview: _preview, ...rest } = value;
- return rest;
-}
-
-function readmePath(
- specifier: string,
- root: string,
- loading: Parameters<typeof loadNodePackageSelection>[2] = {},
-): string | undefined {
- try {
- const located = locateNodePackage(specifier, {
- from: join(root, "__hypit_reference_tools__.mjs"),
- workspaceRoots: [root],
- ...(loading.fallbackRoots === undefined ? {} : { distributionRoots: loading.fallbackRoots }),
- });
- return join(located.root, "README.md");
- } catch { return undefined; }
-}
diff --git a/packages/reference-video-tools/src/transcript.ts b/packages/reference-video-tools/src/transcript.ts
deleted file mode 100644
index 43090ce5..00000000
--- a/packages/reference-video-tools/src/transcript.ts
+++ /dev/null
@@ -1,126 +0,0 @@
-import { access } from "node:fs/promises";
-import { join } from "node:path";
-import { MemoryResourceStore } from "@hypit/driver-node";
-import type { Need, StoredValue } from "@hypit/protocol";
-import type { ResourceStore } from "@hypit/runtime";
-import { sealSpeechEvidenceAudio } from "@hypit/speech";
-import { speechEvidenceTypes } from "@hypit/speech-evidence";
-import type { AlignedTranscriptEvidence } from "@hypit/speech-evidence";
-import { whisperXCapabilities, whisperXRequestForEvidenceAudio } from "@hypit/whisperx";
-import type { WhisperXLanguage } from "@hypit/whisperx";
-
-import { assert, command, readBytes, round, writeJson } from "./media.js";
-import type { Transcript, TranscriptFile, TranscriptPassage, TranscriptWord } from "./types.js";
-
-/**
- * One immediate Need executed through the selected Runtime Profile, outside any Build. The Runtime
- * host provides it; this module never chooses or instantiates a Provider itself, so whichever
- * Endpoint the Profile binds to `whisperx-alignment` — HypiHub, a local service, anything else —
- * measures the transcript.
- */
-export type InvokeNeed = (need: Need, resources: ResourceStore) => Promise<{ readonly value: StoredValue }>;
-
-// WhisperX measures in samples of the canonical evidence rate and refuses audio of any other shape,
-// so this is both what the audio is extracted at and what its word times are divided by.
-const EVIDENCE_SAMPLE_RATE = 16_000;
-
-export async function extractSpeechAudio(videoPath: string, target: string): Promise<string> {
- await command("ffmpeg", ["-hide_banner", "-loglevel", "error", "-y", "-i", videoPath, "-map", "0:a:0", "-vn", "-ac", "1", "-ar", String(EVIDENCE_SAMPLE_RATE), "-c:a", "pcm_s16le", "-bitexact", target], 300_000);
- return target;
-}
-
-// The Provider checks that the declared sample count is exactly the one in the file, so the count is
-// read out of the file rather than derived from a duration that rounds.
-function sampleFrames(bytes: Uint8Array): number {
- const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
- let offset = 12;
- while (offset + 8 <= bytes.byteLength) {
- const size = view.getUint32(offset + 4, true);
- if (String.fromCharCode(...bytes.subarray(offset, offset + 4)) === "data") return size / 2;
- offset += 8 + size + (size % 2);
- }
- throw new Error("extracted speech audio has no WAV data chunk");
-}
-
-function seconds(sample: number | undefined): number | undefined {
- return sample === undefined ? undefined : round(sample / EVIDENCE_SAMPLE_RATE);
-}
-
-// A refused connection arrives as `fetch failed` and says where only in its cause, which is the one
-// thing the reader of an unavailable transcript needs.
-function reason(error: unknown): string {
- if (!(error instanceof Error)) return String(error);
- return error.cause === undefined ? error.message : `${error.message}: ${reason(error.cause)}`;
-}
-
-/** Ask the Profile's WhisperX alignment Endpoint for the word times of one canonical evidence WAV. */
-export async function transcribeSpeechAudio(
- audioPath: string,
- language: WhisperXLanguage,
- invoke: InvokeNeed,
-): Promise<readonly TranscriptPassage[]> {
- const bytes = await readBytes(audioPath);
- const resources = new MemoryResourceStore();
- const artifact = await resources.put(bytes, "audio/wav");
- const evidence = sealSpeechEvidenceAudio({ artifact, sampleFrames: sampleFrames(bytes) });
- const need: Need = {
- id: "need:reference-video-transcript",
- capability: whisperXCapabilities.alignment,
- returns: speechEvidenceTypes.alignedTranscript,
- constraints: whisperXRequestForEvidenceAudio(evidence, { language }),
- result: "record:reference-video-transcript",
- };
- const fulfillment = await invoke(need, resources);
- assert(fulfillment.value.kind === "inline", "WhisperX returned alignment evidence by reference");
- const aligned = fulfillment.value.value as unknown as AlignedTranscriptEvidence;
- return aligned.passages.map((passage): TranscriptPassage => {
- const words = passage.words.map((word): TranscriptWord => {
- const start = seconds(word.startSample);
- const end = seconds(word.endSampleExclusive);
- return {
- text: word.text,
- ...(start === undefined ? {} : { start_seconds: start }),
- ...(end === undefined ? {} : { end_seconds: end }),
- ...(word.score === undefined ? {} : { score: word.score }),
- };
- });
- const start = seconds(passage.startSample);
- const end = seconds(passage.endSampleExclusive);
- return {
- text: words.map((word) => word.text).join(" "),
- ...(start === undefined ? {} : { start_seconds: start }),
- ...(end === undefined ? {} : { end_seconds: end }),
- words,
- };
- });
-}
-
-/**
- * The verbatim transcript is deterministic evidence rather than an observation: it is never cached
- * with the Gemini answers and never sent to Gemini. A Profile whose alignment Endpoint cannot answer
- * still prepares everything else, and is told what did not answer instead of losing the preparation.
- */
-export async function prepareTranscript(
- reference: string,
- videoPath: string,
- root: string,
- hasAudio: boolean,
- language: WhisperXLanguage,
- redo: boolean,
- invoke: InvokeNeed,
-): Promise<Transcript> {
- if (!hasAudio) return { status: "unavailable", transcript_ref: null, word_count: 0, reason: "the reference video has no audio track" };
- try {
- const audioPath = join(root, "speech.wav");
- if (redo || await access(audioPath).then(() => false, () => true)) await extractSpeechAudio(videoPath, audioPath);
- // A refused connection says only "fetch failed"; the reader still has to learn what to start.
- const passages = await transcribeSpeechAudio(audioPath, language, invoke)
- .catch((error: unknown) => { throw new Error(`WhisperX alignment did not answer: ${reason(error)}`); });
- const transcriptPath = join(root, "transcript.json");
- const file: TranscriptFile = { reference_id: reference, audio_ref: audioPath, passages };
- await writeJson(transcriptPath, file);
- return { status: "complete", transcript_ref: transcriptPath, word_count: passages.reduce((total, passage) => total + passage.words.length, 0) };
- } catch (error) {
- return { status: "unavailable", transcript_ref: null, word_count: 0, reason: reason(error) };
- }
-}
diff --git a/packages/reference-video-tools/src/types.ts b/packages/reference-video-tools/src/types.ts
deleted file mode 100644
index be8da9fe..00000000
--- a/packages/reference-video-tools/src/types.ts
+++ /dev/null
@@ -1,127 +0,0 @@
-/**
- * `pending` is not a failure: it is an observation the `agent` observer has been handed and has not
- * answered yet. It is never cached, so the work survives the process that reported it.
- */
-export type Observation = { readonly status: "complete" | "failed" | "pending"; readonly text: string };
-
-export type TranscriptWord = {
- readonly text: string;
- readonly start_seconds?: number;
- readonly end_seconds?: number;
- readonly score?: number;
-};
-
-export type TranscriptPassage = {
- readonly text: string;
- readonly start_seconds?: number;
- readonly end_seconds?: number;
- readonly words: readonly TranscriptWord[];
-};
-
-export type TranscriptFile = {
- readonly reference_id: string;
- readonly audio_ref: string;
- readonly passages: readonly TranscriptPassage[];
-};
-
-// A reference of any length has thousands of words, so the words live in their own file and the
-// prepared result carries the path to them, as it does for the storyboard and the shot media.
-export type Transcript = {
- readonly status: "complete" | "unavailable";
- readonly transcript_ref: string | null;
- readonly word_count: number;
- readonly reason?: string;
-};
-
-export type Shot = {
- readonly shot_id: string;
- readonly index: number;
- readonly start_seconds: number;
- readonly end_seconds: number;
- readonly duration_seconds: number;
- readonly initial_group: number;
- readonly part: number;
- readonly parts: number;
- readonly clip_ref: string;
- readonly representative_frame_ref: string;
- readonly tail_frame_ref: string;
- /**
- * The shot's frames sampled evenly and tiled into one picture, in reading order. Built for the
- * `agent` observer, which reads it in place of the clip, and null for the observer that reads video.
- */
- readonly frames_tile_ref: string | null;
- readonly audio_tail_ref: string | null;
-};
-
-/**
- * Who reads the reference. `gemini` uploads video to the configured Gemini backend; `agent` hands the main agent one tiled
- * picture per shot and takes the answer back. The two produce the same observation keys.
- */
-export type Observer = "gemini" | "agent";
-
-/** One observation the `agent` observer still owes, carrying everything needed to answer it. */
-export type ObservationTaskRequest = {
- readonly key: string;
- readonly instruction: string;
- readonly prompt: string;
- readonly image_refs: readonly string[];
- /**
- * The measured words, for a task whose question needs sound.
- *
- * WhisperX runs locally on both paths, so the words and their times are the one piece of evidence
- * this observer has that its pictures do not carry, and the prompt for such a task tells it to
- * answer from the two together. Naming the file here is what makes that instruction followable:
- * `transcript_words` is what the transcript places inside this task's own stretch, and
- * `transcript_ref` is the whole file for a question asked over the whole reference.
- */
- readonly transcript_ref?: string;
- readonly transcript_words?: string;
- /**
- * The command line that closes this task, ready to run.
- *
- * A sweep hands out four tasks per shot, so a reference of any length produces more of these than
- * anyone will match against a usage string by hand. Carrying the command here is what keeps the
- * choice of recording verb off the caller entirely: every task says how it is answered, the way a
- * comparison and a review already do.
- */
- readonly record_with: string;
-};
-
-export type ReferenceState = {
- readonly reference_id: string;
- readonly video_path: string;
- readonly root: string;
- readonly observer?: Observer;
- readonly video: { readonly duration_seconds: number; readonly width: number; readonly height: number; readonly has_audio: boolean };
- readonly shots: readonly Shot[];
- readonly storyboard_ref: string;
- readonly analysis_video_ref: string;
- readonly transcript?: Transcript;
- readonly people_and_product?: Observation;
- readonly voices?: Observation;
- readonly persistent_systems?: Observation;
- readonly places?: Observation;
-};
-
-export type PrepareResult = {
- readonly reference_id: string;
- readonly status: "ready" | "partial";
- readonly observer?: Observer;
- /** Observations the `agent` observer still owes. Empty on the `gemini` observer. */
- readonly pending_observations?: readonly ObservationTaskRequest[];
- readonly video: ReferenceState["video"];
- readonly shots: readonly Shot[];
- readonly storyboard_ref: string;
- readonly transcript: Transcript;
- /** Canonical whole-reference observation envelope. Legacy direct fields remain for compatibility. */
- readonly observations?: Readonly<{
- readonly people_and_product: Observation;
- readonly voices: Observation;
- readonly persistent_systems: Observation;
- readonly places: Observation;
- }>;
- readonly people_and_product: Observation;
- readonly voices: Observation;
- readonly persistent_systems: Observation;
- readonly places: Observation;
-};
diff --git a/packages/reference-video-tools/test/plan.test.ts b/packages/reference-video-tools/test/plan.test.ts
deleted file mode 100644
index 6069c83b..00000000
--- a/packages/reference-video-tools/test/plan.test.ts
+++ /dev/null
@@ -1,84 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-
-import { scriptBody } from "../src/authoring.js";
-import { reviewPlan } from "../src/plan.js";
-
-const SVML = `<?svml using="@hypit/markup@1"?>
-<svml version="1">
- <import as="story" from="@hypit/script@1"/>
- <import as="media-track" from="@hypit/media-track@1"/>
- <script id="story">
- <one><HOST>alpha bravo @claim charlie delta @/claim echo.</one>
- <two><HOST>foxtrot @proof golf hotel @/proof india.</two>
- </script>
- <media-track:Track id="broll" semantic={speech.semantic} canvas={vertical}>
- <media-track:Item id="one" media={a.media} during={story.selection.claim}
- frame={card-a} appearance={recipes.media.card}/>
- <media-track:Item id="two" media={b.media} during={story.selection.proof}
- frame={card-b} appearance={recipes.media.wide}/>
- </media-track:Track>
-</svml>
-`;
-
-const BROLL = [{ id: "broll", tag: "Track", alias: "media-track", specifier: "@hypit/media-track" }];
-
-function planFor(svml: string) {
- return reviewPlan({ svml, svmlPath: "main.svml", scriptBody: scriptBody(svml), drawn: BROLL });
-}
-
-// Under-covering is the silent direction: the plan looks complete, the round runs, and the second design is
-// simply never drawn. The two exclusion lists this key is built from will grow as packages are added,
-// and widening one by an attribute that actually decides what a picture looks like merges two groups
-// into one. That edit has no visible symptom, so it is the one worth pinning.
-test("two placements that declare different pictures are two looks, and two that declare the same are one", () => {
- const distinct = planFor(SVML);
- assert.equal(distinct.length, 2, "a different appearance and Frame is a different picture");
- assert.deepEqual(distinct.map((entry) => entry.named), ["selection claim", "selection proof"]);
-
- // Only the declaration differs between these two runs. The windows, the ids and the media stay put.
- const same = planFor(SVML
- .replace("appearance={recipes.media.wide}", "appearance={recipes.media.card}")
- .replace("frame={card-b}", "frame={card-a}"));
- assert.equal(same.length, 1, "the same declaration over different words is one design, read once");
-
- // The material each Item names is mocked in the render this plan is for, so two Items differing
- // only in which unbuilt picture they point at draw the identical placeholder.
- const mocked = planFor(SVML
- .replace("appearance={recipes.media.wide}", "appearance={recipes.media.card}")
- .replace("frame={card-b}", "frame={card-a}")
- .replace("media={b.media}", "media={c.media}"));
- assert.equal(mocked.length, 1, "an unbuilt picture is a grey rectangle whichever generation names it");
-});
-
-const CAPTIONS = `<?svml using="@hypit/markup@1"?>
-<svml version="1">
- <import as="story" from="@hypit/script@1"/>
- <import as="caption" from="@hypit/caption@1"/>
- <import as="caption-fine" from="@hypit/caption-fine@1"/>
- <script id="story">
- <one><HOST>alpha bravo charlie delta echo.</one>
- <two><HOST>foxtrot @punch golf hotel @/punch india.</two>
- </script>
- <caption-fine:Style id="plain" recipe={recipes.caption.plain}/>
- <caption-fine:Style id="loud" recipe={recipes.caption.loud}/>
- <caption:Program id="program" document={story.caption} narrative={story} default={plain}>
- <caption:Use style={loud} selection={story.selection.punch}/>
- </caption:Program>
- <caption-fine:Track id="captions" document={story.caption} semantic={speech.semantic} program={program}/>
-</svml>
-`;
-
-// A caption Track is one tag with no window on it, so reading its placements the ordinary way finds
-// one design however many the Program hands out. An override is written inside the Program, and the
-// stretch it covers is a genuinely different picture that nothing on the Track mentions — so leaving
-// it out is a design nobody looks at while the plan reports full coverage.
-test("a caption Style override is its own look, over the stretch it covers", () => {
- const plan = reviewPlan({
- svml: CAPTIONS, svmlPath: "main.svml", scriptBody: scriptBody(CAPTIONS),
- drawn: [{ id: "captions", tag: "Track", alias: "caption-fine", specifier: "@hypit/caption-fine" }],
- });
- const overridden = plan.find((entry) => entry.named === "selection punch");
- assert.ok(overridden, `the overridden stretch is missing; got ${JSON.stringify(plan.map((e) => e.named))}`);
- assert.ok(plan.length >= 2, "the default and the override are two designs, not one");
-});
diff --git a/packages/reference-video-tools/test/tools.test.ts b/packages/reference-video-tools/test/tools.test.ts
deleted file mode 100644
index b924066a..00000000
--- a/packages/reference-video-tools/test/tools.test.ts
+++ /dev/null
@@ -1,270 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-import { mkdtemp, mkdir, writeFile, readFile } from "node:fs/promises";
-import { tmpdir } from "node:os";
-import { join } from "node:path";
-
-import { createReferenceVideoTools } from "../src/tools.js";
-import type { GenerateText } from "../src/tools.js";
-import type { ReferenceState } from "../src/types.js";
-
-type Call = { readonly instruction: string; readonly mimeTypes: readonly string[]; readonly text: string };
-
-const REFERENCE = "ref-fixture";
-
-async function workspace(shotCount: number): Promise<{ readonly root: string; readonly stateRoot: string }> {
- const root = await mkdtemp(join(tmpdir(), "reference-video-tools-"));
- // Prepared state is read from the Hypit tree, which HYPIT_REPOSITORY names. A fixture puts the tree
- // in a directory of its own so the state it writes is the state the tools read.
- process.env["HYPIT_REPOSITORY"] = root;
- const stateRoot = join(root, ".hypit", "reference-video-tools", REFERENCE);
- const shots = join(stateRoot, "shots");
- await mkdir(shots, { recursive: true });
- const media: ReferenceState["shots"][number][] = [];
- for (let index = 0; index < shotCount; index += 1) {
- const id = String(index + 1).padStart(3, "0");
- const clip = join(shots, `${id}.mp4`);
- const representative = join(shots, `${id}-representative.jpg`);
- const tail = join(shots, `${id}-tail.jpg`);
- const framesTile = join(shots, `${id}-frames.jpg`);
- for (const path of [clip, representative, tail, framesTile]) await writeFile(path, `bytes-${id}`, "utf8");
- media.push({
- shot_id: `shot-${id}`,
- index: index + 1,
- start_seconds: index,
- end_seconds: index + 1,
- duration_seconds: 1,
- initial_group: index,
- part: 1,
- parts: 1,
- clip_ref: clip,
- representative_frame_ref: representative,
- tail_frame_ref: tail,
- frames_tile_ref: framesTile,
- audio_tail_ref: null,
- });
- }
- const state: ReferenceState = {
- reference_id: REFERENCE,
- video_path: join(root, "reference.mp4"),
- root: stateRoot,
- video: { duration_seconds: shotCount, width: 360, height: 640, has_audio: true },
- shots: media,
- storyboard_ref: join(stateRoot, "storyboard.jpg"),
- analysis_video_ref: join(stateRoot, "analysis.mp4"),
- people_and_product: { status: "complete", text: "one recurring host" },
- voices: { status: "complete", text: "one voice" },
- persistent_systems: { status: "complete", text: "one caption system for the whole video" },
- places: { status: "complete", text: "one room, one camera position" },
- };
- await writeFile(join(stateRoot, "state.json"), `${JSON.stringify(state, null, 2)}\n`, "utf8");
- await writeFile(join(stateRoot, "observations.json"), "{}\n", "utf8");
- return { root, stateRoot };
-}
-
-function recorder(calls: Call[], answer = "observed"): GenerateText {
- return async ({ parts, instruction }) => {
- calls.push({
- instruction,
- mimeTypes: parts.flatMap((part) => part.inlineData?.mimeType === undefined ? [] : [part.inlineData.mimeType]),
- text: parts.flatMap((part) => part.text === undefined ? [] : [part.text]).join("\n"),
- });
- return answer;
- };
-}
-
-function tools(calls: Call[], answer?: string) {
- return createReferenceVideoTools({ concurrency: 4, launchGapMs: 0, retryDelayMs: 0, generate: recorder(calls, answer) });
-}
-
-test("installed packages can be listed, and one that will not load is reported rather than hidden", async () => {
- const root = await mkdtemp(join(tmpdir(), "reference-video-tools-scope-"));
- const scope = join(root, "node_modules", "@hypit");
- await mkdir(join(scope, "with-activation"), { recursive: true });
- await mkdir(join(scope, "plain-library"), { recursive: true });
- await writeFile(join(scope, "with-activation", "package.json"),
- JSON.stringify({ name: "@hypit/with-activation", description: "declares a Surface", hypit: { activation: "./src/activation.ts" } }), "utf8");
- await writeFile(join(scope, "plain-library", "package.json"),
- JSON.stringify({ name: "@hypit/plain-library" }), "utf8");
-
- const result = await createReferenceVideoTools({ packageRoot: root, generate: async () => "" }).list_svml_packages();
- const packages = result["packages"] as readonly Record<string, unknown>[];
- const named = (name: string) => packages.find((item) => item["package_name"] === name);
- // The listing also reaches the Distribution, since a project holds its own packages and not the
- // installed ones, so this asks about the two written above rather than about the whole answer.
- assert.equal(named("@hypit/plain-library"), undefined,
- "a package without an activation contributes no author vocabulary and is not vocabulary to discover");
- const declared = named("@hypit/with-activation");
- assert.ok(declared, "a package that declares an activation is vocabulary to discover");
- assert.equal(declared["description"], "declares a Surface");
- assert.equal(typeof declared["unreadable"], "string",
- "a package that declares an activation it cannot load is named, not silently dropped");
- assert.equal(declared["tags"] !== undefined
- && (declared["tags"] as readonly unknown[]).includes(null), false,
- "a facet that is not a Markup Surface has no tag, and reading one out of it produced a null entry");
-});
-
-test("observation covers picture, drawn type and sound for every shot and caches every key", async () => {
- const { stateRoot } = await workspace(2);
- const calls: Call[] = [];
- const result = await tools(calls).observe_reference({ reference_id: REFERENCE });
-
- const cache = JSON.parse(await readFile(join(stateRoot, "observations.json"), "utf8")) as Record<string, unknown>;
- assert.deepEqual(Object.keys(cache).sort(), [
- "audio:shot-001", "audio:shot-002",
- "boundary:shot-002",
- "type:shot-001", "type:shot-002",
- "visual:shot-001", "visual:shot-002",
- ], "one cut is one observation, not a same-take question and an overlay question over the same media");
-
- const shots = result["shots"] as readonly Record<string, unknown>[];
- assert.equal(shots.length, 2);
- assert.deepEqual(Object.keys(shots[0]!).sort(), ["audio", "shot_id", "text_appearance", "visual"]);
- assert.deepEqual(result["unresolved"], []);
-
- const picture = calls.find((call) => call.instruction.startsWith("Observe picture only"))!;
- assert.match(picture.text, /does this picture move at all/u,
- "whether a stretch of picture moves decides whether it is generated or held as a still");
- assert.match(picture.text, /depicted scene/u);
- assert.match(picture.text, /flat designed field/u);
- assert.match(picture.text, /the picture inside the frame and the frame itself separately/u);
- assert.match(picture.text, /persistent on-screen system evidence/u);
-
- const type = calls.find((call) => call.instruction.startsWith("Observe the appearance of on-screen text"))!;
- assert.match(type.text, /outline or stroke/u);
- assert.match(type.text, /drop shadow/u);
- assert.match(type.text, /typeface character/u);
-});
-
-test("a second observation reuses the cache and only an explicit reobserve runs the model again", async () => {
- await workspace(2);
- const first: Call[] = [];
- await tools(first).observe_reference({ reference_id: REFERENCE });
- assert.equal(first.length, 7, "two shots: picture, type and sound each, plus one shared boundary");
-
- const second: Call[] = [];
- await tools(second).observe_reference({ reference_id: REFERENCE });
- assert.deepEqual(second, []);
-
- const selected: Call[] = [];
- await tools(selected).observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-002"] });
- assert.deepEqual(selected, [], "naming a shot must not silently re-run its cached observations");
-
- const forced: Call[] = [];
- await tools(forced).observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-002"], reobserve: true });
- assert.equal(forced.length, 4, "reobserve reruns the selected shot's picture, type and sound plus its one boundary, and nothing else");
-});
-
-test("a narrow question costs one call, answers from the named shots and is answered once", async () => {
- await workspace(4);
- const calls: Call[] = [];
- const api = tools(calls, "the caption sits above the lower edge");
- const result = await api.observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-003"], question: "How thick is the caption outline?" });
-
- assert.equal(calls.length, 1);
- assert.deepEqual(calls[0]!.mimeTypes, ["video/mp4", "image/jpeg"]);
- assert.match(calls[0]!.text, /^How thick is the caption outline\?/u);
- assert.match(calls[0]!.text, /platform UI|editing-software watermark/u);
- assert.deepEqual(result["shot_ids"], ["shot-003"]);
- assert.deepEqual(result["result"], { status: "complete", text: "the caption sits above the lower edge" });
-
- // The key carries the question as well as the shots, so a second question over the same shot is a
- // separate entry rather than the first one's answer handed back under a shared key.
- const repeated = await api.observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-003"], question: "How thick is the caption outline?" });
- assert.equal(calls.length, 1, "asking the same question again reads the answer rather than paying for a paraphrase");
- assert.equal(repeated["reused"], true);
- const different = await api.observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-003"], question: "What colour is the background?" });
- assert.equal(calls.length, 2, "a different question over the same shot is a different question");
- assert.notEqual(different["observation_key"], repeated["observation_key"]);
-
- await assert.rejects(
- tools([]).observe_reference({ reference_id: REFERENCE, question: "which shot?" }),
- /question requires at least one shot id/u);
- await assert.rejects(
- tools([]).observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-001", "shot-002"], question: "which shot?" }),
- /exactly one shot id/u);
-});
-
-test("the key a narrow question is handed out under is one record_observation accepts", async () => {
- // A shot id is `shot-003`, and the key that names it was being matched by a pattern with no hyphen
- // in its class — so every real narrow-question key was refused, and the observer that has to record
- // its own answers had nowhere to put one.
- await workspace(4);
- const api = createReferenceVideoTools({ concurrency: 4, launchGapMs: 0, retryDelayMs: 0, generate: async () => "" });
- const state = JSON.parse(await readFile(join(process.env["HYPIT_REPOSITORY"]!, ".hypit", "reference-video-tools", REFERENCE, "state.json"), "utf8")) as ReferenceState;
- await writeFile(join(state.root, "state.json"), `${JSON.stringify({ ...state, observer: "agent" }, null, 2)}\n`, "utf8");
-
- const asked = await api.observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-003"], question: "How thick is the caption outline?" });
- const key = (asked["pending_observations"] as readonly { readonly key: string }[])[0]!.key;
- const stored = await api.record_observation({ reference_id: REFERENCE, key, text: "two pixels" });
- assert.equal(stored["stored_in"], "observations");
-
- const again = await api.observe_reference({ reference_id: REFERENCE, shot_ids: ["shot-003"], question: "How thick is the caption outline?" });
- assert.deepEqual(again["result"], { status: "complete", text: "two pixels" });
-});
-
-test("reconstruction comparison sends an unlabelled pair and accepts rendered PNG frames", async () => {
- const { root } = await workspace(1);
- const rendered = join(root, "rendered.png");
- await writeFile(rendered, "rendered-bytes", "utf8");
- const calls: Call[] = [];
- const result = await tools(calls, "the list starts lower in one image")
- .compare_reconstruction({ reference_id: REFERENCE, shot_id: "shot-001", image_path: rendered, question: "only the full-screen list area" });
-
- assert.equal(calls.length, 1);
- assert.deepEqual(calls[0]!.mimeTypes, ["image/jpeg", "image/png"]);
- assert.match(calls[0]!.text, /supplied in order: one, then two/u);
- assert.match(calls[0]!.text, /only the full-screen list area/u);
- assert.match(calls[0]!.text, /geometry pass/u);
- assert.match(calls[0]!.text, /outer frame/u);
- assert.match(calls[0]!.text, /overflow/u);
- assert.doesNotMatch(calls[0]!.text, /reconstruction|rendered|generated|component|SVML/iu,
- "the comparison must never say which image was built or how");
- assert.match(calls[0]!.instruction, /You are not told how either was made/u);
- assert.deepEqual(result["differences"], { status: "complete", text: "the list starts lower in one image" });
-
- await assert.rejects(
- tools([]).compare_reconstruction({ reference_id: REFERENCE, shot_id: "shot-009", image_path: rendered }),
- /shot shot-009 does not exist/u);
-});
-
-test("an unsupported media extension fails loudly instead of being sent as opaque bytes", async () => {
- const { root } = await workspace(1);
- const rendered = join(root, "rendered.tiff");
- await writeFile(rendered, "rendered-bytes", "utf8");
- await assert.rejects(
- tools([]).compare_reconstruction({ reference_id: REFERENCE, shot_id: "shot-001", image_path: rendered }),
- /unsupported media extension/u);
-});
-
-test("a rejected request fails once instead of being retried until the loop gives up", async () => {
- await workspace(1);
- let attempts = 0;
- const rejecting = createReferenceVideoTools({
- concurrency: 4,
- launchGapMs: 0,
- retryDelayMs: 0,
- generate: async () => {
- attempts += 1;
- throw new Error(`{"error":{"code":400,"message":"Request contains an invalid argument.","status":"INVALID_ARGUMENT"}}`);
- },
- });
- const result = await rejecting.observe_reference({ reference_id: REFERENCE });
- assert.equal(attempts, 3, "one shot has three observations, and each one must give up after a single rejected request");
- assert.deepEqual([...(result["unresolved"] as readonly string[])].sort(), ["audio:shot-001", "type:shot-001", "visual:shot-001"]);
-});
-
-test("a failed observation is reported as unresolved instead of an empty list", async () => {
- await workspace(1);
- const failing = createReferenceVideoTools({
- concurrency: 4,
- launchGapMs: 0,
- retryDelayMs: 0,
- generate: async ({ instruction }) => {
- if (instruction.startsWith("Observe the appearance of on-screen text")) throw new Error("model refused");
- return "observed";
- },
- });
- const result = await failing.observe_reference({ reference_id: REFERENCE });
- assert.deepEqual(result["unresolved"], ["type:shot-001"]);
-});
diff --git a/packages/reference-video-tools/test/transcript.test.ts b/packages/reference-video-tools/test/transcript.test.ts
deleted file mode 100644
index f2ee1d22..00000000
--- a/packages/reference-video-tools/test/transcript.test.ts
+++ /dev/null
@@ -1,122 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-import { mkdtemp, readFile, writeFile } from "node:fs/promises";
-import { tmpdir } from "node:os";
-import { join } from "node:path";
-
-import { canonicalize } from "@hypit/protocol";
-import { sealAlignedTranscriptEvidence } from "@hypit/speech-evidence";
-import { interpretWhisperXTranscript } from "@hypit/whisperx";
-import type { WhisperXAlignmentRequest, WhisperXTranscriptResponse } from "@hypit/whisperx";
-
-import { prepareTranscript, transcribeSpeechAudio } from "../src/transcript.js";
-import type { InvokeNeed } from "../src/transcript.js";
-import type { TranscriptFile } from "../src/types.js";
-
-const REFERENCE = "ref-fixture";
-
-/** The canonical evidence WAV the Provider refuses to normalize a second time: 16 kHz mono s16. */
-function wav(sampleFrames: number): Uint8Array {
- const bytes = new Uint8Array(44 + sampleFrames * 2);
- const view = new DataView(bytes.buffer);
- const write = (offset: number, value: string): void => {
- for (let index = 0; index < value.length; index += 1) bytes[offset + index] = value.charCodeAt(index);
- };
- write(0, "RIFF");
- view.setUint32(4, bytes.byteLength - 8, true);
- write(8, "WAVE");
- write(12, "fmt ");
- view.setUint32(16, 16, true);
- view.setUint16(20, 1, true);
- view.setUint16(22, 1, true);
- view.setUint32(24, 16_000, true);
- view.setUint32(28, 32_000, true);
- view.setUint16(32, 2, true);
- view.setUint16(34, 16, true);
- write(36, "data");
- view.setUint32(40, sampleFrames * 2, true);
- return bytes;
-}
-
-/**
- * The Runtime host's one-shot invocation, answered with a WhisperX-shaped response. These tests
- * exercise request shaping and transcript parsing; which Provider the Profile binds is the Runtime's
- * business and never this module's.
- */
-function answering(response: WhisperXTranscriptResponse): InvokeNeed {
- return async (need) => {
- assert.equal(need.capability.name, "whisperx-alignment");
- const request = need.constraints as unknown as WhisperXAlignmentRequest;
- assert.equal(request.audio.mediaType, "audio/wav");
- assert.equal(request.language, "en");
- return {
- value: {
- kind: "inline",
- value: canonicalize(sealAlignedTranscriptEvidence({
- passages: interpretWhisperXTranscript(response, request.sampleFrames),
- })),
- },
- };
- };
-}
-
-const unanswered: InvokeNeed = async () => {
- throw new Error("no Endpoint in the selected Runtime Profile serves @hypit/whisperx#whisperx-alignment");
-};
-
-test("every word carries its own start and end in seconds, not only the passage around it", async () => {
- const root = await mkdtemp(join(tmpdir(), "reference-video-transcript-"));
- const audioPath = join(root, "speech.wav");
- await writeFile(audioPath, wav(32_000));
-
- const passages = await transcribeSpeechAudio(audioPath, "en", answering({
- language: "en",
- segments: [{ start: 0, end: 2, words: [
- { text: "number", start: 0.1, end: 0.4, score: 0.9 },
- { text: "five", start: 1.2, end: 1.6 },
- ] }],
- }));
- assert.equal(passages.length, 1);
- assert.equal(passages[0]!.text, "number five");
- assert.deepEqual(passages[0]!.words, [
- { text: "number", start_seconds: 0.1, end_seconds: 0.4, score: 0.9 },
- { text: "five", start_seconds: 1.2, end_seconds: 1.6 },
- ], "placing an on-screen reveal needs the time of the word, not of the sentence it sits in");
-});
-
-test("a complete transcript is written beside the other prepared artifacts and summarized by its word count", async () => {
- const root = await mkdtemp(join(tmpdir(), "reference-video-transcript-"));
- await writeFile(join(root, "speech.wav"), wav(32_000));
-
- const result = await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, true, "en", false, answering({
- language: "en",
- segments: [{ start: 0, end: 2, words: [{ text: "hello", start: 0.1, end: 0.4 }, { text: "world", start: 1.2, end: 1.6 }] }],
- }));
- assert.deepEqual(result, { status: "complete", transcript_ref: join(root, "transcript.json"), word_count: 2 });
-
- const file = JSON.parse(await readFile(join(root, "transcript.json"), "utf8")) as TranscriptFile;
- assert.equal(file.reference_id, REFERENCE);
- assert.equal(file.audio_ref, join(root, "speech.wav"));
- assert.equal(file.passages[0]!.words.length, 2);
-});
-
-test("an alignment Endpoint that cannot answer leaves the transcript unavailable with its reason instead of failing preparation", async () => {
- const root = await mkdtemp(join(tmpdir(), "reference-video-transcript-"));
- assert.deepEqual(
- await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, false, "en", false, unanswered),
- { status: "unavailable", transcript_ref: null, word_count: 0, reason: "the reference video has no audio track" },
- "a silent reference is prepared, not refused");
-
- await writeFile(join(root, "speech.wav"), wav(32_000));
- const refused = await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, true, "en", false, unanswered);
- assert.equal(refused.status, "unavailable");
- assert.match(refused.reason ?? "", /WhisperX alignment did not answer: no Endpoint in the selected Runtime Profile/u,
- "the Profile, not this tool, decides who measures the transcript; the reader learns exactly what is missing");
-
- await writeFile(join(root, "reference.mp4"), "not a video", "utf8");
- const failed = await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, true, "en", true, unanswered);
- assert.equal(failed.status, "unavailable");
- assert.equal(failed.transcript_ref, null);
- assert.equal(typeof failed.reason === "string" && failed.reason.length > 0, true,
- "the whole preparation survives, so the one thing that did not work has to say why");
-});
diff --git a/packages/script/src/cue-lint.ts b/packages/script/src/cue-lint.ts
deleted file mode 100644
index 5b6acb70..00000000
--- a/packages/script/src/cue-lint.ts
+++ /dev/null
@@ -1,69 +0,0 @@
-import type { ParsedNarrative } from "./types.js";
-import { captionDocument } from "./narrative.js";
-
-export type CaptionCueLengthViolation = {
- readonly segment: string;
- readonly cue: number;
- readonly wordCount: number;
- readonly maxWords: number;
- readonly sourceRange: { readonly start: number; readonly end: number };
- readonly message: string;
-};
-
-/**
- * Mechanical layout guard for authored caption Cues. The count is based on visible display words,
- * not punctuation or renderer line wrapping, and therefore also covers Dual Text aliases.
- */
-export function validateCaptionCueLengths(
- parsed: ParsedNarrative,
- options: { readonly maxWords?: number } = {},
-): readonly CaptionCueLengthViolation[] {
- const maxWords = options.maxWords ?? 4;
- if (!Number.isSafeInteger(maxWords) || maxWords < 1) throw new Error("maxWords must be a positive integer");
- const document = captionDocument(parsed, "__cue_lint__", parsed.segments[0]?.id ?? "__cue_lint__");
- const breaks = new Set(document.cueBreaks.map((item) => item.afterUnitId));
- const violations: CaptionCueLengthViolation[] = [];
- let cueWords = 0;
- let cueStart = 0;
- let cueOrdinal = 1;
- let cueSegment = document.units[0]?.segmentId;
- const flush = (endExclusive: number): void => {
- if (cueWords <= maxWords || document.units.length === 0) return;
- const first = document.units[cueStart];
- const last = document.units[Math.max(cueStart, endExclusive - 1)];
- const tokenIds = [...(first?.sourceTokenIds ?? []), ...(last?.sourceTokenIds ?? [])];
- const tokenById = new Map(parsed.tokens.map((token) => [token.id, token]));
- const ranges = tokenIds.map((id) => tokenById.get(id)?.range).filter((range): range is { start: number; end: number } => range !== undefined);
- const sourceRange = ranges.length === 0
- ? { start: parsed.sourceRange.start, end: parsed.sourceRange.end }
- : { start: Math.min(...ranges.map((range) => range.start)), end: Math.max(...ranges.map((range) => range.end)) };
- violations.push({
- segment: first?.segmentId ?? "unknown",
- cue: cueOrdinal,
- wordCount: cueWords,
- maxWords,
- sourceRange,
- message: "Split this Cue with ||; captions normally contain no more than four spoken words.",
- });
- };
- for (let index = 0; index < document.units.length; index += 1) {
- const unit = document.units[index]!;
- if (cueSegment !== undefined && unit.segmentId !== cueSegment) {
- // Segment close is an implicit hard boundary. Diagnostics are numbered within the
- // Segment that owns the Cue, rather than carrying an ordinal across independent takes.
- cueWords = 0;
- cueStart = index;
- cueOrdinal = 1;
- cueSegment = unit.segmentId;
- }
- cueWords += unit.wordIds.length;
- if (breaks.has(unit.id)) {
- flush(index + 1);
- cueWords = 0;
- cueStart = index + 1;
- cueOrdinal += 1;
- }
- }
- flush(document.units.length);
- return violations;
-}
diff --git a/packages/script/src/index.ts b/packages/script/src/index.ts
index 5d2d642c..f502de57 100644
--- a/packages/script/src/index.ts
+++ b/packages/script/src/index.ts
@@ -16,6 +16,5 @@ export {
serializeSpeech,
} from "./narrative.js";
export { parseScript } from "./parser.js";
-export { validateCaptionCueLengths } from "./cue-lint.js";
export { decodeScriptSurface } from "./surface.js";
export type * from "./types.js";
diff --git a/packages/script/src/surface.ts b/packages/script/src/surface.ts
index bfdc2692..096cdca8 100644
--- a/packages/script/src/surface.ts
+++ b/packages/script/src/surface.ts
@@ -17,7 +17,6 @@ import {
} from "./manifest.js";
import { textTypes } from "@hypit/text";
import { parseScript } from "./parser.js";
-import { validateCaptionCueLengths } from "./cue-lint.js";
import type { ScriptSurfaceInput, ScriptSurfaceOutput } from "./types.js";
const RECORD_ID = /^[a-z][a-z0-9_-]{0,63}$/u;
@@ -74,17 +73,6 @@ export function decodeScriptSurface(input: ScriptSurfaceInput): ScriptSurfaceOut
input.source.slice(input.contentStart, close.start),
input.contentStart,
);
- if (process.env.HYPIT_STRICT_SCRIPT_CUES === "1") {
- const cueViolation = validateCaptionCueLengths(parsed)[0];
- if (cueViolation !== undefined) {
- throw new ScriptSyntaxError(
- "SCRIPT_CUE_TOO_LONG",
- `${cueViolation.message} Cue ${cueViolation.cue} in Segment ${cueViolation.segment} has ${cueViolation.wordCount} words (maximum ${cueViolation.maxWords}).`,
- input.sourceName,
- cueViolation.sourceRange.start,
- );
- }
- }
const captionId = `${rawId}.caption`;
return {
nextOffset: close.end,
diff --git a/packages/script/test/cue-lint.test.ts b/packages/script/test/cue-lint.test.ts
deleted file mode 100644
index 9695fdad..00000000
--- a/packages/script/test/cue-lint.test.ts
+++ /dev/null
@@ -1,27 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-import { parseScript, validateCaptionCueLengths } from "../src/index.js";
-
-function parsed(body: string) { return parseScript("test.svml", body); }
-
-test("Cue lint accepts four words and rejects an unbroken five-word Cue", () => {
- const good = validateCaptionCueLengths(parsed("<one>one two three four</one>"));
- assert.deepEqual(good, []);
- const bad = validateCaptionCueLengths(parsed("<one>one two three four five</one>"));
- assert.equal(bad.length, 1);
- assert.equal(bad[0]?.wordCount, 5);
- assert.equal(bad[0]?.segment, "one");
-});
-
-test("Cue lint counts visible Dual Text words and respects authored breaks", () => {
- assert.equal(validateCaptionCueLengths(parsed("<one>one two || three four</one>")).length, 0);
- const bad = validateCaptionCueLengths(parsed("<one>one two three four five || six</one>"));
- assert.equal(bad[0]?.wordCount, 5);
- const dual = validateCaptionCueLengths(parsed("<one><one two three four five | uno dos tres cuatro cinco></one>"));
- assert.equal(dual[0]?.wordCount, 5);
-});
-
-test("Segment boundaries are hard Cue boundaries", () => {
- const violations = validateCaptionCueLengths(parsed("<one>one two three</one><two>four five</two>"));
- assert.equal(violations.length, 0);
-});
diff --git a/packages/video-cli/README.md b/packages/video-cli/README.md
index 4b878374..93cd2719 100644
--- a/packages/video-cli/README.md
+++ b/packages/video-cli/README.md
@@ -48,6 +48,30 @@ Profile and spends nothing. `--runtime <profile>` names the Profile;
otherwise the project's `hypit runtime use` selection is read. Slow paid generation — pictures and
clips — is not a creation-time tool: declare it in the Source and go through `plan` and `build`.
+Two more families are local, stateless and spend nothing. `hypit media` prepares what the eyes will
+look at when a whole video is too long or too dense to hand over at once, and `hypit vocabulary`
+prints what a Source may write:
+
+```bash
+hypit media probe reference.mp4
+hypit media cut reference.mp4 --start 12 --end 19.5 --to notes/hook.mp4
+hypit media frames reference.mp4 --at 12.4,13.1 --to notes/hook-frames
+hypit media tile reference.mp4 --start 12 --end 19.5 --to notes/hook-grid.jpg
+hypit media shots reference.mp4
+hypit media fetch https://… --to reference/source.mp4
+hypit vocabulary
+hypit vocabulary @hypit/media-pipeline --tag StillVideo
+hypit vocabulary --visual text
+```
+
+`cut` seeks to the exact frame; `frames` writes one JPEG per named second; `tile` draws four to nine
+frames of a stretch in one grid, never wider than the source, so small type stays legible for an
+observer that reads pictures; `shots` lists where the picture jumps, which is a starting point when
+nothing is known yet and not a shot list; `fetch` turns a link into a file with the pinned yt-dlp.
+Every command writes only what `--to` names and refuses to overwrite. `vocabulary` reads the installed
+manifests: every package with its tags and models, or one package's Surfaces with their attributes,
+children and example, or the value shapes a drawing Producer must emit.
+
Install the `hypit` Distribution globally once. It resolves its own TypeScript loader and CLI, so it
neither invokes npm per command nor requires a project to contain Hypit's `package.json`.
diff --git a/packages/video-cli/package.json b/packages/video-cli/package.json
index 3aff72fe..9dd2ad52 100644
--- a/packages/video-cli/package.json
+++ b/packages/video-cli/package.json
@@ -11,6 +11,7 @@
"dependencies": {
"@hypit/cli": "workspace:*",
"@hypit/compiler-markup-node": "workspace:*",
+ "@hypit/composition": "workspace:*",
"@hypit/credential-store-os": "workspace:*",
"@hypit/driver-node": "workspace:*",
"@hypit/estimate": "workspace:*",
@@ -18,6 +19,7 @@
"@hypit/generation": "workspace:*",
"@hypit/markup": "workspace:*",
"@hypit/mimo-tts": "workspace:*",
+ "@hypit/model-kit": "workspace:*",
"@hypit/package-loader-node": "workspace:*",
"@hypit/protocol": "workspace:*",
"@hypit/provider-hypihub": "workspace:*",
@@ -27,7 +29,9 @@
"@hypit/speech": "workspace:*",
"@hypit/speech-evidence": "workspace:*",
"@hypit/text": "workspace:*",
+ "@hypit/visual-ir": "workspace:*",
"@hypit/whisperx": "workspace:*",
- "@hypit/workspace-fs-node": "workspace:*"
+ "@hypit/workspace-fs-node": "workspace:*",
+ "@hypit/yt-dlp": "workspace:*"
}
}
diff --git a/packages/video-cli/src/cli.ts b/packages/video-cli/src/cli.ts
index 0d49f564..8bd416fc 100644
--- a/packages/video-cli/src/cli.ts
+++ b/packages/video-cli/src/cli.ts
@@ -2,6 +2,8 @@
import { renderCliError, writeCliHelp } from "@hypit/cli";
import type { CliIo } from "@hypit/cli";
import { creationCommands, isCreationCommand, writeCreationHelp } from "./creation.js";
+import { isMediaCommand, mediaCommands, writeMediaHelp } from "./media.js";
+import { writeVocabularyHelp } from "./vocabulary.js";
const argv = process.argv.slice(2);
const json = argv.includes("--json");
@@ -64,10 +66,20 @@ async function main(): Promise<void> {
writeCreationHelp(io, topic);
return;
}
+ if (topic === "media") {
+ const sub = argv[0] === "help" ? argv[2] : argv[1];
+ writeMediaHelp(io, isMediaCommand(sub) ? sub : undefined);
+ return;
+ }
+ if (topic === "vocabulary") {
+ writeVocabularyHelp(io);
+ return;
+ }
writeCliHelp(io, topic);
if (topic === undefined) {
io.write(`\nCreation tools (one request through the selected Runtime Profile, no Build)\n${
- creationCommands.map((item) => ` ${item}`).join("\n")}\n hypit help <tool> for each\n`);
+ creationCommands.map((item) => ` ${item}`).join("\n")}\n hypit help <tool> for each\n`
+ + `\nPreparation (local, no request, no state)\n media ${mediaCommands.join(" | ")}\n vocabulary\n hypit help media, hypit help vocabulary\n`);
}
return;
}
diff --git a/packages/video-cli/src/creation.ts b/packages/video-cli/src/creation.ts
index 2fbd620c..7e595cba 100644
--- a/packages/video-cli/src/creation.ts
+++ b/packages/video-cli/src/creation.ts
@@ -1,4 +1,3 @@
-import { spawn } from "node:child_process";
import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, extname, join, resolve } from "node:path";
@@ -28,6 +27,7 @@ import { whisperXCapabilities, whisperXRequestForEvidenceAudio } from "@hypit/wh
import type { WhisperXLanguage } from "@hypit/whisperx";
import { videoCliDistribution } from "./distribution.js";
+import { runProcess } from "./process.js";
/**
* Creation-time tools: see a picture, hear a recording, speak a line.
@@ -220,19 +220,8 @@ function mediaType(path: string): string {
return type;
}
-function run(executable: string, args: readonly string[]): Promise<string> {
- return new Promise((resolveRun, reject) => {
- const child = spawn(executable, args, { stdio: ["ignore", "pipe", "pipe"] });
- const out: Buffer[] = [];
- const err: Buffer[] = [];
- child.stdout.on("data", (chunk: Buffer) => out.push(chunk));
- child.stderr.on("data", (chunk: Buffer) => err.push(chunk));
- child.on("error", (error) => reject(new Error(`${executable} could not start: ${error.message}`)));
- child.on("close", (code) => {
- if (code === 0) resolveRun(Buffer.concat(out).toString("utf8"));
- else reject(new Error(`${executable} exited with ${code}: ${Buffer.concat(err).toString("utf8").trim()}`));
- });
- });
+async function run(executable: string, args: readonly string[]): Promise<string> {
+ return (await runProcess(executable, args)).toString("utf8");
}
const EVIDENCE_SAMPLE_RATE = 16_000;
diff --git a/packages/video-cli/src/index.ts b/packages/video-cli/src/index.ts
index 7a55d30d..8b9433e8 100644
--- a/packages/video-cli/src/index.ts
+++ b/packages/video-cli/src/index.ts
@@ -9,6 +9,8 @@ import type { LoadedPackage } from "@hypit/package-loader-node";
import { isCreationCommand, runCreationCli } from "./creation.js";
import { videoCliDistribution } from "./distribution.js";
+import { runMediaCli } from "./media.js";
+import { runVocabularyCli } from "./vocabulary.js";
export {
createVideoCompiler,
@@ -17,6 +19,10 @@ export { videoCliDistribution } from "./distribution.js";
export { discoverVideoSourcePackages } from "./package-selection.js";
export { creationCommands, isCreationCommand, runCreationCli, writeCreationHelp } from "./creation.js";
export type { CreationCommand, CreationEnvironment, CreationHost } from "./creation.js";
+export { isMediaCommand, mediaCommands, runMediaCli, writeMediaHelp } from "./media.js";
+export type { MediaCommand, MediaProbe } from "./media.js";
+export { listPackages, listSurfaces, runVocabularyCli, visualSchema, writeVocabularyHelp } from "./vocabulary.js";
+export type { PackageListing, SurfaceListing } from "./vocabulary.js";
/** The project's selected Runtime Profile, read the way `hypit` reads it, for tools that run beside the CLI. */
export { findRuntimeProfile } from "@hypit/cli";
@@ -30,6 +36,8 @@ export function runVideoCli(
: [videoCliDistribution.packageRoot]);
installExternalPackageResolution([hypitHostPackageRoot()]);
if (isCreationCommand(argv[0])) return runCreationCli(argv, io);
+ if (argv[0] === "media") return runMediaCli(argv, io);
+ if (argv[0] === "vocabulary") return runVocabularyCli(argv, io);
return runCli(argv, io, {
...videoCliDistribution,
bootstrapPackages: packages,
diff --git a/packages/video-cli/src/media.ts b/packages/video-cli/src/media.ts
new file mode 100644
index 00000000..c145f831
--- /dev/null
+++ b/packages/video-cli/src/media.ts
@@ -0,0 +1,354 @@
+import { mkdir, stat } from "node:fs/promises";
+import { dirname, extname, join, resolve } from "node:path";
+
+import type { CliIo } from "@hypit/cli";
+import { downloadVideo, isVideoUrl } from "@hypit/yt-dlp";
+
+import { runProcess } from "./process.js";
+
+/**
+ * Media preparation for the eyes: dumb, local, stateless.
+ *
+ * A reference video is read by the Gemini Endpoint through `hypit observe`. These commands only make
+ * the pieces that request needs when the whole video is too long or too dense to read at once: a
+ * stretch cut to a clip, frames at chosen seconds, a grid of frames drawn large enough to read small
+ * type, the pixel-jump boundaries a video shows before anyone knows what it is, and a link turned into
+ * a file. Each writes exactly what the caller named and nothing else; none decides what the video is.
+ */
+
+export const mediaCommands = ["probe", "cut", "frames", "tile", "shots", "fetch"] as const;
+export type MediaCommand = typeof mediaCommands[number];
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+export function isMediaCommand(value: string | undefined): value is MediaCommand {
+ return (mediaCommands as readonly string[]).includes(value ?? "");
+}
+
+function round(value: number): number { return Number(value.toFixed(3)); }
+function clamp(value: number, low: number, high: number): number { return Math.max(low, Math.min(high, value)); }
+
+// ---------------------------------------------------------------------------------------------------
+// Arguments
+
+type Parsed = {
+ readonly positionals: readonly string[];
+ readonly options: ReadonlyMap<string, string>;
+ readonly json: boolean;
+};
+
+function parseArguments(argv: readonly string[], allowed: readonly string[]): Parsed {
+ const positionals: string[] = [];
+ const options = new Map<string, string>();
+ let json = false;
+ for (let index = 0; index < argv.length; index += 1) {
+ const item = argv[index]!;
+ if (item === "--json") { json = true; continue; }
+ if (item === "--debug" || item === "--verbose" || item === "--no-color") continue;
+ if (item === "--color") { index += 1; continue; }
+ if (!item.startsWith("--")) { positionals.push(item); continue; }
+ if (!allowed.includes(item)) throw new Error(`unknown option ${item}`);
+ if (options.has(item)) throw new Error(`${item} cannot be repeated`);
+ const value = argv[index + 1];
+ if (value === undefined || (value.startsWith("--") && value.length > 2)) throw new Error(`${item} requires a value`);
+ options.set(item, value);
+ index += 1;
+ }
+ return { positionals, options, json };
+}
+
+function required(parsed: Parsed, option: string, hint: string): string {
+ const value = parsed.options.get(option);
+ if (value === undefined) throw new Error(`${option} is required: ${hint}`);
+ return value;
+}
+
+function secondsOption(parsed: Parsed, option: string, fallback?: number): number {
+ const raw = parsed.options.get(option);
+ if (raw === undefined) {
+ if (fallback !== undefined) return fallback;
+ throw new Error(`${option} is required: a time in seconds`);
+ }
+ const value = Number(raw);
+ assert(Number.isFinite(value) && value >= 0, `${option} must be a non-negative number of seconds, got ${raw}`);
+ return value;
+}
+
+function integerOption(parsed: Parsed, option: string, fallback: number, low: number): number {
+ const raw = parsed.options.get(option);
+ if (raw === undefined) return fallback;
+ const value = Number(raw);
+ assert(Number.isSafeInteger(value) && value >= low, `${option} must be a whole number of at least ${low}, got ${raw}`);
+ return value;
+}
+
+async function sourceFile(parsed: Parsed, cwd: string): Promise<string> {
+ const [source] = parsed.positionals;
+ assert(source !== undefined, "name the media file first");
+ const path = resolve(cwd, source);
+ assert(await stat(path).then((item) => item.isFile(), () => false), `cannot read ${path}`);
+ return path;
+}
+
+async function destination(parsed: Parsed, cwd: string, hint: string): Promise<string> {
+ const to = resolve(cwd, required(parsed, "--to", hint));
+ assert(!(await stat(to).then(() => true, () => false)), `Destination ${to} already exists`);
+ await mkdir(dirname(to), { recursive: true });
+ return to;
+}
+
+// ---------------------------------------------------------------------------------------------------
+// ffmpeg
+
+export type MediaProbe = {
+ readonly duration: number;
+ readonly width: number;
+ readonly height: number;
+ readonly frameRate: number;
+ readonly hasAudio: boolean;
+};
+
+export async function probeMedia(path: string): Promise<MediaProbe> {
+ const raw = await runProcess("ffprobe", [
+ "-v", "error", "-show_entries", "format=duration:stream=codec_type,width,height,r_frame_rate", "-of", "json", path,
+ ]);
+ const parsed = JSON.parse(raw.toString("utf8")) as {
+ format?: { duration?: string };
+ streams?: readonly { codec_type?: string; width?: number; height?: number; r_frame_rate?: string }[];
+ };
+ const video = parsed.streams?.find((item) => item.codec_type === "video");
+ const duration = Number(parsed.format?.duration);
+ assert(Number.isFinite(duration) && duration > 0, `${path}: duration is unavailable`);
+ assert(video?.width !== undefined && video.height !== undefined, `${path}: no video stream`);
+ // ffprobe reports the rate as a ratio; a still image reports none.
+ const ratio = (video.r_frame_rate ?? "").split("/");
+ const rate = Number(ratio[0]) / Number(ratio[1] ?? 1);
+ return {
+ duration: round(duration),
+ width: video.width,
+ height: video.height,
+ frameRate: Number.isFinite(rate) && rate > 0 ? round(rate) : 0,
+ hasAudio: parsed.streams?.some((item) => item.codec_type === "audio") ?? false,
+ };
+}
+
+/**
+ * Open `source` on the frame at `at` seconds.
+ *
+ * `-ss` after `-i` alone decodes every frame up to the mark; `-ss` before `-i` alone lands on the
+ * keyframe before it. Jumping to two seconds early and decoding the short run-up lands on the exact
+ * frame: measured on a 1080p source nine minutes in, the same JPEG to the byte, 11.6 s to 0.2 s.
+ */
+const SEEK_RUN_UP = 2;
+function openAt(source: string, at: number): readonly string[] {
+ const jump = Math.max(0, at - SEEK_RUN_UP);
+ const runUp = at - jump;
+ return [...(jump > 0 ? ["-ss", String(round(jump))] : []), "-i", source, ...(runUp > 0 ? ["-ss", String(round(runUp))] : [])];
+}
+
+// JPEG is a full-range format; web video is routinely full-range YUV and the encoder refuses it
+// unless told which it is writing (`Non full-range YUV is non-standard`).
+const JPEG = ["-pix_fmt", "yuvj420p", "-q:v", "3"] as const;
+
+export async function cutClip(source: string, start: number, end: number, target: string): Promise<void> {
+ assert(end > start, `the clip must end after it starts (${start} to ${end})`);
+ await runProcess("ffmpeg", [
+ "-hide_banner", "-loglevel", "error", "-y", ...openAt(source, start), "-t", String(round(end - start)),
+ "-c:v", "libx264", "-preset", "veryfast", "-crf", "23", "-c:a", "aac", "-b:a", "96k", "-movflags", "+faststart", target,
+ ]);
+}
+
+export async function cutFrame(source: string, at: number, target: string): Promise<void> {
+ await runProcess("ffmpeg", ["-hide_banner", "-loglevel", "error", "-y", ...openAt(source, at), "-frames:v", "1", ...JPEG, target]);
+}
+
+const TILE_COLUMNS = 3;
+const TILE_CELL_WIDTH = 480;
+/** Four frames show what moves in a one-second shot; nine keep a fifteen-second one legible. */
+export function tileFrameCount(seconds: number): number { return clamp(Math.round(seconds * 1.5), 4, 9); }
+
+export async function tileFrames(
+ source: string,
+ start: number,
+ end: number,
+ target: string,
+ cellWidth: number,
+ frameCount: number,
+): Promise<void> {
+ assert(end > start, `the stretch must end after it starts (${start} to ${end})`);
+ const seconds = end - start;
+ const rows = Math.ceil(frameCount / TILE_COLUMNS);
+ const rate = frameCount / Math.max(seconds, 0.1);
+ await runProcess("ffmpeg", [
+ "-hide_banner", "-loglevel", "error", "-y", ...openAt(source, start), "-t", String(round(seconds)),
+ "-vf", `fps=${rate},scale='min(${Math.round(cellWidth)},iw)':-2,tile=layout=${TILE_COLUMNS}x${rows}:padding=8:margin=8:color=black`,
+ "-frames:v", "1", ...JPEG, target,
+ ]);
+}
+
+/**
+ * Where the picture jumps, at 12 samples per second of 32×32 signatures.
+ *
+ * At three samples a second a boundary overshot by a third of a second, long enough for an observer
+ * to describe a shot as the picture before it. A jump closer than one second to the previous one is a
+ * flicker, flash frame or fast pan rather than a shot, and a clip that short is refused by the model
+ * anyway; a final fragment that short belongs to the shot before it.
+ */
+const SAMPLE_RATE = 12;
+const SHORTEST_SHOT = 1;
+export async function shotBoundaries(source: string, duration: number): Promise<readonly { readonly start: number; readonly end: number }[]> {
+ const bytes = await runProcess("ffmpeg", [
+ "-hide_banner", "-loglevel", "error", "-i", source, "-vf", `fps=${SAMPLE_RATE},scale=32:32,format=rgb24`, "-f", "rawvideo", "-",
+ ]);
+ const stride = 32 * 32 * 3;
+ const frames: Uint8Array[] = [];
+ for (let index = 0; index + stride <= bytes.byteLength; index += stride) frames.push(bytes.subarray(index, index + stride));
+ const starts = [0];
+ if (frames.length >= 2) {
+ let anchor = frames[0]!;
+ for (let index = 1; index < frames.length; index += 1) {
+ const frame = frames[index]!;
+ let total = 0;
+ for (let byte = 0; byte < frame.length; byte += 1) total += Math.abs(frame[byte]! - anchor[byte]!);
+ if (total / frame.length / 255 > 0.1) {
+ const at = round(index / SAMPLE_RATE);
+ if (at - starts.at(-1)! >= SHORTEST_SHOT) starts.push(at);
+ anchor = frame;
+ }
+ }
+ }
+ if (starts.length > 1 && duration - starts.at(-1)! < SHORTEST_SHOT) starts.pop();
+ return starts.map((start, index) => ({ start, end: round(starts[index + 1] ?? duration) }));
+}
+
+// ---------------------------------------------------------------------------------------------------
+// Commands
+
+async function probe(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
+ const parsed = parseArguments(argv, []);
+ const source = await sourceFile(parsed, cwd);
+ const info = await probeMedia(source);
+ if (parsed.json) { io.write(`${JSON.stringify({ path: source, ...info }, null, 2)}\n`); return; }
+ io.write(`${source}\n ${info.duration} s ${info.width}×${info.height} ${info.frameRate > 0 ? `${info.frameRate} fps` : "still"} ${info.hasAudio ? "with audio" : "no audio"}\n`);
+}
+
+async function cut(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
+ const parsed = parseArguments(argv, ["--start", "--end", "--to"]);
+ const source = await sourceFile(parsed, cwd);
+ const start = secondsOption(parsed, "--start", 0);
+ const info = await probeMedia(source);
+ const end = secondsOption(parsed, "--end", info.duration);
+ assert(end <= info.duration + 0.001, `--end ${end} is past the end of the file (${info.duration} s)`);
+ const target = await destination(parsed, cwd, "the clip to write, for example notes/hook.mp4");
+ await cutClip(source, start, end, target);
+ if (parsed.json) { io.write(`${JSON.stringify({ path: target, start, end, seconds: round(end - start) }, null, 2)}\n`); return; }
+ io.write(`${target}\n ${start} s to ${end} s (${round(end - start)} s)\n`);
+}
+
+async function frames(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
+ const parsed = parseArguments(argv, ["--at", "--every", "--to"]);
+ const source = await sourceFile(parsed, cwd);
+ const info = await probeMedia(source);
+ const at = parsed.options.get("--at");
+ const every = parsed.options.get("--every");
+ assert((at === undefined) !== (every === undefined), "name the seconds with --at <s,s,…> or sample with --every <seconds>, not both");
+ let times: number[];
+ if (at !== undefined) {
+ times = at.split(",").map((item) => Number(item.trim()));
+ assert(times.every((value) => Number.isFinite(value) && value >= 0), `--at must list seconds, got ${at}`);
+ } else {
+ const step = Number(every);
+ assert(Number.isFinite(step) && step > 0, `--every must be a positive number of seconds, got ${every}`);
+ times = [];
+ for (let time = 0; time < info.duration; time += step) times.push(round(time));
+ }
+ assert(times.every((value) => value <= info.duration), `a time is past the end of the file (${info.duration} s)`);
+ const to = resolve(cwd, required(parsed, "--to", "the directory to write the frames into"));
+ assert(!(await stat(to).then((item) => item.isFile(), () => false)), `${to} is a file; --to names a directory`);
+ await mkdir(to, { recursive: true });
+ const written: { readonly at: number; readonly path: string }[] = [];
+ for (const time of times) {
+ const target = join(to, `frame-${time.toFixed(2).replace(".", "_")}s.jpg`);
+ assert(!(await stat(target).then(() => true, () => false)), `${target} already exists`);
+ await cutFrame(source, time, target);
+ written.push({ at: time, path: target });
+ }
+ if (parsed.json) { io.write(`${JSON.stringify({ directory: to, frames: written }, null, 2)}\n`); return; }
+ io.write(`${to}\n${written.map((item) => ` ${item.at} s ${item.path}`).join("\n")}\n`);
+}
+
+async function tile(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
+ const parsed = parseArguments(argv, ["--start", "--end", "--frames", "--cell", "--to"]);
+ const source = await sourceFile(parsed, cwd);
+ const info = await probeMedia(source);
+ const start = secondsOption(parsed, "--start", 0);
+ const end = secondsOption(parsed, "--end", info.duration);
+ assert(end <= info.duration + 0.001, `--end ${end} is past the end of the file (${info.duration} s)`);
+ const count = integerOption(parsed, "--frames", tileFrameCount(end - start), 2);
+ // Never enlarge a small source: a grid drawn wider than the video carries the same pixels as more.
+ const cell = integerOption(parsed, "--cell", Math.min(TILE_CELL_WIDTH, info.width), 64);
+ const target = await destination(parsed, cwd, "the grid image to write, for example notes/hook-grid.jpg");
+ await tileFrames(source, start, end, target, cell, count);
+ const rows = Math.ceil(count / TILE_COLUMNS);
+ if (parsed.json) { io.write(`${JSON.stringify({ path: target, start, end, frames: count, columns: TILE_COLUMNS, rows, cellWidth: cell }, null, 2)}\n`); return; }
+ io.write(`${target}\n ${count} frames from ${start} s to ${end} s, ${TILE_COLUMNS}×${rows}, ${cell} px cells, reading order left to right then top to bottom\n`);
+}
+
+async function shots(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
+ const parsed = parseArguments(argv, []);
+ const source = await sourceFile(parsed, cwd);
+ const info = await probeMedia(source);
+ const bounds = await shotBoundaries(source, info.duration);
+ if (parsed.json) { io.write(`${JSON.stringify({ path: source, shots: bounds }, null, 2)}\n`); return; }
+ io.write(`${source}\n ${bounds.length} pixel-jump ${bounds.length === 1 ? "stretch" : "stretches"}; a jump is not a shot until the eyes say so\n${
+ bounds.map((bound, index) => ` ${String(index + 1).padStart(3, " ")} ${bound.start} s to ${bound.end} s (${round(bound.end - bound.start)} s)`).join("\n")}\n`);
+}
+
+async function fetch(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
+ const parsed = parseArguments(argv, ["--to"]);
+ const [url] = parsed.positionals;
+ assert(url !== undefined && isVideoUrl(url), "name an http or https link first");
+ const target = await destination(parsed, cwd, "the video file to write, for example reference/source.mp4");
+ assert([".mp4", ".mkv", ".webm", ".mov"].includes(extname(target).toLowerCase()), "--to must end in .mp4, .mkv, .webm or .mov");
+ await downloadVideo(url, target);
+ const info = await probeMedia(target);
+ if (parsed.json) { io.write(`${JSON.stringify({ path: target, url, ...info }, null, 2)}\n`); return; }
+ io.write(`${target}\n ${info.duration} s ${info.width}×${info.height} ${info.hasAudio ? "with audio" : "no audio"}\n`);
+}
+
+// ---------------------------------------------------------------------------------------------------
+// Entry
+
+export function writeMediaHelp(io: CliIo, topic?: MediaCommand): void {
+ const sections: Record<MediaCommand, readonly string[]> = {
+ probe: [" hypit media probe <file>", " Duration, size, frame rate and whether there is audio."],
+ cut: [" hypit media cut <file> --start <s> --end <s> --to <clip.mp4>", " One stretch as a clip of its own, cut on the exact frames."],
+ frames: [" hypit media frames <file> (--at <s,s,…> | --every <s>) --to <dir>", " Single frames at those seconds, as JPEG files named by their time."],
+ tile: [" hypit media tile <file> [--start <s> --end <s>] [--frames <n>] [--cell <px>] --to <grid.jpg>",
+ " Frames sampled evenly across a stretch, in one grid read left to right then top to bottom.",
+ " Defaults: 4 to 9 frames at 1.5 per second, 480 px cells, never wider than the source."],
+ shots: [" hypit media shots <file>", " Where the picture jumps. A starting point when nothing is known yet; not a shot list."],
+ fetch: [" hypit media fetch <url> --to <video.mp4>", " A link turned into a file with the pinned yt-dlp, video and audio together."],
+ };
+ const chosen = topic === undefined ? mediaCommands : [topic];
+ io.write(`hypit media\nPrepare what the eyes will look at. Local ffmpeg work; no Runtime Profile, no request, no state.\n\n${
+ chosen.map((item) => sections[item].join("\n")).join("\n\n")}\n\nEvery command writes only what --to names and refuses to overwrite. Add --json for a machine view.\n`);
+}
+
+export async function runMediaCli(argv: readonly string[], io: CliIo, cwd = process.cwd()): Promise<void> {
+ const command = argv[1];
+ if (command === undefined || command === "--help" || argv.includes("--help")) {
+ writeMediaHelp(io, isMediaCommand(command) ? command : undefined);
+ return;
+ }
+ assert(isMediaCommand(command), `unknown media command ${command}; one of ${mediaCommands.join(", ")}`);
+ const rest = argv.slice(2);
+ if (command === "probe") await probe(rest, io, cwd);
+ else if (command === "cut") await cut(rest, io, cwd);
+ else if (command === "frames") await frames(rest, io, cwd);
+ else if (command === "tile") await tile(rest, io, cwd);
+ else if (command === "shots") await shots(rest, io, cwd);
+ else await fetch(rest, io, cwd);
+}
diff --git a/packages/video-cli/src/process.ts b/packages/video-cli/src/process.ts
new file mode 100644
index 00000000..87f1f86c
--- /dev/null
+++ b/packages/video-cli/src/process.ts
@@ -0,0 +1,26 @@
+import { spawn } from "node:child_process";
+
+/** Run one external program to completion and return its stdout; stderr becomes the error text. */
+export function runProcess(executable: string, args: readonly string[], timeoutMs = 300_000): Promise<Buffer> {
+ return new Promise((resolveRun, reject) => {
+ const child = spawn(executable, [...args], { stdio: ["ignore", "pipe", "pipe"], windowsHide: true });
+ const out: Buffer[] = [];
+ let err = "";
+ let settled = false;
+ const finish = (error?: Error): void => {
+ if (settled) return;
+ settled = true;
+ clearTimeout(timer);
+ if (error === undefined) resolveRun(Buffer.concat(out));
+ else reject(error);
+ };
+ const timer = setTimeout(() => { child.kill("SIGKILL"); finish(new Error(`${executable} timed out after ${timeoutMs} ms`)); }, timeoutMs);
+ child.stdout.on("data", (chunk: Buffer) => out.push(chunk));
+ child.stderr.on("data", (chunk: Buffer) => { err = `${err}${chunk.toString("utf8")}`.slice(-100_000); });
+ child.on("error", (error) => finish(new Error(`${executable} could not start: ${error.message}`)));
+ child.on("close", (code) => {
+ if (code === 0) finish();
+ else finish(new Error(`${executable} exited with ${code}: ${err.trim()}`));
+ });
+ });
+}
diff --git a/packages/video-cli/src/vocabulary.ts b/packages/video-cli/src/vocabulary.ts
new file mode 100644
index 00000000..16df04c4
--- /dev/null
+++ b/packages/video-cli/src/vocabulary.ts
@@ -0,0 +1,358 @@
+import { readdir, readFile, stat } from "node:fs/promises";
+import { dirname, join, resolve } from "node:path";
+
+import type { CliIo } from "@hypit/cli";
+import {
+ animatableLocalStyles,
+ visualBoxSchema, visualElementSchema, visualImageSchema, visualMaskSchema, visualPathCommandSchema,
+ visualSurfaceSchema, visualTextDocumentSchema, visualTextFlowSchema, visualTextPaintSchema,
+ visualTextSchema, visualTextTypographySchema, visualTrackSchema, visualVideoSchema,
+} from "@hypit/composition";
+import { markupSurfaceHostFacetAbi } from "@hypit/markup";
+import type { RegisteredSurface, SurfaceAttributeVocabulary, SurfaceVocabulary } from "@hypit/markup";
+import { exactModelHostAbi } from "@hypit/model-kit";
+import { loadNodePackageSelection, locateNodePackage } from "@hypit/package-loader-node";
+import type { NodePackageLoadOptions } from "@hypit/package-loader-node";
+import type { ValueSchema } from "@hypit/protocol";
+import { VISUAL_STYLE_ENUM_VALUES_V1, VISUAL_STYLE_NAMES_V1 } from "@hypit/visual-ir";
+
+import { videoCliDistribution } from "./distribution.js";
+
+/**
+ * What a Source may write: the installed packages, and the Surfaces each one declares.
+ *
+ * The manifest is the authority; this prints it. A system you never inspected is a system you are
+ * about to invent, so the listing shows every scope and the project's own `packages/`, and a named
+ * package answers with its own Surfaces only, not with everything it happens to depend on.
+ */
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+async function nearestPackageRoot(start: string): Promise<string> {
+ let directory = resolve(start);
+ while (true) {
+ if (await stat(resolve(directory, "package.json")).then((item) => item.isFile(), () => false)) return directory;
+ const parent = dirname(directory);
+ if (parent === directory) return resolve(start);
+ directory = parent;
+ }
+}
+
+function loading(): NodePackageLoadOptions {
+ return videoCliDistribution.packageRoot === undefined ? {} : { fallbackRoots: [videoCliDistribution.packageRoot] };
+}
+
+async function readJson<T>(path: string): Promise<T | undefined> {
+ try { return JSON.parse(await readFile(path, "utf8")) as T; } catch { return undefined; }
+}
+
+// ---------------------------------------------------------------------------------------------------
+// Listing
+
+export type PackageListing = {
+ readonly name: string;
+ readonly description?: string;
+ readonly tags: readonly string[];
+ readonly models?: readonly string[];
+ readonly unreadable?: string;
+};
+
+export async function listPackages(projectRoot: string): Promise<readonly PackageListing[]> {
+ // Every scope, and the project's own directory, from both the project's node_modules and the
+ // Distribution's: the packages a project does not carry are in the Distribution, and the ones it
+ // does are beside it.
+ const roots = [...new Set([
+ join(projectRoot, "node_modules"),
+ ...(videoCliDistribution.packageRoot === undefined ? [] : [join(videoCliDistribution.packageRoot, "node_modules")]),
+ ])];
+ const candidates: { readonly name: string; readonly directory: string }[] = [];
+ for (const modules of roots) {
+ const scopes = (await readdir(modules, { withFileTypes: true }).catch(() => []))
+ .filter((entry) => entry.isDirectory() && entry.name.startsWith("@"))
+ .map((entry) => entry.name)
+ .sort();
+ for (const scope of scopes) {
+ for (const inside of (await readdir(join(modules, scope)).catch(() => [] as string[])).sort()) {
+ const name = `${scope}/${inside}`;
+ if (!candidates.some((item) => item.name === name)) candidates.push({ name, directory: join(modules, scope, inside) });
+ }
+ }
+ }
+ // A project's own packages are its `packages/<name>/`, where the loader finds them by name whether
+ // or not a package manager linked them.
+ for (const entry of (await readdir(join(projectRoot, "packages")).catch(() => [] as string[])).sort()) {
+ const directory = join(projectRoot, "packages", entry);
+ const own = await readJson<{ readonly name?: string }>(join(directory, "package.json"));
+ if (own?.name !== undefined && !candidates.some((item) => item.name === own.name)) candidates.push({ name: own.name, directory });
+ }
+ const listing: PackageListing[] = [];
+ for (const { name, directory } of candidates) {
+ const manifest = await readJson<{ readonly hypit?: { readonly activation?: string }; readonly description?: string }>(join(directory, "package.json"));
+ if (manifest?.hypit?.activation === undefined) continue;
+ const tags: string[] = [];
+ const models: string[] = [];
+ let unreadable: string | undefined;
+ try {
+ for (const pack of await loadNodePackageSelection([name], projectRoot, loading())) {
+ for (const facet of pack.contribution.hostFacets ?? []) {
+ if (facet.abi === markupSurfaceHostFacetAbi) tags.push((facet.implementation as RegisteredSurface).tag);
+ else if (facet.abi === exactModelHostAbi) models.push(...(facet as { readonly offers?: readonly string[] }).offers ?? []);
+ }
+ }
+ } catch (error) { unreadable = error instanceof Error ? error.message : String(error); }
+ listing.push({
+ name,
+ ...(manifest.description === undefined ? {} : { description: manifest.description }),
+ tags: [...new Set(tags)].sort(),
+ ...(models.length === 0 ? {} : { models: [...new Set(models)].sort() }),
+ ...(unreadable === undefined ? {} : { unreadable }),
+ });
+ }
+ return listing;
+}
+
+// ---------------------------------------------------------------------------------------------------
+// Surfaces
+
+export type SurfaceListing = {
+ readonly package: string;
+ readonly module: RegisteredSurface["module"];
+ readonly surface: string;
+ readonly tag: string;
+ readonly mode: RegisteredSurface["mode"];
+ readonly outputs: RegisteredSurface["outputs"];
+ readonly vocabulary?: unknown;
+ readonly readme?: string;
+};
+
+function printableVocabulary(value: SurfaceVocabulary | undefined): unknown {
+ if (value === undefined) return undefined;
+ // The poster is a file the manifest opens; the listing names it and does not read it.
+ return { ...value, ...(value.preview === undefined ? {} : { preview: { mediaType: value.preview.mediaType, path: value.preview.path } }) };
+}
+
+function readmePath(name: string, projectRoot: string): string | undefined {
+ try {
+ const located = locateNodePackage(name, {
+ from: join(projectRoot, "__hypit_vocabulary__.mjs"),
+ workspaceRoots: [projectRoot],
+ ...(videoCliDistribution.packageRoot === undefined ? {} : { distributionRoots: [videoCliDistribution.packageRoot] }),
+ });
+ return join(located.root, "README.md");
+ } catch { return undefined; }
+}
+
+export async function listSurfaces(projectRoot: string, packages: readonly string[], tags?: readonly string[]): Promise<readonly SurfaceListing[]> {
+ assert(packages.length > 0, "name at least one package");
+ const loaded = await loadNodePackageSelection(packages, projectRoot, loading());
+ const named = new Set(packages);
+ const wanted = tags === undefined ? undefined : new Set(tags);
+ const surfaces: SurfaceListing[] = [];
+ for (const pack of loaded) {
+ // Loading a package brings its dependencies with it; the answer is about the ones named.
+ if (!named.has(pack.specifier)) continue;
+ for (const facet of pack.contribution.hostFacets ?? []) {
+ if (facet.abi !== markupSurfaceHostFacetAbi) continue;
+ const surface = facet.implementation as RegisteredSurface;
+ if (wanted !== undefined && !wanted.has(surface.tag) && !wanted.has(surface.surface)) continue;
+ const readme = readmePath(pack.specifier, projectRoot);
+ surfaces.push({
+ package: pack.specifier,
+ module: surface.module,
+ surface: surface.surface,
+ tag: surface.tag,
+ mode: surface.mode,
+ outputs: surface.outputs,
+ ...(surface.vocabulary === undefined ? {} : { vocabulary: printableVocabulary(surface.vocabulary) }),
+ ...(readme === undefined ? {} : { readme }),
+ });
+ }
+ }
+ return surfaces;
+}
+
+// ---------------------------------------------------------------------------------------------------
+// Visual schema, for a component that draws
+
+/** Render a declared value shape as prose an author can act on. */
+export function describeSchema(schema: ValueSchema, indent = ""): string[] {
+ const step = `${indent} `;
+ switch (schema.kind) {
+ case "object": {
+ const lines: string[] = [];
+ for (const [name, field] of Object.entries(schema.fields)) {
+ const inner = describeSchema(field.schema, step);
+ lines.push(`${indent}${name}${field.optional === true ? " (optional)" : ""}: ${inner[0]!.trimStart()}`, ...inner.slice(1));
+ }
+ if (schema.allowUnknown === true) lines.push(`${indent}… further fields are admitted`);
+ return lines;
+ }
+ case "oneOf": {
+ const pinned = schema.variants.map((variant) => {
+ if (variant.kind !== "object") return undefined;
+ const literal = Object.entries(variant.fields).find(([, field]) => field.schema.kind === "literal");
+ return literal === undefined ? undefined : String((literal[1].schema as { value: unknown }).value);
+ });
+ return [pinned.every((value) => value !== undefined) ? `${indent}one of ${pinned.join(", ")}` : `${indent}one of ${schema.variants.length} shapes`];
+ }
+ case "array": return [`${indent}a list${schema.minItems === undefined ? "" : ` (at least ${schema.minItems})`} of:`, ...describeSchema(schema.items, step)];
+ case "string": return [schema.enum !== undefined ? `${indent}one of ${schema.enum.join(", ")}` : `${indent}${schema.minLength === undefined ? "text" : `text, at least ${schema.minLength} characters`}`];
+ case "number": {
+ const parts = [schema.integer === true ? "a whole number" : "a number"];
+ if (schema.minimum !== undefined) parts.push(`at least ${schema.minimum}`);
+ if (schema.maximum !== undefined) parts.push(`at most ${schema.maximum}`);
+ return [`${indent}${parts.join(", ")}`];
+ }
+ case "literal": return [`${indent}exactly ${JSON.stringify(schema.value)}`];
+ case "boolean": return [`${indent}true or false`];
+ case "null": return [`${indent}null`];
+ default: return [`${indent}${(schema as { kind: string }).kind}`];
+ }
+}
+
+const VISUAL_SHAPES: Readonly<Record<string, ValueSchema>> = {
+ "visual-track": visualTrackSchema,
+ "visual-element": visualElementSchema,
+ box: visualBoxSchema,
+ mask: visualMaskSchema,
+ text: visualTextSchema,
+ image: visualImageSchema,
+ video: visualVideoSchema,
+ surface: visualSurfaceSchema,
+ "text-flow": visualTextFlowSchema,
+ "text-typography": visualTextTypographySchema,
+ "text-paint": visualTextPaintSchema,
+ "text-document": visualTextDocumentSchema,
+ "path-command": visualPathCommandSchema,
+};
+
+// Each of these is refused somewhere in composition, or follows from how the emitted CSS is written.
+const VISUAL_RULES = [
+ "A Present holds exactly one element with no `parent`; every other element names one, and it must be a box or a mask.",
+ "`order` is unique across the whole Present, not among siblings.",
+ "A child's position is measured from its parent's box, not from the Canvas.",
+ "An animation carries at least two keyframes, and every keyframe of one animation declares the same properties; `atFrame` is Present-relative.",
+ "A plain text element provides exact ordered font artifacts and cannot combine them with raw `font`, `font-family`, `font-style`, `font-synthesis` or `font-weight` styles.",
+ "A text element using glyph Paint cannot also declare raw stroke or paint-order styles.",
+ "A media element's artifact media type must match image/video, and still images cannot carry sampling.",
+ "A Surface with still timing cannot carry sampling; frame-based Surface sampling must match its typed timing.",
+ "Mask roots must be direct children of one Present and both mask/content references must resolve to owned elements.",
+] as const;
+
+export function visualSchema(shape?: string): {
+ readonly shapes: readonly { readonly shape: string; readonly describes: string }[];
+ readonly animatableLocalStyles: readonly string[];
+ readonly styleNames: readonly string[];
+ readonly styleEnumValues: typeof VISUAL_STYLE_ENUM_VALUES_V1;
+ readonly rules: readonly string[];
+} {
+ const asked = shape?.trim();
+ assert(asked === undefined || asked.length === 0 || Object.hasOwn(VISUAL_SHAPES, asked), `shape must be one of ${Object.keys(VISUAL_SHAPES).join(", ")}`);
+ const chosen = asked === undefined || asked.length === 0 ? Object.keys(VISUAL_SHAPES) : [asked];
+ return {
+ shapes: chosen.map((name) => ({ shape: name, describes: describeSchema(VISUAL_SHAPES[name]!).join("\n") })),
+ animatableLocalStyles: [...animatableLocalStyles],
+ styleNames: [...VISUAL_STYLE_NAMES_V1],
+ styleEnumValues: VISUAL_STYLE_ENUM_VALUES_V1,
+ rules: VISUAL_RULES,
+ };
+}
+
+// ---------------------------------------------------------------------------------------------------
+// Command
+
+function attributeLine(attribute: SurfaceAttributeVocabulary): string {
+ const shape = attribute.values !== undefined ? attribute.values.join(" | ")
+ : attribute.accepts !== undefined ? attribute.accepts.map((type) => type.name).join(" | ")
+ : attribute.kind;
+ return ` ${attribute.name}${attribute.required ? "" : "?"} ${shape} ${attribute.summary}`;
+}
+
+function surfaceText(surface: SurfaceListing): string {
+ const vocabulary = surface.vocabulary as SurfaceVocabulary | undefined;
+ const lines = [` <${surface.tag}> ${surface.mode} from ${surface.package}`];
+ if (vocabulary !== undefined) {
+ lines.push(` ${vocabulary.summary}`);
+ if (vocabulary.appearance !== undefined) lines.push(` ${vocabulary.appearance}`);
+ if (vocabulary.attributes.length > 0) lines.push(" attributes", ...vocabulary.attributes.map(attributeLine));
+ for (const child of vocabulary.children ?? []) {
+ lines.push(` child <${child.tag}> (${child.cardinality}) ${child.summary}`, ...(child.attributes ?? []).map(attributeLine));
+ }
+ if (vocabulary.ports !== undefined && vocabulary.ports.length > 0) {
+ lines.push(" ports", ...vocabulary.ports.map((port) => ` ${JSON.stringify(port)}`));
+ }
+ lines.push(" example", ...vocabulary.example.split("\n").map((line) => ` ${line}`));
+ for (const note of vocabulary.notes ?? []) lines.push(` note: ${note}`);
+ }
+ if (surface.readme !== undefined) lines.push(` README ${surface.readme}`);
+ return lines.join("\n");
+}
+
+export function writeVocabularyHelp(io: CliIo): void {
+ io.write([
+ "hypit vocabulary",
+ "What a Source may write. Prints installed package manifests; no Runtime Profile, no request, no state.",
+ "",
+ " hypit vocabulary every installed package with its tags and models",
+ " hypit vocabulary <package…> [--tag <tag>] the Surfaces a package declares: attributes, children, example",
+ " hypit vocabulary --visual [<shape>] the value shapes and rules a drawing Producer must emit",
+ "",
+ "Read the package README the listing names before writing its elements. Add --json for a machine view.",
+ "",
+ ].join("\n"));
+}
+
+export async function runVocabularyCli(argv: readonly string[], io: CliIo, cwd = process.cwd()): Promise<void> {
+ const rest = argv.slice(1);
+ if (rest.includes("--help")) { writeVocabularyHelp(io); return; }
+ const json = rest.includes("--json");
+ const words = rest.filter((item) => item !== "--json" && item !== "--debug" && item !== "--verbose" && item !== "--no-color");
+ const projectRoot = await nearestPackageRoot(cwd);
+
+ if (words[0] === "--visual") {
+ const shape = words[1];
+ assert(words.length <= 2, "hypit vocabulary --visual takes at most one shape name");
+ const schema = visualSchema(shape);
+ if (json) { io.write(`${JSON.stringify(schema, null, 2)}\n`); return; }
+ io.write(`${schema.shapes.map((item) => `${item.shape}\n${item.describes.split("\n").map((line) => ` ${line}`).join("\n")}`).join("\n\n")}\n\n`
+ + `animatable local styles: ${schema.animatableLocalStyles.join(", ")}\nstyle names: ${schema.styleNames.join(", ")}\n\nrules\n${schema.rules.map((rule) => ` ${rule}`).join("\n")}\n`);
+ return;
+ }
+
+ const tags: string[] = [];
+ const packages: string[] = [];
+ for (let index = 0; index < words.length; index += 1) {
+ const word = words[index]!;
+ if (word === "--tag") {
+ const value = words[index + 1];
+ assert(value !== undefined && !value.startsWith("--"), "--tag requires a value");
+ tags.push(value);
+ index += 1;
+ } else {
+ assert(!word.startsWith("--"), `unknown option ${word}`);
+ packages.push(word);
+ }
+ }
+
+ if (packages.length === 0) {
+ assert(tags.length === 0, "--tag narrows a named package; name the package first");
+ const listing = await listPackages(projectRoot);
+ if (json) { io.write(`${JSON.stringify({ projectRoot, packages: listing }, null, 2)}\n`); return; }
+ assert(listing.length > 0, `no Hypit packages installed under ${projectRoot} or the Distribution`);
+ io.write(`${listing.map((item) => {
+ const parts = [item.name];
+ if (item.tags.length > 0) parts.push(`tags: ${item.tags.join(", ")}`);
+ if (item.models !== undefined) parts.push(`models: ${item.models.join(", ")}`);
+ if (item.unreadable !== undefined) parts.push(`unreadable: ${item.unreadable}`);
+ return `${parts.join("\n ")}${item.description === undefined ? "" : `\n ${item.description}`}`;
+ }).join("\n")}\n`);
+ return;
+ }
+
+ const surfaces = await listSurfaces(projectRoot, packages, tags.length === 0 ? undefined : tags);
+ if (json) { io.write(`${JSON.stringify({ projectRoot, packages, surfaces }, null, 2)}\n`); return; }
+ assert(surfaces.length > 0, `${packages.join(", ")} ${packages.length === 1 ? "declares" : "declare"} no Markup Surface${tags.length === 0 ? "" : ` tagged ${tags.join(", ")}`}`);
+ io.write(`${surfaces.map(surfaceText).join("\n\n")}\n`);
+}
diff --git a/packages/video-cli/test/media.test.ts b/packages/video-cli/test/media.test.ts
new file mode 100644
index 00000000..3ab31ef9
--- /dev/null
+++ b/packages/video-cli/test/media.test.ts
@@ -0,0 +1,109 @@
+import assert from "node:assert/strict";
+import { spawnSync } from "node:child_process";
+import { mkdtemp, readdir, rm, stat } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import test from "node:test";
+
+import type { CliIo } from "@hypit/cli";
+
+import { probeMedia, runMediaCli, shotBoundaries, tileFrameCount } from "../src/media.js";
+
+const ffmpeg = spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0
+ && spawnSync("ffprobe", ["-version"], { stdio: "ignore" }).status === 0;
+
+function io(): { readonly io: CliIo; text: () => string } {
+ let out = "";
+ return {
+ io: {
+ write: (chunk) => { out += chunk; },
+ setExitCode: () => {},
+ readSecret: async () => "",
+ terminal: { isTTY: false, color: false, unicode: false, columns: 100 },
+ },
+ text: () => out,
+ };
+}
+
+/** Three seconds of two flat colours with a hard cut at 1.5 s, 24 fps, no audio. */
+async function sample(directory: string): Promise<string> {
+ const path = join(directory, "sample.mp4");
+ const result = spawnSync("ffmpeg", [
+ "-hide_banner", "-loglevel", "error", "-y",
+ "-f", "lavfi", "-i", "color=c=red:s=320x240:r=24:d=1.5",
+ "-f", "lavfi", "-i", "color=c=blue:s=320x240:r=24:d=1.5",
+ "-filter_complex", "[0:v][1:v]concat=n=2:v=1:a=0",
+ "-c:v", "libx264", "-preset", "veryfast", "-pix_fmt", "yuv420p", path,
+ ], { encoding: "utf8" });
+ assert.equal(result.status, 0, result.stderr);
+ return path;
+}
+
+test("tile frame counts follow the measured bounds", () => {
+ assert.equal(tileFrameCount(0.5), 4);
+ assert.equal(tileFrameCount(4), 6);
+ assert.equal(tileFrameCount(30), 9);
+});
+
+test("media help names every command and no state", () => {
+ const out = io();
+ runMediaCli(["media"], out.io);
+ for (const name of ["probe", "cut", "frames", "tile", "shots", "fetch"]) assert.match(out.text(), new RegExp(`hypit media ${name}`));
+ assert.match(out.text(), /no state/);
+});
+
+test("probe, cut, frames, tile and shots read one file and write only what --to names", { skip: !ffmpeg && "ffmpeg is not installed" }, async () => {
+ const work = await mkdtemp(join(tmpdir(), "hypit-media-"));
+ try {
+ const source = await sample(work);
+ const info = await probeMedia(source);
+ assert.equal(info.width, 320);
+ assert.equal(info.height, 240);
+ assert.equal(info.hasAudio, false);
+ assert.ok(Math.abs(info.duration - 3) < 0.1, `duration ${info.duration}`);
+
+ const probe = io();
+ await runMediaCli(["media", "probe", source, "--json"], probe.io, work);
+ assert.equal((JSON.parse(probe.text()) as { width: number }).width, 320);
+
+ const cut = io();
+ await runMediaCli(["media", "cut", source, "--start", "0.5", "--end", "2", "--to", "out/piece.mp4"], cut.io, work);
+ const piece = await probeMedia(join(work, "out", "piece.mp4"));
+ assert.ok(Math.abs(piece.duration - 1.5) < 0.15, `cut duration ${piece.duration}`);
+ await assert.rejects(
+ runMediaCli(["media", "cut", source, "--start", "0", "--end", "1", "--to", "out/piece.mp4"], io().io, work),
+ /already exists/,
+ );
+
+ const frames = io();
+ await runMediaCli(["media", "frames", source, "--at", "0.2,2.5", "--to", "out/frames"], frames.io, work);
+ const written = (await readdir(join(work, "out", "frames"))).sort();
+ assert.deepEqual(written, ["frame-0_20s.jpg", "frame-2_50s.jpg"]);
+
+ const every = io();
+ await runMediaCli(["media", "frames", source, "--every", "1", "--to", "out/every", "--json"], every.io, work);
+ assert.equal((JSON.parse(every.text()) as { frames: unknown[] }).frames.length, 3);
+
+ const tile = io();
+ await runMediaCli(["media", "tile", source, "--to", "out/grid.jpg", "--json"], tile.io, work);
+ const grid = JSON.parse(tile.text()) as { frames: number; columns: number; rows: number; cellWidth: number };
+ assert.equal(grid.frames, tileFrameCount(info.duration));
+ assert.equal(grid.columns, 3);
+ assert.equal(grid.cellWidth, 320, "a grid never enlarges a small source");
+ assert.ok((await stat(join(work, "out", "grid.jpg"))).size > 0);
+
+ const bounds = await shotBoundaries(source, info.duration);
+ assert.equal(bounds.length, 2, JSON.stringify(bounds));
+ assert.ok(Math.abs(bounds[1]!.start - 1.5) < 0.2, `cut found at ${bounds[1]!.start}`);
+ const shots = io();
+ await runMediaCli(["media", "shots", source], shots.io, work);
+ assert.match(shots.text(), /2 pixel-jump stretches/);
+ } finally {
+ await rm(work, { recursive: true, force: true });
+ }
+});
+
+test("fetch refuses anything but an http link and an explicit video destination", async () => {
+ await assert.rejects(runMediaCli(["media", "fetch", "./local.mp4", "--to", "x.mp4"], io().io), /http or https link/);
+ await assert.rejects(runMediaCli(["media", "fetch", "https://example.com/v", "--to", "x.txt"], io().io, tmpdir()), /must end in/);
+});
diff --git a/packages/video-cli/test/vocabulary.test.ts b/packages/video-cli/test/vocabulary.test.ts
new file mode 100644
index 00000000..171906b7
--- /dev/null
+++ b/packages/video-cli/test/vocabulary.test.ts
@@ -0,0 +1,64 @@
+import assert from "node:assert/strict";
+import { dirname, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+import test from "node:test";
+
+import type { CliIo } from "@hypit/cli";
+
+import { listPackages, listSurfaces, runVocabularyCli, visualSchema } from "../src/vocabulary.js";
+
+const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
+
+function io(): { readonly io: CliIo; text: () => string } {
+ let out = "";
+ return {
+ io: {
+ write: (chunk) => { out += chunk; },
+ setExitCode: () => {},
+ readSecret: async () => "",
+ terminal: { isTTY: false, color: false, unicode: false, columns: 100 },
+ },
+ text: () => out,
+ };
+}
+
+test("the listing sees every activatable package from the repository root", async () => {
+ const listing = await listPackages(repositoryRoot);
+ const pipeline = listing.find((item) => item.name === "@hypit/media-pipeline");
+ assert.ok(pipeline !== undefined, "media-pipeline is listed");
+ assert.ok(pipeline.tags.includes("StillVideo"), `tags: ${pipeline.tags.join(", ")}`);
+ assert.equal(pipeline.unreadable, undefined);
+});
+
+test("a named package answers with its own Surfaces only", async () => {
+ const surfaces = await listSurfaces(repositoryRoot, ["@hypit/media-pipeline"], ["StillVideo"]);
+ assert.equal(surfaces.length, 1);
+ const [still] = surfaces;
+ assert.equal(still!.package, "@hypit/media-pipeline");
+ assert.equal(still!.tag, "StillVideo");
+ const vocabulary = still!.vocabulary as { attributes: readonly { name: string }[] };
+ assert.ok(vocabulary.attributes.some((attribute) => attribute.name === "duration"));
+ assert.match(still!.readme ?? "", /media-pipeline\/README\.md$/);
+});
+
+test("the visual schema prints every shape and refuses an unknown one", () => {
+ const all = visualSchema();
+ assert.ok(all.shapes.some((item) => item.shape === "text"));
+ assert.ok(all.rules.length > 0);
+ assert.equal(visualSchema("box").shapes.length, 1);
+ assert.throws(() => visualSchema("sprite"), /shape must be one of/);
+});
+
+test("the command prints human and JSON views", async () => {
+ const human = io();
+ await runVocabularyCli(["vocabulary", "@hypit/media-pipeline", "--tag", "StillVideo"], human.io, repositoryRoot);
+ assert.match(human.text(), /<StillVideo>/);
+ assert.match(human.text(), /duration/);
+
+ const json = io();
+ await runVocabularyCli(["vocabulary", "--json"], json.io, repositoryRoot);
+ const parsed = JSON.parse(json.text()) as { packages: readonly { name: string }[] };
+ assert.ok(parsed.packages.some((item) => item.name === "@hypit/media-pipeline"));
+
+ await assert.rejects(runVocabularyCli(["vocabulary", "--tag", "x"], io().io, repositoryRoot), /name the package first/);
+});
diff --git a/packages/yt-dlp/src/download.ts b/packages/yt-dlp/src/download.ts
index 05966142..fedaf89f 100644
--- a/packages/yt-dlp/src/download.ts
+++ b/packages/yt-dlp/src/download.ts
@@ -1,45 +1,30 @@
/**
- * Fetch a reference video that lives at a link rather than on disk.
+ * Fetch a video that lives at a link rather than on disk.
*
- * A reconstruction starts from a video, and most of the videos anybody wants to reconstruct are on
- * TikTok, YouTube, Instagram or Bilibili rather than in a folder. `yt-dlp` is what turns one into a
- * file; everything after that — shot detection, the transcript, the observations, the comparison —
- * reads the file and never learns where it came from.
- *
- * The download is cached by the link. Re-running a route that was interrupted, or preparing the same
- * reference twice, reaches the same bytes at the same path, which is what keeps the reference id
- * stable: it is derived from the file's path, size and modification time, so a second download to a
- * fresh temporary directory would present the same video as a different reference and throw away
- * every observation already paid for.
+ * Most of the videos anybody wants to reconstruct are on TikTok, YouTube, Instagram or Bilibili
+ * rather than in a folder. `yt-dlp` is what turns one into a file; everything after that reads the
+ * file and never learns where it came from.
*
* The tool is a pinned Python dependency under `services/yt-dlp`, reached through `uv`, and not a
* binary the machine happens to carry. `yt-dlp` releases constantly because it is chasing sites that
- * keep changing, so an unpinned copy makes the same link fetch differently on two machines — which
- * would make a reconstruction unreproducible for a reason that has nothing to do with the video. This
- * is the same shape WhisperX and OpenCV already use for their Python programs.
+ * keep changing, so an unpinned copy makes the same link fetch differently on two machines. This is
+ * the same shape WhisperX and OpenCV already use for their Python programs.
*/
import { spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
-import { mkdir, readFile, readdir, writeFile } from "node:fs/promises";
-import { dirname, join } from "node:path";
+import { mkdtemp, readdir, rename, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { dirname, extname, join } from "node:path";
import { fileURLToPath } from "node:url";
-/** What a downloaded reference is, once it is a file like any other. */
-export type DownloadedReference = {
- readonly path: string;
- readonly url: string;
- /** True when the bytes were already on disk from an earlier run. */
- readonly cached: boolean;
-};
-
/**
* Whether this is a link to fetch rather than a path to open.
*
* Only `http` and `https`. A Windows path opens as a URL with a single-letter scheme (`c:\clip.mp4`
- * parses with protocol `c:`), and a route that read that as a link would hand the whole path to
- * `yt-dlp` and report a network failure for a file sitting on the disk.
+ * parses with protocol `c:`), and reading that as a link would hand the whole path to `yt-dlp` and
+ * report a network failure for a file sitting on the disk.
*/
-export function isReferenceUrl(value: string): boolean {
+export function isVideoUrl(value: string): boolean {
try {
const parsed = new URL(value.trim());
return parsed.protocol === "http:" || parsed.protocol === "https:";
@@ -49,10 +34,8 @@ export function isReferenceUrl(value: string): boolean {
}
/**
- * The uv project holding the pinned `yt-dlp`.
- *
- * Found by walking up from this module rather than from the working directory, which is wherever the
- * author happened to run the command from and is routinely outside the tree.
+ * The uv project holding the pinned `yt-dlp`, found by walking up from this module rather than from
+ * the working directory, which is wherever the author happened to run the command from.
*/
function serviceProject(): string {
let directory = dirname(fileURLToPath(import.meta.url));
@@ -67,79 +50,46 @@ function serviceProject(): string {
}
}
-/** Find or allocate one readable cache directory whose `source.url` records the original locator. */
-async function cacheDirectory(root: string, url: string): Promise<string> {
- await mkdir(root, { recursive: true });
- const entries = await readdir(root, { withFileTypes: true });
- for (const entry of entries) {
- if (!entry.isDirectory() || !/^reference-[1-9][0-9]*$/u.test(entry.name)) continue;
- const directory = join(root, entry.name);
- if (await readFile(join(directory, "source.url"), "utf8").catch(() => undefined) === url) return directory;
- }
- const numbers = entries.flatMap((entry) => {
- const match = /^reference-([1-9][0-9]*)$/u.exec(entry.name);
- return match === null ? [] : [Number(match[1])];
- });
- const directory = join(root, `reference-${Math.max(0, ...numbers) + 1}`);
- await mkdir(directory, { recursive: false });
- await writeFile(join(directory, "source.url"), url, "utf8");
- return directory;
-}
-
-async function existingDownload(directory: string): Promise<string | undefined> {
- const entries = await readdir(directory).catch(() => []);
- // `yt-dlp` writes `.part` files while it works, so a directory holding only those is an interrupted
- // download rather than a finished one, and it is fetched again.
- const finished = entries.filter((name) => !name.endsWith(".part") && /\.(mp4|mkv|webm|mov)$/iu.test(name));
- return finished.length === 0 ? undefined : join(directory, finished.sort()[0]!);
-}
-
/**
- * Fetch one video and return the file it landed in.
+ * Fetch one video into exactly the file the caller named.
*
- * The format is asked for as a single muxed file rather than the best video and audio separately,
- * because everything downstream probes one file for both streams. `--no-playlist` keeps a link that
- * happens to sit inside a playlist from fetching the playlist.
+ * Video and audio are asked for together and muxed: sites serve the two separately now, so the best
+ * single file resolves to a video-only stream, and a video with no audio has no transcript. H.264 at
+ * 1080 is preferred rather than required; a `height<=1080` filter would refuse a link that offers
+ * nothing under the bound. `--no-playlist` keeps a link inside a playlist from fetching the playlist.
+ * The container follows the target's extension.
*/
-export async function downloadReferenceVideo(url: string, root: string): Promise<DownloadedReference> {
- const directory = await cacheDirectory(root, url);
-
- const held = await existingDownload(directory);
- if (held !== undefined) return { path: held, url, cached: true };
-
- const result = spawnSync("uv", [
- "run", "--project", serviceProject(), "--frozen", "yt-dlp",
- "--no-playlist",
- "--no-progress",
- "--quiet",
- // Video and audio together, muxed. Sites serve the two separately now, so asking for the best
- // single file resolves to a video-only stream — and a reference with no audio has no transcript,
- // which is half of what the observers read. ffmpeg does the muxing and is already required by
- // everything that opens this file next.
- "--format", "bv*+ba/b",
- "--merge-output-format", "mp4",
- // Prefer H.264 at 1080 rather than requiring either. Nothing downstream reads above 720 — the
- // analysis video, the representative frames and the tails are all `scale='min(720,iw)'` — so
- // fetching a 4K source spends several hundred megabytes to make a 720p working copy. A
- // `height<=1080` filter would refuse outright when a link offers nothing under the bound, which
- // is a download that fails rather than one that is merely larger; a sort expresses the preference
- // and still resolves to whatever the link actually has.
- "--format-sort", "res:1080,vcodec:h264",
- "--output", join(directory, "%(id)s.%(ext)s"),
- url,
- ], { encoding: "utf8", windowsHide: true, timeout: 900_000 });
-
- if (result.error !== undefined && (result.error as NodeJS.ErrnoException).code === "ENOENT") {
- throw new Error(
- "uv is not installed, and a reference given as a link is fetched by a pinned yt-dlp that uv "
- + "installs. It is the same tool the WhisperX and OpenCV programs need. Install it "
- + "(https://docs.astral.sh/uv/), or download the video yourself and pass the path instead.");
- }
- if (result.status !== 0) {
- throw new Error(`yt-dlp could not fetch ${url}: ${(result.stderr ?? "").trim().slice(-2000)}`);
+export async function downloadVideo(url: string, target: string): Promise<void> {
+ const container = extname(target).slice(1).toLowerCase();
+ if (!["mp4", "mkv", "webm", "mov"].includes(container)) {
+ throw new Error(`${target} must end in .mp4, .mkv, .webm or .mov`);
}
+ const work = await mkdtemp(join(tmpdir(), "hypit-fetch-"));
+ try {
+ const result = spawnSync("uv", [
+ "run", "--project", serviceProject(), "--frozen", "yt-dlp",
+ "--no-playlist", "--no-progress", "--quiet",
+ "--format", "bv*+ba/b",
+ "--merge-output-format", container,
+ "--format-sort", "res:1080,vcodec:h264",
+ "--output", join(work, "video.%(ext)s"),
+ url,
+ ], { encoding: "utf8", windowsHide: true, timeout: 900_000 });
- const written = await existingDownload(directory);
- if (written === undefined) throw new Error(`yt-dlp reported success but wrote no video for ${url}`);
- return { path: written, url, cached: false };
+ if (result.error !== undefined && (result.error as NodeJS.ErrnoException).code === "ENOENT") {
+ throw new Error(
+ "uv is not installed, and a link is fetched by a pinned yt-dlp that uv installs. It is the same "
+ + "tool the WhisperX and OpenCV programs need. Install it (https://docs.astral.sh/uv/), or download "
+ + "the video yourself and pass the path instead.");
+ }
+ if (result.status !== 0) {
+ throw new Error(`yt-dlp could not fetch ${url}: ${(result.stderr ?? "").trim().slice(-2000)}`);
+ }
+ const finished = (await readdir(work)).filter((name) => !name.endsWith(".part"));
+ const [file] = finished.sort();
+ if (file === undefined) throw new Error(`yt-dlp reported success for ${url} but wrote no file`);
+ await rename(join(work, file), target);
+ } finally {
+ await rm(work, { recursive: true, force: true });
+ }
}
diff --git a/packages/yt-dlp/src/index.ts b/packages/yt-dlp/src/index.ts
index f659c907..7581402c 100644
--- a/packages/yt-dlp/src/index.ts
+++ b/packages/yt-dlp/src/index.ts
@@ -1 +1 @@
-export { downloadReferenceVideo, isReferenceUrl } from "./download.js";
+export { downloadVideo, isVideoUrl } from "./download.js";
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 7a3095c8..942d1a32 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -2399,94 +2399,6 @@ importers:
specifier: workspace:*
version: link:../spatial
- packages/reference-video-tools:
- dependencies:
- '@hypit/composition':
- specifier: workspace:*
- version: link:../composition
- '@hypit/core':
- specifier: workspace:*
- version: link:../core
- '@hypit/driver-node':
- specifier: workspace:*
- version: link:../driver-node
- '@hypit/gemini':
- specifier: workspace:*
- version: link:../gemini
- '@hypit/hyperframes':
- specifier: workspace:*
- version: link:../hyperframes
- '@hypit/markup':
- specifier: workspace:*
- version: link:../markup
- '@hypit/media':
- specifier: workspace:*
- version: link:../media
- '@hypit/model-kit':
- specifier: workspace:*
- version: link:../model-kit
- '@hypit/package-loader-node':
- specifier: workspace:*
- version: link:../package-loader-node
- '@hypit/preview-mock':
- specifier: workspace:*
- version: link:../preview-mock
- '@hypit/program-space':
- specifier: workspace:*
- version: link:../program-space
- '@hypit/protocol':
- specifier: workspace:*
- version: link:../protocol
- '@hypit/provider-media-local':
- specifier: workspace:*
- version: link:../provider-media-local
- '@hypit/run-markup':
- specifier: workspace:*
- version: link:../run-markup
- '@hypit/runtime':
- specifier: workspace:*
- version: link:../runtime
- '@hypit/script':
- specifier: workspace:*
- version: link:../script
- '@hypit/speech':
- specifier: workspace:*
- version: link:../speech
- '@hypit/speech-evidence':
- specifier: workspace:*
- version: link:../speech-evidence
- '@hypit/studio':
- specifier: workspace:*
- version: link:../studio
- '@hypit/svs':
- specifier: workspace:*
- version: link:../svs
- '@hypit/text':
- specifier: workspace:*
- version: link:../text
- '@hypit/video-cli':
- specifier: workspace:*
- version: link:../video-cli
- '@hypit/visual-ir':
- specifier: workspace:*
- version: link:../visual-ir
- '@hypit/whisperx':
- specifier: workspace:*
- version: link:../whisperx
- '@hypit/yt-dlp':
- specifier: workspace:*
- version: link:../yt-dlp
- puppeteer-core:
- specifier: 25.5.0
- version: 25.5.0
- tsx:
- specifier: 4.21.0
- version: 4.21.0
- devDependencies:
- '@types/node':
- specifier: 24.10.1
- version: 24.10.1
-
packages/render-hyperframes:
dependencies:
'@hypit/artifact':
@@ -3373,6 +3285,9 @@ importers:
'@hypit/compiler-markup-node':
specifier: workspace:*
version: link:../compiler-markup-node
+ '@hypit/composition':
+ specifier: workspace:*
+ version: link:../composition
'@hypit/credential-store-os':
specifier: workspace:*
version: link:../credential-store-os
@@ -3394,6 +3309,9 @@ importers:
'@hypit/mimo-tts':
specifier: workspace:*
version: link:../mimo-tts
+ '@hypit/model-kit':
+ specifier: workspace:*
+ version: link:../model-kit
'@hypit/package-loader-node':
specifier: workspace:*
version: link:../package-loader-node
@@ -3421,12 +3339,18 @@ importers:
'@hypit/text':
specifier: workspace:*
version: link:../text
+ '@hypit/visual-ir':
+ specifier: workspace:*
+ version: link:../visual-ir
'@hypit/whisperx':
specifier: workspace:*
version: link:../whisperx
'@hypit/workspace-fs-node':
specifier: workspace:*
version: link:../workspace-fs-node
+ '@hypit/yt-dlp':
+ specifier: workspace:*
+ version: link:../yt-dlp
packages/visual-ir:
dependencies: