Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

refactor: ground reference understanding in timed frames and semantic intent

kashorin34b2229e662b
diff --git a/docs/guide/packages.md b/docs/guide/packages.md
index 9ea53847..2383ec6e 100644
--- a/docs/guide/packages.md
+++ b/docs/guide/packages.md
@@ -125,7 +125,7 @@ never on exact-model packages or the CLI.
```text
@hypit/provider-kie KIE generation plus background removal
-@hypit/provider-hypihub HypiHub generation, Gemini VLM and WhisperX gateway
+@hypit/provider-hypihub HypiHub generation, WhisperX gateway
@hypit/provider-media-local local ffprobe/ffmpeg
@hypit/provider-whisperx-local local WhisperX service
@hypit/provider-hyperframes-local local Chrome rendering
diff --git a/docs/guide/runtime.md b/docs/guide/runtime.md
index 10ff9cf3..183a419c 100644
--- a/docs/guide/runtime.md
+++ b/docs/guide/runtime.md
@@ -213,7 +213,7 @@ hypit runtime down
`runtime init` writes the video Distribution's starter Profile to `hypit.runtime.json` and selects
it for the resolved project. It refuses to overwrite an existing file. This is a local file operation:
it installs no package, contacts no service, requests no credential and starts no Worker. The official
-starter selects HypiHub for remote generation, Gemini and WhisperX, plus local media processing and
+starter selects HypiHub for remote generation, WhisperX, plus local media processing and
HyperFrames rendering. This is a Distribution default, not a Core rule; edit the Profile or select a
different one when using BYOK or local Providers.
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 1e0a7270..043d9219 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -86,7 +86,7 @@ sequence depends on the video, but it generally includes:
1. **Breaking the request into shots.** The Agent identifies the spoken sections, visual beats,
transitions, captions, B-roll opportunities and any persistent elements such as a ranking board.
-2. **Analyzing the reference or brief.** For a clone, it uses Gemini and the available media tools to
+2. **Analyzing the reference or brief.** For a clone, it watches the reference and uses timed frames and transcripts to
inspect timing, composition, text, speakers and visual continuity. For an original, it resolves
the same questions from your description and the selected creative direction.
3. **Resolving details.** The Agent uses the reference, brief and observation results to settle
diff --git a/docs/quickstart/run.md b/docs/quickstart/run.md
index ecb712ef..6451f0ef 100644
--- a/docs/quickstart/run.md
+++ b/docs/quickstart/run.md
@@ -251,7 +251,7 @@ variables referenced by the selected Runtime Profile:
| Variable | Provider/use |
|---|---|
-| HypiHub OAuth | HypiHub paid generation, Gemini VLM and WhisperX alignment; run `hypit auth login hypihub.default --runtime hypit.runtime.json` |
+| HypiHub OAuth | HypiHub paid generation, WhisperX alignment; run `hypit auth login hypihub.default --runtime hypit.runtime.json` |
| `KIE_API_KEY` | Explicit KIE Provider only |
| `MIMO_API_KEY` | Xiaomi MiMo Voice Design or Voice Clone, only when the official Endpoint is explicitly selected |
diff --git a/docs/zh/guide/packages.md b/docs/zh/guide/packages.md
index 0901ff10..5e26d396 100644
--- a/docs/zh/guide/packages.md
+++ b/docs/zh/guide/packages.md
@@ -117,7 +117,7 @@ Frontend;不存在一个认识全部语法的中央解析器。
```text
@hypit/provider-kie KIE 生成与去背景
-@hypit/provider-hypihub HypiHub 生成、Gemini VLM 与 WhisperX 网关
+@hypit/provider-hypihub HypiHub 生成、WhisperX 网关
@hypit/provider-media-local local ffprobe/ffmpeg
@hypit/provider-whisperx-local local WhisperX service
@hypit/provider-hyperframes-local local Chrome rendering
diff --git a/docs/zh/guide/runtime.md b/docs/zh/guide/runtime.md
index 0ced6a4d..883aaac1 100644
--- a/docs/zh/guide/runtime.md
+++ b/docs/zh/guide/runtime.md
@@ -191,7 +191,7 @@ hypit runtime down
`runtime init` 会把视频 Distribution 提供的起始 Profile 写入 `hypit.runtime.json`,并为已经解析
出的项目选择它;已有文件一律拒绝覆盖。这只是本地文件操作:不安装包、不连接服务、不索取凭据,
-也不启动 Worker。官方起始 Profile 使用 HypiHub 提供远程生成、Gemini 与 WhisperX,使用本地
+也不启动 Worker。官方起始 Profile 使用 HypiHub 提供远程生成、WhisperX,使用本地
Endpoint 处理媒体并通过 HyperFrames 渲染。这是 Distribution 的开箱选择,不是 Core 规则;使用
BYOK 或本地 Provider 时可以编辑 Profile 或选择另一份 Profile。
diff --git a/docs/zh/quickstart.md b/docs/zh/quickstart.md
index 8b744074..7f866b9e 100644
--- a/docs/zh/quickstart.md
+++ b/docs/zh/quickstart.md
@@ -67,7 +67,7 @@ npx skills add hypit-ai/hypit -g
收到 brief 后,Agent 会在不要求你编写源码的情况下自动推进项目。制作过程中不会询问任何问题,你只需等待 Studio mock。具体顺序会因视频而不同,但通常包括:
1. **拆分镜头。** Agent 识别口播段落、视觉节拍、转场、字幕、B-roll 机会,以及 ranking 板等需要持续保持的元素。
-2. **分析参考视频或 brief。** 复刻路线会使用 Gemini 和可用的媒体工具检查节奏、构图、文字、说话人和视觉连续性;原创路线则根据你的描述和确定的创作方向回答同样的问题。
+2. **分析参考视频或 brief。** 复刻路线会观看参考视频,并用带时间的帧图与转录检查节奏、构图、文字、说话人和视觉连续性;原创路线则根据你的描述和确定的创作方向回答同样的问题。
3. **自行确定细节。** Agent 会根据参考视频、brief 和观察结果自动确定时机、语言、产品位置和主持人处理方式。
4. **读取现有 package 声明。** 在自行发明实现之前,Agent 会检查项目和 Hypit 官方 package 中已有的组件。如果已有合适的字幕、ranking、主持人、B-roll 或渲染组件,就直接复用。
5. **必要时编写新 package。** 如果确实缺少所需的视觉行为,Agent 会创建一个足够小、可复用的新组件,并记录它的使用方式。你不需要自行设计 package 接口。
diff --git a/docs/zh/quickstart/run.md b/docs/zh/quickstart/run.md
index c7736a50..dbfc3290 100644
--- a/docs/zh/quickstart/run.md
+++ b/docs/zh/quickstart/run.md
@@ -216,7 +216,7 @@ CLI 必须先确定项目:显式 `--workspace` 直接给出边界;否则使
| 变量 | Provider / 用途 |
|---|---|
-| HypiHub OAuth | HypiHub 付费生成、Gemini VLM 与 WhisperX 对齐;运行 `hypit auth login hypihub.default --runtime hypit.runtime.json` 并在 https://hypit.ai 登录 |
+| HypiHub OAuth | HypiHub 付费生成、WhisperX 对齐;运行 `hypit auth login hypihub.default --runtime hypit.runtime.json` 并在 https://hypit.ai 登录 |
| `KIE_API_KEY` | 仅在显式选择 KIE Provider 时使用 |
| `MIMO_API_KEY` | Xiaomi MiMo 音色设计或音色克隆;只有明确选择官方 Endpoint 时才需要 |
diff --git a/package.json b/package.json
index 1784f820..af0ebde0 100644
--- a/package.json
+++ b/package.json
@@ -164,7 +164,6 @@
"@hypit/film": "workspace:*",
"@hypit/film-studio": "workspace:*",
"@hypit/fonts-open": "workspace:*",
- "@hypit/gemini": "workspace:*",
"@hypit/generation": "workspace:*",
"@hypit/gpt-image": "workspace:*",
"@hypit/gpt-image-kits": "workspace:*",
@@ -189,7 +188,6 @@
"@hypit/provider-image-opencv-local": "workspace:*",
"@hypit/provider-kie": "workspace:*",
"@hypit/provider-media-local": "workspace:*",
- "@hypit/provider-vertex": "workspace:*",
"@hypit/provider-whisperx-local": "workspace:*",
"@hypit/provider-xiaomi-mimo": "workspace:*",
"@hypit/ranking": "workspace:*",
diff --git a/packages/gemini/README.md b/packages/gemini/README.md
deleted file mode 100644
index ede2464f..00000000
--- a/packages/gemini/README.md
+++ /dev/null
@@ -1,22 +0,0 @@
-# `@hypit/gemini`
-
-Provider-neutral Gemini multimodal observation requests.
-
-`<gemini:Generate>` accepts a system instruction, a text prompt and any number of image, video or
-audio Artifact references. The model package owns the semantic request and exact model capability;
-the selected Runtime Endpoint owns credentials, uploads and the service-specific wire format.
-
-```xml
-<gemini:Generate id="describe" model="gemini-3.1-pro"
- instruction={instruction} prompt={prompt}>
- <gemini:Reference media={reference-video}/>
-</gemini:Generate>
-```
-
-The answer is a nominal `VisualObservation`, published as `<id>.observation`. It deliberately is not
-`@hypit/text` Prompt input: Gemini supplies visual evidence to the author; it does not silently write
-production prompts inside a Build. Reference-video work should normally call the creation-time
-`hypit observe` command, inspect the observation, and then author the production Source.
-
-Switching between
-`@hypit/provider-hypihub` and `@hypit/provider-vertex` requires only a Runtime Profile change.
diff --git a/packages/gemini/package.json b/packages/gemini/package.json
deleted file mode 100644
index 700a48c4..00000000
--- a/packages/gemini/package.json
+++ /dev/null
@@ -1,17 +0,0 @@
-{
- "name": "@hypit/gemini",
- "version": "0.0.0-dev",
- "license": "SEE LICENSE IN LICENSE",
- "private": true,
- "type": "module",
- "exports": { ".": "./src/index.ts" },
- "hypit": { "activation": "./src/activation.ts" },
- "dependencies": {
- "@hypit/artifact": "workspace:*",
- "@hypit/component-kit": "workspace:*",
- "@hypit/elaborator": "workspace:*",
- "@hypit/markup": "workspace:*",
- "@hypit/protocol": "workspace:*",
- "@hypit/text": "workspace:*"
- }
-}
diff --git a/packages/gemini/src/activation.ts b/packages/gemini/src/activation.ts
deleted file mode 100644
index 38eaa234..00000000
--- a/packages/gemini/src/activation.ts
+++ /dev/null
@@ -1,22 +0,0 @@
-import { createMarkupSurfaceHostFacet } from "@hypit/markup";
-
-import {
- decodeGeminiGenerateSurface,
- geminiComponent,
- geminiManifest,
- geminiMarkupSurfaces,
- geminiModuleRef,
-} from "./index.js";
-
-export const hypitPackage = {
- format: "hypit.node-package@1" as const,
- modules: [{ manifest: geminiManifest }],
- components: [geminiComponent],
- hostFacets: [createMarkupSurfaceHostFacet({
- module: geminiModuleRef,
- declaration: geminiMarkupSurfaces[0],
- handler: decodeGeminiGenerateSurface,
- })],
-};
-
-export default hypitPackage;
diff --git a/packages/gemini/src/component.ts b/packages/gemini/src/component.ts
deleted file mode 100644
index deaf03ba..00000000
--- a/packages/gemini/src/component.ts
+++ /dev/null
@@ -1,142 +0,0 @@
-import { plannedNeedInputs } from "@hypit/component-kit";
-import type { ComponentPackage, PlannedNeedFacet, ProducerHandlerContext } from "@hypit/component-kit";
-import { canonicalize } from "@hypit/protocol";
-import type { BuildState, CanonicalValue, StoredValue } from "@hypit/protocol";
-import type { Text } from "@hypit/text";
-import { verifyText } from "@hypit/text";
-
-import { geminiCapabilities, geminiModels, geminiProducers, geminiTypes } from "./manifest.js";
-import { sealGeminiRequest, verifyGeminiRequest } from "./request.js";
-import type { GeminiRequest } from "./request.js";
-import { verifyVisualObservation } from "./observation.js";
-
-function inline<T>(value: StoredValue | undefined, subject: string): T {
- if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);
- return value.value as unknown as T;
-}
-
-function text(value: StoredValue | undefined, subject: string): string {
- const result = inline<Text>(value, subject);
- verifyText(result);
- return result.value;
-}
-
-function plannedGeminiText(state: BuildState, requestStep: string): {
- readonly instruction: string;
- readonly prompt: string;
-} | undefined {
- const producedBy = new Map(state.plan.steps.flatMap((step) =>
- Object.values(step.outputs).map((record) => [record, step] as const)));
- const request = state.plan.steps.find((step) => step.id === requestStep);
- let draft = request?.inputs.request;
- const seen = new Set<string>();
- while (draft !== undefined && !seen.has(draft)) {
- seen.add(draft);
- const existing = state.records.find((record) => record.id === draft)?.value;
- if (existing?.kind === "inline") {
- const value = existing.value as Readonly<Record<string, CanonicalValue>>;
- if (typeof value.instruction === "string" && typeof value.prompt === "string") {
- return { instruction: value.instruction, prompt: value.prompt };
- }
- }
- const producer = producedBy.get(draft);
- if (producer === undefined) return undefined;
- if (producer.producer.name === geminiProducers.finalize.name) {
- draft = producer.inputs.draft;
- continue;
- }
- if (producer.producer.name === geminiProducers.bindMedia.name) {
- draft = producer.inputs.draft;
- continue;
- }
- if (producer.producer.name !== geminiProducers.start.name) return undefined;
- const readText = (input: string): string | undefined => {
- const record = producer.inputs[input];
- const stored = record === undefined ? undefined : state.records.find((item) => item.id === record)?.value;
- if (stored?.kind !== "inline" || stored.value === null || typeof stored.value !== "object" || Array.isArray(stored.value)) return undefined;
- const value = (stored.value as Readonly<Record<string, CanonicalValue>>).value;
- return typeof value === "string" ? value : undefined;
- };
- const instruction = readText("instruction");
- const prompt = readText("prompt");
- return instruction === undefined || prompt === undefined ? undefined : { instruction, prompt };
- }
- return undefined;
-}
-
-const plannedGeminiNeeds: readonly PlannedNeedFacet[] = geminiModels.map((model) => ({
- producer: geminiProducers[model],
- port: "observation",
- capability: geminiCapabilities[model],
- plan({ state, step }) {
- const authored = plannedGeminiText(state, step);
- if (authored === undefined) return undefined;
- return {
- constraints: authored,
- pendingInputs: plannedNeedInputs(state, step),
- };
- },
- present(specification) {
- const fields = specification.constraints as Readonly<Record<string, CanonicalValue>>;
- return {
- fields: {
- ...(typeof fields.instruction === "string" ? { instruction: [fields.instruction] } : {}),
- ...(typeof fields.prompt === "string" ? { prompt: [fields.prompt] } : {}),
- },
- references: {},
- };
- },
-}));
-
-export const geminiComponent = {
- validators: [
- { type: geminiTypes.request, handler: ({ value }) => verifyGeminiRequest(inline(value, "Gemini request")) },
- { type: geminiTypes.draft, handler: ({ value }) => verifyGeminiRequest(inline(value, "Gemini request draft")) },
- { type: geminiTypes.visualObservation, handler: ({ value }) => verifyVisualObservation(inline(value, "Gemini Visual Observation")) },
- ],
- producers: [
- {
- producer: geminiProducers.start,
- handler: ({ inputs }: ProducerHandlerContext) => ({
- outputs: { draft: { kind: "inline", value: canonicalize(sealGeminiRequest({
- instruction: text(inputs.instruction?.value, "Gemini instruction"),
- prompt: text(inputs.prompt?.value, "Gemini prompt"),
- media: [],
- })) } },
- needs: {},
- }),
- },
- {
- producer: geminiProducers.bindMedia,
- handler: ({ inputs }: ProducerHandlerContext) => {
- const draft = inline<GeminiRequest>(inputs.draft?.value, "Gemini request draft");
- verifyGeminiRequest(draft);
- const artifact = inputs.media?.value;
- if (artifact?.kind !== "blob") throw new Error("Gemini media must be a Blob Artifact");
- return {
- outputs: { draft: { kind: "inline", value: canonicalize(sealGeminiRequest({
- ...draft, media: [...draft.media, { artifact }],
- })) } },
- needs: {},
- };
- },
- },
- {
- producer: geminiProducers.finalize,
- handler: ({ inputs }: ProducerHandlerContext) => {
- const draft = inline<GeminiRequest>(inputs.draft?.value, "Gemini request draft");
- verifyGeminiRequest(draft);
- return { outputs: { request: { kind: "inline", value: canonicalize(draft) } }, needs: {} };
- },
- },
- ...geminiModels.map((model) => ({
- producer: geminiProducers[model],
- handler: ({ inputs }: ProducerHandlerContext) => {
- const request = inline<GeminiRequest>(inputs.request?.value, `${model} request`);
- verifyGeminiRequest(request);
- return { outputs: {}, needs: { observation: canonicalize(request) } };
- },
- })),
- ],
- plannedNeeds: plannedGeminiNeeds,
-} satisfies ComponentPackage;
diff --git a/packages/gemini/src/fragment.ts b/packages/gemini/src/fragment.ts
deleted file mode 100644
index 14ec2929..00000000
--- a/packages/gemini/src/fragment.ts
+++ /dev/null
@@ -1,50 +0,0 @@
-import { artifactTypes } from "@hypit/artifact";
-import { sealGraphFragment } from "@hypit/elaborator";
-import type { FragmentOperation } from "@hypit/elaborator";
-import { textTypes } from "@hypit/text";
-
-import type { GeminiModel } from "./manifest.js";
-import { geminiProducers, geminiTypes } from "./manifest.js";
-
-const input = (name: string) => ({ kind: "fragment-input" as const, name });
-const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
-
-export function createGeminiFragment(model: GeminiModel, mediaCount: number) {
- if (!Number.isSafeInteger(mediaCount) || mediaCount < 0) throw new Error("Gemini media count is invalid");
- const inputs = [
- { name: "instruction", type: textTypes.text },
- { name: "prompt", type: textTypes.text },
- ...Array.from({ length: mediaCount }, (_item, index) => ({
- name: `media-${String(index + 1).padStart(4, "0")}`,
- type: artifactTypes.blob,
- })),
- ];
- const operations: FragmentOperation[] = [{
- id: "start-request", producer: geminiProducers.start,
- inputs: { instruction: input("instruction"), prompt: input("prompt") },
- result: { kind: "output" as const, name: "draft" },
- }];
- let draft = operation("start-request");
- for (let index = 0; index < mediaCount; index += 1) {
- const id = `bind-media-${String(index + 1).padStart(4, "0")}`;
- operations.push({
- id, producer: geminiProducers.bindMedia,
- inputs: { draft, media: input(`media-${String(index + 1).padStart(4, "0")}`) },
- result: { kind: "output" as const, name: "draft" },
- });
- draft = operation(id);
- }
- operations.push({
- id: "finalize-request", producer: geminiProducers.finalize, inputs: { draft },
- result: { kind: "output" as const, name: "request" },
- });
- operations.push({
- id: "generate", producer: geminiProducers[model], inputs: { request: operation("finalize-request") },
- result: { kind: "need" as const, name: "observation" },
- });
- return sealGraphFragment({
- inputs,
- operations,
- exports: [{ name: "observation", type: geminiTypes.visualObservation, root: operation("generate") }],
- });
-}
diff --git a/packages/gemini/src/index.ts b/packages/gemini/src/index.ts
deleted file mode 100644
index c4206471..00000000
--- a/packages/gemini/src/index.ts
+++ /dev/null
@@ -1,19 +0,0 @@
-export { geminiComponent } from "./component.js";
-export { createGeminiFragment } from "./fragment.js";
-export {
- geminiCapabilities,
- geminiDependency,
- geminiManifest,
- geminiMarkupSurfaces,
- geminiModels,
- geminiModuleRef,
- geminiProducers,
- geminiTypes,
-} from "./manifest.js";
-export type { GeminiModel } from "./manifest.js";
-export { sealGeminiRequest, verifyGeminiRequest } from "./request.js";
-export type { GeminiMediaPart, GeminiRequest } from "./request.js";
-export type { GeminiGenerateInput, GeminiInlinePart } from "./parts.js";
-export { sealVisualObservation, verifyVisualObservation } from "./observation.js";
-export type { VisualObservation } from "./observation.js";
-export { decodeGeminiGenerateSurface } from "./surface.js";
diff --git a/packages/gemini/src/manifest.ts b/packages/gemini/src/manifest.ts
deleted file mode 100644
index f68b3c41..00000000
--- a/packages/gemini/src/manifest.ts
+++ /dev/null
@@ -1,101 +0,0 @@
-import { artifactDependency, artifactTypes } from "@hypit/artifact";
-import type { CapabilityRef, ModuleManifest, ProducerRef, TypeRef } from "@hypit/protocol";
-import { textDependency, textTypes } from "@hypit/text";
-
-export const geminiModuleRef = { name: "@hypit/gemini", version: "1" } as const;
-
-export const geminiModels = ["gemini-3.1-pro"] as const;
-export type GeminiModel = typeof geminiModels[number];
-
-export const geminiTypes = {
- request: { module: geminiModuleRef, name: "GeminiRequest" },
- draft: { module: geminiModuleRef, name: "GeminiRequestDraft" },
- visualObservation: { module: geminiModuleRef, name: "VisualObservation" },
-} satisfies Record<string, TypeRef>;
-
-export const geminiCapabilities = Object.fromEntries(geminiModels.map((model) => [model, {
- module: geminiModuleRef,
- name: model,
-} satisfies CapabilityRef])) as Record<GeminiModel, CapabilityRef>;
-
-export const geminiProducers = {
- start: { module: geminiModuleRef, name: "start-request" },
- bindMedia: { module: geminiModuleRef, name: "bind-media" },
- finalize: { module: geminiModuleRef, name: "finalize-request" },
- ...Object.fromEntries(geminiModels.map((model) => [model, {
- module: geminiModuleRef,
- name: `request-${model}`,
- } satisfies ProducerRef])),
-} as { readonly start: ProducerRef; readonly bindMedia: ProducerRef; readonly finalize: ProducerRef } & Record<GeminiModel, ProducerRef>;
-
-export const geminiMarkupSurfaces = [{
- name: "generate", tag: "Generate", mode: "structured",
- outputs: [geminiTypes.draft],
- vocabulary: {
- summary: "Asks one exact Gemini model to answer a Text prompt with optional image, video or audio references.",
- attributes: [
- { name: "id", kind: "identifier", required: true,
- summary: "Names this request and its generated Text." },
- { name: "model", kind: "literal", required: true, values: [...geminiModels],
- summary: "Selects the exact Gemini capability without choosing its Provider." },
- { name: "instruction", kind: "reference", required: true, accepts: [textTypes.text],
- summary: "Selects the system instruction Text." },
- { name: "prompt", kind: "reference", required: true, accepts: [textTypes.text],
- summary: "Selects the user prompt Text." },
- ],
- children: [{ tag: "Reference", cardinality: "many",
- summary: "Attaches one image, video or audio Artifact to the request in authored order.",
- attributes: [{ name: "media", kind: "reference", required: true, accepts: [artifactTypes.blob],
- summary: "Selects the media Artifact Gemini reads." }] }],
- ports: [{ name: "observation", type: geminiTypes.visualObservation,
- summary: "What Gemini observed, addressed as `<id>.observation`; it is not generation Prompt Text." }],
- example: `<gemini:Generate id="describe" model="gemini-3.1-pro" instruction={instruction} prompt={prompt}>
- <gemini:Reference media={reference-video}/>
-</gemini:Generate>`,
- notes: [
- "The Source declares no API key, upload URL or Provider-specific file representation.",
- "Every Reference remains a graph edge; the selected Provider decides how its Artifact bytes travel.",
- ],
- },
-}] as const;
-
-export const geminiManifest: ModuleManifest = {
- format: "hypit.module@1",
- name: geminiModuleRef.name,
- version: geminiModuleRef.version,
- dependencies: [artifactDependency, textDependency],
- types: [
- { name: geminiTypes.request.name },
- { name: geminiTypes.draft.name },
- { name: geminiTypes.visualObservation.name },
- ],
- capabilities: geminiModels.map((model) => ({ name: model, returns: geminiTypes.visualObservation })),
- producers: [
- {
- name: geminiProducers.start.name,
- inputs: [{ name: "instruction", type: textTypes.text }, { name: "prompt", type: textTypes.text }],
- outputs: [{ name: "draft", type: geminiTypes.draft }],
- needs: [],
- },
- {
- name: geminiProducers.bindMedia.name,
- inputs: [{ name: "draft", type: geminiTypes.draft }, { name: "media", type: artifactTypes.blob }],
- outputs: [{ name: "draft", type: geminiTypes.draft }],
- needs: [],
- },
- {
- name: geminiProducers.finalize.name,
- inputs: [{ name: "draft", type: geminiTypes.draft }],
- outputs: [{ name: "request", type: geminiTypes.request }],
- needs: [],
- },
- ...geminiModels.map((model) => ({
- name: geminiProducers[model].name,
- inputs: [{ name: "request", type: geminiTypes.request }],
- outputs: [],
- needs: [{ name: "observation", capability: geminiCapabilities[model], returns: geminiTypes.visualObservation }],
- })),
- ],
-};
-
-export const geminiDependency = { module: geminiModuleRef } as const;
diff --git a/packages/gemini/src/observation.ts b/packages/gemini/src/observation.ts
deleted file mode 100644
index 1373eebd..00000000
--- a/packages/gemini/src/observation.ts
+++ /dev/null
@@ -1,18 +0,0 @@
-import { canonicalize } from "@hypit/protocol";
-
-/** What Gemini saw in the supplied media. It is evidence for the author, not production Text. */
-export type VisualObservation = { readonly text: string };
-
-export function verifyVisualObservation(value: unknown): asserts value is VisualObservation {
- if (value === null || typeof value !== "object" || Array.isArray(value)
- || typeof (value as { readonly text?: unknown }).text !== "string"
- || (value as { readonly text: string }).text.trim().length === 0) {
- throw new Error("Visual Observation must contain non-empty text");
- }
-}
-
-export function sealVisualObservation(text: string): VisualObservation {
- const result = canonicalize({ text }) as unknown as VisualObservation;
- verifyVisualObservation(result);
- return result;
-}
diff --git a/packages/gemini/src/parts.ts b/packages/gemini/src/parts.ts
deleted file mode 100644
index 23906715..00000000
--- a/packages/gemini/src/parts.ts
+++ /dev/null
@@ -1,9 +0,0 @@
-/** Provider-neutral inline parts used by embedded Gemini callers before a Runtime Artifact exists. */
-export type GeminiInlinePart =
- | { readonly text: string; readonly inlineData?: never }
- | { readonly inlineData: { readonly mimeType: string; readonly data: string }; readonly text?: never };
-
-export type GeminiGenerateInput = {
- readonly parts: readonly GeminiInlinePart[];
- readonly instruction: string;
-};
diff --git a/packages/gemini/src/request.ts b/packages/gemini/src/request.ts
deleted file mode 100644
index 6ed16dba..00000000
--- a/packages/gemini/src/request.ts
+++ /dev/null
@@ -1,43 +0,0 @@
-import { canonicalize } from "@hypit/protocol";
-import type { BlobRef } from "@hypit/protocol";
-
-export type GeminiMediaPart = { readonly artifact: BlobRef };
-export type GeminiRequest = {
- readonly instruction: string;
- readonly prompt: string;
- readonly media: readonly GeminiMediaPart[];
-};
-
-function assert(condition: unknown, message: string): asserts condition {
- if (!condition) throw new Error(message);
-}
-
-function object(value: unknown, subject: string): Record<string, unknown> {
- assert(value !== null && typeof value === "object" && !Array.isArray(value), `${subject} must be an object`);
- return value as Record<string, unknown>;
-}
-
-export function verifyGeminiRequest(value: unknown): asserts value is GeminiRequest {
- const request = object(value, "Gemini request");
- assert(typeof request.instruction === "string" && request.instruction.trim().length > 0,
- "Gemini request instruction must be non-empty text");
- assert(typeof request.prompt === "string" && request.prompt.trim().length > 0,
- "Gemini request prompt must be non-empty text");
- assert(Array.isArray(request.media), "Gemini request media must be an array");
- for (const [index, raw] of request.media.entries()) {
- const item = object(raw, `Gemini request media[${index}]`);
- const artifact = object(item.artifact, `Gemini request media[${index}].artifact`);
- assert(artifact.kind === "blob" && typeof artifact.resource === "string"
- && typeof artifact.size === "number" && Number.isSafeInteger(artifact.size)
- && typeof artifact.mediaType === "string",
- `Gemini request media[${index}].artifact must be a BlobRef`);
- assert(/^(?:image|video|audio)\//u.test(artifact.mediaType),
- `Gemini request media[${index}] must be image, video or audio media`);
- }
-}
-
-export function sealGeminiRequest(value: GeminiRequest): GeminiRequest {
- const result = canonicalize(value) as unknown as GeminiRequest;
- verifyGeminiRequest(result);
- return result;
-}
diff --git a/packages/gemini/src/surface.ts b/packages/gemini/src/surface.ts
deleted file mode 100644
index 5bba6d0e..00000000
--- a/packages/gemini/src/surface.ts
+++ /dev/null
@@ -1,73 +0,0 @@
-import { artifactTypes } from "@hypit/artifact";
-import type {
- MarkupAttributeValue,
- StructuredElement,
- StructuredSurfaceHandler,
- SurfaceResolvedReference,
-} from "@hypit/markup";
-import type { TypeRef } from "@hypit/protocol";
-import { textTypes } from "@hypit/text";
-
-import { createGeminiFragment } from "./fragment.js";
-import { geminiModels } from "./manifest.js";
-import type { GeminiModel } from "./manifest.js";
-
-function assert(condition: unknown, message: string): asserts condition { if (!condition) throw new Error(message); }
-function localName(value: string): string { return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value; }
-function sameType(left: TypeRef, right: TypeRef): boolean {
- return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
-}
-function reference(value: MarkupAttributeValue | undefined, subject: string,
- expected: TypeRef, resolveReference: (path: string) => SurfaceResolvedReference | undefined): SurfaceResolvedReference {
- assert(typeof value === "object" && value.kind === "reference", `${subject} must be a reference`);
- const resolved = resolveReference(value.path);
- assert(resolved !== undefined && sameType(resolved.type, expected), `${subject} has the wrong type`);
- return resolved;
-}
-function literal(element: StructuredElement, name: string): string {
- const value = element.attributes[name];
- assert(typeof value === "string" && value.trim().length > 0, `${element.name}.${name} must be non-empty text`);
- return value.trim();
-}
-
-export const decodeGeminiGenerateSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
- const allowed = ["id", "instruction", "model", "prompt"];
- const unknown = Object.keys(element.attributes).find((name) => !allowed.includes(name));
- assert(unknown === undefined, `${element.name} does not accept ${unknown}`);
- const id = literal(element, "id");
- const model = literal(element, "model");
- assert(geminiModels.includes(model as GeminiModel), `${element.name}.model is unsupported`);
- const instruction = reference(element.attributes.instruction, `${element.name}.instruction`, textTypes.text, resolveReference);
- const prompt = reference(element.attributes.prompt, `${element.name}.prompt`, textTypes.text, resolveReference);
- const media: SurfaceResolvedReference[] = [];
- for (const child of element.children) {
- if (child.kind === "text") {
- assert(child.value.trim().length === 0, `${element.name} accepts only Reference children`);
- continue;
- }
- assert(localName(child.name) === "Reference", `${element.name} accepts only Reference children`);
- assert(Object.keys(child.attributes).join(",") === "media", `${child.name} requires only media`);
- assert(!child.children.some((item) => item.kind === "element" || item.value.trim().length > 0), `${child.name} must be empty`);
- const resolved = reference(child.attributes.media, `${child.name}.media`, artifactTypes.blob, resolveReference);
- if (resolved.record?.value.kind === "blob") {
- assert(/^(?:image|video|audio)\//u.test(resolved.record.value.mediaType), `${child.name}.media must be image, video or audio`);
- }
- media.push(resolved);
- }
- const fragment = createGeminiFragment(model as GeminiModel, media.length);
- return {
- records: [],
- fragments: [fragment],
- components: [{
- id,
- fragment: fragment.id,
- inputs: {
- instruction: instruction.ref,
- prompt: prompt.ref,
- ...Object.fromEntries(media.map((item, index) => [`media-${String(index + 1).padStart(4, "0")}`, item.ref])),
- },
- outputs: { observation: `${id}.observation` },
- range: element.range,
- }],
- };
-};
diff --git a/packages/gemini/test/surface.test.ts b/packages/gemini/test/surface.test.ts
deleted file mode 100644
index 627e1654..00000000
--- a/packages/gemini/test/surface.test.ts
+++ /dev/null
@@ -1,39 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-import { fixtureResource } from "../../../test/fixture-resource.js";
-
-import { decodeGeminiGenerateSurface, geminiCapabilities } from "@hypit/gemini";
-import { artifactTypes } from "@hypit/artifact";
-import { textTypes } from "@hypit/text";
-
-test("Gemini Surface builds one provider-neutral need with ordered media edges", async () => {
- const fragmentResult = await decodeGeminiGenerateSurface({
- sourceName: "main.svml",
- element: {
- kind: "element", name: "gemini:Generate",
- attributes: {
- id: "read", model: "gemini-3.1-pro",
- instruction: { kind: "reference", path: "instruction" },
- prompt: { kind: "reference", path: "prompt" },
- },
- children: [{
- kind: "element", name: "gemini:Reference",
- attributes: { media: { kind: "reference", path: "video" } }, children: [], range: { start: 20, end: 40 },
- }], range: { start: 0, end: 50 },
- },
- resolveAsset: () => { throw new Error("not used"); },
- resolveReference: (path) => path === "video" ? {
- path, ref: { kind: "record", id: "video" }, type: artifactTypes.blob,
- record: { id: "video", type: artifactTypes.blob, value: {
- kind: "blob", resource: fixtureResource("gemini-video"), size: 10, mediaType: "video/mp4",
- } },
- } : {
- path, ref: { kind: "record", id: path }, type: textTypes.text,
- record: { id: path, type: textTypes.text, value: { kind: "inline", value: { value: path } } },
- },
- });
- assert.equal(fragmentResult.fragments.length, 1);
- const operation = fragmentResult.fragments[0]!.operations.find((item) => item.id === "generate");
- assert.deepEqual(operation?.producer, { module: geminiCapabilities["gemini-3.1-pro"].module, name: "request-gemini-3.1-pro" });
- assert.deepEqual(fragmentResult.components[0]?.outputs, { observation: "read.observation" });
-});
diff --git a/packages/provider-hypihub/README.md b/packages/provider-hypihub/README.md
index 649bc259..a8e1ca3a 100644
--- a/packages/provider-hypihub/README.md
+++ b/packages/provider-hypihub/README.md
@@ -1,7 +1,7 @@
# `@hypit/provider-hypihub`
Thin Hypit Runtime Provider for a HypiHub deployment. It is an optional default gateway for
-paid generation, Gemini VLM and WhisperX alignment requests; callers may keep their own Provider and select HypiHub only
+paid generation and WhisperX alignment requests; callers may keep their own Provider and select HypiHub only
when its OAuth login is configured.
It maps the currently shipped image/video model capabilities to HypiHub, including image edits and
@@ -9,8 +9,7 @@ image-to-video first-frame inputs, submits jobs, polls them, downloads the first
admits them into the current Build's working byte area. Image references use HypiHub's documented
`reference_images` object shape (`[{ "url": "…" }]`); video references use the public
`reference_image_urls`, `reference_videos`, and `reference_audios` fields (with `ref_video_url`
-for one video). First/last-frame images use `first_frame` and `last_frame`. It also
-exports a small Gemini-native VLM generator for callers that previously used Vertex.
+for one video). First/last-frame images use `first_frame` and `last_frame`.
The current HypiHub GPT Image 2 route exposes the same request surface as its KIE upstream:
@@ -54,18 +53,17 @@ Runtime Profile example:
}
```
-Gemini VLM is exposed as the Provider-neutral `@hypit/gemini` Runtime capability, and remote
-transcription is exposed as the same `@hypit/whisperx` alignment capability implemented by the local
-WhisperX Provider. An Author
-Source can select the exact model while the Runtime chooses HypiHub or Vertex. Existing embedded
-callers can still use the exported `createHypiHubGeminiGenerator`. Run `hypit auth login hypihub.default --runtime hypit.runtime.json` to sign in with HypiHub OAuth only when choosing HypiHub. A Profile may set `baseUrl`
+Remote transcription exposes the same `@hypit/whisperx` alignment capability implemented by the local
+WhisperX Provider. The Runtime Profile selects which Endpoint serves it. Run
+`hypit auth login hypihub.default --runtime hypit.runtime.json` to sign in with HypiHub OAuth when
+choosing HypiHub. A Profile may set `baseUrl`
to the selected deployment's origin or an existing `/v1`/`/v1beta` base. The Runtime Provider
normalizes it to `/v1`; missing or insufficient user
credentials should be resolved at [hypit.ai](https://hypit.ai). Referenced image, audio and video
Resources are uploaded through a session from `POST /v1/files/uploads`, followed by the private
regional multipart instructions returned by HypiHub. The Provider follows the server-selected part
size and concurrency, retries only a failed part with a fresh signed URL, completes or cancels that
-one upload, and then passes the returned HTTPS URL to generation, Gemini or transcription. One
+one upload, and then passes the returned HTTPS URL to generation or transcription. One
Resource identity is uploaded once within one Runtime operation. Hypit keeps no upload catalog or
cross-Build cache. Embedded callers may replace this transport with `publicAssetUrl`.
@@ -95,7 +93,7 @@ the Provider does not maintain a second list of billing formulas or calculate a
HypiHub declares MiMo Voice Design and Voice Clone together with every other capability it serves; it
never hides one. Voice Design produces an accepted voice-reference Resource, and Voice Clone uses
that reference to produce independent speech. Hypit does not expose MiMo preset voices. When another
-selected Endpoint offers the same capability (a local WhisperX, a Vertex Gemini, or the official MiMo
+selected Endpoint offers the same capability (a local WhisperX or the official MiMo
Provider), the Runtime Profile's `bindings` say which Endpoint serves it.
Execution policy remains local to this Provider:
@@ -109,20 +107,18 @@ Execution policy remains local to this Provider:
| `uploadPartTimeoutMs` | 5 minutes | one upload part |
| `uploadPartAttempts` | 3 | attempts for one upload part |
| `downloadAttempts` | 3 | attempts to collect one result |
-| `geminiRateLimitAttempts` | 4 | Gemini attempts after rate limits |
-| `geminiRateLimitRetryDelayMs` | 2 seconds | delay between those attempts |
`defaultConcurrency` controls the total shared capacity of this Profile's HypiHub pool. Optional `capabilityConcurrency`
-sets narrower group limits, for example `{ "seedance-2-mini": 2, "gemini": 3, "transcription": 1 }`.
-Image/video/speech groups use the exact capability name; all Gemini capabilities share `gemini`, and
-WhisperX uses `transcription`. These limits coordinate this Runtime's requests; HypiHub remains
-responsible for service-wide account limits. Immediate speech/Gemini/transcription slots cover the
+sets narrower group limits, for example `{ "seedance-2-mini": 2, "transcription": 1 }`.
+Image/video/speech groups use the exact capability name; WhisperX uses `transcription`. These limits
+coordinate this Runtime's requests; HypiHub remains
+responsible for service-wide account limits. Immediate speech/transcription slots cover the
active HTTP invocation, while asynchronous image/video slots cover remote work until completion or local execution failure.
For asynchronous image/video jobs, `actionLimits` configures the common `submit`, `poll` and `collect`
admission budgets. Each accepts `concurrency` and `rate: { limit, periodMs }`, shared by the pool.
These limits count lifecycle actions; Provider-specific upload parts and HTTP requests remain inside
-those actions. Synchronous speech, Gemini and transcription retain their ordinary request capacity.
+those actions. Synchronous speech and transcription retain their ordinary request capacity.
A submission, polling or collection error ends the local attempt. Known job IDs and credential
references remain available in Result receipts; a timeout with no ID is recorded as such. A job can
diff --git a/packages/provider-hypihub/package.json b/packages/provider-hypihub/package.json
index fd84facc..d11ca557 100644
--- a/packages/provider-hypihub/package.json
+++ b/packages/provider-hypihub/package.json
@@ -8,7 +8,6 @@
"hypit": { "activation": "./src/activation.ts" },
"dependencies": {
"@hypit/endpoint-kit": "workspace:*",
- "@hypit/gemini": "workspace:*",
"@hypit/generation": "workspace:*",
"@hypit/protocol": "workspace:*",
"@hypit/runtime": "workspace:*",
diff --git a/packages/provider-hypihub/src/activation.ts b/packages/provider-hypihub/src/activation.ts
index 8a0fdb4e..f0550ea0 100644
--- a/packages/provider-hypihub/src/activation.ts
+++ b/packages/provider-hypihub/src/activation.ts
@@ -29,8 +29,6 @@ const adapter = createRuntimeEndpointAdapterFacet({
"uploadPartTimeoutMs",
"uploadPartAttempts",
"downloadAttempts",
- "geminiRateLimitAttempts",
- "geminiRateLimitRetryDelayMs",
"transcriptionModel",
], "HypiHub");
const baseUrl = runtimeConfigString(config.baseUrl, "HypiHub baseUrl");
@@ -52,8 +50,6 @@ const adapter = createRuntimeEndpointAdapterFacet({
const uploadPartTimeoutMs = runtimeConfigPositiveInteger(config.uploadPartTimeoutMs, "HypiHub uploadPartTimeoutMs");
const uploadPartAttempts = runtimeConfigPositiveInteger(config.uploadPartAttempts, "HypiHub uploadPartAttempts");
const downloadAttempts = runtimeConfigPositiveInteger(config.downloadAttempts, "HypiHub downloadAttempts");
- const geminiRateLimitAttempts = runtimeConfigPositiveInteger(config.geminiRateLimitAttempts, "HypiHub geminiRateLimitAttempts");
- const geminiRateLimitRetryDelayMs = runtimeConfigPositiveInteger(config.geminiRateLimitRetryDelayMs, "HypiHub geminiRateLimitRetryDelayMs");
const transcriptionModel = runtimeConfigString(config.transcriptionModel, "HypiHub transcriptionModel");
const capabilityConcurrency = config.capabilityConcurrency === undefined ? undefined : Object.fromEntries(
Object.entries(runtimeConfigObject(config.capabilityConcurrency, "HypiHub capabilityConcurrency"))
@@ -75,8 +71,6 @@ const adapter = createRuntimeEndpointAdapterFacet({
...(uploadPartTimeoutMs === undefined ? {} : { uploadPartTimeoutMs }),
...(uploadPartAttempts === undefined ? {} : { uploadPartAttempts }),
...(downloadAttempts === undefined ? {} : { downloadAttempts }),
- ...(geminiRateLimitAttempts === undefined ? {} : { geminiRateLimitAttempts }),
- ...(geminiRateLimitRetryDelayMs === undefined ? {} : { geminiRateLimitRetryDelayMs }),
...(transcriptionModel === undefined ? {} : { transcriptionModel }),
};
return {
diff --git a/packages/provider-hypihub/src/gemini.ts b/packages/provider-hypihub/src/gemini.ts
deleted file mode 100644
index 4aa76def..00000000
--- a/packages/provider-hypihub/src/gemini.ts
+++ /dev/null
@@ -1,138 +0,0 @@
-import type { GeminiInlinePart } from "@hypit/gemini";
-import { HypiHubUploader } from "./upload.js";
-
-export type HypiHubGeminiPart = GeminiInlinePart
- | { readonly fileData: { readonly mimeType?: string; readonly fileUri: string } };
-
-export type HypiHubGeminiGeneratorOptions = {
- readonly apiKey: string;
- readonly model?: string;
- readonly baseUrl?: string;
- readonly requestTimeoutMs?: number;
- readonly uploadPartTimeoutMs?: number;
- readonly uploadPartAttempts?: number;
- readonly maxRateLimitRetries?: number;
- readonly rateLimitRetryDelayMs?: number;
- readonly fetch?: typeof globalThis.fetch;
-};
-
-export type HypiHubGeminiGenerateInput = {
- readonly parts: readonly HypiHubGeminiPart[];
- readonly instruction: string;
-};
-
-function assert(condition: unknown, message: string): asserts condition {
- if (!condition) throw new Error(message);
-}
-
-function positiveInteger(value: number, subject: string): number {
- assert(Number.isSafeInteger(value) && value > 0, `${subject} must be a positive integer`);
- return value;
-}
-
-function retryDelay(response: Response, fallback: number, attempt: number): number {
- const retryAfter = response.headers.get("retry-after")?.trim();
- if (retryAfter !== undefined && retryAfter !== "") {
- const seconds = Number(retryAfter);
- if (Number.isFinite(seconds) && seconds >= 0) return Math.max(250, Math.round(seconds * 1_000));
- const date = Date.parse(retryAfter);
- if (Number.isFinite(date)) return Math.max(250, date - Date.now());
- }
- return fallback * (attempt + 1);
-}
-
-function partValue(part: HypiHubGeminiPart,
- upload: (mimeType: string, data: string) => Promise<string>): Promise<Record<string, unknown>> {
- if ("text" in part) return Promise.resolve({ text: part.text });
- if ("fileData" in part) {
- assert(typeof part.fileData.fileUri === "string" && part.fileData.fileUri.length > 0,
- "HypiHub Gemini file media part is incomplete");
- return Promise.resolve({ fileData: { ...(part.fileData.mimeType === undefined ? {} : { mimeType: part.fileData.mimeType }), fileUri: part.fileData.fileUri } });
- }
- if ("inlineData" in part) {
- assert(typeof part.inlineData.mimeType === "string" && typeof part.inlineData.data === "string",
- "HypiHub Gemini inline media part is incomplete");
- return upload(part.inlineData.mimeType, part.inlineData.data)
- .then((fileUri) => ({ fileData: { mimeType: part.inlineData.mimeType, fileUri } }));
- }
- throw new Error("HypiHub Gemini only supports text, inline media and file media parts");
-}
-
-/** Create a small native-Gemini generator for callers that previously used Vertex. */
-export function createHypiHubGeminiGenerator(options: HypiHubGeminiGeneratorOptions) {
- const apiKey = options.apiKey.trim();
- assert(apiKey.length > 0, "HypiHub API key is empty");
- // The native Gemini endpoint lives at /v1beta while the paid image/video
- // client lives at /v1. Accept either commonly configured HypiHub base URL
- // and strip only the version suffix before adding /v1beta below.
- const baseUrl = (options.baseUrl ?? "https://hypit.ai")
- .replace(/\/(?:v1beta|v1)\/?$/iu, "")
- .replace(/\/$/u, "");
- const model = options.model?.trim() || "gemini-3.1-pro";
- const timeout = options.requestTimeoutMs ?? 120_000;
- const maxRateLimitRetries = options.maxRateLimitRetries ?? 3;
- assert(Number.isSafeInteger(maxRateLimitRetries) && maxRateLimitRetries >= 0,
- "HypiHub Gemini maxRateLimitRetries must be a non-negative integer");
- const rateLimitRetryDelayMs = positiveInteger(options.rateLimitRetryDelayMs ?? 2_000,
- "HypiHub Gemini rateLimitRetryDelayMs");
- const fetcher = options.fetch ?? globalThis.fetch;
- const uploader = new HypiHubUploader({
- baseUrl,
- requestTimeoutMs: timeout,
- uploadPartTimeoutMs: positiveInteger(options.uploadPartTimeoutMs ?? 5 * 60_000,
- "HypiHub Gemini uploadPartTimeoutMs"),
- uploadPartAttempts: positiveInteger(options.uploadPartAttempts ?? 3,
- "HypiHub Gemini uploadPartAttempts"),
- fetch: fetcher,
- });
-
- return async (input: HypiHubGeminiGenerateInput): Promise<string> => {
- const upload = async (mimeType: string, encoded: string): Promise<string> => {
- let bytes: Uint8Array;
- try { bytes = new Uint8Array(Buffer.from(encoded, "base64")); }
- catch (error) { throw new Error("HypiHub Gemini inline media is not valid base64", { cause: error }); }
- assert(bytes.byteLength > 0, "HypiHub Gemini inline media is empty");
- return await uploader.upload({ bytes, mediaType: mimeType }, apiKey);
- };
- const parts = await Promise.all(input.parts.map((part) => partValue(part, upload)));
- const controller = new AbortController();
- const timer = setTimeout(() => controller.abort(), timeout);
- try {
- let response: Response | undefined;
- let text = "";
- for (let attempt = 0; attempt <= maxRateLimitRetries; attempt += 1) {
- response = await fetcher(`${baseUrl}/v1beta/models/${encodeURIComponent(model)}:generateContent`, {
- method: "POST",
- signal: controller.signal,
- headers: { "x-goog-api-key": apiKey, "content-type": "application/json" },
- body: JSON.stringify({
- systemInstruction: { parts: [{ text: input.instruction }] },
- contents: [{ role: "user", parts }],
- generationConfig: { temperature: 1, responseMimeType: "text/plain" },
- }),
- });
- text = await response.text();
- if (response.status !== 429 || attempt === maxRateLimitRetries) break;
- await new Promise((resolve) => setTimeout(resolve, retryDelay(response!, rateLimitRetryDelayMs, attempt)));
- }
- assert(response !== undefined, "HypiHub Gemini made no request");
- if (!response.ok) {
- const message = `HypiHub Gemini returned HTTP ${response.status}: ${text.slice(0, 300)}`;
- if (response.status === 401 || response.status === 403 || response.status === 404) {
- throw new Error(`${message}. Sign in to HypiHub at https://hypit.ai with hypit auth login`);
- }
- throw new Error(message);
- }
- let body: unknown;
- try { body = JSON.parse(text); } catch { throw new Error("HypiHub Gemini returned invalid JSON"); }
- const candidates = (body as { readonly candidates?: readonly { readonly content?: { readonly parts?: readonly { readonly text?: unknown }[] } }[] }).candidates;
- const output = candidates?.flatMap((candidate) => candidate.content?.parts ?? [])
- .map((part) => typeof part.text === "string" ? part.text : "")
- .join("").trim() ?? "";
- assert(output.length > 0, "HypiHub Gemini returned an empty response");
- return output;
- } finally {
- clearTimeout(timer);
- }
- };
-}
diff --git a/packages/provider-hypihub/src/index.ts b/packages/provider-hypihub/src/index.ts
index 1e853459..7a2d86aa 100644
--- a/packages/provider-hypihub/src/index.ts
+++ b/packages/provider-hypihub/src/index.ts
@@ -1,4 +1,2 @@
export { createHypiHubProvider, hypiHubProviderModuleRef } from "./provider.js";
export type { CreateHypiHubProviderOptions } from "./provider.js";
-export { createHypiHubGeminiGenerator } from "./gemini.js";
-export type { HypiHubGeminiGenerateInput, HypiHubGeminiGeneratorOptions, HypiHubGeminiPart } from "./gemini.js";
diff --git a/packages/provider-hypihub/src/provider.ts b/packages/provider-hypihub/src/provider.ts
index c4973c32..d3a55ef0 100644
--- a/packages/provider-hypihub/src/provider.ts
+++ b/packages/provider-hypihub/src/provider.ts
@@ -1,8 +1,6 @@
import { requestDeadline } from "@hypit/runtime-kit";
import type { AsyncEndpoint, EndpointCredential, EndpointFulfillment, EndpointInvocationContext, EndpointPollContext, EndpointPricingReader, EndpointStartContext, EndpointOutcome, ImmediateEndpointHandler } from "@hypit/endpoint-kit";
import { defineEndpointPackage, wakeAfter } from "@hypit/endpoint-kit";
-import { geminiCapabilities, geminiModels, geminiTypes, sealVisualObservation, verifyGeminiRequest } from "@hypit/gemini";
-import type { GeminiRequest } from "@hypit/gemini";
import { selectWireModelForRequest } from "@hypit/generation";
import type { GenerationRequest } from "@hypit/generation";
import { canonicalize } from "@hypit/protocol";
@@ -17,7 +15,6 @@ import {
whisperXCapabilities,
} from "@hypit/whisperx";
import type { WhisperXTranscriptResponse } from "@hypit/whisperx";
-import { createHypiHubGeminiGenerator } from "./gemini.js";
import { hypiHubRouteForCapability, hypiHubRoutes } from "./routes.js";
import { HypiHubUploader } from "./upload.js";
import type { RuntimeDoctorDiagnostic } from "@hypit/runtime-kit";
@@ -42,8 +39,6 @@ export type CreateHypiHubProviderOptions = {
readonly uploadPartTimeoutMs?: number;
readonly uploadPartAttempts?: number;
readonly downloadAttempts?: number;
- readonly geminiRateLimitAttempts?: number;
- readonly geminiRateLimitRetryDelayMs?: number;
/** HypiHub model used for the Provider-neutral WhisperX alignment capability. */
readonly transcriptionModel?: string;
readonly fetch?: typeof globalThis.fetch;
@@ -81,18 +76,13 @@ function failure(error: unknown): EndpointOutcome {
return { status: "failed", failure: { code: "HYPIHUB_ERROR", message } };
}
-function inlineGeminiRequest(context: EndpointInvocationContext): GeminiRequest {
- const request = context.need.constraints as unknown;
- verifyGeminiRequest(request);
- return request;
-}
function jobId(value: Record<string, unknown>): string {
const id = value.id ?? value.job_id;
assert(typeof id === "string" && id.length > 0, "HypiHub response has no job id");
return id;
}
-type HypiHubModelOperation = "images" | "image_edits" | "videos" | "audio_speech" | "transcriptions" | "gemini";
+type HypiHubModelOperation = "images" | "image_edits" | "videos" | "audio_speech" | "transcriptions";
async function verifyModelRoute(client: HypiHubClient, auth: HypiHubAuth, model: string, operation: HypiHubModelOperation): Promise<void> {
const card = await client.json(`/models/${encodeURIComponent(model)}`, auth);
@@ -254,11 +244,6 @@ function pricingModel(request: import("@hypit/endpoint-kit").EndpointRequest, tr
if (capabilityKey(request.capability) === capabilityKey(whisperXCapabilities.alignment)) {
return transcriptionModel;
}
- if (request.capability.module.name === geminiCapabilities[geminiModels[0]].module.name
- && request.capability.module.version === geminiCapabilities[geminiModels[0]].module.version
- && geminiModels.some((model) => model === request.capability.name)) {
- return request.capability.name;
- }
const route = hypiHubRouteForCapability(request.capability);
if (route === undefined) return undefined;
const generation = request.constraints as unknown as GenerationRequest;
@@ -350,10 +335,6 @@ export async function diagnoseHypiHubProvider(
let available = false;
if (capabilityKey(capability) === capabilityKey(whisperXCapabilities.alignment)) {
available = cardEndpoints(cards.get(transcriptionModel)).includes("transcriptions");
- } else if (geminiModels.some((model) => model === capability.name)
- && capability.module.name === geminiCapabilities[geminiModels[0]].module.name
- && capability.module.version === geminiCapabilities[geminiModels[0]].module.version) {
- available = cardEndpoints(cards.get(capability.name)).includes("gemini");
} else {
const route = hypiHubRouteForCapability(capability);
if (route !== undefined) {
@@ -471,7 +452,7 @@ function endpoint(client: HypiHubClient, pollIntervalMs: number, maxOperationMs:
}
export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}) {
- const capacityNames = new Set([...hypiHubRoutes.map((route) => route.capability.name), "gemini", "transcription"]);
+ const capacityNames = new Set([...hypiHubRoutes.map((route) => route.capability.name), "transcription"]);
const capabilityConcurrency = options.capabilityConcurrency ?? {};
for (const [name, limit] of Object.entries(capabilityConcurrency)) {
assert(capacityNames.has(name), `unknown HypiHub capacity ${name}`);
@@ -484,8 +465,6 @@ export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}
const uploadPartTimeoutMs = options.uploadPartTimeoutMs ?? 5 * 60_000;
const uploadPartAttempts = options.uploadPartAttempts ?? 3;
const downloadAttempts = options.downloadAttempts ?? 3;
- const geminiRateLimitAttempts = options.geminiRateLimitAttempts ?? 4;
- const geminiRateLimitRetryDelayMs = options.geminiRateLimitRetryDelayMs ?? 2_000;
for (const [name, value] of Object.entries({
requestTimeoutMs,
oauthRequestTimeoutMs,
@@ -494,8 +473,6 @@ export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}
uploadPartTimeoutMs,
uploadPartAttempts,
downloadAttempts,
- geminiRateLimitAttempts,
- geminiRateLimitRetryDelayMs,
})) {
assert(Number.isSafeInteger(value) && value > 0, `HypiHub ${name} must be a positive integer`);
}
@@ -524,42 +501,6 @@ export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}
throw new Error(guidedMessage(error), { cause: error });
}
};
- const geminiEndpoint: ImmediateEndpointHandler = async (context) => {
- const request = inlineGeminiRequest(context);
- const auth = authFor(context, client);
- const generate = async (): Promise<string> => await createHypiHubGeminiGenerator({
- apiKey: await auth.token(),
- model: context.need.capability.name,
- ...(options.baseUrl === undefined ? {} : { baseUrl: options.baseUrl }),
- requestTimeoutMs,
- uploadPartTimeoutMs,
- uploadPartAttempts,
- ...(options.fetch === undefined ? {} : { fetch: options.fetch }),
- maxRateLimitRetries: geminiRateLimitAttempts - 1,
- rateLimitRetryDelayMs: geminiRateLimitRetryDelayMs,
- })({ parts, instruction: request.instruction });
- const parts: Array<{ text: string } | { fileData: { mimeType: string; fileUri: string } }> = [{ text: request.prompt }];
- const uploaded = new Map<string, Promise<string>>();
- for (const item of request.media) {
- let url = uploaded.get(item.artifact.resource);
- if (url === undefined) {
- url = options.publicAssetUrl === undefined
- ? client.upload(item.artifact, context.resources, auth)
- : options.publicAssetUrl(item.artifact, context.resources);
- uploaded.set(item.artifact.resource, url);
- }
- parts.push({ fileData: { mimeType: item.artifact.mediaType, fileUri: await url } });
- }
- let value: string;
- try {
- value = await generate();
- } catch (error) {
- if (!auth.canRefresh() || !/HTTP 401\b/u.test(error instanceof Error ? error.message : String(error))) throw error;
- await auth.refresh();
- value = await generate();
- }
- return { value: { kind: "inline", value: canonicalize(sealVisualObservation(value)) } };
- };
const transcriptionModel = options.transcriptionModel?.trim() || "victor-upmeet/whisperx";
const whisperXEndpoint: ImmediateEndpointHandler = async (context) => {
try {
@@ -612,14 +553,6 @@ export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}
.map((route) => route.media === "audio"
? { capability: route.capability, returns: route.returns, lifecycle: "immediate" as const, handler: audioEndpoint, capacity: route.capability.name, ...(route.supports === undefined ? {} : { supports: route.supports }), ...(capabilityConcurrency[route.capability.name] === undefined ? {} : { maxConcurrency: capabilityConcurrency[route.capability.name]! }) }
: { capability: route.capability, returns: route.returns, lifecycle: "asynchronous" as const, endpoint: asyncEndpoint, capacity: route.capability.name, ...(route.supports === undefined ? {} : { supports: route.supports }), ...(capabilityConcurrency[route.capability.name] === undefined ? {} : { maxConcurrency: capabilityConcurrency[route.capability.name]! }) }),
- ...geminiModels.map((model) => ({
- capability: geminiCapabilities[model],
- returns: geminiTypes.visualObservation,
- lifecycle: "immediate" as const,
- handler: geminiEndpoint,
- capacity: "gemini",
- ...(capabilityConcurrency.gemini === undefined ? {} : { maxConcurrency: capabilityConcurrency.gemini }),
- })),
{
capability: whisperXCapabilities.alignment,
returns: speechEvidenceTypes.alignedTranscript,
diff --git a/packages/provider-hypihub/test/gemini.test.ts b/packages/provider-hypihub/test/gemini.test.ts
deleted file mode 100644
index 4563ef75..00000000
--- a/packages/provider-hypihub/test/gemini.test.ts
+++ /dev/null
@@ -1,124 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-
-import { createHypiHubGeminiGenerator } from "../src/gemini.js";
-
-test("HypiHub Gemini uses native wire format for text, image and video parts", async () => {
- let seenUrl = "";
- let seenBody: Record<string, unknown> | undefined;
- let uploads = 0;
- const generate = createHypiHubGeminiGenerator({
- apiKey: "test-key",
- model: "gemini-3.1-pro",
- baseUrl: "https://hypit.ai/v1",
- fetch: async (input, init) => {
- seenUrl = String(input);
- if (seenUrl.endsWith("/v1/files/uploads")) {
- uploads += 1;
- return Response.json({
- upload_mode: "s3_multipart",
- upload_id: `up_native_${uploads}`,
- part_size: 16,
- part_count: 1,
- concurrency: 2,
- });
- }
- const parts = /\/v1\/files\/uploads\/(up_native_\d+)\/parts$/u.exec(seenUrl);
- if (parts) {
- const body = JSON.parse(String(init?.body)) as {
- readonly parts: readonly { readonly bytes: number; readonly checksum_sha256: string }[];
- };
- return Response.json({ parts: [{
- part_number: 1,
- url: `https://s3.example/${parts[1]}?signature=secret`,
- headers: {
- "content-length": String(body.parts[0]?.bytes),
- "x-amz-checksum-sha256": body.parts[0]?.checksum_sha256,
- },
- }] });
- }
- if (seenUrl.startsWith("https://s3.example/up_native_")) {
- return new Response(null, { headers: { etag: "part" } });
- }
- const completed = /\/v1\/files\/uploads\/(up_native_\d+)\/complete$/u.exec(seenUrl);
- if (completed) {
- return Response.json({ url: `https://hypit.ai/files/ref-${completed[1]}.png` });
- }
- seenBody = JSON.parse(String(init?.body)) as Record<string, unknown>;
- return Response.json({ candidates: [{ content: { parts: [{ text: "OK" }] } }] });
- },
- });
-
- assert.equal(await generate({ instruction: "Answer briefly.", parts: [
- { text: "inspect" },
- { inlineData: { mimeType: "image/png", data: "aW1hZ2U=" } },
- { inlineData: { mimeType: "video/mp4", data: "dmlkZW8=" } },
- ] }), "OK");
- assert.equal(seenUrl, "https://hypit.ai/v1beta/models/gemini-3.1-pro:generateContent");
- assert.equal(uploads, 2);
- assert.deepEqual(seenBody?.contents, [{ role: "user", parts: [
- { text: "inspect" },
- { fileData: { mimeType: "image/png", fileUri: "https://hypit.ai/files/ref-up_native_1.png" } },
- { fileData: { mimeType: "video/mp4", fileUri: "https://hypit.ai/files/ref-up_native_2.png" } },
- ] }]);
-});
-
-test("HypiHub Gemini points unavailable models to HypiHub", async () => {
- const generate = createHypiHubGeminiGenerator({
- apiKey: "test-key",
- fetch: async () => new Response("missing", { status: 404 }),
- });
- await assert.rejects(() => generate({ instruction: "x", parts: [{ text: "x" }] }),
- /sign in to HypiHub at https:\/\/hypit\.ai with hypit auth login/iu);
-});
-
-test("HypiHub Gemini retries upstream rate limits without reuploading files", async () => {
- let uploads = 0;
- let generations = 0;
- const generate = createHypiHubGeminiGenerator({
- apiKey: "test-key",
- maxRateLimitRetries: 2,
- rateLimitRetryDelayMs: 1,
- fetch: async (input, init) => {
- const url = String(input);
- if (url.endsWith("/v1/files/uploads")) {
- uploads += 1;
- return Response.json({
- upload_mode: "s3_multipart",
- upload_id: "up_retry",
- part_size: 16,
- part_count: 1,
- concurrency: 1,
- });
- }
- if (url.endsWith("/v1/files/uploads/up_retry/parts")) {
- const body = JSON.parse(String(init?.body)) as {
- readonly parts: readonly { readonly checksum_sha256: string }[];
- };
- return Response.json({ parts: [{
- part_number: 1,
- url: "https://s3.example/retry?signature=secret",
- headers: {
- "content-length": "5",
- "x-amz-checksum-sha256": body.parts[0]?.checksum_sha256,
- },
- }] });
- }
- if (url.startsWith("https://s3.example/retry")) {
- return new Response(null, { headers: { etag: "part" } });
- }
- if (url.endsWith("/v1/files/uploads/up_retry/complete")) {
- return Response.json({ url: "https://hypit.ai/files/ref" });
- }
- generations += 1;
- return generations < 3
- ? new Response("rate limited", { status: 429, headers: { "retry-after": "0" } })
- : Response.json({ candidates: [{ content: { parts: [{ text: "OK" }] } }] });
- },
- });
- assert.equal(await generate({ instruction: "x", parts: [
- { text: "x" }, { inlineData: { mimeType: "image/png", data: "aW1hZ2U=" } },
- ] }), "OK");
- assert.equal(uploads, 1);
- assert.equal(generations, 3);
-});
diff --git a/packages/provider-hypihub/test/provider.test.ts b/packages/provider-hypihub/test/provider.test.ts
index d49b9867..126b6fae 100644
--- a/packages/provider-hypihub/test/provider.test.ts
+++ b/packages/provider-hypihub/test/provider.test.ts
@@ -4,7 +4,6 @@ import test from "node:test";
import { EndpointRegistry, MemoryResourceStore } from "@hypit/driver-node";
import { defineEndpointPackage } from "@hypit/endpoint-kit";
import type { AsyncEndpoint } from "@hypit/endpoint-kit";
-import { geminiCapabilities, geminiTypes, sealGeminiRequest } from "@hypit/gemini";
import type { CanonicalValue, Need } from "@hypit/protocol";
import { mimoSpeechEndpoints, sealMimoSpeechRequest } from "@hypit/mimo-speech";
import { generationTypes } from "@hypit/generation";
@@ -505,80 +504,6 @@ test("HypiHub stops before paid submission when a reference upload fails", async
assert.doesNotMatch(outcome.status === "failed" ? outcome.failure.message : "", /must-not-leak/u);
});
-test("HypiHub fulfills Gemini through the Runtime endpoint and uploads every media Resource", async () => {
- const resources = new MemoryResourceStore();
- const image = await resources.put(new Uint8Array([1, 2, 3]), "image/png");
- const video = await resources.put(new Uint8Array([4, 5, 6]), "video/mp4");
- const request: Need = {
- id: "need:hypihub-gemini",
- capability: geminiCapabilities["gemini-3.1-pro"],
- returns: geminiTypes.visualObservation,
- constraints: sealGeminiRequest({
- instruction: "Answer briefly.", prompt: "Inspect both references.",
- media: [{ artifact: image }, { artifact: video }],
- }) as unknown as CanonicalValue,
- result: "record:hypihub-gemini",
- };
- let uploadSessions = 0;
- let completedUploads = 0;
- let generationBody: Record<string, unknown> | undefined;
- const registry = new EndpointRegistry();
- await createHypiHubProvider({ fetch: async (input, init) => {
- const url = String(input);
- if (url.endsWith("/v1/files/uploads")) {
- uploadSessions += 1;
- return Response.json({
- upload_mode: "s3_multipart",
- upload_id: `up_gemini_${uploadSessions}`,
- part_size: 16,
- part_count: 1,
- concurrency: 2,
- });
- }
- const parts = /\/v1\/files\/uploads\/(up_gemini_\d+)\/parts$/u.exec(url);
- if (parts) {
- const body = JSON.parse(String(init?.body)) as {
- readonly parts: readonly { readonly checksum_sha256: string }[];
- };
- return Response.json({ parts: [{
- part_number: 1,
- url: `https://s3.example/${parts[1]}?signature=secret`,
- headers: {
- "content-length": "3",
- "x-amz-checksum-sha256": body.parts[0]?.checksum_sha256,
- },
- }] });
- }
- if (url.startsWith("https://s3.example/up_gemini_")) {
- return new Response(null, { headers: { etag: "part" } });
- }
- if (/\/v1\/files\/uploads\/up_gemini_\d+\/complete$/u.test(url)) {
- completedUploads += 1;
- return Response.json({ url: `https://hypit.ai/files/${completedUploads}` });
- }
- if (url.includes(":generateContent")) {
- generationBody = JSON.parse(String(init?.body)) as Record<string, unknown>;
- return Response.json({ candidates: [{ content: { parts: [{ text: "provider works" }] } }] });
- }
- throw new Error(`Unexpected URL ${url}`);
- } }).install(registry);
- const resolution = registry.resolve(request);
- assert.equal(resolution.status, "resolved");
- assert.equal(resolution.registration.kind, "immediate");
- const result = await resolution.registration.handler({
- command: { kind: "fulfill-need", id: "command:hypihub-gemini", need: request },
- need: request, resources, credentials: { apiKey: { secret: "test-key" } },
- });
- assert.deepEqual(result.value, { kind: "inline", value: { text: "provider works" } });
- assert.equal(uploadSessions, 2);
- assert.equal(completedUploads, 2);
- assert.deepEqual((generationBody?.contents as readonly unknown[]), [{ role: "user", parts: [
- { text: "Inspect both references." },
- { fileData: { mimeType: "image/png", fileUri: "https://hypit.ai/files/1" } },
- { fileData: { mimeType: "video/mp4", fileUri: "https://hypit.ai/files/2" } },
- ] }]);
-});
-
test("HypiHub fulfills the Provider-neutral WhisperX alignment capability", async () => {
const resources = new MemoryResourceStore();
const bytes = wav(32_000);
@@ -671,10 +596,9 @@ test("HypiHub fulfills the Provider-neutral WhisperX alignment capability", asyn
test("HypiHub exposes model groups beneath its own total capacity", async () => {
const registry = new EndpointRegistry();
await createHypiHubProvider({ pool: "hub-account", defaultConcurrency: 8,
- capabilityConcurrency: { "seedance-2-mini": 2, gemini: 3, transcription: 1 } }).install(registry);
+ capabilityConcurrency: { "seedance-2-mini": 2, transcription: 1 } }).install(registry);
const requests = [
need({}),
- { ...need({}), capability: Object.values(geminiCapabilities)[0]!, returns: geminiTypes.visualObservation },
{ ...need({}), capability: whisperXCapabilities.alignment, returns: speechEvidenceTypes.alignedTranscript },
];
for (const [index, request] of requests.entries()) {
@@ -682,7 +606,7 @@ test("HypiHub exposes model groups beneath its own total capacity", async () =>
assert.equal(selected.status, "resolved");
assert.deepEqual(selected.registration.scheduling?.resources, [
{ id: "pool:hub-account", limit: 8 },
- { id: `capacity:hub-account/${["seedance-2-mini", "gemini", "transcription"][index]}`, limit: [2, 3, 1][index] },
+ { id: `capacity:hub-account/${["seedance-2-mini", "transcription"][index]}`, limit: [2, 1][index] },
]);
}
assert.throws(() => createHypiHubProvider({ capabilityConcurrency: { invented: 1 } }), /unknown HypiHub capacity/);
diff --git a/packages/provider-vertex/README.md b/packages/provider-vertex/README.md
deleted file mode 100644
index 58c5a22f..00000000
--- a/packages/provider-vertex/README.md
+++ /dev/null
@@ -1,58 +0,0 @@
-# `@hypit/provider-vertex`
-
-Vertex Gemini Runtime Provider. It fulfils the same Provider-neutral `@hypit/gemini` capabilities
-as HypiHub, so a Runtime Profile chooses the backend without changing Author Source. The package
-owns the Google GenAI client, credential parsing and the Vertex model ids.
-
-## Runtime Profile
-
-```json
-{
- "credentials": {
- "env": { "use": "@hypit/credential-store-env" }
- },
- "endpoints": {
- "vertex.default": {
- "use": "@hypit/provider-vertex",
- "config": {
- "project": { "store": "env", "key": "GOOGLE_CLOUD_PROJECT" },
- "credentials": { "store": "env", "key": "GOOGLE_APPLICATION_CREDENTIALS_JSON" },
- "location": "global",
- "requestTimeoutMs": 300000
- }
- }
- }
-}
-```
-
-`project` and `credentials` are both required CredentialRefs. `credentials` is the JSON of a Google
-credential: a service account key, or the `application_default_credentials.json` that
-`gcloud auth application-default login` writes (an `authorized_user` credential). `location`
-defaults to `global`, where the current Gemini models are served. `requestTimeoutMs` bounds each
-request; the default is five minutes.
-
-When another selected Endpoint also offers Gemini (HypiHub does), the Profile's `bindings` say which
-one serves `@hypit/gemini@1#gemini-3.1-pro`; neither Provider hides the capability.
-
-## Model ids
-
-The capability names the model the author chose; this package maps it to the id Vertex publishes
-it under (`gemini-3.1-pro` becomes `gemini-3.1-pro-preview`). A capability without a Vertex id is
-refused at activation rather than sent under a name Vertex may not recognise.
-
-## Doctor
-
-`hypit doctor` asks Vertex for each declared model with the configured project and credentials, so a
-project without the Vertex AI API enabled, without access to the model, or with a wrong location is
-reported before the first paid call, with Vertex's own explanation.
-
-## Embedded use
-
-```ts
-import { createVertexGeminiGenerator } from "@hypit/provider-vertex";
-
-const generate = createVertexGeminiGenerator({
- project: process.env.GOOGLE_CLOUD_PROJECT!,
- credentials: process.env.GOOGLE_APPLICATION_CREDENTIALS_JSON!,
-});
-```
diff --git a/packages/provider-vertex/package.json b/packages/provider-vertex/package.json
deleted file mode 100644
index 82c2fcb4..00000000
--- a/packages/provider-vertex/package.json
+++ /dev/null
@@ -1,21 +0,0 @@
-{
- "name": "@hypit/provider-vertex",
- "version": "0.0.0-dev",
- "license": "SEE LICENSE IN LICENSE",
- "private": true,
- "type": "module",
- "exports": { ".": "./src/index.ts" },
- "hypit": { "activation": "./src/activation.ts" },
- "dependencies": {
- "@google/genai": "1.52.0",
- "@hypit/endpoint-kit": "workspace:*",
- "@hypit/gemini": "workspace:*",
- "@hypit/protocol": "workspace:*",
- "@hypit/runtime": "workspace:*",
- "@hypit/runtime-kit": "workspace:*",
- "@hypit/text": "workspace:*"
- },
- "devDependencies": {
- "@hypit/driver-node": "workspace:*"
- }
-}
diff --git a/packages/provider-vertex/src/activation.ts b/packages/provider-vertex/src/activation.ts
deleted file mode 100644
index 8415478d..00000000
--- a/packages/provider-vertex/src/activation.ts
+++ /dev/null
@@ -1,42 +0,0 @@
-import {
- createRuntimeEndpointAdapterFacet,
- runtimeConfigCredentialRef,
- runtimeConfigExact,
- runtimeConfigObject,
- runtimeConfigPositiveInteger,
- runtimeConfigString,
-} from "@hypit/runtime-kit";
-
-import { createVertexProvider, diagnoseVertexProvider } from "./provider.js";
-
-const adapter = createRuntimeEndpointAdapterFacet({
- use: "@hypit/provider-vertex",
- activate(context) {
- if (context.pool === undefined) throw new Error("Vertex Provider Pool is required");
- const config = runtimeConfigObject(context.config, "Vertex");
- runtimeConfigExact(config, ["project", "credentials", "location", "defaultConcurrency", "requestTimeoutMs"], "Vertex");
- const project = runtimeConfigCredentialRef(config.project, "Vertex project");
- if (project === undefined) throw new Error("Vertex project CredentialRef is required");
- const credentials = runtimeConfigCredentialRef(config.credentials, "Vertex credentials");
- if (credentials === undefined) throw new Error("Vertex credentials CredentialRef is required");
- const location = runtimeConfigString(config.location, "Vertex location");
- const defaultConcurrency = runtimeConfigPositiveInteger(config.defaultConcurrency, "Vertex defaultConcurrency");
- const requestTimeoutMs = runtimeConfigPositiveInteger(config.requestTimeoutMs, "Vertex requestTimeoutMs");
- const options = {
- instance: context.instance,
- pool: context.pool,
- project,
- credentials,
- ...(location === undefined ? {} : { location }),
- ...(defaultConcurrency === undefined ? {} : { defaultConcurrency }),
- ...(requestTimeoutMs === undefined ? {} : { requestTimeoutMs }),
- };
- return {
- endpoint: createVertexProvider(options),
- diagnose: async (doctor) => await diagnoseVertexProvider(options, doctor),
- };
- },
-});
-
-export const hypitPackage = { format: "hypit.node-package@1" as const, hostFacets: [adapter] };
-export default hypitPackage;
diff --git a/packages/provider-vertex/src/gemini.ts b/packages/provider-vertex/src/gemini.ts
deleted file mode 100644
index 095095b9..00000000
--- a/packages/provider-vertex/src/gemini.ts
+++ /dev/null
@@ -1,85 +0,0 @@
-import { GoogleGenAI } from "@google/genai";
-import type { Content } from "@google/genai";
-import type { GeminiGenerateInput, GeminiInlinePart } from "@hypit/gemini";
-
-export type VertexGeminiPart = GeminiInlinePart;
-export type VertexGeminiGenerateInput = GeminiGenerateInput;
-
-export type VertexGeminiGeneratorOptions = {
- readonly project: string;
- readonly credentials: Record<string, unknown> | string;
- readonly location?: string;
- readonly model?: string;
- readonly requestTimeoutMs?: number;
-};
-
-function assert(condition: unknown, message: string): asserts condition {
- if (!condition) throw new Error(message);
-}
-
-function parseCredentials(value: Record<string, unknown> | string): Record<string, unknown> {
- if (typeof value !== "string") return value;
- let parsed: unknown;
- try { parsed = JSON.parse(value); } catch { throw new Error("GOOGLE_APPLICATION_CREDENTIALS_JSON is not valid JSON"); }
- assert(parsed !== null && typeof parsed === "object" && !Array.isArray(parsed), "Google credentials must be an object");
- return parsed as Record<string, unknown>;
-}
-
-/**
- * The Vertex publisher id behind each Hypit Gemini capability. The capability names the model the
- * author chose; the id Vertex serves it under is this Provider's fact. A capability without an
- * entry is refused rather than sent under a name Vertex may not recognise.
- */
-const VERTEX_MODEL_IDS: Readonly<Record<string, string>> = {
- "gemini-3.1-pro": "gemini-3.1-pro-preview",
-};
-
-export function vertexModelId(capabilityName: string): string {
- const id = VERTEX_MODEL_IDS[capabilityName];
- if (id === undefined) throw new Error(`Vertex Provider has no published model id for ${capabilityName}`);
- return id;
-}
-
-export type VertexClientOptions = {
- readonly project: string;
- readonly credentials: Record<string, unknown> | string;
- readonly location?: string;
- readonly requestTimeoutMs?: number;
-};
-
-export function createVertexClient(options: VertexClientOptions): GoogleGenAI {
- const project = options.project.trim();
- assert(project.length > 0, "Vertex project is empty");
- return new GoogleGenAI({
- vertexai: true,
- project,
- location: options.location?.trim() || "global",
- googleAuthOptions: {
- credentials: parseCredentials(options.credentials),
- scopes: ["https://www.googleapis.com/auth/cloud-platform"],
- },
- httpOptions: { timeout: options.requestTimeoutMs ?? 300_000 },
- });
-}
-
-/** Ask Vertex whether this project can reach the model; the service's own message says why not. */
-export async function probeVertexModel(options: VertexClientOptions, modelId: string): Promise<void> {
- await createVertexClient(options).models.get({ model: modelId });
-}
-
-/** Vertex Gemini VLM generator for text, image and video inline parts. */
-export function createVertexGeminiGenerator(options: VertexGeminiGeneratorOptions) {
- const model = options.model?.trim() || vertexModelId("gemini-3.1-pro");
- const client = createVertexClient(options);
-
- return async (input: VertexGeminiGenerateInput): Promise<string> => {
- const result = await client.models.generateContent({
- model,
- contents: [{ role: "user", parts: input.parts as unknown as Content["parts"] }] as never,
- config: { systemInstruction: input.instruction, temperature: 1.0, responseMimeType: "text/plain" },
- });
- const text = result.text?.trim() ?? "";
- assert(text.length > 0, "Vertex Gemini returned an empty response");
- return text;
- };
-}
diff --git a/packages/provider-vertex/src/index.ts b/packages/provider-vertex/src/index.ts
deleted file mode 100644
index f8ae7798..00000000
--- a/packages/provider-vertex/src/index.ts
+++ /dev/null
@@ -1,8 +0,0 @@
-export { createVertexGeminiGenerator } from "./gemini.js";
-export type {
- VertexGeminiGenerateInput,
- VertexGeminiGeneratorOptions,
- VertexGeminiPart,
-} from "./gemini.js";
-export { createVertexProvider, vertexProviderModuleRef } from "./provider.js";
-export type { CreateVertexProviderOptions } from "./provider.js";
diff --git a/packages/provider-vertex/src/provider.ts b/packages/provider-vertex/src/provider.ts
deleted file mode 100644
index c577e626..00000000
--- a/packages/provider-vertex/src/provider.ts
+++ /dev/null
@@ -1,116 +0,0 @@
-import { defineEndpointPackage } from "@hypit/endpoint-kit";
-import type { EndpointInvocationContext, ImmediateEndpointHandler } from "@hypit/endpoint-kit";
-import { geminiCapabilities, geminiModels, geminiTypes, sealVisualObservation, verifyGeminiRequest } from "@hypit/gemini";
-import type { CapabilityRef } from "@hypit/protocol";
-import type { RuntimeDoctorDiagnostic } from "@hypit/runtime-kit";
-import type { GeminiRequest } from "@hypit/gemini";
-import { canonicalize } from "@hypit/protocol";
-import type { CredentialRef } from "@hypit/runtime";
-
-import { createVertexGeminiGenerator, probeVertexModel, vertexModelId } from "./gemini.js";
-
-export const vertexProviderModuleRef = { name: "@hypit/provider-vertex", version: "1" } as const;
-
-export type CreateVertexProviderOptions = {
- readonly instance?: string;
- readonly pool?: string;
- readonly project: CredentialRef;
- readonly credentials: CredentialRef;
- readonly location?: string;
- readonly defaultConcurrency?: number;
- readonly requestTimeoutMs?: number;
-};
-
-function secret(context: EndpointInvocationContext, name: "project" | "credentials"): string {
- const value = context.credentials[name]?.secret;
- if (typeof value !== "string" || value.trim().length === 0) throw new Error(`Vertex ${name} credential is unavailable`);
- return value;
-}
-
-function capabilityKey(capability: CapabilityRef): string {
- return `${capability.module.name}@${capability.module.version}#${capability.name}`;
-}
-
-/** Active doctor: can this project reach each declared model on Vertex? Vertex says why when not. */
-export async function diagnoseVertexProvider(
- options: Pick<CreateVertexProviderOptions, "location" | "requestTimeoutMs">,
- context: {
- readonly credentials: Readonly<Record<string, { readonly secret: string }>>;
- readonly capabilities?: readonly CapabilityRef[];
- },
-): Promise<readonly RuntimeDoctorDiagnostic[]> {
- const project = context.credentials.project?.secret ?? "";
- const credentials = context.credentials.credentials?.secret ?? "";
- if (project.trim().length === 0 || credentials.trim().length === 0) {
- return [{ severity: "error", code: "VERTEX_CREDENTIALS_MISSING", message: "Vertex project and credentials are not both configured" }];
- }
- const requested = context.capabilities ?? geminiModels.map((model) => geminiCapabilities[model]);
- const diagnostics: RuntimeDoctorDiagnostic[] = [];
- for (const capability of requested) {
- if (!geminiModels.some((model) => capabilityKey(geminiCapabilities[model]) === capabilityKey(capability))) continue;
- try {
- await probeVertexModel({
- project,
- credentials,
- ...(options.location === undefined ? {} : { location: options.location }),
- ...(options.requestTimeoutMs === undefined ? {} : { requestTimeoutMs: options.requestTimeoutMs }),
- }, vertexModelId(capability.name));
- } catch (error) {
- diagnostics.push({
- severity: "error",
- code: "VERTEX_MODEL_UNAVAILABLE",
- message: `${project} cannot reach ${vertexModelId(capability.name)} on Vertex: ${error instanceof Error ? error.message.slice(0, 400) : String(error)}`,
- subject: capabilityKey(capability),
- });
- }
- }
- return diagnostics;
-}
-
-export function createVertexProvider(options: CreateVertexProviderOptions) {
- // Every declared capability must have a Vertex id before anything is registered.
- for (const model of geminiModels) vertexModelId(model);
- const handler: ImmediateEndpointHandler = async (context) => {
- const request = context.need.constraints as unknown;
- verifyGeminiRequest(request);
- const typed = request as GeminiRequest;
- const generate = createVertexGeminiGenerator({
- project: secret(context, "project"),
- credentials: secret(context, "credentials"),
- model: vertexModelId(context.need.capability.name),
- ...(options.location === undefined ? {} : { location: options.location }),
- ...(options.requestTimeoutMs === undefined ? {} : { requestTimeoutMs: options.requestTimeoutMs }),
- });
- const parts: Array<{ text: string } | { inlineData: { mimeType: string; data: string } }> = [{ text: typed.prompt }];
- for (const item of typed.media) {
- const bytes = await context.resources.get(item.artifact.resource);
- if (bytes === undefined) throw new Error(`Vertex Gemini reference artifact ${item.artifact.resource} is unavailable`);
- parts.push({ inlineData: { mimeType: item.artifact.mediaType, data: Buffer.from(bytes).toString("base64") } });
- }
- const value = await generate({ parts, instruction: typed.instruction });
- return { value: { kind: "inline", value: canonicalize(sealVisualObservation(value)) } };
- };
- return defineEndpointPackage({
- module: vertexProviderModuleRef,
- facet: "gemini",
- instance: options.instance ?? "vertex.default",
- pool: options.pool ?? options.instance ?? "vertex.default",
- pricing: { kind: "page", url: "https://cloud.google.com/vertex-ai/generative-ai/pricing" },
- credentials: {
- project: options.project,
- credentials: options.credentials,
- },
- credentialInputs: {
- project: { label: "Google Cloud project" },
- credentials: { label: "Google application credentials", kind: "json" },
- },
- defaultConcurrency: options.defaultConcurrency ?? 4,
- capabilities: geminiModels.map((model) => ({
- capability: geminiCapabilities[model],
- returns: geminiTypes.visualObservation,
- lifecycle: "immediate" as const,
- handler,
- capacity: model,
- })),
- });
-}
diff --git a/packages/provider-vertex/test/provider.test.ts b/packages/provider-vertex/test/provider.test.ts
deleted file mode 100644
index 8d7cdfaa..00000000
--- a/packages/provider-vertex/test/provider.test.ts
+++ /dev/null
@@ -1,36 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-
-import { EndpointRegistry } from "@hypit/driver-node";
-import { geminiCapabilities, geminiTypes } from "@hypit/gemini";
-import { credentialRef } from "@hypit/runtime";
-
-import { createVertexProvider } from "../src/provider.js";
-import { vertexModelId } from "../src/gemini.js";
-
-test("Vertex Runtime provider exposes the same provider-neutral Gemini capabilities", async () => {
- const registry = new EndpointRegistry();
- const provider = createVertexProvider({
- project: credentialRef("env", "GOOGLE_CLOUD_PROJECT"),
- credentials: credentialRef("env", "GOOGLE_APPLICATION_CREDENTIALS_JSON"),
- });
- await provider.install(registry);
- for (const model of ["gemini-3.1-pro"] as const) {
- const resolution = registry.resolve({
- capability: geminiCapabilities[model],
- returns: geminiTypes.visualObservation,
- constraints: { instruction: "x", prompt: "x", media: [] },
- });
- assert.equal(resolution.status, "resolved");
- assert.equal(resolution.registration.kind, "immediate");
- }
- assert.deepEqual(provider.credentials.map((item) => ({ slot: item.slot, kind: item.kind })), [
- { slot: "credentials", kind: "json" },
- { slot: "project", kind: "secret" },
- ]);
-});
-
-test("Vertex model ids fail closed", () => {
- assert.equal(vertexModelId("gemini-3.1-pro"), "gemini-3.1-pro-preview");
- assert.throws(() => vertexModelId("gemini-99"), /no published model id for gemini-99/u);
-});
diff --git a/packages/video-cli/README.md b/packages/video-cli/README.md
index e5faf8bb..fbd3de51 100644
--- a/packages/video-cli/README.md
+++ b/packages/video-cli/README.md
@@ -32,17 +32,16 @@ hypit cancel <build-id>
hypit doctor
```
-Creation-time tools run one immediate request through the selected Runtime Profile, with no Build,
-Result or state. Each names the Endpoint and its price page before it runs, and writes exactly one
-file the caller chose:
+`transcribe` runs one immediate request through the selected Runtime Profile, with no Build,
+Result or state. It names the Endpoint and its price page before it runs, and writes the transcript
+to the chosen file. `measure` estimates a passage locally:
```bash
-hypit observe <media…> --instruction <text|file> --prompt <text|file> --to notes/frame.md
hypit transcribe reference.mp4 --to notes/reference.transcript.json [--language en|zh|es]
hypit measure main.svml --segment hook --language en --pace normal --rounding round
```
-`observe` uses the Profile's Gemini Endpoint and `transcribe` its `whisperx-alignment` Endpoint (after
+`transcribe` uses the Profile's `whisperx-alignment` Endpoint (after
extracting 16 kHz mono speech audio with ffmpeg). `measure` counts a Segment's pronunciation units at a
delivery policy and prints the seconds to write as the literal `duration`; it opens no Profile and
spends nothing. `--runtime <profile>` names the Profile; otherwise the project's `hypit runtime use`
@@ -52,8 +51,8 @@ that produced them, so they are declared in the Source and go through `plan` and
voice or learn a passage's real length before authoring the rest, build a Run whose target is that
speech output and reuse it as a Candidate.
-Two more families are local, stateless and spend nothing. `hypit media` prepares what the eyes will
-look at when a whole video is too long or too dense to hand over at once, and `hypit vocabulary`
+Two more families are local, stateless and spend nothing. `hypit media` exposes the source at chosen
+times and scales, and `hypit vocabulary`
prints what a Source may write:
```bash
@@ -62,6 +61,8 @@ hypit media cut reference.mp4 --start 12 --end 19.5 --label-time --to notes/hook
hypit media frames reference.mp4 --at 12.4,13.1 --label-time --to notes/hook-frames
hypit media tile reference.mp4 --start 12 --end 19.5 --to notes/hook-grid.jpg
hypit media tile reference.mp4 --at 12.4,13.1,14.8 --columns 3 --to notes/exact-grid.jpg
+hypit media tile reference.mp4 --start 12 --end 14 --every 0.1 --transcript notes/reference.transcript.json --to notes/detail.jpg
+hypit media tiles reference.mp4 --around "your next idea" --transcript notes/reference.transcript.json --every 0.1 --columns 3 --rows 2 --to notes/phrase
hypit media tiles reference.mp4 --ranges notes/ranges.json --to notes/grids
hypit media boundaries reference.mp4
hypit media fetch https://… --to reference/source.mp4
@@ -70,11 +71,33 @@ hypit vocabulary @hypit/media-pipeline --tag StillVideo
hypit vocabulary --visual text
```
-`cut` seeks to the exact frame and can visibly overlay absolute source time on the evidence copy;
-`frames` writes one JPEG per named second and can add the same visible label. `tile` always draws the
-absolute source time below every cell. It accepts either one range sampled evenly or exact `--at`
-times; `tiles` repeats that operation for an ordinary JSON array of `{ start, end, id?, frames? }`
-ranges. Columns, cell width and sample count remain caller choices. `boundaries` reports adjacent-frame
+`cut` isolates the requested interval and can visibly overlay source time on the evidence copy.
+`frames` writes one JPEG per requested time, selecting the first decoded frame at or after it.
+Visible frame labels use that frame's actual timestamp, as do the labels below each `tile` cell.
+Sampling is shared by `frames`, `tile` and `tiles`:
+
+- `--at` names exact sample times; `--start` and `--end` choose a range in seconds.
+- `--every` samples from the start at that interval, excluding the end. Times use millisecond precision.
+- Grids can instead use `--frames` evenly spaced bin midpoints. Without a sampling option they provide
+ a compact overview; choose the interval explicitly when inspecting fast motion.
+- `--transcript` reads `hypit.transcript@1` from `hypit transcribe`. The transcript and input media must
+ share the same clock. Labels show source time, active words with their start/end, and nearby words.
+ Active spans use `[start, end)`; overlapping words are all shown. Missing times stay missing, and a
+ frame without a timed word is identified without inferring silence. Text is rendered below the
+ source picture using Sharp/Pango and the machine's fonts, including font fallback for multilingual text.
+- `--around "a phrase"` with `--transcript` selects that phrase's word boundaries plus `--padding`
+ seconds on either side (0.3 by default), clipped to the input duration. Matching uses whole consecutive
+ words, ignoring case, whitespace and punctuation. Repeated matches list their times and require an
+ explicit `--occurrence` (one-based), or a numeric range. This locates evidence; it does not interpret it.
+- `tiles` accepts the same selection as `tile`, or `--ranges` with a JSON array of
+ `{ start, end, id?, frames?, every? }`. A range's sampling choice overrides the command default.
+ It paginates into `--columns` × `--rows` cells (3 × 3 by default); the final page may have fewer cells.
+ `--cell` chooses picture width. `tile` keeps all requested samples in one image.
+
+`--json` reports each frame's `requestedAt` and actual `at`, plus its active/context words when supplied.
+Grid `samples` retain the requested times; `frames` contain the actual extracted-frame information.
+It also reports every page path for `tiles`. The media layer reads existing timed text; transcription
+and its Endpoint remain separate. `boundaries` reports adjacent-frame
change candidates and their measured scores; it does not suppress short changes or call them shots.
`fetch` turns a link into a file with the pinned yt-dlp. Commands that create evidence write only
what `--to` names and refuse to overwrite. `vocabulary` reads the installed
diff --git a/packages/video-cli/package.json b/packages/video-cli/package.json
index 0b943ad7..cde55822 100644
--- a/packages/video-cli/package.json
+++ b/packages/video-cli/package.json
@@ -15,7 +15,6 @@
"@hypit/credential-store-os": "workspace:*",
"@hypit/driver-node": "workspace:*",
"@hypit/estimate": "workspace:*",
- "@hypit/gemini": "workspace:*",
"@hypit/generation": "workspace:*",
"@hypit/markup": "workspace:*",
"@hypit/model-kit": "workspace:*",
@@ -31,6 +30,7 @@
"@hypit/visual-ir": "workspace:*",
"@hypit/whisperx": "workspace:*",
"@hypit/workspace-fs-node": "workspace:*",
- "@hypit/yt-dlp": "workspace:*"
+ "@hypit/yt-dlp": "workspace:*",
+ "sharp": "0.35.3"
}
}
diff --git a/packages/video-cli/src/creation.ts b/packages/video-cli/src/creation.ts
index 2f7f91d8..c65eee5d 100644
--- a/packages/video-cli/src/creation.ts
+++ b/packages/video-cli/src/creation.ts
@@ -1,6 +1,6 @@
import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
-import { dirname, extname, join, resolve } from "node:path";
+import { dirname, join, resolve } from "node:path";
import { findRuntimeProfile, loadDiscoveredSourcePackages, resolveProjectRoot } from "@hypit/cli";
import type { CliIo } from "@hypit/cli";
@@ -11,8 +11,6 @@ import {
resolveSpeechEstimateLanguage,
speechEstimatePolicyFromAttributes,
} from "@hypit/estimate";
-import { geminiCapabilities, geminiModels, geminiTypes, sealGeminiRequest, verifyVisualObservation } from "@hypit/gemini";
-import type { GeminiMediaPart, GeminiModel, VisualObservation } from "@hypit/gemini";
import type { CanonicalValue, CapabilityRef, Need, StoredValue } from "@hypit/protocol";
import type { ResourceStore } from "@hypit/runtime";
import type { RuntimeHostCapabilityProvider, RuntimeHostProviderQuery } from "@hypit/runtime-host-node";
@@ -26,18 +24,8 @@ import type { WhisperXLanguage } from "@hypit/whisperx";
import { videoCliDistribution } from "./distribution.js";
import { runProcess } from "./process.js";
-/**
- * Creation-time tools: see a picture, hear a recording, measure a script.
- *
- * Each is one immediate Need executed through the selected Runtime Profile, exactly as a Build would
- * resolve it — same Endpoint, same credential, same Provider — without a Build, Result or state. The
- * command names the Endpoint and its price page before it spends anything, writes one file the caller
- * chose, and nothing else. Everything the Author Graph declares as an output — pictures, clips, the
- * spoken A-roll — is a Build, however fast it comes back: an output carries the identity of the Source
- * that produced it, and a command's file does not.
- */
-
-export const creationCommands = ["observe", "transcribe", "measure"] as const;
+/** Transcribe source media through a selected Endpoint, or estimate authored speech locally. */
+export const creationCommands = ["transcribe", "measure"] as const;
export type CreationCommand = typeof creationCommands[number];
/** The slice of the Runtime host these commands use; tests hand in a fake. */
@@ -196,29 +184,6 @@ function providerLine(provider: RuntimeHostCapabilityProvider): string {
// ---------------------------------------------------------------------------------------------------
// Media
-const MEDIA_TYPES: Readonly<Record<string, string>> = {
- ".png": "image/png",
- ".jpg": "image/jpeg",
- ".jpeg": "image/jpeg",
- ".webp": "image/webp",
- ".gif": "image/gif",
- ".mp4": "video/mp4",
- ".m4v": "video/mp4",
- ".mov": "video/quicktime",
- ".webm": "video/webm",
- ".wav": "audio/wav",
- ".mp3": "audio/mpeg",
- ".m4a": "audio/mp4",
- ".flac": "audio/flac",
- ".ogg": "audio/ogg",
-};
-
-function mediaType(path: string): string {
- const type = MEDIA_TYPES[extname(path).toLowerCase()];
- assert(type !== undefined, `${path} has no media type this command knows; supported: ${Object.keys(MEDIA_TYPES).join(", ")}`);
- return type;
-}
-
async function run(executable: string, args: readonly string[]): Promise<string> {
return (await runProcess(executable, args)).toString("utf8");
}
@@ -277,56 +242,6 @@ function seconds(sample: number | undefined): number | undefined {
return sample === undefined ? undefined : round(sample / EVIDENCE_SAMPLE_RATE);
}
-// ---------------------------------------------------------------------------------------------------
-// observe
-
-async function observe(argv: readonly string[], io: CliIo, environment: CreationEnvironment): Promise<void> {
- const parsed = parseArguments(argv, ["--instruction", "--prompt", "--model", "--to", "--runtime", "--workspace"]);
- assert(parsed.positionals.length > 0, "observe requires at least one media file to look at");
- const modelName = parsed.options.get("--model") ?? geminiModels[0];
- assert((geminiModels as readonly string[]).includes(modelName),
- `Gemini model ${modelName} is not declared by @hypit/gemini; declared models: ${geminiModels.join(", ")}`);
- const model = modelName as GeminiModel;
- const instruction = await textOrFile(required(parsed, "--instruction", "who the observer is and what it returns"), "--instruction", environment.cwd);
- const prompt = await textOrFile(required(parsed, "--prompt", "the question to answer about the media"), "--prompt", environment.cwd);
- const to = await destination(parsed, environment.cwd);
- const { profile, host } = await environment.openHost(
- parsed.options.get("--runtime"),
- parsed.options.get("--workspace"),
- );
- const capability = geminiCapabilities[model];
- const resources = new MemoryResourceStore();
- const media: GeminiMediaPart[] = [];
- for (const item of parsed.positionals) {
- const path = resolve(environment.cwd, item);
- media.push({ artifact: await resources.put(new Uint8Array(await readFile(path)), mediaType(path)) });
- }
- const need: Need = {
- id: "need:hypit-observe",
- capability,
- returns: geminiTypes.visualObservation,
- constraints: sealGeminiRequest({ instruction, prompt, media }) as unknown as CanonicalValue,
- result: "record:hypit-observe",
- };
- const provider = await selectedProvider(host, need, profile);
- if (!parsed.json) io.write(`Observing with ${model} through ${providerLine(provider)}\n`);
- const fulfillment = await host.invoke(need, resources);
- assert(fulfillment.value.kind === "inline", "the observation came back by reference");
- verifyVisualObservation(fulfillment.value.value);
- const text = (fulfillment.value.value as unknown as VisualObservation).text;
- await writeNew(to, `${text}\n`);
- const view = {
- format: "hypit.video-cli-observe@1",
- model,
- media: parsed.positionals.map((item) => resolve(environment.cwd, item)),
- ...providerView(provider),
- characters: text.length,
- path: to,
- };
- if (parsed.json) io.write(`${JSON.stringify(view, null, 2)}\n`);
- else io.write(`✓ Observation written\n\n ${text.length} characters\n ${to}\n`);
-}
-
// ---------------------------------------------------------------------------------------------------
// transcribe
@@ -497,18 +412,9 @@ async function measure(argv: readonly string[], io: CliIo, environment: Creation
export function writeCreationHelp(io: CliIo, topic?: CreationCommand): void {
const sections: Record<CreationCommand, readonly string[]> = {
- observe: [
- "hypit observe",
- "Look at pictures, clips or recordings with the Gemini Endpoint of the selected Runtime Profile.",
- "",
- " hypit observe <media…> --instruction <text|file> --prompt <text|file> --to <file>",
- " [--model <gemini model>] [--runtime <profile>] [--workspace <project>]",
- "",
- "Writes the observation as text to --to. One immediate request; no Build, Result or state.",
- ],
transcribe: [
"hypit transcribe",
- "Hear a recording with the whisperx-alignment Endpoint of the selected Runtime Profile.",
+ "Establish word times with the whisperx-alignment Endpoint of the selected Runtime Profile.",
"",
" hypit transcribe <audio|video> --to <transcript.json> [--language en|zh|es] [--runtime <profile>] [--workspace <project>]",
"",
@@ -530,7 +436,7 @@ export function writeCreationHelp(io: CliIo, topic?: CreationCommand): void {
};
const chosen = topic === undefined ? creationCommands : [topic];
io.write(`${chosen.map((item) => sections[item].join("\n")).join("\n\n")}\n\n`
- + "observe and transcribe name the Endpoint and its price page before they run; the Profile\n"
+ + "transcribe names the Endpoint and its price page before it runs; the Profile\n"
+ "comes from --runtime or the project's `hypit runtime use` selection. measure spends nothing.\n");
}
@@ -538,7 +444,6 @@ export async function runCreationCli(argv: readonly string[], io: CliIo, environ
const command = argv[0];
assert(isCreationCommand(command), `unknown creation command ${command}`);
if (argv.includes("--help")) { writeCreationHelp(io, command); return; }
- if (command === "observe") await observe(argv, io, environment);
- else if (command === "transcribe") await transcribe(argv, io, environment);
+ if (command === "transcribe") await transcribe(argv, io, environment);
else await measure(argv, io, environment);
}
diff --git a/packages/video-cli/src/media.ts b/packages/video-cli/src/media.ts
index 9968bb48..82b35f90 100644
--- a/packages/video-cli/src/media.ts
+++ b/packages/video-cli/src/media.ts
@@ -4,17 +4,15 @@ import { dirname, extname, join, resolve } from "node:path";
import type { CliIo } from "@hypit/cli";
import { downloadVideo, isVideoUrl } from "@hypit/yt-dlp";
+import sharp from "sharp";
-import { runProcess, runProcessWithInput } from "./process.js";
+import { runProcess, runProcessOutput, runProcessWithInput } from "./process.js";
+import { phraseRanges, readTranscript, wordsAt } from "./transcript.js";
+import type { FrameWords, TranscriptWord } from "./transcript.js";
/**
- * Media preparation for the eyes: dumb, local, stateless.
- *
- * A reference video is read by the Gemini Endpoint through `hypit observe`. These commands only make
- * the pieces that request needs when the whole video is too long or too dense to read at once: a
- * stretch cut to a clip, frames at chosen seconds, a grid of frames drawn large enough to read small
- * type, the pixel-jump boundaries a video shows before anyone knows what it is, and a link turned into
- * a file. Each writes exactly what the caller named and nothing else; none decides what the video is.
+ * Local views of media at chosen times and scales. Transcript annotations use the same seconds as
+ * the input media. These commands expose evidence; editorial interpretation belongs to the author.
*/
export const mediaCommands = ["probe", "cut", "frames", "tile", "tiles", "boundaries", "fetch"] as const;
@@ -291,20 +289,29 @@ export async function cutClip(
}
}
-export async function cutFrame(source: string, at: number, target: string, labelTime = false): Promise<void> {
+export async function cutFrame(source: string, at: number, target: string, labelTime = false): Promise<number> {
if (!labelTime) {
- await runProcess("ffmpeg", ["-hide_banner", "-loglevel", "error", "-y", ...openAt(source, at), "-frames:v", "1", ...JPEG, target]);
- return;
+ const jump = Math.max(0, at - SEEK_RUN_UP);
+ const output = await runProcessOutput("ffmpeg", [
+ "-hide_banner", "-loglevel", "info", "-y", "-copyts", "-start_at_zero",
+ ...(jump > 0 ? ["-ss", String(jump)] : []), "-i", source,
+ "-vf", `trim=start=${at},showinfo`, "-frames:v", "1", "-fps_mode", "passthrough", ...JPEG, target,
+ ]);
+ const base = /config in time_base:\s*(\d+)\/(\d+)/u.exec(output.stderr);
+ const pts = /\bn:\s*0\s+pts:\s*(-?\d+)/u.exec(output.stderr);
+ assert(base !== null && pts !== null, `no video frame exists at or after ${at} s`);
+ return Number(pts[1]) * Number(base[1]) / Number(base[2]);
}
const temporary = await mkdtemp(join(tmpdir(), "hypit-labeled-frame-"));
const clean = join(temporary, "clean.jpg");
try {
- await cutFrame(source, at, clean);
- const label = rasterLabel(timecode(at));
+ const actual = await cutFrame(source, at, clean);
+ const label = rasterLabel(timecode(actual));
await runProcessWithInput("ffmpeg", [
"-hide_banner", "-loglevel", "error", "-y", "-i", clean, ...rawVideoInput(label, 1),
"-filter_complex", "[0:v][1:v]overlay=8:8[out]", "-map", "[out]", "-frames:v", "1", ...JPEG, target,
], label.bytes);
+ return actual;
} finally {
await rm(temporary, { recursive: true, force: true });
}
@@ -312,7 +319,7 @@ export async function cutFrame(source: string, at: number, target: string, label
const TILE_COLUMNS = 3;
const TILE_CELL_WIDTH = 480;
-/** Four frames show what moves in a one-second shot; nine keep a fifteen-second one legible. */
+/** A compact overview default. Callers choose denser samples for fast changes. */
export function tileFrameCount(seconds: number): number { return clamp(Math.round(seconds * 1.5), 4, 9); }
export function tileSampleTimes(start: number, end: number, frameCount: number): readonly number[] {
@@ -321,41 +328,64 @@ export function tileSampleTimes(start: number, end: number, frameCount: number):
return Array.from({ length: frameCount }, (_, index) => round(start + (index + 0.5) * (end - start) / frameCount));
}
-async function labeledTileCell(source: string, at: number, target: string, cellWidth: number): Promise<void> {
- const clean = `${target}.clean.jpg`;
- try {
- await cutFrame(source, at, clean);
- const label = rasterLabel(timecode(at), cellWidth);
- await runProcessWithInput("ffmpeg", [
- "-hide_banner", "-loglevel", "error", "-y", "-i", clean, ...rawVideoInput(label, 1),
- "-filter_complex", `[0:v]scale=${cellWidth}:-2[frame];[frame][1:v]vstack=inputs=2[out]`,
- "-map", "[out]", "-frames:v", "1", ...JPEG, target,
- ], label.bytes);
- } finally {
- await rm(clean, { force: true });
+function escapeMarkup(value: string): string {
+ return value.replaceAll("&", "&amp;").replaceAll("<", "&lt;").replaceAll(">", "&gt;");
+}
+
+async function frameLabel(at: number, width: number, words?: FrameWords): Promise<Buffer> {
+ const lines = [timecode(at)];
+ if (words !== undefined) {
+ lines.push(words.active.length === 0 ? "No timed word at this frame" : words.active.map((word) =>
+ `<b>${escapeMarkup(word.text)}</b> ${word.start!.toFixed(3)}–${word.end!.toFixed(3)} s`).join("\n"));
+ if (words.context.length > 0) lines.push(words.context.map((word) => words.active.includes(word)
+ ? `<span foreground="#ffdc80"><b>${escapeMarkup(word.text)}</b></span>` : escapeMarkup(word.text)).join(" "));
}
+ const text = await sharp({ text: {
+ text: `<span foreground="#f5f5f5">${lines.join("\n")}</span>`,
+ font: `sans ${Math.max(12, Math.round(width / 28))}`, width: width - 16, rgba: true, wrap: "word-char",
+ } }).png().toBuffer({ resolveWithObject: true });
+ return sharp({ create: { width, height: text.info.height + 16, channels: 3, background: "#0c0c0c" } })
+ .composite([{ input: text.data, left: 8, top: 8 }]).png().toBuffer();
}
+type SampledFrame = { readonly requestedAt: number; readonly at: number; readonly words?: FrameWords };
+
export async function tileFrames(
source: string,
times: readonly number[],
target: string,
cellWidth: number,
columns: number,
-): Promise<void> {
- assert(times.length >= 2, "a grid needs at least two frames");
+ words?: readonly TranscriptWord[],
+): Promise<readonly SampledFrame[]> {
+ assert(times.length >= 1, "a grid needs at least one frame");
const temporary = await mkdtemp(join(tmpdir(), "hypit-tile-"));
try {
+ const frames: SampledFrame[] = [];
+ const cells: { picture: Buffer; label: Buffer; height: number; labelHeight: number }[] = [];
for (let index = 0; index < times.length; index += 1) {
- await labeledTileCell(source, times[index]!, join(temporary, `cell-${String(index).padStart(4, "0")}.jpg`), cellWidth);
+ const at = times[index]!;
+ const path = join(temporary, `${index}.jpg`);
+ const actual = await cutFrame(source, at, path);
+ const frameWords = words === undefined ? undefined : wordsAt(words, actual);
+ frames.push({ requestedAt: at, at: actual, ...(frameWords === undefined ? {} : { words: frameWords }) });
+ const picture = await sharp(path).resize({ width: cellWidth }).png().toBuffer({ resolveWithObject: true });
+ const label = await frameLabel(actual, cellWidth, frameWords);
+ const labelHeight = (await sharp(label).metadata()).height!;
+ cells.push({ picture: picture.data, height: picture.info.height, label, labelHeight });
}
const rows = Math.ceil(times.length / columns);
- await runProcess("ffmpeg", [
- "-hide_banner", "-loglevel", "error", "-y", "-framerate", "1", "-pattern_type", "glob",
- "-i", join(temporary, "cell-*.jpg"),
- "-vf", `tile=layout=${columns}x${rows}:nb_frames=${times.length}:padding=8:margin=8:color=black`,
- "-frames:v", "1", ...JPEG, target,
- ]);
+ const pictureHeight = Math.max(...cells.map((cell) => cell.height));
+ const rowHeight = pictureHeight + Math.max(...cells.map((cell) => cell.labelHeight));
+ await sharp({ create: {
+ width: columns * (cellWidth + 8) + 8, height: rows * (rowHeight + 8) + 8,
+ channels: 3, background: "#0c0c0c",
+ } }).composite(cells.flatMap((cell, index) => {
+ const left = 8 + (index % columns) * (cellWidth + 8);
+ const top = 8 + Math.floor(index / columns) * (rowHeight + 8);
+ return [{ input: cell.picture, left, top }, { input: cell.label, left, top: top + pictureHeight }];
+ })).jpeg({ quality: 90 }).toFile(target);
+ return frames;
} finally {
await rm(temporary, { recursive: true, force: true });
}
@@ -393,6 +423,68 @@ export async function visualBoundaries(
// ---------------------------------------------------------------------------------------------------
// Commands
+const SAMPLE_OPTIONS = ["--start", "--end", "--at", "--every", "--transcript", "--around", "--occurrence", "--padding"];
+
+async function transcriptOption(parsed: Parsed, cwd: string): Promise<readonly TranscriptWord[] | undefined> {
+ const path = parsed.options.get("--transcript");
+ return path === undefined ? undefined : readTranscript(resolve(cwd, path));
+}
+
+export function intervalSamples(start: number, end: number, every: number): readonly number[] {
+ assert(Number.isFinite(start) && start >= 0 && Number.isFinite(end) && end > start, "a sample range needs 0 <= start < end");
+ assert(Number.isFinite(every) && every >= 0.001, "--every must be at least 0.001 seconds");
+ return Array.from({ length: Math.ceil((end - start) / every) }, (_, index) => round(start + index * every))
+ .filter((time) => time < end);
+}
+
+function sampleRange(parsed: Parsed, duration: number, words?: readonly TranscriptWord[]): { start: number; end: number } {
+ const around = parsed.options.get("--around");
+ if (around !== undefined) {
+ assert(words !== undefined, "--around requires --transcript");
+ assert(!parsed.options.has("--start") && !parsed.options.has("--end"), "choose --around or --start/--end");
+ const matches = phraseRanges(words, around);
+ assert(matches.length > 0, `No timed phrase matches ${JSON.stringify(around)}`);
+ assert(matches.length === 1 || parsed.options.has("--occurrence"),
+ `The phrase occurs ${matches.length} times: ${matches.map((match, index) => `${index + 1}: ${match.start}–${match.end}s`).join(", ")}. Choose --occurrence <n> or --start/--end.`);
+ const occurrence = integerOption(parsed, "--occurrence", 1, 1);
+ const match = matches[occurrence - 1];
+ assert(match !== undefined, `--occurrence ${occurrence} is beyond the ${matches.length} matches`);
+ const padding = secondsOption(parsed, "--padding", 0.3);
+ const start = Math.max(0, round(match.start - padding));
+ const end = Math.min(duration, round(match.end + padding));
+ assert(end > start, "the phrase lies outside the input media; use its matching transcript");
+ return { start, end };
+ }
+ assert(!parsed.options.has("--occurrence") && !parsed.options.has("--padding"), "--occurrence and --padding apply to --around");
+ const start = secondsOption(parsed, "--start", 0);
+ const end = secondsOption(parsed, "--end", duration);
+ assert(end > start && end <= duration, `range must satisfy 0 <= start < end <= ${duration} s`);
+ return { start, end };
+}
+
+function sampleTimes(parsed: Parsed, duration: number, words?: readonly TranscriptWord[], requireEvery = false): readonly number[] {
+ const at = parsed.options.get("--at");
+ if (at !== undefined) {
+ for (const key of ["--start", "--end", "--every", "--frames", "--around", "--occurrence", "--padding"]) {
+ assert(!parsed.options.has(key), `--at cannot be combined with ${key}`);
+ }
+ const times = secondsList(at, "--at", 1);
+ assert(times.every((time) => time < duration), `a frame time must be before the end of the file (${duration} s)`);
+ assert(times.every((time, index) => index === 0 || time > times[index - 1]!), "sample times must be strictly increasing");
+ return times;
+ }
+ const { start, end } = sampleRange(parsed, duration, words);
+ const every = parsed.options.get("--every");
+ assert(every === undefined || !parsed.options.has("--frames"), "choose --every or --frames");
+ assert(!requireEvery || every !== undefined, "frames requires --at <s,s,…> or --every <seconds>");
+ const times = every === undefined
+ ? tileSampleTimes(start, end, integerOption(parsed, "--frames", tileFrameCount(end - start), 2))
+ : intervalSamples(start, end, Number(every));
+ assert(times.length > 0 && times.every((time) => time < duration), "the range needs sample times before the media ends");
+ assert(times.every((time, index) => index === 0 || time > times[index - 1]!), "sample times must be distinct to the nearest millisecond");
+ return times;
+}
+
async function probe(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
const parsed = parseArguments(argv, []);
const source = await sourceFile(parsed, cwd);
@@ -416,33 +508,22 @@ async function cut(argv: readonly string[], io: CliIo, cwd: string): Promise<voi
}
async function frames(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
- const parsed = parseArguments(argv, ["--at", "--every", "--to"], ["--label-time"]);
+ const parsed = parseArguments(argv, [...SAMPLE_OPTIONS, "--to"], ["--label-time"]);
const source = await sourceFile(parsed, cwd);
const info = await probeMedia(source);
- const at = parsed.options.get("--at");
- const every = parsed.options.get("--every");
- assert((at === undefined) !== (every === undefined), "name the seconds with --at <s,s,…> or sample with --every <seconds>, not both");
- let times: number[];
- if (at !== undefined) {
- times = secondsList(at, "--at", 1);
- } else {
- const step = Number(every);
- assert(Number.isFinite(step) && step > 0, `--every must be a positive number of seconds, got ${every}`);
- times = [];
- for (let time = 0; time < info.duration; time += step) times.push(round(time));
- }
- assert(times.length > 0, "name at least one frame time");
- assert(times.every((value) => value < info.duration), `a frame time must be before the end of the file (${info.duration} s)`);
- assert(new Set(times.map((value) => value.toFixed(3))).size === times.length, "frame times must be distinct to the nearest millisecond");
+ const words = await transcriptOption(parsed, cwd);
+ const times = sampleTimes(parsed, info.duration, words, true);
const to = await destinationDirectory(parsed, cwd, "the new directory to write the frames into");
const staging = await mkdtemp(join(dirname(to), ".hypit-frames-"));
- const labeled = parsed.flags.has("--label-time");
- const written: { readonly at: number; readonly path: string }[] = [];
+ const labeled = parsed.flags.has("--label-time") || words !== undefined;
+ const written: (SampledFrame & { readonly path: string })[] = [];
try {
for (const time of times) {
const name = `frame-${time.toFixed(3).replace(".", "_")}s.jpg`;
- await cutFrame(source, time, join(staging, name), labeled);
- written.push({ at: time, path: join(to, name) });
+ const frame = words === undefined
+ ? { requestedAt: time, at: await cutFrame(source, time, join(staging, name), labeled) }
+ : (await tileFrames(source, [time], join(staging, name), info.width, 1, words))[0]!;
+ written.push({ ...frame, path: join(to, name) });
}
await rename(staging, to);
} catch (error) {
@@ -454,30 +535,18 @@ async function frames(argv: readonly string[], io: CliIo, cwd: string): Promise<
}
async function tile(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
- const parsed = parseArguments(argv, ["--start", "--end", "--at", "--frames", "--cell", "--columns", "--to"]);
+ const parsed = parseArguments(argv, [...SAMPLE_OPTIONS, "--frames", "--cell", "--columns", "--to"]);
const source = await sourceFile(parsed, cwd);
const info = await probeMedia(source);
- const explicit = parsed.options.get("--at");
- let times: readonly number[];
- if (explicit !== undefined) {
- assert(!parsed.options.has("--start") && !parsed.options.has("--end") && !parsed.options.has("--frames"), "--at cannot be combined with --start, --end or --frames");
- times = secondsList(explicit, "--at", 2);
- } else {
- const start = secondsOption(parsed, "--start", 0);
- const end = secondsOption(parsed, "--end", info.duration);
- assert(end <= info.duration + 0.001, `--end ${end} is past the end of the file (${info.duration} s)`);
- const count = integerOption(parsed, "--frames", tileFrameCount(end - start), 2);
- times = tileSampleTimes(start, end, count);
- }
- assert(times.every((value) => value < info.duration), `a sample time must be before the end of the file (${info.duration} s)`);
- assert(times.every((value, index) => index === 0 || value > times[index - 1]!), "sample times must be strictly increasing");
+ const words = await transcriptOption(parsed, cwd);
+ const times = sampleTimes(parsed, info.duration, words);
// Preserve source width unless an exceptionally tiny input needs room for a legible timecode.
const cell = integerOption(parsed, "--cell", Math.max(80, Math.min(TILE_CELL_WIDTH, info.width)), 80);
const columns = integerOption(parsed, "--columns", TILE_COLUMNS, 1);
const target = await destination(parsed, cwd, "the grid image to write, for example notes/hook-grid.jpg");
- await tileFrames(source, times, target, cell, columns);
+ const frames = await tileFrames(source, times, target, cell, columns, words);
const rows = Math.ceil(times.length / columns);
- if (parsed.json) { io.write(`${JSON.stringify({ path: target, samples: times, columns, rows, cellWidth: cell }, null, 2)}\n`); return; }
+ if (parsed.json) { io.write(`${JSON.stringify({ path: target, samples: times, columns, rows, cellWidth: cell, frames }, null, 2)}\n`); return; }
io.write(`${target}\n ${times.length} labeled frames, ${columns}×${rows}, ${cell} px cells\n ${times.map((at) => `${at} s`).join(" ")}\n`);
}
@@ -486,6 +555,7 @@ type TileRange = {
readonly start: number;
readonly end: number;
readonly frames?: number;
+ readonly every?: number;
};
function tileRange(value: unknown, index: number, duration: number): TileRange {
@@ -493,45 +563,72 @@ function tileRange(value: unknown, index: number, duration: number): TileRange {
const raw = value as Record<string, unknown>;
assert(typeof raw.start === "number" && Number.isFinite(raw.start) && raw.start >= 0, `range ${index + 1}.start must be non-negative seconds`);
assert(typeof raw.end === "number" && Number.isFinite(raw.end) && raw.end > raw.start, `range ${index + 1}.end must be after its start`);
- assert(raw.end <= duration + 0.001, `range ${index + 1}.end is past the end of the file (${duration} s)`);
+ assert(raw.end <= duration, `range ${index + 1}.end is past the end of the file (${duration} s)`);
assert(raw.id === undefined || (typeof raw.id === "string" && /^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(raw.id)), `range ${index + 1}.id must be a safe file name`);
assert(raw.frames === undefined || (Number.isSafeInteger(raw.frames) && (raw.frames as number) >= 2), `range ${index + 1}.frames must be a whole number of at least 2`);
- return { start: raw.start, end: raw.end, ...(raw.id === undefined ? {} : { id: raw.id as string }), ...(raw.frames === undefined ? {} : { frames: raw.frames as number }) };
+ assert(raw.every === undefined || (typeof raw.every === "number" && Number.isFinite(raw.every) && raw.every >= 0.001), `range ${index + 1}.every must be at least 0.001 seconds`);
+ assert(raw.frames === undefined || raw.every === undefined, `range ${index + 1} must choose frames or every`);
+ return { start: raw.start, end: raw.end, ...(raw.id === undefined ? {} : { id: raw.id as string }),
+ ...(raw.frames === undefined ? {} : { frames: raw.frames as number }), ...(raw.every === undefined ? {} : { every: raw.every as number }) };
}
async function tiles(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
- const parsed = parseArguments(argv, ["--ranges", "--frames", "--cell", "--columns", "--to"]);
+ const parsed = parseArguments(argv, [...SAMPLE_OPTIONS, "--ranges", "--frames", "--cell", "--columns", "--rows", "--to"]);
const source = await sourceFile(parsed, cwd);
const info = await probeMedia(source);
- const rangesPath = resolve(cwd, required(parsed, "--ranges", "a JSON array of { start, end, id?, frames? } ranges"));
- const rangesRaw = JSON.parse(await readFile(rangesPath, "utf8")) as unknown;
- assert(Array.isArray(rangesRaw) && rangesRaw.length > 0, `${rangesPath} must contain a non-empty JSON array`);
- const ranges = rangesRaw.map((value, index) => tileRange(value, index, info.duration));
- const ids = ranges.flatMap((range) => range.id === undefined ? [] : [range.id]);
- assert(new Set(ids).size === ids.length, "range ids must be unique");
- const defaultFrames = parsed.options.has("--frames") ? integerOption(parsed, "--frames", 2, 2) : undefined;
+ const words = await transcriptOption(parsed, cwd);
+ const rangesPath = parsed.options.get("--ranges");
+ let ranges: { id?: string; start: number; end: number; times: readonly number[] }[];
+ if (rangesPath === undefined) {
+ const times = sampleTimes(parsed, info.duration, words);
+ const range = parsed.options.has("--at")
+ ? { start: times[0]!, end: times.at(-1)! }
+ : sampleRange(parsed, info.duration, words);
+ ranges = [{ ...range, times }];
+ } else {
+ for (const key of ["--start", "--end", "--at", "--around", "--padding", "--occurrence"]) {
+ assert(!parsed.options.has(key), `--ranges cannot be combined with ${key}`);
+ }
+ assert(!parsed.options.has("--frames") || !parsed.options.has("--every"), "choose --frames or --every");
+ const raw = JSON.parse(await readFile(resolve(cwd, rangesPath), "utf8")) as unknown;
+ assert(Array.isArray(raw) && raw.length > 0, `${rangesPath} must contain a non-empty JSON array`);
+ ranges = raw.map((value, index) => {
+ const range = tileRange(value, index, info.duration);
+ const options = new Map(parsed.options);
+ options.set("--start", String(range.start)); options.set("--end", String(range.end));
+ if (range.frames !== undefined) { options.delete("--every"); options.set("--frames", String(range.frames)); }
+ if (range.every !== undefined) { options.delete("--frames"); options.set("--every", String(range.every)); }
+ return { ...range, times: sampleTimes({ ...parsed, options }, info.duration, words) };
+ });
+ const ids = ranges.flatMap((range) => range.id === undefined ? [] : [range.id]);
+ assert(new Set(ids).size === ids.length, "range ids must be unique");
+ }
const cell = integerOption(parsed, "--cell", Math.max(80, Math.min(TILE_CELL_WIDTH, info.width)), 80);
const columns = integerOption(parsed, "--columns", TILE_COLUMNS, 1);
+ const rows = integerOption(parsed, "--rows", 3, 1);
+ const perPage = columns * rows;
const to = await destinationDirectory(parsed, cwd, "the new directory to write the grids into");
const staging = await mkdtemp(join(dirname(to), ".hypit-tiles-"));
- const written: { readonly id?: string; readonly start: number; readonly end: number; readonly samples: readonly number[]; readonly path: string }[] = [];
+ const written: { id?: string; start: number; end: number; samples: readonly number[]; path: string; frames: readonly SampledFrame[] }[] = [];
try {
- for (let index = 0; index < ranges.length; index += 1) {
- const range = ranges[index]!;
- const count = range.frames ?? defaultFrames ?? tileFrameCount(range.end - range.start);
- const times = tileSampleTimes(range.start, range.end, count);
+ for (const [index, range] of ranges.entries()) {
const stem = range.id ?? `${range.start.toFixed(3).replace(".", "_")}s-${range.end.toFixed(3).replace(".", "_")}s`;
- const name = `${String(index + 1).padStart(3, "0")}-${stem}.jpg`;
- await tileFrames(source, times, join(staging, name), cell, columns);
- written.push({ ...(range.id === undefined ? {} : { id: range.id }), start: range.start, end: range.end, samples: times, path: join(to, name) });
+ for (let offset = 0; offset < range.times.length; offset += perPage) {
+ const times = range.times.slice(offset, offset + perPage);
+ const page = range.times.length > perPage ? `-p${String(offset / perPage + 1).padStart(3, "0")}` : "";
+ const name = `${String(index + 1).padStart(3, "0")}-${stem}${page}.jpg`;
+ const frames = await tileFrames(source, times, join(staging, name), cell, columns, words);
+ written.push({ ...(range.id === undefined ? {} : { id: range.id }), start: range.start, end: range.end,
+ samples: times, path: join(to, name), frames });
+ }
}
await rename(staging, to);
} catch (error) {
await rm(staging, { recursive: true, force: true });
throw error;
}
- if (parsed.json) { io.write(`${JSON.stringify({ directory: to, grids: written }, null, 2)}\n`); return; }
- io.write(`${to}\n ${written.length} time-labeled ${written.length === 1 ? "grid" : "grids"} from ${written[0]!.start} s to ${written.at(-1)!.end} s\n`);
+ if (parsed.json) { io.write(`${JSON.stringify({ directory: to, columns, rows, cellWidth: cell, grids: written }, null, 2)}\n`); return; }
+ io.write(`${to}\n ${written.length} time-labeled ${written.length === 1 ? "grid" : "grids"}\n`);
}
async function boundaries(argv: readonly string[], io: CliIo, cwd: string): Promise<void> {
@@ -568,19 +665,22 @@ export function writeMediaHelp(io: CliIo, topic?: MediaCommand): void {
probe: [" hypit media probe <file>", " Duration, size, frame rate and whether there is audio."],
cut: [" hypit media cut <file> --start <s> --end <s> [--label-time] --to <clip.mp4>",
" One exact stretch. --label-time visibly overlays absolute source time on the evidence copy."],
- frames: [" hypit media frames <file> (--at <s,s,…> | --every <s>) [--label-time] --to <dir>",
- " Single frames named by absolute source time; optional visible labels."],
- tile: [" hypit media tile <file> ([--start <s> --end <s>] | --at <s,s,…>) [--frames <n>] [--cell <px>] [--columns <n>] --to <grid.jpg>",
- " One time-labeled grid. Range samples are evenly spaced; --at names exact samples."],
- tiles: [" hypit media tiles <file> --ranges <ranges.json> [--frames <n>] [--cell <px>] [--columns <n>] --to <dir>",
- " One labeled grid per { start, end, id?, frames? } in a JSON array."],
+ frames: [" hypit media frames <file> (--at <s,s,…> | --every <s>) [--start <s> --end <s>] [--label-time] [--transcript <json>] --to <dir>",
+ " Individual frames named by source time; --transcript adds word times and context below each picture."],
+ tile: [" hypit media tile <file> [--start <s> --end <s> | --at <s,s,…>] [--every <s> | --frames <n>] [--transcript <json>] [--cell <px>] [--columns <n>] --to <grid.jpg>",
+ " One grid. --every samples from start, excluding end; --frames chooses evenly spaced midpoints."],
+ tiles: [" hypit media tiles <file> [--start <s> --end <s> | --at <s,s,…> | --ranges <json>] [--every <s> | --frames <n>] [--transcript <json>] [--cell <px>] [--columns <n>] [--rows <n>] --to <dir>",
+ " Paginated grids (3 rows by default). Range files contain { start, end, id?, frames?, every? }."],
boundaries: [" hypit media boundaries <file> [--rate <samples/s>] [--threshold <0..1>]",
" Mechanical adjacent-frame change candidates with scores; never editorial shot labels."],
fetch: [" hypit media fetch <url> --to <video.mp4>", " A link turned into a file with the pinned yt-dlp, video and audio together."],
};
const chosen = topic === undefined ? mediaCommands : [topic];
- io.write(`hypit media\nPrepare what the eyes will look at. Local ffmpeg work; no Runtime Profile, no request, no state.\n\n${
- chosen.map((item) => sections[item].join("\n")).join("\n\n")}\n\nCommands that create evidence write only what --to names and refuse to overwrite. Add --json for the complete machine view.\n`);
+ io.write(`hypit media\nInspect media at chosen times and scales. Local work; no Runtime Profile, no request, no state.\n\n${
+ chosen.map((item) => sections[item].join("\n")).join("\n\n")}\n\nFrames and grids accept --around <phrase> with --transcript instead of --start/--end.\n`
+ + "Use --occurrence <n> for a repeated phrase; --padding <s> adds time on each side (default 0.3).\n"
+ + "Transcript times must share the input media's clock. An untimed gap is labeled as such, without inferring silence.\n"
+ + "Commands that create evidence write only what --to names and refuse to overwrite. Add --json for the complete machine view.\n");
}
export async function runMediaCli(argv: readonly string[], io: CliIo, cwd = process.cwd()): Promise<void> {
diff --git a/packages/video-cli/src/process.ts b/packages/video-cli/src/process.ts
index b963f719..b92defa5 100644
--- a/packages/video-cli/src/process.ts
+++ b/packages/video-cli/src/process.ts
@@ -2,13 +2,14 @@ import { spawn } from "node:child_process";
import { once } from "node:events";
type ProcessInput = Uint8Array | Iterable<Uint8Array> | AsyncIterable<Uint8Array>;
+type ProcessOutput = { readonly stdout: Buffer; readonly stderr: string };
function run(
executable: string,
args: readonly string[],
input: ProcessInput | undefined,
timeoutMs: number,
-): Promise<Buffer> {
+): Promise<ProcessOutput> {
return new Promise((resolveRun, reject) => {
const child = spawn(executable, [...args], { stdio: [input === undefined ? "ignore" : "pipe", "pipe", "pipe"], windowsHide: true });
const out: Buffer[] = [];
@@ -18,7 +19,7 @@ function run(
if (settled) return;
settled = true;
clearTimeout(timer);
- if (error === undefined) resolveRun(Buffer.concat(out));
+ if (error === undefined) resolveRun({ stdout: Buffer.concat(out), stderr: err });
else reject(error);
};
const timer = setTimeout(() => { child.kill("SIGKILL"); finish(new Error(`${executable} timed out after ${timeoutMs} ms`)); }, timeoutMs);
@@ -50,16 +51,21 @@ function run(
}
/** Run one external program to completion and return its stdout; stderr becomes the error text. */
-export function runProcess(executable: string, args: readonly string[], timeoutMs = 300_000): Promise<Buffer> {
+export async function runProcess(executable: string, args: readonly string[], timeoutMs = 300_000): Promise<Buffer> {
+ return (await run(executable, args, undefined, timeoutMs)).stdout;
+}
+
+/** Successful stderr can carry tool metadata, such as the timestamp of an extracted frame. */
+export function runProcessOutput(executable: string, args: readonly string[], timeoutMs = 300_000): Promise<ProcessOutput> {
return run(executable, args, undefined, timeoutMs);
}
/** The same process boundary when a deterministic byte stream is one of the program's inputs. */
-export function runProcessWithInput(
+export async function runProcessWithInput(
executable: string,
args: readonly string[],
input: ProcessInput,
timeoutMs = 300_000,
): Promise<Buffer> {
- return run(executable, args, input, timeoutMs);
+ return (await run(executable, args, input, timeoutMs)).stdout;
}
diff --git a/packages/video-cli/src/transcript.ts b/packages/video-cli/src/transcript.ts
new file mode 100644
index 00000000..578450ce
--- /dev/null
+++ b/packages/video-cli/src/transcript.ts
@@ -0,0 +1,71 @@
+import { readFile } from "node:fs/promises";
+
+/** The seconds-based file written by `hypit transcribe`. Missing word times stay missing. */
+export type TranscriptWord = {
+ readonly text: string;
+ readonly start?: number;
+ readonly end?: number;
+};
+export type FrameWords = {
+ readonly active: readonly TranscriptWord[];
+ readonly context: readonly TranscriptWord[];
+};
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+export async function readTranscript(path: string): Promise<readonly TranscriptWord[]> {
+ const raw = JSON.parse(await readFile(path, "utf8"));
+ assert(raw?.format === "hypit.transcript@1" && Array.isArray(raw.passages), `${path}: expected hypit.transcript@1`);
+ return raw.passages.flatMap((passage: { words?: unknown }) => {
+ assert(passage !== null && Array.isArray(passage.words), `${path}: each passage needs words`);
+ return passage.words.map((word: Record<string, unknown>) => {
+ assert(word !== null && typeof word.text === "string", `${path}: each word needs text`);
+ const start = word.start_seconds;
+ const end = word.end_seconds;
+ assert(start === undefined || (typeof start === "number" && Number.isFinite(start) && start >= 0), `${path}: invalid word start`);
+ assert(end === undefined || (typeof end === "number" && Number.isFinite(end) && end >= 0), `${path}: invalid word end`);
+ assert(start === undefined || end === undefined || end >= start, `${path}: word end precedes start`);
+ return { text: word.text, ...(start === undefined ? {} : { start }), ...(end === undefined ? {} : { end }) };
+ });
+ });
+}
+
+export function wordsAt(words: readonly TranscriptWord[], at: number): FrameWords {
+ const active = words.filter((word) => word.start !== undefined && word.end !== undefined && word.start <= at && at < word.end);
+ let anchor = active.length > 0 ? words.indexOf(active[0]!) : -1;
+ if (anchor < 0) {
+ let distance = Infinity;
+ for (const [index, word] of words.entries()) {
+ const time = word.start ?? word.end;
+ if (time !== undefined && Math.abs(time - at) < distance) { anchor = index; distance = Math.abs(time - at); }
+ }
+ }
+ const last = active.length > 0 ? words.indexOf(active.at(-1)!) : anchor;
+ return { active, context: anchor < 0 ? [] : words.slice(Math.max(0, anchor - 3), last + 4) };
+}
+
+function searchable(text: string): string {
+ return text.normalize("NFKC").toLowerCase().replace(/[\p{P}\p{Z}\s]/gu, "");
+}
+
+/** Find whole, consecutive transcript words; punctuation and case do not affect matching. */
+export function phraseRanges(words: readonly TranscriptWord[], text: string): readonly { start: number; end: number }[] {
+ const query = searchable(text);
+ assert(query.length > 0, "--around needs a word or phrase");
+ const matches: { start: number; end: number }[] = [];
+ for (let first = 0; first < words.length; first += 1) {
+ if (searchable(words[first]!.text).length === 0) continue;
+ let joined = "";
+ for (let last = first; last < words.length && joined.length < query.length; last += 1) {
+ joined += searchable(words[last]!.text);
+ if (joined !== query) continue;
+ const start = words[first]!.start;
+ const end = words[last]!.end;
+ assert(start !== undefined && end !== undefined, `The phrase ${JSON.stringify(text)} has missing boundary times; inspect its passage and choose --start/--end`);
+ matches.push({ start, end });
+ }
+ }
+ return matches;
+}
diff --git a/packages/video-cli/test/creation.test.ts b/packages/video-cli/test/creation.test.ts
index 7e853861..47493e9c 100644
--- a/packages/video-cli/test/creation.test.ts
+++ b/packages/video-cli/test/creation.test.ts
@@ -8,7 +8,6 @@ import { canonicalize } from "@hypit/protocol";
import type { Need } from "@hypit/protocol";
import type { ResourceStore } from "@hypit/runtime";
import { sealAlignedTranscriptEvidence } from "@hypit/speech-evidence";
-import { sealVisualObservation } from "@hypit/gemini";
import { interpretWhisperXTranscript } from "@hypit/whisperx";
import type { WhisperXAlignmentRequest } from "@hypit/whisperx";
@@ -52,8 +51,7 @@ function host(seen: Need[]): CreationHost {
}, request.sampleFrames),
})) } };
}
- const request = need.constraints as { readonly prompt: string; readonly media: readonly unknown[] };
- return { value: { kind: "inline" as const, value: canonicalize(sealVisualObservation(`saw ${request.media.length} media; asked: ${request.prompt}`)) } };
+ throw new Error(`Unexpected capability ${need.capability.name}`);
},
};
}
@@ -67,34 +65,6 @@ function capture() {
return { io: { write: (text: string) => { output += text; } }, text: () => output };
}
-test("observe seals one Gemini request from the media and writes the answer beside the Provider it named", async () => {
- const root = await mkdtemp(join(tmpdir(), "hypit-observe-"));
- const seen: Need[] = [];
- try {
- await writeFile(join(root, "frame.png"), Uint8Array.from([137, 80, 78, 71]));
- await writeFile(join(root, "prompt.txt"), "What is the presenter holding?\n", "utf8");
- const out = capture();
- await runCreationCli([
- "observe", "frame.png", "--instruction", "You only observe.", "--prompt", "prompt.txt", "--to", "notes/frame.md", "--json",
- ], out.io, environment(root, seen));
- const view = JSON.parse(out.text()) as Record<string, unknown>;
- assert.equal(view.endpoint, "paid.default");
- assert.deepEqual(view.pricing, page);
- assert.equal(await readFile(join(root, "notes", "frame.md"), "utf8"), "saw 1 media; asked: What is the presenter holding?\n");
- assert.equal(seen[0]?.capability.name, "gemini-3.1-pro");
- const constraints = seen[0]!.constraints as { readonly instruction: string; readonly media: readonly { readonly artifact: { readonly mediaType: string } }[] };
- assert.equal(constraints.instruction, "You only observe.");
- assert.equal(constraints.media[0]?.artifact.mediaType, "image/png");
-
- const again = capture();
- await assert.rejects(runCreationCli([
- "observe", "frame.png", "--instruction", "x", "--prompt", "y", "--to", "notes/frame.md",
- ], again.io, environment(root, seen)), /already exists/u);
- } finally {
- await rm(root, { recursive: true, force: true });
- }
-});
-
test("transcribe writes every word in seconds from the Profile's alignment Endpoint", async () => {
const root = await mkdtemp(join(tmpdir(), "hypit-transcribe-"));
const seen: Need[] = [];
diff --git a/packages/video-cli/test/media.test.ts b/packages/video-cli/test/media.test.ts
index 73015c5a..7c8ff357 100644
--- a/packages/video-cli/test/media.test.ts
+++ b/packages/video-cli/test/media.test.ts
@@ -4,6 +4,7 @@ import { mkdtemp, readdir, rm, stat, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
+import sharp from "sharp";
import type { CliIo } from "@hypit/cli";
@@ -130,3 +131,49 @@ test("fetch refuses anything but an http link and an explicit video destination"
await assert.rejects(runMediaCli(["media", "fetch", "./local.mp4", "--to", "x.mp4"], io().io), /http or https link/);
await assert.rejects(runMediaCli(["media", "fetch", "https://example.com/v", "--to", "x.txt"], io().io, tmpdir()), /must end in/);
});
+
+test("word-located grids paginate dense samples without covering the source picture", { skip: !ffmpeg && "ffmpeg is not installed" }, async () => {
+ const work = await mkdtemp(join(tmpdir(), "hypit-word-grids-"));
+ try {
+ const source = await sample(work);
+ const transcript = join(work, "words.json");
+ await writeFile(transcript, JSON.stringify({ format: "hypit.transcript@1", passages: [{ words: [
+ { text: "Hello", start_seconds: 0.1, end_seconds: 0.4 },
+ { text: "world!", start_seconds: 0.4, end_seconds: 0.8 },
+ { text: "<你好 & café>" },
+ { text: "Hello", start_seconds: 2.1, end_seconds: 2.4 },
+ { text: "world!", start_seconds: 2.4, end_seconds: 2.8 },
+ ] }] }));
+ await assert.rejects(runMediaCli(["media", "tile", source, "--around", "hello world", "--transcript", transcript,
+ "--to", "ambiguous.jpg"], io().io, work), /occurs 2 times.*--occurrence/);
+ const around = io();
+ await runMediaCli(["media", "tile", source, "--around", "hello world", "--occurrence", "2", "--padding", "0.1",
+ "--every", "0.2", "--transcript", transcript, "--to", "around.jpg", "--json"], around.io, work);
+ const located = JSON.parse(around.text());
+ assert.deepEqual(located.samples, [2, 2.2, 2.4, 2.6, 2.8]);
+ assert.deepEqual(located.frames[0].words.active, []);
+ assert.equal(located.frames[1].words.active[0].text, "Hello");
+ assert.equal(located.frames[1].requestedAt, 2.2);
+ assert.ok(Math.abs(located.frames[1].at - 53 / 24) < 0.000001, "labels use the decoded frame time, not the requested seek time");
+ assert.equal(located.frames[2].words.active[0].text, "world!");
+
+ const pages = io();
+ await runMediaCli(["media", "tiles", source, "--start", "0", "--end", "3", "--every", "0.6",
+ "--transcript", transcript, "--columns", "2", "--rows", "1", "--to", "pages", "--json"], pages.io, work);
+ const grids = JSON.parse(pages.text()).grids;
+ assert.deepEqual(grids.map((grid: { samples: number[] }) => grid.samples), [[0, 0.6], [1.2, 1.8], [2.4]]);
+ assert.ok(grids.every((grid: { start: number; end: number }) => grid.start === 0 && grid.end === 3));
+ assert.equal((await readdir(join(work, "pages"))).length, 3);
+ const image = await sharp(grids[0].path).raw().toBuffer({ resolveWithObject: true });
+ const center = (128 * image.info.width + 168) * image.info.channels;
+ assert.ok(image.data[center]! > 220 && image.data[center + 1]! < 20, "the first source frame stays red and unobscured");
+ assert.ok(image.info.height > 240 + 32, "word annotations have their own space beneath the picture");
+
+ const frames = io();
+ await runMediaCli(["media", "frames", source, "--start", "0.5", "--end", "1.5", "--every", "0.4",
+ "--transcript", transcript, "--to", "frames", "--json"], frames.io, work);
+ assert.deepEqual(JSON.parse(frames.text()).frames.map((frame: { requestedAt: number }) => frame.requestedAt), [0.5, 0.9, 1.3]);
+ } finally {
+ await rm(work, { recursive: true, force: true });
+ }
+});
diff --git a/packages/video-cli/test/transcript.test.ts b/packages/video-cli/test/transcript.test.ts
new file mode 100644
index 00000000..a4b9e7b7
--- /dev/null
+++ b/packages/video-cli/test/transcript.test.ts
@@ -0,0 +1,26 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { phraseRanges, wordsAt } from "../src/transcript.js";
+
+test("word annotations retain missing times and distinguish word boundaries, overlap and gaps", () => {
+ const words = [
+ { text: "hello", start: 0, end: 0.4 }, { text: "world", start: 0.4, end: 0.8 },
+ { text: "yes", start: 0.6, end: 0.9 }, { text: "untimed" }, { text: "again", start: 1.2, end: 1.5 },
+ ];
+ assert.deepEqual(wordsAt(words, 0.4).active.map((word) => word.text), ["world"]);
+ assert.deepEqual(wordsAt(words, 0.7).active.map((word) => word.text), ["world", "yes"]);
+ assert.deepEqual(wordsAt(words, 1).active, []);
+ assert.ok(wordsAt(words, 1).context.some((word) => word.text === "untimed" && word.start === undefined));
+});
+
+test("phrase location preserves repeated occurrences and multilingual word boundaries", () => {
+ const words = [
+ { text: "Hello,", start: 0.1, end: 0.3 }, { text: "world!", start: 0.3, end: 0.8 },
+ { text: "hello", start: 2.1, end: 2.4 }, { text: "world", start: 2.4, end: 2.8 },
+ { text: "你好", start: 3, end: 3.2 }, { text: "世界。", start: 3.2, end: 3.5 },
+ ];
+ assert.deepEqual(phraseRanges(words, "HELLO world"), [{ start: 0.1, end: 0.8 }, { start: 2.1, end: 2.8 }]);
+ assert.deepEqual(phraseRanges(words, "你好世界"), [{ start: 3, end: 3.5 }]);
+ assert.deepEqual(phraseRanges(words, "hell"), []);
+ assert.throws(() => phraseRanges([{ text: "hello" }], "hello"), /missing boundary times/);
+});
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index a21c0d11..33cfde3d 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -90,9 +90,6 @@ importers:
'@hypit/fonts-open':
specifier: workspace:*
version: link:packages/fonts-open
- '@hypit/gemini':
- specifier: workspace:*
- version: link:packages/gemini
'@hypit/generation':
specifier: workspace:*
version: link:packages/generation
@@ -165,9 +162,6 @@ importers:
'@hypit/provider-media-local':
specifier: workspace:*
version: link:packages/provider-media-local
- '@hypit/provider-vertex':
- specifier: workspace:*
- version: link:packages/provider-vertex
'@hypit/provider-whisperx-local':
specifier: workspace:*
version: link:packages/provider-whisperx-local
@@ -1362,27 +1356,6 @@ importers:
specifier: 0.2.0
version: 0.2.0
- packages/gemini:
- dependencies:
- '@hypit/artifact':
- specifier: workspace:*
- version: link:../artifact
- '@hypit/component-kit':
- specifier: workspace:*
- version: link:../component-kit
- '@hypit/elaborator':
- specifier: workspace:*
- version: link:../elaborator
- '@hypit/markup':
- specifier: workspace:*
- version: link:../markup
- '@hypit/protocol':
- specifier: workspace:*
- version: link:../protocol
- '@hypit/text':
- specifier: workspace:*
- version: link:../text
-
packages/generation:
dependencies:
'@hypit/artifact':
@@ -2006,9 +1979,6 @@ importers:
'@hypit/endpoint-kit':
specifier: workspace:*
version: link:../endpoint-kit
- '@hypit/gemini':
- specifier: workspace:*
- version: link:../gemini
'@hypit/generation':
specifier: workspace:*
version: link:../generation
@@ -2203,34 +2173,6 @@ importers:
specifier: workspace:*
version: link:../driver-node
- packages/provider-vertex:
- dependencies:
- '@google/genai':
- specifier: 1.52.0
- version: 1.52.0
- '@hypit/endpoint-kit':
- specifier: workspace:*
- version: link:../endpoint-kit
- '@hypit/gemini':
- specifier: workspace:*
- version: link:../gemini
- '@hypit/protocol':
- specifier: workspace:*
- version: link:../protocol
- '@hypit/runtime':
- specifier: workspace:*
- version: link:../runtime
- '@hypit/runtime-kit':
- specifier: workspace:*
- version: link:../runtime-kit
- '@hypit/text':
- specifier: workspace:*
- version: link:../text
- devDependencies:
- '@hypit/driver-node':
- specifier: workspace:*
- version: link:../driver-node
-
packages/provider-whisperx-local:
dependencies:
'@hypit/endpoint-kit':
@@ -3294,9 +3236,6 @@ importers:
'@hypit/estimate':
specifier: workspace:*
version: link:../estimate
- '@hypit/gemini':
- specifier: workspace:*
- version: link:../gemini
'@hypit/generation':
specifier: workspace:*
version: link:../generation
@@ -3345,6 +3284,9 @@ importers:
'@hypit/yt-dlp':
specifier: workspace:*
version: link:../yt-dlp
+ sharp:
+ specifier: 0.35.3
+ version: 0.35.3(@types/[email protected])
packages/visual-ir:
dependencies:
@@ -7207,6 +7149,7 @@ snapshots:
- bufferutil
- supports-color
- utf-8-validate
+ optional: true
dependencies:
@@ -7453,25 +7396,34 @@ snapshots:
'@oxc-project/[email protected]': {}
- '@protobufjs/[email protected]': {}
+ '@protobufjs/[email protected]':
+ optional: true
- '@protobufjs/[email protected]': {}
+ '@protobufjs/[email protected]':
+ optional: true
- '@protobufjs/[email protected]': {}
+ '@protobufjs/[email protected]':
+ optional: true
- '@protobufjs/[email protected]': {}
+ '@protobufjs/[email protected]':
+ optional: true
'@protobufjs/[email protected]':
dependencies:
'@protobufjs/aspromise': 1.1.2
+ optional: true
- '@protobufjs/[email protected]': {}
+ '@protobufjs/[email protected]':
+ optional: true
- '@protobufjs/[email protected]': {}
+ '@protobufjs/[email protected]':
+ optional: true
- '@protobufjs/[email protected]': {}
+ '@protobufjs/[email protected]':
+ optional: true
- '@protobufjs/[email protected]': {}
+ '@protobufjs/[email protected]':
+ optional: true
'@puppeteer/[email protected]':
dependencies:
@@ -7710,7 +7662,8 @@ snapshots:
dependencies:
undici-types: 7.16.0
- '@types/[email protected]': {}
+ '@types/[email protected]':
+ optional: true
'@types/[email protected]': {}
@@ -7836,7 +7789,8 @@ snapshots:
transitivePeerDependencies:
- supports-color
+ optional: true
dependencies:
@@ -7898,7 +7852,8 @@ snapshots:
+ optional: true
@@ -7914,7 +7869,8 @@ snapshots:
dependencies:
base64-js: 1.5.1
+ optional: true
@@ -7981,7 +7937,8 @@ snapshots:
+ optional: true
dependencies:
@@ -8061,6 +8018,7 @@ snapshots:
dependencies:
safe-buffer: 5.2.1
+ optional: true
@@ -8182,7 +8140,8 @@ snapshots:
transitivePeerDependencies:
- bare-abort-controller
+ optional: true
@@ -8196,6 +8155,7 @@ snapshots:
dependencies:
node-domexception: 1.0.0
web-streams-polyfill: 3.3.3
+ optional: true
dependencies:
@@ -8227,6 +8187,7 @@ snapshots:
dependencies:
fetch-blob: 3.2.0
+ optional: true
optional: true
@@ -8238,6 +8199,7 @@ snapshots:
node-fetch: 3.3.2
transitivePeerDependencies:
- supports-color
+ optional: true
dependencies:
@@ -8246,6 +8208,7 @@ snapshots:
json-bigint: 1.0.0
transitivePeerDependencies:
- supports-color
+ optional: true
@@ -8281,8 +8244,10 @@ snapshots:
jws: 4.0.1
transitivePeerDependencies:
- supports-color
+ optional: true
+ optional: true
@@ -8340,6 +8305,7 @@ snapshots:
debug: 4.4.3
transitivePeerDependencies:
- supports-color
+ optional: true
dependencies:
@@ -8393,6 +8359,7 @@ snapshots:
dependencies:
bignumber.js: 9.3.1
+ optional: true
@@ -8401,11 +8368,13 @@ snapshots:
buffer-equal-constant-time: 1.0.1
ecdsa-sig-formatter: 1.0.11
safe-buffer: 5.2.1
+ optional: true
dependencies:
jwa: 2.0.1
safe-buffer: 5.2.1
+ optional: true
optional: true
@@ -8466,7 +8435,8 @@ snapshots:
htmlparser2: 10.1.0
uhyphen: 0.2.0
+ optional: true
dependencies:
@@ -8525,13 +8495,15 @@ snapshots:
+ optional: true
dependencies:
data-uri-to-buffer: 4.0.1
fetch-blob: 3.2.0
formdata-polyfill: 4.0.10
+ optional: true
dependencies:
@@ -8568,6 +8540,7 @@ snapshots:
dependencies:
'@types/retry': 0.12.0
retry: 0.13.1
+ optional: true
@@ -8617,6 +8590,7 @@ snapshots:
'@protobufjs/utf8': 1.1.2
'@types/node': 24.10.1
long: 5.3.2
+ optional: true
dependencies:
@@ -8679,7 +8653,8 @@ snapshots:
+ optional: true
@@ -9068,7 +9043,8 @@ snapshots:
dependencies:
argparse: 2.0.1
+ optional: true
diff --git a/skills/hypit/SKILL.md b/skills/hypit/SKILL.md
index 81a7f632..653486dd 100644
--- a/skills/hypit/SKILL.md
+++ b/skills/hypit/SKILL.md
@@ -1,21 +1,23 @@
---
name: hypit
-description: Make, adapt, and revise videos with Hypit from references or briefs, including SVML/SVS/SVRun authoring, project components, and Runtime or credential setup. Use for video production and project-local components, not Hypit framework development.
+description: Make, adapt, and revise videos with Hypit from references or briefs, including SVML/SVS/SVRun authoring, project components, and Runtime or credential setup.
---
# Hypit
You are the director and producer entrusted with delivering the user's video. Look at the material
they give you, understand the intention behind their request, and develop a creative answer you can
-stand behind. Gemini and WhisperX are your eyes and ears; image, video, and audio models are your
-crew; Author Packages are your craft; Studio is your editing room; the Runtime is the production
-facility. You make sense of the evidence, discover why the work holds attention, and make the
+stand behind. Watch the reference, inspect its frames, and read the words in time. Image, video, and
+audio models produce your directed material; Author Packages express the composition; Studio is your
+editing room; the Runtime runs the production. Discover why the work holds attention, and make the
aesthetic and technical choices that bring the new piece to life.
Bring the sensibility the video calls for: quick internet wit, warmth, social intuition, restraint,
or playful absurdity. Let it shape your ideas, images, words, and performances. Through every style,
-remain grounded, curious, imaginative, and discerning. Care about the details and stay honest about
-what the work actually achieves. Speak in the user's language, with warmth and specifics from what
+remain grounded, curious, imaginative, and discerning. Be perceptive about the whole, careful and
+honest about the details, and curious when something does not yet make sense. Look for connections
+that explain an unexpected choice. Draw on the relevant Skill references and keep what you learn
+in the project files as you go. Speak in the user's language, with warmth and specifics from what
you have understood, what you are considering, and what the next useful action will resolve.
Most work Hypit is asked to clone is creator-led, phone-captured short-form UGC: spoken expression is
@@ -45,45 +47,52 @@ GPT Image 2 and Seedance 2 Mini at 720p are the usual starting points for genera
performances, balancing capability and cost. Choose for the actual work; the image and video
crafts explain resolution and duration choices.
-## Establish the working crew
-
-At the beginning of reference reconstruction, read the
-[Runtime Profile](references/environment/profile.md) and establish the capabilities needed to
-understand the supplied work and produce the new piece: moving-image observation, word alignment for
-spoken work, and the image, video, voice or audio generation demanded by the Treatment. Continue when
-they are reachable. When they are not, present the material gaps together, explain their consequence
-for this work, and let the user choose among the real account, local and hosted options before relying
-on the missing capability.
-
-## Two loops, one director
-
-Understanding a reference is a loop between coarse and fine: the whole-piece reading tells your eyes
-and ears where a closer look would change the work, and close evidence rewrites the whole-piece
-reading. A limitation accepted earlier does not freeze the work at that level. When a newly reachable
-capability or a new Result can materially improve an earlier judgment, return to that judgment and
-upgrade it before relying on it for further creative or paid work. Preserve what remains sound and
-revise what the new evidence changes.
-
-Making the new piece is a loop between intention and result: the Brief holds what the user asked for,
-the Treatment is your directorial answer, and Source, Recipe, Run, and components make it exact.
-Clone work often arrives as "make this with my face" or "use my product." Think through what the new
-request means for the whole piece, carrying forward the reference's useful relationships and
-reshaping the story, performance, and visual world where the new intention calls for it. Use
-Studio and rendered Results to repair the implementation; reconsider Treatment when the creative
-design itself changes. Brief changes when the user's goal or constraints change.
-
-Create project components as normal production work. Let the authorized media requests complete and
-continue downstream; refine deterministic Caption, MG, Effects, and composition through Studio and
-rendering, reusing produced media through Run Candidates. Supplied and produced images and videos
-are the primary visual evidence for the work. A preview substitute is a short-lived answer to a
-current composition or wiring question, not a required stage, approval gate, or production state;
-select the most representative available media as soon as it exists. Environment, creation, and
-production are rooms you enter whenever the current question leads there. They are not stages that
-close behind the work.
+## Prepare for the work at hand
+
+Use the [Runtime Profile](references/environment/profile.md) to establish word alignment for spoken
+work and the generation, processing and rendering that the production needs. Inspect the existing
+setup before asking for it. Explain related missing capabilities together, using the actual account,
+local and hosted options and their consequences for this piece. Watching supplied material and
+developing its interpretation can proceed while that setup is being resolved.
+
+## Understand and direct
+
+Understand the whole reference and the designed behavior that makes it work. Follow its argument or
+story from opening to close: what should the viewer feel, learn or decide, and how does the piece
+bring that about? Inspect its distinct visual systems through their content, placement, entry,
+movement, persistence and exit, tied to words or actions and surrounding elements. Close reading
+can reveal relationships that change your understanding of the whole. Use
+[reference understanding](references/creation/reference-video.md) to make both scales concrete.
+Record the whole-piece explanation in Analysis and the timed details and their meaning in Timeline
+as you discover them. Compression means explaining the relationships precisely with fewer, better
+ideas while preserving the details that make them work.
+
+The Brief holds the user's goal; the Treatment is your creative answer. Clone work often means
+"make this with my face" or "use my product." Reconsider the argument, words, images and graphic
+relationships for that request. The reference helps explain what works; the new performance supplies
+the new timing. Give generated media strong direction through Craft, reference images and prompts
+before submitting the requests.
+
+## Compose and refine
+
+Build around the directed material and existing Outputs. Watch how MG, Caption, B-roll, Typography
+and Effects work together in Studio or rendered Results: the graphics make the idea clear,
+the Caption is easy to read, and the pictures and effects appear where and when they serve the
+passage. Adjust the owning Source or component to improve those relationships. Reuse produced media
+through Run Candidates. A useful change makes a comparison readable, lands a reveal on its word, or
+lets B-roll cover the explanation it supports. The composition is ready when its layout and timing
+carry the Treatment clearly and compellingly. Reconsider Treatment when the design itself needs to
+change; change Brief when the user's goal changes.
+
+Create project components as normal production work. Use the most representative available media
+for the question being explored, and carry useful findings back to their owning project documents.
+Environment, creation and production are rooms to revisit whenever the current question leads there.
+When new material, tools or understanding change an earlier judgment or make a better solution
+practical, revisit that choice and carry the improvement into the work and its notes.
## Standing responsibilities
-- **Money.** Before paid work, including observation and transcription, explain the actual requests,
+- **Money.** Before paid work, including transcription, explain the actual requests,
selected Endpoints, and the current Provider pricing information available for them. Use `hypit plan`
with the selected Runtime Profile, then `hypit pricing` when its Providers expose machine-readable
material; interpret that material against the listed Needs instead of treating it as a system verdict.
@@ -107,10 +116,12 @@ close behind the work.
- **Evidence and reading.** Write what a tile, frame, clip, or transcript shows, and write your
interpretation as your interpretation. Unsupported claims about observed media remain unknown;
choose another view when resolving them would change the work.
-- **Files are the memory.** Keep reference understanding, the user's Brief, your Treatment, and
- current progress in their own project documents. Record useful discoveries and decisions while
- they are fresh, updating current notes in place. After an interruption, resume from those notes,
- Sources, Runs, project Results, and Runtime status. Continue using produced work already available.
+- **Files are the memory.** Make the project's understanding and decisions durable as they develop.
+ Keep the whole-piece reading in Analysis, its timed realization and meaning in Timeline, the user's
+ goal in Brief, and your creative answer in Treatment. Progress carries the current question, what
+ remains to examine, and the next useful action. Keep these files current enough for another
+ conversation to continue the work. Resume by reading them with the relevant Skill references,
+ Sources, Runs, project Results and Runtime status. Continue using produced work already available.
The project-files reference below owns document responsibilities and the suggested layout.
- **Who decides.** The user owns the goal, private facts, real value choices, and spending. You own
casting, art direction, shots, performance, Caption, components, prompts, and implementation. Ask
@@ -120,8 +131,8 @@ close behind the work.
Make production changes at their owning source; keep Result media intact. New reusable behavior
belongs in a project component, without patching the installed Distribution for one video.
- **Done means watched.** Watch the actual deliverable and judge it against the Brief, Treatment,
- and relevant reference relationships. Also judge the work's clarity, performance, rhythm, visual
- and sonic coherence, and suitability for publishing. Deliver the work with the important choices
+ and relevant reference relationships. Judge its clarity, visual hierarchy, timing, character,
+ and suitability for publishing. Deliver the work with the important choices
and limitations explained.
## Where the current question is answered
@@ -148,7 +159,7 @@ close behind the work.
| directing music, sound effects, ambience, gain, ducking or the completed mix | `references/playbooks/craft/sound-mix.md` |
| writing Sources, Recipes, and Runs, reusing produced work, adding a component | `references/production/authoring.md` |
| imports, output references, literal values or Recipe rules | `references/production/source-syntax.md` |
-| assembling reusable prompt wording or authoring a Prompt Kit | `references/production/prompt-kits.md` |
+| choosing an existing Prompt Kit, assembling its wording or authoring a new one | `references/production/prompt-kits.md` |
| Run syntax, Targets, Candidates, Run Fragments or substitute media for a preview | `references/production/runs.md` |
| admitting media, normalization, SemanticTakes, still clips, trims or extraction | `references/production/media.md` |
| composing, correcting, resizing, cropping or cutting out an image | `references/production/image-operations.md` |
@@ -158,6 +169,7 @@ close behind the work.
| which installed Surface to use, or whether to write a project component | `references/production/vocabulary.md` |
| sharing a component, Prompt Kit, Model, or Provider across projects | `references/production/component-sharing.md` |
| the A-roll semantic timeline and composing Media, Audio, Caption, Text, MG and Effect Tracks | `references/production/tracks.md` |
+| designing a good component: its visual idea, semantic behavior, useful controls and fit in the work | `references/production/component-design.md` |
| writing a project Track with new layout, semantic events or persistent state | `references/production/track-authoring.md` |
| drawing a component's elements, animation, resources or prepared surfaces | `references/production/component-visuals.md` |
| writing a Caption family with new word relationships, scheduling or layout | `references/production/caption-authoring.md` |
diff --git a/skills/hypit/references/creation/brief.md b/skills/hypit/references/creation/brief.md
index 685598a4..ce371d8c 100644
--- a/skills/hypit/references/creation/brief.md
+++ b/skills/hypit/references/creation/brief.md
@@ -72,6 +72,6 @@ Selections, and Moments.
- New evidence about the reference changes its Analysis or Timeline; it may then cause the director to
reconsider Treatment.
-Studio and built Results participate in this loop. If the work looks wrong because the design itself
-was wrong, revise Treatment. If the design is sound and its implementation is wrong, repair the
-production. Brief changes only when the user's truth changes.
+Studio and built Results make the current design visible. If the work looks wrong because the
+design itself was wrong, revise Treatment. If the design is sound and its implementation is wrong,
+repair the production. Brief changes only when the user's truth changes.
diff --git a/skills/hypit/references/creation/project-files.md b/skills/hypit/references/creation/project-files.md
index 6c629728..c40b8c76 100644
--- a/skills/hypit/references/creation/project-files.md
+++ b/skills/hypit/references/creation/project-files.md
@@ -93,17 +93,20 @@ referenced in place; its documentary role does not require a duplicate.
- `ANALYSIS.md` is the current whole-piece understanding of a reference: form, story, recurring
systems, relationships, function, and why the work holds together.
-- `TIMELINE.md` says what happens when in the reference, including concurrent picture, speech,
- Caption, Typography, MG, Effect, and Audio behavior.
+- `TIMELINE.md` connects what happens when with what it does for the viewer. Preserve the content,
+ placement, entry, changes, persistence and exit of the reference's visual systems, their word or
+ action relationships, source times and useful evidence paths. Concurrent picture, speech, Caption,
+ Typography, MG, Effect and Audio behavior belong to the same connected account.
- `transcript.json` is word-level speech evidence. It is evidence, not the director's interpretation.
- `evidence/` contains only media worth reopening, with ordinary human-readable names.
- `BRIEF.md` preserves the user's goal, facts, constraints, and requested changes.
- `TREATMENT.md` is the director's current answer to the Brief: the intended new piece in complete
creative terms, before implementation details.
- `authors/`, `recipes/`, and `runs/` are the exact production implementation.
-- `PROGRESS.md` is a short photograph of the work now: the live question, next useful action, real
- blockers, active Build ids and reusable Results. When handing work over, retain the relevant Run
- and Runtime Profile, exact Build id and public Output names needed to continue. Run Candidates own
+- `PROGRESS.md` is a short photograph of the work now: the live question, what remains to examine or
+ make, next useful action, real blockers, active Build ids and reusable Results. When handing work
+ over, retain the relevant Run and Runtime Profile, exact Build id and public Output names needed
+ to continue. Run Candidates own
the actual reuse choices; the note points to them. Established conclusions belong in their owning
document instead.
@@ -111,6 +114,12 @@ Rewrite these files when the current truth changes. They are not logs. `PROGRESS
a reference or a production because either kind of work can span conversations; it does not mark a
stage and can disappear when there is nothing useful to hand over.
+Use these files as the working memory across conversations. Write discoveries and decisions into
+their owning documents as the work develops, preserving both the explanation and its concrete
+details. Resume by reading the current project files and relevant Skill pages; use Progress to find
+the next question and the saved evidence, Sources, Runs or Results to continue it. A new discovery
+updates the account it changes so the next session inherits the improved understanding.
+
## Drafts and Results are different things
`drafts/` holds unpublished authoring material that is not part of the current Source: trial copy,
diff --git a/skills/hypit/references/creation/reference-video.md b/skills/hypit/references/creation/reference-video.md
index 46f6a362..9c91208e 100644
--- a/skills/hypit/references/creation/reference-video.md
+++ b/skills/hypit/references/creation/reference-video.md
@@ -1,172 +1,157 @@
# Understanding a reference video
Read this when a video or link is evidence for what the new work should preserve, adapt, or learn
-from.
-
-## Direct the investigation
-
-The Agent is the mind and director. Moving-image observation is its broad eye, transcription and
-alignment are its ears, and local media commands prepare closer views that the Agent can inspect
-itself. The tools report evidence; the Agent connects events across time, resolves contradictions,
-explains why they matter, and decides what to inspect next. Gemini supplies moving-image observations;
-the Agent connects them into a whole-piece understanding and authors the target work.
-
-Begin by watching the whole piece. For ordinary reconstruction, give the whole work or meaningful
-spans to a capable moving-image observer so motion, order and distant relationships are actually in
-evidence. Spoken language carries most work Hypit is asked to clone, so obtain and read its word-level
-transcript and use it as the normal time spine. If the piece is genuinely speechless, establish that
-spine from its actions, visual changes, music, effects and silence. Form a provisional account of:
-
-- the Hook, story movement, payoff, and intended viewer response;
-- how speech or silence carries the piece, who speaks when present, and the roles of A-roll and B-roll;
-- the persistent and recurring Caption, Typography, MG, Effect, and Audio systems;
-- changes, accumulations, replacements, reveals, cover relationships, and callbacks;
-- the few relationships that make the piece recognizably itself.
-
-Then spend detail attention where an answer can change the production. A ranking board that already
-contains players while a new portrait appears nearby raises a structural question: whether the new
-portrait enters, replaces, or accumulates on the board. That matters before the exact direction of an
-incidental slide. Coarse understanding tells you what deserves a close look; close evidence is allowed
-to overturn the coarse understanding.
-
-## Choose evidence for the question
-
-Use `hypit transcribe` for speech and word time. Use `hypit observe` as the normal broad visual pass
-for reference reconstruction, over the whole short work or meaningful spans of a longer or denser
-work. Motion, sequence, persistence, transitions and whole-piece context need moving-image evidence.
-The local `hypit media` family prepares views without interpreting them:
-
-- `probe` reports media facts;
-- `cut` isolates a short temporal question;
-- `frames` exposes exact moments or small details;
-- `tile` and `tiles` show change across a span with visible time labels;
-- `boundaries` offers possible visual-change locations for navigation;
-- `fetch` turns a supported link into a local source file.
-
-For example, these commands preserve original-media time while exposing different evidence:
+from. The useful result is a timecode-level semantic reading: what the piece communicates, and how
+its exact audiovisual choices make that communication work.
+
+## Read the whole through its details
+
+Watch the whole reference from opening to close. Follow its hook, argument or story, shifts of
+attention, payoff and intended viewer response. Read the spoken words with their times. Notice who
+establishes the performance, what the pictures contribute, and which Caption, MG, Typography, Effect
+and Audio systems persist or recur. Form a provisional explanation of why the piece works.
+
+Read that meaning through its concrete realization. Each designed element deserves an account of its
+content and appearance, spatial relationship, entry, active behavior, persistence and exit. Locate
+these changes against the words, pauses or actions they serve. Explain what the choice does for the
+viewer: establishes a person, contrasts two claims, accumulates proof, amplifies a reaction, makes
+a process legible, or hands attention to the next idea. Meaning directs the close look; the close
+look supplies the detail needed to recreate the expression. Close reading also reveals purposes and
+connections that the first whole-piece impression missed.
+
+When a choice is puzzling, widen the view: relate it to the surrounding words, nearby events, earlier
+appearances and later consequences. Consider what those connections could explain, then examine the
+relevant sequence. Read the Format or Craft for the relationship you are investigating; use it to
+sharpen the question and recognize what the reference is doing.
+
+A motion's name is a starting point. Inspect its path, scale or opacity change, pace, overshoot or
+settling, and relationship to neighboring events when those make the effect distinctive. Likewise,
+understand a graphic's hierarchy, typography, palette and spacing through what the viewer must read
+first and compare next. Compression comes from identifying a coherent behavior that explains many
+frames. A dimmer copy during an exit may be one object's fade, for example; inspect the sequence to
+understand the change.
+
+Whole-piece and close readings revise each other. An object that survives a camera cut belongs to a
+system with a longer lifetime than that shot. Follow its state across the piece: a new entry may
+replace an item, add to a collection, or temporarily cover it. Document the system once and locate
+its changes in time. Camera boundaries help navigation; meaning and continuity determine which
+things belong together.
+
+## Make time visible
+
+Use `hypit transcribe` to establish word-level times for spoken work. Keep that transcript beside the
+reference: it is the temporal spine for connecting speech to cuts, illustrations, reveals and
+emphasis. For speechless work, locate meaning through actions and changes in the scene.
+
+The local `hypit media` commands expose the source at the scale needed:
+
+- `probe` gives duration, dimensions, frame rate and audio presence;
+- `cut` saves a selected passage as a clip;
+- `frames` extracts chosen moments or a range at a chosen interval;
+- `tile` and `tiles` arrange time-labeled frames for inspecting change, with optional word context;
+- `boundaries` locates abrupt visual changes worth inspecting;
+- `fetch` saves a supported video link locally.
+
+Use broad grids across the full reference to follow its development. For a long reference, read
+manageable passages while keeping their place in the whole clear. Inspect dense sequences around
+the entries, changes and exits of its distinct visual systems. Choose the range and sampling
+interval for the behavior being investigated. A brief animation needs neighboring frames; small
+writing needs a larger cell or a full-resolution frame. A broad grid can miss brief events between
+samples. Follow the developing idea and the visual systems through closer sequences to understand
+their changes and handoffs.
```bash
hypit transcribe references/ad/source.mp4 \
--to references/ad/transcript.json
-hypit media tile references/ad/source.mp4 --start 0 --end 12 \
+hypit media tile references/ad/source.mp4 --start 0 --end 12 --every 1 \
--to references/ad/evidence/opening.jpg
+hypit media tile references/ad/source.mp4 --start 6.8 --end 8.4 --every 0.1 \
+ --transcript references/ad/transcript.json --columns 4 --cell 480 \
+ --to references/ad/evidence/list-change.jpg
+
hypit media cut references/ad/source.mp4 --start 6.8 --end 8.4 --label-time \
--to references/ad/evidence/list-change.mp4
hypit media frames references/ad/source.mp4 --at 6.9,7.3,7.8 --label-time \
--to references/ad/evidence/list-frames
-
-hypit observe references/ad/source.mp4 \
- --instruction "Observe the supplied work and report evidence relevant to the question." \
- --prompt "How do the picture, Caption, MG and sound establish and pay off the hook?" \
- --to references/ad/drafts/whole-piece-observation.md
```
-`transcribe` writes word evidence used beside `TIMELINE.md`. Media commands write only the clips,
-frames or grids named by `--to`; keep the ones worth reopening under `evidence/`. An `observe` report
-is an observer's account, not the reference archive itself. Read it against the media, then write the
-connected whole-piece judgment in `ANALYSIS.md` and locatable facts in `TIMELINE.md`. A narrower
-question can instead use one clip, grid or frame as its input. The command forms are composable; the
-current uncertainty decides which of them is useful.
-
-Frames and tiles complement moving-image observation by supplying direct static evidence. If no
-moving-image observer is reachable, distinguish the evidence that remains direct from the
-relationships that are not yet well established. Frames and tiles still show appearance, text,
-composition, geometry, and sampled states; they provide less evidence about motion, transitions,
-persistence, and relationships across distant moments. Explain that limitation in terms of the
-current work, then use the
-[Runtime Profile](../environment/profile.md) to present the actual ways to strengthen the missing
-moving-image evidence. A user may knowingly choose to continue with that evidence limit, and a truly
-static question may need only static evidence, but the Agent does not present the limited account as
-a complete moving-image reading.
-
-When moving-image observation becomes reachable later, use it to revisit the parts of the reference
-that were understood from static samples alone. Update `ANALYSIS.md` and `TIMELINE.md` where the
-evidence changes them, then reconsider affected Treatment, Script, shots, components, or prompts.
-Existing work that the stronger evidence still supports remains useful. Observation may itself be a paid request;
-make its Endpoint and price source visible and use only spending authority that covers it.
-
-No evidence form is a substitute for every other one. A still cannot establish movement or sound. A
-single representative frame cannot establish entry, exit, replacement, or persistence. A long video
-pass can lose small text and rapid order. A tile shows sampled order but may miss what happens between
-cells. When motion direction or a transition is important, inspect a short clip or denser adjacent
-frames. When text or geometry is important, inspect full-resolution frames. Keep an unsupported claim
-unknown instead of completing the prose by imagination.
-
-Visual reports can be wrong even when fluently written, especially about changes over time, small UI,
-rapid cuts, and which neighboring shot owns a detail. Compare the report with the transcript and the
-actual media. Ask one narrower question over one relevant span when the answer would change the work.
-Platform watermarks and player chrome are viewing context unless the video itself demonstrates that
-the author designed them.
-
-For Caption systems, sample every meaningfully different configuration: speakers with different
-colors or positions, normal and emphasized states, different regions, or different motion. Repeated
-uses of one configuration do not need separate style investigations. A fixed quota of frames cannot
-prove that all configurations were seen.
-
-## Write two complementary reference documents
-
-`ANALYSIS.md` carries the whole-piece model. Write what kind of work it is, its story and pacing, what
-each layer contributes, which systems persist or recur, how distant moments relate, and why the piece
-works. Distinguish visible or audible facts from the Agent's reading in ordinary prose.
-
-`TIMELINE.md` carries locatable behavior. Organize it with loose second-level sections named by an
-original-media time range and a human-readable content phase:
+Time labels identify positions in the input media. Give `--transcript` the transcript of that same
+media; its word times share that clock. Word labels sit below the picture so the original Caption
+and MG remain visible. Keep useful evidence under `evidence/` and use descriptive names that make it
+easy to reopen the relevant question.
-```md
-## 6.07–8.27 · Pain-point list accelerates
+To inspect a spoken phrase, use `--around "the phrase" --transcript references/ad/transcript.json`
+in place of `--start` and `--end`. `--padding` adds surrounding time; repeated phrases can be selected
+with `--occurrence`. Use `tiles` with `--columns` and `--rows` to page a dense sequence into readable
+grids. Keep the range broad enough to see the incoming and outgoing handoffs.
+
+For Caption, inspect each meaningfully different configuration: speaker treatment, placement,
+emphasis, Cue shape and animation. Follow transitions between configurations too. Repeated uses of
+the same behavior can share its description, with their differing content and times recorded.
+
+When two readings conflict, reopen the relevant source interval and make the disputed detail legible.
+State what is visible or audible separately from what you infer it means. Player controls and other
+viewing context belong to the viewing surface; distinguish them from the designed video content.
-Speech: the list reaches “videos / voiceovers / ads / scripts”. WhisperX places each word in this
-span.
+## Preserve a connected account
-Picture: each item uses a new B-roll shot. Tile 003 shows the cuts clustered around the spoken list.
+Write the understanding into the project as it develops. Preserve both the whole-piece explanation
+and the detailed reading, with source times and useful evidence paths so someone can pick up the
+same work from the files. Include what a choice does for the viewer alongside how it appears.
+
+`ANALYSIS.md` carries the whole-piece model: what the work is trying to achieve, how its story and
+pacing work, what each visual or sound system contributes, which systems persist or recur, and how
+distant moments relate. Keep the reference's facts and your interpretation distinguishable in
+ordinary prose.
+
+`TIMELINE.md` carries time-locatable realization. Organize it into sections named by source-media
+time and a meaningful phase. Within each, connect the active layers and their detailed behavior to
+the words or actions they serve. An account should let someone find the event, understand its
+expression and implement an appropriate counterpart. For example:
+
+```md
+## 6.07–8.27 · The workload accelerates
-Caption: a marker-style word appears with each listed item; the lower spoken Caption is absent here.
+The spoken list reaches “videos / voiceovers / ads / scripts”. Each noun brings a new full-frame
+illustration and a marker-style word at its center, replacing the preceding pair. The lower spoken
+Caption gives way to these central labels so each example reads as one unit.
-Reading: the one-item/one-picture rhythm makes the workload feel excessive.
+The word and picture enter together on each noun; the labels pop to size, settle briefly, and leave
+with their picture on the next cut. The increasingly short holds make the workload feel excessive.
+The 6.8–8.4 clip and list-change grid show the handoffs; word times are in transcript.json.
```
-These sections are navigation, not mutually exclusive rows. They may overlap, leave an unresolved
-gap, or contain smaller timed notes where density warrants it. Within a section, describe every layer
-that is actually active together. A title, music bed, Caption style, or MG object may continue across
-several sections; say that it continues rather than pretending the document boundary reset it.
-
-Use open production vocabulary—A-roll, B-roll, Speech, Voice-over, Silence, Caption, Typography,
-Text Track, MG, UI, Effect, Audio—and introduce a clearer term when a work needs one. Caption is the
-spoken display system: words, Cues, speaker or role, activation, placement, and style. Typography is
-independent visual writing such as a title, label, step, or poster text. UI and MG may contain text of
-their own. Classify by function instead of collapsing everything visible into a generic text track.
-
-Name the material for an important observed fact when its provenance will help another Agent trust,
-reopen or correct it: “tile 004 shows…”, “the 0:12.4 frame shows…”, “the 0:12–0:14 clip shows…”, or
-“the WhisperX transcript gives…”. Several materials may support one conclusion. Ordinary connected
-description need not repeat the citation in every sentence. When a fact remains unsettled, name the
-question that would change the work.
-
-## Preserve time without turning it into target code
-
-Look for the editorial intention behind timing. A portrait may arrive when a player is named, an
-icon may settle when a verdict lands, and the next speaker's voice may begin while the previous
-picture remains. Explain those relationships from the actual evidence. For the target, prefer
-Selections for meaningful spans and Moments for events, then let the accepted Takes locate them.
-If the new product changes the argument, reconsider the picture or reveal that serves it as well.
-
-Reference time establishes order, overlap, duration, and relation to spoken words. Record both the
-original seconds and the meaningful relation: a picture illustrates the word `videos`, a reveal lands
-after a pause, a Caption Cue spans a phrase, or an effect continues through a cut. In the usual spoken
-work, word-level evidence lets the director identify what an event means and later bind it to Script.
-A genuinely speechless work locates meaning through action, visual change, music, effects and silence
-instead. Read `script-and-time.md` when turning those observed relationships into the target's Script
-identities and timing.
-
-## Keep the archive complete and revisable
-
-“Complete” means another Agent can recover every important element, when and how it appears, which
-system it belongs to, what it relates to, what role it plays, and why those relationships matter. It
-rests on accurate temporal and stylistic evidence without becoming a frame-by-frame transcription.
-
-Store only evidence worth reopening. Update Analysis and Timeline in place when new evidence changes
-the account. Put the current unresolved question and next useful observation in the reference's
-`PROGRESS.md`. Understanding the reference does not create the user's Brief.
+These sections can overlap and contain finer timed notes where the work is dense. A persistent
+title, board, sound bed or Caption system can span several sections; refer to the same system and
+describe what changes. Keep exact text, meaningful colors, positions, motion phases and timings where
+they are needed to understand or reproduce the design. Reopen the media for facts still in doubt.
+
+Use production vocabulary by function. A-roll establishes the semantic performance even inside a
+small inset or behind a full-frame B-roll. Caption displays the speech; Typography carries independent
+writing. MG and UI can contain text of their own. A sound can support a continuing argument across
+several picture changes. Name the roles that clarify this particular work.
+
+## Turn understanding into new direction
+
+Source time locates evidence. Record both the original seconds and the expressive relationship:
+a reveal answers a question, an image illustrates a phrase, an exit makes room for the next claim,
+or an audio handoff begins the next speaker under the previous picture.
+
+The user's request determines how those relationships should live in the new piece. A different
+person or product may change the argument, copy, number of examples, graphic content, placement and
+duration. Use [transformations](transformations.md) to think through that adaptation. Prefer
+Selections for meaningful spans and Moments for events in the target Script, and let the accepted
+performance establish their time. [Script and time](script-and-time.md) owns that authoring language.
+
+The recorded account should explain the whole piece from opening to close and make its distinct
+visual systems and their changes locatable. It should be concrete enough to direct the new work:
+what to preserve or adapt, how it is expressed, and why it belongs. Continue investigating an
+unexplained relationship where it could change that direction.
+
+Keep the current question, passages or systems still to examine, and next useful action in
+`PROGRESS.md`. Write discoveries into Analysis and Timeline while they are fresh, and revise those
+accounts when the reading changes. When resuming, read these files and reopen the source at the
+recorded locations. The user's goal remains in Brief and your new design in Treatment.
diff --git a/skills/hypit/references/creation/script-and-time.md b/skills/hypit/references/creation/script-and-time.md
index a1434934..360d4aa9 100644
--- a/skills/hypit/references/creation/script-and-time.md
+++ b/skills/hypit/references/creation/script-and-time.md
@@ -210,10 +210,11 @@ owns that gap through the explicit affinities; [B-roll craft](../playbooks/craft
shared-boundary forms.
The consuming component's Surface projects the authored identity through the real SemanticTrack.
-For example, `during={story.selection.proof}` on a visual Item makes that Surface construct the
-resulting Window, while `at={story.moment.claim}` on an MG item or sound makes its Surface construct
-the resulting Instant. The component's Fragment and Producers consume that value and own playback,
-visible duration, animation, and state behavior; Script supplies the meaning and its Anchors.
+For example, `during={story.selection.proof}` on a visual Item makes that Surface construct a Window.
+A persistent MG reveal can use `at={story.moment.claim}` to consume an Instant; an Audio Clip uses
+`at={story.moment.claim} for="600ms"` to occupy a Window. The component's Fragment and Producers
+consume that value and own playback, visible duration, animation, and state behavior; Script supplies
+the meaning and its Anchors.
Surfaces that expose Hypit's shared temporal vocabulary accept the forms appropriate to their role.
A Window occupies an interval:
@@ -251,7 +252,7 @@ serves. The target Source names the intended relation against the target Script.
actual audio is aligned, the SemanticTrack supplies its frames. Do not copy a reference timestamp into
the target or preserve an incidental lead/lag unless that offset itself is part of the design.
-## Let the creation and production loops meet honestly
+## Connect reference understanding to audiovisual composition
Timing seen in Studio or a real Result may expose one of several different problems. A wrong Cue or
semantic anchor changes Script. A sound performance that changes the intended rhythm may change the
diff --git a/skills/hypit/references/creation/transformations.md b/skills/hypit/references/creation/transformations.md
index 1fdc1a7c..f47ce313 100644
--- a/skills/hypit/references/creation/transformations.md
+++ b/skills/hypit/references/creation/transformations.md
@@ -114,6 +114,7 @@ Otherwise, avoid chains of intermediate pictures that exist only to feed the nex
single shot split by a model's duration limit, reuse the same accepted visual authorities for each
part unless the cut itself is designed around a literal boundary frame.
-Studio and real Results close the decision loop. Adopt a generated item by using it in a later Run,
-Source, or request. If it fails the intended role, change the prompt, reference relation, shot design,
-or Treatment according to what is actually wrong.
+Carry accepted media into the composition through Source and Run Candidates. Use Studio and rendered
+Results to develop Caption, MG, B-roll coverage, effects and sound around those assets. If a generated
+item fails its intended role, improve the responsible prompt, reference relation or shot design before
+deciding on another authorized request. Reconsider Treatment when the creative answer itself changes.
diff --git a/skills/hypit/references/playbooks/craft/sound-mix.md b/skills/hypit/references/playbooks/craft/sound-mix.md
index 83ec3368..5d8eb973 100644
--- a/skills/hypit/references/playbooks/craft/sound-mix.md
+++ b/skills/hypit/references/playbooks/craft/sound-mix.md
@@ -13,8 +13,10 @@ Treat the soundtrack as a composition of independently authored contributions:
- **ambience** gives places and edits a believable acoustic world;
- **sound effects** answer discrete visible or editorial events.
-The Treatment and reference Timeline can describe what each layer contributes and where its role
-changes. The Source expresses the chosen relationships with ordinary audio Resources and Tracks.
+Choose sounds for what the new piece is doing: a reveal can earn a short accent, a comparison can
+share a recognizable motif, and a transition can carry the music into the next idea. Let the video's
+character guide the sound palette. The Treatment describes these choices; the Source expresses them
+with ordinary audio Resources and Tracks.
An audio library, generated sound, recorded material, or an earlier accepted Output can all supply a
layer; choose from the work's need rather than treating origin as an audible category.
@@ -65,17 +67,6 @@ can bind a montage, and a speaking Take can remain audible while B-roll or MG ch
Conversely, a change of place, speaker perspective, narrative state, or designed silence may deserve
an audible transition even when the picture changes gently.
-Listen at boundaries for clipped words, missing breaths, doubled speech, abrupt room changes, music
-that restarts accidentally, and effect tails cut by the next shot. Fades and overlaps should express
-the intended handoff rather than hide an unexplained discontinuity.
-
-## Review the complete listening experience
-
-First establish that each important source is the right performance or sound. Then hear speech with
-the surrounding layers, the densest passage, every structural transition, and the opening and close.
-Finally listen to the complete Film without stopping. Use the available listening setup that represents
-the delivery, and compare headphones with ordinary speakers when translation between them matters.
-
-Judge cumulative pace and fatigue as well as isolated clarity. Repair the owning layer: performance
-problems in the speech source, semantic placement in Source, event design in the effect, arrangement
-and balance in the mix, and clipping or encoding in the relevant media or final output.
+Give speech its complete phrase and let effect tails finish naturally. Use fades and overlaps to
+carry the intended handoff. Keep the mix coherent across the piece: clear speech, a consistent sound
+palette, and emphasis reserved for the moments that deserve it.
diff --git a/skills/hypit/references/playbooks/craft/voice-and-performance.md b/skills/hypit/references/playbooks/craft/voice-and-performance.md
index b73004f2..36e717bd 100644
--- a/skills/hypit/references/playbooks/craft/voice-and-performance.md
+++ b/skills/hypit/references/playbooks/craft/voice-and-performance.md
@@ -188,12 +188,11 @@ accepted performance has semantic time, its dependent layers follow the delivery
happened. Repair a weak performance at its direction, wording, reference, or generation according to
the visible and audible problem.
-## Review the person, not just the waveform
+## Carry the character through the performance
-Judge whether the voice and visible performance express the same character: pronunciation, pace,
-cadence, emotion, mouth movement, gesture, gaze, and interaction with the scene. Across recurring
-Takes, listen for identity drift and room discontinuity while allowing intentional changes in energy
-or delivery.
+Direct voice and visible performance toward the same character: pace, emotion, gesture, gaze and
+interaction with the scene. Carry that identity across recurring Takes while allowing the energy
+and delivery to develop with the conversation.
Review covered passages with their picture and with the underlying speech source made clear. A clean
mix cannot rescue the wrong performer, and a good voice sample cannot rescue an unconvincing Take.
diff --git a/skills/hypit/references/playbooks/formats/narration-led-demo.md b/skills/hypit/references/playbooks/formats/narration-led-demo.md
index 49b1c4fd..93304c05 100644
--- a/skills/hypit/references/playbooks/formats/narration-led-demo.md
+++ b/skills/hypit/references/playbooks/formats/narration-led-demo.md
@@ -78,9 +78,7 @@ their own Typography, Text, screen, or MG roles unless they are also meant to be
Watch the passage first as an argument: does every visual change clarify, prove, pace, or intensify
what is being said? Then inspect exact product and screen truth, hand and device geometry, Caption
-timing, persistent component states, visual handoffs, and audio continuity.
+timing, persistent component states and visual handoffs.
-Listen once without the picture to judge the speech performance, then watch without relying on the
-words to judge whether the visual sequence has its own intelligible progression. Finally review the
-complete Film. Repair weak narration at its performance source, weak proof in the shot or component
-design, wrong timing at the authored semantic relation, and weak balance in the mix.
+Let the visual sequence develop a clear progression alongside the narration. Repair weak proof in
+the shot or component design and wrong timing at the authored semantic relation.
diff --git a/skills/hypit/references/production/review.md b/skills/hypit/references/production/review.md
index 5011eb2e..7ebb439f 100644
--- a/skills/hypit/references/production/review.md
+++ b/skills/hypit/references/production/review.md
@@ -1,11 +1,24 @@
# Reviewing the production
-Read this when opening Studio, judging an encoded Result, comparing a reconstruction with its
-reference, or deciding where a visible problem should be repaired.
+Read this when opening Studio, judging an encoded Result, or improving the layout, timing and
+expression of the new Film.
[Studio](studio.md) explains launch, session reuse, displayed information, Source writeback and
project Companions. [Builds and Results](builds.md) explains retrieval and explicit Output reuse.
+## Make the composition work
+
+Compose MG, Caption, B-roll, Typography and Effects around the directed material and existing Outputs.
+Watch their actual arrangement in Studio or a rendered Result: is the comparison clear, is the
+Caption readable, does the B-roll cover the intended explanation, and do entries and exits land on
+the right words or actions? Adjust the layout, timing or component behavior that will make the
+passage work better. Judge a local change in the surrounding composition.
+
+Use Craft to give generated pictures and performances strong reference and prompt direction before
+requesting them. Composition usually develops around those produced assets. When a generated item
+fails its intended role, identify the concrete defect and improve the responsible direction before
+deciding on another authorized request. That decision has its own purpose and cost.
+
## Review the work the Run actually selects
Open Studio with the ordinary Run intended for the work. That Run binds the Author Source, Targets,
@@ -34,7 +47,7 @@ use does not create an approval state that the work must preserve.
`hypit check` can establish that the Source and graph are legal. Studio can establish what the
configured composition displays. A completed Build can establish what the selected Endpoints actually
-produced. Creative review asks a different question: does this visible and audible work perform the
+produced. Creative review asks a different question: does this composition perform the
Treatment and the useful relationships learned from the reference?
Look at the complete Film at its intended delivery size as well as the components inside it. A component that looks
@@ -50,7 +63,7 @@ location remains an open question.
Use whichever view can answer the current question:
-- An Agent with browser interaction can open Studio, seek and play the ordinary Run, look at the
+- Open Studio, seek and play the ordinary Run, look at the
current frame, and discover where a layout or motion deserves closer inspection. This is usually
the cheapest way to work on Caption, MG, Effects, and composition; a Card can supply the minimal
visible unknown when no more representative media exists yet.
@@ -61,25 +74,24 @@ Use whichever view can answer the current question:
relationship to upstream Candidate reuse.
- On an encoded Result, focused media operations such as frames, cut, and tile can expose exact
pixels, adjacent frames, or a short passage.
-- Watch the whole deliverable when the question concerns Hook clarity, story movement, pace, payoff,
- CTA, performance, continuity, or whether A-roll, B-roll, Caption, MG, Effects, and Audio cooperate.
+- Watch the whole deliverable when the question concerns Hook clarity, story movement, payoff,
+ CTA, or how A-roll, B-roll, Caption, MG and Effects work together.
Choose and combine these views according to the current question. A component name alone does not
prove that a particular state is stable or that a change occurs at a guessed time; Studio or actual
media supplies that evidence.
-Inspect the visible and audible relationships that could change the work:
+Inspect the relationships that make the composition work:
-- **picture and performance** — identity, camera relationship, action, eye line, continuity, useful
- detail, and whether the shot serves its passage;
-- **semantic timing** — cuts, Caption Cues, MG states, Effects, and sounds occur on the intended word,
- phrase, pause, or clock event;
+- **picture and coverage** — B-roll supports the passage and its display window carries the intended
+ explanation or handoff;
+- **semantic timing and motion** — cuts, Caption Cues, MG states and Effects occur on the
+ intended word, phrase, pause, or clock event; entry, settling, active behavior and exit make the
+ intended emphasis and handoff perceptible;
- **Caption** — every meaningfully different speaker, position, color, emphasis, Cue shape, and motion
configuration remains readable, belongs to the speech, and preserves the intended face and action;
- **Typography and UI** — independent writing has the correct hierarchy, content, persistence, and
relationship to the picture;
-- **audio** — speech is intelligible, speaker and voice are right, music and effects support the beat,
- and transitions sound continuous;
- **composition** — each element has enough room and the full frame preserves the intended visual
hierarchy.
@@ -107,9 +119,11 @@ For a supplied person or product, compare the new appearances with those actual
as the intended camera image. Judge recognizable identity, appeal, and how naturally the new subject
inhabits the piece; check that dialogue, demonstrations, and graphic details make sense for this target.
-Fresh visual attention is useful when familiarity hides an obvious problem. Another observer can be
-asked a neutral, bounded question about the actual media, while the commissioned Agent remains
-responsible for combining that evidence with the Brief, Treatment, and graph.
+Inspect the detailed realization of each relationship: the entry catches attention, the hold gives
+enough reading time, the motion explains a change, and the exit clears the next idea. Compare source
+and target at their corresponding words or actions; the target's new speech can place that event at
+a different second. A reference gives precise design evidence, while Brief and Treatment determine
+which of those choices belong in this piece.
## Repair the owning fact
@@ -121,9 +135,15 @@ responsible for combining that evidence with the Brief, Treatment, and graph.
| The wrong file, earlier Output, or stand-in is selected | Run Source |
| A reusable visual role cannot express or render its intended design | the project Author Package |
-Use the smallest correction that changes the failed fact and preserve existing work around it. Review
-again when that correction produces new visible or audible evidence. A request for different paid
-media is a new production decision whose additional calls must be visible in `hypit plan`; it is not
-an automatic consequence of routine review. Build completion is operational success; the production
-closes when the Agent has watched the actual deliverable and can explain why it satisfies the Brief
-and Treatment.
+Before changing the work, connect the composition problem to its consequence and the intended
+improvement. For instance, extending a hold can make a comparison readable; delaying an exit can keep
+proof visible through the claim it supports. Correct the owning fact and preserve existing work
+around it. Watch the affected interval and its handoffs after the change, then judge it in the whole
+Film. If the result did not improve, revisit the explanation before making another adjustment.
+
+A request for different paid media is a production decision whose additional calls must be visible
+in `hypit plan`. Reuse the accepted Outputs for composition changes.
+
+The composition is ready when its layout, readability and timing carry the Brief and Treatment
+clearly and compellingly. Continue when a change will improve that expression; deliver when it works,
+with the important choices and any material limitations explained.