Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

feat(runtime): execute creation-time Needs through the selected Profile

kashorin5793277bf26a
diff --git a/packages/cli/test/build-programs.test.ts b/packages/cli/test/build-programs.test.ts
index 2ffc55b9..91c1f87c 100644
--- a/packages/cli/test/build-programs.test.ts
+++ b/packages/cli/test/build-programs.test.ts
@@ -124,6 +124,7 @@ function distribution(
preflight: async () => ({ dataRoot: "/tmp", diagnostics }),
doctor: async () => ({ dataRoot: "/tmp", diagnostics: [] }),
providers: async () => [],
+ invoke: async () => { throw new Error("creation-time invocation is not part of this test"); },
createRuntime: async () => {
if (execution !== undefined) return execution;
throw new Error("createRuntime is unavailable");
diff --git a/packages/reference-video-tools/README.md b/packages/reference-video-tools/README.md
index 6df15e17..5b334f68 100644
--- a/packages/reference-video-tools/README.md
+++ b/packages/reference-video-tools/README.md
@@ -55,8 +55,7 @@ It also produces `transcript`: the verbatim speech of the whole reference with a
every single word. Placing an on-screen text reveal against the line that triggers it needs the time
of the word, not of the sentence around it, and that question comes up in every reconstruction. The
speech audio is extracted to `speech.wav` beside the shot media and measured through the
-`@hypit/whisperx#whisperx-alignment` capability. This tool currently instantiates the local WhisperX
-Provider directly; routing it through the selected Runtime Profile is pending.
+`@hypit/whisperx#whisperx-alignment` Endpoint of the selected Runtime Profile.
The result reports `status`, `transcript_ref` and `word_count`, and the words themselves live in
`transcript.json` as passages, each with a `words` array of `{ text, start_seconds, end_seconds, score }`.
The transcript is deterministic local evidence rather than an observation: it is never written to the
@@ -85,12 +84,12 @@ depend on a separate state database.
## Environment
-Reference observation uses `HYPIT_GEMINI_PROVIDER=auto|hypihub|vertex`. HypiHub first reads the
-`hypihub.oauth` session written by `hypit auth login` from the OS credential store, then falls back to
-`HYPIHUB_API_KEY`; `HYPIHUB_BASE_URL` remains optional. Vertex reads `GOOGLE_CLOUD_PROJECT`,
-`GOOGLE_APPLICATION_CREDENTIALS_JSON` and optional `GOOGLE_CLOUD_LOCATION`. Gemini defaults to
-`gemini-3.1-pro`.
-
-Transcript preparation uses the selected local WhisperX service. Start services through the normal
-Runtime configuration with `hypit runtime up`. `HYPIT_REFERENCE_CONCURRENCY` and
-`HYPIT_REFERENCE_LAUNCH_GAP_MS` only control observation request pacing.
+Observation and transcription run through the selected Runtime Profile, the same way a Build does:
+`--runtime <profile>` names it, otherwise the project's `.hypit/runtime` selection is read. The
+Profile's Gemini Endpoint (HypiHub, Vertex or another Provider) answers `--observer gemini`, and its
+`@hypit/whisperx#whisperx-alignment` Endpoint measures the transcript. This tool never chooses a
+Provider, reads a credential store or contacts a service itself; `hypit plan --runtime <profile>` shows
+which Endpoint serves each capability and where it publishes prices. The Gemini model defaults to the
+first model `@hypit/gemini` declares; `GEMINI_MODEL` selects another declared model.
+`HYPIT_REFERENCE_CONCURRENCY` and `HYPIT_REFERENCE_LAUNCH_GAP_MS` only control observation request
+pacing.
diff --git a/packages/reference-video-tools/package.json b/packages/reference-video-tools/package.json
index 5d375070..d1f739b5 100644
--- a/packages/reference-video-tools/package.json
+++ b/packages/reference-video-tools/package.json
@@ -13,7 +13,6 @@
},
"dependencies": {
"@hypit/composition": "workspace:*",
- "@hypit/credential-store-os": "workspace:*",
"@hypit/core": "workspace:*",
"@hypit/driver-node": "workspace:*",
"@hypit/estimate": "workspace:*",
@@ -29,14 +28,12 @@
"@hypit/run-markup": "workspace:*",
"@hypit/runtime": "workspace:*",
"@hypit/provider-media-local": "workspace:*",
- "@hypit/provider-hypihub": "workspace:*",
- "@hypit/provider-vertex": "workspace:*",
- "@hypit/provider-whisperx-local": "workspace:*",
"@hypit/script": "workspace:*",
"@hypit/speech": "workspace:*",
"@hypit/speech-evidence": "workspace:*",
"@hypit/studio": "workspace:*",
"@hypit/svs": "workspace:*",
+ "@hypit/text": "workspace:*",
"@hypit/video-cli": "workspace:*",
"@hypit/visual-ir": "workspace:*",
"@hypit/whisperx": "workspace:*",
diff --git a/packages/reference-video-tools/src/cli.ts b/packages/reference-video-tools/src/cli.ts
index 3b40bb8c..190b4a72 100644
--- a/packages/reference-video-tools/src/cli.ts
+++ b/packages/reference-video-tools/src/cli.ts
@@ -26,7 +26,7 @@ function usage(): string {
return [
"Usage:",
" hypit-reference-video-tools list_svml_packages",
- " hypit-reference-video-tools prepare_reference --video-path <path or link> [--observer gemini|agent] [--redo media|transcript|people|voices|systems|places|all]",
+ " hypit-reference-video-tools prepare_reference --video-path <path or link> [--observer gemini|agent] [--runtime <profile>] [--redo media|transcript|people|voices|systems|places|all]",
" hypit-reference-video-tools observe_reference --reference-id <id> [--shot-id <id> ...] [--reobserve]",
" hypit-reference-video-tools observe_reference --reference-id <id> --shot-id <id> --question <text>",
" hypit-reference-video-tools observe_reference --reference-id <id> --batch <questions.json>",
@@ -158,9 +158,10 @@ function usage(): string {
"same bytes and therefore the same reference, and reported back as `source_url`. Everything after",
"the download reads the file without knowing it was ever a link.",
"",
- "--observer picks who reads the reference, once per reference. `gemini` uploads video to the configured Gemini backend.",
- "With HYPIT_GEMINI_PROVIDER=auto it uses HYPIHUB_API_KEY when present, otherwise GOOGLE_CLOUD_PROJECT and GOOGLE_APPLICATION_CREDENTIALS_JSON for Vertex.",
- "If no usable credential is available, sign in to HypiHub with `hypit auth login` at https://hypit.ai or use `agent`. `agent` needs no credentials: it",
+ "--observer picks who reads the reference, once per reference. `gemini` uploads video to the Gemini Endpoint",
+ "the selected Runtime Profile binds (--runtime <profile>, else the project's `hypit runtime use` selection);",
+ "the Profile's whisperx-alignment Endpoint measures the transcript the same way. If that Endpoint's credential",
+ "is missing, sign in with `hypit auth login` or use `agent`. `agent` needs no credentials: it",
"returns each observation as a task carrying its prompt and one tiled picture per shot, which the",
"calling agent answers with record_observation. A task whose question needs sound also carries the",
"words WhisperX measured — `transcript_words` for its own stretch, `transcript_ref` for the whole",
@@ -278,9 +279,10 @@ async function main(): Promise<void> {
const input = supplied ?? {
video_path: required(flags, "video-path"),
...(observer === undefined ? {} : { observer }),
+ ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
...(one(flags, "redo") === undefined ? {} : { redo: one(flags, "redo") }),
};
- result = await tools.prepare_reference(input as { video_path: string; observer?: "gemini" | "agent"; redo?: "media" | "transcript" | "people" | "voices" | "systems" | "places" | "all" });
+ result = await tools.prepare_reference(input as { video_path: string; observer?: "gemini" | "agent"; runtime?: string; redo?: "media" | "transcript" | "people" | "voices" | "systems" | "places" | "all" });
} else if (command === "observe_reference") {
// A round of narrow questions is a list, and a list is too long for flags.
const askFile = one(flags, "batch");
@@ -288,8 +290,10 @@ async function main(): Promise<void> {
const parsed: unknown = JSON.parse(await readFile(askFile, "utf8"));
const list = Array.isArray(parsed) ? parsed : (parsed as { questions?: unknown }).questions;
if (!Array.isArray(list)) throw new Error(`${askFile} must hold a JSON array of {shot_ids, question}, or an object with a "questions" array`);
+ const runtime = one(flags, "runtime");
result = await tools.observe_reference({
reference_id: required(flags, "reference-id"),
+ ...(runtime === undefined ? {} : { runtime }),
questions: list as ObserveReferenceInput["questions"] & object,
});
report(`${JSON.stringify(result, null, 2)}\n`);
@@ -299,9 +303,10 @@ async function main(): Promise<void> {
reference_id: required(flags, "reference-id"),
...(many(flags, "shot-id").length === 0 ? {} : { shot_ids: many(flags, "shot-id") }),
...(one(flags, "question") === undefined ? {} : { question: one(flags, "question") }),
+ ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
...(flags.get("reobserve") === true ? { reobserve: true } : {}),
};
- result = await tools.observe_reference(input as { reference_id: string; shot_ids?: readonly string[]; question?: string; reobserve?: boolean });
+ result = await tools.observe_reference(input as { reference_id: string; shot_ids?: readonly string[]; question?: string; runtime?: string; reobserve?: boolean });
} else if (command === "record_observation") {
// A whole sweep's answers at once. Each entry carries its own text, either inline or as a file the
// answer was written to, which is read here the way --text-file is.
@@ -371,8 +376,10 @@ async function main(): Promise<void> {
const parsed: unknown = JSON.parse(await readFile(batchFile, "utf8"));
const list = Array.isArray(parsed) ? parsed : (parsed as { comparisons?: unknown }).comparisons;
if (!Array.isArray(list)) throw new Error(`${batchFile} must hold a JSON array of comparisons, or an object with a "comparisons" array`);
+ const runtime = one(flags, "runtime");
result = await tools.compare_reconstruction({
reference_id: required(flags, "reference-id"),
+ ...(runtime === undefined ? {} : { runtime }),
comparisons: list as CompareReconstructionInput["comparisons"] & object,
});
report(`${JSON.stringify(result, null, 2)}\n`);
@@ -385,6 +392,7 @@ async function main(): Promise<void> {
const range = segment !== undefined || selection !== undefined || tokens !== undefined;
const input = supplied ?? {
reference_id: required(flags, "reference-id"),
+ ...(one(flags, "runtime") === undefined ? {} : { runtime: one(flags, "runtime") }),
...(range
? {
run: required(flags, "run"),
diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.ts
index 25986804..02af47ac 100644
--- a/packages/reference-video-tools/src/tools.ts
+++ b/packages/reference-video-tools/src/tools.ts
@@ -4,12 +4,10 @@ import { cpus } from "node:os";
import { deflateSync } from "node:zlib";
import { basename, dirname, join, resolve } from "node:path";
import { loadNodePackageSelection, locateNodePackage, physicalPackageName } from "@hypit/package-loader-node";
-import { OsCredentialStore } from "@hypit/credential-store-os";
-import { credentialRef } from "@hypit/runtime";
import { markupSurfaceHostFacetAbi } from "@hypit/markup";
import type { RegisteredSurface, SurfaceVocabulary } from "@hypit/markup";
import { exactModelHostAbi } from "@hypit/model-kit";
-import type { ValueSchema } from "@hypit/protocol";
+import type { CanonicalValue, Need, ValueSchema } from "@hypit/protocol";
import {
visualPathCommandSchema, visualTextDocumentSchema, visualTextFlowSchema,
visualTextPaintSchema, visualTextTypographySchema, visualTrackSchema,
@@ -20,10 +18,12 @@ import {
import { VISUAL_STYLE_ENUM_VALUES_V1, VISUAL_STYLE_NAMES_V1 } from "@hypit/visual-ir";
import { describeSchema } from "./schema.js";
-import { videoCliDistribution } from "@hypit/video-cli";
-import { createHypiHubGeminiGenerator } from "@hypit/provider-hypihub";
-import { createVertexGeminiGenerator } from "@hypit/provider-vertex";
-import type { GeminiInlinePart } from "@hypit/gemini";
+import { findRuntimeProfile, videoCliDistribution } from "@hypit/video-cli";
+import { MemoryResourceStore } from "@hypit/driver-node";
+import { geminiCapabilities, geminiModels, sealGeminiRequest } from "@hypit/gemini";
+import type { GeminiInlinePart, GeminiMediaPart, GeminiModel } from "@hypit/gemini";
+import { textTypes } from "@hypit/text";
+import { whisperXCapabilities } from "@hypit/whisperx";
import { authorSource, invokedFrom, nearestPackageRoot, referenceRoot, referenceWords, renderElement, renderPreviews, spokenRange, standInSidecarPath, tokenWindow } from "./authoring.js";
import type { RenderElementInput, RenderPreviewsInput, SpokenRange, StandInFocus, StandInSidecar } from "./authoring.js";
@@ -52,19 +52,24 @@ import {
tileFrames,
writeJson,
} from "./media.js";
-import { prepareTranscript, whisperxHealth } from "./transcript.js";
+import { prepareTranscript } from "./transcript.js";
+import type { InvokeNeed } from "./transcript.js";
import type { Observation, ObservationTaskRequest, Observer, PrepareResult, ReferenceState, Shot, Transcript } from "./types.js";
export type PrepareReferenceInput = {
readonly video_path: string;
readonly redo?: "media" | "transcript" | "people" | "voices" | "systems" | "places" | "all";
readonly observer?: Observer;
+ /** Runtime Profile whose Endpoints observe and transcribe; defaults to the project's selection. */
+ readonly runtime?: string;
};
export type ObserveReferenceInput = {
readonly reference_id: string;
readonly shot_ids?: readonly string[];
readonly question?: string;
readonly reobserve?: boolean;
+ /** Runtime Profile whose Gemini Endpoint answers; defaults to the project's selection. */
+ readonly runtime?: string;
/**
* A round of narrow questions, asked together. Each entry names its own shots and its own question
* and inherits `reference_id`; each `shot_ids` array must contain exactly one shot. No narrow
@@ -102,6 +107,8 @@ export type ValidateLocalAuthorPackagesInput = {
};
export type CompareReconstructionInput = {
readonly reference_id: string;
+ /** Runtime Profile whose Gemini Endpoint compares; defaults to the project's selection. */
+ readonly runtime?: string;
/**
* A stretch of the reference to compare against, named one of two ways. Exactly one is given.
*
@@ -435,65 +442,83 @@ function positiveInt(value: number, label: string): number {
return value;
}
-async function defaultGenerate(model: string): Promise<GenerateText> {
- const hypiHubKey = await resolveHypiHubApiKey();
- const backend = process.env.HYPIT_GEMINI_PROVIDER?.trim().toLowerCase() || "auto";
- if (backend !== "auto" && backend !== "hypihub" && backend !== "vertex") {
- throw new Error("HYPIT_GEMINI_PROVIDER must be auto, hypihub or vertex");
- }
- if (backend === "hypihub" || (backend === "auto" && hypiHubKey)) {
- if (!hypiHubKey) throw new Error("HYPIT_GEMINI_PROVIDER=hypihub requires HypiHub OAuth; run hypit auth login hypihub.default");
- const generate = createHypiHubGeminiGenerator({
- apiKey: hypiHubKey,
- model,
- baseUrl: process.env.HYPIHUB_BASE_URL?.trim() || "https://hypit.ai",
- });
- return async ({ parts, instruction }) => {
- try {
- return await generate({ parts: parts as unknown as Parameters<typeof generate>[0]["parts"], instruction });
- } catch (error) {
- const message = error instanceof Error ? error.message : String(error);
- if (!/hypit\.ai/iu.test(message) && /HTTP (401|403|404)\b|model_not_found|no_capable_provider/iu.test(message)) {
- throw new Error(`${message}. Sign in at https://hypit.ai with hypit auth login to enable this Gemini model`);
- }
- throw error;
- }
- };
- }
- const project = process.env.GOOGLE_CLOUD_PROJECT?.trim();
- const credentials = process.env.GOOGLE_APPLICATION_CREDENTIALS_JSON?.trim();
- if (!project || !credentials) {
- throw new Error("Gemini credentials are unavailable. Configure Vertex credentials or sign in to HypiHub with hypit auth login.");
- }
- const generate = createVertexGeminiGenerator({
- project,
- credentials,
- model,
- location: process.env.GOOGLE_CLOUD_LOCATION?.trim() || "global",
+type RuntimeHost = Awaited<ReturnType<typeof videoCliDistribution.openRuntimeHost>>;
+
+/**
+ * The Runtime Profile a creation-time call executes through: the one the caller named, otherwise the
+ * project's own selection, read the way `hypit` reads it. Observation and transcription never pick a
+ * Provider, read a credential store or contact a service themselves.
+ */
+async function runtimeProfileFor(runtime: string | undefined): Promise<string> {
+ if (runtime !== undefined) return resolve(invokedFrom(), runtime);
+ const projectRoot = nearestPackageRoot(invokedFrom()) ?? invokedFrom();
+ const selected = await findRuntimeProfile(projectRoot);
+ assert(selected !== undefined,
+ `No Runtime Profile is selected for ${projectRoot}; run hypit runtime use <profile> there, or pass --runtime <profile>`);
+ return selected.profile;
+}
+
+async function runtimeHostFor(runtime: string | undefined): Promise<RuntimeHost> {
+ const profile = await runtimeProfileFor(runtime);
+ const projectRoot = nearestPackageRoot(dirname(profile)) ?? dirname(profile);
+ return await videoCliDistribution.openRuntimeHost(profile, {
+ packageRoot: projectRoot,
+ ...(videoCliDistribution.packageRoot === undefined ? {} : { distributionPackageRoot: videoCliDistribution.packageRoot }),
});
+}
+
+function geminiModel(model: string): GeminiModel {
+ assert((geminiModels as readonly string[]).includes(model),
+ `Gemini model ${model} is not declared by @hypit/gemini; declared models: ${geminiModels.join(", ")}`);
+ return model as GeminiModel;
+}
+
+/**
+ * Observation through the selected Runtime Profile: the same Need, Endpoint, credential and Provider
+ * resolution a Build uses, without a Build. The inline parts this file assembles become one Gemini
+ * request whose media live in a per-call resource store; nothing reaches Build state or a Result.
+ */
+function runtimeGenerate(host: RuntimeHost, model: GeminiModel): GenerateText {
return async ({ parts, instruction }) => {
- try {
- return await generate({ parts, instruction });
- } catch (error) {
- const message = error instanceof Error ? error.message : String(error);
- if (!/hypit\.ai/iu.test(message) && /\b(?:401|403|404)\b|permission denied|unauthenticated|not found|failed precondition/iu.test(message)) {
- throw new Error(`${message}. This Gemini model or Vertex credential is unavailable; sign in at https://hypit.ai with hypit auth login`);
- }
- throw error;
+ const resources = new MemoryResourceStore();
+ const prompt: string[] = [];
+ const media: GeminiMediaPart[] = [];
+ for (const part of parts) {
+ if (part.text !== undefined) { prompt.push(part.text); continue; }
+ media.push({ artifact: await resources.put(Buffer.from(part.inlineData.data, "base64"), part.inlineData.mimeType) });
}
+ const need: Need = {
+ id: "need:reference-video-observation",
+ capability: geminiCapabilities[model],
+ returns: textTypes.text,
+ constraints: sealGeminiRequest({ instruction, prompt: prompt.join("\n\n"), media }) as unknown as CanonicalValue,
+ result: "record:reference-video-observation",
+ };
+ const fulfillment = await host.invoke(need, resources);
+ assert(fulfillment.value.kind === "inline", "Gemini returned its observation by reference");
+ const text = (fulfillment.value.value as { readonly value?: unknown } | null)?.value;
+ assert(typeof text === "string", "Gemini observation is not Text");
+ return text;
};
}
-/** Resolve the HypiHub credential written by `hypit auth login`, with env fallback. */
-async function resolveHypiHubApiKey(): Promise<string | undefined> {
- if (process.platform === "darwin" || process.platform === "win32") {
- const value = await new OsCredentialStore()
- .resolve(credentialRef("os", "hypihub.oauth"))
- .catch(() => undefined);
- const secret = value?.secret?.trim();
- if (secret) return secret;
+/**
+ * How a transcript is measured: the `whisperx-alignment` Endpoint of the selected Runtime Profile.
+ * Preparation reports a missing selection or Endpoint beside everything else it prepared instead of
+ * refusing; the transcript stays recoverable with `--redo transcript` once the Profile serves it.
+ */
+async function alignmentThrough(runtime: string | undefined): Promise<{ readonly invoke: InvokeNeed; readonly note?: string }> {
+ const refusing = (note: string) => ({ invoke: (async () => { throw new Error(note); }) as InvokeNeed, note });
+ try {
+ const host = await runtimeHostFor(runtime);
+ const [provider] = await host.providers([whisperXCapabilities.alignment]);
+ if (provider?.status === "resolved") return { invoke: async (need, resources) => await host.invoke(need, resources) };
+ return refusing(provider?.status === "ambiguous"
+ ? `several Endpoints serve @hypit/whisperx#whisperx-alignment: ${(provider.endpoints ?? []).join(", ")}; keep exactly one in the Runtime Profile`
+ : "no Endpoint in the selected Runtime Profile serves @hypit/whisperx#whisperx-alignment; hypit plan --runtime <profile> shows what each capability needs");
+ } catch (error) {
+ return refusing(error instanceof Error ? error.message : String(error));
}
- return process.env.HYPIHUB_API_KEY?.trim() || undefined;
}
const MIME_TYPES: Readonly<Record<string, string>> = {
@@ -548,6 +573,8 @@ async function callSafely(retryDelayMs: number, generate: GenerateText, parts: r
if (permanent(error)) break;
if (attempt === 6) break;
const wait = rateLimited(error) ? Math.min(900_000, 45_000 * 2 ** (attempt - 1)) : retryDelayMs * attempt;
+ // The retry is visible, not hidden: each failed attempt and the wait before the next one is reported.
+ process.stderr.write(`observation attempt ${attempt}/6 failed: ${message(error)}; retrying in ${Math.round(wait / 1000)}s\n`);
await new Promise((resolveWait) => setTimeout(resolveWait, wait));
}
}
@@ -1037,7 +1064,9 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
...(projectPackageDirectories.length === 0 && expected.size === 0 ? { note: "no project-owned packages declared" } : {}),
};
}
- const model = options.model ?? process.env.GEMINI_MODEL?.trim() ?? "gemini-3.1-pro";
+ // The model is the author's choice among those @hypit/gemini declares; which Provider answers is the
+ // Runtime Profile's.
+ const model = options.model ?? process.env.GEMINI_MODEL?.trim() ?? geminiModels[0];
// Pacing is deployment policy, not author intent: it depends on the quota behind the credentials,
// which the calling agent has no way to know. It is settable here and through the environment, and
// deliberately not through a CLI flag.
@@ -1047,8 +1076,19 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
const concurrency = options.concurrency ?? positiveEnv("HYPIT_REFERENCE_CONCURRENCY") ?? 4;
const gapMs = options.launchGapMs ?? positiveEnv("HYPIT_REFERENCE_LAUNCH_GAP_MS") ?? 1_500;
const retryDelayMs = options.retryDelayMs ?? 2_000;
- let generatorPromise: Promise<GenerateText> | undefined;
- const generator = async (): Promise<GenerateText> => generatorPromise ??= options.generate === undefined ? defaultGenerate(model) : Promise.resolve(options.generate);
+ // One observer per Runtime Profile: an injected generator serves every call, otherwise the Profile's
+ // Gemini Endpoint is opened once per Profile path and reused.
+ const generators = new Map<string, Promise<GenerateText>>();
+ const generator = async (runtime: string | undefined): Promise<GenerateText> => {
+ if (options.generate !== undefined) return options.generate;
+ const profile = await runtimeProfileFor(runtime);
+ let opened = generators.get(profile);
+ if (opened === undefined) {
+ opened = runtimeHostFor(profile).then((host) => runtimeGenerate(host, geminiModel(model)));
+ generators.set(profile, opened);
+ }
+ return await opened;
+ };
const loadState = async (reference: string): Promise<ReferenceState> => {
const state = await readJson<ReferenceState>(join(stateRoot(reference), "state.json"));
@@ -1076,6 +1116,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
observer: Observer,
state: ReferenceState,
comparisonRun?: string,
+ runtime?: string,
): Promise<{ readonly ask: Asker; readonly pending: readonly ObservationTaskRequest[]; readonly paced?: boolean }> => {
if (observer === "agent") {
const pending: ObservationTaskRequest[] = [];
@@ -1115,7 +1156,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
},
};
}
- const generate = await generator();
+ const generate = await generator(runtime);
const mediaPart = mediaParts();
return {
pending: [],
@@ -1271,11 +1312,11 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
// everything except the transcript, which stays recoverable — and points at the repair. A
// reference whose transcript is already complete and unchanged needs no probe.
const needsTranscript = !(state.transcript?.status === "complete" && !redoTranscript);
- const whisperx = needsTranscript ? await whisperxHealth() : undefined;
+ const alignment = needsTranscript ? await alignmentThrough(input.runtime) : undefined;
const transcribing = state.transcript?.status === "complete" && !redoTranscript
? Promise.resolve(state.transcript)
- : prepareTranscript(reference, videoPath, root, info.hasAudio, "en", redoTranscript);
- const { ask, pending } = await askerFor(observer, state);
+ : prepareTranscript(reference, videoPath, root, info.hasAudio, "en", redoTranscript, alignment!.invoke);
+ const { ask, pending } = await askerFor(observer, state, undefined, input.runtime);
const whole = [analysisVideo];
const [people, voices, systems, places] = await Promise.all([
state.people_and_product?.status === "complete" && !redoPeople
@@ -1320,9 +1361,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
observer,
...(fetched === undefined ? {} : { source_url: fetched.url, downloaded: !fetched.cached }),
pending_observations: pending,
- ...(whisperx?.ok === false
- ? { whisperx: `${whisperx.reason}. Read <skill-root>/references/host-setup.md for the failure branches.` }
- : {}),
+ ...(alignment?.note === undefined ? {} : { whisperx: alignment.note }),
};
},
@@ -1349,7 +1388,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
const selected = input.shot_ids === undefined ? state.shots : state.shots.filter((shot) => input.shot_ids!.includes(shot.shot_id));
assert(selected.length > 0, "no requested shot ids exist");
const observer: Observer = state.observer ?? "gemini";
- const { ask, pending, paced } = await askerFor(observer, state);
+ const { ask, pending, paced } = await askerFor(observer, state, undefined, input.runtime);
// An observer that answers out of band produces a list rather than a request, so the launch gap
// and the concurrency cap have nothing to pace.
const rate = paced === false ? { concurrency: Number.MAX_SAFE_INTEGER, gapMs: 0 } : { concurrency, gapMs };
@@ -1711,7 +1750,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
const slot = eventId(basename(renderedPath).replace(/\.[^.]*$/u, ""), new Date(startedAt));
const standIn = await standInBeside(renderedPath);
const observer: Observer = state.observer ?? "gemini";
- const { ask, pending } = await askerFor(observer, state, input.run);
+ const { ask, pending } = await askerFor(observer, state, input.run, input.runtime);
// Which stretch of the reference the render is put beside. A render covers the words a Segment
// or a Selection marks, so that is what the reference is cut to; a shot is a cut in the picture
diff --git a/packages/reference-video-tools/src/transcript.ts b/packages/reference-video-tools/src/transcript.ts
index da254a32..43090ce5 100644
--- a/packages/reference-video-tools/src/transcript.ts
+++ b/packages/reference-video-tools/src/transcript.ts
@@ -1,8 +1,8 @@
import { access } from "node:fs/promises";
import { join } from "node:path";
-import { EndpointRegistry, MemoryResourceStore } from "@hypit/driver-node";
-import type { Need } from "@hypit/protocol";
-import { createLocalWhisperXProvider } from "@hypit/provider-whisperx-local";
+import { MemoryResourceStore } from "@hypit/driver-node";
+import type { Need, StoredValue } from "@hypit/protocol";
+import type { ResourceStore } from "@hypit/runtime";
import { sealSpeechEvidenceAudio } from "@hypit/speech";
import { speechEvidenceTypes } from "@hypit/speech-evidence";
import type { AlignedTranscriptEvidence } from "@hypit/speech-evidence";
@@ -13,28 +13,12 @@ import { assert, command, readBytes, round, writeJson } from "./media.js";
import type { Transcript, TranscriptFile, TranscriptPassage, TranscriptWord } from "./types.js";
/**
- * Whether the local WhisperX service answers.
- *
- * The route used to probe the service by hand, as three commands, before `prepare_reference`. The
- * service answers `/health`, which is the same call the Provider makes before transcribing — made
- * early enough to be reported before the minutes a first start spends loading the model. It reports
- * rather than refuses: a machine without the service still prepares everything except the transcript,
- * which `prepareTranscript` leaves recoverable.
+ * One immediate Need executed through the selected Runtime Profile, outside any Build. The Runtime
+ * host provides it; this module never chooses or instantiates a Provider itself, so whichever
+ * Endpoint the Profile binds to `whisperx-alignment` — HypiHub, a local service, anything else —
+ * measures the transcript.
*/
-export async function whisperxHealth(baseUrl = "http://127.0.0.1:8765"): Promise<{ readonly ok: boolean; readonly reason: string }> {
- const normalized = baseUrl.replace(/\/+$/u, "");
- try {
- const response = await fetch(`${normalized}/health`, { signal: AbortSignal.timeout(5_000) });
- if (!response.ok) return { ok: false, reason: `WhisperX answered ${response.status}` };
- const body = await response.json().catch(() => undefined) as { readonly ok?: unknown } | null | undefined;
- if (body === null || typeof body !== "object" || body.ok !== true) {
- return { ok: false, reason: "WhisperX is up but not ready; a first start spends a few minutes loading the model" };
- }
- return { ok: true, reason: "" };
- } catch {
- return { ok: false, reason: `WhisperX is not reachable at ${normalized}` };
- }
-}
+export type InvokeNeed = (need: Need, resources: ResourceStore) => Promise<{ readonly value: StoredValue }>;
// WhisperX measures in samples of the canonical evidence rate and refuses audio of any other shape,
// so this is both what the audio is extracted at and what its word times are divided by.
@@ -69,10 +53,11 @@ function reason(error: unknown): string {
return error.cause === undefined ? error.message : `${error.message}: ${reason(error.cause)}`;
}
-/** Ask the local WhisperX Provider for the word times of one canonical evidence WAV. */
+/** Ask the Profile's WhisperX alignment Endpoint for the word times of one canonical evidence WAV. */
export async function transcribeSpeechAudio(
audioPath: string,
language: WhisperXLanguage,
+ invoke: InvokeNeed,
): Promise<readonly TranscriptPassage[]> {
const bytes = await readBytes(audioPath);
const resources = new MemoryResourceStore();
@@ -85,17 +70,7 @@ export async function transcribeSpeechAudio(
constraints: whisperXRequestForEvidenceAudio(evidence, { language }),
result: "record:reference-video-transcript",
};
- const registry = new EndpointRegistry();
- await createLocalWhisperXProvider({ instance: "whisperx.local", pool: "whisperx.local" }).install(registry);
- const resolved = registry.resolve(need);
- assert(resolved.status === "resolved", "the local WhisperX Provider does not offer word alignment");
- assert(resolved.registration.kind === "immediate", "the local WhisperX Provider answers word alignment asynchronously");
- const fulfillment = await resolved.registration.handler({
- command: { kind: "fulfill-need", id: "command:reference-video-transcript", need },
- need,
- resources,
- credentials: {},
- });
+ const fulfillment = await invoke(need, resources);
assert(fulfillment.value.kind === "inline", "WhisperX returned alignment evidence by reference");
const aligned = fulfillment.value.value as unknown as AlignedTranscriptEvidence;
return aligned.passages.map((passage): TranscriptPassage => {
@@ -121,8 +96,8 @@ export async function transcribeSpeechAudio(
}
/**
- * The verbatim transcript is local, deterministic evidence rather than an observation: it is never
- * cached with the Gemini answers and never sent to Gemini. A machine that is not running WhisperX
+ * The verbatim transcript is deterministic evidence rather than an observation: it is never cached
+ * with the Gemini answers and never sent to Gemini. A Profile whose alignment Endpoint cannot answer
* still prepares everything else, and is told what did not answer instead of losing the preparation.
*/
export async function prepareTranscript(
@@ -132,14 +107,15 @@ export async function prepareTranscript(
hasAudio: boolean,
language: WhisperXLanguage,
redo: boolean,
+ invoke: InvokeNeed,
): Promise<Transcript> {
if (!hasAudio) return { status: "unavailable", transcript_ref: null, word_count: 0, reason: "the reference video has no audio track" };
try {
const audioPath = join(root, "speech.wav");
if (redo || await access(audioPath).then(() => false, () => true)) await extractSpeechAudio(videoPath, audioPath);
// A refused connection says only "fetch failed"; the reader still has to learn what to start.
- const passages = await transcribeSpeechAudio(audioPath, language)
- .catch((error: unknown) => { throw new Error(`local WhisperX did not answer: ${reason(error)}`); });
+ const passages = await transcribeSpeechAudio(audioPath, language, invoke)
+ .catch((error: unknown) => { throw new Error(`WhisperX alignment did not answer: ${reason(error)}`); });
const transcriptPath = join(root, "transcript.json");
const file: TranscriptFile = { reference_id: reference, audio_ref: audioPath, passages };
await writeJson(transcriptPath, file);
diff --git a/packages/reference-video-tools/test/transcript.test.ts b/packages/reference-video-tools/test/transcript.test.ts
index 0147b2d2..f2ee1d22 100644
--- a/packages/reference-video-tools/test/transcript.test.ts
+++ b/packages/reference-video-tools/test/transcript.test.ts
@@ -4,7 +4,13 @@ import { mkdtemp, readFile, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
+import { canonicalize } from "@hypit/protocol";
+import { sealAlignedTranscriptEvidence } from "@hypit/speech-evidence";
+import { interpretWhisperXTranscript } from "@hypit/whisperx";
+import type { WhisperXAlignmentRequest, WhisperXTranscriptResponse } from "@hypit/whisperx";
+
import { prepareTranscript, transcribeSpeechAudio } from "../src/transcript.js";
+import type { InvokeNeed } from "../src/transcript.js";
import type { TranscriptFile } from "../src/types.js";
const REFERENCE = "ref-fixture";
@@ -32,89 +38,83 @@ function wav(sampleFrames: number): Uint8Array {
return bytes;
}
-async function serving(response: unknown, run: () => Promise<void>): Promise<void> {
- const original = globalThis.fetch;
- const originalApiKey = process.env.HYPIHUB_API_KEY;
- // The provider call is fully mocked below; give it a deterministic credential so the
- // tests exercise request shaping and transcript parsing rather than the developer's
- // local credential store (which is intentionally absent on CI runners).
- process.env.HYPIHUB_API_KEY = "test-hypihub-key";
- globalThis.fetch = (async (input: RequestInfo | URL) => {
- const url = String(input);
- const body = url.endsWith("/health")
- ? {
- ok: true,
- protocol: "hypit.whisperx-service@1",
- serviceVersion: "0.1.0",
- whisperxVersion: "3.8.6",
- model: "small",
- device: "cpu",
- compute: "int8",
- batchSize: 8,
- }
- : url.includes("/models/")
- ? { name: "victor-upmeet/whisperx", endpoints: ["transcriptions"] }
- : url.endsWith("/files")
- ? { url: "https://hypit.ai/test-reference.wav" }
- : response;
- return new Response(JSON.stringify(body), { headers: { "content-type": "application/json" } });
- }) as typeof fetch;
- try { await run(); } finally {
- globalThis.fetch = original;
- if (originalApiKey === undefined) delete process.env.HYPIHUB_API_KEY;
- else process.env.HYPIHUB_API_KEY = originalApiKey;
- }
+/**
+ * The Runtime host's one-shot invocation, answered with a WhisperX-shaped response. These tests
+ * exercise request shaping and transcript parsing; which Provider the Profile binds is the Runtime's
+ * business and never this module's.
+ */
+function answering(response: WhisperXTranscriptResponse): InvokeNeed {
+ return async (need) => {
+ assert.equal(need.capability.name, "whisperx-alignment");
+ const request = need.constraints as unknown as WhisperXAlignmentRequest;
+ assert.equal(request.audio.mediaType, "audio/wav");
+ assert.equal(request.language, "en");
+ return {
+ value: {
+ kind: "inline",
+ value: canonicalize(sealAlignedTranscriptEvidence({
+ passages: interpretWhisperXTranscript(response, request.sampleFrames),
+ })),
+ },
+ };
+ };
}
+const unanswered: InvokeNeed = async () => {
+ throw new Error("no Endpoint in the selected Runtime Profile serves @hypit/whisperx#whisperx-alignment");
+};
+
test("every word carries its own start and end in seconds, not only the passage around it", async () => {
const root = await mkdtemp(join(tmpdir(), "reference-video-transcript-"));
const audioPath = join(root, "speech.wav");
await writeFile(audioPath, wav(32_000));
- await serving({
+ const passages = await transcribeSpeechAudio(audioPath, "en", answering({
language: "en",
segments: [{ start: 0, end: 2, words: [
{ text: "number", start: 0.1, end: 0.4, score: 0.9 },
{ text: "five", start: 1.2, end: 1.6 },
] }],
- }, async () => {
- const passages = await transcribeSpeechAudio(audioPath, "en");
- assert.equal(passages.length, 1);
- assert.equal(passages[0]!.text, "number five");
- assert.deepEqual(passages[0]!.words, [
- { text: "number", start_seconds: 0.1, end_seconds: 0.4, score: 0.9 },
- { text: "five", start_seconds: 1.2, end_seconds: 1.6 },
- ], "placing an on-screen reveal needs the time of the word, not of the sentence it sits in");
- });
+ }));
+ assert.equal(passages.length, 1);
+ assert.equal(passages[0]!.text, "number five");
+ assert.deepEqual(passages[0]!.words, [
+ { text: "number", start_seconds: 0.1, end_seconds: 0.4, score: 0.9 },
+ { text: "five", start_seconds: 1.2, end_seconds: 1.6 },
+ ], "placing an on-screen reveal needs the time of the word, not of the sentence it sits in");
});
test("a complete transcript is written beside the other prepared artifacts and summarized by its word count", async () => {
const root = await mkdtemp(join(tmpdir(), "reference-video-transcript-"));
await writeFile(join(root, "speech.wav"), wav(32_000));
- await serving({
+ const result = await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, true, "en", false, answering({
language: "en",
segments: [{ start: 0, end: 2, words: [{ text: "hello", start: 0.1, end: 0.4 }, { text: "world", start: 1.2, end: 1.6 }] }],
- }, async () => {
- const result = await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, true, "en", false);
- assert.deepEqual(result, { status: "complete", transcript_ref: join(root, "transcript.json"), word_count: 2 });
+ }));
+ assert.deepEqual(result, { status: "complete", transcript_ref: join(root, "transcript.json"), word_count: 2 });
- const file = JSON.parse(await readFile(join(root, "transcript.json"), "utf8")) as TranscriptFile;
- assert.equal(file.reference_id, REFERENCE);
- assert.equal(file.audio_ref, join(root, "speech.wav"));
- assert.equal(file.passages[0]!.words.length, 2);
- });
+ const file = JSON.parse(await readFile(join(root, "transcript.json"), "utf8")) as TranscriptFile;
+ assert.equal(file.reference_id, REFERENCE);
+ assert.equal(file.audio_ref, join(root, "speech.wav"));
+ assert.equal(file.passages[0]!.words.length, 2);
});
-test("a WhisperX that cannot answer leaves the transcript unavailable with its reason instead of failing preparation", async () => {
+test("an alignment Endpoint that cannot answer leaves the transcript unavailable with its reason instead of failing preparation", async () => {
const root = await mkdtemp(join(tmpdir(), "reference-video-transcript-"));
assert.deepEqual(
- await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, false, "en", false),
+ await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, false, "en", false, unanswered),
{ status: "unavailable", transcript_ref: null, word_count: 0, reason: "the reference video has no audio track" },
"a silent reference is prepared, not refused");
+ await writeFile(join(root, "speech.wav"), wav(32_000));
+ const refused = await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, true, "en", false, unanswered);
+ assert.equal(refused.status, "unavailable");
+ assert.match(refused.reason ?? "", /WhisperX alignment did not answer: no Endpoint in the selected Runtime Profile/u,
+ "the Profile, not this tool, decides who measures the transcript; the reader learns exactly what is missing");
+
await writeFile(join(root, "reference.mp4"), "not a video", "utf8");
- const failed = await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, true, "en", false);
+ const failed = await prepareTranscript(REFERENCE, join(root, "reference.mp4"), root, true, "en", true, unanswered);
assert.equal(failed.status, "unavailable");
assert.equal(failed.transcript_ref, null);
assert.equal(typeof failed.reason === "string" && failed.reason.length > 0, true,
diff --git a/packages/runtime-host-node/src/index.ts b/packages/runtime-host-node/src/index.ts
index 32c016df..891ab7fe 100644
--- a/packages/runtime-host-node/src/index.ts
+++ b/packages/runtime-host-node/src/index.ts
@@ -5,7 +5,7 @@ import { delimiter, isAbsolute, join, resolve } from "node:path";
import type { ArtifactAttachment } from "@hypit/workspace";
import type { BuildResultForward } from "@hypit/build-result";
import type { BuildResultRepositoryLocation } from "@hypit/build-result-kit";
-import type { BuildDefinition, BuildState, CapabilityRef } from "@hypit/protocol";
+import type { BuildDefinition, BuildState, CapabilityRef, Need, StoredValue } from "@hypit/protocol";
import type {
BuildCatalogDescriptor,
BuildCompletion,
@@ -13,6 +13,7 @@ import type {
CredentialAcquisition,
CredentialRef,
OperationSnapshot,
+ ResourceStore,
RuntimeWorkerRunOptions,
} from "@hypit/runtime";
import type { RuntimeDoctorDiagnostic } from "@hypit/runtime-kit";
@@ -228,6 +229,12 @@ export type NodeRuntimeHost = {
* Reads the Profile and Endpoint declarations only; never resolves a credential or contacts a service.
*/
providers(capabilities: readonly CapabilityRef[]): Promise<readonly RuntimeHostCapabilityProvider[]>;
+ /**
+ * Execute one immediate Need through the selected Endpoint and its credentials, outside any Build.
+ * The creation-time boundary for observation, transcription and other quick capabilities; it
+ * creates no Build, Result or state, and refuses asynchronous capabilities.
+ */
+ invoke(need: Need, resources: ResourceStore): Promise<{ readonly value: StoredValue }>;
runWorker(readyFile: string, owner: string): Promise<void>;
};
diff --git a/packages/runtime-local/src/config.ts b/packages/runtime-local/src/config.ts
index 6b0509c2..0e0bdf1d 100644
--- a/packages/runtime-local/src/config.ts
+++ b/packages/runtime-local/src/config.ts
@@ -7,12 +7,14 @@ import {
loadNodePackageSelection,
} from "@hypit/package-loader-node";
import type { NodePackageSelectionRequest } from "@hypit/package-loader-node";
+import { EndpointRegistry } from "@hypit/driver-node";
+import type { EndpointFulfillment } from "@hypit/endpoint-kit";
import { assertBuildId, canonicalize, canonicalStringify } from "@hypit/protocol";
-import type { CanonicalValue, CapabilityRef } from "@hypit/protocol";
+import type { CanonicalValue, CapabilityRef, Need } from "@hypit/protocol";
import {
CompositeCredentialStore,
} from "@hypit/runtime";
-import type { CredentialStore, CredentialValue } from "@hypit/runtime";
+import type { CredentialStore, CredentialValue, ResourceStore } from "@hypit/runtime";
import { FileResourceStore } from "@hypit/resource-store-fs";
import { FileBuildResultRepository } from "@hypit/build-result";
import {
@@ -472,6 +474,62 @@ export async function describeRuntimeConfigProviders(
});
}
+/**
+ * Execute one immediate Need through the selected Profile, outside any Build.
+ *
+ * This is the creation-time boundary: the same Endpoint activation, capability resolution and
+ * credential resolution a Build uses, with no Build id, Worker, Result or stored state. The caller
+ * owns the ResourceStore, so whatever the Endpoint reads or returns stays in the caller's hands.
+ * Asynchronous capabilities are refused: slow paid generation goes through a Build.
+ */
+export async function invokeRuntimeConfigNeed(
+ path: string,
+ need: Need,
+ resources: ResourceStore,
+ options: LoadRuntimeConfigOptions = {},
+): Promise<EndpointFulfillment> {
+ const { absolute, document, root, packageRoot } = await openRuntimeConfig(path, options.packageRoot);
+ const hostStateRoot = resolve(options.hostStateRoot ?? hypitHostStateRoot());
+ const registry = options.registry ?? new RuntimeAdapterRegistry();
+ await installRuntimeAdapters(registry, packageRoot, runtimePackageSelection(document), options.distributionPackageRoot);
+ const activations = await activatedEndpoints(document, root, hostStateRoot, registry);
+ const endpoints = new EndpointRegistry();
+ for (const { activation } of activations) await activation.endpoint.install(endpoints);
+ const subject = capabilityKey(need.capability);
+ const resolution = endpoints.resolve(need);
+ if (resolution.status === "missing") {
+ throw new Error(`No Endpoint in ${absolute} serves ${subject}; hypit plan --runtime ${absolute} shows which Endpoint each capability needs`);
+ }
+ if (resolution.status === "ambiguous") {
+ throw new Error(`Several Endpoints in ${absolute} serve ${subject}: ${resolution.endpointIds.join(", ")}; keep exactly one`);
+ }
+ const registration = resolution.registration;
+ if (registration.kind !== "immediate") {
+ throw new Error(`${subject} is an asynchronous capability; creation-time calls only use immediate capabilities, submit a Build for it`);
+ }
+ const activation = activations.find(({ activation: item }) => item.endpoint.instance.id === registration.id)?.activation;
+ if (activation === undefined) throw new Error(`Endpoint ${registration.id} is not declared by ${absolute}`);
+ const stores = await openCredentialStores(document, root, hostStateRoot, registry);
+ try {
+ const credentials: Record<string, CredentialValue> = {};
+ for (const slot of activation.endpoint.credentials) {
+ const value = await stores.store.resolve(slot.ref);
+ if (value === undefined) {
+ throw new Error(`${slot.label} for Endpoint ${slot.endpoint} is not configured. Configure it with: hypit auth login ${slot.endpoint} --runtime ${absolute}`);
+ }
+ credentials[slot.slot] = value;
+ }
+ return await registration.handler({
+ command: { kind: "fulfill-need", id: `command:${need.id}`, need },
+ need,
+ resources,
+ credentials,
+ });
+ } finally {
+ await stores.close();
+ }
+}
+
function diagnostic(error: unknown, code: string, subject?: string): RuntimeDoctorDiagnostic {
return {
severity: "error",
diff --git a/packages/runtime-local/src/host.ts b/packages/runtime-local/src/host.ts
index df3e10e9..ec4fbdb1 100644
--- a/packages/runtime-local/src/host.ts
+++ b/packages/runtime-local/src/host.ts
@@ -14,6 +14,7 @@ import {
createRuntimeFromConfig,
describeRuntimeConfigProviders,
doctorRuntimeConfig,
+ invokeRuntimeConfigNeed,
prepareRuntimeConfigPackages,
preflightRuntimeConfig,
resolveRuntimeConfigPaths,
@@ -135,6 +136,10 @@ export async function openLocalRuntimeHost(
packageRoot: basePackageRoot,
...distribution,
}),
+ invoke: async (need, resources) => await invokeRuntimeConfigNeed(profile, need, resources, {
+ packageRoot: basePackageRoot,
+ ...distribution,
+ }),
runWorker: async (readyFile, owner) => {
const selected = await resolveRuntimeConfigPaths(profile, { packageRoot: basePackageRoot, ...distribution });
const leaseState = new SqliteRuntimeState(join(selected.dataRoot, "runtime.sqlite"));
diff --git a/packages/runtime-local/src/index.ts b/packages/runtime-local/src/index.ts
index 77b50ed5..31711dfb 100644
--- a/packages/runtime-local/src/index.ts
+++ b/packages/runtime-local/src/index.ts
@@ -18,6 +18,7 @@ export {
declaredManagedPrograms,
describeRuntimeConfigProviders,
doctorRuntimeConfig,
+ invokeRuntimeConfigNeed,
prepareRuntimeConfigPackages,
preflightRuntimeConfig,
parseLocalRuntimeProfile,
diff --git a/packages/runtime-local/test/config.test.ts b/packages/runtime-local/test/config.test.ts
index 728c2aa3..61053df6 100644
--- a/packages/runtime-local/test/config.test.ts
+++ b/packages/runtime-local/test/config.test.ts
@@ -24,6 +24,7 @@ import {
describeRuntimeConfigProviders,
doctorProjectBuildResultRepository,
doctorRuntimeConfig,
+ invokeRuntimeConfigNeed,
openProjectBuildResultRepository,
parseLocalRuntimeProfile,
preflightRuntimeConfig,
@@ -316,3 +317,85 @@ test("Runtime providers name the selected Endpoint and its declared price source
}
});
+test("Runtime invoke executes one immediate Need through the selected Endpoint and its credential, outside any Build", async () => {
+ const root = await mkdtemp(join(tmpdir(), "hypit-runtime-invoke-"));
+ const path = join(root, "hypit.runtime.json");
+ await writeFile(path, JSON.stringify(profile({
+ dataRoot: ".",
+ credentials: { secrets: { use: "example.credentials" } },
+ endpoints: { paid: { use: "example.paid" }, slow: { use: "example.slow" } },
+ })));
+ const observe = { module: { name: "example.model", version: "1" }, name: "observe" } as const;
+ const generate = { module: { name: "example.model", version: "1" }, name: "generate" } as const;
+ const missing = { module: { name: "example.model", version: "1" }, name: "transcribe" } as const;
+ const returns = { module: { name: "example.value", version: "1" }, name: "Output" } as const;
+ let configured = true;
+ const registry = new RuntimeAdapterRegistry();
+ registry.registerFacet(createRuntimeCredentialStoreAdapterFacet({
+ use: "example.credentials",
+ validate() {},
+ open: () => ({
+ value: {
+ async resolve(ref) {
+ return configured && ref.store === "secrets" && ref.key === "paid.key" ? { secret: "configured" } : undefined;
+ },
+ },
+ }),
+ }));
+ registry.registerFacet(createRuntimeEndpointAdapterFacet({
+ use: "example.paid",
+ activate: (context) => ({
+ endpoint: defineEndpointPackage({
+ module: { name: "example.provider", version: "1" },
+ facet: "paid",
+ instance: context.instance,
+ pool: context.pool ?? context.instance,
+ credentials: { apiKey: credentialRef("secrets", "paid.key") },
+ capabilities: [{
+ capability: observe,
+ returns,
+ lifecycle: "immediate",
+ handler: ({ need, credentials }) => ({
+ value: { kind: "inline", value: { seen: need.constraints, key: credentials.apiKey?.secret ?? null } },
+ }),
+ }],
+ }),
+ }),
+ }));
+ registry.registerFacet(createRuntimeEndpointAdapterFacet({
+ use: "example.slow",
+ activate: (context) => ({
+ endpoint: defineEndpointPackage({
+ module: { name: "example.slow-provider", version: "1" },
+ facet: "slow",
+ instance: context.instance,
+ pool: context.pool ?? context.instance,
+ capabilities: [{
+ capability: generate,
+ returns,
+ lifecycle: "asynchronous",
+ endpoint: {
+ start: async () => ({ status: "completed" as const, result: { value: { kind: "inline" as const, value: null } } }),
+ poll: async () => ({ status: "completed" as const, result: { value: { kind: "inline" as const, value: null } } }),
+ },
+ }],
+ }),
+ }),
+ }));
+ const resources = { async get() { return undefined; }, async put() { throw new Error("unused"); } } as never;
+ const need = (capability: typeof observe | typeof generate | typeof missing) => ({
+ id: "need:creation-time", capability, returns, constraints: { question: "what happens?" }, result: "record:creation-time",
+ });
+ try {
+ assert.deepEqual(await invokeRuntimeConfigNeed(path, need(observe), resources, { registry }), {
+ value: { kind: "inline", value: { seen: { question: "what happens?" }, key: "configured" } },
+ });
+ await assert.rejects(invokeRuntimeConfigNeed(path, need(missing), resources, { registry }), /No Endpoint in .* serves example\.model@1#transcribe/u);
+ await assert.rejects(invokeRuntimeConfigNeed(path, need(generate), resources, { registry }), /asynchronous capability/u);
+ configured = false;
+ await assert.rejects(invokeRuntimeConfigNeed(path, need(observe), resources, { registry }), /apiKey for Endpoint paid is not configured/u);
+ } finally {
+ await rm(root, { recursive: true, force: true });
+ }
+});
+
diff --git a/packages/video-cli/src/index.ts b/packages/video-cli/src/index.ts
index 2cebe101..418464d8 100644
--- a/packages/video-cli/src/index.ts
+++ b/packages/video-cli/src/index.ts
@@ -16,6 +16,8 @@ export {
} from "./compiler.js";
export { videoCliDistribution } from "./distribution.js";
export { discoverVideoSourcePackages } from "./package-selection.js";
+/** The project's selected Runtime Profile, read the way `hypit` reads it, for tools that run beside the CLI. */
+export { findRuntimeProfile } from "@hypit/cli";
export function runVideoCli(
argv: readonly string[],
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index e20253f0..f1c9505d 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -2422,9 +2422,6 @@ importers:
'@hypit/core':
specifier: workspace:*
version: link:../core
- '@hypit/credential-store-os':
- specifier: workspace:*
- version: link:../credential-store-os
'@hypit/driver-node':
specifier: workspace:*
version: link:../driver-node
@@ -2458,18 +2455,9 @@ importers:
'@hypit/protocol':
specifier: workspace:*
version: link:../protocol
- '@hypit/provider-hypihub':
- specifier: workspace:*
- version: link:../provider-hypihub
'@hypit/provider-media-local':
specifier: workspace:*
version: link:../provider-media-local
- '@hypit/provider-vertex':
- specifier: workspace:*
- version: link:../provider-vertex
- '@hypit/provider-whisperx-local':
- specifier: workspace:*
- version: link:../provider-whisperx-local
'@hypit/run-markup':
specifier: workspace:*
version: link:../run-markup
@@ -2491,6 +2479,9 @@ importers:
'@hypit/svs':
specifier: workspace:*
version: link:../svs
+ '@hypit/text':
+ specifier: workspace:*
+ version: link:../text
'@hypit/video-cli':
specifier: workspace:*
version: link:../video-cli
diff --git a/skills/hypit/references/credentials.md b/skills/hypit/references/credentials.md
index 96c1b9a8..3448e942 100644
--- a/skills/hypit/references/credentials.md
+++ b/skills/hypit/references/credentials.md
@@ -34,9 +34,9 @@ HypiHub is the default for supported paid capabilities and Gemini VLM. It can be
author explicitly selects another Provider.
For ordinary `@hypit/gemini` Author Source, the Runtime Profile selects
`@hypit/provider-hypihub` or `@hypit/provider-vertex`; changing that Endpoint never changes Source.
-For the reference-video preprocessing observer, `HYPIT_GEMINI_PROVIDER=auto` (the default) uses the
-completed HypiHub OAuth environment. Set `HYPIT_GEMINI_PROVIDER=vertex` only when the author explicitly
-requested Vertex and both Google variables are configured. If a configured key cannot reach the requested
+The reference-video preprocessing observer uses the same Gemini Endpoint the selected Runtime Profile
+binds: pass `--runtime <profile>` or select the Profile with `hypit runtime use`. There is no separate
+environment switch for it. If a configured key cannot reach the requested
model, return to environment setup and use HypiHub OAuth when that model is available there; do not ask
them to change Author Source.
@@ -101,8 +101,8 @@ curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $MIMO_API_KEY
https://api.xiaomimimo.com/v1/models
```
-HypiHub OAuth or the Vertex pair are what the reference-video route's `gemini` observer needs,
-depending on `HYPIT_GEMINI_PROVIDER`. Without either backend, that route can run its `agent` observer
+The reference-video route's `gemini` observer needs whatever credential the selected Runtime Profile's
+Gemini Endpoint declares (HypiHub OAuth by default). Without one, that route can run its `agent` observer
instead, which reaches no Provider — `reconstruction/observers.md` says how the author chooses.
`hypit runtime down` stops the Worker for the whole project, so a Build running in another terminal
diff --git a/skills/hypit/references/reconstruction/observers.md b/skills/hypit/references/reconstruction/observers.md
index ae46a946..4e40f37b 100644
--- a/skills/hypit/references/reconstruction/observers.md
+++ b/skills/hypit/references/reconstruction/observers.md
@@ -10,10 +10,9 @@ without knowing which observer produced it.
## The two
-**`gemini`** uploads the shot clips and the whole reference to the configured Gemini backend. It sees
-motion as motion and hears the sound. With the default `HYPIT_GEMINI_PROVIDER=auto`, the completed
-HypiHub OAuth environment uses HypiHub. Vertex is used only when the author explicitly selected it and
-both Google credentials are configured. Each observation is a paid request; do not silently fall back
+**`gemini`** uploads the shot clips and the whole reference to the Gemini Endpoint the selected Runtime
+Profile binds (HypiHub by default; Vertex only when the author's Profile binds it). It sees motion as
+motion and hears the sound. Each observation is a paid request; do not silently fall back
to `agent` because a credential is missing.
**`agent`** hands the observations to you. It needs no credentials and reaches no Provider: the CLI
diff --git a/skills/hypit/references/runtime.md b/skills/hypit/references/runtime.md
index 69b3e317..e2d1b929 100644
--- a/skills/hypit/references/runtime.md
+++ b/skills/hypit/references/runtime.md
@@ -59,9 +59,8 @@ author explicitly asks not to use HypiHub or explicitly selects that Provider.
An Author Source importing `@hypit/gemini` remains Provider-neutral. Bind those capabilities to
`@hypit/provider-hypihub` for HypiHub's upload-backed Gemini endpoint, or to
-`@hypit/provider-vertex` for direct Vertex. Never make this Build-time choice with
-`HYPIT_GEMINI_PROVIDER`; that environment switch belongs only to the reference-video preprocessing
-observer, which runs outside a Build.
+`@hypit/provider-vertex` for direct Vertex. The reference-video preprocessing observer runs outside a
+Build but reads the same Profile binding; there is no separate environment switch.
Provider-specific input limits are also part of preflight. For the KIE GPT Image 2 route, do not
submit `4:3`, `3:4` or `4:5`; the route rejects those aspect ratios before upload or paid submission.