Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

feat: route Gemini through runtime providers

rpone6264a57f256d
diff --git a/.agents/skills/hypit/references/credentials.md b/.agents/skills/hypit/references/credentials.md
index dc180901..3e131747 100644
--- a/.agents/skills/hypit/references/credentials.md
+++ b/.agents/skills/hypit/references/credentials.md
@@ -21,7 +21,9 @@ for a HypiHub key when that model is available there. Do not ask them to change
switch Provider.
HypiHub is optional and is never required when the selected Runtime Profile has another Provider.
-For Gemini VLM/reference observation, `HYPIT_GEMINI_PROVIDER=auto` (the default) uses HypiHub when
+For ordinary `@hypit/gemini` Author Source, the Runtime Profile selects
+`@hypit/provider-hypihub` or `@hypit/provider-vertex`; changing that Endpoint never changes Source.
+For the reference-video preprocessing observer only, `HYPIT_GEMINI_PROVIDER=auto` (the default) uses HypiHub when
`HYPIHUB_API_KEY` is present and otherwise uses Vertex when its two Google variables are present.
Set `HYPIT_GEMINI_PROVIDER=hypihub` or `vertex` to select one explicitly. If a user's configured key
cannot reach the requested model, point them to [hypit.ai](https://hypit.ai) for a HypiHub key instead
diff --git a/.agents/skills/hypit/references/runtime.md b/.agents/skills/hypit/references/runtime.md
index 458fe007..b8866475 100644
--- a/.agents/skills/hypit/references/runtime.md
+++ b/.agents/skills/hypit/references/runtime.md
@@ -12,7 +12,7 @@ Source.
Start from the Runtime Profile template in `docs/guide/runtime.md` (the repository does not ship a
canonical `examples/*/hypit.runtime.json`). Bind one Endpoint per capability your Sources actually
-reach: the picture and video models, speech, alignment, the media Provider and the local renderer.
+reach: the picture and video models, Gemini VLM, speech, alignment, the media Provider and the local renderer.
Read each Provider's README for the shape of its `config`, and reference credentials through the store
rather than writing any secret into the file.
@@ -51,10 +51,17 @@ gateway, so a typical profile looks like this:
```
`defaultConcurrency` is the total HypiHub pool shared by its model capabilities. Tune it to the
-quota behind the key; the Provider keeps each exact capability in its own lane. An explicit KIE
+quota behind the key; image/video capabilities keep exact-model lanes while Gemini requests share
+one conservative lane. An explicit KIE
profile remains valid when a project deliberately chooses `@hypit/provider-kie`, but it is not the
default route.
+An Author Source importing `@hypit/gemini` remains Provider-neutral. Bind those capabilities to
+`@hypit/provider-hypihub` for HypiHub's upload-backed Gemini endpoint, or to
+`@hypit/provider-vertex` for direct Vertex. Never make this Build-time choice with
+`HYPIT_GEMINI_PROVIDER`; that environment switch belongs only to the reference-video preprocessing
+observer, which runs outside a Build.
+
Provider-specific input limits are also part of preflight. For the KIE GPT Image 2 route, do not
submit `4:3`, `3:4` or `4:5`; the route rejects those aspect ratios before upload or paid submission.
diff --git a/package.json b/package.json
index 00639431..a2bbb537 100644
--- a/package.json
+++ b/package.json
@@ -79,6 +79,7 @@
"@hypit/film-studio": "workspace:*",
"@hypit/fonts-open": "workspace:*",
"@hypit/generation": "workspace:*",
+ "@hypit/gemini": "workspace:*",
"@hypit/gpt-image": "workspace:*",
"@hypit/grok-imagine": "workspace:*",
"@hypit/hyperframes": "workspace:*",
diff --git a/packages/gemini/README.md b/packages/gemini/README.md
new file mode 100644
index 00000000..e69db86e
--- /dev/null
+++ b/packages/gemini/README.md
@@ -0,0 +1,17 @@
+# `@hypit/gemini`
+
+Provider-neutral Gemini text and multimodal requests for Hypit Author Source.
+
+`<gemini:Generate>` accepts a system instruction, a text prompt and any number of image, video or
+audio Artifact references. The model package owns the semantic request and exact model capability;
+the selected Runtime Endpoint owns credentials, uploads and the service-specific wire format.
+
+```xml
+<gemini:Generate id="describe" model="gemini-3.7-flash"
+ instruction={instruction} prompt={prompt}>
+ <gemini:Reference media={reference-video}/>
+</gemini:Generate>
+```
+
+The generated Text is published as `<id>.text`. Switching between
+`@hypit/provider-hypihub` and `@hypit/provider-vertex` requires only a Runtime Profile change.
diff --git a/packages/gemini/package.json b/packages/gemini/package.json
new file mode 100644
index 00000000..700a48c4
--- /dev/null
+++ b/packages/gemini/package.json
@@ -0,0 +1,17 @@
+{
+ "name": "@hypit/gemini",
+ "version": "0.0.0-dev",
+ "license": "SEE LICENSE IN LICENSE",
+ "private": true,
+ "type": "module",
+ "exports": { ".": "./src/index.ts" },
+ "hypit": { "activation": "./src/activation.ts" },
+ "dependencies": {
+ "@hypit/artifact": "workspace:*",
+ "@hypit/component-kit": "workspace:*",
+ "@hypit/elaborator": "workspace:*",
+ "@hypit/markup": "workspace:*",
+ "@hypit/protocol": "workspace:*",
+ "@hypit/text": "workspace:*"
+ }
+}
diff --git a/packages/gemini/src/activation.ts b/packages/gemini/src/activation.ts
new file mode 100644
index 00000000..38eaa234
--- /dev/null
+++ b/packages/gemini/src/activation.ts
@@ -0,0 +1,22 @@
+import { createMarkupSurfaceHostFacet } from "@hypit/markup";
+
+import {
+ decodeGeminiGenerateSurface,
+ geminiComponent,
+ geminiManifest,
+ geminiMarkupSurfaces,
+ geminiModuleRef,
+} from "./index.js";
+
+export const hypitPackage = {
+ format: "hypit.node-package@1" as const,
+ modules: [{ manifest: geminiManifest }],
+ components: [geminiComponent],
+ hostFacets: [createMarkupSurfaceHostFacet({
+ module: geminiModuleRef,
+ declaration: geminiMarkupSurfaces[0],
+ handler: decodeGeminiGenerateSurface,
+ })],
+};
+
+export default hypitPackage;
diff --git a/packages/gemini/src/component.ts b/packages/gemini/src/component.ts
new file mode 100644
index 00000000..72534072
--- /dev/null
+++ b/packages/gemini/src/component.ts
@@ -0,0 +1,71 @@
+import type { ComponentPackage, ProducerHandlerContext } from "@hypit/component-kit";
+import { canonicalize } from "@hypit/protocol";
+import type { StoredValue } from "@hypit/protocol";
+import type { Text } from "@hypit/text";
+import { verifyText } from "@hypit/text";
+
+import { geminiModels, geminiProducers, geminiTypes } from "./manifest.js";
+import { sealGeminiRequest, verifyGeminiRequest } from "./request.js";
+import type { GeminiRequest } from "./request.js";
+
+function inline<T>(value: StoredValue | undefined, subject: string): T {
+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);
+ return value.value as unknown as T;
+}
+
+function text(value: StoredValue | undefined, subject: string): string {
+ const result = inline<Text>(value, subject);
+ verifyText(result);
+ return result.value;
+}
+
+export const geminiComponent = {
+ validators: [
+ { type: geminiTypes.request, handler: ({ value }) => verifyGeminiRequest(inline(value, "Gemini request")) },
+ { type: geminiTypes.draft, handler: ({ value }) => verifyGeminiRequest(inline(value, "Gemini request draft")) },
+ ],
+ producers: [
+ {
+ producer: geminiProducers.start,
+ handler: ({ inputs }: ProducerHandlerContext) => ({
+ outputs: { draft: { kind: "inline", value: canonicalize(sealGeminiRequest({
+ instruction: text(inputs.instruction?.value, "Gemini instruction"),
+ prompt: text(inputs.prompt?.value, "Gemini prompt"),
+ media: [],
+ })) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: geminiProducers.bindMedia,
+ handler: ({ inputs }: ProducerHandlerContext) => {
+ const draft = inline<GeminiRequest>(inputs.draft?.value, "Gemini request draft");
+ verifyGeminiRequest(draft);
+ const artifact = inputs.media?.value;
+ if (artifact?.kind !== "blob") throw new Error("Gemini media must be a Blob Artifact");
+ return {
+ outputs: { draft: { kind: "inline", value: canonicalize(sealGeminiRequest({
+ ...draft, media: [...draft.media, { artifact }],
+ })) } },
+ needs: {},
+ };
+ },
+ },
+ {
+ producer: geminiProducers.finalize,
+ handler: ({ inputs }: ProducerHandlerContext) => {
+ const draft = inline<GeminiRequest>(inputs.draft?.value, "Gemini request draft");
+ verifyGeminiRequest(draft);
+ return { outputs: { request: { kind: "inline", value: canonicalize(draft) } }, needs: {} };
+ },
+ },
+ ...geminiModels.map((model) => ({
+ producer: geminiProducers[model],
+ handler: ({ inputs }: ProducerHandlerContext) => {
+ const request = inline<GeminiRequest>(inputs.request?.value, `${model} request`);
+ verifyGeminiRequest(request);
+ return { outputs: {}, needs: { text: canonicalize(request) } };
+ },
+ })),
+ ],
+} satisfies ComponentPackage;
diff --git a/packages/gemini/src/fragment.ts b/packages/gemini/src/fragment.ts
new file mode 100644
index 00000000..a442c80b
--- /dev/null
+++ b/packages/gemini/src/fragment.ts
@@ -0,0 +1,50 @@
+import { artifactTypes } from "@hypit/artifact";
+import { sealGraphFragment } from "@hypit/elaborator";
+import type { FragmentOperation } from "@hypit/elaborator";
+import { textTypes } from "@hypit/text";
+
+import type { GeminiModel } from "./manifest.js";
+import { geminiProducers, geminiTypes } from "./manifest.js";
+
+const input = (name: string) => ({ kind: "fragment-input" as const, name });
+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
+
+export function createGeminiFragment(model: GeminiModel, mediaCount: number) {
+ if (!Number.isSafeInteger(mediaCount) || mediaCount < 0) throw new Error("Gemini media count is invalid");
+ const inputs = [
+ { name: "instruction", type: textTypes.text },
+ { name: "prompt", type: textTypes.text },
+ ...Array.from({ length: mediaCount }, (_item, index) => ({
+ name: `media-${String(index + 1).padStart(4, "0")}`,
+ type: artifactTypes.blob,
+ })),
+ ];
+ const operations: FragmentOperation[] = [{
+ id: "start-request", producer: geminiProducers.start,
+ inputs: { instruction: input("instruction"), prompt: input("prompt") },
+ result: { kind: "output" as const, name: "draft" },
+ }];
+ let draft = operation("start-request");
+ for (let index = 0; index < mediaCount; index += 1) {
+ const id = `bind-media-${String(index + 1).padStart(4, "0")}`;
+ operations.push({
+ id, producer: geminiProducers.bindMedia,
+ inputs: { draft, media: input(`media-${String(index + 1).padStart(4, "0")}`) },
+ result: { kind: "output" as const, name: "draft" },
+ });
+ draft = operation(id);
+ }
+ operations.push({
+ id: "finalize-request", producer: geminiProducers.finalize, inputs: { draft },
+ result: { kind: "output" as const, name: "request" },
+ });
+ operations.push({
+ id: "generate", producer: geminiProducers[model], inputs: { request: operation("finalize-request") },
+ result: { kind: "need" as const, name: "text" },
+ });
+ return sealGraphFragment({
+ inputs,
+ operations,
+ exports: [{ name: "text", type: textTypes.text, root: operation("generate") }],
+ });
+}
diff --git a/packages/gemini/src/index.ts b/packages/gemini/src/index.ts
new file mode 100644
index 00000000..cd120cd1
--- /dev/null
+++ b/packages/gemini/src/index.ts
@@ -0,0 +1,17 @@
+export { geminiComponent } from "./component.js";
+export { createGeminiFragment } from "./fragment.js";
+export {
+ geminiCapabilities,
+ geminiDependency,
+ geminiManifest,
+ geminiMarkupSurfaces,
+ geminiModels,
+ geminiModuleRef,
+ geminiProducers,
+ geminiTypes,
+} from "./manifest.js";
+export type { GeminiModel } from "./manifest.js";
+export { sealGeminiRequest, verifyGeminiRequest } from "./request.js";
+export type { GeminiMediaPart, GeminiRequest } from "./request.js";
+export type { GeminiGenerateInput, GeminiInlinePart } from "./parts.js";
+export { decodeGeminiGenerateSurface } from "./surface.js";
diff --git a/packages/gemini/src/manifest.ts b/packages/gemini/src/manifest.ts
new file mode 100644
index 00000000..2fba2147
--- /dev/null
+++ b/packages/gemini/src/manifest.ts
@@ -0,0 +1,96 @@
+import { artifactDependency, artifactTypes } from "@hypit/artifact";
+import type { CapabilityRef, ModuleManifest, ProducerRef, TypeRef } from "@hypit/protocol";
+import { textDependency, textTypes } from "@hypit/text";
+
+export const geminiModuleRef = { name: "@hypit/gemini", version: "1" } as const;
+
+export const geminiModels = ["gemini-3.1-pro-preview", "gemini-3.7-flash"] as const;
+export type GeminiModel = typeof geminiModels[number];
+
+export const geminiTypes = {
+ request: { module: geminiModuleRef, name: "GeminiRequest" },
+ draft: { module: geminiModuleRef, name: "GeminiRequestDraft" },
+} satisfies Record<string, TypeRef>;
+
+export const geminiCapabilities = Object.fromEntries(geminiModels.map((model) => [model, {
+ module: geminiModuleRef,
+ name: model,
+} satisfies CapabilityRef])) as Record<GeminiModel, CapabilityRef>;
+
+export const geminiProducers = {
+ start: { module: geminiModuleRef, name: "start-request" },
+ bindMedia: { module: geminiModuleRef, name: "bind-media" },
+ finalize: { module: geminiModuleRef, name: "finalize-request" },
+ ...Object.fromEntries(geminiModels.map((model) => [model, {
+ module: geminiModuleRef,
+ name: `request-${model}`,
+ } satisfies ProducerRef])),
+} as { readonly start: ProducerRef; readonly bindMedia: ProducerRef; readonly finalize: ProducerRef } & Record<GeminiModel, ProducerRef>;
+
+export const geminiMarkupSurfaces = [{
+ name: "generate", tag: "Generate", mode: "structured",
+ outputs: [geminiTypes.draft],
+ vocabulary: {
+ summary: "Asks one exact Gemini model to answer a Text prompt with optional image, video or audio references.",
+ attributes: [
+ { name: "id", kind: "identifier", required: true,
+ summary: "Names this request and its generated Text." },
+ { name: "model", kind: "literal", required: true, values: [...geminiModels],
+ summary: "Selects the exact Gemini capability without choosing its Provider." },
+ { name: "instruction", kind: "reference", required: true, accepts: [textTypes.text],
+ summary: "Selects the system instruction Text." },
+ { name: "prompt", kind: "reference", required: true, accepts: [textTypes.text],
+ summary: "Selects the user prompt Text." },
+ ],
+ children: [{ tag: "Reference", cardinality: "many",
+ summary: "Attaches one image, video or audio Artifact to the request in authored order.",
+ attributes: [{ name: "media", kind: "reference", required: true, accepts: [artifactTypes.blob],
+ summary: "Selects the media Artifact Gemini reads." }] }],
+ ports: [{ name: "text", type: textTypes.text,
+ summary: "The generated answer, addressed as `<id>.text`." }],
+ example: `<gemini:Generate id="describe" model="gemini-3.7-flash" instruction={instruction} prompt={prompt}>
+ <gemini:Reference media={reference-video}/>
+</gemini:Generate>`,
+ notes: [
+ "The Source declares no API key, upload URL or Provider-specific file representation.",
+ "Every Reference remains a graph edge; the selected Provider decides how its Artifact bytes travel.",
+ ],
+ },
+}] as const;
+
+export const geminiManifest: ModuleManifest = {
+ format: "hypit.module@1",
+ name: geminiModuleRef.name,
+ version: geminiModuleRef.version,
+ dependencies: [artifactDependency, textDependency],
+ types: [{ name: geminiTypes.request.name }, { name: geminiTypes.draft.name }],
+ capabilities: geminiModels.map((model) => ({ name: model, returns: textTypes.text })),
+ producers: [
+ {
+ name: geminiProducers.start.name,
+ inputs: [{ name: "instruction", type: textTypes.text }, { name: "prompt", type: textTypes.text }],
+ outputs: [{ name: "draft", type: geminiTypes.draft }],
+ needs: [],
+ },
+ {
+ name: geminiProducers.bindMedia.name,
+ inputs: [{ name: "draft", type: geminiTypes.draft }, { name: "media", type: artifactTypes.blob }],
+ outputs: [{ name: "draft", type: geminiTypes.draft }],
+ needs: [],
+ },
+ {
+ name: geminiProducers.finalize.name,
+ inputs: [{ name: "draft", type: geminiTypes.draft }],
+ outputs: [{ name: "request", type: geminiTypes.request }],
+ needs: [],
+ },
+ ...geminiModels.map((model) => ({
+ name: geminiProducers[model].name,
+ inputs: [{ name: "request", type: geminiTypes.request }],
+ outputs: [],
+ needs: [{ name: "text", capability: geminiCapabilities[model], returns: textTypes.text }],
+ })),
+ ],
+};
+
+export const geminiDependency = { module: geminiModuleRef } as const;
diff --git a/packages/gemini/src/parts.ts b/packages/gemini/src/parts.ts
new file mode 100644
index 00000000..23906715
--- /dev/null
+++ b/packages/gemini/src/parts.ts
@@ -0,0 +1,9 @@
+/** Provider-neutral inline parts used by embedded Gemini callers before a Runtime Artifact exists. */
+export type GeminiInlinePart =
+ | { readonly text: string; readonly inlineData?: never }
+ | { readonly inlineData: { readonly mimeType: string; readonly data: string }; readonly text?: never };
+
+export type GeminiGenerateInput = {
+ readonly parts: readonly GeminiInlinePart[];
+ readonly instruction: string;
+};
diff --git a/packages/gemini/src/request.ts b/packages/gemini/src/request.ts
new file mode 100644
index 00000000..ac8870b9
--- /dev/null
+++ b/packages/gemini/src/request.ts
@@ -0,0 +1,43 @@
+import { canonicalize } from "@hypit/protocol";
+import type { BlobRef } from "@hypit/protocol";
+
+export type GeminiMediaPart = { readonly artifact: BlobRef };
+export type GeminiRequest = {
+ readonly instruction: string;
+ readonly prompt: string;
+ readonly media: readonly GeminiMediaPart[];
+};
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function object(value: unknown, subject: string): Record<string, unknown> {
+ assert(value !== null && typeof value === "object" && !Array.isArray(value), `${subject} must be an object`);
+ return value as Record<string, unknown>;
+}
+
+export function verifyGeminiRequest(value: unknown): asserts value is GeminiRequest {
+ const request = object(value, "Gemini request");
+ assert(typeof request.instruction === "string" && request.instruction.trim().length > 0,
+ "Gemini request instruction must be non-empty text");
+ assert(typeof request.prompt === "string" && request.prompt.trim().length > 0,
+ "Gemini request prompt must be non-empty text");
+ assert(Array.isArray(request.media), "Gemini request media must be an array");
+ for (const [index, raw] of request.media.entries()) {
+ const item = object(raw, `Gemini request media[${index}]`);
+ const artifact = object(item.artifact, `Gemini request media[${index}].artifact`);
+ assert(artifact.kind === "blob" && typeof artifact.digest === "string"
+ && typeof artifact.size === "number" && Number.isSafeInteger(artifact.size)
+ && typeof artifact.mediaType === "string",
+ `Gemini request media[${index}].artifact must be a BlobRef`);
+ assert(/^(?:image|video|audio)\//u.test(artifact.mediaType),
+ `Gemini request media[${index}] must be image, video or audio media`);
+ }
+}
+
+export function sealGeminiRequest(value: GeminiRequest): GeminiRequest {
+ const result = canonicalize(value) as unknown as GeminiRequest;
+ verifyGeminiRequest(result);
+ return result;
+}
diff --git a/packages/gemini/src/surface.ts b/packages/gemini/src/surface.ts
new file mode 100644
index 00000000..c61fde38
--- /dev/null
+++ b/packages/gemini/src/surface.ts
@@ -0,0 +1,73 @@
+import { artifactTypes } from "@hypit/artifact";
+import type {
+ MarkupAttributeValue,
+ StructuredElement,
+ StructuredSurfaceHandler,
+ SurfaceResolvedReference,
+} from "@hypit/markup";
+import type { TypeRef } from "@hypit/protocol";
+import { textTypes } from "@hypit/text";
+
+import { createGeminiFragment } from "./fragment.js";
+import { geminiModels } from "./manifest.js";
+import type { GeminiModel } from "./manifest.js";
+
+function assert(condition: unknown, message: string): asserts condition { if (!condition) throw new Error(message); }
+function localName(value: string): string { return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value; }
+function sameType(left: TypeRef, right: TypeRef): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+}
+function reference(value: MarkupAttributeValue | undefined, subject: string,
+ expected: TypeRef, resolveReference: (path: string) => SurfaceResolvedReference | undefined): SurfaceResolvedReference {
+ assert(typeof value === "object" && value.kind === "reference", `${subject} must be a reference`);
+ const resolved = resolveReference(value.path);
+ assert(resolved !== undefined && sameType(resolved.type, expected), `${subject} has the wrong type`);
+ return resolved;
+}
+function literal(element: StructuredElement, name: string): string {
+ const value = element.attributes[name];
+ assert(typeof value === "string" && value.trim().length > 0, `${element.name}.${name} must be non-empty text`);
+ return value.trim();
+}
+
+export const decodeGeminiGenerateSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ const allowed = ["id", "instruction", "model", "prompt"];
+ const unknown = Object.keys(element.attributes).find((name) => !allowed.includes(name));
+ assert(unknown === undefined, `${element.name} does not accept ${unknown}`);
+ const id = literal(element, "id");
+ const model = literal(element, "model");
+ assert(geminiModels.includes(model as GeminiModel), `${element.name}.model is unsupported`);
+ const instruction = reference(element.attributes.instruction, `${element.name}.instruction`, textTypes.text, resolveReference);
+ const prompt = reference(element.attributes.prompt, `${element.name}.prompt`, textTypes.text, resolveReference);
+ const media: SurfaceResolvedReference[] = [];
+ for (const child of element.children) {
+ if (child.kind === "text") {
+ assert(child.value.trim().length === 0, `${element.name} accepts only Reference children`);
+ continue;
+ }
+ assert(localName(child.name) === "Reference", `${element.name} accepts only Reference children`);
+ assert(Object.keys(child.attributes).join(",") === "media", `${child.name} requires only media`);
+ assert(!child.children.some((item) => item.kind === "element" || item.value.trim().length > 0), `${child.name} must be empty`);
+ const resolved = reference(child.attributes.media, `${child.name}.media`, artifactTypes.blob, resolveReference);
+ if (resolved.record?.value.kind === "blob") {
+ assert(/^(?:image|video|audio)\//u.test(resolved.record.value.mediaType), `${child.name}.media must be image, video or audio`);
+ }
+ media.push(resolved);
+ }
+ const fragment = createGeminiFragment(model as GeminiModel, media.length);
+ return {
+ records: [],
+ fragments: [fragment],
+ components: [{
+ id,
+ fragment: fragment.id,
+ inputs: {
+ instruction: instruction.ref,
+ prompt: prompt.ref,
+ ...Object.fromEntries(media.map((item, index) => [`media-${String(index + 1).padStart(4, "0")}`, item.ref])),
+ },
+ outputs: { text: `${id}.text` },
+ range: element.range,
+ }],
+ };
+};
diff --git a/packages/gemini/test/surface.test.ts b/packages/gemini/test/surface.test.ts
new file mode 100644
index 00000000..6e8d61fc
--- /dev/null
+++ b/packages/gemini/test/surface.test.ts
@@ -0,0 +1,39 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { fixtureDigest } from "../../../test/fixture-digest.js";
+
+import { decodeGeminiGenerateSurface, geminiCapabilities } from "@hypit/gemini";
+import { artifactTypes } from "@hypit/artifact";
+import { textTypes } from "@hypit/text";
+
+test("Gemini Surface builds one provider-neutral need with ordered media edges", async () => {
+ const fragmentResult = await decodeGeminiGenerateSurface({
+ sourceName: "main.svml",
+ element: {
+ kind: "element", name: "gemini:Generate",
+ attributes: {
+ id: "read", model: "gemini-3.7-flash",
+ instruction: { kind: "reference", path: "instruction" },
+ prompt: { kind: "reference", path: "prompt" },
+ },
+ children: [{
+ kind: "element", name: "gemini:Reference",
+ attributes: { media: { kind: "reference", path: "video" } }, children: [], range: { start: 20, end: 40 },
+ }], range: { start: 0, end: 50 },
+ },
+ resolveAsset: () => { throw new Error("not used"); },
+ resolveReference: (path) => path === "video" ? {
+ path, ref: { kind: "record", id: "video" }, type: artifactTypes.blob,
+ record: { id: "video", type: artifactTypes.blob, value: {
+ kind: "blob", digest: fixtureDigest("gemini-video"), size: 10, mediaType: "video/mp4",
+ } },
+ } : {
+ path, ref: { kind: "record", id: path }, type: textTypes.text,
+ record: { id: path, type: textTypes.text, value: { kind: "inline", value: { value: path } } },
+ },
+ });
+ assert.equal(fragmentResult.fragments.length, 1);
+ const operation = fragmentResult.fragments[0]!.operations.find((item) => item.id === "generate");
+ assert.deepEqual(operation?.producer, { module: geminiCapabilities["gemini-3.7-flash"].module, name: "request-gemini-3.7-flash" });
+ assert.deepEqual(fragmentResult.components[0]?.outputs, { text: "read.text" });
+});
diff --git a/packages/media/README.md b/packages/media/README.md
index 7ea4f5fe..be864467 100644
--- a/packages/media/README.md
+++ b/packages/media/README.md
@@ -2,15 +2,16 @@
Author-facing media declarations for the neutral `@hypit/media` contract module.
-`<media:Image>`, `<media:Audio>` and `<media:Font>` request bytes through the Host-owned source-asset
-capability. Image and Audio emit one content-addressed `BlobArtifact`; Font wraps the same immutable
+`<media:Image>`, `<media:Audio>`, `<media:Video>` and `<media:Font>` request bytes through the Host-owned source-asset
+capability. Image, Audio and Video emit one content-addressed `BlobArtifact`; Font wraps the same immutable
bytes with exact weight/style metadata as a one-source `FontArtifactRef`. A logical font face may
also contain several exact Unicode-range sources when another package, such as
`@hypit/fonts-open`, owns a sharded installed font. `FontStackRef` is the generic ordered value
for one primary face followed by exact glyph fallbacks. Media declarations do not open files themselves,
inspect media, call a Provider or promote declared audio to speech. The consuming author package
decides whether an audio artifact is a voice reference, soundtrack, evidence source or something
-else, and which visual component uses an exact Font.
+else, whether a video is a generation reference or another input, and which visual component uses
+an exact Font.
The same module owns the deliberately narrow `SynchronizedMedia` contract used after technical
normalization. It contains one common `frameRate`/`frameCount`, an optional visual Artifact with its
diff --git a/packages/media/src/activation.ts b/packages/media/src/activation.ts
index 7d5604f0..13261311 100644
--- a/packages/media/src/activation.ts
+++ b/packages/media/src/activation.ts
@@ -1,7 +1,7 @@
import { createMarkupSurfaceHostFacet } from "@hypit/markup";
import {
- decodeMediaAudioSurface, decodeMediaFontSurface, decodeMediaImageSurface, mediaComponent,
+ decodeMediaAudioSurface, decodeMediaFontSurface, decodeMediaImageSurface, decodeMediaVideoSurface, mediaComponent,
mediaManifest,
mediaModuleRef,
mediaMarkupSurfaces,
@@ -22,6 +22,11 @@ export const hypitPackage = {
declaration: mediaMarkupSurfaces.find((item) => item.name === "audio")!,
handler: decodeMediaAudioSurface,
}),
+ createMarkupSurfaceHostFacet({
+ module: mediaModuleRef,
+ declaration: mediaMarkupSurfaces.find((item) => item.name === "video")!,
+ handler: decodeMediaVideoSurface,
+ }),
createMarkupSurfaceHostFacet({
module: mediaModuleRef,
declaration: mediaMarkupSurfaces.find((item) => item.name === "font")!,
diff --git a/packages/media/src/index.ts b/packages/media/src/index.ts
index 108b5dc1..5bbfd30b 100644
--- a/packages/media/src/index.ts
+++ b/packages/media/src/index.ts
@@ -2,6 +2,7 @@ export {
decodeMediaAudioSurface,
decodeMediaFontSurface,
decodeMediaImageSurface,
+ decodeMediaVideoSurface,
} from "./surface.js";
export { mediaComponent } from "./component.js";
export { sealMediaInspection, sealMediaStreamSelection, sealMuxedMedia, sealRenderedVisual, sealSynchronizedMedia, sealTimelineAudio, synchronizedMediaSampleFrames, verifyMediaInspection, verifyMediaStreamSelection, verifyMuxedMedia, verifyRenderedVisual, verifySynchronizedMedia, verifyTimelineAudio } from "./identity.js";
diff --git a/packages/media/src/manifest.ts b/packages/media/src/manifest.ts
index 2a05dcc9..e3a33f24 100644
--- a/packages/media/src/manifest.ts
+++ b/packages/media/src/manifest.ts
@@ -56,6 +56,27 @@ export const mediaMarkupSurfaces = [
],
},
},
+ {
+ name: "video", tag: "Video", mode: "structured", outputs: [artifactTypes.blob],
+ vocabulary: {
+ summary: "Requests one authored video file from the Host and publishes it as a content-addressed Artifact.",
+ attributes: [
+ { name: "id", kind: "identifier", required: true,
+ summary: "Names the Artifact Record this element publishes." },
+ { name: "src", kind: "literal", required: true,
+ summary: "Points at the video file, resolved by the Host against the source that declares it." },
+ { name: "media-type", kind: "literal", required: false,
+ summary: "States the video media type when the src extension does not name one." },
+ ],
+ example: `<media:Video id="reference" src="./assets/reference.mp4"/>`,
+ notes: [
+ "The element is empty; it accepts no children and no text.",
+ "`.m4v`, `.mov`, `.mp4` and `.webm` name their own media type; any other file needs `media-type`.",
+ "A written `media-type` must begin with `video/`, and the resolved bytes must arrive as a video Artifact.",
+ "The Artifact is published under the bare `id`, and the consuming package decides what the video means.",
+ ],
+ },
+ },
{
name: "font", tag: "Font", mode: "structured", outputs: [mediaTypes.fontArtifact],
vocabulary: {
diff --git a/packages/media/src/surface.ts b/packages/media/src/surface.ts
index 40f2f987..25651998 100644
--- a/packages/media/src/surface.ts
+++ b/packages/media/src/surface.ts
@@ -27,6 +27,13 @@ const AUDIO_MEDIA_TYPES = new Map([
[".wav", "audio/wav"],
]);
+const VIDEO_MEDIA_TYPES = new Map([
+ [".m4v", "video/x-m4v"],
+ [".mov", "video/quicktime"],
+ [".mp4", "video/mp4"],
+ [".webm", "video/webm"],
+]);
+
const FONT_MEDIA_TYPES = new Map([
[".otf", "font/otf"],
[".ttf", "font/ttf"],
@@ -51,7 +58,7 @@ function assertChildrenEmpty(element: StructuredElement): void {
function mediaTypeFor(
element: StructuredElement,
source: string,
- kind: "image" | "audio" | "font",
+ kind: "image" | "audio" | "video" | "font",
known: ReadonlyMap<string, string>,
): string {
const explicit = element.attributes["media-type"];
@@ -74,7 +81,7 @@ function mediaTypeFor(
async function decodeMediaAssetSurface(
element: StructuredElement,
resolveAsset: Parameters<StructuredSurfaceHandler>[0]["resolveAsset"],
- kind: "image" | "audio",
+ kind: "image" | "audio" | "video",
known: ReadonlyMap<string, string>,
) {
const names = Object.keys(element.attributes).sort();
@@ -107,6 +114,9 @@ export const decodeMediaImageSurface: StructuredSurfaceHandler = async ({ elemen
export const decodeMediaAudioSurface: StructuredSurfaceHandler = async ({ element, resolveAsset }) =>
await decodeMediaAssetSurface(element, resolveAsset, "audio", AUDIO_MEDIA_TYPES);
+export const decodeMediaVideoSurface: StructuredSurfaceHandler = async ({ element, resolveAsset }) =>
+ await decodeMediaAssetSurface(element, resolveAsset, "video", VIDEO_MEDIA_TYPES);
+
export const decodeMediaFontSurface: StructuredSurfaceHandler = async ({ element, resolveAsset }) => {
const names = Object.keys(element.attributes).sort();
if (names.join(",") !== "id,src,style,weight" && names.join(",") !== "id,media-type,src,style,weight") {
diff --git a/packages/media/test/asset-surface.test.ts b/packages/media/test/asset-surface.test.ts
new file mode 100644
index 00000000..63ca0653
--- /dev/null
+++ b/packages/media/test/asset-surface.test.ts
@@ -0,0 +1,62 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { fixtureDigest } from "../../../test/fixture-digest.js";
+
+import {
+ decodeMediaAudioSurface,
+ decodeMediaImageSurface,
+ decodeMediaVideoSurface,
+} from "@hypit/media";
+
+const range = { start: 0, end: 80 };
+
+for (const fixture of [
+ { label: "Image", decode: decodeMediaImageSurface, src: "./reference.png", mediaType: "image/png" },
+ { label: "Audio", decode: decodeMediaAudioSurface, src: "./reference.wav", mediaType: "audio/wav" },
+ { label: "Video", decode: decodeMediaVideoSurface, src: "./reference.mp4", mediaType: "video/mp4" },
+] as const) {
+ test(`${fixture.label} Surface publishes the resolved BlobArtifact`, async () => {
+ const artifact = {
+ kind: "blob" as const,
+ digest: fixtureDigest(`media:${fixture.label.toLowerCase()}`),
+ size: 2_048,
+ mediaType: fixture.mediaType,
+ };
+ const requests: unknown[] = [];
+ const result = await fixture.decode({
+ sourceName: "main.svml",
+ element: {
+ kind: "element",
+ name: `media:${fixture.label}`,
+ attributes: { id: "reference", src: fixture.src },
+ children: [],
+ range,
+ },
+ resolveReference: () => undefined,
+ resolveAsset: (request) => {
+ requests.push(request);
+ return { artifact };
+ },
+ });
+
+ assert.deepEqual(requests, [{ from: fixture.src, mediaType: fixture.mediaType, range }]);
+ assert.deepEqual(result.records, [{ id: "reference", type: {
+ module: { name: "@hypit/artifact", version: "1" }, name: "BlobArtifact",
+ }, value: artifact, range }]);
+ });
+}
+
+test("Video Surface rejects ambiguous extensions and non-video resolved artifacts", async () => {
+ const decode = (attributes: Readonly<Record<string, string>>, mediaType = "video/mp4") => decodeMediaVideoSurface({
+ sourceName: "main.svml",
+ element: { kind: "element", name: "media:Video", attributes, children: [], range },
+ resolveReference: () => undefined,
+ resolveAsset: (request) => ({ artifact: {
+ kind: "blob", digest: fixtureDigest("media:video-invalid"), size: 10, mediaType,
+ } }),
+ });
+
+ await assert.rejects(async () => await decode({ id: "video", src: "./clip.bin" }), /known video extension/u);
+ await assert.rejects(async () => await decode({ id: "video", src: "./clip.mp4" }, "image/png"), /video artifact/u);
+ await assert.rejects(async () => await decode({ id: "video", src: "./clip.bin", "media-type": "image/png" }), /video media type/u);
+});
diff --git a/packages/provider-hypihub/README.md b/packages/provider-hypihub/README.md
index 656f1a49..fc3aeda8 100644
--- a/packages/provider-hypihub/README.md
+++ b/packages/provider-hypihub/README.md
@@ -30,8 +30,9 @@ Runtime Profile example:
}
```
-Gemini VLM callers can use the exported `createHypiHubGeminiGenerator` without changing their
-provider-facing code. Set `HYPIHUB_API_KEY` (and optionally `HYPIHUB_BASE_URL`; either the origin or
+Gemini VLM is also exposed as the provider-neutral `@hypit/gemini` Runtime capability, so an Author
+Source can select the exact model while the Runtime chooses HypiHub or Vertex. Existing embedded
+callers can still use the exported `createHypiHubGeminiGenerator`. Set `HYPIHUB_API_KEY` (and optionally `HYPIHUB_BASE_URL`; either the origin or
an existing `/v1`/`/v1beta` base is accepted) only when choosing HypiHub. The Runtime Provider also
accepts the origin or either versioned base and normalizes it to `/v1`; missing or insufficient user
credentials should be resolved at [hypit.ai](https://hypit.ai). Referenced image, audio and video
diff --git a/packages/provider-hypihub/package.json b/packages/provider-hypihub/package.json
index 138fbb4d..fc837de1 100644
--- a/packages/provider-hypihub/package.json
+++ b/packages/provider-hypihub/package.json
@@ -8,10 +8,12 @@
"hypit": { "activation": "./src/activation.ts" },
"dependencies": {
"@hypit/endpoint-kit": "workspace:*",
+ "@hypit/gemini": "workspace:*",
"@hypit/generation": "workspace:*",
"@hypit/protocol": "workspace:*",
"@hypit/runtime": "workspace:*",
- "@hypit/runtime-kit": "workspace:*"
+ "@hypit/runtime-kit": "workspace:*",
+ "@hypit/text": "workspace:*"
},
"devDependencies": {
"@hypit/driver-node": "workspace:*",
diff --git a/packages/provider-hypihub/src/gemini.ts b/packages/provider-hypihub/src/gemini.ts
index dfe67f87..fe880c50 100644
--- a/packages/provider-hypihub/src/gemini.ts
+++ b/packages/provider-hypihub/src/gemini.ts
@@ -1,6 +1,6 @@
-export type HypiHubGeminiPart =
- | { readonly text: string }
- | { readonly inlineData: { readonly mimeType: string; readonly data: string } }
+import type { GeminiInlinePart } from "@hypit/gemini";
+
+export type HypiHubGeminiPart = GeminiInlinePart
| { readonly fileData: { readonly mimeType?: string; readonly fileUri: string } };
export type HypiHubGeminiGeneratorOptions = {
@@ -8,6 +8,8 @@ export type HypiHubGeminiGeneratorOptions = {
readonly model?: string;
readonly baseUrl?: string;
readonly requestTimeoutMs?: number;
+ readonly maxRateLimitRetries?: number;
+ readonly rateLimitRetryDelayMs?: number;
readonly fetch?: typeof globalThis.fetch;
};
@@ -20,6 +22,22 @@ function assert(condition: unknown, message: string): asserts condition {
if (!condition) throw new Error(message);
}
+function positiveInteger(value: number, subject: string): number {
+ assert(Number.isSafeInteger(value) && value > 0, `${subject} must be a positive integer`);
+ return value;
+}
+
+function retryDelay(response: Response, fallback: number, attempt: number): number {
+ const retryAfter = response.headers.get("retry-after")?.trim();
+ if (retryAfter !== undefined && retryAfter !== "") {
+ const seconds = Number(retryAfter);
+ if (Number.isFinite(seconds) && seconds >= 0) return Math.max(250, Math.round(seconds * 1_000));
+ const date = Date.parse(retryAfter);
+ if (Number.isFinite(date)) return Math.max(250, date - Date.now());
+ }
+ return fallback * (attempt + 1);
+}
+
function partValue(part: HypiHubGeminiPart, uploaded: Map<string, Promise<string>>,
upload: (mimeType: string, data: string) => Promise<string>): Promise<Record<string, unknown>> {
if ("text" in part) return Promise.resolve({ text: part.text });
@@ -54,6 +72,11 @@ export function createHypiHubGeminiGenerator(options: HypiHubGeminiGeneratorOpti
.replace(/\/$/u, "");
const model = options.model?.trim() || "gemini-3.1-pro-preview";
const timeout = options.requestTimeoutMs ?? 120_000;
+ const maxRateLimitRetries = options.maxRateLimitRetries ?? 3;
+ assert(Number.isSafeInteger(maxRateLimitRetries) && maxRateLimitRetries >= 0,
+ "HypiHub Gemini maxRateLimitRetries must be a non-negative integer");
+ const rateLimitRetryDelayMs = positiveInteger(options.rateLimitRetryDelayMs ?? 2_000,
+ "HypiHub Gemini rateLimitRetryDelayMs");
const fetcher = options.fetch ?? globalThis.fetch;
return async (input: HypiHubGeminiGenerateInput): Promise<string> => {
@@ -84,17 +107,24 @@ export function createHypiHubGeminiGenerator(options: HypiHubGeminiGeneratorOpti
return url;
};
const parts = await Promise.all(input.parts.map((part) => partValue(part, uploaded, upload)));
- const response = await fetcher(`${baseUrl}/v1beta/models/${encodeURIComponent(model)}:generateContent`, {
- method: "POST",
- signal: controller.signal,
- headers: { "x-goog-api-key": apiKey, "content-type": "application/json" },
- body: JSON.stringify({
- systemInstruction: { parts: [{ text: input.instruction }] },
- contents: [{ role: "user", parts }],
- generationConfig: { temperature: 1, responseMimeType: "text/plain" },
- }),
- });
- const text = await response.text();
+ let response: Response | undefined;
+ let text = "";
+ for (let attempt = 0; attempt <= maxRateLimitRetries; attempt += 1) {
+ response = await fetcher(`${baseUrl}/v1beta/models/${encodeURIComponent(model)}:generateContent`, {
+ method: "POST",
+ signal: controller.signal,
+ headers: { "x-goog-api-key": apiKey, "content-type": "application/json" },
+ body: JSON.stringify({
+ systemInstruction: { parts: [{ text: input.instruction }] },
+ contents: [{ role: "user", parts }],
+ generationConfig: { temperature: 1, responseMimeType: "text/plain" },
+ }),
+ });
+ text = await response.text();
+ if (response.status !== 429 || attempt === maxRateLimitRetries) break;
+ await new Promise((resolve) => setTimeout(resolve, retryDelay(response!, rateLimitRetryDelayMs, attempt)));
+ }
+ assert(response !== undefined, "HypiHub Gemini made no request");
if (!response.ok) {
const message = `HypiHub Gemini returned HTTP ${response.status}: ${text.slice(0, 300)}`;
if (response.status === 401 || response.status === 403 || response.status === 404) {
diff --git a/packages/provider-hypihub/src/provider.ts b/packages/provider-hypihub/src/provider.ts
index b8d158cf..949616de 100644
--- a/packages/provider-hypihub/src/provider.ts
+++ b/packages/provider-hypihub/src/provider.ts
@@ -1,9 +1,14 @@
import type { AsyncEndpoint, EndpointFulfillment, EndpointInvocationContext, EndpointPollContext, EndpointStartContext, EndpointOutcome, ImmediateEndpointHandler } from "@hypit/endpoint-kit";
import { defineEndpointPackage, wakeAfter } from "@hypit/endpoint-kit";
+import { geminiCapabilities, geminiModels, verifyGeminiRequest } from "@hypit/gemini";
+import type { GeminiRequest } from "@hypit/gemini";
import { canonicalize } from "@hypit/protocol";
import type { BlobRef, CapabilityRef } from "@hypit/protocol";
import { credentialRef } from "@hypit/runtime";
import type { CredentialRef, ArtifactStore } from "@hypit/runtime";
+import { sealText } from "@hypit/text";
+import { textTypes } from "@hypit/text";
+import { createHypiHubGeminiGenerator } from "./gemini.js";
import { hypiHubRouteForCapability, hypiHubRoutes } from "./routes.js";
export const hypiHubProviderModuleRef = { name: "@hypit/provider-hypihub", version: "1" } as const;
@@ -52,6 +57,12 @@ function failure(error: unknown): EndpointOutcome {
const message = guidedMessage(error);
return { status: "failed", failure: { code: "HYPIHUB_ERROR", message } };
}
+
+function inlineGeminiRequest(context: EndpointInvocationContext): GeminiRequest {
+ const request = context.need.constraints as unknown;
+ verifyGeminiRequest(request);
+ return request;
+}
function jobId(value: Record<string, unknown>): string {
const id = value.id ?? value.job_id;
assert(typeof id === "string" && id.length > 0, "HypiHub response has no job id");
@@ -219,10 +230,31 @@ export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}
throw new Error(guidedMessage(error), { cause: error });
}
};
+ const geminiEndpoint: ImmediateEndpointHandler = async (context) => {
+ const request = inlineGeminiRequest(context);
+ const generate = createHypiHubGeminiGenerator({
+ apiKey: credential(context),
+ model: context.need.capability.name,
+ ...(options.baseUrl === undefined ? {} : { baseUrl: options.baseUrl }),
+ ...(options.requestTimeoutMs === undefined ? {} : { requestTimeoutMs: options.requestTimeoutMs }),
+ ...(options.fetch === undefined ? {} : { fetch: options.fetch }),
+ maxRateLimitRetries: 5,
+ rateLimitRetryDelayMs: 3_000,
+ });
+ const parts: Array<{ text: string } | { inlineData: { mimeType: string; data: string } }> = [{ text: request.prompt }];
+ for (const item of request.media) {
+ const bytes = await context.artifacts.get(item.artifact.digest);
+ assert(bytes !== undefined, `HypiHub Gemini reference artifact ${item.artifact.digest} is unavailable`);
+ parts.push({ inlineData: { mimeType: item.artifact.mediaType, data: Buffer.from(bytes).toString("base64") } });
+ }
+ const value = await generate({ parts, instruction: request.instruction });
+ return { value: { kind: "inline", value: canonicalize(sealText(value)) } };
+ };
return defineEndpointPackage({
module: hypiHubProviderModuleRef, facet: "gateway", instance: options.instance ?? "hypihub.default", pool: options.pool ?? options.instance ?? "hypihub.default",
credentials: { apiKey: options.apiKey ?? credentialRef("env", "HYPIHUB_API_KEY") }, credentialInputs: { apiKey: { label: "HypiHub API key" } }, defaultConcurrency: options.defaultConcurrency ?? 4,
- capabilities: hypiHubRoutes
+ capabilities: [
+ ...hypiHubRoutes
// HypiHub keys do not necessarily include the MiMo audio models. Keep
// official Xiaomi MiMo as the safe default; opting into HypiHub audio is
// an explicit Runtime decision for a key that actually has those models.
@@ -230,5 +262,14 @@ export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}
.map((route) => route.media === "audio"
? { capability: route.capability, returns: route.returns, lifecycle: "immediate" as const, handler: audioEndpoint, lane: route.capability.name }
: { capability: route.capability, returns: route.returns, lifecycle: "asynchronous" as const, endpoint: asyncEndpoint, lane: route.capability.name }),
+ ...geminiModels.map((model) => ({
+ capability: geminiCapabilities[model],
+ returns: textTypes.text,
+ lifecycle: "immediate" as const,
+ handler: geminiEndpoint,
+ lane: "gemini",
+ maxConcurrency: 1,
+ })),
+ ],
});
}
diff --git a/packages/provider-hypihub/test/gemini.test.ts b/packages/provider-hypihub/test/gemini.test.ts
index c1d07e7a..f51381ee 100644
--- a/packages/provider-hypihub/test/gemini.test.ts
+++ b/packages/provider-hypihub/test/gemini.test.ts
@@ -46,3 +46,29 @@ test("HypiHub Gemini points unavailable models to HypiHub", async () => {
await assert.rejects(() => generate({ instruction: "x", parts: [{ text: "x" }] }),
/get a HypiHub key with this model enabled at https:\/\/hypit\.ai/iu);
});
+
+test("HypiHub Gemini retries upstream rate limits without reuploading files", async () => {
+ let uploads = 0;
+ let generations = 0;
+ const generate = createHypiHubGeminiGenerator({
+ apiKey: "test-key",
+ maxRateLimitRetries: 2,
+ rateLimitRetryDelayMs: 1,
+ fetch: async (input) => {
+ const url = String(input);
+ if (url.endsWith("/v1/files")) {
+ uploads += 1;
+ return Response.json({ url: "https://hypit.ai/files/ref" });
+ }
+ generations += 1;
+ return generations < 3
+ ? new Response("rate limited", { status: 429, headers: { "retry-after": "0" } })
+ : Response.json({ candidates: [{ content: { parts: [{ text: "OK" }] } }] });
+ },
+ });
+ assert.equal(await generate({ instruction: "x", parts: [
+ { text: "x" }, { inlineData: { mimeType: "image/png", data: "aW1hZ2U=" } },
+ ] }), "OK");
+ assert.equal(uploads, 1);
+ assert.equal(generations, 3);
+});
diff --git a/packages/provider-hypihub/test/provider.test.ts b/packages/provider-hypihub/test/provider.test.ts
index f273b6b9..4162fcbb 100644
--- a/packages/provider-hypihub/test/provider.test.ts
+++ b/packages/provider-hypihub/test/provider.test.ts
@@ -3,8 +3,10 @@ import test from "node:test";
import { EndpointRegistry, MemoryArtifactStore } from "@hypit/driver-node";
import type { AsyncEndpoint } from "@hypit/endpoint-kit";
+import { geminiCapabilities, sealGeminiRequest } from "@hypit/gemini";
import type { CanonicalValue, Need } from "@hypit/protocol";
import { mimoTtsEndpoints } from "@hypit/mimo-tts";
+import { textTypes } from "@hypit/text";
import { sealSeedanceRequest, seedanceEndpoints } from "@hypit/seedance";
import { createHypiHubProvider } from "../src/provider.js";
@@ -114,3 +116,51 @@ test("HypiHub uploads referenced Artifacts once, submits their HTTPS URLs, and p
assert.equal(completed.status, "completed");
assert.equal(calls.length, 6);
});
+
+test("HypiHub fulfills Gemini through the Runtime endpoint and uploads every media Artifact", async () => {
+ const artifacts = new MemoryArtifactStore();
+ const image = await artifacts.put(new Uint8Array([1, 2, 3]), "image/png");
+ const video = await artifacts.put(new Uint8Array([4, 5, 6]), "video/mp4");
+ const request: Need = {
+ id: "need:hypihub-gemini",
+ capability: geminiCapabilities["gemini-3.7-flash"],
+ returns: textTypes.text,
+ constraints: sealGeminiRequest({
+ instruction: "Answer briefly.", prompt: "Inspect both references.",
+ media: [{ artifact: image }, { artifact: video }],
+ }) as unknown as CanonicalValue,
+ result: "record:hypihub-gemini",
+ };
+ const uploads: string[] = [];
+ let generationBody: Record<string, unknown> | undefined;
+ const registry = new EndpointRegistry();
+ await createHypiHubProvider({ fetch: async (input, init) => {
+ const url = String(input);
+ if (url.endsWith("/v1/files")) {
+ assert.ok(init?.body instanceof FormData);
+ const file = init.body.get("file");
+ assert.ok(file instanceof File);
+ uploads.push(file.type);
+ return Response.json({ url: `https://hypit.ai/files/${uploads.length}` });
+ }
+ if (url.includes(":generateContent")) {
+ generationBody = JSON.parse(String(init?.body)) as Record<string, unknown>;
+ return Response.json({ candidates: [{ content: { parts: [{ text: "provider works" }] } }] });
+ }
+ throw new Error(`Unexpected URL ${url}`);
+ } }).install(registry);
+ const resolution = registry.resolve(request);
+ assert.equal(resolution.status, "resolved");
+ assert.equal(resolution.registration.kind, "immediate");
+ const result = await resolution.registration.handler({
+ command: { kind: "fulfill-need", id: "command:hypihub-gemini", need: request },
+ need: request, artifacts, credentials: { apiKey: { secret: "test-key" } },
+ });
+ assert.deepEqual(result.value, { kind: "inline", value: { value: "provider works" } });
+ assert.deepEqual(uploads, ["image/png", "video/mp4"]);
+ assert.deepEqual((generationBody?.contents as readonly unknown[]), [{ role: "user", parts: [
+ { text: "Inspect both references." },
+ { fileData: { mimeType: "image/png", fileUri: "https://hypit.ai/files/1" } },
+ { fileData: { mimeType: "video/mp4", fileUri: "https://hypit.ai/files/2" } },
+ ] }]);
+});
diff --git a/packages/provider-vertex/README.md b/packages/provider-vertex/README.md
index 20a12c11..398de16c 100644
--- a/packages/provider-vertex/README.md
+++ b/packages/provider-vertex/README.md
@@ -1,8 +1,9 @@
# `@hypit/provider-vertex`
-Vertex Gemini backend for callers that need text, image or video inline-data input. The package owns
-the Google GenAI client and credential parsing; callers only select the model and provide a generator
-input.
+Vertex Gemini Runtime Provider and embedded backend for callers that need text, image or video input.
+It fulfills the same provider-neutral `@hypit/gemini` capabilities as HypiHub, so Runtime Profiles
+choose the backend without changing Author Source. The package owns the Google GenAI client and
+credential parsing.
```ts
import { createVertexGeminiGenerator } from "@hypit/provider-vertex";
diff --git a/packages/provider-vertex/package.json b/packages/provider-vertex/package.json
index 856b3673..82c2fcb4 100644
--- a/packages/provider-vertex/package.json
+++ b/packages/provider-vertex/package.json
@@ -5,7 +5,17 @@
"private": true,
"type": "module",
"exports": { ".": "./src/index.ts" },
+ "hypit": { "activation": "./src/activation.ts" },
"dependencies": {
- "@google/genai": "1.52.0"
+ "@google/genai": "1.52.0",
+ "@hypit/endpoint-kit": "workspace:*",
+ "@hypit/gemini": "workspace:*",
+ "@hypit/protocol": "workspace:*",
+ "@hypit/runtime": "workspace:*",
+ "@hypit/runtime-kit": "workspace:*",
+ "@hypit/text": "workspace:*"
+ },
+ "devDependencies": {
+ "@hypit/driver-node": "workspace:*"
}
}
diff --git a/packages/provider-vertex/src/activation.ts b/packages/provider-vertex/src/activation.ts
new file mode 100644
index 00000000..5cd04c3e
--- /dev/null
+++ b/packages/provider-vertex/src/activation.ts
@@ -0,0 +1,38 @@
+import {
+ createRuntimeEndpointAdapterFacet,
+ runtimeConfigCredentialRef,
+ runtimeConfigExact,
+ runtimeConfigObject,
+ runtimeConfigPositiveInteger,
+ runtimeConfigString,
+} from "@hypit/runtime-kit";
+
+import { createVertexProvider } from "./provider.js";
+
+const adapter = createRuntimeEndpointAdapterFacet({
+ use: "@hypit/provider-vertex",
+ activate(context) {
+ if (context.pool === undefined) throw new Error("Vertex Provider Pool is required");
+ const config = runtimeConfigObject(context.config, "Vertex");
+ runtimeConfigExact(config, ["project", "credentials", "location", "defaultConcurrency", "requestTimeoutMs"], "Vertex");
+ const project = runtimeConfigCredentialRef(config.project, "Vertex project");
+ if (project === undefined) throw new Error("Vertex project CredentialRef is required");
+ const credentials = runtimeConfigCredentialRef(config.credentials, "Vertex credentials");
+ if (credentials === undefined) throw new Error("Vertex credentials CredentialRef is required");
+ const location = runtimeConfigString(config.location, "Vertex location");
+ const defaultConcurrency = runtimeConfigPositiveInteger(config.defaultConcurrency, "Vertex defaultConcurrency");
+ const requestTimeoutMs = runtimeConfigPositiveInteger(config.requestTimeoutMs, "Vertex requestTimeoutMs");
+ return { endpoint: createVertexProvider({
+ instance: context.instance,
+ pool: context.pool,
+ project,
+ credentials,
+ ...(location === undefined ? {} : { location }),
+ ...(defaultConcurrency === undefined ? {} : { defaultConcurrency }),
+ ...(requestTimeoutMs === undefined ? {} : { requestTimeoutMs }),
+ }) };
+ },
+});
+
+export const hypitPackage = { format: "hypit.node-package@1" as const, hostFacets: [adapter] };
+export default hypitPackage;
diff --git a/packages/provider-vertex/src/gemini.ts b/packages/provider-vertex/src/gemini.ts
index b0aade1f..3d07d84d 100644
--- a/packages/provider-vertex/src/gemini.ts
+++ b/packages/provider-vertex/src/gemini.ts
@@ -1,14 +1,9 @@
import { GoogleGenAI } from "@google/genai";
import type { Content } from "@google/genai";
+import type { GeminiGenerateInput, GeminiInlinePart } from "@hypit/gemini";
-export type VertexGeminiPart =
- | { readonly text?: string; readonly inlineData?: never }
- | { readonly inlineData?: { readonly mimeType: string; readonly data: string }; readonly text?: never };
-
-export type VertexGeminiGenerateInput = {
- readonly parts: readonly VertexGeminiPart[];
- readonly instruction: string;
-};
+export type VertexGeminiPart = GeminiInlinePart;
+export type VertexGeminiGenerateInput = GeminiGenerateInput;
export type VertexGeminiGeneratorOptions = {
readonly project: string;
diff --git a/packages/provider-vertex/src/index.ts b/packages/provider-vertex/src/index.ts
index 49605787..f8ae7798 100644
--- a/packages/provider-vertex/src/index.ts
+++ b/packages/provider-vertex/src/index.ts
@@ -4,3 +4,5 @@ export type {
VertexGeminiGeneratorOptions,
VertexGeminiPart,
} from "./gemini.js";
+export { createVertexProvider, vertexProviderModuleRef } from "./provider.js";
+export type { CreateVertexProviderOptions } from "./provider.js";
diff --git a/packages/provider-vertex/src/provider.ts b/packages/provider-vertex/src/provider.ts
new file mode 100644
index 00000000..82c6d02c
--- /dev/null
+++ b/packages/provider-vertex/src/provider.ts
@@ -0,0 +1,73 @@
+import { defineEndpointPackage } from "@hypit/endpoint-kit";
+import type { EndpointInvocationContext, ImmediateEndpointHandler } from "@hypit/endpoint-kit";
+import { geminiCapabilities, geminiModels, verifyGeminiRequest } from "@hypit/gemini";
+import type { GeminiRequest } from "@hypit/gemini";
+import { canonicalize } from "@hypit/protocol";
+import { credentialRef } from "@hypit/runtime";
+import type { CredentialRef } from "@hypit/runtime";
+import { sealText, textTypes } from "@hypit/text";
+
+import { createVertexGeminiGenerator } from "./gemini.js";
+
+export const vertexProviderModuleRef = { name: "@hypit/provider-vertex", version: "1" } as const;
+
+export type CreateVertexProviderOptions = {
+ readonly instance?: string;
+ readonly pool?: string;
+ readonly project: CredentialRef;
+ readonly credentials: CredentialRef;
+ readonly location?: string;
+ readonly defaultConcurrency?: number;
+ readonly requestTimeoutMs?: number;
+};
+
+function secret(context: EndpointInvocationContext, name: "project" | "credentials"): string {
+ const value = context.credentials[name]?.secret;
+ if (typeof value !== "string" || value.trim().length === 0) throw new Error(`Vertex ${name} credential is unavailable`);
+ return value;
+}
+
+export function createVertexProvider(options: CreateVertexProviderOptions) {
+ const handler: ImmediateEndpointHandler = async (context) => {
+ const request = context.need.constraints as unknown;
+ verifyGeminiRequest(request);
+ const typed = request as GeminiRequest;
+ const generate = createVertexGeminiGenerator({
+ project: secret(context, "project"),
+ credentials: secret(context, "credentials"),
+ model: context.need.capability.name,
+ ...(options.location === undefined ? {} : { location: options.location }),
+ ...(options.requestTimeoutMs === undefined ? {} : { requestTimeoutMs: options.requestTimeoutMs }),
+ });
+ const parts: Array<{ text: string } | { inlineData: { mimeType: string; data: string } }> = [{ text: typed.prompt }];
+ for (const item of typed.media) {
+ const bytes = await context.artifacts.get(item.artifact.digest);
+ if (bytes === undefined) throw new Error(`Vertex Gemini reference artifact ${item.artifact.digest} is unavailable`);
+ parts.push({ inlineData: { mimeType: item.artifact.mediaType, data: Buffer.from(bytes).toString("base64") } });
+ }
+ const value = await generate({ parts, instruction: typed.instruction });
+ return { value: { kind: "inline", value: canonicalize(sealText(value)) } };
+ };
+ return defineEndpointPackage({
+ module: vertexProviderModuleRef,
+ facet: "gemini",
+ instance: options.instance ?? "vertex.default",
+ pool: options.pool ?? options.instance ?? "vertex.default",
+ credentials: {
+ project: options.project ?? credentialRef("env", "GOOGLE_CLOUD_PROJECT"),
+ credentials: options.credentials ?? credentialRef("env", "GOOGLE_APPLICATION_CREDENTIALS_JSON"),
+ },
+ credentialInputs: {
+ project: { label: "Google Cloud project" },
+ credentials: { label: "Google application credentials", kind: "json" },
+ },
+ defaultConcurrency: options.defaultConcurrency ?? 4,
+ capabilities: geminiModels.map((model) => ({
+ capability: geminiCapabilities[model],
+ returns: textTypes.text,
+ lifecycle: "immediate" as const,
+ handler,
+ lane: model,
+ })),
+ });
+}
diff --git a/packages/provider-vertex/test/provider.test.ts b/packages/provider-vertex/test/provider.test.ts
new file mode 100644
index 00000000..a15f5b84
--- /dev/null
+++ b/packages/provider-vertex/test/provider.test.ts
@@ -0,0 +1,33 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import { EndpointRegistry } from "@hypit/driver-node";
+import { geminiCapabilities } from "@hypit/gemini";
+import { credentialRef } from "@hypit/runtime";
+import { textTypes } from "@hypit/text";
+
+import { createVertexProvider } from "../src/provider.js";
+
+test("Vertex Runtime provider exposes the same provider-neutral Gemini capabilities", async () => {
+ const registry = new EndpointRegistry();
+ const provider = createVertexProvider({
+ project: credentialRef("env", "GOOGLE_CLOUD_PROJECT"),
+ credentials: credentialRef("env", "GOOGLE_APPLICATION_CREDENTIALS_JSON"),
+ });
+ await provider.install(registry);
+ for (const model of ["gemini-3.1-pro-preview", "gemini-3.7-flash"] as const) {
+ const resolution = registry.resolve({
+ id: `need:${model}`,
+ capability: geminiCapabilities[model],
+ returns: textTypes.text,
+ constraints: { instruction: "x", prompt: "x", media: [] },
+ result: `record:${model}`,
+ });
+ assert.equal(resolution.status, "resolved");
+ assert.equal(resolution.registration.kind, "immediate");
+ }
+ assert.deepEqual(provider.credentials.map((item) => ({ slot: item.slot, kind: item.kind })), [
+ { slot: "credentials", kind: "json" },
+ { slot: "project", kind: "secret" },
+ ]);
+});
diff --git a/packages/reference-video-tools/package.json b/packages/reference-video-tools/package.json
index 76459d81..5495b3f8 100644
--- a/packages/reference-video-tools/package.json
+++ b/packages/reference-video-tools/package.json
@@ -16,6 +16,7 @@
"@hypit/core": "workspace:*",
"@hypit/driver-node": "workspace:*",
"@hypit/estimate": "workspace:*",
+ "@hypit/gemini": "workspace:*",
"@hypit/hyperframes": "workspace:*",
"@hypit/markup": "workspace:*",
"@hypit/media": "workspace:*",
diff --git a/packages/reference-video-tools/src/tools.ts b/packages/reference-video-tools/src/tools.ts
index 50bebd8c..02e6f1ee 100644
--- a/packages/reference-video-tools/src/tools.ts
+++ b/packages/reference-video-tools/src/tools.ts
@@ -22,7 +22,7 @@ import { describeSchema } from "./contract.js";
import { videoCliDistribution } from "@hypit/video-cli";
import { createHypiHubGeminiGenerator } from "@hypit/provider-hypihub";
import { createVertexGeminiGenerator } from "@hypit/provider-vertex";
-import type { VertexGeminiPart } from "@hypit/provider-vertex";
+import type { GeminiInlinePart } from "@hypit/gemini";
import { authorSource, invokedFrom, nearestPackageRoot, referenceRoot, referenceWords, renderElement, renderPreviews, spokenRange, standInSidecarPath, tokenWindow } from "./authoring.js";
import type { RenderElementInput, RenderPreviewsInput, SpokenRange, StandInFocus, StandInSidecar } from "./authoring.js";
@@ -300,7 +300,7 @@ type ToolOptions = {
readonly retryDelayMs?: number;
readonly generate?: GenerateText;
};
-export type GenerateText = (input: { readonly parts: readonly VertexGeminiPart[]; readonly instruction: string }) => Promise<string>;
+export type GenerateText = (input: { readonly parts: readonly GeminiInlinePart[]; readonly instruction: string }) => Promise<string>;
type ObservationTask = { readonly key: string; readonly request: Request };
function routeStateResult(state: RouteState | undefined): Record<string, unknown> {
@@ -850,12 +850,12 @@ const MIME_TYPES: Readonly<Record<string, string>> = {
// The Vertex path cannot upload a file and reference it later, so the bytes travel with every
// request. One shot's clip is asked about by its own picture, type and sound observations and by the
// boundary on each side, so reading and encoding it once per invocation is worth the memory.
-function mediaParts(): (path: string) => Promise<VertexGeminiPart> {
- const encoded = new Map<string, Promise<VertexGeminiPart>>();
+function mediaParts(): (path: string) => Promise<GeminiInlinePart> {
+ const encoded = new Map<string, Promise<GeminiInlinePart>>();
return (path) => {
const held = encoded.get(path);
if (held !== undefined) return held;
- const part = (async (): Promise<VertexGeminiPart> => {
+ const part = (async (): Promise<GeminiInlinePart> => {
const bytes = await readBytes(path);
const extension = path.toLowerCase().split(".").pop() ?? "";
const mimeType = MIME_TYPES[extension];
@@ -881,7 +881,7 @@ function permanent(error: unknown): boolean {
return /invalid[_ ]argument|permission[_ ]denied|unauthenticated|not[_ ]found|failed[_ ]precondition/iu.test(message(error));
}
-async function callSafely(retryDelayMs: number, generate: GenerateText, parts: readonly VertexGeminiPart[], instruction: string): Promise<Observation> {
+async function callSafely(retryDelayMs: number, generate: GenerateText, parts: readonly GeminiInlinePart[], instruction: string): Promise<Observation> {
let last: unknown;
for (let attempt = 1; attempt <= 6; attempt += 1) {
try { return observation("complete", await generate({ parts, instruction })); }
@@ -1461,7 +1461,7 @@ export function createReferenceVideoTools(options: ToolOptions = {}): ReferenceV
return {
pending: [],
ask: async (_key, { media, prompt, instruction }) => {
- const parts: VertexGeminiPart[] = [];
+ const parts: GeminiInlinePart[] = [];
for (const path of media) parts.push(await mediaPart(path));
parts.push({ text: `${prompt}\n\n${WATERMARK_RULE}` });
return await callSafely(retryDelayMs, generate, parts, instruction);
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index c2f8a6f2..657325dd 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -72,6 +72,9 @@ importers:
'@hypit/fonts-open':
specifier: workspace:*
version: link:packages/fonts-open
+ '@hypit/gemini':
+ specifier: workspace:*
+ version: link:packages/gemini
'@hypit/generation':
specifier: workspace:*
version: link:packages/generation
@@ -1313,6 +1316,27 @@ importers:
specifier: 0.2.0
version: 0.2.0
+ packages/gemini:
+ dependencies:
+ '@hypit/artifact':
+ specifier: workspace:*
+ version: link:../artifact
+ '@hypit/component-kit':
+ specifier: workspace:*
+ version: link:../component-kit
+ '@hypit/elaborator':
+ specifier: workspace:*
+ version: link:../elaborator
+ '@hypit/markup':
+ specifier: workspace:*
+ version: link:../markup
+ '@hypit/protocol':
+ specifier: workspace:*
+ version: link:../protocol
+ '@hypit/text':
+ specifier: workspace:*
+ version: link:../text
+
packages/generation:
dependencies:
'@hypit/artifact':
@@ -1948,6 +1972,9 @@ importers:
'@hypit/endpoint-kit':
specifier: workspace:*
version: link:../endpoint-kit
+ '@hypit/gemini':
+ specifier: workspace:*
+ version: link:../gemini
'@hypit/generation':
specifier: workspace:*
version: link:../generation
@@ -1960,6 +1987,9 @@ importers:
'@hypit/runtime-kit':
specifier: workspace:*
version: link:../runtime-kit
+ '@hypit/text':
+ specifier: workspace:*
+ version: link:../text
devDependencies:
'@hypit/driver-node':
specifier: workspace:*
@@ -2163,6 +2193,28 @@ importers:
'@google/genai':
specifier: 1.52.0
version: 1.52.0
+ '@hypit/endpoint-kit':
+ specifier: workspace:*
+ version: link:../endpoint-kit
+ '@hypit/gemini':
+ specifier: workspace:*
+ version: link:../gemini
+ '@hypit/protocol':
+ specifier: workspace:*
+ version: link:../protocol
+ '@hypit/runtime':
+ specifier: workspace:*
+ version: link:../runtime
+ '@hypit/runtime-kit':
+ specifier: workspace:*
+ version: link:../runtime-kit
+ '@hypit/text':
+ specifier: workspace:*
+ version: link:../text
+ devDependencies:
+ '@hypit/driver-node':
+ specifier: workspace:*
+ version: link:../driver-node
packages/provider-whisperx-local:
dependencies:
@@ -2313,6 +2365,9 @@ importers:
'@hypit/estimate':
specifier: workspace:*
version: link:../estimate
+ '@hypit/gemini':
+ specifier: workspace:*
+ version: link:../gemini
'@hypit/hyperframes':
specifier: workspace:*
version: link:../hyperframes
diff --git a/projects/hypihub-provider-smoke/build.svrun b/projects/hypihub-provider-smoke/build.svrun
index 2e8065aa..ba4cfb5e 100644
--- a/projects/hypihub-provider-smoke/build.svrun
+++ b/projects/hypihub-provider-smoke/build.svrun
@@ -6,19 +6,27 @@
<target output="gpt-edit.image"/>
<target output="nano-text.image"/>
<target output="nano-edit.image"/>
- <target output="nano2-text.image"/>
- <target output="nano2-edit.image"/>
<target output="seedream-text.image"/>
<target output="seedream-edit.image"/>
<target output="seedance-text.video"/>
<target output="seedance-image-audio.video"/>
+ <target output="seedance-frame.video"/>
+ <target output="seedance-video-reference.video"/>
<target output="seedance-standard-text.video"/>
<target output="seedance-fast-text.video"/>
<target output="seedance-v25-text.video"/>
<target output="minimax-text.video"/>
<target output="minimax-reference.video"/>
+ <target output="minimax-frame.video"/>
+ <target output="minimax-video-reference.video"/>
<target output="grok-text.video"/>
<target output="grok-image.video"/>
+ <target output="gemini-flash-text.text"/>
+ <target output="gemini-flash-image.text"/>
+ <target output="gemini-flash-video.text"/>
+ <target output="gemini-pro-text.text"/>
+ <target output="gemini-pro-image.text"/>
+ <target output="gemini-pro-video.text"/>
<target output="mimo-voice.audio"/>
<target output="mimo-preset.audio"/>
<target output="mimo-clone.audio"/>
diff --git a/projects/hypihub-provider-smoke/gemini-media.svrun b/projects/hypihub-provider-smoke/gemini-media.svrun
new file mode 100644
index 00000000..d62a3d1c
--- /dev/null
+++ b/projects/hypihub-provider-smoke/gemini-media.svrun
@@ -0,0 +1,9 @@
+<?svml using="@hypit/run-markup@1"?>
+
+<svrun version="1">
+ <author source="./main.svml"/>
+ <target output="gemini-flash-image.text"/>
+ <target output="gemini-flash-video.text"/>
+ <target output="gemini-pro-image.text"/>
+ <target output="gemini-pro-video.text"/>
+</svrun>
diff --git a/projects/hypihub-provider-smoke/gemini.svrun b/projects/hypihub-provider-smoke/gemini.svrun
new file mode 100644
index 00000000..777e816b
--- /dev/null
+++ b/projects/hypihub-provider-smoke/gemini.svrun
@@ -0,0 +1,11 @@
+<?svml using="@hypit/run-markup@1"?>
+
+<svrun version="1">
+ <author source="./main.svml"/>
+ <target output="gemini-flash-text.text"/>
+ <target output="gemini-flash-image.text"/>
+ <target output="gemini-flash-video.text"/>
+ <target output="gemini-pro-text.text"/>
+ <target output="gemini-pro-image.text"/>
+ <target output="gemini-pro-video.text"/>
+</svrun>
diff --git a/projects/hypihub-provider-smoke/hypit.runtime.json b/projects/hypihub-provider-smoke/hypit.runtime.json
index 1f814f8a..31f36a9a 100644
--- a/projects/hypihub-provider-smoke/hypit.runtime.json
+++ b/projects/hypihub-provider-smoke/hypit.runtime.json
@@ -3,17 +3,17 @@
"runtime": {
"use": "@hypit/runtime-local",
"config": {
- "dataRoot": ".hypit/runtimes/local-v4",
+ "dataRoot": ".hypit/runtimes/local",
"artifacts": { "use": "@hypit/artifact-store-fs", "config": { "path": "artifacts" } },
"credentials": { "env": { "use": "@hypit/credential-store-env" } },
"endpoints": {
"hypihub.smoke": {
"use": "@hypit/provider-hypihub",
- "pool": "hypihub.smoke-v4",
+ "pool": "hypihub.smoke",
"config": {
"baseUrl": "https://hypit.ai/v1",
"apiKey": { "store": "env", "key": "HYPIHUB_API_KEY" },
- "defaultConcurrency": 1,
+ "defaultConcurrency": 8,
"pollIntervalMs": 5000,
"requestTimeoutMs": 120000,
"audio": false
diff --git a/projects/hypihub-provider-smoke/images.svrun b/projects/hypihub-provider-smoke/images.svrun
index 6b4a7bdc..607b1992 100644
--- a/projects/hypihub-provider-smoke/images.svrun
+++ b/projects/hypihub-provider-smoke/images.svrun
@@ -6,8 +6,6 @@
<target output="gpt-edit.image"/>
<target output="nano-text.image"/>
<target output="nano-edit.image"/>
- <target output="nano2-text.image"/>
- <target output="nano2-edit.image"/>
<target output="seedream-text.image"/>
<target output="seedream-edit.image"/>
</svrun>
diff --git a/projects/hypihub-provider-smoke/main.svml b/projects/hypihub-provider-smoke/main.svml
index 20e8369b..7d212ef2 100644
--- a/projects/hypihub-provider-smoke/main.svml
+++ b/projects/hypihub-provider-smoke/main.svml
@@ -9,6 +9,7 @@
<import as="seedance" from="@hypit/seedance@1"/>
<import as="h3" from="@hypit/minimax-h3@1"/>
<import as="grok" from="@hypit/grok-imagine@1"/>
+ <import as="gemini" from="@hypit/gemini@1"/>
<import as="mimo" from="@hypit/mimo-tts@1"/>
<import as="recipes" source="./recipes.svs"/>
@@ -16,7 +17,12 @@
<text:Value id="edit-prompt">Keep the paper boat unchanged and make the studio background warm orange, no text.</text:Value>
<text:Value id="motion-prompt">The paper boat glides slowly across the blue table in one continuous locked-camera shot.</text:Value>
<text:Value id="speech">The red paper boat moves across the blue table.</text:Value>
+ <text:Value id="gemini-instruction">Answer with a single short factual sentence.</text:Value>
+ <text:Value id="gemini-text-prompt">Reply with exactly: provider text works</text:Value>
+ <text:Value id="gemini-image-prompt">Name the main object and its color in this image.</text:Value>
+ <text:Value id="gemini-video-prompt">Briefly describe the visible motion in this video.</text:Value>
<asset:Image id="reference-image" src="./assets/reference.png"/>
+ <asset:Video id="reference-video" src="./assets/reference.mp4"/>
<asset:Audio id="reference-audio" src="./assets/reference.wav"/>
<gpt:Image id="gpt-text" prompt={prompt} aspect-ratio="1:1" resolution="1K"/>
@@ -43,6 +49,10 @@
<seedance:Reference image={reference-image}/>
<seedance:Reference audio={reference-audio}/>
</seedance:ReferenceVideo>
+ <seedance:FrameVideo id="seedance-frame" model="mini" prompt={motion-prompt} duration="5" resolution="480p" aspect-ratio="16:9" generate-audio="false" first-frame={reference-image}/>
+ <seedance:ReferenceVideo id="seedance-video-reference" model="mini" prompt={motion-prompt} duration="5" resolution="480p" aspect-ratio="16:9" generate-audio="false">
+ <seedance:Reference video={reference-video}/>
+ </seedance:ReferenceVideo>
<seedance:TextVideo id="seedance-standard-text" model="standard" prompt={motion-prompt} duration="5" resolution="480p" aspect-ratio="16:9" generate-audio="false"/>
<seedance:TextVideo id="seedance-fast-text" model="fast" prompt={motion-prompt} duration="5" resolution="480p" aspect-ratio="16:9" generate-audio="false"/>
<seedance:TextVideo id="seedance-v25-text" model="2.5" prompt={motion-prompt} duration="5" resolution="720p" aspect-ratio="16:9" generate-audio="false"/>
@@ -51,12 +61,31 @@
<h3:ReferenceVideo id="minimax-reference" prompt={motion-prompt} duration="6" resolution="768P" aspect-ratio="16:9">
<h3:Reference image={reference-image}/>
</h3:ReferenceVideo>
+ <h3:FrameVideo id="minimax-frame" prompt={motion-prompt} duration="6" resolution="768P" first-frame={reference-image}/>
+ <h3:ReferenceVideo id="minimax-video-reference" prompt={motion-prompt} duration="6" resolution="768P" aspect-ratio="16:9">
+ <h3:Reference video={reference-video}/>
+ </h3:ReferenceVideo>
<grok:PreviewVideo id="grok-text" prompt={motion-prompt} duration="6" aspect-ratio="16:9" resolution="480p"/>
<grok:Video id="grok-image" prompt={motion-prompt} duration="6" aspect-ratio="16:9" resolution="480p">
<grok:Reference image={reference-image}/>
</grok:Video>
+ <gemini:Generate id="gemini-flash-text" model="gemini-3.7-flash" instruction={gemini-instruction} prompt={gemini-text-prompt}/>
+ <gemini:Generate id="gemini-flash-image" model="gemini-3.7-flash" instruction={gemini-instruction} prompt={gemini-image-prompt}>
+ <gemini:Reference media={reference-image}/>
+ </gemini:Generate>
+ <gemini:Generate id="gemini-flash-video" model="gemini-3.7-flash" instruction={gemini-instruction} prompt={gemini-video-prompt}>
+ <gemini:Reference media={reference-video}/>
+ </gemini:Generate>
+ <gemini:Generate id="gemini-pro-text" model="gemini-3.1-pro-preview" instruction={gemini-instruction} prompt={gemini-text-prompt}/>
+ <gemini:Generate id="gemini-pro-image" model="gemini-3.1-pro-preview" instruction={gemini-instruction} prompt={gemini-image-prompt}>
+ <gemini:Reference media={reference-image}/>
+ </gemini:Generate>
+ <gemini:Generate id="gemini-pro-video" model="gemini-3.1-pro-preview" instruction={gemini-instruction} prompt={gemini-video-prompt}>
+ <gemini:Reference media={reference-video}/>
+ </gemini:Generate>
+
<mimo:VoiceDesign id="mimo-voice" speech={speech}>A clear, warm narrator with a lightly playful delivery.</mimo:VoiceDesign>
<mimo:Preset id="mimo-preset" speech={speech} voice="Chloe"/>
<mimo:VoiceClone id="mimo-clone" speech={speech} sample={reference-audio}/>
diff --git a/projects/hypihub-provider-smoke/unavailable.svrun b/projects/hypihub-provider-smoke/unavailable.svrun
new file mode 100644
index 00000000..20e0c35e
--- /dev/null
+++ b/projects/hypihub-provider-smoke/unavailable.svrun
@@ -0,0 +1,7 @@
+<?svml using="@hypit/run-markup@1"?>
+
+<svrun version="1">
+ <author source="./main.svml"/>
+ <target output="nano2-text.image"/>
+ <target output="nano2-edit.image"/>
+</svrun>
diff --git a/projects/hypihub-provider-smoke/video-references.svrun b/projects/hypihub-provider-smoke/video-references.svrun
new file mode 100644
index 00000000..4ad8ed32
--- /dev/null
+++ b/projects/hypihub-provider-smoke/video-references.svrun
@@ -0,0 +1,9 @@
+<?svml using="@hypit/run-markup@1"?>
+
+<svrun version="1">
+ <author source="./main.svml"/>
+ <target output="seedance-frame.video"/>
+ <target output="seedance-video-reference.video"/>
+ <target output="minimax-frame.video"/>
+ <target output="minimax-video-reference.video"/>
+</svrun>
diff --git a/projects/hypihub-provider-smoke/videos.svrun b/projects/hypihub-provider-smoke/videos.svrun
index a7b1afb1..e3850853 100644
--- a/projects/hypihub-provider-smoke/videos.svrun
+++ b/projects/hypihub-provider-smoke/videos.svrun
@@ -4,11 +4,15 @@
<author source="./main.svml"/>
<target output="seedance-text.video"/>
<target output="seedance-image-audio.video"/>
+ <target output="seedance-frame.video"/>
+ <target output="seedance-video-reference.video"/>
<target output="seedance-standard-text.video"/>
<target output="seedance-fast-text.video"/>
<target output="seedance-v25-text.video"/>
<target output="minimax-text.video"/>
<target output="minimax-reference.video"/>
+ <target output="minimax-frame.video"/>
+ <target output="minimax-video-reference.video"/>
<target output="grok-text.video"/>
<target output="grok-image.video"/>
</svrun>