Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

merge: integrate upstream speech models and CJK captions

kashorin5ae1470fd9ac
diff --cc examples/podcast/reference.svml
index f43f60e6,c9dfaa53..b9b24971
--- a/examples/podcast/reference.svml
+++ b/examples/podcast/reference.svml
@@@ -2,9 -2,8 +2,9 @@@
<svml>
<import from="@hypit/script@1"/>
+ <import as="sound" from="@hypit/sound@1"/>
<import as="asset" from="@hypit/media@1"/>
- <import as="mimo" from="@hypit/mimo-speech@1"/>
+ <import as="fish" from="@hypit/fishaudio-speech@1"/>
<import as="text" from="@hypit/text@1"/>
<import as="gpt" from="@hypit/gpt-image@1"/>
<import as="seedance" from="@hypit/seedance@1"/>
diff --cc examples/ranking-football/reference.svml
index 660fec11,1f82693e..63464cdd
--- a/examples/ranking-football/reference.svml
+++ b/examples/ranking-football/reference.svml
@@@ -2,10 -2,9 +2,10 @@@
<svml>
<import from="@hypit/script@1"/>
+ <import as="sound" from="@hypit/sound@1"/>
<import as="text" from="@hypit/text@1"/>
<import as="asset" from="@hypit/media@1"/>
- <import as="mimo" from="@hypit/mimo-speech@1"/>
+ <import as="fish" from="@hypit/fishaudio-speech@1"/>
<import as="gpt" from="@hypit/gpt-image@1"/>
<import as="seedance" from="@hypit/seedance@1"/>
<import as="pipeline" from="@hypit/media-pipeline@1"/>
diff --cc packages/caption-fine/README.md
index 57330b64,585a32cc..e63bc230
--- a/packages/caption-fine/README.md
+++ b/packages/caption-fine/README.md
@@@ -84,7 -79,7 +84,9 @@@ caption.primary
The same Caption pipeline serves English and Chinese. Script emits English lexical words and
individual Han characters as Display Words; punctuation stays with its display word. Fine uses
--those units for timing and active Paint, while authored Cues remain complete reading phrases.
++those units for timing, while authored Cues remain complete reading phrases. Step highlighting groups
++adjacent units that form a word in the displayed Cue, using `Intl.Segmenter`; the underlying Script
++units and measured times stay intact. This grouping affects emphasis, not authored Cue boundaries.
Dual Text retains its complete alignment unit even when it displays or speaks several words.
`word-gap` applies between Latin words and at Chinese/Latin boundaries. Adjacent Han characters and
@@@ -101,18 -96,18 +103,18 @@@ font fallbacks supply the required glyp
Both count limits are omitted above so the example flows by available width. For a compact
single-line treatment, author coherent Cues and choose a font, size and width that fit those Cues.
--`karaoke: off` keeps a complete Cue steady; `current` and `trail` follow its timed units, which are
--normally individual characters in Chinese.
++`karaoke: off` keeps a complete Cue steady; `current` and `trail` follow speech timing, with step
++emphasis sharing one activation window across each detected word's units.
| Effect | Recipe | Visible behavior |
| --- | --- | --- |
--| Whole-character emphasis | `karaoke: current; karaoke-transition: step` | The complete current unit takes the active Paint at its start, returning to base Paint at its end. |
--| Accumulating emphasis | `karaoke: trail; karaoke-transition: step` | Each complete unit takes the active Paint at its start and keeps it for the rest of the Cue. |
++| Whole-word emphasis | `karaoke: current; karaoke-transition: step` | The complete current word takes the active Paint at its first unit's start, returning to base Paint at the group's end. |
++| Accumulating emphasis | `karaoke: trail; karaoke-transition: step` | Each complete word takes the active Paint at its first unit's start and keeps it for the rest of the Cue. |
| Sweeping fill | `karaoke: trail; karaoke-transition: wipe` | Paint sweeps inside each unit's glyphs during that unit's own time window. |
| Spoken-unit appearance | `atom-reveal: on-start` | Each complete unit appears at its start. |
| Typing appearance | `atom-reveal: typewriter` | Whole graphemes appear in sequence within each unit, starting with its first grapheme at the unit's start. A normal Han unit appears as a complete character. |
--`step` is the default Karaoke transition. Choose it for ordinary Chinese character highlighting;
++`step` is the default Karaoke transition. Choose it for ordinary whole-word highlighting;
`wipe` deliberately produces partial glyph fills. Cue width and duration do not supply a substitute
clock for either mode. `atom-reveal` controls text appearing, independently of Karaoke recoloring
already visible text. A Dual Text span shares one timing unit, so its display words activate together
diff --cc packages/elevenlabs-speech/src/surface.ts
index 00000000,3581ce30..03ab681c
mode 000000,100644..100644
--- a/packages/elevenlabs-speech/src/surface.ts
+++ b/packages/elevenlabs-speech/src/surface.ts
@@@ -1,0 -1,168 +1,167 @@@
+ import { artifactTypes } from "@hypit/artifact";
+ import { generationPort, sealGenerationMediaBinding } from "@hypit/generation";
+ import type { GenerationMediaPort } from "@hypit/generation";
+ import { exactModelMediaInputNames, exactModelTextInputName } from "@hypit/model-kit";
+ import type { ExactModelEndpoint, ExactModelMediaInput, ExactModelTextInput } from "@hypit/model-kit";
+ import type { CanonicalValue } from "@hypit/protocol";
+ import { textTypes, verifyText } from "@hypit/text";
+ import type {
+ MarkupAttributeValue,
+ StructuredElement,
+ StructuredSurfaceHandler,
+ SurfaceResolvedReference,
+ } from "@hypit/markup";
+
+ import { createElevenLabsSpeechAudioFragment } from "./fragment.js";
+ import { elevenLabsSpeechEndpoints, sealElevenLabsSpeechRequestDraft } from "./index.js";
+
+ function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+ }
+
+ function attributes(element: StructuredElement, required: readonly string[]): void {
+ const allowed = new Set(required);
+ const unknown = Object.keys(element.attributes).filter((name) => !allowed.has(name));
+ if (unknown.length > 0 || required.some((name) => element.attributes[name] === undefined)) {
+ throw new Error(`${element.name} requires ${required.join(", ")}`);
+ }
+ }
+
+ function stringAttribute(element: StructuredElement, name: string): string {
+ const value = element.attributes[name];
+ if (typeof value !== "string" || value.trim().length === 0) {
+ throw new Error(`${element.name}.${name} must be a non-empty string`);
+ }
+ return value.trim();
+ }
+
+ function referencePath(element: StructuredElement, name: string): string {
+ const value: MarkupAttributeValue | undefined = element.attributes[name];
+ if (typeof value !== "object" || value.kind !== "reference" || value.path.length === 0) {
+ throw new Error(`${element.name}.${name} must be a whole-value reference`);
+ }
+ return value.path;
+ }
+
+ function resolved(
+ element: StructuredElement,
+ name: string,
+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,
+ ): SurfaceResolvedReference {
+ const path = referencePath(element, name);
+ const value = resolveReference(path);
+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${path}`);
+ return value;
+ }
+
+ function speechText(reference: SurfaceResolvedReference, subject: string): SurfaceResolvedReference {
+ if (!sameType(reference.type, textTypes.text)) throw new Error(`${subject} must reference Text`);
+ const value = reference.record?.value;
+ if (value !== undefined) {
+ if (value.kind !== "inline") throw new Error(`${subject} has an invalid authored Text value`);
+ verifyText(value.value);
+ }
+ return reference;
+ }
+
+
+ function body(element: StructuredElement, required: boolean): string | undefined {
+ if (element.children.some((child) => child.kind === "element")) {
+ throw new Error(`${element.name} accepts instruction text only`);
+ }
+ const lines = element.children.map((child) => child.kind === "text" ? child.value : "").join("")
+ .replaceAll("\r\n", "\n").split("\n");
+ while (lines[0]?.trim() === "") lines.shift();
+ while (lines.at(-1)?.trim() === "") lines.pop();
+ const indents = lines.filter((line) => line.trim()).map((line) => /^\s*/u.exec(line)?.[0].length ?? 0);
+ const indent = indents.length === 0 ? 0 : Math.min(...indents);
+ const value = lines.map((line) => line.slice(indent).trimEnd()).join("\n").trim();
+ if (value.length === 0) {
+ if (required) throw new Error(`${element.name} requires a voice description in its body`);
+ return undefined;
+ }
+ return value;
+ }
+
+ function output(
+ element: StructuredElement,
+ endpoint: ExactModelEndpoint,
+ draft: ReturnType<typeof sealElevenLabsSpeechRequestDraft>,
+ outputName: "reference" | "audio",
+ text: readonly { readonly input: ExactModelTextInput; readonly source: SurfaceResolvedReference }[],
+ media: readonly { readonly input: ExactModelMediaInput; readonly source: SurfaceResolvedReference }[] = [],
+ ) {
+ const id = stringAttribute(element, "id");
+ const draftId = `${id}.draft`;
+ const records: Array<{
+ readonly id: string;
+ readonly type: SurfaceResolvedReference["type"];
+ readonly value: { readonly kind: "inline"; readonly value: CanonicalValue };
+ readonly range: StructuredElement["range"];
+ }> = [{
+ id: draftId,
+ type: endpoint.draftType,
+ value: { kind: "inline", value: draft as unknown as CanonicalValue },
+ range: element.range,
+ }];
+ const inputs: Record<string, SurfaceResolvedReference["ref"] | { readonly kind: "record"; readonly id: string }> = {
+ draft: { kind: "record", id: draftId },
+ };
+ for (const item of text) inputs[exactModelTextInputName(item.input.name)] = item.source.ref;
+ for (const item of media) {
+ const binding = endpoint.mediaBindings[item.input.port];
+ if (binding === undefined) throw new Error(`${endpoint.ports.model} has no media port ${item.input.port}`);
+ const port = generationPort(endpoint.ports, item.input.port);
+ if (port.value.kind !== "media") throw new Error(`${endpoint.ports.model} port ${item.input.port} is not media`);
+ const names = exactModelMediaInputNames(item.input.name);
+ const bindingId = `${id}.${item.input.name}.binding`;
+ records.push({
+ id: bindingId,
+ type: binding.type,
+ value: {
+ kind: "inline",
+ value: sealGenerationMediaBinding(port as GenerationMediaPort, { role: "audio" }) as unknown as CanonicalValue,
+ },
+ range: element.range,
+ });
+ inputs[names.binding] = { kind: "record", id: bindingId };
+ inputs[names.artifact] = item.source.ref;
+ }
+ const fragment = createElevenLabsSpeechAudioFragment(
+ endpoint,
+ media.map((item) => item.input),
+ text.map((item) => item.input),
+ );
+ return {
+ records,
+ components: [{
+ id,
+ fragment: fragment.id,
+ inputs,
+ outputs: { audio: `${id}.${outputName}` },
+ range: element.range,
+ }],
+ fragments: [fragment],
+ };
+ }
+
+ function speechInput(
+ element: StructuredElement,
+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,
+ ) {
+ return {
+ input: { name: "speech", port: "text" },
+ source: speechText(resolved(element, "speech", resolveReference), `${element.name}.speech`),
+ } as const;
+ }
+
+ export const decodeElevenLabsVoiceDesignSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ attributes(element, ["id", "speech"]);
+ return output(
+ element,
+ elevenLabsSpeechEndpoints.voiceDesign,
+ sealElevenLabsSpeechRequestDraft("eleven_ttv_v3", { voiceDescription: [body(element, true)!] }),
+ "reference",
+ [speechInput(element, resolveReference)],
+ );
+ };
-
diff --cc packages/provider-hypihub/README.md
index db58dbfe,f0a30fa4..0b7fa5dc
--- a/packages/provider-hypihub/README.md
+++ b/packages/provider-hypihub/README.md
@@@ -108,13 -91,23 +108,23 @@@ catalog to verify configured capabiliti
declares HypiHub's public pricing page, `https://hypit.ai/commercial/pricing/`, as its price source.
For each selected Need, `readPricing` resolves the corresponding HypiHub model and returns the service's
authenticated `GET /v1/pricing?model=<model>` response unchanged together with that URL. It covers
- generation, alignment, Voice Design, and Voice Clone through the same mechanism;
+ generation, alignment, Voice Design and Voice Clone through the same mechanism;
the Provider does not maintain a second list of billing formulas or calculate a request total.
- HypiHub declares MiMo Voice Design and Voice Clone together with every other capability it serves; it
- never hides one. Voice Design produces an accepted voice-reference Resource, and Voice Clone uses
- that reference to produce independent speech. Hypit does not expose MiMo preset voices. When another
- selected Endpoint offers the same capability (such as local WhisperX or a project-owned speech
-HypiHub declares every speech capability it serves together with every other capability; it never
-hides one. Voice Design produces an accepted voice-reference Resource, and Voice Clone uses that
++The Provider declares its implemented speech capabilities alongside image, video and alignment.
++Voice Design produces an accepted voice-reference Resource, and Voice Clone uses that
+ reference to produce independent speech. All of them use `POST /v1/audio/speech`:
+
+ | Package | Capability | HypiHub model | Request fields |
+ | --- | --- | --- | --- |
+ | `@hypit/mimo-speech` | `mimo-v2.5-tts-voicedesign` | `mimo-v2.5-tts-voicedesign` | `input`, `voice_description` |
+ | `@hypit/mimo-speech` | `mimo-v2.5-tts-voiceclone` | `mimo-v2.5-tts-voiceclone` | `input`, `reference_audio`, optional `prompt` |
+ | `@hypit/fishaudio-speech` | `voice-design-1` | `fishaudio/voice-design-1` | `input`, `voice_description` |
+ | `@hypit/fishaudio-speech` | `voice-clone` | `fishaudio/voice-clone` | `input`, `reference_audio`, optional `prompt`, constant `voice_description` title |
+ | `@hypit/elevenlabs-speech` | `eleven_ttv_v3` | `eleven_ttv_v3` | `input`, `voice_description` |
+
-Each returned preview becomes one member of the audio set. Hypit does not expose preset voices. When
-another selected Endpoint offers the same capability (a local WhisperX or the official MiMo
++Each returned preview becomes one member of the audio set. These Model packages do not expose preset
++voices. When another selected Endpoint offers the same capability (such as local WhisperX or a project-owned speech
Provider), the Runtime Profile's `bindings` say which Endpoint serves it.
Execution policy remains local to this Provider:
diff --cc pnpm-lock.yaml
index 38ef8c75,cc4a30d6..a995dc77
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@@ -307,6 -307,6 +313,24 @@@ importers
specifier: ^1.6.4
++ examples/complex-explainer/packages/launch-scenes: {}
++
++ examples/complex-explainer/packages/opening-system:
++ dependencies:
++ '@explainer/visual-language':
++ specifier: file:../visual-language
++ version: link:../visual-language
++
++ examples/complex-explainer/packages/single-line-captions: {}
++
++ examples/complex-explainer/packages/visual-language: {}
++
++ examples/complex-explainer/packages/web-scenes:
++ dependencies:
++ '@explainer/visual-language':
++ specifier: file:../visual-language
++ version: link:../visual-language
++
examples/minimal-author-package/packages/example-component:
devDependencies:
'@hypit/hypit':
@@@ -1129,10 -1093,31 +1174,31 @@@
'@hypit/studio-adapter':
specifier: workspace:*
version: link:../studio-adapter
+ '@hypit/timeline':
+ specifier: workspace:*
+ version: link:../timeline
+ packages/fishaudio-speech:
+ dependencies:
+ '@hypit/artifact':
+ specifier: workspace:*
+ version: link:../artifact
+ '@hypit/generation':
+ specifier: workspace:*
+ version: link:../generation
+ '@hypit/markup':
+ specifier: workspace:*
+ version: link:../markup
+ '@hypit/model-kit':
+ specifier: workspace:*
+ version: link:../model-kit
+ '@hypit/protocol':
+ specifier: workspace:*
+ version: link:../protocol
+ '@hypit/text':
+ specifier: workspace:*
+ version: link:../text
+
packages/fonts-open:
dependencies:
'@hypit/markup':
diff --cc skills/hypit/references/playbooks/craft/captions.md
index 8055821e,a9dfa9b2..78134988
--- a/skills/hypit/references/playbooks/craft/captions.md
+++ b/skills/hypit/references/playbooks/craft/captions.md
@@@ -114,7 -114,7 +114,7 @@@ or quantity and unit together when sepa
### Language changes the reading unit
English commonly uses a word as its smallest timed display unit; Chinese uses individual Han
--characters. That gives Chinese precise character highlighting, while a Cue still holds a meaningful
++characters. That gives Chinese precise character timing, while a Cue still holds a meaningful
phrase. A Chinese Cue can comfortably contain more characters than an English Cue contains words.
Choose its length from meaning, reading time and the space the actual font occupies. A fixed word
or character count cannot make that choice. Count the display side of Dual Text when judging fit.
@@@ -127,15 -127,15 +127,16 @@@
Fine already leaves adjacent Han characters together without English word gaps. Its `word-gap`
controls spaced boundaries, including Chinese/Latin transitions; `letter-spacing` adjusts tracking.
--The same timing supports a stable complete Cue, current-character color or a trail. For flowing
++The same timing supports a stable complete Cue, current-word color or a trail. For flowing
Chinese speech, a readable phrase with restrained emphasis often works better than a separate bounce
or reveal on every character. Choose the response for the performance's energy.
--For whole-character highlighting, use `karaoke: current` with `karaoke-transition: step`.
--Choose `trail` instead of `current` when the already spoken characters should stay highlighted.
--`step` activates the complete timed unit at its start; `wipe` is a different visual choice that
--sweeps inside its glyphs. Both follow each unit's own speech time, including uneven delivery and
--pauses. Making a Cue longer changes its reading group, not its character timing.
++For whole-word highlighting, use `karaoke: current` with `karaoke-transition: step`.
++Fine groups adjacent display units into words and lights their complete glyphs together from the
++first unit's time. Choose `trail` when already spoken words should stay highlighted. The grouping is
++a presentation choice; Script keeps its individual character times. `wipe` instead sweeps inside
++each unit's glyphs over that unit's speech window. Both use speech timing, including uneven delivery
++and pauses. Making a Cue longer changes its reading group, not the underlying character timing.
Text appearing and text changing color are separate choices. `atom-reveal: all` keeps the complete
Cue available to read while Karaoke supplies emphasis; `on-start` reveals whole spoken units, and