Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
merge: integrate upstream speech models and CJK captions
kashorin5ae1470fd9ac
diff --cc examples/podcast/reference.svmlindex f43f60e6,c9dfaa53..b9b24971--- a/examples/podcast/reference.svml+++ b/examples/podcast/reference.svml@@@ -2,9 -2,8 +2,9 @@@<svml><import from="@hypit/script@1"/>+ <import as="sound" from="@hypit/sound@1"/><import as="asset" from="@hypit/media@1"/>- <import as="mimo" from="@hypit/mimo-speech@1"/>+ <import as="fish" from="@hypit/fishaudio-speech@1"/><import as="text" from="@hypit/text@1"/><import as="gpt" from="@hypit/gpt-image@1"/><import as="seedance" from="@hypit/seedance@1"/>diff --cc examples/ranking-football/reference.svmlindex 660fec11,1f82693e..63464cdd--- a/examples/ranking-football/reference.svml+++ b/examples/ranking-football/reference.svml@@@ -2,10 -2,9 +2,10 @@@<svml><import from="@hypit/script@1"/>+ <import as="sound" from="@hypit/sound@1"/><import as="text" from="@hypit/text@1"/><import as="asset" from="@hypit/media@1"/>- <import as="mimo" from="@hypit/mimo-speech@1"/>+ <import as="fish" from="@hypit/fishaudio-speech@1"/><import as="gpt" from="@hypit/gpt-image@1"/><import as="seedance" from="@hypit/seedance@1"/><import as="pipeline" from="@hypit/media-pipeline@1"/>diff --cc packages/caption-fine/README.mdindex 57330b64,585a32cc..e63bc230--- a/packages/caption-fine/README.md+++ b/packages/caption-fine/README.md@@@ -84,7 -79,7 +84,9 @@@ caption.primaryThe same Caption pipeline serves English and Chinese. Script emits English lexical words andindividual Han characters as Display Words; punctuation stays with its display word. Fine uses--those units for timing and active Paint, while authored Cues remain complete reading phrases.++those units for timing, while authored Cues remain complete reading phrases. Step highlighting groups++adjacent units that form a word in the displayed Cue, using `Intl.Segmenter`; the underlying Script++units and measured times stay intact. This grouping affects emphasis, not authored Cue boundaries.Dual Text retains its complete alignment unit even when it displays or speaks several words.`word-gap` applies between Latin words and at Chinese/Latin boundaries. Adjacent Han characters and@@@ -101,18 -96,18 +103,18 @@@ font fallbacks supply the required glypBoth count limits are omitted above so the example flows by available width. For a compactsingle-line treatment, author coherent Cues and choose a font, size and width that fit those Cues.--`karaoke: off` keeps a complete Cue steady; `current` and `trail` follow its timed units, which are--normally individual characters in Chinese.++`karaoke: off` keeps a complete Cue steady; `current` and `trail` follow speech timing, with step++emphasis sharing one activation window across each detected word's units.| Effect | Recipe | Visible behavior || --- | --- | --- |--| Whole-character emphasis | `karaoke: current; karaoke-transition: step` | The complete current unit takes the active Paint at its start, returning to base Paint at its end. |--| Accumulating emphasis | `karaoke: trail; karaoke-transition: step` | Each complete unit takes the active Paint at its start and keeps it for the rest of the Cue. |++| Whole-word emphasis | `karaoke: current; karaoke-transition: step` | The complete current word takes the active Paint at its first unit's start, returning to base Paint at the group's end. |++| Accumulating emphasis | `karaoke: trail; karaoke-transition: step` | Each complete word takes the active Paint at its first unit's start and keeps it for the rest of the Cue. || Sweeping fill | `karaoke: trail; karaoke-transition: wipe` | Paint sweeps inside each unit's glyphs during that unit's own time window. || Spoken-unit appearance | `atom-reveal: on-start` | Each complete unit appears at its start. || Typing appearance | `atom-reveal: typewriter` | Whole graphemes appear in sequence within each unit, starting with its first grapheme at the unit's start. A normal Han unit appears as a complete character. |--`step` is the default Karaoke transition. Choose it for ordinary Chinese character highlighting;++`step` is the default Karaoke transition. Choose it for ordinary whole-word highlighting;`wipe` deliberately produces partial glyph fills. Cue width and duration do not supply a substituteclock for either mode. `atom-reveal` controls text appearing, independently of Karaoke recoloringalready visible text. A Dual Text span shares one timing unit, so its display words activate togetherdiff --cc packages/elevenlabs-speech/src/surface.tsindex 00000000,3581ce30..03ab681cmode 000000,100644..100644--- a/packages/elevenlabs-speech/src/surface.ts+++ b/packages/elevenlabs-speech/src/surface.ts@@@ -1,0 -1,168 +1,167 @@@+ import { artifactTypes } from "@hypit/artifact";+ import { generationPort, sealGenerationMediaBinding } from "@hypit/generation";+ import type { GenerationMediaPort } from "@hypit/generation";+ import { exactModelMediaInputNames, exactModelTextInputName } from "@hypit/model-kit";+ import type { ExactModelEndpoint, ExactModelMediaInput, ExactModelTextInput } from "@hypit/model-kit";+ import type { CanonicalValue } from "@hypit/protocol";+ import { textTypes, verifyText } from "@hypit/text";+ import type {+ MarkupAttributeValue,+ StructuredElement,+ StructuredSurfaceHandler,+ SurfaceResolvedReference,+ } from "@hypit/markup";++ import { createElevenLabsSpeechAudioFragment } from "./fragment.js";+ import { elevenLabsSpeechEndpoints, sealElevenLabsSpeechRequestDraft } from "./index.js";++ function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;+ }++ function attributes(element: StructuredElement, required: readonly string[]): void {+ const allowed = new Set(required);+ const unknown = Object.keys(element.attributes).filter((name) => !allowed.has(name));+ if (unknown.length > 0 || required.some((name) => element.attributes[name] === undefined)) {+ throw new Error(`${element.name} requires ${required.join(", ")}`);+ }+ }++ function stringAttribute(element: StructuredElement, name: string): string {+ const value = element.attributes[name];+ if (typeof value !== "string" || value.trim().length === 0) {+ throw new Error(`${element.name}.${name} must be a non-empty string`);+ }+ return value.trim();+ }++ function referencePath(element: StructuredElement, name: string): string {+ const value: MarkupAttributeValue | undefined = element.attributes[name];+ if (typeof value !== "object" || value.kind !== "reference" || value.path.length === 0) {+ throw new Error(`${element.name}.${name} must be a whole-value reference`);+ }+ return value.path;+ }++ function resolved(+ element: StructuredElement,+ name: string,+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,+ ): SurfaceResolvedReference {+ const path = referencePath(element, name);+ const value = resolveReference(path);+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${path}`);+ return value;+ }++ function speechText(reference: SurfaceResolvedReference, subject: string): SurfaceResolvedReference {+ if (!sameType(reference.type, textTypes.text)) throw new Error(`${subject} must reference Text`);+ const value = reference.record?.value;+ if (value !== undefined) {+ if (value.kind !== "inline") throw new Error(`${subject} has an invalid authored Text value`);+ verifyText(value.value);+ }+ return reference;+ }+++ function body(element: StructuredElement, required: boolean): string | undefined {+ if (element.children.some((child) => child.kind === "element")) {+ throw new Error(`${element.name} accepts instruction text only`);+ }+ const lines = element.children.map((child) => child.kind === "text" ? child.value : "").join("")+ .replaceAll("\r\n", "\n").split("\n");+ while (lines[0]?.trim() === "") lines.shift();+ while (lines.at(-1)?.trim() === "") lines.pop();+ const indents = lines.filter((line) => line.trim()).map((line) => /^\s*/u.exec(line)?.[0].length ?? 0);+ const indent = indents.length === 0 ? 0 : Math.min(...indents);+ const value = lines.map((line) => line.slice(indent).trimEnd()).join("\n").trim();+ if (value.length === 0) {+ if (required) throw new Error(`${element.name} requires a voice description in its body`);+ return undefined;+ }+ return value;+ }++ function output(+ element: StructuredElement,+ endpoint: ExactModelEndpoint,+ draft: ReturnType<typeof sealElevenLabsSpeechRequestDraft>,+ outputName: "reference" | "audio",+ text: readonly { readonly input: ExactModelTextInput; readonly source: SurfaceResolvedReference }[],+ media: readonly { readonly input: ExactModelMediaInput; readonly source: SurfaceResolvedReference }[] = [],+ ) {+ const id = stringAttribute(element, "id");+ const draftId = `${id}.draft`;+ const records: Array<{+ readonly id: string;+ readonly type: SurfaceResolvedReference["type"];+ readonly value: { readonly kind: "inline"; readonly value: CanonicalValue };+ readonly range: StructuredElement["range"];+ }> = [{+ id: draftId,+ type: endpoint.draftType,+ value: { kind: "inline", value: draft as unknown as CanonicalValue },+ range: element.range,+ }];+ const inputs: Record<string, SurfaceResolvedReference["ref"] | { readonly kind: "record"; readonly id: string }> = {+ draft: { kind: "record", id: draftId },+ };+ for (const item of text) inputs[exactModelTextInputName(item.input.name)] = item.source.ref;+ for (const item of media) {+ const binding = endpoint.mediaBindings[item.input.port];+ if (binding === undefined) throw new Error(`${endpoint.ports.model} has no media port ${item.input.port}`);+ const port = generationPort(endpoint.ports, item.input.port);+ if (port.value.kind !== "media") throw new Error(`${endpoint.ports.model} port ${item.input.port} is not media`);+ const names = exactModelMediaInputNames(item.input.name);+ const bindingId = `${id}.${item.input.name}.binding`;+ records.push({+ id: bindingId,+ type: binding.type,+ value: {+ kind: "inline",+ value: sealGenerationMediaBinding(port as GenerationMediaPort, { role: "audio" }) as unknown as CanonicalValue,+ },+ range: element.range,+ });+ inputs[names.binding] = { kind: "record", id: bindingId };+ inputs[names.artifact] = item.source.ref;+ }+ const fragment = createElevenLabsSpeechAudioFragment(+ endpoint,+ media.map((item) => item.input),+ text.map((item) => item.input),+ );+ return {+ records,+ components: [{+ id,+ fragment: fragment.id,+ inputs,+ outputs: { audio: `${id}.${outputName}` },+ range: element.range,+ }],+ fragments: [fragment],+ };+ }++ function speechInput(+ element: StructuredElement,+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,+ ) {+ return {+ input: { name: "speech", port: "text" },+ source: speechText(resolved(element, "speech", resolveReference), `${element.name}.speech`),+ } as const;+ }++ export const decodeElevenLabsVoiceDesignSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ attributes(element, ["id", "speech"]);+ return output(+ element,+ elevenLabsSpeechEndpoints.voiceDesign,+ sealElevenLabsSpeechRequestDraft("eleven_ttv_v3", { voiceDescription: [body(element, true)!] }),+ "reference",+ [speechInput(element, resolveReference)],+ );+ };-diff --cc packages/provider-hypihub/README.mdindex db58dbfe,f0a30fa4..0b7fa5dc--- a/packages/provider-hypihub/README.md+++ b/packages/provider-hypihub/README.md@@@ -108,13 -91,23 +108,23 @@@ catalog to verify configured capabilitideclares HypiHub's public pricing page, `https://hypit.ai/commercial/pricing/`, as its price source.For each selected Need, `readPricing` resolves the corresponding HypiHub model and returns the service'sauthenticated `GET /v1/pricing?model=<model>` response unchanged together with that URL. It covers- generation, alignment, Voice Design, and Voice Clone through the same mechanism;+ generation, alignment, Voice Design and Voice Clone through the same mechanism;the Provider does not maintain a second list of billing formulas or calculate a request total.- HypiHub declares MiMo Voice Design and Voice Clone together with every other capability it serves; it- never hides one. Voice Design produces an accepted voice-reference Resource, and Voice Clone uses- that reference to produce independent speech. Hypit does not expose MiMo preset voices. When another- selected Endpoint offers the same capability (such as local WhisperX or a project-owned speech-HypiHub declares every speech capability it serves together with every other capability; it never-hides one. Voice Design produces an accepted voice-reference Resource, and Voice Clone uses that++The Provider declares its implemented speech capabilities alongside image, video and alignment.++Voice Design produces an accepted voice-reference Resource, and Voice Clone uses that+ reference to produce independent speech. All of them use `POST /v1/audio/speech`:++ | Package | Capability | HypiHub model | Request fields |+ | --- | --- | --- | --- |+ | `@hypit/mimo-speech` | `mimo-v2.5-tts-voicedesign` | `mimo-v2.5-tts-voicedesign` | `input`, `voice_description` |+ | `@hypit/mimo-speech` | `mimo-v2.5-tts-voiceclone` | `mimo-v2.5-tts-voiceclone` | `input`, `reference_audio`, optional `prompt` |+ | `@hypit/fishaudio-speech` | `voice-design-1` | `fishaudio/voice-design-1` | `input`, `voice_description` |+ | `@hypit/fishaudio-speech` | `voice-clone` | `fishaudio/voice-clone` | `input`, `reference_audio`, optional `prompt`, constant `voice_description` title |+ | `@hypit/elevenlabs-speech` | `eleven_ttv_v3` | `eleven_ttv_v3` | `input`, `voice_description` |+-Each returned preview becomes one member of the audio set. Hypit does not expose preset voices. When-another selected Endpoint offers the same capability (a local WhisperX or the official MiMo++Each returned preview becomes one member of the audio set. These Model packages do not expose preset++voices. When another selected Endpoint offers the same capability (such as local WhisperX or a project-owned speechProvider), the Runtime Profile's `bindings` say which Endpoint serves it.Execution policy remains local to this Provider:diff --cc pnpm-lock.yamlindex 38ef8c75,cc4a30d6..a995dc77--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@@ -307,6 -307,6 +313,24 @@@ importersspecifier: ^1.6.4version: 1.6.4(@algolia/[email protected])(@types/[email protected])([email protected])([email protected])([email protected])([email protected])++ examples/complex-explainer/packages/launch-scenes: {}++++ examples/complex-explainer/packages/opening-system:++ dependencies:++ '@explainer/visual-language':++ specifier: file:../visual-language++ version: link:../visual-language++++ examples/complex-explainer/packages/single-line-captions: {}++++ examples/complex-explainer/packages/visual-language: {}++++ examples/complex-explainer/packages/web-scenes:++ dependencies:++ '@explainer/visual-language':++ specifier: file:../visual-language++ version: link:../visual-language++examples/minimal-author-package/packages/example-component:devDependencies:'@hypit/hypit':@@@ -1129,10 -1093,31 +1174,31 @@@'@hypit/studio-adapter':specifier: workspace:*version: link:../studio-adapter+ '@hypit/timeline':+ specifier: workspace:*+ version: link:../timeline+ packages/fishaudio-speech:+ dependencies:+ '@hypit/artifact':+ specifier: workspace:*+ version: link:../artifact+ '@hypit/generation':+ specifier: workspace:*+ version: link:../generation+ '@hypit/markup':+ specifier: workspace:*+ version: link:../markup+ '@hypit/model-kit':+ specifier: workspace:*+ version: link:../model-kit+ '@hypit/protocol':+ specifier: workspace:*+ version: link:../protocol+ '@hypit/text':+ specifier: workspace:*+ version: link:../text+packages/fonts-open:dependencies:'@hypit/markup':diff --cc skills/hypit/references/playbooks/craft/captions.mdindex 8055821e,a9dfa9b2..78134988--- a/skills/hypit/references/playbooks/craft/captions.md+++ b/skills/hypit/references/playbooks/craft/captions.md@@@ -114,7 -114,7 +114,7 @@@ or quantity and unit together when sepa### Language changes the reading unitEnglish commonly uses a word as its smallest timed display unit; Chinese uses individual Han--characters. That gives Chinese precise character highlighting, while a Cue still holds a meaningful++characters. That gives Chinese precise character timing, while a Cue still holds a meaningfulphrase. A Chinese Cue can comfortably contain more characters than an English Cue contains words.Choose its length from meaning, reading time and the space the actual font occupies. A fixed wordor character count cannot make that choice. Count the display side of Dual Text when judging fit.@@@ -127,15 -127,15 +127,16 @@@Fine already leaves adjacent Han characters together without English word gaps. Its `word-gap`controls spaced boundaries, including Chinese/Latin transitions; `letter-spacing` adjusts tracking.--The same timing supports a stable complete Cue, current-character color or a trail. For flowing++The same timing supports a stable complete Cue, current-word color or a trail. For flowingChinese speech, a readable phrase with restrained emphasis often works better than a separate bounceor reveal on every character. Choose the response for the performance's energy.--For whole-character highlighting, use `karaoke: current` with `karaoke-transition: step`.--Choose `trail` instead of `current` when the already spoken characters should stay highlighted.--`step` activates the complete timed unit at its start; `wipe` is a different visual choice that--sweeps inside its glyphs. Both follow each unit's own speech time, including uneven delivery and--pauses. Making a Cue longer changes its reading group, not its character timing.++For whole-word highlighting, use `karaoke: current` with `karaoke-transition: step`.++Fine groups adjacent display units into words and lights their complete glyphs together from the++first unit's time. Choose `trail` when already spoken words should stay highlighted. The grouping is++a presentation choice; Script keeps its individual character times. `wipe` instead sweeps inside++each unit's glyphs over that unit's speech window. Both use speech timing, including uneven delivery++and pauses. Making a Cue longer changes its reading group, not the underlying character timing.Text appearing and text changing color are separate choices. `atom-reveal: all` keeps the completeCue available to read while Karaoke supplies emphasis; `on-start` reveals whole spoken units, and