Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

feat(speech): separate voice design from voice clone

kashorin41644acb58ca
diff --git a/docs/guide/packages.md b/docs/guide/packages.md
index 63864f5f..9ea53847 100644
--- a/docs/guide/packages.md
+++ b/docs/guide/packages.md
@@ -90,7 +90,7 @@ specific Provider deployment.
@hypit/gpt-image GPT Image model family
@hypit/nano-banana Nano Banana model family
@hypit/seedream Seedream model family
-@hypit/mimo-tts Xiaomi MiMo VoiceDesign model + author Surface
+@hypit/mimo-speech Xiaomi MiMo Voice Design and Voice Clone models + author Surfaces
@hypit/estimate creation-time speech measurement (hypit measure) and the estimate policy type
@hypit/speech shared speech products
@hypit/speech-evidence acoustic evidence products
@@ -132,7 +132,7 @@ never on exact-model packages or the CLI.
@hypit/provider-hyperframes-aws-lambda asynchronous distributed rendering
@hypit/provider-image-opencv-local local OpenCV Raster execution
@hypit/provider-media-aws-lambda synchronous AWS media execution
-@hypit/provider-xiaomi-mimo official Xiaomi MiMo VoiceDesign API
+@hypit/provider-xiaomi-mimo official Xiaomi MiMo Voice Design and Voice Clone API
```
### Layer 6: Runtime
diff --git a/docs/guide/providers.md b/docs/guide/providers.md
index 67ec3d02..d229a9de 100644
--- a/docs/guide/providers.md
+++ b/docs/guide/providers.md
@@ -84,7 +84,7 @@ Look at existing Providers for reference:
- `packages/provider-hyperframes-local/` — local Chrome rendering
- `packages/provider-hyperframes-aws-lambda/` — asynchronous Step Functions/Lambda rendering
- `packages/provider-media-aws-lambda/` — synchronous Lambda media execution over the shared ffmpeg body
-- `packages/provider-xiaomi-mimo/` — immediate official VoiceDesign API without importing the MiMo model package
+- `packages/provider-xiaomi-mimo/` — immediate official Voice Design and Voice Clone API without importing the MiMo model package
## 4. Write the activation descriptor
@@ -239,4 +239,4 @@ hypit doctor hypit.runtime.json
| `provider-hyperframes-aws-lambda` | Remote asynchronous job: Step Functions submission, polling and S3 streaming |
| `provider-image-opencv-local` | Local Python: bounded OpenCV/NumPy with locked Python environment |
| `provider-media-aws-lambda` | Remote synchronous Lambda: the same nine capabilities as local media |
-| `provider-xiaomi-mimo` | Remote immediate API: exact MiMo VoiceDesign requests to persisted audio Artifacts |
+| `provider-xiaomi-mimo` | Remote immediate API: exact MiMo Voice Design and Voice Clone requests to persisted audio Resources |
diff --git a/docs/quickstart/run.md b/docs/quickstart/run.md
index e33d0ff7..9fd9e5d0 100644
--- a/docs/quickstart/run.md
+++ b/docs/quickstart/run.md
@@ -253,7 +253,7 @@ variables referenced by the selected Runtime Profile:
|---|---|
| HypiHub OAuth | HypiHub paid generation, Gemini VLM and WhisperX alignment; run `hypit auth login hypihub.default --runtime hypit.runtime.json` |
| `KIE_API_KEY` | Explicit KIE Provider only |
-| `MIMO_API_KEY` | Xiaomi MiMo VoiceDesign, only when the official Endpoint is explicitly selected |
+| `MIMO_API_KEY` | Xiaomi MiMo Voice Design or Voice Clone, only when the official Endpoint is explicitly selected |
Run only the lines for the Endpoints in your Profile. In macOS/Linux shells:
diff --git a/docs/zh/guide/packages.md b/docs/zh/guide/packages.md
index 9f25fa68..0901ff10 100644
--- a/docs/zh/guide/packages.md
+++ b/docs/zh/guide/packages.md
@@ -83,7 +83,7 @@ Frontend;不存在一个认识全部语法的中央解析器。
@hypit/gpt-image GPT Image model family
@hypit/nano-banana Nano Banana model family
@hypit/seedream Seedream model family
-@hypit/mimo-tts Xiaomi MiMo VoiceDesign 模型及作者 Surface
+@hypit/mimo-speech Xiaomi MiMo 音色设计与音色克隆模型及作者 Surface
@hypit/estimate 创作期量稿(hypit measure)与估算策略类型
@hypit/speech shared speech products
@hypit/speech-evidence acoustic evidence products
@@ -124,7 +124,7 @@ Frontend;不存在一个认识全部语法的中央解析器。
@hypit/provider-hyperframes-aws-lambda asynchronous distributed rendering
@hypit/provider-image-opencv-local 本地 OpenCV 光栅执行
@hypit/provider-media-aws-lambda synchronous AWS media execution
-@hypit/provider-xiaomi-mimo Xiaomi 官方 MiMo VoiceDesign API
+@hypit/provider-xiaomi-mimo Xiaomi 官方 MiMo 音色设计与音色克隆 API
```
### Layer 6:Runtime
diff --git a/docs/zh/guide/providers.md b/docs/zh/guide/providers.md
index fb3d90ec..86daae41 100644
--- a/docs/zh/guide/providers.md
+++ b/docs/zh/guide/providers.md
@@ -81,7 +81,7 @@ export function createMyServiceProvider(options: {
- `packages/provider-hyperframes-local/` — 本地 Chrome 渲染
- `packages/provider-hyperframes-aws-lambda/` — 异步 Step Functions/Lambda 渲染
- `packages/provider-media-aws-lambda/` — 通过共享 ffmpeg 执行体完成同步 Lambda 媒体操作
-- `packages/provider-xiaomi-mimo/` — 不依赖 MiMo 模型包的官方即时 VoiceDesign API
+- `packages/provider-xiaomi-mimo/` — 不依赖 MiMo 模型包的官方即时音色设计与音色克隆 API
## 4. 编写 activation 描述符
@@ -220,4 +220,4 @@ hypit doctor hypit.runtime.json
| `provider-hyperframes-aws-lambda` | 远程异步任务:按 Operation 自行暂存、Step Functions 轮询与结果流式回收 |
| `provider-image-opencv-local` | 本地 Python:有界的 OpenCV/NumPy,配合锁定的 Python 环境 |
| `provider-media-aws-lambda` | 远程同步 Lambda:与本地媒体相同的九项能力 |
-| `provider-xiaomi-mimo` | 远程即时 API:把精确 MiMo VoiceDesign 请求落成持久化音频 Artifact |
+| `provider-xiaomi-mimo` | 远程即时 API:把精确 MiMo 音色设计与音色克隆请求落成持久化音频 Resource |
diff --git a/docs/zh/quickstart/run.md b/docs/zh/quickstart/run.md
index 1fc048b6..c7736a50 100644
--- a/docs/zh/quickstart/run.md
+++ b/docs/zh/quickstart/run.md
@@ -218,7 +218,7 @@ CLI 必须先确定项目:显式 `--workspace` 直接给出边界;否则使
|---|---|
| HypiHub OAuth | HypiHub 付费生成、Gemini VLM 与 WhisperX 对齐;运行 `hypit auth login hypihub.default --runtime hypit.runtime.json` 并在 https://hypit.ai 登录 |
| `KIE_API_KEY` | 仅在显式选择 KIE Provider 时使用 |
-| `MIMO_API_KEY` | Xiaomi MiMo VoiceDesign;只有明确选择官方 Endpoint 时才需要 |
+| `MIMO_API_KEY` | Xiaomi MiMo 音色设计或音色克隆;只有明确选择官方 Endpoint 时才需要 |
只执行 Profile 中所选 Endpoint 对应的行。在 macOS/Linux Shell 中:
diff --git a/examples/interview/reference.svml b/examples/interview/reference.svml
index 0f5648f1..7561736b 100644
--- a/examples/interview/reference.svml
+++ b/examples/interview/reference.svml
@@ -4,7 +4,7 @@
<import from="@hypit/script@1"/>
<import as="text" from="@hypit/text@1"/>
<import as="media" from="@hypit/media@1"/>
- <import as="mimo" from="@hypit/mimo-tts@1"/>
+ <import as="mimo" from="@hypit/mimo-speech@1"/>
<import as="seedance" from="@hypit/seedance@1"/>
<import as="gpt" from="@hypit/gpt-image@1"/>
<import as="program" from="@hypit/program-space@1"/>
@@ -133,8 +133,8 @@
<seedance:Reference image={interviewer-view.image}/>
<seedance:Reference image={guest-view.image}/>
<seedance:Reference image={shared-view.image}/>
- <seedance:Reference audio={boy-voice.audio}/>
- <seedance:Reference audio={wife-voice.audio}/>
+ <seedance:Reference audio={boy-voice.reference}/>
+ <seedance:Reference audio={wife-voice.reference}/>
</seedance:ReferenceVideo>
<seedance:ReferenceVideo id="real-estate-rule-take"
@@ -144,8 +144,8 @@
<seedance:Reference image={interviewer-view.image}/>
<seedance:Reference image={guest-view.image}/>
<seedance:Reference image={shared-view.image}/>
- <seedance:Reference audio={boy-voice.audio}/>
- <seedance:Reference audio={wife-voice.audio}/>
+ <seedance:Reference audio={boy-voice.reference}/>
+ <seedance:Reference audio={wife-voice.reference}/>
</seedance:ReferenceVideo>
<seedance:ReferenceVideo id="bitcoin-rule-take"
@@ -155,8 +155,8 @@
<seedance:Reference image={interviewer-view.image}/>
<seedance:Reference image={guest-view.image}/>
<seedance:Reference image={shared-view.image}/>
- <seedance:Reference audio={boy-voice.audio}/>
- <seedance:Reference audio={wife-voice.audio}/>
+ <seedance:Reference audio={boy-voice.reference}/>
+ <seedance:Reference audio={wife-voice.reference}/>
</seedance:ReferenceVideo>
<space:Canvas id="vertical" width="720" height="1280"/>
diff --git a/examples/podcast/reference.svml b/examples/podcast/reference.svml
index aba0d86f..bf287f7d 100644
--- a/examples/podcast/reference.svml
+++ b/examples/podcast/reference.svml
@@ -3,7 +3,7 @@
<svml>
<import from="@hypit/script@1"/>
<import as="asset" from="@hypit/media@1"/>
- <import as="mimo" from="@hypit/mimo-tts@1"/>
+ <import as="mimo" from="@hypit/mimo-speech@1"/>
<import as="text" from="@hypit/text@1"/>
<import as="gpt" from="@hypit/gpt-image@1"/>
<import as="seedance" from="@hypit/seedance@1"/>
@@ -163,7 +163,7 @@
prompt={opening-take-prompt} duration="4"
resolution="720p" aspect-ratio="9:16" generate-audio="true">
<seedance:Reference image={split.image}/>
- <seedance:Reference audio={guy-voice.audio}/>
+ <seedance:Reference audio={guy-voice.reference}/>
</seedance:ReferenceVideo>
<seedance:ReferenceVideo id="daily-creatine-take" model="mini"
@@ -171,8 +171,8 @@
resolution="720p" aspect-ratio="9:16" generate-audio="true">
<seedance:Reference image={girl-hold.image}/>
<seedance:Reference image={guy.image}/>
- <seedance:Reference audio={girl-voice.audio}/>
- <seedance:Reference audio={guy-voice.audio}/>
+ <seedance:Reference audio={girl-voice.reference}/>
+ <seedance:Reference audio={guy-voice.reference}/>
</seedance:ReferenceVideo>
<seedance:ReferenceVideo id="arms-are-asking-take" model="mini"
@@ -180,8 +180,8 @@
resolution="720p" aspect-ratio="9:16" generate-audio="true">
<seedance:Reference image={girl-hold.image}/>
<seedance:Reference image={guy-hold.image}/>
- <seedance:Reference audio={girl-voice.audio}/>
- <seedance:Reference audio={guy-voice.audio}/>
+ <seedance:Reference audio={girl-voice.reference}/>
+ <seedance:Reference audio={guy-voice.reference}/>
</seedance:ReferenceVideo>
<space:Canvas id="vertical" width="720" height="1280"/>
diff --git a/examples/ranking-football/reference.svml b/examples/ranking-football/reference.svml
index d4743a48..b0940fe2 100644
--- a/examples/ranking-football/reference.svml
+++ b/examples/ranking-football/reference.svml
@@ -4,7 +4,7 @@
<import from="@hypit/script@1"/>
<import as="text" from="@hypit/text@1"/>
<import as="asset" from="@hypit/media@1"/>
- <import as="mimo" from="@hypit/mimo-tts@1"/>
+ <import as="mimo" from="@hypit/mimo-speech@1"/>
<import as="gpt" from="@hypit/gpt-image@1"/>
<import as="seedance" from="@hypit/seedance@1"/>
<import as="pipeline" from="@hypit/media-pipeline@1"/>
@@ -138,13 +138,13 @@
prompt={ronaldo-prompt} duration="10"
resolution="720p" aspect-ratio="9:16" generate-audio="true">
<seedance:Reference image={presenter.image}/>
- <seedance:Reference audio={presenter-voice.audio}/>
+ <seedance:Reference audio={presenter-voice.reference}/>
</seedance:ReferenceVideo>
<seedance:ReferenceVideo id="messi-take" model="mini"
prompt={messi-prompt} duration="10"
resolution="720p" aspect-ratio="9:16" generate-audio="true">
<seedance:Reference image={presenter.image}/>
- <seedance:Reference audio={presenter-voice.audio}/>
+ <seedance:Reference audio={presenter-voice.reference}/>
</seedance:ReferenceVideo>
<space:Canvas id="vertical" width="720" height="1280"/>
diff --git a/package.json b/package.json
index 20f1ca44..d58f3f64 100644
--- a/package.json
+++ b/package.json
@@ -86,7 +86,7 @@
"@hypit/media-pipeline": "workspace:*",
"@hypit/media-track": "workspace:*",
"@hypit/media-track-studio": "workspace:*",
- "@hypit/mimo-tts": "workspace:*",
+ "@hypit/mimo-speech": "workspace:*",
"@hypit/minimax-h3": "workspace:*",
"@hypit/nano-banana": "workspace:*",
"@hypit/narrative": "workspace:*",
diff --git a/packages/mimo-speech/README.md b/packages/mimo-speech/README.md
new file mode 100644
index 00000000..15ea1a7e
--- /dev/null
+++ b/packages/mimo-speech/README.md
@@ -0,0 +1,33 @@
+# `@hypit/mimo-speech`
+
+Exact model contracts and author Surfaces for Xiaomi MiMo V2.5 Voice Design and Voice Clone.
+
+The package owns two distinct speech operations:
+
+- `mimo-v2.5-tts-voicedesign` creates an ordinary audio voice reference from a natural-language
+ description and a short authored speech sample;
+- `mimo-v2.5-tts-voiceclone` uses one accepted audio voice reference to create independent speech.
+
+It contains no API URL, credential, retry, queue or Xiaomi wire encoding. Those belong to a Runtime
+Endpoint such as `@hypit/provider-xiaomi-mimo`. Both models return the shared `GeneratedAudioSet`;
+the author Surfaces expose its primary member as an ordinary audio Resource.
+
+```xml
+<import as="mimo" from="@hypit/mimo-speech@1"/>
+
+<mimo:VoiceDesign id="host" speech={story.segment.voiceSample.speech}>
+ A clear young woman with a grounded, confident conversational delivery.
+</mimo:VoiceDesign>
+
+<mimo:VoiceClone id="narration" speech={story.segment.reveal.speech} voice={host.reference}>
+ Quietly confident, with a short pause before the final word.
+</mimo:VoiceClone>
+```
+
+`host.reference` is not a special identity record. It is a normal audio Resource, so the same
+accepted reference can also feed an A-roll video model that accepts reference audio.
+
+Official API reference: <https://mimo.mi.com/docs/zh-CN/quick-start/usage-guide/audio/speech-synthesis-v2.5>
+
+The Source import selects author syntax and exact model semantics only. The Runtime Profile
+independently selects the Provider that fulfils either capability.
diff --git a/packages/mimo-tts/package.json b/packages/mimo-speech/package.json
similarity index 93%
rename from packages/mimo-tts/package.json
rename to packages/mimo-speech/package.json
index 6ad3259d..b62279a4 100644
--- a/packages/mimo-tts/package.json
+++ b/packages/mimo-speech/package.json
@@ -1,5 +1,5 @@
{
- "name": "@hypit/mimo-tts",
+ "name": "@hypit/mimo-speech",
"version": "0.0.0-dev",
"license": "SEE LICENSE IN LICENSE",
"private": true,
diff --git a/packages/mimo-speech/src/activation.ts b/packages/mimo-speech/src/activation.ts
new file mode 100644
index 00000000..c7bcc397
--- /dev/null
+++ b/packages/mimo-speech/src/activation.ts
@@ -0,0 +1,32 @@
+import { createMarkupSurfaceHostFacet } from "@hypit/markup";
+
+import {
+ decodeMimoVoiceCloneSurface,
+ decodeMimoVoiceDesignSurface,
+ mimoSpeechComponent,
+ mimoSpeechDefinition,
+ mimoSpeechManifest,
+ mimoSpeechMarkupSurfaces,
+ mimoSpeechModuleRef,
+} from "./index.js";
+
+export const hypitPackage = {
+ format: "hypit.node-package@1" as const,
+ modules: [{ manifest: mimoSpeechManifest }],
+ components: [mimoSpeechComponent],
+ hostFacets: [
+ mimoSpeechDefinition.hostFacet,
+ createMarkupSurfaceHostFacet({
+ module: mimoSpeechModuleRef,
+ declaration: mimoSpeechMarkupSurfaces.find((item) => item.name === "voiceDesign")!,
+ handler: decodeMimoVoiceDesignSurface,
+ }),
+ createMarkupSurfaceHostFacet({
+ module: mimoSpeechModuleRef,
+ declaration: mimoSpeechMarkupSurfaces.find((item) => item.name === "voiceClone")!,
+ handler: decodeMimoVoiceCloneSurface,
+ }),
+ ],
+};
+
+export default hypitPackage;
diff --git a/packages/mimo-tts/src/fragment.ts b/packages/mimo-speech/src/fragment.ts
similarity index 91%
rename from packages/mimo-tts/src/fragment.ts
rename to packages/mimo-speech/src/fragment.ts
index 4f36dfe8..11c7b817 100644
--- a/packages/mimo-tts/src/fragment.ts
+++ b/packages/mimo-speech/src/fragment.ts
@@ -2,7 +2,7 @@ import { createExactModelPrimaryGenerationFragment } from "@hypit/model-kit";
import type { ExactModelEndpoint, ExactModelMediaInput, ExactModelTextInput } from "@hypit/model-kit";
/** Model result projection over the same graph-native request assembly as every exact model. */
-export function createMimoTtsAudioFragment(
+export function createMimoSpeechAudioFragment(
endpoint: ExactModelEndpoint,
mediaInputs: readonly ExactModelMediaInput[] = [],
textInputs: readonly ExactModelTextInput[] = [],
diff --git a/packages/mimo-speech/src/index.ts b/packages/mimo-speech/src/index.ts
new file mode 100644
index 00000000..5a9fbf9e
--- /dev/null
+++ b/packages/mimo-speech/src/index.ts
@@ -0,0 +1,160 @@
+import { artifactTypes } from "@hypit/artifact";
+import { sealGenerationPortRequest, sealGenerationRequestDraft, sealGenerationPortTable } from "@hypit/generation";
+import type { GenerationPortTable, GenerationPortValue, GenerationRequest } from "@hypit/generation";
+import type { SurfaceAttributeVocabulary, SurfacePortVocabulary } from "@hypit/markup";
+import { defineExactModelModule } from "@hypit/model-kit";
+import { textTypes } from "@hypit/text";
+
+export const mimoSpeechModuleRef = { name: "@hypit/mimo-speech", version: "1" } as const;
+export const mimoSpeechModels = [
+ "mimo-v2.5-tts-voicedesign",
+ "mimo-v2.5-tts-voiceclone",
+] as const;
+export type MimoSpeechModel = typeof mimoSpeechModels[number];
+
+// Xiaomi documents no fixed character ceiling. Service-side limits stay service-side
+// instead of becoming an invented model constraint in Author Source.
+const spokenText = { kind: "text" } as const;
+const instruction = { kind: "text" } as const;
+
+function table(model: MimoSpeechModel): GenerationPortTable {
+ if (model === "mimo-v2.5-tts-voicedesign") {
+ return sealGenerationPortTable({
+ model,
+ result: "audio",
+ ports: [
+ { name: "text", value: spokenText, minItems: 1, maxItems: 1 },
+ { name: "voiceDescription", value: instruction, minItems: 1, maxItems: 1 },
+ ],
+ requires: [],
+ });
+ }
+ return sealGenerationPortTable({
+ model,
+ result: "audio",
+ ports: [
+ { name: "text", value: spokenText, minItems: 1, maxItems: 1 },
+ { name: "instruction", value: instruction, minItems: 0, maxItems: 1 },
+ { name: "voiceReference", value: { kind: "media", accepts: ["audio"] }, minItems: 1, maxItems: 1 },
+ ],
+ requires: [],
+ });
+}
+
+export const mimoSpeechPorts: Readonly<Record<MimoSpeechModel, GenerationPortTable>> = {
+ "mimo-v2.5-tts-voicedesign": table("mimo-v2.5-tts-voicedesign"),
+ "mimo-v2.5-tts-voiceclone": table("mimo-v2.5-tts-voiceclone"),
+};
+
+export function sealMimoSpeechRequest(
+ model: MimoSpeechModel,
+ ports: Readonly<Record<string, readonly GenerationPortValue[]>>,
+): GenerationRequest {
+ return sealGenerationPortRequest(mimoSpeechPorts[model], ports);
+}
+
+export function sealMimoSpeechRequestDraft(
+ model: MimoSpeechModel,
+ ports: Readonly<Record<string, readonly GenerationPortValue[]>>,
+) {
+ return sealGenerationRequestDraft(mimoSpeechPorts[model], ports);
+}
+
+const base = defineExactModelModule({
+ module: mimoSpeechModuleRef,
+ endpoints: ([
+ ["voiceDesign", "mimo-v2.5-tts-voicedesign", "MimoVoiceDesignRequest"],
+ ["voiceClone", "mimo-v2.5-tts-voiceclone", "MimoVoiceCloneRequest"],
+ ] as const).map(([key, model, requestTypeName]) => ({
+ key,
+ requestTypeName,
+ producerName: `request-${model}`,
+ ports: mimoSpeechPorts[model],
+ })),
+});
+
+export const mimoSpeechEndpoints = base.endpoints;
+
+const spokenAttributes: readonly SurfaceAttributeVocabulary[] = [
+ {
+ name: "id",
+ kind: "identifier",
+ required: true,
+ summary: "Names this speech request and prefixes the binding it publishes.",
+ },
+ {
+ name: "speech",
+ kind: "reference",
+ required: true,
+ summary: "The Text edge whose exact words are spoken.",
+ accepts: [textTypes.text],
+ },
+];
+
+const referencePort: readonly SurfacePortVocabulary[] = [{
+ name: "reference",
+ type: artifactTypes.blob,
+ summary: "The designed voice reference, addressed as `<id>.reference`.",
+}];
+
+const audioPort: readonly SurfacePortVocabulary[] = [{
+ name: "audio",
+ type: artifactTypes.blob,
+ summary: "The independent speech, addressed as `<id>.audio`.",
+}];
+
+export const mimoSpeechMarkupSurfaces = [
+ {
+ name: "voiceDesign", tag: "VoiceDesign", mode: "structured",
+ outputs: [mimoSpeechEndpoints.voiceDesign.draftType],
+ vocabulary: {
+ summary: "Creates a reusable voice reference from a natural-language voice description.",
+ attributes: spokenAttributes,
+ ports: referencePort,
+ text: "The element's own text is the voice description and is required.",
+ example: `<mimo:VoiceDesign id="host" speech={story.segment.voiceSample.speech}>
+ A clear young woman with a grounded, confident conversational delivery.
+</mimo:VoiceDesign>`,
+ notes: [
+ "The element accepts no child elements; only its text is read.",
+ "The result is an ordinary audio Resource that can be supplied anywhere an audio reference is accepted.",
+ ],
+ },
+ },
+ {
+ name: "voiceClone", tag: "VoiceClone", mode: "structured",
+ outputs: [
+ mimoSpeechEndpoints.voiceClone.draftType,
+ ...Object.values(mimoSpeechEndpoints.voiceClone.mediaBindings).map((binding) => binding.type),
+ ],
+ vocabulary: {
+ summary: "Creates independent speech in the voice heard in one accepted audio reference.",
+ attributes: [
+ ...spokenAttributes,
+ {
+ name: "voice",
+ kind: "reference",
+ required: true,
+ summary: "The audio Resource carrying the voice identity to reproduce.",
+ accepts: [artifactTypes.blob],
+ },
+ ],
+ ports: audioPort,
+ text: "The element's own text is an optional delivery instruction.",
+ example: `<mimo:VoiceClone id="narration" speech={story.segment.reveal.speech} voice={host.reference}>
+ Quietly confident, with a short pause before the final word.
+</mimo:VoiceClone>`,
+ notes: [
+ "The element accepts no child elements; only its text is read.",
+ "The voice reference remains a normal audio Resource rather than a separate identity record.",
+ ],
+ },
+ },
+] as const;
+
+export const mimoSpeechManifest = { ...base.manifest } as const;
+export const mimoSpeechComponent = base.component;
+export const mimoSpeechDefinition = { ...base, manifest: mimoSpeechManifest };
+
+export { createMimoSpeechAudioFragment } from "./fragment.js";
+export { decodeMimoVoiceCloneSurface, decodeMimoVoiceDesignSurface } from "./surface.js";
diff --git a/packages/mimo-speech/src/surface.ts b/packages/mimo-speech/src/surface.ts
new file mode 100644
index 00000000..4ae8b424
--- /dev/null
+++ b/packages/mimo-speech/src/surface.ts
@@ -0,0 +1,193 @@
+import { artifactTypes } from "@hypit/artifact";
+import { generationPort, sealGenerationMediaBinding } from "@hypit/generation";
+import type { GenerationMediaPort, GenerationPortValue } from "@hypit/generation";
+import { exactModelMediaInputNames, exactModelTextInputName } from "@hypit/model-kit";
+import type { ExactModelEndpoint, ExactModelMediaInput, ExactModelTextInput } from "@hypit/model-kit";
+import type { CanonicalValue } from "@hypit/protocol";
+import { textTypes, verifyText } from "@hypit/text";
+import type {
+ MarkupAttributeValue,
+ StructuredElement,
+ StructuredSurfaceHandler,
+ SurfaceResolvedReference,
+} from "@hypit/markup";
+
+import { createMimoSpeechAudioFragment } from "./fragment.js";
+import { mimoSpeechEndpoints, sealMimoSpeechRequestDraft } from "./index.js";
+
+function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+}
+
+function attributes(element: StructuredElement, required: readonly string[]): void {
+ const allowed = new Set(required);
+ const unknown = Object.keys(element.attributes).filter((name) => !allowed.has(name));
+ if (unknown.length > 0 || required.some((name) => element.attributes[name] === undefined)) {
+ throw new Error(`${element.name} requires ${required.join(", ")}`);
+ }
+}
+
+function stringAttribute(element: StructuredElement, name: string): string {
+ const value = element.attributes[name];
+ if (typeof value !== "string" || value.trim().length === 0) {
+ throw new Error(`${element.name}.${name} must be a non-empty string`);
+ }
+ return value.trim();
+}
+
+function referencePath(element: StructuredElement, name: string): string {
+ const value: MarkupAttributeValue | undefined = element.attributes[name];
+ if (typeof value !== "object" || value.kind !== "reference" || value.path.length === 0) {
+ throw new Error(`${element.name}.${name} must be a whole-value reference`);
+ }
+ return value.path;
+}
+
+function resolved(
+ element: StructuredElement,
+ name: string,
+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,
+): SurfaceResolvedReference {
+ const path = referencePath(element, name);
+ const value = resolveReference(path);
+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${path}`);
+ return value;
+}
+
+function speechText(reference: SurfaceResolvedReference, subject: string): SurfaceResolvedReference {
+ if (!sameType(reference.type, textTypes.text)) throw new Error(`${subject} must reference Text`);
+ const value = reference.record?.value;
+ if (value !== undefined) {
+ if (value.kind !== "inline") throw new Error(`${subject} has an invalid authored Text value`);
+ verifyText(value.value);
+ }
+ return reference;
+}
+
+function voiceReference(reference: SurfaceResolvedReference, subject: string): SurfaceResolvedReference {
+ if (!sameType(reference.type, artifactTypes.blob)) throw new Error(`${subject} must reference an audio Resource`);
+ const value = reference.record?.value;
+ if (value !== undefined && (value.kind !== "blob" || !value.mediaType.startsWith("audio/"))) {
+ throw new Error(`${subject} must reference audio media`);
+ }
+ return reference;
+}
+
+function body(element: StructuredElement, required: boolean): string | undefined {
+ if (element.children.some((child) => child.kind === "element")) {
+ throw new Error(`${element.name} accepts instruction text only`);
+ }
+ const lines = element.children.map((child) => child.kind === "text" ? child.value : "").join("")
+ .replaceAll("\r\n", "\n").split("\n");
+ while (lines[0]?.trim() === "") lines.shift();
+ while (lines.at(-1)?.trim() === "") lines.pop();
+ const indents = lines.filter((line) => line.trim()).map((line) => /^\s*/u.exec(line)?.[0].length ?? 0);
+ const indent = indents.length === 0 ? 0 : Math.min(...indents);
+ const value = lines.map((line) => line.slice(indent).trimEnd()).join("\n").trim();
+ if (value.length === 0) {
+ if (required) throw new Error(`${element.name} requires a voice description in its body`);
+ return undefined;
+ }
+ return value;
+}
+
+function output(
+ element: StructuredElement,
+ endpoint: ExactModelEndpoint,
+ draft: ReturnType<typeof sealMimoSpeechRequestDraft>,
+ outputName: "reference" | "audio",
+ text: readonly { readonly input: ExactModelTextInput; readonly source: SurfaceResolvedReference }[],
+ media: readonly { readonly input: ExactModelMediaInput; readonly source: SurfaceResolvedReference }[] = [],
+) {
+ const id = stringAttribute(element, "id");
+ const draftId = `${id}.draft`;
+ const records: Array<{
+ readonly id: string;
+ readonly type: SurfaceResolvedReference["type"];
+ readonly value: { readonly kind: "inline"; readonly value: CanonicalValue };
+ readonly range: StructuredElement["range"];
+ }> = [{
+ id: draftId,
+ type: endpoint.draftType,
+ value: { kind: "inline", value: draft as unknown as CanonicalValue },
+ range: element.range,
+ }];
+ const inputs: Record<string, SurfaceResolvedReference["ref"] | { readonly kind: "record"; readonly id: string }> = {
+ draft: { kind: "record", id: draftId },
+ };
+ for (const item of text) inputs[exactModelTextInputName(item.input.name)] = item.source.ref;
+ for (const item of media) {
+ const binding = endpoint.mediaBindings[item.input.port];
+ if (binding === undefined) throw new Error(`${endpoint.ports.model} has no media port ${item.input.port}`);
+ const port = generationPort(endpoint.ports, item.input.port);
+ if (port.value.kind !== "media") throw new Error(`${endpoint.ports.model} port ${item.input.port} is not media`);
+ const names = exactModelMediaInputNames(item.input.name);
+ const bindingId = `${id}.${item.input.name}.binding`;
+ records.push({
+ id: bindingId,
+ type: binding.type,
+ value: {
+ kind: "inline",
+ value: sealGenerationMediaBinding(port as GenerationMediaPort, { role: "audio" }) as unknown as CanonicalValue,
+ },
+ range: element.range,
+ });
+ inputs[names.binding] = { kind: "record", id: bindingId };
+ inputs[names.artifact] = item.source.ref;
+ }
+ const fragment = createMimoSpeechAudioFragment(
+ endpoint,
+ media.map((item) => item.input),
+ text.map((item) => item.input),
+ );
+ return {
+ records,
+ components: [{
+ id,
+ fragment: fragment.id,
+ inputs,
+ outputs: { [outputName]: `${id}.${outputName}` },
+ range: element.range,
+ }],
+ fragments: [fragment],
+ };
+}
+
+function speechInput(
+ element: StructuredElement,
+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,
+) {
+ return {
+ input: { name: "speech", port: "text" },
+ source: speechText(resolved(element, "speech", resolveReference), `${element.name}.speech`),
+ } as const;
+}
+
+export const decodeMimoVoiceDesignSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ attributes(element, ["id", "speech"]);
+ return output(
+ element,
+ mimoSpeechEndpoints.voiceDesign,
+ sealMimoSpeechRequestDraft("mimo-v2.5-tts-voicedesign", { voiceDescription: [body(element, true)!] }),
+ "reference",
+ [speechInput(element, resolveReference)],
+ );
+};
+
+export const decodeMimoVoiceCloneSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ attributes(element, ["id", "speech", "voice"]);
+ const instruction = body(element, false);
+ const ports: Record<string, readonly GenerationPortValue[]> = {};
+ if (instruction !== undefined) ports.instruction = [instruction];
+ return output(
+ element,
+ mimoSpeechEndpoints.voiceClone,
+ sealMimoSpeechRequestDraft("mimo-v2.5-tts-voiceclone", ports),
+ "audio",
+ [speechInput(element, resolveReference)],
+ [{
+ input: { name: "voice", port: "voiceReference" },
+ source: voiceReference(resolved(element, "voice", resolveReference), `${element.name}.voice`),
+ }],
+ );
+};
diff --git a/packages/mimo-speech/test/mimo-speech.test.ts b/packages/mimo-speech/test/mimo-speech.test.ts
new file mode 100644
index 00000000..f0e61529
--- /dev/null
+++ b/packages/mimo-speech/test/mimo-speech.test.ts
@@ -0,0 +1,110 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import { artifactTypes } from "@hypit/artifact";
+import { generationTypes } from "@hypit/generation";
+import { parseStructuredElement } from "@hypit/markup";
+import type { SurfaceResolvedReference } from "@hypit/markup";
+import { sealText, textTypes } from "@hypit/text";
+
+import mimoNodePackage from "../src/activation.js";
+import {
+ decodeMimoVoiceCloneSurface,
+ decodeMimoVoiceDesignSurface,
+ mimoSpeechEndpoints,
+ mimoSpeechMarkupSurfaces,
+ mimoSpeechModels,
+ mimoSpeechPorts,
+ sealMimoSpeechRequest,
+} from "../src/index.js";
+
+function parsed(source: string) {
+ return parseStructuredElement({ name: "mimo.svml", text: source }, 0).element;
+}
+
+const speech = sealText("Exact authored words stay exact.");
+const speechReference: SurfaceResolvedReference = {
+ path: "story.segment.opening.speech",
+ ref: { kind: "record", id: "story.segment.opening.speech" },
+ type: textTypes.text,
+ record: {
+ id: "story.segment.opening.speech",
+ type: textTypes.text,
+ value: { kind: "inline", value: speech },
+ },
+};
+const voiceReference: SurfaceResolvedReference = {
+ path: "host.reference",
+ ref: { kind: "record", id: "host.reference" },
+ type: artifactTypes.blob,
+ record: {
+ id: "host.reference",
+ type: artifactTypes.blob,
+ value: { kind: "blob", resource: "res_voice-reference", size: 4, mediaType: "audio/wav" },
+ },
+};
+
+const context = (source: string, voice: SurfaceResolvedReference = voiceReference) => ({
+ sourceName: "mimo.svml",
+ element: parsed(source),
+ resolveReference: (path: string) => path === speechReference.path
+ ? speechReference : path === voice.path ? voice : undefined,
+ resolveAsset: () => { throw new Error("no asset"); },
+});
+
+test("MiMo Speech declares Voice Design and Voice Clone without Provider facts", () => {
+ assert.deepEqual(mimoSpeechModels, ["mimo-v2.5-tts-voicedesign", "mimo-v2.5-tts-voiceclone"]);
+ assert.ok(Object.values(mimoSpeechPorts).every((ports) => ports.result === "audio"));
+ assert.ok(Object.values(mimoSpeechEndpoints).every((endpoint) => endpoint.returns.name === generationTypes.audioSet.name));
+ assert.equal(JSON.stringify(mimoSpeechPorts).includes("xiaomimimo.com"), false);
+ assert.throws(() => sealMimoSpeechRequest("mimo-v2.5-tts-voicedesign", {
+ text: ["Do not rewrite me."],
+ }), /voiceDescription is required/u);
+ assert.throws(() => sealMimoSpeechRequest("mimo-v2.5-tts-voiceclone", {
+ text: ["Do not rewrite me."],
+ }), /voiceReference is required/u);
+});
+
+test("the installed author package contributes the two explicit speech operations", () => {
+ assert.equal(mimoNodePackage.format, "hypit.node-package@1");
+ assert.equal(mimoNodePackage.modules[0]?.manifest.version, "1");
+ assert.deepEqual(mimoSpeechMarkupSurfaces.map((surface) => surface.name), ["voiceDesign", "voiceClone"]);
+ assert.equal(JSON.stringify(mimoNodePackage).includes("MIMO_API_KEY"), false);
+});
+
+test("VoiceDesign publishes one reusable voice reference", async () => {
+ const result = await decodeMimoVoiceDesignSurface(context(`<mimo:VoiceDesign id="host" speech={story.segment.opening.speech}>
+ A clear, confident young woman with a grounded conversational tone.
+ </mimo:VoiceDesign>`));
+ assert.deepEqual(result.components[0]!.outputs, { reference: "host.reference" });
+ assert.deepEqual(result.components[0]!.inputs["speech:text"], {
+ kind: "record",
+ id: "story.segment.opening.speech",
+ });
+ const request = result.records[0]!.value.kind === "inline"
+ ? result.records[0]!.value.value as Record<string, unknown> : {};
+ assert.equal((request.ports as Record<string, unknown[]>).text, undefined);
+});
+
+test("VoiceClone binds an accepted audio Resource and publishes independent speech", async () => {
+ const result = await decodeMimoVoiceCloneSurface(context(`<mimo:VoiceClone id="narration" speech={story.segment.opening.speech} voice={host.reference}>
+ Calm and restrained.
+ </mimo:VoiceClone>`));
+ assert.deepEqual(result.components[0]!.outputs, { audio: "narration.audio" });
+ assert.deepEqual(result.components[0]!.inputs["voice:artifact"], { kind: "record", id: "host.reference" });
+ assert.ok(result.components[0]!.inputs["voice:binding"] !== undefined);
+});
+
+test("VoiceClone refuses a known non-audio Resource", () => {
+ const image: SurfaceResolvedReference = {
+ ...voiceReference,
+ record: {
+ ...voiceReference.record!,
+ value: { kind: "blob", resource: "res_image", size: 4, mediaType: "image/png" },
+ },
+ };
+ assert.throws(() => decodeMimoVoiceCloneSurface(context(
+ `<mimo:VoiceClone id="narration" speech={story.segment.opening.speech} voice={host.reference}/>` ,
+ image,
+ )), /must reference audio media/u);
+});
diff --git a/packages/mimo-tts/README.md b/packages/mimo-tts/README.md
deleted file mode 100644
index e4d73f20..00000000
--- a/packages/mimo-tts/README.md
+++ /dev/null
@@ -1,27 +0,0 @@
-# `@hypit/mimo-tts`
-
-Exact model contract and author Surface for Xiaomi MiMo V2.5 VoiceDesign.
-
-The package owns only `mimo-v2.5-tts-voicedesign`: exact speech text plus a
-natural-language voice description.
-
-It contains no API URL, credential, retry, queue or Xiaomi wire encoding. Those belong to a Runtime
-Endpoint such as `@hypit/provider-xiaomi-mimo`. The model returns the shared
-`GeneratedAudioSet`; author Surfaces expose its primary member as an ordinary `BlobArtifact`.
-The `speech` attribute is an ordinary `Text` edge attached through the exact model's `text` port at
-execution time; the Frontend never copies Script words into a request draft.
-
-`optimize_text_preview` is intentionally absent: MiMo may not rewrite Script's exact speech truth.
-
-```xml
-<import as="mimo" from="@hypit/mimo-tts@1"/>
-
-<mimo:VoiceDesign id="designed" speech={story.segment.answer.speech}>
- A clear young woman with a grounded, confident delivery.
-</mimo:VoiceDesign>
-```
-
-Official API reference: <https://mimo.mi.com/docs/zh-CN/quick-start/usage-guide/audio/speech-synthesis-v2.5>
-
-The Source import selects author syntax and exact model semantics only. It does not select an API
-service; the Runtime Profile independently selects a Provider.
diff --git a/packages/mimo-tts/src/activation.ts b/packages/mimo-tts/src/activation.ts
deleted file mode 100644
index fd437f2f..00000000
--- a/packages/mimo-tts/src/activation.ts
+++ /dev/null
@@ -1,25 +0,0 @@
-import { createMarkupSurfaceHostFacet } from "@hypit/markup";
-
-import {
- decodeMimoVoiceDesignSurface,
- mimoTtsComponent,
- mimoTtsDefinition,
- mimoTtsManifest,
- mimoTtsModuleRef,
- mimoTtsMarkupSurfaces,
-} from "./index.js";
-
-export const hypitPackage = {
- format: "hypit.node-package@1" as const,
- modules: [{ manifest: mimoTtsManifest }],
- components: [mimoTtsComponent],
- hostFacets: [
- mimoTtsDefinition.hostFacet,
- createMarkupSurfaceHostFacet({
- module: mimoTtsModuleRef,
- declaration: mimoTtsMarkupSurfaces.find((item) => item.name === "voiceDesign")!, handler: decodeMimoVoiceDesignSurface,
- }),
- ],
-};
-
-export default hypitPackage;
diff --git a/packages/mimo-tts/src/index.ts b/packages/mimo-tts/src/index.ts
deleted file mode 100644
index e8091147..00000000
--- a/packages/mimo-tts/src/index.ts
+++ /dev/null
@@ -1,104 +0,0 @@
-import { artifactTypes } from "@hypit/artifact";
-import { sealGenerationPortRequest, sealGenerationRequestDraft, sealGenerationPortTable } from "@hypit/generation";
-import type { GenerationPortTable, GenerationPortValue, GenerationRequest } from "@hypit/generation";
-import type { SurfaceAttributeVocabulary, SurfacePortVocabulary } from "@hypit/markup";
-import { defineExactModelModule } from "@hypit/model-kit";
-import { textTypes } from "@hypit/text";
-
-export const mimoTtsModuleRef = { name: "@hypit/mimo-tts", version: "1" } as const;
-export const mimoTtsModels = ["mimo-v2.5-tts-voicedesign"] as const;
-export type MimoTtsModel = typeof mimoTtsModels[number];
-
-// Xiaomi documents no fixed character ceiling. Service-side limits stay service-side
-// instead of becoming an invented model constraint in Author Source.
-const spokenText = { kind: "text" } as const;
-const instruction = { kind: "text" } as const;
-
-function table(model: MimoTtsModel): GenerationPortTable {
- return sealGenerationPortTable({
- model,
- result: "audio",
- ports: [
- { name: "text", value: spokenText, minItems: 1, maxItems: 1 },
- { name: "voiceDescription", value: instruction, minItems: 1, maxItems: 1 },
- ],
- requires: [],
- });
-}
-
-export const mimoTtsPorts: Readonly<Record<MimoTtsModel, GenerationPortTable>> = {
- "mimo-v2.5-tts-voicedesign": table("mimo-v2.5-tts-voicedesign"),
-};
-
-export function sealMimoTtsRequest(
- model: MimoTtsModel,
- ports: Readonly<Record<string, readonly GenerationPortValue[]>>,
-): GenerationRequest {
- return sealGenerationPortRequest(mimoTtsPorts[model], ports);
-}
-
-export function sealMimoTtsRequestDraft(
- model: MimoTtsModel,
- ports: Readonly<Record<string, readonly GenerationPortValue[]>>,
-) {
- return sealGenerationRequestDraft(mimoTtsPorts[model], ports);
-}
-
-const base = defineExactModelModule({
- module: mimoTtsModuleRef,
- endpoints: [{
- key: "voiceDesign",
- requestTypeName: "MimoVoiceDesignTtsRequest",
- producerName: "request-mimo-v2.5-tts-voicedesign",
- ports: mimoTtsPorts["mimo-v2.5-tts-voicedesign"],
- }],
-});
-
-export const mimoTtsEndpoints = base.endpoints;
-
-const mimoSpokenAttributes: readonly SurfaceAttributeVocabulary[] = [
- {
- name: "id",
- kind: "identifier",
- required: true,
- summary: "Names this synthesis and prefixes the bindings it publishes.",
- },
- {
- name: "speech",
- kind: "reference",
- required: true,
- summary: "The Text edge whose exact words the model speaks.",
- accepts: [textTypes.text],
- },
-];
-
-const mimoAudioPort: readonly SurfacePortVocabulary[] = [{
- name: "audio",
- type: artifactTypes.blob,
- summary: "The synthesized speech, addressed as `<id>.audio`.",
-}];
-
-export const mimoTtsMarkupSurfaces = [{
- name: "voiceDesign", tag: "VoiceDesign", mode: "structured",
- outputs: [mimoTtsEndpoints.voiceDesign.draftType],
- vocabulary: {
- summary: "Speaks a Text with a voice described in natural language.",
- attributes: mimoSpokenAttributes,
- ports: mimoAudioPort,
- text: "The element's own text is the voice description and is required.",
- example: `<mimo:VoiceDesign id="designed" speech={story.segment.answer.speech}>
- A clear young woman with a grounded, confident delivery.
-</mimo:VoiceDesign>`,
- notes: [
- "The element accepts no child elements; only its text is read.",
- "An empty body is refused, because this model has no voice to fall back on.",
- ],
- },
-}] as const;
-
-export const mimoTtsManifest = { ...base.manifest } as const;
-export const mimoTtsComponent = base.component;
-export const mimoTtsDefinition = { ...base, manifest: mimoTtsManifest };
-
-export { createMimoTtsAudioFragment } from "./fragment.js";
-export { decodeMimoVoiceDesignSurface } from "./surface.js";
diff --git a/packages/mimo-tts/src/surface.ts b/packages/mimo-tts/src/surface.ts
deleted file mode 100644
index 22c80050..00000000
--- a/packages/mimo-tts/src/surface.ts
+++ /dev/null
@@ -1,109 +0,0 @@
-import type { CanonicalValue } from "@hypit/protocol";
-import { exactModelTextInputName } from "@hypit/model-kit";
-import type { ExactModelTextInput } from "@hypit/model-kit";
-import { textTypes, verifyText } from "@hypit/text";
-import type {
- StructuredElement,
- StructuredSurfaceHandler,
- SurfaceResolvedReference,
- MarkupAttributeValue,
-} from "@hypit/markup";
-
-import { createMimoTtsAudioFragment } from "./fragment.js";
-import { mimoTtsEndpoints, sealMimoTtsRequestDraft } from "./index.js";
-
-function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
- return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
-}
-
-function attributes(element: StructuredElement, required: readonly string[]): void {
- const allowed = new Set(required);
- const unknown = Object.keys(element.attributes).filter((name) => !allowed.has(name));
- if (unknown.length > 0 || required.some((name) => element.attributes[name] === undefined)) {
- throw new Error(`${element.name} requires ${required.join(", ")}`);
- }
-}
-
-function stringAttribute(element: StructuredElement, name: string): string {
- const value = element.attributes[name];
- if (typeof value !== "string" || value.trim().length === 0) {
- throw new Error(`${element.name}.${name} must be a non-empty string`);
- }
- return value.trim();
-}
-
-function referencePath(element: StructuredElement, name: string): string {
- const value: MarkupAttributeValue | undefined = element.attributes[name];
- if (typeof value !== "object" || value.kind !== "reference" || value.path.length === 0) {
- throw new Error(`${element.name}.${name} must be a whole-value reference`);
- }
- return value.path;
-}
-
-function resolved(
- element: StructuredElement,
- name: string,
- resolveReference: (path: string) => SurfaceResolvedReference | undefined,
-): SurfaceResolvedReference {
- const path = referencePath(element, name);
- const value = resolveReference(path);
- if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${path}`);
- return value;
-}
-
-function speechText(reference: SurfaceResolvedReference, subject: string): SurfaceResolvedReference {
- if (!sameType(reference.type, textTypes.text)) throw new Error(`${subject} must reference Text`);
- const value = reference.record?.value;
- if (value !== undefined) {
- if (value.kind !== "inline") throw new Error(`${subject} has an invalid authored Text value`);
- verifyText(value.value);
- }
- return reference;
-}
-
-function body(element: StructuredElement): string {
- if (element.children.some((child) => child.kind === "element")) {
- throw new Error(`${element.name} accepts instruction text only`);
- }
- const lines = element.children.map((child) => child.kind === "text" ? child.value : "").join("")
- .replaceAll("\r\n", "\n").split("\n");
- while (lines[0]?.trim() === "") lines.shift();
- while (lines.at(-1)?.trim() === "") lines.pop();
- const indents = lines.filter((line) => line.trim()).map((line) => /^\s*/u.exec(line)?.[0].length ?? 0);
- const indent = indents.length === 0 ? 0 : Math.min(...indents);
- const value = lines.map((line) => line.slice(indent).trimEnd()).join("\n").trim();
- if (value.length === 0) throw new Error(`${element.name} requires a voice description in its body`);
- return value;
-}
-
-export const decodeMimoVoiceDesignSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
- attributes(element, ["id", "speech"]);
- const id = stringAttribute(element, "id");
- const speech = speechText(resolved(element, "speech", resolveReference), `${element.name}.speech`);
- const endpoint = mimoTtsEndpoints.voiceDesign;
- const draftId = `${id}.draft`;
- const draft = sealMimoTtsRequestDraft("mimo-v2.5-tts-voicedesign", {
- voiceDescription: [body(element)],
- });
- const textInput: ExactModelTextInput = { name: "speech", port: "text" };
- const fragment = createMimoTtsAudioFragment(endpoint, [], [textInput]);
- return {
- records: [{
- id: draftId,
- type: endpoint.draftType,
- value: { kind: "inline", value: draft as unknown as CanonicalValue },
- range: element.range,
- }],
- components: [{
- id,
- fragment: fragment.id,
- inputs: {
- draft: { kind: "record", id: draftId },
- [exactModelTextInputName(textInput.name)]: speech.ref,
- },
- outputs: { audio: `${id}.audio` },
- range: element.range,
- }],
- fragments: [fragment],
- };
-};
diff --git a/packages/mimo-tts/test/mimo-tts.test.ts b/packages/mimo-tts/test/mimo-tts.test.ts
deleted file mode 100644
index 9de58898..00000000
--- a/packages/mimo-tts/test/mimo-tts.test.ts
+++ /dev/null
@@ -1,79 +0,0 @@
-import assert from "node:assert/strict";
-import test from "node:test";
-
-import { generationProducers, generationTypes } from "@hypit/generation";
-import { parseStructuredElement } from "@hypit/markup";
-import type { SurfaceResolvedReference } from "@hypit/markup";
-import { sealText, textTypes } from "@hypit/text";
-
-import mimoNodePackage from "../src/activation.js";
-import {
- decodeMimoVoiceDesignSurface,
- mimoTtsEndpoints,
- mimoTtsMarkupSurfaces,
- mimoTtsModels,
- mimoTtsPorts,
- sealMimoTtsRequest,
-} from "../src/index.js";
-
-function parsed(source: string) {
- return parseStructuredElement({ name: "mimo.svml", text: source }, 0).element;
-}
-
-const speech = sealText("Exact authored words stay exact.");
-const reference: SurfaceResolvedReference = {
- path: "story.segment.opening.speech",
- ref: { kind: "record", id: "story.segment.opening.speech" },
- type: textTypes.text,
- record: {
- id: "story.segment.opening.speech",
- type: textTypes.text,
- value: { kind: "inline", value: speech },
- },
-};
-const context = (source: string) => ({
- sourceName: "mimo.svml",
- element: parsed(source),
- resolveReference: (path: string) => path === reference.path ? reference : undefined,
- resolveAsset: () => { throw new Error("no asset"); },
-});
-
-test("MiMo declares only the VoiceDesign audio model without Provider facts", () => {
- assert.deepEqual(mimoTtsModels, ["mimo-v2.5-tts-voicedesign"]);
- assert.deepEqual(Object.values(mimoTtsPorts).map((ports) => ports.result), ["audio"]);
- assert.deepEqual(Object.values(mimoTtsEndpoints).map((endpoint) => endpoint.returns), [generationTypes.audioSet]);
- assert.equal(JSON.stringify(mimoTtsPorts).includes("xiaomimimo.com"), false);
- assert.throws(() => sealMimoTtsRequest("mimo-v2.5-tts-voicedesign", {
- text: ["Do not rewrite me."],
- }), /voiceDescription is required/u);
-});
-
-test("the installed author package contributes only VoiceDesign", () => {
- assert.equal(mimoNodePackage.format, "hypit.node-package@1");
- assert.equal(mimoNodePackage.modules[0]?.manifest.version, "1");
- assert.deepEqual(mimoTtsMarkupSurfaces.map((surface) => surface.name), ["voiceDesign"]);
- assert.equal(JSON.stringify(mimoNodePackage).includes("MIMO_API_KEY"), false);
-});
-
-test("VoiceDesign produces one ordinary audio output", async () => {
- const result = await decodeMimoVoiceDesignSurface(context(`<mimo:VoiceDesign id="designed" speech={story.segment.opening.speech}>
- A clear, confident young woman with a grounded conversational tone.
- </mimo:VoiceDesign>`));
- assert.equal(result.components.length, 1);
- assert.deepEqual(Object.keys(result.components[0]!.outputs), ["audio"]);
- assert.deepEqual(result.fragments[0]!.operations.map((operation) => operation.producer.name), [
- "bind-request-mimo-v2.5-tts-voicedesign-text-text",
- "finalize-request-mimo-v2.5-tts-voicedesign",
- "request-mimo-v2.5-tts-voicedesign",
- generationProducers.primaryAudio.name,
- ]);
- assert.equal(result.records[0]!.value.kind, "inline");
- const request = result.records[0]!.value.kind === "inline"
- ? result.records[0]!.value.value as Record<string, unknown> : {};
- assert.equal((request.ports as Record<string, unknown[]>).text, undefined);
- assert.deepEqual(result.components[0]!.inputs["speech:text"], {
- kind: "record",
- id: "story.segment.opening.speech",
- });
- assert.equal(JSON.stringify(request).includes("optimize_text_preview"), false);
-});
diff --git a/packages/provider-hypihub/README.md b/packages/provider-hypihub/README.md
index 66532dc0..1b5e99f8 100644
--- a/packages/provider-hypihub/README.md
+++ b/packages/provider-hypihub/README.md
@@ -58,14 +58,27 @@ declares HypiHub's public pricing page, `https://hypit.ai/commercial/pricing/`,
`hypit plan --runtime <profile>` prints it beside each request this Endpoint would serve. Per-model
credit costs are never copied into Hypit.
-HypiHub declares MiMo VoiceDesign together with every other capability it serves; it never hides
-one. When another selected Endpoint offers the same capability (a local WhisperX, a Vertex Gemini,
-the official MiMo Provider), the Runtime Profile's `bindings` say which Endpoint serves it. Hypit does
-not expose MiMo preset-voice or voice-cloning models.
+HypiHub declares MiMo Voice Design and Voice Clone together with every other capability it serves; it
+never hides one. Voice Design produces an accepted voice-reference Resource, and Voice Clone uses
+that reference to produce independent speech. Hypit does not expose MiMo preset voices. When another
+selected Endpoint offers the same capability (a local WhisperX, a Vertex Gemini, or the official MiMo
+Provider), the Runtime Profile's `bindings` say which Endpoint serves it.
Execution policy remains local to this Provider. A profile may set `requestTimeoutMs`,
`operationTimeoutMs`, `uploadPartTimeoutMs`, `uploadPartAttempts`, `downloadAttempts`,
`geminiRateLimitAttempts` and `geminiRateLimitRetryDelayMs`; defaults are respectively 300 seconds,
20 minutes, 5 minutes, 3 attempts, 3 attempts, 4 attempts and 2 seconds. `defaultConcurrency`
-controls this Endpoint's shared Runtime capacity. Multipart concurrency is different: HypiHub
+controls the total shared capacity of this Profile's HypiHub pool. Optional `capabilityConcurrency`
+sets narrower group limits, for example `{ "seedance-2-mini": 2, "gemini": 3, "transcription": 1 }`.
+Image/video/speech groups use the exact capability name; all Gemini capabilities share `gemini`, and
+WhisperX uses `transcription`. These limits coordinate this Runtime's requests; HypiHub remains
+responsible for service-wide account limits. Immediate speech/Gemini/transcription slots cover the
+active HTTP invocation, while asynchronous image/video slots remain held until the remote job ends.
+
+For asynchronous jobs, a polling error retains the same job and its capacity. An operation deadline
+records failure but continues observing remote termination without downloading the output. Unknown
+submission acknowledgements are never automatically resubmitted. Runtime bindings never switch from
+a user's own Provider to HypiHub after a key, quota or transport failure.
+
+Multipart concurrency is different: HypiHub
selects it for one upload session, and it does not create a Build queue or a second Runtime pool.
diff --git a/packages/provider-hypihub/package.json b/packages/provider-hypihub/package.json
index 5825b163..d3a24e8c 100644
--- a/packages/provider-hypihub/package.json
+++ b/packages/provider-hypihub/package.json
@@ -19,7 +19,7 @@
},
"devDependencies": {
"@hypit/driver-node": "workspace:*",
- "@hypit/mimo-tts": "workspace:*",
+ "@hypit/mimo-speech": "workspace:*",
"@hypit/seedance": "workspace:*",
"@hypit/speech": "workspace:*"
}
diff --git a/packages/provider-hypihub/src/activation.ts b/packages/provider-hypihub/src/activation.ts
index c9e82d03..62779e5b 100644
--- a/packages/provider-hypihub/src/activation.ts
+++ b/packages/provider-hypihub/src/activation.ts
@@ -18,6 +18,7 @@ const adapter = createRuntimeEndpointAdapterFacet({
"baseUrl",
"apiKey",
"defaultConcurrency",
+ "capabilityConcurrency",
"pollIntervalMs",
"requestTimeoutMs",
"operationTimeoutMs",
@@ -47,12 +48,17 @@ const adapter = createRuntimeEndpointAdapterFacet({
const geminiRateLimitAttempts = runtimeConfigPositiveInteger(config.geminiRateLimitAttempts, "HypiHub geminiRateLimitAttempts");
const geminiRateLimitRetryDelayMs = runtimeConfigPositiveInteger(config.geminiRateLimitRetryDelayMs, "HypiHub geminiRateLimitRetryDelayMs");
const transcriptionModel = runtimeConfigString(config.transcriptionModel, "HypiHub transcriptionModel");
+ const capabilityConcurrency = config.capabilityConcurrency === undefined ? undefined : Object.fromEntries(
+ Object.entries(runtimeConfigObject(config.capabilityConcurrency, "HypiHub capabilityConcurrency"))
+ .map(([name, value]) => [name, runtimeConfigPositiveInteger(value, `HypiHub ${name} capacity`)!]),
+ );
const options = {
instance: context.instance,
pool: context.pool,
...(baseUrl === undefined ? {} : { baseUrl }),
apiKey,
...(defaultConcurrency === undefined ? {} : { defaultConcurrency }),
+ ...(capabilityConcurrency === undefined ? {} : { capabilityConcurrency }),
...(pollIntervalMs === undefined ? {} : { pollIntervalMs }),
...(requestTimeoutMs === undefined ? {} : { requestTimeoutMs }),
...(operationTimeoutMs === undefined ? {} : { operationTimeoutMs }),
diff --git a/packages/provider-hypihub/src/mapping.ts b/packages/provider-hypihub/src/mapping.ts
index eb57cd19..03d95e9a 100644
--- a/packages/provider-hypihub/src/mapping.ts
+++ b/packages/provider-hypihub/src/mapping.ts
@@ -8,7 +8,7 @@ const NANO_BANANA: ModuleRef = { name: "@hypit/nano-banana", version: "1" };
const SEEDREAM: ModuleRef = { name: "@hypit/seedream", version: "1" };
const MINIMAX: ModuleRef = { name: "@hypit/minimax-h3", version: "1" };
const GROK: ModuleRef = { name: "@hypit/grok-imagine", version: "1" };
-const MIMO_TTS: ModuleRef = { name: "@hypit/mimo-tts", version: "1" };
+const MIMO_SPEECH: ModuleRef = { name: "@hypit/mimo-speech", version: "1" };
const seedance = (name: string, model: string): GenerationWireMapping => ({
capability: { module: SEEDANCE, name }, result: "video", routes: [{ model }],
@@ -109,12 +109,20 @@ export const hypiHubMappings: readonly GenerationWireMapping[] = [
},
},
{
- capability: { module: MIMO_TTS, name: "mimo-v2.5-tts-voicedesign" }, result: "audio", routes: [{ model: "mimo-v2.5-tts-voicedesign" }],
+ capability: { module: MIMO_SPEECH, name: "mimo-v2.5-tts-voicedesign" }, result: "audio", routes: [{ model: "mimo-v2.5-tts-voicedesign" }],
fields: {
text: { as: "value", field: "input" },
voiceDescription: { as: "value", field: "voice_description" },
},
},
+ {
+ capability: { module: MIMO_SPEECH, name: "mimo-v2.5-tts-voiceclone" }, result: "audio", routes: [{ model: "mimo-v2.5-tts-voiceclone" }],
+ fields: {
+ text: { as: "value", field: "input" },
+ instruction: { as: "value", field: "prompt" },
+ voiceReference: { as: "urlArray", field: "reference_audio" },
+ },
+ },
];
function capabilityKey(ref: CapabilityRef): string {
diff --git a/packages/provider-hypihub/src/provider.ts b/packages/provider-hypihub/src/provider.ts
index 0e88b503..130a3675 100644
--- a/packages/provider-hypihub/src/provider.ts
+++ b/packages/provider-hypihub/src/provider.ts
@@ -27,6 +27,7 @@ export type CreateHypiHubProviderOptions = {
readonly baseUrl?: string;
readonly apiKey?: CredentialRef;
readonly defaultConcurrency?: number;
+ readonly capabilityConcurrency?: Readonly<Record<string, number>>;
readonly pollIntervalMs?: number;
readonly requestTimeoutMs?: number;
readonly operationTimeoutMs?: number;
@@ -92,6 +93,10 @@ async function verifyModelRoute(client: HypiHubClient, apiKey: string, model: st
`HypiHub model ${model} is not enabled for ${operation} with this API key`);
}
+class HypiHubHttpError extends Error {
+ constructor(readonly status: number, message: string) { super(message); }
+}
+
class HypiHubClient {
readonly baseUrl: string;
readonly timeout: number;
@@ -123,8 +128,8 @@ class HypiHubClient {
try {
const response = await this.fetcher(`${this.baseUrl}${path}`, { ...init, signal: controller.signal, headers: { authorization: `Bearer ${apiKey}`, ...(init.headers ?? {}) } });
const text = await response.text(); let body: unknown = {};
+ if (!response.ok) throw new HypiHubHttpError(response.status, `HypiHub returned HTTP ${response.status}: ${text.slice(0, 300)}`);
try { body = text.length === 0 ? {} : JSON.parse(text); } catch { throw new Error(`HypiHub returned invalid JSON (${response.status})`); }
- if (!response.ok) throw new Error(`HypiHub returned HTTP ${response.status}: ${text.slice(0, 300)}`);
return object(body, "HypiHub response");
} finally { clearTimeout(timer); }
}
@@ -158,18 +163,47 @@ class HypiHubClient {
body: JSON.stringify(body),
});
}
- async binary(path: string, apiKey: string, body: Record<string, unknown>): Promise<{ readonly bytes: Uint8Array; readonly mediaType: string }> {
+ async speech(apiKey: string, body: Record<string, unknown>): Promise<readonly { readonly bytes: Uint8Array; readonly mediaType: string }[]> {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), this.timeout);
try {
- const response = await this.fetcher(`${this.baseUrl}${path}`, {
+ const response = await this.fetcher(`${this.baseUrl}/audio/speech`, {
method: "POST",
headers: { authorization: `Bearer ${apiKey}`, "content-type": "application/json" },
- body: JSON.stringify(body),
+ body: JSON.stringify({ ...body, output: "b64_json" }),
signal: controller.signal,
});
- if (!response.ok) throw new Error(`HypiHub returned HTTP ${response.status}: ${(await response.text()).slice(0, 300)}`);
- return { bytes: new Uint8Array(await response.arrayBuffer()), mediaType: response.headers.get("content-type")?.split(";", 1)[0] ?? "audio/wav" };
+ const bytes = new Uint8Array(await response.arrayBuffer());
+ if (!response.ok) {
+ throw new Error(`HypiHub returned HTTP ${response.status}: ${Buffer.from(bytes).toString("utf8").slice(0, 300)}`);
+ }
+ const responseType = response.headers.get("content-type")?.split(";", 1)[0] ?? "audio/mpeg";
+ if (!responseType.includes("json")) return [{ bytes, mediaType: responseType }];
+
+ let decoded: unknown;
+ try {
+ decoded = JSON.parse(Buffer.from(bytes).toString("utf8"));
+ } catch {
+ throw new Error("HypiHub returned invalid audio JSON");
+ }
+ const root = object(decoded, "HypiHub audio response");
+ const values = Array.isArray(root.previews) ? root.previews : [root];
+ assert(values.length > 0, "HypiHub audio response contains no audio");
+ return await Promise.all(values.map(async (value, index) => {
+ const item = object(value, `HypiHub audio response ${index + 1}`);
+ const mediaType = typeof item.mime_type === "string" && item.mime_type.startsWith("audio/")
+ ? item.mime_type : "audio/mpeg";
+ if (typeof item.b64_json === "string" && item.b64_json.length > 0) {
+ assert(item.b64_json.length % 4 === 0 && /^[A-Za-z0-9+/]*={0,2}$/u.test(item.b64_json),
+ `HypiHub audio response ${index + 1} contains malformed Base64`);
+ const audio = Uint8Array.from(Buffer.from(item.b64_json, "base64"));
+ assert(audio.byteLength > 0, `HypiHub audio response ${index + 1} is empty`);
+ return { bytes: audio, mediaType };
+ }
+ assert(typeof item.url === "string" && /^https?:\/\//u.test(item.url),
+ `HypiHub audio response ${index + 1} contains neither audio bytes nor a URL`);
+ return await this.download(item.url);
+ }));
} finally {
clearTimeout(timer);
}
@@ -261,14 +295,16 @@ async function synthesizeAudio(client: HypiHubClient, context: EndpointInvocatio
? await client.upload(artifact, context.resources, credential(context))
: await publicAssetUrl(artifact, context.resources));
await verifyModelRoute(client, credential(context), compiled.model, "audio_speech");
- const audio = await client.binary("/audio/speech", credential(context), { model: compiled.model, ...(compiled.input as Record<string, unknown>) });
- const artifact = await context.resources.put(audio.bytes, audio.mediaType);
- return { value: route.packageResult([artifact]) };
+ const audio = await client.speech(credential(context), { model: compiled.model, ...(compiled.input as Record<string, unknown>) });
+ const artifacts = await Promise.all(audio.map(async (item) => await context.resources.put(item.bytes, item.mediaType)));
+ return { value: route.packageResult(artifacts) };
}
function endpoint(client: HypiHubClient, pollIntervalMs: number, maxOperationMs: number, publicAssetUrl: CreateHypiHubProviderOptions["publicAssetUrl"]): AsyncEndpoint {
return {
async start(context: EndpointStartContext) {
+ let submitting = false;
+ let remoteEnded = false;
try {
const route = hypiHubRouteForCapability(context.need.capability);
assert(route !== undefined, "HypiHub does not implement this exact capability"); const apiKey = credential(context);
@@ -289,32 +325,56 @@ function endpoint(client: HypiHubClient, pollIntervalMs: number, maxOperationMs:
const path = operation === "image_edits" ? "/images/edits"
: operation === "images" ? "/images/generations" : "/videos";
await verifyModelRoute(client, apiKey, compiled.model, operation);
+ submitting = true;
const response = await client.json(path, apiKey, { method: "POST", headers: { "content-type": "application/json", "idempotency-key": context.operation }, body: JSON.stringify({ model: compiled.model, ...input }) });
- const status = response.status; if (status === "succeeded" || status === "completed") return await complete(client, apiKey, route, jobId(response), context.resources);
+ const status = response.status;
+ remoteEnded = status === "succeeded" || status === "completed";
+ if (remoteEnded) return await complete(client, apiKey, route, jobId(response), context.resources);
const handle: Handle = { contract: "hypit.hypihub-operation@1", jobId: jobId(response), route: capabilityKey(route.capability), startedAt: Date.now() };
return wakeAfter(canonicalize(handle), pollIntervalMs, Date.now(), { phase: "submitted" });
- } catch (error) { return failure(error); }
+ } catch (error) {
+ if (submitting && !remoteEnded && !(error instanceof HypiHubHttpError && error.status >= 400 && error.status < 500)) {
+ return { status: "pending", wakeAt: Date.now() + 30_000, progress: { phase: "submission-unknown" },
+ failure: { code: "SUBMISSION_UNKNOWN", message: "HypiHub submission acknowledgement is unknown; the same Build will not submit again" } };
+ }
+ return failure(error);
+ }
},
async poll(context: EndpointPollContext) {
+ let remoteEnded = false;
try {
const handle = object(context.handle, "HypiHub handle") as unknown as Handle; const route = hypiHubRouteForCapability(context.need.capability);
assert(route !== undefined && handle.contract === "hypit.hypihub-operation@1" && handle.route === capabilityKey(route.capability), "HypiHub handle is invalid");
- if (Date.now() - handle.startedAt > maxOperationMs) throw new Error("HypiHub operation timed out");
+ if (!context.settling && Date.now() - handle.startedAt > maxOperationMs) {
+ return { ...wakeAfter(canonicalize(handle), pollIntervalMs, Date.now(), { phase: "settling" }),
+ failure: { code: "HYPIHUB_OPERATION_TIMEOUT", message: "HypiHub operation timed out; waiting for remote termination" } };
+ }
const job = await client.json(`/jobs/${encodeURIComponent(handle.jobId)}`, credential(context)); const status = job.status;
if (status === "queued" || status === "running" || status === "in_progress") return wakeAfter(canonicalize(handle), pollIntervalMs, Date.now(), { phase: String(status) });
- if (status === "failed" || status === "queue_expired") {
+ remoteEnded = ["failed", "queue_expired", "cancelled", "succeeded", "completed"].includes(String(status));
+ if (remoteEnded && context.settling) return { status: "settled" };
+ if (status === "failed" || status === "queue_expired" || status === "cancelled") {
const detail = [job.error, job.message, job.reason, job.detail]
.find((value) => typeof value === "string" && value.length > 0);
throw new Error(`HypiHub job ${status}${typeof detail === "string" ? `: ${detail}` : ""}`);
}
if (status !== "succeeded" && status !== "completed") throw new Error(`HypiHub returned unknown job status ${String(status)}`);
return await complete(client, credential(context), route, handle.jobId, context.resources);
- } catch (error) { return failure(error); }
+ } catch (error) {
+ if (remoteEnded) return failure(error);
+ return wakeAfter(context.handle, pollIntervalMs, Date.now(), { phase: "poll-unavailable" });
+ }
},
};
}
export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}) {
+ const capacityNames = new Set([...hypiHubRoutes.map((route) => route.capability.name), "gemini", "transcription"]);
+ const capabilityConcurrency = options.capabilityConcurrency ?? {};
+ for (const [name, limit] of Object.entries(capabilityConcurrency)) {
+ assert(capacityNames.has(name), `unknown HypiHub capacity ${name}`);
+ assert(Number.isSafeInteger(limit) && limit > 0, `HypiHub ${name} capacity must be a positive integer`);
+ }
const requestTimeoutMs = options.requestTimeoutMs ?? 300_000;
const operationTimeoutMs = options.operationTimeoutMs ?? 20 * 60_000;
const uploadPartTimeoutMs = options.uploadPartTimeoutMs ?? 5 * 60_000;
@@ -424,14 +484,15 @@ export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}
capabilities: [
...hypiHubRoutes
.map((route) => route.media === "audio"
- ? { capability: route.capability, returns: route.returns, lifecycle: "immediate" as const, handler: audioEndpoint, capacity: route.capability.name }
- : { capability: route.capability, returns: route.returns, lifecycle: "asynchronous" as const, endpoint: asyncEndpoint, capacity: route.capability.name }),
+ ? { capability: route.capability, returns: route.returns, lifecycle: "immediate" as const, handler: audioEndpoint, capacity: route.capability.name, ...(capabilityConcurrency[route.capability.name] === undefined ? {} : { maxConcurrency: capabilityConcurrency[route.capability.name]! }) }
+ : { capability: route.capability, returns: route.returns, lifecycle: "asynchronous" as const, endpoint: asyncEndpoint, capacity: route.capability.name, ...(capabilityConcurrency[route.capability.name] === undefined ? {} : { maxConcurrency: capabilityConcurrency[route.capability.name]! }) }),
...geminiModels.map((model) => ({
capability: geminiCapabilities[model],
returns: geminiTypes.visualObservation,
lifecycle: "immediate" as const,
handler: geminiEndpoint,
capacity: "gemini",
+ ...(capabilityConcurrency.gemini === undefined ? {} : { maxConcurrency: capabilityConcurrency.gemini }),
})),
{
capability: whisperXCapabilities.alignment,
@@ -439,6 +500,7 @@ export function createHypiHubProvider(options: CreateHypiHubProviderOptions = {}
lifecycle: "immediate" as const,
handler: whisperXEndpoint,
capacity: "transcription",
+ ...(capabilityConcurrency.transcription === undefined ? {} : { maxConcurrency: capabilityConcurrency.transcription }),
},
],
});
diff --git a/packages/provider-hypihub/test/provider.test.ts b/packages/provider-hypihub/test/provider.test.ts
index 65136900..d5011978 100644
--- a/packages/provider-hypihub/test/provider.test.ts
+++ b/packages/provider-hypihub/test/provider.test.ts
@@ -6,7 +6,7 @@ import { defineEndpointPackage } from "@hypit/endpoint-kit";
import type { AsyncEndpoint } from "@hypit/endpoint-kit";
import { geminiCapabilities, geminiTypes, sealGeminiRequest } from "@hypit/gemini";
import type { CanonicalValue, Need } from "@hypit/protocol";
-import { mimoTtsEndpoints } from "@hypit/mimo-tts";
+import { mimoSpeechEndpoints, sealMimoSpeechRequest } from "@hypit/mimo-speech";
import { sealSeedanceRequest, seedanceEndpoints } from "@hypit/seedance";
import { sealSpeechEvidenceAudio } from "@hypit/speech";
import { speechEvidenceTypes } from "@hypit/speech-evidence";
@@ -77,17 +77,17 @@ test("HypiHub doctor checks the authenticated catalogue only when actively invok
assert.deepEqual(diagnostics, []);
});
-test("HypiHub declares VoiceDesign like everything else it can do; who serves it is the Profile's binding", async () => {
+test("HypiHub declares both MiMo speech capabilities; who serves them is the Profile's binding", async () => {
const registry = new EndpointRegistry();
await createHypiHubProvider({ fetch: async () => { throw new Error("audio must not call fetch"); } }).install(registry);
- const need = {
- id: "need:hypihub-audio-default",
- capability: mimoTtsEndpoints.voiceDesign.capability,
- returns: mimoTtsEndpoints.voiceDesign.returns,
+ const needs = Object.values(mimoSpeechEndpoints).map((endpoint) => ({
+ id: `need:hypihub-${endpoint.key}`,
+ capability: endpoint.capability,
+ returns: endpoint.returns,
constraints: { ports: {} },
- result: "record:hypihub-audio-default",
- };
- assert.equal(registry.resolve(need).status, "resolved");
+ result: `record:hypihub-${endpoint.key}`,
+ }));
+ assert.ok(needs.every((item) => registry.resolve(item).status === "resolved"));
// A second Endpoint offering the same capability makes the choice the deployment's, not the Provider's.
const other = defineEndpointPackage({
@@ -96,20 +96,85 @@ test("HypiHub declares VoiceDesign like everything else it can do; who serves it
instance: "mimo.official",
pool: "mimo.official",
capabilities: [{
- capability: mimoTtsEndpoints.voiceDesign.capability,
- returns: mimoTtsEndpoints.voiceDesign.returns,
+ capability: mimoSpeechEndpoints.voiceDesign.capability,
+ returns: mimoSpeechEndpoints.voiceDesign.returns,
lifecycle: "immediate",
handler: () => ({ value: { kind: "inline", value: null } }),
}],
});
await other.install(registry);
- assert.equal(registry.resolve(need).status, "ambiguous");
- registry.bind(mimoTtsEndpoints.voiceDesign.capability, "mimo.official");
- const resolved = registry.resolve(need);
+ assert.equal(registry.resolve(needs[0]!).status, "ambiguous");
+ registry.bind(mimoSpeechEndpoints.voiceDesign.capability, "mimo.official");
+ const resolved = registry.resolve(needs[0]!);
assert.equal(resolved.status, "resolved");
assert.equal(resolved.status === "resolved" ? resolved.registration.id : undefined, "mimo.official");
});
+test("HypiHub stores both preview JSON and ordinary speech JSON as audio Resources", async () => {
+ const resources = new MemoryResourceStore();
+ const voiceReference = await resources.put(new Uint8Array([1, 2, 3, 4]), "audio/wav");
+ const submitted: Record<string, unknown>[] = [];
+ const provider = createHypiHubProvider({
+ publicAssetUrl: async (artifact) => {
+ assert.equal(artifact.resource, voiceReference.resource);
+ return "https://hypit.ai/assets/voice.wav";
+ },
+ fetch: async (input, init) => {
+ const url = String(input);
+ if (url.includes("/models/")) return Response.json({ endpoints: ["audio_speech"] });
+ assert.match(url, /\/audio\/speech$/u);
+ const body = JSON.parse(String(init?.body)) as Record<string, unknown>;
+ submitted.push(body);
+ const data = Buffer.from([9, 8, 7, submitted.length]).toString("base64");
+ return submitted.length === 1
+ ? Response.json({ object: "audio.voice_previews", previews: [{ b64_json: data, mime_type: "audio/wav" }] })
+ : Response.json({ object: "audio.speech", b64_json: data, mime_type: "audio/wav" });
+ },
+ });
+ const registry = new EndpointRegistry();
+ await provider.install(registry);
+ const cases = [
+ {
+ endpoint: mimoSpeechEndpoints.voiceDesign,
+ constraints: sealMimoSpeechRequest("mimo-v2.5-tts-voicedesign", {
+ text: ["A short voice sample."], voiceDescription: ["Warm and confident."],
+ }),
+ },
+ {
+ endpoint: mimoSpeechEndpoints.voiceClone,
+ constraints: sealMimoSpeechRequest("mimo-v2.5-tts-voiceclone", {
+ text: ["Independent narration."], instruction: ["Quietly direct."],
+ voiceReference: [{ role: "audio", artifact: voiceReference }],
+ }),
+ },
+ ];
+ for (const [index, item] of cases.entries()) {
+ const request = {
+ id: `need:hypihub-speech-${index}`,
+ capability: item.endpoint.capability,
+ returns: item.endpoint.returns,
+ constraints: item.constraints as unknown as CanonicalValue,
+ result: `record:hypihub-speech-${index}`,
+ };
+ const resolution = registry.resolve(request);
+ assert.equal(resolution.status, "resolved");
+ assert.equal(resolution.registration.kind, "immediate");
+ const result = await resolution.registration.handler({
+ command: { kind: "fulfill-need", id: `command:hypihub-speech-${index}`, need: request },
+ need: request,
+ resources,
+ credentials: { apiKey: { secret: "test-key" } },
+ });
+ assert.equal(result.value.kind, "inline");
+ const set = result.value.kind === "inline" ? result.value.value as Record<string, unknown> : {};
+ const audios = set.audios as Array<{ resource: `res_${string}` }>;
+ assert.equal(await resources.has(audios[0]!.resource), true);
+ }
+ assert.equal(submitted[0]!.output, "b64_json");
+ assert.deepEqual(submitted[1]!.reference_audio, ["https://hypit.ai/assets/voice.wav"]);
+ assert.equal(submitted[1]!.prompt, "Quietly direct.");
+});
+
test("HypiHub uploads one referenced Resource once and submits its HTTPS URL", async () => {
const resources = new MemoryResourceStore();
const referenceBytes = new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]);
@@ -217,7 +282,7 @@ test("HypiHub uploads one referenced Resource once and submits its HTTPS URL", a
assert.deepEqual(signedBatches, [[1, 2, 3], [2]]);
assert.deepEqual([...partAttempts.entries()].sort(), [[1, 1], [2, 2], [3, 1]]);
if (started.status !== "pending") return;
- const completed = await endpoint.poll({ ...common, handle: started.handle });
+ const completed = await endpoint.poll({ ...common, handle: started.handle! });
assert.equal(completed.status, "completed");
});
@@ -458,3 +523,52 @@ test("HypiHub fulfills the Provider-neutral WhisperX alignment capability", asyn
chars: [],
}] } });
});
+
+test("HypiHub exposes model groups beneath its own total capacity", async () => {
+ const registry = new EndpointRegistry();
+ await createHypiHubProvider({ pool: "hub-account", defaultConcurrency: 8,
+ capabilityConcurrency: { "seedance-2-mini": 2, gemini: 3, transcription: 1 } }).install(registry);
+ const requests = [
+ need({}),
+ { ...need({}), capability: Object.values(geminiCapabilities)[0]!, returns: geminiTypes.visualObservation },
+ { ...need({}), capability: whisperXCapabilities.alignment, returns: speechEvidenceTypes.alignedTranscript },
+ ];
+ for (const [index, request] of requests.entries()) {
+ const selected = registry.resolve(request);
+ assert.equal(selected.status, "resolved");
+ assert.deepEqual(selected.registration.scheduling?.resources, [
+ { id: "pool:hub-account", limit: 8 },
+ { id: `capacity:hub-account/${["seedance-2-mini", "gemini", "transcription"][index]}`, limit: [2, 3, 1][index] },
+ ]);
+ }
+ assert.throws(() => createHypiHubProvider({ capabilityConcurrency: { invented: 1 } }), /unknown HypiHub capacity/);
+});
+
+test("HypiHub polling errors and local deadlines retain remote work until settlement", async () => {
+ const request = need({});
+ let unavailable = true, status = "running", requests = 0;
+ const registry = new EndpointRegistry();
+ await createHypiHubProvider({ pollIntervalMs: 0, operationTimeoutMs: 10,
+ fetch: async (url) => {
+ requests++; assert.match(String(url), /\/jobs\/test$/);
+ if (unavailable) throw new Error("offline");
+ return Response.json({ status });
+ } }).install(registry);
+ const selected = registry.resolve(request);
+ assert.equal(selected.status, "resolved"); assert.equal(selected.registration.kind, "asynchronous");
+ const endpoint = selected.registration.endpoint;
+ const common = { command: { kind: "fulfill-need", id: "need:test", need: request } as const,
+ need: request, resources: new MemoryResourceStore(), credentials: { apiKey: { secret: "test" } }, operation: "op:test",
+ handle: { contract: "hypit.hypihub-operation@1", jobId: "test",
+ route: `${request.capability.module.name}@${request.capability.module.version}#${request.capability.name}`, startedAt: 0 } };
+ const timedOut = await endpoint.poll(common);
+ assert.equal(timedOut.status, "pending");
+ assert.equal(timedOut.status === "pending" && timedOut.failure?.code, "HYPIHUB_OPERATION_TIMEOUT");
+ assert.equal(requests, 0);
+ assert.equal((await endpoint.poll({ ...common, settling: true })).status, "pending");
+ unavailable = false;
+ assert.equal((await endpoint.poll({ ...common, settling: true })).status, "pending");
+ status = "completed";
+ assert.deepEqual(await endpoint.poll({ ...common, settling: true }), { status: "settled" });
+ assert.equal(requests, 3, "settling must only query the existing job and never download or submit");
+});
diff --git a/packages/provider-hypihub/test/routes.test.ts b/packages/provider-hypihub/test/routes.test.ts
index 740df5a8..f0a3faa1 100644
--- a/packages/provider-hypihub/test/routes.test.ts
+++ b/packages/provider-hypihub/test/routes.test.ts
@@ -140,7 +140,7 @@ test("HypiHub uses ref_video_url for one video and the public array for multiple
});
});
-test("HypiHub MiMo TTS mappings use the public audio speech fields", async () => {
+test("HypiHub MiMo Speech mappings use the public audio speech fields", async () => {
const voiceDesign = hypiHubRoutes.find((item) => item.capability.name === "mimo-v2.5-tts-voicedesign");
assert.ok(voiceDesign);
const result = await voiceDesign.compile({
@@ -148,4 +148,20 @@ test("HypiHub MiMo TTS mappings use the public audio speech fields", async () =>
}, resolve);
assert.equal(result.model, "mimo-v2.5-tts-voicedesign");
assert.deepEqual(result.input, { input: "hello", voice_description: "warm and calm" });
+
+ const voiceClone = hypiHubRoutes.find((item) => item.capability.name === "mimo-v2.5-tts-voiceclone");
+ assert.ok(voiceClone);
+ const cloned = await voiceClone.compile({
+ ports: {
+ text: ["hello again"],
+ instruction: ["quiet and direct"],
+ voiceReference: [{ role: "audio", artifact: { ...image, mediaType: "audio/wav" } }],
+ },
+ }, resolveAudio);
+ assert.equal(cloned.model, "mimo-v2.5-tts-voiceclone");
+ assert.deepEqual(cloned.input, {
+ input: "hello again",
+ prompt: "quiet and direct",
+ reference_audio: ["data:audio/wav;base64,AQID"],
+ });
});
diff --git a/packages/provider-xiaomi-mimo/README.md b/packages/provider-xiaomi-mimo/README.md
index 602d9dce..e890d36e 100644
--- a/packages/provider-xiaomi-mimo/README.md
+++ b/packages/provider-xiaomi-mimo/README.md
@@ -1,10 +1,10 @@
# `@hypit/provider-xiaomi-mimo`
-Immediate Runtime Endpoint for Xiaomi's official MiMo V2.5 VoiceDesign API.
+Immediate Runtime Endpoint for Xiaomi's official MiMo V2.5 Voice Design and Voice Clone APIs.
This package owns the `chat/completions` wire shape, `api-key` credential, timeout, response bounds,
-and ResourceStore ingestion. It does not import `@hypit/mimo-tts`: the VoiceDesign capability and
-port names are bound as versioned data, keeping the exact model contract independent from this service.
+voice-reference encoding and ResourceStore ingestion. It names `@hypit/mimo-speech@1` capabilities
+as versioned data and does not import the model package at runtime.
Runtime Profile example:
@@ -23,9 +23,10 @@ Runtime Profile example:
}
```
-The Provider always requests final WAV bytes. Streaming is a service transport optimization and is
-not part of the author model.
+Voice Design sends the authored voice description. Voice Clone reads the selected audio Resource,
+checks Xiaomi's accepted MP3/WAV formats and sends it as the vendor's data-URL voice reference.
+The Provider always requests final WAV bytes.
-Selecting the Provider in the Runtime Profile activates it independently from author model syntax.
-It declares total and exact-capability concurrency resources. The shared Runtime coordinates those
-claims across Builds; this package does not create a private scheduler.
+Selecting this Provider in the Runtime Profile is independent from author syntax. Its pool and
+capability claims participate in the shared Runtime capacity model; the package has no private
+scheduler.
diff --git a/packages/provider-xiaomi-mimo/package.json b/packages/provider-xiaomi-mimo/package.json
index b5d65a3b..024877c1 100644
--- a/packages/provider-xiaomi-mimo/package.json
+++ b/packages/provider-xiaomi-mimo/package.json
@@ -19,6 +19,6 @@
},
"devDependencies": {
"@hypit/driver-node": "workspace:*",
- "@hypit/mimo-tts": "workspace:*"
+ "@hypit/mimo-speech": "workspace:*"
}
}
diff --git a/packages/provider-xiaomi-mimo/src/activation.ts b/packages/provider-xiaomi-mimo/src/activation.ts
index ec71b2a4..3c6b6873 100644
--- a/packages/provider-xiaomi-mimo/src/activation.ts
+++ b/packages/provider-xiaomi-mimo/src/activation.ts
@@ -16,7 +16,7 @@ const adapter = createRuntimeEndpointAdapterFacet({
const config = runtimeConfigObject(context.config, "Xiaomi MiMo");
runtimeConfigExact(config, [
"apiBaseUrl", "apiKey", "defaultConcurrency", "requestTimeoutMs",
- "maxResponseBytes",
+ "maxResponseBytes", "maxVoiceReferenceBase64Bytes",
], "Xiaomi MiMo");
const apiBaseUrl = runtimeConfigString(config.apiBaseUrl, "Xiaomi MiMo apiBaseUrl");
if (apiBaseUrl !== undefined) {
@@ -30,6 +30,10 @@ const adapter = createRuntimeEndpointAdapterFacet({
const defaultConcurrency = runtimeConfigPositiveInteger(config.defaultConcurrency, "Xiaomi MiMo defaultConcurrency");
const requestTimeoutMs = runtimeConfigPositiveInteger(config.requestTimeoutMs, "Xiaomi MiMo requestTimeoutMs");
const maxResponseBytes = runtimeConfigPositiveInteger(config.maxResponseBytes, "Xiaomi MiMo maxResponseBytes");
+ const maxVoiceReferenceBase64Bytes = runtimeConfigPositiveInteger(
+ config.maxVoiceReferenceBase64Bytes,
+ "Xiaomi MiMo maxVoiceReferenceBase64Bytes",
+ );
return {
endpoint: createXiaomiMimoProvider({
instance: context.instance,
@@ -39,6 +43,7 @@ const adapter = createRuntimeEndpointAdapterFacet({
...(defaultConcurrency === undefined ? {} : { defaultConcurrency }),
...(requestTimeoutMs === undefined ? {} : { requestTimeoutMs }),
...(maxResponseBytes === undefined ? {} : { maxResponseBytes }),
+ ...(maxVoiceReferenceBase64Bytes === undefined ? {} : { maxVoiceReferenceBase64Bytes }),
}),
};
},
diff --git a/packages/provider-xiaomi-mimo/src/provider.ts b/packages/provider-xiaomi-mimo/src/provider.ts
index 20c18e13..bd7b328c 100644
--- a/packages/provider-xiaomi-mimo/src/provider.ts
+++ b/packages/provider-xiaomi-mimo/src/provider.ts
@@ -4,15 +4,14 @@ import {
generationTypes,
sealGeneratedAudioSet,
} from "@hypit/generation";
-import type { GenerationRequest } from "@hypit/generation";
-import { canonicalize } from "@hypit/protocol";
+import type { GenerationMediaValue, GenerationRequest } from "@hypit/generation";
import type { CanonicalValue, CapabilityRef } from "@hypit/protocol";
import { credentialRef } from "@hypit/runtime";
import type { CredentialRef } from "@hypit/runtime";
export const xiaomiMimoProviderModuleRef = { name: "@hypit/provider-xiaomi-mimo", version: "1" } as const;
-const mimoModelModule = { name: "@hypit/mimo-tts", version: "1" } as const;
-const modelNames = ["mimo-v2.5-tts-voicedesign"] as const;
+const mimoModelModule = { name: "@hypit/mimo-speech", version: "1" } as const;
+const modelNames = ["mimo-v2.5-tts-voicedesign", "mimo-v2.5-tts-voiceclone"] as const;
type Model = typeof modelNames[number];
const capabilities = Object.fromEntries(modelNames.map((name) => [name, {
@@ -30,6 +29,7 @@ export type CreateXiaomiMimoProviderOptions = {
readonly defaultConcurrency?: number;
readonly requestTimeoutMs?: number;
readonly maxResponseBytes?: number;
+ readonly maxVoiceReferenceBase64Bytes?: number;
readonly fetch?: Fetch;
};
@@ -52,7 +52,9 @@ function normalizeBaseUrl(value: string): string {
function request(value: CanonicalValue, model: Model): GenerationRequest {
const result = value as unknown as GenerationRequest;
assert(result.ports !== null && typeof result.ports === "object", "Xiaomi MiMo request has no ports");
- const allowed = new Set(["text", "voiceDescription"]);
+ const allowed = model === "mimo-v2.5-tts-voicedesign"
+ ? new Set(["text", "voiceDescription"])
+ : new Set(["text", "instruction", "voiceReference"]);
assert(Object.keys(result.ports).every((name) => allowed.has(name)),
`${model} request contains an unsupported port`);
return result;
@@ -74,6 +76,32 @@ function credential(context: EndpointInvocationContext): string {
return value;
}
+async function readVoiceReference(
+ context: EndpointInvocationContext,
+ req: GenerationRequest,
+ maxBase64Bytes: number,
+): Promise<string> {
+ const values = req.ports.voiceReference;
+ assert(values?.length === 1, "MiMo Voice Clone requires exactly one voiceReference");
+ const value = values[0] as GenerationMediaValue;
+ assert(value.role === "audio", "MiMo Voice Clone voiceReference must be audio");
+ const supported = new Set(["audio/mpeg", "audio/mp3", "audio/wav", "audio/x-wav"]);
+ assert(supported.has(value.artifact.mediaType), "Xiaomi MiMo accepts only MP3 or WAV voice references");
+ const encodedSize = 4 * Math.ceil(value.artifact.size / 3);
+ assert(encodedSize <= maxBase64Bytes,
+ `MiMo voice reference exceeds the configured ${maxBase64Bytes}-byte Base64 limit`);
+ const bytes = await context.resources.get(value.artifact.resource);
+ assert(bytes !== undefined, `MiMo voice reference ${value.artifact.resource} is unavailable`);
+ assert(bytes.byteLength === value.artifact.size, "MiMo voice reference size differs from its BlobRef");
+ const mediaType = value.artifact.mediaType === "audio/x-wav"
+ ? "audio/wav"
+ : value.artifact.mediaType === "audio/mp3" ? "audio/mpeg" : value.artifact.mediaType;
+ const encoded = Buffer.from(bytes).toString("base64");
+ assert(Buffer.byteLength(encoded, "utf8") <= maxBase64Bytes,
+ `MiMo voice reference exceeds the configured ${maxBase64Bytes}-byte Base64 limit`);
+ return `data:${mediaType};base64,${encoded}`;
+}
+
function parseAudio(text: string, maxAudioBytes: number): Uint8Array {
let raw: unknown;
try {
@@ -127,6 +155,10 @@ export function createXiaomiMimoProvider(options: CreateXiaomiMimoProviderOption
const apiBaseUrl = normalizeBaseUrl(options.apiBaseUrl ?? "https://api.xiaomimimo.com/v1");
const requestTimeoutMs = positiveInteger(options.requestTimeoutMs ?? 180_000, "requestTimeoutMs");
const maxResponseBytes = positiveInteger(options.maxResponseBytes ?? 64 * 1024 * 1024, "maxResponseBytes");
+ const maxVoiceReferenceBase64Bytes = positiveInteger(
+ options.maxVoiceReferenceBase64Bytes ?? 10_000_000,
+ "maxVoiceReferenceBase64Bytes",
+ );
const fetcher = options.fetch ?? globalThis.fetch;
const endpointCapabilities = modelNames.map((model) => ({
capability: capabilities[model],
@@ -135,10 +167,15 @@ export function createXiaomiMimoProvider(options: CreateXiaomiMimoProviderOption
handler: async (context: EndpointInvocationContext) => {
const req = request(context.need.constraints, model);
const text = scalar(req, model, "text")!;
- const messages: Array<{ role: "user" | "assistant"; content: string }> = [
- { role: "user", content: scalar(req, model, "voiceDescription")! },
- ];
+ const messages: Array<{ role: "user" | "assistant"; content: string }> = [];
const audio: Record<string, CanonicalValue> = { format: "wav" };
+ if (model === "mimo-v2.5-tts-voicedesign") {
+ messages.push({ role: "user", content: scalar(req, model, "voiceDescription")! });
+ } else {
+ const instruction = scalar(req, model, "instruction", false);
+ if (instruction !== undefined) messages.push({ role: "user", content: instruction });
+ audio.voice = await readVoiceReference(context, req, maxVoiceReferenceBase64Bytes);
+ }
messages.push({ role: "assistant", content: text });
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(new Error("Xiaomi MiMo request timed out")), requestTimeoutMs);
@@ -165,7 +202,7 @@ export function createXiaomiMimoProvider(options: CreateXiaomiMimoProviderOption
return defineEndpointPackage({
module: xiaomiMimoProviderModuleRef,
- facet: "tts",
+ facet: "speech",
instance: options.instance ?? "xiaomi-mimo.default",
pool: options.pool ?? options.instance ?? "xiaomi-mimo.default",
pricing: { kind: "page", url: "https://mimo.mi.com/docs/en-US/pricing" },
diff --git a/packages/provider-xiaomi-mimo/test/provider.test.ts b/packages/provider-xiaomi-mimo/test/provider.test.ts
index 4ef894e3..1903c07a 100644
--- a/packages/provider-xiaomi-mimo/test/provider.test.ts
+++ b/packages/provider-xiaomi-mimo/test/provider.test.ts
@@ -3,60 +3,103 @@ import test from "node:test";
import { EndpointRegistry, MemoryResourceStore } from "@hypit/driver-node";
import { generationTypes } from "@hypit/generation";
-import { mimoTtsEndpoints, sealMimoTtsRequest } from "@hypit/mimo-tts";
+import { mimoSpeechEndpoints, sealMimoSpeechRequest } from "@hypit/mimo-speech";
import type { CanonicalValue, Need } from "@hypit/protocol";
import { createXiaomiMimoProvider } from "@hypit/provider-xiaomi-mimo";
-function need(constraints: CanonicalValue): Need {
+function need(
+ endpoint: (typeof mimoSpeechEndpoints)[keyof typeof mimoSpeechEndpoints],
+ constraints: CanonicalValue,
+ id: string,
+): Need {
return {
- id: "need:mimo-voicedesign",
- capability: mimoTtsEndpoints.voiceDesign.capability,
- returns: mimoTtsEndpoints.voiceDesign.returns,
+ id: `need:${id}`,
+ capability: endpoint.capability,
+ returns: endpoint.returns,
constraints,
- result: "record:mimo-voicedesign",
+ result: `record:${id}`,
};
}
-test("the official Provider maps only the VoiceDesign contract", async () => {
- const resources = new MemoryResourceStore();
- const provider = createXiaomiMimoProvider({
- fetch: async (input, init) => {
- assert.equal(String(input), "https://api.xiaomimimo.com/v1/chat/completions");
- assert.equal(new Headers(init?.headers).get("api-key"), "test-key");
- const body = JSON.parse(String(init?.body)) as Record<string, unknown>;
- assert.equal(body.model, "mimo-v2.5-tts-voicedesign");
- assert.deepEqual(body.audio, { format: "wav" });
- assert.deepEqual(body.messages, [
- { role: "user", content: "A clear, grounded female voice." },
- { role: "assistant", content: "Keep every authored word." },
- ]);
- return Response.json({ choices: [{ message: { audio: { data: Buffer.from([9, 8, 7]).toString("base64") } } }] });
- },
- });
+async function invoke(request: Need, resources: MemoryResourceStore, fetch: typeof globalThis.fetch) {
+ const provider = createXiaomiMimoProvider({ fetch });
const registry = new EndpointRegistry();
await provider.install(registry);
- const request = need(sealMimoTtsRequest("mimo-v2.5-tts-voicedesign", {
- text: ["Keep every authored word."],
- voiceDescription: ["A clear, grounded female voice."],
- }) as unknown as CanonicalValue);
const resolution = registry.resolve(request);
assert.equal(resolution.status, "resolved");
assert.equal(resolution.registration.kind, "immediate");
- const result = await resolution.registration.handler({
- command: { kind: "fulfill-need", id: "command:mimo-voicedesign", need: request },
+ return await resolution.registration.handler({
+ command: { kind: "fulfill-need", id: `command:${request.id}`, need: request },
need: request,
resources,
credentials: { apiKey: { secret: "test-key" } },
});
- assert.equal(result.value.kind, "inline");
- const set = result.value.kind === "inline" ? result.value.value as Record<string, unknown> : {};
- const audios = set.audios as Array<{ mediaType: string; resource: string }>;
- assert.equal(audios[0]?.mediaType, "audio/wav");
- assert.equal(await resources.has(audios[0]!.resource as `res_${string}`), true);
+}
+
+function audioResponse(bytes: Uint8Array) {
+ return Response.json({ choices: [{ message: { audio: { data: Buffer.from(bytes).toString("base64") } } }] });
+}
+
+test("the official Provider maps Voice Design and Voice Clone to Xiaomi's wire", async () => {
+ const resources = new MemoryResourceStore();
+ const voiceReference = await resources.put(new Uint8Array([1, 2, 3, 4]), "audio/wav");
+ const cases = [
+ {
+ endpoint: mimoSpeechEndpoints.voiceDesign,
+ request: sealMimoSpeechRequest("mimo-v2.5-tts-voicedesign", {
+ text: ["Keep every authored word."],
+ voiceDescription: ["A clear, grounded female voice."],
+ }),
+ assertBody(body: Record<string, unknown>) {
+ assert.deepEqual(body.audio, { format: "wav" });
+ assert.deepEqual(body.messages, [
+ { role: "user", content: "A clear, grounded female voice." },
+ { role: "assistant", content: "Keep every authored word." },
+ ]);
+ },
+ },
+ {
+ endpoint: mimoSpeechEndpoints.voiceClone,
+ request: sealMimoSpeechRequest("mimo-v2.5-tts-voiceclone", {
+ text: ["Keep every authored word."],
+ instruction: ["Calm and restrained."],
+ voiceReference: [{ role: "audio", artifact: voiceReference }],
+ }),
+ assertBody(body: Record<string, unknown>) {
+ const audio = body.audio as Record<string, string>;
+ assert.equal(audio.format, "wav");
+ assert.match(audio.voice!, /^data:audio\/wav;base64,/u);
+ assert.deepEqual(body.messages, [
+ { role: "user", content: "Calm and restrained." },
+ { role: "assistant", content: "Keep every authored word." },
+ ]);
+ },
+ },
+ ];
+
+ for (const [index, item] of cases.entries()) {
+ const result = await invoke(
+ need(item.endpoint, item.request as unknown as CanonicalValue, String(index)),
+ resources,
+ async (input, init) => {
+ assert.equal(String(input), "https://api.xiaomimimo.com/v1/chat/completions");
+ assert.equal(new Headers(init?.headers).get("api-key"), "test-key");
+ const body = JSON.parse(String(init?.body)) as Record<string, unknown>;
+ assert.equal(body.model, item.endpoint.capability.name);
+ item.assertBody(body);
+ return audioResponse(new Uint8Array([9, 8, 7, index]));
+ },
+ );
+ assert.equal(result.value.kind, "inline");
+ const set = result.value.kind === "inline" ? result.value.value as Record<string, unknown> : {};
+ const audios = set.audios as Array<{ mediaType: string; resource: string }>;
+ assert.equal(audios[0]?.mediaType, "audio/wav");
+ assert.equal(await resources.has(audios[0]!.resource as `res_${string}`), true);
+ }
});
-test("Provider configuration owns credentials and queue policy, not model semantics", () => {
+test("Provider configuration owns credentials and shared pool policy, not model semantics", () => {
const provider = createXiaomiMimoProvider({ defaultConcurrency: 3 });
- assert.equal(provider.offers.length, 1);
- assert.equal(provider.offers[0]?.returns.name, generationTypes.audioSet.name);
+ assert.equal(provider.offers.length, 2);
+ assert.ok(provider.offers.every((offer) => offer.returns.name === generationTypes.audioSet.name));
});
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 1830cae2..6d8d4b81 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -114,9 +114,9 @@ importers:
'@hypit/media-track-studio':
specifier: workspace:*
version: link:packages/media-track-studio
- '@hypit/mimo-tts':
+ '@hypit/mimo-speech':
specifier: workspace:*
- version: link:packages/mimo-tts
+ version: link:packages/mimo-speech
'@hypit/minimax-h3':
specifier: workspace:*
version: link:packages/minimax-h3
@@ -1772,7 +1772,7 @@ importers:
specifier: workspace:*
version: link:../studio-adapter
- packages/mimo-tts:
+ packages/mimo-speech:
dependencies:
'@hypit/artifact':
specifier: workspace:*
@@ -1936,6 +1936,12 @@ importers:
packages/provider-hyperframes-local:
dependencies:
+ '@hyperframes/engine':
+ specifier: 0.7.101
+ version: 0.7.101
+ '@hyperframes/producer':
+ specifier: 0.7.101
+ version: 0.7.101
'@hypit/endpoint-kit':
specifier: workspace:*
version: link:../endpoint-kit
@@ -2013,9 +2019,9 @@ importers:
'@hypit/driver-node':
specifier: workspace:*
version: link:../driver-node
- '@hypit/mimo-tts':
+ '@hypit/mimo-speech':
specifier: workspace:*
- version: link:../mimo-tts
+ version: link:../mimo-speech
'@hypit/seedance':
specifier: workspace:*
version: link:../seedance
@@ -2262,9 +2268,9 @@ importers:
'@hypit/driver-node':
specifier: workspace:*
version: link:../driver-node
- '@hypit/mimo-tts':
+ '@hypit/mimo-speech':
specifier: workspace:*
- version: link:../mimo-tts
+ version: link:../mimo-speech
packages/ranking:
dependencies: