Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
feat: complete estimate and media authoring paths
kashorincdb29b16e0aa
diff --git a/docs/quickstart/composition.md b/docs/quickstart/composition.mdindex a2d46056..ea53cc1f 100644--- a/docs/quickstart/composition.md+++ b/docs/quickstart/composition.md@@ -99,7 +99,9 @@ The renderer:4. Mixes the audio Tracks5. Muxes video + audio into the final MP4-**Output:** `{final.video}` — the finished video file. This is the most common Build Target.+**Output:** `{final.video}` — the finished video as an ordinary content-addressed `BlobArtifact`.+This is the most common Build Target, and it can also be connected directly to later Blob consumers+such as media trimming, audio/frame extraction or a model reference input.## Full pipeline walkthrough@@ -138,9 +140,9 @@ The complete data flow from Script to rendered video. This example is based on<wording:Value id="direction">Locked medium close-up in a quiet daylight studio. Spoken dialogue — say exactly: Meaning becomes the source.</wording:Value>- <seedance:Speech id="take" model="mini"- prompt={direction} duration="5"/>- <seedance:Video id="motion" model="mini"+ <seedance:TextVideo id="take" model="mini"+ prompt={direction} duration="5" generate-audio="true"/>+ <seedance:TextVideo id="motion" model="mini"prompt={direction} duration="5"/><space:Canvas id="vertical" width="1080" height="1920"/>@@ -151,7 +153,7 @@ The complete data flow from Script to rendered video. This example is based on<!-- 3. Timing: assemble spine and align words --><speech:Spine id="speech" canvas={vertical}>- <speech:Take source={take} segment={story.segment.opening}/>+ <speech:Take source={take.video} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>diff --git a/docs/quickstart/generation.md b/docs/quickstart/generation.mdindex f0858e17..36bea8c4 100644--- a/docs/quickstart/generation.md+++ b/docs/quickstart/generation.md@@ -12,6 +12,7 @@ Every component shown here must be imported by its package specifier before use:```svml<import as="media" from="@narratage/media@1"/>+<import as="mediaop" from="@narratage/media-pipeline@1"/><import as="estimate" from="@narratage/estimate@1"/><import as="seedance" from="@narratage/seedance@1"/><import as="speaker" from="@narratage/seedance-speaker@1"/>@@ -31,7 +32,7 @@ Declares a content-addressed image asset from a local file.| `src` | yes | Path to the image file, relative to the `.svml` source |The image is referenced downstream via `{presenter}` — for example, as a character reference in-`seedance:Speech` or as a B-roll source.+`seedance:ReferenceVideo` or as a B-roll source.## media:Audio@@ -50,8 +51,8 @@ Typically used as a voice-timbre reference for `speaker:Take`.## estimate:Speech-Deterministic speech duration estimation from Script text. No external service call — the estimate is-computed locally from word count and pace parameters.+Deterministic speech duration planning from Script pronunciation text. No external service call — the+duration is computed locally from pronunciation units and a delivery-density policy.```svml<estimate:Speech id="hook-duration"@@ -63,7 +64,7 @@ computed locally from word count and pace parameters.|---|---|---|| `id` | yes | Unique identifier || `source` | yes | Script text to estimate — typically `{script.segment.NAME.speech}` |-| `policy` | yes | SVS speech Recipe controlling pace and bounds |+| `policy` | no | SVS speech Recipe controlling pace and bounds; omit it to use inline parameters |The `policy` references an SVS Recipe (see [SVS Stylesheets](./styles.md#speech-estimation)):@@ -71,10 +72,9 @@ The `policy` references an SVS Recipe (see [SVS Stylesheets](./styles.md#speech-speech.normal {language: en;pace: normal;- padding: 0.3;min: 4;max: 15;- rounding: ceil;+ rounding: round;}```@@ -83,9 +83,17 @@ You can also specify estimation parameters inline instead of using an SVS policy```svml<estimate:Speech id="opening-duration"source={story.segment.opening.speech}- language="en" pace="normal" padding="0.3" min="4" max="15" rounding="ceil"/>+ language="en" pace="normal" min="4" max="15" rounding="round"/>```+For English, the official pace presets are `slow = 4.2`, `normal = 4.6`, and+`fast = 5.0` syllables per second. `rate="4.75"` may be used instead of `pace`+when a project needs a value between the named presets. `pace` and `rate` are+mutually exclusive.++There are no implicit policy values: `language`, `min`, `max`, `rounding`, and+exactly one of `pace` or `rate` must be present either inline or in the referenced Recipe.+**Output:** `{hook-duration.duration}` — the estimated duration in seconds, passed to generationcomponents.@@ -111,63 +119,69 @@ declared text port.The element body is the exact Text value. `text:Render` can produce the same type from a templateand explicit graph inputs.-## seedance:Speech+## Seedance invocation shapes++Seedance exposes model capabilities, not creative usages. `standard`, `fast` and `mini` choose the+model variant independently of three invocation shapes. All shapes consume a complete ordinary+`Text` prompt and output `{id.video}`.-Generates a talking-head video clip via the Seedance model. This is the low-level generation-component — its complete model input Text already includes any spoken dialogue.+### seedance:TextVideo++Prompt-only generation. This is the only shape that accepts `web-search`.```svml-<seedance:Speech id="alice-take" model="mini"- prompt={alice-direction}- duration="8">- <seedance:Reference image={alice-reference}/>-</seedance:Speech>+<seedance:TextVideo id="ambient" model="mini"+ prompt={ambient-direction} duration="5" web-search="false"/>```-| Attribute | Required | Description |-|---|---|---|-| `id` | yes | Unique identifier |-| `model` | yes | Seedance model name: `mini` |-| `prompt` | yes | Complete model input — reference to ordinary `Text` |-| `duration` | yes | Clip duration in seconds (number or `{estimate.duration}` reference) |-| `resolution` | no | Output resolution: `480p`, `720p` (default varies by model) |-| `aspect-ratio` | no | Output aspect ratio: `9:16`, `16:9`, `1:1` |+### seedance:FrameVideo-### seedance:Reference--Child element that provides a reference image for character consistency:+First-frame generation with an optional last frame:```svml-<seedance:Reference image={alice-reference}/>+<seedance:FrameVideo id="transition" model="fast"+ prompt={transition-direction} duration="5"+ first-frame={opening-image} last-frame={closing-image}/>```-| Attribute | Required | Description |-|---|---|---|-| `image` | yes | Reference to a `media:Image` component |+### seedance:ReferenceVideo-**Output:** `{alice-take}` or `{alice-take.video}` — the generated video, passed to `speech:Spine`.--## seedance:Video--Generates a standalone video clip (not a talking-head — no dialogue lip-sync).+Multimodal reference generation. It requires at least one `Reference` child and accepts image,+video and audio references within the model's declared limits.```svml-<seedance:Video id="product-motion" model="mini"- prompt={product-direction} duration="5">- <seedance:Reference image={product-reference}/>-</seedance:Video>+<seedance:ReferenceVideo id="alice-take" model="mini"+ prompt={alice-direction}+ duration={alice-duration.duration}+ generate-audio="true">+ <seedance:Reference image={alice-reference}/>+ <seedance:Reference audio={alice-voice}/>+</seedance:ReferenceVideo>```-| Attribute | Required | Description |-|---|---|---|-| `id` | yes | Unique identifier |-| `model` | yes | Seedance model name: `mini` |-| `prompt` | yes | Complete model input — reference to ordinary `Text` |-| `duration` | yes | Clip duration in seconds |+The component does not know that this is a talking head. That meaning lives in the supplied Text or+a higher-level package such as `speaker:Take`.++Common attributes are `id`, `model`, `prompt`, `duration`, `resolution`, `aspect-ratio` and+`generate-audio`. `duration` may be literal or an explicit `{estimate.duration}` edge.++The audio generated in an earlier take can be reused as a later reference through an ordinary graph+edge. Extraction does not turn it into speech evidence or attach speaker meaning:-Also accepts `<seedance:Reference>` children for reference images.+```svml+<mediaop:ExtractAudio id="voice-from-opening"+ source={opening.video} audio="default"/>++<seedance:ReferenceVideo id="follow-up" model="mini"+ prompt={follow-up-direction} duration="5" generate-audio="true">+ <seedance:Reference image={presenter-reference}/>+ <seedance:Reference audio={voice-from-opening.audio}/>+</seedance:ReferenceVideo>+```-**Output:** `{product-motion.video}` — used as a B-roll source.+The same media-operation package exposes `Transform` for ordered trim/retime and `ExtractFrame` for+first, last, indexed or timestamped still extraction. Local FFmpeg and AWS Lambda are interchangeable+Runtime Endpoints for these exact Needs; neither changes the author graph.## speaker:Takediff --git a/docs/quickstart/script.md b/docs/quickstart/script.mdindex b6202333..de941b6b 100644--- a/docs/quickstart/script.md+++ b/docs/quickstart/script.md@@ -79,7 +79,7 @@ Role Cues produce different text projections:| **caption** | `What time is it?`<br>`It's 8:30.` |The dialogue `Text` includes Role Cue prefixes. Speech `Text` and caption projection strip them.-Components like `seedance:Speech` use `{story.segment.dialogue.dialogue}` (with labels). Script also+Prompt programs feeding `seedance:ReferenceVideo` may use `{story.segment.dialogue.dialogue}` (with labels). Script alsoemits `{story.caption}` as the ordered display Atom/Word universe and`{story.caption.correspondence}` as the separate Atom-to-speech edge used only when timing joins.diff --git a/docs/quickstart/styles.md b/docs/quickstart/styles.mdindex a6edd195..3af1de1a 100644--- a/docs/quickstart/styles.md+++ b/docs/quickstart/styles.md@@ -251,10 +251,9 @@ Parameters for deterministic speech duration estimation.speech.normal {language: en;pace: normal;- padding: 0.3;min: 4;max: 15;- rounding: ceil;+ rounding: round;}```@@ -262,10 +261,16 @@ speech.normal {|---|---|| `language` | Language code (e.g. `en`) || `pace` | Speaking pace: `slow`, `normal`, `fast` |-| `padding` | Extra padding in seconds added to the estimate |+| `rate` | Positive pronunciation units per second; mutually exclusive with `pace` || `min` | Minimum duration in seconds || `max` | Maximum duration in seconds |-| `rounding` | Rounding mode: `ceil`, `floor`, `round` |+| `rounding` | Rounding mode: `none`, `round`, `ceil` |++The English named presets resolve to `4.2`, `4.6`, and `5.0` syllables per+second. Use a numeric `rate` in place of `pace` when the project needs a+continuous author-controlled value.+Every property is explicit: `language`, `min`, `max`, `rounding`, and exactly+one of `pace` or `rate` are required. The Estimate package supplies no hidden policy defaults.Referenced by `estimate:Speech` via the `policy` attribute:@@ -371,10 +376,9 @@ A complete `studio.svs` file for a four-take talking-head project:speech.normal {language: en;pace: normal;- padding: 0.3;min: 4;max: 15;- rounding: ceil;+ rounding: round;}speaker.host {diff --git a/docs/quickstart/timing.md b/docs/quickstart/timing.mdindex bab46646..64667493 100644--- a/docs/quickstart/timing.md+++ b/docs/quickstart/timing.md@@ -41,7 +41,7 @@ Each `<speech:Take>` child binds a generated video to a Script Segment:| Attribute | Required | Description ||---|---|---|-| `source` | yes | Generated video — from `seedance:Speech`, `speaker:Take`, etc. |+| `source` | yes | Generated video — from `seedance:ReferenceVideo`, `speaker:Take`, etc. || `segment` | yes | Script Segment this take corresponds to — e.g. `{story.segment.hook}` |The order of `<speech:Take>` children **determines the program order**. The first take starts at@@ -135,8 +135,8 @@ The complete timing stage, from generated takes to map and space:<!-- Assemble takes in program order --><space:Canvas id="vertical" width="1080" height="1920"/><speech:Spine id="speech" canvas={vertical}>- <speech:Take source={opening-take} segment={story.segment.opening}/>- <speech:Take source={answer-take} segment={story.segment.answer}/>+ <speech:Take source={opening-take.video} segment={story.segment.opening}/>+ <speech:Take source={answer-take.video} segment={story.segment.answer}/></speech:Spine><!-- Measure word timing -->diff --git a/docs/quickstart/tracks.md b/docs/quickstart/tracks.mdindex d8e968dd..5cbcf3ec 100644--- a/docs/quickstart/tracks.md+++ b/docs/quickstart/tracks.md@@ -172,10 +172,10 @@ Placement is an explicit Spatial Frame edge; appearance and motion remain reusabthen those regions assemble into a finished video.</wording:Value>-<seedance:Video id="product-motion" model="mini"+<seedance:ReferenceVideo id="product-motion" model="mini"prompt={product-direction} duration="5"><seedance:Reference image={product-reference}/>-</seedance:Video>+</seedance:ReferenceVideo><pipeline:Normalize id="product-media" source={product-motion.video}video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>diff --git a/docs/zh/quickstart/composition.md b/docs/zh/quickstart/composition.mdindex f0f3a9d7..9dc37ac9 100644--- a/docs/zh/quickstart/composition.md+++ b/docs/zh/quickstart/composition.md@@ -96,7 +96,8 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T4. 混合音频 Track5. 将视频和音频混合封装为最终的 MP4-**输出:** `{final.video}`——最终的视频文件。这是最常见的 Build Target。+**输出:** `{final.video}`——以普通内容寻址 `BlobArtifact` 表示的最终视频。这是最常见的+Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参考输入等后续 Blob 消费者。## 完整的管线流程@@ -135,9 +136,9 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T<wording:Value id="direction">Locked medium close-up in a quiet daylight studio. Spoken dialogue — say exactly: Meaning becomes the source.</wording:Value>- <seedance:Speech id="take" model="mini"- prompt={direction} duration="5"/>- <seedance:Video id="motion" model="mini"+ <seedance:TextVideo id="take" model="mini"+ prompt={direction} duration="5" generate-audio="true"/>+ <seedance:TextVideo id="motion" model="mini"prompt={direction} duration="5"/><space:Canvas id="vertical" width="1080" height="1920"/>@@ -148,7 +149,7 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T<!-- 3. Timing: assemble spine and align words --><speech:Spine id="speech" canvas={vertical}>- <speech:Take source={take} segment={story.segment.opening}/>+ <speech:Take source={take.video} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>diff --git a/docs/zh/quickstart/generation.md b/docs/zh/quickstart/generation.mdindex 670a10bb..b6e87757 100644--- a/docs/zh/quickstart/generation.md+++ b/docs/zh/quickstart/generation.md@@ -11,6 +11,7 @@ description: 声明媒体资源并使用 Seedance 生成视频。```svml<import as="media" from="@narratage/media@1"/>+<import as="mediaop" from="@narratage/media-pipeline@1"/><import as="estimate" from="@narratage/estimate@1"/><import as="seedance" from="@narratage/seedance@1"/><import as="speaker" from="@narratage/seedance-speaker@1"/>@@ -29,7 +30,7 @@ description: 声明媒体资源并使用 Seedance 生成视频。| `id` | 是 | 组件的唯一标识符 || `src` | 是 | 图片文件路径,相对于 `.svml` 源文件 |-该图片在下游通过 `{presenter}` 引用——例如,作为 `seedance:Speech` 中的角色参考或作为 B-roll 来源。+该图片在下游通过 `{presenter}` 引用——例如,作为 `seedance:ReferenceVideo` 中的角色参考或作为 B-roll 来源。## media:Audio@@ -48,7 +49,7 @@ description: 声明媒体资源并使用 Seedance 生成视频。## estimate:Speech-基于 Script 文本的确定性语音时长估算。无需外部服务调用——估算根据字数和语速参数在本地计算。+基于 Script 实际读音文本的确定性语音时长规划。无需外部服务调用——时长根据读音单位和口播密度策略在本地计算。```svml<estimate:Speech id="hook-duration"@@ -60,7 +61,7 @@ description: 声明媒体资源并使用 Seedance 生成视频。|---|---|---|| `id` | 是 | 唯一标识符 || `source` | 是 | 要估算的 Script 文本——通常为 `{script.segment.NAME.speech}` |-| `policy` | 是 | 控制语速和边界的 SVS 语音 Recipe |+| `policy` | 否 | 控制语速和边界的 SVS 语音 Recipe;省略时使用内联参数 |`policy` 引用一个 SVS Recipe(参见 [SVS 样式表](./styles.md#speech-estimation)):@@ -68,10 +69,9 @@ description: 声明媒体资源并使用 Seedance 生成视频。speech.normal {language: en;pace: normal;- padding: 0.3;min: 4;max: 15;- rounding: ceil;+ rounding: round;}```@@ -80,9 +80,14 @@ speech.normal {```svml<estimate:Speech id="opening-duration"source={story.segment.opening.speech}- language="en" pace="normal" padding="0.3" min="4" max="15" rounding="ceil"/>+ language="en" pace="normal" min="4" max="15" rounding="round"/>```+英语官方档位为 `slow = 4.2`、`normal = 4.6`、`fast = 5.0` 音节/秒。+项目需要档位之间的连续值时,可以用 `rate="4.75"` 代替 `pace`;二者不能同时出现。+策略没有隐式值:无论内联还是引用 Recipe,都必须写明 `language`、`min`、`max`、+`rounding`,并且在 `pace` 和 `rate` 中恰好选择一个。+**输出:** `{hook-duration.duration}`——估算的时长(秒),传递给生成组件。## text:Value@@ -105,62 +110,66 @@ speech.normal {元素主体就是精确的 Text 值。`text:Render` 也能用模板和显式图输入产出同一类型。-## seedance:Speech+## Seedance 三种调用形状++Seedance 只暴露模型能力,不暴露“口播”“B-roll”等创作用途。`standard`、`fast`、`mini`+选择模型版本;调用形状则独立分为三种。三者都消费完整的普通 `Text` Prompt,并输出+`{id.video}`。++### seedance:TextVideo-通过 Seedance 模型生成口播视频片段。这是低层生成组件——完整的模型输入 Text 已包含需要朗读的台词。+纯 Prompt 生成。只有这种形状允许 `web-search`:```svml-<seedance:Speech id="alice-take" model="mini"- prompt={alice-direction}- duration="8">- <seedance:Reference image={alice-reference}/>-</seedance:Speech>+<seedance:TextVideo id="ambient" model="mini"+ prompt={ambient-direction} duration="5" web-search="false"/>```-| 属性 | 必填 | 说明 |-|---|---|---|-| `id` | 是 | 唯一标识符 |-| `model` | 是 | Seedance 模型名称:`mini` |-| `prompt` | 是 | 完整模型输入——引用普通 `Text` |-| `duration` | 是 | 片段时长(秒)(数字或 `{estimate.duration}` 引用) |-| `resolution` | 否 | 输出分辨率:`480p`、`720p`(默认值因模型而异) |-| `aspect-ratio` | 否 | 输出宽高比:`9:16`、`16:9`、`1:1` |--### seedance:Reference+### seedance:FrameVideo-提供参考图片以保持角色一致性的子元素:+必须给首帧,可以额外给尾帧:```svml-<seedance:Reference image={alice-reference}/>+<seedance:FrameVideo id="transition" model="fast"+ prompt={transition-direction} duration="5"+ first-frame={opening-image} last-frame={closing-image}/>```-| 属性 | 必填 | 说明 |-|---|---|---|-| `image` | 是 | 引用 `media:Image` 组件 |+### seedance:ReferenceVideo-**输出:** `{alice-take}` 或 `{alice-take.video}`——生成的视频,传递给 `speech:Spine`。--## seedance:Video--生成独立的视频片段(非说话人头部——无对话口型同步)。+多模态参考生成。至少需要一个 `Reference` 子元素,可以显式接入图片、视频和音频:```svml-<seedance:Video id="product-motion" model="mini"- prompt={product-direction} duration="5">- <seedance:Reference image={product-reference}/>-</seedance:Video>+<seedance:ReferenceVideo id="alice-take" model="mini"+ prompt={alice-direction}+ duration={alice-duration.duration}+ generate-audio="true">+ <seedance:Reference image={alice-reference}/>+ <seedance:Reference audio={alice-voice}/>+</seedance:ReferenceVideo>```-| 属性 | 必填 | 说明 |-|---|---|---|-| `id` | 是 | 唯一标识符 |-| `model` | 是 | Seedance 模型名称:`mini` |-| `prompt` | 是 | 完整模型输入——引用普通 `Text` |-| `duration` | 是 | 片段时长(秒) |+这个低层组件并不知道它被用来做口播;用途只存在于传入的 Text 或 `speaker:Take` 之类的+高层包中。公共属性包括 `id`、`model`、`prompt`、`duration`、`resolution`、+`aspect-ratio`、`generate-audio`;`duration` 可以是字面量或显式 `{estimate.duration}` 边。++可以直接抽取前一段生成视频里的音频,并通过普通图边给后续片段当作参考。这个操作不会+把音频提升成语音证据,也不会凭空附加说话人语义:-同样接受 `<seedance:Reference>` 子元素作为参考图片。+```svml+<mediaop:ExtractAudio id="voice-from-opening"+ source={opening.video} audio="default"/>++<seedance:ReferenceVideo id="follow-up" model="mini"+ prompt={follow-up-direction} duration="5" generate-audio="true">+ <seedance:Reference image={presenter-reference}/>+ <seedance:Reference audio={voice-from-opening.audio}/>+</seedance:ReferenceVideo>+```-**输出:** `{product-motion.video}`——用作 B-roll 来源。+同一个媒体操作包还提供 `Transform`(按顺序截取、变速)和 `ExtractFrame`(首帧、尾帧、+指定帧或指定时间取图)。本地 FFmpeg 与 AWS Lambda 只是这些精确 Need 的可互换 Runtime+Endpoint,不会改变作者图。## speaker:Takediff --git a/docs/zh/quickstart/script.md b/docs/zh/quickstart/script.mdindex 33846b75..35484f9b 100644--- a/docs/zh/quickstart/script.md+++ b/docs/zh/quickstart/script.md@@ -71,7 +71,7 @@ Role Cue 会产生不同的文本投影:| **speech** | `What time is it?`<br>`It's 8:30.` || **caption** | `What time is it?`<br>`It's 8:30.` |-dialogue `Text` 包含 Role Cue 前缀,speech `Text` 和 caption 投影会去除前缀。像 `seedance:Speech` 这样的组件使用 `{story.segment.dialogue.dialogue}`(带标签)。Script 还会显式输出 `{story.caption}` 作为有序的显示 Atom/Word 全集,并以 `{story.caption.correspondence}` 单独提供 Atom 到口播 token 的对应;只有定时汇合处需要后者。+dialogue `Text` 包含 Role Cue 前缀,speech `Text` 和 caption 投影会去除前缀。给 `seedance:ReferenceVideo` 提供输入的 Prompt Program 可以使用 `{story.segment.dialogue.dialogue}`(带标签)。Script 还会显式输出 `{story.caption}` 作为有序的显示 Atom/Word 全集,并以 `{story.caption.correspondence}` 单独提供 Atom 到口播 token 的对应;只有定时汇合处需要后者。## Dual Textdiff --git a/docs/zh/quickstart/styles.md b/docs/zh/quickstart/styles.mdindex b9b3000b..53ca19ed 100644--- a/docs/zh/quickstart/styles.md+++ b/docs/zh/quickstart/styles.md@@ -246,10 +246,9 @@ text.title {speech.normal {language: en;pace: normal;- padding: 0.3;min: 4;max: 15;- rounding: ceil;+ rounding: round;}```@@ -257,10 +256,15 @@ speech.normal {|---|---|| `language` | 语言代码(如 `en`) || `pace` | 语速:`slow`、`normal`、`fast` |-| `padding` | 添加到估算值的额外填充时间(秒) |+| `rate` | 正数的每秒读音单位数;不能和 `pace` 同时使用 || `min` | 最小时长(秒) || `max` | 最大时长(秒) |-| `rounding` | 取整模式:`ceil`、`floor`、`round` |+| `rounding` | 取整模式:`none`、`round`、`ceil` |++英语三个具名档位分别解析为每秒 `4.2`、`4.6`、`5.0` 个音节。项目需要连续可调值时,+用数值 `rate` 代替 `pace`。+所有属性都必须显式提供:`language`、`min`、`max`、`rounding`,并且在 `pace` 与+`rate` 中恰好选择一个。Estimate 包不会补充隐藏策略默认值。通过 `estimate:Speech` 的 `policy` 属性引用:@@ -363,10 +367,9 @@ SVS 描述字体策略,但不选择或打开字体字节。常用开源字体speech.normal {language: en;pace: normal;- padding: 0.3;min: 4;max: 15;- rounding: ceil;+ rounding: round;}speaker.host {diff --git a/docs/zh/quickstart/timing.md b/docs/zh/quickstart/timing.mdindex b9834080..1cca7c32 100644--- a/docs/zh/quickstart/timing.md+++ b/docs/zh/quickstart/timing.md@@ -38,7 +38,7 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连| 属性 | 必填 | 描述 ||---|---|---|-| `source` | 是 | 生成的视频——来自 `seedance:Speech`、`speaker:Take` 等 |+| `source` | 是 | 生成的视频——来自 `seedance:ReferenceVideo`、`speaker:Take` 等 || `segment` | 是 | 此片段对应的 Script Segment——例如 `{story.segment.hook}` |`<speech:Take>` 子元素的排列顺序**决定了节目顺序**。第一个片段从时间零点开始;后续片段依次紧接。@@ -122,8 +122,8 @@ Map 只包含最终词窗口和语义锚点,不传播“测量、推导、估<!-- Assemble takes in program order --><space:Canvas id="vertical" width="1080" height="1920"/><speech:Spine id="speech" canvas={vertical}>- <speech:Take source={opening-take} segment={story.segment.opening}/>- <speech:Take source={answer-take} segment={story.segment.answer}/>+ <speech:Take source={opening-take.video} segment={story.segment.opening}/>+ <speech:Take source={answer-take.video} segment={story.segment.answer}/></speech:Spine><!-- Measure word timing -->diff --git a/docs/zh/quickstart/tracks.md b/docs/zh/quickstart/tracks.mdindex 83d9d3b0..1cb04b23 100644--- a/docs/zh/quickstart/tracks.md+++ b/docs/zh/quickstart/tracks.md@@ -142,10 +142,10 @@ B-roll 是通用 Media Track 的一种剪辑用途,不是独立 Track 家族then those regions assemble into a finished video.</wording:Value>-<seedance:Video id="product-motion" model="mini"+<seedance:ReferenceVideo id="product-motion" model="mini"prompt={product-direction} duration="5"><seedance:Reference image={product-reference}/>-</seedance:Video>+</seedance:ReferenceVideo><pipeline:Normalize id="product-media" source={product-motion.video}video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>diff --git a/examples/talking-film-golden/README.md b/examples/talking-film-golden/README.mdindex c8a1d425..3f91daa8 100644--- a/examples/talking-film-golden/README.md+++ b/examples/talking-film-golden/README.md@@ -69,10 +69,9 @@ Candidates and Operations.| `<script>` | `@narratage/script` | authored `Narrative` and projections | implemented || `media:Image` | `@narratage/media` | content-addressed authored Artifact | implemented || `wording:Value` | `@narratage/text` | model-neutral immutable Text value | implemented |-| `seedance:Speech model="mini"` | `@narratage/seedance` | explicit Seedance Mini Need plus primary-video projection | implemented; explicit duration remains authored until Speech scheduling exists |+| `seedance:ReferenceVideo model="mini"` | `@narratage/seedance` | exact multimodal Seedance Mini Need plus primary-video projection | implemented || `speech:Spine` | `@narratage/speech` | ordered clips -> normalized Takes, one `SpeechBasis`, then ordinary projections | Surface, fold, media normalization and projection components implemented || `whisperx:Alignment` | `@narratage/whisperx` | 48k speech master -> explicit 16k evidence Need -> WhisperX -> provider-neutral `@narratage/speech-alignment` -> Map | Surface, Graph Fragment, deterministic components and local Provider/service implemented |-| `seedance:Video model="mini"` | `@narratage/seedance` | explicit Seedance Mini video Need plus primary-video projection | implemented || `media-track:Track` | `@narratage/media-track` | semantic windows + normalized media + explicit Frame/Recipe -> peer Visual/optional Audio Tracks | Item/Sequence Surface and deterministic lowering implemented || `caption-fine:Style` / `caption:Program` | Fine + common Caption | explicit default over all words + ordered whole-Style replacement by Role or word subset | implemented || `caption-ai:Planner` | `@narratage/caption-gemini` | immutable display Atoms/Words + per-run requirements -> whole-Atom Cue cuts and optional per-Word fields | implemented; Google Vertex Endpoint implemented separately |diff --git a/examples/talking-film-golden/main.svml b/examples/talking-film-golden/main.svmlindex a68c466a..de56321e 100644--- a/examples/talking-film-golden/main.svml+++ b/examples/talking-film-golden/main.svml@@ -93,41 +93,43 @@/><!-- The author explicitly chooses Seedance Mini. Runtime only binds a matching endpoint. -->- <seedance:Speech+ <seedance:ReferenceVideoid="alice-take"model="mini"prompt={alice-direction}duration="8"+ generate-audio="true"><seedance:Reference image={alice-reference}/>- </seedance:Speech>+ </seedance:ReferenceVideo>- <seedance:Speech+ <seedance:ReferenceVideoid="bob-take"model="mini"prompt={bob-direction}duration="8"+ generate-audio="true"><seedance:Reference image={bob-reference}/>- </seedance:Speech>+ </seedance:ReferenceVideo><!-- Ordered generated Takes become one audio basis plus its peer visual/audio Tracks. --><speech:Spine id="speech" canvas={vertical}>- <speech:Take source={alice-take} segment={script.segment.opening}/>- <speech:Take source={bob-take} segment={script.segment.answer}/>+ <speech:Take source={alice-take.video} segment={script.segment.opening}/>+ <speech:Take source={bob-take.video} segment={script.segment.answer}/></speech:Spine><!-- One explicit WhisperX path: request, normalize and deterministic Script alignment. --><whisperx:Alignment id="timing" narrative={script} audio={speech.audio}/>- <seedance:Video+ <seedance:ReferenceVideoid="product-motion"model="mini"prompt={product-direction}duration="5"><seedance:Reference image={product-reference}/>- </seedance:Video>+ </seedance:ReferenceVideo><pipeline:Normalize id="product-media" source={product-motion.video}video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>diff --git a/examples/talking-film-graph-check/main.svml b/examples/talking-film-graph-check/main.svmlindex a473678d..f8aeb3cc 100644--- a/examples/talking-film-graph-check/main.svml+++ b/examples/talking-film-graph-check/main.svml@@ -29,11 +29,11 @@left="10%" top="20%" right="10%" bottom="30%"/><wording:Value id="direction">Locked medium close-up in a quiet daylight studio. Spoken dialogue — say exactly: Meaning becomes the source.</wording:Value>- <seedance:Speech id="take" model="mini" prompt={direction} duration="5"/>- <seedance:Video id="motion" model="mini" prompt={direction} duration="5"/>+ <seedance:TextVideo id="take" model="mini" prompt={direction} duration="5" generate-audio="true"/>+ <seedance:TextVideo id="motion" model="mini" prompt={direction} duration="5"/><speech:Spine id="speech" canvas={vertical}>- <speech:Take source={take} segment={story.segment.opening}/>+ <speech:Take source={take.video} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>diff --git a/examples/talking-film-graph-check/svml.packages.lock b/examples/talking-film-graph-check/svml.packages.lockindex 6455b1b9..3b3a6977 100644--- a/examples/talking-film-graph-check/svml.packages.lock+++ b/examples/talking-film-graph-check/svml.packages.lock@@ -619,7 +619,7 @@{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:08b62e21bd0f6f03fd7e2b481f820c56006a0ff92e6a7318927bd0c3607223f5"+ "digest": "sha256:6e4dbfde67625679cce91e137ed6b0efecee062915d802467ad7e106e75f55fc"},{"name": "@narratage/media-track",@@ -654,7 +654,7 @@{"name": "@narratage/render-hyperframes","version": "0.0.0-dev",- "digest": "sha256:28847756a67fc7a9c304d9ed358964e422956d86da6f53e3b7d2ebccdff54cad"+ "digest": "sha256:18d5f3a63c71135771af5e9b47f8bcd6cf41662adcb3f1d1628dce6138f75931"},{"name": "@narratage/run-markup",@@ -674,7 +674,7 @@{"name": "@narratage/seedance","version": "0.0.0-dev",- "digest": "sha256:66cdf600171833956e1f5c5494f6757471d8f795a4d8fe7cfabd791303127ecf"+ "digest": "sha256:72e172d81787cb632a903ab5f3f56ff7bb12daf57f3e2f927c70045db8140b88"},{"name": "@narratage/semantic-map",@@ -849,8 +849,8 @@"name": "@narratage/media-pipeline","version": "0.0.0-dev"},- "facetsDigest": "sha256:b7bf62142837adc7bad4f8c4b74b60ee6b4ade74140788cf4b682c140893b4af",- "closureDigest": "sha256:3ed0d2f8a7e8d0f3bbb3c4a1ee354967afe24311cd567f562ad766bdca52387c"+ "facetsDigest": "sha256:976b989d377ddaeee207cdf64d76bac74ed3c95f6a6b00e4b9220f4aa2680c50",+ "closureDigest": "sha256:4496f7f9544715877ba1c75e5954a0aa70d48902522ebd6cb89c1d24b34a0f75"},{"specifier": "@narratage/media-track",@@ -885,8 +885,8 @@"name": "@narratage/render-hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:965f293a64a8b1c7ae87e6e13986a751d4c26a2fcded9dd81fc24a7790980b45",- "closureDigest": "sha256:7f0830d7c6714b686acbd3159b4980ffe95f99100e330ac252506e8972b8186a"+ "facetsDigest": "sha256:4210736adc699b3b5f86bf8b0cd84cd7a3c946754c8f19ceae5e0390a3d1d9c5",+ "closureDigest": "sha256:deb47bd7c049ff1ad23b9e2e97caeda6a224067aa0bde9b2336224ef211165c3"},{"specifier": "@narratage/run-markup",@@ -912,8 +912,8 @@"name": "@narratage/seedance","version": "0.0.0-dev"},- "facetsDigest": "sha256:4750e07548e9706f3a0d8da1f8ca1960a27227e1cae647f743934ff2338e11ae",- "closureDigest": "sha256:c329b039f8560a54b56749f7b96329c6c06605d58165a0c6ba5e81a34e84fd9f"+ "facetsDigest": "sha256:1b11ad1dd6e65d9b836dec7fc5b9b03e40c8178a2c2a579144c25a57896d9832",+ "closureDigest": "sha256:b05b8f83592def14f8f2fd0b59279627941b9ec9b76a1d8d9e2bcc4a1333f903"},{"specifier": "@narratage/semantic-map",@@ -975,8 +975,8 @@"name": "@narratage/speech-spine","version": "0.0.0-dev"},- "facetsDigest": "sha256:c2d3ae7c5a676c66368c9225fd516aa6cfa7dab749123f7ecd31007628ee4666",- "closureDigest": "sha256:38365376d684cb70f6d0e939042320c7ac309f8c4272affb0c2a91890419e45c"+ "facetsDigest": "sha256:4af25fdcd03b5dde7c6151419885646c6862978aa5c88fe3df66c7daec1b0199",+ "closureDigest": "sha256:ce29d7f041624590d03d9f0d2358c730b14464b63247f8df84a701e7000581b6"},{"specifier": "@narratage/svs",@@ -1029,9 +1029,9 @@"name": "@narratage/whisperx","version": "0.0.0-dev"},- "facetsDigest": "sha256:be22a14fb7f6eb437aa17b541a35320f5e26b77d7d9925702b2977609f874fe3",- "closureDigest": "sha256:4b2e7e4d12e50b6b8dad43c2c080dd6d5d04475a155e22cf35bb61069bba1f8a"+ "facetsDigest": "sha256:73fceef213e8d6bcad92532c5d712842e62921743b81bc7bb4d9022a92067359",+ "closureDigest": "sha256:32f2a12214d47f6539084d13a87fd95104f458340fae2fa3aee58eee0e111f92"}],- "digest": "sha256:643d86abe83dd2e9db3744c77307118287fdda5088f143b81a4e8b685d893958"+ "digest": "sha256:1ba6eac04981a6edb7ead304a917c485cf187d250e37261c97080a52c6832cc6"}diff --git a/examples/talking-film-live/README.md b/examples/talking-film-live/README.mdindex 195b688c..da2f63ad 100644--- a/examples/talking-film-live/README.md+++ b/examples/talking-film-live/README.md@@ -9,8 +9,8 @@ Script -> deterministic Estimate -> two Seedance Mini 480p generations-> Selection-located Text Track -> Film -> HyperFrames -> audio mix -> mux```-Both English lines resolve to five seconds with `pace="normal"`, `padding="0.3"`, a four-second-minimum and `ceil` rounding. The generated presenter image is intentionally ignored by Git under+Both English lines use a fully explicit duration policy. The package does not add a hidden pause,+padding interval, model bound, or rounding choice. The generated presenter image is intentionally ignored by Git under`assets/`; supply or regenerate `assets/presenter.png` before checking or building.Prerequisites for the exact checked-in Runtime Profile:diff --git a/examples/talking-film-live/main.svml b/examples/talking-film-live/main.svmlindex e6190c2e..ba4cc5dd 100644--- a/examples/talking-film-live/main.svml+++ b/examples/talking-film-live/main.svml@@ -32,10 +32,9 @@source={story.segment.opening.speech}language="en"pace="normal"- padding="0.3"min="4"max="15"- rounding="ceil"+ rounding="round"/><estimate:Speech@@ -43,10 +42,9 @@source={story.segment.answer.speech}language="en"pace="normal"- padding="0.3"min="4"max="15"- rounding="ceil"+ rounding="round"/><wording:Value id="opening-direction">@@ -63,33 +61,35 @@Spoken dialogue — say exactly: Then every caption and visual can arrive at exactly the right moment.</wording:Value>- <seedance:Speech+ <seedance:ReferenceVideoid="opening-take"model="mini"prompt={opening-direction}duration={opening-duration.duration}resolution="480p"aspect-ratio="9:16"+ generate-audio="true"><seedance:Reference image={presenter}/>- </seedance:Speech>+ </seedance:ReferenceVideo>- <seedance:Speech+ <seedance:ReferenceVideoid="answer-take"model="mini"prompt={answer-direction}duration={answer-duration.duration}resolution="480p"aspect-ratio="9:16"+ generate-audio="true"><seedance:Reference image={presenter}/>- </seedance:Speech>+ </seedance:ReferenceVideo><space:Canvas id="vertical" width="480" height="854"/><speech:Spine id="speech" canvas={vertical}>- <speech:Take source={opening-take} segment={story.segment.opening}/>- <speech:Take source={answer-take} segment={story.segment.answer}/>+ <speech:Take source={opening-take.video} segment={story.segment.opening}/>+ <speech:Take source={answer-take.video} segment={story.segment.answer}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>diff --git a/examples/talking-film-live/svml.packages.lock b/examples/talking-film-live/svml.packages.lockindex 4ed91413..88e1e01a 100644--- a/examples/talking-film-live/svml.packages.lock+++ b/examples/talking-film-live/svml.packages.lock@@ -589,7 +589,7 @@{"name": "@narratage/estimate","version": "0.0.0-dev",- "digest": "sha256:6f162eddb9c274d4c1a4f1637a8c154c5e516e2e466c383f0b3e7a312a10aecf"+ "digest": "sha256:f6f41462aee1e8972b44989de8c669ace6b4caf1b2f8fd2867edc8fcd3d77508"},{"name": "@narratage/film",@@ -624,7 +624,7 @@{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:08b62e21bd0f6f03fd7e2b481f820c56006a0ff92e6a7318927bd0c3607223f5"+ "digest": "sha256:6e4dbfde67625679cce91e137ed6b0efecee062915d802467ad7e106e75f55fc"},{"name": "@narratage/media-track",@@ -659,7 +659,7 @@{"name": "@narratage/render-hyperframes","version": "0.0.0-dev",- "digest": "sha256:28847756a67fc7a9c304d9ed358964e422956d86da6f53e3b7d2ebccdff54cad"+ "digest": "sha256:18d5f3a63c71135771af5e9b47f8bcd6cf41662adcb3f1d1628dce6138f75931"},{"name": "@narratage/run-markup",@@ -679,7 +679,7 @@{"name": "@narratage/seedance","version": "0.0.0-dev",- "digest": "sha256:66cdf600171833956e1f5c5494f6757471d8f795a4d8fe7cfabd791303127ecf"+ "digest": "sha256:72e172d81787cb632a903ab5f3f56ff7bb12daf57f3e2f927c70045db8140b88"},{"name": "@narratage/semantic-map",@@ -755,6 +755,11 @@"name": "@narratage/whisperx","version": "0.0.0-dev","digest": "sha256:7209d52d61e9c9237b59695a30f2f04a055d83053f06439ba9e9a31dd6cd189c"+ },+ {+ "name": "cmu-pronouncing-dictionary",+ "version": "3.0.0",+ "digest": "sha256:11f46faa5a54006cd9ffd84ceaa6177765922a7646640c5f84ee1a16d04815d8"}],"packages": [@@ -809,8 +814,8 @@"name": "@narratage/estimate","version": "0.0.0-dev"},- "facetsDigest": "sha256:55e886419b6c25c3ab08c21401896d0352c90dfa32666f71882918309461e5c2",- "closureDigest": "sha256:46469d00067903d023a07077c0d06bac11668e54e00768dea9f0820504ef2e7a"+ "facetsDigest": "sha256:7abde7f9ceaa235797bede77ed5ac9ea34c3b8cde8ab2aaa037f6b705608d282",+ "closureDigest": "sha256:fd705f372501de14e37daa827353d60c5a090cc294c752c9e026e41360beaceb"},{"specifier": "@narratage/film",@@ -863,8 +868,8 @@"name": "@narratage/media-pipeline","version": "0.0.0-dev"},- "facetsDigest": "sha256:b7bf62142837adc7bad4f8c4b74b60ee6b4ade74140788cf4b682c140893b4af",- "closureDigest": "sha256:3ed0d2f8a7e8d0f3bbb3c4a1ee354967afe24311cd567f562ad766bdca52387c"+ "facetsDigest": "sha256:976b989d377ddaeee207cdf64d76bac74ed3c95f6a6b00e4b9220f4aa2680c50",+ "closureDigest": "sha256:4496f7f9544715877ba1c75e5954a0aa70d48902522ebd6cb89c1d24b34a0f75"},{"specifier": "@narratage/media-track",@@ -899,8 +904,8 @@"name": "@narratage/render-hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:965f293a64a8b1c7ae87e6e13986a751d4c26a2fcded9dd81fc24a7790980b45",- "closureDigest": "sha256:7f0830d7c6714b686acbd3159b4980ffe95f99100e330ac252506e8972b8186a"+ "facetsDigest": "sha256:4210736adc699b3b5f86bf8b0cd84cd7a3c946754c8f19ceae5e0390a3d1d9c5",+ "closureDigest": "sha256:deb47bd7c049ff1ad23b9e2e97caeda6a224067aa0bde9b2336224ef211165c3"},{"specifier": "@narratage/run-markup",@@ -926,8 +931,8 @@"name": "@narratage/seedance","version": "0.0.0-dev"},- "facetsDigest": "sha256:4750e07548e9706f3a0d8da1f8ca1960a27227e1cae647f743934ff2338e11ae",- "closureDigest": "sha256:c329b039f8560a54b56749f7b96329c6c06605d58165a0c6ba5e81a34e84fd9f"+ "facetsDigest": "sha256:1b11ad1dd6e65d9b836dec7fc5b9b03e40c8178a2c2a579144c25a57896d9832",+ "closureDigest": "sha256:b05b8f83592def14f8f2fd0b59279627941b9ec9b76a1d8d9e2bcc4a1333f903"},{"specifier": "@narratage/semantic-map",@@ -989,8 +994,8 @@"name": "@narratage/speech-spine","version": "0.0.0-dev"},- "facetsDigest": "sha256:c2d3ae7c5a676c66368c9225fd516aa6cfa7dab749123f7ecd31007628ee4666",- "closureDigest": "sha256:38365376d684cb70f6d0e939042320c7ac309f8c4272affb0c2a91890419e45c"+ "facetsDigest": "sha256:4af25fdcd03b5dde7c6151419885646c6862978aa5c88fe3df66c7daec1b0199",+ "closureDigest": "sha256:ce29d7f041624590d03d9f0d2358c730b14464b63247f8df84a701e7000581b6"},{"specifier": "@narratage/svs",@@ -1043,9 +1048,9 @@"name": "@narratage/whisperx","version": "0.0.0-dev"},- "facetsDigest": "sha256:be22a14fb7f6eb437aa17b541a35320f5e26b77d7d9925702b2977609f874fe3",- "closureDigest": "sha256:4b2e7e4d12e50b6b8dad43c2c080dd6d5d04475a155e22cf35bb61069bba1f8a"+ "facetsDigest": "sha256:73fceef213e8d6bcad92532c5d712842e62921743b81bc7bb4d9022a92067359",+ "closureDigest": "sha256:32f2a12214d47f6539084d13a87fd95104f458340fae2fa3aee58eee0e111f92"}],- "digest": "sha256:5259a51ad29b38c836b2ef3edb02aded6fe5a7020da013d99fc74eb6967c6ac4"+ "digest": "sha256:75066d609497771f755c489447106bf86cc166b0a42d547f96d28e3527865b19"}diff --git a/examples/talking-film-live/svml.runtime-packages.lock b/examples/talking-film-live/svml.runtime-packages.lockindex b5e433c7..8b71d264 100644--- a/examples/talking-film-live/svml.runtime-packages.lock+++ b/examples/talking-film-live/svml.runtime-packages.lock@@ -54,7 +54,7 @@{"name": "@narratage/composition","version": "0.0.0-dev",- "digest": "sha256:acc55c73eaf46558b695532a495bc078a72337ac727b97eaf412bc7147336d76"+ "digest": "sha256:c9bfd8e6d07dc38a14f622b4ce545752dee7df09c2458d4e4798dc87eb9ce997"},{"name": "@narratage/core",@@ -84,22 +84,22 @@{"name": "@narratage/hyperframes","version": "0.0.0-dev",- "digest": "sha256:cd4b8c4cfb11307e0a487fa5d065a961c1cfb701a48f6d293f98b90c6d01b5cb"+ "digest": "sha256:cbe9359d2a59fcde65e3b8d4fef43c9965b5e825b2d517fb25babad4c18ed512"},{"name": "@narratage/markup","version": "0.0.0-dev",- "digest": "sha256:8a8f2e935fa5b3248661b426b8429df007296f786dacf147c829b964410ce93d"+ "digest": "sha256:d038863cedf4d22162dc0b390485131caf0977be11032861415a766d068e3745"},{"name": "@narratage/media-execution","version": "0.0.0-dev",- "digest": "sha256:3a876a5c4cd879fd5155ff81c1c7455af7c122ebe6900f67fec1e9e7d76b49c9"+ "digest": "sha256:2bca4da3fe0325847811888d2ded010761d4dc447b8ec79bec9710bca5011c3e"},{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:08b62e21bd0f6f03fd7e2b481f820c56006a0ff92e6a7318927bd0c3607223f5"+ "digest": "sha256:27b943e3fd51adefaf329c84717dae9174b5126a745116f6cf5e9df372fd886e"},{"name": "@narratage/media",@@ -109,7 +109,7 @@{"name": "@narratage/narrative","version": "0.0.0-dev",- "digest": "sha256:410b1c0b30dfe4b8a81e67bbb5833dc915418337cce71325fb1bb8b93f227fac"+ "digest": "sha256:c25b30b1881baa443ca91447f2529ac744feebe54eb2e4fbf3d2ba1d797a457d"},{"name": "@narratage/program-space",@@ -139,7 +139,7 @@{"name": "@narratage/provider-media-local","version": "0.0.0-dev",- "digest": "sha256:8c5065798511380666b03f65b40fd0acbf02d7fa5c6c76ad695631d0ce023d49"+ "digest": "sha256:afedec770515f6cf0e06d42321406a1ecdf9e06c249c0575791518f924381480"},{"name": "@narratage/provider-whisperx-local",@@ -204,7 +204,7 @@{"name": "@narratage/text","version": "0.0.0-dev",- "digest": "sha256:989b56f87a21e907aca26f768262d43e5fe4a4ab3efee17595d774d4ccbca141"+ "digest": "sha256:ec9b6917a6f9d52efa6f62a9f40c4d1e3ae85623b15bdc7dffa60f96c50a7b4b"},{"name": "@narratage/validation",@@ -870,7 +870,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b2636492ab0b0b500d674899ab2ff14fc61c7f2b22ef44eb8f816f141b21ec4d",- "closureDigest": "sha256:d61896f51d88c5d07a1d198a1879055e62986b23dd50f00437486da512e6faa2"+ "closureDigest": "sha256:4f2f3ddd8072d04af19f9b3f2dcec5b4a9c909c1dc3fe4aa789b190992fea9ee"},{"specifier": "@narratage/provider-hyperframes-local",@@ -879,7 +879,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b14b721bdc0c9d5d9ced1c345107c92a3a5878f3383dea2ecd3e310442c2a40c",- "closureDigest": "sha256:a76597166b1e27453d77f3b209498f88e6ae1204249b51d531b71258b75d6f3d"+ "closureDigest": "sha256:fa04cd3045315c2331c4466967c4e6e1c411bff578b6afd5e451dd6074f7940c"},{"specifier": "@narratage/provider-kie",@@ -888,7 +888,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:e61ebb911a33d4c5d0e4e4ee4fb69cdb790f4ced050ef35f7472bfbd72013d9b",- "closureDigest": "sha256:606dc72a6f601a7aec856efa5436a0f74e3f0b505569a4017f8f30675f444fb8"+ "closureDigest": "sha256:081710459182986ed07fc269457303e6063c21acf0ee418d1edb083ec82bb53a"},{"specifier": "@narratage/provider-media-local",@@ -897,7 +897,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:701abcc923bf8a711c20b6e308d8cc8be27bf560f249b8c5a801e1dbd9cb3979",- "closureDigest": "sha256:5dcea5c6536c78a52a8900b8dc864a4c9c2353ef57cf8122a2590c7be62e8e76"+ "closureDigest": "sha256:bacffa564f65f413425b2bf895e4cb01dcd2266fa64e1836b7f7fd92d42af186"},{"specifier": "@narratage/provider-whisperx-local",@@ -906,8 +906,8 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:3686b84c42cc2b1bcaccf57cbdcda25a15c7d9d3778fb92d25437f9880ed8f5d",- "closureDigest": "sha256:9d00849016e4f24e679f7176207db44d410cfa4a46cf65df761d5cc397502655"+ "closureDigest": "sha256:bd4ab81b2d7e24531824ff85c4a97013f08f34c95570dc674884026c0ce18018"}],- "digest": "sha256:96871b6a5f84cb7c60d835eebc0f384025b641d7cb0a2f7239db7f77be4a7632"+ "digest": "sha256:9f01815de837f7af502b90eda6f7f37d3a48016671cb010cedb23892fbd4c5c6"}diff --git a/examples/talking-head-aroll/studio.svs b/examples/talking-head-aroll/studio.svsindex 1341ccff..405086c2 100644--- a/examples/talking-head-aroll/studio.svs+++ b/examples/talking-head-aroll/studio.svs@@ -4,10 +4,9 @@speech.normal {language: en;pace: normal;- padding: 0.3;min: 4;max: 15;- rounding: ceil;+ rounding: round;}speaker.host {diff --git a/examples/talking-head-aroll/svml.packages.lock b/examples/talking-head-aroll/svml.packages.lockindex 1b55f832..6683c1ff 100644--- a/examples/talking-head-aroll/svml.packages.lock+++ b/examples/talking-head-aroll/svml.packages.lock@@ -589,7 +589,7 @@{"name": "@narratage/estimate","version": "0.0.0-dev",- "digest": "sha256:6f162eddb9c274d4c1a4f1637a8c154c5e516e2e466c383f0b3e7a312a10aecf"+ "digest": "sha256:f6f41462aee1e8972b44989de8c669ace6b4caf1b2f8fd2867edc8fcd3d77508"},{"name": "@narratage/film",@@ -624,7 +624,7 @@{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:08b62e21bd0f6f03fd7e2b481f820c56006a0ff92e6a7318927bd0c3607223f5"+ "digest": "sha256:6e4dbfde67625679cce91e137ed6b0efecee062915d802467ad7e106e75f55fc"},{"name": "@narratage/media-track",@@ -659,7 +659,7 @@{"name": "@narratage/render-hyperframes","version": "0.0.0-dev",- "digest": "sha256:28847756a67fc7a9c304d9ed358964e422956d86da6f53e3b7d2ebccdff54cad"+ "digest": "sha256:18d5f3a63c71135771af5e9b47f8bcd6cf41662adcb3f1d1628dce6138f75931"},{"name": "@narratage/run-markup",@@ -679,12 +679,12 @@{"name": "@narratage/seedance-speaker","version": "0.0.0-dev",- "digest": "sha256:e8a18e87c2cf40a900d0bc6fb4e151f2fe2d89922ece069812e89aea2c2bedfd"+ "digest": "sha256:6d57cf8c86233c8381a16248cbe917e64aafecaa2b661d685274b755b9b0b0e9"},{"name": "@narratage/seedance","version": "0.0.0-dev",- "digest": "sha256:66cdf600171833956e1f5c5494f6757471d8f795a4d8fe7cfabd791303127ecf"+ "digest": "sha256:72e172d81787cb632a903ab5f3f56ff7bb12daf57f3e2f927c70045db8140b88"},{"name": "@narratage/semantic-map",@@ -755,6 +755,11 @@"name": "@narratage/whisperx","version": "0.0.0-dev","digest": "sha256:7209d52d61e9c9237b59695a30f2f04a055d83053f06439ba9e9a31dd6cd189c"+ },+ {+ "name": "cmu-pronouncing-dictionary",+ "version": "3.0.0",+ "digest": "sha256:11f46faa5a54006cd9ffd84ceaa6177765922a7646640c5f84ee1a16d04815d8"}],"packages": [@@ -809,8 +814,8 @@"name": "@narratage/estimate","version": "0.0.0-dev"},- "facetsDigest": "sha256:55e886419b6c25c3ab08c21401896d0352c90dfa32666f71882918309461e5c2",- "closureDigest": "sha256:46469d00067903d023a07077c0d06bac11668e54e00768dea9f0820504ef2e7a"+ "facetsDigest": "sha256:7abde7f9ceaa235797bede77ed5ac9ea34c3b8cde8ab2aaa037f6b705608d282",+ "closureDigest": "sha256:fd705f372501de14e37daa827353d60c5a090cc294c752c9e026e41360beaceb"},{"specifier": "@narratage/film",@@ -863,8 +868,8 @@"name": "@narratage/media-pipeline","version": "0.0.0-dev"},- "facetsDigest": "sha256:b7bf62142837adc7bad4f8c4b74b60ee6b4ade74140788cf4b682c140893b4af",- "closureDigest": "sha256:3ed0d2f8a7e8d0f3bbb3c4a1ee354967afe24311cd567f562ad766bdca52387c"+ "facetsDigest": "sha256:976b989d377ddaeee207cdf64d76bac74ed3c95f6a6b00e4b9220f4aa2680c50",+ "closureDigest": "sha256:4496f7f9544715877ba1c75e5954a0aa70d48902522ebd6cb89c1d24b34a0f75"},{"specifier": "@narratage/narrative",@@ -890,8 +895,8 @@"name": "@narratage/render-hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:965f293a64a8b1c7ae87e6e13986a751d4c26a2fcded9dd81fc24a7790980b45",- "closureDigest": "sha256:7f0830d7c6714b686acbd3159b4980ffe95f99100e330ac252506e8972b8186a"+ "facetsDigest": "sha256:4210736adc699b3b5f86bf8b0cd84cd7a3c946754c8f19ceae5e0390a3d1d9c5",+ "closureDigest": "sha256:deb47bd7c049ff1ad23b9e2e97caeda6a224067aa0bde9b2336224ef211165c3"},{"specifier": "@narratage/run-markup",@@ -917,8 +922,8 @@"name": "@narratage/seedance","version": "0.0.0-dev"},- "facetsDigest": "sha256:4750e07548e9706f3a0d8da1f8ca1960a27227e1cae647f743934ff2338e11ae",- "closureDigest": "sha256:c329b039f8560a54b56749f7b96329c6c06605d58165a0c6ba5e81a34e84fd9f"+ "facetsDigest": "sha256:1b11ad1dd6e65d9b836dec7fc5b9b03e40c8178a2c2a579144c25a57896d9832",+ "closureDigest": "sha256:b05b8f83592def14f8f2fd0b59279627941b9ec9b76a1d8d9e2bcc4a1333f903"},{"specifier": "@narratage/seedance-speaker",@@ -926,8 +931,8 @@"name": "@narratage/seedance-speaker","version": "0.0.0-dev"},- "facetsDigest": "sha256:56ba468bb0aba538382354da37eb7fb51b0c1d1c8b61bbe46bd279861cb8fc7d",- "closureDigest": "sha256:c16a1aa701a3f793f4a79988f9b860d1fe3dd81c1dce0ab81e3633cd24801cff"+ "facetsDigest": "sha256:b86103d4ccac16408533f2c9d5fd34edf4016f9323554c4550cb1a8fbfbe2e38",+ "closureDigest": "sha256:94a78d68a238bc96ceb414d1d8ee87ab5b96edb6a03aae3947edb097d5112d86"},{"specifier": "@narratage/semantic-map",@@ -989,8 +994,8 @@"name": "@narratage/speech-spine","version": "0.0.0-dev"},- "facetsDigest": "sha256:c2d3ae7c5a676c66368c9225fd516aa6cfa7dab749123f7ecd31007628ee4666",- "closureDigest": "sha256:38365376d684cb70f6d0e939042320c7ac309f8c4272affb0c2a91890419e45c"+ "facetsDigest": "sha256:4af25fdcd03b5dde7c6151419885646c6862978aa5c88fe3df66c7daec1b0199",+ "closureDigest": "sha256:ce29d7f041624590d03d9f0d2358c730b14464b63247f8df84a701e7000581b6"},{"specifier": "@narratage/svs",@@ -1025,9 +1030,9 @@"name": "@narratage/whisperx","version": "0.0.0-dev"},- "facetsDigest": "sha256:be22a14fb7f6eb437aa17b541a35320f5e26b77d7d9925702b2977609f874fe3",- "closureDigest": "sha256:4b2e7e4d12e50b6b8dad43c2c080dd6d5d04475a155e22cf35bb61069bba1f8a"+ "facetsDigest": "sha256:73fceef213e8d6bcad92532c5d712842e62921743b81bc7bb4d9022a92067359",+ "closureDigest": "sha256:32f2a12214d47f6539084d13a87fd95104f458340fae2fa3aee58eee0e111f92"}],- "digest": "sha256:3f47adb10eda3a03bec6e8017ab1197b7ccf6448b21a289a9cc9c0f621e18478"+ "digest": "sha256:c1e337ed5515348f2f7dfb7b584280a1dd6c037ce2da58a5e8ac014f4e9f4c9e"}diff --git a/examples/talking-head-aroll/svml.runtime-packages.lock b/examples/talking-head-aroll/svml.runtime-packages.lockindex b5e433c7..8b71d264 100644--- a/examples/talking-head-aroll/svml.runtime-packages.lock+++ b/examples/talking-head-aroll/svml.runtime-packages.lock@@ -54,7 +54,7 @@{"name": "@narratage/composition","version": "0.0.0-dev",- "digest": "sha256:acc55c73eaf46558b695532a495bc078a72337ac727b97eaf412bc7147336d76"+ "digest": "sha256:c9bfd8e6d07dc38a14f622b4ce545752dee7df09c2458d4e4798dc87eb9ce997"},{"name": "@narratage/core",@@ -84,22 +84,22 @@{"name": "@narratage/hyperframes","version": "0.0.0-dev",- "digest": "sha256:cd4b8c4cfb11307e0a487fa5d065a961c1cfb701a48f6d293f98b90c6d01b5cb"+ "digest": "sha256:cbe9359d2a59fcde65e3b8d4fef43c9965b5e825b2d517fb25babad4c18ed512"},{"name": "@narratage/markup","version": "0.0.0-dev",- "digest": "sha256:8a8f2e935fa5b3248661b426b8429df007296f786dacf147c829b964410ce93d"+ "digest": "sha256:d038863cedf4d22162dc0b390485131caf0977be11032861415a766d068e3745"},{"name": "@narratage/media-execution","version": "0.0.0-dev",- "digest": "sha256:3a876a5c4cd879fd5155ff81c1c7455af7c122ebe6900f67fec1e9e7d76b49c9"+ "digest": "sha256:2bca4da3fe0325847811888d2ded010761d4dc447b8ec79bec9710bca5011c3e"},{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:08b62e21bd0f6f03fd7e2b481f820c56006a0ff92e6a7318927bd0c3607223f5"+ "digest": "sha256:27b943e3fd51adefaf329c84717dae9174b5126a745116f6cf5e9df372fd886e"},{"name": "@narratage/media",@@ -109,7 +109,7 @@{"name": "@narratage/narrative","version": "0.0.0-dev",- "digest": "sha256:410b1c0b30dfe4b8a81e67bbb5833dc915418337cce71325fb1bb8b93f227fac"+ "digest": "sha256:c25b30b1881baa443ca91447f2529ac744feebe54eb2e4fbf3d2ba1d797a457d"},{"name": "@narratage/program-space",@@ -139,7 +139,7 @@{"name": "@narratage/provider-media-local","version": "0.0.0-dev",- "digest": "sha256:8c5065798511380666b03f65b40fd0acbf02d7fa5c6c76ad695631d0ce023d49"+ "digest": "sha256:afedec770515f6cf0e06d42321406a1ecdf9e06c249c0575791518f924381480"},{"name": "@narratage/provider-whisperx-local",@@ -204,7 +204,7 @@{"name": "@narratage/text","version": "0.0.0-dev",- "digest": "sha256:989b56f87a21e907aca26f768262d43e5fe4a4ab3efee17595d774d4ccbca141"+ "digest": "sha256:ec9b6917a6f9d52efa6f62a9f40c4d1e3ae85623b15bdc7dffa60f96c50a7b4b"},{"name": "@narratage/validation",@@ -870,7 +870,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b2636492ab0b0b500d674899ab2ff14fc61c7f2b22ef44eb8f816f141b21ec4d",- "closureDigest": "sha256:d61896f51d88c5d07a1d198a1879055e62986b23dd50f00437486da512e6faa2"+ "closureDigest": "sha256:4f2f3ddd8072d04af19f9b3f2dcec5b4a9c909c1dc3fe4aa789b190992fea9ee"},{"specifier": "@narratage/provider-hyperframes-local",@@ -879,7 +879,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b14b721bdc0c9d5d9ced1c345107c92a3a5878f3383dea2ecd3e310442c2a40c",- "closureDigest": "sha256:a76597166b1e27453d77f3b209498f88e6ae1204249b51d531b71258b75d6f3d"+ "closureDigest": "sha256:fa04cd3045315c2331c4466967c4e6e1c411bff578b6afd5e451dd6074f7940c"},{"specifier": "@narratage/provider-kie",@@ -888,7 +888,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:e61ebb911a33d4c5d0e4e4ee4fb69cdb790f4ced050ef35f7472bfbd72013d9b",- "closureDigest": "sha256:606dc72a6f601a7aec856efa5436a0f74e3f0b505569a4017f8f30675f444fb8"+ "closureDigest": "sha256:081710459182986ed07fc269457303e6063c21acf0ee418d1edb083ec82bb53a"},{"specifier": "@narratage/provider-media-local",@@ -897,7 +897,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:701abcc923bf8a711c20b6e308d8cc8be27bf560f249b8c5a801e1dbd9cb3979",- "closureDigest": "sha256:5dcea5c6536c78a52a8900b8dc864a4c9c2353ef57cf8122a2590c7be62e8e76"+ "closureDigest": "sha256:bacffa564f65f413425b2bf895e4cb01dcd2266fa64e1836b7f7fd92d42af186"},{"specifier": "@narratage/provider-whisperx-local",@@ -906,8 +906,8 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:3686b84c42cc2b1bcaccf57cbdcda25a15c7d9d3778fb92d25437f9880ed8f5d",- "closureDigest": "sha256:9d00849016e4f24e679f7176207db44d410cfa4a46cf65df761d5cc397502655"+ "closureDigest": "sha256:bd4ab81b2d7e24531824ff85c4a97013f08f34c95570dc674884026c0ce18018"}],- "digest": "sha256:96871b6a5f84cb7c60d835eebc0f384025b641d7cb0a2f7239db7f77be4a7632"+ "digest": "sha256:9f01815de837f7af502b90eda6f7f37d3a48016671cb010cedb23892fbd4c5c6"}diff --git a/packages/estimate/README.md b/packages/estimate/README.mdindex facf1d39..d67e8030 100644--- a/packages/estimate/README.md+++ b/packages/estimate/README.md@@ -3,8 +3,10 @@Provider-free deterministic speech-duration planning.`<estimate:Speech>` consumes an ordinary `Text` value and produces the public `SpeechDuration`-contract. It uses a language-aware syllable/character/mora rate, then applies an-explicit tail padding, bounds and rounding policy. It performs no network or media work.+contract. It uses a language-aware pronunciation-unit delivery density, then applies explicit+bounds and rounding. English dictionary words use CMUdict;+unknown words use a deterministic spelling fallback. The consumed Text remains authoritative:+the estimator never invents how numbers, names or acronyms should be pronounced.```xml<estimate:Speech@@ -12,12 +14,21 @@ explicit tail padding, bounds and rounding policy. It performs no network or medsource={story.segment.opening.speech}language="en"pace="normal"- padding="0.3"min="4"max="15"- rounding="ceil"+ rounding="round"/>```+The English pace presets are `slow = 4.2`, `normal = 4.6` and `fast = 5.0`+syllables per second. A policy may use a positive numeric `rate` instead of `pace`:++```xml+<estimate:Speech id="opening-duration" source={story.segment.opening.speech} rate="4.75"/>+```++Every policy field is explicit. `round` selects the nearest provider-compatible integer second. `ceil` remains available+when a project explicitly prefers a conservative, never-shorter duration.+The output is `opening-duration.duration`. Estimate is only one possible producer of this sharedvalue; model packages consume the contract rather than depending on this package.diff --git a/packages/estimate/package.json b/packages/estimate/package.jsonindex 661675ad..62f63449 100644--- a/packages/estimate/package.json+++ b/packages/estimate/package.json@@ -12,10 +12,11 @@"dependencies": {"@narratage/component-kit": "workspace:*","@narratage/elaborator": "workspace:*",+ "@narratage/markup": "workspace:*","@narratage/protocol": "workspace:*","@narratage/speech": "workspace:*","@narratage/svs": "workspace:*","@narratage/text": "workspace:*",- "@narratage/markup": "workspace:*"+ "cmu-pronouncing-dictionary": "3.0.0"}}diff --git a/packages/estimate/src/index.ts b/packages/estimate/src/index.tsindex 1f354d79..c231d927 100644--- a/packages/estimate/src/index.ts+++ b/packages/estimate/src/index.ts@@ -15,6 +15,7 @@ export {detectSpeechEstimateLanguage,estimateSpeechDuration,estimateSpeechImplementationDigest,+ resolveSpeechEstimateRate,resolveSpeechEstimateLanguage,sealSpeechEstimatePolicy,} from "./program.js";diff --git a/packages/estimate/src/manifest.ts b/packages/estimate/src/manifest.tsindex 2e57ab73..c258b083 100644--- a/packages/estimate/src/manifest.ts+++ b/packages/estimate/src/manifest.ts@@ -13,7 +13,9 @@ export const estimateTypes = {export const estimateProducers = {speech: { module: estimateModuleRef, name: "estimate-speech-duration" },} satisfies Record<string, ProducerRef>;-export const estimateSurfaceImplementationDigest = digestOf("@narratage/estimate/speech-surface@1");+export const estimateSurfaceImplementationDigest = digestOf(+ "@narratage/estimate/speech-surface@1:explicit-policy-without-padding",+);const number = { kind: "number" } as const satisfies ValueSchema;const speechEstimatePolicySchema: ValueSchema = {@@ -21,8 +23,8 @@ const speechEstimatePolicySchema: ValueSchema = {fields: {contract: { schema: { kind: "literal", value: "svml.speech-estimate-policy@1" } },language: { schema: { kind: "string", enum: ["auto", "en", "zh", "ja", "es"] } },- pace: { schema: { kind: "string", enum: ["slow", "normal", "fast"] } },- paddingSec: { schema: number },+ pace: { schema: { kind: "string", enum: ["slow", "normal", "fast"] }, optional: true },+ rate: { schema: { kind: "number", minimum: 0.000001 }, optional: true },minimumSec: { schema: number },maximumSec: { schema: number },rounding: { schema: { kind: "string", enum: ["none", "round", "ceil"] } },diff --git a/packages/estimate/src/program.ts b/packages/estimate/src/program.tsindex 39bee859..9b9aed98 100644--- a/packages/estimate/src/program.ts+++ b/packages/estimate/src/program.ts@@ -2,30 +2,33 @@ import type { Text } from "@narratage/text";import { assertSpeechDurationIdentity, sealSpeechDuration } from "@narratage/speech";import type { SpeechDuration } from "@narratage/speech";import { digestOf } from "@narratage/protocol";+import { dictionary as cmuPronouncingDictionary } from "cmu-pronouncing-dictionary";import type {ResolvedSpeechEstimateLanguage,SpeechEstimateLanguage,+ SpeechEstimatePace,SpeechEstimatePolicy,} from "./types.js";-export const estimateSpeechImplementationDigest = digestOf("@narratage/estimate/estimate-speech@1");+export const estimateSpeechImplementationDigest = digestOf(+ "@narratage/estimate/estimate-speech@1:explicit-delivery-density-without-padding",+);-const BASE_RATE: Readonly<Record<ResolvedSpeechEstimateLanguage, number>> = {- en: 4.5,- zh: 5.25,- ja: 7.5,- es: 5.9,+const PACE_RATE: Readonly<Record<+ ResolvedSpeechEstimateLanguage,+ Readonly<Record<SpeechEstimatePace, number>>+>> = {+ // English is calibrated as delivery density, not pause-free articulation rate.+ en: { slow: 4.2, normal: 4.6, fast: 5 },+ // Preserve the existing non-English behavior until those languages are audited.+ zh: { slow: 4.2, normal: 5.25, fast: 6.5625 },+ ja: { slow: 6, normal: 7.5, fast: 9.375 },+ es: { slow: 4.72, normal: 5.9, fast: 7.375 },};-const PACE_MULTIPLIER = {- slow: 0.8,- normal: 1,- fast: 1.25,-} as const;-function words(text: string): string[] {- return text.match(/[\p{L}\p{N}]+(?:['-][\p{L}\p{N}]+)?/gu) ?? [];+ return text.match(/[\p{L}\p{N}]+(?:['’\-‐‑–][\p{L}\p{N}]+)*/gu) ?? [];}function looksSpanish(text: string): boolean {@@ -46,8 +49,23 @@ export function detectSpeechEstimateLanguage(text: string): ResolvedSpeechEstimareturn "en";}-function englishSyllables(word: string): number {- const normalized = word.toLowerCase().replace(/[^a-z]/gu, "");+function normalizedEnglishWord(word: string): string {+ return word+ .toLowerCase()+ .replace(/’/gu, "'")+ .replace(/[‐‑–]/gu, "-")+ .replace(/[^a-z'-]/gu, "");+}++function dictionarySyllables(word: string): number | undefined {+ const pronunciation = cmuPronouncingDictionary[word];+ if (pronunciation === undefined) return undefined;+ const syllables = pronunciation.split(" ").filter((phoneme) => /\d$/u.test(phoneme)).length;+ return syllables > 0 ? syllables : undefined;+}++function heuristicEnglishSyllables(word: string): number {+ const normalized = word.replace(/[^a-z]/gu, "");if (!normalized) return 0;if (normalized.length <= 3) return 1;const stripped = normalized@@ -56,6 +74,20 @@ function englishSyllables(word: string): number {return Math.max(1, stripped.match(/[aeiouy]{1,2}/gu)?.length ?? 1);}+function englishSyllables(word: string): number {+ const normalized = normalizedEnglishWord(word);+ if (!normalized) return 0;+ const direct = dictionarySyllables(normalized);+ if (direct !== undefined) return direct;+ if (normalized.includes("-")) {+ return normalized+ .split("-")+ .filter(Boolean)+ .reduce((sum, part) => sum + (dictionarySyllables(part) ?? heuristicEnglishSyllables(part)), 0);+ }+ return heuristicEnglishSyllables(normalized);+}+function spanishVowelNuclei(run: string): number {let nuclei = 0;let pending = "";@@ -108,13 +140,15 @@ export function sealSpeechEstimatePolicy(value: SpeechEstimatePolicy): SpeechEst}export function assertSpeechEstimatePolicy(value: SpeechEstimatePolicy): void {+ const hasPace = value.pace !== undefined;+ const hasRate = value.rate !== undefined;if (value.contract !== "svml.speech-estimate-policy@1"|| !(["auto", "en", "zh", "ja", "es"] as const).includes(value.language)- || !(["slow", "normal", "fast"] as const).includes(value.pace)+ || hasPace === hasRate+ || (hasPace && !(["slow", "normal", "fast"] as const).includes(value.pace))+ || (hasRate && (!Number.isFinite(value.rate) || value.rate <= 0))|| !(["none", "round", "ceil"] as const).includes(value.rounding)- || !Number.isFinite(value.paddingSec)- || value.paddingSec < 0|| !Number.isFinite(value.minimumSec)|| value.minimumSec < 0|| !Number.isFinite(value.maximumSec)@@ -125,6 +159,14 @@ export function assertSpeechEstimatePolicy(value: SpeechEstimatePolicy): void {}}+export function resolveSpeechEstimateRate(+ policy: SpeechEstimatePolicy,+ language: ResolvedSpeechEstimateLanguage,+): number {+ assertSpeechEstimatePolicy(policy);+ return policy.rate ?? PACE_RATE[language][policy.pace];+}+function assertSpeechText(value: Text): void {if (value.contract !== "svml.text@1"@@ -149,7 +191,7 @@ export function estimateSpeechDuration(const language = policy.language === "auto" ? detectSpeechEstimateLanguage(text.value) : policy.language;const units = countSpeechEstimateUnits(text.value, language);if (units < 1) throw new Error("Speech Text contains no countable speech units");- const raw = units / (BASE_RATE[language] * PACE_MULTIPLIER[policy.pace]) + policy.paddingSec;+ const raw = units / resolveSpeechEstimateRate(policy, language);const firstClamp = Math.min(policy.maximumSec, Math.max(policy.minimumSec, raw));const durationSec = Math.min(policy.maximumSec, Math.max(policy.minimumSec, rounded(firstClamp, policy.rounding)));const duration = sealSpeechDuration({diff --git a/packages/estimate/src/surface.ts b/packages/estimate/src/surface.tsindex 8ae64ed5..436cd5a2 100644--- a/packages/estimate/src/surface.ts+++ b/packages/estimate/src/surface.ts@@ -24,7 +24,7 @@ function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedfunction attributes(element: StructuredElement): void {const required = ["id", "source"];- const direct = ["language", "pace", "padding", "min", "max", "rounding"];+ const direct = ["language", "pace", "rate", "min", "max", "rounding"];const optional = ["policy", ...direct];const allowed = new Set([...required, ...optional]);if (@@ -36,6 +36,18 @@ function attributes(element: StructuredElement): void {if (element.attributes.policy !== undefined && direct.some((name) => element.attributes[name] !== undefined)) {throw new Error(`${element.name}.policy cannot be combined with inline estimate parameters`);}+ if (element.attributes.policy === undefined) {+ const missing = ["language", "min", "max", "rounding"]+ .filter((name) => element.attributes[name] === undefined);+ if (missing.length > 0) {+ throw new Error(`${element.name} inline policy requires ${missing.join(", ")}`);+ }+ const hasPace = element.attributes.pace !== undefined;+ const hasRate = element.attributes.rate !== undefined;+ if (hasPace === hasRate) {+ throw new Error(`${element.name} inline policy requires exactly one of pace or rate`);+ }+ }if (element.children.some((child) => child.kind === "element" || child.value.trim().length > 0)) {throw new Error(`${element.name} must be empty`);}@@ -74,32 +86,41 @@ export function speechEstimatePolicyFromRecipe(subject = `SVS Recipe ${recipe.path}`,) {if (recipe.contract !== "svml.svs-recipe@1") throw new Error(`${subject} is invalid`);- const allowed = new Set(["language", "pace", "padding", "min", "max", "rounding"]);+ const allowed = new Set(["language", "pace", "rate", "min", "max", "rounding"]);const unknown = Object.keys(recipe.properties).filter((name) => !allowed.has(name));if (unknown.length > 0) throw new Error(`${subject} contains unknown property ${unknown[0]}`);- const string = (name: string, fallback: string): string => {- const value = recipe.properties[name] ?? fallback;+ const string = (name: string): string => {+ const value = recipe.properties[name];if (typeof value !== "string" || value.trim().length === 0) {throw new Error(`${subject}.${name} must be a non-empty string`);}return value.trim();};- const finite = (name: string, fallback: number): number => {- const value = recipe.properties[name] ?? fallback;+ const finite = (name: string): number => {+ const value = recipe.properties[name];if (typeof value !== "number" || !Number.isFinite(value)) {throw new Error(`${subject}.${name} must be a finite number`);}return value;};- return sealSpeechEstimatePolicy({+ const pace = recipe.properties.pace;+ const rate = recipe.properties.rate;+ if ((pace !== undefined) === (rate !== undefined)) {+ throw new Error(`${subject} requires exactly one of pace or rate`);+ }+ if (rate !== undefined && (typeof rate !== "number" || !Number.isFinite(rate))) {+ throw new Error(`${subject}.rate must be a finite number`);+ }+ const common = {contract: "svml.speech-estimate-policy@1",- language: string("language", "auto") as SpeechEstimateLanguage,- pace: string("pace", "normal") as SpeechEstimatePace,- paddingSec: finite("padding", 0.3),- minimumSec: finite("min", 4),- maximumSec: finite("max", 15),- rounding: string("rounding", "ceil") as SpeechEstimateRounding,- });+ language: string("language") as SpeechEstimateLanguage,+ minimumSec: finite("min"),+ maximumSec: finite("max"),+ rounding: string("rounding") as SpeechEstimateRounding,+ } as const;+ return rate === undefined+ ? sealSpeechEstimatePolicy({ ...common, pace: string("pace") as SpeechEstimatePace })+ : sealSpeechEstimatePolicy({ ...common, rate });}function text(element: StructuredElement, name: string, fallback?: string): string {@@ -110,8 +131,8 @@ function text(element: StructuredElement, name: string, fallback?: string): strireturn value.trim();}-function number(element: StructuredElement, name: string, fallback: number): number {- const value = Number(text(element, name, String(fallback)));+function number(element: StructuredElement, name: string): number {+ const value = Number(text(element, name));if (!Number.isFinite(value)) throw new Error(`${element.name}.${name} must be a finite number`);return value;}@@ -138,12 +159,13 @@ export const decodeSpeechEstimateSurface: StructuredSurfaceHandler = ({ element,const policy = element.attributes.policy === undefined? sealSpeechEstimatePolicy({contract: "svml.speech-estimate-policy@1",- language: text(element, "language", "auto") as SpeechEstimateLanguage,- pace: text(element, "pace", "normal") as SpeechEstimatePace,- paddingSec: number(element, "padding", 0.3),- minimumSec: number(element, "min", 4),- maximumSec: number(element, "max", 15),- rounding: text(element, "rounding", "ceil") as SpeechEstimateRounding,+ language: text(element, "language") as SpeechEstimateLanguage,+ ...(element.attributes.rate === undefined+ ? { pace: text(element, "pace") as SpeechEstimatePace }+ : { rate: number(element, "rate") }),+ minimumSec: number(element, "min"),+ maximumSec: number(element, "max"),+ rounding: text(element, "rounding") as SpeechEstimateRounding,}): recipePolicy(element, resolveReference);const policyId = `${id}.policy`;diff --git a/packages/estimate/src/types.ts b/packages/estimate/src/types.tsindex d3524a0a..a2a43243 100644--- a/packages/estimate/src/types.ts+++ b/packages/estimate/src/types.ts@@ -3,12 +3,15 @@ export type ResolvedSpeechEstimateLanguage = Exclude<SpeechEstimateLanguage, "auexport type SpeechEstimatePace = "slow" | "normal" | "fast";export type SpeechEstimateRounding = "none" | "round" | "ceil";-export type SpeechEstimatePolicy = {+type SpeechEstimatePolicyBase = {readonly contract: "svml.speech-estimate-policy@1";readonly language: SpeechEstimateLanguage;- readonly pace: SpeechEstimatePace;- readonly paddingSec: number;readonly minimumSec: number;readonly maximumSec: number;readonly rounding: SpeechEstimateRounding;};++export type SpeechEstimatePolicy = SpeechEstimatePolicyBase & (+ | { readonly pace: SpeechEstimatePace; readonly rate?: never }+ | { readonly pace?: never; readonly rate: number }+);diff --git a/packages/estimate/test/estimate.test.ts b/packages/estimate/test/estimate.test.tsindex dd907fad..93823c2f 100644--- a/packages/estimate/test/estimate.test.ts+++ b/packages/estimate/test/estimate.test.ts@@ -1,31 +1,91 @@import assert from "node:assert/strict";import test from "node:test";-import { sealText } from "@narratage/text";+import { parseStructuredElement } from "@narratage/markup";+import { sealText, textTypes } from "@narratage/text";import {countSpeechEstimateUnits,+ decodeSpeechEstimateSurface,estimateSpeechDuration,sealSpeechEstimatePolicy,speechEstimatePolicyFromRecipe,} from "@narratage/estimate";+function estimateSurface(source: string) {+ return decodeSpeechEstimateSurface({+ sourceName: "estimate.svml",+ element: parseStructuredElement({ name: "estimate.svml", text: source }, 0).element,+ resolveReference: (path) => path === "speech"+ ? { path, ref: { kind: "record", id: path }, type: textTypes.text }+ : undefined,+ resolveAsset: () => { throw new Error("Estimate has no assets"); },+ });+}+const normal = sealSpeechEstimatePolicy({contract: "svml.speech-estimate-policy@1",language: "en",pace: "normal",- paddingSec: 0.3,minimumSec: 4,maximumSec: 15,- rounding: "ceil",+ rounding: "round",+});++test("English dictionary pronunciation drives ordinary word syllables", () => {+ assert.equal(countSpeechEstimateUnits("Video presented needed ideas queue.", "en"), 12);+ assert.equal(countSpeechEstimateUnits("I'm going back-and-forth.", "en"), 6);+ assert.equal(countSpeechEstimateUnits("I’m going back-and-forth.", "en"), 6);+ assert.equal(countSpeechEstimateUnits("45%", "en"), 0);});-test("normal English speech estimate follows the syllable-rate policy", () => {+test("normal English speech estimate follows the delivery-density policy", () => {const source = sealText("Video editing begins with meaning, not a pile of clips on a timeline.",);- assert.equal(countSpeechEstimateUnits(source.value, "en"), 21);+ assert.equal(countSpeechEstimateUnits(source.value, "en"), 20);const result = estimateSpeechDuration(source, normal);- assert.equal(result.durationSec, 5);+ assert.equal(result.durationSec, 4);+});++test("English pace presets occupy adjacent integer durations", () => {+ const source = sealText(Array.from({ length: 49 }, () => "day").join(" "));+ const estimate = (pace: "slow" | "normal" | "fast") => estimateSpeechDuration(+ source,+ sealSpeechEstimatePolicy({+ contract: normal.contract,+ language: normal.language,+ pace,+ minimumSec: normal.minimumSec,+ maximumSec: normal.maximumSec,+ rounding: normal.rounding,+ }),+ ).durationSec;+ assert.deepEqual([estimate("slow"), estimate("normal"), estimate("fast")], [12, 11, 10]);+});++test("a numeric rate gives SVS a continuous author-controlled pace", () => {+ const source = sealText(Array.from({ length: 49 }, () => "day").join(" "));+ const result = estimateSpeechDuration(source, sealSpeechEstimatePolicy({+ contract: normal.contract,+ language: normal.language,+ rate: 4.75,+ minimumSec: normal.minimumSec,+ maximumSec: normal.maximumSec,+ rounding: normal.rounding,+ }));+ assert.equal(result.durationSec, 10);+});++test("the author Surface requires one complete explicit inline policy", async () => {+ const output = await estimateSurface('<estimate:Speech id="duration" source={speech} language="en" rate="4.75" min="4" max="15" rounding="round"/>');+ const value = output.records[0]!.value;+ assert.equal(value.kind, "inline");+ assert.equal(value.kind === "inline" && (value.value as { readonly rate?: number }).rate, 4.75);+ assert.throws(+ () => estimateSurface('<estimate:Speech id="duration" source={speech} language="en" pace="normal" rate="4.75" min="4" max="15" rounding="round"/>'),+ /exactly one/u,+ );+ assert.throws(() => estimateSurface('<estimate:Speech id="duration" source={speech} rate="4.75"/>'), /requires/u);});test("the minimum applies before rounding and the maximum applies after rounding", () => {@@ -40,15 +100,20 @@ test("an SVS Recipe configures one reusable estimate policy without becoming exepath: "speech.normal",properties: {language: "en",- pace: "normal",- padding: 0.3,+ rate: 4.75,min: 4,max: 15,- rounding: "ceil",+ rounding: "round",},});assert.equal(policy.language, "en");+ assert.equal(policy.rate, 4.75);assert.equal(policy.maximumSec, 15);+ assert.throws(() => speechEstimatePolicyFromRecipe({+ contract: "svml.svs-recipe@1",+ path: "speech.ambiguous",+ properties: { pace: "normal", rate: 4.6 },+ }), /exactly one/u);assert.throws(() => speechEstimatePolicyFromRecipe({contract: "svml.svs-recipe@1",path: "speech.invalid",diff --git a/packages/media-execution/README.md b/packages/media-execution/README.mdindex 82094367..9e895133 100644--- a/packages/media-execution/README.md+++ b/packages/media-execution/README.md@@ -3,10 +3,13 @@Shared FFmpeg execution body for Narratage media Providers.`@narratage/media-pipeline` owns provider-neutral Needs, plans and result contracts. This package-owns the byte-level implementations of the five current operations:+owns the byte-level implementations of the eight current operations:- inspect every media stream;- normalize selected audio/video streams;+- transform synchronized media with ordered trim and pitch-preserving retime operations;+- extract one selected audio stream as an ordinary 48 kHz stereo WAV Artifact;+- extract a first, last, indexed or time-selected frame as PNG;- project canonical 16 kHz mono speech-evidence audio;- render an exact sample-domain audio program;- mux rendered visual and audio products.diff --git a/packages/media-execution/src/execute.ts b/packages/media-execution/src/execute.tsindex 54c6ae71..69b9a561 100644--- a/packages/media-execution/src/execute.ts+++ b/packages/media-execution/src/execute.ts@@ -3,26 +3,32 @@ import { createHash } from "node:crypto";import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";import { tmpdir } from "node:os";import { join } from "node:path";-import { mediaTypes, sealMediaInspection, sealMuxedMedia, sealSynchronizedMedia, sealTimelineAudio, verifyMediaInspection, verifyMediaStreamSelection, verifyRenderedVisual, verifyTimelineAudio } from "@narratage/media";-import type { MediaAudioStream, MediaInspection, MediaRational, MediaStream, MediaStreamSelection, MediaTimestamp, MediaVideoStream, MuxedMedia, RenderedVisual, TimelineAudio } from "@narratage/media";+import { mediaTypes, sealMediaInspection, sealMuxedMedia, sealSynchronizedMedia, sealTimelineAudio, verifyMediaInspection, verifyMediaStreamSelection, verifyRenderedVisual, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";+import type { MediaAudioStream, MediaInspection, MediaRational, MediaStream, MediaStreamSelection, MediaTimestamp, MediaVideoStream, MuxedMedia, RenderedVisual, SynchronizedMedia, TimelineAudio } from "@narratage/media";import type { ProgramSpace } from "@narratage/program-space";import { assertSpeechEvidenceAudioIdentity, sealSpeechEvidenceAudio, speechEvidenceSampleBoundary, speechTypes } from "@narratage/speech";import type { SpeechEvidenceAudio } from "@narratage/speech";import {mediaPipelineCapabilities,+ verifyAudioExtractionRequest,verifyAudioProgramPlan,+ verifyFrameExtractionRequest,+ verifyMediaTransformProgram,+ type ExtractAudioNeed,+ type ExtractFrameNeed,type InspectMediaNeed,type MuxMediaNeed,type NormalizeMediaNeed,type ProjectSpeechEvidenceAudioNeed,type RenderAudioNeed,+ type TransformMediaNeed,} from "@narratage/media-pipeline";-import type { AudioProgramClip, AudioProgramPlan } from "@narratage/media-pipeline";+import type { AudioProgramClip, AudioProgramPlan, MediaTransformOperation } from "@narratage/media-pipeline";import {canonicalize,digestOf,} from "@narratage/protocol";-import type { BlobRef, CanonicalValue } from "@narratage/protocol";+import type { BlobRef, CanonicalValue, StoredValue } from "@narratage/protocol";import { parseMediaInspection } from "./probe.js";import {@@ -36,7 +42,7 @@ import type { AnimatedWebp } from "./webp.js";/*** Where the bytes live and which binaries transform them.*- * These five operations are the whole of Narratage's media execution, and they+ * These byte operations are the whole of Narratage's media execution, and they* are written once. A local Provider supplies the Build's own ArtifactStore and* the ffmpeg on its PATH; a Lambda Provider supplies an S3-backed gateway and* the ffmpeg carried by its deployment. Nothing below knows which it is, so the two@@ -60,7 +66,7 @@ export type MediaExecutionEnvironment = {};export type MediaOperationResult = {- readonly value: CanonicalValue;+ readonly value: StoredValue;readonly metadata: CanonicalValue;};@@ -68,6 +74,9 @@ export type MediaOperationResult = {export const mediaOperationContracts = {inspect: "svml.inspect-media-request@1",normalize: "svml.normalize-media-request@1",+ transform: "svml.transform-media-request@1",+ extractAudio: "svml.extract-audio-request@1",+ extractFrame: "svml.extract-frame-request@1",projectSpeechEvidenceAudio: "svml.project-speech-evidence-audio-request@1",renderAudio: "svml.render-audio-request@1",mux: "svml.mux-media-request@1",@@ -423,7 +432,11 @@ async function outputInspection(args: {...(args.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: args.sharedLibraryPath }) });}-function result(value: CanonicalValue, metadata: CanonicalValue): MediaOperationResult {+function inlineResult(value: CanonicalValue, metadata: CanonicalValue): MediaOperationResult {+ return { value: { kind: "inline", value }, metadata };+}++function artifactResult(value: BlobRef, metadata: CanonicalValue): MediaOperationResult {return { value, metadata };}@@ -446,6 +459,48 @@ function normalizeNeed(value: CanonicalValue): NormalizeMediaNeed {return item;}+function transformNeed(value: CanonicalValue): TransformMediaNeed {+ const item = object(value, "TransformMediaNeed") as unknown as TransformMediaNeed;+ assert(item.contract === "svml.transform-media-request@1", "TransformMediaNeed is invalid");+ verifySynchronizedMedia(item.media);+ verifyMediaTransformProgram(item.program);+ assert(item.media.visual !== undefined, "TransformMediaNeed requires synchronized visual media");+ return item;+}++function extractAudioNeed(value: CanonicalValue): ExtractAudioNeed {+ const item = object(value, "ExtractAudioNeed") as unknown as ExtractAudioNeed;+ assert(item.contract === "svml.extract-audio-request@1" && item.source?.kind === "blob",+ "ExtractAudioNeed is invalid");+ assert(Number.isSafeInteger(item.streamIndex) && item.streamIndex >= 0,+ "ExtractAudioNeed streamIndex is invalid");+ verifyAudioExtractionRequest({+ contract: "svml.audio-extraction-request@1",+ audio: { mode: "stream-index", streamIndex: item.streamIndex },+ output: item.output,+ });+ return item;+}++function extractFrameNeed(value: CanonicalValue): ExtractFrameNeed {+ const item = object(value, "ExtractFrameNeed") as unknown as ExtractFrameNeed;+ assert(item.contract === "svml.extract-frame-request@1" && item.source?.kind === "blob",+ "ExtractFrameNeed is invalid");+ assert(Number.isSafeInteger(item.streamIndex) && item.streamIndex >= 0+ && Number.isSafeInteger(item.sourceFrameCount) && item.sourceFrameCount > 0,+ "ExtractFrameNeed stream domain is invalid");+ verifyFrameExtractionRequest({+ contract: "svml.frame-extraction-request@1",+ video: { mode: "stream-index", streamIndex: item.streamIndex },+ at: item.at,+ output: item.output,+ });+ if (item.at.kind === "frame") {+ assert(item.at.index < item.sourceFrameCount, "ExtractFrameNeed frame lies outside the source stream");+ }+ return item;+}+function evidenceAudioNeed(value: CanonicalValue): ProjectSpeechEvidenceAudioNeed {const item = object(value, "ProjectSpeechEvidenceAudioNeed") as unknown as ProjectSpeechEvidenceAudioNeed;assert(item.contract === "svml.project-speech-evidence-audio-request@1" && item.source?.kind === "blob",@@ -566,7 +621,7 @@ export async function executeInspectMedia(...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),}): animatedWebpInspection(need.source, animation);- return result(canonicalize(inspection), canonicalize({ provider: env.label, operation: "inspect" }));+ return inlineResult(canonicalize(inspection), canonicalize({ provider: env.label, operation: "inspect" }));} finally {await rm(work, { recursive: true, force: true }).catch(() => {});}@@ -708,13 +763,237 @@ export async function executeNormalizeMedia(},}),});- return result(canonicalize(media), canonicalize({ provider: env.label, operation: "normalize",+ return inlineResult(canonicalize(media), canonicalize({ provider: env.label, operation: "normalize",visual: visualArtifact !== undefined, audio: audioArtifact !== undefined }));} finally {await rm(work, { recursive: true, force: true }).catch(() => {});}}+type TransformPlan = {+ readonly videoFilters: readonly string[];+ readonly audioFilters: readonly string[];+ readonly frameCount: number;+ readonly sampleFrames: number;+ readonly durationSec: number;+};++function decimal(value: number): string {+ assert(Number.isFinite(value), "Media transform produced a non-finite time");+ return value.toPrecision(15);+}++function compileTransformPlan(media: SynchronizedMedia, operations: readonly MediaTransformOperation[]): TransformPlan {+ const rate = media.timeline.frameRate.numerator / media.timeline.frameRate.denominator;+ let durationSec = media.timeline.frameCount / rate;+ const videoFilters: string[] = ["setpts=PTS-STARTPTS"];+ const audioFilters: string[] = ["asetpts=N/SR/TB"];+ for (const [index, operation] of operations.entries()) {+ if (operation.kind === "trim") {+ const start = operation.startSec ?? 0;+ const end = operation.endSec ?? (operation.tailSec === undefined+ ? durationSec+ : durationSec - operation.tailSec);+ assert(start >= 0 && start < durationSec,+ `Media transform trim ${index} starts outside its current ${durationSec}s timeline`);+ assert(end > start && end <= durationSec + 1e-9,+ `Media transform trim ${index} ends outside its current ${durationSec}s timeline`);+ const boundedEnd = Math.min(end, durationSec);+ videoFilters.push(`trim=start=${decimal(start)}:end=${decimal(boundedEnd)}`, "setpts=PTS-STARTPTS");+ audioFilters.push(`atrim=start=${decimal(start)}:end=${decimal(boundedEnd)}`, "asetpts=N/SR/TB");+ durationSec = boundedEnd - start;+ continue;+ }+ videoFilters.push(`setpts=PTS/${decimal(operation.rate)}`);+ audioFilters.push(...atempo(operation.rate));+ durationSec /= operation.rate;+ }+ const frameCount = Math.max(1, Math.round(durationSec * rate));+ assert(Number.isSafeInteger(frameCount), "Media transform output frame count exceeds safe arithmetic");+ const sampleFrames = roundPositive(+ BigInt(frameCount) * 48_000n * BigInt(media.timeline.frameRate.denominator),+ BigInt(media.timeline.frameRate.numerator),+ );+ const fps = `${media.timeline.frameRate.numerator}/${media.timeline.frameRate.denominator}`;+ videoFilters.push(+ `fps=fps=${fps}:round=near:start_time=0:eof_action=round`,+ `trim=start_frame=0:end_frame=${frameCount}`,+ `setpts=N*${media.timeline.frameRate.denominator}/(${media.timeline.frameRate.numerator}*TB)`,+ );+ audioFilters.push(+ `apad=whole_len=${sampleFrames}`,+ `atrim=start_sample=0:end_sample=${sampleFrames}`,+ "asetpts=N/SR/TB",+ );+ return { videoFilters, audioFilters, frameCount, sampleFrames, durationSec };+}++/** Ordered trim/retime over one exact synchronized A/V value. */+export async function executeTransformMedia(+ env: MediaExecutionEnvironment,+ constraints: CanonicalValue,+): Promise<MediaOperationResult> {+ const need = transformNeed(constraints);+ const media = need.media;+ const plan = compileTransformPlan(media, need.program.operations);+ const work = await mkdtemp(join(tmpdir(), "svml-media-transform-"));+ try {+ const visualPath = join(work, "visual.mp4");+ const audioPath = join(work, "audio.wav");+ const output = join(work, "transformed.mp4");+ await stageArtifact(env, media.visual!.artifact, visualPath);+ if (media.audio !== undefined) await stageArtifact(env, media.audio.artifact, audioPath);+ const argv = ["-y", "-i", visualPath];+ if (media.audio !== undefined) {+ argv.push(+ "-i", audioPath,+ "-filter_complex",+ `[0:v:0]${plan.videoFilters.join(",")}[video];[1:a:0]${plan.audioFilters.join(",")}[audio]`,+ "-map", "[video]", "-map", "[audio]",+ "-c:a", "aac", "-ar", "48000", "-ac", "2",+ );+ } else {+ argv.push("-map", "0:v:0", "-an", "-vf", plan.videoFilters.join(","));+ }+ argv.push(+ "-frames:v", String(plan.frameCount), "-fps_mode", "cfr",+ "-c:v", "libx264", "-preset", "veryfast", "-pix_fmt", "yuv420p",+ "-movflags", "+faststart", output,+ );+ await runProcess({+ executable: env.ffmpegPath,+ argv,+ timeoutMs: env.processTimeoutMs,+ maxStdoutBytes: 64 * 1024,+ ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),+ });+ const inspected = await outputInspection({+ path: output,+ mediaType: "video/mp4",+ ffprobePath: env.ffprobePath,+ timeoutMs: env.processTimeoutMs,+ maxProbeOutputBytes: env.maxProbeOutputBytes,+ ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),+ });+ const videos = inspected.streams.filter((item): item is MediaVideoStream => item.kind === "video");+ const audios = inspected.streams.filter((item): item is MediaAudioStream => item.kind === "audio");+ assert(videos.length === 1 && videos[0]!.role === "moving"+ && videos[0]!.decodedUnitCount === plan.frameCount,+ "Transformed media video differs from its compiled frame domain");+ assert(audios.length === (media.audio === undefined ? 0 : 1),+ "Transformed media audio presence differs from its synchronized input");+ if (audios[0] !== undefined) {+ assert(audios[0].sampleRate === 48_000 && audios[0].channels === 2,+ "Transformed media audio is not 48 kHz stereo");+ }+ const artifact = await env.artifacts.put(await readFile(output), "video/mp4");+ return artifactResult(artifact, canonicalize({+ provider: env.label,+ operation: "transform",+ operations: need.program.operations.length,+ frameCount: plan.frameCount,+ sampleFrames: plan.sampleFrames,+ }));+ } finally {+ await rm(work, { recursive: true, force: true }).catch(() => {});+ }+}++/** Generic model-reference audio: no Narrative, SpeechBasis or alignment claim is introduced. */+export async function executeExtractAudio(+ env: MediaExecutionEnvironment,+ constraints: CanonicalValue,+): Promise<MediaOperationResult> {+ const need = extractAudioNeed(constraints);+ const work = await mkdtemp(join(tmpdir(), "svml-media-extract-audio-"));+ try {+ const input = join(work, "source.bin");+ const output = join(work, "audio.wav");+ await stageArtifact(env, need.source, input);+ await runProcess({+ executable: env.ffmpegPath,+ argv: [+ "-y", "-i", input, "-map", `0:${need.streamIndex}`, "-vn",+ "-af", "asetpts=PTS-STARTPTS,aresample=48000:async=0:first_pts=0,aformat=sample_rates=48000:channel_layouts=stereo",+ "-c:a", "pcm_s16le", "-ar", "48000", "-ac", "2", output,+ ],+ timeoutMs: env.processTimeoutMs,+ maxStdoutBytes: 64 * 1024,+ ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),+ });+ const bytes = await readFile(output);+ const artifact = await env.artifacts.put(bytes, "audio/wav");+ await assertCanonicalWav({+ path: output,+ source: artifact,+ ffprobePath: env.ffprobePath,+ timeoutMs: env.processTimeoutMs,+ maxProbeOutputBytes: env.maxProbeOutputBytes,+ ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),+ });+ return artifactResult(artifact, canonicalize({+ provider: env.label,+ operation: "extract-audio",+ sourceStream: need.streamIndex,+ }));+ } finally {+ await rm(work, { recursive: true, force: true }).catch(() => {});+ }+}++export async function executeExtractFrame(+ env: MediaExecutionEnvironment,+ constraints: CanonicalValue,+): Promise<MediaOperationResult> {+ const need = extractFrameNeed(constraints);+ const work = await mkdtemp(join(tmpdir(), "svml-media-extract-frame-"));+ try {+ const input = join(work, "source.bin");+ const output = join(work, "frame.png");+ await stageArtifact(env, need.source, input);+ const selection = need.at.kind === "first"+ ? "eq(n\\,0)"+ : need.at.kind === "last"+ ? `eq(n\\,${need.sourceFrameCount - 1})`+ : need.at.kind === "frame"+ ? `eq(n\\,${need.at.index})`+ : `gte(t\\,${decimal(need.at.seconds)})`;+ await runProcess({+ executable: env.ffmpegPath,+ argv: [+ "-y", "-i", input, "-map", `0:${need.streamIndex}`, "-an",+ "-vf", `setpts=PTS-STARTPTS,select=${selection}`,+ "-frames:v", "1", "-fps_mode", "vfr", "-c:v", "png", output,+ ],+ timeoutMs: env.processTimeoutMs,+ maxStdoutBytes: 64 * 1024,+ ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),+ });+ const bytes = await readFile(output);+ assert(bytes.byteLength > 0, "Frame extraction produced no image");+ const inspected = await outputInspection({+ path: output,+ mediaType: "image/png",+ ffprobePath: env.ffprobePath,+ timeoutMs: env.processTimeoutMs,+ maxProbeOutputBytes: env.maxProbeOutputBytes,+ ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),+ });+ const images = inspected.streams.filter((item): item is MediaVideoStream => item.kind === "video");+ assert(images.length === 1 && images[0]!.decodedUnitCount === 1,+ "Frame extraction output must decode to exactly one image");+ const artifact = await env.artifacts.put(bytes, "image/png");+ return artifactResult(artifact, canonicalize({+ provider: env.label,+ operation: "extract-frame",+ sourceStream: need.streamIndex,+ at: need.at,+ }));+ } finally {+ await rm(work, { recursive: true, force: true }).catch(() => {});+ }+}+export async function executeProjectSpeechEvidenceAudio(env: MediaExecutionEnvironment,constraints: CanonicalValue,@@ -787,7 +1066,7 @@ export async function executeProjectSpeechEvidenceAudio(},});assertSpeechEvidenceAudioIdentity(evidence);- return result(canonicalize(evidence), canonicalize({+ return inlineResult(canonicalize(evidence), canonicalize({provider: env.label,operation: "project-speech-evidence-audio",sourceSampleFrames: need.sourceSampleFrames,@@ -885,7 +1164,7 @@ export async function executeRenderTimelineAudio(sampleFrames: plan.sampleFrames,loudness: "planned",});- return result(canonicalize(value), canonicalize({+ return inlineResult(canonicalize(value), canonicalize({provider: env.label,operation: "render-audio",clips: plan.clips.length,@@ -984,7 +1263,7 @@ export async function executeMuxProgramMedia(presentationSampleFrames: need.audio.sampleFrames,artifact,});- return result(canonicalize(value), canonicalize({+ return inlineResult(canonicalize(value), canonicalize({provider: env.label,operation: "mux",videoStream: finalVideo[0]!.index,diff --git a/packages/media-execution/src/index.ts b/packages/media-execution/src/index.tsindex 9636bbee..b9e2b58c 100644--- a/packages/media-execution/src/index.ts+++ b/packages/media-execution/src/index.ts@@ -1,9 +1,12 @@export {executeInspectMedia,+ executeExtractAudio,+ executeExtractFrame,executeMuxProgramMedia,executeNormalizeMedia,executeProjectSpeechEvidenceAudio,executeRenderTimelineAudio,+ executeTransformMedia,mediaNeedHasContract,mediaOperationContracts,} from "./execute.js";diff --git a/packages/media-pipeline/README.md b/packages/media-pipeline/README.mdindex 985c35d9..257b05b8 100644--- a/packages/media-pipeline/README.md+++ b/packages/media-pipeline/README.md@@ -25,6 +25,24 @@ The current normalization profile uses one source presentation origin, a requestrate, and a 48 kHz stereo PCM render stem. It preserves input level: loudness/mastering remains aseparate author policy.+Three ordinary author operations reuse that same inspection/execution boundary:++```xml+<media:Transform id="prepared" source={shot.video}+ video="primary-moving" audio="default" span-authority="video" frame-rate="30">+ <media:Trim tail="0.25s"/>+ <media:Retime rate="1.05" pitch="preserve"/>+</media:Transform>++<media:ExtractAudio id="voice-reference" source={prepared.video} audio="default"/>+<media:ExtractFrame id="continuity" source={prepared.video} video="primary-moving" at="last"/>+```++Every result is an ordinary `BlobArtifact`. Audio extraction emits a deterministic 48 kHz stereo PCM+WAV but makes no `SpeechBasis`, speaker or alignment claim; it can therefore feed a later model+reference port directly. Frame extraction supports `first`, `last`, `frame:<index>` and+`time:<seconds>`. Transform operations are ordered author meaning and never an arbitrary FFmpeg string.+The same package also owns two provider-neutral finalization plans/capabilities:```textdiff --git a/packages/media-pipeline/src/activation.ts b/packages/media-pipeline/src/activation.tsindex 7bfe1cb0..0d6f9255 100644--- a/packages/media-pipeline/src/activation.ts+++ b/packages/media-pipeline/src/activation.ts@@ -1,6 +1,10 @@import { createMarkupSurfaceHostFacet } from "@narratage/markup";import {decodeSynchronizedMediaSurface,+ decodeExtractAudioSurface,+ decodeExtractFrameSurface,+ decodeTransformMediaSurface,+ mediaOperationSurfaceImplementationDigests,mediaPipelineComponent,mediaPipelineManifest,mediaPipelineModuleRef,@@ -18,6 +22,24 @@ export const svmlPackage = {mode: "structured",implementationDigest: synchronizedMediaSurfaceImplementationDigest,handler: decodeSynchronizedMediaSurface,+ }), createMarkupSurfaceHostFacet({+ module: mediaPipelineModuleRef,+ surface: "transform-media",+ mode: "structured",+ implementationDigest: mediaOperationSurfaceImplementationDigests.transform,+ handler: decodeTransformMediaSurface,+ }), createMarkupSurfaceHostFacet({+ module: mediaPipelineModuleRef,+ surface: "extract-audio",+ mode: "structured",+ implementationDigest: mediaOperationSurfaceImplementationDigests.extractAudio,+ handler: decodeExtractAudioSurface,+ }), createMarkupSurfaceHostFacet({+ module: mediaPipelineModuleRef,+ surface: "extract-frame",+ mode: "structured",+ implementationDigest: mediaOperationSurfaceImplementationDigests.extractFrame,+ handler: decodeExtractFrameSurface,})],};export default svmlPackage;diff --git a/packages/media-pipeline/src/component.ts b/packages/media-pipeline/src/component.tsindex 4ebd8be4..859ffc94 100644--- a/packages/media-pipeline/src/component.ts+++ b/packages/media-pipeline/src/component.ts@@ -1,6 +1,6 @@import { mediaComponent } from "@narratage/media";import type { ComponentPackage } from "@narratage/component-kit";-import { verifyMediaInspection, verifyMediaStreamSelection, verifyMuxedMedia, verifyRenderedVisual, verifyTimelineAudio } from "@narratage/media";+import { verifyMediaInspection, verifyMediaStreamSelection, verifyMuxedMedia, verifyRenderedVisual, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";import type { MediaInspection, MediaStreamSelection, MuxedMedia, RenderedVisual, TimelineAudio } from "@narratage/media";import { programSpaceSampleFrames } from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";@@ -23,13 +23,25 @@ import {selectMediaStreams,verifyMediaSelectionRequest,} from "./selection.js";+import {+ selectAudioStream,+ selectVideoStream,+ verifyAudioExtractionRequest,+ verifyFrameExtractionRequest,+ verifyMediaTransformProgram,+} from "./operations.js";import type {+ AudioExtractionRequest,+ ExtractAudioNeed,+ ExtractFrameNeed,+ FrameExtractionRequest,InspectMediaNeed,MuxMediaNeed,MediaSelectionRequest,NormalizeMediaNeed,ProjectSpeechEvidenceAudioNeed,RenderAudioNeed,+ TransformMediaNeed,} from "./types.js";function inline(value: StoredValue, subject: string): CanonicalValue {@@ -59,6 +71,27 @@ export const mediaPipelineComponent = {verifyAudioProgramPlan(inline(value, "AudioProgramPlan"));},},+ {+ type: mediaPipelineTypes.transformProgram,+ implementationDigest: mediaPipelineImplementationDigests.transformProgramValidator,+ handler: ({ value }) => {+ verifyMediaTransformProgram(inline(value, "MediaTransformProgram"));+ },+ },+ {+ type: mediaPipelineTypes.audioExtractionRequest,+ implementationDigest: mediaPipelineImplementationDigests.audioExtractionValidator,+ handler: ({ value }) => {+ verifyAudioExtractionRequest(inline(value, "AudioExtractionRequest"));+ },+ },+ {+ type: mediaPipelineTypes.frameExtractionRequest,+ implementationDigest: mediaPipelineImplementationDigests.frameExtractionValidator,+ handler: ({ value }) => {+ verifyFrameExtractionRequest(inline(value, "FrameExtractionRequest"));+ },+ },],producers: [{@@ -108,6 +141,66 @@ export const mediaPipelineComponent = {return { outputs: {}, needs: { media: canonicalize(need) } };},},+ {+ producer: mediaPipelineProducers.transform,+ implementationDigest: mediaPipelineImplementationDigests.transform,+ handler: ({ inputs }) => {+ const media = inline(inputs.media!.value, "SynchronizedMedia");+ const program = inline(inputs.program!.value, "MediaTransformProgram");+ verifySynchronizedMedia(media);+ verifyMediaTransformProgram(program);+ if (media.visual === undefined) throw new Error("Media transform requires a visual stream");+ const need: TransformMediaNeed = {+ contract: "svml.transform-media-request@1",+ media,+ program,+ };+ return { outputs: {}, needs: { video: canonicalize(need) } };+ },+ },+ {+ producer: mediaPipelineProducers.extractAudio,+ implementationDigest: mediaPipelineImplementationDigests.extractAudio,+ handler: ({ inputs }) => {+ const source = blob(inputs.source!.value, "Audio extraction source");+ const inspection = inline(inputs.inspection!.value, "MediaInspection");+ const request = inline(inputs.request!.value, "AudioExtractionRequest") as unknown as AudioExtractionRequest;+ verifyMediaInspection(inspection);+ verifyAudioExtractionRequest(request);+ const selected = selectAudioStream(inspection, request.audio);+ const need: ExtractAudioNeed = {+ contract: "svml.extract-audio-request@1",+ source,+ streamIndex: selected.index,+ output: request.output,+ };+ return { outputs: {}, needs: { audio: canonicalize(need) } };+ },+ },+ {+ producer: mediaPipelineProducers.extractFrame,+ implementationDigest: mediaPipelineImplementationDigests.extractFrame,+ handler: ({ inputs }) => {+ const source = blob(inputs.source!.value, "Frame extraction source");+ const inspection = inline(inputs.inspection!.value, "MediaInspection");+ const request = inline(inputs.request!.value, "FrameExtractionRequest") as unknown as FrameExtractionRequest;+ verifyMediaInspection(inspection);+ verifyFrameExtractionRequest(request);+ const selected = selectVideoStream(inspection, request.video);+ if (request.at.kind === "frame" && request.at.index >= selected.decodedUnitCount) {+ throw new Error(`Frame ${request.at.index} is outside the selected stream (${selected.decodedUnitCount} frames)`);+ }+ const need: ExtractFrameNeed = {+ contract: "svml.extract-frame-request@1",+ source,+ streamIndex: selected.index,+ sourceFrameCount: selected.decodedUnitCount,+ at: request.at,+ output: request.output,+ };+ return { outputs: {}, needs: { image: canonicalize(need) } };+ },+ },{producer: mediaPipelineProducers.projectSpeechEvidenceAudio,implementationDigest: mediaPipelineImplementationDigests.projectSpeechEvidenceAudio,@@ -186,19 +279,8 @@ export const mediaPipelineComponent = {handler: ({ inputs }) => {const media = inline(inputs.media!.value, "MuxedMedia");verifyMuxedMedia(media);- const durationSec = media.frameCount * media.frameRate.denominator / media.frameRate.numerator;return {- outputs: {- video: {- kind: "inline",- value: canonicalize({- digest: media.artifact.digest,- size: media.artifact.size,- mediaType: media.artifact.mediaType,- durationSec,- }),- },- },+ outputs: { video: media.artifact },needs: {},};},diff --git a/packages/media-pipeline/src/fragment.ts b/packages/media-pipeline/src/fragment.tsindex d1a8bbf0..723a1235 100644--- a/packages/media-pipeline/src/fragment.ts+++ b/packages/media-pipeline/src/fragment.ts@@ -49,3 +49,108 @@ export const synchronizedMediaFragment = sealGraphFragment({fidelity: "exact",}],});++export const transformMediaFragment = sealGraphFragment({+ name: "@narratage/media-pipeline/transform-media@1",+ inputs: [+ { name: "source", type: artifactTypes.blob },+ { name: "selection", type: mediaPipelineTypes.selectionRequest },+ { name: "program", type: mediaPipelineTypes.transformProgram },+ ],+ operations: [+ {+ id: "inspect",+ producer: mediaPipelineProducers.inspect,+ inputs: { source: input("source") },+ result: { kind: "need", name: "inspection", accepts: "exact" },+ },+ {+ id: "select",+ producer: mediaPipelineProducers.select,+ inputs: { inspection: operation("inspect"), request: input("selection") },+ result: { kind: "output", name: "selection" },+ },+ {+ id: "normalize",+ producer: mediaPipelineProducers.normalize,+ inputs: {+ source: input("source"),+ inspection: operation("inspect"),+ selection: operation("select"),+ request: input("selection"),+ },+ result: { kind: "need", name: "media", accepts: "exact" },+ },+ {+ id: "transform",+ producer: mediaPipelineProducers.transform,+ inputs: { media: operation("normalize"), program: input("program") },+ result: { kind: "need", name: "video", accepts: "exact" },+ },+ ],+ exports: [{+ name: "video",+ type: artifactTypes.blob,+ root: operation("transform"),+ semanticInputs: ["program", "selection", "source"],+ fidelity: "exact",+ }],+});++export const extractAudioFragment = sealGraphFragment({+ name: "@narratage/media-pipeline/extract-audio@1",+ inputs: [+ { name: "source", type: artifactTypes.blob },+ { name: "request", type: mediaPipelineTypes.audioExtractionRequest },+ ],+ operations: [+ {+ id: "inspect",+ producer: mediaPipelineProducers.inspect,+ inputs: { source: input("source") },+ result: { kind: "need", name: "inspection", accepts: "exact" },+ },+ {+ id: "extract",+ producer: mediaPipelineProducers.extractAudio,+ inputs: { source: input("source"), inspection: operation("inspect"), request: input("request") },+ result: { kind: "need", name: "audio", accepts: "exact" },+ },+ ],+ exports: [{+ name: "audio",+ type: artifactTypes.blob,+ root: operation("extract"),+ semanticInputs: ["request", "source"],+ fidelity: "exact",+ }],+});++export const extractFrameFragment = sealGraphFragment({+ name: "@narratage/media-pipeline/extract-frame@1",+ inputs: [+ { name: "source", type: artifactTypes.blob },+ { name: "request", type: mediaPipelineTypes.frameExtractionRequest },+ ],+ operations: [+ {+ id: "inspect",+ producer: mediaPipelineProducers.inspect,+ inputs: { source: input("source") },+ result: { kind: "need", name: "inspection", accepts: "exact" },+ },+ {+ id: "extract",+ producer: mediaPipelineProducers.extractFrame,+ inputs: { source: input("source"), inspection: operation("inspect"), request: input("request") },+ result: { kind: "need", name: "image", accepts: "exact" },+ },+ ],+ exports: [{+ name: "image",+ type: artifactTypes.blob,+ root: operation("extract"),+ semanticInputs: ["request", "source"],+ fidelity: "exact",+ }],+});diff --git a/packages/media-pipeline/src/index.ts b/packages/media-pipeline/src/index.tsindex 28079f08..7f613e16 100644--- a/packages/media-pipeline/src/index.ts+++ b/packages/media-pipeline/src/index.ts@@ -2,7 +2,12 @@ export {mediaPipelineComponent,mediaPipelineComponents,} from "./component.js";-export { synchronizedMediaFragment } from "./fragment.js";+export {+ extractAudioFragment,+ extractFrameFragment,+ synchronizedMediaFragment,+ transformMediaFragment,+} from "./fragment.js";export {compileAudioProgramPlan,sealAudioProgramPlan,@@ -19,6 +24,10 @@ export {audioProgramPlanSchema,mediaSelectionRequestSchema,synchronizedMediaSurfaceImplementationDigest,+ mediaOperationSurfaceImplementationDigests,+ audioExtractionRequestSchema,+ frameExtractionRequestSchema,+ mediaTransformProgramSchema,} from "./manifest.js";export {sealMediaSelectionRequest,@@ -26,4 +35,19 @@ export {verifyMediaSelectionRequest,} from "./selection.js";export { decodeSynchronizedMediaSurface } from "./surface.js";+export {+ decodeExtractAudioSurface,+ decodeExtractFrameSurface,+ decodeTransformMediaSurface,+} from "./surface.js";+export {+ sealAudioExtractionRequest,+ sealFrameExtractionRequest,+ sealMediaTransformProgram,+ selectAudioStream,+ selectVideoStream,+ verifyAudioExtractionRequest,+ verifyFrameExtractionRequest,+ verifyMediaTransformProgram,+} from "./operations.js";export type * from "./types.js";diff --git a/packages/media-pipeline/src/manifest.ts b/packages/media-pipeline/src/manifest.tsindex 6c1ccb26..3a0ee81e 100644--- a/packages/media-pipeline/src/manifest.ts+++ b/packages/media-pipeline/src/manifest.ts@@ -17,10 +17,16 @@ export const mediaPipelineModuleRef = { name: "@narratage/media-pipeline", versiexport const mediaPipelineTypes = {selectionRequest: { module: mediaPipelineModuleRef, name: "MediaSelectionRequest" },audioProgramPlan: { module: mediaPipelineModuleRef, name: "AudioProgramPlan" },+ transformProgram: { module: mediaPipelineModuleRef, name: "MediaTransformProgram" },+ audioExtractionRequest: { module: mediaPipelineModuleRef, name: "AudioExtractionRequest" },+ frameExtractionRequest: { module: mediaPipelineModuleRef, name: "FrameExtractionRequest" },} satisfies Record<string, TypeRef>;export const mediaPipelineCapabilities = {inspect: { module: mediaPipelineModuleRef, name: "inspect-media" },normalize: { module: mediaPipelineModuleRef, name: "normalize-media" },+ transform: { module: mediaPipelineModuleRef, name: "transform-media" },+ extractAudio: { module: mediaPipelineModuleRef, name: "extract-media-audio" },+ extractFrame: { module: mediaPipelineModuleRef, name: "extract-media-frame" },projectSpeechEvidenceAudio: { module: mediaPipelineModuleRef, name: "project-speech-evidence-audio" },renderAudio: { module: mediaPipelineModuleRef, name: "render-timeline-audio" },mux: { module: mediaPipelineModuleRef, name: "mux-program-media" },@@ -29,6 +35,9 @@ export const mediaPipelineProducers = {inspect: { module: mediaPipelineModuleRef, name: "request-media-inspection" },select: { module: mediaPipelineModuleRef, name: "select-media-streams" },normalize: { module: mediaPipelineModuleRef, name: "request-media-normalization" },+ transform: { module: mediaPipelineModuleRef, name: "request-media-transform" },+ extractAudio: { module: mediaPipelineModuleRef, name: "request-audio-extraction" },+ extractFrame: { module: mediaPipelineModuleRef, name: "request-frame-extraction" },projectSpeechEvidenceAudio: { module: mediaPipelineModuleRef, name: "request-speech-evidence-audio" },planAudio: { module: mediaPipelineModuleRef, name: "compile-audio-program" },renderAudio: { module: mediaPipelineModuleRef, name: "request-audio-render" },@@ -39,6 +48,9 @@ export const mediaPipelineImplementationDigests = {inspect: digestOf("@narratage/media-pipeline/request-media-inspection@1"),select: digestOf("@narratage/media-pipeline/select-media-streams@1"),normalize: digestOf("@narratage/media-pipeline/request-media-normalization@1"),+ transform: digestOf("@narratage/media-pipeline/request-media-transform@1"),+ extractAudio: digestOf("@narratage/media-pipeline/request-audio-extraction@1"),+ extractFrame: digestOf("@narratage/media-pipeline/request-frame-extraction@1"),projectSpeechEvidenceAudio: digestOf("@narratage/media-pipeline/request-speech-evidence-audio@1"),requestValidator: digestOf("@narratage/media-pipeline/validate-selection-request@1"),planAudio: digestOf("@narratage/media-pipeline/compile-audio-program@1"),@@ -46,8 +58,16 @@ export const mediaPipelineImplementationDigests = {mux: digestOf("@narratage/media-pipeline/request-media-mux@1"),projectMuxed: digestOf("@narratage/media-pipeline/project-muxed-media@1"),audioPlanValidator: digestOf("@narratage/media-pipeline/validate-audio-program-plan@1"),+ transformProgramValidator: digestOf("@narratage/media-pipeline/validate-media-transform-program@1"),+ audioExtractionValidator: digestOf("@narratage/media-pipeline/validate-audio-extraction-request@1"),+ frameExtractionValidator: digestOf("@narratage/media-pipeline/validate-frame-extraction-request@1"),} as const;export const synchronizedMediaSurfaceImplementationDigest = digestOf("@narratage/media-pipeline/synchronized-media-surface@1");+export const mediaOperationSurfaceImplementationDigests = {+ transform: digestOf("@narratage/media-pipeline/transform-media-surface@1"),+ extractAudio: digestOf("@narratage/media-pipeline/extract-audio-surface@1"),+ extractFrame: digestOf("@narratage/media-pipeline/extract-frame-surface@1"),+} as const;const integer = { kind: "number", integer: true, minimum: 0 } as const;const mode = (name: string): ValueSchema => ({@@ -78,6 +98,81 @@ export const mediaSelectionRequestSchema: ValueSchema = {},};+const videoSelectorSchema: ValueSchema = {+ kind: "oneOf",+ variants: [mode("primary-moving"), streamMode],+};+const audioSelectorSchema: ValueSchema = {+ kind: "oneOf",+ variants: [mode("default"), streamMode],+};+const nonNegativeNumber = { kind: "number", minimum: 0 } as const;+const trimOperationSchema: ValueSchema = {+ kind: "object",+ fields: {+ kind: { schema: { kind: "literal", value: "trim" } },+ startSec: { schema: nonNegativeNumber, optional: true },+ endSec: { schema: nonNegativeNumber, optional: true },+ tailSec: { schema: nonNegativeNumber, optional: true },+ },+};+const retimeOperationSchema: ValueSchema = {+ kind: "object",+ fields: {+ kind: { schema: { kind: "literal", value: "retime" } },+ rate: { schema: { kind: "number", minimum: 0.000001, maximum: 100 } },+ pitch: { schema: { kind: "literal", value: "preserve" } },+ },+};+export const mediaTransformProgramSchema: ValueSchema = {+ kind: "object",+ fields: {+ contract: { schema: { kind: "literal", value: "svml.media-transform-program@1" } },+ operations: { schema: { kind: "array", minItems: 1, items: {+ kind: "oneOf", variants: [trimOperationSchema, retimeOperationSchema],+ } } },+ },+};+export const audioExtractionRequestSchema: ValueSchema = {+ kind: "object",+ fields: {+ contract: { schema: { kind: "literal", value: "svml.audio-extraction-request@1" } },+ audio: { schema: audioSelectorSchema },+ output: { schema: {+ kind: "object",+ fields: {+ container: { schema: { kind: "literal", value: "wav" } },+ codec: { schema: { kind: "literal", value: "pcm_s16le" } },+ sampleRate: { schema: { kind: "literal", value: 48_000 } },+ channels: { schema: { kind: "literal", value: 2 } },+ },+ } },+ },+};+export const frameExtractionRequestSchema: ValueSchema = {+ kind: "object",+ fields: {+ contract: { schema: { kind: "literal", value: "svml.frame-extraction-request@1" } },+ video: { schema: videoSelectorSchema },+ at: { schema: { kind: "oneOf", variants: [+ { kind: "object", fields: { kind: { schema: { kind: "literal", value: "first" } } } },+ { kind: "object", fields: { kind: { schema: { kind: "literal", value: "last" } } } },+ { kind: "object", fields: {+ kind: { schema: { kind: "literal", value: "frame" } },+ index: { schema: integer },+ } },+ { kind: "object", fields: {+ kind: { schema: { kind: "literal", value: "time" } },+ seconds: { schema: nonNegativeNumber },+ } },+ ] } },+ output: { schema: {+ kind: "object",+ fields: { format: { schema: { kind: "literal", value: "png" } } },+ } },+ },+};+const blobRefSchema: ValueSchema = {kind: "object",fields: {@@ -167,19 +262,75 @@ export const mediaPipelineManifest: ModuleManifest = {},},},+ {+ name: mediaPipelineTypes.transformProgram.name,+ schema: mediaTransformProgramSchema,+ validator: {+ abi: "svml.type-validator@1",+ implementation: {+ kind: "registered",+ locator: "@narratage/media-pipeline/validate-media-transform-program",+ digest: mediaPipelineImplementationDigests.transformProgramValidator,+ },+ },+ },+ {+ name: mediaPipelineTypes.audioExtractionRequest.name,+ schema: audioExtractionRequestSchema,+ validator: {+ abi: "svml.type-validator@1",+ implementation: {+ kind: "registered",+ locator: "@narratage/media-pipeline/validate-audio-extraction-request",+ digest: mediaPipelineImplementationDigests.audioExtractionValidator,+ },+ },+ },+ {+ name: mediaPipelineTypes.frameExtractionRequest.name,+ schema: frameExtractionRequestSchema,+ validator: {+ abi: "svml.type-validator@1",+ implementation: {+ kind: "registered",+ locator: "@narratage/media-pipeline/validate-frame-extraction-request",+ digest: mediaPipelineImplementationDigests.frameExtractionValidator,+ },+ },+ },],capabilities: [{ name: mediaPipelineCapabilities.inspect.name, returns: mediaTypes.inspection },{ name: mediaPipelineCapabilities.normalize.name, returns: mediaTypes.synchronized },+ { name: mediaPipelineCapabilities.transform.name, returns: artifactTypes.blob },+ { name: mediaPipelineCapabilities.extractAudio.name, returns: artifactTypes.blob },+ { name: mediaPipelineCapabilities.extractFrame.name, returns: artifactTypes.blob },{ name: mediaPipelineCapabilities.projectSpeechEvidenceAudio.name, returns: speechTypes.evidenceAudio },{ name: mediaPipelineCapabilities.renderAudio.name, returns: mediaTypes.timelineAudio },{ name: mediaPipelineCapabilities.mux.name, returns: mediaTypes.muxed },],- surfaces: [{- name: "synchronized-media", tag: "Normalize", mode: "structured",- outputs: [mediaPipelineTypes.selectionRequest, mediaTypes.synchronized],- implementation: { kind: "trusted-frontend-surface", locator: "@narratage/media-pipeline/synchronized-media-surface", digest: synchronizedMediaSurfaceImplementationDigest },- }],+ surfaces: [+ {+ name: "synchronized-media", tag: "Normalize", mode: "structured",+ outputs: [mediaPipelineTypes.selectionRequest, mediaTypes.synchronized],+ implementation: { kind: "trusted-frontend-surface", locator: "@narratage/media-pipeline/synchronized-media-surface", digest: synchronizedMediaSurfaceImplementationDigest },+ },+ {+ name: "transform-media", tag: "Transform", mode: "structured",+ outputs: [mediaPipelineTypes.selectionRequest, mediaPipelineTypes.transformProgram, artifactTypes.blob],+ implementation: { kind: "trusted-frontend-surface", locator: "@narratage/media-pipeline/transform-media-surface", digest: mediaOperationSurfaceImplementationDigests.transform },+ },+ {+ name: "extract-audio", tag: "ExtractAudio", mode: "structured",+ outputs: [mediaPipelineTypes.audioExtractionRequest, artifactTypes.blob],+ implementation: { kind: "trusted-frontend-surface", locator: "@narratage/media-pipeline/extract-audio-surface", digest: mediaOperationSurfaceImplementationDigests.extractAudio },+ },+ {+ name: "extract-frame", tag: "ExtractFrame", mode: "structured",+ outputs: [mediaPipelineTypes.frameExtractionRequest, artifactTypes.blob],+ implementation: { kind: "trusted-frontend-surface", locator: "@narratage/media-pipeline/extract-frame-surface", digest: mediaOperationSurfaceImplementationDigests.extractFrame },+ },+ ],producers: [{name: mediaPipelineProducers.inspect.name,@@ -233,6 +384,62 @@ export const mediaPipelineManifest: ModuleManifest = {digest: mediaPipelineImplementationDigests.normalize,},},+ {+ name: mediaPipelineProducers.transform.name,+ inputs: [+ { name: "media", type: mediaTypes.synchronized },+ { name: "program", type: mediaPipelineTypes.transformProgram },+ ],+ outputs: [],+ needs: [{+ name: "video",+ capability: mediaPipelineCapabilities.transform,+ returns: artifactTypes.blob,+ }],+ implementation: {+ kind: "registered",+ locator: "@narratage/media-pipeline/request-media-transform",+ digest: mediaPipelineImplementationDigests.transform,+ },+ },+ {+ name: mediaPipelineProducers.extractAudio.name,+ inputs: [+ { name: "source", type: artifactTypes.blob },+ { name: "inspection", type: mediaTypes.inspection },+ { name: "request", type: mediaPipelineTypes.audioExtractionRequest },+ ],+ outputs: [],+ needs: [{+ name: "audio",+ capability: mediaPipelineCapabilities.extractAudio,+ returns: artifactTypes.blob,+ }],+ implementation: {+ kind: "registered",+ locator: "@narratage/media-pipeline/request-audio-extraction",+ digest: mediaPipelineImplementationDigests.extractAudio,+ },+ },+ {+ name: mediaPipelineProducers.extractFrame.name,+ inputs: [+ { name: "source", type: artifactTypes.blob },+ { name: "inspection", type: mediaTypes.inspection },+ { name: "request", type: mediaPipelineTypes.frameExtractionRequest },+ ],+ outputs: [],+ needs: [{+ name: "image",+ capability: mediaPipelineCapabilities.extractFrame,+ returns: artifactTypes.blob,+ }],+ implementation: {+ kind: "registered",+ locator: "@narratage/media-pipeline/request-frame-extraction",+ digest: mediaPipelineImplementationDigests.extractFrame,+ },+ },{name: mediaPipelineProducers.projectSpeechEvidenceAudio.name,inputs: [{ name: "audio", type: speechTypes.audioBasis }],@@ -298,7 +505,7 @@ export const mediaPipelineManifest: ModuleManifest = {{name: mediaPipelineProducers.projectMuxed.name,inputs: [{ name: "media", type: mediaTypes.muxed }],- outputs: [{ name: "video", type: mediaTypes.artifact }],+ outputs: [{ name: "video", type: artifactTypes.blob }],needs: [],implementation: {kind: "registered",diff --git a/packages/media-pipeline/src/operations.ts b/packages/media-pipeline/src/operations.tsnew file mode 100644index 00000000..a4116a60--- /dev/null+++ b/packages/media-pipeline/src/operations.ts@@ -0,0 +1,156 @@+import { verifyMediaInspection } from "@narratage/media";+import type { MediaAudioStream, MediaInspection, MediaVideoStream } from "@narratage/media";+import { canonicalize } from "@narratage/protocol";++import type {+ AudioExtractionRequest,+ FrameExtractionRequest,+ MediaAudioSelector,+ MediaTransformProgram,+ MediaVideoSelector,+} from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function finiteNonNegative(value: unknown, subject: string): number {+ assert(typeof value === "number" && Number.isFinite(value) && value >= 0,+ `${subject} must be a finite non-negative number`);+ return value;+}++function streamIndex(value: unknown, subject: string): number {+ assert(Number.isSafeInteger(value) && (value as number) >= 0, `${subject} must be a non-negative integer`);+ return value as number;+}++function object(value: unknown, subject: string): Record<string, unknown> {+ assert(value !== null && typeof value === "object" && !Array.isArray(value), `${subject} must be an object`);+ return value as Record<string, unknown>;+}++export function verifyMediaTransformProgram(value: unknown): asserts value is MediaTransformProgram {+ const program = object(value, "MediaTransformProgram");+ assert(program.contract === "svml.media-transform-program@1", "MediaTransformProgram contract is invalid");+ assert(Array.isArray(program.operations) && program.operations.length > 0,+ "MediaTransformProgram requires at least one operation");+ for (const [index, raw] of program.operations.entries()) {+ const operation = object(raw, `MediaTransformProgram operation ${index}`);+ if (operation.kind === "trim") {+ const start = operation.startSec;+ const end = operation.endSec;+ const tail = operation.tailSec;+ assert(start !== undefined || end !== undefined || tail !== undefined,+ `MediaTransformProgram trim ${index} must specify startSec, endSec or tailSec`);+ if (start !== undefined) finiteNonNegative(start, `MediaTransformProgram trim ${index}.startSec`);+ if (end !== undefined) finiteNonNegative(end, `MediaTransformProgram trim ${index}.endSec`);+ if (tail !== undefined) finiteNonNegative(tail, `MediaTransformProgram trim ${index}.tailSec`);+ assert(end === undefined || tail === undefined,+ `MediaTransformProgram trim ${index} cannot specify both endSec and tailSec`);+ if (start !== undefined && end !== undefined) {+ assert((end as number) > (start as number),+ `MediaTransformProgram trim ${index}.endSec must be after startSec`);+ }+ continue;+ }+ assert(operation.kind === "retime", `MediaTransformProgram operation ${index} kind is invalid`);+ assert(typeof operation.rate === "number" && Number.isFinite(operation.rate) && operation.rate > 0+ && operation.rate <= 100,+ `MediaTransformProgram retime ${index}.rate must be in (0, 100]`);+ assert(operation.pitch === "preserve", `MediaTransformProgram retime ${index}.pitch must be preserve`);+ }+}++export function sealMediaTransformProgram(value: MediaTransformProgram): MediaTransformProgram {+ const result = canonicalize(value) as unknown as MediaTransformProgram;+ verifyMediaTransformProgram(result);+ return result;+}++function verifyVideoSelector(value: unknown, subject: string): asserts value is MediaVideoSelector {+ const selector = object(value, subject);+ assert(selector.mode === "primary-moving" || selector.mode === "stream-index", `${subject}.mode is invalid`);+ if (selector.mode === "stream-index") streamIndex(selector.streamIndex, `${subject}.streamIndex`);+}++function verifyAudioSelector(value: unknown, subject: string, allowNone: boolean): asserts value is MediaAudioSelector {+ const selector = object(value, subject);+ assert(selector.mode === "default" || selector.mode === "stream-index" || (allowNone && selector.mode === "none"),+ `${subject}.mode is invalid`);+ if (selector.mode === "stream-index") streamIndex(selector.streamIndex, `${subject}.streamIndex`);+}++export function verifyAudioExtractionRequest(value: unknown): asserts value is AudioExtractionRequest {+ const request = object(value, "AudioExtractionRequest");+ assert(request.contract === "svml.audio-extraction-request@1", "AudioExtractionRequest contract is invalid");+ verifyAudioSelector(request.audio, "AudioExtractionRequest.audio", false);+ const output = object(request.output, "AudioExtractionRequest.output");+ assert(output.container === "wav" && output.codec === "pcm_s16le"+ && output.sampleRate === 48_000 && output.channels === 2,+ "AudioExtractionRequest output profile is unsupported");+}++export function sealAudioExtractionRequest(value: AudioExtractionRequest): AudioExtractionRequest {+ const result = canonicalize(value) as unknown as AudioExtractionRequest;+ verifyAudioExtractionRequest(result);+ return result;+}++export function verifyFrameExtractionRequest(value: unknown): asserts value is FrameExtractionRequest {+ const request = object(value, "FrameExtractionRequest");+ assert(request.contract === "svml.frame-extraction-request@1", "FrameExtractionRequest contract is invalid");+ verifyVideoSelector(request.video, "FrameExtractionRequest.video");+ const at = object(request.at, "FrameExtractionRequest.at");+ assert(at.kind === "first" || at.kind === "last" || at.kind === "frame" || at.kind === "time",+ "FrameExtractionRequest.at.kind is invalid");+ if (at.kind === "frame") streamIndex(at.index, "FrameExtractionRequest.at.index");+ if (at.kind === "time") finiteNonNegative(at.seconds, "FrameExtractionRequest.at.seconds");+ const output = object(request.output, "FrameExtractionRequest.output");+ assert(output.format === "png", "FrameExtractionRequest output format is unsupported");+}++export function sealFrameExtractionRequest(value: FrameExtractionRequest): FrameExtractionRequest {+ const result = canonicalize(value) as unknown as FrameExtractionRequest;+ verifyFrameExtractionRequest(result);+ return result;+}++function uniqueDefault<T extends { readonly disposition: { readonly default: boolean }; readonly index: number }>(+ values: readonly T[],+ subject: string,+): T {+ assert(values.length > 0, `${subject} has no eligible stream`);+ const defaults = values.filter((item) => item.disposition.default);+ if (defaults.length === 1) return defaults[0]!;+ assert(defaults.length === 0, `${subject} has multiple default streams: ${defaults.map((item) => item.index).join(", ")}`);+ assert(values.length === 1, `${subject} is ambiguous: ${values.map((item) => item.index).join(", ")}`);+ return values[0]!;+}++export function selectVideoStream(inspection: MediaInspection, selector: MediaVideoSelector): MediaVideoStream {+ verifyMediaInspection(inspection);+ verifyVideoSelector(selector, "Video selector");+ const moving = inspection.streams.filter((item): item is MediaVideoStream =>+ item.kind === "video" && item.role === "moving" && !item.disposition.attachedPicture+ && item.timingStatus === "admissible" && item.startPts !== undefined && item.endPts !== undefined);+ if (selector.mode === "primary-moving") return uniqueDefault(moving, "Primary moving video");+ const result = moving.find((item) => item.index === selector.streamIndex);+ assert(result !== undefined, `Video stream ${selector.streamIndex} is not an eligible moving stream`);+ return result;+}++export function selectAudioStream(+ inspection: MediaInspection,+ selector: Exclude<MediaAudioSelector, { readonly mode: "none" }>,+): MediaAudioStream {+ verifyMediaInspection(inspection);+ verifyAudioSelector(selector, "Audio selector", false);+ const audio = inspection.streams.filter((item): item is MediaAudioStream =>+ item.kind === "audio" && item.timingStatus === "admissible"+ && item.startPts !== undefined && item.endPts !== undefined);+ if (selector.mode === "default") return uniqueDefault(audio, "Default audio");+ const result = audio.find((item) => item.index === selector.streamIndex);+ assert(result !== undefined, `Audio stream ${selector.streamIndex} is not eligible`);+ return result;+}diff --git a/packages/media-pipeline/src/surface.ts b/packages/media-pipeline/src/surface.tsindex d1c71622..963592ed 100644--- a/packages/media-pipeline/src/surface.ts+++ b/packages/media-pipeline/src/surface.ts@@ -1,10 +1,25 @@import { artifactTypes } from "@narratage/artifact";import type { StructuredElement, StructuredSurfaceHandler, SurfaceResolvedReference, MarkupAttributeValue } from "@narratage/markup";-import { synchronizedMediaFragment } from "./fragment.js";+import {+ extractAudioFragment,+ extractFrameFragment,+ synchronizedMediaFragment,+ transformMediaFragment,+} from "./fragment.js";import { mediaPipelineTypes } from "./manifest.js";+import {+ sealAudioExtractionRequest,+ sealFrameExtractionRequest,+ sealMediaTransformProgram,+} from "./operations.js";import { sealMediaSelectionRequest } from "./selection.js";-import type { MediaSelectionRequest } from "./types.js";+import type {+ AudioExtractionRequest,+ FrameExtractionRequest,+ MediaSelectionRequest,+ MediaTransformOperation,+} from "./types.js";function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;@@ -49,6 +64,92 @@ function frameRate(value: string): { readonly numerator: number; readonly denomireturn { numerator, denominator };}+function localName(value: string): string {+ return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value;+}++function exactAttributes(element: StructuredElement, required: readonly string[], optional: readonly string[] = []): void {+ const expected = new Set([...required, ...optional]);+ const unknown = Object.keys(element.attributes).filter((name) => !expected.has(name));+ if (unknown.length > 0 || required.some((name) => element.attributes[name] === undefined)) {+ throw new Error(`${element.name} requires ${required.join(", ")}`+ + (optional.length === 0 ? "" : `; optional: ${optional.join(", ")}`));+ }+}++function seconds(value: string, subject: string, allowZero = true): number {+ const match = /^(\d+(?:\.\d+)?)(?:s)?$/u.exec(value.trim());+ if (match === null) throw new Error(`${subject} must be seconds such as 0.25s or 2`);+ const result = Number(match[1]);+ if (!Number.isFinite(result) || result < 0 || (!allowZero && result === 0)) {+ throw new Error(`${subject} must be ${allowZero ? "non-negative" : "positive"}`);+ }+ return result;+}++function audioSelector(value: string): AudioExtractionRequest["audio"] {+ if (value === "default") return { mode: "default" };+ const match = /^stream:(\d+)$/u.exec(value);+ if (match === null) throw new Error("ExtractAudio.audio must be default or stream:<index>.");+ return { mode: "stream-index", streamIndex: Number(match[1]) };+}++function videoSelector(value: string): FrameExtractionRequest["video"] {+ if (value === "primary-moving") return { mode: "primary-moving" };+ const match = /^stream:(\d+)$/u.exec(value);+ if (match === null) throw new Error("video must be primary-moving or stream:<index>.");+ return { mode: "stream-index", streamIndex: Number(match[1]) };+}++function empty(element: StructuredElement): void {+ if (element.children.some((child) => child.kind === "element" || child.value.trim().length > 0)) {+ throw new Error(`${element.name} must be empty.`);+ }+}++function transformOperations(element: StructuredElement): readonly MediaTransformOperation[] {+ const result: MediaTransformOperation[] = [];+ for (const child of element.children) {+ if (child.kind === "text") {+ if (child.value.trim().length > 0) throw new Error(`${element.name} accepts only Trim and Retime children.`);+ continue;+ }+ const name = localName(child.name);+ empty(child);+ if (name === "Trim") {+ exactAttributes(child, [], ["start", "end", "tail"]);+ const start = child.attributes.start;+ const end = child.attributes.end;+ const tail = child.attributes.tail;+ if (start === undefined && end === undefined && tail === undefined) {+ throw new Error(`${child.name} requires at least one of start, end or tail.`);+ }+ if (end !== undefined && tail !== undefined) throw new Error(`${child.name} cannot combine end and tail.`);+ for (const [attribute, value] of [["start", start], ["end", end], ["tail", tail]] as const) {+ if (value !== undefined && typeof value !== "string") throw new Error(`${child.name}.${attribute} must be text.`);+ }+ result.push({+ kind: "trim",+ ...(typeof start === "string" ? { startSec: seconds(start, `${child.name}.start`) } : {}),+ ...(typeof end === "string" ? { endSec: seconds(end, `${child.name}.end`, false) } : {}),+ ...(typeof tail === "string" ? { tailSec: seconds(tail, `${child.name}.tail`, false) } : {}),+ });+ continue;+ }+ if (name === "Retime") {+ exactAttributes(child, ["rate", "pitch"]);+ const rate = Number(text(child, "rate"));+ if (!Number.isFinite(rate) || rate <= 0 || rate > 100) throw new Error(`${child.name}.rate must be in (0, 100].`);+ if (text(child, "pitch") !== "preserve") throw new Error(`${child.name}.pitch must be preserve.`);+ result.push({ kind: "retime", rate, pitch: "preserve" });+ continue;+ }+ throw new Error(`${element.name} accepts only Trim and Retime children.`);+ }+ if (result.length === 0) throw new Error(`${element.name} requires at least one Trim or Retime child.`);+ return result;+}+export const decodeSynchronizedMediaSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {const expected = ["audio", "frame-rate", "id", "source", "span-authority", "video"];if (Object.keys(element.attributes).sort().join("\0") !== expected.sort().join("\0")) {@@ -83,3 +184,106 @@ export const decodeSynchronizedMediaSurface: StructuredSurfaceHandler = ({ elemefragments: [synchronizedMediaFragment],};};++export const decodeTransformMediaSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ exactAttributes(element, ["id", "source", "video", "audio", "span-authority", "frame-rate"]);+ const id = text(element, "id");+ const source = ref(element.attributes.source, `${element.name}.source`, resolveReference);+ const video = stream(text(element, "video"), "video");+ if (video.mode === "none") throw new Error(`${element.name}.video cannot be none.`);+ const audio = stream(text(element, "audio"), "audio");+ const spanAuthority = text(element, "span-authority");+ if (spanAuthority !== "video") throw new Error(`${element.name}.span-authority must be video.`);+ const selection = sealMediaSelectionRequest({+ contract: "svml.media-selection-request@1",+ video,+ audio,+ spanAuthority,+ frameRate: frameRate(text(element, "frame-rate")),+ });+ const program = sealMediaTransformProgram({+ contract: "svml.media-transform-program@1",+ operations: transformOperations(element),+ });+ const selectionId = `${id}.selection`;+ const programId = `${id}.program`;+ return {+ records: [+ { id: selectionId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline", value: selection }, range: element.range },+ { id: programId, type: mediaPipelineTypes.transformProgram, value: { kind: "inline", value: program }, range: element.range },+ ],+ components: [{+ id,+ fragment: transformMediaFragment.id,+ inputs: {+ source: source.ref,+ selection: { kind: "record", id: selectionId },+ program: { kind: "record", id: programId },+ },+ outputs: { video: `${id}.video` },+ range: element.range,+ }],+ fragments: [transformMediaFragment],+ };+};++export const decodeExtractAudioSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ exactAttributes(element, ["id", "source", "audio"]);+ empty(element);+ const id = text(element, "id");+ const source = ref(element.attributes.source, `${element.name}.source`, resolveReference);+ const request = sealAudioExtractionRequest({+ contract: "svml.audio-extraction-request@1",+ audio: audioSelector(text(element, "audio")),+ output: { container: "wav", codec: "pcm_s16le", sampleRate: 48_000, channels: 2 },+ });+ const requestId = `${id}.request`;+ return {+ records: [{ id: requestId, type: mediaPipelineTypes.audioExtractionRequest,+ value: { kind: "inline", value: request }, range: element.range }],+ components: [{+ id,+ fragment: extractAudioFragment.id,+ inputs: { source: source.ref, request: { kind: "record", id: requestId } },+ outputs: { audio: `${id}.audio` },+ range: element.range,+ }],+ fragments: [extractAudioFragment],+ };+};++function frameSelector(value: string, subject: string): FrameExtractionRequest["at"] {+ if (value === "first") return { kind: "first" };+ if (value === "last") return { kind: "last" };+ const frame = /^frame:(\d+)$/u.exec(value);+ if (frame !== null) return { kind: "frame", index: Number(frame[1]) };+ const time = /^time:(.+)$/u.exec(value);+ if (time !== null) return { kind: "time", seconds: seconds(time[1]!, subject) };+ throw new Error(`${subject} must be first, last, frame:<index> or time:<seconds>.`);+}++export const decodeExtractFrameSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ exactAttributes(element, ["id", "source", "video", "at"]);+ empty(element);+ const id = text(element, "id");+ const source = ref(element.attributes.source, `${element.name}.source`, resolveReference);+ const request = sealFrameExtractionRequest({+ contract: "svml.frame-extraction-request@1",+ video: videoSelector(text(element, "video")),+ at: frameSelector(text(element, "at"), `${element.name}.at`),+ output: { format: "png" },+ });+ const requestId = `${id}.request`;+ return {+ records: [{ id: requestId, type: mediaPipelineTypes.frameExtractionRequest,+ value: { kind: "inline", value: request }, range: element.range }],+ components: [{+ id,+ fragment: extractFrameFragment.id,+ inputs: { source: source.ref, request: { kind: "record", id: requestId } },+ outputs: { image: `${id}.image` },+ range: element.range,+ }],+ fragments: [extractFrameFragment],+ };+};diff --git a/packages/media-pipeline/src/types.ts b/packages/media-pipeline/src/types.tsindex 186a74ec..52336473 100644--- a/packages/media-pipeline/src/types.ts+++ b/packages/media-pipeline/src/types.ts@@ -21,6 +21,80 @@ export type InspectMediaNeed = {readonly source: BlobRef;};+export type MediaVideoSelector =+ | { readonly mode: "primary-moving" }+ | { readonly mode: "stream-index"; readonly streamIndex: number };++export type MediaAudioSelector =+ | { readonly mode: "default" }+ | { readonly mode: "stream-index"; readonly streamIndex: number }+ | { readonly mode: "none" };++export type MediaTransformOperation =+ | {+ readonly kind: "trim";+ /** Seconds in the operation's current timeline, not necessarily the source timeline. */+ readonly startSec?: number;+ readonly endSec?: number;+ readonly tailSec?: number;+ }+ | {+ readonly kind: "retime";+ /** 2 means twice as fast and therefore half as long. */+ readonly rate: number;+ readonly pitch: "preserve";+ };++/** Ordered, deterministic A/V operations over already synchronized media. */+export type MediaTransformProgram = {+ readonly contract: "svml.media-transform-program@1";+ readonly operations: readonly MediaTransformOperation[];+};++export type AudioExtractionRequest = {+ readonly contract: "svml.audio-extraction-request@1";+ readonly audio: Exclude<MediaAudioSelector, { readonly mode: "none" }>;+ readonly output: {+ readonly container: "wav";+ readonly codec: "pcm_s16le";+ readonly sampleRate: 48_000;+ readonly channels: 2;+ };+};++export type FrameExtractionRequest = {+ readonly contract: "svml.frame-extraction-request@1";+ readonly video: MediaVideoSelector;+ readonly at:+ | { readonly kind: "first" }+ | { readonly kind: "last" }+ | { readonly kind: "frame"; readonly index: number }+ | { readonly kind: "time"; readonly seconds: number };+ readonly output: { readonly format: "png" };+};++export type TransformMediaNeed = {+ readonly contract: "svml.transform-media-request@1";+ readonly media: import("@narratage/media").SynchronizedMedia;+ readonly program: MediaTransformProgram;+};++export type ExtractAudioNeed = {+ readonly contract: "svml.extract-audio-request@1";+ readonly source: BlobRef;+ readonly streamIndex: number;+ readonly output: AudioExtractionRequest["output"];+};++export type ExtractFrameNeed = {+ readonly contract: "svml.extract-frame-request@1";+ readonly source: BlobRef;+ readonly streamIndex: number;+ readonly sourceFrameCount: number;+ readonly at: FrameExtractionRequest["at"];+ readonly output: FrameExtractionRequest["output"];+};+export type NormalizeMediaNeed = {readonly contract: "svml.normalize-media-request@1";readonly source: BlobRef;diff --git a/packages/media-pipeline/test/manifest.test.ts b/packages/media-pipeline/test/manifest.test.tsindex 582d46e6..690ed23f 100644--- a/packages/media-pipeline/test/manifest.test.ts+++ b/packages/media-pipeline/test/manifest.test.ts@@ -17,7 +17,13 @@ import {mediaPipelineManifest,mediaPipelineTypes,decodeSynchronizedMediaSurface,+ decodeExtractAudioSurface,+ decodeExtractFrameSurface,+ decodeTransformMediaSurface,+ extractAudioFragment,+ extractFrameFragment,synchronizedMediaFragment,+ transformMediaFragment,} from "../src/index.js";test("the media pipeline is an ordinary Fragment over public media contracts", () => {@@ -78,4 +84,64 @@ test("media identities and selection requests have package-owned semantic validaassert.ok(registry.resolve(mediaTypes.muxed));assert.ok(registry.resolve(mediaPipelineTypes.selectionRequest));assert.ok(registry.resolve(mediaPipelineTypes.audioProgramPlan));+ assert.ok(registry.resolve(mediaPipelineTypes.transformProgram));+ assert.ok(registry.resolve(mediaPipelineTypes.audioExtractionRequest));+ assert.ok(registry.resolve(mediaPipelineTypes.frameExtractionRequest));+});++test("media operations are ordinary graph branches over BlobArtifact", async () => {+ const range = { source: "operations.svml", start: 0, end: 1 };+ const resolveReference = (path: string) => path === "generated.video"+ ? { path, ref: { kind: "record" as const, id: path }, type: artifactTypes.blob }+ : undefined;+ const context = { sourceName: range.source, resolveReference,+ resolveAsset: async () => { throw new Error("no asset resolution expected"); } };+ const transform = await decodeTransformMediaSurface({+ ...context,+ element: {+ kind: "element", name: "media:Transform", range,+ attributes: {+ id: "prepared", source: { kind: "reference", path: "generated.video" },+ video: "primary-moving", audio: "default", "span-authority": "video", "frame-rate": "30",+ },+ children: [+ { kind: "element", name: "media:Trim", attributes: { tail: "0.25s" }, children: [], range },+ { kind: "element", name: "media:Retime", attributes: { rate: "1.05", pitch: "preserve" }, children: [], range },+ ],+ },+ });+ assert.equal(transform.fragments[0]?.id, transformMediaFragment.id);+ assert.deepEqual(transform.components[0]?.outputs, { video: "prepared.video" });+ const program = transform.records.find((record) => record.id === "prepared.program");+ assert.ok(program?.value.kind === "inline");+ assert.deepEqual(program.value.value, {+ contract: "svml.media-transform-program@1",+ operations: [+ { kind: "trim", tailSec: 0.25 },+ { kind: "retime", rate: 1.05, pitch: "preserve" },+ ],+ });++ const audio = await decodeExtractAudioSurface({+ ...context,+ element: {+ kind: "element", name: "media:ExtractAudio", range,+ attributes: { id: "voice-reference", source: { kind: "reference", path: "generated.video" }, audio: "default" },+ children: [],+ },+ });+ assert.equal(audio.fragments[0]?.id, extractAudioFragment.id);+ assert.deepEqual(audio.components[0]?.outputs, { audio: "voice-reference.audio" });++ const frame = await decodeExtractFrameSurface({+ ...context,+ element: {+ kind: "element", name: "media:ExtractFrame", range,+ attributes: { id: "continuity", source: { kind: "reference", path: "generated.video" },+ video: "primary-moving", at: "last" },+ children: [],+ },+ });+ assert.equal(frame.fragments[0]?.id, extractFrameFragment.id);+ assert.deepEqual(frame.components[0]?.outputs, { image: "continuity.image" });});diff --git a/packages/provider-media-aws-lambda/README.md b/packages/provider-media-aws-lambda/README.mdindex be0336b0..4ebfdd9d 100644--- a/packages/provider-media-aws-lambda/README.md+++ b/packages/provider-media-aws-lambda/README.md@@ -1,6 +1,6 @@# `@narratage/provider-media-aws-lambda`-AWS Lambda Endpoint package for the five exact capabilities declared by+AWS Lambda Endpoint package for the eight exact capabilities declared by`@narratage/media-pipeline` and implemented by `@narratage/media-execution`.The Provider invokes one versioned or aliased Lambda ARN synchronously. Source and result bytes staydiff --git a/packages/provider-media-aws-lambda/package.json b/packages/provider-media-aws-lambda/package.jsonindex e504ea9d..622886af 100644--- a/packages/provider-media-aws-lambda/package.json+++ b/packages/provider-media-aws-lambda/package.json@@ -10,6 +10,7 @@"activation": "./src/activation.ts"},"dependencies": {+ "@narratage/artifact": "workspace:*","@narratage/endpoint-kit": "workspace:*","@narratage/media": "workspace:*","@narratage/media-execution": "workspace:*",diff --git a/packages/provider-media-aws-lambda/src/contract.ts b/packages/provider-media-aws-lambda/src/contract.tsindex 689b6fc5..daf35386 100644--- a/packages/provider-media-aws-lambda/src/contract.ts+++ b/packages/provider-media-aws-lambda/src/contract.ts@@ -1,5 +1,5 @@import { canonicalize } from "@narratage/protocol";-import type { CanonicalValue } from "@narratage/protocol";+import type { CanonicalValue, StoredValue } from "@narratage/protocol";/*** The envelope between the Endpoint and the function it invokes.@@ -15,6 +15,9 @@ export const MEDIA_LAMBDA_RESPONSE = "svml.media-lambda-response@1";export const mediaLambdaOperations = ["inspect","normalize",+ "transform",+ "extract-audio",+ "extract-frame","project-speech-evidence-audio","render-audio","mux",@@ -48,7 +51,7 @@ export type MediaLambdaResponse =readonly contract: typeof MEDIA_LAMBDA_RESPONSE;readonly operation: MediaLambdaOperation;readonly ok: true;- readonly value: CanonicalValue;+ readonly value: StoredValue;readonly metadata: CanonicalValue;}| {@@ -79,6 +82,19 @@ function operation(value: unknown, subject: string): MediaLambdaOperation {return value as MediaLambdaOperation;}+function storedValue(value: unknown, subject: string): StoredValue {+ const item = object(value, subject);+ if (item.kind === "inline") {+ assert(item.value !== null && typeof item.value === "object", `${subject}.value must be canonical data`);+ return item as unknown as StoredValue;+ }+ assert(item.kind === "blob" && typeof item.digest === "string"+ && Number.isSafeInteger(item.size) && (item.size as number) >= 0+ && typeof item.mediaType === "string" && item.mediaType.length > 0,+ `${subject} must be an inline or blob StoredValue`);+ return item as unknown as StoredValue;+}+export function sealMediaLambdaRequest(request: MediaLambdaRequest): CanonicalValue {parseMediaLambdaRequest(request);return canonicalize(request as unknown as CanonicalValue);@@ -129,14 +145,13 @@ export function parseMediaLambdaResponse(value: unknown): MediaLambdaResponse {};}assert(item.ok === true, "MediaLambdaResponse.ok must be a boolean");- assert(item.value !== null && typeof item.value === "object", "MediaLambdaResponse.value must be an object");assert(item.metadata !== null && typeof item.metadata === "object","MediaLambdaResponse.metadata must be an object");return {contract: MEDIA_LAMBDA_RESPONSE,operation: named,ok: true,- value: item.value as CanonicalValue,+ value: storedValue(item.value, "MediaLambdaResponse.value"),metadata: item.metadata as CanonicalValue,};}diff --git a/packages/provider-media-aws-lambda/src/provider.ts b/packages/provider-media-aws-lambda/src/provider.tsindex f2a5415b..f23fb1a7 100644--- a/packages/provider-media-aws-lambda/src/provider.ts+++ b/packages/provider-media-aws-lambda/src/provider.ts@@ -1,4 +1,5 @@import { mediaTypes } from "@narratage/media";+import { artifactTypes } from "@narratage/artifact";import { mediaNeedHasContract, mediaOperationContracts } from "@narratage/media-execution";import { mediaPipelineCapabilities } from "@narratage/media-pipeline";import { canonicalize, digestOf } from "@narratage/protocol";@@ -108,7 +109,7 @@ export function createAwsLambdaMediaProvider(config: CreateAwsLambdaMediaProvide+ ` — the Provider's bucket and the ArtifactStore's bucket are probably not the same`);}return {- value: { kind: "inline", value: reply.value },+ value: reply.value,conformance: "exact",delivery: "executed",metadata: reply.metadata,@@ -142,6 +143,27 @@ export function createAwsLambdaMediaProvider(config: CreateAwsLambdaMediaProvidesupports: (need) => mediaNeedHasContract(need.constraints, mediaOperationContracts.normalize),handler: operation("normalize"),},+ {+ lifecycle: "immediate" as const,+ capability: mediaPipelineCapabilities.transform,+ returns: artifactTypes.blob,+ supports: (need) => mediaNeedHasContract(need.constraints, mediaOperationContracts.transform),+ handler: operation("transform"),+ },+ {+ lifecycle: "immediate" as const,+ capability: mediaPipelineCapabilities.extractAudio,+ returns: artifactTypes.blob,+ supports: (need) => mediaNeedHasContract(need.constraints, mediaOperationContracts.extractAudio),+ handler: operation("extract-audio"),+ },+ {+ lifecycle: "immediate" as const,+ capability: mediaPipelineCapabilities.extractFrame,+ returns: artifactTypes.blob,+ supports: (need) => mediaNeedHasContract(need.constraints, mediaOperationContracts.extractFrame),+ handler: operation("extract-frame"),+ },{lifecycle: "immediate" as const,capability: mediaPipelineCapabilities.projectSpeechEvidenceAudio,diff --git a/packages/provider-media-aws-lambda/test/provider.test.ts b/packages/provider-media-aws-lambda/test/provider.test.tsindex a0f6f2ca..a050e2dc 100644--- a/packages/provider-media-aws-lambda/test/provider.test.ts+++ b/packages/provider-media-aws-lambda/test/provider.test.ts@@ -80,7 +80,7 @@ test("the Provider sends the Need verbatim with the bucket the function should ucontract: "svml.media-lambda-response@1",operation: "inspect",ok: true,- value: { contract: "svml.media-inspection@1", artifact },+ value: { kind: "inline", value: { contract: "svml.media-inspection@1", artifact } },metadata: { provider: "media.aws-lambda" },} as unknown as CanonicalValue));@@ -133,15 +133,10 @@ test("the remote Provider receives the exact AudioProgramPlan compiled for localcontract: "svml.media-lambda-response@1",operation: "render-audio",ok: true,- value: {- contract: "svml.timeline-audio@1",- artifact: output,- codec: "pcm_s16le",- sampleRate: 48_000,- channels: 2,- sampleFrames: 48_048,- loudness: "planned",- },+ value: { kind: "inline", value: {+ contract: "svml.timeline-audio@1", artifact: output, codec: "pcm_s16le",+ sampleRate: 48_000, channels: 2, sampleFrames: 48_048, loudness: "planned",+ } },metadata: { provider: "media.aws-lambda" },} as unknown as CanonicalValue));const capability = await endpointFor(invoker, { capability: "render-timeline-audio" });@@ -158,10 +153,10 @@ test("a Provider aimed at another bucket fails naming the Artifact, not later wicontract: "svml.media-lambda-response@1",operation: "inspect",ok: true,- value: {+ value: { kind: "inline", value: {contract: "svml.media-inspection@1",artifact: { kind: "blob", digest: stranger, size: 1, mediaType: "video/mp4" },- },+ } },metadata: {},} as unknown as CanonicalValue));diff --git a/packages/provider-media-local/README.md b/packages/provider-media-local/README.mdindex 7f7f56aa..920bdeb9 100644--- a/packages/provider-media-local/README.md+++ b/packages/provider-media-local/README.md@@ -1,6 +1,6 @@# `@narratage/provider-media-local`-Trusted local ffprobe/ffmpeg implementation of `@narratage/media-pipeline`'s four exact byte-operation+Trusted local ffprobe/ffmpeg implementation of `@narratage/media-pipeline`'s eight exact byte-operationcapabilities. It is Runtime configuration and is never imported by author `.svml`.```ts@@ -23,6 +23,7 @@ The Endpoint:- uses the selected authority stream's presentation interval;- preserves source A/V offset through deterministic trim, delay, pad and crop operations;- emits a silent CFR H.264 visual and, when selected, an exact-length 48 kHz stereo PCM WAV.+- transforms synchronized A/V, extracts generic reference audio and extracts exact source frames;- renders an explicit `AudioProgramPlan` into one exact-length 48 kHz stereo PCM `TimelineAudio`;- muxes exactly one verified silent visual stream and one verified program-audio stream into MP4;- distinguishes AAC coding-frame padding from the authoritative packet presentation span.diff --git a/packages/provider-media-local/package.json b/packages/provider-media-local/package.jsonindex 30016096..524a14f5 100644--- a/packages/provider-media-local/package.json+++ b/packages/provider-media-local/package.json@@ -10,6 +10,7 @@"activation": "./src/activation.ts"},"dependencies": {+ "@narratage/artifact": "workspace:*","@narratage/endpoint-kit": "workspace:*","@narratage/media": "workspace:*","@narratage/media-execution": "workspace:*",diff --git a/packages/provider-media-local/src/provider.ts b/packages/provider-media-local/src/provider.tsindex f227844b..9dddab91 100644--- a/packages/provider-media-local/src/provider.ts+++ b/packages/provider-media-local/src/provider.ts@@ -1,10 +1,14 @@import { mediaTypes } from "@narratage/media";+import { artifactTypes } from "@narratage/artifact";import {+ executeExtractAudio,+ executeExtractFrame,executeInspectMedia,executeMuxProgramMedia,executeNormalizeMedia,executeProjectSpeechEvidenceAudio,executeRenderTimelineAudio,+ executeTransformMedia,mediaNeedHasContract,mediaOperationContracts,} from "@narratage/media-execution";@@ -39,7 +43,7 @@ function positiveInteger(value: number, subject: string): number {function fulfillment(result: MediaOperationResult): EndpointFulfillment {return {- value: { kind: "inline", value: result.value },+ value: result.value,conformance: "exact",delivery: "executed",metadata: result.metadata,@@ -98,6 +102,27 @@ export function createLocalMediaProvider(config: CreateLocalMediaProviderOptionssupports: (need) => mediaNeedHasContract(need.constraints, mediaOperationContracts.normalize),handler: operation(executeNormalizeMedia),},+ {+ lifecycle: "immediate" as const,+ capability: mediaPipelineCapabilities.transform,+ returns: artifactTypes.blob,+ supports: (need) => mediaNeedHasContract(need.constraints, mediaOperationContracts.transform),+ handler: operation(executeTransformMedia),+ },+ {+ lifecycle: "immediate" as const,+ capability: mediaPipelineCapabilities.extractAudio,+ returns: artifactTypes.blob,+ supports: (need) => mediaNeedHasContract(need.constraints, mediaOperationContracts.extractAudio),+ handler: operation(executeExtractAudio),+ },+ {+ lifecycle: "immediate" as const,+ capability: mediaPipelineCapabilities.extractFrame,+ returns: artifactTypes.blob,+ supports: (need) => mediaNeedHasContract(need.constraints, mediaOperationContracts.extractFrame),+ handler: operation(executeExtractFrame),+ },{lifecycle: "immediate" as const,capability: mediaPipelineCapabilities.projectSpeechEvidenceAudio,diff --git a/packages/provider-media-local/test/provider.test.ts b/packages/provider-media-local/test/provider.test.tsindex 69db6329..bca885a9 100644--- a/packages/provider-media-local/test/provider.test.ts+++ b/packages/provider-media-local/test/provider.test.ts@@ -3,6 +3,7 @@ import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";import { tmpdir } from "node:os";import { join } from "node:path";import test from "node:test";+import { artifactTypes } from "@narratage/artifact";import { mediaTypes, sealRenderedVisual, verifyMediaInspection, verifyMuxedMedia, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";import type { MediaAudioStream, MediaInspection, MuxedMedia, SynchronizedMedia, TimelineAudio } from "@narratage/media";import { assertSpeechEvidenceAudioIdentity, speechTypes } from "@narratage/speech";@@ -527,6 +528,96 @@ test("a silent generated MP4 remains a visual-only product and cannot satisfy a}});+test("local media Provider transforms A/V and extracts ordinary audio and frame Artifacts", {+ skip: !hasMediaBinaries,+}, async () => {+ const root = await mkdtemp(join(tmpdir(), "svml-provider-media-ordinary-ops-"));+ try {+ const sourcePath = await fixture(root);+ const artifacts = new MemoryArtifactStore();+ const source = await artifacts.put(await readFile(sourcePath), "video/mp4");+ const inspection = await inspectArtifact(artifacts, source);+ const selectionRequest = sealMediaSelectionRequest({+ contract: "svml.media-selection-request@1",+ video: { mode: "primary-moving" },+ audio: { mode: "default" },+ spanAuthority: "video",+ frameRate: { numerator: 30, denominator: 1 },+ });+ const selection = selectMediaStreams(inspection, selectionRequest);+ const normalized = await normalizeArtifact({+ artifacts, source, inspection, selection, frameRate: selectionRequest.frameRate,+ });+ const executeArtifact = async (request: Need) => {+ const provider = await handlerFor(request);+ const result = await provider.handler({+ command: { kind: "fulfill-need", id: `command:${request.id}`, need: request },+ need: request,+ artifacts,+ credentials: {},+ });+ assert.equal(result.value.kind, "blob");+ return result.value.kind === "blob" ? result.value : (() => { throw new Error("expected BlobArtifact"); })();+ };++ const transformed = await executeArtifact(need(+ "need:transform",+ mediaPipelineCapabilities.transform,+ artifactTypes.blob,+ canonicalize({+ contract: "svml.transform-media-request@1",+ media: normalized,+ program: {+ contract: "svml.media-transform-program@1",+ operations: [+ { kind: "trim", tailSec: 0.2 },+ { kind: "retime", rate: 2, pitch: "preserve" },+ ],+ },+ }),+ ));+ const transformedInspection = await inspectArtifact(artifacts, transformed);+ assert.equal(transformedInspection.streams.find((item) => item.kind === "video")?.decodedUnitCount, 12);++ const audioIndex = inspection.streams.find((item) => item.kind === "audio")!.index;+ const extractedAudio = await executeArtifact(need(+ "need:extract-audio",+ mediaPipelineCapabilities.extractAudio,+ artifactTypes.blob,+ canonicalize({+ contract: "svml.extract-audio-request@1",+ source,+ streamIndex: audioIndex,+ output: { container: "wav", codec: "pcm_s16le", sampleRate: 48_000, channels: 2 },+ }),+ ));+ assert.equal(extractedAudio.mediaType, "audio/wav");+ const audioInspection = await inspectArtifact(artifacts, extractedAudio);+ const audio = audioInspection.streams.find((item) => item.kind === "audio");+ assert.ok(audio?.kind === "audio" && audio.sampleRate === 48_000 && audio.channels === 2);++ const video = inspection.streams.find((item) => item.kind === "video" && item.role === "moving")!;+ const extractedFrame = await executeArtifact(need(+ "need:extract-frame",+ mediaPipelineCapabilities.extractFrame,+ artifactTypes.blob,+ canonicalize({+ contract: "svml.extract-frame-request@1",+ source,+ streamIndex: video.index,+ sourceFrameCount: video.decodedUnitCount,+ at: { kind: "last" },+ output: { format: "png" },+ }),+ ));+ assert.equal(extractedFrame.mediaType, "image/png");+ const frameInspection = await inspectArtifact(artifacts, extractedFrame);+ assert.equal(frameInspection.streams.find((item) => item.kind === "video")?.decodedUnitCount, 1);+ } finally {+ await rm(root, { recursive: true, force: true }).catch(() => {});+ }+});+test("local media Provider renders one frame-domain audio plan and muxes exactly one silent visual with it", {skip: !hasMediaBinaries,}, async () => {diff --git a/packages/render-hyperframes/README.md b/packages/render-hyperframes/README.mdindex 020a48b6..02d5b5de 100644--- a/packages/render-hyperframes/README.md+++ b/packages/render-hyperframes/README.md@@ -9,7 +9,7 @@ The package owns `<render:Video composition={...}/>` and lowers it to six ordina3. compile every peer `AudioTrack` into one content-addressed `AudioProgramPlan`;4. request an exact 48 kHz `TimelineAudio`;5. request one `MuxedMedia` from those two independently verified Products;-6. project that Product into the shared `MediaArtifactRef` contract.+6. expose the verified mux Artifact unchanged as a domain-neutral `BlobArtifact`.The package is not a renderer and contains no queue, credentials or deployment choice. A localHyperFrames process, a hosted Hypit endpoint or another conforming execution package may register a@@ -17,6 +17,10 @@ Provider for the exact visual capability. Media Providers independently realizemux. None parses SVML or decides which Composition to render. Every result is bound to the sameProgramSpace and exact frame/sample domain before it can become a final video Record.+The output does not carry copied duration or lineage metadata. It is an ordinary content-addressed+Blob and can therefore be connected directly to any later component that accepts Blob bytes. A+consumer that needs stream or duration facts must request explicit media inspection.+Core therefore sees three real byte-operation Needs, not one opaque mega-render. Frame workers,contiguous or interleaved chunks, retries and image-sequence assembly remain inside the selectedvisual Endpoint; clip decoding/mixing remains inside the audio Endpoint; container encoding remainsdiff --git a/packages/render-hyperframes/package.json b/packages/render-hyperframes/package.jsonindex 33b201f5..6eeb9f25 100644--- a/packages/render-hyperframes/package.json+++ b/packages/render-hyperframes/package.json@@ -10,6 +10,7 @@"activation": "./src/activation.ts"},"dependencies": {+ "@narratage/artifact": "workspace:*","@narratage/component-kit": "workspace:*","@narratage/composition": "workspace:*","@narratage/elaborator": "workspace:*",diff --git a/packages/render-hyperframes/src/fragment.ts b/packages/render-hyperframes/src/fragment.tsindex d616e700..2c04e70b 100644--- a/packages/render-hyperframes/src/fragment.ts+++ b/packages/render-hyperframes/src/fragment.ts@@ -1,4 +1,4 @@-import { mediaTypes } from "@narratage/media";+import { artifactTypes } from "@narratage/artifact";import { programSpaceTypes } from "@narratage/program-space";import { compositionTypes } from "@narratage/composition";import { sealGraphFragment } from "@narratage/elaborator";@@ -61,7 +61,7 @@ export const renderHyperframesFragment = sealGraphFragment({],exports: [{name: "video",- type: mediaTypes.artifact,+ type: artifactTypes.blob,root: operation("project-video"),semanticInputs: ["composition", "space"],fidelity: "exact",diff --git a/packages/render-hyperframes/src/manifest.ts b/packages/render-hyperframes/src/manifest.tsindex c0748142..a8717b6e 100644--- a/packages/render-hyperframes/src/manifest.ts+++ b/packages/render-hyperframes/src/manifest.ts@@ -1,3 +1,4 @@+import { artifactDependency } from "@narratage/artifact";import { mediaDependency, mediaTypes } from "@narratage/media";import { compositionDependency } from "@narratage/composition";import {@@ -35,6 +36,7 @@ export const renderHyperframesManifest: ModuleManifest = {name: renderHyperframesModuleRef.name,version: renderHyperframesModuleRef.version,dependencies: [+ artifactDependency,mediaDependency,compositionDependency,{ module: hyperframesModuleRef, digest: hyperframesManifestDigest },diff --git a/packages/render-hyperframes/test/render.test.ts b/packages/render-hyperframes/test/render.test.tsindex 0d3a6048..de8a52eb 100644--- a/packages/render-hyperframes/test/render.test.ts+++ b/packages/render-hyperframes/test/render.test.ts@@ -1,4 +1,5 @@import { compositionComponent, videoContractManifests } from "../../test-support/video-domain.js";+import { artifactTypes } from "@narratage/artifact";import {registerProducerFacets,registerTypeValidatorFacets,@@ -52,10 +53,13 @@ import {} from "@narratage/render-hyperframes";import {compileAudioProgramPlan,+ decodeExtractFrameSurface,mediaPipelineComponent,mediaPipelineComponents,mediaPipelineCapabilities,mediaPipelineManifest,+ mediaPipelineModuleRef,+ mediaOperationSurfaceImplementationDigests,mediaPipelineProducers,} from "@narratage/media-pipeline";import {@@ -280,14 +284,12 @@ test("separate visual, audio and mux Endpoints complete one author-visible rende}).run(build());assert.equal(result.status, "complete");const video = result.state.records.find((record) =>- record.type.module.name === mediaTypes.artifact.module.name- && record.type.name === mediaTypes.artifact.name);- assert.deepEqual(inline(video), {- digest: finalArtifact.digest,- size: finalArtifact.size,- mediaType: finalArtifact.mediaType,- durationSec: space.durationSec,- });+ record.type.module.name === artifactTypes.blob.module.name+ && record.type.name === artifactTypes.blob.name+ && record.value.kind === "blob"+ && record.value.digest === finalArtifact.digest);+ assert(video, "missing final BlobArtifact");+ assert.deepEqual(video.value, finalArtifact);});test("a render Product with another frame domain is rejected by the explicit downstream join", async () => {@@ -387,7 +389,7 @@ function source(text: string): AuthorSourceUnit {};}-test("the official render Surface lowers real author source to the same BuildPlan", async () => {+test("the final rendered video is an ordinary BlobArtifact that can feed another author component", async () => {const sourceClosure = createResolvedClosure([...videoContractManifests,hyperframesManifest,@@ -410,21 +412,29 @@ test("the official render Surface lowers real author source to the same BuildPlarenderHyperframesSurfaceImplementationDigest,decodeHyperframesRenderSurface,);+ surfaces.registerStructured(+ mediaPipelineModuleRef,+ "extract-frame",+ mediaOperationSurfaceImplementationDigests.extractFrame,+ decodeExtractFrameSurface,+ );const frontends = new AuthorFrontendRegistry();frontends.register(createMarkupAuthorFrontend({registry: surfaces,resolveModule(request) {- return request.from.startsWith("@narratage/render-hyperframes")- ? renderHyperframesModuleRef- : fixtureModule;+ if (request.from.startsWith("@narratage/render-hyperframes")) return renderHyperframesModuleRef;+ if (request.from.startsWith("@narratage/media-pipeline")) return mediaPipelineModuleRef;+ return fixtureModule;},}));const compiled = await compileSourceClosure({entry: source(`<svml><import as="fixture" from="example.composition-fixture@1"/><import as="render" from="@narratage/render-hyperframes@1"/>+ <import as="media" from="@narratage/media-pipeline@1"/><fixture:Composition/><render:Video id="final" composition={composition} space={space}/>+ <media:ExtractFrame id="poster" source={final.video} video="primary-moving" at="last"/></svml>`),closure: sourceClosure,frontends,@@ -433,7 +443,7 @@ test("the official render Surface lowers real author source to the same BuildPla},admitRecord: createRecordAdmitter(validatorRegistry()),});- const target = resolveCompiledSourceExport(compiled, "final.video", mediaTypes.artifact);+ const target = resolveCompiledSourceExport(compiled, "poster.image", artifactTypes.blob);assert.equal(target.ref.kind, "logical-output");const state = start(compiled.program, compiled.elaboration.graph, sealBuildRequest({graph: compiled.elaboration.graph.id,@@ -447,5 +457,7 @@ test("the official render Surface lowers real author source to the same BuildPlamediaPipelineProducers.renderAudio.name,mediaPipelineProducers.mux.name,mediaPipelineProducers.projectMuxed.name,+ mediaPipelineProducers.inspect.name,+ mediaPipelineProducers.extractFrame.name,].sort());});diff --git a/packages/seedance-speaker/README.md b/packages/seedance-speaker/README.mdindex b4339c11..20cfb902 100644--- a/packages/seedance-speaker/README.md+++ b/packages/seedance-speaker/README.md@@ -9,7 +9,7 @@ directly to a visible `@narratage/text` render component in the graph. Optionaland `extra` values are also ordinary Text edges; they are nothidden Recipe strings. The rendered `Text` output is wired to Seedance's exact `prompt` port. Thesame Surface authors the low-level-`@narratage/seedance` `SpeechProgram`; generation reuses the exact Seedance request contracts. The+`@narratage/seedance` `DurationProgram`; generation reuses the exact Seedance ReferenceVideo request shape. Theofficial Text Template source is[`official-ugc-v1.svs`](./kits/official-ugc-v1.svs).@@ -29,7 +29,7 @@ Script dialogue Text ───────────────────optional action/extra Text ────────────────────┼──► render TextSVS-selected TextBindings + TextTemplate ──────┘ ││ exact prompt edge-authored Blob references + Seedance SpeechProgram ──────┼──► generation Need+authored Blob references + Seedance DurationProgram ────┼──► generation Need````take.prompt` and `take.video` are ordinary Logical Outputs; `take.program` is authored configuration.diff --git a/packages/seedance-speaker/src/fragment.ts b/packages/seedance-speaker/src/fragment.tsindex a5d77e0f..0f25c631 100644--- a/packages/seedance-speaker/src/fragment.ts+++ b/packages/seedance-speaker/src/fragment.ts@@ -1,6 +1,6 @@import type { ExactModelEndpoint, ExactModelMediaInput } from "@narratage/model-kit";import type { ProducerRef } from "@narratage/protocol";-import { createSeedanceSpeechGenerationFragment } from "@narratage/seedance";+import { createSeedanceDurationGenerationFragment } from "@narratage/seedance";/*** Speaker is an authoring Kit, not another execution protocol. Once its prompt@@ -12,7 +12,7 @@ export function createSeedanceSpeakerTakeFragment(compileRequestProducer: ProducerRef,mediaInputs: readonly ExactModelMediaInput[] = [],) {- return createSeedanceSpeechGenerationFragment(+ return createSeedanceDurationGenerationFragment(endpoint,compileRequestProducer,mediaInputs,diff --git a/packages/seedance-speaker/src/index.ts b/packages/seedance-speaker/src/index.tsindex 8b760015..44333178 100644--- a/packages/seedance-speaker/src/index.ts+++ b/packages/seedance-speaker/src/index.ts@@ -1,6 +1,6 @@export { createSeedanceSpeakerTakeFragment } from "./fragment.js";export {- createSpeakerSpeechProgram,+ createSpeakerDurationProgram,createSpeakerTextBindings,sealSpeakerTakeIntent,speakerMethodDefaults,diff --git a/packages/seedance-speaker/src/kit.ts b/packages/seedance-speaker/src/kit.tsindex df56f2b6..bff1f993 100644--- a/packages/seedance-speaker/src/kit.ts+++ b/packages/seedance-speaker/src/kit.ts@@ -1,10 +1,10 @@import { sealTextBindings } from "@narratage/text";import type { TextBindings, TextScalar } from "@narratage/text";import {- sealSeedanceSpeechProgram,+ sealSeedanceDurationProgram,seedanceModels,} from "@narratage/seedance";-import type { SeedanceSpeechProgram } from "@narratage/seedance";+import type { SeedanceDurationProgram } from "@narratage/seedance";import type { SpeakerTakeIntent } from "./types.js";@@ -84,10 +84,10 @@ export function createSpeakerTextBindings(intent: SpeakerTakeIntent): TextBindin});}-export function createSpeakerSpeechProgram(intent: SpeakerTakeIntent): SeedanceSpeechProgram {+export function createSpeakerDurationProgram(intent: SpeakerTakeIntent): SeedanceDurationProgram {verifySpeakerTakeIntent(intent);- return sealSeedanceSpeechProgram({- contract: "svml.seedance-speech-spine@1",+ return sealSeedanceDurationProgram({+ contract: "svml.seedance-duration-program@1",model: intent.model,ports: {resolution: [intent.resolution],diff --git a/packages/seedance-speaker/src/manifest.ts b/packages/seedance-speaker/src/manifest.tsindex 7639229c..c08a0102 100644--- a/packages/seedance-speaker/src/manifest.ts+++ b/packages/seedance-speaker/src/manifest.ts@@ -39,7 +39,7 @@ export const seedanceSpeakerManifest: ModuleManifest = {textTypes.bindings,textTypes.binding,textTypes.text,- seedanceTypes.speechSpine,+ seedanceTypes.durationProgram,artifactTypes.blob,...Object.values(seedanceEndpointsByModel).flatMap((endpoint) =>Object.values(endpoint.mediaBindings).map((binding) => binding.type)),diff --git a/packages/seedance-speaker/src/surface.ts b/packages/seedance-speaker/src/surface.tsindex 282ea7af..ad55e477 100644--- a/packages/seedance-speaker/src/surface.ts+++ b/packages/seedance-speaker/src/surface.ts@@ -16,7 +16,7 @@ import {import type { TextTemplate } from "@narratage/text";import {seedanceEndpointsByModel,- seedanceSpeechCompileProducers,+ seedanceDurationCompileProducers,seedanceTypes,} from "@narratage/seedance";import type { SeedanceModel } from "@narratage/seedance";@@ -31,7 +31,7 @@ import type {import { createSeedanceSpeakerTakeFragment } from "./fragment.js";import {- createSpeakerSpeechProgram,+ createSpeakerDurationProgram,createSpeakerTextBindings,sealSpeakerTakeIntent,speakerMethodDefaults,@@ -300,7 +300,7 @@ export const decodeSeedanceSpeakerTakeSurface: StructuredSurfaceHandler = ({ ele});const bindings = createSpeakerTextBindings(intent);// Static template semantics remain an ordinary deterministic graph step.- const program = createSpeakerSpeechProgram(intent);+ const program = createSpeakerDurationProgram(intent);const duration = resolved(element, "duration", resolveReference);if (!sameType(duration.type, speechTypes.duration)) {throw new Error(`${element.name}.duration must reference a SpeechDuration`);@@ -317,7 +317,7 @@ export const decodeSeedanceSpeakerTakeSurface: StructuredSurfaceHandler = ({ eleconst promptFragment = createTextRenderFragment(dynamicText.map(({ name }) => ({ name })));const generationFragment = createSeedanceSpeakerTakeFragment(endpoint,- seedanceSpeechCompileProducers[selectedModel],+ seedanceDurationCompileProducers[selectedModel],assembled.mediaInputs,);return {@@ -328,7 +328,7 @@ export const decodeSeedanceSpeakerTakeSurface: StructuredSurfaceHandler = ({ elerange: element.range,}, {id: programId,- type: seedanceTypes.speechSpine,+ type: seedanceTypes.durationProgram,value: { kind: "inline", value: program as unknown as CanonicalValue },range: element.range,}, ...dynamicText.map(({ name }) => ({diff --git a/packages/seedance-speaker/test/kit.test.ts b/packages/seedance-speaker/test/kit.test.tsindex 6a0b9d87..86b32f11 100644--- a/packages/seedance-speaker/test/kit.test.ts+++ b/packages/seedance-speaker/test/kit.test.ts@@ -12,7 +12,7 @@ import {} from "@narratage/text";import { parseSvs } from "@narratage/svs";import {- createSpeakerSpeechProgram,+ createSpeakerDurationProgram,createSpeakerTextBindings,sealSpeakerTakeIntent,speakerMethodDefaults,@@ -67,7 +67,7 @@ test("official UGC Kit renders one ordinary ordered Text value", () => {test("Speaker method preserves exact Seedance settings while prompt and media stay on graph edges", () => {const take = intent();- const program = createSpeakerSpeechProgram(take);+ const program = createSpeakerDurationProgram(take);assert.equal(program.model, "seedance-2-mini");assert.deepEqual(program.ports.resolution, ["720p"]);assert.deepEqual(program.ports.aspectRatio, ["9:16"]);diff --git a/packages/seedance/README.md b/packages/seedance/README.mdindex 3132ca83..5188b2b9 100644--- a/packages/seedance/README.md+++ b/packages/seedance/README.md@@ -1,12 +1,18 @@# `@narratage/seedance`-Exact Seedance author model module. It owns the request schema and the namespaced `Prompt`, `Video`-and `Speech` Surfaces; it does not contain KIE credentials, HTTP code, queues or runtime routing.+Exact Seedance author model module. It owns the request schema and exactly three invocation Surfaces;+it does not contain KIE credentials, HTTP code, queues, runtime routing or usage-specific Prompt assembly.-`Video` and `Speech` preserve the remote result as one atomic `GeneratedVideoSet`, then expose its-first ordered member through the shared deterministic primary-video projection. Their prompt ports-consume ordinary `Text`, so Script, a generic Text Template or any third-party author Surface can-feed them without importing the official Script parser.+The three model invocation shapes are deliberately separate:-Duration is explicit in this low-level model Surface. A later Speech scheduling package may wrap it-with estimation without changing Core or the exact Seedance/KIE contracts.+- `TextVideo`: Prompt only; this is the only shape that exposes Web Search.+- `FrameVideo`: required first frame and optional last frame.+- `ReferenceVideo`: one or more image, video or audio references within the model's port limits.++All three preserve the remote result as one atomic `GeneratedVideoSet`, then expose its first ordered+member as an ordinary `BlobArtifact`. Their prompt ports consume ordinary `Text`, so a Script projection,+generic Text Template or third-party author module can feed them without becoming part of Seedance.++`standard`, `fast` and `mini` select model variants independently of the invocation shape. Duration is+either literal or supplied by an explicit `SpeechDuration` edge; the generic internal `DurationProgram`+only delays that scalar binding and does not introduce a Speaker/Speech usage into the model interface.diff --git a/packages/seedance/src/activation.ts b/packages/seedance/src/activation.tsindex e90a8abf..171e1dd3 100644--- a/packages/seedance/src/activation.ts+++ b/packages/seedance/src/activation.ts@@ -1,8 +1,9 @@import { createMarkupSurfaceHostFacet } from "@narratage/markup";import {- decodeSeedanceSpeechSurface,- decodeSeedanceVideoSurface,+ decodeSeedanceFrameVideoSurface,+ decodeSeedanceReferenceVideoSurface,+ decodeSeedanceTextVideoSurface,seedanceComponent,seedanceManifest,seedanceModuleRef,@@ -17,17 +18,24 @@ export const svmlPackage = {hostFacets: [createMarkupSurfaceHostFacet({module: seedanceModuleRef,- surface: "speech",+ surface: "text-video",mode: "structured",- implementationDigest: seedanceSurfaceImplementationDigests.speech,- handler: decodeSeedanceSpeechSurface,+ implementationDigest: seedanceSurfaceImplementationDigests.textVideo,+ handler: decodeSeedanceTextVideoSurface,}),createMarkupSurfaceHostFacet({module: seedanceModuleRef,- surface: "video",+ surface: "frame-video",mode: "structured",- implementationDigest: seedanceSurfaceImplementationDigests.video,- handler: decodeSeedanceVideoSurface,+ implementationDigest: seedanceSurfaceImplementationDigests.frameVideo,+ handler: decodeSeedanceFrameVideoSurface,+ }),+ createMarkupSurfaceHostFacet({+ module: seedanceModuleRef,+ surface: "reference-video",+ mode: "structured",+ implementationDigest: seedanceSurfaceImplementationDigests.referenceVideo,+ handler: decodeSeedanceReferenceVideoSurface,}),],};diff --git a/packages/seedance/src/fragment.ts b/packages/seedance/src/fragment.tsindex 74371564..504921b8 100644--- a/packages/seedance/src/fragment.ts+++ b/packages/seedance/src/fragment.ts@@ -55,14 +55,14 @@ export function createSeedanceAssembledGenerationFragment(return createExactModelPrimaryGenerationFragment(endpoint, mediaInputs, textInputs);}-export function createSeedanceSpeechGenerationFragment(+export function createSeedanceDurationGenerationFragment(endpoint: ExactModelEndpoint,compileProducer: ProducerRef,mediaInputs: readonly ExactModelMediaInput[] = [],textInputs: readonly ExactModelTextInput[] = [],) {const inputs = [- { name: "program", type: { module: endpoint.producer.module, name: "SpeechProgram" } },+ { name: "program", type: { module: endpoint.producer.module, name: "DurationProgram" } },{ name: "duration", type: speechTypes.duration },];const operations: Array<import("@narratage/elaborator").FragmentOperation> = [{@@ -125,7 +125,7 @@ export function createSeedanceSpeechGenerationFragment(...mediaInputs.map((item) => `${item.name}=${item.port}:media`),].join(",") || "no-dynamic-inputs";return sealGraphFragment({- name: `@narratage/seedance/${endpoint.key}-speech-assembled-primary-video[${shape}]@1`,+ name: `@narratage/seedance/${endpoint.key}-duration-assembled-primary-video[${shape}]@1`,inputs,operations,exports: [{diff --git a/packages/seedance/src/index.ts b/packages/seedance/src/index.tsindex a077e756..0ca303c6 100644--- a/packages/seedance/src/index.ts+++ b/packages/seedance/src/index.ts@@ -86,31 +86,32 @@ export function sealSeedanceRequest(model: SeedanceModel, ports: SeedancePortMapreturn sealGenerationPortRequest(seedancePorts[model], ports);}-/** One authored generation minus the duration only speech estimation can supply. */-export type SeedanceSpeechProgram = {- readonly contract: "svml.seedance-speech-spine@1";+/** One authored generation minus a duration supplied by an explicit graph edge. */+export type SeedanceDurationProgram = {+ readonly contract: "svml.seedance-duration-program@1";readonly model: SeedanceModel;readonly ports: SeedancePortMap;};export const seedanceTypes = {- speechSpine: { module: seedanceModuleRef, name: "SpeechProgram" },+ durationProgram: { module: seedanceModuleRef, name: "DurationProgram" },} satisfies Record<string, TypeRef>;-export const seedanceSpeechCompileProducers = Object.fromEntries(+export const seedanceDurationCompileProducers = Object.fromEntries(seedanceModels.map((model) => [model, {module: seedanceModuleRef,- name: `compile-${model}-speech-request`,+ name: `compile-${model}-duration-request`,}]),) as Record<SeedanceModel, ProducerRef>;-export const seedanceSpeechCompileImplementationDigests = Object.fromEntries(- seedanceModels.map((model) => [model, digestOf(`@narratage/seedance/compile-${model}-speech-request@1`)]),+export const seedanceDurationCompileImplementationDigests = Object.fromEntries(+ seedanceModels.map((model) => [model, digestOf(`@narratage/seedance/compile-${model}-duration-request@1`)]),) as Record<SeedanceModel, Digest>;export const seedanceSurfaceImplementationDigests = {- speech: digestOf("@narratage/seedance/speech-surface@1"),- video: digestOf("@narratage/seedance/video-surface@1"),+ textVideo: digestOf("@narratage/seedance/text-video-surface@1"),+ frameVideo: digestOf("@narratage/seedance/frame-video-surface@1"),+ referenceVideo: digestOf("@narratage/seedance/reference-video-surface@1"),} as const;function assertObject(value: unknown): asserts value is Record<string, unknown> {@@ -119,11 +120,11 @@ function assertObject(value: unknown): asserts value is Record<string, unknown>}}-export function verifySeedanceSpeechProgram(value: unknown): asserts value is SeedanceSpeechProgram {+export function verifySeedanceDurationProgram(value: unknown): asserts value is SeedanceDurationProgram {assertObject(value);- if (value.contract !== "svml.seedance-speech-spine@1"+ if (value.contract !== "svml.seedance-duration-program@1"|| !seedanceModels.includes(value.model as SeedanceModel)) {- throw new Error("Seedance SpeechProgram identity is invalid");+ throw new Error("Seedance DurationProgram identity is invalid");}const table = seedancePorts[value.model as SeedanceModel];const later = table.ports@@ -131,20 +132,19 @@ export function verifySeedanceSpeechProgram(value: unknown): asserts value is Se.map((port) => port.name);verifyPortsAgainstTable(table, value.ports, { omit: ["duration", ...later] });const generateAudio = (value.ports as SeedancePortMap).generateAudio;- if (generateAudio?.[0] !== true) throw new Error("Seedance SpeechProgram must generate audio");}-export function sealSeedanceSpeechProgram(value: SeedanceSpeechProgram): SeedanceSpeechProgram {- const result = canonicalize(value) as unknown as SeedanceSpeechProgram;- verifySeedanceSpeechProgram(result);+export function sealSeedanceDurationProgram(value: SeedanceDurationProgram): SeedanceDurationProgram {+ const result = canonicalize(value) as unknown as SeedanceDurationProgram;+ verifySeedanceDurationProgram(result);return result;}-export function compileSeedanceSpeechRequestDraft(- program: SeedanceSpeechProgram,+export function compileSeedanceDurationRequestDraft(+ program: SeedanceDurationProgram,duration: SpeechDuration,): GenerationRequestDraft {- verifySeedanceSpeechProgram(program);+ verifySeedanceDurationProgram(program);assertSpeechDurationIdentity(duration);return sealGenerationRequestDraft(seedancePorts[program.model], {...program.ports,@@ -152,8 +152,8 @@ export function compileSeedanceSpeechRequestDraft(});}-const speechSpineSchema = (model: SeedanceModel): ValueSchema => generationObjectSchema({- contract: { schema: { kind: "literal", value: "svml.seedance-speech-spine@1" } },+const durationProgramSchema = (model: SeedanceModel): ValueSchema => generationObjectSchema({+ contract: { schema: { kind: "literal", value: "svml.seedance-duration-program@1" } },model: { schema: { kind: "literal", value: model } },ports: {schema: portsObjectSchema(seedancePorts[model], {@@ -197,17 +197,17 @@ export const seedanceManifest = {types: [...seedanceBaseDefinition.manifest.types,{- name: seedanceTypes.speechSpine.name,- schema: { kind: "oneOf", variants: seedanceModels.map(speechSpineSchema) } satisfies ValueSchema,+ name: seedanceTypes.durationProgram.name,+ schema: { kind: "oneOf", variants: seedanceModels.map(durationProgramSchema) } satisfies ValueSchema,},],surfaces: [{- name: "speech",- tag: "Speech",+ name: "text-video",+ tag: "TextVideo",mode: "structured",outputs: [- seedanceTypes.speechSpine,+ seedanceTypes.durationProgram,...Object.values(seedanceEndpoints).flatMap((endpoint) => [endpoint.draftType,...Object.values(endpoint.mediaBindings).map((binding) => binding.type),@@ -215,39 +215,51 @@ export const seedanceManifest = {],implementation: {kind: "trusted-frontend-surface",- locator: "@narratage/seedance/speech-surface",- digest: seedanceSurfaceImplementationDigests.speech,+ locator: "@narratage/seedance/text-video-surface",+ digest: seedanceSurfaceImplementationDigests.textVideo,},},{- name: "video",- tag: "Video",+ name: "frame-video",+ tag: "FrameVideo",mode: "structured",- outputs: Object.values(seedanceEndpoints).flatMap((endpoint) => [- endpoint.draftType,- ...Object.values(endpoint.mediaBindings).map((binding) => binding.type),- ]),+ outputs: [seedanceTypes.durationProgram, ...Object.values(seedanceEndpoints).flatMap((endpoint) => [+ endpoint.draftType, ...Object.values(endpoint.mediaBindings).map((binding) => binding.type),+ ])],implementation: {kind: "trusted-frontend-surface",- locator: "@narratage/seedance/video-surface",- digest: seedanceSurfaceImplementationDigests.video,+ locator: "@narratage/seedance/frame-video-surface",+ digest: seedanceSurfaceImplementationDigests.frameVideo,+ },+ },+ {+ name: "reference-video",+ tag: "ReferenceVideo",+ mode: "structured",+ outputs: [seedanceTypes.durationProgram, ...Object.values(seedanceEndpoints).flatMap((endpoint) => [+ endpoint.draftType, ...Object.values(endpoint.mediaBindings).map((binding) => binding.type),+ ])],+ implementation: {+ kind: "trusted-frontend-surface",+ locator: "@narratage/seedance/reference-video-surface",+ digest: seedanceSurfaceImplementationDigests.referenceVideo,},},],producers: [...seedanceBaseDefinition.manifest.producers,...seedanceModels.map((model) => ({- name: seedanceSpeechCompileProducers[model].name,+ name: seedanceDurationCompileProducers[model].name,inputs: [- { name: "program", type: seedanceTypes.speechSpine },+ { name: "program", type: seedanceTypes.durationProgram },{ name: "duration", type: speechTypes.duration },],outputs: [{ name: "draft", type: seedanceEndpointsByModel[model].draftType }],needs: [],implementation: {kind: "registered" as const,- locator: `@narratage/seedance/compile-${model}-speech-request`,- digest: seedanceSpeechCompileImplementationDigests[model],+ locator: `@narratage/seedance/compile-${model}-duration-request`,+ digest: seedanceDurationCompileImplementationDigests[model],},})),],@@ -258,16 +270,16 @@ export const seedanceComponent = {producers: [...seedanceBaseDefinition.component.producers,...seedanceModels.map((model) => ({- producer: seedanceSpeechCompileProducers[model],- implementationDigest: seedanceSpeechCompileImplementationDigests[model],+ producer: seedanceDurationCompileProducers[model],+ implementationDigest: seedanceDurationCompileImplementationDigests[model],handler: ({ inputs }: { readonly inputs: Readonly<Record<string, { readonly value: import("@narratage/protocol").StoredValue }>> }) => ({outputs: {draft: {kind: "inline" as const,- value: canonicalize(compileSeedanceSpeechRequestDraft(+ value: canonicalize(compileSeedanceDurationRequestDraft(inputs.program?.value.kind === "inline"- ? inputs.program.value.value as unknown as SeedanceSpeechProgram- : (() => { throw new Error("Seedance SpeechProgram must be inline"); })(),+ ? inputs.program.value.value as unknown as SeedanceDurationProgram+ : (() => { throw new Error("Seedance DurationProgram must be inline"); })(),inputs.duration?.value.kind === "inline"? inputs.duration.value.value as unknown as SpeechDuration: (() => { throw new Error("SpeechDuration must be inline"); })(),@@ -288,9 +300,10 @@ export const seedanceDefinition = {export {createSeedanceAssembledGenerationFragment,createSeedanceGenerationFragment,- createSeedanceSpeechGenerationFragment,+ createSeedanceDurationGenerationFragment,} from "./fragment.js";export {- decodeSeedanceSpeechSurface,- decodeSeedanceVideoSurface,+ decodeSeedanceFrameVideoSurface,+ decodeSeedanceReferenceVideoSurface,+ decodeSeedanceTextVideoSurface,} from "./surface.js";diff --git a/packages/seedance/src/surface.ts b/packages/seedance/src/surface.tsindex 5ef23869..63b4f9c0 100644--- a/packages/seedance/src/surface.ts+++ b/packages/seedance/src/surface.ts@@ -1,40 +1,41 @@-import { speechTypes } from "@narratage/speech";import { artifactTypes } from "@narratage/artifact";+import {+ generationPort,+ sealGenerationMediaBinding,+ sealGenerationRequestDraft,+} from "@narratage/generation";+import type {+ GenerationMediaPort,+ GenerationMediaRole,+ GenerationPortTable,+} from "@narratage/generation";+import { exactModelMediaInputNames, exactModelTextInputName } from "@narratage/model-kit";+import type { ExactModelEndpoint, ExactModelMediaInput } from "@narratage/model-kit";import type { CanonicalValue } from "@narratage/protocol";+import { speechTypes } from "@narratage/speech";+import { textTypes, verifyText } from "@narratage/text";import type {+ MarkupAttributeValue,StructuredElement,StructuredSurfaceHandler,SurfaceResolvedReference,- MarkupAttributeValue,} from "@narratage/markup";import {createSeedanceAssembledGenerationFragment,- createSeedanceSpeechGenerationFragment,+ createSeedanceDurationGenerationFragment,} from "./fragment.js";import {- generationPort,- sealGenerationMediaBinding,- sealGenerationRequestDraft,-} from "@narratage/generation";-import type { GenerationMediaPort, GenerationMediaRole, GenerationPortTable } from "@narratage/generation";-import { exactModelMediaInputNames, exactModelTextInputName } from "@narratage/model-kit";-import type { ExactModelEndpoint, ExactModelMediaInput } from "@narratage/model-kit";-import { textTypes, verifyText } from "@narratage/text";--import {- sealSeedanceSpeechProgram,+ sealSeedanceDurationProgram,+ seedanceDurationCompileProducers,seedanceEndpoints,seedancePorts,- seedanceSpeechCompileProducers,seedanceTypes,} from "./index.js";-import type {- SeedanceModel,- SeedancePortMap,-} from "./index.js";+import type { SeedanceModel, SeedancePortMap } from "./index.js";-type ReferenceInput = {+type MediaInput = {+ readonly port: "referenceImage" | "referenceVideo" | "referenceAudio" | "firstFrame" | "lastFrame";readonly role: GenerationMediaRole;readonly source: SurfaceResolvedReference;};@@ -43,18 +44,18 @@ function localName(value: string): string {return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value;}-function attributes(- element: StructuredElement,- required: readonly string[],- optional: readonly string[] = [],-): void {+function attributes(element: StructuredElement, required: readonly string[], optional: readonly string[] = []): void {const allowed = new Set([...required, ...optional]);const unknown = Object.keys(element.attributes).filter((name) => !allowed.has(name));if (unknown.length > 0 || required.some((name) => element.attributes[name] === undefined)) {- throw new Error(- `${element.name} requires ${required.join(", ")}`- + (optional.length === 0 ? "" : `; optional: ${optional.join(", ")}`),- );+ throw new Error(`${element.name} requires ${required.join(", ")}`+ + (optional.length === 0 ? "" : `; optional: ${optional.join(", ")}`));+ }+}++function empty(element: StructuredElement): void {+ if (element.children.some((child) => child.kind === "element" || child.value.trim().length > 0)) {+ throw new Error(`${element.name} must be empty`);}}@@ -75,7 +76,7 @@ function optionalStringAttribute(element: StructuredElement, name: string): strireturn value.trim();}-function referenceAttribute(element: StructuredElement, name: string): string {+function referencePath(element: StructuredElement, name: string): string {const value: MarkupAttributeValue | undefined = element.attributes[name];if (typeof value !== "object" || value.kind !== "reference" || value.path.length === 0) {throw new Error(`${element.name}.${name} must be a whole-value reference`);@@ -84,9 +85,7 @@ function referenceAttribute(element: StructuredElement, name: string): string {}function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {- return left.module.name === right.module.name- && left.module.version === right.module.version- && left.name === right.name;+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;}function resolved(@@ -94,18 +93,12 @@ function resolved(name: string,resolveReference: (path: string) => SurfaceResolvedReference | undefined,): SurfaceResolvedReference {- const path = referenceAttribute(element, name);+ const path = referencePath(element, name);const value = resolveReference(path);if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${path}`);return value;}-function inline(reference: SurfaceResolvedReference, subject: string): CanonicalValue {- const value = reference.record?.value;- if (value?.kind !== "inline") throw new Error(`${subject} must reference an authored value declared before use`);- return value.value;-}-function prompt(reference: SurfaceResolvedReference, subject: string): void {if (!sameType(reference.type, textTypes.text)) throw new Error(`${subject} must reference Text`);const value = reference.record?.value;@@ -115,7 +108,11 @@ function prompt(reference: SurfaceResolvedReference, subject: string): void {}}-function mediaReference(reference: SurfaceResolvedReference, role: GenerationMediaRole, subject: string): SurfaceResolvedReference {+function mediaReference(+ reference: SurfaceResolvedReference,+ role: GenerationMediaRole,+ subject: string,+): SurfaceResolvedReference {if (!sameType(reference.type, artifactTypes.blob)) throw new Error(`${subject} must reference a BlobArtifact`);const value = reference.record?.value;if (value !== undefined && (value.kind !== "blob" || !value.mediaType.startsWith(`${role}/`))) {@@ -152,7 +149,6 @@ function booleanAttribute(element: StructuredElement, name: string, fallback: bothrow new Error(`${element.name}.${name} must be true or false`);}-/** The model's own port table is the only source of truth for these bounds. */function enumeratedPort(table: GenerationPortTable, name: string): readonly (string | number)[] {const port = generationPort(table, name);if (port.value.kind !== "enum") throw new Error(`${table.model} port ${name} is not enumerated`);@@ -179,13 +175,12 @@ function generationSettings(}const aspectRatios = enumeratedPort(table, "aspectRatio");const aspectRatio = optionalStringAttribute(element, "aspect-ratio") ?? "9:16";- if (!aspectRatios.includes(aspectRatio)) {- throw new Error(`${element.name}.aspect-ratio is not supported by Seedance`);- }+ if (!aspectRatios.includes(aspectRatio)) throw new Error(`${element.name}.aspect-ratio is not supported by Seedance`);return {duration: [durationSec],resolution: [resolution],aspectRatio: [aspectRatio],+ generateAudio: [booleanAttribute(element, "generate-audio", false)],webSearch: [booleanAttribute(element, "web-search", false)],};}@@ -194,16 +189,15 @@ const REFERENCE_PORTS = {image: "referenceImage",video: "referenceVideo",audio: "referenceAudio",-} as const satisfies Record<GenerationMediaRole, string>;+} as const;-function references(+function referenceInputs(element: StructuredElement,model: SeedanceModel,resolveReference: (path: string) => SurfaceResolvedReference | undefined,-): ReferenceInput[] {- const table = seedancePorts[model];+): MediaInput[] {const accepted = Object.keys(REFERENCE_PORTS) as readonly GenerationMediaRole[];- const result: ReferenceInput[] = [];+ const result: MediaInput[] = [];for (const child of element.children) {if (child.kind === "text") {if (child.value.trim().length > 0) throw new Error(`${element.name} accepts only Reference children`);@@ -211,69 +205,80 @@ function references(}if (localName(child.name) !== "Reference") throw new Error(`${element.name} accepts only Reference children`);attributes(child, [], accepted);+ empty(child);const kinds = accepted.filter((kind) => child.attributes[kind] !== undefined);if (kinds.length !== 1) throw new Error(`${child.name} requires exactly one of ${accepted.join(", ")}`);const role = kinds[0]!;- const source = mediaReference(resolved(child, role, resolveReference), role, `${child.name}.${role}`);- result.push({ role, source });+ result.push({+ role,+ port: REFERENCE_PORTS[role],+ source: mediaReference(resolved(child, role, resolveReference), role, `${child.name}.${role}`),+ });}- // Every bound below is read from the model's own port table, never repeated here.+ if (result.length === 0) throw new Error(`${element.name} requires at least one Reference child`);+ const table = seedancePorts[model];for (const role of accepted) {const port = generationPort(table, REFERENCE_PORTS[role]);const used = result.filter((item) => item.role === role).length;- if (used > port.maxItems) {- throw new Error(`${element.name} accepts at most ${port.maxItems} ${role} references`);- }+ if (used > port.maxItems) throw new Error(`${element.name} accepts at most ${port.maxItems} ${role} references`);}return result;}-type AssembledReferences = {- readonly mediaInputs: readonly ExactModelMediaInput[];- readonly records: readonly {- readonly id: string;- readonly type: SurfaceResolvedReference["type"];- readonly value: { readonly kind: "inline"; readonly value: CanonicalValue };- readonly range: StructuredElement["range"];- }[];- readonly inputs: Readonly<Record<string, SurfaceResolvedReference["ref"] | { readonly kind: "record"; readonly id: string }>>;-};+function frameInputs(+ element: StructuredElement,+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,+): MediaInput[] {+ const first = mediaReference(resolved(element, "first-frame", resolveReference), "image", `${element.name}.first-frame`);+ const result: MediaInput[] = [{ port: "firstFrame", role: "image", source: first }];+ if (element.attributes["last-frame"] !== undefined) {+ result.push({+ port: "lastFrame",+ role: "image",+ source: mediaReference(resolved(element, "last-frame", resolveReference), "image", `${element.name}.last-frame`),+ });+ }+ return result;+}-function assembleReferences(+function assembleMedia(id: string,endpoint: ExactModelEndpoint,- values: readonly ReferenceInput[],+ values: readonly MediaInput[],range: StructuredElement["range"],-): AssembledReferences {- const records: AssembledReferences["records"][number][] = [];+) {+ const records: Array<{+ readonly id: string;+ readonly type: SurfaceResolvedReference["type"];+ readonly value: { readonly kind: "inline"; readonly value: CanonicalValue };+ readonly range: StructuredElement["range"];+ }> = [];const inputs: Record<string, SurfaceResolvedReference["ref"] | { readonly kind: "record"; readonly id: string }> = {};const mediaInputs = values.map((value, index): ExactModelMediaInput => {const name = `media-${String(index + 1).padStart(4, "0")}`;- const portName = REFERENCE_PORTS[value.role];- const binding = endpoint.mediaBindings[portName];- if (binding === undefined) throw new Error(`${endpoint.ports.model} has no media port ${portName}`);- const port = generationPort(endpoint.ports, portName);- if (port.value.kind !== "media") throw new Error(`${endpoint.ports.model} port ${portName} is not media`);- const mediaPort = port as GenerationMediaPort;+ const binding = endpoint.mediaBindings[value.port];+ if (binding === undefined) throw new Error(`${endpoint.ports.model} has no media port ${value.port}`);+ const port = generationPort(endpoint.ports, value.port);+ if (port.value.kind !== "media") throw new Error(`${endpoint.ports.model} port ${value.port} is not media`);const bindingId = `${id}.${name}.binding`;records.push({id: bindingId,type: binding.type,value: {kind: "inline",- value: sealGenerationMediaBinding(mediaPort, { role: value.role }) as unknown as CanonicalValue,+ value: sealGenerationMediaBinding(port as GenerationMediaPort, { role: value.role }) as unknown as CanonicalValue,},range,});const names = exactModelMediaInputNames(name);inputs[names.binding] = { kind: "record", id: bindingId };inputs[names.artifact] = value.source.ref;- return { name, port: portName };+ return { name, port: value.port };});return { mediaInputs, records, inputs };}-function speechDurationReference(+function durationReference(element: StructuredElement,resolveReference: (path: string) => SurfaceResolvedReference | undefined,): SurfaceResolvedReference | undefined {@@ -285,16 +290,55 @@ function speechDurationReference(return result;}-function generationOutput(- element: StructuredElement,- endpoint: ExactModelEndpoint,- draft: ReturnType<typeof sealGenerationRequestDraft>,- promptSource: SurfaceResolvedReference,- references: readonly ReferenceInput[],- output: string,-) {+function generationOutput(args: {+ readonly element: StructuredElement;+ readonly endpoint: ExactModelEndpoint;+ readonly model: SeedanceModel;+ readonly promptSource: SurfaceResolvedReference;+ readonly media: readonly MediaInput[];+ readonly resolveReference: (path: string) => SurfaceResolvedReference | undefined;+}) {+ const { element, endpoint, model, promptSource } = args;const id = stringAttribute(element, "id");- const assembled = assembleReferences(id, endpoint, references, element.range);+ const assembled = assembleMedia(id, endpoint, args.media, element.range);+ const duration = durationReference(element, args.resolveReference);+ if (duration !== undefined) {+ const { duration: _later, ...ports } = generationSettings(element, model, 4);+ const program = sealSeedanceDurationProgram({+ contract: "svml.seedance-duration-program@1",+ model,+ ports,+ });+ const programId = `${id}.duration-program`;+ const fragment = createSeedanceDurationGenerationFragment(+ endpoint,+ seedanceDurationCompileProducers[model],+ assembled.mediaInputs,+ [{ name: "prompt", port: "prompt" }],+ );+ return {+ records: [{+ id: programId,+ type: seedanceTypes.durationProgram,+ value: { kind: "inline" as const, value: program as unknown as CanonicalValue },+ range: element.range,+ }, ...assembled.records],+ components: [{+ id,+ fragment: fragment.id,+ inputs: {+ program: { kind: "record" as const, id: programId },+ duration: duration.ref,+ [exactModelTextInputName("prompt")]: promptSource.ref,+ ...assembled.inputs,+ },+ outputs: { video: `${id}.video` },+ range: element.range,+ }],+ fragments: [fragment],+ };+ }+ const draft = sealGenerationRequestDraft(seedancePorts[model], generationSettings(element, model));const fragment = createSeedanceAssembledGenerationFragment(endpoint,assembled.mediaInputs,@@ -316,81 +360,45 @@ function generationOutput([exactModelTextInputName("prompt")]: promptSource.ref,...assembled.inputs,},- outputs: { video: output },+ outputs: { video: `${id}.video` },range: element.range,}],fragments: [fragment],};}-export const decodeSeedanceVideoSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- attributes(element, ["id", "model", "prompt", "duration"], [- "resolution", "aspect-ratio", "generate-audio", "web-search",- ]);+const COMMON_OPTIONAL = ["resolution", "aspect-ratio", "generate-audio"] as const;++export const decodeSeedanceTextVideoSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ attributes(element, ["id", "model", "prompt", "duration"], [...COMMON_OPTIONAL, "web-search"]);+ empty(element);const selected = modelSelection(element);const promptSource = resolved(element, "prompt", resolveReference);prompt(promptSource, `${element.name}.prompt`);- const refs = references(element, selected.model, resolveReference);- const draft = sealGenerationRequestDraft(seedancePorts[selected.model], {- ...generationSettings(element, selected.model),- generateAudio: [booleanAttribute(element, "generate-audio", false)],+ return generationOutput({ element, ...selected, promptSource, media: [], resolveReference });+};++export const decodeSeedanceFrameVideoSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ attributes(element, ["id", "model", "prompt", "duration", "first-frame"], [...COMMON_OPTIONAL, "last-frame"]);+ empty(element);+ const selected = modelSelection(element);+ const promptSource = resolved(element, "prompt", resolveReference);+ prompt(promptSource, `${element.name}.prompt`);+ return generationOutput({+ element, ...selected, promptSource, media: frameInputs(element, resolveReference), resolveReference,});- return generationOutput(element, selected.endpoint, draft, promptSource, refs, `${stringAttribute(element, "id")}.video`);};-export const decodeSeedanceSpeechSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- attributes(element, ["id", "model", "prompt", "duration"], [- "resolution", "aspect-ratio", "web-search",- ]);+export const decodeSeedanceReferenceVideoSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ attributes(element, ["id", "model", "prompt", "duration"], COMMON_OPTIONAL);const selected = modelSelection(element);const promptSource = resolved(element, "prompt", resolveReference);prompt(promptSource, `${element.name}.prompt`);- const refs = references(element, selected.model, resolveReference);- const duration = speechDurationReference(element, resolveReference);- if (duration !== undefined) {- const { duration: _later, ...settings } = generationSettings(element, selected.model, 4);- const program = sealSeedanceSpeechProgram({- contract: "svml.seedance-speech-spine@1",- model: selected.model,- ports: {- ...settings,- generateAudio: [true],- },- });- const id = stringAttribute(element, "id");- const programId = `${id}.program`;- const assembled = assembleReferences(id, selected.endpoint, refs, element.range);- const fragment = createSeedanceSpeechGenerationFragment(- selected.endpoint,- seedanceSpeechCompileProducers[selected.model],- assembled.mediaInputs,- [{ name: "prompt", port: "prompt" }],- );- return {- records: [{- id: programId,- type: seedanceTypes.speechSpine,- value: { kind: "inline", value: program as unknown as CanonicalValue },- range: element.range,- }, ...assembled.records],- components: [{- id,- fragment: fragment.id,- inputs: {- program: { kind: "record", id: programId },- duration: duration.ref,- [exactModelTextInputName("prompt")]: promptSource.ref,- ...assembled.inputs,- },- outputs: { video: id },- range: element.range,- }],- fragments: [fragment],- };- }- const draft = sealGenerationRequestDraft(seedancePorts[selected.model], {- ...generationSettings(element, selected.model),- generateAudio: [true],+ return generationOutput({+ element,+ ...selected,+ promptSource,+ media: referenceInputs(element, selected.model, resolveReference),+ resolveReference,});- return generationOutput(element, selected.endpoint, draft, promptSource, refs, stringAttribute(element, "id"));};diff --git a/packages/seedance/test/surface.test.ts b/packages/seedance/test/surface.test.tsindex 76028ec0..2f2ed411 100644--- a/packages/seedance/test/surface.test.ts+++ b/packages/seedance/test/surface.test.ts@@ -3,47 +3,104 @@ import test from "node:test";import { artifactTypes } from "@narratage/artifact";import { parseStructuredElement } from "@narratage/markup";-import type { SurfaceResolvedReference } from "@narratage/markup";+import type { StructuredSurfaceHandler, SurfaceResolvedReference } from "@narratage/markup";import { textTypes } from "@narratage/text";import {- decodeSeedanceVideoSurface,+ decodeSeedanceFrameVideoSurface,+ decodeSeedanceReferenceVideoSurface,+ decodeSeedanceTextVideoSurface,} from "../src/index.js";-test("Seedance keeps upstream generated Text and image as graph edges", async () => {- const element = parseStructuredElement({- name: "dynamic-reference.svml",- text: `<seedance:Video id="motion" model="mini" prompt={direction} duration="6" resolution="720p" aspect-ratio="9:16">- <seedance:Reference image={generated.image}/>- </seedance:Video>`,- }, 0).element;- const refs = new Map<string, SurfaceResolvedReference>([- ["direction", {- path: "direction",- ref: { kind: "component-output", component: "assembled-direction", output: "text" },- type: textTypes.text,- }],- ["generated.image", {- path: "generated.image",- ref: { kind: "component-output", component: "generated", output: "image" },- type: artifactTypes.blob,- }],- ]);- const result = await decodeSeedanceVideoSurface({- sourceName: "dynamic-reference.svml",+const refs = new Map<string, SurfaceResolvedReference>([+ ["direction", {+ path: "direction",+ ref: { kind: "component-output", component: "assembled-direction", output: "text" },+ type: textTypes.text,+ }],+ ...["first.image", "last.image", "generated.image", "voice.audio"].map((path) => [path, {+ path,+ ref: { kind: "component-output" as const, component: path.split(".")[0]!, output: path.split(".")[1]! },+ type: artifactTypes.blob,+ }] as const),+]);++async function decode(source: string, handler: StructuredSurfaceHandler) {+ const element = parseStructuredElement({ name: "seedance.svml", text: source }, 0).element;+ return await handler({+ sourceName: "seedance.svml",element,resolveReference: (path) => refs.get(path),resolveAsset: () => { throw new Error("no asset"); },});+}++test("TextVideo exposes prompt-only generation without inventing a usage", async () => {+ const result = await decode(+ '<seedance:TextVideo id="motion" model="mini" prompt={direction} duration="6" resolution="720p" aspect-ratio="9:16" generate-audio="false" web-search="true"/>',+ decodeSeedanceTextVideoSurface,+ );+ assert.deepEqual(result.components[0]?.inputs["prompt:text"], {+ kind: "component-output", component: "assembled-direction", output: "text",+ });+ assert.deepEqual(result.components[0]?.outputs, { video: "motion.video" });+ assert.equal(Object.keys(result.components[0]?.inputs ?? {}).some((name) => name.includes("media")), false);+});++test("FrameVideo exposes first-frame and optional last-frame as exact model ports", async () => {+ const result = await decode(+ '<seedance:FrameVideo id="bridge" model="fast" prompt={direction} duration="5" first-frame={first.image} last-frame={last.image}/>',+ decodeSeedanceFrameVideoSurface,+ );+ const component = result.components[0]!;+ assert.deepEqual(component.inputs["media-0001:artifact"], {+ kind: "component-output", component: "first", output: "image",+ });+ assert.deepEqual(component.inputs["media-0002:artifact"], {+ kind: "component-output", component: "last", output: "image",+ });+ const bindings = result.records.filter((record) => record.id.includes(".binding"));+ assert.equal(bindings.length, 2);+ assert.equal(Object.keys(component.inputs).filter((name) => name.endsWith(":artifact")).length, 2);+});++test("ReferenceVideo keeps generated Text and heterogeneous references on explicit graph edges", async () => {+ const result = await decode(+ `<seedance:ReferenceVideo id="speaker" model="mini" prompt={direction} duration="6" generate-audio="true">+ <seedance:Reference image={generated.image}/>+ <seedance:Reference audio={voice.audio}/>+ </seedance:ReferenceVideo>`,+ decodeSeedanceReferenceVideoSurface,+ );const component = result.components[0]!;assert.deepEqual(component.inputs["media-0001:artifact"], {kind: "component-output", component: "generated", output: "image",});- assert.equal(component.inputs["media-0001:binding"]?.kind, "record");+ assert.deepEqual(component.inputs["media-0002:artifact"], {+ kind: "component-output", component: "voice", output: "audio",+ });assert.deepEqual(component.inputs["prompt:text"], {kind: "component-output", component: "assembled-direction", output: "text",});- const draft = result.records.find((record) => record.id === "motion.draft");+ const draft = result.records.find((record) => record.id === "speaker.draft");assert.ok(draft?.value.kind === "inline");- assert.equal(JSON.stringify(draft.value).includes("referenceImage"), false);+ assert.equal(JSON.stringify(draft.value).includes("referenceImage"), false,+ "runtime references must remain graph edges rather than authored draft metadata");+});++test("the three Surfaces make incompatible invocation shapes unrepresentable", async () => {+ await assert.rejects(+ async () => await decode(+ '<seedance:ReferenceVideo id="empty" model="mini" prompt={direction} duration="5"/>',+ decodeSeedanceReferenceVideoSurface,+ ),+ /requires at least one Reference/u,+ );+ await assert.rejects(+ async () => await decode(+ '<seedance:ReferenceVideo id="search" model="mini" prompt={direction} duration="5" web-search="true"><seedance:Reference image={generated.image}/></seedance:ReferenceVideo>',+ decodeSeedanceReferenceVideoSurface,+ ),+ /requires/u,+ );});diff --git a/packages/video-cli/test/cli.test.ts b/packages/video-cli/test/cli.test.tsindex 291618e9..da3a8892 100644--- a/packages/video-cli/test/cli.test.ts+++ b/packages/video-cli/test/cli.test.ts@@ -197,7 +197,7 @@ test("Text Template and Speaker keep prompt assembly visible in the Run Graph",await writeFile(join(root, "official-ugc-v1.svs"), kitSource, "utf8");await writeFile(join(root, "studio.svs"), `<?svml using="@narratage/svs@1"?><sheet version="1">- speech.normal { language: en; pace: normal; padding: 0.3; min: 4; max: 15; rounding: ceil; }+ speech.normal { language: en; pace: normal; min: 4; max: 15; rounding: round; }speaker.default { kind: ugc-talking-head; model: mini; resolution: 720p; aspect-ratio: 9:16; }</sheet>`, "utf8");await writeFile(join(root, "host.png"), new Uint8Array([137, 80, 78, 71]));@@ -239,7 +239,7 @@ test("Text Template and Speaker keep prompt assembly visible in the Run Graph","bind-request-seedance-2-mini-referenceImage","bind-request-seedance-2-mini-prompt-text","bind-text",- "compile-seedance-2-mini-speech-request",+ "compile-seedance-2-mini-duration-request","estimate-speech-duration","finalize-request-seedance-2-mini","request-seedance-2-mini",diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex ac2f4439..d2dacf91 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -704,6 +704,9 @@ importers:'@narratage/text':specifier: workspace:*version: link:../text+ cmu-pronouncing-dictionary:+ specifier: 3.0.0+ version: 3.0.0packages/film:dependencies:@@ -1768,6 +1771,9 @@ importers:packages/provider-media-aws-lambda:dependencies:+ '@narratage/artifact':+ specifier: workspace:*+ version: link:../artifact'@narratage/endpoint-kit':specifier: workspace:*version: link:../endpoint-kit@@ -1802,6 +1808,9 @@ importers:packages/provider-media-local:dependencies:+ '@narratage/artifact':+ specifier: workspace:*+ version: link:../artifact'@narratage/endpoint-kit':specifier: workspace:*version: link:../endpoint-kit@@ -1954,6 +1963,9 @@ importers:packages/render-hyperframes:dependencies:+ '@narratage/artifact':+ specifier: workspace:*+ version: link:../artifact'@narratage/component-kit':specifier: workspace:*version: link:../component-kit@@ -4377,6 +4389,9 @@ packages:resolution: {integrity: sha512-3Pe/CF1Nn94hyhIYpjtiLhdCoEoz0DqQ+988E9gmeEdQZlojxnOb74wctFyuwWQHzqyf9X7C7MG8juUpqBJT8w==}engines: {node: '>=0.8'}+ resolution: {integrity: sha512-6y5X4OxsYFnHERo6WSVPnLrk53G07MOQ8aUb2z555JgwynpbZsHR0a//WdN6DUcmE1j2auTF2TtL4PpoKze8QQ==}+resolution: {integrity: sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg==}@@ -6706,6 +6721,8 @@ snapshots:+ [email protected]: {}+diff --git a/services/media-lambda/src/handler.ts b/services/media-lambda/src/handler.tsindex 50e54769..3b476684 100644--- a/services/media-lambda/src/handler.ts+++ b/services/media-lambda/src/handler.ts@@ -3,10 +3,13 @@ import { AwsS3ObjectClient } from "@narratage/artifact-store-s3";import type { S3ObjectClient } from "@narratage/artifact-store-s3";import {executeInspectMedia,+ executeExtractAudio,+ executeExtractFrame,executeMuxProgramMedia,executeNormalizeMedia,executeProjectSpeechEvidenceAudio,executeRenderTimelineAudio,+ executeTransformMedia,} from "@narratage/media-execution";import type { MediaExecutionEnvironment, MediaOperationResult } from "@narratage/media-execution";import { execFile } from "node:child_process";@@ -33,6 +36,9 @@ const OPERATIONS: Record<> = {"inspect": executeInspectMedia,"normalize": executeNormalizeMedia,+ "transform": executeTransformMedia,+ "extract-audio": executeExtractAudio,+ "extract-frame": executeExtractFrame,"project-speech-evidence-audio": executeProjectSpeechEvidenceAudio,"render-audio": executeRenderTimelineAudio,"mux": executeMuxProgramMedia,diff --git a/services/media-lambda/test/handler.test.ts b/services/media-lambda/test/handler.test.tsindex 641a8e13..69984c21 100644--- a/services/media-lambda/test/handler.test.ts+++ b/services/media-lambda/test/handler.test.ts@@ -69,7 +69,9 @@ test("the function answers a real Need with real media, over a bucket it addressassert.equal(reply.ok, true, `the function failed: ${JSON.stringify(reply)}`);assert.equal(reply.operation, "inspect");- const value = reply.value as { contract: string; streams: { kind: string; sampleRate: number }[] };+ const stored = reply.value as { kind: string; value: { contract: string; streams: { kind: string; sampleRate: number }[] } };+ assert.equal(stored.kind, "inline");+ const value = stored.value;assert.equal(value.contract, "svml.media-inspection@1");assert.equal(value.streams.length, 1);assert.equal(value.streams[0]!.kind, "audio");diff --git a/spec/ranking-track.md b/spec/ranking-track.mdindex 42354a4e..02f87b9c 100644--- a/spec/ranking-track.md+++ b/spec/ranking-track.md@@ -242,7 +242,7 @@ Recipe properties fail. There is no arbitrary CSS escape hatch.## 7. Media, fonts and sound-Images and sounds enter through explicit `MediaArtifactRef` edges. Fonts enter through explicit,+Images and sounds enter through explicit `BlobArtifact` edges. Fonts enter through explicit,content-addressed font stack edges. Ranking never stores a URL, asks the Runtime to choose a font,or names a renderer component.diff --git a/spec/spatial-layout.md b/spec/spatial-layout.mdindex e390e61d..b0465f4a 100644--- a/spec/spatial-layout.md+++ b/spec/spatial-layout.md@@ -403,7 +403,7 @@ authored ContentFit ────────────────────package Paint/motion/clip -> VisualTrack```-Every dependency is a graph edge. `MediaArtifactRef` does not gain placement fields. `SpatialFrame`+Every dependency is a graph edge. `BlobArtifact` does not gain placement fields. `SpatialFrame`does not gain source, role, timing or Artifact digests. `VisualTrack` does not gain `focal`, B-rollor Text fields. Affinity is validated by the multi-input fitting/lowering Operation rather than bycopying lineage metadata through every value.diff --git a/spec/track-authoring.md b/spec/track-authoring.mdindex 48d02c8b..aea28044 100644--- a/spec/track-authoring.md+++ b/spec/track-authoring.md@@ -7,9 +7,10 @@ Audio and Screen Overlay are now specified in [`spatial-layout.md`](./spatial-la[`typography-track.md`](./typography-track.md), [`ranking-track.md`](./ranking-track.md),[`media-track.md`](./media-track.md), [`deck-track.md`](./deck-track.md),[`audio-track.md`](./audio-track.md) and-[`screen-overlay.md`](./screen-overlay.md). Comment Sticker remains independent future work and-does not block the packages already specified here. This is deliberately not a compatibility-promise for a released public ABI.+[`screen-overlay.md`](./screen-overlay.md). The independently designed+`@narratage/comment-sticker` package now uses the same Temporal and Spatial foundations and lowers+to one peer VisualTrack. This is deliberately not a compatibility promise for a released public+ABI.## Purpose@@ -446,11 +447,11 @@ The complete Ranking audit, component split, group schedule and executable migrasupport cumulative state, exclusive current stages and a final settled suffix without addingRanking or `Stage` meaning to Core or the public Track contract.-## 10. Remaining design and implementation work+## 10. Migration state-One independently designed author package remains outside the current migration:--1. design Comment Sticker content/layout/presentation independently when a real migration starts;+Comment Sticker content, layout and presentation are implemented as an independent author package;+it shares only the common Temporal, Spatial and terminal VisualTrack vocabulary. No currently+specified Track family remains blocked on this common authoring model.For every currently specified official Track package, the implemented acceptance matrix provesText, Caption, Media, Deck, Ranking and Screen Overlay lower to peer VisualTrack values, while