Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
refactor(video): normalize media at track boundaries
kashorin29c031b28cff
diff --git a/docs/guide/conventions.md b/docs/guide/conventions.mdindex b0539a52..783ef971 100644--- a/docs/guide/conventions.md+++ b/docs/guide/conventions.md@@ -21,7 +21,7 @@ description: Naming, module boundaries, TypeScript configuration and wire data.- Each package has exactly one public entry point: `src/index.ts`.- Internal modules use explicit `.js` extensions (NodeNext resolution).- Cross-package imports use `@narratage/*`, never relative paths across package boundaries.-- Circular dependencies are forbidden (enforced by `tools/package-boundaries.test.mjs`).+- Circular production dependencies are forbidden.## TypeScript configurationdiff --git a/docs/guide/packages.md b/docs/guide/packages.mdindex f07cc636..264e3a57 100644--- a/docs/guide/packages.md+++ b/docs/guide/packages.md@@ -141,7 +141,7 @@ never on exact-model packages or the CLI.## Dependency rules-`tools/package-boundaries.test.mjs` enforces three invariants on every commit:+The package layout follows three dependency rules:1. **Acyclic production graph.** No dependency cycle among any `@narratage/*` packages.@@ -152,7 +152,8 @@ never on exact-model packages or the CLI.Provider package. The video CLI also does not depend on any author-level video package(`@narratage/script`, `@narratage/seedance`, `@narratage/media-track`, `@narratage/typography-track`, `@narratage/film`).Author packages are activated through the explicit package lock, not compile-time CLI- dependencies.+ dependencies. These rules are kept visible in package manifests and reviewed as architecture,+ rather than approximated by source-text regex tests.## Package anatomydiff --git a/docs/guide/testing.md b/docs/guide/testing.mdindex 243d1943..eeae7892 100644--- a/docs/guide/testing.md+++ b/docs/guide/testing.md@@ -75,19 +75,13 @@ test("generates a video", async (t) => {});```-### Boundary tests+### Architecture boundaries-`tools/package-boundaries.test.mjs` and `tools/graph-first-value-boundary.test.mjs` verify-structural invariants across the entire workspace:+Package boundaries are expressed by package manifests, public entry points and shared contracts.+Tests exercise observable behavior at those boundaries. The repository deliberately does not use+source-text regex tests as a substitute for dependency analysis or architectural review.-- Acyclic dependency graph-- Domain-neutral closure contains only domain-neutral packages-- CLI has no Provider or author-package dependency--These run as part of `pnpm test` on every commit.--These tests remain when the repository is opened: protocol, graph, package-boundary and deterministic-unit tests are the executable definition of the decoupling rules. Before publication, remove customer+Before publication, remove customeror brand fixtures, credential traces, paid output artifacts, absolute workstation paths and one-offacceptance harnesses. Generic live tests may remain only when they are explicitly opt-in, use nocommitted secret and fail before spending money unless their gate is set.diff --git a/docs/quickstart/composition.md b/docs/quickstart/composition.mdindex a75409dc..9c4e7c4b 100644--- a/docs/quickstart/composition.md+++ b/docs/quickstart/composition.md@@ -50,7 +50,7 @@ Common Track sources:| Source | Type | From ||---|---|---|-| `{speech.visual}` | VisualTrack | `speech:Spine` — full-screen talking head |+| `{speech.visual}` | VisualTrack | `speech:Spine` — sparse same-source speech visuals || `{speech.audioTrack}` | AudioTrack | `speech:Spine` — synchronized audio || `{captions.track}` | VisualTrack | a Caption Style-family Track — timed captions || `{cards.visual}` | VisualTrack | `media-track:Track` — media overlays or B-roll |@@ -66,7 +66,7 @@ Typical stacking order:| stack-order | Content ||---|---|-| 10 | Speech visual (full-screen talking head) |+| 10 | Example Speech visual (author-selected, not a built-in default) || 40 | Media overlays || 70 | Captions || 90 | Text overlays |@@ -145,13 +145,16 @@ The complete data flow from Script to rendered video. This example is based onprompt={direction} duration="5"/><space:Canvas id="vertical" width="1080" height="1920"/>+ <space:Frame id="speech-frame" within={vertical}+ left="0%" top="0%" right="100%" bottom="100%"/><space:Frame id="title-frame" within={vertical}left="6%" top="6%" right="6%" bottom="84%"/><space:Frame id="card-frame" within={vertical}left="10%" top="20%" right="10%" bottom="30%"/><!-- 3. Timing: assemble spine and align words -->- <speech:Spine id="speech" canvas={vertical} frame-rate="30">+ <speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={take.video} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>@@ -204,6 +207,7 @@ The complete data flow from Script to rendered video. This example is based onfilm.vertical {background: #09090B;}+ speech.visual { fit: cover; }media.card {stack-order: 40; fit: cover; playback: hold-start;frame-paint: #111116; clip: rounded; radius: 20;diff --git a/docs/quickstart/timing.md b/docs/quickstart/timing.mdindex 2693cf59..c4a971bb 100644--- a/docs/quickstart/timing.md+++ b/docs/quickstart/timing.md@@ -13,6 +13,7 @@ steps produce the **ProgramSpace** and **SemanticMap** that every downstream com<import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/><import as="space" from="@narratage/spatial@1"/>+<import as="studio" source="./studio.svs"/>```## speech:Spine@@ -22,7 +23,9 @@ program order — the final sequence of Segments in the finished video.```svml<space:Canvas id="vertical" width="1080" height="1920"/>-<speech:Spine id="speech" canvas={vertical} frame-rate="30">+<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>+<speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={hook-take.video} segment={story.segment.hook}/><speech:Take video={meeting-take.video} segment={story.segment.meeting}/><speech:Take video={evidence-take.video} segment={story.segment.evidence}/>@@ -33,20 +36,26 @@ program order — the final sequence of Segments in the finished video.| Attribute | Required | Description ||---|---|---|| `id` | yes | Unique identifier |-| `canvas` | yes | Explicit CanvasSpace used by the restricted Media visual projection || `frame-rate` | yes | Program rate as an integer or rational, such as `30` or `30000/1001` |+| `visual-frame` | yes | Explicit base SpatialFrame for same-source Take visuals |+| `visual-appearance` | yes | SVS Recipe containing only spatial fit properties |+| `visual-z` | yes | Base absolute stacking order for same-source Take visuals |### speech:Take-Each `<speech:Take>` child binds a generated video to a Script Segment:+Each `<speech:Take>` child binds one speech-bearing source to a Script Segment:| Attribute | Required | Description ||---|---|---|-| `video` | exactly one | Generated/raw video Blob — for example `{take.video}` from Seedance |+| `video` | exactly one | Generated/raw A/V Blob — for example `{take.video}` from Seedance |+| `audio` | exactly one | Voice-only Blob; contributes time and master speech audio, but no visual || `media` | exactly one | Already prepared `SynchronizedMedia`; bypasses automatic normalization || `segment` | yes | Script Segment this take corresponds to — e.g. `{story.segment.hook}` |+| `frame` | visual only | Override the Spine's `visual-frame` for this Take |+| `appearance` | visual only | Override the Spine's `visual-appearance` for this Take |+| `z` | visual only | Override the Spine's `visual-z` for this Take |-`video` and `media` are mutually exclusive. The normal AIGC path is `video={take.video}`. The+`video`, `audio` and `media` are mutually exclusive. The normal AIGC path is `video={take.video}`. TheSpeech Surface expands that readable declaration into ordinary Media Pipeline Operations:inspect the container, select its primary moving video and default audio stream, then normalizeboth to the Spine's declared frame rate. A 30 fps source connected to a 60 fps Spine keeps its@@ -54,6 +63,17 @@ duration and is deterministically resampled to a 60 fps frame sequence; it is nofast. Use `media=` only when another graph branch has already produced the exact synchronized valueyou intend to assemble.+The base is deliberately explicit rather than a hidden full-screen default. A visual Take inherits+all three values unless it overrides them. An audio Take cannot declare visual overrides: while it+is playing, `speech.visual` simply has no Present, so the Film background or peer Tracks remain+visible.++The fit Recipe is ordinary compile-time SVS data, for example:++```svs+speech.visual { fit: cover; }+```+The order of `<speech:Take>` children **determines the program order**. The first take starts attime zero; each subsequent take follows immediately.@@ -63,7 +83,7 @@ The Spine produces four outputs used by downstream components:| Output | Type | Used by ||---|---|---|-| `{speech.visual}` | VisualTrack | `film:Film` — the full-screen talking-head video |+| `{speech.visual}` | VisualTrack | `film:Film` — sparse same-source Take visuals || `{speech.audio}` | Audio | `whisperx:Alignment` — raw audio for word timing || `{speech.audioTrack}` | AudioTrack | `film:Film` — the synchronized audio || `{speech.space}` | ProgramSpace | Everything — the unified duration and frame domain |@@ -122,6 +142,9 @@ The SemanticMap is the typed bridge between Script text and physical time. Whenexact frame range that Selection covers. Without a SemanticMap, Selections and Moments have nophysical meaning.+A whole Segment needs no synthetic Selection. `during={story.segment.answer}` addresses the+Segment's existing structural start/end anchors directly.+Components that use the map take it via the `map` attribute:```svml@@ -142,10 +165,13 @@ The complete timing stage, from generated takes to map and space:<import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/><import as="space" from="@narratage/spatial@1"/>+<import as="studio" source="./studio.svs"/><!-- Assemble takes in program order --><space:Canvas id="vertical" width="1080" height="1920"/>-<speech:Spine id="speech" canvas={vertical} frame-rate="30">+<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>+<speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={opening-take.video} segment={story.segment.opening}/><speech:Take video={answer-take.video} segment={story.segment.answer}/></speech:Spine>@@ -155,7 +181,7 @@ The complete timing stage, from generated takes to map and space:<!-- Downstream components now reference: --><!-- {speech.space} — ProgramSpace for duration/frame domain -->-<!-- {speech.visual} — VisualTrack for the talking-head video -->+<!-- {speech.visual} — sparse same-source speech VisualTrack --><!-- {speech.audioTrack} — AudioTrack for synchronized audio --><!-- {timing.map} — SemanticMap for Selection/Moment timing -->```diff --git a/docs/zh/guide/conventions.md b/docs/zh/guide/conventions.mdindex 15f87ad7..b88b9bd7 100644--- a/docs/zh/guide/conventions.md+++ b/docs/zh/guide/conventions.md@@ -21,7 +21,7 @@ description: 命名、模块边界、TypeScript 配置与 wire 数据。- 每个包有且只有一个公开入口点:`src/index.ts`。- 内部模块使用显式的 `.js` 扩展名(NodeNext 解析)。- 跨包导入使用 `@narratage/*`,绝不跨包边界使用相对路径。-- 禁止循环依赖(由 `tools/package-boundaries.test.mjs` 强制执行)。+- 禁止生产依赖形成循环。## TypeScript 配置diff --git a/docs/zh/guide/packages.md b/docs/zh/guide/packages.mdindex b6752e67..bf479a6b 100644--- a/docs/zh/guide/packages.md+++ b/docs/zh/guide/packages.md@@ -135,13 +135,13 @@ Sticker、Deck 或任何模型家族。## 依赖规则-`tools/package-boundaries.test.mjs` 在每次提交时强制执行三条不变式:+包结构遵循三条依赖规则:1. **无环的生产依赖图。** 任何 `@narratage/*` 包之间都不存在依赖环。2. **领域无关闭包。** 每个 Layer 1 包的传递闭包只包含 Layer 1 的包。`@narratage/core` 只依赖 `@narratage/protocol`。-3. **CLI 独立性。** `@narratage/cli` 和 `@narratage/video-cli` 都不会传递依赖任何 Provider 包。video CLI 同样不依赖任何作者层的视频包(`@narratage/script`、`@narratage/seedance`、`@narratage/media-track`、`@narratage/typography-track`、`@narratage/film`)。作者包通过显式的 package lock 被 activate,而不是通过编译期的 CLI 依赖。+3. **CLI 独立性。** `@narratage/cli` 和 `@narratage/video-cli` 都不会传递依赖任何 Provider 包。video CLI 同样不依赖任何作者层的视频包(`@narratage/script`、`@narratage/seedance`、`@narratage/media-track`、`@narratage/typography-track`、`@narratage/film`)。作者包通过显式的 package lock 被 activate,而不是通过编译期的 CLI 依赖。这些规则直接体现在 package manifest 中并接受架构审查,不再用源码文本正则来近似证明。## 包的结构diff --git a/docs/zh/guide/testing.md b/docs/zh/guide/testing.mdindex 09a1514d..d3805c49 100644--- a/docs/zh/guide/testing.md+++ b/docs/zh/guide/testing.md@@ -66,17 +66,11 @@ test("generates a video", async (t) => {});```-### Boundary 测试+### 架构边界-`tools/package-boundaries.test.mjs` 和 `tools/graph-first-value-boundary.test.mjs` 会在整个 workspace 范围内校验结构性不变量:+包边界由 package manifest、公开入口和共享合同表达;测试只验证边界上可观察的行为。+仓库不再把源码文本正则当作依赖分析或架构审查的替代品。-- 依赖图无环-- 领域中立闭包中只包含领域中立的包-- CLI 不依赖任何 Provider 或作者包--它们作为 `pnpm test` 的一部分,在每次提交时运行。--这些测试在仓库开源时应当保留:协议、图、包边界与确定性单元测试是解耦规则的可执行定义。发布前应删除客户或品牌 fixture、凭据痕迹、付费产物、工作站绝对路径与一次性验收脚本。通用 live test 只有在显式 opt-in、不提交秘密,并且未设置开关时能在花钱前退出的情况下才保留。diff --git a/docs/zh/quickstart/composition.md b/docs/zh/quickstart/composition.mdindex e0820c47..b704166a 100644--- a/docs/zh/quickstart/composition.md+++ b/docs/zh/quickstart/composition.md@@ -49,7 +49,7 @@ Composition。然后渲染器将该 Composition 编译为 MP4 视频。| 来源 | 类型 | 来自 ||---|---|---|-| `{speech.visual}` | VisualTrack | `speech:Spine`——全屏说话人画面 |+| `{speech.visual}` | VisualTrack | `speech:Spine`——稀疏的同源口播视觉 || `{speech.audioTrack}` | AudioTrack | `speech:Spine`——同步音频 || `{captions.track}` | VisualTrack | Caption 样式族 Track——定时字幕 || `{cards.visual}` | VisualTrack | `media-track:Track`——Media 叠加层或 B-roll |@@ -64,7 +64,7 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T| stack-order | 内容 ||---|---|-| 10 | 语音画面(全屏说话人画面) |+| 10 | 示例口播视觉(作者显式选择,不是内置默认值) || 40 | Media 叠加层 || 70 | 字幕 || 90 | 文字叠加层 |@@ -141,13 +141,16 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参prompt={direction} duration="5"/><space:Canvas id="vertical" width="1080" height="1920"/>+ <space:Frame id="speech-frame" within={vertical}+ left="0%" top="0%" right="100%" bottom="100%"/><space:Frame id="title-frame" within={vertical}left="6%" top="6%" right="6%" bottom="84%"/><space:Frame id="card-frame" within={vertical}left="10%" top="20%" right="10%" bottom="30%"/><!-- 3. Timing: assemble spine and align words -->- <speech:Spine id="speech" canvas={vertical} frame-rate="30">+ <speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={take.video} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>@@ -200,6 +203,7 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参film.vertical {background: #09090B;}+ speech.visual { fit: cover; }media.card {stack-order: 40; fit: cover; playback: hold-start;frame-paint: #111116; clip: rounded; radius: 20;diff --git a/docs/zh/quickstart/timing.md b/docs/zh/quickstart/timing.mdindex 3922e29c..b1a93d98 100644--- a/docs/zh/quickstart/timing.md+++ b/docs/zh/quickstart/timing.md@@ -11,6 +11,7 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连<import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/><import as="space" from="@narratage/spatial@1"/>+<import as="studio" source="./studio.svs"/>```## speech:Spine@@ -19,7 +20,9 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连```svml<space:Canvas id="vertical" width="1080" height="1920"/>-<speech:Spine id="speech" canvas={vertical} frame-rate="30">+<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>+<speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={hook-take.video} segment={story.segment.hook}/><speech:Take video={meeting-take.video} segment={story.segment.meeting}/><speech:Take video={evidence-take.video} segment={story.segment.evidence}/>@@ -30,25 +33,39 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连| 属性 | 必填 | 描述 ||---|---|---|| `id` | 是 | 唯一标识符 |-| `canvas` | 是 | 受限 Media 视觉投影显式使用的 CanvasSpace || `frame-rate` | 是 | 节目帧率,写整数或有理数,例如 `30`、`30000/1001` |+| `visual-frame` | 是 | 同源 Take 视觉的显式基础 SpatialFrame |+| `visual-appearance` | 是 | 只包含空间 fit 属性的 SVS Recipe |+| `visual-z` | 是 | 同源 Take 视觉的基础绝对层级 |### speech:Take-每个 `<speech:Take>` 子元素将一个生成的视频绑定到一个 Script Segment:+每个 `<speech:Take>` 子元素将一个带口播的来源绑定到一个 Script Segment:| 属性 | 必填 | 描述 ||---|---|---|-| `video` | 二选一 | 生成/原始视频 Blob,例如 Seedance 产出的 `{take.video}` |-| `media` | 二选一 | 已准备好的 `SynchronizedMedia`;跳过自动规范化 |+| `video` | 三选一 | 生成/原始视频 Blob,例如 Seedance 产出的 `{take.video}` |+| `audio` | 三选一 | 纯语音 Blob;贡献节目时长与主音频,不贡献视觉 |+| `media` | 三选一 | 已准备好的 `SynchronizedMedia`;跳过自动规范化 || `segment` | 是 | 此片段对应的 Script Segment——例如 `{story.segment.hook}` |+| `frame` | 仅视觉 | 覆盖本 Take 的基础 `visual-frame` |+| `appearance` | 仅视觉 | 覆盖本 Take 的基础 `visual-appearance` |+| `z` | 仅视觉 | 覆盖本 Take 的基础 `visual-z` |-`video` 与 `media` 必须且只能选一个。AIGC 的正常路径就是 `video={take.video}`。Speech+`video`、`audio` 与 `media` 必须且只能选一个。AIGC 的正常路径就是 `video={take.video}`。SpeechSurface 会把这句易读声明展开为普通 Media Pipeline Operation:检查容器、选择主动态视频流和默认音轨,再按 Spine 声明的帧率规范化。把 30 fps 素材接入 60 fps Spine 时,时长不变,帧序列会被确定性重采样为 60 fps,而不是把视频播放加速一倍。只有上游图已经明确产出所需`SynchronizedMedia` 时才使用 `media=`。+视觉 Take 默认继承 Spine 上显式写出的三个视觉值,也可以逐段覆盖。纯音频 Take 禁止写+视觉覆盖;它播放时 `speech.visual` 就没有 Present,Film 背景或平级 Track 会自然露出,+系统不会伪造黑场。fit Recipe 只是普通编译期 SVS 数据,例如:++```svs+speech.visual { fit: cover; }+```+`<speech:Take>` 子元素的排列顺序**决定了节目顺序**。第一个片段从时间零点开始;后续片段依次紧接。### 输出@@ -57,7 +74,7 @@ Spine 产生四个输出,供下游组件使用:| 输出 | 类型 | 使用方 ||---|---|---|-| `{speech.visual}` | VisualTrack | `film:Film`——全屏真人出镜视频 |+| `{speech.visual}` | VisualTrack | `film:Film`——稀疏的同源 Take 视觉 || `{speech.audio}` | Audio | `whisperx:Alignment`——用于词级时序的原始音频 || `{speech.audioTrack}` | AudioTrack | `film:Film`——同步音频 || `{speech.space}` | ProgramSpace | 所有组件——统一的时长和帧域 |@@ -108,6 +125,8 @@ ProgramSpace 包含:SemanticMap 是连接 Script 文本与物理时间的类型化桥梁。当你在 Media Item 上写 `during={story.selection.demo}` 时,组件会使用 SemanticMap 查找该 Selection 覆盖的精确帧范围。没有 SemanticMap,Selection 和 Moment 就没有物理意义。+整段使用无需伪造 Selection:`during={story.segment.answer}` 直接消费 Segment 已有的首尾结构锚点。+使用该映射的组件通过 `map` 属性接收它:```svml@@ -127,10 +146,13 @@ Map 只包含最终词窗口和语义锚点,不传播“测量、推导、估<import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/><import as="space" from="@narratage/spatial@1"/>+<import as="studio" source="./studio.svs"/><!-- Assemble takes in program order --><space:Canvas id="vertical" width="1080" height="1920"/>-<speech:Spine id="speech" canvas={vertical} frame-rate="30">+<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>+<speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={opening-take.video} segment={story.segment.opening}/><speech:Take video={answer-take.video} segment={story.segment.answer}/></speech:Spine>@@ -140,7 +162,7 @@ Map 只包含最终词窗口和语义锚点,不传播“测量、推导、估<!-- Downstream components now reference: --><!-- {speech.space} — ProgramSpace for duration/frame domain -->-<!-- {speech.visual} — VisualTrack for the talking-head video -->+<!-- {speech.visual} — 稀疏的同源口播 VisualTrack --><!-- {speech.audioTrack} — AudioTrack for synchronized audio --><!-- {timing.map} — SemanticMap for Selection/Moment timing -->```diff --git a/examples/talking-film-golden/main.svml b/examples/talking-film-golden/main.svmlindex 8ba56335..3caa19e2 100644--- a/examples/talking-film-golden/main.svml+++ b/examples/talking-film-golden/main.svml@@ -46,6 +46,8 @@<media:Image id="product-reference" src="./assets/product.png"/><space:Canvas id="vertical" width="1080" height="1920"/>+ <space:Frame id="speech-frame" within={vertical}+ left="0%" top="0%" right="100%" bottom="100%"/><space:Frame id="title-frame" within={vertical}left="6%" top="6%" right="6%" bottom="84%"/><space:Frame id="product-frame" within={vertical}@@ -113,7 +115,8 @@</seedance:ReferenceVideo><!-- Ordered generated Takes become one audio basis plus its peer visual/audio Tracks. -->- <speech:Spine id="speech" canvas={vertical} frame-rate="30">+ <speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={alice-take.video} segment={script.segment.opening}/><speech:Take video={bob-take.video} segment={script.segment.answer}/></speech:Spine>diff --git a/examples/talking-film-golden/studio.svs b/examples/talking-film-golden/studio.svsindex 44279403..9ed2389c 100644--- a/examples/talking-film-golden/studio.svs+++ b/examples/talking-film-golden/studio.svs@@ -19,14 +19,7 @@background: #09090B;}- speech.full {- stack-order: 10;- x: 0;- y: 0;- width: 1;- height: 1;- fit: cover;- }+ speech.visual { fit: cover; }media.product {stack-order: 40;diff --git a/examples/talking-film-graph-check/main.svml b/examples/talking-film-graph-check/main.svmlindex d65384ec..775c4574 100644--- a/examples/talking-film-graph-check/main.svml+++ b/examples/talking-film-graph-check/main.svml@@ -22,6 +22,8 @@</script><space:Canvas id="vertical" width="1080" height="1920"/>+ <space:Frame id="speech-frame" within={vertical}+ left="0%" top="0%" right="100%" bottom="100%"/><space:Frame id="title-frame" within={vertical}left="6%" top="6%" right="6%" bottom="84%"/><space:Frame id="card-frame" within={vertical}@@ -31,7 +33,8 @@<seedance:TextVideo id="take" model="mini" prompt={direction} duration="5" generate-audio="true"/><seedance:TextVideo id="motion" model="mini" prompt={direction} duration="5"/>- <speech:Spine id="speech" canvas={vertical} frame-rate="30">+ <speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={take.video} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>diff --git a/examples/talking-film-graph-check/studio.svs b/examples/talking-film-graph-check/studio.svsindex 99821095..79a68cfb 100644--- a/examples/talking-film-graph-check/studio.svs+++ b/examples/talking-film-graph-check/studio.svs@@ -2,6 +2,7 @@<sheet version="1">film.vertical { background: #09090B; }+ speech.visual { fit: cover; }media.card {stack-order: 40; fit: cover; playback: hold-start;frame-paint: #111116; clip: rounded; radius: 20;diff --git a/examples/talking-film-graph-check/svml.packages.lock b/examples/talking-film-graph-check/svml.packages.lockindex eaa02f72..55005928 100644--- a/examples/talking-film-graph-check/svml.packages.lock+++ b/examples/talking-film-graph-check/svml.packages.lock@@ -597,7 +597,7 @@{"name": "@narratage/caption","version": "0.0.0-dev",- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"},{"name": "@narratage/component-kit",@@ -617,7 +617,7 @@{"name": "@narratage/elaborator","version": "0.0.0-dev",- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"},{"name": "@narratage/film",@@ -652,17 +652,17 @@{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"},{"name": "@narratage/media-track","version": "0.0.0-dev",- "digest": "sha256:b99121eb2f0f5307c19115a00af5fc7b218440c87e4b4caeded9d4570d6547d0"+ "digest": "sha256:08af7b82b3f4182fba2f3cc4cd12aee34452c2c2cf0cdbf2c3da48422a5b5d5e"},{"name": "@narratage/media","version": "0.0.0-dev",- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"},{"name": "@narratage/model-kit",@@ -712,7 +712,7 @@{"name": "@narratage/semantic-map","version": "0.0.0-dev",- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"},{"name": "@narratage/source",@@ -722,17 +722,17 @@{"name": "@narratage/spatial","version": "0.0.0-dev",- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"},{"name": "@narratage/speech-alignment","version": "0.0.0-dev",- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"},{"name": "@narratage/speech-basis","version": "0.0.0-dev",- "digest": "sha256:01532b3694db77f9d090b6474440144bbf1245682befa00a4cd44e77e2395b0a"+ "digest": "sha256:3e6952237c7bc44fb85c873fae2850999fbb77f90e397a0f9441046aa64ad9d7"},{"name": "@narratage/speech-evidence",@@ -742,12 +742,12 @@{"name": "@narratage/speech-spine","version": "0.0.0-dev",- "digest": "sha256:84898416ea156ceefd5c090e18b560355741f95451cbf90e99e8f6a639986da1"+ "digest": "sha256:03a18dbf36764e3cd951fb46eb8badb69ead040da5f7b6d258a5bb125950f0a1"},{"name": "@narratage/speech","version": "0.0.0-dev",- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"},{"name": "@narratage/svs",@@ -757,7 +757,7 @@{"name": "@narratage/temporal","version": "0.0.0-dev",- "digest": "sha256:b20cb315f5fc365a2e9994c63989a0cc1edc46b5843e06a337827d21d74068a1"+ "digest": "sha256:6af6c0bf4e2cdb23240125815dddd831dcacfe793055150c58f4dfd37ce9e350"},{"name": "@narratage/text",@@ -782,7 +782,7 @@{"name": "@narratage/whisperx","version": "0.0.0-dev",- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"}],"packages": [@@ -801,8 +801,8 @@"name": "@narratage/caption","version": "0.0.0-dev"},- "facetsDigest": "sha256:6a2991ec75f9fd30e8d286213fc7993aae2c570b85a8faae5d3c2af661c9fdfc",- "closureDigest": "sha256:ee79cd66af6e561432e52f1bf78f597f9dc01cd09d12d68c26e7cf61e7361aac"+ "facetsDigest": "sha256:b8434ae33703de6d3c41c1426ae76083b19e398e14efe8589f2c01a92a9d4e8d",+ "closureDigest": "sha256:acd4bf7e979294b1b69fbab08a67b627be720c7e90226d94460490e76a9b4b07"},{"specifier": "@narratage/caption-fine",@@ -810,8 +810,8 @@"name": "@narratage/caption-fine","version": "0.0.0-dev"},- "facetsDigest": "sha256:c5616408dd56ce9c903523319b2a697ff6b8af5bfe1dc66fe793a11e4629f3cf",- "closureDigest": "sha256:782a9f2c0345431e06c8ab009d518933c964465b34d790b4064f87170e0495ff"+ "facetsDigest": "sha256:2fad0cbda2ae17fe67f5816e99f493d6ebfce232e2f526c5c9299a35ff0429f9",+ "closureDigest": "sha256:0aa2796ab56a70cb4e7db6fab3394cdb2d119397f3144f0f1f18cc59f30af29b"},{"specifier": "@narratage/caption-gemini",@@ -819,8 +819,8 @@"name": "@narratage/caption-gemini","version": "0.0.0-dev"},- "facetsDigest": "sha256:22fed1f1748349c11b6f28448be38cbea68b1dd5a51a3dc2d548ae0a376b9c71",- "closureDigest": "sha256:3cc6fd7d5250e5457db548fa2432f33d62012db8f77e1c8fe4f590e38a3439ab"+ "facetsDigest": "sha256:d4f1cf46a646ff5cf6294307af7f77a453a000eaf2b168099d42f41d1fbbcf50",+ "closureDigest": "sha256:4f6bddb12ebf1b61a1297d09ca1ab4080bab20523487e73f66d53be2a28f99e4"},{"specifier": "@narratage/composition",@@ -828,8 +828,8 @@"name": "@narratage/composition","version": "0.0.0-dev"},- "facetsDigest": "sha256:0a988b665baf0846246e309d9faf39ffb3ef6ec455bf37f7c438eb30429ff46e",- "closureDigest": "sha256:15c9088a176b04a6051b2398e308bde8fab6fb15d0147786e9e40abfce772662"+ "facetsDigest": "sha256:d0c91a61caa7303210d898183ae355ab6c5e0eb1de8bd5a0c237cd4388693104",+ "closureDigest": "sha256:937caaaab137467eaaef1e9974b103498a76d0b915690ffe404d800b5f84b706"},{"specifier": "@narratage/film",@@ -837,8 +837,8 @@"name": "@narratage/film","version": "0.0.0-dev"},- "facetsDigest": "sha256:8e58b5630b9a68a8ce7314620f5f1c7cda973a05c179c18d61e10d68002964f0",- "closureDigest": "sha256:2a4b2b66d9f3913bc839f052e35e6c5cd6afed08821a3308c643b16dcd011f44"+ "facetsDigest": "sha256:f6e47bdf84726556e6ee174da07b7dd435b6e7b9865065f7d7cb29ec0ed5b166",+ "closureDigest": "sha256:cfb335a738929c1153350cdc14f7a579679566a558232b0786945a41cf18eaa5"},{"specifier": "@narratage/fonts-open",@@ -846,8 +846,8 @@"name": "@narratage/fonts-open","version": "0.0.0-dev"},- "facetsDigest": "sha256:31b51352edcb4f34beccc9d9ac8cf7397125a840971a72c912df3d34d1f3feda",- "closureDigest": "sha256:4594c947a3425c6c42738e2e7d0f7ce56a6efaced91fcf6085e282c3ad98b797"+ "facetsDigest": "sha256:2117122b06d7c1cd6bee519e72aff22bc3b50f179b50da21dcc84ddf7014457f",+ "closureDigest": "sha256:65fe769fd8fe5de6a8ef1f33a63bd843ce94fc63dd9bacafecef862b2fadbc25"},{"specifier": "@narratage/generation",@@ -856,7 +856,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:d74a57a3210267e665366513324db5c313b3ca0f770da66e74cc38aee5048efb",- "closureDigest": "sha256:3bd4bfdd124001b83602627b0305d6652e96f43cbb9345e3b9bece51a3d701ce"+ "closureDigest": "sha256:dd00aba4da27ff3bfcec6e5bc24d86a07ac94344701c9936972b66ab5fc64628"},{"specifier": "@narratage/hyperframes",@@ -864,8 +864,8 @@"name": "@narratage/hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:b8b8773d03f19844cdc8d714650380145056649245e33c607f30220fff194157",- "closureDigest": "sha256:88d7a9fedb614166e149e29d064be1dca18f1c4a102759f1a2600afba4953b8b"+ "facetsDigest": "sha256:520234d0b029d7229e6595483e2f0d0227d20cf662363163975691e2471a767e",+ "closureDigest": "sha256:118b9b787b8df4bea1fbf4cfc09d65627d7670f96db1984a1ae6f0c649e5d43b"},{"specifier": "@narratage/media",@@ -873,8 +873,8 @@"name": "@narratage/media","version": "0.0.0-dev"},- "facetsDigest": "sha256:6c7cf563a92aa8e48db6da224697fd5f56c50b636b0327efc17fbbc264502b3f",- "closureDigest": "sha256:e8edbbfb7bec2efa7ad05fa273549a3b033861eb9ef20345430d9be617b26674"+ "facetsDigest": "sha256:f0974c7d88918ca4796dbb86dea24b6d29f8f5f44430c4845f9928f1df5b22e5",+ "closureDigest": "sha256:007255553cdbe2c9b82bd71ab920faaae9d107e54e6548adc5e26853505d9146"},{"specifier": "@narratage/media-pipeline",@@ -882,8 +882,8 @@"name": "@narratage/media-pipeline","version": "0.0.0-dev"},- "facetsDigest": "sha256:21f817f8e6eea9e9e4361005ed5ab9e69117fd77ec237fd467fb11afe9965019",- "closureDigest": "sha256:3555f1939c82dfa216902fd600e2616ecfce6a244e240b060080621367333d97"+ "facetsDigest": "sha256:e83eed5903110d1e8fa38aeac1487f63737f1e2a773f6e3c794dff54ad66122a",+ "closureDigest": "sha256:fde796ee9bd9d9992fe2cb37d3cf18f539d74f16e55e847a586ad013d7f87705"},{"specifier": "@narratage/media-track",@@ -891,8 +891,8 @@"name": "@narratage/media-track","version": "0.0.0-dev"},- "facetsDigest": "sha256:bb26c54c0ad8802ee8c7cd6f13314c90dd7de7e937ded1c501367e19693fe3b1",- "closureDigest": "sha256:5b8e32d627c9f55e5dbe8a4a85348b44852070f2e6796e3d19bbd7a0e42a49b6"+ "facetsDigest": "sha256:ed087e0f28c7767b3481fea40c886413bbbc9963a3963ce4480182265140c31a",+ "closureDigest": "sha256:8c82d8c186f0dd286467056cb79d7210f560b22aa20065e92836e60d0a885f95"},{"specifier": "@narratage/narrative",@@ -918,8 +918,8 @@"name": "@narratage/render-hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:aa47f04cf0652ee3fa2e114eb6d9a80b0522828a47869103ca33e95027b21634",- "closureDigest": "sha256:1c9dcb543a3afa6a94c9757fd47a3e0bfb68ed7d5f3c87b3ef0f0e52ed286332"+ "facetsDigest": "sha256:b5a0bad5b8809158caea96274f94c397e18086febcd8b11eb3f4360bb869f06e",+ "closureDigest": "sha256:67e343cebb44e6d3cfe929356e5363b317f9462e7240fc411c5bb0df87bbf17f"},{"specifier": "@narratage/run-markup",@@ -928,7 +928,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:fb3a8a5be9a0ef0aa914026a199be641f4e20df27f9b284be196f52fd260be55",- "closureDigest": "sha256:20b6fcbacf36352d6c180e1a466eaf368c0bef183eec09396456e2e2375ea16f"+ "closureDigest": "sha256:b3d5afceb67cb1275867e99f0aa384f4d339d2b759443d6822352bc8714b8bc7"},{"specifier": "@narratage/script",@@ -937,7 +937,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:69e1c08a5f07c73a506ca8286426a0386485738649e86463cca0b291f319ee4e",- "closureDigest": "sha256:696bb2e69c89d9e1079a76ce68d26fe2c99615275a4266bf24fa905ad77c1bb4"+ "closureDigest": "sha256:cc5ebec9ea7bf21d23d67536dac321618679037e811042d2b349688a373ffc4d"},{"specifier": "@narratage/seedance",@@ -945,8 +945,8 @@"name": "@narratage/seedance","version": "0.0.0-dev"},- "facetsDigest": "sha256:e889ca3808a19f544e7aa5e27453beb5a09464c645633b0b1b1d9a4b7ce6591a",- "closureDigest": "sha256:e1c45192a92620ec8ea03d9cf1b19073bae2dbfde12475f8436cd5c008c854bd"+ "facetsDigest": "sha256:0832f344b8c327e316a153d8ca4559da5eba452f3e334ffb49c7e94a29f36ecf",+ "closureDigest": "sha256:3ef4370d01ffcfef671cae12e1a250b1216caaa373b335246d2bfb5d11b148cf"},{"specifier": "@narratage/semantic-map",@@ -954,8 +954,8 @@"name": "@narratage/semantic-map","version": "0.0.0-dev"},- "facetsDigest": "sha256:a9d81ddfa09dfa23c9c3c0817b60299431ca9ec640252a659d490be55d92ba68",- "closureDigest": "sha256:3ac5deb912231d3092666c692d73a379129d1cf2ed035d43a95bef6d5101d525"+ "facetsDigest": "sha256:836a539f60d3a77a122d0bae169f03e229566dfc4696001143e9bd32c313dc12",+ "closureDigest": "sha256:a6313c3c407510323d7b5f856aaae3935f47cd80869b099b961aa34c9eb6c1ba"},{"specifier": "@narratage/spatial",@@ -964,7 +964,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:5062281ffadb46495ec12e1a67185b57fe13879e90f03bcf544e93fd1d273716",- "closureDigest": "sha256:3209c05b608c20aedab35299235974d74ee4374ead722ec125f2b6ea70829c77"+ "closureDigest": "sha256:e328230a8de70bd5c4ec4af65db9f964f0ffe20900358ba4a59fff0d29dd8ac7"},{"specifier": "@narratage/speech",@@ -972,8 +972,8 @@"name": "@narratage/speech","version": "0.0.0-dev"},- "facetsDigest": "sha256:75b1537c9b9729c2e9197445c34b295bc9f72dbef754a4702e1ca792dceea472",- "closureDigest": "sha256:a5be05750141d51e4f4460945b779bf0196548d7b7c908eca34d8c63c2ec7b70"+ "facetsDigest": "sha256:ed94af4fb62357b30f8faec7f16ccb2b2d055deecd300c7ca948b112b56b710c",+ "closureDigest": "sha256:cae66d28f2f5598824877a845f996798a35b17c84043dc261678ead81c602e86"},{"specifier": "@narratage/speech-alignment",@@ -981,8 +981,8 @@"name": "@narratage/speech-alignment","version": "0.0.0-dev"},- "facetsDigest": "sha256:bdfe219081310041d11c6d54440e6ccb80398bfb97f93afc28e9d1ed101b4a79",- "closureDigest": "sha256:de73bc9d1411af8319a099df658dfc51f6e7b44dcbf3173cb5f9db18edb449cf"+ "facetsDigest": "sha256:fe041c994c1974eddf582703869f1b8718eab817c1a0eb5e2edd0ae98644f8d9",+ "closureDigest": "sha256:8d611e7523965c7a446d95beead34533db5eb94c6ed7666d14381cf9c77629fc"},{"specifier": "@narratage/speech-basis",@@ -990,8 +990,8 @@"name": "@narratage/speech-basis","version": "0.0.0-dev"},- "facetsDigest": "sha256:f01f21badb0ccdd5cc62b131be05f46030bffee96042c7323a0c572786d012ef",- "closureDigest": "sha256:843a7922a70d5df180ade4ec3a8becf8fb110f6137fa6a38d86cc5aaa14830e1"+ "facetsDigest": "sha256:5badee4556d51ff8075eb2a4531b5bf9ff65069ce9cce724b870b21e135c2129",+ "closureDigest": "sha256:151aff0a543fef15b471e6914324594b333f68f081e9835c29bc52826be2c825"},{"specifier": "@narratage/speech-evidence",@@ -999,8 +999,8 @@"name": "@narratage/speech-evidence","version": "0.0.0-dev"},- "facetsDigest": "sha256:60f366de074a7fba20c71dcbd825083fddc54ea7de602a903eb144eab5555425",- "closureDigest": "sha256:918172938ecc1e9b6b1cff9465048bdfe2dcbfe8748971101f78f12ee36f1d75"+ "facetsDigest": "sha256:f6dfc644f0dd9975af48081fb4b1b7ac23154c4bccd2aeceff19cee05697b757",+ "closureDigest": "sha256:62f0174c12061f87acd28f1aa0d816cb1ff4c80956040737ad55f1d4a3bf4838"},{"specifier": "@narratage/speech-spine",@@ -1008,8 +1008,8 @@"name": "@narratage/speech-spine","version": "0.0.0-dev"},- "facetsDigest": "sha256:a27dd4349d6bb1aaa7f382ad09f883e9894157579e48285b00e8fa24456190f7",- "closureDigest": "sha256:f056e14a44f4f2cc0751b28e1ecb9831b7ff472034a02cac8089b9ff45d67854"+ "facetsDigest": "sha256:d20cdc013e61ea8d7a632d7cea2c10a9186a0bf0736ece2ff0f307c1320b6459",+ "closureDigest": "sha256:16eb190b79628257a3100ca81b7cbe26ccb1e4d6e95b6f93c629dafac07a2c8d"},{"specifier": "@narratage/svs",@@ -1018,7 +1018,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:9cc7c397d2cd97b5a3e814886b6553b6d236fd8825df0d056e328b7f0cf20163",- "closureDigest": "sha256:2be69621b66211f48f0f7181bda377dd9e6f9a90067e1d0640fa2d762a0f19d1"+ "closureDigest": "sha256:9358fe74dcc74e0bd730d9043292e044843e2e9eb003ee9db24e73f81ac495df"},{"specifier": "@narratage/temporal",@@ -1027,7 +1027,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:2e9c4804f142e08e87b2e11af79aa17770109fb573e865397bed14ed7a4f1d0e",- "closureDigest": "sha256:7a2df8dba92e1005388b373be07f4fe3065bb3d4b7106e19139a4fe1cadc090b"+ "closureDigest": "sha256:fb131fd446a2e565a29a1850df39e5ba0bc709ee5f128346f9dcc3b1583c4189"},{"specifier": "@narratage/text",@@ -1036,7 +1036,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:4435dd4bd38e5e017013f775e28e379e3e235211649ee8f3913378f7f9e60b58",- "closureDigest": "sha256:c6ba3301621d63598421b7d3f0dc2e1bcc8967f16624407d37a20c2c0703a246"+ "closureDigest": "sha256:9cab1e68a2273b7cf749c06033d63c2bdc28690b55fc0c755921512f0c3a701f"},{"specifier": "@narratage/typography-track",@@ -1044,8 +1044,8 @@"name": "@narratage/typography-track","version": "0.0.0-dev"},- "facetsDigest": "sha256:cc8349fec027f43e01a8832ba8a7050490e924851cee3325d3568cb5bc5fd57f",- "closureDigest": "sha256:840cec885e08123174809ef2dcf357120203b3fb5e2af3b22e0c6177d2e61af1"+ "facetsDigest": "sha256:a9a59e547f2db875f17fb52a87a1b3ff6921c87f065a5964c74be00f3cf1ac8f",+ "closureDigest": "sha256:6c8598a278b61a2deae144c2a9745e379611796a268c3a25a020e27b9df9d4c0"},{"specifier": "@narratage/visual-ir",@@ -1062,9 +1062,9 @@"name": "@narratage/whisperx","version": "0.0.0-dev"},- "facetsDigest": "sha256:c8cb487d25e567a081c74a7abc97ce2b94dc0c47c36c90ba6b44d2ec21a9e80b",- "closureDigest": "sha256:98e68de949bf7cfb9e1c483de66a38de6f347fc5032122d10e9af971a28fa534"+ "facetsDigest": "sha256:9e765c9776d31ceeb1f0898fa3df9d7345747fa0fd5a60d46728f8d78c8a69df",+ "closureDigest": "sha256:d0abf29a19cbc43aa0e0cfe239670d4e339da684f89ed5ad413b1aabea18e165"}],- "digest": "sha256:c2d6ceb3e801c9045047d59040be4e3f9a297f3a245f51cc8425ce7e5d9c3043"+ "digest": "sha256:4c4ab7a6d2674b5a3a6f0078c51539c0b995852e6bff422cf57f795f8382baa0"}diff --git a/examples/talking-film-live/main.svml b/examples/talking-film-live/main.svmlindex da813b6e..842982ba 100644--- a/examples/talking-film-live/main.svml+++ b/examples/talking-film-live/main.svml@@ -86,8 +86,10 @@</seedance:ReferenceVideo><space:Canvas id="vertical" width="480" height="854"/>+ <space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>- <speech:Spine id="speech" canvas={vertical} frame-rate="30">+ <speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={opening-take.video} segment={story.segment.opening}/><speech:Take video={answer-take.video} segment={story.segment.answer}/></speech:Spine>diff --git a/examples/talking-film-live/studio.svs b/examples/talking-film-live/studio.svsindex 92261acb..adf36b48 100644--- a/examples/talking-film-live/studio.svs+++ b/examples/talking-film-live/studio.svs@@ -5,6 +5,8 @@background: #09090B;}+ speech.visual { fit: cover; }+caption.primary {cue-min-words: 2;cue-max-words: 5;diff --git a/examples/talking-film-live/svml.packages.lock b/examples/talking-film-live/svml.packages.lockindex 71244e89..4a08bb9b 100644--- a/examples/talking-film-live/svml.packages.lock+++ b/examples/talking-film-live/svml.packages.lock@@ -598,7 +598,7 @@{"name": "@narratage/caption","version": "0.0.0-dev",- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"},{"name": "@narratage/component-kit",@@ -618,7 +618,7 @@{"name": "@narratage/elaborator","version": "0.0.0-dev",- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"},{"name": "@narratage/estimate",@@ -658,17 +658,17 @@{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"},{"name": "@narratage/media-track","version": "0.0.0-dev",- "digest": "sha256:b99121eb2f0f5307c19115a00af5fc7b218440c87e4b4caeded9d4570d6547d0"+ "digest": "sha256:08af7b82b3f4182fba2f3cc4cd12aee34452c2c2cf0cdbf2c3da48422a5b5d5e"},{"name": "@narratage/media","version": "0.0.0-dev",- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"},{"name": "@narratage/model-kit",@@ -718,7 +718,7 @@{"name": "@narratage/semantic-map","version": "0.0.0-dev",- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"},{"name": "@narratage/source",@@ -728,17 +728,17 @@{"name": "@narratage/spatial","version": "0.0.0-dev",- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"},{"name": "@narratage/speech-alignment","version": "0.0.0-dev",- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"},{"name": "@narratage/speech-basis","version": "0.0.0-dev",- "digest": "sha256:01532b3694db77f9d090b6474440144bbf1245682befa00a4cd44e77e2395b0a"+ "digest": "sha256:3e6952237c7bc44fb85c873fae2850999fbb77f90e397a0f9441046aa64ad9d7"},{"name": "@narratage/speech-evidence",@@ -748,12 +748,12 @@{"name": "@narratage/speech-spine","version": "0.0.0-dev",- "digest": "sha256:84898416ea156ceefd5c090e18b560355741f95451cbf90e99e8f6a639986da1"+ "digest": "sha256:03a18dbf36764e3cd951fb46eb8badb69ead040da5f7b6d258a5bb125950f0a1"},{"name": "@narratage/speech","version": "0.0.0-dev",- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"},{"name": "@narratage/svs",@@ -763,7 +763,7 @@{"name": "@narratage/temporal","version": "0.0.0-dev",- "digest": "sha256:b20cb315f5fc365a2e9994c63989a0cc1edc46b5843e06a337827d21d74068a1"+ "digest": "sha256:6af6c0bf4e2cdb23240125815dddd831dcacfe793055150c58f4dfd37ce9e350"},{"name": "@narratage/text",@@ -788,7 +788,7 @@{"name": "@narratage/whisperx","version": "0.0.0-dev",- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"},{"name": "cmu-pronouncing-dictionary",@@ -812,8 +812,8 @@"name": "@narratage/caption","version": "0.0.0-dev"},- "facetsDigest": "sha256:6a2991ec75f9fd30e8d286213fc7993aae2c570b85a8faae5d3c2af661c9fdfc",- "closureDigest": "sha256:ee79cd66af6e561432e52f1bf78f597f9dc01cd09d12d68c26e7cf61e7361aac"+ "facetsDigest": "sha256:b8434ae33703de6d3c41c1426ae76083b19e398e14efe8589f2c01a92a9d4e8d",+ "closureDigest": "sha256:acd4bf7e979294b1b69fbab08a67b627be720c7e90226d94460490e76a9b4b07"},{"specifier": "@narratage/caption-fine",@@ -821,8 +821,8 @@"name": "@narratage/caption-fine","version": "0.0.0-dev"},- "facetsDigest": "sha256:c5616408dd56ce9c903523319b2a697ff6b8af5bfe1dc66fe793a11e4629f3cf",- "closureDigest": "sha256:782a9f2c0345431e06c8ab009d518933c964465b34d790b4064f87170e0495ff"+ "facetsDigest": "sha256:2fad0cbda2ae17fe67f5816e99f493d6ebfce232e2f526c5c9299a35ff0429f9",+ "closureDigest": "sha256:0aa2796ab56a70cb4e7db6fab3394cdb2d119397f3144f0f1f18cc59f30af29b"},{"specifier": "@narratage/caption-gemini",@@ -830,8 +830,8 @@"name": "@narratage/caption-gemini","version": "0.0.0-dev"},- "facetsDigest": "sha256:22fed1f1748349c11b6f28448be38cbea68b1dd5a51a3dc2d548ae0a376b9c71",- "closureDigest": "sha256:3cc6fd7d5250e5457db548fa2432f33d62012db8f77e1c8fe4f590e38a3439ab"+ "facetsDigest": "sha256:d4f1cf46a646ff5cf6294307af7f77a453a000eaf2b168099d42f41d1fbbcf50",+ "closureDigest": "sha256:4f6bddb12ebf1b61a1297d09ca1ab4080bab20523487e73f66d53be2a28f99e4"},{"specifier": "@narratage/composition",@@ -839,8 +839,8 @@"name": "@narratage/composition","version": "0.0.0-dev"},- "facetsDigest": "sha256:0a988b665baf0846246e309d9faf39ffb3ef6ec455bf37f7c438eb30429ff46e",- "closureDigest": "sha256:15c9088a176b04a6051b2398e308bde8fab6fb15d0147786e9e40abfce772662"+ "facetsDigest": "sha256:d0c91a61caa7303210d898183ae355ab6c5e0eb1de8bd5a0c237cd4388693104",+ "closureDigest": "sha256:937caaaab137467eaaef1e9974b103498a76d0b915690ffe404d800b5f84b706"},{"specifier": "@narratage/estimate",@@ -848,8 +848,8 @@"name": "@narratage/estimate","version": "0.0.0-dev"},- "facetsDigest": "sha256:9975815b3109520eb6bba276554a90767a9aebff6498c2768d9faf4763219bef",- "closureDigest": "sha256:7bbdc658746e65e2bb4199b3de84d1197376f16120708e2eea31558898de662f"+ "facetsDigest": "sha256:018228974baa7623415df07f4703025f2d09449969439724027836a13e61caf1",+ "closureDigest": "sha256:9424d9a12d4fcb0b7886c82aabed10a21f6707b3e42ded94f960396818fa8412"},{"specifier": "@narratage/film",@@ -857,8 +857,8 @@"name": "@narratage/film","version": "0.0.0-dev"},- "facetsDigest": "sha256:8e58b5630b9a68a8ce7314620f5f1c7cda973a05c179c18d61e10d68002964f0",- "closureDigest": "sha256:2a4b2b66d9f3913bc839f052e35e6c5cd6afed08821a3308c643b16dcd011f44"+ "facetsDigest": "sha256:f6e47bdf84726556e6ee174da07b7dd435b6e7b9865065f7d7cb29ec0ed5b166",+ "closureDigest": "sha256:cfb335a738929c1153350cdc14f7a579679566a558232b0786945a41cf18eaa5"},{"specifier": "@narratage/fonts-open",@@ -866,8 +866,8 @@"name": "@narratage/fonts-open","version": "0.0.0-dev"},- "facetsDigest": "sha256:31b51352edcb4f34beccc9d9ac8cf7397125a840971a72c912df3d34d1f3feda",- "closureDigest": "sha256:4594c947a3425c6c42738e2e7d0f7ce56a6efaced91fcf6085e282c3ad98b797"+ "facetsDigest": "sha256:2117122b06d7c1cd6bee519e72aff22bc3b50f179b50da21dcc84ddf7014457f",+ "closureDigest": "sha256:65fe769fd8fe5de6a8ef1f33a63bd843ce94fc63dd9bacafecef862b2fadbc25"},{"specifier": "@narratage/generation",@@ -876,7 +876,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:d74a57a3210267e665366513324db5c313b3ca0f770da66e74cc38aee5048efb",- "closureDigest": "sha256:3bd4bfdd124001b83602627b0305d6652e96f43cbb9345e3b9bece51a3d701ce"+ "closureDigest": "sha256:dd00aba4da27ff3bfcec6e5bc24d86a07ac94344701c9936972b66ab5fc64628"},{"specifier": "@narratage/hyperframes",@@ -884,8 +884,8 @@"name": "@narratage/hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:b8b8773d03f19844cdc8d714650380145056649245e33c607f30220fff194157",- "closureDigest": "sha256:88d7a9fedb614166e149e29d064be1dca18f1c4a102759f1a2600afba4953b8b"+ "facetsDigest": "sha256:520234d0b029d7229e6595483e2f0d0227d20cf662363163975691e2471a767e",+ "closureDigest": "sha256:118b9b787b8df4bea1fbf4cfc09d65627d7670f96db1984a1ae6f0c649e5d43b"},{"specifier": "@narratage/media",@@ -893,8 +893,8 @@"name": "@narratage/media","version": "0.0.0-dev"},- "facetsDigest": "sha256:6c7cf563a92aa8e48db6da224697fd5f56c50b636b0327efc17fbbc264502b3f",- "closureDigest": "sha256:e8edbbfb7bec2efa7ad05fa273549a3b033861eb9ef20345430d9be617b26674"+ "facetsDigest": "sha256:f0974c7d88918ca4796dbb86dea24b6d29f8f5f44430c4845f9928f1df5b22e5",+ "closureDigest": "sha256:007255553cdbe2c9b82bd71ab920faaae9d107e54e6548adc5e26853505d9146"},{"specifier": "@narratage/media-pipeline",@@ -902,8 +902,8 @@"name": "@narratage/media-pipeline","version": "0.0.0-dev"},- "facetsDigest": "sha256:21f817f8e6eea9e9e4361005ed5ab9e69117fd77ec237fd467fb11afe9965019",- "closureDigest": "sha256:3555f1939c82dfa216902fd600e2616ecfce6a244e240b060080621367333d97"+ "facetsDigest": "sha256:e83eed5903110d1e8fa38aeac1487f63737f1e2a773f6e3c794dff54ad66122a",+ "closureDigest": "sha256:fde796ee9bd9d9992fe2cb37d3cf18f539d74f16e55e847a586ad013d7f87705"},{"specifier": "@narratage/media-track",@@ -911,8 +911,8 @@"name": "@narratage/media-track","version": "0.0.0-dev"},- "facetsDigest": "sha256:bb26c54c0ad8802ee8c7cd6f13314c90dd7de7e937ded1c501367e19693fe3b1",- "closureDigest": "sha256:5b8e32d627c9f55e5dbe8a4a85348b44852070f2e6796e3d19bbd7a0e42a49b6"+ "facetsDigest": "sha256:ed087e0f28c7767b3481fea40c886413bbbc9963a3963ce4480182265140c31a",+ "closureDigest": "sha256:8c82d8c186f0dd286467056cb79d7210f560b22aa20065e92836e60d0a885f95"},{"specifier": "@narratage/narrative",@@ -938,8 +938,8 @@"name": "@narratage/render-hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:aa47f04cf0652ee3fa2e114eb6d9a80b0522828a47869103ca33e95027b21634",- "closureDigest": "sha256:1c9dcb543a3afa6a94c9757fd47a3e0bfb68ed7d5f3c87b3ef0f0e52ed286332"+ "facetsDigest": "sha256:b5a0bad5b8809158caea96274f94c397e18086febcd8b11eb3f4360bb869f06e",+ "closureDigest": "sha256:67e343cebb44e6d3cfe929356e5363b317f9462e7240fc411c5bb0df87bbf17f"},{"specifier": "@narratage/run-markup",@@ -948,7 +948,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:fb3a8a5be9a0ef0aa914026a199be641f4e20df27f9b284be196f52fd260be55",- "closureDigest": "sha256:20b6fcbacf36352d6c180e1a466eaf368c0bef183eec09396456e2e2375ea16f"+ "closureDigest": "sha256:b3d5afceb67cb1275867e99f0aa384f4d339d2b759443d6822352bc8714b8bc7"},{"specifier": "@narratage/script",@@ -957,7 +957,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:69e1c08a5f07c73a506ca8286426a0386485738649e86463cca0b291f319ee4e",- "closureDigest": "sha256:696bb2e69c89d9e1079a76ce68d26fe2c99615275a4266bf24fa905ad77c1bb4"+ "closureDigest": "sha256:cc5ebec9ea7bf21d23d67536dac321618679037e811042d2b349688a373ffc4d"},{"specifier": "@narratage/seedance",@@ -965,8 +965,8 @@"name": "@narratage/seedance","version": "0.0.0-dev"},- "facetsDigest": "sha256:e889ca3808a19f544e7aa5e27453beb5a09464c645633b0b1b1d9a4b7ce6591a",- "closureDigest": "sha256:e1c45192a92620ec8ea03d9cf1b19073bae2dbfde12475f8436cd5c008c854bd"+ "facetsDigest": "sha256:0832f344b8c327e316a153d8ca4559da5eba452f3e334ffb49c7e94a29f36ecf",+ "closureDigest": "sha256:3ef4370d01ffcfef671cae12e1a250b1216caaa373b335246d2bfb5d11b148cf"},{"specifier": "@narratage/semantic-map",@@ -974,8 +974,8 @@"name": "@narratage/semantic-map","version": "0.0.0-dev"},- "facetsDigest": "sha256:a9d81ddfa09dfa23c9c3c0817b60299431ca9ec640252a659d490be55d92ba68",- "closureDigest": "sha256:3ac5deb912231d3092666c692d73a379129d1cf2ed035d43a95bef6d5101d525"+ "facetsDigest": "sha256:836a539f60d3a77a122d0bae169f03e229566dfc4696001143e9bd32c313dc12",+ "closureDigest": "sha256:a6313c3c407510323d7b5f856aaae3935f47cd80869b099b961aa34c9eb6c1ba"},{"specifier": "@narratage/spatial",@@ -984,7 +984,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:5062281ffadb46495ec12e1a67185b57fe13879e90f03bcf544e93fd1d273716",- "closureDigest": "sha256:3209c05b608c20aedab35299235974d74ee4374ead722ec125f2b6ea70829c77"+ "closureDigest": "sha256:e328230a8de70bd5c4ec4af65db9f964f0ffe20900358ba4a59fff0d29dd8ac7"},{"specifier": "@narratage/speech",@@ -992,8 +992,8 @@"name": "@narratage/speech","version": "0.0.0-dev"},- "facetsDigest": "sha256:75b1537c9b9729c2e9197445c34b295bc9f72dbef754a4702e1ca792dceea472",- "closureDigest": "sha256:a5be05750141d51e4f4460945b779bf0196548d7b7c908eca34d8c63c2ec7b70"+ "facetsDigest": "sha256:ed94af4fb62357b30f8faec7f16ccb2b2d055deecd300c7ca948b112b56b710c",+ "closureDigest": "sha256:cae66d28f2f5598824877a845f996798a35b17c84043dc261678ead81c602e86"},{"specifier": "@narratage/speech-alignment",@@ -1001,8 +1001,8 @@"name": "@narratage/speech-alignment","version": "0.0.0-dev"},- "facetsDigest": "sha256:bdfe219081310041d11c6d54440e6ccb80398bfb97f93afc28e9d1ed101b4a79",- "closureDigest": "sha256:de73bc9d1411af8319a099df658dfc51f6e7b44dcbf3173cb5f9db18edb449cf"+ "facetsDigest": "sha256:fe041c994c1974eddf582703869f1b8718eab817c1a0eb5e2edd0ae98644f8d9",+ "closureDigest": "sha256:8d611e7523965c7a446d95beead34533db5eb94c6ed7666d14381cf9c77629fc"},{"specifier": "@narratage/speech-basis",@@ -1010,8 +1010,8 @@"name": "@narratage/speech-basis","version": "0.0.0-dev"},- "facetsDigest": "sha256:f01f21badb0ccdd5cc62b131be05f46030bffee96042c7323a0c572786d012ef",- "closureDigest": "sha256:843a7922a70d5df180ade4ec3a8becf8fb110f6137fa6a38d86cc5aaa14830e1"+ "facetsDigest": "sha256:5badee4556d51ff8075eb2a4531b5bf9ff65069ce9cce724b870b21e135c2129",+ "closureDigest": "sha256:151aff0a543fef15b471e6914324594b333f68f081e9835c29bc52826be2c825"},{"specifier": "@narratage/speech-evidence",@@ -1019,8 +1019,8 @@"name": "@narratage/speech-evidence","version": "0.0.0-dev"},- "facetsDigest": "sha256:60f366de074a7fba20c71dcbd825083fddc54ea7de602a903eb144eab5555425",- "closureDigest": "sha256:918172938ecc1e9b6b1cff9465048bdfe2dcbfe8748971101f78f12ee36f1d75"+ "facetsDigest": "sha256:f6dfc644f0dd9975af48081fb4b1b7ac23154c4bccd2aeceff19cee05697b757",+ "closureDigest": "sha256:62f0174c12061f87acd28f1aa0d816cb1ff4c80956040737ad55f1d4a3bf4838"},{"specifier": "@narratage/speech-spine",@@ -1028,8 +1028,8 @@"name": "@narratage/speech-spine","version": "0.0.0-dev"},- "facetsDigest": "sha256:a27dd4349d6bb1aaa7f382ad09f883e9894157579e48285b00e8fa24456190f7",- "closureDigest": "sha256:f056e14a44f4f2cc0751b28e1ecb9831b7ff472034a02cac8089b9ff45d67854"+ "facetsDigest": "sha256:d20cdc013e61ea8d7a632d7cea2c10a9186a0bf0736ece2ff0f307c1320b6459",+ "closureDigest": "sha256:16eb190b79628257a3100ca81b7cbe26ccb1e4d6e95b6f93c629dafac07a2c8d"},{"specifier": "@narratage/svs",@@ -1038,7 +1038,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:9cc7c397d2cd97b5a3e814886b6553b6d236fd8825df0d056e328b7f0cf20163",- "closureDigest": "sha256:2be69621b66211f48f0f7181bda377dd9e6f9a90067e1d0640fa2d762a0f19d1"+ "closureDigest": "sha256:9358fe74dcc74e0bd730d9043292e044843e2e9eb003ee9db24e73f81ac495df"},{"specifier": "@narratage/temporal",@@ -1047,7 +1047,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:2e9c4804f142e08e87b2e11af79aa17770109fb573e865397bed14ed7a4f1d0e",- "closureDigest": "sha256:7a2df8dba92e1005388b373be07f4fe3065bb3d4b7106e19139a4fe1cadc090b"+ "closureDigest": "sha256:fb131fd446a2e565a29a1850df39e5ba0bc709ee5f128346f9dcc3b1583c4189"},{"specifier": "@narratage/text",@@ -1056,7 +1056,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:4435dd4bd38e5e017013f775e28e379e3e235211649ee8f3913378f7f9e60b58",- "closureDigest": "sha256:c6ba3301621d63598421b7d3f0dc2e1bcc8967f16624407d37a20c2c0703a246"+ "closureDigest": "sha256:9cab1e68a2273b7cf749c06033d63c2bdc28690b55fc0c755921512f0c3a701f"},{"specifier": "@narratage/typography-track",@@ -1064,8 +1064,8 @@"name": "@narratage/typography-track","version": "0.0.0-dev"},- "facetsDigest": "sha256:cc8349fec027f43e01a8832ba8a7050490e924851cee3325d3568cb5bc5fd57f",- "closureDigest": "sha256:840cec885e08123174809ef2dcf357120203b3fb5e2af3b22e0c6177d2e61af1"+ "facetsDigest": "sha256:a9a59e547f2db875f17fb52a87a1b3ff6921c87f065a5964c74be00f3cf1ac8f",+ "closureDigest": "sha256:6c8598a278b61a2deae144c2a9745e379611796a268c3a25a020e27b9df9d4c0"},{"specifier": "@narratage/visual-ir",@@ -1082,9 +1082,9 @@"name": "@narratage/whisperx","version": "0.0.0-dev"},- "facetsDigest": "sha256:c8cb487d25e567a081c74a7abc97ce2b94dc0c47c36c90ba6b44d2ec21a9e80b",- "closureDigest": "sha256:98e68de949bf7cfb9e1c483de66a38de6f347fc5032122d10e9af971a28fa534"+ "facetsDigest": "sha256:9e765c9776d31ceeb1f0898fa3df9d7345747fa0fd5a60d46728f8d78c8a69df",+ "closureDigest": "sha256:d0abf29a19cbc43aa0e0cfe239670d4e339da684f89ed5ad413b1aabea18e165"}],- "digest": "sha256:a224a78bd87743c777eadf978452e86d4cba400b1cf5075057cf0aafac26a406"+ "digest": "sha256:ff48f6c0eaef7beb70cd23ed010836d513a4e0d678c7450d164fd792b75e5288"}diff --git a/examples/talking-film-live/svml.runtime-packages.lock b/examples/talking-film-live/svml.runtime-packages.lockindex ef94a1d9..a27d165c 100644--- a/examples/talking-film-live/svml.runtime-packages.lock+++ b/examples/talking-film-live/svml.runtime-packages.lock@@ -60,12 +60,12 @@{"name": "@narratage/caption","version": "0.0.0-dev",- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"},{"name": "@narratage/compiler-node","version": "0.0.0-dev",- "digest": "sha256:4626882f405a083281ce7ffec73fa598640223669908b36715d8281fa19be92e"+ "digest": "sha256:804eac87761fe0465234647c2309b6396ed4fd0a886346afef9167d327c3bb9f"},{"name": "@narratage/component-kit",@@ -95,7 +95,7 @@{"name": "@narratage/elaborator","version": "0.0.0-dev",- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"},{"name": "@narratage/endpoint-kit",@@ -120,7 +120,7 @@{"name": "@narratage/local","version": "0.0.0-dev",- "digest": "sha256:cfbd3c13a81dbc21378bebd22662e93a201480e1bfe725f04e78534764f86c0e"+ "digest": "sha256:d2bece5a791dc3b7a88f890a54142b5e0cc5a534485ff4e20d5740787a8b90a1"},{"name": "@narratage/markup",@@ -130,17 +130,17 @@{"name": "@narratage/media-execution","version": "0.0.0-dev",- "digest": "sha256:8fe6aca65ca4ebcc1efa71376e986e97741310d836f85647fcb223f39fda9754"+ "digest": "sha256:3f07960e6723ad386b7fd8858438f2f9e72f6bab23120ffb10462944ac896020"},{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"},{"name": "@narratage/media","version": "0.0.0-dev",- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"},{"name": "@narratage/narrative",@@ -150,7 +150,7 @@{"name": "@narratage/package-loader-node","version": "0.0.0-dev",- "digest": "sha256:db78f8abd9a08a7655f7ed2d24860b1f848b3763421984847c303afa316b9003"+ "digest": "sha256:64a8daed4ebf1c561be55e4f26d2439235cf7b7d8b8fec26f995e26489415675"},{"name": "@narratage/program-space",@@ -165,12 +165,12 @@{"name": "@narratage/provider-google-vertex","version": "0.0.0-dev",- "digest": "sha256:e6f2594a0636b527fa7472c66fb3134eaa92d7e9903a8cebf0065fc7b4af4377"+ "digest": "sha256:87cc370c56a3730dd94f9d2e74481d91db25a3a9ca0483b3077e66f69d5f3c10"},{"name": "@narratage/provider-hyperframes-local","version": "0.0.0-dev",- "digest": "sha256:1d3c090bc9d13fc266b65120f3e0439be824b1d05d4746a670aa929da49a560b"+ "digest": "sha256:5e466e9bc267266768904c4bd873c2f98a5a9dffbc2dc67c545710e391a60242"},{"name": "@narratage/provider-kie",@@ -180,7 +180,7 @@{"name": "@narratage/provider-media-local","version": "0.0.0-dev",- "digest": "sha256:9bfc485ddbdeca9cd747925b4ef04f85315e41746a2cf4a99ac0d96baf237014"+ "digest": "sha256:4dcefebb3b6baf6436ba607ce434328a2d50eb88542a96f6220671715cddd9f1"},{"name": "@narratage/provider-whisperx-local",@@ -205,7 +205,7 @@{"name": "@narratage/runtime-adapter","version": "0.0.0-dev",- "digest": "sha256:7bb0b1ef43eb3fa2e67725c990b17f585a2fccd98fc71eae6c7a8da29b3e06a4"+ "digest": "sha256:b67dfc214ab9ae8cec350cee2e479aa70e7c62bf19a01d7c1d091d6e7df7c913"},{"name": "@narratage/runtime",@@ -215,7 +215,7 @@{"name": "@narratage/semantic-map","version": "0.0.0-dev",- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"},{"name": "@narratage/source",@@ -225,12 +225,12 @@{"name": "@narratage/spatial","version": "0.0.0-dev",- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"},{"name": "@narratage/speech-alignment","version": "0.0.0-dev",- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"},{"name": "@narratage/speech-evidence",@@ -240,7 +240,7 @@{"name": "@narratage/speech","version": "0.0.0-dev",- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"},{"name": "@narratage/store-sqlite",@@ -270,7 +270,7 @@{"name": "@narratage/whisperx","version": "0.0.0-dev",- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"},{"name": "@narratage/workspace-fs-node",@@ -926,7 +926,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:914018276b9218afc13940ca5c362dc95fa19d574fc07b06b6fe005c5eb30315",- "closureDigest": "sha256:fadbd7bdb4fb5af49e762cb92c1e22d0cf3e2bf10fd5f0de3d594ef3f0ca8d93"+ "closureDigest": "sha256:466bbc1b42c06f020dfe64ebd701f037771f0c61a37ce5614cdff03d4bcf0943"},{"specifier": "@narratage/credential-store-env",@@ -935,7 +935,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:ee33ba638bbf2f88eee8f34a2f3179f435a0af9ccbae3fbb7d2a81bd6aeac612",- "closureDigest": "sha256:d416ef4d798b0948dcc2cd16c962784297b6e2f20daf8626f28c5d90e37b1dec"+ "closureDigest": "sha256:7521e30257c61ecaae22198b985900b135e6348172bf1a699eae06c57782458c"},{"specifier": "@narratage/local",@@ -944,7 +944,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:6689aed79b0947c7315aef65e46806ce73da01de3d73b454e1497b8d847ffa61",- "closureDigest": "sha256:2e6704a8d53b79c750c1dc1ada0196737e13cb1e41bec5fac233abbd62b70326"+ "closureDigest": "sha256:e0730ba07fbef51517fb77ea985b83d12a42eedde998852e29db5cdb199a55c2"},{"specifier": "@narratage/provider-google-vertex",@@ -953,7 +953,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b2636492ab0b0b500d674899ab2ff14fc61c7f2b22ef44eb8f816f141b21ec4d",- "closureDigest": "sha256:fdfd1b6d25ae82b3eca28bc7f1e94ca2a4c2930f6171388afb861ca9a06990da"+ "closureDigest": "sha256:4bcb114a29f4bd429dbcea883b7a8ee0f178a30c3488d261f2485214f953acd8"},{"specifier": "@narratage/provider-hyperframes-local",@@ -962,7 +962,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b14b721bdc0c9d5d9ced1c345107c92a3a5878f3383dea2ecd3e310442c2a40c",- "closureDigest": "sha256:22819933ee6b109e794ca94cee669cddf3288c8fe5c3d4e7e2bd727fdeab8bda"+ "closureDigest": "sha256:0eb5def07621f9a75934160af6372d8c97087dd6e8434e40cb8ad4497de38218"},{"specifier": "@narratage/provider-kie",@@ -971,7 +971,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:e61ebb911a33d4c5d0e4e4ee4fb69cdb790f4ced050ef35f7472bfbd72013d9b",- "closureDigest": "sha256:90589aebf0631d19b59f258bd1e4c20638f4325f946260aa71968350e9e4efc0"+ "closureDigest": "sha256:94ed9baec91ef62d316be9296803aa26d523164cddbdcb48b92b8a743ab04a09"},{"specifier": "@narratage/provider-media-local",@@ -980,7 +980,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:701abcc923bf8a711c20b6e308d8cc8be27bf560f249b8c5a801e1dbd9cb3979",- "closureDigest": "sha256:f40546545b842197e77982ceba94c7f52f2389bad33d68ff98ad62e28eb576e1"+ "closureDigest": "sha256:d59a831fdeea751342aba28e48465d2f62b13d436dafe84bd47529aa2a7542f0"},{"specifier": "@narratage/provider-whisperx-local",@@ -989,7 +989,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:3686b84c42cc2b1bcaccf57cbdcda25a15c7d9d3778fb92d25437f9880ed8f5d",- "closureDigest": "sha256:8c03021a0c809fbc076cd0c09c797151ec6cb48d473453f48993a8807ceb216f"+ "closureDigest": "sha256:2125f3b538132060328c2435e6bda42f86e5356edef9705c954f32d162ab268f"},{"specifier": "@narratage/store-sqlite",@@ -998,8 +998,8 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:5bccf73cd953710fc31ef44672fef1bfc396fd4aa2eff4b53108e8ff87e749c3",- "closureDigest": "sha256:d512ec9a51a70ce2c020a4a1cb4d1f67b66b484615c6212a4a85918983415718"+ "closureDigest": "sha256:84f9770cc535fb1e01a038e6f626bb5d32beb83d8b2169dc0c02b914334ec4c4"}],- "digest": "sha256:2ef941c08a6fde4e0fc9b9527358186402e5bffbc262b1c35e98a6678b3f9b99"+ "digest": "sha256:f1d098a85cfd67ba365732a7635f15e45c72ea45e2eb7807f7afcac914cc2faf"}diff --git a/examples/talking-head-aroll/main.svml b/examples/talking-head-aroll/main.svmlindex 90a76028..f3465ca4 100644--- a/examples/talking-head-aroll/main.svml+++ b/examples/talking-head-aroll/main.svml@@ -152,9 +152,11 @@</seedance:ReferenceVideo><space:Canvas id="vertical" width="720" height="1280"/>+ <space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/><!-- Four equal Takes become one ordered speech coordinate space and two peer Tracks. -->- <speech:Spine id="speech" canvas={vertical} frame-rate="30">+ <speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0"><speech:Take video={hook-take.video} segment={story.segment.hook}/><speech:Take video={meeting-take.video} segment={story.segment.meeting}/><speech:Take video={evidence-take.video} segment={story.segment.evidence}/>diff --git a/examples/talking-head-aroll/studio.svs b/examples/talking-head-aroll/studio.svsindex c10c3e58..7efbf882 100644--- a/examples/talking-head-aroll/studio.svs+++ b/examples/talking-head-aroll/studio.svs@@ -21,6 +21,8 @@background: #09090B;}+ speech.visual { fit: cover; }+caption.primary {cue-min-words: 2;cue-max-words: 7;diff --git a/examples/talking-head-aroll/svml.packages.lock b/examples/talking-head-aroll/svml.packages.lockindex 8c11d380..90151134 100644--- a/examples/talking-head-aroll/svml.packages.lock+++ b/examples/talking-head-aroll/svml.packages.lock@@ -595,7 +595,7 @@{"name": "@narratage/caption","version": "0.0.0-dev",- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"},{"name": "@narratage/component-kit",@@ -615,7 +615,7 @@{"name": "@narratage/elaborator","version": "0.0.0-dev",- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"},{"name": "@narratage/estimate",@@ -655,17 +655,17 @@{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"},{"name": "@narratage/media-track","version": "0.0.0-dev",- "digest": "sha256:b99121eb2f0f5307c19115a00af5fc7b218440c87e4b4caeded9d4570d6547d0"+ "digest": "sha256:08af7b82b3f4182fba2f3cc4cd12aee34452c2c2cf0cdbf2c3da48422a5b5d5e"},{"name": "@narratage/media","version": "0.0.0-dev",- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"},{"name": "@narratage/model-kit",@@ -715,7 +715,7 @@{"name": "@narratage/semantic-map","version": "0.0.0-dev",- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"},{"name": "@narratage/source",@@ -725,17 +725,17 @@{"name": "@narratage/spatial","version": "0.0.0-dev",- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"},{"name": "@narratage/speech-alignment","version": "0.0.0-dev",- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"},{"name": "@narratage/speech-basis","version": "0.0.0-dev",- "digest": "sha256:01532b3694db77f9d090b6474440144bbf1245682befa00a4cd44e77e2395b0a"+ "digest": "sha256:3e6952237c7bc44fb85c873fae2850999fbb77f90e397a0f9441046aa64ad9d7"},{"name": "@narratage/speech-evidence",@@ -745,12 +745,12 @@{"name": "@narratage/speech-spine","version": "0.0.0-dev",- "digest": "sha256:84898416ea156ceefd5c090e18b560355741f95451cbf90e99e8f6a639986da1"+ "digest": "sha256:03a18dbf36764e3cd951fb46eb8badb69ead040da5f7b6d258a5bb125950f0a1"},{"name": "@narratage/speech","version": "0.0.0-dev",- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"},{"name": "@narratage/svs",@@ -760,7 +760,7 @@{"name": "@narratage/temporal","version": "0.0.0-dev",- "digest": "sha256:b20cb315f5fc365a2e9994c63989a0cc1edc46b5843e06a337827d21d74068a1"+ "digest": "sha256:6af6c0bf4e2cdb23240125815dddd831dcacfe793055150c58f4dfd37ce9e350"},{"name": "@narratage/text",@@ -780,7 +780,7 @@{"name": "@narratage/whisperx","version": "0.0.0-dev",- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"},{"name": "cmu-pronouncing-dictionary",@@ -804,8 +804,8 @@"name": "@narratage/caption","version": "0.0.0-dev"},- "facetsDigest": "sha256:6a2991ec75f9fd30e8d286213fc7993aae2c570b85a8faae5d3c2af661c9fdfc",- "closureDigest": "sha256:ee79cd66af6e561432e52f1bf78f597f9dc01cd09d12d68c26e7cf61e7361aac"+ "facetsDigest": "sha256:b8434ae33703de6d3c41c1426ae76083b19e398e14efe8589f2c01a92a9d4e8d",+ "closureDigest": "sha256:acd4bf7e979294b1b69fbab08a67b627be720c7e90226d94460490e76a9b4b07"},{"specifier": "@narratage/caption-fine",@@ -813,8 +813,8 @@"name": "@narratage/caption-fine","version": "0.0.0-dev"},- "facetsDigest": "sha256:c5616408dd56ce9c903523319b2a697ff6b8af5bfe1dc66fe793a11e4629f3cf",- "closureDigest": "sha256:782a9f2c0345431e06c8ab009d518933c964465b34d790b4064f87170e0495ff"+ "facetsDigest": "sha256:2fad0cbda2ae17fe67f5816e99f493d6ebfce232e2f526c5c9299a35ff0429f9",+ "closureDigest": "sha256:0aa2796ab56a70cb4e7db6fab3394cdb2d119397f3144f0f1f18cc59f30af29b"},{"specifier": "@narratage/caption-gemini",@@ -822,8 +822,8 @@"name": "@narratage/caption-gemini","version": "0.0.0-dev"},- "facetsDigest": "sha256:22fed1f1748349c11b6f28448be38cbea68b1dd5a51a3dc2d548ae0a376b9c71",- "closureDigest": "sha256:3cc6fd7d5250e5457db548fa2432f33d62012db8f77e1c8fe4f590e38a3439ab"+ "facetsDigest": "sha256:d4f1cf46a646ff5cf6294307af7f77a453a000eaf2b168099d42f41d1fbbcf50",+ "closureDigest": "sha256:4f6bddb12ebf1b61a1297d09ca1ab4080bab20523487e73f66d53be2a28f99e4"},{"specifier": "@narratage/composition",@@ -831,8 +831,8 @@"name": "@narratage/composition","version": "0.0.0-dev"},- "facetsDigest": "sha256:0a988b665baf0846246e309d9faf39ffb3ef6ec455bf37f7c438eb30429ff46e",- "closureDigest": "sha256:15c9088a176b04a6051b2398e308bde8fab6fb15d0147786e9e40abfce772662"+ "facetsDigest": "sha256:d0c91a61caa7303210d898183ae355ab6c5e0eb1de8bd5a0c237cd4388693104",+ "closureDigest": "sha256:937caaaab137467eaaef1e9974b103498a76d0b915690ffe404d800b5f84b706"},{"specifier": "@narratage/estimate",@@ -840,8 +840,8 @@"name": "@narratage/estimate","version": "0.0.0-dev"},- "facetsDigest": "sha256:9975815b3109520eb6bba276554a90767a9aebff6498c2768d9faf4763219bef",- "closureDigest": "sha256:7bbdc658746e65e2bb4199b3de84d1197376f16120708e2eea31558898de662f"+ "facetsDigest": "sha256:018228974baa7623415df07f4703025f2d09449969439724027836a13e61caf1",+ "closureDigest": "sha256:9424d9a12d4fcb0b7886c82aabed10a21f6707b3e42ded94f960396818fa8412"},{"specifier": "@narratage/film",@@ -849,8 +849,8 @@"name": "@narratage/film","version": "0.0.0-dev"},- "facetsDigest": "sha256:8e58b5630b9a68a8ce7314620f5f1c7cda973a05c179c18d61e10d68002964f0",- "closureDigest": "sha256:2a4b2b66d9f3913bc839f052e35e6c5cd6afed08821a3308c643b16dcd011f44"+ "facetsDigest": "sha256:f6e47bdf84726556e6ee174da07b7dd435b6e7b9865065f7d7cb29ec0ed5b166",+ "closureDigest": "sha256:cfb335a738929c1153350cdc14f7a579679566a558232b0786945a41cf18eaa5"},{"specifier": "@narratage/fonts-open",@@ -858,8 +858,8 @@"name": "@narratage/fonts-open","version": "0.0.0-dev"},- "facetsDigest": "sha256:31b51352edcb4f34beccc9d9ac8cf7397125a840971a72c912df3d34d1f3feda",- "closureDigest": "sha256:4594c947a3425c6c42738e2e7d0f7ce56a6efaced91fcf6085e282c3ad98b797"+ "facetsDigest": "sha256:2117122b06d7c1cd6bee519e72aff22bc3b50f179b50da21dcc84ddf7014457f",+ "closureDigest": "sha256:65fe769fd8fe5de6a8ef1f33a63bd843ce94fc63dd9bacafecef862b2fadbc25"},{"specifier": "@narratage/generation",@@ -868,7 +868,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:d74a57a3210267e665366513324db5c313b3ca0f770da66e74cc38aee5048efb",- "closureDigest": "sha256:3bd4bfdd124001b83602627b0305d6652e96f43cbb9345e3b9bece51a3d701ce"+ "closureDigest": "sha256:dd00aba4da27ff3bfcec6e5bc24d86a07ac94344701c9936972b66ab5fc64628"},{"specifier": "@narratage/hyperframes",@@ -876,8 +876,8 @@"name": "@narratage/hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:b8b8773d03f19844cdc8d714650380145056649245e33c607f30220fff194157",- "closureDigest": "sha256:88d7a9fedb614166e149e29d064be1dca18f1c4a102759f1a2600afba4953b8b"+ "facetsDigest": "sha256:520234d0b029d7229e6595483e2f0d0227d20cf662363163975691e2471a767e",+ "closureDigest": "sha256:118b9b787b8df4bea1fbf4cfc09d65627d7670f96db1984a1ae6f0c649e5d43b"},{"specifier": "@narratage/media",@@ -885,8 +885,8 @@"name": "@narratage/media","version": "0.0.0-dev"},- "facetsDigest": "sha256:6c7cf563a92aa8e48db6da224697fd5f56c50b636b0327efc17fbbc264502b3f",- "closureDigest": "sha256:e8edbbfb7bec2efa7ad05fa273549a3b033861eb9ef20345430d9be617b26674"+ "facetsDigest": "sha256:f0974c7d88918ca4796dbb86dea24b6d29f8f5f44430c4845f9928f1df5b22e5",+ "closureDigest": "sha256:007255553cdbe2c9b82bd71ab920faaae9d107e54e6548adc5e26853505d9146"},{"specifier": "@narratage/media-pipeline",@@ -894,8 +894,8 @@"name": "@narratage/media-pipeline","version": "0.0.0-dev"},- "facetsDigest": "sha256:21f817f8e6eea9e9e4361005ed5ab9e69117fd77ec237fd467fb11afe9965019",- "closureDigest": "sha256:3555f1939c82dfa216902fd600e2616ecfce6a244e240b060080621367333d97"+ "facetsDigest": "sha256:e83eed5903110d1e8fa38aeac1487f63737f1e2a773f6e3c794dff54ad66122a",+ "closureDigest": "sha256:fde796ee9bd9d9992fe2cb37d3cf18f539d74f16e55e847a586ad013d7f87705"},{"specifier": "@narratage/narrative",@@ -921,8 +921,8 @@"name": "@narratage/render-hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:aa47f04cf0652ee3fa2e114eb6d9a80b0522828a47869103ca33e95027b21634",- "closureDigest": "sha256:1c9dcb543a3afa6a94c9757fd47a3e0bfb68ed7d5f3c87b3ef0f0e52ed286332"+ "facetsDigest": "sha256:b5a0bad5b8809158caea96274f94c397e18086febcd8b11eb3f4360bb869f06e",+ "closureDigest": "sha256:67e343cebb44e6d3cfe929356e5363b317f9462e7240fc411c5bb0df87bbf17f"},{"specifier": "@narratage/run-markup",@@ -931,7 +931,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:fb3a8a5be9a0ef0aa914026a199be641f4e20df27f9b284be196f52fd260be55",- "closureDigest": "sha256:20b6fcbacf36352d6c180e1a466eaf368c0bef183eec09396456e2e2375ea16f"+ "closureDigest": "sha256:b3d5afceb67cb1275867e99f0aa384f4d339d2b759443d6822352bc8714b8bc7"},{"specifier": "@narratage/script",@@ -940,7 +940,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:69e1c08a5f07c73a506ca8286426a0386485738649e86463cca0b291f319ee4e",- "closureDigest": "sha256:696bb2e69c89d9e1079a76ce68d26fe2c99615275a4266bf24fa905ad77c1bb4"+ "closureDigest": "sha256:cc5ebec9ea7bf21d23d67536dac321618679037e811042d2b349688a373ffc4d"},{"specifier": "@narratage/seedance",@@ -948,8 +948,8 @@"name": "@narratage/seedance","version": "0.0.0-dev"},- "facetsDigest": "sha256:e889ca3808a19f544e7aa5e27453beb5a09464c645633b0b1b1d9a4b7ce6591a",- "closureDigest": "sha256:e1c45192a92620ec8ea03d9cf1b19073bae2dbfde12475f8436cd5c008c854bd"+ "facetsDigest": "sha256:0832f344b8c327e316a153d8ca4559da5eba452f3e334ffb49c7e94a29f36ecf",+ "closureDigest": "sha256:3ef4370d01ffcfef671cae12e1a250b1216caaa373b335246d2bfb5d11b148cf"},{"specifier": "@narratage/semantic-map",@@ -957,8 +957,8 @@"name": "@narratage/semantic-map","version": "0.0.0-dev"},- "facetsDigest": "sha256:a9d81ddfa09dfa23c9c3c0817b60299431ca9ec640252a659d490be55d92ba68",- "closureDigest": "sha256:3ac5deb912231d3092666c692d73a379129d1cf2ed035d43a95bef6d5101d525"+ "facetsDigest": "sha256:836a539f60d3a77a122d0bae169f03e229566dfc4696001143e9bd32c313dc12",+ "closureDigest": "sha256:a6313c3c407510323d7b5f856aaae3935f47cd80869b099b961aa34c9eb6c1ba"},{"specifier": "@narratage/spatial",@@ -967,7 +967,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:5062281ffadb46495ec12e1a67185b57fe13879e90f03bcf544e93fd1d273716",- "closureDigest": "sha256:3209c05b608c20aedab35299235974d74ee4374ead722ec125f2b6ea70829c77"+ "closureDigest": "sha256:e328230a8de70bd5c4ec4af65db9f964f0ffe20900358ba4a59fff0d29dd8ac7"},{"specifier": "@narratage/speech",@@ -975,8 +975,8 @@"name": "@narratage/speech","version": "0.0.0-dev"},- "facetsDigest": "sha256:75b1537c9b9729c2e9197445c34b295bc9f72dbef754a4702e1ca792dceea472",- "closureDigest": "sha256:a5be05750141d51e4f4460945b779bf0196548d7b7c908eca34d8c63c2ec7b70"+ "facetsDigest": "sha256:ed94af4fb62357b30f8faec7f16ccb2b2d055deecd300c7ca948b112b56b710c",+ "closureDigest": "sha256:cae66d28f2f5598824877a845f996798a35b17c84043dc261678ead81c602e86"},{"specifier": "@narratage/speech-alignment",@@ -984,8 +984,8 @@"name": "@narratage/speech-alignment","version": "0.0.0-dev"},- "facetsDigest": "sha256:bdfe219081310041d11c6d54440e6ccb80398bfb97f93afc28e9d1ed101b4a79",- "closureDigest": "sha256:de73bc9d1411af8319a099df658dfc51f6e7b44dcbf3173cb5f9db18edb449cf"+ "facetsDigest": "sha256:fe041c994c1974eddf582703869f1b8718eab817c1a0eb5e2edd0ae98644f8d9",+ "closureDigest": "sha256:8d611e7523965c7a446d95beead34533db5eb94c6ed7666d14381cf9c77629fc"},{"specifier": "@narratage/speech-basis",@@ -993,8 +993,8 @@"name": "@narratage/speech-basis","version": "0.0.0-dev"},- "facetsDigest": "sha256:f01f21badb0ccdd5cc62b131be05f46030bffee96042c7323a0c572786d012ef",- "closureDigest": "sha256:843a7922a70d5df180ade4ec3a8becf8fb110f6137fa6a38d86cc5aaa14830e1"+ "facetsDigest": "sha256:5badee4556d51ff8075eb2a4531b5bf9ff65069ce9cce724b870b21e135c2129",+ "closureDigest": "sha256:151aff0a543fef15b471e6914324594b333f68f081e9835c29bc52826be2c825"},{"specifier": "@narratage/speech-evidence",@@ -1002,8 +1002,8 @@"name": "@narratage/speech-evidence","version": "0.0.0-dev"},- "facetsDigest": "sha256:60f366de074a7fba20c71dcbd825083fddc54ea7de602a903eb144eab5555425",- "closureDigest": "sha256:918172938ecc1e9b6b1cff9465048bdfe2dcbfe8748971101f78f12ee36f1d75"+ "facetsDigest": "sha256:f6dfc644f0dd9975af48081fb4b1b7ac23154c4bccd2aeceff19cee05697b757",+ "closureDigest": "sha256:62f0174c12061f87acd28f1aa0d816cb1ff4c80956040737ad55f1d4a3bf4838"},{"specifier": "@narratage/speech-spine",@@ -1011,8 +1011,8 @@"name": "@narratage/speech-spine","version": "0.0.0-dev"},- "facetsDigest": "sha256:a27dd4349d6bb1aaa7f382ad09f883e9894157579e48285b00e8fa24456190f7",- "closureDigest": "sha256:f056e14a44f4f2cc0751b28e1ecb9831b7ff472034a02cac8089b9ff45d67854"+ "facetsDigest": "sha256:d20cdc013e61ea8d7a632d7cea2c10a9186a0bf0736ece2ff0f307c1320b6459",+ "closureDigest": "sha256:16eb190b79628257a3100ca81b7cbe26ccb1e4d6e95b6f93c629dafac07a2c8d"},{"specifier": "@narratage/svs",@@ -1021,7 +1021,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:9cc7c397d2cd97b5a3e814886b6553b6d236fd8825df0d056e328b7f0cf20163",- "closureDigest": "sha256:2be69621b66211f48f0f7181bda377dd9e6f9a90067e1d0640fa2d762a0f19d1"+ "closureDigest": "sha256:9358fe74dcc74e0bd730d9043292e044843e2e9eb003ee9db24e73f81ac495df"},{"specifier": "@narratage/text",@@ -1030,7 +1030,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:4435dd4bd38e5e017013f775e28e379e3e235211649ee8f3913378f7f9e60b58",- "closureDigest": "sha256:c6ba3301621d63598421b7d3f0dc2e1bcc8967f16624407d37a20c2c0703a246"+ "closureDigest": "sha256:9cab1e68a2273b7cf749c06033d63c2bdc28690b55fc0c755921512f0c3a701f"},{"specifier": "@narratage/visual-ir",@@ -1047,9 +1047,9 @@"name": "@narratage/whisperx","version": "0.0.0-dev"},- "facetsDigest": "sha256:c8cb487d25e567a081c74a7abc97ce2b94dc0c47c36c90ba6b44d2ec21a9e80b",- "closureDigest": "sha256:98e68de949bf7cfb9e1c483de66a38de6f347fc5032122d10e9af971a28fa534"+ "facetsDigest": "sha256:9e765c9776d31ceeb1f0898fa3df9d7345747fa0fd5a60d46728f8d78c8a69df",+ "closureDigest": "sha256:d0abf29a19cbc43aa0e0cfe239670d4e339da684f89ed5ad413b1aabea18e165"}],- "digest": "sha256:828557259e164093a055111cff451aaf48d1d8519cc02352804066c5841347ee"+ "digest": "sha256:c88ce3ec0a540451c5be1d4d1357d2a7ca82b036c6922c6455a55a908a8009c6"}diff --git a/examples/talking-head-aroll/svml.runtime-packages.lock b/examples/talking-head-aroll/svml.runtime-packages.lockindex ef94a1d9..a27d165c 100644--- a/examples/talking-head-aroll/svml.runtime-packages.lock+++ b/examples/talking-head-aroll/svml.runtime-packages.lock@@ -60,12 +60,12 @@{"name": "@narratage/caption","version": "0.0.0-dev",- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"},{"name": "@narratage/compiler-node","version": "0.0.0-dev",- "digest": "sha256:4626882f405a083281ce7ffec73fa598640223669908b36715d8281fa19be92e"+ "digest": "sha256:804eac87761fe0465234647c2309b6396ed4fd0a886346afef9167d327c3bb9f"},{"name": "@narratage/component-kit",@@ -95,7 +95,7 @@{"name": "@narratage/elaborator","version": "0.0.0-dev",- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"},{"name": "@narratage/endpoint-kit",@@ -120,7 +120,7 @@{"name": "@narratage/local","version": "0.0.0-dev",- "digest": "sha256:cfbd3c13a81dbc21378bebd22662e93a201480e1bfe725f04e78534764f86c0e"+ "digest": "sha256:d2bece5a791dc3b7a88f890a54142b5e0cc5a534485ff4e20d5740787a8b90a1"},{"name": "@narratage/markup",@@ -130,17 +130,17 @@{"name": "@narratage/media-execution","version": "0.0.0-dev",- "digest": "sha256:8fe6aca65ca4ebcc1efa71376e986e97741310d836f85647fcb223f39fda9754"+ "digest": "sha256:3f07960e6723ad386b7fd8858438f2f9e72f6bab23120ffb10462944ac896020"},{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"},{"name": "@narratage/media","version": "0.0.0-dev",- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"},{"name": "@narratage/narrative",@@ -150,7 +150,7 @@{"name": "@narratage/package-loader-node","version": "0.0.0-dev",- "digest": "sha256:db78f8abd9a08a7655f7ed2d24860b1f848b3763421984847c303afa316b9003"+ "digest": "sha256:64a8daed4ebf1c561be55e4f26d2439235cf7b7d8b8fec26f995e26489415675"},{"name": "@narratage/program-space",@@ -165,12 +165,12 @@{"name": "@narratage/provider-google-vertex","version": "0.0.0-dev",- "digest": "sha256:e6f2594a0636b527fa7472c66fb3134eaa92d7e9903a8cebf0065fc7b4af4377"+ "digest": "sha256:87cc370c56a3730dd94f9d2e74481d91db25a3a9ca0483b3077e66f69d5f3c10"},{"name": "@narratage/provider-hyperframes-local","version": "0.0.0-dev",- "digest": "sha256:1d3c090bc9d13fc266b65120f3e0439be824b1d05d4746a670aa929da49a560b"+ "digest": "sha256:5e466e9bc267266768904c4bd873c2f98a5a9dffbc2dc67c545710e391a60242"},{"name": "@narratage/provider-kie",@@ -180,7 +180,7 @@{"name": "@narratage/provider-media-local","version": "0.0.0-dev",- "digest": "sha256:9bfc485ddbdeca9cd747925b4ef04f85315e41746a2cf4a99ac0d96baf237014"+ "digest": "sha256:4dcefebb3b6baf6436ba607ce434328a2d50eb88542a96f6220671715cddd9f1"},{"name": "@narratage/provider-whisperx-local",@@ -205,7 +205,7 @@{"name": "@narratage/runtime-adapter","version": "0.0.0-dev",- "digest": "sha256:7bb0b1ef43eb3fa2e67725c990b17f585a2fccd98fc71eae6c7a8da29b3e06a4"+ "digest": "sha256:b67dfc214ab9ae8cec350cee2e479aa70e7c62bf19a01d7c1d091d6e7df7c913"},{"name": "@narratage/runtime",@@ -215,7 +215,7 @@{"name": "@narratage/semantic-map","version": "0.0.0-dev",- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"},{"name": "@narratage/source",@@ -225,12 +225,12 @@{"name": "@narratage/spatial","version": "0.0.0-dev",- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"},{"name": "@narratage/speech-alignment","version": "0.0.0-dev",- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"},{"name": "@narratage/speech-evidence",@@ -240,7 +240,7 @@{"name": "@narratage/speech","version": "0.0.0-dev",- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"},{"name": "@narratage/store-sqlite",@@ -270,7 +270,7 @@{"name": "@narratage/whisperx","version": "0.0.0-dev",- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"},{"name": "@narratage/workspace-fs-node",@@ -926,7 +926,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:914018276b9218afc13940ca5c362dc95fa19d574fc07b06b6fe005c5eb30315",- "closureDigest": "sha256:fadbd7bdb4fb5af49e762cb92c1e22d0cf3e2bf10fd5f0de3d594ef3f0ca8d93"+ "closureDigest": "sha256:466bbc1b42c06f020dfe64ebd701f037771f0c61a37ce5614cdff03d4bcf0943"},{"specifier": "@narratage/credential-store-env",@@ -935,7 +935,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:ee33ba638bbf2f88eee8f34a2f3179f435a0af9ccbae3fbb7d2a81bd6aeac612",- "closureDigest": "sha256:d416ef4d798b0948dcc2cd16c962784297b6e2f20daf8626f28c5d90e37b1dec"+ "closureDigest": "sha256:7521e30257c61ecaae22198b985900b135e6348172bf1a699eae06c57782458c"},{"specifier": "@narratage/local",@@ -944,7 +944,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:6689aed79b0947c7315aef65e46806ce73da01de3d73b454e1497b8d847ffa61",- "closureDigest": "sha256:2e6704a8d53b79c750c1dc1ada0196737e13cb1e41bec5fac233abbd62b70326"+ "closureDigest": "sha256:e0730ba07fbef51517fb77ea985b83d12a42eedde998852e29db5cdb199a55c2"},{"specifier": "@narratage/provider-google-vertex",@@ -953,7 +953,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b2636492ab0b0b500d674899ab2ff14fc61c7f2b22ef44eb8f816f141b21ec4d",- "closureDigest": "sha256:fdfd1b6d25ae82b3eca28bc7f1e94ca2a4c2930f6171388afb861ca9a06990da"+ "closureDigest": "sha256:4bcb114a29f4bd429dbcea883b7a8ee0f178a30c3488d261f2485214f953acd8"},{"specifier": "@narratage/provider-hyperframes-local",@@ -962,7 +962,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b14b721bdc0c9d5d9ced1c345107c92a3a5878f3383dea2ecd3e310442c2a40c",- "closureDigest": "sha256:22819933ee6b109e794ca94cee669cddf3288c8fe5c3d4e7e2bd727fdeab8bda"+ "closureDigest": "sha256:0eb5def07621f9a75934160af6372d8c97087dd6e8434e40cb8ad4497de38218"},{"specifier": "@narratage/provider-kie",@@ -971,7 +971,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:e61ebb911a33d4c5d0e4e4ee4fb69cdb790f4ced050ef35f7472bfbd72013d9b",- "closureDigest": "sha256:90589aebf0631d19b59f258bd1e4c20638f4325f946260aa71968350e9e4efc0"+ "closureDigest": "sha256:94ed9baec91ef62d316be9296803aa26d523164cddbdcb48b92b8a743ab04a09"},{"specifier": "@narratage/provider-media-local",@@ -980,7 +980,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:701abcc923bf8a711c20b6e308d8cc8be27bf560f249b8c5a801e1dbd9cb3979",- "closureDigest": "sha256:f40546545b842197e77982ceba94c7f52f2389bad33d68ff98ad62e28eb576e1"+ "closureDigest": "sha256:d59a831fdeea751342aba28e48465d2f62b13d436dafe84bd47529aa2a7542f0"},{"specifier": "@narratage/provider-whisperx-local",@@ -989,7 +989,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:3686b84c42cc2b1bcaccf57cbdcda25a15c7d9d3778fb92d25437f9880ed8f5d",- "closureDigest": "sha256:8c03021a0c809fbc076cd0c09c797151ec6cb48d473453f48993a8807ceb216f"+ "closureDigest": "sha256:2125f3b538132060328c2435e6bda42f86e5356edef9705c954f32d162ab268f"},{"specifier": "@narratage/store-sqlite",@@ -998,8 +998,8 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:5bccf73cd953710fc31ef44672fef1bfc396fd4aa2eff4b53108e8ff87e749c3",- "closureDigest": "sha256:d512ec9a51a70ce2c020a4a1cb4d1f67b66b484615c6212a4a85918983415718"+ "closureDigest": "sha256:84f9770cc535fb1e01a038e6f626bb5d32beb83d8b2169dc0c02b914334ec4c4"}],- "digest": "sha256:2ef941c08a6fde4e0fc9b9527358186402e5bffbc262b1c35e98a6678b3f9b99"+ "digest": "sha256:f1d098a85cfd67ba365732a7635f15e45c72ea45e2eb7807f7afcac914cc2faf"}diff --git a/package.json b/package.jsonindex e75e28c5..902a0286 100644--- a/package.json+++ b/package.json@@ -9,7 +9,8 @@"docs:dev": "vitepress dev docs","docs:build": "vitepress build docs","docs:preview": "vitepress preview docs",- "test": "node --import tsx --test 'packages/*/test/**/*.test.ts' 'services/*/test/**/*.test.ts' 'test/**/*.test.ts' 'tools/*.test.mjs'",+ "test": "node --import tsx --test 'packages/*/test/**/*.test.ts' 'services/*/test/**/*.test.ts' 'test/**/*.test.ts'",+ "test:release": "node --import tsx --test tools/repository-hygiene.test.mjs tools/recipe-defaults.test.mjs","test:whisperx-service": "uv run --project services/whisperx --frozen python -m unittest discover -s services/whisperx/tests -p 'test_*.py'","test:image-opencv": "SVML_OPENCV_TESTS=1 SVML_OPENCV_PYTHON=services/image-opencv/.venv/bin/python node --import tsx --test packages/provider-image-opencv-local/test/provider.test.ts","smoke:kie": "node --import tsx packages/provider-kie/test/live-smoke.ts",diff --git a/packages/audio-track/src/program.ts b/packages/audio-track/src/program.tsindex 85e63664..0b914d0a 100644--- a/packages/audio-track/src/program.ts+++ b/packages/audio-track/src/program.ts@@ -1,6 +1,6 @@import { assertAudioTrackIdentity, sealAudioTrack } from "@narratage/composition";import type { AudioClip, AudioTrack } from "@narratage/composition";-import { verifySynchronizedMedia } from "@narratage/media";+import { synchronizedMediaSampleFrames, verifySynchronizedMedia } from "@narratage/media";import type { SynchronizedMedia } from "@narratage/media";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";import {@@ -114,10 +114,9 @@ export function assertAudioTrackSet(value: AudioTrackSet): void {function sourceFacts(media: SynchronizedMedia): AudioItemProgram["source"] {verifySynchronizedMedia(media);assert(media.audio !== undefined, "Audio Track source has no explicitly normalized audio member.");- const { artifact, sampleFrames } = media.audio;return {- artifact: structuredClone(artifact),- sampleFrames,+ artifact: structuredClone(media.audio.artifact),+ sampleFrames: synchronizedMediaSampleFrames(media),};}diff --git a/packages/audio-track/test/audio-track.test.ts b/packages/audio-track/test/audio-track.test.tsindex b837516a..95db5533 100644--- a/packages/audio-track/test/audio-track.test.ts+++ b/packages/audio-track/test/audio-track.test.ts@@ -53,29 +53,11 @@ function media(id: string, sampleFrames: number): SynchronizedMedia {return {contract: "svml.synchronized-media@1",timeline: {- spanAuthority: "audio",frameRate: { numerator: 30, denominator: 1 },frameCount: Math.max(1, Math.round(sampleFrames / 1_600)),- sampleRate: 48_000,- sampleFrames,- },- sourceMap: {- sourceOriginPts: { ticks: "0", timeBase: { numerator: 1, denominator: 48_000 } },- sourceEndPts: { ticks: String(sampleFrames), timeBase: { numerator: 1, denominator: 48_000 } },- audioTrimStartSamples: 0,- audioTrimEndSamples: 0,- audioHeadSamples: 0,- audioContentSamples: sampleFrames,- audioTailSamples: 0,},audio: {artifact: { kind: "blob", digest: digestOf(`audio:${id}`), size: sampleFrames * 4, mediaType: "audio/wav" },- sourceStreamIndex: 0,- codec: "pcm_s16le",- sampleRate: 48_000,- channels: 2,- sampleFrames,- loudness: "preserved",},};}diff --git a/packages/caption/test/caption.test.ts b/packages/caption/test/caption.test.tsindex 10da5469..587997c6 100644--- a/packages/caption/test/caption.test.ts+++ b/packages/caption/test/caption.test.ts@@ -45,13 +45,7 @@ function locate(narrative: Narrative, durationSec: number, segments: readonly Alcontract: "svml.speech-basis@1",programSpace: space,audio,- visualTrack: { clips: basisSegments.map((segment) => ({- segmentId: segment.segmentId,- artifact: { kind: "blob", digest: digestOf(segment.segmentId), size: 1, mediaType: "video/mp4" },- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },- frameRate: { ...space.frameRate },- frameCount: Math.round((segment.endSec - segment.startSec) * 30),- })) },+ visualTrack: { clips: [] },segments: basisSegments,});const evidence = sealAlignedTranscriptEvidence({ contract: "svml.aligned-transcript-evidence@1", durationSec, segments });diff --git a/packages/elaborator/test/fragment.test.ts b/packages/elaborator/test/fragment.test.tsindex bd2a4854..0cd08916 100644--- a/packages/elaborator/test/fragment.test.ts+++ b/packages/elaborator/test/fragment.test.ts@@ -140,7 +140,7 @@ function speechFragment(): GraphFragment {{id: "visual",producer: speechBasisProducers.projectVisual,- inputs: { basis: operation("generate"), canvas: input("canvas") },+ inputs: { basis: operation("generate") },result: { kind: "output", name: "visual" },},],@@ -163,7 +163,7 @@ function speechFragment(): GraphFragment {name: "visual",type: compositionTypes.visualTrack,root: operation("visual"),- semanticInputs: ["request", "style", "canvas"],+ semanticInputs: ["request", "style"],fidelity: "exact",},],diff --git a/packages/media-execution/src/execute.ts b/packages/media-execution/src/execute.tsindex d13a3254..d74c8caa 100644--- a/packages/media-execution/src/execute.ts+++ b/packages/media-execution/src/execute.ts@@ -260,8 +260,6 @@ function stream<T extends MediaStream["kind"]>(type NormalizationPlan = {readonly video?: MediaVideoStream;readonly audio?: MediaAudioStream;- readonly origin: MediaTimestamp;- readonly end: MediaTimestamp;readonly frameCount: number;readonly sampleFrames: number;readonly audioTrimStartSamples: number;@@ -300,8 +298,6 @@ function normalizationPlan(if (audio === undefined) {return {...(video === undefined ? {} : { video }),- origin: authority.startPts,- end: authority.endPts,frameCount,sampleFrames,audioTrimStartSamples: 0,@@ -320,8 +316,6 @@ function normalizationPlan(return {...(video === undefined ? {} : { video }),audio,- origin: authority.startPts,- end: authority.endPts,frameCount,sampleFrames,audioTrimStartSamples: Math.max(0, -audioStart),@@ -749,41 +743,19 @@ export async function executeNormalizeMedia(const media = sealSynchronizedMedia({contract: "svml.synchronized-media@1",timeline: {- spanAuthority: need.selection.spanAuthority,frameRate: need.frameRate,frameCount: plan.frameCount,- sampleRate: 48_000,- sampleFrames: plan.sampleFrames,- },- sourceMap: {- sourceOriginPts: plan.origin,- sourceEndPts: plan.end,- audioTrimStartSamples: plan.audioTrimStartSamples,- audioTrimEndSamples: plan.audioTrimEndSamples,- audioHeadSamples: plan.audioHeadSamples,- audioContentSamples: plan.audioContentSamples,- audioTailSamples: plan.audioTailSamples,},...(visualArtifact === undefined ? {} : {visual: {artifact: visualArtifact,- sourceStreamIndex: plan.video!.index,width: visualWidth!,height: visualHeight!,- frameRate: need.frameRate,- frameCount: plan.frameCount,- muted: true as const,},}),...(audioArtifact === undefined ? {} : {audio: {artifact: audioArtifact,- sourceStreamIndex: plan.audio!.index,- codec: "pcm_s16le" as const,- sampleRate: 48_000 as const,- channels: 2 as const,- sampleFrames: plan.sampleFrames,- loudness: "preserved" as const,},}),});diff --git a/packages/media-pipeline/README.md b/packages/media-pipeline/README.mdindex 4aad1ac0..b7338559 100644--- a/packages/media-pipeline/README.md+++ b/packages/media-pipeline/README.md@@ -23,7 +23,10 @@ Segment identities.The current normalization profile uses one source presentation origin, a requested rational video framerate, and a 48 kHz stereo PCM render stem. It preserves input level: loudness/mastering remains a-separate author policy.+separate author policy. Its public `SynchronizedMedia` waist carries only the common frame domain,+optional visual Artifact plus intrinsic extent, and optional audio Artifact. Stream indexes,+authority choice and the trim/pad ledger remain in Selection and execution instead of travelling+through every consumer.This package is the shared execution vocabulary, not mandatory authoring boilerplate. A consumingSurface that already owns the needed policy may expand the same finite graph itself. For example,diff --git a/packages/media-track/README.md b/packages/media-track/README.mdindex c2a24957..27f11dbe 100644--- a/packages/media-track/README.md+++ b/packages/media-track/README.md@@ -5,9 +5,10 @@ Official provider-free Media Item and Sequence authoring package.`Track` accepts explicit ProgramSpace, CanvasSpace and, when semantic points are used, aCompleteSemanticMap. Each independently timed `Item` receives an explicit SpatialFrame and eitherone direct source or ordered Paint/sample layers. A visual source is always named by its actual-form: `image`, generated/raw `video`, prepared `media`, or compositable `surface`. Raw video is-automatically inspected and normalized to the connected ProgramSpace; `audio="include"` opts its-audio into the same normalization. `Sequence` owns an ordered replacement surface with explicit+form: `image`, generated/raw `video`, prepared `media`, or compositable `surface`. For raw video the+Track Surface declares the connected ProgramSpace as its consumption policy and expands ordinary+Media Pipeline inspect/select/normalize Operations; it does not execute normalization itself.+`audio="include"` opts audio into that same graph. `Sequence` owns an ordered replacement surface with explicitactivation points and pairwise Handoffs. Package-owned Recipes cover fitting, source occupancy,frame Paint, clipping, borders, shadows, lifecycle motion and sampling motion.diff --git a/packages/media-track/src/author.ts b/packages/media-track/src/author.tsindex b10b109d..54b936f2 100644--- a/packages/media-track/src/author.ts+++ b/packages/media-track/src/author.ts@@ -1,5 +1,5 @@import type { SvsRecipe } from "@narratage/svs";-import { sealContentFit } from "@narratage/spatial";+import { decodeContentFitProperties } from "@narratage/spatial";import type { ContentFit } from "@narratage/spatial";import {@@ -125,14 +125,7 @@ const FRAME_KEYS = ["stack-order", "clip", "radius", "padding", "border-width",const MOTION_KEYS = ["enter", "enter-frames", "enter-easing", "enter-direction", "enter-amount", "enter-origin", "sustain", "exit", "exit-frames", "exit-easing", "exit-direction", "exit-amount", "exit-origin"] as const;export function decodeMediaFit(recipe: SvsRecipe): ContentFit {- return sealContentFit({- contract: "svml.content-fit@1",- sizing: oneOf(recipe, "fit", ["contain", "cover", "fit-width", "fit-height", "native", "scale-down", "stretch"] as const, "contain"),- framePoint: { x: number(recipe, "frame-x", 0.5), y: number(recipe, "frame-y", 0.5) },- contentPoint: { x: number(recipe, "content-x", 0.5), y: number(recipe, "content-y", 0.5) },- offsetPx: { x: number(recipe, "fit-offset-x", 0), y: number(recipe, "fit-offset-y", 0) },- constraint: oneOf(recipe, "fit-constraint", ["bounded", "free"] as const, "bounded"),- });+ return decodeContentFitProperties(recipe.properties, `Media Recipe ${recipe.path}`);}function occupancy(recipe: SvsRecipe): MediaVisualOccupancy {diff --git a/packages/media-track/src/component.ts b/packages/media-track/src/component.tsindex 21c05518..2bac8700 100644--- a/packages/media-track/src/component.ts+++ b/packages/media-track/src/component.ts@@ -1,6 +1,6 @@import type { ComponentPackage, ProducerHandlerContext } from "@narratage/component-kit";import type { CompositableSurfaceRef, SynchronizedMedia } from "@narratage/media";-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";import type { ProgramSpace } from "@narratage/program-space";import { canonicalize } from "@narratage/protocol";import type { BlobRef, StoredValue } from "@narratage/protocol";@@ -21,6 +21,7 @@ import {appendMediaSequenceUntilSelection,appendMomentMediaItem,appendProgramMediaItem,+ appendSegmentMediaItem,appendSelectionMediaItem,bindMediaItemClipPath,bindMediaSequenceClipPath,@@ -140,6 +141,12 @@ export const mediaTrackComponent = {inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), value.spec, value.sounds)) }, needs: {} };} },+ { producer: mediaTrackProducers.appendSegmentItem, implementationDigest: mediaTrackImplementationDigests.appendSegmentItem, handler: ({ inputs }) => {+ const value = itemInputs(inputs);+ return { outputs: { set: output(appendSegmentMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<NarrativeExcerpt>(inputs.segment?.value, "NarrativeExcerpt"), value.spec, value.sounds)) }, needs: {} };+ } },{ producer: mediaTrackProducers.appendMomentItem, implementationDigest: mediaTrackImplementationDigests.appendMomentItem, handler: ({ inputs }) => {const value = itemInputs(inputs);return { outputs: { set: output(appendMomentMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,diff --git a/packages/media-track/src/index.ts b/packages/media-track/src/index.tsindex a80b05d9..84b3bb3b 100644--- a/packages/media-track/src/index.ts+++ b/packages/media-track/src/index.ts@@ -73,6 +73,7 @@ export {appendMediaSequenceUntilProgramEnd,appendMediaSequenceUntilSelection,appendProgramMediaItem,+ appendSegmentMediaItem,appendSelectionMediaItem,bindMediaItemClipPath,bindMediaSequenceClipPath,diff --git a/packages/media-track/src/layers.ts b/packages/media-track/src/layers.tsindex c797172a..c3f65968 100644--- a/packages/media-track/src/layers.ts+++ b/packages/media-track/src/layers.ts@@ -1,5 +1,6 @@import {assertCompositableSurfaceRef,+ synchronizedMediaSampleFrames,verifySynchronizedMedia,} from "@narratage/media";import type { CompositableSurfaceRef, SynchronizedMedia } from "@narratage/media";@@ -276,11 +277,11 @@ export function appendTimedMediaLayer(widthPx: media.visual.width,heightPx: media.visual.height,},- frameRate: { ...media.visual.frameRate },- frameCount: media.visual.frameCount,+ frameRate: { ...media.timeline.frameRate },+ frameCount: media.timeline.frameCount,...(media.audio === undefined ? {} : { audio: {artifact: structuredClone(media.audio.artifact),- sampleFrames: media.audio.sampleFrames,+ sampleFrames: synchronizedMediaSampleFrames(media),} }),};return append(set, sampleLayer(source, fit, spec));diff --git a/packages/media-track/src/manifest.ts b/packages/media-track/src/manifest.tsindex efc20fbb..1e85512d 100644--- a/packages/media-track/src/manifest.ts+++ b/packages/media-track/src/manifest.ts@@ -50,6 +50,7 @@ export const mediaTrackProducers = {createSet: { module: mediaTrackModuleRef, name: "create-media-track-set" },appendProgramItem: { module: mediaTrackModuleRef, name: "append-program-media-item" },appendSelectionItem: { module: mediaTrackModuleRef, name: "append-selection-media-item" },+ appendSegmentItem: { module: mediaTrackModuleRef, name: "append-segment-media-item" },appendMomentItem: { module: mediaTrackModuleRef, name: "append-moment-media-item" },bindItemClipPath: { module: mediaTrackModuleRef, name: "bind-media-item-clip-path" },bindSequenceClipPath: { module: mediaTrackModuleRef, name: "bind-media-sequence-clip-path" },@@ -150,7 +151,7 @@ const signedDuration: ValueSchema = { kind: "oneOf", variants: [object({ unit: { schema: { kind: "literal", value: "seconds" } }, numerator: { schema: integer }, denominator: { schema: positiveInteger } }),] };const point: ValueSchema = { kind: "oneOf", variants: [- ...["program.start", "program.end", "selection.start", "selection.end", "moment.cue"].map((ref) => object({+ ...["program.start", "program.end", "selection.start", "selection.end", "segment.start", "segment.end", "moment.cue"].map((ref) => object({ref: { schema: { kind: "literal", value: ref } }, offset: { schema: signedDuration, optional: true },})),object({ ref: { schema: { kind: "literal", value: "absolute" } }, at: { schema: duration } }),@@ -343,6 +344,7 @@ export const mediaTrackManifest: ModuleManifest = {{ name: mediaTrackProducers.createSet.name, inputs: [], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/create-set", digest: mediaTrackImplementationDigests.createSet } },{ name: mediaTrackProducers.appendProgramItem.name, inputs: [...itemInputs], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-program-item", digest: mediaTrackImplementationDigests.appendProgramItem } },{ name: mediaTrackProducers.appendSelectionItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-selection-item", digest: mediaTrackImplementationDigests.appendSelectionItem } },+ { name: mediaTrackProducers.appendSegmentItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "segment", type: narrativeTypes.excerpt }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-segment-item", digest: mediaTrackImplementationDigests.appendSegmentItem } },{ name: mediaTrackProducers.appendMomentItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-moment-item", digest: mediaTrackImplementationDigests.appendMomentItem } },{ name: mediaTrackProducers.bindItemClipPath.name, inputs: [{ name: "spec", type: mediaTrackTypes.itemSpec }, { name: "path", type: spatialTypes.path }], outputs: [{ name: "spec", type: mediaTrackTypes.itemSpec }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/bind-item-clip-path", digest: mediaTrackImplementationDigests.bindItemClipPath } },{ name: mediaTrackProducers.bindSequenceClipPath.name, inputs: [{ name: "spec", type: mediaTrackTypes.sequenceSpec }, { name: "path", type: spatialTypes.path }], outputs: [{ name: "spec", type: mediaTrackTypes.sequenceSpec }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/bind-sequence-clip-path", digest: mediaTrackImplementationDigests.bindSequenceClipPath } },diff --git a/packages/media-track/src/program.ts b/packages/media-track/src/program.tsindex 929c86e4..399fedab 100644--- a/packages/media-track/src/program.ts+++ b/packages/media-track/src/program.ts@@ -5,7 +5,7 @@ import {sealVisualTrack,} from "@narratage/composition";import type { AudioClip, AudioTrack, VisualTrack } from "@narratage/composition";-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";import {assertProgramSpaceIdentity,programFrameSampleBoundary,@@ -24,6 +24,7 @@ import {locateSelectionOccurrences,projectMomentWindows,projectProgramWindow,+ projectSegmentWindow,projectSelectionWindows,} from "@narratage/temporal";import type { ProjectedOccurrence, TemporalDuration, TemporalPointExpression } from "@narratage/temporal";@@ -74,6 +75,7 @@ export const mediaTrackImplementationDigests = {createSet: digestOf("@narratage/media-track/create-set@1"),appendProgramItem: digestOf("@narratage/media-track/append-program-item@1"),appendSelectionItem: digestOf("@narratage/media-track/append-selection-item@1"),+ appendSegmentItem: digestOf("@narratage/media-track/append-segment-item@1"),appendMomentItem: digestOf("@narratage/media-track/append-moment-item@1"),bindItemClipPath: digestOf("@narratage/media-track/bind-item-clip-path@1"),bindSequenceClipPath: digestOf("@narratage/media-track/bind-sequence-clip-path@1"),@@ -130,7 +132,7 @@ function assertDuration(value: TemporalDuration, label: string, signed: boolean)}function assertPoint(value: TemporalPointExpression, label: string): void {- assert(["program.start", "program.end", "selection.start", "selection.end", "moment.cue", "absolute"].includes(value.ref),+ assert(["program.start", "program.end", "selection.start", "selection.end", "segment.start", "segment.end", "moment.cue", "absolute"].includes(value.ref),`${label}.ref is invalid.`);if (value.ref === "absolute") assertDuration(value.at, `${label}.at`, false);else if (value.offset !== undefined) assertDuration(value.offset, `${label}.offset`, true);@@ -272,6 +274,24 @@ export function appendSelectionMediaItem(}));}+export function appendSegmentMediaItem(+ set: MediaTrackSet,+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ layers: MediaLayerSet,+ frame: MediaItemProgram["frame"],+ map: CompleteSemanticMap,+ segment: NarrativeExcerpt,+ spec: MediaItemSpec,+ sounds: MediaSoundSet,+): MediaTrackSet {+ assert(spec.expansion.kind === "one", `Segment Media Item ${spec.id} must use one occurrence.`);+ return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, [projectSegmentWindow({+ itemId: spec.id, map, segment, space, projection: spec.projection,+ })]);+}+export function appendMomentMediaItem(set: MediaTrackSet,header: MediaTrackHeader,diff --git a/packages/media-track/src/restricted-speech.ts b/packages/media-track/src/restricted-speech.tsindex 02ea0da0..fc051b6a 100644--- a/packages/media-track/src/restricted-speech.ts+++ b/packages/media-track/src/restricted-speech.ts@@ -2,8 +2,8 @@ import type { VisualPresent } from "@narratage/composition";import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";import type { BlobRef } from "@narratage/protocol";-import { assertCanvasSpace, assertContentFit, assertIntrinsicExtent } from "@narratage/spatial";-import type { CanvasSpace, ContentFit, IntrinsicExtent } from "@narratage/spatial";+import { assertContentFit, assertIntrinsicExtent, assertSpatialFrame } from "@narratage/spatial";+import type { ContentFit, IntrinsicExtent, SpatialFrame } from "@narratage/spatial";import { lowerMediaItemElements } from "./lower.js";import type { MediaItemProgram } from "./types.js";@@ -15,29 +15,32 @@ export type RestrictedSpeechVisualClip = {readonly extent: IntrinsicExtent;readonly frameRate: { readonly numerator: number; readonly denominator: number };readonly frameCount: number;+ readonly frame: SpatialFrame;+ readonly fit: ContentFit;+ readonly stackingOrder: number;};/**- * Focused reuse point for Speech Spine: one normalized muted take, one full-Canvas- * foreground layer and no Media author semantics beyond the shared lowering laws.+ * Focused reuse point for Speech Spine: one normalized muted take and one+ * explicitly placed foreground layer, without Media Track motion or sequence+ * semantics.*/export function lowerRestrictedSpeechVisualPresents(trackId: string,space: ProgramSpace,- canvas: CanvasSpace,clips: readonly RestrictedSpeechVisualClip[],- fit: ContentFit,): readonly VisualPresent[] {assertProgramSpaceIdentity(space);- assertCanvasSpace(canvas);- assertContentFit(fit);- if (!trackId || clips.length === 0) throw new Error("Restricted Speech visual requires an id and clips.");+ if (!trackId) throw new Error("Restricted Speech visual requires an id.");const totalFrames = programSpaceFrameCount(space);return clips.map((clip, index) => {if (clip.artifact.kind !== "blob" || !clip.artifact.mediaType.startsWith("video/")) {throw new Error(`Speech visual ${clip.id} is not a normalized moving-video Blob.`);}assertIntrinsicExtent(clip.extent);+ assertSpatialFrame(clip.frame);+ assertContentFit(clip.fit);+ if (!Number.isSafeInteger(clip.stackingOrder)) throw new Error(`Speech visual ${clip.id} has an invalid stacking order.`);if (clip.frameRate.numerator !== space.frameRate.numerator|| clip.frameRate.denominator !== space.frameRate.denominator) {throw new Error(`Speech visual ${clip.id} is not normalized to ProgramSpace frame rate.`);@@ -52,13 +55,7 @@ export function lowerRestrictedSpeechVisualPresents(id: `${trackId}:${clip.id}`,sourceOccurrenceId: clip.id,span: { ...clip.span },- frame: {- contract: "svml.spatial-frame@1",- xPx: 0,- yPx: 0,- widthPx: canvas.widthPx,- heightPx: canvas.heightPx,- },+ frame: structuredClone(clip.frame),presentation: {clip: { kind: "frame" },padding: { topPx: 0, rightPx: 0, bottomPx: 0, leftPx: 0 },@@ -74,7 +71,7 @@ export function lowerRestrictedSpeechVisualPresents(frameRate: { ...clip.frameRate },frameCount: clip.frameCount,},- fit: structuredClone(fit),+ fit: structuredClone(clip.fit),occupancy: { mode: "once", align: "start" },appearance: {opacity: 1,@@ -83,7 +80,7 @@ export function lowerRestrictedSpeechVisualPresents(}],motion: { sustain: [] },stacking: {- order: 0,+ order: clip.stackingOrder,tieBreak: `${trackId}:${String(index + 1).padStart(4, "0")}:${clip.id}`,},sounds: [],diff --git a/packages/media-track/src/sounds.ts b/packages/media-track/src/sounds.tsindex 93ffb940..e592c4cd 100644--- a/packages/media-track/src/sounds.ts+++ b/packages/media-track/src/sounds.ts@@ -1,4 +1,4 @@-import { verifySynchronizedMedia } from "@narratage/media";+import { synchronizedMediaSampleFrames, verifySynchronizedMedia } from "@narratage/media";import type { SynchronizedMedia } from "@narratage/media";import { canonicalize } from "@narratage/protocol";@@ -62,7 +62,7 @@ export function appendMediaSound(trigger: structuredClone(spec.trigger),source: {artifact: structuredClone(media.audio.artifact),- sampleFrames: media.audio.sampleFrames,+ sampleFrames: synchronizedMediaSampleFrames(media),},gain: spec.gain,}],diff --git a/packages/media-track/src/surface.ts b/packages/media-track/src/surface.tsindex 304c1676..93df0f21 100644--- a/packages/media-track/src/surface.ts+++ b/packages/media-track/src/surface.ts@@ -160,7 +160,7 @@ function negate(value: TemporalDuration): TemporalDuration {function point(value: string, label: string): TemporalPointExpression {const trimmed = value.trim();- const refs = ["program.start", "program.end", "selection.start", "selection.end", "moment.cue"] as const;+ const refs = ["program.start", "program.end", "selection.start", "selection.end", "segment.start", "segment.end", "moment.cue"] as const;for (const ref of refs) {if (trimmed === ref) return { ref };const match = new RegExp(`^${ref.replace(".", "\\.")}\\s*([+-])\\s*(.+)$`, "u").exec(trimmed);@@ -173,7 +173,7 @@ function point(value: string, label: string): TemporalPointExpression {}type TemporalBinding = {- readonly kind: "program" | "selection" | "moment";+ readonly kind: "program" | "selection" | "segment" | "moment";readonly projection: TemporalWindowProjection;readonly source?: SurfaceResolvedReference;};@@ -187,6 +187,7 @@ function temporalBinding(const start = optionalText(element, "start");const end = optionalText(element, "end");const selection = element.attributes.selection;+ const segment = element.attributes.segment;const moment = element.attributes.moment;const forms = Number(during !== undefined) + Number(at !== undefined) + Number(start !== undefined || end !== undefined);if (forms !== 1) throw new Error(`${element.name} requires exactly one of during, at/for, or start/end.`);@@ -195,10 +196,15 @@ function temporalBinding(if (during.trim() !== "program") throw new Error(`${element.name}.during text must be program.`);return { kind: "program", projection: { start: { ref: "program.start" }, end: { ref: "program.end" } } };}- return {+ const source = oneOfReferences(during, `${element.name}.during`, [narrativeTypes.selection, narrativeTypes.excerpt], resolve);+ return sameType(source.type, narrativeTypes.selection) ? {kind: "selection",- source: reference(during, `${element.name}.during`, narrativeTypes.selection, resolve),+ source,projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },+ } : {+ kind: "segment",+ source,+ projection: { start: { ref: "segment.start" }, end: { ref: "segment.end" } },};}if (at !== undefined) {@@ -210,7 +216,9 @@ function temporalBinding(};}if (start === undefined || end === undefined) throw new Error(`${element.name} explicit timing requires start and end.`);- if (selection !== undefined && moment !== undefined) throw new Error(`${element.name} cannot bind Selection and Moment together.`);+ if (Number(selection !== undefined) + Number(segment !== undefined) + Number(moment !== undefined) > 1) {+ throw new Error(`${element.name} cannot bind Selection, Segment and Moment together.`);+ }if (selection !== undefined) return {kind: "selection",source: reference(selection, `${element.name}.selection`, narrativeTypes.selection, resolve),@@ -221,6 +229,11 @@ function temporalBinding(source: reference(moment, `${element.name}.moment`, narrativeTypes.moment, resolve),projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) },};+ if (segment !== undefined) return {+ kind: "segment",+ source: reference(segment, `${element.name}.segment`, narrativeTypes.excerpt, resolve),+ projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) },+ };return { kind: "program", projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) } };}@@ -361,7 +374,8 @@ function createMediaTrackSurfaceFragment(inputTypes: readonly { readonly name: soperations.push({id: appendId,producer: item.binding === "program" ? mediaTrackProducers.appendProgramItem- : item.binding === "selection" ? mediaTrackProducers.appendSelectionItem : mediaTrackProducers.appendMomentItem,+ : item.binding === "selection" ? mediaTrackProducers.appendSelectionItem+ : item.binding === "segment" ? mediaTrackProducers.appendSegmentItem : mediaTrackProducers.appendMomentItem,inputs: item.binding === "program" ? common : {...common, map: input("map"), [item.binding]: input(item.sourceName!),},@@ -729,7 +743,7 @@ export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, resconst itemSuffix = suffix(itemIndex);allowed(child, ["id", "image", "video", "media", "surface", "extent", "audio", "frame", "appearance", "motion", "source-audio", "audio-gain",- "clip", "during", "at", "for", "start", "end", "selection", "moment", "occurrences",+ "clip", "during", "at", "for", "start", "end", "selection", "segment", "moment", "occurrences",]);const id = optionalText(child, "id") ?? `${trackId}.item.${itemSuffix}`;const appearance = recipe(reference(child.attributes.appearance, `${child.name}.appearance`, svsRecipeType, resolveReference), `${child.name}.appearance`);diff --git a/packages/media-track/test/media-track.test.ts b/packages/media-track/test/media-track.test.tsindex 44f7fe6c..05479e58 100644--- a/packages/media-track/test/media-track.test.ts+++ b/packages/media-track/test/media-track.test.ts@@ -16,6 +16,7 @@ import {appendMediaSequenceMember,appendMomentMediaItem,appendProgramMediaItem,+ appendSegmentMediaItem,appendSelectionMediaItem,appendStillMediaLayer,appendSurfaceMediaLayer,@@ -139,38 +140,16 @@ function timed(id = "timed", withAudio = true): SynchronizedMedia {return {contract: "svml.synchronized-media@1",timeline: {- spanAuthority: "video",frameRate: { numerator: 30, denominator: 1 },frameCount: 60,- sampleRate: 48_000,- sampleFrames,- },- sourceMap: {- sourceOriginPts: { ticks: "0", timeBase: { numerator: 1, denominator: 30 } },- sourceEndPts: { ticks: "60", timeBase: { numerator: 1, denominator: 30 } },- audioTrimStartSamples: 0,- audioTrimEndSamples: 0,- audioHeadSamples: 0,- audioContentSamples: withAudio ? sampleFrames : 0,- audioTailSamples: 0,},visual: {artifact: { kind: "blob", digest: digestOf(`video:${id}`), size: 10_000, mediaType: "video/mp4" },- sourceStreamIndex: 0,width: 720,height: 1280,- frameRate: { numerator: 30, denominator: 1 },- frameCount: 60,- muted: true,},...(withAudio ? { audio: {artifact: { kind: "blob" as const, digest: digestOf(`audio:${id}`), size: sampleFrames * 4, mediaType: "audio/wav" },- sourceStreamIndex: 1,- codec: "pcm_s16le" as const,- sampleRate: 48_000 as const,- channels: 2 as const,- sampleFrames,- loudness: "preserved" as const,} } : {}),};}@@ -559,6 +538,28 @@ test("Media Items consume the shared one/each temporal algebra without becomingassert.deepEqual(absolute.items[0]?.span, { startFrame: 7, endFrameExclusive: 19 });});+test("a Media Item can consume one whole Narrative Segment without a synthetic Selection", () => {+ const semanticMap: CompleteSemanticMap = {+ contract: "svml.complete-semantic-map@1",+ tokens: [],+ anchors: [+ { identity: "segment:answer:start", timeSec: 1, frame: 30 },+ { identity: "segment:answer:end", timeSec: 3, frame: 90 },+ ],+ };+ const result = appendSegmentMediaItem(+ createMediaTrackSet(), header, space, canvas, stillLayers(), frame,+ semanticMap,+ { contract: "svml.narrative-excerpt@1", kind: "segment", id: "answer", tokenStart: 0, tokenEndExclusive: 1 },+ itemSpec({+ id: "whole-answer",+ projection: { start: { ref: "segment.start" }, end: { ref: "segment.end" } },+ }),+ createMediaSoundSet(),+ );+ assert.deepEqual(result.items.map((item) => item.span), [{ startFrame: 30, endFrameExclusive: 90 }]);+});+test("every declared lifecycle and sustain operator lowers, and outside-canvas motion resolves from explicit geometry", () => {for (const operator of ["fade", "slide", "scale", "pop", "bounce", "blur-reveal", "wipe", "flip", "spin"] as const) {const direction = ["slide", "wipe", "flip"].includes(operator) ? { direction: "up" as const } : {};@@ -709,6 +710,7 @@ test("the Media author Surface emits explicit graph edges for layers, semantic t["surface", plain("surface", mediaTypes.compositableSurface)],["sfx", plain("sfx", mediaTypes.synchronized)],["selection", plain("selection", narrativeTypes.selection)],+ ["answer-segment", plain("answer-segment", narrativeTypes.excerpt)],["terminal", plain("terminal", narrativeTypes.selection)],["cue1", plain("cue1", narrativeTypes.moment)],["cue2", plain("cue2", narrativeTypes.moment)],@@ -723,6 +725,7 @@ test("the Media author Surface emits explicit graph edges for layers, semantic t]);const root = node("media:Track", { id: "editorial", space: ref("space"), canvas: ref("canvas"), map: ref("map") }, [node("media:Item", { id: "still-card", image: ref("still"), extent: ref("extent"), frame: ref("frame"), clip: ref("clip-path"), appearance: ref("still-style"), during: "program" }),+ node("media:Item", { id: "segment-card", image: ref("still"), extent: ref("extent"), frame: ref("frame"), appearance: ref("still-style"), during: ref("answer-segment") }),node("media:Item", { id: "proof", frame: ref("frame"), appearance: ref("card-style"), motion: ref("motion"), during: ref("selection"), "source-audio": "video", "audio-gain": "0.8" }, [node("media:Paint", { id: "backing", appearance: ref("paint-style") }),node("media:Layer", { id: "video", video: ref("raw-video"), audio: "include", appearance: ref("video-style") }, [@@ -749,6 +752,7 @@ test("the Media author Surface emits explicit graph edges for layers, semantic tconst fragment = result.fragments[0]!;const producers = fragment.operations.map((entry) => entry.producer.name);assert.ok(producers.includes("append-still-media-layer"));+ assert.ok(producers.includes("append-segment-media-item"));assert.ok(producers.includes("append-timed-media-layer"));assert.ok(producers.includes(mediaPipelineProducers.bindAvRequest.name));assert.ok(producers.includes(mediaPipelineProducers.inspect.name));@@ -765,9 +769,9 @@ test("the Media author Surface emits explicit graph edges for layers, semantic tassert.ok(fragment.inputs.some((entry) => entry.type.name === narrativeTypes.selection.name));assert.ok(fragment.inputs.some((entry) => entry.type.name === spatialTypes.path.name));const itemSpecs = result.records.filter((entry) => entry.type.name === "MediaItemSpec");- assert.equal(itemSpecs.length, 2);- const selected = itemSpecs[1]!.value.kind === "inline"- ? itemSpecs[1]!.value.value as unknown as MediaItemSpec+ assert.equal(itemSpecs.length, 3);+ const selected = itemSpecs[2]!.value.kind === "inline"+ ? itemSpecs[2]!.value.value as unknown as MediaItemSpec: undefined;assert.equal(selected?.sourceAudio?.fromLayer, "video");assert.ok(mediaTrackManifest.surfaces.some((surface) => surface.name === "track"));diff --git a/packages/media/README.md b/packages/media/README.mdindex 85429efe..b8e3db9f 100644--- a/packages/media/README.md+++ b/packages/media/README.md@@ -11,3 +11,8 @@ for one primary face followed by exact glyph fallbacks. Media declarations do noinspect media, call a Provider or promote declared audio to speech. The consuming author packagedecides whether an audio artifact is a voice reference, soundtrack, evidence source or somethingelse, and which visual component uses an exact Font.++The same module owns the deliberately narrow `SynchronizedMedia` contract used after technical+normalization. It contains one common `frameRate`/`frameCount`, an optional visual Artifact with its+intrinsic pixel extent, and an optional audio Artifact. Source stream indexes, selection authority,+normalization ledgers and repeated codec/rate constants are not downstream media fields.diff --git a/packages/media/src/identity.ts b/packages/media/src/identity.tsindex 5363e929..abe6ad57 100644--- a/packages/media/src/identity.ts+++ b/packages/media/src/identity.ts@@ -159,62 +159,33 @@ export function sealSynchronizedMedia(value: SynchronizedMedia): SynchronizedMedreturn canonicalize(value) as unknown as SynchronizedMedia;}+/** Exact 48 kHz sample span implied by one normalized frame domain. */+export function synchronizedMediaSampleFrames(value: SynchronizedMedia): number {+ const numerator = BigInt(value.timeline.frameCount) * 48_000n * BigInt(value.timeline.frameRate.denominator);+ const denominator = BigInt(value.timeline.frameRate.numerator);+ const rounded = (numerator * 2n + denominator) / (denominator * 2n);+ assert(rounded > 0n && rounded <= BigInt(Number.MAX_SAFE_INTEGER),+ "SynchronizedMedia frame domain has an invalid sample span");+ return Number(rounded);+}+export function verifySynchronizedMedia(value: unknown): asserts value is SynchronizedMedia {const item = object(value, "SynchronizedMedia") as unknown as SynchronizedMedia;assert(item.contract === "svml.synchronized-media@1", "SynchronizedMedia contract is invalid");- assert(item.timeline.spanAuthority === "video" || item.timeline.spanAuthority === "audio",- "SynchronizedMedia authority is invalid");verifyRational(item.timeline.frameRate, "SynchronizedMedia.timeline.frameRate");positiveInteger(item.timeline.frameCount, "SynchronizedMedia.timeline.frameCount");- assert(item.timeline.sampleRate === 48_000, "SynchronizedMedia sample rate must be 48000");- positiveInteger(item.timeline.sampleFrames, "SynchronizedMedia.timeline.sampleFrames");- verifyTimestamp(item.sourceMap.sourceOriginPts, "SynchronizedMedia.sourceMap.sourceOriginPts");- verifyTimestamp(item.sourceMap.sourceEndPts, "SynchronizedMedia.sourceMap.sourceEndPts");- assert(compareTimestamp(item.sourceMap.sourceOriginPts, item.sourceMap.sourceEndPts) < 0,- "SynchronizedMedia source interval is empty or reversed");- nonNegativeInteger(item.sourceMap.audioTrimStartSamples, "SynchronizedMedia.sourceMap.audioTrimStartSamples");- nonNegativeInteger(item.sourceMap.audioTrimEndSamples, "SynchronizedMedia.sourceMap.audioTrimEndSamples");- nonNegativeInteger(item.sourceMap.audioHeadSamples, "SynchronizedMedia.sourceMap.audioHeadSamples");- nonNegativeInteger(item.sourceMap.audioContentSamples, "SynchronizedMedia.sourceMap.audioContentSamples");- nonNegativeInteger(item.sourceMap.audioTailSamples, "SynchronizedMedia.sourceMap.audioTailSamples");+ synchronizedMediaSampleFrames(item);assert(item.visual !== undefined || item.audio !== undefined, "SynchronizedMedia contains no media projection");if (item.visual !== undefined) {verifyBlob(item.visual.artifact, "SynchronizedMedia.visual.artifact");assert(item.visual.artifact.mediaType.startsWith("video/"), "SynchronizedMedia visual artifact is not video");- nonNegativeInteger(item.visual.sourceStreamIndex, "SynchronizedMedia.visual.sourceStreamIndex");positiveInteger(item.visual.width, "SynchronizedMedia.visual.width");positiveInteger(item.visual.height, "SynchronizedMedia.visual.height");- verifyRational(item.visual.frameRate, "SynchronizedMedia.visual.frameRate");- assert(item.visual.frameRate.numerator === item.timeline.frameRate.numerator- && item.visual.frameRate.denominator === item.timeline.frameRate.denominator,- "SynchronizedMedia visual frame rate differs from its timeline");- assert(item.visual.frameCount === item.timeline.frameCount, "SynchronizedMedia visual frame count differs");- assert(item.visual.muted === true, "SynchronizedMedia visual must be muted");}if (item.audio !== undefined) {verifyBlob(item.audio.artifact, "SynchronizedMedia.audio.artifact");assert(item.audio.artifact.mediaType === "audio/wav", "SynchronizedMedia audio artifact must be WAV");- nonNegativeInteger(item.audio.sourceStreamIndex, "SynchronizedMedia.audio.sourceStreamIndex");- assert(item.audio.codec === "pcm_s16le" && item.audio.sampleRate === 48_000 && item.audio.channels === 2,- "SynchronizedMedia audio shape is invalid");- assert(item.audio.sampleFrames === item.timeline.sampleFrames,- "SynchronizedMedia audio sample count differs from its timeline");- assert(item.audio.loudness === "preserved", "SynchronizedMedia must not silently normalize loudness");- assert(item.sourceMap.audioHeadSamples + item.sourceMap.audioContentSamples + item.sourceMap.audioTailSamples- === item.timeline.sampleFrames,- "SynchronizedMedia audio transform does not cover its complete timeline");- } else {- assert(item.sourceMap.audioTrimStartSamples === 0- && item.sourceMap.audioTrimEndSamples === 0- && item.sourceMap.audioHeadSamples === 0- && item.sourceMap.audioContentSamples === 0- && item.sourceMap.audioTailSamples === 0,- "SynchronizedMedia without audio has an audio transform");}- assert(item.timeline.spanAuthority !== "video" || item.visual !== undefined,- "SynchronizedMedia video authority has no visual projection");- assert(item.timeline.spanAuthority !== "audio" || item.audio !== undefined,- "SynchronizedMedia audio authority has no audio projection");}export function sealRenderedVisual(value: RenderedVisual): RenderedVisual {diff --git a/packages/media/src/index.ts b/packages/media/src/index.tsindex 7c248428..774077a6 100644--- a/packages/media/src/index.ts+++ b/packages/media/src/index.ts@@ -4,7 +4,7 @@ export {decodeMediaImageSurface,} from "./surface.js";export { mediaComponent } from "./component.js";-export { sealMediaInspection, sealMediaStreamSelection, sealMuxedMedia, sealRenderedVisual, sealSynchronizedMedia, sealTimelineAudio, verifyMediaInspection, verifyMediaStreamSelection, verifyMuxedMedia, verifyRenderedVisual, verifySynchronizedMedia, verifyTimelineAudio } from "./identity.js";+export { sealMediaInspection, sealMediaStreamSelection, sealMuxedMedia, sealRenderedVisual, sealSynchronizedMedia, sealTimelineAudio, synchronizedMediaSampleFrames, verifyMediaInspection, verifyMediaStreamSelection, verifyMuxedMedia, verifyRenderedVisual, verifySynchronizedMedia, verifyTimelineAudio } from "./identity.js";export { mediaDependency, mediaManifest, mediaManifestDigest, mediaModuleRef, mediaSurfaceImplementationDigests, mediaTypes, mediaValidatorDigests } from "./manifest.js";export { compositableSurfaceSchema, fontArtifactSchema, fontStackSchema, mediaArtifactSchema, mediaInspectionSchema, mediaStreamSelectionSchema, muxedMediaSchema, renderedVisualSchema, synchronizedMediaSchema, timelineAudioSchema } from "./schema.js";export { assertCompositableSurfaceRef, assertFontArtifactRef, assertFontStackRef } from "./render.js";diff --git a/packages/media/src/schema.ts b/packages/media/src/schema.tsindex 98f76296..027bcdc3 100644--- a/packages/media/src/schema.ts+++ b/packages/media/src/schema.ts@@ -43,17 +43,12 @@ export const mediaStreamSelectionSchema: ValueSchema = object({});export const synchronizedMediaSchema: ValueSchema = object({contract: { schema: { kind: "literal", value: "svml.synchronized-media@1" } },- timeline: { schema: object({ spanAuthority: { schema: { kind: "string", enum: ["video", "audio"] } }, frameRate: { schema: rational },- frameCount: { schema: { kind: "number", integer: true, minimum: 1 } }, sampleRate: { schema: { kind: "literal", value: 48_000 } },- sampleFrames: { schema: { kind: "number", integer: true, minimum: 1 } } }) },- sourceMap: { schema: object({ sourceOriginPts: { schema: timestamp }, sourceEndPts: { schema: timestamp }, audioTrimStartSamples: { schema: integer },- audioTrimEndSamples: { schema: integer }, audioHeadSamples: { schema: integer }, audioContentSamples: { schema: integer }, audioTailSamples: { schema: integer } }) },- visual: { schema: object({ artifact: { schema: blobArtifactSchema() }, sourceStreamIndex: { schema: integer },+ timeline: { schema: object({ frameRate: { schema: rational },+ frameCount: { schema: { kind: "number", integer: true, minimum: 1 } } }) },+ visual: { schema: object({ artifact: { schema: blobArtifactSchema() },width: { schema: { kind: "number", integer: true, minimum: 1 } }, height: { schema: { kind: "number", integer: true, minimum: 1 } },- frameRate: { schema: rational }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } }, muted: { schema: { kind: "literal", value: true } } }), optional: true },- audio: { schema: object({ artifact: { schema: blobArtifactSchema(["audio/wav"]) }, sourceStreamIndex: { schema: integer }, codec: { schema: { kind: "literal", value: "pcm_s16le" } },- sampleRate: { schema: { kind: "literal", value: 48_000 } }, channels: { schema: { kind: "literal", value: 2 } },- sampleFrames: { schema: { kind: "number", integer: true, minimum: 1 } }, loudness: { schema: { kind: "literal", value: "preserved" } } }), optional: true },+ }), optional: true },+ audio: { schema: object({ artifact: { schema: blobArtifactSchema(["audio/wav"]) } }), optional: true },});export const renderedVisualSchema: ValueSchema = object({ contract: { schema: { kind: "literal", value: "svml.rendered-visual@1" } },frameRate: { schema: rational }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } },diff --git a/packages/media/src/types.ts b/packages/media/src/types.tsindex 0e5a0253..a94da3be 100644--- a/packages/media/src/types.ts+++ b/packages/media/src/types.ts@@ -88,39 +88,16 @@ export type MediaStreamSelection = {export type SynchronizedMedia = {readonly contract: "svml.synchronized-media@1";readonly timeline: {- readonly spanAuthority: "video" | "audio";readonly frameRate: MediaRational;readonly frameCount: number;- readonly sampleRate: 48_000;- readonly sampleFrames: number;- };- readonly sourceMap: {- readonly sourceOriginPts: MediaTimestamp;- readonly sourceEndPts: MediaTimestamp;- readonly audioTrimStartSamples: number;- readonly audioTrimEndSamples: number;- readonly audioHeadSamples: number;- readonly audioContentSamples: number;- readonly audioTailSamples: number;};readonly visual?: {readonly artifact: BlobRef;- readonly sourceStreamIndex: number;readonly width: number;readonly height: number;- readonly frameRate: MediaRational;- readonly frameCount: number;- readonly muted: true;};readonly audio?: {readonly artifact: BlobRef;- readonly sourceStreamIndex: number;- readonly codec: "pcm_s16le";- readonly sampleRate: 48_000;- readonly channels: 2;- readonly sampleFrames: number;- /** Media normalization preserves level. Loudness/mastering is a separate author policy. */- readonly loudness: "preserved";};};diff --git a/packages/provider-media-local/test/provider.test.ts b/packages/provider-media-local/test/provider.test.tsindex bca885a9..89adc7c3 100644--- a/packages/provider-media-local/test/provider.test.ts+++ b/packages/provider-media-local/test/provider.test.ts@@ -4,7 +4,7 @@ import { tmpdir } from "node:os";import { join } from "node:path";import test from "node:test";import { artifactTypes } from "@narratage/artifact";-import { mediaTypes, sealRenderedVisual, verifyMediaInspection, verifyMuxedMedia, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";+import { mediaTypes, sealRenderedVisual, synchronizedMediaSampleFrames, verifyMediaInspection, verifyMuxedMedia, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";import type { MediaAudioStream, MediaInspection, MuxedMedia, SynchronizedMedia, TimelineAudio } from "@narratage/media";import { assertSpeechEvidenceAudioIdentity, speechTypes } from "@narratage/speech";import type { SpeechEvidenceAudio } from "@narratage/speech";@@ -70,6 +70,47 @@ function rampWav(sampleFrames: number): Buffer {return bytes;}+function pcm16StereoAudibleSpan(bytes: Uint8Array): {+ readonly firstSample: number;+ readonly endSampleExclusive: number;+ readonly sampleFrames: number;+} {+ const wav = Buffer.from(bytes);+ assert.equal(wav.toString("ascii", 0, 4), "RIFF");+ assert.equal(wav.toString("ascii", 8, 12), "WAVE");+ let dataStart = -1;+ let dataBytes = 0;+ for (let offset = 12; offset + 8 <= wav.length;) {+ const id = wav.toString("ascii", offset, offset + 4);+ const size = wav.readUInt32LE(offset + 4);+ if (id === "fmt ") {+ assert.equal(wav.readUInt16LE(offset + 8), 1);+ assert.equal(wav.readUInt16LE(offset + 10), 2);+ assert.equal(wav.readUInt32LE(offset + 12), 48_000);+ assert.equal(wav.readUInt16LE(offset + 22), 16);+ }+ if (id === "data") {+ dataStart = offset + 8;+ dataBytes = size;+ break;+ }+ offset += 8 + size + (size % 2);+ }+ assert(dataStart >= 0 && dataStart + dataBytes <= wav.length, "normalized WAV has no complete data chunk");+ const sampleFrames = Math.floor(dataBytes / 4);+ let firstSample = sampleFrames;+ let endSampleExclusive = 0;+ for (let frame = 0; frame < sampleFrames; frame += 1) {+ const offset = dataStart + frame * 4;+ if (Math.max(Math.abs(wav.readInt16LE(offset)), Math.abs(wav.readInt16LE(offset + 2))) > 64) {+ firstSample = Math.min(firstSample, frame);+ endSampleExclusive = frame + 1;+ }+ }+ assert(firstSample < endSampleExclusive, "normalized WAV contains no audible samples");+ return { firstSample, endSampleExclusive, sampleFrames };+}+function presentationSampleFrames(stream: MediaAudioStream): number {assert(stream.startPts !== undefined && stream.endPts !== undefined);const startNumerator = BigInt(stream.startPts.ticks) * BigInt(stream.startPts.timeBase.numerator);@@ -223,16 +264,7 @@ test("local media Provider enumerates attached pictures and jointly normalizes 3const typed = await normalizeArtifact({ artifacts, source, inspection: typedInspection,selection, frameRate: selectionRequest.frameRate });assert.equal(typed.timeline.frameCount, 30);- assert.equal(typed.timeline.sampleFrames, 48_000);- assert.equal(typed.visual?.sourceStreamIndex, 0);- assert.equal(typed.visual?.muted, true);- assert.equal(typed.audio?.sourceStreamIndex, 1);- assert.equal(typed.audio?.sampleRate, 48_000);- assert.equal(typed.audio?.channels, 2);- assert.equal(typed.sourceMap.audioHeadSamples + typed.sourceMap.audioContentSamples- + typed.sourceMap.audioTailSamples, typed.timeline.sampleFrames);- assert.ok(typed.sourceMap.audioTrimEndSamples > 0, "AAC tail beyond the final picture must be trimmed");- assert.equal(typed.sourceMap.audioHeadSamples, 0);+ assert.equal(synchronizedMediaSampleFrames(typed), 48_000);assert.equal("basisDigest" in typed, false);assert.equal("narrativeDigest" in typed, false);assert.equal(await artifacts.has(typed.visual!.artifact.digest), true);@@ -265,6 +297,11 @@ test("local media Provider enumerates attached pictures and jointly normalizes 3};assert.deepEqual(outputProbe.streams.map((stream) => stream.codec_type), ["video"]);assert.equal(outputProbe.streams[0]?.nb_frames, "30");+ const audioOutput = await inspectArtifact(artifacts, typed.audio!.artifact);+ const normalizedAudio = audioOutput.streams.find((stream) => stream.kind === "audio");+ assert.equal(normalizedAudio?.kind === "audio" && normalizedAudio.sampleRate, 48_000);+ assert.equal(normalizedAudio?.kind === "audio" && normalizedAudio.channels, 2);+ assert.equal(normalizedAudio?.kind === "audio" && normalizedAudio.decodedSampleFrames, 48_000);} finally {await rm(root, { recursive: true, force: true });}@@ -335,7 +372,6 @@ test("animated WebP keeps its authored frame timing before fixed-rate normalizatconst selection = selectMediaStreams(inspection, request);const normalized = await normalizeArtifact({ artifacts, source, inspection, selection, frameRate: request.frameRate });assert.equal(normalized.timeline.frameCount, 10);- assert.equal(normalized.visual?.frameCount, 10);assert.equal(normalized.visual?.width, 64);assert.equal(normalized.visual?.height, 48);const output = await inspectArtifact(artifacts, normalized.visual!.artifact);@@ -469,14 +505,14 @@ test("local media Provider preserves one source A/V origin when audio starts latconst expectedEnd = Math.round((seconds(audio.endPts) - seconds(video.startPts)) * 48_000);const expectedTail = 48_000 - Math.min(48_000, expectedEnd);assert.equal(normalized.timeline.frameCount, 30);- assert.equal(normalized.timeline.sampleFrames, 48_000);- assert.equal(normalized.sourceMap.audioHeadSamples, expectedHead,- "normalized audio head must preserve the measured stream-start offset");- assert.equal(normalized.sourceMap.audioTailSamples, expectedTail,- "normalized audio tail must preserve the measured stream end");- assert.ok(normalized.sourceMap.audioHeadSamples > 9_000, "fixture must retain a substantial audio lag");- assert.equal(normalized.sourceMap.audioTrimStartSamples, 0);- assert.equal(normalized.sourceMap.audioTrimEndSamples, 0);+ const bytes = await artifacts.get(normalized.audio!.artifact.digest);+ assert(bytes !== undefined);+ const audible = pcm16StereoAudibleSpan(bytes);+ assert.equal(audible.sampleFrames, 48_000);+ assert.ok(Math.abs(audible.firstSample - expectedHead) <= 64,+ `normalized audio begins at ${audible.firstSample}, expected ${expectedHead}`);+ assert.ok(Math.abs(audible.endSampleExclusive - (48_000 - expectedTail)) <= 64,+ `normalized audio ends at ${audible.endSampleExclusive}, expected ${48_000 - expectedTail}`);} finally {await rm(root, { recursive: true, force: true });}@@ -507,13 +543,6 @@ test("a silent generated MP4 remains a visual-only product and cannot satisfy aassert.equal(normalized.timeline.frameCount, 15);assert.ok(normalized.visual);assert.equal(normalized.audio, undefined);- assert.deepEqual({- trimStart: normalized.sourceMap.audioTrimStartSamples,- trimEnd: normalized.sourceMap.audioTrimEndSamples,- head: normalized.sourceMap.audioHeadSamples,- content: normalized.sourceMap.audioContentSamples,- tail: normalized.sourceMap.audioTailSamples,- }, { trimStart: 0, trimEnd: 0, head: 0, content: 0, tail: 0 });const invalid = sealMediaSelectionRequest({contract: "svml.media-selection-request@1",diff --git a/packages/ranking/src/render.ts b/packages/ranking/src/render.tsindex 40d06037..a02296cf 100644--- a/packages/ranking/src/render.ts+++ b/packages/ranking/src/render.ts@@ -17,7 +17,7 @@ import type {VisualTextTypography,VisualTrack,} from "@narratage/composition";-import { verifySynchronizedMedia } from "@narratage/media";+import { synchronizedMediaSampleFrames, verifySynchronizedMedia } from "@narratage/media";import {assertProgramSpaceIdentity,programFrameSampleBoundary,@@ -684,14 +684,15 @@ export function renderRankingAudio(const audio = media.audio;assert(audio !== undefined, `Ranking ${event.kind} sound has no normalized audio.`);const startSample = programFrameSampleBoundary(space, event.frame, 48_000);- const length = Math.min(audio.sampleFrames, totalSamples - startSample);+ const sourceSampleFrames = synchronizedMediaSampleFrames(media);+ const length = Math.min(sourceSampleFrames, totalSamples - startSample);assert(length > 0, `Ranking sound ${event.id} starts after ProgramSpace.`);assert(fadeSamples <= length, `Ranking sound ${event.id} fade exceeds its audible interval.`);clips.push({id: event.id,artifact: structuredClone(audio.artifact),target: { startSample, endSampleExclusive: startSample + length },- source: { sampleFrames: audio.sampleFrames, startSample: 0, endSampleExclusive: length, loop: false, phaseSample: 0 },+ source: { sampleFrames: sourceSampleFrames, startSample: 0, endSampleExclusive: length, loop: false, phaseSample: 0 },playbackRate: 1, pitch: "preserve",gain: event.kind === "appear" ? style.appearGain : style.moveGain,fadeInSamples: fadeSamples, fadeOutSamples: 0,diff --git a/packages/ranking/test/ranking.test.ts b/packages/ranking/test/ranking.test.tsindex 3eb28e46..716f9c99 100644--- a/packages/ranking/test/ranking.test.ts+++ b/packages/ranking/test/ranking.test.ts@@ -284,19 +284,11 @@ test("Typewriter uses Unicode graphemes, explicit emphasis and winner timing wit}, set));});-const sound = (id: string, sampleFrames = 4_800): SynchronizedMedia => ({+const sound = (id: string): SynchronizedMedia => ({contract: "svml.synchronized-media@1",- timeline: { spanAuthority: "audio", frameRate: { numerator: 30, denominator: 1 }, frameCount: 3,- sampleRate: 48_000, sampleFrames },- sourceMap: {- sourceOriginPts: { ticks: "0", timeBase: { numerator: 1, denominator: 48_000 } },- sourceEndPts: { ticks: String(sampleFrames), timeBase: { numerator: 1, denominator: 48_000 } },- audioTrimStartSamples: 0, audioTrimEndSamples: 0, audioHeadSamples: 0,- audioContentSamples: sampleFrames, audioTailSamples: 0,- },+ timeline: { frameRate: { numerator: 30, denominator: 1 }, frameCount: 3 },audio: {artifact: { kind: "blob", digest: digestOf(`ranking-sound:${id}`), size: 128, mediaType: "audio/wav" },- sourceStreamIndex: 0, codec: "pcm_s16le", sampleRate: 48_000, channels: 2, sampleFrames, loudness: "preserved",},});diff --git a/packages/semantic-map/src/index.ts b/packages/semantic-map/src/index.tsindex f01755d2..bb2dadd1 100644--- a/packages/semantic-map/src/index.ts+++ b/packages/semantic-map/src/index.ts@@ -3,6 +3,7 @@ export {assertNarrativeMomentIdentity,assertNarrativeSelectionIdentity,momentFrames,+ segmentFrameSpan,selectionFrameSpans,tokenSpanSeconds,} from "./location.js";diff --git a/packages/semantic-map/src/location.ts b/packages/semantic-map/src/location.tsindex c100bd9a..b78a6aa3 100644--- a/packages/semantic-map/src/location.ts+++ b/packages/semantic-map/src/location.ts@@ -1,4 +1,4 @@-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";import { assertProgramSpaceIdentity } from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";import type { CompleteSemanticMap } from "./types.js";@@ -77,6 +77,24 @@ export function selectionFrameSpans(});}+/** Locate one authored Segment by its two structural anchors. */+export function segmentFrameSpan(+ map: CompleteSemanticMap,+ segment: NarrativeExcerpt,+ programSpace: ProgramSpace,+): LocatedFrameSpan {+ assertCompleteSemanticMapIdentity(map);+ assertProgramSpaceIdentity(programSpace);+ if (segment.contract !== "svml.narrative-excerpt@1" || segment.kind !== "segment" || segment.id.length === 0) {+ throw new Error("Narrative Segment excerpt is invalid");+ }+ const frames = anchorFrames(map);+ return {+ startFrame: frameFor(frames, `segment:${segment.id}:start`, `Narrative Segment ${segment.id}`),+ endFrameExclusive: frameFor(frames, `segment:${segment.id}:end`, `Narrative Segment ${segment.id}`),+ };+}+/*** The measured window a run of Script tokens occupies: the first token's start* and the last token's end, exactly as located. Undefined when any named tokendiff --git a/packages/semantic-map/test/location.test.ts b/packages/semantic-map/test/location.test.tsindex aada1467..10427b03 100644--- a/packages/semantic-map/test/location.test.ts+++ b/packages/semantic-map/test/location.test.ts@@ -1,7 +1,7 @@import assert from "node:assert/strict";import test from "node:test";-import { momentFrames, selectionFrameSpans } from "@narratage/semantic-map";+import { momentFrames, segmentFrameSpan, selectionFrameSpans } from "@narratage/semantic-map";import type { CompleteSemanticMap } from "@narratage/semantic-map";import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";import type { ProgramSpace } from "@narratage/program-space";@@ -86,6 +86,12 @@ test("a Segment cut is an ordinary anchor, so a Selection may start at one", ());});+test("a whole Segment is exactly its two structural anchors", () => {+ assert.deepEqual(segmentFrameSpan(map, {+ contract: "svml.narrative-excerpt@1", kind: "segment", id: "second", tokenStart: 2, tokenEndExclusive: 4,+ }, space), { startFrame: 5, endFrameExclusive: 9 });+});+test("a Moment locates one instant per occurrence", () => {const moment = {contract: "svml.narrative-moment@1",diff --git a/packages/spatial/src/author.ts b/packages/spatial/src/author.tsnew file mode 100644index 00000000..e4d4d7b1--- /dev/null+++ b/packages/spatial/src/author.ts@@ -0,0 +1,60 @@+import type { CanonicalValue } from "@narratage/protocol";++import { sealContentFit } from "./geometry.js";+import type { ContentFit } from "./types.js";++export const contentFitPropertyNames = [+ "fit", "frame-x", "frame-y", "content-x", "content-y",+ "fit-offset-x", "fit-offset-y", "fit-constraint",+] as const;++function number(+ properties: Readonly<Record<string, CanonicalValue>>,+ name: string,+ fallback: number,+ owner: string,+): number {+ const value = properties[name];+ if (value === undefined) return fallback;+ if (typeof value !== "number" || !Number.isFinite(value)) throw new Error(`${owner} ${name} must be a finite number.`);+ return value;+}++function oneOf<T extends string>(+ properties: Readonly<Record<string, CanonicalValue>>,+ name: string,+ allowed: readonly T[],+ fallback: T,+ owner: string,+): T {+ const value = properties[name];+ if (value === undefined) return fallback;+ if (typeof value !== "string" || !allowed.includes(value as T)) {+ throw new Error(`${owner} ${name} must be ${allowed.join(" | ")}.`);+ }+ return value as T;+}++/** Decode the shared spatial-fit vocabulary from any author-side property map. */+export function decodeContentFitProperties(+ properties: Readonly<Record<string, CanonicalValue>>,+ owner = "Content-fit properties",+): ContentFit {+ return sealContentFit({+ contract: "svml.content-fit@1",+ sizing: oneOf(properties, "fit", ["contain", "cover", "fit-width", "fit-height", "native", "scale-down", "stretch"] as const, "contain", owner),+ framePoint: {+ x: number(properties, "frame-x", 0.5, owner),+ y: number(properties, "frame-y", 0.5, owner),+ },+ contentPoint: {+ x: number(properties, "content-x", 0.5, owner),+ y: number(properties, "content-y", 0.5, owner),+ },+ offsetPx: {+ x: number(properties, "fit-offset-x", 0, owner),+ y: number(properties, "fit-offset-y", 0, owner),+ },+ constraint: oneOf(properties, "fit-constraint", ["bounded", "free"] as const, "bounded", owner),+ });+}diff --git a/packages/spatial/src/index.ts b/packages/spatial/src/index.tsindex 6d982619..5fdf6dc3 100644--- a/packages/spatial/src/index.ts+++ b/packages/spatial/src/index.ts@@ -1,4 +1,5 @@export { spatialComponent } from "./component.js";+export { contentFitPropertyNames, decodeContentFitProperties } from "./author.js";export {anchoredFrameFragment,aspectFrameFragment,diff --git a/packages/speech-alignment/test/speech-align.test.ts b/packages/speech-alignment/test/speech-align.test.tsindex cc132c47..292136fa 100644--- a/packages/speech-alignment/test/speech-align.test.ts+++ b/packages/speech-alignment/test/speech-align.test.ts@@ -76,20 +76,7 @@ function speechBasis(contract: "svml.speech-basis@1",programSpace,audio: { kind: "blob", digest: audioDigest, size: 1, mediaType: "audio/wav" },- visualTrack: {- clips: segments.map((segment) => ({- segmentId: segment.segmentId,- artifact: {- kind: "blob",- digest: digestOf(`fixture:clip:${segment.segmentId}`),- size: 1,- mediaType: "video/mp4",- },- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },- frameRate: { ...programSpace.frameRate },- frameCount: Math.round((segment.endSec - segment.startSec) * 1_000),- })),- },+ visualTrack: { clips: [] },segments,});return audioProjection(take);diff --git a/packages/speech-basis/package.json b/packages/speech-basis/package.jsonindex ff66222d..e38fea46 100644--- a/packages/speech-basis/package.json+++ b/packages/speech-basis/package.json@@ -16,8 +16,7 @@"@narratage/media-track": "workspace:*","@narratage/program-space": "workspace:*","@narratage/protocol": "workspace:*",- "@narratage/speech": "workspace:*",- "@narratage/spatial": "workspace:*"+ "@narratage/speech": "workspace:*"},"devDependencies": {"@narratage/core": "workspace:*",diff --git a/packages/speech-basis/src/component.ts b/packages/speech-basis/src/component.tsindex 0d8712af..17464667 100644--- a/packages/speech-basis/src/component.ts+++ b/packages/speech-basis/src/component.ts@@ -2,7 +2,6 @@ import type { ComponentPackage } from "@narratage/component-kit";import type { SpeechBasis } from "@narratage/speech";import type { StoredValue } from "@narratage/protocol";import { canonicalize } from "@narratage/protocol";-import type { CanvasSpace } from "@narratage/spatial";import {speechBasisProducers,@@ -23,11 +22,6 @@ function speechBasis(value: StoredValue | undefined): SpeechBasis {return value.value as unknown as SpeechBasis;}-function canvas(value: StoredValue | undefined): CanvasSpace {- if (value?.kind !== "inline") throw new Error("CanvasSpace must be inline");- return value.value as unknown as CanvasSpace;-}-/** Pure projections of one atomic SpeechBasis; no Artifact read or external capability is hidden here. */export const speechBasisComponent = {name: "@narratage/speech-basis",@@ -65,7 +59,7 @@ export const speechBasisComponent = {outputs: {visual: {kind: "inline",- value: canonicalize(projectSpeechVisual(speechBasis(inputs.basis?.value), canvas(inputs.canvas?.value))),+ value: canonicalize(projectSpeechVisual(speechBasis(inputs.basis?.value))),},},needs: {},diff --git a/packages/speech-basis/src/fragment.ts b/packages/speech-basis/src/fragment.tsindex d1c75968..582b9848 100644--- a/packages/speech-basis/src/fragment.ts+++ b/packages/speech-basis/src/fragment.ts@@ -3,7 +3,6 @@ import { speechTypes } from "@narratage/speech";import type { SpeechBasis } from "@narratage/speech";import { compositionTypes } from "@narratage/composition";import { sealGraphFragment } from "@narratage/elaborator";-import { spatialTypes } from "@narratage/spatial";import { speechBasisProducers } from "./manifest.js";@@ -13,7 +12,7 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operat/** Project the independently demandable facts carried by one SpeechBasis. */export const speechBasisProjectionFragment = sealGraphFragment({name: "@narratage/speech-basis/projections@1",- inputs: [{ name: "basis", type: speechTypes.basis }, { name: "canvas", type: spatialTypes.canvas }],+ inputs: [{ name: "basis", type: speechTypes.basis }],operations: [{id: "project-program-space",@@ -30,7 +29,7 @@ export const speechBasisProjectionFragment = sealGraphFragment({{id: "project-visual",producer: speechBasisProducers.projectVisual,- inputs: { basis: input("basis"), canvas: input("canvas") },+ inputs: { basis: input("basis") },result: { kind: "output", name: "visual" },},{@@ -59,7 +58,7 @@ export const speechBasisProjectionFragment = sealGraphFragment({name: "visual",type: compositionTypes.visualTrack,root: operation("project-visual"),- semanticInputs: ["basis", "canvas"],+ semanticInputs: ["basis"],fidelity: "exact",},{diff --git a/packages/speech-basis/src/manifest.ts b/packages/speech-basis/src/manifest.tsindex a3f4af9b..511710b2 100644--- a/packages/speech-basis/src/manifest.ts+++ b/packages/speech-basis/src/manifest.ts@@ -1,7 +1,6 @@import { programSpaceDependency, programSpaceTypes } from "@narratage/program-space";import { speechDependency, speechTypes } from "@narratage/speech";import { compositionDependency, compositionTypes } from "@narratage/composition";-import { spatialDependency, spatialTypes } from "@narratage/spatial";import { digestOf } from "@narratage/protocol";import type { ModuleManifest, ProducerRef } from "@narratage/protocol";@@ -29,7 +28,6 @@ export const speechBasisManifest: ModuleManifest = {speechDependency,programSpaceDependency,compositionDependency,- spatialDependency,],types: [],capabilities: [],@@ -59,7 +57,7 @@ export const speechBasisManifest: ModuleManifest = {},{name: speechBasisProducers.projectVisual.name,- inputs: [{ name: "basis", type: speechTypes.basis }, { name: "canvas", type: spatialTypes.canvas }],+ inputs: [{ name: "basis", type: speechTypes.basis }],outputs: [{ name: "visual", type: compositionTypes.visualTrack }],needs: [],implementation: {diff --git a/packages/speech-basis/src/projection.ts b/packages/speech-basis/src/projection.tsindex d297738b..3369350b 100644--- a/packages/speech-basis/src/projection.ts+++ b/packages/speech-basis/src/projection.ts@@ -1,4 +1,4 @@-import { programSpaceFrameCount, programSpaceSampleFrames } from "@narratage/program-space";+import { programSpaceSampleFrames } from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";import { assertSpeechBasisIdentity } from "@narratage/speech";import type { SpeechAudioBasis, SpeechBasis } from "@narratage/speech";@@ -6,7 +6,6 @@ import { lowerRestrictedSpeechVisualPresents } from "@narratage/media-track";import { sealAudioTrack, sealVisualTrack } from "@narratage/composition";import type { AudioTrack, VisualTrack } from "@narratage/composition";import { digestOf } from "@narratage/protocol";-import type { CanvasSpace } from "@narratage/spatial";export const projectSpeechAudioImplementationDigest = digestOf("@narratage/speech-basis/project-audio@1",@@ -39,14 +38,9 @@ export function projectSpeechAudio(basis: SpeechBasis): SpeechAudioBasis {};}-function frameAt(basis: SpeechBasis, seconds: number): number {- return Math.round(seconds * basis.programSpace.frameRate.numerator / basis.programSpace.frameRate.denominator);-}--export function projectSpeechVisual(basis: SpeechBasis, canvas: CanvasSpace): VisualTrack {+export function projectSpeechVisual(basis: SpeechBasis): VisualTrack {assertSpeechBasisIdentity(basis);const trackId = `speech-visual:${basis.segments.map((segment) => segment.segmentId).join("+")}`;- const segments = new Map(basis.segments.map((segment) => [segment.segmentId, segment]));return sealVisualTrack({contract: "svml.visual-track@1",visualIr: "svml.visual-ir@1",@@ -54,26 +48,17 @@ export function projectSpeechVisual(basis: SpeechBasis, canvas: CanvasSpace): Vipresents: lowerRestrictedSpeechVisualPresents(trackId,basis.programSpace,- canvas,- basis.visualTrack.clips.map((clip) => {- const segment = segments.get(clip.segmentId)!;- return {- id: clip.segmentId,- span: { startFrame: frameAt(basis, segment.startSec), endFrameExclusive: frameAt(basis, segment.endSec) },- artifact: clip.artifact,- extent: clip.extent,- frameRate: clip.frameRate,- frameCount: clip.frameCount,- };- }),- {- contract: "svml.content-fit@1",- sizing: "cover",- framePoint: { x: 0.5, y: 0.5 },- contentPoint: { x: 0.5, y: 0.5 },- offsetPx: { x: 0, y: 0 },- constraint: "bounded",- },+ basis.visualTrack.clips.map((clip) => ({+ id: clip.segmentId,+ span: structuredClone(clip.span),+ artifact: clip.artifact,+ extent: clip.extent,+ frameRate: clip.frameRate,+ frameCount: clip.frameCount,+ frame: clip.frame,+ fit: clip.fit,+ stackingOrder: clip.stackingOrder,+ })),),});}diff --git a/packages/speech-basis/test/product.test.ts b/packages/speech-basis/test/product.test.tsindex 07d70d16..e6ea7721 100644--- a/packages/speech-basis/test/product.test.ts+++ b/packages/speech-basis/test/product.test.ts@@ -42,20 +42,8 @@ import {speechBasisProducers,} from "@narratage/speech-basis";import { TypeValidatorRegistry } from "@narratage/validation";-import { spatialTypes } from "@narratage/spatial";-import { spatialValidatorDigests } from "@narratage/spatial";import { mediaTrackManifest } from "@narratage/media-track";-const canvas = {- contract: "svml.canvas-space@1" as const,- widthPx: 720,- heightPx: 1280,- origin: "top-left" as const,- xDirection: "right" as const,- yDirection: "down" as const,- pixelAspect: "square" as const,-};-const testModule = { name: "example.speech-basis-product", version: "0.0.0" } as const;const requestType = { module: testModule, name: "SpeechRequest" } satisfies TypeRef;const generateProducer = { module: testModule, name: "generate-speech-basis" } satisfies ProducerRef;@@ -123,10 +111,18 @@ function sampleTake(label = "generated"): SpeechBasis {visualTrack: {clips: [{segmentId: "opening",+ span: { startFrame: 0, endFrameExclusive: 60 },artifact: visual,extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },frameRate: { ...programSpace.frameRate },frameCount: 60,+ frame: { contract: "svml.spatial-frame@1", xPx: 0, yPx: 0, widthPx: 720, heightPx: 1280 },+ fit: {+ contract: "svml.content-fit@1", sizing: "cover",+ framePoint: { x: 0.5, y: 0.5 }, contentPoint: { x: 0.5, y: 0.5 },+ offsetPx: { x: 0, y: 0 }, constraint: "bounded",+ },+ stackingOrder: 0,}],},segments: [{@@ -149,29 +145,10 @@ function createProgram(): LinkedProgram {frontendClosureDigest: digestOf("frontend:speech-basis-product"),},});- const rawCanvasRecord = sealRecord({- id: "canvas:vertical",- type: spatialTypes.canvas,- value: { kind: "inline", value: canvas },- conformance: "exact",- origin: {- kind: "authored",- sourceDigest: digestOf("source:speech-basis-product"),- frontendClosureDigest: digestOf("frontend:speech-basis-product"),- },- });- const canvasRecord = {- ...rawCanvasRecord,- validation: sealTypeValidationReceipt({- type: rawCanvasRecord.type,- recordDigest: rawCanvasRecord.digest,- validatorDigest: spatialValidatorDigests.canvas,- }),- };return link(closure, [sealTypedModule({id: "author:speech-basis-product",closureDigest: closure.digest,- records: [request, canvasRecord],+ records: [request],})]);}@@ -180,7 +157,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {const output = (id: string) => ({ kind: "logical-output" as const, id });const operation = (id: string) => ({ kind: "operation-result" as const, operation: id });const existingTake = sampleTake("approved");- const existingVisual = projectSpeechVisual(existingTake, canvas);+ const existingVisual = projectSpeechVisual(existingTake);const existingVisualValue = { kind: "inline" as const, value: existingVisual };const existingVisualValidation = sealTypeValidationReceipt({type: compositionTypes.visualTrack,@@ -206,7 +183,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {id: "opening.visual",type: compositionTypes.visualTrack,primary: "opening.visual.project",- semanticInputs: [output("opening.take"), record("canvas:vertical")],+ semanticInputs: [output("opening.take")],},],candidates: [@@ -267,7 +244,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {{id: "project-opening-visual",producer: speechBasisProducers.projectVisual,- inputs: { basis: output("opening.take"), canvas: record("canvas:vertical") },+ inputs: { basis: output("opening.take") },result: { kind: "output", name: "visual", record: "visual:opening" },},],@@ -316,7 +293,7 @@ test("SpeechBasis is one Product and audio/visual are ordinary shared projectiontest("SpeechBasis projects to peer generic visual and audio Tracks", () => {const take = sampleTake();- const visual = projectSpeechVisual(take, canvas);+ const visual = projectSpeechVisual(take);const audio = projectSpeechAudioTrack(take);const programSpace = projectSpeechProgramSpace(take);assert.equal(visual.contract, "svml.visual-track@1");diff --git a/packages/speech-spine/README.md b/packages/speech-spine/README.mdindex 8444325f..a63adf41 100644--- a/packages/speech-spine/README.md+++ b/packages/speech-spine/README.md@@ -10,13 +10,20 @@ auditable reusable graph contributions:- WhisperX evidence plus the provider-neutral speech SemanticMap;- timed caption projection.-The package also owns `<speech:Spine>`. The Spine explicitly declares its `frame-rate`; each+The package also owns `<speech:Spine>`. The Spine explicitly declares its `frame-rate` and its+restricted visual base (`visual-frame`, `visual-appearance`, `visual-z`); each`<speech:Take>` connects one `NarrativeExcerpt` and exactly one of:- `video={...}` — an ordinary generated video Blob. The Surface expands inspection, streamselection and A/V normalization before assembly;+- `audio={...}` — an ordinary voice Blob. The Surface expands audio-authoritative normalization;+ this Take contributes no visual clip and never invents a black frame;- `media={...}` — an already prepared `SynchronizedMedia` value, connected directly.+Visual Takes may override the Spine base with `frame`, `appearance` and `z`. That is the complete+visual authority of Speech Spine: one same-source visual clip may be placed and stacked. Motion,+transitions, sequences and independent pictures remain ordinary Media Tracks.+The Surface then expands an immutable append fold, one timeline-audio render and ordinarySpeechBasis projections. Whitespace and child layout do not create ports, and Core receives novariadic node, media-normalization rule or Speech special case.diff --git a/packages/speech-spine/package.json b/packages/speech-spine/package.jsonindex 161ce210..47ae7f25 100644--- a/packages/speech-spine/package.json+++ b/packages/speech-spine/package.json@@ -22,6 +22,7 @@"@narratage/speech": "workspace:*","@narratage/speech-basis": "workspace:*","@narratage/spatial": "workspace:*",+ "@narratage/svs": "workspace:*","@narratage/markup": "workspace:*"},"devDependencies": {diff --git a/packages/speech-spine/src/component.ts b/packages/speech-spine/src/component.tsindex f52df1fa..db739958 100644--- a/packages/speech-spine/src/component.ts+++ b/packages/speech-spine/src/component.ts@@ -3,11 +3,14 @@ import type { NarrativeExcerpt } from "@narratage/narrative";import type { SynchronizedMedia, TimelineAudio } from "@narratage/media";import type { StoredValue } from "@narratage/protocol";import { canonicalize } from "@narratage/protocol";+import type { ContentFit, SpatialFrame } from "@narratage/spatial";import { speechSpineProducers } from "./manifest.js";import {- appendSpeechSpineTake,- appendSpeechSpineTakeImplementationDigest,+ appendSpeechSpineAudioTake,+ appendSpeechSpineAudioTakeImplementationDigest,+ appendSpeechSpineVisualTake,+ appendSpeechSpineVisualTakeImplementationDigest,assembleSpeechBasis,assembleSpeechBasisImplementationDigest,compileSpeechSpineAudio,@@ -15,7 +18,7 @@ import {createSpeechSpineSet,createSpeechSpineSetImplementationDigest,} from "./program.js";-import type { SpeechSpineProgram, SpeechSpineSet } from "./types.js";+import type { SpeechSpineProgram, SpeechSpineSet, SpeechSpineVisualSpec } from "./types.js";function inline<T>(value: StoredValue | undefined, subject: string): T {if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);@@ -34,10 +37,10 @@ export const speechSpineComponent = {}),},{- producer: speechSpineProducers.appendTake,- implementationDigest: appendSpeechSpineTakeImplementationDigest,+ producer: speechSpineProducers.appendAudioTake,+ implementationDigest: appendSpeechSpineAudioTakeImplementationDigest,handler: ({ inputs }) => ({- outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineTake(+ outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineAudioTake(inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"),inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),@@ -46,6 +49,22 @@ export const speechSpineComponent = {needs: {},}),},+ {+ producer: speechSpineProducers.appendVisualTake,+ implementationDigest: appendSpeechSpineVisualTakeImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineVisualTake(+ inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),+ inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"),+ inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),+ inline<NarrativeExcerpt>(inputs.segment?.value, "NarrativeExcerpt"),+ inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),+ inline<ContentFit>(inputs.fit?.value, "ContentFit"),+ inline<SpeechSpineVisualSpec>(inputs.visualSpec?.value, "SpeechSpineVisualSpec"),+ )) } },+ needs: {},+ }),+ },{producer: speechSpineProducers.compileAudio,implementationDigest: compileSpeechSpineAudioImplementationDigest,diff --git a/packages/speech-spine/src/fragment.ts b/packages/speech-spine/src/fragment.tsindex 9ed61e30..1b8f3d7b 100644--- a/packages/speech-spine/src/fragment.ts+++ b/packages/speech-spine/src/fragment.ts@@ -8,6 +8,7 @@ import type { FragmentOperation } from "@narratage/elaborator";import { mediaPipelineProducers } from "@narratage/media-pipeline";import { speechBasisProducers } from "@narratage/speech-basis";import { spatialTypes } from "@narratage/spatial";+import type { TypeRef } from "@narratage/protocol";import { speechSpineProducers, speechSpineTypes } from "./manifest.js";import type { SpeechSpineFragmentOptions } from "./types.js";@@ -17,13 +18,25 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operatexport function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {if (options.takes.length === 0) throw new Error("Speech Spine requires at least one Take");- const names = new Set(["program", "canvas"]);+ const declaredInputs = new Map<string, TypeRef>();+ const addInput = (name: string, type: TypeRef): void => {+ if (!name) throw new Error("Speech Spine Fragment input names must not be empty");+ const previous = declaredInputs.get(name);+ if (previous !== undefined && (previous.module.name !== type.module.name+ || previous.module.version !== type.module.version || previous.name !== type.name)) {+ throw new Error(`Speech Spine Fragment input ${name} is reused with another Type`);+ }+ declaredInputs.set(name, type);+ };+ addInput("program", speechSpineTypes.spineProgram);for (const take of options.takes) {- if (!take.mediaName || !take.segmentName || names.has(take.mediaName) || names.has(take.segmentName)) {- throw new Error("Speech Spine Fragment input names are empty or duplicated");+ addInput(take.mediaName, mediaTypes.synchronized);+ addInput(take.segmentName, narrativeTypes.excerpt);+ if (take.visual !== undefined) {+ addInput(take.visual.frameName, spatialTypes.frame);+ addInput(take.visual.fitName, spatialTypes.fit);+ addInput(take.visual.visualSpecName, speechSpineTypes.visualSpec);}- names.add(take.mediaName);- names.add(take.segmentName);}const operations: FragmentOperation[] = [{id: "spine:set:empty",@@ -36,12 +49,17 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {const id = `spine:set:append:${String(index + 1).padStart(4, "0")}`;operations.push({id,- producer: speechSpineProducers.appendTake,+ producer: take.visual === undefined ? speechSpineProducers.appendAudioTake : speechSpineProducers.appendVisualTake,inputs: {set: operation(current),program: input("program"),media: input(take.mediaName),segment: input(take.segmentName),+ ...(take.visual === undefined ? {} : {+ frame: input(take.visual.frameName),+ fit: input(take.visual.fitName),+ visualSpec: input(take.visual.visualSpecName),+ }),},result: { kind: "output", name: "set" },});@@ -81,7 +99,7 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {{id: "spine:visual-track",producer: speechBasisProducers.projectVisual,- inputs: { basis: operation("spine:basis"), canvas: input("canvas") },+ inputs: { basis: operation("spine:basis") },result: { kind: "output", name: "visual" },},{@@ -91,17 +109,14 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {result: { kind: "output", name: "track" },},);- const semanticInputs = ["program", ...options.takes.flatMap((take) => [take.mediaName, take.segmentName])];+ const semanticInputs = [...new Set(["program", ...options.takes.flatMap((take) => [+ take.mediaName, take.segmentName, ...(take.visual === undefined ? [] : [+ take.visual.frameName, take.visual.fitName, take.visual.visualSpecName,+ ]),+ ])])];return sealGraphFragment({name: options.name?.trim() || "@narratage/speech-spine/spine@1",- inputs: [- { name: "program", type: speechSpineTypes.spineProgram },- { name: "canvas", type: spatialTypes.canvas },- ...options.takes.flatMap((take) => [- { name: take.mediaName, type: mediaTypes.synchronized },- { name: take.segmentName, type: narrativeTypes.excerpt },- ]),- ],+ inputs: [...declaredInputs.entries()].map(([name, type]) => ({ name, type })),operations,exports: [{ name: "basis", type: speechTypes.basis, root: operation("spine:basis"), semanticInputs, fidelity: "exact" },@@ -114,7 +129,7 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {fidelity: "exact",},{- name: "visual", type: compositionTypes.visualTrack, root: operation("spine:visual-track"), semanticInputs: [...semanticInputs, "canvas"],+ name: "visual", type: compositionTypes.visualTrack, root: operation("spine:visual-track"), semanticInputs,fidelity: "exact",},{diff --git a/packages/speech-spine/src/index.ts b/packages/speech-spine/src/index.tsindex 5c50b720..ae3fac4a 100644--- a/packages/speech-spine/src/index.ts+++ b/packages/speech-spine/src/index.ts@@ -9,20 +9,25 @@ export {speechSpineTypes,speechSpineProgramSchema,speechSpineSetSchema,+ speechSpineVisualSpecSchema,speechSpineSurfaceImplementationDigest,} from "./manifest.js";export {- appendSpeechSpineTake,- appendSpeechSpineTakeImplementationDigest,+ appendSpeechSpineAudioTake,+ appendSpeechSpineAudioTakeImplementationDigest,+ appendSpeechSpineVisualTake,+ appendSpeechSpineVisualTakeImplementationDigest,assembleSpeechBasis,assembleSpeechBasisImplementationDigest,assertSpeechSpineProgram,assertSpeechSpineSet,+ assertSpeechSpineVisualSpec,compileSpeechSpineAudio,compileSpeechSpineAudioImplementationDigest,createSpeechSpineSet,createSpeechSpineSetImplementationDigest,sealSpeechSpineProgram,+ sealSpeechSpineVisualSpec,} from "./program.js";export { decodeSpeechSpineSurface } from "./surface.js";export type * from "./types.js";diff --git a/packages/speech-spine/src/manifest.ts b/packages/speech-spine/src/manifest.tsindex 7af92363..72b4a37e 100644--- a/packages/speech-spine/src/manifest.ts+++ b/packages/speech-spine/src/manifest.ts@@ -12,9 +12,17 @@ import {import { digestOf } from "@narratage/protocol";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@narratage/protocol";import { speechBasisManifest, speechBasisModuleRef } from "@narratage/speech-basis";+import {+ contentFitSchema,+ spatialDependency,+ spatialFrameSchema,+ spatialTypes,+} from "@narratage/spatial";+import { svsManifest, svsModuleRef } from "@narratage/svs";import {- appendSpeechSpineTakeImplementationDigest,+ appendSpeechSpineAudioTakeImplementationDigest,+ appendSpeechSpineVisualTakeImplementationDigest,assembleSpeechBasisImplementationDigest,compileSpeechSpineAudioImplementationDigest,createSpeechSpineSetImplementationDigest,@@ -25,10 +33,12 @@ export const speechSpineSurfaceImplementationDigest = digestOf("@narratage/speecexport const speechSpineTypes = {spineProgram: { module: speechSpineModuleRef, name: "SpeechSpineProgram" },spineSet: { module: speechSpineModuleRef, name: "SpeechSpineSet" },+ visualSpec: { module: speechSpineModuleRef, name: "SpeechSpineVisualSpec" },} satisfies Record<string, TypeRef>;export const speechSpineProducers = {createSet: { module: speechSpineModuleRef, name: "create-spine-set" },- appendTake: { module: speechSpineModuleRef, name: "append-spine-take" },+ appendAudioTake: { module: speechSpineModuleRef, name: "append-spine-audio-take" },+ appendVisualTake: { module: speechSpineModuleRef, name: "append-spine-visual-take" },compileAudio: { module: speechSpineModuleRef, name: "compile-spine-audio" },assembleBasis: { module: speechSpineModuleRef, name: "assemble-speech-basis" },} satisfies Record<string, ProducerRef>;@@ -52,9 +62,19 @@ export const speechSpineSetSchema: ValueSchema = object({takes: { schema: { kind: "array", items: object({segment: { schema: narrativeExcerptSchema },media: { schema: synchronizedMediaSchema },+ visual: { schema: object({+ frame: { schema: spatialFrameSchema },+ fit: { schema: contentFitSchema },+ stackingOrder: { schema: { kind: "number", integer: true } },+ }), optional: true },}) } },});+export const speechSpineVisualSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.speech-spine-visual-spec@1" } },+ stackingOrder: { schema: { kind: "number", integer: true } },+});+export const speechSpineManifest: ModuleManifest = {format: "svml.module@1",name: speechSpineModuleRef.name,@@ -66,19 +86,23 @@ export const speechSpineManifest: ModuleManifest = {programSpaceDependency,speechDependency,compositionDependency,+ spatialDependency,+ { module: svsModuleRef, digest: digestOf(svsManifest) },{ module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },{ module: speechBasisModuleRef, digest: digestOf(speechBasisManifest) },],types: [{ name: speechSpineTypes.spineProgram.name, schema: speechSpineProgramSchema },{ name: speechSpineTypes.spineSet.name, schema: speechSpineSetSchema },+ { name: speechSpineTypes.visualSpec.name, schema: speechSpineVisualSpecSchema },],capabilities: [],surfaces: [{name: "spine",tag: "Spine",mode: "structured",- outputs: [speechSpineTypes.spineProgram, mediaPipelineTypes.selectionRequest,+ outputs: [speechSpineTypes.spineProgram, speechSpineTypes.visualSpec, spatialTypes.fit,+ mediaPipelineTypes.selectionRequest,speechTypes.basis, programSpaceTypes.programSpace, speechTypes.audioBasis,compositionTypes.visualTrack, compositionTypes.audioTrack],implementation: {@@ -96,16 +120,31 @@ export const speechSpineManifest: ModuleManifest = {implementation: { kind: "registered", locator: "@narratage/speech-spine/create-spine-set", digest: createSpeechSpineSetImplementationDigest },},{- name: speechSpineProducers.appendTake.name,+ name: speechSpineProducers.appendAudioTake.name,+ inputs: [+ { name: "set", type: speechSpineTypes.spineSet },+ { name: "program", type: speechSpineTypes.spineProgram },+ { name: "media", type: mediaTypes.synchronized },+ { name: "segment", type: narrativeTypes.excerpt },+ ],+ outputs: [{ name: "set", type: speechSpineTypes.spineSet }],+ needs: [],+ implementation: { kind: "registered", locator: "@narratage/speech-spine/append-spine-audio-take", digest: appendSpeechSpineAudioTakeImplementationDigest },+ },+ {+ name: speechSpineProducers.appendVisualTake.name,inputs: [{ name: "set", type: speechSpineTypes.spineSet },{ name: "program", type: speechSpineTypes.spineProgram },{ name: "media", type: mediaTypes.synchronized },{ name: "segment", type: narrativeTypes.excerpt },+ { name: "frame", type: spatialTypes.frame },+ { name: "fit", type: spatialTypes.fit },+ { name: "visualSpec", type: speechSpineTypes.visualSpec },],outputs: [{ name: "set", type: speechSpineTypes.spineSet }],needs: [],- implementation: { kind: "registered", locator: "@narratage/speech-spine/append-spine-take", digest: appendSpeechSpineTakeImplementationDigest },+ implementation: { kind: "registered", locator: "@narratage/speech-spine/append-spine-visual-take", digest: appendSpeechSpineVisualTakeImplementationDigest },},{name: speechSpineProducers.compileAudio.name,diff --git a/packages/speech-spine/src/program.ts b/packages/speech-spine/src/program.tsindex b5f93f9e..0c60994a 100644--- a/packages/speech-spine/src/program.ts+++ b/packages/speech-spine/src/program.ts@@ -1,5 +1,5 @@import type { NarrativeExcerpt } from "@narratage/narrative";-import { verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";+import { synchronizedMediaSampleFrames, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";import type { SynchronizedMedia, TimelineAudio } from "@narratage/media";import { assertProgramSpaceIdentity, sealProgramSpace } from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";@@ -11,15 +11,19 @@ import {} from "@narratage/media-pipeline";import type { AudioProgramPlan } from "@narratage/media-pipeline";import { canonicalize, digestOf } from "@narratage/protocol";+import { assertContentFit, assertSpatialFrame } from "@narratage/spatial";+import type { ContentFit, SpatialFrame } from "@narratage/spatial";import type {SpeechSpineProgram,SpeechSpineSet,SpeechSpineTake,+ SpeechSpineVisualSpec,} from "./types.js";export const createSpeechSpineSetImplementationDigest = digestOf("@narratage/speech-spine/create-spine-set@1");-export const appendSpeechSpineTakeImplementationDigest = digestOf("@narratage/speech-spine/append-spine-take@1");+export const appendSpeechSpineAudioTakeImplementationDigest = digestOf("@narratage/speech-spine/append-spine-audio-take@1");+export const appendSpeechSpineVisualTakeImplementationDigest = digestOf("@narratage/speech-spine/append-spine-visual-take@1");export const compileSpeechSpineAudioImplementationDigest = digestOf("@narratage/speech-spine/compile-spine-audio@1");export const assembleSpeechBasisImplementationDigest = digestOf("@narratage/speech-spine/assemble-speech-basis@1");@@ -69,11 +73,29 @@ function sealSpeechSpineSet(value: SpeechSpineSet): SpeechSpineSet {function assertTake(take: SpeechSpineTake, program: SpeechSpineProgram): void {verifyExcerpt(take.segment);verifySynchronizedMedia(take.media);- assert(take.media.visual !== undefined, `Speech Segment ${take.segment.id} has no moving visual`);assert(take.media.audio !== undefined, `Speech Segment ${take.segment.id} has no speech audio`);assert(take.media.timeline.frameRate.numerator === program.frameRate.numerator&& take.media.timeline.frameRate.denominator === program.frameRate.denominator,`Speech Segment ${take.segment.id} uses another frame rate`);+ assert((take.media.visual === undefined) === (take.visual === undefined),+ `Speech Segment ${take.segment.id} visual policy does not match its media`);+ if (take.visual !== undefined) {+ assertSpatialFrame(take.visual.frame);+ assertContentFit(take.visual.fit);+ assert(Number.isSafeInteger(take.visual.stackingOrder),+ `Speech Segment ${take.segment.id} visual stacking order is invalid`);+ }+}++export function sealSpeechSpineVisualSpec(value: SpeechSpineVisualSpec): SpeechSpineVisualSpec {+ const result = structuredClone(value);+ assertSpeechSpineVisualSpec(result);+ return result;+}++export function assertSpeechSpineVisualSpec(value: SpeechSpineVisualSpec): void {+ assert(value.contract === "svml.speech-spine-visual-spec@1", "Unsupported SpeechSpineVisualSpec contract");+ assert(Number.isSafeInteger(value.stackingOrder), "SpeechSpineVisualSpec stacking order is invalid");}export function assertSpeechSpineSet(value: SpeechSpineSet): void {@@ -94,23 +116,58 @@ export function createSpeechSpineSet(): SpeechSpineSet {});}-export function appendSpeechSpineTake(+function appendTake(set: SpeechSpineSet,program: SpeechSpineProgram,- media: SynchronizedMedia,- segment: NarrativeExcerpt,+ take: SpeechSpineTake,): SpeechSpineSet {assertSpeechSpineSet(set);assertSpeechSpineProgram(program);- const take = { media, segment } satisfies SpeechSpineTake;assertTake(take, program);- assert(!set.takes.some((item) => item.segment.id === segment.id), `Speech Spine repeats Segment ${segment.id}`);+ assert(!set.takes.some((item) => item.segment.id === take.segment.id), `Speech Spine repeats Segment ${take.segment.id}`);return sealSpeechSpineSet({contract: "svml.speech-spine-set@1",takes: [...set.takes, take],});}+export function appendSpeechSpineAudioTake(+ set: SpeechSpineSet,+ program: SpeechSpineProgram,+ media: SynchronizedMedia,+ segment: NarrativeExcerpt,+): SpeechSpineSet {+ verifySynchronizedMedia(media);+ assert(media.visual === undefined, `Speech Segment ${segment.id} audio Take unexpectedly contains a visual stream`);+ return appendTake(set, program, { media, segment });+}++export function appendSpeechSpineVisualTake(+ set: SpeechSpineSet,+ program: SpeechSpineProgram,+ media: SynchronizedMedia,+ segment: NarrativeExcerpt,+ frame: SpatialFrame,+ fit: ContentFit,+ visualSpec: SpeechSpineVisualSpec,+): SpeechSpineSet {+ assertSpatialFrame(frame);+ assertContentFit(fit);+ assertSpeechSpineVisualSpec(visualSpec);+ verifySynchronizedMedia(media);+ assert(media.visual !== undefined, `Speech Segment ${segment.id} visual Take has no visual stream`);+ const take = {+ media,+ segment,+ visual: {+ frame: structuredClone(frame),+ fit: structuredClone(fit),+ stackingOrder: visualSpec.stackingOrder,+ },+ } satisfies SpeechSpineTake;+ return appendTake(set, program, take);+}+function frameSample(frame: number, frameRate: SpeechSpineProgram["frameRate"]): number {const numerator = BigInt(frame) * 48_000n * BigInt(frameRate.denominator);const denominator = BigInt(frameRate.numerator);@@ -143,16 +200,17 @@ export function compileSpeechSpineAudio(program: SpeechSpineProgram, set: Speechframe += take.media.timeline.frameCount;const targetStartSample = frameSample(startFrame, program.frameRate);const targetEndSampleExclusive = frameSample(frame, program.frameRate);- assert(targetEndSampleExclusive - targetStartSample === audio.sampleFrames,+ const sourceSampleFrames = synchronizedMediaSampleFrames(take.media);+ assert(targetEndSampleExclusive - targetStartSample === sourceSampleFrames,`Speech Segment ${take.segment.id} audio does not exactly cover its normalized frame span`);return {id: `${String(index + 1).padStart(4, "0")}:${take.segment.id}`,artifact: audio.artifact,targetStartSample,targetEndSampleExclusive,- sourceSampleFrames: audio.sampleFrames,+ sourceSampleFrames,sourceStartSample: 0,- sourceEndSampleExclusive: audio.sampleFrames,+ sourceEndSampleExclusive: sourceSampleFrames,sourceLoop: false,sourcePhaseSample: 0,playbackRate: 1,@@ -193,17 +251,27 @@ export function assembleSpeechBasis(frame += take.media.timeline.frameCount;return { segmentId: take.segment.id, startSec: seconds(startFrame), endSec: seconds(frame) };});- const visualClips = set.takes.map((take, index) => ({- segmentId: take.segment.id,- artifact: structuredClone(take.media.visual!.artifact),- extent: {- contract: "svml.intrinsic-extent@1" as const,- widthPx: take.media.visual!.width,- heightPx: take.media.visual!.height,- },- frameRate: { ...take.media.visual!.frameRate },- frameCount: take.media.visual!.frameCount,- }));+ frame = 0;+ const visualClips = set.takes.flatMap((take) => {+ const startFrame = frame;+ frame += take.media.timeline.frameCount;+ if (take.media.visual === undefined || take.visual === undefined) return [];+ return [{+ segmentId: take.segment.id,+ span: { startFrame, endFrameExclusive: frame },+ artifact: structuredClone(take.media.visual.artifact),+ extent: {+ contract: "svml.intrinsic-extent@1" as const,+ widthPx: take.media.visual.width,+ heightPx: take.media.visual.height,+ },+ frameRate: { ...take.media.timeline.frameRate },+ frameCount: take.media.timeline.frameCount,+ frame: structuredClone(take.visual.frame),+ fit: structuredClone(take.visual.fit),+ stackingOrder: take.visual.stackingOrder,+ }];+ });return sealSpeechBasis({contract: "svml.speech-basis@1",programSpace: space,diff --git a/packages/speech-spine/src/surface.ts b/packages/speech-spine/src/surface.tsindex eadc2e7d..f88eeb75 100644--- a/packages/speech-spine/src/surface.ts+++ b/packages/speech-spine/src/surface.ts@@ -2,9 +2,12 @@ import { narrativeTypes } from "@narratage/narrative";import { artifactTypes } from "@narratage/artifact";import { mediaTypes } from "@narratage/media";import {+ mediaPipelineTypes,sealMediaSelectionRequest,synchronizedMediaFragment,} from "@narratage/media-pipeline";+import type { SvsRecipe } from "@narratage/svs";+import { svsRecipeType } from "@narratage/svs";import type {StructuredElement,StructuredSurfaceHandler,@@ -14,8 +17,12 @@ import type {import { createSpeechSpineFragment } from "./fragment.js";import { speechSpineTypes } from "./manifest.js";-import { sealSpeechSpineProgram } from "./program.js";-import { spatialTypes } from "@narratage/spatial";+import { sealSpeechSpineProgram, sealSpeechSpineVisualSpec } from "./program.js";+import {+ contentFitPropertyNames,+ decodeContentFitProperties,+ spatialTypes,+} from "@narratage/spatial";function localName(value: string): string {return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value;@@ -27,6 +34,15 @@ function exactAttributes(element: StructuredElement, names: readonly string[]):}}+function allowedAttributes(element: StructuredElement, required: readonly string[], optional: readonly string[]): void {+ const keys = Object.keys(element.attributes);+ const missing = required.filter((name) => !keys.includes(name));+ const unknown = keys.filter((name) => !required.includes(name) && !optional.includes(name));+ if (missing.length > 0 || unknown.length > 0) {+ throw new Error(`${element.name} requires ${required.join(", ")}${optional.length === 0 ? "" : ` and optionally ${optional.join(", ")}`}`);+ }+}+function stringAttribute(element: StructuredElement, name: string): string {const value = element.attributes[name];if (typeof value !== "string" || value.trim().length === 0) throw new Error(`${element.name}.${name} must be a non-empty string`);@@ -45,6 +61,13 @@ function frameRate(element: StructuredElement): { readonly numerator: number; rereturn { numerator, denominator };}+function integerAttribute(element: StructuredElement, name: string): number {+ const raw = stringAttribute(element, name);+ const value = Number(raw);+ if (!Number.isSafeInteger(value)) throw new Error(`${element.name}.${name} must be an integer`);+ return value;+}+function referencePath(element: StructuredElement, name: string): string {const value: MarkupAttributeValue | undefined = element.attributes[name];if (typeof value !== "object" || value.kind !== "reference" || !value.path) {@@ -70,6 +93,18 @@ function resolve(return value;}+function recipe(value: SurfaceResolvedReference, label: string): SvsRecipe {+ if (!sameType(value.type, svsRecipeType) || value.record?.value.kind !== "inline") {+ throw new Error(`${label} must be an authored SVS Recipe`);+ }+ const result = value.record.value.value as unknown as SvsRecipe;+ const unknown = Object.keys(result.properties).filter((name) => !contentFitPropertyNames.includes(+ name as (typeof contentFitPropertyNames)[number],+ ));+ if (unknown.length > 0) throw new Error(`${label} only accepts spatial fit properties; found ${unknown.join(", ")}`);+ return result;+}+function takes(element: StructuredElement): StructuredElement[] {const result: StructuredElement[] = [];for (const child of element.children) {@@ -85,56 +120,165 @@ function takes(element: StructuredElement): StructuredElement[] {}export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- exactAttributes(element, ["id", "canvas", "frame-rate"]);+ exactAttributes(element, ["id", "frame-rate", "visual-frame", "visual-appearance", "visual-z"]);const id = stringAttribute(element, "id");- const canvas = resolve(element, "canvas", spatialTypes.canvas, resolveReference);+ const baseFrame = resolve(element, "visual-frame", spatialTypes.frame, resolveReference);+ const baseAppearanceReference = resolve(element, "visual-appearance", svsRecipeType, resolveReference);+ const baseAppearance = recipe(+ baseAppearanceReference,+ `${element.name}.visual-appearance`,+ );+ const baseZ = integerAttribute(element, "visual-z");const rate = frameRate(element);+ const frameInputNames = new Map<string, string>();+ const frameInputName = (value: SurfaceResolvedReference, fallback: string): string => {+ const key = JSON.stringify(value.ref);+ const previous = frameInputNames.get(key);+ if (previous !== undefined) return previous;+ frameInputNames.set(key, fallback);+ return fallback;+ };+ const fitInputNames = new Map<string, string>();+ const fitInputName = (value: SurfaceResolvedReference, fallback: string): string => {+ const key = JSON.stringify(value.ref);+ const previous = fitInputNames.get(key);+ if (previous !== undefined) return previous;+ fitInputNames.set(key, fallback);+ return fallback;+ };+ const visualSpecInputNames = new Map<number, string>();+ const visualSpecInputName = (stackingOrder: number, fallback: string): string => {+ const previous = visualSpecInputNames.get(stackingOrder);+ if (previous !== undefined) return previous;+ visualSpecInputNames.set(stackingOrder, fallback);+ return fallback;+ };const declaredTakes = takes(element).map((take, index) => {const hasVideo = take.attributes.video !== undefined;+ const hasAudio = take.attributes.audio !== undefined;const hasMedia = take.attributes.media !== undefined;- if (Number(hasVideo) + Number(hasMedia) !== 1) {- throw new Error(`${take.name} requires exactly one of video or media`);+ if (Number(hasVideo) + Number(hasAudio) + Number(hasMedia) !== 1) {+ throw new Error(`${take.name} requires exactly one of video, audio or media`);+ }+ const sourceAttribute = hasVideo ? "video" : hasAudio ? "audio" : "media";+ allowedAttributes(take, [sourceAttribute, "segment"], ["frame", "appearance", "z"]);+ if (hasAudio && (take.attributes.frame !== undefined || take.attributes.appearance !== undefined || take.attributes.z !== undefined)) {+ throw new Error(`${take.name} audio Take cannot declare visual frame, appearance or z`);}- exactAttributes(take, [hasVideo ? "video" : "media", "segment"]);+ const suffix = String(index + 1).padStart(4, "0");+ const effectiveFrame = take.attributes.frame === undefined+ ? baseFrame+ : resolve(take, "frame", spatialTypes.frame, resolveReference);+ const effectiveAppearanceReference = take.attributes.appearance === undefined+ ? baseAppearanceReference+ : resolve(take, "appearance", svsRecipeType, resolveReference);+ const effectiveAppearance = take.attributes.appearance === undefined+ ? baseAppearance+ : recipe(effectiveAppearanceReference, `${take.name}.appearance`);+ const effectiveZ = take.attributes.z === undefined ? baseZ : integerAttribute(take, "z");+ const visual = hasAudio ? undefined : {+ frameName: frameInputName(effectiveFrame, take.attributes.frame === undefined ? "visual-base-frame" : `take-${suffix}-frame`),+ fitName: fitInputName(effectiveAppearanceReference, take.attributes.appearance === undefined ? "visual-base-fit" : `take-${suffix}-fit`),+ visualSpecName: visualSpecInputName(effectiveZ, take.attributes.z === undefined ? "visual-base-spec" : `take-${suffix}-visual-spec`),+ frame: effectiveFrame,+ appearance: effectiveAppearance,+ stackingOrder: effectiveZ,+ };return {- mediaName: `take-${String(index + 1).padStart(4, "0")}-media`,- segmentName: `take-${String(index + 1).padStart(4, "0")}-segment`,- sourceKind: hasVideo ? "video" as const : "media" as const,- source: resolve(take, hasVideo ? "video" : "media", hasVideo ? artifactTypes.blob : mediaTypes.synchronized, resolveReference),+ suffix,+ mediaName: `take-${suffix}-media`,+ segmentName: `take-${suffix}-segment`,+ sourceKind: hasVideo ? "video" as const : hasAudio ? "audio" as const : "media" as const,+ source: resolve(take, sourceAttribute, hasMedia ? mediaTypes.synchronized : artifactTypes.blob, resolveReference),segment: resolve(take, "segment", narrativeTypes.excerpt, resolveReference),+ visual,range: take.range,};});const programId = `${id}.program`;- const requestId = `${id}.selection`;+ const videoRequestId = `${id}.selection.video`;+ const audioRequestId = `${id}.selection.audio`;const program = sealSpeechSpineProgram({contract: "svml.speech-spine-program@1",id,frameRate: rate,});- const request = sealMediaSelectionRequest({+ const videoRequest = sealMediaSelectionRequest({contract: "svml.media-selection-request@1",video: { mode: "primary-moving" },audio: { mode: "default" },spanAuthority: "video",frameRate: rate,});+ const audioRequest = sealMediaSelectionRequest({+ contract: "svml.media-selection-request@1",+ video: { mode: "none" },+ audio: { mode: "default" },+ spanAuthority: "audio",+ frameRate: rate,+ });const assembly = createSpeechSpineFragment({name: `@narratage/speech-spine/surface/${id}@1`,- takes: declaredTakes.map(({ mediaName, segmentName }) => ({ mediaName, segmentName })),+ takes: declaredTakes.map(({ mediaName, segmentName, visual }) => ({+ mediaName,+ segmentName,+ ...(visual === undefined ? {} : { visual: {+ frameName: visual.frameName,+ fitName: visual.fitName,+ visualSpecName: visual.visualSpecName,+ } }),+ })),});const normalizationComponents = declaredTakes.flatMap((take, index) => take.sourceKind === "media" ? [] : [{id: `${id}.normalize.${String(index + 1).padStart(4, "0")}`,fragment: synchronizedMediaFragment.id,- inputs: { source: take.source.ref, request: { kind: "record" as const, id: requestId } },+ inputs: {+ source: take.source.ref,+ request: { kind: "record" as const, id: take.sourceKind === "audio" ? audioRequestId : videoRequestId },+ },outputs: { media: `${id}.normalized.${String(index + 1).padStart(4, "0")}` },range: take.range,}]);const normalizationByTake = new Map(normalizationComponents.map((component) => [component.id.split(".").at(-1), component]));+ const emittedFits = new Set<string>();+ const emittedSpecs = new Set<string>();+ const visualRecords = declaredTakes.flatMap((take) => {+ if (take.visual === undefined) return [];+ const records = [];+ if (!emittedFits.has(take.visual.fitName)) {+ emittedFits.add(take.visual.fitName);+ records.push({+ id: `${id}.${take.visual.fitName}`,+ type: spatialTypes.fit,+ value: { kind: "inline" as const, value: decodeContentFitProperties(+ take.visual.appearance.properties,+ `Speech visual Recipe ${take.visual.appearance.path}`,+ ) },+ range: take.range,+ });+ }+ if (!emittedSpecs.has(take.visual.visualSpecName)) {+ emittedSpecs.add(take.visual.visualSpecName);+ records.push({+ id: `${id}.${take.visual.visualSpecName}`,+ type: speechSpineTypes.visualSpec,+ value: { kind: "inline" as const, value: sealSpeechSpineVisualSpec({+ contract: "svml.speech-spine-visual-spec@1",+ stackingOrder: take.visual.stackingOrder,+ }) },+ range: take.range,+ });+ }+ return records;+ });+ const hasVideoNormalization = declaredTakes.some((take) => take.sourceKind === "video");+ const hasAudioNormalization = declaredTakes.some((take) => take.sourceKind === "audio");return {records: [{ id: programId, type: speechSpineTypes.spineProgram, value: { kind: "inline", value: program }, range: element.range },- ...(normalizationComponents.length === 0 ? [] : [{ id: requestId, type: { module: { name: "@narratage/media-pipeline", version: "1" }, name: "MediaSelectionRequest" }, value: { kind: "inline" as const, value: request }, range: element.range }]),+ ...(hasVideoNormalization ? [{ id: videoRequestId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline" as const, value: videoRequest }, range: element.range }] : []),+ ...(hasAudioNormalization ? [{ id: audioRequestId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline" as const, value: audioRequest }, range: element.range }] : []),+ ...visualRecords,],components: [...normalizationComponents,@@ -143,7 +287,6 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, refragment: assembly.id,inputs: {program: { kind: "record", id: programId },- canvas: canvas.ref,...Object.fromEntries(declaredTakes.flatMap((take, index) => [[take.mediaName, take.sourceKind === "media" ? take.source.ref : {kind: "component-output" as const,@@ -151,6 +294,11 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, reoutput: "media",}],[take.segmentName, take.segment.ref],+ ...(take.visual === undefined ? [] : [+ [take.visual.frameName, take.visual.frame.ref],+ [take.visual.fitName, { kind: "record" as const, id: `${id}.${take.visual.fitName}` }],+ [take.visual.visualSpecName, { kind: "record" as const, id: `${id}.${take.visual.visualSpecName}` }],+ ]),])),},outputs: {diff --git a/packages/speech-spine/src/types.ts b/packages/speech-spine/src/types.tsindex 806cfafa..7c8b1ee2 100644--- a/packages/speech-spine/src/types.ts+++ b/packages/speech-spine/src/types.ts@@ -1,5 +1,6 @@import type { NarrativeExcerpt } from "@narratage/narrative";import type { SynchronizedMedia } from "@narratage/media";+import type { ContentFit, SpatialFrame } from "@narratage/spatial";/** Author-owned policy for one ordered Speech Spine. */export type SpeechSpineProgram = {@@ -14,6 +15,16 @@ export type SpeechSpineProgram = {export type SpeechSpineTake = {readonly segment: NarrativeExcerpt;readonly media: SynchronizedMedia;+ readonly visual?: {+ readonly frame: SpatialFrame;+ readonly fit: ContentFit;+ readonly stackingOrder: number;+ };+};++export type SpeechSpineVisualSpec = {+ readonly contract: "svml.speech-spine-visual-spec@1";+ readonly stackingOrder: number;};/**@@ -28,6 +39,11 @@ export type SpeechSpineSet = {export type SpeechSpineInput = {readonly mediaName: string;readonly segmentName: string;+ readonly visual?: {+ readonly frameName: string;+ readonly fitName: string;+ readonly visualSpecName: string;+ };};export type SpeechSpineFragmentOptions = {diff --git a/packages/speech-spine/test/program.test.ts b/packages/speech-spine/test/program.test.tsnew file mode 100644index 00000000..2aff12b7--- /dev/null+++ b/packages/speech-spine/test/program.test.ts@@ -0,0 +1,89 @@+import assert from "node:assert/strict";+import test from "node:test";++import type { SynchronizedMedia, TimelineAudio } from "@narratage/media";+import type { NarrativeExcerpt } from "@narratage/narrative";+import { digestOf } from "@narratage/protocol";+import {+ appendSpeechSpineAudioTake,+ appendSpeechSpineVisualTake,+ assembleSpeechBasis,+ createSpeechSpineSet,+ sealSpeechSpineProgram,+ sealSpeechSpineVisualSpec,+} from "@narratage/speech-spine";++const frame = { contract: "svml.spatial-frame@1" as const, xPx: 40, yPx: 80, widthPx: 640, heightPx: 900 };+const fit = {+ contract: "svml.content-fit@1" as const,+ sizing: "cover" as const,+ framePoint: { x: 0.5, y: 0.5 },+ contentPoint: { x: 0.5, y: 0.5 },+ offsetPx: { x: 0, y: 0 },+ constraint: "bounded" as const,+};++function segment(id: string, index: number): NarrativeExcerpt {+ return { contract: "svml.narrative-excerpt@1", kind: "segment", id, tokenStart: index, tokenEndExclusive: index + 1 };+}++function synchronized(id: string, visual: boolean): SynchronizedMedia {+ return {+ contract: "svml.synchronized-media@1",+ timeline: {+ frameRate: { numerator: 30, denominator: 1 },+ frameCount: 30,+ },+ ...(visual ? { visual: {+ artifact: { kind: "blob" as const, digest: digestOf(`${id}:video`), size: 1, mediaType: "video/mp4" },+ width: 720,+ height: 1280,+ } } : {}),+ audio: {+ artifact: { kind: "blob", digest: digestOf(`${id}:audio`), size: 1, mediaType: "audio/wav" },+ },+ };+}++test("audio Takes lengthen the speech program without inventing a visual clip", () => {+ const program = sealSpeechSpineProgram({+ contract: "svml.speech-spine-program@1",+ id: "speech",+ frameRate: { numerator: 30, denominator: 1 },+ });+ let set = createSpeechSpineSet();+ set = appendSpeechSpineAudioTake(set, program, synchronized("voice", false), segment("voiceover", 0));+ set = appendSpeechSpineVisualTake(+ set,+ program,+ synchronized("presenter", true),+ segment("answer", 1),+ frame,+ fit,+ sealSpeechSpineVisualSpec({ contract: "svml.speech-spine-visual-spec@1", stackingOrder: 30 }),+ );+ const audio: TimelineAudio = {+ contract: "svml.timeline-audio@1",+ artifact: { kind: "blob", digest: digestOf("speech:mix"), size: 1, mediaType: "audio/wav" },+ codec: "pcm_s16le",+ sampleRate: 48_000,+ channels: 2,+ sampleFrames: 96_000,+ loudness: "planned",+ };+ const basis = assembleSpeechBasis(program, set, audio);+ assert.deepEqual(basis.segments.map((item) => item.segmentId), ["voiceover", "answer"]);+ assert.equal(basis.programSpace.durationSec, 2);+ assert.equal(basis.visualTrack.clips.length, 1);+ assert.deepEqual(basis.visualTrack.clips[0], {+ segmentId: "answer",+ span: { startFrame: 30, endFrameExclusive: 60 },+ artifact: synchronized("presenter", true).visual!.artifact,+ extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },+ frameRate: { numerator: 30, denominator: 1 },+ frameCount: 30,+ frame,+ fit,+ stackingOrder: 30,+ });+});diff --git a/packages/speech-spine/test/surface.test.ts b/packages/speech-spine/test/surface.test.tsindex 1133ef26..e151f555 100644--- a/packages/speech-spine/test/surface.test.ts+++ b/packages/speech-spine/test/surface.test.ts@@ -29,6 +29,7 @@ import {import { speechBasisManifest, speechBasisProducers } from "@narratage/speech-basis";import { textComponent, textManifest } from "@narratage/text";import { mediaTrackManifest } from "@narratage/media-track";+import { svsManifest, svsModuleRef, svsRecipeType } from "@narratage/svs";import {decodeSpeechSpineSurface,speechSpineManifest,@@ -38,6 +39,7 @@ import {} from "@narratage/speech-spine";import {decodeCanvasSurface,+ decodeFrameSurface,spatialComponent,spatialModuleRef,spatialSurfaceDigests,@@ -54,10 +56,10 @@ const fixtureManifest: ModuleManifest = {format: "svml.module@1",name: fixtureModule.name,version: fixtureModule.version,- dependencies: [artifactDependency, mediaDependency],+ dependencies: [artifactDependency, mediaDependency, { module: svsModuleRef, digest: digestOf(svsManifest) }],types: [], capabilities: [], producers: [],surfaces: [{- name: "media", tag: "Media", mode: "structured", outputs: [artifactTypes.blob],+ name: "media", tag: "Media", mode: "structured", outputs: [artifactTypes.blob, svsRecipeType],implementation: { kind: "trusted-frontend-surface", locator: "example.speech-media/surface", digest: fixtureSurfaceDigest },}],};@@ -75,6 +77,7 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio plan...videoContractManifests,mediaPipelineManifest,mediaTrackManifest,+ svsManifest,speechBasisManifest,speechSpineManifest,textManifest,@@ -85,12 +88,23 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio plansurfaces.registerRaw(scriptModuleRef, "script", scriptSurfaceImplementationDigest, decodeScriptSurface);surfaces.registerStructured(speechSpineModuleRef, "spine", speechSpineSurfaceImplementationDigest, decodeSpeechSpineSurface);surfaces.registerStructured(spatialModuleRef, "canvas", spatialSurfaceDigests.canvas, decodeCanvasSurface);+ surfaces.registerStructured(spatialModuleRef, "frame", spatialSurfaceDigests.frame, decodeFrameSurface);surfaces.registerStructured(fixtureModule, "media", fixtureSurfaceDigest, ({ element }) => ({- records: ["take-one", "take-two"].map((id) => ({- id, type: artifactTypes.blob,- value: { kind: "blob" as const, digest: digestOf(id), size: 128, mediaType: "video/mp4" },- range: element.range,- })),+ records: [+ ...["take-one", "take-two", "voice-one"].map((id) => ({+ id, type: artifactTypes.blob,+ value: { kind: "blob" as const, digest: digestOf(id), size: 128,+ mediaType: id === "voice-one" ? "audio/mpeg" : "video/mp4" },+ range: element.range,+ })),+ {+ id: "speech-style", type: svsRecipeType,+ value: { kind: "inline" as const, value: {+ contract: "svml.svs-recipe@1", path: "speech.base", properties: { fit: "cover" },+ } },+ range: element.range,+ },+ ],components: [], fragments: [],}));const frontends = new AuthorFrontendRegistry();@@ -119,8 +133,10 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio plan</script><fixture:Media/><space:Canvas id="vertical" width="720" height="1280"/>- <speech:Spine id="speech" canvas={vertical} frame-rate="30">- <speech:Take video={take-one} segment={story.segment.opening}/>+ <space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>+ <speech:Spine id="speech" frame-rate="30"+ visual-frame={speech-frame} visual-appearance={speech-style} visual-z="0">+ <speech:Take audio={voice-one} segment={story.segment.opening}/><speech:Take video={take-two} segment={story.segment.answer}/></speech:Spine></svml>`),@@ -138,7 +154,8 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio planconst names = build.plan.steps.map((step) => step.producer.name);assert.equal(names.filter((name) => name === mediaPipelineProducers.inspect.name).length, 2);assert.equal(names.filter((name) => name === mediaPipelineProducers.normalize.name).length, 2);- assert.equal(names.filter((name) => name === speechSpineProducers.appendTake.name).length, 2);+ assert.equal(names.filter((name) => name === speechSpineProducers.appendAudioTake.name).length, 1);+ assert.equal(names.filter((name) => name === speechSpineProducers.appendVisualTake.name).length, 1);assert.equal(names.filter((name) => name === speechSpineProducers.compileAudio.name).length, 1);assert.equal(names.filter((name) => name === mediaPipelineProducers.renderAudio.name).length, 1);assert.equal(names.filter((name) => name === speechBasisProducers.projectVisual.name).length, 1);diff --git a/packages/speech-spine/test/video-pipeline-fixture.ts b/packages/speech-spine/test/video-pipeline-fixture.tsdeleted file mode 100644index ddcef5d7..00000000--- a/packages/speech-spine/test/video-pipeline-fixture.ts+++ /dev/null@@ -1,504 +0,0 @@-import { videoContractManifests } from "../../test-support/video-domain.js";-import {- captionManifest,- captionTypes,- captionValidatorDigests,- plannedCaptionTimingFragment,- resolveCaptionProgram,- sealCaptionPlan,- sealCaptionStyle,-} from "@narratage/caption";-import { narrativeDependency, narrativeTypes } from "@narratage/narrative";-import { speechDependency, speechTypes } from "@narratage/speech";-import { sealText, textDependency, textImplementationDigests, textManifest, textTypes } from "@narratage/text";-import {- createResolvedClosure,- canonicalize,- digestOf,- link,- sealBuildRequest,- sealCompiledGraph,- sealRecord,- sealTypeValidationReceipt,- sealTypedModule,- start,-} from "@narratage/core";-import {- bindAuthorFragment,- bindCandidateFragment,- elaborateGraphFragment,- mergeFragmentContributions,- sealGraphFragment,-} from "@narratage/elaborator";-import type {- BuildState,- CapabilityRef,- CompiledGraph,- LinkedProgram,- ModuleManifest,- ProducerRef,- Satisfaction,- TypeRef,- ValueSchema,-} from "@narratage/protocol";-import { resolveRealization, sealRealizationOverlay } from "@narratage/run";-import { spatialTypes, spatialValidatorDigests } from "@narratage/spatial";-import {- captionCorrespondence,- captionDisplaySequence,- narrativeValue,- parseScript,-} from "@narratage/script";-import { speechAlignmentManifest } from "@narratage/speech-alignment";-import { speechBasisManifest, speechBasisProjectionFragment } from "@narratage/speech-basis";-import { mediaPipelineManifest } from "@narratage/media-pipeline";-import { mediaTrackManifest } from "@narratage/media-track";-import {- whisperXManifest,- whisperXSpeechAlignmentFragment,- whisperXTypes,-} from "@narratage/whisperx";--export const videoModule = { name: "example.video-pipeline", version: "0.0.0" } as const;--export const videoTypes = {- estimate: { module: videoModule, name: "OfficialSpeechDurationEstimate" },- seedanceMiniMedia: { module: videoModule, name: "SeedanceMiniSpeechMedia" },-} satisfies Record<string, TypeRef>;--export const videoCapabilities = {- estimate: { module: videoModule, name: "official-speech-duration-estimate" },- seedanceMini: { module: videoModule, name: "seedance-mini-speech-media" },-} satisfies Record<string, CapabilityRef>;--export const videoProducers = {- requestEstimate: { module: videoModule, name: "request-official-speech-estimate" },- placeholderEstimate: { module: videoModule, name: "placeholder-speech-estimate" },- requestSeedanceMini: { module: videoModule, name: "request-seedance-mini-speech" },- assembleBasis: { module: videoModule, name: "assemble-speech-basis" },-} satisfies Record<string, ProducerRef>;--export const videoImplementations = Object.fromEntries(- Object.entries(videoProducers).map(([name]) => [name, digestOf(`example.video-pipeline/${name}@1`)]),-) as Readonly<Record<keyof typeof videoProducers, ReturnType<typeof digestOf>>>;--const string = { kind: "string", minLength: 1 } as const;-const number = { kind: "number", minimum: 0 } as const;-const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema }>>): ValueSchema => ({- kind: "object",- fields,-});--export const videoManifest: ModuleManifest = {- format: "svml.module@1",- name: videoModule.name,- version: videoModule.version,- dependencies: [- narrativeDependency,- speechDependency,- textDependency,- ],- types: [- { name: videoTypes.estimate.name, schema: object({ durationSec: { schema: number } }) },- {- name: videoTypes.seedanceMiniMedia.name,- schema: object({- model: { schema: { kind: "literal", value: "mini" } },- audioDigest: { schema: string },- visualDigest: { schema: string },- durationSec: { schema: number },- }),- },- ],- capabilities: [- { name: videoCapabilities.estimate.name, returns: videoTypes.estimate },- { name: videoCapabilities.seedanceMini.name, returns: videoTypes.seedanceMiniMedia },- ],- surfaces: [],- producers: [- {- name: videoProducers.requestEstimate.name,- inputs: [{ name: "speech", type: textTypes.text }],- outputs: [],- needs: [{- name: "estimate",- capability: videoCapabilities.estimate,- returns: videoTypes.estimate,- }],- implementation: {- kind: "registered",- locator: "example.video-pipeline/request-estimate",- digest: videoImplementations.requestEstimate,- },- },- {- name: videoProducers.requestSeedanceMini.name,- inputs: [- { name: "dialogue", type: textTypes.text },- { name: "estimate", type: videoTypes.estimate },- ],- outputs: [],- needs: [{- name: "media",- capability: videoCapabilities.seedanceMini,- returns: videoTypes.seedanceMiniMedia,- }],- implementation: {- kind: "registered",- locator: "example.video-pipeline/request-seedance-mini",- digest: videoImplementations.requestSeedanceMini,- },- },- {- name: videoProducers.placeholderEstimate.name,- inputs: [{ name: "speech", type: textTypes.text }],- outputs: [{ name: "estimate", type: videoTypes.estimate }],- needs: [],- implementation: {- kind: "registered",- locator: "example.video-pipeline/placeholder-estimate",- digest: videoImplementations.placeholderEstimate,- },- },- {- name: videoProducers.assembleBasis.name,- inputs: [- { name: "narrative", type: narrativeTypes.narrative },- { name: "media", type: videoTypes.seedanceMiniMedia },- ],- outputs: [{- name: "basis",- type: speechTypes.basis,- }],- needs: [],- implementation: {- kind: "registered",- locator: "example.video-pipeline/assemble-basis",- digest: videoImplementations.assembleBasis,- },- },- ],-};--const input = (name: string) => ({ kind: "fragment-input" as const, name });-const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });--const durationEstimateFragment = sealGraphFragment({- name: "example.video-pipeline/duration-estimate@1",- inputs: [{ name: "speech", type: textTypes.text }],- operations: [{- id: "estimate",- producer: videoProducers.requestEstimate,- inputs: { speech: input("speech") },- result: { kind: "need", name: "estimate", accepts: "exact" },- }],- exports: [{- name: "estimate",- type: videoTypes.estimate,- root: operation("estimate"),- semanticInputs: ["speech"],- fidelity: "exact",- }],-});--const speechBasisGenerationFragment = sealGraphFragment({- name: "example.video-pipeline/seedance-mini-speech-basis@1",- inputs: [- { name: "narrative", type: narrativeTypes.narrative },- { name: "dialogue", type: textTypes.text },- { name: "estimate", type: videoTypes.estimate },- ],- operations: [- {- id: "request-seedance-mini",- producer: videoProducers.requestSeedanceMini,- inputs: { dialogue: input("dialogue"), estimate: input("estimate") },- result: { kind: "need", name: "media", accepts: "exact" },- },- {- id: "assemble-take",- producer: videoProducers.assembleBasis,- inputs: { narrative: input("narrative"), media: operation("request-seedance-mini") },- result: { kind: "output", name: "basis" },- },- ],- exports: [{- name: "take",- type: speechTypes.basis,- root: operation("assemble-take"),- semanticInputs: ["narrative", "dialogue", "estimate"],- fidelity: "exact",- }],-});--const placeholderEstimateFragment = sealGraphFragment({- name: "example.video-pipeline/placeholder-estimate@1",- inputs: [{ name: "speech", type: textTypes.text }],- operations: [{- id: "placeholder",- producer: videoProducers.placeholderEstimate,- inputs: { speech: input("speech") },- result: { kind: "output", name: "estimate" },- }],- exports: [{- name: "estimate",- type: videoTypes.estimate,- root: operation("placeholder"),- semanticInputs: ["speech"],- fidelity: "substitute",- }],-});--export const videoOutputs = {- estimate: "speech.estimate",- take: "speech.take",- programSpace: "speech.program-space",- audio: "speech.audio",- audioTrack: "speech.audio-track",- visual: "speech.visual",- rawEvidence: "speech.whisperx-evidence",- evidence: "speech.evidence",- map: "speech.map",- caption: "speech.caption",-} as const;--export type VideoOutputName = keyof typeof videoOutputs;--export const videoClosure = createResolvedClosure([- ...videoContractManifests,- textManifest,- videoManifest,- mediaTrackManifest,- speechBasisManifest,- mediaPipelineManifest,- whisperXManifest,- speechAlignmentManifest,- captionManifest,-]);--function authorProgram(): LinkedProgram {- const parsed = parseScript("pipeline.svml", "<line><that was insane | what the fuck></line>");- const display = captionDisplaySequence(parsed, "caption:display");- const correspondence = captionCorrespondence(parsed, display.id);- const style = sealCaptionStyle({- contract: "svml.caption-style@1",- id: "fixture",- planning: {- cue: { minimumWords: 1, maximumWords: 7, instruction: "Use one authored Atom." },- fields: [],- },- rendering: { family: "fixture-caption@1", parameters: {} },- });- const captionProgram = resolveCaptionProgram(display, "caption-program", style, []);- const captionPlan = sealCaptionPlan({- contract: "svml.caption-plan@1",- runs: [{- id: captionProgram.runs[0]!.id,- styleId: style.id,- cues: [{ id: "caption:cue:1", atomIds: display.atoms.map((atom) => atom.id), fields: [] }],- }],- });- const origin = {- kind: "authored" as const,- sourceDigest: digestOf("source:video-pipeline"),- frontendClosureDigest: digestOf("frontend:script"),- sourceName: "pipeline.svml",- };- const narrative = sealRecord({- id: "narrative:root",- type: narrativeTypes.narrative,- value: { kind: "inline", value: narrativeValue(parsed) },- conformance: "exact",- origin,- });- const rawSpeech = sealRecord({- id: "narrative:speech",- type: textTypes.text,- value: { kind: "inline", value: sealText(parsed.serializations.speech) as never },- conformance: "exact",- origin,- });- const speech = { ...rawSpeech, validation: sealTypeValidationReceipt({- type: rawSpeech.type,- recordDigest: rawSpeech.digest,- validatorDigest: textImplementationDigests.validateText,- }) };- const rawDialogue = sealRecord({- id: "narrative:dialogue",- type: textTypes.text,- value: { kind: "inline", value: sealText(parsed.serializations.dialogue) as never },- conformance: "exact",- origin,- });- const dialogue = { ...rawDialogue, validation: sealTypeValidationReceipt({- type: rawDialogue.type,- recordDigest: rawDialogue.digest,- validatorDigest: textImplementationDigests.validateText,- }) };- const programRecord = sealRecord({ id: "caption:program", type: captionTypes.program,- value: { kind: "inline", value: canonicalize(captionProgram) }, conformance: "exact", origin });- const planRecord = sealRecord({ id: "caption:plan", type: captionTypes.plan,- value: { kind: "inline", value: canonicalize(captionPlan) }, conformance: "exact", origin });- const captionRecords = [- sealRecord({ id: "caption:display", type: narrativeTypes.captionDisplay,- value: { kind: "inline", value: canonicalize(display) }, conformance: "exact", origin }),- sealRecord({ id: "caption:correspondence", type: narrativeTypes.captionCorrespondence,- value: { kind: "inline", value: canonicalize(correspondence) }, conformance: "exact", origin }),- { ...programRecord, validation: sealTypeValidationReceipt({- type: programRecord.type,- recordDigest: programRecord.digest,- validatorDigest: captionValidatorDigests.program,- }) },- { ...planRecord, validation: sealTypeValidationReceipt({- type: planRecord.type,- recordDigest: planRecord.digest,- validatorDigest: captionValidatorDigests.plan,- }) },- ];- const rawCanvas = sealRecord({- id: "canvas:vertical",- type: spatialTypes.canvas,- value: { kind: "inline", value: {- contract: "svml.canvas-space@1",- widthPx: 720,- heightPx: 1280,- origin: "top-left",- xDirection: "right",- yDirection: "down",- pixelAspect: "square",- } },- conformance: "exact",- origin,- });- const canvas = {- ...rawCanvas,- validation: sealTypeValidationReceipt({- type: rawCanvas.type,- recordDigest: rawCanvas.digest,- validatorDigest: spatialValidatorDigests.canvas,- }),- };- return link(videoClosure, [sealTypedModule({- id: "author:video-pipeline",- closureDigest: videoClosure.digest,- records: [narrative, speech, dialogue, canvas, ...captionRecords],- })]);-}--export function createVideoGraph(program: LinkedProgram): CompiledGraph {- const narrative = { kind: "record" as const, id: "narrative:root" };- const speech = { kind: "record" as const, id: "narrative:speech" };- const dialogue = { kind: "record" as const, id: "narrative:dialogue" };- const estimate = elaborateGraphFragment(program, durationEstimateFragment, {- id: "opening.estimate",- fragment: durationEstimateFragment.id,- inputs: { speech },- });- const take = elaborateGraphFragment(program, speechBasisGenerationFragment, {- id: "opening.seedance-mini",- fragment: speechBasisGenerationFragment.id,- inputs: {- narrative,- dialogue,- estimate: { kind: "logical-output", id: videoOutputs.estimate },- },- });- const projections = elaborateGraphFragment(program, speechBasisProjectionFragment, {- id: "opening.take-projections",- fragment: speechBasisProjectionFragment.id,- inputs: {- basis: { kind: "logical-output", id: videoOutputs.take },- canvas: { kind: "record", id: "canvas:vertical" },- },- });- const alignment = elaborateGraphFragment(program, whisperXSpeechAlignmentFragment, {- id: "opening.whisperx-alignment",- fragment: whisperXSpeechAlignmentFragment.id,- inputs: {- narrative,- audio: { kind: "logical-output", id: videoOutputs.audio },- },- });- const caption = elaborateGraphFragment(program, plannedCaptionTimingFragment, {- id: "opening.caption",- fragment: plannedCaptionTimingFragment.id,- inputs: {- display: { kind: "record", id: "caption:display" },- correspondence: { kind: "record", id: "caption:correspondence" },- map: { kind: "logical-output", id: videoOutputs.map },- program: { kind: "record", id: "caption:program" },- plan: { kind: "record", id: "caption:plan" },- },- });- const contribution = mergeFragmentContributions(- { outputs: [], candidates: [], operations: [] },- bindAuthorFragment(estimate, { estimate: videoOutputs.estimate }),- bindAuthorFragment(take, { take: videoOutputs.take }),- bindAuthorFragment(projections, {- programSpace: videoOutputs.programSpace,- audio: videoOutputs.audio,- audioTrack: videoOutputs.audioTrack,- visual: videoOutputs.visual,- }),- bindAuthorFragment(alignment, {- rawEvidence: videoOutputs.rawEvidence,- evidence: videoOutputs.evidence,- map: videoOutputs.map,- }),- bindAuthorFragment(caption, { caption: videoOutputs.caption }),- );- return sealCompiledGraph({ program: program.semanticDigest, ...contribution });-}--export type VideoBuildOptions = {- readonly estimateRealization?: "primary" | "placeholder";- readonly goalAccepts?: "exact" | "substitute";- readonly targets?: readonly VideoOutputName[];-};--export type VideoFixture = {- readonly program: LinkedProgram;- readonly source: CompiledGraph;- readonly graph: CompiledGraph;- readonly satisfactions: readonly Satisfaction[];-};--export function createVideoFixture(options: VideoBuildOptions = {}): VideoFixture {- const program = authorProgram();- const source = createVideoGraph(program);- if (options.estimateRealization !== "placeholder") {- return { program, source, graph: source, satisfactions: [] };- }- const placeholder = elaborateGraphFragment(program, placeholderEstimateFragment, {- id: "opening.placeholder-estimate",- fragment: placeholderEstimateFragment.id,- inputs: { speech: { kind: "record", id: "narrative:speech" } },- });- const contribution = bindCandidateFragment(placeholder, { estimate: videoOutputs.estimate });- const overlay = sealRealizationOverlay({- sourceGraph: source.id,- candidates: contribution.candidates,- operations: contribution.operations,- });- const realized = resolveRealization(program, source, [overlay]);- return {- program,- source,- graph: realized.graph,- satisfactions: contribution.satisfactions ?? [],- };-}--export function createVideoBuild(options: VideoBuildOptions = {}): BuildState {- const fixture = createVideoFixture(options);- const accepts = options.goalAccepts ?? "exact";- const targets = options.targets ?? ["caption"];- const request = sealBuildRequest({- graph: fixture.graph.id,- targets: targets.map((name) => ({ output: videoOutputs[name], accepts })),- satisfactions: fixture.satisfactions,- });- return start(fixture.program, fixture.graph, request);-}--export { whisperXTypes };diff --git a/packages/speech-spine/test/video-pipeline.test.ts b/packages/speech-spine/test/video-pipeline.test.tsdeleted file mode 100644index d3fb05b1..00000000--- a/packages/speech-spine/test/video-pipeline.test.ts+++ /dev/null@@ -1,672 +0,0 @@-import { compositionComponent, spatialComponent } from "../../test-support/video-domain.js";-import {- registerProducerFacets,- registerTypeValidatorFacets,-} from "@narratage/component-kit";-import {- captionComponent,- captionProducers,-} from "@narratage/caption";-import type { TimedCaptionProjection } from "@narratage/caption";-import type { Narrative } from "@narratage/narrative";-import type { Text } from "@narratage/text";-import { sealProgramSpace } from "@narratage/program-space";-import { sealSpeechBasis, sealSpeechEvidenceAudio, speechTypes } from "@narratage/speech";-import type { SpeechAudioBasis, SpeechBasis, SpeechEvidenceAudio } from "@narratage/speech";-import { speechEvidenceTypes } from "@narratage/speech-evidence";-import { compositionTypes, sealVisualTrack } from "@narratage/composition";-import type { VisualTrack } from "@narratage/composition";-import assert from "node:assert/strict";-import test from "node:test";--import { digestOf, sealBuildRequest, start } from "@narratage/core";-import {- ProducerRegistry,- NodeDriver,- EndpointRegistry,- parseBuildState,- serializeBuildState,-} from "@narratage/driver-node";-import type { BuildState, CapabilityRef, ProducerRef } from "@narratage/protocol";-import {- createProvidedCandidate,- resolveRealization,- sealRealizationOverlay,-} from "@narratage/run";-import {- speechAlignmentComponent,- speechAlignmentProducers,-} from "@narratage/speech-alignment";-import {- mediaPipelineCapabilities,- mediaPipelineImplementationDigests,- mediaPipelineProducers,-} from "@narratage/media-pipeline";-import type { ProjectSpeechEvidenceAudioNeed } from "@narratage/media-pipeline";-import {- projectSpeechAudio,- projectSpeechAudioImplementationDigest,- projectSpeechVisual,- projectSpeechVisualImplementationDigest,- speechBasisProducers,-} from "@narratage/speech-basis";-import {- normalizeWhisperXAlignment,- sealWhisperXAlignmentEvidence,- whisperXCapabilities,- whisperXImplementationDigests,- whisperXProducers,- whisperXRequestForEvidenceAudio,-} from "@narratage/whisperx";-import type { WhisperXAlignmentEvidence } from "@narratage/whisperx";-import { TypeValidatorRegistry, validateValue } from "@narratage/validation";--import {- createVideoBuild,- createVideoFixture,- videoCapabilities,- videoImplementations,- videoOutputs,- videoProducers,- videoTypes,- whisperXTypes,-} from "./video-pipeline-fixture.js";--function inlineObject(value: unknown): Readonly<Record<string, unknown>> {- if (value === null || Array.isArray(value) || typeof value !== "object") throw new Error("expected object");- return value as Readonly<Record<string, unknown>>;-}--function validatorRegistry(): TypeValidatorRegistry {- const registry = new TypeValidatorRegistry();- registerTypeValidatorFacets(registry, captionComponent.validators ?? []);- registerTypeValidatorFacets(registry, compositionComponent.validators);- registerTypeValidatorFacets(registry, spatialComponent.validators);- return registry;-}--function inlineValue<T>(value: unknown): T {- return value as T;-}--function sameCapability(left: CapabilityRef, right: CapabilityRef): boolean {- return left.name === right.name- && left.module.name === right.module.name- && left.module.version === right.module.version;-}--function selectedRecord(state: BuildState, output: string): string {- const selection = state.plan.selections.find((item) => item.output === output);- assert(selection, `missing selection for ${output}`);- return selection.record;-}--function producerCount(state: BuildState, producer: ProducerRef): number {- return state.plan.steps.filter((step) =>- step.producer.name === producer.name- && step.producer.module.name === producer.module.name- && step.producer.module.version === producer.module.version).length;-}--test("the Speech Spine pipeline resumes without repeating paid calls", async () => {- const host = new ProducerRegistry();- const endpoints = new EndpointRegistry();- const calls = {- estimate: 0,- seedance: 0,- assemble: 0,- projectAudio: 0,- projectVisual: 0,- evidenceAudioRequest: 0,- whisperRequest: 0,- whisperNormalize: 0,- };-- host.registerProducer(videoProducers.requestEstimate, videoImplementations.requestEstimate, ({ inputs }) => {- calls.estimate += 1;- assert.equal(inputs.speech?.value.kind, "inline");- const speech = inlineValue<Text>(inputs.speech.value.value);- return {- outputs: {},- needs: {- estimate: {- contract: "example.official-speech-estimate-request@1",- speech: speech.value,- },- },- };- });- host.registerProducer(videoProducers.requestSeedanceMini, videoImplementations.requestSeedanceMini, ({ inputs }) => {- calls.seedance += 1;- assert.equal(inputs.dialogue?.value.kind, "inline");- assert.equal(inputs.estimate?.value.kind, "inline");- const dialogue = inlineValue<Text>(inputs.dialogue.value.value);- const estimate = inlineObject(inputs.estimate.value.value);- return {- outputs: {},- needs: {- media: {- contract: "example.seedance-mini-speech-request@1",- model: "mini",- script: dialogue.value,- durationSec: estimate.durationSec as number,- },- },- };- });- host.registerProducer(videoProducers.assembleBasis, videoImplementations.assembleBasis, ({ inputs }) => {- calls.assemble += 1;- assert.equal(inputs.narrative?.value.kind, "inline");- assert.equal(inputs.media?.value.kind, "inline");- const narrative = inlineValue<Narrative>(inputs.narrative.value.value);- const media = inlineObject(inputs.media.value.value);- const durationSec = media.durationSec as number;- const programSpace = sealProgramSpace({- contract: "svml.program-space@1",- durationSec,- frameRate: { numerator: 1_000, denominator: 1 },- });- const audioDigest = media.audioDigest as SpeechBasis["audio"]["digest"];- const visualDigest = media.visualDigest as SpeechBasis["audio"]["digest"];- const basis = sealSpeechBasis({- contract: "svml.speech-basis@1",- programSpace,- audio: { kind: "blob", digest: audioDigest, size: 1, mediaType: "audio/wav" },- visualTrack: { clips: [{- segmentId: "line",- artifact: { kind: "blob", digest: visualDigest, size: 1, mediaType: "video/mp4" },- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },- frameRate: { ...programSpace.frameRate },- frameCount: Math.round(durationSec * 1_000),- }] },- segments: [{ segmentId: "line", startSec: 0, endSec: durationSec }],- });- return { outputs: { basis: { kind: "inline", value: basis } }, needs: {} };- });- host.registerProducer(- speechBasisProducers.projectAudio,- projectSpeechAudioImplementationDigest,- ({ inputs }) => {- calls.projectAudio += 1;- assert.equal(inputs.basis?.value.kind, "inline");- return {- outputs: {- audio: {- kind: "inline",- value: projectSpeechAudio(inlineValue<SpeechBasis>(inputs.basis.value.value)),- },- },- needs: {},- };- },- );- host.registerProducer(- mediaPipelineProducers.projectSpeechEvidenceAudio,- mediaPipelineImplementationDigests.projectSpeechEvidenceAudio,- ({ inputs }) => {- calls.evidenceAudioRequest += 1;- assert.equal(inputs.audio?.value.kind, "inline");- const audio = inlineValue<SpeechAudioBasis>(inputs.audio.value.value);- const sourceSampleFrames = Math.round(audio.programSpace.durationSec * 48_000);- const need: ProjectSpeechEvidenceAudioNeed = {- contract: "svml.project-speech-evidence-audio-request@1",- source: {- kind: "blob",- digest: audio.audio.digest,- size: audio.audio.size,- mediaType: audio.audio.mediaType,- },- sourceSampleRate: 48_000,- sourceChannels: 2,- sourceCodec: "pcm_s16le",- sourceSampleFrames,- evidenceSampleRate: 16_000,- evidenceChannels: 1,- evidenceCodec: "pcm_s16le",- evidenceSampleFrames: Math.round(sourceSampleFrames / 3),- durationSec: audio.programSpace.durationSec,- segments: audio.segments,- };- return { outputs: {}, needs: { evidenceAudio: need } };- },- );- host.registerProducer(- speechBasisProducers.projectVisual,- projectSpeechVisualImplementationDigest,- ({ inputs }) => {- calls.projectVisual += 1;- assert.equal(inputs.basis?.value.kind, "inline");- assert(inputs.canvas?.value.kind === "inline");- return {- outputs: {- visual: {- kind: "inline",- value: projectSpeechVisual(- inlineValue<SpeechBasis>(inputs.basis.value.value),- inlineValue(inputs.canvas.value.value),- ),- },- },- needs: {},- };- },- );- host.registerProducer(whisperXProducers.request, whisperXImplementationDigests.request, ({ inputs }) => {- calls.whisperRequest += 1;- assert.equal(inputs.audio?.value.kind, "inline");- return {- outputs: {},- needs: {- alignment: whisperXRequestForEvidenceAudio(- inlineValue<SpeechEvidenceAudio>(inputs.audio.value.value),- { language: "en" },- ),- },- };- });- host.registerProducer(whisperXProducers.normalize, whisperXImplementationDigests.normalize, ({ inputs }) => {- calls.whisperNormalize += 1;- assert.equal(inputs.whisperx?.value.kind, "inline");- const evidence = normalizeWhisperXAlignment(- inlineValue<WhisperXAlignmentEvidence>(inputs.whisperx.value.value),- );- return { outputs: { evidence: { kind: "inline", value: evidence } }, needs: {} };- });- registerProducerFacets(host, speechAlignmentComponent.producers);- registerProducerFacets(host, captionComponent.producers);-- const driver = new NodeDriver({ producers: host, endpoints, validators: validatorRegistry() });- const atEstimate = await driver.run(createVideoBuild());- assert.equal(atEstimate.blocked[0]?.reason, "missing-endpoint");- assert.match(atEstimate.blocked[0]?.subject ?? "", /OfficialSpeechDurationEstimate/u);-- endpoints.registerImmediateEndpoint("runtime:official-estimate", videoCapabilities.estimate, videoTypes.estimate, () => ({- value: { kind: "inline", value: { durationSec: 1 } },- conformance: "exact",- delivery: "executed",- metadata: {},- }));- const atSeedance = await driver.run(parseBuildState(serializeBuildState(atEstimate.state)));- assert.equal(atSeedance.blocked[0]?.reason, "missing-endpoint");- assert.match(atSeedance.blocked[0]?.subject ?? "", /SeedanceMiniSpeechMedia/u);-- endpoints.registerImmediateEndpoint(- "runtime:kie-seedance-mini",- videoCapabilities.seedanceMini,- videoTypes.seedanceMiniMedia,- ({ need }) => {- assert.deepEqual(need.constraints, {- contract: "example.seedance-mini-speech-request@1",- durationSec: 1,- model: "mini",- script: "what the fuck",- });- return {- value: { kind: "inline", value: {- model: "mini",- audioDigest: digestOf("artifact:audio"),- visualDigest: digestOf("artifact:visual"),- durationSec: 1,- } },- conformance: "exact",- delivery: "executed",- metadata: { endpoint: "kie", model: "seedance-mini" },- };- },- );- const atEvidenceAudio = await driver.run(parseBuildState(serializeBuildState(atSeedance.state)));- assert.equal(atEvidenceAudio.blocked[0]?.reason, "missing-endpoint");- assert.match(atEvidenceAudio.blocked[0]?.subject ?? "", /SpeechEvidenceAudio/u);-- endpoints.registerImmediateEndpoint(- "runtime:media-local",- mediaPipelineCapabilities.projectSpeechEvidenceAudio,- speechTypes.evidenceAudio,- ({ need }) => {- const request = inlineValue<ProjectSpeechEvidenceAudioNeed>(need.constraints);- return {- value: { kind: "inline", value: sealSpeechEvidenceAudio({- contract: "svml.speech-evidence-audio@1",- artifact: {- kind: "blob",- digest: digestOf("artifact:evidence-audio"),- size: 32_044,- mediaType: "audio/wav",- },- codec: "pcm_s16le",- sampleRate: 16_000,- channels: 1,- sampleFrames: request.evidenceSampleFrames,- durationSec: request.durationSec,- segments: request.segments,- sampleMap: {- algorithm: "rational-boundary-round@1",- sourceSampleRate: 48_000,- evidenceSampleRate: 16_000,- sourceSampleFrames: request.sourceSampleFrames,- evidenceSampleFrames: request.evidenceSampleFrames,- sourceOriginSample: 0,- evidenceOriginSample: 0,- },- }) },- conformance: "exact",- delivery: "executed",- metadata: {},- };- },- );- const atWhisperX = await driver.run(parseBuildState(serializeBuildState(atEvidenceAudio.state)));- assert.equal(atWhisperX.blocked[0]?.reason, "missing-endpoint");- assert.match(atWhisperX.blocked[0]?.subject ?? "", /WhisperXAlignmentEvidence/u);-- const wrongMeasurementEndpoints = new EndpointRegistry();- wrongMeasurementEndpoints.registerImmediateEndpoint(- "runtime:wrong-measurement-whisperx",- whisperXCapabilities.alignment,- whisperXTypes.alignmentEvidence,- () => ({- value: {- kind: "inline",- value: {- contract: "invalid-whisperx-evidence",- durationSec: 1,- segments: [{ sourceSegmentId: "line", startSec: 0, endSec: 1, words: [], chars: [] }],- },- },- conformance: "exact",- delivery: "executed",- metadata: {},- }),- );- const rejectedEvidence = await new NodeDriver({- producers: host,- endpoints: wrongMeasurementEndpoints,- validators: validatorRegistry(),- }).run(- parseBuildState(serializeBuildState(atWhisperX.state)),- );- assert.equal(rejectedEvidence.status, "paused");- assert.match(rejectedEvidence.journal.at(-1)?.message ?? "", /contract/u);- const rejectedWhisperNeed = rejectedEvidence.state.needs.find((need) =>- sameCapability(need.capability, whisperXCapabilities.alignment));- assert(rejectedWhisperNeed);- assert.equal(- rejectedEvidence.state.records.some((record) =>- record.id === rejectedWhisperNeed.result),- false,- "Invalid Evidence must not enter BuildState",- );-- endpoints.registerImmediateEndpoint(- "runtime:whisperx-local",- whisperXCapabilities.alignment,- whisperXTypes.alignmentEvidence,- () => {- return {- value: {- kind: "inline",- value: sealWhisperXAlignmentEvidence({- contract: "svml.whisperx-alignment-evidence@1",- durationSec: 1,- segments: [{- sourceSegmentId: "line",- startSec: 0,- endSec: 1,- words: [- { text: "what", startSec: 0.1, endSec: 0.25, score: 0.98 },- { text: "the", startSec: 0.3, endSec: 0.42, score: 0.98 },- { text: "fuck", startSec: 0.48, endSec: 0.72, score: 0.97 },- ],- chars: [],- }],- }),- },- conformance: "exact",- delivery: "executed",- metadata: { adapter: "whisperx.local", model: "fixture" },- };- },- );- const completed = await driver.run(parseBuildState(serializeBuildState(atWhisperX.state)));-- assert.equal(completed.status, "complete");- assert.deepEqual(calls, {- estimate: 1,- seedance: 1,- assemble: 1,- projectAudio: 1,- projectVisual: 0,- evidenceAudioRequest: 1,- whisperRequest: 1,- whisperNormalize: 1,- });- assert.deepEqual(completed.state.receipts.map((receipt) => receipt.fulfiller), [- "runtime:official-estimate",- "runtime:kie-seedance-mini",- "runtime:media-local",- "runtime:whisperx-local",- ]);- const captionRecord = completed.state.records.find((record) =>- record.id === selectedRecord(completed.state, videoOutputs.caption));- assert.equal(captionRecord?.value.kind, "inline");- const value = inlineValue<TimedCaptionProjection>(- captionRecord?.value.kind === "inline" ? captionRecord.value.value : {},- );- assert.deepEqual(- [value.cues[0]?.startSec, value.cues[0]?.endSec, value.cues[0]?.atoms.length],- [0.1, 0.72, 1],- );-- const mapRecordId = selectedRecord(completed.state, videoOutputs.map);- const mapRecord = completed.state.records.find((record) => record.id === mapRecordId);- assert.equal(mapRecord?.value.kind, "inline");- const mapValue = inlineObject(mapRecord?.value.kind === "inline" ? mapRecord.value.value : {});- for (const forbiddenLineageField of [- "narrativeDigest",- "semanticIndexDigest",- "basisDigest",- "audioArtifactDigest",- "evidenceDigest",- "locatorDigest",- ]) {- assert.equal(- forbiddenLineageField in mapValue,- false,- `SemanticMap must not serialize Graph lineage as ${forbiddenLineageField}`,- );- }- assert.equal(mapRecord?.origin.kind, "derived");- const mapDerivation = completed.state.derivations.find((derivation) =>- derivation.id === (mapRecord?.origin.kind === "derived" ? mapRecord.origin.derivation : undefined));- assert(mapDerivation);- const evidenceInput = mapDerivation.inputs.find((input) => {- const record = completed.state.records.find((candidate) => candidate.id === input.id);- return record?.type.name === speechEvidenceTypes.alignedTranscript.name- && record.type.module.name === speechEvidenceTypes.alignedTranscript.module.name- && record.type.module.version === speechEvidenceTypes.alignedTranscript.module.version;- });- assert(evidenceInput, "the Map fan-in must include the internal normalized Evidence Record");- assert.deepEqual(- new Set(mapDerivation.inputs.map((input) => input.id)),- new Set([- "narrative:root",- selectedRecord(completed.state, videoOutputs.audio),- evidenceInput.id,- ]),- "the fan-in belongs to the executed Graph/Derivation, not the SemanticMap payload",- );-});--test("Targets prune official Fragments while shared SpeechBasis generation stays singular", () => {- const visual = createVideoBuild({ targets: ["visual"] });- assert.equal(producerCount(visual, videoProducers.requestEstimate), 1);- assert.equal(producerCount(visual, videoProducers.requestSeedanceMini), 1);- assert.equal(producerCount(visual, videoProducers.assembleBasis), 1);- assert.equal(producerCount(visual, speechBasisProducers.projectVisual), 1);- assert.equal(producerCount(visual, speechBasisProducers.projectAudio), 0);- assert.equal(producerCount(visual, whisperXProducers.request), 0);- assert.equal(producerCount(visual, captionProducers.temporalizePlan), 0);-- const map = createVideoBuild({ targets: ["map"] });- assert.equal(producerCount(map, speechBasisProducers.projectAudio), 1);- assert.equal(producerCount(map, speechBasisProducers.projectVisual), 0);- assert.equal(producerCount(map, whisperXProducers.request), 1);- assert.equal(producerCount(map, whisperXProducers.normalize), 1);- assert.equal(producerCount(map, speechAlignmentProducers.locate), 1);- assert.equal(producerCount(map, captionProducers.temporalizePlan), 0);-- const bothTakeProjections = createVideoBuild({ targets: ["audio", "visual"] });- assert.equal(producerCount(bothTakeProjections, videoProducers.requestSeedanceMini), 1);- assert.equal(producerCount(bothTakeProjections, videoProducers.assembleBasis), 1);- assert.equal(producerCount(bothTakeProjections, speechBasisProducers.projectAudio), 1);- assert.equal(producerCount(bothTakeProjections, speechBasisProducers.projectVisual), 1);-- const bothEvidenceExports = createVideoBuild({ targets: ["rawEvidence", "map"] });- assert.equal(producerCount(bothEvidenceExports, whisperXProducers.request), 1);- assert.equal(producerCount(bothEvidenceExports, whisperXProducers.normalize), 1);- assert.equal(producerCount(bothEvidenceExports, speechAlignmentProducers.locate), 1);-});--test("an Existing SpeechBasis cuts generation while a visual substitute cuts the whole upstream", async () => {- const fixture = createVideoFixture();- const narrativeRecord = fixture.program.records.find((record) => record.id === "narrative:root");- assert.equal(narrativeRecord?.value.kind, "inline");- const narrative = inlineValue<Narrative>(- narrativeRecord?.value.kind === "inline" ? narrativeRecord.value.value : {},- );- const programSpace = sealProgramSpace({- contract: "svml.program-space@1",- durationSec: 1,- frameRate: { numerator: 1_000, denominator: 1 },- });- const visualDigest = digestOf("existing-take:visual");- const existingTake = sealSpeechBasis({- contract: "svml.speech-basis@1",- programSpace,- audio: {- kind: "blob",- digest: digestOf("existing-take:audio"),- size: 1,- mediaType: "audio/wav",- },- visualTrack: { clips: [{- segmentId: "line",- artifact: { kind: "blob", digest: visualDigest, size: 1, mediaType: "video/mp4" },- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },- frameRate: { ...programSpace.frameRate },- frameCount: 1_000,- }] },- segments: [{ segmentId: "line", startSec: 0, endSec: 1 }],- });- const takeCandidate = createProvidedCandidate({- type: speechTypes.basis,- value: { kind: "inline", value: existingTake },- provenance: { source: "approved-library", name: "opening-v3" },- });- const takeOverlay = sealRealizationOverlay({- sourceGraph: fixture.source.id,- candidates: [takeCandidate],- operations: [],- });- const withExistingTake = resolveRealization(fixture.program, fixture.source, [takeOverlay]);- const captionFromExisting = start(fixture.program, withExistingTake.graph, sealBuildRequest({- graph: withExistingTake.graph.id,- targets: [{ output: videoOutputs.caption, accepts: "exact" }],- satisfactions: [{ output: videoOutputs.take, candidate: takeCandidate.id, fidelity: "exact" }],- }));- assert.equal(producerCount(captionFromExisting, videoProducers.requestEstimate), 0);- assert.equal(producerCount(captionFromExisting, videoProducers.requestSeedanceMini), 0);- assert.equal(producerCount(captionFromExisting, videoProducers.assembleBasis), 0);- assert.equal(producerCount(captionFromExisting, speechBasisProducers.projectAudio), 1);- assert.equal(producerCount(captionFromExisting, whisperXProducers.request), 1);- assert.equal(producerCount(captionFromExisting, captionProducers.temporalizePlan), 1);-- const black: VisualTrack = sealVisualTrack({- contract: "svml.visual-track@1",- visualIr: "svml.visual-ir@1",- id: "preview:black",- presents: [],- });- const blackValue = { kind: "inline" as const, value: black };- const blackValidation = await validateValue(- fixture.program.closure,- compositionTypes.visualTrack,- blackValue,- validatorRegistry(),- );- const blackCandidate = createProvidedCandidate({- type: compositionTypes.visualTrack,- value: blackValue,- provenance: { method: "black-preview" },- ...(blackValidation === undefined ? {} : { validation: blackValidation }),- });- const blackOverlay = sealRealizationOverlay({- sourceGraph: fixture.source.id,- candidates: [blackCandidate],- operations: [],- });- const withBlack = resolveRealization(fixture.program, fixture.source, [blackOverlay]);- const preview = start(fixture.program, withBlack.graph, sealBuildRequest({- graph: withBlack.graph.id,- targets: [{ output: videoOutputs.visual, accepts: "substitute" }],- satisfactions: [{ output: videoOutputs.visual, candidate: blackCandidate.id, fidelity: "substitute" }],- }));- assert.deepEqual(preview.plan.steps, []);- assert.equal(preview.plan.initialValues.length, 1);-});--test("an exact Endpoint result cannot wash a substitute input back to exact", async () => {- const initial = createVideoBuild({- estimateRealization: "placeholder",- goalAccepts: "substitute",- });- const host = new ProducerRegistry();- host.registerProducer(- videoProducers.placeholderEstimate,- videoImplementations.placeholderEstimate,- () => ({ outputs: { estimate: { kind: "inline", value: { durationSec: 1 } } }, needs: {} }),- );- host.registerProducer(videoProducers.requestSeedanceMini, videoImplementations.requestSeedanceMini, () => ({- outputs: {},- needs: { media: { model: "mini" } },- }));- const endpoints = new EndpointRegistry();- endpoints.registerImmediateEndpoint(- "runtime:exact-seedance",- videoCapabilities.seedanceMini,- videoTypes.seedanceMiniMedia,- () => ({- value: { kind: "inline", value: {- model: "mini",- audioDigest: digestOf("floor:audio"),- visualDigest: digestOf("floor:visual"),- durationSec: 1,- } },- conformance: "exact",- delivery: "executed",- metadata: {},- }),- );-- const result = await new NodeDriver({- producers: host,- endpoints,- validators: validatorRegistry(),- }).run(initial);- assert.equal(result.status, "paused", "the unregistered Basis assembler should pause after Seedance");- const seedanceNeed = result.state.needs.find((need) =>- sameCapability(need.capability, videoCapabilities.seedanceMini));- const seedanceReceipt = result.state.receipts.find((receipt) => receipt.need === seedanceNeed?.id);- const media = result.state.records.find((record) => record.id === seedanceReceipt?.output);- assert.equal(seedanceNeed?.conformanceFloor, "substitute");- assert.equal(seedanceReceipt?.fulfillmentConformance, "exact");- assert.equal(seedanceReceipt?.conformance, "substitute");- assert.equal(media?.conformance, "substitute");-});--test("an exact Target rejects an explicitly selected substitute realization", () => {- assert.throws(- () => createVideoBuild({ estimateRealization: "placeholder" }),- /selects a substitute path/u,- );-});diff --git a/packages/speech/src/identity.ts b/packages/speech/src/identity.tsindex c2f5b650..7e131932 100644--- a/packages/speech/src/identity.ts+++ b/packages/speech/src/identity.ts@@ -1,6 +1,6 @@import { isDigest } from "@narratage/protocol";import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";-import { assertIntrinsicExtent } from "@narratage/spatial";+import { assertContentFit, assertIntrinsicExtent, assertSpatialFrame } from "@narratage/spatial";import type { SpeechAudioBasis, SpeechBasis, SpeechDuration, SpeechEvidenceAudio } from "./types.js";function assertAudioBlob(value: SpeechBasis["audio"], label: string): void {if (value.kind !== "blob" || !isDigest(value.digest) || !Number.isSafeInteger(value.size)@@ -34,39 +34,48 @@ export function assertSpeechBasisIdentity(basis: SpeechBasis): void {if (basis.contract !== "svml.speech-basis@1") throw new Error("Unsupported SpeechBasis contract.");assertProgramSpaceIdentity(basis.programSpace);assertAudioBlob(basis.audio, "SpeechBasis audio");- if (basis.segments.length === 0 || basis.visualTrack.clips.length !== basis.segments.length) {- throw new Error("SpeechBasis must pair every Segment with one visual clip.");- }+ if (basis.segments.length === 0) throw new Error("SpeechBasis must contain at least one Segment.");let previousEnd = 0;- let accumulatedFrames = 0;const frameAt = (seconds: number): number => Math.round(seconds * basis.programSpace.frameRate.numerator / basis.programSpace.frameRate.denominator,);- for (const [index, segment] of basis.segments.entries()) {- const clip = basis.visualTrack.clips[index]!;+ const segmentSpans = new Map<string, { readonly startFrame: number; readonly endFrameExclusive: number }>();+ for (const segment of basis.segments) {if (!segment.segmentId || segment.startSec !== previousEnd || !Number.isFinite(segment.endSec)|| segment.endSec <= segment.startSec || segment.endSec > basis.programSpace.durationSec) {throw new Error("SpeechBasis Segment is invalid or non-contiguous.");}- if (clip.segmentId !== segment.segmentId || clip.artifact.kind !== "blob" || !isDigest(clip.artifact.digest)+ if (segmentSpans.has(segment.segmentId)) throw new Error(`SpeechBasis repeats Segment ${segment.segmentId}.`);+ segmentSpans.set(segment.segmentId, {+ startFrame: frameAt(segment.startSec),+ endFrameExclusive: frameAt(segment.endSec),+ });+ previousEnd = segment.endSec;+ }+ if (frameAt(previousEnd) !== programSpaceFrameCount(basis.programSpace)) {+ throw new Error("SpeechBasis Segments do not cover ProgramSpace.");+ }+ const seenVisuals = new Set<string>();+ for (const clip of basis.visualTrack.clips) {+ const segmentSpan = segmentSpans.get(clip.segmentId);+ if (segmentSpan === undefined || seenVisuals.has(clip.segmentId)+ || clip.artifact.kind !== "blob" || !isDigest(clip.artifact.digest)|| !Number.isSafeInteger(clip.artifact.size) || clip.artifact.size < 0|| !clip.artifact.mediaType.startsWith("video/")|| clip.frameRate.numerator !== basis.programSpace.frameRate.numerator|| clip.frameRate.denominator !== basis.programSpace.frameRate.denominator- || !Number.isSafeInteger(clip.frameCount) || clip.frameCount <= 0) {+ || !Number.isSafeInteger(clip.frameCount) || clip.frameCount <= 0+ || !Number.isSafeInteger(clip.span.startFrame) || !Number.isSafeInteger(clip.span.endFrameExclusive)+ || clip.span.startFrame !== segmentSpan?.startFrame+ || clip.span.endFrameExclusive !== segmentSpan?.endFrameExclusive+ || clip.frameCount !== clip.span.endFrameExclusive - clip.span.startFrame+ || !Number.isSafeInteger(clip.stackingOrder)) {throw new Error(`SpeechBasis visual clip ${clip.segmentId} is invalid.`);}+ seenVisuals.add(clip.segmentId);assertIntrinsicExtent(clip.extent);- const startFrame = frameAt(segment.startSec);- const endFrame = frameAt(segment.endSec);- if (startFrame !== accumulatedFrames || endFrame - startFrame !== clip.frameCount) {- throw new Error(`SpeechBasis visual clip ${clip.segmentId} does not cover its Segment.`);- }- accumulatedFrames = endFrame;- previousEnd = segment.endSec;- }- if (accumulatedFrames !== programSpaceFrameCount(basis.programSpace)) {- throw new Error("SpeechBasis visuals do not cover ProgramSpace.");+ assertSpatialFrame(clip.frame);+ assertContentFit(clip.fit);}}export function assertSpeechAudioBasisIdentity(basis: SpeechAudioBasis): void {diff --git a/packages/speech/src/schema.ts b/packages/speech/src/schema.tsindex bd497935..d4142753 100644--- a/packages/speech/src/schema.ts+++ b/packages/speech/src/schema.ts@@ -1,6 +1,6 @@import { programSpaceSchema } from "@narratage/program-space";import type { ValueSchema } from "@narratage/protocol";-import { intrinsicExtentSchema } from "@narratage/spatial";+import { contentFitSchema, intrinsicExtentSchema, spatialFrameSchema } from "@narratage/spatial";const string = { kind: "string", minLength: 1 } as const;const number = { kind: "number", minimum: 0 } as const;const integer = { kind: "number", integer: true, minimum: 0 } as const;@@ -18,10 +18,15 @@ export const speechBasisSchema: ValueSchema = object({contract: { schema: { kind: "literal", value: "svml.speech-basis@1" } }, programSpace: { schema: programSpaceSchema }, audio: { schema: audioBlobRef },visualTrack: { schema: object({ clips: { schema: { kind: "array", items: object({ segmentId: { schema: string }, artifact: { schema: object({kind: { schema: { kind: "literal", value: "blob" } }, digest: { schema: digest }, size: { schema: integer }, mediaType: { schema: string },+ }) }, span: { schema: object({+ startFrame: { schema: integer }, endFrameExclusive: { schema: integer },}) }, extent: { schema: intrinsicExtentSchema }, frameRate: { schema: object({numerator: { schema: { kind: "number", integer: true, minimum: 1 } },denominator: { schema: { kind: "number", integer: true, minimum: 1 } },- }) }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } } }) } } }) },+ }) }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } },+ frame: { schema: spatialFrameSchema }, fit: { schema: contentFitSchema },+ stackingOrder: { schema: { kind: "number", integer: true } },+ }) } } }) },segments: { schema: { kind: "array", minItems: 1, items: segment } },});export const speechAudioBasisSchema: ValueSchema = object({ contract: { schema: { kind: "literal", value: "svml.speech-audio-basis@1" } },diff --git a/packages/speech/src/types.ts b/packages/speech/src/types.tsindex 9b8a2e3d..c3298817 100644--- a/packages/speech/src/types.ts+++ b/packages/speech/src/types.ts@@ -1,16 +1,20 @@import type { ProgramSpace } from "@narratage/program-space";import type { BlobRef } from "@narratage/protocol";-import type { IntrinsicExtent } from "@narratage/spatial";+import type { ContentFit, IntrinsicExtent, SpatialFrame } from "@narratage/spatial";export type SpeechDuration = { readonly contract: "svml.speech-duration@1"; readonly durationSec: number };export type SpeechBasisSegment = { readonly segmentId: string; readonly startSec: number; readonly endSec: number };export type SpeechBasis = {readonly contract: "svml.speech-basis@1"; readonly programSpace: ProgramSpace; readonly audio: BlobRef;readonly visualTrack: { readonly clips: readonly {readonly segmentId: string;+ readonly span: { readonly startFrame: number; readonly endFrameExclusive: number };readonly artifact: BlobRef;readonly extent: IntrinsicExtent;readonly frameRate: { readonly numerator: number; readonly denominator: number };readonly frameCount: number;+ readonly frame: SpatialFrame;+ readonly fit: ContentFit;+ readonly stackingOrder: number;}[] };readonly segments: readonly SpeechBasisSegment[];};diff --git a/packages/temporal/src/location.ts b/packages/temporal/src/location.tsindex 8bd5c01c..5c1ae690 100644--- a/packages/temporal/src/location.ts+++ b/packages/temporal/src/location.ts@@ -1,12 +1,13 @@-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";-import { momentFrames, selectionFrameSpans } from "@narratage/semantic-map";+import { momentFrames, segmentFrameSpan, selectionFrameSpans } from "@narratage/semantic-map";import type { CompleteSemanticMap } from "@narratage/semantic-map";import type {LocatedMomentOccurrence,LocatedProgramOccurrence,+ LocatedSegmentOccurrence,LocatedSelectionOccurrence,} from "./types.js";@@ -79,3 +80,21 @@ export function locateProgramOccurrence(space: ProgramSpace): LocatedProgramOccuassertProgramSpaceIdentity(space);return { id: "program", start: { frame: 0 }, end: { frame: programSpaceFrameCount(space) } };}++export function locateSegmentOccurrence(+ map: CompleteSemanticMap,+ segment: NarrativeExcerpt,+ space: ProgramSpace,+): LocatedSegmentOccurrence {+ assertProgramSpaceIdentity(space);+ const totalFrames = programSpaceFrameCount(space);+ const span = segmentFrameSpan(map, segment, space);+ assertLocatedFrame(span.startFrame, totalFrames, `Narrative Segment ${segment.id} start`);+ assertLocatedFrame(span.endFrameExclusive, totalFrames, `Narrative Segment ${segment.id} end`);+ if (span.endFrameExclusive <= span.startFrame) throw new Error(`Narrative Segment ${segment.id} has no positive frame span.`);+ return {+ id: segment.id,+ start: { frame: span.startFrame },+ end: { frame: span.endFrameExclusive },+ };+}diff --git a/packages/temporal/src/projection.ts b/packages/temporal/src/projection.tsindex 59b780e8..bae800e9 100644--- a/packages/temporal/src/projection.ts+++ b/packages/temporal/src/projection.ts@@ -1,9 +1,9 @@-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";import type { CompleteSemanticMap } from "@narratage/semantic-map";-import { locateMomentOccurrences, locateProgramOccurrence, locateSelectionOccurrences } from "./location.js";+import { locateMomentOccurrences, locateProgramOccurrence, locateSegmentOccurrence, locateSelectionOccurrences } from "./location.js";import {add,compare,@@ -18,6 +18,7 @@ import type {FrameSpan,LocatedMomentOccurrence,LocatedProgramOccurrence,+ LocatedSegmentOccurrence,LocatedSelectionOccurrence,OccurrenceExpansion,ProjectedOccurrence,@@ -30,6 +31,7 @@ type PointEnvironment = {readonly program: LocatedProgramOccurrence;readonly selection?: LocatedSelectionOccurrence;readonly moment?: LocatedMomentOccurrence;+ readonly segment?: LocatedSegmentOccurrence;};function usesLocalPoint(expression: TemporalPointExpression, source: "selection" | "moment"): boolean {@@ -82,6 +84,16 @@ function evaluatePoint(base = environment.selection.end.frame;break;}+ case "segment.start": {+ if (environment.segment === undefined) throw new Error("segment.start requires a Segment occurrence.");+ base = environment.segment.start.frame;+ break;+ }+ case "segment.end": {+ if (environment.segment === undefined) throw new Error("segment.end requires a Segment occurrence.");+ base = environment.segment.end.frame;+ break;+ }case "moment.cue": {if (environment.moment === undefined) throw new Error("moment.cue requires a Moment occurrence.");base = environment.moment.cue.frame;@@ -183,6 +195,22 @@ export function projectProgramWindow(input: {};}+export function projectSegmentWindow(input: {+ readonly itemId: string;+ readonly map: CompleteSemanticMap;+ readonly segment: NarrativeExcerpt;+ readonly space: ProgramSpace;+ readonly projection: TemporalWindowProjection;+}): ProjectedOccurrence {+ const program = locateProgramOccurrence(input.space);+ const segment = locateSegmentOccurrence(input.map, input.segment, input.space);+ return {+ id: projectedId(input.itemId, segment.id),+ sourceOccurrenceId: segment.id,+ span: projectTemporalWindow(input.projection, { program, segment }, input.space),+ };+}+export function assertWindowRelation(occurrences: readonly ProjectedOccurrence[],relation: WindowRelation,diff --git a/packages/temporal/src/types.ts b/packages/temporal/src/types.tsindex ff703888..f848ba09 100644--- a/packages/temporal/src/types.ts+++ b/packages/temporal/src/types.ts@@ -21,6 +21,12 @@ export type LocatedProgramOccurrence = {readonly end: FramePoint;};+export type LocatedSegmentOccurrence = {+ readonly id: string;+ readonly start: FramePoint;+ readonly end: FramePoint;+};+/** Exact author duration. Decimal source spelling is reduced before reaching this value. */export type TemporalDuration =| { readonly unit: "frames"; readonly value: number }@@ -32,6 +38,8 @@ export type TemporalPointExpression =| { readonly ref: "program.end"; readonly offset?: TemporalDuration }| { readonly ref: "selection.start"; readonly offset?: TemporalDuration }| { readonly ref: "selection.end"; readonly offset?: TemporalDuration }+ | { readonly ref: "segment.start"; readonly offset?: TemporalDuration }+ | { readonly ref: "segment.end"; readonly offset?: TemporalDuration }| { readonly ref: "moment.cue"; readonly offset?: TemporalDuration }| { readonly ref: "absolute"; readonly at: TemporalDuration };diff --git a/packages/temporal/test/temporal.test.ts b/packages/temporal/test/temporal.test.tsindex 02567d0a..b0b9b7e1 100644--- a/packages/temporal/test/temporal.test.ts+++ b/packages/temporal/test/temporal.test.ts@@ -11,6 +11,7 @@ import {locateSelectionOccurrences,projectMomentWindows,projectProgramWindow,+ projectSegmentWindow,projectSelectionWindows,resolveTriggeredSchedule,temporalDurationInSamples,@@ -31,6 +32,8 @@ const map: CompleteSemanticMap = {{ identity: "c", timeSec: 3, frame: 90 },{ identity: "d", timeSec: 4, frame: 120 },{ identity: "late", timeSec: 8, frame: 240 },+ { identity: "segment:answer:start", timeSec: 2, frame: 60 },+ { identity: "segment:answer:end", timeSec: 4, frame: 120 },],};@@ -59,6 +62,21 @@ test("one Selection projects exact local points and stable occurrence identity",}]);});+test("one Segment projects from its own structural start and end anchors", () => {+ const result = projectSegmentWindow({+ itemId: "answer-card",+ map,+ segment: { contract: "svml.narrative-excerpt@1", kind: "segment", id: "answer", tokenStart: 0, tokenEndExclusive: 1 },+ space,+ projection: { start: { ref: "segment.start" }, end: { ref: "segment.end" } },+ });+ assert.deepEqual(result, {+ id: "answer-card::answer",+ sourceOccurrenceId: "answer",+ span: { startFrame: 60, endFrameExclusive: 120 },+ });+});+test("each preserves source order, even when physical time is reversed between occurrences", () => {const result = projectSelectionWindows({itemId: "repeat",diff --git a/packages/whisperx/test/whisperx.test.ts b/packages/whisperx/test/whisperx.test.tsindex d11fd3c0..18573973 100644--- a/packages/whisperx/test/whisperx.test.ts+++ b/packages/whisperx/test/whisperx.test.ts@@ -24,18 +24,11 @@ function basis() {durationSec: 1,frameRate: { numerator: 30, denominator: 1 },});- const visual = digestOf("whisperx-test:visual");return sealSpeechBasis({contract: "svml.speech-basis@1",programSpace,audio: { kind: "blob", digest: digestOf("whisperx-test:audio"), size: 1, mediaType: "audio/wav" },- visualTrack: { clips: [{- segmentId: "line",- artifact: { kind: "blob", digest: visual, size: 1, mediaType: "video/mp4" },- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },- frameRate: { ...programSpace.frameRate },- frameCount: 30,- }] },+ visualTrack: { clips: [] },segments: [{ segmentId: "line", startSec: 0, endSec: 1 }],});}diff --git a/spec/media-track.md b/spec/media-track.mdindex 5ebd817f..c068c564 100644--- a/spec/media-track.md+++ b/spec/media-track.md@@ -514,11 +514,11 @@ Speech Spine should stop maintaining a second ad-hoc media renderer. Its visualsame Media lowering implementation with a deliberately restricted generated Program:```text-one normalized A/V take per speech Segment+one normalized speech-bearing Take per Segment; its visual stream is optionalexact already-established Segment frame span-Canvas Placement Frame+explicit authored Spatial Frameone foreground layer-explicit contain/cover fit chosen by the speech package+explicit fit and absolute stacking order chosen by the authormuted visualcut between contiguous takesno entry, sustain or exit motion@@ -528,7 +528,10 @@ no source-audio projectionThe Speech author Surface accepts raw `video=` and expands the same generic inspection andnormalization graph before assembly, selecting primary moving video plus default audio at the-Spine's explicit frame rate. It also accepts prepared `media=` directly. After assembly, Speech+Spine's explicit frame rate. Raw `audio=` expands an audio-authoritative normalization graph and+contributes no visual Present; it never manufactures black media. Prepared `media=` remains a+direct exact input. Visual Takes inherit the Spine's explicit `visual-frame`,+`visual-appearance` and `visual-z`, with per-Take overrides for those same limited axes. After assembly, Speechaudio remains the separate canonical `SpeechAudioBasis -> AudioTrack` projection; the restrictedvisual lowerer itself remains muted and does not rediscover container audio.@@ -540,6 +543,10 @@ still does not register Media meaning.The author imports Speech Spine, whose manifest declares its package dependency. Authors do notneed to import a second parser merely because the implementation reuses Media lowering.+Ordinary Media Items may consume `during={story.segment.answer}` directly. This is not an implicit+Selection: Temporal resolves the Segment's already-existing structural start/end anchors. The same+Item can still consume authored Selection and Moment values when those are the actual intent.+## 14. Legacy audit: retained and retiredRetain:@@ -705,8 +712,8 @@ The matrix above is closed by executable evidence rather than by package status- `packages/hyperframes/test/browser-visual.test.ts` renders opaque images, a straight-alphaSurface, two-frame fitting, local motion and a Sequence handoff through real Chromium, thencompares every decoded RGBA frame across different worker partitions;-- `tools/package-boundaries.test.mjs` and `tools/graph-first-value-boundary.test.mjs` enforce the- absence of a Media family registry and of hidden lineage/provider metadata in values.+- package manifests contain no Media family registry, while the public value types contain no+ hidden lineage or Provider metadata.The local package is therefore complete as a pre-release Media implementation over the frozenrepository-internal `svml.visual-track@1` / `svml.visual-ir@1` waist. This does not publish ordiff --git a/spec/track-authoring.md b/spec/track-authoring.mdindex e1621aa7..a11ed8d4 100644--- a/spec/track-authoring.md+++ b/spec/track-authoring.md@@ -412,8 +412,9 @@ Audio Track ─────────┘```Caption's component-owned timed Cue projection may enter the final frame-window stage directly.-Speech Spine establishes ProgramSpace and contiguous speech takes rather than pretending those-structural cuts are Narrative Selection consumption.+Speech Spine establishes ProgramSpace and contiguous speech takes. Other Tracks may address one+whole Narrative Segment through its two structural anchors directly, without pretending it is an+authored Selection.## 8. Migration consequencesdiff --git a/tools/caption-playground-sha256.test.mjs b/tools/caption-playground-sha256.test.mjsdeleted file mode 100644index d772c740..00000000--- a/tools/caption-playground-sha256.test.mjs+++ /dev/null@@ -1,17 +0,0 @@-import assert from "node:assert/strict";-import { createHash } from "node:crypto";-import test from "node:test";--import { createHash as shimCreateHash } from "./caption-playground/src/shims/node-crypto.ts";--const cases = ["", "abc", "x".repeat(55), "x".repeat(56), "x".repeat(64), "字幕轨道与画布"];-test("Caption Playground browser sha256 agrees with Node", () => {- for (const value of cases) {- assert.equal(shimCreateHash("sha256").update(value).digest("hex"), createHash("sha256").update(value).digest("hex"));- }-});--test("Caption Playground browser sha256 rejects unsupported operations", () => {- assert.throws(() => shimCreateHash("sha512"), /sha256/u);- assert.throws(() => shimCreateHash("sha256").update("a").digest("base64"), /hex/u);-});diff --git a/tools/caption-playground-source-bytes.test.mjs b/tools/caption-playground-source-bytes.test.mjsdeleted file mode 100644index 2809f841..00000000--- a/tools/caption-playground-source-bytes.test.mjs+++ /dev/null@@ -1,25 +0,0 @@-import assert from "node:assert/strict";-import { execFileSync } from "node:child_process";-import { existsSync, readFileSync } from "node:fs";-import test from "node:test";-import { fileURLToPath } from "node:url";--const root = fileURLToPath(new URL("..", import.meta.url));-const paths = execFileSync("git", ["ls-files", "-c", "-o", "--exclude-standard", "tools/caption-playground", "test"], {- cwd: root, encoding: "utf8",-}).trim().split("\n").filter((path) => /\.(?:ts|mjs|js|html|css|json)$/u.test(path));--test("Caption Playground source carries no raw control byte", () => {- const offenders = [];- for (const path of paths) {- const file = new URL(path, new URL("..", import.meta.url));- if (!existsSync(file)) continue;- for (const [index, byte] of readFileSync(file).entries()) {- if (byte < 0x20 && byte !== 0x09 && byte !== 0x0a && byte !== 0x0d) {- offenders.push(`${path}: 0x${byte.toString(16)} at ${index}`);- break;- }- }- }- assert.deepEqual(offenders, []);-});diff --git a/tools/graph-first-value-boundary.test.mjs b/tools/graph-first-value-boundary.test.mjsdeleted file mode 100644index e371d083..00000000--- a/tools/graph-first-value-boundary.test.mjs+++ /dev/null@@ -1,118 +0,0 @@-import assert from "node:assert/strict";-import {- readdir,- readFile,-} from "node:fs/promises";-import test from "node:test";--const packageRoot = new URL("../packages/", import.meta.url);--async function sourceFiles(root) {- const files = [];- for (const entry of await readdir(root, { withFileTypes: true })) {- const path = new URL(`./${entry.name}${entry.isDirectory() ? "/" : ""}`, root);- if (entry.isDirectory()) files.push(...await sourceFiles(path));- else if (entry.isFile() && entry.name.endsWith(".ts")) files.push(path);- }- return files;-}--async function contents(packageNames) {- const result = [];- for (const name of packageNames) {- const root = new URL(`./${name}/src/`, packageRoot);- for (const path of await sourceFiles(root)) {- result.push({ path, source: await readFile(path, "utf8") });- }- }- return result;-}--const domainValuePackages = [- "media-track",- "caption",- "caption-gemini",- "estimate",- "film",- "generation",- "hyperframes",- "render-hyperframes",- "image-transform",- "media-pipeline",- "text",- "script",- "seedance",- "speech-alignment",- "speech-spine",- "speech-basis",- "typography-track",- "whisperx",-];--const forbiddenLineageFields = [- "programSpaceDigest",- "narrativeDigest",- "mapDigest",- "selectionDigest",- "excerptDigest",- "basisDigest",- "resultDigest",- "productDigest",- "planDigest",- "captionProgramDigest",- "sourceAudioArtifactDigest",- "renderInputDigest",- "visualDigest",- "audioDigest",- "inspectionDigest",- "previousSetDigest",- "trackDigest",-];--test("domain Products do not copy graph lineage into payload fields", async () => {- for (const { path, source } of await contents(domainValuePackages)) {- for (const field of forbiddenLineageFields) {- assert.doesNotMatch(source, new RegExp(`\\b${field}\\b`, "u"), `${path.pathname} reintroduced ${field}`);- }- assert.doesNotMatch(- source,- /readonly\s+digest\s*:\s*Digest\b/u,- `${path.pathname} reintroduced a domain Product self-digest`,- );- }-});--test("the generic graph stack has no payload-affinity side channel", async () => {- const genericSources = await contents(["protocol", "core", "elaborator"]);- const removedApi = [- "ResultAffinityDeclaration",- "AffinityConstraint",- "FragmentAffinityConstraint",- "verifyRecordAffinity",- ];- for (const { path, source } of genericSources) {- for (const symbol of removedApi) {- assert.doesNotMatch(source, new RegExp(`\\b${symbol}\\b`, "u"), `${path.pathname} reintroduced ${symbol}`);- }- }-});--test("fold values retain members, not hidden identity or policy context", async () => {- const declarations = [- ["media-track", "types.ts", "MediaTrackSet"],- ["film", "types.ts", "FilmTrackSet"],- ["speech-spine", "types.ts", "SpeechSpineSet"],- ["typography-track", "types.ts", "TypographyTrackSet"],- ];- for (const [packageName, fileName, typeName] of declarations) {- const path = new URL(`./${packageName}/src/${fileName}`, packageRoot);- const source = await readFile(path, "utf8");- const declaration = new RegExp(`export type ${typeName} = \\{([\\s\\S]*?)\\n\\};`, "u").exec(source)?.[1];- assert.ok(declaration, `${typeName} declaration is missing`);- assert.doesNotMatch(- declaration,- /readonly\s+(?:id|digest|program|programSpace)\s*:/u,- `${typeName} carries hidden identity or policy context`,- );- }-});diff --git a/tools/package-boundaries.test.mjs b/tools/package-boundaries.test.mjsdeleted file mode 100644index 135116c9..00000000--- a/tools/package-boundaries.test.mjs+++ /dev/null@@ -1,343 +0,0 @@-import assert from "node:assert/strict";-import {- readdir,- readFile,-} from "node:fs/promises";-import test from "node:test";--const packageRoot = new URL("../packages/", import.meta.url);--async function workspacePackages() {- const manifests = new Map();- for (const entry of await readdir(packageRoot, { withFileTypes: true })) {- if (!entry.isDirectory()) continue;- const path = new URL(`./${entry.name}/package.json`, packageRoot);- let manifest;- try {- manifest = JSON.parse(await readFile(path, "utf8"));- } catch (error) {- if (error?.code === "ENOENT") continue;- throw error;- }- manifests.set(manifest.name, manifest);- }- return manifests;-}--async function sourceFiles(directory) {- const found = [];- let entries;- try {- entries = await readdir(directory, { withFileTypes: true });- } catch (error) {- if (error?.code === "ENOENT") return found;- throw error;- }- for (const entry of entries) {- const path = new URL(`./${entry.name}${entry.isDirectory() ? "/" : ""}`, directory);- if (entry.isDirectory()) found.push(...await sourceFiles(path));- else if (entry.name.endsWith(".ts")) found.push(path);- }- return found;-}--function productionGraph(packages) {- return new Map([...packages].map(([name, manifest]) => [- name,- Object.keys({- ...(manifest.dependencies ?? {}),- ...(manifest.optionalDependencies ?? {}),- }).filter((dependency) => packages.has(dependency)),- ]));-}--function transitive(graph, entry) {- const found = new Set();- const visit = (name) => {- for (const dependency of graph.get(name) ?? []) {- if (found.has(dependency)) continue;- found.add(dependency);- visit(dependency);- }- };- visit(entry);- return found;-}--const domainNeutralPackages = new Set([- "@narratage/protocol",- "@narratage/artifact",- "@narratage/core",- "@narratage/source",- "@narratage/elaborator",- "@narratage/run",- "@narratage/validation",- "@narratage/host",- "@narratage/compiler-node",- "@narratage/workspace-fs-node",- "@narratage/component-kit",- "@narratage/runtime",- "@narratage/runtime-adapter",- "@narratage/runtime-adapter-node",- "@narratage/endpoint-kit",- "@narratage/driver-node",- "@narratage/package-loader-node",- "@narratage/store-sqlite",- "@narratage/artifact-store-fs",- "@narratage/artifact-store-s3",- "@narratage/credential-store-env",- "@narratage/credential-store-keychain",- "@narratage/transport",- "@narratage/transport-aws-lambda",- "@narratage/local",-]);--test("official production packages have no dependency cycle", async () => {- const graph = productionGraph(await workspacePackages());- const visiting = new Set();- const complete = new Set();- const visit = (name, trail) => {- if (visiting.has(name)) assert.fail(`production dependency cycle: ${[...trail, name].join(" -> ")}`);- if (complete.has(name)) return;- visiting.add(name);- for (const dependency of graph.get(name) ?? []) visit(dependency, [...trail, name]);- visiting.delete(name);- complete.add(name);- };- for (const name of graph.keys()) visit(name, []);-});--test("Caption Playground cannot add semantics or metadata to production packages", async () => {- let productionSource = "";- for (const entry of await readdir(packageRoot, { withFileTypes: true })) {- if (!entry.isDirectory()) continue;- const sourceRoot = new URL(`./${entry.name}/src/`, packageRoot);- for (const file of await sourceFiles(sourceRoot)) {- const source = await readFile(file, "utf8");- productionSource += source;- assert.ok(!/\bplayground\b/iu.test(source),- `${file.pathname} mentions Playground; the dependency must point from the tool to production only`);- }- }-- const typeDeclarations = await readFile(new URL("./protocol/src/module.ts", packageRoot), "utf8");- assert.ok(!/readonly\s+default\??\s*:/u.test(typeDeclarations),- "TypeDeclaration must not carry preview defaults");-- const valueSchemas = await readFile(new URL("./protocol/src/value.ts", packageRoot), "utf8");- assert.ok(!/readonly\s+format\??\s*:/u.test(valueSchemas),- "ValueSchema must not carry presentation hints for a tool");-- const componentKit = await readFile(new URL("./component-kit/src/index.ts", packageRoot), "utf8");- assert.ok(!/\bRecipeFacet\b|readonly\s+recipes\??\s*:/u.test(componentKit),- "ComponentPackage must not grow an editor or Playground Recipe facet");-- for (const helper of [- "defaultProgramSpace", "defaultCanvasSpace", "defaultFilmProgram", "defaultScreenOverlayProgram",- ]) {- assert.ok(!productionSource.includes(helper),- `${helper} is a preview starting value and must stay outside production packages`);- }-});--test("Caption Playground is a selected source editor, not a component registry", async () => {- const root = new URL("../tools/caption-playground/src/", import.meta.url);- const allowedPackages = new Set([- "@narratage/caption",- "@narratage/caption-fine",- "@narratage/composition",- "@narratage/fonts-open",- "@narratage/host",- "@narratage/narrative",- "@narratage/package-loader-node",- "@narratage/program-space",- "@narratage/protocol",- "@narratage/svs",- "@narratage/video-cli",- ]);- for (const file of await sourceFiles(root)) {- const source = await readFile(file, "utf8");- for (const match of source.matchAll(/["'](@narratage\/[a-z0-9-]+)["']/gu)) {- assert.ok(allowedPackages.has(match[1]), `${file.pathname} unexpectedly names ${match[1]}`);- }- assert.ok(!/\b(?:Story|Scenario|Recipe)Registry\b/u.test(source),- `${file.pathname} introduces a parallel Caption Playground registry`);- assert.ok(!/import\.meta\.glob|discoverPreviewProducers/u.test(source),- `${file.pathname} scans the workspace instead of editing the explicitly selected Caption source`);- }-});--test("Markup syntax, graph Text and video Typography keep distinct package identities", async () => {- const packages = await workspacePackages();- for (const name of [- "@narratage/markup",- "@narratage/run-markup",- "@narratage/compiler-markup-node",- "@narratage/text",- "@narratage/typography-track",- ]) assert.ok(packages.has(name), `${name} is absent`);- for (const retired of [- "@narratage/run-text",- "@narratage/compiler-text-node",- "@narratage/text-track",- "@narratage/prompt-kit",- ]) assert.ok(!packages.has(retired), `${retired} must stay retired`);-});--test("the declared domain-neutral distribution closes without syntax, AIGC or video packages", async () => {- const packages = await workspacePackages();- const graph = productionGraph(packages);- for (const name of domainNeutralPackages) {- assert.ok(packages.has(name), `domain-neutral package ${name} is absent`);- const outside = [...transitive(graph, name)].filter((dependency) =>- dependency.startsWith("@narratage/") && !domainNeutralPackages.has(dependency));- assert.deepEqual(outside, [], `${name} reaches packages outside the domain-neutral distribution`);- }- assert.deepEqual(graph.get("@narratage/core"), ["@narratage/protocol"]);-});--test("a Provider reaches no exact-model package, so models and services stay N+M", async () => {- const packages = await workspacePackages();- const graph = productionGraph(packages);- const models = [- "@narratage/gemini-omni",- "@narratage/gpt-image",- "@narratage/grok-imagine",- "@narratage/minimax-h3",- "@narratage/nano-banana",- "@narratage/seedance",- "@narratage/seedream",- ];- for (const name of [...packages.keys()].filter((item) => item.startsWith("@narratage/provider-"))) {- const reached = transitive(graph, name);- assert.deepEqual(- models.filter((model) => reached.has(model)),- [],- `${name} depends on an exact-model package; a Provider must map declared ports instead`,- );- }- // The shared port and wire-mapping vocabulary is the only legitimate meeting point.- assert.ok(transitive(graph, "@narratage/provider-kie").has("@narratage/generation"));- assert.ok(transitive(graph, "@narratage/seedance").has("@narratage/generation"));-});--test("the speech time map is an opaque handle: no consumer reads its fields", async () => {- const { readdir, readFile } = await import("node:fs/promises");- const packages = new URL("../packages/", import.meta.url);- // Locating is a lookup through semantic-map. A consumer that destructured the- // map could sort, merge or clip located spans against one another; those are- // the caller's decisions to make on its own values, never the map's to make- // for it.- const reach = /\bmap\.(tokens|anchors|segments)\b/u;- for (const entry of await readdir(packages, { withFileTypes: true })) {- if (!entry.isDirectory() || entry.name === "semantic-map" || entry.name === "speech-alignment") continue;- const source = new URL(`./${entry.name}/src/`, packages);- let files = [];- try {- files = await readdir(source);- } catch {- continue;- }- for (const file of files.filter((name) => name.endsWith(".ts"))) {- const content = await readFile(new URL(file, source), "utf8");- assert.ok(!reach.test(content),- `${entry.name}/src/${file} reads a SemanticMap field directly; call a semantic-map lookup instead`);- }- }-});--test("Markup compilation is one explicit leaf assembly, not a Package Loader or Local Runtime dependency", async () => {- const graph = productionGraph(await workspacePackages());- assert.ok(transitive(graph, "@narratage/compiler-markup-node").has("@narratage/markup"));- assert.ok(!transitive(graph, "@narratage/package-loader-node").has("@narratage/markup"));- assert.ok(!transitive(graph, "@narratage/local").has("@narratage/markup"));-});--test("generic and video CLIs reach no Provider package and video CLI activates no author aggregate", async () => {- const packages = await workspacePackages();- const graph = productionGraph(packages);- const dependencies = transitive(graph, "@narratage/cli");- const providers = [...packages.keys()].filter((name) => name.startsWith("@narratage/provider-"));- assert.deepEqual(providers.filter((name) => dependencies.has(name)), []);- const videoDependencies = transitive(graph, "@narratage/video-cli");- assert.deepEqual(providers.filter((name) => videoDependencies.has(name)), []);- assert.ok(!videoDependencies.has("@narratage/script"));- assert.ok(!videoDependencies.has("@narratage/media-track"));- assert.ok(!videoDependencies.has("@narratage/typography-track"));- assert.ok(!videoDependencies.has("@narratage/film"));- assert.ok(videoDependencies.has("@narratage/cli"));-});--test("domain packages confine their Markup dependency to Surface and activation entries", async () => {- const { readdir, readFile } = await import("node:fs/promises");- const surfaceOnly = [- "caption", "caption-gemini", "estimate", "film", "render-hyperframes",- "image-transform", "media", "media-pipeline", "media-track", "seedance", "speech-spine", "whisperx",- ];- const allowed = new Set(["surface.ts", "activation.ts"]);- for (const name of surfaceOnly) {- const root = new URL(`../packages/${name}/src/`, import.meta.url);- for (const file of await readdir(root)) {- if (!file.endsWith(".ts") || allowed.has(file)) continue;- const content = await readFile(new URL(file, root), "utf8");- assert.ok(!content.includes("\"@narratage/markup\""),- `${name}/src/${file} imports @narratage/markup outside its Surface boundary`);- }- }-});--test("every package is reachable: imported, activatable, or a declared entry point", async () => {- const { readdir, readFile } = await import("node:fs/promises");- const packages = await workspacePackages();- // A CLI is nobody's dependency by construction; it is what a person runs.- const entryPoints = new Set(["@narratage/video-cli", "@narratage/test-support"]);-- // Anything imported by non-test source anywhere in the workspace, including- // the deployment services, which are real consumers even though they are not- // workspace packages.- const imported = new Set();- const scan = async (directory, owner) => {- let entries;- try {- entries = await readdir(directory, { withFileTypes: true });- } catch {- return;- }- for (const entry of entries) {- const path = `${directory}/${entry.name}`;- if (entry.isDirectory() && entry.name !== "node_modules") await scan(path, owner);- else if (entry.name.endsWith(".ts")) {- for (const match of (await readFile(path, "utf8")).matchAll(/"(@narratage\/[a-z0-9-]+)"/gu)) {- if (match[1] !== owner) imported.add(match[1]);- }- }- }- };- for (const name of packages.keys()) await scan(`packages/${name.replace("@narratage/", "")}/src`, name);- await scan("services", undefined);-- // A package with an activation entry is loaded by the Package Loader from a- // lock, never by an import, so having no importer is what it is supposed to- // look like.- const activatable = new Set([...packages]- .filter(([, manifest]) => manifest.svml?.activation !== undefined)- .map(([name]) => name));-- // A data-only author package can deliberately export self-describing Source- // Modules without shipping executable activation code. Those exports are- // public entry points just as surely as a CLI binary is.- const sourceResources = new Set([...packages]- .filter(([, manifest]) => manifest.exports !== null- && typeof manifest.exports === "object"- && Object.values(manifest.exports).some((target) => typeof target === "string" && target.endsWith(".svs")))- .map(([name]) => name));-- const unreachable = [...packages.keys()]- .filter((name) => !imported.has(name)- && !activatable.has(name)- && !sourceResources.has(name)- && !entryPoints.has(name))- .sort();- assert.deepEqual(unreachable, [],- "packages nothing imports, nothing can activate, and no one runs — delete them or give them a consumer");-});diff --git a/tools/visual-ir-boundary.test.mjs b/tools/visual-ir-boundary.test.mjsdeleted file mode 100644index 9e11ba78..00000000--- a/tools/visual-ir-boundary.test.mjs+++ /dev/null@@ -1,103 +0,0 @@-import assert from "node:assert/strict";-import { readdir, readFile } from "node:fs/promises";-import test from "node:test";--import { compositionSchema } from "@narratage/composition";-import { VISUAL_STYLE_NAMES_V1 } from "@narratage/visual-ir";--function schemaFieldNames(schema, found = new Set()) {- if (schema === null || typeof schema !== "object") return found;- if (schema.kind === "object") {- for (const [name, field] of Object.entries(schema.fields ?? {})) {- found.add(name);- schemaFieldNames(field.schema, found);- }- }- if (schema.kind === "array") schemaFieldNames(schema.items, found);- if (schema.kind === "oneOf") for (const variant of schema.variants ?? []) schemaFieldNames(variant, found);- return found;-}--async function packageManifest(name) {- return JSON.parse(await readFile(new URL(`../packages/${name}/package.json`, import.meta.url), "utf8"));-}--async function typescriptSources(directory) {- const result = [];- const visit = async (root) => {- for (const entry of await readdir(root, { withFileTypes: true })) {- const child = new URL(`./${entry.name}${entry.isDirectory() ? "/" : ""}`, root);- if (entry.isDirectory()) await visit(child);- else if (entry.name.endsWith(".ts")) result.push({ path: child, source: await readFile(child, "utf8") });- }- };- await visit(new URL(`../packages/${directory}/src/`, import.meta.url));- return result;-}--test("the public Composition waist has no author-family, Provider or cross-Track input field", () => {- const fields = schemaFieldNames(compositionSchema);- const forbidden = [- "authorFamily",- "caption",- "component",- "family",- "lowerComposite",- "mediaTrack",- "provider",- "ranking",- "recipe",- "selector",- "siblingTrack",- "sourceTrack",- "trackRef",- ];- assert.deepEqual(forbidden.filter((field) => fields.has(field)), []);- assert.ok(fields.has("visualIr") && fields.has("presents") && fields.has("elements"),- "the audit must be inspecting the actual public VisualTrack schema");- assert.equal(VISUAL_STYLE_NAMES_V1.includes("backdrop-filter"), false);- assert.equal(VISUAL_STYLE_NAMES_V1.includes("mix-blend-mode"), false);-});--test("Composition and render adapters reach no author-package family", async () => {- const authorPackages = new Set([- "@narratage/audio-track",- "@narratage/caption",- "@narratage/caption-fine",- "@narratage/deck-track",- "@narratage/media-track",- "@narratage/ranking",- "@narratage/screen-overlay",- "@narratage/typography-track",- ]);- for (const name of ["composition", "hyperframes", "render-hyperframes", "provider-hyperframes-local"]) {- const manifest = await packageManifest(name);- const dependencies = new Set([- ...Object.keys(manifest.dependencies ?? {}),- ...Object.keys(manifest.optionalDependencies ?? {}),- ]);- assert.deepEqual([...authorPackages].filter((dependency) => dependencies.has(dependency)), [],- `${manifest.name} depends on an author-package family`);- }-});--test("the terminal compiler contains no hidden accumulated-composite sampling path", async () => {- const forbidden = [- /backdrop-filter/u,- /mix-blend-mode/u,- /lowerComposite/u,- /siblingTrack/u,- /sourceTrack/u,- /trackRef/u,- /@narratage\/(?:caption(?:-fine)?|deck-track|media-track|ranking|screen-overlay|text-track)/u,- ];- const failures = [];- for (const directory of ["hyperframes", "render-hyperframes"]) {- for (const item of await typescriptSources(directory)) {- for (const pattern of forbidden) {- if (pattern.test(item.source)) failures.push(`${item.path.pathname}: ${pattern}`);- }- }- }- assert.deepEqual(failures, [], `terminal renderer boundary leaked author/cross-Track semantics:\n${failures.join("\n")}`);-});