Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

refactor(video): normalize media at track boundaries

kashorin29c031b28cff
diff --git a/docs/guide/conventions.md b/docs/guide/conventions.md
index b0539a52..783ef971 100644
--- a/docs/guide/conventions.md
+++ b/docs/guide/conventions.md
@@ -21,7 +21,7 @@ description: Naming, module boundaries, TypeScript configuration and wire data.
- Each package has exactly one public entry point: `src/index.ts`.
- Internal modules use explicit `.js` extensions (NodeNext resolution).
- Cross-package imports use `@narratage/*`, never relative paths across package boundaries.
-- Circular dependencies are forbidden (enforced by `tools/package-boundaries.test.mjs`).
+- Circular production dependencies are forbidden.
## TypeScript configuration
diff --git a/docs/guide/packages.md b/docs/guide/packages.md
index f07cc636..264e3a57 100644
--- a/docs/guide/packages.md
+++ b/docs/guide/packages.md
@@ -141,7 +141,7 @@ never on exact-model packages or the CLI.
## Dependency rules
-`tools/package-boundaries.test.mjs` enforces three invariants on every commit:
+The package layout follows three dependency rules:
1. **Acyclic production graph.** No dependency cycle among any `@narratage/*` packages.
@@ -152,7 +152,8 @@ never on exact-model packages or the CLI.
Provider package. The video CLI also does not depend on any author-level video package
(`@narratage/script`, `@narratage/seedance`, `@narratage/media-track`, `@narratage/typography-track`, `@narratage/film`).
Author packages are activated through the explicit package lock, not compile-time CLI
- dependencies.
+ dependencies. These rules are kept visible in package manifests and reviewed as architecture,
+ rather than approximated by source-text regex tests.
## Package anatomy
diff --git a/docs/guide/testing.md b/docs/guide/testing.md
index 243d1943..eeae7892 100644
--- a/docs/guide/testing.md
+++ b/docs/guide/testing.md
@@ -75,19 +75,13 @@ test("generates a video", async (t) => {
});
```
-### Boundary tests
+### Architecture boundaries
-`tools/package-boundaries.test.mjs` and `tools/graph-first-value-boundary.test.mjs` verify
-structural invariants across the entire workspace:
+Package boundaries are expressed by package manifests, public entry points and shared contracts.
+Tests exercise observable behavior at those boundaries. The repository deliberately does not use
+source-text regex tests as a substitute for dependency analysis or architectural review.
-- Acyclic dependency graph
-- Domain-neutral closure contains only domain-neutral packages
-- CLI has no Provider or author-package dependency
-
-These run as part of `pnpm test` on every commit.
-
-These tests remain when the repository is opened: protocol, graph, package-boundary and deterministic
-unit tests are the executable definition of the decoupling rules. Before publication, remove customer
+Before publication, remove customer
or brand fixtures, credential traces, paid output artifacts, absolute workstation paths and one-off
acceptance harnesses. Generic live tests may remain only when they are explicitly opt-in, use no
committed secret and fail before spending money unless their gate is set.
diff --git a/docs/quickstart/composition.md b/docs/quickstart/composition.md
index a75409dc..9c4e7c4b 100644
--- a/docs/quickstart/composition.md
+++ b/docs/quickstart/composition.md
@@ -50,7 +50,7 @@ Common Track sources:
| Source | Type | From |
|---|---|---|
-| `{speech.visual}` | VisualTrack | `speech:Spine` — full-screen talking head |
+| `{speech.visual}` | VisualTrack | `speech:Spine` — sparse same-source speech visuals |
| `{speech.audioTrack}` | AudioTrack | `speech:Spine` — synchronized audio |
| `{captions.track}` | VisualTrack | a Caption Style-family Track — timed captions |
| `{cards.visual}` | VisualTrack | `media-track:Track` — media overlays or B-roll |
@@ -66,7 +66,7 @@ Typical stacking order:
| stack-order | Content |
|---|---|
-| 10 | Speech visual (full-screen talking head) |
+| 10 | Example Speech visual (author-selected, not a built-in default) |
| 40 | Media overlays |
| 70 | Captions |
| 90 | Text overlays |
@@ -145,13 +145,16 @@ The complete data flow from Script to rendered video. This example is based on
prompt={direction} duration="5"/>
<space:Canvas id="vertical" width="1080" height="1920"/>
+ <space:Frame id="speech-frame" within={vertical}
+ left="0%" top="0%" right="100%" bottom="100%"/>
<space:Frame id="title-frame" within={vertical}
left="6%" top="6%" right="6%" bottom="84%"/>
<space:Frame id="card-frame" within={vertical}
left="10%" top="20%" right="10%" bottom="30%"/>
<!-- 3. Timing: assemble spine and align words -->
- <speech:Spine id="speech" canvas={vertical} frame-rate="30">
+ <speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={take.video} segment={story.segment.opening}/>
</speech:Spine>
<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>
@@ -204,6 +207,7 @@ The complete data flow from Script to rendered video. This example is based on
film.vertical {
background: #09090B;
}
+ speech.visual { fit: cover; }
media.card {
stack-order: 40; fit: cover; playback: hold-start;
frame-paint: #111116; clip: rounded; radius: 20;
diff --git a/docs/quickstart/timing.md b/docs/quickstart/timing.md
index 2693cf59..c4a971bb 100644
--- a/docs/quickstart/timing.md
+++ b/docs/quickstart/timing.md
@@ -13,6 +13,7 @@ steps produce the **ProgramSpace** and **SemanticMap** that every downstream com
<import as="speech" from="@narratage/speech-spine@1"/>
<import as="whisperx" from="@narratage/whisperx@1"/>
<import as="space" from="@narratage/spatial@1"/>
+<import as="studio" source="./studio.svs"/>
```
## speech:Spine
@@ -22,7 +23,9 @@ program order — the final sequence of Segments in the finished video.
```svml
<space:Canvas id="vertical" width="1080" height="1920"/>
-<speech:Spine id="speech" canvas={vertical} frame-rate="30">
+<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>
+<speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={hook-take.video} segment={story.segment.hook}/>
<speech:Take video={meeting-take.video} segment={story.segment.meeting}/>
<speech:Take video={evidence-take.video} segment={story.segment.evidence}/>
@@ -33,20 +36,26 @@ program order — the final sequence of Segments in the finished video.
| Attribute | Required | Description |
|---|---|---|
| `id` | yes | Unique identifier |
-| `canvas` | yes | Explicit CanvasSpace used by the restricted Media visual projection |
| `frame-rate` | yes | Program rate as an integer or rational, such as `30` or `30000/1001` |
+| `visual-frame` | yes | Explicit base SpatialFrame for same-source Take visuals |
+| `visual-appearance` | yes | SVS Recipe containing only spatial fit properties |
+| `visual-z` | yes | Base absolute stacking order for same-source Take visuals |
### speech:Take
-Each `<speech:Take>` child binds a generated video to a Script Segment:
+Each `<speech:Take>` child binds one speech-bearing source to a Script Segment:
| Attribute | Required | Description |
|---|---|---|
-| `video` | exactly one | Generated/raw video Blob — for example `{take.video}` from Seedance |
+| `video` | exactly one | Generated/raw A/V Blob — for example `{take.video}` from Seedance |
+| `audio` | exactly one | Voice-only Blob; contributes time and master speech audio, but no visual |
| `media` | exactly one | Already prepared `SynchronizedMedia`; bypasses automatic normalization |
| `segment` | yes | Script Segment this take corresponds to — e.g. `{story.segment.hook}` |
+| `frame` | visual only | Override the Spine's `visual-frame` for this Take |
+| `appearance` | visual only | Override the Spine's `visual-appearance` for this Take |
+| `z` | visual only | Override the Spine's `visual-z` for this Take |
-`video` and `media` are mutually exclusive. The normal AIGC path is `video={take.video}`. The
+`video`, `audio` and `media` are mutually exclusive. The normal AIGC path is `video={take.video}`. The
Speech Surface expands that readable declaration into ordinary Media Pipeline Operations:
inspect the container, select its primary moving video and default audio stream, then normalize
both to the Spine's declared frame rate. A 30 fps source connected to a 60 fps Spine keeps its
@@ -54,6 +63,17 @@ duration and is deterministically resampled to a 60 fps frame sequence; it is no
fast. Use `media=` only when another graph branch has already produced the exact synchronized value
you intend to assemble.
+The base is deliberately explicit rather than a hidden full-screen default. A visual Take inherits
+all three values unless it overrides them. An audio Take cannot declare visual overrides: while it
+is playing, `speech.visual` simply has no Present, so the Film background or peer Tracks remain
+visible.
+
+The fit Recipe is ordinary compile-time SVS data, for example:
+
+```svs
+speech.visual { fit: cover; }
+```
+
The order of `<speech:Take>` children **determines the program order**. The first take starts at
time zero; each subsequent take follows immediately.
@@ -63,7 +83,7 @@ The Spine produces four outputs used by downstream components:
| Output | Type | Used by |
|---|---|---|
-| `{speech.visual}` | VisualTrack | `film:Film` — the full-screen talking-head video |
+| `{speech.visual}` | VisualTrack | `film:Film` — sparse same-source Take visuals |
| `{speech.audio}` | Audio | `whisperx:Alignment` — raw audio for word timing |
| `{speech.audioTrack}` | AudioTrack | `film:Film` — the synchronized audio |
| `{speech.space}` | ProgramSpace | Everything — the unified duration and frame domain |
@@ -122,6 +142,9 @@ The SemanticMap is the typed bridge between Script text and physical time. When
exact frame range that Selection covers. Without a SemanticMap, Selections and Moments have no
physical meaning.
+A whole Segment needs no synthetic Selection. `during={story.segment.answer}` addresses the
+Segment's existing structural start/end anchors directly.
+
Components that use the map take it via the `map` attribute:
```svml
@@ -142,10 +165,13 @@ The complete timing stage, from generated takes to map and space:
<import as="speech" from="@narratage/speech-spine@1"/>
<import as="whisperx" from="@narratage/whisperx@1"/>
<import as="space" from="@narratage/spatial@1"/>
+<import as="studio" source="./studio.svs"/>
<!-- Assemble takes in program order -->
<space:Canvas id="vertical" width="1080" height="1920"/>
-<speech:Spine id="speech" canvas={vertical} frame-rate="30">
+<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>
+<speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={opening-take.video} segment={story.segment.opening}/>
<speech:Take video={answer-take.video} segment={story.segment.answer}/>
</speech:Spine>
@@ -155,7 +181,7 @@ The complete timing stage, from generated takes to map and space:
<!-- Downstream components now reference: -->
<!-- {speech.space} — ProgramSpace for duration/frame domain -->
-<!-- {speech.visual} — VisualTrack for the talking-head video -->
+<!-- {speech.visual} — sparse same-source speech VisualTrack -->
<!-- {speech.audioTrack} — AudioTrack for synchronized audio -->
<!-- {timing.map} — SemanticMap for Selection/Moment timing -->
```
diff --git a/docs/zh/guide/conventions.md b/docs/zh/guide/conventions.md
index 15f87ad7..b88b9bd7 100644
--- a/docs/zh/guide/conventions.md
+++ b/docs/zh/guide/conventions.md
@@ -21,7 +21,7 @@ description: 命名、模块边界、TypeScript 配置与 wire 数据。
- 每个包有且只有一个公开入口点:`src/index.ts`。
- 内部模块使用显式的 `.js` 扩展名(NodeNext 解析)。
- 跨包导入使用 `@narratage/*`,绝不跨包边界使用相对路径。
-- 禁止循环依赖(由 `tools/package-boundaries.test.mjs` 强制执行)。
+- 禁止生产依赖形成循环。
## TypeScript 配置
diff --git a/docs/zh/guide/packages.md b/docs/zh/guide/packages.md
index b6752e67..bf479a6b 100644
--- a/docs/zh/guide/packages.md
+++ b/docs/zh/guide/packages.md
@@ -135,13 +135,13 @@ Sticker、Deck 或任何模型家族。
## 依赖规则
-`tools/package-boundaries.test.mjs` 在每次提交时强制执行三条不变式:
+包结构遵循三条依赖规则:
1. **无环的生产依赖图。** 任何 `@narratage/*` 包之间都不存在依赖环。
2. **领域无关闭包。** 每个 Layer 1 包的传递闭包只包含 Layer 1 的包。`@narratage/core` 只依赖 `@narratage/protocol`。
-3. **CLI 独立性。** `@narratage/cli` 和 `@narratage/video-cli` 都不会传递依赖任何 Provider 包。video CLI 同样不依赖任何作者层的视频包(`@narratage/script`、`@narratage/seedance`、`@narratage/media-track`、`@narratage/typography-track`、`@narratage/film`)。作者包通过显式的 package lock 被 activate,而不是通过编译期的 CLI 依赖。
+3. **CLI 独立性。** `@narratage/cli` 和 `@narratage/video-cli` 都不会传递依赖任何 Provider 包。video CLI 同样不依赖任何作者层的视频包(`@narratage/script`、`@narratage/seedance`、`@narratage/media-track`、`@narratage/typography-track`、`@narratage/film`)。作者包通过显式的 package lock 被 activate,而不是通过编译期的 CLI 依赖。这些规则直接体现在 package manifest 中并接受架构审查,不再用源码文本正则来近似证明。
## 包的结构
diff --git a/docs/zh/guide/testing.md b/docs/zh/guide/testing.md
index 09a1514d..d3805c49 100644
--- a/docs/zh/guide/testing.md
+++ b/docs/zh/guide/testing.md
@@ -66,17 +66,11 @@ test("generates a video", async (t) => {
});
```
-### Boundary 测试
+### 架构边界
-`tools/package-boundaries.test.mjs` 和 `tools/graph-first-value-boundary.test.mjs` 会在整个 workspace 范围内校验结构性不变量:
+包边界由 package manifest、公开入口和共享合同表达;测试只验证边界上可观察的行为。
+仓库不再把源码文本正则当作依赖分析或架构审查的替代品。
-- 依赖图无环
-- 领域中立闭包中只包含领域中立的包
-- CLI 不依赖任何 Provider 或作者包
-
-它们作为 `pnpm test` 的一部分,在每次提交时运行。
-
-这些测试在仓库开源时应当保留:协议、图、包边界与确定性单元测试是解耦规则的可执行定义。
发布前应删除客户或品牌 fixture、凭据痕迹、付费产物、工作站绝对路径与一次性验收脚本。
通用 live test 只有在显式 opt-in、不提交秘密,并且未设置开关时能在花钱前退出的情况下才保留。
diff --git a/docs/zh/quickstart/composition.md b/docs/zh/quickstart/composition.md
index e0820c47..b704166a 100644
--- a/docs/zh/quickstart/composition.md
+++ b/docs/zh/quickstart/composition.md
@@ -49,7 +49,7 @@ Composition。然后渲染器将该 Composition 编译为 MP4 视频。
| 来源 | 类型 | 来自 |
|---|---|---|
-| `{speech.visual}` | VisualTrack | `speech:Spine`——全屏说话人画面 |
+| `{speech.visual}` | VisualTrack | `speech:Spine`——稀疏的同源口播视觉 |
| `{speech.audioTrack}` | AudioTrack | `speech:Spine`——同步音频 |
| `{captions.track}` | VisualTrack | Caption 样式族 Track——定时字幕 |
| `{cards.visual}` | VisualTrack | `media-track:Track`——Media 叠加层或 B-roll |
@@ -64,7 +64,7 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T
| stack-order | 内容 |
|---|---|
-| 10 | 语音画面(全屏说话人画面) |
+| 10 | 示例口播视觉(作者显式选择,不是内置默认值) |
| 40 | Media 叠加层 |
| 70 | 字幕 |
| 90 | 文字叠加层 |
@@ -141,13 +141,16 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参
prompt={direction} duration="5"/>
<space:Canvas id="vertical" width="1080" height="1920"/>
+ <space:Frame id="speech-frame" within={vertical}
+ left="0%" top="0%" right="100%" bottom="100%"/>
<space:Frame id="title-frame" within={vertical}
left="6%" top="6%" right="6%" bottom="84%"/>
<space:Frame id="card-frame" within={vertical}
left="10%" top="20%" right="10%" bottom="30%"/>
<!-- 3. Timing: assemble spine and align words -->
- <speech:Spine id="speech" canvas={vertical} frame-rate="30">
+ <speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={take.video} segment={story.segment.opening}/>
</speech:Spine>
<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>
@@ -200,6 +203,7 @@ Build Target,也可以直接接到媒体裁切、音频/帧提取或模型参
film.vertical {
background: #09090B;
}
+ speech.visual { fit: cover; }
media.card {
stack-order: 40; fit: cover; playback: hold-start;
frame-paint: #111116; clip: rounded; radius: 20;
diff --git a/docs/zh/quickstart/timing.md b/docs/zh/quickstart/timing.md
index 3922e29c..b1a93d98 100644
--- a/docs/zh/quickstart/timing.md
+++ b/docs/zh/quickstart/timing.md
@@ -11,6 +11,7 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连
<import as="speech" from="@narratage/speech-spine@1"/>
<import as="whisperx" from="@narratage/whisperx@1"/>
<import as="space" from="@narratage/spatial@1"/>
+<import as="studio" source="./studio.svs"/>
```
## speech:Spine
@@ -19,7 +20,9 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连
```svml
<space:Canvas id="vertical" width="1080" height="1920"/>
-<speech:Spine id="speech" canvas={vertical} frame-rate="30">
+<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>
+<speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={hook-take.video} segment={story.segment.hook}/>
<speech:Take video={meeting-take.video} segment={story.segment.meeting}/>
<speech:Take video={evidence-take.video} segment={story.segment.evidence}/>
@@ -30,25 +33,39 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连
| 属性 | 必填 | 描述 |
|---|---|---|
| `id` | 是 | 唯一标识符 |
-| `canvas` | 是 | 受限 Media 视觉投影显式使用的 CanvasSpace |
| `frame-rate` | 是 | 节目帧率,写整数或有理数,例如 `30`、`30000/1001` |
+| `visual-frame` | 是 | 同源 Take 视觉的显式基础 SpatialFrame |
+| `visual-appearance` | 是 | 只包含空间 fit 属性的 SVS Recipe |
+| `visual-z` | 是 | 同源 Take 视觉的基础绝对层级 |
### speech:Take
-每个 `<speech:Take>` 子元素将一个生成的视频绑定到一个 Script Segment:
+每个 `<speech:Take>` 子元素将一个带口播的来源绑定到一个 Script Segment:
| 属性 | 必填 | 描述 |
|---|---|---|
-| `video` | 二选一 | 生成/原始视频 Blob,例如 Seedance 产出的 `{take.video}` |
-| `media` | 二选一 | 已准备好的 `SynchronizedMedia`;跳过自动规范化 |
+| `video` | 三选一 | 生成/原始视频 Blob,例如 Seedance 产出的 `{take.video}` |
+| `audio` | 三选一 | 纯语音 Blob;贡献节目时长与主音频,不贡献视觉 |
+| `media` | 三选一 | 已准备好的 `SynchronizedMedia`;跳过自动规范化 |
| `segment` | 是 | 此片段对应的 Script Segment——例如 `{story.segment.hook}` |
+| `frame` | 仅视觉 | 覆盖本 Take 的基础 `visual-frame` |
+| `appearance` | 仅视觉 | 覆盖本 Take 的基础 `visual-appearance` |
+| `z` | 仅视觉 | 覆盖本 Take 的基础 `visual-z` |
-`video` 与 `media` 必须且只能选一个。AIGC 的正常路径就是 `video={take.video}`。Speech
+`video`、`audio` 与 `media` 必须且只能选一个。AIGC 的正常路径就是 `video={take.video}`。Speech
Surface 会把这句易读声明展开为普通 Media Pipeline Operation:检查容器、选择主动态视频流
和默认音轨,再按 Spine 声明的帧率规范化。把 30 fps 素材接入 60 fps Spine 时,时长不变,
帧序列会被确定性重采样为 60 fps,而不是把视频播放加速一倍。只有上游图已经明确产出所需
`SynchronizedMedia` 时才使用 `media=`。
+视觉 Take 默认继承 Spine 上显式写出的三个视觉值,也可以逐段覆盖。纯音频 Take 禁止写
+视觉覆盖;它播放时 `speech.visual` 就没有 Present,Film 背景或平级 Track 会自然露出,
+系统不会伪造黑场。fit Recipe 只是普通编译期 SVS 数据,例如:
+
+```svs
+speech.visual { fit: cover; }
+```
+
`<speech:Take>` 子元素的排列顺序**决定了节目顺序**。第一个片段从时间零点开始;后续片段依次紧接。
### 输出
@@ -57,7 +74,7 @@ Spine 产生四个输出,供下游组件使用:
| 输出 | 类型 | 使用方 |
|---|---|---|
-| `{speech.visual}` | VisualTrack | `film:Film`——全屏真人出镜视频 |
+| `{speech.visual}` | VisualTrack | `film:Film`——稀疏的同源 Take 视觉 |
| `{speech.audio}` | Audio | `whisperx:Alignment`——用于词级时序的原始音频 |
| `{speech.audioTrack}` | AudioTrack | `film:Film`——同步音频 |
| `{speech.space}` | ProgramSpace | 所有组件——统一的时长和帧域 |
@@ -108,6 +125,8 @@ ProgramSpace 包含:
SemanticMap 是连接 Script 文本与物理时间的类型化桥梁。当你在 Media Item 上写 `during={story.selection.demo}` 时,组件会使用 SemanticMap 查找该 Selection 覆盖的精确帧范围。没有 SemanticMap,Selection 和 Moment 就没有物理意义。
+整段使用无需伪造 Selection:`during={story.segment.answer}` 直接消费 Segment 已有的首尾结构锚点。
+
使用该映射的组件通过 `map` 属性接收它:
```svml
@@ -127,10 +146,13 @@ Map 只包含最终词窗口和语义锚点,不传播“测量、推导、估
<import as="speech" from="@narratage/speech-spine@1"/>
<import as="whisperx" from="@narratage/whisperx@1"/>
<import as="space" from="@narratage/spatial@1"/>
+<import as="studio" source="./studio.svs"/>
<!-- Assemble takes in program order -->
<space:Canvas id="vertical" width="1080" height="1920"/>
-<speech:Spine id="speech" canvas={vertical} frame-rate="30">
+<space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>
+<speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={opening-take.video} segment={story.segment.opening}/>
<speech:Take video={answer-take.video} segment={story.segment.answer}/>
</speech:Spine>
@@ -140,7 +162,7 @@ Map 只包含最终词窗口和语义锚点,不传播“测量、推导、估
<!-- Downstream components now reference: -->
<!-- {speech.space} — ProgramSpace for duration/frame domain -->
-<!-- {speech.visual} — VisualTrack for the talking-head video -->
+<!-- {speech.visual} — 稀疏的同源口播 VisualTrack -->
<!-- {speech.audioTrack} — AudioTrack for synchronized audio -->
<!-- {timing.map} — SemanticMap for Selection/Moment timing -->
```
diff --git a/examples/talking-film-golden/main.svml b/examples/talking-film-golden/main.svml
index 8ba56335..3caa19e2 100644
--- a/examples/talking-film-golden/main.svml
+++ b/examples/talking-film-golden/main.svml
@@ -46,6 +46,8 @@
<media:Image id="product-reference" src="./assets/product.png"/>
<space:Canvas id="vertical" width="1080" height="1920"/>
+ <space:Frame id="speech-frame" within={vertical}
+ left="0%" top="0%" right="100%" bottom="100%"/>
<space:Frame id="title-frame" within={vertical}
left="6%" top="6%" right="6%" bottom="84%"/>
<space:Frame id="product-frame" within={vertical}
@@ -113,7 +115,8 @@
</seedance:ReferenceVideo>
<!-- Ordered generated Takes become one audio basis plus its peer visual/audio Tracks. -->
- <speech:Spine id="speech" canvas={vertical} frame-rate="30">
+ <speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={alice-take.video} segment={script.segment.opening}/>
<speech:Take video={bob-take.video} segment={script.segment.answer}/>
</speech:Spine>
diff --git a/examples/talking-film-golden/studio.svs b/examples/talking-film-golden/studio.svs
index 44279403..9ed2389c 100644
--- a/examples/talking-film-golden/studio.svs
+++ b/examples/talking-film-golden/studio.svs
@@ -19,14 +19,7 @@
background: #09090B;
}
- speech.full {
- stack-order: 10;
- x: 0;
- y: 0;
- width: 1;
- height: 1;
- fit: cover;
- }
+ speech.visual { fit: cover; }
media.product {
stack-order: 40;
diff --git a/examples/talking-film-graph-check/main.svml b/examples/talking-film-graph-check/main.svml
index d65384ec..775c4574 100644
--- a/examples/talking-film-graph-check/main.svml
+++ b/examples/talking-film-graph-check/main.svml
@@ -22,6 +22,8 @@
</script>
<space:Canvas id="vertical" width="1080" height="1920"/>
+ <space:Frame id="speech-frame" within={vertical}
+ left="0%" top="0%" right="100%" bottom="100%"/>
<space:Frame id="title-frame" within={vertical}
left="6%" top="6%" right="6%" bottom="84%"/>
<space:Frame id="card-frame" within={vertical}
@@ -31,7 +33,8 @@
<seedance:TextVideo id="take" model="mini" prompt={direction} duration="5" generate-audio="true"/>
<seedance:TextVideo id="motion" model="mini" prompt={direction} duration="5"/>
- <speech:Spine id="speech" canvas={vertical} frame-rate="30">
+ <speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={take.video} segment={story.segment.opening}/>
</speech:Spine>
<whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>
diff --git a/examples/talking-film-graph-check/studio.svs b/examples/talking-film-graph-check/studio.svs
index 99821095..79a68cfb 100644
--- a/examples/talking-film-graph-check/studio.svs
+++ b/examples/talking-film-graph-check/studio.svs
@@ -2,6 +2,7 @@
<sheet version="1">
film.vertical { background: #09090B; }
+ speech.visual { fit: cover; }
media.card {
stack-order: 40; fit: cover; playback: hold-start;
frame-paint: #111116; clip: rounded; radius: 20;
diff --git a/examples/talking-film-graph-check/svml.packages.lock b/examples/talking-film-graph-check/svml.packages.lock
index eaa02f72..55005928 100644
--- a/examples/talking-film-graph-check/svml.packages.lock
+++ b/examples/talking-film-graph-check/svml.packages.lock
@@ -597,7 +597,7 @@
{
"name": "@narratage/caption",
"version": "0.0.0-dev",
- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"
+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"
},
{
"name": "@narratage/component-kit",
@@ -617,7 +617,7 @@
{
"name": "@narratage/elaborator",
"version": "0.0.0-dev",
- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"
+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"
},
{
"name": "@narratage/film",
@@ -652,17 +652,17 @@
{
"name": "@narratage/media-pipeline",
"version": "0.0.0-dev",
- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"
+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"
},
{
"name": "@narratage/media-track",
"version": "0.0.0-dev",
- "digest": "sha256:b99121eb2f0f5307c19115a00af5fc7b218440c87e4b4caeded9d4570d6547d0"
+ "digest": "sha256:08af7b82b3f4182fba2f3cc4cd12aee34452c2c2cf0cdbf2c3da48422a5b5d5e"
},
{
"name": "@narratage/media",
"version": "0.0.0-dev",
- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"
+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"
},
{
"name": "@narratage/model-kit",
@@ -712,7 +712,7 @@
{
"name": "@narratage/semantic-map",
"version": "0.0.0-dev",
- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"
+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"
},
{
"name": "@narratage/source",
@@ -722,17 +722,17 @@
{
"name": "@narratage/spatial",
"version": "0.0.0-dev",
- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"
+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"
},
{
"name": "@narratage/speech-alignment",
"version": "0.0.0-dev",
- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"
+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"
},
{
"name": "@narratage/speech-basis",
"version": "0.0.0-dev",
- "digest": "sha256:01532b3694db77f9d090b6474440144bbf1245682befa00a4cd44e77e2395b0a"
+ "digest": "sha256:3e6952237c7bc44fb85c873fae2850999fbb77f90e397a0f9441046aa64ad9d7"
},
{
"name": "@narratage/speech-evidence",
@@ -742,12 +742,12 @@
{
"name": "@narratage/speech-spine",
"version": "0.0.0-dev",
- "digest": "sha256:84898416ea156ceefd5c090e18b560355741f95451cbf90e99e8f6a639986da1"
+ "digest": "sha256:03a18dbf36764e3cd951fb46eb8badb69ead040da5f7b6d258a5bb125950f0a1"
},
{
"name": "@narratage/speech",
"version": "0.0.0-dev",
- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"
+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"
},
{
"name": "@narratage/svs",
@@ -757,7 +757,7 @@
{
"name": "@narratage/temporal",
"version": "0.0.0-dev",
- "digest": "sha256:b20cb315f5fc365a2e9994c63989a0cc1edc46b5843e06a337827d21d74068a1"
+ "digest": "sha256:6af6c0bf4e2cdb23240125815dddd831dcacfe793055150c58f4dfd37ce9e350"
},
{
"name": "@narratage/text",
@@ -782,7 +782,7 @@
{
"name": "@narratage/whisperx",
"version": "0.0.0-dev",
- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"
+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"
}
],
"packages": [
@@ -801,8 +801,8 @@
"name": "@narratage/caption",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:6a2991ec75f9fd30e8d286213fc7993aae2c570b85a8faae5d3c2af661c9fdfc",
- "closureDigest": "sha256:ee79cd66af6e561432e52f1bf78f597f9dc01cd09d12d68c26e7cf61e7361aac"
+ "facetsDigest": "sha256:b8434ae33703de6d3c41c1426ae76083b19e398e14efe8589f2c01a92a9d4e8d",
+ "closureDigest": "sha256:acd4bf7e979294b1b69fbab08a67b627be720c7e90226d94460490e76a9b4b07"
},
{
"specifier": "@narratage/caption-fine",
@@ -810,8 +810,8 @@
"name": "@narratage/caption-fine",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:c5616408dd56ce9c903523319b2a697ff6b8af5bfe1dc66fe793a11e4629f3cf",
- "closureDigest": "sha256:782a9f2c0345431e06c8ab009d518933c964465b34d790b4064f87170e0495ff"
+ "facetsDigest": "sha256:2fad0cbda2ae17fe67f5816e99f493d6ebfce232e2f526c5c9299a35ff0429f9",
+ "closureDigest": "sha256:0aa2796ab56a70cb4e7db6fab3394cdb2d119397f3144f0f1f18cc59f30af29b"
},
{
"specifier": "@narratage/caption-gemini",
@@ -819,8 +819,8 @@
"name": "@narratage/caption-gemini",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:22fed1f1748349c11b6f28448be38cbea68b1dd5a51a3dc2d548ae0a376b9c71",
- "closureDigest": "sha256:3cc6fd7d5250e5457db548fa2432f33d62012db8f77e1c8fe4f590e38a3439ab"
+ "facetsDigest": "sha256:d4f1cf46a646ff5cf6294307af7f77a453a000eaf2b168099d42f41d1fbbcf50",
+ "closureDigest": "sha256:4f6bddb12ebf1b61a1297d09ca1ab4080bab20523487e73f66d53be2a28f99e4"
},
{
"specifier": "@narratage/composition",
@@ -828,8 +828,8 @@
"name": "@narratage/composition",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:0a988b665baf0846246e309d9faf39ffb3ef6ec455bf37f7c438eb30429ff46e",
- "closureDigest": "sha256:15c9088a176b04a6051b2398e308bde8fab6fb15d0147786e9e40abfce772662"
+ "facetsDigest": "sha256:d0c91a61caa7303210d898183ae355ab6c5e0eb1de8bd5a0c237cd4388693104",
+ "closureDigest": "sha256:937caaaab137467eaaef1e9974b103498a76d0b915690ffe404d800b5f84b706"
},
{
"specifier": "@narratage/film",
@@ -837,8 +837,8 @@
"name": "@narratage/film",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:8e58b5630b9a68a8ce7314620f5f1c7cda973a05c179c18d61e10d68002964f0",
- "closureDigest": "sha256:2a4b2b66d9f3913bc839f052e35e6c5cd6afed08821a3308c643b16dcd011f44"
+ "facetsDigest": "sha256:f6e47bdf84726556e6ee174da07b7dd435b6e7b9865065f7d7cb29ec0ed5b166",
+ "closureDigest": "sha256:cfb335a738929c1153350cdc14f7a579679566a558232b0786945a41cf18eaa5"
},
{
"specifier": "@narratage/fonts-open",
@@ -846,8 +846,8 @@
"name": "@narratage/fonts-open",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:31b51352edcb4f34beccc9d9ac8cf7397125a840971a72c912df3d34d1f3feda",
- "closureDigest": "sha256:4594c947a3425c6c42738e2e7d0f7ce56a6efaced91fcf6085e282c3ad98b797"
+ "facetsDigest": "sha256:2117122b06d7c1cd6bee519e72aff22bc3b50f179b50da21dcc84ddf7014457f",
+ "closureDigest": "sha256:65fe769fd8fe5de6a8ef1f33a63bd843ce94fc63dd9bacafecef862b2fadbc25"
},
{
"specifier": "@narratage/generation",
@@ -856,7 +856,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:d74a57a3210267e665366513324db5c313b3ca0f770da66e74cc38aee5048efb",
- "closureDigest": "sha256:3bd4bfdd124001b83602627b0305d6652e96f43cbb9345e3b9bece51a3d701ce"
+ "closureDigest": "sha256:dd00aba4da27ff3bfcec6e5bc24d86a07ac94344701c9936972b66ab5fc64628"
},
{
"specifier": "@narratage/hyperframes",
@@ -864,8 +864,8 @@
"name": "@narratage/hyperframes",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:b8b8773d03f19844cdc8d714650380145056649245e33c607f30220fff194157",
- "closureDigest": "sha256:88d7a9fedb614166e149e29d064be1dca18f1c4a102759f1a2600afba4953b8b"
+ "facetsDigest": "sha256:520234d0b029d7229e6595483e2f0d0227d20cf662363163975691e2471a767e",
+ "closureDigest": "sha256:118b9b787b8df4bea1fbf4cfc09d65627d7670f96db1984a1ae6f0c649e5d43b"
},
{
"specifier": "@narratage/media",
@@ -873,8 +873,8 @@
"name": "@narratage/media",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:6c7cf563a92aa8e48db6da224697fd5f56c50b636b0327efc17fbbc264502b3f",
- "closureDigest": "sha256:e8edbbfb7bec2efa7ad05fa273549a3b033861eb9ef20345430d9be617b26674"
+ "facetsDigest": "sha256:f0974c7d88918ca4796dbb86dea24b6d29f8f5f44430c4845f9928f1df5b22e5",
+ "closureDigest": "sha256:007255553cdbe2c9b82bd71ab920faaae9d107e54e6548adc5e26853505d9146"
},
{
"specifier": "@narratage/media-pipeline",
@@ -882,8 +882,8 @@
"name": "@narratage/media-pipeline",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:21f817f8e6eea9e9e4361005ed5ab9e69117fd77ec237fd467fb11afe9965019",
- "closureDigest": "sha256:3555f1939c82dfa216902fd600e2616ecfce6a244e240b060080621367333d97"
+ "facetsDigest": "sha256:e83eed5903110d1e8fa38aeac1487f63737f1e2a773f6e3c794dff54ad66122a",
+ "closureDigest": "sha256:fde796ee9bd9d9992fe2cb37d3cf18f539d74f16e55e847a586ad013d7f87705"
},
{
"specifier": "@narratage/media-track",
@@ -891,8 +891,8 @@
"name": "@narratage/media-track",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:bb26c54c0ad8802ee8c7cd6f13314c90dd7de7e937ded1c501367e19693fe3b1",
- "closureDigest": "sha256:5b8e32d627c9f55e5dbe8a4a85348b44852070f2e6796e3d19bbd7a0e42a49b6"
+ "facetsDigest": "sha256:ed087e0f28c7767b3481fea40c886413bbbc9963a3963ce4480182265140c31a",
+ "closureDigest": "sha256:8c82d8c186f0dd286467056cb79d7210f560b22aa20065e92836e60d0a885f95"
},
{
"specifier": "@narratage/narrative",
@@ -918,8 +918,8 @@
"name": "@narratage/render-hyperframes",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:aa47f04cf0652ee3fa2e114eb6d9a80b0522828a47869103ca33e95027b21634",
- "closureDigest": "sha256:1c9dcb543a3afa6a94c9757fd47a3e0bfb68ed7d5f3c87b3ef0f0e52ed286332"
+ "facetsDigest": "sha256:b5a0bad5b8809158caea96274f94c397e18086febcd8b11eb3f4360bb869f06e",
+ "closureDigest": "sha256:67e343cebb44e6d3cfe929356e5363b317f9462e7240fc411c5bb0df87bbf17f"
},
{
"specifier": "@narratage/run-markup",
@@ -928,7 +928,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:fb3a8a5be9a0ef0aa914026a199be641f4e20df27f9b284be196f52fd260be55",
- "closureDigest": "sha256:20b6fcbacf36352d6c180e1a466eaf368c0bef183eec09396456e2e2375ea16f"
+ "closureDigest": "sha256:b3d5afceb67cb1275867e99f0aa384f4d339d2b759443d6822352bc8714b8bc7"
},
{
"specifier": "@narratage/script",
@@ -937,7 +937,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:69e1c08a5f07c73a506ca8286426a0386485738649e86463cca0b291f319ee4e",
- "closureDigest": "sha256:696bb2e69c89d9e1079a76ce68d26fe2c99615275a4266bf24fa905ad77c1bb4"
+ "closureDigest": "sha256:cc5ebec9ea7bf21d23d67536dac321618679037e811042d2b349688a373ffc4d"
},
{
"specifier": "@narratage/seedance",
@@ -945,8 +945,8 @@
"name": "@narratage/seedance",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:e889ca3808a19f544e7aa5e27453beb5a09464c645633b0b1b1d9a4b7ce6591a",
- "closureDigest": "sha256:e1c45192a92620ec8ea03d9cf1b19073bae2dbfde12475f8436cd5c008c854bd"
+ "facetsDigest": "sha256:0832f344b8c327e316a153d8ca4559da5eba452f3e334ffb49c7e94a29f36ecf",
+ "closureDigest": "sha256:3ef4370d01ffcfef671cae12e1a250b1216caaa373b335246d2bfb5d11b148cf"
},
{
"specifier": "@narratage/semantic-map",
@@ -954,8 +954,8 @@
"name": "@narratage/semantic-map",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:a9d81ddfa09dfa23c9c3c0817b60299431ca9ec640252a659d490be55d92ba68",
- "closureDigest": "sha256:3ac5deb912231d3092666c692d73a379129d1cf2ed035d43a95bef6d5101d525"
+ "facetsDigest": "sha256:836a539f60d3a77a122d0bae169f03e229566dfc4696001143e9bd32c313dc12",
+ "closureDigest": "sha256:a6313c3c407510323d7b5f856aaae3935f47cd80869b099b961aa34c9eb6c1ba"
},
{
"specifier": "@narratage/spatial",
@@ -964,7 +964,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:5062281ffadb46495ec12e1a67185b57fe13879e90f03bcf544e93fd1d273716",
- "closureDigest": "sha256:3209c05b608c20aedab35299235974d74ee4374ead722ec125f2b6ea70829c77"
+ "closureDigest": "sha256:e328230a8de70bd5c4ec4af65db9f964f0ffe20900358ba4a59fff0d29dd8ac7"
},
{
"specifier": "@narratage/speech",
@@ -972,8 +972,8 @@
"name": "@narratage/speech",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:75b1537c9b9729c2e9197445c34b295bc9f72dbef754a4702e1ca792dceea472",
- "closureDigest": "sha256:a5be05750141d51e4f4460945b779bf0196548d7b7c908eca34d8c63c2ec7b70"
+ "facetsDigest": "sha256:ed94af4fb62357b30f8faec7f16ccb2b2d055deecd300c7ca948b112b56b710c",
+ "closureDigest": "sha256:cae66d28f2f5598824877a845f996798a35b17c84043dc261678ead81c602e86"
},
{
"specifier": "@narratage/speech-alignment",
@@ -981,8 +981,8 @@
"name": "@narratage/speech-alignment",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:bdfe219081310041d11c6d54440e6ccb80398bfb97f93afc28e9d1ed101b4a79",
- "closureDigest": "sha256:de73bc9d1411af8319a099df658dfc51f6e7b44dcbf3173cb5f9db18edb449cf"
+ "facetsDigest": "sha256:fe041c994c1974eddf582703869f1b8718eab817c1a0eb5e2edd0ae98644f8d9",
+ "closureDigest": "sha256:8d611e7523965c7a446d95beead34533db5eb94c6ed7666d14381cf9c77629fc"
},
{
"specifier": "@narratage/speech-basis",
@@ -990,8 +990,8 @@
"name": "@narratage/speech-basis",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:f01f21badb0ccdd5cc62b131be05f46030bffee96042c7323a0c572786d012ef",
- "closureDigest": "sha256:843a7922a70d5df180ade4ec3a8becf8fb110f6137fa6a38d86cc5aaa14830e1"
+ "facetsDigest": "sha256:5badee4556d51ff8075eb2a4531b5bf9ff65069ce9cce724b870b21e135c2129",
+ "closureDigest": "sha256:151aff0a543fef15b471e6914324594b333f68f081e9835c29bc52826be2c825"
},
{
"specifier": "@narratage/speech-evidence",
@@ -999,8 +999,8 @@
"name": "@narratage/speech-evidence",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:60f366de074a7fba20c71dcbd825083fddc54ea7de602a903eb144eab5555425",
- "closureDigest": "sha256:918172938ecc1e9b6b1cff9465048bdfe2dcbfe8748971101f78f12ee36f1d75"
+ "facetsDigest": "sha256:f6dfc644f0dd9975af48081fb4b1b7ac23154c4bccd2aeceff19cee05697b757",
+ "closureDigest": "sha256:62f0174c12061f87acd28f1aa0d816cb1ff4c80956040737ad55f1d4a3bf4838"
},
{
"specifier": "@narratage/speech-spine",
@@ -1008,8 +1008,8 @@
"name": "@narratage/speech-spine",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:a27dd4349d6bb1aaa7f382ad09f883e9894157579e48285b00e8fa24456190f7",
- "closureDigest": "sha256:f056e14a44f4f2cc0751b28e1ecb9831b7ff472034a02cac8089b9ff45d67854"
+ "facetsDigest": "sha256:d20cdc013e61ea8d7a632d7cea2c10a9186a0bf0736ece2ff0f307c1320b6459",
+ "closureDigest": "sha256:16eb190b79628257a3100ca81b7cbe26ccb1e4d6e95b6f93c629dafac07a2c8d"
},
{
"specifier": "@narratage/svs",
@@ -1018,7 +1018,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:9cc7c397d2cd97b5a3e814886b6553b6d236fd8825df0d056e328b7f0cf20163",
- "closureDigest": "sha256:2be69621b66211f48f0f7181bda377dd9e6f9a90067e1d0640fa2d762a0f19d1"
+ "closureDigest": "sha256:9358fe74dcc74e0bd730d9043292e044843e2e9eb003ee9db24e73f81ac495df"
},
{
"specifier": "@narratage/temporal",
@@ -1027,7 +1027,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:2e9c4804f142e08e87b2e11af79aa17770109fb573e865397bed14ed7a4f1d0e",
- "closureDigest": "sha256:7a2df8dba92e1005388b373be07f4fe3065bb3d4b7106e19139a4fe1cadc090b"
+ "closureDigest": "sha256:fb131fd446a2e565a29a1850df39e5ba0bc709ee5f128346f9dcc3b1583c4189"
},
{
"specifier": "@narratage/text",
@@ -1036,7 +1036,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:4435dd4bd38e5e017013f775e28e379e3e235211649ee8f3913378f7f9e60b58",
- "closureDigest": "sha256:c6ba3301621d63598421b7d3f0dc2e1bcc8967f16624407d37a20c2c0703a246"
+ "closureDigest": "sha256:9cab1e68a2273b7cf749c06033d63c2bdc28690b55fc0c755921512f0c3a701f"
},
{
"specifier": "@narratage/typography-track",
@@ -1044,8 +1044,8 @@
"name": "@narratage/typography-track",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:cc8349fec027f43e01a8832ba8a7050490e924851cee3325d3568cb5bc5fd57f",
- "closureDigest": "sha256:840cec885e08123174809ef2dcf357120203b3fb5e2af3b22e0c6177d2e61af1"
+ "facetsDigest": "sha256:a9a59e547f2db875f17fb52a87a1b3ff6921c87f065a5964c74be00f3cf1ac8f",
+ "closureDigest": "sha256:6c8598a278b61a2deae144c2a9745e379611796a268c3a25a020e27b9df9d4c0"
},
{
"specifier": "@narratage/visual-ir",
@@ -1062,9 +1062,9 @@
"name": "@narratage/whisperx",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:c8cb487d25e567a081c74a7abc97ce2b94dc0c47c36c90ba6b44d2ec21a9e80b",
- "closureDigest": "sha256:98e68de949bf7cfb9e1c483de66a38de6f347fc5032122d10e9af971a28fa534"
+ "facetsDigest": "sha256:9e765c9776d31ceeb1f0898fa3df9d7345747fa0fd5a60d46728f8d78c8a69df",
+ "closureDigest": "sha256:d0abf29a19cbc43aa0e0cfe239670d4e339da684f89ed5ad413b1aabea18e165"
}
],
- "digest": "sha256:c2d6ceb3e801c9045047d59040be4e3f9a297f3a245f51cc8425ce7e5d9c3043"
+ "digest": "sha256:4c4ab7a6d2674b5a3a6f0078c51539c0b995852e6bff422cf57f795f8382baa0"
}
diff --git a/examples/talking-film-live/main.svml b/examples/talking-film-live/main.svml
index da813b6e..842982ba 100644
--- a/examples/talking-film-live/main.svml
+++ b/examples/talking-film-live/main.svml
@@ -86,8 +86,10 @@
</seedance:ReferenceVideo>
<space:Canvas id="vertical" width="480" height="854"/>
+ <space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>
- <speech:Spine id="speech" canvas={vertical} frame-rate="30">
+ <speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={opening-take.video} segment={story.segment.opening}/>
<speech:Take video={answer-take.video} segment={story.segment.answer}/>
</speech:Spine>
diff --git a/examples/talking-film-live/studio.svs b/examples/talking-film-live/studio.svs
index 92261acb..adf36b48 100644
--- a/examples/talking-film-live/studio.svs
+++ b/examples/talking-film-live/studio.svs
@@ -5,6 +5,8 @@
background: #09090B;
}
+ speech.visual { fit: cover; }
+
caption.primary {
cue-min-words: 2;
cue-max-words: 5;
diff --git a/examples/talking-film-live/svml.packages.lock b/examples/talking-film-live/svml.packages.lock
index 71244e89..4a08bb9b 100644
--- a/examples/talking-film-live/svml.packages.lock
+++ b/examples/talking-film-live/svml.packages.lock
@@ -598,7 +598,7 @@
{
"name": "@narratage/caption",
"version": "0.0.0-dev",
- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"
+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"
},
{
"name": "@narratage/component-kit",
@@ -618,7 +618,7 @@
{
"name": "@narratage/elaborator",
"version": "0.0.0-dev",
- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"
+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"
},
{
"name": "@narratage/estimate",
@@ -658,17 +658,17 @@
{
"name": "@narratage/media-pipeline",
"version": "0.0.0-dev",
- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"
+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"
},
{
"name": "@narratage/media-track",
"version": "0.0.0-dev",
- "digest": "sha256:b99121eb2f0f5307c19115a00af5fc7b218440c87e4b4caeded9d4570d6547d0"
+ "digest": "sha256:08af7b82b3f4182fba2f3cc4cd12aee34452c2c2cf0cdbf2c3da48422a5b5d5e"
},
{
"name": "@narratage/media",
"version": "0.0.0-dev",
- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"
+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"
},
{
"name": "@narratage/model-kit",
@@ -718,7 +718,7 @@
{
"name": "@narratage/semantic-map",
"version": "0.0.0-dev",
- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"
+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"
},
{
"name": "@narratage/source",
@@ -728,17 +728,17 @@
{
"name": "@narratage/spatial",
"version": "0.0.0-dev",
- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"
+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"
},
{
"name": "@narratage/speech-alignment",
"version": "0.0.0-dev",
- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"
+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"
},
{
"name": "@narratage/speech-basis",
"version": "0.0.0-dev",
- "digest": "sha256:01532b3694db77f9d090b6474440144bbf1245682befa00a4cd44e77e2395b0a"
+ "digest": "sha256:3e6952237c7bc44fb85c873fae2850999fbb77f90e397a0f9441046aa64ad9d7"
},
{
"name": "@narratage/speech-evidence",
@@ -748,12 +748,12 @@
{
"name": "@narratage/speech-spine",
"version": "0.0.0-dev",
- "digest": "sha256:84898416ea156ceefd5c090e18b560355741f95451cbf90e99e8f6a639986da1"
+ "digest": "sha256:03a18dbf36764e3cd951fb46eb8badb69ead040da5f7b6d258a5bb125950f0a1"
},
{
"name": "@narratage/speech",
"version": "0.0.0-dev",
- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"
+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"
},
{
"name": "@narratage/svs",
@@ -763,7 +763,7 @@
{
"name": "@narratage/temporal",
"version": "0.0.0-dev",
- "digest": "sha256:b20cb315f5fc365a2e9994c63989a0cc1edc46b5843e06a337827d21d74068a1"
+ "digest": "sha256:6af6c0bf4e2cdb23240125815dddd831dcacfe793055150c58f4dfd37ce9e350"
},
{
"name": "@narratage/text",
@@ -788,7 +788,7 @@
{
"name": "@narratage/whisperx",
"version": "0.0.0-dev",
- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"
+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"
},
{
"name": "cmu-pronouncing-dictionary",
@@ -812,8 +812,8 @@
"name": "@narratage/caption",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:6a2991ec75f9fd30e8d286213fc7993aae2c570b85a8faae5d3c2af661c9fdfc",
- "closureDigest": "sha256:ee79cd66af6e561432e52f1bf78f597f9dc01cd09d12d68c26e7cf61e7361aac"
+ "facetsDigest": "sha256:b8434ae33703de6d3c41c1426ae76083b19e398e14efe8589f2c01a92a9d4e8d",
+ "closureDigest": "sha256:acd4bf7e979294b1b69fbab08a67b627be720c7e90226d94460490e76a9b4b07"
},
{
"specifier": "@narratage/caption-fine",
@@ -821,8 +821,8 @@
"name": "@narratage/caption-fine",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:c5616408dd56ce9c903523319b2a697ff6b8af5bfe1dc66fe793a11e4629f3cf",
- "closureDigest": "sha256:782a9f2c0345431e06c8ab009d518933c964465b34d790b4064f87170e0495ff"
+ "facetsDigest": "sha256:2fad0cbda2ae17fe67f5816e99f493d6ebfce232e2f526c5c9299a35ff0429f9",
+ "closureDigest": "sha256:0aa2796ab56a70cb4e7db6fab3394cdb2d119397f3144f0f1f18cc59f30af29b"
},
{
"specifier": "@narratage/caption-gemini",
@@ -830,8 +830,8 @@
"name": "@narratage/caption-gemini",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:22fed1f1748349c11b6f28448be38cbea68b1dd5a51a3dc2d548ae0a376b9c71",
- "closureDigest": "sha256:3cc6fd7d5250e5457db548fa2432f33d62012db8f77e1c8fe4f590e38a3439ab"
+ "facetsDigest": "sha256:d4f1cf46a646ff5cf6294307af7f77a453a000eaf2b168099d42f41d1fbbcf50",
+ "closureDigest": "sha256:4f6bddb12ebf1b61a1297d09ca1ab4080bab20523487e73f66d53be2a28f99e4"
},
{
"specifier": "@narratage/composition",
@@ -839,8 +839,8 @@
"name": "@narratage/composition",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:0a988b665baf0846246e309d9faf39ffb3ef6ec455bf37f7c438eb30429ff46e",
- "closureDigest": "sha256:15c9088a176b04a6051b2398e308bde8fab6fb15d0147786e9e40abfce772662"
+ "facetsDigest": "sha256:d0c91a61caa7303210d898183ae355ab6c5e0eb1de8bd5a0c237cd4388693104",
+ "closureDigest": "sha256:937caaaab137467eaaef1e9974b103498a76d0b915690ffe404d800b5f84b706"
},
{
"specifier": "@narratage/estimate",
@@ -848,8 +848,8 @@
"name": "@narratage/estimate",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:9975815b3109520eb6bba276554a90767a9aebff6498c2768d9faf4763219bef",
- "closureDigest": "sha256:7bbdc658746e65e2bb4199b3de84d1197376f16120708e2eea31558898de662f"
+ "facetsDigest": "sha256:018228974baa7623415df07f4703025f2d09449969439724027836a13e61caf1",
+ "closureDigest": "sha256:9424d9a12d4fcb0b7886c82aabed10a21f6707b3e42ded94f960396818fa8412"
},
{
"specifier": "@narratage/film",
@@ -857,8 +857,8 @@
"name": "@narratage/film",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:8e58b5630b9a68a8ce7314620f5f1c7cda973a05c179c18d61e10d68002964f0",
- "closureDigest": "sha256:2a4b2b66d9f3913bc839f052e35e6c5cd6afed08821a3308c643b16dcd011f44"
+ "facetsDigest": "sha256:f6e47bdf84726556e6ee174da07b7dd435b6e7b9865065f7d7cb29ec0ed5b166",
+ "closureDigest": "sha256:cfb335a738929c1153350cdc14f7a579679566a558232b0786945a41cf18eaa5"
},
{
"specifier": "@narratage/fonts-open",
@@ -866,8 +866,8 @@
"name": "@narratage/fonts-open",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:31b51352edcb4f34beccc9d9ac8cf7397125a840971a72c912df3d34d1f3feda",
- "closureDigest": "sha256:4594c947a3425c6c42738e2e7d0f7ce56a6efaced91fcf6085e282c3ad98b797"
+ "facetsDigest": "sha256:2117122b06d7c1cd6bee519e72aff22bc3b50f179b50da21dcc84ddf7014457f",
+ "closureDigest": "sha256:65fe769fd8fe5de6a8ef1f33a63bd843ce94fc63dd9bacafecef862b2fadbc25"
},
{
"specifier": "@narratage/generation",
@@ -876,7 +876,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:d74a57a3210267e665366513324db5c313b3ca0f770da66e74cc38aee5048efb",
- "closureDigest": "sha256:3bd4bfdd124001b83602627b0305d6652e96f43cbb9345e3b9bece51a3d701ce"
+ "closureDigest": "sha256:dd00aba4da27ff3bfcec6e5bc24d86a07ac94344701c9936972b66ab5fc64628"
},
{
"specifier": "@narratage/hyperframes",
@@ -884,8 +884,8 @@
"name": "@narratage/hyperframes",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:b8b8773d03f19844cdc8d714650380145056649245e33c607f30220fff194157",
- "closureDigest": "sha256:88d7a9fedb614166e149e29d064be1dca18f1c4a102759f1a2600afba4953b8b"
+ "facetsDigest": "sha256:520234d0b029d7229e6595483e2f0d0227d20cf662363163975691e2471a767e",
+ "closureDigest": "sha256:118b9b787b8df4bea1fbf4cfc09d65627d7670f96db1984a1ae6f0c649e5d43b"
},
{
"specifier": "@narratage/media",
@@ -893,8 +893,8 @@
"name": "@narratage/media",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:6c7cf563a92aa8e48db6da224697fd5f56c50b636b0327efc17fbbc264502b3f",
- "closureDigest": "sha256:e8edbbfb7bec2efa7ad05fa273549a3b033861eb9ef20345430d9be617b26674"
+ "facetsDigest": "sha256:f0974c7d88918ca4796dbb86dea24b6d29f8f5f44430c4845f9928f1df5b22e5",
+ "closureDigest": "sha256:007255553cdbe2c9b82bd71ab920faaae9d107e54e6548adc5e26853505d9146"
},
{
"specifier": "@narratage/media-pipeline",
@@ -902,8 +902,8 @@
"name": "@narratage/media-pipeline",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:21f817f8e6eea9e9e4361005ed5ab9e69117fd77ec237fd467fb11afe9965019",
- "closureDigest": "sha256:3555f1939c82dfa216902fd600e2616ecfce6a244e240b060080621367333d97"
+ "facetsDigest": "sha256:e83eed5903110d1e8fa38aeac1487f63737f1e2a773f6e3c794dff54ad66122a",
+ "closureDigest": "sha256:fde796ee9bd9d9992fe2cb37d3cf18f539d74f16e55e847a586ad013d7f87705"
},
{
"specifier": "@narratage/media-track",
@@ -911,8 +911,8 @@
"name": "@narratage/media-track",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:bb26c54c0ad8802ee8c7cd6f13314c90dd7de7e937ded1c501367e19693fe3b1",
- "closureDigest": "sha256:5b8e32d627c9f55e5dbe8a4a85348b44852070f2e6796e3d19bbd7a0e42a49b6"
+ "facetsDigest": "sha256:ed087e0f28c7767b3481fea40c886413bbbc9963a3963ce4480182265140c31a",
+ "closureDigest": "sha256:8c82d8c186f0dd286467056cb79d7210f560b22aa20065e92836e60d0a885f95"
},
{
"specifier": "@narratage/narrative",
@@ -938,8 +938,8 @@
"name": "@narratage/render-hyperframes",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:aa47f04cf0652ee3fa2e114eb6d9a80b0522828a47869103ca33e95027b21634",
- "closureDigest": "sha256:1c9dcb543a3afa6a94c9757fd47a3e0bfb68ed7d5f3c87b3ef0f0e52ed286332"
+ "facetsDigest": "sha256:b5a0bad5b8809158caea96274f94c397e18086febcd8b11eb3f4360bb869f06e",
+ "closureDigest": "sha256:67e343cebb44e6d3cfe929356e5363b317f9462e7240fc411c5bb0df87bbf17f"
},
{
"specifier": "@narratage/run-markup",
@@ -948,7 +948,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:fb3a8a5be9a0ef0aa914026a199be641f4e20df27f9b284be196f52fd260be55",
- "closureDigest": "sha256:20b6fcbacf36352d6c180e1a466eaf368c0bef183eec09396456e2e2375ea16f"
+ "closureDigest": "sha256:b3d5afceb67cb1275867e99f0aa384f4d339d2b759443d6822352bc8714b8bc7"
},
{
"specifier": "@narratage/script",
@@ -957,7 +957,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:69e1c08a5f07c73a506ca8286426a0386485738649e86463cca0b291f319ee4e",
- "closureDigest": "sha256:696bb2e69c89d9e1079a76ce68d26fe2c99615275a4266bf24fa905ad77c1bb4"
+ "closureDigest": "sha256:cc5ebec9ea7bf21d23d67536dac321618679037e811042d2b349688a373ffc4d"
},
{
"specifier": "@narratage/seedance",
@@ -965,8 +965,8 @@
"name": "@narratage/seedance",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:e889ca3808a19f544e7aa5e27453beb5a09464c645633b0b1b1d9a4b7ce6591a",
- "closureDigest": "sha256:e1c45192a92620ec8ea03d9cf1b19073bae2dbfde12475f8436cd5c008c854bd"
+ "facetsDigest": "sha256:0832f344b8c327e316a153d8ca4559da5eba452f3e334ffb49c7e94a29f36ecf",
+ "closureDigest": "sha256:3ef4370d01ffcfef671cae12e1a250b1216caaa373b335246d2bfb5d11b148cf"
},
{
"specifier": "@narratage/semantic-map",
@@ -974,8 +974,8 @@
"name": "@narratage/semantic-map",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:a9d81ddfa09dfa23c9c3c0817b60299431ca9ec640252a659d490be55d92ba68",
- "closureDigest": "sha256:3ac5deb912231d3092666c692d73a379129d1cf2ed035d43a95bef6d5101d525"
+ "facetsDigest": "sha256:836a539f60d3a77a122d0bae169f03e229566dfc4696001143e9bd32c313dc12",
+ "closureDigest": "sha256:a6313c3c407510323d7b5f856aaae3935f47cd80869b099b961aa34c9eb6c1ba"
},
{
"specifier": "@narratage/spatial",
@@ -984,7 +984,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:5062281ffadb46495ec12e1a67185b57fe13879e90f03bcf544e93fd1d273716",
- "closureDigest": "sha256:3209c05b608c20aedab35299235974d74ee4374ead722ec125f2b6ea70829c77"
+ "closureDigest": "sha256:e328230a8de70bd5c4ec4af65db9f964f0ffe20900358ba4a59fff0d29dd8ac7"
},
{
"specifier": "@narratage/speech",
@@ -992,8 +992,8 @@
"name": "@narratage/speech",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:75b1537c9b9729c2e9197445c34b295bc9f72dbef754a4702e1ca792dceea472",
- "closureDigest": "sha256:a5be05750141d51e4f4460945b779bf0196548d7b7c908eca34d8c63c2ec7b70"
+ "facetsDigest": "sha256:ed94af4fb62357b30f8faec7f16ccb2b2d055deecd300c7ca948b112b56b710c",
+ "closureDigest": "sha256:cae66d28f2f5598824877a845f996798a35b17c84043dc261678ead81c602e86"
},
{
"specifier": "@narratage/speech-alignment",
@@ -1001,8 +1001,8 @@
"name": "@narratage/speech-alignment",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:bdfe219081310041d11c6d54440e6ccb80398bfb97f93afc28e9d1ed101b4a79",
- "closureDigest": "sha256:de73bc9d1411af8319a099df658dfc51f6e7b44dcbf3173cb5f9db18edb449cf"
+ "facetsDigest": "sha256:fe041c994c1974eddf582703869f1b8718eab817c1a0eb5e2edd0ae98644f8d9",
+ "closureDigest": "sha256:8d611e7523965c7a446d95beead34533db5eb94c6ed7666d14381cf9c77629fc"
},
{
"specifier": "@narratage/speech-basis",
@@ -1010,8 +1010,8 @@
"name": "@narratage/speech-basis",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:f01f21badb0ccdd5cc62b131be05f46030bffee96042c7323a0c572786d012ef",
- "closureDigest": "sha256:843a7922a70d5df180ade4ec3a8becf8fb110f6137fa6a38d86cc5aaa14830e1"
+ "facetsDigest": "sha256:5badee4556d51ff8075eb2a4531b5bf9ff65069ce9cce724b870b21e135c2129",
+ "closureDigest": "sha256:151aff0a543fef15b471e6914324594b333f68f081e9835c29bc52826be2c825"
},
{
"specifier": "@narratage/speech-evidence",
@@ -1019,8 +1019,8 @@
"name": "@narratage/speech-evidence",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:60f366de074a7fba20c71dcbd825083fddc54ea7de602a903eb144eab5555425",
- "closureDigest": "sha256:918172938ecc1e9b6b1cff9465048bdfe2dcbfe8748971101f78f12ee36f1d75"
+ "facetsDigest": "sha256:f6dfc644f0dd9975af48081fb4b1b7ac23154c4bccd2aeceff19cee05697b757",
+ "closureDigest": "sha256:62f0174c12061f87acd28f1aa0d816cb1ff4c80956040737ad55f1d4a3bf4838"
},
{
"specifier": "@narratage/speech-spine",
@@ -1028,8 +1028,8 @@
"name": "@narratage/speech-spine",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:a27dd4349d6bb1aaa7f382ad09f883e9894157579e48285b00e8fa24456190f7",
- "closureDigest": "sha256:f056e14a44f4f2cc0751b28e1ecb9831b7ff472034a02cac8089b9ff45d67854"
+ "facetsDigest": "sha256:d20cdc013e61ea8d7a632d7cea2c10a9186a0bf0736ece2ff0f307c1320b6459",
+ "closureDigest": "sha256:16eb190b79628257a3100ca81b7cbe26ccb1e4d6e95b6f93c629dafac07a2c8d"
},
{
"specifier": "@narratage/svs",
@@ -1038,7 +1038,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:9cc7c397d2cd97b5a3e814886b6553b6d236fd8825df0d056e328b7f0cf20163",
- "closureDigest": "sha256:2be69621b66211f48f0f7181bda377dd9e6f9a90067e1d0640fa2d762a0f19d1"
+ "closureDigest": "sha256:9358fe74dcc74e0bd730d9043292e044843e2e9eb003ee9db24e73f81ac495df"
},
{
"specifier": "@narratage/temporal",
@@ -1047,7 +1047,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:2e9c4804f142e08e87b2e11af79aa17770109fb573e865397bed14ed7a4f1d0e",
- "closureDigest": "sha256:7a2df8dba92e1005388b373be07f4fe3065bb3d4b7106e19139a4fe1cadc090b"
+ "closureDigest": "sha256:fb131fd446a2e565a29a1850df39e5ba0bc709ee5f128346f9dcc3b1583c4189"
},
{
"specifier": "@narratage/text",
@@ -1056,7 +1056,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:4435dd4bd38e5e017013f775e28e379e3e235211649ee8f3913378f7f9e60b58",
- "closureDigest": "sha256:c6ba3301621d63598421b7d3f0dc2e1bcc8967f16624407d37a20c2c0703a246"
+ "closureDigest": "sha256:9cab1e68a2273b7cf749c06033d63c2bdc28690b55fc0c755921512f0c3a701f"
},
{
"specifier": "@narratage/typography-track",
@@ -1064,8 +1064,8 @@
"name": "@narratage/typography-track",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:cc8349fec027f43e01a8832ba8a7050490e924851cee3325d3568cb5bc5fd57f",
- "closureDigest": "sha256:840cec885e08123174809ef2dcf357120203b3fb5e2af3b22e0c6177d2e61af1"
+ "facetsDigest": "sha256:a9a59e547f2db875f17fb52a87a1b3ff6921c87f065a5964c74be00f3cf1ac8f",
+ "closureDigest": "sha256:6c8598a278b61a2deae144c2a9745e379611796a268c3a25a020e27b9df9d4c0"
},
{
"specifier": "@narratage/visual-ir",
@@ -1082,9 +1082,9 @@
"name": "@narratage/whisperx",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:c8cb487d25e567a081c74a7abc97ce2b94dc0c47c36c90ba6b44d2ec21a9e80b",
- "closureDigest": "sha256:98e68de949bf7cfb9e1c483de66a38de6f347fc5032122d10e9af971a28fa534"
+ "facetsDigest": "sha256:9e765c9776d31ceeb1f0898fa3df9d7345747fa0fd5a60d46728f8d78c8a69df",
+ "closureDigest": "sha256:d0abf29a19cbc43aa0e0cfe239670d4e339da684f89ed5ad413b1aabea18e165"
}
],
- "digest": "sha256:a224a78bd87743c777eadf978452e86d4cba400b1cf5075057cf0aafac26a406"
+ "digest": "sha256:ff48f6c0eaef7beb70cd23ed010836d513a4e0d678c7450d164fd792b75e5288"
}
diff --git a/examples/talking-film-live/svml.runtime-packages.lock b/examples/talking-film-live/svml.runtime-packages.lock
index ef94a1d9..a27d165c 100644
--- a/examples/talking-film-live/svml.runtime-packages.lock
+++ b/examples/talking-film-live/svml.runtime-packages.lock
@@ -60,12 +60,12 @@
{
"name": "@narratage/caption",
"version": "0.0.0-dev",
- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"
+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"
},
{
"name": "@narratage/compiler-node",
"version": "0.0.0-dev",
- "digest": "sha256:4626882f405a083281ce7ffec73fa598640223669908b36715d8281fa19be92e"
+ "digest": "sha256:804eac87761fe0465234647c2309b6396ed4fd0a886346afef9167d327c3bb9f"
},
{
"name": "@narratage/component-kit",
@@ -95,7 +95,7 @@
{
"name": "@narratage/elaborator",
"version": "0.0.0-dev",
- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"
+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"
},
{
"name": "@narratage/endpoint-kit",
@@ -120,7 +120,7 @@
{
"name": "@narratage/local",
"version": "0.0.0-dev",
- "digest": "sha256:cfbd3c13a81dbc21378bebd22662e93a201480e1bfe725f04e78534764f86c0e"
+ "digest": "sha256:d2bece5a791dc3b7a88f890a54142b5e0cc5a534485ff4e20d5740787a8b90a1"
},
{
"name": "@narratage/markup",
@@ -130,17 +130,17 @@
{
"name": "@narratage/media-execution",
"version": "0.0.0-dev",
- "digest": "sha256:8fe6aca65ca4ebcc1efa71376e986e97741310d836f85647fcb223f39fda9754"
+ "digest": "sha256:3f07960e6723ad386b7fd8858438f2f9e72f6bab23120ffb10462944ac896020"
},
{
"name": "@narratage/media-pipeline",
"version": "0.0.0-dev",
- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"
+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"
},
{
"name": "@narratage/media",
"version": "0.0.0-dev",
- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"
+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"
},
{
"name": "@narratage/narrative",
@@ -150,7 +150,7 @@
{
"name": "@narratage/package-loader-node",
"version": "0.0.0-dev",
- "digest": "sha256:db78f8abd9a08a7655f7ed2d24860b1f848b3763421984847c303afa316b9003"
+ "digest": "sha256:64a8daed4ebf1c561be55e4f26d2439235cf7b7d8b8fec26f995e26489415675"
},
{
"name": "@narratage/program-space",
@@ -165,12 +165,12 @@
{
"name": "@narratage/provider-google-vertex",
"version": "0.0.0-dev",
- "digest": "sha256:e6f2594a0636b527fa7472c66fb3134eaa92d7e9903a8cebf0065fc7b4af4377"
+ "digest": "sha256:87cc370c56a3730dd94f9d2e74481d91db25a3a9ca0483b3077e66f69d5f3c10"
},
{
"name": "@narratage/provider-hyperframes-local",
"version": "0.0.0-dev",
- "digest": "sha256:1d3c090bc9d13fc266b65120f3e0439be824b1d05d4746a670aa929da49a560b"
+ "digest": "sha256:5e466e9bc267266768904c4bd873c2f98a5a9dffbc2dc67c545710e391a60242"
},
{
"name": "@narratage/provider-kie",
@@ -180,7 +180,7 @@
{
"name": "@narratage/provider-media-local",
"version": "0.0.0-dev",
- "digest": "sha256:9bfc485ddbdeca9cd747925b4ef04f85315e41746a2cf4a99ac0d96baf237014"
+ "digest": "sha256:4dcefebb3b6baf6436ba607ce434328a2d50eb88542a96f6220671715cddd9f1"
},
{
"name": "@narratage/provider-whisperx-local",
@@ -205,7 +205,7 @@
{
"name": "@narratage/runtime-adapter",
"version": "0.0.0-dev",
- "digest": "sha256:7bb0b1ef43eb3fa2e67725c990b17f585a2fccd98fc71eae6c7a8da29b3e06a4"
+ "digest": "sha256:b67dfc214ab9ae8cec350cee2e479aa70e7c62bf19a01d7c1d091d6e7df7c913"
},
{
"name": "@narratage/runtime",
@@ -215,7 +215,7 @@
{
"name": "@narratage/semantic-map",
"version": "0.0.0-dev",
- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"
+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"
},
{
"name": "@narratage/source",
@@ -225,12 +225,12 @@
{
"name": "@narratage/spatial",
"version": "0.0.0-dev",
- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"
+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"
},
{
"name": "@narratage/speech-alignment",
"version": "0.0.0-dev",
- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"
+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"
},
{
"name": "@narratage/speech-evidence",
@@ -240,7 +240,7 @@
{
"name": "@narratage/speech",
"version": "0.0.0-dev",
- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"
+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"
},
{
"name": "@narratage/store-sqlite",
@@ -270,7 +270,7 @@
{
"name": "@narratage/whisperx",
"version": "0.0.0-dev",
- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"
+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"
},
{
"name": "@narratage/workspace-fs-node",
@@ -926,7 +926,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:914018276b9218afc13940ca5c362dc95fa19d574fc07b06b6fe005c5eb30315",
- "closureDigest": "sha256:fadbd7bdb4fb5af49e762cb92c1e22d0cf3e2bf10fd5f0de3d594ef3f0ca8d93"
+ "closureDigest": "sha256:466bbc1b42c06f020dfe64ebd701f037771f0c61a37ce5614cdff03d4bcf0943"
},
{
"specifier": "@narratage/credential-store-env",
@@ -935,7 +935,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:ee33ba638bbf2f88eee8f34a2f3179f435a0af9ccbae3fbb7d2a81bd6aeac612",
- "closureDigest": "sha256:d416ef4d798b0948dcc2cd16c962784297b6e2f20daf8626f28c5d90e37b1dec"
+ "closureDigest": "sha256:7521e30257c61ecaae22198b985900b135e6348172bf1a699eae06c57782458c"
},
{
"specifier": "@narratage/local",
@@ -944,7 +944,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:6689aed79b0947c7315aef65e46806ce73da01de3d73b454e1497b8d847ffa61",
- "closureDigest": "sha256:2e6704a8d53b79c750c1dc1ada0196737e13cb1e41bec5fac233abbd62b70326"
+ "closureDigest": "sha256:e0730ba07fbef51517fb77ea985b83d12a42eedde998852e29db5cdb199a55c2"
},
{
"specifier": "@narratage/provider-google-vertex",
@@ -953,7 +953,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:b2636492ab0b0b500d674899ab2ff14fc61c7f2b22ef44eb8f816f141b21ec4d",
- "closureDigest": "sha256:fdfd1b6d25ae82b3eca28bc7f1e94ca2a4c2930f6171388afb861ca9a06990da"
+ "closureDigest": "sha256:4bcb114a29f4bd429dbcea883b7a8ee0f178a30c3488d261f2485214f953acd8"
},
{
"specifier": "@narratage/provider-hyperframes-local",
@@ -962,7 +962,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:b14b721bdc0c9d5d9ced1c345107c92a3a5878f3383dea2ecd3e310442c2a40c",
- "closureDigest": "sha256:22819933ee6b109e794ca94cee669cddf3288c8fe5c3d4e7e2bd727fdeab8bda"
+ "closureDigest": "sha256:0eb5def07621f9a75934160af6372d8c97087dd6e8434e40cb8ad4497de38218"
},
{
"specifier": "@narratage/provider-kie",
@@ -971,7 +971,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:e61ebb911a33d4c5d0e4e4ee4fb69cdb790f4ced050ef35f7472bfbd72013d9b",
- "closureDigest": "sha256:90589aebf0631d19b59f258bd1e4c20638f4325f946260aa71968350e9e4efc0"
+ "closureDigest": "sha256:94ed9baec91ef62d316be9296803aa26d523164cddbdcb48b92b8a743ab04a09"
},
{
"specifier": "@narratage/provider-media-local",
@@ -980,7 +980,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:701abcc923bf8a711c20b6e308d8cc8be27bf560f249b8c5a801e1dbd9cb3979",
- "closureDigest": "sha256:f40546545b842197e77982ceba94c7f52f2389bad33d68ff98ad62e28eb576e1"
+ "closureDigest": "sha256:d59a831fdeea751342aba28e48465d2f62b13d436dafe84bd47529aa2a7542f0"
},
{
"specifier": "@narratage/provider-whisperx-local",
@@ -989,7 +989,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:3686b84c42cc2b1bcaccf57cbdcda25a15c7d9d3778fb92d25437f9880ed8f5d",
- "closureDigest": "sha256:8c03021a0c809fbc076cd0c09c797151ec6cb48d473453f48993a8807ceb216f"
+ "closureDigest": "sha256:2125f3b538132060328c2435e6bda42f86e5356edef9705c954f32d162ab268f"
},
{
"specifier": "@narratage/store-sqlite",
@@ -998,8 +998,8 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:5bccf73cd953710fc31ef44672fef1bfc396fd4aa2eff4b53108e8ff87e749c3",
- "closureDigest": "sha256:d512ec9a51a70ce2c020a4a1cb4d1f67b66b484615c6212a4a85918983415718"
+ "closureDigest": "sha256:84f9770cc535fb1e01a038e6f626bb5d32beb83d8b2169dc0c02b914334ec4c4"
}
],
- "digest": "sha256:2ef941c08a6fde4e0fc9b9527358186402e5bffbc262b1c35e98a6678b3f9b99"
+ "digest": "sha256:f1d098a85cfd67ba365732a7635f15e45c72ea45e2eb7807f7afcac914cc2faf"
}
diff --git a/examples/talking-head-aroll/main.svml b/examples/talking-head-aroll/main.svml
index 90a76028..f3465ca4 100644
--- a/examples/talking-head-aroll/main.svml
+++ b/examples/talking-head-aroll/main.svml
@@ -152,9 +152,11 @@
</seedance:ReferenceVideo>
<space:Canvas id="vertical" width="720" height="1280"/>
+ <space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>
<!-- Four equal Takes become one ordered speech coordinate space and two peer Tracks. -->
- <speech:Spine id="speech" canvas={vertical} frame-rate="30">
+ <speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={studio.speech.visual} visual-z="0">
<speech:Take video={hook-take.video} segment={story.segment.hook}/>
<speech:Take video={meeting-take.video} segment={story.segment.meeting}/>
<speech:Take video={evidence-take.video} segment={story.segment.evidence}/>
diff --git a/examples/talking-head-aroll/studio.svs b/examples/talking-head-aroll/studio.svs
index c10c3e58..7efbf882 100644
--- a/examples/talking-head-aroll/studio.svs
+++ b/examples/talking-head-aroll/studio.svs
@@ -21,6 +21,8 @@
background: #09090B;
}
+ speech.visual { fit: cover; }
+
caption.primary {
cue-min-words: 2;
cue-max-words: 7;
diff --git a/examples/talking-head-aroll/svml.packages.lock b/examples/talking-head-aroll/svml.packages.lock
index 8c11d380..90151134 100644
--- a/examples/talking-head-aroll/svml.packages.lock
+++ b/examples/talking-head-aroll/svml.packages.lock
@@ -595,7 +595,7 @@
{
"name": "@narratage/caption",
"version": "0.0.0-dev",
- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"
+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"
},
{
"name": "@narratage/component-kit",
@@ -615,7 +615,7 @@
{
"name": "@narratage/elaborator",
"version": "0.0.0-dev",
- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"
+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"
},
{
"name": "@narratage/estimate",
@@ -655,17 +655,17 @@
{
"name": "@narratage/media-pipeline",
"version": "0.0.0-dev",
- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"
+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"
},
{
"name": "@narratage/media-track",
"version": "0.0.0-dev",
- "digest": "sha256:b99121eb2f0f5307c19115a00af5fc7b218440c87e4b4caeded9d4570d6547d0"
+ "digest": "sha256:08af7b82b3f4182fba2f3cc4cd12aee34452c2c2cf0cdbf2c3da48422a5b5d5e"
},
{
"name": "@narratage/media",
"version": "0.0.0-dev",
- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"
+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"
},
{
"name": "@narratage/model-kit",
@@ -715,7 +715,7 @@
{
"name": "@narratage/semantic-map",
"version": "0.0.0-dev",
- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"
+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"
},
{
"name": "@narratage/source",
@@ -725,17 +725,17 @@
{
"name": "@narratage/spatial",
"version": "0.0.0-dev",
- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"
+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"
},
{
"name": "@narratage/speech-alignment",
"version": "0.0.0-dev",
- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"
+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"
},
{
"name": "@narratage/speech-basis",
"version": "0.0.0-dev",
- "digest": "sha256:01532b3694db77f9d090b6474440144bbf1245682befa00a4cd44e77e2395b0a"
+ "digest": "sha256:3e6952237c7bc44fb85c873fae2850999fbb77f90e397a0f9441046aa64ad9d7"
},
{
"name": "@narratage/speech-evidence",
@@ -745,12 +745,12 @@
{
"name": "@narratage/speech-spine",
"version": "0.0.0-dev",
- "digest": "sha256:84898416ea156ceefd5c090e18b560355741f95451cbf90e99e8f6a639986da1"
+ "digest": "sha256:03a18dbf36764e3cd951fb46eb8badb69ead040da5f7b6d258a5bb125950f0a1"
},
{
"name": "@narratage/speech",
"version": "0.0.0-dev",
- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"
+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"
},
{
"name": "@narratage/svs",
@@ -760,7 +760,7 @@
{
"name": "@narratage/temporal",
"version": "0.0.0-dev",
- "digest": "sha256:b20cb315f5fc365a2e9994c63989a0cc1edc46b5843e06a337827d21d74068a1"
+ "digest": "sha256:6af6c0bf4e2cdb23240125815dddd831dcacfe793055150c58f4dfd37ce9e350"
},
{
"name": "@narratage/text",
@@ -780,7 +780,7 @@
{
"name": "@narratage/whisperx",
"version": "0.0.0-dev",
- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"
+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"
},
{
"name": "cmu-pronouncing-dictionary",
@@ -804,8 +804,8 @@
"name": "@narratage/caption",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:6a2991ec75f9fd30e8d286213fc7993aae2c570b85a8faae5d3c2af661c9fdfc",
- "closureDigest": "sha256:ee79cd66af6e561432e52f1bf78f597f9dc01cd09d12d68c26e7cf61e7361aac"
+ "facetsDigest": "sha256:b8434ae33703de6d3c41c1426ae76083b19e398e14efe8589f2c01a92a9d4e8d",
+ "closureDigest": "sha256:acd4bf7e979294b1b69fbab08a67b627be720c7e90226d94460490e76a9b4b07"
},
{
"specifier": "@narratage/caption-fine",
@@ -813,8 +813,8 @@
"name": "@narratage/caption-fine",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:c5616408dd56ce9c903523319b2a697ff6b8af5bfe1dc66fe793a11e4629f3cf",
- "closureDigest": "sha256:782a9f2c0345431e06c8ab009d518933c964465b34d790b4064f87170e0495ff"
+ "facetsDigest": "sha256:2fad0cbda2ae17fe67f5816e99f493d6ebfce232e2f526c5c9299a35ff0429f9",
+ "closureDigest": "sha256:0aa2796ab56a70cb4e7db6fab3394cdb2d119397f3144f0f1f18cc59f30af29b"
},
{
"specifier": "@narratage/caption-gemini",
@@ -822,8 +822,8 @@
"name": "@narratage/caption-gemini",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:22fed1f1748349c11b6f28448be38cbea68b1dd5a51a3dc2d548ae0a376b9c71",
- "closureDigest": "sha256:3cc6fd7d5250e5457db548fa2432f33d62012db8f77e1c8fe4f590e38a3439ab"
+ "facetsDigest": "sha256:d4f1cf46a646ff5cf6294307af7f77a453a000eaf2b168099d42f41d1fbbcf50",
+ "closureDigest": "sha256:4f6bddb12ebf1b61a1297d09ca1ab4080bab20523487e73f66d53be2a28f99e4"
},
{
"specifier": "@narratage/composition",
@@ -831,8 +831,8 @@
"name": "@narratage/composition",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:0a988b665baf0846246e309d9faf39ffb3ef6ec455bf37f7c438eb30429ff46e",
- "closureDigest": "sha256:15c9088a176b04a6051b2398e308bde8fab6fb15d0147786e9e40abfce772662"
+ "facetsDigest": "sha256:d0c91a61caa7303210d898183ae355ab6c5e0eb1de8bd5a0c237cd4388693104",
+ "closureDigest": "sha256:937caaaab137467eaaef1e9974b103498a76d0b915690ffe404d800b5f84b706"
},
{
"specifier": "@narratage/estimate",
@@ -840,8 +840,8 @@
"name": "@narratage/estimate",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:9975815b3109520eb6bba276554a90767a9aebff6498c2768d9faf4763219bef",
- "closureDigest": "sha256:7bbdc658746e65e2bb4199b3de84d1197376f16120708e2eea31558898de662f"
+ "facetsDigest": "sha256:018228974baa7623415df07f4703025f2d09449969439724027836a13e61caf1",
+ "closureDigest": "sha256:9424d9a12d4fcb0b7886c82aabed10a21f6707b3e42ded94f960396818fa8412"
},
{
"specifier": "@narratage/film",
@@ -849,8 +849,8 @@
"name": "@narratage/film",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:8e58b5630b9a68a8ce7314620f5f1c7cda973a05c179c18d61e10d68002964f0",
- "closureDigest": "sha256:2a4b2b66d9f3913bc839f052e35e6c5cd6afed08821a3308c643b16dcd011f44"
+ "facetsDigest": "sha256:f6e47bdf84726556e6ee174da07b7dd435b6e7b9865065f7d7cb29ec0ed5b166",
+ "closureDigest": "sha256:cfb335a738929c1153350cdc14f7a579679566a558232b0786945a41cf18eaa5"
},
{
"specifier": "@narratage/fonts-open",
@@ -858,8 +858,8 @@
"name": "@narratage/fonts-open",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:31b51352edcb4f34beccc9d9ac8cf7397125a840971a72c912df3d34d1f3feda",
- "closureDigest": "sha256:4594c947a3425c6c42738e2e7d0f7ce56a6efaced91fcf6085e282c3ad98b797"
+ "facetsDigest": "sha256:2117122b06d7c1cd6bee519e72aff22bc3b50f179b50da21dcc84ddf7014457f",
+ "closureDigest": "sha256:65fe769fd8fe5de6a8ef1f33a63bd843ce94fc63dd9bacafecef862b2fadbc25"
},
{
"specifier": "@narratage/generation",
@@ -868,7 +868,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:d74a57a3210267e665366513324db5c313b3ca0f770da66e74cc38aee5048efb",
- "closureDigest": "sha256:3bd4bfdd124001b83602627b0305d6652e96f43cbb9345e3b9bece51a3d701ce"
+ "closureDigest": "sha256:dd00aba4da27ff3bfcec6e5bc24d86a07ac94344701c9936972b66ab5fc64628"
},
{
"specifier": "@narratage/hyperframes",
@@ -876,8 +876,8 @@
"name": "@narratage/hyperframes",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:b8b8773d03f19844cdc8d714650380145056649245e33c607f30220fff194157",
- "closureDigest": "sha256:88d7a9fedb614166e149e29d064be1dca18f1c4a102759f1a2600afba4953b8b"
+ "facetsDigest": "sha256:520234d0b029d7229e6595483e2f0d0227d20cf662363163975691e2471a767e",
+ "closureDigest": "sha256:118b9b787b8df4bea1fbf4cfc09d65627d7670f96db1984a1ae6f0c649e5d43b"
},
{
"specifier": "@narratage/media",
@@ -885,8 +885,8 @@
"name": "@narratage/media",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:6c7cf563a92aa8e48db6da224697fd5f56c50b636b0327efc17fbbc264502b3f",
- "closureDigest": "sha256:e8edbbfb7bec2efa7ad05fa273549a3b033861eb9ef20345430d9be617b26674"
+ "facetsDigest": "sha256:f0974c7d88918ca4796dbb86dea24b6d29f8f5f44430c4845f9928f1df5b22e5",
+ "closureDigest": "sha256:007255553cdbe2c9b82bd71ab920faaae9d107e54e6548adc5e26853505d9146"
},
{
"specifier": "@narratage/media-pipeline",
@@ -894,8 +894,8 @@
"name": "@narratage/media-pipeline",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:21f817f8e6eea9e9e4361005ed5ab9e69117fd77ec237fd467fb11afe9965019",
- "closureDigest": "sha256:3555f1939c82dfa216902fd600e2616ecfce6a244e240b060080621367333d97"
+ "facetsDigest": "sha256:e83eed5903110d1e8fa38aeac1487f63737f1e2a773f6e3c794dff54ad66122a",
+ "closureDigest": "sha256:fde796ee9bd9d9992fe2cb37d3cf18f539d74f16e55e847a586ad013d7f87705"
},
{
"specifier": "@narratage/narrative",
@@ -921,8 +921,8 @@
"name": "@narratage/render-hyperframes",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:aa47f04cf0652ee3fa2e114eb6d9a80b0522828a47869103ca33e95027b21634",
- "closureDigest": "sha256:1c9dcb543a3afa6a94c9757fd47a3e0bfb68ed7d5f3c87b3ef0f0e52ed286332"
+ "facetsDigest": "sha256:b5a0bad5b8809158caea96274f94c397e18086febcd8b11eb3f4360bb869f06e",
+ "closureDigest": "sha256:67e343cebb44e6d3cfe929356e5363b317f9462e7240fc411c5bb0df87bbf17f"
},
{
"specifier": "@narratage/run-markup",
@@ -931,7 +931,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:fb3a8a5be9a0ef0aa914026a199be641f4e20df27f9b284be196f52fd260be55",
- "closureDigest": "sha256:20b6fcbacf36352d6c180e1a466eaf368c0bef183eec09396456e2e2375ea16f"
+ "closureDigest": "sha256:b3d5afceb67cb1275867e99f0aa384f4d339d2b759443d6822352bc8714b8bc7"
},
{
"specifier": "@narratage/script",
@@ -940,7 +940,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:69e1c08a5f07c73a506ca8286426a0386485738649e86463cca0b291f319ee4e",
- "closureDigest": "sha256:696bb2e69c89d9e1079a76ce68d26fe2c99615275a4266bf24fa905ad77c1bb4"
+ "closureDigest": "sha256:cc5ebec9ea7bf21d23d67536dac321618679037e811042d2b349688a373ffc4d"
},
{
"specifier": "@narratage/seedance",
@@ -948,8 +948,8 @@
"name": "@narratage/seedance",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:e889ca3808a19f544e7aa5e27453beb5a09464c645633b0b1b1d9a4b7ce6591a",
- "closureDigest": "sha256:e1c45192a92620ec8ea03d9cf1b19073bae2dbfde12475f8436cd5c008c854bd"
+ "facetsDigest": "sha256:0832f344b8c327e316a153d8ca4559da5eba452f3e334ffb49c7e94a29f36ecf",
+ "closureDigest": "sha256:3ef4370d01ffcfef671cae12e1a250b1216caaa373b335246d2bfb5d11b148cf"
},
{
"specifier": "@narratage/semantic-map",
@@ -957,8 +957,8 @@
"name": "@narratage/semantic-map",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:a9d81ddfa09dfa23c9c3c0817b60299431ca9ec640252a659d490be55d92ba68",
- "closureDigest": "sha256:3ac5deb912231d3092666c692d73a379129d1cf2ed035d43a95bef6d5101d525"
+ "facetsDigest": "sha256:836a539f60d3a77a122d0bae169f03e229566dfc4696001143e9bd32c313dc12",
+ "closureDigest": "sha256:a6313c3c407510323d7b5f856aaae3935f47cd80869b099b961aa34c9eb6c1ba"
},
{
"specifier": "@narratage/spatial",
@@ -967,7 +967,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:5062281ffadb46495ec12e1a67185b57fe13879e90f03bcf544e93fd1d273716",
- "closureDigest": "sha256:3209c05b608c20aedab35299235974d74ee4374ead722ec125f2b6ea70829c77"
+ "closureDigest": "sha256:e328230a8de70bd5c4ec4af65db9f964f0ffe20900358ba4a59fff0d29dd8ac7"
},
{
"specifier": "@narratage/speech",
@@ -975,8 +975,8 @@
"name": "@narratage/speech",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:75b1537c9b9729c2e9197445c34b295bc9f72dbef754a4702e1ca792dceea472",
- "closureDigest": "sha256:a5be05750141d51e4f4460945b779bf0196548d7b7c908eca34d8c63c2ec7b70"
+ "facetsDigest": "sha256:ed94af4fb62357b30f8faec7f16ccb2b2d055deecd300c7ca948b112b56b710c",
+ "closureDigest": "sha256:cae66d28f2f5598824877a845f996798a35b17c84043dc261678ead81c602e86"
},
{
"specifier": "@narratage/speech-alignment",
@@ -984,8 +984,8 @@
"name": "@narratage/speech-alignment",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:bdfe219081310041d11c6d54440e6ccb80398bfb97f93afc28e9d1ed101b4a79",
- "closureDigest": "sha256:de73bc9d1411af8319a099df658dfc51f6e7b44dcbf3173cb5f9db18edb449cf"
+ "facetsDigest": "sha256:fe041c994c1974eddf582703869f1b8718eab817c1a0eb5e2edd0ae98644f8d9",
+ "closureDigest": "sha256:8d611e7523965c7a446d95beead34533db5eb94c6ed7666d14381cf9c77629fc"
},
{
"specifier": "@narratage/speech-basis",
@@ -993,8 +993,8 @@
"name": "@narratage/speech-basis",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:f01f21badb0ccdd5cc62b131be05f46030bffee96042c7323a0c572786d012ef",
- "closureDigest": "sha256:843a7922a70d5df180ade4ec3a8becf8fb110f6137fa6a38d86cc5aaa14830e1"
+ "facetsDigest": "sha256:5badee4556d51ff8075eb2a4531b5bf9ff65069ce9cce724b870b21e135c2129",
+ "closureDigest": "sha256:151aff0a543fef15b471e6914324594b333f68f081e9835c29bc52826be2c825"
},
{
"specifier": "@narratage/speech-evidence",
@@ -1002,8 +1002,8 @@
"name": "@narratage/speech-evidence",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:60f366de074a7fba20c71dcbd825083fddc54ea7de602a903eb144eab5555425",
- "closureDigest": "sha256:918172938ecc1e9b6b1cff9465048bdfe2dcbfe8748971101f78f12ee36f1d75"
+ "facetsDigest": "sha256:f6dfc644f0dd9975af48081fb4b1b7ac23154c4bccd2aeceff19cee05697b757",
+ "closureDigest": "sha256:62f0174c12061f87acd28f1aa0d816cb1ff4c80956040737ad55f1d4a3bf4838"
},
{
"specifier": "@narratage/speech-spine",
@@ -1011,8 +1011,8 @@
"name": "@narratage/speech-spine",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:a27dd4349d6bb1aaa7f382ad09f883e9894157579e48285b00e8fa24456190f7",
- "closureDigest": "sha256:f056e14a44f4f2cc0751b28e1ecb9831b7ff472034a02cac8089b9ff45d67854"
+ "facetsDigest": "sha256:d20cdc013e61ea8d7a632d7cea2c10a9186a0bf0736ece2ff0f307c1320b6459",
+ "closureDigest": "sha256:16eb190b79628257a3100ca81b7cbe26ccb1e4d6e95b6f93c629dafac07a2c8d"
},
{
"specifier": "@narratage/svs",
@@ -1021,7 +1021,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:9cc7c397d2cd97b5a3e814886b6553b6d236fd8825df0d056e328b7f0cf20163",
- "closureDigest": "sha256:2be69621b66211f48f0f7181bda377dd9e6f9a90067e1d0640fa2d762a0f19d1"
+ "closureDigest": "sha256:9358fe74dcc74e0bd730d9043292e044843e2e9eb003ee9db24e73f81ac495df"
},
{
"specifier": "@narratage/text",
@@ -1030,7 +1030,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:4435dd4bd38e5e017013f775e28e379e3e235211649ee8f3913378f7f9e60b58",
- "closureDigest": "sha256:c6ba3301621d63598421b7d3f0dc2e1bcc8967f16624407d37a20c2c0703a246"
+ "closureDigest": "sha256:9cab1e68a2273b7cf749c06033d63c2bdc28690b55fc0c755921512f0c3a701f"
},
{
"specifier": "@narratage/visual-ir",
@@ -1047,9 +1047,9 @@
"name": "@narratage/whisperx",
"version": "0.0.0-dev"
},
- "facetsDigest": "sha256:c8cb487d25e567a081c74a7abc97ce2b94dc0c47c36c90ba6b44d2ec21a9e80b",
- "closureDigest": "sha256:98e68de949bf7cfb9e1c483de66a38de6f347fc5032122d10e9af971a28fa534"
+ "facetsDigest": "sha256:9e765c9776d31ceeb1f0898fa3df9d7345747fa0fd5a60d46728f8d78c8a69df",
+ "closureDigest": "sha256:d0abf29a19cbc43aa0e0cfe239670d4e339da684f89ed5ad413b1aabea18e165"
}
],
- "digest": "sha256:828557259e164093a055111cff451aaf48d1d8519cc02352804066c5841347ee"
+ "digest": "sha256:c88ce3ec0a540451c5be1d4d1357d2a7ca82b036c6922c6455a55a908a8009c6"
}
diff --git a/examples/talking-head-aroll/svml.runtime-packages.lock b/examples/talking-head-aroll/svml.runtime-packages.lock
index ef94a1d9..a27d165c 100644
--- a/examples/talking-head-aroll/svml.runtime-packages.lock
+++ b/examples/talking-head-aroll/svml.runtime-packages.lock
@@ -60,12 +60,12 @@
{
"name": "@narratage/caption",
"version": "0.0.0-dev",
- "digest": "sha256:d365d74d385b56d25bbe9fa08e0d70a58dafaef3cc51683a43821d9cf8da08de"
+ "digest": "sha256:5e98428e8db5970a8a699209f98aa1d108225132f3ad99aba0daf0cea6dbbb77"
},
{
"name": "@narratage/compiler-node",
"version": "0.0.0-dev",
- "digest": "sha256:4626882f405a083281ce7ffec73fa598640223669908b36715d8281fa19be92e"
+ "digest": "sha256:804eac87761fe0465234647c2309b6396ed4fd0a886346afef9167d327c3bb9f"
},
{
"name": "@narratage/component-kit",
@@ -95,7 +95,7 @@
{
"name": "@narratage/elaborator",
"version": "0.0.0-dev",
- "digest": "sha256:d433a2298b3afcf12d41575a9f0d3e7fd6a3feec297ba66466cc9d98b5aa8f1f"
+ "digest": "sha256:16ccb3999473a950aded72e9563bddeec455b2da05d620f063875a7577be2558"
},
{
"name": "@narratage/endpoint-kit",
@@ -120,7 +120,7 @@
{
"name": "@narratage/local",
"version": "0.0.0-dev",
- "digest": "sha256:cfbd3c13a81dbc21378bebd22662e93a201480e1bfe725f04e78534764f86c0e"
+ "digest": "sha256:d2bece5a791dc3b7a88f890a54142b5e0cc5a534485ff4e20d5740787a8b90a1"
},
{
"name": "@narratage/markup",
@@ -130,17 +130,17 @@
{
"name": "@narratage/media-execution",
"version": "0.0.0-dev",
- "digest": "sha256:8fe6aca65ca4ebcc1efa71376e986e97741310d836f85647fcb223f39fda9754"
+ "digest": "sha256:3f07960e6723ad386b7fd8858438f2f9e72f6bab23120ffb10462944ac896020"
},
{
"name": "@narratage/media-pipeline",
"version": "0.0.0-dev",
- "digest": "sha256:5b11d105aa3e51cf8bd5edf30c2e4ff885b1d4b337d8c2cb401abd7b51d92c0d"
+ "digest": "sha256:40d13cafc0a5a9c768dedb8540ee92dbde47b3762104f7f515a93ba0da0814c4"
},
{
"name": "@narratage/media",
"version": "0.0.0-dev",
- "digest": "sha256:3a6915c8344e579ea32ce0d93221485dbf35eda882de800246a41325ed31923d"
+ "digest": "sha256:ba242b944bfea8ef85d25b194749f3d09b887697515abc67d76805b140052614"
},
{
"name": "@narratage/narrative",
@@ -150,7 +150,7 @@
{
"name": "@narratage/package-loader-node",
"version": "0.0.0-dev",
- "digest": "sha256:db78f8abd9a08a7655f7ed2d24860b1f848b3763421984847c303afa316b9003"
+ "digest": "sha256:64a8daed4ebf1c561be55e4f26d2439235cf7b7d8b8fec26f995e26489415675"
},
{
"name": "@narratage/program-space",
@@ -165,12 +165,12 @@
{
"name": "@narratage/provider-google-vertex",
"version": "0.0.0-dev",
- "digest": "sha256:e6f2594a0636b527fa7472c66fb3134eaa92d7e9903a8cebf0065fc7b4af4377"
+ "digest": "sha256:87cc370c56a3730dd94f9d2e74481d91db25a3a9ca0483b3077e66f69d5f3c10"
},
{
"name": "@narratage/provider-hyperframes-local",
"version": "0.0.0-dev",
- "digest": "sha256:1d3c090bc9d13fc266b65120f3e0439be824b1d05d4746a670aa929da49a560b"
+ "digest": "sha256:5e466e9bc267266768904c4bd873c2f98a5a9dffbc2dc67c545710e391a60242"
},
{
"name": "@narratage/provider-kie",
@@ -180,7 +180,7 @@
{
"name": "@narratage/provider-media-local",
"version": "0.0.0-dev",
- "digest": "sha256:9bfc485ddbdeca9cd747925b4ef04f85315e41746a2cf4a99ac0d96baf237014"
+ "digest": "sha256:4dcefebb3b6baf6436ba607ce434328a2d50eb88542a96f6220671715cddd9f1"
},
{
"name": "@narratage/provider-whisperx-local",
@@ -205,7 +205,7 @@
{
"name": "@narratage/runtime-adapter",
"version": "0.0.0-dev",
- "digest": "sha256:7bb0b1ef43eb3fa2e67725c990b17f585a2fccd98fc71eae6c7a8da29b3e06a4"
+ "digest": "sha256:b67dfc214ab9ae8cec350cee2e479aa70e7c62bf19a01d7c1d091d6e7df7c913"
},
{
"name": "@narratage/runtime",
@@ -215,7 +215,7 @@
{
"name": "@narratage/semantic-map",
"version": "0.0.0-dev",
- "digest": "sha256:2edb5924fb765570a4404654844646b308d819bb8c4580689513130d78c36b78"
+ "digest": "sha256:4833471d964249e4b045dd1341ac129f1de7ef0f273453f21db0d186f542dcbf"
},
{
"name": "@narratage/source",
@@ -225,12 +225,12 @@
{
"name": "@narratage/spatial",
"version": "0.0.0-dev",
- "digest": "sha256:4b41917023c9a9fc9b1e4a3a2c6d82fafeb3d732733c6e4ca6721ae2009edbdf"
+ "digest": "sha256:c4bdffd24369c3062f73df2f3db13b0bb69162841cdb19030c2df3c76f03e470"
},
{
"name": "@narratage/speech-alignment",
"version": "0.0.0-dev",
- "digest": "sha256:e62baaf3a77d41cdd4f245169f48b4311f6a02d232625cdbc44523455d7d7c17"
+ "digest": "sha256:92aa160a554a9a9284730e10c2938fcefd6b09c6f8f47f00cd75edb5cf7a8144"
},
{
"name": "@narratage/speech-evidence",
@@ -240,7 +240,7 @@
{
"name": "@narratage/speech",
"version": "0.0.0-dev",
- "digest": "sha256:ad66df5763802600b43d5df3538e5dc43bfe52cb7f1dabe63497e79834cfbe09"
+ "digest": "sha256:e4836c9ae05726fe80ec549a3553675835f40bdca304bec039111a2fda53afec"
},
{
"name": "@narratage/store-sqlite",
@@ -270,7 +270,7 @@
{
"name": "@narratage/whisperx",
"version": "0.0.0-dev",
- "digest": "sha256:7ee7f559fdc0508b6a7447838eb0a121cbe60c03522c6969638e0e8d3c04068f"
+ "digest": "sha256:b580c93b5850fe24e33af2bdf983af75381cd35aa913cf68bdc3b41f712a7198"
},
{
"name": "@narratage/workspace-fs-node",
@@ -926,7 +926,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:914018276b9218afc13940ca5c362dc95fa19d574fc07b06b6fe005c5eb30315",
- "closureDigest": "sha256:fadbd7bdb4fb5af49e762cb92c1e22d0cf3e2bf10fd5f0de3d594ef3f0ca8d93"
+ "closureDigest": "sha256:466bbc1b42c06f020dfe64ebd701f037771f0c61a37ce5614cdff03d4bcf0943"
},
{
"specifier": "@narratage/credential-store-env",
@@ -935,7 +935,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:ee33ba638bbf2f88eee8f34a2f3179f435a0af9ccbae3fbb7d2a81bd6aeac612",
- "closureDigest": "sha256:d416ef4d798b0948dcc2cd16c962784297b6e2f20daf8626f28c5d90e37b1dec"
+ "closureDigest": "sha256:7521e30257c61ecaae22198b985900b135e6348172bf1a699eae06c57782458c"
},
{
"specifier": "@narratage/local",
@@ -944,7 +944,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:6689aed79b0947c7315aef65e46806ce73da01de3d73b454e1497b8d847ffa61",
- "closureDigest": "sha256:2e6704a8d53b79c750c1dc1ada0196737e13cb1e41bec5fac233abbd62b70326"
+ "closureDigest": "sha256:e0730ba07fbef51517fb77ea985b83d12a42eedde998852e29db5cdb199a55c2"
},
{
"specifier": "@narratage/provider-google-vertex",
@@ -953,7 +953,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:b2636492ab0b0b500d674899ab2ff14fc61c7f2b22ef44eb8f816f141b21ec4d",
- "closureDigest": "sha256:fdfd1b6d25ae82b3eca28bc7f1e94ca2a4c2930f6171388afb861ca9a06990da"
+ "closureDigest": "sha256:4bcb114a29f4bd429dbcea883b7a8ee0f178a30c3488d261f2485214f953acd8"
},
{
"specifier": "@narratage/provider-hyperframes-local",
@@ -962,7 +962,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:b14b721bdc0c9d5d9ced1c345107c92a3a5878f3383dea2ecd3e310442c2a40c",
- "closureDigest": "sha256:22819933ee6b109e794ca94cee669cddf3288c8fe5c3d4e7e2bd727fdeab8bda"
+ "closureDigest": "sha256:0eb5def07621f9a75934160af6372d8c97087dd6e8434e40cb8ad4497de38218"
},
{
"specifier": "@narratage/provider-kie",
@@ -971,7 +971,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:e61ebb911a33d4c5d0e4e4ee4fb69cdb790f4ced050ef35f7472bfbd72013d9b",
- "closureDigest": "sha256:90589aebf0631d19b59f258bd1e4c20638f4325f946260aa71968350e9e4efc0"
+ "closureDigest": "sha256:94ed9baec91ef62d316be9296803aa26d523164cddbdcb48b92b8a743ab04a09"
},
{
"specifier": "@narratage/provider-media-local",
@@ -980,7 +980,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:701abcc923bf8a711c20b6e308d8cc8be27bf560f249b8c5a801e1dbd9cb3979",
- "closureDigest": "sha256:f40546545b842197e77982ceba94c7f52f2389bad33d68ff98ad62e28eb576e1"
+ "closureDigest": "sha256:d59a831fdeea751342aba28e48465d2f62b13d436dafe84bd47529aa2a7542f0"
},
{
"specifier": "@narratage/provider-whisperx-local",
@@ -989,7 +989,7 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:3686b84c42cc2b1bcaccf57cbdcda25a15c7d9d3778fb92d25437f9880ed8f5d",
- "closureDigest": "sha256:8c03021a0c809fbc076cd0c09c797151ec6cb48d473453f48993a8807ceb216f"
+ "closureDigest": "sha256:2125f3b538132060328c2435e6bda42f86e5356edef9705c954f32d162ab268f"
},
{
"specifier": "@narratage/store-sqlite",
@@ -998,8 +998,8 @@
"version": "0.0.0-dev"
},
"facetsDigest": "sha256:5bccf73cd953710fc31ef44672fef1bfc396fd4aa2eff4b53108e8ff87e749c3",
- "closureDigest": "sha256:d512ec9a51a70ce2c020a4a1cb4d1f67b66b484615c6212a4a85918983415718"
+ "closureDigest": "sha256:84f9770cc535fb1e01a038e6f626bb5d32beb83d8b2169dc0c02b914334ec4c4"
}
],
- "digest": "sha256:2ef941c08a6fde4e0fc9b9527358186402e5bffbc262b1c35e98a6678b3f9b99"
+ "digest": "sha256:f1d098a85cfd67ba365732a7635f15e45c72ea45e2eb7807f7afcac914cc2faf"
}
diff --git a/package.json b/package.json
index e75e28c5..902a0286 100644
--- a/package.json
+++ b/package.json
@@ -9,7 +9,8 @@
"docs:dev": "vitepress dev docs",
"docs:build": "vitepress build docs",
"docs:preview": "vitepress preview docs",
- "test": "node --import tsx --test 'packages/*/test/**/*.test.ts' 'services/*/test/**/*.test.ts' 'test/**/*.test.ts' 'tools/*.test.mjs'",
+ "test": "node --import tsx --test 'packages/*/test/**/*.test.ts' 'services/*/test/**/*.test.ts' 'test/**/*.test.ts'",
+ "test:release": "node --import tsx --test tools/repository-hygiene.test.mjs tools/recipe-defaults.test.mjs",
"test:whisperx-service": "uv run --project services/whisperx --frozen python -m unittest discover -s services/whisperx/tests -p 'test_*.py'",
"test:image-opencv": "SVML_OPENCV_TESTS=1 SVML_OPENCV_PYTHON=services/image-opencv/.venv/bin/python node --import tsx --test packages/provider-image-opencv-local/test/provider.test.ts",
"smoke:kie": "node --import tsx packages/provider-kie/test/live-smoke.ts",
diff --git a/packages/audio-track/src/program.ts b/packages/audio-track/src/program.ts
index 85e63664..0b914d0a 100644
--- a/packages/audio-track/src/program.ts
+++ b/packages/audio-track/src/program.ts
@@ -1,6 +1,6 @@
import { assertAudioTrackIdentity, sealAudioTrack } from "@narratage/composition";
import type { AudioClip, AudioTrack } from "@narratage/composition";
-import { verifySynchronizedMedia } from "@narratage/media";
+import { synchronizedMediaSampleFrames, verifySynchronizedMedia } from "@narratage/media";
import type { SynchronizedMedia } from "@narratage/media";
import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
import {
@@ -114,10 +114,9 @@ export function assertAudioTrackSet(value: AudioTrackSet): void {
function sourceFacts(media: SynchronizedMedia): AudioItemProgram["source"] {
verifySynchronizedMedia(media);
assert(media.audio !== undefined, "Audio Track source has no explicitly normalized audio member.");
- const { artifact, sampleFrames } = media.audio;
return {
- artifact: structuredClone(artifact),
- sampleFrames,
+ artifact: structuredClone(media.audio.artifact),
+ sampleFrames: synchronizedMediaSampleFrames(media),
};
}
diff --git a/packages/audio-track/test/audio-track.test.ts b/packages/audio-track/test/audio-track.test.ts
index b837516a..95db5533 100644
--- a/packages/audio-track/test/audio-track.test.ts
+++ b/packages/audio-track/test/audio-track.test.ts
@@ -53,29 +53,11 @@ function media(id: string, sampleFrames: number): SynchronizedMedia {
return {
contract: "svml.synchronized-media@1",
timeline: {
- spanAuthority: "audio",
frameRate: { numerator: 30, denominator: 1 },
frameCount: Math.max(1, Math.round(sampleFrames / 1_600)),
- sampleRate: 48_000,
- sampleFrames,
- },
- sourceMap: {
- sourceOriginPts: { ticks: "0", timeBase: { numerator: 1, denominator: 48_000 } },
- sourceEndPts: { ticks: String(sampleFrames), timeBase: { numerator: 1, denominator: 48_000 } },
- audioTrimStartSamples: 0,
- audioTrimEndSamples: 0,
- audioHeadSamples: 0,
- audioContentSamples: sampleFrames,
- audioTailSamples: 0,
},
audio: {
artifact: { kind: "blob", digest: digestOf(`audio:${id}`), size: sampleFrames * 4, mediaType: "audio/wav" },
- sourceStreamIndex: 0,
- codec: "pcm_s16le",
- sampleRate: 48_000,
- channels: 2,
- sampleFrames,
- loudness: "preserved",
},
};
}
diff --git a/packages/caption/test/caption.test.ts b/packages/caption/test/caption.test.ts
index 10da5469..587997c6 100644
--- a/packages/caption/test/caption.test.ts
+++ b/packages/caption/test/caption.test.ts
@@ -45,13 +45,7 @@ function locate(narrative: Narrative, durationSec: number, segments: readonly Al
contract: "svml.speech-basis@1",
programSpace: space,
audio,
- visualTrack: { clips: basisSegments.map((segment) => ({
- segmentId: segment.segmentId,
- artifact: { kind: "blob", digest: digestOf(segment.segmentId), size: 1, mediaType: "video/mp4" },
- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },
- frameRate: { ...space.frameRate },
- frameCount: Math.round((segment.endSec - segment.startSec) * 30),
- })) },
+ visualTrack: { clips: [] },
segments: basisSegments,
});
const evidence = sealAlignedTranscriptEvidence({ contract: "svml.aligned-transcript-evidence@1", durationSec, segments });
diff --git a/packages/elaborator/test/fragment.test.ts b/packages/elaborator/test/fragment.test.ts
index bd2a4854..0cd08916 100644
--- a/packages/elaborator/test/fragment.test.ts
+++ b/packages/elaborator/test/fragment.test.ts
@@ -140,7 +140,7 @@ function speechFragment(): GraphFragment {
{
id: "visual",
producer: speechBasisProducers.projectVisual,
- inputs: { basis: operation("generate"), canvas: input("canvas") },
+ inputs: { basis: operation("generate") },
result: { kind: "output", name: "visual" },
},
],
@@ -163,7 +163,7 @@ function speechFragment(): GraphFragment {
name: "visual",
type: compositionTypes.visualTrack,
root: operation("visual"),
- semanticInputs: ["request", "style", "canvas"],
+ semanticInputs: ["request", "style"],
fidelity: "exact",
},
],
diff --git a/packages/media-execution/src/execute.ts b/packages/media-execution/src/execute.ts
index d13a3254..d74c8caa 100644
--- a/packages/media-execution/src/execute.ts
+++ b/packages/media-execution/src/execute.ts
@@ -260,8 +260,6 @@ function stream<T extends MediaStream["kind"]>(
type NormalizationPlan = {
readonly video?: MediaVideoStream;
readonly audio?: MediaAudioStream;
- readonly origin: MediaTimestamp;
- readonly end: MediaTimestamp;
readonly frameCount: number;
readonly sampleFrames: number;
readonly audioTrimStartSamples: number;
@@ -300,8 +298,6 @@ function normalizationPlan(
if (audio === undefined) {
return {
...(video === undefined ? {} : { video }),
- origin: authority.startPts,
- end: authority.endPts,
frameCount,
sampleFrames,
audioTrimStartSamples: 0,
@@ -320,8 +316,6 @@ function normalizationPlan(
return {
...(video === undefined ? {} : { video }),
audio,
- origin: authority.startPts,
- end: authority.endPts,
frameCount,
sampleFrames,
audioTrimStartSamples: Math.max(0, -audioStart),
@@ -749,41 +743,19 @@ export async function executeNormalizeMedia(
const media = sealSynchronizedMedia({
contract: "svml.synchronized-media@1",
timeline: {
- spanAuthority: need.selection.spanAuthority,
frameRate: need.frameRate,
frameCount: plan.frameCount,
- sampleRate: 48_000,
- sampleFrames: plan.sampleFrames,
- },
- sourceMap: {
- sourceOriginPts: plan.origin,
- sourceEndPts: plan.end,
- audioTrimStartSamples: plan.audioTrimStartSamples,
- audioTrimEndSamples: plan.audioTrimEndSamples,
- audioHeadSamples: plan.audioHeadSamples,
- audioContentSamples: plan.audioContentSamples,
- audioTailSamples: plan.audioTailSamples,
},
...(visualArtifact === undefined ? {} : {
visual: {
artifact: visualArtifact,
- sourceStreamIndex: plan.video!.index,
width: visualWidth!,
height: visualHeight!,
- frameRate: need.frameRate,
- frameCount: plan.frameCount,
- muted: true as const,
},
}),
...(audioArtifact === undefined ? {} : {
audio: {
artifact: audioArtifact,
- sourceStreamIndex: plan.audio!.index,
- codec: "pcm_s16le" as const,
- sampleRate: 48_000 as const,
- channels: 2 as const,
- sampleFrames: plan.sampleFrames,
- loudness: "preserved" as const,
},
}),
});
diff --git a/packages/media-pipeline/README.md b/packages/media-pipeline/README.md
index 4aad1ac0..b7338559 100644
--- a/packages/media-pipeline/README.md
+++ b/packages/media-pipeline/README.md
@@ -23,7 +23,10 @@ Segment identities.
The current normalization profile uses one source presentation origin, a requested rational video frame
rate, and a 48 kHz stereo PCM render stem. It preserves input level: loudness/mastering remains a
-separate author policy.
+separate author policy. Its public `SynchronizedMedia` waist carries only the common frame domain,
+optional visual Artifact plus intrinsic extent, and optional audio Artifact. Stream indexes,
+authority choice and the trim/pad ledger remain in Selection and execution instead of travelling
+through every consumer.
This package is the shared execution vocabulary, not mandatory authoring boilerplate. A consuming
Surface that already owns the needed policy may expand the same finite graph itself. For example,
diff --git a/packages/media-track/README.md b/packages/media-track/README.md
index c2a24957..27f11dbe 100644
--- a/packages/media-track/README.md
+++ b/packages/media-track/README.md
@@ -5,9 +5,10 @@ Official provider-free Media Item and Sequence authoring package.
`Track` accepts explicit ProgramSpace, CanvasSpace and, when semantic points are used, a
CompleteSemanticMap. Each independently timed `Item` receives an explicit SpatialFrame and either
one direct source or ordered Paint/sample layers. A visual source is always named by its actual
-form: `image`, generated/raw `video`, prepared `media`, or compositable `surface`. Raw video is
-automatically inspected and normalized to the connected ProgramSpace; `audio="include"` opts its
-audio into the same normalization. `Sequence` owns an ordered replacement surface with explicit
+form: `image`, generated/raw `video`, prepared `media`, or compositable `surface`. For raw video the
+Track Surface declares the connected ProgramSpace as its consumption policy and expands ordinary
+Media Pipeline inspect/select/normalize Operations; it does not execute normalization itself.
+`audio="include"` opts audio into that same graph. `Sequence` owns an ordered replacement surface with explicit
activation points and pairwise Handoffs. Package-owned Recipes cover fitting, source occupancy,
frame Paint, clipping, borders, shadows, lifecycle motion and sampling motion.
diff --git a/packages/media-track/src/author.ts b/packages/media-track/src/author.ts
index b10b109d..54b936f2 100644
--- a/packages/media-track/src/author.ts
+++ b/packages/media-track/src/author.ts
@@ -1,5 +1,5 @@
import type { SvsRecipe } from "@narratage/svs";
-import { sealContentFit } from "@narratage/spatial";
+import { decodeContentFitProperties } from "@narratage/spatial";
import type { ContentFit } from "@narratage/spatial";
import {
@@ -125,14 +125,7 @@ const FRAME_KEYS = ["stack-order", "clip", "radius", "padding", "border-width",
const MOTION_KEYS = ["enter", "enter-frames", "enter-easing", "enter-direction", "enter-amount", "enter-origin", "sustain", "exit", "exit-frames", "exit-easing", "exit-direction", "exit-amount", "exit-origin"] as const;
export function decodeMediaFit(recipe: SvsRecipe): ContentFit {
- return sealContentFit({
- contract: "svml.content-fit@1",
- sizing: oneOf(recipe, "fit", ["contain", "cover", "fit-width", "fit-height", "native", "scale-down", "stretch"] as const, "contain"),
- framePoint: { x: number(recipe, "frame-x", 0.5), y: number(recipe, "frame-y", 0.5) },
- contentPoint: { x: number(recipe, "content-x", 0.5), y: number(recipe, "content-y", 0.5) },
- offsetPx: { x: number(recipe, "fit-offset-x", 0), y: number(recipe, "fit-offset-y", 0) },
- constraint: oneOf(recipe, "fit-constraint", ["bounded", "free"] as const, "bounded"),
- });
+ return decodeContentFitProperties(recipe.properties, `Media Recipe ${recipe.path}`);
}
function occupancy(recipe: SvsRecipe): MediaVisualOccupancy {
diff --git a/packages/media-track/src/component.ts b/packages/media-track/src/component.ts
index 21c05518..2bac8700 100644
--- a/packages/media-track/src/component.ts
+++ b/packages/media-track/src/component.ts
@@ -1,6 +1,6 @@
import type { ComponentPackage, ProducerHandlerContext } from "@narratage/component-kit";
import type { CompositableSurfaceRef, SynchronizedMedia } from "@narratage/media";
-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
import type { ProgramSpace } from "@narratage/program-space";
import { canonicalize } from "@narratage/protocol";
import type { BlobRef, StoredValue } from "@narratage/protocol";
@@ -21,6 +21,7 @@ import {
appendMediaSequenceUntilSelection,
appendMomentMediaItem,
appendProgramMediaItem,
+ appendSegmentMediaItem,
appendSelectionMediaItem,
bindMediaItemClipPath,
bindMediaSequenceClipPath,
@@ -140,6 +141,12 @@ export const mediaTrackComponent = {
inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),
inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), value.spec, value.sounds)) }, needs: {} };
} },
+ { producer: mediaTrackProducers.appendSegmentItem, implementationDigest: mediaTrackImplementationDigests.appendSegmentItem, handler: ({ inputs }) => {
+ const value = itemInputs(inputs);
+ return { outputs: { set: output(appendSegmentMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,
+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),
+ inline<NarrativeExcerpt>(inputs.segment?.value, "NarrativeExcerpt"), value.spec, value.sounds)) }, needs: {} };
+ } },
{ producer: mediaTrackProducers.appendMomentItem, implementationDigest: mediaTrackImplementationDigests.appendMomentItem, handler: ({ inputs }) => {
const value = itemInputs(inputs);
return { outputs: { set: output(appendMomentMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,
diff --git a/packages/media-track/src/index.ts b/packages/media-track/src/index.ts
index a80b05d9..84b3bb3b 100644
--- a/packages/media-track/src/index.ts
+++ b/packages/media-track/src/index.ts
@@ -73,6 +73,7 @@ export {
appendMediaSequenceUntilProgramEnd,
appendMediaSequenceUntilSelection,
appendProgramMediaItem,
+ appendSegmentMediaItem,
appendSelectionMediaItem,
bindMediaItemClipPath,
bindMediaSequenceClipPath,
diff --git a/packages/media-track/src/layers.ts b/packages/media-track/src/layers.ts
index c797172a..c3f65968 100644
--- a/packages/media-track/src/layers.ts
+++ b/packages/media-track/src/layers.ts
@@ -1,5 +1,6 @@
import {
assertCompositableSurfaceRef,
+ synchronizedMediaSampleFrames,
verifySynchronizedMedia,
} from "@narratage/media";
import type { CompositableSurfaceRef, SynchronizedMedia } from "@narratage/media";
@@ -276,11 +277,11 @@ export function appendTimedMediaLayer(
widthPx: media.visual.width,
heightPx: media.visual.height,
},
- frameRate: { ...media.visual.frameRate },
- frameCount: media.visual.frameCount,
+ frameRate: { ...media.timeline.frameRate },
+ frameCount: media.timeline.frameCount,
...(media.audio === undefined ? {} : { audio: {
artifact: structuredClone(media.audio.artifact),
- sampleFrames: media.audio.sampleFrames,
+ sampleFrames: synchronizedMediaSampleFrames(media),
} }),
};
return append(set, sampleLayer(source, fit, spec));
diff --git a/packages/media-track/src/manifest.ts b/packages/media-track/src/manifest.ts
index efc20fbb..1e85512d 100644
--- a/packages/media-track/src/manifest.ts
+++ b/packages/media-track/src/manifest.ts
@@ -50,6 +50,7 @@ export const mediaTrackProducers = {
createSet: { module: mediaTrackModuleRef, name: "create-media-track-set" },
appendProgramItem: { module: mediaTrackModuleRef, name: "append-program-media-item" },
appendSelectionItem: { module: mediaTrackModuleRef, name: "append-selection-media-item" },
+ appendSegmentItem: { module: mediaTrackModuleRef, name: "append-segment-media-item" },
appendMomentItem: { module: mediaTrackModuleRef, name: "append-moment-media-item" },
bindItemClipPath: { module: mediaTrackModuleRef, name: "bind-media-item-clip-path" },
bindSequenceClipPath: { module: mediaTrackModuleRef, name: "bind-media-sequence-clip-path" },
@@ -150,7 +151,7 @@ const signedDuration: ValueSchema = { kind: "oneOf", variants: [
object({ unit: { schema: { kind: "literal", value: "seconds" } }, numerator: { schema: integer }, denominator: { schema: positiveInteger } }),
] };
const point: ValueSchema = { kind: "oneOf", variants: [
- ...["program.start", "program.end", "selection.start", "selection.end", "moment.cue"].map((ref) => object({
+ ...["program.start", "program.end", "selection.start", "selection.end", "segment.start", "segment.end", "moment.cue"].map((ref) => object({
ref: { schema: { kind: "literal", value: ref } }, offset: { schema: signedDuration, optional: true },
})),
object({ ref: { schema: { kind: "literal", value: "absolute" } }, at: { schema: duration } }),
@@ -343,6 +344,7 @@ export const mediaTrackManifest: ModuleManifest = {
{ name: mediaTrackProducers.createSet.name, inputs: [], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/create-set", digest: mediaTrackImplementationDigests.createSet } },
{ name: mediaTrackProducers.appendProgramItem.name, inputs: [...itemInputs], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-program-item", digest: mediaTrackImplementationDigests.appendProgramItem } },
{ name: mediaTrackProducers.appendSelectionItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-selection-item", digest: mediaTrackImplementationDigests.appendSelectionItem } },
+ { name: mediaTrackProducers.appendSegmentItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "segment", type: narrativeTypes.excerpt }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-segment-item", digest: mediaTrackImplementationDigests.appendSegmentItem } },
{ name: mediaTrackProducers.appendMomentItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-moment-item", digest: mediaTrackImplementationDigests.appendMomentItem } },
{ name: mediaTrackProducers.bindItemClipPath.name, inputs: [{ name: "spec", type: mediaTrackTypes.itemSpec }, { name: "path", type: spatialTypes.path }], outputs: [{ name: "spec", type: mediaTrackTypes.itemSpec }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/bind-item-clip-path", digest: mediaTrackImplementationDigests.bindItemClipPath } },
{ name: mediaTrackProducers.bindSequenceClipPath.name, inputs: [{ name: "spec", type: mediaTrackTypes.sequenceSpec }, { name: "path", type: spatialTypes.path }], outputs: [{ name: "spec", type: mediaTrackTypes.sequenceSpec }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/bind-sequence-clip-path", digest: mediaTrackImplementationDigests.bindSequenceClipPath } },
diff --git a/packages/media-track/src/program.ts b/packages/media-track/src/program.ts
index 929c86e4..399fedab 100644
--- a/packages/media-track/src/program.ts
+++ b/packages/media-track/src/program.ts
@@ -5,7 +5,7 @@ import {
sealVisualTrack,
} from "@narratage/composition";
import type { AudioClip, AudioTrack, VisualTrack } from "@narratage/composition";
-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
import {
assertProgramSpaceIdentity,
programFrameSampleBoundary,
@@ -24,6 +24,7 @@ import {
locateSelectionOccurrences,
projectMomentWindows,
projectProgramWindow,
+ projectSegmentWindow,
projectSelectionWindows,
} from "@narratage/temporal";
import type { ProjectedOccurrence, TemporalDuration, TemporalPointExpression } from "@narratage/temporal";
@@ -74,6 +75,7 @@ export const mediaTrackImplementationDigests = {
createSet: digestOf("@narratage/media-track/create-set@1"),
appendProgramItem: digestOf("@narratage/media-track/append-program-item@1"),
appendSelectionItem: digestOf("@narratage/media-track/append-selection-item@1"),
+ appendSegmentItem: digestOf("@narratage/media-track/append-segment-item@1"),
appendMomentItem: digestOf("@narratage/media-track/append-moment-item@1"),
bindItemClipPath: digestOf("@narratage/media-track/bind-item-clip-path@1"),
bindSequenceClipPath: digestOf("@narratage/media-track/bind-sequence-clip-path@1"),
@@ -130,7 +132,7 @@ function assertDuration(value: TemporalDuration, label: string, signed: boolean)
}
function assertPoint(value: TemporalPointExpression, label: string): void {
- assert(["program.start", "program.end", "selection.start", "selection.end", "moment.cue", "absolute"].includes(value.ref),
+ assert(["program.start", "program.end", "selection.start", "selection.end", "segment.start", "segment.end", "moment.cue", "absolute"].includes(value.ref),
`${label}.ref is invalid.`);
if (value.ref === "absolute") assertDuration(value.at, `${label}.at`, false);
else if (value.offset !== undefined) assertDuration(value.offset, `${label}.offset`, true);
@@ -272,6 +274,24 @@ export function appendSelectionMediaItem(
}));
}
+export function appendSegmentMediaItem(
+ set: MediaTrackSet,
+ header: MediaTrackHeader,
+ space: ProgramSpace,
+ canvas: CanvasSpace,
+ layers: MediaLayerSet,
+ frame: MediaItemProgram["frame"],
+ map: CompleteSemanticMap,
+ segment: NarrativeExcerpt,
+ spec: MediaItemSpec,
+ sounds: MediaSoundSet,
+): MediaTrackSet {
+ assert(spec.expansion.kind === "one", `Segment Media Item ${spec.id} must use one occurrence.`);
+ return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, [projectSegmentWindow({
+ itemId: spec.id, map, segment, space, projection: spec.projection,
+ })]);
+}
+
export function appendMomentMediaItem(
set: MediaTrackSet,
header: MediaTrackHeader,
diff --git a/packages/media-track/src/restricted-speech.ts b/packages/media-track/src/restricted-speech.ts
index 02ea0da0..fc051b6a 100644
--- a/packages/media-track/src/restricted-speech.ts
+++ b/packages/media-track/src/restricted-speech.ts
@@ -2,8 +2,8 @@ import type { VisualPresent } from "@narratage/composition";
import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";
import type { ProgramSpace } from "@narratage/program-space";
import type { BlobRef } from "@narratage/protocol";
-import { assertCanvasSpace, assertContentFit, assertIntrinsicExtent } from "@narratage/spatial";
-import type { CanvasSpace, ContentFit, IntrinsicExtent } from "@narratage/spatial";
+import { assertContentFit, assertIntrinsicExtent, assertSpatialFrame } from "@narratage/spatial";
+import type { ContentFit, IntrinsicExtent, SpatialFrame } from "@narratage/spatial";
import { lowerMediaItemElements } from "./lower.js";
import type { MediaItemProgram } from "./types.js";
@@ -15,29 +15,32 @@ export type RestrictedSpeechVisualClip = {
readonly extent: IntrinsicExtent;
readonly frameRate: { readonly numerator: number; readonly denominator: number };
readonly frameCount: number;
+ readonly frame: SpatialFrame;
+ readonly fit: ContentFit;
+ readonly stackingOrder: number;
};
/**
- * Focused reuse point for Speech Spine: one normalized muted take, one full-Canvas
- * foreground layer and no Media author semantics beyond the shared lowering laws.
+ * Focused reuse point for Speech Spine: one normalized muted take and one
+ * explicitly placed foreground layer, without Media Track motion or sequence
+ * semantics.
*/
export function lowerRestrictedSpeechVisualPresents(
trackId: string,
space: ProgramSpace,
- canvas: CanvasSpace,
clips: readonly RestrictedSpeechVisualClip[],
- fit: ContentFit,
): readonly VisualPresent[] {
assertProgramSpaceIdentity(space);
- assertCanvasSpace(canvas);
- assertContentFit(fit);
- if (!trackId || clips.length === 0) throw new Error("Restricted Speech visual requires an id and clips.");
+ if (!trackId) throw new Error("Restricted Speech visual requires an id.");
const totalFrames = programSpaceFrameCount(space);
return clips.map((clip, index) => {
if (clip.artifact.kind !== "blob" || !clip.artifact.mediaType.startsWith("video/")) {
throw new Error(`Speech visual ${clip.id} is not a normalized moving-video Blob.`);
}
assertIntrinsicExtent(clip.extent);
+ assertSpatialFrame(clip.frame);
+ assertContentFit(clip.fit);
+ if (!Number.isSafeInteger(clip.stackingOrder)) throw new Error(`Speech visual ${clip.id} has an invalid stacking order.`);
if (clip.frameRate.numerator !== space.frameRate.numerator
|| clip.frameRate.denominator !== space.frameRate.denominator) {
throw new Error(`Speech visual ${clip.id} is not normalized to ProgramSpace frame rate.`);
@@ -52,13 +55,7 @@ export function lowerRestrictedSpeechVisualPresents(
id: `${trackId}:${clip.id}`,
sourceOccurrenceId: clip.id,
span: { ...clip.span },
- frame: {
- contract: "svml.spatial-frame@1",
- xPx: 0,
- yPx: 0,
- widthPx: canvas.widthPx,
- heightPx: canvas.heightPx,
- },
+ frame: structuredClone(clip.frame),
presentation: {
clip: { kind: "frame" },
padding: { topPx: 0, rightPx: 0, bottomPx: 0, leftPx: 0 },
@@ -74,7 +71,7 @@ export function lowerRestrictedSpeechVisualPresents(
frameRate: { ...clip.frameRate },
frameCount: clip.frameCount,
},
- fit: structuredClone(fit),
+ fit: structuredClone(clip.fit),
occupancy: { mode: "once", align: "start" },
appearance: {
opacity: 1,
@@ -83,7 +80,7 @@ export function lowerRestrictedSpeechVisualPresents(
}],
motion: { sustain: [] },
stacking: {
- order: 0,
+ order: clip.stackingOrder,
tieBreak: `${trackId}:${String(index + 1).padStart(4, "0")}:${clip.id}`,
},
sounds: [],
diff --git a/packages/media-track/src/sounds.ts b/packages/media-track/src/sounds.ts
index 93ffb940..e592c4cd 100644
--- a/packages/media-track/src/sounds.ts
+++ b/packages/media-track/src/sounds.ts
@@ -1,4 +1,4 @@
-import { verifySynchronizedMedia } from "@narratage/media";
+import { synchronizedMediaSampleFrames, verifySynchronizedMedia } from "@narratage/media";
import type { SynchronizedMedia } from "@narratage/media";
import { canonicalize } from "@narratage/protocol";
@@ -62,7 +62,7 @@ export function appendMediaSound(
trigger: structuredClone(spec.trigger),
source: {
artifact: structuredClone(media.audio.artifact),
- sampleFrames: media.audio.sampleFrames,
+ sampleFrames: synchronizedMediaSampleFrames(media),
},
gain: spec.gain,
}],
diff --git a/packages/media-track/src/surface.ts b/packages/media-track/src/surface.ts
index 304c1676..93df0f21 100644
--- a/packages/media-track/src/surface.ts
+++ b/packages/media-track/src/surface.ts
@@ -160,7 +160,7 @@ function negate(value: TemporalDuration): TemporalDuration {
function point(value: string, label: string): TemporalPointExpression {
const trimmed = value.trim();
- const refs = ["program.start", "program.end", "selection.start", "selection.end", "moment.cue"] as const;
+ const refs = ["program.start", "program.end", "selection.start", "selection.end", "segment.start", "segment.end", "moment.cue"] as const;
for (const ref of refs) {
if (trimmed === ref) return { ref };
const match = new RegExp(`^${ref.replace(".", "\\.")}\\s*([+-])\\s*(.+)$`, "u").exec(trimmed);
@@ -173,7 +173,7 @@ function point(value: string, label: string): TemporalPointExpression {
}
type TemporalBinding = {
- readonly kind: "program" | "selection" | "moment";
+ readonly kind: "program" | "selection" | "segment" | "moment";
readonly projection: TemporalWindowProjection;
readonly source?: SurfaceResolvedReference;
};
@@ -187,6 +187,7 @@ function temporalBinding(
const start = optionalText(element, "start");
const end = optionalText(element, "end");
const selection = element.attributes.selection;
+ const segment = element.attributes.segment;
const moment = element.attributes.moment;
const forms = Number(during !== undefined) + Number(at !== undefined) + Number(start !== undefined || end !== undefined);
if (forms !== 1) throw new Error(`${element.name} requires exactly one of during, at/for, or start/end.`);
@@ -195,10 +196,15 @@ function temporalBinding(
if (during.trim() !== "program") throw new Error(`${element.name}.during text must be program.`);
return { kind: "program", projection: { start: { ref: "program.start" }, end: { ref: "program.end" } } };
}
- return {
+ const source = oneOfReferences(during, `${element.name}.during`, [narrativeTypes.selection, narrativeTypes.excerpt], resolve);
+ return sameType(source.type, narrativeTypes.selection) ? {
kind: "selection",
- source: reference(during, `${element.name}.during`, narrativeTypes.selection, resolve),
+ source,
projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },
+ } : {
+ kind: "segment",
+ source,
+ projection: { start: { ref: "segment.start" }, end: { ref: "segment.end" } },
};
}
if (at !== undefined) {
@@ -210,7 +216,9 @@ function temporalBinding(
};
}
if (start === undefined || end === undefined) throw new Error(`${element.name} explicit timing requires start and end.`);
- if (selection !== undefined && moment !== undefined) throw new Error(`${element.name} cannot bind Selection and Moment together.`);
+ if (Number(selection !== undefined) + Number(segment !== undefined) + Number(moment !== undefined) > 1) {
+ throw new Error(`${element.name} cannot bind Selection, Segment and Moment together.`);
+ }
if (selection !== undefined) return {
kind: "selection",
source: reference(selection, `${element.name}.selection`, narrativeTypes.selection, resolve),
@@ -221,6 +229,11 @@ function temporalBinding(
source: reference(moment, `${element.name}.moment`, narrativeTypes.moment, resolve),
projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) },
};
+ if (segment !== undefined) return {
+ kind: "segment",
+ source: reference(segment, `${element.name}.segment`, narrativeTypes.excerpt, resolve),
+ projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) },
+ };
return { kind: "program", projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) } };
}
@@ -361,7 +374,8 @@ function createMediaTrackSurfaceFragment(inputTypes: readonly { readonly name: s
operations.push({
id: appendId,
producer: item.binding === "program" ? mediaTrackProducers.appendProgramItem
- : item.binding === "selection" ? mediaTrackProducers.appendSelectionItem : mediaTrackProducers.appendMomentItem,
+ : item.binding === "selection" ? mediaTrackProducers.appendSelectionItem
+ : item.binding === "segment" ? mediaTrackProducers.appendSegmentItem : mediaTrackProducers.appendMomentItem,
inputs: item.binding === "program" ? common : {
...common, map: input("map"), [item.binding]: input(item.sourceName!),
},
@@ -729,7 +743,7 @@ export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, res
const itemSuffix = suffix(itemIndex);
allowed(child, [
"id", "image", "video", "media", "surface", "extent", "audio", "frame", "appearance", "motion", "source-audio", "audio-gain",
- "clip", "during", "at", "for", "start", "end", "selection", "moment", "occurrences",
+ "clip", "during", "at", "for", "start", "end", "selection", "segment", "moment", "occurrences",
]);
const id = optionalText(child, "id") ?? `${trackId}.item.${itemSuffix}`;
const appearance = recipe(reference(child.attributes.appearance, `${child.name}.appearance`, svsRecipeType, resolveReference), `${child.name}.appearance`);
diff --git a/packages/media-track/test/media-track.test.ts b/packages/media-track/test/media-track.test.ts
index 44f7fe6c..05479e58 100644
--- a/packages/media-track/test/media-track.test.ts
+++ b/packages/media-track/test/media-track.test.ts
@@ -16,6 +16,7 @@ import {
appendMediaSequenceMember,
appendMomentMediaItem,
appendProgramMediaItem,
+ appendSegmentMediaItem,
appendSelectionMediaItem,
appendStillMediaLayer,
appendSurfaceMediaLayer,
@@ -139,38 +140,16 @@ function timed(id = "timed", withAudio = true): SynchronizedMedia {
return {
contract: "svml.synchronized-media@1",
timeline: {
- spanAuthority: "video",
frameRate: { numerator: 30, denominator: 1 },
frameCount: 60,
- sampleRate: 48_000,
- sampleFrames,
- },
- sourceMap: {
- sourceOriginPts: { ticks: "0", timeBase: { numerator: 1, denominator: 30 } },
- sourceEndPts: { ticks: "60", timeBase: { numerator: 1, denominator: 30 } },
- audioTrimStartSamples: 0,
- audioTrimEndSamples: 0,
- audioHeadSamples: 0,
- audioContentSamples: withAudio ? sampleFrames : 0,
- audioTailSamples: 0,
},
visual: {
artifact: { kind: "blob", digest: digestOf(`video:${id}`), size: 10_000, mediaType: "video/mp4" },
- sourceStreamIndex: 0,
width: 720,
height: 1280,
- frameRate: { numerator: 30, denominator: 1 },
- frameCount: 60,
- muted: true,
},
...(withAudio ? { audio: {
artifact: { kind: "blob" as const, digest: digestOf(`audio:${id}`), size: sampleFrames * 4, mediaType: "audio/wav" },
- sourceStreamIndex: 1,
- codec: "pcm_s16le" as const,
- sampleRate: 48_000 as const,
- channels: 2 as const,
- sampleFrames,
- loudness: "preserved" as const,
} } : {}),
};
}
@@ -559,6 +538,28 @@ test("Media Items consume the shared one/each temporal algebra without becoming
assert.deepEqual(absolute.items[0]?.span, { startFrame: 7, endFrameExclusive: 19 });
});
+test("a Media Item can consume one whole Narrative Segment without a synthetic Selection", () => {
+ const semanticMap: CompleteSemanticMap = {
+ contract: "svml.complete-semantic-map@1",
+ tokens: [],
+ anchors: [
+ { identity: "segment:answer:start", timeSec: 1, frame: 30 },
+ { identity: "segment:answer:end", timeSec: 3, frame: 90 },
+ ],
+ };
+ const result = appendSegmentMediaItem(
+ createMediaTrackSet(), header, space, canvas, stillLayers(), frame,
+ semanticMap,
+ { contract: "svml.narrative-excerpt@1", kind: "segment", id: "answer", tokenStart: 0, tokenEndExclusive: 1 },
+ itemSpec({
+ id: "whole-answer",
+ projection: { start: { ref: "segment.start" }, end: { ref: "segment.end" } },
+ }),
+ createMediaSoundSet(),
+ );
+ assert.deepEqual(result.items.map((item) => item.span), [{ startFrame: 30, endFrameExclusive: 90 }]);
+});
+
test("every declared lifecycle and sustain operator lowers, and outside-canvas motion resolves from explicit geometry", () => {
for (const operator of ["fade", "slide", "scale", "pop", "bounce", "blur-reveal", "wipe", "flip", "spin"] as const) {
const direction = ["slide", "wipe", "flip"].includes(operator) ? { direction: "up" as const } : {};
@@ -709,6 +710,7 @@ test("the Media author Surface emits explicit graph edges for layers, semantic t
["surface", plain("surface", mediaTypes.compositableSurface)],
["sfx", plain("sfx", mediaTypes.synchronized)],
["selection", plain("selection", narrativeTypes.selection)],
+ ["answer-segment", plain("answer-segment", narrativeTypes.excerpt)],
["terminal", plain("terminal", narrativeTypes.selection)],
["cue1", plain("cue1", narrativeTypes.moment)],
["cue2", plain("cue2", narrativeTypes.moment)],
@@ -723,6 +725,7 @@ test("the Media author Surface emits explicit graph edges for layers, semantic t
]);
const root = node("media:Track", { id: "editorial", space: ref("space"), canvas: ref("canvas"), map: ref("map") }, [
node("media:Item", { id: "still-card", image: ref("still"), extent: ref("extent"), frame: ref("frame"), clip: ref("clip-path"), appearance: ref("still-style"), during: "program" }),
+ node("media:Item", { id: "segment-card", image: ref("still"), extent: ref("extent"), frame: ref("frame"), appearance: ref("still-style"), during: ref("answer-segment") }),
node("media:Item", { id: "proof", frame: ref("frame"), appearance: ref("card-style"), motion: ref("motion"), during: ref("selection"), "source-audio": "video", "audio-gain": "0.8" }, [
node("media:Paint", { id: "backing", appearance: ref("paint-style") }),
node("media:Layer", { id: "video", video: ref("raw-video"), audio: "include", appearance: ref("video-style") }, [
@@ -749,6 +752,7 @@ test("the Media author Surface emits explicit graph edges for layers, semantic t
const fragment = result.fragments[0]!;
const producers = fragment.operations.map((entry) => entry.producer.name);
assert.ok(producers.includes("append-still-media-layer"));
+ assert.ok(producers.includes("append-segment-media-item"));
assert.ok(producers.includes("append-timed-media-layer"));
assert.ok(producers.includes(mediaPipelineProducers.bindAvRequest.name));
assert.ok(producers.includes(mediaPipelineProducers.inspect.name));
@@ -765,9 +769,9 @@ test("the Media author Surface emits explicit graph edges for layers, semantic t
assert.ok(fragment.inputs.some((entry) => entry.type.name === narrativeTypes.selection.name));
assert.ok(fragment.inputs.some((entry) => entry.type.name === spatialTypes.path.name));
const itemSpecs = result.records.filter((entry) => entry.type.name === "MediaItemSpec");
- assert.equal(itemSpecs.length, 2);
- const selected = itemSpecs[1]!.value.kind === "inline"
- ? itemSpecs[1]!.value.value as unknown as MediaItemSpec
+ assert.equal(itemSpecs.length, 3);
+ const selected = itemSpecs[2]!.value.kind === "inline"
+ ? itemSpecs[2]!.value.value as unknown as MediaItemSpec
: undefined;
assert.equal(selected?.sourceAudio?.fromLayer, "video");
assert.ok(mediaTrackManifest.surfaces.some((surface) => surface.name === "track"));
diff --git a/packages/media/README.md b/packages/media/README.md
index 85429efe..b8e3db9f 100644
--- a/packages/media/README.md
+++ b/packages/media/README.md
@@ -11,3 +11,8 @@ for one primary face followed by exact glyph fallbacks. Media declarations do no
inspect media, call a Provider or promote declared audio to speech. The consuming author package
decides whether an audio artifact is a voice reference, soundtrack, evidence source or something
else, and which visual component uses an exact Font.
+
+The same module owns the deliberately narrow `SynchronizedMedia` contract used after technical
+normalization. It contains one common `frameRate`/`frameCount`, an optional visual Artifact with its
+intrinsic pixel extent, and an optional audio Artifact. Source stream indexes, selection authority,
+normalization ledgers and repeated codec/rate constants are not downstream media fields.
diff --git a/packages/media/src/identity.ts b/packages/media/src/identity.ts
index 5363e929..abe6ad57 100644
--- a/packages/media/src/identity.ts
+++ b/packages/media/src/identity.ts
@@ -159,62 +159,33 @@ export function sealSynchronizedMedia(value: SynchronizedMedia): SynchronizedMed
return canonicalize(value) as unknown as SynchronizedMedia;
}
+/** Exact 48 kHz sample span implied by one normalized frame domain. */
+export function synchronizedMediaSampleFrames(value: SynchronizedMedia): number {
+ const numerator = BigInt(value.timeline.frameCount) * 48_000n * BigInt(value.timeline.frameRate.denominator);
+ const denominator = BigInt(value.timeline.frameRate.numerator);
+ const rounded = (numerator * 2n + denominator) / (denominator * 2n);
+ assert(rounded > 0n && rounded <= BigInt(Number.MAX_SAFE_INTEGER),
+ "SynchronizedMedia frame domain has an invalid sample span");
+ return Number(rounded);
+}
+
export function verifySynchronizedMedia(value: unknown): asserts value is SynchronizedMedia {
const item = object(value, "SynchronizedMedia") as unknown as SynchronizedMedia;
assert(item.contract === "svml.synchronized-media@1", "SynchronizedMedia contract is invalid");
- assert(item.timeline.spanAuthority === "video" || item.timeline.spanAuthority === "audio",
- "SynchronizedMedia authority is invalid");
verifyRational(item.timeline.frameRate, "SynchronizedMedia.timeline.frameRate");
positiveInteger(item.timeline.frameCount, "SynchronizedMedia.timeline.frameCount");
- assert(item.timeline.sampleRate === 48_000, "SynchronizedMedia sample rate must be 48000");
- positiveInteger(item.timeline.sampleFrames, "SynchronizedMedia.timeline.sampleFrames");
- verifyTimestamp(item.sourceMap.sourceOriginPts, "SynchronizedMedia.sourceMap.sourceOriginPts");
- verifyTimestamp(item.sourceMap.sourceEndPts, "SynchronizedMedia.sourceMap.sourceEndPts");
- assert(compareTimestamp(item.sourceMap.sourceOriginPts, item.sourceMap.sourceEndPts) < 0,
- "SynchronizedMedia source interval is empty or reversed");
- nonNegativeInteger(item.sourceMap.audioTrimStartSamples, "SynchronizedMedia.sourceMap.audioTrimStartSamples");
- nonNegativeInteger(item.sourceMap.audioTrimEndSamples, "SynchronizedMedia.sourceMap.audioTrimEndSamples");
- nonNegativeInteger(item.sourceMap.audioHeadSamples, "SynchronizedMedia.sourceMap.audioHeadSamples");
- nonNegativeInteger(item.sourceMap.audioContentSamples, "SynchronizedMedia.sourceMap.audioContentSamples");
- nonNegativeInteger(item.sourceMap.audioTailSamples, "SynchronizedMedia.sourceMap.audioTailSamples");
+ synchronizedMediaSampleFrames(item);
assert(item.visual !== undefined || item.audio !== undefined, "SynchronizedMedia contains no media projection");
if (item.visual !== undefined) {
verifyBlob(item.visual.artifact, "SynchronizedMedia.visual.artifact");
assert(item.visual.artifact.mediaType.startsWith("video/"), "SynchronizedMedia visual artifact is not video");
- nonNegativeInteger(item.visual.sourceStreamIndex, "SynchronizedMedia.visual.sourceStreamIndex");
positiveInteger(item.visual.width, "SynchronizedMedia.visual.width");
positiveInteger(item.visual.height, "SynchronizedMedia.visual.height");
- verifyRational(item.visual.frameRate, "SynchronizedMedia.visual.frameRate");
- assert(item.visual.frameRate.numerator === item.timeline.frameRate.numerator
- && item.visual.frameRate.denominator === item.timeline.frameRate.denominator,
- "SynchronizedMedia visual frame rate differs from its timeline");
- assert(item.visual.frameCount === item.timeline.frameCount, "SynchronizedMedia visual frame count differs");
- assert(item.visual.muted === true, "SynchronizedMedia visual must be muted");
}
if (item.audio !== undefined) {
verifyBlob(item.audio.artifact, "SynchronizedMedia.audio.artifact");
assert(item.audio.artifact.mediaType === "audio/wav", "SynchronizedMedia audio artifact must be WAV");
- nonNegativeInteger(item.audio.sourceStreamIndex, "SynchronizedMedia.audio.sourceStreamIndex");
- assert(item.audio.codec === "pcm_s16le" && item.audio.sampleRate === 48_000 && item.audio.channels === 2,
- "SynchronizedMedia audio shape is invalid");
- assert(item.audio.sampleFrames === item.timeline.sampleFrames,
- "SynchronizedMedia audio sample count differs from its timeline");
- assert(item.audio.loudness === "preserved", "SynchronizedMedia must not silently normalize loudness");
- assert(item.sourceMap.audioHeadSamples + item.sourceMap.audioContentSamples + item.sourceMap.audioTailSamples
- === item.timeline.sampleFrames,
- "SynchronizedMedia audio transform does not cover its complete timeline");
- } else {
- assert(item.sourceMap.audioTrimStartSamples === 0
- && item.sourceMap.audioTrimEndSamples === 0
- && item.sourceMap.audioHeadSamples === 0
- && item.sourceMap.audioContentSamples === 0
- && item.sourceMap.audioTailSamples === 0,
- "SynchronizedMedia without audio has an audio transform");
}
- assert(item.timeline.spanAuthority !== "video" || item.visual !== undefined,
- "SynchronizedMedia video authority has no visual projection");
- assert(item.timeline.spanAuthority !== "audio" || item.audio !== undefined,
- "SynchronizedMedia audio authority has no audio projection");
}
export function sealRenderedVisual(value: RenderedVisual): RenderedVisual {
diff --git a/packages/media/src/index.ts b/packages/media/src/index.ts
index 7c248428..774077a6 100644
--- a/packages/media/src/index.ts
+++ b/packages/media/src/index.ts
@@ -4,7 +4,7 @@ export {
decodeMediaImageSurface,
} from "./surface.js";
export { mediaComponent } from "./component.js";
-export { sealMediaInspection, sealMediaStreamSelection, sealMuxedMedia, sealRenderedVisual, sealSynchronizedMedia, sealTimelineAudio, verifyMediaInspection, verifyMediaStreamSelection, verifyMuxedMedia, verifyRenderedVisual, verifySynchronizedMedia, verifyTimelineAudio } from "./identity.js";
+export { sealMediaInspection, sealMediaStreamSelection, sealMuxedMedia, sealRenderedVisual, sealSynchronizedMedia, sealTimelineAudio, synchronizedMediaSampleFrames, verifyMediaInspection, verifyMediaStreamSelection, verifyMuxedMedia, verifyRenderedVisual, verifySynchronizedMedia, verifyTimelineAudio } from "./identity.js";
export { mediaDependency, mediaManifest, mediaManifestDigest, mediaModuleRef, mediaSurfaceImplementationDigests, mediaTypes, mediaValidatorDigests } from "./manifest.js";
export { compositableSurfaceSchema, fontArtifactSchema, fontStackSchema, mediaArtifactSchema, mediaInspectionSchema, mediaStreamSelectionSchema, muxedMediaSchema, renderedVisualSchema, synchronizedMediaSchema, timelineAudioSchema } from "./schema.js";
export { assertCompositableSurfaceRef, assertFontArtifactRef, assertFontStackRef } from "./render.js";
diff --git a/packages/media/src/schema.ts b/packages/media/src/schema.ts
index 98f76296..027bcdc3 100644
--- a/packages/media/src/schema.ts
+++ b/packages/media/src/schema.ts
@@ -43,17 +43,12 @@ export const mediaStreamSelectionSchema: ValueSchema = object({
});
export const synchronizedMediaSchema: ValueSchema = object({
contract: { schema: { kind: "literal", value: "svml.synchronized-media@1" } },
- timeline: { schema: object({ spanAuthority: { schema: { kind: "string", enum: ["video", "audio"] } }, frameRate: { schema: rational },
- frameCount: { schema: { kind: "number", integer: true, minimum: 1 } }, sampleRate: { schema: { kind: "literal", value: 48_000 } },
- sampleFrames: { schema: { kind: "number", integer: true, minimum: 1 } } }) },
- sourceMap: { schema: object({ sourceOriginPts: { schema: timestamp }, sourceEndPts: { schema: timestamp }, audioTrimStartSamples: { schema: integer },
- audioTrimEndSamples: { schema: integer }, audioHeadSamples: { schema: integer }, audioContentSamples: { schema: integer }, audioTailSamples: { schema: integer } }) },
- visual: { schema: object({ artifact: { schema: blobArtifactSchema() }, sourceStreamIndex: { schema: integer },
+ timeline: { schema: object({ frameRate: { schema: rational },
+ frameCount: { schema: { kind: "number", integer: true, minimum: 1 } } }) },
+ visual: { schema: object({ artifact: { schema: blobArtifactSchema() },
width: { schema: { kind: "number", integer: true, minimum: 1 } }, height: { schema: { kind: "number", integer: true, minimum: 1 } },
- frameRate: { schema: rational }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } }, muted: { schema: { kind: "literal", value: true } } }), optional: true },
- audio: { schema: object({ artifact: { schema: blobArtifactSchema(["audio/wav"]) }, sourceStreamIndex: { schema: integer }, codec: { schema: { kind: "literal", value: "pcm_s16le" } },
- sampleRate: { schema: { kind: "literal", value: 48_000 } }, channels: { schema: { kind: "literal", value: 2 } },
- sampleFrames: { schema: { kind: "number", integer: true, minimum: 1 } }, loudness: { schema: { kind: "literal", value: "preserved" } } }), optional: true },
+ }), optional: true },
+ audio: { schema: object({ artifact: { schema: blobArtifactSchema(["audio/wav"]) } }), optional: true },
});
export const renderedVisualSchema: ValueSchema = object({ contract: { schema: { kind: "literal", value: "svml.rendered-visual@1" } },
frameRate: { schema: rational }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } },
diff --git a/packages/media/src/types.ts b/packages/media/src/types.ts
index 0e5a0253..a94da3be 100644
--- a/packages/media/src/types.ts
+++ b/packages/media/src/types.ts
@@ -88,39 +88,16 @@ export type MediaStreamSelection = {
export type SynchronizedMedia = {
readonly contract: "svml.synchronized-media@1";
readonly timeline: {
- readonly spanAuthority: "video" | "audio";
readonly frameRate: MediaRational;
readonly frameCount: number;
- readonly sampleRate: 48_000;
- readonly sampleFrames: number;
- };
- readonly sourceMap: {
- readonly sourceOriginPts: MediaTimestamp;
- readonly sourceEndPts: MediaTimestamp;
- readonly audioTrimStartSamples: number;
- readonly audioTrimEndSamples: number;
- readonly audioHeadSamples: number;
- readonly audioContentSamples: number;
- readonly audioTailSamples: number;
};
readonly visual?: {
readonly artifact: BlobRef;
- readonly sourceStreamIndex: number;
readonly width: number;
readonly height: number;
- readonly frameRate: MediaRational;
- readonly frameCount: number;
- readonly muted: true;
};
readonly audio?: {
readonly artifact: BlobRef;
- readonly sourceStreamIndex: number;
- readonly codec: "pcm_s16le";
- readonly sampleRate: 48_000;
- readonly channels: 2;
- readonly sampleFrames: number;
- /** Media normalization preserves level. Loudness/mastering is a separate author policy. */
- readonly loudness: "preserved";
};
};
diff --git a/packages/provider-media-local/test/provider.test.ts b/packages/provider-media-local/test/provider.test.ts
index bca885a9..89adc7c3 100644
--- a/packages/provider-media-local/test/provider.test.ts
+++ b/packages/provider-media-local/test/provider.test.ts
@@ -4,7 +4,7 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
import { artifactTypes } from "@narratage/artifact";
-import { mediaTypes, sealRenderedVisual, verifyMediaInspection, verifyMuxedMedia, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";
+import { mediaTypes, sealRenderedVisual, synchronizedMediaSampleFrames, verifyMediaInspection, verifyMuxedMedia, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";
import type { MediaAudioStream, MediaInspection, MuxedMedia, SynchronizedMedia, TimelineAudio } from "@narratage/media";
import { assertSpeechEvidenceAudioIdentity, speechTypes } from "@narratage/speech";
import type { SpeechEvidenceAudio } from "@narratage/speech";
@@ -70,6 +70,47 @@ function rampWav(sampleFrames: number): Buffer {
return bytes;
}
+function pcm16StereoAudibleSpan(bytes: Uint8Array): {
+ readonly firstSample: number;
+ readonly endSampleExclusive: number;
+ readonly sampleFrames: number;
+} {
+ const wav = Buffer.from(bytes);
+ assert.equal(wav.toString("ascii", 0, 4), "RIFF");
+ assert.equal(wav.toString("ascii", 8, 12), "WAVE");
+ let dataStart = -1;
+ let dataBytes = 0;
+ for (let offset = 12; offset + 8 <= wav.length;) {
+ const id = wav.toString("ascii", offset, offset + 4);
+ const size = wav.readUInt32LE(offset + 4);
+ if (id === "fmt ") {
+ assert.equal(wav.readUInt16LE(offset + 8), 1);
+ assert.equal(wav.readUInt16LE(offset + 10), 2);
+ assert.equal(wav.readUInt32LE(offset + 12), 48_000);
+ assert.equal(wav.readUInt16LE(offset + 22), 16);
+ }
+ if (id === "data") {
+ dataStart = offset + 8;
+ dataBytes = size;
+ break;
+ }
+ offset += 8 + size + (size % 2);
+ }
+ assert(dataStart >= 0 && dataStart + dataBytes <= wav.length, "normalized WAV has no complete data chunk");
+ const sampleFrames = Math.floor(dataBytes / 4);
+ let firstSample = sampleFrames;
+ let endSampleExclusive = 0;
+ for (let frame = 0; frame < sampleFrames; frame += 1) {
+ const offset = dataStart + frame * 4;
+ if (Math.max(Math.abs(wav.readInt16LE(offset)), Math.abs(wav.readInt16LE(offset + 2))) > 64) {
+ firstSample = Math.min(firstSample, frame);
+ endSampleExclusive = frame + 1;
+ }
+ }
+ assert(firstSample < endSampleExclusive, "normalized WAV contains no audible samples");
+ return { firstSample, endSampleExclusive, sampleFrames };
+}
+
function presentationSampleFrames(stream: MediaAudioStream): number {
assert(stream.startPts !== undefined && stream.endPts !== undefined);
const startNumerator = BigInt(stream.startPts.ticks) * BigInt(stream.startPts.timeBase.numerator);
@@ -223,16 +264,7 @@ test("local media Provider enumerates attached pictures and jointly normalizes 3
const typed = await normalizeArtifact({ artifacts, source, inspection: typedInspection,
selection, frameRate: selectionRequest.frameRate });
assert.equal(typed.timeline.frameCount, 30);
- assert.equal(typed.timeline.sampleFrames, 48_000);
- assert.equal(typed.visual?.sourceStreamIndex, 0);
- assert.equal(typed.visual?.muted, true);
- assert.equal(typed.audio?.sourceStreamIndex, 1);
- assert.equal(typed.audio?.sampleRate, 48_000);
- assert.equal(typed.audio?.channels, 2);
- assert.equal(typed.sourceMap.audioHeadSamples + typed.sourceMap.audioContentSamples
- + typed.sourceMap.audioTailSamples, typed.timeline.sampleFrames);
- assert.ok(typed.sourceMap.audioTrimEndSamples > 0, "AAC tail beyond the final picture must be trimmed");
- assert.equal(typed.sourceMap.audioHeadSamples, 0);
+ assert.equal(synchronizedMediaSampleFrames(typed), 48_000);
assert.equal("basisDigest" in typed, false);
assert.equal("narrativeDigest" in typed, false);
assert.equal(await artifacts.has(typed.visual!.artifact.digest), true);
@@ -265,6 +297,11 @@ test("local media Provider enumerates attached pictures and jointly normalizes 3
};
assert.deepEqual(outputProbe.streams.map((stream) => stream.codec_type), ["video"]);
assert.equal(outputProbe.streams[0]?.nb_frames, "30");
+ const audioOutput = await inspectArtifact(artifacts, typed.audio!.artifact);
+ const normalizedAudio = audioOutput.streams.find((stream) => stream.kind === "audio");
+ assert.equal(normalizedAudio?.kind === "audio" && normalizedAudio.sampleRate, 48_000);
+ assert.equal(normalizedAudio?.kind === "audio" && normalizedAudio.channels, 2);
+ assert.equal(normalizedAudio?.kind === "audio" && normalizedAudio.decodedSampleFrames, 48_000);
} finally {
await rm(root, { recursive: true, force: true });
}
@@ -335,7 +372,6 @@ test("animated WebP keeps its authored frame timing before fixed-rate normalizat
const selection = selectMediaStreams(inspection, request);
const normalized = await normalizeArtifact({ artifacts, source, inspection, selection, frameRate: request.frameRate });
assert.equal(normalized.timeline.frameCount, 10);
- assert.equal(normalized.visual?.frameCount, 10);
assert.equal(normalized.visual?.width, 64);
assert.equal(normalized.visual?.height, 48);
const output = await inspectArtifact(artifacts, normalized.visual!.artifact);
@@ -469,14 +505,14 @@ test("local media Provider preserves one source A/V origin when audio starts lat
const expectedEnd = Math.round((seconds(audio.endPts) - seconds(video.startPts)) * 48_000);
const expectedTail = 48_000 - Math.min(48_000, expectedEnd);
assert.equal(normalized.timeline.frameCount, 30);
- assert.equal(normalized.timeline.sampleFrames, 48_000);
- assert.equal(normalized.sourceMap.audioHeadSamples, expectedHead,
- "normalized audio head must preserve the measured stream-start offset");
- assert.equal(normalized.sourceMap.audioTailSamples, expectedTail,
- "normalized audio tail must preserve the measured stream end");
- assert.ok(normalized.sourceMap.audioHeadSamples > 9_000, "fixture must retain a substantial audio lag");
- assert.equal(normalized.sourceMap.audioTrimStartSamples, 0);
- assert.equal(normalized.sourceMap.audioTrimEndSamples, 0);
+ const bytes = await artifacts.get(normalized.audio!.artifact.digest);
+ assert(bytes !== undefined);
+ const audible = pcm16StereoAudibleSpan(bytes);
+ assert.equal(audible.sampleFrames, 48_000);
+ assert.ok(Math.abs(audible.firstSample - expectedHead) <= 64,
+ `normalized audio begins at ${audible.firstSample}, expected ${expectedHead}`);
+ assert.ok(Math.abs(audible.endSampleExclusive - (48_000 - expectedTail)) <= 64,
+ `normalized audio ends at ${audible.endSampleExclusive}, expected ${48_000 - expectedTail}`);
} finally {
await rm(root, { recursive: true, force: true });
}
@@ -507,13 +543,6 @@ test("a silent generated MP4 remains a visual-only product and cannot satisfy a
assert.equal(normalized.timeline.frameCount, 15);
assert.ok(normalized.visual);
assert.equal(normalized.audio, undefined);
- assert.deepEqual({
- trimStart: normalized.sourceMap.audioTrimStartSamples,
- trimEnd: normalized.sourceMap.audioTrimEndSamples,
- head: normalized.sourceMap.audioHeadSamples,
- content: normalized.sourceMap.audioContentSamples,
- tail: normalized.sourceMap.audioTailSamples,
- }, { trimStart: 0, trimEnd: 0, head: 0, content: 0, tail: 0 });
const invalid = sealMediaSelectionRequest({
contract: "svml.media-selection-request@1",
diff --git a/packages/ranking/src/render.ts b/packages/ranking/src/render.ts
index 40d06037..a02296cf 100644
--- a/packages/ranking/src/render.ts
+++ b/packages/ranking/src/render.ts
@@ -17,7 +17,7 @@ import type {
VisualTextTypography,
VisualTrack,
} from "@narratage/composition";
-import { verifySynchronizedMedia } from "@narratage/media";
+import { synchronizedMediaSampleFrames, verifySynchronizedMedia } from "@narratage/media";
import {
assertProgramSpaceIdentity,
programFrameSampleBoundary,
@@ -684,14 +684,15 @@ export function renderRankingAudio(
const audio = media.audio;
assert(audio !== undefined, `Ranking ${event.kind} sound has no normalized audio.`);
const startSample = programFrameSampleBoundary(space, event.frame, 48_000);
- const length = Math.min(audio.sampleFrames, totalSamples - startSample);
+ const sourceSampleFrames = synchronizedMediaSampleFrames(media);
+ const length = Math.min(sourceSampleFrames, totalSamples - startSample);
assert(length > 0, `Ranking sound ${event.id} starts after ProgramSpace.`);
assert(fadeSamples <= length, `Ranking sound ${event.id} fade exceeds its audible interval.`);
clips.push({
id: event.id,
artifact: structuredClone(audio.artifact),
target: { startSample, endSampleExclusive: startSample + length },
- source: { sampleFrames: audio.sampleFrames, startSample: 0, endSampleExclusive: length, loop: false, phaseSample: 0 },
+ source: { sampleFrames: sourceSampleFrames, startSample: 0, endSampleExclusive: length, loop: false, phaseSample: 0 },
playbackRate: 1, pitch: "preserve",
gain: event.kind === "appear" ? style.appearGain : style.moveGain,
fadeInSamples: fadeSamples, fadeOutSamples: 0,
diff --git a/packages/ranking/test/ranking.test.ts b/packages/ranking/test/ranking.test.ts
index 3eb28e46..716f9c99 100644
--- a/packages/ranking/test/ranking.test.ts
+++ b/packages/ranking/test/ranking.test.ts
@@ -284,19 +284,11 @@ test("Typewriter uses Unicode graphemes, explicit emphasis and winner timing wit
}, set));
});
-const sound = (id: string, sampleFrames = 4_800): SynchronizedMedia => ({
+const sound = (id: string): SynchronizedMedia => ({
contract: "svml.synchronized-media@1",
- timeline: { spanAuthority: "audio", frameRate: { numerator: 30, denominator: 1 }, frameCount: 3,
- sampleRate: 48_000, sampleFrames },
- sourceMap: {
- sourceOriginPts: { ticks: "0", timeBase: { numerator: 1, denominator: 48_000 } },
- sourceEndPts: { ticks: String(sampleFrames), timeBase: { numerator: 1, denominator: 48_000 } },
- audioTrimStartSamples: 0, audioTrimEndSamples: 0, audioHeadSamples: 0,
- audioContentSamples: sampleFrames, audioTailSamples: 0,
- },
+ timeline: { frameRate: { numerator: 30, denominator: 1 }, frameCount: 3 },
audio: {
artifact: { kind: "blob", digest: digestOf(`ranking-sound:${id}`), size: 128, mediaType: "audio/wav" },
- sourceStreamIndex: 0, codec: "pcm_s16le", sampleRate: 48_000, channels: 2, sampleFrames, loudness: "preserved",
},
});
diff --git a/packages/semantic-map/src/index.ts b/packages/semantic-map/src/index.ts
index f01755d2..bb2dadd1 100644
--- a/packages/semantic-map/src/index.ts
+++ b/packages/semantic-map/src/index.ts
@@ -3,6 +3,7 @@ export {
assertNarrativeMomentIdentity,
assertNarrativeSelectionIdentity,
momentFrames,
+ segmentFrameSpan,
selectionFrameSpans,
tokenSpanSeconds,
} from "./location.js";
diff --git a/packages/semantic-map/src/location.ts b/packages/semantic-map/src/location.ts
index c100bd9a..b78a6aa3 100644
--- a/packages/semantic-map/src/location.ts
+++ b/packages/semantic-map/src/location.ts
@@ -1,4 +1,4 @@
-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
import { assertProgramSpaceIdentity } from "@narratage/program-space";
import type { ProgramSpace } from "@narratage/program-space";
import type { CompleteSemanticMap } from "./types.js";
@@ -77,6 +77,24 @@ export function selectionFrameSpans(
});
}
+/** Locate one authored Segment by its two structural anchors. */
+export function segmentFrameSpan(
+ map: CompleteSemanticMap,
+ segment: NarrativeExcerpt,
+ programSpace: ProgramSpace,
+): LocatedFrameSpan {
+ assertCompleteSemanticMapIdentity(map);
+ assertProgramSpaceIdentity(programSpace);
+ if (segment.contract !== "svml.narrative-excerpt@1" || segment.kind !== "segment" || segment.id.length === 0) {
+ throw new Error("Narrative Segment excerpt is invalid");
+ }
+ const frames = anchorFrames(map);
+ return {
+ startFrame: frameFor(frames, `segment:${segment.id}:start`, `Narrative Segment ${segment.id}`),
+ endFrameExclusive: frameFor(frames, `segment:${segment.id}:end`, `Narrative Segment ${segment.id}`),
+ };
+}
+
/**
* The measured window a run of Script tokens occupies: the first token's start
* and the last token's end, exactly as located. Undefined when any named token
diff --git a/packages/semantic-map/test/location.test.ts b/packages/semantic-map/test/location.test.ts
index aada1467..10427b03 100644
--- a/packages/semantic-map/test/location.test.ts
+++ b/packages/semantic-map/test/location.test.ts
@@ -1,7 +1,7 @@
import assert from "node:assert/strict";
import test from "node:test";
-import { momentFrames, selectionFrameSpans } from "@narratage/semantic-map";
+import { momentFrames, segmentFrameSpan, selectionFrameSpans } from "@narratage/semantic-map";
import type { CompleteSemanticMap } from "@narratage/semantic-map";
import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
import type { ProgramSpace } from "@narratage/program-space";
@@ -86,6 +86,12 @@ test("a Segment cut is an ordinary anchor, so a Selection may start at one", ()
);
});
+test("a whole Segment is exactly its two structural anchors", () => {
+ assert.deepEqual(segmentFrameSpan(map, {
+ contract: "svml.narrative-excerpt@1", kind: "segment", id: "second", tokenStart: 2, tokenEndExclusive: 4,
+ }, space), { startFrame: 5, endFrameExclusive: 9 });
+});
+
test("a Moment locates one instant per occurrence", () => {
const moment = {
contract: "svml.narrative-moment@1",
diff --git a/packages/spatial/src/author.ts b/packages/spatial/src/author.ts
new file mode 100644
index 00000000..e4d4d7b1
--- /dev/null
+++ b/packages/spatial/src/author.ts
@@ -0,0 +1,60 @@
+import type { CanonicalValue } from "@narratage/protocol";
+
+import { sealContentFit } from "./geometry.js";
+import type { ContentFit } from "./types.js";
+
+export const contentFitPropertyNames = [
+ "fit", "frame-x", "frame-y", "content-x", "content-y",
+ "fit-offset-x", "fit-offset-y", "fit-constraint",
+] as const;
+
+function number(
+ properties: Readonly<Record<string, CanonicalValue>>,
+ name: string,
+ fallback: number,
+ owner: string,
+): number {
+ const value = properties[name];
+ if (value === undefined) return fallback;
+ if (typeof value !== "number" || !Number.isFinite(value)) throw new Error(`${owner} ${name} must be a finite number.`);
+ return value;
+}
+
+function oneOf<T extends string>(
+ properties: Readonly<Record<string, CanonicalValue>>,
+ name: string,
+ allowed: readonly T[],
+ fallback: T,
+ owner: string,
+): T {
+ const value = properties[name];
+ if (value === undefined) return fallback;
+ if (typeof value !== "string" || !allowed.includes(value as T)) {
+ throw new Error(`${owner} ${name} must be ${allowed.join(" | ")}.`);
+ }
+ return value as T;
+}
+
+/** Decode the shared spatial-fit vocabulary from any author-side property map. */
+export function decodeContentFitProperties(
+ properties: Readonly<Record<string, CanonicalValue>>,
+ owner = "Content-fit properties",
+): ContentFit {
+ return sealContentFit({
+ contract: "svml.content-fit@1",
+ sizing: oneOf(properties, "fit", ["contain", "cover", "fit-width", "fit-height", "native", "scale-down", "stretch"] as const, "contain", owner),
+ framePoint: {
+ x: number(properties, "frame-x", 0.5, owner),
+ y: number(properties, "frame-y", 0.5, owner),
+ },
+ contentPoint: {
+ x: number(properties, "content-x", 0.5, owner),
+ y: number(properties, "content-y", 0.5, owner),
+ },
+ offsetPx: {
+ x: number(properties, "fit-offset-x", 0, owner),
+ y: number(properties, "fit-offset-y", 0, owner),
+ },
+ constraint: oneOf(properties, "fit-constraint", ["bounded", "free"] as const, "bounded", owner),
+ });
+}
diff --git a/packages/spatial/src/index.ts b/packages/spatial/src/index.ts
index 6d982619..5fdf6dc3 100644
--- a/packages/spatial/src/index.ts
+++ b/packages/spatial/src/index.ts
@@ -1,4 +1,5 @@
export { spatialComponent } from "./component.js";
+export { contentFitPropertyNames, decodeContentFitProperties } from "./author.js";
export {
anchoredFrameFragment,
aspectFrameFragment,
diff --git a/packages/speech-alignment/test/speech-align.test.ts b/packages/speech-alignment/test/speech-align.test.ts
index cc132c47..292136fa 100644
--- a/packages/speech-alignment/test/speech-align.test.ts
+++ b/packages/speech-alignment/test/speech-align.test.ts
@@ -76,20 +76,7 @@ function speechBasis(
contract: "svml.speech-basis@1",
programSpace,
audio: { kind: "blob", digest: audioDigest, size: 1, mediaType: "audio/wav" },
- visualTrack: {
- clips: segments.map((segment) => ({
- segmentId: segment.segmentId,
- artifact: {
- kind: "blob",
- digest: digestOf(`fixture:clip:${segment.segmentId}`),
- size: 1,
- mediaType: "video/mp4",
- },
- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },
- frameRate: { ...programSpace.frameRate },
- frameCount: Math.round((segment.endSec - segment.startSec) * 1_000),
- })),
- },
+ visualTrack: { clips: [] },
segments,
});
return audioProjection(take);
diff --git a/packages/speech-basis/package.json b/packages/speech-basis/package.json
index ff66222d..e38fea46 100644
--- a/packages/speech-basis/package.json
+++ b/packages/speech-basis/package.json
@@ -16,8 +16,7 @@
"@narratage/media-track": "workspace:*",
"@narratage/program-space": "workspace:*",
"@narratage/protocol": "workspace:*",
- "@narratage/speech": "workspace:*",
- "@narratage/spatial": "workspace:*"
+ "@narratage/speech": "workspace:*"
},
"devDependencies": {
"@narratage/core": "workspace:*",
diff --git a/packages/speech-basis/src/component.ts b/packages/speech-basis/src/component.ts
index 0d8712af..17464667 100644
--- a/packages/speech-basis/src/component.ts
+++ b/packages/speech-basis/src/component.ts
@@ -2,7 +2,6 @@ import type { ComponentPackage } from "@narratage/component-kit";
import type { SpeechBasis } from "@narratage/speech";
import type { StoredValue } from "@narratage/protocol";
import { canonicalize } from "@narratage/protocol";
-import type { CanvasSpace } from "@narratage/spatial";
import {
speechBasisProducers,
@@ -23,11 +22,6 @@ function speechBasis(value: StoredValue | undefined): SpeechBasis {
return value.value as unknown as SpeechBasis;
}
-function canvas(value: StoredValue | undefined): CanvasSpace {
- if (value?.kind !== "inline") throw new Error("CanvasSpace must be inline");
- return value.value as unknown as CanvasSpace;
-}
-
/** Pure projections of one atomic SpeechBasis; no Artifact read or external capability is hidden here. */
export const speechBasisComponent = {
name: "@narratage/speech-basis",
@@ -65,7 +59,7 @@ export const speechBasisComponent = {
outputs: {
visual: {
kind: "inline",
- value: canonicalize(projectSpeechVisual(speechBasis(inputs.basis?.value), canvas(inputs.canvas?.value))),
+ value: canonicalize(projectSpeechVisual(speechBasis(inputs.basis?.value))),
},
},
needs: {},
diff --git a/packages/speech-basis/src/fragment.ts b/packages/speech-basis/src/fragment.ts
index d1c75968..582b9848 100644
--- a/packages/speech-basis/src/fragment.ts
+++ b/packages/speech-basis/src/fragment.ts
@@ -3,7 +3,6 @@ import { speechTypes } from "@narratage/speech";
import type { SpeechBasis } from "@narratage/speech";
import { compositionTypes } from "@narratage/composition";
import { sealGraphFragment } from "@narratage/elaborator";
-import { spatialTypes } from "@narratage/spatial";
import { speechBasisProducers } from "./manifest.js";
@@ -13,7 +12,7 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operat
/** Project the independently demandable facts carried by one SpeechBasis. */
export const speechBasisProjectionFragment = sealGraphFragment({
name: "@narratage/speech-basis/projections@1",
- inputs: [{ name: "basis", type: speechTypes.basis }, { name: "canvas", type: spatialTypes.canvas }],
+ inputs: [{ name: "basis", type: speechTypes.basis }],
operations: [
{
id: "project-program-space",
@@ -30,7 +29,7 @@ export const speechBasisProjectionFragment = sealGraphFragment({
{
id: "project-visual",
producer: speechBasisProducers.projectVisual,
- inputs: { basis: input("basis"), canvas: input("canvas") },
+ inputs: { basis: input("basis") },
result: { kind: "output", name: "visual" },
},
{
@@ -59,7 +58,7 @@ export const speechBasisProjectionFragment = sealGraphFragment({
name: "visual",
type: compositionTypes.visualTrack,
root: operation("project-visual"),
- semanticInputs: ["basis", "canvas"],
+ semanticInputs: ["basis"],
fidelity: "exact",
},
{
diff --git a/packages/speech-basis/src/manifest.ts b/packages/speech-basis/src/manifest.ts
index a3f4af9b..511710b2 100644
--- a/packages/speech-basis/src/manifest.ts
+++ b/packages/speech-basis/src/manifest.ts
@@ -1,7 +1,6 @@
import { programSpaceDependency, programSpaceTypes } from "@narratage/program-space";
import { speechDependency, speechTypes } from "@narratage/speech";
import { compositionDependency, compositionTypes } from "@narratage/composition";
-import { spatialDependency, spatialTypes } from "@narratage/spatial";
import { digestOf } from "@narratage/protocol";
import type { ModuleManifest, ProducerRef } from "@narratage/protocol";
@@ -29,7 +28,6 @@ export const speechBasisManifest: ModuleManifest = {
speechDependency,
programSpaceDependency,
compositionDependency,
- spatialDependency,
],
types: [],
capabilities: [],
@@ -59,7 +57,7 @@ export const speechBasisManifest: ModuleManifest = {
},
{
name: speechBasisProducers.projectVisual.name,
- inputs: [{ name: "basis", type: speechTypes.basis }, { name: "canvas", type: spatialTypes.canvas }],
+ inputs: [{ name: "basis", type: speechTypes.basis }],
outputs: [{ name: "visual", type: compositionTypes.visualTrack }],
needs: [],
implementation: {
diff --git a/packages/speech-basis/src/projection.ts b/packages/speech-basis/src/projection.ts
index d297738b..3369350b 100644
--- a/packages/speech-basis/src/projection.ts
+++ b/packages/speech-basis/src/projection.ts
@@ -1,4 +1,4 @@
-import { programSpaceFrameCount, programSpaceSampleFrames } from "@narratage/program-space";
+import { programSpaceSampleFrames } from "@narratage/program-space";
import type { ProgramSpace } from "@narratage/program-space";
import { assertSpeechBasisIdentity } from "@narratage/speech";
import type { SpeechAudioBasis, SpeechBasis } from "@narratage/speech";
@@ -6,7 +6,6 @@ import { lowerRestrictedSpeechVisualPresents } from "@narratage/media-track";
import { sealAudioTrack, sealVisualTrack } from "@narratage/composition";
import type { AudioTrack, VisualTrack } from "@narratage/composition";
import { digestOf } from "@narratage/protocol";
-import type { CanvasSpace } from "@narratage/spatial";
export const projectSpeechAudioImplementationDigest = digestOf(
"@narratage/speech-basis/project-audio@1",
@@ -39,14 +38,9 @@ export function projectSpeechAudio(basis: SpeechBasis): SpeechAudioBasis {
};
}
-function frameAt(basis: SpeechBasis, seconds: number): number {
- return Math.round(seconds * basis.programSpace.frameRate.numerator / basis.programSpace.frameRate.denominator);
-}
-
-export function projectSpeechVisual(basis: SpeechBasis, canvas: CanvasSpace): VisualTrack {
+export function projectSpeechVisual(basis: SpeechBasis): VisualTrack {
assertSpeechBasisIdentity(basis);
const trackId = `speech-visual:${basis.segments.map((segment) => segment.segmentId).join("+")}`;
- const segments = new Map(basis.segments.map((segment) => [segment.segmentId, segment]));
return sealVisualTrack({
contract: "svml.visual-track@1",
visualIr: "svml.visual-ir@1",
@@ -54,26 +48,17 @@ export function projectSpeechVisual(basis: SpeechBasis, canvas: CanvasSpace): Vi
presents: lowerRestrictedSpeechVisualPresents(
trackId,
basis.programSpace,
- canvas,
- basis.visualTrack.clips.map((clip) => {
- const segment = segments.get(clip.segmentId)!;
- return {
- id: clip.segmentId,
- span: { startFrame: frameAt(basis, segment.startSec), endFrameExclusive: frameAt(basis, segment.endSec) },
- artifact: clip.artifact,
- extent: clip.extent,
- frameRate: clip.frameRate,
- frameCount: clip.frameCount,
- };
- }),
- {
- contract: "svml.content-fit@1",
- sizing: "cover",
- framePoint: { x: 0.5, y: 0.5 },
- contentPoint: { x: 0.5, y: 0.5 },
- offsetPx: { x: 0, y: 0 },
- constraint: "bounded",
- },
+ basis.visualTrack.clips.map((clip) => ({
+ id: clip.segmentId,
+ span: structuredClone(clip.span),
+ artifact: clip.artifact,
+ extent: clip.extent,
+ frameRate: clip.frameRate,
+ frameCount: clip.frameCount,
+ frame: clip.frame,
+ fit: clip.fit,
+ stackingOrder: clip.stackingOrder,
+ })),
),
});
}
diff --git a/packages/speech-basis/test/product.test.ts b/packages/speech-basis/test/product.test.ts
index 07d70d16..e6ea7721 100644
--- a/packages/speech-basis/test/product.test.ts
+++ b/packages/speech-basis/test/product.test.ts
@@ -42,20 +42,8 @@ import {
speechBasisProducers,
} from "@narratage/speech-basis";
import { TypeValidatorRegistry } from "@narratage/validation";
-import { spatialTypes } from "@narratage/spatial";
-import { spatialValidatorDigests } from "@narratage/spatial";
import { mediaTrackManifest } from "@narratage/media-track";
-const canvas = {
- contract: "svml.canvas-space@1" as const,
- widthPx: 720,
- heightPx: 1280,
- origin: "top-left" as const,
- xDirection: "right" as const,
- yDirection: "down" as const,
- pixelAspect: "square" as const,
-};
-
const testModule = { name: "example.speech-basis-product", version: "0.0.0" } as const;
const requestType = { module: testModule, name: "SpeechRequest" } satisfies TypeRef;
const generateProducer = { module: testModule, name: "generate-speech-basis" } satisfies ProducerRef;
@@ -123,10 +111,18 @@ function sampleTake(label = "generated"): SpeechBasis {
visualTrack: {
clips: [{
segmentId: "opening",
+ span: { startFrame: 0, endFrameExclusive: 60 },
artifact: visual,
extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },
frameRate: { ...programSpace.frameRate },
frameCount: 60,
+ frame: { contract: "svml.spatial-frame@1", xPx: 0, yPx: 0, widthPx: 720, heightPx: 1280 },
+ fit: {
+ contract: "svml.content-fit@1", sizing: "cover",
+ framePoint: { x: 0.5, y: 0.5 }, contentPoint: { x: 0.5, y: 0.5 },
+ offsetPx: { x: 0, y: 0 }, constraint: "bounded",
+ },
+ stackingOrder: 0,
}],
},
segments: [{
@@ -149,29 +145,10 @@ function createProgram(): LinkedProgram {
frontendClosureDigest: digestOf("frontend:speech-basis-product"),
},
});
- const rawCanvasRecord = sealRecord({
- id: "canvas:vertical",
- type: spatialTypes.canvas,
- value: { kind: "inline", value: canvas },
- conformance: "exact",
- origin: {
- kind: "authored",
- sourceDigest: digestOf("source:speech-basis-product"),
- frontendClosureDigest: digestOf("frontend:speech-basis-product"),
- },
- });
- const canvasRecord = {
- ...rawCanvasRecord,
- validation: sealTypeValidationReceipt({
- type: rawCanvasRecord.type,
- recordDigest: rawCanvasRecord.digest,
- validatorDigest: spatialValidatorDigests.canvas,
- }),
- };
return link(closure, [sealTypedModule({
id: "author:speech-basis-product",
closureDigest: closure.digest,
- records: [request, canvasRecord],
+ records: [request],
})]);
}
@@ -180,7 +157,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {
const output = (id: string) => ({ kind: "logical-output" as const, id });
const operation = (id: string) => ({ kind: "operation-result" as const, operation: id });
const existingTake = sampleTake("approved");
- const existingVisual = projectSpeechVisual(existingTake, canvas);
+ const existingVisual = projectSpeechVisual(existingTake);
const existingVisualValue = { kind: "inline" as const, value: existingVisual };
const existingVisualValidation = sealTypeValidationReceipt({
type: compositionTypes.visualTrack,
@@ -206,7 +183,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {
id: "opening.visual",
type: compositionTypes.visualTrack,
primary: "opening.visual.project",
- semanticInputs: [output("opening.take"), record("canvas:vertical")],
+ semanticInputs: [output("opening.take")],
},
],
candidates: [
@@ -267,7 +244,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {
{
id: "project-opening-visual",
producer: speechBasisProducers.projectVisual,
- inputs: { basis: output("opening.take"), canvas: record("canvas:vertical") },
+ inputs: { basis: output("opening.take") },
result: { kind: "output", name: "visual", record: "visual:opening" },
},
],
@@ -316,7 +293,7 @@ test("SpeechBasis is one Product and audio/visual are ordinary shared projection
test("SpeechBasis projects to peer generic visual and audio Tracks", () => {
const take = sampleTake();
- const visual = projectSpeechVisual(take, canvas);
+ const visual = projectSpeechVisual(take);
const audio = projectSpeechAudioTrack(take);
const programSpace = projectSpeechProgramSpace(take);
assert.equal(visual.contract, "svml.visual-track@1");
diff --git a/packages/speech-spine/README.md b/packages/speech-spine/README.md
index 8444325f..a63adf41 100644
--- a/packages/speech-spine/README.md
+++ b/packages/speech-spine/README.md
@@ -10,13 +10,20 @@ auditable reusable graph contributions:
- WhisperX evidence plus the provider-neutral speech SemanticMap;
- timed caption projection.
-The package also owns `<speech:Spine>`. The Spine explicitly declares its `frame-rate`; each
+The package also owns `<speech:Spine>`. The Spine explicitly declares its `frame-rate` and its
+restricted visual base (`visual-frame`, `visual-appearance`, `visual-z`); each
`<speech:Take>` connects one `NarrativeExcerpt` and exactly one of:
- `video={...}` — an ordinary generated video Blob. The Surface expands inspection, stream
selection and A/V normalization before assembly;
+- `audio={...}` — an ordinary voice Blob. The Surface expands audio-authoritative normalization;
+ this Take contributes no visual clip and never invents a black frame;
- `media={...}` — an already prepared `SynchronizedMedia` value, connected directly.
+Visual Takes may override the Spine base with `frame`, `appearance` and `z`. That is the complete
+visual authority of Speech Spine: one same-source visual clip may be placed and stacked. Motion,
+transitions, sequences and independent pictures remain ordinary Media Tracks.
+
The Surface then expands an immutable append fold, one timeline-audio render and ordinary
SpeechBasis projections. Whitespace and child layout do not create ports, and Core receives no
variadic node, media-normalization rule or Speech special case.
diff --git a/packages/speech-spine/package.json b/packages/speech-spine/package.json
index 161ce210..47ae7f25 100644
--- a/packages/speech-spine/package.json
+++ b/packages/speech-spine/package.json
@@ -22,6 +22,7 @@
"@narratage/speech": "workspace:*",
"@narratage/speech-basis": "workspace:*",
"@narratage/spatial": "workspace:*",
+ "@narratage/svs": "workspace:*",
"@narratage/markup": "workspace:*"
},
"devDependencies": {
diff --git a/packages/speech-spine/src/component.ts b/packages/speech-spine/src/component.ts
index f52df1fa..db739958 100644
--- a/packages/speech-spine/src/component.ts
+++ b/packages/speech-spine/src/component.ts
@@ -3,11 +3,14 @@ import type { NarrativeExcerpt } from "@narratage/narrative";
import type { SynchronizedMedia, TimelineAudio } from "@narratage/media";
import type { StoredValue } from "@narratage/protocol";
import { canonicalize } from "@narratage/protocol";
+import type { ContentFit, SpatialFrame } from "@narratage/spatial";
import { speechSpineProducers } from "./manifest.js";
import {
- appendSpeechSpineTake,
- appendSpeechSpineTakeImplementationDigest,
+ appendSpeechSpineAudioTake,
+ appendSpeechSpineAudioTakeImplementationDigest,
+ appendSpeechSpineVisualTake,
+ appendSpeechSpineVisualTakeImplementationDigest,
assembleSpeechBasis,
assembleSpeechBasisImplementationDigest,
compileSpeechSpineAudio,
@@ -15,7 +18,7 @@ import {
createSpeechSpineSet,
createSpeechSpineSetImplementationDigest,
} from "./program.js";
-import type { SpeechSpineProgram, SpeechSpineSet } from "./types.js";
+import type { SpeechSpineProgram, SpeechSpineSet, SpeechSpineVisualSpec } from "./types.js";
function inline<T>(value: StoredValue | undefined, subject: string): T {
if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);
@@ -34,10 +37,10 @@ export const speechSpineComponent = {
}),
},
{
- producer: speechSpineProducers.appendTake,
- implementationDigest: appendSpeechSpineTakeImplementationDigest,
+ producer: speechSpineProducers.appendAudioTake,
+ implementationDigest: appendSpeechSpineAudioTakeImplementationDigest,
handler: ({ inputs }) => ({
- outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineTake(
+ outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineAudioTake(
inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),
inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"),
inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),
@@ -46,6 +49,22 @@ export const speechSpineComponent = {
needs: {},
}),
},
+ {
+ producer: speechSpineProducers.appendVisualTake,
+ implementationDigest: appendSpeechSpineVisualTakeImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineVisualTake(
+ inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),
+ inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"),
+ inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),
+ inline<NarrativeExcerpt>(inputs.segment?.value, "NarrativeExcerpt"),
+ inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),
+ inline<ContentFit>(inputs.fit?.value, "ContentFit"),
+ inline<SpeechSpineVisualSpec>(inputs.visualSpec?.value, "SpeechSpineVisualSpec"),
+ )) } },
+ needs: {},
+ }),
+ },
{
producer: speechSpineProducers.compileAudio,
implementationDigest: compileSpeechSpineAudioImplementationDigest,
diff --git a/packages/speech-spine/src/fragment.ts b/packages/speech-spine/src/fragment.ts
index 9ed61e30..1b8f3d7b 100644
--- a/packages/speech-spine/src/fragment.ts
+++ b/packages/speech-spine/src/fragment.ts
@@ -8,6 +8,7 @@ import type { FragmentOperation } from "@narratage/elaborator";
import { mediaPipelineProducers } from "@narratage/media-pipeline";
import { speechBasisProducers } from "@narratage/speech-basis";
import { spatialTypes } from "@narratage/spatial";
+import type { TypeRef } from "@narratage/protocol";
import { speechSpineProducers, speechSpineTypes } from "./manifest.js";
import type { SpeechSpineFragmentOptions } from "./types.js";
@@ -17,13 +18,25 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operat
export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {
if (options.takes.length === 0) throw new Error("Speech Spine requires at least one Take");
- const names = new Set(["program", "canvas"]);
+ const declaredInputs = new Map<string, TypeRef>();
+ const addInput = (name: string, type: TypeRef): void => {
+ if (!name) throw new Error("Speech Spine Fragment input names must not be empty");
+ const previous = declaredInputs.get(name);
+ if (previous !== undefined && (previous.module.name !== type.module.name
+ || previous.module.version !== type.module.version || previous.name !== type.name)) {
+ throw new Error(`Speech Spine Fragment input ${name} is reused with another Type`);
+ }
+ declaredInputs.set(name, type);
+ };
+ addInput("program", speechSpineTypes.spineProgram);
for (const take of options.takes) {
- if (!take.mediaName || !take.segmentName || names.has(take.mediaName) || names.has(take.segmentName)) {
- throw new Error("Speech Spine Fragment input names are empty or duplicated");
+ addInput(take.mediaName, mediaTypes.synchronized);
+ addInput(take.segmentName, narrativeTypes.excerpt);
+ if (take.visual !== undefined) {
+ addInput(take.visual.frameName, spatialTypes.frame);
+ addInput(take.visual.fitName, spatialTypes.fit);
+ addInput(take.visual.visualSpecName, speechSpineTypes.visualSpec);
}
- names.add(take.mediaName);
- names.add(take.segmentName);
}
const operations: FragmentOperation[] = [{
id: "spine:set:empty",
@@ -36,12 +49,17 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {
const id = `spine:set:append:${String(index + 1).padStart(4, "0")}`;
operations.push({
id,
- producer: speechSpineProducers.appendTake,
+ producer: take.visual === undefined ? speechSpineProducers.appendAudioTake : speechSpineProducers.appendVisualTake,
inputs: {
set: operation(current),
program: input("program"),
media: input(take.mediaName),
segment: input(take.segmentName),
+ ...(take.visual === undefined ? {} : {
+ frame: input(take.visual.frameName),
+ fit: input(take.visual.fitName),
+ visualSpec: input(take.visual.visualSpecName),
+ }),
},
result: { kind: "output", name: "set" },
});
@@ -81,7 +99,7 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {
{
id: "spine:visual-track",
producer: speechBasisProducers.projectVisual,
- inputs: { basis: operation("spine:basis"), canvas: input("canvas") },
+ inputs: { basis: operation("spine:basis") },
result: { kind: "output", name: "visual" },
},
{
@@ -91,17 +109,14 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {
result: { kind: "output", name: "track" },
},
);
- const semanticInputs = ["program", ...options.takes.flatMap((take) => [take.mediaName, take.segmentName])];
+ const semanticInputs = [...new Set(["program", ...options.takes.flatMap((take) => [
+ take.mediaName, take.segmentName, ...(take.visual === undefined ? [] : [
+ take.visual.frameName, take.visual.fitName, take.visual.visualSpecName,
+ ]),
+ ])])];
return sealGraphFragment({
name: options.name?.trim() || "@narratage/speech-spine/spine@1",
- inputs: [
- { name: "program", type: speechSpineTypes.spineProgram },
- { name: "canvas", type: spatialTypes.canvas },
- ...options.takes.flatMap((take) => [
- { name: take.mediaName, type: mediaTypes.synchronized },
- { name: take.segmentName, type: narrativeTypes.excerpt },
- ]),
- ],
+ inputs: [...declaredInputs.entries()].map(([name, type]) => ({ name, type })),
operations,
exports: [
{ name: "basis", type: speechTypes.basis, root: operation("spine:basis"), semanticInputs, fidelity: "exact" },
@@ -114,7 +129,7 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {
fidelity: "exact",
},
{
- name: "visual", type: compositionTypes.visualTrack, root: operation("spine:visual-track"), semanticInputs: [...semanticInputs, "canvas"],
+ name: "visual", type: compositionTypes.visualTrack, root: operation("spine:visual-track"), semanticInputs,
fidelity: "exact",
},
{
diff --git a/packages/speech-spine/src/index.ts b/packages/speech-spine/src/index.ts
index 5c50b720..ae3fac4a 100644
--- a/packages/speech-spine/src/index.ts
+++ b/packages/speech-spine/src/index.ts
@@ -9,20 +9,25 @@ export {
speechSpineTypes,
speechSpineProgramSchema,
speechSpineSetSchema,
+ speechSpineVisualSpecSchema,
speechSpineSurfaceImplementationDigest,
} from "./manifest.js";
export {
- appendSpeechSpineTake,
- appendSpeechSpineTakeImplementationDigest,
+ appendSpeechSpineAudioTake,
+ appendSpeechSpineAudioTakeImplementationDigest,
+ appendSpeechSpineVisualTake,
+ appendSpeechSpineVisualTakeImplementationDigest,
assembleSpeechBasis,
assembleSpeechBasisImplementationDigest,
assertSpeechSpineProgram,
assertSpeechSpineSet,
+ assertSpeechSpineVisualSpec,
compileSpeechSpineAudio,
compileSpeechSpineAudioImplementationDigest,
createSpeechSpineSet,
createSpeechSpineSetImplementationDigest,
sealSpeechSpineProgram,
+ sealSpeechSpineVisualSpec,
} from "./program.js";
export { decodeSpeechSpineSurface } from "./surface.js";
export type * from "./types.js";
diff --git a/packages/speech-spine/src/manifest.ts b/packages/speech-spine/src/manifest.ts
index 7af92363..72b4a37e 100644
--- a/packages/speech-spine/src/manifest.ts
+++ b/packages/speech-spine/src/manifest.ts
@@ -12,9 +12,17 @@ import {
import { digestOf } from "@narratage/protocol";
import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@narratage/protocol";
import { speechBasisManifest, speechBasisModuleRef } from "@narratage/speech-basis";
+import {
+ contentFitSchema,
+ spatialDependency,
+ spatialFrameSchema,
+ spatialTypes,
+} from "@narratage/spatial";
+import { svsManifest, svsModuleRef } from "@narratage/svs";
import {
- appendSpeechSpineTakeImplementationDigest,
+ appendSpeechSpineAudioTakeImplementationDigest,
+ appendSpeechSpineVisualTakeImplementationDigest,
assembleSpeechBasisImplementationDigest,
compileSpeechSpineAudioImplementationDigest,
createSpeechSpineSetImplementationDigest,
@@ -25,10 +33,12 @@ export const speechSpineSurfaceImplementationDigest = digestOf("@narratage/speec
export const speechSpineTypes = {
spineProgram: { module: speechSpineModuleRef, name: "SpeechSpineProgram" },
spineSet: { module: speechSpineModuleRef, name: "SpeechSpineSet" },
+ visualSpec: { module: speechSpineModuleRef, name: "SpeechSpineVisualSpec" },
} satisfies Record<string, TypeRef>;
export const speechSpineProducers = {
createSet: { module: speechSpineModuleRef, name: "create-spine-set" },
- appendTake: { module: speechSpineModuleRef, name: "append-spine-take" },
+ appendAudioTake: { module: speechSpineModuleRef, name: "append-spine-audio-take" },
+ appendVisualTake: { module: speechSpineModuleRef, name: "append-spine-visual-take" },
compileAudio: { module: speechSpineModuleRef, name: "compile-spine-audio" },
assembleBasis: { module: speechSpineModuleRef, name: "assemble-speech-basis" },
} satisfies Record<string, ProducerRef>;
@@ -52,9 +62,19 @@ export const speechSpineSetSchema: ValueSchema = object({
takes: { schema: { kind: "array", items: object({
segment: { schema: narrativeExcerptSchema },
media: { schema: synchronizedMediaSchema },
+ visual: { schema: object({
+ frame: { schema: spatialFrameSchema },
+ fit: { schema: contentFitSchema },
+ stackingOrder: { schema: { kind: "number", integer: true } },
+ }), optional: true },
}) } },
});
+export const speechSpineVisualSpecSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.speech-spine-visual-spec@1" } },
+ stackingOrder: { schema: { kind: "number", integer: true } },
+});
+
export const speechSpineManifest: ModuleManifest = {
format: "svml.module@1",
name: speechSpineModuleRef.name,
@@ -66,19 +86,23 @@ export const speechSpineManifest: ModuleManifest = {
programSpaceDependency,
speechDependency,
compositionDependency,
+ spatialDependency,
+ { module: svsModuleRef, digest: digestOf(svsManifest) },
{ module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },
{ module: speechBasisModuleRef, digest: digestOf(speechBasisManifest) },
],
types: [
{ name: speechSpineTypes.spineProgram.name, schema: speechSpineProgramSchema },
{ name: speechSpineTypes.spineSet.name, schema: speechSpineSetSchema },
+ { name: speechSpineTypes.visualSpec.name, schema: speechSpineVisualSpecSchema },
],
capabilities: [],
surfaces: [{
name: "spine",
tag: "Spine",
mode: "structured",
- outputs: [speechSpineTypes.spineProgram, mediaPipelineTypes.selectionRequest,
+ outputs: [speechSpineTypes.spineProgram, speechSpineTypes.visualSpec, spatialTypes.fit,
+ mediaPipelineTypes.selectionRequest,
speechTypes.basis, programSpaceTypes.programSpace, speechTypes.audioBasis,
compositionTypes.visualTrack, compositionTypes.audioTrack],
implementation: {
@@ -96,16 +120,31 @@ export const speechSpineManifest: ModuleManifest = {
implementation: { kind: "registered", locator: "@narratage/speech-spine/create-spine-set", digest: createSpeechSpineSetImplementationDigest },
},
{
- name: speechSpineProducers.appendTake.name,
+ name: speechSpineProducers.appendAudioTake.name,
+ inputs: [
+ { name: "set", type: speechSpineTypes.spineSet },
+ { name: "program", type: speechSpineTypes.spineProgram },
+ { name: "media", type: mediaTypes.synchronized },
+ { name: "segment", type: narrativeTypes.excerpt },
+ ],
+ outputs: [{ name: "set", type: speechSpineTypes.spineSet }],
+ needs: [],
+ implementation: { kind: "registered", locator: "@narratage/speech-spine/append-spine-audio-take", digest: appendSpeechSpineAudioTakeImplementationDigest },
+ },
+ {
+ name: speechSpineProducers.appendVisualTake.name,
inputs: [
{ name: "set", type: speechSpineTypes.spineSet },
{ name: "program", type: speechSpineTypes.spineProgram },
{ name: "media", type: mediaTypes.synchronized },
{ name: "segment", type: narrativeTypes.excerpt },
+ { name: "frame", type: spatialTypes.frame },
+ { name: "fit", type: spatialTypes.fit },
+ { name: "visualSpec", type: speechSpineTypes.visualSpec },
],
outputs: [{ name: "set", type: speechSpineTypes.spineSet }],
needs: [],
- implementation: { kind: "registered", locator: "@narratage/speech-spine/append-spine-take", digest: appendSpeechSpineTakeImplementationDigest },
+ implementation: { kind: "registered", locator: "@narratage/speech-spine/append-spine-visual-take", digest: appendSpeechSpineVisualTakeImplementationDigest },
},
{
name: speechSpineProducers.compileAudio.name,
diff --git a/packages/speech-spine/src/program.ts b/packages/speech-spine/src/program.ts
index b5f93f9e..0c60994a 100644
--- a/packages/speech-spine/src/program.ts
+++ b/packages/speech-spine/src/program.ts
@@ -1,5 +1,5 @@
import type { NarrativeExcerpt } from "@narratage/narrative";
-import { verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";
+import { synchronizedMediaSampleFrames, verifySynchronizedMedia, verifyTimelineAudio } from "@narratage/media";
import type { SynchronizedMedia, TimelineAudio } from "@narratage/media";
import { assertProgramSpaceIdentity, sealProgramSpace } from "@narratage/program-space";
import type { ProgramSpace } from "@narratage/program-space";
@@ -11,15 +11,19 @@ import {
} from "@narratage/media-pipeline";
import type { AudioProgramPlan } from "@narratage/media-pipeline";
import { canonicalize, digestOf } from "@narratage/protocol";
+import { assertContentFit, assertSpatialFrame } from "@narratage/spatial";
+import type { ContentFit, SpatialFrame } from "@narratage/spatial";
import type {
SpeechSpineProgram,
SpeechSpineSet,
SpeechSpineTake,
+ SpeechSpineVisualSpec,
} from "./types.js";
export const createSpeechSpineSetImplementationDigest = digestOf("@narratage/speech-spine/create-spine-set@1");
-export const appendSpeechSpineTakeImplementationDigest = digestOf("@narratage/speech-spine/append-spine-take@1");
+export const appendSpeechSpineAudioTakeImplementationDigest = digestOf("@narratage/speech-spine/append-spine-audio-take@1");
+export const appendSpeechSpineVisualTakeImplementationDigest = digestOf("@narratage/speech-spine/append-spine-visual-take@1");
export const compileSpeechSpineAudioImplementationDigest = digestOf("@narratage/speech-spine/compile-spine-audio@1");
export const assembleSpeechBasisImplementationDigest = digestOf("@narratage/speech-spine/assemble-speech-basis@1");
@@ -69,11 +73,29 @@ function sealSpeechSpineSet(value: SpeechSpineSet): SpeechSpineSet {
function assertTake(take: SpeechSpineTake, program: SpeechSpineProgram): void {
verifyExcerpt(take.segment);
verifySynchronizedMedia(take.media);
- assert(take.media.visual !== undefined, `Speech Segment ${take.segment.id} has no moving visual`);
assert(take.media.audio !== undefined, `Speech Segment ${take.segment.id} has no speech audio`);
assert(take.media.timeline.frameRate.numerator === program.frameRate.numerator
&& take.media.timeline.frameRate.denominator === program.frameRate.denominator,
`Speech Segment ${take.segment.id} uses another frame rate`);
+ assert((take.media.visual === undefined) === (take.visual === undefined),
+ `Speech Segment ${take.segment.id} visual policy does not match its media`);
+ if (take.visual !== undefined) {
+ assertSpatialFrame(take.visual.frame);
+ assertContentFit(take.visual.fit);
+ assert(Number.isSafeInteger(take.visual.stackingOrder),
+ `Speech Segment ${take.segment.id} visual stacking order is invalid`);
+ }
+}
+
+export function sealSpeechSpineVisualSpec(value: SpeechSpineVisualSpec): SpeechSpineVisualSpec {
+ const result = structuredClone(value);
+ assertSpeechSpineVisualSpec(result);
+ return result;
+}
+
+export function assertSpeechSpineVisualSpec(value: SpeechSpineVisualSpec): void {
+ assert(value.contract === "svml.speech-spine-visual-spec@1", "Unsupported SpeechSpineVisualSpec contract");
+ assert(Number.isSafeInteger(value.stackingOrder), "SpeechSpineVisualSpec stacking order is invalid");
}
export function assertSpeechSpineSet(value: SpeechSpineSet): void {
@@ -94,23 +116,58 @@ export function createSpeechSpineSet(): SpeechSpineSet {
});
}
-export function appendSpeechSpineTake(
+function appendTake(
set: SpeechSpineSet,
program: SpeechSpineProgram,
- media: SynchronizedMedia,
- segment: NarrativeExcerpt,
+ take: SpeechSpineTake,
): SpeechSpineSet {
assertSpeechSpineSet(set);
assertSpeechSpineProgram(program);
- const take = { media, segment } satisfies SpeechSpineTake;
assertTake(take, program);
- assert(!set.takes.some((item) => item.segment.id === segment.id), `Speech Spine repeats Segment ${segment.id}`);
+ assert(!set.takes.some((item) => item.segment.id === take.segment.id), `Speech Spine repeats Segment ${take.segment.id}`);
return sealSpeechSpineSet({
contract: "svml.speech-spine-set@1",
takes: [...set.takes, take],
});
}
+export function appendSpeechSpineAudioTake(
+ set: SpeechSpineSet,
+ program: SpeechSpineProgram,
+ media: SynchronizedMedia,
+ segment: NarrativeExcerpt,
+): SpeechSpineSet {
+ verifySynchronizedMedia(media);
+ assert(media.visual === undefined, `Speech Segment ${segment.id} audio Take unexpectedly contains a visual stream`);
+ return appendTake(set, program, { media, segment });
+}
+
+export function appendSpeechSpineVisualTake(
+ set: SpeechSpineSet,
+ program: SpeechSpineProgram,
+ media: SynchronizedMedia,
+ segment: NarrativeExcerpt,
+ frame: SpatialFrame,
+ fit: ContentFit,
+ visualSpec: SpeechSpineVisualSpec,
+): SpeechSpineSet {
+ assertSpatialFrame(frame);
+ assertContentFit(fit);
+ assertSpeechSpineVisualSpec(visualSpec);
+ verifySynchronizedMedia(media);
+ assert(media.visual !== undefined, `Speech Segment ${segment.id} visual Take has no visual stream`);
+ const take = {
+ media,
+ segment,
+ visual: {
+ frame: structuredClone(frame),
+ fit: structuredClone(fit),
+ stackingOrder: visualSpec.stackingOrder,
+ },
+ } satisfies SpeechSpineTake;
+ return appendTake(set, program, take);
+}
+
function frameSample(frame: number, frameRate: SpeechSpineProgram["frameRate"]): number {
const numerator = BigInt(frame) * 48_000n * BigInt(frameRate.denominator);
const denominator = BigInt(frameRate.numerator);
@@ -143,16 +200,17 @@ export function compileSpeechSpineAudio(program: SpeechSpineProgram, set: Speech
frame += take.media.timeline.frameCount;
const targetStartSample = frameSample(startFrame, program.frameRate);
const targetEndSampleExclusive = frameSample(frame, program.frameRate);
- assert(targetEndSampleExclusive - targetStartSample === audio.sampleFrames,
+ const sourceSampleFrames = synchronizedMediaSampleFrames(take.media);
+ assert(targetEndSampleExclusive - targetStartSample === sourceSampleFrames,
`Speech Segment ${take.segment.id} audio does not exactly cover its normalized frame span`);
return {
id: `${String(index + 1).padStart(4, "0")}:${take.segment.id}`,
artifact: audio.artifact,
targetStartSample,
targetEndSampleExclusive,
- sourceSampleFrames: audio.sampleFrames,
+ sourceSampleFrames,
sourceStartSample: 0,
- sourceEndSampleExclusive: audio.sampleFrames,
+ sourceEndSampleExclusive: sourceSampleFrames,
sourceLoop: false,
sourcePhaseSample: 0,
playbackRate: 1,
@@ -193,17 +251,27 @@ export function assembleSpeechBasis(
frame += take.media.timeline.frameCount;
return { segmentId: take.segment.id, startSec: seconds(startFrame), endSec: seconds(frame) };
});
- const visualClips = set.takes.map((take, index) => ({
- segmentId: take.segment.id,
- artifact: structuredClone(take.media.visual!.artifact),
- extent: {
- contract: "svml.intrinsic-extent@1" as const,
- widthPx: take.media.visual!.width,
- heightPx: take.media.visual!.height,
- },
- frameRate: { ...take.media.visual!.frameRate },
- frameCount: take.media.visual!.frameCount,
- }));
+ frame = 0;
+ const visualClips = set.takes.flatMap((take) => {
+ const startFrame = frame;
+ frame += take.media.timeline.frameCount;
+ if (take.media.visual === undefined || take.visual === undefined) return [];
+ return [{
+ segmentId: take.segment.id,
+ span: { startFrame, endFrameExclusive: frame },
+ artifact: structuredClone(take.media.visual.artifact),
+ extent: {
+ contract: "svml.intrinsic-extent@1" as const,
+ widthPx: take.media.visual.width,
+ heightPx: take.media.visual.height,
+ },
+ frameRate: { ...take.media.timeline.frameRate },
+ frameCount: take.media.timeline.frameCount,
+ frame: structuredClone(take.visual.frame),
+ fit: structuredClone(take.visual.fit),
+ stackingOrder: take.visual.stackingOrder,
+ }];
+ });
return sealSpeechBasis({
contract: "svml.speech-basis@1",
programSpace: space,
diff --git a/packages/speech-spine/src/surface.ts b/packages/speech-spine/src/surface.ts
index eadc2e7d..f88eeb75 100644
--- a/packages/speech-spine/src/surface.ts
+++ b/packages/speech-spine/src/surface.ts
@@ -2,9 +2,12 @@ import { narrativeTypes } from "@narratage/narrative";
import { artifactTypes } from "@narratage/artifact";
import { mediaTypes } from "@narratage/media";
import {
+ mediaPipelineTypes,
sealMediaSelectionRequest,
synchronizedMediaFragment,
} from "@narratage/media-pipeline";
+import type { SvsRecipe } from "@narratage/svs";
+import { svsRecipeType } from "@narratage/svs";
import type {
StructuredElement,
StructuredSurfaceHandler,
@@ -14,8 +17,12 @@ import type {
import { createSpeechSpineFragment } from "./fragment.js";
import { speechSpineTypes } from "./manifest.js";
-import { sealSpeechSpineProgram } from "./program.js";
-import { spatialTypes } from "@narratage/spatial";
+import { sealSpeechSpineProgram, sealSpeechSpineVisualSpec } from "./program.js";
+import {
+ contentFitPropertyNames,
+ decodeContentFitProperties,
+ spatialTypes,
+} from "@narratage/spatial";
function localName(value: string): string {
return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value;
@@ -27,6 +34,15 @@ function exactAttributes(element: StructuredElement, names: readonly string[]):
}
}
+function allowedAttributes(element: StructuredElement, required: readonly string[], optional: readonly string[]): void {
+ const keys = Object.keys(element.attributes);
+ const missing = required.filter((name) => !keys.includes(name));
+ const unknown = keys.filter((name) => !required.includes(name) && !optional.includes(name));
+ if (missing.length > 0 || unknown.length > 0) {
+ throw new Error(`${element.name} requires ${required.join(", ")}${optional.length === 0 ? "" : ` and optionally ${optional.join(", ")}`}`);
+ }
+}
+
function stringAttribute(element: StructuredElement, name: string): string {
const value = element.attributes[name];
if (typeof value !== "string" || value.trim().length === 0) throw new Error(`${element.name}.${name} must be a non-empty string`);
@@ -45,6 +61,13 @@ function frameRate(element: StructuredElement): { readonly numerator: number; re
return { numerator, denominator };
}
+function integerAttribute(element: StructuredElement, name: string): number {
+ const raw = stringAttribute(element, name);
+ const value = Number(raw);
+ if (!Number.isSafeInteger(value)) throw new Error(`${element.name}.${name} must be an integer`);
+ return value;
+}
+
function referencePath(element: StructuredElement, name: string): string {
const value: MarkupAttributeValue | undefined = element.attributes[name];
if (typeof value !== "object" || value.kind !== "reference" || !value.path) {
@@ -70,6 +93,18 @@ function resolve(
return value;
}
+function recipe(value: SurfaceResolvedReference, label: string): SvsRecipe {
+ if (!sameType(value.type, svsRecipeType) || value.record?.value.kind !== "inline") {
+ throw new Error(`${label} must be an authored SVS Recipe`);
+ }
+ const result = value.record.value.value as unknown as SvsRecipe;
+ const unknown = Object.keys(result.properties).filter((name) => !contentFitPropertyNames.includes(
+ name as (typeof contentFitPropertyNames)[number],
+ ));
+ if (unknown.length > 0) throw new Error(`${label} only accepts spatial fit properties; found ${unknown.join(", ")}`);
+ return result;
+}
+
function takes(element: StructuredElement): StructuredElement[] {
const result: StructuredElement[] = [];
for (const child of element.children) {
@@ -85,56 +120,165 @@ function takes(element: StructuredElement): StructuredElement[] {
}
export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
- exactAttributes(element, ["id", "canvas", "frame-rate"]);
+ exactAttributes(element, ["id", "frame-rate", "visual-frame", "visual-appearance", "visual-z"]);
const id = stringAttribute(element, "id");
- const canvas = resolve(element, "canvas", spatialTypes.canvas, resolveReference);
+ const baseFrame = resolve(element, "visual-frame", spatialTypes.frame, resolveReference);
+ const baseAppearanceReference = resolve(element, "visual-appearance", svsRecipeType, resolveReference);
+ const baseAppearance = recipe(
+ baseAppearanceReference,
+ `${element.name}.visual-appearance`,
+ );
+ const baseZ = integerAttribute(element, "visual-z");
const rate = frameRate(element);
+ const frameInputNames = new Map<string, string>();
+ const frameInputName = (value: SurfaceResolvedReference, fallback: string): string => {
+ const key = JSON.stringify(value.ref);
+ const previous = frameInputNames.get(key);
+ if (previous !== undefined) return previous;
+ frameInputNames.set(key, fallback);
+ return fallback;
+ };
+ const fitInputNames = new Map<string, string>();
+ const fitInputName = (value: SurfaceResolvedReference, fallback: string): string => {
+ const key = JSON.stringify(value.ref);
+ const previous = fitInputNames.get(key);
+ if (previous !== undefined) return previous;
+ fitInputNames.set(key, fallback);
+ return fallback;
+ };
+ const visualSpecInputNames = new Map<number, string>();
+ const visualSpecInputName = (stackingOrder: number, fallback: string): string => {
+ const previous = visualSpecInputNames.get(stackingOrder);
+ if (previous !== undefined) return previous;
+ visualSpecInputNames.set(stackingOrder, fallback);
+ return fallback;
+ };
const declaredTakes = takes(element).map((take, index) => {
const hasVideo = take.attributes.video !== undefined;
+ const hasAudio = take.attributes.audio !== undefined;
const hasMedia = take.attributes.media !== undefined;
- if (Number(hasVideo) + Number(hasMedia) !== 1) {
- throw new Error(`${take.name} requires exactly one of video or media`);
+ if (Number(hasVideo) + Number(hasAudio) + Number(hasMedia) !== 1) {
+ throw new Error(`${take.name} requires exactly one of video, audio or media`);
+ }
+ const sourceAttribute = hasVideo ? "video" : hasAudio ? "audio" : "media";
+ allowedAttributes(take, [sourceAttribute, "segment"], ["frame", "appearance", "z"]);
+ if (hasAudio && (take.attributes.frame !== undefined || take.attributes.appearance !== undefined || take.attributes.z !== undefined)) {
+ throw new Error(`${take.name} audio Take cannot declare visual frame, appearance or z`);
}
- exactAttributes(take, [hasVideo ? "video" : "media", "segment"]);
+ const suffix = String(index + 1).padStart(4, "0");
+ const effectiveFrame = take.attributes.frame === undefined
+ ? baseFrame
+ : resolve(take, "frame", spatialTypes.frame, resolveReference);
+ const effectiveAppearanceReference = take.attributes.appearance === undefined
+ ? baseAppearanceReference
+ : resolve(take, "appearance", svsRecipeType, resolveReference);
+ const effectiveAppearance = take.attributes.appearance === undefined
+ ? baseAppearance
+ : recipe(effectiveAppearanceReference, `${take.name}.appearance`);
+ const effectiveZ = take.attributes.z === undefined ? baseZ : integerAttribute(take, "z");
+ const visual = hasAudio ? undefined : {
+ frameName: frameInputName(effectiveFrame, take.attributes.frame === undefined ? "visual-base-frame" : `take-${suffix}-frame`),
+ fitName: fitInputName(effectiveAppearanceReference, take.attributes.appearance === undefined ? "visual-base-fit" : `take-${suffix}-fit`),
+ visualSpecName: visualSpecInputName(effectiveZ, take.attributes.z === undefined ? "visual-base-spec" : `take-${suffix}-visual-spec`),
+ frame: effectiveFrame,
+ appearance: effectiveAppearance,
+ stackingOrder: effectiveZ,
+ };
return {
- mediaName: `take-${String(index + 1).padStart(4, "0")}-media`,
- segmentName: `take-${String(index + 1).padStart(4, "0")}-segment`,
- sourceKind: hasVideo ? "video" as const : "media" as const,
- source: resolve(take, hasVideo ? "video" : "media", hasVideo ? artifactTypes.blob : mediaTypes.synchronized, resolveReference),
+ suffix,
+ mediaName: `take-${suffix}-media`,
+ segmentName: `take-${suffix}-segment`,
+ sourceKind: hasVideo ? "video" as const : hasAudio ? "audio" as const : "media" as const,
+ source: resolve(take, sourceAttribute, hasMedia ? mediaTypes.synchronized : artifactTypes.blob, resolveReference),
segment: resolve(take, "segment", narrativeTypes.excerpt, resolveReference),
+ visual,
range: take.range,
};
});
const programId = `${id}.program`;
- const requestId = `${id}.selection`;
+ const videoRequestId = `${id}.selection.video`;
+ const audioRequestId = `${id}.selection.audio`;
const program = sealSpeechSpineProgram({
contract: "svml.speech-spine-program@1",
id,
frameRate: rate,
});
- const request = sealMediaSelectionRequest({
+ const videoRequest = sealMediaSelectionRequest({
contract: "svml.media-selection-request@1",
video: { mode: "primary-moving" },
audio: { mode: "default" },
spanAuthority: "video",
frameRate: rate,
});
+ const audioRequest = sealMediaSelectionRequest({
+ contract: "svml.media-selection-request@1",
+ video: { mode: "none" },
+ audio: { mode: "default" },
+ spanAuthority: "audio",
+ frameRate: rate,
+ });
const assembly = createSpeechSpineFragment({
name: `@narratage/speech-spine/surface/${id}@1`,
- takes: declaredTakes.map(({ mediaName, segmentName }) => ({ mediaName, segmentName })),
+ takes: declaredTakes.map(({ mediaName, segmentName, visual }) => ({
+ mediaName,
+ segmentName,
+ ...(visual === undefined ? {} : { visual: {
+ frameName: visual.frameName,
+ fitName: visual.fitName,
+ visualSpecName: visual.visualSpecName,
+ } }),
+ })),
});
const normalizationComponents = declaredTakes.flatMap((take, index) => take.sourceKind === "media" ? [] : [{
id: `${id}.normalize.${String(index + 1).padStart(4, "0")}`,
fragment: synchronizedMediaFragment.id,
- inputs: { source: take.source.ref, request: { kind: "record" as const, id: requestId } },
+ inputs: {
+ source: take.source.ref,
+ request: { kind: "record" as const, id: take.sourceKind === "audio" ? audioRequestId : videoRequestId },
+ },
outputs: { media: `${id}.normalized.${String(index + 1).padStart(4, "0")}` },
range: take.range,
}]);
const normalizationByTake = new Map(normalizationComponents.map((component) => [component.id.split(".").at(-1), component]));
+ const emittedFits = new Set<string>();
+ const emittedSpecs = new Set<string>();
+ const visualRecords = declaredTakes.flatMap((take) => {
+ if (take.visual === undefined) return [];
+ const records = [];
+ if (!emittedFits.has(take.visual.fitName)) {
+ emittedFits.add(take.visual.fitName);
+ records.push({
+ id: `${id}.${take.visual.fitName}`,
+ type: spatialTypes.fit,
+ value: { kind: "inline" as const, value: decodeContentFitProperties(
+ take.visual.appearance.properties,
+ `Speech visual Recipe ${take.visual.appearance.path}`,
+ ) },
+ range: take.range,
+ });
+ }
+ if (!emittedSpecs.has(take.visual.visualSpecName)) {
+ emittedSpecs.add(take.visual.visualSpecName);
+ records.push({
+ id: `${id}.${take.visual.visualSpecName}`,
+ type: speechSpineTypes.visualSpec,
+ value: { kind: "inline" as const, value: sealSpeechSpineVisualSpec({
+ contract: "svml.speech-spine-visual-spec@1",
+ stackingOrder: take.visual.stackingOrder,
+ }) },
+ range: take.range,
+ });
+ }
+ return records;
+ });
+ const hasVideoNormalization = declaredTakes.some((take) => take.sourceKind === "video");
+ const hasAudioNormalization = declaredTakes.some((take) => take.sourceKind === "audio");
return {
records: [
{ id: programId, type: speechSpineTypes.spineProgram, value: { kind: "inline", value: program }, range: element.range },
- ...(normalizationComponents.length === 0 ? [] : [{ id: requestId, type: { module: { name: "@narratage/media-pipeline", version: "1" }, name: "MediaSelectionRequest" }, value: { kind: "inline" as const, value: request }, range: element.range }]),
+ ...(hasVideoNormalization ? [{ id: videoRequestId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline" as const, value: videoRequest }, range: element.range }] : []),
+ ...(hasAudioNormalization ? [{ id: audioRequestId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline" as const, value: audioRequest }, range: element.range }] : []),
+ ...visualRecords,
],
components: [
...normalizationComponents,
@@ -143,7 +287,6 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, re
fragment: assembly.id,
inputs: {
program: { kind: "record", id: programId },
- canvas: canvas.ref,
...Object.fromEntries(declaredTakes.flatMap((take, index) => [
[take.mediaName, take.sourceKind === "media" ? take.source.ref : {
kind: "component-output" as const,
@@ -151,6 +294,11 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, re
output: "media",
}],
[take.segmentName, take.segment.ref],
+ ...(take.visual === undefined ? [] : [
+ [take.visual.frameName, take.visual.frame.ref],
+ [take.visual.fitName, { kind: "record" as const, id: `${id}.${take.visual.fitName}` }],
+ [take.visual.visualSpecName, { kind: "record" as const, id: `${id}.${take.visual.visualSpecName}` }],
+ ]),
])),
},
outputs: {
diff --git a/packages/speech-spine/src/types.ts b/packages/speech-spine/src/types.ts
index 806cfafa..7c8b1ee2 100644
--- a/packages/speech-spine/src/types.ts
+++ b/packages/speech-spine/src/types.ts
@@ -1,5 +1,6 @@
import type { NarrativeExcerpt } from "@narratage/narrative";
import type { SynchronizedMedia } from "@narratage/media";
+import type { ContentFit, SpatialFrame } from "@narratage/spatial";
/** Author-owned policy for one ordered Speech Spine. */
export type SpeechSpineProgram = {
@@ -14,6 +15,16 @@ export type SpeechSpineProgram = {
export type SpeechSpineTake = {
readonly segment: NarrativeExcerpt;
readonly media: SynchronizedMedia;
+ readonly visual?: {
+ readonly frame: SpatialFrame;
+ readonly fit: ContentFit;
+ readonly stackingOrder: number;
+ };
+};
+
+export type SpeechSpineVisualSpec = {
+ readonly contract: "svml.speech-spine-visual-spec@1";
+ readonly stackingOrder: number;
};
/**
@@ -28,6 +39,11 @@ export type SpeechSpineSet = {
export type SpeechSpineInput = {
readonly mediaName: string;
readonly segmentName: string;
+ readonly visual?: {
+ readonly frameName: string;
+ readonly fitName: string;
+ readonly visualSpecName: string;
+ };
};
export type SpeechSpineFragmentOptions = {
diff --git a/packages/speech-spine/test/program.test.ts b/packages/speech-spine/test/program.test.ts
new file mode 100644
index 00000000..2aff12b7
--- /dev/null
+++ b/packages/speech-spine/test/program.test.ts
@@ -0,0 +1,89 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import type { SynchronizedMedia, TimelineAudio } from "@narratage/media";
+import type { NarrativeExcerpt } from "@narratage/narrative";
+import { digestOf } from "@narratage/protocol";
+import {
+ appendSpeechSpineAudioTake,
+ appendSpeechSpineVisualTake,
+ assembleSpeechBasis,
+ createSpeechSpineSet,
+ sealSpeechSpineProgram,
+ sealSpeechSpineVisualSpec,
+} from "@narratage/speech-spine";
+
+const frame = { contract: "svml.spatial-frame@1" as const, xPx: 40, yPx: 80, widthPx: 640, heightPx: 900 };
+const fit = {
+ contract: "svml.content-fit@1" as const,
+ sizing: "cover" as const,
+ framePoint: { x: 0.5, y: 0.5 },
+ contentPoint: { x: 0.5, y: 0.5 },
+ offsetPx: { x: 0, y: 0 },
+ constraint: "bounded" as const,
+};
+
+function segment(id: string, index: number): NarrativeExcerpt {
+ return { contract: "svml.narrative-excerpt@1", kind: "segment", id, tokenStart: index, tokenEndExclusive: index + 1 };
+}
+
+function synchronized(id: string, visual: boolean): SynchronizedMedia {
+ return {
+ contract: "svml.synchronized-media@1",
+ timeline: {
+ frameRate: { numerator: 30, denominator: 1 },
+ frameCount: 30,
+ },
+ ...(visual ? { visual: {
+ artifact: { kind: "blob" as const, digest: digestOf(`${id}:video`), size: 1, mediaType: "video/mp4" },
+ width: 720,
+ height: 1280,
+ } } : {}),
+ audio: {
+ artifact: { kind: "blob", digest: digestOf(`${id}:audio`), size: 1, mediaType: "audio/wav" },
+ },
+ };
+}
+
+test("audio Takes lengthen the speech program without inventing a visual clip", () => {
+ const program = sealSpeechSpineProgram({
+ contract: "svml.speech-spine-program@1",
+ id: "speech",
+ frameRate: { numerator: 30, denominator: 1 },
+ });
+ let set = createSpeechSpineSet();
+ set = appendSpeechSpineAudioTake(set, program, synchronized("voice", false), segment("voiceover", 0));
+ set = appendSpeechSpineVisualTake(
+ set,
+ program,
+ synchronized("presenter", true),
+ segment("answer", 1),
+ frame,
+ fit,
+ sealSpeechSpineVisualSpec({ contract: "svml.speech-spine-visual-spec@1", stackingOrder: 30 }),
+ );
+ const audio: TimelineAudio = {
+ contract: "svml.timeline-audio@1",
+ artifact: { kind: "blob", digest: digestOf("speech:mix"), size: 1, mediaType: "audio/wav" },
+ codec: "pcm_s16le",
+ sampleRate: 48_000,
+ channels: 2,
+ sampleFrames: 96_000,
+ loudness: "planned",
+ };
+ const basis = assembleSpeechBasis(program, set, audio);
+ assert.deepEqual(basis.segments.map((item) => item.segmentId), ["voiceover", "answer"]);
+ assert.equal(basis.programSpace.durationSec, 2);
+ assert.equal(basis.visualTrack.clips.length, 1);
+ assert.deepEqual(basis.visualTrack.clips[0], {
+ segmentId: "answer",
+ span: { startFrame: 30, endFrameExclusive: 60 },
+ artifact: synchronized("presenter", true).visual!.artifact,
+ extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },
+ frameRate: { numerator: 30, denominator: 1 },
+ frameCount: 30,
+ frame,
+ fit,
+ stackingOrder: 30,
+ });
+});
diff --git a/packages/speech-spine/test/surface.test.ts b/packages/speech-spine/test/surface.test.ts
index 1133ef26..e151f555 100644
--- a/packages/speech-spine/test/surface.test.ts
+++ b/packages/speech-spine/test/surface.test.ts
@@ -29,6 +29,7 @@ import {
import { speechBasisManifest, speechBasisProducers } from "@narratage/speech-basis";
import { textComponent, textManifest } from "@narratage/text";
import { mediaTrackManifest } from "@narratage/media-track";
+import { svsManifest, svsModuleRef, svsRecipeType } from "@narratage/svs";
import {
decodeSpeechSpineSurface,
speechSpineManifest,
@@ -38,6 +39,7 @@ import {
} from "@narratage/speech-spine";
import {
decodeCanvasSurface,
+ decodeFrameSurface,
spatialComponent,
spatialModuleRef,
spatialSurfaceDigests,
@@ -54,10 +56,10 @@ const fixtureManifest: ModuleManifest = {
format: "svml.module@1",
name: fixtureModule.name,
version: fixtureModule.version,
- dependencies: [artifactDependency, mediaDependency],
+ dependencies: [artifactDependency, mediaDependency, { module: svsModuleRef, digest: digestOf(svsManifest) }],
types: [], capabilities: [], producers: [],
surfaces: [{
- name: "media", tag: "Media", mode: "structured", outputs: [artifactTypes.blob],
+ name: "media", tag: "Media", mode: "structured", outputs: [artifactTypes.blob, svsRecipeType],
implementation: { kind: "trusted-frontend-surface", locator: "example.speech-media/surface", digest: fixtureSurfaceDigest },
}],
};
@@ -75,6 +77,7 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio plan
...videoContractManifests,
mediaPipelineManifest,
mediaTrackManifest,
+ svsManifest,
speechBasisManifest,
speechSpineManifest,
textManifest,
@@ -85,12 +88,23 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio plan
surfaces.registerRaw(scriptModuleRef, "script", scriptSurfaceImplementationDigest, decodeScriptSurface);
surfaces.registerStructured(speechSpineModuleRef, "spine", speechSpineSurfaceImplementationDigest, decodeSpeechSpineSurface);
surfaces.registerStructured(spatialModuleRef, "canvas", spatialSurfaceDigests.canvas, decodeCanvasSurface);
+ surfaces.registerStructured(spatialModuleRef, "frame", spatialSurfaceDigests.frame, decodeFrameSurface);
surfaces.registerStructured(fixtureModule, "media", fixtureSurfaceDigest, ({ element }) => ({
- records: ["take-one", "take-two"].map((id) => ({
- id, type: artifactTypes.blob,
- value: { kind: "blob" as const, digest: digestOf(id), size: 128, mediaType: "video/mp4" },
- range: element.range,
- })),
+ records: [
+ ...["take-one", "take-two", "voice-one"].map((id) => ({
+ id, type: artifactTypes.blob,
+ value: { kind: "blob" as const, digest: digestOf(id), size: 128,
+ mediaType: id === "voice-one" ? "audio/mpeg" : "video/mp4" },
+ range: element.range,
+ })),
+ {
+ id: "speech-style", type: svsRecipeType,
+ value: { kind: "inline" as const, value: {
+ contract: "svml.svs-recipe@1", path: "speech.base", properties: { fit: "cover" },
+ } },
+ range: element.range,
+ },
+ ],
components: [], fragments: [],
}));
const frontends = new AuthorFrontendRegistry();
@@ -119,8 +133,10 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio plan
</script>
<fixture:Media/>
<space:Canvas id="vertical" width="720" height="1280"/>
- <speech:Spine id="speech" canvas={vertical} frame-rate="30">
- <speech:Take video={take-one} segment={story.segment.opening}/>
+ <space:Frame id="speech-frame" within={vertical} left="0%" top="0%" right="100%" bottom="100%"/>
+ <speech:Spine id="speech" frame-rate="30"
+ visual-frame={speech-frame} visual-appearance={speech-style} visual-z="0">
+ <speech:Take audio={voice-one} segment={story.segment.opening}/>
<speech:Take video={take-two} segment={story.segment.answer}/>
</speech:Spine>
</svml>`),
@@ -138,7 +154,8 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio plan
const names = build.plan.steps.map((step) => step.producer.name);
assert.equal(names.filter((name) => name === mediaPipelineProducers.inspect.name).length, 2);
assert.equal(names.filter((name) => name === mediaPipelineProducers.normalize.name).length, 2);
- assert.equal(names.filter((name) => name === speechSpineProducers.appendTake.name).length, 2);
+ assert.equal(names.filter((name) => name === speechSpineProducers.appendAudioTake.name).length, 1);
+ assert.equal(names.filter((name) => name === speechSpineProducers.appendVisualTake.name).length, 1);
assert.equal(names.filter((name) => name === speechSpineProducers.compileAudio.name).length, 1);
assert.equal(names.filter((name) => name === mediaPipelineProducers.renderAudio.name).length, 1);
assert.equal(names.filter((name) => name === speechBasisProducers.projectVisual.name).length, 1);
diff --git a/packages/speech-spine/test/video-pipeline-fixture.ts b/packages/speech-spine/test/video-pipeline-fixture.ts
deleted file mode 100644
index ddcef5d7..00000000
--- a/packages/speech-spine/test/video-pipeline-fixture.ts
+++ /dev/null
@@ -1,504 +0,0 @@
-import { videoContractManifests } from "../../test-support/video-domain.js";
-import {
- captionManifest,
- captionTypes,
- captionValidatorDigests,
- plannedCaptionTimingFragment,
- resolveCaptionProgram,
- sealCaptionPlan,
- sealCaptionStyle,
-} from "@narratage/caption";
-import { narrativeDependency, narrativeTypes } from "@narratage/narrative";
-import { speechDependency, speechTypes } from "@narratage/speech";
-import { sealText, textDependency, textImplementationDigests, textManifest, textTypes } from "@narratage/text";
-import {
- createResolvedClosure,
- canonicalize,
- digestOf,
- link,
- sealBuildRequest,
- sealCompiledGraph,
- sealRecord,
- sealTypeValidationReceipt,
- sealTypedModule,
- start,
-} from "@narratage/core";
-import {
- bindAuthorFragment,
- bindCandidateFragment,
- elaborateGraphFragment,
- mergeFragmentContributions,
- sealGraphFragment,
-} from "@narratage/elaborator";
-import type {
- BuildState,
- CapabilityRef,
- CompiledGraph,
- LinkedProgram,
- ModuleManifest,
- ProducerRef,
- Satisfaction,
- TypeRef,
- ValueSchema,
-} from "@narratage/protocol";
-import { resolveRealization, sealRealizationOverlay } from "@narratage/run";
-import { spatialTypes, spatialValidatorDigests } from "@narratage/spatial";
-import {
- captionCorrespondence,
- captionDisplaySequence,
- narrativeValue,
- parseScript,
-} from "@narratage/script";
-import { speechAlignmentManifest } from "@narratage/speech-alignment";
-import { speechBasisManifest, speechBasisProjectionFragment } from "@narratage/speech-basis";
-import { mediaPipelineManifest } from "@narratage/media-pipeline";
-import { mediaTrackManifest } from "@narratage/media-track";
-import {
- whisperXManifest,
- whisperXSpeechAlignmentFragment,
- whisperXTypes,
-} from "@narratage/whisperx";
-
-export const videoModule = { name: "example.video-pipeline", version: "0.0.0" } as const;
-
-export const videoTypes = {
- estimate: { module: videoModule, name: "OfficialSpeechDurationEstimate" },
- seedanceMiniMedia: { module: videoModule, name: "SeedanceMiniSpeechMedia" },
-} satisfies Record<string, TypeRef>;
-
-export const videoCapabilities = {
- estimate: { module: videoModule, name: "official-speech-duration-estimate" },
- seedanceMini: { module: videoModule, name: "seedance-mini-speech-media" },
-} satisfies Record<string, CapabilityRef>;
-
-export const videoProducers = {
- requestEstimate: { module: videoModule, name: "request-official-speech-estimate" },
- placeholderEstimate: { module: videoModule, name: "placeholder-speech-estimate" },
- requestSeedanceMini: { module: videoModule, name: "request-seedance-mini-speech" },
- assembleBasis: { module: videoModule, name: "assemble-speech-basis" },
-} satisfies Record<string, ProducerRef>;
-
-export const videoImplementations = Object.fromEntries(
- Object.entries(videoProducers).map(([name]) => [name, digestOf(`example.video-pipeline/${name}@1`)]),
-) as Readonly<Record<keyof typeof videoProducers, ReturnType<typeof digestOf>>>;
-
-const string = { kind: "string", minLength: 1 } as const;
-const number = { kind: "number", minimum: 0 } as const;
-const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema }>>): ValueSchema => ({
- kind: "object",
- fields,
-});
-
-export const videoManifest: ModuleManifest = {
- format: "svml.module@1",
- name: videoModule.name,
- version: videoModule.version,
- dependencies: [
- narrativeDependency,
- speechDependency,
- textDependency,
- ],
- types: [
- { name: videoTypes.estimate.name, schema: object({ durationSec: { schema: number } }) },
- {
- name: videoTypes.seedanceMiniMedia.name,
- schema: object({
- model: { schema: { kind: "literal", value: "mini" } },
- audioDigest: { schema: string },
- visualDigest: { schema: string },
- durationSec: { schema: number },
- }),
- },
- ],
- capabilities: [
- { name: videoCapabilities.estimate.name, returns: videoTypes.estimate },
- { name: videoCapabilities.seedanceMini.name, returns: videoTypes.seedanceMiniMedia },
- ],
- surfaces: [],
- producers: [
- {
- name: videoProducers.requestEstimate.name,
- inputs: [{ name: "speech", type: textTypes.text }],
- outputs: [],
- needs: [{
- name: "estimate",
- capability: videoCapabilities.estimate,
- returns: videoTypes.estimate,
- }],
- implementation: {
- kind: "registered",
- locator: "example.video-pipeline/request-estimate",
- digest: videoImplementations.requestEstimate,
- },
- },
- {
- name: videoProducers.requestSeedanceMini.name,
- inputs: [
- { name: "dialogue", type: textTypes.text },
- { name: "estimate", type: videoTypes.estimate },
- ],
- outputs: [],
- needs: [{
- name: "media",
- capability: videoCapabilities.seedanceMini,
- returns: videoTypes.seedanceMiniMedia,
- }],
- implementation: {
- kind: "registered",
- locator: "example.video-pipeline/request-seedance-mini",
- digest: videoImplementations.requestSeedanceMini,
- },
- },
- {
- name: videoProducers.placeholderEstimate.name,
- inputs: [{ name: "speech", type: textTypes.text }],
- outputs: [{ name: "estimate", type: videoTypes.estimate }],
- needs: [],
- implementation: {
- kind: "registered",
- locator: "example.video-pipeline/placeholder-estimate",
- digest: videoImplementations.placeholderEstimate,
- },
- },
- {
- name: videoProducers.assembleBasis.name,
- inputs: [
- { name: "narrative", type: narrativeTypes.narrative },
- { name: "media", type: videoTypes.seedanceMiniMedia },
- ],
- outputs: [{
- name: "basis",
- type: speechTypes.basis,
- }],
- needs: [],
- implementation: {
- kind: "registered",
- locator: "example.video-pipeline/assemble-basis",
- digest: videoImplementations.assembleBasis,
- },
- },
- ],
-};
-
-const input = (name: string) => ({ kind: "fragment-input" as const, name });
-const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
-
-const durationEstimateFragment = sealGraphFragment({
- name: "example.video-pipeline/duration-estimate@1",
- inputs: [{ name: "speech", type: textTypes.text }],
- operations: [{
- id: "estimate",
- producer: videoProducers.requestEstimate,
- inputs: { speech: input("speech") },
- result: { kind: "need", name: "estimate", accepts: "exact" },
- }],
- exports: [{
- name: "estimate",
- type: videoTypes.estimate,
- root: operation("estimate"),
- semanticInputs: ["speech"],
- fidelity: "exact",
- }],
-});
-
-const speechBasisGenerationFragment = sealGraphFragment({
- name: "example.video-pipeline/seedance-mini-speech-basis@1",
- inputs: [
- { name: "narrative", type: narrativeTypes.narrative },
- { name: "dialogue", type: textTypes.text },
- { name: "estimate", type: videoTypes.estimate },
- ],
- operations: [
- {
- id: "request-seedance-mini",
- producer: videoProducers.requestSeedanceMini,
- inputs: { dialogue: input("dialogue"), estimate: input("estimate") },
- result: { kind: "need", name: "media", accepts: "exact" },
- },
- {
- id: "assemble-take",
- producer: videoProducers.assembleBasis,
- inputs: { narrative: input("narrative"), media: operation("request-seedance-mini") },
- result: { kind: "output", name: "basis" },
- },
- ],
- exports: [{
- name: "take",
- type: speechTypes.basis,
- root: operation("assemble-take"),
- semanticInputs: ["narrative", "dialogue", "estimate"],
- fidelity: "exact",
- }],
-});
-
-const placeholderEstimateFragment = sealGraphFragment({
- name: "example.video-pipeline/placeholder-estimate@1",
- inputs: [{ name: "speech", type: textTypes.text }],
- operations: [{
- id: "placeholder",
- producer: videoProducers.placeholderEstimate,
- inputs: { speech: input("speech") },
- result: { kind: "output", name: "estimate" },
- }],
- exports: [{
- name: "estimate",
- type: videoTypes.estimate,
- root: operation("placeholder"),
- semanticInputs: ["speech"],
- fidelity: "substitute",
- }],
-});
-
-export const videoOutputs = {
- estimate: "speech.estimate",
- take: "speech.take",
- programSpace: "speech.program-space",
- audio: "speech.audio",
- audioTrack: "speech.audio-track",
- visual: "speech.visual",
- rawEvidence: "speech.whisperx-evidence",
- evidence: "speech.evidence",
- map: "speech.map",
- caption: "speech.caption",
-} as const;
-
-export type VideoOutputName = keyof typeof videoOutputs;
-
-export const videoClosure = createResolvedClosure([
- ...videoContractManifests,
- textManifest,
- videoManifest,
- mediaTrackManifest,
- speechBasisManifest,
- mediaPipelineManifest,
- whisperXManifest,
- speechAlignmentManifest,
- captionManifest,
-]);
-
-function authorProgram(): LinkedProgram {
- const parsed = parseScript("pipeline.svml", "<line><that was insane | what the fuck></line>");
- const display = captionDisplaySequence(parsed, "caption:display");
- const correspondence = captionCorrespondence(parsed, display.id);
- const style = sealCaptionStyle({
- contract: "svml.caption-style@1",
- id: "fixture",
- planning: {
- cue: { minimumWords: 1, maximumWords: 7, instruction: "Use one authored Atom." },
- fields: [],
- },
- rendering: { family: "fixture-caption@1", parameters: {} },
- });
- const captionProgram = resolveCaptionProgram(display, "caption-program", style, []);
- const captionPlan = sealCaptionPlan({
- contract: "svml.caption-plan@1",
- runs: [{
- id: captionProgram.runs[0]!.id,
- styleId: style.id,
- cues: [{ id: "caption:cue:1", atomIds: display.atoms.map((atom) => atom.id), fields: [] }],
- }],
- });
- const origin = {
- kind: "authored" as const,
- sourceDigest: digestOf("source:video-pipeline"),
- frontendClosureDigest: digestOf("frontend:script"),
- sourceName: "pipeline.svml",
- };
- const narrative = sealRecord({
- id: "narrative:root",
- type: narrativeTypes.narrative,
- value: { kind: "inline", value: narrativeValue(parsed) },
- conformance: "exact",
- origin,
- });
- const rawSpeech = sealRecord({
- id: "narrative:speech",
- type: textTypes.text,
- value: { kind: "inline", value: sealText(parsed.serializations.speech) as never },
- conformance: "exact",
- origin,
- });
- const speech = { ...rawSpeech, validation: sealTypeValidationReceipt({
- type: rawSpeech.type,
- recordDigest: rawSpeech.digest,
- validatorDigest: textImplementationDigests.validateText,
- }) };
- const rawDialogue = sealRecord({
- id: "narrative:dialogue",
- type: textTypes.text,
- value: { kind: "inline", value: sealText(parsed.serializations.dialogue) as never },
- conformance: "exact",
- origin,
- });
- const dialogue = { ...rawDialogue, validation: sealTypeValidationReceipt({
- type: rawDialogue.type,
- recordDigest: rawDialogue.digest,
- validatorDigest: textImplementationDigests.validateText,
- }) };
- const programRecord = sealRecord({ id: "caption:program", type: captionTypes.program,
- value: { kind: "inline", value: canonicalize(captionProgram) }, conformance: "exact", origin });
- const planRecord = sealRecord({ id: "caption:plan", type: captionTypes.plan,
- value: { kind: "inline", value: canonicalize(captionPlan) }, conformance: "exact", origin });
- const captionRecords = [
- sealRecord({ id: "caption:display", type: narrativeTypes.captionDisplay,
- value: { kind: "inline", value: canonicalize(display) }, conformance: "exact", origin }),
- sealRecord({ id: "caption:correspondence", type: narrativeTypes.captionCorrespondence,
- value: { kind: "inline", value: canonicalize(correspondence) }, conformance: "exact", origin }),
- { ...programRecord, validation: sealTypeValidationReceipt({
- type: programRecord.type,
- recordDigest: programRecord.digest,
- validatorDigest: captionValidatorDigests.program,
- }) },
- { ...planRecord, validation: sealTypeValidationReceipt({
- type: planRecord.type,
- recordDigest: planRecord.digest,
- validatorDigest: captionValidatorDigests.plan,
- }) },
- ];
- const rawCanvas = sealRecord({
- id: "canvas:vertical",
- type: spatialTypes.canvas,
- value: { kind: "inline", value: {
- contract: "svml.canvas-space@1",
- widthPx: 720,
- heightPx: 1280,
- origin: "top-left",
- xDirection: "right",
- yDirection: "down",
- pixelAspect: "square",
- } },
- conformance: "exact",
- origin,
- });
- const canvas = {
- ...rawCanvas,
- validation: sealTypeValidationReceipt({
- type: rawCanvas.type,
- recordDigest: rawCanvas.digest,
- validatorDigest: spatialValidatorDigests.canvas,
- }),
- };
- return link(videoClosure, [sealTypedModule({
- id: "author:video-pipeline",
- closureDigest: videoClosure.digest,
- records: [narrative, speech, dialogue, canvas, ...captionRecords],
- })]);
-}
-
-export function createVideoGraph(program: LinkedProgram): CompiledGraph {
- const narrative = { kind: "record" as const, id: "narrative:root" };
- const speech = { kind: "record" as const, id: "narrative:speech" };
- const dialogue = { kind: "record" as const, id: "narrative:dialogue" };
- const estimate = elaborateGraphFragment(program, durationEstimateFragment, {
- id: "opening.estimate",
- fragment: durationEstimateFragment.id,
- inputs: { speech },
- });
- const take = elaborateGraphFragment(program, speechBasisGenerationFragment, {
- id: "opening.seedance-mini",
- fragment: speechBasisGenerationFragment.id,
- inputs: {
- narrative,
- dialogue,
- estimate: { kind: "logical-output", id: videoOutputs.estimate },
- },
- });
- const projections = elaborateGraphFragment(program, speechBasisProjectionFragment, {
- id: "opening.take-projections",
- fragment: speechBasisProjectionFragment.id,
- inputs: {
- basis: { kind: "logical-output", id: videoOutputs.take },
- canvas: { kind: "record", id: "canvas:vertical" },
- },
- });
- const alignment = elaborateGraphFragment(program, whisperXSpeechAlignmentFragment, {
- id: "opening.whisperx-alignment",
- fragment: whisperXSpeechAlignmentFragment.id,
- inputs: {
- narrative,
- audio: { kind: "logical-output", id: videoOutputs.audio },
- },
- });
- const caption = elaborateGraphFragment(program, plannedCaptionTimingFragment, {
- id: "opening.caption",
- fragment: plannedCaptionTimingFragment.id,
- inputs: {
- display: { kind: "record", id: "caption:display" },
- correspondence: { kind: "record", id: "caption:correspondence" },
- map: { kind: "logical-output", id: videoOutputs.map },
- program: { kind: "record", id: "caption:program" },
- plan: { kind: "record", id: "caption:plan" },
- },
- });
- const contribution = mergeFragmentContributions(
- { outputs: [], candidates: [], operations: [] },
- bindAuthorFragment(estimate, { estimate: videoOutputs.estimate }),
- bindAuthorFragment(take, { take: videoOutputs.take }),
- bindAuthorFragment(projections, {
- programSpace: videoOutputs.programSpace,
- audio: videoOutputs.audio,
- audioTrack: videoOutputs.audioTrack,
- visual: videoOutputs.visual,
- }),
- bindAuthorFragment(alignment, {
- rawEvidence: videoOutputs.rawEvidence,
- evidence: videoOutputs.evidence,
- map: videoOutputs.map,
- }),
- bindAuthorFragment(caption, { caption: videoOutputs.caption }),
- );
- return sealCompiledGraph({ program: program.semanticDigest, ...contribution });
-}
-
-export type VideoBuildOptions = {
- readonly estimateRealization?: "primary" | "placeholder";
- readonly goalAccepts?: "exact" | "substitute";
- readonly targets?: readonly VideoOutputName[];
-};
-
-export type VideoFixture = {
- readonly program: LinkedProgram;
- readonly source: CompiledGraph;
- readonly graph: CompiledGraph;
- readonly satisfactions: readonly Satisfaction[];
-};
-
-export function createVideoFixture(options: VideoBuildOptions = {}): VideoFixture {
- const program = authorProgram();
- const source = createVideoGraph(program);
- if (options.estimateRealization !== "placeholder") {
- return { program, source, graph: source, satisfactions: [] };
- }
- const placeholder = elaborateGraphFragment(program, placeholderEstimateFragment, {
- id: "opening.placeholder-estimate",
- fragment: placeholderEstimateFragment.id,
- inputs: { speech: { kind: "record", id: "narrative:speech" } },
- });
- const contribution = bindCandidateFragment(placeholder, { estimate: videoOutputs.estimate });
- const overlay = sealRealizationOverlay({
- sourceGraph: source.id,
- candidates: contribution.candidates,
- operations: contribution.operations,
- });
- const realized = resolveRealization(program, source, [overlay]);
- return {
- program,
- source,
- graph: realized.graph,
- satisfactions: contribution.satisfactions ?? [],
- };
-}
-
-export function createVideoBuild(options: VideoBuildOptions = {}): BuildState {
- const fixture = createVideoFixture(options);
- const accepts = options.goalAccepts ?? "exact";
- const targets = options.targets ?? ["caption"];
- const request = sealBuildRequest({
- graph: fixture.graph.id,
- targets: targets.map((name) => ({ output: videoOutputs[name], accepts })),
- satisfactions: fixture.satisfactions,
- });
- return start(fixture.program, fixture.graph, request);
-}
-
-export { whisperXTypes };
diff --git a/packages/speech-spine/test/video-pipeline.test.ts b/packages/speech-spine/test/video-pipeline.test.ts
deleted file mode 100644
index d3fb05b1..00000000
--- a/packages/speech-spine/test/video-pipeline.test.ts
+++ /dev/null
@@ -1,672 +0,0 @@
-import { compositionComponent, spatialComponent } from "../../test-support/video-domain.js";
-import {
- registerProducerFacets,
- registerTypeValidatorFacets,
-} from "@narratage/component-kit";
-import {
- captionComponent,
- captionProducers,
-} from "@narratage/caption";
-import type { TimedCaptionProjection } from "@narratage/caption";
-import type { Narrative } from "@narratage/narrative";
-import type { Text } from "@narratage/text";
-import { sealProgramSpace } from "@narratage/program-space";
-import { sealSpeechBasis, sealSpeechEvidenceAudio, speechTypes } from "@narratage/speech";
-import type { SpeechAudioBasis, SpeechBasis, SpeechEvidenceAudio } from "@narratage/speech";
-import { speechEvidenceTypes } from "@narratage/speech-evidence";
-import { compositionTypes, sealVisualTrack } from "@narratage/composition";
-import type { VisualTrack } from "@narratage/composition";
-import assert from "node:assert/strict";
-import test from "node:test";
-
-import { digestOf, sealBuildRequest, start } from "@narratage/core";
-import {
- ProducerRegistry,
- NodeDriver,
- EndpointRegistry,
- parseBuildState,
- serializeBuildState,
-} from "@narratage/driver-node";
-import type { BuildState, CapabilityRef, ProducerRef } from "@narratage/protocol";
-import {
- createProvidedCandidate,
- resolveRealization,
- sealRealizationOverlay,
-} from "@narratage/run";
-import {
- speechAlignmentComponent,
- speechAlignmentProducers,
-} from "@narratage/speech-alignment";
-import {
- mediaPipelineCapabilities,
- mediaPipelineImplementationDigests,
- mediaPipelineProducers,
-} from "@narratage/media-pipeline";
-import type { ProjectSpeechEvidenceAudioNeed } from "@narratage/media-pipeline";
-import {
- projectSpeechAudio,
- projectSpeechAudioImplementationDigest,
- projectSpeechVisual,
- projectSpeechVisualImplementationDigest,
- speechBasisProducers,
-} from "@narratage/speech-basis";
-import {
- normalizeWhisperXAlignment,
- sealWhisperXAlignmentEvidence,
- whisperXCapabilities,
- whisperXImplementationDigests,
- whisperXProducers,
- whisperXRequestForEvidenceAudio,
-} from "@narratage/whisperx";
-import type { WhisperXAlignmentEvidence } from "@narratage/whisperx";
-import { TypeValidatorRegistry, validateValue } from "@narratage/validation";
-
-import {
- createVideoBuild,
- createVideoFixture,
- videoCapabilities,
- videoImplementations,
- videoOutputs,
- videoProducers,
- videoTypes,
- whisperXTypes,
-} from "./video-pipeline-fixture.js";
-
-function inlineObject(value: unknown): Readonly<Record<string, unknown>> {
- if (value === null || Array.isArray(value) || typeof value !== "object") throw new Error("expected object");
- return value as Readonly<Record<string, unknown>>;
-}
-
-function validatorRegistry(): TypeValidatorRegistry {
- const registry = new TypeValidatorRegistry();
- registerTypeValidatorFacets(registry, captionComponent.validators ?? []);
- registerTypeValidatorFacets(registry, compositionComponent.validators);
- registerTypeValidatorFacets(registry, spatialComponent.validators);
- return registry;
-}
-
-function inlineValue<T>(value: unknown): T {
- return value as T;
-}
-
-function sameCapability(left: CapabilityRef, right: CapabilityRef): boolean {
- return left.name === right.name
- && left.module.name === right.module.name
- && left.module.version === right.module.version;
-}
-
-function selectedRecord(state: BuildState, output: string): string {
- const selection = state.plan.selections.find((item) => item.output === output);
- assert(selection, `missing selection for ${output}`);
- return selection.record;
-}
-
-function producerCount(state: BuildState, producer: ProducerRef): number {
- return state.plan.steps.filter((step) =>
- step.producer.name === producer.name
- && step.producer.module.name === producer.module.name
- && step.producer.module.version === producer.module.version).length;
-}
-
-test("the Speech Spine pipeline resumes without repeating paid calls", async () => {
- const host = new ProducerRegistry();
- const endpoints = new EndpointRegistry();
- const calls = {
- estimate: 0,
- seedance: 0,
- assemble: 0,
- projectAudio: 0,
- projectVisual: 0,
- evidenceAudioRequest: 0,
- whisperRequest: 0,
- whisperNormalize: 0,
- };
-
- host.registerProducer(videoProducers.requestEstimate, videoImplementations.requestEstimate, ({ inputs }) => {
- calls.estimate += 1;
- assert.equal(inputs.speech?.value.kind, "inline");
- const speech = inlineValue<Text>(inputs.speech.value.value);
- return {
- outputs: {},
- needs: {
- estimate: {
- contract: "example.official-speech-estimate-request@1",
- speech: speech.value,
- },
- },
- };
- });
- host.registerProducer(videoProducers.requestSeedanceMini, videoImplementations.requestSeedanceMini, ({ inputs }) => {
- calls.seedance += 1;
- assert.equal(inputs.dialogue?.value.kind, "inline");
- assert.equal(inputs.estimate?.value.kind, "inline");
- const dialogue = inlineValue<Text>(inputs.dialogue.value.value);
- const estimate = inlineObject(inputs.estimate.value.value);
- return {
- outputs: {},
- needs: {
- media: {
- contract: "example.seedance-mini-speech-request@1",
- model: "mini",
- script: dialogue.value,
- durationSec: estimate.durationSec as number,
- },
- },
- };
- });
- host.registerProducer(videoProducers.assembleBasis, videoImplementations.assembleBasis, ({ inputs }) => {
- calls.assemble += 1;
- assert.equal(inputs.narrative?.value.kind, "inline");
- assert.equal(inputs.media?.value.kind, "inline");
- const narrative = inlineValue<Narrative>(inputs.narrative.value.value);
- const media = inlineObject(inputs.media.value.value);
- const durationSec = media.durationSec as number;
- const programSpace = sealProgramSpace({
- contract: "svml.program-space@1",
- durationSec,
- frameRate: { numerator: 1_000, denominator: 1 },
- });
- const audioDigest = media.audioDigest as SpeechBasis["audio"]["digest"];
- const visualDigest = media.visualDigest as SpeechBasis["audio"]["digest"];
- const basis = sealSpeechBasis({
- contract: "svml.speech-basis@1",
- programSpace,
- audio: { kind: "blob", digest: audioDigest, size: 1, mediaType: "audio/wav" },
- visualTrack: { clips: [{
- segmentId: "line",
- artifact: { kind: "blob", digest: visualDigest, size: 1, mediaType: "video/mp4" },
- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },
- frameRate: { ...programSpace.frameRate },
- frameCount: Math.round(durationSec * 1_000),
- }] },
- segments: [{ segmentId: "line", startSec: 0, endSec: durationSec }],
- });
- return { outputs: { basis: { kind: "inline", value: basis } }, needs: {} };
- });
- host.registerProducer(
- speechBasisProducers.projectAudio,
- projectSpeechAudioImplementationDigest,
- ({ inputs }) => {
- calls.projectAudio += 1;
- assert.equal(inputs.basis?.value.kind, "inline");
- return {
- outputs: {
- audio: {
- kind: "inline",
- value: projectSpeechAudio(inlineValue<SpeechBasis>(inputs.basis.value.value)),
- },
- },
- needs: {},
- };
- },
- );
- host.registerProducer(
- mediaPipelineProducers.projectSpeechEvidenceAudio,
- mediaPipelineImplementationDigests.projectSpeechEvidenceAudio,
- ({ inputs }) => {
- calls.evidenceAudioRequest += 1;
- assert.equal(inputs.audio?.value.kind, "inline");
- const audio = inlineValue<SpeechAudioBasis>(inputs.audio.value.value);
- const sourceSampleFrames = Math.round(audio.programSpace.durationSec * 48_000);
- const need: ProjectSpeechEvidenceAudioNeed = {
- contract: "svml.project-speech-evidence-audio-request@1",
- source: {
- kind: "blob",
- digest: audio.audio.digest,
- size: audio.audio.size,
- mediaType: audio.audio.mediaType,
- },
- sourceSampleRate: 48_000,
- sourceChannels: 2,
- sourceCodec: "pcm_s16le",
- sourceSampleFrames,
- evidenceSampleRate: 16_000,
- evidenceChannels: 1,
- evidenceCodec: "pcm_s16le",
- evidenceSampleFrames: Math.round(sourceSampleFrames / 3),
- durationSec: audio.programSpace.durationSec,
- segments: audio.segments,
- };
- return { outputs: {}, needs: { evidenceAudio: need } };
- },
- );
- host.registerProducer(
- speechBasisProducers.projectVisual,
- projectSpeechVisualImplementationDigest,
- ({ inputs }) => {
- calls.projectVisual += 1;
- assert.equal(inputs.basis?.value.kind, "inline");
- assert(inputs.canvas?.value.kind === "inline");
- return {
- outputs: {
- visual: {
- kind: "inline",
- value: projectSpeechVisual(
- inlineValue<SpeechBasis>(inputs.basis.value.value),
- inlineValue(inputs.canvas.value.value),
- ),
- },
- },
- needs: {},
- };
- },
- );
- host.registerProducer(whisperXProducers.request, whisperXImplementationDigests.request, ({ inputs }) => {
- calls.whisperRequest += 1;
- assert.equal(inputs.audio?.value.kind, "inline");
- return {
- outputs: {},
- needs: {
- alignment: whisperXRequestForEvidenceAudio(
- inlineValue<SpeechEvidenceAudio>(inputs.audio.value.value),
- { language: "en" },
- ),
- },
- };
- });
- host.registerProducer(whisperXProducers.normalize, whisperXImplementationDigests.normalize, ({ inputs }) => {
- calls.whisperNormalize += 1;
- assert.equal(inputs.whisperx?.value.kind, "inline");
- const evidence = normalizeWhisperXAlignment(
- inlineValue<WhisperXAlignmentEvidence>(inputs.whisperx.value.value),
- );
- return { outputs: { evidence: { kind: "inline", value: evidence } }, needs: {} };
- });
- registerProducerFacets(host, speechAlignmentComponent.producers);
- registerProducerFacets(host, captionComponent.producers);
-
- const driver = new NodeDriver({ producers: host, endpoints, validators: validatorRegistry() });
- const atEstimate = await driver.run(createVideoBuild());
- assert.equal(atEstimate.blocked[0]?.reason, "missing-endpoint");
- assert.match(atEstimate.blocked[0]?.subject ?? "", /OfficialSpeechDurationEstimate/u);
-
- endpoints.registerImmediateEndpoint("runtime:official-estimate", videoCapabilities.estimate, videoTypes.estimate, () => ({
- value: { kind: "inline", value: { durationSec: 1 } },
- conformance: "exact",
- delivery: "executed",
- metadata: {},
- }));
- const atSeedance = await driver.run(parseBuildState(serializeBuildState(atEstimate.state)));
- assert.equal(atSeedance.blocked[0]?.reason, "missing-endpoint");
- assert.match(atSeedance.blocked[0]?.subject ?? "", /SeedanceMiniSpeechMedia/u);
-
- endpoints.registerImmediateEndpoint(
- "runtime:kie-seedance-mini",
- videoCapabilities.seedanceMini,
- videoTypes.seedanceMiniMedia,
- ({ need }) => {
- assert.deepEqual(need.constraints, {
- contract: "example.seedance-mini-speech-request@1",
- durationSec: 1,
- model: "mini",
- script: "what the fuck",
- });
- return {
- value: { kind: "inline", value: {
- model: "mini",
- audioDigest: digestOf("artifact:audio"),
- visualDigest: digestOf("artifact:visual"),
- durationSec: 1,
- } },
- conformance: "exact",
- delivery: "executed",
- metadata: { endpoint: "kie", model: "seedance-mini" },
- };
- },
- );
- const atEvidenceAudio = await driver.run(parseBuildState(serializeBuildState(atSeedance.state)));
- assert.equal(atEvidenceAudio.blocked[0]?.reason, "missing-endpoint");
- assert.match(atEvidenceAudio.blocked[0]?.subject ?? "", /SpeechEvidenceAudio/u);
-
- endpoints.registerImmediateEndpoint(
- "runtime:media-local",
- mediaPipelineCapabilities.projectSpeechEvidenceAudio,
- speechTypes.evidenceAudio,
- ({ need }) => {
- const request = inlineValue<ProjectSpeechEvidenceAudioNeed>(need.constraints);
- return {
- value: { kind: "inline", value: sealSpeechEvidenceAudio({
- contract: "svml.speech-evidence-audio@1",
- artifact: {
- kind: "blob",
- digest: digestOf("artifact:evidence-audio"),
- size: 32_044,
- mediaType: "audio/wav",
- },
- codec: "pcm_s16le",
- sampleRate: 16_000,
- channels: 1,
- sampleFrames: request.evidenceSampleFrames,
- durationSec: request.durationSec,
- segments: request.segments,
- sampleMap: {
- algorithm: "rational-boundary-round@1",
- sourceSampleRate: 48_000,
- evidenceSampleRate: 16_000,
- sourceSampleFrames: request.sourceSampleFrames,
- evidenceSampleFrames: request.evidenceSampleFrames,
- sourceOriginSample: 0,
- evidenceOriginSample: 0,
- },
- }) },
- conformance: "exact",
- delivery: "executed",
- metadata: {},
- };
- },
- );
- const atWhisperX = await driver.run(parseBuildState(serializeBuildState(atEvidenceAudio.state)));
- assert.equal(atWhisperX.blocked[0]?.reason, "missing-endpoint");
- assert.match(atWhisperX.blocked[0]?.subject ?? "", /WhisperXAlignmentEvidence/u);
-
- const wrongMeasurementEndpoints = new EndpointRegistry();
- wrongMeasurementEndpoints.registerImmediateEndpoint(
- "runtime:wrong-measurement-whisperx",
- whisperXCapabilities.alignment,
- whisperXTypes.alignmentEvidence,
- () => ({
- value: {
- kind: "inline",
- value: {
- contract: "invalid-whisperx-evidence",
- durationSec: 1,
- segments: [{ sourceSegmentId: "line", startSec: 0, endSec: 1, words: [], chars: [] }],
- },
- },
- conformance: "exact",
- delivery: "executed",
- metadata: {},
- }),
- );
- const rejectedEvidence = await new NodeDriver({
- producers: host,
- endpoints: wrongMeasurementEndpoints,
- validators: validatorRegistry(),
- }).run(
- parseBuildState(serializeBuildState(atWhisperX.state)),
- );
- assert.equal(rejectedEvidence.status, "paused");
- assert.match(rejectedEvidence.journal.at(-1)?.message ?? "", /contract/u);
- const rejectedWhisperNeed = rejectedEvidence.state.needs.find((need) =>
- sameCapability(need.capability, whisperXCapabilities.alignment));
- assert(rejectedWhisperNeed);
- assert.equal(
- rejectedEvidence.state.records.some((record) =>
- record.id === rejectedWhisperNeed.result),
- false,
- "Invalid Evidence must not enter BuildState",
- );
-
- endpoints.registerImmediateEndpoint(
- "runtime:whisperx-local",
- whisperXCapabilities.alignment,
- whisperXTypes.alignmentEvidence,
- () => {
- return {
- value: {
- kind: "inline",
- value: sealWhisperXAlignmentEvidence({
- contract: "svml.whisperx-alignment-evidence@1",
- durationSec: 1,
- segments: [{
- sourceSegmentId: "line",
- startSec: 0,
- endSec: 1,
- words: [
- { text: "what", startSec: 0.1, endSec: 0.25, score: 0.98 },
- { text: "the", startSec: 0.3, endSec: 0.42, score: 0.98 },
- { text: "fuck", startSec: 0.48, endSec: 0.72, score: 0.97 },
- ],
- chars: [],
- }],
- }),
- },
- conformance: "exact",
- delivery: "executed",
- metadata: { adapter: "whisperx.local", model: "fixture" },
- };
- },
- );
- const completed = await driver.run(parseBuildState(serializeBuildState(atWhisperX.state)));
-
- assert.equal(completed.status, "complete");
- assert.deepEqual(calls, {
- estimate: 1,
- seedance: 1,
- assemble: 1,
- projectAudio: 1,
- projectVisual: 0,
- evidenceAudioRequest: 1,
- whisperRequest: 1,
- whisperNormalize: 1,
- });
- assert.deepEqual(completed.state.receipts.map((receipt) => receipt.fulfiller), [
- "runtime:official-estimate",
- "runtime:kie-seedance-mini",
- "runtime:media-local",
- "runtime:whisperx-local",
- ]);
- const captionRecord = completed.state.records.find((record) =>
- record.id === selectedRecord(completed.state, videoOutputs.caption));
- assert.equal(captionRecord?.value.kind, "inline");
- const value = inlineValue<TimedCaptionProjection>(
- captionRecord?.value.kind === "inline" ? captionRecord.value.value : {},
- );
- assert.deepEqual(
- [value.cues[0]?.startSec, value.cues[0]?.endSec, value.cues[0]?.atoms.length],
- [0.1, 0.72, 1],
- );
-
- const mapRecordId = selectedRecord(completed.state, videoOutputs.map);
- const mapRecord = completed.state.records.find((record) => record.id === mapRecordId);
- assert.equal(mapRecord?.value.kind, "inline");
- const mapValue = inlineObject(mapRecord?.value.kind === "inline" ? mapRecord.value.value : {});
- for (const forbiddenLineageField of [
- "narrativeDigest",
- "semanticIndexDigest",
- "basisDigest",
- "audioArtifactDigest",
- "evidenceDigest",
- "locatorDigest",
- ]) {
- assert.equal(
- forbiddenLineageField in mapValue,
- false,
- `SemanticMap must not serialize Graph lineage as ${forbiddenLineageField}`,
- );
- }
- assert.equal(mapRecord?.origin.kind, "derived");
- const mapDerivation = completed.state.derivations.find((derivation) =>
- derivation.id === (mapRecord?.origin.kind === "derived" ? mapRecord.origin.derivation : undefined));
- assert(mapDerivation);
- const evidenceInput = mapDerivation.inputs.find((input) => {
- const record = completed.state.records.find((candidate) => candidate.id === input.id);
- return record?.type.name === speechEvidenceTypes.alignedTranscript.name
- && record.type.module.name === speechEvidenceTypes.alignedTranscript.module.name
- && record.type.module.version === speechEvidenceTypes.alignedTranscript.module.version;
- });
- assert(evidenceInput, "the Map fan-in must include the internal normalized Evidence Record");
- assert.deepEqual(
- new Set(mapDerivation.inputs.map((input) => input.id)),
- new Set([
- "narrative:root",
- selectedRecord(completed.state, videoOutputs.audio),
- evidenceInput.id,
- ]),
- "the fan-in belongs to the executed Graph/Derivation, not the SemanticMap payload",
- );
-});
-
-test("Targets prune official Fragments while shared SpeechBasis generation stays singular", () => {
- const visual = createVideoBuild({ targets: ["visual"] });
- assert.equal(producerCount(visual, videoProducers.requestEstimate), 1);
- assert.equal(producerCount(visual, videoProducers.requestSeedanceMini), 1);
- assert.equal(producerCount(visual, videoProducers.assembleBasis), 1);
- assert.equal(producerCount(visual, speechBasisProducers.projectVisual), 1);
- assert.equal(producerCount(visual, speechBasisProducers.projectAudio), 0);
- assert.equal(producerCount(visual, whisperXProducers.request), 0);
- assert.equal(producerCount(visual, captionProducers.temporalizePlan), 0);
-
- const map = createVideoBuild({ targets: ["map"] });
- assert.equal(producerCount(map, speechBasisProducers.projectAudio), 1);
- assert.equal(producerCount(map, speechBasisProducers.projectVisual), 0);
- assert.equal(producerCount(map, whisperXProducers.request), 1);
- assert.equal(producerCount(map, whisperXProducers.normalize), 1);
- assert.equal(producerCount(map, speechAlignmentProducers.locate), 1);
- assert.equal(producerCount(map, captionProducers.temporalizePlan), 0);
-
- const bothTakeProjections = createVideoBuild({ targets: ["audio", "visual"] });
- assert.equal(producerCount(bothTakeProjections, videoProducers.requestSeedanceMini), 1);
- assert.equal(producerCount(bothTakeProjections, videoProducers.assembleBasis), 1);
- assert.equal(producerCount(bothTakeProjections, speechBasisProducers.projectAudio), 1);
- assert.equal(producerCount(bothTakeProjections, speechBasisProducers.projectVisual), 1);
-
- const bothEvidenceExports = createVideoBuild({ targets: ["rawEvidence", "map"] });
- assert.equal(producerCount(bothEvidenceExports, whisperXProducers.request), 1);
- assert.equal(producerCount(bothEvidenceExports, whisperXProducers.normalize), 1);
- assert.equal(producerCount(bothEvidenceExports, speechAlignmentProducers.locate), 1);
-});
-
-test("an Existing SpeechBasis cuts generation while a visual substitute cuts the whole upstream", async () => {
- const fixture = createVideoFixture();
- const narrativeRecord = fixture.program.records.find((record) => record.id === "narrative:root");
- assert.equal(narrativeRecord?.value.kind, "inline");
- const narrative = inlineValue<Narrative>(
- narrativeRecord?.value.kind === "inline" ? narrativeRecord.value.value : {},
- );
- const programSpace = sealProgramSpace({
- contract: "svml.program-space@1",
- durationSec: 1,
- frameRate: { numerator: 1_000, denominator: 1 },
- });
- const visualDigest = digestOf("existing-take:visual");
- const existingTake = sealSpeechBasis({
- contract: "svml.speech-basis@1",
- programSpace,
- audio: {
- kind: "blob",
- digest: digestOf("existing-take:audio"),
- size: 1,
- mediaType: "audio/wav",
- },
- visualTrack: { clips: [{
- segmentId: "line",
- artifact: { kind: "blob", digest: visualDigest, size: 1, mediaType: "video/mp4" },
- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },
- frameRate: { ...programSpace.frameRate },
- frameCount: 1_000,
- }] },
- segments: [{ segmentId: "line", startSec: 0, endSec: 1 }],
- });
- const takeCandidate = createProvidedCandidate({
- type: speechTypes.basis,
- value: { kind: "inline", value: existingTake },
- provenance: { source: "approved-library", name: "opening-v3" },
- });
- const takeOverlay = sealRealizationOverlay({
- sourceGraph: fixture.source.id,
- candidates: [takeCandidate],
- operations: [],
- });
- const withExistingTake = resolveRealization(fixture.program, fixture.source, [takeOverlay]);
- const captionFromExisting = start(fixture.program, withExistingTake.graph, sealBuildRequest({
- graph: withExistingTake.graph.id,
- targets: [{ output: videoOutputs.caption, accepts: "exact" }],
- satisfactions: [{ output: videoOutputs.take, candidate: takeCandidate.id, fidelity: "exact" }],
- }));
- assert.equal(producerCount(captionFromExisting, videoProducers.requestEstimate), 0);
- assert.equal(producerCount(captionFromExisting, videoProducers.requestSeedanceMini), 0);
- assert.equal(producerCount(captionFromExisting, videoProducers.assembleBasis), 0);
- assert.equal(producerCount(captionFromExisting, speechBasisProducers.projectAudio), 1);
- assert.equal(producerCount(captionFromExisting, whisperXProducers.request), 1);
- assert.equal(producerCount(captionFromExisting, captionProducers.temporalizePlan), 1);
-
- const black: VisualTrack = sealVisualTrack({
- contract: "svml.visual-track@1",
- visualIr: "svml.visual-ir@1",
- id: "preview:black",
- presents: [],
- });
- const blackValue = { kind: "inline" as const, value: black };
- const blackValidation = await validateValue(
- fixture.program.closure,
- compositionTypes.visualTrack,
- blackValue,
- validatorRegistry(),
- );
- const blackCandidate = createProvidedCandidate({
- type: compositionTypes.visualTrack,
- value: blackValue,
- provenance: { method: "black-preview" },
- ...(blackValidation === undefined ? {} : { validation: blackValidation }),
- });
- const blackOverlay = sealRealizationOverlay({
- sourceGraph: fixture.source.id,
- candidates: [blackCandidate],
- operations: [],
- });
- const withBlack = resolveRealization(fixture.program, fixture.source, [blackOverlay]);
- const preview = start(fixture.program, withBlack.graph, sealBuildRequest({
- graph: withBlack.graph.id,
- targets: [{ output: videoOutputs.visual, accepts: "substitute" }],
- satisfactions: [{ output: videoOutputs.visual, candidate: blackCandidate.id, fidelity: "substitute" }],
- }));
- assert.deepEqual(preview.plan.steps, []);
- assert.equal(preview.plan.initialValues.length, 1);
-});
-
-test("an exact Endpoint result cannot wash a substitute input back to exact", async () => {
- const initial = createVideoBuild({
- estimateRealization: "placeholder",
- goalAccepts: "substitute",
- });
- const host = new ProducerRegistry();
- host.registerProducer(
- videoProducers.placeholderEstimate,
- videoImplementations.placeholderEstimate,
- () => ({ outputs: { estimate: { kind: "inline", value: { durationSec: 1 } } }, needs: {} }),
- );
- host.registerProducer(videoProducers.requestSeedanceMini, videoImplementations.requestSeedanceMini, () => ({
- outputs: {},
- needs: { media: { model: "mini" } },
- }));
- const endpoints = new EndpointRegistry();
- endpoints.registerImmediateEndpoint(
- "runtime:exact-seedance",
- videoCapabilities.seedanceMini,
- videoTypes.seedanceMiniMedia,
- () => ({
- value: { kind: "inline", value: {
- model: "mini",
- audioDigest: digestOf("floor:audio"),
- visualDigest: digestOf("floor:visual"),
- durationSec: 1,
- } },
- conformance: "exact",
- delivery: "executed",
- metadata: {},
- }),
- );
-
- const result = await new NodeDriver({
- producers: host,
- endpoints,
- validators: validatorRegistry(),
- }).run(initial);
- assert.equal(result.status, "paused", "the unregistered Basis assembler should pause after Seedance");
- const seedanceNeed = result.state.needs.find((need) =>
- sameCapability(need.capability, videoCapabilities.seedanceMini));
- const seedanceReceipt = result.state.receipts.find((receipt) => receipt.need === seedanceNeed?.id);
- const media = result.state.records.find((record) => record.id === seedanceReceipt?.output);
- assert.equal(seedanceNeed?.conformanceFloor, "substitute");
- assert.equal(seedanceReceipt?.fulfillmentConformance, "exact");
- assert.equal(seedanceReceipt?.conformance, "substitute");
- assert.equal(media?.conformance, "substitute");
-});
-
-test("an exact Target rejects an explicitly selected substitute realization", () => {
- assert.throws(
- () => createVideoBuild({ estimateRealization: "placeholder" }),
- /selects a substitute path/u,
- );
-});
diff --git a/packages/speech/src/identity.ts b/packages/speech/src/identity.ts
index c2f5b650..7e131932 100644
--- a/packages/speech/src/identity.ts
+++ b/packages/speech/src/identity.ts
@@ -1,6 +1,6 @@
import { isDigest } from "@narratage/protocol";
import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";
-import { assertIntrinsicExtent } from "@narratage/spatial";
+import { assertContentFit, assertIntrinsicExtent, assertSpatialFrame } from "@narratage/spatial";
import type { SpeechAudioBasis, SpeechBasis, SpeechDuration, SpeechEvidenceAudio } from "./types.js";
function assertAudioBlob(value: SpeechBasis["audio"], label: string): void {
if (value.kind !== "blob" || !isDigest(value.digest) || !Number.isSafeInteger(value.size)
@@ -34,39 +34,48 @@ export function assertSpeechBasisIdentity(basis: SpeechBasis): void {
if (basis.contract !== "svml.speech-basis@1") throw new Error("Unsupported SpeechBasis contract.");
assertProgramSpaceIdentity(basis.programSpace);
assertAudioBlob(basis.audio, "SpeechBasis audio");
- if (basis.segments.length === 0 || basis.visualTrack.clips.length !== basis.segments.length) {
- throw new Error("SpeechBasis must pair every Segment with one visual clip.");
- }
+ if (basis.segments.length === 0) throw new Error("SpeechBasis must contain at least one Segment.");
let previousEnd = 0;
- let accumulatedFrames = 0;
const frameAt = (seconds: number): number => Math.round(
seconds * basis.programSpace.frameRate.numerator / basis.programSpace.frameRate.denominator,
);
- for (const [index, segment] of basis.segments.entries()) {
- const clip = basis.visualTrack.clips[index]!;
+ const segmentSpans = new Map<string, { readonly startFrame: number; readonly endFrameExclusive: number }>();
+ for (const segment of basis.segments) {
if (!segment.segmentId || segment.startSec !== previousEnd || !Number.isFinite(segment.endSec)
|| segment.endSec <= segment.startSec || segment.endSec > basis.programSpace.durationSec) {
throw new Error("SpeechBasis Segment is invalid or non-contiguous.");
}
- if (clip.segmentId !== segment.segmentId || clip.artifact.kind !== "blob" || !isDigest(clip.artifact.digest)
+ if (segmentSpans.has(segment.segmentId)) throw new Error(`SpeechBasis repeats Segment ${segment.segmentId}.`);
+ segmentSpans.set(segment.segmentId, {
+ startFrame: frameAt(segment.startSec),
+ endFrameExclusive: frameAt(segment.endSec),
+ });
+ previousEnd = segment.endSec;
+ }
+ if (frameAt(previousEnd) !== programSpaceFrameCount(basis.programSpace)) {
+ throw new Error("SpeechBasis Segments do not cover ProgramSpace.");
+ }
+ const seenVisuals = new Set<string>();
+ for (const clip of basis.visualTrack.clips) {
+ const segmentSpan = segmentSpans.get(clip.segmentId);
+ if (segmentSpan === undefined || seenVisuals.has(clip.segmentId)
+ || clip.artifact.kind !== "blob" || !isDigest(clip.artifact.digest)
|| !Number.isSafeInteger(clip.artifact.size) || clip.artifact.size < 0
|| !clip.artifact.mediaType.startsWith("video/")
|| clip.frameRate.numerator !== basis.programSpace.frameRate.numerator
|| clip.frameRate.denominator !== basis.programSpace.frameRate.denominator
- || !Number.isSafeInteger(clip.frameCount) || clip.frameCount <= 0) {
+ || !Number.isSafeInteger(clip.frameCount) || clip.frameCount <= 0
+ || !Number.isSafeInteger(clip.span.startFrame) || !Number.isSafeInteger(clip.span.endFrameExclusive)
+ || clip.span.startFrame !== segmentSpan?.startFrame
+ || clip.span.endFrameExclusive !== segmentSpan?.endFrameExclusive
+ || clip.frameCount !== clip.span.endFrameExclusive - clip.span.startFrame
+ || !Number.isSafeInteger(clip.stackingOrder)) {
throw new Error(`SpeechBasis visual clip ${clip.segmentId} is invalid.`);
}
+ seenVisuals.add(clip.segmentId);
assertIntrinsicExtent(clip.extent);
- const startFrame = frameAt(segment.startSec);
- const endFrame = frameAt(segment.endSec);
- if (startFrame !== accumulatedFrames || endFrame - startFrame !== clip.frameCount) {
- throw new Error(`SpeechBasis visual clip ${clip.segmentId} does not cover its Segment.`);
- }
- accumulatedFrames = endFrame;
- previousEnd = segment.endSec;
- }
- if (accumulatedFrames !== programSpaceFrameCount(basis.programSpace)) {
- throw new Error("SpeechBasis visuals do not cover ProgramSpace.");
+ assertSpatialFrame(clip.frame);
+ assertContentFit(clip.fit);
}
}
export function assertSpeechAudioBasisIdentity(basis: SpeechAudioBasis): void {
diff --git a/packages/speech/src/schema.ts b/packages/speech/src/schema.ts
index bd497935..d4142753 100644
--- a/packages/speech/src/schema.ts
+++ b/packages/speech/src/schema.ts
@@ -1,6 +1,6 @@
import { programSpaceSchema } from "@narratage/program-space";
import type { ValueSchema } from "@narratage/protocol";
-import { intrinsicExtentSchema } from "@narratage/spatial";
+import { contentFitSchema, intrinsicExtentSchema, spatialFrameSchema } from "@narratage/spatial";
const string = { kind: "string", minLength: 1 } as const;
const number = { kind: "number", minimum: 0 } as const;
const integer = { kind: "number", integer: true, minimum: 0 } as const;
@@ -18,10 +18,15 @@ export const speechBasisSchema: ValueSchema = object({
contract: { schema: { kind: "literal", value: "svml.speech-basis@1" } }, programSpace: { schema: programSpaceSchema }, audio: { schema: audioBlobRef },
visualTrack: { schema: object({ clips: { schema: { kind: "array", items: object({ segmentId: { schema: string }, artifact: { schema: object({
kind: { schema: { kind: "literal", value: "blob" } }, digest: { schema: digest }, size: { schema: integer }, mediaType: { schema: string },
+ }) }, span: { schema: object({
+ startFrame: { schema: integer }, endFrameExclusive: { schema: integer },
}) }, extent: { schema: intrinsicExtentSchema }, frameRate: { schema: object({
numerator: { schema: { kind: "number", integer: true, minimum: 1 } },
denominator: { schema: { kind: "number", integer: true, minimum: 1 } },
- }) }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } } }) } } }) },
+ }) }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } },
+ frame: { schema: spatialFrameSchema }, fit: { schema: contentFitSchema },
+ stackingOrder: { schema: { kind: "number", integer: true } },
+ }) } } }) },
segments: { schema: { kind: "array", minItems: 1, items: segment } },
});
export const speechAudioBasisSchema: ValueSchema = object({ contract: { schema: { kind: "literal", value: "svml.speech-audio-basis@1" } },
diff --git a/packages/speech/src/types.ts b/packages/speech/src/types.ts
index 9b8a2e3d..c3298817 100644
--- a/packages/speech/src/types.ts
+++ b/packages/speech/src/types.ts
@@ -1,16 +1,20 @@
import type { ProgramSpace } from "@narratage/program-space";
import type { BlobRef } from "@narratage/protocol";
-import type { IntrinsicExtent } from "@narratage/spatial";
+import type { ContentFit, IntrinsicExtent, SpatialFrame } from "@narratage/spatial";
export type SpeechDuration = { readonly contract: "svml.speech-duration@1"; readonly durationSec: number };
export type SpeechBasisSegment = { readonly segmentId: string; readonly startSec: number; readonly endSec: number };
export type SpeechBasis = {
readonly contract: "svml.speech-basis@1"; readonly programSpace: ProgramSpace; readonly audio: BlobRef;
readonly visualTrack: { readonly clips: readonly {
readonly segmentId: string;
+ readonly span: { readonly startFrame: number; readonly endFrameExclusive: number };
readonly artifact: BlobRef;
readonly extent: IntrinsicExtent;
readonly frameRate: { readonly numerator: number; readonly denominator: number };
readonly frameCount: number;
+ readonly frame: SpatialFrame;
+ readonly fit: ContentFit;
+ readonly stackingOrder: number;
}[] };
readonly segments: readonly SpeechBasisSegment[];
};
diff --git a/packages/temporal/src/location.ts b/packages/temporal/src/location.ts
index 8bd5c01c..5c1ae690 100644
--- a/packages/temporal/src/location.ts
+++ b/packages/temporal/src/location.ts
@@ -1,12 +1,13 @@
-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";
import type { ProgramSpace } from "@narratage/program-space";
-import { momentFrames, selectionFrameSpans } from "@narratage/semantic-map";
+import { momentFrames, segmentFrameSpan, selectionFrameSpans } from "@narratage/semantic-map";
import type { CompleteSemanticMap } from "@narratage/semantic-map";
import type {
LocatedMomentOccurrence,
LocatedProgramOccurrence,
+ LocatedSegmentOccurrence,
LocatedSelectionOccurrence,
} from "./types.js";
@@ -79,3 +80,21 @@ export function locateProgramOccurrence(space: ProgramSpace): LocatedProgramOccu
assertProgramSpaceIdentity(space);
return { id: "program", start: { frame: 0 }, end: { frame: programSpaceFrameCount(space) } };
}
+
+export function locateSegmentOccurrence(
+ map: CompleteSemanticMap,
+ segment: NarrativeExcerpt,
+ space: ProgramSpace,
+): LocatedSegmentOccurrence {
+ assertProgramSpaceIdentity(space);
+ const totalFrames = programSpaceFrameCount(space);
+ const span = segmentFrameSpan(map, segment, space);
+ assertLocatedFrame(span.startFrame, totalFrames, `Narrative Segment ${segment.id} start`);
+ assertLocatedFrame(span.endFrameExclusive, totalFrames, `Narrative Segment ${segment.id} end`);
+ if (span.endFrameExclusive <= span.startFrame) throw new Error(`Narrative Segment ${segment.id} has no positive frame span.`);
+ return {
+ id: segment.id,
+ start: { frame: span.startFrame },
+ end: { frame: span.endFrameExclusive },
+ };
+}
diff --git a/packages/temporal/src/projection.ts b/packages/temporal/src/projection.ts
index 59b780e8..bae800e9 100644
--- a/packages/temporal/src/projection.ts
+++ b/packages/temporal/src/projection.ts
@@ -1,9 +1,9 @@
-import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
+import type { NarrativeExcerpt, NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";
import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";
import type { ProgramSpace } from "@narratage/program-space";
import type { CompleteSemanticMap } from "@narratage/semantic-map";
-import { locateMomentOccurrences, locateProgramOccurrence, locateSelectionOccurrences } from "./location.js";
+import { locateMomentOccurrences, locateProgramOccurrence, locateSegmentOccurrence, locateSelectionOccurrences } from "./location.js";
import {
add,
compare,
@@ -18,6 +18,7 @@ import type {
FrameSpan,
LocatedMomentOccurrence,
LocatedProgramOccurrence,
+ LocatedSegmentOccurrence,
LocatedSelectionOccurrence,
OccurrenceExpansion,
ProjectedOccurrence,
@@ -30,6 +31,7 @@ type PointEnvironment = {
readonly program: LocatedProgramOccurrence;
readonly selection?: LocatedSelectionOccurrence;
readonly moment?: LocatedMomentOccurrence;
+ readonly segment?: LocatedSegmentOccurrence;
};
function usesLocalPoint(expression: TemporalPointExpression, source: "selection" | "moment"): boolean {
@@ -82,6 +84,16 @@ function evaluatePoint(
base = environment.selection.end.frame;
break;
}
+ case "segment.start": {
+ if (environment.segment === undefined) throw new Error("segment.start requires a Segment occurrence.");
+ base = environment.segment.start.frame;
+ break;
+ }
+ case "segment.end": {
+ if (environment.segment === undefined) throw new Error("segment.end requires a Segment occurrence.");
+ base = environment.segment.end.frame;
+ break;
+ }
case "moment.cue": {
if (environment.moment === undefined) throw new Error("moment.cue requires a Moment occurrence.");
base = environment.moment.cue.frame;
@@ -183,6 +195,22 @@ export function projectProgramWindow(input: {
};
}
+export function projectSegmentWindow(input: {
+ readonly itemId: string;
+ readonly map: CompleteSemanticMap;
+ readonly segment: NarrativeExcerpt;
+ readonly space: ProgramSpace;
+ readonly projection: TemporalWindowProjection;
+}): ProjectedOccurrence {
+ const program = locateProgramOccurrence(input.space);
+ const segment = locateSegmentOccurrence(input.map, input.segment, input.space);
+ return {
+ id: projectedId(input.itemId, segment.id),
+ sourceOccurrenceId: segment.id,
+ span: projectTemporalWindow(input.projection, { program, segment }, input.space),
+ };
+}
+
export function assertWindowRelation(
occurrences: readonly ProjectedOccurrence[],
relation: WindowRelation,
diff --git a/packages/temporal/src/types.ts b/packages/temporal/src/types.ts
index ff703888..f848ba09 100644
--- a/packages/temporal/src/types.ts
+++ b/packages/temporal/src/types.ts
@@ -21,6 +21,12 @@ export type LocatedProgramOccurrence = {
readonly end: FramePoint;
};
+export type LocatedSegmentOccurrence = {
+ readonly id: string;
+ readonly start: FramePoint;
+ readonly end: FramePoint;
+};
+
/** Exact author duration. Decimal source spelling is reduced before reaching this value. */
export type TemporalDuration =
| { readonly unit: "frames"; readonly value: number }
@@ -32,6 +38,8 @@ export type TemporalPointExpression =
| { readonly ref: "program.end"; readonly offset?: TemporalDuration }
| { readonly ref: "selection.start"; readonly offset?: TemporalDuration }
| { readonly ref: "selection.end"; readonly offset?: TemporalDuration }
+ | { readonly ref: "segment.start"; readonly offset?: TemporalDuration }
+ | { readonly ref: "segment.end"; readonly offset?: TemporalDuration }
| { readonly ref: "moment.cue"; readonly offset?: TemporalDuration }
| { readonly ref: "absolute"; readonly at: TemporalDuration };
diff --git a/packages/temporal/test/temporal.test.ts b/packages/temporal/test/temporal.test.ts
index 02567d0a..b0b9b7e1 100644
--- a/packages/temporal/test/temporal.test.ts
+++ b/packages/temporal/test/temporal.test.ts
@@ -11,6 +11,7 @@ import {
locateSelectionOccurrences,
projectMomentWindows,
projectProgramWindow,
+ projectSegmentWindow,
projectSelectionWindows,
resolveTriggeredSchedule,
temporalDurationInSamples,
@@ -31,6 +32,8 @@ const map: CompleteSemanticMap = {
{ identity: "c", timeSec: 3, frame: 90 },
{ identity: "d", timeSec: 4, frame: 120 },
{ identity: "late", timeSec: 8, frame: 240 },
+ { identity: "segment:answer:start", timeSec: 2, frame: 60 },
+ { identity: "segment:answer:end", timeSec: 4, frame: 120 },
],
};
@@ -59,6 +62,21 @@ test("one Selection projects exact local points and stable occurrence identity",
}]);
});
+test("one Segment projects from its own structural start and end anchors", () => {
+ const result = projectSegmentWindow({
+ itemId: "answer-card",
+ map,
+ segment: { contract: "svml.narrative-excerpt@1", kind: "segment", id: "answer", tokenStart: 0, tokenEndExclusive: 1 },
+ space,
+ projection: { start: { ref: "segment.start" }, end: { ref: "segment.end" } },
+ });
+ assert.deepEqual(result, {
+ id: "answer-card::answer",
+ sourceOccurrenceId: "answer",
+ span: { startFrame: 60, endFrameExclusive: 120 },
+ });
+});
+
test("each preserves source order, even when physical time is reversed between occurrences", () => {
const result = projectSelectionWindows({
itemId: "repeat",
diff --git a/packages/whisperx/test/whisperx.test.ts b/packages/whisperx/test/whisperx.test.ts
index d11fd3c0..18573973 100644
--- a/packages/whisperx/test/whisperx.test.ts
+++ b/packages/whisperx/test/whisperx.test.ts
@@ -24,18 +24,11 @@ function basis() {
durationSec: 1,
frameRate: { numerator: 30, denominator: 1 },
});
- const visual = digestOf("whisperx-test:visual");
return sealSpeechBasis({
contract: "svml.speech-basis@1",
programSpace,
audio: { kind: "blob", digest: digestOf("whisperx-test:audio"), size: 1, mediaType: "audio/wav" },
- visualTrack: { clips: [{
- segmentId: "line",
- artifact: { kind: "blob", digest: visual, size: 1, mediaType: "video/mp4" },
- extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },
- frameRate: { ...programSpace.frameRate },
- frameCount: 30,
- }] },
+ visualTrack: { clips: [] },
segments: [{ segmentId: "line", startSec: 0, endSec: 1 }],
});
}
diff --git a/spec/media-track.md b/spec/media-track.md
index 5ebd817f..c068c564 100644
--- a/spec/media-track.md
+++ b/spec/media-track.md
@@ -514,11 +514,11 @@ Speech Spine should stop maintaining a second ad-hoc media renderer. Its visual
same Media lowering implementation with a deliberately restricted generated Program:
```text
-one normalized A/V take per speech Segment
+one normalized speech-bearing Take per Segment; its visual stream is optional
exact already-established Segment frame span
-Canvas Placement Frame
+explicit authored Spatial Frame
one foreground layer
-explicit contain/cover fit chosen by the speech package
+explicit fit and absolute stacking order chosen by the author
muted visual
cut between contiguous takes
no entry, sustain or exit motion
@@ -528,7 +528,10 @@ no source-audio projection
The Speech author Surface accepts raw `video=` and expands the same generic inspection and
normalization graph before assembly, selecting primary moving video plus default audio at the
-Spine's explicit frame rate. It also accepts prepared `media=` directly. After assembly, Speech
+Spine's explicit frame rate. Raw `audio=` expands an audio-authoritative normalization graph and
+contributes no visual Present; it never manufactures black media. Prepared `media=` remains a
+direct exact input. Visual Takes inherit the Spine's explicit `visual-frame`,
+`visual-appearance` and `visual-z`, with per-Take overrides for those same limited axes. After assembly, Speech
audio remains the separate canonical `SpeechAudioBasis -> AudioTrack` projection; the restricted
visual lowerer itself remains muted and does not rediscover container audio.
@@ -540,6 +543,10 @@ still does not register Media meaning.
The author imports Speech Spine, whose manifest declares its package dependency. Authors do not
need to import a second parser merely because the implementation reuses Media lowering.
+Ordinary Media Items may consume `during={story.segment.answer}` directly. This is not an implicit
+Selection: Temporal resolves the Segment's already-existing structural start/end anchors. The same
+Item can still consume authored Selection and Moment values when those are the actual intent.
+
## 14. Legacy audit: retained and retired
Retain:
@@ -705,8 +712,8 @@ The matrix above is closed by executable evidence rather than by package status
- `packages/hyperframes/test/browser-visual.test.ts` renders opaque images, a straight-alpha
Surface, two-frame fitting, local motion and a Sequence handoff through real Chromium, then
compares every decoded RGBA frame across different worker partitions;
-- `tools/package-boundaries.test.mjs` and `tools/graph-first-value-boundary.test.mjs` enforce the
- absence of a Media family registry and of hidden lineage/provider metadata in values.
+- package manifests contain no Media family registry, while the public value types contain no
+ hidden lineage or Provider metadata.
The local package is therefore complete as a pre-release Media implementation over the frozen
repository-internal `svml.visual-track@1` / `svml.visual-ir@1` waist. This does not publish or
diff --git a/spec/track-authoring.md b/spec/track-authoring.md
index e1621aa7..a11ed8d4 100644
--- a/spec/track-authoring.md
+++ b/spec/track-authoring.md
@@ -412,8 +412,9 @@ Audio Track ─────────┘
```
Caption's component-owned timed Cue projection may enter the final frame-window stage directly.
-Speech Spine establishes ProgramSpace and contiguous speech takes rather than pretending those
-structural cuts are Narrative Selection consumption.
+Speech Spine establishes ProgramSpace and contiguous speech takes. Other Tracks may address one
+whole Narrative Segment through its two structural anchors directly, without pretending it is an
+authored Selection.
## 8. Migration consequences
diff --git a/tools/caption-playground-sha256.test.mjs b/tools/caption-playground-sha256.test.mjs
deleted file mode 100644
index d772c740..00000000
--- a/tools/caption-playground-sha256.test.mjs
+++ /dev/null
@@ -1,17 +0,0 @@
-import assert from "node:assert/strict";
-import { createHash } from "node:crypto";
-import test from "node:test";
-
-import { createHash as shimCreateHash } from "./caption-playground/src/shims/node-crypto.ts";
-
-const cases = ["", "abc", "x".repeat(55), "x".repeat(56), "x".repeat(64), "字幕轨道与画布"];
-test("Caption Playground browser sha256 agrees with Node", () => {
- for (const value of cases) {
- assert.equal(shimCreateHash("sha256").update(value).digest("hex"), createHash("sha256").update(value).digest("hex"));
- }
-});
-
-test("Caption Playground browser sha256 rejects unsupported operations", () => {
- assert.throws(() => shimCreateHash("sha512"), /sha256/u);
- assert.throws(() => shimCreateHash("sha256").update("a").digest("base64"), /hex/u);
-});
diff --git a/tools/caption-playground-source-bytes.test.mjs b/tools/caption-playground-source-bytes.test.mjs
deleted file mode 100644
index 2809f841..00000000
--- a/tools/caption-playground-source-bytes.test.mjs
+++ /dev/null
@@ -1,25 +0,0 @@
-import assert from "node:assert/strict";
-import { execFileSync } from "node:child_process";
-import { existsSync, readFileSync } from "node:fs";
-import test from "node:test";
-import { fileURLToPath } from "node:url";
-
-const root = fileURLToPath(new URL("..", import.meta.url));
-const paths = execFileSync("git", ["ls-files", "-c", "-o", "--exclude-standard", "tools/caption-playground", "test"], {
- cwd: root, encoding: "utf8",
-}).trim().split("\n").filter((path) => /\.(?:ts|mjs|js|html|css|json)$/u.test(path));
-
-test("Caption Playground source carries no raw control byte", () => {
- const offenders = [];
- for (const path of paths) {
- const file = new URL(path, new URL("..", import.meta.url));
- if (!existsSync(file)) continue;
- for (const [index, byte] of readFileSync(file).entries()) {
- if (byte < 0x20 && byte !== 0x09 && byte !== 0x0a && byte !== 0x0d) {
- offenders.push(`${path}: 0x${byte.toString(16)} at ${index}`);
- break;
- }
- }
- }
- assert.deepEqual(offenders, []);
-});
diff --git a/tools/graph-first-value-boundary.test.mjs b/tools/graph-first-value-boundary.test.mjs
deleted file mode 100644
index e371d083..00000000
--- a/tools/graph-first-value-boundary.test.mjs
+++ /dev/null
@@ -1,118 +0,0 @@
-import assert from "node:assert/strict";
-import {
- readdir,
- readFile,
-} from "node:fs/promises";
-import test from "node:test";
-
-const packageRoot = new URL("../packages/", import.meta.url);
-
-async function sourceFiles(root) {
- const files = [];
- for (const entry of await readdir(root, { withFileTypes: true })) {
- const path = new URL(`./${entry.name}${entry.isDirectory() ? "/" : ""}`, root);
- if (entry.isDirectory()) files.push(...await sourceFiles(path));
- else if (entry.isFile() && entry.name.endsWith(".ts")) files.push(path);
- }
- return files;
-}
-
-async function contents(packageNames) {
- const result = [];
- for (const name of packageNames) {
- const root = new URL(`./${name}/src/`, packageRoot);
- for (const path of await sourceFiles(root)) {
- result.push({ path, source: await readFile(path, "utf8") });
- }
- }
- return result;
-}
-
-const domainValuePackages = [
- "media-track",
- "caption",
- "caption-gemini",
- "estimate",
- "film",
- "generation",
- "hyperframes",
- "render-hyperframes",
- "image-transform",
- "media-pipeline",
- "text",
- "script",
- "seedance",
- "speech-alignment",
- "speech-spine",
- "speech-basis",
- "typography-track",
- "whisperx",
-];
-
-const forbiddenLineageFields = [
- "programSpaceDigest",
- "narrativeDigest",
- "mapDigest",
- "selectionDigest",
- "excerptDigest",
- "basisDigest",
- "resultDigest",
- "productDigest",
- "planDigest",
- "captionProgramDigest",
- "sourceAudioArtifactDigest",
- "renderInputDigest",
- "visualDigest",
- "audioDigest",
- "inspectionDigest",
- "previousSetDigest",
- "trackDigest",
-];
-
-test("domain Products do not copy graph lineage into payload fields", async () => {
- for (const { path, source } of await contents(domainValuePackages)) {
- for (const field of forbiddenLineageFields) {
- assert.doesNotMatch(source, new RegExp(`\\b${field}\\b`, "u"), `${path.pathname} reintroduced ${field}`);
- }
- assert.doesNotMatch(
- source,
- /readonly\s+digest\s*:\s*Digest\b/u,
- `${path.pathname} reintroduced a domain Product self-digest`,
- );
- }
-});
-
-test("the generic graph stack has no payload-affinity side channel", async () => {
- const genericSources = await contents(["protocol", "core", "elaborator"]);
- const removedApi = [
- "ResultAffinityDeclaration",
- "AffinityConstraint",
- "FragmentAffinityConstraint",
- "verifyRecordAffinity",
- ];
- for (const { path, source } of genericSources) {
- for (const symbol of removedApi) {
- assert.doesNotMatch(source, new RegExp(`\\b${symbol}\\b`, "u"), `${path.pathname} reintroduced ${symbol}`);
- }
- }
-});
-
-test("fold values retain members, not hidden identity or policy context", async () => {
- const declarations = [
- ["media-track", "types.ts", "MediaTrackSet"],
- ["film", "types.ts", "FilmTrackSet"],
- ["speech-spine", "types.ts", "SpeechSpineSet"],
- ["typography-track", "types.ts", "TypographyTrackSet"],
- ];
- for (const [packageName, fileName, typeName] of declarations) {
- const path = new URL(`./${packageName}/src/${fileName}`, packageRoot);
- const source = await readFile(path, "utf8");
- const declaration = new RegExp(`export type ${typeName} = \\{([\\s\\S]*?)\\n\\};`, "u").exec(source)?.[1];
- assert.ok(declaration, `${typeName} declaration is missing`);
- assert.doesNotMatch(
- declaration,
- /readonly\s+(?:id|digest|program|programSpace)\s*:/u,
- `${typeName} carries hidden identity or policy context`,
- );
- }
-});
diff --git a/tools/package-boundaries.test.mjs b/tools/package-boundaries.test.mjs
deleted file mode 100644
index 135116c9..00000000
--- a/tools/package-boundaries.test.mjs
+++ /dev/null
@@ -1,343 +0,0 @@
-import assert from "node:assert/strict";
-import {
- readdir,
- readFile,
-} from "node:fs/promises";
-import test from "node:test";
-
-const packageRoot = new URL("../packages/", import.meta.url);
-
-async function workspacePackages() {
- const manifests = new Map();
- for (const entry of await readdir(packageRoot, { withFileTypes: true })) {
- if (!entry.isDirectory()) continue;
- const path = new URL(`./${entry.name}/package.json`, packageRoot);
- let manifest;
- try {
- manifest = JSON.parse(await readFile(path, "utf8"));
- } catch (error) {
- if (error?.code === "ENOENT") continue;
- throw error;
- }
- manifests.set(manifest.name, manifest);
- }
- return manifests;
-}
-
-async function sourceFiles(directory) {
- const found = [];
- let entries;
- try {
- entries = await readdir(directory, { withFileTypes: true });
- } catch (error) {
- if (error?.code === "ENOENT") return found;
- throw error;
- }
- for (const entry of entries) {
- const path = new URL(`./${entry.name}${entry.isDirectory() ? "/" : ""}`, directory);
- if (entry.isDirectory()) found.push(...await sourceFiles(path));
- else if (entry.name.endsWith(".ts")) found.push(path);
- }
- return found;
-}
-
-function productionGraph(packages) {
- return new Map([...packages].map(([name, manifest]) => [
- name,
- Object.keys({
- ...(manifest.dependencies ?? {}),
- ...(manifest.optionalDependencies ?? {}),
- }).filter((dependency) => packages.has(dependency)),
- ]));
-}
-
-function transitive(graph, entry) {
- const found = new Set();
- const visit = (name) => {
- for (const dependency of graph.get(name) ?? []) {
- if (found.has(dependency)) continue;
- found.add(dependency);
- visit(dependency);
- }
- };
- visit(entry);
- return found;
-}
-
-const domainNeutralPackages = new Set([
- "@narratage/protocol",
- "@narratage/artifact",
- "@narratage/core",
- "@narratage/source",
- "@narratage/elaborator",
- "@narratage/run",
- "@narratage/validation",
- "@narratage/host",
- "@narratage/compiler-node",
- "@narratage/workspace-fs-node",
- "@narratage/component-kit",
- "@narratage/runtime",
- "@narratage/runtime-adapter",
- "@narratage/runtime-adapter-node",
- "@narratage/endpoint-kit",
- "@narratage/driver-node",
- "@narratage/package-loader-node",
- "@narratage/store-sqlite",
- "@narratage/artifact-store-fs",
- "@narratage/artifact-store-s3",
- "@narratage/credential-store-env",
- "@narratage/credential-store-keychain",
- "@narratage/transport",
- "@narratage/transport-aws-lambda",
- "@narratage/local",
-]);
-
-test("official production packages have no dependency cycle", async () => {
- const graph = productionGraph(await workspacePackages());
- const visiting = new Set();
- const complete = new Set();
- const visit = (name, trail) => {
- if (visiting.has(name)) assert.fail(`production dependency cycle: ${[...trail, name].join(" -> ")}`);
- if (complete.has(name)) return;
- visiting.add(name);
- for (const dependency of graph.get(name) ?? []) visit(dependency, [...trail, name]);
- visiting.delete(name);
- complete.add(name);
- };
- for (const name of graph.keys()) visit(name, []);
-});
-
-test("Caption Playground cannot add semantics or metadata to production packages", async () => {
- let productionSource = "";
- for (const entry of await readdir(packageRoot, { withFileTypes: true })) {
- if (!entry.isDirectory()) continue;
- const sourceRoot = new URL(`./${entry.name}/src/`, packageRoot);
- for (const file of await sourceFiles(sourceRoot)) {
- const source = await readFile(file, "utf8");
- productionSource += source;
- assert.ok(!/\bplayground\b/iu.test(source),
- `${file.pathname} mentions Playground; the dependency must point from the tool to production only`);
- }
- }
-
- const typeDeclarations = await readFile(new URL("./protocol/src/module.ts", packageRoot), "utf8");
- assert.ok(!/readonly\s+default\??\s*:/u.test(typeDeclarations),
- "TypeDeclaration must not carry preview defaults");
-
- const valueSchemas = await readFile(new URL("./protocol/src/value.ts", packageRoot), "utf8");
- assert.ok(!/readonly\s+format\??\s*:/u.test(valueSchemas),
- "ValueSchema must not carry presentation hints for a tool");
-
- const componentKit = await readFile(new URL("./component-kit/src/index.ts", packageRoot), "utf8");
- assert.ok(!/\bRecipeFacet\b|readonly\s+recipes\??\s*:/u.test(componentKit),
- "ComponentPackage must not grow an editor or Playground Recipe facet");
-
- for (const helper of [
- "defaultProgramSpace", "defaultCanvasSpace", "defaultFilmProgram", "defaultScreenOverlayProgram",
- ]) {
- assert.ok(!productionSource.includes(helper),
- `${helper} is a preview starting value and must stay outside production packages`);
- }
-});
-
-test("Caption Playground is a selected source editor, not a component registry", async () => {
- const root = new URL("../tools/caption-playground/src/", import.meta.url);
- const allowedPackages = new Set([
- "@narratage/caption",
- "@narratage/caption-fine",
- "@narratage/composition",
- "@narratage/fonts-open",
- "@narratage/host",
- "@narratage/narrative",
- "@narratage/package-loader-node",
- "@narratage/program-space",
- "@narratage/protocol",
- "@narratage/svs",
- "@narratage/video-cli",
- ]);
- for (const file of await sourceFiles(root)) {
- const source = await readFile(file, "utf8");
- for (const match of source.matchAll(/["'](@narratage\/[a-z0-9-]+)["']/gu)) {
- assert.ok(allowedPackages.has(match[1]), `${file.pathname} unexpectedly names ${match[1]}`);
- }
- assert.ok(!/\b(?:Story|Scenario|Recipe)Registry\b/u.test(source),
- `${file.pathname} introduces a parallel Caption Playground registry`);
- assert.ok(!/import\.meta\.glob|discoverPreviewProducers/u.test(source),
- `${file.pathname} scans the workspace instead of editing the explicitly selected Caption source`);
- }
-});
-
-test("Markup syntax, graph Text and video Typography keep distinct package identities", async () => {
- const packages = await workspacePackages();
- for (const name of [
- "@narratage/markup",
- "@narratage/run-markup",
- "@narratage/compiler-markup-node",
- "@narratage/text",
- "@narratage/typography-track",
- ]) assert.ok(packages.has(name), `${name} is absent`);
- for (const retired of [
- "@narratage/run-text",
- "@narratage/compiler-text-node",
- "@narratage/text-track",
- "@narratage/prompt-kit",
- ]) assert.ok(!packages.has(retired), `${retired} must stay retired`);
-});
-
-test("the declared domain-neutral distribution closes without syntax, AIGC or video packages", async () => {
- const packages = await workspacePackages();
- const graph = productionGraph(packages);
- for (const name of domainNeutralPackages) {
- assert.ok(packages.has(name), `domain-neutral package ${name} is absent`);
- const outside = [...transitive(graph, name)].filter((dependency) =>
- dependency.startsWith("@narratage/") && !domainNeutralPackages.has(dependency));
- assert.deepEqual(outside, [], `${name} reaches packages outside the domain-neutral distribution`);
- }
- assert.deepEqual(graph.get("@narratage/core"), ["@narratage/protocol"]);
-});
-
-test("a Provider reaches no exact-model package, so models and services stay N+M", async () => {
- const packages = await workspacePackages();
- const graph = productionGraph(packages);
- const models = [
- "@narratage/gemini-omni",
- "@narratage/gpt-image",
- "@narratage/grok-imagine",
- "@narratage/minimax-h3",
- "@narratage/nano-banana",
- "@narratage/seedance",
- "@narratage/seedream",
- ];
- for (const name of [...packages.keys()].filter((item) => item.startsWith("@narratage/provider-"))) {
- const reached = transitive(graph, name);
- assert.deepEqual(
- models.filter((model) => reached.has(model)),
- [],
- `${name} depends on an exact-model package; a Provider must map declared ports instead`,
- );
- }
- // The shared port and wire-mapping vocabulary is the only legitimate meeting point.
- assert.ok(transitive(graph, "@narratage/provider-kie").has("@narratage/generation"));
- assert.ok(transitive(graph, "@narratage/seedance").has("@narratage/generation"));
-});
-
-test("the speech time map is an opaque handle: no consumer reads its fields", async () => {
- const { readdir, readFile } = await import("node:fs/promises");
- const packages = new URL("../packages/", import.meta.url);
- // Locating is a lookup through semantic-map. A consumer that destructured the
- // map could sort, merge or clip located spans against one another; those are
- // the caller's decisions to make on its own values, never the map's to make
- // for it.
- const reach = /\bmap\.(tokens|anchors|segments)\b/u;
- for (const entry of await readdir(packages, { withFileTypes: true })) {
- if (!entry.isDirectory() || entry.name === "semantic-map" || entry.name === "speech-alignment") continue;
- const source = new URL(`./${entry.name}/src/`, packages);
- let files = [];
- try {
- files = await readdir(source);
- } catch {
- continue;
- }
- for (const file of files.filter((name) => name.endsWith(".ts"))) {
- const content = await readFile(new URL(file, source), "utf8");
- assert.ok(!reach.test(content),
- `${entry.name}/src/${file} reads a SemanticMap field directly; call a semantic-map lookup instead`);
- }
- }
-});
-
-test("Markup compilation is one explicit leaf assembly, not a Package Loader or Local Runtime dependency", async () => {
- const graph = productionGraph(await workspacePackages());
- assert.ok(transitive(graph, "@narratage/compiler-markup-node").has("@narratage/markup"));
- assert.ok(!transitive(graph, "@narratage/package-loader-node").has("@narratage/markup"));
- assert.ok(!transitive(graph, "@narratage/local").has("@narratage/markup"));
-});
-
-test("generic and video CLIs reach no Provider package and video CLI activates no author aggregate", async () => {
- const packages = await workspacePackages();
- const graph = productionGraph(packages);
- const dependencies = transitive(graph, "@narratage/cli");
- const providers = [...packages.keys()].filter((name) => name.startsWith("@narratage/provider-"));
- assert.deepEqual(providers.filter((name) => dependencies.has(name)), []);
- const videoDependencies = transitive(graph, "@narratage/video-cli");
- assert.deepEqual(providers.filter((name) => videoDependencies.has(name)), []);
- assert.ok(!videoDependencies.has("@narratage/script"));
- assert.ok(!videoDependencies.has("@narratage/media-track"));
- assert.ok(!videoDependencies.has("@narratage/typography-track"));
- assert.ok(!videoDependencies.has("@narratage/film"));
- assert.ok(videoDependencies.has("@narratage/cli"));
-});
-
-test("domain packages confine their Markup dependency to Surface and activation entries", async () => {
- const { readdir, readFile } = await import("node:fs/promises");
- const surfaceOnly = [
- "caption", "caption-gemini", "estimate", "film", "render-hyperframes",
- "image-transform", "media", "media-pipeline", "media-track", "seedance", "speech-spine", "whisperx",
- ];
- const allowed = new Set(["surface.ts", "activation.ts"]);
- for (const name of surfaceOnly) {
- const root = new URL(`../packages/${name}/src/`, import.meta.url);
- for (const file of await readdir(root)) {
- if (!file.endsWith(".ts") || allowed.has(file)) continue;
- const content = await readFile(new URL(file, root), "utf8");
- assert.ok(!content.includes("\"@narratage/markup\""),
- `${name}/src/${file} imports @narratage/markup outside its Surface boundary`);
- }
- }
-});
-
-test("every package is reachable: imported, activatable, or a declared entry point", async () => {
- const { readdir, readFile } = await import("node:fs/promises");
- const packages = await workspacePackages();
- // A CLI is nobody's dependency by construction; it is what a person runs.
- const entryPoints = new Set(["@narratage/video-cli", "@narratage/test-support"]);
-
- // Anything imported by non-test source anywhere in the workspace, including
- // the deployment services, which are real consumers even though they are not
- // workspace packages.
- const imported = new Set();
- const scan = async (directory, owner) => {
- let entries;
- try {
- entries = await readdir(directory, { withFileTypes: true });
- } catch {
- return;
- }
- for (const entry of entries) {
- const path = `${directory}/${entry.name}`;
- if (entry.isDirectory() && entry.name !== "node_modules") await scan(path, owner);
- else if (entry.name.endsWith(".ts")) {
- for (const match of (await readFile(path, "utf8")).matchAll(/"(@narratage\/[a-z0-9-]+)"/gu)) {
- if (match[1] !== owner) imported.add(match[1]);
- }
- }
- }
- };
- for (const name of packages.keys()) await scan(`packages/${name.replace("@narratage/", "")}/src`, name);
- await scan("services", undefined);
-
- // A package with an activation entry is loaded by the Package Loader from a
- // lock, never by an import, so having no importer is what it is supposed to
- // look like.
- const activatable = new Set([...packages]
- .filter(([, manifest]) => manifest.svml?.activation !== undefined)
- .map(([name]) => name));
-
- // A data-only author package can deliberately export self-describing Source
- // Modules without shipping executable activation code. Those exports are
- // public entry points just as surely as a CLI binary is.
- const sourceResources = new Set([...packages]
- .filter(([, manifest]) => manifest.exports !== null
- && typeof manifest.exports === "object"
- && Object.values(manifest.exports).some((target) => typeof target === "string" && target.endsWith(".svs")))
- .map(([name]) => name));
-
- const unreachable = [...packages.keys()]
- .filter((name) => !imported.has(name)
- && !activatable.has(name)
- && !sourceResources.has(name)
- && !entryPoints.has(name))
- .sort();
- assert.deepEqual(unreachable, [],
- "packages nothing imports, nothing can activate, and no one runs — delete them or give them a consumer");
-});
diff --git a/tools/visual-ir-boundary.test.mjs b/tools/visual-ir-boundary.test.mjs
deleted file mode 100644
index 9e11ba78..00000000
--- a/tools/visual-ir-boundary.test.mjs
+++ /dev/null
@@ -1,103 +0,0 @@
-import assert from "node:assert/strict";
-import { readdir, readFile } from "node:fs/promises";
-import test from "node:test";
-
-import { compositionSchema } from "@narratage/composition";
-import { VISUAL_STYLE_NAMES_V1 } from "@narratage/visual-ir";
-
-function schemaFieldNames(schema, found = new Set()) {
- if (schema === null || typeof schema !== "object") return found;
- if (schema.kind === "object") {
- for (const [name, field] of Object.entries(schema.fields ?? {})) {
- found.add(name);
- schemaFieldNames(field.schema, found);
- }
- }
- if (schema.kind === "array") schemaFieldNames(schema.items, found);
- if (schema.kind === "oneOf") for (const variant of schema.variants ?? []) schemaFieldNames(variant, found);
- return found;
-}
-
-async function packageManifest(name) {
- return JSON.parse(await readFile(new URL(`../packages/${name}/package.json`, import.meta.url), "utf8"));
-}
-
-async function typescriptSources(directory) {
- const result = [];
- const visit = async (root) => {
- for (const entry of await readdir(root, { withFileTypes: true })) {
- const child = new URL(`./${entry.name}${entry.isDirectory() ? "/" : ""}`, root);
- if (entry.isDirectory()) await visit(child);
- else if (entry.name.endsWith(".ts")) result.push({ path: child, source: await readFile(child, "utf8") });
- }
- };
- await visit(new URL(`../packages/${directory}/src/`, import.meta.url));
- return result;
-}
-
-test("the public Composition waist has no author-family, Provider or cross-Track input field", () => {
- const fields = schemaFieldNames(compositionSchema);
- const forbidden = [
- "authorFamily",
- "caption",
- "component",
- "family",
- "lowerComposite",
- "mediaTrack",
- "provider",
- "ranking",
- "recipe",
- "selector",
- "siblingTrack",
- "sourceTrack",
- "trackRef",
- ];
- assert.deepEqual(forbidden.filter((field) => fields.has(field)), []);
- assert.ok(fields.has("visualIr") && fields.has("presents") && fields.has("elements"),
- "the audit must be inspecting the actual public VisualTrack schema");
- assert.equal(VISUAL_STYLE_NAMES_V1.includes("backdrop-filter"), false);
- assert.equal(VISUAL_STYLE_NAMES_V1.includes("mix-blend-mode"), false);
-});
-
-test("Composition and render adapters reach no author-package family", async () => {
- const authorPackages = new Set([
- "@narratage/audio-track",
- "@narratage/caption",
- "@narratage/caption-fine",
- "@narratage/deck-track",
- "@narratage/media-track",
- "@narratage/ranking",
- "@narratage/screen-overlay",
- "@narratage/typography-track",
- ]);
- for (const name of ["composition", "hyperframes", "render-hyperframes", "provider-hyperframes-local"]) {
- const manifest = await packageManifest(name);
- const dependencies = new Set([
- ...Object.keys(manifest.dependencies ?? {}),
- ...Object.keys(manifest.optionalDependencies ?? {}),
- ]);
- assert.deepEqual([...authorPackages].filter((dependency) => dependencies.has(dependency)), [],
- `${manifest.name} depends on an author-package family`);
- }
-});
-
-test("the terminal compiler contains no hidden accumulated-composite sampling path", async () => {
- const forbidden = [
- /backdrop-filter/u,
- /mix-blend-mode/u,
- /lowerComposite/u,
- /siblingTrack/u,
- /sourceTrack/u,
- /trackRef/u,
- /@narratage\/(?:caption(?:-fine)?|deck-track|media-track|ranking|screen-overlay|text-track)/u,
- ];
- const failures = [];
- for (const directory of ["hyperframes", "render-hyperframes"]) {
- for (const item of await typescriptSources(directory)) {
- for (const pattern of forbidden) {
- if (pattern.test(item.source)) failures.push(`${item.path.pathname}: ${pattern}`);
- }
- }
- }
- assert.deepEqual(failures, [], `terminal renderer boundary leaked author/cross-Track semantics:\n${failures.join("\n")}`);
-});