Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
feat: complete unified media track
kashorin56e3e8f6db0e
diff --git a/docs/guide/author-packages.md b/docs/guide/author-packages.mdindex 6676e322..d366a990 100644--- a/docs/guide/author-packages.md+++ b/docs/guide/author-packages.md@@ -81,7 +81,7 @@ export const decodeMyComponentSurface: TextSurfaceDecoder = (element, context) =Look at existing Surface implementations for reference:- `packages/seedance/src/surface.ts` — Prompt, Speech and Video Surfaces- `packages/caption/src/surface.ts` — the common Program Surface; concrete Style/Track Surfaces live in Style-family packages-- `packages/broll/src/surface.ts` — Track and Item Surfaces+- `packages/media-track/src/surface.ts` — Track, Item and Sequence Surfaces## 5. Write the activation descriptor@@ -159,6 +159,6 @@ authority.| `packages/seedance-speaker/` | Higher-level binding that composes Script, Prompt Kit and Seedance || `packages/caption/` | common Program, Cue/field contracts and whole-Atom timing || `packages/caption-fine/` | one field-free Style and Track Surface family |-| `packages/broll/` | Track with Item/transition behavior |+| `packages/media-track/` | Track with Item/Sequence, layer, motion and handoff behavior || `packages/text-track/` | Simple text overlay Track || `packages/film/` | Composition target that consumes peer Tracks |diff --git a/docs/guide/packages.md b/docs/guide/packages.mdindex d63df416..a0389be4 100644--- a/docs/guide/packages.md+++ b/docs/guide/packages.md@@ -86,7 +86,7 @@ Depends on Layer 1 and 2 but not on any Provider.@narratage/whisperx WhisperX component@narratage/caption caption planning and Track@narratage/caption-gemini Gemini caption planner-@narratage/broll B-roll Track+@narratage/media-track unified Media Item/Sequence Track@narratage/text-track text overlay Track@narratage/film Film composition@narratage/composition peer Track composition@@ -132,7 +132,7 @@ on the CLI.3. **CLI independence.** Neither `@narratage/cli` nor `@narratage/video-cli` transitively depends on anyProvider package. The video CLI also does not depend on any author-level video package- (`@narratage/script`, `@narratage/seedance-speaker`, `@narratage/broll`, `@narratage/text-track`, `@narratage/film`).+ (`@narratage/script`, `@narratage/seedance-speaker`, `@narratage/media-track`, `@narratage/text-track`, `@narratage/film`).Author packages are activated through the explicit package lock, not compile-time CLIdependencies.diff --git a/docs/quickstart.md b/docs/quickstart.mdindex 97e1e605..87fc4346 100644--- a/docs/quickstart.md+++ b/docs/quickstart.md@@ -40,7 +40,7 @@ Author Source says *what*. Run Source says *which*. Runtime Profile says *where*## Your first graph checkThe `talking-film-graph-check` example compiles a complete video graph — Script, Seedance, Speech,-WhisperX, Gemini Caption, B-roll, Text, Film, HyperFrames — without calling any external service.+WhisperX, Gemini Caption, Media Track, Text, Film, HyperFrames — without calling any external service.```bashpnpm narratage check examples/talking-film-graph-check/main.svml \@@ -62,7 +62,7 @@ before spending money.| [SVS Stylesheets](./quickstart/styles.md) | CSS-like Recipes for film, caption, B-roll, text, speech and fonts || [Media & Generation](./quickstart/generation.md) | media:Image, media:Audio, estimate:Speech, Seedance, speaker:Take || [Timing & Assembly](./quickstart/timing.md) | speech:Spine, whisperx:Alignment, ProgramSpace, SemanticMap |-| [Caption, B-roll & Text](./quickstart/tracks.md) | caption-fine:Style/Track, caption:Program, Planner, B-roll and Text |+| [Caption, Media & Text](./quickstart/tracks.md) | caption-fine:Style/Track, caption:Program, Planner, Media Track and Text || [Film & Rendering](./quickstart/composition.md) | film:Film, render:Video, full pipeline walkthrough || [Run Source & Builds](./quickstart/run.md) | .svrun syntax, targets, reuse, runtime profile, build workflow |diff --git a/docs/quickstart/composition.md b/docs/quickstart/composition.mdindex 5ed453fc..f0263a14 100644--- a/docs/quickstart/composition.md+++ b/docs/quickstart/composition.md@@ -17,7 +17,7 @@ Composition. The renderer then compiles that Composition into an MP4 video.## film:FilmAssembles all Tracks into a single Composition. Film itself has no domain knowledge — it does not-know what a caption is, what B-roll is, or what speech is. It takes any VisualTrack or AudioTrack+know what a caption is, what Media is, or what speech is. It takes any VisualTrack or AudioTrackand layers them by stacking order.```svml@@ -53,7 +53,7 @@ Common Track sources:| `{speech.visual}` | VisualTrack | `speech:Spine` — full-screen talking head || `{speech.audioTrack}` | AudioTrack | `speech:Spine` — synchronized audio || `{captions.track}` | VisualTrack | a Caption Style-family Track — timed captions |-| `{cards.visual}` | VisualTrack | `broll:Track` — B-roll overlays |+| `{cards.visual}` | VisualTrack | `media-track:Track` — media overlays or B-roll || `{titles.track}` | VisualTrack | `text:Track` — text overlays |### Track stacking@@ -67,7 +67,7 @@ Typical stacking order:| stack-order | Content ||---|---|| 10 | Speech visual (full-screen talking head) |-| 40 | B-roll overlays |+| 40 | Media overlays || 70 | Captions || 90 | Text overlays |@@ -119,7 +119,8 @@ The complete data flow from Script to rendered video. This example is based on<import as="caption" from="@narratage/caption@1"/><import as="caption-fine" from="@narratage/caption-fine@1"/><import as="caption-ai" from="@narratage/caption-gemini@1"/>- <import as="broll" from="@narratage/broll@1"/>+ <import as="pipeline" from="@narratage/media-pipeline@1"/>+ <import as="media-track" from="@narratage/media-track@1"/><import as="text" from="@narratage/text-track@1"/><import as="space" from="@narratage/spatial@1"/><import as="film" from="@narratage/film@1"/>@@ -141,13 +142,19 @@ The complete data flow from Script to rendered video. This example is based on<seedance:Video id="motion" model="mini"prompt={direction} duration="5"/>+ <space:Canvas id="vertical" width="1080" height="1920"/>+ <space:Frame id="title-frame" within={vertical}+ left="6%" top="6%" right="6%" bottom="84%"/>+ <space:Frame id="card-frame" within={vertical}+ left="10%" top="20%" right="10%" bottom="30%"/>+<!-- 3. Timing: assemble spine and align words -->- <speech:Spine id="speech">+ <speech:Spine id="speech" canvas={vertical}><speech:Take source={take} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>- <!-- 4. Tracks: captions, B-roll, text -->+ <!-- 4. Tracks: captions, Media, text --><caption-fine:Style id="base-caption" recipe={studio.caption.base}/><caption:Program id="caption-program" display={story.caption}default={base-caption}/>@@ -156,14 +163,13 @@ The complete data flow from Script to rendered video. This example is based on<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}space={speech.space} plan={cue-plan.plan} program={caption-program}/>- <broll:Track id="cards" map={timing.map} space={speech.space}>- <broll:Item source={motion.video} during={story.selection.demo}- appearance={studio.broll.card}/>- </broll:Track>-- <space:Canvas id="vertical" width="1080" height="1920"/>- <space:Frame id="title-frame" within={vertical}- left="6%" top="6%" right="6%" bottom="84%"/>+ <pipeline:Normalize id="motion-media" source={motion.video}+ video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>+ <media-track:Track id="cards" map={timing.map}+ space={speech.space} canvas={vertical}>+ <media-track:Item source={motion-media.media} during={story.selection.demo}+ frame={card-frame} appearance={studio.media.card} motion={studio.motion.card}/>+ </media-track:Track><text:Track id="titles" space={speech.space}><text:Item text="MEANING" during="full"frame={title-frame}@@ -195,10 +201,13 @@ The complete data flow from Script to rendered video. This example is based onfilm.vertical {background: #09090B;}- broll.card {- stack-order: 40; x: 0.1; y: 0.2; width: 0.8; height: 0.5;- fit: cover; background: #111116; radius: 20;- enter: slide-up 4f; exit: fade 4f;+ media.card {+ stack-order: 40; fit: cover; playback: hold-start;+ frame-paint: #111116; clip: rounded; radius: 20;+ }+ motion.card {+ enter: slide; enter-frames: 4; enter-direction: up; enter-easing: ease-out;+ exit: fade; exit-frames: 4; exit-easing: ease-in;}caption.base {cue-min-words: 1; cue-max-words: 5;diff --git a/docs/quickstart/styles.md b/docs/quickstart/styles.mdindex 4d7fb943..ce2e2998 100644--- a/docs/quickstart/styles.md+++ b/docs/quickstart/styles.md@@ -1,12 +1,12 @@---title: SVS Stylesheets-description: The SVS Recipe language — CSS-like stylesheets for film, caption, B-roll, text and generation settings.+description: The SVS Recipe language — CSS-like stylesheets for film, caption, media, text and generation settings.---# SVS StylesheetsSVS (`.svs`) files define reusable, typed configuration values using a CSS-like syntax. They-configure Film appearance, caption appearance, B-roll layout, text styling, speech estimation+configure Film appearance, caption appearance, Media presentation and motion, text styling, speech estimationparameters, generation settings, and typography choices. SVS values are called **Recipes** — they areimmutable typed Records that consuming components validate and interpret.@@ -153,41 +153,59 @@ Then assign them via `caption:Program`:</caption:Program>```-## B-roll+## Media Track-B-roll item appearance — position, fit, container, and enter/exit animations.+Media keeps spatial placement, frame presentation and lifecycle motion separate. A `SpatialFrame`+owns position and size; the appearance Recipe owns fitting and the frame material; an optional+motion Recipe owns enter, sustain and exit behavior.```svs-broll.product {+media.product {stack-order: 40;- x: 0.08;- y: 0.20;- width: 0.84;- height: 0.48;fit: contain;- background: #111116;+ playback: hold-start;+ frame-paint: #111116;+ clip: rounded;radius: 28;- enter: slide-up 8f;- exit: fade 6f;+ padding: 0;+ border-width: 1;+ border-style: solid;+ border-color: #FFFFFF20;+ shadows: 0 10 24 0 #00000066;+}++motion.product {+ enter: slide;+ enter-frames: 8;+ enter-direction: up;+ enter-easing: ease-out;+ exit: fade;+ exit-frames: 6;+ exit-easing: ease-in;}```| Property | Description ||---|---|| `stack-order` | Z-stacking order |-| `x`, `y` | Position as fraction of canvas |-| `width`, `height` | Size as fraction of canvas |-| `fit` | How the source fits the container: `cover`, `contain` |-| `background` | Container background color |-| `radius` | Container border radius |-| `enter` | Enter animation: `slide-up Nf`, `fade Nf` (N = frames) |-| `exit` | Exit animation: `fade Nf`, `slide-down Nf` |--Referenced by `broll:Item` via the `appearance` attribute:+| `fit` | `contain`, `cover`, `fit-width`, `fit-height`, `native`, `scale-down`, or `stretch` |+| `frame-x`, `frame-y` | Alignment point inside the placement Frame |+| `content-x`, `content-y` | Independently selected focal point inside the source |+| `playback` | Timed-source occupancy such as `once-start`, `hold-start`, `loop-end`, or `stretch` |+| `frame-paint` | Solid or gradient Paint behind the sampled source |+| `clip`, `radius`, `padding` | Frame clipping and inset |+| `border-*`, `shadows` | Frame-owned border and ordered shadows |+| `enter`, `exit` | Lifecycle operator; its frame count, easing and direction use separate properties |+| `sustain` | Zero or more deterministic local motions such as `float 12 2 up` |++Position remains an explicit graph edge:```svml-<broll:Item source={motion.video} during={story.selection.demo}- appearance={studio.broll.product}/>+<space:Frame id="product-frame" within={vertical}+ left="8%" top="20%" right="8%" bottom="32%"/>+<media-track:Item source={product-media.media}+ during={story.selection.demo} frame={product-frame}+ appearance={studio.media.product} motion={studio.motion.product}/>```## Textdiff --git a/docs/quickstart/timing.md b/docs/quickstart/timing.mdindex f91cc789..bab46646 100644--- a/docs/quickstart/timing.md+++ b/docs/quickstart/timing.md@@ -12,6 +12,7 @@ steps produce the **ProgramSpace** and **SemanticMap** that every downstream com```svml<import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/>+<import as="space" from="@narratage/spatial@1"/>```## speech:Spine@@ -20,7 +21,8 @@ Concatenates multiple takes into one ordered audio/visual coordinate space. Theprogram order — the final sequence of Segments in the finished video.```svml-<speech:Spine id="speech">+<space:Canvas id="vertical" width="1080" height="1920"/>+<speech:Spine id="speech" canvas={vertical}><speech:Take source={hook-take.video} segment={story.segment.hook}/><speech:Take source={meeting-take.video} segment={story.segment.meeting}/><speech:Take source={evidence-take.video} segment={story.segment.evidence}/>@@ -31,6 +33,7 @@ program order — the final sequence of Segments in the finished video.| Attribute | Required | Description ||---|---|---|| `id` | yes | Unique identifier |+| `canvas` | yes | Explicit CanvasSpace used by the restricted Media visual projection |### speech:Take@@ -74,7 +77,7 @@ This produces the **SemanticMap** — the bridge between Script text and physica| Output | Type | Used by ||---|---|---|-| `{timing.map}` | CompleteSemanticMap | Caption Style-family Tracks, `broll:Track`, `text:Track` — timed placement |+| `{timing.map}` | CompleteSemanticMap | Caption Style-family Tracks, `media-track:Track`, `text:Track` — timed placement |The SemanticMap maps every authored Script anchor to a time point. It covers all `2M + 2N` identities(where M = total speech tokens, N = number of Segments). This is how Selections and Moments declared@@ -104,22 +107,21 @@ Every component that operates in the time domain takes a `space` attribute point## SemanticMapThe SemanticMap is the typed bridge between Script text and physical time. When you write-`during={story.selection.demo}` on a B-roll item, the component uses the SemanticMap to look up the+`during={story.selection.demo}` on a Media Item, the component uses the SemanticMap to look up theexact frame range that Selection covers. Without a SemanticMap, Selections and Moments have nophysical meaning.Components that use the map take it via the `map` attribute:```svml-<broll:Track id="cards" map={timing.map} ...>+<media-track:Track id="cards" map={timing.map} ...><caption-fine:Track id="captions" ... map={timing.map} .../>```-Each point in the map can be:--- **measured** — directly observed by WhisperX alignment-- **derived** — computed from measured points via interpolation-- **estimated** — from the initial duration estimate (used before alignment runs)+The map contains final token windows and semantic anchor points only. It does not propagate+`measured`, `derived` or `estimated` labels. WhisperX evidence and the deterministic M:N aligner are+responsible for using the available recording evidence; downstream Tracks receive one complete map+and do not reinterpret how each point was obtained.## Combination example@@ -128,9 +130,11 @@ The complete timing stage, from generated takes to map and space:```svml<import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/>+<import as="space" from="@narratage/spatial@1"/><!-- Assemble takes in program order -->-<speech:Spine id="speech">+<space:Canvas id="vertical" width="1080" height="1920"/>+<speech:Spine id="speech" canvas={vertical}><speech:Take source={opening-take} segment={story.segment.opening}/><speech:Take source={answer-take} segment={story.segment.answer}/></speech:Spine>@@ -154,7 +158,7 @@ speaker:Take outputs ──► speech:Spine ──► whisperx:Alignment.audio │.audioTrack ▼.space ──────────► caption-fine:Track- │ broll:Track+ │ media-track:Track│ text:Track▼ film:Filmfilm:Film render:Videodiff --git a/docs/quickstart/tracks.md b/docs/quickstart/tracks.mdindex 514f6d4c..3404a4a8 100644--- a/docs/quickstart/tracks.md+++ b/docs/quickstart/tracks.md@@ -1,18 +1,17 @@----title: Caption, B-roll & Text-description: Visual track components — captions, B-roll overlays and text overlays.+title: Caption, Media & Text+description: Visual track components — captions, media overlays and text overlays.----# Caption, B-roll & Text+# Caption, Media & Text-> **Executable-slice note:** Caption Fine is the current complete package. The B-roll and Text-> sections document today's executable vertical slices. Their intended replacement author models-> are specified in `spec/media-track.md`, `spec/spatial-layout.md` and `spec/text-track.md`; examples-> using those future Surfaces will replace these sections only after the packages execute.+> **Pre-freeze note:** Caption Fine and Media Track execute their declared author Surfaces. Text is+> still the small graph witness described here; its complete author model remains governed by+> `spec/text-track.md`. None of these package contracts is a frozen public ABI yet.Every audiovisual contribution entering the final composition is a peer **Track**. Tracks are flat(no nesting), and their z-order is determined by the `stack-order` property in SVS. This page-covers the three main visual Track types: captions, B-roll, and text overlays.+covers three official visual Track packages: captions, media, and text overlays.## Caption system@@ -153,71 +152,19 @@ planner cannot rewrite text, select Styles, see audio or invent time. Its outputThe common Caption timing step joins the Plan to the independent SemanticMap. Fine then renders alldefault and override Styles into one ordinary peer `VisualTrack`: `{captions.track}`.-## B-roll+## Media overlays and B-roll-B-roll overlays generated or provided video/images at semantic time positions.+B-roll is an editorial use of the generic Media Track, not a separate Track family. One Item can+place a normalized image, video, animation or compositable Surface at a semantic or absolute window.```svml-<import as="broll" from="@narratage/broll@1"/>+<import as="pipeline" from="@narratage/media-pipeline@1"/>+<import as="media-track" from="@narratage/media-track@1"/>```-### broll:Track+### media-track:Track and media-track:Item-Container for B-roll items. Takes the SemanticMap for timing resolution.--```svml-<broll:Track id="product-broll" map={timing.map} space={speech.space}>- <broll:Item source={product-motion.video}- during={story.selection.product-demo}- appearance={studio.broll.product}/>-</broll:Track>-```--| Attribute | Required | Description |-|---|---|---|-| `id` | yes | Unique identifier |-| `map` | yes | SemanticMap from `whisperx:Alignment` |-| `space` | no | ProgramSpace (required by some configurations) |--### broll:Item--Each item places a source at a semantic time position with styled appearance:--```svml-<broll:Item source={product-motion.video}- during={story.selection.product-demo}- appearance={studio.broll.product}/>-```--| Attribute | Required | Description |-|---|---|---|-| `source` | yes | Video or image — from `seedance:Video`, `media:Image`, etc. |-| `during` | yes | Selection reference — when this item appears |-| `appearance` | yes | SVS broll Recipe — position, size, fit, animations |--The `during` attribute takes a Selection reference like `{story.selection.product-demo}`. The B-roll-item appears on screen for exactly the duration of that Selection, as resolved through the-SemanticMap.--The appearance Recipe controls enter/exit animations:--```svs-broll.product {- stack-order: 40;- x: 0.08; y: 0.20; width: 0.84; height: 0.48;- fit: contain;- background: #111116;- radius: 28;- enter: slide-up 8f;- exit: fade 6f;-}-```--**Output:** `{product-broll.visual}` — a VisualTrack added to `film:Film`.--### B-roll example--B-roll with a generated Seedance video appearing during a Script Selection:+Placement is an explicit Spatial Frame edge; appearance and motion remain reusable SVS values.```svml<seedance:Prompt id="product-direction">@@ -230,13 +177,27 @@ B-roll with a generated Seedance video appearing during a Script Selection:<seedance:Reference image={product-reference} role="subject"/></seedance:Video>-<broll:Track id="product-broll" map={timing.map}>- <broll:Item source={product-motion.video}+<pipeline:Normalize id="product-media" source={product-motion.video}+ video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>++<space:Frame id="product-frame" within={vertical}+ left="8%" top="20%" right="8%" bottom="32%"/>++<media-track:Track id="product-broll" map={timing.map}+ space={speech.space} canvas={vertical}>+ <media-track:Item source={product-media.media} frame={product-frame}during={story.selection.product-demo}- appearance={studio.broll.product}/>-</broll:Track>+ appearance={studio.media.product}+ motion={studio.motion.product}/>+</media-track:Track>```+The Selection contributes semantic points; Media performs the package-owned window projection.+The same Item model also covers full-canvas cutaways, split screens and corner overlays. Ordered+child layers, source occupancy and explicit Sequences are available when one source is not enough.++**Outputs:** `{product-broll.visual}` and, only when explicitly authored, `{product-broll.audio}`.+## Text overlaysStatic or timed text displayed on screen — titles, callouts, lower thirds.@@ -304,7 +265,8 @@ All three track types together in one source file:<import as="caption" from="@narratage/caption@1"/><import as="caption-fine" from="@narratage/caption-fine@1"/><import as="caption-ai" from="@narratage/caption-gemini@1"/>-<import as="broll" from="@narratage/broll@1"/>+<import as="pipeline" from="@narratage/media-pipeline@1"/>+<import as="media-track" from="@narratage/media-track@1"/><import as="text" from="@narratage/text-track@1"/><import as="space" from="@narratage/spatial@1"/>@@ -316,16 +278,20 @@ All three track types together in one source file:<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}space={speech.space} plan={cue-plan.plan} program={caption-program}/>-<!-- B-roll: generated video during a Selection -->-<broll:Track id="cards" map={timing.map} space={speech.space}>- <broll:Item source={motion.video} during={story.selection.demo}- appearance={studio.broll.card}/>-</broll:Track>-<!-- Shared placement is an explicit edge, separate from Text appearance. --><space:Canvas id="vertical" width="1080" height="1920"/><space:Frame id="title-frame" within={vertical}left="6%" top="6%" right="6%" bottom="84%"/>+<space:Frame id="card-frame" within={vertical}+ left="10%" top="20%" right="10%" bottom="30%"/>++<!-- Media: one ordinary Item used editorially as B-roll -->+<pipeline:Normalize id="motion-media" source={motion.video}+ video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>+<media-track:Track id="cards" map={timing.map} space={speech.space} canvas={vertical}>+ <media-track:Item source={motion-media.media} frame={card-frame}+ during={story.selection.demo} appearance={studio.media.card} motion={studio.motion.card}/>+</media-track:Track><!-- Text: persistent title overlay --><text:Track id="titles" space={speech.space}>@@ -343,5 +309,5 @@ All three track types together in one source file:</film:Film>```-The `stack-order` in each SVS Recipe determines z-ordering: speech visual at 10, B-roll at 40,+The `stack-order` in each SVS Recipe determines z-ordering: speech visual at 10, media at 40,captions at 70, text at 90. Higher values render on top.diff --git a/docs/zh/guide/author-packages.md b/docs/zh/guide/author-packages.mdindex 15dc3d35..80db6720 100644--- a/docs/zh/guide/author-packages.md+++ b/docs/zh/guide/author-packages.md@@ -77,7 +77,7 @@ export const decodeMyComponentSurface: TextSurfaceDecoder = (element, context) =可以参考已有的 Surface 实现:- `packages/seedance/src/surface.ts` —— Prompt、Speech 和 Video Surface- `packages/caption/src/surface.ts` —— 公共 Program Surface;具体 Style/Track Surface 属于各样式族包-- `packages/broll/src/surface.ts` —— Track 和 Item Surface+- `packages/media-track/src/surface.ts` —— Track、Item 和 Sequence Surface## 5. 编写 activation 描述符@@ -153,6 +153,6 @@ pnpm narratage lock-packages <lock-file> \| `packages/seedance-speaker/` | 组合 Script、Prompt Kit 和 Seedance 的更高层绑定 || `packages/caption/` | 公共 Program、Cue/字段合同和整 Atom 定时 || `packages/caption-fine/` | 一种无字段的 Style 与 Track Surface 样式族 |-| `packages/broll/` | 带 Item/转场行为的 Track |+| `packages/media-track/` | 带 Item/Sequence、图层、动效和交接行为的 Track || `packages/text-track/` | 简单的文字叠加 Track || `packages/film/` | 消费同级 Track 的合成 target |diff --git a/docs/zh/guide/packages.md b/docs/zh/guide/packages.mdindex c0c86223..f415594d 100644--- a/docs/zh/guide/packages.md+++ b/docs/zh/guide/packages.md@@ -81,7 +81,7 @@ description: 四个分层、依赖边界、包的结构与 facet。@narratage/whisperx WhisperX component@narratage/caption caption planning and Track@narratage/caption-gemini Gemini caption planner-@narratage/broll B-roll Track+@narratage/media-track 统一的 Media Item/Sequence Track@narratage/text-track text overlay Track@narratage/film Film composition@narratage/composition peer Track composition@@ -123,7 +123,7 @@ description: 四个分层、依赖边界、包的结构与 facet。2. **领域无关闭包。** 每个 Layer 1 包的传递闭包只包含 Layer 1 的包。`@narratage/core` 只依赖 `@narratage/protocol`。-3. **CLI 独立性。** `@narratage/cli` 和 `@narratage/video-cli` 都不会传递依赖任何 Provider 包。video CLI 同样不依赖任何作者层的视频包(`@narratage/script`、`@narratage/seedance-speaker`、`@narratage/broll`、`@narratage/text-track`、`@narratage/film`)。作者包通过显式的 package lock 被 activate,而不是通过编译期的 CLI 依赖。+3. **CLI 独立性。** `@narratage/cli` 和 `@narratage/video-cli` 都不会传递依赖任何 Provider 包。video CLI 同样不依赖任何作者层的视频包(`@narratage/script`、`@narratage/seedance-speaker`、`@narratage/media-track`、`@narratage/text-track`、`@narratage/film`)。作者包通过显式的 package lock 被 activate,而不是通过编译期的 CLI 依赖。## 包的结构diff --git a/docs/zh/quickstart.md b/docs/zh/quickstart.mdindex 75c441bd..eb96ba64 100644--- a/docs/zh/quickstart.md+++ b/docs/zh/quickstart.md@@ -59,7 +59,7 @@ pnpm narratage plan examples/talking-film-graph-check/build.svrun \| [SVS 样式表](./quickstart/styles.md) | CSS 风格的 Recipe:film、caption、B-roll、text、speech、字体 || [媒体与生成](./quickstart/generation.md) | media:Image、media:Audio、estimate:Speech、Seedance、speaker:Take || [时序与装配](./quickstart/timing.md) | speech:Spine、whisperx:Alignment、ProgramSpace、SemanticMap |-| [字幕、B-roll 与文字](./quickstart/tracks.md) | caption-fine:Style/Track、caption:Program、Planner、B-roll 与 Text |+| [字幕、Media 与文字](./quickstart/tracks.md) | caption-fine:Style/Track、caption:Program、Planner、Media Track 与 Text || [Film 与渲染](./quickstart/composition.md) | film:Film、render:Video、完整流水线演练 || [Run Source 与 Build](./quickstart/run.md) | .svrun 语法、targets、复用、runtime profile、build 工作流 |diff --git a/docs/zh/quickstart/composition.md b/docs/zh/quickstart/composition.mdindex 8ed6847c..8ff53897 100644--- a/docs/zh/quickstart/composition.md+++ b/docs/zh/quickstart/composition.md@@ -17,7 +17,7 @@ Composition。然后渲染器将该 Composition 编译为 MP4 视频。## film:Film将所有 Track 组装为单一的 Composition。Film 本身没有领域知识——它不知道什么是字幕、-什么是 B-roll、什么是语音。它接收任何 VisualTrack 或 AudioTrack,并按堆叠顺序将它们分层。+什么是 Media、什么是语音。它接收任何 VisualTrack 或 AudioTrack,并按堆叠顺序将它们分层。```svml<space:Canvas id="vertical" width="1080" height="1920"/>@@ -52,7 +52,7 @@ Composition。然后渲染器将该 Composition 编译为 MP4 视频。| `{speech.visual}` | VisualTrack | `speech:Spine`——全屏说话人画面 || `{speech.audioTrack}` | AudioTrack | `speech:Spine`——同步音频 || `{captions.track}` | VisualTrack | Caption 样式族 Track——定时字幕 |-| `{cards.visual}` | VisualTrack | `broll:Track`——B-roll 叠加层 |+| `{cards.visual}` | VisualTrack | `media-track:Track`——Media 叠加层或 B-roll || `{titles.track}` | VisualTrack | `text:Track`——文字叠加层 |### Track 堆叠@@ -65,7 +65,7 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T| stack-order | 内容 ||---|---|| 10 | 语音画面(全屏说话人画面) |-| 40 | B-roll 叠加层 |+| 40 | Media 叠加层 || 70 | 字幕 || 90 | 文字叠加层 |@@ -116,7 +116,8 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T<import as="caption" from="@narratage/caption@1"/><import as="caption-fine" from="@narratage/caption-fine@1"/><import as="caption-ai" from="@narratage/caption-gemini@1"/>- <import as="broll" from="@narratage/broll@1"/>+ <import as="pipeline" from="@narratage/media-pipeline@1"/>+ <import as="media-track" from="@narratage/media-track@1"/><import as="text" from="@narratage/text-track@1"/><import as="space" from="@narratage/spatial@1"/><import as="film" from="@narratage/film@1"/>@@ -138,13 +139,19 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T<seedance:Video id="motion" model="mini"prompt={direction} duration="5"/>+ <space:Canvas id="vertical" width="1080" height="1920"/>+ <space:Frame id="title-frame" within={vertical}+ left="6%" top="6%" right="6%" bottom="84%"/>+ <space:Frame id="card-frame" within={vertical}+ left="10%" top="20%" right="10%" bottom="30%"/>+<!-- 3. Timing: assemble spine and align words -->- <speech:Spine id="speech">+ <speech:Spine id="speech" canvas={vertical}><speech:Take source={take} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>- <!-- 4. Tracks: captions, B-roll, text -->+ <!-- 4. Tracks: captions, Media, text --><caption-fine:Style id="base-caption" recipe={studio.caption.base}/><caption:Program id="caption-program" display={story.caption}default={base-caption}/>@@ -153,14 +160,13 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 T<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}space={speech.space} plan={cue-plan.plan} program={caption-program}/>- <broll:Track id="cards" map={timing.map} space={speech.space}>- <broll:Item source={motion.video} during={story.selection.demo}- appearance={studio.broll.card}/>- </broll:Track>-- <space:Canvas id="vertical" width="1080" height="1920"/>- <space:Frame id="title-frame" within={vertical}- left="6%" top="6%" right="6%" bottom="84%"/>+ <pipeline:Normalize id="motion-media" source={motion.video}+ video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>+ <media-track:Track id="cards" map={timing.map}+ space={speech.space} canvas={vertical}>+ <media-track:Item source={motion-media.media} during={story.selection.demo}+ frame={card-frame} appearance={studio.media.card} motion={studio.motion.card}/>+ </media-track:Track><text:Track id="titles" space={speech.space}><text:Item text="MEANING" during="full"frame={title-frame}@@ -192,10 +198,13 @@ Track 是**扁平的**——没有嵌套或分组。Z 轴排序完全由每个 Tfilm.vertical {background: #09090B;}- broll.card {- stack-order: 40; x: 0.1; y: 0.2; width: 0.8; height: 0.5;- fit: cover; background: #111116; radius: 20;- enter: slide-up 4f; exit: fade 4f;+ media.card {+ stack-order: 40; fit: cover; playback: hold-start;+ frame-paint: #111116; clip: rounded; radius: 20;+ }+ motion.card {+ enter: slide; enter-frames: 4; enter-direction: up; enter-easing: ease-out;+ exit: fade; exit-frames: 4; exit-easing: ease-in;}caption.base {cue-min-words: 1; cue-max-words: 5;diff --git a/docs/zh/quickstart/styles.md b/docs/zh/quickstart/styles.mdindex 584d9966..4ee60cc1 100644--- a/docs/zh/quickstart/styles.md+++ b/docs/zh/quickstart/styles.md@@ -1,11 +1,11 @@---title: SVS 样式表-description: SVS Recipe 语言——用于影片、字幕、B-roll、文本及生成设置的类 CSS 样式表。+description: SVS Recipe 语言——用于影片、字幕、媒体、文本及生成设置的类 CSS 样式表。---# SVS 样式表-SVS(`.svs`)文件使用类 CSS 语法定义可复用的类型化配置值。它们用于配置影片尺寸、字幕外观、B-roll 布局、文本样式、语音估算参数、生成设置和字体选择。SVS 中的值称为 **Recipe**——它们是不可变的类型化记录,由消费组件进行验证和解释。+SVS(`.svs`)文件使用类 CSS 语法定义可复用的类型化配置值。它们用于配置影片外观、字幕外观、Media 呈现与运动、文本样式、语音估算参数、生成设置和字体选择。SVS 中的值称为 **Recipe**——它们是不可变的类型化记录,由消费组件进行验证和解释。## 基本语法@@ -149,41 +149,58 @@ caption.bob {</caption:Program>```-## B-roll+## Media Track-B-roll 项目外观——位置、适配方式、容器以及进入/退出动画。+Media 将空间位置、框呈现与生命周期运动分开。`SpatialFrame` 负责位置和尺寸;外观 Recipe+负责素材适配与框材质;可选的 motion Recipe 负责入场、持续和退场。```svs-broll.product {+media.product {stack-order: 40;- x: 0.08;- y: 0.20;- width: 0.84;- height: 0.48;fit: contain;- background: #111116;+ playback: hold-start;+ frame-paint: #111116;+ clip: rounded;radius: 28;- enter: slide-up 8f;- exit: fade 6f;+ padding: 0;+ border-width: 1;+ border-style: solid;+ border-color: #FFFFFF20;+ shadows: 0 10 24 0 #00000066;+}++motion.product {+ enter: slide;+ enter-frames: 8;+ enter-direction: up;+ enter-easing: ease-out;+ exit: fade;+ exit-frames: 6;+ exit-easing: ease-in;}```| 属性 | 描述 ||---|---|| `stack-order` | Z 轴层叠顺序 |-| `x`、`y` | 位置,以画布比例表示 |-| `width`、`height` | 尺寸,以画布比例表示 |-| `fit` | 源内容适配容器的方式:`cover`、`contain` |-| `background` | 容器背景颜色 |-| `radius` | 容器圆角半径 |-| `enter` | 进入动画:`slide-up Nf`、`fade Nf`(N = 帧数) |-| `exit` | 退出动画:`fade Nf`、`slide-down Nf` |--通过 `broll:Item` 的 `appearance` 属性引用:+| `fit` | `contain`、`cover`、`fit-width`、`fit-height`、`native`、`scale-down` 或 `stretch` |+| `frame-x`、`frame-y` | 放置 Frame 内的对齐点 |+| `content-x`、`content-y` | 素材内部独立选择的焦点 |+| `playback` | `once-start`、`hold-start`、`loop-end`、`stretch` 等有时长素材占用方式 |+| `frame-paint` | 采样素材背后的纯色或渐变 Paint |+| `clip`、`radius`、`padding` | 框裁切与内缩 |+| `border-*`、`shadows` | 框自有的边框与有序阴影 |+| `enter`、`exit` | 生命周期算子;帧数、缓动和方向使用独立属性 |+| `sustain` | 零个或多个确定性局部运动,例如 `float 12 2 up` |++位置始终是一条显式图边:```svml-<broll:Item source={motion.video} during={story.selection.demo}- appearance={studio.broll.product}/>+<space:Frame id="product-frame" within={vertical}+ left="8%" top="20%" right="8%" bottom="32%"/>+<media-track:Item source={product-media.media}+ during={story.selection.demo} frame={product-frame}+ appearance={studio.media.product} motion={studio.motion.product}/>```## 文本diff --git a/docs/zh/quickstart/timing.md b/docs/zh/quickstart/timing.mdindex fbef90db..b9834080 100644--- a/docs/zh/quickstart/timing.md+++ b/docs/zh/quickstart/timing.md@@ -10,6 +10,7 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连```svml<import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/>+<import as="space" from="@narratage/spatial@1"/>```## speech:Spine@@ -17,7 +18,8 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连将多个片段拼接成一个有序的音视频坐标空间。Spine 定义了节目顺序——即最终视频中 Segment 的排列序列。```svml-<speech:Spine id="speech">+<space:Canvas id="vertical" width="1080" height="1920"/>+<speech:Spine id="speech" canvas={vertical}><speech:Take source={hook-take.video} segment={story.segment.hook}/><speech:Take source={meeting-take.video} segment={story.segment.meeting}/><speech:Take source={evidence-take.video} segment={story.segment.evidence}/>@@ -28,6 +30,7 @@ description: Speech Spine 装配与 WhisperX 对齐——将生成的片段连| 属性 | 必填 | 描述 ||---|---|---|| `id` | 是 | 唯一标识符 |+| `canvas` | 是 | 受限 Media 视觉投影显式使用的 CanvasSpace |### speech:Take@@ -69,7 +72,7 @@ Spine 产生四个输出,供下游组件使用:| 输出 | 类型 | 使用方 ||---|---|---|-| `{timing.map}` | CompleteSemanticMap | Caption 样式族 Track、`broll:Track`、`text:Track`——定时放置 |+| `{timing.map}` | CompleteSemanticMap | Caption 样式族 Track、`media-track:Track`、`text:Track`——定时放置 |SemanticMap 将每个 Script 中标注的锚点映射到一个时间点。它覆盖所有 `2M + 2N` 个标识(其中 M = 语音词元总数,N = Segment 数量)。这就是 Script 中声明的 Selection 和 Moment 如何转化为下游视觉组件所需的实际时间范围和时间点。@@ -94,20 +97,18 @@ ProgramSpace 包含:## SemanticMap-SemanticMap 是连接 Script 文本与物理时间的类型化桥梁。当你在 B-roll 项上写 `during={story.selection.demo}` 时,组件会使用 SemanticMap 查找该 Selection 覆盖的精确帧范围。没有 SemanticMap,Selection 和 Moment 就没有物理意义。+SemanticMap 是连接 Script 文本与物理时间的类型化桥梁。当你在 Media Item 上写 `during={story.selection.demo}` 时,组件会使用 SemanticMap 查找该 Selection 覆盖的精确帧范围。没有 SemanticMap,Selection 和 Moment 就没有物理意义。使用该映射的组件通过 `map` 属性接收它:```svml-<broll:Track id="cards" map={timing.map} ...>+<media-track:Track id="cards" map={timing.map} ...><caption-fine:Track id="captions" ... map={timing.map} .../>```-映射中的每个点可以是:--- **测量值**——由 WhisperX 对齐直接观测得到-- **推导值**——通过插值从测量值计算得出-- **估算值**——来自初始时长估计(在对齐运行之前使用)+Map 只包含最终词窗口和语义锚点,不传播“测量、推导、估算”标签。WhisperX Evidence+与确定性的 M:N 对齐器负责充分使用录音证据;下游 Track 只接收一份完整 Map,不再解释+每个点是如何获得的。## 组合示例@@ -116,9 +117,11 @@ SemanticMap 是连接 Script 文本与物理时间的类型化桥梁。当你在```svml<import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/>+<import as="space" from="@narratage/spatial@1"/><!-- Assemble takes in program order -->-<speech:Spine id="speech">+<space:Canvas id="vertical" width="1080" height="1920"/>+<speech:Spine id="speech" canvas={vertical}><speech:Take source={opening-take} segment={story.segment.opening}/><speech:Take source={answer-take} segment={story.segment.answer}/></speech:Spine>@@ -142,7 +145,7 @@ speaker:Take outputs ──► speech:Spine ──► whisperx:Alignment.audio │.audioTrack ▼.space ──────────► caption-fine:Track- │ broll:Track+ │ media-track:Track│ text:Track▼ film:Filmfilm:Film render:Videodiff --git a/docs/zh/quickstart/tracks.md b/docs/zh/quickstart/tracks.mdindex 214983b6..74fe7743 100644--- a/docs/zh/quickstart/tracks.md+++ b/docs/zh/quickstart/tracks.md@@ -1,15 +1,15 @@----title: 字幕、B-roll 与文字-description: 视觉 Track 组件——字幕、B-roll 叠加层和文字叠加层。+title: 字幕、Media 与文字+description: 视觉 Track 组件——字幕、媒体叠加层和文字叠加层。----# 字幕、B-roll 与文字+# 字幕、Media 与文字-> **当前可执行切片:** Caption Fine 已完整实现。B-roll 与文字章节描述的是今天仍可运行的-> 纵向切片;它们的目标模型分别记录在 `spec/media-track.md`、`spec/spatial-layout.md` 与-> `spec/text-track.md`。只有对应新包真正可运行后,教程才会改用新语法。+> **冻结前说明:** Caption Fine 与 Media Track 已执行各自声明的作者 Surface。Text+> 仍是本页展示的小型图链路证明;完整作者模型由 `spec/text-track.md` 约束。当前没有任何+> 一个包被宣布为冻结的公开 ABI。-每个进入最终合成的视听内容都是一个对等的 **Track**。Track 是扁平的(无嵌套),其 z 轴顺序由 SVS 中的 `stack-order` 属性决定。本页介绍三种主要的视觉 Track 类型:字幕、B-roll 和文字叠加层。+每个进入最终合成的视听内容都是一个对等的 **Track**。Track 是扁平的(无嵌套),其 z 轴顺序由 SVS 中的 `stack-order` 属性决定。本页介绍三个官方视觉 Track 包:字幕、Media 和文字叠加层。## 字幕系统@@ -121,69 +121,19 @@ Dual Text 右侧。公共 Caption 先把 Plan 与独立 SemanticMap 拼接,Fine 再把所有默认/覆盖样式渲染成一个普通的对等 `VisualTrack`:`{captions.track}`。-## B-roll+## Media 叠加层与 B-roll-B-roll 在语义时间位置叠加生成的或预先提供的视频/图像。+B-roll 是通用 Media Track 的一种剪辑用途,不是独立 Track 家族。一个 Item 可以在语义+或绝对窗口内放置规范化图片、视频、动画或 Compositable Surface。```svml-<import as="broll" from="@narratage/broll@1"/>+<import as="pipeline" from="@narratage/media-pipeline@1"/>+<import as="media-track" from="@narratage/media-track@1"/>```-### broll:Track+### media-track:Track 与 media-track:Item-B-roll 项目的容器。接收 SemanticMap 用于时间解析。--```svml-<broll:Track id="product-broll" map={timing.map} space={speech.space}>- <broll:Item source={product-motion.video}- during={story.selection.product-demo}- appearance={studio.broll.product}/>-</broll:Track>-```--| 属性 | 必填 | 描述 |-|---|---|---|-| `id` | 是 | 唯一标识符 |-| `map` | 是 | 来自 `whisperx:Alignment` 的 SemanticMap |-| `space` | 否 | ProgramSpace(某些配置需要) |--### broll:Item--每个项目将一个来源放置在语义时间位置上,并附带样式外观:--```svml-<broll:Item source={product-motion.video}- during={story.selection.product-demo}- appearance={studio.broll.product}/>-```--| 属性 | 必填 | 描述 |-|---|---|---|-| `source` | 是 | 视频或图像——来自 `seedance:Video`、`media:Image` 等 |-| `during` | 是 | Selection 引用——该项目何时出现 |-| `appearance` | 是 | SVS B-roll Recipe——位置、大小、适配方式、动画 |--`during` 属性接收一个 Selection 引用,如 `{story.selection.product-demo}`。B-roll 项目在屏幕上显示的时长恰好等于该 Selection 的持续时间,通过 SemanticMap 解析确定。--外观 Recipe 控制进场/退场动画:--```svs-broll.product {- stack-order: 40;- x: 0.08; y: 0.20; width: 0.84; height: 0.48;- fit: contain;- background: #111116;- radius: 28;- enter: slide-up 8f;- exit: fade 6f;-}-```--**输出:**`{product-broll.visual}` —— 添加到 `film:Film` 的 VisualTrack。--### B-roll 示例--B-roll 搭配生成的 Seedance 视频,在 Script Selection 期间出现:+位置是一条显式 Spatial Frame 边,外观和运动则是可复用的 SVS 值:```svml<seedance:Prompt id="product-direction">@@ -196,13 +146,27 @@ B-roll 搭配生成的 Seedance 视频,在 Script Selection 期间出现:<seedance:Reference image={product-reference} role="subject"/></seedance:Video>-<broll:Track id="product-broll" map={timing.map}>- <broll:Item source={product-motion.video}+<pipeline:Normalize id="product-media" source={product-motion.video}+ video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>++<space:Frame id="product-frame" within={vertical}+ left="8%" top="20%" right="8%" bottom="32%"/>++<media-track:Track id="product-broll" map={timing.map}+ space={speech.space} canvas={vertical}>+ <media-track:Item source={product-media.media} frame={product-frame}during={story.selection.product-demo}- appearance={studio.broll.product}/>-</broll:Track>+ appearance={studio.media.product}+ motion={studio.motion.product}/>+</media-track:Track>```+Selection 只贡献语义点;Media 包负责将这些点投影为窗口。同一个 Item 模型也能表达全屏+切换、分屏和角落小窗。需要多个素材时,可以使用有序局部 Layer 或显式 Sequence。++**输出:**`{product-broll.visual}`;只有作者显式选择了源音频或 SFX 时才会出现+`{product-broll.audio}`。+## 文字叠加层在屏幕上显示的静态或定时文字——标题、标注、下方三分之一字幕条。@@ -269,7 +233,8 @@ B-roll 搭配生成的 Seedance 视频,在 Script Selection 期间出现:<import as="caption" from="@narratage/caption@1"/><import as="caption-fine" from="@narratage/caption-fine@1"/><import as="caption-ai" from="@narratage/caption-gemini@1"/>-<import as="broll" from="@narratage/broll@1"/>+<import as="pipeline" from="@narratage/media-pipeline@1"/>+<import as="media-track" from="@narratage/media-track@1"/><import as="text" from="@narratage/text-track@1"/><import as="space" from="@narratage/spatial@1"/>@@ -281,16 +246,20 @@ B-roll 搭配生成的 Seedance 视频,在 Script Selection 期间出现:<caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence} map={timing.map}space={speech.space} plan={cue-plan.plan} program={caption-program}/>-<!-- B-roll: generated video during a Selection -->-<broll:Track id="cards" map={timing.map} space={speech.space}>- <broll:Item source={motion.video} during={story.selection.demo}- appearance={studio.broll.card}/>-</broll:Track>--<!-- 共享位置是显式边,与 Text 外观分开。 -->+<!-- 共享位置是显式边,与 Media/Text 外观分开。 --><space:Canvas id="vertical" width="1080" height="1920"/><space:Frame id="title-frame" within={vertical}left="6%" top="6%" right="6%" bottom="84%"/>+<space:Frame id="card-frame" within={vertical}+ left="10%" top="20%" right="10%" bottom="30%"/>++<!-- Media:Selection 期间显示一个普通 Item -->+<pipeline:Normalize id="motion-media" source={motion.video}+ video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>+<media-track:Track id="cards" map={timing.map} space={speech.space} canvas={vertical}>+ <media-track:Item source={motion-media.media} frame={card-frame}+ during={story.selection.demo} appearance={studio.media.card} motion={studio.motion.card}/>+</media-track:Track><!-- Text: persistent title overlay --><text:Track id="titles" space={speech.space}>@@ -308,4 +277,4 @@ B-roll 搭配生成的 Seedance 视频,在 Script Selection 期间出现:</film:Film>```-每个 SVS Recipe 中的 `stack-order` 决定 z 轴排序:语音视觉层为 10,B-roll 为 40,字幕为 70,文字为 90。数值越高,渲染层越靠上。+每个 SVS Recipe 中的 `stack-order` 决定 z 轴排序:语音视觉层为 10,Media 为 40,字幕为 70,文字为 90。数值越高,渲染层越靠上。diff --git a/examples/README.md b/examples/README.mdindex 47922e37..3c4d18af 100644--- a/examples/README.md+++ b/examples/README.md@@ -1,7 +1,7 @@# Examples- [`talking-film-graph-check`](./talking-film-graph-check/README.md) compiles the complete- provider-free author graph through Speech, WhisperX, Caption, B-roll, Text, Film and final render+ provider-free author graph through Speech, WhisperX, Caption, Media Track, Text, Film and final renderrequirements.- [`talking-film-live`](./talking-film-live/README.md) executes the real paid two-take path with KIE,local media processing, local WhisperX, Vertex Caption planning and local HyperFrames rendering.diff --git a/examples/talking-film-golden/README.md b/examples/talking-film-golden/README.mdindex d84b260a..4bdcd2a6 100644--- a/examples/talking-film-golden/README.md+++ b/examples/talking-film-golden/README.md@@ -14,7 +14,7 @@ This directory answers one question:The source explicitly chooses:- two Seedance **Mini** speech generations, each with its own Script Segment, Prompt and reference;-- one Seedance Mini B-roll generation;+- one Seedance Mini media generation used editorially as B-roll;- one WhisperX measurement path;- the Gemini-backed official Caption package;- Hyperframes as the final rendering method.@@ -50,9 +50,9 @@ Script dialogue + Prompt + reference ──> two Seedance Speech Needs▼Script left display + Caption Program ──> Gemini Plan ──> Caption Track-Script selections + generated B-roll ───────────────────> B-roll Track+Script selections + normalized generated media ─────────> Media Track-Seedance B-roll + Text Track + every Track above+Media Track + Text Track + every Track above-> Film Composition-> explicit Hyperframes compile + render component-> final video Need@@ -73,7 +73,7 @@ Candidates and Operations.| `speech:Spine` | `@narratage/speech` | ordered clips -> normalized Takes, one `SpeechBasis`, then ordinary projections | Surface, fold, media normalization and projection components implemented || `whisperx:Alignment` | `@narratage/whisperx` | 48k speech master -> explicit 16k evidence Need -> WhisperX -> provider-neutral `@narratage/speech-alignment` -> Map | Surface, Graph Fragment, deterministic components and local Provider/service implemented || `seedance:Video model="mini"` | `@narratage/seedance` | explicit Seedance Mini video Need plus primary-video projection | implemented |-| `broll:Track` | `@narratage/broll` | semantic windows + normalized media + recipe -> peer Visual/Audio Tracks | Surface and deterministic lowering implemented |+| `media-track:Track` | `@narratage/media-track` | semantic windows + normalized media + explicit Frame/Recipe -> peer Visual/optional Audio Tracks | Item/Sequence Surface and deterministic lowering implemented || `caption-fine:Style` / `caption:Program` | Fine + common Caption | explicit default over all words + ordered whole-Style replacement by Role or word subset | implemented || `caption-ai:Planner` | `@narratage/caption-gemini` | immutable display Atoms/Words + per-run requirements -> whole-Atom Cue cuts and optional per-Word fields | implemented; Google Vertex Endpoint implemented separately || `caption-fine:Track` | `@narratage/caption-fine` | CaptionPlan + independent SemanticMap + complete Fine Styles -> VisualTrack | implemented |diff --git a/examples/talking-film-golden/main.svml b/examples/talking-film-golden/main.svmlindex 62d4d41e..b145127b 100644--- a/examples/talking-film-golden/main.svml+++ b/examples/talking-film-golden/main.svml@@ -15,7 +15,8 @@<import as="caption" from="@narratage/caption@1"/><import as="caption-fine" from="@narratage/caption-fine@1"/><import as="caption-ai" from="@narratage/caption-gemini@1"/>- <import as="broll" from="@narratage/broll@1"/>+ <import as="pipeline" from="@narratage/media-pipeline@1"/>+ <import as="media-track" from="@narratage/media-track@1"/><import as="text" from="@narratage/text-track@1"/><import as="space" from="@narratage/spatial@1"/><import as="film" from="@narratage/film@1"/>@@ -43,6 +44,12 @@<media:Image id="bob-reference" src="./assets/bob.png"/><media:Image id="product-reference" src="./assets/product.png"/>+ <space:Canvas id="vertical" width="1080" height="1920"/>+ <space:Frame id="title-frame" within={vertical}+ left="6%" top="6%" right="6%" bottom="84%"/>+ <space:Frame id="product-frame" within={vertical}+ left="8%" top="20%" right="8%" bottom="32%"/>+<seedance:Prompt id="alice-direction">Locked medium close-up. Alice speaks directly to camera in a quiet daylight studio.Calm, curious delivery; natural breathing and restrained hand movement.@@ -101,7 +108,7 @@</seedance:Speech><!-- Ordered generated Takes become one audio basis plus its peer visual/audio Tracks. -->- <speech:Spine id="speech">+ <speech:Spine id="speech" canvas={vertical}><speech:Take source={alice-take} segment={script.segment.opening}/><speech:Take source={bob-take} segment={script.segment.answer}/></speech:Spine>@@ -118,13 +125,19 @@<seedance:Reference image={product-reference} role="subject"/></seedance:Video>- <broll:Track id="product-broll" map={timing.map}>- <broll:Item- source={product-motion.video}+ <pipeline:Normalize id="product-media" source={product-motion.video}+ video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>++ <media-track:Track id="product-broll" map={timing.map}+ space={speech.space} canvas={vertical}>+ <media-track:Item+ source={product-media.media}during={script.selection.product-demo}- appearance={studio.broll.product}+ frame={product-frame}+ appearance={studio.media.product}+ motion={studio.motion.product}/>- </broll:Track>+ </media-track:Track><!-- This imported package explicitly chooses Gemini cue planning; it cannot rewrite Script. --><caption-fine:Track@@ -137,10 +150,6 @@plan={caption-plan.plan}/>- <space:Canvas id="vertical" width="1080" height="1920"/>- <space:Frame id="title-frame" within={vertical}- left="6%" top="6%" right="6%" bottom="84%"/>-<text:Track id="titles" space={speech.space}><text:Itemtext="EDIT MEANING, NOT TIMELINES"diff --git a/examples/talking-film-golden/studio.svs b/examples/talking-film-golden/studio.svsindex 2f645be1..87beae8b 100644--- a/examples/talking-film-golden/studio.svs+++ b/examples/talking-film-golden/studio.svs@@ -28,17 +28,23 @@fit: cover;}- broll.product {+ media.product {stack-order: 40;- x: 0.08;- y: 0.20;- width: 0.84;- height: 0.48;fit: contain;- background: #111116;+ playback: hold-start;+ frame-paint: #111116;+ clip: rounded;radius: 28;- enter: slide-up 8f;- exit: fade 6f;+ }++ motion.product {+ enter: slide;+ enter-frames: 8;+ enter-direction: up;+ enter-easing: ease-out;+ exit: fade;+ exit-frames: 6;+ exit-easing: ease-in;}caption.short-cues {diff --git a/examples/talking-film-graph-check/README.md b/examples/talking-film-graph-check/README.mdindex 162e96e3..ff179e35 100644--- a/examples/talking-film-graph-check/README.md+++ b/examples/talking-film-graph-check/README.md@@ -1,7 +1,7 @@# Talking-film graph checkThis fixture compiles the complete author graph: Script, generated-media requests, Speech Spine,-WhisperX alignment, display-only Gemini Caption planning, B-roll, Caption, Text, Film and+WhisperX alignment, display-only Gemini Caption planning, Media Track, Caption, Text, Film andexplicit HyperFrames rendering. `check` and `plan` do not invoke Seedance, WhisperX, Gemini orHyperFrames.diff --git a/examples/talking-film-graph-check/main.svml b/examples/talking-film-graph-check/main.svmlindex 3b3f09d1..07825b57 100644--- a/examples/talking-film-graph-check/main.svml+++ b/examples/talking-film-graph-check/main.svml@@ -5,7 +5,8 @@<import as="seedance" from="@narratage/seedance@1"/><import as="speech" from="@narratage/speech-spine@1"/><import as="whisperx" from="@narratage/whisperx@1"/>- <import as="broll" from="@narratage/broll@1"/>+ <import as="pipeline" from="@narratage/media-pipeline@1"/>+ <import as="media-track" from="@narratage/media-track@1"/><import as="caption" from="@narratage/caption@1"/><import as="caption-fine" from="@narratage/caption-fine@1"/><import as="caption-ai" from="@narratage/caption-gemini@1"/>@@ -19,11 +20,17 @@<opening><HOST>Meaning @demo becomes the source @/demo.</opening></script>+ <space:Canvas id="vertical" width="1080" height="1920"/>+ <space:Frame id="title-frame" within={vertical}+ left="6%" top="6%" right="6%" bottom="84%"/>+ <space:Frame id="card-frame" within={vertical}+ left="10%" top="20%" right="10%" bottom="30%"/>+<seedance:Prompt id="direction">Locked medium close-up in a quiet daylight studio.</seedance:Prompt><seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue} prompt={direction} duration="5"/><seedance:Video id="motion" model="mini" prompt={direction} duration="5"/>- <speech:Spine id="speech">+ <speech:Spine id="speech" canvas={vertical}><speech:Take source={take} segment={story.segment.opening}/></speech:Spine><whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>@@ -32,15 +39,15 @@<caption:Program id="caption-program" display={story.caption} default={base-caption}/><caption-ai:Planner id="cue-plan" display={story.caption} program={caption-program} model="gemini-2.5-flash"/>- <broll:Track id="cards" map={timing.map} space={speech.space}>- <broll:Item source={motion.video} during={story.selection.demo} appearance={studio.broll.card}/>- </broll:Track>+ <pipeline:Normalize id="motion-media" source={motion.video}+ video="primary-moving" audio="none" span-authority="video" frame-rate="30"/>+ <media-track:Track id="cards" map={timing.map} space={speech.space} canvas={vertical}>+ <media-track:Item source={motion-media.media} during={story.selection.demo}+ frame={card-frame} appearance={studio.media.card} motion={studio.motion.card}/>+ </media-track:Track><caption-fine:Track id="captions" display={story.caption} correspondence={story.caption.correspondence}map={timing.map} space={speech.space}plan={cue-plan.plan} program={caption-program}/>- <space:Canvas id="vertical" width="1080" height="1920"/>- <space:Frame id="title-frame" within={vertical}- left="6%" top="6%" right="6%" bottom="84%"/><text:Track id="titles" space={speech.space}><text:Item text="MEANING" during="full" frame={title-frame}appearance={studio.text.title}/>diff --git a/examples/talking-film-graph-check/studio.svs b/examples/talking-film-graph-check/studio.svsindex e37174c6..8bee8108 100644--- a/examples/talking-film-graph-check/studio.svs+++ b/examples/talking-film-graph-check/studio.svs@@ -2,9 +2,13 @@<sheet version="1">film.vertical { background: #09090B; }- broll.card {- stack-order: 40; x: 0.1; y: 0.2; width: 0.8; height: 0.5; fit: cover;- background: #111116; radius: 20; enter: slide-up 4f; exit: fade 4f;+ media.card {+ stack-order: 40; fit: cover; playback: hold-start;+ frame-paint: #111116; clip: rounded; radius: 20;+ }+ motion.card {+ enter: slide; enter-frames: 4; enter-direction: up; enter-easing: ease-out;+ exit: fade; exit-frames: 4; exit-easing: ease-in;}caption.base {cue-min-words: 1; cue-max-words: 5;diff --git a/examples/talking-film-graph-check/svml.packages.lock b/examples/talking-film-graph-check/svml.packages.lockindex 9b217405..d5ba9f20 100644--- a/examples/talking-film-graph-check/svml.packages.lock+++ b/examples/talking-film-graph-check/svml.packages.lock@@ -6,11 +6,6 @@"version": "0.0.0-dev","digest": "sha256:b16107fdad5256905782fe28142cd19e77673f190691e839fd12d4f1f536ea0f"},- {- "name": "@narratage/broll",- "version": "0.0.0-dev",- "digest": "sha256:3b29285798f91a041a02b19abea52c09e7d954541186a51ea752368d0f5241a2"- },{"name": "@narratage/caption-fine","version": "0.0.0-dev",@@ -24,7 +19,7 @@{"name": "@narratage/caption","version": "0.0.0-dev",- "digest": "sha256:5853264aff66206c602d427ab31bfce5f44814f061e4fb46875d4176afd0ce7d"+ "digest": "sha256:71d9337b12bd73e962a4f21cf08ba5b70d6551e38f86b5927aa25ceb05ec2a20"},{"name": "@narratage/component-kit",@@ -34,7 +29,7 @@{"name": "@narratage/composition","version": "0.0.0-dev",- "digest": "sha256:b02b20d71e8ee4006056cb05d567cb4087971aab2d7f7823357c413b1319325e"+ "digest": "sha256:3a08807cfdbd28ca332c46644a6d807004d23aff9b683d46fca08e92ecad2368"},{"name": "@narratage/core",@@ -44,7 +39,7 @@{"name": "@narratage/elaborator","version": "0.0.0-dev",- "digest": "sha256:9d207daffcefb3d8e12c458ef6a40114b7bb53b3d6f42119c8c5c67e4d0ed8a9"+ "digest": "sha256:1a59d1179cf5ea91d05f4c3976c50502a2a8faad07a15bda8b51d5741668777b"},{"name": "@narratage/estimate",@@ -69,12 +64,17 @@{"name": "@narratage/hyperframes","version": "0.0.0-dev",- "digest": "sha256:bb074d98e7f4a43adce3ea2860486798f9a5be6a30c5e412adec8be6cf5fd031"+ "digest": "sha256:93c7a9a5ca241784d10356593c0be1575df0a846a9ae263888646a2f57385a36"},{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:7da4f1b8f84fa6c7ef3a463ade22eabd50b37857fa8d9d3b7813dd34b95de714"+ "digest": "sha256:dad3a50c6ca9b1f1c18b5d7906387be8645c903190357b0944cf8d7f08546824"+ },+ {+ "name": "@narratage/media-track",+ "version": "0.0.0-dev",+ "digest": "sha256:f6a874bf30529539a9956e1965a04c564f65a128d9e96c39bb0cf2c12b164163"},{"name": "@narratage/media",@@ -139,17 +139,17 @@{"name": "@narratage/spatial","version": "0.0.0-dev",- "digest": "sha256:f0bc551c5ccf6897181e05aa89e2265dbe98f8a71e02cd2b10cb5f946a7037a3"+ "digest": "sha256:ed17712ab17d3714befe14ba5faa9324bf1d9aff624817415cf3199eb26c915a"},{"name": "@narratage/speech-alignment","version": "0.0.0-dev",- "digest": "sha256:98d4d047d49998e6a7370f9e44bb8c3fea7576225068a5a8c0144bfa616547d8"+ "digest": "sha256:c672058b0f1ffad58801d138511f835c07ff4a576dd1c70d3b022e7bb3926389"},{"name": "@narratage/speech-basis","version": "0.0.0-dev",- "digest": "sha256:7df71fda858bf23f32a563e7ca308bf98a8293a63146b8ba341e769fe9de62d6"+ "digest": "sha256:cb7b02dcb2fd7a834505f08c8c62ae00b4da9c806fba10dc8a653bc77a13c724"},{"name": "@narratage/speech-evidence",@@ -159,17 +159,17 @@{"name": "@narratage/speech-spine","version": "0.0.0-dev",- "digest": "sha256:5b7fde204276f70d5b6f5b141055f4b77de6b13c5cd204b41050910aaeee4b0b"+ "digest": "sha256:3becadbd81d7ce0e2caa19049d0d56fad988a875bcfe51c56e9ea6d040294163"},{"name": "@narratage/speech","version": "0.0.0-dev",- "digest": "sha256:0a6efec25a5f16e1f33dd4683955ccd26fc56f0471018ce87245dbb97a4a4a16"+ "digest": "sha256:baf3cc5d2142b87240f2b9ebebc14d58ffd30d6b57691e0e36323f52c59a179c"},{"name": "@narratage/svs","version": "0.0.0-dev",- "digest": "sha256:6054e6dc0fed6195e881ba3d980eed86b33168252b1f38757618343c41e76898"+ "digest": "sha256:351ff3b7bde621cdd258ed48a7906aabe4158a21c1e7ecad227e93655d8842d9"},{"name": "@narratage/temporal",@@ -199,7 +199,7 @@{"name": "@narratage/whisperx","version": "0.0.0-dev",- "digest": "sha256:a72627ed3b15fa02239ba7670b03cd581dfa05ef92451fbd04dff7a5e3b5bb56"+ "digest": "sha256:fb6014901948f977c8e388c07694568419aa5bb0daffd95370a0968a2e840430"}],"packages": [@@ -212,23 +212,14 @@"facetsDigest": "sha256:cf6435732ab7b25c8d0c225042482d940344d5541fe37a4d9cbf905a0d0dad7a","closureDigest": "sha256:d4386b53042e584f162c2882bdc3b515c19f10a87b23c3e9634852fdad75c1ae"},- {- "specifier": "@narratage/broll",- "package": {- "name": "@narratage/broll",- "version": "0.0.0-dev"- },- "facetsDigest": "sha256:61f962f6ce0478a71baafc4e043ca9e977d746914126a2363047ca14ffa7db4c",- "closureDigest": "sha256:9a91f192c5bd38044d51ed58e1c23d5ca742c098baaf049c52db583f9f1e38f6"- },{"specifier": "@narratage/caption","package": {"name": "@narratage/caption","version": "0.0.0-dev"},- "facetsDigest": "sha256:83e25dd21523d0986c23ec090b17bef9b1b8e01414b966caaebde4f238399648",- "closureDigest": "sha256:3a29f5d83eea4f3f145efc4a0ed71632b40b047a55369ec9e268994ab1cbd7c9"+ "facetsDigest": "sha256:071c416e7db22ff4be9436f6abc67621ee1895a6f50313f3b6d2d53e6bf34e77",+ "closureDigest": "sha256:36879850299b9d44be9893443cbf0aad6ca7cf5f9c34e52b0f1086313b7a2f33"},{"specifier": "@narratage/caption-fine",@@ -236,8 +227,8 @@"name": "@narratage/caption-fine","version": "0.0.0-dev"},- "facetsDigest": "sha256:1364a1a82036a5506c658a4482ae6351a3f3a9b7f20e53baf6a717a7a03281cd",- "closureDigest": "sha256:996ea794bc4df1de51020e33514d5a1aa5736c5a7b1d873eae9f30f35f9b035a"+ "facetsDigest": "sha256:b69e22b722cd997dde42023c51b32211e456e35a9ef9ba637ad925f3cb1e7a1c",+ "closureDigest": "sha256:b251ce56441ef56ff21dcb468f79d9c65865be921e62287e09e364a7e83d1b52"},{"specifier": "@narratage/caption-gemini",@@ -245,8 +236,8 @@"name": "@narratage/caption-gemini","version": "0.0.0-dev"},- "facetsDigest": "sha256:e3a0c9a52ab4ad3012b4c35983a6a35bd6c44b81276d2b025aff505bff302d2c",- "closureDigest": "sha256:3c3f55d03c860687ff472dbb499b6b9bf64e0381bf9fc01d1be9b81e79de0b12"+ "facetsDigest": "sha256:e0a2507fb60a673c7bb624261e4ef840ca10d4c484d9f9f82e1ef129ba1dbe0b",+ "closureDigest": "sha256:54cb664fd62afe3d7ec0c6ceecb53df4e25bfeb033702fc86d4045298e045c3c"},{"specifier": "@narratage/composition",@@ -254,8 +245,8 @@"name": "@narratage/composition","version": "0.0.0-dev"},- "facetsDigest": "sha256:63033490f1be1fb90998152809079acf6ae1a20e359146ef1d1bbbfc8f8e608f",- "closureDigest": "sha256:feaeb83ebb2c6150eebe666ee7bd15f234ab8ceba9f083b27b37c5266acb8f7b"+ "facetsDigest": "sha256:e7fecbcebc9a7f20e8db6760ff69a9056735e7ed1f6454e12770af48700b41dc",+ "closureDigest": "sha256:911cb6188bd23cb63490673b1927d08ee0fcdbf7b6a236be5d24d94c57d6ae72"},{"specifier": "@narratage/estimate",@@ -263,8 +254,8 @@"name": "@narratage/estimate","version": "0.0.0-dev"},- "facetsDigest": "sha256:2bae43e32fb4e857294c13363b7d834d81bbfdb2992534c8418b7ca1de5e9cb4",- "closureDigest": "sha256:ddf4ea9e7db521f3a786f5bb45818df8a42643328956510d66cceeda4237fe17"+ "facetsDigest": "sha256:5f4b2fc1016f1a2d4b7d7a065f7b0082164ddb7af7e17030036b21b05c984bb2",+ "closureDigest": "sha256:da7e7ffc106205ba28874399ed484db32063c00256ca6c2e57c6a948823c92cc"},{"specifier": "@narratage/film",@@ -272,8 +263,8 @@"name": "@narratage/film","version": "0.0.0-dev"},- "facetsDigest": "sha256:c05f05c0f91cc84e9393749ef03cab2fd261d326e4f03d54747f8c132aecd36f",- "closureDigest": "sha256:6a25be2681ce047899cfa06e7cf6af7544e60f43477d99d93a4818f1a4a2fb30"+ "facetsDigest": "sha256:0b02cd7057b1778da0fc2949967913787feab879aa301a9453352b5c6c10d439",+ "closureDigest": "sha256:5311d9ae01b6ecb8fcaf49cb6a34175eebf2801b47806bbca9bd1707d2a7bcd2"},{"specifier": "@narratage/generation",@@ -290,8 +281,8 @@"name": "@narratage/hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:be96ac7d0e7b980426fa5596272b326453a819c99ceb581d9280d76f0d58c009",- "closureDigest": "sha256:9614017ee055d0f166a8a73e0e75938a4ccebec4c40f83c33ac39317b7f94d39"+ "facetsDigest": "sha256:ee88f89eb47a7c06a1df3523a15a8ced2b1acbaac78ff6cc297ef8234b807bbf",+ "closureDigest": "sha256:be5513a7d2ac203069a5b926d8661dec80574f4060a9c96ee874715faf7c7103"},{"specifier": "@narratage/media",@@ -300,7 +291,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b365049728f4fd4eaa8cf93792697c89cf7871b1f333b1d4b5bef1113dcfb218",- "closureDigest": "sha256:4b4573f6fcd2ddacaf39c2c37a2aebebf5b6959aa9fb3a14df4e3dd819712b99"+ "closureDigest": "sha256:d8cbdc19dac06b36788e22c98e82deae4683b40739f77f05b026f261a1e48a7d"},{"specifier": "@narratage/media-pipeline",@@ -308,8 +299,17 @@"name": "@narratage/media-pipeline","version": "0.0.0-dev"},- "facetsDigest": "sha256:295304e9d68a8e02bf8ef51b0ebb2a4828337b8ddb665e9dee360c9c965c8ba4",- "closureDigest": "sha256:1aecf7ff98e966329cd1fa8bef018739b6ea0de30a53b3ef2e66e70ce062571a"+ "facetsDigest": "sha256:0ab71314be714a96b94a79ffdb541aebc6613f707b45a5a896285985628f93f2",+ "closureDigest": "sha256:eece5ab677d8946755def49fb433e88f80a60a15cfda9e0be10057e7868faec5"+ },+ {+ "specifier": "@narratage/media-track",+ "package": {+ "name": "@narratage/media-track",+ "version": "0.0.0-dev"+ },+ "facetsDigest": "sha256:db4ee70c3b876853d007cbf0b31856fde7ae3cb4e4b77e3323540c6deb730757",+ "closureDigest": "sha256:f26ab1bc9987ac567a2f898c7a2e658bd32a855d89b5470b54058e6bdbb60c7a"},{"specifier": "@narratage/narrative",@@ -335,8 +335,8 @@"name": "@narratage/render-hyperframes","version": "0.0.0-dev"},- "facetsDigest": "sha256:0ab12fa11602bb87353e5d6ad8e21dc0801a9b79e1dd6a45f7f1812047d3ba84",- "closureDigest": "sha256:34d96f4abbfbcd795765578140162f3409a0d55f98e9b23c05ad65eb404c23f2"+ "facetsDigest": "sha256:136723c3a4cc5fdd1bc64fc65b2362ff48ee9512a1a4567bc243591b6b5eaf44",+ "closureDigest": "sha256:2238ece63ae0c148594c8b346970d97f0e741fccd26faf0d39037ce1af0d9fa3"},{"specifier": "@narratage/run-text",@@ -345,7 +345,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:fbcab7cea86223fe00ea04b51c2160bb534aafbb99067f5877e6825b0ce4eefc",- "closureDigest": "sha256:698ff071b25054a67f6f69e485857664613027b4711d64285a21cc569591414a"+ "closureDigest": "sha256:e1b12334e545c3c06354a822c481e3fa94a96ef6b70bb7f415a50c02d03693b0"},{"specifier": "@narratage/script",@@ -354,7 +354,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:1407d1c367c4131e3419837a66d0b81ae7a13ff44413540f8f63ef9663f6792f",- "closureDigest": "sha256:db7038d84b8155caa5ec2d61525e41ae20677ae753456825599f320215c0992e"+ "closureDigest": "sha256:75ebf01982a17d22f0cc87b23ceb2f3966e2edc20ec273250d4114379647716b"},{"specifier": "@narratage/seedance",@@ -362,8 +362,8 @@"name": "@narratage/seedance","version": "0.0.0-dev"},- "facetsDigest": "sha256:c0a306fdfdf2dfa9489b6bdbd00e48275e2989ce1cbd6d32bc467618c7ae13cd",- "closureDigest": "sha256:4bff3f4e973dc3be6317ae6d3e01e9306d4b696829ad553070a6fe9c78e6db39"+ "facetsDigest": "sha256:15f48aee0a7176aa0daf74332b0d331752d5b0e0f0694ba1b58b95b4a0575c95",+ "closureDigest": "sha256:280e8fdfa3d1cfb0420b454ed88bcf75648348faa682ee897db87f84d65fd88a"},{"specifier": "@narratage/semantic-map",@@ -371,8 +371,8 @@"name": "@narratage/semantic-map","version": "0.0.0-dev"},- "facetsDigest": "sha256:3598b75d25da49ae97aae0a74e0e97d2632902c88445994264671eb4d5d59edd",- "closureDigest": "sha256:b5728f9b4f90b6692d18bed8aae3f52f9dc232928f7137f070af8195a9fd72d2"+ "facetsDigest": "sha256:91e600a9b2e1f8764f743874c9cb3d7a37135ebe3d832428ce55b820a9baacd1",+ "closureDigest": "sha256:38a57ebc3fd9de38a1bdb5ae34b07234bf797337e641f1c2c889c6e672fa83d6"},{"specifier": "@narratage/spatial",@@ -380,8 +380,8 @@"name": "@narratage/spatial","version": "0.0.0-dev"},- "facetsDigest": "sha256:1a3004524c7cdb09be7c84dfdfac577a7e133b7f7b6e6c2301298aaad91053aa",- "closureDigest": "sha256:fe56db252c01c4ec6df7b2521e19e2405f072283711724cd82d9708460d06b9b"+ "facetsDigest": "sha256:d72520078836b2fb2d5956d246f05707d7da7f566e7fa6c69f13f67659d22266",+ "closureDigest": "sha256:2af38c47ef4d854c4117e7637e0b6167012d54a50aae73de8e96de0df543db28"},{"specifier": "@narratage/speech",@@ -389,8 +389,8 @@"name": "@narratage/speech","version": "0.0.0-dev"},- "facetsDigest": "sha256:3d05fd0f8e2238518bf0e26a8ca0b89af9ec213c8a2569f318a484bd692b18d7",- "closureDigest": "sha256:92017fe3ea14c52c33c7e1ef3e30946bdcdd6441512c55fd762b9fa18d025a7c"+ "facetsDigest": "sha256:cde5bb74f05f98e093728c4d80d818adfa3f2fc3d000b6f47b16150c9d3bb953",+ "closureDigest": "sha256:925eb115b12d4cf17501862ff901b46e1cf3b79979795c20c738e7198cae16ce"},{"specifier": "@narratage/speech-alignment",@@ -398,8 +398,8 @@"name": "@narratage/speech-alignment","version": "0.0.0-dev"},- "facetsDigest": "sha256:068f7e40d6c82e122d28e105830e8e746e7a383e9bfe8e975004d56173d47f16",- "closureDigest": "sha256:9a29d77058bd3bc507f1d69548888827c4c0d1b98ade5ab767a6d00e64179ec6"+ "facetsDigest": "sha256:78ec7602fd8b161e1b323e5680c758ef0c3765f2eeb93fafb9fd2408356d73dc",+ "closureDigest": "sha256:55f68ba65ddc1802cbc20e1f204bf3804cd5c3b50ab424666e6490d631c69157"},{"specifier": "@narratage/speech-basis",@@ -407,8 +407,8 @@"name": "@narratage/speech-basis","version": "0.0.0-dev"},- "facetsDigest": "sha256:a05cf6c4ed4839bd7c82aec72d414c0a969127c7f03c23d7b82860a7a6c4f831",- "closureDigest": "sha256:46700e5257dcb675c816a73f40674e237a76168d8ef4228e334ea36d36199200"+ "facetsDigest": "sha256:65b67f0e22be29ade11cafcd59f956cb11e2a0b28ae6d2ae25b40194e237f941",+ "closureDigest": "sha256:b01d15af2279db1efdbda7daec6c63d17a11a67211fdbc192f25e3ce731d628a"},{"specifier": "@narratage/speech-evidence",@@ -416,8 +416,8 @@"name": "@narratage/speech-evidence","version": "0.0.0-dev"},- "facetsDigest": "sha256:8f6bae5441065ef2c99bbb09bcf2b9b2bf70a7b1ca38fae2211baaecff3e6539",- "closureDigest": "sha256:e22ab9b6ef69591df0432dd915b62f8009a158718ca9875a770c8657b83faa81"+ "facetsDigest": "sha256:c555f832f6246d57523fd6d6837f1a191667f6a8efedbaf41ee7108d94e93952",+ "closureDigest": "sha256:91debf95bc3f59504f6026fe37d195299c988699145630e5aa37966a328e26fb"},{"specifier": "@narratage/speech-spine",@@ -425,8 +425,8 @@"name": "@narratage/speech-spine","version": "0.0.0-dev"},- "facetsDigest": "sha256:780a944f7151a67ab92fd3830c153a867544e733f100106b4b1a133408808221",- "closureDigest": "sha256:0c3a6e7d936edd49005e241765de4446009912e72da5ea9041bbb650d00e1cd1"+ "facetsDigest": "sha256:473a49c922d0caae73b1d3102f82b74baed609c0a6730d4f0909f7cad2b4e49a",+ "closureDigest": "sha256:350fc90323c26b69aad40cc4d34e08502e2f06259d3ffefca032bb6b287174b2"},{"specifier": "@narratage/svs",@@ -435,7 +435,16 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:9cc7c397d2cd97b5a3e814886b6553b6d236fd8825df0d056e328b7f0cf20163",- "closureDigest": "sha256:6acbe73fe544447499f51bb74ee3c6e1d67ba34cbef12a33bbee8b34f8a1bc49"+ "closureDigest": "sha256:4ca6299ca8ff060e690e3fd64ef135b913bc03d8008ed9c6b4775be51ff1aa28"+ },+ {+ "specifier": "@narratage/temporal",+ "package": {+ "name": "@narratage/temporal",+ "version": "0.0.0-dev"+ },+ "facetsDigest": "sha256:baed248cfe1d4ea0e6924201a62ae7a5409bba5e995fd4a4b37eeb5c0738a4a2",+ "closureDigest": "sha256:ca1540d21f4275e22c9c81eafdffe6103b4100519ec09419444a7aedcb25191a"},{"specifier": "@narratage/text-track",@@ -443,8 +452,8 @@"name": "@narratage/text-track","version": "0.0.0-dev"},- "facetsDigest": "sha256:c62ee46204b26647aea09b0110ac59724119763fcae5b7a04549b6a1e3eaa66f",- "closureDigest": "sha256:b7b8a63ece30b1c22853801d8ae5853a015a6173253bfb08d354c96f58e66c20"+ "facetsDigest": "sha256:9bb698436ebc871b4a3ffddd1608d84f8d618ebe2e2017933fd2d8a34e762424",+ "closureDigest": "sha256:b8fae0b60355fa49ec445d9cbed076c11269c94323f7eaf41f3dae21c6298cd5"},{"specifier": "@narratage/visual-ir",@@ -461,9 +470,9 @@"name": "@narratage/whisperx","version": "0.0.0-dev"},- "facetsDigest": "sha256:58bd17f5beb706ba6d4ee8c9221a85fa81fb5a65772a17747c480dd881b21cde",- "closureDigest": "sha256:dae2140029b7054846648d38c991857adf2160ed1afb8a3720a71f6f741d5ae2"+ "facetsDigest": "sha256:14922531807c6de8acb17c869731efdc487c1e8b61a878bf8d65154381b9b980",+ "closureDigest": "sha256:038e0a971ed6cec9e12a20ca5b491deaada98a92f3429f465a536791888a7c74"}],- "digest": "sha256:b43c4e571daf47f059f942956a443159112c587f42f1655866f7f4e0d170f113"+ "digest": "sha256:81700cabd86f435dd16447efb7c48bb4db03490b8894801e03c512bdb99dc7e2"}diff --git a/examples/talking-film-live/main.svml b/examples/talking-film-live/main.svmlindex 8ae24a72..ea8e397c 100644--- a/examples/talking-film-live/main.svml+++ b/examples/talking-film-live/main.svml@@ -83,7 +83,7 @@<seedance:Reference image={presenter} role="character"/></seedance:Speech>- <speech:Spine id="speech">+ <speech:Spine id="speech" canvas={vertical}><speech:Take source={opening-take} segment={story.segment.opening}/><speech:Take source={answer-take} segment={story.segment.answer}/></speech:Spine>diff --git a/examples/talking-film-live/svml.runtime-packages.lock b/examples/talking-film-live/svml.runtime-packages.lockindex 141f80bf..fed7092f 100644--- a/examples/talking-film-live/svml.runtime-packages.lock+++ b/examples/talking-film-live/svml.runtime-packages.lock@@ -34,12 +34,12 @@{"name": "@narratage/caption-gemini","version": "0.0.0-dev",- "digest": "sha256:418af81230bf3ac8a7e307484b6098f283575bbae0b18a94cb525c3b52ce26e8"+ "digest": "sha256:9744f6fee5bb6f69c232ca1dcff1db21e20f5d5e28257dae1944d70e5358a6bd"},{"name": "@narratage/caption","version": "0.0.0-dev",- "digest": "sha256:3a569665a5bc60fde2f049786f76928e0729b4fd723c91a38d568014f53cdbb1"+ "digest": "sha256:71d9337b12bd73e962a4f21cf08ba5b70d6551e38f86b5927aa25ceb05ec2a20"},{"name": "@narratage/component-kit",@@ -49,7 +49,7 @@{"name": "@narratage/composition","version": "0.0.0-dev",- "digest": "sha256:7184d720c545663536476115bfc2004f92625eacdeeeac5958f0c087e5fa7d5a"+ "digest": "sha256:3a08807cfdbd28ca332c46644a6d807004d23aff9b683d46fca08e92ecad2368"},{"name": "@narratage/core",@@ -59,7 +59,7 @@{"name": "@narratage/elaborator","version": "0.0.0-dev",- "digest": "sha256:9d207daffcefb3d8e12c458ef6a40114b7bb53b3d6f42119c8c5c67e4d0ed8a9"+ "digest": "sha256:1a59d1179cf5ea91d05f4c3976c50502a2a8faad07a15bda8b51d5741668777b"},{"name": "@narratage/endpoint-kit",@@ -79,17 +79,17 @@{"name": "@narratage/hyperframes","version": "0.0.0-dev",- "digest": "sha256:48f6b2bb2f90ea9f5d11fb8b1d94b5e7ccdb552dd5eb4f08cdbb5631c69de8c0"+ "digest": "sha256:93c7a9a5ca241784d10356593c0be1575df0a846a9ae263888646a2f57385a36"},{"name": "@narratage/media-execution","version": "0.0.0-dev",- "digest": "sha256:f1998bb2b93e848c4adebf797bb645d10361aa7ba68e449c09f03709ceeb9ad1"+ "digest": "sha256:d00a933d8a5ca93366be96fdb1bd83da7557a311d11e50e930652872d3f421c2"},{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:4ceb5c9ef785e456f3807809b0384cbd40d463c4e5f1878375dcc8b017799922"+ "digest": "sha256:dad3a50c6ca9b1f1c18b5d7906387be8645c903190357b0944cf8d7f08546824"},{"name": "@narratage/media",@@ -104,7 +104,7 @@{"name": "@narratage/program-space","version": "0.0.0-dev",- "digest": "sha256:3a321f064e7f763ae056662ed07bbe4b0fdf1b1628502891c4c73beb5a20edd6"+ "digest": "sha256:18ff5d1109b91ac83733c69ca5d0d5162a07a6f1e99d1473128c12382d116eaf"},{"name": "@narratage/protocol",@@ -119,7 +119,7 @@{"name": "@narratage/provider-hyperframes-local","version": "0.0.0-dev",- "digest": "sha256:0281e943dbf30074f75ba4551c462ea634f8d14d7490a75bf60cc672aaceee91"+ "digest": "sha256:32b6d1fb16ccf37dedb6af205a9b1548a18b96ceb90cfce012ce3cd95945c2fe"},{"name": "@narratage/provider-kie",@@ -129,7 +129,7 @@{"name": "@narratage/provider-media-local","version": "0.0.0-dev",- "digest": "sha256:fc1812c1603897d98073a9a42224bb3051d4d533b41edbdcc102c943eda3393e"+ "digest": "sha256:561c3321959c46194574f7d3147462c3d3986008f4fc1c028eac1610e9a3edff"},{"name": "@narratage/provider-whisperx-local",@@ -166,10 +166,15 @@"version": "0.0.0-dev","digest": "sha256:8bbfa48c0fde45639578902c907d9190bd5bbb8a03bd59fca8fd364b820a3184"},+ {+ "name": "@narratage/spatial",+ "version": "0.0.0-dev",+ "digest": "sha256:ed17712ab17d3714befe14ba5faa9324bf1d9aff624817415cf3199eb26c915a"+ },{"name": "@narratage/speech-alignment","version": "0.0.0-dev",- "digest": "sha256:9f54aefc38b06344902d0a9f1bb0e76f03cd9e9e29d1bd8d79263979c73da6d2"+ "digest": "sha256:c672058b0f1ffad58801d138511f835c07ff4a576dd1c70d3b022e7bb3926389"},{"name": "@narratage/speech-evidence",@@ -179,7 +184,7 @@{"name": "@narratage/speech","version": "0.0.0-dev",- "digest": "sha256:1296f5241ab87621cbed2019ffd09e51a2ecd79a49848715ade628a9d722d8c1"+ "digest": "sha256:baf3cc5d2142b87240f2b9ebebc14d58ffd30d6b57691e0e36323f52c59a179c"},{"name": "@narratage/text",@@ -199,7 +204,7 @@{"name": "@narratage/whisperx","version": "0.0.0-dev",- "digest": "sha256:df11edd20f992955c1d0851828e1df94dcfefeecf8ece037a0f5a0d740fe72ee"+ "digest": "sha256:fb6014901948f977c8e388c07694568419aa5bb0daffd95370a0968a2e840430"},{"name": "@protobufjs/aspromise",@@ -248,8 +253,8 @@},{"name": "@puppeteer/browsers",- "version": "3.0.6",- "digest": "sha256:1fc1212e7cf17104bd0e45e8e63c90b984df83e9a2b11ef8104b162bb140893f"+ "version": "3.1.0",+ "digest": "sha256:2d169f7add89058eb6af9a73a87836111a24e33188e0871b337a80321e7fc1c6"},{"name": "@swc/helpers",@@ -533,8 +538,8 @@},{"name": "hyperframes",- "version": "0.7.84",- "digest": "sha256:d61bdd821dc513adcddeb339559f6a165ab7a0e8702ace8e97b6494f51d92b6e"+ "version": "0.7.101",+ "digest": "sha256:04483607a6da0af75c880505ce230b31c986b23d23b3a400c15b1fd8445d33d1"},{"name": "ignore",@@ -678,8 +683,8 @@},{"name": "puppeteer-core",- "version": "25.4.0",- "digest": "sha256:05bbe4cff28c147c6f78aa83edfd1180321d4734ec4180077ab4ac5f8e8c031e"+ "version": "25.5.0",+ "digest": "sha256:dde144a3ef6ff39badc532b8f34132f74efba2544f6889fb173c62a763f55cae"},{"name": "restructure",@@ -850,7 +855,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b2636492ab0b0b500d674899ab2ff14fc61c7f2b22ef44eb8f816f141b21ec4d",- "closureDigest": "sha256:5c92620e7d9365cf43721684b39ea5dc8a3807bd24491f340607243511e65213"+ "closureDigest": "sha256:720506401ab75076e47584e35fb4a27e918421679fd54fc51aa60284f4f05f26"},{"specifier": "@narratage/provider-hyperframes-local",@@ -859,7 +864,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b14b721bdc0c9d5d9ced1c345107c92a3a5878f3383dea2ecd3e310442c2a40c",- "closureDigest": "sha256:292da8784b8c8465a66ee6849209bc0c908691b28cf7aceaf2e9fb7d4e506e39"+ "closureDigest": "sha256:a67551d2aebf7f61ffd2d301013bab5f544761f526fb9e45f4013bad6fc40a4f"},{"specifier": "@narratage/provider-kie",@@ -868,7 +873,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:e61ebb911a33d4c5d0e4e4ee4fb69cdb790f4ced050ef35f7472bfbd72013d9b",- "closureDigest": "sha256:9d4675de6cf7ea19cee9f4a0ba2481a160b5af77d1f35678ff27262e97fcb57b"+ "closureDigest": "sha256:6380c880940a8824f8c9e2fe0eefa571e7b9207f6b31d1b16c023fdd8cfab9dc"},{"specifier": "@narratage/provider-media-local",@@ -877,7 +882,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:701abcc923bf8a711c20b6e308d8cc8be27bf560f249b8c5a801e1dbd9cb3979",- "closureDigest": "sha256:1c0b1ad30ee02c1637e25de62ec4d5c207f30b5f927ee11de297e4acc114eeaa"+ "closureDigest": "sha256:fe5a264314799b8bc56ed8fe3d70cc2b04d5812d294b5f488f8f3e7a9ca9931f"},{"specifier": "@narratage/provider-whisperx-local",@@ -886,8 +891,8 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:3686b84c42cc2b1bcaccf57cbdcda25a15c7d9d3778fb92d25437f9880ed8f5d",- "closureDigest": "sha256:c229c953e6eda641deb1116ba4cdf782a94eccbf56537710cbab40a8279207b7"+ "closureDigest": "sha256:63ab771f758da52c4e574545e1ae9f42ae63e1c5f2b1f3197fd67d116c29c5cc"}],- "digest": "sha256:19529231d0dcb2d7045c51c9a2cd5324e788a18b0b9ee5830c7966bc07a6e08f"+ "digest": "sha256:a01b137a018945b40ac5dad764e82da557dcf12bdd6252e0d951cc01f1b9b659"}diff --git a/examples/talking-head-aroll/main.svml b/examples/talking-head-aroll/main.svmlindex db69de2a..8a37751a 100644--- a/examples/talking-head-aroll/main.svml+++ b/examples/talking-head-aroll/main.svml@@ -91,7 +91,7 @@</speaker:Take><!-- Four equal Takes become one ordered speech coordinate space and two peer Tracks. -->- <speech:Spine id="speech">+ <speech:Spine id="speech" canvas={vertical}><speech:Take source={hook-take.video} segment={story.segment.hook}/><speech:Take source={meeting-take.video} segment={story.segment.meeting}/><speech:Take source={evidence-take.video} segment={story.segment.evidence}/>diff --git a/examples/talking-head-aroll/svml.runtime-packages.lock b/examples/talking-head-aroll/svml.runtime-packages.lockindex 141f80bf..fed7092f 100644--- a/examples/talking-head-aroll/svml.runtime-packages.lock+++ b/examples/talking-head-aroll/svml.runtime-packages.lock@@ -34,12 +34,12 @@{"name": "@narratage/caption-gemini","version": "0.0.0-dev",- "digest": "sha256:418af81230bf3ac8a7e307484b6098f283575bbae0b18a94cb525c3b52ce26e8"+ "digest": "sha256:9744f6fee5bb6f69c232ca1dcff1db21e20f5d5e28257dae1944d70e5358a6bd"},{"name": "@narratage/caption","version": "0.0.0-dev",- "digest": "sha256:3a569665a5bc60fde2f049786f76928e0729b4fd723c91a38d568014f53cdbb1"+ "digest": "sha256:71d9337b12bd73e962a4f21cf08ba5b70d6551e38f86b5927aa25ceb05ec2a20"},{"name": "@narratage/component-kit",@@ -49,7 +49,7 @@{"name": "@narratage/composition","version": "0.0.0-dev",- "digest": "sha256:7184d720c545663536476115bfc2004f92625eacdeeeac5958f0c087e5fa7d5a"+ "digest": "sha256:3a08807cfdbd28ca332c46644a6d807004d23aff9b683d46fca08e92ecad2368"},{"name": "@narratage/core",@@ -59,7 +59,7 @@{"name": "@narratage/elaborator","version": "0.0.0-dev",- "digest": "sha256:9d207daffcefb3d8e12c458ef6a40114b7bb53b3d6f42119c8c5c67e4d0ed8a9"+ "digest": "sha256:1a59d1179cf5ea91d05f4c3976c50502a2a8faad07a15bda8b51d5741668777b"},{"name": "@narratage/endpoint-kit",@@ -79,17 +79,17 @@{"name": "@narratage/hyperframes","version": "0.0.0-dev",- "digest": "sha256:48f6b2bb2f90ea9f5d11fb8b1d94b5e7ccdb552dd5eb4f08cdbb5631c69de8c0"+ "digest": "sha256:93c7a9a5ca241784d10356593c0be1575df0a846a9ae263888646a2f57385a36"},{"name": "@narratage/media-execution","version": "0.0.0-dev",- "digest": "sha256:f1998bb2b93e848c4adebf797bb645d10361aa7ba68e449c09f03709ceeb9ad1"+ "digest": "sha256:d00a933d8a5ca93366be96fdb1bd83da7557a311d11e50e930652872d3f421c2"},{"name": "@narratage/media-pipeline","version": "0.0.0-dev",- "digest": "sha256:4ceb5c9ef785e456f3807809b0384cbd40d463c4e5f1878375dcc8b017799922"+ "digest": "sha256:dad3a50c6ca9b1f1c18b5d7906387be8645c903190357b0944cf8d7f08546824"},{"name": "@narratage/media",@@ -104,7 +104,7 @@{"name": "@narratage/program-space","version": "0.0.0-dev",- "digest": "sha256:3a321f064e7f763ae056662ed07bbe4b0fdf1b1628502891c4c73beb5a20edd6"+ "digest": "sha256:18ff5d1109b91ac83733c69ca5d0d5162a07a6f1e99d1473128c12382d116eaf"},{"name": "@narratage/protocol",@@ -119,7 +119,7 @@{"name": "@narratage/provider-hyperframes-local","version": "0.0.0-dev",- "digest": "sha256:0281e943dbf30074f75ba4551c462ea634f8d14d7490a75bf60cc672aaceee91"+ "digest": "sha256:32b6d1fb16ccf37dedb6af205a9b1548a18b96ceb90cfce012ce3cd95945c2fe"},{"name": "@narratage/provider-kie",@@ -129,7 +129,7 @@{"name": "@narratage/provider-media-local","version": "0.0.0-dev",- "digest": "sha256:fc1812c1603897d98073a9a42224bb3051d4d533b41edbdcc102c943eda3393e"+ "digest": "sha256:561c3321959c46194574f7d3147462c3d3986008f4fc1c028eac1610e9a3edff"},{"name": "@narratage/provider-whisperx-local",@@ -166,10 +166,15 @@"version": "0.0.0-dev","digest": "sha256:8bbfa48c0fde45639578902c907d9190bd5bbb8a03bd59fca8fd364b820a3184"},+ {+ "name": "@narratage/spatial",+ "version": "0.0.0-dev",+ "digest": "sha256:ed17712ab17d3714befe14ba5faa9324bf1d9aff624817415cf3199eb26c915a"+ },{"name": "@narratage/speech-alignment","version": "0.0.0-dev",- "digest": "sha256:9f54aefc38b06344902d0a9f1bb0e76f03cd9e9e29d1bd8d79263979c73da6d2"+ "digest": "sha256:c672058b0f1ffad58801d138511f835c07ff4a576dd1c70d3b022e7bb3926389"},{"name": "@narratage/speech-evidence",@@ -179,7 +184,7 @@{"name": "@narratage/speech","version": "0.0.0-dev",- "digest": "sha256:1296f5241ab87621cbed2019ffd09e51a2ecd79a49848715ade628a9d722d8c1"+ "digest": "sha256:baf3cc5d2142b87240f2b9ebebc14d58ffd30d6b57691e0e36323f52c59a179c"},{"name": "@narratage/text",@@ -199,7 +204,7 @@{"name": "@narratage/whisperx","version": "0.0.0-dev",- "digest": "sha256:df11edd20f992955c1d0851828e1df94dcfefeecf8ece037a0f5a0d740fe72ee"+ "digest": "sha256:fb6014901948f977c8e388c07694568419aa5bb0daffd95370a0968a2e840430"},{"name": "@protobufjs/aspromise",@@ -248,8 +253,8 @@},{"name": "@puppeteer/browsers",- "version": "3.0.6",- "digest": "sha256:1fc1212e7cf17104bd0e45e8e63c90b984df83e9a2b11ef8104b162bb140893f"+ "version": "3.1.0",+ "digest": "sha256:2d169f7add89058eb6af9a73a87836111a24e33188e0871b337a80321e7fc1c6"},{"name": "@swc/helpers",@@ -533,8 +538,8 @@},{"name": "hyperframes",- "version": "0.7.84",- "digest": "sha256:d61bdd821dc513adcddeb339559f6a165ab7a0e8702ace8e97b6494f51d92b6e"+ "version": "0.7.101",+ "digest": "sha256:04483607a6da0af75c880505ce230b31c986b23d23b3a400c15b1fd8445d33d1"},{"name": "ignore",@@ -678,8 +683,8 @@},{"name": "puppeteer-core",- "version": "25.4.0",- "digest": "sha256:05bbe4cff28c147c6f78aa83edfd1180321d4734ec4180077ab4ac5f8e8c031e"+ "version": "25.5.0",+ "digest": "sha256:dde144a3ef6ff39badc532b8f34132f74efba2544f6889fb173c62a763f55cae"},{"name": "restructure",@@ -850,7 +855,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b2636492ab0b0b500d674899ab2ff14fc61c7f2b22ef44eb8f816f141b21ec4d",- "closureDigest": "sha256:5c92620e7d9365cf43721684b39ea5dc8a3807bd24491f340607243511e65213"+ "closureDigest": "sha256:720506401ab75076e47584e35fb4a27e918421679fd54fc51aa60284f4f05f26"},{"specifier": "@narratage/provider-hyperframes-local",@@ -859,7 +864,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:b14b721bdc0c9d5d9ced1c345107c92a3a5878f3383dea2ecd3e310442c2a40c",- "closureDigest": "sha256:292da8784b8c8465a66ee6849209bc0c908691b28cf7aceaf2e9fb7d4e506e39"+ "closureDigest": "sha256:a67551d2aebf7f61ffd2d301013bab5f544761f526fb9e45f4013bad6fc40a4f"},{"specifier": "@narratage/provider-kie",@@ -868,7 +873,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:e61ebb911a33d4c5d0e4e4ee4fb69cdb790f4ced050ef35f7472bfbd72013d9b",- "closureDigest": "sha256:9d4675de6cf7ea19cee9f4a0ba2481a160b5af77d1f35678ff27262e97fcb57b"+ "closureDigest": "sha256:6380c880940a8824f8c9e2fe0eefa571e7b9207f6b31d1b16c023fdd8cfab9dc"},{"specifier": "@narratage/provider-media-local",@@ -877,7 +882,7 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:701abcc923bf8a711c20b6e308d8cc8be27bf560f249b8c5a801e1dbd9cb3979",- "closureDigest": "sha256:1c0b1ad30ee02c1637e25de62ec4d5c207f30b5f927ee11de297e4acc114eeaa"+ "closureDigest": "sha256:fe5a264314799b8bc56ed8fe3d70cc2b04d5812d294b5f488f8f3e7a9ca9931f"},{"specifier": "@narratage/provider-whisperx-local",@@ -886,8 +891,8 @@"version": "0.0.0-dev"},"facetsDigest": "sha256:3686b84c42cc2b1bcaccf57cbdcda25a15c7d9d3778fb92d25437f9880ed8f5d",- "closureDigest": "sha256:c229c953e6eda641deb1116ba4cdf782a94eccbf56537710cbab40a8279207b7"+ "closureDigest": "sha256:63ab771f758da52c4e574545e1ae9f42ae63e1c5f2b1f3197fd67d116c29c5cc"}],- "digest": "sha256:19529231d0dcb2d7045c51c9a2cd5324e788a18b0b9ee5830c7966bc07a6e08f"+ "digest": "sha256:a01b137a018945b40ac5dad764e82da557dcf12bdd6252e0d951cc01f1b9b659"}diff --git a/package.json b/package.jsonindex baab1950..de83c270 100644--- a/package.json+++ b/package.json@@ -23,7 +23,6 @@"devDependencies": {"@narratage/artifact": "workspace:*","@narratage/audio-track": "workspace:*",- "@narratage/broll": "workspace:*","@narratage/caption": "workspace:*","@narratage/caption-fine": "workspace:*","@narratage/caption-gemini": "workspace:*",diff --git a/packages/audio-track/test/audio-track.test.ts b/packages/audio-track/test/audio-track.test.tsindex 892874a3..29314bde 100644--- a/packages/audio-track/test/audio-track.test.ts+++ b/packages/audio-track/test/audio-track.test.ts@@ -35,6 +35,7 @@ import type { CompleteSemanticMap } from "@narratage/semantic-map";import { semanticMapManifest } from "@narratage/semantic-map";import { speechEvidenceManifest } from "@narratage/speech-evidence";import { speechManifest } from "@narratage/speech";+import { spatialManifest } from "@narratage/spatial";import { temporalManifest } from "@narratage/temporal";import { TextSurfaceRegistry, createTextAuthorFrontend } from "@narratage/text";import { createRecordAdmitter, TypeValidatorRegistry } from "@narratage/validation";@@ -278,6 +279,7 @@ test("the self-described Audio Surface parses into the same finite Producer grapnarrativeManifest,programSpaceManifest,speechManifest,+ spatialManifest,speechEvidenceManifest,semanticMapManifest,temporalManifest,diff --git a/packages/broll/README.md b/packages/broll/README.mddeleted file mode 100644index 56055fce..00000000--- a/packages/broll/README.md+++ /dev/null@@ -1,17 +0,0 @@-# `@narratage/broll`--Official B-roll authoring and lowering package. It owns B-roll items, local-entrance/exit motion, pair transitions, source audio and transition SFX. It-emits ordinary `VisualTrack` and `AudioTrack` values whose Presents participate-in Composition by their own absolute z.--The `<broll:Track>` Surface consumes explicit `CompleteSemanticMap`, authored Script selection,-source Blob and SVS Recipe edges. Each source is inspected and normalized through the shared media-Fragment; semantic-window placement and local motion are deterministic package Operations.--`BrollProgram@1` remains an executable vertical-slice fixture. The author-facing package is not-frozen until the old content-frame plus independent foreground/backdrop sampling witnesses lower-through the closed SVML Visual IR.--It does not generate media, select a Provider, read another Track, sample the-accumulated lower composite or implement Base/Screen FX.diff --git a/packages/broll/package.json b/packages/broll/package.jsondeleted file mode 100644index 9ba49961..00000000--- a/packages/broll/package.json+++ /dev/null@@ -1,29 +0,0 @@-{- "name": "@narratage/broll",- "version": "0.0.0-dev",- "private": true,- "type": "module",- "exports": {- ".": "./src/index.ts"- },- "svml": {- "activation": "./src/activation.ts"- },- "dependencies": {- "@narratage/artifact": "workspace:*",- "@narratage/component-kit": "workspace:*",- "@narratage/composition": "workspace:*",- "@narratage/elaborator": "workspace:*",- "@narratage/media": "workspace:*",- "@narratage/media-pipeline": "workspace:*",- "@narratage/narrative": "workspace:*",- "@narratage/program-space": "workspace:*",- "@narratage/protocol": "workspace:*",- "@narratage/semantic-map": "workspace:*",- "@narratage/svs": "workspace:*",- "@narratage/text": "workspace:*"- },- "devDependencies": {- "@narratage/hyperframes": "workspace:*"- }-}diff --git a/packages/broll/src/activation.ts b/packages/broll/src/activation.tsdeleted file mode 100644index 7b9256ee..00000000--- a/packages/broll/src/activation.ts+++ /dev/null@@ -1,17 +0,0 @@-import { createTextSurfaceHostFacet } from "@narratage/text";-import {- brollComponent, brollManifest, brollModuleRef, brollSurfaceImplementationDigest,- decodeBrollTrackSurface,-} from "./index.js";--export const svmlPackage = {- format: "svml.node-package@1" as const,- name: "@narratage/broll",- modules: [{ manifest: brollManifest, specifiers: ["@narratage/broll", "@narratage/broll@1"] }],- components: [brollComponent],- hostFacets: [createTextSurfaceHostFacet({- module: brollModuleRef, surface: "track", mode: "structured",- implementationDigest: brollSurfaceImplementationDigest, handler: decodeBrollTrackSurface,- })],-};-export default svmlPackage;diff --git a/packages/broll/src/author.ts b/packages/broll/src/author.tsdeleted file mode 100644index e8ed4aef..00000000--- a/packages/broll/src/author.ts+++ /dev/null@@ -1,109 +0,0 @@-import type { NarrativeSelectionRef } from "@narratage/narrative";-import { verifySynchronizedMedia } from "@narratage/media";-import type { SynchronizedMedia } from "@narratage/media";-import type { ProgramSpace } from "@narratage/program-space";-import { assertCompleteSemanticMapIdentity, assertNarrativeSelectionIdentity, selectionFrameSpans } from "@narratage/semantic-map";-import type { CompleteSemanticMap } from "@narratage/semantic-map";-import type { Track } from "@narratage/composition";-import { canonicalize, digestOf } from "@narratage/protocol";--import { sealBrollProgram } from "./program.js";-import type {- BrollItemSpec,- BrollProgram,- BrollSet,- BrollTrackSpec,-} from "./types.js";--export const createBrollSetImplementationDigest = digestOf("@narratage/broll/create-set@1");-export const appendBrollItemImplementationDigest = digestOf("@narratage/broll/append-item@1");-export const finalizeBrollProgramImplementationDigest = digestOf("@narratage/broll/finalize-program@1");--function assert(condition: unknown, message: string): asserts condition {- if (!condition) throw new Error(message);-}--export function sealBrollItemSpec(value: BrollItemSpec): BrollItemSpec {- return canonicalize(value) as unknown as BrollItemSpec;-}--export function assertBrollItemSpec(value: BrollItemSpec): void {- assert(value.contract === "svml.broll-item-spec@1" && value.id.length > 0, "BrollItemSpec identity is invalid");- assert(Number.isSafeInteger(value.z), "BrollItemSpec z must be an integer");- assert(value.fit === "contain" || value.fit === "cover", "BrollItemSpec fit is invalid");-}--export function sealBrollTrackSpec(value: BrollTrackSpec): BrollTrackSpec {- return canonicalize(value) as unknown as BrollTrackSpec;-}--function assertBrollTrackSpec(value: BrollTrackSpec): void {- assert(value.contract === "svml.broll-track-spec@1" && value.id.length > 0, "BrollTrackSpec is invalid");-}--export function assertBrollSet(value: BrollSet): void {- assert(value.contract === "svml.broll-set@1", "BrollSet is invalid");-}--export function createBrollSet(): BrollSet {- return { contract: "svml.broll-set@1", items: [] };-}--export function appendBrollItem(- set: BrollSet,- track: BrollTrackSpec,- map: CompleteSemanticMap,- programSpace: ProgramSpace,- media: SynchronizedMedia,- selection: NarrativeSelectionRef,- spec: BrollItemSpec,-): BrollSet {- assertBrollSet(set);- assertBrollTrackSpec(track);- assertCompleteSemanticMapIdentity(map);- verifySynchronizedMedia(media);- assertNarrativeSelectionIdentity(selection);- assertBrollItemSpec(spec);- assert(media.visual !== undefined, `B-roll ${spec.id} source has no visual stream`);- assert(media.timeline.frameRate.numerator === programSpace.frameRate.numerator- && media.timeline.frameRate.denominator === programSpace.frameRate.denominator,- `B-roll ${spec.id} media uses another frame rate`);- const durationSec = media.timeline.frameCount * media.timeline.frameRate.denominator / media.timeline.frameRate.numerator;- const additions = selectionFrameSpans(map, selection, programSpace).map((span, index) => {- return {- id: selection.occurrences.length === 1 ? spec.id : `${spec.id}:${index + 1}`,- artifact: {- digest: media.visual!.artifact.digest,- size: media.visual!.artifact.size,- mediaType: media.visual!.artifact.mediaType,- durationSec,- },- span,- z: spec.z,- tieBreak: `${track.id}:${spec.id}:${index + 1}`,- box: spec.box,- fit: spec.fit,- playback: "stretch" as const,- ...(spec.backgroundColor === undefined ? {} : { backgroundColor: spec.backgroundColor }),- ...(spec.borderRadiusPx === undefined ? {} : { borderRadiusPx: spec.borderRadiusPx }),- ...(spec.enter === undefined ? {} : { enter: spec.enter }),- ...(spec.exit === undefined ? {} : { exit: spec.exit }),- };- });- return {- contract: "svml.broll-set@1",- items: [...set.items, ...additions],- };-}--export function finalizeBrollProgram(set: BrollSet, track: BrollTrackSpec): BrollProgram {- assertBrollSet(set);- assertBrollTrackSpec(track);- assert(set.items.length > 0, "B-roll Track requires at least one Item");- return sealBrollProgram({- contract: "svml.broll-program@1",- id: track.id,- items: set.items,- transitions: [],- });-}diff --git a/packages/broll/src/component.ts b/packages/broll/src/component.tsdeleted file mode 100644index e96f2ebc..00000000--- a/packages/broll/src/component.ts+++ /dev/null@@ -1,95 +0,0 @@-import type { ComponentPackage } from "@narratage/component-kit";-import type { NarrativeSelection, NarrativeSelectionRef } from "@narratage/narrative";-import type { SynchronizedMedia } from "@narratage/media";-import type { ProgramSpace } from "@narratage/program-space";-import type { CompleteSemanticMap } from "@narratage/semantic-map";-import type { StoredValue } from "@narratage/protocol";-import { canonicalize } from "@narratage/protocol";--import { brollProducers } from "./manifest.js";-import {- appendBrollItem,- appendBrollItemImplementationDigest,- createBrollSet,- createBrollSetImplementationDigest,- finalizeBrollProgram,- finalizeBrollProgramImplementationDigest,-} from "./author.js";-import {- compileBrollImplementationDigest,- compileBrollProduct,- projectBrollAudio,- projectBrollAudioImplementationDigest,- projectBrollVisual,- projectBrollVisualImplementationDigest,-} from "./program.js";-import type { BrollItemSpec, BrollProduct, BrollProgram, BrollSet, BrollTrackSpec } from "./types.js";--function inline<T>(value: StoredValue | undefined, subject: string): T {- if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);- return value.value as unknown as T;-}--export const brollComponent = {- name: "@narratage/broll",- producers: [- {- producer: brollProducers.createSet,- implementationDigest: createBrollSetImplementationDigest,- handler: () => ({- outputs: { set: { kind: "inline", value: canonicalize(createBrollSet()) } }, needs: {},- }),- },- {- producer: brollProducers.appendItem,- implementationDigest: appendBrollItemImplementationDigest,- handler: ({ inputs }) => ({- outputs: { set: { kind: "inline", value: canonicalize(appendBrollItem(- inline<BrollSet>(inputs.set?.value, "BrollSet"),- inline<BrollTrackSpec>(inputs.track?.value, "BrollTrackSpec"),- inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),- inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),- inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"),- inline<BrollItemSpec>(inputs.spec?.value, "BrollItemSpec"),- )) } }, needs: {},- }),- },- {- producer: brollProducers.finalize,- implementationDigest: finalizeBrollProgramImplementationDigest,- handler: ({ inputs }) => ({- outputs: { program: { kind: "inline", value: canonicalize(finalizeBrollProgram(- inline<BrollSet>(inputs.set?.value, "BrollSet"),- inline<BrollTrackSpec>(inputs.track?.value, "BrollTrackSpec"),- )) } }, needs: {},- }),- },- {- producer: brollProducers.compile,- implementationDigest: compileBrollImplementationDigest,- handler: ({ inputs }) => ({- outputs: { product: { kind: "inline", value: canonicalize(compileBrollProduct(- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<BrollProgram>(inputs.program?.value, "BrollProgram"),- )) } },- needs: {},- }),- },- {- producer: brollProducers.projectVisual,- implementationDigest: projectBrollVisualImplementationDigest,- handler: ({ inputs }) => ({- outputs: { visual: { kind: "inline", value: canonicalize(projectBrollVisual(inline<BrollProduct>(inputs.product?.value, "BrollProduct"))) } },- needs: {},- }),- },- {- producer: brollProducers.projectAudio,- implementationDigest: projectBrollAudioImplementationDigest,- handler: ({ inputs }) => ({- outputs: { audio: { kind: "inline", value: canonicalize(projectBrollAudio(inline<BrollProduct>(inputs.product?.value, "BrollProduct"))) } },- needs: {},- }),- },- ],-} satisfies ComponentPackage;diff --git a/packages/broll/src/index.ts b/packages/broll/src/index.tsdeleted file mode 100644index 4a24edf3..00000000--- a/packages/broll/src/index.ts+++ /dev/null@@ -1,39 +0,0 @@-export {- assertBrollProductIdentity,- assertBrollProgramIdentity,- compileBrollImplementationDigest,- compileBrollProduct,- projectBrollAudio,- projectBrollAudioImplementationDigest,- projectBrollVisual,- projectBrollVisualImplementationDigest,- sealBrollProgram,-} from "./program.js";-export { brollComponent } from "./component.js";-export {- appendBrollItem,- appendBrollItemImplementationDigest,- assertBrollItemSpec,- assertBrollSet,- createBrollSet,- createBrollSetImplementationDigest,- finalizeBrollProgram,- finalizeBrollProgramImplementationDigest,- sealBrollItemSpec,- sealBrollTrackSpec,-} from "./author.js";-export {- brollManifest,- brollManifestDigest,- brollModuleRef,- brollProductSchema,- brollProgramSchema,- brollSetSchema,- brollItemSpecSchema,- brollTrackSpecSchema,- brollSurfaceImplementationDigest,- brollProducers,- brollTypes,-} from "./manifest.js";-export { decodeBrollTrackSurface } from "./surface.js";-export type * from "./types.js";diff --git a/packages/broll/src/manifest.ts b/packages/broll/src/manifest.tsdeleted file mode 100644index 07f119a0..00000000--- a/packages/broll/src/manifest.ts+++ /dev/null@@ -1,239 +0,0 @@-import { artifactDependency } from "@narratage/artifact";-import { narrativeDependency, narrativeTypes } from "@narratage/narrative";-import { mediaArtifactSchema, mediaDependency, mediaTypes } from "@narratage/media";-import { programSpaceDependency, programSpaceTypes } from "@narratage/program-space";-import { semanticMapDependency, semanticMapTypes } from "@narratage/semantic-map";-import { audioTrackSchema, compositionDependency, compositionTypes, visualTrackSchema } from "@narratage/composition";-import type { Track } from "@narratage/composition";-import { digestOf } from "@narratage/protocol";-import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@narratage/protocol";-import { mediaPipelineManifest, mediaPipelineModuleRef, mediaPipelineTypes } from "@narratage/media-pipeline";-import { svsManifest, svsModuleRef } from "@narratage/svs";--import {- appendBrollItemImplementationDigest,- createBrollSetImplementationDigest,- finalizeBrollProgramImplementationDigest,-} from "./author.js";--import {- compileBrollImplementationDigest,- projectBrollAudioImplementationDigest,- projectBrollVisualImplementationDigest,-} from "./program.js";--export const brollModuleRef = { name: "@narratage/broll", version: "0.0.0-dev" } as const;-export const brollTypes = {- program: { module: brollModuleRef, name: "BrollProgram" },- product: { module: brollModuleRef, name: "BrollProduct" },- trackSpec: { module: brollModuleRef, name: "BrollTrackSpec" },- itemSpec: { module: brollModuleRef, name: "BrollItemSpec" },- set: { module: brollModuleRef, name: "BrollSet" },-} satisfies Record<string, TypeRef>;-export const brollProducers = {- createSet: { module: brollModuleRef, name: "create-broll-set" },- appendItem: { module: brollModuleRef, name: "append-broll-item" },- finalize: { module: brollModuleRef, name: "finalize-broll-program" },- compile: { module: brollModuleRef, name: "compile-broll" },- projectVisual: { module: brollModuleRef, name: "project-broll-visual" },- projectAudio: { module: brollModuleRef, name: "project-broll-audio" },-} satisfies Record<string, ProducerRef>;--const string = { kind: "string", minLength: 1 } as const;-const number = { kind: "number" } as const;-const unsignedInteger = { kind: "number", integer: true, minimum: 0 } as const;-const signedInteger = { kind: "number", integer: true } as const;-const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>): ValueSchema => ({- kind: "object",- fields,-});-const audioBlobRef = object({- kind: { schema: { kind: "literal", value: "blob" } },- digest: { schema: { kind: "string", minLength: 71, maxLength: 71 } },- size: { schema: unsignedInteger },- mediaType: { schema: { kind: "literal", value: "audio/wav" } },-});--const frameSpan = object({- startFrame: { schema: unsignedInteger },- endFrameExclusive: { schema: unsignedInteger },-});-const box = object({- xPercent: { schema: number },- yPercent: { schema: number },- widthPercent: { schema: number },- heightPercent: { schema: number },-});-const motion = object({- operator: { schema: { kind: "string", enum: ["fade", "pop", "slide-up", "slide-down"] } },- durationFrames: { schema: unsignedInteger },- amount: { schema: number, optional: true },-});-const item = object({- id: { schema: string },- artifact: { schema: mediaArtifactSchema },- span: { schema: frameSpan },- z: { schema: signedInteger },- tieBreak: { schema: string },- box: { schema: box },- fit: { schema: { kind: "string", enum: ["contain", "cover"] } },- playback: { schema: { kind: "string", enum: ["freeze", "native", "loop", "stretch"] } },- mediaStartSec: { schema: number, optional: true },- audio: { schema: object({- artifact: { schema: audioBlobRef },- sampleFrames: { schema: unsignedInteger },- gain: { schema: number },- }), optional: true },- enter: { schema: motion, optional: true },- exit: { schema: motion, optional: true },- backgroundColor: { schema: string, optional: true },- borderRadiusPx: { schema: number, optional: true },-});-const itemSpec = object({- contract: { schema: { kind: "literal", value: "svml.broll-item-spec@1" } },- id: { schema: string },- z: { schema: signedInteger },- box: { schema: box },- fit: { schema: { kind: "string", enum: ["contain", "cover"] } },- backgroundColor: { schema: string, optional: true },- borderRadiusPx: { schema: number, optional: true },- enter: { schema: motion, optional: true },- exit: { schema: motion, optional: true },-});-const transition = object({- id: { schema: string },- fromItemId: { schema: string },- toItemId: { schema: string },- span: { schema: frameSpan },- operator: { schema: { kind: "string", enum: ["push", "page-turn"] } },- direction: { schema: { kind: "string", enum: ["left", "right", "up", "down"] } },- sfx: { schema: object({- artifact: { schema: audioBlobRef },- sampleFrames: { schema: unsignedInteger },- gain: { schema: number },- }), optional: true },-});--export const brollProgramSchema: ValueSchema = object({- contract: { schema: { kind: "literal", value: "svml.broll-program@1" } },- id: { schema: string },- items: { schema: { kind: "array", minItems: 1, items: item } },- transitions: { schema: { kind: "array", items: transition } },-});--export const brollProductSchema: ValueSchema = object({- contract: { schema: { kind: "literal", value: "svml.broll-product@1" } },- visualTrack: { schema: visualTrackSchema },- audioTrack: { schema: audioTrackSchema },-});--export const brollTrackSpecSchema: ValueSchema = object({- contract: { schema: { kind: "literal", value: "svml.broll-track-spec@1" } },- id: { schema: string },-});-export const brollItemSpecSchema: ValueSchema = itemSpec;-export const brollSetSchema: ValueSchema = object({- contract: { schema: { kind: "literal", value: "svml.broll-set@1" } },- items: { schema: { kind: "array", items: item } },-});--export const brollSurfaceImplementationDigest = digestOf("@narratage/broll/track-surface@1");--export const brollManifest: ModuleManifest = {- format: "svml.module@1",- name: brollModuleRef.name,- version: brollModuleRef.version,- dependencies: [- artifactDependency,- narrativeDependency,- mediaDependency,- semanticMapDependency,- programSpaceDependency,- compositionDependency,- { module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },- { module: svsModuleRef, digest: digestOf(svsManifest) },- ],- types: [- { name: brollTypes.program.name, schema: brollProgramSchema },- { name: brollTypes.product.name, schema: brollProductSchema },- { name: brollTypes.trackSpec.name, schema: brollTrackSpecSchema },- { name: brollTypes.itemSpec.name, schema: brollItemSpecSchema },- { name: brollTypes.set.name, schema: brollSetSchema },- ],- capabilities: [],- surfaces: [{- name: "track", tag: "Track", mode: "structured",- outputs: [brollTypes.trackSpec, brollTypes.itemSpec, mediaPipelineTypes.selectionRequest,- brollTypes.program, brollTypes.product, compositionTypes.visualTrack, compositionTypes.audioTrack],- implementation: { kind: "trusted-frontend-surface", locator: "@narratage/broll/track-surface", digest: brollSurfaceImplementationDigest },- }],- producers: [- {- name: brollProducers.createSet.name,- inputs: [],- outputs: [{ name: "set", type: brollTypes.set }], needs: [],- implementation: { kind: "registered", locator: "@narratage/broll/create-set", digest: createBrollSetImplementationDigest },- },- {- name: brollProducers.appendItem.name,- inputs: [- { name: "set", type: brollTypes.set },- { name: "track", type: brollTypes.trackSpec },- { name: "map", type: semanticMapTypes.complete },- { name: "space", type: programSpaceTypes.programSpace },- { name: "media", type: mediaTypes.synchronized },- { name: "selection", type: narrativeTypes.selection },- { name: "spec", type: brollTypes.itemSpec },- ],- outputs: [{ name: "set", type: brollTypes.set }], needs: [],- implementation: { kind: "registered", locator: "@narratage/broll/append-item", digest: appendBrollItemImplementationDigest },- },- {- name: brollProducers.finalize.name,- inputs: [- { name: "set", type: brollTypes.set },- { name: "track", type: brollTypes.trackSpec },- ],- outputs: [{ name: "program", type: brollTypes.program }], needs: [],- implementation: { kind: "registered", locator: "@narratage/broll/finalize-program", digest: finalizeBrollProgramImplementationDigest },- },- {- name: brollProducers.compile.name,- inputs: [- { name: "space", type: programSpaceTypes.programSpace },- { name: "program", type: brollTypes.program },- ],- outputs: [{ name: "product", type: brollTypes.product }],- needs: [],- implementation: {- kind: "registered",- locator: "@narratage/broll/compile",- digest: compileBrollImplementationDigest,- },- },- {- name: brollProducers.projectVisual.name,- inputs: [{ name: "product", type: brollTypes.product }],- outputs: [{ name: "visual", type: compositionTypes.visualTrack }],- needs: [],- implementation: {- kind: "registered",- locator: "@narratage/broll/project-visual",- digest: projectBrollVisualImplementationDigest,- },- },- {- name: brollProducers.projectAudio.name,- inputs: [{ name: "product", type: brollTypes.product }],- outputs: [{ name: "audio", type: compositionTypes.audioTrack }],- needs: [],- implementation: {- kind: "registered",- locator: "@narratage/broll/project-audio",- digest: projectBrollAudioImplementationDigest,- },- },- ],-};--export const brollManifestDigest = digestOf(brollManifest);diff --git a/packages/broll/src/program.ts b/packages/broll/src/program.tsdeleted file mode 100644index 24b4ddcf..00000000--- a/packages/broll/src/program.ts+++ /dev/null@@ -1,460 +0,0 @@-import type { MediaArtifactRef } from "@narratage/media";-import type { BlobRef } from "@narratage/protocol";-import {- assertProgramSpaceIdentity,- programFrameSampleBoundary,- programSpaceFrameCount,-} from "@narratage/program-space";-import type { ProgramSpace } from "@narratage/program-space";-import { assertAudioTrackIdentity, assertVisualTrackIdentity, sealAudioTrack, sealVisualTrack } from "@narratage/composition";-import type { AudioClip, AudioTrack, VisualAnimation, VisualElement, VisualStyleDeclaration } from "@narratage/composition";-import { digestOf, isDigest } from "@narratage/protocol";--import type {- BrollItem,- BrollMotion,- BrollPairTransition,- BrollProduct,- BrollProgram,-} from "./types.js";--export const compileBrollImplementationDigest = digestOf("@narratage/broll/compile@1");-export const projectBrollVisualImplementationDigest = digestOf("@narratage/broll/project-visual@1");-export const projectBrollAudioImplementationDigest = digestOf("@narratage/broll/project-audio@1");--type MotionState = {- readonly opacity: number;- readonly transform: string;-};--type StateMark = MotionState & {- readonly atFrame: number;- readonly easing?: "linear" | "ease-in" | "ease-out" | "ease-in-out";-};--const STABLE: MotionState = { opacity: 1, transform: "translate3d(0%,0%,0) scale(1) rotateX(0deg) rotateY(0deg)" };--function normalizeProgram(value: BrollProgram): BrollProgram {- return {- contract: "svml.broll-program@1",- id: value.id,- items: [...value.items].map((item) => structuredClone(item)).sort((left, right) => left.id.localeCompare(right.id)),- transitions: [...value.transitions].map((transition) => structuredClone(transition)).sort((left, right) => left.id.localeCompare(right.id)),- };-}--export function sealBrollProgram(value: BrollProgram): BrollProgram {- return normalizeProgram(value);-}--function assertArtifact(artifact: MediaArtifactRef, label: string, kinds: readonly string[]): void {- if (- !isDigest(artifact.digest)- || !Number.isSafeInteger(artifact.size)- || artifact.size < 0- || !Number.isFinite(artifact.durationSec)- || artifact.durationSec < 0- || !kinds.some((kind) => artifact.mediaType.startsWith(`${kind}/`))- ) {- throw new Error(`${label} Artifact is invalid.`);- }-}--function assertAudioArtifact(artifact: BlobRef, label: string): void {- if (- artifact.kind !== "blob"- || !isDigest(artifact.digest)- || !Number.isSafeInteger(artifact.size)- || artifact.size < 0- || artifact.mediaType !== "audio/wav"- ) {- throw new Error(`${label} must be a canonical WAV BlobRef.`);- }-}--function assertSpan(item: { readonly startFrame: number; readonly endFrameExclusive: number }, totalFrames: number, label: string): void {- if (- !Number.isSafeInteger(item.startFrame)- || !Number.isSafeInteger(item.endFrameExclusive)- || item.startFrame < 0- || item.endFrameExclusive <= item.startFrame- || item.endFrameExclusive > totalFrames- ) throw new Error(`${label} span is outside ProgramSpace.`);-}--function assertMotion(motion: BrollMotion | undefined, itemFrames: number, label: string): void {- if (motion === undefined) return;- if (!Number.isSafeInteger(motion.durationFrames) || motion.durationFrames <= 0 || motion.durationFrames > itemFrames) {- throw new Error(`${label} duration is invalid.`);- }- if (!["fade", "pop", "slide-up", "slide-down"].includes(motion.operator)) {- throw new Error(`${label} operator is unsupported.`);- }- if (motion.amount !== undefined && (!Number.isFinite(motion.amount) || motion.amount <= 0 || motion.amount > 4)) {- throw new Error(`${label} amount is invalid.`);- }-}--function assertItem(item: BrollItem, totalFrames: number): void {- if (!item.id || !item.tieBreak) throw new Error("B-roll item identity must not be empty.");- assertArtifact(item.artifact, `B-roll item ${item.id}`, ["image", "video"]);- assertSpan(item.span, totalFrames, `B-roll item ${item.id}`);- if (!Number.isSafeInteger(item.z)) throw new Error(`B-roll item ${item.id} z is invalid.`);- for (const [name, value] of Object.entries(item.box)) {- if (!Number.isFinite(value)) throw new Error(`B-roll item ${item.id} ${name} is invalid.`);- }- if (item.box.widthPercent <= 0 || item.box.heightPercent <= 0) {- throw new Error(`B-roll item ${item.id} box must have positive size.`);- }- const isImage = item.artifact.mediaType.startsWith("image/");- if (isImage && item.playback !== "freeze") throw new Error(`B-roll image ${item.id} must use freeze playback.`);- if (!isImage && item.playback === "freeze") throw new Error(`B-roll video ${item.id} cannot use freeze playback.`);- if (item.mediaStartSec !== undefined && (!Number.isFinite(item.mediaStartSec) || item.mediaStartSec < 0)) {- throw new Error(`B-roll item ${item.id} mediaStartSec is invalid.`);- }- if (item.backgroundColor !== undefined && !/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/iu.test(item.backgroundColor)) {- throw new Error(`B-roll item ${item.id} background color is invalid.`);- }- if (item.borderRadiusPx !== undefined && (!Number.isFinite(item.borderRadiusPx) || item.borderRadiusPx < 0)) {- throw new Error(`B-roll item ${item.id} border radius is invalid.`);- }- if (item.audio !== undefined) {- assertAudioArtifact(item.audio.artifact, `B-roll item ${item.id} source audio`);- if (isImage- || !Number.isSafeInteger(item.audio.sampleFrames) || item.audio.sampleFrames <= 0- || !Number.isFinite(item.audio.gain) || item.audio.gain < 0) {- throw new Error(`B-roll item ${item.id} source audio is invalid.`);- }- }- const duration = item.span.endFrameExclusive - item.span.startFrame;- assertMotion(item.enter, duration, `B-roll item ${item.id} enter motion`);- assertMotion(item.exit, duration, `B-roll item ${item.id} exit motion`);- if ((item.enter?.durationFrames ?? 0) + (item.exit?.durationFrames ?? 0) > duration) {- throw new Error(`B-roll item ${item.id} entrance and exit overlap.`);- }-}--function sameBox(left: BrollItem["box"], right: BrollItem["box"]): boolean {- return left.xPercent === right.xPercent- && left.yPercent === right.yPercent- && left.widthPercent === right.widthPercent- && left.heightPercent === right.heightPercent;-}--function transitionMaps(program: BrollProgram, totalFrames: number) {- const items = new Map(program.items.map((item) => [item.id, item]));- const incoming = new Map<string, BrollPairTransition>();- const outgoing = new Map<string, BrollPairTransition>();- const ids = new Set<string>();- for (const transition of program.transitions) {- if (!transition.id || ids.has(transition.id)) throw new Error(`B-roll transition id ${transition.id} is empty or duplicated.`);- ids.add(transition.id);- const from = items.get(transition.fromItemId);- const to = items.get(transition.toItemId);- if (!from || !to || from === to) throw new Error(`B-roll transition ${transition.id} has invalid endpoints.`);- assertSpan(transition.span, totalFrames, `B-roll transition ${transition.id}`);- if (- transition.span.endFrameExclusive !== from.span.endFrameExclusive- || transition.span.startFrame !== to.span.startFrame- || transition.span.startFrame < from.span.startFrame- || transition.span.endFrameExclusive > to.span.endFrameExclusive- ) {- throw new Error(`B-roll transition ${transition.id} must be the outgoing and incoming boundary of its two items.`);- }- if (!sameBox(from.box, to.box)) {- throw new Error(`B-roll transition ${transition.id} requires both items to share one presentation box.`);- }- if (from.exit || to.enter) throw new Error(`B-roll transition ${transition.id} conflicts with item edge motion.`);- if (outgoing.has(from.id) || incoming.has(to.id)) throw new Error(`B-roll transition ${transition.id} duplicates an item edge.`);- if (!["push", "page-turn"].includes(transition.operator)) throw new Error(`B-roll transition ${transition.id} operator is unsupported.`);- if (!["left", "right", "up", "down"].includes(transition.direction)) throw new Error(`B-roll transition ${transition.id} direction is unsupported.`);- if (transition.sfx) {- assertAudioArtifact(transition.sfx.artifact, `B-roll transition ${transition.id} SFX`);- if (!Number.isSafeInteger(transition.sfx.sampleFrames) || transition.sfx.sampleFrames <= 0- || !Number.isFinite(transition.sfx.gain) || transition.sfx.gain < 0) {- throw new Error(`B-roll transition ${transition.id} SFX gain is invalid.`);- }- }- outgoing.set(from.id, transition);- incoming.set(to.id, transition);- }- for (const item of program.items) {- const before = incoming.get(item.id);- const after = outgoing.get(item.id);- if (before && after && before.span.endFrameExclusive > after.span.startFrame) {- throw new Error(`B-roll item ${item.id} has overlapping pair transitions.`);- }- }- return { incoming, outgoing };-}--export function assertBrollProgramIdentity(program: BrollProgram, programSpace: ProgramSpace): void {- assertProgramSpaceIdentity(programSpace);- if (program.contract !== "svml.broll-program@1") throw new Error("Unsupported BrollProgram contract.");- if (!program.id) throw new Error("BrollProgram identity is invalid.");- if (program.items.length === 0) throw new Error("BrollProgram must contain at least one item.");- const totalFrames = programSpaceFrameCount(programSpace);- const ids = new Set<string>();- for (const item of program.items) {- if (ids.has(item.id)) throw new Error(`BrollProgram contains duplicate item ${item.id}.`);- ids.add(item.id);- assertItem(item, totalFrames);- }- transitionMaps(program, totalFrames);-}--function motionState(motion: BrollMotion, channel: "enter" | "exit"): MotionState {- if (motion.operator === "fade") return { opacity: 0, transform: STABLE.transform };- if (motion.operator === "pop") {- const scale = motion.amount ?? 0.72;- return { opacity: 0, transform: `translate3d(0%,0%,0) scale(${scale}) rotateX(0deg) rotateY(0deg)` };- }- const distance = (motion.amount ?? 1) * 100;- if (motion.operator === "slide-down") {- const y = channel === "enter" ? -distance : distance;- return { opacity: 1, transform: `translate3d(0%,${y}%,0) scale(1) rotateX(0deg) rotateY(0deg)` };- }- const y = channel === "enter" ? distance : -distance;- return { opacity: 1, transform: `translate3d(0%,${y}%,0) scale(1) rotateX(0deg) rotateY(0deg)` };-}--function pairState(- transition: BrollPairTransition,- endpoint: "incoming" | "outgoing",-): MotionState {- const incoming = endpoint === "incoming";- const sign = incoming ? -1 : 1;- if (transition.operator === "push") {- const vectors = {- left: [-100, 0],- right: [100, 0],- up: [0, -100],- down: [0, 100],- } as const;- const [x, y] = vectors[transition.direction];- return { opacity: 1, transform: `translate3d(${x * sign}%,${y * sign}%,0) scale(1) rotateX(0deg) rotateY(0deg)` };- }- const angle = (transition.direction === "left" || transition.direction === "up" ? -90 : 90) * sign;- const transform = transition.direction === "left" || transition.direction === "right"- ? `perspective(1200px) rotateY(${angle}deg)`- : `perspective(1200px) rotateX(${angle}deg)`;- return { opacity: incoming ? 0 : 0, transform };-}--function sameState(left: MotionState, right: MotionState): boolean {- return left.opacity === right.opacity && left.transform === right.transform;-}--function addMark(marks: Map<number, StateMark>, mark: StateMark, label: string): void {- const previous = marks.get(mark.atFrame);- if (previous && !sameState(previous, mark)) throw new Error(`${label} has conflicting animation states at frame ${mark.atFrame}.`);- marks.set(mark.atFrame, previous ? { ...previous, ...(mark.easing === undefined ? {} : { easing: mark.easing }) } : mark);-}--function itemAnimation(- item: BrollItem,- incoming: BrollPairTransition | undefined,- outgoing: BrollPairTransition | undefined,-): VisualAnimation | undefined {- const duration = item.span.endFrameExclusive - item.span.startFrame;- const marks = new Map<number, StateMark>();- addMark(marks, { ...STABLE, atFrame: 0 }, item.id);- addMark(marks, { ...STABLE, atFrame: duration }, item.id);-- if (item.enter) {- marks.set(0, { ...motionState(item.enter, "enter"), atFrame: 0, easing: "ease-out" });- addMark(marks, { ...STABLE, atFrame: item.enter.durationFrames }, item.id);- }- if (item.exit) {- const start = duration - item.exit.durationFrames;- addMark(marks, { ...STABLE, atFrame: start, easing: "ease-in" }, item.id);- marks.set(duration, { ...motionState(item.exit, "exit"), atFrame: duration });- }- if (incoming) {- const end = incoming.span.endFrameExclusive - item.span.startFrame;- marks.set(0, { ...pairState(incoming, "incoming"), atFrame: 0, easing: "ease-in-out" });- addMark(marks, { ...STABLE, atFrame: end }, item.id);- }- if (outgoing) {- const start = outgoing.span.startFrame - item.span.startFrame;- addMark(marks, { ...STABLE, atFrame: start, easing: "ease-in-out" }, item.id);- marks.set(duration, { ...pairState(outgoing, "outgoing"), atFrame: duration });- }-- const ordered = [...marks.values()].sort((left, right) => left.atFrame - right.atFrame);- if (ordered.every((mark) => sameState(mark, STABLE))) return undefined;- return {- keyframes: ordered.map((mark) => ({- atFrame: mark.atFrame,- ...(mark.easing === undefined ? {} : { easing: mark.easing }),- style: [- { name: "opacity", value: mark.opacity },- { name: "transform", value: mark.transform },- ],- })),- };-}--function itemElements(- item: BrollItem,- animation: VisualAnimation | undefined,- programSpace: ProgramSpace,-): readonly VisualElement[] {- const presentSeconds = (item.span.endFrameExclusive - item.span.startFrame)- * programSpace.frameRate.denominator / programSpace.frameRate.numerator;- const isImage = item.artifact.mediaType.startsWith("image/");- const playbackRate = item.playback === "stretch" && item.artifact.durationSec > 0- ? item.artifact.durationSec / presentSeconds- : undefined;- const rootStyle: VisualStyleDeclaration[] = [- { name: "height", value: `${item.box.heightPercent}%` },- { name: "left", value: `${item.box.xPercent}%` },- { name: "overflow", value: "hidden" },- { name: "position", value: "absolute" },- { name: "top", value: `${item.box.yPercent}%` },- { name: "transform-origin", value: "center center" },- { name: "width", value: `${item.box.widthPercent}%` },- ...(item.backgroundColor === undefined ? [] : [{ name: "background-color", value: item.backgroundColor }]),- ...(item.borderRadiusPx === undefined ? [] : [{ name: "border-radius", value: `${item.borderRadiusPx}px` }]),- ];- return [- {- id: "root",- order: 0,- kind: "box",- style: rootStyle,- ...(animation === undefined ? {} : { animation }),- },- {- id: "media",- parent: "root",- order: 1,- kind: isImage ? "image" : "video",- artifact: item.artifact,- style: [- { name: "height", value: "100%" },- { name: "object-fit", value: item.fit },- { name: "position", value: "absolute" },- { name: "width", value: "100%" },- ],- ...(item.mediaStartSec === undefined ? {} : { mediaStartSec: item.mediaStartSec }),- ...(playbackRate === undefined ? {} : { playbackRate }),- ...(!isImage && item.playback === "loop" ? { loop: true } : {}),- ...(!isImage ? { muted: true } : {}),- },- ];-}--function audioClips(program: BrollProgram, programSpace: ProgramSpace): AudioClip[] {- const clips: AudioClip[] = program.items.flatMap((item) => {- if (item.audio === undefined) return [];- const targetStart = programFrameSampleBoundary(programSpace, item.span.startFrame, 48_000);- const targetEnd = programFrameSampleBoundary(programSpace, item.span.endFrameExclusive, 48_000);- const targetLength = targetEnd - targetStart;- const playbackRate = item.playback === "stretch"- ? item.audio.sampleFrames / targetLength- : 1;- return [{- id: `source:${item.id}`,- artifact: item.audio.artifact,- target: { startSample: targetStart, endSampleExclusive: targetEnd },- source: {- sampleFrames: item.audio.sampleFrames,- startSample: 0,- endSampleExclusive: item.audio.sampleFrames,- loop: item.playback === "loop",- phaseSample: 0,- },- playbackRate,- pitch: "preserve" as const,- gain: item.audio.gain,- fadeInSamples: 0,- fadeOutSamples: 0,- }];- });- const totalFrames = programSpaceFrameCount(programSpace);- for (const transition of program.transitions) {- if (!transition.sfx) continue;- const startSample = programFrameSampleBoundary(programSpace, transition.span.startFrame, 48_000);- const programSamples = programFrameSampleBoundary(programSpace, totalFrames, 48_000);- const endSample = Math.min(programSamples, startSample + transition.sfx.sampleFrames);- clips.push({- id: `transition:${transition.id}`,- artifact: transition.sfx.artifact,- target: { startSample, endSampleExclusive: endSample },- source: {- sampleFrames: transition.sfx.sampleFrames,- startSample: 0,- endSampleExclusive: endSample - startSample,- loop: false,- phaseSample: 0,- },- playbackRate: 1,- pitch: "preserve",- gain: transition.sfx.gain,- fadeInSamples: 0,- fadeOutSamples: 0,- });- }- return clips;-}--function productContent(value: BrollProduct): BrollProduct {- return {- contract: "svml.broll-product@1",- visualTrack: structuredClone(value.visualTrack),- audioTrack: structuredClone(value.audioTrack),- };-}--export function compileBrollProduct(programSpace: ProgramSpace, program: BrollProgram): BrollProduct {- assertBrollProgramIdentity(program, programSpace);- const { incoming, outgoing } = transitionMaps(program, programSpaceFrameCount(programSpace));- const visualTrack = sealVisualTrack({- contract: "svml.visual-track@1",- visualIr: "svml.visual-ir@1",- id: `${program.id}:visual`,- presents: program.items.map((item) => ({- id: item.id,- span: { ...item.span },- stacking: { order: item.z, tieBreak: item.tieBreak },- elements: itemElements(item, itemAnimation(item, incoming.get(item.id), outgoing.get(item.id)), programSpace),- })),- });- const audioTrack = sealAudioTrack({- contract: "svml.audio-track@1",- id: `${program.id}:audio`,- clips: audioClips(program, programSpace),- });- const content = productContent({- contract: "svml.broll-product@1",- visualTrack,- audioTrack,- });- const product = content;- assertBrollProductIdentity(product, programSpace);- return product;-}--export function assertBrollProductIdentity(product: BrollProduct, programSpace: ProgramSpace): void {- assertProgramSpaceIdentity(programSpace);- if (product.contract !== "svml.broll-product@1") throw new Error("Unsupported BrollProduct contract.");- assertVisualTrackIdentity(product.visualTrack, programSpace);- assertAudioTrackIdentity(product.audioTrack, programSpace);-}--function assertStandaloneProduct(product: BrollProduct): void {- if (product.contract !== "svml.broll-product@1") throw new Error("Unsupported BrollProduct contract.");- assertVisualTrackIdentity(product.visualTrack);- assertAudioTrackIdentity(product.audioTrack);-}--export function projectBrollVisual(product: BrollProduct) {- assertStandaloneProduct(product);- return product.visualTrack;-}--export function projectBrollAudio(product: BrollProduct) {- assertStandaloneProduct(product);- return product.audioTrack;-}diff --git a/packages/broll/src/surface.ts b/packages/broll/src/surface.tsdeleted file mode 100644index f9007cac..00000000--- a/packages/broll/src/surface.ts+++ /dev/null@@ -1,171 +0,0 @@-import { narrativeTypes } from "@narratage/narrative";-import { mediaTypes } from "@narratage/media";-import { programSpaceTypes } from "@narratage/program-space";-import { semanticMapTypes } from "@narratage/semantic-map";-import { compositionTypes } from "@narratage/composition";-import { artifactTypes } from "@narratage/artifact";-import { sealGraphFragment } from "@narratage/elaborator";-import type { FragmentOperation } from "@narratage/elaborator";-import {- mediaPipelineTypes,- sealMediaSelectionRequest,- synchronizedMediaFragment,-} from "@narratage/media-pipeline";-import { svsRecipeType } from "@narratage/svs";-import type { SvsRecipe } from "@narratage/svs";-import type { StructuredElement, StructuredSurfaceHandler, SurfaceResolvedReference, TextAttributeValue } from "@narratage/text";--import { sealBrollItemSpec, sealBrollTrackSpec } from "./author.js";-import { brollProducers, brollTypes } from "./manifest.js";-import type { BrollMotion, BrollSurfaceItemInput } from "./types.js";--const input = (name: string) => ({ kind: "fragment-input" as const, name });-const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });--function createBrollSurfaceFragment(items: readonly BrollSurfaceItemInput[], name: string) {- const operations: FragmentOperation[] = [- { id: "broll:set:empty", producer: brollProducers.createSet, inputs: {}, result: { kind: "output", name: "set" } },- ];- let current = "broll:set:empty";- items.forEach((item, index) => {- const id = `broll:set:append:${String(index + 1).padStart(4, "0")}`;- operations.push({- id, producer: brollProducers.appendItem,- inputs: { set: operation(current), track: input("track-spec"), map: input("map"), space: input("space"), media: input(item.mediaName), selection: input(item.selectionName), spec: input(item.specName) },- result: { kind: "output", name: "set" },- });- current = id;- });- operations.push(- { id: "broll:program", producer: brollProducers.finalize, inputs: { set: operation(current), track: input("track-spec") }, result: { kind: "output", name: "program" } },- { id: "broll:product", producer: brollProducers.compile, inputs: { space: input("space"), program: operation("broll:program") }, result: { kind: "output", name: "product" } },- { id: "broll:visual", producer: brollProducers.projectVisual, inputs: { product: operation("broll:product") }, result: { kind: "output", name: "visual" } },- { id: "broll:audio", producer: brollProducers.projectAudio, inputs: { product: operation("broll:product") }, result: { kind: "output", name: "audio" } },- );- const semanticInputs = ["map", "space", "track-spec", ...items.flatMap((item) => [item.mediaName, item.selectionName, item.specName])];- return sealGraphFragment({- name,- inputs: [- { name: "map", type: semanticMapTypes.complete },- { name: "space", type: programSpaceTypes.programSpace },- { name: "track-spec", type: brollTypes.trackSpec },- ...items.flatMap((item) => [- { name: item.mediaName, type: mediaTypes.synchronized },- { name: item.selectionName, type: narrativeTypes.selection },- { name: item.specName, type: brollTypes.itemSpec },- ]),- ],- operations,- exports: [- { name: "visual", type: compositionTypes.visualTrack, root: operation("broll:visual"), semanticInputs, fidelity: "exact" },- { name: "audio", type: compositionTypes.audioTrack, root: operation("broll:audio"), semanticInputs, fidelity: "exact" },- ],- });-}--function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {- return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;-}-function exact(element: StructuredElement, names: readonly string[]): void {- if (Object.keys(element.attributes).sort().join("\0") !== [...names].sort().join("\0")) throw new Error(`${element.name} requires exactly ${names.join(", ")}`);-}-function text(element: StructuredElement, name: string): string {- const value = element.attributes[name];- if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);- return value.trim();-}-function ref(element: StructuredElement, name: string, expected: SurfaceResolvedReference["type"], resolve: (path: string) => SurfaceResolvedReference | undefined) {- const raw: TextAttributeValue | undefined = element.attributes[name];- if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a reference`);- const value = resolve(raw.path);- if (!value || !sameType(value.type, expected)) throw new Error(`${element.name}.${name} cannot resolve a value of the required type`);- return value;-}-function recipe(value: SurfaceResolvedReference): SvsRecipe {- if (!sameType(value.type, svsRecipeType) || value.record?.value.kind !== "inline") throw new Error("B-roll appearance must be an authored SVS Recipe");- return value.record.value.value as unknown as SvsRecipe;-}-function num(value: SvsRecipe, name: string): number {- const result = value.properties[name];- if (typeof result !== "number" || !Number.isFinite(result)) throw new Error(`B-roll Recipe ${name} must be a number`);- return result;-}-function str(value: SvsRecipe, name: string): string {- const result = value.properties[name];- if (typeof result !== "string" || !result) throw new Error(`B-roll Recipe ${name} must be a string`);- return result;-}-function motion(value: string): BrollMotion {- const match = /^(fade|pop|slide-up|slide-down)\s+([1-9][0-9]*)f$/u.exec(value.trim());- if (!match) throw new Error(`B-roll motion ${value} is invalid`);- return { operator: match[1] as BrollMotion["operator"], durationFrames: Number(match[2]) };-}--export const decodeBrollTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- exact(element, ["id", "map", "space"]);- const id = text(element, "id");- const map = ref(element, "map", semanticMapTypes.complete, resolveReference);- const space = ref(element, "space", programSpaceTypes.programSpace, resolveReference);- const trackSpecId = `${id}.spec`;- const requestId = `${id}.selection`;- const records: Array<{ id: string; type: typeof brollTypes.itemSpec | typeof brollTypes.trackSpec | typeof mediaPipelineTypes.selectionRequest; value: { kind: "inline"; value: ReturnType<typeof sealBrollItemSpec> | ReturnType<typeof sealBrollTrackSpec> | ReturnType<typeof sealMediaSelectionRequest> }; range: typeof element.range }> = [- { id: trackSpecId, type: brollTypes.trackSpec, value: { kind: "inline", value: sealBrollTrackSpec({ contract: "svml.broll-track-spec@1", id }) }, range: element.range },- { id: requestId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline", value: sealMediaSelectionRequest({- contract: "svml.media-selection-request@1", video: { mode: "primary-moving" }, audio: { mode: "none" },- spanAuthority: "video", frameRate: { numerator: 30, denominator: 1 },- }) }, range: element.range },- ];- let itemIndex = 0;- const declared = element.children.flatMap((child) => {- if (child.kind === "text") {- if (child.value.trim()) throw new Error(`${element.name} accepts only Item children`);- return [];- }- if (!child.name.endsWith(":Item") && child.name !== "Item") throw new Error(`${element.name} accepts only Item children`);- itemIndex += 1;- exact(child, ["source", "during", "appearance"]);- const source = ref(child, "source", artifactTypes.blob, resolveReference);- const selection = ref(child, "during", narrativeTypes.selection, resolveReference);- const appearance = recipe(ref(child, "appearance", svsRecipeType, resolveReference));- const expected = ["background", "enter", "exit", "fit", "height", "radius", "stack-order", "width", "x", "y"];- if (Object.keys(appearance.properties).sort().join("\0") !== expected.sort().join("\0")) throw new Error(`B-roll Recipe requires exactly ${expected.join(", ")}`);- const fit = str(appearance, "fit");- if (fit !== "contain" && fit !== "cover") throw new Error("B-roll Recipe fit is invalid");- const suffix = String(itemIndex).padStart(4, "0");- const specId = `${id}.item.${suffix}.spec`;- records.push({- id: specId, type: brollTypes.itemSpec,- value: { kind: "inline", value: sealBrollItemSpec({- contract: "svml.broll-item-spec@1", id: `${id}.item.${suffix}`, z: num(appearance, "stack-order"),- box: { xPercent: num(appearance, "x") * 100, yPercent: num(appearance, "y") * 100,- widthPercent: num(appearance, "width") * 100, heightPercent: num(appearance, "height") * 100 },- fit, backgroundColor: str(appearance, "background"), borderRadiusPx: num(appearance, "radius"),- enter: motion(str(appearance, "enter")), exit: motion(str(appearance, "exit")),- }) }, range: child.range,- });- return [{ suffix, source, selection, specId, range: child.range,- input: { mediaName: `item-${suffix}-media`, selectionName: `item-${suffix}-selection`, specName: `item-${suffix}-spec` } }];- });- if (declared.length === 0) throw new Error(`${element.name} requires at least one Item`);- const fragment = createBrollSurfaceFragment(declared.map((item) => item.input), `@narratage/broll/surface/${id}@1`);- const normalization = declared.map((item) => ({- id: `${id}.normalize.${item.suffix}`, fragment: synchronizedMediaFragment.id,- inputs: { source: item.source.ref, request: { kind: "record" as const, id: requestId } },- outputs: { media: `${id}.normalized.${item.suffix}` }, range: item.range,- }));- return {- records,- components: [- ...normalization,- { id, fragment: fragment.id, inputs: {- map: map.ref, space: space.ref, "track-spec": { kind: "record", id: trackSpecId },- ...Object.fromEntries(declared.flatMap((item, index) => [- [item.input.mediaName, { kind: "component-output" as const, component: normalization[index]!.id, output: "media" }],- [item.input.selectionName, item.selection.ref],- [item.input.specName, { kind: "record" as const, id: item.specId }],- ])),- }, outputs: { visual: `${id}.visual`, audio: `${id}.audio` }, range: element.range },- ],- fragments: [synchronizedMediaFragment, fragment],- };-};diff --git a/packages/broll/src/types.ts b/packages/broll/src/types.tsdeleted file mode 100644index 650c620f..00000000--- a/packages/broll/src/types.ts+++ /dev/null@@ -1,97 +0,0 @@-import type { MediaArtifactRef } from "@narratage/media";-import type { BlobRef } from "@narratage/protocol";-import type { AudioTrack, FrameSpan, VisualTrack } from "@narratage/composition";--export type BrollBox = {- readonly xPercent: number;- readonly yPercent: number;- readonly widthPercent: number;- readonly heightPercent: number;-};--export type BrollMotionOperator = "fade" | "pop" | "slide-up" | "slide-down";--export type BrollMotion = {- readonly operator: BrollMotionOperator;- readonly durationFrames: number;- /** Slide distance or pop start/end scale. Package-defined and deterministic. */- readonly amount?: number;-};--export type BrollItem = {- readonly id: string;- readonly artifact: MediaArtifactRef;- readonly span: FrameSpan;- readonly z: number;- readonly tieBreak: string;- readonly box: BrollBox;- readonly fit: "contain" | "cover";- readonly playback: "freeze" | "native" | "loop" | "stretch";- readonly mediaStartSec?: number;- readonly audio?: {- readonly artifact: BlobRef;- readonly sampleFrames: number;- readonly gain: number;- };- readonly enter?: BrollMotion;- readonly exit?: BrollMotion;- readonly backgroundColor?: string;- readonly borderRadiusPx?: number;-};--export type BrollPairTransition = {- readonly id: string;- readonly fromItemId: string;- readonly toItemId: string;- readonly span: FrameSpan;- readonly operator: "push" | "page-turn";- readonly direction: "left" | "right" | "up" | "down";- readonly sfx?: {- readonly artifact: BlobRef;- readonly sampleFrames: number;- readonly gain: number;- };-};--/** Official B-roll authoring/lowering value. It is not a Core contract. */-export type BrollProgram = {- readonly contract: "svml.broll-program@1";- readonly id: string;- readonly items: readonly BrollItem[];- readonly transitions: readonly BrollPairTransition[];-};--/** One atomic compilation result followed by ordinary deterministic projections. */-export type BrollProduct = {- readonly contract: "svml.broll-product@1";- readonly visualTrack: VisualTrack;- readonly audioTrack: AudioTrack;-};--export type BrollItemSpec = {- readonly contract: "svml.broll-item-spec@1";- readonly id: string;- readonly z: number;- readonly box: BrollBox;- readonly fit: "contain" | "cover";- readonly backgroundColor?: string;- readonly borderRadiusPx?: number;- readonly enter?: BrollMotion;- readonly exit?: BrollMotion;-};--export type BrollTrackSpec = {- readonly contract: "svml.broll-track-spec@1";- readonly id: string;-};--export type BrollSet = {- readonly contract: "svml.broll-set@1";- readonly items: readonly BrollItem[];-};--export type BrollSurfaceItemInput = {- readonly mediaName: string;- readonly selectionName: string;- readonly specName: string;-};diff --git a/packages/broll/test/broll.test.ts b/packages/broll/test/broll.test.tsdeleted file mode 100644index 72435170..00000000--- a/packages/broll/test/broll.test.ts+++ /dev/null@@ -1,193 +0,0 @@-import {- assertBrollProductIdentity,- assertBrollProgramIdentity,- compileBrollProduct,- projectBrollAudio,- projectBrollVisual,- sealBrollProgram,-} from "@narratage/broll";-import type { MediaArtifactRef } from "@narratage/media";-import { sealProgramSpace } from "@narratage/program-space";-import type { ProgramSpace } from "@narratage/program-space";-import { sealComposition, sealVisualTrack } from "@narratage/composition";-import type { AudioTrack, Track } from "@narratage/composition";-import assert from "node:assert/strict";-import test from "node:test";--import type { BrollItem, BrollPairTransition } from "@narratage/broll";-import { compileHyperframesDocument } from "@narratage/hyperframes";-import { digestOf } from "@narratage/protocol";-import type { BlobRef } from "@narratage/protocol";--const programSpace = sealProgramSpace({- contract: "svml.program-space@1",- durationSec: 10,- frameRate: { numerator: 30, denominator: 1 },-});--function artifact(id: string, mediaType = "video/mp4", durationSec = 4): MediaArtifactRef {- return { digest: digestOf(`broll:${id}`), size: 100, mediaType, durationSec };-}--function audioArtifact(id: string): BlobRef {- return { kind: "blob", digest: digestOf(`broll:${id}`), size: 100, mediaType: "audio/wav" };-}--function item(overrides: Partial<BrollItem> & Pick<BrollItem, "id" | "span" | "z">): BrollItem {- return {- artifact: artifact(overrides.id),- tieBreak: overrides.id,- box: { xPercent: 0, yPercent: 0, widthPercent: 100, heightPercent: 100 },- fit: "cover",- playback: "native",- ...overrides,- };-}--function program(items: readonly BrollItem[], transitions: readonly BrollPairTransition[] = []) {- return sealBrollProgram({- contract: "svml.broll-program@1",- id: "story-broll",- items,- transitions,- });-}--test("B-roll owns local motion while every item remains an independently stacked Present", () => {- const authored = program([- item({- id: "board",- artifact: artifact("board", "image/png", 0),- span: { startFrame: 0, endFrameExclusive: 300 },- z: 30,- playback: "freeze",- enter: { operator: "pop", durationFrames: 15 },- exit: { operator: "fade", durationFrames: 15 },- }),- item({- id: "cutaway",- span: { startFrame: 30, endFrameExclusive: 150 },- z: 80,- enter: { operator: "slide-down", durationFrames: 12 },- exit: { operator: "slide-up", durationFrames: 12 },- audio: { artifact: audioArtifact("cutaway-audio"), sampleFrames: 192_000, gain: 0.7 },- }),- ]);- assert.doesNotThrow(() => assertBrollProgramIdentity(authored, programSpace));- const product = compileBrollProduct(programSpace, authored);- assert.doesNotThrow(() => assertBrollProductIdentity(product, programSpace));- const board = product.visualTrack.presents.find((present) => present.id === "board")!;- const cutaway = product.visualTrack.presents.find((present) => present.id === "cutaway")!;- assert.equal(board.stacking.order, 30);- assert.equal(cutaway.stacking.order, 80);- assert.deepEqual(board.elements[0]!.animation?.keyframes.map((keyframe) => keyframe.atFrame), [0, 15, 285, 300]);- assert.deepEqual(cutaway.elements[0]!.animation?.keyframes.map((keyframe) => keyframe.atFrame), [0, 12, 108, 120]);- assert.deepEqual(product.audioTrack.clips.map((clip) => clip.id), ["source:cutaway"]);-- const middle = sealVisualTrack({- contract: "svml.visual-track@1",- visualIr: "svml.visual-ir@1",- id: "middle-overlay",- presents: [{- id: "middle",- span: { startFrame: 0, endFrameExclusive: 300 },- stacking: { order: 50, tieBreak: "middle" },- elements: [{ id: "root", order: 0, kind: "box", style: [] }],- }],- });- const document = compileHyperframesDocument(sealComposition({- contract: "svml.composition@1",- id: "interleaved-broll",- canvas: { width: 1080, height: 1920, clearColor: "#000000" },- tracks: [projectBrollVisual(product), middle, projectBrollAudio(product)],- }), programSpace);- const boardAt = document.html.indexOf('data-svml-present-id="board"');- const middleAt = document.html.indexOf('data-svml-present-id="middle"');- const cutawayAt = document.html.indexOf('data-svml-present-id="cutaway"');- assert.ok(boardAt < middleAt && middleAt < cutawayAt, "Presents from one B-roll Track must interleave with peers by absolute z");-});--test("push is lowered into complementary animations inside one B-roll-owned handoff", () => {- const authored = program([- item({ id: "a", span: { startFrame: 0, endFrameExclusive: 120 }, z: 70 }),- item({ id: "b", span: { startFrame: 105, endFrameExclusive: 240 }, z: 71 }),- ], [{- id: "a-to-b",- fromItemId: "a",- toItemId: "b",- span: { startFrame: 105, endFrameExclusive: 120 },- operator: "push",- direction: "left",- }]);- const product = compileBrollProduct(programSpace, authored);- const from = product.visualTrack.presents.find((present) => present.id === "a")!;- const to = product.visualTrack.presents.find((present) => present.id === "b")!;- assert.deepEqual(from.elements[0]!.animation?.keyframes.map((keyframe) => keyframe.atFrame), [0, 105, 120]);- assert.deepEqual(to.elements[0]!.animation?.keyframes.map((keyframe) => keyframe.atFrame), [0, 15, 135]);- assert.match(String(from.elements[0]!.animation?.keyframes.at(-1)?.style[1]?.value), /translate3d\(-100%/u);- assert.match(String(to.elements[0]!.animation?.keyframes[0]?.style[1]?.value), /translate3d\(100%/u);-});--test("page-turn and transition SFX remain B-roll-owned and lower to generic visual/audio facts", () => {- const sfx = audioArtifact("page-sfx");- const authored = program([- item({ id: "page-a", span: { startFrame: 0, endFrameExclusive: 90 }, z: 40 }),- item({ id: "page-b", span: { startFrame: 75, endFrameExclusive: 180 }, z: 41 }),- ], [{- id: "turn",- fromItemId: "page-a",- toItemId: "page-b",- span: { startFrame: 75, endFrameExclusive: 90 },- operator: "page-turn",- direction: "left",- sfx: { artifact: sfx, sampleFrames: 9_600, gain: 0.8 },- }]);- const product = compileBrollProduct(programSpace, authored);- assert.equal(product.audioTrack.clips[0]?.id, "transition:turn");- assert.deepEqual(product.audioTrack.clips[0]?.target, {- startSample: 120_000,- endSampleExclusive: 129_600,- });- assert.equal(product.audioTrack.clips[0]?.artifact.digest, sfx.digest);- const document = compileHyperframesDocument(sealComposition({- contract: "svml.composition@1",- id: "page-turn",- canvas: { width: 1080, height: 1920, clearColor: "#000000" },- tracks: [product.visualTrack, product.audioTrack],- }), programSpace);- assert.match(document.html, /perspective\(1200px\) rotateY/u);- assert.doesNotMatch(document.html, new RegExp(sfx.digest.slice("sha256:".length), "u"),- "HyperFrames is visual-only; the media pipeline renders this proven AudioTrack separately");-});--test("B-roll rejects a pair transition that is not both items' exact shared boundary", () => {- const authored = program([- item({ id: "a", span: { startFrame: 0, endFrameExclusive: 120 }, z: 70 }),- item({ id: "b", span: { startFrame: 105, endFrameExclusive: 240 }, z: 71 }),- ], [{- id: "invalid",- fromItemId: "a",- toItemId: "b",- span: { startFrame: 106, endFrameExclusive: 119 },- operator: "push",- direction: "down",- }]);- assert.throws(() => compileBrollProduct(programSpace, authored), /must be the outgoing and incoming boundary/);-});--test("a BrollProduct is validated against the explicitly connected ProgramSpace", () => {- const product = compileBrollProduct(programSpace, program([- item({ id: "bounded", span: { startFrame: 0, endFrameExclusive: 120 }, z: 20 }),- ]));- const present = product.visualTrack.presents[0]!;- const invalidVisual = sealVisualTrack({- ...product.visualTrack,- presents: [{ ...present, span: { startFrame: 0, endFrameExclusive: 301 } }],- });- const tampered = {- contract: product.contract,- visualTrack: invalidVisual,- audioTrack: product.audioTrack,- };- assert.throws(() => assertBrollProductIdentity(tampered, programSpace), /outside ProgramSpace/);-});diff --git a/packages/caption/test/caption.test.ts b/packages/caption/test/caption.test.tsindex eae0c60b..7fa183c6 100644--- a/packages/caption/test/caption.test.ts+++ b/packages/caption/test/caption.test.ts@@ -47,9 +47,10 @@ function locate(narrative: Narrative, durationSec: number, segments: readonly Alaudio,visualTrack: { clips: basisSegments.map((segment) => ({segmentId: segment.segmentId,- artifact: { digest: digestOf(segment.segmentId), size: 1, mediaType: "video/mp4", durationSec: segment.endSec - segment.startSec },- startSec: segment.startSec,- endSec: segment.endSec,+ artifact: { kind: "blob", digest: digestOf(segment.segmentId), size: 1, mediaType: "video/mp4" },+ extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },+ frameRate: { ...space.frameRate },+ frameCount: Math.round((segment.endSec - segment.startSec) * 30),})) },segments: basisSegments,});diff --git a/packages/composition/src/schema.ts b/packages/composition/src/schema.tsindex f7455e25..c629494a 100644--- a/packages/composition/src/schema.ts+++ b/packages/composition/src/schema.ts@@ -12,6 +12,23 @@ const audioBlobRef = object({size: { schema: integer },mediaType: { schema: { kind: "literal", value: "audio/wav" } },});+const mediaBlobRef = object({+ kind: { schema: { kind: "literal", value: "blob" } },+ digest: { schema: { kind: "string", minLength: 71, maxLength: 71 } },+ size: { schema: integer },+ mediaType: { schema: string },+});+const positiveInteger = { kind: "number", integer: true, minimum: 1 } as const;+const rational = object({ numerator: { schema: integer }, denominator: { schema: positiveInteger } });+const samplingSegment = object({+ target: { schema: object({ startFrame: { schema: integer }, endFrameExclusive: { schema: positiveInteger } }) },+ sourceFrame: { schema: rational }, rate: { schema: rational },+ loop: { schema: object({ startFrame: { schema: integer }, endFrameExclusive: { schema: positiveInteger } }), optional: true },+});+const sampling = object({+ sourceFrameRate: { schema: rational }, sourceFrameCount: { schema: positiveInteger },+ segments: { schema: { kind: "array", minItems: 1, items: samplingSegment } },+});const styleDeclaration: ValueSchema = { kind: "oneOf", variants: VISUAL_STYLE_NAMES_V1.map((name) => object({name: { schema: { kind: "literal", value: name } }, value: { schema: Object.hasOwn(VISUAL_STYLE_ENUM_VALUES_V1, name)? { kind: "string", enum: VISUAL_STYLE_ENUM_VALUES_V1[name as keyof typeof VISUAL_STYLE_ENUM_VALUES_V1] }@@ -23,8 +40,8 @@ const animation = object({ keyframes: { schema: { kind: "array", minItems: 2, itconst base = { id: { schema: string }, parent: { schema: string, optional: true }, order: { schema: integer }, style: { schema: { kind: "array", items: styleDeclaration } }, attributes: { schema: { kind: "array", items: attribute }, optional: true }, animation: { schema: animation, optional: true } } as const;const box = object({ ...base, kind: { schema: { kind: "literal", value: "box" } } });const text = object({ ...base, kind: { schema: { kind: "literal", value: "text" } }, text: { schema: { kind: "string" } }, fonts: { schema: { kind: "array", minItems: 1, items: fontArtifactSchema }, optional: true } });-const media = (kind: "image" | "video") => object({ ...base, kind: { schema: { kind: "literal", value: kind } }, artifact: { schema: mediaArtifactSchema }, mediaStartSec: { schema: number, optional: true }, playbackRate: { schema: number, optional: true }, loop: { schema: { kind: "boolean" }, optional: true }, muted: { schema: { kind: "boolean" }, optional: true } });-const surface = object({ ...base, kind: { schema: { kind: "literal", value: "surface" } }, surface: { schema: compositableSurfaceSchema } });+const media = (kind: "image" | "video") => object({ ...base, kind: { schema: { kind: "literal", value: kind } }, artifact: { schema: mediaBlobRef }, sampling: { schema: sampling, optional: true }, mediaStartSec: { schema: number, optional: true }, playbackRate: { schema: number, optional: true }, loop: { schema: { kind: "boolean" }, optional: true }, muted: { schema: { kind: "boolean" }, optional: true } });+const surface = object({ ...base, kind: { schema: { kind: "literal", value: "surface" } }, surface: { schema: compositableSurfaceSchema }, sampling: { schema: sampling, optional: true } });const element: ValueSchema = { kind: "oneOf", variants: [box, text, media("image"), media("video"), surface] };const span = object({ startFrame: { schema: integer }, endFrameExclusive: { schema: integer } });const present = object({ id: { schema: string }, span: { schema: span }, stacking: { schema: object({ order: { schema: signedInteger }, tieBreak: { schema: string } }) }, elements: { schema: { kind: "array", minItems: 1, items: element } } });diff --git a/packages/composition/src/track.ts b/packages/composition/src/track.tsindex a843479f..5b9b98b5 100644--- a/packages/composition/src/track.ts+++ b/packages/composition/src/track.ts@@ -12,7 +12,7 @@ import {} from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";import { assertCompositableSurfaceRef, assertFontArtifactRef } from "@narratage/media";-import type { CompositableSurfaceRef, FontArtifactRef, MediaArtifactRef } from "@narratage/media";+import type { CompositableSurfaceRef, FontArtifactRef } from "@narratage/media";export type FrameSpan = {readonly startFrame: number;@@ -67,9 +67,36 @@ export type VisualTextElement = VisualElementBase & {readonly fonts?: readonly FontArtifactRef[];};+export type VisualSamplingRational = {+ readonly numerator: number;+ readonly denominator: number;+};++/**+ * One exact piece of a timed visual's source-time function. Target frames are+ * relative to the containing Present. A gap means the material is invisible.+ */+export type VisualSamplingSegment = {+ readonly target: FrameSpan;+ /** Absolute source-frame position at target.startFrame. */+ readonly sourceFrame: VisualSamplingRational;+ /** Source frames advanced per Program frame; zero is an exact held frame. */+ readonly rate: VisualSamplingRational;+ readonly loop?: FrameSpan;+};++export type VisualTimedSampling = {+ readonly sourceFrameRate: VisualSamplingRational;+ readonly sourceFrameCount: number;+ readonly segments: readonly VisualSamplingSegment[];+};+export type VisualMediaElement = VisualElementBase & {readonly kind: "image" | "video";- readonly artifact: MediaArtifactRef;+ readonly artifact: BlobRef;+ /** Exact frame-domain mapping superseding the candidate-era scalar playback fields. */+ readonly sampling?: VisualTimedSampling;+ /** Candidate-era compatibility path; new timed lowerers use sampling. */readonly mediaStartSec?: number;readonly playbackRate?: number;readonly loop?: boolean;@@ -80,6 +107,7 @@ export type VisualMediaElement = VisualElementBase & {export type VisualSurfaceElement = VisualElementBase & {readonly kind: "surface";readonly surface: CompositableSurfaceRef;+ readonly sampling?: VisualTimedSampling;};/**@@ -178,19 +206,74 @@ function assertFrameSpan(span: FrameSpan, totalFrames: number, label: string): v}}-function assertMediaArtifact(artifact: MediaArtifactRef, label: string): void {+function assertMediaArtifact(artifact: BlobRef, label: string): void {if (- !isDigest(artifact.digest)+ artifact.kind !== "blob"+ || !isDigest(artifact.digest)|| !Number.isSafeInteger(artifact.size)|| artifact.size < 0|| !artifact.mediaType- || !Number.isFinite(artifact.durationSec)- || artifact.durationSec < 0) {throw new Error(`${label} is invalid.`);}}+function assertRational(value: VisualSamplingRational, label: string, allowZero: boolean): void {+ if (!Number.isSafeInteger(value.numerator) || (!allowZero && value.numerator <= 0) || (allowZero && value.numerator < 0)+ || !Number.isSafeInteger(value.denominator) || value.denominator <= 0) {+ throw new Error(`${label} must be a non-negative safe rational.`);+ }+}++function compareRationalToInteger(value: VisualSamplingRational, integer: number): number {+ const difference = BigInt(value.numerator) - BigInt(integer) * BigInt(value.denominator);+ return difference < 0n ? -1 : difference > 0n ? 1 : 0;+}++function sourceAt(segment: VisualSamplingSegment, targetOffset: number): VisualSamplingRational {+ const denominator = BigInt(segment.sourceFrame.denominator) * BigInt(segment.rate.denominator);+ const numerator = BigInt(segment.sourceFrame.numerator) * BigInt(segment.rate.denominator)+ + BigInt(targetOffset) * BigInt(segment.rate.numerator) * BigInt(segment.sourceFrame.denominator);+ if (numerator > BigInt(Number.MAX_SAFE_INTEGER) || denominator > BigInt(Number.MAX_SAFE_INTEGER)) {+ throw new Error("Visual sampling arithmetic exceeds the safe wire domain.");+ }+ return { numerator: Number(numerator), denominator: Number(denominator) };+}++function assertSampling(value: VisualTimedSampling, durationFrames: number, label: string): void {+ assertRational(value.sourceFrameRate, `${label}.sourceFrameRate`, false);+ if (!Number.isSafeInteger(value.sourceFrameCount) || value.sourceFrameCount <= 0 || value.segments.length === 0) {+ throw new Error(`${label} source frame domain is invalid.`);+ }+ let previousEnd = 0;+ for (const [index, segment] of value.segments.entries()) {+ const item = `${label}.segments.${index}`;+ if (!Number.isSafeInteger(segment.target.startFrame) || !Number.isSafeInteger(segment.target.endFrameExclusive)+ || segment.target.startFrame < previousEnd || segment.target.endFrameExclusive <= segment.target.startFrame+ || segment.target.endFrameExclusive > durationFrames) throw new Error(`${item} target interval is invalid.`);+ assertRational(segment.sourceFrame, `${item}.sourceFrame`, true);+ assertRational(segment.rate, `${item}.rate`, true);+ if (segment.loop !== undefined) {+ if (!Number.isSafeInteger(segment.loop.startFrame) || !Number.isSafeInteger(segment.loop.endFrameExclusive)+ || segment.loop.startFrame < 0 || segment.loop.endFrameExclusive <= segment.loop.startFrame+ || segment.loop.endFrameExclusive > value.sourceFrameCount+ || compareRationalToInteger(segment.sourceFrame, segment.loop.startFrame) < 0+ || compareRationalToInteger(segment.sourceFrame, segment.loop.endFrameExclusive) >= 0) {+ throw new Error(`${item} loop interval or phase is invalid.`);+ }+ } else {+ const last = sourceAt(segment, segment.target.endFrameExclusive - segment.target.startFrame - 1);+ if (compareRationalToInteger(segment.sourceFrame, 0) < 0+ || compareRationalToInteger(segment.sourceFrame, value.sourceFrameCount) >= 0+ || compareRationalToInteger(last, 0) < 0+ || compareRationalToInteger(last, value.sourceFrameCount) >= 0) {+ throw new Error(`${item} samples outside its source frame domain.`);+ }+ }+ previousEnd = segment.target.endFrameExclusive;+ }+}+function assertAudioArtifact(artifact: BlobRef, label: string): void {if (artifact.kind !== "blob"@@ -299,6 +382,19 @@ function assertPresent(present: VisualPresent, programSpace: ProgramSpace | undeif (element.playbackRate !== undefined && (!Number.isFinite(element.playbackRate) || element.playbackRate <= 0)) {throw new Error(`${trackId}.${present.id}.${element.id} has invalid playbackRate.`);}+ if (element.kind === "image" && element.sampling !== undefined) {+ throw new Error(`${trackId}.${present.id}.${element.id} cannot sample a durationless image.`);+ }+ if (element.sampling !== undefined) {+ if (element.mediaStartSec !== undefined || element.playbackRate !== undefined || element.loop !== undefined) {+ throw new Error(`${trackId}.${present.id}.${element.id} mixes exact sampling with scalar playback fields.`);+ }+ if (element.animation !== undefined) {+ throw new Error(`${trackId}.${present.id}.${element.id} sampling motion must live on an owned wrapper.`);+ }+ assertSampling(element.sampling, present.span.endFrameExclusive - present.span.startFrame,+ `${trackId}.${present.id}.${element.id}.sampling`);+ }}if (element.kind === "text" && element.fonts !== undefined) {if (element.fonts.length === 0) throw new Error(`${trackId}.${present.id}.${element.id} has an empty font stack.`);@@ -316,12 +412,28 @@ function assertPresent(present: VisualPresent, programSpace: ProgramSpace | unde}if (element.kind === "surface") {assertCompositableSurfaceRef(element.surface, `${trackId}.${present.id}.${element.id}.surface`);+ if (element.surface.timing.kind === "still" && element.sampling !== undefined) {+ throw new Error(`${trackId}.${present.id}.${element.id} cannot sample a still Surface.`);+ }+ if (element.sampling !== undefined) {+ if (element.animation !== undefined) {+ throw new Error(`${trackId}.${present.id}.${element.id} sampling motion must live on an owned wrapper.`);+ }+ if (element.surface.timing.kind !== "frames"+ || element.sampling.sourceFrameCount !== element.surface.timing.frameCount+ || element.sampling.sourceFrameRate.numerator !== element.surface.timing.frameRate.numerator+ || element.sampling.sourceFrameRate.denominator !== element.surface.timing.frameRate.denominator) {+ throw new Error(`${trackId}.${present.id}.${element.id} Surface sampling differs from its typed timing.`);+ }+ assertSampling(element.sampling, present.span.endFrameExclusive - present.span.startFrame,+ `${trackId}.${present.id}.${element.id}.sampling`);+ }if (element.surface.timing.kind === "frames") {const durationFrames = present.span.endFrameExclusive - present.span.startFrame;- if (element.surface.timing.frameCount !== durationFrames) {+ if (element.sampling === undefined && element.surface.timing.frameCount !== durationFrames) {throw new Error(`${trackId}.${present.id}.${element.id} Surface must exactly match its Present frame domain.`);}- if (programSpace !== undefined && (+ if (element.sampling === undefined && programSpace !== undefined && (element.surface.timing.frameRate.numerator !== programSpace.frameRate.numerator|| element.surface.timing.frameRate.denominator !== programSpace.frameRate.denominator)) throw new Error(`${trackId}.${present.id}.${element.id} Surface must exactly match its Present frame domain.`);@@ -409,12 +521,28 @@ function normalizeElement(element: VisualElement): VisualElement {? { kind: "still" }: { ...element.surface.timing, frameRate: { ...element.surface.timing.frameRate } },},+ ...(element.sampling === undefined ? {} : { sampling: {+ sourceFrameRate: { ...element.sampling.sourceFrameRate },+ sourceFrameCount: element.sampling.sourceFrameCount,+ segments: element.sampling.segments.map((segment) => ({+ target: { ...segment.target }, sourceFrame: { ...segment.sourceFrame }, rate: { ...segment.rate },+ ...(segment.loop === undefined ? {} : { loop: { ...segment.loop } }),+ })),+ } }),};}return {...common,kind: element.kind,artifact: { ...element.artifact },+ ...(element.sampling === undefined ? {} : { sampling: {+ sourceFrameRate: { ...element.sampling.sourceFrameRate },+ sourceFrameCount: element.sampling.sourceFrameCount,+ segments: element.sampling.segments.map((segment) => ({+ target: { ...segment.target }, sourceFrame: { ...segment.sourceFrame }, rate: { ...segment.rate },+ ...(segment.loop === undefined ? {} : { loop: { ...segment.loop } }),+ })),+ } }),...(element.mediaStartSec === undefined ? {} : { mediaStartSec: element.mediaStartSec }),...(element.playbackRate === undefined ? {} : { playbackRate: element.playbackRate }),...(element.loop === undefined ? {} : { loop: element.loop }),diff --git a/packages/composition/test/track.test.ts b/packages/composition/test/track.test.tsindex adb9bf1c..1db31e59 100644--- a/packages/composition/test/track.test.ts+++ b/packages/composition/test/track.test.ts@@ -14,6 +14,7 @@ import { programSpaceManifest, sealProgramSpace } from "@narratage/program-spaceimport { speechManifest } from "@narratage/speech";import { speechEvidenceManifest } from "@narratage/speech-evidence";import { semanticMapManifest } from "@narratage/semantic-map";+import { spatialManifest } from "@narratage/spatial";import { VISUAL_IR_V1, visualIrManifest } from "@narratage/visual-ir";import {@@ -29,7 +30,7 @@ import {import type { VisualTrack } from "../src/index.js";const videoContractManifests = [artifactManifest, narrativeManifest, mediaManifest, programSpaceManifest,- speechManifest, speechEvidenceManifest, semanticMapManifest, visualIrManifest, compositionManifest] as const;+ speechManifest, speechEvidenceManifest, semanticMapManifest, spatialManifest, visualIrManifest, compositionManifest] as const;const image: MediaArtifactRef = {digest: digestOf("image"),diff --git a/packages/elaborator/test/fragment.test.ts b/packages/elaborator/test/fragment.test.tsindex b85e74db..bd2a4854 100644--- a/packages/elaborator/test/fragment.test.ts+++ b/packages/elaborator/test/fragment.test.ts@@ -1,6 +1,7 @@import { videoContractManifests } from "../../test-support/video-domain.js";import { speechDependency, speechTypes } from "@narratage/speech";import { compositionTypes } from "@narratage/composition";+import { spatialTypes, spatialValidatorDigests } from "@narratage/spatial";import assert from "node:assert/strict";import test from "node:test";@@ -11,6 +12,7 @@ import {sealBuildRequest,sealCompiledGraph,sealRecord,+ sealTypeValidationReceipt,sealTypedModule,start,} from "@narratage/core";@@ -74,6 +76,21 @@ function program(): LinkedProgram {sourceDigest: digestOf("source:fragment"),frontendClosureDigest: digestOf("frontend:fragment"),};+ const rawCanvas = sealRecord({+ id: "canvas:root",+ type: spatialTypes.canvas,+ value: { kind: "inline", value: {+ contract: "svml.canvas-space@1", widthPx: 1080, heightPx: 1920,+ origin: "top-left", xDirection: "right", yDirection: "down", pixelAspect: "square",+ } },+ conformance: "exact",+ origin,+ });+ const canvas = { ...rawCanvas, validation: sealTypeValidationReceipt({+ type: rawCanvas.type,+ recordDigest: rawCanvas.digest,+ validatorDigest: spatialValidatorDigests.canvas,+ }) };return link(closure, [sealTypedModule({id: "author:fragment",closureDigest: closure.digest,@@ -92,6 +109,7 @@ function program(): LinkedProgram {conformance: "exact",origin,}),+ canvas,],})]);}@@ -104,6 +122,7 @@ function speechFragment(): GraphFragment {inputs: [{ name: "request", type: requestType },{ name: "style", type: requestType },+ { name: "canvas", type: spatialTypes.canvas },],operations: [{@@ -121,7 +140,7 @@ function speechFragment(): GraphFragment {{id: "visual",producer: speechBasisProducers.projectVisual,- inputs: { basis: operation("generate") },+ inputs: { basis: operation("generate"), canvas: input("canvas") },result: { kind: "output", name: "visual" },},],@@ -144,7 +163,7 @@ function speechFragment(): GraphFragment {name: "visual",type: compositionTypes.visualTrack,root: operation("visual"),- semanticInputs: ["request", "style"],+ semanticInputs: ["request", "style", "canvas"],fidelity: "exact",},],@@ -159,6 +178,7 @@ function instance(programValue: LinkedProgram, id: string) {inputs: {request: { kind: "record", id: "request:root" },style: { kind: "record", id: "style:root" },+ canvas: { kind: "record", id: "canvas:root" },},});}diff --git a/packages/hyperframes/src/document.ts b/packages/hyperframes/src/document.tsindex 0b0c3786..78eb0fc8 100644--- a/packages/hyperframes/src/document.ts+++ b/packages/hyperframes/src/document.ts@@ -2,7 +2,17 @@ import type { FontArtifactRef } from "@narratage/media";import { programSpaceFrameCount } from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";import { assertCompositionIdentity } from "@narratage/composition";-import type { Composition, Track, VisualAttribute, VisualElement, VisualPresent, VisualStyleDeclaration, VisualTrack } from "@narratage/composition";+import type {+ Composition,+ Track,+ VisualAttribute,+ VisualElement,+ VisualPresent,+ VisualSamplingRational,+ VisualSamplingSegment,+ VisualStyleDeclaration,+ VisualTrack,+} from "@narratage/composition";import { digestOf, isDigest } from "@narratage/protocol";import type { BlobRef, Digest } from "@narratage/protocol";import { VISUAL_IR_V1 } from "@narratage/visual-ir";@@ -48,6 +58,108 @@ function fpsRational(numerator: number, denominator: number): string {return denominator === 1 ? String(numerator) : `${numerator}/${denominator}`;}+function rationalDecimal(numerator: bigint, denominator: bigint): string {+ if (denominator <= 0n) throw new Error("HyperFrames rational denominator must be positive.");+ const scale = 1_000_000_000_000n;+ const scaled = numerator * scale / denominator;+ const whole = scaled / scale;+ const remainder = scaled % scale;+ if (remainder === 0n) return String(whole);+ return `${whole}.${String(remainder).padStart(12, "0").replace(/0+$/u, "")}`;+}++function sourceSeconds(frame: VisualSamplingRational, frameRate: VisualSamplingRational): string {+ return rationalDecimal(+ BigInt(frame.numerator) * BigInt(frameRate.denominator),+ BigInt(frame.denominator) * BigInt(frameRate.numerator),+ );+}++function sourcePosition(segment: VisualSamplingSegment, offset: number): {+ readonly position: VisualSamplingRational;+ readonly cycle: bigint;+} {+ const denominator = BigInt(segment.sourceFrame.denominator) * BigInt(segment.rate.denominator);+ const raw = BigInt(segment.sourceFrame.numerator) * BigInt(segment.rate.denominator)+ + BigInt(offset) * BigInt(segment.rate.numerator) * BigInt(segment.sourceFrame.denominator);+ if (segment.loop === undefined) {+ if (raw < 0n || raw > BigInt(Number.MAX_SAFE_INTEGER) || denominator > BigInt(Number.MAX_SAFE_INTEGER)) {+ throw new Error("HyperFrames visual sampling exceeds safe arithmetic.");+ }+ return { position: { numerator: Number(raw), denominator: Number(denominator) }, cycle: 0n };+ }+ const start = BigInt(segment.loop.startFrame) * denominator;+ const length = BigInt(segment.loop.endFrameExclusive - segment.loop.startFrame) * denominator;+ const delta = raw - start;+ const cycle = delta / length;+ const wrapped = ((delta % length) + length) % length;+ const numerator = start + wrapped;+ if (numerator > BigInt(Number.MAX_SAFE_INTEGER) || denominator > BigInt(Number.MAX_SAFE_INTEGER)) {+ throw new Error("HyperFrames visual sampling exceeds safe arithmetic.");+ }+ return { position: { numerator: Number(numerator), denominator: Number(denominator) }, cycle };+}++function samplingRuns(segment: VisualSamplingSegment): Array<{+ readonly startFrame: number;+ readonly endFrameExclusive: number;+ readonly sourceFrame: VisualSamplingRational;+}> {+ const length = segment.target.endFrameExclusive - segment.target.startFrame;+ const runs: Array<{ startFrame: number; endFrameExclusive: number; sourceFrame: VisualSamplingRational }> = [];+ // HyperFrames and HTMLMediaElement require a positive playback rate. A held+ // visual frame is therefore represented as one independently addressed+ // target-frame clip per Program frame. Every clip starts at the same exact+ // source position; no browser-specific zero-rate behavior is assumed.+ if (segment.rate.numerator === 0) {+ const source = sourcePosition(segment, 0).position;+ for (let offset = 0; offset < length; offset += 1) {+ runs.push({+ startFrame: segment.target.startFrame + offset,+ endFrameExclusive: segment.target.startFrame + offset + 1,+ sourceFrame: source,+ });+ }+ return runs;+ }+ let runStart = 0;+ let runSource = sourcePosition(segment, 0);+ for (let offset = 1; offset < length; offset += 1) {+ const next = sourcePosition(segment, offset);+ if (next.cycle !== runSource.cycle) {+ runs.push({+ startFrame: segment.target.startFrame + runStart,+ endFrameExclusive: segment.target.startFrame + offset,+ sourceFrame: runSource.position,+ });+ runStart = offset;+ runSource = next;+ }+ }+ runs.push({+ startFrame: segment.target.startFrame + runStart,+ endFrameExclusive: segment.target.endFrameExclusive,+ sourceFrame: runSource.position,+ });+ return runs;+}++function sampledPlaybackRate(+ rate: VisualSamplingRational,+ sourceFrameRate: VisualSamplingRational,+ programNumerator: number,+ programDenominator: number,+): string {+ // Zero-rate segments have already been split into one-frame clips above.+ // Their media clock may advance inside that single frame, while every+ // independently rendered target frame still starts at the exact held source.+ if (rate.numerator === 0) return "1";+ return rationalDecimal(+ BigInt(rate.numerator) * BigInt(programNumerator) * BigInt(sourceFrameRate.denominator),+ BigInt(rate.denominator) * BigInt(programDenominator) * BigInt(sourceFrameRate.numerator),+ );+}+function percentage(frame: number, totalFrames: number): string {const scale = 1_000_000_000n;const scaled = BigInt(frame) * 100n * scale / BigInt(totalFrames);@@ -97,6 +209,9 @@ function renderElement(readonly presentId: string;readonly presentStart: string;readonly presentDuration: string;+ readonly presentStartFrame: number;+ readonly programNumerator: number;+ readonly programDenominator: number;readonly stackIndex: number;},): string {@@ -111,6 +226,11 @@ function renderElement(`animation-name:${animationName}`,`animation-duration:${context.presentDuration}s`,"animation-fill-mode:both",+ // An independently launched render worker may begin at any frame. Keep+ // CSS animations inert from first paint so HyperFrames' exact seek is+ // the only clock; otherwise page-load time leaks into the first frame of+ // each partition before the runtime pauses the animation.+ "animation-play-state:paused","animation-timing-function:linear",]),].filter(Boolean).join(";");@@ -120,6 +240,40 @@ function renderElement(.join("");if (element.kind === "box") return `<div ${common}>${descendants}</div>`;if (element.kind === "text") return `<div ${common}>${escapeHtml(element.text)}${descendants}</div>`;+ if ((element.kind === "video" || element.kind === "surface") && element.sampling !== undefined) {+ const artifact = element.kind === "surface" ? element.surface.artifact : element.artifact;+ const source = escapeHtml(hyperframesArtifactUri(artifact.digest));+ let part = 0;+ return element.sampling.segments.flatMap((segment) => samplingRuns(segment).map((run) => {+ part += 1;+ const startFrame = context.presentStartFrame + run.startFrame;+ const durationFrames = run.endFrameExclusive - run.startFrame;+ const partId = `${id}-sample-${String(part).padStart(4, "0")}`;+ const media = [+ `id="${partId}"`,+ `data-svml-element-id="${escapeHtml(element.id)}"`,+ `data-svml-sampling-part="${part}"`,+ `data-start="${frameSeconds(startFrame, context.programNumerator, context.programDenominator)}"`,+ `data-duration="${frameSeconds(durationFrames, context.programNumerator, context.programDenominator)}"`,+ `data-track-index="${context.stackIndex}"`,+ `data-media-start="${sourceSeconds(run.sourceFrame, element.sampling!.sourceFrameRate)}"`,+ `data-playback-rate="${sampledPlaybackRate(segment.rate, element.sampling!.sourceFrameRate, context.programNumerator, context.programDenominator)}"`,+ `data-svml-source-frame="${run.sourceFrame.numerator}/${run.sourceFrame.denominator}"`,+ `data-svml-source-rate="${segment.rate.numerator}/${segment.rate.denominator}"`,+ `style="${escapeHtml(inlineStyle)}"`,+ attributes(element.attributes).trim(),+ "muted",+ "playsinline",+ ...(element.kind === "surface" ? [+ `data-svml-alpha-mode="${element.surface.alphaMode}"`,+ `data-svml-color-space="${element.surface.colorSpace}"`,+ `width="${element.surface.width}"`,+ `height="${element.surface.height}"`,+ ] : []),+ ].filter(Boolean).join(" ");+ return `<video ${media} src="${source}"></video>`;+ })).join("");+ }if (element.kind === "surface") {const source = escapeHtml(hyperframesArtifactUri(element.surface.artifact.digest));const surface = [@@ -173,6 +327,9 @@ function renderVisualPresent(presentId: present.id,presentStart: start,presentDuration: duration,+ presentStartFrame: present.span.startFrame,+ programNumerator: numerator,+ programDenominator: denominator,stackIndex,});return `<div class="clip svml-visual-present" data-svml-track-id="${escapeHtml(track.id)}" data-svml-present-id="${escapeHtml(present.id)}" data-svml-stack-order="${present.stacking.order}" data-svml-stack-tie="${escapeHtml(present.stacking.tieBreak)}" data-track-index="${stackIndex}" data-start="${start}" data-duration="${duration}" style="position:absolute;inset:0;z-index:${stackIndex};overflow:hidden;pointer-events:none">${contents}</div>`;@@ -294,6 +451,17 @@ function emitHtml(composition: Composition, programSpace: ProgramSpace): string<div data-composition-id="${escapeHtml(composition.id)}" data-start="0" data-no-timeline data-width="${composition.canvas.width}" data-height="${composition.canvas.height}" data-duration="${duration}" data-fps="${fps}" data-svml-frame-count="${frameCount}">${visualHtml}</div>+ <script>+ window.addEventListener("hf-seek", (event) => {+ // HyperFrames 0.7.101 exposes a per-seek readiness barrier. One browser+ // task gives Chromium's compositor the same commit opportunity even+ // when this worker begins at a non-zero frame, so partition boundaries+ // cannot leak first-paint state into captured pixels.+ if (event && event.detail && typeof event.detail.waitUntil === "function") {+ event.detail.waitUntil(new Promise((resolve) => setTimeout(resolve, 16)));+ }+ });+ </script></body></html>`;diff --git a/packages/hyperframes/test/browser-visual.test.ts b/packages/hyperframes/test/browser-visual.test.tsindex bc3aeeb0..83f93c89 100644--- a/packages/hyperframes/test/browser-visual.test.ts+++ b/packages/hyperframes/test/browser-visual.test.ts@@ -10,7 +10,8 @@ import { resolveCaptionProgram } from "@narratage/caption";import type { TimedCaptionProjection } from "@narratage/caption";import { fineCaptionStyle, renderFineCaption } from "@narratage/caption-fine";import { decodeOpenFontFaceSurface } from "@narratage/fonts-open";-import type { FontArtifactRef } from "@narratage/media";+import type { CompositableSurfaceRef, FontArtifactRef } from "@narratage/media";+import * as mediaTrack from "@narratage/media-track";import { sealProgramSpace } from "@narratage/program-space";import { sealComposition, sealVisualTrack } from "@narratage/composition";import assert from "node:assert/strict";@@ -18,7 +19,7 @@ import {compileHyperframesDocument,materializeHyperframesHtml,} from "@narratage/hyperframes";-import type { Digest } from "@narratage/protocol";+import type { BlobRef, Digest } from "@narratage/protocol";import { captionDisplaySequence, parseScript } from "@narratage/script";import {appendProgramScreenOverlay,@@ -659,6 +660,144 @@ test("installed open fonts render CJK, emoji and independent stroke, shadow and}});+test("Media two-frame sampling, alpha, local motion and handoff survive partitioned browser rendering", {+ skip: !enabled,+ timeout: 120_000,+}, async () => {+ const temp = await mkdtemp(path.join(os.tmpdir(), "svml-media-track-visual-"));+ try {+ const width = 160;+ const height = 120;+ const redBytes = rgbaPng(80, 120, [245, 35, 35, 255]);+ const greenBytes = rgbaPng(80, 120, [20, 235, 80, 255]);+ const alphaBytes = rgbaPng(64, 64, [35, 70, 255, 176]);+ const red: BlobRef = { kind: "blob", digest: digest(redBytes), size: redBytes.byteLength, mediaType: "image/png" };+ const green: BlobRef = { kind: "blob", digest: digest(greenBytes), size: greenBytes.byteLength, mediaType: "image/png" };+ const alpha: CompositableSurfaceRef = {+ contract: "svml.compositable-surface@1",+ artifact: { kind: "blob", digest: digest(alphaBytes), size: alphaBytes.byteLength, mediaType: "image/png" },+ width: 64,+ height: 64,+ colorSpace: "srgb",+ alphaMode: "straight",+ timing: { kind: "still" },+ };+ const paths = new Map<Digest, string>();+ for (const [name, bytes, artifactDigest] of [+ ["red.png", redBytes, red.digest],+ ["green.png", greenBytes, green.digest],+ ["alpha.png", alphaBytes, alpha.artifact.digest],+ ] as const) {+ await writeFile(path.join(temp, name), bytes);+ paths.set(artifactDigest, `./${name}`);+ }+ const canvas = sealCanvasSpace({+ contract: "svml.canvas-space@1", widthPx: width, heightPx: height,+ origin: "top-left", xDirection: "right", yDirection: "down", pixelAspect: "square",+ });+ const space = sealProgramSpace({+ contract: "svml.program-space@1", durationSec: 1, frameRate: { numerator: 12, denominator: 1 },+ });+ const header = mediaTrack.sealMediaTrackHeader({ contract: "svml.media-track-header@1", id: "browser-media" });+ const sampleAppearance = { opacity: 1, filter: { blurPx: 0, brightness: 1, contrast: 1, saturation: 1 } };+ const contentFit = (sizing: "contain" | "cover") => ({+ contract: "svml.content-fit@1" as const, sizing,+ framePoint: { x: 0.5, y: 0.5 }, contentPoint: { x: 0.5, y: 0.5 },+ offsetPx: { x: 0, y: 0 }, constraint: "bounded" as const,+ });+ const extent = { contract: "svml.intrinsic-extent@1" as const, widthPx: 80, heightPx: 120 };+ let itemLayers = mediaTrack.createMediaLayerSet();+ itemLayers = mediaTrack.appendStillMediaLayer(itemLayers, red, extent, contentFit("cover"), mediaTrack.sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "blurred-backdrop",+ appearance: { opacity: 1, filter: { blurPx: 6, brightness: 0.8, contrast: 1, saturation: 1 } },+ }));+ itemLayers = mediaTrack.appendSurfaceMediaLayer(itemLayers, alpha, contentFit("contain"), mediaTrack.sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "alpha-foreground", appearance: sampleAppearance,+ samplingMotion: { keyframes: [+ { atProgress: 0, zoom: 0.9, offsetX: 0, offsetY: 5, rotationDeg: -2 },+ { atProgress: 1, zoom: 1.05, offsetX: 0, offsetY: -3, rotationDeg: 2, easing: "ease-in-out" },+ ] },+ }));+ let set = mediaTrack.appendProgramMediaItem(+ mediaTrack.createMediaTrackSet(), header, space, canvas, itemLayers,+ { contract: "svml.spatial-frame@1", xPx: 0, yPx: 0, widthPx: 80, heightPx: 120 },+ mediaTrack.sealMediaItemSpec({+ contract: "svml.media-item-spec@1", id: "two-frame",+ projection: { start: { ref: "program.start" }, end: { ref: "program.end" } }, expansion: { kind: "one" },+ presentation: { clip: { kind: "frame" }, padding: { topPx: 0, rightPx: 0, bottomPx: 0, leftPx: 0 }, shadows: [] },+ motion: {+ enter: { operator: "fade", durationFrames: 2, easing: "ease-out" },+ sustain: [{ operator: "breathe", amount: 0.04, cycles: 1 }],+ exit: { operator: "fade", durationFrames: 2, easing: "ease-in" },+ },+ stackingOrder: 10,+ }),+ mediaTrack.createMediaSoundSet(),+ );+ const stillLayers = (id: string, artifact: BlobRef) => mediaTrack.appendStillMediaLayer(+ mediaTrack.createMediaLayerSet(), artifact, extent, contentFit("cover"), mediaTrack.sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id, appearance: sampleAppearance,+ }),+ );+ let members = mediaTrack.createMediaSequenceMemberSet();+ members = mediaTrack.appendMediaSequenceMember(members, stillLayers("red-member", red),+ mediaTrack.sealMediaSequenceMemberSpec({ contract: "svml.media-sequence-member-spec@1", id: "red" }), 0);+ members = mediaTrack.appendMediaSequenceMember(members, stillLayers("green-member", green),+ mediaTrack.sealMediaSequenceMemberSpec({ contract: "svml.media-sequence-member-spec@1", id: "green" }), 6);+ set = mediaTrack.appendMediaSequence(+ set, header, space, canvas, members,+ { contract: "svml.spatial-frame@1", xPx: 80, yPx: 0, widthPx: 80, heightPx: 120 },+ mediaTrack.sealMediaSequenceSpec({+ contract: "svml.media-sequence-spec@1", id: "handoff",+ presentation: { clip: { kind: "frame" }, padding: { topPx: 0, rightPx: 0, bottomPx: 0, leftPx: 0 }, shadows: [] },+ motion: { sustain: [] }, stackingOrder: 20,+ handoffs: [mediaTrack.sealMediaHandoffSpec({+ contract: "svml.media-handoff-spec@1", id: "red-green", fromMemberId: "red", toMemberId: "green",+ operator: "crossfade", durationFrames: 4, boundaryRatio: 0.5, audio: "cut",+ })],+ }),+ mediaTrack.createMediaSoundSet(), 12,+ );+ const track = mediaTrack.projectMediaVisualTrack(space, mediaTrack.finalizeMediaTrack(set, header, space));+ const document = compileHyperframesDocument(sealComposition({+ contract: "svml.composition@1", id: "media-browser-proof",+ canvas: { width, height, clearColor: "#000000" }, tracks: [track],+ }), space);+ await writeFile(path.join(temp, "index.html"), materializeHyperframesHtml(document, (artifact) => {+ const materialized = paths.get(artifact.digest);+ if (materialized === undefined) throw new Error(`Unexpected Media Artifact ${artifact.digest}`);+ return materialized;+ }));+ const render = async (workers: number, name: string): Promise<Buffer[]> => {+ const output = path.join(temp, name);+ await mkdir(output);+ const result = spawnSync(process.execPath, [+ hyperframesCli, "render", temp,+ "--format", "png-sequence", "--output", output, "--fps", "12", "--workers", String(workers),+ "--no-browser-gpu", "--no-best-effort", "--quiet",+ ], { encoding: "utf8", timeout: 110_000 });+ assert.equal(result.status, 0, `${result.stdout}\n${result.stderr}`);+ return Promise.all((await collectPngs(output)).map(async (file) => decodedRgba(file, width, height)));+ };+ const sequential = await render(2, "sequential");+ const partitioned = await render(3, "partitioned");+ assert.equal(sequential.length, 12);+ assert.equal(partitioned.length, 12);+ for (let frameIndex = 0; frameIndex < sequential.length; frameIndex += 1) {+ assert.deepEqual(partitioned[frameIndex], sequential[frameIndex], `Media frame ${frameIndex} changed under partitioning`);+ }+ assert.notDeepEqual(sequential[1], sequential[10], "Media lifecycle and handoff painted no temporal change");+ assert.ok(matchingPixels(sequential[6]!, width, { left: 0, top: 0, right: 80, bottom: 120 },+ (redValue, _greenValue, blueValue) => redValue > 50 && blueValue > 80) > 400,+ "the straight-alpha foreground did not composite over its explicit backdrop");+ assert.ok(matchingPixels(sequential[10]!, width, { left: 80, top: 0, right: 160, bottom: 120 },+ (_redValue, greenValue) => greenValue > 150) > 4_000,+ "the Sequence handoff never reached the incoming material");+ } finally {+ await rm(temp, { recursive: true, force: true });+ }+});+test("every official Screen Overlay survives real sequential and parallel browser frames identically", {skip: !enabled,timeout: 120_000,diff --git a/packages/hyperframes/test/document.test.ts b/packages/hyperframes/test/document.test.tsindex 99f2b57e..fee31b0e 100644--- a/packages/hyperframes/test/document.test.ts+++ b/packages/hyperframes/test/document.test.ts@@ -23,10 +23,10 @@ function fixture() {frameRate: { numerator: 30_000, denominator: 1_001 },});const picture = {+ kind: "blob" as const,digest: digestOf("hyperframes:picture"),size: 10,mediaType: "image/png",- durationSec: 0,};const sound = {kind: "blob" as const,@@ -271,3 +271,63 @@ test("content-bound fonts and typed compositable Surfaces cross the same Artifacassert.match(materialized, new RegExp(font.sources[0]!.artifact.digest, "u"));assert.match(materialized, new RegExp(surfaceDigest, "u"));});++test("exact timed sampling lowers loop boundaries and held frames without zero-rate browser media", () => {+ const programSpace = sealProgramSpace({+ contract: "svml.program-space@1",+ durationSec: 8 / 30,+ frameRate: { numerator: 30, denominator: 1 },+ });+ const artifact = {+ kind: "blob" as const,+ digest: digestOf("hyperframes:sampled-video"),+ size: 1_000,+ mediaType: "video/mp4",+ };+ const track = sealVisualTrack({+ contract: "svml.visual-track@1",+ visualIr: "svml.visual-ir@1",+ id: "sampled",+ presents: [{+ id: "sampled",+ span: { startFrame: 0, endFrameExclusive: 8 },+ stacking: { order: 1, tieBreak: "sampled" },+ elements: [{+ id: "video",+ order: 0,+ kind: "video",+ artifact,+ muted: true,+ sampling: {+ sourceFrameRate: { numerator: 30, denominator: 1 },+ sourceFrameCount: 4,+ segments: [+ {+ target: { startFrame: 0, endFrameExclusive: 6 },+ sourceFrame: { numerator: 2, denominator: 1 },+ rate: { numerator: 1, denominator: 1 },+ loop: { startFrame: 0, endFrameExclusive: 4 },+ },+ {+ target: { startFrame: 6, endFrameExclusive: 8 },+ sourceFrame: { numerator: 3, denominator: 1 },+ rate: { numerator: 0, denominator: 1 },+ },+ ],+ },+ style: [{ name: "position", value: "absolute" }, { name: "inset", value: 0 }],+ }],+ }],+ });+ const document = compileHyperframesDocument(sealComposition({+ contract: "svml.composition@1",+ id: "sampled",+ canvas: { width: 100, height: 100, clearColor: "#000000" },+ tracks: [track],+ }), programSpace);+ assert.equal((document.html.match(/data-svml-sampling-part=/gu) ?? []).length, 4);+ assert.match(document.html, /data-media-start="0\.066666666666"/u);+ assert.match(document.html, /data-media-start="0"/u);+ assert.equal((document.html.match(/data-playback-rate="1"/gu) ?? []).length, 4);+ assert.doesNotMatch(document.html, /data-playback-rate="0"/u);+});diff --git a/packages/hyperframes/test/track-expressiveness.test.ts b/packages/hyperframes/test/track-expressiveness.test.tsindex e5db34eb..a600af17 100644--- a/packages/hyperframes/test/track-expressiveness.test.ts+++ b/packages/hyperframes/test/track-expressiveness.test.ts@@ -1,4 +1,4 @@-import type { CompositableSurfaceRef, MediaArtifactRef } from "@narratage/media";+import type { CompositableSurfaceRef } from "@narratage/media";import { sealProgramSpace } from "@narratage/program-space";import { sealComposition, sealVisualTrack } from "@narratage/composition";import type { Track, VisualElement, VisualPresent, VisualTrack } from "@narratage/composition";@@ -10,6 +10,7 @@ import {compileHyperframesDocument,} from "@narratage/hyperframes";import { digestOf } from "@narratage/protocol";+import type { BlobRef } from "@narratage/protocol";const programSpace = sealProgramSpace({contract: "svml.program-space@1",@@ -244,11 +245,11 @@ test("Caption range/cue/content boxes and word-local timing remain an ordinary V});test("one content box lowers independent backdrop and foreground samples of one media Artifact", () => {- const media: MediaArtifactRef = {+ const media: BlobRef = {+ kind: "blob",digest: digestOf("two-box-source"),size: 1_024,mediaType: "video/mp4",- durationSec: 4,};const track = sealVisualTrack({contract: "svml.visual-track@1",diff --git a/packages/media-execution/src/execute.ts b/packages/media-execution/src/execute.tsindex 9bd63ebe..ad7fb27f 100644--- a/packages/media-execution/src/execute.ts+++ b/packages/media-execution/src/execute.ts@@ -3,7 +3,7 @@ import { createHash } from "node:crypto";import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";import { tmpdir } from "node:os";import { join } from "node:path";-import { mediaTypes, sealMuxedMedia, sealSynchronizedMedia, sealTimelineAudio, verifyMediaInspection, verifyMediaStreamSelection, verifyRenderedVisual, verifyTimelineAudio } from "@narratage/media";+import { mediaTypes, sealMediaInspection, sealMuxedMedia, sealSynchronizedMedia, sealTimelineAudio, verifyMediaInspection, verifyMediaStreamSelection, verifyRenderedVisual, verifyTimelineAudio } from "@narratage/media";import type { MediaAudioStream, MediaInspection, MediaRational, MediaStream, MediaStreamSelection, MediaTimestamp, MediaVideoStream, MuxedMedia, RenderedVisual, TimelineAudio } from "@narratage/media";import type { ProgramSpace } from "@narratage/program-space";import { assertSpeechEvidenceAudioIdentity, sealSpeechEvidenceAudio, speechEvidenceSampleBoundary, speechTypes } from "@narratage/speech";@@ -25,6 +25,13 @@ import {import type { BlobRef, CanonicalValue } from "@narratage/protocol";import { parseMediaInspection } from "./probe.js";+import {+ compositeAnimatedWebpFrame,+ createAnimatedWebpCanvas,+ pamRgba,+ parseAnimatedWebp,+} from "./webp.js";+import type { AnimatedWebp } from "./webp.js";/*** Where the bytes live and which binaries transform them.@@ -325,6 +332,75 @@ async function inspectFile(args: {return parseMediaInspection({ source: args.source, ffprobeVersion, value });}+function divisor(left: number, right: number): number {+ let a = Math.abs(left);+ let b = Math.abs(right);+ while (b !== 0) [a, b] = [b, a % b];+ return a;+}++function animatedWebpInspection(source: BlobRef, animation: AnimatedWebp): MediaInspection {+ const durationMs = animation.frames.reduce((sum, frame) => sum + frame.durationMs, 0);+ const numerator = animation.frames.length * 1_000;+ const factor = divisor(numerator, durationMs);+ return sealMediaInspection({+ contract: "svml.media-inspection@1",+ container: { formatNames: ["webp", "webp-animation"] },+ streams: [{+ kind: "video",+ index: 0,+ codecType: "video",+ codecName: "webp",+ disposition: { default: true, attachedPicture: false },+ timingStatus: "admissible",+ timeBase: { numerator: 1, denominator: 1_000 },+ startPts: { ticks: "0", timeBase: { numerator: 1, denominator: 1_000 } },+ endPts: { ticks: String(durationMs), timeBase: { numerator: 1, denominator: 1_000 } },+ decodedUnitCount: animation.frames.length,+ role: "moving",+ width: animation.width,+ height: animation.height,+ averageFrameRate: { numerator: numerator / factor, denominator: durationMs / factor },+ }],+ });+}++async function animatedWebpConcat(+ env: MediaExecutionEnvironment,+ animation: AnimatedWebp,+ work: string,+): Promise<string> {+ assert(animation.frames.length <= 10_000, "Animated WebP has too many frames");+ const canvasBytes = animation.width * animation.height * 4;+ assert(Number.isSafeInteger(canvasBytes) && canvasBytes > 0 && canvasBytes <= 512 * 1024 * 1024,+ "Animated WebP canvas is outside the supported memory bound");+ const canvas = createAnimatedWebpCanvas(animation);+ const paths: string[] = [];+ for (const [index, frame] of animation.frames.entries()) {+ const imagePath = join(work, `webp-source-${String(index).padStart(6, "0")}.webp`);+ await writeFile(imagePath, frame.image);+ const decoded = await runProcess({+ executable: env.ffmpegPath,+ argv: ["-v", "error", "-i", imagePath, "-frames:v", "1", "-f", "rawvideo", "-pix_fmt", "rgba", "-"],+ timeoutMs: env.processTimeoutMs,+ maxStdoutBytes: frame.width * frame.height * 4,+ ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),+ });+ const snapshot = compositeAnimatedWebpFrame(canvas, animation, frame, decoded);+ const path = join(work, `webp-frame-${String(index).padStart(6, "0")}.pam`);+ await writeFile(path, pamRgba(animation.width, animation.height, snapshot));+ paths.push(path);+ }+ const lines = ["ffconcat version 1.0"];+ for (const [index, path] of paths.entries()) {+ lines.push(`file '${path}'`, `duration ${(animation.frames[index]!.durationMs / 1_000).toFixed(6)}`);+ }+ lines.push(`file '${paths.at(-1)!}'`);+ const manifest = join(work, "animated-webp.ffconcat");+ await writeFile(manifest, `${lines.join("\n")}\n`, "utf8");+ return manifest;+}+async function outputInspection(args: {readonly path: string;readonly mediaType: string;@@ -475,15 +551,19 @@ export async function executeInspectMedia(const work = await mkdtemp(join(tmpdir(), "svml-media-inspect-"));try {const input = join(work, "source.bin");- await writeFile(input, await sourceBytes(env, need.source));- const inspection = await inspectFile({- source: need.source,- input,- ffprobePath: env.ffprobePath,- timeoutMs: env.processTimeoutMs,- maxProbeOutputBytes: env.maxProbeOutputBytes,- ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),- });+ const bytes = await sourceBytes(env, need.source);+ await writeFile(input, bytes);+ const animation = parseAnimatedWebp(bytes);+ const inspection = animation === undefined+ ? await inspectFile({+ source: need.source,+ input,+ ffprobePath: env.ffprobePath,+ timeoutMs: env.processTimeoutMs,+ maxProbeOutputBytes: env.maxProbeOutputBytes,+ ...(env.sharedLibraryPath === undefined ? {} : { sharedLibraryPath: env.sharedLibraryPath }),+ })+ : animatedWebpInspection(need.source, animation);return result(canonicalize(inspection), canonicalize({ provider: env.label, operation: "inspect" }));} finally {await rm(work, { recursive: true, force: true }).catch(() => {});@@ -499,7 +579,9 @@ export async function executeNormalizeMedia(const work = await mkdtemp(join(tmpdir(), "svml-media-normalize-"));try {const input = join(work, "source.bin");- await writeFile(input, await sourceBytes(env, need.source));+ const bytes = await sourceBytes(env, need.source);+ await writeFile(input, bytes);+ const animation = parseAnimatedWebp(bytes);let visualArtifact: BlobRef | undefined;let visualWidth: number | undefined;let visualHeight: number | undefined;@@ -513,9 +595,12 @@ export async function executeNormalizeMedia(`setpts=N*${need.frameRate.denominator}/(${need.frameRate.numerator}*TB)`,"scale=trunc(iw/2)*2:trunc(ih/2)*2",].join(",");+ const visualInput = animation === undefined+ ? ["-i", input, "-map", `0:${plan.video.index}`]+ : ["-f", "concat", "-safe", "0", "-i", await animatedWebpConcat(env, animation, work), "-map", "0:v:0"];await runProcess({executable: env.ffmpegPath,- argv: ["-y", "-i", input, "-map", `0:${plan.video.index}`, "-an", "-vf", filter,+ argv: ["-y", ...visualInput, "-an", "-vf", filter,"-frames:v", String(plan.frameCount), "-fps_mode", "cfr", "-c:v", "libx264","-preset", "veryfast", "-pix_fmt", "yuv420p", "-movflags", "+faststart", output],timeoutMs: env.processTimeoutMs,diff --git a/packages/media-execution/src/webp.ts b/packages/media-execution/src/webp.tsnew file mode 100644index 00000000..65c286cd--- /dev/null+++ b/packages/media-execution/src/webp.ts@@ -0,0 +1,200 @@+export type AnimatedWebpFrame = {+ readonly x: number;+ readonly y: number;+ readonly width: number;+ readonly height: number;+ readonly durationMs: number;+ readonly disposeToBackground: boolean;+ readonly blend: boolean;+ readonly image: Uint8Array;+};++export type AnimatedWebp = {+ readonly width: number;+ readonly height: number;+ readonly background: readonly [number, number, number, number];+ readonly loopCount: number;+ readonly frames: readonly AnimatedWebpFrame[];+};++function ascii(bytes: Uint8Array, start: number, length: number): string {+ return String.fromCharCode(...bytes.subarray(start, start + length));+}++function u24(bytes: Uint8Array, offset: number): number {+ return bytes[offset]! | bytes[offset + 1]! << 8 | bytes[offset + 2]! << 16;+}++function u32(bytes: Uint8Array, offset: number): number {+ return new DataView(bytes.buffer, bytes.byteOffset + offset, 4).getUint32(0, true);+}++function chunk(id: string, payload: Uint8Array): Uint8Array {+ const result = new Uint8Array(8 + payload.byteLength + (payload.byteLength & 1));+ for (let index = 0; index < 4; index += 1) result[index] = id.charCodeAt(index);+ new DataView(result.buffer).setUint32(4, payload.byteLength, true);+ result.set(payload, 8);+ return result;+}++function riff(chunks: readonly Uint8Array[]): Uint8Array {+ const payloadBytes = chunks.reduce((sum, value) => sum + value.byteLength, 0);+ const result = new Uint8Array(12 + payloadBytes);+ result.set([0x52, 0x49, 0x46, 0x46], 0);+ new DataView(result.buffer).setUint32(4, 4 + payloadBytes, true);+ result.set([0x57, 0x45, 0x42, 0x50], 8);+ let offset = 12;+ for (const value of chunks) {+ result.set(value, offset);+ offset += value.byteLength;+ }+ return result;+}++function frameImage(payload: Uint8Array, width: number, height: number): Uint8Array {+ const chunks: Uint8Array[] = [];+ let offset = 0;+ let hasAlpha = false;+ while (offset + 8 <= payload.byteLength) {+ const id = ascii(payload, offset, 4);+ const size = u32(payload, offset + 4);+ const end = offset + 8 + size;+ if (end > payload.byteLength) throw new Error("Animated WebP frame chunk is truncated");+ if (id === "ALPH") hasAlpha = true;+ if (id === "ALPH" || id === "VP8 " || id === "VP8L") {+ chunks.push(chunk(id, payload.subarray(offset + 8, end)));+ }+ offset = end + (size & 1);+ }+ if (!chunks.some((value) => ascii(value, 0, 4) === "VP8 " || ascii(value, 0, 4) === "VP8L")) {+ throw new Error("Animated WebP frame has no image payload");+ }+ if (hasAlpha && chunks.some((value) => ascii(value, 0, 4) === "VP8 ")) {+ const extended = new Uint8Array(10);+ extended[0] = 0x10;+ const widthMinusOne = width - 1;+ const heightMinusOne = height - 1;+ extended[4] = widthMinusOne & 0xff;+ extended[5] = widthMinusOne >> 8 & 0xff;+ extended[6] = widthMinusOne >> 16 & 0xff;+ extended[7] = heightMinusOne & 0xff;+ extended[8] = heightMinusOne >> 8 & 0xff;+ extended[9] = heightMinusOne >> 16 & 0xff;+ chunks.unshift(chunk("VP8X", extended));+ }+ return riff(chunks);+}++/** Parse only the animated WebP container facts FFmpeg currently cannot expose. */+export function parseAnimatedWebp(bytes: Uint8Array): AnimatedWebp | undefined {+ if (bytes.byteLength < 12 || ascii(bytes, 0, 4) !== "RIFF" || ascii(bytes, 8, 4) !== "WEBP") return undefined;+ const declaredEnd = 8 + u32(bytes, 4);+ if (declaredEnd > bytes.byteLength || declaredEnd < 12) throw new Error("WebP RIFF size is invalid");+ let width: number | undefined;+ let height: number | undefined;+ let background: readonly [number, number, number, number] = [0, 0, 0, 0];+ let loopCount = 0;+ const frames: AnimatedWebpFrame[] = [];+ let offset = 12;+ while (offset + 8 <= declaredEnd) {+ const id = ascii(bytes, offset, 4);+ const size = u32(bytes, offset + 4);+ const start = offset + 8;+ const end = start + size;+ if (end > declaredEnd) throw new Error(`WebP ${id} chunk is truncated`);+ const payload = bytes.subarray(start, end);+ if (id === "VP8X") {+ if (size !== 10) throw new Error("WebP VP8X chunk is invalid");+ width = u24(payload, 4) + 1;+ height = u24(payload, 7) + 1;+ } else if (id === "ANIM") {+ if (size !== 6) throw new Error("WebP ANIM chunk is invalid");+ background = [payload[2]!, payload[1]!, payload[0]!, payload[3]!];+ loopCount = payload[4]! | payload[5]! << 8;+ } else if (id === "ANMF") {+ if (size < 16) throw new Error("WebP ANMF chunk is invalid");+ const frameWidth = u24(payload, 6) + 1;+ const frameHeight = u24(payload, 9) + 1;+ const durationMs = u24(payload, 12);+ if (durationMs < 1) throw new Error("Animated WebP frame duration must be positive");+ frames.push({+ x: u24(payload, 0) * 2,+ y: u24(payload, 3) * 2,+ width: frameWidth,+ height: frameHeight,+ durationMs,+ disposeToBackground: (payload[15]! & 1) !== 0,+ blend: (payload[15]! & 2) === 0,+ image: frameImage(payload.subarray(16), frameWidth, frameHeight),+ });+ }+ offset = end + (size & 1);+ }+ if (frames.length === 0) return undefined;+ if (width === undefined || height === undefined) throw new Error("Animated WebP has no canvas size");+ if (frames.some((frame) => frame.x + frame.width > width! || frame.y + frame.height > height!)) {+ throw new Error("Animated WebP frame escapes its canvas");+ }+ return { width, height, background, loopCount, frames };+}++function clear(canvas: Uint8Array, animation: AnimatedWebp, frame: AnimatedWebpFrame): void {+ for (let y = frame.y; y < frame.y + frame.height; y += 1) {+ for (let x = frame.x; x < frame.x + frame.width; x += 1) {+ const target = (y * animation.width + x) * 4;+ canvas.set(animation.background, target);+ }+ }+}++function blendPixel(canvas: Uint8Array, target: number, source: Uint8Array, sourceOffset: number): void {+ const sourceAlpha = source[sourceOffset + 3]!;+ if (sourceAlpha === 255) {+ canvas.set(source.subarray(sourceOffset, sourceOffset + 4), target);+ return;+ }+ if (sourceAlpha === 0) return;+ const destinationAlpha = canvas[target + 3]!;+ const inverse = 255 - sourceAlpha;+ const outputAlpha255 = sourceAlpha * 255 + destinationAlpha * inverse;+ for (let channel = 0; channel < 3; channel += 1) {+ const numerator = source[sourceOffset + channel]! * sourceAlpha * 255+ + canvas[target + channel]! * destinationAlpha * inverse;+ canvas[target + channel] = Math.round(numerator / outputAlpha255);+ }+ canvas[target + 3] = Math.round(outputAlpha255 / 255);+}++export function compositeAnimatedWebpFrame(+ canvas: Uint8Array,+ animation: AnimatedWebp,+ frame: AnimatedWebpFrame,+ decoded: Uint8Array,+): Uint8Array {+ if (decoded.byteLength !== frame.width * frame.height * 4) {+ throw new Error("Decoded WebP frame dimensions differ from the container");+ }+ for (let y = 0; y < frame.height; y += 1) {+ for (let x = 0; x < frame.width; x += 1) {+ const source = (y * frame.width + x) * 4;+ const target = ((frame.y + y) * animation.width + frame.x + x) * 4;+ if (frame.blend) blendPixel(canvas, target, decoded, source);+ else canvas.set(decoded.subarray(source, source + 4), target);+ }+ }+ const snapshot = canvas.slice();+ if (frame.disposeToBackground) clear(canvas, animation, frame);+ return snapshot;+}++export function createAnimatedWebpCanvas(animation: AnimatedWebp): Uint8Array {+ const canvas = new Uint8Array(animation.width * animation.height * 4);+ for (let offset = 0; offset < canvas.byteLength; offset += 4) canvas.set(animation.background, offset);+ return canvas;+}++export function pamRgba(width: number, height: number, rgba: Uint8Array): Uint8Array {+ if (rgba.byteLength !== width * height * 4) throw new Error("PAM frame byte count is invalid");+ const header = Buffer.from(`P7\nWIDTH ${width}\nHEIGHT ${height}\nDEPTH 4\nMAXVAL 255\nTUPLTYPE RGB_ALPHA\nENDHDR\n`, "ascii");+ return Buffer.concat([header, rgba]);+}diff --git a/packages/media-execution/test/webp.test.ts b/packages/media-execution/test/webp.test.tsnew file mode 100644index 00000000..428b607a--- /dev/null+++ b/packages/media-execution/test/webp.test.ts@@ -0,0 +1,53 @@+import assert from "node:assert/strict";+import test from "node:test";++import {+ compositeAnimatedWebpFrame,+ createAnimatedWebpCanvas,+} from "../src/webp.js";+import type { AnimatedWebp, AnimatedWebpFrame } from "../src/webp.js";++function animation(frame: AnimatedWebpFrame): AnimatedWebp {+ return {+ width: 3,+ height: 1,+ background: [10, 20, 30, 255],+ loopCount: 0,+ frames: [frame],+ };+}++test("animated WebP composition honors no-blend and dispose-to-background independently", () => {+ const frame: AnimatedWebpFrame = {+ x: 1,+ y: 0,+ width: 1,+ height: 1,+ durationMs: 100,+ disposeToBackground: true,+ blend: false,+ image: new Uint8Array(),+ };+ const source = animation(frame);+ const canvas = createAnimatedWebpCanvas(source);+ const snapshot = compositeAnimatedWebpFrame(canvas, source, frame, new Uint8Array([200, 100, 50, 128]));+ assert.deepEqual([...snapshot], [10, 20, 30, 255, 200, 100, 50, 128, 10, 20, 30, 255]);+ assert.deepEqual([...canvas], [10, 20, 30, 255, 10, 20, 30, 255, 10, 20, 30, 255]);+});++test("animated WebP alpha blending uses the existing straight-alpha canvas", () => {+ const frame: AnimatedWebpFrame = {+ x: 0,+ y: 0,+ width: 1,+ height: 1,+ durationMs: 100,+ disposeToBackground: false,+ blend: true,+ image: new Uint8Array(),+ };+ const source = animation(frame);+ const canvas = createAnimatedWebpCanvas(source);+ const snapshot = compositeAnimatedWebpFrame(canvas, source, frame, new Uint8Array([210, 120, 70, 128]));+ assert.deepEqual([...snapshot.slice(0, 4)], [110, 70, 50, 255]);+});diff --git a/packages/media-pipeline/package.json b/packages/media-pipeline/package.jsonindex acc0eb32..4e8db81b 100644--- a/packages/media-pipeline/package.json+++ b/packages/media-pipeline/package.json@@ -18,7 +18,8 @@"@narratage/narrative": "workspace:*","@narratage/program-space": "workspace:*","@narratage/protocol": "workspace:*",- "@narratage/speech": "workspace:*"+ "@narratage/speech": "workspace:*",+ "@narratage/text": "workspace:*"},"devDependencies": {"@narratage/core": "workspace:*",diff --git a/packages/media-pipeline/src/activation.ts b/packages/media-pipeline/src/activation.tsindex 47f92555..240a2155 100644--- a/packages/media-pipeline/src/activation.ts+++ b/packages/media-pipeline/src/activation.ts@@ -1,9 +1,23 @@-import { mediaPipelineComponent, mediaPipelineManifest } from "./index.js";+import { createTextSurfaceHostFacet } from "@narratage/text";+import {+ decodeSynchronizedMediaSurface,+ mediaPipelineComponent,+ mediaPipelineManifest,+ mediaPipelineModuleRef,+ synchronizedMediaSurfaceImplementationDigest,+} from "./index.js";export const svmlPackage = {format: "svml.node-package@1" as const,name: "@narratage/media-pipeline",- modules: [{ manifest: mediaPipelineManifest }],+ modules: [{ manifest: mediaPipelineManifest, specifiers: [mediaPipelineModuleRef.name, `${mediaPipelineModuleRef.name}@1`] }],components: [mediaPipelineComponent],+ hostFacets: [createTextSurfaceHostFacet({+ module: mediaPipelineModuleRef,+ surface: "synchronized-media",+ mode: "structured",+ implementationDigest: synchronizedMediaSurfaceImplementationDigest,+ handler: decodeSynchronizedMediaSurface,+ })],};export default svmlPackage;diff --git a/packages/media-pipeline/src/index.ts b/packages/media-pipeline/src/index.tsindex 736bcce1..28079f08 100644--- a/packages/media-pipeline/src/index.ts+++ b/packages/media-pipeline/src/index.ts@@ -18,10 +18,12 @@ export {mediaPipelineTypes,audioProgramPlanSchema,mediaSelectionRequestSchema,+ synchronizedMediaSurfaceImplementationDigest,} from "./manifest.js";export {sealMediaSelectionRequest,selectMediaStreams,verifyMediaSelectionRequest,} from "./selection.js";+export { decodeSynchronizedMediaSurface } from "./surface.js";export type * from "./types.js";diff --git a/packages/media-pipeline/src/manifest.ts b/packages/media-pipeline/src/manifest.tsindex 098803f6..0f19fc76 100644--- a/packages/media-pipeline/src/manifest.ts+++ b/packages/media-pipeline/src/manifest.ts@@ -47,6 +47,7 @@ export const mediaPipelineImplementationDigests = {projectMuxed: digestOf("@narratage/media-pipeline/project-muxed-media@1"),audioPlanValidator: digestOf("@narratage/media-pipeline/validate-audio-program-plan@1"),} as const;+export const synchronizedMediaSurfaceImplementationDigest = digestOf("@narratage/media-pipeline/synchronized-media-surface@1");const integer = { kind: "number", integer: true, minimum: 0 } as const;const mode = (name: string): ValueSchema => ({@@ -174,7 +175,11 @@ export const mediaPipelineManifest: ModuleManifest = {{ name: mediaPipelineCapabilities.renderAudio.name, returns: mediaTypes.timelineAudio },{ name: mediaPipelineCapabilities.mux.name, returns: mediaTypes.muxed },],- surfaces: [],+ surfaces: [{+ name: "synchronized-media", tag: "Normalize", mode: "structured",+ outputs: [mediaPipelineTypes.selectionRequest, mediaTypes.synchronized],+ implementation: { kind: "trusted-frontend-surface", locator: "@narratage/media-pipeline/synchronized-media-surface", digest: synchronizedMediaSurfaceImplementationDigest },+ }],producers: [{name: mediaPipelineProducers.inspect.name,diff --git a/packages/media-pipeline/src/surface.ts b/packages/media-pipeline/src/surface.tsnew file mode 100644index 00000000..600b52bd--- /dev/null+++ b/packages/media-pipeline/src/surface.ts@@ -0,0 +1,85 @@+import { artifactTypes } from "@narratage/artifact";+import type { StructuredElement, StructuredSurfaceHandler, SurfaceResolvedReference, TextAttributeValue } from "@narratage/text";++import { synchronizedMediaFragment } from "./fragment.js";+import { mediaPipelineTypes } from "./manifest.js";+import { sealMediaSelectionRequest } from "./selection.js";+import type { MediaSelectionRequest } from "./types.js";++function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;+}++function text(element: StructuredElement, name: string): string {+ const value = element.attributes[name];+ if (typeof value !== "string" || value.trim().length === 0) throw new Error(`${element.name}.${name} must be non-empty text.`);+ return value.trim();+}++function ref(+ raw: TextAttributeValue | undefined,+ label: string,+ resolve: (path: string) => SurfaceResolvedReference | undefined,+): SurfaceResolvedReference {+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${label} must be a reference.`);+ const value = resolve(raw.path);+ if (value === undefined || !sameType(value.type, artifactTypes.blob)) throw new Error(`${label} must resolve to BlobArtifact.`);+ return value;+}++function stream(value: string, kind: "video"): MediaSelectionRequest["video"];+function stream(value: string, kind: "audio"): MediaSelectionRequest["audio"];+function stream(value: string, kind: "video" | "audio"): MediaSelectionRequest["video"] | MediaSelectionRequest["audio"] {+ if (kind === "video" && value === "primary-moving") return { mode: "primary-moving" };+ if (kind === "audio" && value === "default") return { mode: "default" };+ if (value === "none") return { mode: "none" };+ const match = /^stream:(\d+)$/u.exec(value);+ if (match === null) throw new Error(`${kind} must be ${kind === "video" ? "primary-moving" : "default"}, none or stream:<index>.`);+ return { mode: "stream-index", streamIndex: Number(match[1]) };+}++function frameRate(value: string): { readonly numerator: number; readonly denominator: number } {+ const match = /^(\d+)(?:\/(\d+))?$/u.exec(value);+ if (match === null) throw new Error("Normalize.frame-rate must be a positive rational such as 30 or 30000/1001.");+ const numerator = Number(match[1]);+ const denominator = Number(match[2] ?? "1");+ if (!Number.isSafeInteger(numerator) || numerator < 1 || !Number.isSafeInteger(denominator) || denominator < 1) {+ throw new Error("Normalize.frame-rate is invalid.");+ }+ return { numerator, denominator };+}++export const decodeSynchronizedMediaSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ const expected = ["audio", "frame-rate", "id", "source", "span-authority", "video"];+ if (Object.keys(element.attributes).sort().join("\0") !== expected.sort().join("\0")) {+ throw new Error(`${element.name} requires exactly id, source, video, audio, span-authority and frame-rate.`);+ }+ if (element.children.some((child) => child.kind === "element" || child.value.trim().length > 0)) {+ throw new Error(`${element.name} must be empty.`);+ }+ const id = text(element, "id");+ const source = ref(element.attributes.source, `${element.name}.source`, resolveReference);+ const video = stream(text(element, "video"), "video");+ const audio = stream(text(element, "audio"), "audio");+ const spanAuthority = text(element, "span-authority");+ if (spanAuthority !== "video" && spanAuthority !== "audio") throw new Error(`${element.name}.span-authority must be video or audio.`);+ const request = sealMediaSelectionRequest({+ contract: "svml.media-selection-request@1",+ video,+ audio,+ spanAuthority,+ frameRate: frameRate(text(element, "frame-rate")),+ });+ const requestId = `${id}.request`;+ return {+ records: [{ id: requestId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline", value: request }, range: element.range }],+ components: [{+ id,+ fragment: synchronizedMediaFragment.id,+ inputs: { source: source.ref, request: { kind: "record", id: requestId } },+ outputs: { media: `${id}.media` },+ range: element.range,+ }],+ fragments: [synchronizedMediaFragment],+ };+};diff --git a/packages/media-pipeline/test/manifest.test.ts b/packages/media-pipeline/test/manifest.test.tsindex 2bc2f000..582d46e6 100644--- a/packages/media-pipeline/test/manifest.test.ts+++ b/packages/media-pipeline/test/manifest.test.ts@@ -1,6 +1,7 @@import { videoContractManifests } from "../../test-support/video-domain.js";import { registerTypeValidatorFacets } from "@narratage/component-kit";import { mediaTypes } from "@narratage/media";+import { artifactTypes } from "@narratage/artifact";import assert from "node:assert/strict";import test from "node:test";@@ -15,6 +16,7 @@ import {mediaPipelineComponents,mediaPipelineManifest,mediaPipelineTypes,+ decodeSynchronizedMediaSurface,synchronizedMediaFragment,} from "../src/index.js";@@ -28,6 +30,42 @@ test("the media pipeline is an ordinary Fragment over public media contracts", (assert.equal(synchronizedMediaFragment.exports[0]?.type.name, mediaTypes.synchronized.name);});+test("the Normalize Surface makes inspection and normalization an explicit author graph branch", async () => {+ const range = { source: "normalize.svml", start: 0, end: 1 };+ const result = await decodeSynchronizedMediaSurface({+ sourceName: range.source,+ element: {+ kind: "element",+ name: "pipeline:Normalize",+ attributes: {+ id: "motion",+ source: { kind: "reference", path: "generated" },+ video: "primary-moving",+ audio: "none",+ "span-authority": "video",+ "frame-rate": "30000/1001",+ },+ children: [],+ range,+ },+ resolveReference: (path) => path === "generated"+ ? { path, ref: { kind: "record", id: path }, type: artifactTypes.blob }+ : undefined,+ resolveAsset: async () => { throw new Error("no asset resolution expected"); },+ });+ assert.equal(result.fragments[0]?.id, synchronizedMediaFragment.id);+ assert.deepEqual(result.components[0]?.outputs, { media: "motion.media" });+ const request = result.records[0]?.value;+ assert.ok(request?.kind === "inline");+ assert.deepEqual(request.value, {+ contract: "svml.media-selection-request@1",+ video: { mode: "primary-moving" },+ audio: { mode: "none" },+ spanAuthority: "video",+ frameRate: { numerator: 30_000, denominator: 1_001 },+ });+});+test("media identities and selection requests have package-owned semantic validators", () => {const registry = new TypeValidatorRegistry();for (const component of mediaPipelineComponents) {diff --git a/packages/media-track/README.md b/packages/media-track/README.mdindex 50cd67f7..4fa5b7fe 100644--- a/packages/media-track/README.md+++ b/packages/media-track/README.md@@ -2,11 +2,17 @@Official provider-free Media Item and Sequence authoring package.-The current executable slice implements one independently timed still-image Item. Its Artifact,-intrinsic extent, Placement Frame, Content Fit and ProgramSpace are separate graph inputs. The-resolved package-owned Program lowers to an ordinary peer `VisualTrack`; it does not add Media,-B-roll or Provider meaning to Core, Film, Composition or HyperFrames.--Timed visual sources, frame Paint, motion, Sequence handoffs, explicit audio projection and the-author Surface remain governed by `spec/media-track.md` and are intentionally not claimed by this-first witness.+`Track` accepts explicit ProgramSpace, CanvasSpace and, when semantic points are used, a+CompleteSemanticMap. Each independently timed `Item` receives an explicit SpatialFrame and either+one direct normalized source or ordered Paint/sample layers. `Sequence` owns an ordered replacement+surface with explicit activation points and pairwise Handoffs. Package-owned Recipes cover fitting,+source occupancy, frame Paint, clipping, borders, shadows, lifecycle motion and sampling motion.++Visual and audio are separate deterministic projections. A visual-only Track exports only+`VisualTrack`; selecting one source layer's audio or adding explicit enter/exit/handoff sounds also+exports a peer `AudioTrack`. Speech Spine reuses the same lowering laws through a deliberately+restricted internal projection.++The package does not add Media, B-roll or Provider meaning to Core, Film, Composition or+HyperFrames. “B-roll” is an editorial use of an ordinary Item or Sequence. The complete pre-release+contract and acceptance matrix live in `spec/media-track.md`.diff --git a/packages/media-track/package.json b/packages/media-track/package.jsonindex 9b73fa4c..e74e19eb 100644--- a/packages/media-track/package.json+++ b/packages/media-track/package.json@@ -15,9 +15,13 @@"@narratage/composition": "workspace:*","@narratage/elaborator": "workspace:*","@narratage/media": "workspace:*",+ "@narratage/narrative": "workspace:*","@narratage/program-space": "workspace:*","@narratage/protocol": "workspace:*",+ "@narratage/semantic-map": "workspace:*","@narratage/spatial": "workspace:*",+ "@narratage/svs": "workspace:*",+ "@narratage/text": "workspace:*","@narratage/temporal": "workspace:*"}}diff --git a/packages/media-track/src/activation.ts b/packages/media-track/src/activation.tsindex 1a330b84..b2fa5864 100644--- a/packages/media-track/src/activation.ts+++ b/packages/media-track/src/activation.ts@@ -1,9 +1,23 @@-import { mediaTrackComponent, mediaTrackManifest, mediaTrackModuleRef } from "./index.js";+import { createTextSurfaceHostFacet } from "@narratage/text";+import {+ decodeMediaTrackSurface,+ mediaTrackComponent,+ mediaTrackManifest,+ mediaTrackModuleRef,+ mediaTrackSurfaceImplementationDigest,+} from "./index.js";export const svmlPackage = {format: "svml.node-package@1" as const,name: "@narratage/media-track",modules: [{ manifest: mediaTrackManifest, specifiers: [mediaTrackModuleRef.name, `${mediaTrackModuleRef.name}@1`] }],components: [mediaTrackComponent],+ hostFacets: [createTextSurfaceHostFacet({+ module: mediaTrackModuleRef,+ surface: "track",+ mode: "structured",+ implementationDigest: mediaTrackSurfaceImplementationDigest,+ handler: decodeMediaTrackSurface,+ })],};export default svmlPackage;diff --git a/packages/media-track/src/author.ts b/packages/media-track/src/author.tsnew file mode 100644index 00000000..b10b109d--- /dev/null+++ b/packages/media-track/src/author.ts@@ -0,0 +1,320 @@+import type { SvsRecipe } from "@narratage/svs";+import { sealContentFit } from "@narratage/spatial";+import type { ContentFit } from "@narratage/spatial";++import {+ sealMediaPaintLayerSpec,+ sealMediaSampleLayerSpec,+} from "./layers.js";+import { sealMediaItemSpec } from "./program.js";+import { sealMediaHandoffSpec, sealMediaSequenceSpec } from "./sequence.js";+import type {+ MediaFramePresentation,+ MediaHandoffSpec,+ MediaItemSpec,+ MediaLifecycleMotion,+ MediaPaint,+ MediaPaintLayerSpec,+ MediaSampleLayerSpec,+ MediaSequenceSpec,+ MediaVisualOccupancy,+ MediaVisualTrim,+} from "./types.js";++function fail(recipe: SvsRecipe, message: string): never {+ throw new Error(`Media Recipe ${recipe.path} ${message}`);+}++function optionalNumber(recipe: SvsRecipe, name: string): number | undefined {+ const value = recipe.properties[name];+ if (value === undefined) return undefined;+ if (typeof value !== "number" || !Number.isFinite(value)) fail(recipe, `${name} must be a finite number.`);+ return value;+}++function number(recipe: SvsRecipe, name: string, fallback?: number): number {+ const value = optionalNumber(recipe, name);+ if (value === undefined && fallback !== undefined) return fallback;+ if (value === undefined) fail(recipe, `requires ${name}.`);+ return value;+}++function optionalString(recipe: SvsRecipe, name: string): string | undefined {+ const value = recipe.properties[name];+ if (value === undefined) return undefined;+ if (typeof value !== "string" || value.trim().length === 0) fail(recipe, `${name} must be text.`);+ return value.trim();+}++function string(recipe: SvsRecipe, name: string, fallback?: string): string {+ const value = optionalString(recipe, name);+ if (value === undefined && fallback !== undefined) return fallback;+ if (value === undefined) fail(recipe, `requires ${name}.`);+ return value;+}++function oneOf<T extends string>(recipe: SvsRecipe, name: string, values: readonly T[], fallback?: T): T {+ const value = optionalString(recipe, name);+ if (value === undefined && fallback !== undefined) return fallback;+ if (value === undefined || !values.includes(value as T)) fail(recipe, `${name} must be ${values.join(" | ")}.`);+ return value as T;+}++function assertKeys(recipe: SvsRecipe, allowed: readonly string[]): void {+ const unknown = Object.keys(recipe.properties).filter((name) => !allowed.includes(name));+ if (unknown.length > 0) fail(recipe, `does not accept ${unknown.join(", ")}.`);+}++function scalarList(recipe: SvsRecipe, name: string, allowed: readonly number[]): number[] {+ const source = optionalString(recipe, name);+ if (source === undefined) return [];+ const values = source.split(/\s+/u).map(Number);+ if (!allowed.includes(values.length) || values.some((value) => !Number.isFinite(value))) {+ fail(recipe, `${name} requires ${allowed.join(" or ")} finite numbers.`);+ }+ return values;+}++function padding(recipe: SvsRecipe): MediaFramePresentation["padding"] {+ const values = scalarList(recipe, "padding", [1, 2, 4]);+ if (values.length === 0) return { topPx: 0, rightPx: 0, bottomPx: 0, leftPx: 0 };+ if (values.length === 1) return { topPx: values[0]!, rightPx: values[0]!, bottomPx: values[0]!, leftPx: values[0]! };+ if (values.length === 2) return { topPx: values[0]!, rightPx: values[1]!, bottomPx: values[0]!, leftPx: values[1]! };+ return { topPx: values[0]!, rightPx: values[1]!, bottomPx: values[2]!, leftPx: values[3]! };+}++function gradientStops(recipe: SvsRecipe, source: string): readonly { readonly offset: number; readonly color: string }[] {+ const stops = source.split(",").map((entry) => {+ const match = /^(.+?)@((?:0(?:\.\d+)?)|(?:1(?:\.0+)?))$/u.exec(entry.trim());+ if (match === null) fail(recipe, "gradient stops must be color@offset inside [0,1].");+ return { color: match[1]!.trim(), offset: Number(match[2]) };+ });+ if (stops.length < 2) fail(recipe, "gradient requires at least two stops.");+ return stops;+}++function paint(recipe: SvsRecipe, property = "paint"): MediaPaint {+ const source = string(recipe, property);+ const linear = /^linear\(([-+]?\d+(?:\.\d+)?);(.+)\)$/u.exec(source);+ if (linear !== null) {+ return { kind: "linear-gradient", angleDeg: Number(linear[1]), stops: gradientStops(recipe, linear[2]!) };+ }+ const radial = /^radial\(((?:0(?:\.\d+)?)|(?:1(?:\.0+)?)),((?:0(?:\.\d+)?)|(?:1(?:\.0+)?));(.+)\)$/u.exec(source);+ if (radial !== null) {+ return { kind: "radial-gradient", center: { x: Number(radial[1]), y: Number(radial[2]) }, stops: gradientStops(recipe, radial[3]!) };+ }+ return { kind: "solid", color: source };+}++function shadows(recipe: SvsRecipe): MediaFramePresentation["shadows"] {+ const source = optionalString(recipe, "shadows");+ if (source === undefined || source === "none") return [];+ return source.split(";").map((entry) => {+ const match = /^([-+]?\d+(?:\.\d+)?)\s+([-+]?\d+(?:\.\d+)?)\s+(\d+(?:\.\d+)?)\s+([-+]?\d+(?:\.\d+)?)\s+(.+)$/u.exec(entry.trim());+ if (match === null) fail(recipe, "shadows entries require x y blur spread color.");+ return {+ offsetX: Number(match[1]), offsetY: Number(match[2]), blurPx: Number(match[3]),+ spreadPx: Number(match[4]), color: match[5]!.trim(),+ };+ });+}++const FIT_KEYS = ["fit", "frame-x", "frame-y", "content-x", "content-y", "fit-offset-x", "fit-offset-y", "fit-constraint"] as const;+const SAMPLE_KEYS = ["opacity", "blur", "brightness", "contrast", "saturation", "playback", "trim-start", "trim-end"] as const;+const FRAME_KEYS = ["stack-order", "clip", "radius", "padding", "border-width", "border-style", "border-color", "shadows", "frame-paint"] as const;+const MOTION_KEYS = ["enter", "enter-frames", "enter-easing", "enter-direction", "enter-amount", "enter-origin", "sustain", "exit", "exit-frames", "exit-easing", "exit-direction", "exit-amount", "exit-origin"] as const;++export function decodeMediaFit(recipe: SvsRecipe): ContentFit {+ return sealContentFit({+ contract: "svml.content-fit@1",+ sizing: oneOf(recipe, "fit", ["contain", "cover", "fit-width", "fit-height", "native", "scale-down", "stretch"] as const, "contain"),+ framePoint: { x: number(recipe, "frame-x", 0.5), y: number(recipe, "frame-y", 0.5) },+ contentPoint: { x: number(recipe, "content-x", 0.5), y: number(recipe, "content-y", 0.5) },+ offsetPx: { x: number(recipe, "fit-offset-x", 0), y: number(recipe, "fit-offset-y", 0) },+ constraint: oneOf(recipe, "fit-constraint", ["bounded", "free"] as const, "bounded"),+ });+}++function occupancy(recipe: SvsRecipe): MediaVisualOccupancy {+ switch (oneOf(recipe, "playback", ["once-start", "once-end", "hold-start", "hold-end", "loop-start", "loop-end", "stretch"] as const, "once-start")) {+ case "once-start": return { mode: "once", align: "start" };+ case "once-end": return { mode: "once", align: "end" };+ case "hold-start": return { mode: "hold", align: "start" };+ case "hold-end": return { mode: "hold", align: "end" };+ case "loop-start": return { mode: "loop", align: "start" };+ case "loop-end": return { mode: "loop", align: "end" };+ case "stretch": return { mode: "stretch" };+ }+}++function trim(recipe: SvsRecipe): MediaVisualTrim | undefined {+ const startFrame = optionalNumber(recipe, "trim-start");+ const endFrameExclusive = optionalNumber(recipe, "trim-end");+ if (startFrame === undefined && endFrameExclusive === undefined) return undefined;+ if (+ startFrame === undefined+ || endFrameExclusive === undefined+ || !Number.isSafeInteger(startFrame)+ || !Number.isSafeInteger(endFrameExclusive)+ ) {+ fail(recipe, "trim-start and trim-end must both be integer frames.");+ }+ return { startFrame, endFrameExclusive };+}++export function decodeMediaSampleSpec(+ recipe: SvsRecipe,+ id: string,+ sourceKind: "still" | "timed" | "surface",+ samplingMotion?: MediaSampleLayerSpec["samplingMotion"],+ combinedItemAppearance = false,+): MediaSampleLayerSpec {+ assertKeys(recipe, [...FIT_KEYS, ...SAMPLE_KEYS, ...(combinedItemAppearance ? FRAME_KEYS : [])]);+ const hasTimedProperties = recipe.properties.playback !== undefined+ || recipe.properties["trim-start"] !== undefined+ || recipe.properties["trim-end"] !== undefined;+ if (sourceKind === "still" && hasTimedProperties) {+ fail(recipe, "cannot apply playback or trim to durationless still material.");+ }+ const timed = sourceKind === "timed" || (sourceKind === "surface" && hasTimedProperties);+ const sourceTrim = timed ? trim(recipe) : undefined;+ return sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1",+ id,+ ...(timed ? { occupancy: occupancy(recipe), ...(sourceTrim === undefined ? {} : { trim: sourceTrim }) } : {}),+ appearance: {+ opacity: number(recipe, "opacity", 1),+ filter: {+ blurPx: number(recipe, "blur", 0), brightness: number(recipe, "brightness", 1),+ contrast: number(recipe, "contrast", 1), saturation: number(recipe, "saturation", 1),+ },+ },+ ...(samplingMotion === undefined ? {} : { samplingMotion }),+ });+}++function edge(recipe: SvsRecipe, prefix: "enter" | "exit"): MediaLifecycleMotion["enter"] {+ const operator = optionalString(recipe, prefix);+ if (operator === undefined || operator === "none") return undefined;+ const allowed = ["fade", "slide", "scale", "pop", "bounce", "blur-reveal", "wipe", "flip", "spin"] as const;+ if (!allowed.includes(operator as typeof allowed[number])) fail(recipe, `${prefix} operator is invalid.`);+ const direction = optionalString(recipe, `${prefix}-direction`);+ const origin = optionalString(recipe, `${prefix}-origin`);+ return {+ operator: operator as typeof allowed[number],+ durationFrames: number(recipe, `${prefix}-frames`),+ easing: oneOf(recipe, `${prefix}-easing`, ["linear", "ease-in", "ease-out", "ease-in-out"] as const, "ease-in-out"),+ ...(direction === undefined ? {} : { direction: oneOf(recipe, `${prefix}-direction`, ["left", "right", "up", "down"] as const) }),+ ...(origin === undefined ? {} : { origin: oneOf(recipe, `${prefix}-origin`, ["outside-canvas"] as const) }),+ ...(optionalNumber(recipe, `${prefix}-amount`) === undefined ? {} : { amount: number(recipe, `${prefix}-amount`) }),+ };+}++function sustain(recipe: SvsRecipe): MediaLifecycleMotion["sustain"] {+ const source = optionalString(recipe, "sustain");+ if (source === undefined || source === "none") return [];+ return source.split(",").map((entry) => {+ const match = /^(float|breathe|pulse|wobble|shake|drift)\s+([-+]?\d+(?:\.\d+)?)\s+([1-9]\d*)(?:\s+(left|right|up|down))?$/u.exec(entry.trim());+ if (match === null) fail(recipe, "sustain entries require operator amount cycles [direction].");+ return {+ operator: match[1] as MediaLifecycleMotion["sustain"][number]["operator"],+ amount: Number(match[2]), cycles: Number(match[3]),+ ...(match[4] === undefined ? {} : { direction: match[4] as "left" | "right" | "up" | "down" }),+ };+ });+}++export function decodeMediaMotion(recipe: SvsRecipe | undefined): MediaLifecycleMotion {+ if (recipe === undefined) return { sustain: [] };+ assertKeys(recipe, MOTION_KEYS);+ const enter = edge(recipe, "enter");+ const exit = edge(recipe, "exit");+ return {+ ...(enter === undefined ? {} : { enter }),+ sustain: sustain(recipe),+ ...(exit === undefined ? {} : { exit }),+ };+}++export function decodeMediaPresentation(recipe: SvsRecipe): MediaFramePresentation {+ const clip = oneOf(recipe, "clip", ["none", "frame", "rounded"] as const, "frame");+ const borderWidth = number(recipe, "border-width", 0);+ return {+ clip: clip === "rounded" ? { kind: "rounded", radiusPx: number(recipe, "radius", 0) } : { kind: clip },+ padding: padding(recipe),+ ...(borderWidth === 0 ? {} : { border: {+ widthPx: borderWidth,+ style: oneOf(recipe, "border-style", ["solid", "dashed", "dotted"] as const, "solid"),+ color: string(recipe, "border-color"),+ } }),+ shadows: shadows(recipe),+ };+}++export function decodeMediaItemSpec(+ recipe: SvsRecipe,+ input: Pick<MediaItemSpec, "id" | "projection" | "expansion"> & {+ readonly motion: MediaLifecycleMotion;+ readonly sourceAudio?: MediaItemSpec["sourceAudio"];+ },+): MediaItemSpec {+ assertKeys(recipe, [...FIT_KEYS, ...SAMPLE_KEYS, ...FRAME_KEYS]);+ return sealMediaItemSpec({+ contract: "svml.media-item-spec@1",+ id: input.id,+ projection: input.projection,+ expansion: input.expansion,+ presentation: decodeMediaPresentation(recipe),+ motion: input.motion,+ stackingOrder: number(recipe, "stack-order"),+ ...(input.sourceAudio === undefined ? {} : { sourceAudio: input.sourceAudio }),+ });+}++export function decodeMediaFramePaint(recipe: SvsRecipe, id: string): MediaPaintLayerSpec | undefined {+ const source = optionalString(recipe, "frame-paint");+ if (source === undefined || source === "transparent") return undefined;+ const adapted: SvsRecipe = { ...recipe, properties: { paint: source } };+ return sealMediaPaintLayerSpec({ contract: "svml.media-paint-layer-spec@1", id, paint: paint(adapted), opacity: 1 });+}++export function decodeMediaPaintSpec(recipe: SvsRecipe, id: string): MediaPaintLayerSpec {+ assertKeys(recipe, ["paint", "opacity"]);+ return sealMediaPaintLayerSpec({+ contract: "svml.media-paint-layer-spec@1", id, paint: paint(recipe), opacity: number(recipe, "opacity", 1),+ });+}++export function decodeMediaHandoffSpec(+ recipe: SvsRecipe,+ id: string,+ fromMemberId: string,+ toMemberId: string,+): MediaHandoffSpec {+ assertKeys(recipe, ["operator", "duration-frames", "boundary-ratio", "direction", "audio"]);+ const operator = oneOf(recipe, "operator", ["cut", "crossfade", "push", "wipe", "cover", "page-turn"] as const);+ const direction = optionalString(recipe, "direction");+ return sealMediaHandoffSpec({+ contract: "svml.media-handoff-spec@1", id, fromMemberId, toMemberId, operator,+ durationFrames: number(recipe, "duration-frames"),+ boundaryRatio: number(recipe, "boundary-ratio", 0.5),+ ...(direction === undefined ? {} : { direction: oneOf(recipe, "direction", ["left", "right", "up", "down"] as const) }),+ audio: oneOf(recipe, "audio", ["cut", "crossfade"] as const, "cut"),+ });+}++export function decodeMediaSequenceSpec(+ recipe: SvsRecipe,+ id: string,+ motion: MediaLifecycleMotion,+ handoffs: readonly MediaHandoffSpec[],+): MediaSequenceSpec {+ assertKeys(recipe, [...FIT_KEYS, ...SAMPLE_KEYS, ...FRAME_KEYS]);+ return sealMediaSequenceSpec({+ contract: "svml.media-sequence-spec@1", id,+ presentation: decodeMediaPresentation(recipe), motion,+ stackingOrder: number(recipe, "stack-order"), handoffs,+ });+}++export const mediaAppearanceKeys = { fit: FIT_KEYS, sample: SAMPLE_KEYS, frame: FRAME_KEYS, motion: MOTION_KEYS } as const;diff --git a/packages/media-track/src/component.ts b/packages/media-track/src/component.tsindex 253f27c7..21c05518 100644--- a/packages/media-track/src/component.ts+++ b/packages/media-track/src/component.ts@@ -1,20 +1,60 @@-import type { ComponentPackage } from "@narratage/component-kit";+import type { ComponentPackage, ProducerHandlerContext } from "@narratage/component-kit";+import type { CompositableSurfaceRef, SynchronizedMedia } from "@narratage/media";+import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";import type { ProgramSpace } from "@narratage/program-space";import { canonicalize } from "@narratage/protocol";import type { BlobRef, StoredValue } from "@narratage/protocol";-import type { ContentFit, IntrinsicExtent, SpatialFrame } from "@narratage/spatial";+import type { CompleteSemanticMap } from "@narratage/semantic-map";+import type { CanvasSpace, ContentFit, IntrinsicExtent, SpatialFrame, SpatialPath } from "@narratage/spatial";+import {+ appendMediaPaintLayer,+ appendStillMediaLayer,+ appendSurfaceMediaLayer,+ appendTimedMediaLayer,+ createMediaLayerSet,+} from "./layers.js";import { mediaTrackProducers, mediaTrackTypes } from "./manifest.js";import {- appendFullStillMediaItem,+ appendMediaSequenceUntilMoment,+ appendMediaSequenceUntilProgramEnd,+ appendMediaSequenceUntilSelection,+ appendMomentMediaItem,+ appendProgramMediaItem,+ appendSelectionMediaItem,+ bindMediaItemClipPath,+ bindMediaSequenceClipPath,assertMediaTrackProgram,createMediaTrackSet,finalizeMediaTrack,mediaTrackImplementationDigests,mediaTrackValidatorDigests,- renderMediaTrack,+ projectMediaAudioTrack,+ projectMediaVisualTrack,} from "./program.js";-import type { MediaStillItemSpec, MediaTrackHeader, MediaTrackProgram, MediaTrackSet } from "./types.js";+import {+ appendMediaSequenceMomentMember,+ appendMediaSequenceSelectionMember,+ createMediaSequenceMemberSet,+} from "./sequence.js";+import {+ appendMediaSound,+ createMediaSoundSet,+} from "./sounds.js";+import type {+ MediaItemSpec,+ MediaLayerSet,+ MediaPaintLayerSpec,+ MediaSampleLayerSpec,+ MediaSoundSet,+ MediaSoundSpec,+ MediaSequenceMemberSet,+ MediaSequenceMemberSpec,+ MediaSequenceSpec,+ MediaTrackHeader,+ MediaTrackProgram,+ MediaTrackSet,+} from "./types.js";function inline<T>(value: StoredValue | undefined, label: string): T {if (value?.kind !== "inline") throw new Error(`${label} must be inline.`);@@ -26,27 +66,155 @@ function blob(value: StoredValue | undefined): BlobRef {}const output = (value: unknown) => ({ kind: "inline" as const, value: canonicalize(value) });+function itemInputs(inputs: ProducerHandlerContext["inputs"]) {+ return {+ set: inline<MediaTrackSet>(inputs.set?.value, "MediaTrackSet"),+ header: inline<MediaTrackHeader>(inputs.header?.value, "MediaTrackHeader"),+ space: inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ canvas: inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"),+ layers: inline<MediaLayerSet>(inputs.layers?.value, "MediaLayerSet"),+ frame: inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),+ spec: inline<MediaItemSpec>(inputs.spec?.value, "MediaItemSpec"),+ sounds: inline<MediaSoundSet>(inputs.sounds?.value, "MediaSoundSet"),+ };+}++function memberInputs(inputs: ProducerHandlerContext["inputs"]) {+ return {+ members: inline<MediaSequenceMemberSet>(inputs.members?.value, "MediaSequenceMemberSet"),+ layers: inline<MediaLayerSet>(inputs.layers?.value, "MediaLayerSet"),+ spec: inline<MediaSequenceMemberSpec>(inputs.spec?.value, "MediaSequenceMemberSpec"),+ map: inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ space: inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ };+}++function sequenceInputs(inputs: ProducerHandlerContext["inputs"]) {+ return {+ set: inline<MediaTrackSet>(inputs.set?.value, "MediaTrackSet"),+ header: inline<MediaTrackHeader>(inputs.header?.value, "MediaTrackHeader"),+ space: inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ canvas: inline<CanvasSpace>(inputs.canvas?.value, "CanvasSpace"),+ members: inline<MediaSequenceMemberSet>(inputs.members?.value, "MediaSequenceMemberSet"),+ frame: inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),+ spec: inline<MediaSequenceSpec>(inputs.spec?.value, "MediaSequenceSpec"),+ sounds: inline<MediaSoundSet>(inputs.sounds?.value, "MediaSoundSet"),+ };+}+export const mediaTrackComponent = {name: "@narratage/media-track",producers: [+ { producer: mediaTrackProducers.createLayers, implementationDigest: mediaTrackImplementationDigests.createLayers, handler: () => ({ outputs: { layers: output(createMediaLayerSet()) }, needs: {} }) },+ { producer: mediaTrackProducers.appendPaintLayer, implementationDigest: mediaTrackImplementationDigests.appendPaintLayer, handler: ({ inputs }) => ({ outputs: { layers: output(appendMediaPaintLayer(+ inline<MediaLayerSet>(inputs.layers?.value, "MediaLayerSet"),+ inline<MediaPaintLayerSpec>(inputs.spec?.value, "MediaPaintLayerSpec"),+ )) }, needs: {} }) },+ { producer: mediaTrackProducers.appendStillLayer, implementationDigest: mediaTrackImplementationDigests.appendStillLayer, handler: ({ inputs }) => ({ outputs: { layers: output(appendStillMediaLayer(+ inline<MediaLayerSet>(inputs.layers?.value, "MediaLayerSet"), blob(inputs.source?.value),+ inline<IntrinsicExtent>(inputs.extent?.value, "IntrinsicExtent"), inline<ContentFit>(inputs.fit?.value, "ContentFit"),+ inline<MediaSampleLayerSpec>(inputs.spec?.value, "MediaSampleLayerSpec"),+ )) }, needs: {} }) },+ { producer: mediaTrackProducers.appendTimedLayer, implementationDigest: mediaTrackImplementationDigests.appendTimedLayer, handler: ({ inputs }) => ({ outputs: { layers: output(appendTimedMediaLayer(+ inline<MediaLayerSet>(inputs.layers?.value, "MediaLayerSet"), inline<SynchronizedMedia>(inputs.source?.value, "SynchronizedMedia"),+ inline<ContentFit>(inputs.fit?.value, "ContentFit"), inline<MediaSampleLayerSpec>(inputs.spec?.value, "MediaSampleLayerSpec"),+ )) }, needs: {} }) },+ { producer: mediaTrackProducers.appendSurfaceLayer, implementationDigest: mediaTrackImplementationDigests.appendSurfaceLayer, handler: ({ inputs }) => ({ outputs: { layers: output(appendSurfaceMediaLayer(+ inline<MediaLayerSet>(inputs.layers?.value, "MediaLayerSet"), inline<CompositableSurfaceRef>(inputs.source?.value, "CompositableSurfaceRef"),+ inline<ContentFit>(inputs.fit?.value, "ContentFit"), inline<MediaSampleLayerSpec>(inputs.spec?.value, "MediaSampleLayerSpec"),+ )) }, needs: {} }) },+ { producer: mediaTrackProducers.createSounds, implementationDigest: mediaTrackImplementationDigests.createSounds, handler: () => ({ outputs: { sounds: output(createMediaSoundSet()) }, needs: {} }) },+ { producer: mediaTrackProducers.appendSound, implementationDigest: mediaTrackImplementationDigests.appendSound, handler: ({ inputs }) => ({ outputs: { sounds: output(appendMediaSound(+ inline<MediaSoundSet>(inputs.sounds?.value, "MediaSoundSet"),+ inline<SynchronizedMedia>(inputs.source?.value, "SynchronizedMedia"),+ inline<MediaSoundSpec>(inputs.spec?.value, "MediaSoundSpec"),+ )) }, needs: {} }) },{ producer: mediaTrackProducers.createSet, implementationDigest: mediaTrackImplementationDigests.createSet, handler: () => ({ outputs: { set: output(createMediaTrackSet()) }, needs: {} }) },- { producer: mediaTrackProducers.appendFullStill, implementationDigest: mediaTrackImplementationDigests.appendFullStill, handler: ({ inputs }) => ({ outputs: { set: output(appendFullStillMediaItem(- inline<MediaTrackSet>(inputs.set?.value, "MediaTrackSet"),- inline<MediaTrackHeader>(inputs.header?.value, "MediaTrackHeader"),- inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),- blob(inputs.source?.value),- inline<IntrinsicExtent>(inputs.extent?.value, "IntrinsicExtent"),- inline<SpatialFrame>(inputs.frame?.value, "SpatialFrame"),- inline<ContentFit>(inputs.fit?.value, "ContentFit"),- inline<MediaStillItemSpec>(inputs.spec?.value, "MediaStillItemSpec"),+ { producer: mediaTrackProducers.appendProgramItem, implementationDigest: mediaTrackImplementationDigests.appendProgramItem, handler: ({ inputs }) => {+ const value = itemInputs(inputs);+ return { outputs: { set: output(appendProgramMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame, value.spec, value.sounds)) }, needs: {} };+ } },+ { producer: mediaTrackProducers.appendSelectionItem, implementationDigest: mediaTrackImplementationDigests.appendSelectionItem, handler: ({ inputs }) => {+ const value = itemInputs(inputs);+ return { outputs: { set: output(appendSelectionMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), value.spec, value.sounds)) }, needs: {} };+ } },+ { producer: mediaTrackProducers.appendMomentItem, implementationDigest: mediaTrackImplementationDigests.appendMomentItem, handler: ({ inputs }) => {+ const value = itemInputs(inputs);+ return { outputs: { set: output(appendMomentMediaItem(value.set, value.header, value.space, value.canvas, value.layers, value.frame,+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMomentRef"), value.spec, value.sounds)) }, needs: {} };+ } },+ { producer: mediaTrackProducers.bindItemClipPath, implementationDigest: mediaTrackImplementationDigests.bindItemClipPath, handler: ({ inputs }) => ({ outputs: { spec: output(bindMediaItemClipPath(+ inline<MediaItemSpec>(inputs.spec?.value, "MediaItemSpec"),+ inline<SpatialPath>(inputs.path?.value, "SpatialPath"),)) }, needs: {} }) },- { producer: mediaTrackProducers.finalize, implementationDigest: mediaTrackImplementationDigests.finalize, handler: ({ inputs }) => ({ outputs: { program: output(finalizeMediaTrack(- inline<MediaTrackSet>(inputs.set?.value, "MediaTrackSet"),- inline<MediaTrackHeader>(inputs.header?.value, "MediaTrackHeader"),+ { producer: mediaTrackProducers.bindSequenceClipPath, implementationDigest: mediaTrackImplementationDigests.bindSequenceClipPath, handler: ({ inputs }) => ({ outputs: { spec: output(bindMediaSequenceClipPath(+ inline<MediaSequenceSpec>(inputs.spec?.value, "MediaSequenceSpec"),+ inline<SpatialPath>(inputs.path?.value, "SpatialPath"),)) }, needs: {} }) },- { producer: mediaTrackProducers.render, implementationDigest: mediaTrackImplementationDigests.render, handler: ({ inputs }) => ({ outputs: { track: output(renderMediaTrack(+ { producer: mediaTrackProducers.createMembers, implementationDigest: mediaTrackImplementationDigests.createMembers, handler: () => ({ outputs: { members: output(createMediaSequenceMemberSet()) }, needs: {} }) },+ { producer: mediaTrackProducers.appendMomentMember, implementationDigest: mediaTrackImplementationDigests.appendMomentMember, handler: ({ inputs }) => {+ const value = memberInputs(inputs);+ return { outputs: { members: output(appendMediaSequenceMomentMember(+ value.members, value.layers, value.spec, value.map,+ inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMomentRef"), value.space,+ )) }, needs: {} };+ } },+ { producer: mediaTrackProducers.appendSelectionStartMember, implementationDigest: mediaTrackImplementationDigests.appendSelectionStartMember, handler: ({ inputs }) => {+ const value = memberInputs(inputs);+ return { outputs: { members: output(appendMediaSequenceSelectionMember(+ value.members, value.layers, value.spec, value.map,+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), value.space, "start",+ )) }, needs: {} };+ } },+ { producer: mediaTrackProducers.appendSelectionEndMember, implementationDigest: mediaTrackImplementationDigests.appendSelectionEndMember, handler: ({ inputs }) => {+ const value = memberInputs(inputs);+ return { outputs: { members: output(appendMediaSequenceSelectionMember(+ value.members, value.layers, value.spec, value.map,+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), value.space, "end",+ )) }, needs: {} };+ } },+ { producer: mediaTrackProducers.appendSequenceProgramEnd, implementationDigest: mediaTrackImplementationDigests.appendSequenceProgramEnd, handler: ({ inputs }) => {+ const value = sequenceInputs(inputs);+ return { outputs: { set: output(appendMediaSequenceUntilProgramEnd(+ value.set, value.header, value.space, value.canvas, value.members, value.frame, value.spec, value.sounds,+ )) }, needs: {} };+ } },+ { producer: mediaTrackProducers.appendSequenceUntilMoment, implementationDigest: mediaTrackImplementationDigests.appendSequenceUntilMoment, handler: ({ inputs }) => {+ const value = sequenceInputs(inputs);+ return { outputs: { set: output(appendMediaSequenceUntilMoment(+ value.set, value.header, value.space, value.canvas, value.members, value.frame, value.spec, value.sounds,+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<NarrativeMomentRef>(inputs.moment?.value, "NarrativeMomentRef"),+ )) }, needs: {} };+ } },+ { producer: mediaTrackProducers.appendSequenceUntilSelectionStart, implementationDigest: mediaTrackImplementationDigests.appendSequenceUntilSelectionStart, handler: ({ inputs }) => {+ const value = sequenceInputs(inputs);+ return { outputs: { set: output(appendMediaSequenceUntilSelection(+ value.set, value.header, value.space, value.canvas, value.members, value.frame, value.spec, value.sounds,+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), "start",+ )) }, needs: {} };+ } },+ { producer: mediaTrackProducers.appendSequenceUntilSelectionEnd, implementationDigest: mediaTrackImplementationDigests.appendSequenceUntilSelectionEnd, handler: ({ inputs }) => {+ const value = sequenceInputs(inputs);+ return { outputs: { set: output(appendMediaSequenceUntilSelection(+ value.set, value.header, value.space, value.canvas, value.members, value.frame, value.spec, value.sounds,+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelectionRef"), "end",+ )) }, needs: {} };+ } },+ { producer: mediaTrackProducers.finalize, implementationDigest: mediaTrackImplementationDigests.finalize, handler: ({ inputs }) => ({ outputs: { program: output(finalizeMediaTrack(+ inline<MediaTrackSet>(inputs.set?.value, "MediaTrackSet"), inline<MediaTrackHeader>(inputs.header?.value, "MediaTrackHeader"),inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),- inline<MediaTrackProgram>(inputs.program?.value, "MediaTrackProgram"),+ )) }, needs: {} }) },+ { producer: mediaTrackProducers.projectVisual, implementationDigest: mediaTrackImplementationDigests.projectVisual, handler: ({ inputs }) => ({ outputs: { track: output(projectMediaVisualTrack(+ inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<MediaTrackProgram>(inputs.program?.value, "MediaTrackProgram"),+ )) }, needs: {} }) },+ { producer: mediaTrackProducers.projectAudio, implementationDigest: mediaTrackImplementationDigests.projectAudio, handler: ({ inputs }) => ({ outputs: { track: output(projectMediaAudioTrack(+ inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<MediaTrackProgram>(inputs.program?.value, "MediaTrackProgram"),)) }, needs: {} }) },],validators: [{diff --git a/packages/media-track/src/fragment.ts b/packages/media-track/src/fragment.tsindex c7761405..1451ef1c 100644--- a/packages/media-track/src/fragment.ts+++ b/packages/media-track/src/fragment.ts@@ -9,35 +9,47 @@ import { mediaTrackProducers, mediaTrackTypes } from "./manifest.js";const input = (name: string) => ({ kind: "fragment-input" as const, name });const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });+/** Minimal complete graph witness; richer Surfaces generate the same primitive operations dynamically. */export const stillMediaTrackFragment = sealGraphFragment({name: "@narratage/media-track/one-full-still@1",inputs: [{ name: "header", type: mediaTrackTypes.header },{ name: "space", type: programSpaceTypes.programSpace },+ { name: "canvas", type: spatialTypes.canvas },{ name: "source", type: artifactTypes.blob },{ name: "extent", type: spatialTypes.extent },{ name: "frame", type: spatialTypes.frame },{ name: "fit", type: spatialTypes.fit },- { name: "spec", type: mediaTrackTypes.stillItemSpec },+ { name: "sample-spec", type: mediaTrackTypes.sampleLayerSpec },+ { name: "item-spec", type: mediaTrackTypes.itemSpec },],operations: [+ { id: "layers", producer: mediaTrackProducers.createLayers, inputs: {}, result: { kind: "output", name: "layers" } },+ { id: "sample", producer: mediaTrackProducers.appendStillLayer, inputs: {+ layers: operation("layers"), source: input("source"), extent: input("extent"), fit: input("fit"), spec: input("sample-spec"),+ }, result: { kind: "output", name: "layers" } },{ id: "set", producer: mediaTrackProducers.createSet, inputs: {}, result: { kind: "output", name: "set" } },- { id: "append", producer: mediaTrackProducers.appendFullStill, inputs: {- set: operation("set"), header: input("header"), space: input("space"), source: input("source"),- extent: input("extent"), frame: input("frame"), fit: input("fit"), spec: input("spec"),+ { id: "sounds", producer: mediaTrackProducers.createSounds, inputs: {}, result: { kind: "output", name: "sounds" } },+ { id: "append", producer: mediaTrackProducers.appendProgramItem, inputs: {+ set: operation("set"), header: input("header"), space: input("space"), canvas: input("canvas"), layers: operation("sample"),+ frame: input("frame"), spec: input("item-spec"), sounds: operation("sounds"),}, result: { kind: "output", name: "set" } },- { id: "finalize", producer: mediaTrackProducers.finalize, inputs: { set: operation("append"), header: input("header") }, result: { kind: "output", name: "program" } },- { id: "render", producer: mediaTrackProducers.render, inputs: { space: input("space"), program: operation("finalize") }, result: { kind: "output", name: "track" } },+ { id: "finalize", producer: mediaTrackProducers.finalize, inputs: {+ set: operation("append"), header: input("header"), space: input("space"),+ }, result: { kind: "output", name: "program" } },+ { id: "visual", producer: mediaTrackProducers.projectVisual, inputs: {+ space: input("space"), program: operation("finalize"),+ }, result: { kind: "output", name: "track" } },],exports: [- { name: "program", type: mediaTrackTypes.program, root: operation("finalize"), semanticInputs: ["header", "space", "source", "extent", "frame", "fit", "spec"], fidelity: "exact" },- { name: "track", type: compositionTypes.visualTrack, root: operation("render"), semanticInputs: ["header", "space", "source", "extent", "frame", "fit", "spec"], fidelity: "exact" },+ { name: "program", type: mediaTrackTypes.program, root: operation("finalize"), semanticInputs: ["header", "space", "canvas", "source", "extent", "frame", "fit", "sample-spec", "item-spec"], fidelity: "exact" },+ { name: "track", type: compositionTypes.visualTrack, root: operation("visual"), semanticInputs: ["header", "space", "canvas", "source", "extent", "frame", "fit", "sample-spec", "item-spec"], fidelity: "exact" },],});export const renderMediaTrackFragment = sealGraphFragment({- name: "@narratage/media-track/render@1",+ name: "@narratage/media-track/project-visual@1",inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: mediaTrackTypes.program }],- operations: [{ id: "render", producer: mediaTrackProducers.render, inputs: { space: input("space"), program: input("program") }, result: { kind: "output", name: "track" } }],- exports: [{ name: "track", type: compositionTypes.visualTrack, root: operation("render"), semanticInputs: ["space", "program"], fidelity: "exact" }],+ operations: [{ id: "visual", producer: mediaTrackProducers.projectVisual, inputs: { space: input("space"), program: input("program") }, result: { kind: "output", name: "track" } }],+ exports: [{ name: "track", type: compositionTypes.visualTrack, root: operation("visual"), semanticInputs: ["space", "program"], fidelity: "exact" }],});diff --git a/packages/media-track/src/index.ts b/packages/media-track/src/index.tsindex 7eee9f42..dff41d96 100644--- a/packages/media-track/src/index.ts+++ b/packages/media-track/src/index.ts@@ -1,20 +1,72 @@export { mediaTrackComponent } from "./component.js";export { renderMediaTrackFragment, stillMediaTrackFragment } from "./fragment.js";export {- mediaStillItemSpecSchema,+ appendMediaPaintLayer,+ appendStillMediaLayer,+ appendSurfaceMediaLayer,+ appendTimedMediaLayer,+ assertMediaLayerSet,+ assertMediaPaintLayerSpec,+ assertMediaSampleLayerSpec,+ assertMediaVisualOccupancy,+ assertMediaVisualSource,+ createMediaLayerSet,+ sealMediaPaintLayerSpec,+ sealMediaSampleLayerSpec,+} from "./layers.js";+export { lowerMediaItemElements } from "./lower.js";+export {+ mediaItemSpecSchema,+ mediaLayerSetSchema,+ mediaHandoffSpecSchema,+ mediaPaintLayerSpecSchema,+ mediaSampleLayerSpecSchema,+ mediaSoundSetSchema,+ mediaSoundSpecSchema,+ mediaSequenceMemberSetSchema,+ mediaSequenceMemberSpecSchema,+ mediaSequenceSpecSchema,mediaTrackDependency,mediaTrackHeaderSchema,mediaTrackManifest,mediaTrackManifestDigest,mediaTrackModuleRef,mediaTrackProducers,+ mediaTrackSurfaceImplementationDigest,mediaTrackProgramSchema,mediaTrackSetSchema,mediaTrackTypes,} from "./manifest.js";export {- appendFullStillMediaItem,- assertMediaStillItemSpec,+ assertMediaEdgeMotion,+ assertMediaLifecycleMotion,+ assertMediaSustainMotion,+ lifecycleAnimation,+ samplingAnimation,+ lifecycleAnimationWindow,+ resolveMediaLifecycleMotion,+ sustainAnimation,+ sustainAnimationWindow,+} from "./motion.js";+export { assertMediaFramePresentation } from "./presentation.js";+export {+ appendMediaSound,+ assertMediaSoundSet,+ assertMediaSoundSpec,+ createMediaSoundSet,+ sealMediaSoundSpec,+} from "./sounds.js";+export {+ appendMomentMediaItem,+ appendMediaSequence,+ appendMediaSequenceUntilMoment,+ appendMediaSequenceUntilProgramEnd,+ appendMediaSequenceUntilSelection,+ appendProgramMediaItem,+ appendSelectionMediaItem,+ bindMediaItemClipPath,+ bindMediaSequenceClipPath,+ assertMediaItemSpec,assertMediaTrackHeader,assertMediaTrackProgram,assertMediaTrackProgramIdentity,@@ -23,9 +75,31 @@ export {finalizeMediaTrack,mediaTrackImplementationDigests,mediaTrackValidatorDigests,+ projectMediaAudioTrack,+ projectMediaVisualTrack,renderMediaTrack,- sealMediaStillItemSpec,+ sealMediaItemSpec,sealMediaTrackHeader,sealMediaTrackProgram,} from "./program.js";+export { resolveVisualSampling } from "./sampling.js";+export { lowerRestrictedSpeechVisualPresents } from "./restricted-speech.js";+export type { RestrictedSpeechVisualClip } from "./restricted-speech.js";+export { decodeMediaTrackSurface } from "./surface.js";+export {+ appendMediaSequenceMember,+ appendMediaSequenceMomentMember,+ appendMediaSequenceSelectionMember,+ assertMediaHandoffSpec,+ assertMediaSequenceMemberSet,+ assertMediaSequenceMemberSpec,+ assertMediaSequenceSpec,+ createMediaSequenceMemberSet,+ lowerMediaSequencePresents,+ resolveMediaSequence,+ sealMediaHandoffSpec,+ sealMediaSequenceMemberSpec,+ sealMediaSequenceSpec,+ sequenceMemberHandoffAnimation,+} from "./sequence.js";export type * from "./types.js";diff --git a/packages/media-track/src/layers.ts b/packages/media-track/src/layers.tsnew file mode 100644index 00000000..c797172a--- /dev/null+++ b/packages/media-track/src/layers.ts@@ -0,0 +1,310 @@+import {+ assertCompositableSurfaceRef,+ verifySynchronizedMedia,+} from "@narratage/media";+import type { CompositableSurfaceRef, SynchronizedMedia } from "@narratage/media";+import { canonicalize, isDigest } from "@narratage/protocol";+import type { BlobRef } from "@narratage/protocol";+import {+ assertContentFit,+ assertIntrinsicExtent,+} from "@narratage/spatial";+import type { ContentFit, IntrinsicExtent } from "@narratage/spatial";++import type {+ MediaGradientStop,+ MediaLayerProgram,+ MediaLayerSet,+ MediaPaint,+ MediaPaintLayerSpec,+ MediaSampleLayerProgram,+ MediaSampleLayerSpec,+ MediaSamplingMotion,+ MediaVisualOccupancy,+ MediaVisualSource,+ MediaVisualTrim,+} from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++export function assertMediaIdentity(value: string, label: string): void {+ assert(/^[A-Za-z][A-Za-z0-9_.:#-]{0,191}$/u.test(value), `${label} is invalid.`);+}++function assertBlob(value: BlobRef, label: string, prefix?: string): void {+ assert(value.kind === "blob" && isDigest(value.digest)+ && Number.isSafeInteger(value.size) && value.size >= 0 && value.mediaType.length > 0,+ `${label} is not a valid BlobArtifact.`);+ if (prefix !== undefined) assert(value.mediaType.startsWith(prefix), `${label} must be ${prefix} bytes.`);+}++function finite(value: number, label: string): void {+ assert(Number.isFinite(value), `${label} must be finite.`);+}++function unit(value: number, label: string): void {+ finite(value, label);+ assert(value >= 0 && value <= 1, `${label} must be inside [0, 1].`);+}++function assertColor(value: string, label: string): void {+ assert(typeof value === "string" && value.trim().length > 0 && !/[;{}]/u.test(value), `${label} is invalid.`);+}++function assertStops(stops: readonly MediaGradientStop[], label: string): void {+ assert(stops.length >= 2, `${label} needs at least two stops.`);+ let previous = -1;+ for (const [index, stop] of stops.entries()) {+ unit(stop.offset, `${label}.${index}.offset`);+ assert(stop.offset > previous, `${label} offsets must be strictly increasing.`);+ assertColor(stop.color, `${label}.${index}.color`);+ previous = stop.offset;+ }+}++export function assertMediaPaint(value: MediaPaint, label: string): void {+ if (value.kind === "solid") {+ assertColor(value.color, `${label}.color`);+ return;+ }+ assertStops(value.stops, `${label}.stops`);+ if (value.kind === "linear-gradient") {+ finite(value.angleDeg, `${label}.angleDeg`);+ return;+ }+ assert(value.kind === "radial-gradient", `${label}.kind is unsupported.`);+ unit(value.center.x, `${label}.center.x`);+ unit(value.center.y, `${label}.center.y`);+}++export function assertMediaVisualTrim(value: MediaVisualTrim, sourceFrames: number, label: string): void {+ assert(Number.isSafeInteger(value.startFrame) && Number.isSafeInteger(value.endFrameExclusive)+ && value.startFrame >= 0 && value.endFrameExclusive > value.startFrame+ && value.endFrameExclusive <= sourceFrames, `${label} is outside its source.`);+}++export function assertMediaVisualOccupancy(value: MediaVisualOccupancy, label: string): void {+ if (value.mode === "stretch") return;+ assert(["once", "hold", "loop"].includes(value.mode)+ && (value.align === "start" || value.align === "end"), `${label} is invalid.`);+}++export function assertMediaSamplingMotion(value: MediaSamplingMotion, label: string): void {+ assert(value.keyframes.length >= 2, `${label} needs at least two keyframes.`);+ let previous = -1;+ for (const [index, keyframe] of value.keyframes.entries()) {+ finite(keyframe.atProgress, `${label}.${index}.atProgress`);+ assert(keyframe.atProgress >= 0 && keyframe.atProgress <= 1 && keyframe.atProgress > previous,+ `${label}.${index}.atProgress is invalid.`);+ finite(keyframe.zoom, `${label}.${index}.zoom`);+ assert(keyframe.zoom > 0 && keyframe.zoom <= 100, `${label}.${index}.zoom is invalid.`);+ finite(keyframe.offsetX, `${label}.${index}.offsetX`);+ finite(keyframe.offsetY, `${label}.${index}.offsetY`);+ finite(keyframe.rotationDeg, `${label}.${index}.rotationDeg`);+ if (keyframe.easing !== undefined) {+ assert(["linear", "ease-in", "ease-out", "ease-in-out"].includes(keyframe.easing),+ `${label}.${index}.easing is invalid.`);+ }+ previous = keyframe.atProgress;+ }+ assert(value.keyframes[0]!.atProgress === 0 && value.keyframes.at(-1)!.atProgress === 1,+ `${label} must cover normalized progress [0, 1].`);+}++export function assertMediaVisualSource(value: MediaVisualSource, label: string): void {+ assertIntrinsicExtent(value.extent);+ if (value.kind === "still") {+ assertBlob(value.artifact, `${label}.artifact`, "image/");+ return;+ }+ if (value.kind === "surface") {+ assertCompositableSurfaceRef(value.surface, `${label}.surface`);+ assert(value.extent.widthPx === value.surface.width && value.extent.heightPx === value.surface.height,+ `${label} Surface extent disagrees with its intrinsic dimensions.`);+ return;+ }+ assert(value.kind === "timed", `${label}.kind is unsupported.`);+ assertBlob(value.artifact, `${label}.artifact`, "video/");+ assert(Number.isSafeInteger(value.frameRate.numerator) && value.frameRate.numerator > 0+ && Number.isSafeInteger(value.frameRate.denominator) && value.frameRate.denominator > 0+ && Number.isSafeInteger(value.frameCount) && value.frameCount > 0, `${label} timing is invalid.`);+ if (value.audio !== undefined) {+ assertBlob(value.audio.artifact, `${label}.audio.artifact`, "audio/");+ assert(Number.isSafeInteger(value.audio.sampleFrames) && value.audio.sampleFrames > 0,+ `${label}.audio.sampleFrames is invalid.`);+ }+}++export function assertMediaSampleLayerSpec(value: MediaSampleLayerSpec): void {+ assert(value.contract === "svml.media-sample-layer-spec@1", "Unsupported MediaSampleLayerSpec contract.");+ assertMediaIdentity(value.id, "MediaSampleLayerSpec.id");+ unit(value.appearance.opacity, "MediaSampleLayerSpec.appearance.opacity");+ const filter = value.appearance.filter;+ finite(filter.blurPx, "MediaSampleLayerSpec.filter.blurPx");+ finite(filter.brightness, "MediaSampleLayerSpec.filter.brightness");+ finite(filter.contrast, "MediaSampleLayerSpec.filter.contrast");+ finite(filter.saturation, "MediaSampleLayerSpec.filter.saturation");+ assert(filter.blurPx >= 0 && filter.brightness >= 0 && filter.contrast >= 0 && filter.saturation >= 0,+ "MediaSampleLayerSpec filter values must be non-negative.");+ if (value.occupancy !== undefined) assertMediaVisualOccupancy(value.occupancy, "MediaSampleLayerSpec.occupancy");+ if (value.samplingMotion !== undefined) assertMediaSamplingMotion(value.samplingMotion, "MediaSampleLayerSpec.samplingMotion");+}++export function sealMediaSampleLayerSpec(value: MediaSampleLayerSpec): MediaSampleLayerSpec {+ assertMediaSampleLayerSpec(value);+ return canonicalize(value) as unknown as MediaSampleLayerSpec;+}++export function assertMediaPaintLayerSpec(value: MediaPaintLayerSpec): void {+ assert(value.contract === "svml.media-paint-layer-spec@1", "Unsupported MediaPaintLayerSpec contract.");+ assertMediaIdentity(value.id, "MediaPaintLayerSpec.id");+ assertMediaPaint(value.paint, "MediaPaintLayerSpec.paint");+ unit(value.opacity, "MediaPaintLayerSpec.opacity");+}++export function sealMediaPaintLayerSpec(value: MediaPaintLayerSpec): MediaPaintLayerSpec {+ assertMediaPaintLayerSpec(value);+ return canonicalize(value) as unknown as MediaPaintLayerSpec;+}++export function createMediaLayerSet(): MediaLayerSet {+ return { contract: "svml.media-layer-set@1", layers: [] };+}++export function assertMediaLayerSet(value: MediaLayerSet): void {+ assert(value.contract === "svml.media-layer-set@1" && Array.isArray(value.layers), "MediaLayerSet is invalid.");+ const ids = new Set<string>();+ for (const layer of value.layers) {+ assertMediaIdentity(layer.id, "Media layer id");+ assert(!ids.has(layer.id), `MediaLayerSet repeats layer ${layer.id}.`);+ ids.add(layer.id);+ if (layer.kind === "paint") {+ assertMediaPaint(layer.paint, `Media layer ${layer.id}.paint`);+ unit(layer.opacity, `Media layer ${layer.id}.opacity`);+ continue;+ }+ assert(layer.kind === "sample", `Media layer ${layer.id} kind is unsupported.`);+ assertMediaVisualSource(layer.source, `Media layer ${layer.id}.source`);+ assertContentFit(layer.fit);+ if (layer.source.kind === "still") {+ assert(layer.trim === undefined && layer.occupancy === undefined,+ `Still Media layer ${layer.id} cannot have trim or occupancy.`);+ } else {+ const frames = layer.source.kind === "timed"+ ? layer.source.frameCount+ : layer.source.surface.timing.kind === "frames" ? layer.source.surface.timing.frameCount : undefined;+ if (frames === undefined) {+ assert(layer.trim === undefined && layer.occupancy === undefined,+ `Still Surface layer ${layer.id} cannot have trim or occupancy.`);+ } else {+ if (layer.trim !== undefined) assertMediaVisualTrim(layer.trim, frames, `Media layer ${layer.id}.trim`);+ assert(layer.occupancy !== undefined, `Timed Media layer ${layer.id} requires occupancy.`);+ assertMediaVisualOccupancy(layer.occupancy, `Media layer ${layer.id}.occupancy`);+ }+ }+ assertMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1",+ id: layer.id,+ ...(layer.trim === undefined ? {} : { trim: layer.trim }),+ ...(layer.occupancy === undefined ? {} : { occupancy: layer.occupancy }),+ appearance: layer.appearance,+ ...(layer.samplingMotion === undefined ? {} : { samplingMotion: layer.samplingMotion }),+ });+ }+}++function append(set: MediaLayerSet, layer: MediaLayerProgram): MediaLayerSet {+ assertMediaLayerSet(set);+ assert(!set.layers.some((item) => item.id === layer.id), `MediaLayerSet already contains ${layer.id}.`);+ const result = { contract: "svml.media-layer-set@1" as const, layers: [...set.layers, layer] };+ assertMediaLayerSet(result);+ return canonicalize(result) as unknown as MediaLayerSet;+}++export function appendMediaPaintLayer(set: MediaLayerSet, spec: MediaPaintLayerSpec): MediaLayerSet {+ assertMediaPaintLayerSpec(spec);+ return append(set, { id: spec.id, kind: "paint", paint: structuredClone(spec.paint), opacity: spec.opacity });+}++function sampleLayer(source: MediaVisualSource, fit: ContentFit, spec: MediaSampleLayerSpec): MediaSampleLayerProgram {+ assertMediaVisualSource(source, "Media sample source");+ assertContentFit(fit);+ assertMediaSampleLayerSpec(spec);+ const layer: MediaSampleLayerProgram = {+ id: spec.id,+ kind: "sample",+ source: structuredClone(source),+ fit: structuredClone(fit),+ ...(spec.trim === undefined ? {} : { trim: { ...spec.trim } }),+ ...(spec.occupancy === undefined ? {} : { occupancy: structuredClone(spec.occupancy) }),+ appearance: structuredClone(spec.appearance),+ ...(spec.samplingMotion === undefined ? {} : { samplingMotion: structuredClone(spec.samplingMotion) }),+ };+ return layer;+}++export function appendStillMediaLayer(+ set: MediaLayerSet,+ source: BlobRef,+ extent: IntrinsicExtent,+ fit: ContentFit,+ spec: MediaSampleLayerSpec,+): MediaLayerSet {+ assertBlob(source, "Still Media source", "image/");+ assertIntrinsicExtent(extent);+ assert(spec.trim === undefined && spec.occupancy === undefined,+ "Still Media source cannot have trim or occupancy.");+ return append(set, sampleLayer({ kind: "still", artifact: structuredClone(source), extent: { ...extent } }, fit, spec));+}++export function appendTimedMediaLayer(+ set: MediaLayerSet,+ media: SynchronizedMedia,+ fit: ContentFit,+ spec: MediaSampleLayerSpec,+): MediaLayerSet {+ verifySynchronizedMedia(media);+ assert(media.visual !== undefined, "Timed Media source has no normalized visual member.");+ assert(spec.occupancy !== undefined, "Timed Media source requires explicit occupancy.");+ const source: MediaVisualSource = {+ kind: "timed",+ artifact: structuredClone(media.visual.artifact),+ extent: {+ contract: "svml.intrinsic-extent@1",+ widthPx: media.visual.width,+ heightPx: media.visual.height,+ },+ frameRate: { ...media.visual.frameRate },+ frameCount: media.visual.frameCount,+ ...(media.audio === undefined ? {} : { audio: {+ artifact: structuredClone(media.audio.artifact),+ sampleFrames: media.audio.sampleFrames,+ } }),+ };+ return append(set, sampleLayer(source, fit, spec));+}++export function appendSurfaceMediaLayer(+ set: MediaLayerSet,+ surface: CompositableSurfaceRef,+ fit: ContentFit,+ spec: MediaSampleLayerSpec,+): MediaLayerSet {+ assertCompositableSurfaceRef(surface);+ let effectiveSpec = spec;+ if (surface.timing.kind === "frames") {+ effectiveSpec = spec.occupancy === undefined+ ? sealMediaSampleLayerSpec({ ...spec, occupancy: { mode: "once", align: "start" } })+ : spec;+ } else {+ assert(spec.trim === undefined && spec.occupancy === undefined,+ "Still Media Surface cannot have trim or occupancy.");+ }+ return append(set, sampleLayer({+ kind: "surface",+ surface: structuredClone(surface),+ extent: { contract: "svml.intrinsic-extent@1", widthPx: surface.width, heightPx: surface.height },+ }, fit, effectiveSpec));+}diff --git a/packages/media-track/src/lower.ts b/packages/media-track/src/lower.tsnew file mode 100644index 00000000..a06ed02b--- /dev/null+++ b/packages/media-track/src/lower.ts@@ -0,0 +1,286 @@+import type {+ VisualAnimation,+ VisualElement,+ VisualStyleDeclaration,+} from "@narratage/composition";+import type { ProgramSpace } from "@narratage/program-space";+import { fitContent } from "@narratage/spatial";+import type { SpatialFrame, SpatialPath } from "@narratage/spatial";++import { lifecycleAnimation, samplingAnimation, sustainAnimation } from "./motion.js";+import { resolveVisualSampling } from "./sampling.js";+import type {+ MediaFramePresentation,+ MediaGradientStop,+ MediaItemProgram,+ MediaLayerProgram,+ MediaPaint,+ MediaSampleLayerProgram,+} from "./types.js";++function px(value: number): string { return `${value}px`; }++function box(id: string, order: number, parent: string | undefined, animation?: VisualAnimation): VisualElement {+ return {+ id,+ ...(parent === undefined ? {} : { parent }),+ order,+ kind: "box",+ style: [+ { name: "height", value: "100%" },+ { name: "left", value: 0 },+ { name: "position", value: "absolute" },+ { name: "top", value: 0 },+ { name: "transform-origin", value: "center center" },+ { name: "width", value: "100%" },+ ],+ ...(animation === undefined ? {} : { animation }),+ };+}++function stop(stopValue: MediaGradientStop): string {+ return `${stopValue.color} ${stopValue.offset * 100}%`;+}++function paint(value: MediaPaint): string {+ if (value.kind === "solid") return value.color;+ if (value.kind === "linear-gradient") {+ return `linear-gradient(${value.angleDeg}deg,${value.stops.map(stop).join(",")})`;+ }+ return `radial-gradient(circle at ${value.center.x * 100}% ${value.center.y * 100}%,${value.stops.map(stop).join(",")})`;+}++function pathData(path: SpatialPath): string {+ return path.commands.map((command) => {+ switch (command.kind) {+ case "move": return `M ${command.xPx} ${command.yPx}`;+ case "line": return `L ${command.xPx} ${command.yPx}`;+ case "quadratic": return `Q ${command.controlX} ${command.controlY} ${command.xPx} ${command.yPx}`;+ case "cubic": return `C ${command.control1X} ${command.control1Y} ${command.control2X} ${command.control2Y} ${command.xPx} ${command.yPx}`;+ case "close": return "Z";+ }+ }).join(" ");+}++function frameStyles(value: MediaFramePresentation): VisualStyleDeclaration[] {+ const styles: VisualStyleDeclaration[] = [+ { name: "box-sizing", value: "border-box" },+ { name: "height", value: "100%" },+ { name: "left", value: 0 },+ { name: "position", value: "absolute" },+ { name: "top", value: 0 },+ { name: "width", value: "100%" },+ ];+ if (value.clip.kind === "none") styles.push({ name: "overflow", value: "visible" });+ else {+ styles.push({ name: "overflow", value: "hidden" });+ if (value.clip.kind === "rounded") styles.push({ name: "border-radius", value: px(value.clip.radiusPx) });+ if (value.clip.kind === "path") styles.push({ name: "clip-path", value: `path("${pathData(value.clip.path)}")` });+ }+ if (value.border !== undefined) {+ styles.push({ name: "border", value: `${value.border.widthPx}px ${value.border.style} ${value.border.color}` });+ }+ if (value.shadows.length > 0) {+ styles.push({+ name: "box-shadow",+ value: value.shadows.map((shadow) =>+ `${shadow.offsetX}px ${shadow.offsetY}px ${shadow.blurPx}px ${shadow.spreadPx}px ${shadow.color}`).join(","),+ });+ }+ return styles;+}++function innerFrame(item: MediaItemProgram): SpatialFrame {+ const { padding } = item.presentation;+ return {+ contract: "svml.spatial-frame@1",+ xPx: item.frame.xPx + padding.leftPx,+ yPx: item.frame.yPx + padding.topPx,+ widthPx: item.frame.widthPx - padding.leftPx - padding.rightPx,+ heightPx: item.frame.heightPx - padding.topPx - padding.bottomPx,+ };+}++function filter(layer: MediaSampleLayerProgram): string {+ const value = layer.appearance.filter;+ return [+ value.blurPx === 0 ? "" : `blur(${value.blurPx}px)`,+ value.brightness === 1 ? "" : `brightness(${value.brightness})`,+ value.contrast === 1 ? "" : `contrast(${value.contrast})`,+ value.saturation === 1 ? "" : `saturate(${value.saturation})`,+ ].filter(Boolean).join(" ") || "none";+}++function sampleElements(+ layer: MediaSampleLayerProgram,+ item: MediaItemProgram,+ parent: string,+ order: { value: number },+ space: ProgramSpace,+): VisualElement[] {+ const content = fitContent(innerFrame(item), layer.source.extent, layer.fit).contentFrame;+ const wrapper = `${layer.id}:sampling`;+ const durationFrames = item.span.endFrameExclusive - item.span.startFrame;+ const wrapperStyle: VisualStyleDeclaration[] = [+ { name: "height", value: px(content.heightPx) },+ { name: "left", value: px(content.xPx - item.frame.xPx) },+ { name: "position", value: "absolute" },+ { name: "top", value: px(content.yPx - item.frame.yPx) },+ { name: "transform-origin", value: "center center" },+ { name: "width", value: px(content.widthPx) },+ ];+ const wrapperElement: VisualElement = {+ id: wrapper,+ parent,+ order: order.value++,+ kind: "box",+ style: wrapperStyle,+ ...(layer.samplingMotion === undefined ? {} : {+ animation: samplingAnimation(layer.samplingMotion, durationFrames),+ }),+ };+ // Blur samples are overscanned inside the owned frame before clipping. The+ // Gaussian kernel therefore never invents a transparent/dark border and still+ // cannot observe any pixel from a sibling Track.+ const overscanPx = layer.appearance.filter.blurPx === 0+ ? 0 : Math.ceil(layer.appearance.filter.blurPx * 2);+ const mediaStyle: VisualStyleDeclaration[] = [+ { name: "filter", value: filter(layer) },+ { name: "height", value: overscanPx === 0 ? "100%" : px(content.heightPx + (overscanPx * 2)) },+ { name: "left", value: overscanPx === 0 ? 0 : px(-overscanPx) },+ { name: "opacity", value: layer.appearance.opacity },+ { name: "position", value: "absolute" },+ { name: "top", value: overscanPx === 0 ? 0 : px(-overscanPx) },+ { name: "width", value: overscanPx === 0 ? "100%" : px(content.widthPx + (overscanPx * 2)) },+ ];+ if (layer.source.kind === "still") {+ return [wrapperElement, {+ id: layer.id,+ parent: wrapper,+ order: order.value++,+ kind: "image",+ artifact: structuredClone(layer.source.artifact),+ style: mediaStyle,+ }];+ }+ const sourceTiming = layer.source.kind === "timed"+ ? { frameRate: layer.source.frameRate, frameCount: layer.source.frameCount }+ : layer.source.kind === "surface" && layer.source.surface.timing.kind === "frames"+ ? { frameRate: layer.source.surface.timing.frameRate, frameCount: layer.source.surface.timing.frameCount }+ : undefined;+ if (sourceTiming === undefined) {+ if (layer.source.kind !== "surface") throw new Error(`Timed Media layer ${layer.id} has no timing.`);+ return [wrapperElement, {+ id: layer.id,+ parent: wrapper,+ order: order.value++,+ kind: "surface",+ surface: structuredClone(layer.source.surface),+ style: mediaStyle,+ }];+ }+ const sampling = resolveVisualSampling({+ space,+ sourceFrameRate: sourceTiming.frameRate,+ sourceFrameCount: sourceTiming.frameCount,+ targetFrameCount: durationFrames,+ ...(layer.trim === undefined ? {} : { trim: layer.trim }),+ occupancy: layer.occupancy!,+ });+ if (layer.source.kind === "timed") {+ return [wrapperElement, {+ id: layer.id,+ parent: wrapper,+ order: order.value++,+ kind: "video",+ artifact: structuredClone(layer.source.artifact),+ sampling,+ muted: true,+ style: mediaStyle,+ }];+ }+ return [wrapperElement, {+ id: layer.id,+ parent: wrapper,+ order: order.value++,+ kind: "surface",+ surface: structuredClone(layer.source.surface),+ sampling,+ style: mediaStyle,+ }];+}++function layerElements(+ layer: MediaLayerProgram,+ item: MediaItemProgram,+ parent: string,+ order: { value: number },+ space: ProgramSpace,+): VisualElement[] {+ if (layer.kind === "sample") return sampleElements(layer, item, parent, order, space);+ return [{+ id: layer.id,+ parent,+ order: order.value++,+ kind: "box",+ style: [+ { name: "background", value: paint(layer.paint) },+ { name: "height", value: "100%" },+ { name: "left", value: 0 },+ { name: "opacity", value: layer.opacity },+ { name: "position", value: "absolute" },+ { name: "top", value: 0 },+ { name: "width", value: "100%" },+ ],+ }];+}++/** Lower one independent Item through the fixed wrapper stack from the Media spec. */+export function lowerMediaItemElements(+ item: MediaItemProgram,+ space: ProgramSpace,+ options: {+ readonly includeHandoffWrapper?: boolean;+ readonly handoffAnimation?: VisualAnimation;+ readonly lifecycleAnimationOverride?: VisualAnimation | null;+ readonly sustainAnimationsOverride?: readonly (VisualAnimation | null)[];+ } = {},+): readonly VisualElement[] {+ const durationFrames = item.span.endFrameExclusive - item.span.startFrame;+ const order = { value: 0 };+ const root = `${item.id}:placement`;+ const elements: VisualElement[] = [{+ id: root,+ order: order.value++,+ kind: "box",+ style: [+ { name: "height", value: px(item.frame.heightPx) },+ { name: "left", value: px(item.frame.xPx) },+ { name: "position", value: "absolute" },+ { name: "top", value: px(item.frame.yPx) },+ { name: "width", value: px(item.frame.widthPx) },+ ],+ }];+ let parent = `${item.id}:lifecycle`;+ const lifecycle = options.lifecycleAnimationOverride === undefined+ ? lifecycleAnimation(item.motion, durationFrames)+ : options.lifecycleAnimationOverride ?? undefined;+ elements.push(box(parent, order.value++, root, lifecycle));+ for (const [index, sustain] of item.motion.sustain.entries()) {+ const id = `${item.id}:sustain:${index + 1}`;+ const animation = options.sustainAnimationsOverride === undefined+ ? sustainAnimation(sustain, durationFrames)+ : options.sustainAnimationsOverride[index] ?? undefined;+ elements.push(box(id, order.value++, parent, animation ?? undefined));+ parent = id;+ }+ if (options.includeHandoffWrapper === true) {+ const handoff = `${item.id}:handoff`;+ elements.push(box(handoff, order.value++, parent, options.handoffAnimation));+ parent = handoff;+ }+ const frame = `${item.id}:frame`;+ elements.push({ id: frame, parent, order: order.value++, kind: "box", style: frameStyles(item.presentation) });+ for (const layer of item.layers) elements.push(...layerElements(layer, item, frame, order, space));+ return elements;+}diff --git a/packages/media-track/src/manifest.ts b/packages/media-track/src/manifest.tsindex 0e845990..f6550322 100644--- a/packages/media-track/src/manifest.ts+++ b/packages/media-track/src/manifest.ts@@ -1,9 +1,15 @@import { artifactDependency, artifactTypes } from "@narratage/artifact";import { compositionDependency, compositionTypes } from "@narratage/composition";-import { mediaArtifactSchema, mediaDependency } from "@narratage/media";+import {+ compositableSurfaceSchema,+ mediaDependency,+ mediaTypes,+} from "@narratage/media";+import { narrativeDependency, narrativeTypes } from "@narratage/narrative";import { programSpaceDependency, programSpaceTypes } from "@narratage/program-space";import { digestOf } from "@narratage/protocol";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@narratage/protocol";+import { semanticMapDependency, semanticMapTypes } from "@narratage/semantic-map";import {contentFitSchema,intrinsicExtentSchema,@@ -11,90 +17,337 @@ import {spatialFrameSchema,spatialTypes,} from "@narratage/spatial";+import { temporalDependency } from "@narratage/temporal";import { mediaTrackImplementationDigests, mediaTrackValidatorDigests } from "./program.js";export const mediaTrackModuleRef = { name: "@narratage/media-track", version: "0.0.0-dev" } as const;export const mediaTrackTypes = {- program: { module: mediaTrackModuleRef, name: "MediaTrackProgram" },header: { module: mediaTrackModuleRef, name: "MediaTrackHeader" },- stillItemSpec: { module: mediaTrackModuleRef, name: "MediaStillItemSpec" },+ paintLayerSpec: { module: mediaTrackModuleRef, name: "MediaPaintLayerSpec" },+ sampleLayerSpec: { module: mediaTrackModuleRef, name: "MediaSampleLayerSpec" },+ layerSet: { module: mediaTrackModuleRef, name: "MediaLayerSet" },+ soundSpec: { module: mediaTrackModuleRef, name: "MediaSoundSpec" },+ soundSet: { module: mediaTrackModuleRef, name: "MediaSoundSet" },+ itemSpec: { module: mediaTrackModuleRef, name: "MediaItemSpec" },+ memberSpec: { module: mediaTrackModuleRef, name: "MediaSequenceMemberSpec" },+ memberSet: { module: mediaTrackModuleRef, name: "MediaSequenceMemberSet" },+ handoffSpec: { module: mediaTrackModuleRef, name: "MediaHandoffSpec" },+ sequenceSpec: { module: mediaTrackModuleRef, name: "MediaSequenceSpec" },set: { module: mediaTrackModuleRef, name: "MediaTrackSet" },+ program: { module: mediaTrackModuleRef, name: "MediaTrackProgram" },} satisfies Record<string, TypeRef>;+export const mediaTrackProducers = {+ createLayers: { module: mediaTrackModuleRef, name: "create-media-layer-set" },+ appendPaintLayer: { module: mediaTrackModuleRef, name: "append-media-paint-layer" },+ appendStillLayer: { module: mediaTrackModuleRef, name: "append-still-media-layer" },+ appendTimedLayer: { module: mediaTrackModuleRef, name: "append-timed-media-layer" },+ appendSurfaceLayer: { module: mediaTrackModuleRef, name: "append-surface-media-layer" },+ createSounds: { module: mediaTrackModuleRef, name: "create-media-sound-set" },+ appendSound: { module: mediaTrackModuleRef, name: "append-media-sound" },createSet: { module: mediaTrackModuleRef, name: "create-media-track-set" },- appendFullStill: { module: mediaTrackModuleRef, name: "append-full-still-media-item" },+ appendProgramItem: { module: mediaTrackModuleRef, name: "append-program-media-item" },+ appendSelectionItem: { module: mediaTrackModuleRef, name: "append-selection-media-item" },+ appendMomentItem: { module: mediaTrackModuleRef, name: "append-moment-media-item" },+ bindItemClipPath: { module: mediaTrackModuleRef, name: "bind-media-item-clip-path" },+ bindSequenceClipPath: { module: mediaTrackModuleRef, name: "bind-media-sequence-clip-path" },+ createMembers: { module: mediaTrackModuleRef, name: "create-media-sequence-member-set" },+ appendMomentMember: { module: mediaTrackModuleRef, name: "append-moment-media-sequence-member" },+ appendSelectionStartMember: { module: mediaTrackModuleRef, name: "append-selection-start-media-sequence-member" },+ appendSelectionEndMember: { module: mediaTrackModuleRef, name: "append-selection-end-media-sequence-member" },+ appendSequenceProgramEnd: { module: mediaTrackModuleRef, name: "append-media-sequence-until-program-end" },+ appendSequenceUntilMoment: { module: mediaTrackModuleRef, name: "append-media-sequence-until-moment" },+ appendSequenceUntilSelectionStart: { module: mediaTrackModuleRef, name: "append-media-sequence-until-selection-start" },+ appendSequenceUntilSelectionEnd: { module: mediaTrackModuleRef, name: "append-media-sequence-until-selection-end" },finalize: { module: mediaTrackModuleRef, name: "finalize-media-track" },- render: { module: mediaTrackModuleRef, name: "render-media-track" },+ projectVisual: { module: mediaTrackModuleRef, name: "project-media-visual-track" },+ projectAudio: { module: mediaTrackModuleRef, name: "project-media-audio-track" },} satisfies Record<string, ProducerRef>;+export const mediaTrackSurfaceImplementationDigest = digestOf("@narratage/media-track/track-surface@1");+const string = { kind: "string", minLength: 1 } as const;+const number = { kind: "number" } as const;+const unsigned = { kind: "number", minimum: 0 } as const;const integer = { kind: "number", integer: true } as const;const unsignedInteger = { kind: "number", integer: true, minimum: 0 } as const;+const positiveInteger = { kind: "number", integer: true, minimum: 1 } as const;const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>): ValueSchema => ({ kind: "object", fields });-const frameSpan = object({ startFrame: { schema: unsignedInteger }, endFrameExclusive: { schema: unsignedInteger } });+const blob = object({+ kind: { schema: { kind: "literal", value: "blob" } },+ digest: { schema: { kind: "string", minLength: 71, maxLength: 71 } },+ size: { schema: unsignedInteger },+ mediaType: { schema: string },+});+const rational = object({ numerator: { schema: positiveInteger }, denominator: { schema: positiveInteger } });+const extent = intrinsicExtentSchema;+const trim = object({ startFrame: { schema: unsignedInteger }, endFrameExclusive: { schema: positiveInteger } });+const occupancy: ValueSchema = { kind: "oneOf", variants: [+ ...["once", "hold", "loop"].map((mode) => object({+ mode: { schema: { kind: "literal", value: mode } },+ align: { schema: { kind: "string", enum: ["start", "end"] } },+ })),+ object({ mode: { schema: { kind: "literal", value: "stretch" } } }),+] };+const stops = { kind: "array", minItems: 2, items: object({ offset: { schema: unsigned }, color: { schema: string } }) } as const;+const paint: ValueSchema = { kind: "oneOf", variants: [+ object({ kind: { schema: { kind: "literal", value: "solid" } }, color: { schema: string } }),+ object({ kind: { schema: { kind: "literal", value: "linear-gradient" } }, angleDeg: { schema: number }, stops: { schema: stops } }),+ object({ kind: { schema: { kind: "literal", value: "radial-gradient" } }, center: { schema: object({ x: { schema: unsigned }, y: { schema: unsigned } }) }, stops: { schema: stops } }),+] };+const appearance = object({+ opacity: { schema: unsigned },+ filter: { schema: object({+ blurPx: { schema: unsigned }, brightness: { schema: unsigned }, contrast: { schema: unsigned }, saturation: { schema: unsigned },+ }) },+});+const samplingKeyframe = object({+ atProgress: { schema: { kind: "number", minimum: 0, maximum: 1 } }, zoom: { schema: { kind: "number", minimum: 0.000001 } },+ offsetX: { schema: number }, offsetY: { schema: number }, rotationDeg: { schema: number },+ easing: { schema: { kind: "string", enum: ["linear", "ease-in", "ease-out", "ease-in-out"] }, optional: true },+});+const samplingMotion = object({ keyframes: { schema: { kind: "array", minItems: 2, items: samplingKeyframe } } });+export const mediaPaintLayerSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-paint-layer-spec@1" } },+ id: { schema: string }, paint: { schema: paint }, opacity: { schema: unsigned },+});+export const mediaSampleLayerSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-sample-layer-spec@1" } },+ id: { schema: string }, trim: { schema: trim, optional: true }, occupancy: { schema: occupancy, optional: true },+ appearance: { schema: appearance }, samplingMotion: { schema: samplingMotion, optional: true },+});+const audioSource = object({ artifact: { schema: blob }, sampleFrames: { schema: positiveInteger } });+const visualSource: ValueSchema = { kind: "oneOf", variants: [+ object({ kind: { schema: { kind: "literal", value: "still" } }, artifact: { schema: blob }, extent: { schema: extent } }),+ object({+ kind: { schema: { kind: "literal", value: "timed" } }, artifact: { schema: blob }, extent: { schema: extent },+ frameRate: { schema: rational }, frameCount: { schema: positiveInteger }, audio: { schema: audioSource, optional: true },+ }),+ object({ kind: { schema: { kind: "literal", value: "surface" } }, surface: { schema: compositableSurfaceSchema }, extent: { schema: extent } }),+] };+const paintLayer = object({ id: { schema: string }, kind: { schema: { kind: "literal", value: "paint" } }, paint: { schema: paint }, opacity: { schema: unsigned } });+const sampleLayer = object({+ id: { schema: string }, kind: { schema: { kind: "literal", value: "sample" } }, source: { schema: visualSource },+ fit: { schema: contentFitSchema }, trim: { schema: trim, optional: true }, occupancy: { schema: occupancy, optional: true },+ appearance: { schema: appearance }, samplingMotion: { schema: samplingMotion, optional: true },+});+const layer = { kind: "oneOf", variants: [paintLayer, sampleLayer] } as const;+export const mediaLayerSetSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-layer-set@1" } },+ layers: { schema: { kind: "array", items: layer } },+});++const duration: ValueSchema = { kind: "oneOf", variants: [+ object({ unit: { schema: { kind: "literal", value: "frames" } }, value: { schema: unsignedInteger } }),+ object({ unit: { schema: { kind: "literal", value: "milliseconds" } }, value: { schema: unsignedInteger } }),+ object({ unit: { schema: { kind: "literal", value: "seconds" } }, numerator: { schema: unsignedInteger }, denominator: { schema: positiveInteger } }),+] };+const signedDuration: ValueSchema = { kind: "oneOf", variants: [+ object({ unit: { schema: { kind: "literal", value: "frames" } }, value: { schema: integer } }),+ object({ unit: { schema: { kind: "literal", value: "milliseconds" } }, value: { schema: integer } }),+ object({ unit: { schema: { kind: "literal", value: "seconds" } }, numerator: { schema: integer }, denominator: { schema: positiveInteger } }),+] };+const point: ValueSchema = { kind: "oneOf", variants: [+ ...["program.start", "program.end", "selection.start", "selection.end", "moment.cue"].map((ref) => object({+ ref: { schema: { kind: "literal", value: ref } }, offset: { schema: signedDuration, optional: true },+ })),+ object({ ref: { schema: { kind: "literal", value: "absolute" } }, at: { schema: duration } }),+] };+const projection = object({ start: { schema: point }, end: { schema: point } });+const path = { kind: "object", fields: {}, allowUnknown: true } as const;+const clip: ValueSchema = { kind: "oneOf", variants: [+ object({ kind: { schema: { kind: "literal", value: "none" } } }),+ object({ kind: { schema: { kind: "literal", value: "frame" } } }),+ object({ kind: { schema: { kind: "literal", value: "rounded" } }, radiusPx: { schema: unsigned } }),+ object({ kind: { schema: { kind: "literal", value: "path" } }, path: { schema: path } }),+] };+const padding = object({ topPx: { schema: unsigned }, rightPx: { schema: unsigned }, bottomPx: { schema: unsigned }, leftPx: { schema: unsigned } });const presentation = object({- clip: { schema: { kind: "literal", value: "frame" } },- fill: { schema: object({ kind: { schema: { kind: "literal", value: "transparent" } } }) },+ clip: { schema: clip }, padding: { schema: padding },+ border: { schema: object({ widthPx: { schema: unsigned }, style: { schema: { kind: "string", enum: ["solid", "dashed", "dotted"] } }, color: { schema: string } }), optional: true },+ shadows: { schema: { kind: "array", items: object({ offsetX: { schema: number }, offsetY: { schema: number }, blurPx: { schema: unsigned }, spreadPx: { schema: number }, color: { schema: string } }) } },+});+const edgeMotion = object({+ operator: { schema: { kind: "string", enum: ["fade", "slide", "scale", "pop", "bounce", "blur-reveal", "wipe", "flip", "spin"] } },+ durationFrames: { schema: positiveInteger }, easing: { schema: { kind: "string", enum: ["linear", "ease-in", "ease-out", "ease-in-out"] } },+ direction: { schema: { kind: "string", enum: ["left", "right", "up", "down"] }, optional: true }, amount: { schema: number, optional: true },+ origin: { schema: { kind: "literal", value: "outside-canvas" }, optional: true },+});+const sustainMotion = object({+ operator: { schema: { kind: "string", enum: ["float", "breathe", "pulse", "wobble", "shake", "drift"] } },+ amount: { schema: unsigned }, cycles: { schema: positiveInteger },+ direction: { schema: { kind: "string", enum: ["left", "right", "up", "down"] }, optional: true },+});+const lifecycle = object({+ enter: { schema: edgeMotion, optional: true }, sustain: { schema: { kind: "array", items: sustainMotion } },+ exit: { schema: edgeMotion, optional: true },+});+const soundTrigger: ValueSchema = { kind: "oneOf", variants: [+ object({ kind: { schema: { kind: "literal", value: "enter" } } }),+ object({ kind: { schema: { kind: "literal", value: "exit" } } }),+ object({ kind: { schema: { kind: "literal", value: "handoff" } }, handoffId: { schema: string } }),+] };+export const mediaSoundSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-sound-spec@1" } },+ id: { schema: string }, trigger: { schema: soundTrigger }, gain: { schema: unsigned },});-const stillLayer = object({+const soundEvent = object({ id: { schema: string }, trigger: { schema: soundTrigger }, source: { schema: audioSource }, gain: { schema: unsigned } });+export const mediaSoundSetSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-sound-set@1" } },+ sounds: { schema: { kind: "array", items: soundEvent } },+});+export const mediaItemSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-item-spec@1" } }, id: { schema: string },+ projection: { schema: projection }, expansion: { schema: object({ kind: { schema: { kind: "string", enum: ["one", "each"] } } }) },+ presentation: { schema: presentation }, motion: { schema: lifecycle }, stackingOrder: { schema: integer },+ sourceAudio: { schema: object({ fromLayer: { schema: string }, gain: { schema: unsigned } }), optional: true },+});+const sourceAudio = object({ fromLayer: { schema: string }, gain: { schema: unsigned } });+export const mediaSequenceMemberSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-sequence-member-spec@1" } },id: { schema: string },- kind: { schema: { kind: "literal", value: "still" } },- artifact: { schema: mediaArtifactSchema },- extent: { schema: intrinsicExtentSchema },- fit: { schema: contentFitSchema },-});-const item = object({- id: { schema: string }, span: { schema: frameSpan }, frame: { schema: spatialFrameSchema },- presentation: { schema: presentation },- layers: { schema: { kind: "array", minItems: 1, items: stillLayer } },+ sourceAudio: { schema: sourceAudio, optional: true },+});+const unresolvedMember = object({+ id: { schema: string }, activationFrame: { schema: unsignedInteger },+ layers: { schema: { kind: "array", minItems: 1, items: layer } },+ sourceAudio: { schema: sourceAudio, optional: true },+});+export const mediaSequenceMemberSetSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-sequence-member-set@1" } },+ members: { schema: { kind: "array", items: unresolvedMember } },+});+const direction = { kind: "string", enum: ["left", "right", "up", "down"] } as const;+const handoffFields = {+ id: { schema: string }, fromMemberId: { schema: string }, toMemberId: { schema: string },+ operator: { schema: { kind: "string", enum: ["cut", "crossfade", "push", "wipe", "cover", "page-turn"] } },+ durationFrames: { schema: unsignedInteger }, boundaryRatio: { schema: unsigned },+ direction: { schema: direction, optional: true }, audio: { schema: { kind: "string", enum: ["cut", "crossfade"] } },+} as const;+export const mediaHandoffSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-handoff-spec@1" } }, ...handoffFields,+});+export const mediaSequenceSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-sequence-spec@1" } },+ id: { schema: string }, presentation: { schema: presentation },+ motion: { schema: lifecycle }, stackingOrder: { schema: integer },+ handoffs: { schema: { kind: "array", minItems: 1, items: mediaHandoffSpecSchema } },+});+const frameSpan = object({ startFrame: { schema: unsignedInteger }, endFrameExclusive: { schema: positiveInteger } });+const resolvedItem = object({+ id: { schema: string }, sourceOccurrenceId: { schema: string }, span: { schema: frameSpan }, frame: { schema: spatialFrameSchema },+ presentation: { schema: presentation }, layers: { schema: { kind: "array", minItems: 1, items: layer } }, motion: { schema: lifecycle },stacking: { schema: object({ order: { schema: integer }, tieBreak: { schema: string } }) },+ sourceAudio: { schema: object({ fromLayer: { schema: string }, gain: { schema: unsigned } }), optional: true },+ sounds: { schema: { kind: "array", items: soundEvent } },+});+const resolvedMember = object({+ id: { schema: string }, activationFrame: { schema: unsignedInteger }, logicalSpan: { schema: frameSpan },+ visualSpan: { schema: frameSpan }, layers: { schema: { kind: "array", minItems: 1, items: layer } },+ sourceAudio: { schema: sourceAudio, optional: true },+});+const resolvedHandoff = object({ ...handoffFields, span: { schema: frameSpan } });+const resolvedSequence = object({+ id: { schema: string }, span: { schema: frameSpan }, terminalFrame: { schema: positiveInteger },+ frame: { schema: spatialFrameSchema }, presentation: { schema: presentation },+ members: { schema: { kind: "array", minItems: 2, items: resolvedMember } },+ handoffs: { schema: { kind: "array", minItems: 1, items: resolvedHandoff } },+ motion: { schema: lifecycle }, stacking: { schema: object({ order: { schema: integer }, tieBreak: { schema: string } }) },+ sounds: { schema: { kind: "array", items: soundEvent } },+});+export const mediaTrackSetSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.media-track-set@1" } }, items: { schema: { kind: "array", items: resolvedItem } },+ sequences: { schema: { kind: "array", items: resolvedSequence } },});export const mediaTrackProgramSchema: ValueSchema = object({- contract: { schema: { kind: "literal", value: "svml.media-track-program@1" } },- id: { schema: string }, items: { schema: { kind: "array", minItems: 1, items: item } },+ contract: { schema: { kind: "literal", value: "svml.media-track-program@1" } }, id: { schema: string },+ items: { schema: { kind: "array", items: resolvedItem } }, sequences: { schema: { kind: "array", items: resolvedSequence } },});export const mediaTrackHeaderSchema: ValueSchema = object({contract: { schema: { kind: "literal", value: "svml.media-track-header@1" } }, id: { schema: string },});-export const mediaStillItemSpecSchema: ValueSchema = object({- contract: { schema: { kind: "literal", value: "svml.media-still-item-spec@1" } },- id: { schema: string }, stackingOrder: { schema: integer }, presentation: { schema: presentation },-});-export const mediaTrackSetSchema: ValueSchema = object({- contract: { schema: { kind: "literal", value: "svml.media-track-set@1" } },- items: { schema: { kind: "array", items: item } },-});const validator = (locator: string, digest: ReturnType<typeof digestOf>) => ({abi: "svml.type-validator@1" as const,implementation: { kind: "registered" as const, locator, digest },});+const itemInputs = [+ { name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header },+ { name: "space", type: programSpaceTypes.programSpace }, { name: "canvas", type: spatialTypes.canvas },+ { name: "layers", type: mediaTrackTypes.layerSet },+ { name: "frame", type: spatialTypes.frame }, { name: "spec", type: mediaTrackTypes.itemSpec },+ { name: "sounds", type: mediaTrackTypes.soundSet },+] as const;+const memberInputs = [+ { name: "members", type: mediaTrackTypes.memberSet }, { name: "layers", type: mediaTrackTypes.layerSet },+ { name: "spec", type: mediaTrackTypes.memberSpec }, { name: "map", type: semanticMapTypes.complete },+] as const;+const sequenceInputs = [+ { name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header },+ { name: "space", type: programSpaceTypes.programSpace }, { name: "canvas", type: spatialTypes.canvas },+ { name: "members", type: mediaTrackTypes.memberSet },+ { name: "frame", type: spatialTypes.frame }, { name: "spec", type: mediaTrackTypes.sequenceSpec },+ { name: "sounds", type: mediaTrackTypes.soundSet },+] as const;export const mediaTrackManifest: ModuleManifest = {format: "svml.module@1",name: mediaTrackModuleRef.name,version: mediaTrackModuleRef.version,- dependencies: [artifactDependency, mediaDependency, programSpaceDependency, spatialDependency, compositionDependency],+ dependencies: [artifactDependency, mediaDependency, narrativeDependency, semanticMapDependency, programSpaceDependency, temporalDependency, spatialDependency, compositionDependency],types: [- { name: mediaTrackTypes.program.name, schema: mediaTrackProgramSchema, validator: validator("@narratage/media-track/validate-program", mediaTrackValidatorDigests.program) },{ name: mediaTrackTypes.header.name, schema: mediaTrackHeaderSchema },- { name: mediaTrackTypes.stillItemSpec.name, schema: mediaStillItemSpecSchema },+ { name: mediaTrackTypes.paintLayerSpec.name, schema: mediaPaintLayerSpecSchema },+ { name: mediaTrackTypes.sampleLayerSpec.name, schema: mediaSampleLayerSpecSchema },+ { name: mediaTrackTypes.layerSet.name, schema: mediaLayerSetSchema },+ { name: mediaTrackTypes.soundSpec.name, schema: mediaSoundSpecSchema },+ { name: mediaTrackTypes.soundSet.name, schema: mediaSoundSetSchema },+ { name: mediaTrackTypes.itemSpec.name, schema: mediaItemSpecSchema },+ { name: mediaTrackTypes.memberSpec.name, schema: mediaSequenceMemberSpecSchema },+ { name: mediaTrackTypes.memberSet.name, schema: mediaSequenceMemberSetSchema },+ { name: mediaTrackTypes.handoffSpec.name, schema: mediaHandoffSpecSchema },+ { name: mediaTrackTypes.sequenceSpec.name, schema: mediaSequenceSpecSchema },{ name: mediaTrackTypes.set.name, schema: mediaTrackSetSchema },+ { name: mediaTrackTypes.program.name, schema: mediaTrackProgramSchema, validator: validator("@narratage/media-track/validate-program", mediaTrackValidatorDigests.program) },],capabilities: [],- surfaces: [],+ surfaces: [{+ name: "track", tag: "Track", mode: "structured",+ outputs: [spatialTypes.fit, mediaTrackTypes.header, mediaTrackTypes.paintLayerSpec, mediaTrackTypes.sampleLayerSpec,+ mediaTrackTypes.layerSet, mediaTrackTypes.soundSpec, mediaTrackTypes.soundSet,+ mediaTrackTypes.itemSpec, mediaTrackTypes.memberSpec, mediaTrackTypes.memberSet,+ mediaTrackTypes.handoffSpec, mediaTrackTypes.sequenceSpec, mediaTrackTypes.set, mediaTrackTypes.program,+ compositionTypes.visualTrack, compositionTypes.audioTrack],+ implementation: { kind: "trusted-frontend-surface", locator: "@narratage/media-track/track-surface", digest: mediaTrackSurfaceImplementationDigest },+ }],producers: [+ { name: mediaTrackProducers.createLayers.name, inputs: [], outputs: [{ name: "layers", type: mediaTrackTypes.layerSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/create-layers", digest: mediaTrackImplementationDigests.createLayers } },+ { name: mediaTrackProducers.appendPaintLayer.name, inputs: [{ name: "layers", type: mediaTrackTypes.layerSet }, { name: "spec", type: mediaTrackTypes.paintLayerSpec }], outputs: [{ name: "layers", type: mediaTrackTypes.layerSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-paint-layer", digest: mediaTrackImplementationDigests.appendPaintLayer } },+ { name: mediaTrackProducers.appendStillLayer.name, inputs: [{ name: "layers", type: mediaTrackTypes.layerSet }, { name: "source", type: artifactTypes.blob }, { name: "extent", type: spatialTypes.extent }, { name: "fit", type: spatialTypes.fit }, { name: "spec", type: mediaTrackTypes.sampleLayerSpec }], outputs: [{ name: "layers", type: mediaTrackTypes.layerSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-still-layer", digest: mediaTrackImplementationDigests.appendStillLayer } },+ { name: mediaTrackProducers.appendTimedLayer.name, inputs: [{ name: "layers", type: mediaTrackTypes.layerSet }, { name: "source", type: mediaTypes.synchronized }, { name: "fit", type: spatialTypes.fit }, { name: "spec", type: mediaTrackTypes.sampleLayerSpec }], outputs: [{ name: "layers", type: mediaTrackTypes.layerSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-timed-layer", digest: mediaTrackImplementationDigests.appendTimedLayer } },+ { name: mediaTrackProducers.appendSurfaceLayer.name, inputs: [{ name: "layers", type: mediaTrackTypes.layerSet }, { name: "source", type: mediaTypes.compositableSurface }, { name: "fit", type: spatialTypes.fit }, { name: "spec", type: mediaTrackTypes.sampleLayerSpec }], outputs: [{ name: "layers", type: mediaTrackTypes.layerSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-surface-layer", digest: mediaTrackImplementationDigests.appendSurfaceLayer } },+ { name: mediaTrackProducers.createSounds.name, inputs: [], outputs: [{ name: "sounds", type: mediaTrackTypes.soundSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/create-sounds", digest: mediaTrackImplementationDigests.createSounds } },+ { name: mediaTrackProducers.appendSound.name, inputs: [{ name: "sounds", type: mediaTrackTypes.soundSet }, { name: "source", type: mediaTypes.synchronized }, { name: "spec", type: mediaTrackTypes.soundSpec }], outputs: [{ name: "sounds", type: mediaTrackTypes.soundSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-sound", digest: mediaTrackImplementationDigests.appendSound } },{ name: mediaTrackProducers.createSet.name, inputs: [], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/create-set", digest: mediaTrackImplementationDigests.createSet } },- { name: mediaTrackProducers.appendFullStill.name, inputs: [- { name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header },- { name: "space", type: programSpaceTypes.programSpace }, { name: "source", type: artifactTypes.blob },- { name: "extent", type: spatialTypes.extent }, { name: "frame", type: spatialTypes.frame },- { name: "fit", type: spatialTypes.fit }, { name: "spec", type: mediaTrackTypes.stillItemSpec },- ], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-full-still", digest: mediaTrackImplementationDigests.appendFullStill } },- { name: mediaTrackProducers.finalize.name, inputs: [{ name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header }], outputs: [{ name: "program", type: mediaTrackTypes.program }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/finalize", digest: mediaTrackImplementationDigests.finalize } },- { name: mediaTrackProducers.render.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: mediaTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/render", digest: mediaTrackImplementationDigests.render } },+ { name: mediaTrackProducers.appendProgramItem.name, inputs: [...itemInputs], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-program-item", digest: mediaTrackImplementationDigests.appendProgramItem } },+ { name: mediaTrackProducers.appendSelectionItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-selection-item", digest: mediaTrackImplementationDigests.appendSelectionItem } },+ { name: mediaTrackProducers.appendMomentItem.name, inputs: [...itemInputs, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-moment-item", digest: mediaTrackImplementationDigests.appendMomentItem } },+ { name: mediaTrackProducers.bindItemClipPath.name, inputs: [{ name: "spec", type: mediaTrackTypes.itemSpec }, { name: "path", type: spatialTypes.path }], outputs: [{ name: "spec", type: mediaTrackTypes.itemSpec }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/bind-item-clip-path", digest: mediaTrackImplementationDigests.bindItemClipPath } },+ { name: mediaTrackProducers.bindSequenceClipPath.name, inputs: [{ name: "spec", type: mediaTrackTypes.sequenceSpec }, { name: "path", type: spatialTypes.path }], outputs: [{ name: "spec", type: mediaTrackTypes.sequenceSpec }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/bind-sequence-clip-path", digest: mediaTrackImplementationDigests.bindSequenceClipPath } },+ { name: mediaTrackProducers.createMembers.name, inputs: [], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/create-members", digest: mediaTrackImplementationDigests.createMembers } },+ { name: mediaTrackProducers.appendMomentMember.name, inputs: [...memberInputs, { name: "moment", type: narrativeTypes.moment }, { name: "space", type: programSpaceTypes.programSpace }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-moment-member", digest: mediaTrackImplementationDigests.appendMomentMember } },+ { name: mediaTrackProducers.appendSelectionStartMember.name, inputs: [...memberInputs, { name: "selection", type: narrativeTypes.selection }, { name: "space", type: programSpaceTypes.programSpace }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-selection-start-member", digest: mediaTrackImplementationDigests.appendSelectionStartMember } },+ { name: mediaTrackProducers.appendSelectionEndMember.name, inputs: [...memberInputs, { name: "selection", type: narrativeTypes.selection }, { name: "space", type: programSpaceTypes.programSpace }], outputs: [{ name: "members", type: mediaTrackTypes.memberSet }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-selection-end-member", digest: mediaTrackImplementationDigests.appendSelectionEndMember } },+ { name: mediaTrackProducers.appendSequenceProgramEnd.name, inputs: sequenceInputs, outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-sequence-program-end", digest: mediaTrackImplementationDigests.appendSequenceProgramEnd } },+ { name: mediaTrackProducers.appendSequenceUntilMoment.name, inputs: [...sequenceInputs, { name: "map", type: semanticMapTypes.complete }, { name: "moment", type: narrativeTypes.moment }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-sequence-until-moment", digest: mediaTrackImplementationDigests.appendSequenceUntilMoment } },+ { name: mediaTrackProducers.appendSequenceUntilSelectionStart.name, inputs: [...sequenceInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-sequence-until-selection-start", digest: mediaTrackImplementationDigests.appendSequenceUntilSelectionStart } },+ { name: mediaTrackProducers.appendSequenceUntilSelectionEnd.name, inputs: [...sequenceInputs, { name: "map", type: semanticMapTypes.complete }, { name: "selection", type: narrativeTypes.selection }], outputs: [{ name: "set", type: mediaTrackTypes.set }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/append-sequence-until-selection-end", digest: mediaTrackImplementationDigests.appendSequenceUntilSelectionEnd } },+ { name: mediaTrackProducers.finalize.name, inputs: [{ name: "set", type: mediaTrackTypes.set }, { name: "header", type: mediaTrackTypes.header }, { name: "space", type: programSpaceTypes.programSpace }], outputs: [{ name: "program", type: mediaTrackTypes.program }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/finalize", digest: mediaTrackImplementationDigests.finalize } },+ { name: mediaTrackProducers.projectVisual.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: mediaTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.visualTrack }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/project-visual", digest: mediaTrackImplementationDigests.projectVisual } },+ { name: mediaTrackProducers.projectAudio.name, inputs: [{ name: "space", type: programSpaceTypes.programSpace }, { name: "program", type: mediaTrackTypes.program }], outputs: [{ name: "track", type: compositionTypes.audioTrack }], needs: [], implementation: { kind: "registered", locator: "@narratage/media-track/project-audio", digest: mediaTrackImplementationDigests.projectAudio } },],};+export const mediaTrackManifestDigest = digestOf(mediaTrackManifest);export const mediaTrackDependency = { module: mediaTrackModuleRef, digest: mediaTrackManifestDigest } as const;diff --git a/packages/media-track/src/motion.ts b/packages/media-track/src/motion.tsnew file mode 100644index 00000000..8fe3171a--- /dev/null+++ b/packages/media-track/src/motion.ts@@ -0,0 +1,407 @@+import type {+ VisualAnimation,+ VisualEasing,+ VisualStyleDeclaration,+} from "@narratage/composition";++import type {+ MediaEdgeMotion,+ MediaLifecycleMotion,+ MediaMotionDirection,+ MediaSamplingMotion,+ MediaSustainMotion,+} from "./types.js";+import type { CanvasSpace, SpatialFrame } from "@narratage/spatial";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function finite(value: number, label: string): void {+ assert(Number.isFinite(value), `${label} must be finite.`);+}++function direction(value: MediaMotionDirection | undefined, label: string): void {+ if (value !== undefined) assert(["left", "right", "up", "down"].includes(value), `${label} is invalid.`);+}++export function assertMediaEdgeMotion(value: MediaEdgeMotion, label: string): void {+ assert(["fade", "slide", "scale", "pop", "bounce", "blur-reveal", "wipe", "flip", "spin"].includes(value.operator),+ `${label}.operator is invalid.`);+ assert(Number.isSafeInteger(value.durationFrames) && value.durationFrames > 0,+ `${label}.durationFrames must be positive.`);+ assert(["linear", "ease-in", "ease-out", "ease-in-out"].includes(value.easing), `${label}.easing is invalid.`);+ direction(value.direction, `${label}.direction`);+ if (["slide", "wipe", "flip"].includes(value.operator)) {+ assert(value.direction !== undefined, `${label}.${value.operator} requires direction.`);+ }+ if (value.amount !== undefined) finite(value.amount, `${label}.amount`);+ if (value.origin !== undefined) {+ assert(value.origin === "outside-canvas" && value.operator === "slide",+ `${label}.origin is supported only by slide.`);+ assert(value.amount === undefined, `${label} cannot combine amount with outside-canvas.`);+ }+}++function outsideCanvasAmount(directionValue: MediaMotionDirection, frame: SpatialFrame, canvas: CanvasSpace): number {+ if (directionValue === "left") return frame.xPx + frame.widthPx;+ if (directionValue === "right") return canvas.widthPx - frame.xPx;+ if (directionValue === "up") return frame.yPx + frame.heightPx;+ return canvas.heightPx - frame.yPx;+}++export function resolveMediaLifecycleMotion(+ value: MediaLifecycleMotion,+ frame: SpatialFrame,+ canvas: CanvasSpace,+): MediaLifecycleMotion {+ const edge = (motion: MediaEdgeMotion | undefined): MediaEdgeMotion | undefined => {+ if (motion === undefined || motion.origin === undefined) return motion === undefined ? undefined : structuredClone(motion);+ return {+ operator: motion.operator,+ durationFrames: motion.durationFrames,+ easing: motion.easing,+ ...(motion.direction === undefined ? {} : { direction: motion.direction }),+ amount: outsideCanvasAmount(motion.direction!, frame, canvas),+ };+ };+ return {+ ...(value.enter === undefined ? {} : { enter: edge(value.enter)! }),+ sustain: structuredClone(value.sustain),+ ...(value.exit === undefined ? {} : { exit: edge(value.exit)! }),+ };+}++export function assertMediaSustainMotion(value: MediaSustainMotion, label: string): void {+ assert(["float", "breathe", "pulse", "wobble", "shake", "drift"].includes(value.operator),+ `${label}.operator is invalid.`);+ finite(value.amount, `${label}.amount`);+ assert(value.amount >= 0, `${label}.amount must be non-negative.`);+ assert(Number.isSafeInteger(value.cycles) && value.cycles > 0 && value.cycles <= 100,+ `${label}.cycles must be inside [1, 100].`);+ direction(value.direction, `${label}.direction`);+ if (value.operator === "drift") assert(value.direction !== undefined, `${label}.drift requires direction.`);+}++export function assertMediaLifecycleMotion(value: MediaLifecycleMotion, durationFrames: number, label: string): void {+ assert(Number.isSafeInteger(durationFrames) && durationFrames > 0, `${label} duration is invalid.`);+ if (value.enter !== undefined) assertMediaEdgeMotion(value.enter, `${label}.enter`);+ if (value.exit !== undefined) assertMediaEdgeMotion(value.exit, `${label}.exit`);+ for (const [index, sustain] of value.sustain.entries()) assertMediaSustainMotion(sustain, `${label}.sustain.${index}`);+ const enter = value.enter?.durationFrames ?? 0;+ const exit = value.exit?.durationFrames ?? 0;+ assert(enter + exit <= durationFrames, `${label} entry and exit durations overlap.`);+}++type MotionState = {+ readonly opacity: number;+ readonly transform: string;+ readonly filter: string;+ readonly clipPath: string;+};++const neutral: MotionState = {+ opacity: 1,+ transform: "none",+ filter: "none",+ clipPath: "inset(0% 0% 0% 0%)",+};++function translate(directionValue: MediaMotionDirection, amount: number): string {+ if (directionValue === "left") return `translateX(${-amount}px)`;+ if (directionValue === "right") return `translateX(${amount}px)`;+ if (directionValue === "up") return `translateY(${-amount}px)`;+ return `translateY(${amount}px)`;+}++function wipe(directionValue: MediaMotionDirection): string {+ if (directionValue === "left") return "inset(0% 0% 0% 100%)";+ if (directionValue === "right") return "inset(0% 100% 0% 0%)";+ if (directionValue === "up") return "inset(0% 0% 100% 0%)";+ return "inset(100% 0% 0% 0%)";+}++function edgeState(value: MediaEdgeMotion): MotionState {+ const amount = value.amount ?? (value.operator === "blur-reveal" ? 16 : value.operator === "scale" ? 0.8 : 100);+ switch (value.operator) {+ case "fade": return { ...neutral, opacity: 0 };+ case "slide": return { ...neutral, transform: translate(value.direction!, amount) };+ case "scale": return { ...neutral, transform: `scale(${amount})` };+ case "pop": return { ...neutral, opacity: 0, transform: `scale(${value.amount ?? 0.7})` };+ case "bounce": return { ...neutral, opacity: 0, transform: `translateY(${value.amount ?? 80}px) scale(0.92)` };+ case "blur-reveal": return { ...neutral, opacity: 0, filter: `blur(${amount}px)` };+ case "wipe": return { ...neutral, clipPath: wipe(value.direction!) };+ case "flip": {+ const axis = value.direction === "left" || value.direction === "right" ? "Y" : "X";+ const sign = value.direction === "left" || value.direction === "up" ? -1 : 1;+ return { ...neutral, opacity: 0, transform: `perspective(800px) rotate${axis}(${sign * (value.amount ?? 90)}deg)` };+ }+ case "spin": return { ...neutral, opacity: 0, transform: `rotate(${value.amount ?? 180}deg) scale(0.8)` };+ }+}++function style(state: MotionState): VisualStyleDeclaration[] {+ return [+ { name: "clip-path", value: state.clipPath },+ { name: "filter", value: state.filter },+ { name: "opacity", value: state.opacity },+ { name: "transform", value: state.transform },+ ];+}++function addState(+ states: Map<number, { readonly state: MotionState; readonly easing?: VisualEasing }>,+ frame: number,+ state: MotionState,+ easing?: VisualEasing,+): void {+ const previous = states.get(frame);+ if (previous !== undefined) {+ assert(JSON.stringify(previous.state) === JSON.stringify(state),+ `Media lifecycle produces conflicting states at frame ${frame}.`);+ }+ states.set(frame, { state, ...(easing === undefined ? {} : { easing }) });+}++export function lifecycleAnimation(value: MediaLifecycleMotion, durationFrames: number): VisualAnimation | undefined {+ assertMediaLifecycleMotion(value, durationFrames, "Media lifecycle");+ if (value.enter === undefined && value.exit === undefined) return undefined;+ const states = new Map<number, { readonly state: MotionState; readonly easing?: VisualEasing }>();+ if (value.enter === undefined) addState(states, 0, neutral);+ else {+ addState(states, 0, edgeState(value.enter), value.enter.easing);+ addState(states, value.enter.durationFrames, neutral);+ }+ if (value.exit === undefined) addState(states, durationFrames, neutral);+ else {+ const exitStart = durationFrames - value.exit.durationFrames;+ addState(states, exitStart, neutral, value.exit.easing);+ addState(states, durationFrames, edgeState(value.exit));+ }+ return {+ keyframes: [...states.entries()].sort(([left], [right]) => left - right).map(([atFrame, item]) => ({+ atFrame,+ ...(item.easing === undefined ? {} : { easing: item.easing }),+ style: style(item.state),+ })),+ };+}++function clean(value: number): number {+ const rounded = Math.round(value * 1_000_000) / 1_000_000;+ return Object.is(rounded, -0) ? 0 : rounded;+}++function sustainTransform(value: MediaSustainMotion, phase: number): string {+ const amount = value.amount * phase;+ switch (value.operator) {+ case "float": return `translateY(${amount}px)`;+ case "breathe": return `scale(${1 + amount})`;+ case "pulse": return `scale(${1 + amount})`;+ case "wobble": return `rotate(${amount}deg)`;+ case "shake": return `translateX(${amount}px)`;+ case "drift": return translate(value.direction!, Math.abs(amount) * (phase < 0 ? -1 : 1));+ }+}++function sustainPhases(value: MediaSustainMotion, durationFrames: number): ReadonlyMap<number, number> {+ const states = new Map<number, number>([[0, 0], [durationFrames, 0]]);+ for (let cycle = 0; cycle < value.cycles; cycle += 1) {+ for (const [phaseOffset, phase] of [[0.25, 1], [0.75, -1], [1, 0]] as const) {+ states.set(Math.min(durationFrames, Math.round(+ ((cycle + phaseOffset) / value.cycles) * durationFrames,+ )), phase);+ }+ }+ return states;+}++function phaseAt(states: ReadonlyMap<number, number>, frame: number): number {+ const entries = [...states.entries()].sort(([left], [right]) => left - right);+ const exact = states.get(frame);+ if (exact !== undefined) return exact;+ const rightIndex = entries.findIndex(([atFrame]) => atFrame > frame);+ const right = entries[rightIndex]!;+ const left = entries[rightIndex - 1]!;+ const progress = (frame - left[0]) / (right[0] - left[0]);+ return clean(left[1] + ((right[1] - left[1]) * progress));+}++export function sustainAnimation(value: MediaSustainMotion, durationFrames: number): VisualAnimation {+ assertMediaSustainMotion(value, "Media sustain");+ assert(Number.isSafeInteger(durationFrames) && durationFrames > 0, "Media sustain duration is invalid.");+ const states = sustainPhases(value, durationFrames);+ return {+ keyframes: [...states.entries()].sort(([left], [right]) => left - right).map(([atFrame, phase]) => ({+ atFrame,+ easing: "linear",+ style: [{+ name: "transform",+ value: phase === 0 ? "none" : sustainTransform(value, phase),+ }],+ })),+ };+}++/** Slice one group-global sustain channel without restarting its phase for each member Present. */+export function sustainAnimationWindow(+ value: MediaSustainMotion,+ outerDurationFrames: number,+ startOffset: number,+ endOffset: number,+): VisualAnimation {+ assertMediaSustainMotion(value, "Media sustain");+ assert(Number.isSafeInteger(outerDurationFrames) && outerDurationFrames > 0,+ "Media sustain outer duration is invalid.");+ assert(Number.isSafeInteger(startOffset) && Number.isSafeInteger(endOffset)+ && startOffset >= 0 && endOffset > startOffset && endOffset <= outerDurationFrames,+ "Media sustain window is invalid.");+ const global = sustainPhases(value, outerDurationFrames);+ const frames = new Set<number>([startOffset, endOffset]);+ for (const frame of global.keys()) {+ if (frame > startOffset && frame < endOffset) frames.add(frame);+ }+ return {+ keyframes: [...frames].sort((left, right) => left - right).map((frame) => {+ const phase = phaseAt(global, frame);+ return {+ atFrame: frame - startOffset,+ easing: "linear" as const,+ style: [{ name: "transform", value: phase === 0 ? "none" : sustainTransform(value, phase) }],+ };+ }),+ };+}++type Cubic = readonly [number, number, number, number];++function cubicCoordinate(t: number, first: number, second: number): number {+ const inverse = 1 - t;+ return (3 * inverse * inverse * t * first) + (3 * inverse * t * t * second) + (t * t * t);+}++function eased(progress: number, easing: VisualEasing): number {+ if (easing === "linear" || progress <= 0 || progress >= 1) return progress;+ const curve: Cubic = easing === "ease-in" ? [0.42, 0, 1, 1]+ : easing === "ease-out" ? [0, 0, 0.58, 1]+ : [0.42, 0, 0.58, 1];+ let low = 0;+ let high = 1;+ for (let iteration = 0; iteration < 24; iteration += 1) {+ const middle = (low + high) / 2;+ if (cubicCoordinate(middle, curve[0], curve[2]) < progress) low = middle;+ else high = middle;+ }+ return cubicCoordinate((low + high) / 2, curve[1], curve[3]);+}++function edgeStateAt(value: MediaEdgeMotion, progressToNeutral: number): MotionState {+ const progress = clean(progressToNeutral);+ const inverse = clean(1 - progress);+ const amount = value.amount ?? (value.operator === "blur-reveal" ? 16 : value.operator === "scale" ? 0.8 : 100);+ switch (value.operator) {+ case "fade": return { ...neutral, opacity: progress };+ case "slide": return { ...neutral, transform: translate(value.direction!, clean(amount * inverse)) };+ case "scale": return { ...neutral, transform: `scale(${clean(amount + ((1 - amount) * progress))})` };+ case "pop": {+ const start = value.amount ?? 0.7;+ return { ...neutral, opacity: progress, transform: `scale(${clean(start + ((1 - start) * progress))})` };+ }+ case "bounce": return {+ ...neutral,+ opacity: progress,+ transform: `translateY(${clean((value.amount ?? 80) * inverse)}px) scale(${clean(0.92 + (0.08 * progress))})`,+ };+ case "blur-reveal": return { ...neutral, opacity: progress, filter: `blur(${clean(amount * inverse)}px)` };+ case "wipe": {+ const percentage = clean(100 * inverse);+ if (value.direction === "left") return { ...neutral, clipPath: `inset(0% 0% 0% ${percentage}%)` };+ if (value.direction === "right") return { ...neutral, clipPath: `inset(0% ${percentage}% 0% 0%)` };+ if (value.direction === "up") return { ...neutral, clipPath: `inset(0% 0% ${percentage}% 0%)` };+ return { ...neutral, clipPath: `inset(${percentage}% 0% 0% 0%)` };+ }+ case "flip": {+ const axis = value.direction === "left" || value.direction === "right" ? "Y" : "X";+ const sign = value.direction === "left" || value.direction === "up" ? -1 : 1;+ return { ...neutral, opacity: progress,+ transform: `perspective(800px) rotate${axis}(${clean(sign * (value.amount ?? 90) * inverse)}deg)` };+ }+ case "spin": return { ...neutral, opacity: progress,+ transform: `rotate(${clean((value.amount ?? 180) * inverse)}deg) scale(${clean(0.8 + (0.2 * progress))})` };+ }+}++function lifecycleStateAt(value: MediaLifecycleMotion, durationFrames: number, frame: number): MotionState {+ if (value.enter !== undefined && frame < value.enter.durationFrames) {+ return edgeStateAt(value.enter, eased(frame / value.enter.durationFrames, value.enter.easing));+ }+ if (value.exit !== undefined) {+ const exitStart = durationFrames - value.exit.durationFrames;+ if (frame > exitStart) {+ const progress = eased((frame - exitStart) / value.exit.durationFrames, value.exit.easing);+ return edgeStateAt(value.exit, 1 - progress);+ }+ }+ return neutral;+}++/** Slice one outer lifecycle over a member Present; entry/exit never restart at member boundaries. */+export function lifecycleAnimationWindow(+ value: MediaLifecycleMotion,+ outerDurationFrames: number,+ startOffset: number,+ endOffset: number,+): VisualAnimation | undefined {+ assertMediaLifecycleMotion(value, outerDurationFrames, "Media lifecycle");+ assert(Number.isSafeInteger(startOffset) && Number.isSafeInteger(endOffset)+ && startOffset >= 0 && endOffset > startOffset && endOffset <= outerDurationFrames,+ "Media lifecycle window is invalid.");+ if (value.enter === undefined && value.exit === undefined) return undefined;+ const frames = new Set<number>([startOffset, endOffset]);+ if (value.enter !== undefined) {+ const end = value.enter.durationFrames;+ for (let frame = Math.max(startOffset, 0); frame <= Math.min(endOffset, end); frame += 1) frames.add(frame);+ }+ if (value.exit !== undefined) {+ const start = outerDurationFrames - value.exit.durationFrames;+ for (let frame = Math.max(startOffset, start); frame <= Math.min(endOffset, outerDurationFrames); frame += 1) frames.add(frame);+ }+ const ordered = [...frames].sort((left, right) => left - right);+ const compact = ordered.filter((frame, index) => {+ if (index === 0 || index === ordered.length - 1) return true;+ const previous = lifecycleStateAt(value, outerDurationFrames, ordered[index - 1]!);+ const current = lifecycleStateAt(value, outerDurationFrames, frame);+ const next = lifecycleStateAt(value, outerDurationFrames, ordered[index + 1]!);+ return JSON.stringify(previous) !== JSON.stringify(current) || JSON.stringify(current) !== JSON.stringify(next);+ });+ return {+ keyframes: compact.map((frame) => ({+ atFrame: frame - startOffset,+ easing: "linear",+ style: style(lifecycleStateAt(value, outerDurationFrames, frame)),+ })),+ };+}++export function samplingAnimation(value: MediaSamplingMotion, durationFrames: number): VisualAnimation {+ assert(value.keyframes.length >= 2, "Media sampling motion requires keyframes.");+ assert(value.keyframes[0]?.atProgress === 0 && value.keyframes.at(-1)?.atProgress === 1,+ "Media sampling motion must cover normalized progress [0, 1].");+ let previous = -1;+ return {+ keyframes: value.keyframes.map((keyframe, index) => {+ const atFrame = Math.round(keyframe.atProgress * durationFrames);+ assert(atFrame > previous,+ `Media sampling keyframe ${index + 1} collapses after frame quantization; use fewer or wider-spaced keyframes.`);+ previous = atFrame;+ return {+ atFrame,+ ...(keyframe.easing === undefined ? {} : { easing: keyframe.easing }),+ style: [{+ name: "transform",+ value: `translate(${keyframe.offsetX}px,${keyframe.offsetY}px) rotate(${keyframe.rotationDeg}deg) scale(${keyframe.zoom})`,+ }],+ };+ }),+ };+}diff --git a/packages/media-track/src/presentation.ts b/packages/media-track/src/presentation.tsnew file mode 100644index 00000000..ed654df7--- /dev/null+++ b/packages/media-track/src/presentation.ts@@ -0,0 +1,35 @@+import { assertSpatialPath } from "@narratage/spatial";++import type { MediaFramePresentation } from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function finite(value: number, label: string): void {+ assert(Number.isFinite(value), `${label} must be finite.`);+}++export function assertMediaFramePresentation(value: MediaFramePresentation, label: string): void {+ assert(["none", "frame", "rounded", "path"].includes(value.clip.kind), `${label}.clip is invalid.`);+ if (value.clip.kind === "rounded") {+ finite(value.clip.radiusPx, `${label}.clip.radiusPx`);+ assert(value.clip.radiusPx >= 0, `${label}.clip.radiusPx is invalid.`);+ }+ if (value.clip.kind === "path") assertSpatialPath(value.clip.path);+ for (const [name, padding] of Object.entries(value.padding)) {+ finite(padding, `${label}.padding.${name}`);+ assert(padding >= 0, `${label}.padding.${name} is invalid.`);+ }+ if (value.border !== undefined) {+ finite(value.border.widthPx, `${label}.border.widthPx`);+ assert(value.border.widthPx >= 0 && ["solid", "dashed", "dotted"].includes(value.border.style)+ && value.border.color.length > 0, `${label}.border is invalid.`);+ }+ for (const [index, shadow] of value.shadows.entries()) {+ for (const [name, number] of Object.entries(shadow).filter(([, item]) => typeof item === "number")) {+ finite(number as number, `${label}.shadows.${index}.${name}`);+ }+ assert(shadow.blurPx >= 0 && shadow.color.length > 0, `${label}.shadows.${index} is invalid.`);+ }+}diff --git a/packages/media-track/src/program.ts b/packages/media-track/src/program.tsindex 5976a2d4..929c86e4 100644--- a/packages/media-track/src/program.ts+++ b/packages/media-track/src/program.ts@@ -1,33 +1,113 @@-import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";+import {+ assertAudioTrackIdentity,+ assertVisualTrackIdentity,+ sealAudioTrack,+ sealVisualTrack,+} from "@narratage/composition";+import type { AudioClip, AudioTrack, VisualTrack } from "@narratage/composition";+import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";+import {+ assertProgramSpaceIdentity,+ programFrameSampleBoundary,+ programSpaceFrameCount,+ programSpaceSampleFrames,+} from "@narratage/program-space";import type { ProgramSpace } from "@narratage/program-space";-import { sealVisualTrack } from "@narratage/composition";-import type { VisualElement, VisualStyleDeclaration, VisualTrack } from "@narratage/composition";import { canonicalize, digestOf, isDigest } from "@narratage/protocol";-import type { BlobRef } from "@narratage/protocol";+import type { CompleteSemanticMap } from "@narratage/semantic-map";+import { assertCanvasSpace, assertSpatialFrame } from "@narratage/spatial";+import type { CanvasSpace } from "@narratage/spatial";+import { assertSpatialPath } from "@narratage/spatial";+import type { SpatialPath } from "@narratage/spatial";import {- assertContentFit,- assertIntrinsicExtent,- assertSpatialFrame,- fitContent,-} from "@narratage/spatial";-import type { ContentFit, IntrinsicExtent, SpatialFrame } from "@narratage/spatial";-import { projectProgramWindow } from "@narratage/temporal";+ locateMomentOccurrences,+ locateSelectionOccurrences,+ projectMomentWindows,+ projectProgramWindow,+ projectSelectionWindows,+} from "@narratage/temporal";+import type { ProjectedOccurrence, TemporalDuration, TemporalPointExpression } from "@narratage/temporal";+import {+ assertMediaIdentity,+ assertMediaLayerSet,+ assertMediaVisualOccupancy,+ assertMediaVisualSource,+ assertMediaVisualTrim,+} from "./layers.js";+import { lowerMediaItemElements } from "./lower.js";+import { assertMediaLifecycleMotion, resolveMediaLifecycleMotion } from "./motion.js";+import { assertMediaFramePresentation } from "./presentation.js";+import { assertMediaSoundSet } from "./sounds.js";+import {+ assertMediaHandoffSpec,+ assertMediaSequenceMemberSet,+ assertMediaSequenceSpec,+ lowerMediaSequencePresents,+ resolveMediaSequence,+ sealMediaSequenceSpec,+} from "./sequence.js";import type {+ MediaFramePresentation,MediaItemProgram,- MediaStillItemSpec,+ MediaItemSpec,+ MediaLayerSet,+ MediaSampleLayerProgram,+ MediaSequenceProgram,+ MediaSequenceMemberSet,+ MediaSequenceSpec,+ MediaSoundSource,+ MediaSoundSet,MediaTrackHeader,MediaTrackProgram,MediaTrackSet,} from "./types.js";export const mediaTrackImplementationDigests = {+ createLayers: digestOf("@narratage/media-track/create-layers@1"),+ appendPaintLayer: digestOf("@narratage/media-track/append-paint-layer@1"),+ appendStillLayer: digestOf("@narratage/media-track/append-still-layer@1"),+ appendTimedLayer: digestOf("@narratage/media-track/append-timed-layer@1"),+ appendSurfaceLayer: digestOf("@narratage/media-track/append-surface-layer@1"),+ createSounds: digestOf("@narratage/media-track/create-sounds@1"),+ appendSound: digestOf("@narratage/media-track/append-sound@1"),createSet: digestOf("@narratage/media-track/create-set@1"),- appendFullStill: digestOf("@narratage/media-track/append-full-still@1"),+ appendProgramItem: digestOf("@narratage/media-track/append-program-item@1"),+ appendSelectionItem: digestOf("@narratage/media-track/append-selection-item@1"),+ appendMomentItem: digestOf("@narratage/media-track/append-moment-item@1"),+ bindItemClipPath: digestOf("@narratage/media-track/bind-item-clip-path@1"),+ bindSequenceClipPath: digestOf("@narratage/media-track/bind-sequence-clip-path@1"),+ createMembers: digestOf("@narratage/media-track/create-members@1"),+ appendMomentMember: digestOf("@narratage/media-track/append-moment-member@1"),+ appendSelectionStartMember: digestOf("@narratage/media-track/append-selection-start-member@1"),+ appendSelectionEndMember: digestOf("@narratage/media-track/append-selection-end-member@1"),+ appendSequenceProgramEnd: digestOf("@narratage/media-track/append-sequence-program-end@1"),+ appendSequenceUntilMoment: digestOf("@narratage/media-track/append-sequence-until-moment@1"),+ appendSequenceUntilSelectionStart: digestOf("@narratage/media-track/append-sequence-until-selection-start@1"),+ appendSequenceUntilSelectionEnd: digestOf("@narratage/media-track/append-sequence-until-selection-end@1"),finalize: digestOf("@narratage/media-track/finalize@1"),- render: digestOf("@narratage/media-track/render@1"),+ projectVisual: digestOf("@narratage/media-track/project-visual@1"),+ projectAudio: digestOf("@narratage/media-track/project-audio@1"),} as const;+export function bindMediaItemClipPath(spec: MediaItemSpec, path: SpatialPath): MediaItemSpec {+ assertMediaItemSpec(spec);+ assertSpatialPath(path);+ return sealMediaItemSpec({+ ...spec,+ presentation: { ...spec.presentation, clip: { kind: "path", path: structuredClone(path) } },+ });+}++export function bindMediaSequenceClipPath(spec: MediaSequenceSpec, path: SpatialPath): MediaSequenceSpec {+ assertMediaSequenceSpec(spec);+ assertSpatialPath(path);+ return sealMediaSequenceSpec({+ ...spec,+ presentation: { ...spec.presentation, clip: { kind: "path", path: structuredClone(path) } },+ });+}+export const mediaTrackValidatorDigests = {program: digestOf("@narratage/media-track/validate-program@1"),} as const;@@ -36,17 +116,51 @@ function assert(condition: unknown, message: string): asserts condition {if (!condition) throw new Error(message);}-function assertIdentity(value: string, label: string): void {- assert(/^[A-Za-z][A-Za-z0-9_.:-]{0,191}$/u.test(value), `${label} is invalid.`);+function finite(value: number, label: string): void {+ assert(Number.isFinite(value), `${label} must be finite.`);+}++function assertDuration(value: TemporalDuration, label: string, signed: boolean): void {+ if (value.unit === "seconds") {+ assert(Number.isSafeInteger(value.numerator) && (signed || value.numerator >= 0)+ && Number.isSafeInteger(value.denominator) && value.denominator > 0, `${label} is invalid.`);+ } else {+ assert(Number.isSafeInteger(value.value) && (signed || value.value >= 0), `${label} is invalid.`);+ }+}++function assertPoint(value: TemporalPointExpression, label: string): void {+ assert(["program.start", "program.end", "selection.start", "selection.end", "moment.cue", "absolute"].includes(value.ref),+ `${label}.ref is invalid.`);+ if (value.ref === "absolute") assertDuration(value.at, `${label}.at`, false);+ else if (value.offset !== undefined) assertDuration(value.offset, `${label}.offset`, true);}-function assertPresentation(value: MediaStillItemSpec["presentation"], label: string): void {- assert(value.clip === "frame" && value.fill.kind === "transparent", `${label} is unsupported.`);+function assertSound(value: MediaSoundSource, label: string): void {+ assert(value.artifact.kind === "blob" && isDigest(value.artifact.digest)+ && value.artifact.mediaType.startsWith("audio/")+ && Number.isSafeInteger(value.artifact.size) && value.artifact.size >= 0+ && Number.isSafeInteger(value.sampleFrames) && value.sampleFrames > 0, `${label} is invalid.`);}-function assertArtifact(value: MediaItemProgram["layers"][number]["artifact"], label: string): void {- assert(isDigest(value.digest) && Number.isSafeInteger(value.size) && value.size >= 0- && value.mediaType.startsWith("image/") && value.durationSec === 0, `${label} is not a still-image Artifact.`);+export function assertMediaItemSpec(value: MediaItemSpec): void {+ assert(value.contract === "svml.media-item-spec@1", "Unsupported MediaItemSpec contract.");+ assertMediaIdentity(value.id, "MediaItemSpec.id");+ assertPoint(value.projection.start, "MediaItemSpec.projection.start");+ assertPoint(value.projection.end, "MediaItemSpec.projection.end");+ assert(value.expansion.kind === "one" || value.expansion.kind === "each", "MediaItemSpec.expansion is invalid.");+ assertMediaFramePresentation(value.presentation, "MediaItemSpec.presentation");+ assert(Number.isSafeInteger(value.stackingOrder), "MediaItemSpec.stackingOrder must be an integer.");+ if (value.sourceAudio !== undefined) {+ assertMediaIdentity(value.sourceAudio.fromLayer, "MediaItemSpec.sourceAudio.fromLayer");+ finite(value.sourceAudio.gain, "MediaItemSpec.sourceAudio.gain");+ assert(value.sourceAudio.gain >= 0 && value.sourceAudio.gain <= 64, "MediaItemSpec.sourceAudio.gain is invalid.");+ }+}++export function sealMediaItemSpec(value: MediaItemSpec): MediaItemSpec {+ assertMediaItemSpec(value);+ return canonicalize(value) as unknown as MediaItemSpec;}export function sealMediaTrackHeader(value: MediaTrackHeader): MediaTrackHeader {@@ -56,179 +170,540 @@ export function sealMediaTrackHeader(value: MediaTrackHeader): MediaTrackHeaderexport function assertMediaTrackHeader(value: MediaTrackHeader): void {assert(value.contract === "svml.media-track-header@1", "Unsupported MediaTrackHeader contract.");- assertIdentity(value.id, "MediaTrackHeader.id");-}--export function sealMediaStillItemSpec(value: MediaStillItemSpec): MediaStillItemSpec {- assertMediaStillItemSpec(value);- return canonicalize(value) as unknown as MediaStillItemSpec;-}--export function assertMediaStillItemSpec(value: MediaStillItemSpec): void {- assert(value.contract === "svml.media-still-item-spec@1", "Unsupported MediaStillItemSpec contract.");- assertIdentity(value.id, "MediaStillItemSpec.id");- assert(Number.isSafeInteger(value.stackingOrder), "MediaStillItemSpec.stackingOrder must be an integer.");- assertPresentation(value.presentation, "MediaStillItemSpec.presentation");+ assertMediaIdentity(value.id, "MediaTrackHeader.id");}export function createMediaTrackSet(): MediaTrackSet {- return { contract: "svml.media-track-set@1", items: [] };+ return { contract: "svml.media-track-set@1", items: [], sequences: [] };}export function assertMediaTrackSet(value: MediaTrackSet): void {- assert(value.contract === "svml.media-track-set@1" && Array.isArray(value.items), "MediaTrackSet is invalid.");+ assert(value.contract === "svml.media-track-set@1" && Array.isArray(value.items)+ && Array.isArray(value.sequences), "MediaTrackSet is invalid.");}-export function appendFullStillMediaItem(+function realizedItems(set: MediaTrackSet,header: MediaTrackHeader,space: ProgramSpace,- source: BlobRef,- extent: IntrinsicExtent,- frame: SpatialFrame,- fit: ContentFit,- spec: MediaStillItemSpec,+ canvas: CanvasSpace,+ layers: MediaLayerSet,+ frame: MediaItemProgram["frame"],+ spec: MediaItemSpec,+ sounds: MediaSoundSet,+ occurrences: readonly ProjectedOccurrence[],): MediaTrackSet {assertMediaTrackSet(set);assertMediaTrackHeader(header);assertProgramSpaceIdentity(space);- assert(source.kind === "blob" && source.mediaType.startsWith("image/"), "Media still source must be an image BlobArtifact.");- assertIntrinsicExtent(extent);+ assertCanvasSpace(canvas);+ assertMediaLayerSet(layers);+ assert(layers.layers.length > 0, `Media Item ${spec.id} requires at least one layer.`);assertSpatialFrame(frame);- assertContentFit(fit);- assertMediaStillItemSpec(spec);- assert(!set.items.some((item) => item.id === spec.id), `Media Track already contains Item ${spec.id}.`);- const projected = projectProgramWindow({+ assertMediaItemSpec(spec);+ assertMediaSoundSet(sounds);+ assert(!sounds.sounds.some((sound) => sound.trigger.kind === "handoff"),+ `Media Item ${spec.id} cannot own a Handoff sound.`);+ assert(frame.widthPx > spec.presentation.padding.leftPx + spec.presentation.padding.rightPx+ && frame.heightPx > spec.presentation.padding.topPx + spec.presentation.padding.bottomPx,+ `Media Item ${spec.id} padding consumes its complete Placement Frame.`);+ if (spec.sourceAudio !== undefined) {+ const selected = layers.layers.find((layer) => layer.id === spec.sourceAudio!.fromLayer);+ assert(selected?.kind === "sample" && selected.source.kind === "timed" && selected.source.audio !== undefined,+ `Media Item ${spec.id} source-audio layer is absent or has no normalized audio.`);+ }+ const additions = occurrences.map((occurrence) => {+ const duration = occurrence.span.endFrameExclusive - occurrence.span.startFrame;+ assertMediaLifecycleMotion(spec.motion, duration, `Media Item ${spec.id} motion`);+ const motion = resolveMediaLifecycleMotion(spec.motion, frame, canvas);+ return {+ id: occurrence.id,+ sourceOccurrenceId: occurrence.sourceOccurrenceId,+ span: { ...occurrence.span },+ frame: { ...frame },+ presentation: structuredClone(spec.presentation),+ layers: structuredClone(layers.layers),+ motion,+ stacking: { order: spec.stackingOrder, tieBreak: `${header.id}:${occurrence.id}` },+ ...(spec.sourceAudio === undefined ? {} : { sourceAudio: { ...spec.sourceAudio } }),+ sounds: structuredClone(sounds.sounds),+ } satisfies MediaItemProgram;+ });+ const ids = new Set([...set.items.map((item) => item.id), ...set.sequences.map((item) => item.id)]);+ for (const item of additions) {+ assert(!ids.has(item.id), `Media Track ${header.id} already contains ${item.id}.`);+ ids.add(item.id);+ }+ return { ...set, items: [...set.items, ...additions] };+}++export function appendProgramMediaItem(+ set: MediaTrackSet,+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ layers: MediaLayerSet,+ frame: MediaItemProgram["frame"],+ spec: MediaItemSpec,+ sounds: MediaSoundSet,+): MediaTrackSet {+ assert(spec.expansion.kind === "one", `Program Media Item ${spec.id} must use one occurrence.`);+ return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, [projectProgramWindow({itemId: spec.id,space,- projection: { start: { ref: "program.start" }, end: { ref: "program.end" } },- });- const item: MediaItemProgram = {- id: spec.id,- span: projected.span,- frame: { ...frame },- presentation: structuredClone(spec.presentation),- layers: [{- id: `${spec.id}:content`,- kind: "still",- artifact: {- digest: source.digest,- size: source.size,- mediaType: source.mediaType,- durationSec: 0,- },- extent: { ...extent },- fit: structuredClone(fit),- }],- stacking: { order: spec.stackingOrder, tieBreak: `${header.id}:${spec.id}` },- };- return { contract: "svml.media-track-set@1", items: [...set.items, item] };+ projection: spec.projection,+ })]);+}++export function appendSelectionMediaItem(+ set: MediaTrackSet,+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ layers: MediaLayerSet,+ frame: MediaItemProgram["frame"],+ map: CompleteSemanticMap,+ selection: NarrativeSelectionRef,+ spec: MediaItemSpec,+ sounds: MediaSoundSet,+): MediaTrackSet {+ return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, projectSelectionWindows({+ itemId: spec.id, map, selection, space, expansion: spec.expansion, projection: spec.projection,+ }));+}++export function appendMomentMediaItem(+ set: MediaTrackSet,+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ layers: MediaLayerSet,+ frame: MediaItemProgram["frame"],+ map: CompleteSemanticMap,+ moment: NarrativeMomentRef,+ spec: MediaItemSpec,+ sounds: MediaSoundSet,+): MediaTrackSet {+ return realizedItems(set, header, space, canvas, layers, frame, spec, sounds, projectMomentWindows({+ itemId: spec.id, map, moment, space, expansion: spec.expansion, projection: spec.projection,+ }));+}++function assertSampleLayerForSpace(layer: MediaSampleLayerProgram, space: ProgramSpace, label: string): void {+ assertMediaVisualSource(layer.source, `${label}.source`);+ const timing = layer.source.kind === "timed" ? { frameRate: layer.source.frameRate, frameCount: layer.source.frameCount }+ : layer.source.kind === "surface" && layer.source.surface.timing.kind === "frames"+ ? layer.source.surface.timing : undefined;+ if (timing === undefined) return;+ assert(timing.frameRate.numerator === space.frameRate.numerator+ && timing.frameRate.denominator === space.frameRate.denominator,+ `${label} is not normalized to ProgramSpace frame rate.`);+ if (layer.trim !== undefined) assertMediaVisualTrim(layer.trim, timing.frameCount, `${label}.trim`);+ assert(layer.occupancy !== undefined, `${label} requires occupancy.`);+ assertMediaVisualOccupancy(layer.occupancy, `${label}.occupancy`);+}++function assertItem(item: MediaItemProgram, space: ProgramSpace, label: string): void {+ assertMediaIdentity(item.id, `${label}.id`);+ assert(item.sourceOccurrenceId.length > 0, `${label}.sourceOccurrenceId is empty.`);+ assert(Number.isSafeInteger(item.span.startFrame) && Number.isSafeInteger(item.span.endFrameExclusive)+ && item.span.startFrame >= 0 && item.span.endFrameExclusive > item.span.startFrame+ && item.span.endFrameExclusive <= programSpaceFrameCount(space), `${label}.span is invalid.`);+ assertSpatialFrame(item.frame);+ assertMediaFramePresentation(item.presentation, `${label}.presentation`);+ assertMediaLifecycleMotion(item.motion, item.span.endFrameExclusive - item.span.startFrame, `${label}.motion`);+ assert(item.motion.enter?.origin === undefined && item.motion.exit?.origin === undefined,+ `${label}.motion contains an unresolved Canvas origin.`);+ assert(Number.isSafeInteger(item.stacking.order) && item.stacking.tieBreak.length > 0, `${label}.stacking is invalid.`);+ assert(item.layers.length > 0, `${label} requires layers.`);+ const layerSet: MediaLayerSet = { contract: "svml.media-layer-set@1", layers: item.layers };+ assertMediaLayerSet(layerSet);+ for (const layer of item.layers) if (layer.kind === "sample") assertSampleLayerForSpace(layer, space, `${label}.${layer.id}`);+ if (item.sourceAudio !== undefined) {+ const selected = item.layers.find((layer) => layer.id === item.sourceAudio!.fromLayer);+ assert(selected?.kind === "sample" && selected.source.kind === "timed" && selected.source.audio !== undefined,+ `${label}.sourceAudio is invalid.`);+ }+ assertMediaSoundSet({ contract: "svml.media-sound-set@1", sounds: item.sounds });+ assert(!item.sounds.some((sound) => sound.trigger.kind === "handoff"), `${label} owns a Handoff sound.`);+}++export function appendMediaSequence(+ set: MediaTrackSet,+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ members: MediaSequenceMemberSet,+ frame: MediaSequenceProgram["frame"],+ spec: MediaSequenceSpec,+ sounds: MediaSoundSet,+ terminalFrame: number,+): MediaTrackSet {+ assertMediaTrackSet(set);+ assertMediaSequenceMemberSet(members);+ assertMediaSequenceSpec(spec);+ const sequence = resolveMediaSequence(header, space, canvas, members, frame, spec, sounds, terminalFrame);+ assert(!set.items.some((item) => item.id === sequence.id)+ && !set.sequences.some((item) => item.id === sequence.id),+ `Media Track ${header.id} already contains ${sequence.id}.`);+ return { ...set, sequences: [...set.sequences, sequence] };+}++function exactlyOneTerminalFrame(values: readonly number[], label: string): number {+ assert(values.length === 1, `${label} requires exactly one occurrence; received ${values.length}.`);+ return values[0]!;+}++export function appendMediaSequenceUntilProgramEnd(+ set: MediaTrackSet,+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ members: MediaSequenceMemberSet,+ frame: MediaSequenceProgram["frame"],+ spec: MediaSequenceSpec,+ sounds: MediaSoundSet,+): MediaTrackSet {+ return appendMediaSequence(set, header, space, canvas, members, frame, spec, sounds, programSpaceFrameCount(space));+}++export function appendMediaSequenceUntilMoment(+ set: MediaTrackSet,+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ members: MediaSequenceMemberSet,+ frame: MediaSequenceProgram["frame"],+ spec: MediaSequenceSpec,+ sounds: MediaSoundSet,+ map: CompleteSemanticMap,+ moment: NarrativeMomentRef,+): MediaTrackSet {+ const terminalFrame = exactlyOneTerminalFrame(+ locateMomentOccurrences(map, moment, space).map((occurrence) => occurrence.cue.frame),+ `Media Sequence ${spec.id} terminal Moment`,+ );+ return appendMediaSequence(set, header, space, canvas, members, frame, spec, sounds, terminalFrame);+}++export function appendMediaSequenceUntilSelection(+ set: MediaTrackSet,+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ members: MediaSequenceMemberSet,+ frame: MediaSequenceProgram["frame"],+ spec: MediaSequenceSpec,+ sounds: MediaSoundSet,+ map: CompleteSemanticMap,+ selection: NarrativeSelectionRef,+ boundary: "start" | "end",+): MediaTrackSet {+ const terminalFrame = exactlyOneTerminalFrame(+ locateSelectionOccurrences(map, selection, space).map((occurrence) =>+ boundary === "start" ? occurrence.start.frame : occurrence.end.frame),+ `Media Sequence ${spec.id} terminal Selection`,+ );+ return appendMediaSequence(set, header, space, canvas, members, frame, spec, sounds, terminalFrame);+}++function assertSequence(sequence: MediaSequenceProgram, space: ProgramSpace, label: string): void {+ assertMediaIdentity(sequence.id, `${label}.id`);+ assert(sequence.span.startFrame >= 0 && sequence.span.endFrameExclusive <= programSpaceFrameCount(space)+ && sequence.span.endFrameExclusive > sequence.span.startFrame+ && sequence.terminalFrame === sequence.span.endFrameExclusive, `${label}.span is invalid.`);+ assertSpatialFrame(sequence.frame);+ assertMediaFramePresentation(sequence.presentation, `${label}.presentation`);+ assertMediaLifecycleMotion(sequence.motion, sequence.span.endFrameExclusive - sequence.span.startFrame, `${label}.motion`);+ assert(sequence.motion.enter?.origin === undefined && sequence.motion.exit?.origin === undefined,+ `${label}.motion contains an unresolved Canvas origin.`);+ assert(sequence.members.length >= 2 && sequence.handoffs.length === sequence.members.length - 1,+ `${label} topology is invalid.`);+ for (const [index, member] of sequence.members.entries()) {+ assertMediaIdentity(member.id, `${label}.members.${index}.id`);+ assert(member.activationFrame === member.logicalSpan.startFrame+ && member.logicalSpan.startFrame >= sequence.span.startFrame+ && member.logicalSpan.endFrameExclusive <= sequence.terminalFrame+ && member.logicalSpan.endFrameExclusive > member.logicalSpan.startFrame,+ `${label}.members.${index}.logicalSpan is invalid.`);+ assert(member.visualSpan.startFrame <= member.logicalSpan.startFrame+ && member.visualSpan.endFrameExclusive >= member.logicalSpan.endFrameExclusive,+ `${label}.members.${index}.visualSpan does not contain its logical phase.`);+ const layerSet: MediaLayerSet = { contract: "svml.media-layer-set@1", layers: member.layers };+ assertMediaLayerSet(layerSet);+ for (const layer of member.layers) if (layer.kind === "sample") {+ assertSampleLayerForSpace(layer, space, `${label}.members.${index}.${layer.id}`);+ }+ if (member.sourceAudio !== undefined) {+ const selected = member.layers.find((layer) => layer.id === member.sourceAudio!.fromLayer);+ assert(selected?.kind === "sample" && selected.source.kind === "timed" && selected.source.audio !== undefined,+ `${label}.members.${index}.sourceAudio is invalid.`);+ }+ }+ for (const [index, handoff] of sequence.handoffs.entries()) {+ assertMediaHandoffSpec({ ...handoff, contract: "svml.media-handoff-spec@1" });+ assert(handoff.fromMemberId === sequence.members[index]!.id+ && handoff.toMemberId === sequence.members[index + 1]!.id+ && handoff.span.startFrame <= sequence.members[index]!.logicalSpan.endFrameExclusive+ && handoff.span.endFrameExclusive >= sequence.members[index + 1]!.logicalSpan.startFrame,+ `${label}.handoffs.${index} is disconnected.`);+ }+ assertMediaSoundSet({ contract: "svml.media-sound-set@1", sounds: sequence.sounds });+ for (const sound of sequence.sounds) {+ if (sound.trigger.kind === "handoff") {+ const { handoffId } = sound.trigger;+ assert(sequence.handoffs.some((handoff) => handoff.id === handoffId),+ `${label} sound ${sound.id} targets an unknown Handoff.`);+ }+ }}function normalizeProgram(value: MediaTrackProgram): MediaTrackProgram {return {contract: "svml.media-track-program@1",id: value.id,- items: [...value.items]- .map((item) => structuredClone(item))- .sort((left, right) => left.id.localeCompare(right.id)),+ items: [...value.items].map((item) => structuredClone(item)).sort((left, right) => left.id.localeCompare(right.id)),+ sequences: [...value.sequences].map((item) => structuredClone(item)).sort((left, right) => left.id.localeCompare(right.id)),};}-export function sealMediaTrackProgram(value: MediaTrackProgram): MediaTrackProgram {+export function sealMediaTrackProgram(value: MediaTrackProgram, space: ProgramSpace): MediaTrackProgram {const normalized = normalizeProgram(value);- assertMediaTrackProgram(normalized);+ assertMediaTrackProgramIdentity(normalized, space);return canonicalize(normalized) as unknown as MediaTrackProgram;}-export function finalizeMediaTrack(set: MediaTrackSet, header: MediaTrackHeader): MediaTrackProgram {+export function finalizeMediaTrack(set: MediaTrackSet, header: MediaTrackHeader, space: ProgramSpace): MediaTrackProgram {assertMediaTrackSet(set);assertMediaTrackHeader(header);- assert(set.items.length > 0, "Media Track requires at least one Item.");- return sealMediaTrackProgram({ contract: "svml.media-track-program@1", id: header.id, items: set.items });+ assert(set.items.length + set.sequences.length > 0, "Media Track requires at least one Item or Sequence.");+ return sealMediaTrackProgram({+ contract: "svml.media-track-program@1",+ id: header.id,+ items: set.items,+ sequences: set.sequences,+ }, space);}export function assertMediaTrackProgram(value: MediaTrackProgram): void {assert(value.contract === "svml.media-track-program@1", "Unsupported MediaTrackProgram contract.");- assertIdentity(value.id, "MediaTrackProgram.id");- assert(value.items.length > 0, "MediaTrackProgram requires at least one Item.");- const ids = new Set<string>();- for (const item of value.items) {- assertIdentity(item.id, "MediaItemProgram.id");- assert(!ids.has(item.id), `MediaTrackProgram contains duplicate Item ${item.id}.`);- ids.add(item.id);- assert(Number.isSafeInteger(item.span.startFrame) && Number.isSafeInteger(item.span.endFrameExclusive)- && item.span.startFrame >= 0 && item.span.endFrameExclusive > item.span.startFrame,- `Media Item ${item.id} span is invalid.`);- assertSpatialFrame(item.frame);- assertPresentation(item.presentation, `Media Item ${item.id} presentation`);- assert(Number.isSafeInteger(item.stacking.order) && item.stacking.tieBreak.length > 0,- `Media Item ${item.id} stacking is invalid.`);- assert(item.layers.length > 0, `Media Item ${item.id} requires at least one layer.`);- const layerIds = new Set<string>();- for (const layer of item.layers) {- assertIdentity(layer.id, `Media Item ${item.id} layer id`);- assert(!layerIds.has(layer.id), `Media Item ${item.id} contains duplicate layer ${layer.id}.`);- layerIds.add(layer.id);- assert(layer.kind === "still", `Media Item ${item.id} contains an unsupported layer.`);- assertArtifact(layer.artifact, `Media Item ${item.id} layer ${layer.id}`);- assertIntrinsicExtent(layer.extent);- assertContentFit(layer.fit);- }- }+ assertMediaIdentity(value.id, "MediaTrackProgram.id");+ assert(Array.isArray(value.items) && Array.isArray(value.sequences)+ && value.items.length + value.sequences.length > 0, "MediaTrackProgram is empty.");}export function assertMediaTrackProgramIdentity(value: MediaTrackProgram, space: ProgramSpace): void {assertMediaTrackProgram(value);assertProgramSpaceIdentity(space);- const totalFrames = programSpaceFrameCount(space);+ const ids = new Set<string>();for (const item of value.items) {- assert(item.span.endFrameExclusive <= totalFrames, `Media Item ${item.id} is outside ProgramSpace.`);+ assert(!ids.has(item.id), `MediaTrackProgram repeats ${item.id}.`);+ ids.add(item.id);+ assertItem(item, space, `Media Item ${item.id}`);+ }+ for (const sequence of value.sequences) {+ assert(!ids.has(sequence.id), `MediaTrackProgram repeats ${sequence.id}.`);+ ids.add(sequence.id);+ assertSequence(sequence, space, `Media Sequence ${sequence.id}`);}}-function px(value: number): string { return `${value}px`; }--function itemElements(item: MediaItemProgram): readonly VisualElement[] {- const rootStyle: VisualStyleDeclaration[] = [- { name: "height", value: px(item.frame.heightPx) },- { name: "left", value: px(item.frame.xPx) },- { name: "overflow", value: "hidden" },- { name: "position", value: "absolute" },- { name: "top", value: px(item.frame.yPx) },- { name: "width", value: px(item.frame.widthPx) },- ];- const media = item.layers.map((layer, index) => {- const fitted = fitContent(item.frame, layer.extent, layer.fit).contentFrame;- return {- id: layer.id,- parent: "root",- order: index + 1,- kind: "image" as const,- artifact: layer.artifact,- style: [- { name: "height", value: px(fitted.heightPx) },- { name: "left", value: px(fitted.xPx - item.frame.xPx) },- { name: "position", value: "absolute" },- { name: "top", value: px(fitted.yPx - item.frame.yPx) },- { name: "width", value: px(fitted.widthPx) },- ],- } satisfies VisualElement;- });- return [{ id: "root", order: 0, kind: "box", style: rootStyle }, ...media];-}--export function renderMediaTrack(space: ProgramSpace, program: MediaTrackProgram): VisualTrack {+export function projectMediaVisualTrack(space: ProgramSpace, program: MediaTrackProgram): VisualTrack {assertMediaTrackProgramIdentity(program, space);- return sealVisualTrack({+ const track = sealVisualTrack({contract: "svml.visual-track@1",visualIr: "svml.visual-ir@1",id: program.id,- presents: program.items.map((item) => ({+ presents: [+ ...program.items.map((item) => ({id: item.id,span: { ...item.span },stacking: { ...item.stacking },- elements: itemElements(item),- })),+ elements: lowerMediaItemElements(item, space),+ })),+ ...program.sequences.flatMap((sequence) => lowerMediaSequencePresents(sequence, space)),+ ],+ });+ assertVisualTrackIdentity(track, space);+ return track;+}++function sourceSampleBoundary(frame: number, frameCount: number, sampleFrames: number): number {+ const numerator = BigInt(frame) * BigInt(sampleFrames);+ const denominator = BigInt(frameCount);+ const rounded = (numerator * 2n + denominator) / (denominator * 2n);+ assert(rounded <= BigInt(Number.MAX_SAFE_INTEGER), "Media source sample boundary exceeds safe arithmetic.");+ return Number(rounded);+}++function sourceAudioClip(+ value: Pick<MediaItemProgram, "id" | "span" | "layers" | "sourceAudio">,+ space: ProgramSpace,+ fades: { readonly inSamples: number; readonly outSamples: number } = { inSamples: 0, outSamples: 0 },+): AudioClip | undefined {+ if (value.sourceAudio === undefined) return undefined;+ const layer = value.layers.find((candidate): candidate is MediaSampleLayerProgram =>+ candidate.kind === "sample" && candidate.id === value.sourceAudio!.fromLayer)!;+ assert(layer.source.kind === "timed" && layer.source.audio !== undefined && layer.occupancy !== undefined,+ `Media projection ${value.id} selected audio from an invalid layer.`);+ const source = layer.source;+ const audio = source.audio;+ assert(audio !== undefined, `Media projection ${value.id} selected audio from an invalid layer.`);+ const trim = layer.trim ?? { startFrame: 0, endFrameExclusive: source.frameCount };+ const sourceStart = sourceSampleBoundary(trim.startFrame, source.frameCount, audio.sampleFrames);+ const sourceEnd = sourceSampleBoundary(trim.endFrameExclusive, source.frameCount, audio.sampleFrames);+ const targetWindowStart = programFrameSampleBoundary(space, value.span.startFrame, 48_000);+ const targetWindowEnd = programFrameSampleBoundary(space, value.span.endFrameExclusive, 48_000);+ const targetFrames = value.span.endFrameExclusive - value.span.startFrame;+ const sourceFrames = trim.endFrameExclusive - trim.startFrame;+ let targetStart = targetWindowStart;+ let targetEnd = targetWindowEnd;+ let clipSourceStart = sourceStart;+ let clipSourceEnd = sourceEnd;+ let loop = false;+ let phaseSample = 0;+ let playbackRate = 1;+ if (layer.occupancy.mode === "once" || layer.occupancy.mode === "hold") {+ const playedFrames = Math.min(targetFrames, sourceFrames);+ if (layer.occupancy.align === "start") {+ targetEnd = programFrameSampleBoundary(space, value.span.startFrame + playedFrames, 48_000);+ clipSourceEnd = sourceSampleBoundary(trim.startFrame + playedFrames, source.frameCount, audio.sampleFrames);+ } else {+ targetStart = programFrameSampleBoundary(space, value.span.endFrameExclusive - playedFrames, 48_000);+ clipSourceStart = sourceSampleBoundary(trim.endFrameExclusive - playedFrames, source.frameCount, audio.sampleFrames);+ }+ } else if (layer.occupancy.mode === "loop") {+ loop = true;+ if (layer.occupancy.align === "end") {+ const phaseFrames = (sourceFrames - (targetFrames % sourceFrames)) % sourceFrames;+ phaseSample = sourceSampleBoundary(trim.startFrame + phaseFrames, source.frameCount, audio.sampleFrames) - sourceStart;+ }+ } else {+ playbackRate = (sourceEnd - sourceStart) / (targetWindowEnd - targetWindowStart);+ }+ const audible = targetEnd - targetStart;+ assert(fades.inSamples <= audible && fades.outSamples <= audible,+ `Media projection ${value.id} source audio is too short for its authored crossfade.`);+ return {+ id: `${value.id}:source-audio`,+ artifact: structuredClone(audio.artifact),+ target: { startSample: targetStart, endSampleExclusive: targetEnd },+ source: {+ sampleFrames: audio.sampleFrames,+ startSample: clipSourceStart,+ endSampleExclusive: clipSourceEnd,+ loop,+ phaseSample,+ },+ playbackRate,+ pitch: "preserve",+ gain: value.sourceAudio.gain,+ fadeInSamples: fades.inSamples,+ fadeOutSamples: fades.outSamples,+ };+}++function soundClip(+ id: string,+ source: MediaSoundSource,+ gain: number,+ point: number,+ space: ProgramSpace,+): AudioClip {+ const total = programSpaceSampleFrames(space, 48_000);+ const targetStart = programFrameSampleBoundary(space, point, 48_000);+ const audible = Math.min(source.sampleFrames, Math.max(0, total - targetStart));+ assert(audible > 0, `Media sound ${id} begins outside ProgramSpace.`);+ return {+ id,+ artifact: structuredClone(source.artifact),+ target: { startSample: targetStart, endSampleExclusive: targetStart + audible },+ source: {+ sampleFrames: source.sampleFrames,+ startSample: 0,+ endSampleExclusive: audible,+ loop: false,+ phaseSample: 0,+ },+ playbackRate: 1,+ pitch: "preserve",+ gain,+ fadeInSamples: 0,+ fadeOutSamples: 0,+ };+}++function edgeSoundClips(+ value: Pick<MediaItemProgram, "id" | "span" | "motion" | "sounds">,+ space: ProgramSpace,+): AudioClip[] {+ return value.sounds.flatMap((sound) => {+ if (sound.trigger.kind === "handoff") return [];+ const point = sound.trigger.kind === "enter"+ ? value.span.startFrame+ : value.span.endFrameExclusive - (value.motion.exit?.durationFrames ?? 0);+ return [soundClip(`${value.id}:${sound.id}`, sound.source, sound.gain, point, space)];+ });+}++function sequenceAudioClips(sequence: MediaSequenceProgram, space: ProgramSpace): AudioClip[] {+ const sourceClips = sequence.members.flatMap((member, index) => {+ if (member.sourceAudio === undefined) return [];+ const incoming = sequence.handoffs[index - 1];+ const outgoing = sequence.handoffs[index];+ const span = {+ startFrame: incoming?.audio === "crossfade" ? incoming.span.startFrame : member.logicalSpan.startFrame,+ endFrameExclusive: outgoing?.audio === "crossfade"+ ? outgoing.span.endFrameExclusive : member.logicalSpan.endFrameExclusive,+ };+ const fadeInSamples = incoming?.audio === "crossfade"+ ? programFrameSampleBoundary(space, incoming.span.endFrameExclusive, 48_000)+ - programFrameSampleBoundary(space, incoming.span.startFrame, 48_000)+ : 0;+ const fadeOutSamples = outgoing?.audio === "crossfade"+ ? programFrameSampleBoundary(space, outgoing.span.endFrameExclusive, 48_000)+ - programFrameSampleBoundary(space, outgoing.span.startFrame, 48_000)+ : 0;+ const clip = sourceAudioClip({+ id: `${sequence.id}:${member.id}`,+ span,+ layers: member.layers,+ sourceAudio: member.sourceAudio,+ }, space, { inSamples: fadeInSamples, outSamples: fadeOutSamples });+ return clip === undefined ? [] : [clip];});+ const handoffSounds = sequence.sounds.flatMap((sound) => {+ if (sound.trigger.kind !== "handoff") return [];+ const { handoffId } = sound.trigger;+ const handoff = sequence.handoffs.find((candidate) => candidate.id === handoffId)!;+ return [soundClip(+ `${sequence.id}:${sound.id}`, sound.source, sound.gain,+ sequence.members.find((member) => member.id === handoff.toMemberId)!.activationFrame, space,+ )];+ });+ return [+ ...sourceClips,+ ...edgeSoundClips({+ id: sequence.id,+ span: sequence.span,+ motion: sequence.motion,+ sounds: sequence.sounds,+ }, space),+ ...handoffSounds,+ ];}++export function projectMediaAudioTrack(space: ProgramSpace, program: MediaTrackProgram): AudioTrack {+ assertMediaTrackProgramIdentity(program, space);+ const clips = program.items.flatMap((item) => {+ const source = sourceAudioClip(item, space);+ return [...(source === undefined ? [] : [source]), ...edgeSoundClips(item, space)];+ }).concat(program.sequences.flatMap((sequence) => sequenceAudioClips(sequence, space)));+ assert(clips.length > 0, `Media Program ${program.id} has no explicitly authored audio projection.`);+ const track = sealAudioTrack({ contract: "svml.audio-track@1", id: `${program.id}:audio`, clips });+ assertAudioTrackIdentity(track, space);+ return track;+}++/** Candidate-era name retained only inside this pre-release migration. */+export const renderMediaTrack = projectMediaVisualTrack;diff --git a/packages/media-track/src/restricted-speech.ts b/packages/media-track/src/restricted-speech.tsnew file mode 100644index 00000000..02ea0da0--- /dev/null+++ b/packages/media-track/src/restricted-speech.ts@@ -0,0 +1,98 @@+import type { VisualPresent } from "@narratage/composition";+import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";+import type { ProgramSpace } from "@narratage/program-space";+import type { BlobRef } from "@narratage/protocol";+import { assertCanvasSpace, assertContentFit, assertIntrinsicExtent } from "@narratage/spatial";+import type { CanvasSpace, ContentFit, IntrinsicExtent } from "@narratage/spatial";++import { lowerMediaItemElements } from "./lower.js";+import type { MediaItemProgram } from "./types.js";++export type RestrictedSpeechVisualClip = {+ readonly id: string;+ readonly span: { readonly startFrame: number; readonly endFrameExclusive: number };+ readonly artifact: BlobRef;+ readonly extent: IntrinsicExtent;+ readonly frameRate: { readonly numerator: number; readonly denominator: number };+ readonly frameCount: number;+};++/**+ * Focused reuse point for Speech Spine: one normalized muted take, one full-Canvas+ * foreground layer and no Media author semantics beyond the shared lowering laws.+ */+export function lowerRestrictedSpeechVisualPresents(+ trackId: string,+ space: ProgramSpace,+ canvas: CanvasSpace,+ clips: readonly RestrictedSpeechVisualClip[],+ fit: ContentFit,+): readonly VisualPresent[] {+ assertProgramSpaceIdentity(space);+ assertCanvasSpace(canvas);+ assertContentFit(fit);+ if (!trackId || clips.length === 0) throw new Error("Restricted Speech visual requires an id and clips.");+ const totalFrames = programSpaceFrameCount(space);+ return clips.map((clip, index) => {+ if (clip.artifact.kind !== "blob" || !clip.artifact.mediaType.startsWith("video/")) {+ throw new Error(`Speech visual ${clip.id} is not a normalized moving-video Blob.`);+ }+ assertIntrinsicExtent(clip.extent);+ if (clip.frameRate.numerator !== space.frameRate.numerator+ || clip.frameRate.denominator !== space.frameRate.denominator) {+ throw new Error(`Speech visual ${clip.id} is not normalized to ProgramSpace frame rate.`);+ }+ const duration = clip.span.endFrameExclusive - clip.span.startFrame;+ if (!Number.isSafeInteger(clip.span.startFrame) || !Number.isSafeInteger(clip.span.endFrameExclusive)+ || clip.span.startFrame < 0 || clip.span.endFrameExclusive > totalFrames || duration <= 0+ || clip.frameCount !== duration) {+ throw new Error(`Speech visual ${clip.id} does not exactly cover its Segment frame span.`);+ }+ const item: MediaItemProgram = {+ id: `${trackId}:${clip.id}`,+ sourceOccurrenceId: clip.id,+ span: { ...clip.span },+ frame: {+ contract: "svml.spatial-frame@1",+ xPx: 0,+ yPx: 0,+ widthPx: canvas.widthPx,+ heightPx: canvas.heightPx,+ },+ presentation: {+ clip: { kind: "frame" },+ padding: { topPx: 0, rightPx: 0, bottomPx: 0, leftPx: 0 },+ shadows: [],+ },+ layers: [{+ id: `${trackId}:${clip.id}:foreground`,+ kind: "sample",+ source: {+ kind: "timed",+ artifact: structuredClone(clip.artifact),+ extent: structuredClone(clip.extent),+ frameRate: { ...clip.frameRate },+ frameCount: clip.frameCount,+ },+ fit: structuredClone(fit),+ occupancy: { mode: "once", align: "start" },+ appearance: {+ opacity: 1,+ filter: { blurPx: 0, brightness: 1, contrast: 1, saturation: 1 },+ },+ }],+ motion: { sustain: [] },+ stacking: {+ order: 0,+ tieBreak: `${trackId}:${String(index + 1).padStart(4, "0")}:${clip.id}`,+ },+ sounds: [],+ };+ return {+ id: item.id,+ span: { ...item.span },+ stacking: { ...item.stacking },+ elements: lowerMediaItemElements(item, space),+ };+ });+}diff --git a/packages/media-track/src/sampling.ts b/packages/media-track/src/sampling.tsnew file mode 100644index 00000000..f14a5dd7--- /dev/null+++ b/packages/media-track/src/sampling.ts@@ -0,0 +1,118 @@+import type {+ VisualSamplingRational,+ VisualSamplingSegment,+ VisualTimedSampling,+} from "@narratage/composition";+import { assertProgramSpaceIdentity } from "@narratage/program-space";+import type { ProgramSpace } from "@narratage/program-space";++import type { MediaVisualOccupancy, MediaVisualTrim } from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function gcd(left: bigint, right: bigint): bigint {+ let a = left < 0n ? -left : left;+ let b = right < 0n ? -right : right;+ while (b !== 0n) [a, b] = [b, a % b];+ return a;+}++function rational(numerator: bigint, denominator = 1n): VisualSamplingRational {+ assert(denominator > 0n && numerator >= 0n, "Media visual sampling rational is negative or undefined.");+ const divisor = gcd(numerator, denominator);+ const reducedNumerator = numerator / divisor;+ const reducedDenominator = denominator / divisor;+ assert(reducedNumerator <= BigInt(Number.MAX_SAFE_INTEGER)+ && reducedDenominator <= BigInt(Number.MAX_SAFE_INTEGER),+ "Media visual sampling rational exceeds safe wire arithmetic.");+ return { numerator: Number(reducedNumerator), denominator: Number(reducedDenominator) };+}++function segment(+ startFrame: number,+ endFrameExclusive: number,+ sourceFrame: VisualSamplingRational,+ rate: VisualSamplingRational,+ loop?: MediaVisualTrim,+): VisualSamplingSegment {+ return {+ target: { startFrame, endFrameExclusive },+ sourceFrame,+ rate,+ ...(loop === undefined ? {} : { loop: { ...loop } }),+ };+}++function assertOccupancy(value: MediaVisualOccupancy): void {+ if (value.mode === "stretch") return;+ assert(["once", "hold", "loop"].includes(value.mode), "Media visual occupancy mode is invalid.");+ assert(value.align === "start" || value.align === "end", "Media visual occupancy alignment is invalid.");+}++/**+ * Resolve package-owned trim/occupancy intent into the renderer-neutral exact+ * source-frame function. Timed media is normalized to Program frame rate+ * upstream, so native playback advances exactly one source frame per Program+ * frame. This avoids a second, renderer-specific frame quantizer.+ */+export function resolveVisualSampling(input: {+ readonly space: ProgramSpace;+ readonly sourceFrameRate: { readonly numerator: number; readonly denominator: number };+ readonly sourceFrameCount: number;+ readonly targetFrameCount: number;+ readonly trim?: MediaVisualTrim;+ readonly occupancy: MediaVisualOccupancy;+}): VisualTimedSampling {+ assertProgramSpaceIdentity(input.space);+ assert(Number.isSafeInteger(input.sourceFrameCount) && input.sourceFrameCount > 0,+ "Timed visual source frame count is invalid.");+ assert(Number.isSafeInteger(input.targetFrameCount) && input.targetFrameCount > 0,+ "Timed visual target frame count is invalid.");+ assert(input.sourceFrameRate.numerator === input.space.frameRate.numerator+ && input.sourceFrameRate.denominator === input.space.frameRate.denominator,+ "Timed visual source must be normalized to ProgramSpace frame rate before Media authoring.");+ assertOccupancy(input.occupancy);+ const trim = input.trim ?? { startFrame: 0, endFrameExclusive: input.sourceFrameCount };+ assert(Number.isSafeInteger(trim.startFrame) && Number.isSafeInteger(trim.endFrameExclusive)+ && trim.startFrame >= 0 && trim.endFrameExclusive > trim.startFrame+ && trim.endFrameExclusive <= input.sourceFrameCount,+ "Timed visual trim is outside its source frame domain.");+ const sourceLength = trim.endFrameExclusive - trim.startFrame;+ const targetLength = input.targetFrameCount;+ const one = rational(1n);+ const zero = rational(0n);+ const segments: VisualSamplingSegment[] = [];++ if (input.occupancy.mode === "stretch") {+ segments.push(segment(0, targetLength, rational(BigInt(trim.startFrame)),+ rational(BigInt(sourceLength), BigInt(targetLength))));+ } else if (input.occupancy.mode === "loop") {+ const phase = input.occupancy.align === "start"+ ? 0+ : (sourceLength - (targetLength % sourceLength)) % sourceLength;+ segments.push(segment(0, targetLength, rational(BigInt(trim.startFrame + phase)), one, trim));+ } else {+ const played = Math.min(sourceLength, targetLength);+ if (input.occupancy.align === "start") {+ segments.push(segment(0, played, rational(BigInt(trim.startFrame)), one));+ if (input.occupancy.mode === "hold" && played < targetLength) {+ segments.push(segment(played, targetLength, rational(BigInt(trim.endFrameExclusive - 1)), zero));+ }+ } else {+ const targetStart = targetLength - played;+ if (input.occupancy.mode === "hold" && targetStart > 0) {+ segments.push(segment(0, targetStart, rational(BigInt(trim.startFrame)), zero));+ }+ segments.push(segment(targetStart, targetLength,+ rational(BigInt(trim.endFrameExclusive - played)), one));+ }+ }++ return {+ sourceFrameRate: { ...input.sourceFrameRate },+ sourceFrameCount: input.sourceFrameCount,+ segments,+ };+}diff --git a/packages/media-track/src/sequence.ts b/packages/media-track/src/sequence.tsnew file mode 100644index 00000000..b7f57bb8--- /dev/null+++ b/packages/media-track/src/sequence.ts@@ -0,0 +1,435 @@+import type {+ VisualAnimation,+ VisualEasing,+ VisualPresent,+ VisualStyleDeclaration,+} from "@narratage/composition";+import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";+import type { ProgramSpace } from "@narratage/program-space";+import { canonicalize } from "@narratage/protocol";+import { assertCanvasSpace, assertSpatialFrame } from "@narratage/spatial";+import type { CanvasSpace } from "@narratage/spatial";+import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";+import type { CompleteSemanticMap } from "@narratage/semantic-map";+import {+ locateMomentOccurrences,+ locateSelectionOccurrences,+ resolveTriggeredSchedule,+} from "@narratage/temporal";++import { assertMediaIdentity, assertMediaLayerSet } from "./layers.js";+import { lowerMediaItemElements } from "./lower.js";+import {+ assertMediaLifecycleMotion,+ lifecycleAnimationWindow,+ resolveMediaLifecycleMotion,+ sustainAnimationWindow,+} from "./motion.js";+import { assertMediaFramePresentation } from "./presentation.js";+import { assertMediaSoundSet } from "./sounds.js";+import type {+ MediaHandoffProgram,+ MediaHandoffSpec,+ MediaItemProgram,+ MediaLayerSet,+ MediaMotionDirection,+ MediaSequenceMemberProgram,+ MediaSequenceMemberSet,+ MediaSequenceMemberSpec,+ MediaSequenceProgram,+ MediaSequenceSpec,+ MediaSoundSet,+ MediaTrackHeader,+} from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function finite(value: number, label: string): void {+ assert(Number.isFinite(value), `${label} must be finite.`);+}++export function assertMediaSequenceMemberSpec(value: MediaSequenceMemberSpec): void {+ assert(value.contract === "svml.media-sequence-member-spec@1", "Unsupported MediaSequenceMemberSpec contract.");+ assertMediaIdentity(value.id, "MediaSequenceMemberSpec.id");+ if (value.sourceAudio !== undefined) {+ assertMediaIdentity(value.sourceAudio.fromLayer, "MediaSequenceMemberSpec.sourceAudio.fromLayer");+ finite(value.sourceAudio.gain, "MediaSequenceMemberSpec.sourceAudio.gain");+ assert(value.sourceAudio.gain >= 0 && value.sourceAudio.gain <= 64,+ "MediaSequenceMemberSpec.sourceAudio.gain is invalid.");+ }+}++export function sealMediaSequenceMemberSpec(value: MediaSequenceMemberSpec): MediaSequenceMemberSpec {+ assertMediaSequenceMemberSpec(value);+ return canonicalize(value) as unknown as MediaSequenceMemberSpec;+}++export function createMediaSequenceMemberSet(): MediaSequenceMemberSet {+ return { contract: "svml.media-sequence-member-set@1", members: [] };+}++export function assertMediaSequenceMemberSet(value: MediaSequenceMemberSet): void {+ assert(value.contract === "svml.media-sequence-member-set@1" && Array.isArray(value.members),+ "MediaSequenceMemberSet is invalid.");+ const ids = new Set<string>();+ for (const member of value.members) {+ assertMediaIdentity(member.id, "Media Sequence member id");+ assert(!ids.has(member.id), `MediaSequenceMemberSet repeats ${member.id}.`);+ ids.add(member.id);+ assert(Number.isSafeInteger(member.activationFrame) && member.activationFrame >= 0,+ `Media Sequence member ${member.id} activation is invalid.`);+ assertMediaLayerSet({ contract: "svml.media-layer-set@1", layers: member.layers });+ }+}++export function appendMediaSequenceMember(+ set: MediaSequenceMemberSet,+ layers: MediaLayerSet,+ spec: MediaSequenceMemberSpec,+ activationFrame: number,+): MediaSequenceMemberSet {+ assertMediaSequenceMemberSet(set);+ assertMediaLayerSet(layers);+ assert(layers.layers.length > 0, `Media Sequence member ${spec.id} requires layers.`);+ assertMediaSequenceMemberSpec(spec);+ assert(Number.isSafeInteger(activationFrame) && activationFrame >= 0,+ `Media Sequence member ${spec.id} activation is invalid.`);+ assert(!set.members.some((member) => member.id === spec.id), `Media Sequence already contains ${spec.id}.`);+ if (spec.sourceAudio !== undefined) {+ const selected = layers.layers.find((layer) => layer.id === spec.sourceAudio!.fromLayer);+ assert(selected?.kind === "sample" && selected.source.kind === "timed" && selected.source.audio !== undefined,+ `Media Sequence member ${spec.id} source-audio layer is absent or silent.`);+ }+ return canonicalize({+ contract: "svml.media-sequence-member-set@1",+ members: [...set.members, {+ id: spec.id,+ activationFrame,+ layers: structuredClone(layers.layers),+ ...(spec.sourceAudio === undefined ? {} : { sourceAudio: { ...spec.sourceAudio } }),+ }],+ }) as unknown as MediaSequenceMemberSet;+}++function exactlyOneFrame(values: readonly number[], label: string): number {+ assert(values.length === 1, `${label} requires exactly one occurrence; received ${values.length}.`);+ return values[0]!;+}++export function appendMediaSequenceMomentMember(+ set: MediaSequenceMemberSet,+ layers: MediaLayerSet,+ spec: MediaSequenceMemberSpec,+ map: CompleteSemanticMap,+ moment: NarrativeMomentRef,+ space: ProgramSpace,+): MediaSequenceMemberSet {+ const frame = exactlyOneFrame(+ locateMomentOccurrences(map, moment, space).map((occurrence) => occurrence.cue.frame),+ `Media Sequence member ${spec.id} Moment`,+ );+ return appendMediaSequenceMember(set, layers, spec, frame);+}++export function appendMediaSequenceSelectionMember(+ set: MediaSequenceMemberSet,+ layers: MediaLayerSet,+ spec: MediaSequenceMemberSpec,+ map: CompleteSemanticMap,+ selection: NarrativeSelectionRef,+ space: ProgramSpace,+ boundary: "start" | "end",+): MediaSequenceMemberSet {+ const frame = exactlyOneFrame(+ locateSelectionOccurrences(map, selection, space).map((occurrence) =>+ boundary === "start" ? occurrence.start.frame : occurrence.end.frame),+ `Media Sequence member ${spec.id} Selection`,+ );+ return appendMediaSequenceMember(set, layers, spec, frame);+}++export function assertMediaHandoffSpec(value: MediaHandoffSpec): void {+ assert(value.contract === "svml.media-handoff-spec@1", "Unsupported MediaHandoffSpec contract.");+ assertMediaIdentity(value.id, "MediaHandoffSpec.id");+ assertMediaIdentity(value.fromMemberId, "MediaHandoffSpec.fromMemberId");+ assertMediaIdentity(value.toMemberId, "MediaHandoffSpec.toMemberId");+ assert(["cut", "crossfade", "push", "wipe", "cover", "page-turn"].includes(value.operator),+ "MediaHandoffSpec.operator is invalid.");+ assert(Number.isSafeInteger(value.durationFrames) && value.durationFrames >= 0,+ "MediaHandoffSpec.durationFrames is invalid.");+ assert((value.operator === "cut") === (value.durationFrames === 0),+ "A cut must have zero duration and every visual transition must have positive duration.");+ finite(value.boundaryRatio, "MediaHandoffSpec.boundaryRatio");+ assert(value.boundaryRatio >= 0 && value.boundaryRatio <= 1,+ "MediaHandoffSpec.boundaryRatio must be inside [0, 1].");+ if (["push", "wipe", "cover", "page-turn"].includes(value.operator)) {+ assert(value.direction !== undefined && ["left", "right", "up", "down"].includes(value.direction),+ `MediaHandoffSpec ${value.operator} requires a direction.`);+ }+ assert(value.audio === "cut" || value.audio === "crossfade", "MediaHandoffSpec.audio is invalid.");+ if (value.audio === "crossfade") assert(value.durationFrames > 0, "Audio crossfade requires a non-empty handoff.");+}++export function sealMediaHandoffSpec(value: MediaHandoffSpec): MediaHandoffSpec {+ assertMediaHandoffSpec(value);+ return canonicalize(value) as unknown as MediaHandoffSpec;+}++export function assertMediaSequenceSpec(value: MediaSequenceSpec): void {+ assert(value.contract === "svml.media-sequence-spec@1", "Unsupported MediaSequenceSpec contract.");+ assertMediaIdentity(value.id, "MediaSequenceSpec.id");+ assertMediaFramePresentation(value.presentation, "MediaSequenceSpec.presentation");+ assert(Number.isSafeInteger(value.stackingOrder), "MediaSequenceSpec.stackingOrder must be an integer.");+ for (const handoff of value.handoffs) assertMediaHandoffSpec(handoff);+}++export function sealMediaSequenceSpec(value: MediaSequenceSpec): MediaSequenceSpec {+ assertMediaSequenceSpec(value);+ return canonicalize(value) as unknown as MediaSequenceSpec;+}++function resolvedHandoff(+ spec: MediaHandoffSpec,+ boundary: number,+): MediaHandoffProgram {+ const before = Math.floor(spec.durationFrames * spec.boundaryRatio);+ const span = {+ startFrame: boundary - before,+ endFrameExclusive: boundary + (spec.durationFrames - before),+ };+ return {+ id: spec.id,+ fromMemberId: spec.fromMemberId,+ toMemberId: spec.toMemberId,+ operator: spec.operator,+ durationFrames: spec.durationFrames,+ boundaryRatio: spec.boundaryRatio,+ span,+ ...(spec.direction === undefined ? {} : { direction: spec.direction }),+ audio: spec.audio,+ };+}++export function resolveMediaSequence(+ header: MediaTrackHeader,+ space: ProgramSpace,+ canvas: CanvasSpace,+ memberSet: MediaSequenceMemberSet,+ frame: MediaSequenceProgram["frame"],+ spec: MediaSequenceSpec,+ sounds: MediaSoundSet,+ terminalFrame: number,+): MediaSequenceProgram {+ assertProgramSpaceIdentity(space);+ assertCanvasSpace(canvas);+ assertMediaSequenceMemberSet(memberSet);+ assertMediaSequenceSpec(spec);+ assertMediaSoundSet(sounds);+ assert(Number.isSafeInteger(terminalFrame) && terminalFrame > 0,+ `Media Sequence ${spec.id} terminal frame is invalid.`);+ assertSpatialFrame(frame);+ assert(memberSet.members.length >= 2, `Media Sequence ${spec.id} requires at least two Members.`);+ assert(spec.handoffs.length === memberSet.members.length - 1,+ `Media Sequence ${spec.id} requires exactly one Handoff for every adjacent pair.`);+ const first = memberSet.members[0]!;+ const schedule = resolveTriggeredSchedule({+ outer: { startFrame: first.activationFrame, endFrameExclusive: terminalFrame },+ terminalFrame,+ triggers: memberSet.members.map((member) => ({ id: member.id, frame: member.activationFrame })),+ });+ assert(terminalFrame <= programSpaceFrameCount(space), `Media Sequence ${spec.id} exits ProgramSpace.`);+ const handoffs = spec.handoffs.map((handoff, index) => {+ const from = memberSet.members[index]!;+ const to = memberSet.members[index + 1]!;+ assert(handoff.fromMemberId === from.id && handoff.toMemberId === to.id,+ `Media Handoff ${handoff.id} does not connect authored adjacent Members ${from.id} → ${to.id}.`);+ const result = resolvedHandoff(handoff, to.activationFrame);+ assert(result.span.startFrame >= first.activationFrame && result.span.endFrameExclusive <= terminalFrame,+ `Media Handoff ${handoff.id} exceeds the Sequence envelope.`);+ return result;+ });+ for (const sound of sounds.sounds) {+ if (sound.trigger.kind === "handoff") {+ const { handoffId } = sound.trigger;+ assert(handoffs.some((handoff) => handoff.id === handoffId),+ `Media Sequence ${spec.id} sound ${sound.id} targets an unknown Handoff.`);+ }+ }+ for (let index = 1; index < handoffs.length; index += 1) {+ assert(handoffs[index - 1]!.span.endFrameExclusive <= handoffs[index]!.span.startFrame,+ `Media Sequence ${spec.id} has overlapping Handoffs.`);+ }+ const members: MediaSequenceMemberProgram[] = memberSet.members.map((member, index) => {+ const logicalSpan = schedule.exclusive[index]!.span;+ const incoming = handoffs[index - 1];+ const outgoing = handoffs[index];+ const visualSpan = {+ startFrame: incoming?.span.startFrame ?? logicalSpan.startFrame,+ endFrameExclusive: outgoing?.span.endFrameExclusive ?? logicalSpan.endFrameExclusive,+ };+ return {+ id: member.id,+ activationFrame: member.activationFrame,+ logicalSpan: { ...logicalSpan },+ visualSpan,+ layers: structuredClone(member.layers),+ ...(member.sourceAudio === undefined ? {} : { sourceAudio: { ...member.sourceAudio } }),+ };+ });+ const span = { startFrame: first.activationFrame, endFrameExclusive: terminalFrame };+ assertMediaLifecycleMotion(spec.motion, span.endFrameExclusive - span.startFrame, `Media Sequence ${spec.id} motion`);+ const motion = resolveMediaLifecycleMotion(spec.motion, frame, canvas);+ assert(frame.widthPx > spec.presentation.padding.leftPx + spec.presentation.padding.rightPx+ && frame.heightPx > spec.presentation.padding.topPx + spec.presentation.padding.bottomPx,+ `Media Sequence ${spec.id} padding consumes its complete Placement Frame.`);+ return canonicalize({+ id: spec.id,+ span,+ terminalFrame,+ frame: { ...frame },+ presentation: structuredClone(spec.presentation),+ members,+ handoffs,+ motion,+ stacking: { order: spec.stackingOrder, tieBreak: `${header.id}:${spec.id}` },+ sounds: structuredClone(sounds.sounds),+ }) as unknown as MediaSequenceProgram;+}++type TransitionState = {+ readonly opacity: number;+ readonly transform: string;+ readonly clipPath: string;+};++const neutral: TransitionState = { opacity: 1, transform: "none", clipPath: "inset(0% 0% 0% 0%)" };++function translate(direction: MediaMotionDirection, amount: number): string {+ if (direction === "left") return `translateX(${-amount}%)`;+ if (direction === "right") return `translateX(${amount}%)`;+ if (direction === "up") return `translateY(${-amount}%)`;+ return `translateY(${amount}%)`;+}++function opposite(direction: MediaMotionDirection): MediaMotionDirection {+ if (direction === "left") return "right";+ if (direction === "right") return "left";+ if (direction === "up") return "down";+ return "up";+}++function clipped(direction: MediaMotionDirection): string {+ if (direction === "left") return "inset(0% 0% 0% 100%)";+ if (direction === "right") return "inset(0% 100% 0% 0%)";+ if (direction === "up") return "inset(0% 0% 100% 0%)";+ return "inset(100% 0% 0% 0%)";+}++function incomingState(handoff: MediaHandoffProgram): TransitionState {+ if (handoff.operator === "crossfade") return { ...neutral, opacity: 0 };+ if (handoff.operator === "push" || handoff.operator === "cover") {+ return { ...neutral, transform: translate(opposite(handoff.direction!), 100) };+ }+ if (handoff.operator === "wipe") return { ...neutral, clipPath: clipped(handoff.direction!) };+ if (handoff.operator === "page-turn") {+ const axis = handoff.direction === "left" || handoff.direction === "right" ? "Y" : "X";+ const sign = handoff.direction === "left" || handoff.direction === "up" ? 1 : -1;+ return { ...neutral, opacity: 0, transform: `perspective(900px) rotate${axis}(${sign * 75}deg)` };+ }+ return neutral;+}++function outgoingState(handoff: MediaHandoffProgram): TransitionState {+ if (handoff.operator === "crossfade") return { ...neutral, opacity: 0 };+ if (handoff.operator === "push") return { ...neutral, transform: translate(handoff.direction!, 100) };+ if (handoff.operator === "page-turn") {+ const axis = handoff.direction === "left" || handoff.direction === "right" ? "Y" : "X";+ const sign = handoff.direction === "left" || handoff.direction === "up" ? -1 : 1;+ return { ...neutral, opacity: 0, transform: `perspective(900px) rotate${axis}(${sign * 75}deg)` };+ }+ return neutral;+}++function styles(value: TransitionState): VisualStyleDeclaration[] {+ return [+ { name: "clip-path", value: value.clipPath },+ { name: "opacity", value: value.opacity },+ { name: "transform", value: value.transform },+ ];+}++export function sequenceMemberHandoffAnimation(+ member: MediaSequenceMemberProgram,+ incoming: MediaHandoffProgram | undefined,+ outgoing: MediaHandoffProgram | undefined,+): VisualAnimation | undefined {+ if ((incoming === undefined || incoming.operator === "cut") && (outgoing === undefined || outgoing.operator === "cut")) {+ return undefined;+ }+ const duration = member.visualSpan.endFrameExclusive - member.visualSpan.startFrame;+ const states = new Map<number, { readonly state: TransitionState; readonly easing?: VisualEasing }>();+ const add = (frame: number, state: TransitionState, easing?: VisualEasing): void => {+ const previous = states.get(frame);+ assert(previous === undefined || JSON.stringify(previous.state) === JSON.stringify(state),+ `Media Sequence member ${member.id} has conflicting Handoff states at frame ${frame}.`);+ states.set(frame, { state, ...(easing === undefined ? {} : { easing }) });+ };+ add(0, incoming === undefined || incoming.operator === "cut" ? neutral : incomingState(incoming), "ease-in-out");+ if (incoming !== undefined && incoming.operator !== "cut") {+ add(incoming.span.endFrameExclusive - member.visualSpan.startFrame, neutral);+ }+ if (outgoing !== undefined && outgoing.operator !== "cut") {+ add(outgoing.span.startFrame - member.visualSpan.startFrame, neutral, "ease-in-out");+ add(outgoing.span.endFrameExclusive - member.visualSpan.startFrame, outgoingState(outgoing));+ } else add(duration, neutral);+ return { keyframes: [...states.entries()].sort(([left], [right]) => left - right).map(([atFrame, value]) => ({+ atFrame,+ ...(value.easing === undefined ? {} : { easing: value.easing }),+ style: styles(value.state),+ })) };+}++export function lowerMediaSequencePresents(+ sequence: MediaSequenceProgram,+ space: ProgramSpace,+): readonly VisualPresent[] {+ return sequence.members.map((member, index) => {+ const incoming = sequence.handoffs[index - 1];+ const outgoing = sequence.handoffs[index];+ const item: MediaItemProgram = {+ id: `${sequence.id}:${member.id}`,+ sourceOccurrenceId: member.id,+ span: { ...member.visualSpan },+ frame: { ...sequence.frame },+ presentation: structuredClone(sequence.presentation),+ layers: structuredClone(member.layers),+ motion: structuredClone(sequence.motion),+ stacking: {+ order: sequence.stacking.order,+ tieBreak: `${sequence.stacking.tieBreak}:${String(index + 1).padStart(4, "0")}`,+ },+ ...(member.sourceAudio === undefined ? {} : { sourceAudio: { ...member.sourceAudio } }),+ sounds: [],+ };+ const handoffAnimation = sequenceMemberHandoffAnimation(member, incoming, outgoing);+ const outerDuration = sequence.span.endFrameExclusive - sequence.span.startFrame;+ const startOffset = member.visualSpan.startFrame - sequence.span.startFrame;+ const endOffset = member.visualSpan.endFrameExclusive - sequence.span.startFrame;+ return {+ id: item.id,+ span: { ...item.span },+ stacking: { ...item.stacking },+ elements: lowerMediaItemElements(item, space, {+ includeHandoffWrapper: true,+ ...(handoffAnimation === undefined ? {} : { handoffAnimation }),+ lifecycleAnimationOverride: lifecycleAnimationWindow(+ sequence.motion, outerDuration, startOffset, endOffset,+ ) ?? null,+ sustainAnimationsOverride: sequence.motion.sustain.map((sustain) =>+ sustainAnimationWindow(sustain, outerDuration, startOffset, endOffset)),+ }),+ };+ });+}diff --git a/packages/media-track/src/sounds.ts b/packages/media-track/src/sounds.tsnew file mode 100644index 00000000..93ffb940--- /dev/null+++ b/packages/media-track/src/sounds.ts@@ -0,0 +1,72 @@+import { verifySynchronizedMedia } from "@narratage/media";+import type { SynchronizedMedia } from "@narratage/media";+import { canonicalize } from "@narratage/protocol";++import { assertMediaIdentity } from "./layers.js";+import type { MediaSoundSet, MediaSoundSpec } from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++export function assertMediaSoundSpec(value: MediaSoundSpec): void {+ assert(value.contract === "svml.media-sound-spec@1", "Unsupported MediaSoundSpec contract.");+ assertMediaIdentity(value.id, "MediaSoundSpec.id");+ assert(value.trigger.kind === "enter" || value.trigger.kind === "exit" || value.trigger.kind === "handoff",+ "MediaSoundSpec.trigger is invalid.");+ if (value.trigger.kind === "handoff") {+ assertMediaIdentity(value.trigger.handoffId, "MediaSoundSpec.trigger.handoffId");+ }+ assert(Number.isFinite(value.gain) && value.gain >= 0 && value.gain <= 64,+ "MediaSoundSpec.gain must be inside [0, 64].");+}++export function sealMediaSoundSpec(value: MediaSoundSpec): MediaSoundSpec {+ assertMediaSoundSpec(value);+ return canonicalize(value) as unknown as MediaSoundSpec;+}++export function createMediaSoundSet(): MediaSoundSet {+ return { contract: "svml.media-sound-set@1", sounds: [] };+}++export function assertMediaSoundSet(value: MediaSoundSet): void {+ assert(value.contract === "svml.media-sound-set@1" && Array.isArray(value.sounds),+ "MediaSoundSet is invalid.");+ const ids = new Set<string>();+ for (const sound of value.sounds) {+ assertMediaSoundSpec({ contract: "svml.media-sound-spec@1", id: sound.id, trigger: sound.trigger, gain: sound.gain });+ assert(!ids.has(sound.id), `MediaSoundSet repeats ${sound.id}.`);+ ids.add(sound.id);+ assert(sound.source.artifact.kind === "blob" && sound.source.artifact.mediaType.startsWith("audio/")+ && Number.isSafeInteger(sound.source.sampleFrames) && sound.source.sampleFrames > 0,+ `MediaSoundSet ${sound.id} source is invalid.`);+ }+}++/** Bind one normalized audio source through an ordinary graph input edge. */+export function appendMediaSound(+ set: MediaSoundSet,+ media: SynchronizedMedia,+ spec: MediaSoundSpec,+): MediaSoundSet {+ assertMediaSoundSet(set);+ verifySynchronizedMedia(media);+ assertMediaSoundSpec(spec);+ assert(media.audio !== undefined, `Media sound ${spec.id} requires normalized audio.`);+ assert(!set.sounds.some((sound) => sound.id === spec.id), `MediaSoundSet already contains ${spec.id}.`);+ const result = {+ contract: "svml.media-sound-set@1" as const,+ sounds: [...set.sounds, {+ id: spec.id,+ trigger: structuredClone(spec.trigger),+ source: {+ artifact: structuredClone(media.audio.artifact),+ sampleFrames: media.audio.sampleFrames,+ },+ gain: spec.gain,+ }],+ };+ assertMediaSoundSet(result);+ return canonicalize(result) as unknown as MediaSoundSet;+}diff --git a/packages/media-track/src/surface.ts b/packages/media-track/src/surface.tsnew file mode 100644index 00000000..50dd746c--- /dev/null+++ b/packages/media-track/src/surface.ts@@ -0,0 +1,807 @@+import { artifactTypes } from "@narratage/artifact";+import { compositionTypes } from "@narratage/composition";+import { sealGraphFragment } from "@narratage/elaborator";+import type { FragmentOperation } from "@narratage/elaborator";+import { mediaTypes } from "@narratage/media";+import { narrativeTypes } from "@narratage/narrative";+import { programSpaceTypes } from "@narratage/program-space";+import type { TypeRef } from "@narratage/protocol";+import { semanticMapTypes } from "@narratage/semantic-map";+import { spatialTypes } from "@narratage/spatial";+import { svsRecipeType } from "@narratage/svs";+import type { SvsRecipe } from "@narratage/svs";+import type {+ StructuredElement,+ StructuredSurfaceHandler,+ SurfaceRecordDraft,+ SurfaceResolvedReference,+ TextAttributeValue,+} from "@narratage/text";+import type {+ OccurrenceExpansion,+ TemporalDuration,+ TemporalPointExpression,+ TemporalWindowProjection,+} from "@narratage/temporal";++import {+ decodeMediaFit,+ decodeMediaFramePaint,+ decodeMediaHandoffSpec,+ decodeMediaItemSpec,+ decodeMediaMotion,+ decodeMediaPaintSpec,+ decodeMediaSampleSpec,+ decodeMediaSequenceSpec,+} from "./author.js";+import {+ sealMediaSequenceMemberSpec,+} from "./sequence.js";+import { sealMediaSoundSpec } from "./sounds.js";+import { sealMediaTrackHeader } from "./program.js";+import { mediaTrackProducers, mediaTrackTypes } from "./manifest.js";+import type {+ MediaHandoffSpec,+ MediaSampleLayerSpec,+ MediaSamplingMotion,+ MediaSequenceMemberSpec,+ MediaSoundSpec,+} from "./types.js";++const input = (name: string) => ({ kind: "fragment-input" as const, name });+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });++function sameType(left: TypeRef, right: TypeRef): boolean {+ return left.module.name === right.module.name+ && left.module.version === right.module.version+ && left.name === right.name;+}++function allowed(element: StructuredElement, names: readonly string[]): void {+ const permit = new Set(names);+ const unknown = Object.keys(element.attributes).filter((name) => !permit.has(name));+ if (unknown.length > 0) throw new Error(`${element.name} has unsupported attributes ${unknown.join(", ")}.`);+}++function empty(element: StructuredElement): void {+ if (element.children.some((child) => child.kind === "element" || child.value.trim().length > 0)) {+ throw new Error(`${element.name} must be empty.`);+ }+}++function text(element: StructuredElement, name: string, fallback?: string): string {+ const value = element.attributes[name];+ if (value === undefined && fallback !== undefined) return fallback;+ if (typeof value !== "string" || value.trim().length === 0) {+ throw new Error(`${element.name}.${name} must be non-empty text.`);+ }+ return value.trim();+}++function optionalText(element: StructuredElement, name: string): string | undefined {+ const value = element.attributes[name];+ if (value === undefined) return undefined;+ if (typeof value !== "string" || value.trim().length === 0) {+ throw new Error(`${element.name}.${name} must be non-empty text.`);+ }+ return value.trim();+}++function numberValue(element: StructuredElement, name: string, fallback?: number): number {+ const raw = optionalText(element, name);+ if (raw === undefined && fallback !== undefined) return fallback;+ const value = Number(raw);+ if (!Number.isFinite(value)) throw new Error(`${element.name}.${name} must be a finite number.`);+ return value;+}++function reference(+ raw: TextAttributeValue | undefined,+ label: string,+ expected: TypeRef,+ resolve: (path: string) => SurfaceResolvedReference | undefined,+): SurfaceResolvedReference {+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${label} must be a reference.`);+ const value = resolve(raw.path);+ if (value === undefined || !sameType(value.type, expected)) throw new Error(`${label} has the wrong Type.`);+ return value;+}++function oneOfReferences(+ raw: TextAttributeValue | undefined,+ label: string,+ expected: readonly TypeRef[],+ resolve: (path: string) => SurfaceResolvedReference | undefined,+): SurfaceResolvedReference {+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${label} must be a reference.`);+ const value = resolve(raw.path);+ if (value === undefined || !expected.some((type) => sameType(value.type, type))) {+ throw new Error(`${label} has the wrong Type.`);+ }+ return value;+}++function recipe(value: SurfaceResolvedReference, label: string): SvsRecipe {+ if (!sameType(value.type, svsRecipeType) || value.record?.value.kind !== "inline") {+ throw new Error(`${label} must be an authored SVS Recipe.`);+ }+ return value.record.value.value as unknown as SvsRecipe;+}++function divisor(left: number, right: number): number {+ let a = Math.abs(left);+ let b = Math.abs(right);+ while (b !== 0) [a, b] = [b, a % b];+ return a;+}++function duration(value: string, label: string): TemporalDuration {+ const match = /^(\d+)(?:\.(\d+))?(f|ms|s)$/u.exec(value.trim());+ if (match === null) throw new Error(`${label} must be an exact duration such as 12f, 250ms or 1.5s.`);+ const whole = Number(match[1]);+ const fraction = match[2] ?? "";+ const unit = match[3];+ if (!Number.isSafeInteger(whole)) throw new Error(`${label} is outside safe arithmetic.`);+ if (unit === "f" || unit === "ms") {+ if (fraction.length > 0) throw new Error(`${label} ${unit} duration must be an integer.`);+ return { unit: unit === "f" ? "frames" : "milliseconds", value: whole };+ }+ const scale = 10 ** fraction.length;+ const numerator = whole * scale + (fraction.length === 0 ? 0 : Number(fraction));+ if (!Number.isSafeInteger(numerator) || !Number.isSafeInteger(scale)) throw new Error(`${label} is outside safe arithmetic.`);+ const gcd = divisor(numerator, scale);+ return { unit: "seconds", numerator: numerator / gcd, denominator: scale / gcd };+}++function negate(value: TemporalDuration): TemporalDuration {+ return value.unit === "seconds" ? { ...value, numerator: -value.numerator } : { ...value, value: -value.value };+}++function point(value: string, label: string): TemporalPointExpression {+ const trimmed = value.trim();+ const refs = ["program.start", "program.end", "selection.start", "selection.end", "moment.cue"] as const;+ for (const ref of refs) {+ if (trimmed === ref) return { ref };+ const match = new RegExp(`^${ref.replace(".", "\\.")}\\s*([+-])\\s*(.+)$`, "u").exec(trimmed);+ if (match !== null) {+ const offset = duration(match[2]!, `${label} offset`);+ return { ref, offset: match[1] === "-" ? negate(offset) : offset };+ }+ }+ return { ref: "absolute", at: duration(trimmed, label) };+}++type TemporalBinding = {+ readonly kind: "program" | "selection" | "moment";+ readonly projection: TemporalWindowProjection;+ readonly source?: SurfaceResolvedReference;+};++function temporalBinding(+ element: StructuredElement,+ resolve: (path: string) => SurfaceResolvedReference | undefined,+): TemporalBinding {+ const during = element.attributes.during;+ const at = element.attributes.at;+ const start = optionalText(element, "start");+ const end = optionalText(element, "end");+ const selection = element.attributes.selection;+ const moment = element.attributes.moment;+ const forms = Number(during !== undefined) + Number(at !== undefined) + Number(start !== undefined || end !== undefined);+ if (forms !== 1) throw new Error(`${element.name} requires exactly one of during, at/for, or start/end.`);+ if (during !== undefined) {+ if (typeof during === "string") {+ if (during.trim() !== "program") throw new Error(`${element.name}.during text must be program.`);+ return { kind: "program", projection: { start: { ref: "program.start" }, end: { ref: "program.end" } } };+ }+ return {+ kind: "selection",+ source: reference(during, `${element.name}.during`, narrativeTypes.selection, resolve),+ projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },+ };+ }+ if (at !== undefined) {+ const forDuration = duration(text(element, "for"), `${element.name}.for`);+ return {+ kind: "moment",+ source: reference(at, `${element.name}.at`, narrativeTypes.moment, resolve),+ projection: { start: { ref: "moment.cue" }, end: { ref: "moment.cue", offset: forDuration } },+ };+ }+ if (start === undefined || end === undefined) throw new Error(`${element.name} explicit timing requires start and end.`);+ if (selection !== undefined && moment !== undefined) throw new Error(`${element.name} cannot bind Selection and Moment together.`);+ if (selection !== undefined) return {+ kind: "selection",+ source: reference(selection, `${element.name}.selection`, narrativeTypes.selection, resolve),+ projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) },+ };+ if (moment !== undefined) return {+ kind: "moment",+ source: reference(moment, `${element.name}.moment`, narrativeTypes.moment, resolve),+ projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) },+ };+ return { kind: "program", projection: { start: point(start, `${element.name}.start`), end: point(end, `${element.name}.end`) } };+}++type FragmentLayer =+ | { readonly kind: "paint"; readonly specName: string }+ | { readonly kind: "still"; readonly sourceName: string; readonly extentName: string; readonly fitName: string; readonly specName: string }+ | { readonly kind: "timed" | "surface"; readonly sourceName: string; readonly fitName: string; readonly specName: string };++type FragmentSound = { readonly sourceName: string; readonly specName: string };++type FragmentItem = {+ readonly suffix: string;+ readonly binding: TemporalBinding["kind"];+ readonly sourceName?: string;+ readonly frameName: string;+ readonly specName: string;+ readonly clipPathName?: string;+ readonly layers: readonly FragmentLayer[];+ readonly sounds: readonly FragmentSound[];+};++type FragmentMember = {+ readonly suffix: string;+ readonly binding: "moment" | "selection-start" | "selection-end";+ readonly sourceName: string;+ readonly specName: string;+ readonly layers: readonly FragmentLayer[];+};++type FragmentSequence = {+ readonly suffix: string;+ readonly terminal: "program-end" | "moment" | "selection-start" | "selection-end";+ readonly terminalSourceName?: string;+ readonly frameName: string;+ readonly specName: string;+ readonly clipPathName?: string;+ readonly members: readonly FragmentMember[];+ readonly sounds: readonly FragmentSound[];+};++function appendLayers(operations: FragmentOperation[], prefix: string, layers: readonly FragmentLayer[]): string {+ const emptyId = `${prefix}:layers:empty`;+ operations.push({ id: emptyId, producer: mediaTrackProducers.createLayers, inputs: {}, result: { kind: "output", name: "layers" } });+ let current = emptyId;+ for (const [index, layer] of layers.entries()) {+ const id = `${prefix}:layers:${String(index + 1).padStart(4, "0")}`;+ const common = { layers: operation(current), spec: input(layer.specName) };+ if (layer.kind === "paint") {+ operations.push({ id, producer: mediaTrackProducers.appendPaintLayer, inputs: common, result: { kind: "output", name: "layers" } });+ } else if (layer.kind === "still") {+ operations.push({ id, producer: mediaTrackProducers.appendStillLayer, inputs: {+ ...common, source: input(layer.sourceName), extent: input(layer.extentName), fit: input(layer.fitName),+ }, result: { kind: "output", name: "layers" } });+ } else {+ operations.push({ id, producer: layer.kind === "timed" ? mediaTrackProducers.appendTimedLayer : mediaTrackProducers.appendSurfaceLayer,+ inputs: { ...common, source: input(layer.sourceName), fit: input(layer.fitName) }, result: { kind: "output", name: "layers" } });+ }+ current = id;+ }+ return current;+}++function appendSounds(operations: FragmentOperation[], prefix: string, sounds: readonly FragmentSound[]): string {+ const emptyId = `${prefix}:sounds:empty`;+ operations.push({ id: emptyId, producer: mediaTrackProducers.createSounds, inputs: {}, result: { kind: "output", name: "sounds" } });+ let current = emptyId;+ for (const [index, sound] of sounds.entries()) {+ const id = `${prefix}:sounds:${String(index + 1).padStart(4, "0")}`;+ operations.push({ id, producer: mediaTrackProducers.appendSound, inputs: {+ sounds: operation(current), source: input(sound.sourceName), spec: input(sound.specName),+ }, result: { kind: "output", name: "sounds" } });+ current = id;+ }+ return current;+}++function createMediaTrackSurfaceFragment(inputTypes: readonly { readonly name: string; readonly type: TypeRef }[], items: readonly FragmentItem[], sequences: readonly FragmentSequence[], name: string, audio: boolean) {+ const operations: FragmentOperation[] = [{ id: "track:set:empty", producer: mediaTrackProducers.createSet, inputs: {}, result: { kind: "output", name: "set" } }];+ let set = "track:set:empty";+ for (const item of items) {+ const prefix = `item:${item.suffix}`;+ const layers = appendLayers(operations, prefix, item.layers);+ const sounds = appendSounds(operations, prefix, item.sounds);+ const appendId = `${prefix}:append`;+ let spec: FragmentOperation["inputs"][string] = input(item.specName);+ if (item.clipPathName !== undefined) {+ const bindId = `${prefix}:bind-clip-path`;+ operations.push({ id: bindId, producer: mediaTrackProducers.bindItemClipPath, inputs: {+ spec, path: input(item.clipPathName),+ }, result: { kind: "output", name: "spec" } });+ spec = operation(bindId);+ }+ const common = {+ set: operation(set), header: input("header"), space: input("space"), canvas: input("canvas"),+ layers: operation(layers), frame: input(item.frameName), spec, sounds: operation(sounds),+ };+ operations.push({+ id: appendId,+ producer: item.binding === "program" ? mediaTrackProducers.appendProgramItem+ : item.binding === "selection" ? mediaTrackProducers.appendSelectionItem : mediaTrackProducers.appendMomentItem,+ inputs: item.binding === "program" ? common : {+ ...common, map: input("map"), [item.binding]: input(item.sourceName!),+ },+ result: { kind: "output", name: "set" },+ });+ set = appendId;+ }+ for (const sequence of sequences) {+ const prefix = `sequence:${sequence.suffix}`;+ const emptyMembers = `${prefix}:members:empty`;+ operations.push({ id: emptyMembers, producer: mediaTrackProducers.createMembers, inputs: {}, result: { kind: "output", name: "members" } });+ let members = emptyMembers;+ for (const member of sequence.members) {+ const memberPrefix = `${prefix}:member:${member.suffix}`;+ const layers = appendLayers(operations, memberPrefix, member.layers);+ const appendId = `${memberPrefix}:append`;+ const producer = member.binding === "moment" ? mediaTrackProducers.appendMomentMember+ : member.binding === "selection-start" ? mediaTrackProducers.appendSelectionStartMember+ : mediaTrackProducers.appendSelectionEndMember;+ operations.push({ id: appendId, producer, inputs: {+ members: operation(members), layers: operation(layers), spec: input(member.specName), map: input("map"),+ space: input("space"), [member.binding === "moment" ? "moment" : "selection"]: input(member.sourceName),+ }, result: { kind: "output", name: "members" } });+ members = appendId;+ }+ const sounds = appendSounds(operations, prefix, sequence.sounds);+ const appendId = `${prefix}:append`;+ let spec: FragmentOperation["inputs"][string] = input(sequence.specName);+ if (sequence.clipPathName !== undefined) {+ const bindId = `${prefix}:bind-clip-path`;+ operations.push({ id: bindId, producer: mediaTrackProducers.bindSequenceClipPath, inputs: {+ spec, path: input(sequence.clipPathName),+ }, result: { kind: "output", name: "spec" } });+ spec = operation(bindId);+ }+ const common = {+ set: operation(set), header: input("header"), space: input("space"), canvas: input("canvas"),+ members: operation(members), frame: input(sequence.frameName), spec, sounds: operation(sounds),+ };+ const producer = sequence.terminal === "program-end" ? mediaTrackProducers.appendSequenceProgramEnd+ : sequence.terminal === "moment" ? mediaTrackProducers.appendSequenceUntilMoment+ : sequence.terminal === "selection-start" ? mediaTrackProducers.appendSequenceUntilSelectionStart+ : mediaTrackProducers.appendSequenceUntilSelectionEnd;+ operations.push({ id: appendId, producer, inputs: sequence.terminal === "program-end" ? common : {+ ...common, map: input("map"), [sequence.terminal === "moment" ? "moment" : "selection"]: input(sequence.terminalSourceName!),+ }, result: { kind: "output", name: "set" } });+ set = appendId;+ }+ operations.push(+ { id: "track:finalize", producer: mediaTrackProducers.finalize, inputs: { set: operation(set), header: input("header"), space: input("space") }, result: { kind: "output", name: "program" } },+ { id: "track:visual", producer: mediaTrackProducers.projectVisual, inputs: { space: input("space"), program: operation("track:finalize") }, result: { kind: "output", name: "track" } },+ );+ if (audio) operations.push({ id: "track:audio", producer: mediaTrackProducers.projectAudio, inputs: { space: input("space"), program: operation("track:finalize") }, result: { kind: "output", name: "track" } });+ const semanticInputs = inputTypes.map((entry) => entry.name);+ return sealGraphFragment({+ name,+ inputs: inputTypes,+ operations,+ exports: [+ { name: "visual", type: compositionTypes.visualTrack, root: operation("track:visual"), semanticInputs, fidelity: "exact" },+ ...(audio ? [{ name: "audio" as const, type: compositionTypes.audioTrack, root: operation("track:audio"), semanticInputs, fidelity: "exact" as const }] : []),+ ],+ });+}++type SurfaceBuilder = {+ readonly records: SurfaceRecordDraft[];+ readonly inputs: Record<string, SurfaceResolvedReference["ref"] | { readonly kind: "record"; readonly id: string }>;+ readonly inputTypes: { name: string; type: TypeRef }[];+ addReference(name: string, value: SurfaceResolvedReference): void;+ addRecord(name: string, id: string, type: TypeRef, value: unknown, range: StructuredElement["range"]): void;+};++function builder(): SurfaceBuilder {+ const records: SurfaceRecordDraft[] = [];+ const inputs: SurfaceBuilder["inputs"] = {};+ const inputTypes: SurfaceBuilder["inputTypes"] = [];+ return {+ records,+ inputs,+ inputTypes,+ addReference(name, value) {+ if (inputs[name] !== undefined) throw new Error(`Media Surface repeats input ${name}.`);+ inputs[name] = value.ref;+ inputTypes.push({ name, type: value.type });+ },+ addRecord(name, id, type, value, range) {+ if (inputs[name] !== undefined) throw new Error(`Media Surface repeats input ${name}.`);+ records.push({ id, type, value: { kind: "inline", value: value as never }, range });+ inputs[name] = { kind: "record", id };+ inputTypes.push({ name, type });+ },+ };+}++function suffix(index: number): string {+ return String(index).padStart(4, "0");+}++type SourceLayerContext = {+ readonly trackId: string;+ readonly unitSuffix: string;+ readonly defaultRecipe: SvsRecipe;+ readonly sourceElement: StructuredElement;+ readonly source?: SurfaceResolvedReference;+ readonly extent?: SurfaceResolvedReference;+ readonly ignoreSiblingChildren?: boolean;+ readonly defaultLayerId?: string;+};++function sampling(element: StructuredElement): MediaSamplingMotion["keyframes"][number] {+ allowed(element, ["at", "zoom", "x", "y", "rotate", "easing"]);+ empty(element);+ const at = text(element, "at");+ const percentage = /^(\d+(?:\.\d+)?)%$/u.exec(at);+ const atProgress = at === "start" ? 0 : at === "end" ? 1+ : percentage === null ? Number.NaN : Number(percentage[1]) / 100;+ if (!Number.isFinite(atProgress) || atProgress < 0 || atProgress > 1) {+ throw new Error(`${element.name}.at must be start, end or a percentage inside 0%..100%.`);+ }+ const easing = optionalText(element, "easing");+ if (easing !== undefined && !["linear", "ease-in", "ease-out", "ease-in-out"].includes(easing)) {+ throw new Error(`${element.name}.easing is invalid.`);+ }+ return {+ atProgress, zoom: numberValue(element, "zoom", 1), offsetX: numberValue(element, "x", 0),+ offsetY: numberValue(element, "y", 0), rotationDeg: numberValue(element, "rotate", 0),+ ...(easing === undefined ? {} : { easing: easing as "linear" | "ease-in" | "ease-out" | "ease-in-out" }),+ };+}++function sourceLayer(+ state: SurfaceBuilder,+ context: SourceLayerContext,+ layerIndex: number,+ resolve: (path: string) => SurfaceResolvedReference | undefined,+): FragmentLayer {+ const element = context.sourceElement;+ const source = context.source ?? oneOfReferences(element.attributes.source, `${element.name}.source`, [artifactTypes.blob, mediaTypes.synchronized, mediaTypes.compositableSurface], resolve);+ const extent = context.extent ?? (sameType(source.type, artifactTypes.blob)+ ? reference(element.attributes.extent, `${element.name}.extent`, spatialTypes.extent, resolve)+ : undefined);+ if (!sameType(source.type, artifactTypes.blob) && element.attributes.extent !== undefined) {+ throw new Error(`${element.name}.extent is only valid for a durationless Blob source.`);+ }+ const appearance = element.attributes.appearance === undefined+ ? context.defaultRecipe+ : recipe(reference(element.attributes.appearance, `${element.name}.appearance`, svsRecipeType, resolve), `${element.name}.appearance`);+ const keyframes = element.children.flatMap((child) => {+ if (child.kind === "text") {+ if (child.value.trim().length > 0) throw new Error(`${element.name} accepts only Sampling children.`);+ return [];+ }+ if (!child.name.endsWith(":Sampling") && child.name !== "Sampling") {+ if (context.ignoreSiblingChildren === true) return [];+ throw new Error(`${element.name} accepts only Sampling children.`);+ }+ return [sampling(child)];+ });+ const motion = keyframes.length === 0 ? undefined : { keyframes };+ const layerSuffix = `${context.unitSuffix}-layer-${suffix(layerIndex)}`;+ const layerId = context.defaultLayerId ?? optionalText(element, "id") ?? `${context.trackId}.${layerSuffix}`;+ const fitName = `${layerSuffix}-fit`;+ const specName = `${layerSuffix}-spec`;+ state.addRecord(fitName, `${context.trackId}.${layerSuffix}.fit`, spatialTypes.fit, decodeMediaFit(appearance), element.range);+ const sourceKind = sameType(source.type, artifactTypes.blob) ? "still"+ : sameType(source.type, mediaTypes.synchronized) ? "timed" : "surface";+ const spec = decodeMediaSampleSpec(appearance, layerId, sourceKind, motion, context.source !== undefined);+ state.addRecord(specName, `${context.trackId}.${layerSuffix}.spec`, mediaTrackTypes.sampleLayerSpec, spec, element.range);+ const sourceName = `${layerSuffix}-source`;+ state.addReference(sourceName, source);+ if (sourceKind === "still") {+ const extentName = `${layerSuffix}-extent`;+ state.addReference(extentName, extent!);+ return { kind: "still", sourceName, extentName, fitName, specName };+ }+ return { kind: sourceKind, sourceName, fitName, specName };+}++function unitLayers(+ state: SurfaceBuilder,+ input: {+ readonly trackId: string;+ readonly unitSuffix: string;+ readonly element: StructuredElement;+ readonly appearance: SvsRecipe;+ readonly directSource?: SurfaceResolvedReference;+ readonly directExtent?: SurfaceResolvedReference;+ readonly allowFramePaint: boolean;+ },+ resolve: (path: string) => SurfaceResolvedReference | undefined,+): readonly FragmentLayer[] {+ const layers: FragmentLayer[] = [];+ if (input.allowFramePaint) {+ const framePaint = decodeMediaFramePaint(input.appearance, `${input.trackId}.${input.unitSuffix}.frame-paint`);+ if (framePaint !== undefined) {+ const name = `${input.unitSuffix}-frame-paint`;+ state.addRecord(name, `${input.trackId}.${input.unitSuffix}.frame-paint`, mediaTrackTypes.paintLayerSpec, framePaint, input.element.range);+ layers.push({ kind: "paint", specName: name });+ }+ }+ const explicit = input.element.children.filter((child): child is StructuredElement => child.kind === "element"+ && (child.name.endsWith(":Paint") || child.name === "Paint" || child.name.endsWith(":Layer") || child.name === "Layer"));+ if (input.directSource !== undefined && explicit.length > 0) {+ throw new Error(`${input.element.name} cannot combine a direct source with Paint or Layer children.`);+ }+ if (input.directSource !== undefined) {+ layers.push(sourceLayer(state, {+ trackId: input.trackId, unitSuffix: input.unitSuffix, defaultRecipe: input.appearance,+ sourceElement: input.element, source: input.directSource, ...(input.directExtent === undefined ? {} : { extent: input.directExtent }),+ ignoreSiblingChildren: true,+ defaultLayerId: "content",+ }, 1, resolve));+ return layers;+ }+ let layerIndex = 0;+ for (const child of explicit) {+ layerIndex += 1;+ if (child.name.endsWith(":Paint") || child.name === "Paint") {+ allowed(child, ["id", "appearance"]);+ empty(child);+ const appearance = recipe(reference(child.attributes.appearance, `${child.name}.appearance`, svsRecipeType, resolve), `${child.name}.appearance`);+ const id = optionalText(child, "id") ?? `${input.trackId}.${input.unitSuffix}.paint.${suffix(layerIndex)}`;+ const name = `${input.unitSuffix}-paint-${suffix(layerIndex)}`;+ state.addRecord(name, `${input.trackId}.${input.unitSuffix}.paint.${suffix(layerIndex)}`, mediaTrackTypes.paintLayerSpec,+ decodeMediaPaintSpec(appearance, id), child.range);+ layers.push({ kind: "paint", specName: name });+ continue;+ }+ allowed(child, ["id", "source", "extent", "appearance"]);+ layers.push(sourceLayer(state, {+ trackId: input.trackId, unitSuffix: input.unitSuffix, defaultRecipe: input.appearance, sourceElement: child,+ }, layerIndex, resolve));+ }+ if (layers.length === 0) throw new Error(`${input.element.name} requires a direct source or at least one Paint/Layer child.`);+ return layers;+}++function validateUnitChildren(element: StructuredElement, direct: boolean, allowSound: boolean): void {+ for (const child of element.children) {+ if (child.kind === "text") {+ if (child.value.trim().length > 0) throw new Error(`${element.name} cannot contain text.`);+ continue;+ }+ const samplingChild = child.name.endsWith(":Sampling") || child.name === "Sampling";+ const layerChild = child.name.endsWith(":Layer") || child.name === "Layer";+ const paintChild = child.name.endsWith(":Paint") || child.name === "Paint";+ const soundChild = child.name.endsWith(":Sound") || child.name === "Sound";+ if ((direct && samplingChild) || (!direct && (layerChild || paintChild)) || (allowSound && soundChild)) continue;+ throw new Error(`${element.name} has unsupported child ${child.name}.`);+ }+}++function sourceAudio(element: StructuredElement): MediaSequenceMemberSpec["sourceAudio"] {+ const fromLayer = optionalText(element, "source-audio");+ if (fromLayer === undefined) {+ if (element.attributes["audio-gain"] !== undefined) throw new Error(`${element.name}.audio-gain requires source-audio.`);+ return undefined;+ }+ return { fromLayer, gain: numberValue(element, "audio-gain", 1) };+}++function sound(+ state: SurfaceBuilder,+ element: StructuredElement,+ unitSuffix: string,+ trackId: string,+ index: number,+ handoffIds: ReadonlySet<string>,+ resolve: (path: string) => SurfaceResolvedReference | undefined,+): FragmentSound {+ allowed(element, ["id", "source", "at", "handoff", "gain"]);+ empty(element);+ const at = optionalText(element, "at");+ const handoff = optionalText(element, "handoff");+ if (Number(at !== undefined) + Number(handoff !== undefined) !== 1) {+ throw new Error(`${element.name} requires exactly one of at or handoff.`);+ }+ let trigger: MediaSoundSpec["trigger"];+ if (handoff !== undefined) {+ if (!handoffIds.has(handoff)) throw new Error(`${element.name}.handoff does not name a Handoff in this Sequence.`);+ trigger = { kind: "handoff", handoffId: handoff };+ } else {+ if (at !== "enter" && at !== "exit") throw new Error(`${element.name}.at must be enter or exit.`);+ trigger = { kind: at };+ }+ const soundSuffix = `${unitSuffix}-sound-${suffix(index)}`;+ const specName = `${soundSuffix}-spec`;+ const sourceName = `${soundSuffix}-source`;+ const id = optionalText(element, "id") ?? `${trackId}.${soundSuffix}`;+ state.addRecord(specName, `${trackId}.${soundSuffix}.spec`, mediaTrackTypes.soundSpec,+ sealMediaSoundSpec({ contract: "svml.media-sound-spec@1", id, trigger, gain: numberValue(element, "gain", 1) }), element.range);+ state.addReference(sourceName, reference(element.attributes.source, `${element.name}.source`, mediaTypes.synchronized, resolve));+ return { sourceName, specName };+}++export const decodeMediaTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ allowed(element, ["id", "space", "canvas", "map"]);+ const trackId = text(element, "id");+ const state = builder();+ state.addReference("space", reference(element.attributes.space, `${element.name}.space`, programSpaceTypes.programSpace, resolveReference));+ state.addReference("canvas", reference(element.attributes.canvas, `${element.name}.canvas`, spatialTypes.canvas, resolveReference));+ const headerId = `${trackId}.header`;+ state.addRecord("header", headerId, mediaTrackTypes.header,+ sealMediaTrackHeader({ contract: "svml.media-track-header@1", id: trackId }), element.range);+ const items: FragmentItem[] = [];+ const sequences: FragmentSequence[] = [];+ let itemIndex = 0;+ let sequenceIndex = 0;+ let usesMap = false;+ let hasAudio = false;+ for (const child of element.children) {+ if (child.kind === "text") {+ if (child.value.trim().length > 0) throw new Error(`${element.name} accepts only Item and Sequence children.`);+ continue;+ }+ if (child.name.endsWith(":Item") || child.name === "Item") {+ itemIndex += 1;+ const itemSuffix = suffix(itemIndex);+ allowed(child, [+ "id", "source", "extent", "frame", "appearance", "motion", "source-audio", "audio-gain",+ "clip", "during", "at", "for", "start", "end", "selection", "moment", "occurrences",+ ]);+ const id = optionalText(child, "id") ?? `${trackId}.item.${itemSuffix}`;+ const appearance = recipe(reference(child.attributes.appearance, `${child.name}.appearance`, svsRecipeType, resolveReference), `${child.name}.appearance`);+ const clipPathName = child.attributes.clip === undefined ? undefined : `item-${itemSuffix}-clip-path`;+ if (clipPathName !== undefined) {+ if (appearance.properties.clip !== undefined) throw new Error(`${child.name} cannot combine clip={Path} with Recipe clip.`);+ state.addReference(clipPathName, reference(child.attributes.clip, `${child.name}.clip`, spatialTypes.path, resolveReference));+ }+ const motionRecipe = child.attributes.motion === undefined ? undefined+ : recipe(reference(child.attributes.motion, `${child.name}.motion`, svsRecipeType, resolveReference), `${child.name}.motion`);+ const binding = temporalBinding(child, resolveReference);+ if (binding.kind !== "program") usesMap = true;+ const occurrences = text(child, "occurrences", "one");+ if (occurrences !== "one" && occurrences !== "each") throw new Error(`${child.name}.occurrences must be one or each.`);+ const expansion: OccurrenceExpansion = { kind: occurrences };+ const directSource = child.attributes.source === undefined ? undefined+ : oneOfReferences(child.attributes.source, `${child.name}.source`, [artifactTypes.blob, mediaTypes.synchronized, mediaTypes.compositableSurface], resolveReference);+ const directExtent = directSource !== undefined && sameType(directSource.type, artifactTypes.blob)+ ? reference(child.attributes.extent, `${child.name}.extent`, spatialTypes.extent, resolveReference) : undefined;+ if ((directSource === undefined || !sameType(directSource.type, artifactTypes.blob)) && child.attributes.extent !== undefined) {+ throw new Error(`${child.name}.extent requires a direct Blob source.`);+ }+ validateUnitChildren(child, directSource !== undefined, true);+ const layers = unitLayers(state, { trackId, unitSuffix: `item-${itemSuffix}`, element: child, appearance,+ ...(directSource === undefined ? {} : { directSource }), ...(directExtent === undefined ? {} : { directExtent }), allowFramePaint: true }, resolveReference);+ const selectedAudio = sourceAudio(child);+ const specName = `item-${itemSuffix}-spec`;+ state.addRecord(specName, `${trackId}.item.${itemSuffix}.spec`, mediaTrackTypes.itemSpec,+ decodeMediaItemSpec(appearance, { id, projection: binding.projection, expansion,+ motion: decodeMediaMotion(motionRecipe), ...(selectedAudio === undefined ? {} : { sourceAudio: selectedAudio }) }), child.range);+ const frameName = `item-${itemSuffix}-frame`;+ state.addReference(frameName, reference(child.attributes.frame, `${child.name}.frame`, spatialTypes.frame, resolveReference));+ const fragmentSounds: FragmentSound[] = [];+ for (const soundChild of child.children) {+ if (soundChild.kind === "element" && (soundChild.name.endsWith(":Sound") || soundChild.name === "Sound")) {+ fragmentSounds.push(sound(state, soundChild, `item-${itemSuffix}`, trackId, fragmentSounds.length + 1, new Set(), resolveReference));+ }+ }+ if (selectedAudio !== undefined || fragmentSounds.length > 0) hasAudio = true;+ const sourceName = binding.kind === "program" ? undefined : `item-${itemSuffix}-${binding.kind}`;+ if (binding.source !== undefined) state.addReference(sourceName!, binding.source);+ items.push({ suffix: itemSuffix, binding: binding.kind, ...(sourceName === undefined ? {} : { sourceName }), frameName, specName,+ ...(clipPathName === undefined ? {} : { clipPathName }), layers, sounds: fragmentSounds });+ continue;+ }+ if (!(child.name.endsWith(":Sequence") || child.name === "Sequence")) {+ throw new Error(`${element.name} accepts only Item and Sequence children.`);+ }+ sequenceIndex += 1;+ const sequenceSuffix = suffix(sequenceIndex);+ allowed(child, ["id", "frame", "clip", "appearance", "motion", "until", "until-boundary"]);+ const id = optionalText(child, "id") ?? `${trackId}.sequence.${sequenceSuffix}`;+ const appearance = recipe(reference(child.attributes.appearance, `${child.name}.appearance`, svsRecipeType, resolveReference), `${child.name}.appearance`);+ const clipPathName = child.attributes.clip === undefined ? undefined : `sequence-${sequenceSuffix}-clip-path`;+ if (clipPathName !== undefined) {+ if (appearance.properties.clip !== undefined) throw new Error(`${child.name} cannot combine clip={Path} with Recipe clip.`);+ state.addReference(clipPathName, reference(child.attributes.clip, `${child.name}.clip`, spatialTypes.path, resolveReference));+ }+ const motionRecipe = child.attributes.motion === undefined ? undefined+ : recipe(reference(child.attributes.motion, `${child.name}.motion`, svsRecipeType, resolveReference), `${child.name}.motion`);+ const memberElements = child.children.filter((node): node is StructuredElement => node.kind === "element" && (node.name.endsWith(":Member") || node.name === "Member"));+ if (memberElements.length < 2) throw new Error(`${child.name} requires at least two Member children.`);+ const memberIds = memberElements.map((member, index) => optionalText(member, "id") ?? `${id}.member.${suffix(index + 1)}`);+ const handoffElements = child.children.filter((node): node is StructuredElement => node.kind === "element" && (node.name.endsWith(":Handoff") || node.name === "Handoff"));+ if (handoffElements.length !== memberIds.length - 1) throw new Error(`${child.name} requires exactly one Handoff for every adjacent Member pair.`);+ const handoffs: MediaHandoffSpec[] = [];+ for (const [index, handoff] of handoffElements.entries()) {+ allowed(handoff, ["id", "from", "transition"]);+ empty(handoff);+ const from = text(handoff, "from");+ if (from !== memberIds[index]) throw new Error(`${handoff.name}.from must name Member ${memberIds[index]}.`);+ const handoffId = optionalText(handoff, "id") ?? `${id}.handoff.${suffix(index + 1)}`;+ const transition = recipe(reference(handoff.attributes.transition, `${handoff.name}.transition`, svsRecipeType, resolveReference), `${handoff.name}.transition`);+ handoffs.push(decodeMediaHandoffSpec(transition, handoffId, from, memberIds[index + 1]!));+ }+ const handoffIds = new Set(handoffs.map((handoff) => handoff.id));+ const members: FragmentMember[] = [];+ for (const [index, member] of memberElements.entries()) {+ const memberSuffix = suffix(index + 1);+ allowed(member, ["id", "source", "extent", "appearance", "at", "boundary", "source-audio", "audio-gain"]);+ const at = oneOfReferences(member.attributes.at, `${member.name}.at`, [narrativeTypes.moment, narrativeTypes.selection], resolveReference);+ usesMap = true;+ const boundary = optionalText(member, "boundary");+ const binding = sameType(at.type, narrativeTypes.moment) ? "moment"+ : boundary === "start" ? "selection-start" : boundary === "end" ? "selection-end" : undefined;+ if (binding === undefined) throw new Error(`${member.name}.boundary must be start or end for a Selection activation.`);+ if (sameType(at.type, narrativeTypes.moment) && boundary !== undefined) throw new Error(`${member.name}.boundary is invalid for a Moment activation.`);+ const memberAppearance = member.attributes.appearance === undefined ? appearance+ : recipe(reference(member.attributes.appearance, `${member.name}.appearance`, svsRecipeType, resolveReference), `${member.name}.appearance`);+ const directSource = member.attributes.source === undefined ? undefined+ : oneOfReferences(member.attributes.source, `${member.name}.source`, [artifactTypes.blob, mediaTypes.synchronized, mediaTypes.compositableSurface], resolveReference);+ const directExtent = directSource !== undefined && sameType(directSource.type, artifactTypes.blob)+ ? reference(member.attributes.extent, `${member.name}.extent`, spatialTypes.extent, resolveReference) : undefined;+ if ((directSource === undefined || !sameType(directSource.type, artifactTypes.blob)) && member.attributes.extent !== undefined) {+ throw new Error(`${member.name}.extent requires a direct Blob source.`);+ }+ validateUnitChildren(member, directSource !== undefined, false);+ const unitSuffix = `sequence-${sequenceSuffix}-member-${memberSuffix}`;+ const layers = unitLayers(state, { trackId, unitSuffix, element: member, appearance: memberAppearance,+ ...(directSource === undefined ? {} : { directSource }), ...(directExtent === undefined ? {} : { directExtent }), allowFramePaint: true }, resolveReference);+ const selectedAudio = sourceAudio(member);+ if (selectedAudio !== undefined) hasAudio = true;+ const specName = `${unitSuffix}-spec`;+ state.addRecord(specName, `${trackId}.${unitSuffix}.spec`, mediaTrackTypes.memberSpec,+ sealMediaSequenceMemberSpec({ contract: "svml.media-sequence-member-spec@1", id: memberIds[index]!,+ ...(selectedAudio === undefined ? {} : { sourceAudio: selectedAudio }) }), member.range);+ const sourceName = `${unitSuffix}-${binding === "moment" ? "moment" : "selection"}`;+ state.addReference(sourceName, at);+ members.push({ suffix: memberSuffix, binding, sourceName, specName, layers });+ }+ const sequenceSounds: FragmentSound[] = [];+ for (const soundChild of child.children) {+ if (soundChild.kind === "element" && (soundChild.name.endsWith(":Sound") || soundChild.name === "Sound")) {+ sequenceSounds.push(sound(state, soundChild, `sequence-${sequenceSuffix}`, trackId, sequenceSounds.length + 1, handoffIds, resolveReference));+ } else if (soundChild.kind === "element" && !(+ soundChild.name.endsWith(":Member") || soundChild.name === "Member"+ || soundChild.name.endsWith(":Handoff") || soundChild.name === "Handoff"+ )) throw new Error(`${child.name} accepts only Member, Handoff and Sound children.`);+ }+ if (sequenceSounds.length > 0) hasAudio = true;+ const until = child.attributes.until;+ let terminal: FragmentSequence["terminal"];+ let terminalReference: SurfaceResolvedReference | undefined;+ if (typeof until === "string") {+ if (until.trim() !== "program.end") throw new Error(`${child.name}.until text must be program.end.`);+ terminal = "program-end";+ if (child.attributes["until-boundary"] !== undefined) throw new Error(`${child.name}.until-boundary is invalid for program.end.`);+ } else {+ terminalReference = oneOfReferences(until, `${child.name}.until`, [narrativeTypes.moment, narrativeTypes.selection], resolveReference);+ usesMap = true;+ if (sameType(terminalReference.type, narrativeTypes.moment)) {+ if (child.attributes["until-boundary"] !== undefined) throw new Error(`${child.name}.until-boundary is invalid for a Moment.`);+ terminal = "moment";+ } else {+ const boundary = text(child, "until-boundary", "end");+ if (boundary !== "start" && boundary !== "end") throw new Error(`${child.name}.until-boundary must be start or end.`);+ terminal = boundary === "start" ? "selection-start" : "selection-end";+ }+ }+ const frameName = `sequence-${sequenceSuffix}-frame`;+ state.addReference(frameName, reference(child.attributes.frame, `${child.name}.frame`, spatialTypes.frame, resolveReference));+ const specName = `sequence-${sequenceSuffix}-spec`;+ state.addRecord(specName, `${trackId}.sequence.${sequenceSuffix}.spec`, mediaTrackTypes.sequenceSpec,+ decodeMediaSequenceSpec(appearance, id, decodeMediaMotion(motionRecipe), handoffs), child.range);+ const terminalSourceName = terminalReference === undefined ? undefined : `sequence-${sequenceSuffix}-terminal`;+ if (terminalReference !== undefined) state.addReference(terminalSourceName!, terminalReference);+ sequences.push({ suffix: sequenceSuffix, terminal, ...(terminalSourceName === undefined ? {} : { terminalSourceName }), frameName, specName,+ ...(clipPathName === undefined ? {} : { clipPathName }), members, sounds: sequenceSounds });+ }+ if (items.length === 0 && sequences.length === 0) throw new Error(`${element.name} requires at least one Item or Sequence.`);+ if (usesMap) {+ state.addReference("map", reference(element.attributes.map, `${element.name}.map`, semanticMapTypes.complete, resolveReference));+ } else if (element.attributes.map !== undefined) {+ throw new Error(`${element.name}.map is unused because no child consumes semantic timing.`);+ }+ const fragment = createMediaTrackSurfaceFragment(state.inputTypes, items, sequences, `@narratage/media-track/surface/${trackId}@1`, hasAudio);+ return {+ records: state.records,+ components: [{ id: trackId, fragment: fragment.id, inputs: state.inputs,+ outputs: { visual: `${trackId}.visual`, ...(hasAudio ? { audio: `${trackId}.audio` } : {}) }, range: element.range }],+ fragments: [fragment],+ };+};diff --git a/packages/media-track/src/types.ts b/packages/media-track/src/types.tsindex 099671cf..420ca5f8 100644--- a/packages/media-track/src/types.ts+++ b/packages/media-track/src/types.ts@@ -1,51 +1,338 @@-import type { FrameSpan } from "@narratage/composition";-import type { MediaArtifactRef } from "@narratage/media";-import type { ContentFit, IntrinsicExtent, SpatialFrame } from "@narratage/spatial";+import type { AudioTrack, FrameSpan, VisualTrack } from "@narratage/composition";+import type { CompositableSurfaceRef, MediaRational } from "@narratage/media";+import type { BlobRef } from "@narratage/protocol";+import type {+ ContentFit,+ IntrinsicExtent,+ SpatialFrame,+ SpatialPath,+} from "@narratage/spatial";+import type {+ OccurrenceExpansion,+ TemporalWindowProjection,+} from "@narratage/temporal";-export type MediaFramePresentation = {- readonly clip: "frame";- readonly fill: { readonly kind: "transparent" };+/** Intrinsic visual truth resolved before Media authoring; never a Provider or lineage envelope. */+export type MediaVisualSource =+ | {+ readonly kind: "still";+ readonly artifact: BlobRef;+ readonly extent: IntrinsicExtent;+ }+ | {+ readonly kind: "timed";+ readonly artifact: BlobRef;+ readonly extent: IntrinsicExtent;+ readonly frameRate: MediaRational;+ readonly frameCount: number;+ readonly audio?: {+ readonly artifact: BlobRef;+ readonly sampleFrames: number;+ };+ }+ | {+ readonly kind: "surface";+ readonly surface: CompositableSurfaceRef;+ readonly extent: IntrinsicExtent;+ };++export type MediaVisualTrim = {+ readonly startFrame: number;+ readonly endFrameExclusive: number;+};++export type MediaVisualOccupancy =+ | { readonly mode: "once"; readonly align: "start" | "end" }+ | { readonly mode: "hold"; readonly align: "start" | "end" }+ | { readonly mode: "loop"; readonly align: "start" | "end" }+ | { readonly mode: "stretch" };++export type MediaGradientStop = {+ readonly offset: number;+ readonly color: string;+};++export type MediaPaint =+ | { readonly kind: "solid"; readonly color: string }+ | {+ readonly kind: "linear-gradient";+ readonly angleDeg: number;+ readonly stops: readonly MediaGradientStop[];+ }+ | {+ readonly kind: "radial-gradient";+ readonly center: { readonly x: number; readonly y: number };+ readonly stops: readonly MediaGradientStop[];+ };++export type MediaSampleAppearance = {+ readonly opacity: number;+ readonly filter: {+ readonly blurPx: number;+ readonly brightness: number;+ readonly contrast: number;+ readonly saturation: number;+ };+};++export type MediaSamplingKeyframe = {+ /** Normalized position inside the resolved Item/member envelope. */+ readonly atProgress: number;+ readonly zoom: number;+ readonly offsetX: number;+ readonly offsetY: number;+ readonly rotationDeg: number;+ readonly easing?: "linear" | "ease-in" | "ease-out" | "ease-in-out";+};++export type MediaSamplingMotion = {+ readonly keyframes: readonly MediaSamplingKeyframe[];+};++export type MediaPaintLayerProgram = {+ readonly id: string;+ readonly kind: "paint";+ readonly paint: MediaPaint;+ readonly opacity: number;};-export type MediaStillLayerProgram = {+export type MediaSampleLayerProgram = {readonly id: string;- readonly kind: "still";- readonly artifact: MediaArtifactRef;- readonly extent: IntrinsicExtent;+ readonly kind: "sample";+ readonly source: MediaVisualSource;readonly fit: ContentFit;+ readonly trim?: MediaVisualTrim;+ readonly occupancy?: MediaVisualOccupancy;+ readonly appearance: MediaSampleAppearance;+ readonly samplingMotion?: MediaSamplingMotion;+};++export type MediaLayerProgram = MediaPaintLayerProgram | MediaSampleLayerProgram;++export type MediaPaintLayerSpec = {+ readonly contract: "svml.media-paint-layer-spec@1";+ readonly id: string;+ readonly paint: MediaPaint;+ readonly opacity: number;+};++export type MediaSampleLayerSpec = {+ readonly contract: "svml.media-sample-layer-spec@1";+ readonly id: string;+ readonly trim?: MediaVisualTrim;+ readonly occupancy?: MediaVisualOccupancy;+ readonly appearance: MediaSampleAppearance;+ readonly samplingMotion?: MediaSamplingMotion;+};++export type MediaLayerSet = {+ readonly contract: "svml.media-layer-set@1";+ readonly layers: readonly MediaLayerProgram[];+};++export type MediaPadding = {+ readonly topPx: number;+ readonly rightPx: number;+ readonly bottomPx: number;+ readonly leftPx: number;+};++export type MediaFrameClip =+ | { readonly kind: "none" }+ | { readonly kind: "frame" }+ | { readonly kind: "rounded"; readonly radiusPx: number }+ | { readonly kind: "path"; readonly path: SpatialPath };++export type MediaFramePresentation = {+ readonly clip: MediaFrameClip;+ readonly padding: MediaPadding;+ readonly border?: {+ readonly widthPx: number;+ readonly style: "solid" | "dashed" | "dotted";+ readonly color: string;+ };+ readonly shadows: readonly {+ readonly offsetX: number;+ readonly offsetY: number;+ readonly blurPx: number;+ readonly spreadPx: number;+ readonly color: string;+ }[];+};++export type MediaMotionDirection = "left" | "right" | "up" | "down";++export type MediaEdgeMotion = {+ readonly operator: "fade" | "slide" | "scale" | "pop" | "bounce" | "blur-reveal" | "wipe" | "flip" | "spin";+ readonly durationFrames: number;+ readonly easing: "linear" | "ease-in" | "ease-out" | "ease-in-out";+ readonly direction?: MediaMotionDirection;+ readonly amount?: number;+ readonly origin?: "outside-canvas";+};++export type MediaSustainMotion = {+ readonly operator: "float" | "breathe" | "pulse" | "wobble" | "shake" | "drift";+ readonly amount: number;+ readonly cycles: number;+ readonly direction?: MediaMotionDirection;+};++export type MediaLifecycleMotion = {+ readonly enter?: MediaEdgeMotion;+ readonly sustain: readonly MediaSustainMotion[];+ readonly exit?: MediaEdgeMotion;+};++export type MediaSourceAudioProjection = {+ readonly fromLayer: string;+ readonly gain: number;+};++export type MediaSoundSource = {+ readonly artifact: BlobRef;+ readonly sampleFrames: number;+};++export type MediaSoundTrigger =+ | { readonly kind: "enter" }+ | { readonly kind: "exit" }+ | { readonly kind: "handoff"; readonly handoffId: string };++export type MediaSoundSpec = {+ readonly contract: "svml.media-sound-spec@1";+ readonly id: string;+ readonly trigger: MediaSoundTrigger;+ readonly gain: number;+};++export type MediaSoundEvent = Omit<MediaSoundSpec, "contract"> & {+ readonly source: MediaSoundSource;+};++export type MediaSoundSet = {+ readonly contract: "svml.media-sound-set@1";+ readonly sounds: readonly MediaSoundEvent[];+};++export type MediaAbsoluteStacking = {+ readonly order: number;+ readonly tieBreak: string;+};++export type MediaItemSpec = {+ readonly contract: "svml.media-item-spec@1";+ readonly id: string;+ readonly projection: TemporalWindowProjection;+ readonly expansion: OccurrenceExpansion;+ readonly presentation: MediaFramePresentation;+ readonly motion: MediaLifecycleMotion;+ readonly stackingOrder: number;+ readonly sourceAudio?: MediaSourceAudioProjection;};export type MediaItemProgram = {readonly id: string;+ readonly sourceOccurrenceId: string;readonly span: FrameSpan;readonly frame: SpatialFrame;readonly presentation: MediaFramePresentation;- readonly layers: readonly MediaStillLayerProgram[];- readonly stacking: {- readonly order: number;- readonly tieBreak: string;- };+ readonly layers: readonly MediaLayerProgram[];+ readonly motion: MediaLifecycleMotion;+ readonly stacking: MediaAbsoluteStacking;+ readonly sourceAudio?: MediaSourceAudioProjection;+ readonly sounds: readonly MediaSoundEvent[];};-export type MediaTrackProgram = {- readonly contract: "svml.media-track-program@1";+export type MediaHandoffOperator = "cut" | "crossfade" | "push" | "wipe" | "cover" | "page-turn";++export type MediaHandoffProgram = {readonly id: string;- readonly items: readonly MediaItemProgram[];+ readonly fromMemberId: string;+ readonly toMemberId: string;+ readonly operator: MediaHandoffOperator;+ readonly durationFrames: number;+ readonly boundaryRatio: number;+ readonly span: FrameSpan;+ readonly direction?: MediaMotionDirection;+ readonly audio: "cut" | "crossfade";};-export type MediaTrackHeader = {- readonly contract: "svml.media-track-header@1";+export type MediaSequenceMemberProgram = {readonly id: string;+ readonly activationFrame: number;+ readonly logicalSpan: FrameSpan;+ readonly visualSpan: FrameSpan;+ readonly layers: readonly MediaLayerProgram[];+ readonly sourceAudio?: MediaSourceAudioProjection;};-export type MediaStillItemSpec = {- readonly contract: "svml.media-still-item-spec@1";+export type MediaSequenceMemberSpec = {+ readonly contract: "svml.media-sequence-member-spec@1";readonly id: string;+ readonly sourceAudio?: MediaSourceAudioProjection;+};++export type MediaSequenceMemberSet = {+ readonly contract: "svml.media-sequence-member-set@1";+ readonly members: readonly {+ readonly id: string;+ readonly activationFrame: number;+ readonly layers: readonly MediaLayerProgram[];+ readonly sourceAudio?: MediaSourceAudioProjection;+ }[];+};++export type MediaHandoffSpec = Omit<MediaHandoffProgram, "span"> & {+ readonly contract: "svml.media-handoff-spec@1";+};++export type MediaSequenceSpec = {+ readonly contract: "svml.media-sequence-spec@1";+ readonly id: string;+ readonly presentation: MediaFramePresentation;+ readonly motion: MediaLifecycleMotion;readonly stackingOrder: number;+ readonly handoffs: readonly MediaHandoffSpec[];+};++export type MediaSequenceProgram = {+ readonly id: string;+ readonly span: FrameSpan;+ readonly terminalFrame: number;+ readonly frame: SpatialFrame;readonly presentation: MediaFramePresentation;+ readonly members: readonly MediaSequenceMemberProgram[];+ readonly handoffs: readonly MediaHandoffProgram[];+ readonly motion: MediaLifecycleMotion;+ readonly stacking: MediaAbsoluteStacking;+ readonly sounds: readonly MediaSoundEvent[];+};++export type MediaTrackProgram = {+ readonly contract: "svml.media-track-program@1";+ readonly id: string;+ readonly items: readonly MediaItemProgram[];+ readonly sequences: readonly MediaSequenceProgram[];+};++export type MediaTrackHeader = {+ readonly contract: "svml.media-track-header@1";+ readonly id: string;};export type MediaTrackSet = {readonly contract: "svml.media-track-set@1";readonly items: readonly MediaItemProgram[];+ readonly sequences: readonly MediaSequenceProgram[];+};++export type MediaVisualProjection = {+ readonly contract: "svml.media-visual-projection@1";+ readonly track: VisualTrack;+};++export type MediaAudioProjection = {+ readonly contract: "svml.media-audio-projection@1";+ readonly track: AudioTrack;};diff --git a/packages/media-track/test/media-track.test.ts b/packages/media-track/test/media-track.test.tsindex 6f56672a..b59a656c 100644--- a/packages/media-track/test/media-track.test.ts+++ b/packages/media-track/test/media-track.test.ts@@ -1,24 +1,81 @@import assert from "node:assert/strict";import test from "node:test";-import { sealProgramSpace } from "@narratage/program-space";-import { digestOf } from "@narratage/protocol";-import type { BlobRef } from "@narratage/protocol";+import { sealComposition } from "@narratage/composition";+import type { AudioTrack } from "@narratage/composition";+import { compileHyperframesDocument } from "@narratage/hyperframes";+import { mediaTypes } from "@narratage/media";+import type { CompositableSurfaceRef, SynchronizedMedia } from "@narratage/media";+import { artifactTypes } from "@narratage/artifact";+import { narrativeTypes } from "@narratage/narrative";import {- appendFullStillMediaItem,+ appendMediaPaintLayer,+ appendMediaSound,+ appendMediaSequence,+ appendMediaSequenceMember,+ appendMomentMediaItem,+ appendProgramMediaItem,+ appendSelectionMediaItem,+ appendStillMediaLayer,+ appendSurfaceMediaLayer,+ appendTimedMediaLayer,+ bindMediaItemClipPath,+ createMediaLayerSet,+ createMediaSequenceMemberSet,+ createMediaSoundSet,createMediaTrackSet,+ decodeMediaTrackSurface,finalizeMediaTrack,- renderMediaTrack,- sealMediaStillItemSpec,+ projectMediaAudioTrack,+ projectMediaVisualTrack,+ lifecycleAnimation,+ resolveMediaLifecycleMotion,+ sustainAnimation,+ resolveVisualSampling,+ sealMediaItemSpec,+ sealMediaHandoffSpec,+ sealMediaPaintLayerSpec,+ sealMediaSampleLayerSpec,+ sealMediaSoundSpec,+ sealMediaSequenceMemberSpec,+ sealMediaSequenceSpec,sealMediaTrackHeader,stillMediaTrackFragment,+ mediaTrackManifest,} from "@narratage/media-track";+import type {+ MediaHandoffOperator,+ MediaItemSpec,+ MediaLayerSet,+ MediaSequenceSpec,+} from "@narratage/media-track";+import type { NarrativeMomentRef, NarrativeSelectionRef } from "@narratage/narrative";+import { sealProgramSpace } from "@narratage/program-space";+import { programSpaceTypes } from "@narratage/program-space";+import { digestOf } from "@narratage/protocol";+import type { BlobRef } from "@narratage/protocol";+import { semanticMapTypes } from "@narratage/semantic-map";+import type { CompleteSemanticMap } from "@narratage/semantic-map";+import { spatialTypes } from "@narratage/spatial";+import { svsRecipeType } from "@narratage/svs";+import type { SvsRecipe } from "@narratage/svs";+import type { StructuredElement, StructuredNode, SurfaceResolvedReference, TextAttributeValue } from "@narratage/text";const space = sealProgramSpace({contract: "svml.program-space@1",durationSec: 4,frameRate: { numerator: 30, denominator: 1 },});+const canvas = {+ contract: "svml.canvas-space@1" as const,+ widthPx: 1080,+ heightPx: 1920,+ origin: "top-left" as const,+ xDirection: "right" as const,+ yDirection: "down" as const,+ pixelAspect: "square" as const,+};+const header = sealMediaTrackHeader({ contract: "svml.media-track-header@1", id: "proof" });const source: BlobRef = {kind: "blob",digest: digestOf("media-track:still"),@@ -35,61 +92,846 @@ const fit = {offsetPx: { x: 0, y: 0 },constraint: "bounded" as const,};+const appearance = {+ opacity: 1,+ filter: { blurPx: 0, brightness: 1, contrast: 1, saturation: 1 },+};-test("one still Media Item consumes explicit time, source, extent, Frame and Fit edges", () => {- const header = sealMediaTrackHeader({ contract: "svml.media-track-header@1", id: "proof" });- const spec = sealMediaStillItemSpec({- contract: "svml.media-still-item-spec@1",+function itemSpec(overrides: Partial<MediaItemSpec> = {}): MediaItemSpec {+ return sealMediaItemSpec({+ contract: "svml.media-item-spec@1",id: "product",+ projection: { start: { ref: "program.start" }, end: { ref: "program.end" } },+ expansion: { kind: "one" },+ presentation: {+ clip: { kind: "rounded", radiusPx: 20 },+ padding: { topPx: 10, rightPx: 10, bottomPx: 10, leftPx: 10 },+ border: { widthPx: 2, style: "solid", color: "#ffffff" },+ shadows: [{ offsetX: 0, offsetY: 8, blurPx: 20, spreadPx: 0, color: "#00000080" }],+ },+ motion: { sustain: [] },stackingOrder: 40,- presentation: { clip: "frame", fill: { kind: "transparent" } },+ ...overrides,});- const program = finalizeMediaTrack(appendFullStillMediaItem(- createMediaTrackSet(), header, space, source, extent, frame, fit, spec,- ), header);+}++function stillLayers(): MediaLayerSet {+ let layers = createMediaLayerSet();+ layers = appendMediaPaintLayer(layers, sealMediaPaintLayerSpec({+ contract: "svml.media-paint-layer-spec@1",+ id: "backing",+ paint: { kind: "linear-gradient", angleDeg: 90, stops: [+ { offset: 0, color: "#101010" }, { offset: 1, color: "#303030" },+ ] },+ opacity: 1,+ }));+ layers = appendStillMediaLayer(layers, source, extent, fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1",+ id: "content",+ appearance,+ }));+ return layers;+}++function timed(id = "timed", withAudio = true): SynchronizedMedia {+ const sampleFrames = 96_000;+ return {+ contract: "svml.synchronized-media@1",+ timeline: {+ spanAuthority: "video",+ frameRate: { numerator: 30, denominator: 1 },+ frameCount: 60,+ sampleRate: 48_000,+ sampleFrames,+ },+ sourceMap: {+ sourceOriginPts: { ticks: "0", timeBase: { numerator: 1, denominator: 30 } },+ sourceEndPts: { ticks: "60", timeBase: { numerator: 1, denominator: 30 } },+ audioTrimStartSamples: 0,+ audioTrimEndSamples: 0,+ audioHeadSamples: 0,+ audioContentSamples: withAudio ? sampleFrames : 0,+ audioTailSamples: 0,+ },+ visual: {+ artifact: { kind: "blob", digest: digestOf(`video:${id}`), size: 10_000, mediaType: "video/mp4" },+ sourceStreamIndex: 0,+ width: 720,+ height: 1280,+ frameRate: { numerator: 30, denominator: 1 },+ frameCount: 60,+ muted: true,+ },+ ...(withAudio ? { audio: {+ artifact: { kind: "blob" as const, digest: digestOf(`audio:${id}`), size: sampleFrames * 4, mediaType: "audio/wav" },+ sourceStreamIndex: 1,+ codec: "pcm_s16le" as const,+ sampleRate: 48_000 as const,+ channels: 2 as const,+ sampleFrames,+ loudness: "preserved" as const,+ } } : {}),+ };+}++function timedLayers(id: string, withAudio = true): MediaLayerSet {+ return appendTimedMediaLayer(createMediaLayerSet(), timed(id, withAudio), fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1",+ id: "video",+ occupancy: { mode: "loop", align: "start" },+ appearance,+ }));+}++function sequenceMembers(+ activations: readonly number[] = [0, 40, 80],+ layerFactory: (index: number) => MediaLayerSet = (index) => timedLayers(`member-${index + 1}`),+) {+ let members = createMediaSequenceMemberSet();+ for (const [index, activationFrame] of activations.entries()) {+ members = appendMediaSequenceMember(members, layerFactory(index), sealMediaSequenceMemberSpec({+ contract: "svml.media-sequence-member-spec@1",+ id: `member-${index + 1}`,+ sourceAudio: { fromLayer: "video", gain: 1 - (index * 0.1) },+ }), activationFrame);+ }+ return members;+}++function sequenceSpec(overrides: Partial<MediaSequenceSpec> = {}): MediaSequenceSpec {+ return sealMediaSequenceSpec({+ contract: "svml.media-sequence-spec@1",+ id: "steps",+ presentation: itemSpec().presentation,+ motion: {+ enter: { operator: "fade", durationFrames: 12, easing: "ease-out" },+ sustain: [{ operator: "float", amount: 4, cycles: 2 }],+ exit: { operator: "slide", durationFrames: 10, easing: "ease-in", direction: "down", amount: 60 },+ },+ stackingOrder: 50,+ handoffs: [+ sealMediaHandoffSpec({+ contract: "svml.media-handoff-spec@1", id: "one-two", fromMemberId: "member-1", toMemberId: "member-2",+ operator: "crossfade", durationFrames: 20, boundaryRatio: 0.5, audio: "crossfade",+ }),+ sealMediaHandoffSpec({+ contract: "svml.media-handoff-spec@1", id: "two-three", fromMemberId: "member-2", toMemberId: "member-3",+ operator: "push", durationFrames: 20, boundaryRatio: 0.5, direction: "left", audio: "cut",+ }),+ ],+ ...overrides,+ });+}++function sequenceSounds() {+ let sounds = createMediaSoundSet();+ for (const [id, trigger] of [+ ["enter-sound", { kind: "enter" as const }],+ ["handoff-out-sound", { kind: "handoff" as const, handoffId: "one-two" }],+ ["handoff-in-sound", { kind: "handoff" as const, handoffId: "one-two" }],+ ["exit-sound", { kind: "exit" as const }],+ ] as const) {+ sounds = appendMediaSound(sounds, timed(`sfx-${id}`), sealMediaSoundSpec({+ contract: "svml.media-sound-spec@1", id, trigger, gain: 0.5,+ }));+ }+ return sounds;+}++test("an Item keeps ordered Paint/sample layers, two-frame fit and frame presentation package-owned", () => {+ const program = finalizeMediaTrack(appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, stillLayers(), frame, itemSpec(), createMediaSoundSet(),+ ), header, space);assert.deepEqual(program.items[0]?.span, { startFrame: 0, endFrameExclusive: 120 });- assert.equal(program.items[0]?.layers[0]?.artifact.digest, source.digest);- const track = renderMediaTrack(space, program);- assert.equal(track.presents.length, 1);- assert.deepEqual(track.presents[0]?.stacking, { order: 40, tieBreak: "proof:product" });- const root = track.presents[0]?.elements[0];- const image = track.presents[0]?.elements[1];- assert.equal(root?.kind, "box");- assert.deepEqual(root?.style, [- { name: "height", value: "300px" },- { name: "left", value: "100px" },- { name: "overflow", value: "hidden" },- { name: "position", value: "absolute" },- { name: "top", value: "200px" },- { name: "width", value: "400px" },+ assert.deepEqual(program.items[0]?.layers.map((layer) => layer.id), ["backing", "content"]);+ const track = projectMediaVisualTrack(space, program);+ assert.deepEqual(track.presents[0]?.stacking, { order: 40, tieBreak: "proof:product::program" });+ const elements = track.presents[0]!.elements;+ assert.deepEqual(elements.slice(0, 3).map((element) => element.id), [+ "product::program:placement", "product::program:lifecycle", "product::program:frame",]);- assert.equal(image?.kind, "image");- assert.deepEqual(image?.style, [- { name: "height", value: "300px" },- { name: "left", value: "50px" },+ const media = elements.find((element) => element.id === "content");+ assert.equal(media?.kind, "image");+ const wrapper = elements.find((element) => element.id === "content:sampling");+ assert.deepEqual(wrapper?.style, [+ { name: "height", value: "280px" },+ { name: "left", value: "60px" },{ name: "position", value: "absolute" },- { name: "top", value: "0px" },- { name: "width", value: "300px" },+ { name: "top", value: "10px" },+ { name: "transform-origin", value: "center center" },+ { name: "width", value: "280px" },+ ]);+});++test("an owned clip path is an explicit Media input and lowers only over the Item's pixels", () => {+ const clipped = bindMediaItemClipPath(itemSpec(), {+ contract: "svml.spatial-path@1",+ commands: [+ { kind: "move", xPx: 0, yPx: 0 },+ { kind: "line", xPx: 400, yPx: 0 },+ { kind: "line", xPx: 360, yPx: 300 },+ { kind: "line", xPx: 40, yPx: 300 },+ { kind: "close" },+ ],+ });+ const track = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, stillLayers(), frame, clipped, createMediaSoundSet(),+ ), header, space));+ const frameElement = track.presents[0]!.elements.find((element) => element.id.endsWith(":frame"));+ assert.equal(frameElement?.style.find((entry) => entry.name === "clip-path")?.value,+ 'path("M 0 0 L 400 0 L 360 300 L 40 300 Z")');+});++test("self-blur is two explicit samples of one Artifact and Artifact collection deduplicates bytes", () => {+ let layers = createMediaLayerSet();+ layers = appendStillMediaLayer(layers, source, extent, { ...fit, sizing: "cover" }, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "blurred", appearance: {+ opacity: 1, filter: { blurPx: 24, brightness: 0.7, contrast: 1, saturation: 0.8 },+ },+ }));+ layers = appendStillMediaLayer(layers, source, extent, fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "foreground", appearance,+ }));+ const track = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec(), createMediaSoundSet(),+ ), header, space));+ const document = compileHyperframesDocument(sealComposition({+ contract: "svml.composition@1",+ id: "media",+ canvas: { width: 1080, height: 1920, clearColor: "#000000" },+ tracks: [track],+ }), space);+ assert.equal(document.artifacts.length, 1);+ assert.equal((document.html.match(new RegExp(source.digest, "gu")) ?? []).length, 0,+ "HTML uses artifact URIs without the digest prefix spelling");+ assert.equal((document.html.match(/svml-artifact:\/\/sha256\//gu) ?? []).length, 2);+ const blurred = track.presents[0]!.elements.find((element) => element.id === "blurred");+ assert.equal(blurred?.style.find((entry) => entry.name === "left")?.value, "-48px");+ assert.equal(blurred?.style.find((entry) => entry.name === "top")?.value, "-48px");+});++test("timed layer trim/hold, lifecycle and sampling motion remain separate wrapper channels", () => {+ let layers = createMediaLayerSet();+ layers = appendTimedMediaLayer(layers, timed(), fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1",+ id: "video",+ trim: { startFrame: 10, endFrameExclusive: 40 },+ occupancy: { mode: "hold", align: "start" },+ appearance,+ samplingMotion: { keyframes: [+ { atProgress: 0, zoom: 1, offsetX: 0, offsetY: 0, rotationDeg: 0 },+ { atProgress: 0.5, zoom: 1.1, offsetX: 10, offsetY: -5, rotationDeg: 1 },+ { atProgress: 1, zoom: 1.2, offsetX: 20, offsetY: -10, rotationDeg: 2, easing: "ease-out" },+ ] },+ }));+ const program = finalizeMediaTrack(appendProgramMediaItem(createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec({+ motion: {+ enter: { operator: "slide", durationFrames: 12, easing: "ease-out", direction: "down", amount: 120 },+ sustain: [{ operator: "wobble", amount: 2, cycles: 2 }],+ exit: { operator: "fade", durationFrames: 10, easing: "ease-in" },+ },+ }), createMediaSoundSet()), header, space);+ const elements = projectMediaVisualTrack(space, program).presents[0]!.elements;+ assert.ok(elements.find((element) => element.id.endsWith(":lifecycle"))?.animation);+ assert.ok(elements.find((element) => element.id.includes(":sustain:"))?.animation);+ const samplingWrapper = elements.find((element) => element.id === "video:sampling");+ assert.ok(samplingWrapper?.animation);+ assert.deepEqual(samplingWrapper?.animation?.keyframes.map((entry) => entry.atFrame), [0, 60, 120]);+ const video = elements.find((element) => element.id === "video");+ assert.equal(video?.kind, "video");+ assert.deepEqual(video.kind === "video" ? video.sampling?.segments : undefined, [+ { target: { startFrame: 0, endFrameExclusive: 30 }, sourceFrame: { numerator: 10, denominator: 1 }, rate: { numerator: 1, denominator: 1 } },+ { target: { startFrame: 30, endFrameExclusive: 120 }, sourceFrame: { numerator: 39, denominator: 1 }, rate: { numerator: 0, denominator: 1 } },]);+ assert.throws(() => appendProgramMediaItem(createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec({+ motion: {+ enter: { operator: "fade", durationFrames: 70, easing: "linear" },+ sustain: [],+ exit: { operator: "fade", durationFrames: 60, easing: "linear" },+ },+ }), createMediaSoundSet()), /overlap/u);});-test("the first Media Fragment exposes every external dependency as a semantic input", () => {+test("source audio and edge SFX project separately from the visual Track", () => {+ let layers = createMediaLayerSet();+ layers = appendTimedMediaLayer(layers, timed("audio"), fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1",+ id: "video",+ trim: { startFrame: 10, endFrameExclusive: 40 },+ occupancy: { mode: "hold", align: "end" },+ appearance,+ }));+ let sounds = createMediaSoundSet();+ sounds = appendMediaSound(sounds, timed("sfx-enter"), sealMediaSoundSpec({+ contract: "svml.media-sound-spec@1", id: "enter-sound", trigger: { kind: "enter" }, gain: 0.5,+ }));+ sounds = appendMediaSound(sounds, timed("sfx-exit"), sealMediaSoundSpec({+ contract: "svml.media-sound-spec@1", id: "exit-sound", trigger: { kind: "exit" }, gain: 0.25,+ }));+ const program = finalizeMediaTrack(appendProgramMediaItem(createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec({+ sourceAudio: { fromLayer: "video", gain: 0.75 },+ motion: { sustain: [], exit: { operator: "fade", durationFrames: 10, easing: "ease-in" } },+ }), sounds), header, space);+ const visual = projectMediaVisualTrack(space, program);+ const audio = projectMediaAudioTrack(space, program);+ assert.equal(visual.presents.length, 1);+ assert.equal(audio.clips.length, 3);+ const sourceClip = audio.clips.find((clip) => clip.id.endsWith(":source-audio"));+ assert.deepEqual(sourceClip?.target, { startSample: 144_000, endSampleExclusive: 192_000 });+ assert.deepEqual(sourceClip?.source, {+ sampleFrames: 96_000,+ startSample: 16_000,+ endSampleExclusive: 64_000,+ loop: false,+ phaseSample: 0,+ });+ assert.equal(sourceClip?.gain, 0.75);+ assert.throws(() => projectMediaAudioTrack(space, finalizeMediaTrack(appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, stillLayers(), frame, itemSpec(), createMediaSoundSet(),+ ), header, space)), /no explicitly authored audio/u);++ const bgm: AudioTrack = {+ contract: "svml.audio-track@1",+ id: "independent-bgm",+ clips: [{+ id: "bed",+ artifact: timed("bgm").audio!.artifact,+ target: { startSample: 0, endSampleExclusive: 192_000 },+ source: {+ sampleFrames: 96_000,+ startSample: 0,+ endSampleExclusive: 96_000,+ loop: true,+ phaseSample: 0,+ },+ playbackRate: 1,+ pitch: "preserve",+ gain: 0.2,+ fadeInSamples: 0,+ fadeOutSamples: 0,+ }],+ };+ const composition = sealComposition({+ contract: "svml.composition@1",+ id: "media-with-peer-bgm",+ canvas: { width: 1080, height: 1920, clearColor: "#000000" },+ tracks: [visual, audio, bgm],+ });+ assert.deepEqual(composition.tracks.find((candidate) => candidate.id === bgm.id), bgm,+ "Media visual/audio projection must not rewrite an independent peer BGM Track");+});++test("still and animated typed Surfaces use the same layer law without browser format guesses", () => {+ const still: CompositableSurfaceRef = {+ contract: "svml.compositable-surface@1",+ artifact: { kind: "blob", digest: digestOf("surface:still"), size: 500, mediaType: "image/png" },+ width: 100,+ height: 100,+ colorSpace: "srgb",+ alphaMode: "straight",+ timing: { kind: "still" },+ };+ const animated: CompositableSurfaceRef = {+ ...still,+ artifact: { kind: "blob", digest: digestOf("surface:animated"), size: 2_000, mediaType: "video/webm" },+ timing: { kind: "frames", frameRate: { numerator: 30, denominator: 1 }, frameCount: 30 },+ };+ let layers = appendSurfaceMediaLayer(createMediaLayerSet(), still, fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "still", appearance,+ }));+ layers = appendSurfaceMediaLayer(layers, animated, fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "animated", occupancy: { mode: "loop", align: "start" }, appearance,+ }));+ const elements = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec(), createMediaSoundSet(),+ ), header, space)).presents[0]!.elements;+ assert.equal(elements.find((element) => element.id === "still")?.kind, "surface");+ const moving = elements.find((element) => element.id === "animated");+ assert.equal(moving?.kind, "surface");+ assert.ok(moving?.kind === "surface" && moving.sampling !== undefined);+ const defaulted = appendSurfaceMediaLayer(createMediaLayerSet(), animated, fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "defaulted", appearance,+ }));+ assert.deepEqual(defaulted.layers[0]?.kind === "sample" ? defaulted.layers[0].occupancy : undefined,+ { mode: "once", align: "start" });+});++test("Media Items consume the shared one/each temporal algebra without becoming an exclusive lane", () => {+ const semanticMap: CompleteSemanticMap = {+ contract: "svml.complete-semantic-map@1",+ tokens: [],+ anchors: [+ { identity: "a", timeSec: 0.5, frame: 15 },+ { identity: "b", timeSec: 1.5, frame: 45 },+ { identity: "c", timeSec: 2, frame: 60 },+ { identity: "d", timeSec: 3.5, frame: 105 },+ ],+ };+ const selection: NarrativeSelectionRef = {+ contract: "svml.narrative-selection@1",+ id: "mentions",+ occurrences: [+ { occurrence: 0, startAnchorId: "a", endAnchorId: "b" },+ { occurrence: 1, startAnchorId: "c", endAnchorId: "d" },+ ],+ };+ const repeated = appendSelectionMediaItem(createMediaTrackSet(), header, space, canvas, stillLayers(), frame,+ semanticMap, selection, itemSpec({+ id: "mention",+ expansion: { kind: "each" },+ projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },+ }), createMediaSoundSet());+ assert.deepEqual(repeated.items.map((item) => item.span), [+ { startFrame: 15, endFrameExclusive: 45 },+ { startFrame: 60, endFrameExclusive: 105 },+ ]);+ const moment: NarrativeMomentRef = {+ contract: "svml.narrative-moment@1", id: "cue", occurrences: [{ occurrence: 0, anchorId: "b" }],+ };+ const overlapping = appendMomentMediaItem(repeated, header, space, canvas, stillLayers(), frame,+ semanticMap, moment, itemSpec({+ id: "popup",+ projection: { start: { ref: "moment.cue", offset: { unit: "frames", value: -10 } }, end: { ref: "moment.cue", offset: { unit: "frames", value: 20 } } },+ stackingOrder: 41,+ }), createMediaSoundSet());+ assert.deepEqual(overlapping.items[2]?.span, { startFrame: 35, endFrameExclusive: 65 });+ const track = projectMediaVisualTrack(space, finalizeMediaTrack(overlapping, header, space));+ // Terminal Presents are canonically ordered by their resolved start frame. The+ // overlapping Moment remains an independent absolute layer; it does not turn+ // the two Selection occurrences into an exclusive lane.+ assert.deepEqual(track.presents.map((present) => present.stacking.order), [40, 41, 40]);++ const singleSelection: NarrativeSelectionRef = {+ ...selection,+ id: "single-mention",+ occurrences: [selection.occurrences[0]!],+ };+ const selectedOnce = appendSelectionMediaItem(+ createMediaTrackSet(), header, space, canvas, stillLayers(), frame,+ semanticMap, singleSelection, itemSpec({+ id: "selected-once",+ expansion: { kind: "one" },+ projection: { start: { ref: "selection.start" }, end: { ref: "selection.end" } },+ }),+ createMediaSoundSet(),+ );+ assert.deepEqual(selectedOnce.items.map((item) => item.span), [+ { startFrame: 15, endFrameExclusive: 45 },+ ]);++ const repeatedMoment: NarrativeMomentRef = {+ contract: "svml.narrative-moment@1",+ id: "cues",+ occurrences: [+ { occurrence: 0, anchorId: "b" },+ { occurrence: 1, anchorId: "d" },+ ],+ };+ const moments = appendMomentMediaItem(+ createMediaTrackSet(), header, space, canvas, stillLayers(), frame,+ semanticMap, repeatedMoment, itemSpec({+ id: "moment-each",+ expansion: { kind: "each" },+ projection: {+ start: { ref: "moment.cue" },+ end: { ref: "moment.cue", offset: { unit: "frames", value: 5 } },+ },+ }), createMediaSoundSet(),+ );+ assert.deepEqual(moments.items.map((item) => item.span), [+ { startFrame: 45, endFrameExclusive: 50 },+ { startFrame: 105, endFrameExclusive: 110 },+ ]);++ const absolute = appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, stillLayers(), frame,+ itemSpec({+ id: "absolute",+ projection: {+ start: { ref: "absolute", at: { unit: "frames", value: 7 } },+ end: { ref: "absolute", at: { unit: "frames", value: 19 } },+ },+ }), createMediaSoundSet(),+ );+ assert.deepEqual(absolute.items[0]?.span, { startFrame: 7, endFrameExclusive: 19 });+});++test("every declared lifecycle and sustain operator lowers, and outside-canvas motion resolves from explicit geometry", () => {+ for (const operator of ["fade", "slide", "scale", "pop", "bounce", "blur-reveal", "wipe", "flip", "spin"] as const) {+ const direction = ["slide", "wipe", "flip"].includes(operator) ? { direction: "up" as const } : {};+ assert.ok(lifecycleAnimation({ enter: { operator, durationFrames: 10, easing: "ease-out", ...direction }, sustain: [] }, 30));+ }+ for (const operator of ["float", "breathe", "pulse", "wobble", "shake", "drift"] as const) {+ assert.ok(sustainAnimation({ operator, amount: 2, cycles: 2, ...(operator === "drift" ? { direction: "right" as const } : {}) }, 30));+ }+ const resolved = resolveMediaLifecycleMotion({+ enter: { operator: "slide", durationFrames: 8, easing: "ease-out", direction: "up", origin: "outside-canvas" },+ sustain: [],+ exit: { operator: "slide", durationFrames: 8, easing: "ease-in", direction: "down", origin: "outside-canvas" },+ }, frame, canvas);+ assert.deepEqual(resolved.enter, { operator: "slide", durationFrames: 8, easing: "ease-out", direction: "up", amount: 500 });+ assert.deepEqual(resolved.exit, { operator: "slide", durationFrames: 8, easing: "ease-in", direction: "down", amount: 1_720 });+});++test("every documented Media frame and fit remains one ordinary Item instead of a mode", () => {+ const frames = [+ { contract: "svml.spatial-frame@1" as const, xPx: 0, yPx: 0, widthPx: 1080, heightPx: 1920 },+ { contract: "svml.spatial-frame@1" as const, xPx: 0, yPx: 0, widthPx: 540, heightPx: 1920 },+ { contract: "svml.spatial-frame@1" as const, xPx: 80, yPx: 1280, widthPx: 920, heightPx: 500 },+ { contract: "svml.spatial-frame@1" as const, xPx: 760, yPx: 80, widthPx: 260, heightPx: 360 },+ { contract: "svml.spatial-frame@1" as const, xPx: -120, yPx: 1400, widthPx: 500, heightPx: 600 },+ ];+ const sizings = ["contain", "cover", "fit-width", "fit-height", "native", "scale-down", "stretch"] as const;+ let set = createMediaTrackSet();+ for (const [index, sizing] of sizings.entries()) {+ const localFit = { ...fit, sizing,+ framePoint: { x: index % 2 === 0 ? 0.2 : 0.8, y: 0.75 },+ contentPoint: { x: 0.65, y: 0.1 },+ constraint: index % 2 === 0 ? "bounded" as const : "free" as const,+ };+ const layers = appendStillMediaLayer(createMediaLayerSet(), source,+ { contract: "svml.intrinsic-extent@1", widthPx: index % 3 === 0 ? 400 : index % 3 === 1 ? 1200 : 800,+ heightPx: index % 3 === 0 ? 1200 : index % 3 === 1 ? 400 : 800 },+ localFit, sealMediaSampleLayerSpec({ contract: "svml.media-sample-layer-spec@1", id: `sample-${index}`, appearance }));+ set = appendProgramMediaItem(set, header, space, canvas, layers, frames[index % frames.length]!, itemSpec({+ id: `fit-${sizing}`, stackingOrder: 100 + index,+ presentation: { clip: { kind: "none" }, padding: { topPx: 0, rightPx: 0, bottomPx: 0, leftPx: 0 }, shadows: [] },+ }), createMediaSoundSet());+ }+ const track = projectMediaVisualTrack(space, finalizeMediaTrack(set, header, space));+ assert.equal(track.presents.length, sizings.length);+ for (const present of track.presents) {+ const placement = present.elements[0]!;+ assert.equal(placement.kind, "box");+ assert.ok(placement.style.some((entry) => entry.name === "width" && typeof entry.value === "string"));+ assert.ok(placement.style.some((entry) => entry.name === "height" && typeof entry.value === "string"));+ }+});++test("transparent, Paint, self-blur and alternate-source backing are only ordered owned layers", () => {+ const alternate: BlobRef = { kind: "blob", digest: digestOf("media-track:alternate"), size: 2_048, mediaType: "image/webp" };+ let layers = createMediaLayerSet();+ layers = appendMediaPaintLayer(layers, sealMediaPaintLayerSpec({+ contract: "svml.media-paint-layer-spec@1", id: "solid", paint: { kind: "solid", color: "#101018" }, opacity: 1,+ }));+ layers = appendStillMediaLayer(layers, source, extent, { ...fit, sizing: "cover" }, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "self-blur", appearance: {+ opacity: 1, filter: { blurPx: 32, brightness: 0.7, contrast: 1.1, saturation: 0.8 },+ },+ }));+ layers = appendStillMediaLayer(layers, alternate, { ...extent, widthPx: 1200, heightPx: 600 }, { ...fit, sizing: "cover" },+ sealMediaSampleLayerSpec({ contract: "svml.media-sample-layer-spec@1", id: "alternate", appearance }));+ layers = appendStillMediaLayer(layers, source, extent, fit,+ sealMediaSampleLayerSpec({ contract: "svml.media-sample-layer-spec@1", id: "foreground", appearance }));+ const track = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, layers, frame, itemSpec(), createMediaSoundSet(),+ ), header, space));+ const document = compileHyperframesDocument(sealComposition({+ contract: "svml.composition@1", id: "layer-matrix", canvas: { width: 1080, height: 1920, clearColor: "#000000" }, tracks: [track],+ }), space);+ assert.deepEqual(track.presents[0]!.elements.filter((element) => element.kind === "image").map((element) => element.id),+ ["self-blur", "alternate", "foreground"]);+ assert.equal(document.artifacts.length, 2, "repeated samples share bytes while the alternate source remains explicit");++ const transparent = projectMediaVisualTrack(space, finalizeMediaTrack(appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas,+ appendStillMediaLayer(createMediaLayerSet(), source, extent, fit,+ sealMediaSampleLayerSpec({ contract: "svml.media-sample-layer-spec@1", id: "only", appearance })),+ frame, itemSpec(), createMediaSoundSet(),+ ), header, space));+ assert.equal(transparent.presents[0]!.elements.some((element) =>+ element.kind === "box" && element.style.some((entry) => entry.name === "background")), false);+});++test("visual-only normalized video never creates an implicit audio branch", () => {+ const silent = timedLayers("silent", false);+ const program = finalizeMediaTrack(appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, silent, frame, itemSpec(), createMediaSoundSet(),+ ), header, space);+ assert.equal(projectMediaVisualTrack(space, program).presents[0]!.elements.some((element) => element.kind === "video"), true);+ assert.throws(() => projectMediaAudioTrack(space, program), /no explicitly authored audio projection/u);+ assert.throws(() => appendProgramMediaItem(+ createMediaTrackSet(), header, space, canvas, silent, frame,+ itemSpec({ sourceAudio: { fromLayer: "video", gain: 1 } }), createMediaSoundSet(),+ ), /has no normalized audio/u);+});++test("timed visual occupancy resolves every alignment into exact source-frame segments", () => {+ const sampling = (occupancy: Parameters<typeof resolveVisualSampling>[0]["occupancy"]) =>+ resolveVisualSampling({ space, sourceFrameRate: { numerator: 30, denominator: 1 }, sourceFrameCount: 30, targetFrameCount: 50, occupancy });+ assert.deepEqual(sampling({ mode: "once", align: "start" }).segments.map((item) => item.target), [{ startFrame: 0, endFrameExclusive: 30 }]);+ assert.deepEqual(sampling({ mode: "once", align: "end" }).segments.map((item) => item.target), [{ startFrame: 20, endFrameExclusive: 50 }]);+ assert.deepEqual(sampling({ mode: "loop", align: "end" }).segments[0]?.sourceFrame, { numerator: 10, denominator: 1 });+ assert.deepEqual(sampling({ mode: "stretch" }).segments[0]?.rate, { numerator: 3, denominator: 5 });+ assert.throws(() => resolveVisualSampling({+ space, sourceFrameRate: { numerator: 24, denominator: 1 }, sourceFrameCount: 24, targetFrameCount: 30,+ occupancy: { mode: "once", align: "start" },+ }), /normalized to ProgramSpace frame rate/u);+});++test("the graph witness keeps every source, extent, fit, frame, time and appearance input explicit", () => {assert.deepEqual(stillMediaTrackFragment.inputs.map((input) => input.name), [- "extent", "fit", "frame", "header", "source", "space", "spec",+ "canvas", "extent", "fit", "frame", "header", "item-spec", "sample-spec", "source", "space",]);assert.deepEqual(stillMediaTrackFragment.exports[0]?.semanticInputs, [- "extent", "fit", "frame", "header", "source", "space", "spec",+ "canvas", "extent", "fit", "frame", "header", "item-spec", "sample-spec", "source", "space",]);});-test("a still Item rejects non-image bytes instead of inferring from a filename or Provider", () => {- const header = sealMediaTrackHeader({ contract: "svml.media-track-header@1", id: "proof" });- const spec = sealMediaStillItemSpec({- contract: "svml.media-still-item-spec@1", id: "bad", stackingOrder: 0,- presentation: { clip: "frame", fill: { kind: "transparent" } },+test("the Media author Surface emits explicit graph edges for layers, semantic time, Sequence topology and sound", async () => {+ const range = { source: "media-surface.svml", start: 0, end: 1 };+ const ref = (path: string): TextAttributeValue => ({ kind: "reference", path });+ const node = (name: string, attributes: Record<string, TextAttributeValue>, children: StructuredNode[] = []): StructuredElement => ({+ kind: "element", name, attributes, children, range,+ });+ const appearance = (path: string, properties: SvsRecipe["properties"]): SurfaceResolvedReference => ({+ path,+ ref: { kind: "record", id: path },+ type: svsRecipeType,+ record: { value: { kind: "inline", value: { contract: "svml.svs-recipe@1", path, properties } } } as never,+ });+ const plain = (path: string, type: SurfaceResolvedReference["type"]): SurfaceResolvedReference => ({+ path, ref: { kind: "record", id: path }, type,+ });+ const references = new Map<string, SurfaceResolvedReference>([+ ["space", plain("space", programSpaceTypes.programSpace)],+ ["canvas", plain("canvas", spatialTypes.canvas)],+ ["map", plain("map", semanticMapTypes.complete)],+ ["frame", plain("frame", spatialTypes.frame)],+ ["clip-path", plain("clip-path", spatialTypes.path)],+ ["still", plain("still", artifactTypes.blob)],+ ["extent", plain("extent", spatialTypes.extent)],+ ["video", plain("video", mediaTypes.synchronized)],+ ["surface", plain("surface", mediaTypes.compositableSurface)],+ ["sfx", plain("sfx", mediaTypes.synchronized)],+ ["selection", plain("selection", narrativeTypes.selection)],+ ["terminal", plain("terminal", narrativeTypes.selection)],+ ["cue1", plain("cue1", narrativeTypes.moment)],+ ["cue2", plain("cue2", narrativeTypes.moment)],+ ["still-style", appearance("still-style", { "stack-order": 20, fit: "contain" })],+ ["card-style", appearance("card-style", { "stack-order": 30, clip: "frame", "frame-paint": "#111111" })],+ ["video-style", appearance("video-style", { fit: "cover", playback: "hold-start", opacity: 1 })],+ ["paint-style", appearance("paint-style", { paint: "linear(90;#111111@0,#333333@1)", opacity: 1 })],+ ["sequence-style", appearance("sequence-style", { "stack-order": 40, fit: "contain", clip: "frame" })],+ ["surface-style", appearance("surface-style", { fit: "cover", playback: "loop-start" })],+ ["motion", appearance("motion", { enter: "slide", "enter-frames": 12, "enter-easing": "ease-out", "enter-direction": "up", "enter-origin": "outside-canvas" })],+ ["handoff", appearance("handoff", { operator: "crossfade", "duration-frames": 10, "boundary-ratio": 0.5, audio: "cut" })],+ ]);+ const root = node("media:Track", { id: "editorial", space: ref("space"), canvas: ref("canvas"), map: ref("map") }, [+ node("media:Item", { id: "still-card", source: ref("still"), extent: ref("extent"), frame: ref("frame"), clip: ref("clip-path"), appearance: ref("still-style"), during: "program" }),+ node("media:Item", { id: "proof", frame: ref("frame"), appearance: ref("card-style"), motion: ref("motion"), during: ref("selection"), "source-audio": "video", "audio-gain": "0.8" }, [+ node("media:Paint", { id: "backing", appearance: ref("paint-style") }),+ node("media:Layer", { id: "video", source: ref("video"), appearance: ref("video-style") }, [+ node("media:Sampling", { at: "start", zoom: "1" }),+ node("media:Sampling", { at: "end", zoom: "1.1", y: "-10", easing: "ease-out" }),+ ]),+ node("media:Sound", { id: "proof-enter", source: ref("sfx"), at: "enter", gain: "0.5" }),+ ]),+ node("media:Sequence", { id: "steps", frame: ref("frame"), appearance: ref("sequence-style"), until: ref("terminal"), "until-boundary": "end" }, [+ node("media:Member", { id: "one", source: ref("still"), extent: ref("extent"), at: ref("cue1") }),+ node("media:Member", { id: "two", source: ref("surface"), appearance: ref("surface-style"), at: ref("cue2") }),+ node("media:Handoff", { id: "one-two", from: "one", transition: ref("handoff") }),+ node("media:Sound", { id: "transition", source: ref("sfx"), handoff: "one-two" }),+ ]),+ ]);+ const result = await decodeMediaTrackSurface({+ sourceName: "media-surface.svml",+ element: root,+ resolveReference: (path) => references.get(path),+ resolveAsset: async () => { throw new Error("no asset resolution expected"); },});- assert.throws(() => appendFullStillMediaItem(- createMediaTrackSet(), header, space,- { ...source, mediaType: "video/mp4" }, extent, frame, fit, spec,- ), /image BlobArtifact/u);+ assert.equal(result.components.length, 1);+ assert.deepEqual(Object.keys(result.components[0]!.outputs).sort(), ["audio", "visual"]);+ const fragment = result.fragments[0]!;+ const producers = fragment.operations.map((entry) => entry.producer.name);+ assert.ok(producers.includes("append-still-media-layer"));+ assert.ok(producers.includes("append-timed-media-layer"));+ assert.ok(producers.includes("append-surface-media-layer"));+ assert.ok(producers.includes("append-media-paint-layer"));+ assert.ok(producers.includes("bind-media-item-clip-path"));+ assert.ok(producers.includes("append-selection-media-item"));+ assert.ok(producers.includes("append-moment-media-sequence-member"));+ assert.ok(producers.includes("append-media-sequence-until-selection-end"));+ assert.ok(producers.includes("append-media-sound"));+ assert.ok(fragment.inputs.some((entry) => entry.type.name === artifactTypes.blob.name));+ assert.ok(fragment.inputs.some((entry) => entry.type.name === mediaTypes.synchronized.name));+ assert.ok(fragment.inputs.some((entry) => entry.type.name === narrativeTypes.selection.name));+ assert.ok(fragment.inputs.some((entry) => entry.type.name === spatialTypes.path.name));+ const itemSpecs = result.records.filter((entry) => entry.type.name === "MediaItemSpec");+ assert.equal(itemSpecs.length, 2);+ const selected = itemSpecs[1]!.value.kind === "inline"+ ? itemSpecs[1]!.value.value as unknown as MediaItemSpec+ : undefined;+ assert.equal(selected?.sourceAudio?.fromLayer, "video");+ assert.ok(mediaTrackManifest.surfaces.some((surface) => surface.name === "track"));+});++test("Sequence resolves strict logical phases, expanded handoffs and one uninterrupted outer lifecycle", () => {+ const set = appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers(), frame, sequenceSpec(), sequenceSounds(), 120,+ );+ const program = finalizeMediaTrack(set, header, space);+ const sequence = program.sequences[0]!;+ assert.deepEqual(sequence.members.map((member) => member.logicalSpan), [+ { startFrame: 0, endFrameExclusive: 40 },+ { startFrame: 40, endFrameExclusive: 80 },+ { startFrame: 80, endFrameExclusive: 120 },+ ]);+ assert.deepEqual(sequence.members.map((member) => member.visualSpan), [+ { startFrame: 0, endFrameExclusive: 50 },+ { startFrame: 30, endFrameExclusive: 90 },+ { startFrame: 70, endFrameExclusive: 120 },+ ]);+ assert.deepEqual(sequence.handoffs.map((handoff) => handoff.span), [+ { startFrame: 30, endFrameExclusive: 50 },+ { startFrame: 70, endFrameExclusive: 90 },+ ]);++ const presents = projectMediaVisualTrack(space, program).presents;+ assert.equal(presents.length, 3);+ const middleLifecycle = presents[1]!.elements.find((element) => element.id.endsWith(":lifecycle"));+ assert.deepEqual(middleLifecycle?.animation?.keyframes.map((keyframe) => keyframe.style.find((item) => item.name === "opacity")?.value), [1, 1],+ "the middle member observes the neutral slice of the outer lifecycle instead of restarting entry");+ const middleSustain = presents[1]!.elements.find((element) => element.id.includes(":sustain:"));+ assert.equal(middleSustain?.animation?.keyframes.length, 5,+ "the middle member keeps only its exact boundary and group-global phase rows");+ assert.equal(middleSustain?.animation?.keyframes[1]?.atFrame, 15,+ "the middle member resumes the next group-global phase boundary instead of starting a new cycle");+ assert.notEqual(middleSustain?.animation?.keyframes[1]?.style[0]?.value, "none");+ assert.ok(presents[1]!.elements.find((element) => element.id.endsWith(":handoff"))?.animation);+ const sampled = presents[1]!.elements.find((element) => element.id === "video");+ assert.equal(sampled?.kind, "video");+ assert.deepEqual(sampled.kind === "video" ? sampled.sampling?.segments[0]?.target : undefined,+ { startFrame: 0, endFrameExclusive: 60 });+});++test("Sequence handoffs accept still, timed and alpha Surface members through one lowering path", () => {+ const alphaSurface: CompositableSurfaceRef = {+ contract: "svml.compositable-surface@1",+ artifact: { kind: "blob", digest: digestOf("surface:alpha-member"), size: 800, mediaType: "image/png" },+ width: 200, height: 300, colorSpace: "srgb", alphaMode: "straight", timing: { kind: "still" },+ };+ const surfaceLayers = () => appendSurfaceMediaLayer(createMediaLayerSet(), alphaSurface, fit, sealMediaSampleLayerSpec({+ contract: "svml.media-sample-layer-spec@1", id: "surface", appearance,+ }));+ for (const pair of [+ [stillLayers(), stillLayers()],+ [timedLayers("left"), timedLayers("right")],+ [stillLayers(), timedLayers("moving")],+ [surfaceLayers(), stillLayers()],+ ] as const) {+ let members = createMediaSequenceMemberSet();+ members = appendMediaSequenceMember(members, pair[0], sealMediaSequenceMemberSpec({ contract: "svml.media-sequence-member-spec@1", id: "a" }), 0);+ members = appendMediaSequenceMember(members, pair[1], sealMediaSequenceMemberSpec({ contract: "svml.media-sequence-member-spec@1", id: "b" }), 60);+ const spec = sequenceSpec({ motion: { sustain: [] }, handoffs: [sealMediaHandoffSpec({+ contract: "svml.media-handoff-spec@1", id: "a-b", fromMemberId: "a", toMemberId: "b",+ operator: "crossfade", durationFrames: 10, boundaryRatio: 0.5, audio: "cut",+ })] });+ const program = finalizeMediaTrack(appendMediaSequence(createMediaTrackSet(), header, space, canvas,+ members, frame, spec, createMediaSoundSet(), 120), header, space);+ assert.equal(projectMediaVisualTrack(space, program).presents.length, 2);+ }+});++test("Sequence projects source audio, visual-independent cut/crossfade, edge sounds and boundary sound", () => {+ const program = finalizeMediaTrack(appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers(), frame, sequenceSpec(), sequenceSounds(), 120,+ ), header, space);+ const track = projectMediaAudioTrack(space, program);+ const sources = track.clips.filter((clip) => clip.id.endsWith(":source-audio"));+ assert.equal(sources.length, 3);+ assert.deepEqual(sources.map((clip) => clip.target), [+ { startSample: 0, endSampleExclusive: 80_000 },+ { startSample: 48_000, endSampleExclusive: 128_000 },+ { startSample: 128_000, endSampleExclusive: 192_000 },+ ]);+ assert.deepEqual(sources.map((clip) => [clip.fadeInSamples, clip.fadeOutSamples]), [+ [0, 32_000], [32_000, 0], [0, 0],+ ]);+ assert.deepEqual(Object.fromEntries(track.clips.filter((clip) => !clip.id.endsWith(":source-audio"))+ .map((clip) => [clip.id, clip.target.startSample])), {+ "steps:enter-sound": 0,+ "steps:handoff-out-sound": 64_000,+ "steps:handoff-in-sound": 64_000,+ "steps:exit-sound": 176_000,+ });+});++test("every Sequence handoff operator and boundary ratio lowers without changing the authored boundary", () => {+ const operators: readonly MediaHandoffOperator[] = ["cut", "crossfade", "push", "wipe", "cover", "page-turn"];+ for (const operator of operators) {+ for (const boundaryRatio of [0, 0.5, 1]) {+ const durationFrames = operator === "cut" ? 0 : 20;+ const handoff = sealMediaHandoffSpec({+ contract: "svml.media-handoff-spec@1",+ id: `${operator}-${boundaryRatio}`,+ fromMemberId: "member-1",+ toMemberId: "member-2",+ operator,+ durationFrames,+ boundaryRatio,+ ...(["push", "wipe", "cover", "page-turn"].includes(operator) ? { direction: "right" as const } : {}),+ audio: "cut",+ });+ const spec = sequenceSpec({+ handoffs: [handoff],+ motion: { sustain: [] },+ });+ const program = finalizeMediaTrack(appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers([0, 60]), frame, spec, createMediaSoundSet(), 120,+ ), header, space);+ const resolved = program.sequences[0]!.handoffs[0]!;+ assert.equal(resolved.span.endFrameExclusive - resolved.span.startFrame, durationFrames);+ assert.equal(program.sequences[0]!.members[1]!.activationFrame, 60);+ assert.equal(projectMediaVisualTrack(space, program).presents.length, 2);+ }+ }++ for (const durationFrames of [20, 60, 80]) {+ const spec = sequenceSpec({+ motion: { sustain: [] },+ handoffs: [sealMediaHandoffSpec({+ contract: "svml.media-handoff-spec@1",+ id: `duration-${durationFrames}`,+ fromMemberId: "member-1",+ toMemberId: "member-2",+ operator: "crossfade",+ durationFrames,+ boundaryRatio: 0.5,+ audio: "crossfade",+ })],+ });+ const sequence = finalizeMediaTrack(appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers([0, 60]), frame,+ spec, createMediaSoundSet(), 120,+ ), header, space).sequences[0]!;+ assert.equal(sequence.handoffs[0]!.span.endFrameExclusive+ - sequence.handoffs[0]!.span.startFrame, durationFrames);+ }+});++test("Sequence fails atomically on malformed order, topology, envelope and three-member overlap", () => {+ assert.throws(() => appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers(), frame,+ sequenceSpec({ handoffs: sequenceSpec().handoffs.slice(0, 1) }), createMediaSoundSet(), 120,+ ), /exactly one Handoff/u);+ assert.throws(() => appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers([0, 40, 40]), frame, sequenceSpec(), createMediaSoundSet(), 120,+ ), /strictly increasing/u);+ assert.throws(() => appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers([0, 80, 40]), frame, sequenceSpec(), createMediaSoundSet(), 120,+ ), /authored order/u);+ assert.throws(() => appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers([0, 40, 70]), frame,+ sequenceSpec({ handoffs: [+ sealMediaHandoffSpec({ contract: "svml.media-handoff-spec@1", id: "wide-one", fromMemberId: "member-1", toMemberId: "member-2", operator: "crossfade", durationFrames: 40, boundaryRatio: 0.5, audio: "cut" }),+ sealMediaHandoffSpec({ contract: "svml.media-handoff-spec@1", id: "wide-two", fromMemberId: "member-2", toMemberId: "member-3", operator: "crossfade", durationFrames: 40, boundaryRatio: 0.5, audio: "cut" }),+ ] }), createMediaSoundSet(), 120,+ ), /overlapping Handoffs/u);+ assert.throws(() => appendMediaSequence(+ createMediaTrackSet(), header, space, canvas, sequenceMembers([0, 20]), frame,+ sequenceSpec({ handoffs: [+ sealMediaHandoffSpec({ contract: "svml.media-handoff-spec@1", id: "outside", fromMemberId: "member-1", toMemberId: "member-2", operator: "wipe", durationFrames: 60, boundaryRatio: 1, direction: "left", audio: "cut" }),+ ] }), createMediaSoundSet(), 120,+ ), /envelope/u);});diff --git a/packages/provider-hyperframes-aws-lambda/package.json b/packages/provider-hyperframes-aws-lambda/package.jsonindex d461e3ee..83f51aab 100644--- a/packages/provider-hyperframes-aws-lambda/package.json+++ b/packages/provider-hyperframes-aws-lambda/package.json@@ -15,7 +15,7 @@"dependencies": {"@aws-sdk/client-s3": "^3.700.0","@aws-sdk/client-sfn": "^3.700.0",- "@hyperframes/aws-lambda": "0.7.84",+ "@hyperframes/aws-lambda": "0.7.101","@narratage/endpoint-kit": "workspace:*","@narratage/hyperframes": "workspace:*","@narratage/media": "workspace:*",diff --git a/packages/provider-hyperframes-aws-lambda/src/provider.ts b/packages/provider-hyperframes-aws-lambda/src/provider.tsindex 21de7add..dfef8851 100644--- a/packages/provider-hyperframes-aws-lambda/src/provider.ts+++ b/packages/provider-hyperframes-aws-lambda/src/provider.ts@@ -48,7 +48,7 @@ import type {HyperframesLambdaSite,} from "./client.js";-const HYPERFRAMES_VERSION = "0.7.84";+const HYPERFRAMES_VERSION = "0.7.101";const REQUEST_CONTRACT = "svml.hyperframes-visual-render-request@1";const CHECKPOINT_CONTRACT = "svml.hyperframes-aws-lambda-operation@1";const SUPPORTED_FPS = new Set([24, 30, 60]);diff --git a/packages/provider-hyperframes-local/package.json b/packages/provider-hyperframes-local/package.jsonindex 3e27a488..045ce729 100644--- a/packages/provider-hyperframes-local/package.json+++ b/packages/provider-hyperframes-local/package.json@@ -17,7 +17,7 @@"@narratage/render-hyperframes": "workspace:*","@narratage/runtime-adapter": "workspace:*","@narratage/runtime-adapter-node": "workspace:*",- "hyperframes": "0.7.84"+ "hyperframes": "0.7.101"},"devDependencies": {"@narratage/composition": "workspace:*",diff --git a/packages/provider-hyperframes-local/src/provider.ts b/packages/provider-hyperframes-local/src/provider.tsindex 42959609..d28d86c3 100644--- a/packages/provider-hyperframes-local/src/provider.ts+++ b/packages/provider-hyperframes-local/src/provider.ts@@ -13,7 +13,7 @@ import { canonicalize, digestOf } from "@narratage/protocol";import type { BlobRef, CanonicalValue } from "@narratage/protocol";import { defineEndpointPackage } from "@narratage/endpoint-kit";-const HYPERFRAMES_VERSION = "0.7.84";+const HYPERFRAMES_VERSION = "0.7.101";export const localHyperframesProviderModuleRef = {name: "@narratage/provider-hyperframes-local",diff --git a/packages/provider-media-local/test/provider.test.ts b/packages/provider-media-local/test/provider.test.tsindex 96d51fed..d4ab9f6b 100644--- a/packages/provider-media-local/test/provider.test.ts+++ b/packages/provider-media-local/test/provider.test.ts@@ -28,6 +28,12 @@ import { createLocalMediaProvider } from "../src/index.js";const hasMediaBinaries = spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0&& spawnSync("ffprobe", ["-version"], { stdio: "ignore" }).status === 0;+// Five 64x48 lossless frames at 100 ms each. Later frames are partial alpha+// rectangles, so this fixture exercises WebP blend and canvas persistence.+const animatedWebp = Buffer.from([+ "UklGRjQEAABXRUJQVlA4WAoAAAASAAAAPwAALwAAQU5JTQYAAAAAbPz/AABBTk1GpgIAAAAAAAAAAD8AAC8AAGQAAAJWUDhMjQIAAC8/wAsAp8EQkiRJeQ9KwxzeARbKgjBq20iQd/egFMzCOSgH58AUyjwIBCEy0kojzX8gd0AYXnAa/lMIRRkGIQiDELwMG5RBcaEUJQhCDyEIwkotw3l4P34IQlA+o4RdINm27bZt6KLGC9aIhJAnMGqge/f/fxzee1C06Iwyiei/A7dtI6nGCh0PvOk1g3nD02opOIu6Seg2ofsrKS4wj3qYs66vH6+l1G/59+0qqXlC1wktVyYwAaBPqCorqTI7pYgRASOzP2iulqdwCiEYGHS9ZlFpnpIKo+bxoJmAiej6XvoxX3C/qs4gqG6I+WMS5nQpMgWEUfqq30sg0I72QFk473xR9L1MtLXzzhtjgttxAYyCAaPzzpd35Z3zq2UI5mWXg3Z59+kauK9Sox42v7B9NlkcFxfQ9/Hq17l7LrtPv57NALIADBFZThD1XetdU1Vk2SYiy/eJEPI3D4xk+T/FOA+zmYElA85Z4gTnGj+0RWUp2hmPhwOWwgi/y8NoiROWuIBEvnMgS7Yeftdk9bkj23hveDx44QL63PH4hlqOcyGA7S63ZKn+HmpL2rbUuD9iPYznAtq2VH981WKcH0nbKttGu6rqhG2VbUVf20eyl9j1R8I+KfsUAby+a/uQsvFKeb73KL9c0/jPItqxXFW2g1vDD8a8OHABjHjzHHjj8X3dlc+uYQAb2hHyrnS0cUUZ0dYRdbuh3TYzmdmJAmFEDoQR8G7H4yu9EzDA+edOhItLnjvBo7SZFzx32UkqU8891HPfy2dPAMB0752FwnTvHW1P9d5J2VO9dy6wKxk+B/3eSduzye3/gOdOabrvNm6LSZvGdN+dE178mKYTix8NHeCE8H/UxT9/7q4BAEFOTUZSAAAAAAAAAAAABwAACQAAZAAAAFZQOEw6AAAALwdAAhA/EIraSIFWzCkFgchDCg+BQJK/ypDPf4C3zAiyOaCQkSTmFA7mCd6k+0M0hIj+JwfeYjBtNEFOTUZOAAAAAAAAAQAACQAABwAAZAAAAFZQOEw2AAAALwnAARA3MGxq2jZgCrD8NaH0mKr5D8A1EMgogKBt25jCxuUIqvE4fxolEdH/GGRxg/T6q+kDQU5NRl4AAAABAAABAAAJAAAJAABkAAAAVlA4TEYAAAAvCUACEEcgFkzmfwUYmEK2EaCDmT/PfTBTNf9BQOA5mQ8F4KiRJEeKdUCq1r0dkMV01I9ERP+jbSWJ4gre4grOQv78FnwHQU5NRkQAAAACAAACAAAHAAAJAABkAAAAVlA4TCwAAAAvB0ACECcQAoFkmPyJhpmq+Y8gbqCQjSRoGNZgzuDz5zuEiP5nKqXKvxvuAA==",+].join(""), "base64");+async function run(executable: string, args: readonly string[]): Promise<void> {await new Promise<void>((resolve, reject) => {const child = spawn(executable, [...args], { stdio: ["ignore", "ignore", "pipe"] });@@ -253,6 +259,68 @@ test("local media Provider enumerates attached pictures and jointly normalizes 3}});+test("animated WebP keeps its authored frame timing before fixed-rate normalization", {+ skip: !hasMediaBinaries,+}, async () => {+ const artifacts = new MemoryArtifactStore();+ const source = await artifacts.put(animatedWebp, "image/webp");+ const inspection = await inspectArtifact(artifacts, source);+ assert.deepEqual(inspection.container.formatNames, ["webp", "webp-animation"]);+ assert.deepEqual(inspection.streams.map((stream) => [stream.kind, stream.kind === "video" ? stream.role : undefined,+ stream.decodedUnitCount]), [["video", "moving", 5]]);+ const video = inspection.streams[0];+ assert.equal(video?.kind, "video");+ assert.equal(video?.startPts?.ticks, "0");+ assert.equal(video?.endPts?.ticks, "500");+ const request = sealMediaSelectionRequest({+ contract: "svml.media-selection-request@1",+ video: { mode: "primary-moving" },+ audio: { mode: "none" },+ spanAuthority: "video",+ frameRate: { numerator: 20, denominator: 1 },+ });+ const selection = selectMediaStreams(inspection, request);+ const normalized = await normalizeArtifact({ artifacts, source, inspection, selection, frameRate: request.frameRate });+ assert.equal(normalized.timeline.frameCount, 10);+ assert.equal(normalized.visual?.frameCount, 10);+ assert.equal(normalized.visual?.width, 64);+ assert.equal(normalized.visual?.height, 48);+ const output = await inspectArtifact(artifacts, normalized.visual!.artifact);+ assert.equal(output.streams[0]?.decodedUnitCount, 10);+});++test("animated GIF keeps its authored frame timing before fixed-rate normalization", {+ skip: !hasMediaBinaries,+}, async () => {+ const root = await mkdtemp(join(tmpdir(), "svml-provider-media-gif-"));+ try {+ const path = join(root, "animated.gif");+ await run("ffmpeg", [+ "-v", "error", "-y", "-f", "lavfi", "-i", "testsrc2=s=64x48:r=10:d=0.5",+ "-frames:v", "5", path,+ ]);+ const artifacts = new MemoryArtifactStore();+ const source = await artifacts.put(await readFile(path), "image/gif");+ const inspection = await inspectArtifact(artifacts, source);+ assert.deepEqual(inspection.streams.map((stream) => [stream.kind, stream.kind === "video" ? stream.role : undefined,+ stream.decodedUnitCount]), [["video", "moving", 5]]);+ const request = sealMediaSelectionRequest({+ contract: "svml.media-selection-request@1",+ video: { mode: "primary-moving" },+ audio: { mode: "none" },+ spanAuthority: "video",+ frameRate: { numerator: 20, denominator: 1 },+ });+ const selection = selectMediaStreams(inspection, request);+ const normalized = await normalizeArtifact({ artifacts, source, inspection, selection, frameRate: request.frameRate });+ assert.equal(normalized.timeline.frameCount, 10);+ const output = await inspectArtifact(artifacts, normalized.visual!.artifact);+ assert.equal(output.streams[0]?.decodedUnitCount, 10);+ } finally {+ await rm(root, { recursive: true, force: true });+ }+});+test("local media Provider derives one exact 16 kHz mono WhisperX evidence artifact without a hidden second transcode", {skip: !hasMediaBinaries,}, async () => {diff --git a/packages/spatial/src/activation.ts b/packages/spatial/src/activation.tsindex 69d69fba..83d5a691 100644--- a/packages/spatial/src/activation.ts+++ b/packages/spatial/src/activation.ts@@ -4,6 +4,7 @@ import {decodeAnchoredFrameSurface,decodeAspectFrameSurface,decodeCanvasSurface,+ decodeExtentSurface,decodeFrameSurface,} from "./surface.js";import {@@ -20,6 +21,7 @@ export const svmlPackage = {components: [spatialComponent],hostFacets: [createTextSurfaceHostFacet({ module: spatialModuleRef, surface: "canvas", mode: "structured", implementationDigest: spatialSurfaceDigests.canvas, handler: decodeCanvasSurface }),+ createTextSurfaceHostFacet({ module: spatialModuleRef, surface: "extent", mode: "structured", implementationDigest: spatialSurfaceDigests.extent, handler: decodeExtentSurface }),createTextSurfaceHostFacet({ module: spatialModuleRef, surface: "frame", mode: "structured", implementationDigest: spatialSurfaceDigests.frame, handler: decodeFrameSurface }),createTextSurfaceHostFacet({ module: spatialModuleRef, surface: "anchored-frame", mode: "structured", implementationDigest: spatialSurfaceDigests.anchoredFrame, handler: decodeAnchoredFrameSurface }),createTextSurfaceHostFacet({ module: spatialModuleRef, surface: "aspect-frame", mode: "structured", implementationDigest: spatialSurfaceDigests.aspectFrame, handler: decodeAspectFrameSurface }),diff --git a/packages/spatial/src/index.ts b/packages/spatial/src/index.tsindex ac3c90ab..7b33f503 100644--- a/packages/spatial/src/index.ts+++ b/packages/spatial/src/index.ts@@ -21,6 +21,7 @@ export {decodeAnchoredFrameSurface,decodeAspectFrameSurface,decodeCanvasSurface,+ decodeExtentSurface,decodeFrameSurface,} from "./surface.js";export type * from "./types.js";diff --git a/packages/spatial/src/manifest.ts b/packages/spatial/src/manifest.tsindex 8af8465b..eab3615f 100644--- a/packages/spatial/src/manifest.ts+++ b/packages/spatial/src/manifest.ts@@ -18,6 +18,7 @@ import {export const spatialModuleRef = { name: "@narratage/spatial", version: "0.0.0-dev" } as const;export const spatialSurfaceDigests = {canvas: digestOf("@narratage/spatial/canvas-surface@1"),+ extent: digestOf("@narratage/spatial/extent-surface@1"),frame: digestOf("@narratage/spatial/frame-surface@1"),anchoredFrame: digestOf("@narratage/spatial/anchored-frame-surface@1"),aspectFrame: digestOf("@narratage/spatial/aspect-frame-surface@1"),@@ -67,6 +68,7 @@ export const spatialManifest: ModuleManifest = {capabilities: [],surfaces: [{ name: "canvas", tag: "Canvas", mode: "structured", outputs: [spatialTypes.canvas], implementation: { kind: "trusted-frontend-surface", locator: "@narratage/spatial/canvas-surface", digest: spatialSurfaceDigests.canvas } },+ { name: "extent", tag: "Extent", mode: "structured", outputs: [spatialTypes.extent], implementation: { kind: "trusted-frontend-surface", locator: "@narratage/spatial/extent-surface", digest: spatialSurfaceDigests.extent } },{ name: "frame", tag: "Frame", mode: "structured", outputs: [spatialTypes.frame, spatialTypes.frameEdgesProgram], implementation: { kind: "trusted-frontend-surface", locator: "@narratage/spatial/frame-surface", digest: spatialSurfaceDigests.frame } },{ name: "anchored-frame", tag: "AnchoredFrame", mode: "structured", outputs: [spatialTypes.frame, spatialTypes.anchoredFrameProgram], implementation: { kind: "trusted-frontend-surface", locator: "@narratage/spatial/anchored-frame-surface", digest: spatialSurfaceDigests.anchoredFrame } },{ name: "aspect-frame", tag: "AspectFrame", mode: "structured", outputs: [spatialTypes.frame, spatialTypes.extent, spatialTypes.aspectFrameProgram], implementation: { kind: "trusted-frontend-surface", locator: "@narratage/spatial/aspect-frame-surface", digest: spatialSurfaceDigests.aspectFrame } },diff --git a/packages/spatial/src/surface.ts b/packages/spatial/src/surface.tsindex e7b0f909..0888c161 100644--- a/packages/spatial/src/surface.ts+++ b/packages/spatial/src/surface.ts@@ -122,6 +122,17 @@ export const decodeCanvasSurface: StructuredSurfaceHandler = ({ element }) => {return { records: [{ id, type: spatialTypes.canvas, value: { kind: "inline", value: canvas as unknown as CanonicalValue }, range: element.range }], components: [], fragments: [] };};+export const decodeExtentSurface: StructuredSurfaceHandler = ({ element }) => {+ exact(element, ["id", "width", "height"], ["id", "width", "height"]);+ const id = text(element, "id");+ const extent = sealIntrinsicExtent({+ contract: "svml.intrinsic-extent@1",+ widthPx: positiveInteger(element, "width"),+ heightPx: positiveInteger(element, "height"),+ });+ return { records: [{ id, type: spatialTypes.extent, value: { kind: "inline", value: extent as unknown as CanonicalValue }, range: element.range }], components: [], fragments: [] };+};+function frameSurface(element: StructuredElement,resolveReference: (path: string) => SurfaceResolvedReference | undefined,diff --git a/packages/speech-alignment/test/speech-align.test.ts b/packages/speech-alignment/test/speech-align.test.tsindex 0e5fde25..cc132c47 100644--- a/packages/speech-alignment/test/speech-align.test.ts+++ b/packages/speech-alignment/test/speech-align.test.ts@@ -80,13 +80,14 @@ function speechBasis(clips: segments.map((segment) => ({segmentId: segment.segmentId,artifact: {+ kind: "blob",digest: digestOf(`fixture:clip:${segment.segmentId}`),size: 1,mediaType: "video/mp4",- durationSec: segment.endSec - segment.startSec,},- startSec: segment.startSec,- endSec: segment.endSec,+ extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },+ frameRate: { ...programSpace.frameRate },+ frameCount: Math.round((segment.endSec - segment.startSec) * 1_000),})),},segments,diff --git a/packages/speech-basis/package.json b/packages/speech-basis/package.jsonindex 51c78614..ff66222d 100644--- a/packages/speech-basis/package.json+++ b/packages/speech-basis/package.json@@ -13,9 +13,11 @@"@narratage/component-kit": "workspace:*","@narratage/composition": "workspace:*","@narratage/elaborator": "workspace:*",+ "@narratage/media-track": "workspace:*","@narratage/program-space": "workspace:*","@narratage/protocol": "workspace:*",- "@narratage/speech": "workspace:*"+ "@narratage/speech": "workspace:*",+ "@narratage/spatial": "workspace:*"},"devDependencies": {"@narratage/core": "workspace:*",diff --git a/packages/speech-basis/src/component.ts b/packages/speech-basis/src/component.tsindex 17464667..0d8712af 100644--- a/packages/speech-basis/src/component.ts+++ b/packages/speech-basis/src/component.ts@@ -2,6 +2,7 @@ import type { ComponentPackage } from "@narratage/component-kit";import type { SpeechBasis } from "@narratage/speech";import type { StoredValue } from "@narratage/protocol";import { canonicalize } from "@narratage/protocol";+import type { CanvasSpace } from "@narratage/spatial";import {speechBasisProducers,@@ -22,6 +23,11 @@ function speechBasis(value: StoredValue | undefined): SpeechBasis {return value.value as unknown as SpeechBasis;}+function canvas(value: StoredValue | undefined): CanvasSpace {+ if (value?.kind !== "inline") throw new Error("CanvasSpace must be inline");+ return value.value as unknown as CanvasSpace;+}+/** Pure projections of one atomic SpeechBasis; no Artifact read or external capability is hidden here. */export const speechBasisComponent = {name: "@narratage/speech-basis",@@ -59,7 +65,7 @@ export const speechBasisComponent = {outputs: {visual: {kind: "inline",- value: canonicalize(projectSpeechVisual(speechBasis(inputs.basis?.value))),+ value: canonicalize(projectSpeechVisual(speechBasis(inputs.basis?.value), canvas(inputs.canvas?.value))),},},needs: {},diff --git a/packages/speech-basis/src/fragment.ts b/packages/speech-basis/src/fragment.tsindex 582b9848..d1c75968 100644--- a/packages/speech-basis/src/fragment.ts+++ b/packages/speech-basis/src/fragment.ts@@ -3,6 +3,7 @@ import { speechTypes } from "@narratage/speech";import type { SpeechBasis } from "@narratage/speech";import { compositionTypes } from "@narratage/composition";import { sealGraphFragment } from "@narratage/elaborator";+import { spatialTypes } from "@narratage/spatial";import { speechBasisProducers } from "./manifest.js";@@ -12,7 +13,7 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operat/** Project the independently demandable facts carried by one SpeechBasis. */export const speechBasisProjectionFragment = sealGraphFragment({name: "@narratage/speech-basis/projections@1",- inputs: [{ name: "basis", type: speechTypes.basis }],+ inputs: [{ name: "basis", type: speechTypes.basis }, { name: "canvas", type: spatialTypes.canvas }],operations: [{id: "project-program-space",@@ -29,7 +30,7 @@ export const speechBasisProjectionFragment = sealGraphFragment({{id: "project-visual",producer: speechBasisProducers.projectVisual,- inputs: { basis: input("basis") },+ inputs: { basis: input("basis"), canvas: input("canvas") },result: { kind: "output", name: "visual" },},{@@ -58,7 +59,7 @@ export const speechBasisProjectionFragment = sealGraphFragment({name: "visual",type: compositionTypes.visualTrack,root: operation("project-visual"),- semanticInputs: ["basis"],+ semanticInputs: ["basis", "canvas"],fidelity: "exact",},{diff --git a/packages/speech-basis/src/manifest.ts b/packages/speech-basis/src/manifest.tsindex 3282450a..e5832adb 100644--- a/packages/speech-basis/src/manifest.ts+++ b/packages/speech-basis/src/manifest.ts@@ -1,6 +1,7 @@import { programSpaceDependency, programSpaceTypes } from "@narratage/program-space";import { speechDependency, speechTypes } from "@narratage/speech";import { compositionDependency, compositionTypes } from "@narratage/composition";+import { spatialDependency, spatialTypes } from "@narratage/spatial";import { digestOf } from "@narratage/protocol";import type { ModuleManifest, ProducerRef } from "@narratage/protocol";@@ -28,6 +29,7 @@ export const speechBasisManifest: ModuleManifest = {speechDependency,programSpaceDependency,compositionDependency,+ spatialDependency,],types: [],capabilities: [],@@ -57,7 +59,7 @@ export const speechBasisManifest: ModuleManifest = {},{name: speechBasisProducers.projectVisual.name,- inputs: [{ name: "basis", type: speechTypes.basis }],+ inputs: [{ name: "basis", type: speechTypes.basis }, { name: "canvas", type: spatialTypes.canvas }],outputs: [{ name: "visual", type: compositionTypes.visualTrack }],needs: [],implementation: {diff --git a/packages/speech-basis/src/projection.ts b/packages/speech-basis/src/projection.tsindex 444efaab..d297738b 100644--- a/packages/speech-basis/src/projection.ts+++ b/packages/speech-basis/src/projection.ts@@ -2,9 +2,11 @@ import { programSpaceFrameCount, programSpaceSampleFrames } from "@narratage/proimport type { ProgramSpace } from "@narratage/program-space";import { assertSpeechBasisIdentity } from "@narratage/speech";import type { SpeechAudioBasis, SpeechBasis } from "@narratage/speech";+import { lowerRestrictedSpeechVisualPresents } from "@narratage/media-track";import { sealAudioTrack, sealVisualTrack } from "@narratage/composition";-import type { AudioTrack, VisualElement, VisualTrack } from "@narratage/composition";+import type { AudioTrack, VisualTrack } from "@narratage/composition";import { digestOf } from "@narratage/protocol";+import type { CanvasSpace } from "@narratage/spatial";export const projectSpeechAudioImplementationDigest = digestOf("@narratage/speech-basis/project-audio@1",@@ -41,41 +43,38 @@ function frameAt(basis: SpeechBasis, seconds: number): number {return Math.round(seconds * basis.programSpace.frameRate.numerator / basis.programSpace.frameRate.denominator);}-export function projectSpeechVisual(basis: SpeechBasis): VisualTrack {+export function projectSpeechVisual(basis: SpeechBasis, canvas: CanvasSpace): VisualTrack {assertSpeechBasisIdentity(basis);- const totalFrames = programSpaceFrameCount(basis.programSpace);const trackId = `speech-visual:${basis.segments.map((segment) => segment.segmentId).join("+")}`;+ const segments = new Map(basis.segments.map((segment) => [segment.segmentId, segment]));return sealVisualTrack({contract: "svml.visual-track@1",visualIr: "svml.visual-ir@1",id: trackId,- presents: basis.visualTrack.clips.map((clip, index) => {- const startFrame = frameAt(basis, clip.startSec);- const endFrameExclusive = Math.min(totalFrames, frameAt(basis, clip.endSec));- const kind: VisualElement["kind"] = clip.artifact.mediaType.startsWith("image/") ? "image" : "video";- if (kind !== "image" && !clip.artifact.mediaType.startsWith("video/")) {- throw new Error(`Speech visual ${clip.segmentId} is not an image or video Artifact.`);- }- if (endFrameExclusive <= startFrame) throw new Error(`Speech visual ${clip.segmentId} has an empty frame span.`);- return {- id: `${clip.segmentId}:${index + 1}`,- span: { startFrame, endFrameExclusive },- stacking: { order: 0, tieBreak: `${trackId}:${clip.segmentId}:${index + 1}` },- elements: [{- id: "media",- order: 0,- kind,+ presents: lowerRestrictedSpeechVisualPresents(+ trackId,+ basis.programSpace,+ canvas,+ basis.visualTrack.clips.map((clip) => {+ const segment = segments.get(clip.segmentId)!;+ return {+ id: clip.segmentId,+ span: { startFrame: frameAt(basis, segment.startSec), endFrameExclusive: frameAt(basis, segment.endSec) },artifact: clip.artifact,- style: [- { name: "height", value: "100%" },- { name: "object-fit", value: "cover" },- { name: "position", value: "absolute" },- { name: "width", value: "100%" },- ],- ...(kind === "video" ? { muted: true } : {}),- }],- };- }),+ extent: clip.extent,+ frameRate: clip.frameRate,+ frameCount: clip.frameCount,+ };+ }),+ {+ contract: "svml.content-fit@1",+ sizing: "cover",+ framePoint: { x: 0.5, y: 0.5 },+ contentPoint: { x: 0.5, y: 0.5 },+ offsetPx: { x: 0, y: 0 },+ constraint: "bounded",+ },+ ),});}diff --git a/packages/speech-basis/test/product.test.ts b/packages/speech-basis/test/product.test.tsindex 4b216718..af527d55 100644--- a/packages/speech-basis/test/product.test.ts+++ b/packages/speech-basis/test/product.test.ts@@ -1,4 +1,4 @@-import { compositionComponent, videoContractManifests } from "../../test-support/video-domain.js";+import { compositionComponent, spatialComponent, videoContractManifests } from "../../test-support/video-domain.js";import {registerProducerFacets,registerTypeValidatorFacets,@@ -41,6 +41,19 @@ import {speechBasisProducers,} from "@narratage/speech-basis";import { TypeValidatorRegistry } from "@narratage/validation";+import { spatialTypes } from "@narratage/spatial";+import { spatialValidatorDigests } from "@narratage/spatial";+import { mediaTrackManifest } from "@narratage/media-track";++const canvas = {+ contract: "svml.canvas-space@1" as const,+ widthPx: 720,+ heightPx: 1280,+ origin: "top-left" as const,+ xDirection: "right" as const,+ yDirection: "down" as const,+ pixelAspect: "square" as const,+};const testModule = { name: "example.speech-basis-product", version: "0.0.0" } as const;const requestType = { module: testModule, name: "SpeechRequest" } satisfies TypeRef;@@ -68,11 +81,12 @@ const testManifest: ModuleManifest = {}],};-const closure = createResolvedClosure([...videoContractManifests, speechBasisManifest, testManifest]);+const closure = createResolvedClosure([...videoContractManifests, mediaTrackManifest, speechBasisManifest, testManifest]);function validatorRegistry(): TypeValidatorRegistry {const registry = new TypeValidatorRegistry();registerTypeValidatorFacets(registry, compositionComponent.validators);+ registerTypeValidatorFacets(registry, spatialComponent.validators);return registry;}@@ -90,17 +104,23 @@ function sampleTake(label = "generated"): SpeechBasis {mediaType: "audio/wav",};const visual = {+ kind: "blob" as const,digest: digestOf(`${label}:visual`),size: 24,mediaType: "video/mp4",- durationSec,};return sealSpeechBasis({contract: "svml.speech-basis@1",programSpace,audio,visualTrack: {- clips: [{ segmentId: "opening", artifact: visual, startSec: 0, endSec: durationSec }],+ clips: [{+ segmentId: "opening",+ artifact: visual,+ extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },+ frameRate: { ...programSpace.frameRate },+ frameCount: 60,+ }],},segments: [{segmentId: "opening",@@ -122,10 +142,29 @@ function createProgram(): LinkedProgram {frontendClosureDigest: digestOf("frontend:speech-basis-product"),},});+ const rawCanvasRecord = sealRecord({+ id: "canvas:vertical",+ type: spatialTypes.canvas,+ value: { kind: "inline", value: canvas },+ conformance: "exact",+ origin: {+ kind: "authored",+ sourceDigest: digestOf("source:speech-basis-product"),+ frontendClosureDigest: digestOf("frontend:speech-basis-product"),+ },+ });+ const canvasRecord = {+ ...rawCanvasRecord,+ validation: sealTypeValidationReceipt({+ type: rawCanvasRecord.type,+ recordDigest: rawCanvasRecord.digest,+ validatorDigest: spatialValidatorDigests.canvas,+ }),+ };return link(closure, [sealTypedModule({id: "author:speech-basis-product",closureDigest: closure.digest,- records: [request],+ records: [request, canvasRecord],})]);}@@ -134,7 +173,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {const output = (id: string) => ({ kind: "logical-output" as const, id });const operation = (id: string) => ({ kind: "operation-result" as const, operation: id });const existingTake = sampleTake("approved");- const existingVisual = projectSpeechVisual(existingTake);+ const existingVisual = projectSpeechVisual(existingTake, canvas);const existingVisualValue = { kind: "inline" as const, value: existingVisual };const existingVisualValidation = sealTypeValidationReceipt({type: compositionTypes.visualTrack,@@ -160,7 +199,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {id: "opening.visual",type: compositionTypes.visualTrack,primary: "opening.visual.project",- semanticInputs: [output("opening.take")],+ semanticInputs: [output("opening.take"), record("canvas:vertical")],},],candidates: [@@ -221,7 +260,7 @@ function createGraph(program: LinkedProgram): CompiledGraph {{id: "project-opening-visual",producer: speechBasisProducers.projectVisual,- inputs: { basis: output("opening.take") },+ inputs: { basis: output("opening.take"), canvas: record("canvas:vertical") },result: { kind: "output", name: "visual", record: "visual:opening" },},],@@ -270,7 +309,7 @@ test("SpeechBasis is one Product and audio/visual are ordinary shared projectiontest("SpeechBasis projects to peer generic visual and audio Tracks", () => {const take = sampleTake();- const visual = projectSpeechVisual(take);+ const visual = projectSpeechVisual(take, canvas);const audio = projectSpeechAudioTrack(take);const programSpace = projectSpeechProgramSpace(take);assert.equal(visual.contract, "svml.visual-track@1");diff --git a/packages/speech-spine/package.json b/packages/speech-spine/package.jsonindex d2742020..f2359d4e 100644--- a/packages/speech-spine/package.json+++ b/packages/speech-spine/package.json@@ -21,6 +21,7 @@"@narratage/protocol": "workspace:*","@narratage/speech": "workspace:*","@narratage/speech-basis": "workspace:*",+ "@narratage/spatial": "workspace:*","@narratage/text": "workspace:*"},"devDependencies": {diff --git a/packages/speech-spine/src/fragment.ts b/packages/speech-spine/src/fragment.tsindex 59267fd9..9ed61e30 100644--- a/packages/speech-spine/src/fragment.ts+++ b/packages/speech-spine/src/fragment.ts@@ -7,6 +7,7 @@ import { sealGraphFragment } from "@narratage/elaborator";import type { FragmentOperation } from "@narratage/elaborator";import { mediaPipelineProducers } from "@narratage/media-pipeline";import { speechBasisProducers } from "@narratage/speech-basis";+import { spatialTypes } from "@narratage/spatial";import { speechSpineProducers, speechSpineTypes } from "./manifest.js";import type { SpeechSpineFragmentOptions } from "./types.js";@@ -16,7 +17,7 @@ const operation = (id: string) => ({ kind: "fragment-operation" as const, operatexport function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {if (options.takes.length === 0) throw new Error("Speech Spine requires at least one Take");- const names = new Set(["program"]);+ const names = new Set(["program", "canvas"]);for (const take of options.takes) {if (!take.mediaName || !take.segmentName || names.has(take.mediaName) || names.has(take.segmentName)) {throw new Error("Speech Spine Fragment input names are empty or duplicated");@@ -80,7 +81,7 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {{id: "spine:visual-track",producer: speechBasisProducers.projectVisual,- inputs: { basis: operation("spine:basis") },+ inputs: { basis: operation("spine:basis"), canvas: input("canvas") },result: { kind: "output", name: "visual" },},{@@ -95,6 +96,7 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {name: options.name?.trim() || "@narratage/speech-spine/spine@1",inputs: [{ name: "program", type: speechSpineTypes.spineProgram },+ { name: "canvas", type: spatialTypes.canvas },...options.takes.flatMap((take) => [{ name: take.mediaName, type: mediaTypes.synchronized },{ name: take.segmentName, type: narrativeTypes.excerpt },@@ -112,7 +114,7 @@ export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {fidelity: "exact",},{- name: "visual", type: compositionTypes.visualTrack, root: operation("spine:visual-track"), semanticInputs,+ name: "visual", type: compositionTypes.visualTrack, root: operation("spine:visual-track"), semanticInputs: [...semanticInputs, "canvas"],fidelity: "exact",},{diff --git a/packages/speech-spine/src/program.ts b/packages/speech-spine/src/program.tsindex d11fe662..b5f93f9e 100644--- a/packages/speech-spine/src/program.ts+++ b/packages/speech-spine/src/program.ts@@ -195,14 +195,14 @@ export function assembleSpeechBasis(});const visualClips = set.takes.map((take, index) => ({segmentId: take.segment.id,- artifact: {- digest: take.media.visual!.artifact.digest,- size: take.media.visual!.artifact.size,- mediaType: take.media.visual!.artifact.mediaType,- durationSec: seconds(take.media.timeline.frameCount),+ artifact: structuredClone(take.media.visual!.artifact),+ extent: {+ contract: "svml.intrinsic-extent@1" as const,+ widthPx: take.media.visual!.width,+ heightPx: take.media.visual!.height,},- startSec: segments[index]!.startSec,- endSec: segments[index]!.endSec,+ frameRate: { ...take.media.visual!.frameRate },+ frameCount: take.media.visual!.frameCount,}));return sealSpeechBasis({contract: "svml.speech-basis@1",diff --git a/packages/speech-spine/src/surface.ts b/packages/speech-spine/src/surface.tsindex b8cccc43..c5c9dff0 100644--- a/packages/speech-spine/src/surface.ts+++ b/packages/speech-spine/src/surface.ts@@ -14,6 +14,7 @@ import type {import { createSpeechSpineFragment } from "./fragment.js";import { speechSpineTypes } from "./manifest.js";import { sealSpeechSpineProgram } from "./program.js";+import { spatialTypes } from "@narratage/spatial";function localName(value: string): string {return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value;@@ -72,8 +73,9 @@ function takes(element: StructuredElement): StructuredElement[] {}export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- exactAttributes(element, ["id"]);+ exactAttributes(element, ["id", "canvas"]);const id = stringAttribute(element, "id");+ const canvas = resolve(element, "canvas", spatialTypes.canvas, resolveReference);const declaredTakes = takes(element).map((take, index) => ({mediaName: `take-${String(index + 1).padStart(4, "0")}-media`,segmentName: `take-${String(index + 1).padStart(4, "0")}-segment`,@@ -118,6 +120,7 @@ export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, refragment: assembly.id,inputs: {program: { kind: "record", id: programId },+ canvas: canvas.ref,...Object.fromEntries(declaredTakes.flatMap((take, index) => [[take.mediaName, { kind: "component-output" as const, component: normalizationComponents[index]!.id, output: "media" }],[take.segmentName, take.segment.ref],diff --git a/packages/speech-spine/test/surface.test.ts b/packages/speech-spine/test/surface.test.tsindex 753e403b..01dbcd69 100644--- a/packages/speech-spine/test/surface.test.ts+++ b/packages/speech-spine/test/surface.test.ts@@ -27,6 +27,7 @@ import {scriptSurfaceImplementationDigest,} from "@narratage/script";import { speechBasisManifest, speechBasisProducers } from "@narratage/speech-basis";+import { mediaTrackManifest } from "@narratage/media-track";import {decodeSpeechSpineSurface,speechSpineManifest,@@ -34,6 +35,12 @@ import {speechSpineProducers,speechSpineSurfaceImplementationDigest,} from "@narratage/speech-spine";+import {+ decodeCanvasSurface,+ spatialComponent,+ spatialModuleRef,+ spatialSurfaceDigests,+} from "@narratage/spatial";import {TextSurfaceRegistry,createTextAuthorFrontend,@@ -66,6 +73,7 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio planconst closure = createResolvedClosure([...videoContractManifests,mediaPipelineManifest,+ mediaTrackManifest,speechBasisManifest,speechSpineManifest,scriptManifest,@@ -74,6 +82,7 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio planconst surfaces = new TextSurfaceRegistry();surfaces.registerRaw(scriptModuleRef, "script", scriptSurfaceImplementationDigest, decodeScriptSurface);surfaces.registerStructured(speechSpineModuleRef, "spine", speechSpineSurfaceImplementationDigest, decodeSpeechSpineSurface);+ surfaces.registerStructured(spatialModuleRef, "canvas", spatialSurfaceDigests.canvas, decodeCanvasSurface);surfaces.registerStructured(fixtureModule, "media", fixtureSurfaceDigest, ({ element }) => ({records: ["take-one", "take-two"].map((id) => ({id, type: artifactTypes.blob,@@ -88,22 +97,26 @@ test("Speech Spine lowers ordered Takes into media normalization, one audio planresolveModule(request) {if (request.from.startsWith("@narratage/script")) return scriptModuleRef;if (request.from.startsWith("@narratage/speech")) return speechSpineModuleRef;+ if (request.from.startsWith("@narratage/spatial")) return spatialModuleRef;return fixtureModule;},}));const validators = new TypeValidatorRegistry();registerTypeValidatorFacets(validators, mediaPipelineComponent.validators ?? []);+ registerTypeValidatorFacets(validators, spatialComponent.validators ?? []);const compiled = await compileSourceClosure({entry: source(`<svml><import from="@narratage/script@1"/><import as="fixture" from="example.speech-media@1"/><import as="speech" from="@narratage/speech-spine@1"/>+ <import as="space" from="@narratage/spatial@1"/><script id="story"><opening><ALICE> Hello from Alice.</opening><answer><BOB> Hello from Bob.</answer></script><fixture:Media/>- <speech:Spine id="speech">+ <space:Canvas id="vertical" width="720" height="1280"/>+ <speech:Spine id="speech" canvas={vertical}><speech:Take source={take-one} segment={story.segment.opening}/><speech:Take source={take-two} segment={story.segment.answer}/></speech:Spine>diff --git a/packages/speech-spine/test/video-pipeline-fixture.ts b/packages/speech-spine/test/video-pipeline-fixture.tsindex 5b6494e4..e528d200 100644--- a/packages/speech-spine/test/video-pipeline-fixture.ts+++ b/packages/speech-spine/test/video-pipeline-fixture.ts@@ -41,6 +41,7 @@ import type {ValueSchema,} from "@narratage/protocol";import { resolveRealization, sealRealizationOverlay } from "@narratage/run";+import { spatialTypes, spatialValidatorDigests } from "@narratage/spatial";import {captionCorrespondence,captionDisplaySequence,@@ -50,6 +51,7 @@ import {import { speechAlignmentManifest } from "@narratage/speech-alignment";import { speechBasisManifest, speechBasisProjectionFragment } from "@narratage/speech-basis";import { mediaPipelineManifest } from "@narratage/media-pipeline";+import { mediaTrackManifest } from "@narratage/media-track";import {whisperXManifest,whisperXSpeechAlignmentFragment,@@ -262,6 +264,7 @@ export type VideoOutputName = keyof typeof videoOutputs;export const videoClosure = createResolvedClosure([...videoContractManifests,videoManifest,+ mediaTrackManifest,speechBasisManifest,mediaPipelineManifest,whisperXManifest,@@ -324,10 +327,33 @@ function authorProgram(): LinkedProgram {validatorDigest: captionValidatorDigests.plan,}) },];+ const rawCanvas = sealRecord({+ id: "canvas:vertical",+ type: spatialTypes.canvas,+ value: { kind: "inline", value: {+ contract: "svml.canvas-space@1",+ widthPx: 720,+ heightPx: 1280,+ origin: "top-left",+ xDirection: "right",+ yDirection: "down",+ pixelAspect: "square",+ } },+ conformance: "exact",+ origin,+ });+ const canvas = {+ ...rawCanvas,+ validation: sealTypeValidationReceipt({+ type: rawCanvas.type,+ recordDigest: rawCanvas.digest,+ validatorDigest: spatialValidatorDigests.canvas,+ }),+ };return link(videoClosure, [sealTypedModule({id: "author:video-pipeline",closureDigest: videoClosure.digest,- records: [narrative, ...captionRecords],+ records: [narrative, canvas, ...captionRecords],})]);}@@ -349,7 +375,10 @@ export function createVideoGraph(program: LinkedProgram): CompiledGraph {const projections = elaborateGraphFragment(program, speechBasisProjectionFragment, {id: "opening.take-projections",fragment: speechBasisProjectionFragment.id,- inputs: { basis: { kind: "logical-output", id: videoOutputs.take } },+ inputs: {+ basis: { kind: "logical-output", id: videoOutputs.take },+ canvas: { kind: "record", id: "canvas:vertical" },+ },});const alignment = elaborateGraphFragment(program, whisperXSpeechAlignmentFragment, {id: "opening.whisperx-alignment",diff --git a/packages/speech-spine/test/video-pipeline.test.ts b/packages/speech-spine/test/video-pipeline.test.tsindex 4e976204..b9960f78 100644--- a/packages/speech-spine/test/video-pipeline.test.ts+++ b/packages/speech-spine/test/video-pipeline.test.ts@@ -1,4 +1,4 @@-import { compositionComponent } from "../../test-support/video-domain.js";+import { compositionComponent, spatialComponent } from "../../test-support/video-domain.js";import {registerProducerFacets,registerTypeValidatorFacets,@@ -80,6 +80,7 @@ function validatorRegistry(): TypeValidatorRegistry {const registry = new TypeValidatorRegistry();registerTypeValidatorFacets(registry, captionComponent.validators ?? []);registerTypeValidatorFacets(registry, compositionComponent.validators);+ registerTypeValidatorFacets(registry, spatialComponent.validators);return registry;}@@ -172,9 +173,10 @@ test("the Speech Spine pipeline resumes without repeating paid calls", async ()audio: { kind: "blob", digest: audioDigest, size: 1, mediaType: "audio/wav" },visualTrack: { clips: [{segmentId: "line",- artifact: { digest: visualDigest, size: 1, mediaType: "video/mp4", durationSec },- startSec: 0,- endSec: durationSec,+ artifact: { kind: "blob", digest: visualDigest, size: 1, mediaType: "video/mp4" },+ extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },+ frameRate: { ...programSpace.frameRate },+ frameCount: Math.round(durationSec * 1_000),}] },segments: [{ segmentId: "line", startSec: 0, endSec: durationSec }],});@@ -233,11 +235,15 @@ test("the Speech Spine pipeline resumes without repeating paid calls", async ()({ inputs }) => {calls.projectVisual += 1;assert.equal(inputs.basis?.value.kind, "inline");+ assert(inputs.canvas?.value.kind === "inline");return {outputs: {visual: {kind: "inline",- value: projectSpeechVisual(inlineValue<SpeechBasis>(inputs.basis.value.value)),+ value: projectSpeechVisual(+ inlineValue<SpeechBasis>(inputs.basis.value.value),+ inlineValue(inputs.canvas.value.value),+ ),},},needs: {},@@ -544,9 +550,10 @@ test("an Existing SpeechBasis cuts generation while a visual substitute cuts the},visualTrack: { clips: [{segmentId: "line",- artifact: { digest: visualDigest, size: 1, mediaType: "video/mp4", durationSec: 1 },- startSec: 0,- endSec: 1,+ artifact: { kind: "blob", digest: visualDigest, size: 1, mediaType: "video/mp4" },+ extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },+ frameRate: { ...programSpace.frameRate },+ frameCount: 1_000,}] },segments: [{ segmentId: "line", startSec: 0, endSec: 1 }],});diff --git a/packages/speech/package.json b/packages/speech/package.jsonindex b5b6d0d2..68f7be11 100644--- a/packages/speech/package.json+++ b/packages/speech/package.json@@ -13,6 +13,7 @@"@narratage/media": "workspace:*","@narratage/narrative": "workspace:*","@narratage/program-space": "workspace:*",- "@narratage/protocol": "workspace:*"+ "@narratage/protocol": "workspace:*",+ "@narratage/spatial": "workspace:*"}}diff --git a/packages/speech/src/identity.ts b/packages/speech/src/identity.tsindex e7b38fbc..c2f5b650 100644--- a/packages/speech/src/identity.ts+++ b/packages/speech/src/identity.ts@@ -1,5 +1,6 @@import { isDigest } from "@narratage/protocol";-import { assertProgramSpaceIdentity } from "@narratage/program-space";+import { assertProgramSpaceIdentity, programSpaceFrameCount } from "@narratage/program-space";+import { assertIntrinsicExtent } from "@narratage/spatial";import type { SpeechAudioBasis, SpeechBasis, SpeechDuration, SpeechEvidenceAudio } from "./types.js";function assertAudioBlob(value: SpeechBasis["audio"], label: string): void {if (value.kind !== "blob" || !isDigest(value.digest) || !Number.isSafeInteger(value.size)@@ -33,6 +34,40 @@ export function assertSpeechBasisIdentity(basis: SpeechBasis): void {if (basis.contract !== "svml.speech-basis@1") throw new Error("Unsupported SpeechBasis contract.");assertProgramSpaceIdentity(basis.programSpace);assertAudioBlob(basis.audio, "SpeechBasis audio");+ if (basis.segments.length === 0 || basis.visualTrack.clips.length !== basis.segments.length) {+ throw new Error("SpeechBasis must pair every Segment with one visual clip.");+ }+ let previousEnd = 0;+ let accumulatedFrames = 0;+ const frameAt = (seconds: number): number => Math.round(+ seconds * basis.programSpace.frameRate.numerator / basis.programSpace.frameRate.denominator,+ );+ for (const [index, segment] of basis.segments.entries()) {+ const clip = basis.visualTrack.clips[index]!;+ if (!segment.segmentId || segment.startSec !== previousEnd || !Number.isFinite(segment.endSec)+ || segment.endSec <= segment.startSec || segment.endSec > basis.programSpace.durationSec) {+ throw new Error("SpeechBasis Segment is invalid or non-contiguous.");+ }+ if (clip.segmentId !== segment.segmentId || clip.artifact.kind !== "blob" || !isDigest(clip.artifact.digest)+ || !Number.isSafeInteger(clip.artifact.size) || clip.artifact.size < 0+ || !clip.artifact.mediaType.startsWith("video/")+ || clip.frameRate.numerator !== basis.programSpace.frameRate.numerator+ || clip.frameRate.denominator !== basis.programSpace.frameRate.denominator+ || !Number.isSafeInteger(clip.frameCount) || clip.frameCount <= 0) {+ throw new Error(`SpeechBasis visual clip ${clip.segmentId} is invalid.`);+ }+ assertIntrinsicExtent(clip.extent);+ const startFrame = frameAt(segment.startSec);+ const endFrame = frameAt(segment.endSec);+ if (startFrame !== accumulatedFrames || endFrame - startFrame !== clip.frameCount) {+ throw new Error(`SpeechBasis visual clip ${clip.segmentId} does not cover its Segment.`);+ }+ accumulatedFrames = endFrame;+ previousEnd = segment.endSec;+ }+ if (accumulatedFrames !== programSpaceFrameCount(basis.programSpace)) {+ throw new Error("SpeechBasis visuals do not cover ProgramSpace.");+ }}export function assertSpeechAudioBasisIdentity(basis: SpeechAudioBasis): void {if (basis.contract !== "svml.speech-audio-basis@1") throw new Error("Unsupported SpeechAudioBasis contract.");diff --git a/packages/speech/src/manifest.ts b/packages/speech/src/manifest.tsindex 896440cd..f1e91f99 100644--- a/packages/speech/src/manifest.ts+++ b/packages/speech/src/manifest.ts@@ -3,6 +3,7 @@ import { narrativeDependency } from "@narratage/narrative";import { programSpaceDependency } from "@narratage/program-space";import { digestOf } from "@narratage/protocol";import type { ModuleManifest, TypeRef } from "@narratage/protocol";+import { spatialDependency } from "@narratage/spatial";import { speechAudioBasisSchema, speechBasisSchema, speechDurationSchema, speechEvidenceAudioSchema } from "./schema.js";export const speechModuleRef = { name: "@narratage/speech", version: "0.0.0-dev" } as const;export const speechTypes = {@@ -10,7 +11,7 @@ export const speechTypes = {audioBasis: { module: speechModuleRef, name: "SpeechAudioBasis" }, evidenceAudio: { module: speechModuleRef, name: "SpeechEvidenceAudio" },} satisfies Record<string, TypeRef>;export const speechManifest: ModuleManifest = { format: "svml.module@1", name: speechModuleRef.name, version: speechModuleRef.version,- dependencies: [narrativeDependency, mediaDependency, programSpaceDependency],+ dependencies: [narrativeDependency, mediaDependency, programSpaceDependency, spatialDependency],types: [{ name: speechTypes.duration.name, schema: speechDurationSchema }, { name: speechTypes.basis.name, schema: speechBasisSchema },{ name: speechTypes.audioBasis.name, schema: speechAudioBasisSchema }, { name: speechTypes.evidenceAudio.name, schema: speechEvidenceAudioSchema }],capabilities: [], surfaces: [], producers: [] };diff --git a/packages/speech/src/schema.ts b/packages/speech/src/schema.tsindex 6b562b4c..bd497935 100644--- a/packages/speech/src/schema.ts+++ b/packages/speech/src/schema.ts@@ -1,6 +1,6 @@-import { mediaArtifactSchema } from "@narratage/media";import { programSpaceSchema } from "@narratage/program-space";import type { ValueSchema } from "@narratage/protocol";+import { intrinsicExtentSchema } from "@narratage/spatial";const string = { kind: "string", minLength: 1 } as const;const number = { kind: "number", minimum: 0 } as const;const integer = { kind: "number", integer: true, minimum: 0 } as const;@@ -16,7 +16,12 @@ const segment = object({ segmentId: { schema: string }, startSec: { schema: numbexport const speechDurationSchema: ValueSchema = object({ contract: { schema: { kind: "literal", value: "svml.speech-duration@1" } }, durationSec: { schema: number } });export const speechBasisSchema: ValueSchema = object({contract: { schema: { kind: "literal", value: "svml.speech-basis@1" } }, programSpace: { schema: programSpaceSchema }, audio: { schema: audioBlobRef },- visualTrack: { schema: object({ clips: { schema: { kind: "array", items: object({ segmentId: { schema: string }, artifact: { schema: mediaArtifactSchema }, startSec: { schema: number }, endSec: { schema: number } }) } } }) },+ visualTrack: { schema: object({ clips: { schema: { kind: "array", items: object({ segmentId: { schema: string }, artifact: { schema: object({+ kind: { schema: { kind: "literal", value: "blob" } }, digest: { schema: digest }, size: { schema: integer }, mediaType: { schema: string },+ }) }, extent: { schema: intrinsicExtentSchema }, frameRate: { schema: object({+ numerator: { schema: { kind: "number", integer: true, minimum: 1 } },+ denominator: { schema: { kind: "number", integer: true, minimum: 1 } },+ }) }, frameCount: { schema: { kind: "number", integer: true, minimum: 1 } } }) } } }) },segments: { schema: { kind: "array", minItems: 1, items: segment } },});export const speechAudioBasisSchema: ValueSchema = object({ contract: { schema: { kind: "literal", value: "svml.speech-audio-basis@1" } },diff --git a/packages/speech/src/types.ts b/packages/speech/src/types.tsindex f8ca8e16..9b8a2e3d 100644--- a/packages/speech/src/types.ts+++ b/packages/speech/src/types.ts@@ -1,11 +1,17 @@-import type { MediaArtifactRef } from "@narratage/media";import type { ProgramSpace } from "@narratage/program-space";import type { BlobRef } from "@narratage/protocol";+import type { IntrinsicExtent } from "@narratage/spatial";export type SpeechDuration = { readonly contract: "svml.speech-duration@1"; readonly durationSec: number };export type SpeechBasisSegment = { readonly segmentId: string; readonly startSec: number; readonly endSec: number };export type SpeechBasis = {readonly contract: "svml.speech-basis@1"; readonly programSpace: ProgramSpace; readonly audio: BlobRef;- readonly visualTrack: { readonly clips: readonly { readonly segmentId: string; readonly artifact: MediaArtifactRef; readonly startSec: number; readonly endSec: number }[] };+ readonly visualTrack: { readonly clips: readonly {+ readonly segmentId: string;+ readonly artifact: BlobRef;+ readonly extent: IntrinsicExtent;+ readonly frameRate: { readonly numerator: number; readonly denominator: number };+ readonly frameCount: number;+ }[] };readonly segments: readonly SpeechBasisSegment[];};export type SpeechAudioBasis = {diff --git a/packages/svs/test/source-closure.test.ts b/packages/svs/test/source-closure.test.tsindex 8f97714b..972586a8 100644--- a/packages/svs/test/source-closure.test.ts+++ b/packages/svs/test/source-closure.test.ts@@ -389,7 +389,7 @@ test("Source Closure rejects duplicate aliases and unknown Frontends before decotest("the golden studio.svs parses as generic Recipes without video knowledge", () => {const source = readFileSync(new URL("../../../examples/talking-film-golden/studio.svs", import.meta.url), "utf8");const parsed = parseSvs("studio.svs", maskSourceHeader(source, parseSourceHeader("studio.svs", source)));- assert.equal(parsed.recipes.length, 10);+ assert.equal(parsed.recipes.length, 11);assert.equal(parsed.recipes.find((recipe) => recipe.value.path === "caption.short-cues")?.value.properties.model, "gemini-2.5-flash");assert.equal(parsed.recipes.find((recipe) => recipe.value.path === "film.vertical")?.value.properties.background, "#09090B");});diff --git a/packages/test-support/video-domain.ts b/packages/test-support/video-domain.tsindex 014eaeab..30a62a89 100644--- a/packages/test-support/video-domain.ts+++ b/packages/test-support/video-domain.ts@@ -8,6 +8,7 @@ import { spatialComponent, spatialManifest } from "@narratage/spatial";import { speechManifest } from "@narratage/speech";import { speechEvidenceManifest } from "@narratage/speech-evidence";import { visualIrManifest } from "@narratage/visual-ir";+import { temporalManifest } from "@narratage/temporal";/** Test-only explicit video contract closure. Production packages import only the contracts they use. */export const videoContractManifests = [@@ -19,6 +20,7 @@ export const videoContractManifests = [speechEvidenceManifest,semanticMapManifest,spatialManifest,+ temporalManifest,visualIrManifest,compositionManifest,] as const;diff --git a/packages/video-cli/test/cli.test.ts b/packages/video-cli/test/cli.test.tsindex 461f36f3..55a8e2c1 100644--- a/packages/video-cli/test/cli.test.ts+++ b/packages/video-cli/test/cli.test.ts@@ -311,7 +311,9 @@ test("one checked-in fixture closes the complete provider-free video plan", asyn"assemble-speech-basis","request-whisperx-alignment","request-caption-gemini-plan",- "compile-broll",+ "request-media-inspection",+ "request-media-normalization",+ "project-media-visual-track","render-fine-caption","render-text-track","compile-composition",diff --git a/packages/video-cli/test/packages.ts b/packages/video-cli/test/packages.tsindex dba9bf5e..5ba2be29 100644--- a/packages/video-cli/test/packages.ts+++ b/packages/video-cli/test/packages.ts@@ -1,5 +1,4 @@import artifact from "../../artifact/src/activation.js";-import broll from "../../broll/src/activation.js";import captionGemini from "../../caption-gemini/src/activation.js";import captionFine from "../../caption-fine/src/activation.js";import caption from "../../caption/src/activation.js";@@ -11,6 +10,7 @@ import generation from "../../generation/src/activation.js";import renderHyperframes from "../../render-hyperframes/src/activation.js";import hyperframes from "../../hyperframes/src/activation.js";import mediaPipeline from "../../media-pipeline/src/activation.js";+import mediaTrack from "../../media-track/src/activation.js";import media from "../../media/src/activation.js";import narrative from "../../narrative/src/activation.js";import programSpace from "../../program-space/src/activation.js";@@ -25,6 +25,8 @@ import speechBasis from "../../speech-basis/src/activation.js";import speech from "../../speech/src/activation.js";import speechEvidence from "../../speech-evidence/src/activation.js";import semanticMap from "../../semantic-map/src/activation.js";+import spatial from "../../spatial/src/activation.js";+import temporal from "../../temporal/src/activation.js";import visualIr from "../../visual-ir/src/activation.js";import svs from "../../svs/src/activation.js";import textTrack from "../../text-track/src/activation.js";@@ -56,7 +58,9 @@ export const videoTestPackages = [speechBasis,speechSpine,whisperX,- broll,+ spatial,+ temporal,+ mediaTrack,textTrack,film,hyperframes,diff --git a/packages/whisperx/test/whisperx.test.ts b/packages/whisperx/test/whisperx.test.tsindex e9edd80e..d11fd3c0 100644--- a/packages/whisperx/test/whisperx.test.ts+++ b/packages/whisperx/test/whisperx.test.ts@@ -31,9 +31,10 @@ function basis() {audio: { kind: "blob", digest: digestOf("whisperx-test:audio"), size: 1, mediaType: "audio/wav" },visualTrack: { clips: [{segmentId: "line",- artifact: { digest: visual, size: 1, mediaType: "video/mp4", durationSec: 1 },- startSec: 0,- endSec: 1,+ artifact: { kind: "blob", digest: visual, size: 1, mediaType: "video/mp4" },+ extent: { contract: "svml.intrinsic-extent@1", widthPx: 720, heightPx: 1280 },+ frameRate: { ...programSpace.frameRate },+ frameCount: 30,}] },segments: [{ segmentId: "line", startSec: 0, endSec: 1 }],});diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 795d2879..5d26119b 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -14,9 +14,6 @@ importers:'@narratage/audio-track':specifier: workspace:*version: link:packages/audio-track- '@narratage/broll':- specifier: workspace:*- version: link:packages/broll'@narratage/caption':specifier: workspace:*version: link:packages/caption@@ -222,49 +219,6 @@ importers:specifier: workspace:*version: link:../text- packages/broll:- dependencies:- '@narratage/artifact':- specifier: workspace:*- version: link:../artifact- '@narratage/component-kit':- specifier: workspace:*- version: link:../component-kit- '@narratage/composition':- specifier: workspace:*- version: link:../composition- '@narratage/elaborator':- specifier: workspace:*- version: link:../elaborator- '@narratage/media':- specifier: workspace:*- version: link:../media- '@narratage/media-pipeline':- specifier: workspace:*- version: link:../media-pipeline- '@narratage/narrative':- specifier: workspace:*- version: link:../narrative- '@narratage/program-space':- specifier: workspace:*- version: link:../program-space- '@narratage/protocol':- specifier: workspace:*- version: link:../protocol- '@narratage/semantic-map':- specifier: workspace:*- version: link:../semantic-map- '@narratage/svs':- specifier: workspace:*- version: link:../svs- '@narratage/text':- specifier: workspace:*- version: link:../text- devDependencies:- '@narratage/hyperframes':- specifier: workspace:*- version: link:../hyperframes-packages/caption:dependencies:'@narratage/component-kit':@@ -1209,6 +1163,9 @@ importers:'@narratage/speech':specifier: workspace:*version: link:../speech+ '@narratage/text':+ specifier: workspace:*+ version: link:../textdevDependencies:'@narratage/core':specifier: workspace:*@@ -1234,18 +1191,30 @@ importers:'@narratage/media':specifier: workspace:*version: link:../media+ '@narratage/narrative':+ specifier: workspace:*+ version: link:../narrative'@narratage/program-space':specifier: workspace:*version: link:../program-space'@narratage/protocol':specifier: workspace:*version: link:../protocol+ '@narratage/semantic-map':+ specifier: workspace:*+ version: link:../semantic-map'@narratage/spatial':specifier: workspace:*version: link:../spatial+ '@narratage/svs':+ specifier: workspace:*+ version: link:../svs'@narratage/temporal':specifier: workspace:*version: link:../temporal+ '@narratage/text':+ specifier: workspace:*+ version: link:../textpackages/minimax-h3:dependencies:@@ -1411,8 +1380,8 @@ importers:specifier: ^3.700.0version: 3.1106.0'@hyperframes/aws-lambda':- specifier: 0.7.84- version: 0.7.84+ specifier: 0.7.101+ version: 0.7.101'@narratage/endpoint-kit':specifier: workspace:*version: link:../endpoint-kit@@ -1469,8 +1438,8 @@ importers:specifier: workspace:*version: link:../runtime-adapter-nodehyperframes:- specifier: 0.7.84- version: 0.7.84(@types/[email protected])+ specifier: 0.7.101+ version: 0.7.101(@types/[email protected])devDependencies:'@narratage/composition':specifier: workspace:*@@ -1970,6 +1939,9 @@ importers:'@narratage/protocol':specifier: workspace:*version: link:../protocol+ '@narratage/spatial':+ specifier: workspace:*+ version: link:../spatialpackages/speech-alignment:dependencies:@@ -2010,12 +1982,18 @@ importers:'@narratage/elaborator':specifier: workspace:*version: link:../elaborator+ '@narratage/media-track':+ specifier: workspace:*+ version: link:../media-track'@narratage/program-space':specifier: workspace:*version: link:../program-space'@narratage/protocol':specifier: workspace:*version: link:../protocol+ '@narratage/spatial':+ specifier: workspace:*+ version: link:../spatial'@narratage/speech':specifier: workspace:*version: link:../speech@@ -2068,6 +2046,9 @@ importers:'@narratage/protocol':specifier: workspace:*version: link:../protocol+ '@narratage/spatial':+ specifier: workspace:*+ version: link:../spatial'@narratage/speech':specifier: workspace:*version: link:../speech@@ -3363,8 +3344,8 @@ packages:peerDependencies:hono: ^4- '@hyperframes/[email protected]':- resolution: {integrity: sha512-ampI/JKXixe/ilmbLgo9Mg5BaWvrBSEWC2Ovc9rzIFkJ3V+6z736tOWhK5hQ70r6xmSfw2a+As/+trhGsujl7Q==}+ '@hyperframes/[email protected]':+ resolution: {integrity: sha512-SNv2b2cn/TKEuSWDeJ6POz29DYqFEnbhDkQ+57+6dv3Ztu4344ChGKlofzK8LxyVXGqLimOnxI8RG0cRxXBxFg==}engines: {node: '>=22'}peerDependencies:aws-cdk-lib: ^2.130.0@@ -3607,19 +3588,6 @@ packages:'@protobufjs/[email protected]':resolution: {integrity: sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug==}- '@puppeteer/[email protected]':- resolution: {integrity: sha512-B/gKoqlFkzhvzsI6jo9K1cZz9o5ypviVv/xu8CwA4grZzyVwN+XfkT+tu8T1zrauuEXv6VhS2oGX+6NL95WcKA==}- engines: {node: '>=22.12.0'}- hasBin: true- peerDependencies:- proxy-agent: '>=8.0.1'- yauzl: ^2.10.0 || ^3.4.0- peerDependenciesMeta:- proxy-agent:- optional: true- yauzl:- optional: true-'@puppeteer/[email protected]':resolution: {integrity: sha512-RDLpio3fH/qrj5k4DVY6eyiN8tCS0Zovd/6jW//n605oeqkWcUjn+3k+9ZtZBnbwMpsu0F7xDIiKXvVmG5c5Bw==}engines: {node: '>=22.12.0'}@@ -4411,8 +4379,8 @@ packages:resolution: {integrity: sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==}engines: {node: '>= 14'}- resolution: {integrity: sha512-JkfJtHRjPezHPdXoyoHjgnBEGQOffzjvAe4+bsHgbo1VBp3vYQI47GdL2bn3EydPa7p8c/shs7Ffs503ZrO4xA==}+ resolution: {integrity: sha512-Y6+qLyEo5rwHfMJzPXLACDvV5QlGo0qs2BgVmq/Jn2pgWg+N9VXsF9drBXnIG7v5DpDvqGumCttfsv/tmq3ixQ==}engines: {node: '>=22'}hasBin: true@@ -4608,10 +4576,6 @@ packages:resolution: {integrity: sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==}- resolution: {integrity: sha512-K1plkLOdeoUnGeT1OvdqF3qxl33v+Ra/uH5VyPEhXdMcpvGiEskHzxxEU3fgpccJpJLIipB/rPUsvkZRWeKqOA==}- engines: {node: '>=22.12.0'}-resolution: {integrity: sha512-XPNT0dQJtphqQ4I29zxlG4IIPbg1iEHAQKWuQgtMJGXjACV77pZSmJvDi51IIIfd+DTKICcopJwUx4upVQ4XbA==}engines: {node: '>=22.12.0'}@@ -5756,7 +5720,7 @@ snapshots:dependencies:hono: 4.12.32- '@hyperframes/[email protected]':+ '@hyperframes/[email protected]':dependencies:'@aws-sdk/client-s3': 3.1103.0'@aws-sdk/client-sfn': 3.1106.0@@ -5764,7 +5728,7 @@ snapshots:'@sparticuz/chromium': 148.0.0ffmpeg-static: 5.3.0ffprobe-static: 3.1.0- puppeteer-core: 25.4.0+ puppeteer-core: 25.5.0tar: 7.5.22transitivePeerDependencies:- bare-abort-controller@@ -5799,7 +5763,7 @@ snapshots:hono: 4.12.32linkedom: 0.18.13puppeteer: 25.5.0- puppeteer-core: 25.4.0+ puppeteer-core: 25.5.0transitivePeerDependencies:- bufferutil- canvas@@ -5851,7 +5815,7 @@ snapshots:linkedom: 0.18.13postcss: 8.5.25puppeteer: 25.5.0- puppeteer-core: 25.4.0+ puppeteer-core: 25.5.0wawoff2: 2.0.1transitivePeerDependencies:- bufferutil@@ -6014,11 +5978,6 @@ snapshots:'@protobufjs/[email protected]': {}- '@puppeteer/[email protected]':- dependencies:- modern-tar: 0.7.7- yargs: 18.1.0-'@puppeteer/[email protected]':dependencies:modern-tar: 0.7.7@@ -6834,10 +6793,10 @@ snapshots:transitivePeerDependencies:- supports-color- [email protected](@types/[email protected]):+ [email protected](@types/[email protected]):dependencies:'@hono/node-server': 2.0.12([email protected])- '@puppeteer/browsers': 3.0.6+ '@puppeteer/browsers': 3.1.0adm-zip: 0.6.0citty: 0.2.2compare-versions: 6.1.1@@ -6851,7 +6810,7 @@ snapshots:open: 10.2.0postcss: 8.5.25prettier: 3.9.6- puppeteer-core: 25.4.0+ puppeteer-core: 25.5.0sharp: 0.35.3(@types/[email protected])optionalDependencies:'@google/genai': 1.52.0@@ -7054,20 +7013,6 @@ snapshots:end-of-stream: 1.4.5once: 1.4.0- dependencies:- '@puppeteer/browsers': 3.0.6- chromium-bidi: 17.0.2([email protected])- devtools-protocol: 0.0.1653615- typed-query-selector: 2.12.2- webdriver-bidi-protocol: 0.4.2- ws: 8.21.1- transitivePeerDependencies:- - bufferutil- - proxy-agent- - utf-8-validate- - yauzl-dependencies:'@puppeteer/browsers': 3.1.0diff --git a/spec/author-surface-binding.md b/spec/author-surface-binding.mdindex 4be67467..dc1fb765 100644--- a/spec/author-surface-binding.md+++ b/spec/author-surface-binding.md@@ -7,7 +7,7 @@ not yet implemented.The generic Markup Frontend owns element syntax. A package-owned Surface owns what one imported tagmeans. SVS owns only its stylesheet syntax and produces generic immutable Recipe records. Neither-Markup, SVS nor Core owns Caption, B-roll, Film or another package's property vocabulary.+Markup, SVS nor Core owns Caption, Media, Film or another package's property vocabulary.The missing bridge is compile-time reference inspection: a Surface such as Caption must be able toread an explicitly referenced public Recipe and lower it into a nominal `CaptionProgram` before thediff --git a/spec/media-track.md b/spec/media-track.mdindex 4203433a..3c2d2087 100644--- a/spec/media-track.md+++ b/spec/media-track.md@@ -1,9 +1,9 @@# SVML Media Track Authoring-Status: design authority for the unified official Media Track migration. Its first explicit-still-image Item witness is executable; timed material, layers, motion, Sequence, audio projection-and the author Surface remain unimplemented. It supersedes the intended author model of the current-`@narratage/broll` vertical slice and is not a frozen public ABI.+Status: executable pre-release authority for the unified official Media Track. Item/Sequence+Surfaces, timed material, ordered layers, motion, handoffs, separate audio projection and the+restricted Speech projection are implemented. The former `@narratage/broll` vertical slice has+been retired rather than preserved as compatibility. The contract is not a frozen public ABI.## 1. Conclusion@@ -589,9 +589,10 @@ The package may use internal focused files or future implementation-only libraricreate one umbrella authoring library shared by Text, Caption, Ranking and Media, and it must notrequire a release of Core when a new media Recipe, motion operator or component package is added.-The existing `@narratage/broll` package remains an executable regression witness until the new path-passes. Because the project is pre-release, the migration may replace it rather than preserving a-public compatibility layer. Source assets and generated files remain untouched.+The former `@narratage/broll` package was used as a regression witness during migration and is now+retired. Because the project is pre-release, no compatibility package remains. “B-roll” is still a+useful editorial description for an Item or Sequence, not a terminal Type or author package.+Source assets and generated files remain untouched.## 16. Implementation order@@ -599,14 +600,16 @@ Implement only after the shared Temporal and Spatial slices exist:1. **Implemented:** introduce `@narratage/media-track` with one still-image Item whose BlobArtifact,IntrinsicExtent, SpatialFrame, ContentFit and ProgramSpace are separate semantic inputs;-2. add timed source trim and every occupancy/alignment policy;-3. add ordered Paint/media layers, clipping and the two-frame fit model;-4. add lifecycle and sampling motion with the fixed transform stack;-5. add explicit Sequence scheduling and pair transitions;-6. add separate source-audio and SFX projection;-7. route Speech Spine visual projection through the restricted lowerer;-8. migrate real B-roll examples and then retire the old package;-9. freeze only after the complete acceptance matrix passes.+2. **Implemented:** add timed source trim and every occupancy/alignment policy;+3. **Implemented:** add ordered Paint/media layers, clipping and the two-frame fit model;+4. **Implemented:** add lifecycle and sampling motion with the fixed transform stack;+5. **Implemented:** add explicit Sequence scheduling and pair transitions;+6. **Implemented:** add separate source-audio and SFX projection;+7. **Implemented:** route Speech Spine visual projection through the restricted lowerer;+8. **Implemented:** migrate the checked-in B-roll examples and retire the old package;+9. **Implemented:** close the package-local acceptance matrix with structural, real-media and+ partitioned-browser evidence. The shared Track/Visual IR remains pre-freeze for the independent+ gates in [`track-expressiveness.md`](./track-expressiveness.md).No step requires a Core, Runtime, queue or Provider change. Surface materialization may use anexisting explicitly registered Provider capability, but ordinary Item/Sequence lowering is@@ -625,7 +628,7 @@ deterministic local compilation.### Item-- Selection, Moment, Program and absolute windows under `one` and `each`;+- Program/absolute windows under `one`, and Selection/Moment windows under `one` and `each`;- independent overlapping Items at unrelated absolute stacking keys;- every source trim and shorter/equal/longer occupancy case;- full Canvas, split, lower card, corner and off-canvas Frames;@@ -641,7 +644,8 @@ deterministic local compilation.- cut, crossfade, push, wipe, cover and page turn;- start/center/end boundary-position ratios;- continuous source sampling across handoff overlap;-- transition duration at, below and above available separation;+- transition duration below, equal to and above the adjacent logical-phase separation when the+ explicit Sequence envelope and no-three-source laws still hold; invalid envelope/overlap fails;- missing, repeated, reversed and same-frame activation failure;- no implicit three-member overlap and no lower-composite input;- group entry/exit operating only on owned pixels.@@ -665,3 +669,33 @@ deterministic local compilation.- no Track contains provider, path, Narrative digest or source-family metadata;- no operation samples the accumulated lower composite;- the Speech Spine restricted projection and ordinary Item use the same lowering laws.++### Executable evidence++The matrix above is closed by executable evidence rather than by package status prose:++- `packages/media-track/test/media-track.test.ts` covers every Item binding, strict occurrence+ expansion, all documented Frames and Content Fits, ordered Paint/sample/backing layers, trim and+ occupancy, lifecycle/sustain/sampling motion, every Handoff operator and ratio, exact source+ audio/SFX projection, malformed Sequence rejection and explicit author-graph edges;+- `packages/provider-media-local/test/provider.test.ts` runs real FFmpeg inspection and+ normalization for moving video with and without audio, attached pictures, source A/V offsets,+ animated GIF, animated WebP, exact audio rendering and final muxing;+- `packages/media-execution/test/webp.test.ts` separately proves animated WebP straight-alpha+ blend and dispose behavior used by both local and Lambda execution;+- `packages/provider-media-aws-lambda/test/provider.test.ts` proves that remote execution receives+ the same exact `AudioProgramPlan`; both local and Lambda media environments execute+ `@narratage/media-execution` rather than owning alternate media semantics;+- `packages/speech-spine/test/surface.test.ts` and `packages/speech-basis/test/product.test.ts`+ prove the restricted Speech projection, one shared generated Product and independent visual/audio+ Candidate satisfaction;+- `packages/hyperframes/test/browser-visual.test.ts` renders opaque images, a straight-alpha+ Surface, two-frame fitting, local motion and a Sequence handoff through real Chromium, then+ compares every decoded RGBA frame across different worker partitions;+- `tools/package-boundaries.test.mjs` and `tools/graph-first-value-boundary.test.mjs` enforce the+ absence of a Media family registry and of hidden lineage/provider metadata in values.++The local package is therefore complete as a pre-freeze Media implementation. This does not claim+that the shared `svml.visual-track@1` or `svml.visual-ir@1` waist is frozen; renderer receipts,+Surface-byte validation and the remaining independent Text/Deck/Ranking witnesses still govern+that later decision.diff --git a/spec/track-authoring.md b/spec/track-authoring.mdindex 95cae35f..8fe77bee 100644--- a/spec/track-authoring.md+++ b/spec/track-authoring.md@@ -416,9 +416,9 @@ structural cuts are Narrative Selection consumption.## 8. Migration consequences-The current `text-track` and `broll` author Surfaces support only `during="full"` or an identity-Selection mapping. They must not independently grow another timing enum. Their rewrites should-consume the shared temporal package.+The current simple `text-track` Surface still supports only `during="full"` or an identity+Selection mapping. It must not independently grow another timing enum. Its rewrite must consume the+shared temporal package. Media Track already consumes that package for Item and Sequence timing.The first implementation batch must include tests for:diff --git a/spec/track-composition.md b/spec/track-composition.mdindex f1fab1f7..7d5c3972 100644--- a/spec/track-composition.md+++ b/spec/track-composition.md@@ -7,11 +7,11 @@ Status: first executable contract candidate; public freeze is gated byExcept for the shared ProgramSpace and canvas geometry, every audiovisual contribution entering aComposition is a peer Track. Composition validates, orders and combines Tracks. It does not know-Caption, Speech, B-roll, Text, Seedance or any author-package family.+Caption, Speech, Media, Text, Seedance or any author-package family.```textSpeech visual ───────────────> VisualTrack ─┐-B-roll ──────────────────────> VisualTrack ─┤+Media ───────────────────────> VisualTrack ─┤TimedCaption + style/cues ───> VisualTrack ─┤Text + style ────────────────> VisualTrack ─┼─> CompositionVignette / overlay ──────────> VisualTrack ─┤@@ -40,13 +40,13 @@ names.Its ProgramSpace identity is intrinsic because the Track cannot be interpreted without a clock.Which Records produced it is Graph/Derivation truth and is deliberately absent from the Track.-Composition flattens Presents across every Track before ordering them. One B-roll package may+Composition flattens Presents across every Track before ordering them. One Media package maytherefore own a board at z=30 and an icon at z=80 while a Text Present from another Track sits atz=50. A z change over time is represented by two non-overlapping Presents with different stackingkeys. Track ownership never creates a render stacking context or a provenance chain.Every element may optionally carry frame-exact local keyframes over opacity, transform, filter or-clip-path. Those keyframes operate only on that Present's element tree. A B-roll package may lower a+clip-path. Those keyframes operate only on that Present's element tree. A Media package may lower apair transition into complementary animations on two of its own Presents without teachingComposition the name or semantics of that transition.@@ -92,7 +92,7 @@ lowering must finish before the caption enters Composition as an ordinary Visual`@narratage/hyperframes` is the current reference compiler for Composition and its one versionedSVML Visual IR. It never-switches on Speech, Caption, B-roll or author-package identity. It deterministically emits a+switches on Speech, Caption, Media or author-package identity. It deterministically emits acontent-addressed `HyperframesDocument` whose HTML interleaves VisualPresents by absolute stackingkey. It must not mount an authoring Track as one isolated visual wrapper. AudioTrack compilationand final mux are separate media operations over the same ProgramSpace; HyperFrames is a silentdiff --git a/spec/track-expressiveness.md b/spec/track-expressiveness.mdindex 1d9b3966..9e9ebe21 100644--- a/spec/track-expressiveness.md+++ b/spec/track-expressiveness.md@@ -5,7 +5,7 @@ Status: executable validation gate; not yet a frozen public Track ABI.## PurposeOne public Track contract must solve final audiovisual composition without pretending that every-author package shares one authoring model. Text, Caption, B-roll, Ranking and future packages own+author package shares one authoring model. Text, Caption, Media, Ranking and future packages owntheir Programs and may revise those Programs independently. They may enter Composition only afterlowering their package-specific meanings into self-contained, frame-exact contributions.@@ -128,7 +128,7 @@ the Screen Overlay package. See [`screen-overlay.md`](./screen-overlay.md).| explicit media replacement group | Media Sequence | coordinated owned outgoing/incoming Presents || visible card history and depth reflow | Depth-Stack Deck Track | package-owned collection state over ordinary Presents || Ranking board and icons at unrelated z | Ranking Program | several absolute-stack Presents |-| B-roll enter/exit and pair transition | B-roll Program | local frame-exact keyframes over owned Presents |+| Media enter/exit and pair transition | Media Program | local frame-exact keyframes over owned Presents || arbitrary self-contained visual | component/provider | materialized compositable Surface Artifact || music/SFX/additional voice placement | Audio Program | ordinary frame-exact AudioTrack clips || flash/vignette/grain/veil | Screen Overlay Program | owned full-canvas Presents or alpha Surface |@@ -137,8 +137,8 @@ the Screen Overlay package. See [`screen-overlay.md`](./screen-overlay.md).## Current executable evidence`packages/hyperframes/test/track-expressiveness.test.ts` constructs package-private lowering-witnesses rather than adding Text, Caption, B-roll or Ranking fields to public contracts. Existing-package tests separately prove ordinary Text, Caption and B-roll Programs lower through the same+witnesses rather than adding Text, Caption, Media or Ranking fields to public contracts. Existing+package tests separately prove ordinary Text, Caption and Media Programs lower through the sameTrack and Composition path.The current `VisualTrack / VisualPresent / VisualElement` candidate explicitly binds@@ -167,7 +167,7 @@ property, environment-dependent value or alternate browser language without chanversion. The candidate is intentionally still not frozen. Unbound text remains temporarily legal forcandidate-era package migration, and a production Runtime still needs to bind its exact rendererimplementation and validate that Surface bytes satisfy the declared metadata. Those remaining-facts must stay generic; they must not be patched with Caption-, Text- or B-roll-specific fields.+facts must stay generic; they must not be patched with Caption-, Text- or Media-specific fields.E7 and E8 now have their official package witnesses. `@narratage/audio-track` proves the exact48 kHz source/target mapping, overlap law, bounded stretch and one shared local/remote@@ -185,7 +185,7 @@ Track v1 may be called a stable public waist only when all of the following hold3. E6 has a typed compositable-Surface path whose Runtime validates the declared media facts andrecords the renderer implementation;4. installing a new Track package requires no Core, Film, Composition or family registry change;-5. public Track data contains no Caption, Text, B-roll, Ranking or provider discriminator;+5. public Track data contains no Caption, Text, Media, Ranking or provider discriminator;6. a package can revise its Program schema while preserving or versioning only its own lowerer;7. unsupported cross-Track sampling fails closed.diff --git a/spec/visual-ir.md b/spec/visual-ir.mdindex d64e4d39..65d8233d 100644--- a/spec/visual-ir.md+++ b/spec/visual-ir.md@@ -12,7 +12,7 @@ SVML video stack. It is:- not a Provider API;- not part of the domain-neutral Core.-Text, Caption, B-roll, Ranking and third-party visual packages may own unrelated author Programs.+Text, Caption, Media, Ranking and third-party visual packages may own unrelated author Programs.Before entering a `Composition`, each package must lower its resolved result into a `VisualTrack`that explicitly names this IR. A renderer adapter validates and compiles that common language; itnever learns the originating component family. `@narratage/hyperframes` is the first reference@@ -21,7 +21,7 @@ adapter, not the owner of the protocol.```textauthor Program shared terminal protocol final-render route───────────────────── ───────────────────────── ─────────────────────-Text / Caption / B-roll ───> SVML Visual IR in Track ───┬─> HyperFrames+Text / Caption / Media ───> SVML Visual IR in Track ───┬─> HyperFramescustom visual producer ───> typed CompositableSurface ───┤├─> future Remotion└─> future render API@@ -100,9 +100,9 @@ Installing a new visual component must not modify this protocol. The component eOnly a capability that is both broadly reusable and impossible to represent through those twopaths may justify changing the `svml.visual-ir@1` contract. While Narratage is pre-release theidentifier remains `@1`, with exact implementation identity carried by digests and package locks.-Adding a Text style, Caption mode, B-roll recipe, transition or Provider never qualifies by itself.+Adding a Text style, Caption mode, Media recipe, transition or Provider never qualifies by itself.-Old-system Text three-box, Caption range/cue/content and media content-frame sampling models are+Old-system Text three-box, Caption range/cue/content and Media content-frame sampling models aretherefore package-owned Programs and lowering evidence. They are expressiveness witnesses for thisIR, not fields added to the IR.diff --git a/tools/graph-first-value-boundary.test.mjs b/tools/graph-first-value-boundary.test.mjsindex 257f6a9c..0406c0d7 100644--- a/tools/graph-first-value-boundary.test.mjs+++ b/tools/graph-first-value-boundary.test.mjs@@ -29,7 +29,7 @@ async function contents(packageNames) {}const domainValuePackages = [- "broll",+ "media-track","caption","caption-gemini","estimate",@@ -100,7 +100,7 @@ test("the generic graph stack has no payload-affinity side channel", async () =>test("fold values retain members, not hidden identity or policy context", async () => {const declarations = [- ["broll", "types.ts", "BrollSet"],+ ["media-track", "types.ts", "MediaTrackSet"],["film", "types.ts", "FilmTrackSet"],["speech-spine", "types.ts", "SpeechSpineSet"],["text-track", "types.ts", "TextTrackSet"],diff --git a/tools/package-boundaries.test.mjs b/tools/package-boundaries.test.mjsindex 168af42b..2e7d1a4d 100644--- a/tools/package-boundaries.test.mjs+++ b/tools/package-boundaries.test.mjs@@ -169,7 +169,7 @@ test("generic and video CLIs reach no Provider package and video CLI activates nassert.deepEqual(providers.filter((name) => videoDependencies.has(name)), []);assert.ok(!videoDependencies.has("@narratage/script"));assert.ok(!videoDependencies.has("@narratage/seedance-speaker"));- assert.ok(!videoDependencies.has("@narratage/broll"));+ assert.ok(!videoDependencies.has("@narratage/media-track"));assert.ok(!videoDependencies.has("@narratage/text-track"));assert.ok(!videoDependencies.has("@narratage/film"));assert.ok(videoDependencies.has("@narratage/cli"));@@ -178,8 +178,8 @@ test("generic and video CLIs reach no Provider package and video CLI activates ntest("domain packages confine their Text dependency to Surface and activation entries", async () => {const { readdir, readFile } = await import("node:fs/promises");const surfaceOnly = [- "broll", "caption", "caption-gemini", "estimate", "film", "render-hyperframes",- "image-transform", "media", "seedance", "seedance-speaker", "speech-spine", "whisperx",+ "caption", "caption-gemini", "estimate", "film", "render-hyperframes",+ "image-transform", "media", "media-pipeline", "media-track", "seedance", "seedance-speaker", "speech-spine", "whisperx",];const allowed = new Set(["surface.ts", "activation.ts"]);for (const name of surfaceOnly) {diff --git a/tsconfig.json b/tsconfig.jsonindex 15228956..6fffac0f 100644--- a/tsconfig.json+++ b/tsconfig.json@@ -204,9 +204,6 @@"@narratage/render-hyperframes": ["packages/render-hyperframes/src/index.ts"],- "@narratage/broll": [- "packages/broll/src/index.ts"- ],"@narratage/film": ["packages/film/src/index.ts"],