Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

feat(v2): close the talking-film author graph

kashorin0a0d2aebf676
diff --git a/README.md b/README.md
index 261fe2f8..cb691fa3 100644
--- a/README.md
+++ b/README.md
@@ -14,9 +14,10 @@
> Fragments with hygienic instance identities;
> `packages/realization` attaches locked Existing-Value or Fragment Candidates without changing the
> author graph identity. These are executable reference layers outside Core, not new Kernel magic.
-> `packages/speech-program` now composes the first real speech vertical from static Fragments:
-> SpeechTake projections, one-pass WhisperX evidence, a provider-neutral SemanticMap and timed
-> captions. Producer-declared affinity protects internal Fragment results before they enter state;
+> `packages/speech-program` now owns the namespaced Speech Spine Surface and composes normalized
+> generated Takes into one audio master, one ProgramSpace and peer Tracks. `packages/whisperx`
+> owns the explicit alignment Surface; provider-neutral SemanticMap and Caption timing remain
+> separately auditable Operations. Producer-declared affinity protects internal Fragment results before they enter state;
> Logical-Output affinity separately protects selected Candidates.
> `packages/broll` lowers provider-neutral B-roll Programs into independently stacked visual
> Presents and ordinary source-audio/SFX tracks. Local pop/fade/slide motion and push/page-turn
@@ -39,7 +40,7 @@
> distribution implement the trusted local path. Hosted adapters remain Runtime packages outside
> this slice. `packages/elaborator` now compiles recursive Source Closures,
> while `packages/svs` implements a deliberately generic record-only Recipe Frontend; package-owned
-> video Recipe validation remains undecided. `packages/compiler-node` now supplies the domain-free
+> video Recipe validation is package-owned rather than built into SVS. `packages/compiler-node` now supplies the domain-free
> registered-manifest resolver, injected Workspace compiler and file-to-Graph/
> BuildPlan facade; `packages/cli` assembles the first trusted Text + Script + SVS + Film + Render
> command-line prelude. `packages/validation` now supplies package-owned semantic Type admission: authored,
@@ -53,8 +54,10 @@
> author packages can now be activated from a byte-locked physical dependency closure without
> changing Core or the official CLI; automatic installation, untrusted parser/validator sandboxing,
> distributed leases and hosted WhisperX /
-> HyperFrames Endpoint adapters are not implemented yet. The first KIE generation Provider and its
-> seven explicit model families are implemented. `@svml/store-sqlite`, `@svml/artifact-store-fs` and
+> HyperFrames Endpoint adapters are not implemented yet; the local WhisperX and HyperFrames
+> Providers are. The KIE generation Provider and its seven explicit model families are implemented,
+> as are display-only Gemini Caption planning and its Google Vertex Provider.
+> `@svml/store-sqlite`, `@svml/artifact-store-fs` and
> `@svml/local` now provide the durable zero-service developer assembly.
> `@svml/endpoint-kit` and `@svml/credential-store-env` now bind static Endpoint identity,
> non-secret configuration, exact credential slots and recoverable wake/retry/cancel behavior;
@@ -66,7 +69,7 @@
> `@svml/media-pipeline` and `@svml/provider-media-local` now provide all-stream ffprobe inspection,
> attached-picture-safe stream selection and shared-origin A/V normalization. Embedded AAC remains
> an ordinary media fact until an explicit Narrative-bound speech component promotes it;
-> the v2 CLI exposes local follow, status and cancel without adding another authoritative queue. The existing
+> the v2 CLI exposes local follow, status, cancel and content-addressed final Artifact export without adding another authoritative queue. The existing
> root compiler and standard library remain the executable v1 research oracle during migration.
`@svml/component-kit` is the narrow compute-host port: deterministic model, media, WhisperX and
@@ -229,9 +232,11 @@ generation provider is invoked by tests or compilation.
[`examples/talking-film-golden`](examples/talking-film-golden/README.md) preserves the implemented
Script Surface and specifies the complete namespaced author experience for two Seedance Mini speech
takes, Seedance B-roll, explicit WhisperX alignment, two speaker caption styles, editorial text,
-peer Tracks, Film and Hyperframes rendering. Its README marks every implemented and missing lowering;
-Script, authored Image, Seedance generation, Film and Render are accepted by the current compiler,
-while Speech assembly and Track Surfaces remain the next target.
+peer Tracks, Film and Hyperframes rendering. Provider-free Speech, WhisperX, Caption, B-roll and
+Text Surfaces now compile. Display-only Gemini planning, the ordered Caption Style cascade and the
+Google Vertex Provider are also implemented;
+[`examples/talking-film-graph-check`](examples/talking-film-graph-check/README.md) is the fully
+checkable graph counterpart without production assets or paid execution.
## Specification and implementation record
diff --git a/docs/README.md b/docs/README.md
index 243985da..73c2b28b 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -24,33 +24,37 @@ inside a historical design record does not make that record normative.
the current Track is a contract candidate rather than a public compatibility promise.
9. [`../spec/hyperframes-visual-ir-v1.md`](../spec/hyperframes-visual-ir-v1.md) — the one versioned,
closed terminal visual language shared by video components; not a Core or author-component API.
-10. [`../examples/talking-film-golden/README.md`](../examples/talking-film-golden/README.md) — the
- canonical v2 authoring target: implemented Script, authored Image, Seedance generation, Film,
- SVS and render syntax surrounded by Speech assembly, WhisperX, Caption, B-roll and Text Surfaces
- that remain to be completed.
-11. [`package-vocabulary-and-ownership-v1.md`](./package-vocabulary-and-ownership-v1.md) — target
+10. [`../spec/caption-program-v1.md`](../spec/caption-program-v1.md) — implemented total default
+ Style, ordered Role/Selection replacement, display-atom planner and timing-join laws.
+11. [`../examples/talking-film-golden/README.md`](../examples/talking-film-golden/README.md) — the
+ canonical two-speaker v2 authoring target. Its Script projections, Caption Style cascade,
+ display-only Gemini planning and graph lowering are implemented; real assets and credentials
+ remain deployment inputs.
+12. [`package-vocabulary-and-ownership-v1.md`](./package-vocabulary-and-ownership-v1.md) — target
naming and ownership for framework, video, Provider, Runtime, queues, stores and distributions.
-12. [`../spec/author-surface-binding-v1.md`](../spec/author-surface-binding-v1.md) — implemented
+13. [`../spec/author-surface-binding-v1.md`](../spec/author-surface-binding-v1.md) — implemented
package-owned binding from generic imported Records such as SVS Recipes into typed Programs,
AuthorComponents and Graph Fragments.
-13. [`local-developer-runtime-v1.md`](./local-developer-runtime-v1.md) — implemented local
+14. [`local-developer-runtime-v1.md`](./local-developer-runtime-v1.md) — implemented local
SQLite/filesystem Runtime assembly, trusted configuration, recovery law and environment swaps.
-14. [`kie-generation-modules-v1.md`](./kie-generation-modules-v1.md) — implemented seven exact
+15. [`kie-generation-modules-v1.md`](./kie-generation-modules-v1.md) — implemented seven exact
generation model modules, shared generated-media Products and the recoverable KIE Provider.
-15. [`kie-live-smoke-2026-08-06.md`](./kie-live-smoke-2026-08-06.md) — credential-free deployment
+16. [`kie-live-smoke-2026-08-06.md`](./kie-live-smoke-2026-08-06.md) — credential-free deployment
evidence from seven representative live model calls and one synthetic-reference upload.
-16. [`media-inspection-and-normalization-v1.md`](./media-inspection-and-normalization-v1.md) —
+17. [`media-inspection-and-normalization-v1.md`](./media-inspection-and-normalization-v1.md) —
implemented all-stream inspection, attached-picture-safe selection, shared-origin A/V
normalization and the boundary that prevents embedded AAC from becoming speech by inference.
-17. [`media-execution-boundary-v1.md`](./media-execution-boundary-v1.md) — the implemented rule that
+18. [`media-execution-boundary-v1.md`](./media-execution-boundary-v1.md) — the implemented rule that
planning is ordinary code while probe/decode/mix/render/mux are explicit Provider Needs, plus
the visual/audio/mux split for final HyperFrames output, canonical WhisperX evidence-audio path
and implemented local Providers.
-18. [`../services/whisperx/README.md`](../services/whisperx/README.md) — installation, runtime
+19. [`../services/whisperx/README.md`](../services/whisperx/README.md) — installation, runtime
identity, path confinement and queue boundary for the implemented warm local WhisperX service.
-19. [`node-package-activation-v1.md`](./node-package-activation-v1.md) — implemented trusted author
+20. [`node-package-activation-v1.md`](./node-package-activation-v1.md) — implemented trusted author
and deterministic compute facet locking, with the exact boundary that keeps source imports away
from Provider and Runtime authority.
+21. [`caption-gemini-provider-contract.md`](./caption-gemini-provider-contract.md) — implemented
+ display-atom planning boundary, old Vertex execution evidence and no-text-rewrite contract.
The executable v2 Node entrypoints are documented in
[`../packages/compiler-node/README.md`](../packages/compiler-node/README.md),
diff --git a/examples/README.md b/examples/README.md
index bcd18b9c..21a4a0d2 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -8,12 +8,14 @@ and the complete v2 authoring target.
- [`v2-bootstrap/main.svml`](./v2-bootstrap/main.svml) is accepted by `pnpm svml:v2 check`. It
exercises the real Node Source Host, official Text/Script prelude and recursive SVS Frontend. It
deliberately stops before package-owned Recipe consumption and component Graph construction.
+- [`talking-film-graph-check/main.svml`](./talking-film-graph-check/main.svml) is the complete
+ provider-free author graph through Speech, WhisperX, B-roll, Caption, Text, Film and Render.
## v2 authoring target
- [`talking-film-golden/main.svml`](./talking-film-golden/main.svml) is the canonical visual and
- semantic target for the next author-facing Surfaces. Its Script syntax is implemented; its outer
- package Surfaces are tracked line by line in the adjacent README and are not executable yet.
+ semantic target for two-speaker Gemini captions. Its author and paid Caption packages are
+ implemented; the adjacent README tracks real assets and deployment bindings.
## Executable v1 research fixtures
diff --git a/examples/talking-film-golden/README.md b/examples/talking-film-golden/README.md
index 92a10007..fa0d201d 100644
--- a/examples/talking-film-golden/README.md
+++ b/examples/talking-film-golden/README.md
@@ -1,8 +1,8 @@
# Talking-film authoring golden fixture
-Status: **v2 authoring target, not executable end to end yet.** Script, authored Image, Seedance
-generation, Film, generic SVS and the explicit HyperFrames Render Surface are implemented. Speech
-assembly and Track components are the concrete package-local work remaining.
+Status: **two-speaker production target, not executable end to end with the checked-in fixture.**
+Every author Surface and the Gemini planning/Vertex execution package now exists. Real image/font
+assets, credentials and a selected local Runtime profile are not checked into this fixture.
This example replaces neither the executable v1 fixtures nor their regression value. In particular,
`examples/flat-track-launch` remains the capability oracle for selections, moments, B-roll motion,
@@ -38,23 +38,21 @@ Frontend named by `using`.
## Author data flow
```text
-Script ────────────────────────────────────────────────┐
- ├─ Segment + Prompt + reference -> Seedance Speech ─┤
- ├─ Segment + Prompt + reference -> Seedance Speech ─┤
- │ ▼
- │ Speech Spine
- │ ┌──────────┴──────────┐
- │ │ │
- │ audio basis visual/audio Tracks
- │ │
- ├──────────────────────────────> WhisperX + speech-align
- │ │
- │ CompleteSemanticMap
- │ ┌───────┴────────┐
- │ │ │
- │ Caption Track B-roll Track
- │
- └─ caption display projection
+Script dialogue + Prompt + reference ──> two Seedance Speech Needs
+ │
+ ▼
+ Speech Spine
+ ┌─────────┴─────────┐
+ │ │
+ audio basis visual/audio Tracks
+ │
+ ▼
+ WhisperX + align ───────────┐
+ CompleteSemanticMap │
+ ▼
+Script left display + Caption Program ──> Gemini Plan ──> Caption Track
+
+Script selections + generated B-roll ───────────────────> B-roll Track
Seedance B-roll + Text Track + every Track above
-> Film Composition
@@ -74,15 +72,17 @@ Candidates and Operations.
| `media:Image` | `@svml/media` | content-addressed authored Artifact | implemented |
| `seedance:Prompt` | `@svml/seedance` | package-private immutable direction value | implemented |
| `seedance:Speech model="mini"` | `@svml/seedance` | explicit Seedance Mini Need plus primary-video projection | implemented; explicit duration remains authored until Speech scheduling exists |
-| `speech:Spine` | `@svml/speech` | ordered clips -> one `SpeechBasis`, then ordinary projections | basis and four locked projection facets implemented; author assembly missing |
-| `whisperx:Alignment` | `@svml/whisperx` | 48k speech master -> explicit 16k evidence Need -> WhisperX -> provider-neutral `@svml/speech-align` -> Map | Graph Fragment, locked deterministic components, local media projection and local Provider/service implemented; Surface missing |
+| `speech:Spine` | `@svml/speech` | ordered clips -> normalized Takes, one `SpeechBasis`, then ordinary projections | Surface, fold, media normalization and projection components implemented |
+| `whisperx:Alignment` | `@svml/whisperx` | 48k speech master -> explicit 16k evidence Need -> WhisperX -> provider-neutral `@svml/speech-align` -> Map | Surface, Graph Fragment, deterministic components and local Provider/service implemented |
| `seedance:Video model="mini"` | `@svml/seedance` | explicit Seedance Mini video Need plus primary-video projection | implemented |
-| `broll:Track` | `@svml/broll` | semantic windows + media + recipe -> peer Visual/Audio Tracks | Program/lowering implemented; Surface missing |
-| `caption:Track` | `@svml/caption-gemini` | timed projection + Gemini cue grouping + role recipe -> VisualTrack | timing/lowering and owner validators are locked components; complete Recipe audit and Gemini/role Surface missing |
-| `text:Track` | `@svml/text-track` | package Program -> VisualTrack | Program/lowering implemented; Surface/exact-font use missing |
+| `broll:Track` | `@svml/broll` | semantic windows + normalized media + recipe -> peer Visual/Audio Tracks | Surface and deterministic lowering implemented |
+| `caption:Style` / `caption:Program` | `@svml/caption` | default total Style + ordered whole-Style replacement by Role or Selection | implemented |
+| `caption-ai:Planner` | `@svml/caption-gemini` | immutable display atoms + per-run requirements -> cue cuts and per-atom fields | implemented; Google Vertex Endpoint implemented separately |
+| `caption:Track` | `@svml/caption` | CaptionPlan + independent SemanticMap + complete Styles -> VisualTrack | implemented |
+| `text:Track` | `@svml/text-track` | package Spec + ProgramSpace -> VisualTrack | provider-free Surface/lowering implemented; exact-font use remains |
| `film:Film` | `@svml/film` | finite TrackSet fold -> Composition | Graph Fragment and official Surface implemented |
| `render:Video` | `@svml/hyperframes-render` | Composition -> silent HyperframesDocument render + explicit program audio + mux -> final video Artifact + Receipt | Surface, Fragment and all local execution Providers implemented |
-| `studio.svs` | `@svml/svs` | generic immutable Recipe Records; consuming packages must validate and lower them | parser and recursive source import implemented; video recipe consumers/exact font lowering missing |
+| `studio.svs` | `@svml/svs` | generic immutable Recipe Records; consuming packages validate and lower them | parser, imports and current package consumers implemented; exact font assets remain |
The remaining work is intentionally package-local. None of these rows requires a new Core video type,
a component-family registry in Core, a privileged Film root or another Track kind.
@@ -110,5 +110,6 @@ The intended final command is:
svml build main.svml --target final.video
```
-Before the remaining Surfaces and Providers are implemented, tools must report that the complete
-file is a design fixture rather than pretending to compile it with the v1 engine.
+Until real assets, credentials and a Runtime profile are supplied, tools must report this file as a
+production design fixture rather than promise a rendered file. `../talking-film-graph-check` is the
+fully checkable graph today.
diff --git a/examples/talking-film-golden/main.svml b/examples/talking-film-golden/main.svml
index 6276c88a..8bc84dae 100644
--- a/examples/talking-film-golden/main.svml
+++ b/examples/talking-film-golden/main.svml
@@ -1,8 +1,8 @@
<!--
SVML authoring golden fixture.
- Script, authored media, Seedance generation, Film and HyperFrames Render
- Surfaces are implemented. Speech assembly and Track Surfaces remain the
- package-local design target.
+ All provider-free author Surfaces are implemented. This golden intentionally
+ explicitly names caption-gemini because paid cue planning is author-selected,
+ not a Runtime guess.
-->
<svml>
<import from="@svml/script@1"/>
@@ -10,7 +10,8 @@
<import as="seedance" from="@svml/seedance@1"/>
<import as="speech" from="@svml/speech@1"/>
<import as="whisperx" from="@svml/whisperx@1"/>
- <import as="caption" from="@svml/caption-gemini@1"/>
+ <import as="caption" from="@svml/caption@1"/>
+ <import as="caption-ai" from="@svml/caption-gemini@1"/>
<import as="broll" from="@svml/broll@1"/>
<import as="text" from="@svml/text-track@1"/>
<import as="film" from="@svml/film@1"/>
@@ -53,11 +54,41 @@
regions assemble into a finished video. No readable UI text and no camera shake.
</seedance:Prompt>
+ <!-- Complete styles are declared once. Selection and use remain separate. -->
+ <caption:Style id="dialogue-caption" appearance={studio.caption.dialogue}>
+ <caption:Cues>Use short complete semantic phrases, usually between two and five words.</caption:Cues>
+ </caption:Style>
+
+ <caption:Style id="alice-caption" appearance={studio.caption.alice}>
+ <caption:Cues>Use short complete semantic phrases, usually between two and five words.</caption:Cues>
+ <caption:Field id="important" type="boolean" min-per-cue="0" max-per-cue="2">
+ Select at most two words whose emphasis best communicates this Cue.
+ </caption:Field>
+ </caption:Style>
+
+ <caption:Style id="bob-caption" appearance={studio.caption.bob}>
+ <caption:Cues>Use short complete semantic phrases, usually between two and five words.</caption:Cues>
+ </caption:Style>
+
+ <caption:Program id="caption-program" narrative={script} default={dialogue-caption}>
+ <caption:Use role="ALICE" style={alice-caption}/>
+ <caption:Use role="BOB" style={bob-caption}/>
+ <!-- Ordered whole-style replacement: no complement Selection is required. -->
+ <caption:Use on={script.selection.product-demo} style={dialogue-caption}/>
+ </caption:Program>
+
+ <caption-ai:Planner
+ id="caption-plan"
+ narrative={script}
+ program={caption-program}
+ model="gemini-2.5-flash"
+ />
+
<!-- The author explicitly chooses Seedance Mini. Runtime only binds a matching endpoint. -->
<seedance:Speech
id="alice-take"
model="mini"
- script={script.segment.opening}
+ dialogue={script.segment.opening.dialogue}
prompt={alice-direction}
duration="8"
>
@@ -67,7 +98,7 @@
<seedance:Speech
id="bob-take"
model="mini"
- script={script.segment.answer}
+ dialogue={script.segment.answer.dialogue}
prompt={bob-direction}
duration="8"
>
@@ -75,9 +106,9 @@
</seedance:Speech>
<!-- Ordered generated Takes become one audio basis plus its peer visual/audio Tracks. -->
- <speech:Spine id="speech" narrative={script} appearance={studio.speech.full}>
- <speech:Take source={alice-take}/>
- <speech:Take source={bob-take}/>
+ <speech:Spine id="speech">
+ <speech:Take source={alice-take} segment={script.segment.opening}/>
+ <speech:Take source={bob-take} segment={script.segment.answer}/>
</speech:Spine>
<!-- One explicit WhisperX path: request, normalize and deterministic Script alignment. -->
@@ -105,12 +136,9 @@
id="captions"
narrative={script}
map={timing.map}
- grouping={studio.caption.short-cues}
- appearance={studio.caption.dialogue}
- >
- <caption:Role role="ALICE" appearance={studio.caption.alice}/>
- <caption:Role role="BOB" appearance={studio.caption.bob}/>
- </caption:Track>
+ program={caption-program}
+ plan={caption-plan.plan}
+ />
<text:Track id="titles" space={speech.space}>
<text:Item
diff --git a/examples/talking-film-golden/studio.svs b/examples/talking-film-golden/studio.svs
index c9ec02ba..d18ce33e 100644
--- a/examples/talking-film-golden/studio.svs
+++ b/examples/talking-film-golden/studio.svs
@@ -54,6 +54,7 @@
y: 0.76;
width: 0.84;
font: inter-semibold;
+ weight: 600;
size: 58;
line-height: 0.96;
align: center;
@@ -64,11 +65,35 @@
}
caption.alice {
- active-fill: #73FBD3;
+ stack-order: 70;
+ x: 0.08;
+ y: 0.76;
+ width: 0.84;
+ font: inter-semibold;
+ weight: 600;
+ size: 58;
+ line-height: 0.96;
+ align: center;
+ fill: #73FBD3;
+ background: #09090BCC;
+ padding: 16 24;
+ radius: 18;
}
caption.bob {
- active-fill: #FFD166;
+ stack-order: 70;
+ x: 0.08;
+ y: 0.76;
+ width: 0.84;
+ font: inter-semibold;
+ weight: 600;
+ size: 58;
+ line-height: 0.96;
+ align: center;
+ fill: #FFD166;
+ background: #09090BCC;
+ padding: 16 24;
+ radius: 18;
}
text.title {
@@ -78,6 +103,7 @@
width: 0.88;
height: 0.10;
font: inter-black;
+ weight: 900;
size: 64;
align: center;
fill: #FFFFFF;
diff --git a/examples/talking-film-graph-check/README.md b/examples/talking-film-graph-check/README.md
new file mode 100644
index 00000000..d00f0df5
--- /dev/null
+++ b/examples/talking-film-graph-check/README.md
@@ -0,0 +1,15 @@
+# Talking-film graph check
+
+This fixture compiles the complete author graph: Script, generated-media requests, Speech Spine,
+WhisperX alignment, display-only Gemini Caption planning, B-roll, Caption, Text, Film and
+explicit HyperFrames rendering. `check` and `plan` do not invoke Seedance, WhisperX, Gemini or
+HyperFrames.
+
+```bash
+pnpm svml:v2 check examples/talking-film-graph-check/main.svml
+pnpm svml:v2 plan examples/talking-film-graph-check/main.svml --target final.video
+```
+
+The Gemini planner receives immutable left-side display atoms and resolved style runs only. Its generic
+`CaptionPlan` joins the independent `CompleteSemanticMap` in `@svml/caption`; neither the planner
+nor the Vertex Provider has text-rewriting or timing authority.
diff --git a/examples/talking-film-graph-check/main.svml b/examples/talking-film-graph-check/main.svml
new file mode 100644
index 00000000..034c43ed
--- /dev/null
+++ b/examples/talking-film-graph-check/main.svml
@@ -0,0 +1,54 @@
+<svml>
+ <import from="@svml/script@1"/>
+ <import as="seedance" from="@svml/seedance@1"/>
+ <import as="speech" from="@svml/speech@1"/>
+ <import as="whisperx" from="@svml/whisperx@1"/>
+ <import as="broll" from="@svml/broll@1"/>
+ <import as="caption" from="@svml/caption@1"/>
+ <import as="caption-ai" from="@svml/caption-gemini@1"/>
+ <import as="text" from="@svml/text-track@1"/>
+ <import as="film" from="@svml/film@1"/>
+ <import as="render" from="@svml/hyperframes-render@1"/>
+ <import as="studio" from="./studio.svs" using="@svml/svs@1"/>
+
+ <script id="story">
+ <opening><HOST>Meaning @demo becomes the source @/demo.</opening>
+ </script>
+
+ <seedance:Prompt id="direction">Locked medium close-up in a quiet daylight studio.</seedance:Prompt>
+ <seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue} prompt={direction} duration="5"/>
+ <seedance:Video id="motion" model="mini" prompt={direction} duration="5"/>
+
+ <speech:Spine id="speech">
+ <speech:Take source={take} segment={story.segment.opening}/>
+ </speech:Spine>
+ <whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>
+
+ <caption:Style id="base-caption" appearance={studio.caption.base}>
+ <caption:Cues>Prefer short complete semantic phrases. Never cross clause punctuation.</caption:Cues>
+ <caption:Field id="important" type="boolean"
+ min-per-cue="0" max-per-cue="2">
+ Select zero, one, or two words whose emphasis best communicates this Cue.
+ </caption:Field>
+ </caption:Style>
+ <caption:Program id="caption-program" narrative={story} default={base-caption}/>
+ <caption-ai:Planner id="cue-plan" narrative={story} program={caption-program} model="gemini-2.5-flash"/>
+
+ <broll:Track id="cards" map={timing.map}>
+ <broll:Item source={motion.video} during={story.selection.demo} appearance={studio.broll.card}/>
+ </broll:Track>
+ <caption:Track id="captions" narrative={story} map={timing.map}
+ plan={cue-plan.plan} program={caption-program}/>
+ <text:Track id="titles" space={speech.space}>
+ <text:Item text="MEANING" during="full" appearance={studio.text.title}/>
+ </text:Track>
+
+ <film:Film id="main" space={speech.space} appearance={studio.film.vertical}>
+ <film:Track source={speech.visual}/>
+ <film:Track source={speech.audioTrack}/>
+ <film:Track source={cards.visual}/>
+ <film:Track source={captions.track}/>
+ <film:Track source={titles.track}/>
+ </film:Film>
+ <render:Video id="final" composition={main.composition}/>
+</svml>
diff --git a/examples/talking-film-graph-check/studio.svs b/examples/talking-film-graph-check/studio.svs
new file mode 100644
index 00000000..a97bea8c
--- /dev/null
+++ b/examples/talking-film-graph-check/studio.svs
@@ -0,0 +1,16 @@
+<sheet version="1">
+ film.vertical { width: 1080; height: 1920; frame-rate: 30; background: #09090B; }
+ broll.card {
+ stack-order: 40; x: 0.1; y: 0.2; width: 0.8; height: 0.5; fit: cover;
+ background: #111116; radius: 20; enter: slide-up 4f; exit: fade 4f;
+ }
+ caption.base {
+ stack-order: 70; x: 0.08; y: 0.76; width: 0.84; font: Inter; weight: 600;
+ size: 58; line-height: 1; align: center; fill: #FFFFFF;
+ background: #09090BCC; padding: 16 24; radius: 18;
+ }
+ text.title {
+ stack-order: 90; x: 0.06; y: 0.06; width: 0.88; height: 0.1; font: Inter;
+ weight: 900; size: 64; align: center; fill: #FFFFFF; tracking: -1;
+ }
+</sheet>
diff --git a/packages/broll/README.md b/packages/broll/README.md
index 71a980e4..77f5d1f0 100644
--- a/packages/broll/README.md
+++ b/packages/broll/README.md
@@ -5,6 +5,10 @@ entrance/exit motion, pair transitions, source audio and transition SFX. It
emits ordinary `VisualTrack` and `AudioTrack` values whose Presents participate
in Composition by their own absolute z.
+The `<broll:Track>` Surface consumes explicit `CompleteSemanticMap`, authored Script selection,
+source Blob and SVS Recipe edges. Each source is inspected and normalized through the shared media
+Fragment; semantic-window placement and local motion are deterministic package Operations.
+
`BrollProgram@1` remains an executable vertical-slice fixture. The author-facing package is not
frozen until the old content-frame plus independent foreground/backdrop sampling witnesses lower
through the closed HyperFrames Visual IR.
diff --git a/packages/broll/package.json b/packages/broll/package.json
index 92bc3d7d..bb715551 100644
--- a/packages/broll/package.json
+++ b/packages/broll/package.json
@@ -7,8 +7,13 @@
".": "./src/index.ts"
},
"dependencies": {
+ "@svml/component-kit": "workspace:*",
"@svml/contracts": "workspace:*",
- "@svml/protocol": "workspace:*"
+ "@svml/elaborator": "workspace:*",
+ "@svml/media-pipeline": "workspace:*",
+ "@svml/protocol": "workspace:*",
+ "@svml/svs": "workspace:*",
+ "@svml/text": "workspace:*"
},
"devDependencies": {
"@svml/hyperframes": "workspace:*"
diff --git a/packages/broll/src/author.ts b/packages/broll/src/author.ts
new file mode 100644
index 00000000..9827ec52
--- /dev/null
+++ b/packages/broll/src/author.ts
@@ -0,0 +1,175 @@
+import {
+ assertProgramSpaceIdentity,
+ verifySynchronizedMedia,
+} from "@svml/contracts";
+import type {
+ CompleteSemanticMap,
+ NarrativeSelectionRef,
+ SynchronizedMedia,
+} from "@svml/contracts";
+import { canonicalize, digestOf, isDigest } from "@svml/protocol";
+
+import { sealBrollProgram } from "./program.js";
+import type {
+ BrollItemSpec,
+ BrollProgram,
+ BrollSet,
+ BrollTrackSpec,
+} from "./types.js";
+
+export const createBrollSetImplementationDigest = digestOf("@svml/broll/create-set@1");
+export const appendBrollItemImplementationDigest = digestOf("@svml/broll/append-item@1");
+export const finalizeBrollProgramImplementationDigest = digestOf("@svml/broll/finalize-program@1");
+export const projectBrollProgramSpaceImplementationDigest = digestOf("@svml/broll/project-program-space@1");
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function verifyMap(map: CompleteSemanticMap): void {
+ assert(map.contract === "svml.complete-semantic-map@1", "B-roll requires CompleteSemanticMap");
+ assertProgramSpaceIdentity(map.programSpace);
+ const { mapDigest: _digest, ...content } = map;
+ assert(isDigest(map.mapDigest) && map.mapDigest === digestOf(canonicalize(content)),
+ "CompleteSemanticMap digest differs from its contents");
+}
+
+export function projectBrollProgramSpace(map: CompleteSemanticMap) {
+ verifyMap(map);
+ return map.programSpace;
+}
+
+export function sealBrollItemSpec(value: Omit<BrollItemSpec, "digest">): BrollItemSpec {
+ const content = canonicalize(value) as unknown as Omit<BrollItemSpec, "digest">;
+ return { ...content, digest: digestOf(content) };
+}
+
+export function assertBrollItemSpec(value: BrollItemSpec): void {
+ assert(value.contract === "svml.broll-item-spec@1" && value.id.length > 0, "BrollItemSpec identity is invalid");
+ const { digest: _digest, ...content } = value;
+ assert(isDigest(value.digest) && value.digest === digestOf(canonicalize(content)), "BrollItemSpec digest differs");
+ assert(Number.isSafeInteger(value.z), "BrollItemSpec z must be an integer");
+ assert(value.fit === "contain" || value.fit === "cover", "BrollItemSpec fit is invalid");
+}
+
+export function sealBrollTrackSpec(value: Omit<BrollTrackSpec, "digest">): BrollTrackSpec {
+ const content = canonicalize(value) as unknown as Omit<BrollTrackSpec, "digest">;
+ return { ...content, digest: digestOf(content) };
+}
+
+function assertBrollTrackSpec(value: BrollTrackSpec): void {
+ const { digest: _digest, ...content } = value;
+ assert(value.contract === "svml.broll-track-spec@1" && value.id.length > 0
+ && isDigest(value.digest) && value.digest === digestOf(canonicalize(content)), "BrollTrackSpec is invalid");
+}
+
+function verifySelection(value: NarrativeSelectionRef): void {
+ assert(value.contract === "svml.narrative-selection@1" && value.id.length > 0 && value.occurrences.length > 0,
+ "B-roll selection is invalid");
+ const { selectionDigest: _digest, ...content } = value;
+ assert(isDigest(value.selectionDigest) && value.selectionDigest === digestOf(canonicalize(content)),
+ "B-roll selection digest differs");
+}
+
+function setContent(value: Omit<BrollSet, "digest">): Omit<BrollSet, "digest"> {
+ return canonicalize(value) as unknown as Omit<BrollSet, "digest">;
+}
+
+function sealSet(value: Omit<BrollSet, "digest">): BrollSet {
+ const content = setContent(value);
+ return { ...content, digest: digestOf(content) };
+}
+
+export function assertBrollSet(value: BrollSet): void {
+ assert(value.contract === "svml.broll-set@1" && value.id.length > 0, "BrollSet identity is invalid");
+ verifyMap(value.map);
+ const { digest: _digest, ...content } = value;
+ assert(isDigest(value.digest) && value.digest === digestOf(setContent(content)), "BrollSet digest differs");
+ assert(value.items.length === 0 ? value.lastAddition === undefined : value.lastAddition !== undefined,
+ "BrollSet last addition is inconsistent");
+}
+
+export function createBrollSet(map: CompleteSemanticMap, spec: BrollTrackSpec): BrollSet {
+ verifyMap(map);
+ assertBrollTrackSpec(spec);
+ return sealSet({ contract: "svml.broll-set@1", id: spec.id, map, items: [] });
+}
+
+function boundaryFrame(
+ map: CompleteSemanticMap,
+ edge: NarrativeSelectionRef["occurrences"][number]["open"],
+ side: "open" | "close",
+): number {
+ const index = edge.boundary.tokenIndex;
+ if (side === "open") {
+ if (edge.affinity === "left" && index > 0) return map.tokens[index - 1]?.startFrame ?? 0;
+ return map.tokens[index]?.startFrame ?? Math.round(map.durationSec * map.programSpace.frameRate.numerator / map.programSpace.frameRate.denominator);
+ }
+ if (edge.affinity === "right" && index < map.tokens.length) return map.tokens[index]?.endFrame ?? 0;
+ return index > 0 ? map.tokens[index - 1]?.endFrame ?? 0 : 0;
+}
+
+export function appendBrollItem(
+ set: BrollSet,
+ media: SynchronizedMedia,
+ selection: NarrativeSelectionRef,
+ spec: BrollItemSpec,
+): BrollSet {
+ assertBrollSet(set);
+ verifySynchronizedMedia(media);
+ verifySelection(selection);
+ assertBrollItemSpec(spec);
+ assert(media.visual !== undefined, `B-roll ${spec.id} source has no visual stream`);
+ assert(media.timeline.frameRate.numerator === set.map.programSpace.frameRate.numerator
+ && media.timeline.frameRate.denominator === set.map.programSpace.frameRate.denominator,
+ `B-roll ${spec.id} media uses another frame rate`);
+ const durationSec = media.timeline.frameCount * media.timeline.frameRate.denominator / media.timeline.frameRate.numerator;
+ const additions = selection.occurrences.map((occurrence, index) => {
+ const startFrame = boundaryFrame(set.map, occurrence.open, "open");
+ const endFrameExclusive = boundaryFrame(set.map, occurrence.close, "close");
+ assert(endFrameExclusive > startFrame, `B-roll ${spec.id} selection occurrence is empty`);
+ return {
+ id: selection.occurrences.length === 1 ? spec.id : `${spec.id}:${index + 1}`,
+ artifact: {
+ digest: media.visual!.artifact.digest,
+ size: media.visual!.artifact.size,
+ mediaType: media.visual!.artifact.mediaType,
+ durationSec,
+ },
+ span: { startFrame, endFrameExclusive },
+ z: spec.z,
+ tieBreak: `${set.id}:${spec.id}:${index + 1}`,
+ box: spec.box,
+ fit: spec.fit,
+ playback: "stretch" as const,
+ ...(spec.backgroundColor === undefined ? {} : { backgroundColor: spec.backgroundColor }),
+ ...(spec.borderRadiusPx === undefined ? {} : { borderRadiusPx: spec.borderRadiusPx }),
+ ...(spec.enter === undefined ? {} : { enter: spec.enter }),
+ ...(spec.exit === undefined ? {} : { exit: spec.exit }),
+ };
+ });
+ return sealSet({
+ contract: "svml.broll-set@1",
+ id: set.id,
+ map: set.map,
+ items: [...set.items, ...additions],
+ lastAddition: {
+ previousSetDigest: set.digest,
+ mediaDigest: media.synchronizedMediaDigest,
+ selectionDigest: selection.selectionDigest,
+ specDigest: spec.digest,
+ },
+ });
+}
+
+export function finalizeBrollProgram(set: BrollSet): BrollProgram {
+ assertBrollSet(set);
+ assert(set.items.length > 0, "B-roll Track requires at least one Item");
+ return sealBrollProgram({
+ contract: "svml.broll-program@1",
+ id: set.id,
+ programSpaceDigest: set.map.programSpace.digest,
+ items: set.items,
+ transitions: [],
+ });
+}
diff --git a/packages/broll/src/component.ts b/packages/broll/src/component.ts
new file mode 100644
index 00000000..ed627982
--- /dev/null
+++ b/packages/broll/src/component.ts
@@ -0,0 +1,103 @@
+import type { ComponentPackage } from "@svml/component-kit";
+import type { ProgramSpace } from "@svml/contracts";
+import type { CompleteSemanticMap, NarrativeSelectionRef, SynchronizedMedia } from "@svml/contracts";
+import type { StoredValue } from "@svml/protocol";
+import { canonicalize } from "@svml/protocol";
+
+import { brollProducers } from "./manifest.js";
+import {
+ appendBrollItem,
+ appendBrollItemImplementationDigest,
+ createBrollSet,
+ createBrollSetImplementationDigest,
+ finalizeBrollProgram,
+ finalizeBrollProgramImplementationDigest,
+ projectBrollProgramSpace,
+ projectBrollProgramSpaceImplementationDigest,
+} from "./author.js";
+import {
+ compileBrollImplementationDigest,
+ compileBrollProduct,
+ projectBrollAudio,
+ projectBrollAudioImplementationDigest,
+ projectBrollVisual,
+ projectBrollVisualImplementationDigest,
+} from "./program.js";
+import type { BrollItemSpec, BrollProduct, BrollProgram, BrollSet, BrollTrackSpec } from "./types.js";
+
+function inline<T>(value: StoredValue | undefined, subject: string): T {
+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);
+ return value.value as unknown as T;
+}
+
+export const brollComponent = {
+ name: "@svml/broll",
+ producers: [
+ {
+ producer: brollProducers.projectSpace,
+ implementationDigest: projectBrollProgramSpaceImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { space: { kind: "inline", value: canonicalize(projectBrollProgramSpace(
+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),
+ )) } }, needs: {},
+ }),
+ },
+ {
+ producer: brollProducers.createSet,
+ implementationDigest: createBrollSetImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { set: { kind: "inline", value: canonicalize(createBrollSet(
+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),
+ inline<BrollTrackSpec>(inputs.spec?.value, "BrollTrackSpec"),
+ )) } }, needs: {},
+ }),
+ },
+ {
+ producer: brollProducers.appendItem,
+ implementationDigest: appendBrollItemImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { set: { kind: "inline", value: canonicalize(appendBrollItem(
+ inline<BrollSet>(inputs.set?.value, "BrollSet"),
+ inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),
+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"),
+ inline<BrollItemSpec>(inputs.spec?.value, "BrollItemSpec"),
+ )) } }, needs: {},
+ }),
+ },
+ {
+ producer: brollProducers.finalize,
+ implementationDigest: finalizeBrollProgramImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { program: { kind: "inline", value: canonicalize(finalizeBrollProgram(
+ inline<BrollSet>(inputs.set?.value, "BrollSet"),
+ )) } }, needs: {},
+ }),
+ },
+ {
+ producer: brollProducers.compile,
+ implementationDigest: compileBrollImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { product: { kind: "inline", value: canonicalize(compileBrollProduct(
+ inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<BrollProgram>(inputs.program?.value, "BrollProgram"),
+ )) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: brollProducers.projectVisual,
+ implementationDigest: projectBrollVisualImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { visual: { kind: "inline", value: canonicalize(projectBrollVisual(inline<BrollProduct>(inputs.product?.value, "BrollProduct"))) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: brollProducers.projectAudio,
+ implementationDigest: projectBrollAudioImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { audio: { kind: "inline", value: canonicalize(projectBrollAudio(inline<BrollProduct>(inputs.product?.value, "BrollProduct"))) } },
+ needs: {},
+ }),
+ },
+ ],
+} satisfies ComponentPackage;
diff --git a/packages/broll/src/index.ts b/packages/broll/src/index.ts
index b147f342..7dd9db8d 100644
--- a/packages/broll/src/index.ts
+++ b/packages/broll/src/index.ts
@@ -11,13 +11,33 @@ export {
projectBrollVisualImplementationDigest,
sealBrollProgram,
} from "./program.js";
+export { brollComponent } from "./component.js";
+export {
+ appendBrollItem,
+ appendBrollItemImplementationDigest,
+ assertBrollItemSpec,
+ assertBrollSet,
+ createBrollSet,
+ createBrollSetImplementationDigest,
+ finalizeBrollProgram,
+ finalizeBrollProgramImplementationDigest,
+ projectBrollProgramSpace,
+ projectBrollProgramSpaceImplementationDigest,
+ sealBrollItemSpec,
+ sealBrollTrackSpec,
+} from "./author.js";
export {
brollManifest,
brollManifestDigest,
brollModuleRef,
brollProductSchema,
brollProgramSchema,
+ brollSetSchema,
+ brollItemSpecSchema,
+ brollTrackSpecSchema,
+ brollSurfaceImplementationDigest,
brollProducers,
brollTypes,
} from "./manifest.js";
+export { decodeBrollTrackSurface } from "./surface.js";
export type * from "./types.js";
diff --git a/packages/broll/src/manifest.ts b/packages/broll/src/manifest.ts
index 807ad900..fbb818e2 100644
--- a/packages/broll/src/manifest.ts
+++ b/packages/broll/src/manifest.ts
@@ -1,13 +1,25 @@
import {
audioTrackSchema,
+ completeSemanticMapSchema,
contractTypes,
mediaArtifactSchema,
+ narrativeSelectionSchema,
programSpaceSchema,
+ synchronizedMediaSchema,
videoContractDependencies,
visualTrackSchema,
} from "@svml/contracts";
import { digestOf } from "@svml/protocol";
import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";
+import { mediaPipelineManifest, mediaPipelineModuleRef, mediaPipelineTypes } from "@svml/media-pipeline";
+import { svsManifest, svsModuleRef } from "@svml/svs";
+
+import {
+ appendBrollItemImplementationDigest,
+ createBrollSetImplementationDigest,
+ finalizeBrollProgramImplementationDigest,
+ projectBrollProgramSpaceImplementationDigest,
+} from "./author.js";
import {
compileBrollImplementationDigest,
@@ -19,8 +31,15 @@ export const brollModuleRef = { name: "@svml/broll", version: "0.0.0-dev" } as c
export const brollTypes = {
program: { module: brollModuleRef, name: "BrollProgram" },
product: { module: brollModuleRef, name: "BrollProduct" },
+ trackSpec: { module: brollModuleRef, name: "BrollTrackSpec" },
+ itemSpec: { module: brollModuleRef, name: "BrollItemSpec" },
+ set: { module: brollModuleRef, name: "BrollSet" },
} satisfies Record<string, TypeRef>;
export const brollProducers = {
+ createSet: { module: brollModuleRef, name: "create-broll-set" },
+ appendItem: { module: brollModuleRef, name: "append-broll-item" },
+ finalize: { module: brollModuleRef, name: "finalize-broll-program" },
+ projectSpace: { module: brollModuleRef, name: "project-broll-program-space" },
compile: { module: brollModuleRef, name: "compile-broll" },
projectVisual: { module: brollModuleRef, name: "project-broll-visual" },
projectAudio: { module: brollModuleRef, name: "project-broll-audio" },
@@ -65,6 +84,20 @@ const item = object({
audioGain: { schema: number, optional: true },
enter: { schema: motion, optional: true },
exit: { schema: motion, optional: true },
+ backgroundColor: { schema: string, optional: true },
+ borderRadiusPx: { schema: number, optional: true },
+});
+const itemSpec = object({
+ contract: { schema: { kind: "literal", value: "svml.broll-item-spec@1" } },
+ digest: { schema: digest },
+ id: { schema: string },
+ z: { schema: signedInteger },
+ box: { schema: box },
+ fit: { schema: { kind: "string", enum: ["contain", "cover"] } },
+ backgroundColor: { schema: string, optional: true },
+ borderRadiusPx: { schema: number, optional: true },
+ enter: { schema: motion, optional: true },
+ exit: { schema: motion, optional: true },
});
const transition = object({
id: { schema: string },
@@ -96,21 +129,95 @@ export const brollProductSchema: ValueSchema = object({
audioTrack: { schema: audioTrackSchema },
});
+export const brollTrackSpecSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.broll-track-spec@1" } },
+ digest: { schema: digest },
+ id: { schema: string },
+});
+export const brollItemSpecSchema: ValueSchema = itemSpec;
+export const brollSetSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.broll-set@1" } },
+ digest: { schema: digest },
+ id: { schema: string },
+ map: { schema: completeSemanticMapSchema },
+ items: { schema: { kind: "array", items: item } },
+ lastAddition: { schema: object({
+ previousSetDigest: { schema: digest }, mediaDigest: { schema: digest },
+ selectionDigest: { schema: digest }, specDigest: { schema: digest },
+ }), optional: true },
+});
+
+export const brollSurfaceImplementationDigest = digestOf("@svml/broll/track-surface@1");
+
export const brollManifest: ModuleManifest = {
format: "svml.module@0",
name: brollModuleRef.name,
version: brollModuleRef.version,
dependencies: [
+ videoContractDependencies.narrative,
+ videoContractDependencies.media,
+ videoContractDependencies.semanticTime,
videoContractDependencies.programSpace,
videoContractDependencies.composition,
+ { module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },
+ { module: svsModuleRef, digest: digestOf(svsManifest) },
],
types: [
{ name: brollTypes.program.name, schema: brollProgramSchema },
{ name: brollTypes.product.name, schema: brollProductSchema },
+ { name: brollTypes.trackSpec.name, schema: brollTrackSpecSchema },
+ { name: brollTypes.itemSpec.name, schema: brollItemSpecSchema },
+ { name: brollTypes.set.name, schema: brollSetSchema },
],
capabilities: [],
- surfaces: [],
+ surfaces: [{
+ name: "track", tag: "Track", mode: "structured",
+ outputs: [brollTypes.trackSpec, brollTypes.itemSpec, mediaPipelineTypes.selectionRequest,
+ brollTypes.program, brollTypes.product, contractTypes.visualTrack, contractTypes.audioTrack],
+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/broll/track-surface", digest: brollSurfaceImplementationDigest },
+ }],
producers: [
+ {
+ name: brollProducers.projectSpace.name,
+ inputs: [{ name: "map", type: contractTypes.completeSemanticMap }],
+ outputs: [{ name: "space", type: contractTypes.programSpace, affinity: [
+ { resultPointer: "/digest", input: "map", inputPointer: "/programSpace/digest" },
+ ] }], needs: [],
+ implementation: { kind: "registered", locator: "@svml/broll/project-program-space", digest: projectBrollProgramSpaceImplementationDigest },
+ },
+ {
+ name: brollProducers.createSet.name,
+ inputs: [
+ { name: "map", type: contractTypes.completeSemanticMap },
+ { name: "spec", type: brollTypes.trackSpec },
+ ],
+ outputs: [{ name: "set", type: brollTypes.set }], needs: [],
+ implementation: { kind: "registered", locator: "@svml/broll/create-set", digest: createBrollSetImplementationDigest },
+ },
+ {
+ name: brollProducers.appendItem.name,
+ inputs: [
+ { name: "set", type: brollTypes.set },
+ { name: "media", type: contractTypes.synchronizedMedia },
+ { name: "selection", type: contractTypes.narrativeSelection },
+ { name: "spec", type: brollTypes.itemSpec },
+ ],
+ outputs: [{ name: "set", type: brollTypes.set, affinity: [
+ { resultPointer: "/lastAddition/previousSetDigest", input: "set", inputPointer: "/digest" },
+ { resultPointer: "/lastAddition/mediaDigest", input: "media", inputPointer: "/synchronizedMediaDigest" },
+ { resultPointer: "/lastAddition/selectionDigest", input: "selection", inputPointer: "/selectionDigest" },
+ { resultPointer: "/lastAddition/specDigest", input: "spec", inputPointer: "/digest" },
+ ] }], needs: [],
+ implementation: { kind: "registered", locator: "@svml/broll/append-item", digest: appendBrollItemImplementationDigest },
+ },
+ {
+ name: brollProducers.finalize.name,
+ inputs: [{ name: "set", type: brollTypes.set }],
+ outputs: [{ name: "program", type: brollTypes.program, affinity: [
+ { resultPointer: "/programSpaceDigest", input: "set", inputPointer: "/map/programSpace/digest" },
+ ] }], needs: [],
+ implementation: { kind: "registered", locator: "@svml/broll/finalize-program", digest: finalizeBrollProgramImplementationDigest },
+ },
{
name: brollProducers.compile.name,
inputs: [
diff --git a/packages/broll/src/program.ts b/packages/broll/src/program.ts
index 0c2efaf0..95661b34 100644
--- a/packages/broll/src/program.ts
+++ b/packages/broll/src/program.ts
@@ -116,6 +116,12 @@ function assertItem(item: BrollItem, totalFrames: number): void {
if (item.audioGain !== undefined && (!Number.isFinite(item.audioGain) || item.audioGain < 0)) {
throw new Error(`B-roll item ${item.id} audioGain is invalid.`);
}
+ if (item.backgroundColor !== undefined && !/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/iu.test(item.backgroundColor)) {
+ throw new Error(`B-roll item ${item.id} background color is invalid.`);
+ }
+ if (item.borderRadiusPx !== undefined && (!Number.isFinite(item.borderRadiusPx) || item.borderRadiusPx < 0)) {
+ throw new Error(`B-roll item ${item.id} border radius is invalid.`);
+ }
if (item.includeAudio && isImage) throw new Error(`B-roll image ${item.id} cannot contribute source audio.`);
if (item.includeAudio && item.playback === "loop") {
throw new Error(`B-roll item ${item.id} cannot loop source audio in the v1 AudioTrack contract.`);
@@ -311,6 +317,8 @@ function itemElements(
{ name: "top", value: `${item.box.yPercent}%` },
{ name: "transform-origin", value: "center center" },
{ name: "width", value: `${item.box.widthPercent}%` },
+ ...(item.backgroundColor === undefined ? [] : [{ name: "background-color", value: item.backgroundColor }]),
+ ...(item.borderRadiusPx === undefined ? [] : [{ name: "border-radius", value: `${item.borderRadiusPx}px` }]),
];
return [
{
diff --git a/packages/broll/src/surface.ts b/packages/broll/src/surface.ts
new file mode 100644
index 00000000..01a49513
--- /dev/null
+++ b/packages/broll/src/surface.ts
@@ -0,0 +1,167 @@
+import { contractTypes } from "@svml/contracts";
+import { sealGraphFragment } from "@svml/elaborator";
+import type { FragmentOperation } from "@svml/elaborator";
+import {
+ mediaPipelineTypes,
+ sealMediaSelectionRequest,
+ synchronizedMediaFragment,
+} from "@svml/media-pipeline";
+import { svsRecipeType } from "@svml/svs";
+import type { SvsRecipe } from "@svml/svs";
+import type { StructuredElement, StructuredSurfaceHandler, SurfaceResolvedReference, TextAttributeValue } from "@svml/text";
+
+import { sealBrollItemSpec, sealBrollTrackSpec } from "./author.js";
+import { brollProducers, brollTypes } from "./manifest.js";
+import type { BrollMotion, BrollSurfaceItemInput } from "./types.js";
+
+const input = (name: string) => ({ kind: "fragment-input" as const, name });
+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
+
+function createBrollSurfaceFragment(items: readonly BrollSurfaceItemInput[], name: string) {
+ const operations: FragmentOperation[] = [
+ { id: "broll:space", producer: brollProducers.projectSpace, inputs: { map: input("map") }, result: { kind: "output", name: "space" } },
+ { id: "broll:set:empty", producer: brollProducers.createSet, inputs: { map: input("map"), spec: input("track-spec") }, result: { kind: "output", name: "set" } },
+ ];
+ let current = "broll:set:empty";
+ items.forEach((item, index) => {
+ const id = `broll:set:append:${String(index + 1).padStart(4, "0")}`;
+ operations.push({
+ id, producer: brollProducers.appendItem,
+ inputs: { set: operation(current), media: input(item.mediaName), selection: input(item.selectionName), spec: input(item.specName) },
+ result: { kind: "output", name: "set" },
+ });
+ current = id;
+ });
+ operations.push(
+ { id: "broll:program", producer: brollProducers.finalize, inputs: { set: operation(current) }, result: { kind: "output", name: "program" } },
+ { id: "broll:product", producer: brollProducers.compile, inputs: { space: operation("broll:space"), program: operation("broll:program") }, result: { kind: "output", name: "product" } },
+ { id: "broll:visual", producer: brollProducers.projectVisual, inputs: { product: operation("broll:product") }, result: { kind: "output", name: "visual" } },
+ { id: "broll:audio", producer: brollProducers.projectAudio, inputs: { product: operation("broll:product") }, result: { kind: "output", name: "audio" } },
+ );
+ const semanticInputs = ["map", "track-spec", ...items.flatMap((item) => [item.mediaName, item.selectionName, item.specName])];
+ return sealGraphFragment({
+ name,
+ inputs: [
+ { name: "map", type: contractTypes.completeSemanticMap },
+ { name: "track-spec", type: brollTypes.trackSpec },
+ ...items.flatMap((item) => [
+ { name: item.mediaName, type: contractTypes.synchronizedMedia },
+ { name: item.selectionName, type: contractTypes.narrativeSelection },
+ { name: item.specName, type: brollTypes.itemSpec },
+ ]),
+ ],
+ operations,
+ exports: [
+ { name: "visual", type: contractTypes.visualTrack, root: operation("broll:visual"), semanticInputs,
+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("map"), sourcePointer: "/programSpace/digest" }], fidelity: "exact" },
+ { name: "audio", type: contractTypes.audioTrack, root: operation("broll:audio"), semanticInputs,
+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("map"), sourcePointer: "/programSpace/digest" }], fidelity: "exact" },
+ ],
+ });
+}
+
+function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+}
+function exact(element: StructuredElement, names: readonly string[]): void {
+ if (Object.keys(element.attributes).sort().join("\0") !== [...names].sort().join("\0")) throw new Error(`${element.name} requires exactly ${names.join(", ")}`);
+}
+function text(element: StructuredElement, name: string): string {
+ const value = element.attributes[name];
+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);
+ return value.trim();
+}
+function ref(element: StructuredElement, name: string, expected: SurfaceResolvedReference["type"], resolve: (path: string) => SurfaceResolvedReference | undefined) {
+ const raw: TextAttributeValue | undefined = element.attributes[name];
+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a reference`);
+ const value = resolve(raw.path);
+ if (!value || !sameType(value.type, expected)) throw new Error(`${element.name}.${name} cannot resolve a value of the required type`);
+ return value;
+}
+function recipe(value: SurfaceResolvedReference): SvsRecipe {
+ if (!sameType(value.type, svsRecipeType) || value.record?.value.kind !== "inline") throw new Error("B-roll appearance must be an authored SVS Recipe");
+ return value.record.value.value as unknown as SvsRecipe;
+}
+function num(value: SvsRecipe, name: string): number {
+ const result = value.properties[name];
+ if (typeof result !== "number" || !Number.isFinite(result)) throw new Error(`B-roll Recipe ${name} must be a number`);
+ return result;
+}
+function str(value: SvsRecipe, name: string): string {
+ const result = value.properties[name];
+ if (typeof result !== "string" || !result) throw new Error(`B-roll Recipe ${name} must be a string`);
+ return result;
+}
+function motion(value: string): BrollMotion {
+ const match = /^(fade|pop|slide-up|slide-down)\s+([1-9][0-9]*)f$/u.exec(value.trim());
+ if (!match) throw new Error(`B-roll motion ${value} is invalid`);
+ return { operator: match[1] as BrollMotion["operator"], durationFrames: Number(match[2]) };
+}
+
+export const decodeBrollTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ exact(element, ["id", "map"]);
+ const id = text(element, "id");
+ const map = ref(element, "map", contractTypes.completeSemanticMap, resolveReference);
+ const trackSpecId = `${id}.spec`;
+ const requestId = `${id}.selection`;
+ const records: Array<{ id: string; type: typeof brollTypes.itemSpec | typeof brollTypes.trackSpec | typeof mediaPipelineTypes.selectionRequest; value: { kind: "inline"; value: ReturnType<typeof sealBrollItemSpec> | ReturnType<typeof sealBrollTrackSpec> | ReturnType<typeof sealMediaSelectionRequest> }; range: typeof element.range }> = [
+ { id: trackSpecId, type: brollTypes.trackSpec, value: { kind: "inline", value: sealBrollTrackSpec({ contract: "svml.broll-track-spec@1", id }) }, range: element.range },
+ { id: requestId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline", value: sealMediaSelectionRequest({
+ contract: "svml.media-selection-request@1", video: { mode: "primary-moving" }, audio: { mode: "none" },
+ spanAuthority: "video", frameRate: { numerator: 30, denominator: 1 },
+ }) }, range: element.range },
+ ];
+ let itemIndex = 0;
+ const declared = element.children.flatMap((child) => {
+ if (child.kind === "text") {
+ if (child.value.trim()) throw new Error(`${element.name} accepts only Item children`);
+ return [];
+ }
+ if (!child.name.endsWith(":Item") && child.name !== "Item") throw new Error(`${element.name} accepts only Item children`);
+ itemIndex += 1;
+ exact(child, ["source", "during", "appearance"]);
+ const source = ref(child, "source", contractTypes.blobArtifact, resolveReference);
+ const selection = ref(child, "during", contractTypes.narrativeSelection, resolveReference);
+ const appearance = recipe(ref(child, "appearance", svsRecipeType, resolveReference));
+ const expected = ["background", "enter", "exit", "fit", "height", "radius", "stack-order", "width", "x", "y"];
+ if (Object.keys(appearance.properties).sort().join("\0") !== expected.sort().join("\0")) throw new Error(`B-roll Recipe requires exactly ${expected.join(", ")}`);
+ const fit = str(appearance, "fit");
+ if (fit !== "contain" && fit !== "cover") throw new Error("B-roll Recipe fit is invalid");
+ const suffix = String(itemIndex).padStart(4, "0");
+ const specId = `${id}.item.${suffix}.spec`;
+ records.push({
+ id: specId, type: brollTypes.itemSpec,
+ value: { kind: "inline", value: sealBrollItemSpec({
+ contract: "svml.broll-item-spec@1", id: `${id}.item.${suffix}`, z: num(appearance, "stack-order"),
+ box: { xPercent: num(appearance, "x") * 100, yPercent: num(appearance, "y") * 100,
+ widthPercent: num(appearance, "width") * 100, heightPercent: num(appearance, "height") * 100 },
+ fit, backgroundColor: str(appearance, "background"), borderRadiusPx: num(appearance, "radius"),
+ enter: motion(str(appearance, "enter")), exit: motion(str(appearance, "exit")),
+ }) }, range: child.range,
+ });
+ return [{ suffix, source, selection, specId, range: child.range,
+ input: { mediaName: `item-${suffix}-media`, selectionName: `item-${suffix}-selection`, specName: `item-${suffix}-spec` } }];
+ });
+ if (declared.length === 0) throw new Error(`${element.name} requires at least one Item`);
+ const fragment = createBrollSurfaceFragment(declared.map((item) => item.input), `@svml/broll/surface/${id}@1`);
+ const normalization = declared.map((item) => ({
+ id: `${id}.normalize.${item.suffix}`, fragment: synchronizedMediaFragment.id,
+ inputs: { source: item.source.ref, request: { kind: "record" as const, id: requestId } },
+ outputs: { media: `${id}.normalized.${item.suffix}` }, range: item.range,
+ }));
+ return {
+ records,
+ components: [
+ ...normalization,
+ { id, fragment: fragment.id, inputs: {
+ map: map.ref, "track-spec": { kind: "record", id: trackSpecId },
+ ...Object.fromEntries(declared.flatMap((item, index) => [
+ [item.input.mediaName, { kind: "component-output" as const, component: normalization[index]!.id, output: "media" }],
+ [item.input.selectionName, item.selection.ref],
+ [item.input.specName, { kind: "record" as const, id: item.specId }],
+ ])),
+ }, outputs: { visual: `${id}.visual`, audio: `${id}.audio` }, range: element.range },
+ ],
+ fragments: [synchronizedMediaFragment, fragment],
+ };
+};
diff --git a/packages/broll/src/types.ts b/packages/broll/src/types.ts
index 6c0044ee..49bed3c8 100644
--- a/packages/broll/src/types.ts
+++ b/packages/broll/src/types.ts
@@ -37,6 +37,8 @@ export type BrollItem = {
readonly audioGain?: number;
readonly enter?: BrollMotion;
readonly exit?: BrollMotion;
+ readonly backgroundColor?: string;
+ readonly borderRadiusPx?: number;
};
export type BrollPairTransition = {
@@ -70,3 +72,42 @@ export type BrollProduct = {
readonly visualTrack: VisualTrack;
readonly audioTrack: AudioTrack;
};
+
+export type BrollItemSpec = {
+ readonly contract: "svml.broll-item-spec@1";
+ readonly digest: Digest;
+ readonly id: string;
+ readonly z: number;
+ readonly box: BrollBox;
+ readonly fit: "contain" | "cover";
+ readonly backgroundColor?: string;
+ readonly borderRadiusPx?: number;
+ readonly enter?: BrollMotion;
+ readonly exit?: BrollMotion;
+};
+
+export type BrollTrackSpec = {
+ readonly contract: "svml.broll-track-spec@1";
+ readonly digest: Digest;
+ readonly id: string;
+};
+
+export type BrollSet = {
+ readonly contract: "svml.broll-set@1";
+ readonly digest: Digest;
+ readonly id: string;
+ readonly map: import("@svml/contracts").CompleteSemanticMap;
+ readonly items: readonly BrollItem[];
+ readonly lastAddition?: {
+ readonly previousSetDigest: Digest;
+ readonly mediaDigest: Digest;
+ readonly selectionDigest: Digest;
+ readonly specDigest: Digest;
+ };
+};
+
+export type BrollSurfaceItemInput = {
+ readonly mediaName: string;
+ readonly selectionName: string;
+ readonly specName: string;
+};
diff --git a/packages/caption-gemini/README.md b/packages/caption-gemini/README.md
new file mode 100644
index 00000000..359a2de2
--- /dev/null
+++ b/packages/caption-gemini/README.md
@@ -0,0 +1,32 @@
+# `@svml/caption-gemini`
+
+Exact Gemini Caption planning, without transcript correction, Style selection or timing authority.
+
+The package consumes an already-resolved `CaptionProgram`. Gemini sees only its immutable left-side
+display atoms and the planning requirements for each resolved run. It never receives WhisperX
+text, timestamps or the right side of Dual Text, and it cannot return replacement wording.
+
+```xml
+<caption-ai:Planner
+ id="caption-plan"
+ narrative={story}
+ program={caption-program}
+ model="gemini-2.5-flash"
+/>
+```
+
+The output is `caption-plan.plan`. Its response has one entry per Program run. Within each run the
+model may only:
+
+- choose `after_atom_id` Cue boundaries that partition the run exactly once and in order;
+- attach declared field values to atoms inside that Cue.
+
+Fields are generic per-atom attributes, not hard-coded emphasis spans. A Style may declare no
+fields, one `important` flag, independent `best` and `medium` flags, enums, bounded numbers, or any
+other combination expressible by its declarations. Each atom may receive zero, one or multiple
+attributes. The validator enforces ids, value types and per-Cue cardinality before a `CaptionPlan`
+enters the graph.
+
+Model choice is author-visible. Network execution is separate: a Runtime explicitly binds the
+exact Gemini capability to `@svml/provider-google-vertex` or another compatible endpoint. The local
+Scheduler owns concurrency; credentials and transport never enter `.svml` or BuildState.
diff --git a/packages/caption-gemini/package.json b/packages/caption-gemini/package.json
new file mode 100644
index 00000000..c1e0d5b0
--- /dev/null
+++ b/packages/caption-gemini/package.json
@@ -0,0 +1,20 @@
+{
+ "name": "@svml/caption-gemini",
+ "version": "0.0.0-dev",
+ "private": true,
+ "type": "module",
+ "exports": {
+ ".": "./src/index.ts"
+ },
+ "dependencies": {
+ "@svml/caption": "workspace:*",
+ "@svml/component-kit": "workspace:*",
+ "@svml/contracts": "workspace:*",
+ "@svml/elaborator": "workspace:*",
+ "@svml/protocol": "workspace:*",
+ "@svml/text": "workspace:*"
+ },
+ "devDependencies": {
+ "@svml/script": "workspace:*"
+ }
+}
diff --git a/packages/caption-gemini/src/component.ts b/packages/caption-gemini/src/component.ts
new file mode 100644
index 00000000..394db912
--- /dev/null
+++ b/packages/caption-gemini/src/component.ts
@@ -0,0 +1,59 @@
+import type { ComponentPackage } from "@svml/component-kit";
+import type { CaptionProgram } from "@svml/caption";
+import type { Narrative } from "@svml/contracts";
+import type { StoredValue } from "@svml/protocol";
+import { canonicalize } from "@svml/protocol";
+
+import { captionGeminiImplementationDigests, captionGeminiProducers, captionGeminiTypes } from "./manifest.js";
+import { verifyCaptionGeminiProgram } from "./program.js";
+import { compileCaptionGeminiRequest, verifyCaptionGeminiRequest } from "./request.js";
+import type { CaptionGeminiProgram, CaptionGeminiRequest } from "./types.js";
+
+function inline<T>(value: StoredValue | undefined, subject: string): T {
+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);
+ return value.value as T;
+}
+
+export const captionGeminiComponent = {
+ name: "@svml/caption-gemini",
+ producers: [
+ {
+ producer: captionGeminiProducers.compile,
+ implementationDigest: captionGeminiImplementationDigests.compile,
+ handler: ({ inputs }) => ({
+ outputs: {
+ request: {
+ kind: "inline",
+ value: canonicalize(compileCaptionGeminiRequest(
+ inline<Narrative>(inputs.narrative?.value, "Narrative"),
+ inline<CaptionProgram>(inputs.captionProgram?.value, "CaptionProgram"),
+ inline<CaptionGeminiProgram>(inputs.program?.value, "CaptionGeminiProgram"),
+ )),
+ },
+ },
+ needs: {},
+ }),
+ },
+ {
+ producer: captionGeminiProducers.request,
+ implementationDigest: captionGeminiImplementationDigests.request,
+ handler: ({ inputs }) => {
+ const request = inline<CaptionGeminiRequest>(inputs.request?.value, "CaptionGeminiRequest");
+ verifyCaptionGeminiRequest(request);
+ return { outputs: {}, needs: { plan: canonicalize(request) } };
+ },
+ },
+ ],
+ validators: [
+ {
+ type: captionGeminiTypes.program,
+ implementationDigest: captionGeminiImplementationDigests.programValidator,
+ handler: ({ value }) => verifyCaptionGeminiProgram(inline(value, "CaptionGeminiProgram")),
+ },
+ {
+ type: captionGeminiTypes.request,
+ implementationDigest: captionGeminiImplementationDigests.requestValidator,
+ handler: ({ value }) => verifyCaptionGeminiRequest(inline(value, "CaptionGeminiRequest")),
+ },
+ ],
+} satisfies ComponentPackage;
diff --git a/packages/caption-gemini/src/fragment.ts b/packages/caption-gemini/src/fragment.ts
new file mode 100644
index 00000000..f1cf7e67
--- /dev/null
+++ b/packages/caption-gemini/src/fragment.ts
@@ -0,0 +1,43 @@
+import { contractTypes } from "@svml/contracts";
+import { captionTypes } from "@svml/caption";
+import { sealGraphFragment } from "@svml/elaborator";
+
+import { captionGeminiProducers, captionGeminiTypes } from "./manifest.js";
+
+const input = (name: string) => ({ kind: "fragment-input" as const, name });
+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
+
+export const captionGeminiPlanningFragment = sealGraphFragment({
+ name: "@svml/caption-gemini/planning@1",
+ inputs: [
+ { name: "narrative", type: contractTypes.narrative },
+ { name: "captionProgram", type: captionTypes.program },
+ { name: "program", type: captionGeminiTypes.program },
+ ],
+ operations: [
+ {
+ id: "caption-gemini:compile",
+ producer: captionGeminiProducers.compile,
+ inputs: { narrative: input("narrative"), captionProgram: input("captionProgram"), program: input("program") },
+ result: { kind: "output", name: "request" },
+ },
+ {
+ id: "caption-gemini:plan",
+ producer: captionGeminiProducers.request,
+ inputs: { request: operation("caption-gemini:compile") },
+ result: { kind: "need", name: "plan", accepts: "exact" },
+ },
+ ],
+ exports: [{
+ name: "plan",
+ type: captionTypes.plan,
+ root: operation("caption-gemini:plan"),
+ semanticInputs: ["narrative", "captionProgram", "program"],
+ affinity: [{
+ resultPointer: "/planningRequestDigest",
+ source: operation("caption-gemini:compile"),
+ sourcePointer: "/requestDigest",
+ }],
+ fidelity: "exact",
+ }],
+});
diff --git a/packages/caption-gemini/src/index.ts b/packages/caption-gemini/src/index.ts
new file mode 100644
index 00000000..1901561a
--- /dev/null
+++ b/packages/caption-gemini/src/index.ts
@@ -0,0 +1,22 @@
+export { captionGeminiComponent } from "./component.js";
+export { captionGeminiPlanningFragment } from "./fragment.js";
+export {
+ captionGeminiCapabilities,
+ captionGeminiImplementationDigests,
+ captionGeminiManifest,
+ captionGeminiManifestDigest,
+ captionGeminiModuleRef,
+ captionGeminiProducers,
+ captionGeminiProgramSchema,
+ captionGeminiRequestSchema,
+ captionGeminiTypes,
+} from "./manifest.js";
+export { sealCaptionGeminiPlan, verifyCaptionGeminiPlan } from "./plan.js";
+export { sealCaptionGeminiProgram, verifyCaptionGeminiProgram } from "./program.js";
+export {
+ captionGeminiSystemInstruction,
+ compileCaptionGeminiRequest,
+ verifyCaptionGeminiRequest,
+} from "./request.js";
+export { decodeCaptionGeminiPlannerSurface } from "./surface.js";
+export type * from "./types.js";
diff --git a/packages/caption-gemini/src/manifest.ts b/packages/caption-gemini/src/manifest.ts
new file mode 100644
index 00000000..1643e057
--- /dev/null
+++ b/packages/caption-gemini/src/manifest.ts
@@ -0,0 +1,134 @@
+import { captionManifest, captionModuleRef, captionTypes } from "@svml/caption";
+import { contractTypes, videoContractDependencies } from "@svml/contracts";
+import { digestOf } from "@svml/protocol";
+import type { CapabilityRef, ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";
+
+export const captionGeminiModuleRef = { name: "@svml/caption-gemini", version: "0.0.0-dev" } as const;
+export const captionGeminiTypes = {
+ program: { module: captionGeminiModuleRef, name: "CaptionGeminiProgram" },
+ request: { module: captionGeminiModuleRef, name: "CaptionGeminiRequest" },
+} satisfies Record<string, TypeRef>;
+export const captionGeminiCapabilities = {
+ plan: { module: captionGeminiModuleRef, name: "gemini-caption-planning" },
+} satisfies Record<string, CapabilityRef>;
+export const captionGeminiProducers = {
+ compile: { module: captionGeminiModuleRef, name: "compile-caption-gemini-request" },
+ request: { module: captionGeminiModuleRef, name: "request-caption-gemini-plan" },
+} satisfies Record<string, ProducerRef>;
+export const captionGeminiImplementationDigests = {
+ compile: digestOf("@svml/caption-gemini/compile-request@2"),
+ request: digestOf("@svml/caption-gemini/request-plan@2"),
+ programValidator: digestOf("@svml/caption-gemini/validate-program@2"),
+ requestValidator: digestOf("@svml/caption-gemini/validate-request@2"),
+ plannerSurface: digestOf("@svml/caption-gemini/planner-surface@2"),
+} as const;
+
+const string = { kind: "string", minLength: 1 } as const satisfies ValueSchema;
+const digest = { kind: "string", minLength: 71, maxLength: 71 } as const satisfies ValueSchema;
+const nonNegativeInteger = { kind: "number", integer: true, minimum: 0 } as const satisfies ValueSchema;
+const object = (
+ fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>,
+): ValueSchema => ({ kind: "object", fields });
+const model = { kind: "string", enum: ["gemini-2.5-flash", "gemini-3.1-pro-preview"] } as const satisfies ValueSchema;
+const fieldValueSchema: ValueSchema = {
+ kind: "oneOf",
+ variants: [
+ object({ kind: { schema: { kind: "literal", value: "boolean" } } }),
+ object({
+ kind: { schema: { kind: "literal", value: "enum" } },
+ values: { schema: { kind: "array", minItems: 1, items: string } },
+ }),
+ object({
+ kind: { schema: { kind: "literal", value: "number" } },
+ minimum: { schema: { kind: "number" }, optional: true },
+ maximum: { schema: { kind: "number" }, optional: true },
+ }),
+ ],
+};
+const fieldDeclaration = object({
+ id: { schema: string }, value: { schema: fieldValueSchema }, instruction: { schema: string },
+ minimumPerCue: { schema: nonNegativeInteger }, maximumPerCue: { schema: nonNegativeInteger },
+});
+const planningRun = object({
+ id: { schema: string }, styleId: { schema: string },
+ atomIds: { schema: { kind: "array", minItems: 1, items: string } },
+ cueInstruction: { schema: string }, fields: { schema: { kind: "array", items: fieldDeclaration } },
+});
+export const captionGeminiProgramSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.caption-gemini-program@1" } },
+ model: { schema: model }, programDigest: { schema: digest },
+});
+export const captionGeminiRequestSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.caption-gemini-request@1" } },
+ model: { schema: model }, narrativeDigest: { schema: digest }, captionProgramDigest: { schema: digest },
+ atoms: { schema: { kind: "array", minItems: 1, items: object({ id: { schema: string }, text: { schema: string } }) } },
+ runs: { schema: { kind: "array", minItems: 1, items: planningRun } },
+ systemInstruction: { schema: string }, prompt: { schema: string },
+ temperature: { schema: { kind: "literal", value: 0.2 } }, requestDigest: { schema: digest },
+});
+
+function ownedType(name: string, schema: ValueSchema, digestValue: ReturnType<typeof digestOf>) {
+ return {
+ name,
+ schema,
+ validator: {
+ abi: "svml.type-validator@1" as const,
+ implementation: { kind: "registered" as const, locator: `@svml/caption-gemini/validate-${name}`, digest: digestValue },
+ },
+ };
+}
+
+export const captionGeminiManifest: ModuleManifest = {
+ format: "svml.module@0",
+ name: captionGeminiModuleRef.name,
+ version: captionGeminiModuleRef.version,
+ dependencies: [
+ videoContractDependencies.narrative,
+ { module: captionModuleRef, digest: digestOf(captionManifest) },
+ ],
+ types: [
+ ownedType(captionGeminiTypes.program.name, captionGeminiProgramSchema, captionGeminiImplementationDigests.programValidator),
+ ownedType(captionGeminiTypes.request.name, captionGeminiRequestSchema, captionGeminiImplementationDigests.requestValidator),
+ ],
+ capabilities: [{ name: captionGeminiCapabilities.plan.name, returns: captionTypes.plan }],
+ surfaces: [{
+ name: "planner",
+ tag: "Planner",
+ mode: "structured",
+ outputs: [captionGeminiTypes.program, captionTypes.plan],
+ implementation: {
+ kind: "trusted-frontend-surface",
+ locator: "@svml/caption-gemini/planner-surface",
+ digest: captionGeminiImplementationDigests.plannerSurface,
+ },
+ }],
+ producers: [
+ {
+ name: captionGeminiProducers.compile.name,
+ inputs: [
+ { name: "narrative", type: contractTypes.narrative },
+ { name: "captionProgram", type: captionTypes.program },
+ { name: "program", type: captionGeminiTypes.program },
+ ],
+ outputs: [{ name: "request", type: captionGeminiTypes.request }],
+ needs: [],
+ implementation: {
+ kind: "registered", locator: "@svml/caption-gemini/compile-request", digest: captionGeminiImplementationDigests.compile,
+ },
+ },
+ {
+ name: captionGeminiProducers.request.name,
+ inputs: [{ name: "request", type: captionGeminiTypes.request }],
+ outputs: [],
+ needs: [{
+ name: "plan", capability: captionGeminiCapabilities.plan, returns: captionTypes.plan,
+ affinity: [{ resultPointer: "/planningRequestDigest", input: "request", inputPointer: "/requestDigest" }],
+ }],
+ implementation: {
+ kind: "registered", locator: "@svml/caption-gemini/request-plan", digest: captionGeminiImplementationDigests.request,
+ },
+ },
+ ],
+};
+
+export const captionGeminiManifestDigest = digestOf(captionGeminiManifest);
diff --git a/packages/caption-gemini/src/plan.ts b/packages/caption-gemini/src/plan.ts
new file mode 100644
index 00000000..3ed8b779
--- /dev/null
+++ b/packages/caption-gemini/src/plan.ts
@@ -0,0 +1,126 @@
+import { assertCaptionPlan, sealCaptionPlan } from "@svml/caption";
+import type { CaptionFieldDeclaration, CaptionPlannedCue } from "@svml/caption";
+
+import { verifyCaptionGeminiRequest } from "./request.js";
+import type {
+ CaptionGeminiPlan,
+ CaptionGeminiRequest,
+ RawCaptionGeminiResponse,
+} from "./types.js";
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function object(value: unknown, subject: string): Record<string, unknown> {
+ assert(value !== null && typeof value === "object" && !Array.isArray(value), `${subject} must be an object`);
+ return value as Record<string, unknown>;
+}
+
+function exactKeys(value: Record<string, unknown>, keys: readonly string[], subject: string): void {
+ assert(Object.keys(value).sort().join("\0") === [...keys].sort().join("\0"), `${subject} has unknown or missing fields`);
+}
+
+function fieldValue(declaration: CaptionFieldDeclaration, value: unknown): string {
+ assert(typeof value === "string", `Caption field ${declaration.id} value must be a string`);
+ if (declaration.value.kind === "boolean") {
+ assert(value === "true", `Caption field ${declaration.id} is sparse: absence means false`);
+ } else if (declaration.value.kind === "enum") {
+ assert(declaration.value.values.includes(value), `Caption field ${declaration.id} enum value is invalid`);
+ } else {
+ const number = Number(value);
+ assert(Number.isFinite(number), `Caption field ${declaration.id} number value is invalid`);
+ assert(declaration.value.minimum === undefined || number >= declaration.value.minimum,
+ `Caption field ${declaration.id} value is below its minimum`);
+ assert(declaration.value.maximum === undefined || number <= declaration.value.maximum,
+ `Caption field ${declaration.id} value is above its maximum`);
+ }
+ return value;
+}
+
+function cueFromRaw(
+ raw: unknown,
+ runId: string,
+ cueIndex: number,
+ runAtomIds: readonly string[],
+ cursor: number,
+ declarations: ReadonlyMap<string, CaptionFieldDeclaration>,
+): { readonly cue: CaptionPlannedCue; readonly nextCursor: number } {
+ const value = object(raw, `Caption cue ${runId}/${cueIndex + 1}`);
+ exactKeys(value, ["after_atom_id", "fields"], `Caption cue ${runId}/${cueIndex + 1}`);
+ assert(typeof value.after_atom_id === "string", `Caption cue ${runId}/${cueIndex + 1} endpoint is invalid`);
+ const endpoint = runAtomIds.indexOf(value.after_atom_id, cursor);
+ assert(endpoint >= cursor, `Caption cue ${runId}/${cueIndex + 1} endpoint is outside its remaining run`);
+ const atomIds = runAtomIds.slice(cursor, endpoint + 1);
+ assert(Array.isArray(value.fields), `Caption cue ${runId}/${cueIndex + 1} fields are invalid`);
+ const assignments = value.fields.map((rawField, fieldIndex) => {
+ const field = object(rawField, `Caption field ${runId}/${cueIndex + 1}/${fieldIndex + 1}`);
+ exactKeys(field, ["declaration_id", "atom_id", "value"], "Caption field assignment");
+ assert(typeof field.declaration_id === "string", "Caption field declaration id is invalid");
+ const declaration = declarations.get(field.declaration_id);
+ assert(declaration !== undefined, `Caption field ${field.declaration_id} was not declared for run ${runId}`);
+ assert(typeof field.atom_id === "string" && atomIds.includes(field.atom_id),
+ `Caption field ${declaration.id} atom lies outside its cue`);
+ return {
+ declarationId: declaration.id,
+ atomId: field.atom_id,
+ value: fieldValue(declaration, field.value),
+ };
+ });
+ const unique = new Set(assignments.map((field) => `${field.declarationId}\0${field.atomId}`));
+ assert(unique.size === assignments.length, `Caption cue ${runId}/${cueIndex + 1} repeats a field assignment`);
+ for (const declaration of declarations.values()) {
+ const count = assignments.filter((field) => field.declarationId === declaration.id).length;
+ assert(count >= declaration.minimumPerCue && count <= declaration.maximumPerCue,
+ `Caption field ${declaration.id} count ${count} violates ${declaration.minimumPerCue}..${declaration.maximumPerCue} per cue`);
+ }
+ return {
+ cue: { id: `${runId}:cue:${cueIndex + 1}`, atomIds, fields: assignments },
+ nextCursor: endpoint + 1,
+ };
+}
+
+export function sealCaptionGeminiPlan(
+ request: CaptionGeminiRequest,
+ response: unknown,
+): CaptionGeminiPlan {
+ verifyCaptionGeminiRequest(request);
+ const root = object(response, "Caption Gemini response");
+ exactKeys(root, ["runs"], "Caption Gemini response");
+ assert(Array.isArray(root.runs), "Caption Gemini response runs are invalid");
+ const rawRuns = new Map<string, Record<string, unknown>>();
+ for (const rawRun of root.runs) {
+ const run = object(rawRun, "Caption Gemini response run");
+ exactKeys(run, ["run_id", "cues"], "Caption Gemini response run");
+ assert(typeof run.run_id === "string" && !rawRuns.has(run.run_id), "Caption Gemini response run id is invalid or repeated");
+ rawRuns.set(run.run_id, run);
+ }
+ assert(rawRuns.size === request.runs.length, "Caption Gemini response does not contain the exact run set");
+ const runs = request.runs.map((expectedRun) => {
+ const rawRun = rawRuns.get(expectedRun.id);
+ assert(rawRun !== undefined && Array.isArray(rawRun.cues) && rawRun.cues.length > 0,
+ `Caption run ${expectedRun.id} has no Cues`);
+ const declarations = new Map(expectedRun.fields.map((field) => [field.id, field]));
+ let cursor = 0;
+ const cues = rawRun.cues.map((rawCue, cueIndex) => {
+ const planned = cueFromRaw(rawCue, expectedRun.id, cueIndex, expectedRun.atomIds, cursor, declarations);
+ cursor = planned.nextCursor;
+ return planned.cue;
+ });
+ assert(cursor === expectedRun.atomIds.length, `Caption run ${expectedRun.id} Cues do not cover its final atom`);
+ return { id: expectedRun.id, styleId: expectedRun.styleId, cues };
+ });
+ return sealCaptionPlan({
+ contract: "svml.caption-plan@1",
+ narrativeDigest: request.narrativeDigest,
+ captionProgramDigest: request.captionProgramDigest,
+ planningRequestDigest: request.requestDigest,
+ runs,
+ });
+}
+
+export function verifyCaptionGeminiPlan(value: unknown): asserts value is CaptionGeminiPlan {
+ assertCaptionPlan(value);
+}
+
+export const rawCaptionGeminiResponseShape = null as unknown as RawCaptionGeminiResponse;
diff --git a/packages/caption-gemini/src/program.ts b/packages/caption-gemini/src/program.ts
new file mode 100644
index 00000000..b013e966
--- /dev/null
+++ b/packages/caption-gemini/src/program.ts
@@ -0,0 +1,34 @@
+import { digestOf, isDigest } from "@svml/protocol";
+
+import type { CaptionGeminiProgram } from "./types.js";
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function content(program: Omit<CaptionGeminiProgram, "programDigest">) {
+ return {
+ contract: "svml.caption-gemini-program@1" as const,
+ model: program.model,
+ };
+}
+
+export function verifyCaptionGeminiProgram(value: unknown): asserts value is CaptionGeminiProgram {
+ assert(value !== null && typeof value === "object" && !Array.isArray(value), "Caption Gemini Program must be an object");
+ const program = value as CaptionGeminiProgram;
+ assert(program.contract === "svml.caption-gemini-program@1", "Caption Gemini Program contract is invalid");
+ assert(program.model === "gemini-2.5-flash" || program.model === "gemini-3.1-pro-preview",
+ "Caption Gemini model is unsupported");
+ assert(isDigest(program.programDigest), "Caption Gemini Program digest is invalid");
+ const { programDigest: _digest, ...withoutDigest } = program;
+ assert(program.programDigest === digestOf(content(withoutDigest)), "Caption Gemini Program digest does not match its contents");
+}
+
+export function sealCaptionGeminiProgram(
+ value: Omit<CaptionGeminiProgram, "programDigest">,
+): CaptionGeminiProgram {
+ const normalized = content(value);
+ const program = { ...normalized, programDigest: digestOf(normalized) };
+ verifyCaptionGeminiProgram(program);
+ return program;
+}
diff --git a/packages/caption-gemini/src/request.ts b/packages/caption-gemini/src/request.ts
new file mode 100644
index 00000000..9c2cf11d
--- /dev/null
+++ b/packages/caption-gemini/src/request.ts
@@ -0,0 +1,136 @@
+import { assertCaptionProgramForNarrative } from "@svml/caption";
+import type { CaptionFieldDeclaration, CaptionProgram } from "@svml/caption";
+import type { Narrative } from "@svml/contracts";
+import { digestOf, isDigest } from "@svml/protocol";
+
+import { verifyCaptionGeminiProgram } from "./program.js";
+import type {
+ CaptionGeminiProgram,
+ CaptionGeminiRequest,
+ CaptionPlanningRun,
+} from "./types.js";
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function fieldInstruction(field: CaptionFieldDeclaration): string {
+ const value = field.value.kind === "boolean"
+ ? 'the string "true"; omit the assignment when the field does not apply'
+ : field.value.kind === "enum"
+ ? `one of ${field.value.values.map((item) => JSON.stringify(item)).join(", ")}`
+ : "a finite base-10 number encoded as a string";
+ return `${field.id}: ${field.minimumPerCue}..${field.maximumPerCue} atom assignments per cue; value=${value}. ${field.instruction}`;
+}
+
+export function captionGeminiSystemInstruction(runs: readonly CaptionPlanningRun[]): string {
+ const styleRules = new Map<string, CaptionPlanningRun>();
+ runs.forEach((run) => styleRules.set(run.styleId, run));
+ return [
+ "You are a caption cue and per-word attribute planner. Display atoms are immutable author truth.",
+ "Return only response-schema JSON. Never return corrected text, replacement text, pronunciation text, timestamps, Markdown, or explanations.",
+ "Each run is a hard boundary. Plan every run independently; a cue may never cross a run boundary.",
+ "Within each run, cues must preserve and exactly partition atom_ids. Return only each cue's final after_atom_id; the final cue must end at the run's final atom.",
+ "Every field assignment targets exactly one atom inside its cue and uses a field declared by that run's style.",
+ ...[...styleRules.values()].flatMap((run) => [
+ `Style ${run.styleId} cue judgment: ${run.cueInstruction.trim()}`,
+ ...run.fields.map((field) => `Style ${run.styleId} field ${fieldInstruction(field)}`),
+ ]),
+ "Echo every run exactly once. Use an empty fields array when no declaration applies. Never invent ids.",
+ ].join("\n");
+}
+
+function promptContent(request: Pick<CaptionGeminiRequest, "atoms" | "runs">): string {
+ return JSON.stringify({
+ display_atoms: request.atoms,
+ runs: request.runs.map((run) => ({
+ run_id: run.id,
+ style_id: run.styleId,
+ atom_ids: run.atomIds,
+ cue_instruction: run.cueInstruction,
+ field_declarations: run.fields,
+ })),
+ }, null, 2);
+}
+
+function requestContent(value: Omit<CaptionGeminiRequest, "requestDigest">) {
+ return {
+ contract: "svml.caption-gemini-request@1" as const,
+ model: value.model,
+ narrativeDigest: value.narrativeDigest,
+ captionProgramDigest: value.captionProgramDigest,
+ atoms: value.atoms.map((atom) => ({ ...atom })),
+ runs: value.runs.map((run) => ({
+ id: run.id,
+ styleId: run.styleId,
+ atomIds: [...run.atomIds],
+ cueInstruction: run.cueInstruction.trim(),
+ fields: run.fields.map((field) => structuredClone(field)),
+ })),
+ systemInstruction: value.systemInstruction,
+ prompt: value.prompt,
+ temperature: 0.2 as const,
+ };
+}
+
+export function compileCaptionGeminiRequest(
+ narrative: Narrative,
+ captionProgram: CaptionProgram,
+ program: CaptionGeminiProgram,
+): CaptionGeminiRequest {
+ assertCaptionProgramForNarrative(captionProgram, narrative);
+ verifyCaptionGeminiProgram(program);
+ const styles = new Map(captionProgram.styles.map((style) => [style.id, style]));
+ const runs = captionProgram.runs.map((run) => {
+ const style = styles.get(run.styleId);
+ assert(style !== undefined, `Caption run ${run.id} references an unknown Style`);
+ return {
+ id: run.id,
+ styleId: run.styleId,
+ atomIds: [...run.atomIds],
+ cueInstruction: style.planning.cueInstruction,
+ fields: style.planning.fields.map((field) => structuredClone(field)),
+ };
+ });
+ const base = {
+ contract: "svml.caption-gemini-request@1" as const,
+ model: program.model,
+ narrativeDigest: captionProgram.narrativeDigest,
+ captionProgramDigest: captionProgram.digest,
+ atoms: captionProgram.atoms.map((atom) => ({ id: atom.id, text: atom.text })),
+ runs,
+ };
+ const systemInstruction = captionGeminiSystemInstruction(runs);
+ const prompt = promptContent(base);
+ const content = requestContent({ ...base, systemInstruction, prompt, temperature: 0.2 });
+ const request = { ...content, requestDigest: digestOf(content) };
+ verifyCaptionGeminiRequest(request);
+ return request;
+}
+
+export function verifyCaptionGeminiRequest(value: unknown): asserts value is CaptionGeminiRequest {
+ assert(value !== null && typeof value === "object" && !Array.isArray(value), "Caption Gemini request must be an object");
+ const request = value as CaptionGeminiRequest;
+ assert(request.contract === "svml.caption-gemini-request@1", "Caption Gemini request contract is invalid");
+ assert(request.model === "gemini-2.5-flash" || request.model === "gemini-3.1-pro-preview", "Caption Gemini request model is unsupported");
+ assert(isDigest(request.narrativeDigest) && isDigest(request.captionProgramDigest) && isDigest(request.requestDigest),
+ "Caption Gemini request identity is invalid");
+ assert(request.temperature === 0.2, "Caption Gemini temperature is not the package-owned value");
+ assert(Array.isArray(request.atoms) && request.atoms.length > 0, "Caption Gemini request atoms are empty");
+ const atomIds = request.atoms.map((atom) => atom.id);
+ assert(atomIds.every((id) => typeof id === "string" && id.length > 0) && new Set(atomIds).size === atomIds.length,
+ "Caption Gemini atom ids are invalid or repeated");
+ assert(request.atoms.every((atom) => typeof atom.text === "string" && atom.text.length > 0),
+ "Caption Gemini display atom text is invalid");
+ const atomSet = new Set(atomIds);
+ const planned = request.runs.flatMap((run) => run.atomIds);
+ assert(request.runs.length > 0 && planned.length === atomIds.length && new Set(planned).size === atomIds.length
+ && planned.every((id) => atomSet.has(id)), "Caption Gemini runs are not an exact display-atom partition");
+ assert(request.runs.every((run) => run.atomIds.length > 0 && run.cueInstruction.trim().length > 0),
+ "Caption Gemini request contains an empty run");
+ assert(request.systemInstruction === captionGeminiSystemInstruction(request.runs),
+ "Caption Gemini system instruction differs from the model package");
+ assert(request.prompt === promptContent(request), "Caption Gemini prompt differs from immutable display facts");
+ const { requestDigest: _digest, ...withoutDigest } = request;
+ assert(request.requestDigest === digestOf(requestContent(withoutDigest)), "Caption Gemini request digest does not match its contents");
+}
diff --git a/packages/caption-gemini/src/surface.ts b/packages/caption-gemini/src/surface.ts
new file mode 100644
index 00000000..3f21d021
--- /dev/null
+++ b/packages/caption-gemini/src/surface.ts
@@ -0,0 +1,68 @@
+import { captionTypes } from "@svml/caption";
+import { contractTypes } from "@svml/contracts";
+import type {
+ StructuredSurfaceHandler,
+ SurfaceResolvedReference,
+ TextAttributeValue,
+} from "@svml/text";
+
+import { captionGeminiPlanningFragment } from "./fragment.js";
+import { captionGeminiTypes } from "./manifest.js";
+import { sealCaptionGeminiProgram } from "./program.js";
+import type { CaptionGeminiModel } from "./types.js";
+
+function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+}
+
+function reference(
+ element: Parameters<StructuredSurfaceHandler>[0]["element"],
+ name: string,
+ expected: SurfaceResolvedReference["type"],
+ resolve: (path: string) => SurfaceResolvedReference | undefined,
+): SurfaceResolvedReference {
+ const raw: TextAttributeValue | undefined = element.attributes[name];
+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a whole-value reference`);
+ const value = resolve(raw.path);
+ if (value === undefined || !sameType(value.type, expected)) throw new Error(`${element.name}.${name} cannot resolve the required type`);
+ return value;
+}
+
+export const decodeCaptionGeminiPlannerSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ const required = ["id", "narrative", "program", "model"];
+ if (required.some((name) => element.attributes[name] === undefined)
+ || Object.keys(element.attributes).some((name) => !required.includes(name))) {
+ throw new Error(`${element.name} requires ${required.join(", ")}`);
+ }
+ if (element.children.some((child) => child.kind === "element" || child.value.trim())) {
+ throw new Error(`${element.name} does not accept children`);
+ }
+ const id = element.attributes.id;
+ const model = element.attributes.model;
+ if (typeof id !== "string" || !id.trim()) throw new Error(`${element.name}.id must be a non-empty string`);
+ if (model !== "gemini-2.5-flash" && model !== "gemini-3.1-pro-preview") {
+ throw new Error(`${element.name}.model is unsupported`);
+ }
+ const narrative = reference(element, "narrative", contractTypes.narrative, resolveReference);
+ const captionProgram = reference(element, "program", captionTypes.program, resolveReference);
+ const optionsId = `${id}.gemini`;
+ const options = sealCaptionGeminiProgram({
+ contract: "svml.caption-gemini-program@1",
+ model: model as CaptionGeminiModel,
+ });
+ return {
+ records: [{ id: optionsId, type: captionGeminiTypes.program, value: { kind: "inline", value: options }, range: element.range }],
+ components: [{
+ id,
+ fragment: captionGeminiPlanningFragment.id,
+ inputs: {
+ narrative: narrative.ref,
+ captionProgram: captionProgram.ref,
+ program: { kind: "record", id: optionsId },
+ },
+ outputs: { plan: `${id}.plan` },
+ range: element.range,
+ }],
+ fragments: [captionGeminiPlanningFragment],
+ };
+};
diff --git a/packages/caption-gemini/src/types.ts b/packages/caption-gemini/src/types.ts
new file mode 100644
index 00000000..fdc13a3c
--- /dev/null
+++ b/packages/caption-gemini/src/types.ts
@@ -0,0 +1,69 @@
+import type {
+ CaptionFieldAssignment,
+ CaptionFieldDeclaration,
+ CaptionFieldValueSchema,
+ CaptionPlan,
+ CaptionPlannedCue,
+ CaptionPlannedRun,
+ CaptionProgram,
+} from "@svml/caption";
+import type { Digest } from "@svml/protocol";
+
+export type {
+ CaptionFieldAssignment,
+ CaptionFieldDeclaration,
+ CaptionFieldValueSchema,
+ CaptionPlannedCue,
+ CaptionPlannedRun,
+ CaptionProgram,
+};
+export type CaptionGeminiPlan = CaptionPlan;
+
+export type CaptionGeminiModel = "gemini-2.5-flash" | "gemini-3.1-pro-preview";
+
+/** Explicit model choice. Cue and field meaning comes from the Caption Program, not the Provider. */
+export type CaptionGeminiProgram = {
+ readonly contract: "svml.caption-gemini-program@1";
+ readonly model: CaptionGeminiModel;
+ readonly programDigest: Digest;
+};
+
+export type CaptionPlanningAtom = {
+ readonly id: string;
+ readonly text: string;
+};
+
+export type CaptionPlanningRun = {
+ readonly id: string;
+ readonly styleId: string;
+ readonly atomIds: readonly string[];
+ readonly cueInstruction: string;
+ readonly fields: readonly CaptionFieldDeclaration[];
+};
+
+export type CaptionGeminiRequest = {
+ readonly contract: "svml.caption-gemini-request@1";
+ readonly model: CaptionGeminiModel;
+ readonly narrativeDigest: Digest;
+ readonly captionProgramDigest: Digest;
+ readonly atoms: readonly CaptionPlanningAtom[];
+ readonly runs: readonly CaptionPlanningRun[];
+ readonly systemInstruction: string;
+ readonly prompt: string;
+ readonly temperature: 0.2;
+ readonly requestDigest: Digest;
+};
+
+export type RawCaptionGeminiResponse = {
+ readonly runs: readonly {
+ readonly run_id: string;
+ readonly cues: readonly {
+ readonly after_atom_id: string;
+ readonly fields: readonly {
+ readonly declaration_id: string;
+ readonly atom_id: string;
+ readonly value: string;
+ }[];
+ }[];
+ }[];
+};
diff --git a/packages/caption-gemini/test/planning.test.ts b/packages/caption-gemini/test/planning.test.ts
new file mode 100644
index 00000000..3c02650e
--- /dev/null
+++ b/packages/caption-gemini/test/planning.test.ts
@@ -0,0 +1,173 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import {
+ defaultCaptionTrackProgram,
+ resolveCaptionProgram,
+ sealCaptionProgram,
+ sealCaptionStyle,
+} from "@svml/caption";
+import type { CaptionStyleIntent } from "@svml/caption";
+import {
+ captionGeminiComponent,
+ captionGeminiManifest,
+ compileCaptionGeminiRequest,
+ sealCaptionGeminiPlan,
+ sealCaptionGeminiProgram,
+} from "@svml/caption-gemini";
+import type { CaptionGeminiRequest, RawCaptionGeminiResponse } from "@svml/caption-gemini";
+import { parseScript } from "@svml/script";
+
+function style(id: string, fields: CaptionStyleIntent["planning"]["fields"] = []) {
+ const base = defaultCaptionTrackProgram(id);
+ return sealCaptionStyle({
+ contract: "svml.caption-style@1",
+ id,
+ planning: {
+ cueInstruction: "Prefer short complete semantic phrases and never cross clause punctuation.",
+ fields,
+ },
+ presentation: { mode: "whole", stackingOrder: base.stacking.order, style: base.style },
+ });
+}
+
+function options() {
+ return sealCaptionGeminiProgram({
+ contract: "svml.caption-gemini-program@1",
+ model: "gemini-2.5-flash",
+ });
+}
+
+function validResponse(request: CaptionGeminiRequest): RawCaptionGeminiResponse {
+ return {
+ runs: request.runs.map((run) => ({
+ run_id: run.id,
+ cues: [{
+ after_atom_id: run.atomIds.at(-1)!,
+ fields: run.fields.length === 0 ? [] : [{
+ declaration_id: run.fields[0]!.id,
+ atom_id: run.atomIds[0]!,
+ value: "true",
+ }],
+ }],
+ })),
+ };
+}
+
+test("Gemini sees only immutable display atoms while Role and roleless runs come from Caption Program", () => {
+ const narrative = parseScript("roles.svml", `
+ <opening>Meet <SVML | semantic video markup language>.</opening>
+ <answer><ALICE>Meaning becomes the source.</answer>
+ `);
+ const normal = style("normal");
+ const alice = style("alice", [{
+ id: "important",
+ value: { kind: "boolean" },
+ instruction: "Select zero or one word whose emphasis best communicates this Cue.",
+ minimumPerCue: 0,
+ maximumPerCue: 1,
+ }]);
+ const program = resolveCaptionProgram(narrative, "captions", normal, [{
+ id: "alice-use",
+ selector: { kind: "role", role: "ALICE" },
+ style: alice,
+ }]);
+ const request = compileCaptionGeminiRequest(narrative, program, options());
+
+ assert.deepEqual(request.runs.map((run) => run.styleId), ["normal", "alice"]);
+ assert.equal(request.atoms.some((atom) => atom.text === "SVML"), true);
+ assert.equal(request.atoms.some((atom) => atom.text === "semantic"), false);
+ assert.equal(request.prompt.includes("semantic video markup language"), false);
+ assert.equal(request.prompt.includes("pronunciation"), false);
+ assert.equal("whisperx" in request, false);
+ assert.equal("transcript" in request, false);
+
+ const { digest: _digest, ...content } = program;
+ const tampered = sealCaptionProgram({
+ ...content,
+ atoms: program.atoms.map((atom, index) => index === 0 ? { ...atom, text: "TAMPERED" } : atom),
+ });
+ assert.throws(() => compileCaptionGeminiRequest(narrative, tampered, options()),
+ /display atoms differ from its Narrative/u);
+});
+
+test("Gemini may only cut each resolved run and assign declared attributes to atoms inside its Cue", () => {
+ const narrative = parseScript("plan.svml", "<line><ALICE>Meaning becomes the source.</line>");
+ const emphasis = style("emphasis", [{
+ id: "important",
+ value: { kind: "boolean" },
+ instruction: "Select exactly one important word.",
+ minimumPerCue: 1,
+ maximumPerCue: 1,
+ }]);
+ const program = resolveCaptionProgram(narrative, "captions", emphasis, []);
+ const request = compileCaptionGeminiRequest(narrative, program, options());
+ const plan = sealCaptionGeminiPlan(request, validResponse(request));
+ assert.equal(plan.captionProgramDigest, program.digest);
+ assert.deepEqual(
+ plan.runs.flatMap((run) => run.cues.flatMap((cue) => cue.atomIds)),
+ request.runs.flatMap((run) => run.atomIds),
+ );
+ assert.equal(plan.runs[0]?.cues[0]?.fields[0]?.declarationId, "important");
+
+ assert.throws(
+ () => sealCaptionGeminiPlan(request, { ...validResponse(request), corrected_text: "tampered" }),
+ /unknown or missing fields/u,
+ );
+ const outside = structuredClone(validResponse(request));
+ const field = outside.runs[0]!.cues[0]!.fields[0]! as { atom_id: string };
+ field.atom_id = "foreign";
+ assert.throws(() => sealCaptionGeminiPlan(request, outside), /outside its cue/u);
+});
+
+test("one atom may carry multiple independent fields and fields need not form contiguous spans", () => {
+ const narrative = parseScript("multi-field.svml", "<line>one two three four five.</line>");
+ const multi = style("multi", [
+ {
+ id: "best",
+ value: { kind: "boolean" },
+ instruction: "Select up to two best words.",
+ minimumPerCue: 0,
+ maximumPerCue: 2,
+ },
+ {
+ id: "medium",
+ value: { kind: "boolean" },
+ instruction: "Independently select up to two medium words.",
+ minimumPerCue: 0,
+ maximumPerCue: 2,
+ },
+ ]);
+ const program = resolveCaptionProgram(narrative, "captions", multi, []);
+ const request = compileCaptionGeminiRequest(narrative, program, options());
+ const atoms = request.runs[0]!.atomIds;
+ const plan = sealCaptionGeminiPlan(request, {
+ runs: [{
+ run_id: request.runs[0]!.id,
+ cues: [{
+ after_atom_id: atoms.at(-1)!,
+ fields: [
+ { declaration_id: "best", atom_id: atoms[0]!, value: "true" },
+ { declaration_id: "best", atom_id: atoms[2]!, value: "true" },
+ { declaration_id: "medium", atom_id: atoms[1]!, value: "true" },
+ { declaration_id: "medium", atom_id: atoms[2]!, value: "true" },
+ ],
+ }],
+ }],
+ });
+ const fields = plan.runs[0]!.cues[0]!.fields;
+ assert.equal(fields.filter((field) => field.atomId === atoms[2]).length, 2);
+ assert.deepEqual(fields.filter((field) => field.declarationId === "best").map((field) => field.atomId),
+ [atoms[0], atoms[2]]);
+});
+
+test("the package enumerates every Manifest Producer and validator", () => {
+ assert.deepEqual(
+ captionGeminiComponent.producers?.map((facet) => [facet.producer.name, facet.implementationDigest]),
+ captionGeminiManifest.producers.map((producer) => [producer.name, producer.implementation.digest]),
+ );
+ assert.deepEqual(
+ captionGeminiComponent.validators?.map((facet) => [facet.type.name, facet.implementationDigest]),
+ captionGeminiManifest.types.map((type) => [type.name, type.validator?.implementation.digest]),
+ );
+});
diff --git a/packages/caption/README.md b/packages/caption/README.md
index dc21afbb..17d77786 100644
--- a/packages/caption/README.md
+++ b/packages/caption/README.md
@@ -1,24 +1,58 @@
# `@svml/caption`
-Pure composition between the common Narrative's untimed Caption Projection and a CompleteSemanticMap.
-The authored display region receives the evidence-backed envelope of the speech tokens it owns.
-Exact internal correspondences may reuse evidence timing; unresolved display words remain untimed
-until a track explicitly chooses a local presentation policy.
+Provider-neutral Caption planning, timing projection and lowering to an ordinary peer
+`VisualTrack`.
-`planCaptionPresentation()` is that optional track-local policy. `whole` preserves each authored
-region; `proportional-word` and `character-flow` create visibly labelled estimates. They never
-write those estimates back into the global speech map or Caption Projection.
+Caption has two independent inputs:
-`captionComponent` exposes two enumerable deterministic Producers:
+- `Narrative.captionProjection` is the immutable display truth. Only the left side of Dual Text is
+ visible here; `<SVML | semantic video markup language>` contributes the display atom `SVML`, not
+ its pronunciation words.
+- `CaptionProgram` is the complete author intent for planning and presentation. It assigns exactly
+ one full `CaptionStyle` to every visible display atom.
-- `temporalize-caption`: `Narrative + CompleteSemanticMap -> TimedCaptionProjection`;
-- `render-caption-track`: `TimedCaptionProjection + CaptionTrackProgram -> VisualTrack`.
+A Style contains both its planning requirements and its complete visual appearance. The Program
+names one default Style and then applies ordered whole-Style replacements by Role or explicit
+Script Selection. Later matching applications win. Consequently roleless prose and every
+unmentioned word receive the default automatically, while changing one sentence needs only one
+Selection and one `Use`; the author never writes the complement.
-It also owns and validates both Caption-specific Types. Their validator identities and both
-Producer implementations are checked against `captionManifest` and enter the installed package
-lock. The package has no Provider, LLM, queue, credential or Core authority.
+```xml
+<caption:Style id="plain" appearance={studio.caption.plain}>
+ <caption:Cues>Use complete phrases of two to five words.</caption:Cues>
+</caption:Style>
-The current `CaptionTrackProgram` is an executable vertical-slice program, not the frozen public
-style language. Exact font Artifacts, the complete production positioning/style matrix, role
-overrides, package-owned SVS Recipes, Gemini cue grouping and the final author Surface require a
-separate capability audit before compatibility freeze.
+<caption:Style id="impact" appearance={studio.caption.impact}>
+ <caption:Cues>Use complete phrases of two to seven words.</caption:Cues>
+ <caption:Field id="important" type="boolean" min-per-cue="0" max-per-cue="2">
+ Select at most two words whose emphasis best communicates this Cue.
+ </caption:Field>
+</caption:Style>
+
+<caption:Program id="captions" narrative={story} default={plain}>
+ <caption:Use role="ALICE" style={impact}/>
+ <caption:Use on={story.selection.special} style={plain}/>
+</caption:Program>
+```
+
+`Use` order is semantic. The second rule above replaces the entire Style for `special`; it does not
+merge arbitrary fields from two styles. A Role Cue is optional in Script, and Role state never
+crosses a Segment boundary.
+
+The planner-neutral `CaptionPlan` has only two freedoms:
+
+1. partition each already-resolved Program run into ordered Cues;
+2. assign zero, one or more declared attributes to each display atom.
+
+It cannot rewrite text, change Style assignment or invent time. `temporalize-caption-plan` later
+joins those atom identities with the independent `CompleteSemanticMap`. Exact display/speech
+correspondence reuses measured timing. A display alias without word-level audio evidence receives
+an explicitly estimated local projection and never contaminates the global speech map.
+
+The final `render-caption-program` Producer lowers the timed result and each selected full Style to
+one self-contained `VisualTrack`. Caption is not a privileged Composition layer: Film consumes it
+exactly like Speech, B-roll or Text Track output.
+
+The package owns deterministic Types, validators and Producers only. It contains no LLM, Provider,
+credential, queue or Core authority. A planning package such as `@svml/caption-gemini` may fulfill
+the narrow `CaptionPlan` contract; presentation and timing remain here.
diff --git a/packages/caption/package.json b/packages/caption/package.json
index c8d509a3..e3f6747e 100644
--- a/packages/caption/package.json
+++ b/packages/caption/package.json
@@ -9,7 +9,10 @@
"dependencies": {
"@svml/component-kit": "workspace:*",
"@svml/contracts": "workspace:*",
- "@svml/protocol": "workspace:*"
+ "@svml/elaborator": "workspace:*",
+ "@svml/protocol": "workspace:*",
+ "@svml/svs": "workspace:*",
+ "@svml/text": "workspace:*"
},
"devDependencies": {
"@svml/script": "workspace:*",
diff --git a/packages/caption/src/component.ts b/packages/caption/src/component.ts
index eda6cd21..bfef4207 100644
--- a/packages/caption/src/component.ts
+++ b/packages/caption/src/component.ts
@@ -8,18 +8,26 @@ import { canonicalize } from "@svml/protocol";
import {
captionImplementationDigest,
+ captionPlanImplementationDigest,
captionProducers,
captionTypes,
captionValidatorDigests,
} from "./manifest.js";
-import { temporalizeCaption } from "./temporalize.js";
+import { assertCaptionPlan } from "./plan.js";
+import { assertCaptionProgram, assertCaptionStyle } from "./style.js";
+import { temporalizeCaption, temporalizeCaptionPlan } from "./temporalize.js";
import {
assertCaptionTrackProgram,
assertTimedCaptionProjection,
+ renderCaptionProgram,
renderCaptionTrack,
+ renderCaptionProgramImplementationDigest,
renderCaptionTrackImplementationDigest,
} from "./track.js";
import type {
+ CaptionPlan,
+ CaptionProgram,
+ CaptionStyleIntent,
CaptionTrackProgram,
TimedCaptionProjection,
} from "./types.js";
@@ -49,6 +57,40 @@ export const captionComponent = {
needs: {},
}),
},
+ {
+ producer: captionProducers.temporalizePlan,
+ implementationDigest: captionPlanImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: {
+ caption: {
+ kind: "inline",
+ value: canonicalize(temporalizeCaptionPlan(
+ inline<Narrative>(inputs.narrative?.value, "Narrative"),
+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),
+ inline<CaptionProgram>(inputs.program?.value, "CaptionProgram"),
+ inline<CaptionPlan>(inputs.plan?.value, "CaptionPlan"),
+ )),
+ },
+ },
+ needs: {},
+ }),
+ },
+ {
+ producer: captionProducers.renderProgram,
+ implementationDigest: renderCaptionProgramImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: {
+ track: {
+ kind: "inline",
+ value: canonicalize(renderCaptionProgram(
+ inline<TimedCaptionProjection>(inputs.caption?.value, "TimedCaptionProjection"),
+ inline<CaptionProgram>(inputs.program?.value, "CaptionProgram"),
+ )),
+ },
+ },
+ needs: {},
+ }),
+ },
{
producer: captionProducers.renderTrack,
implementationDigest: renderCaptionTrackImplementationDigest,
@@ -67,6 +109,21 @@ export const captionComponent = {
},
],
validators: [
+ {
+ type: captionTypes.style,
+ implementationDigest: captionValidatorDigests.style,
+ handler: ({ value }) => assertCaptionStyle(inline<CaptionStyleIntent>(value, "CaptionStyle")),
+ },
+ {
+ type: captionTypes.program,
+ implementationDigest: captionValidatorDigests.program,
+ handler: ({ value }) => assertCaptionProgram(inline<CaptionProgram>(value, "CaptionProgram")),
+ },
+ {
+ type: captionTypes.plan,
+ implementationDigest: captionValidatorDigests.plan,
+ handler: ({ value }) => assertCaptionPlan(inline(value, "CaptionPlan")),
+ },
{
type: captionTypes.timedProjection,
implementationDigest: captionValidatorDigests.timedProjection,
diff --git a/packages/caption/src/display.ts b/packages/caption/src/display.ts
new file mode 100644
index 00000000..d02c635e
--- /dev/null
+++ b/packages/caption/src/display.ts
@@ -0,0 +1,106 @@
+import type {
+ CaptionRegion,
+ Narrative,
+ NarrativeSelectionRef,
+} from "@svml/contracts";
+
+import type { CaptionDisplayAtom } from "./types.js";
+
+const WORD =
+ /[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}]\p{M}*|[\p{L}\p{M}\p{N}]+(?:['’.-][\p{L}\p{M}\p{N}]+)*/gu;
+
+function displayWords(value: string): Array<{ readonly text: string; readonly start: number; readonly end: number }> {
+ const result: Array<{ text: string; start: number; end: number }> = [];
+ WORD.lastIndex = 0;
+ let match: RegExpExecArray | null;
+ while ((match = WORD.exec(value))) {
+ result.push({ text: match[0], start: match.index, end: match.index + match[0].length });
+ }
+ return result;
+}
+
+function turnForRegion(narrative: Narrative, region: CaptionRegion): Narrative["turns"][number] {
+ const turn = narrative.turns.find((candidate) =>
+ candidate.segmentId === region.segmentId
+ && candidate.tokenStart < region.endTokenExclusive
+ && candidate.tokenEndExclusive > region.startToken);
+ if (turn === undefined) throw new Error(`Caption region ${region.id} is not owned by a Narrative Turn`);
+ return turn;
+}
+
+/** The only word universe visible to Caption planning. Speech-side alias text never enters it. */
+export function captionDisplayAtoms(narrative: Narrative): CaptionDisplayAtom[] {
+ const atoms: CaptionDisplayAtom[] = [];
+ for (const region of narrative.captionProjection.regions) {
+ if (region.kind === "hidden") continue;
+ const turn = turnForRegion(narrative, region);
+ for (const word of displayWords(region.display)) {
+ const exact = region.refinements.find((refinement) =>
+ refinement.displayStart === word.start && refinement.displayEnd === word.end);
+ atoms.push({
+ id: `${region.id}:display:${atoms.length + 1}`,
+ index: atoms.length,
+ regionId: region.id,
+ segmentId: region.segmentId,
+ turnId: turn.id,
+ ...(turn.role === undefined ? {} : { role: turn.role }),
+ text: word.text,
+ displayStart: word.start,
+ displayEnd: word.end,
+ sourceTokenStart: exact?.startToken ?? region.startToken,
+ sourceTokenEndExclusive: exact?.endTokenExclusive ?? region.endTokenExclusive,
+ correspondence: exact === undefined ? "region-envelope" : "exact",
+ });
+ }
+ }
+ if (atoms.length === 0) throw new Error("Caption display projection contains no visible words");
+ return atoms;
+}
+
+function occurrenceRange(
+ occurrence: NarrativeSelectionRef["occurrences"][number],
+ tokenCount: number,
+): { readonly start: number; readonly endExclusive: number } {
+ const start = occurrence.open.affinity === "left"
+ ? Math.max(0, occurrence.open.boundary.tokenIndex - 1)
+ : occurrence.open.boundary.tokenIndex;
+ const endExclusive = occurrence.close.affinity === "right"
+ ? Math.min(tokenCount, occurrence.close.boundary.tokenIndex + 1)
+ : occurrence.close.boundary.tokenIndex;
+ return { start, endExclusive };
+}
+
+export function displayAtomMatchesSelection(
+ atom: CaptionDisplayAtom,
+ selection: NarrativeSelectionRef,
+ tokenCount: number,
+): boolean {
+ return selection.occurrences.some((occurrence) => {
+ const range = occurrenceRange(occurrence, tokenCount);
+ return atom.sourceTokenStart < range.endExclusive && atom.sourceTokenEndExclusive > range.start;
+ });
+}
+
+export function displayTextForAtoms(
+ narrative: Narrative,
+ selected: readonly CaptionDisplayAtom[],
+): string {
+ const byRegion = new Map<string, CaptionDisplayAtom[]>();
+ for (const atom of selected) {
+ const values = byRegion.get(atom.regionId) ?? [];
+ values.push(atom);
+ byRegion.set(atom.regionId, values);
+ }
+ const parts: string[] = [];
+ for (const region of narrative.captionProjection.regions) {
+ const atoms = byRegion.get(region.id);
+ if (atoms === undefined || atoms.length === 0) continue;
+ const ordered = [...atoms].sort((left, right) => left.displayStart - right.displayStart);
+ const all = displayWords(region.display);
+ const first = ordered[0]!;
+ const last = ordered.at(-1)!;
+ const after = all.find((word) => word.start > last.displayStart);
+ parts.push(region.display.slice(first.displayStart, after?.start ?? region.display.length).trim());
+ }
+ return parts.join(" ").replace(/\s+/gu, " ").replace(/\s+([,.;:!?])/gu, "$1").trim();
+}
diff --git a/packages/caption/src/fragment.ts b/packages/caption/src/fragment.ts
new file mode 100644
index 00000000..b169b337
--- /dev/null
+++ b/packages/caption/src/fragment.ts
@@ -0,0 +1,72 @@
+import { contractTypes } from "@svml/contracts";
+import { sealGraphFragment } from "@svml/elaborator";
+
+import { captionProducers, captionTypes } from "./manifest.js";
+
+const input = (name: string) => ({ kind: "fragment-input" as const, name });
+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
+
+/** Provider-free Caption base: authored display text + measured map -> peer VisualTrack. */
+export const captionTrackSurfaceFragment = sealGraphFragment({
+ name: "@svml/caption/track-surface@1",
+ inputs: [
+ { name: "narrative", type: contractTypes.narrative },
+ { name: "map", type: contractTypes.completeSemanticMap },
+ { name: "program", type: captionTypes.trackProgram },
+ ],
+ operations: [
+ {
+ id: "caption:temporalize",
+ producer: captionProducers.temporalize,
+ inputs: { narrative: input("narrative"), map: input("map") },
+ result: { kind: "output", name: "caption" },
+ },
+ {
+ id: "caption:render",
+ producer: captionProducers.renderTrack,
+ inputs: { caption: operation("caption:temporalize"), program: input("program") },
+ result: { kind: "output", name: "track" },
+ },
+ ],
+ exports: [{
+ name: "track",
+ type: contractTypes.visualTrack,
+ root: operation("caption:render"),
+ semanticInputs: ["narrative", "map", "program"],
+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("map"), sourcePointer: "/programSpace/digest" }],
+ fidelity: "exact",
+ }],
+});
+
+/** Planner-neutral Cue/field facts join the independent measured map only at Caption lowering. */
+export const plannedCaptionTrackSurfaceFragment = sealGraphFragment({
+ name: "@svml/caption/planned-track-surface@2",
+ inputs: [
+ { name: "narrative", type: contractTypes.narrative },
+ { name: "map", type: contractTypes.completeSemanticMap },
+ { name: "plan", type: captionTypes.plan },
+ { name: "program", type: captionTypes.program },
+ ],
+ operations: [
+ {
+ id: "caption:temporalize-plan",
+ producer: captionProducers.temporalizePlan,
+ inputs: { narrative: input("narrative"), map: input("map"), program: input("program"), plan: input("plan") },
+ result: { kind: "output", name: "caption" },
+ },
+ {
+ id: "caption:render-plan",
+ producer: captionProducers.renderProgram,
+ inputs: { caption: operation("caption:temporalize-plan"), program: input("program") },
+ result: { kind: "output", name: "track" },
+ },
+ ],
+ exports: [{
+ name: "track",
+ type: contractTypes.visualTrack,
+ root: operation("caption:render-plan"),
+ semanticInputs: ["narrative", "map", "plan", "program"],
+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("map"), sourcePointer: "/programSpace/digest" }],
+ fidelity: "exact",
+ }],
+});
diff --git a/packages/caption/src/index.ts b/packages/caption/src/index.ts
index a269037f..1228d9ea 100644
--- a/packages/caption/src/index.ts
+++ b/packages/caption/src/index.ts
@@ -1,7 +1,16 @@
export { CaptionProjectionError } from "./error.js";
+export { captionDisplayAtoms, displayAtomMatchesSelection, displayTextForAtoms } from "./display.js";
export { captionComponent } from "./component.js";
+export { captionTrackSurfaceFragment, plannedCaptionTrackSurfaceFragment } from "./fragment.js";
export {
captionImplementationDigest,
+ captionProgramSchema,
+ captionProgramSurfaceImplementationDigest,
+ captionPlanImplementationDigest,
+ captionPlanSchema,
+ captionSurfaceImplementationDigest,
+ captionStyleSchema,
+ captionStyleSurfaceImplementationDigest,
captionManifest,
captionModuleRef,
captionProducers,
@@ -10,15 +19,32 @@ export {
captionValidatorDigests,
timedCaptionProjectionSchema,
} from "./manifest.js";
+export {
+ decodeCaptionProgramSurface,
+ decodeCaptionStyleSurface,
+ decodeCaptionTrackSurface,
+} from "./surface.js";
export { planCaptionPresentation } from "./presentation.js";
-export { temporalizeCaption } from "./temporalize.js";
+export { assertCaptionPlan, assertCaptionPlanForProgram, sealCaptionPlan } from "./plan.js";
+export {
+ assertCaptionProgram,
+ assertCaptionProgramForNarrative,
+ assertCaptionStyle,
+ resolveCaptionProgram,
+ sealCaptionProgram,
+ sealCaptionStyle,
+} from "./style.js";
+export type { CaptionStyleApplication, CaptionStyleSelector } from "./style.js";
+export { temporalizeCaption, temporalizeCaptionPlan } from "./temporalize.js";
export {
assertCaptionTrackProgram,
assertTimedCaptionProjection,
computeCaptionTrackProgramDigest,
defaultCaptionTrackProgram,
renderCaptionTrack,
+ renderCaptionProgram,
renderCaptionTrackImplementationDigest,
+ renderCaptionProgramImplementationDigest,
sealCaptionTrackProgram,
} from "./track.js";
export type * from "./types.js";
diff --git a/packages/caption/src/manifest.ts b/packages/caption/src/manifest.ts
index 41fbb961..a8f7c172 100644
--- a/packages/caption/src/manifest.ts
+++ b/packages/caption/src/manifest.ts
@@ -6,22 +6,37 @@ import {
import { digestOf } from "@svml/protocol";
import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";
-import { renderCaptionTrackImplementationDigest } from "./track.js";
+import {
+ renderCaptionProgramImplementationDigest,
+ renderCaptionTrackImplementationDigest,
+} from "./track.js";
export const captionModuleRef = { name: "@svml/caption", version: "0.0.0-dev" } as const;
export const captionProducers = {
temporalize: { module: captionModuleRef, name: "temporalize-caption" },
+ temporalizePlan: { module: captionModuleRef, name: "temporalize-caption-plan" },
+ renderProgram: { module: captionModuleRef, name: "render-caption-program" },
renderTrack: { module: captionModuleRef, name: "render-caption-track" },
} satisfies Record<string, ProducerRef>;
export const captionTypes = {
+ style: { module: captionModuleRef, name: "CaptionStyle" },
+ program: { module: captionModuleRef, name: "CaptionProgram" },
+ plan: { module: captionModuleRef, name: "CaptionPlan" },
timedProjection: { module: captionModuleRef, name: "TimedCaptionProjection" },
trackProgram: { module: captionModuleRef, name: "CaptionTrackProgram" },
} satisfies Record<string, TypeRef>;
export const captionImplementationDigest = digestOf("@svml/caption/temporalize@1");
+export const captionPlanImplementationDigest = digestOf("@svml/caption/temporalize-plan@2");
export const captionValidatorDigests = {
- timedProjection: digestOf("@svml/caption/validate-timed-projection@1"),
+ style: digestOf("@svml/caption/validate-style@2"),
+ program: digestOf("@svml/caption/validate-program@2"),
+ plan: digestOf("@svml/caption/validate-plan@2"),
+ timedProjection: digestOf("@svml/caption/validate-timed-projection@2"),
trackProgram: digestOf("@svml/caption/validate-track-program@1"),
} as const;
+export const captionSurfaceImplementationDigest = digestOf("@svml/caption/track-surface@2");
+export const captionStyleSurfaceImplementationDigest = digestOf("@svml/caption/style-surface@1");
+export const captionProgramSurfaceImplementationDigest = digestOf("@svml/caption/program-surface@1");
const string = { kind: "string", minLength: 1 } as const;
const number = { kind: "number", minimum: 0 } as const;
@@ -46,6 +61,8 @@ const timedCaptionRefinement = object({
});
const timedCaptionRegion = object({
id: { schema: string },
+ runId: { schema: string, optional: true },
+ styleId: { schema: string, optional: true },
display: { schema: { kind: "string" } },
segmentId: { schema: string },
kind: { schema: { kind: "string", enum: ["identity", "alias", "hidden"] } },
@@ -55,6 +72,34 @@ const timedCaptionRegion = object({
startQuality: { schema: quality },
endQuality: { schema: quality },
refinements: { schema: { kind: "array", items: timedCaptionRefinement } },
+ fields: { schema: { kind: "array", items: object({
+ declarationId: { schema: string },
+ atomId: { schema: string },
+ value: { schema: string },
+ }) }, optional: true },
+});
+const captionFieldAssignment = object({
+ declarationId: { schema: string },
+ atomId: { schema: string },
+ value: { schema: string },
+});
+const captionPlannedCue = object({
+ id: { schema: string },
+ atomIds: { schema: { kind: "array", minItems: 1, items: string } },
+ fields: { schema: { kind: "array", items: captionFieldAssignment } },
+});
+const captionPlannedRun = object({
+ id: { schema: string },
+ styleId: { schema: string },
+ cues: { schema: { kind: "array", minItems: 1, items: captionPlannedCue } },
+});
+export const captionPlanSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.caption-plan@1" } },
+ narrativeDigest: { schema: digest },
+ captionProgramDigest: { schema: digest },
+ planningRequestDigest: { schema: digest },
+ runs: { schema: { kind: "array", minItems: 1, items: captionPlannedRun } },
+ planDigest: { schema: digest },
});
export const timedCaptionProjectionSchema: ValueSchema = object({
contract: { schema: { kind: "literal", value: "svml.timed-caption-projection@1" } },
@@ -64,13 +109,7 @@ export const timedCaptionProjectionSchema: ValueSchema = object({
projectionDigest: { schema: digest },
});
-export const captionTrackProgramSchema: ValueSchema = object({
- contract: { schema: { kind: "literal", value: "svml.caption-track-program@1" } },
- digest: { schema: digest },
- id: { schema: string },
- mode: { schema: { kind: "string", enum: ["whole", "proportional-word", "character-flow"] } },
- stacking: { schema: object({ order: { schema: integer }, tieBreak: { schema: string } }) },
- style: { schema: object({
+const captionTrackStyleSchema = object({
fontFamily: { schema: string },
fontSizePx: { schema: number },
fontWeight: { schema: number },
@@ -81,8 +120,76 @@ export const captionTrackProgramSchema: ValueSchema = object({
borderRadiusPx: { schema: number },
bottomPercent: { schema: number },
maxWidthPercent: { schema: number },
+ leftPercent: { schema: number, optional: true },
+ topPercent: { schema: number, optional: true },
+ widthPercent: { schema: number, optional: true },
+ lineHeight: { schema: number, optional: true },
textAlign: { schema: { kind: "string", enum: ["left", "center", "right"] } },
+ });
+
+export const captionTrackProgramSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.caption-track-program@1" } },
+ digest: { schema: digest },
+ id: { schema: string },
+ mode: { schema: { kind: "string", enum: ["whole", "proportional-word", "character-flow"] } },
+ stacking: { schema: object({ order: { schema: integer }, tieBreak: { schema: string } }) },
+ style: { schema: captionTrackStyleSchema },
+});
+
+const captionFieldValueSchema: ValueSchema = {
+ kind: "oneOf",
+ variants: [
+ object({ kind: { schema: { kind: "literal", value: "boolean" } } }),
+ object({
+ kind: { schema: { kind: "literal", value: "enum" } },
+ values: { schema: { kind: "array", minItems: 1, items: string } },
+ }),
+ object({
+ kind: { schema: { kind: "literal", value: "number" } },
+ minimum: { schema: { kind: "number" }, optional: true },
+ maximum: { schema: { kind: "number" }, optional: true },
+ }),
+ ],
+};
+const captionFieldDeclaration = object({
+ id: { schema: string },
+ value: { schema: captionFieldValueSchema },
+ instruction: { schema: string },
+ minimumPerCue: { schema: integer },
+ maximumPerCue: { schema: integer },
+});
+export const captionStyleSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.caption-style@1" } },
+ id: { schema: string },
+ planning: { schema: object({
+ cueInstruction: { schema: string },
+ fields: { schema: { kind: "array", items: captionFieldDeclaration } },
+ }) },
+ presentation: { schema: object({
+ mode: { schema: { kind: "string", enum: ["whole", "proportional-word", "character-flow"] } },
+ stackingOrder: { schema: integer },
+ style: { schema: captionTrackStyleSchema },
}) },
+ digest: { schema: digest },
+});
+const captionDisplayAtom = object({
+ id: { schema: string }, index: { schema: integer }, regionId: { schema: string }, segmentId: { schema: string },
+ turnId: { schema: string }, role: { schema: string, optional: true }, text: { schema: string },
+ displayStart: { schema: integer }, displayEnd: { schema: integer }, sourceTokenStart: { schema: integer },
+ sourceTokenEndExclusive: { schema: integer },
+ correspondence: { schema: { kind: "string", enum: ["exact", "region-envelope"] } },
+});
+const captionProgramRun = object({
+ id: { schema: string }, styleId: { schema: string },
+ atomIds: { schema: { kind: "array", minItems: 1, items: string } },
+});
+export const captionProgramSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.caption-program@1" } },
+ id: { schema: string }, narrativeDigest: { schema: digest }, defaultStyleId: { schema: string },
+ styles: { schema: { kind: "array", minItems: 1, items: captionStyleSchema } },
+ atoms: { schema: { kind: "array", minItems: 1, items: captionDisplayAtom } },
+ runs: { schema: { kind: "array", minItems: 1, items: captionProgramRun } },
+ digest: { schema: digest },
});
export const captionManifest: ModuleManifest = {
@@ -96,6 +203,32 @@ export const captionManifest: ModuleManifest = {
videoContractDependencies.composition,
],
types: [
+ {
+ name: captionTypes.style.name,
+ schema: captionStyleSchema,
+ validator: { abi: "svml.type-validator@1", implementation: {
+ kind: "registered", locator: "@svml/caption/validate-style", digest: captionValidatorDigests.style,
+ } },
+ },
+ {
+ name: captionTypes.program.name,
+ schema: captionProgramSchema,
+ validator: { abi: "svml.type-validator@1", implementation: {
+ kind: "registered", locator: "@svml/caption/validate-program", digest: captionValidatorDigests.program,
+ } },
+ },
+ {
+ name: captionTypes.plan.name,
+ schema: captionPlanSchema,
+ validator: {
+ abi: "svml.type-validator@1",
+ implementation: {
+ kind: "registered",
+ locator: "@svml/caption/validate-plan",
+ digest: captionValidatorDigests.plan,
+ },
+ },
+ },
{
name: captionTypes.timedProjection.name,
schema: timedCaptionProjectionSchema,
@@ -122,7 +255,21 @@ export const captionManifest: ModuleManifest = {
},
],
capabilities: [],
- surfaces: [],
+ surfaces: [
+ {
+ name: "style", tag: "Style", mode: "structured", outputs: [captionTypes.style],
+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/caption/style-surface", digest: captionStyleSurfaceImplementationDigest },
+ },
+ {
+ name: "program", tag: "Program", mode: "structured", outputs: [captionTypes.program],
+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/caption/program-surface", digest: captionProgramSurfaceImplementationDigest },
+ },
+ {
+ name: "track", tag: "Track", mode: "structured",
+ outputs: [captionTypes.trackProgram, contractTypes.visualTrack],
+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/caption/track-surface", digest: captionSurfaceImplementationDigest },
+ },
+ ],
producers: [
{
name: captionProducers.temporalize.name,
@@ -144,6 +291,41 @@ export const captionManifest: ModuleManifest = {
digest: captionImplementationDigest,
},
},
+ {
+ name: captionProducers.temporalizePlan.name,
+ inputs: [
+ { name: "narrative", type: contractTypes.narrative },
+ { name: "map", type: contractTypes.completeSemanticMap },
+ { name: "program", type: captionTypes.program },
+ { name: "plan", type: captionTypes.plan },
+ ],
+ outputs: [{
+ name: "caption",
+ type: captionTypes.timedProjection,
+ affinity: [
+ { resultPointer: "/programSpace/digest", input: "map", inputPointer: "/programSpace/digest" },
+ ],
+ }],
+ needs: [],
+ implementation: {
+ kind: "registered",
+ locator: "@svml/caption/temporalize-plan",
+ digest: captionPlanImplementationDigest,
+ },
+ },
+ {
+ name: captionProducers.renderProgram.name,
+ inputs: [
+ { name: "caption", type: captionTypes.timedProjection },
+ { name: "program", type: captionTypes.program },
+ ],
+ outputs: [{
+ name: "track", type: contractTypes.visualTrack,
+ affinity: [{ resultPointer: "/programSpaceDigest", input: "caption", inputPointer: "/programSpace/digest" }],
+ }],
+ needs: [],
+ implementation: { kind: "registered", locator: "@svml/caption/render-program", digest: renderCaptionProgramImplementationDigest },
+ },
{
name: captionProducers.renderTrack.name,
inputs: [
diff --git a/packages/caption/src/plan.ts b/packages/caption/src/plan.ts
new file mode 100644
index 00000000..dfb09ddf
--- /dev/null
+++ b/packages/caption/src/plan.ts
@@ -0,0 +1,118 @@
+import { digestOf, isDigest } from "@svml/protocol";
+
+import type { CaptionFieldDeclaration, CaptionPlan, CaptionProgram } from "./types.js";
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function content(value: Omit<CaptionPlan, "planDigest">) {
+ return {
+ contract: "svml.caption-plan@1" as const,
+ narrativeDigest: value.narrativeDigest,
+ captionProgramDigest: value.captionProgramDigest,
+ planningRequestDigest: value.planningRequestDigest,
+ runs: value.runs.map((run) => ({
+ id: run.id,
+ styleId: run.styleId,
+ cues: run.cues.map((cue) => ({
+ id: cue.id,
+ atomIds: [...cue.atomIds],
+ fields: cue.fields.map((field) => ({ ...field })),
+ })),
+ })),
+ };
+}
+
+export function sealCaptionPlan(value: Omit<CaptionPlan, "planDigest">): CaptionPlan {
+ const normalized = content(value);
+ const plan = { ...normalized, planDigest: digestOf(normalized) };
+ assertCaptionPlan(plan);
+ return plan;
+}
+
+export function assertCaptionPlan(value: unknown): asserts value is CaptionPlan {
+ assert(value !== null && typeof value === "object" && !Array.isArray(value), "CaptionPlan must be an object");
+ const plan = value as CaptionPlan;
+ assert(plan.contract === "svml.caption-plan@1", "Unsupported CaptionPlan contract");
+ assert(isDigest(plan.narrativeDigest) && isDigest(plan.captionProgramDigest)
+ && isDigest(plan.planningRequestDigest) && isDigest(plan.planDigest), "CaptionPlan identity is invalid");
+ assert(Array.isArray(plan.runs) && plan.runs.length > 0, "CaptionPlan runs are empty");
+ const runIds = new Set<string>();
+ const atomIds = new Set<string>();
+ const cueIds = new Set<string>();
+ for (const run of plan.runs) {
+ assert(run.id.length > 0 && !runIds.has(run.id) && run.styleId.length > 0,
+ "CaptionPlan run id or style is invalid or repeated");
+ runIds.add(run.id);
+ assert(run.cues.length > 0, `CaptionPlan run ${run.id} has no Cues`);
+ for (const cue of run.cues) {
+ assert(cue.id.length > 0 && !cueIds.has(cue.id) && cue.atomIds.length > 0,
+ `CaptionPlan run ${run.id} contains an empty or repeated Cue`);
+ cueIds.add(cue.id);
+ assert(new Set(cue.atomIds).size === cue.atomIds.length, `CaptionPlan Cue ${cue.id} repeats a display atom`);
+ for (const atomId of cue.atomIds) {
+ assert(!atomIds.has(atomId), `CaptionPlan repeats display atom ${atomId}`);
+ atomIds.add(atomId);
+ }
+ for (const field of cue.fields) {
+ assert(field.declarationId.length > 0 && field.atomId.length > 0 && field.value.length > 0,
+ `CaptionPlan Cue ${cue.id} field is invalid`);
+ assert(cue.atomIds.includes(field.atomId), `CaptionPlan Cue ${cue.id} field lies outside its Cue`);
+ }
+ }
+ }
+ const { planDigest: _digest, ...withoutDigest } = plan;
+ assert(plan.planDigest === digestOf(content(withoutDigest)), "CaptionPlan digest does not match its contents");
+}
+
+function assertFieldValue(declaration: CaptionFieldDeclaration, value: string): void {
+ if (declaration.value.kind === "boolean") {
+ assert(value === "true", `Caption field ${declaration.id} is sparse and only accepts true`);
+ } else if (declaration.value.kind === "enum") {
+ assert(declaration.value.values.includes(value), `Caption field ${declaration.id} enum value is invalid`);
+ } else {
+ const numeric = Number(value);
+ assert(Number.isFinite(numeric), `Caption field ${declaration.id} number value is invalid`);
+ assert(declaration.value.minimum === undefined || numeric >= declaration.value.minimum,
+ `Caption field ${declaration.id} number is below its minimum`);
+ assert(declaration.value.maximum === undefined || numeric <= declaration.value.maximum,
+ `Caption field ${declaration.id} number is above its maximum`);
+ }
+}
+
+/** Validate planner freedom against one resolved Program; no Provider-specific assumptions. */
+export function assertCaptionPlanForProgram(plan: CaptionPlan, program: CaptionProgram): void {
+ assertCaptionPlan(plan);
+ assert(plan.narrativeDigest === program.narrativeDigest && plan.captionProgramDigest === program.digest,
+ "CaptionPlan belongs to another Caption Program");
+ assert(plan.runs.length === program.runs.length, "CaptionPlan does not contain the exact Program run set");
+ const styles = new Map(program.styles.map((style) => [style.id, style]));
+ for (let index = 0; index < program.runs.length; index += 1) {
+ const expected = program.runs[index]!;
+ const actual = plan.runs[index]!;
+ assert(actual.id === expected.id && actual.styleId === expected.styleId,
+ `CaptionPlan run ${actual.id} differs from Program order or Style`);
+ assert(actual.cues.flatMap((cue) => cue.atomIds).join("\0") === expected.atomIds.join("\0"),
+ `CaptionPlan run ${actual.id} does not partition its Program atoms exactly in order`);
+ const style = styles.get(expected.styleId);
+ assert(style !== undefined, `Caption Program Style ${expected.styleId} is absent`);
+ const declarations = new Map(style.planning.fields.map((field) => [field.id, field]));
+ for (const cue of actual.cues) {
+ const unique = new Set<string>();
+ for (const field of cue.fields) {
+ const declaration = declarations.get(field.declarationId);
+ assert(declaration !== undefined, `Caption field ${field.declarationId} is not declared by Style ${style.id}`);
+ const key = `${field.declarationId}\0${field.atomId}`;
+ assert(!unique.has(key), `Caption Cue ${cue.id} repeats field ${field.declarationId} on ${field.atomId}`);
+ unique.add(key);
+ assertFieldValue(declaration, field.value);
+ }
+ for (const declaration of declarations.values()) {
+ const count = cue.fields.filter((field) => field.declarationId === declaration.id).length;
+ assert(count >= declaration.minimumPerCue && count <= declaration.maximumPerCue,
+ `Caption Cue ${cue.id} field ${declaration.id} violates its per-Cue cardinality`);
+ }
+ }
+ }
+}
diff --git a/packages/caption/src/presentation.ts b/packages/caption/src/presentation.ts
index 62cfb0a6..f6976555 100644
--- a/packages/caption/src/presentation.ts
+++ b/packages/caption/src/presentation.ts
@@ -31,6 +31,8 @@ function wholeRegion(region: TimedCaptionRegion): CaptionPresentationUnit[] {
return [{
id: `${region.id}:whole`,
regionId: region.id,
+ ...(region.runId === undefined ? {} : { runId: region.runId }),
+ ...(region.styleId === undefined ? {} : { styleId: region.styleId }),
display: region.display,
displayStart: 0,
displayEnd: region.display.length,
@@ -38,6 +40,7 @@ function wholeRegion(region: TimedCaptionRegion): CaptionPresentationUnit[] {
endSec: region.endSec,
basis: "region-envelope",
timingQuality: region.startQuality === "estimated" || region.endQuality === "estimated" ? "estimated" : "derived",
+ ...(region.fields === undefined ? {} : { fields: region.fields.map((field) => ({ ...field })) }),
}];
}
@@ -50,6 +53,8 @@ function proportionalWords(region: TimedCaptionRegion): CaptionPresentationUnit[
return refinement ? {
id: `${region.id}:word:${index + 1}`,
regionId: region.id,
+ ...(region.runId === undefined ? {} : { runId: region.runId }),
+ ...(region.styleId === undefined ? {} : { styleId: region.styleId }),
display: word.display,
displayStart: word.start,
displayEnd: word.end,
@@ -83,6 +88,8 @@ function proportionalWords(region: TimedCaptionRegion): CaptionPresentationUnit[
units[index] = {
id: `${region.id}:word:${index + 1}`,
regionId: region.id,
+ ...(region.runId === undefined ? {} : { runId: region.runId }),
+ ...(region.styleId === undefined ? {} : { styleId: region.styleId }),
display: word.display,
displayStart: word.start,
displayEnd: word.end,
@@ -109,6 +116,8 @@ function characterFlow(region: TimedCaptionRegion): CaptionPresentationUnit[] {
units.push({
id: `${region.id}:character:${index + 1}`,
regionId: region.id,
+ ...(region.runId === undefined ? {} : { runId: region.runId }),
+ ...(region.styleId === undefined ? {} : { styleId: region.styleId }),
display: character,
displayStart: start,
displayEnd: sourceOffset,
diff --git a/packages/caption/src/style.ts b/packages/caption/src/style.ts
new file mode 100644
index 00000000..ccd38750
--- /dev/null
+++ b/packages/caption/src/style.ts
@@ -0,0 +1,239 @@
+import type { Narrative, NarrativeSelectionRef } from "@svml/contracts";
+import { canonicalize, digestOf, isDigest } from "@svml/protocol";
+
+import { captionDisplayAtoms, displayAtomMatchesSelection } from "./display.js";
+import type {
+ CaptionDisplayAtom,
+ CaptionFieldDeclaration,
+ CaptionProgram,
+ CaptionStyleIntent,
+} from "./types.js";
+
+export type CaptionStyleSelector =
+ | { readonly kind: "role"; readonly role: string }
+ | { readonly kind: "selection"; readonly selection: NarrativeSelectionRef };
+
+export type CaptionStyleApplication = {
+ readonly id: string;
+ readonly selector: CaptionStyleSelector;
+ readonly style: CaptionStyleIntent;
+};
+
+const ID = /^[A-Za-z][A-Za-z0-9_.-]{0,127}$/u;
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function normalizedField(field: CaptionFieldDeclaration): CaptionFieldDeclaration {
+ return {
+ id: field.id,
+ value: field.value.kind === "enum"
+ ? { kind: "enum", values: [...field.value.values] }
+ : field.value.kind === "number"
+ ? {
+ kind: "number",
+ ...(field.value.minimum === undefined ? {} : { minimum: field.value.minimum }),
+ ...(field.value.maximum === undefined ? {} : { maximum: field.value.maximum }),
+ }
+ : { kind: "boolean" },
+ instruction: field.instruction.trim(),
+ minimumPerCue: field.minimumPerCue,
+ maximumPerCue: field.maximumPerCue,
+ };
+}
+
+function styleContent(value: Omit<CaptionStyleIntent, "digest">) {
+ return canonicalize({
+ contract: "svml.caption-style@1",
+ id: value.id,
+ planning: {
+ cueInstruction: value.planning.cueInstruction.trim(),
+ fields: value.planning.fields.map(normalizedField),
+ },
+ presentation: canonicalize(value.presentation),
+ }) as unknown as Omit<CaptionStyleIntent, "digest">;
+}
+
+export function sealCaptionStyle(value: Omit<CaptionStyleIntent, "digest">): CaptionStyleIntent {
+ const content = styleContent(value);
+ const result = { ...content, digest: digestOf(content) };
+ assertCaptionStyle(result);
+ return result;
+}
+
+export function assertCaptionStyle(value: CaptionStyleIntent): void {
+ assert(value.contract === "svml.caption-style@1" && ID.test(value.id), "Caption Style identity is invalid");
+ assert(value.planning.cueInstruction.length > 0, `Caption Style ${value.id} Cue instruction is empty`);
+ const fields = new Set<string>();
+ for (const field of value.planning.fields) {
+ assert(ID.test(field.id) && !fields.has(field.id), `Caption Style ${value.id} field id is invalid or repeated`);
+ fields.add(field.id);
+ assert(field.instruction.trim().length > 0, `Caption Style ${value.id} field instruction is empty`);
+ assert(Number.isSafeInteger(field.minimumPerCue) && field.minimumPerCue >= 0,
+ `Caption Style ${value.id} field minimum is invalid`);
+ assert(Number.isSafeInteger(field.maximumPerCue) && field.maximumPerCue >= field.minimumPerCue,
+ `Caption Style ${value.id} field maximum is invalid`);
+ if (field.value.kind === "enum") {
+ assert(field.value.values.length > 0 && new Set(field.value.values).size === field.value.values.length,
+ `Caption Style ${value.id} field enum is invalid`);
+ } else if (field.value.kind === "number") {
+ assert(field.value.minimum === undefined || Number.isFinite(field.value.minimum),
+ `Caption Style ${value.id} field numeric minimum is invalid`);
+ assert(field.value.maximum === undefined || Number.isFinite(field.value.maximum),
+ `Caption Style ${value.id} field numeric maximum is invalid`);
+ }
+ }
+ assert(Number.isSafeInteger(value.presentation.stackingOrder) && value.presentation.stackingOrder >= 0,
+ `Caption Style ${value.id} stacking order is invalid`);
+ assert(["whole", "proportional-word", "character-flow"].includes(value.presentation.mode),
+ `Caption Style ${value.id} presentation mode is invalid`);
+ const appearance = value.presentation.style;
+ assert(appearance.fontFamily.trim().length > 0, `Caption Style ${value.id} font family is empty`);
+ assert(appearance.textAlign === "left" || appearance.textAlign === "center" || appearance.textAlign === "right",
+ `Caption Style ${value.id} text alignment is invalid`);
+ const numeric = [
+ appearance.fontSizePx,
+ appearance.fontWeight,
+ appearance.paddingXPx,
+ appearance.paddingYPx,
+ appearance.borderRadiusPx,
+ appearance.bottomPercent,
+ appearance.maxWidthPercent,
+ appearance.leftPercent,
+ appearance.topPercent,
+ appearance.widthPercent,
+ appearance.lineHeight,
+ ].filter((item): item is number => item !== undefined);
+ assert(numeric.every((item) => Number.isFinite(item) && item >= 0),
+ `Caption Style ${value.id} appearance contains an invalid number`);
+ assert(appearance.fontSizePx > 0 && appearance.maxWidthPercent > 0 && appearance.maxWidthPercent <= 100
+ && appearance.bottomPercent <= 100,
+ `Caption Style ${value.id} appearance bounds are invalid`);
+ for (const color of [appearance.color, appearance.backgroundColor].filter((item): item is string => item !== undefined)) {
+ assert(/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/iu.test(color), `Caption Style ${value.id} color ${color} is invalid`);
+ }
+ assert(isDigest(value.digest), `Caption Style ${value.id} digest is invalid`);
+ const { digest: _digest, ...withoutDigest } = value;
+ assert(value.digest === digestOf(styleContent(withoutDigest)), `Caption Style ${value.id} digest differs`);
+}
+
+function matches(
+ atom: CaptionDisplayAtom,
+ application: CaptionStyleApplication,
+ narrative: Narrative,
+): boolean {
+ if (application.selector.kind === "role") return atom.role === application.selector.role;
+ return displayAtomMatchesSelection(atom, application.selector.selection, narrative.tokens.length);
+}
+
+function programContent(value: Omit<CaptionProgram, "digest">) {
+ return canonicalize(value) as unknown as Omit<CaptionProgram, "digest">;
+}
+
+export function sealCaptionProgram(value: Omit<CaptionProgram, "digest">): CaptionProgram {
+ const content = programContent(value);
+ const result = { ...content, digest: digestOf(content) };
+ assertCaptionProgram(result);
+ return result;
+}
+
+export function assertCaptionProgram(value: CaptionProgram): void {
+ assert(value.contract === "svml.caption-program@1" && ID.test(value.id), "Caption Program identity is invalid");
+ assert(isDigest(value.narrativeDigest) && isDigest(value.digest), "Caption Program digest is invalid");
+ assert(value.atoms.length > 0 && value.runs.length > 0 && value.styles.length > 0, "Caption Program is empty");
+ const styles = new Map(value.styles.map((style) => [style.id, style]));
+ assert(styles.size === value.styles.length && styles.has(value.defaultStyleId), "Caption Program styles are invalid");
+ value.styles.forEach(assertCaptionStyle);
+ const atomIds = value.atoms.map((atom) => atom.id);
+ assert(new Set(atomIds).size === atomIds.length, "Caption Program display atom ids are repeated");
+ value.atoms.forEach((atom, index) => {
+ assert(atom.id.length > 0 && atom.index === index && atom.regionId.length > 0 && atom.segmentId.length > 0
+ && atom.turnId.length > 0 && atom.text.length > 0, "Caption Program display atom identity is invalid");
+ assert(Number.isSafeInteger(atom.displayStart) && Number.isSafeInteger(atom.displayEnd)
+ && atom.displayStart >= 0 && atom.displayEnd > atom.displayStart,
+ `Caption Program display atom ${atom.id} display range is invalid`);
+ assert(Number.isSafeInteger(atom.sourceTokenStart) && Number.isSafeInteger(atom.sourceTokenEndExclusive)
+ && atom.sourceTokenStart >= 0 && atom.sourceTokenEndExclusive > atom.sourceTokenStart,
+ `Caption Program display atom ${atom.id} source range is invalid`);
+ assert(atom.correspondence === "exact" || atom.correspondence === "region-envelope",
+ `Caption Program display atom ${atom.id} correspondence is invalid`);
+ });
+ assert(new Set(value.runs.map((run) => run.id)).size === value.runs.length,
+ "Caption Program run ids are repeated");
+ const planned = value.runs.flatMap((run) => run.atomIds);
+ assert(planned.join("\0") === atomIds.join("\0"),
+ "Caption Program runs must partition every display atom exactly once and in order");
+ assert(value.runs.every((run) => styles.has(run.styleId) && run.atomIds.length > 0), "Caption Program run style is invalid");
+ const { digest: _digest, ...withoutDigest } = value;
+ assert(value.digest === digestOf(programContent(withoutDigest)), "Caption Program digest differs");
+}
+
+/** Bind a Program's immutable display universe to the exact Narrative consumed by a Producer. */
+export function assertCaptionProgramForNarrative(value: CaptionProgram, narrative: Narrative): void {
+ assertCaptionProgram(value);
+ assert(value.narrativeDigest === digestOf(narrative), "Caption Program belongs to another Narrative");
+ assert(digestOf(value.atoms) === digestOf(captionDisplayAtoms(narrative)),
+ "Caption Program display atoms differ from its Narrative");
+}
+
+/** Resolve a base style plus ordered, whole-style replacements. Later matching applications win. */
+export function resolveCaptionProgram(
+ narrative: Narrative,
+ id: string,
+ defaultStyle: CaptionStyleIntent,
+ applications: readonly CaptionStyleApplication[],
+): CaptionProgram {
+ assertCaptionStyle(defaultStyle);
+ applications.forEach((application) => {
+ assert(ID.test(application.id), "Caption Style Application identity is invalid");
+ assertCaptionStyle(application.style);
+ if (application.selector.kind === "role") {
+ assert(application.selector.role.trim().length > 0, `Caption application ${application.id} Role is empty`);
+ }
+ });
+ const atoms = captionDisplayAtoms(narrative);
+ const styles = new Map<string, CaptionStyleIntent>([[defaultStyle.id, defaultStyle]]);
+ applications.forEach((application) => {
+ const previous = styles.get(application.style.id);
+ assert(previous === undefined || previous.digest === application.style.digest,
+ `Caption Style ${application.style.id} has conflicting definitions`);
+ styles.set(application.style.id, application.style);
+ });
+ for (const application of applications) {
+ assert(atoms.some((atom) => matches(atom, application, narrative)),
+ `Caption application ${application.id} selects no visible display atom`);
+ }
+ const assignments = atoms.map((atom) => {
+ let styleId = defaultStyle.id;
+ for (const application of applications) {
+ if (matches(atom, application, narrative)) styleId = application.style.id;
+ }
+ return { atom, styleId };
+ });
+ const runs: Array<{ id: string; styleId: string; atomIds: string[]; turnId: string; segmentId: string }> = [];
+ for (const assignment of assignments) {
+ const current = runs.at(-1);
+ if (current === undefined || current.styleId !== assignment.styleId
+ || current.turnId !== assignment.atom.turnId || current.segmentId !== assignment.atom.segmentId) {
+ runs.push({
+ id: `${id}:run:${runs.length + 1}`,
+ styleId: assignment.styleId,
+ atomIds: [assignment.atom.id],
+ turnId: assignment.atom.turnId,
+ segmentId: assignment.atom.segmentId,
+ });
+ } else {
+ current.atomIds.push(assignment.atom.id);
+ }
+ }
+ return sealCaptionProgram({
+ contract: "svml.caption-program@1",
+ id,
+ narrativeDigest: digestOf(narrative),
+ defaultStyleId: defaultStyle.id,
+ styles: [...styles.values()],
+ atoms,
+ runs: runs.map(({ turnId: _turn, segmentId: _segment, ...run }) => run),
+ });
+}
diff --git a/packages/caption/src/surface.ts b/packages/caption/src/surface.ts
new file mode 100644
index 00000000..e239d71a
--- /dev/null
+++ b/packages/caption/src/surface.ts
@@ -0,0 +1,321 @@
+import { contractTypes } from "@svml/contracts";
+import type { Narrative, NarrativeSelectionRef } from "@svml/contracts";
+import { svsRecipeType } from "@svml/svs";
+import type { SvsRecipe } from "@svml/svs";
+import type {
+ StructuredElement,
+ StructuredSurfaceHandler,
+ SurfaceResolvedReference,
+ TextAttributeValue,
+} from "@svml/text";
+
+import { captionTrackSurfaceFragment, plannedCaptionTrackSurfaceFragment } from "./fragment.js";
+import { captionTypes } from "./manifest.js";
+import { sealCaptionTrackProgram } from "./track.js";
+import { resolveCaptionProgram, sealCaptionStyle } from "./style.js";
+import type { CaptionStyleApplication } from "./style.js";
+import type {
+ CaptionFieldDeclaration,
+ CaptionFieldValueSchema,
+ CaptionPresentationMode,
+ CaptionStyleIntent,
+} from "./types.js";
+
+function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+}
+
+function attributes(element: StructuredElement, required: readonly string[], optional: readonly string[] = []): void {
+ const allowed = new Set([...required, ...optional]);
+ const actual = Object.keys(element.attributes);
+ if (required.some((name) => element.attributes[name] === undefined) || actual.some((name) => !allowed.has(name))) {
+ throw new Error(`${element.name} requires ${required.join(", ")}${optional.length ? `; optional: ${optional.join(", ")}` : ""}`);
+ }
+}
+
+function stringAttribute(element: StructuredElement, name: string): string {
+ const value = element.attributes[name];
+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);
+ return value.trim();
+}
+
+function reference(
+ element: StructuredElement,
+ name: string,
+ expected: SurfaceResolvedReference["type"],
+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,
+): SurfaceResolvedReference {
+ const raw: TextAttributeValue | undefined = element.attributes[name];
+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a whole-value reference`);
+ const value = resolveReference(raw.path);
+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${raw.path}`);
+ if (!sameType(value.type, expected)) throw new Error(`${element.name}.${name} has the wrong type`);
+ return value;
+}
+
+function recipe(reference: SurfaceResolvedReference): SvsRecipe {
+ if (!sameType(reference.type, svsRecipeType) || reference.record?.value.kind !== "inline") {
+ throw new Error("Caption appearance must reference an authored SVS Recipe");
+ }
+ return reference.record.value.value as unknown as SvsRecipe;
+}
+
+function number(value: SvsRecipe, name: string): number {
+ const property = value.properties[name];
+ if (typeof property !== "number" || !Number.isFinite(property)) throw new Error(`Caption Recipe ${name} must be a number`);
+ return property;
+}
+
+function integer(value: SvsRecipe, name: string): number {
+ const property = number(value, name);
+ if (!Number.isSafeInteger(property)) throw new Error(`Caption Recipe ${name} must be an integer`);
+ return property;
+}
+
+function string(value: SvsRecipe, name: string): string {
+ const property = value.properties[name];
+ if (typeof property !== "string" || !property.trim()) throw new Error(`Caption Recipe ${name} must be a string`);
+ return property.trim();
+}
+
+function padding(value: string): { readonly x: number; readonly y: number } {
+ const parts = value.trim().split(/\s+/u).map(Number);
+ if ((parts.length !== 1 && parts.length !== 2) || parts.some((item) => !Number.isFinite(item) || item < 0)) {
+ throw new Error("Caption Recipe padding must contain one or two non-negative pixel numbers");
+ }
+ return { y: parts[0]!, x: parts[1] ?? parts[0]! };
+}
+
+function mode(element: StructuredElement): CaptionPresentationMode {
+ const value = element.attributes.mode;
+ if (value === undefined) return "whole";
+ if (typeof value === "string" && ["whole", "proportional-word", "character-flow"].includes(value)) {
+ return value as CaptionPresentationMode;
+ }
+ throw new Error(`${element.name}.mode is invalid`);
+}
+
+function inline<T>(reference: SurfaceResolvedReference, subject: string): T {
+ if (reference.record?.value.kind !== "inline") throw new Error(`${subject} must reference an authored inline Record`);
+ return reference.record.value.value as unknown as T;
+}
+
+function localName(name: string): string {
+ return name.includes(":") ? name.slice(name.lastIndexOf(":") + 1) : name;
+}
+
+function body(element: StructuredElement): string {
+ if (element.children.some((child) => child.kind === "element")) throw new Error(`${element.name} accepts only natural-language text`);
+ const value = element.children.map((child) => child.kind === "text" ? child.value : "").join("").trim();
+ if (!value) throw new Error(`${element.name} instruction is empty`);
+ return value.replace(/\s+/gu, " ");
+}
+
+function optionalString(element: StructuredElement, name: string): string | undefined {
+ const value = element.attributes[name];
+ if (value === undefined) return undefined;
+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);
+ return value.trim();
+}
+
+function nonNegativeInteger(element: StructuredElement, name: string): number {
+ const value = Number(stringAttribute(element, name));
+ if (!Number.isSafeInteger(value) || value < 0) throw new Error(`${element.name}.${name} must be a non-negative integer`);
+ return value;
+}
+
+function fieldValue(element: StructuredElement): CaptionFieldValueSchema {
+ const type = stringAttribute(element, "type");
+ if (type === "boolean") return { kind: "boolean" };
+ if (type === "enum") {
+ const values = stringAttribute(element, "values").split(",").map((item) => item.trim()).filter(Boolean);
+ if (!values.length) throw new Error(`${element.name}.values is empty`);
+ return { kind: "enum", values };
+ }
+ if (type === "number") {
+ const minimum = optionalString(element, "minimum");
+ const maximum = optionalString(element, "maximum");
+ const min = minimum === undefined ? undefined : Number(minimum);
+ const max = maximum === undefined ? undefined : Number(maximum);
+ if ((min !== undefined && !Number.isFinite(min)) || (max !== undefined && !Number.isFinite(max))) {
+ throw new Error(`${element.name} number bounds must be finite`);
+ }
+ return { kind: "number", ...(min === undefined ? {} : { minimum: min }), ...(max === undefined ? {} : { maximum: max }) };
+ }
+ throw new Error(`${element.name}.type must be boolean, enum or number`);
+}
+
+function captionAppearance(element: StructuredElement, value: SvsRecipe) {
+ const expected = ["align", "background", "fill", "font", "line-height", "padding", "radius", "size", "stack-order", "weight", "width", "x", "y"];
+ if (Object.keys(value.properties).sort().join("\u0000") !== expected.sort().join("\u0000")) {
+ throw new Error(`Caption Recipe requires exactly ${expected.join(", ")}`);
+ }
+ const align = string(value, "align");
+ if (align !== "left" && align !== "center" && align !== "right") throw new Error("Caption Recipe align is invalid");
+ const pad = padding(string(value, "padding"));
+ return {
+ mode: mode(element),
+ stackingOrder: integer(value, "stack-order"),
+ style: {
+ fontFamily: string(value, "font"),
+ fontSizePx: number(value, "size"),
+ fontWeight: integer(value, "weight"),
+ color: string(value, "fill"),
+ backgroundColor: string(value, "background"),
+ paddingXPx: pad.x,
+ paddingYPx: pad.y,
+ borderRadiusPx: number(value, "radius"),
+ bottomPercent: 0,
+ maxWidthPercent: number(value, "width") * 100,
+ leftPercent: number(value, "x") * 100,
+ topPercent: number(value, "y") * 100,
+ widthPercent: number(value, "width") * 100,
+ lineHeight: number(value, "line-height"),
+ textAlign: align,
+ },
+ } as const;
+}
+
+export const decodeCaptionStyleSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ attributes(element, ["id", "appearance"], ["mode"]);
+ const id = stringAttribute(element, "id");
+ const appearance = recipe(reference(element, "appearance", svsRecipeType, resolveReference));
+ let cueInstruction: string | undefined;
+ const fields: CaptionFieldDeclaration[] = [];
+ for (const child of element.children) {
+ if (child.kind === "text") {
+ if (child.value.trim()) throw new Error(`${element.name} accepts only Cues and Field children`);
+ continue;
+ }
+ if (localName(child.name) === "Cues") {
+ attributes(child, []);
+ if (cueInstruction !== undefined) throw new Error(`${element.name} repeats Cues`);
+ cueInstruction = body(child);
+ } else if (localName(child.name) === "Field") {
+ attributes(child, ["id", "type", "min-per-cue", "max-per-cue"], ["values", "minimum", "maximum"]);
+ fields.push({
+ id: stringAttribute(child, "id"),
+ value: fieldValue(child),
+ instruction: body(child),
+ minimumPerCue: nonNegativeInteger(child, "min-per-cue"),
+ maximumPerCue: nonNegativeInteger(child, "max-per-cue"),
+ });
+ } else {
+ throw new Error(`${element.name} accepts only Cues and Field children`);
+ }
+ }
+ if (cueInstruction === undefined) throw new Error(`${element.name} requires one Cues instruction`);
+ const style = sealCaptionStyle({
+ contract: "svml.caption-style@1",
+ id,
+ planning: { cueInstruction, fields },
+ presentation: captionAppearance(element, appearance),
+ });
+ return {
+ records: [{ id, type: captionTypes.style, value: { kind: "inline", value: style }, range: element.range }],
+ components: [],
+ fragments: [],
+ };
+};
+
+export const decodeCaptionProgramSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ attributes(element, ["id", "narrative", "default"]);
+ const id = stringAttribute(element, "id");
+ const narrativeRef = reference(element, "narrative", contractTypes.narrative, resolveReference);
+ const defaultRef = reference(element, "default", captionTypes.style, resolveReference);
+ const applications: CaptionStyleApplication[] = [];
+ for (const child of element.children) {
+ if (child.kind === "text") {
+ if (child.value.trim()) throw new Error(`${element.name} accepts only Use children`);
+ continue;
+ }
+ if (localName(child.name) !== "Use") throw new Error(`${element.name} accepts only Use children`);
+ attributes(child, ["style"], ["role", "on"]);
+ const role = optionalString(child, "role");
+ const on = child.attributes.on;
+ if ((role === undefined) === (on === undefined)) throw new Error(`${child.name} requires exactly one of role or on`);
+ const styleRef = reference(child, "style", captionTypes.style, resolveReference);
+ const selector = role !== undefined
+ ? { kind: "role" as const, role }
+ : {
+ kind: "selection" as const,
+ selection: inline<NarrativeSelectionRef>(
+ reference(child, "on", contractTypes.narrativeSelection, resolveReference),
+ `${child.name}.on`,
+ ),
+ };
+ applications.push({
+ id: `${id}.use.${applications.length + 1}`,
+ selector,
+ style: inline<CaptionStyleIntent>(styleRef, `${child.name}.style`),
+ });
+ }
+ const program = resolveCaptionProgram(
+ inline<Narrative>(narrativeRef, `${element.name}.narrative`),
+ id,
+ inline<CaptionStyleIntent>(defaultRef, `${element.name}.default`),
+ applications,
+ );
+ return {
+ records: [{ id, type: captionTypes.program, value: { kind: "inline", value: program }, range: element.range }],
+ components: [],
+ fragments: [],
+ };
+};
+
+export const decodeCaptionTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ if (element.attributes.program !== undefined) {
+ attributes(element, ["id", "narrative", "map", "program", "plan"]);
+ if (element.children.some((child) => child.kind === "element" || child.value.trim())) {
+ throw new Error(`${element.name} with a Caption Program does not accept children`);
+ }
+ const id = stringAttribute(element, "id");
+ const narrative = reference(element, "narrative", contractTypes.narrative, resolveReference);
+ const map = reference(element, "map", contractTypes.completeSemanticMap, resolveReference);
+ const program = reference(element, "program", captionTypes.program, resolveReference);
+ const plan = reference(element, "plan", captionTypes.plan, resolveReference);
+ return {
+ records: [],
+ components: [{
+ id,
+ fragment: plannedCaptionTrackSurfaceFragment.id,
+ inputs: { narrative: narrative.ref, map: map.ref, program: program.ref, plan: plan.ref },
+ outputs: { track: `${id}.track` },
+ range: element.range,
+ }],
+ fragments: [plannedCaptionTrackSurfaceFragment],
+ };
+ }
+ attributes(element, ["id", "narrative", "map", "appearance"], ["mode"]);
+ if (element.children.some((child) => child.kind === "element" || child.value.trim())) {
+ throw new Error(`${element.name} provider-free base does not accept Role children`);
+ }
+ const id = stringAttribute(element, "id");
+ const narrative = reference(element, "narrative", contractTypes.narrative, resolveReference);
+ const map = reference(element, "map", contractTypes.completeSemanticMap, resolveReference);
+ const appearance = recipe(reference(element, "appearance", svsRecipeType, resolveReference));
+ const resolvedAppearance = captionAppearance(element, appearance);
+ const program = sealCaptionTrackProgram({
+ contract: "svml.caption-track-program@1",
+ id,
+ mode: resolvedAppearance.mode,
+ stacking: { order: resolvedAppearance.stackingOrder, tieBreak: id },
+ style: resolvedAppearance.style,
+ });
+ const programId = `${id}.program`;
+ return {
+ records: [{ id: programId, type: captionTypes.trackProgram, value: { kind: "inline", value: program }, range: element.range }],
+ components: [{
+ id,
+ fragment: captionTrackSurfaceFragment.id,
+ inputs: {
+ narrative: narrative.ref,
+ map: map.ref,
+ program: { kind: "record", id: programId },
+ },
+ outputs: { track: `${id}.track` },
+ range: element.range,
+ }],
+ fragments: [captionTrackSurfaceFragment],
+ };
+};
diff --git a/packages/caption/src/temporalize.ts b/packages/caption/src/temporalize.ts
index 33d9c31e..5c3d3614 100644
--- a/packages/caption/src/temporalize.ts
+++ b/packages/caption/src/temporalize.ts
@@ -3,10 +3,16 @@ import type { CompleteSemanticMap, Narrative, TimingQuality } from "@svml/contra
import { CaptionProjectionError } from "./error.js";
import type {
+ CaptionPlan,
+ CaptionProgram,
+ CaptionDisplayAtom,
TimedCaptionProjection,
TimedCaptionRefinement,
TimedCaptionRegion,
} from "./types.js";
+import { assertCaptionPlanForProgram } from "./plan.js";
+import { displayTextForAtoms } from "./display.js";
+import { assertCaptionProgramForNarrative } from "./style.js";
const QUALITY_RANK: Readonly<Record<TimingQuality, number>> = {
measured: 0,
@@ -69,3 +75,134 @@ export function temporalizeCaption(
};
return { ...payload, projectionDigest: digestOf(payload) };
}
+
+function cleanDisplay(value: string): string {
+ return value.replace(/\s+/gu, " ").replace(/\s+([,.;:!?])/gu, "$1").trim();
+}
+
+/** Join planner-neutral Cue/field facts with the one authoritative Script map. */
+export function temporalizeCaptionPlan(
+ narrative: Narrative,
+ map: CompleteSemanticMap,
+ program: CaptionProgram,
+ plan: CaptionPlan,
+): TimedCaptionProjection {
+ assertCaptionProgramForNarrative(program, narrative);
+ assertCaptionPlanForProgram(plan, program);
+ if (plan.narrativeDigest !== digestOf(narrative)) {
+ throw new CaptionProjectionError("CAPTION_PLAN_NARRATIVE", "CaptionPlan belongs to another Narrative.");
+ }
+ if (program.narrativeDigest !== plan.narrativeDigest || plan.captionProgramDigest !== program.digest) {
+ throw new CaptionProjectionError("CAPTION_PLAN_PROGRAM", "CaptionPlan belongs to another Caption Program.");
+ }
+ const atomById = new Map(program.atoms.map((atom) => [atom.id, atom]));
+ const timingById = new Map(map.tokens.map((token) => [token.tokenId, token]));
+ const regionById = new Map(narrative.captionProjection.regions.map((region) => [region.id, region]));
+ const atomsByRegion = new Map<string, CaptionDisplayAtom[]>();
+ for (const atom of program.atoms) {
+ const values = atomsByRegion.get(atom.regionId) ?? [];
+ values.push(atom);
+ atomsByRegion.set(atom.regionId, values);
+ }
+ const atomWindow = (atom: CaptionDisplayAtom) => {
+ const firstSource = narrative.tokens[atom.sourceTokenStart];
+ const lastSource = narrative.tokens[atom.sourceTokenEndExclusive - 1];
+ const first = firstSource === undefined ? undefined : timingById.get(firstSource.id);
+ const last = lastSource === undefined ? undefined : timingById.get(lastSource.id);
+ if (first === undefined || last === undefined) {
+ throw new CaptionProjectionError("CAPTION_SPEECH_COVERAGE", `Caption display atom ${atom.id} is absent from the complete speech map.`);
+ }
+ if (atom.correspondence === "exact") {
+ return {
+ startSec: first.startSec,
+ endSec: last.endSec,
+ startQuality: composedQuality(first.startQuality),
+ endQuality: composedQuality(last.endQuality),
+ };
+ }
+ const region = regionById.get(atom.regionId);
+ if (region === undefined || region.display.length === 0) {
+ throw new CaptionProjectionError("CAPTION_DISPLAY_REGION", `Caption display atom ${atom.id} has no owning display region.`);
+ }
+ const siblings = atomsByRegion.get(atom.regionId) ?? [];
+ const next = siblings.find((candidate) => candidate.displayStart > atom.displayStart);
+ const duration = last.endSec - first.startSec;
+ return {
+ startSec: first.startSec + duration * atom.displayStart / region.display.length,
+ endSec: first.startSec + duration * (next?.displayStart ?? region.display.length) / region.display.length,
+ startQuality: "estimated" as const,
+ endQuality: "estimated" as const,
+ };
+ };
+ const visible = new Set(program.atoms.map((atom) => atom.id));
+ const plannedIds = plan.runs.flatMap((run) => run.cues.flatMap((cue) => cue.atomIds));
+ if (plannedIds.length !== visible.size || new Set(plannedIds).size !== visible.size || plannedIds.some((id) => !visible.has(id))) {
+ throw new CaptionProjectionError(
+ "CAPTION_PLAN_COVERAGE",
+ "CaptionPlan Cues must be disjoint and cover every display atom exactly once.",
+ );
+ }
+ const programRuns = new Map(program.runs.map((run) => [run.id, run]));
+ const regions: TimedCaptionRegion[] = [];
+ for (const run of plan.runs) {
+ const expectedRun = programRuns.get(run.id);
+ if (expectedRun === undefined || expectedRun.styleId !== run.styleId) {
+ throw new CaptionProjectionError("CAPTION_PLAN_RUN", `Caption run ${run.id} differs from its Program.`);
+ }
+ const runAtoms = run.cues.flatMap((cue) => cue.atomIds);
+ if (runAtoms.join("\0") !== expectedRun.atomIds.join("\0")) {
+ throw new CaptionProjectionError("CAPTION_PLAN_RUN", `Caption run ${run.id} changes its Program atom order.`);
+ }
+ for (const cue of run.cues) {
+ const source = cue.atomIds.map((id) => atomById.get(id));
+ if (source.some((atom) => atom === undefined)) {
+ throw new CaptionProjectionError("CAPTION_PLAN_ATOM", `Caption Cue ${cue.id} references an unknown display atom.`);
+ }
+ const atoms = source as CaptionDisplayAtom[];
+ if (atoms.some((atom, index) => index > 0 && atom.index <= atoms[index - 1]!.index)) {
+ throw new CaptionProjectionError("CAPTION_PLAN_ORDER", `Caption Cue ${cue.id} changes display atom order.`);
+ }
+ if (new Set(atoms.map((atom) => atom.segmentId)).size !== 1) {
+ throw new CaptionProjectionError("CAPTION_PLAN_SEGMENT", `Caption Cue ${cue.id} crosses a Script Segment.`);
+ }
+ const sourceIndexes = [...new Set(atoms.flatMap((atom) => Array.from(
+ { length: atom.sourceTokenEndExclusive - atom.sourceTokenStart },
+ (_, offset) => atom.sourceTokenStart + offset,
+ )))].sort((left, right) => left - right);
+ const sourceTokens = sourceIndexes.map((index) => narrative.tokens[index]);
+ const timed = sourceTokens.map((token) => token === undefined ? undefined : timingById.get(token.id));
+ if (timed.length === 0 || timed.some((token) => token === undefined)) {
+ throw new CaptionProjectionError("CAPTION_SPEECH_COVERAGE", `Caption Cue ${cue.id} is absent from the complete speech map.`);
+ }
+ const firstWindow = atomWindow(atoms[0]!);
+ const lastWindow = atomWindow(atoms.at(-1)!);
+ for (const field of cue.fields) {
+ if (!cue.atomIds.includes(field.atomId)) {
+ throw new CaptionProjectionError("CAPTION_PLAN_FIELD", `Caption field ${field.declarationId} lies outside Cue ${cue.id}.`);
+ }
+ }
+ regions.push({
+ id: cue.id,
+ runId: run.id,
+ styleId: run.styleId,
+ display: displayTextForAtoms(narrative, atoms),
+ segmentId: atoms[0]!.segmentId,
+ kind: atoms.some((atom) => atom.correspondence === "region-envelope") ? "alias" : "identity",
+ sourceTokenIds: sourceTokens.map((token) => token!.id),
+ startSec: firstWindow.startSec,
+ endSec: lastWindow.endSec,
+ startQuality: firstWindow.startQuality,
+ endQuality: lastWindow.endQuality,
+ refinements: [],
+ fields: cue.fields.map((field) => ({ ...field })),
+ });
+ }
+ }
+ const payload = {
+ contract: "svml.timed-caption-projection@1" as const,
+ programSpace: map.programSpace,
+ text: cleanDisplay(regions.map((region) => region.display).join(" ")),
+ regions,
+ };
+ return { ...payload, projectionDigest: digestOf(payload) };
+}
diff --git a/packages/caption/src/track.ts b/packages/caption/src/track.ts
index 12040150..8fc4bbbf 100644
--- a/packages/caption/src/track.ts
+++ b/packages/caption/src/track.ts
@@ -9,10 +9,11 @@ import { digestOf, isDigest } from "@svml/protocol";
import type { Digest } from "@svml/protocol";
import { planCaptionPresentation } from "./presentation.js";
-import type { TimedCaptionProjection } from "./types.js";
-import type { CaptionTrackProgram } from "./types.js";
+import { assertCaptionProgram } from "./style.js";
+import type { CaptionProgram, CaptionTrackProgram, TimedCaptionProjection } from "./types.js";
export const renderCaptionTrackImplementationDigest = digestOf("@svml/caption/render-track@1");
+export const renderCaptionProgramImplementationDigest = digestOf("@svml/caption/render-program@1");
function normalizedProgram(value: Omit<CaptionTrackProgram, "digest">): Omit<CaptionTrackProgram, "digest"> {
return {
@@ -75,6 +76,10 @@ export function assertCaptionTrackProgram(program: CaptionTrackProgram): void {
borderRadiusPx: program.style.borderRadiusPx,
bottomPercent: program.style.bottomPercent,
maxWidthPercent: program.style.maxWidthPercent,
+ ...(program.style.leftPercent === undefined ? {} : { leftPercent: program.style.leftPercent }),
+ ...(program.style.topPercent === undefined ? {} : { topPercent: program.style.topPercent }),
+ ...(program.style.widthPercent === undefined ? {} : { widthPercent: program.style.widthPercent }),
+ ...(program.style.lineHeight === undefined ? {} : { lineHeight: program.style.lineHeight }),
};
for (const [name, value] of Object.entries(numeric)) {
if (!Number.isFinite(value) || value < 0) throw new Error(`CaptionTrackProgram ${name} is invalid.`);
@@ -84,7 +89,7 @@ export function assertCaptionTrackProgram(program: CaptionTrackProgram): void {
}
if (program.style.bottomPercent > 100) throw new Error("CaptionTrackProgram bottomPercent is invalid.");
for (const color of [program.style.color, program.style.backgroundColor].filter((value): value is string => value !== undefined)) {
- if (!/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/u.test(color)) throw new Error(`CaptionTrackProgram color ${color} is invalid.`);
+ if (!/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/iu.test(color)) throw new Error(`CaptionTrackProgram color ${color} is invalid.`);
}
}
@@ -105,16 +110,17 @@ function frameAt(projection: TimedCaptionProjection, seconds: number): number {
/ projection.programSpace.frameRate.denominator);
}
-function textStyle(program: CaptionTrackProgram): VisualStyleDeclaration[] {
+function textStyle(style: CaptionTrackProgram["style"]): VisualStyleDeclaration[] {
return [
- { name: "background", value: program.style.backgroundColor ?? "transparent" },
- { name: "border-radius", value: `${program.style.borderRadiusPx}px` },
- { name: "color", value: program.style.color },
- { name: "font-family", value: program.style.fontFamily },
- { name: "font-size", value: `${program.style.fontSizePx}px` },
- { name: "font-weight", value: program.style.fontWeight },
- { name: "padding", value: `${program.style.paddingYPx}px ${program.style.paddingXPx}px` },
- { name: "text-align", value: program.style.textAlign },
+ { name: "background", value: style.backgroundColor ?? "transparent" },
+ { name: "border-radius", value: `${style.borderRadiusPx}px` },
+ { name: "color", value: style.color },
+ { name: "font-family", value: style.fontFamily },
+ { name: "font-size", value: `${style.fontSizePx}px` },
+ { name: "font-weight", value: style.fontWeight },
+ ...(style.lineHeight === undefined ? [] : [{ name: "line-height", value: style.lineHeight }]),
+ { name: "padding", value: `${style.paddingYPx}px ${style.paddingXPx}px` },
+ { name: "text-align", value: style.textAlign },
{ name: "white-space", value: "pre-wrap" },
];
}
@@ -146,7 +152,7 @@ export function renderCaptionTrack(
id: "root",
order: 0,
kind: "box" as const,
- style: [
+ style: program.style.leftPercent === undefined || program.style.topPercent === undefined || program.style.widthPercent === undefined ? [
{ name: "align-items", value: "center" },
{ name: "bottom", value: `${program.style.bottomPercent}%` },
{ name: "display", value: "flex" },
@@ -155,6 +161,93 @@ export function renderCaptionTrack(
{ name: "max-width", value: `${program.style.maxWidthPercent}%` },
{ name: "position", value: "absolute" },
{ name: "transform", value: "translateX(-50%)" },
+ ] : [
+ { name: "align-items", value: "center" },
+ { name: "display", value: "flex" },
+ { name: "justify-content", value: "center" },
+ { name: "left", value: `${program.style.leftPercent}%` },
+ { name: "position", value: "absolute" },
+ { name: "top", value: `${program.style.topPercent}%` },
+ { name: "width", value: `${program.style.widthPercent}%` },
+ ],
+ },
+ {
+ id: "text",
+ parent: "root",
+ order: 1,
+ kind: "text" as const,
+ text: unit.display,
+ style: textStyle(program.style),
+ attributes: [
+ { name: "data-caption-basis", value: unit.basis },
+ { name: "data-caption-quality", value: unit.timingQuality },
+ ...(unit.runId === undefined ? [] : [{ name: "data-caption-run", value: unit.runId }]),
+ ...(unit.fields === undefined ? [] : [{ name: "data-caption-fields", value: JSON.stringify(unit.fields) }]),
+ ],
+ },
+ ],
+ }];
+ }),
+ });
+ assertVisualTrackIdentity(track, projection.programSpace);
+ return track;
+}
+
+/** Render one peer Track whose Presents may each select a complete, resolved Caption Style. */
+export function renderCaptionProgram(
+ projection: TimedCaptionProjection,
+ program: CaptionProgram,
+): VisualTrack {
+ assertTimedCaptionProjection(projection);
+ assertCaptionProgram(program);
+ const styles = new Map(program.styles.map((style) => [style.id, style]));
+ const totalFrames = programSpaceFrameCount(projection.programSpace);
+ const units = projection.regions.flatMap((region) => {
+ const styleId = region.styleId ?? program.defaultStyleId;
+ const style = styles.get(styleId);
+ if (style === undefined) throw new Error(`Caption region ${region.id} references unknown Style ${styleId}`);
+ return planCaptionPresentation(
+ { projectionDigest: projection.projectionDigest, regions: [region] },
+ style.presentation.mode,
+ ).units.map((unit) => ({ unit, style }));
+ });
+ const track = sealVisualTrack({
+ contract: "svml.visual-track@1",
+ visualIr: "svml.hyperframes-visual-ir@1",
+ id: program.id,
+ programSpaceDigest: projection.programSpace.digest,
+ presents: units.flatMap(({ unit, style }) => {
+ const startFrame = Math.max(0, frameAt(projection, unit.startSec));
+ const measuredEnd = Math.min(totalFrames, frameAt(projection, unit.endSec));
+ const endFrameExclusive = Math.min(totalFrames, Math.max(startFrame + 1, measuredEnd));
+ if (!unit.display || startFrame >= totalFrames || endFrameExclusive <= startFrame) return [];
+ const appearance = style.presentation.style;
+ return [{
+ id: unit.id,
+ span: { startFrame, endFrameExclusive },
+ stacking: { order: style.presentation.stackingOrder, tieBreak: `${program.id}:${unit.id}` },
+ elements: [
+ {
+ id: "root",
+ order: 0,
+ kind: "box" as const,
+ style: appearance.leftPercent === undefined || appearance.topPercent === undefined || appearance.widthPercent === undefined ? [
+ { name: "align-items", value: "center" },
+ { name: "bottom", value: `${appearance.bottomPercent}%` },
+ { name: "display", value: "flex" },
+ { name: "justify-content", value: "center" },
+ { name: "left", value: "50%" },
+ { name: "max-width", value: `${appearance.maxWidthPercent}%` },
+ { name: "position", value: "absolute" },
+ { name: "transform", value: "translateX(-50%)" },
+ ] : [
+ { name: "align-items", value: "center" },
+ { name: "display", value: "flex" },
+ { name: "justify-content", value: "center" },
+ { name: "left", value: `${appearance.leftPercent}%` },
+ { name: "position", value: "absolute" },
+ { name: "top", value: `${appearance.topPercent}%` },
+ { name: "width", value: `${appearance.widthPercent}%` },
],
},
{
@@ -163,10 +256,13 @@ export function renderCaptionTrack(
order: 1,
kind: "text" as const,
text: unit.display,
- style: textStyle(program),
+ style: textStyle(appearance),
attributes: [
{ name: "data-caption-basis", value: unit.basis },
{ name: "data-caption-quality", value: unit.timingQuality },
+ { name: "data-caption-style", value: style.id },
+ ...(unit.runId === undefined ? [] : [{ name: "data-caption-run", value: unit.runId }]),
+ ...(unit.fields === undefined ? [] : [{ name: "data-caption-fields", value: JSON.stringify(unit.fields) }]),
],
},
],
diff --git a/packages/caption/src/types.ts b/packages/caption/src/types.ts
index 942e252a..859ef2b5 100644
--- a/packages/caption/src/types.ts
+++ b/packages/caption/src/types.ts
@@ -1,6 +1,97 @@
import type { ProgramSpace, TimingQuality } from "@svml/contracts";
import type { Digest } from "@svml/protocol";
+export type CaptionFieldValueSchema =
+ | { readonly kind: "boolean" }
+ | { readonly kind: "enum"; readonly values: readonly string[] }
+ | { readonly kind: "number"; readonly minimum?: number; readonly maximum?: number };
+
+export type CaptionFieldDeclaration = {
+ readonly id: string;
+ readonly value: CaptionFieldValueSchema;
+ readonly instruction: string;
+ readonly minimumPerCue: number;
+ readonly maximumPerCue: number;
+};
+
+export type CaptionPlanningRequirements = {
+ readonly cueInstruction: string;
+ readonly fields: readonly CaptionFieldDeclaration[];
+};
+
+export type CaptionDisplayAtom = {
+ readonly id: string;
+ readonly index: number;
+ readonly regionId: string;
+ readonly segmentId: string;
+ readonly turnId: string;
+ readonly role?: string;
+ readonly text: string;
+ readonly displayStart: number;
+ readonly displayEnd: number;
+ readonly sourceTokenStart: number;
+ readonly sourceTokenEndExclusive: number;
+ readonly correspondence: "exact" | "region-envelope";
+};
+
+export type CaptionFieldAssignment = {
+ readonly declarationId: string;
+ readonly atomId: string;
+ readonly value: string;
+};
+
+export type CaptionPlannedCue = {
+ readonly id: string;
+ readonly atomIds: readonly string[];
+ readonly fields: readonly CaptionFieldAssignment[];
+};
+
+export type CaptionPlannedRun = {
+ readonly id: string;
+ readonly styleId: string;
+ readonly cues: readonly CaptionPlannedCue[];
+};
+
+/** Planner-neutral result. Model/provider identity remains in the producing Graph and Receipt. */
+export type CaptionPlan = {
+ readonly contract: "svml.caption-plan@1";
+ readonly narrativeDigest: Digest;
+ readonly captionProgramDigest: Digest;
+ readonly planningRequestDigest: Digest;
+ readonly runs: readonly CaptionPlannedRun[];
+ readonly planDigest: Digest;
+};
+
+export type CaptionStyleIntent = {
+ readonly contract: "svml.caption-style@1";
+ readonly id: string;
+ readonly planning: CaptionPlanningRequirements;
+ readonly presentation: {
+ readonly mode: CaptionPresentationMode;
+ readonly stackingOrder: number;
+ readonly style: CaptionTrackProgram["style"];
+ };
+ readonly digest: Digest;
+};
+
+export type CaptionProgramRun = {
+ readonly id: string;
+ readonly styleId: string;
+ readonly atomIds: readonly string[];
+};
+
+/** Total, ordered style assignment over one Caption display universe. */
+export type CaptionProgram = {
+ readonly contract: "svml.caption-program@1";
+ readonly id: string;
+ readonly narrativeDigest: Digest;
+ readonly defaultStyleId: string;
+ readonly styles: readonly CaptionStyleIntent[];
+ readonly atoms: readonly CaptionDisplayAtom[];
+ readonly runs: readonly CaptionProgramRun[];
+ readonly digest: Digest;
+};
+
export type TimedCaptionRefinement = {
readonly id: string;
readonly display: string;
@@ -16,6 +107,8 @@ export type TimedCaptionRefinement = {
export type TimedCaptionRegion = {
readonly id: string;
+ readonly runId?: string;
+ readonly styleId?: string;
readonly display: string;
readonly segmentId: string;
readonly kind: "identity" | "alias" | "hidden";
@@ -25,6 +118,7 @@ export type TimedCaptionRegion = {
readonly startQuality: TimingQuality;
readonly endQuality: TimingQuality;
readonly refinements: readonly TimedCaptionRefinement[];
+ readonly fields?: readonly CaptionFieldAssignment[];
};
export type TimedCaptionProjection = {
@@ -40,6 +134,8 @@ export type CaptionPresentationMode = "whole" | "proportional-word" | "character
export type CaptionPresentationUnit = {
readonly id: string;
readonly regionId: string;
+ readonly runId?: string;
+ readonly styleId?: string;
readonly display: string;
readonly displayStart: number;
readonly displayEnd: number;
@@ -47,6 +143,7 @@ export type CaptionPresentationUnit = {
readonly endSec: number;
readonly basis: "region-envelope" | "exact-correspondence" | "presentation-estimate";
readonly timingQuality: TimingQuality;
+ readonly fields?: readonly CaptionFieldAssignment[];
};
export type CaptionPresentationPlan = {
@@ -77,6 +174,10 @@ export type CaptionTrackProgram = {
readonly borderRadiusPx: number;
readonly bottomPercent: number;
readonly maxWidthPercent: number;
+ readonly leftPercent?: number;
+ readonly topPercent?: number;
+ readonly widthPercent?: number;
+ readonly lineHeight?: number;
readonly textAlign: "left" | "center" | "right";
};
};
diff --git a/packages/caption/test/caption.test.ts b/packages/caption/test/caption.test.ts
index 57777409..e8a085d7 100644
--- a/packages/caption/test/caption.test.ts
+++ b/packages/caption/test/caption.test.ts
@@ -6,18 +6,23 @@ import {
captionManifest,
defaultCaptionTrackProgram,
planCaptionPresentation,
+ renderCaptionProgram,
renderCaptionTrack,
+ resolveCaptionProgram,
+ sealCaptionPlan,
+ sealCaptionStyle,
sealCaptionTrackProgram,
temporalizeCaption,
+ temporalizeCaptionPlan,
} from "@svml/caption";
import {
sealAlignedTranscriptEvidence,
sealProgramSpace,
sealSpeechBasis,
} from "@svml/contracts";
-import type { AlignedTranscriptSegment, Narrative, SpeechAudioBasis } from "@svml/contracts";
+import type { AlignedTranscriptSegment, Narrative, NarrativeSelectionRef, SpeechAudioBasis } from "@svml/contracts";
import { digestOf } from "@svml/protocol";
-import { parseScript } from "@svml/script";
+import { narrativeSelectionValue, parseScript } from "@svml/script";
import { locateSpeechTiming } from "@svml/speech-align";
test("the component enumerates every Manifest Producer and owned Type validator", () => {
@@ -166,6 +171,52 @@ test("Caption temporalization preserves evidence envelopes and labels local esti
assert.deepEqual(map, originalMap, "caption presentation must not modify the global speech map");
});
+test("multiple Cues inside one display alias receive ordered local estimates, not duplicate envelopes", () => {
+ const narrative = parseScript("alias-plan.svml", "<line><that was insane | what the fuck></line>");
+ const map = locate(narrative, 1, [{
+ sourceSegmentId: "line",
+ startSec: 0,
+ endSec: 1,
+ words: [
+ { text: "what", startSec: 0.1, endSec: 0.25 },
+ { text: "the", startSec: 0.3, endSec: 0.42 },
+ { text: "fuck", startSec: 0.48, endSec: 0.72 },
+ ],
+ chars: [],
+ }]);
+ const base = defaultCaptionTrackProgram("alias-base");
+ const style = sealCaptionStyle({
+ contract: "svml.caption-style@1",
+ id: "alias-style",
+ planning: { cueInstruction: "One word per Cue for this test.", fields: [] },
+ presentation: { mode: "whole", stackingOrder: 100, style: base.style },
+ });
+ const program = resolveCaptionProgram(narrative, "alias-caption", style, []);
+ const run = program.runs[0]!;
+ const plan = sealCaptionPlan({
+ contract: "svml.caption-plan@1",
+ narrativeDigest: digestOf(narrative),
+ captionProgramDigest: program.digest,
+ planningRequestDigest: digestOf("alias-plan-request"),
+ runs: [{
+ id: run.id,
+ styleId: run.styleId,
+ cues: run.atomIds.map((atomId, index) => ({ id: `alias-cue:${index + 1}`, atomIds: [atomId], fields: [] })),
+ }],
+ });
+ const projection = temporalizeCaptionPlan(narrative, map, program, plan);
+
+ assert.deepEqual(projection.regions.map((region) => region.display), ["that", "was", "insane"]);
+ assert.equal(projection.regions.every((region) =>
+ region.startQuality === "estimated" && region.endQuality === "estimated"), true);
+ assert.equal(projection.regions[0]!.endSec <= projection.regions[1]!.startSec, true);
+ assert.equal(projection.regions[1]!.endSec <= projection.regions[2]!.startSec, true);
+ assert.deepEqual(
+ [projection.regions[0]!.startSec, projection.regions[2]!.endSec],
+ [0.1, 0.72],
+ );
+});
+
test("hidden speech owns time but emits no visible presentation unit", () => {
const narrative = parseScript("hidden.svml", "<line>Hello < | um> world.</line>");
const map = locate(narrative, 1, [{
@@ -237,3 +288,119 @@ test("two caption styles become two peer Tracks without mutating one another", (
assert.equal(firstTrack.presents[0]?.stacking.order, 100);
assert.equal(secondTrack.presents[0]?.stacking.order, 101);
});
+
+test("a planner-neutral CaptionPlan joins SemanticMap only after Cue and field planning", () => {
+ const narrative = parseScript("planned.svml", "<line><ALICE>Meaning becomes the source.</line>");
+ const map = locate(narrative, 2, [{
+ sourceSegmentId: "line",
+ startSec: 0,
+ endSec: 2,
+ words: [
+ { text: "Meaning", startSec: 0.1, endSec: 0.35 },
+ { text: "becomes", startSec: 0.4, endSec: 0.7 },
+ { text: "the", startSec: 0.8, endSec: 1.0 },
+ { text: "source", startSec: 1.1, endSec: 1.5 },
+ ],
+ chars: [],
+ }]);
+ const base = defaultCaptionTrackProgram("planned-caption");
+ const style = sealCaptionStyle({
+ contract: "svml.caption-style@1",
+ id: "alice",
+ planning: {
+ cueInstruction: "Use short semantic phrases.",
+ fields: [{
+ id: "important",
+ value: { kind: "boolean" },
+ instruction: "Select at most one important word.",
+ minimumPerCue: 0,
+ maximumPerCue: 1,
+ }],
+ },
+ presentation: { mode: "whole", stackingOrder: 100, style: base.style },
+ });
+ const program = resolveCaptionProgram(narrative, "planned-caption", style, []);
+ const ids = program.atoms.map((atom) => atom.id);
+ const plan = sealCaptionPlan({
+ contract: "svml.caption-plan@1",
+ narrativeDigest: digestOf(narrative),
+ captionProgramDigest: program.digest,
+ planningRequestDigest: digestOf("planned-caption-request"),
+ runs: [{
+ id: program.runs[0]!.id,
+ styleId: style.id,
+ cues: [
+ {
+ id: "cue:1",
+ atomIds: ids.slice(0, 2),
+ fields: [{ declarationId: "important", atomId: ids[0]!, value: "true" }],
+ },
+ { id: "cue:2", atomIds: ids.slice(2), fields: [] },
+ ],
+ }],
+ });
+ const projection = temporalizeCaptionPlan(narrative, map, program, plan);
+ assert.deepEqual(projection.regions.map((region) => region.display), ["Meaning becomes", "the source."]);
+ assert.deepEqual(projection.regions[0]?.fields?.map((field) => field.declarationId), ["important"]);
+ const track = renderCaptionProgram(projection, program);
+ const firstText = track.presents[0]?.elements.find((element) => element.kind === "text");
+ assert.equal(firstText?.attributes?.some((attribute) =>
+ attribute.name === "data-caption-style" && attribute.value === "alice"), true);
+ assert.equal(firstText?.attributes?.some((attribute) =>
+ attribute.name === "data-caption-fields" && attribute.value.includes("important")), true);
+});
+
+test("a default Style covers roleless text and ordered whole-style overrides use last match", () => {
+ const narrative = parseScript("cascade.svml", `
+ <intro>Roleless words stay readable.</intro>
+ <answer><ALICE>Only this sentence changes.</answer>
+ `);
+ const base = defaultCaptionTrackProgram("base");
+ const make = (id: string, color: string) => sealCaptionStyle({
+ contract: "svml.caption-style@1",
+ id,
+ planning: { cueInstruction: `Plan ${id}.`, fields: [] },
+ presentation: { mode: "whole", stackingOrder: 100, style: { ...base.style, color } },
+ });
+ const normal = make("normal", "#ffffff");
+ const alice = make("alice", "#00ff00");
+ const final = make("final", "#ff00ff");
+ const program = resolveCaptionProgram(narrative, "captions", normal, [
+ { id: "alice", selector: { kind: "role", role: "ALICE" }, style: alice },
+ { id: "alice-later", selector: { kind: "role", role: "ALICE" }, style: final },
+ ]);
+ assert.deepEqual(program.runs.map((run) => run.styleId), ["normal", "final"]);
+ assert.equal(program.styles.some((style) => style.id === "alice"), true, "declared overrides stay auditable");
+ assert.throws(() => resolveCaptionProgram(narrative, "invalid", normal, [{
+ id: "typo",
+ selector: { kind: "role", role: "ALIEC" },
+ style: alice,
+ }]), /selects no visible display atom/u);
+});
+
+test("an explicit Selection replaces only its words without authoring the complement", () => {
+ const narrative = parseScript(
+ "selection-style.svml",
+ "<line>Keep this @special one sentence different @/special and return.</line>",
+ );
+ const base = defaultCaptionTrackProgram("base");
+ const make = (id: string, color: string) => sealCaptionStyle({
+ contract: "svml.caption-style@1",
+ id,
+ planning: { cueInstruction: `Plan ${id}.`, fields: [] },
+ presentation: { mode: "whole", stackingOrder: 100, style: { ...base.style, color } },
+ });
+ const normal = make("normal", "#ffffff");
+ const special = make("special", "#ff00ff");
+ const selection = narrativeSelectionValue(narrative.selections[0]!) as unknown as NarrativeSelectionRef;
+ const program = resolveCaptionProgram(narrative, "captions", normal, [{
+ id: "special-use",
+ selector: { kind: "selection", selection },
+ style: special,
+ }]);
+ assert.deepEqual(program.runs.map((run) => [run.styleId, run.atomIds.length]), [
+ ["normal", 2],
+ ["special", 3],
+ ["normal", 2],
+ ]);
+});
diff --git a/packages/cli/README.md b/packages/cli/README.md
index fac82473..96f28694 100644
--- a/packages/cli/README.md
+++ b/packages/cli/README.md
@@ -9,7 +9,8 @@ itself. The current prelude contributes:
- the official Text Frontend;
- the Script raw Surface and its audiovisual Narrative contract dependency;
- the SVS alternate Frontend and generic Recipe manifest;
-- the official Film and HyperFrames Render Structured Surfaces plus their exact static dependencies.
+- the official Speech, WhisperX, provider-free Caption, B-roll, Text, Film and HyperFrames Render
+ Structured Surfaces plus their exact static dependencies.
It does not place these packages in Core and does not grant source imports network, credential or
process permissions.
@@ -23,14 +24,15 @@ pnpm svml:v2 check path/to/main.svml --package-lock ./svml.packages.lock --root
pnpm svml:v2 plan path/to/main.svml --target component.result
pnpm svml:v2 build path/to/main.svml --target component.result --runtime ./svml.runtime.ts
pnpm svml:v2 build path/to/main.svml --target component.result --runtime ./svml.runtime.ts --follow
+pnpm svml:v2 build path/to/main.svml --target final.video --runtime ./svml.runtime.ts --follow --out ./final.mp4
pnpm svml:v2 status <build-id> --runtime ./svml.runtime.ts
pnpm svml:v2 cancel <build-id> --runtime ./svml.runtime.ts
```
-`check` is usable for the implemented Script, authored Image, Seedance generation, Film, Render and
-recursively imported `.svs` sources. `plan` is fully implemented by the generic Node compiler. A
-complete author video still awaits Speech assembly and Track Surfaces that produce Film's inputs;
-the CLI does not fabricate those missing facts.
+`check` is usable for the complete provider-free author graph in
+`examples/talking-film-graph-check`. `plan` is fully implemented by the generic Node compiler. The
+production golden still awaits the explicit Gemini Caption package and real assets; the CLI does
+not fabricate those missing facts.
`build` compiles the same locked BuildState and passes it to a trusted local Runtime config module.
The default Build identity is content-derived, so the same invocation resumes durable local state;
@@ -43,6 +45,10 @@ non-retryable failure, cancellation or `--max-wait-ms`. `status` reads durable v
`cancel` invokes active endpoints and then records the resulting terminal Build failure. None of
these commands creates a second ready-command queue.
+`--out` accepts exactly one completed media target, reads its content-addressed bytes through the
+selected Runtime ArtifactStore and writes the requested file. It does not treat a CAS path as a
+public filename or bypass Record identity verification.
+
`@svml/package-loader-node` supports explicitly trusted installed implementation packages. It
verifies the complete physical dependency closure before executing an activation entry, then checks
exact Module, Frontend, Surface, Producer and Validator identities. `plan` and `build` bind the lock
diff --git a/packages/cli/package.json b/packages/cli/package.json
index 86c40e58..29505769 100644
--- a/packages/cli/package.json
+++ b/packages/cli/package.json
@@ -10,6 +10,7 @@
"@svml/compiler-node": "workspace:*",
"@svml/local": "workspace:*",
"@svml/package-loader-node": "workspace:*",
- "@svml/prelude-video": "workspace:*"
+ "@svml/prelude-video": "workspace:*",
+ "@svml/protocol": "workspace:*"
}
}
diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts
index 924cce86..5f2d466d 100644
--- a/packages/cli/src/index.ts
+++ b/packages/cli/src/index.ts
@@ -1,3 +1,3 @@
export { createOfficialNodeCompiler } from "./host.js";
export type { OfficialCompilerOptions } from "./host.js";
-export { runCli } from "./main.js";
+export { materializeSingleGoal, runCli } from "./main.js";
diff --git a/packages/cli/src/main.ts b/packages/cli/src/main.ts
index fbac9f30..467da4f9 100644
--- a/packages/cli/src/main.ts
+++ b/packages/cli/src/main.ts
@@ -1,7 +1,9 @@
+import { mkdir, writeFile } from "node:fs/promises";
import { dirname, resolve } from "node:path";
import { pathToFileURL } from "node:url";
import type { LocalRuntime } from "@svml/local";
+import { isDigest } from "@svml/protocol";
import {
createNodePackageLock,
loadNodePackageSet,
@@ -26,6 +28,7 @@ type ParsedArgs = {
readonly maxWaitMs: number | undefined;
readonly packageLock: string | undefined;
readonly packages: readonly string[];
+ readonly out: string | undefined;
};
function parseArgs(argv: readonly string[]): ParsedArgs {
@@ -39,6 +42,7 @@ function parseArgs(argv: readonly string[]): ParsedArgs {
let packageLock: string | undefined;
const packages: string[] = [];
let substitute = false;
+ let out: string | undefined;
for (let index = 0; index < rest.length; index += 1) {
const item = rest[index];
if (item === "--target") {
@@ -80,6 +84,13 @@ function parseArgs(argv: readonly string[]): ParsedArgs {
index += 1;
continue;
}
+ if (item === "--out") {
+ const value = rest[index + 1];
+ if (value === undefined || value.startsWith("--")) throw new Error("--out requires a file path");
+ out = resolve(value);
+ index += 1;
+ continue;
+ }
if (item === "--build-id") {
const value = rest[index + 1];
if (value === undefined || value.startsWith("--")) throw new Error("--build-id requires a stable identity");
@@ -115,6 +126,7 @@ function parseArgs(argv: readonly string[]): ParsedArgs {
maxWaitMs,
packageLock,
packages,
+ out,
};
}
@@ -124,7 +136,7 @@ function usage(): string {
" svml-v2 lock-packages <svml.packages.lock> --package installed-name [--package installed-name] [--root directory]",
" svml-v2 check <file.svml> [--package-lock file] [--root directory]",
" svml-v2 plan <file.svml> --target export [--target export] [--accept-substitute] [--package-lock file]",
- " svml-v2 build <file.svml> --target export --runtime ./svml.runtime.ts [--package-lock file] [--follow]",
+ " svml-v2 build <file.svml> --target export --runtime ./svml.runtime.ts [--package-lock file] [--follow] [--out video.mp4]",
" svml-v2 status <build-id> --runtime ./svml.runtime.ts",
" svml-v2 cancel <build-id> --runtime ./svml.runtime.ts",
].join("\n");
@@ -135,6 +147,8 @@ function isLocalRuntime(value: unknown): value is LocalRuntime {
&& value !== null
&& "build" in value
&& typeof value.build === "function"
+ && "readArtifact" in value
+ && typeof value.readArtifact === "function"
&& "close" in value
&& typeof value.close === "function";
}
@@ -154,6 +168,33 @@ async function loadLocalRuntime(path: string): Promise<LocalRuntime> {
return candidate;
}
+export async function materializeSingleGoal(
+ runtime: Pick<LocalRuntime, "readArtifact">,
+ built: Awaited<ReturnType<LocalRuntime["build"]>>,
+ output: string,
+): Promise<{ readonly path: string; readonly digest: string; readonly size: number }> {
+ if (built.status !== "complete") throw new Error("--out requires a completed Build");
+ if (built.state.plan.goals.length !== 1) throw new Error("--out requires exactly one target");
+ const goal = built.state.plan.goals[0]!;
+ const record = built.state.records.find((item) => item.id === goal.record);
+ const value = record?.value.kind === "inline" ? record.value.value : undefined;
+ if (value === null || Array.isArray(value) || typeof value !== "object") {
+ throw new Error("--out target is not a materializable media Artifact");
+ }
+ const artifact = value as Readonly<Record<string, unknown>>;
+ const digest = artifact.digest;
+ const size = artifact.size;
+ if (typeof digest !== "string" || !isDigest(digest) || typeof size !== "number" || !Number.isSafeInteger(size)) {
+ throw new Error("--out target has no valid Artifact identity");
+ }
+ const bytes = await runtime.readArtifact(digest);
+ if (bytes === undefined) throw new Error(`Artifact ${digest} is absent from the selected ArtifactStore`);
+ if (bytes.byteLength !== size) throw new Error(`Artifact ${digest} size differs from its target Record`);
+ await mkdir(dirname(output), { recursive: true });
+ await writeFile(output, bytes);
+ return { path: output, digest, size };
+}
+
export async function runCli(argv: readonly string[], io: CliIo): Promise<void> {
const args = parseArgs(argv);
if (args.file === undefined
@@ -171,6 +212,7 @@ export async function runCli(argv: readonly string[], io: CliIo): Promise<void>
io.write(`${JSON.stringify({ ok: true, packageLock: output, digest: lock.digest, packages: lock.packages }, null, 2)}\n`);
return;
}
+ if (args.out !== undefined && args.command !== "build") throw new Error("--out is only valid for build");
if (args.command === "status" || args.command === "cancel") {
if (args.runtime === undefined) throw new Error(`${args.command} requires --runtime`);
const runtime = await loadLocalRuntime(args.runtime);
@@ -248,6 +290,10 @@ export async function runCli(argv: readonly string[], io: CliIo): Promise<void>
follow: args.follow,
...(args.maxWaitMs === undefined ? {} : { maxWaitMs: args.maxWaitMs }),
});
+ let materialized: { readonly path: string; readonly digest: string; readonly size: number } | undefined;
+ if (args.out !== undefined) {
+ materialized = await materializeSingleGoal(runtime, built, args.out);
+ }
io.write(`${JSON.stringify({
build: built.id,
core: built.state.id,
@@ -263,6 +309,7 @@ export async function runCli(argv: readonly string[], io: CliIo): Promise<void>
}),
blocked: built.blocked,
journal: built.journal,
+ ...(materialized === undefined ? {} : { output: materialized }),
}, null, 2)}\n`);
} finally {
await runtime.close();
diff --git a/packages/cli/test/cli.test.ts b/packages/cli/test/cli.test.ts
index b3604fa9..033e5863 100644
--- a/packages/cli/test/cli.test.ts
+++ b/packages/cli/test/cli.test.ts
@@ -1,10 +1,11 @@
import assert from "node:assert/strict";
-import { mkdir, mkdtemp, writeFile } from "node:fs/promises";
+import { createHash } from "node:crypto";
+import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
-import { runCli } from "@svml/cli";
+import { materializeSingleGoal, runCli } from "@svml/cli";
test("official v2 CLI checks a real Script source through the Node compiler host", async () => {
const root = await mkdtemp(join(tmpdir(), "svml-cli-"));
@@ -44,10 +45,22 @@ test("official v2 CLI checks a real Script source through the Node compiler host
module: { name: "@svml/narrative", version: "0.0.0-dev" },
name: "Narrative",
}, kind: "record" },
+ { name: "story.caption", type: {
+ module: { name: "@svml/narrative", version: "0.0.0-dev" },
+ name: "CaptionProjection",
+ }, kind: "record" },
{ name: "story.segment.opening", type: {
module: { name: "@svml/narrative", version: "0.0.0-dev" },
name: "NarrativeExcerpt",
}, kind: "record" },
+ { name: "story.segment.opening.dialogue", type: {
+ module: { name: "@svml/narrative", version: "0.0.0-dev" },
+ name: "NarrativeDialogueExcerpt",
+ }, kind: "record" },
+ { name: "story.segment.opening.speech", type: {
+ module: { name: "@svml/narrative", version: "0.0.0-dev" },
+ name: "NarrativeSpeechExcerpt",
+ }, kind: "record" },
]);
});
@@ -64,7 +77,7 @@ test("official media and Seedance Surfaces lower author intent into exact genera
<seedance:Prompt id="direction">
Locked medium close-up in a quiet daylight studio.
</seedance:Prompt>
- <seedance:Speech id="take" model="mini" script={story.segment.opening}
+ <seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue}
prompt={direction} duration="5">
<seedance:Reference image={host} role="character"/>
</seedance:Speech>
@@ -86,7 +99,10 @@ test("official media and Seedance Surfaces lower author intent into exact genera
"motion.request",
"motion.video",
"story",
+ "story.caption",
"story.segment.opening",
+ "story.segment.opening.dialogue",
+ "story.segment.opening.speech",
"take",
"take.request",
]);
@@ -113,6 +129,132 @@ test("official media and Seedance Surfaces lower author intent into exact genera
);
});
+test("official prelude lowers Speech Spine and explicit WhisperX alignment without provider calls", async () => {
+ const root = await mkdtemp(join(tmpdir(), "svml-cli-speech-spine-"));
+ const file = join(root, "main.svml");
+ await writeFile(file, `<svml>
+ <import from="@svml/script@1"/>
+ <import as="seedance" from="@svml/seedance@1"/>
+ <import as="speech" from="@svml/speech@1"/>
+ <import as="whisperx" from="@svml/whisperx@1"/>
+ <script id="story"><opening><HOST>Meaning becomes the source.</opening></script>
+ <seedance:Prompt id="direction">Locked medium close-up.</seedance:Prompt>
+ <seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue}
+ prompt={direction} duration="5"/>
+ <speech:Spine id="speech">
+ <speech:Take source={take} segment={story.segment.opening}/>
+ </speech:Spine>
+ <whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>
+ </svml>`, "utf8");
+ let output = "";
+ await runCli(["check", file], { write: (text) => { output += text; } });
+ const checked = JSON.parse(output) as { readonly exports: readonly { readonly name: string }[] };
+ assert.equal(checked.exports.some((item) => item.name === "speech.visual"), true);
+ assert.equal(checked.exports.some((item) => item.name === "timing.map"), true);
+});
+
+test("Caption Gemini Surface lowers immutable display-atom runs into one explicit planning Need", async () => {
+ const root = await mkdtemp(join(tmpdir(), "svml-cli-caption-gemini-"));
+ const file = join(root, "main.svml");
+ await writeFile(join(root, "studio.svs"), `<sheet version="1">
+ caption.base { stack-order: 70; x: 0.08; y: 0.76; width: 0.84; font: Inter;
+ weight: 600; size: 58; line-height: 1; align: center; fill: #FFFFFF;
+ background: #09090BCC; padding: 16 24; radius: 18; }
+ </sheet>`, "utf8");
+ await writeFile(file, `<svml>
+ <import from="@svml/script@1"/>
+ <import as="caption" from="@svml/caption@1"/>
+ <import as="gemini" from="@svml/caption-gemini@1"/>
+ <import as="studio" from="./studio.svs" using="@svml/svs@1"/>
+ <script id="story"><dialogue><ALICE>Meaning becomes the source.<BOB>Then the graph stays explicit.</dialogue></script>
+ <caption:Style id="plain" appearance={studio.caption.base}>
+ <caption:Cues>Prefer short complete semantic phrases. Never cross clause punctuation.</caption:Cues>
+ </caption:Style>
+ <caption:Style id="emphasis" appearance={studio.caption.base}>
+ <caption:Cues>Prefer short complete semantic phrases. Never cross clause punctuation.</caption:Cues>
+ <caption:Field id="important" type="boolean" min-per-cue="0" max-per-cue="2">
+ Select zero, one, or two words whose emphasis best communicates this Cue.
+ </caption:Field>
+ </caption:Style>
+ <caption:Program id="caption-program" narrative={story} default={plain}>
+ <caption:Use role="ALICE" style={emphasis}/>
+ </caption:Program>
+ <gemini:Planner id="dialogue-plan" narrative={story} program={caption-program} model="gemini-2.5-flash"/>
+ </svml>`, "utf8");
+ let checkedOutput = "";
+ await runCli(["check", file], { write: (text) => { checkedOutput += text; } });
+ const checked = JSON.parse(checkedOutput) as { readonly exports: readonly { readonly name: string }[] };
+ assert.equal(checked.exports.some((item) => item.name === "caption-program"), true);
+ assert.equal(checked.exports.some((item) => item.name === "dialogue-plan.plan"), true);
+ let planOutput = "";
+ await runCli(["plan", file, "--target", "dialogue-plan.plan"], { write: (text) => { planOutput += text; } });
+ const plan = JSON.parse(planOutput) as { readonly steps: readonly { readonly producer: { readonly name: string } }[] };
+ assert.deepEqual(plan.steps.map((step) => step.producer.name).sort(), [
+ "compile-caption-gemini-request",
+ "request-caption-gemini-plan",
+ ].sort());
+});
+
+test("Track Surfaces close one complete author graph before any external execution", async () => {
+ const root = await mkdtemp(join(tmpdir(), "svml-cli-complete-author-"));
+ const file = join(root, "main.svml");
+ await writeFile(join(root, "studio.svs"), `<sheet version="1">
+ film.vertical { width: 1080; height: 1920; frame-rate: 30; background: #09090B; }
+ broll.card { stack-order: 40; x: 0.1; y: 0.2; width: 0.8; height: 0.5; fit: cover;
+ background: #111116; radius: 20; enter: slide-up 4f; exit: fade 4f; }
+ caption.base { stack-order: 70; x: 0.08; y: 0.76; width: 0.84; font: Inter;
+ weight: 600; size: 58; line-height: 1; align: center; fill: #FFFFFF;
+ background: #09090BCC; padding: 16 24; radius: 18; }
+ text.title { stack-order: 90; x: 0.06; y: 0.06; width: 0.88; height: 0.1;
+ font: Inter; weight: 900; size: 64; align: center; fill: #FFFFFF; tracking: -1; }
+ </sheet>`, "utf8");
+ await writeFile(file, `<svml>
+ <import from="@svml/script@1"/>
+ <import as="seedance" from="@svml/seedance@1"/>
+ <import as="speech" from="@svml/speech@1"/>
+ <import as="whisperx" from="@svml/whisperx@1"/>
+ <import as="broll" from="@svml/broll@1"/>
+ <import as="caption" from="@svml/caption@1"/>
+ <import as="caption-ai" from="@svml/caption-gemini@1"/>
+ <import as="text" from="@svml/text-track@1"/>
+ <import as="film" from="@svml/film@1"/>
+ <import as="render" from="@svml/hyperframes-render@1"/>
+ <import as="studio" from="./studio.svs" using="@svml/svs@1"/>
+ <script id="story"><opening><HOST>Meaning @demo becomes the source @/demo.</opening></script>
+ <seedance:Prompt id="direction">Locked medium close-up.</seedance:Prompt>
+ <seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue} prompt={direction} duration="5"/>
+ <seedance:Video id="motion" model="mini" prompt={direction} duration="5"/>
+ <speech:Spine id="speech"><speech:Take source={take} segment={story.segment.opening}/></speech:Spine>
+ <whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>
+ <caption:Style id="base-caption" appearance={studio.caption.base}>
+ <caption:Cues>Prefer short complete semantic phrases.</caption:Cues>
+ <caption:Field id="important" type="boolean" min-per-cue="0" max-per-cue="2">
+ Select zero, one, or two words whose emphasis best communicates this Cue.
+ </caption:Field>
+ </caption:Style>
+ <caption:Program id="caption-program" narrative={story} default={base-caption}/>
+ <caption-ai:Planner id="cue-plan" narrative={story} program={caption-program} model="gemini-2.5-flash"/>
+ <broll:Track id="cards" map={timing.map}>
+ <broll:Item source={motion.video} during={story.selection.demo} appearance={studio.broll.card}/>
+ </broll:Track>
+ <caption:Track id="captions" narrative={story} map={timing.map} plan={cue-plan.plan} program={caption-program}/>
+ <text:Track id="titles" space={speech.space}>
+ <text:Item text="MEANING" during="full" appearance={studio.text.title}/>
+ </text:Track>
+ <film:Film id="main" space={speech.space} appearance={studio.film.vertical}>
+ <film:Track source={speech.visual}/><film:Track source={speech.audioTrack}/>
+ <film:Track source={cards.visual}/><film:Track source={captions.track}/><film:Track source={titles.track}/>
+ </film:Film>
+ <render:Video id="final" composition={main.composition}/>
+ </svml>`, "utf8");
+ let output = "";
+ await runCli(["check", file], { write: (text) => { output += text; } });
+ const checked = JSON.parse(output) as { readonly exports: readonly { readonly name: string }[] };
+ for (const name of ["story.selection.demo", "cue-plan.plan", "cards.visual", "captions.track", "titles.track", "final.video"]) {
+ assert.equal(checked.exports.some((item) => item.name === name), true, name);
+ }
+});
+
test("official v2 CLI closes the explicit HyperFrames render package without loading a Provider", async () => {
const root = await mkdtemp(join(tmpdir(), "svml-cli-render-"));
const file = join(root, "main.svml");
@@ -194,3 +336,26 @@ test("CLI package lock activates an installed package without changing the offic
assert.equal(checked.ok, true);
assert.deepEqual(checked.modules, ["example.empty@1"]);
});
+
+test("materializeSingleGoal writes one verified target through the Runtime ArtifactStore seam", async () => {
+ const root = await mkdtemp(join(tmpdir(), "svml-cli-out-"));
+ const bytes = Buffer.from("final-video-bytes");
+ const artifactDigest = `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
+ const output = join(root, "out", "final.mp4");
+ const runtime = {
+ async readArtifact(digest: string) { return digest === artifactDigest ? bytes : undefined; },
+ } as Parameters<typeof materializeSingleGoal>[0];
+ const built = {
+ status: "complete",
+ state: {
+ plan: { goals: [{ record: "final.video" }] },
+ records: [{
+ id: "final.video",
+ value: { kind: "inline", value: { digest: artifactDigest, size: bytes.byteLength, mediaType: "video/mp4" } },
+ }],
+ },
+ } as unknown as Parameters<typeof materializeSingleGoal>[1];
+ const result = await materializeSingleGoal(runtime, built, output);
+ assert.deepEqual(await readFile(output), bytes);
+ assert.deepEqual(result, { path: output, digest: artifactDigest, size: bytes.byteLength });
+});
diff --git a/packages/contracts/README.md b/packages/contracts/README.md
index bc979e2e..e32ab5c0 100644
--- a/packages/contracts/README.md
+++ b/packages/contracts/README.md
@@ -14,10 +14,13 @@ and may later become `@svml/video` without changing those logical identities. Th
contains Provider- and Frontend-neutral data contracts, identity validators and static schemas
only. It does not parse SVML, execute external services, align speech or render Tracks.
-`NarrativeExcerpt` is the shared contiguous author-selection contract used by
-`script.segment.<id>` and speech-capable generation packages. It keeps model packages independent
-of the official Script parser; the actual `<media:Image>` Surface implementation likewise lives in
-the separate physical `@svml/media` package.
+The Narrative module provides distinct narrow views instead of making every consumer depend on one
+parser-specific object: `NarrativeExcerpt` associates a Take with a Segment,
+`NarrativeDialogueExcerpt` carries Role-aware spoken prompt text,
+`NarrativeSpeechExcerpt` carries pronunciation-only estimate text, and `CaptionProjectionRef`
+identifies the whole display-side Caption truth. They keep model and Caption packages independent of
+the official Script parser. The actual `<media:Image>` Surface implementation likewise lives in the
+separate physical `@svml/media` package.
`VisualTrack`, `AudioTrack` and `Composition` implement the flat composition law in
[`../../spec/track-composition-v1.md`](../../spec/track-composition-v1.md). Caption, Speech and
diff --git a/packages/contracts/src/index.ts b/packages/contracts/src/index.ts
index d9f4c0c3..2ace3cdc 100644
--- a/packages/contracts/src/index.ts
+++ b/packages/contracts/src/index.ts
@@ -80,6 +80,10 @@ export {
timelineAudioSchema,
narrativeSchema,
narrativeExcerptSchema,
+ narrativeDialogueExcerptSchema,
+ narrativeSpeechExcerptSchema,
+ narrativeSelectionSchema,
+ captionProjectionSchema,
programSpaceSchema,
speechAudioBasisSchema,
speechEvidenceAudioSchema,
diff --git a/packages/contracts/src/manifest.ts b/packages/contracts/src/manifest.ts
index 34b196c1..9dbb659c 100644
--- a/packages/contracts/src/manifest.ts
+++ b/packages/contracts/src/manifest.ts
@@ -17,6 +17,10 @@ export const compositionModuleRef = { name: "@svml/composition", version: "0.0.0
export const contractTypes = {
narrative: { module: narrativeModuleRef, name: "Narrative" },
narrativeExcerpt: { module: narrativeModuleRef, name: "NarrativeExcerpt" },
+ narrativeDialogueExcerpt: { module: narrativeModuleRef, name: "NarrativeDialogueExcerpt" },
+ narrativeSpeechExcerpt: { module: narrativeModuleRef, name: "NarrativeSpeechExcerpt" },
+ narrativeSelection: { module: narrativeModuleRef, name: "NarrativeSelection" },
+ captionProjection: { module: narrativeModuleRef, name: "CaptionProjection" },
blobArtifact: { module: mediaModuleRef, name: "BlobArtifact" },
mediaArtifact: { module: mediaModuleRef, name: "MediaArtifactRef" },
mediaInspection: { module: mediaModuleRef, name: "MediaInspection" },
@@ -127,6 +131,51 @@ export const narrativeExcerptSchema: ValueSchema = object({
excerptDigest: { schema: digest },
});
+export const narrativeDialogueExcerptSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.narrative-dialogue-excerpt@1" } },
+ kind: { schema: { kind: "literal", value: "segment" } },
+ id: { schema: string },
+ tokenStart: { schema: integer },
+ tokenEndExclusive: { schema: integer },
+ dialogue: { schema: string },
+ excerptDigest: { schema: digest },
+});
+
+export const narrativeSpeechExcerptSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.narrative-speech-excerpt@1" } },
+ kind: { schema: { kind: "literal", value: "segment" } },
+ id: { schema: string },
+ tokenStart: { schema: integer },
+ tokenEndExclusive: { schema: integer },
+ speech: { schema: string },
+ excerptDigest: { schema: digest },
+});
+
+export const captionProjectionSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.caption-projection@0" } },
+ text: { schema: { kind: "string" } },
+ regions: { schema: { kind: "array", items: object({
+ id: { schema: string }, display: { schema: { kind: "string" } }, segmentId: { schema: string },
+ startToken: { schema: integer }, endTokenExclusive: { schema: integer },
+ kind: { schema: { kind: "string", enum: ["identity", "alias", "hidden"] } },
+ refinements: { schema: { kind: "array", items: object({
+ id: { schema: string }, display: { schema: string }, displayStart: { schema: integer },
+ displayEnd: { schema: integer }, startToken: { schema: integer }, endTokenExclusive: { schema: integer },
+ relation: { schema: { kind: "literal", value: "exact" } },
+ }) } },
+ }) } },
+ projectionDigest: { schema: digest },
+});
+
+export const narrativeSelectionSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.narrative-selection@1" } },
+ id: { schema: string },
+ occurrences: { schema: { kind: "array", minItems: 1, items: object({
+ occurrence: { schema: integer }, open: { schema: markerEdge }, close: { schema: markerEdge },
+ }) } },
+ selectionDigest: { schema: digest },
+});
+
export const mediaArtifactSchema = object({
digest: { schema: digest }, size: { schema: integer }, mediaType: { schema: string }, durationSec: { schema: number },
});
@@ -543,6 +592,10 @@ export const narrativeManifest: ModuleManifest = {
types: [
{ name: contractTypes.narrative.name, schema: narrativeSchema },
{ name: contractTypes.narrativeExcerpt.name, schema: narrativeExcerptSchema },
+ { name: contractTypes.narrativeDialogueExcerpt.name, schema: narrativeDialogueExcerptSchema },
+ { name: contractTypes.narrativeSpeechExcerpt.name, schema: narrativeSpeechExcerptSchema },
+ { name: contractTypes.narrativeSelection.name, schema: narrativeSelectionSchema },
+ { name: contractTypes.captionProjection.name, schema: captionProjectionSchema },
],
capabilities: [],
surfaces: [],
diff --git a/packages/contracts/src/narrative.ts b/packages/contracts/src/narrative.ts
index 6054ae19..4b80943b 100644
--- a/packages/contracts/src/narrative.ts
+++ b/packages/contracts/src/narrative.ts
@@ -45,6 +45,12 @@ export type NarrativeSelection = {
readonly occurrences: readonly NarrativeSelectionOccurrence[];
};
+/** One explicitly authored semantic window, independently referenceable by graph edges. */
+export type NarrativeSelectionRef = NarrativeSelection & {
+ readonly contract: "svml.narrative-selection@1";
+ readonly selectionDigest: Digest;
+};
+
export type NarrativeMomentOccurrence = {
readonly occurrence: number;
readonly affinity: Affinity;
@@ -82,6 +88,11 @@ export type CaptionProjection = {
readonly regions: readonly CaptionRegion[];
};
+/** Whole authored display projection. It contains no pronunciation replacement text. */
+export type CaptionProjectionRef = CaptionProjection & {
+ readonly projectionDigest: Digest;
+};
+
export type SemanticAnchor = {
readonly id: string;
readonly kind: "segment-start" | "token-start" | "token-end" | "segment-end";
@@ -126,4 +137,26 @@ export type NarrativeExcerpt = {
};
readonly excerptDigest: Digest;
};
+
+/** The exact dialogue prompt for one Segment, including only explicitly authored Role prefixes. */
+export type NarrativeDialogueExcerpt = {
+ readonly contract: "svml.narrative-dialogue-excerpt@1";
+ readonly kind: "segment";
+ readonly id: string;
+ readonly tokenStart: number;
+ readonly tokenEndExclusive: number;
+ readonly dialogue: string;
+ readonly excerptDigest: Digest;
+};
+
+/** The exact spoken wording for one Segment, without Role prefixes. */
+export type NarrativeSpeechExcerpt = {
+ readonly contract: "svml.narrative-speech-excerpt@1";
+ readonly kind: "segment";
+ readonly id: string;
+ readonly tokenStart: number;
+ readonly tokenEndExclusive: number;
+ readonly speech: string;
+ readonly excerptDigest: Digest;
+};
import type { Digest } from "@svml/protocol";
diff --git a/packages/film/package.json b/packages/film/package.json
index 5a72c96e..fd2fd720 100644
--- a/packages/film/package.json
+++ b/packages/film/package.json
@@ -7,6 +7,7 @@
".": "./src/index.ts"
},
"dependencies": {
+ "@svml/component-kit": "workspace:*",
"@svml/contracts": "workspace:*",
"@svml/elaborator": "workspace:*",
"@svml/protocol": "workspace:*",
diff --git a/packages/film/src/component.ts b/packages/film/src/component.ts
new file mode 100644
index 00000000..2c7e1267
--- /dev/null
+++ b/packages/film/src/component.ts
@@ -0,0 +1,66 @@
+import type { ComponentPackage } from "@svml/component-kit";
+import type { AudioTrack, ProgramSpace, VisualTrack } from "@svml/contracts";
+import type { StoredValue } from "@svml/protocol";
+import { canonicalize } from "@svml/protocol";
+
+import { filmProducers } from "./manifest.js";
+import {
+ appendFilmAudioTrack,
+ appendFilmAudioTrackImplementationDigest,
+ appendFilmVisualTrack,
+ appendFilmVisualTrackImplementationDigest,
+ compileFilmComposition,
+ compileFilmCompositionImplementationDigest,
+ createFilmTrackSet,
+ createFilmTrackSetImplementationDigest,
+} from "./program.js";
+import type { FilmProgram, FilmTrackSet } from "./types.js";
+
+function inline<T>(value: StoredValue | undefined, subject: string): T {
+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);
+ return value.value as unknown as T;
+}
+
+export const filmComponent = {
+ name: "@svml/film",
+ producers: [
+ {
+ producer: filmProducers.createTrackSet,
+ implementationDigest: createFilmTrackSetImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { set: { kind: "inline", value: canonicalize(createFilmTrackSet(inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"))) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: filmProducers.appendVisualTrack,
+ implementationDigest: appendFilmVisualTrackImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { set: { kind: "inline", value: canonicalize(appendFilmVisualTrack(
+ inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"), inline<VisualTrack>(inputs.track?.value, "VisualTrack"),
+ )) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: filmProducers.appendAudioTrack,
+ implementationDigest: appendFilmAudioTrackImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { set: { kind: "inline", value: canonicalize(appendFilmAudioTrack(
+ inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"), inline<AudioTrack>(inputs.track?.value, "AudioTrack"),
+ )) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: filmProducers.compileComposition,
+ implementationDigest: compileFilmCompositionImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { composition: { kind: "inline", value: canonicalize(compileFilmComposition(
+ inline<FilmProgram>(inputs.program?.value, "FilmProgram"), inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"),
+ )) } },
+ needs: {},
+ }),
+ },
+ ],
+} satisfies ComponentPackage;
diff --git a/packages/film/src/index.ts b/packages/film/src/index.ts
index cba24996..f83b4a77 100644
--- a/packages/film/src/index.ts
+++ b/packages/film/src/index.ts
@@ -1,4 +1,5 @@
export { createFilmAssemblyFragment } from "./fragment.js";
+export { filmComponent } from "./component.js";
export {
filmManifest,
filmManifestDigest,
diff --git a/packages/local/README.md b/packages/local/README.md
index c65fd1fc..85c12848 100644
--- a/packages/local/README.md
+++ b/packages/local/README.md
@@ -46,6 +46,9 @@ When several instances provide the same role, `runtimeSelection` must name the e
The same mechanism covers Postgres, S3, keychains and replacement Schedulers; none requires a
change to `@svml/local`. Advanced hosts may still call `createLocalRuntime` with raw ports.
+`LocalRuntime.readArtifact(digest)` is the generic egress seam used by CLI `--out`. It exposes
+content-addressed bytes, not filesystem layout, so filesystem and S3 stores remain interchangeable.
+
`LocalBuildRequest.attachments` is the explicit ingress from a trusted Host into the selected
ArtifactStore. Each attachment carries claimed `BlobRef` metadata plus bytes; Local Runtime copies
the bytes, stores them content-addressably and requires the Store's returned digest, size and media
diff --git a/packages/local/src/runtime.ts b/packages/local/src/runtime.ts
index 8d8ba53a..54ecaafb 100644
--- a/packages/local/src/runtime.ts
+++ b/packages/local/src/runtime.ts
@@ -222,6 +222,9 @@ export async function createLocalRuntime(
const [result] = await scheduler.run([{ id: build, state: snapshot.state }]);
return result;
},
+ async readArtifact(digest) {
+ return await options.artifactStore.get(digest);
+ },
close() {},
};
}
@@ -302,6 +305,7 @@ export async function createProjectLocalRuntime(
buildMany: runtime.buildMany,
status: runtime.status,
cancel: runtime.cancel,
+ readArtifact: runtime.readArtifact,
close() {
return services.close();
},
diff --git a/packages/local/src/types.ts b/packages/local/src/types.ts
index 0abbb4ca..a5535430 100644
--- a/packages/local/src/types.ts
+++ b/packages/local/src/types.ts
@@ -2,6 +2,7 @@ import type { Awaitable, ComponentPackage } from "@svml/component-kit";
import type { ArtifactAttachment } from "@svml/host";
import type { EndpointPackage } from "@svml/endpoint-kit";
import type { BuildState } from "@svml/protocol";
+import type { Digest } from "@svml/protocol";
import type {
ArtifactStore,
BuildSchedulerFactory,
@@ -103,5 +104,6 @@ export type LocalRuntime = {
buildMany(requests: readonly LocalBuildRequest[]): Promise<readonly ScheduledBuildResult[]>;
status(build: string): Promise<LocalRuntimeStatus>;
cancel(build: string): Promise<ScheduledBuildResult | undefined>;
+ readArtifact(digest: Digest): Promise<Uint8Array | undefined>;
close(): Awaitable<void>;
};
diff --git a/packages/prelude-video/package.json b/packages/prelude-video/package.json
index a706fdff..7bbcc007 100644
--- a/packages/prelude-video/package.json
+++ b/packages/prelude-video/package.json
@@ -10,7 +10,9 @@
"activation": "./src/index.ts"
},
"dependencies": {
+ "@svml/broll": "workspace:*",
"@svml/caption": "workspace:*",
+ "@svml/caption-gemini": "workspace:*",
"@svml/contracts": "workspace:*",
"@svml/generation": "workspace:*",
"@svml/film": "workspace:*",
@@ -21,8 +23,11 @@
"@svml/script": "workspace:*",
"@svml/seedance": "workspace:*",
"@svml/speech-align": "workspace:*",
+ "@svml/speech-program": "workspace:*",
"@svml/speech-take": "workspace:*",
- "@svml/svs": "workspace:*"
+ "@svml/svs": "workspace:*",
+ "@svml/text-track": "workspace:*",
+ "@svml/whisperx": "workspace:*"
},
"devDependencies": {
"@svml/package-loader-node": "workspace:*"
diff --git a/packages/prelude-video/src/index.ts b/packages/prelude-video/src/index.ts
index e4b39952..fee77157 100644
--- a/packages/prelude-video/src/index.ts
+++ b/packages/prelude-video/src/index.ts
@@ -6,12 +6,34 @@ import {
generationComponent,
generationManifest,
} from "@svml/generation";
+import {
+ brollComponent,
+ brollManifest,
+ brollModuleRef,
+ brollSurfaceImplementationDigest,
+ decodeBrollTrackSurface,
+} from "@svml/broll";
import {
captionComponent,
captionManifest,
+ captionModuleRef,
+ captionProgramSurfaceImplementationDigest,
+ captionSurfaceImplementationDigest,
+ captionStyleSurfaceImplementationDigest,
+ decodeCaptionProgramSurface,
+ decodeCaptionStyleSurface,
+ decodeCaptionTrackSurface,
} from "@svml/caption";
+import {
+ captionGeminiComponent,
+ captionGeminiImplementationDigests,
+ captionGeminiManifest,
+ captionGeminiModuleRef,
+ decodeCaptionGeminiPlannerSurface,
+} from "@svml/caption-gemini";
import {
decodeFilmSurface,
+ filmComponent,
filmManifest,
filmModuleRef,
filmSurfaceImplementationDigest,
@@ -60,7 +82,28 @@ import {
speechTakeComponent,
speechTakeManifest,
} from "@svml/speech-take";
+import {
+ decodeSpeechSpineSurface,
+ speechProgramComponent,
+ speechProgramManifest,
+ speechProgramModuleRef,
+ speechSpineSurfaceImplementationDigest,
+} from "@svml/speech-program";
import { svsFrontend, svsManifest } from "@svml/svs";
+import {
+ decodeTextTrackSurface,
+ textTrackComponent,
+ textTrackManifest,
+ textTrackModuleRef,
+ textTrackSurfaceImplementationDigest,
+} from "@svml/text-track";
+import {
+ decodeWhisperXAlignmentSurface,
+ whisperXComponent,
+ whisperXImplementationDigests,
+ whisperXManifest,
+ whisperXModuleRef,
+} from "@svml/whisperx";
export const svmlPackage: NodePackageActivation = {
format: "svml.node-package@1",
@@ -78,9 +121,17 @@ export const svmlPackage: NodePackageActivation = {
{ manifest: svsManifest },
{ manifest: hyperframesManifest },
{ manifest: mediaPipelineManifest },
- { manifest: captionManifest },
+ { manifest: captionManifest, specifiers: ["@svml/caption", "@svml/caption@1"] },
+ {
+ manifest: captionGeminiManifest,
+ specifiers: ["@svml/caption-gemini", "@svml/caption-gemini@1"],
+ },
{ manifest: speechAlignManifest },
{ manifest: speechTakeManifest },
+ { manifest: speechProgramManifest, specifiers: ["@svml/speech", "@svml/speech@1"] },
+ { manifest: whisperXManifest, specifiers: ["@svml/whisperx", "@svml/whisperx@1"] },
+ { manifest: textTrackManifest, specifiers: ["@svml/text-track", "@svml/text-track@1"] },
+ { manifest: brollManifest, specifiers: ["@svml/broll", "@svml/broll@1"] },
{ manifest: filmManifest, specifiers: ["@svml/film", "@svml/film@1"] },
{
manifest: hyperframesRenderManifest,
@@ -93,9 +144,15 @@ export const svmlPackage: NodePackageActivation = {
generationComponent,
seedanceComponent,
captionComponent,
+ captionGeminiComponent,
mediaPipelineComponent,
speechAlignComponent,
speechTakeComponent,
+ speechProgramComponent,
+ whisperXComponent,
+ textTrackComponent,
+ brollComponent,
+ filmComponent,
hyperframesComponent,
hyperframesRenderComponent,
],
@@ -135,6 +192,62 @@ export const svmlPackage: NodePackageActivation = {
implementationDigest: scriptSurfaceImplementationDigest,
handler: decodeScriptSurface,
},
+ {
+ module: captionModuleRef,
+ surface: "style",
+ mode: "structured",
+ implementationDigest: captionStyleSurfaceImplementationDigest,
+ handler: decodeCaptionStyleSurface,
+ },
+ {
+ module: captionModuleRef,
+ surface: "program",
+ mode: "structured",
+ implementationDigest: captionProgramSurfaceImplementationDigest,
+ handler: decodeCaptionProgramSurface,
+ },
+ {
+ module: captionModuleRef,
+ surface: "track",
+ mode: "structured",
+ implementationDigest: captionSurfaceImplementationDigest,
+ handler: decodeCaptionTrackSurface,
+ },
+ {
+ module: captionGeminiModuleRef,
+ surface: "planner",
+ mode: "structured",
+ implementationDigest: captionGeminiImplementationDigests.plannerSurface,
+ handler: decodeCaptionGeminiPlannerSurface,
+ },
+ {
+ module: speechProgramModuleRef,
+ surface: "spine",
+ mode: "structured",
+ implementationDigest: speechSpineSurfaceImplementationDigest,
+ handler: decodeSpeechSpineSurface,
+ },
+ {
+ module: whisperXModuleRef,
+ surface: "alignment",
+ mode: "structured",
+ implementationDigest: whisperXImplementationDigests.surface,
+ handler: decodeWhisperXAlignmentSurface,
+ },
+ {
+ module: brollModuleRef,
+ surface: "track",
+ mode: "structured",
+ implementationDigest: brollSurfaceImplementationDigest,
+ handler: decodeBrollTrackSurface,
+ },
+ {
+ module: textTrackModuleRef,
+ surface: "track",
+ mode: "structured",
+ implementationDigest: textTrackSurfaceImplementationDigest,
+ handler: decodeTextTrackSurface,
+ },
{
module: filmModuleRef,
surface: "film",
diff --git a/packages/prelude-video/test/activation.test.ts b/packages/prelude-video/test/activation.test.ts
index 9878787e..40cefc5a 100644
--- a/packages/prelude-video/test/activation.test.ts
+++ b/packages/prelude-video/test/activation.test.ts
@@ -66,11 +66,11 @@ test("the official prelude activates Caption compute and owner validators withou
assert.equal(components.includes(captionComponent), true);
assert.deepEqual(
captionComponent.producers.map((facet) => facet.producer.name),
- ["temporalize-caption", "render-caption-track"],
+ ["temporalize-caption", "temporalize-caption-plan", "render-caption-program", "render-caption-track"],
);
assert.deepEqual(
captionComponent.validators?.map((facet) => facet.type.name),
- ["TimedCaptionProjection", "CaptionTrackProgram"],
+ ["CaptionStyle", "CaptionProgram", "CaptionPlan", "TimedCaptionProjection", "CaptionTrackProgram"],
);
});
diff --git a/packages/provider-google-vertex/README.md b/packages/provider-google-vertex/README.md
new file mode 100644
index 00000000..66af05a1
--- /dev/null
+++ b/packages/provider-google-vertex/README.md
@@ -0,0 +1,25 @@
+# @svml/provider-google-vertex
+
+Google Vertex transport for the exact `@svml/caption-gemini` capability.
+
+This package does not build prompts, parse SVML, inspect WhisperX, choose a model, or rewrite text.
+It receives the model package's immutable request, calls `GoogleGenAI({ vertexai: true })` with
+structured JSON output, and returns the response only after `@svml/caption-gemini` validates every
+Run, Cue endpoint, atom assignment, field value and per-Cue cardinality.
+
+```ts
+createGoogleVertexCaptionProvider({
+ project: "my-google-cloud-project",
+ location: "global",
+});
+```
+
+The default credential reference is the explicit environment key
+`GOOGLE_APPLICATION_CREDENTIALS_JSON`, whose value is the JSON contents rather than a path. This
+keeps filesystem authority out of the Provider. A deployment may bind another `CredentialStore`
+and pass its `CredentialRef`. Credentials never enter author source, Runtime Closure, BuildState,
+Operation state, Prompt or Receipt metadata.
+
+The old `twinit` implementation used `GOOGLE_APPLICATION_CREDENTIALS` as an ADC file path. A host
+may preserve that deployment mechanism in a file-backed CredentialStore, but the Provider itself
+does not read arbitrary paths or ambient environment variables.
diff --git a/packages/provider-google-vertex/package.json b/packages/provider-google-vertex/package.json
new file mode 100644
index 00000000..6200a24a
--- /dev/null
+++ b/packages/provider-google-vertex/package.json
@@ -0,0 +1,21 @@
+{
+ "name": "@svml/provider-google-vertex",
+ "version": "0.0.0-dev",
+ "private": true,
+ "type": "module",
+ "exports": {
+ ".": "./src/index.ts"
+ },
+ "dependencies": {
+ "@google/genai": "1.52.0",
+ "@svml/caption": "workspace:*",
+ "@svml/caption-gemini": "workspace:*",
+ "@svml/endpoint-kit": "workspace:*",
+ "@svml/protocol": "workspace:*",
+ "@svml/runtime": "workspace:*"
+ },
+ "devDependencies": {
+ "@svml/driver-node": "workspace:*",
+ "@svml/script": "workspace:*"
+ }
+}
diff --git a/packages/provider-google-vertex/src/index.ts b/packages/provider-google-vertex/src/index.ts
new file mode 100644
index 00000000..5916a153
--- /dev/null
+++ b/packages/provider-google-vertex/src/index.ts
@@ -0,0 +1,9 @@
+export {
+ createGoogleVertexCaptionProvider,
+ googleVertexProviderImplementationDigest,
+ googleVertexProviderModuleRef,
+} from "./provider.js";
+export type {
+ CreateGoogleVertexCaptionProviderOptions,
+ GenerateCaptionContent,
+} from "./provider.js";
diff --git a/packages/provider-google-vertex/src/provider.ts b/packages/provider-google-vertex/src/provider.ts
new file mode 100644
index 00000000..10181cb7
--- /dev/null
+++ b/packages/provider-google-vertex/src/provider.ts
@@ -0,0 +1,268 @@
+import { GoogleGenAI, Type } from "@google/genai";
+import type { GenerateContentResponse, GoogleGenAIOptions, Schema } from "@google/genai";
+import { captionTypes } from "@svml/caption";
+import {
+ captionGeminiCapabilities,
+ sealCaptionGeminiPlan,
+ verifyCaptionGeminiRequest,
+} from "@svml/caption-gemini";
+import type { CaptionGeminiRequest } from "@svml/caption-gemini";
+import { defineEndpointPackage } from "@svml/endpoint-kit";
+import type { EndpointFulfillment, EndpointInvocationContext } from "@svml/endpoint-kit";
+import { canonicalize, digestOf, isDigest } from "@svml/protocol";
+import type { CanonicalValue, Digest } from "@svml/protocol";
+import { credentialRef } from "@svml/runtime";
+import type { CredentialRef } from "@svml/runtime";
+
+export const googleVertexProviderModuleRef = {
+ name: "@svml/provider-google-vertex",
+ version: "0.0.0-dev",
+} as const;
+export const googleVertexProviderImplementationDigest = digestOf(
+ "@svml/provider-google-vertex/caption-gemini@2:@google/[email protected]",
+);
+
+type GenerateCaptionContentInput = {
+ readonly model: string;
+ readonly systemInstruction: string;
+ readonly prompt: string;
+ readonly temperature: number;
+ readonly responseSchema: Schema;
+ readonly abortSignal: AbortSignal;
+ readonly credentialsJson: Readonly<Record<string, unknown>>;
+ readonly project: string;
+ readonly location: string;
+};
+
+export type GenerateCaptionContent = (
+ input: GenerateCaptionContentInput,
+) => Promise<{ readonly text: string; readonly totalTokenCount?: number; readonly modelVersion?: string }>;
+
+export type CreateGoogleVertexCaptionProviderOptions = {
+ readonly project: string;
+ readonly location?: string;
+ readonly instance?: string;
+ readonly lane?: string;
+ /** JSON contents, not a filesystem path. A CredentialStore decides where these bytes live. */
+ readonly credentialsJson?: CredentialRef;
+ readonly defaultConcurrency?: number;
+ readonly requestTimeoutMs?: number;
+ readonly maxResponseBytes?: number;
+ /** Test/private transport injection; its identity must be locked into Runtime configuration. */
+ readonly generateContent?: GenerateCaptionContent;
+ readonly generateContentImplementationDigest?: Digest;
+};
+
+const captionResponseSchema: Schema = {
+ type: Type.OBJECT,
+ properties: {
+ runs: {
+ type: Type.ARRAY,
+ items: {
+ type: Type.OBJECT,
+ properties: {
+ run_id: { type: Type.STRING },
+ cues: {
+ type: Type.ARRAY,
+ items: {
+ type: Type.OBJECT,
+ properties: {
+ after_atom_id: { type: Type.STRING },
+ fields: {
+ type: Type.ARRAY,
+ items: {
+ type: Type.OBJECT,
+ properties: {
+ declaration_id: { type: Type.STRING },
+ atom_id: { type: Type.STRING },
+ value: { type: Type.STRING },
+ },
+ required: ["declaration_id", "atom_id", "value"],
+ propertyOrdering: ["declaration_id", "atom_id", "value"],
+ },
+ },
+ },
+ required: ["after_atom_id", "fields"],
+ propertyOrdering: ["after_atom_id", "fields"],
+ },
+ },
+ },
+ required: ["run_id", "cues"],
+ propertyOrdering: ["run_id", "cues"],
+ },
+ },
+ },
+ required: ["runs"],
+ propertyOrdering: ["runs"],
+};
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function positiveInteger(value: number, subject: string): number {
+ assert(Number.isSafeInteger(value) && value > 0, `${subject} must be a positive integer`);
+ return value;
+}
+
+function credentialJson(context: EndpointInvocationContext): Readonly<Record<string, unknown>> {
+ const secret = context.credentials.googleCredentials?.secret;
+ assert(secret !== undefined && secret.length > 0, "Google Vertex credentials JSON is unavailable");
+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(secret);
+ } catch {
+ throw new Error("Google Vertex credentials are not valid JSON");
+ }
+ assert(parsed !== null && typeof parsed === "object" && !Array.isArray(parsed),
+ "Google Vertex credentials JSON must be an object");
+ const value = parsed as Readonly<Record<string, unknown>>;
+ assert(typeof value.type === "string" && value.type.length > 0,
+ "Google Vertex credentials JSON has no credential type");
+ return value;
+}
+
+function requestValue(value: CanonicalValue): CaptionGeminiRequest {
+ verifyCaptionGeminiRequest(value);
+ return value;
+}
+
+function sdkGenerateContent(): GenerateCaptionContent {
+ return async (input) => {
+ const googleAuthOptions = {
+ credentials: input.credentialsJson,
+ scopes: ["https://www.googleapis.com/auth/cloud-platform"],
+ } as NonNullable<GoogleGenAIOptions["googleAuthOptions"]>;
+ const client = new GoogleGenAI({
+ vertexai: true,
+ project: input.project,
+ location: input.location,
+ googleAuthOptions,
+ });
+ const response: GenerateContentResponse = await client.models.generateContent({
+ model: input.model,
+ contents: [{ role: "user", parts: [{ text: input.prompt }] }],
+ config: {
+ systemInstruction: input.systemInstruction,
+ temperature: input.temperature,
+ responseMimeType: "application/json",
+ responseSchema: input.responseSchema,
+ abortSignal: input.abortSignal,
+ },
+ });
+ return {
+ text: response.text ?? "",
+ ...(response.usageMetadata?.totalTokenCount === undefined
+ ? {} : { totalTokenCount: response.usageMetadata.totalTokenCount }),
+ ...(response.modelVersion === undefined ? {} : { modelVersion: response.modelVersion }),
+ };
+ };
+}
+
+async function withTimeout<T>(timeoutMs: number, task: (signal: AbortSignal) => Promise<T>): Promise<T> {
+ const controller = new AbortController();
+ let timer: ReturnType<typeof setTimeout> | undefined;
+ try {
+ return await Promise.race([
+ task(controller.signal),
+ new Promise<never>((_, reject) => {
+ timer = setTimeout(() => {
+ controller.abort(new Error("Google Vertex request timed out"));
+ reject(new Error("Google Vertex request timed out"));
+ }, timeoutMs);
+ }),
+ ]);
+ } finally {
+ if (timer !== undefined) clearTimeout(timer);
+ }
+}
+
+function fulfillment(value: CanonicalValue, metadata: CanonicalValue): EndpointFulfillment {
+ return {
+ value: { kind: "inline", value },
+ conformance: "exact",
+ delivery: "executed",
+ metadata,
+ };
+}
+
+export function createGoogleVertexCaptionProvider(options: CreateGoogleVertexCaptionProviderOptions) {
+ assert(options.project.trim().length > 0, "Google Vertex project is empty");
+ const project = options.project.trim();
+ const location = (options.location ?? "global").trim();
+ assert(location.length > 0, "Google Vertex location is empty");
+ const requestTimeoutMs = positiveInteger(options.requestTimeoutMs ?? 120_000, "requestTimeoutMs");
+ const maxResponseBytes = positiveInteger(options.maxResponseBytes ?? 2_000_000, "maxResponseBytes");
+ if (options.generateContent !== undefined && options.generateContentImplementationDigest === undefined) {
+ throw new Error("custom Google Vertex generateContent requires generateContentImplementationDigest");
+ }
+ if (options.generateContentImplementationDigest !== undefined && !isDigest(options.generateContentImplementationDigest)) {
+ throw new Error("generateContentImplementationDigest is invalid");
+ }
+ const generateContent = options.generateContent ?? sdkGenerateContent();
+ const transportDigest = options.generateContentImplementationDigest
+ ?? digestOf("@svml/provider-google-vertex/@google-genai-1.52.0@1");
+
+ return defineEndpointPackage({
+ module: googleVertexProviderModuleRef,
+ facet: "caption-gemini",
+ instance: options.instance ?? "google-vertex.caption",
+ ...(options.lane === undefined ? {} : { lane: options.lane }),
+ implementation: {
+ locator: "@svml/provider-google-vertex/caption-gemini",
+ digest: googleVertexProviderImplementationDigest,
+ },
+ permissions: ["network:aiplatform.googleapis.com"],
+ configuration: canonicalize({ project, location, requestTimeoutMs, maxResponseBytes, transportDigest }),
+ credentials: {
+ googleCredentials: options.credentialsJson ?? credentialRef("env", "GOOGLE_APPLICATION_CREDENTIALS_JSON"),
+ },
+ defaultConcurrency: options.defaultConcurrency ?? 2,
+ capabilities: [{
+ capability: captionGeminiCapabilities.plan,
+ returns: captionTypes.plan,
+ lifecycle: "immediate",
+ supports: (need) => {
+ try {
+ verifyCaptionGeminiRequest(need.constraints);
+ return true;
+ } catch {
+ return false;
+ }
+ },
+ handler: async (context) => {
+ const request = requestValue(context.need.constraints);
+ const response = await withTimeout(requestTimeoutMs, (abortSignal) => generateContent({
+ model: request.model,
+ systemInstruction: request.systemInstruction,
+ prompt: request.prompt,
+ temperature: request.temperature,
+ responseSchema: captionResponseSchema,
+ abortSignal,
+ credentialsJson: credentialJson(context),
+ project,
+ location,
+ }));
+ const responseBytes = Buffer.byteLength(response.text, "utf8");
+ assert(responseBytes > 0, "Google Vertex returned an empty Caption plan");
+ assert(responseBytes <= maxResponseBytes, "Google Vertex Caption response exceeded its configured limit");
+ let raw: unknown;
+ try {
+ raw = JSON.parse(response.text);
+ } catch {
+ throw new Error("Google Vertex returned invalid Caption JSON");
+ }
+ const plan = sealCaptionGeminiPlan(request, raw);
+ return fulfillment(canonicalize(plan), canonicalize({
+ provider: "google-vertex",
+ project,
+ location,
+ requestedModel: request.model,
+ ...(response.modelVersion === undefined ? {} : { modelVersion: response.modelVersion }),
+ ...(response.totalTokenCount === undefined ? {} : { totalTokenCount: response.totalTokenCount }),
+ responseDigest: digestOf(response.text),
+ }));
+ },
+ }],
+ });
+}
diff --git a/packages/provider-google-vertex/test/provider.test.ts b/packages/provider-google-vertex/test/provider.test.ts
new file mode 100644
index 00000000..a656731e
--- /dev/null
+++ b/packages/provider-google-vertex/test/provider.test.ts
@@ -0,0 +1,131 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import {
+ captionGeminiCapabilities,
+ compileCaptionGeminiRequest,
+ sealCaptionGeminiProgram,
+} from "@svml/caption-gemini";
+import {
+ captionTypes,
+ defaultCaptionTrackProgram,
+ resolveCaptionProgram,
+ sealCaptionStyle,
+} from "@svml/caption";
+import type { CaptionGeminiRequest, RawCaptionGeminiResponse } from "@svml/caption-gemini";
+import { EndpointRegistry, MemoryArtifactStore } from "@svml/driver-node";
+import type { ImmediateEndpointHandler } from "@svml/endpoint-kit";
+import { digestOf } from "@svml/protocol";
+import type { CanonicalValue, Need } from "@svml/protocol";
+import {
+ createGoogleVertexCaptionProvider,
+ googleVertexProviderImplementationDigest,
+} from "@svml/provider-google-vertex";
+import type { GenerateCaptionContent } from "@svml/provider-google-vertex";
+import { parseScript } from "@svml/script";
+
+function request(): CaptionGeminiRequest {
+ const narrative = parseScript("provider.svml", "<line><ALICE>Meaning becomes the source.</line>");
+ const base = defaultCaptionTrackProgram("important");
+ const style = sealCaptionStyle({
+ contract: "svml.caption-style@1",
+ id: "important",
+ planning: {
+ cueInstruction: "Prefer one short complete semantic phrase.",
+ fields: [{
+ id: "important",
+ value: { kind: "enum", values: ["important"] },
+ instruction: "Select at most two important words.",
+ minimumPerCue: 0,
+ maximumPerCue: 2,
+ }],
+ },
+ presentation: { mode: "whole", stackingOrder: 100, style: base.style },
+ });
+ const captionProgram = resolveCaptionProgram(narrative, "captions", style, []);
+ return compileCaptionGeminiRequest(narrative, captionProgram, sealCaptionGeminiProgram({
+ contract: "svml.caption-gemini-program@1",
+ model: "gemini-2.5-flash",
+ }));
+}
+
+function response(requestValue: CaptionGeminiRequest): RawCaptionGeminiResponse {
+ return { runs: requestValue.runs.map((run) => ({
+ run_id: run.id,
+ cues: [{
+ after_atom_id: run.atomIds.at(-1)!,
+ fields: [{ declaration_id: "important", atom_id: run.atomIds[0]!, value: "important" }],
+ }],
+ })) };
+}
+
+function need(requestValue: CaptionGeminiRequest): Need {
+ const constraints = requestValue as unknown as CanonicalValue;
+ return {
+ id: "need:caption-gemini",
+ capability: captionGeminiCapabilities.plan,
+ returns: captionTypes.plan,
+ constraints,
+ requestedBy: "derivation:caption-gemini",
+ result: "record:caption-gemini",
+ accepts: "exact",
+ conformanceFloor: "exact",
+ requestDigest: digestOf({ capability: captionGeminiCapabilities.plan, constraints }),
+ };
+}
+
+test("Vertex transports the exact model request while the model package validates the result", async () => {
+ const requestValue = request();
+ let captured: Parameters<GenerateCaptionContent>[0] | undefined;
+ const provider = createGoogleVertexCaptionProvider({
+ project: "svml-test-project",
+ location: "global",
+ generateContentImplementationDigest: digestOf("caption-gemini:test-transport"),
+ generateContent: async (input) => {
+ captured = input;
+ return {
+ text: JSON.stringify(response(requestValue)),
+ totalTokenCount: 123,
+ modelVersion: "gemini-2.5-flash-001",
+ };
+ },
+ });
+ const registry = new EndpointRegistry();
+ await provider.install(registry);
+ const requestNeed = need(requestValue);
+ const resolution = registry.resolve(requestNeed);
+ assert.equal(resolution.status, "resolved");
+ assert.equal(resolution.registration.kind, "immediate");
+ const handler: ImmediateEndpointHandler = resolution.registration.handler;
+ const result = await handler({
+ command: { kind: "fulfill-need", id: "command:caption-gemini", need: requestNeed },
+ need: requestNeed,
+ artifacts: new MemoryArtifactStore(),
+ credentials: { googleCredentials: { secret: JSON.stringify({ type: "service_account" }) } },
+ });
+ assert.equal(captured?.model, "gemini-2.5-flash");
+ assert.equal(captured?.prompt, requestValue.prompt);
+ assert.equal(captured?.systemInstruction, requestValue.systemInstruction);
+ assert.equal(captured?.temperature, 0.2);
+ assert.equal(result.value.kind, "inline");
+ const plan = result.value.kind === "inline" ? result.value.value as Record<string, unknown> : {};
+ assert.equal(plan.contract, "svml.caption-plan@1");
+ assert.equal(plan.planningRequestDigest, requestValue.requestDigest);
+ assert.equal((result.metadata as Record<string, unknown>).provider, "google-vertex");
+});
+
+test("Vertex configuration exposes credential/queue policy without changing the model request", () => {
+ const provider = createGoogleVertexCaptionProvider({ project: "svml-test-project", defaultConcurrency: 3 });
+ assert.equal(provider.name, "google-vertex.caption");
+ const facet = provider.manifest.facets[0];
+ assert.equal(facet?.role, "capability-endpoint");
+ assert(facet?.role === "capability-endpoint");
+ assert.equal(facet.defaultConcurrency, 3);
+ assert.deepEqual(facet.credentialSlots, ["googleCredentials"]);
+ assert.deepEqual(provider.bindings, [{
+ capability: captionGeminiCapabilities.plan,
+ returns: captionTypes.plan,
+ endpoint: "google-vertex.caption",
+ }]);
+ assert.equal(facet.implementation.digest, googleVertexProviderImplementationDigest);
+});
diff --git a/packages/script/README.md b/packages/script/README.md
index cfada96a..05603d70 100644
--- a/packages/script/README.md
+++ b/packages/script/README.md
@@ -26,12 +26,25 @@ parser is inside a Segment, a valid bare tag such as `<ALICE>` is a Role Cue. Th
not indentation: the compact spelling
`<opening><ALICE>我先说。<BOB>我回答。</opening>` has the same semantic value.
+A Role Cue is optional. Text before the first Role Cue is a roleless Turn, and Role state is reset
+when every Segment closes; a Role can never leak into the following Segment.
+
The package exports its Manifest, `parseScript`, semantic/source-map projection helpers, a
semantic-preserving formatter and the raw `decodeScriptSurface` handler. Source ranges and parser
state remain private to Script; its authored Narrative Record uses the Frontend-neutral type from
`@svml/contracts`, so third-party author surfaces can feed the same WhisperX, locator and caption
components without importing Script internals.
-Every named Segment is additionally exported as `script.segment.<id>` with the shared
-`NarrativeExcerpt` type. Seedance and future speech packages consume that narrow value rather than
-Script's parser AST; another authoring package may produce the same contract.
+The Surface exports one full Narrative plus narrow, immutable views:
+
+- `script.segment.<id>` is a `NarrativeExcerpt` used to associate a generated Take with one Segment;
+- `script.segment.<id>.dialogue` is display-independent dialogue, including optional Role cues and
+ right-side Dual Text pronunciation, for a speech-video model;
+- `script.segment.<id>.speech` is pronunciation-only text for duration estimation;
+- `script.caption` is the whole left-side `CaptionProjection` and is not split by Segment for the
+ Caption planner;
+- `script.selection.<id>` is a reusable explicit Selection.
+
+Seedance consumes the dialogue excerpt, Speech Spine consumes the Segment excerpt, and Caption
+consumes the whole Narrative/Caption projection. None imports Script's parser AST. Another authoring
+package may produce the same contracts.
diff --git a/packages/script/src/index.ts b/packages/script/src/index.ts
index 1cd92aba..47f45e3b 100644
--- a/packages/script/src/index.ts
+++ b/packages/script/src/index.ts
@@ -1,15 +1,23 @@
export { ScriptSyntaxError } from "./error.js";
export { formatScript } from "./format.js";
export {
+ captionProjectionType,
+ narrativeDialogueExcerptType,
narrativeExcerptType,
+ narrativeSelectionType,
narrativeSchema,
+ narrativeSpeechExcerptType,
narrativeType,
scriptManifest,
scriptModuleRef,
scriptSurfaceImplementationDigest,
} from "./manifest.js";
export {
+ captionProjectionValue,
+ narrativeDialogueExcerptValue,
narrativeSegmentExcerptValue,
+ narrativeSelectionValue,
+ narrativeSpeechExcerptValue,
narrativeSourceMap,
narrativeValue,
serializeCaption,
diff --git a/packages/script/src/manifest.ts b/packages/script/src/manifest.ts
index f61e4d26..ee4d8267 100644
--- a/packages/script/src/manifest.ts
+++ b/packages/script/src/manifest.ts
@@ -9,9 +9,13 @@ import type { ModuleManifest, TypeRef } from "@svml/protocol";
export const scriptModuleRef = { name: "@svml/script", version: "0.0.0-dev" } as const;
export const narrativeType: TypeRef = contractTypes.narrative;
export const narrativeExcerptType: TypeRef = contractTypes.narrativeExcerpt;
+export const narrativeDialogueExcerptType: TypeRef = contractTypes.narrativeDialogueExcerpt;
+export const narrativeSpeechExcerptType: TypeRef = contractTypes.narrativeSpeechExcerpt;
+export const narrativeSelectionType: TypeRef = contractTypes.narrativeSelection;
+export const captionProjectionType: TypeRef = contractTypes.captionProjection;
export { narrativeSchema };
-export const scriptSurfaceImplementationDigest = digestOf("@svml/script/surface@1");
+export const scriptSurfaceImplementationDigest = digestOf("@svml/script/surface@2");
export const scriptManifest: ModuleManifest = {
format: "svml.module@0",
@@ -25,7 +29,14 @@ export const scriptManifest: ModuleManifest = {
name: "script",
tag: "script",
mode: "raw",
- outputs: [narrativeType, narrativeExcerptType],
+ outputs: [
+ narrativeType,
+ narrativeExcerptType,
+ narrativeDialogueExcerptType,
+ narrativeSpeechExcerptType,
+ narrativeSelectionType,
+ captionProjectionType,
+ ],
implementation: {
kind: "trusted-frontend-surface",
locator: "@svml/script/surface",
diff --git a/packages/script/src/narrative.ts b/packages/script/src/narrative.ts
index a7d42635..8871e201 100644
--- a/packages/script/src/narrative.ts
+++ b/packages/script/src/narrative.ts
@@ -42,6 +42,29 @@ function segmentSerializations(segment: ParsedNarrative["segments"][number]): {
return { dialogue: dialogue.join("\n"), speech: joinProjection(speechParts) };
}
+function excerptContent(
+ segment: ParsedNarrative["segments"][number],
+ projection: "dialogue" | "speech",
+): {
+ readonly kind: "segment";
+ readonly id: string;
+ readonly tokenStart: number;
+ readonly tokenEndExclusive: number;
+ readonly dialogue?: string;
+ readonly speech?: string;
+} {
+ const serializations = segmentSerializations(segment);
+ return {
+ kind: "segment",
+ id: segment.id,
+ tokenStart: segment.tokenStart,
+ tokenEndExclusive: segment.tokenEndExclusive,
+ ...(projection === "dialogue"
+ ? { dialogue: serializations.dialogue }
+ : { speech: serializations.speech }),
+ };
+}
+
export function narrativeSegmentExcerptValue(
parsed: ParsedNarrative,
segment: ParsedNarrative["segments"][number],
@@ -57,6 +80,48 @@ export function narrativeSegmentExcerptValue(
return canonicalize({ ...content, excerptDigest: digestOf(canonicalize(content)) });
}
+export function narrativeDialogueExcerptValue(
+ segment: ParsedNarrative["segments"][number],
+): CanonicalValue {
+ const content = {
+ contract: "svml.narrative-dialogue-excerpt@1",
+ ...excerptContent(segment, "dialogue"),
+ } as const;
+ return canonicalize({ ...content, excerptDigest: digestOf(canonicalize(content)) });
+}
+
+export function narrativeSpeechExcerptValue(
+ segment: ParsedNarrative["segments"][number],
+): CanonicalValue {
+ const content = {
+ contract: "svml.narrative-speech-excerpt@1",
+ ...excerptContent(segment, "speech"),
+ } as const;
+ return canonicalize({ ...content, excerptDigest: digestOf(canonicalize(content)) });
+}
+
+export function captionProjectionValue(parsed: ParsedNarrative): CanonicalValue {
+ const content = {
+ contract: parsed.captionProjection.contract,
+ text: parsed.captionProjection.text,
+ regions: parsed.captionProjection.regions.map(({ range: _range, ...region }) => region),
+ } as const;
+ return canonicalize({ ...content, projectionDigest: digestOf(canonicalize(content)) });
+}
+
+export function narrativeSelectionValue(selection: ParsedNarrative["selections"][number]): CanonicalValue {
+ const content = {
+ contract: "svml.narrative-selection@1",
+ id: selection.id,
+ occurrences: selection.occurrences.map((occurrence) => ({
+ occurrence: occurrence.occurrence,
+ open: { affinity: occurrence.open.affinity, boundary: semanticBoundary(occurrence.open.boundary) },
+ close: { affinity: occurrence.close.affinity, boundary: semanticBoundary(occurrence.close.boundary) },
+ })),
+ } as const;
+ return canonicalize({ ...content, selectionDigest: digestOf(canonicalize(content)) });
+}
+
function semanticBoundary(boundary: {
readonly tokenIndex: number;
readonly structuralPosition: number;
diff --git a/packages/script/src/surface.ts b/packages/script/src/surface.ts
index 8d5d56b6..36dc1ad3 100644
--- a/packages/script/src/surface.ts
+++ b/packages/script/src/surface.ts
@@ -2,11 +2,22 @@ import { canonicalize } from "@svml/core";
import { ScriptSyntaxError } from "./error.js";
import {
+ captionProjectionValue,
+ narrativeDialogueExcerptValue,
narrativeSegmentExcerptValue,
+ narrativeSelectionValue,
+ narrativeSpeechExcerptValue,
narrativeSourceMap,
narrativeValue,
} from "./narrative.js";
-import { narrativeExcerptType, narrativeType } from "./manifest.js";
+import {
+ captionProjectionType,
+ narrativeDialogueExcerptType,
+ narrativeExcerptType,
+ narrativeSelectionType,
+ narrativeSpeechExcerptType,
+ narrativeType,
+} from "./manifest.js";
import { parseScript } from "./parser.js";
import type { ScriptSurfaceInput, ScriptSurfaceOutput } from "./types.js";
@@ -79,6 +90,35 @@ export function decodeScriptSurface(input: ScriptSurfaceInput): ScriptSurfaceOut
value: { kind: "inline" as const, value: narrativeSegmentExcerptValue(parsed, segment) },
range: segment.range,
})),
+ ...parsed.segments.flatMap((segment) => [
+ {
+ id: `${rawId}.segment.${segment.id}.dialogue`,
+ type: narrativeDialogueExcerptType,
+ value: { kind: "inline" as const, value: narrativeDialogueExcerptValue(segment) },
+ range: segment.range,
+ },
+ {
+ id: `${rawId}.segment.${segment.id}.speech`,
+ type: narrativeSpeechExcerptType,
+ value: { kind: "inline" as const, value: narrativeSpeechExcerptValue(segment) },
+ range: segment.range,
+ },
+ ]),
+ {
+ id: `${rawId}.caption`,
+ type: captionProjectionType,
+ value: { kind: "inline" as const, value: captionProjectionValue(parsed) },
+ range: { start: input.openingStart, end: close.end },
+ },
+ ...parsed.selections.map((selection) => ({
+ id: `${rawId}.selection.${selection.id}`,
+ type: narrativeSelectionType,
+ value: { kind: "inline" as const, value: narrativeSelectionValue(selection) },
+ range: {
+ start: selection.occurrences[0]!.open.range.start,
+ end: selection.occurrences.at(-1)!.close.range.end,
+ },
+ })),
],
components: [],
fragments: [],
diff --git a/packages/script/test/script.test.ts b/packages/script/test/script.test.ts
index ad6ad82b..ca48ad54 100644
--- a/packages/script/test/script.test.ts
+++ b/packages/script/test/script.test.ts
@@ -36,6 +36,18 @@ test("named blocks are Segments and Role Cues do not depend on line breaks", ()
assert.equal(serializeDialogue(compact), "ALICE: Hello there.\nBOB: Good morning.");
});
+test("Role is optional per Turn and never leaks across Segment boundaries", () => {
+ const parsed = parseScript(
+ "optional-role.svml",
+ "<intro>Roleless narration.</intro><answer><ALICE>Named reply.</answer><close>Roleless close.</close>",
+ );
+ assert.deepEqual(parsed.turns.map((turn) => turn.role), [undefined, "ALICE", undefined]);
+ assert.equal(
+ serializeDialogue(parsed),
+ "Roleless narration.\nALICE: Named reply.\nRoleless close.",
+ );
+});
+
test("Script produces exactly 2M + 2N independent semantic anchors", () => {
const parsed = parseScript("anchors.svml", "<one>One two.</one><silence/><two>Three.</two>");
assert.equal(
diff --git a/packages/seedance/README.md b/packages/seedance/README.md
index b0e658b3..ff9bce3a 100644
--- a/packages/seedance/README.md
+++ b/packages/seedance/README.md
@@ -5,7 +5,7 @@ and `Speech` Surfaces; it does not contain KIE credentials, HTTP code, queues or
`Video` and `Speech` preserve the remote result as one atomic `GeneratedVideoSet`, then expose its
first ordered member through the shared deterministic primary-video projection. `Speech` consumes a
-shared `NarrativeExcerpt`, so a third-party narrative Surface can feed it without importing the
+shared `NarrativeDialogueExcerpt`, so a third-party narrative Surface can feed it without importing the
official Script parser.
Duration is explicit in this low-level model Surface. A later Speech scheduling package may wrap it
diff --git a/packages/seedance/src/index.ts b/packages/seedance/src/index.ts
index 99f6d4f6..4f021250 100644
--- a/packages/seedance/src/index.ts
+++ b/packages/seedance/src/index.ts
@@ -56,7 +56,7 @@ export const seedanceTypes = {
export const seedanceSurfaceImplementationDigests = {
prompt: digestOf("@svml/seedance/prompt-surface@1"),
- speech: digestOf("@svml/seedance/speech-surface@1"),
+ speech: digestOf("@svml/seedance/speech-surface@2"),
video: digestOf("@svml/seedance/video-surface@1"),
} as const;
diff --git a/packages/seedance/src/surface.ts b/packages/seedance/src/surface.ts
index 00122e56..5807d3cf 100644
--- a/packages/seedance/src/surface.ts
+++ b/packages/seedance/src/surface.ts
@@ -211,26 +211,25 @@ function referencePrompt(values: readonly ReferenceInput[]): string {
return roles.length === 0 ? "" : `\n\nReference roles:\n${roles.join("\n")}`;
}
-function excerpt(reference: SurfaceResolvedReference, subject: string): {
- readonly serializations: { readonly dialogue: string; readonly speech: string };
+function dialogueExcerpt(reference: SurfaceResolvedReference, subject: string): {
+ readonly dialogue: string;
} {
- if (!sameType(reference.type, contractTypes.narrativeExcerpt)) {
- throw new Error(`${subject} must reference a NarrativeExcerpt such as script.segment.opening`);
+ if (!sameType(reference.type, contractTypes.narrativeDialogueExcerpt)) {
+ throw new Error(`${subject} must reference a NarrativeDialogueExcerpt such as script.segment.opening.dialogue`);
}
const value = inline(reference, subject) as unknown as {
readonly contract: string;
readonly excerptDigest: string;
- readonly serializations?: { readonly dialogue?: string; readonly speech?: string };
+ readonly dialogue?: string;
readonly [key: string]: CanonicalValue | undefined;
};
- if (value.contract !== "svml.narrative-excerpt@1" || value.serializations === undefined
- || typeof value.serializations.dialogue !== "string" || typeof value.serializations.speech !== "string") {
- throw new Error(`${subject} NarrativeExcerpt is invalid`);
+ if (value.contract !== "svml.narrative-dialogue-excerpt@1" || typeof value.dialogue !== "string") {
+ throw new Error(`${subject} NarrativeDialogueExcerpt is invalid`);
}
const { excerptDigest: _digest, ...content } = value;
- if (value.excerptDigest !== digestOf(content)) throw new Error(`${subject} NarrativeExcerpt digest differs`);
- if (value.serializations.speech.trim().length === 0) throw new Error(`${subject} contains no spoken text`);
- return value as { readonly serializations: { readonly dialogue: string; readonly speech: string } };
+ if (value.excerptDigest !== digestOf(content)) throw new Error(`${subject} NarrativeDialogueExcerpt digest differs`);
+ if (value.dialogue.trim().length === 0) throw new Error(`${subject} contains no spoken text`);
+ return value as { readonly dialogue: string };
}
function generationOutput(
@@ -297,17 +296,17 @@ export const decodeSeedanceVideoSurface: StructuredSurfaceHandler = ({ element,
};
export const decodeSeedanceSpeechSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
- attributes(element, ["id", "model", "script", "prompt", "duration"], [
+ attributes(element, ["id", "model", "dialogue", "prompt", "duration"], [
"resolution", "aspect-ratio", "web-search",
]);
const selected = modelSelection(element);
const declaredPrompt = prompt(resolved(element, "prompt", resolveReference), `${element.name}.prompt`);
- const spoken = excerpt(resolved(element, "script", resolveReference), `${element.name}.script`);
+ const spoken = dialogueExcerpt(resolved(element, "dialogue", resolveReference), `${element.name}.dialogue`);
const refs = references(element, resolveReference);
const request = sealSeedanceRequest({
contract: "svml.seedance-request@1",
model: selected.model as SeedanceModel,
- prompt: `${declaredPrompt.text}${referencePrompt(refs)}\n\nSpoken dialogue — say exactly:\n${spoken.serializations.dialogue}`,
+ prompt: `${declaredPrompt.text}${referencePrompt(refs)}\n\nSpoken dialogue — say exactly:\n${spoken.dialogue}`,
mode: refs.length === 0
? { kind: "text" }
: { kind: "reference", items: refs.map(({ role: _role, ...item }) => item) },
diff --git a/packages/speech-program/README.md b/packages/speech-program/README.md
index c8a802d7..bbfb696a 100644
--- a/packages/speech-program/README.md
+++ b/packages/speech-program/README.md
@@ -10,5 +10,9 @@ auditable reusable graph contributions:
- WhisperX evidence plus the provider-neutral speech SemanticMap;
- timed caption projection.
-The author frontend may expose these Fragments through concise surface syntax.
+The package also owns `<speech:Spine>`. Each `<speech:Take>` explicitly connects a generated Blob
+and its `NarrativeExcerpt`; the Surface expands per-Take media normalization, an immutable append
+fold, one timeline-audio render and ordinary SpeechBasis projections. Whitespace and child layout
+do not create ports, and Core receives no variadic node or Speech special case.
+
The Elaborator expands them hygienically before SVML Core sees the graph.
diff --git a/packages/speech-program/package.json b/packages/speech-program/package.json
index 81586c59..78ec82be 100644
--- a/packages/speech-program/package.json
+++ b/packages/speech-program/package.json
@@ -8,18 +8,19 @@
},
"dependencies": {
"@svml/caption": "workspace:*",
+ "@svml/component-kit": "workspace:*",
"@svml/contracts": "workspace:*",
"@svml/elaborator": "workspace:*",
"@svml/media-pipeline": "workspace:*",
+ "@svml/protocol": "workspace:*",
"@svml/speech-align": "workspace:*",
"@svml/speech-take": "workspace:*",
+ "@svml/text": "workspace:*",
"@svml/whisperx": "workspace:*"
},
"devDependencies": {
- "@svml/component-kit": "workspace:*",
"@svml/core": "workspace:*",
"@svml/driver-node": "workspace:*",
- "@svml/protocol": "workspace:*",
"@svml/realization": "workspace:*",
"@svml/script": "workspace:*",
"@svml/validation": "workspace:*"
diff --git a/packages/speech-program/src/component.ts b/packages/speech-program/src/component.ts
new file mode 100644
index 00000000..5ca3dbef
--- /dev/null
+++ b/packages/speech-program/src/component.ts
@@ -0,0 +1,71 @@
+import type { ComponentPackage } from "@svml/component-kit";
+import type {
+ NarrativeExcerpt,
+ SynchronizedMedia,
+ TimelineAudio,
+} from "@svml/contracts";
+import type { StoredValue } from "@svml/protocol";
+import { canonicalize } from "@svml/protocol";
+
+import { speechProgramProducers } from "./manifest.js";
+import {
+ appendSpeechSpineTake,
+ appendSpeechSpineTakeImplementationDigest,
+ assembleSpeechBasis,
+ assembleSpeechBasisImplementationDigest,
+ compileSpeechSpineAudio,
+ compileSpeechSpineAudioImplementationDigest,
+ createSpeechSpineSet,
+ createSpeechSpineSetImplementationDigest,
+} from "./program.js";
+import type { SpeechSpineProgram, SpeechSpineSet } from "./types.js";
+
+function inline<T>(value: StoredValue | undefined, subject: string): T {
+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);
+ return value.value as unknown as T;
+}
+
+export const speechProgramComponent = {
+ name: "@svml/speech-program",
+ producers: [
+ {
+ producer: speechProgramProducers.createSet,
+ implementationDigest: createSpeechSpineSetImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { set: { kind: "inline", value: canonicalize(createSpeechSpineSet(inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"))) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: speechProgramProducers.appendTake,
+ implementationDigest: appendSpeechSpineTakeImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineTake(
+ inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),
+ inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),
+ inline<NarrativeExcerpt>(inputs.segment?.value, "NarrativeExcerpt"),
+ )) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: speechProgramProducers.compileAudio,
+ implementationDigest: compileSpeechSpineAudioImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { plan: { kind: "inline", value: canonicalize(compileSpeechSpineAudio(inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"))) } },
+ needs: {},
+ }),
+ },
+ {
+ producer: speechProgramProducers.assembleBasis,
+ implementationDigest: assembleSpeechBasisImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { basis: { kind: "inline", value: canonicalize(assembleSpeechBasis(
+ inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),
+ inline<TimelineAudio>(inputs.audio?.value, "TimelineAudio"),
+ )) } },
+ needs: {},
+ }),
+ },
+ ],
+} satisfies ComponentPackage;
diff --git a/packages/speech-program/src/fragment.ts b/packages/speech-program/src/fragment.ts
new file mode 100644
index 00000000..ab212c87
--- /dev/null
+++ b/packages/speech-program/src/fragment.ts
@@ -0,0 +1,115 @@
+import { contractTypes } from "@svml/contracts";
+import { sealGraphFragment } from "@svml/elaborator";
+import type { FragmentOperation } from "@svml/elaborator";
+import { mediaPipelineProducers } from "@svml/media-pipeline";
+import { speechTakeProducers } from "@svml/speech-take";
+
+import { speechProgramProducers, speechProgramTypes } from "./manifest.js";
+import type { SpeechSpineFragmentOptions } from "./types.js";
+
+const input = (name: string) => ({ kind: "fragment-input" as const, name });
+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
+
+export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {
+ if (options.takes.length === 0) throw new Error("Speech Spine requires at least one Take");
+ const names = new Set(["program"]);
+ for (const take of options.takes) {
+ if (!take.mediaName || !take.segmentName || names.has(take.mediaName) || names.has(take.segmentName)) {
+ throw new Error("Speech Spine Fragment input names are empty or duplicated");
+ }
+ names.add(take.mediaName);
+ names.add(take.segmentName);
+ }
+ const operations: FragmentOperation[] = [{
+ id: "spine:set:empty",
+ producer: speechProgramProducers.createSet,
+ inputs: { program: input("program") },
+ result: { kind: "output", name: "set" },
+ }];
+ let current = "spine:set:empty";
+ options.takes.forEach((take, index) => {
+ const id = `spine:set:append:${String(index + 1).padStart(4, "0")}`;
+ operations.push({
+ id,
+ producer: speechProgramProducers.appendTake,
+ inputs: { set: operation(current), media: input(take.mediaName), segment: input(take.segmentName) },
+ result: { kind: "output", name: "set" },
+ });
+ current = id;
+ });
+ operations.push(
+ {
+ id: "spine:audio:plan",
+ producer: speechProgramProducers.compileAudio,
+ inputs: { set: operation(current) },
+ result: { kind: "output", name: "plan" },
+ },
+ {
+ id: "spine:audio:render",
+ producer: mediaPipelineProducers.renderAudio,
+ inputs: { plan: operation("spine:audio:plan") },
+ result: { kind: "need", name: "audio", accepts: "exact" },
+ },
+ {
+ id: "spine:basis",
+ producer: speechProgramProducers.assembleBasis,
+ inputs: { set: operation(current), audio: operation("spine:audio:render") },
+ result: { kind: "output", name: "basis" },
+ },
+ {
+ id: "spine:space",
+ producer: speechTakeProducers.projectProgramSpace,
+ inputs: { basis: operation("spine:basis") },
+ result: { kind: "output", name: "programSpace" },
+ },
+ {
+ id: "spine:audio-basis",
+ producer: speechTakeProducers.projectAudio,
+ inputs: { basis: operation("spine:basis") },
+ result: { kind: "output", name: "audio" },
+ },
+ {
+ id: "spine:visual-track",
+ producer: speechTakeProducers.projectVisual,
+ inputs: { basis: operation("spine:basis") },
+ result: { kind: "output", name: "visual" },
+ },
+ {
+ id: "spine:audio-track",
+ producer: speechTakeProducers.projectAudioTrack,
+ inputs: { basis: operation("spine:basis") },
+ result: { kind: "output", name: "track" },
+ },
+ );
+ const semanticInputs = ["program", ...options.takes.flatMap((take) => [take.mediaName, take.segmentName])];
+ return sealGraphFragment({
+ name: options.name?.trim() || "@svml/speech-program/spine@1",
+ inputs: [
+ { name: "program", type: speechProgramTypes.spineProgram },
+ ...options.takes.flatMap((take) => [
+ { name: take.mediaName, type: contractTypes.synchronizedMedia },
+ { name: take.segmentName, type: contractTypes.narrativeExcerpt },
+ ]),
+ ],
+ operations,
+ exports: [
+ { name: "basis", type: contractTypes.speechBasis, root: operation("spine:basis"), semanticInputs, fidelity: "exact" },
+ {
+ name: "space", type: contractTypes.programSpace, root: operation("spine:space"), semanticInputs,
+ affinity: [{ resultPointer: "/digest", source: operation("spine:basis"), sourcePointer: "/programSpace/digest" }], fidelity: "exact",
+ },
+ {
+ name: "audio", type: contractTypes.speechAudioBasis, root: operation("spine:audio-basis"), semanticInputs,
+ affinity: [{ resultPointer: "/programSpace/digest", source: operation("spine:basis"), sourcePointer: "/programSpace/digest" }], fidelity: "exact",
+ },
+ {
+ name: "visual", type: contractTypes.visualTrack, root: operation("spine:visual-track"), semanticInputs,
+ affinity: [{ resultPointer: "/programSpaceDigest", source: operation("spine:basis"), sourcePointer: "/programSpace/digest" }], fidelity: "exact",
+ },
+ {
+ name: "audioTrack", type: contractTypes.audioTrack, root: operation("spine:audio-track"), semanticInputs,
+ affinity: [{ resultPointer: "/programSpaceDigest", source: operation("spine:basis"), sourcePointer: "/programSpace/digest" }], fidelity: "exact",
+ },
+ ],
+ });
+}
diff --git a/packages/speech-program/src/index.ts b/packages/speech-program/src/index.ts
index 75fb3e00..b7ea1553 100644
--- a/packages/speech-program/src/index.ts
+++ b/packages/speech-program/src/index.ts
@@ -3,5 +3,32 @@ export {
captionTrackFragment,
captionTimingFragment,
speechTakeProjectionFragment,
- whisperXSpeechAlignmentFragment,
} from "./speech.js";
+export { whisperXSpeechAlignmentFragment } from "@svml/whisperx";
+export { speechProgramComponent } from "./component.js";
+export { createSpeechSpineFragment } from "./fragment.js";
+export {
+ speechProgramManifest,
+ speechProgramManifestDigest,
+ speechProgramModuleRef,
+ speechProgramProducers,
+ speechProgramTypes,
+ speechSpineProgramSchema,
+ speechSpineSetSchema,
+ speechSpineSurfaceImplementationDigest,
+} from "./manifest.js";
+export {
+ appendSpeechSpineTake,
+ appendSpeechSpineTakeImplementationDigest,
+ assembleSpeechBasis,
+ assembleSpeechBasisImplementationDigest,
+ assertSpeechSpineProgram,
+ assertSpeechSpineSet,
+ compileSpeechSpineAudio,
+ compileSpeechSpineAudioImplementationDigest,
+ createSpeechSpineSet,
+ createSpeechSpineSetImplementationDigest,
+ sealSpeechSpineProgram,
+} from "./program.js";
+export { decodeSpeechSpineSurface } from "./surface.js";
+export type * from "./types.js";
diff --git a/packages/speech-program/src/manifest.ts b/packages/speech-program/src/manifest.ts
new file mode 100644
index 00000000..491e5148
--- /dev/null
+++ b/packages/speech-program/src/manifest.ts
@@ -0,0 +1,149 @@
+import {
+ contractTypes,
+ narrativeExcerptSchema,
+ synchronizedMediaSchema,
+ videoContractDependencies,
+} from "@svml/contracts";
+import {
+ mediaPipelineManifest,
+ mediaPipelineModuleRef,
+ mediaPipelineTypes,
+} from "@svml/media-pipeline";
+import { digestOf } from "@svml/protocol";
+import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";
+import { speechTakeManifest, speechTakeModuleRef } from "@svml/speech-take";
+
+import {
+ appendSpeechSpineTakeImplementationDigest,
+ assembleSpeechBasisImplementationDigest,
+ compileSpeechSpineAudioImplementationDigest,
+ createSpeechSpineSetImplementationDigest,
+} from "./program.js";
+
+export const speechProgramModuleRef = { name: "@svml/speech-program", version: "0.0.0-dev" } as const;
+export const speechSpineSurfaceImplementationDigest = digestOf("@svml/speech-program/spine-surface@1");
+export const speechProgramTypes = {
+ spineProgram: { module: speechProgramModuleRef, name: "SpeechSpineProgram" },
+ spineSet: { module: speechProgramModuleRef, name: "SpeechSpineSet" },
+} satisfies Record<string, TypeRef>;
+export const speechProgramProducers = {
+ createSet: { module: speechProgramModuleRef, name: "create-spine-set" },
+ appendTake: { module: speechProgramModuleRef, name: "append-spine-take" },
+ compileAudio: { module: speechProgramModuleRef, name: "compile-spine-audio" },
+ assembleBasis: { module: speechProgramModuleRef, name: "assemble-speech-basis" },
+} satisfies Record<string, ProducerRef>;
+
+const string = { kind: "string", minLength: 1 } as const;
+const integer = { kind: "number", integer: true, minimum: 1 } as const;
+const digest = { kind: "string", minLength: 71, maxLength: 71 } as const;
+const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>): ValueSchema => ({
+ kind: "object",
+ fields,
+});
+const frameRate = object({ numerator: { schema: integer }, denominator: { schema: integer } });
+
+export const speechSpineProgramSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.speech-spine-program@1" } },
+ digest: { schema: digest },
+ id: { schema: string },
+ frameRate: { schema: frameRate },
+});
+
+export const speechSpineSetSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.speech-spine-set@1" } },
+ digest: { schema: digest },
+ program: { schema: speechSpineProgramSchema },
+ takes: { schema: { kind: "array", items: object({
+ segment: { schema: narrativeExcerptSchema },
+ media: { schema: synchronizedMediaSchema },
+ }) } },
+ lastAddition: { schema: object({
+ previousSetDigest: { schema: digest },
+ segmentDigest: { schema: digest },
+ mediaDigest: { schema: digest },
+ }), optional: true },
+});
+
+export const speechProgramManifest: ModuleManifest = {
+ format: "svml.module@0",
+ name: speechProgramModuleRef.name,
+ version: speechProgramModuleRef.version,
+ dependencies: [
+ videoContractDependencies.narrative,
+ videoContractDependencies.media,
+ videoContractDependencies.programSpace,
+ videoContractDependencies.speech,
+ videoContractDependencies.composition,
+ { module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },
+ { module: speechTakeModuleRef, digest: digestOf(speechTakeManifest) },
+ ],
+ types: [
+ { name: speechProgramTypes.spineProgram.name, schema: speechSpineProgramSchema },
+ { name: speechProgramTypes.spineSet.name, schema: speechSpineSetSchema },
+ ],
+ capabilities: [],
+ surfaces: [{
+ name: "spine",
+ tag: "Spine",
+ mode: "structured",
+ outputs: [speechProgramTypes.spineProgram, mediaPipelineTypes.selectionRequest,
+ contractTypes.speechBasis, contractTypes.programSpace, contractTypes.speechAudioBasis,
+ contractTypes.visualTrack, contractTypes.audioTrack],
+ implementation: {
+ kind: "trusted-frontend-surface",
+ locator: "@svml/speech-program/spine-surface",
+ digest: speechSpineSurfaceImplementationDigest,
+ },
+ }],
+ producers: [
+ {
+ name: speechProgramProducers.createSet.name,
+ inputs: [{ name: "program", type: speechProgramTypes.spineProgram }],
+ outputs: [{ name: "set", type: speechProgramTypes.spineSet }],
+ needs: [],
+ implementation: { kind: "registered", locator: "@svml/speech-program/create-spine-set", digest: createSpeechSpineSetImplementationDigest },
+ },
+ {
+ name: speechProgramProducers.appendTake.name,
+ inputs: [
+ { name: "set", type: speechProgramTypes.spineSet },
+ { name: "media", type: contractTypes.synchronizedMedia },
+ { name: "segment", type: contractTypes.narrativeExcerpt },
+ ],
+ outputs: [{
+ name: "set",
+ type: speechProgramTypes.spineSet,
+ affinity: [
+ { resultPointer: "/lastAddition/previousSetDigest", input: "set", inputPointer: "/digest" },
+ { resultPointer: "/lastAddition/mediaDigest", input: "media", inputPointer: "/synchronizedMediaDigest" },
+ { resultPointer: "/lastAddition/segmentDigest", input: "segment", inputPointer: "/excerptDigest" },
+ ],
+ }],
+ needs: [],
+ implementation: { kind: "registered", locator: "@svml/speech-program/append-spine-take", digest: appendSpeechSpineTakeImplementationDigest },
+ },
+ {
+ name: speechProgramProducers.compileAudio.name,
+ inputs: [{ name: "set", type: speechProgramTypes.spineSet }],
+ outputs: [{ name: "plan", type: mediaPipelineTypes.audioProgramPlan }],
+ needs: [],
+ implementation: { kind: "registered", locator: "@svml/speech-program/compile-spine-audio", digest: compileSpeechSpineAudioImplementationDigest },
+ },
+ {
+ name: speechProgramProducers.assembleBasis.name,
+ inputs: [
+ { name: "set", type: speechProgramTypes.spineSet },
+ { name: "audio", type: contractTypes.timelineAudio },
+ ],
+ outputs: [{
+ name: "basis",
+ type: contractTypes.speechBasis,
+ affinity: [{ resultPointer: "/audio/digest", input: "audio", inputPointer: "/artifact/digest" }],
+ }],
+ needs: [],
+ implementation: { kind: "registered", locator: "@svml/speech-program/assemble-speech-basis", digest: assembleSpeechBasisImplementationDigest },
+ },
+ ],
+};
+
+export const speechProgramManifestDigest = digestOf(speechProgramManifest);
diff --git a/packages/speech-program/src/program.ts b/packages/speech-program/src/program.ts
new file mode 100644
index 00000000..512c333a
--- /dev/null
+++ b/packages/speech-program/src/program.ts
@@ -0,0 +1,245 @@
+import {
+ assertProgramSpaceIdentity,
+ sealProgramSpace,
+ sealSpeechBasis,
+ verifySynchronizedMedia,
+ verifyTimelineAudio,
+} from "@svml/contracts";
+import type {
+ NarrativeExcerpt,
+ ProgramSpace,
+ SpeechBasis,
+ SynchronizedMedia,
+ TimelineAudio,
+} from "@svml/contracts";
+import {
+ sealAudioProgramPlan,
+ verifyAudioProgramPlan,
+} from "@svml/media-pipeline";
+import type { AudioProgramPlan } from "@svml/media-pipeline";
+import { canonicalize, digestOf, isDigest } from "@svml/protocol";
+import type { Digest } from "@svml/protocol";
+
+import type {
+ SpeechSpineProgram,
+ SpeechSpineSet,
+ SpeechSpineTake,
+} from "./types.js";
+
+export const createSpeechSpineSetImplementationDigest = digestOf("@svml/speech-program/create-spine-set@1");
+export const appendSpeechSpineTakeImplementationDigest = digestOf("@svml/speech-program/append-spine-take@1");
+export const compileSpeechSpineAudioImplementationDigest = digestOf("@svml/speech-program/compile-spine-audio@1");
+export const assembleSpeechBasisImplementationDigest = digestOf("@svml/speech-program/assemble-speech-basis@1");
+
+function assert(condition: unknown, message: string): asserts condition {
+ if (!condition) throw new Error(message);
+}
+
+function programContent(value: Omit<SpeechSpineProgram, "digest">): Omit<SpeechSpineProgram, "digest"> {
+ return {
+ contract: "svml.speech-spine-program@1",
+ id: value.id,
+ frameRate: { ...value.frameRate },
+ };
+}
+
+export function sealSpeechSpineProgram(value: Omit<SpeechSpineProgram, "digest">): SpeechSpineProgram {
+ const content = programContent(value);
+ return { ...content, digest: digestOf(content) };
+}
+
+export function assertSpeechSpineProgram(value: SpeechSpineProgram): void {
+ assert(value.contract === "svml.speech-spine-program@1", "Unsupported SpeechSpineProgram contract");
+ assert(value.id.trim().length > 0, "SpeechSpineProgram id must not be empty");
+ assert(Number.isSafeInteger(value.frameRate.numerator) && value.frameRate.numerator > 0
+ && Number.isSafeInteger(value.frameRate.denominator) && value.frameRate.denominator > 0,
+ "SpeechSpineProgram frame rate is invalid");
+ const { digest: _digest, ...content } = value;
+ assert(isDigest(value.digest) && value.digest === digestOf(canonicalize(content)),
+ "SpeechSpineProgram digest differs from its contents");
+}
+
+function verifyExcerpt(value: NarrativeExcerpt): void {
+ assert(value.contract === "svml.narrative-excerpt@1" && value.kind === "segment",
+ "Speech Spine Take must reference a Segment NarrativeExcerpt");
+ assert(value.id.length > 0 && Number.isSafeInteger(value.tokenStart)
+ && Number.isSafeInteger(value.tokenEndExclusive) && value.tokenEndExclusive > value.tokenStart,
+ "Speech Spine Segment excerpt is invalid");
+ const { excerptDigest: _digest, ...content } = value;
+ assert(isDigest(value.excerptDigest) && value.excerptDigest === digestOf(canonicalize(content)),
+ "Speech Spine Segment excerpt digest differs from its contents");
+}
+
+function setContent(value: Omit<SpeechSpineSet, "digest">): Omit<SpeechSpineSet, "digest"> {
+ return canonicalize({
+ contract: "svml.speech-spine-set@1",
+ program: value.program,
+ takes: value.takes,
+ ...(value.lastAddition === undefined ? {} : { lastAddition: value.lastAddition }),
+ }) as unknown as Omit<SpeechSpineSet, "digest">;
+}
+
+function sealSpeechSpineSet(value: Omit<SpeechSpineSet, "digest">): SpeechSpineSet {
+ const content = setContent(value);
+ return { ...content, digest: digestOf(content) };
+}
+
+function assertTake(take: SpeechSpineTake, program: SpeechSpineProgram): void {
+ verifyExcerpt(take.segment);
+ verifySynchronizedMedia(take.media);
+ assert(take.media.visual !== undefined, `Speech Segment ${take.segment.id} has no moving visual`);
+ assert(take.media.audio !== undefined, `Speech Segment ${take.segment.id} has no speech audio`);
+ assert(take.media.timeline.frameRate.numerator === program.frameRate.numerator
+ && take.media.timeline.frameRate.denominator === program.frameRate.denominator,
+ `Speech Segment ${take.segment.id} uses another frame rate`);
+}
+
+export function assertSpeechSpineSet(value: SpeechSpineSet): void {
+ assert(value.contract === "svml.speech-spine-set@1", "Unsupported SpeechSpineSet contract");
+ assertSpeechSpineProgram(value.program);
+ const { digest: _digest, ...content } = value;
+ assert(isDigest(value.digest) && value.digest === digestOf(setContent(content)),
+ "SpeechSpineSet digest differs from its contents");
+ const segments = new Set<string>();
+ for (const take of value.takes) {
+ assertTake(take, value.program);
+ assert(!segments.has(take.segment.id), `Speech Spine repeats Segment ${take.segment.id}`);
+ segments.add(take.segment.id);
+ }
+ assert(value.takes.length === 0 ? value.lastAddition === undefined : value.lastAddition !== undefined,
+ "SpeechSpineSet last addition is inconsistent");
+ if (value.lastAddition !== undefined) {
+ const last = value.takes.at(-1)!;
+ assert(isDigest(value.lastAddition.previousSetDigest)
+ && value.lastAddition.segmentDigest === last.segment.excerptDigest
+ && value.lastAddition.mediaDigest === last.media.synchronizedMediaDigest,
+ "SpeechSpineSet last addition is invalid");
+ }
+}
+
+export function createSpeechSpineSet(program: SpeechSpineProgram): SpeechSpineSet {
+ assertSpeechSpineProgram(program);
+ return sealSpeechSpineSet({
+ contract: "svml.speech-spine-set@1",
+ program,
+ takes: [],
+ });
+}
+
+export function appendSpeechSpineTake(
+ set: SpeechSpineSet,
+ media: SynchronizedMedia,
+ segment: NarrativeExcerpt,
+): SpeechSpineSet {
+ assertSpeechSpineSet(set);
+ const take = { media, segment } satisfies SpeechSpineTake;
+ assertTake(take, set.program);
+ assert(!set.takes.some((item) => item.segment.id === segment.id), `Speech Spine repeats Segment ${segment.id}`);
+ return sealSpeechSpineSet({
+ contract: "svml.speech-spine-set@1",
+ program: set.program,
+ takes: [...set.takes, take],
+ lastAddition: {
+ previousSetDigest: set.digest,
+ segmentDigest: segment.excerptDigest,
+ mediaDigest: media.synchronizedMediaDigest,
+ },
+ });
+}
+
+function frameSample(frame: number, frameRate: SpeechSpineProgram["frameRate"]): number {
+ const numerator = BigInt(frame) * 48_000n * BigInt(frameRate.denominator);
+ const denominator = BigInt(frameRate.numerator);
+ const rounded = (numerator * 2n + denominator) / (denominator * 2n);
+ assert(rounded <= BigInt(Number.MAX_SAFE_INTEGER), "Speech Spine sample boundary exceeds safe arithmetic");
+ return Number(rounded);
+}
+
+function programSpace(set: SpeechSpineSet): ProgramSpace {
+ assertSpeechSpineSet(set);
+ assert(set.takes.length > 0, "Speech Spine must contain at least one Take");
+ const frameCount = set.takes.reduce((sum, take) => sum + take.media.timeline.frameCount, 0);
+ const space = sealProgramSpace({
+ contract: "svml.program-space@0",
+ durationSec: frameCount * set.program.frameRate.denominator / set.program.frameRate.numerator,
+ frameRate: { ...set.program.frameRate },
+ });
+ assertProgramSpaceIdentity(space);
+ return space;
+}
+
+export function compileSpeechSpineAudio(set: SpeechSpineSet): AudioProgramPlan {
+ const space = programSpace(set);
+ let frame = 0;
+ const clips = set.takes.map((take, index) => {
+ const audio = take.media.audio!;
+ const startFrame = frame;
+ frame += take.media.timeline.frameCount;
+ const targetStartSample = frameSample(startFrame, set.program.frameRate);
+ const targetEndSampleExclusive = frameSample(frame, set.program.frameRate);
+ assert(targetEndSampleExclusive - targetStartSample === audio.sampleFrames,
+ `Speech Segment ${take.segment.id} audio does not exactly cover its normalized frame span`);
+ return {
+ id: `${String(index + 1).padStart(4, "0")}:${take.segment.id}`,
+ artifact: audio.artifact,
+ targetStartSample,
+ targetEndSampleExclusive,
+ sourceStartSample: 0,
+ playbackRate: 1,
+ gain: 1,
+ fadeInSamples: 0,
+ fadeOutSamples: 0,
+ bus: "speech" as const,
+ };
+ });
+ const plan = sealAudioProgramPlan({
+ contract: "svml.audio-program-plan@1",
+ programSpaceDigest: space.digest,
+ frameRate: { ...space.frameRate },
+ frameCount: frame,
+ sampleRate: 48_000,
+ sampleFrames: frameSample(frame, set.program.frameRate),
+ clips,
+ mix: { normalize: false, limiter: "none" },
+ });
+ verifyAudioProgramPlan(plan);
+ return plan;
+}
+
+export function assembleSpeechBasis(set: SpeechSpineSet, audio: TimelineAudio): SpeechBasis {
+ const space = programSpace(set);
+ const plan = compileSpeechSpineAudio(set);
+ verifyTimelineAudio(audio);
+ assert(audio.planDigest === plan.planDigest && audio.programSpaceDigest === space.digest,
+ "TimelineAudio does not realize this Speech Spine plan");
+ let frame = 0;
+ const seconds = (value: number): number => value * space.frameRate.denominator / space.frameRate.numerator;
+ const segments = set.takes.map((take) => {
+ const startFrame = frame;
+ frame += take.media.timeline.frameCount;
+ return { segmentId: take.segment.id, startSec: seconds(startFrame), endSec: seconds(frame) };
+ });
+ const visualClips = set.takes.map((take, index) => ({
+ segmentId: take.segment.id,
+ artifact: {
+ digest: take.media.visual!.artifact.digest,
+ size: take.media.visual!.artifact.size,
+ mediaType: take.media.visual!.artifact.mediaType,
+ durationSec: seconds(take.media.timeline.frameCount),
+ },
+ startSec: segments[index]!.startSec,
+ endSec: segments[index]!.endSec,
+ }));
+ return sealSpeechBasis({
+ contract: "svml.speech-basis@1",
+ programSpace: space,
+ audio: {
+ digest: audio.artifact.digest,
+ size: audio.artifact.size,
+ mediaType: audio.artifact.mediaType,
+ durationSec: space.durationSec,
+ },
+ visualTrack: { clips: visualClips },
+ segments,
+ });
+}
diff --git a/packages/speech-program/src/speech.ts b/packages/speech-program/src/speech.ts
index 186c1f9c..f55e9b0b 100644
--- a/packages/speech-program/src/speech.ts
+++ b/packages/speech-program/src/speech.ts
@@ -1,10 +1,7 @@
import { captionProducers, captionTypes } from "@svml/caption";
import { contractTypes } from "@svml/contracts";
import { sealGraphFragment } from "@svml/elaborator";
-import { speechAlignProducers } from "@svml/speech-align";
import { speechTakeProducers } from "@svml/speech-take";
-import { whisperXProducers, whisperXTypes } from "@svml/whisperx";
-import { mediaPipelineProducers } from "@svml/media-pipeline";
const input = (name: string) => ({ kind: "fragment-input" as const, name });
const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
@@ -82,79 +79,6 @@ export const speechTakeProjectionFragment = sealGraphFragment({
],
});
-/** One WhisperX pass can expose both normalized Evidence and the final SemanticMap. */
-export const whisperXSpeechAlignmentFragment = sealGraphFragment({
- name: "@svml/speech-program/whisperx-speech-alignment@1",
- inputs: [
- { name: "narrative", type: contractTypes.narrative },
- { name: "audio", type: contractTypes.speechAudioBasis },
- ],
- operations: [
- {
- id: "prepare-evidence-audio",
- producer: mediaPipelineProducers.projectSpeechEvidenceAudio,
- inputs: { audio: input("audio") },
- result: { kind: "need", name: "evidenceAudio", accepts: "exact" },
- },
- {
- id: "request-whisperx",
- producer: whisperXProducers.request,
- inputs: { audio: operation("prepare-evidence-audio") },
- result: { kind: "need", name: "alignment", accepts: "exact" },
- },
- {
- id: "normalize-evidence",
- producer: whisperXProducers.normalize,
- inputs: { whisperx: operation("request-whisperx") },
- result: { kind: "output", name: "evidence" },
- },
- {
- id: "locate-speech",
- producer: speechAlignProducers.locate,
- inputs: {
- narrative: input("narrative"),
- audio: input("audio"),
- evidence: operation("normalize-evidence"),
- },
- result: { kind: "output", name: "map" },
- },
- ],
- exports: [
- {
- name: "rawEvidence",
- type: whisperXTypes.alignmentEvidence,
- root: operation("request-whisperx"),
- semanticInputs: ["audio"],
- affinity: [
- { resultPointer: "/audioArtifactDigest", source: operation("prepare-evidence-audio"), sourcePointer: "/artifact/digest" },
- { resultPointer: "/programSpaceDigest", source: input("audio"), sourcePointer: "/programSpace/digest" },
- ],
- fidelity: "exact",
- },
- {
- name: "evidence",
- type: contractTypes.alignedTranscriptEvidence,
- root: operation("normalize-evidence"),
- semanticInputs: ["audio"],
- affinity: [
- { resultPointer: "/audioArtifactDigest", source: operation("prepare-evidence-audio"), sourcePointer: "/artifact/digest" },
- { resultPointer: "/programSpaceDigest", source: input("audio"), sourcePointer: "/programSpace/digest" },
- ],
- fidelity: "exact",
- },
- {
- name: "map",
- type: contractTypes.completeSemanticMap,
- root: operation("locate-speech"),
- semanticInputs: ["narrative", "audio"],
- affinity: [
- { resultPointer: "/programSpace/digest", source: input("audio"), sourcePointer: "/programSpace/digest" },
- ],
- fidelity: "exact",
- },
- ],
-});
-
export const captionTimingFragment = sealGraphFragment({
name: "@svml/speech-program/caption-timing@1",
inputs: [
diff --git a/packages/speech-program/src/surface.ts b/packages/speech-program/src/surface.ts
new file mode 100644
index 00000000..b4c9ad70
--- /dev/null
+++ b/packages/speech-program/src/surface.ts
@@ -0,0 +1,137 @@
+import { contractTypes } from "@svml/contracts";
+import {
+ sealMediaSelectionRequest,
+ synchronizedMediaFragment,
+} from "@svml/media-pipeline";
+import type {
+ StructuredElement,
+ StructuredSurfaceHandler,
+ SurfaceResolvedReference,
+ TextAttributeValue,
+} from "@svml/text";
+
+import { createSpeechSpineFragment } from "./fragment.js";
+import { speechProgramTypes } from "./manifest.js";
+import { sealSpeechSpineProgram } from "./program.js";
+
+function localName(value: string): string {
+ return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value;
+}
+
+function exactAttributes(element: StructuredElement, names: readonly string[]): void {
+ if (Object.keys(element.attributes).sort().join("\u0000") !== [...names].sort().join("\u0000")) {
+ throw new Error(`${element.name} requires exactly ${names.join(", ")}`);
+ }
+}
+
+function stringAttribute(element: StructuredElement, name: string): string {
+ const value = element.attributes[name];
+ if (typeof value !== "string" || value.trim().length === 0) throw new Error(`${element.name}.${name} must be a non-empty string`);
+ return value.trim();
+}
+
+function referencePath(element: StructuredElement, name: string): string {
+ const value: TextAttributeValue | undefined = element.attributes[name];
+ if (typeof value !== "object" || value.kind !== "reference" || !value.path) {
+ throw new Error(`${element.name}.${name} must be a whole-value reference`);
+ }
+ return value.path;
+}
+
+function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+}
+
+function resolve(
+ element: StructuredElement,
+ name: string,
+ expected: SurfaceResolvedReference["type"],
+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,
+): SurfaceResolvedReference {
+ const path = referencePath(element, name);
+ const value = resolveReference(path);
+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${path}`);
+ if (!sameType(value.type, expected)) throw new Error(`${element.name}.${name} has the wrong type`);
+ return value;
+}
+
+function takes(element: StructuredElement): StructuredElement[] {
+ const result: StructuredElement[] = [];
+ for (const child of element.children) {
+ if (child.kind === "text") {
+ if (child.value.trim()) throw new Error(`${element.name} accepts only Take children`);
+ continue;
+ }
+ if (localName(child.name) !== "Take") throw new Error(`${element.name} accepts only Take children`);
+ exactAttributes(child, ["source", "segment"]);
+ result.push(child);
+ }
+ if (result.length === 0) throw new Error(`${element.name} requires at least one Take`);
+ return result;
+}
+
+export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ exactAttributes(element, ["id"]);
+ const id = stringAttribute(element, "id");
+ const declaredTakes = takes(element).map((take, index) => ({
+ mediaName: `take-${String(index + 1).padStart(4, "0")}-media`,
+ segmentName: `take-${String(index + 1).padStart(4, "0")}-segment`,
+ source: resolve(take, "source", contractTypes.blobArtifact, resolveReference),
+ segment: resolve(take, "segment", contractTypes.narrativeExcerpt, resolveReference),
+ range: take.range,
+ }));
+ const programId = `${id}.program`;
+ const requestId = `${id}.selection`;
+ const program = sealSpeechSpineProgram({
+ contract: "svml.speech-spine-program@1",
+ id,
+ frameRate: { numerator: 30, denominator: 1 },
+ });
+ const request = sealMediaSelectionRequest({
+ contract: "svml.media-selection-request@1",
+ video: { mode: "primary-moving" },
+ audio: { mode: "default" },
+ spanAuthority: "video",
+ frameRate: { numerator: 30, denominator: 1 },
+ });
+ const assembly = createSpeechSpineFragment({
+ name: `@svml/speech-program/surface/${id}@1`,
+ takes: declaredTakes.map(({ mediaName, segmentName }) => ({ mediaName, segmentName })),
+ });
+ const normalizationComponents = declaredTakes.map((take, index) => ({
+ id: `${id}.normalize.${String(index + 1).padStart(4, "0")}`,
+ fragment: synchronizedMediaFragment.id,
+ inputs: { source: take.source.ref, request: { kind: "record" as const, id: requestId } },
+ outputs: { media: `${id}.normalized.${String(index + 1).padStart(4, "0")}` },
+ range: take.range,
+ }));
+ return {
+ records: [
+ { id: programId, type: speechProgramTypes.spineProgram, value: { kind: "inline", value: program }, range: element.range },
+ { id: requestId, type: { module: { name: "@svml/media-pipeline", version: "0.0.0-dev" }, name: "MediaSelectionRequest" }, value: { kind: "inline", value: request }, range: element.range },
+ ],
+ components: [
+ ...normalizationComponents,
+ {
+ id,
+ fragment: assembly.id,
+ inputs: {
+ program: { kind: "record", id: programId },
+ ...Object.fromEntries(declaredTakes.flatMap((take, index) => [
+ [take.mediaName, { kind: "component-output" as const, component: normalizationComponents[index]!.id, output: "media" }],
+ [take.segmentName, take.segment.ref],
+ ])),
+ },
+ outputs: {
+ basis: `${id}.basis`,
+ space: `${id}.space`,
+ audio: `${id}.audio`,
+ visual: `${id}.visual`,
+ audioTrack: `${id}.audioTrack`,
+ },
+ range: element.range,
+ },
+ ],
+ fragments: [synchronizedMediaFragment, assembly],
+ };
+};
diff --git a/packages/speech-program/src/types.ts b/packages/speech-program/src/types.ts
new file mode 100644
index 00000000..ea340b95
--- /dev/null
+++ b/packages/speech-program/src/types.ts
@@ -0,0 +1,47 @@
+import type {
+ NarrativeExcerpt,
+ SynchronizedMedia,
+} from "@svml/contracts";
+import type { Digest } from "@svml/protocol";
+
+/** Author-owned policy for one ordered Speech Spine. */
+export type SpeechSpineProgram = {
+ readonly contract: "svml.speech-spine-program@1";
+ readonly digest: Digest;
+ readonly id: string;
+ readonly frameRate: {
+ readonly numerator: number;
+ readonly denominator: number;
+ };
+};
+
+export type SpeechSpineTake = {
+ readonly segment: NarrativeExcerpt;
+ readonly media: SynchronizedMedia;
+};
+
+/**
+ * Package-private immutable fold value. Its order is author meaning; Core sees
+ * only fixed-port append Operations for the concrete Spine declaration.
+ */
+export type SpeechSpineSet = {
+ readonly contract: "svml.speech-spine-set@1";
+ readonly digest: Digest;
+ readonly program: SpeechSpineProgram;
+ readonly takes: readonly SpeechSpineTake[];
+ readonly lastAddition?: {
+ readonly previousSetDigest: Digest;
+ readonly segmentDigest: Digest;
+ readonly mediaDigest: Digest;
+ };
+};
+
+export type SpeechSpineInput = {
+ readonly mediaName: string;
+ readonly segmentName: string;
+};
+
+export type SpeechSpineFragmentOptions = {
+ readonly name?: string;
+ readonly takes: readonly SpeechSpineInput[];
+};
diff --git a/packages/speech-program/test/surface.test.ts b/packages/speech-program/test/surface.test.ts
new file mode 100644
index 00000000..9b0bb2c9
--- /dev/null
+++ b/packages/speech-program/test/surface.test.ts
@@ -0,0 +1,127 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+
+import { registerTypeValidatorFacets } from "@svml/component-kit";
+import {
+ contractTypes,
+ videoContractDependencies,
+ videoContractManifests,
+} from "@svml/contracts";
+import { createResolvedClosure, digestOf, sealBuildRequest, start } from "@svml/core";
+import {
+ AuthorFrontendRegistry,
+ compileSourceClosure,
+ resolveCompiledSourceExport,
+} from "@svml/elaborator";
+import type { AuthorSourceUnit } from "@svml/elaborator";
+import {
+ mediaPipelineComponent,
+ mediaPipelineManifest,
+ mediaPipelineProducers,
+} from "@svml/media-pipeline";
+import type { ModuleManifest } from "@svml/protocol";
+import {
+ decodeScriptSurface,
+ scriptManifest,
+ scriptModuleRef,
+ scriptSurfaceImplementationDigest,
+} from "@svml/script";
+import { speechTakeManifest, speechTakeProducers } from "@svml/speech-take";
+import {
+ decodeSpeechSpineSurface,
+ speechProgramManifest,
+ speechProgramModuleRef,
+ speechProgramProducers,
+ speechSpineSurfaceImplementationDigest,
+} from "@svml/speech-program";
+import {
+ TextSurfaceRegistry,
+ createTextAuthorFrontend,
+ textAuthorFrontendId,
+} from "@svml/text";
+import { createRecordAdmitter, TypeValidatorRegistry } from "@svml/validation";
+
+const fixtureModule = { name: "example.speech-media", version: "1" } as const;
+const fixtureSurfaceDigest = digestOf("example.speech-media/surface@1");
+const fixtureManifest: ModuleManifest = {
+ format: "svml.module@0",
+ name: fixtureModule.name,
+ version: fixtureModule.version,
+ dependencies: [videoContractDependencies.media],
+ types: [], capabilities: [], producers: [],
+ surfaces: [{
+ name: "media", tag: "Media", mode: "structured", outputs: [contractTypes.blobArtifact],
+ implementation: { kind: "trusted-frontend-surface", locator: "example.speech-media/surface", digest: fixtureSurfaceDigest },
+ }],
+};
+
+function source(text: string): AuthorSourceUnit {
+ return { id: "/project/main.svml", name: "main.svml", text };
+}
+
+test("Speech Spine lowers ordered Takes into media normalization, one audio plan and peer projections", async () => {
+ const closure = createResolvedClosure([
+ ...videoContractManifests,
+ mediaPipelineManifest,
+ speechTakeManifest,
+ speechProgramManifest,
+ scriptManifest,
+ fixtureManifest,
+ ]);
+ const surfaces = new TextSurfaceRegistry();
+ surfaces.registerRaw(scriptModuleRef, "script", scriptSurfaceImplementationDigest, decodeScriptSurface);
+ surfaces.registerStructured(speechProgramModuleRef, "spine", speechSpineSurfaceImplementationDigest, decodeSpeechSpineSurface);
+ surfaces.registerStructured(fixtureModule, "media", fixtureSurfaceDigest, ({ element }) => ({
+ records: ["take-one", "take-two"].map((id) => ({
+ id, type: contractTypes.blobArtifact,
+ value: { kind: "blob" as const, digest: digestOf(id), size: 128, mediaType: "video/mp4" },
+ range: element.range,
+ })),
+ components: [], fragments: [],
+ }));
+ const frontends = new AuthorFrontendRegistry();
+ frontends.register(createTextAuthorFrontend({
+ registry: surfaces,
+ resolveModule(request) {
+ if (request.from.startsWith("@svml/script")) return scriptModuleRef;
+ if (request.from.startsWith("@svml/speech")) return speechProgramModuleRef;
+ return fixtureModule;
+ },
+ }));
+ const validators = new TypeValidatorRegistry();
+ registerTypeValidatorFacets(validators, mediaPipelineComponent.validators ?? []);
+ const compiled = await compileSourceClosure({
+ entry: source(`<svml>
+ <import from="@svml/script@1"/>
+ <import as="fixture" from="example.speech-media@1"/>
+ <import as="speech" from="@svml/speech@1"/>
+ <script id="story">
+ <opening><ALICE> Hello from Alice.</opening>
+ <answer><BOB> Hello from Bob.</answer>
+ </script>
+ <fixture:Media/>
+ <speech:Spine id="speech">
+ <speech:Take source={take-one} segment={story.segment.opening}/>
+ <speech:Take source={take-two} segment={story.segment.answer}/>
+ </speech:Spine>
+ </svml>`),
+ frontend: textAuthorFrontendId,
+ closure,
+ frontends,
+ admitRecord: createRecordAdmitter(validators),
+ resolveSource() { throw new Error("fixture has no source imports"); },
+ });
+ const target = resolveCompiledSourceExport(compiled, "speech.visual", contractTypes.visualTrack);
+ const build = start(compiled.program, compiled.elaboration.graph, sealBuildRequest({
+ graph: compiled.elaboration.graph.id,
+ targets: [{ output: target.ref.kind === "logical-output" ? target.ref.id : "", accepts: "exact" }],
+ bindings: [],
+ }));
+ const names = build.plan.steps.map((step) => step.producer.name);
+ assert.equal(names.filter((name) => name === mediaPipelineProducers.inspect.name).length, 2);
+ assert.equal(names.filter((name) => name === mediaPipelineProducers.normalize.name).length, 2);
+ assert.equal(names.filter((name) => name === speechProgramProducers.appendTake.name).length, 2);
+ assert.equal(names.filter((name) => name === speechProgramProducers.compileAudio.name).length, 1);
+ assert.equal(names.filter((name) => name === mediaPipelineProducers.renderAudio.name).length, 1);
+ assert.equal(names.filter((name) => name === speechTakeProducers.projectVisual.name).length, 1);
+});
diff --git a/packages/text-track/README.md b/packages/text-track/README.md
index 50dabb65..489587a0 100644
--- a/packages/text-track/README.md
+++ b/packages/text-track/README.md
@@ -11,5 +11,7 @@ An item spanning the complete ProgramSpace is a persistent overlay; a shorter it
are not different Track kinds. The package exposes semantic typography and box parameters rather
than a rendering callback, arbitrary CSS or cross-Track access.
-No author-facing Surface is frozen yet. A future Text Surface and `.svs` recipes can produce the
-same TextTrackProgram without changing Core, Film, Composition or HyperFrames.
+The provider-free `<text:Track>` Surface now validates `.svs` Recipes, compiles `during="full"`
+against an explicitly connected ProgramSpace and produces the same TextTrackProgram without
+changing Core, Film, Composition or HyperFrames. Timed semantic windows and exact FontArtifact
+lowering remain pre-freeze work.
diff --git a/packages/text-track/package.json b/packages/text-track/package.json
index 085ef7f4..eb11ac4d 100644
--- a/packages/text-track/package.json
+++ b/packages/text-track/package.json
@@ -7,9 +7,12 @@
".": "./src/index.ts"
},
"dependencies": {
+ "@svml/component-kit": "workspace:*",
"@svml/contracts": "workspace:*",
"@svml/elaborator": "workspace:*",
- "@svml/protocol": "workspace:*"
+ "@svml/protocol": "workspace:*",
+ "@svml/svs": "workspace:*",
+ "@svml/text": "workspace:*"
},
"devDependencies": {
"@svml/hyperframes": "workspace:*"
diff --git a/packages/text-track/src/component.ts b/packages/text-track/src/component.ts
new file mode 100644
index 00000000..9601e753
--- /dev/null
+++ b/packages/text-track/src/component.ts
@@ -0,0 +1,43 @@
+import type { ComponentPackage } from "@svml/component-kit";
+import type { ProgramSpace } from "@svml/contracts";
+import type { StoredValue } from "@svml/protocol";
+import { canonicalize } from "@svml/protocol";
+
+import { textTrackProducers } from "./manifest.js";
+import {
+ compileTextTrackImplementationDigest,
+ compileTextTrackProgram,
+ renderTextTrack,
+ renderTextTrackImplementationDigest,
+} from "./program.js";
+import type { TextTrackProgram, TextTrackSpec } from "./types.js";
+
+function inline<T>(value: StoredValue | undefined, subject: string): T {
+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);
+ return value.value as unknown as T;
+}
+
+export const textTrackComponent = {
+ name: "@svml/text-track",
+ producers: [{
+ producer: textTrackProducers.compile,
+ implementationDigest: compileTextTrackImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { program: { kind: "inline", value: canonicalize(compileTextTrackProgram(
+ inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),
+ inline<TextTrackSpec>(inputs.spec?.value, "TextTrackSpec"),
+ )) } },
+ needs: {},
+ }),
+ }, {
+ producer: textTrackProducers.render,
+ implementationDigest: renderTextTrackImplementationDigest,
+ handler: ({ inputs }) => ({
+ outputs: { track: { kind: "inline", value: canonicalize(renderTextTrack(
+ inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),
+ inline<TextTrackProgram>(inputs.program?.value, "TextTrackProgram"),
+ )) } },
+ needs: {},
+ }),
+ }],
+} satisfies ComponentPackage;
diff --git a/packages/text-track/src/index.ts b/packages/text-track/src/index.ts
index 9bec2c0a..56184210 100644
--- a/packages/text-track/src/index.ts
+++ b/packages/text-track/src/index.ts
@@ -1,17 +1,25 @@
export { textTrackFragment } from "./fragment.js";
+export { textTrackComponent } from "./component.js";
export {
textTrackManifest,
textTrackManifestDigest,
textTrackModuleRef,
textTrackProducers,
textTrackProgramSchema,
+ textTrackSpecSchema,
+ textTrackSurfaceImplementationDigest,
textTrackTypes,
} from "./manifest.js";
export {
assertTextTrackProgramIdentity,
+ assertTextTrackSpec,
+ compileTextTrackImplementationDigest,
+ compileTextTrackProgram,
computeTextTrackProgramDigest,
renderTextTrack,
renderTextTrackImplementationDigest,
sealTextTrackProgram,
+ sealTextTrackSpec,
} from "./program.js";
+export { decodeTextTrackSurface } from "./surface.js";
export type * from "./types.js";
diff --git a/packages/text-track/src/manifest.ts b/packages/text-track/src/manifest.ts
index 7a394af8..6b51830d 100644
--- a/packages/text-track/src/manifest.ts
+++ b/packages/text-track/src/manifest.ts
@@ -6,13 +6,15 @@ import {
import { digestOf } from "@svml/protocol";
import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";
-import { renderTextTrackImplementationDigest } from "./program.js";
+import { compileTextTrackImplementationDigest, renderTextTrackImplementationDigest } from "./program.js";
export const textTrackModuleRef = { name: "@svml/text-track", version: "0.0.0-dev" } as const;
export const textTrackTypes = {
program: { module: textTrackModuleRef, name: "TextTrackProgram" },
+ spec: { module: textTrackModuleRef, name: "TextTrackSpec" },
} satisfies Record<string, TypeRef>;
export const textTrackProducers = {
+ compile: { module: textTrackModuleRef, name: "compile-text-track" },
render: { module: textTrackModuleRef, name: "render-text-track" },
} satisfies Record<string, ProducerRef>;
@@ -27,6 +29,20 @@ const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema;
fields,
});
+const textAppearanceSchema = object({
+ color: { schema: string },
+ fontSizePx: { schema: positiveNumber },
+ fontFamily: { schema: string, optional: true },
+ fontWeight: { schema: { kind: "number", integer: true, minimum: 1, maximum: 1000 }, optional: true },
+ lineHeight: { schema: positiveNumber, optional: true },
+ align: { schema: { kind: "string", enum: ["left", "center", "right"] }, optional: true },
+ verticalAlign: { schema: { kind: "string", enum: ["top", "center", "bottom"] }, optional: true },
+ backgroundColor: { schema: string, optional: true },
+ borderRadiusPx: { schema: { kind: "number", minimum: 0 }, optional: true },
+ paddingPx: { schema: { kind: "number", minimum: 0 }, optional: true },
+ letterSpacingPx: { schema: number, optional: true },
+});
+
const textItemSchema = object({
id: { schema: string },
text: { schema: string },
@@ -42,18 +58,7 @@ const textItemSchema = object({
widthPercent: { schema: positiveNumber },
heightPercent: { schema: positiveNumber },
}) },
- appearance: { schema: object({
- color: { schema: string },
- fontSizePx: { schema: positiveNumber },
- fontFamily: { schema: string, optional: true },
- fontWeight: { schema: { kind: "number", integer: true, minimum: 1, maximum: 1000 }, optional: true },
- lineHeight: { schema: positiveNumber, optional: true },
- align: { schema: { kind: "string", enum: ["left", "center", "right"] }, optional: true },
- verticalAlign: { schema: { kind: "string", enum: ["top", "center", "bottom"] }, optional: true },
- backgroundColor: { schema: string, optional: true },
- borderRadiusPx: { schema: { kind: "number", minimum: 0 }, optional: true },
- paddingPx: { schema: { kind: "number", minimum: 0 }, optional: true },
- }) },
+ appearance: { schema: textAppearanceSchema },
});
export const textTrackProgramSchema: ValueSchema = object({
@@ -64,6 +69,25 @@ export const textTrackProgramSchema: ValueSchema = object({
items: { schema: { kind: "array", minItems: 1, items: textItemSchema } },
});
+export const textTrackSpecSchema: ValueSchema = object({
+ contract: { schema: { kind: "literal", value: "svml.text-track-spec@1" } },
+ digest: { schema: digest },
+ id: { schema: string },
+ items: { schema: { kind: "array", minItems: 1, items: object({
+ id: { schema: string },
+ text: { schema: string },
+ during: { schema: { kind: "literal", value: "full" } },
+ z: { schema: signedInteger },
+ box: { schema: object({
+ xPercent: { schema: number }, yPercent: { schema: number },
+ widthPercent: { schema: positiveNumber }, heightPercent: { schema: positiveNumber },
+ }) },
+ appearance: { schema: textAppearanceSchema },
+ }) } },
+});
+
+export const textTrackSurfaceImplementationDigest = digestOf("@svml/text-track/track-surface@1");
+
export const textTrackManifest: ModuleManifest = {
format: "svml.module@0",
name: textTrackModuleRef.name,
@@ -72,10 +96,29 @@ export const textTrackManifest: ModuleManifest = {
videoContractDependencies.programSpace,
videoContractDependencies.composition,
],
- types: [{ name: textTrackTypes.program.name, schema: textTrackProgramSchema }],
+ types: [
+ { name: textTrackTypes.program.name, schema: textTrackProgramSchema },
+ { name: textTrackTypes.spec.name, schema: textTrackSpecSchema },
+ ],
capabilities: [],
- surfaces: [],
+ surfaces: [{
+ name: "track", tag: "Track", mode: "structured",
+ outputs: [textTrackTypes.spec, textTrackTypes.program, contractTypes.visualTrack],
+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/text-track/track-surface", digest: textTrackSurfaceImplementationDigest },
+ }],
producers: [{
+ name: textTrackProducers.compile.name,
+ inputs: [
+ { name: "space", type: contractTypes.programSpace },
+ { name: "spec", type: textTrackTypes.spec },
+ ],
+ outputs: [{
+ name: "program", type: textTrackTypes.program,
+ affinity: [{ resultPointer: "/programSpaceDigest", input: "space", inputPointer: "/digest" }],
+ }],
+ needs: [],
+ implementation: { kind: "registered", locator: "@svml/text-track/compile", digest: compileTextTrackImplementationDigest },
+ }, {
name: textTrackProducers.render.name,
inputs: [
{ name: "space", type: contractTypes.programSpace },
diff --git a/packages/text-track/src/program.ts b/packages/text-track/src/program.ts
index dd6ee428..ad73f0af 100644
--- a/packages/text-track/src/program.ts
+++ b/packages/text-track/src/program.ts
@@ -8,9 +8,10 @@ import type { ProgramSpace, VisualStyleDeclaration, VisualTrack } from "@svml/co
import { digestOf, isDigest } from "@svml/protocol";
import type { Digest } from "@svml/protocol";
-import type { TextAppearance, TextItem, TextTrackProgram } from "./types.js";
+import type { TextAppearance, TextItem, TextTrackProgram, TextTrackSpec } from "./types.js";
export const renderTextTrackImplementationDigest = digestOf("@svml/text-track/render@1");
+export const compileTextTrackImplementationDigest = digestOf("@svml/text-track/compile@1");
function assertNonEmpty(value: string, label: string): void {
if (!value.trim()) throw new Error(`${label} must not be empty.`);
@@ -80,6 +81,7 @@ function assertAppearance(appearance: TextAppearance, label: string): void {
}
}
if (appearance.backgroundColor !== undefined) assertNonEmpty(appearance.backgroundColor, `${label} backgroundColor`);
+ if (appearance.letterSpacingPx !== undefined) assertFinite(appearance.letterSpacingPx, `${label} letterSpacingPx`);
}
export function assertTextTrackProgramIdentity(program: TextTrackProgram, programSpace: ProgramSpace): void {
@@ -154,9 +156,68 @@ function textStyle(appearance: TextAppearance): VisualStyleDeclaration[] {
{ name: "line-height", value: appearance.lineHeight ?? 1.2 },
{ name: "text-align", value: appearance.align ?? "center" },
...(appearance.fontFamily === undefined ? [] : [{ name: "font-family", value: appearance.fontFamily }]),
+ ...(appearance.letterSpacingPx === undefined ? [] : [{ name: "letter-spacing", value: `${appearance.letterSpacingPx}px` }]),
];
}
+export function sealTextTrackSpec(value: Omit<TextTrackSpec, "digest">): TextTrackSpec {
+ const content = canonicalSpec(value);
+ return { ...content, digest: digestOf(content) };
+}
+
+function canonicalSpec(value: Omit<TextTrackSpec, "digest">): Omit<TextTrackSpec, "digest"> {
+ return {
+ contract: "svml.text-track-spec@1",
+ id: value.id,
+ items: value.items.map((item) => ({
+ id: item.id,
+ text: item.text,
+ during: "full",
+ z: item.z,
+ box: { ...item.box },
+ appearance: { ...item.appearance },
+ })),
+ };
+}
+
+export function assertTextTrackSpec(spec: TextTrackSpec): void {
+ if (spec.contract !== "svml.text-track-spec@1" || !spec.id || spec.items.length === 0) {
+ throw new Error("TextTrackSpec identity or items are invalid.");
+ }
+ const { digest: _digest, ...content } = spec;
+ if (!isDigest(spec.digest) || spec.digest !== digestOf(canonicalSpec(content))) {
+ throw new Error("TextTrackSpec digest differs from its contents.");
+ }
+ const ids = new Set<string>();
+ for (const item of spec.items) {
+ if (!item.id || !item.text || item.during !== "full" || ids.has(item.id) || !Number.isSafeInteger(item.z)) {
+ throw new Error("TextTrackSpec contains an invalid item.");
+ }
+ ids.add(item.id);
+ for (const value of Object.values(item.box)) assertFinite(value, `${item.id} box`);
+ assertAppearance(item.appearance, `${item.id} appearance`);
+ }
+}
+
+export function compileTextTrackProgram(programSpace: ProgramSpace, spec: TextTrackSpec): TextTrackProgram {
+ assertProgramSpaceIdentity(programSpace);
+ assertTextTrackSpec(spec);
+ return sealTextTrackProgram({
+ contract: "svml.text-track-program@1",
+ id: spec.id,
+ programSpaceDigest: programSpace.digest,
+ items: spec.items.map((item) => ({
+ id: item.id,
+ text: item.text,
+ span: { startFrame: 0, endFrameExclusive: programSpaceFrameCount(programSpace) },
+ z: item.z,
+ tieBreak: `${spec.id}:${item.id}`,
+ box: item.box,
+ appearance: item.appearance,
+ })),
+ });
+}
+
export function renderTextTrack(programSpace: ProgramSpace, program: TextTrackProgram): VisualTrack {
assertTextTrackProgramIdentity(program, programSpace);
const track = sealVisualTrack({
diff --git a/packages/text-track/src/surface.ts b/packages/text-track/src/surface.ts
new file mode 100644
index 00000000..aa5d2877
--- /dev/null
+++ b/packages/text-track/src/surface.ts
@@ -0,0 +1,105 @@
+import { contractTypes } from "@svml/contracts";
+import { sealGraphFragment } from "@svml/elaborator";
+import { svsRecipeType } from "@svml/svs";
+import type { SvsRecipe } from "@svml/svs";
+import type { StructuredElement, StructuredSurfaceHandler, SurfaceResolvedReference, TextAttributeValue } from "@svml/text";
+
+import { textTrackProducers, textTrackTypes } from "./manifest.js";
+import { sealTextTrackSpec } from "./program.js";
+
+const input = (name: string) => ({ kind: "fragment-input" as const, name });
+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
+
+const textTrackSurfaceFragment = sealGraphFragment({
+ name: "@svml/text-track/track-surface@1",
+ inputs: [
+ { name: "space", type: contractTypes.programSpace },
+ { name: "spec", type: textTrackTypes.spec },
+ ],
+ operations: [
+ { id: "text:compile", producer: textTrackProducers.compile, inputs: { space: input("space"), spec: input("spec") }, result: { kind: "output", name: "program" } },
+ { id: "text:render", producer: textTrackProducers.render, inputs: { space: input("space"), program: operation("text:compile") }, result: { kind: "output", name: "track" } },
+ ],
+ exports: [{
+ name: "track", type: contractTypes.visualTrack, root: operation("text:render"), semanticInputs: ["space", "spec"],
+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("space"), sourcePointer: "/digest" }], fidelity: "exact",
+ }],
+});
+
+function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+}
+function exact(element: StructuredElement, names: readonly string[]): void {
+ if (Object.keys(element.attributes).sort().join("\0") !== [...names].sort().join("\0")) throw new Error(`${element.name} requires exactly ${names.join(", ")}`);
+}
+function text(element: StructuredElement, name: string): string {
+ const value = element.attributes[name];
+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);
+ return value.trim();
+}
+function ref(element: StructuredElement, name: string, expected: SurfaceResolvedReference["type"], resolve: (path: string) => SurfaceResolvedReference | undefined) {
+ const raw: TextAttributeValue | undefined = element.attributes[name];
+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a reference`);
+ const value = resolve(raw.path);
+ if (!value || !sameType(value.type, expected)) throw new Error(`${element.name}.${name} cannot resolve a value of the required type`);
+ return value;
+}
+function recipe(value: SurfaceResolvedReference): SvsRecipe {
+ if (!sameType(value.type, svsRecipeType) || value.record?.value.kind !== "inline") throw new Error("Text appearance must be an authored SVS Recipe");
+ return value.record.value.value as unknown as SvsRecipe;
+}
+function propNumber(value: SvsRecipe, name: string): number {
+ const result = value.properties[name];
+ if (typeof result !== "number" || !Number.isFinite(result)) throw new Error(`Text Recipe ${name} must be a number`);
+ return result;
+}
+function propString(value: SvsRecipe, name: string): string {
+ const result = value.properties[name];
+ if (typeof result !== "string" || !result) throw new Error(`Text Recipe ${name} must be a string`);
+ return result;
+}
+
+export const decodeTextTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ exact(element, ["id", "space"]);
+ const id = text(element, "id");
+ const space = ref(element, "space", contractTypes.programSpace, resolveReference);
+ let itemIndex = 0;
+ const items = element.children.flatMap((child) => {
+ if (child.kind === "text") {
+ if (child.value.trim()) throw new Error(`${element.name} accepts only Item children`);
+ return [];
+ }
+ if (!child.name.endsWith(":Item") && child.name !== "Item") throw new Error(`${element.name} accepts only Item children`);
+ itemIndex += 1;
+ exact(child, ["text", "during", "appearance"]);
+ if (text(child, "during") !== "full") throw new Error(`${child.name}.during currently accepts only full`);
+ const appearance = recipe(ref(child, "appearance", svsRecipeType, resolveReference));
+ const expected = ["align", "fill", "font", "height", "size", "stack-order", "tracking", "weight", "width", "x", "y"];
+ if (Object.keys(appearance.properties).sort().join("\0") !== expected.sort().join("\0")) throw new Error(`Text Recipe requires exactly ${expected.join(", ")}`);
+ const align = propString(appearance, "align");
+ if (align !== "left" && align !== "center" && align !== "right") throw new Error("Text Recipe align is invalid");
+ const textAlign = align as "left" | "center" | "right";
+ return [{
+ id: `item-${String(itemIndex).padStart(4, "0")}`,
+ text: text(child, "text"), during: "full" as const,
+ z: propNumber(appearance, "stack-order"),
+ box: {
+ xPercent: propNumber(appearance, "x") * 100, yPercent: propNumber(appearance, "y") * 100,
+ widthPercent: propNumber(appearance, "width") * 100, heightPercent: propNumber(appearance, "height") * 100,
+ },
+ appearance: {
+ color: propString(appearance, "fill"), fontSizePx: propNumber(appearance, "size"),
+ fontFamily: propString(appearance, "font"), fontWeight: propNumber(appearance, "weight"),
+ letterSpacingPx: propNumber(appearance, "tracking"), align: textAlign,
+ },
+ }];
+ });
+ if (items.length === 0) throw new Error(`${element.name} requires at least one Item`);
+ const spec = sealTextTrackSpec({ contract: "svml.text-track-spec@1", id, items });
+ const specId = `${id}.spec`;
+ return {
+ records: [{ id: specId, type: textTrackTypes.spec, value: { kind: "inline", value: spec }, range: element.range }],
+ components: [{ id, fragment: textTrackSurfaceFragment.id, inputs: { space: space.ref, spec: { kind: "record", id: specId } }, outputs: { track: `${id}.track` }, range: element.range }],
+ fragments: [textTrackSurfaceFragment],
+ };
+};
diff --git a/packages/text-track/src/types.ts b/packages/text-track/src/types.ts
index b90e9fe0..56f17a81 100644
--- a/packages/text-track/src/types.ts
+++ b/packages/text-track/src/types.ts
@@ -19,6 +19,7 @@ export type TextAppearance = {
readonly backgroundColor?: string;
readonly borderRadiusPx?: number;
readonly paddingPx?: number;
+ readonly letterSpacingPx?: number;
};
export type TextItem = {
@@ -39,3 +40,17 @@ export type TextTrackProgram = {
readonly programSpaceDigest: Digest;
readonly items: readonly TextItem[];
};
+
+export type TextTrackSpec = {
+ readonly contract: "svml.text-track-spec@1";
+ readonly digest: Digest;
+ readonly id: string;
+ readonly items: readonly {
+ readonly id: string;
+ readonly text: string;
+ readonly during: "full";
+ readonly z: number;
+ readonly box: TextBox;
+ readonly appearance: TextAppearance;
+ }[];
+};
diff --git a/packages/text/test/text.test.ts b/packages/text/test/text.test.ts
index 6ffcb2ff..07e0ed3f 100644
--- a/packages/text/test/text.test.ts
+++ b/packages/text/test/text.test.ts
@@ -51,11 +51,17 @@ test("Text learns <script> only from an imported Script Manifest", async () => {
scriptContext(),
);
- assert.equal(result.module.records.length, 2);
+ assert.equal(result.module.records.length, 5);
assert.equal(result.module.records[0]?.id, "story");
assert.equal(result.module.records[0]?.type.name, "Narrative");
- assert.equal(result.module.records[1]?.id, "story.segment.opening");
- assert.equal(result.module.records[1]?.type.name, "NarrativeExcerpt");
+ assert.equal(result.module.records.some((record) =>
+ record.id === "story.segment.opening" && record.type.name === "NarrativeExcerpt"), true);
+ assert.equal(result.module.records.some((record) =>
+ record.id === "story.segment.opening.dialogue" && record.type.name === "NarrativeDialogueExcerpt"), true);
+ assert.equal(result.module.records.some((record) =>
+ record.id === "story.segment.opening.speech" && record.type.name === "NarrativeSpeechExcerpt"), true);
+ assert.equal(result.module.records.some((record) =>
+ record.id === "story.caption" && record.type.name === "CaptionProjection"), true);
assert.equal(result.sourceMaps.length, 1);
assert.doesNotThrow(() => link(scriptContext().closure, [result.module]));
});
diff --git a/packages/whisperx/package.json b/packages/whisperx/package.json
index 25a68b69..1fd28c31 100644
--- a/packages/whisperx/package.json
+++ b/packages/whisperx/package.json
@@ -9,6 +9,10 @@
"dependencies": {
"@svml/component-kit": "workspace:*",
"@svml/contracts": "workspace:*",
- "@svml/protocol": "workspace:*"
+ "@svml/elaborator": "workspace:*",
+ "@svml/media-pipeline": "workspace:*",
+ "@svml/protocol": "workspace:*",
+ "@svml/speech-align": "workspace:*",
+ "@svml/text": "workspace:*"
}
}
diff --git a/packages/whisperx/src/fragment.ts b/packages/whisperx/src/fragment.ts
new file mode 100644
index 00000000..a32bb9fa
--- /dev/null
+++ b/packages/whisperx/src/fragment.ts
@@ -0,0 +1,76 @@
+import { contractTypes } from "@svml/contracts";
+import { sealGraphFragment } from "@svml/elaborator";
+import { mediaPipelineProducers } from "@svml/media-pipeline";
+import { speechAlignProducers } from "@svml/speech-align";
+
+import { whisperXProducers, whisperXTypes } from "./manifest.js";
+
+const input = (name: string) => ({ kind: "fragment-input" as const, name });
+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });
+
+/** One measured acoustic pass followed by provider-neutral deterministic Script alignment. */
+export const whisperXSpeechAlignmentFragment = sealGraphFragment({
+ name: "@svml/whisperx/speech-alignment@1",
+ inputs: [
+ { name: "narrative", type: contractTypes.narrative },
+ { name: "audio", type: contractTypes.speechAudioBasis },
+ ],
+ operations: [
+ {
+ id: "prepare-evidence-audio",
+ producer: mediaPipelineProducers.projectSpeechEvidenceAudio,
+ inputs: { audio: input("audio") },
+ result: { kind: "need", name: "evidenceAudio", accepts: "exact" },
+ },
+ {
+ id: "request-whisperx",
+ producer: whisperXProducers.request,
+ inputs: { audio: operation("prepare-evidence-audio") },
+ result: { kind: "need", name: "alignment", accepts: "exact" },
+ },
+ {
+ id: "normalize-evidence",
+ producer: whisperXProducers.normalize,
+ inputs: { whisperx: operation("request-whisperx") },
+ result: { kind: "output", name: "evidence" },
+ },
+ {
+ id: "locate-speech",
+ producer: speechAlignProducers.locate,
+ inputs: { narrative: input("narrative"), audio: input("audio"), evidence: operation("normalize-evidence") },
+ result: { kind: "output", name: "map" },
+ },
+ ],
+ exports: [
+ {
+ name: "rawEvidence",
+ type: whisperXTypes.alignmentEvidence,
+ root: operation("request-whisperx"),
+ semanticInputs: ["audio"],
+ affinity: [
+ { resultPointer: "/audioArtifactDigest", source: operation("prepare-evidence-audio"), sourcePointer: "/artifact/digest" },
+ { resultPointer: "/programSpaceDigest", source: input("audio"), sourcePointer: "/programSpace/digest" },
+ ],
+ fidelity: "exact",
+ },
+ {
+ name: "evidence",
+ type: contractTypes.alignedTranscriptEvidence,
+ root: operation("normalize-evidence"),
+ semanticInputs: ["audio"],
+ affinity: [
+ { resultPointer: "/audioArtifactDigest", source: operation("prepare-evidence-audio"), sourcePointer: "/artifact/digest" },
+ { resultPointer: "/programSpaceDigest", source: input("audio"), sourcePointer: "/programSpace/digest" },
+ ],
+ fidelity: "exact",
+ },
+ {
+ name: "map",
+ type: contractTypes.completeSemanticMap,
+ root: operation("locate-speech"),
+ semanticInputs: ["narrative", "audio"],
+ affinity: [{ resultPointer: "/programSpace/digest", source: input("audio"), sourcePointer: "/programSpace/digest" }],
+ fidelity: "exact",
+ },
+ ],
+});
diff --git a/packages/whisperx/src/index.ts b/packages/whisperx/src/index.ts
index 6f466be7..8c0fa497 100644
--- a/packages/whisperx/src/index.ts
+++ b/packages/whisperx/src/index.ts
@@ -1,6 +1,7 @@
export {
whisperXComponent,
} from "./component.js";
+export { whisperXSpeechAlignmentFragment } from "./fragment.js";
export {
normalizeWhisperXAlignment,
sealWhisperXAlignmentEvidence,
@@ -15,4 +16,5 @@ export {
whisperXProducers,
whisperXTypes,
} from "./manifest.js";
+export { decodeWhisperXAlignmentSurface } from "./surface.js";
export type * from "./types.js";
diff --git a/packages/whisperx/src/manifest.ts b/packages/whisperx/src/manifest.ts
index 1b43b090..6954f479 100644
--- a/packages/whisperx/src/manifest.ts
+++ b/packages/whisperx/src/manifest.ts
@@ -5,6 +5,8 @@ import {
} from "@svml/contracts";
import { digestOf } from "@svml/protocol";
import type { CapabilityRef, ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";
+import { mediaPipelineManifest, mediaPipelineModuleRef } from "@svml/media-pipeline";
+import { speechAlignManifest, speechAlignModuleRef } from "@svml/speech-align";
export const whisperXModuleRef = { name: "@svml/whisperx", version: "0.0.0-dev" } as const;
export const whisperXTypes = {
@@ -20,6 +22,7 @@ export const whisperXProducers = {
export const whisperXImplementationDigests = {
request: digestOf("@svml/whisperx/request@2"),
normalize: digestOf("@svml/whisperx/normalize@2"),
+ surface: digestOf("@svml/whisperx/alignment-surface@1"),
};
const { contract: _commonContract, evidenceDigest: _commonDigest, ...sharedEvidenceFields } =
@@ -41,13 +44,25 @@ export const whisperXManifest: ModuleManifest = {
dependencies: [
videoContractDependencies.speech,
videoContractDependencies.semanticTime,
+ { module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },
+ { module: speechAlignModuleRef, digest: digestOf(speechAlignManifest) },
],
types: [{ name: whisperXTypes.alignmentEvidence.name, schema: whisperXAlignmentEvidenceSchema }],
capabilities: [{
name: whisperXCapabilities.alignment.name,
returns: whisperXTypes.alignmentEvidence,
}],
- surfaces: [],
+ surfaces: [{
+ name: "alignment",
+ tag: "Alignment",
+ mode: "structured",
+ outputs: [whisperXTypes.alignmentEvidence, contractTypes.alignedTranscriptEvidence, contractTypes.completeSemanticMap],
+ implementation: {
+ kind: "trusted-frontend-surface",
+ locator: "@svml/whisperx/alignment-surface",
+ digest: whisperXImplementationDigests.surface,
+ },
+ }],
producers: [
{
name: whisperXProducers.request.name,
diff --git a/packages/whisperx/src/surface.ts b/packages/whisperx/src/surface.ts
new file mode 100644
index 00000000..754d1adb
--- /dev/null
+++ b/packages/whisperx/src/surface.ts
@@ -0,0 +1,64 @@
+import { contractTypes } from "@svml/contracts";
+import type {
+ StructuredElement,
+ StructuredSurfaceHandler,
+ SurfaceResolvedReference,
+ TextAttributeValue,
+} from "@svml/text";
+
+import { whisperXSpeechAlignmentFragment } from "./fragment.js";
+
+function exactAttributes(element: StructuredElement, names: readonly string[]): void {
+ if (Object.keys(element.attributes).sort().join("\u0000") !== [...names].sort().join("\u0000")) {
+ throw new Error(`${element.name} requires exactly ${names.join(", ")}`);
+ }
+}
+
+function stringAttribute(element: StructuredElement, name: string): string {
+ const value = element.attributes[name];
+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);
+ return value.trim();
+}
+
+function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {
+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;
+}
+
+function reference(
+ element: StructuredElement,
+ name: string,
+ expected: SurfaceResolvedReference["type"],
+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,
+): SurfaceResolvedReference {
+ const raw: TextAttributeValue | undefined = element.attributes[name];
+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a whole-value reference`);
+ const value = resolveReference(raw.path);
+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${raw.path}`);
+ if (!sameType(value.type, expected)) throw new Error(`${element.name}.${name} has the wrong type`);
+ return value;
+}
+
+export const decodeWhisperXAlignmentSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {
+ exactAttributes(element, ["id", "narrative", "audio"]);
+ if (element.children.some((child) => child.kind === "element" || child.value.trim())) {
+ throw new Error(`${element.name} does not accept children`);
+ }
+ const id = stringAttribute(element, "id");
+ const narrative = reference(element, "narrative", contractTypes.narrative, resolveReference);
+ const audio = reference(element, "audio", contractTypes.speechAudioBasis, resolveReference);
+ return {
+ records: [],
+ components: [{
+ id,
+ fragment: whisperXSpeechAlignmentFragment.id,
+ inputs: { narrative: narrative.ref, audio: audio.ref },
+ outputs: {
+ rawEvidence: `${id}.rawEvidence`,
+ evidence: `${id}.evidence`,
+ map: `${id}.map`,
+ },
+ range: element.range,
+ }],
+ fragments: [whisperXSpeechAlignmentFragment],
+ };
+};
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 3003d89a..c07fda22 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -45,12 +45,27 @@ importers:
packages/broll:
dependencies:
+ '@svml/component-kit':
+ specifier: workspace:*
+ version: link:../component-kit
'@svml/contracts':
specifier: workspace:*
version: link:../contracts
+ '@svml/elaborator':
+ specifier: workspace:*
+ version: link:../elaborator
+ '@svml/media-pipeline':
+ specifier: workspace:*
+ version: link:../media-pipeline
'@svml/protocol':
specifier: workspace:*
version: link:../protocol
+ '@svml/svs':
+ specifier: workspace:*
+ version: link:../svs
+ '@svml/text':
+ specifier: workspace:*
+ version: link:../text
devDependencies:
'@svml/hyperframes':
specifier: workspace:*
@@ -64,9 +79,18 @@ importers:
'@svml/contracts':
specifier: workspace:*
version: link:../contracts
+ '@svml/elaborator':
+ specifier: workspace:*
+ version: link:../elaborator
'@svml/protocol':
specifier: workspace:*
version: link:../protocol
+ '@svml/svs':
+ specifier: workspace:*
+ version: link:../svs
+ '@svml/text':
+ specifier: workspace:*
+ version: link:../text
devDependencies:
'@svml/script':
specifier: workspace:*
@@ -75,6 +99,31 @@ importers:
specifier: workspace:*
version: link:../speech-align
+ packages/caption-gemini:
+ dependencies:
+ '@svml/caption':
+ specifier: workspace:*
+ version: link:../caption
+ '@svml/component-kit':
+ specifier: workspace:*
+ version: link:../component-kit
+ '@svml/contracts':
+ specifier: workspace:*
+ version: link:../contracts
+ '@svml/elaborator':
+ specifier: workspace:*
+ version: link:../elaborator
+ '@svml/protocol':
+ specifier: workspace:*
+ version: link:../protocol
+ '@svml/text':
+ specifier: workspace:*
+ version: link:../text
+ devDependencies:
+ '@svml/script':
+ specifier: workspace:*
+ version: link:../script
+
packages/cli:
dependencies:
'@svml/compiler-node':
@@ -89,6 +138,9 @@ importers:
'@svml/prelude-video':
specifier: workspace:*
version: link:../prelude-video
+ '@svml/protocol':
+ specifier: workspace:*
+ version: link:../protocol
packages/compiler-node:
dependencies:
@@ -216,6 +268,9 @@ importers:
packages/film:
dependencies:
+ '@svml/component-kit':
+ specifier: workspace:*
+ version: link:../component-kit
'@svml/contracts':
specifier: workspace:*
version: link:../contracts
@@ -232,9 +287,6 @@ importers:
specifier: workspace:*
version: link:../text
devDependencies:
- '@svml/component-kit':
- specifier: workspace:*
- version: link:../component-kit
'@svml/core':
specifier: workspace:*
version: link:../core
@@ -500,9 +552,15 @@ importers:
packages/prelude-video:
dependencies:
+ '@svml/broll':
+ specifier: workspace:*
+ version: link:../broll
'@svml/caption':
specifier: workspace:*
version: link:../caption
+ '@svml/caption-gemini':
+ specifier: workspace:*
+ version: link:../caption-gemini
'@svml/contracts':
specifier: workspace:*
version: link:../contracts
@@ -533,12 +591,21 @@ importers:
'@svml/speech-align':
specifier: workspace:*
version: link:../speech-align
+ '@svml/speech-program':
+ specifier: workspace:*
+ version: link:../speech-program
'@svml/speech-take':
specifier: workspace:*
version: link:../speech-take
'@svml/svs':
specifier: workspace:*
version: link:../svs
+ '@svml/text-track':
+ specifier: workspace:*
+ version: link:../text-track
+ '@svml/whisperx':
+ specifier: workspace:*
+ version: link:../whisperx
devDependencies:
'@svml/package-loader-node':
specifier: workspace:*
@@ -546,6 +613,34 @@ importers:
packages/protocol: {}
+ packages/provider-google-vertex:
+ dependencies:
+ '@google/genai':
+ specifier: 1.52.0
+ version: 1.52.0
+ '@svml/caption':
+ specifier: workspace:*
+ version: link:../caption
+ '@svml/caption-gemini':
+ specifier: workspace:*
+ version: link:../caption-gemini
+ '@svml/endpoint-kit':
+ specifier: workspace:*
+ version: link:../endpoint-kit
+ '@svml/protocol':
+ specifier: workspace:*
+ version: link:../protocol
+ '@svml/runtime':
+ specifier: workspace:*
+ version: link:../runtime
+ devDependencies:
+ '@svml/driver-node':
+ specifier: workspace:*
+ version: link:../driver-node
+ '@svml/script':
+ specifier: workspace:*
+ version: link:../script
+
packages/provider-hyperframes-local:
dependencies:
'@svml/contracts':
@@ -779,6 +874,9 @@ importers:
'@svml/caption':
specifier: workspace:*
version: link:../caption
+ '@svml/component-kit':
+ specifier: workspace:*
+ version: link:../component-kit
'@svml/contracts':
specifier: workspace:*
version: link:../contracts
@@ -788,28 +886,28 @@ importers:
'@svml/media-pipeline':
specifier: workspace:*
version: link:../media-pipeline
+ '@svml/protocol':
+ specifier: workspace:*
+ version: link:../protocol
'@svml/speech-align':
specifier: workspace:*
version: link:../speech-align
'@svml/speech-take':
specifier: workspace:*
version: link:../speech-take
+ '@svml/text':
+ specifier: workspace:*
+ version: link:../text
'@svml/whisperx':
specifier: workspace:*
version: link:../whisperx
devDependencies:
- '@svml/component-kit':
- specifier: workspace:*
- version: link:../component-kit
'@svml/core':
specifier: workspace:*
version: link:../core
'@svml/driver-node':
specifier: workspace:*
version: link:../driver-node
- '@svml/protocol':
- specifier: workspace:*
- version: link:../protocol
'@svml/realization':
specifier: workspace:*
version: link:../realization
@@ -891,6 +989,9 @@ importers:
packages/text-track:
dependencies:
+ '@svml/component-kit':
+ specifier: workspace:*
+ version: link:../component-kit
'@svml/contracts':
specifier: workspace:*
version: link:../contracts
@@ -900,6 +1001,12 @@ importers:
'@svml/protocol':
specifier: workspace:*
version: link:../protocol
+ '@svml/svs':
+ specifier: workspace:*
+ version: link:../svs
+ '@svml/text':
+ specifier: workspace:*
+ version: link:../text
devDependencies:
'@svml/hyperframes':
specifier: workspace:*
@@ -949,9 +1056,21 @@ importers:
'@svml/contracts':
specifier: workspace:*
version: link:../contracts
+ '@svml/elaborator':
+ specifier: workspace:*
+ version: link:../elaborator
+ '@svml/media-pipeline':
+ specifier: workspace:*
+ version: link:../media-pipeline
'@svml/protocol':
specifier: workspace:*
version: link:../protocol
+ '@svml/speech-align':
+ specifier: workspace:*
+ version: link:../speech-align
+ '@svml/text':
+ specifier: workspace:*
+ version: link:../text
packages/workspace-fs-node:
dependencies:
@@ -2440,7 +2559,6 @@ snapshots:
- bufferutil
- supports-color
- utf-8-validate
- optional: true
dependencies:
@@ -2556,34 +2674,25 @@ snapshots:
dependencies:
minipass: 7.1.3
- '@protobufjs/[email protected]':
- optional: true
+ '@protobufjs/[email protected]': {}
- '@protobufjs/[email protected]':
- optional: true
+ '@protobufjs/[email protected]': {}
- '@protobufjs/[email protected]':
- optional: true
+ '@protobufjs/[email protected]': {}
- '@protobufjs/[email protected]':
- optional: true
+ '@protobufjs/[email protected]': {}
'@protobufjs/[email protected]':
dependencies:
'@protobufjs/aspromise': 1.1.2
- optional: true
- '@protobufjs/[email protected]':
- optional: true
+ '@protobufjs/[email protected]': {}
- '@protobufjs/[email protected]':
- optional: true
+ '@protobufjs/[email protected]': {}
- '@protobufjs/[email protected]':
- optional: true
+ '@protobufjs/[email protected]': {}
- '@protobufjs/[email protected]':
- optional: true
+ '@protobufjs/[email protected]': {}
'@puppeteer/[email protected]':
dependencies:
@@ -2631,13 +2740,11 @@ snapshots:
dependencies:
undici-types: 7.16.0
- '@types/[email protected]':
- optional: true
+ '@types/[email protected]': {}
- optional: true
@@ -2645,8 +2752,7 @@ snapshots:
- optional: true
@@ -2656,8 +2762,7 @@ snapshots:
dependencies:
base64-js: 1.5.1
- optional: true
dependencies:
@@ -2683,8 +2788,7 @@ snapshots:
- optional: true
dependencies:
@@ -2722,7 +2826,6 @@ snapshots:
dependencies:
safe-buffer: 5.2.1
- optional: true
@@ -2794,8 +2897,7 @@ snapshots:
- optional: true
@@ -2803,7 +2905,6 @@ snapshots:
dependencies:
node-domexception: 1.0.0
web-streams-polyfill: 3.3.3
- optional: true
dependencies:
@@ -2820,7 +2921,6 @@ snapshots:
dependencies:
fetch-blob: 3.2.0
- optional: true
optional: true
@@ -2832,7 +2932,6 @@ snapshots:
node-fetch: 3.3.2
transitivePeerDependencies:
- supports-color
- optional: true
dependencies:
@@ -2841,7 +2940,6 @@ snapshots:
json-bigint: 1.0.0
transitivePeerDependencies:
- supports-color
- optional: true
@@ -2877,10 +2975,8 @@ snapshots:
jws: 4.0.1
transitivePeerDependencies:
- supports-color
- optional: true
- optional: true
@@ -2896,7 +2992,6 @@ snapshots:
debug: 4.4.3
transitivePeerDependencies:
- supports-color
- optional: true
dependencies:
@@ -2943,7 +3038,6 @@ snapshots:
dependencies:
bignumber.js: 9.3.1
- optional: true
@@ -2952,16 +3046,13 @@ snapshots:
buffer-equal-constant-time: 1.0.1
ecdsa-sig-formatter: 1.0.11
safe-buffer: 5.2.1
- optional: true
dependencies:
jwa: 2.0.1
safe-buffer: 5.2.1
- optional: true
- optional: true
dependencies:
@@ -2981,15 +3072,13 @@ snapshots:
- optional: true
dependencies:
data-uri-to-buffer: 4.0.1
fetch-blob: 3.2.0
formdata-polyfill: 4.0.10
- optional: true
@@ -3012,7 +3101,6 @@ snapshots:
dependencies:
'@types/retry': 0.12.0
retry: 0.13.1
- optional: true
@@ -3039,7 +3127,6 @@ snapshots:
'@protobufjs/utf8': 1.1.2
'@types/node': 24.10.1
long: 5.3.2
- optional: true
dependencies:
@@ -3059,8 +3146,7 @@ snapshots:
- optional: true
dependencies:
@@ -3073,8 +3159,7 @@ snapshots:
- optional: true
@@ -3173,8 +3258,7 @@ snapshots:
pako: 0.2.9
tiny-inflate: 1.0.3
- optional: true
diff --git a/spec/caption-program-v1.md b/spec/caption-program-v1.md
new file mode 100644
index 00000000..4d8a95d8
--- /dev/null
+++ b/spec/caption-program-v1.md
@@ -0,0 +1,133 @@
+# Caption Program v1
+
+Status: implemented pre-release contract. Public names may still be polished before package
+publication; the semantic laws below are executable.
+
+## 1. Scope
+
+Caption is an ordinary video component family. It owns the transformation from immutable authored
+display text to one self-contained `VisualTrack`. It has no authority over Script wording,
+pronunciation, speech alignment, global time, Composition order, Provider selection or Runtime
+reuse policy.
+
+The graph has two independent branches:
+
+```text
+Narrative.captionProjection + CaptionProgram ── planner ─────── CaptionPlan
+speech audio ─────────────────────────────────── evidence/map ─ CompleteSemanticMap
+
+Narrative + Program + Plan + Map ── temporalize ── TimedCaptionProjection
+TimedCaptionProjection + Program ── render ─────── VisualTrack
+```
+
+The branches meet only at temporalization. Planning contains no timestamps, and alignment contains
+no Caption Style or Cue judgment.
+
+## 2. Display truth
+
+Caption planning sees only the visible left side of Script. For Dual Text:
+
+```svml
+<SVML | semantic video markup language>
+```
+
+Caption sees `SVML`; speech generation and alignment see `semantic video markup language`. The
+right side must never enter a Caption planning request.
+
+The Caption package deterministically tokenizes every visible `CaptionProjection` region into an
+ordered `CaptionDisplayAtom` universe. Each atom has a stable id, display text, Segment/Turn
+ownership, optional Role and a source speech-token range. An atom's correspondence is either:
+
+- `exact`: its own source speech-token range is proven;
+- `region-envelope`: only the enclosing display region's speech range is proven.
+
+The package does not invent word timing for the second case at this stage.
+
+## 3. Complete Style intent
+
+A `CaptionStyle` is a complete author declaration with two independent payloads:
+
+- `planning`: one Cue instruction and zero or more typed per-atom field declarations;
+- `presentation`: one complete visual appearance, presentation mode and absolute stacking order.
+
+Styles are declared separately from where they apply. A field declaration has a stable id, value
+schema, natural-language instruction and minimum/maximum assignment count per Cue. Boolean, enum
+and bounded numeric values are supported. One atom may receive zero, one or multiple independent
+fields; fields need not be contiguous.
+
+A Style is a whole value. Applying another Style replaces it completely. The language does not
+implicitly merge arbitrary visual properties, Cue rules or fields from two Styles.
+
+## 4. Total assignment and ordered replacement
+
+Every `CaptionProgram` must name one `default` Style. That Style initially owns every visible atom,
+including roleless Turns. The Program then applies zero or more ordered `Use` rules:
+
+```xml
+<caption:Program id="captions" narrative={story} default={plain}>
+ <caption:Use role="ALICE" style={alice}/>
+ <caption:Use on={story.selection.special} style={impact}/>
+</caption:Program>
+```
+
+A `Use` selector is either one exact Role label or one explicit Script Selection. It cannot contain
+both. Rules run in source order; for every atom, the last matching rule wins. A rule that selects no
+visible atom is an error. Authors never spell a complement merely to preserve the default.
+
+Role is optional per Script Turn and resets at every Segment boundary. A roleless Turn therefore
+receives the default unless an explicit Selection replaces it.
+
+After assignment, the Program deterministically creates maximal contiguous runs. A run never
+crosses a Segment or Turn boundary and every atom in it has one Style. Runs are disjoint, ordered and
+partition the complete display universe exactly once.
+
+## 5. Planner contract
+
+A planner consumes resolved runs rather than interpreting Role, Selection or Style precedence. For
+each run it may only:
+
+1. split the atom sequence into one or more ordered Cues;
+2. assign declared fields to atoms inside each Cue.
+
+The result must contain every Program run exactly once. Cues must preserve order and partition the
+run's atoms exactly once. Field ids, atom ids, value schemas and per-Cue cardinalities fail closed.
+There is no response field for text, pronunciation, time or Style.
+
+`@svml/caption-gemini` is one explicit implementation of this contract. It is not part of the
+Caption package or Core. Other packages may implement the same output law without being called
+Gemini, but the author graph must explicitly choose them.
+
+## 6. Timing projection
+
+`CompleteSemanticMap` remains the only global speech timing truth. Exact display atoms reuse their
+source token windows. For a `region-envelope` alias, a Caption renderer may allocate local windows
+inside that region's measured envelope. Such boundaries are labelled `estimated`, remain downstream
+Caption facts and are never written back into the global map.
+
+Multiple Cues inside one alias must receive ordered, non-overlapping local windows. They must not
+each copy the same whole region envelope.
+
+## 7. Rendering and composition
+
+Each timed Cue selects the full presentation payload of its resolved Style. The renderer may expose
+declared field values to its own visual lowering, but cannot sample or mutate another Track. The
+result is exactly one ordinary `VisualTrack` whose Presents carry their own frame spans and absolute
+stacking facts.
+
+Film does not recognize Caption specially. It folds the returned Track alongside Speech, B-roll,
+Text or any third-party Track satisfying the same public contract.
+
+## 8. Explicit non-goals
+
+Caption Program v1 does not define:
+
+- Script correction or alternative wording;
+- speaker diarization or speaker identity;
+- implicit model routing;
+- automatic cache reuse or prompt-based memoization;
+- hidden Runtime fallback;
+- a global Caption layer in Composition;
+- field-specific visual semantics in Core.
+
+Reuse, Existing Values and substitutes are selected by the external BuildRequest/realization
+closure. They do not change Caption data or planner semantics.
diff --git a/spec/script-surface-v1.md b/spec/script-surface-v1.md
index f43a465b..a114eead 100644
--- a/spec/script-surface-v1.md
+++ b/spec/script-surface-v1.md
@@ -536,9 +536,10 @@ temporal marker、Role Cue 或结构标签内部。注释可出现在 atom 之
- 每个端点的左右 affinity 和端点源码位置。
- 每个 MomentSet 的一个或多个有序 point、左右 affinity 和源码位置。
-外部 Program 的 `role="label"` selector 从上述 turn token ranges 派生一个
-SelectionSet;同一 label 的多次 turn 成为同一集合的多个有序 occurrence。
-它是显式查询,不是 Role Cue 的隐式字幕行为,也不建立 speaker entity。
+外部 Program 的 `role="label"` selector 可以从上述 Turn 范围投影自己的选择集合;
+同一 label 的多次 Turn 是多个有序、互不相邻的命中。它是消费组件内部的显式查询,
+不是 Role Cue 的隐式字幕行为,也不建立 speaker entity。当前 Caption Program 直接把
+该查询投影到可见 Display Atom,不伪造一个作者写下的 Selection Record。
SelectionSet/MomentSet 是消费者边界,不是 Segment 的子对象。典型外部
关系是:
@@ -563,6 +564,11 @@ Caption 的多样式、region、cue segmentation、annotation、mute 和 layout
编译错误;这不是 Script 禁止 Selection 重叠。未来 Caption 若支持 overlay,
只扩展 Caption Program,不扩展正文语法。
+当前 Caption Program 使用一个覆盖全部可见词的默认 Style,再按源码顺序应用 Role
+或显式 Selection 的整 Style 替换,最后命中的规则获胜。无 Role 的 Turn 自动保留默认
+Style;修改一个局部区间不要求作者书写它的补集。完整规则见
+[`caption-program-v1.md`](./caption-program-v1.md)。
+
## 9. v1 明确不在 Script 中表达
以下能力不是遗漏的第七种 inline marker: