Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.
TypeScript agentic-aiaiai-agentsai-videocompilerdslffmpeggenerative-aillmmarkup-languagemonorepoplugin-systemprogramming-languagetext-to-videotypescriptvideovideo-automationvideo-clonevideo-editingvideo-generation
feat(v2): close the talking-film author graph
kashorin0a0d2aebf676
diff --git a/README.md b/README.mdindex 261fe2f8..cb691fa3 100644--- a/README.md+++ b/README.md@@ -14,9 +14,10 @@> Fragments with hygienic instance identities;> `packages/realization` attaches locked Existing-Value or Fragment Candidates without changing the> author graph identity. These are executable reference layers outside Core, not new Kernel magic.-> `packages/speech-program` now composes the first real speech vertical from static Fragments:-> SpeechTake projections, one-pass WhisperX evidence, a provider-neutral SemanticMap and timed-> captions. Producer-declared affinity protects internal Fragment results before they enter state;+> `packages/speech-program` now owns the namespaced Speech Spine Surface and composes normalized+> generated Takes into one audio master, one ProgramSpace and peer Tracks. `packages/whisperx`+> owns the explicit alignment Surface; provider-neutral SemanticMap and Caption timing remain+> separately auditable Operations. Producer-declared affinity protects internal Fragment results before they enter state;> Logical-Output affinity separately protects selected Candidates.> `packages/broll` lowers provider-neutral B-roll Programs into independently stacked visual> Presents and ordinary source-audio/SFX tracks. Local pop/fade/slide motion and push/page-turn@@ -39,7 +40,7 @@> distribution implement the trusted local path. Hosted adapters remain Runtime packages outside> this slice. `packages/elaborator` now compiles recursive Source Closures,> while `packages/svs` implements a deliberately generic record-only Recipe Frontend; package-owned-> video Recipe validation remains undecided. `packages/compiler-node` now supplies the domain-free+> video Recipe validation is package-owned rather than built into SVS. `packages/compiler-node` now supplies the domain-free> registered-manifest resolver, injected Workspace compiler and file-to-Graph/> BuildPlan facade; `packages/cli` assembles the first trusted Text + Script + SVS + Film + Render> command-line prelude. `packages/validation` now supplies package-owned semantic Type admission: authored,@@ -53,8 +54,10 @@> author packages can now be activated from a byte-locked physical dependency closure without> changing Core or the official CLI; automatic installation, untrusted parser/validator sandboxing,> distributed leases and hosted WhisperX /-> HyperFrames Endpoint adapters are not implemented yet. The first KIE generation Provider and its-> seven explicit model families are implemented. `@svml/store-sqlite`, `@svml/artifact-store-fs` and+> HyperFrames Endpoint adapters are not implemented yet; the local WhisperX and HyperFrames+> Providers are. The KIE generation Provider and its seven explicit model families are implemented,+> as are display-only Gemini Caption planning and its Google Vertex Provider.+> `@svml/store-sqlite`, `@svml/artifact-store-fs` and> `@svml/local` now provide the durable zero-service developer assembly.> `@svml/endpoint-kit` and `@svml/credential-store-env` now bind static Endpoint identity,> non-secret configuration, exact credential slots and recoverable wake/retry/cancel behavior;@@ -66,7 +69,7 @@> `@svml/media-pipeline` and `@svml/provider-media-local` now provide all-stream ffprobe inspection,> attached-picture-safe stream selection and shared-origin A/V normalization. Embedded AAC remains> an ordinary media fact until an explicit Narrative-bound speech component promotes it;-> the v2 CLI exposes local follow, status and cancel without adding another authoritative queue. The existing+> the v2 CLI exposes local follow, status, cancel and content-addressed final Artifact export without adding another authoritative queue. The existing> root compiler and standard library remain the executable v1 research oracle during migration.`@svml/component-kit` is the narrow compute-host port: deterministic model, media, WhisperX and@@ -229,9 +232,11 @@ generation provider is invoked by tests or compilation.[`examples/talking-film-golden`](examples/talking-film-golden/README.md) preserves the implementedScript Surface and specifies the complete namespaced author experience for two Seedance Mini speechtakes, Seedance B-roll, explicit WhisperX alignment, two speaker caption styles, editorial text,-peer Tracks, Film and Hyperframes rendering. Its README marks every implemented and missing lowering;-Script, authored Image, Seedance generation, Film and Render are accepted by the current compiler,-while Speech assembly and Track Surfaces remain the next target.+peer Tracks, Film and Hyperframes rendering. Provider-free Speech, WhisperX, Caption, B-roll and+Text Surfaces now compile. Display-only Gemini planning, the ordered Caption Style cascade and the+Google Vertex Provider are also implemented;+[`examples/talking-film-graph-check`](examples/talking-film-graph-check/README.md) is the fully+checkable graph counterpart without production assets or paid execution.## Specification and implementation recorddiff --git a/docs/README.md b/docs/README.mdindex 243985da..73c2b28b 100644--- a/docs/README.md+++ b/docs/README.md@@ -24,33 +24,37 @@ inside a historical design record does not make that record normative.the current Track is a contract candidate rather than a public compatibility promise.9. [`../spec/hyperframes-visual-ir-v1.md`](../spec/hyperframes-visual-ir-v1.md) — the one versioned,closed terminal visual language shared by video components; not a Core or author-component API.-10. [`../examples/talking-film-golden/README.md`](../examples/talking-film-golden/README.md) — the- canonical v2 authoring target: implemented Script, authored Image, Seedance generation, Film,- SVS and render syntax surrounded by Speech assembly, WhisperX, Caption, B-roll and Text Surfaces- that remain to be completed.-11. [`package-vocabulary-and-ownership-v1.md`](./package-vocabulary-and-ownership-v1.md) — target+10. [`../spec/caption-program-v1.md`](../spec/caption-program-v1.md) — implemented total default+ Style, ordered Role/Selection replacement, display-atom planner and timing-join laws.+11. [`../examples/talking-film-golden/README.md`](../examples/talking-film-golden/README.md) — the+ canonical two-speaker v2 authoring target. Its Script projections, Caption Style cascade,+ display-only Gemini planning and graph lowering are implemented; real assets and credentials+ remain deployment inputs.+12. [`package-vocabulary-and-ownership-v1.md`](./package-vocabulary-and-ownership-v1.md) — targetnaming and ownership for framework, video, Provider, Runtime, queues, stores and distributions.-12. [`../spec/author-surface-binding-v1.md`](../spec/author-surface-binding-v1.md) — implemented+13. [`../spec/author-surface-binding-v1.md`](../spec/author-surface-binding-v1.md) — implementedpackage-owned binding from generic imported Records such as SVS Recipes into typed Programs,AuthorComponents and Graph Fragments.-13. [`local-developer-runtime-v1.md`](./local-developer-runtime-v1.md) — implemented local+14. [`local-developer-runtime-v1.md`](./local-developer-runtime-v1.md) — implemented localSQLite/filesystem Runtime assembly, trusted configuration, recovery law and environment swaps.-14. [`kie-generation-modules-v1.md`](./kie-generation-modules-v1.md) — implemented seven exact+15. [`kie-generation-modules-v1.md`](./kie-generation-modules-v1.md) — implemented seven exactgeneration model modules, shared generated-media Products and the recoverable KIE Provider.-15. [`kie-live-smoke-2026-08-06.md`](./kie-live-smoke-2026-08-06.md) — credential-free deployment+16. [`kie-live-smoke-2026-08-06.md`](./kie-live-smoke-2026-08-06.md) — credential-free deploymentevidence from seven representative live model calls and one synthetic-reference upload.-16. [`media-inspection-and-normalization-v1.md`](./media-inspection-and-normalization-v1.md) —+17. [`media-inspection-and-normalization-v1.md`](./media-inspection-and-normalization-v1.md) —implemented all-stream inspection, attached-picture-safe selection, shared-origin A/Vnormalization and the boundary that prevents embedded AAC from becoming speech by inference.-17. [`media-execution-boundary-v1.md`](./media-execution-boundary-v1.md) — the implemented rule that+18. [`media-execution-boundary-v1.md`](./media-execution-boundary-v1.md) — the implemented rule thatplanning is ordinary code while probe/decode/mix/render/mux are explicit Provider Needs, plusthe visual/audio/mux split for final HyperFrames output, canonical WhisperX evidence-audio pathand implemented local Providers.-18. [`../services/whisperx/README.md`](../services/whisperx/README.md) — installation, runtime+19. [`../services/whisperx/README.md`](../services/whisperx/README.md) — installation, runtimeidentity, path confinement and queue boundary for the implemented warm local WhisperX service.-19. [`node-package-activation-v1.md`](./node-package-activation-v1.md) — implemented trusted author+20. [`node-package-activation-v1.md`](./node-package-activation-v1.md) — implemented trusted authorand deterministic compute facet locking, with the exact boundary that keeps source imports awayfrom Provider and Runtime authority.+21. [`caption-gemini-provider-contract.md`](./caption-gemini-provider-contract.md) — implemented+ display-atom planning boundary, old Vertex execution evidence and no-text-rewrite contract.The executable v2 Node entrypoints are documented in[`../packages/compiler-node/README.md`](../packages/compiler-node/README.md),diff --git a/examples/README.md b/examples/README.mdindex bcd18b9c..21a4a0d2 100644--- a/examples/README.md+++ b/examples/README.md@@ -8,12 +8,14 @@ and the complete v2 authoring target.- [`v2-bootstrap/main.svml`](./v2-bootstrap/main.svml) is accepted by `pnpm svml:v2 check`. Itexercises the real Node Source Host, official Text/Script prelude and recursive SVS Frontend. Itdeliberately stops before package-owned Recipe consumption and component Graph construction.+- [`talking-film-graph-check/main.svml`](./talking-film-graph-check/main.svml) is the complete+ provider-free author graph through Speech, WhisperX, B-roll, Caption, Text, Film and Render.## v2 authoring target- [`talking-film-golden/main.svml`](./talking-film-golden/main.svml) is the canonical visual and- semantic target for the next author-facing Surfaces. Its Script syntax is implemented; its outer- package Surfaces are tracked line by line in the adjacent README and are not executable yet.+ semantic target for two-speaker Gemini captions. Its author and paid Caption packages are+ implemented; the adjacent README tracks real assets and deployment bindings.## Executable v1 research fixturesdiff --git a/examples/talking-film-golden/README.md b/examples/talking-film-golden/README.mdindex 92a10007..fa0d201d 100644--- a/examples/talking-film-golden/README.md+++ b/examples/talking-film-golden/README.md@@ -1,8 +1,8 @@# Talking-film authoring golden fixture-Status: **v2 authoring target, not executable end to end yet.** Script, authored Image, Seedance-generation, Film, generic SVS and the explicit HyperFrames Render Surface are implemented. Speech-assembly and Track components are the concrete package-local work remaining.+Status: **two-speaker production target, not executable end to end with the checked-in fixture.**+Every author Surface and the Gemini planning/Vertex execution package now exists. Real image/font+assets, credentials and a selected local Runtime profile are not checked into this fixture.This example replaces neither the executable v1 fixtures nor their regression value. In particular,`examples/flat-track-launch` remains the capability oracle for selections, moments, B-roll motion,@@ -38,23 +38,21 @@ Frontend named by `using`.## Author data flow```text-Script ────────────────────────────────────────────────┐- ├─ Segment + Prompt + reference -> Seedance Speech ─┤- ├─ Segment + Prompt + reference -> Seedance Speech ─┤- │ ▼- │ Speech Spine- │ ┌──────────┴──────────┐- │ │ │- │ audio basis visual/audio Tracks- │ │- ├──────────────────────────────> WhisperX + speech-align- │ │- │ CompleteSemanticMap- │ ┌───────┴────────┐- │ │ │- │ Caption Track B-roll Track- │- └─ caption display projection+Script dialogue + Prompt + reference ──> two Seedance Speech Needs+ │+ ▼+ Speech Spine+ ┌─────────┴─────────┐+ │ │+ audio basis visual/audio Tracks+ │+ ▼+ WhisperX + align ───────────┐+ CompleteSemanticMap │+ ▼+Script left display + Caption Program ──> Gemini Plan ──> Caption Track++Script selections + generated B-roll ───────────────────> B-roll TrackSeedance B-roll + Text Track + every Track above-> Film Composition@@ -74,15 +72,17 @@ Candidates and Operations.| `media:Image` | `@svml/media` | content-addressed authored Artifact | implemented || `seedance:Prompt` | `@svml/seedance` | package-private immutable direction value | implemented || `seedance:Speech model="mini"` | `@svml/seedance` | explicit Seedance Mini Need plus primary-video projection | implemented; explicit duration remains authored until Speech scheduling exists |-| `speech:Spine` | `@svml/speech` | ordered clips -> one `SpeechBasis`, then ordinary projections | basis and four locked projection facets implemented; author assembly missing |-| `whisperx:Alignment` | `@svml/whisperx` | 48k speech master -> explicit 16k evidence Need -> WhisperX -> provider-neutral `@svml/speech-align` -> Map | Graph Fragment, locked deterministic components, local media projection and local Provider/service implemented; Surface missing |+| `speech:Spine` | `@svml/speech` | ordered clips -> normalized Takes, one `SpeechBasis`, then ordinary projections | Surface, fold, media normalization and projection components implemented |+| `whisperx:Alignment` | `@svml/whisperx` | 48k speech master -> explicit 16k evidence Need -> WhisperX -> provider-neutral `@svml/speech-align` -> Map | Surface, Graph Fragment, deterministic components and local Provider/service implemented || `seedance:Video model="mini"` | `@svml/seedance` | explicit Seedance Mini video Need plus primary-video projection | implemented |-| `broll:Track` | `@svml/broll` | semantic windows + media + recipe -> peer Visual/Audio Tracks | Program/lowering implemented; Surface missing |-| `caption:Track` | `@svml/caption-gemini` | timed projection + Gemini cue grouping + role recipe -> VisualTrack | timing/lowering and owner validators are locked components; complete Recipe audit and Gemini/role Surface missing |-| `text:Track` | `@svml/text-track` | package Program -> VisualTrack | Program/lowering implemented; Surface/exact-font use missing |+| `broll:Track` | `@svml/broll` | semantic windows + normalized media + recipe -> peer Visual/Audio Tracks | Surface and deterministic lowering implemented |+| `caption:Style` / `caption:Program` | `@svml/caption` | default total Style + ordered whole-Style replacement by Role or Selection | implemented |+| `caption-ai:Planner` | `@svml/caption-gemini` | immutable display atoms + per-run requirements -> cue cuts and per-atom fields | implemented; Google Vertex Endpoint implemented separately |+| `caption:Track` | `@svml/caption` | CaptionPlan + independent SemanticMap + complete Styles -> VisualTrack | implemented |+| `text:Track` | `@svml/text-track` | package Spec + ProgramSpace -> VisualTrack | provider-free Surface/lowering implemented; exact-font use remains || `film:Film` | `@svml/film` | finite TrackSet fold -> Composition | Graph Fragment and official Surface implemented || `render:Video` | `@svml/hyperframes-render` | Composition -> silent HyperframesDocument render + explicit program audio + mux -> final video Artifact + Receipt | Surface, Fragment and all local execution Providers implemented |-| `studio.svs` | `@svml/svs` | generic immutable Recipe Records; consuming packages must validate and lower them | parser and recursive source import implemented; video recipe consumers/exact font lowering missing |+| `studio.svs` | `@svml/svs` | generic immutable Recipe Records; consuming packages validate and lower them | parser, imports and current package consumers implemented; exact font assets remain |The remaining work is intentionally package-local. None of these rows requires a new Core video type,a component-family registry in Core, a privileged Film root or another Track kind.@@ -110,5 +110,6 @@ The intended final command is:svml build main.svml --target final.video```-Before the remaining Surfaces and Providers are implemented, tools must report that the complete-file is a design fixture rather than pretending to compile it with the v1 engine.+Until real assets, credentials and a Runtime profile are supplied, tools must report this file as a+production design fixture rather than promise a rendered file. `../talking-film-graph-check` is the+fully checkable graph today.diff --git a/examples/talking-film-golden/main.svml b/examples/talking-film-golden/main.svmlindex 6276c88a..8bc84dae 100644--- a/examples/talking-film-golden/main.svml+++ b/examples/talking-film-golden/main.svml@@ -1,8 +1,8 @@<!--SVML authoring golden fixture.- Script, authored media, Seedance generation, Film and HyperFrames Render- Surfaces are implemented. Speech assembly and Track Surfaces remain the- package-local design target.+ All provider-free author Surfaces are implemented. This golden intentionally+ explicitly names caption-gemini because paid cue planning is author-selected,+ not a Runtime guess.--><svml><import from="@svml/script@1"/>@@ -10,7 +10,8 @@<import as="seedance" from="@svml/seedance@1"/><import as="speech" from="@svml/speech@1"/><import as="whisperx" from="@svml/whisperx@1"/>- <import as="caption" from="@svml/caption-gemini@1"/>+ <import as="caption" from="@svml/caption@1"/>+ <import as="caption-ai" from="@svml/caption-gemini@1"/><import as="broll" from="@svml/broll@1"/><import as="text" from="@svml/text-track@1"/><import as="film" from="@svml/film@1"/>@@ -53,11 +54,41 @@regions assemble into a finished video. No readable UI text and no camera shake.</seedance:Prompt>+ <!-- Complete styles are declared once. Selection and use remain separate. -->+ <caption:Style id="dialogue-caption" appearance={studio.caption.dialogue}>+ <caption:Cues>Use short complete semantic phrases, usually between two and five words.</caption:Cues>+ </caption:Style>++ <caption:Style id="alice-caption" appearance={studio.caption.alice}>+ <caption:Cues>Use short complete semantic phrases, usually between two and five words.</caption:Cues>+ <caption:Field id="important" type="boolean" min-per-cue="0" max-per-cue="2">+ Select at most two words whose emphasis best communicates this Cue.+ </caption:Field>+ </caption:Style>++ <caption:Style id="bob-caption" appearance={studio.caption.bob}>+ <caption:Cues>Use short complete semantic phrases, usually between two and five words.</caption:Cues>+ </caption:Style>++ <caption:Program id="caption-program" narrative={script} default={dialogue-caption}>+ <caption:Use role="ALICE" style={alice-caption}/>+ <caption:Use role="BOB" style={bob-caption}/>+ <!-- Ordered whole-style replacement: no complement Selection is required. -->+ <caption:Use on={script.selection.product-demo} style={dialogue-caption}/>+ </caption:Program>++ <caption-ai:Planner+ id="caption-plan"+ narrative={script}+ program={caption-program}+ model="gemini-2.5-flash"+ />+<!-- The author explicitly chooses Seedance Mini. Runtime only binds a matching endpoint. --><seedance:Speechid="alice-take"model="mini"- script={script.segment.opening}+ dialogue={script.segment.opening.dialogue}prompt={alice-direction}duration="8">@@ -67,7 +98,7 @@<seedance:Speechid="bob-take"model="mini"- script={script.segment.answer}+ dialogue={script.segment.answer.dialogue}prompt={bob-direction}duration="8">@@ -75,9 +106,9 @@</seedance:Speech><!-- Ordered generated Takes become one audio basis plus its peer visual/audio Tracks. -->- <speech:Spine id="speech" narrative={script} appearance={studio.speech.full}>- <speech:Take source={alice-take}/>- <speech:Take source={bob-take}/>+ <speech:Spine id="speech">+ <speech:Take source={alice-take} segment={script.segment.opening}/>+ <speech:Take source={bob-take} segment={script.segment.answer}/></speech:Spine><!-- One explicit WhisperX path: request, normalize and deterministic Script alignment. -->@@ -105,12 +136,9 @@id="captions"narrative={script}map={timing.map}- grouping={studio.caption.short-cues}- appearance={studio.caption.dialogue}- >- <caption:Role role="ALICE" appearance={studio.caption.alice}/>- <caption:Role role="BOB" appearance={studio.caption.bob}/>- </caption:Track>+ program={caption-program}+ plan={caption-plan.plan}+ /><text:Track id="titles" space={speech.space}><text:Itemdiff --git a/examples/talking-film-golden/studio.svs b/examples/talking-film-golden/studio.svsindex c9ec02ba..d18ce33e 100644--- a/examples/talking-film-golden/studio.svs+++ b/examples/talking-film-golden/studio.svs@@ -54,6 +54,7 @@y: 0.76;width: 0.84;font: inter-semibold;+ weight: 600;size: 58;line-height: 0.96;align: center;@@ -64,11 +65,35 @@}caption.alice {- active-fill: #73FBD3;+ stack-order: 70;+ x: 0.08;+ y: 0.76;+ width: 0.84;+ font: inter-semibold;+ weight: 600;+ size: 58;+ line-height: 0.96;+ align: center;+ fill: #73FBD3;+ background: #09090BCC;+ padding: 16 24;+ radius: 18;}caption.bob {- active-fill: #FFD166;+ stack-order: 70;+ x: 0.08;+ y: 0.76;+ width: 0.84;+ font: inter-semibold;+ weight: 600;+ size: 58;+ line-height: 0.96;+ align: center;+ fill: #FFD166;+ background: #09090BCC;+ padding: 16 24;+ radius: 18;}text.title {@@ -78,6 +103,7 @@width: 0.88;height: 0.10;font: inter-black;+ weight: 900;size: 64;align: center;fill: #FFFFFF;diff --git a/examples/talking-film-graph-check/README.md b/examples/talking-film-graph-check/README.mdnew file mode 100644index 00000000..d00f0df5--- /dev/null+++ b/examples/talking-film-graph-check/README.md@@ -0,0 +1,15 @@+# Talking-film graph check++This fixture compiles the complete author graph: Script, generated-media requests, Speech Spine,+WhisperX alignment, display-only Gemini Caption planning, B-roll, Caption, Text, Film and+explicit HyperFrames rendering. `check` and `plan` do not invoke Seedance, WhisperX, Gemini or+HyperFrames.++```bash+pnpm svml:v2 check examples/talking-film-graph-check/main.svml+pnpm svml:v2 plan examples/talking-film-graph-check/main.svml --target final.video+```++The Gemini planner receives immutable left-side display atoms and resolved style runs only. Its generic+`CaptionPlan` joins the independent `CompleteSemanticMap` in `@svml/caption`; neither the planner+nor the Vertex Provider has text-rewriting or timing authority.diff --git a/examples/talking-film-graph-check/main.svml b/examples/talking-film-graph-check/main.svmlnew file mode 100644index 00000000..034c43ed--- /dev/null+++ b/examples/talking-film-graph-check/main.svml@@ -0,0 +1,54 @@+<svml>+ <import from="@svml/script@1"/>+ <import as="seedance" from="@svml/seedance@1"/>+ <import as="speech" from="@svml/speech@1"/>+ <import as="whisperx" from="@svml/whisperx@1"/>+ <import as="broll" from="@svml/broll@1"/>+ <import as="caption" from="@svml/caption@1"/>+ <import as="caption-ai" from="@svml/caption-gemini@1"/>+ <import as="text" from="@svml/text-track@1"/>+ <import as="film" from="@svml/film@1"/>+ <import as="render" from="@svml/hyperframes-render@1"/>+ <import as="studio" from="./studio.svs" using="@svml/svs@1"/>++ <script id="story">+ <opening><HOST>Meaning @demo becomes the source @/demo.</opening>+ </script>++ <seedance:Prompt id="direction">Locked medium close-up in a quiet daylight studio.</seedance:Prompt>+ <seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue} prompt={direction} duration="5"/>+ <seedance:Video id="motion" model="mini" prompt={direction} duration="5"/>++ <speech:Spine id="speech">+ <speech:Take source={take} segment={story.segment.opening}/>+ </speech:Spine>+ <whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>++ <caption:Style id="base-caption" appearance={studio.caption.base}>+ <caption:Cues>Prefer short complete semantic phrases. Never cross clause punctuation.</caption:Cues>+ <caption:Field id="important" type="boolean"+ min-per-cue="0" max-per-cue="2">+ Select zero, one, or two words whose emphasis best communicates this Cue.+ </caption:Field>+ </caption:Style>+ <caption:Program id="caption-program" narrative={story} default={base-caption}/>+ <caption-ai:Planner id="cue-plan" narrative={story} program={caption-program} model="gemini-2.5-flash"/>++ <broll:Track id="cards" map={timing.map}>+ <broll:Item source={motion.video} during={story.selection.demo} appearance={studio.broll.card}/>+ </broll:Track>+ <caption:Track id="captions" narrative={story} map={timing.map}+ plan={cue-plan.plan} program={caption-program}/>+ <text:Track id="titles" space={speech.space}>+ <text:Item text="MEANING" during="full" appearance={studio.text.title}/>+ </text:Track>++ <film:Film id="main" space={speech.space} appearance={studio.film.vertical}>+ <film:Track source={speech.visual}/>+ <film:Track source={speech.audioTrack}/>+ <film:Track source={cards.visual}/>+ <film:Track source={captions.track}/>+ <film:Track source={titles.track}/>+ </film:Film>+ <render:Video id="final" composition={main.composition}/>+</svml>diff --git a/examples/talking-film-graph-check/studio.svs b/examples/talking-film-graph-check/studio.svsnew file mode 100644index 00000000..a97bea8c--- /dev/null+++ b/examples/talking-film-graph-check/studio.svs@@ -0,0 +1,16 @@+<sheet version="1">+ film.vertical { width: 1080; height: 1920; frame-rate: 30; background: #09090B; }+ broll.card {+ stack-order: 40; x: 0.1; y: 0.2; width: 0.8; height: 0.5; fit: cover;+ background: #111116; radius: 20; enter: slide-up 4f; exit: fade 4f;+ }+ caption.base {+ stack-order: 70; x: 0.08; y: 0.76; width: 0.84; font: Inter; weight: 600;+ size: 58; line-height: 1; align: center; fill: #FFFFFF;+ background: #09090BCC; padding: 16 24; radius: 18;+ }+ text.title {+ stack-order: 90; x: 0.06; y: 0.06; width: 0.88; height: 0.1; font: Inter;+ weight: 900; size: 64; align: center; fill: #FFFFFF; tracking: -1;+ }+</sheet>diff --git a/packages/broll/README.md b/packages/broll/README.mdindex 71a980e4..77f5d1f0 100644--- a/packages/broll/README.md+++ b/packages/broll/README.md@@ -5,6 +5,10 @@ entrance/exit motion, pair transitions, source audio and transition SFX. Itemits ordinary `VisualTrack` and `AudioTrack` values whose Presents participatein Composition by their own absolute z.+The `<broll:Track>` Surface consumes explicit `CompleteSemanticMap`, authored Script selection,+source Blob and SVS Recipe edges. Each source is inspected and normalized through the shared media+Fragment; semantic-window placement and local motion are deterministic package Operations.+`BrollProgram@1` remains an executable vertical-slice fixture. The author-facing package is notfrozen until the old content-frame plus independent foreground/backdrop sampling witnesses lowerthrough the closed HyperFrames Visual IR.diff --git a/packages/broll/package.json b/packages/broll/package.jsonindex 92bc3d7d..bb715551 100644--- a/packages/broll/package.json+++ b/packages/broll/package.json@@ -7,8 +7,13 @@".": "./src/index.ts"},"dependencies": {+ "@svml/component-kit": "workspace:*","@svml/contracts": "workspace:*",- "@svml/protocol": "workspace:*"+ "@svml/elaborator": "workspace:*",+ "@svml/media-pipeline": "workspace:*",+ "@svml/protocol": "workspace:*",+ "@svml/svs": "workspace:*",+ "@svml/text": "workspace:*"},"devDependencies": {"@svml/hyperframes": "workspace:*"diff --git a/packages/broll/src/author.ts b/packages/broll/src/author.tsnew file mode 100644index 00000000..9827ec52--- /dev/null+++ b/packages/broll/src/author.ts@@ -0,0 +1,175 @@+import {+ assertProgramSpaceIdentity,+ verifySynchronizedMedia,+} from "@svml/contracts";+import type {+ CompleteSemanticMap,+ NarrativeSelectionRef,+ SynchronizedMedia,+} from "@svml/contracts";+import { canonicalize, digestOf, isDigest } from "@svml/protocol";++import { sealBrollProgram } from "./program.js";+import type {+ BrollItemSpec,+ BrollProgram,+ BrollSet,+ BrollTrackSpec,+} from "./types.js";++export const createBrollSetImplementationDigest = digestOf("@svml/broll/create-set@1");+export const appendBrollItemImplementationDigest = digestOf("@svml/broll/append-item@1");+export const finalizeBrollProgramImplementationDigest = digestOf("@svml/broll/finalize-program@1");+export const projectBrollProgramSpaceImplementationDigest = digestOf("@svml/broll/project-program-space@1");++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function verifyMap(map: CompleteSemanticMap): void {+ assert(map.contract === "svml.complete-semantic-map@1", "B-roll requires CompleteSemanticMap");+ assertProgramSpaceIdentity(map.programSpace);+ const { mapDigest: _digest, ...content } = map;+ assert(isDigest(map.mapDigest) && map.mapDigest === digestOf(canonicalize(content)),+ "CompleteSemanticMap digest differs from its contents");+}++export function projectBrollProgramSpace(map: CompleteSemanticMap) {+ verifyMap(map);+ return map.programSpace;+}++export function sealBrollItemSpec(value: Omit<BrollItemSpec, "digest">): BrollItemSpec {+ const content = canonicalize(value) as unknown as Omit<BrollItemSpec, "digest">;+ return { ...content, digest: digestOf(content) };+}++export function assertBrollItemSpec(value: BrollItemSpec): void {+ assert(value.contract === "svml.broll-item-spec@1" && value.id.length > 0, "BrollItemSpec identity is invalid");+ const { digest: _digest, ...content } = value;+ assert(isDigest(value.digest) && value.digest === digestOf(canonicalize(content)), "BrollItemSpec digest differs");+ assert(Number.isSafeInteger(value.z), "BrollItemSpec z must be an integer");+ assert(value.fit === "contain" || value.fit === "cover", "BrollItemSpec fit is invalid");+}++export function sealBrollTrackSpec(value: Omit<BrollTrackSpec, "digest">): BrollTrackSpec {+ const content = canonicalize(value) as unknown as Omit<BrollTrackSpec, "digest">;+ return { ...content, digest: digestOf(content) };+}++function assertBrollTrackSpec(value: BrollTrackSpec): void {+ const { digest: _digest, ...content } = value;+ assert(value.contract === "svml.broll-track-spec@1" && value.id.length > 0+ && isDigest(value.digest) && value.digest === digestOf(canonicalize(content)), "BrollTrackSpec is invalid");+}++function verifySelection(value: NarrativeSelectionRef): void {+ assert(value.contract === "svml.narrative-selection@1" && value.id.length > 0 && value.occurrences.length > 0,+ "B-roll selection is invalid");+ const { selectionDigest: _digest, ...content } = value;+ assert(isDigest(value.selectionDigest) && value.selectionDigest === digestOf(canonicalize(content)),+ "B-roll selection digest differs");+}++function setContent(value: Omit<BrollSet, "digest">): Omit<BrollSet, "digest"> {+ return canonicalize(value) as unknown as Omit<BrollSet, "digest">;+}++function sealSet(value: Omit<BrollSet, "digest">): BrollSet {+ const content = setContent(value);+ return { ...content, digest: digestOf(content) };+}++export function assertBrollSet(value: BrollSet): void {+ assert(value.contract === "svml.broll-set@1" && value.id.length > 0, "BrollSet identity is invalid");+ verifyMap(value.map);+ const { digest: _digest, ...content } = value;+ assert(isDigest(value.digest) && value.digest === digestOf(setContent(content)), "BrollSet digest differs");+ assert(value.items.length === 0 ? value.lastAddition === undefined : value.lastAddition !== undefined,+ "BrollSet last addition is inconsistent");+}++export function createBrollSet(map: CompleteSemanticMap, spec: BrollTrackSpec): BrollSet {+ verifyMap(map);+ assertBrollTrackSpec(spec);+ return sealSet({ contract: "svml.broll-set@1", id: spec.id, map, items: [] });+}++function boundaryFrame(+ map: CompleteSemanticMap,+ edge: NarrativeSelectionRef["occurrences"][number]["open"],+ side: "open" | "close",+): number {+ const index = edge.boundary.tokenIndex;+ if (side === "open") {+ if (edge.affinity === "left" && index > 0) return map.tokens[index - 1]?.startFrame ?? 0;+ return map.tokens[index]?.startFrame ?? Math.round(map.durationSec * map.programSpace.frameRate.numerator / map.programSpace.frameRate.denominator);+ }+ if (edge.affinity === "right" && index < map.tokens.length) return map.tokens[index]?.endFrame ?? 0;+ return index > 0 ? map.tokens[index - 1]?.endFrame ?? 0 : 0;+}++export function appendBrollItem(+ set: BrollSet,+ media: SynchronizedMedia,+ selection: NarrativeSelectionRef,+ spec: BrollItemSpec,+): BrollSet {+ assertBrollSet(set);+ verifySynchronizedMedia(media);+ verifySelection(selection);+ assertBrollItemSpec(spec);+ assert(media.visual !== undefined, `B-roll ${spec.id} source has no visual stream`);+ assert(media.timeline.frameRate.numerator === set.map.programSpace.frameRate.numerator+ && media.timeline.frameRate.denominator === set.map.programSpace.frameRate.denominator,+ `B-roll ${spec.id} media uses another frame rate`);+ const durationSec = media.timeline.frameCount * media.timeline.frameRate.denominator / media.timeline.frameRate.numerator;+ const additions = selection.occurrences.map((occurrence, index) => {+ const startFrame = boundaryFrame(set.map, occurrence.open, "open");+ const endFrameExclusive = boundaryFrame(set.map, occurrence.close, "close");+ assert(endFrameExclusive > startFrame, `B-roll ${spec.id} selection occurrence is empty`);+ return {+ id: selection.occurrences.length === 1 ? spec.id : `${spec.id}:${index + 1}`,+ artifact: {+ digest: media.visual!.artifact.digest,+ size: media.visual!.artifact.size,+ mediaType: media.visual!.artifact.mediaType,+ durationSec,+ },+ span: { startFrame, endFrameExclusive },+ z: spec.z,+ tieBreak: `${set.id}:${spec.id}:${index + 1}`,+ box: spec.box,+ fit: spec.fit,+ playback: "stretch" as const,+ ...(spec.backgroundColor === undefined ? {} : { backgroundColor: spec.backgroundColor }),+ ...(spec.borderRadiusPx === undefined ? {} : { borderRadiusPx: spec.borderRadiusPx }),+ ...(spec.enter === undefined ? {} : { enter: spec.enter }),+ ...(spec.exit === undefined ? {} : { exit: spec.exit }),+ };+ });+ return sealSet({+ contract: "svml.broll-set@1",+ id: set.id,+ map: set.map,+ items: [...set.items, ...additions],+ lastAddition: {+ previousSetDigest: set.digest,+ mediaDigest: media.synchronizedMediaDigest,+ selectionDigest: selection.selectionDigest,+ specDigest: spec.digest,+ },+ });+}++export function finalizeBrollProgram(set: BrollSet): BrollProgram {+ assertBrollSet(set);+ assert(set.items.length > 0, "B-roll Track requires at least one Item");+ return sealBrollProgram({+ contract: "svml.broll-program@1",+ id: set.id,+ programSpaceDigest: set.map.programSpace.digest,+ items: set.items,+ transitions: [],+ });+}diff --git a/packages/broll/src/component.ts b/packages/broll/src/component.tsnew file mode 100644index 00000000..ed627982--- /dev/null+++ b/packages/broll/src/component.ts@@ -0,0 +1,103 @@+import type { ComponentPackage } from "@svml/component-kit";+import type { ProgramSpace } from "@svml/contracts";+import type { CompleteSemanticMap, NarrativeSelectionRef, SynchronizedMedia } from "@svml/contracts";+import type { StoredValue } from "@svml/protocol";+import { canonicalize } from "@svml/protocol";++import { brollProducers } from "./manifest.js";+import {+ appendBrollItem,+ appendBrollItemImplementationDigest,+ createBrollSet,+ createBrollSetImplementationDigest,+ finalizeBrollProgram,+ finalizeBrollProgramImplementationDigest,+ projectBrollProgramSpace,+ projectBrollProgramSpaceImplementationDigest,+} from "./author.js";+import {+ compileBrollImplementationDigest,+ compileBrollProduct,+ projectBrollAudio,+ projectBrollAudioImplementationDigest,+ projectBrollVisual,+ projectBrollVisualImplementationDigest,+} from "./program.js";+import type { BrollItemSpec, BrollProduct, BrollProgram, BrollSet, BrollTrackSpec } from "./types.js";++function inline<T>(value: StoredValue | undefined, subject: string): T {+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);+ return value.value as unknown as T;+}++export const brollComponent = {+ name: "@svml/broll",+ producers: [+ {+ producer: brollProducers.projectSpace,+ implementationDigest: projectBrollProgramSpaceImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { space: { kind: "inline", value: canonicalize(projectBrollProgramSpace(+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ )) } }, needs: {},+ }),+ },+ {+ producer: brollProducers.createSet,+ implementationDigest: createBrollSetImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(createBrollSet(+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<BrollTrackSpec>(inputs.spec?.value, "BrollTrackSpec"),+ )) } }, needs: {},+ }),+ },+ {+ producer: brollProducers.appendItem,+ implementationDigest: appendBrollItemImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(appendBrollItem(+ inline<BrollSet>(inputs.set?.value, "BrollSet"),+ inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),+ inline<NarrativeSelectionRef>(inputs.selection?.value, "NarrativeSelection"),+ inline<BrollItemSpec>(inputs.spec?.value, "BrollItemSpec"),+ )) } }, needs: {},+ }),+ },+ {+ producer: brollProducers.finalize,+ implementationDigest: finalizeBrollProgramImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { program: { kind: "inline", value: canonicalize(finalizeBrollProgram(+ inline<BrollSet>(inputs.set?.value, "BrollSet"),+ )) } }, needs: {},+ }),+ },+ {+ producer: brollProducers.compile,+ implementationDigest: compileBrollImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { product: { kind: "inline", value: canonicalize(compileBrollProduct(+ inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"), inline<BrollProgram>(inputs.program?.value, "BrollProgram"),+ )) } },+ needs: {},+ }),+ },+ {+ producer: brollProducers.projectVisual,+ implementationDigest: projectBrollVisualImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { visual: { kind: "inline", value: canonicalize(projectBrollVisual(inline<BrollProduct>(inputs.product?.value, "BrollProduct"))) } },+ needs: {},+ }),+ },+ {+ producer: brollProducers.projectAudio,+ implementationDigest: projectBrollAudioImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { audio: { kind: "inline", value: canonicalize(projectBrollAudio(inline<BrollProduct>(inputs.product?.value, "BrollProduct"))) } },+ needs: {},+ }),+ },+ ],+} satisfies ComponentPackage;diff --git a/packages/broll/src/index.ts b/packages/broll/src/index.tsindex b147f342..7dd9db8d 100644--- a/packages/broll/src/index.ts+++ b/packages/broll/src/index.ts@@ -11,13 +11,33 @@ export {projectBrollVisualImplementationDigest,sealBrollProgram,} from "./program.js";+export { brollComponent } from "./component.js";+export {+ appendBrollItem,+ appendBrollItemImplementationDigest,+ assertBrollItemSpec,+ assertBrollSet,+ createBrollSet,+ createBrollSetImplementationDigest,+ finalizeBrollProgram,+ finalizeBrollProgramImplementationDigest,+ projectBrollProgramSpace,+ projectBrollProgramSpaceImplementationDigest,+ sealBrollItemSpec,+ sealBrollTrackSpec,+} from "./author.js";export {brollManifest,brollManifestDigest,brollModuleRef,brollProductSchema,brollProgramSchema,+ brollSetSchema,+ brollItemSpecSchema,+ brollTrackSpecSchema,+ brollSurfaceImplementationDigest,brollProducers,brollTypes,} from "./manifest.js";+export { decodeBrollTrackSurface } from "./surface.js";export type * from "./types.js";diff --git a/packages/broll/src/manifest.ts b/packages/broll/src/manifest.tsindex 807ad900..fbb818e2 100644--- a/packages/broll/src/manifest.ts+++ b/packages/broll/src/manifest.ts@@ -1,13 +1,25 @@import {audioTrackSchema,+ completeSemanticMapSchema,contractTypes,mediaArtifactSchema,+ narrativeSelectionSchema,programSpaceSchema,+ synchronizedMediaSchema,videoContractDependencies,visualTrackSchema,} from "@svml/contracts";import { digestOf } from "@svml/protocol";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";+import { mediaPipelineManifest, mediaPipelineModuleRef, mediaPipelineTypes } from "@svml/media-pipeline";+import { svsManifest, svsModuleRef } from "@svml/svs";++import {+ appendBrollItemImplementationDigest,+ createBrollSetImplementationDigest,+ finalizeBrollProgramImplementationDigest,+ projectBrollProgramSpaceImplementationDigest,+} from "./author.js";import {compileBrollImplementationDigest,@@ -19,8 +31,15 @@ export const brollModuleRef = { name: "@svml/broll", version: "0.0.0-dev" } as cexport const brollTypes = {program: { module: brollModuleRef, name: "BrollProgram" },product: { module: brollModuleRef, name: "BrollProduct" },+ trackSpec: { module: brollModuleRef, name: "BrollTrackSpec" },+ itemSpec: { module: brollModuleRef, name: "BrollItemSpec" },+ set: { module: brollModuleRef, name: "BrollSet" },} satisfies Record<string, TypeRef>;export const brollProducers = {+ createSet: { module: brollModuleRef, name: "create-broll-set" },+ appendItem: { module: brollModuleRef, name: "append-broll-item" },+ finalize: { module: brollModuleRef, name: "finalize-broll-program" },+ projectSpace: { module: brollModuleRef, name: "project-broll-program-space" },compile: { module: brollModuleRef, name: "compile-broll" },projectVisual: { module: brollModuleRef, name: "project-broll-visual" },projectAudio: { module: brollModuleRef, name: "project-broll-audio" },@@ -65,6 +84,20 @@ const item = object({audioGain: { schema: number, optional: true },enter: { schema: motion, optional: true },exit: { schema: motion, optional: true },+ backgroundColor: { schema: string, optional: true },+ borderRadiusPx: { schema: number, optional: true },+});+const itemSpec = object({+ contract: { schema: { kind: "literal", value: "svml.broll-item-spec@1" } },+ digest: { schema: digest },+ id: { schema: string },+ z: { schema: signedInteger },+ box: { schema: box },+ fit: { schema: { kind: "string", enum: ["contain", "cover"] } },+ backgroundColor: { schema: string, optional: true },+ borderRadiusPx: { schema: number, optional: true },+ enter: { schema: motion, optional: true },+ exit: { schema: motion, optional: true },});const transition = object({id: { schema: string },@@ -96,21 +129,95 @@ export const brollProductSchema: ValueSchema = object({audioTrack: { schema: audioTrackSchema },});+export const brollTrackSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.broll-track-spec@1" } },+ digest: { schema: digest },+ id: { schema: string },+});+export const brollItemSpecSchema: ValueSchema = itemSpec;+export const brollSetSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.broll-set@1" } },+ digest: { schema: digest },+ id: { schema: string },+ map: { schema: completeSemanticMapSchema },+ items: { schema: { kind: "array", items: item } },+ lastAddition: { schema: object({+ previousSetDigest: { schema: digest }, mediaDigest: { schema: digest },+ selectionDigest: { schema: digest }, specDigest: { schema: digest },+ }), optional: true },+});++export const brollSurfaceImplementationDigest = digestOf("@svml/broll/track-surface@1");+export const brollManifest: ModuleManifest = {format: "svml.module@0",name: brollModuleRef.name,version: brollModuleRef.version,dependencies: [+ videoContractDependencies.narrative,+ videoContractDependencies.media,+ videoContractDependencies.semanticTime,videoContractDependencies.programSpace,videoContractDependencies.composition,+ { module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },+ { module: svsModuleRef, digest: digestOf(svsManifest) },],types: [{ name: brollTypes.program.name, schema: brollProgramSchema },{ name: brollTypes.product.name, schema: brollProductSchema },+ { name: brollTypes.trackSpec.name, schema: brollTrackSpecSchema },+ { name: brollTypes.itemSpec.name, schema: brollItemSpecSchema },+ { name: brollTypes.set.name, schema: brollSetSchema },],capabilities: [],- surfaces: [],+ surfaces: [{+ name: "track", tag: "Track", mode: "structured",+ outputs: [brollTypes.trackSpec, brollTypes.itemSpec, mediaPipelineTypes.selectionRequest,+ brollTypes.program, brollTypes.product, contractTypes.visualTrack, contractTypes.audioTrack],+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/broll/track-surface", digest: brollSurfaceImplementationDigest },+ }],producers: [+ {+ name: brollProducers.projectSpace.name,+ inputs: [{ name: "map", type: contractTypes.completeSemanticMap }],+ outputs: [{ name: "space", type: contractTypes.programSpace, affinity: [+ { resultPointer: "/digest", input: "map", inputPointer: "/programSpace/digest" },+ ] }], needs: [],+ implementation: { kind: "registered", locator: "@svml/broll/project-program-space", digest: projectBrollProgramSpaceImplementationDigest },+ },+ {+ name: brollProducers.createSet.name,+ inputs: [+ { name: "map", type: contractTypes.completeSemanticMap },+ { name: "spec", type: brollTypes.trackSpec },+ ],+ outputs: [{ name: "set", type: brollTypes.set }], needs: [],+ implementation: { kind: "registered", locator: "@svml/broll/create-set", digest: createBrollSetImplementationDigest },+ },+ {+ name: brollProducers.appendItem.name,+ inputs: [+ { name: "set", type: brollTypes.set },+ { name: "media", type: contractTypes.synchronizedMedia },+ { name: "selection", type: contractTypes.narrativeSelection },+ { name: "spec", type: brollTypes.itemSpec },+ ],+ outputs: [{ name: "set", type: brollTypes.set, affinity: [+ { resultPointer: "/lastAddition/previousSetDigest", input: "set", inputPointer: "/digest" },+ { resultPointer: "/lastAddition/mediaDigest", input: "media", inputPointer: "/synchronizedMediaDigest" },+ { resultPointer: "/lastAddition/selectionDigest", input: "selection", inputPointer: "/selectionDigest" },+ { resultPointer: "/lastAddition/specDigest", input: "spec", inputPointer: "/digest" },+ ] }], needs: [],+ implementation: { kind: "registered", locator: "@svml/broll/append-item", digest: appendBrollItemImplementationDigest },+ },+ {+ name: brollProducers.finalize.name,+ inputs: [{ name: "set", type: brollTypes.set }],+ outputs: [{ name: "program", type: brollTypes.program, affinity: [+ { resultPointer: "/programSpaceDigest", input: "set", inputPointer: "/map/programSpace/digest" },+ ] }], needs: [],+ implementation: { kind: "registered", locator: "@svml/broll/finalize-program", digest: finalizeBrollProgramImplementationDigest },+ },{name: brollProducers.compile.name,inputs: [diff --git a/packages/broll/src/program.ts b/packages/broll/src/program.tsindex 0c2efaf0..95661b34 100644--- a/packages/broll/src/program.ts+++ b/packages/broll/src/program.ts@@ -116,6 +116,12 @@ function assertItem(item: BrollItem, totalFrames: number): void {if (item.audioGain !== undefined && (!Number.isFinite(item.audioGain) || item.audioGain < 0)) {throw new Error(`B-roll item ${item.id} audioGain is invalid.`);}+ if (item.backgroundColor !== undefined && !/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/iu.test(item.backgroundColor)) {+ throw new Error(`B-roll item ${item.id} background color is invalid.`);+ }+ if (item.borderRadiusPx !== undefined && (!Number.isFinite(item.borderRadiusPx) || item.borderRadiusPx < 0)) {+ throw new Error(`B-roll item ${item.id} border radius is invalid.`);+ }if (item.includeAudio && isImage) throw new Error(`B-roll image ${item.id} cannot contribute source audio.`);if (item.includeAudio && item.playback === "loop") {throw new Error(`B-roll item ${item.id} cannot loop source audio in the v1 AudioTrack contract.`);@@ -311,6 +317,8 @@ function itemElements({ name: "top", value: `${item.box.yPercent}%` },{ name: "transform-origin", value: "center center" },{ name: "width", value: `${item.box.widthPercent}%` },+ ...(item.backgroundColor === undefined ? [] : [{ name: "background-color", value: item.backgroundColor }]),+ ...(item.borderRadiusPx === undefined ? [] : [{ name: "border-radius", value: `${item.borderRadiusPx}px` }]),];return [{diff --git a/packages/broll/src/surface.ts b/packages/broll/src/surface.tsnew file mode 100644index 00000000..01a49513--- /dev/null+++ b/packages/broll/src/surface.ts@@ -0,0 +1,167 @@+import { contractTypes } from "@svml/contracts";+import { sealGraphFragment } from "@svml/elaborator";+import type { FragmentOperation } from "@svml/elaborator";+import {+ mediaPipelineTypes,+ sealMediaSelectionRequest,+ synchronizedMediaFragment,+} from "@svml/media-pipeline";+import { svsRecipeType } from "@svml/svs";+import type { SvsRecipe } from "@svml/svs";+import type { StructuredElement, StructuredSurfaceHandler, SurfaceResolvedReference, TextAttributeValue } from "@svml/text";++import { sealBrollItemSpec, sealBrollTrackSpec } from "./author.js";+import { brollProducers, brollTypes } from "./manifest.js";+import type { BrollMotion, BrollSurfaceItemInput } from "./types.js";++const input = (name: string) => ({ kind: "fragment-input" as const, name });+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });++function createBrollSurfaceFragment(items: readonly BrollSurfaceItemInput[], name: string) {+ const operations: FragmentOperation[] = [+ { id: "broll:space", producer: brollProducers.projectSpace, inputs: { map: input("map") }, result: { kind: "output", name: "space" } },+ { id: "broll:set:empty", producer: brollProducers.createSet, inputs: { map: input("map"), spec: input("track-spec") }, result: { kind: "output", name: "set" } },+ ];+ let current = "broll:set:empty";+ items.forEach((item, index) => {+ const id = `broll:set:append:${String(index + 1).padStart(4, "0")}`;+ operations.push({+ id, producer: brollProducers.appendItem,+ inputs: { set: operation(current), media: input(item.mediaName), selection: input(item.selectionName), spec: input(item.specName) },+ result: { kind: "output", name: "set" },+ });+ current = id;+ });+ operations.push(+ { id: "broll:program", producer: brollProducers.finalize, inputs: { set: operation(current) }, result: { kind: "output", name: "program" } },+ { id: "broll:product", producer: brollProducers.compile, inputs: { space: operation("broll:space"), program: operation("broll:program") }, result: { kind: "output", name: "product" } },+ { id: "broll:visual", producer: brollProducers.projectVisual, inputs: { product: operation("broll:product") }, result: { kind: "output", name: "visual" } },+ { id: "broll:audio", producer: brollProducers.projectAudio, inputs: { product: operation("broll:product") }, result: { kind: "output", name: "audio" } },+ );+ const semanticInputs = ["map", "track-spec", ...items.flatMap((item) => [item.mediaName, item.selectionName, item.specName])];+ return sealGraphFragment({+ name,+ inputs: [+ { name: "map", type: contractTypes.completeSemanticMap },+ { name: "track-spec", type: brollTypes.trackSpec },+ ...items.flatMap((item) => [+ { name: item.mediaName, type: contractTypes.synchronizedMedia },+ { name: item.selectionName, type: contractTypes.narrativeSelection },+ { name: item.specName, type: brollTypes.itemSpec },+ ]),+ ],+ operations,+ exports: [+ { name: "visual", type: contractTypes.visualTrack, root: operation("broll:visual"), semanticInputs,+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("map"), sourcePointer: "/programSpace/digest" }], fidelity: "exact" },+ { name: "audio", type: contractTypes.audioTrack, root: operation("broll:audio"), semanticInputs,+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("map"), sourcePointer: "/programSpace/digest" }], fidelity: "exact" },+ ],+ });+}++function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;+}+function exact(element: StructuredElement, names: readonly string[]): void {+ if (Object.keys(element.attributes).sort().join("\0") !== [...names].sort().join("\0")) throw new Error(`${element.name} requires exactly ${names.join(", ")}`);+}+function text(element: StructuredElement, name: string): string {+ const value = element.attributes[name];+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);+ return value.trim();+}+function ref(element: StructuredElement, name: string, expected: SurfaceResolvedReference["type"], resolve: (path: string) => SurfaceResolvedReference | undefined) {+ const raw: TextAttributeValue | undefined = element.attributes[name];+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a reference`);+ const value = resolve(raw.path);+ if (!value || !sameType(value.type, expected)) throw new Error(`${element.name}.${name} cannot resolve a value of the required type`);+ return value;+}+function recipe(value: SurfaceResolvedReference): SvsRecipe {+ if (!sameType(value.type, svsRecipeType) || value.record?.value.kind !== "inline") throw new Error("B-roll appearance must be an authored SVS Recipe");+ return value.record.value.value as unknown as SvsRecipe;+}+function num(value: SvsRecipe, name: string): number {+ const result = value.properties[name];+ if (typeof result !== "number" || !Number.isFinite(result)) throw new Error(`B-roll Recipe ${name} must be a number`);+ return result;+}+function str(value: SvsRecipe, name: string): string {+ const result = value.properties[name];+ if (typeof result !== "string" || !result) throw new Error(`B-roll Recipe ${name} must be a string`);+ return result;+}+function motion(value: string): BrollMotion {+ const match = /^(fade|pop|slide-up|slide-down)\s+([1-9][0-9]*)f$/u.exec(value.trim());+ if (!match) throw new Error(`B-roll motion ${value} is invalid`);+ return { operator: match[1] as BrollMotion["operator"], durationFrames: Number(match[2]) };+}++export const decodeBrollTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ exact(element, ["id", "map"]);+ const id = text(element, "id");+ const map = ref(element, "map", contractTypes.completeSemanticMap, resolveReference);+ const trackSpecId = `${id}.spec`;+ const requestId = `${id}.selection`;+ const records: Array<{ id: string; type: typeof brollTypes.itemSpec | typeof brollTypes.trackSpec | typeof mediaPipelineTypes.selectionRequest; value: { kind: "inline"; value: ReturnType<typeof sealBrollItemSpec> | ReturnType<typeof sealBrollTrackSpec> | ReturnType<typeof sealMediaSelectionRequest> }; range: typeof element.range }> = [+ { id: trackSpecId, type: brollTypes.trackSpec, value: { kind: "inline", value: sealBrollTrackSpec({ contract: "svml.broll-track-spec@1", id }) }, range: element.range },+ { id: requestId, type: mediaPipelineTypes.selectionRequest, value: { kind: "inline", value: sealMediaSelectionRequest({+ contract: "svml.media-selection-request@1", video: { mode: "primary-moving" }, audio: { mode: "none" },+ spanAuthority: "video", frameRate: { numerator: 30, denominator: 1 },+ }) }, range: element.range },+ ];+ let itemIndex = 0;+ const declared = element.children.flatMap((child) => {+ if (child.kind === "text") {+ if (child.value.trim()) throw new Error(`${element.name} accepts only Item children`);+ return [];+ }+ if (!child.name.endsWith(":Item") && child.name !== "Item") throw new Error(`${element.name} accepts only Item children`);+ itemIndex += 1;+ exact(child, ["source", "during", "appearance"]);+ const source = ref(child, "source", contractTypes.blobArtifact, resolveReference);+ const selection = ref(child, "during", contractTypes.narrativeSelection, resolveReference);+ const appearance = recipe(ref(child, "appearance", svsRecipeType, resolveReference));+ const expected = ["background", "enter", "exit", "fit", "height", "radius", "stack-order", "width", "x", "y"];+ if (Object.keys(appearance.properties).sort().join("\0") !== expected.sort().join("\0")) throw new Error(`B-roll Recipe requires exactly ${expected.join(", ")}`);+ const fit = str(appearance, "fit");+ if (fit !== "contain" && fit !== "cover") throw new Error("B-roll Recipe fit is invalid");+ const suffix = String(itemIndex).padStart(4, "0");+ const specId = `${id}.item.${suffix}.spec`;+ records.push({+ id: specId, type: brollTypes.itemSpec,+ value: { kind: "inline", value: sealBrollItemSpec({+ contract: "svml.broll-item-spec@1", id: `${id}.item.${suffix}`, z: num(appearance, "stack-order"),+ box: { xPercent: num(appearance, "x") * 100, yPercent: num(appearance, "y") * 100,+ widthPercent: num(appearance, "width") * 100, heightPercent: num(appearance, "height") * 100 },+ fit, backgroundColor: str(appearance, "background"), borderRadiusPx: num(appearance, "radius"),+ enter: motion(str(appearance, "enter")), exit: motion(str(appearance, "exit")),+ }) }, range: child.range,+ });+ return [{ suffix, source, selection, specId, range: child.range,+ input: { mediaName: `item-${suffix}-media`, selectionName: `item-${suffix}-selection`, specName: `item-${suffix}-spec` } }];+ });+ if (declared.length === 0) throw new Error(`${element.name} requires at least one Item`);+ const fragment = createBrollSurfaceFragment(declared.map((item) => item.input), `@svml/broll/surface/${id}@1`);+ const normalization = declared.map((item) => ({+ id: `${id}.normalize.${item.suffix}`, fragment: synchronizedMediaFragment.id,+ inputs: { source: item.source.ref, request: { kind: "record" as const, id: requestId } },+ outputs: { media: `${id}.normalized.${item.suffix}` }, range: item.range,+ }));+ return {+ records,+ components: [+ ...normalization,+ { id, fragment: fragment.id, inputs: {+ map: map.ref, "track-spec": { kind: "record", id: trackSpecId },+ ...Object.fromEntries(declared.flatMap((item, index) => [+ [item.input.mediaName, { kind: "component-output" as const, component: normalization[index]!.id, output: "media" }],+ [item.input.selectionName, item.selection.ref],+ [item.input.specName, { kind: "record" as const, id: item.specId }],+ ])),+ }, outputs: { visual: `${id}.visual`, audio: `${id}.audio` }, range: element.range },+ ],+ fragments: [synchronizedMediaFragment, fragment],+ };+};diff --git a/packages/broll/src/types.ts b/packages/broll/src/types.tsindex 6c0044ee..49bed3c8 100644--- a/packages/broll/src/types.ts+++ b/packages/broll/src/types.ts@@ -37,6 +37,8 @@ export type BrollItem = {readonly audioGain?: number;readonly enter?: BrollMotion;readonly exit?: BrollMotion;+ readonly backgroundColor?: string;+ readonly borderRadiusPx?: number;};export type BrollPairTransition = {@@ -70,3 +72,42 @@ export type BrollProduct = {readonly visualTrack: VisualTrack;readonly audioTrack: AudioTrack;};++export type BrollItemSpec = {+ readonly contract: "svml.broll-item-spec@1";+ readonly digest: Digest;+ readonly id: string;+ readonly z: number;+ readonly box: BrollBox;+ readonly fit: "contain" | "cover";+ readonly backgroundColor?: string;+ readonly borderRadiusPx?: number;+ readonly enter?: BrollMotion;+ readonly exit?: BrollMotion;+};++export type BrollTrackSpec = {+ readonly contract: "svml.broll-track-spec@1";+ readonly digest: Digest;+ readonly id: string;+};++export type BrollSet = {+ readonly contract: "svml.broll-set@1";+ readonly digest: Digest;+ readonly id: string;+ readonly map: import("@svml/contracts").CompleteSemanticMap;+ readonly items: readonly BrollItem[];+ readonly lastAddition?: {+ readonly previousSetDigest: Digest;+ readonly mediaDigest: Digest;+ readonly selectionDigest: Digest;+ readonly specDigest: Digest;+ };+};++export type BrollSurfaceItemInput = {+ readonly mediaName: string;+ readonly selectionName: string;+ readonly specName: string;+};diff --git a/packages/caption-gemini/README.md b/packages/caption-gemini/README.mdnew file mode 100644index 00000000..359a2de2--- /dev/null+++ b/packages/caption-gemini/README.md@@ -0,0 +1,32 @@+# `@svml/caption-gemini`++Exact Gemini Caption planning, without transcript correction, Style selection or timing authority.++The package consumes an already-resolved `CaptionProgram`. Gemini sees only its immutable left-side+display atoms and the planning requirements for each resolved run. It never receives WhisperX+text, timestamps or the right side of Dual Text, and it cannot return replacement wording.++```xml+<caption-ai:Planner+ id="caption-plan"+ narrative={story}+ program={caption-program}+ model="gemini-2.5-flash"+/>+```++The output is `caption-plan.plan`. Its response has one entry per Program run. Within each run the+model may only:++- choose `after_atom_id` Cue boundaries that partition the run exactly once and in order;+- attach declared field values to atoms inside that Cue.++Fields are generic per-atom attributes, not hard-coded emphasis spans. A Style may declare no+fields, one `important` flag, independent `best` and `medium` flags, enums, bounded numbers, or any+other combination expressible by its declarations. Each atom may receive zero, one or multiple+attributes. The validator enforces ids, value types and per-Cue cardinality before a `CaptionPlan`+enters the graph.++Model choice is author-visible. Network execution is separate: a Runtime explicitly binds the+exact Gemini capability to `@svml/provider-google-vertex` or another compatible endpoint. The local+Scheduler owns concurrency; credentials and transport never enter `.svml` or BuildState.diff --git a/packages/caption-gemini/package.json b/packages/caption-gemini/package.jsonnew file mode 100644index 00000000..c1e0d5b0--- /dev/null+++ b/packages/caption-gemini/package.json@@ -0,0 +1,20 @@+{+ "name": "@svml/caption-gemini",+ "version": "0.0.0-dev",+ "private": true,+ "type": "module",+ "exports": {+ ".": "./src/index.ts"+ },+ "dependencies": {+ "@svml/caption": "workspace:*",+ "@svml/component-kit": "workspace:*",+ "@svml/contracts": "workspace:*",+ "@svml/elaborator": "workspace:*",+ "@svml/protocol": "workspace:*",+ "@svml/text": "workspace:*"+ },+ "devDependencies": {+ "@svml/script": "workspace:*"+ }+}diff --git a/packages/caption-gemini/src/component.ts b/packages/caption-gemini/src/component.tsnew file mode 100644index 00000000..394db912--- /dev/null+++ b/packages/caption-gemini/src/component.ts@@ -0,0 +1,59 @@+import type { ComponentPackage } from "@svml/component-kit";+import type { CaptionProgram } from "@svml/caption";+import type { Narrative } from "@svml/contracts";+import type { StoredValue } from "@svml/protocol";+import { canonicalize } from "@svml/protocol";++import { captionGeminiImplementationDigests, captionGeminiProducers, captionGeminiTypes } from "./manifest.js";+import { verifyCaptionGeminiProgram } from "./program.js";+import { compileCaptionGeminiRequest, verifyCaptionGeminiRequest } from "./request.js";+import type { CaptionGeminiProgram, CaptionGeminiRequest } from "./types.js";++function inline<T>(value: StoredValue | undefined, subject: string): T {+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);+ return value.value as T;+}++export const captionGeminiComponent = {+ name: "@svml/caption-gemini",+ producers: [+ {+ producer: captionGeminiProducers.compile,+ implementationDigest: captionGeminiImplementationDigests.compile,+ handler: ({ inputs }) => ({+ outputs: {+ request: {+ kind: "inline",+ value: canonicalize(compileCaptionGeminiRequest(+ inline<Narrative>(inputs.narrative?.value, "Narrative"),+ inline<CaptionProgram>(inputs.captionProgram?.value, "CaptionProgram"),+ inline<CaptionGeminiProgram>(inputs.program?.value, "CaptionGeminiProgram"),+ )),+ },+ },+ needs: {},+ }),+ },+ {+ producer: captionGeminiProducers.request,+ implementationDigest: captionGeminiImplementationDigests.request,+ handler: ({ inputs }) => {+ const request = inline<CaptionGeminiRequest>(inputs.request?.value, "CaptionGeminiRequest");+ verifyCaptionGeminiRequest(request);+ return { outputs: {}, needs: { plan: canonicalize(request) } };+ },+ },+ ],+ validators: [+ {+ type: captionGeminiTypes.program,+ implementationDigest: captionGeminiImplementationDigests.programValidator,+ handler: ({ value }) => verifyCaptionGeminiProgram(inline(value, "CaptionGeminiProgram")),+ },+ {+ type: captionGeminiTypes.request,+ implementationDigest: captionGeminiImplementationDigests.requestValidator,+ handler: ({ value }) => verifyCaptionGeminiRequest(inline(value, "CaptionGeminiRequest")),+ },+ ],+} satisfies ComponentPackage;diff --git a/packages/caption-gemini/src/fragment.ts b/packages/caption-gemini/src/fragment.tsnew file mode 100644index 00000000..f1cf7e67--- /dev/null+++ b/packages/caption-gemini/src/fragment.ts@@ -0,0 +1,43 @@+import { contractTypes } from "@svml/contracts";+import { captionTypes } from "@svml/caption";+import { sealGraphFragment } from "@svml/elaborator";++import { captionGeminiProducers, captionGeminiTypes } from "./manifest.js";++const input = (name: string) => ({ kind: "fragment-input" as const, name });+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });++export const captionGeminiPlanningFragment = sealGraphFragment({+ name: "@svml/caption-gemini/planning@1",+ inputs: [+ { name: "narrative", type: contractTypes.narrative },+ { name: "captionProgram", type: captionTypes.program },+ { name: "program", type: captionGeminiTypes.program },+ ],+ operations: [+ {+ id: "caption-gemini:compile",+ producer: captionGeminiProducers.compile,+ inputs: { narrative: input("narrative"), captionProgram: input("captionProgram"), program: input("program") },+ result: { kind: "output", name: "request" },+ },+ {+ id: "caption-gemini:plan",+ producer: captionGeminiProducers.request,+ inputs: { request: operation("caption-gemini:compile") },+ result: { kind: "need", name: "plan", accepts: "exact" },+ },+ ],+ exports: [{+ name: "plan",+ type: captionTypes.plan,+ root: operation("caption-gemini:plan"),+ semanticInputs: ["narrative", "captionProgram", "program"],+ affinity: [{+ resultPointer: "/planningRequestDigest",+ source: operation("caption-gemini:compile"),+ sourcePointer: "/requestDigest",+ }],+ fidelity: "exact",+ }],+});diff --git a/packages/caption-gemini/src/index.ts b/packages/caption-gemini/src/index.tsnew file mode 100644index 00000000..1901561a--- /dev/null+++ b/packages/caption-gemini/src/index.ts@@ -0,0 +1,22 @@+export { captionGeminiComponent } from "./component.js";+export { captionGeminiPlanningFragment } from "./fragment.js";+export {+ captionGeminiCapabilities,+ captionGeminiImplementationDigests,+ captionGeminiManifest,+ captionGeminiManifestDigest,+ captionGeminiModuleRef,+ captionGeminiProducers,+ captionGeminiProgramSchema,+ captionGeminiRequestSchema,+ captionGeminiTypes,+} from "./manifest.js";+export { sealCaptionGeminiPlan, verifyCaptionGeminiPlan } from "./plan.js";+export { sealCaptionGeminiProgram, verifyCaptionGeminiProgram } from "./program.js";+export {+ captionGeminiSystemInstruction,+ compileCaptionGeminiRequest,+ verifyCaptionGeminiRequest,+} from "./request.js";+export { decodeCaptionGeminiPlannerSurface } from "./surface.js";+export type * from "./types.js";diff --git a/packages/caption-gemini/src/manifest.ts b/packages/caption-gemini/src/manifest.tsnew file mode 100644index 00000000..1643e057--- /dev/null+++ b/packages/caption-gemini/src/manifest.ts@@ -0,0 +1,134 @@+import { captionManifest, captionModuleRef, captionTypes } from "@svml/caption";+import { contractTypes, videoContractDependencies } from "@svml/contracts";+import { digestOf } from "@svml/protocol";+import type { CapabilityRef, ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";++export const captionGeminiModuleRef = { name: "@svml/caption-gemini", version: "0.0.0-dev" } as const;+export const captionGeminiTypes = {+ program: { module: captionGeminiModuleRef, name: "CaptionGeminiProgram" },+ request: { module: captionGeminiModuleRef, name: "CaptionGeminiRequest" },+} satisfies Record<string, TypeRef>;+export const captionGeminiCapabilities = {+ plan: { module: captionGeminiModuleRef, name: "gemini-caption-planning" },+} satisfies Record<string, CapabilityRef>;+export const captionGeminiProducers = {+ compile: { module: captionGeminiModuleRef, name: "compile-caption-gemini-request" },+ request: { module: captionGeminiModuleRef, name: "request-caption-gemini-plan" },+} satisfies Record<string, ProducerRef>;+export const captionGeminiImplementationDigests = {+ compile: digestOf("@svml/caption-gemini/compile-request@2"),+ request: digestOf("@svml/caption-gemini/request-plan@2"),+ programValidator: digestOf("@svml/caption-gemini/validate-program@2"),+ requestValidator: digestOf("@svml/caption-gemini/validate-request@2"),+ plannerSurface: digestOf("@svml/caption-gemini/planner-surface@2"),+} as const;++const string = { kind: "string", minLength: 1 } as const satisfies ValueSchema;+const digest = { kind: "string", minLength: 71, maxLength: 71 } as const satisfies ValueSchema;+const nonNegativeInteger = { kind: "number", integer: true, minimum: 0 } as const satisfies ValueSchema;+const object = (+ fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>,+): ValueSchema => ({ kind: "object", fields });+const model = { kind: "string", enum: ["gemini-2.5-flash", "gemini-3.1-pro-preview"] } as const satisfies ValueSchema;+const fieldValueSchema: ValueSchema = {+ kind: "oneOf",+ variants: [+ object({ kind: { schema: { kind: "literal", value: "boolean" } } }),+ object({+ kind: { schema: { kind: "literal", value: "enum" } },+ values: { schema: { kind: "array", minItems: 1, items: string } },+ }),+ object({+ kind: { schema: { kind: "literal", value: "number" } },+ minimum: { schema: { kind: "number" }, optional: true },+ maximum: { schema: { kind: "number" }, optional: true },+ }),+ ],+};+const fieldDeclaration = object({+ id: { schema: string }, value: { schema: fieldValueSchema }, instruction: { schema: string },+ minimumPerCue: { schema: nonNegativeInteger }, maximumPerCue: { schema: nonNegativeInteger },+});+const planningRun = object({+ id: { schema: string }, styleId: { schema: string },+ atomIds: { schema: { kind: "array", minItems: 1, items: string } },+ cueInstruction: { schema: string }, fields: { schema: { kind: "array", items: fieldDeclaration } },+});+export const captionGeminiProgramSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.caption-gemini-program@1" } },+ model: { schema: model }, programDigest: { schema: digest },+});+export const captionGeminiRequestSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.caption-gemini-request@1" } },+ model: { schema: model }, narrativeDigest: { schema: digest }, captionProgramDigest: { schema: digest },+ atoms: { schema: { kind: "array", minItems: 1, items: object({ id: { schema: string }, text: { schema: string } }) } },+ runs: { schema: { kind: "array", minItems: 1, items: planningRun } },+ systemInstruction: { schema: string }, prompt: { schema: string },+ temperature: { schema: { kind: "literal", value: 0.2 } }, requestDigest: { schema: digest },+});++function ownedType(name: string, schema: ValueSchema, digestValue: ReturnType<typeof digestOf>) {+ return {+ name,+ schema,+ validator: {+ abi: "svml.type-validator@1" as const,+ implementation: { kind: "registered" as const, locator: `@svml/caption-gemini/validate-${name}`, digest: digestValue },+ },+ };+}++export const captionGeminiManifest: ModuleManifest = {+ format: "svml.module@0",+ name: captionGeminiModuleRef.name,+ version: captionGeminiModuleRef.version,+ dependencies: [+ videoContractDependencies.narrative,+ { module: captionModuleRef, digest: digestOf(captionManifest) },+ ],+ types: [+ ownedType(captionGeminiTypes.program.name, captionGeminiProgramSchema, captionGeminiImplementationDigests.programValidator),+ ownedType(captionGeminiTypes.request.name, captionGeminiRequestSchema, captionGeminiImplementationDigests.requestValidator),+ ],+ capabilities: [{ name: captionGeminiCapabilities.plan.name, returns: captionTypes.plan }],+ surfaces: [{+ name: "planner",+ tag: "Planner",+ mode: "structured",+ outputs: [captionGeminiTypes.program, captionTypes.plan],+ implementation: {+ kind: "trusted-frontend-surface",+ locator: "@svml/caption-gemini/planner-surface",+ digest: captionGeminiImplementationDigests.plannerSurface,+ },+ }],+ producers: [+ {+ name: captionGeminiProducers.compile.name,+ inputs: [+ { name: "narrative", type: contractTypes.narrative },+ { name: "captionProgram", type: captionTypes.program },+ { name: "program", type: captionGeminiTypes.program },+ ],+ outputs: [{ name: "request", type: captionGeminiTypes.request }],+ needs: [],+ implementation: {+ kind: "registered", locator: "@svml/caption-gemini/compile-request", digest: captionGeminiImplementationDigests.compile,+ },+ },+ {+ name: captionGeminiProducers.request.name,+ inputs: [{ name: "request", type: captionGeminiTypes.request }],+ outputs: [],+ needs: [{+ name: "plan", capability: captionGeminiCapabilities.plan, returns: captionTypes.plan,+ affinity: [{ resultPointer: "/planningRequestDigest", input: "request", inputPointer: "/requestDigest" }],+ }],+ implementation: {+ kind: "registered", locator: "@svml/caption-gemini/request-plan", digest: captionGeminiImplementationDigests.request,+ },+ },+ ],+};++export const captionGeminiManifestDigest = digestOf(captionGeminiManifest);diff --git a/packages/caption-gemini/src/plan.ts b/packages/caption-gemini/src/plan.tsnew file mode 100644index 00000000..3ed8b779--- /dev/null+++ b/packages/caption-gemini/src/plan.ts@@ -0,0 +1,126 @@+import { assertCaptionPlan, sealCaptionPlan } from "@svml/caption";+import type { CaptionFieldDeclaration, CaptionPlannedCue } from "@svml/caption";++import { verifyCaptionGeminiRequest } from "./request.js";+import type {+ CaptionGeminiPlan,+ CaptionGeminiRequest,+ RawCaptionGeminiResponse,+} from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function object(value: unknown, subject: string): Record<string, unknown> {+ assert(value !== null && typeof value === "object" && !Array.isArray(value), `${subject} must be an object`);+ return value as Record<string, unknown>;+}++function exactKeys(value: Record<string, unknown>, keys: readonly string[], subject: string): void {+ assert(Object.keys(value).sort().join("\0") === [...keys].sort().join("\0"), `${subject} has unknown or missing fields`);+}++function fieldValue(declaration: CaptionFieldDeclaration, value: unknown): string {+ assert(typeof value === "string", `Caption field ${declaration.id} value must be a string`);+ if (declaration.value.kind === "boolean") {+ assert(value === "true", `Caption field ${declaration.id} is sparse: absence means false`);+ } else if (declaration.value.kind === "enum") {+ assert(declaration.value.values.includes(value), `Caption field ${declaration.id} enum value is invalid`);+ } else {+ const number = Number(value);+ assert(Number.isFinite(number), `Caption field ${declaration.id} number value is invalid`);+ assert(declaration.value.minimum === undefined || number >= declaration.value.minimum,+ `Caption field ${declaration.id} value is below its minimum`);+ assert(declaration.value.maximum === undefined || number <= declaration.value.maximum,+ `Caption field ${declaration.id} value is above its maximum`);+ }+ return value;+}++function cueFromRaw(+ raw: unknown,+ runId: string,+ cueIndex: number,+ runAtomIds: readonly string[],+ cursor: number,+ declarations: ReadonlyMap<string, CaptionFieldDeclaration>,+): { readonly cue: CaptionPlannedCue; readonly nextCursor: number } {+ const value = object(raw, `Caption cue ${runId}/${cueIndex + 1}`);+ exactKeys(value, ["after_atom_id", "fields"], `Caption cue ${runId}/${cueIndex + 1}`);+ assert(typeof value.after_atom_id === "string", `Caption cue ${runId}/${cueIndex + 1} endpoint is invalid`);+ const endpoint = runAtomIds.indexOf(value.after_atom_id, cursor);+ assert(endpoint >= cursor, `Caption cue ${runId}/${cueIndex + 1} endpoint is outside its remaining run`);+ const atomIds = runAtomIds.slice(cursor, endpoint + 1);+ assert(Array.isArray(value.fields), `Caption cue ${runId}/${cueIndex + 1} fields are invalid`);+ const assignments = value.fields.map((rawField, fieldIndex) => {+ const field = object(rawField, `Caption field ${runId}/${cueIndex + 1}/${fieldIndex + 1}`);+ exactKeys(field, ["declaration_id", "atom_id", "value"], "Caption field assignment");+ assert(typeof field.declaration_id === "string", "Caption field declaration id is invalid");+ const declaration = declarations.get(field.declaration_id);+ assert(declaration !== undefined, `Caption field ${field.declaration_id} was not declared for run ${runId}`);+ assert(typeof field.atom_id === "string" && atomIds.includes(field.atom_id),+ `Caption field ${declaration.id} atom lies outside its cue`);+ return {+ declarationId: declaration.id,+ atomId: field.atom_id,+ value: fieldValue(declaration, field.value),+ };+ });+ const unique = new Set(assignments.map((field) => `${field.declarationId}\0${field.atomId}`));+ assert(unique.size === assignments.length, `Caption cue ${runId}/${cueIndex + 1} repeats a field assignment`);+ for (const declaration of declarations.values()) {+ const count = assignments.filter((field) => field.declarationId === declaration.id).length;+ assert(count >= declaration.minimumPerCue && count <= declaration.maximumPerCue,+ `Caption field ${declaration.id} count ${count} violates ${declaration.minimumPerCue}..${declaration.maximumPerCue} per cue`);+ }+ return {+ cue: { id: `${runId}:cue:${cueIndex + 1}`, atomIds, fields: assignments },+ nextCursor: endpoint + 1,+ };+}++export function sealCaptionGeminiPlan(+ request: CaptionGeminiRequest,+ response: unknown,+): CaptionGeminiPlan {+ verifyCaptionGeminiRequest(request);+ const root = object(response, "Caption Gemini response");+ exactKeys(root, ["runs"], "Caption Gemini response");+ assert(Array.isArray(root.runs), "Caption Gemini response runs are invalid");+ const rawRuns = new Map<string, Record<string, unknown>>();+ for (const rawRun of root.runs) {+ const run = object(rawRun, "Caption Gemini response run");+ exactKeys(run, ["run_id", "cues"], "Caption Gemini response run");+ assert(typeof run.run_id === "string" && !rawRuns.has(run.run_id), "Caption Gemini response run id is invalid or repeated");+ rawRuns.set(run.run_id, run);+ }+ assert(rawRuns.size === request.runs.length, "Caption Gemini response does not contain the exact run set");+ const runs = request.runs.map((expectedRun) => {+ const rawRun = rawRuns.get(expectedRun.id);+ assert(rawRun !== undefined && Array.isArray(rawRun.cues) && rawRun.cues.length > 0,+ `Caption run ${expectedRun.id} has no Cues`);+ const declarations = new Map(expectedRun.fields.map((field) => [field.id, field]));+ let cursor = 0;+ const cues = rawRun.cues.map((rawCue, cueIndex) => {+ const planned = cueFromRaw(rawCue, expectedRun.id, cueIndex, expectedRun.atomIds, cursor, declarations);+ cursor = planned.nextCursor;+ return planned.cue;+ });+ assert(cursor === expectedRun.atomIds.length, `Caption run ${expectedRun.id} Cues do not cover its final atom`);+ return { id: expectedRun.id, styleId: expectedRun.styleId, cues };+ });+ return sealCaptionPlan({+ contract: "svml.caption-plan@1",+ narrativeDigest: request.narrativeDigest,+ captionProgramDigest: request.captionProgramDigest,+ planningRequestDigest: request.requestDigest,+ runs,+ });+}++export function verifyCaptionGeminiPlan(value: unknown): asserts value is CaptionGeminiPlan {+ assertCaptionPlan(value);+}++export const rawCaptionGeminiResponseShape = null as unknown as RawCaptionGeminiResponse;diff --git a/packages/caption-gemini/src/program.ts b/packages/caption-gemini/src/program.tsnew file mode 100644index 00000000..b013e966--- /dev/null+++ b/packages/caption-gemini/src/program.ts@@ -0,0 +1,34 @@+import { digestOf, isDigest } from "@svml/protocol";++import type { CaptionGeminiProgram } from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function content(program: Omit<CaptionGeminiProgram, "programDigest">) {+ return {+ contract: "svml.caption-gemini-program@1" as const,+ model: program.model,+ };+}++export function verifyCaptionGeminiProgram(value: unknown): asserts value is CaptionGeminiProgram {+ assert(value !== null && typeof value === "object" && !Array.isArray(value), "Caption Gemini Program must be an object");+ const program = value as CaptionGeminiProgram;+ assert(program.contract === "svml.caption-gemini-program@1", "Caption Gemini Program contract is invalid");+ assert(program.model === "gemini-2.5-flash" || program.model === "gemini-3.1-pro-preview",+ "Caption Gemini model is unsupported");+ assert(isDigest(program.programDigest), "Caption Gemini Program digest is invalid");+ const { programDigest: _digest, ...withoutDigest } = program;+ assert(program.programDigest === digestOf(content(withoutDigest)), "Caption Gemini Program digest does not match its contents");+}++export function sealCaptionGeminiProgram(+ value: Omit<CaptionGeminiProgram, "programDigest">,+): CaptionGeminiProgram {+ const normalized = content(value);+ const program = { ...normalized, programDigest: digestOf(normalized) };+ verifyCaptionGeminiProgram(program);+ return program;+}diff --git a/packages/caption-gemini/src/request.ts b/packages/caption-gemini/src/request.tsnew file mode 100644index 00000000..9c2cf11d--- /dev/null+++ b/packages/caption-gemini/src/request.ts@@ -0,0 +1,136 @@+import { assertCaptionProgramForNarrative } from "@svml/caption";+import type { CaptionFieldDeclaration, CaptionProgram } from "@svml/caption";+import type { Narrative } from "@svml/contracts";+import { digestOf, isDigest } from "@svml/protocol";++import { verifyCaptionGeminiProgram } from "./program.js";+import type {+ CaptionGeminiProgram,+ CaptionGeminiRequest,+ CaptionPlanningRun,+} from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function fieldInstruction(field: CaptionFieldDeclaration): string {+ const value = field.value.kind === "boolean"+ ? 'the string "true"; omit the assignment when the field does not apply'+ : field.value.kind === "enum"+ ? `one of ${field.value.values.map((item) => JSON.stringify(item)).join(", ")}`+ : "a finite base-10 number encoded as a string";+ return `${field.id}: ${field.minimumPerCue}..${field.maximumPerCue} atom assignments per cue; value=${value}. ${field.instruction}`;+}++export function captionGeminiSystemInstruction(runs: readonly CaptionPlanningRun[]): string {+ const styleRules = new Map<string, CaptionPlanningRun>();+ runs.forEach((run) => styleRules.set(run.styleId, run));+ return [+ "You are a caption cue and per-word attribute planner. Display atoms are immutable author truth.",+ "Return only response-schema JSON. Never return corrected text, replacement text, pronunciation text, timestamps, Markdown, or explanations.",+ "Each run is a hard boundary. Plan every run independently; a cue may never cross a run boundary.",+ "Within each run, cues must preserve and exactly partition atom_ids. Return only each cue's final after_atom_id; the final cue must end at the run's final atom.",+ "Every field assignment targets exactly one atom inside its cue and uses a field declared by that run's style.",+ ...[...styleRules.values()].flatMap((run) => [+ `Style ${run.styleId} cue judgment: ${run.cueInstruction.trim()}`,+ ...run.fields.map((field) => `Style ${run.styleId} field ${fieldInstruction(field)}`),+ ]),+ "Echo every run exactly once. Use an empty fields array when no declaration applies. Never invent ids.",+ ].join("\n");+}++function promptContent(request: Pick<CaptionGeminiRequest, "atoms" | "runs">): string {+ return JSON.stringify({+ display_atoms: request.atoms,+ runs: request.runs.map((run) => ({+ run_id: run.id,+ style_id: run.styleId,+ atom_ids: run.atomIds,+ cue_instruction: run.cueInstruction,+ field_declarations: run.fields,+ })),+ }, null, 2);+}++function requestContent(value: Omit<CaptionGeminiRequest, "requestDigest">) {+ return {+ contract: "svml.caption-gemini-request@1" as const,+ model: value.model,+ narrativeDigest: value.narrativeDigest,+ captionProgramDigest: value.captionProgramDigest,+ atoms: value.atoms.map((atom) => ({ ...atom })),+ runs: value.runs.map((run) => ({+ id: run.id,+ styleId: run.styleId,+ atomIds: [...run.atomIds],+ cueInstruction: run.cueInstruction.trim(),+ fields: run.fields.map((field) => structuredClone(field)),+ })),+ systemInstruction: value.systemInstruction,+ prompt: value.prompt,+ temperature: 0.2 as const,+ };+}++export function compileCaptionGeminiRequest(+ narrative: Narrative,+ captionProgram: CaptionProgram,+ program: CaptionGeminiProgram,+): CaptionGeminiRequest {+ assertCaptionProgramForNarrative(captionProgram, narrative);+ verifyCaptionGeminiProgram(program);+ const styles = new Map(captionProgram.styles.map((style) => [style.id, style]));+ const runs = captionProgram.runs.map((run) => {+ const style = styles.get(run.styleId);+ assert(style !== undefined, `Caption run ${run.id} references an unknown Style`);+ return {+ id: run.id,+ styleId: run.styleId,+ atomIds: [...run.atomIds],+ cueInstruction: style.planning.cueInstruction,+ fields: style.planning.fields.map((field) => structuredClone(field)),+ };+ });+ const base = {+ contract: "svml.caption-gemini-request@1" as const,+ model: program.model,+ narrativeDigest: captionProgram.narrativeDigest,+ captionProgramDigest: captionProgram.digest,+ atoms: captionProgram.atoms.map((atom) => ({ id: atom.id, text: atom.text })),+ runs,+ };+ const systemInstruction = captionGeminiSystemInstruction(runs);+ const prompt = promptContent(base);+ const content = requestContent({ ...base, systemInstruction, prompt, temperature: 0.2 });+ const request = { ...content, requestDigest: digestOf(content) };+ verifyCaptionGeminiRequest(request);+ return request;+}++export function verifyCaptionGeminiRequest(value: unknown): asserts value is CaptionGeminiRequest {+ assert(value !== null && typeof value === "object" && !Array.isArray(value), "Caption Gemini request must be an object");+ const request = value as CaptionGeminiRequest;+ assert(request.contract === "svml.caption-gemini-request@1", "Caption Gemini request contract is invalid");+ assert(request.model === "gemini-2.5-flash" || request.model === "gemini-3.1-pro-preview", "Caption Gemini request model is unsupported");+ assert(isDigest(request.narrativeDigest) && isDigest(request.captionProgramDigest) && isDigest(request.requestDigest),+ "Caption Gemini request identity is invalid");+ assert(request.temperature === 0.2, "Caption Gemini temperature is not the package-owned value");+ assert(Array.isArray(request.atoms) && request.atoms.length > 0, "Caption Gemini request atoms are empty");+ const atomIds = request.atoms.map((atom) => atom.id);+ assert(atomIds.every((id) => typeof id === "string" && id.length > 0) && new Set(atomIds).size === atomIds.length,+ "Caption Gemini atom ids are invalid or repeated");+ assert(request.atoms.every((atom) => typeof atom.text === "string" && atom.text.length > 0),+ "Caption Gemini display atom text is invalid");+ const atomSet = new Set(atomIds);+ const planned = request.runs.flatMap((run) => run.atomIds);+ assert(request.runs.length > 0 && planned.length === atomIds.length && new Set(planned).size === atomIds.length+ && planned.every((id) => atomSet.has(id)), "Caption Gemini runs are not an exact display-atom partition");+ assert(request.runs.every((run) => run.atomIds.length > 0 && run.cueInstruction.trim().length > 0),+ "Caption Gemini request contains an empty run");+ assert(request.systemInstruction === captionGeminiSystemInstruction(request.runs),+ "Caption Gemini system instruction differs from the model package");+ assert(request.prompt === promptContent(request), "Caption Gemini prompt differs from immutable display facts");+ const { requestDigest: _digest, ...withoutDigest } = request;+ assert(request.requestDigest === digestOf(requestContent(withoutDigest)), "Caption Gemini request digest does not match its contents");+}diff --git a/packages/caption-gemini/src/surface.ts b/packages/caption-gemini/src/surface.tsnew file mode 100644index 00000000..3f21d021--- /dev/null+++ b/packages/caption-gemini/src/surface.ts@@ -0,0 +1,68 @@+import { captionTypes } from "@svml/caption";+import { contractTypes } from "@svml/contracts";+import type {+ StructuredSurfaceHandler,+ SurfaceResolvedReference,+ TextAttributeValue,+} from "@svml/text";++import { captionGeminiPlanningFragment } from "./fragment.js";+import { captionGeminiTypes } from "./manifest.js";+import { sealCaptionGeminiProgram } from "./program.js";+import type { CaptionGeminiModel } from "./types.js";++function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;+}++function reference(+ element: Parameters<StructuredSurfaceHandler>[0]["element"],+ name: string,+ expected: SurfaceResolvedReference["type"],+ resolve: (path: string) => SurfaceResolvedReference | undefined,+): SurfaceResolvedReference {+ const raw: TextAttributeValue | undefined = element.attributes[name];+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a whole-value reference`);+ const value = resolve(raw.path);+ if (value === undefined || !sameType(value.type, expected)) throw new Error(`${element.name}.${name} cannot resolve the required type`);+ return value;+}++export const decodeCaptionGeminiPlannerSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ const required = ["id", "narrative", "program", "model"];+ if (required.some((name) => element.attributes[name] === undefined)+ || Object.keys(element.attributes).some((name) => !required.includes(name))) {+ throw new Error(`${element.name} requires ${required.join(", ")}`);+ }+ if (element.children.some((child) => child.kind === "element" || child.value.trim())) {+ throw new Error(`${element.name} does not accept children`);+ }+ const id = element.attributes.id;+ const model = element.attributes.model;+ if (typeof id !== "string" || !id.trim()) throw new Error(`${element.name}.id must be a non-empty string`);+ if (model !== "gemini-2.5-flash" && model !== "gemini-3.1-pro-preview") {+ throw new Error(`${element.name}.model is unsupported`);+ }+ const narrative = reference(element, "narrative", contractTypes.narrative, resolveReference);+ const captionProgram = reference(element, "program", captionTypes.program, resolveReference);+ const optionsId = `${id}.gemini`;+ const options = sealCaptionGeminiProgram({+ contract: "svml.caption-gemini-program@1",+ model: model as CaptionGeminiModel,+ });+ return {+ records: [{ id: optionsId, type: captionGeminiTypes.program, value: { kind: "inline", value: options }, range: element.range }],+ components: [{+ id,+ fragment: captionGeminiPlanningFragment.id,+ inputs: {+ narrative: narrative.ref,+ captionProgram: captionProgram.ref,+ program: { kind: "record", id: optionsId },+ },+ outputs: { plan: `${id}.plan` },+ range: element.range,+ }],+ fragments: [captionGeminiPlanningFragment],+ };+};diff --git a/packages/caption-gemini/src/types.ts b/packages/caption-gemini/src/types.tsnew file mode 100644index 00000000..fdc13a3c--- /dev/null+++ b/packages/caption-gemini/src/types.ts@@ -0,0 +1,69 @@+import type {+ CaptionFieldAssignment,+ CaptionFieldDeclaration,+ CaptionFieldValueSchema,+ CaptionPlan,+ CaptionPlannedCue,+ CaptionPlannedRun,+ CaptionProgram,+} from "@svml/caption";+import type { Digest } from "@svml/protocol";++export type {+ CaptionFieldAssignment,+ CaptionFieldDeclaration,+ CaptionFieldValueSchema,+ CaptionPlannedCue,+ CaptionPlannedRun,+ CaptionProgram,+};+export type CaptionGeminiPlan = CaptionPlan;++export type CaptionGeminiModel = "gemini-2.5-flash" | "gemini-3.1-pro-preview";++/** Explicit model choice. Cue and field meaning comes from the Caption Program, not the Provider. */+export type CaptionGeminiProgram = {+ readonly contract: "svml.caption-gemini-program@1";+ readonly model: CaptionGeminiModel;+ readonly programDigest: Digest;+};++export type CaptionPlanningAtom = {+ readonly id: string;+ readonly text: string;+};++export type CaptionPlanningRun = {+ readonly id: string;+ readonly styleId: string;+ readonly atomIds: readonly string[];+ readonly cueInstruction: string;+ readonly fields: readonly CaptionFieldDeclaration[];+};++export type CaptionGeminiRequest = {+ readonly contract: "svml.caption-gemini-request@1";+ readonly model: CaptionGeminiModel;+ readonly narrativeDigest: Digest;+ readonly captionProgramDigest: Digest;+ readonly atoms: readonly CaptionPlanningAtom[];+ readonly runs: readonly CaptionPlanningRun[];+ readonly systemInstruction: string;+ readonly prompt: string;+ readonly temperature: 0.2;+ readonly requestDigest: Digest;+};++export type RawCaptionGeminiResponse = {+ readonly runs: readonly {+ readonly run_id: string;+ readonly cues: readonly {+ readonly after_atom_id: string;+ readonly fields: readonly {+ readonly declaration_id: string;+ readonly atom_id: string;+ readonly value: string;+ }[];+ }[];+ }[];+};diff --git a/packages/caption-gemini/test/planning.test.ts b/packages/caption-gemini/test/planning.test.tsnew file mode 100644index 00000000..3c02650e--- /dev/null+++ b/packages/caption-gemini/test/planning.test.ts@@ -0,0 +1,173 @@+import assert from "node:assert/strict";+import test from "node:test";++import {+ defaultCaptionTrackProgram,+ resolveCaptionProgram,+ sealCaptionProgram,+ sealCaptionStyle,+} from "@svml/caption";+import type { CaptionStyleIntent } from "@svml/caption";+import {+ captionGeminiComponent,+ captionGeminiManifest,+ compileCaptionGeminiRequest,+ sealCaptionGeminiPlan,+ sealCaptionGeminiProgram,+} from "@svml/caption-gemini";+import type { CaptionGeminiRequest, RawCaptionGeminiResponse } from "@svml/caption-gemini";+import { parseScript } from "@svml/script";++function style(id: string, fields: CaptionStyleIntent["planning"]["fields"] = []) {+ const base = defaultCaptionTrackProgram(id);+ return sealCaptionStyle({+ contract: "svml.caption-style@1",+ id,+ planning: {+ cueInstruction: "Prefer short complete semantic phrases and never cross clause punctuation.",+ fields,+ },+ presentation: { mode: "whole", stackingOrder: base.stacking.order, style: base.style },+ });+}++function options() {+ return sealCaptionGeminiProgram({+ contract: "svml.caption-gemini-program@1",+ model: "gemini-2.5-flash",+ });+}++function validResponse(request: CaptionGeminiRequest): RawCaptionGeminiResponse {+ return {+ runs: request.runs.map((run) => ({+ run_id: run.id,+ cues: [{+ after_atom_id: run.atomIds.at(-1)!,+ fields: run.fields.length === 0 ? [] : [{+ declaration_id: run.fields[0]!.id,+ atom_id: run.atomIds[0]!,+ value: "true",+ }],+ }],+ })),+ };+}++test("Gemini sees only immutable display atoms while Role and roleless runs come from Caption Program", () => {+ const narrative = parseScript("roles.svml", `+ <opening>Meet <SVML | semantic video markup language>.</opening>+ <answer><ALICE>Meaning becomes the source.</answer>+ `);+ const normal = style("normal");+ const alice = style("alice", [{+ id: "important",+ value: { kind: "boolean" },+ instruction: "Select zero or one word whose emphasis best communicates this Cue.",+ minimumPerCue: 0,+ maximumPerCue: 1,+ }]);+ const program = resolveCaptionProgram(narrative, "captions", normal, [{+ id: "alice-use",+ selector: { kind: "role", role: "ALICE" },+ style: alice,+ }]);+ const request = compileCaptionGeminiRequest(narrative, program, options());++ assert.deepEqual(request.runs.map((run) => run.styleId), ["normal", "alice"]);+ assert.equal(request.atoms.some((atom) => atom.text === "SVML"), true);+ assert.equal(request.atoms.some((atom) => atom.text === "semantic"), false);+ assert.equal(request.prompt.includes("semantic video markup language"), false);+ assert.equal(request.prompt.includes("pronunciation"), false);+ assert.equal("whisperx" in request, false);+ assert.equal("transcript" in request, false);++ const { digest: _digest, ...content } = program;+ const tampered = sealCaptionProgram({+ ...content,+ atoms: program.atoms.map((atom, index) => index === 0 ? { ...atom, text: "TAMPERED" } : atom),+ });+ assert.throws(() => compileCaptionGeminiRequest(narrative, tampered, options()),+ /display atoms differ from its Narrative/u);+});++test("Gemini may only cut each resolved run and assign declared attributes to atoms inside its Cue", () => {+ const narrative = parseScript("plan.svml", "<line><ALICE>Meaning becomes the source.</line>");+ const emphasis = style("emphasis", [{+ id: "important",+ value: { kind: "boolean" },+ instruction: "Select exactly one important word.",+ minimumPerCue: 1,+ maximumPerCue: 1,+ }]);+ const program = resolveCaptionProgram(narrative, "captions", emphasis, []);+ const request = compileCaptionGeminiRequest(narrative, program, options());+ const plan = sealCaptionGeminiPlan(request, validResponse(request));+ assert.equal(plan.captionProgramDigest, program.digest);+ assert.deepEqual(+ plan.runs.flatMap((run) => run.cues.flatMap((cue) => cue.atomIds)),+ request.runs.flatMap((run) => run.atomIds),+ );+ assert.equal(plan.runs[0]?.cues[0]?.fields[0]?.declarationId, "important");++ assert.throws(+ () => sealCaptionGeminiPlan(request, { ...validResponse(request), corrected_text: "tampered" }),+ /unknown or missing fields/u,+ );+ const outside = structuredClone(validResponse(request));+ const field = outside.runs[0]!.cues[0]!.fields[0]! as { atom_id: string };+ field.atom_id = "foreign";+ assert.throws(() => sealCaptionGeminiPlan(request, outside), /outside its cue/u);+});++test("one atom may carry multiple independent fields and fields need not form contiguous spans", () => {+ const narrative = parseScript("multi-field.svml", "<line>one two three four five.</line>");+ const multi = style("multi", [+ {+ id: "best",+ value: { kind: "boolean" },+ instruction: "Select up to two best words.",+ minimumPerCue: 0,+ maximumPerCue: 2,+ },+ {+ id: "medium",+ value: { kind: "boolean" },+ instruction: "Independently select up to two medium words.",+ minimumPerCue: 0,+ maximumPerCue: 2,+ },+ ]);+ const program = resolveCaptionProgram(narrative, "captions", multi, []);+ const request = compileCaptionGeminiRequest(narrative, program, options());+ const atoms = request.runs[0]!.atomIds;+ const plan = sealCaptionGeminiPlan(request, {+ runs: [{+ run_id: request.runs[0]!.id,+ cues: [{+ after_atom_id: atoms.at(-1)!,+ fields: [+ { declaration_id: "best", atom_id: atoms[0]!, value: "true" },+ { declaration_id: "best", atom_id: atoms[2]!, value: "true" },+ { declaration_id: "medium", atom_id: atoms[1]!, value: "true" },+ { declaration_id: "medium", atom_id: atoms[2]!, value: "true" },+ ],+ }],+ }],+ });+ const fields = plan.runs[0]!.cues[0]!.fields;+ assert.equal(fields.filter((field) => field.atomId === atoms[2]).length, 2);+ assert.deepEqual(fields.filter((field) => field.declarationId === "best").map((field) => field.atomId),+ [atoms[0], atoms[2]]);+});++test("the package enumerates every Manifest Producer and validator", () => {+ assert.deepEqual(+ captionGeminiComponent.producers?.map((facet) => [facet.producer.name, facet.implementationDigest]),+ captionGeminiManifest.producers.map((producer) => [producer.name, producer.implementation.digest]),+ );+ assert.deepEqual(+ captionGeminiComponent.validators?.map((facet) => [facet.type.name, facet.implementationDigest]),+ captionGeminiManifest.types.map((type) => [type.name, type.validator?.implementation.digest]),+ );+});diff --git a/packages/caption/README.md b/packages/caption/README.mdindex dc21afbb..17d77786 100644--- a/packages/caption/README.md+++ b/packages/caption/README.md@@ -1,24 +1,58 @@# `@svml/caption`-Pure composition between the common Narrative's untimed Caption Projection and a CompleteSemanticMap.-The authored display region receives the evidence-backed envelope of the speech tokens it owns.-Exact internal correspondences may reuse evidence timing; unresolved display words remain untimed-until a track explicitly chooses a local presentation policy.+Provider-neutral Caption planning, timing projection and lowering to an ordinary peer+`VisualTrack`.-`planCaptionPresentation()` is that optional track-local policy. `whole` preserves each authored-region; `proportional-word` and `character-flow` create visibly labelled estimates. They never-write those estimates back into the global speech map or Caption Projection.+Caption has two independent inputs:-`captionComponent` exposes two enumerable deterministic Producers:+- `Narrative.captionProjection` is the immutable display truth. Only the left side of Dual Text is+ visible here; `<SVML | semantic video markup language>` contributes the display atom `SVML`, not+ its pronunciation words.+- `CaptionProgram` is the complete author intent for planning and presentation. It assigns exactly+ one full `CaptionStyle` to every visible display atom.-- `temporalize-caption`: `Narrative + CompleteSemanticMap -> TimedCaptionProjection`;-- `render-caption-track`: `TimedCaptionProjection + CaptionTrackProgram -> VisualTrack`.+A Style contains both its planning requirements and its complete visual appearance. The Program+names one default Style and then applies ordered whole-Style replacements by Role or explicit+Script Selection. Later matching applications win. Consequently roleless prose and every+unmentioned word receive the default automatically, while changing one sentence needs only one+Selection and one `Use`; the author never writes the complement.-It also owns and validates both Caption-specific Types. Their validator identities and both-Producer implementations are checked against `captionManifest` and enter the installed package-lock. The package has no Provider, LLM, queue, credential or Core authority.+```xml+<caption:Style id="plain" appearance={studio.caption.plain}>+ <caption:Cues>Use complete phrases of two to five words.</caption:Cues>+</caption:Style>-The current `CaptionTrackProgram` is an executable vertical-slice program, not the frozen public-style language. Exact font Artifacts, the complete production positioning/style matrix, role-overrides, package-owned SVS Recipes, Gemini cue grouping and the final author Surface require a-separate capability audit before compatibility freeze.+<caption:Style id="impact" appearance={studio.caption.impact}>+ <caption:Cues>Use complete phrases of two to seven words.</caption:Cues>+ <caption:Field id="important" type="boolean" min-per-cue="0" max-per-cue="2">+ Select at most two words whose emphasis best communicates this Cue.+ </caption:Field>+</caption:Style>++<caption:Program id="captions" narrative={story} default={plain}>+ <caption:Use role="ALICE" style={impact}/>+ <caption:Use on={story.selection.special} style={plain}/>+</caption:Program>+```++`Use` order is semantic. The second rule above replaces the entire Style for `special`; it does not+merge arbitrary fields from two styles. A Role Cue is optional in Script, and Role state never+crosses a Segment boundary.++The planner-neutral `CaptionPlan` has only two freedoms:++1. partition each already-resolved Program run into ordered Cues;+2. assign zero, one or more declared attributes to each display atom.++It cannot rewrite text, change Style assignment or invent time. `temporalize-caption-plan` later+joins those atom identities with the independent `CompleteSemanticMap`. Exact display/speech+correspondence reuses measured timing. A display alias without word-level audio evidence receives+an explicitly estimated local projection and never contaminates the global speech map.++The final `render-caption-program` Producer lowers the timed result and each selected full Style to+one self-contained `VisualTrack`. Caption is not a privileged Composition layer: Film consumes it+exactly like Speech, B-roll or Text Track output.++The package owns deterministic Types, validators and Producers only. It contains no LLM, Provider,+credential, queue or Core authority. A planning package such as `@svml/caption-gemini` may fulfill+the narrow `CaptionPlan` contract; presentation and timing remain here.diff --git a/packages/caption/package.json b/packages/caption/package.jsonindex c8d509a3..e3f6747e 100644--- a/packages/caption/package.json+++ b/packages/caption/package.json@@ -9,7 +9,10 @@"dependencies": {"@svml/component-kit": "workspace:*","@svml/contracts": "workspace:*",- "@svml/protocol": "workspace:*"+ "@svml/elaborator": "workspace:*",+ "@svml/protocol": "workspace:*",+ "@svml/svs": "workspace:*",+ "@svml/text": "workspace:*"},"devDependencies": {"@svml/script": "workspace:*",diff --git a/packages/caption/src/component.ts b/packages/caption/src/component.tsindex eda6cd21..bfef4207 100644--- a/packages/caption/src/component.ts+++ b/packages/caption/src/component.ts@@ -8,18 +8,26 @@ import { canonicalize } from "@svml/protocol";import {captionImplementationDigest,+ captionPlanImplementationDigest,captionProducers,captionTypes,captionValidatorDigests,} from "./manifest.js";-import { temporalizeCaption } from "./temporalize.js";+import { assertCaptionPlan } from "./plan.js";+import { assertCaptionProgram, assertCaptionStyle } from "./style.js";+import { temporalizeCaption, temporalizeCaptionPlan } from "./temporalize.js";import {assertCaptionTrackProgram,assertTimedCaptionProjection,+ renderCaptionProgram,renderCaptionTrack,+ renderCaptionProgramImplementationDigest,renderCaptionTrackImplementationDigest,} from "./track.js";import type {+ CaptionPlan,+ CaptionProgram,+ CaptionStyleIntent,CaptionTrackProgram,TimedCaptionProjection,} from "./types.js";@@ -49,6 +57,40 @@ export const captionComponent = {needs: {},}),},+ {+ producer: captionProducers.temporalizePlan,+ implementationDigest: captionPlanImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: {+ caption: {+ kind: "inline",+ value: canonicalize(temporalizeCaptionPlan(+ inline<Narrative>(inputs.narrative?.value, "Narrative"),+ inline<CompleteSemanticMap>(inputs.map?.value, "CompleteSemanticMap"),+ inline<CaptionProgram>(inputs.program?.value, "CaptionProgram"),+ inline<CaptionPlan>(inputs.plan?.value, "CaptionPlan"),+ )),+ },+ },+ needs: {},+ }),+ },+ {+ producer: captionProducers.renderProgram,+ implementationDigest: renderCaptionProgramImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: {+ track: {+ kind: "inline",+ value: canonicalize(renderCaptionProgram(+ inline<TimedCaptionProjection>(inputs.caption?.value, "TimedCaptionProjection"),+ inline<CaptionProgram>(inputs.program?.value, "CaptionProgram"),+ )),+ },+ },+ needs: {},+ }),+ },{producer: captionProducers.renderTrack,implementationDigest: renderCaptionTrackImplementationDigest,@@ -67,6 +109,21 @@ export const captionComponent = {},],validators: [+ {+ type: captionTypes.style,+ implementationDigest: captionValidatorDigests.style,+ handler: ({ value }) => assertCaptionStyle(inline<CaptionStyleIntent>(value, "CaptionStyle")),+ },+ {+ type: captionTypes.program,+ implementationDigest: captionValidatorDigests.program,+ handler: ({ value }) => assertCaptionProgram(inline<CaptionProgram>(value, "CaptionProgram")),+ },+ {+ type: captionTypes.plan,+ implementationDigest: captionValidatorDigests.plan,+ handler: ({ value }) => assertCaptionPlan(inline(value, "CaptionPlan")),+ },{type: captionTypes.timedProjection,implementationDigest: captionValidatorDigests.timedProjection,diff --git a/packages/caption/src/display.ts b/packages/caption/src/display.tsnew file mode 100644index 00000000..d02c635e--- /dev/null+++ b/packages/caption/src/display.ts@@ -0,0 +1,106 @@+import type {+ CaptionRegion,+ Narrative,+ NarrativeSelectionRef,+} from "@svml/contracts";++import type { CaptionDisplayAtom } from "./types.js";++const WORD =+ /[\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}]\p{M}*|[\p{L}\p{M}\p{N}]+(?:['’.-][\p{L}\p{M}\p{N}]+)*/gu;++function displayWords(value: string): Array<{ readonly text: string; readonly start: number; readonly end: number }> {+ const result: Array<{ text: string; start: number; end: number }> = [];+ WORD.lastIndex = 0;+ let match: RegExpExecArray | null;+ while ((match = WORD.exec(value))) {+ result.push({ text: match[0], start: match.index, end: match.index + match[0].length });+ }+ return result;+}++function turnForRegion(narrative: Narrative, region: CaptionRegion): Narrative["turns"][number] {+ const turn = narrative.turns.find((candidate) =>+ candidate.segmentId === region.segmentId+ && candidate.tokenStart < region.endTokenExclusive+ && candidate.tokenEndExclusive > region.startToken);+ if (turn === undefined) throw new Error(`Caption region ${region.id} is not owned by a Narrative Turn`);+ return turn;+}++/** The only word universe visible to Caption planning. Speech-side alias text never enters it. */+export function captionDisplayAtoms(narrative: Narrative): CaptionDisplayAtom[] {+ const atoms: CaptionDisplayAtom[] = [];+ for (const region of narrative.captionProjection.regions) {+ if (region.kind === "hidden") continue;+ const turn = turnForRegion(narrative, region);+ for (const word of displayWords(region.display)) {+ const exact = region.refinements.find((refinement) =>+ refinement.displayStart === word.start && refinement.displayEnd === word.end);+ atoms.push({+ id: `${region.id}:display:${atoms.length + 1}`,+ index: atoms.length,+ regionId: region.id,+ segmentId: region.segmentId,+ turnId: turn.id,+ ...(turn.role === undefined ? {} : { role: turn.role }),+ text: word.text,+ displayStart: word.start,+ displayEnd: word.end,+ sourceTokenStart: exact?.startToken ?? region.startToken,+ sourceTokenEndExclusive: exact?.endTokenExclusive ?? region.endTokenExclusive,+ correspondence: exact === undefined ? "region-envelope" : "exact",+ });+ }+ }+ if (atoms.length === 0) throw new Error("Caption display projection contains no visible words");+ return atoms;+}++function occurrenceRange(+ occurrence: NarrativeSelectionRef["occurrences"][number],+ tokenCount: number,+): { readonly start: number; readonly endExclusive: number } {+ const start = occurrence.open.affinity === "left"+ ? Math.max(0, occurrence.open.boundary.tokenIndex - 1)+ : occurrence.open.boundary.tokenIndex;+ const endExclusive = occurrence.close.affinity === "right"+ ? Math.min(tokenCount, occurrence.close.boundary.tokenIndex + 1)+ : occurrence.close.boundary.tokenIndex;+ return { start, endExclusive };+}++export function displayAtomMatchesSelection(+ atom: CaptionDisplayAtom,+ selection: NarrativeSelectionRef,+ tokenCount: number,+): boolean {+ return selection.occurrences.some((occurrence) => {+ const range = occurrenceRange(occurrence, tokenCount);+ return atom.sourceTokenStart < range.endExclusive && atom.sourceTokenEndExclusive > range.start;+ });+}++export function displayTextForAtoms(+ narrative: Narrative,+ selected: readonly CaptionDisplayAtom[],+): string {+ const byRegion = new Map<string, CaptionDisplayAtom[]>();+ for (const atom of selected) {+ const values = byRegion.get(atom.regionId) ?? [];+ values.push(atom);+ byRegion.set(atom.regionId, values);+ }+ const parts: string[] = [];+ for (const region of narrative.captionProjection.regions) {+ const atoms = byRegion.get(region.id);+ if (atoms === undefined || atoms.length === 0) continue;+ const ordered = [...atoms].sort((left, right) => left.displayStart - right.displayStart);+ const all = displayWords(region.display);+ const first = ordered[0]!;+ const last = ordered.at(-1)!;+ const after = all.find((word) => word.start > last.displayStart);+ parts.push(region.display.slice(first.displayStart, after?.start ?? region.display.length).trim());+ }+ return parts.join(" ").replace(/\s+/gu, " ").replace(/\s+([,.;:!?])/gu, "$1").trim();+}diff --git a/packages/caption/src/fragment.ts b/packages/caption/src/fragment.tsnew file mode 100644index 00000000..b169b337--- /dev/null+++ b/packages/caption/src/fragment.ts@@ -0,0 +1,72 @@+import { contractTypes } from "@svml/contracts";+import { sealGraphFragment } from "@svml/elaborator";++import { captionProducers, captionTypes } from "./manifest.js";++const input = (name: string) => ({ kind: "fragment-input" as const, name });+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });++/** Provider-free Caption base: authored display text + measured map -> peer VisualTrack. */+export const captionTrackSurfaceFragment = sealGraphFragment({+ name: "@svml/caption/track-surface@1",+ inputs: [+ { name: "narrative", type: contractTypes.narrative },+ { name: "map", type: contractTypes.completeSemanticMap },+ { name: "program", type: captionTypes.trackProgram },+ ],+ operations: [+ {+ id: "caption:temporalize",+ producer: captionProducers.temporalize,+ inputs: { narrative: input("narrative"), map: input("map") },+ result: { kind: "output", name: "caption" },+ },+ {+ id: "caption:render",+ producer: captionProducers.renderTrack,+ inputs: { caption: operation("caption:temporalize"), program: input("program") },+ result: { kind: "output", name: "track" },+ },+ ],+ exports: [{+ name: "track",+ type: contractTypes.visualTrack,+ root: operation("caption:render"),+ semanticInputs: ["narrative", "map", "program"],+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("map"), sourcePointer: "/programSpace/digest" }],+ fidelity: "exact",+ }],+});++/** Planner-neutral Cue/field facts join the independent measured map only at Caption lowering. */+export const plannedCaptionTrackSurfaceFragment = sealGraphFragment({+ name: "@svml/caption/planned-track-surface@2",+ inputs: [+ { name: "narrative", type: contractTypes.narrative },+ { name: "map", type: contractTypes.completeSemanticMap },+ { name: "plan", type: captionTypes.plan },+ { name: "program", type: captionTypes.program },+ ],+ operations: [+ {+ id: "caption:temporalize-plan",+ producer: captionProducers.temporalizePlan,+ inputs: { narrative: input("narrative"), map: input("map"), program: input("program"), plan: input("plan") },+ result: { kind: "output", name: "caption" },+ },+ {+ id: "caption:render-plan",+ producer: captionProducers.renderProgram,+ inputs: { caption: operation("caption:temporalize-plan"), program: input("program") },+ result: { kind: "output", name: "track" },+ },+ ],+ exports: [{+ name: "track",+ type: contractTypes.visualTrack,+ root: operation("caption:render-plan"),+ semanticInputs: ["narrative", "map", "plan", "program"],+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("map"), sourcePointer: "/programSpace/digest" }],+ fidelity: "exact",+ }],+});diff --git a/packages/caption/src/index.ts b/packages/caption/src/index.tsindex a269037f..1228d9ea 100644--- a/packages/caption/src/index.ts+++ b/packages/caption/src/index.ts@@ -1,7 +1,16 @@export { CaptionProjectionError } from "./error.js";+export { captionDisplayAtoms, displayAtomMatchesSelection, displayTextForAtoms } from "./display.js";export { captionComponent } from "./component.js";+export { captionTrackSurfaceFragment, plannedCaptionTrackSurfaceFragment } from "./fragment.js";export {captionImplementationDigest,+ captionProgramSchema,+ captionProgramSurfaceImplementationDigest,+ captionPlanImplementationDigest,+ captionPlanSchema,+ captionSurfaceImplementationDigest,+ captionStyleSchema,+ captionStyleSurfaceImplementationDigest,captionManifest,captionModuleRef,captionProducers,@@ -10,15 +19,32 @@ export {captionValidatorDigests,timedCaptionProjectionSchema,} from "./manifest.js";+export {+ decodeCaptionProgramSurface,+ decodeCaptionStyleSurface,+ decodeCaptionTrackSurface,+} from "./surface.js";export { planCaptionPresentation } from "./presentation.js";-export { temporalizeCaption } from "./temporalize.js";+export { assertCaptionPlan, assertCaptionPlanForProgram, sealCaptionPlan } from "./plan.js";+export {+ assertCaptionProgram,+ assertCaptionProgramForNarrative,+ assertCaptionStyle,+ resolveCaptionProgram,+ sealCaptionProgram,+ sealCaptionStyle,+} from "./style.js";+export type { CaptionStyleApplication, CaptionStyleSelector } from "./style.js";+export { temporalizeCaption, temporalizeCaptionPlan } from "./temporalize.js";export {assertCaptionTrackProgram,assertTimedCaptionProjection,computeCaptionTrackProgramDigest,defaultCaptionTrackProgram,renderCaptionTrack,+ renderCaptionProgram,renderCaptionTrackImplementationDigest,+ renderCaptionProgramImplementationDigest,sealCaptionTrackProgram,} from "./track.js";export type * from "./types.js";diff --git a/packages/caption/src/manifest.ts b/packages/caption/src/manifest.tsindex 41fbb961..a8f7c172 100644--- a/packages/caption/src/manifest.ts+++ b/packages/caption/src/manifest.ts@@ -6,22 +6,37 @@ import {import { digestOf } from "@svml/protocol";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";-import { renderCaptionTrackImplementationDigest } from "./track.js";+import {+ renderCaptionProgramImplementationDigest,+ renderCaptionTrackImplementationDigest,+} from "./track.js";export const captionModuleRef = { name: "@svml/caption", version: "0.0.0-dev" } as const;export const captionProducers = {temporalize: { module: captionModuleRef, name: "temporalize-caption" },+ temporalizePlan: { module: captionModuleRef, name: "temporalize-caption-plan" },+ renderProgram: { module: captionModuleRef, name: "render-caption-program" },renderTrack: { module: captionModuleRef, name: "render-caption-track" },} satisfies Record<string, ProducerRef>;export const captionTypes = {+ style: { module: captionModuleRef, name: "CaptionStyle" },+ program: { module: captionModuleRef, name: "CaptionProgram" },+ plan: { module: captionModuleRef, name: "CaptionPlan" },timedProjection: { module: captionModuleRef, name: "TimedCaptionProjection" },trackProgram: { module: captionModuleRef, name: "CaptionTrackProgram" },} satisfies Record<string, TypeRef>;export const captionImplementationDigest = digestOf("@svml/caption/temporalize@1");+export const captionPlanImplementationDigest = digestOf("@svml/caption/temporalize-plan@2");export const captionValidatorDigests = {- timedProjection: digestOf("@svml/caption/validate-timed-projection@1"),+ style: digestOf("@svml/caption/validate-style@2"),+ program: digestOf("@svml/caption/validate-program@2"),+ plan: digestOf("@svml/caption/validate-plan@2"),+ timedProjection: digestOf("@svml/caption/validate-timed-projection@2"),trackProgram: digestOf("@svml/caption/validate-track-program@1"),} as const;+export const captionSurfaceImplementationDigest = digestOf("@svml/caption/track-surface@2");+export const captionStyleSurfaceImplementationDigest = digestOf("@svml/caption/style-surface@1");+export const captionProgramSurfaceImplementationDigest = digestOf("@svml/caption/program-surface@1");const string = { kind: "string", minLength: 1 } as const;const number = { kind: "number", minimum: 0 } as const;@@ -46,6 +61,8 @@ const timedCaptionRefinement = object({});const timedCaptionRegion = object({id: { schema: string },+ runId: { schema: string, optional: true },+ styleId: { schema: string, optional: true },display: { schema: { kind: "string" } },segmentId: { schema: string },kind: { schema: { kind: "string", enum: ["identity", "alias", "hidden"] } },@@ -55,6 +72,34 @@ const timedCaptionRegion = object({startQuality: { schema: quality },endQuality: { schema: quality },refinements: { schema: { kind: "array", items: timedCaptionRefinement } },+ fields: { schema: { kind: "array", items: object({+ declarationId: { schema: string },+ atomId: { schema: string },+ value: { schema: string },+ }) }, optional: true },+});+const captionFieldAssignment = object({+ declarationId: { schema: string },+ atomId: { schema: string },+ value: { schema: string },+});+const captionPlannedCue = object({+ id: { schema: string },+ atomIds: { schema: { kind: "array", minItems: 1, items: string } },+ fields: { schema: { kind: "array", items: captionFieldAssignment } },+});+const captionPlannedRun = object({+ id: { schema: string },+ styleId: { schema: string },+ cues: { schema: { kind: "array", minItems: 1, items: captionPlannedCue } },+});+export const captionPlanSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.caption-plan@1" } },+ narrativeDigest: { schema: digest },+ captionProgramDigest: { schema: digest },+ planningRequestDigest: { schema: digest },+ runs: { schema: { kind: "array", minItems: 1, items: captionPlannedRun } },+ planDigest: { schema: digest },});export const timedCaptionProjectionSchema: ValueSchema = object({contract: { schema: { kind: "literal", value: "svml.timed-caption-projection@1" } },@@ -64,13 +109,7 @@ export const timedCaptionProjectionSchema: ValueSchema = object({projectionDigest: { schema: digest },});-export const captionTrackProgramSchema: ValueSchema = object({- contract: { schema: { kind: "literal", value: "svml.caption-track-program@1" } },- digest: { schema: digest },- id: { schema: string },- mode: { schema: { kind: "string", enum: ["whole", "proportional-word", "character-flow"] } },- stacking: { schema: object({ order: { schema: integer }, tieBreak: { schema: string } }) },- style: { schema: object({+const captionTrackStyleSchema = object({fontFamily: { schema: string },fontSizePx: { schema: number },fontWeight: { schema: number },@@ -81,8 +120,76 @@ export const captionTrackProgramSchema: ValueSchema = object({borderRadiusPx: { schema: number },bottomPercent: { schema: number },maxWidthPercent: { schema: number },+ leftPercent: { schema: number, optional: true },+ topPercent: { schema: number, optional: true },+ widthPercent: { schema: number, optional: true },+ lineHeight: { schema: number, optional: true },textAlign: { schema: { kind: "string", enum: ["left", "center", "right"] } },+ });++export const captionTrackProgramSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.caption-track-program@1" } },+ digest: { schema: digest },+ id: { schema: string },+ mode: { schema: { kind: "string", enum: ["whole", "proportional-word", "character-flow"] } },+ stacking: { schema: object({ order: { schema: integer }, tieBreak: { schema: string } }) },+ style: { schema: captionTrackStyleSchema },+});++const captionFieldValueSchema: ValueSchema = {+ kind: "oneOf",+ variants: [+ object({ kind: { schema: { kind: "literal", value: "boolean" } } }),+ object({+ kind: { schema: { kind: "literal", value: "enum" } },+ values: { schema: { kind: "array", minItems: 1, items: string } },+ }),+ object({+ kind: { schema: { kind: "literal", value: "number" } },+ minimum: { schema: { kind: "number" }, optional: true },+ maximum: { schema: { kind: "number" }, optional: true },+ }),+ ],+};+const captionFieldDeclaration = object({+ id: { schema: string },+ value: { schema: captionFieldValueSchema },+ instruction: { schema: string },+ minimumPerCue: { schema: integer },+ maximumPerCue: { schema: integer },+});+export const captionStyleSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.caption-style@1" } },+ id: { schema: string },+ planning: { schema: object({+ cueInstruction: { schema: string },+ fields: { schema: { kind: "array", items: captionFieldDeclaration } },+ }) },+ presentation: { schema: object({+ mode: { schema: { kind: "string", enum: ["whole", "proportional-word", "character-flow"] } },+ stackingOrder: { schema: integer },+ style: { schema: captionTrackStyleSchema },}) },+ digest: { schema: digest },+});+const captionDisplayAtom = object({+ id: { schema: string }, index: { schema: integer }, regionId: { schema: string }, segmentId: { schema: string },+ turnId: { schema: string }, role: { schema: string, optional: true }, text: { schema: string },+ displayStart: { schema: integer }, displayEnd: { schema: integer }, sourceTokenStart: { schema: integer },+ sourceTokenEndExclusive: { schema: integer },+ correspondence: { schema: { kind: "string", enum: ["exact", "region-envelope"] } },+});+const captionProgramRun = object({+ id: { schema: string }, styleId: { schema: string },+ atomIds: { schema: { kind: "array", minItems: 1, items: string } },+});+export const captionProgramSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.caption-program@1" } },+ id: { schema: string }, narrativeDigest: { schema: digest }, defaultStyleId: { schema: string },+ styles: { schema: { kind: "array", minItems: 1, items: captionStyleSchema } },+ atoms: { schema: { kind: "array", minItems: 1, items: captionDisplayAtom } },+ runs: { schema: { kind: "array", minItems: 1, items: captionProgramRun } },+ digest: { schema: digest },});export const captionManifest: ModuleManifest = {@@ -96,6 +203,32 @@ export const captionManifest: ModuleManifest = {videoContractDependencies.composition,],types: [+ {+ name: captionTypes.style.name,+ schema: captionStyleSchema,+ validator: { abi: "svml.type-validator@1", implementation: {+ kind: "registered", locator: "@svml/caption/validate-style", digest: captionValidatorDigests.style,+ } },+ },+ {+ name: captionTypes.program.name,+ schema: captionProgramSchema,+ validator: { abi: "svml.type-validator@1", implementation: {+ kind: "registered", locator: "@svml/caption/validate-program", digest: captionValidatorDigests.program,+ } },+ },+ {+ name: captionTypes.plan.name,+ schema: captionPlanSchema,+ validator: {+ abi: "svml.type-validator@1",+ implementation: {+ kind: "registered",+ locator: "@svml/caption/validate-plan",+ digest: captionValidatorDigests.plan,+ },+ },+ },{name: captionTypes.timedProjection.name,schema: timedCaptionProjectionSchema,@@ -122,7 +255,21 @@ export const captionManifest: ModuleManifest = {},],capabilities: [],- surfaces: [],+ surfaces: [+ {+ name: "style", tag: "Style", mode: "structured", outputs: [captionTypes.style],+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/caption/style-surface", digest: captionStyleSurfaceImplementationDigest },+ },+ {+ name: "program", tag: "Program", mode: "structured", outputs: [captionTypes.program],+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/caption/program-surface", digest: captionProgramSurfaceImplementationDigest },+ },+ {+ name: "track", tag: "Track", mode: "structured",+ outputs: [captionTypes.trackProgram, contractTypes.visualTrack],+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/caption/track-surface", digest: captionSurfaceImplementationDigest },+ },+ ],producers: [{name: captionProducers.temporalize.name,@@ -144,6 +291,41 @@ export const captionManifest: ModuleManifest = {digest: captionImplementationDigest,},},+ {+ name: captionProducers.temporalizePlan.name,+ inputs: [+ { name: "narrative", type: contractTypes.narrative },+ { name: "map", type: contractTypes.completeSemanticMap },+ { name: "program", type: captionTypes.program },+ { name: "plan", type: captionTypes.plan },+ ],+ outputs: [{+ name: "caption",+ type: captionTypes.timedProjection,+ affinity: [+ { resultPointer: "/programSpace/digest", input: "map", inputPointer: "/programSpace/digest" },+ ],+ }],+ needs: [],+ implementation: {+ kind: "registered",+ locator: "@svml/caption/temporalize-plan",+ digest: captionPlanImplementationDigest,+ },+ },+ {+ name: captionProducers.renderProgram.name,+ inputs: [+ { name: "caption", type: captionTypes.timedProjection },+ { name: "program", type: captionTypes.program },+ ],+ outputs: [{+ name: "track", type: contractTypes.visualTrack,+ affinity: [{ resultPointer: "/programSpaceDigest", input: "caption", inputPointer: "/programSpace/digest" }],+ }],+ needs: [],+ implementation: { kind: "registered", locator: "@svml/caption/render-program", digest: renderCaptionProgramImplementationDigest },+ },{name: captionProducers.renderTrack.name,inputs: [diff --git a/packages/caption/src/plan.ts b/packages/caption/src/plan.tsnew file mode 100644index 00000000..dfb09ddf--- /dev/null+++ b/packages/caption/src/plan.ts@@ -0,0 +1,118 @@+import { digestOf, isDigest } from "@svml/protocol";++import type { CaptionFieldDeclaration, CaptionPlan, CaptionProgram } from "./types.js";++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function content(value: Omit<CaptionPlan, "planDigest">) {+ return {+ contract: "svml.caption-plan@1" as const,+ narrativeDigest: value.narrativeDigest,+ captionProgramDigest: value.captionProgramDigest,+ planningRequestDigest: value.planningRequestDigest,+ runs: value.runs.map((run) => ({+ id: run.id,+ styleId: run.styleId,+ cues: run.cues.map((cue) => ({+ id: cue.id,+ atomIds: [...cue.atomIds],+ fields: cue.fields.map((field) => ({ ...field })),+ })),+ })),+ };+}++export function sealCaptionPlan(value: Omit<CaptionPlan, "planDigest">): CaptionPlan {+ const normalized = content(value);+ const plan = { ...normalized, planDigest: digestOf(normalized) };+ assertCaptionPlan(plan);+ return plan;+}++export function assertCaptionPlan(value: unknown): asserts value is CaptionPlan {+ assert(value !== null && typeof value === "object" && !Array.isArray(value), "CaptionPlan must be an object");+ const plan = value as CaptionPlan;+ assert(plan.contract === "svml.caption-plan@1", "Unsupported CaptionPlan contract");+ assert(isDigest(plan.narrativeDigest) && isDigest(plan.captionProgramDigest)+ && isDigest(plan.planningRequestDigest) && isDigest(plan.planDigest), "CaptionPlan identity is invalid");+ assert(Array.isArray(plan.runs) && plan.runs.length > 0, "CaptionPlan runs are empty");+ const runIds = new Set<string>();+ const atomIds = new Set<string>();+ const cueIds = new Set<string>();+ for (const run of plan.runs) {+ assert(run.id.length > 0 && !runIds.has(run.id) && run.styleId.length > 0,+ "CaptionPlan run id or style is invalid or repeated");+ runIds.add(run.id);+ assert(run.cues.length > 0, `CaptionPlan run ${run.id} has no Cues`);+ for (const cue of run.cues) {+ assert(cue.id.length > 0 && !cueIds.has(cue.id) && cue.atomIds.length > 0,+ `CaptionPlan run ${run.id} contains an empty or repeated Cue`);+ cueIds.add(cue.id);+ assert(new Set(cue.atomIds).size === cue.atomIds.length, `CaptionPlan Cue ${cue.id} repeats a display atom`);+ for (const atomId of cue.atomIds) {+ assert(!atomIds.has(atomId), `CaptionPlan repeats display atom ${atomId}`);+ atomIds.add(atomId);+ }+ for (const field of cue.fields) {+ assert(field.declarationId.length > 0 && field.atomId.length > 0 && field.value.length > 0,+ `CaptionPlan Cue ${cue.id} field is invalid`);+ assert(cue.atomIds.includes(field.atomId), `CaptionPlan Cue ${cue.id} field lies outside its Cue`);+ }+ }+ }+ const { planDigest: _digest, ...withoutDigest } = plan;+ assert(plan.planDigest === digestOf(content(withoutDigest)), "CaptionPlan digest does not match its contents");+}++function assertFieldValue(declaration: CaptionFieldDeclaration, value: string): void {+ if (declaration.value.kind === "boolean") {+ assert(value === "true", `Caption field ${declaration.id} is sparse and only accepts true`);+ } else if (declaration.value.kind === "enum") {+ assert(declaration.value.values.includes(value), `Caption field ${declaration.id} enum value is invalid`);+ } else {+ const numeric = Number(value);+ assert(Number.isFinite(numeric), `Caption field ${declaration.id} number value is invalid`);+ assert(declaration.value.minimum === undefined || numeric >= declaration.value.minimum,+ `Caption field ${declaration.id} number is below its minimum`);+ assert(declaration.value.maximum === undefined || numeric <= declaration.value.maximum,+ `Caption field ${declaration.id} number is above its maximum`);+ }+}++/** Validate planner freedom against one resolved Program; no Provider-specific assumptions. */+export function assertCaptionPlanForProgram(plan: CaptionPlan, program: CaptionProgram): void {+ assertCaptionPlan(plan);+ assert(plan.narrativeDigest === program.narrativeDigest && plan.captionProgramDigest === program.digest,+ "CaptionPlan belongs to another Caption Program");+ assert(plan.runs.length === program.runs.length, "CaptionPlan does not contain the exact Program run set");+ const styles = new Map(program.styles.map((style) => [style.id, style]));+ for (let index = 0; index < program.runs.length; index += 1) {+ const expected = program.runs[index]!;+ const actual = plan.runs[index]!;+ assert(actual.id === expected.id && actual.styleId === expected.styleId,+ `CaptionPlan run ${actual.id} differs from Program order or Style`);+ assert(actual.cues.flatMap((cue) => cue.atomIds).join("\0") === expected.atomIds.join("\0"),+ `CaptionPlan run ${actual.id} does not partition its Program atoms exactly in order`);+ const style = styles.get(expected.styleId);+ assert(style !== undefined, `Caption Program Style ${expected.styleId} is absent`);+ const declarations = new Map(style.planning.fields.map((field) => [field.id, field]));+ for (const cue of actual.cues) {+ const unique = new Set<string>();+ for (const field of cue.fields) {+ const declaration = declarations.get(field.declarationId);+ assert(declaration !== undefined, `Caption field ${field.declarationId} is not declared by Style ${style.id}`);+ const key = `${field.declarationId}\0${field.atomId}`;+ assert(!unique.has(key), `Caption Cue ${cue.id} repeats field ${field.declarationId} on ${field.atomId}`);+ unique.add(key);+ assertFieldValue(declaration, field.value);+ }+ for (const declaration of declarations.values()) {+ const count = cue.fields.filter((field) => field.declarationId === declaration.id).length;+ assert(count >= declaration.minimumPerCue && count <= declaration.maximumPerCue,+ `Caption Cue ${cue.id} field ${declaration.id} violates its per-Cue cardinality`);+ }+ }+ }+}diff --git a/packages/caption/src/presentation.ts b/packages/caption/src/presentation.tsindex 62cfb0a6..f6976555 100644--- a/packages/caption/src/presentation.ts+++ b/packages/caption/src/presentation.ts@@ -31,6 +31,8 @@ function wholeRegion(region: TimedCaptionRegion): CaptionPresentationUnit[] {return [{id: `${region.id}:whole`,regionId: region.id,+ ...(region.runId === undefined ? {} : { runId: region.runId }),+ ...(region.styleId === undefined ? {} : { styleId: region.styleId }),display: region.display,displayStart: 0,displayEnd: region.display.length,@@ -38,6 +40,7 @@ function wholeRegion(region: TimedCaptionRegion): CaptionPresentationUnit[] {endSec: region.endSec,basis: "region-envelope",timingQuality: region.startQuality === "estimated" || region.endQuality === "estimated" ? "estimated" : "derived",+ ...(region.fields === undefined ? {} : { fields: region.fields.map((field) => ({ ...field })) }),}];}@@ -50,6 +53,8 @@ function proportionalWords(region: TimedCaptionRegion): CaptionPresentationUnit[return refinement ? {id: `${region.id}:word:${index + 1}`,regionId: region.id,+ ...(region.runId === undefined ? {} : { runId: region.runId }),+ ...(region.styleId === undefined ? {} : { styleId: region.styleId }),display: word.display,displayStart: word.start,displayEnd: word.end,@@ -83,6 +88,8 @@ function proportionalWords(region: TimedCaptionRegion): CaptionPresentationUnit[units[index] = {id: `${region.id}:word:${index + 1}`,regionId: region.id,+ ...(region.runId === undefined ? {} : { runId: region.runId }),+ ...(region.styleId === undefined ? {} : { styleId: region.styleId }),display: word.display,displayStart: word.start,displayEnd: word.end,@@ -109,6 +116,8 @@ function characterFlow(region: TimedCaptionRegion): CaptionPresentationUnit[] {units.push({id: `${region.id}:character:${index + 1}`,regionId: region.id,+ ...(region.runId === undefined ? {} : { runId: region.runId }),+ ...(region.styleId === undefined ? {} : { styleId: region.styleId }),display: character,displayStart: start,displayEnd: sourceOffset,diff --git a/packages/caption/src/style.ts b/packages/caption/src/style.tsnew file mode 100644index 00000000..ccd38750--- /dev/null+++ b/packages/caption/src/style.ts@@ -0,0 +1,239 @@+import type { Narrative, NarrativeSelectionRef } from "@svml/contracts";+import { canonicalize, digestOf, isDigest } from "@svml/protocol";++import { captionDisplayAtoms, displayAtomMatchesSelection } from "./display.js";+import type {+ CaptionDisplayAtom,+ CaptionFieldDeclaration,+ CaptionProgram,+ CaptionStyleIntent,+} from "./types.js";++export type CaptionStyleSelector =+ | { readonly kind: "role"; readonly role: string }+ | { readonly kind: "selection"; readonly selection: NarrativeSelectionRef };++export type CaptionStyleApplication = {+ readonly id: string;+ readonly selector: CaptionStyleSelector;+ readonly style: CaptionStyleIntent;+};++const ID = /^[A-Za-z][A-Za-z0-9_.-]{0,127}$/u;++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function normalizedField(field: CaptionFieldDeclaration): CaptionFieldDeclaration {+ return {+ id: field.id,+ value: field.value.kind === "enum"+ ? { kind: "enum", values: [...field.value.values] }+ : field.value.kind === "number"+ ? {+ kind: "number",+ ...(field.value.minimum === undefined ? {} : { minimum: field.value.minimum }),+ ...(field.value.maximum === undefined ? {} : { maximum: field.value.maximum }),+ }+ : { kind: "boolean" },+ instruction: field.instruction.trim(),+ minimumPerCue: field.minimumPerCue,+ maximumPerCue: field.maximumPerCue,+ };+}++function styleContent(value: Omit<CaptionStyleIntent, "digest">) {+ return canonicalize({+ contract: "svml.caption-style@1",+ id: value.id,+ planning: {+ cueInstruction: value.planning.cueInstruction.trim(),+ fields: value.planning.fields.map(normalizedField),+ },+ presentation: canonicalize(value.presentation),+ }) as unknown as Omit<CaptionStyleIntent, "digest">;+}++export function sealCaptionStyle(value: Omit<CaptionStyleIntent, "digest">): CaptionStyleIntent {+ const content = styleContent(value);+ const result = { ...content, digest: digestOf(content) };+ assertCaptionStyle(result);+ return result;+}++export function assertCaptionStyle(value: CaptionStyleIntent): void {+ assert(value.contract === "svml.caption-style@1" && ID.test(value.id), "Caption Style identity is invalid");+ assert(value.planning.cueInstruction.length > 0, `Caption Style ${value.id} Cue instruction is empty`);+ const fields = new Set<string>();+ for (const field of value.planning.fields) {+ assert(ID.test(field.id) && !fields.has(field.id), `Caption Style ${value.id} field id is invalid or repeated`);+ fields.add(field.id);+ assert(field.instruction.trim().length > 0, `Caption Style ${value.id} field instruction is empty`);+ assert(Number.isSafeInteger(field.minimumPerCue) && field.minimumPerCue >= 0,+ `Caption Style ${value.id} field minimum is invalid`);+ assert(Number.isSafeInteger(field.maximumPerCue) && field.maximumPerCue >= field.minimumPerCue,+ `Caption Style ${value.id} field maximum is invalid`);+ if (field.value.kind === "enum") {+ assert(field.value.values.length > 0 && new Set(field.value.values).size === field.value.values.length,+ `Caption Style ${value.id} field enum is invalid`);+ } else if (field.value.kind === "number") {+ assert(field.value.minimum === undefined || Number.isFinite(field.value.minimum),+ `Caption Style ${value.id} field numeric minimum is invalid`);+ assert(field.value.maximum === undefined || Number.isFinite(field.value.maximum),+ `Caption Style ${value.id} field numeric maximum is invalid`);+ }+ }+ assert(Number.isSafeInteger(value.presentation.stackingOrder) && value.presentation.stackingOrder >= 0,+ `Caption Style ${value.id} stacking order is invalid`);+ assert(["whole", "proportional-word", "character-flow"].includes(value.presentation.mode),+ `Caption Style ${value.id} presentation mode is invalid`);+ const appearance = value.presentation.style;+ assert(appearance.fontFamily.trim().length > 0, `Caption Style ${value.id} font family is empty`);+ assert(appearance.textAlign === "left" || appearance.textAlign === "center" || appearance.textAlign === "right",+ `Caption Style ${value.id} text alignment is invalid`);+ const numeric = [+ appearance.fontSizePx,+ appearance.fontWeight,+ appearance.paddingXPx,+ appearance.paddingYPx,+ appearance.borderRadiusPx,+ appearance.bottomPercent,+ appearance.maxWidthPercent,+ appearance.leftPercent,+ appearance.topPercent,+ appearance.widthPercent,+ appearance.lineHeight,+ ].filter((item): item is number => item !== undefined);+ assert(numeric.every((item) => Number.isFinite(item) && item >= 0),+ `Caption Style ${value.id} appearance contains an invalid number`);+ assert(appearance.fontSizePx > 0 && appearance.maxWidthPercent > 0 && appearance.maxWidthPercent <= 100+ && appearance.bottomPercent <= 100,+ `Caption Style ${value.id} appearance bounds are invalid`);+ for (const color of [appearance.color, appearance.backgroundColor].filter((item): item is string => item !== undefined)) {+ assert(/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/iu.test(color), `Caption Style ${value.id} color ${color} is invalid`);+ }+ assert(isDigest(value.digest), `Caption Style ${value.id} digest is invalid`);+ const { digest: _digest, ...withoutDigest } = value;+ assert(value.digest === digestOf(styleContent(withoutDigest)), `Caption Style ${value.id} digest differs`);+}++function matches(+ atom: CaptionDisplayAtom,+ application: CaptionStyleApplication,+ narrative: Narrative,+): boolean {+ if (application.selector.kind === "role") return atom.role === application.selector.role;+ return displayAtomMatchesSelection(atom, application.selector.selection, narrative.tokens.length);+}++function programContent(value: Omit<CaptionProgram, "digest">) {+ return canonicalize(value) as unknown as Omit<CaptionProgram, "digest">;+}++export function sealCaptionProgram(value: Omit<CaptionProgram, "digest">): CaptionProgram {+ const content = programContent(value);+ const result = { ...content, digest: digestOf(content) };+ assertCaptionProgram(result);+ return result;+}++export function assertCaptionProgram(value: CaptionProgram): void {+ assert(value.contract === "svml.caption-program@1" && ID.test(value.id), "Caption Program identity is invalid");+ assert(isDigest(value.narrativeDigest) && isDigest(value.digest), "Caption Program digest is invalid");+ assert(value.atoms.length > 0 && value.runs.length > 0 && value.styles.length > 0, "Caption Program is empty");+ const styles = new Map(value.styles.map((style) => [style.id, style]));+ assert(styles.size === value.styles.length && styles.has(value.defaultStyleId), "Caption Program styles are invalid");+ value.styles.forEach(assertCaptionStyle);+ const atomIds = value.atoms.map((atom) => atom.id);+ assert(new Set(atomIds).size === atomIds.length, "Caption Program display atom ids are repeated");+ value.atoms.forEach((atom, index) => {+ assert(atom.id.length > 0 && atom.index === index && atom.regionId.length > 0 && atom.segmentId.length > 0+ && atom.turnId.length > 0 && atom.text.length > 0, "Caption Program display atom identity is invalid");+ assert(Number.isSafeInteger(atom.displayStart) && Number.isSafeInteger(atom.displayEnd)+ && atom.displayStart >= 0 && atom.displayEnd > atom.displayStart,+ `Caption Program display atom ${atom.id} display range is invalid`);+ assert(Number.isSafeInteger(atom.sourceTokenStart) && Number.isSafeInteger(atom.sourceTokenEndExclusive)+ && atom.sourceTokenStart >= 0 && atom.sourceTokenEndExclusive > atom.sourceTokenStart,+ `Caption Program display atom ${atom.id} source range is invalid`);+ assert(atom.correspondence === "exact" || atom.correspondence === "region-envelope",+ `Caption Program display atom ${atom.id} correspondence is invalid`);+ });+ assert(new Set(value.runs.map((run) => run.id)).size === value.runs.length,+ "Caption Program run ids are repeated");+ const planned = value.runs.flatMap((run) => run.atomIds);+ assert(planned.join("\0") === atomIds.join("\0"),+ "Caption Program runs must partition every display atom exactly once and in order");+ assert(value.runs.every((run) => styles.has(run.styleId) && run.atomIds.length > 0), "Caption Program run style is invalid");+ const { digest: _digest, ...withoutDigest } = value;+ assert(value.digest === digestOf(programContent(withoutDigest)), "Caption Program digest differs");+}++/** Bind a Program's immutable display universe to the exact Narrative consumed by a Producer. */+export function assertCaptionProgramForNarrative(value: CaptionProgram, narrative: Narrative): void {+ assertCaptionProgram(value);+ assert(value.narrativeDigest === digestOf(narrative), "Caption Program belongs to another Narrative");+ assert(digestOf(value.atoms) === digestOf(captionDisplayAtoms(narrative)),+ "Caption Program display atoms differ from its Narrative");+}++/** Resolve a base style plus ordered, whole-style replacements. Later matching applications win. */+export function resolveCaptionProgram(+ narrative: Narrative,+ id: string,+ defaultStyle: CaptionStyleIntent,+ applications: readonly CaptionStyleApplication[],+): CaptionProgram {+ assertCaptionStyle(defaultStyle);+ applications.forEach((application) => {+ assert(ID.test(application.id), "Caption Style Application identity is invalid");+ assertCaptionStyle(application.style);+ if (application.selector.kind === "role") {+ assert(application.selector.role.trim().length > 0, `Caption application ${application.id} Role is empty`);+ }+ });+ const atoms = captionDisplayAtoms(narrative);+ const styles = new Map<string, CaptionStyleIntent>([[defaultStyle.id, defaultStyle]]);+ applications.forEach((application) => {+ const previous = styles.get(application.style.id);+ assert(previous === undefined || previous.digest === application.style.digest,+ `Caption Style ${application.style.id} has conflicting definitions`);+ styles.set(application.style.id, application.style);+ });+ for (const application of applications) {+ assert(atoms.some((atom) => matches(atom, application, narrative)),+ `Caption application ${application.id} selects no visible display atom`);+ }+ const assignments = atoms.map((atom) => {+ let styleId = defaultStyle.id;+ for (const application of applications) {+ if (matches(atom, application, narrative)) styleId = application.style.id;+ }+ return { atom, styleId };+ });+ const runs: Array<{ id: string; styleId: string; atomIds: string[]; turnId: string; segmentId: string }> = [];+ for (const assignment of assignments) {+ const current = runs.at(-1);+ if (current === undefined || current.styleId !== assignment.styleId+ || current.turnId !== assignment.atom.turnId || current.segmentId !== assignment.atom.segmentId) {+ runs.push({+ id: `${id}:run:${runs.length + 1}`,+ styleId: assignment.styleId,+ atomIds: [assignment.atom.id],+ turnId: assignment.atom.turnId,+ segmentId: assignment.atom.segmentId,+ });+ } else {+ current.atomIds.push(assignment.atom.id);+ }+ }+ return sealCaptionProgram({+ contract: "svml.caption-program@1",+ id,+ narrativeDigest: digestOf(narrative),+ defaultStyleId: defaultStyle.id,+ styles: [...styles.values()],+ atoms,+ runs: runs.map(({ turnId: _turn, segmentId: _segment, ...run }) => run),+ });+}diff --git a/packages/caption/src/surface.ts b/packages/caption/src/surface.tsnew file mode 100644index 00000000..e239d71a--- /dev/null+++ b/packages/caption/src/surface.ts@@ -0,0 +1,321 @@+import { contractTypes } from "@svml/contracts";+import type { Narrative, NarrativeSelectionRef } from "@svml/contracts";+import { svsRecipeType } from "@svml/svs";+import type { SvsRecipe } from "@svml/svs";+import type {+ StructuredElement,+ StructuredSurfaceHandler,+ SurfaceResolvedReference,+ TextAttributeValue,+} from "@svml/text";++import { captionTrackSurfaceFragment, plannedCaptionTrackSurfaceFragment } from "./fragment.js";+import { captionTypes } from "./manifest.js";+import { sealCaptionTrackProgram } from "./track.js";+import { resolveCaptionProgram, sealCaptionStyle } from "./style.js";+import type { CaptionStyleApplication } from "./style.js";+import type {+ CaptionFieldDeclaration,+ CaptionFieldValueSchema,+ CaptionPresentationMode,+ CaptionStyleIntent,+} from "./types.js";++function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;+}++function attributes(element: StructuredElement, required: readonly string[], optional: readonly string[] = []): void {+ const allowed = new Set([...required, ...optional]);+ const actual = Object.keys(element.attributes);+ if (required.some((name) => element.attributes[name] === undefined) || actual.some((name) => !allowed.has(name))) {+ throw new Error(`${element.name} requires ${required.join(", ")}${optional.length ? `; optional: ${optional.join(", ")}` : ""}`);+ }+}++function stringAttribute(element: StructuredElement, name: string): string {+ const value = element.attributes[name];+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);+ return value.trim();+}++function reference(+ element: StructuredElement,+ name: string,+ expected: SurfaceResolvedReference["type"],+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,+): SurfaceResolvedReference {+ const raw: TextAttributeValue | undefined = element.attributes[name];+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a whole-value reference`);+ const value = resolveReference(raw.path);+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${raw.path}`);+ if (!sameType(value.type, expected)) throw new Error(`${element.name}.${name} has the wrong type`);+ return value;+}++function recipe(reference: SurfaceResolvedReference): SvsRecipe {+ if (!sameType(reference.type, svsRecipeType) || reference.record?.value.kind !== "inline") {+ throw new Error("Caption appearance must reference an authored SVS Recipe");+ }+ return reference.record.value.value as unknown as SvsRecipe;+}++function number(value: SvsRecipe, name: string): number {+ const property = value.properties[name];+ if (typeof property !== "number" || !Number.isFinite(property)) throw new Error(`Caption Recipe ${name} must be a number`);+ return property;+}++function integer(value: SvsRecipe, name: string): number {+ const property = number(value, name);+ if (!Number.isSafeInteger(property)) throw new Error(`Caption Recipe ${name} must be an integer`);+ return property;+}++function string(value: SvsRecipe, name: string): string {+ const property = value.properties[name];+ if (typeof property !== "string" || !property.trim()) throw new Error(`Caption Recipe ${name} must be a string`);+ return property.trim();+}++function padding(value: string): { readonly x: number; readonly y: number } {+ const parts = value.trim().split(/\s+/u).map(Number);+ if ((parts.length !== 1 && parts.length !== 2) || parts.some((item) => !Number.isFinite(item) || item < 0)) {+ throw new Error("Caption Recipe padding must contain one or two non-negative pixel numbers");+ }+ return { y: parts[0]!, x: parts[1] ?? parts[0]! };+}++function mode(element: StructuredElement): CaptionPresentationMode {+ const value = element.attributes.mode;+ if (value === undefined) return "whole";+ if (typeof value === "string" && ["whole", "proportional-word", "character-flow"].includes(value)) {+ return value as CaptionPresentationMode;+ }+ throw new Error(`${element.name}.mode is invalid`);+}++function inline<T>(reference: SurfaceResolvedReference, subject: string): T {+ if (reference.record?.value.kind !== "inline") throw new Error(`${subject} must reference an authored inline Record`);+ return reference.record.value.value as unknown as T;+}++function localName(name: string): string {+ return name.includes(":") ? name.slice(name.lastIndexOf(":") + 1) : name;+}++function body(element: StructuredElement): string {+ if (element.children.some((child) => child.kind === "element")) throw new Error(`${element.name} accepts only natural-language text`);+ const value = element.children.map((child) => child.kind === "text" ? child.value : "").join("").trim();+ if (!value) throw new Error(`${element.name} instruction is empty`);+ return value.replace(/\s+/gu, " ");+}++function optionalString(element: StructuredElement, name: string): string | undefined {+ const value = element.attributes[name];+ if (value === undefined) return undefined;+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);+ return value.trim();+}++function nonNegativeInteger(element: StructuredElement, name: string): number {+ const value = Number(stringAttribute(element, name));+ if (!Number.isSafeInteger(value) || value < 0) throw new Error(`${element.name}.${name} must be a non-negative integer`);+ return value;+}++function fieldValue(element: StructuredElement): CaptionFieldValueSchema {+ const type = stringAttribute(element, "type");+ if (type === "boolean") return { kind: "boolean" };+ if (type === "enum") {+ const values = stringAttribute(element, "values").split(",").map((item) => item.trim()).filter(Boolean);+ if (!values.length) throw new Error(`${element.name}.values is empty`);+ return { kind: "enum", values };+ }+ if (type === "number") {+ const minimum = optionalString(element, "minimum");+ const maximum = optionalString(element, "maximum");+ const min = minimum === undefined ? undefined : Number(minimum);+ const max = maximum === undefined ? undefined : Number(maximum);+ if ((min !== undefined && !Number.isFinite(min)) || (max !== undefined && !Number.isFinite(max))) {+ throw new Error(`${element.name} number bounds must be finite`);+ }+ return { kind: "number", ...(min === undefined ? {} : { minimum: min }), ...(max === undefined ? {} : { maximum: max }) };+ }+ throw new Error(`${element.name}.type must be boolean, enum or number`);+}++function captionAppearance(element: StructuredElement, value: SvsRecipe) {+ const expected = ["align", "background", "fill", "font", "line-height", "padding", "radius", "size", "stack-order", "weight", "width", "x", "y"];+ if (Object.keys(value.properties).sort().join("\u0000") !== expected.sort().join("\u0000")) {+ throw new Error(`Caption Recipe requires exactly ${expected.join(", ")}`);+ }+ const align = string(value, "align");+ if (align !== "left" && align !== "center" && align !== "right") throw new Error("Caption Recipe align is invalid");+ const pad = padding(string(value, "padding"));+ return {+ mode: mode(element),+ stackingOrder: integer(value, "stack-order"),+ style: {+ fontFamily: string(value, "font"),+ fontSizePx: number(value, "size"),+ fontWeight: integer(value, "weight"),+ color: string(value, "fill"),+ backgroundColor: string(value, "background"),+ paddingXPx: pad.x,+ paddingYPx: pad.y,+ borderRadiusPx: number(value, "radius"),+ bottomPercent: 0,+ maxWidthPercent: number(value, "width") * 100,+ leftPercent: number(value, "x") * 100,+ topPercent: number(value, "y") * 100,+ widthPercent: number(value, "width") * 100,+ lineHeight: number(value, "line-height"),+ textAlign: align,+ },+ } as const;+}++export const decodeCaptionStyleSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ attributes(element, ["id", "appearance"], ["mode"]);+ const id = stringAttribute(element, "id");+ const appearance = recipe(reference(element, "appearance", svsRecipeType, resolveReference));+ let cueInstruction: string | undefined;+ const fields: CaptionFieldDeclaration[] = [];+ for (const child of element.children) {+ if (child.kind === "text") {+ if (child.value.trim()) throw new Error(`${element.name} accepts only Cues and Field children`);+ continue;+ }+ if (localName(child.name) === "Cues") {+ attributes(child, []);+ if (cueInstruction !== undefined) throw new Error(`${element.name} repeats Cues`);+ cueInstruction = body(child);+ } else if (localName(child.name) === "Field") {+ attributes(child, ["id", "type", "min-per-cue", "max-per-cue"], ["values", "minimum", "maximum"]);+ fields.push({+ id: stringAttribute(child, "id"),+ value: fieldValue(child),+ instruction: body(child),+ minimumPerCue: nonNegativeInteger(child, "min-per-cue"),+ maximumPerCue: nonNegativeInteger(child, "max-per-cue"),+ });+ } else {+ throw new Error(`${element.name} accepts only Cues and Field children`);+ }+ }+ if (cueInstruction === undefined) throw new Error(`${element.name} requires one Cues instruction`);+ const style = sealCaptionStyle({+ contract: "svml.caption-style@1",+ id,+ planning: { cueInstruction, fields },+ presentation: captionAppearance(element, appearance),+ });+ return {+ records: [{ id, type: captionTypes.style, value: { kind: "inline", value: style }, range: element.range }],+ components: [],+ fragments: [],+ };+};++export const decodeCaptionProgramSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ attributes(element, ["id", "narrative", "default"]);+ const id = stringAttribute(element, "id");+ const narrativeRef = reference(element, "narrative", contractTypes.narrative, resolveReference);+ const defaultRef = reference(element, "default", captionTypes.style, resolveReference);+ const applications: CaptionStyleApplication[] = [];+ for (const child of element.children) {+ if (child.kind === "text") {+ if (child.value.trim()) throw new Error(`${element.name} accepts only Use children`);+ continue;+ }+ if (localName(child.name) !== "Use") throw new Error(`${element.name} accepts only Use children`);+ attributes(child, ["style"], ["role", "on"]);+ const role = optionalString(child, "role");+ const on = child.attributes.on;+ if ((role === undefined) === (on === undefined)) throw new Error(`${child.name} requires exactly one of role or on`);+ const styleRef = reference(child, "style", captionTypes.style, resolveReference);+ const selector = role !== undefined+ ? { kind: "role" as const, role }+ : {+ kind: "selection" as const,+ selection: inline<NarrativeSelectionRef>(+ reference(child, "on", contractTypes.narrativeSelection, resolveReference),+ `${child.name}.on`,+ ),+ };+ applications.push({+ id: `${id}.use.${applications.length + 1}`,+ selector,+ style: inline<CaptionStyleIntent>(styleRef, `${child.name}.style`),+ });+ }+ const program = resolveCaptionProgram(+ inline<Narrative>(narrativeRef, `${element.name}.narrative`),+ id,+ inline<CaptionStyleIntent>(defaultRef, `${element.name}.default`),+ applications,+ );+ return {+ records: [{ id, type: captionTypes.program, value: { kind: "inline", value: program }, range: element.range }],+ components: [],+ fragments: [],+ };+};++export const decodeCaptionTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ if (element.attributes.program !== undefined) {+ attributes(element, ["id", "narrative", "map", "program", "plan"]);+ if (element.children.some((child) => child.kind === "element" || child.value.trim())) {+ throw new Error(`${element.name} with a Caption Program does not accept children`);+ }+ const id = stringAttribute(element, "id");+ const narrative = reference(element, "narrative", contractTypes.narrative, resolveReference);+ const map = reference(element, "map", contractTypes.completeSemanticMap, resolveReference);+ const program = reference(element, "program", captionTypes.program, resolveReference);+ const plan = reference(element, "plan", captionTypes.plan, resolveReference);+ return {+ records: [],+ components: [{+ id,+ fragment: plannedCaptionTrackSurfaceFragment.id,+ inputs: { narrative: narrative.ref, map: map.ref, program: program.ref, plan: plan.ref },+ outputs: { track: `${id}.track` },+ range: element.range,+ }],+ fragments: [plannedCaptionTrackSurfaceFragment],+ };+ }+ attributes(element, ["id", "narrative", "map", "appearance"], ["mode"]);+ if (element.children.some((child) => child.kind === "element" || child.value.trim())) {+ throw new Error(`${element.name} provider-free base does not accept Role children`);+ }+ const id = stringAttribute(element, "id");+ const narrative = reference(element, "narrative", contractTypes.narrative, resolveReference);+ const map = reference(element, "map", contractTypes.completeSemanticMap, resolveReference);+ const appearance = recipe(reference(element, "appearance", svsRecipeType, resolveReference));+ const resolvedAppearance = captionAppearance(element, appearance);+ const program = sealCaptionTrackProgram({+ contract: "svml.caption-track-program@1",+ id,+ mode: resolvedAppearance.mode,+ stacking: { order: resolvedAppearance.stackingOrder, tieBreak: id },+ style: resolvedAppearance.style,+ });+ const programId = `${id}.program`;+ return {+ records: [{ id: programId, type: captionTypes.trackProgram, value: { kind: "inline", value: program }, range: element.range }],+ components: [{+ id,+ fragment: captionTrackSurfaceFragment.id,+ inputs: {+ narrative: narrative.ref,+ map: map.ref,+ program: { kind: "record", id: programId },+ },+ outputs: { track: `${id}.track` },+ range: element.range,+ }],+ fragments: [captionTrackSurfaceFragment],+ };+};diff --git a/packages/caption/src/temporalize.ts b/packages/caption/src/temporalize.tsindex 33d9c31e..5c3d3614 100644--- a/packages/caption/src/temporalize.ts+++ b/packages/caption/src/temporalize.ts@@ -3,10 +3,16 @@ import type { CompleteSemanticMap, Narrative, TimingQuality } from "@svml/contraimport { CaptionProjectionError } from "./error.js";import type {+ CaptionPlan,+ CaptionProgram,+ CaptionDisplayAtom,TimedCaptionProjection,TimedCaptionRefinement,TimedCaptionRegion,} from "./types.js";+import { assertCaptionPlanForProgram } from "./plan.js";+import { displayTextForAtoms } from "./display.js";+import { assertCaptionProgramForNarrative } from "./style.js";const QUALITY_RANK: Readonly<Record<TimingQuality, number>> = {measured: 0,@@ -69,3 +75,134 @@ export function temporalizeCaption(};return { ...payload, projectionDigest: digestOf(payload) };}++function cleanDisplay(value: string): string {+ return value.replace(/\s+/gu, " ").replace(/\s+([,.;:!?])/gu, "$1").trim();+}++/** Join planner-neutral Cue/field facts with the one authoritative Script map. */+export function temporalizeCaptionPlan(+ narrative: Narrative,+ map: CompleteSemanticMap,+ program: CaptionProgram,+ plan: CaptionPlan,+): TimedCaptionProjection {+ assertCaptionProgramForNarrative(program, narrative);+ assertCaptionPlanForProgram(plan, program);+ if (plan.narrativeDigest !== digestOf(narrative)) {+ throw new CaptionProjectionError("CAPTION_PLAN_NARRATIVE", "CaptionPlan belongs to another Narrative.");+ }+ if (program.narrativeDigest !== plan.narrativeDigest || plan.captionProgramDigest !== program.digest) {+ throw new CaptionProjectionError("CAPTION_PLAN_PROGRAM", "CaptionPlan belongs to another Caption Program.");+ }+ const atomById = new Map(program.atoms.map((atom) => [atom.id, atom]));+ const timingById = new Map(map.tokens.map((token) => [token.tokenId, token]));+ const regionById = new Map(narrative.captionProjection.regions.map((region) => [region.id, region]));+ const atomsByRegion = new Map<string, CaptionDisplayAtom[]>();+ for (const atom of program.atoms) {+ const values = atomsByRegion.get(atom.regionId) ?? [];+ values.push(atom);+ atomsByRegion.set(atom.regionId, values);+ }+ const atomWindow = (atom: CaptionDisplayAtom) => {+ const firstSource = narrative.tokens[atom.sourceTokenStart];+ const lastSource = narrative.tokens[atom.sourceTokenEndExclusive - 1];+ const first = firstSource === undefined ? undefined : timingById.get(firstSource.id);+ const last = lastSource === undefined ? undefined : timingById.get(lastSource.id);+ if (first === undefined || last === undefined) {+ throw new CaptionProjectionError("CAPTION_SPEECH_COVERAGE", `Caption display atom ${atom.id} is absent from the complete speech map.`);+ }+ if (atom.correspondence === "exact") {+ return {+ startSec: first.startSec,+ endSec: last.endSec,+ startQuality: composedQuality(first.startQuality),+ endQuality: composedQuality(last.endQuality),+ };+ }+ const region = regionById.get(atom.regionId);+ if (region === undefined || region.display.length === 0) {+ throw new CaptionProjectionError("CAPTION_DISPLAY_REGION", `Caption display atom ${atom.id} has no owning display region.`);+ }+ const siblings = atomsByRegion.get(atom.regionId) ?? [];+ const next = siblings.find((candidate) => candidate.displayStart > atom.displayStart);+ const duration = last.endSec - first.startSec;+ return {+ startSec: first.startSec + duration * atom.displayStart / region.display.length,+ endSec: first.startSec + duration * (next?.displayStart ?? region.display.length) / region.display.length,+ startQuality: "estimated" as const,+ endQuality: "estimated" as const,+ };+ };+ const visible = new Set(program.atoms.map((atom) => atom.id));+ const plannedIds = plan.runs.flatMap((run) => run.cues.flatMap((cue) => cue.atomIds));+ if (plannedIds.length !== visible.size || new Set(plannedIds).size !== visible.size || plannedIds.some((id) => !visible.has(id))) {+ throw new CaptionProjectionError(+ "CAPTION_PLAN_COVERAGE",+ "CaptionPlan Cues must be disjoint and cover every display atom exactly once.",+ );+ }+ const programRuns = new Map(program.runs.map((run) => [run.id, run]));+ const regions: TimedCaptionRegion[] = [];+ for (const run of plan.runs) {+ const expectedRun = programRuns.get(run.id);+ if (expectedRun === undefined || expectedRun.styleId !== run.styleId) {+ throw new CaptionProjectionError("CAPTION_PLAN_RUN", `Caption run ${run.id} differs from its Program.`);+ }+ const runAtoms = run.cues.flatMap((cue) => cue.atomIds);+ if (runAtoms.join("\0") !== expectedRun.atomIds.join("\0")) {+ throw new CaptionProjectionError("CAPTION_PLAN_RUN", `Caption run ${run.id} changes its Program atom order.`);+ }+ for (const cue of run.cues) {+ const source = cue.atomIds.map((id) => atomById.get(id));+ if (source.some((atom) => atom === undefined)) {+ throw new CaptionProjectionError("CAPTION_PLAN_ATOM", `Caption Cue ${cue.id} references an unknown display atom.`);+ }+ const atoms = source as CaptionDisplayAtom[];+ if (atoms.some((atom, index) => index > 0 && atom.index <= atoms[index - 1]!.index)) {+ throw new CaptionProjectionError("CAPTION_PLAN_ORDER", `Caption Cue ${cue.id} changes display atom order.`);+ }+ if (new Set(atoms.map((atom) => atom.segmentId)).size !== 1) {+ throw new CaptionProjectionError("CAPTION_PLAN_SEGMENT", `Caption Cue ${cue.id} crosses a Script Segment.`);+ }+ const sourceIndexes = [...new Set(atoms.flatMap((atom) => Array.from(+ { length: atom.sourceTokenEndExclusive - atom.sourceTokenStart },+ (_, offset) => atom.sourceTokenStart + offset,+ )))].sort((left, right) => left - right);+ const sourceTokens = sourceIndexes.map((index) => narrative.tokens[index]);+ const timed = sourceTokens.map((token) => token === undefined ? undefined : timingById.get(token.id));+ if (timed.length === 0 || timed.some((token) => token === undefined)) {+ throw new CaptionProjectionError("CAPTION_SPEECH_COVERAGE", `Caption Cue ${cue.id} is absent from the complete speech map.`);+ }+ const firstWindow = atomWindow(atoms[0]!);+ const lastWindow = atomWindow(atoms.at(-1)!);+ for (const field of cue.fields) {+ if (!cue.atomIds.includes(field.atomId)) {+ throw new CaptionProjectionError("CAPTION_PLAN_FIELD", `Caption field ${field.declarationId} lies outside Cue ${cue.id}.`);+ }+ }+ regions.push({+ id: cue.id,+ runId: run.id,+ styleId: run.styleId,+ display: displayTextForAtoms(narrative, atoms),+ segmentId: atoms[0]!.segmentId,+ kind: atoms.some((atom) => atom.correspondence === "region-envelope") ? "alias" : "identity",+ sourceTokenIds: sourceTokens.map((token) => token!.id),+ startSec: firstWindow.startSec,+ endSec: lastWindow.endSec,+ startQuality: firstWindow.startQuality,+ endQuality: lastWindow.endQuality,+ refinements: [],+ fields: cue.fields.map((field) => ({ ...field })),+ });+ }+ }+ const payload = {+ contract: "svml.timed-caption-projection@1" as const,+ programSpace: map.programSpace,+ text: cleanDisplay(regions.map((region) => region.display).join(" ")),+ regions,+ };+ return { ...payload, projectionDigest: digestOf(payload) };+}diff --git a/packages/caption/src/track.ts b/packages/caption/src/track.tsindex 12040150..8fc4bbbf 100644--- a/packages/caption/src/track.ts+++ b/packages/caption/src/track.ts@@ -9,10 +9,11 @@ import { digestOf, isDigest } from "@svml/protocol";import type { Digest } from "@svml/protocol";import { planCaptionPresentation } from "./presentation.js";-import type { TimedCaptionProjection } from "./types.js";-import type { CaptionTrackProgram } from "./types.js";+import { assertCaptionProgram } from "./style.js";+import type { CaptionProgram, CaptionTrackProgram, TimedCaptionProjection } from "./types.js";export const renderCaptionTrackImplementationDigest = digestOf("@svml/caption/render-track@1");+export const renderCaptionProgramImplementationDigest = digestOf("@svml/caption/render-program@1");function normalizedProgram(value: Omit<CaptionTrackProgram, "digest">): Omit<CaptionTrackProgram, "digest"> {return {@@ -75,6 +76,10 @@ export function assertCaptionTrackProgram(program: CaptionTrackProgram): void {borderRadiusPx: program.style.borderRadiusPx,bottomPercent: program.style.bottomPercent,maxWidthPercent: program.style.maxWidthPercent,+ ...(program.style.leftPercent === undefined ? {} : { leftPercent: program.style.leftPercent }),+ ...(program.style.topPercent === undefined ? {} : { topPercent: program.style.topPercent }),+ ...(program.style.widthPercent === undefined ? {} : { widthPercent: program.style.widthPercent }),+ ...(program.style.lineHeight === undefined ? {} : { lineHeight: program.style.lineHeight }),};for (const [name, value] of Object.entries(numeric)) {if (!Number.isFinite(value) || value < 0) throw new Error(`CaptionTrackProgram ${name} is invalid.`);@@ -84,7 +89,7 @@ export function assertCaptionTrackProgram(program: CaptionTrackProgram): void {}if (program.style.bottomPercent > 100) throw new Error("CaptionTrackProgram bottomPercent is invalid.");for (const color of [program.style.color, program.style.backgroundColor].filter((value): value is string => value !== undefined)) {- if (!/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/u.test(color)) throw new Error(`CaptionTrackProgram color ${color} is invalid.`);+ if (!/^#[0-9a-f]{6}(?:[0-9a-f]{2})?$/iu.test(color)) throw new Error(`CaptionTrackProgram color ${color} is invalid.`);}}@@ -105,16 +110,17 @@ function frameAt(projection: TimedCaptionProjection, seconds: number): number {/ projection.programSpace.frameRate.denominator);}-function textStyle(program: CaptionTrackProgram): VisualStyleDeclaration[] {+function textStyle(style: CaptionTrackProgram["style"]): VisualStyleDeclaration[] {return [- { name: "background", value: program.style.backgroundColor ?? "transparent" },- { name: "border-radius", value: `${program.style.borderRadiusPx}px` },- { name: "color", value: program.style.color },- { name: "font-family", value: program.style.fontFamily },- { name: "font-size", value: `${program.style.fontSizePx}px` },- { name: "font-weight", value: program.style.fontWeight },- { name: "padding", value: `${program.style.paddingYPx}px ${program.style.paddingXPx}px` },- { name: "text-align", value: program.style.textAlign },+ { name: "background", value: style.backgroundColor ?? "transparent" },+ { name: "border-radius", value: `${style.borderRadiusPx}px` },+ { name: "color", value: style.color },+ { name: "font-family", value: style.fontFamily },+ { name: "font-size", value: `${style.fontSizePx}px` },+ { name: "font-weight", value: style.fontWeight },+ ...(style.lineHeight === undefined ? [] : [{ name: "line-height", value: style.lineHeight }]),+ { name: "padding", value: `${style.paddingYPx}px ${style.paddingXPx}px` },+ { name: "text-align", value: style.textAlign },{ name: "white-space", value: "pre-wrap" },];}@@ -146,7 +152,7 @@ export function renderCaptionTrack(id: "root",order: 0,kind: "box" as const,- style: [+ style: program.style.leftPercent === undefined || program.style.topPercent === undefined || program.style.widthPercent === undefined ? [{ name: "align-items", value: "center" },{ name: "bottom", value: `${program.style.bottomPercent}%` },{ name: "display", value: "flex" },@@ -155,6 +161,93 @@ export function renderCaptionTrack({ name: "max-width", value: `${program.style.maxWidthPercent}%` },{ name: "position", value: "absolute" },{ name: "transform", value: "translateX(-50%)" },+ ] : [+ { name: "align-items", value: "center" },+ { name: "display", value: "flex" },+ { name: "justify-content", value: "center" },+ { name: "left", value: `${program.style.leftPercent}%` },+ { name: "position", value: "absolute" },+ { name: "top", value: `${program.style.topPercent}%` },+ { name: "width", value: `${program.style.widthPercent}%` },+ ],+ },+ {+ id: "text",+ parent: "root",+ order: 1,+ kind: "text" as const,+ text: unit.display,+ style: textStyle(program.style),+ attributes: [+ { name: "data-caption-basis", value: unit.basis },+ { name: "data-caption-quality", value: unit.timingQuality },+ ...(unit.runId === undefined ? [] : [{ name: "data-caption-run", value: unit.runId }]),+ ...(unit.fields === undefined ? [] : [{ name: "data-caption-fields", value: JSON.stringify(unit.fields) }]),+ ],+ },+ ],+ }];+ }),+ });+ assertVisualTrackIdentity(track, projection.programSpace);+ return track;+}++/** Render one peer Track whose Presents may each select a complete, resolved Caption Style. */+export function renderCaptionProgram(+ projection: TimedCaptionProjection,+ program: CaptionProgram,+): VisualTrack {+ assertTimedCaptionProjection(projection);+ assertCaptionProgram(program);+ const styles = new Map(program.styles.map((style) => [style.id, style]));+ const totalFrames = programSpaceFrameCount(projection.programSpace);+ const units = projection.regions.flatMap((region) => {+ const styleId = region.styleId ?? program.defaultStyleId;+ const style = styles.get(styleId);+ if (style === undefined) throw new Error(`Caption region ${region.id} references unknown Style ${styleId}`);+ return planCaptionPresentation(+ { projectionDigest: projection.projectionDigest, regions: [region] },+ style.presentation.mode,+ ).units.map((unit) => ({ unit, style }));+ });+ const track = sealVisualTrack({+ contract: "svml.visual-track@1",+ visualIr: "svml.hyperframes-visual-ir@1",+ id: program.id,+ programSpaceDigest: projection.programSpace.digest,+ presents: units.flatMap(({ unit, style }) => {+ const startFrame = Math.max(0, frameAt(projection, unit.startSec));+ const measuredEnd = Math.min(totalFrames, frameAt(projection, unit.endSec));+ const endFrameExclusive = Math.min(totalFrames, Math.max(startFrame + 1, measuredEnd));+ if (!unit.display || startFrame >= totalFrames || endFrameExclusive <= startFrame) return [];+ const appearance = style.presentation.style;+ return [{+ id: unit.id,+ span: { startFrame, endFrameExclusive },+ stacking: { order: style.presentation.stackingOrder, tieBreak: `${program.id}:${unit.id}` },+ elements: [+ {+ id: "root",+ order: 0,+ kind: "box" as const,+ style: appearance.leftPercent === undefined || appearance.topPercent === undefined || appearance.widthPercent === undefined ? [+ { name: "align-items", value: "center" },+ { name: "bottom", value: `${appearance.bottomPercent}%` },+ { name: "display", value: "flex" },+ { name: "justify-content", value: "center" },+ { name: "left", value: "50%" },+ { name: "max-width", value: `${appearance.maxWidthPercent}%` },+ { name: "position", value: "absolute" },+ { name: "transform", value: "translateX(-50%)" },+ ] : [+ { name: "align-items", value: "center" },+ { name: "display", value: "flex" },+ { name: "justify-content", value: "center" },+ { name: "left", value: `${appearance.leftPercent}%` },+ { name: "position", value: "absolute" },+ { name: "top", value: `${appearance.topPercent}%` },+ { name: "width", value: `${appearance.widthPercent}%` },],},{@@ -163,10 +256,13 @@ export function renderCaptionTrack(order: 1,kind: "text" as const,text: unit.display,- style: textStyle(program),+ style: textStyle(appearance),attributes: [{ name: "data-caption-basis", value: unit.basis },{ name: "data-caption-quality", value: unit.timingQuality },+ { name: "data-caption-style", value: style.id },+ ...(unit.runId === undefined ? [] : [{ name: "data-caption-run", value: unit.runId }]),+ ...(unit.fields === undefined ? [] : [{ name: "data-caption-fields", value: JSON.stringify(unit.fields) }]),],},],diff --git a/packages/caption/src/types.ts b/packages/caption/src/types.tsindex 942e252a..859ef2b5 100644--- a/packages/caption/src/types.ts+++ b/packages/caption/src/types.ts@@ -1,6 +1,97 @@import type { ProgramSpace, TimingQuality } from "@svml/contracts";import type { Digest } from "@svml/protocol";+export type CaptionFieldValueSchema =+ | { readonly kind: "boolean" }+ | { readonly kind: "enum"; readonly values: readonly string[] }+ | { readonly kind: "number"; readonly minimum?: number; readonly maximum?: number };++export type CaptionFieldDeclaration = {+ readonly id: string;+ readonly value: CaptionFieldValueSchema;+ readonly instruction: string;+ readonly minimumPerCue: number;+ readonly maximumPerCue: number;+};++export type CaptionPlanningRequirements = {+ readonly cueInstruction: string;+ readonly fields: readonly CaptionFieldDeclaration[];+};++export type CaptionDisplayAtom = {+ readonly id: string;+ readonly index: number;+ readonly regionId: string;+ readonly segmentId: string;+ readonly turnId: string;+ readonly role?: string;+ readonly text: string;+ readonly displayStart: number;+ readonly displayEnd: number;+ readonly sourceTokenStart: number;+ readonly sourceTokenEndExclusive: number;+ readonly correspondence: "exact" | "region-envelope";+};++export type CaptionFieldAssignment = {+ readonly declarationId: string;+ readonly atomId: string;+ readonly value: string;+};++export type CaptionPlannedCue = {+ readonly id: string;+ readonly atomIds: readonly string[];+ readonly fields: readonly CaptionFieldAssignment[];+};++export type CaptionPlannedRun = {+ readonly id: string;+ readonly styleId: string;+ readonly cues: readonly CaptionPlannedCue[];+};++/** Planner-neutral result. Model/provider identity remains in the producing Graph and Receipt. */+export type CaptionPlan = {+ readonly contract: "svml.caption-plan@1";+ readonly narrativeDigest: Digest;+ readonly captionProgramDigest: Digest;+ readonly planningRequestDigest: Digest;+ readonly runs: readonly CaptionPlannedRun[];+ readonly planDigest: Digest;+};++export type CaptionStyleIntent = {+ readonly contract: "svml.caption-style@1";+ readonly id: string;+ readonly planning: CaptionPlanningRequirements;+ readonly presentation: {+ readonly mode: CaptionPresentationMode;+ readonly stackingOrder: number;+ readonly style: CaptionTrackProgram["style"];+ };+ readonly digest: Digest;+};++export type CaptionProgramRun = {+ readonly id: string;+ readonly styleId: string;+ readonly atomIds: readonly string[];+};++/** Total, ordered style assignment over one Caption display universe. */+export type CaptionProgram = {+ readonly contract: "svml.caption-program@1";+ readonly id: string;+ readonly narrativeDigest: Digest;+ readonly defaultStyleId: string;+ readonly styles: readonly CaptionStyleIntent[];+ readonly atoms: readonly CaptionDisplayAtom[];+ readonly runs: readonly CaptionProgramRun[];+ readonly digest: Digest;+};+export type TimedCaptionRefinement = {readonly id: string;readonly display: string;@@ -16,6 +107,8 @@ export type TimedCaptionRefinement = {export type TimedCaptionRegion = {readonly id: string;+ readonly runId?: string;+ readonly styleId?: string;readonly display: string;readonly segmentId: string;readonly kind: "identity" | "alias" | "hidden";@@ -25,6 +118,7 @@ export type TimedCaptionRegion = {readonly startQuality: TimingQuality;readonly endQuality: TimingQuality;readonly refinements: readonly TimedCaptionRefinement[];+ readonly fields?: readonly CaptionFieldAssignment[];};export type TimedCaptionProjection = {@@ -40,6 +134,8 @@ export type CaptionPresentationMode = "whole" | "proportional-word" | "characterexport type CaptionPresentationUnit = {readonly id: string;readonly regionId: string;+ readonly runId?: string;+ readonly styleId?: string;readonly display: string;readonly displayStart: number;readonly displayEnd: number;@@ -47,6 +143,7 @@ export type CaptionPresentationUnit = {readonly endSec: number;readonly basis: "region-envelope" | "exact-correspondence" | "presentation-estimate";readonly timingQuality: TimingQuality;+ readonly fields?: readonly CaptionFieldAssignment[];};export type CaptionPresentationPlan = {@@ -77,6 +174,10 @@ export type CaptionTrackProgram = {readonly borderRadiusPx: number;readonly bottomPercent: number;readonly maxWidthPercent: number;+ readonly leftPercent?: number;+ readonly topPercent?: number;+ readonly widthPercent?: number;+ readonly lineHeight?: number;readonly textAlign: "left" | "center" | "right";};};diff --git a/packages/caption/test/caption.test.ts b/packages/caption/test/caption.test.tsindex 57777409..e8a085d7 100644--- a/packages/caption/test/caption.test.ts+++ b/packages/caption/test/caption.test.ts@@ -6,18 +6,23 @@ import {captionManifest,defaultCaptionTrackProgram,planCaptionPresentation,+ renderCaptionProgram,renderCaptionTrack,+ resolveCaptionProgram,+ sealCaptionPlan,+ sealCaptionStyle,sealCaptionTrackProgram,temporalizeCaption,+ temporalizeCaptionPlan,} from "@svml/caption";import {sealAlignedTranscriptEvidence,sealProgramSpace,sealSpeechBasis,} from "@svml/contracts";-import type { AlignedTranscriptSegment, Narrative, SpeechAudioBasis } from "@svml/contracts";+import type { AlignedTranscriptSegment, Narrative, NarrativeSelectionRef, SpeechAudioBasis } from "@svml/contracts";import { digestOf } from "@svml/protocol";-import { parseScript } from "@svml/script";+import { narrativeSelectionValue, parseScript } from "@svml/script";import { locateSpeechTiming } from "@svml/speech-align";test("the component enumerates every Manifest Producer and owned Type validator", () => {@@ -166,6 +171,52 @@ test("Caption temporalization preserves evidence envelopes and labels local estiassert.deepEqual(map, originalMap, "caption presentation must not modify the global speech map");});+test("multiple Cues inside one display alias receive ordered local estimates, not duplicate envelopes", () => {+ const narrative = parseScript("alias-plan.svml", "<line><that was insane | what the fuck></line>");+ const map = locate(narrative, 1, [{+ sourceSegmentId: "line",+ startSec: 0,+ endSec: 1,+ words: [+ { text: "what", startSec: 0.1, endSec: 0.25 },+ { text: "the", startSec: 0.3, endSec: 0.42 },+ { text: "fuck", startSec: 0.48, endSec: 0.72 },+ ],+ chars: [],+ }]);+ const base = defaultCaptionTrackProgram("alias-base");+ const style = sealCaptionStyle({+ contract: "svml.caption-style@1",+ id: "alias-style",+ planning: { cueInstruction: "One word per Cue for this test.", fields: [] },+ presentation: { mode: "whole", stackingOrder: 100, style: base.style },+ });+ const program = resolveCaptionProgram(narrative, "alias-caption", style, []);+ const run = program.runs[0]!;+ const plan = sealCaptionPlan({+ contract: "svml.caption-plan@1",+ narrativeDigest: digestOf(narrative),+ captionProgramDigest: program.digest,+ planningRequestDigest: digestOf("alias-plan-request"),+ runs: [{+ id: run.id,+ styleId: run.styleId,+ cues: run.atomIds.map((atomId, index) => ({ id: `alias-cue:${index + 1}`, atomIds: [atomId], fields: [] })),+ }],+ });+ const projection = temporalizeCaptionPlan(narrative, map, program, plan);++ assert.deepEqual(projection.regions.map((region) => region.display), ["that", "was", "insane"]);+ assert.equal(projection.regions.every((region) =>+ region.startQuality === "estimated" && region.endQuality === "estimated"), true);+ assert.equal(projection.regions[0]!.endSec <= projection.regions[1]!.startSec, true);+ assert.equal(projection.regions[1]!.endSec <= projection.regions[2]!.startSec, true);+ assert.deepEqual(+ [projection.regions[0]!.startSec, projection.regions[2]!.endSec],+ [0.1, 0.72],+ );+});+test("hidden speech owns time but emits no visible presentation unit", () => {const narrative = parseScript("hidden.svml", "<line>Hello < | um> world.</line>");const map = locate(narrative, 1, [{@@ -237,3 +288,119 @@ test("two caption styles become two peer Tracks without mutating one another", (assert.equal(firstTrack.presents[0]?.stacking.order, 100);assert.equal(secondTrack.presents[0]?.stacking.order, 101);});++test("a planner-neutral CaptionPlan joins SemanticMap only after Cue and field planning", () => {+ const narrative = parseScript("planned.svml", "<line><ALICE>Meaning becomes the source.</line>");+ const map = locate(narrative, 2, [{+ sourceSegmentId: "line",+ startSec: 0,+ endSec: 2,+ words: [+ { text: "Meaning", startSec: 0.1, endSec: 0.35 },+ { text: "becomes", startSec: 0.4, endSec: 0.7 },+ { text: "the", startSec: 0.8, endSec: 1.0 },+ { text: "source", startSec: 1.1, endSec: 1.5 },+ ],+ chars: [],+ }]);+ const base = defaultCaptionTrackProgram("planned-caption");+ const style = sealCaptionStyle({+ contract: "svml.caption-style@1",+ id: "alice",+ planning: {+ cueInstruction: "Use short semantic phrases.",+ fields: [{+ id: "important",+ value: { kind: "boolean" },+ instruction: "Select at most one important word.",+ minimumPerCue: 0,+ maximumPerCue: 1,+ }],+ },+ presentation: { mode: "whole", stackingOrder: 100, style: base.style },+ });+ const program = resolveCaptionProgram(narrative, "planned-caption", style, []);+ const ids = program.atoms.map((atom) => atom.id);+ const plan = sealCaptionPlan({+ contract: "svml.caption-plan@1",+ narrativeDigest: digestOf(narrative),+ captionProgramDigest: program.digest,+ planningRequestDigest: digestOf("planned-caption-request"),+ runs: [{+ id: program.runs[0]!.id,+ styleId: style.id,+ cues: [+ {+ id: "cue:1",+ atomIds: ids.slice(0, 2),+ fields: [{ declarationId: "important", atomId: ids[0]!, value: "true" }],+ },+ { id: "cue:2", atomIds: ids.slice(2), fields: [] },+ ],+ }],+ });+ const projection = temporalizeCaptionPlan(narrative, map, program, plan);+ assert.deepEqual(projection.regions.map((region) => region.display), ["Meaning becomes", "the source."]);+ assert.deepEqual(projection.regions[0]?.fields?.map((field) => field.declarationId), ["important"]);+ const track = renderCaptionProgram(projection, program);+ const firstText = track.presents[0]?.elements.find((element) => element.kind === "text");+ assert.equal(firstText?.attributes?.some((attribute) =>+ attribute.name === "data-caption-style" && attribute.value === "alice"), true);+ assert.equal(firstText?.attributes?.some((attribute) =>+ attribute.name === "data-caption-fields" && attribute.value.includes("important")), true);+});++test("a default Style covers roleless text and ordered whole-style overrides use last match", () => {+ const narrative = parseScript("cascade.svml", `+ <intro>Roleless words stay readable.</intro>+ <answer><ALICE>Only this sentence changes.</answer>+ `);+ const base = defaultCaptionTrackProgram("base");+ const make = (id: string, color: string) => sealCaptionStyle({+ contract: "svml.caption-style@1",+ id,+ planning: { cueInstruction: `Plan ${id}.`, fields: [] },+ presentation: { mode: "whole", stackingOrder: 100, style: { ...base.style, color } },+ });+ const normal = make("normal", "#ffffff");+ const alice = make("alice", "#00ff00");+ const final = make("final", "#ff00ff");+ const program = resolveCaptionProgram(narrative, "captions", normal, [+ { id: "alice", selector: { kind: "role", role: "ALICE" }, style: alice },+ { id: "alice-later", selector: { kind: "role", role: "ALICE" }, style: final },+ ]);+ assert.deepEqual(program.runs.map((run) => run.styleId), ["normal", "final"]);+ assert.equal(program.styles.some((style) => style.id === "alice"), true, "declared overrides stay auditable");+ assert.throws(() => resolveCaptionProgram(narrative, "invalid", normal, [{+ id: "typo",+ selector: { kind: "role", role: "ALIEC" },+ style: alice,+ }]), /selects no visible display atom/u);+});++test("an explicit Selection replaces only its words without authoring the complement", () => {+ const narrative = parseScript(+ "selection-style.svml",+ "<line>Keep this @special one sentence different @/special and return.</line>",+ );+ const base = defaultCaptionTrackProgram("base");+ const make = (id: string, color: string) => sealCaptionStyle({+ contract: "svml.caption-style@1",+ id,+ planning: { cueInstruction: `Plan ${id}.`, fields: [] },+ presentation: { mode: "whole", stackingOrder: 100, style: { ...base.style, color } },+ });+ const normal = make("normal", "#ffffff");+ const special = make("special", "#ff00ff");+ const selection = narrativeSelectionValue(narrative.selections[0]!) as unknown as NarrativeSelectionRef;+ const program = resolveCaptionProgram(narrative, "captions", normal, [{+ id: "special-use",+ selector: { kind: "selection", selection },+ style: special,+ }]);+ assert.deepEqual(program.runs.map((run) => [run.styleId, run.atomIds.length]), [+ ["normal", 2],+ ["special", 3],+ ["normal", 2],+ ]);+});diff --git a/packages/cli/README.md b/packages/cli/README.mdindex fac82473..96f28694 100644--- a/packages/cli/README.md+++ b/packages/cli/README.md@@ -9,7 +9,8 @@ itself. The current prelude contributes:- the official Text Frontend;- the Script raw Surface and its audiovisual Narrative contract dependency;- the SVS alternate Frontend and generic Recipe manifest;-- the official Film and HyperFrames Render Structured Surfaces plus their exact static dependencies.+- the official Speech, WhisperX, provider-free Caption, B-roll, Text, Film and HyperFrames Render+ Structured Surfaces plus their exact static dependencies.It does not place these packages in Core and does not grant source imports network, credential orprocess permissions.@@ -23,14 +24,15 @@ pnpm svml:v2 check path/to/main.svml --package-lock ./svml.packages.lock --rootpnpm svml:v2 plan path/to/main.svml --target component.resultpnpm svml:v2 build path/to/main.svml --target component.result --runtime ./svml.runtime.tspnpm svml:v2 build path/to/main.svml --target component.result --runtime ./svml.runtime.ts --follow+pnpm svml:v2 build path/to/main.svml --target final.video --runtime ./svml.runtime.ts --follow --out ./final.mp4pnpm svml:v2 status <build-id> --runtime ./svml.runtime.tspnpm svml:v2 cancel <build-id> --runtime ./svml.runtime.ts```-`check` is usable for the implemented Script, authored Image, Seedance generation, Film, Render and-recursively imported `.svs` sources. `plan` is fully implemented by the generic Node compiler. A-complete author video still awaits Speech assembly and Track Surfaces that produce Film's inputs;-the CLI does not fabricate those missing facts.+`check` is usable for the complete provider-free author graph in+`examples/talking-film-graph-check`. `plan` is fully implemented by the generic Node compiler. The+production golden still awaits the explicit Gemini Caption package and real assets; the CLI does+not fabricate those missing facts.`build` compiles the same locked BuildState and passes it to a trusted local Runtime config module.The default Build identity is content-derived, so the same invocation resumes durable local state;@@ -43,6 +45,10 @@ non-retryable failure, cancellation or `--max-wait-ms`. `status` reads durable v`cancel` invokes active endpoints and then records the resulting terminal Build failure. None ofthese commands creates a second ready-command queue.+`--out` accepts exactly one completed media target, reads its content-addressed bytes through the+selected Runtime ArtifactStore and writes the requested file. It does not treat a CAS path as a+public filename or bypass Record identity verification.+`@svml/package-loader-node` supports explicitly trusted installed implementation packages. Itverifies the complete physical dependency closure before executing an activation entry, then checksexact Module, Frontend, Surface, Producer and Validator identities. `plan` and `build` bind the lockdiff --git a/packages/cli/package.json b/packages/cli/package.jsonindex 86c40e58..29505769 100644--- a/packages/cli/package.json+++ b/packages/cli/package.json@@ -10,6 +10,7 @@"@svml/compiler-node": "workspace:*","@svml/local": "workspace:*","@svml/package-loader-node": "workspace:*",- "@svml/prelude-video": "workspace:*"+ "@svml/prelude-video": "workspace:*",+ "@svml/protocol": "workspace:*"}}diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.tsindex 924cce86..5f2d466d 100644--- a/packages/cli/src/index.ts+++ b/packages/cli/src/index.ts@@ -1,3 +1,3 @@export { createOfficialNodeCompiler } from "./host.js";export type { OfficialCompilerOptions } from "./host.js";-export { runCli } from "./main.js";+export { materializeSingleGoal, runCli } from "./main.js";diff --git a/packages/cli/src/main.ts b/packages/cli/src/main.tsindex fbac9f30..467da4f9 100644--- a/packages/cli/src/main.ts+++ b/packages/cli/src/main.ts@@ -1,7 +1,9 @@+import { mkdir, writeFile } from "node:fs/promises";import { dirname, resolve } from "node:path";import { pathToFileURL } from "node:url";import type { LocalRuntime } from "@svml/local";+import { isDigest } from "@svml/protocol";import {createNodePackageLock,loadNodePackageSet,@@ -26,6 +28,7 @@ type ParsedArgs = {readonly maxWaitMs: number | undefined;readonly packageLock: string | undefined;readonly packages: readonly string[];+ readonly out: string | undefined;};function parseArgs(argv: readonly string[]): ParsedArgs {@@ -39,6 +42,7 @@ function parseArgs(argv: readonly string[]): ParsedArgs {let packageLock: string | undefined;const packages: string[] = [];let substitute = false;+ let out: string | undefined;for (let index = 0; index < rest.length; index += 1) {const item = rest[index];if (item === "--target") {@@ -80,6 +84,13 @@ function parseArgs(argv: readonly string[]): ParsedArgs {index += 1;continue;}+ if (item === "--out") {+ const value = rest[index + 1];+ if (value === undefined || value.startsWith("--")) throw new Error("--out requires a file path");+ out = resolve(value);+ index += 1;+ continue;+ }if (item === "--build-id") {const value = rest[index + 1];if (value === undefined || value.startsWith("--")) throw new Error("--build-id requires a stable identity");@@ -115,6 +126,7 @@ function parseArgs(argv: readonly string[]): ParsedArgs {maxWaitMs,packageLock,packages,+ out,};}@@ -124,7 +136,7 @@ function usage(): string {" svml-v2 lock-packages <svml.packages.lock> --package installed-name [--package installed-name] [--root directory]"," svml-v2 check <file.svml> [--package-lock file] [--root directory]"," svml-v2 plan <file.svml> --target export [--target export] [--accept-substitute] [--package-lock file]",- " svml-v2 build <file.svml> --target export --runtime ./svml.runtime.ts [--package-lock file] [--follow]",+ " svml-v2 build <file.svml> --target export --runtime ./svml.runtime.ts [--package-lock file] [--follow] [--out video.mp4]"," svml-v2 status <build-id> --runtime ./svml.runtime.ts"," svml-v2 cancel <build-id> --runtime ./svml.runtime.ts",].join("\n");@@ -135,6 +147,8 @@ function isLocalRuntime(value: unknown): value is LocalRuntime {&& value !== null&& "build" in value&& typeof value.build === "function"+ && "readArtifact" in value+ && typeof value.readArtifact === "function"&& "close" in value&& typeof value.close === "function";}@@ -154,6 +168,33 @@ async function loadLocalRuntime(path: string): Promise<LocalRuntime> {return candidate;}+export async function materializeSingleGoal(+ runtime: Pick<LocalRuntime, "readArtifact">,+ built: Awaited<ReturnType<LocalRuntime["build"]>>,+ output: string,+): Promise<{ readonly path: string; readonly digest: string; readonly size: number }> {+ if (built.status !== "complete") throw new Error("--out requires a completed Build");+ if (built.state.plan.goals.length !== 1) throw new Error("--out requires exactly one target");+ const goal = built.state.plan.goals[0]!;+ const record = built.state.records.find((item) => item.id === goal.record);+ const value = record?.value.kind === "inline" ? record.value.value : undefined;+ if (value === null || Array.isArray(value) || typeof value !== "object") {+ throw new Error("--out target is not a materializable media Artifact");+ }+ const artifact = value as Readonly<Record<string, unknown>>;+ const digest = artifact.digest;+ const size = artifact.size;+ if (typeof digest !== "string" || !isDigest(digest) || typeof size !== "number" || !Number.isSafeInteger(size)) {+ throw new Error("--out target has no valid Artifact identity");+ }+ const bytes = await runtime.readArtifact(digest);+ if (bytes === undefined) throw new Error(`Artifact ${digest} is absent from the selected ArtifactStore`);+ if (bytes.byteLength !== size) throw new Error(`Artifact ${digest} size differs from its target Record`);+ await mkdir(dirname(output), { recursive: true });+ await writeFile(output, bytes);+ return { path: output, digest, size };+}+export async function runCli(argv: readonly string[], io: CliIo): Promise<void> {const args = parseArgs(argv);if (args.file === undefined@@ -171,6 +212,7 @@ export async function runCli(argv: readonly string[], io: CliIo): Promise<void>io.write(`${JSON.stringify({ ok: true, packageLock: output, digest: lock.digest, packages: lock.packages }, null, 2)}\n`);return;}+ if (args.out !== undefined && args.command !== "build") throw new Error("--out is only valid for build");if (args.command === "status" || args.command === "cancel") {if (args.runtime === undefined) throw new Error(`${args.command} requires --runtime`);const runtime = await loadLocalRuntime(args.runtime);@@ -248,6 +290,10 @@ export async function runCli(argv: readonly string[], io: CliIo): Promise<void>follow: args.follow,...(args.maxWaitMs === undefined ? {} : { maxWaitMs: args.maxWaitMs }),});+ let materialized: { readonly path: string; readonly digest: string; readonly size: number } | undefined;+ if (args.out !== undefined) {+ materialized = await materializeSingleGoal(runtime, built, args.out);+ }io.write(`${JSON.stringify({build: built.id,core: built.state.id,@@ -263,6 +309,7 @@ export async function runCli(argv: readonly string[], io: CliIo): Promise<void>}),blocked: built.blocked,journal: built.journal,+ ...(materialized === undefined ? {} : { output: materialized }),}, null, 2)}\n`);} finally {await runtime.close();diff --git a/packages/cli/test/cli.test.ts b/packages/cli/test/cli.test.tsindex b3604fa9..033e5863 100644--- a/packages/cli/test/cli.test.ts+++ b/packages/cli/test/cli.test.ts@@ -1,10 +1,11 @@import assert from "node:assert/strict";-import { mkdir, mkdtemp, writeFile } from "node:fs/promises";+import { createHash } from "node:crypto";+import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";import { tmpdir } from "node:os";import { join } from "node:path";import test from "node:test";-import { runCli } from "@svml/cli";+import { materializeSingleGoal, runCli } from "@svml/cli";test("official v2 CLI checks a real Script source through the Node compiler host", async () => {const root = await mkdtemp(join(tmpdir(), "svml-cli-"));@@ -44,10 +45,22 @@ test("official v2 CLI checks a real Script source through the Node compiler hostmodule: { name: "@svml/narrative", version: "0.0.0-dev" },name: "Narrative",}, kind: "record" },+ { name: "story.caption", type: {+ module: { name: "@svml/narrative", version: "0.0.0-dev" },+ name: "CaptionProjection",+ }, kind: "record" },{ name: "story.segment.opening", type: {module: { name: "@svml/narrative", version: "0.0.0-dev" },name: "NarrativeExcerpt",}, kind: "record" },+ { name: "story.segment.opening.dialogue", type: {+ module: { name: "@svml/narrative", version: "0.0.0-dev" },+ name: "NarrativeDialogueExcerpt",+ }, kind: "record" },+ { name: "story.segment.opening.speech", type: {+ module: { name: "@svml/narrative", version: "0.0.0-dev" },+ name: "NarrativeSpeechExcerpt",+ }, kind: "record" },]);});@@ -64,7 +77,7 @@ test("official media and Seedance Surfaces lower author intent into exact genera<seedance:Prompt id="direction">Locked medium close-up in a quiet daylight studio.</seedance:Prompt>- <seedance:Speech id="take" model="mini" script={story.segment.opening}+ <seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue}prompt={direction} duration="5"><seedance:Reference image={host} role="character"/></seedance:Speech>@@ -86,7 +99,10 @@ test("official media and Seedance Surfaces lower author intent into exact genera"motion.request","motion.video","story",+ "story.caption","story.segment.opening",+ "story.segment.opening.dialogue",+ "story.segment.opening.speech","take","take.request",]);@@ -113,6 +129,132 @@ test("official media and Seedance Surfaces lower author intent into exact genera);});+test("official prelude lowers Speech Spine and explicit WhisperX alignment without provider calls", async () => {+ const root = await mkdtemp(join(tmpdir(), "svml-cli-speech-spine-"));+ const file = join(root, "main.svml");+ await writeFile(file, `<svml>+ <import from="@svml/script@1"/>+ <import as="seedance" from="@svml/seedance@1"/>+ <import as="speech" from="@svml/speech@1"/>+ <import as="whisperx" from="@svml/whisperx@1"/>+ <script id="story"><opening><HOST>Meaning becomes the source.</opening></script>+ <seedance:Prompt id="direction">Locked medium close-up.</seedance:Prompt>+ <seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue}+ prompt={direction} duration="5"/>+ <speech:Spine id="speech">+ <speech:Take source={take} segment={story.segment.opening}/>+ </speech:Spine>+ <whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>+ </svml>`, "utf8");+ let output = "";+ await runCli(["check", file], { write: (text) => { output += text; } });+ const checked = JSON.parse(output) as { readonly exports: readonly { readonly name: string }[] };+ assert.equal(checked.exports.some((item) => item.name === "speech.visual"), true);+ assert.equal(checked.exports.some((item) => item.name === "timing.map"), true);+});++test("Caption Gemini Surface lowers immutable display-atom runs into one explicit planning Need", async () => {+ const root = await mkdtemp(join(tmpdir(), "svml-cli-caption-gemini-"));+ const file = join(root, "main.svml");+ await writeFile(join(root, "studio.svs"), `<sheet version="1">+ caption.base { stack-order: 70; x: 0.08; y: 0.76; width: 0.84; font: Inter;+ weight: 600; size: 58; line-height: 1; align: center; fill: #FFFFFF;+ background: #09090BCC; padding: 16 24; radius: 18; }+ </sheet>`, "utf8");+ await writeFile(file, `<svml>+ <import from="@svml/script@1"/>+ <import as="caption" from="@svml/caption@1"/>+ <import as="gemini" from="@svml/caption-gemini@1"/>+ <import as="studio" from="./studio.svs" using="@svml/svs@1"/>+ <script id="story"><dialogue><ALICE>Meaning becomes the source.<BOB>Then the graph stays explicit.</dialogue></script>+ <caption:Style id="plain" appearance={studio.caption.base}>+ <caption:Cues>Prefer short complete semantic phrases. Never cross clause punctuation.</caption:Cues>+ </caption:Style>+ <caption:Style id="emphasis" appearance={studio.caption.base}>+ <caption:Cues>Prefer short complete semantic phrases. Never cross clause punctuation.</caption:Cues>+ <caption:Field id="important" type="boolean" min-per-cue="0" max-per-cue="2">+ Select zero, one, or two words whose emphasis best communicates this Cue.+ </caption:Field>+ </caption:Style>+ <caption:Program id="caption-program" narrative={story} default={plain}>+ <caption:Use role="ALICE" style={emphasis}/>+ </caption:Program>+ <gemini:Planner id="dialogue-plan" narrative={story} program={caption-program} model="gemini-2.5-flash"/>+ </svml>`, "utf8");+ let checkedOutput = "";+ await runCli(["check", file], { write: (text) => { checkedOutput += text; } });+ const checked = JSON.parse(checkedOutput) as { readonly exports: readonly { readonly name: string }[] };+ assert.equal(checked.exports.some((item) => item.name === "caption-program"), true);+ assert.equal(checked.exports.some((item) => item.name === "dialogue-plan.plan"), true);+ let planOutput = "";+ await runCli(["plan", file, "--target", "dialogue-plan.plan"], { write: (text) => { planOutput += text; } });+ const plan = JSON.parse(planOutput) as { readonly steps: readonly { readonly producer: { readonly name: string } }[] };+ assert.deepEqual(plan.steps.map((step) => step.producer.name).sort(), [+ "compile-caption-gemini-request",+ "request-caption-gemini-plan",+ ].sort());+});++test("Track Surfaces close one complete author graph before any external execution", async () => {+ const root = await mkdtemp(join(tmpdir(), "svml-cli-complete-author-"));+ const file = join(root, "main.svml");+ await writeFile(join(root, "studio.svs"), `<sheet version="1">+ film.vertical { width: 1080; height: 1920; frame-rate: 30; background: #09090B; }+ broll.card { stack-order: 40; x: 0.1; y: 0.2; width: 0.8; height: 0.5; fit: cover;+ background: #111116; radius: 20; enter: slide-up 4f; exit: fade 4f; }+ caption.base { stack-order: 70; x: 0.08; y: 0.76; width: 0.84; font: Inter;+ weight: 600; size: 58; line-height: 1; align: center; fill: #FFFFFF;+ background: #09090BCC; padding: 16 24; radius: 18; }+ text.title { stack-order: 90; x: 0.06; y: 0.06; width: 0.88; height: 0.1;+ font: Inter; weight: 900; size: 64; align: center; fill: #FFFFFF; tracking: -1; }+ </sheet>`, "utf8");+ await writeFile(file, `<svml>+ <import from="@svml/script@1"/>+ <import as="seedance" from="@svml/seedance@1"/>+ <import as="speech" from="@svml/speech@1"/>+ <import as="whisperx" from="@svml/whisperx@1"/>+ <import as="broll" from="@svml/broll@1"/>+ <import as="caption" from="@svml/caption@1"/>+ <import as="caption-ai" from="@svml/caption-gemini@1"/>+ <import as="text" from="@svml/text-track@1"/>+ <import as="film" from="@svml/film@1"/>+ <import as="render" from="@svml/hyperframes-render@1"/>+ <import as="studio" from="./studio.svs" using="@svml/svs@1"/>+ <script id="story"><opening><HOST>Meaning @demo becomes the source @/demo.</opening></script>+ <seedance:Prompt id="direction">Locked medium close-up.</seedance:Prompt>+ <seedance:Speech id="take" model="mini" dialogue={story.segment.opening.dialogue} prompt={direction} duration="5"/>+ <seedance:Video id="motion" model="mini" prompt={direction} duration="5"/>+ <speech:Spine id="speech"><speech:Take source={take} segment={story.segment.opening}/></speech:Spine>+ <whisperx:Alignment id="timing" narrative={story} audio={speech.audio}/>+ <caption:Style id="base-caption" appearance={studio.caption.base}>+ <caption:Cues>Prefer short complete semantic phrases.</caption:Cues>+ <caption:Field id="important" type="boolean" min-per-cue="0" max-per-cue="2">+ Select zero, one, or two words whose emphasis best communicates this Cue.+ </caption:Field>+ </caption:Style>+ <caption:Program id="caption-program" narrative={story} default={base-caption}/>+ <caption-ai:Planner id="cue-plan" narrative={story} program={caption-program} model="gemini-2.5-flash"/>+ <broll:Track id="cards" map={timing.map}>+ <broll:Item source={motion.video} during={story.selection.demo} appearance={studio.broll.card}/>+ </broll:Track>+ <caption:Track id="captions" narrative={story} map={timing.map} plan={cue-plan.plan} program={caption-program}/>+ <text:Track id="titles" space={speech.space}>+ <text:Item text="MEANING" during="full" appearance={studio.text.title}/>+ </text:Track>+ <film:Film id="main" space={speech.space} appearance={studio.film.vertical}>+ <film:Track source={speech.visual}/><film:Track source={speech.audioTrack}/>+ <film:Track source={cards.visual}/><film:Track source={captions.track}/><film:Track source={titles.track}/>+ </film:Film>+ <render:Video id="final" composition={main.composition}/>+ </svml>`, "utf8");+ let output = "";+ await runCli(["check", file], { write: (text) => { output += text; } });+ const checked = JSON.parse(output) as { readonly exports: readonly { readonly name: string }[] };+ for (const name of ["story.selection.demo", "cue-plan.plan", "cards.visual", "captions.track", "titles.track", "final.video"]) {+ assert.equal(checked.exports.some((item) => item.name === name), true, name);+ }+});+test("official v2 CLI closes the explicit HyperFrames render package without loading a Provider", async () => {const root = await mkdtemp(join(tmpdir(), "svml-cli-render-"));const file = join(root, "main.svml");@@ -194,3 +336,26 @@ test("CLI package lock activates an installed package without changing the officassert.equal(checked.ok, true);assert.deepEqual(checked.modules, ["example.empty@1"]);});++test("materializeSingleGoal writes one verified target through the Runtime ArtifactStore seam", async () => {+ const root = await mkdtemp(join(tmpdir(), "svml-cli-out-"));+ const bytes = Buffer.from("final-video-bytes");+ const artifactDigest = `sha256:${createHash("sha256").update(bytes).digest("hex")}`;+ const output = join(root, "out", "final.mp4");+ const runtime = {+ async readArtifact(digest: string) { return digest === artifactDigest ? bytes : undefined; },+ } as Parameters<typeof materializeSingleGoal>[0];+ const built = {+ status: "complete",+ state: {+ plan: { goals: [{ record: "final.video" }] },+ records: [{+ id: "final.video",+ value: { kind: "inline", value: { digest: artifactDigest, size: bytes.byteLength, mediaType: "video/mp4" } },+ }],+ },+ } as unknown as Parameters<typeof materializeSingleGoal>[1];+ const result = await materializeSingleGoal(runtime, built, output);+ assert.deepEqual(await readFile(output), bytes);+ assert.deepEqual(result, { path: output, digest: artifactDigest, size: bytes.byteLength });+});diff --git a/packages/contracts/README.md b/packages/contracts/README.mdindex bc979e2e..e32ab5c0 100644--- a/packages/contracts/README.md+++ b/packages/contracts/README.md@@ -14,10 +14,13 @@ and may later become `@svml/video` without changing those logical identities. Thcontains Provider- and Frontend-neutral data contracts, identity validators and static schemasonly. It does not parse SVML, execute external services, align speech or render Tracks.-`NarrativeExcerpt` is the shared contiguous author-selection contract used by-`script.segment.<id>` and speech-capable generation packages. It keeps model packages independent-of the official Script parser; the actual `<media:Image>` Surface implementation likewise lives in-the separate physical `@svml/media` package.+The Narrative module provides distinct narrow views instead of making every consumer depend on one+parser-specific object: `NarrativeExcerpt` associates a Take with a Segment,+`NarrativeDialogueExcerpt` carries Role-aware spoken prompt text,+`NarrativeSpeechExcerpt` carries pronunciation-only estimate text, and `CaptionProjectionRef`+identifies the whole display-side Caption truth. They keep model and Caption packages independent of+the official Script parser. The actual `<media:Image>` Surface implementation likewise lives in the+separate physical `@svml/media` package.`VisualTrack`, `AudioTrack` and `Composition` implement the flat composition law in[`../../spec/track-composition-v1.md`](../../spec/track-composition-v1.md). Caption, Speech anddiff --git a/packages/contracts/src/index.ts b/packages/contracts/src/index.tsindex d9f4c0c3..2ace3cdc 100644--- a/packages/contracts/src/index.ts+++ b/packages/contracts/src/index.ts@@ -80,6 +80,10 @@ export {timelineAudioSchema,narrativeSchema,narrativeExcerptSchema,+ narrativeDialogueExcerptSchema,+ narrativeSpeechExcerptSchema,+ narrativeSelectionSchema,+ captionProjectionSchema,programSpaceSchema,speechAudioBasisSchema,speechEvidenceAudioSchema,diff --git a/packages/contracts/src/manifest.ts b/packages/contracts/src/manifest.tsindex 34b196c1..9dbb659c 100644--- a/packages/contracts/src/manifest.ts+++ b/packages/contracts/src/manifest.ts@@ -17,6 +17,10 @@ export const compositionModuleRef = { name: "@svml/composition", version: "0.0.0export const contractTypes = {narrative: { module: narrativeModuleRef, name: "Narrative" },narrativeExcerpt: { module: narrativeModuleRef, name: "NarrativeExcerpt" },+ narrativeDialogueExcerpt: { module: narrativeModuleRef, name: "NarrativeDialogueExcerpt" },+ narrativeSpeechExcerpt: { module: narrativeModuleRef, name: "NarrativeSpeechExcerpt" },+ narrativeSelection: { module: narrativeModuleRef, name: "NarrativeSelection" },+ captionProjection: { module: narrativeModuleRef, name: "CaptionProjection" },blobArtifact: { module: mediaModuleRef, name: "BlobArtifact" },mediaArtifact: { module: mediaModuleRef, name: "MediaArtifactRef" },mediaInspection: { module: mediaModuleRef, name: "MediaInspection" },@@ -127,6 +131,51 @@ export const narrativeExcerptSchema: ValueSchema = object({excerptDigest: { schema: digest },});+export const narrativeDialogueExcerptSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.narrative-dialogue-excerpt@1" } },+ kind: { schema: { kind: "literal", value: "segment" } },+ id: { schema: string },+ tokenStart: { schema: integer },+ tokenEndExclusive: { schema: integer },+ dialogue: { schema: string },+ excerptDigest: { schema: digest },+});++export const narrativeSpeechExcerptSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.narrative-speech-excerpt@1" } },+ kind: { schema: { kind: "literal", value: "segment" } },+ id: { schema: string },+ tokenStart: { schema: integer },+ tokenEndExclusive: { schema: integer },+ speech: { schema: string },+ excerptDigest: { schema: digest },+});++export const captionProjectionSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.caption-projection@0" } },+ text: { schema: { kind: "string" } },+ regions: { schema: { kind: "array", items: object({+ id: { schema: string }, display: { schema: { kind: "string" } }, segmentId: { schema: string },+ startToken: { schema: integer }, endTokenExclusive: { schema: integer },+ kind: { schema: { kind: "string", enum: ["identity", "alias", "hidden"] } },+ refinements: { schema: { kind: "array", items: object({+ id: { schema: string }, display: { schema: string }, displayStart: { schema: integer },+ displayEnd: { schema: integer }, startToken: { schema: integer }, endTokenExclusive: { schema: integer },+ relation: { schema: { kind: "literal", value: "exact" } },+ }) } },+ }) } },+ projectionDigest: { schema: digest },+});++export const narrativeSelectionSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.narrative-selection@1" } },+ id: { schema: string },+ occurrences: { schema: { kind: "array", minItems: 1, items: object({+ occurrence: { schema: integer }, open: { schema: markerEdge }, close: { schema: markerEdge },+ }) } },+ selectionDigest: { schema: digest },+});+export const mediaArtifactSchema = object({digest: { schema: digest }, size: { schema: integer }, mediaType: { schema: string }, durationSec: { schema: number },});@@ -543,6 +592,10 @@ export const narrativeManifest: ModuleManifest = {types: [{ name: contractTypes.narrative.name, schema: narrativeSchema },{ name: contractTypes.narrativeExcerpt.name, schema: narrativeExcerptSchema },+ { name: contractTypes.narrativeDialogueExcerpt.name, schema: narrativeDialogueExcerptSchema },+ { name: contractTypes.narrativeSpeechExcerpt.name, schema: narrativeSpeechExcerptSchema },+ { name: contractTypes.narrativeSelection.name, schema: narrativeSelectionSchema },+ { name: contractTypes.captionProjection.name, schema: captionProjectionSchema },],capabilities: [],surfaces: [],diff --git a/packages/contracts/src/narrative.ts b/packages/contracts/src/narrative.tsindex 6054ae19..4b80943b 100644--- a/packages/contracts/src/narrative.ts+++ b/packages/contracts/src/narrative.ts@@ -45,6 +45,12 @@ export type NarrativeSelection = {readonly occurrences: readonly NarrativeSelectionOccurrence[];};+/** One explicitly authored semantic window, independently referenceable by graph edges. */+export type NarrativeSelectionRef = NarrativeSelection & {+ readonly contract: "svml.narrative-selection@1";+ readonly selectionDigest: Digest;+};+export type NarrativeMomentOccurrence = {readonly occurrence: number;readonly affinity: Affinity;@@ -82,6 +88,11 @@ export type CaptionProjection = {readonly regions: readonly CaptionRegion[];};+/** Whole authored display projection. It contains no pronunciation replacement text. */+export type CaptionProjectionRef = CaptionProjection & {+ readonly projectionDigest: Digest;+};+export type SemanticAnchor = {readonly id: string;readonly kind: "segment-start" | "token-start" | "token-end" | "segment-end";@@ -126,4 +137,26 @@ export type NarrativeExcerpt = {};readonly excerptDigest: Digest;};++/** The exact dialogue prompt for one Segment, including only explicitly authored Role prefixes. */+export type NarrativeDialogueExcerpt = {+ readonly contract: "svml.narrative-dialogue-excerpt@1";+ readonly kind: "segment";+ readonly id: string;+ readonly tokenStart: number;+ readonly tokenEndExclusive: number;+ readonly dialogue: string;+ readonly excerptDigest: Digest;+};++/** The exact spoken wording for one Segment, without Role prefixes. */+export type NarrativeSpeechExcerpt = {+ readonly contract: "svml.narrative-speech-excerpt@1";+ readonly kind: "segment";+ readonly id: string;+ readonly tokenStart: number;+ readonly tokenEndExclusive: number;+ readonly speech: string;+ readonly excerptDigest: Digest;+};import type { Digest } from "@svml/protocol";diff --git a/packages/film/package.json b/packages/film/package.jsonindex 5a72c96e..fd2fd720 100644--- a/packages/film/package.json+++ b/packages/film/package.json@@ -7,6 +7,7 @@".": "./src/index.ts"},"dependencies": {+ "@svml/component-kit": "workspace:*","@svml/contracts": "workspace:*","@svml/elaborator": "workspace:*","@svml/protocol": "workspace:*",diff --git a/packages/film/src/component.ts b/packages/film/src/component.tsnew file mode 100644index 00000000..2c7e1267--- /dev/null+++ b/packages/film/src/component.ts@@ -0,0 +1,66 @@+import type { ComponentPackage } from "@svml/component-kit";+import type { AudioTrack, ProgramSpace, VisualTrack } from "@svml/contracts";+import type { StoredValue } from "@svml/protocol";+import { canonicalize } from "@svml/protocol";++import { filmProducers } from "./manifest.js";+import {+ appendFilmAudioTrack,+ appendFilmAudioTrackImplementationDigest,+ appendFilmVisualTrack,+ appendFilmVisualTrackImplementationDigest,+ compileFilmComposition,+ compileFilmCompositionImplementationDigest,+ createFilmTrackSet,+ createFilmTrackSetImplementationDigest,+} from "./program.js";+import type { FilmProgram, FilmTrackSet } from "./types.js";++function inline<T>(value: StoredValue | undefined, subject: string): T {+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);+ return value.value as unknown as T;+}++export const filmComponent = {+ name: "@svml/film",+ producers: [+ {+ producer: filmProducers.createTrackSet,+ implementationDigest: createFilmTrackSetImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(createFilmTrackSet(inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"))) } },+ needs: {},+ }),+ },+ {+ producer: filmProducers.appendVisualTrack,+ implementationDigest: appendFilmVisualTrackImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(appendFilmVisualTrack(+ inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"), inline<VisualTrack>(inputs.track?.value, "VisualTrack"),+ )) } },+ needs: {},+ }),+ },+ {+ producer: filmProducers.appendAudioTrack,+ implementationDigest: appendFilmAudioTrackImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(appendFilmAudioTrack(+ inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"), inline<AudioTrack>(inputs.track?.value, "AudioTrack"),+ )) } },+ needs: {},+ }),+ },+ {+ producer: filmProducers.compileComposition,+ implementationDigest: compileFilmCompositionImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { composition: { kind: "inline", value: canonicalize(compileFilmComposition(+ inline<FilmProgram>(inputs.program?.value, "FilmProgram"), inline<FilmTrackSet>(inputs.set?.value, "FilmTrackSet"),+ )) } },+ needs: {},+ }),+ },+ ],+} satisfies ComponentPackage;diff --git a/packages/film/src/index.ts b/packages/film/src/index.tsindex cba24996..f83b4a77 100644--- a/packages/film/src/index.ts+++ b/packages/film/src/index.ts@@ -1,4 +1,5 @@export { createFilmAssemblyFragment } from "./fragment.js";+export { filmComponent } from "./component.js";export {filmManifest,filmManifestDigest,diff --git a/packages/local/README.md b/packages/local/README.mdindex c65fd1fc..85c12848 100644--- a/packages/local/README.md+++ b/packages/local/README.md@@ -46,6 +46,9 @@ When several instances provide the same role, `runtimeSelection` must name the eThe same mechanism covers Postgres, S3, keychains and replacement Schedulers; none requires achange to `@svml/local`. Advanced hosts may still call `createLocalRuntime` with raw ports.+`LocalRuntime.readArtifact(digest)` is the generic egress seam used by CLI `--out`. It exposes+content-addressed bytes, not filesystem layout, so filesystem and S3 stores remain interchangeable.+`LocalBuildRequest.attachments` is the explicit ingress from a trusted Host into the selectedArtifactStore. Each attachment carries claimed `BlobRef` metadata plus bytes; Local Runtime copiesthe bytes, stores them content-addressably and requires the Store's returned digest, size and mediadiff --git a/packages/local/src/runtime.ts b/packages/local/src/runtime.tsindex 8d8ba53a..54ecaafb 100644--- a/packages/local/src/runtime.ts+++ b/packages/local/src/runtime.ts@@ -222,6 +222,9 @@ export async function createLocalRuntime(const [result] = await scheduler.run([{ id: build, state: snapshot.state }]);return result;},+ async readArtifact(digest) {+ return await options.artifactStore.get(digest);+ },close() {},};}@@ -302,6 +305,7 @@ export async function createProjectLocalRuntime(buildMany: runtime.buildMany,status: runtime.status,cancel: runtime.cancel,+ readArtifact: runtime.readArtifact,close() {return services.close();},diff --git a/packages/local/src/types.ts b/packages/local/src/types.tsindex 0abbb4ca..a5535430 100644--- a/packages/local/src/types.ts+++ b/packages/local/src/types.ts@@ -2,6 +2,7 @@ import type { Awaitable, ComponentPackage } from "@svml/component-kit";import type { ArtifactAttachment } from "@svml/host";import type { EndpointPackage } from "@svml/endpoint-kit";import type { BuildState } from "@svml/protocol";+import type { Digest } from "@svml/protocol";import type {ArtifactStore,BuildSchedulerFactory,@@ -103,5 +104,6 @@ export type LocalRuntime = {buildMany(requests: readonly LocalBuildRequest[]): Promise<readonly ScheduledBuildResult[]>;status(build: string): Promise<LocalRuntimeStatus>;cancel(build: string): Promise<ScheduledBuildResult | undefined>;+ readArtifact(digest: Digest): Promise<Uint8Array | undefined>;close(): Awaitable<void>;};diff --git a/packages/prelude-video/package.json b/packages/prelude-video/package.jsonindex a706fdff..7bbcc007 100644--- a/packages/prelude-video/package.json+++ b/packages/prelude-video/package.json@@ -10,7 +10,9 @@"activation": "./src/index.ts"},"dependencies": {+ "@svml/broll": "workspace:*","@svml/caption": "workspace:*",+ "@svml/caption-gemini": "workspace:*","@svml/contracts": "workspace:*","@svml/generation": "workspace:*","@svml/film": "workspace:*",@@ -21,8 +23,11 @@"@svml/script": "workspace:*","@svml/seedance": "workspace:*","@svml/speech-align": "workspace:*",+ "@svml/speech-program": "workspace:*","@svml/speech-take": "workspace:*",- "@svml/svs": "workspace:*"+ "@svml/svs": "workspace:*",+ "@svml/text-track": "workspace:*",+ "@svml/whisperx": "workspace:*"},"devDependencies": {"@svml/package-loader-node": "workspace:*"diff --git a/packages/prelude-video/src/index.ts b/packages/prelude-video/src/index.tsindex e4b39952..fee77157 100644--- a/packages/prelude-video/src/index.ts+++ b/packages/prelude-video/src/index.ts@@ -6,12 +6,34 @@ import {generationComponent,generationManifest,} from "@svml/generation";+import {+ brollComponent,+ brollManifest,+ brollModuleRef,+ brollSurfaceImplementationDigest,+ decodeBrollTrackSurface,+} from "@svml/broll";import {captionComponent,captionManifest,+ captionModuleRef,+ captionProgramSurfaceImplementationDigest,+ captionSurfaceImplementationDigest,+ captionStyleSurfaceImplementationDigest,+ decodeCaptionProgramSurface,+ decodeCaptionStyleSurface,+ decodeCaptionTrackSurface,} from "@svml/caption";+import {+ captionGeminiComponent,+ captionGeminiImplementationDigests,+ captionGeminiManifest,+ captionGeminiModuleRef,+ decodeCaptionGeminiPlannerSurface,+} from "@svml/caption-gemini";import {decodeFilmSurface,+ filmComponent,filmManifest,filmModuleRef,filmSurfaceImplementationDigest,@@ -60,7 +82,28 @@ import {speechTakeComponent,speechTakeManifest,} from "@svml/speech-take";+import {+ decodeSpeechSpineSurface,+ speechProgramComponent,+ speechProgramManifest,+ speechProgramModuleRef,+ speechSpineSurfaceImplementationDigest,+} from "@svml/speech-program";import { svsFrontend, svsManifest } from "@svml/svs";+import {+ decodeTextTrackSurface,+ textTrackComponent,+ textTrackManifest,+ textTrackModuleRef,+ textTrackSurfaceImplementationDigest,+} from "@svml/text-track";+import {+ decodeWhisperXAlignmentSurface,+ whisperXComponent,+ whisperXImplementationDigests,+ whisperXManifest,+ whisperXModuleRef,+} from "@svml/whisperx";export const svmlPackage: NodePackageActivation = {format: "svml.node-package@1",@@ -78,9 +121,17 @@ export const svmlPackage: NodePackageActivation = {{ manifest: svsManifest },{ manifest: hyperframesManifest },{ manifest: mediaPipelineManifest },- { manifest: captionManifest },+ { manifest: captionManifest, specifiers: ["@svml/caption", "@svml/caption@1"] },+ {+ manifest: captionGeminiManifest,+ specifiers: ["@svml/caption-gemini", "@svml/caption-gemini@1"],+ },{ manifest: speechAlignManifest },{ manifest: speechTakeManifest },+ { manifest: speechProgramManifest, specifiers: ["@svml/speech", "@svml/speech@1"] },+ { manifest: whisperXManifest, specifiers: ["@svml/whisperx", "@svml/whisperx@1"] },+ { manifest: textTrackManifest, specifiers: ["@svml/text-track", "@svml/text-track@1"] },+ { manifest: brollManifest, specifiers: ["@svml/broll", "@svml/broll@1"] },{ manifest: filmManifest, specifiers: ["@svml/film", "@svml/film@1"] },{manifest: hyperframesRenderManifest,@@ -93,9 +144,15 @@ export const svmlPackage: NodePackageActivation = {generationComponent,seedanceComponent,captionComponent,+ captionGeminiComponent,mediaPipelineComponent,speechAlignComponent,speechTakeComponent,+ speechProgramComponent,+ whisperXComponent,+ textTrackComponent,+ brollComponent,+ filmComponent,hyperframesComponent,hyperframesRenderComponent,],@@ -135,6 +192,62 @@ export const svmlPackage: NodePackageActivation = {implementationDigest: scriptSurfaceImplementationDigest,handler: decodeScriptSurface,},+ {+ module: captionModuleRef,+ surface: "style",+ mode: "structured",+ implementationDigest: captionStyleSurfaceImplementationDigest,+ handler: decodeCaptionStyleSurface,+ },+ {+ module: captionModuleRef,+ surface: "program",+ mode: "structured",+ implementationDigest: captionProgramSurfaceImplementationDigest,+ handler: decodeCaptionProgramSurface,+ },+ {+ module: captionModuleRef,+ surface: "track",+ mode: "structured",+ implementationDigest: captionSurfaceImplementationDigest,+ handler: decodeCaptionTrackSurface,+ },+ {+ module: captionGeminiModuleRef,+ surface: "planner",+ mode: "structured",+ implementationDigest: captionGeminiImplementationDigests.plannerSurface,+ handler: decodeCaptionGeminiPlannerSurface,+ },+ {+ module: speechProgramModuleRef,+ surface: "spine",+ mode: "structured",+ implementationDigest: speechSpineSurfaceImplementationDigest,+ handler: decodeSpeechSpineSurface,+ },+ {+ module: whisperXModuleRef,+ surface: "alignment",+ mode: "structured",+ implementationDigest: whisperXImplementationDigests.surface,+ handler: decodeWhisperXAlignmentSurface,+ },+ {+ module: brollModuleRef,+ surface: "track",+ mode: "structured",+ implementationDigest: brollSurfaceImplementationDigest,+ handler: decodeBrollTrackSurface,+ },+ {+ module: textTrackModuleRef,+ surface: "track",+ mode: "structured",+ implementationDigest: textTrackSurfaceImplementationDigest,+ handler: decodeTextTrackSurface,+ },{module: filmModuleRef,surface: "film",diff --git a/packages/prelude-video/test/activation.test.ts b/packages/prelude-video/test/activation.test.tsindex 9878787e..40cefc5a 100644--- a/packages/prelude-video/test/activation.test.ts+++ b/packages/prelude-video/test/activation.test.ts@@ -66,11 +66,11 @@ test("the official prelude activates Caption compute and owner validators withouassert.equal(components.includes(captionComponent), true);assert.deepEqual(captionComponent.producers.map((facet) => facet.producer.name),- ["temporalize-caption", "render-caption-track"],+ ["temporalize-caption", "temporalize-caption-plan", "render-caption-program", "render-caption-track"],);assert.deepEqual(captionComponent.validators?.map((facet) => facet.type.name),- ["TimedCaptionProjection", "CaptionTrackProgram"],+ ["CaptionStyle", "CaptionProgram", "CaptionPlan", "TimedCaptionProjection", "CaptionTrackProgram"],);});diff --git a/packages/provider-google-vertex/README.md b/packages/provider-google-vertex/README.mdnew file mode 100644index 00000000..66af05a1--- /dev/null+++ b/packages/provider-google-vertex/README.md@@ -0,0 +1,25 @@+# @svml/provider-google-vertex++Google Vertex transport for the exact `@svml/caption-gemini` capability.++This package does not build prompts, parse SVML, inspect WhisperX, choose a model, or rewrite text.+It receives the model package's immutable request, calls `GoogleGenAI({ vertexai: true })` with+structured JSON output, and returns the response only after `@svml/caption-gemini` validates every+Run, Cue endpoint, atom assignment, field value and per-Cue cardinality.++```ts+createGoogleVertexCaptionProvider({+ project: "my-google-cloud-project",+ location: "global",+});+```++The default credential reference is the explicit environment key+`GOOGLE_APPLICATION_CREDENTIALS_JSON`, whose value is the JSON contents rather than a path. This+keeps filesystem authority out of the Provider. A deployment may bind another `CredentialStore`+and pass its `CredentialRef`. Credentials never enter author source, Runtime Closure, BuildState,+Operation state, Prompt or Receipt metadata.++The old `twinit` implementation used `GOOGLE_APPLICATION_CREDENTIALS` as an ADC file path. A host+may preserve that deployment mechanism in a file-backed CredentialStore, but the Provider itself+does not read arbitrary paths or ambient environment variables.diff --git a/packages/provider-google-vertex/package.json b/packages/provider-google-vertex/package.jsonnew file mode 100644index 00000000..6200a24a--- /dev/null+++ b/packages/provider-google-vertex/package.json@@ -0,0 +1,21 @@+{+ "name": "@svml/provider-google-vertex",+ "version": "0.0.0-dev",+ "private": true,+ "type": "module",+ "exports": {+ ".": "./src/index.ts"+ },+ "dependencies": {+ "@google/genai": "1.52.0",+ "@svml/caption": "workspace:*",+ "@svml/caption-gemini": "workspace:*",+ "@svml/endpoint-kit": "workspace:*",+ "@svml/protocol": "workspace:*",+ "@svml/runtime": "workspace:*"+ },+ "devDependencies": {+ "@svml/driver-node": "workspace:*",+ "@svml/script": "workspace:*"+ }+}diff --git a/packages/provider-google-vertex/src/index.ts b/packages/provider-google-vertex/src/index.tsnew file mode 100644index 00000000..5916a153--- /dev/null+++ b/packages/provider-google-vertex/src/index.ts@@ -0,0 +1,9 @@+export {+ createGoogleVertexCaptionProvider,+ googleVertexProviderImplementationDigest,+ googleVertexProviderModuleRef,+} from "./provider.js";+export type {+ CreateGoogleVertexCaptionProviderOptions,+ GenerateCaptionContent,+} from "./provider.js";diff --git a/packages/provider-google-vertex/src/provider.ts b/packages/provider-google-vertex/src/provider.tsnew file mode 100644index 00000000..10181cb7--- /dev/null+++ b/packages/provider-google-vertex/src/provider.ts@@ -0,0 +1,268 @@+import { GoogleGenAI, Type } from "@google/genai";+import type { GenerateContentResponse, GoogleGenAIOptions, Schema } from "@google/genai";+import { captionTypes } from "@svml/caption";+import {+ captionGeminiCapabilities,+ sealCaptionGeminiPlan,+ verifyCaptionGeminiRequest,+} from "@svml/caption-gemini";+import type { CaptionGeminiRequest } from "@svml/caption-gemini";+import { defineEndpointPackage } from "@svml/endpoint-kit";+import type { EndpointFulfillment, EndpointInvocationContext } from "@svml/endpoint-kit";+import { canonicalize, digestOf, isDigest } from "@svml/protocol";+import type { CanonicalValue, Digest } from "@svml/protocol";+import { credentialRef } from "@svml/runtime";+import type { CredentialRef } from "@svml/runtime";++export const googleVertexProviderModuleRef = {+ name: "@svml/provider-google-vertex",+ version: "0.0.0-dev",+} as const;+export const googleVertexProviderImplementationDigest = digestOf(+ "@svml/provider-google-vertex/caption-gemini@2:@google/[email protected]",+);++type GenerateCaptionContentInput = {+ readonly model: string;+ readonly systemInstruction: string;+ readonly prompt: string;+ readonly temperature: number;+ readonly responseSchema: Schema;+ readonly abortSignal: AbortSignal;+ readonly credentialsJson: Readonly<Record<string, unknown>>;+ readonly project: string;+ readonly location: string;+};++export type GenerateCaptionContent = (+ input: GenerateCaptionContentInput,+) => Promise<{ readonly text: string; readonly totalTokenCount?: number; readonly modelVersion?: string }>;++export type CreateGoogleVertexCaptionProviderOptions = {+ readonly project: string;+ readonly location?: string;+ readonly instance?: string;+ readonly lane?: string;+ /** JSON contents, not a filesystem path. A CredentialStore decides where these bytes live. */+ readonly credentialsJson?: CredentialRef;+ readonly defaultConcurrency?: number;+ readonly requestTimeoutMs?: number;+ readonly maxResponseBytes?: number;+ /** Test/private transport injection; its identity must be locked into Runtime configuration. */+ readonly generateContent?: GenerateCaptionContent;+ readonly generateContentImplementationDigest?: Digest;+};++const captionResponseSchema: Schema = {+ type: Type.OBJECT,+ properties: {+ runs: {+ type: Type.ARRAY,+ items: {+ type: Type.OBJECT,+ properties: {+ run_id: { type: Type.STRING },+ cues: {+ type: Type.ARRAY,+ items: {+ type: Type.OBJECT,+ properties: {+ after_atom_id: { type: Type.STRING },+ fields: {+ type: Type.ARRAY,+ items: {+ type: Type.OBJECT,+ properties: {+ declaration_id: { type: Type.STRING },+ atom_id: { type: Type.STRING },+ value: { type: Type.STRING },+ },+ required: ["declaration_id", "atom_id", "value"],+ propertyOrdering: ["declaration_id", "atom_id", "value"],+ },+ },+ },+ required: ["after_atom_id", "fields"],+ propertyOrdering: ["after_atom_id", "fields"],+ },+ },+ },+ required: ["run_id", "cues"],+ propertyOrdering: ["run_id", "cues"],+ },+ },+ },+ required: ["runs"],+ propertyOrdering: ["runs"],+};++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function positiveInteger(value: number, subject: string): number {+ assert(Number.isSafeInteger(value) && value > 0, `${subject} must be a positive integer`);+ return value;+}++function credentialJson(context: EndpointInvocationContext): Readonly<Record<string, unknown>> {+ const secret = context.credentials.googleCredentials?.secret;+ assert(secret !== undefined && secret.length > 0, "Google Vertex credentials JSON is unavailable");+ let parsed: unknown;+ try {+ parsed = JSON.parse(secret);+ } catch {+ throw new Error("Google Vertex credentials are not valid JSON");+ }+ assert(parsed !== null && typeof parsed === "object" && !Array.isArray(parsed),+ "Google Vertex credentials JSON must be an object");+ const value = parsed as Readonly<Record<string, unknown>>;+ assert(typeof value.type === "string" && value.type.length > 0,+ "Google Vertex credentials JSON has no credential type");+ return value;+}++function requestValue(value: CanonicalValue): CaptionGeminiRequest {+ verifyCaptionGeminiRequest(value);+ return value;+}++function sdkGenerateContent(): GenerateCaptionContent {+ return async (input) => {+ const googleAuthOptions = {+ credentials: input.credentialsJson,+ scopes: ["https://www.googleapis.com/auth/cloud-platform"],+ } as NonNullable<GoogleGenAIOptions["googleAuthOptions"]>;+ const client = new GoogleGenAI({+ vertexai: true,+ project: input.project,+ location: input.location,+ googleAuthOptions,+ });+ const response: GenerateContentResponse = await client.models.generateContent({+ model: input.model,+ contents: [{ role: "user", parts: [{ text: input.prompt }] }],+ config: {+ systemInstruction: input.systemInstruction,+ temperature: input.temperature,+ responseMimeType: "application/json",+ responseSchema: input.responseSchema,+ abortSignal: input.abortSignal,+ },+ });+ return {+ text: response.text ?? "",+ ...(response.usageMetadata?.totalTokenCount === undefined+ ? {} : { totalTokenCount: response.usageMetadata.totalTokenCount }),+ ...(response.modelVersion === undefined ? {} : { modelVersion: response.modelVersion }),+ };+ };+}++async function withTimeout<T>(timeoutMs: number, task: (signal: AbortSignal) => Promise<T>): Promise<T> {+ const controller = new AbortController();+ let timer: ReturnType<typeof setTimeout> | undefined;+ try {+ return await Promise.race([+ task(controller.signal),+ new Promise<never>((_, reject) => {+ timer = setTimeout(() => {+ controller.abort(new Error("Google Vertex request timed out"));+ reject(new Error("Google Vertex request timed out"));+ }, timeoutMs);+ }),+ ]);+ } finally {+ if (timer !== undefined) clearTimeout(timer);+ }+}++function fulfillment(value: CanonicalValue, metadata: CanonicalValue): EndpointFulfillment {+ return {+ value: { kind: "inline", value },+ conformance: "exact",+ delivery: "executed",+ metadata,+ };+}++export function createGoogleVertexCaptionProvider(options: CreateGoogleVertexCaptionProviderOptions) {+ assert(options.project.trim().length > 0, "Google Vertex project is empty");+ const project = options.project.trim();+ const location = (options.location ?? "global").trim();+ assert(location.length > 0, "Google Vertex location is empty");+ const requestTimeoutMs = positiveInteger(options.requestTimeoutMs ?? 120_000, "requestTimeoutMs");+ const maxResponseBytes = positiveInteger(options.maxResponseBytes ?? 2_000_000, "maxResponseBytes");+ if (options.generateContent !== undefined && options.generateContentImplementationDigest === undefined) {+ throw new Error("custom Google Vertex generateContent requires generateContentImplementationDigest");+ }+ if (options.generateContentImplementationDigest !== undefined && !isDigest(options.generateContentImplementationDigest)) {+ throw new Error("generateContentImplementationDigest is invalid");+ }+ const generateContent = options.generateContent ?? sdkGenerateContent();+ const transportDigest = options.generateContentImplementationDigest+ ?? digestOf("@svml/provider-google-vertex/@google-genai-1.52.0@1");++ return defineEndpointPackage({+ module: googleVertexProviderModuleRef,+ facet: "caption-gemini",+ instance: options.instance ?? "google-vertex.caption",+ ...(options.lane === undefined ? {} : { lane: options.lane }),+ implementation: {+ locator: "@svml/provider-google-vertex/caption-gemini",+ digest: googleVertexProviderImplementationDigest,+ },+ permissions: ["network:aiplatform.googleapis.com"],+ configuration: canonicalize({ project, location, requestTimeoutMs, maxResponseBytes, transportDigest }),+ credentials: {+ googleCredentials: options.credentialsJson ?? credentialRef("env", "GOOGLE_APPLICATION_CREDENTIALS_JSON"),+ },+ defaultConcurrency: options.defaultConcurrency ?? 2,+ capabilities: [{+ capability: captionGeminiCapabilities.plan,+ returns: captionTypes.plan,+ lifecycle: "immediate",+ supports: (need) => {+ try {+ verifyCaptionGeminiRequest(need.constraints);+ return true;+ } catch {+ return false;+ }+ },+ handler: async (context) => {+ const request = requestValue(context.need.constraints);+ const response = await withTimeout(requestTimeoutMs, (abortSignal) => generateContent({+ model: request.model,+ systemInstruction: request.systemInstruction,+ prompt: request.prompt,+ temperature: request.temperature,+ responseSchema: captionResponseSchema,+ abortSignal,+ credentialsJson: credentialJson(context),+ project,+ location,+ }));+ const responseBytes = Buffer.byteLength(response.text, "utf8");+ assert(responseBytes > 0, "Google Vertex returned an empty Caption plan");+ assert(responseBytes <= maxResponseBytes, "Google Vertex Caption response exceeded its configured limit");+ let raw: unknown;+ try {+ raw = JSON.parse(response.text);+ } catch {+ throw new Error("Google Vertex returned invalid Caption JSON");+ }+ const plan = sealCaptionGeminiPlan(request, raw);+ return fulfillment(canonicalize(plan), canonicalize({+ provider: "google-vertex",+ project,+ location,+ requestedModel: request.model,+ ...(response.modelVersion === undefined ? {} : { modelVersion: response.modelVersion }),+ ...(response.totalTokenCount === undefined ? {} : { totalTokenCount: response.totalTokenCount }),+ responseDigest: digestOf(response.text),+ }));+ },+ }],+ });+}diff --git a/packages/provider-google-vertex/test/provider.test.ts b/packages/provider-google-vertex/test/provider.test.tsnew file mode 100644index 00000000..a656731e--- /dev/null+++ b/packages/provider-google-vertex/test/provider.test.ts@@ -0,0 +1,131 @@+import assert from "node:assert/strict";+import test from "node:test";++import {+ captionGeminiCapabilities,+ compileCaptionGeminiRequest,+ sealCaptionGeminiProgram,+} from "@svml/caption-gemini";+import {+ captionTypes,+ defaultCaptionTrackProgram,+ resolveCaptionProgram,+ sealCaptionStyle,+} from "@svml/caption";+import type { CaptionGeminiRequest, RawCaptionGeminiResponse } from "@svml/caption-gemini";+import { EndpointRegistry, MemoryArtifactStore } from "@svml/driver-node";+import type { ImmediateEndpointHandler } from "@svml/endpoint-kit";+import { digestOf } from "@svml/protocol";+import type { CanonicalValue, Need } from "@svml/protocol";+import {+ createGoogleVertexCaptionProvider,+ googleVertexProviderImplementationDigest,+} from "@svml/provider-google-vertex";+import type { GenerateCaptionContent } from "@svml/provider-google-vertex";+import { parseScript } from "@svml/script";++function request(): CaptionGeminiRequest {+ const narrative = parseScript("provider.svml", "<line><ALICE>Meaning becomes the source.</line>");+ const base = defaultCaptionTrackProgram("important");+ const style = sealCaptionStyle({+ contract: "svml.caption-style@1",+ id: "important",+ planning: {+ cueInstruction: "Prefer one short complete semantic phrase.",+ fields: [{+ id: "important",+ value: { kind: "enum", values: ["important"] },+ instruction: "Select at most two important words.",+ minimumPerCue: 0,+ maximumPerCue: 2,+ }],+ },+ presentation: { mode: "whole", stackingOrder: 100, style: base.style },+ });+ const captionProgram = resolveCaptionProgram(narrative, "captions", style, []);+ return compileCaptionGeminiRequest(narrative, captionProgram, sealCaptionGeminiProgram({+ contract: "svml.caption-gemini-program@1",+ model: "gemini-2.5-flash",+ }));+}++function response(requestValue: CaptionGeminiRequest): RawCaptionGeminiResponse {+ return { runs: requestValue.runs.map((run) => ({+ run_id: run.id,+ cues: [{+ after_atom_id: run.atomIds.at(-1)!,+ fields: [{ declaration_id: "important", atom_id: run.atomIds[0]!, value: "important" }],+ }],+ })) };+}++function need(requestValue: CaptionGeminiRequest): Need {+ const constraints = requestValue as unknown as CanonicalValue;+ return {+ id: "need:caption-gemini",+ capability: captionGeminiCapabilities.plan,+ returns: captionTypes.plan,+ constraints,+ requestedBy: "derivation:caption-gemini",+ result: "record:caption-gemini",+ accepts: "exact",+ conformanceFloor: "exact",+ requestDigest: digestOf({ capability: captionGeminiCapabilities.plan, constraints }),+ };+}++test("Vertex transports the exact model request while the model package validates the result", async () => {+ const requestValue = request();+ let captured: Parameters<GenerateCaptionContent>[0] | undefined;+ const provider = createGoogleVertexCaptionProvider({+ project: "svml-test-project",+ location: "global",+ generateContentImplementationDigest: digestOf("caption-gemini:test-transport"),+ generateContent: async (input) => {+ captured = input;+ return {+ text: JSON.stringify(response(requestValue)),+ totalTokenCount: 123,+ modelVersion: "gemini-2.5-flash-001",+ };+ },+ });+ const registry = new EndpointRegistry();+ await provider.install(registry);+ const requestNeed = need(requestValue);+ const resolution = registry.resolve(requestNeed);+ assert.equal(resolution.status, "resolved");+ assert.equal(resolution.registration.kind, "immediate");+ const handler: ImmediateEndpointHandler = resolution.registration.handler;+ const result = await handler({+ command: { kind: "fulfill-need", id: "command:caption-gemini", need: requestNeed },+ need: requestNeed,+ artifacts: new MemoryArtifactStore(),+ credentials: { googleCredentials: { secret: JSON.stringify({ type: "service_account" }) } },+ });+ assert.equal(captured?.model, "gemini-2.5-flash");+ assert.equal(captured?.prompt, requestValue.prompt);+ assert.equal(captured?.systemInstruction, requestValue.systemInstruction);+ assert.equal(captured?.temperature, 0.2);+ assert.equal(result.value.kind, "inline");+ const plan = result.value.kind === "inline" ? result.value.value as Record<string, unknown> : {};+ assert.equal(plan.contract, "svml.caption-plan@1");+ assert.equal(plan.planningRequestDigest, requestValue.requestDigest);+ assert.equal((result.metadata as Record<string, unknown>).provider, "google-vertex");+});++test("Vertex configuration exposes credential/queue policy without changing the model request", () => {+ const provider = createGoogleVertexCaptionProvider({ project: "svml-test-project", defaultConcurrency: 3 });+ assert.equal(provider.name, "google-vertex.caption");+ const facet = provider.manifest.facets[0];+ assert.equal(facet?.role, "capability-endpoint");+ assert(facet?.role === "capability-endpoint");+ assert.equal(facet.defaultConcurrency, 3);+ assert.deepEqual(facet.credentialSlots, ["googleCredentials"]);+ assert.deepEqual(provider.bindings, [{+ capability: captionGeminiCapabilities.plan,+ returns: captionTypes.plan,+ endpoint: "google-vertex.caption",+ }]);+ assert.equal(facet.implementation.digest, googleVertexProviderImplementationDigest);+});diff --git a/packages/script/README.md b/packages/script/README.mdindex cfada96a..05603d70 100644--- a/packages/script/README.md+++ b/packages/script/README.md@@ -26,12 +26,25 @@ parser is inside a Segment, a valid bare tag such as `<ALICE>` is a Role Cue. Thnot indentation: the compact spelling`<opening><ALICE>我先说。<BOB>我回答。</opening>` has the same semantic value.+A Role Cue is optional. Text before the first Role Cue is a roleless Turn, and Role state is reset+when every Segment closes; a Role can never leak into the following Segment.+The package exports its Manifest, `parseScript`, semantic/source-map projection helpers, asemantic-preserving formatter and the raw `decodeScriptSurface` handler. Source ranges and parserstate remain private to Script; its authored Narrative Record uses the Frontend-neutral type from`@svml/contracts`, so third-party author surfaces can feed the same WhisperX, locator and captioncomponents without importing Script internals.-Every named Segment is additionally exported as `script.segment.<id>` with the shared-`NarrativeExcerpt` type. Seedance and future speech packages consume that narrow value rather than-Script's parser AST; another authoring package may produce the same contract.+The Surface exports one full Narrative plus narrow, immutable views:++- `script.segment.<id>` is a `NarrativeExcerpt` used to associate a generated Take with one Segment;+- `script.segment.<id>.dialogue` is display-independent dialogue, including optional Role cues and+ right-side Dual Text pronunciation, for a speech-video model;+- `script.segment.<id>.speech` is pronunciation-only text for duration estimation;+- `script.caption` is the whole left-side `CaptionProjection` and is not split by Segment for the+ Caption planner;+- `script.selection.<id>` is a reusable explicit Selection.++Seedance consumes the dialogue excerpt, Speech Spine consumes the Segment excerpt, and Caption+consumes the whole Narrative/Caption projection. None imports Script's parser AST. Another authoring+package may produce the same contracts.diff --git a/packages/script/src/index.ts b/packages/script/src/index.tsindex 1cd92aba..47f45e3b 100644--- a/packages/script/src/index.ts+++ b/packages/script/src/index.ts@@ -1,15 +1,23 @@export { ScriptSyntaxError } from "./error.js";export { formatScript } from "./format.js";export {+ captionProjectionType,+ narrativeDialogueExcerptType,narrativeExcerptType,+ narrativeSelectionType,narrativeSchema,+ narrativeSpeechExcerptType,narrativeType,scriptManifest,scriptModuleRef,scriptSurfaceImplementationDigest,} from "./manifest.js";export {+ captionProjectionValue,+ narrativeDialogueExcerptValue,narrativeSegmentExcerptValue,+ narrativeSelectionValue,+ narrativeSpeechExcerptValue,narrativeSourceMap,narrativeValue,serializeCaption,diff --git a/packages/script/src/manifest.ts b/packages/script/src/manifest.tsindex f61e4d26..ee4d8267 100644--- a/packages/script/src/manifest.ts+++ b/packages/script/src/manifest.ts@@ -9,9 +9,13 @@ import type { ModuleManifest, TypeRef } from "@svml/protocol";export const scriptModuleRef = { name: "@svml/script", version: "0.0.0-dev" } as const;export const narrativeType: TypeRef = contractTypes.narrative;export const narrativeExcerptType: TypeRef = contractTypes.narrativeExcerpt;+export const narrativeDialogueExcerptType: TypeRef = contractTypes.narrativeDialogueExcerpt;+export const narrativeSpeechExcerptType: TypeRef = contractTypes.narrativeSpeechExcerpt;+export const narrativeSelectionType: TypeRef = contractTypes.narrativeSelection;+export const captionProjectionType: TypeRef = contractTypes.captionProjection;export { narrativeSchema };-export const scriptSurfaceImplementationDigest = digestOf("@svml/script/surface@1");+export const scriptSurfaceImplementationDigest = digestOf("@svml/script/surface@2");export const scriptManifest: ModuleManifest = {format: "svml.module@0",@@ -25,7 +29,14 @@ export const scriptManifest: ModuleManifest = {name: "script",tag: "script",mode: "raw",- outputs: [narrativeType, narrativeExcerptType],+ outputs: [+ narrativeType,+ narrativeExcerptType,+ narrativeDialogueExcerptType,+ narrativeSpeechExcerptType,+ narrativeSelectionType,+ captionProjectionType,+ ],implementation: {kind: "trusted-frontend-surface",locator: "@svml/script/surface",diff --git a/packages/script/src/narrative.ts b/packages/script/src/narrative.tsindex a7d42635..8871e201 100644--- a/packages/script/src/narrative.ts+++ b/packages/script/src/narrative.ts@@ -42,6 +42,29 @@ function segmentSerializations(segment: ParsedNarrative["segments"][number]): {return { dialogue: dialogue.join("\n"), speech: joinProjection(speechParts) };}+function excerptContent(+ segment: ParsedNarrative["segments"][number],+ projection: "dialogue" | "speech",+): {+ readonly kind: "segment";+ readonly id: string;+ readonly tokenStart: number;+ readonly tokenEndExclusive: number;+ readonly dialogue?: string;+ readonly speech?: string;+} {+ const serializations = segmentSerializations(segment);+ return {+ kind: "segment",+ id: segment.id,+ tokenStart: segment.tokenStart,+ tokenEndExclusive: segment.tokenEndExclusive,+ ...(projection === "dialogue"+ ? { dialogue: serializations.dialogue }+ : { speech: serializations.speech }),+ };+}+export function narrativeSegmentExcerptValue(parsed: ParsedNarrative,segment: ParsedNarrative["segments"][number],@@ -57,6 +80,48 @@ export function narrativeSegmentExcerptValue(return canonicalize({ ...content, excerptDigest: digestOf(canonicalize(content)) });}+export function narrativeDialogueExcerptValue(+ segment: ParsedNarrative["segments"][number],+): CanonicalValue {+ const content = {+ contract: "svml.narrative-dialogue-excerpt@1",+ ...excerptContent(segment, "dialogue"),+ } as const;+ return canonicalize({ ...content, excerptDigest: digestOf(canonicalize(content)) });+}++export function narrativeSpeechExcerptValue(+ segment: ParsedNarrative["segments"][number],+): CanonicalValue {+ const content = {+ contract: "svml.narrative-speech-excerpt@1",+ ...excerptContent(segment, "speech"),+ } as const;+ return canonicalize({ ...content, excerptDigest: digestOf(canonicalize(content)) });+}++export function captionProjectionValue(parsed: ParsedNarrative): CanonicalValue {+ const content = {+ contract: parsed.captionProjection.contract,+ text: parsed.captionProjection.text,+ regions: parsed.captionProjection.regions.map(({ range: _range, ...region }) => region),+ } as const;+ return canonicalize({ ...content, projectionDigest: digestOf(canonicalize(content)) });+}++export function narrativeSelectionValue(selection: ParsedNarrative["selections"][number]): CanonicalValue {+ const content = {+ contract: "svml.narrative-selection@1",+ id: selection.id,+ occurrences: selection.occurrences.map((occurrence) => ({+ occurrence: occurrence.occurrence,+ open: { affinity: occurrence.open.affinity, boundary: semanticBoundary(occurrence.open.boundary) },+ close: { affinity: occurrence.close.affinity, boundary: semanticBoundary(occurrence.close.boundary) },+ })),+ } as const;+ return canonicalize({ ...content, selectionDigest: digestOf(canonicalize(content)) });+}+function semanticBoundary(boundary: {readonly tokenIndex: number;readonly structuralPosition: number;diff --git a/packages/script/src/surface.ts b/packages/script/src/surface.tsindex 8d5d56b6..36dc1ad3 100644--- a/packages/script/src/surface.ts+++ b/packages/script/src/surface.ts@@ -2,11 +2,22 @@ import { canonicalize } from "@svml/core";import { ScriptSyntaxError } from "./error.js";import {+ captionProjectionValue,+ narrativeDialogueExcerptValue,narrativeSegmentExcerptValue,+ narrativeSelectionValue,+ narrativeSpeechExcerptValue,narrativeSourceMap,narrativeValue,} from "./narrative.js";-import { narrativeExcerptType, narrativeType } from "./manifest.js";+import {+ captionProjectionType,+ narrativeDialogueExcerptType,+ narrativeExcerptType,+ narrativeSelectionType,+ narrativeSpeechExcerptType,+ narrativeType,+} from "./manifest.js";import { parseScript } from "./parser.js";import type { ScriptSurfaceInput, ScriptSurfaceOutput } from "./types.js";@@ -79,6 +90,35 @@ export function decodeScriptSurface(input: ScriptSurfaceInput): ScriptSurfaceOutvalue: { kind: "inline" as const, value: narrativeSegmentExcerptValue(parsed, segment) },range: segment.range,})),+ ...parsed.segments.flatMap((segment) => [+ {+ id: `${rawId}.segment.${segment.id}.dialogue`,+ type: narrativeDialogueExcerptType,+ value: { kind: "inline" as const, value: narrativeDialogueExcerptValue(segment) },+ range: segment.range,+ },+ {+ id: `${rawId}.segment.${segment.id}.speech`,+ type: narrativeSpeechExcerptType,+ value: { kind: "inline" as const, value: narrativeSpeechExcerptValue(segment) },+ range: segment.range,+ },+ ]),+ {+ id: `${rawId}.caption`,+ type: captionProjectionType,+ value: { kind: "inline" as const, value: captionProjectionValue(parsed) },+ range: { start: input.openingStart, end: close.end },+ },+ ...parsed.selections.map((selection) => ({+ id: `${rawId}.selection.${selection.id}`,+ type: narrativeSelectionType,+ value: { kind: "inline" as const, value: narrativeSelectionValue(selection) },+ range: {+ start: selection.occurrences[0]!.open.range.start,+ end: selection.occurrences.at(-1)!.close.range.end,+ },+ })),],components: [],fragments: [],diff --git a/packages/script/test/script.test.ts b/packages/script/test/script.test.tsindex ad6ad82b..ca48ad54 100644--- a/packages/script/test/script.test.ts+++ b/packages/script/test/script.test.ts@@ -36,6 +36,18 @@ test("named blocks are Segments and Role Cues do not depend on line breaks", ()assert.equal(serializeDialogue(compact), "ALICE: Hello there.\nBOB: Good morning.");});+test("Role is optional per Turn and never leaks across Segment boundaries", () => {+ const parsed = parseScript(+ "optional-role.svml",+ "<intro>Roleless narration.</intro><answer><ALICE>Named reply.</answer><close>Roleless close.</close>",+ );+ assert.deepEqual(parsed.turns.map((turn) => turn.role), [undefined, "ALICE", undefined]);+ assert.equal(+ serializeDialogue(parsed),+ "Roleless narration.\nALICE: Named reply.\nRoleless close.",+ );+});+test("Script produces exactly 2M + 2N independent semantic anchors", () => {const parsed = parseScript("anchors.svml", "<one>One two.</one><silence/><two>Three.</two>");assert.equal(diff --git a/packages/seedance/README.md b/packages/seedance/README.mdindex b0e658b3..ff9bce3a 100644--- a/packages/seedance/README.md+++ b/packages/seedance/README.md@@ -5,7 +5,7 @@ and `Speech` Surfaces; it does not contain KIE credentials, HTTP code, queues or`Video` and `Speech` preserve the remote result as one atomic `GeneratedVideoSet`, then expose itsfirst ordered member through the shared deterministic primary-video projection. `Speech` consumes a-shared `NarrativeExcerpt`, so a third-party narrative Surface can feed it without importing the+shared `NarrativeDialogueExcerpt`, so a third-party narrative Surface can feed it without importing theofficial Script parser.Duration is explicit in this low-level model Surface. A later Speech scheduling package may wrap itdiff --git a/packages/seedance/src/index.ts b/packages/seedance/src/index.tsindex 99f6d4f6..4f021250 100644--- a/packages/seedance/src/index.ts+++ b/packages/seedance/src/index.ts@@ -56,7 +56,7 @@ export const seedanceTypes = {export const seedanceSurfaceImplementationDigests = {prompt: digestOf("@svml/seedance/prompt-surface@1"),- speech: digestOf("@svml/seedance/speech-surface@1"),+ speech: digestOf("@svml/seedance/speech-surface@2"),video: digestOf("@svml/seedance/video-surface@1"),} as const;diff --git a/packages/seedance/src/surface.ts b/packages/seedance/src/surface.tsindex 00122e56..5807d3cf 100644--- a/packages/seedance/src/surface.ts+++ b/packages/seedance/src/surface.ts@@ -211,26 +211,25 @@ function referencePrompt(values: readonly ReferenceInput[]): string {return roles.length === 0 ? "" : `\n\nReference roles:\n${roles.join("\n")}`;}-function excerpt(reference: SurfaceResolvedReference, subject: string): {- readonly serializations: { readonly dialogue: string; readonly speech: string };+function dialogueExcerpt(reference: SurfaceResolvedReference, subject: string): {+ readonly dialogue: string;} {- if (!sameType(reference.type, contractTypes.narrativeExcerpt)) {- throw new Error(`${subject} must reference a NarrativeExcerpt such as script.segment.opening`);+ if (!sameType(reference.type, contractTypes.narrativeDialogueExcerpt)) {+ throw new Error(`${subject} must reference a NarrativeDialogueExcerpt such as script.segment.opening.dialogue`);}const value = inline(reference, subject) as unknown as {readonly contract: string;readonly excerptDigest: string;- readonly serializations?: { readonly dialogue?: string; readonly speech?: string };+ readonly dialogue?: string;readonly [key: string]: CanonicalValue | undefined;};- if (value.contract !== "svml.narrative-excerpt@1" || value.serializations === undefined- || typeof value.serializations.dialogue !== "string" || typeof value.serializations.speech !== "string") {- throw new Error(`${subject} NarrativeExcerpt is invalid`);+ if (value.contract !== "svml.narrative-dialogue-excerpt@1" || typeof value.dialogue !== "string") {+ throw new Error(`${subject} NarrativeDialogueExcerpt is invalid`);}const { excerptDigest: _digest, ...content } = value;- if (value.excerptDigest !== digestOf(content)) throw new Error(`${subject} NarrativeExcerpt digest differs`);- if (value.serializations.speech.trim().length === 0) throw new Error(`${subject} contains no spoken text`);- return value as { readonly serializations: { readonly dialogue: string; readonly speech: string } };+ if (value.excerptDigest !== digestOf(content)) throw new Error(`${subject} NarrativeDialogueExcerpt digest differs`);+ if (value.dialogue.trim().length === 0) throw new Error(`${subject} contains no spoken text`);+ return value as { readonly dialogue: string };}function generationOutput(@@ -297,17 +296,17 @@ export const decodeSeedanceVideoSurface: StructuredSurfaceHandler = ({ element,};export const decodeSeedanceSpeechSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {- attributes(element, ["id", "model", "script", "prompt", "duration"], [+ attributes(element, ["id", "model", "dialogue", "prompt", "duration"], ["resolution", "aspect-ratio", "web-search",]);const selected = modelSelection(element);const declaredPrompt = prompt(resolved(element, "prompt", resolveReference), `${element.name}.prompt`);- const spoken = excerpt(resolved(element, "script", resolveReference), `${element.name}.script`);+ const spoken = dialogueExcerpt(resolved(element, "dialogue", resolveReference), `${element.name}.dialogue`);const refs = references(element, resolveReference);const request = sealSeedanceRequest({contract: "svml.seedance-request@1",model: selected.model as SeedanceModel,- prompt: `${declaredPrompt.text}${referencePrompt(refs)}\n\nSpoken dialogue — say exactly:\n${spoken.serializations.dialogue}`,+ prompt: `${declaredPrompt.text}${referencePrompt(refs)}\n\nSpoken dialogue — say exactly:\n${spoken.dialogue}`,mode: refs.length === 0? { kind: "text" }: { kind: "reference", items: refs.map(({ role: _role, ...item }) => item) },diff --git a/packages/speech-program/README.md b/packages/speech-program/README.mdindex c8a802d7..bbfb696a 100644--- a/packages/speech-program/README.md+++ b/packages/speech-program/README.md@@ -10,5 +10,9 @@ auditable reusable graph contributions:- WhisperX evidence plus the provider-neutral speech SemanticMap;- timed caption projection.-The author frontend may expose these Fragments through concise surface syntax.+The package also owns `<speech:Spine>`. Each `<speech:Take>` explicitly connects a generated Blob+and its `NarrativeExcerpt`; the Surface expands per-Take media normalization, an immutable append+fold, one timeline-audio render and ordinary SpeechBasis projections. Whitespace and child layout+do not create ports, and Core receives no variadic node or Speech special case.+The Elaborator expands them hygienically before SVML Core sees the graph.diff --git a/packages/speech-program/package.json b/packages/speech-program/package.jsonindex 81586c59..78ec82be 100644--- a/packages/speech-program/package.json+++ b/packages/speech-program/package.json@@ -8,18 +8,19 @@},"dependencies": {"@svml/caption": "workspace:*",+ "@svml/component-kit": "workspace:*","@svml/contracts": "workspace:*","@svml/elaborator": "workspace:*","@svml/media-pipeline": "workspace:*",+ "@svml/protocol": "workspace:*","@svml/speech-align": "workspace:*","@svml/speech-take": "workspace:*",+ "@svml/text": "workspace:*","@svml/whisperx": "workspace:*"},"devDependencies": {- "@svml/component-kit": "workspace:*","@svml/core": "workspace:*","@svml/driver-node": "workspace:*",- "@svml/protocol": "workspace:*","@svml/realization": "workspace:*","@svml/script": "workspace:*","@svml/validation": "workspace:*"diff --git a/packages/speech-program/src/component.ts b/packages/speech-program/src/component.tsnew file mode 100644index 00000000..5ca3dbef--- /dev/null+++ b/packages/speech-program/src/component.ts@@ -0,0 +1,71 @@+import type { ComponentPackage } from "@svml/component-kit";+import type {+ NarrativeExcerpt,+ SynchronizedMedia,+ TimelineAudio,+} from "@svml/contracts";+import type { StoredValue } from "@svml/protocol";+import { canonicalize } from "@svml/protocol";++import { speechProgramProducers } from "./manifest.js";+import {+ appendSpeechSpineTake,+ appendSpeechSpineTakeImplementationDigest,+ assembleSpeechBasis,+ assembleSpeechBasisImplementationDigest,+ compileSpeechSpineAudio,+ compileSpeechSpineAudioImplementationDigest,+ createSpeechSpineSet,+ createSpeechSpineSetImplementationDigest,+} from "./program.js";+import type { SpeechSpineProgram, SpeechSpineSet } from "./types.js";++function inline<T>(value: StoredValue | undefined, subject: string): T {+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);+ return value.value as unknown as T;+}++export const speechProgramComponent = {+ name: "@svml/speech-program",+ producers: [+ {+ producer: speechProgramProducers.createSet,+ implementationDigest: createSpeechSpineSetImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(createSpeechSpineSet(inline<SpeechSpineProgram>(inputs.program?.value, "SpeechSpineProgram"))) } },+ needs: {},+ }),+ },+ {+ producer: speechProgramProducers.appendTake,+ implementationDigest: appendSpeechSpineTakeImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { set: { kind: "inline", value: canonicalize(appendSpeechSpineTake(+ inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),+ inline<SynchronizedMedia>(inputs.media?.value, "SynchronizedMedia"),+ inline<NarrativeExcerpt>(inputs.segment?.value, "NarrativeExcerpt"),+ )) } },+ needs: {},+ }),+ },+ {+ producer: speechProgramProducers.compileAudio,+ implementationDigest: compileSpeechSpineAudioImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { plan: { kind: "inline", value: canonicalize(compileSpeechSpineAudio(inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"))) } },+ needs: {},+ }),+ },+ {+ producer: speechProgramProducers.assembleBasis,+ implementationDigest: assembleSpeechBasisImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { basis: { kind: "inline", value: canonicalize(assembleSpeechBasis(+ inline<SpeechSpineSet>(inputs.set?.value, "SpeechSpineSet"),+ inline<TimelineAudio>(inputs.audio?.value, "TimelineAudio"),+ )) } },+ needs: {},+ }),+ },+ ],+} satisfies ComponentPackage;diff --git a/packages/speech-program/src/fragment.ts b/packages/speech-program/src/fragment.tsnew file mode 100644index 00000000..ab212c87--- /dev/null+++ b/packages/speech-program/src/fragment.ts@@ -0,0 +1,115 @@+import { contractTypes } from "@svml/contracts";+import { sealGraphFragment } from "@svml/elaborator";+import type { FragmentOperation } from "@svml/elaborator";+import { mediaPipelineProducers } from "@svml/media-pipeline";+import { speechTakeProducers } from "@svml/speech-take";++import { speechProgramProducers, speechProgramTypes } from "./manifest.js";+import type { SpeechSpineFragmentOptions } from "./types.js";++const input = (name: string) => ({ kind: "fragment-input" as const, name });+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });++export function createSpeechSpineFragment(options: SpeechSpineFragmentOptions) {+ if (options.takes.length === 0) throw new Error("Speech Spine requires at least one Take");+ const names = new Set(["program"]);+ for (const take of options.takes) {+ if (!take.mediaName || !take.segmentName || names.has(take.mediaName) || names.has(take.segmentName)) {+ throw new Error("Speech Spine Fragment input names are empty or duplicated");+ }+ names.add(take.mediaName);+ names.add(take.segmentName);+ }+ const operations: FragmentOperation[] = [{+ id: "spine:set:empty",+ producer: speechProgramProducers.createSet,+ inputs: { program: input("program") },+ result: { kind: "output", name: "set" },+ }];+ let current = "spine:set:empty";+ options.takes.forEach((take, index) => {+ const id = `spine:set:append:${String(index + 1).padStart(4, "0")}`;+ operations.push({+ id,+ producer: speechProgramProducers.appendTake,+ inputs: { set: operation(current), media: input(take.mediaName), segment: input(take.segmentName) },+ result: { kind: "output", name: "set" },+ });+ current = id;+ });+ operations.push(+ {+ id: "spine:audio:plan",+ producer: speechProgramProducers.compileAudio,+ inputs: { set: operation(current) },+ result: { kind: "output", name: "plan" },+ },+ {+ id: "spine:audio:render",+ producer: mediaPipelineProducers.renderAudio,+ inputs: { plan: operation("spine:audio:plan") },+ result: { kind: "need", name: "audio", accepts: "exact" },+ },+ {+ id: "spine:basis",+ producer: speechProgramProducers.assembleBasis,+ inputs: { set: operation(current), audio: operation("spine:audio:render") },+ result: { kind: "output", name: "basis" },+ },+ {+ id: "spine:space",+ producer: speechTakeProducers.projectProgramSpace,+ inputs: { basis: operation("spine:basis") },+ result: { kind: "output", name: "programSpace" },+ },+ {+ id: "spine:audio-basis",+ producer: speechTakeProducers.projectAudio,+ inputs: { basis: operation("spine:basis") },+ result: { kind: "output", name: "audio" },+ },+ {+ id: "spine:visual-track",+ producer: speechTakeProducers.projectVisual,+ inputs: { basis: operation("spine:basis") },+ result: { kind: "output", name: "visual" },+ },+ {+ id: "spine:audio-track",+ producer: speechTakeProducers.projectAudioTrack,+ inputs: { basis: operation("spine:basis") },+ result: { kind: "output", name: "track" },+ },+ );+ const semanticInputs = ["program", ...options.takes.flatMap((take) => [take.mediaName, take.segmentName])];+ return sealGraphFragment({+ name: options.name?.trim() || "@svml/speech-program/spine@1",+ inputs: [+ { name: "program", type: speechProgramTypes.spineProgram },+ ...options.takes.flatMap((take) => [+ { name: take.mediaName, type: contractTypes.synchronizedMedia },+ { name: take.segmentName, type: contractTypes.narrativeExcerpt },+ ]),+ ],+ operations,+ exports: [+ { name: "basis", type: contractTypes.speechBasis, root: operation("spine:basis"), semanticInputs, fidelity: "exact" },+ {+ name: "space", type: contractTypes.programSpace, root: operation("spine:space"), semanticInputs,+ affinity: [{ resultPointer: "/digest", source: operation("spine:basis"), sourcePointer: "/programSpace/digest" }], fidelity: "exact",+ },+ {+ name: "audio", type: contractTypes.speechAudioBasis, root: operation("spine:audio-basis"), semanticInputs,+ affinity: [{ resultPointer: "/programSpace/digest", source: operation("spine:basis"), sourcePointer: "/programSpace/digest" }], fidelity: "exact",+ },+ {+ name: "visual", type: contractTypes.visualTrack, root: operation("spine:visual-track"), semanticInputs,+ affinity: [{ resultPointer: "/programSpaceDigest", source: operation("spine:basis"), sourcePointer: "/programSpace/digest" }], fidelity: "exact",+ },+ {+ name: "audioTrack", type: contractTypes.audioTrack, root: operation("spine:audio-track"), semanticInputs,+ affinity: [{ resultPointer: "/programSpaceDigest", source: operation("spine:basis"), sourcePointer: "/programSpace/digest" }], fidelity: "exact",+ },+ ],+ });+}diff --git a/packages/speech-program/src/index.ts b/packages/speech-program/src/index.tsindex 75fb3e00..b7ea1553 100644--- a/packages/speech-program/src/index.ts+++ b/packages/speech-program/src/index.ts@@ -3,5 +3,32 @@ export {captionTrackFragment,captionTimingFragment,speechTakeProjectionFragment,- whisperXSpeechAlignmentFragment,} from "./speech.js";+export { whisperXSpeechAlignmentFragment } from "@svml/whisperx";+export { speechProgramComponent } from "./component.js";+export { createSpeechSpineFragment } from "./fragment.js";+export {+ speechProgramManifest,+ speechProgramManifestDigest,+ speechProgramModuleRef,+ speechProgramProducers,+ speechProgramTypes,+ speechSpineProgramSchema,+ speechSpineSetSchema,+ speechSpineSurfaceImplementationDigest,+} from "./manifest.js";+export {+ appendSpeechSpineTake,+ appendSpeechSpineTakeImplementationDigest,+ assembleSpeechBasis,+ assembleSpeechBasisImplementationDigest,+ assertSpeechSpineProgram,+ assertSpeechSpineSet,+ compileSpeechSpineAudio,+ compileSpeechSpineAudioImplementationDigest,+ createSpeechSpineSet,+ createSpeechSpineSetImplementationDigest,+ sealSpeechSpineProgram,+} from "./program.js";+export { decodeSpeechSpineSurface } from "./surface.js";+export type * from "./types.js";diff --git a/packages/speech-program/src/manifest.ts b/packages/speech-program/src/manifest.tsnew file mode 100644index 00000000..491e5148--- /dev/null+++ b/packages/speech-program/src/manifest.ts@@ -0,0 +1,149 @@+import {+ contractTypes,+ narrativeExcerptSchema,+ synchronizedMediaSchema,+ videoContractDependencies,+} from "@svml/contracts";+import {+ mediaPipelineManifest,+ mediaPipelineModuleRef,+ mediaPipelineTypes,+} from "@svml/media-pipeline";+import { digestOf } from "@svml/protocol";+import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";+import { speechTakeManifest, speechTakeModuleRef } from "@svml/speech-take";++import {+ appendSpeechSpineTakeImplementationDigest,+ assembleSpeechBasisImplementationDigest,+ compileSpeechSpineAudioImplementationDigest,+ createSpeechSpineSetImplementationDigest,+} from "./program.js";++export const speechProgramModuleRef = { name: "@svml/speech-program", version: "0.0.0-dev" } as const;+export const speechSpineSurfaceImplementationDigest = digestOf("@svml/speech-program/spine-surface@1");+export const speechProgramTypes = {+ spineProgram: { module: speechProgramModuleRef, name: "SpeechSpineProgram" },+ spineSet: { module: speechProgramModuleRef, name: "SpeechSpineSet" },+} satisfies Record<string, TypeRef>;+export const speechProgramProducers = {+ createSet: { module: speechProgramModuleRef, name: "create-spine-set" },+ appendTake: { module: speechProgramModuleRef, name: "append-spine-take" },+ compileAudio: { module: speechProgramModuleRef, name: "compile-spine-audio" },+ assembleBasis: { module: speechProgramModuleRef, name: "assemble-speech-basis" },+} satisfies Record<string, ProducerRef>;++const string = { kind: "string", minLength: 1 } as const;+const integer = { kind: "number", integer: true, minimum: 1 } as const;+const digest = { kind: "string", minLength: 71, maxLength: 71 } as const;+const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema; readonly optional?: boolean }>>): ValueSchema => ({+ kind: "object",+ fields,+});+const frameRate = object({ numerator: { schema: integer }, denominator: { schema: integer } });++export const speechSpineProgramSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.speech-spine-program@1" } },+ digest: { schema: digest },+ id: { schema: string },+ frameRate: { schema: frameRate },+});++export const speechSpineSetSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.speech-spine-set@1" } },+ digest: { schema: digest },+ program: { schema: speechSpineProgramSchema },+ takes: { schema: { kind: "array", items: object({+ segment: { schema: narrativeExcerptSchema },+ media: { schema: synchronizedMediaSchema },+ }) } },+ lastAddition: { schema: object({+ previousSetDigest: { schema: digest },+ segmentDigest: { schema: digest },+ mediaDigest: { schema: digest },+ }), optional: true },+});++export const speechProgramManifest: ModuleManifest = {+ format: "svml.module@0",+ name: speechProgramModuleRef.name,+ version: speechProgramModuleRef.version,+ dependencies: [+ videoContractDependencies.narrative,+ videoContractDependencies.media,+ videoContractDependencies.programSpace,+ videoContractDependencies.speech,+ videoContractDependencies.composition,+ { module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },+ { module: speechTakeModuleRef, digest: digestOf(speechTakeManifest) },+ ],+ types: [+ { name: speechProgramTypes.spineProgram.name, schema: speechSpineProgramSchema },+ { name: speechProgramTypes.spineSet.name, schema: speechSpineSetSchema },+ ],+ capabilities: [],+ surfaces: [{+ name: "spine",+ tag: "Spine",+ mode: "structured",+ outputs: [speechProgramTypes.spineProgram, mediaPipelineTypes.selectionRequest,+ contractTypes.speechBasis, contractTypes.programSpace, contractTypes.speechAudioBasis,+ contractTypes.visualTrack, contractTypes.audioTrack],+ implementation: {+ kind: "trusted-frontend-surface",+ locator: "@svml/speech-program/spine-surface",+ digest: speechSpineSurfaceImplementationDigest,+ },+ }],+ producers: [+ {+ name: speechProgramProducers.createSet.name,+ inputs: [{ name: "program", type: speechProgramTypes.spineProgram }],+ outputs: [{ name: "set", type: speechProgramTypes.spineSet }],+ needs: [],+ implementation: { kind: "registered", locator: "@svml/speech-program/create-spine-set", digest: createSpeechSpineSetImplementationDigest },+ },+ {+ name: speechProgramProducers.appendTake.name,+ inputs: [+ { name: "set", type: speechProgramTypes.spineSet },+ { name: "media", type: contractTypes.synchronizedMedia },+ { name: "segment", type: contractTypes.narrativeExcerpt },+ ],+ outputs: [{+ name: "set",+ type: speechProgramTypes.spineSet,+ affinity: [+ { resultPointer: "/lastAddition/previousSetDigest", input: "set", inputPointer: "/digest" },+ { resultPointer: "/lastAddition/mediaDigest", input: "media", inputPointer: "/synchronizedMediaDigest" },+ { resultPointer: "/lastAddition/segmentDigest", input: "segment", inputPointer: "/excerptDigest" },+ ],+ }],+ needs: [],+ implementation: { kind: "registered", locator: "@svml/speech-program/append-spine-take", digest: appendSpeechSpineTakeImplementationDigest },+ },+ {+ name: speechProgramProducers.compileAudio.name,+ inputs: [{ name: "set", type: speechProgramTypes.spineSet }],+ outputs: [{ name: "plan", type: mediaPipelineTypes.audioProgramPlan }],+ needs: [],+ implementation: { kind: "registered", locator: "@svml/speech-program/compile-spine-audio", digest: compileSpeechSpineAudioImplementationDigest },+ },+ {+ name: speechProgramProducers.assembleBasis.name,+ inputs: [+ { name: "set", type: speechProgramTypes.spineSet },+ { name: "audio", type: contractTypes.timelineAudio },+ ],+ outputs: [{+ name: "basis",+ type: contractTypes.speechBasis,+ affinity: [{ resultPointer: "/audio/digest", input: "audio", inputPointer: "/artifact/digest" }],+ }],+ needs: [],+ implementation: { kind: "registered", locator: "@svml/speech-program/assemble-speech-basis", digest: assembleSpeechBasisImplementationDigest },+ },+ ],+};++export const speechProgramManifestDigest = digestOf(speechProgramManifest);diff --git a/packages/speech-program/src/program.ts b/packages/speech-program/src/program.tsnew file mode 100644index 00000000..512c333a--- /dev/null+++ b/packages/speech-program/src/program.ts@@ -0,0 +1,245 @@+import {+ assertProgramSpaceIdentity,+ sealProgramSpace,+ sealSpeechBasis,+ verifySynchronizedMedia,+ verifyTimelineAudio,+} from "@svml/contracts";+import type {+ NarrativeExcerpt,+ ProgramSpace,+ SpeechBasis,+ SynchronizedMedia,+ TimelineAudio,+} from "@svml/contracts";+import {+ sealAudioProgramPlan,+ verifyAudioProgramPlan,+} from "@svml/media-pipeline";+import type { AudioProgramPlan } from "@svml/media-pipeline";+import { canonicalize, digestOf, isDigest } from "@svml/protocol";+import type { Digest } from "@svml/protocol";++import type {+ SpeechSpineProgram,+ SpeechSpineSet,+ SpeechSpineTake,+} from "./types.js";++export const createSpeechSpineSetImplementationDigest = digestOf("@svml/speech-program/create-spine-set@1");+export const appendSpeechSpineTakeImplementationDigest = digestOf("@svml/speech-program/append-spine-take@1");+export const compileSpeechSpineAudioImplementationDigest = digestOf("@svml/speech-program/compile-spine-audio@1");+export const assembleSpeechBasisImplementationDigest = digestOf("@svml/speech-program/assemble-speech-basis@1");++function assert(condition: unknown, message: string): asserts condition {+ if (!condition) throw new Error(message);+}++function programContent(value: Omit<SpeechSpineProgram, "digest">): Omit<SpeechSpineProgram, "digest"> {+ return {+ contract: "svml.speech-spine-program@1",+ id: value.id,+ frameRate: { ...value.frameRate },+ };+}++export function sealSpeechSpineProgram(value: Omit<SpeechSpineProgram, "digest">): SpeechSpineProgram {+ const content = programContent(value);+ return { ...content, digest: digestOf(content) };+}++export function assertSpeechSpineProgram(value: SpeechSpineProgram): void {+ assert(value.contract === "svml.speech-spine-program@1", "Unsupported SpeechSpineProgram contract");+ assert(value.id.trim().length > 0, "SpeechSpineProgram id must not be empty");+ assert(Number.isSafeInteger(value.frameRate.numerator) && value.frameRate.numerator > 0+ && Number.isSafeInteger(value.frameRate.denominator) && value.frameRate.denominator > 0,+ "SpeechSpineProgram frame rate is invalid");+ const { digest: _digest, ...content } = value;+ assert(isDigest(value.digest) && value.digest === digestOf(canonicalize(content)),+ "SpeechSpineProgram digest differs from its contents");+}++function verifyExcerpt(value: NarrativeExcerpt): void {+ assert(value.contract === "svml.narrative-excerpt@1" && value.kind === "segment",+ "Speech Spine Take must reference a Segment NarrativeExcerpt");+ assert(value.id.length > 0 && Number.isSafeInteger(value.tokenStart)+ && Number.isSafeInteger(value.tokenEndExclusive) && value.tokenEndExclusive > value.tokenStart,+ "Speech Spine Segment excerpt is invalid");+ const { excerptDigest: _digest, ...content } = value;+ assert(isDigest(value.excerptDigest) && value.excerptDigest === digestOf(canonicalize(content)),+ "Speech Spine Segment excerpt digest differs from its contents");+}++function setContent(value: Omit<SpeechSpineSet, "digest">): Omit<SpeechSpineSet, "digest"> {+ return canonicalize({+ contract: "svml.speech-spine-set@1",+ program: value.program,+ takes: value.takes,+ ...(value.lastAddition === undefined ? {} : { lastAddition: value.lastAddition }),+ }) as unknown as Omit<SpeechSpineSet, "digest">;+}++function sealSpeechSpineSet(value: Omit<SpeechSpineSet, "digest">): SpeechSpineSet {+ const content = setContent(value);+ return { ...content, digest: digestOf(content) };+}++function assertTake(take: SpeechSpineTake, program: SpeechSpineProgram): void {+ verifyExcerpt(take.segment);+ verifySynchronizedMedia(take.media);+ assert(take.media.visual !== undefined, `Speech Segment ${take.segment.id} has no moving visual`);+ assert(take.media.audio !== undefined, `Speech Segment ${take.segment.id} has no speech audio`);+ assert(take.media.timeline.frameRate.numerator === program.frameRate.numerator+ && take.media.timeline.frameRate.denominator === program.frameRate.denominator,+ `Speech Segment ${take.segment.id} uses another frame rate`);+}++export function assertSpeechSpineSet(value: SpeechSpineSet): void {+ assert(value.contract === "svml.speech-spine-set@1", "Unsupported SpeechSpineSet contract");+ assertSpeechSpineProgram(value.program);+ const { digest: _digest, ...content } = value;+ assert(isDigest(value.digest) && value.digest === digestOf(setContent(content)),+ "SpeechSpineSet digest differs from its contents");+ const segments = new Set<string>();+ for (const take of value.takes) {+ assertTake(take, value.program);+ assert(!segments.has(take.segment.id), `Speech Spine repeats Segment ${take.segment.id}`);+ segments.add(take.segment.id);+ }+ assert(value.takes.length === 0 ? value.lastAddition === undefined : value.lastAddition !== undefined,+ "SpeechSpineSet last addition is inconsistent");+ if (value.lastAddition !== undefined) {+ const last = value.takes.at(-1)!;+ assert(isDigest(value.lastAddition.previousSetDigest)+ && value.lastAddition.segmentDigest === last.segment.excerptDigest+ && value.lastAddition.mediaDigest === last.media.synchronizedMediaDigest,+ "SpeechSpineSet last addition is invalid");+ }+}++export function createSpeechSpineSet(program: SpeechSpineProgram): SpeechSpineSet {+ assertSpeechSpineProgram(program);+ return sealSpeechSpineSet({+ contract: "svml.speech-spine-set@1",+ program,+ takes: [],+ });+}++export function appendSpeechSpineTake(+ set: SpeechSpineSet,+ media: SynchronizedMedia,+ segment: NarrativeExcerpt,+): SpeechSpineSet {+ assertSpeechSpineSet(set);+ const take = { media, segment } satisfies SpeechSpineTake;+ assertTake(take, set.program);+ assert(!set.takes.some((item) => item.segment.id === segment.id), `Speech Spine repeats Segment ${segment.id}`);+ return sealSpeechSpineSet({+ contract: "svml.speech-spine-set@1",+ program: set.program,+ takes: [...set.takes, take],+ lastAddition: {+ previousSetDigest: set.digest,+ segmentDigest: segment.excerptDigest,+ mediaDigest: media.synchronizedMediaDigest,+ },+ });+}++function frameSample(frame: number, frameRate: SpeechSpineProgram["frameRate"]): number {+ const numerator = BigInt(frame) * 48_000n * BigInt(frameRate.denominator);+ const denominator = BigInt(frameRate.numerator);+ const rounded = (numerator * 2n + denominator) / (denominator * 2n);+ assert(rounded <= BigInt(Number.MAX_SAFE_INTEGER), "Speech Spine sample boundary exceeds safe arithmetic");+ return Number(rounded);+}++function programSpace(set: SpeechSpineSet): ProgramSpace {+ assertSpeechSpineSet(set);+ assert(set.takes.length > 0, "Speech Spine must contain at least one Take");+ const frameCount = set.takes.reduce((sum, take) => sum + take.media.timeline.frameCount, 0);+ const space = sealProgramSpace({+ contract: "svml.program-space@0",+ durationSec: frameCount * set.program.frameRate.denominator / set.program.frameRate.numerator,+ frameRate: { ...set.program.frameRate },+ });+ assertProgramSpaceIdentity(space);+ return space;+}++export function compileSpeechSpineAudio(set: SpeechSpineSet): AudioProgramPlan {+ const space = programSpace(set);+ let frame = 0;+ const clips = set.takes.map((take, index) => {+ const audio = take.media.audio!;+ const startFrame = frame;+ frame += take.media.timeline.frameCount;+ const targetStartSample = frameSample(startFrame, set.program.frameRate);+ const targetEndSampleExclusive = frameSample(frame, set.program.frameRate);+ assert(targetEndSampleExclusive - targetStartSample === audio.sampleFrames,+ `Speech Segment ${take.segment.id} audio does not exactly cover its normalized frame span`);+ return {+ id: `${String(index + 1).padStart(4, "0")}:${take.segment.id}`,+ artifact: audio.artifact,+ targetStartSample,+ targetEndSampleExclusive,+ sourceStartSample: 0,+ playbackRate: 1,+ gain: 1,+ fadeInSamples: 0,+ fadeOutSamples: 0,+ bus: "speech" as const,+ };+ });+ const plan = sealAudioProgramPlan({+ contract: "svml.audio-program-plan@1",+ programSpaceDigest: space.digest,+ frameRate: { ...space.frameRate },+ frameCount: frame,+ sampleRate: 48_000,+ sampleFrames: frameSample(frame, set.program.frameRate),+ clips,+ mix: { normalize: false, limiter: "none" },+ });+ verifyAudioProgramPlan(plan);+ return plan;+}++export function assembleSpeechBasis(set: SpeechSpineSet, audio: TimelineAudio): SpeechBasis {+ const space = programSpace(set);+ const plan = compileSpeechSpineAudio(set);+ verifyTimelineAudio(audio);+ assert(audio.planDigest === plan.planDigest && audio.programSpaceDigest === space.digest,+ "TimelineAudio does not realize this Speech Spine plan");+ let frame = 0;+ const seconds = (value: number): number => value * space.frameRate.denominator / space.frameRate.numerator;+ const segments = set.takes.map((take) => {+ const startFrame = frame;+ frame += take.media.timeline.frameCount;+ return { segmentId: take.segment.id, startSec: seconds(startFrame), endSec: seconds(frame) };+ });+ const visualClips = set.takes.map((take, index) => ({+ segmentId: take.segment.id,+ artifact: {+ digest: take.media.visual!.artifact.digest,+ size: take.media.visual!.artifact.size,+ mediaType: take.media.visual!.artifact.mediaType,+ durationSec: seconds(take.media.timeline.frameCount),+ },+ startSec: segments[index]!.startSec,+ endSec: segments[index]!.endSec,+ }));+ return sealSpeechBasis({+ contract: "svml.speech-basis@1",+ programSpace: space,+ audio: {+ digest: audio.artifact.digest,+ size: audio.artifact.size,+ mediaType: audio.artifact.mediaType,+ durationSec: space.durationSec,+ },+ visualTrack: { clips: visualClips },+ segments,+ });+}diff --git a/packages/speech-program/src/speech.ts b/packages/speech-program/src/speech.tsindex 186c1f9c..f55e9b0b 100644--- a/packages/speech-program/src/speech.ts+++ b/packages/speech-program/src/speech.ts@@ -1,10 +1,7 @@import { captionProducers, captionTypes } from "@svml/caption";import { contractTypes } from "@svml/contracts";import { sealGraphFragment } from "@svml/elaborator";-import { speechAlignProducers } from "@svml/speech-align";import { speechTakeProducers } from "@svml/speech-take";-import { whisperXProducers, whisperXTypes } from "@svml/whisperx";-import { mediaPipelineProducers } from "@svml/media-pipeline";const input = (name: string) => ({ kind: "fragment-input" as const, name });const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });@@ -82,79 +79,6 @@ export const speechTakeProjectionFragment = sealGraphFragment({],});-/** One WhisperX pass can expose both normalized Evidence and the final SemanticMap. */-export const whisperXSpeechAlignmentFragment = sealGraphFragment({- name: "@svml/speech-program/whisperx-speech-alignment@1",- inputs: [- { name: "narrative", type: contractTypes.narrative },- { name: "audio", type: contractTypes.speechAudioBasis },- ],- operations: [- {- id: "prepare-evidence-audio",- producer: mediaPipelineProducers.projectSpeechEvidenceAudio,- inputs: { audio: input("audio") },- result: { kind: "need", name: "evidenceAudio", accepts: "exact" },- },- {- id: "request-whisperx",- producer: whisperXProducers.request,- inputs: { audio: operation("prepare-evidence-audio") },- result: { kind: "need", name: "alignment", accepts: "exact" },- },- {- id: "normalize-evidence",- producer: whisperXProducers.normalize,- inputs: { whisperx: operation("request-whisperx") },- result: { kind: "output", name: "evidence" },- },- {- id: "locate-speech",- producer: speechAlignProducers.locate,- inputs: {- narrative: input("narrative"),- audio: input("audio"),- evidence: operation("normalize-evidence"),- },- result: { kind: "output", name: "map" },- },- ],- exports: [- {- name: "rawEvidence",- type: whisperXTypes.alignmentEvidence,- root: operation("request-whisperx"),- semanticInputs: ["audio"],- affinity: [- { resultPointer: "/audioArtifactDigest", source: operation("prepare-evidence-audio"), sourcePointer: "/artifact/digest" },- { resultPointer: "/programSpaceDigest", source: input("audio"), sourcePointer: "/programSpace/digest" },- ],- fidelity: "exact",- },- {- name: "evidence",- type: contractTypes.alignedTranscriptEvidence,- root: operation("normalize-evidence"),- semanticInputs: ["audio"],- affinity: [- { resultPointer: "/audioArtifactDigest", source: operation("prepare-evidence-audio"), sourcePointer: "/artifact/digest" },- { resultPointer: "/programSpaceDigest", source: input("audio"), sourcePointer: "/programSpace/digest" },- ],- fidelity: "exact",- },- {- name: "map",- type: contractTypes.completeSemanticMap,- root: operation("locate-speech"),- semanticInputs: ["narrative", "audio"],- affinity: [- { resultPointer: "/programSpace/digest", source: input("audio"), sourcePointer: "/programSpace/digest" },- ],- fidelity: "exact",- },- ],-});-export const captionTimingFragment = sealGraphFragment({name: "@svml/speech-program/caption-timing@1",inputs: [diff --git a/packages/speech-program/src/surface.ts b/packages/speech-program/src/surface.tsnew file mode 100644index 00000000..b4c9ad70--- /dev/null+++ b/packages/speech-program/src/surface.ts@@ -0,0 +1,137 @@+import { contractTypes } from "@svml/contracts";+import {+ sealMediaSelectionRequest,+ synchronizedMediaFragment,+} from "@svml/media-pipeline";+import type {+ StructuredElement,+ StructuredSurfaceHandler,+ SurfaceResolvedReference,+ TextAttributeValue,+} from "@svml/text";++import { createSpeechSpineFragment } from "./fragment.js";+import { speechProgramTypes } from "./manifest.js";+import { sealSpeechSpineProgram } from "./program.js";++function localName(value: string): string {+ return value.includes(":") ? value.slice(value.lastIndexOf(":") + 1) : value;+}++function exactAttributes(element: StructuredElement, names: readonly string[]): void {+ if (Object.keys(element.attributes).sort().join("\u0000") !== [...names].sort().join("\u0000")) {+ throw new Error(`${element.name} requires exactly ${names.join(", ")}`);+ }+}++function stringAttribute(element: StructuredElement, name: string): string {+ const value = element.attributes[name];+ if (typeof value !== "string" || value.trim().length === 0) throw new Error(`${element.name}.${name} must be a non-empty string`);+ return value.trim();+}++function referencePath(element: StructuredElement, name: string): string {+ const value: TextAttributeValue | undefined = element.attributes[name];+ if (typeof value !== "object" || value.kind !== "reference" || !value.path) {+ throw new Error(`${element.name}.${name} must be a whole-value reference`);+ }+ return value.path;+}++function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;+}++function resolve(+ element: StructuredElement,+ name: string,+ expected: SurfaceResolvedReference["type"],+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,+): SurfaceResolvedReference {+ const path = referencePath(element, name);+ const value = resolveReference(path);+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${path}`);+ if (!sameType(value.type, expected)) throw new Error(`${element.name}.${name} has the wrong type`);+ return value;+}++function takes(element: StructuredElement): StructuredElement[] {+ const result: StructuredElement[] = [];+ for (const child of element.children) {+ if (child.kind === "text") {+ if (child.value.trim()) throw new Error(`${element.name} accepts only Take children`);+ continue;+ }+ if (localName(child.name) !== "Take") throw new Error(`${element.name} accepts only Take children`);+ exactAttributes(child, ["source", "segment"]);+ result.push(child);+ }+ if (result.length === 0) throw new Error(`${element.name} requires at least one Take`);+ return result;+}++export const decodeSpeechSpineSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ exactAttributes(element, ["id"]);+ const id = stringAttribute(element, "id");+ const declaredTakes = takes(element).map((take, index) => ({+ mediaName: `take-${String(index + 1).padStart(4, "0")}-media`,+ segmentName: `take-${String(index + 1).padStart(4, "0")}-segment`,+ source: resolve(take, "source", contractTypes.blobArtifact, resolveReference),+ segment: resolve(take, "segment", contractTypes.narrativeExcerpt, resolveReference),+ range: take.range,+ }));+ const programId = `${id}.program`;+ const requestId = `${id}.selection`;+ const program = sealSpeechSpineProgram({+ contract: "svml.speech-spine-program@1",+ id,+ frameRate: { numerator: 30, denominator: 1 },+ });+ const request = sealMediaSelectionRequest({+ contract: "svml.media-selection-request@1",+ video: { mode: "primary-moving" },+ audio: { mode: "default" },+ spanAuthority: "video",+ frameRate: { numerator: 30, denominator: 1 },+ });+ const assembly = createSpeechSpineFragment({+ name: `@svml/speech-program/surface/${id}@1`,+ takes: declaredTakes.map(({ mediaName, segmentName }) => ({ mediaName, segmentName })),+ });+ const normalizationComponents = declaredTakes.map((take, index) => ({+ id: `${id}.normalize.${String(index + 1).padStart(4, "0")}`,+ fragment: synchronizedMediaFragment.id,+ inputs: { source: take.source.ref, request: { kind: "record" as const, id: requestId } },+ outputs: { media: `${id}.normalized.${String(index + 1).padStart(4, "0")}` },+ range: take.range,+ }));+ return {+ records: [+ { id: programId, type: speechProgramTypes.spineProgram, value: { kind: "inline", value: program }, range: element.range },+ { id: requestId, type: { module: { name: "@svml/media-pipeline", version: "0.0.0-dev" }, name: "MediaSelectionRequest" }, value: { kind: "inline", value: request }, range: element.range },+ ],+ components: [+ ...normalizationComponents,+ {+ id,+ fragment: assembly.id,+ inputs: {+ program: { kind: "record", id: programId },+ ...Object.fromEntries(declaredTakes.flatMap((take, index) => [+ [take.mediaName, { kind: "component-output" as const, component: normalizationComponents[index]!.id, output: "media" }],+ [take.segmentName, take.segment.ref],+ ])),+ },+ outputs: {+ basis: `${id}.basis`,+ space: `${id}.space`,+ audio: `${id}.audio`,+ visual: `${id}.visual`,+ audioTrack: `${id}.audioTrack`,+ },+ range: element.range,+ },+ ],+ fragments: [synchronizedMediaFragment, assembly],+ };+};diff --git a/packages/speech-program/src/types.ts b/packages/speech-program/src/types.tsnew file mode 100644index 00000000..ea340b95--- /dev/null+++ b/packages/speech-program/src/types.ts@@ -0,0 +1,47 @@+import type {+ NarrativeExcerpt,+ SynchronizedMedia,+} from "@svml/contracts";+import type { Digest } from "@svml/protocol";++/** Author-owned policy for one ordered Speech Spine. */+export type SpeechSpineProgram = {+ readonly contract: "svml.speech-spine-program@1";+ readonly digest: Digest;+ readonly id: string;+ readonly frameRate: {+ readonly numerator: number;+ readonly denominator: number;+ };+};++export type SpeechSpineTake = {+ readonly segment: NarrativeExcerpt;+ readonly media: SynchronizedMedia;+};++/**+ * Package-private immutable fold value. Its order is author meaning; Core sees+ * only fixed-port append Operations for the concrete Spine declaration.+ */+export type SpeechSpineSet = {+ readonly contract: "svml.speech-spine-set@1";+ readonly digest: Digest;+ readonly program: SpeechSpineProgram;+ readonly takes: readonly SpeechSpineTake[];+ readonly lastAddition?: {+ readonly previousSetDigest: Digest;+ readonly segmentDigest: Digest;+ readonly mediaDigest: Digest;+ };+};++export type SpeechSpineInput = {+ readonly mediaName: string;+ readonly segmentName: string;+};++export type SpeechSpineFragmentOptions = {+ readonly name?: string;+ readonly takes: readonly SpeechSpineInput[];+};diff --git a/packages/speech-program/test/surface.test.ts b/packages/speech-program/test/surface.test.tsnew file mode 100644index 00000000..9b0bb2c9--- /dev/null+++ b/packages/speech-program/test/surface.test.ts@@ -0,0 +1,127 @@+import assert from "node:assert/strict";+import test from "node:test";++import { registerTypeValidatorFacets } from "@svml/component-kit";+import {+ contractTypes,+ videoContractDependencies,+ videoContractManifests,+} from "@svml/contracts";+import { createResolvedClosure, digestOf, sealBuildRequest, start } from "@svml/core";+import {+ AuthorFrontendRegistry,+ compileSourceClosure,+ resolveCompiledSourceExport,+} from "@svml/elaborator";+import type { AuthorSourceUnit } from "@svml/elaborator";+import {+ mediaPipelineComponent,+ mediaPipelineManifest,+ mediaPipelineProducers,+} from "@svml/media-pipeline";+import type { ModuleManifest } from "@svml/protocol";+import {+ decodeScriptSurface,+ scriptManifest,+ scriptModuleRef,+ scriptSurfaceImplementationDigest,+} from "@svml/script";+import { speechTakeManifest, speechTakeProducers } from "@svml/speech-take";+import {+ decodeSpeechSpineSurface,+ speechProgramManifest,+ speechProgramModuleRef,+ speechProgramProducers,+ speechSpineSurfaceImplementationDigest,+} from "@svml/speech-program";+import {+ TextSurfaceRegistry,+ createTextAuthorFrontend,+ textAuthorFrontendId,+} from "@svml/text";+import { createRecordAdmitter, TypeValidatorRegistry } from "@svml/validation";++const fixtureModule = { name: "example.speech-media", version: "1" } as const;+const fixtureSurfaceDigest = digestOf("example.speech-media/surface@1");+const fixtureManifest: ModuleManifest = {+ format: "svml.module@0",+ name: fixtureModule.name,+ version: fixtureModule.version,+ dependencies: [videoContractDependencies.media],+ types: [], capabilities: [], producers: [],+ surfaces: [{+ name: "media", tag: "Media", mode: "structured", outputs: [contractTypes.blobArtifact],+ implementation: { kind: "trusted-frontend-surface", locator: "example.speech-media/surface", digest: fixtureSurfaceDigest },+ }],+};++function source(text: string): AuthorSourceUnit {+ return { id: "/project/main.svml", name: "main.svml", text };+}++test("Speech Spine lowers ordered Takes into media normalization, one audio plan and peer projections", async () => {+ const closure = createResolvedClosure([+ ...videoContractManifests,+ mediaPipelineManifest,+ speechTakeManifest,+ speechProgramManifest,+ scriptManifest,+ fixtureManifest,+ ]);+ const surfaces = new TextSurfaceRegistry();+ surfaces.registerRaw(scriptModuleRef, "script", scriptSurfaceImplementationDigest, decodeScriptSurface);+ surfaces.registerStructured(speechProgramModuleRef, "spine", speechSpineSurfaceImplementationDigest, decodeSpeechSpineSurface);+ surfaces.registerStructured(fixtureModule, "media", fixtureSurfaceDigest, ({ element }) => ({+ records: ["take-one", "take-two"].map((id) => ({+ id, type: contractTypes.blobArtifact,+ value: { kind: "blob" as const, digest: digestOf(id), size: 128, mediaType: "video/mp4" },+ range: element.range,+ })),+ components: [], fragments: [],+ }));+ const frontends = new AuthorFrontendRegistry();+ frontends.register(createTextAuthorFrontend({+ registry: surfaces,+ resolveModule(request) {+ if (request.from.startsWith("@svml/script")) return scriptModuleRef;+ if (request.from.startsWith("@svml/speech")) return speechProgramModuleRef;+ return fixtureModule;+ },+ }));+ const validators = new TypeValidatorRegistry();+ registerTypeValidatorFacets(validators, mediaPipelineComponent.validators ?? []);+ const compiled = await compileSourceClosure({+ entry: source(`<svml>+ <import from="@svml/script@1"/>+ <import as="fixture" from="example.speech-media@1"/>+ <import as="speech" from="@svml/speech@1"/>+ <script id="story">+ <opening><ALICE> Hello from Alice.</opening>+ <answer><BOB> Hello from Bob.</answer>+ </script>+ <fixture:Media/>+ <speech:Spine id="speech">+ <speech:Take source={take-one} segment={story.segment.opening}/>+ <speech:Take source={take-two} segment={story.segment.answer}/>+ </speech:Spine>+ </svml>`),+ frontend: textAuthorFrontendId,+ closure,+ frontends,+ admitRecord: createRecordAdmitter(validators),+ resolveSource() { throw new Error("fixture has no source imports"); },+ });+ const target = resolveCompiledSourceExport(compiled, "speech.visual", contractTypes.visualTrack);+ const build = start(compiled.program, compiled.elaboration.graph, sealBuildRequest({+ graph: compiled.elaboration.graph.id,+ targets: [{ output: target.ref.kind === "logical-output" ? target.ref.id : "", accepts: "exact" }],+ bindings: [],+ }));+ const names = build.plan.steps.map((step) => step.producer.name);+ assert.equal(names.filter((name) => name === mediaPipelineProducers.inspect.name).length, 2);+ assert.equal(names.filter((name) => name === mediaPipelineProducers.normalize.name).length, 2);+ assert.equal(names.filter((name) => name === speechProgramProducers.appendTake.name).length, 2);+ assert.equal(names.filter((name) => name === speechProgramProducers.compileAudio.name).length, 1);+ assert.equal(names.filter((name) => name === mediaPipelineProducers.renderAudio.name).length, 1);+ assert.equal(names.filter((name) => name === speechTakeProducers.projectVisual.name).length, 1);+});diff --git a/packages/text-track/README.md b/packages/text-track/README.mdindex 50dabb65..489587a0 100644--- a/packages/text-track/README.md+++ b/packages/text-track/README.md@@ -11,5 +11,7 @@ An item spanning the complete ProgramSpace is a persistent overlay; a shorter itare not different Track kinds. The package exposes semantic typography and box parameters ratherthan a rendering callback, arbitrary CSS or cross-Track access.-No author-facing Surface is frozen yet. A future Text Surface and `.svs` recipes can produce the-same TextTrackProgram without changing Core, Film, Composition or HyperFrames.+The provider-free `<text:Track>` Surface now validates `.svs` Recipes, compiles `during="full"`+against an explicitly connected ProgramSpace and produces the same TextTrackProgram without+changing Core, Film, Composition or HyperFrames. Timed semantic windows and exact FontArtifact+lowering remain pre-freeze work.diff --git a/packages/text-track/package.json b/packages/text-track/package.jsonindex 085ef7f4..eb11ac4d 100644--- a/packages/text-track/package.json+++ b/packages/text-track/package.json@@ -7,9 +7,12 @@".": "./src/index.ts"},"dependencies": {+ "@svml/component-kit": "workspace:*","@svml/contracts": "workspace:*","@svml/elaborator": "workspace:*",- "@svml/protocol": "workspace:*"+ "@svml/protocol": "workspace:*",+ "@svml/svs": "workspace:*",+ "@svml/text": "workspace:*"},"devDependencies": {"@svml/hyperframes": "workspace:*"diff --git a/packages/text-track/src/component.ts b/packages/text-track/src/component.tsnew file mode 100644index 00000000..9601e753--- /dev/null+++ b/packages/text-track/src/component.ts@@ -0,0 +1,43 @@+import type { ComponentPackage } from "@svml/component-kit";+import type { ProgramSpace } from "@svml/contracts";+import type { StoredValue } from "@svml/protocol";+import { canonicalize } from "@svml/protocol";++import { textTrackProducers } from "./manifest.js";+import {+ compileTextTrackImplementationDigest,+ compileTextTrackProgram,+ renderTextTrack,+ renderTextTrackImplementationDigest,+} from "./program.js";+import type { TextTrackProgram, TextTrackSpec } from "./types.js";++function inline<T>(value: StoredValue | undefined, subject: string): T {+ if (value?.kind !== "inline") throw new Error(`${subject} must be inline`);+ return value.value as unknown as T;+}++export const textTrackComponent = {+ name: "@svml/text-track",+ producers: [{+ producer: textTrackProducers.compile,+ implementationDigest: compileTextTrackImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { program: { kind: "inline", value: canonicalize(compileTextTrackProgram(+ inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ inline<TextTrackSpec>(inputs.spec?.value, "TextTrackSpec"),+ )) } },+ needs: {},+ }),+ }, {+ producer: textTrackProducers.render,+ implementationDigest: renderTextTrackImplementationDigest,+ handler: ({ inputs }) => ({+ outputs: { track: { kind: "inline", value: canonicalize(renderTextTrack(+ inline<ProgramSpace>(inputs.space?.value, "ProgramSpace"),+ inline<TextTrackProgram>(inputs.program?.value, "TextTrackProgram"),+ )) } },+ needs: {},+ }),+ }],+} satisfies ComponentPackage;diff --git a/packages/text-track/src/index.ts b/packages/text-track/src/index.tsindex 9bec2c0a..56184210 100644--- a/packages/text-track/src/index.ts+++ b/packages/text-track/src/index.ts@@ -1,17 +1,25 @@export { textTrackFragment } from "./fragment.js";+export { textTrackComponent } from "./component.js";export {textTrackManifest,textTrackManifestDigest,textTrackModuleRef,textTrackProducers,textTrackProgramSchema,+ textTrackSpecSchema,+ textTrackSurfaceImplementationDigest,textTrackTypes,} from "./manifest.js";export {assertTextTrackProgramIdentity,+ assertTextTrackSpec,+ compileTextTrackImplementationDigest,+ compileTextTrackProgram,computeTextTrackProgramDigest,renderTextTrack,renderTextTrackImplementationDigest,sealTextTrackProgram,+ sealTextTrackSpec,} from "./program.js";+export { decodeTextTrackSurface } from "./surface.js";export type * from "./types.js";diff --git a/packages/text-track/src/manifest.ts b/packages/text-track/src/manifest.tsindex 7a394af8..6b51830d 100644--- a/packages/text-track/src/manifest.ts+++ b/packages/text-track/src/manifest.ts@@ -6,13 +6,15 @@ import {import { digestOf } from "@svml/protocol";import type { ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";-import { renderTextTrackImplementationDigest } from "./program.js";+import { compileTextTrackImplementationDigest, renderTextTrackImplementationDigest } from "./program.js";export const textTrackModuleRef = { name: "@svml/text-track", version: "0.0.0-dev" } as const;export const textTrackTypes = {program: { module: textTrackModuleRef, name: "TextTrackProgram" },+ spec: { module: textTrackModuleRef, name: "TextTrackSpec" },} satisfies Record<string, TypeRef>;export const textTrackProducers = {+ compile: { module: textTrackModuleRef, name: "compile-text-track" },render: { module: textTrackModuleRef, name: "render-text-track" },} satisfies Record<string, ProducerRef>;@@ -27,6 +29,20 @@ const object = (fields: Readonly<Record<string, { readonly schema: ValueSchema;fields,});+const textAppearanceSchema = object({+ color: { schema: string },+ fontSizePx: { schema: positiveNumber },+ fontFamily: { schema: string, optional: true },+ fontWeight: { schema: { kind: "number", integer: true, minimum: 1, maximum: 1000 }, optional: true },+ lineHeight: { schema: positiveNumber, optional: true },+ align: { schema: { kind: "string", enum: ["left", "center", "right"] }, optional: true },+ verticalAlign: { schema: { kind: "string", enum: ["top", "center", "bottom"] }, optional: true },+ backgroundColor: { schema: string, optional: true },+ borderRadiusPx: { schema: { kind: "number", minimum: 0 }, optional: true },+ paddingPx: { schema: { kind: "number", minimum: 0 }, optional: true },+ letterSpacingPx: { schema: number, optional: true },+});+const textItemSchema = object({id: { schema: string },text: { schema: string },@@ -42,18 +58,7 @@ const textItemSchema = object({widthPercent: { schema: positiveNumber },heightPercent: { schema: positiveNumber },}) },- appearance: { schema: object({- color: { schema: string },- fontSizePx: { schema: positiveNumber },- fontFamily: { schema: string, optional: true },- fontWeight: { schema: { kind: "number", integer: true, minimum: 1, maximum: 1000 }, optional: true },- lineHeight: { schema: positiveNumber, optional: true },- align: { schema: { kind: "string", enum: ["left", "center", "right"] }, optional: true },- verticalAlign: { schema: { kind: "string", enum: ["top", "center", "bottom"] }, optional: true },- backgroundColor: { schema: string, optional: true },- borderRadiusPx: { schema: { kind: "number", minimum: 0 }, optional: true },- paddingPx: { schema: { kind: "number", minimum: 0 }, optional: true },- }) },+ appearance: { schema: textAppearanceSchema },});export const textTrackProgramSchema: ValueSchema = object({@@ -64,6 +69,25 @@ export const textTrackProgramSchema: ValueSchema = object({items: { schema: { kind: "array", minItems: 1, items: textItemSchema } },});+export const textTrackSpecSchema: ValueSchema = object({+ contract: { schema: { kind: "literal", value: "svml.text-track-spec@1" } },+ digest: { schema: digest },+ id: { schema: string },+ items: { schema: { kind: "array", minItems: 1, items: object({+ id: { schema: string },+ text: { schema: string },+ during: { schema: { kind: "literal", value: "full" } },+ z: { schema: signedInteger },+ box: { schema: object({+ xPercent: { schema: number }, yPercent: { schema: number },+ widthPercent: { schema: positiveNumber }, heightPercent: { schema: positiveNumber },+ }) },+ appearance: { schema: textAppearanceSchema },+ }) } },+});++export const textTrackSurfaceImplementationDigest = digestOf("@svml/text-track/track-surface@1");+export const textTrackManifest: ModuleManifest = {format: "svml.module@0",name: textTrackModuleRef.name,@@ -72,10 +96,29 @@ export const textTrackManifest: ModuleManifest = {videoContractDependencies.programSpace,videoContractDependencies.composition,],- types: [{ name: textTrackTypes.program.name, schema: textTrackProgramSchema }],+ types: [+ { name: textTrackTypes.program.name, schema: textTrackProgramSchema },+ { name: textTrackTypes.spec.name, schema: textTrackSpecSchema },+ ],capabilities: [],- surfaces: [],+ surfaces: [{+ name: "track", tag: "Track", mode: "structured",+ outputs: [textTrackTypes.spec, textTrackTypes.program, contractTypes.visualTrack],+ implementation: { kind: "trusted-frontend-surface", locator: "@svml/text-track/track-surface", digest: textTrackSurfaceImplementationDigest },+ }],producers: [{+ name: textTrackProducers.compile.name,+ inputs: [+ { name: "space", type: contractTypes.programSpace },+ { name: "spec", type: textTrackTypes.spec },+ ],+ outputs: [{+ name: "program", type: textTrackTypes.program,+ affinity: [{ resultPointer: "/programSpaceDigest", input: "space", inputPointer: "/digest" }],+ }],+ needs: [],+ implementation: { kind: "registered", locator: "@svml/text-track/compile", digest: compileTextTrackImplementationDigest },+ }, {name: textTrackProducers.render.name,inputs: [{ name: "space", type: contractTypes.programSpace },diff --git a/packages/text-track/src/program.ts b/packages/text-track/src/program.tsindex dd6ee428..ad73f0af 100644--- a/packages/text-track/src/program.ts+++ b/packages/text-track/src/program.ts@@ -8,9 +8,10 @@ import type { ProgramSpace, VisualStyleDeclaration, VisualTrack } from "@svml/coimport { digestOf, isDigest } from "@svml/protocol";import type { Digest } from "@svml/protocol";-import type { TextAppearance, TextItem, TextTrackProgram } from "./types.js";+import type { TextAppearance, TextItem, TextTrackProgram, TextTrackSpec } from "./types.js";export const renderTextTrackImplementationDigest = digestOf("@svml/text-track/render@1");+export const compileTextTrackImplementationDigest = digestOf("@svml/text-track/compile@1");function assertNonEmpty(value: string, label: string): void {if (!value.trim()) throw new Error(`${label} must not be empty.`);@@ -80,6 +81,7 @@ function assertAppearance(appearance: TextAppearance, label: string): void {}}if (appearance.backgroundColor !== undefined) assertNonEmpty(appearance.backgroundColor, `${label} backgroundColor`);+ if (appearance.letterSpacingPx !== undefined) assertFinite(appearance.letterSpacingPx, `${label} letterSpacingPx`);}export function assertTextTrackProgramIdentity(program: TextTrackProgram, programSpace: ProgramSpace): void {@@ -154,9 +156,68 @@ function textStyle(appearance: TextAppearance): VisualStyleDeclaration[] {{ name: "line-height", value: appearance.lineHeight ?? 1.2 },{ name: "text-align", value: appearance.align ?? "center" },...(appearance.fontFamily === undefined ? [] : [{ name: "font-family", value: appearance.fontFamily }]),+ ...(appearance.letterSpacingPx === undefined ? [] : [{ name: "letter-spacing", value: `${appearance.letterSpacingPx}px` }]),];}+export function sealTextTrackSpec(value: Omit<TextTrackSpec, "digest">): TextTrackSpec {+ const content = canonicalSpec(value);+ return { ...content, digest: digestOf(content) };+}++function canonicalSpec(value: Omit<TextTrackSpec, "digest">): Omit<TextTrackSpec, "digest"> {+ return {+ contract: "svml.text-track-spec@1",+ id: value.id,+ items: value.items.map((item) => ({+ id: item.id,+ text: item.text,+ during: "full",+ z: item.z,+ box: { ...item.box },+ appearance: { ...item.appearance },+ })),+ };+}++export function assertTextTrackSpec(spec: TextTrackSpec): void {+ if (spec.contract !== "svml.text-track-spec@1" || !spec.id || spec.items.length === 0) {+ throw new Error("TextTrackSpec identity or items are invalid.");+ }+ const { digest: _digest, ...content } = spec;+ if (!isDigest(spec.digest) || spec.digest !== digestOf(canonicalSpec(content))) {+ throw new Error("TextTrackSpec digest differs from its contents.");+ }+ const ids = new Set<string>();+ for (const item of spec.items) {+ if (!item.id || !item.text || item.during !== "full" || ids.has(item.id) || !Number.isSafeInteger(item.z)) {+ throw new Error("TextTrackSpec contains an invalid item.");+ }+ ids.add(item.id);+ for (const value of Object.values(item.box)) assertFinite(value, `${item.id} box`);+ assertAppearance(item.appearance, `${item.id} appearance`);+ }+}++export function compileTextTrackProgram(programSpace: ProgramSpace, spec: TextTrackSpec): TextTrackProgram {+ assertProgramSpaceIdentity(programSpace);+ assertTextTrackSpec(spec);+ return sealTextTrackProgram({+ contract: "svml.text-track-program@1",+ id: spec.id,+ programSpaceDigest: programSpace.digest,+ items: spec.items.map((item) => ({+ id: item.id,+ text: item.text,+ span: { startFrame: 0, endFrameExclusive: programSpaceFrameCount(programSpace) },+ z: item.z,+ tieBreak: `${spec.id}:${item.id}`,+ box: item.box,+ appearance: item.appearance,+ })),+ });+}+export function renderTextTrack(programSpace: ProgramSpace, program: TextTrackProgram): VisualTrack {assertTextTrackProgramIdentity(program, programSpace);const track = sealVisualTrack({diff --git a/packages/text-track/src/surface.ts b/packages/text-track/src/surface.tsnew file mode 100644index 00000000..aa5d2877--- /dev/null+++ b/packages/text-track/src/surface.ts@@ -0,0 +1,105 @@+import { contractTypes } from "@svml/contracts";+import { sealGraphFragment } from "@svml/elaborator";+import { svsRecipeType } from "@svml/svs";+import type { SvsRecipe } from "@svml/svs";+import type { StructuredElement, StructuredSurfaceHandler, SurfaceResolvedReference, TextAttributeValue } from "@svml/text";++import { textTrackProducers, textTrackTypes } from "./manifest.js";+import { sealTextTrackSpec } from "./program.js";++const input = (name: string) => ({ kind: "fragment-input" as const, name });+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });++const textTrackSurfaceFragment = sealGraphFragment({+ name: "@svml/text-track/track-surface@1",+ inputs: [+ { name: "space", type: contractTypes.programSpace },+ { name: "spec", type: textTrackTypes.spec },+ ],+ operations: [+ { id: "text:compile", producer: textTrackProducers.compile, inputs: { space: input("space"), spec: input("spec") }, result: { kind: "output", name: "program" } },+ { id: "text:render", producer: textTrackProducers.render, inputs: { space: input("space"), program: operation("text:compile") }, result: { kind: "output", name: "track" } },+ ],+ exports: [{+ name: "track", type: contractTypes.visualTrack, root: operation("text:render"), semanticInputs: ["space", "spec"],+ affinity: [{ resultPointer: "/programSpaceDigest", source: input("space"), sourcePointer: "/digest" }], fidelity: "exact",+ }],+});++function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;+}+function exact(element: StructuredElement, names: readonly string[]): void {+ if (Object.keys(element.attributes).sort().join("\0") !== [...names].sort().join("\0")) throw new Error(`${element.name} requires exactly ${names.join(", ")}`);+}+function text(element: StructuredElement, name: string): string {+ const value = element.attributes[name];+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);+ return value.trim();+}+function ref(element: StructuredElement, name: string, expected: SurfaceResolvedReference["type"], resolve: (path: string) => SurfaceResolvedReference | undefined) {+ const raw: TextAttributeValue | undefined = element.attributes[name];+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a reference`);+ const value = resolve(raw.path);+ if (!value || !sameType(value.type, expected)) throw new Error(`${element.name}.${name} cannot resolve a value of the required type`);+ return value;+}+function recipe(value: SurfaceResolvedReference): SvsRecipe {+ if (!sameType(value.type, svsRecipeType) || value.record?.value.kind !== "inline") throw new Error("Text appearance must be an authored SVS Recipe");+ return value.record.value.value as unknown as SvsRecipe;+}+function propNumber(value: SvsRecipe, name: string): number {+ const result = value.properties[name];+ if (typeof result !== "number" || !Number.isFinite(result)) throw new Error(`Text Recipe ${name} must be a number`);+ return result;+}+function propString(value: SvsRecipe, name: string): string {+ const result = value.properties[name];+ if (typeof result !== "string" || !result) throw new Error(`Text Recipe ${name} must be a string`);+ return result;+}++export const decodeTextTrackSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ exact(element, ["id", "space"]);+ const id = text(element, "id");+ const space = ref(element, "space", contractTypes.programSpace, resolveReference);+ let itemIndex = 0;+ const items = element.children.flatMap((child) => {+ if (child.kind === "text") {+ if (child.value.trim()) throw new Error(`${element.name} accepts only Item children`);+ return [];+ }+ if (!child.name.endsWith(":Item") && child.name !== "Item") throw new Error(`${element.name} accepts only Item children`);+ itemIndex += 1;+ exact(child, ["text", "during", "appearance"]);+ if (text(child, "during") !== "full") throw new Error(`${child.name}.during currently accepts only full`);+ const appearance = recipe(ref(child, "appearance", svsRecipeType, resolveReference));+ const expected = ["align", "fill", "font", "height", "size", "stack-order", "tracking", "weight", "width", "x", "y"];+ if (Object.keys(appearance.properties).sort().join("\0") !== expected.sort().join("\0")) throw new Error(`Text Recipe requires exactly ${expected.join(", ")}`);+ const align = propString(appearance, "align");+ if (align !== "left" && align !== "center" && align !== "right") throw new Error("Text Recipe align is invalid");+ const textAlign = align as "left" | "center" | "right";+ return [{+ id: `item-${String(itemIndex).padStart(4, "0")}`,+ text: text(child, "text"), during: "full" as const,+ z: propNumber(appearance, "stack-order"),+ box: {+ xPercent: propNumber(appearance, "x") * 100, yPercent: propNumber(appearance, "y") * 100,+ widthPercent: propNumber(appearance, "width") * 100, heightPercent: propNumber(appearance, "height") * 100,+ },+ appearance: {+ color: propString(appearance, "fill"), fontSizePx: propNumber(appearance, "size"),+ fontFamily: propString(appearance, "font"), fontWeight: propNumber(appearance, "weight"),+ letterSpacingPx: propNumber(appearance, "tracking"), align: textAlign,+ },+ }];+ });+ if (items.length === 0) throw new Error(`${element.name} requires at least one Item`);+ const spec = sealTextTrackSpec({ contract: "svml.text-track-spec@1", id, items });+ const specId = `${id}.spec`;+ return {+ records: [{ id: specId, type: textTrackTypes.spec, value: { kind: "inline", value: spec }, range: element.range }],+ components: [{ id, fragment: textTrackSurfaceFragment.id, inputs: { space: space.ref, spec: { kind: "record", id: specId } }, outputs: { track: `${id}.track` }, range: element.range }],+ fragments: [textTrackSurfaceFragment],+ };+};diff --git a/packages/text-track/src/types.ts b/packages/text-track/src/types.tsindex b90e9fe0..56f17a81 100644--- a/packages/text-track/src/types.ts+++ b/packages/text-track/src/types.ts@@ -19,6 +19,7 @@ export type TextAppearance = {readonly backgroundColor?: string;readonly borderRadiusPx?: number;readonly paddingPx?: number;+ readonly letterSpacingPx?: number;};export type TextItem = {@@ -39,3 +40,17 @@ export type TextTrackProgram = {readonly programSpaceDigest: Digest;readonly items: readonly TextItem[];};++export type TextTrackSpec = {+ readonly contract: "svml.text-track-spec@1";+ readonly digest: Digest;+ readonly id: string;+ readonly items: readonly {+ readonly id: string;+ readonly text: string;+ readonly during: "full";+ readonly z: number;+ readonly box: TextBox;+ readonly appearance: TextAppearance;+ }[];+};diff --git a/packages/text/test/text.test.ts b/packages/text/test/text.test.tsindex 6ffcb2ff..07e0ed3f 100644--- a/packages/text/test/text.test.ts+++ b/packages/text/test/text.test.ts@@ -51,11 +51,17 @@ test("Text learns <script> only from an imported Script Manifest", async () => {scriptContext(),);- assert.equal(result.module.records.length, 2);+ assert.equal(result.module.records.length, 5);assert.equal(result.module.records[0]?.id, "story");assert.equal(result.module.records[0]?.type.name, "Narrative");- assert.equal(result.module.records[1]?.id, "story.segment.opening");- assert.equal(result.module.records[1]?.type.name, "NarrativeExcerpt");+ assert.equal(result.module.records.some((record) =>+ record.id === "story.segment.opening" && record.type.name === "NarrativeExcerpt"), true);+ assert.equal(result.module.records.some((record) =>+ record.id === "story.segment.opening.dialogue" && record.type.name === "NarrativeDialogueExcerpt"), true);+ assert.equal(result.module.records.some((record) =>+ record.id === "story.segment.opening.speech" && record.type.name === "NarrativeSpeechExcerpt"), true);+ assert.equal(result.module.records.some((record) =>+ record.id === "story.caption" && record.type.name === "CaptionProjection"), true);assert.equal(result.sourceMaps.length, 1);assert.doesNotThrow(() => link(scriptContext().closure, [result.module]));});diff --git a/packages/whisperx/package.json b/packages/whisperx/package.jsonindex 25a68b69..1fd28c31 100644--- a/packages/whisperx/package.json+++ b/packages/whisperx/package.json@@ -9,6 +9,10 @@"dependencies": {"@svml/component-kit": "workspace:*","@svml/contracts": "workspace:*",- "@svml/protocol": "workspace:*"+ "@svml/elaborator": "workspace:*",+ "@svml/media-pipeline": "workspace:*",+ "@svml/protocol": "workspace:*",+ "@svml/speech-align": "workspace:*",+ "@svml/text": "workspace:*"}}diff --git a/packages/whisperx/src/fragment.ts b/packages/whisperx/src/fragment.tsnew file mode 100644index 00000000..a32bb9fa--- /dev/null+++ b/packages/whisperx/src/fragment.ts@@ -0,0 +1,76 @@+import { contractTypes } from "@svml/contracts";+import { sealGraphFragment } from "@svml/elaborator";+import { mediaPipelineProducers } from "@svml/media-pipeline";+import { speechAlignProducers } from "@svml/speech-align";++import { whisperXProducers, whisperXTypes } from "./manifest.js";++const input = (name: string) => ({ kind: "fragment-input" as const, name });+const operation = (id: string) => ({ kind: "fragment-operation" as const, operation: id });++/** One measured acoustic pass followed by provider-neutral deterministic Script alignment. */+export const whisperXSpeechAlignmentFragment = sealGraphFragment({+ name: "@svml/whisperx/speech-alignment@1",+ inputs: [+ { name: "narrative", type: contractTypes.narrative },+ { name: "audio", type: contractTypes.speechAudioBasis },+ ],+ operations: [+ {+ id: "prepare-evidence-audio",+ producer: mediaPipelineProducers.projectSpeechEvidenceAudio,+ inputs: { audio: input("audio") },+ result: { kind: "need", name: "evidenceAudio", accepts: "exact" },+ },+ {+ id: "request-whisperx",+ producer: whisperXProducers.request,+ inputs: { audio: operation("prepare-evidence-audio") },+ result: { kind: "need", name: "alignment", accepts: "exact" },+ },+ {+ id: "normalize-evidence",+ producer: whisperXProducers.normalize,+ inputs: { whisperx: operation("request-whisperx") },+ result: { kind: "output", name: "evidence" },+ },+ {+ id: "locate-speech",+ producer: speechAlignProducers.locate,+ inputs: { narrative: input("narrative"), audio: input("audio"), evidence: operation("normalize-evidence") },+ result: { kind: "output", name: "map" },+ },+ ],+ exports: [+ {+ name: "rawEvidence",+ type: whisperXTypes.alignmentEvidence,+ root: operation("request-whisperx"),+ semanticInputs: ["audio"],+ affinity: [+ { resultPointer: "/audioArtifactDigest", source: operation("prepare-evidence-audio"), sourcePointer: "/artifact/digest" },+ { resultPointer: "/programSpaceDigest", source: input("audio"), sourcePointer: "/programSpace/digest" },+ ],+ fidelity: "exact",+ },+ {+ name: "evidence",+ type: contractTypes.alignedTranscriptEvidence,+ root: operation("normalize-evidence"),+ semanticInputs: ["audio"],+ affinity: [+ { resultPointer: "/audioArtifactDigest", source: operation("prepare-evidence-audio"), sourcePointer: "/artifact/digest" },+ { resultPointer: "/programSpaceDigest", source: input("audio"), sourcePointer: "/programSpace/digest" },+ ],+ fidelity: "exact",+ },+ {+ name: "map",+ type: contractTypes.completeSemanticMap,+ root: operation("locate-speech"),+ semanticInputs: ["narrative", "audio"],+ affinity: [{ resultPointer: "/programSpace/digest", source: input("audio"), sourcePointer: "/programSpace/digest" }],+ fidelity: "exact",+ },+ ],+});diff --git a/packages/whisperx/src/index.ts b/packages/whisperx/src/index.tsindex 6f466be7..8c0fa497 100644--- a/packages/whisperx/src/index.ts+++ b/packages/whisperx/src/index.ts@@ -1,6 +1,7 @@export {whisperXComponent,} from "./component.js";+export { whisperXSpeechAlignmentFragment } from "./fragment.js";export {normalizeWhisperXAlignment,sealWhisperXAlignmentEvidence,@@ -15,4 +16,5 @@ export {whisperXProducers,whisperXTypes,} from "./manifest.js";+export { decodeWhisperXAlignmentSurface } from "./surface.js";export type * from "./types.js";diff --git a/packages/whisperx/src/manifest.ts b/packages/whisperx/src/manifest.tsindex 1b43b090..6954f479 100644--- a/packages/whisperx/src/manifest.ts+++ b/packages/whisperx/src/manifest.ts@@ -5,6 +5,8 @@ import {} from "@svml/contracts";import { digestOf } from "@svml/protocol";import type { CapabilityRef, ModuleManifest, ProducerRef, TypeRef, ValueSchema } from "@svml/protocol";+import { mediaPipelineManifest, mediaPipelineModuleRef } from "@svml/media-pipeline";+import { speechAlignManifest, speechAlignModuleRef } from "@svml/speech-align";export const whisperXModuleRef = { name: "@svml/whisperx", version: "0.0.0-dev" } as const;export const whisperXTypes = {@@ -20,6 +22,7 @@ export const whisperXProducers = {export const whisperXImplementationDigests = {request: digestOf("@svml/whisperx/request@2"),normalize: digestOf("@svml/whisperx/normalize@2"),+ surface: digestOf("@svml/whisperx/alignment-surface@1"),};const { contract: _commonContract, evidenceDigest: _commonDigest, ...sharedEvidenceFields } =@@ -41,13 +44,25 @@ export const whisperXManifest: ModuleManifest = {dependencies: [videoContractDependencies.speech,videoContractDependencies.semanticTime,+ { module: mediaPipelineModuleRef, digest: digestOf(mediaPipelineManifest) },+ { module: speechAlignModuleRef, digest: digestOf(speechAlignManifest) },],types: [{ name: whisperXTypes.alignmentEvidence.name, schema: whisperXAlignmentEvidenceSchema }],capabilities: [{name: whisperXCapabilities.alignment.name,returns: whisperXTypes.alignmentEvidence,}],- surfaces: [],+ surfaces: [{+ name: "alignment",+ tag: "Alignment",+ mode: "structured",+ outputs: [whisperXTypes.alignmentEvidence, contractTypes.alignedTranscriptEvidence, contractTypes.completeSemanticMap],+ implementation: {+ kind: "trusted-frontend-surface",+ locator: "@svml/whisperx/alignment-surface",+ digest: whisperXImplementationDigests.surface,+ },+ }],producers: [{name: whisperXProducers.request.name,diff --git a/packages/whisperx/src/surface.ts b/packages/whisperx/src/surface.tsnew file mode 100644index 00000000..754d1adb--- /dev/null+++ b/packages/whisperx/src/surface.ts@@ -0,0 +1,64 @@+import { contractTypes } from "@svml/contracts";+import type {+ StructuredElement,+ StructuredSurfaceHandler,+ SurfaceResolvedReference,+ TextAttributeValue,+} from "@svml/text";++import { whisperXSpeechAlignmentFragment } from "./fragment.js";++function exactAttributes(element: StructuredElement, names: readonly string[]): void {+ if (Object.keys(element.attributes).sort().join("\u0000") !== [...names].sort().join("\u0000")) {+ throw new Error(`${element.name} requires exactly ${names.join(", ")}`);+ }+}++function stringAttribute(element: StructuredElement, name: string): string {+ const value = element.attributes[name];+ if (typeof value !== "string" || !value.trim()) throw new Error(`${element.name}.${name} must be a non-empty string`);+ return value.trim();+}++function sameType(left: SurfaceResolvedReference["type"], right: SurfaceResolvedReference["type"]): boolean {+ return left.module.name === right.module.name && left.module.version === right.module.version && left.name === right.name;+}++function reference(+ element: StructuredElement,+ name: string,+ expected: SurfaceResolvedReference["type"],+ resolveReference: (path: string) => SurfaceResolvedReference | undefined,+): SurfaceResolvedReference {+ const raw: TextAttributeValue | undefined = element.attributes[name];+ if (typeof raw !== "object" || raw.kind !== "reference") throw new Error(`${element.name}.${name} must be a whole-value reference`);+ const value = resolveReference(raw.path);+ if (value === undefined) throw new Error(`${element.name}.${name} cannot resolve ${raw.path}`);+ if (!sameType(value.type, expected)) throw new Error(`${element.name}.${name} has the wrong type`);+ return value;+}++export const decodeWhisperXAlignmentSurface: StructuredSurfaceHandler = ({ element, resolveReference }) => {+ exactAttributes(element, ["id", "narrative", "audio"]);+ if (element.children.some((child) => child.kind === "element" || child.value.trim())) {+ throw new Error(`${element.name} does not accept children`);+ }+ const id = stringAttribute(element, "id");+ const narrative = reference(element, "narrative", contractTypes.narrative, resolveReference);+ const audio = reference(element, "audio", contractTypes.speechAudioBasis, resolveReference);+ return {+ records: [],+ components: [{+ id,+ fragment: whisperXSpeechAlignmentFragment.id,+ inputs: { narrative: narrative.ref, audio: audio.ref },+ outputs: {+ rawEvidence: `${id}.rawEvidence`,+ evidence: `${id}.evidence`,+ map: `${id}.map`,+ },+ range: element.range,+ }],+ fragments: [whisperXSpeechAlignmentFragment],+ };+};diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 3003d89a..c07fda22 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -45,12 +45,27 @@ importers:packages/broll:dependencies:+ '@svml/component-kit':+ specifier: workspace:*+ version: link:../component-kit'@svml/contracts':specifier: workspace:*version: link:../contracts+ '@svml/elaborator':+ specifier: workspace:*+ version: link:../elaborator+ '@svml/media-pipeline':+ specifier: workspace:*+ version: link:../media-pipeline'@svml/protocol':specifier: workspace:*version: link:../protocol+ '@svml/svs':+ specifier: workspace:*+ version: link:../svs+ '@svml/text':+ specifier: workspace:*+ version: link:../textdevDependencies:'@svml/hyperframes':specifier: workspace:*@@ -64,9 +79,18 @@ importers:'@svml/contracts':specifier: workspace:*version: link:../contracts+ '@svml/elaborator':+ specifier: workspace:*+ version: link:../elaborator'@svml/protocol':specifier: workspace:*version: link:../protocol+ '@svml/svs':+ specifier: workspace:*+ version: link:../svs+ '@svml/text':+ specifier: workspace:*+ version: link:../textdevDependencies:'@svml/script':specifier: workspace:*@@ -75,6 +99,31 @@ importers:specifier: workspace:*version: link:../speech-align+ packages/caption-gemini:+ dependencies:+ '@svml/caption':+ specifier: workspace:*+ version: link:../caption+ '@svml/component-kit':+ specifier: workspace:*+ version: link:../component-kit+ '@svml/contracts':+ specifier: workspace:*+ version: link:../contracts+ '@svml/elaborator':+ specifier: workspace:*+ version: link:../elaborator+ '@svml/protocol':+ specifier: workspace:*+ version: link:../protocol+ '@svml/text':+ specifier: workspace:*+ version: link:../text+ devDependencies:+ '@svml/script':+ specifier: workspace:*+ version: link:../script+packages/cli:dependencies:'@svml/compiler-node':@@ -89,6 +138,9 @@ importers:'@svml/prelude-video':specifier: workspace:*version: link:../prelude-video+ '@svml/protocol':+ specifier: workspace:*+ version: link:../protocolpackages/compiler-node:dependencies:@@ -216,6 +268,9 @@ importers:packages/film:dependencies:+ '@svml/component-kit':+ specifier: workspace:*+ version: link:../component-kit'@svml/contracts':specifier: workspace:*version: link:../contracts@@ -232,9 +287,6 @@ importers:specifier: workspace:*version: link:../textdevDependencies:- '@svml/component-kit':- specifier: workspace:*- version: link:../component-kit'@svml/core':specifier: workspace:*version: link:../core@@ -500,9 +552,15 @@ importers:packages/prelude-video:dependencies:+ '@svml/broll':+ specifier: workspace:*+ version: link:../broll'@svml/caption':specifier: workspace:*version: link:../caption+ '@svml/caption-gemini':+ specifier: workspace:*+ version: link:../caption-gemini'@svml/contracts':specifier: workspace:*version: link:../contracts@@ -533,12 +591,21 @@ importers:'@svml/speech-align':specifier: workspace:*version: link:../speech-align+ '@svml/speech-program':+ specifier: workspace:*+ version: link:../speech-program'@svml/speech-take':specifier: workspace:*version: link:../speech-take'@svml/svs':specifier: workspace:*version: link:../svs+ '@svml/text-track':+ specifier: workspace:*+ version: link:../text-track+ '@svml/whisperx':+ specifier: workspace:*+ version: link:../whisperxdevDependencies:'@svml/package-loader-node':specifier: workspace:*@@ -546,6 +613,34 @@ importers:packages/protocol: {}+ packages/provider-google-vertex:+ dependencies:+ '@google/genai':+ specifier: 1.52.0+ version: 1.52.0+ '@svml/caption':+ specifier: workspace:*+ version: link:../caption+ '@svml/caption-gemini':+ specifier: workspace:*+ version: link:../caption-gemini+ '@svml/endpoint-kit':+ specifier: workspace:*+ version: link:../endpoint-kit+ '@svml/protocol':+ specifier: workspace:*+ version: link:../protocol+ '@svml/runtime':+ specifier: workspace:*+ version: link:../runtime+ devDependencies:+ '@svml/driver-node':+ specifier: workspace:*+ version: link:../driver-node+ '@svml/script':+ specifier: workspace:*+ version: link:../script+packages/provider-hyperframes-local:dependencies:'@svml/contracts':@@ -779,6 +874,9 @@ importers:'@svml/caption':specifier: workspace:*version: link:../caption+ '@svml/component-kit':+ specifier: workspace:*+ version: link:../component-kit'@svml/contracts':specifier: workspace:*version: link:../contracts@@ -788,28 +886,28 @@ importers:'@svml/media-pipeline':specifier: workspace:*version: link:../media-pipeline+ '@svml/protocol':+ specifier: workspace:*+ version: link:../protocol'@svml/speech-align':specifier: workspace:*version: link:../speech-align'@svml/speech-take':specifier: workspace:*version: link:../speech-take+ '@svml/text':+ specifier: workspace:*+ version: link:../text'@svml/whisperx':specifier: workspace:*version: link:../whisperxdevDependencies:- '@svml/component-kit':- specifier: workspace:*- version: link:../component-kit'@svml/core':specifier: workspace:*version: link:../core'@svml/driver-node':specifier: workspace:*version: link:../driver-node- '@svml/protocol':- specifier: workspace:*- version: link:../protocol'@svml/realization':specifier: workspace:*version: link:../realization@@ -891,6 +989,9 @@ importers:packages/text-track:dependencies:+ '@svml/component-kit':+ specifier: workspace:*+ version: link:../component-kit'@svml/contracts':specifier: workspace:*version: link:../contracts@@ -900,6 +1001,12 @@ importers:'@svml/protocol':specifier: workspace:*version: link:../protocol+ '@svml/svs':+ specifier: workspace:*+ version: link:../svs+ '@svml/text':+ specifier: workspace:*+ version: link:../textdevDependencies:'@svml/hyperframes':specifier: workspace:*@@ -949,9 +1056,21 @@ importers:'@svml/contracts':specifier: workspace:*version: link:../contracts+ '@svml/elaborator':+ specifier: workspace:*+ version: link:../elaborator+ '@svml/media-pipeline':+ specifier: workspace:*+ version: link:../media-pipeline'@svml/protocol':specifier: workspace:*version: link:../protocol+ '@svml/speech-align':+ specifier: workspace:*+ version: link:../speech-align+ '@svml/text':+ specifier: workspace:*+ version: link:../textpackages/workspace-fs-node:dependencies:@@ -2440,7 +2559,6 @@ snapshots:- bufferutil- supports-color- utf-8-validate- optional: true'@hono/[email protected]([email protected])':dependencies:@@ -2556,34 +2674,25 @@ snapshots:dependencies:minipass: 7.1.3- '@protobufjs/[email protected]':- optional: true+ '@protobufjs/[email protected]': {}- '@protobufjs/[email protected]':- optional: true+ '@protobufjs/[email protected]': {}- '@protobufjs/[email protected]':- optional: true+ '@protobufjs/[email protected]': {}- '@protobufjs/[email protected]':- optional: true+ '@protobufjs/[email protected]': {}'@protobufjs/[email protected]':dependencies:'@protobufjs/aspromise': 1.1.2- optional: true- '@protobufjs/[email protected]':- optional: true+ '@protobufjs/[email protected]': {}- '@protobufjs/[email protected]':- optional: true+ '@protobufjs/[email protected]': {}- '@protobufjs/[email protected]':- optional: true+ '@protobufjs/[email protected]': {}- '@protobufjs/[email protected]':- optional: true+ '@protobufjs/[email protected]': {}'@puppeteer/[email protected]':dependencies:@@ -2631,13 +2740,11 @@ snapshots:dependencies:undici-types: 7.16.0- '@types/[email protected]':- optional: true+ '@types/[email protected]': {}- optional: true+ [email protected]: {}@@ -2645,8 +2752,7 @@ snapshots:- optional: true+ [email protected]: {}@@ -2656,8 +2762,7 @@ snapshots:dependencies:base64-js: 1.5.1- optional: true+ [email protected]: {}dependencies:@@ -2683,8 +2788,7 @@ snapshots:- optional: true+ [email protected]: {}dependencies:@@ -2722,7 +2826,6 @@ snapshots:dependencies:safe-buffer: 5.2.1- optional: true@@ -2794,8 +2897,7 @@ snapshots:- optional: true+ [email protected]: {}@@ -2803,7 +2905,6 @@ snapshots:dependencies:node-domexception: 1.0.0web-streams-polyfill: 3.3.3- optional: truedependencies:@@ -2820,7 +2921,6 @@ snapshots:dependencies:fetch-blob: 3.2.0- optional: trueoptional: true@@ -2832,7 +2932,6 @@ snapshots:node-fetch: 3.3.2transitivePeerDependencies:- supports-color- optional: truedependencies:@@ -2841,7 +2940,6 @@ snapshots:json-bigint: 1.0.0transitivePeerDependencies:- supports-color- optional: true@@ -2877,10 +2975,8 @@ snapshots:jws: 4.0.1transitivePeerDependencies:- supports-color- optional: true- optional: true+ [email protected]: {}@@ -2896,7 +2992,6 @@ snapshots:debug: 4.4.3transitivePeerDependencies:- supports-color- optional: true[email protected](@types/[email protected]):dependencies:@@ -2943,7 +3038,6 @@ snapshots:dependencies:bignumber.js: 9.3.1- optional: true@@ -2952,16 +3046,13 @@ snapshots:buffer-equal-constant-time: 1.0.1ecdsa-sig-formatter: 1.0.11safe-buffer: 5.2.1- optional: truedependencies:jwa: 2.0.1safe-buffer: 5.2.1- optional: true- optional: true+ [email protected]: {}dependencies:@@ -2981,15 +3072,13 @@ snapshots:- optional: true+ [email protected]: {}dependencies:data-uri-to-buffer: 4.0.1fetch-blob: 3.2.0formdata-polyfill: 4.0.10- optional: true@@ -3012,7 +3101,6 @@ snapshots:dependencies:'@types/retry': 0.12.0retry: 0.13.1- optional: true@@ -3039,7 +3127,6 @@ snapshots:'@protobufjs/utf8': 1.1.2'@types/node': 24.10.1long: 5.3.2- optional: truedependencies:@@ -3059,8 +3146,7 @@ snapshots:- optional: true+ [email protected]: {}dependencies:@@ -3073,8 +3159,7 @@ snapshots:- optional: true+ [email protected]: {}@@ -3173,8 +3258,7 @@ snapshots:pako: 0.2.9tiny-inflate: 1.0.3- optional: true+ [email protected]: {}diff --git a/spec/caption-program-v1.md b/spec/caption-program-v1.mdnew file mode 100644index 00000000..4d8a95d8--- /dev/null+++ b/spec/caption-program-v1.md@@ -0,0 +1,133 @@+# Caption Program v1++Status: implemented pre-release contract. Public names may still be polished before package+publication; the semantic laws below are executable.++## 1. Scope++Caption is an ordinary video component family. It owns the transformation from immutable authored+display text to one self-contained `VisualTrack`. It has no authority over Script wording,+pronunciation, speech alignment, global time, Composition order, Provider selection or Runtime+reuse policy.++The graph has two independent branches:++```text+Narrative.captionProjection + CaptionProgram ── planner ─────── CaptionPlan+speech audio ─────────────────────────────────── evidence/map ─ CompleteSemanticMap++Narrative + Program + Plan + Map ── temporalize ── TimedCaptionProjection+TimedCaptionProjection + Program ── render ─────── VisualTrack+```++The branches meet only at temporalization. Planning contains no timestamps, and alignment contains+no Caption Style or Cue judgment.++## 2. Display truth++Caption planning sees only the visible left side of Script. For Dual Text:++```svml+<SVML | semantic video markup language>+```++Caption sees `SVML`; speech generation and alignment see `semantic video markup language`. The+right side must never enter a Caption planning request.++The Caption package deterministically tokenizes every visible `CaptionProjection` region into an+ordered `CaptionDisplayAtom` universe. Each atom has a stable id, display text, Segment/Turn+ownership, optional Role and a source speech-token range. An atom's correspondence is either:++- `exact`: its own source speech-token range is proven;+- `region-envelope`: only the enclosing display region's speech range is proven.++The package does not invent word timing for the second case at this stage.++## 3. Complete Style intent++A `CaptionStyle` is a complete author declaration with two independent payloads:++- `planning`: one Cue instruction and zero or more typed per-atom field declarations;+- `presentation`: one complete visual appearance, presentation mode and absolute stacking order.++Styles are declared separately from where they apply. A field declaration has a stable id, value+schema, natural-language instruction and minimum/maximum assignment count per Cue. Boolean, enum+and bounded numeric values are supported. One atom may receive zero, one or multiple independent+fields; fields need not be contiguous.++A Style is a whole value. Applying another Style replaces it completely. The language does not+implicitly merge arbitrary visual properties, Cue rules or fields from two Styles.++## 4. Total assignment and ordered replacement++Every `CaptionProgram` must name one `default` Style. That Style initially owns every visible atom,+including roleless Turns. The Program then applies zero or more ordered `Use` rules:++```xml+<caption:Program id="captions" narrative={story} default={plain}>+ <caption:Use role="ALICE" style={alice}/>+ <caption:Use on={story.selection.special} style={impact}/>+</caption:Program>+```++A `Use` selector is either one exact Role label or one explicit Script Selection. It cannot contain+both. Rules run in source order; for every atom, the last matching rule wins. A rule that selects no+visible atom is an error. Authors never spell a complement merely to preserve the default.++Role is optional per Script Turn and resets at every Segment boundary. A roleless Turn therefore+receives the default unless an explicit Selection replaces it.++After assignment, the Program deterministically creates maximal contiguous runs. A run never+crosses a Segment or Turn boundary and every atom in it has one Style. Runs are disjoint, ordered and+partition the complete display universe exactly once.++## 5. Planner contract++A planner consumes resolved runs rather than interpreting Role, Selection or Style precedence. For+each run it may only:++1. split the atom sequence into one or more ordered Cues;+2. assign declared fields to atoms inside each Cue.++The result must contain every Program run exactly once. Cues must preserve order and partition the+run's atoms exactly once. Field ids, atom ids, value schemas and per-Cue cardinalities fail closed.+There is no response field for text, pronunciation, time or Style.++`@svml/caption-gemini` is one explicit implementation of this contract. It is not part of the+Caption package or Core. Other packages may implement the same output law without being called+Gemini, but the author graph must explicitly choose them.++## 6. Timing projection++`CompleteSemanticMap` remains the only global speech timing truth. Exact display atoms reuse their+source token windows. For a `region-envelope` alias, a Caption renderer may allocate local windows+inside that region's measured envelope. Such boundaries are labelled `estimated`, remain downstream+Caption facts and are never written back into the global map.++Multiple Cues inside one alias must receive ordered, non-overlapping local windows. They must not+each copy the same whole region envelope.++## 7. Rendering and composition++Each timed Cue selects the full presentation payload of its resolved Style. The renderer may expose+declared field values to its own visual lowering, but cannot sample or mutate another Track. The+result is exactly one ordinary `VisualTrack` whose Presents carry their own frame spans and absolute+stacking facts.++Film does not recognize Caption specially. It folds the returned Track alongside Speech, B-roll,+Text or any third-party Track satisfying the same public contract.++## 8. Explicit non-goals++Caption Program v1 does not define:++- Script correction or alternative wording;+- speaker diarization or speaker identity;+- implicit model routing;+- automatic cache reuse or prompt-based memoization;+- hidden Runtime fallback;+- a global Caption layer in Composition;+- field-specific visual semantics in Core.++Reuse, Existing Values and substitutes are selected by the external BuildRequest/realization+closure. They do not change Caption data or planner semantics.diff --git a/spec/script-surface-v1.md b/spec/script-surface-v1.mdindex f43a465b..a114eead 100644--- a/spec/script-surface-v1.md+++ b/spec/script-surface-v1.md@@ -536,9 +536,10 @@ temporal marker、Role Cue 或结构标签内部。注释可出现在 atom 之- 每个端点的左右 affinity 和端点源码位置。- 每个 MomentSet 的一个或多个有序 point、左右 affinity 和源码位置。-外部 Program 的 `role="label"` selector 从上述 turn token ranges 派生一个-SelectionSet;同一 label 的多次 turn 成为同一集合的多个有序 occurrence。-它是显式查询,不是 Role Cue 的隐式字幕行为,也不建立 speaker entity。+外部 Program 的 `role="label"` selector 可以从上述 Turn 范围投影自己的选择集合;+同一 label 的多次 Turn 是多个有序、互不相邻的命中。它是消费组件内部的显式查询,+不是 Role Cue 的隐式字幕行为,也不建立 speaker entity。当前 Caption Program 直接把+该查询投影到可见 Display Atom,不伪造一个作者写下的 Selection Record。SelectionSet/MomentSet 是消费者边界,不是 Segment 的子对象。典型外部关系是:@@ -563,6 +564,11 @@ Caption 的多样式、region、cue segmentation、annotation、mute 和 layout编译错误;这不是 Script 禁止 Selection 重叠。未来 Caption 若支持 overlay,只扩展 Caption Program,不扩展正文语法。+当前 Caption Program 使用一个覆盖全部可见词的默认 Style,再按源码顺序应用 Role+或显式 Selection 的整 Style 替换,最后命中的规则获胜。无 Role 的 Turn 自动保留默认+Style;修改一个局部区间不要求作者书写它的补集。完整规则见+[`caption-program-v1.md`](./caption-program-v1.md)。+## 9. v1 明确不在 Script 中表达以下能力不是遗漏的第七种 inline marker: