Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

feat: implement decoupled SVML v1 compiler

kashorin61f8fc848faa
diff --git a/README.md b/README.md
index d6c4d77c..05899a86 100644
--- a/README.md
+++ b/README.md
@@ -1,56 +1,60 @@
# SVML
Semantic Video Markup Language is a semantic source language for information-flow
-video. Script creates stable semantic addresses; a Composition selects one
-content-addressed ProgramBasis and one imported Locator Component whose SemanticMap
-binds those addresses to that basis; Track Components then lower the program into
-one deterministic HyperFrames document.
+video. Script creates stable semantic addresses. A Composition selects one
+content-addressed `ProgramBasis` and one exact `SemanticMap`; flat Track
+Components then lower every audiovisual contribution into one deterministic
+HyperFrames HTML document.
```text
-.svml + .svc + .svs + .svk + lock + evidence
- │
- ├─ Plan IR ─────────────────────> Canvas view
- ├─ Basis + SemanticMap ─────────> TemporalBinding
- ├─ TemporalBinding ─────────────> Timeline view
- └─ flat Track[] ────────────────> HyperFrames HTML
+.svml + transitive .svk/.svs/.svc imports + lock + evidence
+ │
+ ├─ typed Plan IR ─────────────────────────────> Canvas view
+ ├─ Basis Producer ─────> TemporalBasisProduction
+ ├─ Locator ────────────> ExactSemanticMap
+ ├─ Basis + Map ────────> TemporalBinding ─────> Timeline view
+ └─ flat Track[] ───────> Composition ─────────> HyperFrames HTML
```
-Canvas and Timeline are views of the same source closure. They are not additional
-authoring truths. Provider tasks, pins, takes and caches are runtime state and do
-not enter the source language.
+Canvas and Timeline are views of the same source closure, not additional
+authoring truths. Provider tasks, pins, takes, caches and queue state belong to a
+Runtime Host and do not enter the source language.
+
+## Implemented v1 slice
+
+The TypeScript compiler and local standard library implement the current v1
+architecture end to end:
+
+- readable Script with Role Cues, Dual Text, Slots, closed/disconnected
+ Selections and zero-width Moments;
+- an exact `SemanticIndex` with independent word and Segment endpoints
+ (`2M + 2N` identities);
+- replaceable `TemporalBasisProduction` and `ExactSemanticMap` Components,
+ validated and bound before any Track lowering;
+- hard cut, gap and audio/visual crossfade joins with distinct source-to-program
+ maps;
+- typed `.svc` content modules and `.svs` classes, including nested Component
+ fields;
+- stable Plan identity, separate execution digests and a full temporal
+ `svml.lock`;
+- isolated `.svk` projectors, verified offline capability artifacts and finite,
+ auditable `composite-v1` expansion;
+- manifest-enforced temporal `one` / `each` / `set` consumption;
+- flat media, ranking, B-roll, caption, text and audio Tracks in one absolute
+ ProgramSpace;
+- Film as the only `Track[]` consumer and HyperFrames HTML as the sole formal
+ video compilation target.
-## Current prototype
+There is no live provider adapter in this repository. A reachable
+`profile="capability-v1"` Component must be satisfied by an exact,
+content-verified `svml.artifacts.v1` binding; compilation never falls back to a
+provider call. Existing local `Image`, `Video`, `Audio` and Evidence values work
+directly.
-The repository contains a strict TypeScript compiler and an intentionally small
-stdlib. The deterministic slice is working end to end:
+SVML has not been publicly released. There is no legacy v1 compatibility layer:
+the current architecture and Script Surface are the only v1 target.
-- an early Script parser with dialogue, speech and caption projections;
-- closed, possibly disconnected Selection sets and zero-width Moments;
-- `.svc` content modules and typed `.svs` parameter classes;
-- stable Plan identities, separate execution digests and a verified `svml.lock`;
-- isolated `.svk` component lowering with scoped CSS and a JSON-only ABI;
-- typed `.svk` ports, parameters and recursive child-content schemas;
-- manifest-enforced temporal `one` / `each` / `set` consumption;
-- a temporary Speech Spine prototype, media/presentation mapping, ranking, B-roll,
- captions, text and audio;
-- frame-exact Located IR, Canvas/Timeline views and HyperFrames HTML.
-
-The current runtime deliberately has no live generation provider host. A
-`profile="capability-v1"` Component remains visible in Plan, but compilation requires
-an exact, content-verified `svml.artifacts.v1` binding for its outputs and never
-falls back to a provider call. Direct existing `Image`, `Video` and `Audio` values
-work as usual. This keeps the first end-to-end reference deterministic while
-preserving the eventual capability boundary.
-
-SVML has not been publicly released, so the repository does not preserve the early
-prototype as a compatibility contract. The sole v1 target is the architecture draft
-and Script Surface v1: a replaceable Basis Component, an imported Locator Component,
-and `2M + 2N` independent Segment/word endpoint identities. The executable prototype
-still uses shared adjacent Segment cuts and a temporary `speech-spine`; it is an
-incomplete implementation scheduled for direct replacement, not an older language
-version that v1 must support.
-
-## Commands
+## Quick start
Requires Node.js 22 and pnpm.
@@ -61,70 +65,73 @@ pnpm test
pnpm build
pnpm svml check examples/regen-ranking/regen-ranking.svml
-pnpm svml fmt examples/regen-ranking/regen-ranking.svml --check
+pnpm svml script examples/regen-ranking/regen-ranking.svml --out narrative.json
+pnpm svml plan examples/regen-ranking/regen-ranking.svml --out plan.json
pnpm svml canvas examples/regen-ranking/regen-ranking.svml --out canvas.json
-pnpm svml estimate examples/regen-ranking/regen-ranking.svml \
- --out estimated-alignment.json
-pnpm svml timeline examples/regen-ranking/regen-ranking.svml \
- --evidence examples/regen-ranking/evidence/alignment.json \
- --out timeline.json
+pnpm svml estimate examples/regen-ranking/regen-ranking.svml --out estimate.json
+pnpm svml timeline examples/regen-ranking/regen-ranking.svml --out timeline.json
+
+# Produce a full frozen lock, including the selected Basis, Map and Evidence.
pnpm svml lock examples/regen-ranking/regen-ranking.svml --out svml.lock
+
pnpm svml compile examples/regen-ranking/regen-ranking.svml \
- --evidence examples/regen-ranking/evidence/alignment.json \
--lock svml.lock \
--out build/index.html
pnpm svml render build/index.html --out build/video.mp4
```
-`--artifacts` is needed only when the reachable Plan contains capability Components.
-`fmt` currently canonicalizes only the prototype Script body and refuses any rewrite
-whose reparsed semantic IR differs.
-`canvas` requires no timing or provider result. `estimate` deterministically
-creates provisional syllable-based alignment; `timeline` and `compile` consume
-either that preview evidence or measured Speech Spine evidence. The current
-prototype verifies source closure and bound capability artifacts. It does not yet
-claim complete temporal reproducibility because Evidence, Locator, ProgramBasis and
-SemanticMap digests are not all part of the executable lock path.
-
-## Reference implementation
-
-[`examples/regen-ranking/regen-ranking.svml`](examples/regen-ranking/regen-ranking.svml)
-reconstructs a real production video using the temporary Speech Spine prototype,
-a five-item ranking track, five B-roll items, captions, title, music and sound effects. The media files
-are intentionally gitignored; their production provenance and frozen assertions
-are recorded in
-[`examples/regen-ranking/reference.json`](examples/regen-ranking/reference.json).
-The regression test compiles the example to 27 Canvas nodes, 29 typed topology
-edges, 1083 frames, 155 visual fragments and 14 audio fragments.
-The default syllable estimator predicts 36.267 seconds for the frozen 36.1-second
-speech program (0.46% duration error); it remains preview evidence, not measured
-alignment.
-
-[`examples/flat-track-launch/flat-track-launch.svml`](examples/flat-track-launch/flat-track-launch.svml)
-is the readable fixture for the v1 architecture target: separate Basis and
-Locator Components, Script Surface v1, continuous A-roll Present changes, B-roll
-transition audio, disconnected caption selections, absolute z and Film as the sole
-`Track[]` consumer. It is a draft fixture not yet accepted by the early compiler
-prototype.
-
-[`examples/composite-speech-program/minimal.svml`](examples/composite-speech-program/minimal.svml)
-and its adjacent `speech-program.svk` define the proposed `composite-v1` author
-surface: one transparent Component replaces repeated Basis/Locator wiring while
-both internal instances remain independently visible in Plan and lock. Its
-`SpeechProgram` output is only a typed alias bundle, per-Segment joins are author
-data consumed by one assembler, and the Component has no ambient document access.
-
-## Specification
-
-- [Script Surface v1](spec/script-surface-v1.md) — the sole draft target for the
- human-readable Script, including independent Segment endpoints and `2M + 2N`.
-- [Source Architecture Draft](spec/source-architecture-draft.md) — the current
- draft for `.svk` components, `.svs` parameter sheets, `.svc` content modules,
- Script-dependent generation, five public output boundaries, `composite-v1`, flat
- Tracks, and a Composition root.
-- [Compiler prototype record](docs/compiler-prototype.md) — implemented
- boundaries, real-video evidence and remaining work.
-- [External Runtime Host Architecture Draft](docs/runtime-host-architecture-draft.md)
- — portable runtime, Artifact, Effect/Receipt, Worker, queue, provider and
- self-hosting boundaries; TemporalBasisProduction acquisition remains
- intentionally deferred.
+`--artifacts` is required only when the reachable Plan contains capability
+Components. Exact alignment is an explicit typed value referenced by the
+Locator in source; it is not an ambient CLI argument. `estimate` returns an
+`EstimatedSemanticMap` for preview and cannot be supplied where final HTML
+requires an `ExactSemanticMap`.
+
+`fmt` currently canonicalizes only the Script body and refuses a rewrite if the
+reparsed semantic IR differs. `canvas` needs no materialized time evidence.
+`timeline`, `compile` and `lock` execute the deterministic local Plan and validate
+the selected Basis/Map pair.
+
+## Four source files
+
+| Suffix | Responsibility |
+|---|---|
+| `.svml` | One film's Script, local values, Component calls, Tracks and Composition |
+| `.svk` | An importable Component vocabulary, public typed ABI and lowering |
+| `.svs` | Typed parameter classes for public Component parameters and fields |
+| `.svc` | Context-free reusable content values and content subgraphs |
+
+The compiler owns universal language laws. Libraries own replaceable vocabulary
+and algorithms. A Runtime Host owns effects, credentials, queues and storage.
+Canvas owns only a human-readable view.
+
+## Executable examples
+
+- [`examples/regen-ranking/regen-ranking.svml`](examples/regen-ranking/regen-ranking.svml)
+ reconstructs a pinned production video with five ranking items, five B-roll
+ items, captions, title, BGM and sound effects. It compiles to 29 Plan nodes,
+ 33 typed edges, 1083 frames, 153 visual fragments and 14 audio fragments.
+- [`examples/flat-track-launch/flat-track-launch.svml`](examples/flat-track-launch/flat-track-launch.svml)
+ is the comprehensive language fixture: explicit Basis/Locator, nested SVS,
+ overlapping Presents, manual ProgramSpans, a Moment, B-roll source audio and
+ transition SFX, disconnected caption styling and absolute z.
+- [`examples/composite-speech-program/minimal.svml`](examples/composite-speech-program/minimal.svml)
+ proves that an author-facing `speech-program` can hide repetitive wiring while
+ expanding to separately auditable `voice::basis` and `voice::locator`
+ instances. The compiler has no special knowledge of that Component name.
+- [`examples/media-basis`](test/fixtures/media-basis) proves that precomposed
+ media can implement the same Basis contract and reuse the same Locator and
+ Composition path.
+
+The local media used by the examples are already-materialized artifacts; no
+generation provider is invoked by tests or compilation.
+
+## Specification and implementation record
+
+- [Script Surface v1](spec/script-surface-v1.md)
+- [Source Architecture v1 draft](spec/source-architecture-draft.md)
+- [Compiler implementation record](docs/compiler-prototype.md)
+- [External Runtime Host architecture](docs/runtime-host-architecture-draft.md)
+
+VLM, bbox, face tracking and visual reverse-location are intentionally outside
+SVML v1. They may be implemented later as explicit post-processing extensions;
+the semantic audio timeline remains the language's locating foundation.
diff --git a/examples/composite-speech-program/README.md b/examples/composite-speech-program/README.md
index b8e0a439..1d416eb5 100644
--- a/examples/composite-speech-program/README.md
+++ b/examples/composite-speech-program/README.md
@@ -1,49 +1,40 @@
-# Composite Speech Program draft fixture
+# Composite Speech Program executable fixture
-This directory demonstrates the proposed, unimplemented `composite-v1` SVK
-profile.
+This directory proves the implemented `composite-v1` profile.
- `minimal.svml` is the author-facing source.
-- `speech-program.svk` defines a callable Component that statically expands
- `speech-assemble` and `speech-locator`.
-- `house.svs` configures only the Composite's public join shorthand.
+- `speech-program.svk` statically expands `speech-assemble` and
+ `speech-locator`.
+- `house.svs` configures only public Composite parameters.
+- `alignment.json` is explicit typed Evidence.
-For `<speech-program id="voice">`, canonical expansion creates the internal
-instances `voice::basis` and `voice::locator`. Their implementations, digests,
-Evidence and Artifacts remain independently visible in Plan and lock. The outer
-ports are aliases:
+For `<speech-program id="voice">`, expansion creates:
```text
-voice.program.production = voice::basis.production
-voice.program.semantic = voice::locator.map
-voice.production = voice::basis.production
-voice.map = voice::locator.map
-voice.facets.visual = voice::basis.facets.visual
-```
+voice::basis → TemporalBasisProduction + ProgramBoundVideoSequence
+ │
+ └──────────────> voice::locator → ExactSemanticMap
-`voice.program` is a typed port bundle whose fields alias the two internal
-outputs. It does not copy audio, facets, Evidence or Artifacts, and it does not
-replace the compiler's basis/map affinity check. The imported Film facade accepts
-the bundle as author-facing shorthand; after Composite expansion the Plan still
-contains the selected Basis and ExactSemanticMap separately.
+voice.production = voice::basis.production
+voice.map = voice::locator.map
+voice.facets.visual = voice::basis.facets.visual
+```
-The default join and each `<join after="...">` are author data. They normalize to
-per-boundary `JoinSpec` values consumed by one `speech-assemble` instance; they do
-not select a language-level family or mode. `after` names a stable Segment id, and
-the following Segment comes from source order. A join cannot declare both `gap`
-and `overlap`.
+The two internal instances, implementation digests, Evidence dependencies and
+outputs remain visible in Plan and lock. The outer ports are aliases; the
+Composite does not copy Artifacts or merge execution boundaries.
-The v1 stdlib publishes `speech-program` as its chosen author surface. It remains
-an ordinary imported Composite, never a compiler keyword or a required singleton.
-Alternative Locators use explicit low-level wiring or another Composite instead
-of extending a closed `locator="..."` enum.
+`join` and `joinDuration` are author shorthand forwarded to one assembler. Each
+explicit `<join after="...">` may override the corresponding boundary. They do
+not select a compiler family or language-level mode.
The bracket expression in `ports.script.segment[item.id]` is a stable id-keyed
-lookup, not a positional array lookup. The `for` iterates only over finite children
-declared at the call site.
-
-The Composite receives its Script through an explicit typed input. Reading
-`document.script` or any other ambient caller state is forbidden.
-
-The current compiler must reject or clearly report unsupported `composite-v1`;
-these files are architecture fixtures, not executable regression inputs.
+lookup over finite declared children, not positional indexing. The Composite
+receives Script and Evidence through explicit typed inputs and cannot read
+ambient document state.
+
+```sh
+pnpm svml check examples/composite-speech-program/minimal.svml
+pnpm svml compile examples/composite-speech-program/minimal.svml \
+ --out /tmp/svml-composite/index.html
+```
diff --git a/examples/composite-speech-program/alignment.json b/examples/composite-speech-program/alignment.json
new file mode 100644
index 00000000..5cc43427
--- /dev/null
+++ b/examples/composite-speech-program/alignment.json
@@ -0,0 +1,21 @@
+{
+ "contract": "svml.speech-alignment.v1",
+ "durationSec": 9.5,
+ "fps": 30,
+ "words": [
+ { "text": "Hello", "startSec": 0.2, "endSec": 0.8, "segmentId": "intro" },
+ { "text": "this", "startSec": 1.0, "endSec": 1.5, "segmentId": "intro" },
+ { "text": "is", "startSec": 1.6, "endSec": 1.9, "segmentId": "intro" },
+ { "text": "SVML", "startSec": 2.0, "endSec": 2.8, "segmentId": "intro" },
+ { "text": "Write", "startSec": 4.7, "endSec": 5.2, "segmentId": "close" },
+ { "text": "the", "startSec": 5.3, "endSec": 5.6, "segmentId": "close" },
+ { "text": "script", "startSec": 5.7, "endSec": 6.2, "segmentId": "close" },
+ { "text": "get", "startSec": 6.4, "endSec": 6.7, "segmentId": "close" },
+ { "text": "the", "startSec": 6.8, "endSec": 7.1, "segmentId": "close" },
+ { "text": "video", "startSec": 7.2, "endSec": 8.0, "segmentId": "close" }
+ ],
+ "segments": [
+ { "id": "intro", "startSec": 0, "endSec": 5 },
+ { "id": "close", "startSec": 4.5, "endSec": 9.5 }
+ ]
+}
diff --git a/examples/composite-speech-program/minimal.svml b/examples/composite-speech-program/minimal.svml
index 02a13028..a15d9958 100644
--- a/examples/composite-speech-program/minimal.svml
+++ b/examples/composite-speech-program/minimal.svml
@@ -1,12 +1,7 @@
-<!--
- Proposed composite-v1 author surface. The current compiler does not implement
- this profile or the complete Script Surface v1 yet.
--->
<svml version="1">
-
<import from="./speech-program.svk"/>
- <import from="@svml/std/media-track.svk"/>
- <import from="@svml/std/film.svk"/>
+ <import from="../../stdlib/media-track.svk"/>
+ <import from="../../stdlib/film.svk"/>
<import as="house" from="./house.svs"/>
<script>
@@ -19,42 +14,21 @@
</segment>
</script>
- <video id="host" src="./host.mp4"/>
-
- <speech-program
- id="voice"
- script={script}
- class="house.default"
- >
- <segment
- id="intro"
- source={host}
- sourceWindow="0s .. 5s"
- />
- <segment
- id="close"
- source={host}
- sourceWindow="5s .. 10s"
- />
+ <video id="host" src="../regen-ranking/assets/avatar-1.mp4"/>
+ <audio id="host-audio" src="../regen-ranking/assets/avatar-1.mp3"/>
+ <alignment id="alignment" src="./alignment.json"/>
- <join
- after="intro"
- overlap="500ms"
- audio="crossfade"
- visual="dissolve"
- />
+ <speech-program id="voice" script={script} evidence={alignment} class="house.default">
+ <segment id="intro" visual={host} audio={host-audio} sourceWindow="0s .. 5s"/>
+ <segment id="close" visual={host} audio={host-audio} sourceWindow="5s .. 10s"/>
+ <join after="intro" overlap="500ms" audio="crossfade" visual="dissolve"/>
</speech-program>
- <media-track id="aroll">
- <item
- source={voice.facets.visual}
- during="full"
- playback="program-map"
- />
+ <media-track id="aroll" z="100">
+ <item id="host" source={voice.facets.visual} during="full" playback="program-map"/>
</media-track>
- <film program={voice.program}>
+ <film id="main" basis={voice.production} semantic={voice.map} width="1080" height="1920">
<track ref={aroll.track}/>
</film>
-
</svml>
diff --git a/examples/composite-speech-program/speech-program.svk b/examples/composite-speech-program/speech-program.svk
index a00423d6..0dc9970f 100644
--- a/examples/composite-speech-program/speech-program.svk
+++ b/examples/composite-speech-program/speech-program.svk
@@ -1,62 +1,18 @@
-<!-- Proposed composite-v1 syntax; not implemented by the current compiler. -->
-<kernel
- name="speech-program"
- abi="1"
- profile="composite-v1"
->
+<kernel name="speech-program" abi="1" profile="composite-v1">
+ <port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
+ <port name="evidence" direction="input" type="AlignmentEvidence" cardinality="one"/>
+ <port name="production" direction="output" type="TemporalBasisProduction" cardinality="one"/>
+ <port name="map" direction="output" type="ExactSemanticMap" cardinality="one"/>
+ <port name="facets.visual" direction="output" type="ProgramBoundVideoSequence" cardinality="one"/>
- <port
- name="script"
- direction="input"
- type="Script"
- cardinality="one"
- />
-
- <port
- name="production"
- direction="output"
- type="TemporalBasisProduction"
- cardinality="one"
- />
-
- <port
- name="program"
- direction="output"
- type="SpeechProgram"
- cardinality="one"
- />
-
- <port
- name="map"
- direction="output"
- type="ExactSemanticMap"
- cardinality="one"
- />
-
- <port
- name="facets.visual"
- direction="output"
- type="ProgramBoundVideo"
- cardinality="zero-or-one"
- />
-
- <param
- name="join"
- type="enum(cut,crossfade,gap)"
- default="cut"
- />
-
- <param
- name="joinDuration"
- style="join-duration"
- type="duration"
- default="0ms"
- />
+ <param name="join" type="enum(cut,crossfade,gap)" default="cut"/>
+ <param name="joinDuration" style="join-duration" type="duration" default="0ms"/>
<child name="segment" cardinality="one-or-more">
<field name="id" type="id" required="true"/>
- <field name="source" type="Video" required="true"/>
- <field name="sourceWindow" type="SourceRange"/>
+ <field name="visual" type="Video"/>
+ <field name="audio" type="Audio" required="true"/>
+ <field name="sourceWindow" type="SourceRange" required="true"/>
</child>
<child name="join" cardinality="zero-or-more">
@@ -67,32 +23,20 @@
<field name="visual" type="enum(cut,dissolve)"/>
</child>
- <import
- as="basis-component"
- from="@svml/std/speech-assemble.svk"
- />
-
- <import
- as="locator-component"
- from="@svml/std/speech-locator.svk"
- />
+ <import as="assemble" from="../../stdlib/speech-assemble.svk"/>
+ <import as="locate" from="../../stdlib/speech-locator.svk"/>
<compose>
-
- <basis-component
- id="basis"
- defaultJoin={params.join}
- defaultJoinDuration={params.joinDuration}
- >
+ <assemble id="basis" defaultJoin={params.join} defaultJoinDuration={params.joinDuration}>
<for each={children.segment} as="item">
<segment
id={item.id}
script={ports.script.segment[item.id]}
- source={item.source}
+ visual={item.visual}
+ audio={item.audio}
sourceWindow={item.sourceWindow}
/>
</for>
-
<for each={children.join} as="join">
<join
after={join.after}
@@ -102,36 +46,12 @@
visual={join.visual}
/>
</for>
- </basis-component>
-
- <locator-component
- id="locator"
- script={ports.script}
- basis={basis.production}
- />
-
- <export
- port="production"
- value={basis.production}
- />
+ </assemble>
- <export
- port="map"
- value={locator.map}
- />
-
- <export port="program">
- <value
- production={basis.production}
- semantic={locator.map}
- />
- </export>
-
- <export
- port="facets.visual"
- value={basis.facets.visual}
- />
+ <locate id="locator" script={ports.script} basis={basis.production} evidence={ports.evidence}/>
+ <export port="production" value={basis.production}/>
+ <export port="map" value={locator.map}/>
+ <export port="facets.visual" value={basis.facets.visual}/>
</compose>
-
</kernel>
diff --git a/examples/flat-track-launch/README.md b/examples/flat-track-launch/README.md
index 9940f1ec..4984925a 100644
--- a/examples/flat-track-launch/README.md
+++ b/examples/flat-track-launch/README.md
@@ -1,20 +1,23 @@
-# Flat Track Launch architecture fixture
+# Flat Track Launch executable fixture
-This source is the readable fixture for the unimplemented Source Architecture
-Draft. It deliberately exercises v1 contracts that the early compiler does not
-yet accept:
+This is the end-to-end v1 architecture fixture. It compiles without generation
+providers and exercises the contracts together:
-- Script Surface v1 independent Segment endpoints (`2M + 2N`);
-- a Basis Component that overlaps adjacent Segment media;
-- an imported Locator Component that produces the selected ExactSemanticMap;
-- one ProgramBasis and one absolute ProgramSpace;
-- flat visual/audio Track contributions with no Track nesting;
-- continuous A-roll source mapping across overlapping Present windows;
-- B-roll transition audio inside the B-roll Track;
-- disconnected caption selections, Moments and manual ProgramSpans;
-- Film as the sole `Track[]` consumer.
+- independent Script Segment endpoints (`2M + 2N`);
+- replaceable Basis Producer and Locator Components;
+- one explicit `ProgramBasis` plus `ExactSemanticMap`;
+- flat Tracks and absolute `z`, with Film as the only `Track[]` consumer;
+- one continuous A-roll mapping with overlapping Present windows;
+- manual `ProgramSpan`, a `Moment`, B-roll audio and a visual crossfade;
+- a disconnected SelectionSet driving one caption style rule;
+- top-level and nested Component fields configured by `.svs`.
-The `@svml/std/*` imports and local assets are illustrative package paths. The
-adjacent `launch.svs` makes every ProgramSpace placement explicit. Do not use this
-fixture as the executable regression until the flat Track milestone lands; use
-`../regen-ranking/regen-ranking.svml` for the current compiler.
+It reuses the checked-in local artifacts and exact alignment evidence from the
+adjacent `regen-ranking` example. No provider is called.
+
+```sh
+pnpm svml compile examples/flat-track-launch/flat-track-launch.svml \
+ --out /tmp/svml-flat-track-launch/index.html
+pnpm svml render examples/flat-track-launch/flat-track-launch.svml \
+ --out /tmp/svml-flat-track-launch.mp4
+```
diff --git a/examples/flat-track-launch/flat-track-launch.svml b/examples/flat-track-launch/flat-track-launch.svml
index 22bccfbb..a3d82c5f 100644
--- a/examples/flat-track-launch/flat-track-launch.svml
+++ b/examples/flat-track-launch/flat-track-launch.svml
@@ -1,263 +1,122 @@
-<!--
- Architecture fixture for Source Architecture Draft.
- It is intentionally not accepted by the current early compiler prototype.
--->
<svml version="1">
- <import from="@svml/std/speech-assemble.svk"/>
- <import from="@svml/std/speech-locator.svk"/>
- <import from="@svml/std/media-track.svk"/>
- <import from="@svml/std/broll-track.svk"/>
- <import from="@svml/std/ranking-tier-list.svk"/>
- <import from="@svml/std/text-track.svk"/>
- <import from="@svml/std/caption-track.svk"/>
- <import from="@svml/std/audio-track.svk"/>
- <import from="@svml/std/film.svk"/>
-
+ <import from="../../stdlib/speech-assemble.svk"/>
+ <import from="../../stdlib/speech-locator.svk"/>
+ <import from="../../stdlib/media-track.svk"/>
+ <import from="../../stdlib/broll-track.svk"/>
+ <import from="../../stdlib/ranking-column.svk"/>
+ <import from="../../stdlib/text-track.svk"/>
+ <import from="../../stdlib/caption-track.svk"/>
+ <import from="../../stdlib/audio-track.svk"/>
+ <import from="../../stdlib/film.svk"/>
<import as="house" from="./launch.svs"/>
<script>
@whole
- @prelude
- <segment id="prelude"/>
- @/prelude
+ <segment id="scene-1">
+ <NARRATOR> @photoshop @emphasis Photoshop @/emphasis.
+ <SPEAKER> Powerful, but only if you know how to use it. Otherwise, your new profile
+ picture becomes a three-hour design project @/photoshop.
+ <NARRATOR> @facetune Facetune.
+ <SPEAKER> Fast and great for touch-ups. But it still leaves you with the same awkward
+ pose and the same dorm-room background @/facetune.
+ </segment>
- <segment id="intro">
- <HOST> @promise @emphasis Video creation is about to change @/emphasis.
- @/promise~ @ranking! @rank-svml SVML starts with meaning, not pixels.
- @/rank-svml
+ <segment id="scene-2">
+ <NARRATOR> @remini Remini.
+ <SPEAKER> Great for blurry photos. It restores detail, but it can't fix the pose,
+ outfit, or background. You get the same bad photo in HD @/remini.
+ <NARRATOR> @chatgpt ChatGPT.
+ <SPEAKER> More flexible. It can create a new shot, but getting your face and the
+ details right becomes a prompt-writing group project @/chatgpt.
</segment>
- <segment id="close">
- <HOST> @reveal One semantic source becomes @emphasis every frame @/emphasis,
- every track, and every version. @/reveal
+ <segment id="scene-3">
+ <NARRATOR> @ranking! @regen ReGen.
+ <SPEAKER> Pick a photo you love, and the app @handsome-1 rebuilds it around you with a
+ @/handsome-1~ @handsome-2 @emphasis better pose @/emphasis, outfit, and setting. @/handsome-2
+ From one selfie, you can make @dating-photo a dating photo, @/dating-photo~
+ @linkedin-headshot LinkedIn headshot, @/linkedin-headshot~
+ @instagram-post or Instagram post, @/instagram-post
+ so you get what you wanted instead of fixing a bad shot @/regen.
</segment>
@/whole~
</script>
- <!-- Plain values can fan out to Basis and Track Components. -->
- <video id="host-long" src="./assets/host-50s.mp4"/>
- <image id="poster" src="./assets/poster.png"/>
- <video id="product-demo" src="./assets/product-demo.mp4"/>
- <image id="rank-icon-svml" src="./assets/svml.png"/>
- <image id="rank-icon-editors" src="./assets/editors.png"/>
- <audio id="whoosh" src="./assets/whoosh.wav"/>
- <audio id="music" src="./assets/music.mp3"/>
-
- <!--
- Basis production assembles physical media only. It emits ProgramBasis,
- source maps, program audio and a program-bound visual facet. The Locator
- Component separately binds this Script's 2M + 2N identities to the basis.
- -->
- <speech-assemble id="voice" defaultJoin="cut">
- <segment
- id="prelude"
- script={script.segment.prelude}
- source={host-long}
- sourceWindow="0s .. 10s"
- />
- <segment
- id="intro"
- script={script.segment.intro}
- source={host-long}
- sourceWindow="10s .. 30s"
- />
- <segment
- id="close"
- script={script.segment.close}
- source={host-long}
- sourceWindow="30s .. 50s"
- />
-
- <join
- after="prelude"
- overlap="500ms"
- audio="crossfade"
- visual="dissolve"
- />
-
- <join
- after="intro"
- overlap="500ms"
- audio="crossfade"
- visual="dissolve"
- />
+ <!-- These local artifacts are inputs; the example invokes no generation model. -->
+ <video id="avatar-1" src="../regen-ranking/assets/avatar-1.mp4"/>
+ <video id="avatar-2" src="../regen-ranking/assets/avatar-2.mp4"/>
+ <video id="avatar-3" src="../regen-ranking/assets/avatar-3.mp4"/>
+ <audio id="avatar-1-audio" src="../regen-ranking/assets/avatar-1.mp3"/>
+ <audio id="avatar-2-audio" src="../regen-ranking/assets/avatar-2.mp3"/>
+ <audio id="avatar-3-audio" src="../regen-ranking/assets/avatar-3.mp3"/>
+ <image id="ranking-icon-1" src="../regen-ranking/assets/ranking-icon-1.jpg"/>
+ <image id="ranking-icon-2" src="../regen-ranking/assets/ranking-icon-2.jpg"/>
+ <image id="ranking-icon-3" src="../regen-ranking/assets/ranking-icon-3.jpg"/>
+ <image id="ranking-icon-4" src="../regen-ranking/assets/ranking-icon-4.jpg"/>
+ <image id="ranking-icon-5" src="../regen-ranking/assets/ranking-icon-5.jpg"/>
+ <image id="broll-1" src="../regen-ranking/assets/broll-5.png"/>
+ <audio id="ranking-appear" src="../regen-ranking/assets/ranking-appear.wav"/>
+ <audio id="ranking-move" src="../regen-ranking/assets/ranking-move.wav"/>
+ <audio id="music" src="../regen-ranking/assets/background.mp3"/>
+ <alignment id="alignment" src="../regen-ranking/evidence/alignment.json"/>
+
+ <!-- The Basis Producer and Locator are replaceable Components with typed outputs. -->
+ <speech-assemble id="voice" fps="30" defaultJoin="cut">
+ <segment id="scene-1" script={script.segment.scene-1} visual={avatar-1} audio={avatar-1-audio} sourceWindow="0s .. 12.033333333333333s"/>
+ <segment id="scene-2" script={script.segment.scene-2} visual={avatar-2} audio={avatar-2-audio} sourceWindow="0s .. 12.033333333333333s"/>
+ <segment id="scene-3" script={script.segment.scene-3} visual={avatar-3} audio={avatar-3-audio} sourceWindow="0s .. 12.033333333333333s"/>
</speech-assemble>
-
- <!--
- SemanticMap is the fifth SVK output role. A simple speech-locator may expose
- one atomic capability; a modular alternative can consume explicit Evidence
- Value Components while preserving this exact output contract.
- -->
- <speech-locator
- id="location"
- script={script}
- basis={voice.production}
- />
-
- <!--
- One program-bound host facet, one continuous source mapping. Present changes
- only x/y/size/mask/z; it never restarts or seeks the video.
- -->
- <media-track id="aroll" z="200">
- <item
- id="host"
- source={voice.facets.visual}
- during={script.selection.whole}
- playback="program-map"
- class="house.host-full"
- >
- <!-- Ten seconds without words: manual projections remain legal. -->
- <present
- id="prelude-top-left"
- during={script.selection.prelude}
- window="start .. start+2.2s"
- z="210"
- class="house.face-top-left"
- exit="fade 200ms"
- />
- <present
- id="prelude-top-right"
- during={script.selection.prelude}
- window="start+2s .. start+4.2s"
- z="211"
- class="house.face-top-right"
- enter="fade 200ms"
- exit="fade 200ms"
- />
- <present
- id="prelude-bottom-right"
- during={script.selection.prelude}
- window="start+4s .. start+6.2s"
- z="212"
- class="house.face-bottom-right"
- enter="fade 200ms"
- exit="fade 200ms"
- />
- <present
- id="prelude-bottom-left"
- during={script.selection.prelude}
- window="start+6s .. start+8.2s"
- z="213"
- class="house.face-bottom-left"
- enter="fade 200ms"
- exit="fade 200ms"
- />
- <present
- id="prelude-center"
- during={script.selection.prelude}
- window="start+8s .. end"
- z="214"
- class="house.face-center"
- enter="fade 200ms"
- />
-
- <present
- id="promise-split"
- during={script.selection.promise}
- z="220"
- class="house.host-lower-split"
- />
- <present
- id="reveal-pip"
- during={script.selection.reveal}
- window="start-200ms .. end"
- z="221"
- class="house.host-corner-circle"
- enter="fade 200ms"
- />
+ <speech-locator id="location" script={script} basis={voice.production} evidence={alignment}/>
+
+ <!-- One source mapping; Present only changes its projection in ProgramSpace. -->
+ <media-track id="aroll" z="20">
+ <item id="host" source={voice.facets.visual} during={script.selection.whole} class="house.host-full">
+ <present id="opening-left" during="0s .. 2.2s" z="31" class="house.face-top-left" exit="fade 200ms"/>
+ <present id="opening-right" during="2s .. 4.2s" z="32" class="house.face-top-right" enter="fade 200ms" exit="fade 200ms"/>
+ <present id="opening-low" during="4s .. 6.2s" z="33" class="house.face-bottom-right" enter="fade 200ms" exit="fade 200ms"/>
+ <present id="opening-full" during="6s .. 8s" z="34" class="house.face-center" enter="fade 200ms"/>
+ <present id="photoshop-split" during={script.selection.photoshop} z="35" class="house.host-lower-split"/>
+ <present id="regen-pip" during={script.selection.regen} window="start-200ms .. end" z="36" class="house.host-corner-circle" enter="fade 200ms"/>
</item>
</media-track>
- <!-- B-roll owns item composition, visual transition, source audio and SFX. -->
- <broll-track id="broll" z="100" class="house.broll-full">
- <item
- id="poster"
- source={poster}
- during={script.selection.promise}
- playback="hold"
- />
- <item
- id="product-demo"
- source={product-demo}
- during={script.selection.reveal}
- playback="stretch"
- sourceAudio="on"
- sourceGain="0.35"
- />
- <transition
- from="poster"
- to="product-demo"
- kind="crossfade"
- duration="200ms"
- sound={whoosh}
- soundGain="0.7"
- />
+ <!-- B-roll keeps its own source audio and transition sound in the same Track. -->
+ <broll-track id="broll" z="100">
+ <item id="poster" source={broll-1} during={script.selection.handsome-1} playback="hold" class="house.broll-full"/>
+ <item id="product-demo" source={avatar-3} during={script.selection.handsome-2} playback="stretch" sourceDuration="12.033333333333333" audio="source" volume="0.15" class="house.broll-full"/>
+ <transition from="poster" to="product-demo" kind="crossfade" duration="200ms" sound={ranking-appear} volume="0.7"/>
</broll-track>
- <!-- Ranking is another flat Track, not a child of either media Track. -->
- <ranking-tier-list
- id="ranking"
- z="300"
- at={script.moment.ranking}
- class="house.ranking"
- >
- <item
- id="svml"
- label="SVML"
- rank="1"
- icon={rank-icon-svml}
- during={script.selection.rank-svml}
- />
- <item
- id="editors"
- label="Pixel-first editors"
- rank="2"
- icon={rank-icon-editors}
- during={script.selection.reveal}
- />
- </ranking-tier-list>
-
- <!-- Manual ProgramSpan is intentionally legal. -->
+ <!-- A Moment gates one independent Track; every item still owns its Selection. -->
+ <ranking-column id="ranking" z="300" at={script.moment.ranking} class="house.ranking"
+ appearSound={ranking-appear} moveSound={ranking-move}>
+ <item id="regen" rank="1" image={ranking-icon-1} during={script.selection.regen}/>
+ <item id="chatgpt" rank="2" image={ranking-icon-2} during={script.selection.chatgpt}/>
+ <item id="remini" rank="3" image={ranking-icon-3} during={script.selection.remini}/>
+ <item id="facetune" rank="4" image={ranking-icon-4} during={script.selection.facetune}/>
+ <item id="photoshop" rank="5" image={ranking-icon-5} during={script.selection.photoshop}/>
+ </ranking-column>
+
+ <!-- Manual ProgramSpans remain a deliberate escape hatch. -->
<text-track id="overlays" z="400">
- <item id="opening-title" during="0s .. 3s" class="house.title">
- Semantic video starts with meaning.
- </item>
- <item id="ending-arrow" during="end-2s .. end" class="house.arrow">
- ↓
- </item>
+ <item id="opening-title" text="Semantic video starts with meaning." during="0s .. 3s" class="house.title"/>
+ <item id="ending-arrow" text="↓" during="end-2s .. end" class="house.arrow"/>
</text-track>
- <!-- The repeated emphasis Selection is disconnected and consumed as a set. -->
- <caption-track id="captions" z="500" script={script} class="house.captions">
- <style during={script.selection.emphasis} class="house.caption-emphasis"/>
+ <!-- The repeated emphasis Selection is disconnected and consumed as one set. -->
+ <caption-track id="captions" z="500" script={script} semantic={location.map} class="house.captions">
+ <style id="emphasis" during={script.selection.emphasis} class="house.caption-emphasis"/>
</caption-track>
<audio-track id="background-music">
- <item
- id="music"
- source={music}
- during="full"
- volume="0.28"
- bus="music"
- duckUnder="voice"
- />
+ <item id="music" source={music} during="full" volume="0.28" bus="music"/>
</audio-track>
- <!--
- Child order is deliberately not z order. Film is the only Track[] consumer.
- Grain and master mix are Film-level because they need the complete program.
- -->
- <film
- id="launch"
- basis={voice.production}
- semantic={location.map}
- class="house.vertical"
- globalEffect="grain-soft"
- masterMix="house.launch-mix"
- >
+ <!-- Child order is not paint order: absolute z across flat Tracks is authoritative. -->
+ <film id="launch" basis={voice.production} semantic={location.map} class="house.vertical">
<track ref={captions.track}/>
<track ref={background-music.track}/>
<track ref={broll.track}/>
diff --git a/examples/flat-track-launch/launch.svs b/examples/flat-track-launch/launch.svs
index f90b811c..a1099acd 100644
--- a/examples/flat-track-launch/launch.svs
+++ b/examples/flat-track-launch/launch.svs
@@ -1,7 +1,8 @@
<sheet version="1" id="house">
film.vertical {
- resolution: 1080x1920;
+ width: 1080;
+ height: 1920;
frame-rate: 30;
background: #09090B;
}
@@ -41,15 +42,6 @@
mask: circle;
}
- media-track.face-bottom-left {
- x: 0.05;
- y: 0.64;
- width: 0.30;
- height: 0.30;
- fit: cover;
- mask: circle;
- }
-
media-track.face-center {
x: 0.35;
y: 0.35;
@@ -84,21 +76,24 @@
fit: cover;
}
- ranking-tier-list.ranking {
+ ranking-column.ranking {
x: 0.06;
y: 0.14;
- width: 0.88;
- row-gap: 0.018;
- board-color: #141418;
- accent: #8B5CF6;
+ width: 0.42;
+ row-height: 0.071;
+ gap: 0.018;
+ stage-x: 0.66;
+ stage-y: 0.73;
+ stage-size: 0.33;
}
text-track.title {
x: 0.06;
y: 0.08;
width: 0.88;
- font-family: Inter;
- font-size: 64px;
+ height: 0.12;
+ font-family: Inter, Arial, sans-serif;
+ font-size: 64;
font-weight: 850;
fill: #FFFFFF;
}
@@ -107,7 +102,8 @@
x: 0.42;
y: 0.80;
width: 0.16;
- font-size: 120px;
+ height: 0.14;
+ font-size: 120;
fill: #A78BFA;
}
@@ -115,14 +111,15 @@
x: 0.08;
y: 0.78;
width: 0.84;
- font-family: Inter;
- font-size: 56px;
+ font-family: Inter, Arial, sans-serif;
+ font-size: 56;
font-weight: 800;
fill: #FFFFFF;
}
caption-track.caption-emphasis {
fill: #FDE047;
+ active-fill: #FFFFFF;
scale: 1.08;
}
diff --git a/examples/regen-ranking/reference.json b/examples/regen-ranking/reference.json
index 00b38ee8..9360a94e 100644
--- a/examples/regen-ranking/reference.json
+++ b/examples/regen-ranking/reference.json
@@ -14,9 +14,9 @@
},
"plan": {
"root": "regen-ranking",
- "values": 19,
- "instances": 8,
- "edges": 29
+ "values": 20,
+ "instances": 9,
+ "edges": 33
},
"selectionFrames": {
"handsome-1": [[791, 831]],
@@ -26,7 +26,22 @@
"instagram-post": [[977, 1009]]
},
"locatedOutput": {
- "visualFragments": 155,
+ "visualFragments": 153,
"audioFragments": 14
+ },
+ "renderComparison": {
+ "reference": "/svml-reference/regen-ranking-column-east-asian/final.mp4",
+ "durationSec": 36.1,
+ "video": {
+ "width": 1080,
+ "height": 1920,
+ "fps": 30,
+ "ssimAll": 0.9658
+ },
+ "audio": {
+ "sampleRate": 48000,
+ "channels": 2,
+ "apsnrDb": [174.207, 174.208]
+ }
}
}
diff --git a/examples/regen-ranking/regen-ranking.svml b/examples/regen-ranking/regen-ranking.svml
index 32f6fc6a..5149f76a 100644
--- a/examples/regen-ranking/regen-ranking.svml
+++ b/examples/regen-ranking/regen-ranking.svml
@@ -1,6 +1,7 @@
<svml version="1">
- <import from="../../stdlib/speech-spine.svk"/>
+ <import from="../../stdlib/speech-assemble.svk"/>
+ <import from="../../stdlib/speech-locator.svk"/>
<import from="../../stdlib/media-track.svk"/>
<import from="../../stdlib/ranking-column.svk"/>
<import from="../../stdlib/broll-track.svk"/>
@@ -61,17 +62,20 @@
<audio id="ranking-appear" src="./assets/ranking-appear.wav"/>
<audio id="ranking-move" src="./assets/ranking-move.wav"/>
<audio id="background-music" src="./assets/background.mp3"/>
+ <alignment id="alignment" src="./evidence/alignment.json"/>
- <speech-spine id="speech">
- <segment id="scene-1" script={script.segment.scene-1} visual={avatar-1} audio={avatar-1-audio}/>
- <segment id="scene-2" script={script.segment.scene-2} visual={avatar-2} audio={avatar-2-audio}/>
- <segment id="scene-3" script={script.segment.scene-3} visual={avatar-3} audio={avatar-3-audio}/>
- </speech-spine>
+ <speech-assemble id="voice" fps="30" defaultJoin="cut">
+ <segment id="scene-1" script={script.segment.scene-1} visual={avatar-1} audio={avatar-1-audio} sourceWindow="0s .. 12.033333333333333s"/>
+ <segment id="scene-2" script={script.segment.scene-2} visual={avatar-2} audio={avatar-2-audio} sourceWindow="0s .. 12.033333333333333s"/>
+ <segment id="scene-3" script={script.segment.scene-3} visual={avatar-3} audio={avatar-3-audio} sourceWindow="0s .. 12.033333333333333s"/>
+ </speech-assemble>
+
+ <speech-locator id="location" script={script} basis={voice.production} evidence={alignment}/>
<media-track id="base" z="20">
- <item id="scene-1" source={speech.segment.scene-1.visual} fit="cover"/>
- <item id="scene-2" source={speech.segment.scene-2.visual} fit="cover"/>
- <item id="scene-3" source={speech.segment.scene-3.visual} fit="cover"/>
+ <item id="scene-1" source={voice.segment.scene-1.visual} fit="cover"/>
+ <item id="scene-2" source={voice.segment.scene-2.visual} fit="cover"/>
+ <item id="scene-3" source={voice.segment.scene-3.visual} fit="cover"/>
</media-track>
<ranking-column
@@ -128,6 +132,7 @@
<caption-track
id="captions"
script={script}
+ semantic={location.map}
z="500"
x="0.37"
y="0.54"
@@ -148,7 +153,8 @@
<film
id="regen-ranking"
- speech={speech.speech}
+ basis={voice.production}
+ semantic={location.map}
width="1080"
height="1920"
fps="30"
diff --git a/src/artifacts.ts b/src/artifacts.ts
index 73dfb249..00639bc4 100644
--- a/src/artifacts.ts
+++ b/src/artifacts.ts
@@ -8,9 +8,9 @@ import type { KernelProjection } from "./runtime-contract.js";
import { sha256, stableJson } from "./util.js";
type ArtifactOutput = {
- type: "Image" | "Video" | "Audio" | "Text";
+ type: string;
uri?: string;
- value?: string;
+ value?: unknown;
contentDigest: string;
};
@@ -51,6 +51,16 @@ async function resolveOutput(
}
return { type: "Text", value: output.value, contentDigest: digest };
}
+ if (!["Image", "Video", "Audio"].includes(output.type)) {
+ if (output.uri !== undefined || output.value === undefined) {
+ fail("artifact_typed_value", `${output.type} artifact output requires value and forbids uri.`);
+ }
+ const digest = sha256(stableJson(output.value));
+ if (digest !== output.contentDigest) {
+ fail("artifact_digest", `${output.type} artifact contentDigest does not match its value.`);
+ }
+ return output.value as Record<string, unknown>;
+ }
if (typeof output.uri !== "string" || output.value !== undefined) {
fail("artifact_media", `${output.type} artifact output requires uri and forbids value.`);
}
diff --git a/src/cli.ts b/src/cli.ts
index 72347360..972b13c0 100644
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -94,7 +94,12 @@ async function run(): Promise<void> {
}
if (args.command === "lock") {
if (!file) throw new Error("usage: svml lock <file.svml> [--out svml.lock]");
- const result = await checkSource(file);
+ const result = await compileSource({
+ file,
+ ...(option(args, "artifacts")
+ ? { artifactsFile: option(args, "artifacts")! }
+ : {}),
+ });
const output = option(args, "out") ?? resolve(dirname(resolve(file)), "svml.lock");
await writeJson(output, result.lock);
process.stdout.write(`${JSON.stringify({
@@ -188,9 +193,14 @@ async function run(): Promise<void> {
: {}),
...(numeric("fps") !== undefined ? { fps: numeric("fps")! } : {}),
});
+ const payload = {
+ basis: result.basis,
+ map: result.map,
+ timeline: result.located,
+ };
const output = option(args, "out");
- if (output) await writeJson(output, result.evidence);
- else process.stdout.write(`${JSON.stringify(result.evidence, null, 2)}\n`);
+ if (output) await writeJson(output, payload);
+ else process.stdout.write(`${JSON.stringify(payload, null, 2)}\n`);
return;
}
if (args.command === "canvas") {
@@ -204,12 +214,10 @@ async function run(): Promise<void> {
}
if (args.command === "timeline") {
if (!file) {
- throw new Error("usage: svml timeline <file.svml> --evidence alignment.json [--out timeline.json]");
+ throw new Error("usage: svml timeline <file.svml> [--out timeline.json]");
}
- const evidenceFile = option(args, "evidence", true);
const compilation = await compileSource({
file,
- evidenceFile: evidenceFile!,
...(option(args, "lock") ? { lockFile: option(args, "lock")! } : {}),
...(option(args, "artifacts")
? { artifactsFile: option(args, "artifacts")! }
@@ -223,13 +231,11 @@ async function run(): Promise<void> {
}
if (args.command === "compile") {
if (!file) {
- throw new Error("usage: svml compile <file.svml> --evidence alignment.json --out index.html");
+ throw new Error("usage: svml compile <file.svml> --out index.html");
}
- const evidenceFile = option(args, "evidence", true);
const outputFile = option(args, "out", true);
const compilation = await compileSource({
file,
- evidenceFile: evidenceFile!,
outputFile: outputFile!,
...(option(args, "lock") ? { lockFile: option(args, "lock")! } : {}),
...(option(args, "artifacts")
@@ -278,14 +284,14 @@ async function run(): Promise<void> {
usage:
svml check <file.svml>
- svml lock <file.svml> [--out svml.lock]
+ svml lock <file.svml> [--out svml.lock] [--artifacts artifacts.json]
svml script <file.svml> [--out narrative.json]
svml plan <file.svml> [--out plan.json]
svml fmt <file.svml> [--check|--write|--out formatted.svml]
svml estimate <file.svml> [--out alignment.json] [--fps 30]
svml canvas <file.svml> [--out canvas.json]
- svml timeline <file.svml> --evidence alignment.json [--out timeline.json] [--lock svml.lock] [--artifacts artifacts.json]
- svml compile <file.svml> --evidence alignment.json --out index.html [--lock svml.lock] [--artifacts artifacts.json]
+ svml timeline <file.svml> [--out timeline.json] [--lock svml.lock] [--artifacts artifacts.json]
+ svml compile <file.svml> --out index.html [--lock svml.lock] [--artifacts artifacts.json]
svml render <index.html> --out video.mp4
The current runtime is deterministic-only: it has no generation provider host.
diff --git a/src/compiler.ts b/src/compiler.ts
index 2b3b5d8d..1f753ff8 100644
--- a/src/compiler.ts
+++ b/src/compiler.ts
@@ -2,6 +2,7 @@ import { copyFile, readFile, writeFile, mkdir } from "node:fs/promises";
import { basename, dirname, resolve } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { fail } from "./diagnostics.js";
+import { expandComposites } from "./composite.js";
import {
resolveArtifactFile,
verifyArtifactExecutionDigests,
@@ -13,13 +14,16 @@ import {
loadKernels,
type KernelManifest,
} from "./kernel.js";
-import { createLock, verifyLockFile, type SvmlLock } from "./lock.js";
+import { createLock, freezeLock, verifyLockFile, type SvmlLock } from "./lock.js";
import type {
AlignmentEvidence,
AttributeValue,
+ ExactSemanticMap,
+ LocatedScript,
PlanIR,
SourceDocument,
SourceElement,
+ TemporalBasisProduction,
} from "./model.js";
import { buildPlan, topologicalInstances } from "./plan.js";
import type {
@@ -29,24 +33,34 @@ import type {
} from "./runtime-contract.js";
import { projectKernelIsolated } from "./sandbox.js";
import { validateKernelProjection } from "./runtime-validation.js";
-import { locateScript } from "./script/locate.js";
+import { bindSemanticMap, locateScript } from "./script/locate.js";
import { estimateAlignment, type EstimateOptions } from "./script/estimate.js";
import { parseScript } from "./script/parse.js";
import { loadSourceClosure } from "./source/modules.js";
import { childElements, textContent } from "./source/query.js";
import { applyStyleSheets } from "./source/styles.js";
-import { sha256 } from "./util.js";
+import { sha256, stableJson } from "./util.js";
+import {
+ createProgramBasis,
+ semanticMapFromAlignment,
+ validateSemanticMap,
+ validateTemporalBasisProduction,
+} from "./temporal.js";
export type Compilation = {
document: SourceDocument;
narrative: ReturnType<typeof parseScript>;
plan: PlanIR;
- located: ReturnType<typeof locateScript>;
+ basis: TemporalBasisProduction;
+ semanticMap: ExactSemanticMap;
+ located: LocatedScript;
html: string;
target: HyperframesDocument;
outputFile?: string;
projections: Map<string, KernelProjection>;
lock: SvmlLock;
+ sourceClosureVerified: boolean;
+ temporalEvidenceVerified: boolean;
lockVerified: boolean;
artifactsVerified: boolean;
artifactManifestHash?: string;
@@ -97,6 +111,43 @@ function isHyperframesDocument(value: unknown): value is HyperframesDocument {
&& (value as HyperframesDocument).contract === "svml.hyperframes-document.v1";
}
+function validateTrackOutput(
+ instanceId: string,
+ portName: string,
+ value: unknown,
+ durationFrames: number,
+): void {
+ if (!value || typeof value !== "object") {
+ fail("track_output_shape", `Track output "${instanceId}.${portName}" is not an object.`);
+ }
+ const track = value as { visuals?: unknown; audios?: unknown; styles?: unknown };
+ if (!Array.isArray(track.visuals) || !Array.isArray(track.audios) || !Array.isArray(track.styles)) {
+ fail(
+ "track_output_shape",
+ `Track output "${instanceId}.${portName}" requires flat visuals, audios and styles arrays.`,
+ );
+ }
+ for (const contribution of [...track.visuals, ...track.audios] as Array<{
+ id?: unknown;
+ startFrame?: unknown;
+ endFrameExclusive?: unknown;
+ }>) {
+ if (
+ typeof contribution.id !== "string"
+ || !Number.isInteger(contribution.startFrame)
+ || !Number.isInteger(contribution.endFrameExclusive)
+ || (contribution.startFrame as number) < 0
+ || (contribution.endFrameExclusive as number) > durationFrames
+ || (contribution.endFrameExclusive as number) <= (contribution.startFrame as number)
+ ) {
+ fail(
+ "track_contribution_window",
+ `Track "${instanceId}" contribution "${String(contribution.id)}" has invalid Program range ${String(contribution.startFrame)}..${String(contribution.endFrameExclusive)} for duration ${durationFrames}.`,
+ );
+ }
+ }
+}
+
async function stageDocumentAssets(
document: HyperframesDocument,
outputFile: string,
@@ -136,7 +187,8 @@ export async function checkSource(file: string): Promise<{
}> {
const closure = await loadSourceClosure(file);
const kernels = await loadKernels(closure);
- const document = await applyStyleSheets(closure, kernels);
+ const styled = await applyStyleSheets(closure, kernels);
+ const document = await applyStyleSheets(expandComposites(styled, kernels), kernels);
const narrative = parseScript(
document.file,
document.scriptSource,
@@ -160,29 +212,44 @@ export async function estimateSource(
options: EstimateOptions = {},
): Promise<{
evidence: AlignmentEvidence;
- located: ReturnType<typeof locateScript>;
+ basis: ReturnType<typeof createProgramBasis>;
+ map: ReturnType<typeof semanticMapFromAlignment>;
+ located: LocatedScript;
}> {
const checked = await checkSource(file);
const evidence = estimateAlignment(checked.narrative, options);
+ const basis = createProgramBasis({
+ fps: evidence.fps,
+ durationFrames: Math.round(evidence.durationSec * evidence.fps),
+ outcomeDigest: sha256(JSON.stringify({
+ kind: "svml.estimated-program-basis.v1",
+ evidence,
+ })),
+ });
+ const map = semanticMapFromAlignment(
+ checked.narrative,
+ basis,
+ evidence,
+ { precision: "estimated" },
+ );
return {
evidence,
- located: locateScript(checked.narrative, evidence),
+ basis,
+ map,
+ located: bindSemanticMap(checked.narrative, basis, map),
};
}
export async function compileSource(args: {
file: string;
- evidenceFile: string;
outputFile?: string;
lockFile?: string;
artifactsFile?: string;
}): Promise<Compilation> {
const checked = await checkSource(args.file);
- if (args.lockFile) await verifyLockFile(resolve(args.lockFile), checked.lock);
- const evidence = JSON.parse(
- await readFile(resolve(args.evidenceFile), "utf8"),
- ) as AlignmentEvidence;
- const located = locateScript(checked.narrative, evidence);
+ const suppliedLock = args.lockFile
+ ? await verifyLockFile(resolve(args.lockFile), checked.lock)
+ : undefined;
const kernels = await loadKernels(checked.document);
const artifacts = await resolveArtifactFile(args.artifactsFile, checked.plan, kernels);
await attachExecutionDigests(checked.plan, kernels, artifacts.outputDigests);
@@ -192,35 +259,57 @@ export async function compileSource(args: {
if (!root) fail("runtime_root_missing", `Root instance "${checked.plan.root}" is missing.`);
const rootElement = elements.get(root.id);
if (!rootElement) fail("runtime_root_element", `Root element "${root.id}" is missing.`);
+ const rootManifest = kernels.get(root.kernel);
+ if (!rootManifest) fail("runtime_kernel_missing", `Kernel "${root.kernel}" is missing.`);
+ const basisPort = rootManifest.ports.find((port) =>
+ port.direction === "input" && port.type.split("|").includes("TemporalBasisProduction"));
+ const semanticPort = rootManifest.ports.find((port) =>
+ port.direction === "input" && port.type.split("|").includes("ExactSemanticMap"));
+ if (!basisPort || !semanticPort) {
+ fail("runtime_composition_contract", "Composition does not expose basis/map input ports.");
+ }
const width = Number(root.attributes.width ?? 1080);
const height = Number(root.attributes.height ?? 1920);
if (!Number.isFinite(width) || !Number.isFinite(height)) {
fail("runtime_dimensions", "Film width and height must be finite numbers.");
}
- const values = new Map<string, RuntimeValue>(checked.plan.values.map((value) => {
+ const valueEntries = await Promise.all(checked.plan.values.map(async (value) => {
const absoluteSource = value.source
? pathToFileURL(resolve(dirname(value.module), value.source)).href
: undefined;
+ const data = value.type === "AlignmentEvidence" && value.source
+ ? JSON.parse(await readFile(resolve(dirname(value.module), value.source), "utf8"))
+ : undefined;
return [
value.id,
- { ...value, ...(absoluteSource ? { absoluteSource } : {}) },
- ];
+ {
+ ...value,
+ ...(absoluteSource ? { absoluteSource } : {}),
+ ...(data !== undefined ? { data } : {}),
+ },
+ ] as const;
}));
+ const values = new Map<string, RuntimeValue>(valueEntries);
const results = new Map<string, KernelProjection>();
+ let located: LocatedScript | undefined;
const resolvePath = (path: string): unknown => {
if (path === "script") {
- return { narrative: checked.narrative, located };
+ return checked.narrative;
}
if (path.startsWith("script.selection.")) {
+ if (!located) fail("runtime_temporal_unbound", `Temporal reference "${path}" is not bound yet.`);
return located.selections[path.slice("script.selection.".length)];
}
if (path.startsWith("script.moment.")) {
+ if (!located) fail("runtime_temporal_unbound", `Temporal reference "${path}" is not bound yet.`);
return located.moments[path.slice("script.moment.".length)];
}
if (path.startsWith("script.segment.")) {
- return located.segments[path.slice("script.segment.".length)];
+ return checked.narrative.segments.find(
+ (segment) => segment.id === path.slice("script.segment.".length),
+ );
}
const declarationIds = [
...checked.plan.values.map((value) => value.id),
@@ -253,7 +342,7 @@ export async function compileSource(args: {
};
};
- for (const instance of topologicalInstances(checked.plan)) {
+ const executeInstance = async (instance: PlanIR["instances"][number]): Promise<void> => {
const manifest = kernels.get(instance.kernel);
if (!manifest) fail("runtime_kernel_missing", `Kernel "${instance.kernel}" is missing.`);
if (manifest.abiVersion !== "1") {
@@ -268,7 +357,7 @@ export async function compileSource(args: {
fail("artifact_projection_missing", `Capability "${instance.id}" has no resolved outputs.`);
}
results.set(instance.id, projection);
- continue;
+ return;
}
const sourceElement = elements.get(instance.id);
if (!sourceElement) fail("runtime_instance_element", `Element "${instance.id}" is missing.`);
@@ -285,19 +374,73 @@ export async function compileSource(args: {
const context = {
instance,
element: resolvedElement,
- narrative: checked.narrative,
- located,
- alignment: evidence,
- plan: checked.plan,
temporalContracts: kernelTemporalContracts(manifest),
- fps: evidence.fps,
+ ...(located ? {
+ program: {
+ basisDigest: located.basisDigest,
+ fps: located.fps,
+ durationFrames: located.durationFrames,
+ durationSec: located.durationSec,
+ },
+ } : {}),
+ fps: located?.fps ?? Number(root.attributes.fps ?? 30),
width,
height,
};
- results.set(
- instance.id,
- validateKernelProjection(await projectKernelIsolated(manifest, context)),
- );
+ const projection = validateKernelProjection(await projectKernelIsolated(manifest, context));
+ if (located) {
+ for (const port of manifest.ports.filter((candidate) =>
+ candidate.direction === "output" && candidate.type === "Track")) {
+ validateTrackOutput(instance.id, port.name, projection.outputs[port.name], located.durationFrames);
+ }
+ }
+ results.set(instance.id, projection);
+ };
+
+ const ordered = topologicalInstances(checked.plan);
+ const instanceIds = ordered.map((instance) => instance.id)
+ .sort((left, right) => right.length - left.length);
+ const producerOf = (value: AttributeValue | undefined, label: string): string => {
+ if (!value || typeof value !== "object" || value.kind !== "reference") {
+ fail("runtime_composition_reference", `Composition ${label} must be an instance output reference.`);
+ }
+ const owner = instanceIds.find((id) => value.path === id || value.path.startsWith(`${id}.`));
+ if (!owner) fail("runtime_composition_reference", `Composition ${label} has no producer.`);
+ return owner;
+ };
+ const preBinding = new Set<string>();
+ const addPreBinding = (id: string): void => {
+ if (preBinding.has(id)) return;
+ preBinding.add(id);
+ const instance = ordered.find((candidate) => candidate.id === id);
+ for (const dependency of instance?.dependencies ?? []) {
+ if (ordered.some((candidate) => candidate.id === dependency)) addPreBinding(dependency);
+ }
+ };
+ addPreBinding(producerOf(rootElement.attributes[basisPort.name], basisPort.name));
+ addPreBinding(producerOf(rootElement.attributes[semanticPort.name], semanticPort.name));
+ for (const instance of ordered) {
+ if (preBinding.has(instance.id)) await executeInstance(instance);
+ }
+
+ const production = resolveValue(
+ rootElement.attributes[basisPort.name],
+ ) as TemporalBasisProduction | undefined;
+ const semanticMap = resolveValue(
+ rootElement.attributes[semanticPort.name],
+ ) as ExactSemanticMap | undefined;
+ if (production?.contract !== "svml.temporal-basis-production.v1") {
+ fail("runtime_basis_missing", "Composition root must select one TemporalBasisProduction.");
+ }
+ if (semanticMap?.contract !== "svml.exact-semantic-map.v1") {
+ fail("runtime_semantic_map_missing", "Composition root must select one ExactSemanticMap.");
+ }
+ validateTemporalBasisProduction(production);
+ validateSemanticMap(checked.narrative, production.basis, semanticMap);
+ located = locateScript(checked.narrative, production.basis, semanticMap);
+
+ for (const instance of ordered) {
+ if (!preBinding.has(instance.id)) await executeInstance(instance);
}
const rootProjection = results.get(checked.plan.root);
@@ -308,6 +451,78 @@ export async function compileSource(args: {
`Root Kernel "${root.kernel}" did not output svml.hyperframes-document.v1.`,
);
}
+ const sourceDigests = new Map<string, string>();
+ for (const value of values.values()) {
+ const source = value.absoluteSource ?? value.source;
+ if (source && value.contentDigest) sourceDigests.set(source, value.contentDigest);
+ }
+ const collectArtifactSources = (value: unknown): void => {
+ if (Array.isArray(value)) {
+ value.forEach(collectArtifactSources);
+ return;
+ }
+ if (!value || typeof value !== "object") return;
+ const record = value as Record<string, unknown>;
+ if (typeof record.source === "string" && typeof record.contentDigest === "string") {
+ sourceDigests.set(record.source, record.contentDigest);
+ }
+ Object.values(record).forEach(collectArtifactSources);
+ };
+ results.forEach((projection) => collectArtifactSources(projection.outputs));
+ const canonicalTarget = {
+ ...documentOutput,
+ visuals: documentOutput.visuals.map((fragment) => ({
+ ...fragment,
+ ...(fragment.source
+ ? { source: sourceDigests.get(fragment.source) ?? fragment.source }
+ : {}),
+ })),
+ audios: documentOutput.audios.map((fragment) => ({
+ ...fragment,
+ source: sourceDigests.get(fragment.source) ?? fragment.source,
+ })),
+ };
+ const planDigest = sha256(stableJson({
+ root: checked.plan.root,
+ instances: checked.plan.instances.map((instance) => ({
+ identity: instance.identity,
+ executionDigest: instance.executionDigest,
+ })),
+ edges: checked.plan.edges,
+ expansions: checked.plan.expansions,
+ }));
+ const frozenLock = freezeLock(checked.lock, {
+ planDigest,
+ ...(artifacts.manifestHash ? { artifactManifestHash: artifacts.manifestHash } : {}),
+ basisDigest: production.basis.basisDigest,
+ productionDigest: production.productionDigest,
+ semanticIndexDigest: semanticMap.semanticIndexDigest,
+ semanticMapDigest: semanticMap.mapDigest,
+ locatorDigest: semanticMap.locatorDigest,
+ evidenceDigests: semanticMap.evidenceDigests,
+ quantizationPolicy: semanticMap.quantizationPolicy,
+ anchors: semanticMap.anchors.map((anchor) => ({
+ identity: anchor.identity,
+ frame: anchor.point.frame,
+ quality: anchor.quality,
+ })),
+ programSpace: {
+ width: documentOutput.width,
+ height: documentOutput.height,
+ fps: documentOutput.fps,
+ durationFrames: documentOutput.durationFrames,
+ },
+ targetDigest: sha256(stableJson(canonicalTarget)),
+ htmlDigest: sha256(emitHyperframesHtml(canonicalTarget)),
+ }, artifacts.verified);
+ const temporalEvidenceVerified = !!suppliedLock?.execution
+ && stableJson(suppliedLock) === stableJson(frozenLock);
+ if (suppliedLock?.execution && !temporalEvidenceVerified) {
+ fail(
+ "lock_execution_mismatch",
+ `${resolve(args.lockFile!)} does not match the compiled Basis, SemanticMap or HyperFrames target.`,
+ );
+ }
const outputFile = args.outputFile ? resolve(args.outputFile) : undefined;
const materializedDocument = outputFile
? await stageDocumentAssets(documentOutput, outputFile)
@@ -321,20 +536,24 @@ export async function compileSource(args: {
document: checked.document,
narrative: checked.narrative,
plan: checked.plan,
+ basis: production,
+ semanticMap,
located,
html,
target: materializedDocument,
...(outputFile ? { outputFile } : {}),
projections: results,
- lock: checked.lock,
- lockVerified: args.lockFile !== undefined,
+ lock: frozenLock,
+ sourceClosureVerified: args.lockFile !== undefined,
+ temporalEvidenceVerified,
+ lockVerified: temporalEvidenceVerified,
artifactsVerified: artifacts.verified,
...(artifacts.manifestHash
? { artifactManifestHash: artifacts.manifestHash }
: {}),
reproducible:
- args.lockFile !== undefined
- && checked.lock.reproducible
+ temporalEvidenceVerified
+ && frozenLock.reproducible
&& artifacts.verified,
};
}
diff --git a/src/composite.ts b/src/composite.ts
new file mode 100644
index 00000000..b03e6e5c
--- /dev/null
+++ b/src/composite.ts
@@ -0,0 +1,285 @@
+import { pathToFileURL } from "node:url";
+import { fail } from "./diagnostics.js";
+import type { KernelManifest, KernelRegistry } from "./kernel.js";
+import { validateKernelShape } from "./plan.js";
+import type {
+ AttributeValue,
+ Reference,
+ SourceDocument,
+ SourceElement,
+ SourceNode,
+} from "./model.js";
+import { childElements } from "./source/query.js";
+import { sha256, stableJson } from "./util.js";
+
+type LoopBinding = {
+ name: string;
+ value: SourceElement;
+};
+
+function reference(value: AttributeValue | undefined): Reference | undefined {
+ return value && typeof value === "object" && value.kind === "reference"
+ ? value
+ : undefined;
+}
+
+function canonicalNode(node: SourceNode): unknown {
+ if (node.kind === "text") return { kind: "text", value: node.value };
+ return {
+ kind: "element",
+ name: node.name,
+ attributes: node.attributes,
+ children: node.children.map(canonicalNode),
+ };
+}
+
+function localReference(path: string, outerId: string, localIds: Set<string>): Reference | undefined {
+ const owner = [...localIds]
+ .sort((left, right) => right.length - left.length)
+ .find((candidate) => path === candidate || path.startsWith(`${candidate}.`));
+ if (!owner) return undefined;
+ return {
+ kind: "reference",
+ path: `${outerId}::${owner}${path.slice(owner.length)}`,
+ };
+}
+
+function evaluate(
+ value: AttributeValue,
+ args: {
+ outer: SourceElement;
+ outerId: string;
+ localIds: Set<string>;
+ loop?: LoopBinding;
+ },
+): AttributeValue | undefined {
+ const ref = reference(value);
+ if (!ref) return value;
+ const computedScript = /^ports\.([A-Za-z_][A-Za-z0-9_.:-]*)\.segment\[([A-Za-z_][A-Za-z0-9_.:-]*)\.id\]$/u
+ .exec(ref.path);
+ if (computedScript) {
+ if (!args.loop || args.loop.name !== computedScript[2]) {
+ fail("composite_expression", `Composite expression "${ref.path}" has no matching loop.`);
+ }
+ const script = reference(args.outer.attributes[computedScript[1]!]);
+ const id = args.loop.value.attributes.id;
+ if (!script || typeof id !== "string") {
+ fail("composite_expression", `Composite expression "${ref.path}" requires Script and child id.`);
+ }
+ return { kind: "reference", path: `${script.path}.segment.${id}` };
+ }
+ const [head, ...tail] = ref.path.split(".");
+ if (head === "params" || head === "ports") {
+ const name = tail.join(".");
+ return args.outer.attributes[name];
+ }
+ if (args.loop && head === args.loop.name) {
+ return args.loop.value.attributes[tail.join(".")];
+ }
+ return localReference(ref.path, args.outerId, args.localIds) ?? ref;
+}
+
+function instantiateNode(
+ template: SourceNode,
+ args: {
+ outer: SourceElement;
+ outerId: string;
+ localIds: Set<string>;
+ aliases: ReadonlyMap<string, string>;
+ expansionDigest: string;
+ definitionFile: string;
+ componentRoot: boolean;
+ loop?: LoopBinding;
+ },
+): SourceNode[] {
+ if (template.kind === "text") return template.value.trim() ? [{ ...template }] : [];
+ if (template.name === "for") {
+ const each = reference(template.attributes.each)?.path;
+ const alias = template.attributes.as;
+ const match = /^children\.([A-Za-z_][A-Za-z0-9_.:-]*)$/u.exec(each ?? "");
+ if (!match || typeof alias !== "string") {
+ fail("composite_for", "Composite <for> requires each={children.name} and a quoted as name.");
+ }
+ return childElements(args.outer, match[1]).flatMap((value) =>
+ template.children.flatMap((child) => instantiateNode(child, {
+ ...args,
+ componentRoot: false,
+ loop: { name: alias, value },
+ })));
+ }
+ const name = args.componentRoot
+ ? args.aliases.get(template.name) ?? template.name
+ : template.name;
+ const attributes = Object.fromEntries(Object.entries(template.attributes).flatMap(([key, value]) => {
+ const evaluated = evaluate(value, args);
+ if (evaluated === undefined) return [];
+ if (key === "id" && args.componentRoot) {
+ if (typeof evaluated !== "string") fail("composite_id", "Composite internal id must be literal.");
+ return [[key, `${args.outerId}::${evaluated}`]];
+ }
+ return [[key, evaluated]];
+ }));
+ return [{
+ ...template,
+ name,
+ attributes,
+ children: template.children.flatMap((child) => instantiateNode(child, {
+ ...args,
+ componentRoot: false,
+ })),
+ origin: {
+ file: args.outer.origin?.file ?? "",
+ ...(typeof attributes.id === "string" ? { localId: attributes.id } : {}),
+ expansionDigest: args.expansionDigest,
+ callSite: {
+ file: args.outer.origin?.file ?? "",
+ start: args.outer.start,
+ end: args.outer.end,
+ },
+ definitionSite: {
+ file: args.definitionFile,
+ start: template.start,
+ end: template.end,
+ },
+ },
+ }];
+}
+
+function rewriteReference(value: AttributeValue, aliases: ReadonlyMap<string, string>): AttributeValue {
+ const ref = reference(value);
+ if (!ref) return value;
+ let path = ref.path;
+ for (let pass = 0; pass < 32; pass += 1) {
+ const owner = [...aliases.keys()]
+ .sort((left, right) => right.length - left.length)
+ .find((candidate) => path === candidate || path.startsWith(`${candidate}.`));
+ if (!owner) break;
+ const next = `${aliases.get(owner)!}${path.slice(owner.length)}`;
+ if (next === path) break;
+ path = next;
+ }
+ return { kind: "reference", path };
+}
+
+function rewriteNode(node: SourceNode, aliases: ReadonlyMap<string, string>): SourceNode {
+ if (node.kind === "text") return node;
+ return {
+ ...node,
+ attributes: Object.fromEntries(Object.entries(node.attributes)
+ .map(([name, value]) => [name, rewriteReference(value, aliases)])),
+ children: node.children.map((child) => rewriteNode(child, aliases)),
+ };
+}
+
+function expandOne(
+ element: SourceElement,
+ manifest: KernelManifest,
+): {
+ nodes: SourceElement[];
+ aliases: Record<string, string>;
+ expansion: NonNullable<SourceDocument["expansions"]>[number];
+} {
+ const outerId = element.attributes.id;
+ if (typeof outerId !== "string") fail("composite_id", `<${manifest.name}> requires a stable id.`);
+ validateKernelShape(element, manifest);
+ const compose = manifest.compose;
+ if (!compose) fail("composite_shape", `Composite "${manifest.name}" has no compose body.`);
+ const templates = childElements(compose).filter((child) => child.name !== "export");
+ const localIds = new Set(templates.flatMap((template) => {
+ const id = template.attributes.id;
+ return typeof id === "string" ? [id] : [];
+ }));
+ const expansionDigest = sha256(stableJson({
+ contract: "svml.composite-expansion.v1",
+ manifestHash: manifest.sourceHash,
+ instanceId: outerId,
+ attributes: element.attributes,
+ children: element.children.map(canonicalNode),
+ }));
+ const importAliases = new Map((manifest.compositeImports ?? []).map((item) => {
+ if (!item.kernelName) fail("composite_import", `Composite import "${item.alias}" is unresolved.`);
+ return [item.alias, item.kernelName];
+ }));
+ const nodes = templates.flatMap((template) => instantiateNode(template, {
+ outer: element,
+ outerId,
+ localIds,
+ aliases: importAliases,
+ expansionDigest,
+ definitionFile: manifest.sourcePath,
+ componentRoot: true,
+ })) as SourceElement[];
+ const exports: Record<string, string> = {};
+ for (const item of childElements(compose, "export")) {
+ const port = item.attributes.port;
+ const value = item.attributes.value;
+ if (typeof port !== "string" || !reference(value)) {
+ fail("composite_export", "Composite <export> requires port and reference value attributes.");
+ }
+ const evaluated = evaluate(value!, { outer: element, outerId, localIds });
+ const target = reference(evaluated)?.path;
+ if (!target || exports[port]) fail("composite_export", `Invalid or duplicate export "${port}".`);
+ exports[port] = target;
+ }
+ for (const port of manifest.ports.filter((candidate) => candidate.direction === "output")) {
+ if (!exports[port.name]) {
+ fail("composite_export_missing", `Composite "${manifest.name}" does not export "${port.name}".`);
+ }
+ }
+ return {
+ nodes,
+ aliases: Object.fromEntries(Object.entries(exports).map(([port, target]) => [
+ `${outerId}.${port}`,
+ target,
+ ])),
+ expansion: {
+ instanceId: outerId,
+ component: manifest.name,
+ manifestHash: manifest.sourceHash,
+ expansionDigest,
+ internalIds: nodes.map((node) => String(node.attributes.id)),
+ exports,
+ },
+ };
+}
+
+export function expandComposites(
+ document: SourceDocument,
+ kernels: KernelRegistry,
+): SourceDocument {
+ let children = document.root.children;
+ const aliases = new Map<string, string>();
+ const expansions: NonNullable<SourceDocument["expansions"]> = [];
+ for (let pass = 0; pass < 32; pass += 1) {
+ let changed = false;
+ const next = children.flatMap((node): SourceNode[] => {
+ if (node.kind !== "element") return [node];
+ const manifest = kernels.get(node.name);
+ if (manifest?.profile !== "composite-v1") return [node];
+ changed = true;
+ const expanded = expandOne(node, manifest);
+ Object.entries(expanded.aliases).forEach(([from, to]) => aliases.set(from, to));
+ expansions.push(expanded.expansion);
+ return expanded.nodes;
+ });
+ children = next.map((node) => rewriteNode(node, aliases));
+ if (!changed) {
+ const compositeModules = [...kernels.values()]
+ .filter((manifest) => manifest.profile === "composite-v1")
+ .map((manifest) => ({
+ kind: "svk" as const,
+ uri: pathToFileURL(manifest.sourcePath).href,
+ contentHash: manifest.sourceHash,
+ dependencies: (manifest.compositeImports ?? []).map((item) =>
+ pathToFileURL(item.sourcePath).href),
+ }));
+ return {
+ ...document,
+ root: { ...document.root, children },
+ modules: [...(document.modules ?? []), ...compositeModules],
+ expansions,
+ };
+ }
+ }
+ fail("composite_expansion_limit", "Composite expansion exceeded 32 finite passes.");
+}
diff --git a/src/digests.ts b/src/digests.ts
index dd31e506..23d5a94a 100644
--- a/src/digests.ts
+++ b/src/digests.ts
@@ -108,6 +108,7 @@ export async function attachExecutionDigests(
abi: manifest.abiVersion,
manifest: manifest.sourceHash,
implementation: manifest.implementationHash,
+ expansion: instance.expansionDigest,
attributes: Object.fromEntries(Object.entries(instance.attributes)
.filter(([name]) => !["id", "class"].includes(name))
.map(([name, value]) => [name, canonicalValue(value)])),
diff --git a/src/hyperframes.ts b/src/hyperframes.ts
index fdd2d8b7..bf1c15b5 100644
--- a/src/hyperframes.ts
+++ b/src/hyperframes.ts
@@ -96,8 +96,6 @@ export function emitHyperframesHtml(args: {
}): string {
const orderedVisuals = [...args.visuals].sort((left, right) =>
left.z - right.z
- || (left.layer ?? 0) - (right.layer ?? 0)
- || left.startFrame - right.startFrame
|| left.id.localeCompare(right.id));
const visuals = orderedVisuals.map((fragment, index) =>
visualHtml(fragment, args.fps, index));
diff --git a/src/kernel.ts b/src/kernel.ts
index 7f34a0ec..f7bf3022 100644
--- a/src/kernel.ts
+++ b/src/kernel.ts
@@ -18,7 +18,7 @@ export type KernelPort = {
export type KernelParameter = {
name: string;
styleName: string;
- type: "string" | "number" | "boolean";
+ type: string;
defaultValue?: AttributeValue;
};
@@ -46,12 +46,14 @@ export type KernelManifest = {
sourceHash: string;
implementationPath?: string;
implementationHash: string;
- profile: "isolated-projector-v1" | "capability-v1";
+ profile: "isolated-projector-v1" | "capability-v1" | "composite-v1";
capability?: string;
permissions: string[];
ports: KernelPort[];
parameters: KernelParameter[];
children: KernelChildSchema[];
+ compositeImports?: Array<{ alias: string; sourcePath: string; kernelName?: string }>;
+ compose?: SourceElement;
};
export type KernelRegistry = Map<string, KernelManifest>;
@@ -124,7 +126,14 @@ function booleanAttribute(
function parseParameter(sourcePath: string, element: SourceElement): KernelParameter {
const name = stringAttribute(element, "name");
const type = stringAttribute(element, "type");
- if (!name || !type || !["string", "number", "boolean"].includes(type)) {
+ if (
+ !name
+ || !type
+ || !(
+ ["string", "number", "boolean", "duration"].includes(type)
+ || /^enum\([A-Za-z_][A-Za-z0-9_.:-]*(?:,[A-Za-z_][A-Za-z0-9_.:-]*)+\)$/u.test(type)
+ )
+ ) {
fail("kernel_manifest_parameter", `${sourcePath} contains an invalid <param>.`);
}
let defaultValue = element.attributes.default;
@@ -141,6 +150,22 @@ function parseParameter(sourcePath: string, element: SourceElement): KernelParam
`${sourcePath} parameter ${name} has a non-boolean default.`,
);
}
+ if (type === "duration" && (
+ typeof defaultValue !== "string"
+ || !/^[+-]?\d+(?:\.\d+)?(?:ms|s)$/u.test(defaultValue)
+ )) {
+ fail(
+ "kernel_manifest_parameter_default",
+ `${sourcePath} parameter ${name} has an invalid duration default.`,
+ );
+ }
+ const enumValues = /^enum\((.+)\)$/u.exec(type)?.[1]?.split(",");
+ if (enumValues && (typeof defaultValue !== "string" || !enumValues.includes(defaultValue))) {
+ fail(
+ "kernel_manifest_parameter_default",
+ `${sourcePath} parameter ${name} has an invalid enum default.`,
+ );
+ }
if (type === "string" && typeof defaultValue !== "string") {
defaultValue = String(defaultValue);
}
@@ -212,7 +237,7 @@ function parseManifest(sourcePath: string, source: string): KernelManifest {
if (!name || !/^[A-Za-z_][A-Za-z0-9_.:-]*$/u.test(name)) {
fail("kernel_manifest_name", `${sourcePath} has an invalid or missing Kernel name.`);
}
- if (profile !== "isolated-projector-v1" && profile !== "capability-v1") {
+ if (!["isolated-projector-v1", "capability-v1", "composite-v1"].includes(profile ?? "")) {
fail("kernel_manifest_profile", `${sourcePath} has unsupported profile "${profile ?? ""}".`);
}
if (profile === "isolated-projector-v1" && !implementation) {
@@ -262,7 +287,28 @@ function parseManifest(sourcePath: string, source: string): KernelManifest {
if (new Set(children.map((child) => child.name)).size !== children.length) {
fail("kernel_manifest_child_duplicate", `${sourcePath} repeats a root child schema.`);
}
- const known = new Set(["port", "param", "child"]);
+ const compositeImports = childElements(root, "import").map((element) => ({
+ alias: stringAttribute(element, "as") ?? "",
+ sourcePath: resolve(dirname(sourcePath), stringAttribute(element, "from") ?? ""),
+ }));
+ const compose = childElements(root, "compose")[0];
+ if (profile === "composite-v1") {
+ if (implementation || capability || !compose || !compositeImports.length) {
+ fail(
+ "kernel_composite_shape",
+ `${sourcePath} composite-v1 requires imports and one <compose>, without implementation/capability.`,
+ );
+ }
+ if (compositeImports.some((item) => !item.alias)) {
+ fail("kernel_composite_import", `${sourcePath} Composite imports require explicit aliases.`);
+ }
+ }
+ const known = new Set([
+ "port",
+ "param",
+ "child",
+ ...(profile === "composite-v1" ? ["import", "compose"] : []),
+ ]);
const unknown = childElements(root).find((element) => !known.has(element.name));
if (unknown) {
fail("kernel_manifest_element", `${sourcePath} contains unknown <${unknown.name}>.`);
@@ -275,7 +321,7 @@ function parseManifest(sourcePath: string, source: string): KernelManifest {
sourceHash: sha256(source),
...(implementation ? { implementationPath: resolve(dirname(sourcePath), implementation) } : {}),
implementationHash: "",
- profile,
+ profile: profile as KernelManifest["profile"],
...(capability ? { capability } : {}),
permissions: (stringAttribute(root, "permissions", "") ?? "")
.split(/[\s,]+/u)
@@ -283,27 +329,70 @@ function parseManifest(sourcePath: string, source: string): KernelManifest {
ports,
parameters,
children,
+ ...(profile === "composite-v1" ? { compositeImports, compose } : {}),
};
}
export async function loadKernels(document: SourceDocument): Promise<KernelRegistry> {
const registry: KernelRegistry = new Map();
const imports = childElements(document.root, "import");
- for (const element of imports) {
- const from = attributeString(element, "from");
- if (!from.endsWith(".svk")) continue;
- const sourcePath = resolve(dirname(document.file), from);
+ const loading = new Set<string>();
+ const definitions = new Map<string, KernelManifest>();
+ const load = async (sourcePath: string): Promise<KernelManifest> => {
+ sourcePath = resolve(sourcePath);
+ if (loading.has(sourcePath)) {
+ fail("kernel_import_cycle", `Composite Kernel import cycle at ${sourcePath}.`);
+ }
+ const byPath = definitions.get(sourcePath);
+ if (byPath) return byPath;
+ loading.add(sourcePath);
const manifest = parseManifest(sourcePath, await readFile(sourcePath, "utf8"));
- manifest.implementationHash = manifest.implementationPath
- ? sha256(await readFile(manifest.implementationPath))
- : sha256(`capability\0${manifest.capability ?? manifest.name}`);
- if (registry.has(manifest.name)) {
+ if (manifest.profile === "composite-v1") {
+ for (const imported of manifest.compositeImports ?? []) {
+ const nested = await load(imported.sourcePath);
+ imported.kernelName = nested.name;
+ }
+ manifest.implementationHash = sha256(`composite\0${manifest.sourceHash}`);
+ } else {
+ manifest.implementationHash = manifest.implementationPath
+ ? sha256(await readFile(manifest.implementationPath))
+ : sha256(`capability\0${manifest.capability ?? manifest.name}`);
+ }
+ definitions.set(sourcePath, manifest);
+ loading.delete(sourcePath);
+ return manifest;
+ };
+ const bind = (binding: string, manifest: KernelManifest): void => {
+ if (!/^[A-Za-z_][A-Za-z0-9_.:-]*$/u.test(binding)) {
+ fail("kernel_import_alias", `Kernel import alias "${binding}" is invalid.`);
+ }
+ const existing = registry.get(binding);
+ if (existing && existing.sourcePath !== manifest.sourcePath) {
fail(
"kernel_duplicate_binding",
- `Kernel "${manifest.name}" is imported more than once; use explicit aliases once namespace syntax is enabled.`,
+ `Kernel binding "${binding}" is defined by both ${existing.sourcePath} and ${manifest.sourcePath}.`,
);
}
- registry.set(manifest.name, manifest);
+ registry.set(binding, binding === manifest.name ? manifest : { ...manifest, name: binding });
+ };
+ const bindCompositeDependencies = (manifest: KernelManifest): void => {
+ for (const imported of manifest.compositeImports ?? []) {
+ const nested = definitions.get(resolve(imported.sourcePath));
+ if (!nested) fail("kernel_import_resolution", `Kernel import ${imported.sourcePath} is unresolved.`);
+ bind(nested.name, nested);
+ bindCompositeDependencies(nested);
+ }
+ };
+ for (const element of imports) {
+ const from = attributeString(element, "from");
+ if (!from.endsWith(".svk")) continue;
+ const manifest = await load(resolve(dirname(document.file), from));
+ bindCompositeDependencies(manifest);
+ const alias = element.attributes.as;
+ if (alias !== undefined && typeof alias !== "string") {
+ fail("kernel_import_alias", `${document.file} Kernel import alias must be a string.`);
+ }
+ bind(alias ?? manifest.name, manifest);
}
return registry;
}
diff --git a/src/lock.ts b/src/lock.ts
index d2d04450..2e9b870b 100644
--- a/src/lock.ts
+++ b/src/lock.ts
@@ -23,6 +23,31 @@ export type SvmlLock = {
};
reproducible: boolean;
modules: NonNullable<SourceDocument["modules"]>;
+ expansions: PlanIR["expansions"];
+ execution?: {
+ planDigest: string;
+ artifactManifestHash?: string;
+ basisDigest: string;
+ productionDigest: string;
+ semanticIndexDigest: string;
+ semanticMapDigest: string;
+ locatorDigest: string;
+ evidenceDigests: string[];
+ quantizationPolicy: string;
+ anchors: Array<{
+ identity: string;
+ frame: number;
+ quality: string;
+ }>;
+ programSpace: {
+ width: number;
+ height: number;
+ fps: number;
+ durationFrames: number;
+ };
+ targetDigest: string;
+ htmlDigest: string;
+ };
materials: Array<{
identity: string;
type: string;
@@ -89,12 +114,13 @@ export function createLock(
name: "hyperframes",
version: "0.7.84",
},
- reproducible: materials.every((material) => material.status === "content"),
+ reproducible: false,
modules: (document.modules ?? []).map((module) => ({
...module,
uri: lockUri(module.uri),
dependencies: module.dependencies.map(lockUri),
})).sort((left, right) => left.uri.localeCompare(right.uri)),
+ expansions: plan.expansions,
materials,
kernels: manifests.map((manifest) => ({
name: manifest.name,
@@ -112,10 +138,24 @@ export function createLock(
};
}
+export function freezeLock(
+ sourceLock: SvmlLock,
+ execution: NonNullable<SvmlLock["execution"]>,
+ artifactsVerified: boolean,
+): SvmlLock {
+ return {
+ ...sourceLock,
+ reproducible:
+ artifactsVerified
+ && sourceLock.materials.every((material) => material.status === "content"),
+ execution,
+ };
+}
+
export async function verifyLockFile(
file: string,
expected: SvmlLock,
-): Promise<void> {
+): Promise<SvmlLock> {
let actual: unknown;
try {
actual = JSON.parse(await readFile(file, "utf8"));
@@ -128,10 +168,17 @@ export async function verifyLockFile(
if (!actual || typeof actual !== "object" || (actual as { contract?: string }).contract !== "svml.lock.v1") {
fail("lock_contract", `${file} is not an svml.lock.v1 document.`);
}
- if (stableJson(actual) !== stableJson(expected)) {
+ const parsed = actual as SvmlLock;
+ const sourceOnly: SvmlLock = {
+ ...parsed,
+ reproducible: false,
+ };
+ delete sourceOnly.execution;
+ if (stableJson(sourceOnly) !== stableJson(expected)) {
fail(
"lock_mismatch",
`${file} does not match the resolved source closure; regenerate it with "svml lock".`,
);
}
+ return parsed;
}
diff --git a/src/model.ts b/src/model.ts
index 1e783e22..27946dee 100644
--- a/src/model.ts
+++ b/src/model.ts
@@ -23,6 +23,9 @@ export type SourceElement = {
origin?: {
file: string;
localId?: string;
+ expansionDigest?: string;
+ callSite?: { file: string; start: number; end: number };
+ definitionSite?: { file: string; start: number; end: number };
};
parameterSources?: Record<string, ParameterSource[]>;
};
@@ -37,7 +40,7 @@ export type SourceText = {
export type SourceNode = SourceElement | SourceText;
export type SourceModuleRecord = {
- kind: "svml" | "svc" | "svs";
+ kind: "svml" | "svc" | "svs" | "svk";
uri: string;
contentHash: string;
dependencies: string[];
@@ -50,6 +53,14 @@ export type SourceDocument = {
scriptSource: string;
scriptOffset: number;
modules?: SourceModuleRecord[];
+ expansions?: Array<{
+ instanceId: string;
+ component: string;
+ manifestHash: string;
+ expansionDigest: string;
+ internalIds: string[];
+ exports: Record<string, string>;
+ }>;
};
export type ScriptAtom =
@@ -60,6 +71,9 @@ export type ScriptToken = {
id: string;
index: number;
segmentId: string;
+ segmentTokenIndex: number;
+ startAnchorId: string;
+ endAnchorId: string;
text: string;
normalized: string;
sourceStart: number;
@@ -69,7 +83,7 @@ export type ScriptToken = {
export type MarkerBoundary = {
tokenIndex: number;
segmentId?: string;
- structuralCut?: number;
+ structuralPosition: number;
};
export type SelectionOccurrence = {
@@ -98,6 +112,8 @@ export type MomentOccurrence = {
export type ScriptSegment = {
id: string;
index: number;
+ startAnchorId: string;
+ endAnchorId: string;
atoms: ScriptAtom[];
tokenStart: number;
tokenEnd: number;
@@ -105,6 +121,34 @@ export type ScriptSegment = {
sourceEnd: number;
};
+export type SemanticAnchorKind =
+ | "segment-start"
+ | "token-start"
+ | "token-end"
+ | "segment-end";
+
+export type SemanticAnchorIdentity = {
+ id: string;
+ kind: SemanticAnchorKind;
+ segmentId: string;
+ tokenId?: string;
+ segmentTokenIndex?: number;
+};
+
+export type SemanticIndex = {
+ contract: "svml.semantic-index.v1";
+ anchors: SemanticAnchorIdentity[];
+ tokens: Array<{
+ id: string;
+ segmentId: string;
+ segmentTokenIndex: number;
+ normalized: string;
+ startAnchorId: string;
+ endAnchorId: string;
+ }>;
+ digest: string;
+};
+
export type CaptionAtom = {
id: string;
display: string;
@@ -121,6 +165,7 @@ export type NarrativeIR = {
selections: Record<string, SelectionOccurrence[]>;
moments: Record<string, MomentOccurrence[]>;
captionAtoms: CaptionAtom[];
+ semanticIndex: SemanticIndex;
projections: {
dialogue: string;
speech: string;
@@ -128,6 +173,93 @@ export type NarrativeIR = {
};
};
+export type Rational = {
+ numerator: number;
+ denominator: number;
+};
+
+export type ProgramBasis = {
+ contract: "svml.program-basis.v1";
+ frameRate: Rational;
+ originFrame: 0;
+ durationFrames: number;
+ basisDigest: string;
+};
+
+export type ProgramPoint = {
+ basisDigest: string;
+ frame: number;
+};
+
+export type SemanticPointQuality = "estimated" | "derived" | "measured";
+
+export type SemanticAnchorPoint = {
+ identity: string;
+ point: ProgramPoint;
+ quality: SemanticPointQuality;
+};
+
+export type SemanticMapBase = {
+ semanticIndexDigest: string;
+ basisDigest: string;
+ anchors: SemanticAnchorPoint[];
+ evidenceDigests: string[];
+ locatorDigest: string;
+ quantizationPolicy: "nearest-frame";
+ captionCues?: AlignmentCaptionCue[];
+ mapDigest: string;
+};
+
+export type EstimatedSemanticMap = SemanticMapBase & {
+ contract: "svml.estimated-semantic-map.v1";
+};
+
+export type ExactSemanticMap = SemanticMapBase & {
+ contract: "svml.exact-semantic-map.v1";
+};
+
+export type SemanticMap = EstimatedSemanticMap | ExactSemanticMap;
+
+export type SourceToProgramMap = {
+ id: string;
+ sourceId: string;
+ sourceDigest: string;
+ sourceStartFrame: number;
+ sourceEndFrameExclusive: number;
+ programStartFrame: number;
+ programEndFrameExclusive: number;
+};
+
+export type ProgramAudioContribution = {
+ id: string;
+ sourceId: string;
+ sourceDigest: string;
+ source: string;
+ sourceMapId: string;
+ sourceStartFrame: number;
+ sourceEndFrameExclusive: number;
+ programStartFrame: number;
+ programEndFrameExclusive: number;
+ gain: number;
+ fadeInFrames?: number;
+ fadeOutFrames?: number;
+};
+
+export type TemporalBasisProduction = {
+ contract: "svml.temporal-basis-production.v1";
+ basis: ProgramBasis;
+ alignmentSubjects: Array<{
+ id: string;
+ sourceId: string;
+ sourceDigest: string;
+ sourceMapId: string;
+ }>;
+ sourceMaps: SourceToProgramMap[];
+ audio: ProgramAudioContribution[];
+ facets: Record<string, unknown>;
+ productionDigest: string;
+};
+
export type AlignmentWord = {
text: string;
startSec: number;
@@ -183,6 +315,10 @@ export type LocatedCaptionCue = ProgramRange & {
export type LocatedCaptionAtom = CaptionAtom & ProgramRange;
export type LocatedScript = {
+ contract: "svml.temporal-binding.v1";
+ precision: "estimated" | "exact";
+ basisDigest: string;
+ semanticMapDigest: string;
durationFrames: number;
durationSec: number;
fps: number;
@@ -196,7 +332,7 @@ export type LocatedScript = {
export type PlanValue = {
id: string;
- type: "Image" | "Video" | "Audio" | "Text";
+ type: "Image" | "Video" | "Audio" | "Text" | "AlignmentEvidence";
source?: string;
value?: string;
identity: string;
@@ -220,6 +356,11 @@ export type PlanInstance = {
children: SourceNode[];
dependencies: string[];
executionDigest?: string;
+ expansionDigest?: string;
+ sourceMap?: {
+ callSite: { file: string; start: number; end: number };
+ definitionSite: { file: string; start: number; end: number };
+ };
};
export type PlanIR = {
@@ -232,7 +373,7 @@ export type PlanIR = {
abiVersion: string;
manifestHash: string;
implementationHash: string;
- profile: "isolated-projector-v1" | "capability-v1";
+ profile: "isolated-projector-v1" | "capability-v1" | "composite-v1";
capability?: string;
ports: Array<{
name: string;
@@ -244,7 +385,7 @@ export type PlanIR = {
parameters: Array<{
name: string;
styleName: string;
- type: "string" | "number" | "boolean";
+ type: string;
defaultValue?: AttributeValue;
}>;
children: Array<{
@@ -262,6 +403,7 @@ export type PlanIR = {
}>;
}>;
root: string;
+ expansions: NonNullable<SourceDocument["expansions"]>;
edges: Array<{
from: string;
to: string;
diff --git a/src/plan.ts b/src/plan.ts
index 88819f7e..fb25fdad 100644
--- a/src/plan.ts
+++ b/src/plan.ts
@@ -24,11 +24,12 @@ const VALUE_TYPES: Record<string, PlanValue["type"]> = {
video: "Video",
audio: "Audio",
text: "Text",
+ alignment: "AlignmentEvidence",
};
function idOf(element: SourceElement): string {
const id = element.attributes.id;
- if (typeof id !== "string" || !/^[A-Za-z_][A-Za-z0-9_.-]*$/u.test(id)) {
+ if (typeof id !== "string" || !/^[A-Za-z_][A-Za-z0-9_.:-]*$/u.test(id)) {
fail("plan_declaration_id", `<${element.name}> requires a stable id.`);
}
return id;
@@ -75,9 +76,17 @@ function scalarAccepts(type: string, value: AttributeValue): boolean {
if (type === "number") return typeof value === "number" && Number.isFinite(value);
if (type === "boolean") return typeof value === "boolean";
if (type === "id") {
- return typeof value === "string" && /^[A-Za-z_][A-Za-z0-9_.-]*$/u.test(value);
+ return typeof value === "string" && /^[A-Za-z_][A-Za-z0-9_.:-]*$/u.test(value);
}
if (type === "ProgramSpan" || type === "ProgramPoint") return typeof value === "string";
+ if (type === "duration") {
+ return typeof value === "string" && /^[+-]?\d+(?:\.\d+)?(?:ms|s)$/u.test(value);
+ }
+ if (type === "SourceRange") {
+ return typeof value === "string" && /^\s*\d+(?:\.\d+)?(?:ms|s)\s*\.\.\s*\d+(?:\.\d+)?(?:ms|s)\s*$/u.test(value);
+ }
+ const enumValues = /^enum\((.+)\)$/u.exec(type)?.[1]?.split(",");
+ if (enumValues) return typeof value === "string" && enumValues.includes(value);
return false;
}
@@ -144,7 +153,7 @@ function validateChild(
}
}
-function validateKernelShape(element: SourceElement, manifest: KernelManifest): void {
+export function validateKernelShape(element: SourceElement, manifest: KernelManifest): void {
const inputPorts = manifest.ports.filter((port) => port.direction === "input");
const allowed = new Set([
"id",
@@ -338,6 +347,17 @@ export function buildPlan(
effectiveParameters,
children: element.children,
dependencies: [...new Set(dependencies)],
+ ...(element.origin?.expansionDigest
+ ? { expansionDigest: element.origin.expansionDigest }
+ : {}),
+ ...(element.origin?.callSite && element.origin.definitionSite
+ ? {
+ sourceMap: {
+ callSite: element.origin.callSite,
+ definitionSite: element.origin.definitionSite,
+ },
+ }
+ : {}),
});
}
@@ -482,6 +502,39 @@ export function buildPlan(
};
visit(root);
const reachableInstances = instances.filter((instance) => reachable.has(instance.id));
+ const reachableEdges = edges.filter((edge) => reachable.has(edge.from) && reachable.has(edge.to));
+ const rootManifest = kernels.get(rootCandidates[0]!.kernel)!;
+ const basisInputs = rootManifest.ports.filter((port) =>
+ port.direction === "input" && port.type.split("|").includes("TemporalBasisProduction"));
+ const semanticInputs = rootManifest.ports.filter((port) =>
+ port.direction === "input" && port.type.split("|").includes("ExactSemanticMap"));
+ if (
+ basisInputs.length !== 1
+ || basisInputs[0]?.cardinality !== "one"
+ || semanticInputs.length !== 1
+ || semanticInputs[0]?.cardinality !== "one"
+ ) {
+ fail(
+ "composition_temporal_contract",
+ `Composition Kernel "${rootManifest.name}" must select exactly one TemporalBasisProduction and one ExactSemanticMap.`,
+ );
+ }
+ const trackEdges = reachableEdges.filter((edge) => edge.type === "Track");
+ const illegalTrackConsumer = trackEdges.find((edge) => edge.to !== root);
+ if (illegalTrackConsumer) {
+ fail(
+ "track_terminal_contract",
+ `Track "${illegalTrackConsumer.from}" is consumed by non-Composition instance "${illegalTrackConsumer.to}".`,
+ );
+ }
+ const duplicateTrack = trackEdges.find((edge, index) =>
+ trackEdges.findIndex((candidate) => candidate.from === edge.from && candidate.to === edge.to) !== index);
+ if (duplicateTrack) {
+ fail(
+ "duplicate_track_in_composition",
+ `Composition "${root}" references Track "${duplicateTrack.from}" more than once.`,
+ );
+ }
const usedKernels = new Set(reachableInstances.map((instance) => instance.kernel));
return {
contract: "svml.plan.v1",
@@ -503,7 +556,8 @@ export function buildPlan(
}))
.sort((left, right) => left.name.localeCompare(right.name)),
root,
- edges: edges.filter((edge) => reachable.has(edge.from) && reachable.has(edge.to)),
+ expansions: document.expansions ?? [],
+ edges: reachableEdges,
};
}
diff --git a/src/runtime-contract.ts b/src/runtime-contract.ts
index e3f14565..49ac1bca 100644
--- a/src/runtime-contract.ts
+++ b/src/runtime-contract.ts
@@ -1,9 +1,5 @@
import type {
- AlignmentEvidence,
AttributeValue,
- LocatedScript,
- NarrativeIR,
- PlanIR,
PlanInstance,
PlanValue,
ProgramRange,
@@ -16,7 +12,6 @@ export type VisualFragment = {
startFrame: number;
endFrameExclusive: number;
z: number;
- layer?: number;
source?: string;
mediaStartSec?: number;
playbackRate?: number;
@@ -65,19 +60,20 @@ export type HyperframesDocument = {
export type RuntimeValue = PlanValue & {
absoluteSource?: string;
+ data?: unknown;
};
export type KernelContext = {
instance: PlanInstance;
element: SourceElement;
- narrative: NarrativeIR;
- located: LocatedScript;
- alignment: AlignmentEvidence;
- plan: PlanIR;
- values: Map<string, RuntimeValue>;
- results: Map<string, KernelProjection>;
resolve: (value: AttributeValue | undefined) => unknown;
- resolvePath: (path: string) => unknown;
+ digest: (value: unknown) => string;
+ program?: {
+ basisDigest: string;
+ fps: number;
+ durationFrames: number;
+ durationSec: number;
+ };
temporalContracts: Record<string, {
kind: "selection" | "moment";
consume: "one" | "each" | "set";
diff --git a/src/runtime-validation.ts b/src/runtime-validation.ts
index 284e522f..72b789bf 100644
--- a/src/runtime-validation.ts
+++ b/src/runtime-validation.ts
@@ -149,7 +149,7 @@ function normalizeSurface(value: unknown): void {
}
const audioIds = (surface.audios ?? []).map((fragment) => fragment.id);
if (new Set(audioIds).size !== audioIds.length) {
- fail("kernel_audio_duplicate_id", "A projected audio tree repeats a fragment id.");
+ fail("kernel_audio_duplicate_id", "A projected audio contribution set repeats a fragment id.");
}
const styles = Array.isArray(surface.styles) ? surface.styles : [];
if (surface.visuals) surface.visuals = surface.visuals.map((fragment) =>
diff --git a/src/sandbox-worker.mjs b/src/sandbox-worker.mjs
index ee285b1a..b38f2b69 100644
--- a/src/sandbox-worker.mjs
+++ b/src/sandbox-worker.mjs
@@ -1,4 +1,5 @@
import { readFile, realpath } from "node:fs/promises";
+import { createHash } from "node:crypto";
import { dirname, relative, resolve } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import vm from "node:vm";
@@ -25,6 +26,7 @@ async function main() {
name: "svml-kernel-projector",
codeGeneration: { strings: false, wasm: false },
});
+ context.__svmlSha256 = value => createHash("sha256").update(value).digest("hex");
new vm.Script(`
Object.defineProperty(Math, "random", {
value() { throw new Error("SVK projector cannot use ambient randomness"); },
@@ -84,9 +86,14 @@ async function main() {
(() => {
const context = JSON.parse(globalThis.__svmlInput);
context.resolve = value => value;
- context.resolvePath = () => {
- throw new Error("resolvePath is unavailable in isolated projector ABI v1");
+ const stableJson = value => {
+ if (Array.isArray(value)) return "[" + value.map(stableJson).join(",") + "]";
+ if (value && typeof value === "object") {
+ return "{" + Object.keys(value).sort().map(key => JSON.stringify(key) + ":" + stableJson(value[key])).join(",") + "}";
+ }
+ return JSON.stringify(value);
};
+ context.digest = value => globalThis.__svmlSha256(stableJson(value));
const frameRange = (startSec, endSec) => {
const startFrame = Math.round(startSec * context.fps);
const endFrameExclusive = Math.round(endSec * context.fps);
@@ -101,10 +108,10 @@ async function main() {
const endpoint = raw => {
const value = String(raw).trim();
if (value === "start") return 0;
- if (value === "end") return context.located.durationSec;
+ if (value === "end") return context.program.durationSec;
const relative = /^(start|end)([+-])(\\d+(?:\\.\\d+)?)(ms|s)$/.exec(value);
if (relative) {
- const base = relative[1] === "start" ? 0 : context.located.durationSec;
+ const base = relative[1] === "start" ? 0 : context.program.durationSec;
const amount = Number(relative[3]) * (relative[4] === "ms" ? 0.001 : 1);
return base + (relative[2] === "+" ? amount : -amount);
}
@@ -123,7 +130,7 @@ async function main() {
const contract = temporalContract(path, "selection");
let ranges;
if (typeof value === "string") {
- if (value === "full") ranges = [frameRange(0, context.located.durationSec)];
+ if (value === "full") ranges = [frameRange(0, context.program.durationSec)];
else {
const parts = value.split("..");
if (parts.length !== 2) throw new Error("manual span requires start .. end");
diff --git a/src/script/locate.ts b/src/script/locate.ts
index 949eaae1..40a39211 100644
--- a/src/script/locate.ts
+++ b/src/script/locate.ts
@@ -1,23 +1,29 @@
import { fail } from "../diagnostics.js";
import type {
- AlignmentEvidence,
+ ExactSemanticMap,
LocatedScript,
MarkerBoundary,
NarrativeIR,
+ ProgramBasis,
ProgramRange,
+ SemanticMap,
} from "../model.js";
-import { normalizeWord } from "../util.js";
+import { basisFps, semanticPoints, validateSemanticMap } from "../temporal.js";
-function frameAt(seconds: number, fps: number): number {
- return Math.round(seconds * fps);
-}
-
-function range(startSec: number, endSec: number, fps: number): ProgramRange {
- const startFrame = frameAt(startSec, fps);
- const endFrameExclusive = frameAt(endSec, fps);
- if (endFrameExclusive < startFrame) {
- fail("locate_negative_range", `Range ${startSec}..${endSec} resolves backwards.`);
+function range(
+ startFrame: number,
+ endFrameExclusive: number,
+ basis: ProgramBasis,
+ label: string,
+ requirePositive = false,
+): ProgramRange {
+ if (endFrameExclusive < startFrame || (requirePositive && endFrameExclusive === startFrame)) {
+ fail(
+ requirePositive ? "locate_nonpositive_range" : "locate_negative_range",
+ `${label} resolves to non-positive range ${startFrame}..${endFrameExclusive}.`,
+ );
}
+ const fps = basisFps(basis);
return {
startFrame,
endFrameExclusive,
@@ -26,180 +32,72 @@ function range(startSec: number, endSec: number, fps: number): ProgramRange {
};
}
-function boundaryTime(
+export function boundaryAnchor(
+ narrative: NarrativeIR,
boundary: MarkerBoundary,
affinity: "left" | "right",
- narrative: NarrativeIR,
- evidence: AlignmentEvidence,
-): number {
- const left = boundary.tokenIndex > 0 ? evidence.words[boundary.tokenIndex - 1] : undefined;
- const right = evidence.words[boundary.tokenIndex];
- if (affinity === "left" && left) return left.endSec;
- if (affinity === "right" && right) return right.startSec;
-
- const segment = boundary.segmentId
- ? evidence.segments.find((item) => item.id === boundary.segmentId)
- : undefined;
- if (segment) return affinity === "left" ? segment.endSec : segment.startSec;
+): string {
+ if (boundary.segmentId) {
+ const segment = narrative.segments.find((item) => item.id === boundary.segmentId);
+ if (!segment) fail("locate_boundary_segment", `Unknown Segment "${boundary.segmentId}".`);
+ if (affinity === "left") {
+ const token = narrative.tokens[boundary.tokenIndex - 1];
+ if (token && token.segmentId === segment.id) return token.endAnchorId;
+ return segment.startAnchorId;
+ }
+ const token = narrative.tokens[boundary.tokenIndex];
+ if (token && token.segmentId === segment.id) return token.startAnchorId;
+ return segment.endAnchorId;
+ }
- const cut = boundary.structuralCut ?? 0;
- if (cut <= 0) return 0;
- if (cut >= narrative.segments.length) return evidence.durationSec;
- const rightSegment = evidence.segments.find((item) => item.id === narrative.segments[cut]?.id);
- const leftSegment = evidence.segments.find((item) => item.id === narrative.segments[cut - 1]?.id);
- if (affinity === "right" && rightSegment) return rightSegment.startSec;
- if (leftSegment) return leftSegment.endSec;
- fail("locate_unresolved_boundary", `Could not resolve structural cut ${cut}.`);
+ const position = boundary.structuralPosition;
+ if (affinity === "left") {
+ return narrative.segments[position - 1]?.endAnchorId
+ ?? narrative.segments[position]?.startAnchorId
+ ?? fail("locate_boundary_empty", "Script has no structural boundary candidate.");
+ }
+ return narrative.segments[position]?.startAnchorId
+ ?? narrative.segments[position - 1]?.endAnchorId
+ ?? fail("locate_boundary_empty", "Script has no structural boundary candidate.");
}
-export function locateScript(
+function validateCaptionCues(
narrative: NarrativeIR,
- evidence: AlignmentEvidence,
-): LocatedScript {
- if (evidence.contract !== "svml.speech-alignment.v1") {
- fail("locate_alignment_contract", `Unsupported alignment contract "${evidence.contract}".`);
- }
- if (
- !Number.isFinite(evidence.fps)
- || evidence.fps <= 0
- || !Number.isFinite(evidence.durationSec)
- || evidence.durationSec <= 0
- ) {
- fail("locate_alignment_clock", "Alignment evidence requires positive finite fps and durationSec.");
- }
- if (narrative.tokens.length !== evidence.words.length) {
- fail(
- "locate_alignment_cardinality",
- `Script has ${narrative.tokens.length} words but alignment has ${evidence.words.length}.`,
- );
- }
- for (let index = 0; index < evidence.words.length; index += 1) {
- const word = evidence.words[index];
- if (!word) continue;
- if (
- !Number.isFinite(word.startSec)
- || !Number.isFinite(word.endSec)
- || word.startSec < 0
- || word.endSec < word.startSec
- || word.endSec > evidence.durationSec
- ) {
- fail(
- "locate_alignment_word_window",
- `Alignment word ${index + 1} has invalid window ${word.startSec}..${word.endSec}.`,
- );
- }
- const previous = evidence.words[index - 1];
- if (previous && word.startSec < previous.endSec - 1e-6) {
- fail(
- "locate_alignment_word_overlap",
- `Alignment word ${index + 1} starts at ${word.startSec} before word ${index} ends at ${previous.endSec}.`,
- );
- }
- const segment = evidence.segments.find((item) => item.id === word.segmentId);
- if (
- !segment
- || word.startSec < segment.startSec - 1e-6
- || word.endSec > segment.endSec + 1e-6
- ) {
- fail(
- "locate_alignment_word_segment",
- `Alignment word ${index + 1} falls outside segment "${word.segmentId}".`,
- );
- }
- }
- for (let index = 0; index < narrative.tokens.length; index += 1) {
- const token = narrative.tokens[index];
- const word = evidence.words[index];
- if (!token || !word) continue;
- if (token.normalized !== normalizeWord(word.text)) {
- fail(
- "locate_alignment_word_mismatch",
- `Word ${index + 1} is "${token.text}" in Script and "${word.text}" in alignment evidence.`,
- );
- }
- if (token.segmentId !== word.segmentId) {
- fail(
- "locate_alignment_segment_mismatch",
- `Word ${index + 1} belongs to ${token.segmentId} in Script and ${word.segmentId} in evidence.`,
- );
- }
- }
- const fps = evidence.fps;
- const words = narrative.tokens.map((token, index) => {
- const word = evidence.words[index];
- if (!word) fail("locate_alignment_missing_word", `Missing aligned word ${index + 1}.`);
- return { ...token, ...range(word.startSec, word.endSec, fps) };
- });
- const segments = Object.fromEntries(evidence.segments.map((segment) => [
- segment.id,
- range(segment.startSec, segment.endSec, fps),
- ]));
- const selections = Object.fromEntries(Object.entries(narrative.selections).map(([id, occurrences]) => [
- id,
- {
- id,
- ranges: occurrences.map((occurrence) => range(
- boundaryTime(occurrence.open.boundary, occurrence.open.affinity, narrative, evidence),
- boundaryTime(occurrence.close.boundary, occurrence.close.affinity, narrative, evidence),
- fps,
- )),
- },
- ]));
- const moments = Object.fromEntries(Object.entries(narrative.moments).map(([id, occurrences]) => [
- id,
- {
- id,
- frames: occurrences.map((occurrence) => frameAt(
- boundaryTime(occurrence.boundary, occurrence.affinity, narrative, evidence),
- fps,
- )),
- },
- ]));
- const captionAtoms = narrative.captionAtoms.map((atom) => {
- const first = words[atom.startWord];
- const last = words[atom.endWordExclusive - 1];
- if (!first || !last || first.segmentId !== last.segmentId) {
- fail("locate_caption_atom", `Caption atom "${atom.id}" has an invalid speech span.`);
- }
- return {
- ...atom,
- ...range(first.startSec, last.endSec, fps),
- };
- });
- const captionCues = evidence.captionCues?.map((cue, index) => {
+ locatedWords: LocatedScript["words"],
+ locatedAtoms: LocatedScript["captionAtoms"],
+ map: SemanticMap,
+ basis: ProgramBasis,
+): LocatedScript["captionCues"] {
+ const cues = map.captionCues;
+ if (!cues) return undefined;
+ const located = cues.map((cue, index) => {
if (!Number.isInteger(cue.startWord) || !Number.isInteger(cue.endWordExclusive)) {
- fail("locate_caption_cue_index", `Caption cue ${index + 1} must use integer word indexes.`);
+ fail("locate_caption_cue_index", `Caption cue ${index + 1} must use integer token indexes.`);
}
if (
cue.startWord < 0
|| cue.endWordExclusive <= cue.startWord
- || cue.endWordExclusive > words.length
+ || cue.endWordExclusive > locatedWords.length
) {
fail(
"locate_caption_cue_range",
- `Caption cue ${index + 1} has invalid word range ${cue.startWord}..${cue.endWordExclusive}.`,
+ `Caption cue ${index + 1} has invalid token range ${cue.startWord}..${cue.endWordExclusive}.`,
);
}
- const previous = evidence.captionCues?.[index - 1];
+ const previous = cues[index - 1];
const expectedStart = previous?.endWordExclusive ?? 0;
if (cue.startWord !== expectedStart) {
fail(
"locate_caption_cue_partition",
- `Caption cue ${index + 1} starts at word ${cue.startWord}; expected ${expectedStart}.`,
+ `Caption cue ${index + 1} starts at token ${cue.startWord}; expected ${expectedStart}.`,
);
}
- const first = words[cue.startWord];
- const last = words[cue.endWordExclusive - 1];
- if (!first || !last) {
- fail("locate_caption_cue_word", `Caption cue ${index + 1} references a missing word.`);
- }
- if (first.segmentId !== last.segmentId) {
- fail(
- "locate_caption_cue_segment",
- `Caption cue ${index + 1} crosses Script segments ${first.segmentId} and ${last.segmentId}.`,
- );
+ const first = locatedWords[cue.startWord];
+ const last = locatedWords[cue.endWordExclusive - 1];
+ if (!first || !last || first.segmentId !== last.segmentId) {
+ fail("locate_caption_cue_segment", `Caption cue ${index + 1} crosses Script Segments.`);
}
- for (const atom of captionAtoms) {
+ for (const atom of locatedAtoms) {
const overlaps = cue.startWord < atom.endWordExclusive
&& cue.endWordExclusive > atom.startWord;
const contains = cue.startWord <= atom.startWord
@@ -215,21 +113,105 @@ export function locateScript(
id: cue.id ?? `cue-${index + 1}`,
startWord: cue.startWord,
endWordExclusive: cue.endWordExclusive,
- ...range(first.startSec, last.endSec, fps),
+ ...range(first.startFrame, last.endFrameExclusive, basis, `Caption cue ${index + 1}`),
};
});
- if (
- captionCues
- && (captionCues.at(-1)?.endWordExclusive ?? 0) !== words.length
- ) {
+ if ((located.at(-1)?.endWordExclusive ?? 0) !== narrative.tokens.length) {
fail(
"locate_caption_cue_partition",
- `Caption cues end at word ${captionCues.at(-1)?.endWordExclusive ?? 0}; expected ${words.length}.`,
+ `Caption cues end at token ${located.at(-1)?.endWordExclusive ?? 0}; expected ${narrative.tokens.length}.`,
);
}
+ return located;
+}
+
+export function locateScript(
+ narrative: NarrativeIR,
+ basis: ProgramBasis,
+ map: ExactSemanticMap,
+): LocatedScript {
+ if (map.contract !== "svml.exact-semantic-map.v1") {
+ fail("locate_exact_required", "Located HTML compilation requires ExactSemanticMap.");
+ }
+ return bindSemanticMap(narrative, basis, map);
+}
+
+export function bindSemanticMap(
+ narrative: NarrativeIR,
+ basis: ProgramBasis,
+ map: SemanticMap,
+): LocatedScript {
+ validateSemanticMap(narrative, basis, map);
+ const points = semanticPoints(map);
+ const point = (identity: string): number => {
+ const value = points.get(identity);
+ if (value === undefined) fail("locate_anchor_missing", `SemanticMap is missing "${identity}".`);
+ return value;
+ };
+ const fps = basisFps(basis);
+ const words = narrative.tokens.map((token) => ({
+ ...token,
+ ...range(
+ point(token.startAnchorId),
+ point(token.endAnchorId),
+ basis,
+ `Token "${token.text}"`,
+ ),
+ }));
+ const segments = Object.fromEntries(narrative.segments.map((segment) => [
+ segment.id,
+ range(
+ point(segment.startAnchorId),
+ point(segment.endAnchorId),
+ basis,
+ `Segment "${segment.id}"`,
+ ),
+ ]));
+ const selections = Object.fromEntries(Object.entries(narrative.selections).map(
+ ([id, occurrences]) => [
+ id,
+ {
+ id,
+ ranges: occurrences.map((occurrence, index) => range(
+ point(boundaryAnchor(narrative, occurrence.open.boundary, occurrence.open.affinity)),
+ point(boundaryAnchor(narrative, occurrence.close.boundary, occurrence.close.affinity)),
+ basis,
+ `Selection "${id}" occurrence ${index + 1}`,
+ true,
+ )),
+ },
+ ],
+ ));
+ const moments = Object.fromEntries(Object.entries(narrative.moments).map(
+ ([id, occurrences]) => [
+ id,
+ {
+ id,
+ frames: occurrences.map((occurrence) => point(
+ boundaryAnchor(narrative, occurrence.boundary, occurrence.affinity),
+ )),
+ },
+ ],
+ ));
+ const captionAtoms = narrative.captionAtoms.map((atom) => {
+ const first = words[atom.startWord];
+ const last = words[atom.endWordExclusive - 1];
+ if (!first || !last || first.segmentId !== last.segmentId) {
+ fail("locate_caption_atom", `Caption atom "${atom.id}" has an invalid speech span.`);
+ }
+ return {
+ ...atom,
+ ...range(first.startFrame, last.endFrameExclusive, basis, `Caption atom "${atom.id}"`),
+ };
+ });
+ const captionCues = validateCaptionCues(narrative, words, captionAtoms, map, basis);
return {
- durationFrames: frameAt(evidence.durationSec, fps),
- durationSec: frameAt(evidence.durationSec, fps) / fps,
+ contract: "svml.temporal-binding.v1",
+ precision: map.contract === "svml.exact-semantic-map.v1" ? "exact" : "estimated",
+ basisDigest: basis.basisDigest,
+ semanticMapDigest: map.mapDigest,
+ durationFrames: basis.durationFrames,
+ durationSec: basis.durationFrames / fps,
fps,
words,
segments,
diff --git a/src/script/parse.ts b/src/script/parse.ts
index a3cbdc5f..4f626dab 100644
--- a/src/script/parse.ts
+++ b/src/script/parse.ts
@@ -8,7 +8,7 @@ import type {
ScriptToken,
SelectionOccurrence,
} from "../model.js";
-import { normalizeWord } from "../util.js";
+import { normalizeWord, sha256, stableJson } from "../util.js";
type Marker = {
id: string;
@@ -77,15 +77,27 @@ function joinProjection(parts: string[]): string {
function markerBoundary(
tokenIndex: number,
segment: MutableSegment | undefined,
- structuralCut: number,
+ structuralPosition: number,
): MarkerBoundary {
return {
tokenIndex,
- structuralCut,
+ structuralPosition,
...(segment ? { segmentId: segment.id } : {}),
};
}
+function segmentAnchorId(segmentId: string, edge: "start" | "end"): string {
+ return `segment:${segmentId}:${edge}`;
+}
+
+function tokenAnchorId(
+ segmentId: string,
+ segmentTokenIndex: number,
+ edge: "start" | "end",
+): string {
+ return `segment:${segmentId}:token:${segmentTokenIndex + 1}:${edge}`;
+}
+
export function parseScript(
file: string,
scriptSource: string,
@@ -105,7 +117,7 @@ export function parseScript(
const captionAtoms: NarrativeIR["captionAtoms"] = [];
let current: MutableSegment | undefined;
let offset = 0;
- let structuralCut = 0;
+ let structuralPosition = 0;
let lineStart = true;
const location = (at: number) => sourceLocation(file, scriptSource, at);
@@ -196,7 +208,7 @@ export function parseScript(
literalPieces(raw, absoluteStart, dual).map((piece) => piece.value).join("");
const addMarker = (marker: Marker, at: number): void => {
- const boundary = markerBoundary(tokens.length, current, structuralCut);
+ const boundary = markerBoundary(tokens.length, current, structuralPosition);
if (marker.kind === "moment") {
if (openSelections.has(marker.id) || selections[marker.id]) {
fail(
@@ -292,10 +304,14 @@ export function parseScript(
const normalized = normalizeWord(match[0]);
if (!normalized) continue;
const index = tokens.length;
+ const segmentTokenIndex = index - current.tokenStart;
tokens.push({
id: `w${index + 1}`,
index,
segmentId: current.id,
+ segmentTokenIndex,
+ startAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "start"),
+ endAnchorId: tokenAnchorId(current.id, segmentTokenIndex, "end"),
text: match[0],
normalized,
sourceStart: sourceOffset + start + match.index,
@@ -402,11 +418,13 @@ export function parseScript(
if (segmentOpen[3] === "/") {
segments.push({
...current,
+ startAnchorId: segmentAnchorId(current.id, "start"),
+ endAnchorId: segmentAnchorId(current.id, "end"),
tokenEnd: tokens.length,
sourceEnd: sourceOffset + offset,
});
current = undefined;
- structuralCut += 1;
+ structuralPosition += 1;
}
continue;
}
@@ -419,11 +437,13 @@ export function parseScript(
offset += segmentClose[0].length;
segments.push({
...current,
+ startAnchorId: segmentAnchorId(current.id, "start"),
+ endAnchorId: segmentAnchorId(current.id, "end"),
tokenEnd: tokens.length,
sourceEnd: sourceOffset + offset,
});
current = undefined;
- structuralCut += 1;
+ structuralPosition += 1;
lineStart = true;
continue;
}
@@ -561,12 +581,58 @@ export function parseScript(
if (caption) captionSegments.push(caption);
}
+ const semanticAnchors = segments.flatMap((segment) => [
+ {
+ id: segment.startAnchorId,
+ kind: "segment-start" as const,
+ segmentId: segment.id,
+ },
+ ...tokens.slice(segment.tokenStart, segment.tokenEnd).flatMap((token) => [
+ {
+ id: token.startAnchorId,
+ kind: "token-start" as const,
+ segmentId: segment.id,
+ tokenId: token.id,
+ segmentTokenIndex: token.segmentTokenIndex,
+ },
+ {
+ id: token.endAnchorId,
+ kind: "token-end" as const,
+ segmentId: segment.id,
+ tokenId: token.id,
+ segmentTokenIndex: token.segmentTokenIndex,
+ },
+ ]),
+ {
+ id: segment.endAnchorId,
+ kind: "segment-end" as const,
+ segmentId: segment.id,
+ },
+ ]);
+ const semanticTokens = tokens.map((token) => ({
+ id: token.id,
+ segmentId: token.segmentId,
+ segmentTokenIndex: token.segmentTokenIndex,
+ normalized: token.normalized,
+ startAnchorId: token.startAnchorId,
+ endAnchorId: token.endAnchorId,
+ }));
+ const semanticIndexPayload = {
+ contract: "svml.semantic-index.v1" as const,
+ anchors: semanticAnchors,
+ tokens: semanticTokens,
+ };
+
return {
segments,
tokens,
selections,
moments,
captionAtoms,
+ semanticIndex: {
+ ...semanticIndexPayload,
+ digest: sha256(stableJson(semanticIndexPayload)),
+ },
projections: {
dialogue: dialogueTurns.join("\n"),
speech: speechSegments.join("\n"),
diff --git a/src/source/modules.ts b/src/source/modules.ts
index b2ce8318..1b9a683c 100644
--- a/src/source/modules.ts
+++ b/src/source/modules.ts
@@ -78,7 +78,7 @@ function cloneNode(
if (localId) attributes.id = `${args.prefix}${localId}`;
if (
args.topLevel
- && ["image", "video", "audio"].includes(node.name)
+ && ["image", "video", "audio", "alignment"].includes(node.name)
&& typeof attributes.src === "string"
&& !isAbsolute(attributes.src)
&& !isExternalSource(attributes.src)
diff --git a/src/source/parse-document.ts b/src/source/parse-document.ts
index dc1a45c4..80896096 100644
--- a/src/source/parse-document.ts
+++ b/src/source/parse-document.ts
@@ -73,7 +73,7 @@ function parseReference(cursor: Cursor): AttributeValue {
}
const path = cursor.source.slice(start, cursor.offset).trim();
cursor.offset += 1;
- if (!/^[A-Za-z_][A-Za-z0-9_.:-]*$/u.test(path)) {
+ if (!/^[A-Za-z_][A-Za-z0-9_.:-]*(?:\[[A-Za-z_][A-Za-z0-9_.:-]*\])?$/u.test(path)) {
fail(
"source_invalid_reference",
`Invalid reference path "${path}".`,
diff --git a/src/source/styles.ts b/src/source/styles.ts
index a858e893..2e08053c 100644
--- a/src/source/styles.ts
+++ b/src/source/styles.ts
@@ -10,6 +10,8 @@ import type {
SourceNode,
} from "../model.js";
import type {
+ KernelChildSchema,
+ KernelField,
KernelManifest,
KernelParameter,
KernelRegistry,
@@ -57,8 +59,20 @@ function typedValue(
if (typeof value !== "boolean") {
fail("style_parameter_type", `${source} expects boolean ${parameter.styleName}.`);
}
- } else if (typeof value !== "string") {
- fail("style_parameter_type", `${source} expects string ${parameter.styleName}.`);
+ } else {
+ if (typeof value !== "string") {
+ fail("style_parameter_type", `${source} expects string ${parameter.styleName}.`);
+ }
+ if (
+ parameter.type === "duration"
+ && !/^[+-]?\d+(?:\.\d+)?(?:ms|s)$/u.test(value)
+ ) {
+ fail("style_parameter_type", `${source} expects duration ${parameter.styleName}.`);
+ }
+ const enumValues = /^enum\((.+)\)$/u.exec(parameter.type)?.[1]?.split(",");
+ if (enumValues && !enumValues.includes(value)) {
+ fail("style_parameter_type", `${source} expects ${parameter.type} ${parameter.styleName}.`);
+ }
}
return value;
}
@@ -140,6 +154,7 @@ function findRule(
token: string,
element: SourceElement,
sheets: Sheet[],
+ selector: string,
): StyleRule {
const dot = token.indexOf(".");
if (dot > 0) {
@@ -148,18 +163,18 @@ function findRule(
const sheet = sheets.find((candidate) => candidate.binding === binding);
if (!sheet) fail("style_unknown_binding", `Unknown style binding "${binding}".`);
const rule = sheet.rules.find(
- (candidate) => candidate.selector === element.name && candidate.className === className,
+ (candidate) => candidate.selector === selector && candidate.className === className,
);
if (!rule) {
fail(
"style_unknown_class",
- `${sheet.file} does not define ${element.name}.${className}.`,
+ `${sheet.file} does not define ${selector}.${className}.`,
);
}
return rule;
}
const matches = sheets.flatMap((sheet) => sheet.rules.filter(
- (candidate) => candidate.selector === element.name && candidate.className === token,
+ (candidate) => candidate.selector === selector && candidate.className === token,
));
if (matches.length !== 1) {
fail(
@@ -180,6 +195,23 @@ function parameterFor(
);
}
+function fieldFor(
+ schemas: KernelChildSchema[],
+ elementName: string,
+ styleName: string,
+): KernelField | undefined {
+ for (const schema of schemas) {
+ if (schema.name === elementName) {
+ const field = schema.fields.find((candidate) =>
+ candidate.styleName === styleName || candidate.name === styleName);
+ if (field) return field;
+ }
+ const nested = fieldFor(schema.children, elementName, styleName);
+ if (nested) return nested;
+ }
+ return undefined;
+}
+
function styleElement(
element: SourceElement,
sheets: Sheet[],
@@ -187,7 +219,16 @@ function styleElement(
ownerKernel?: KernelManifest,
): SourceElement {
const manifest = kernels.get(element.name) ?? ownerKernel;
+ if (element.parameterSources) {
+ return {
+ ...element,
+ children: element.children.map((child) => child.kind === "element"
+ ? styleElement(child, sheets, kernels, manifest)
+ : child),
+ };
+ }
const topLevelKernel = kernels.get(element.name);
+ const selector = topLevelKernel?.name ?? ownerKernel?.name ?? element.name;
const classValue = element.attributes.class;
if (classValue !== undefined && typeof classValue !== "string") {
fail("style_class_type", `<${element.name}> class must be a string.`);
@@ -195,19 +236,27 @@ function styleElement(
const attributes: Record<string, AttributeValue> = {};
const parameterSources: Record<string, ParameterSource[]> = {};
for (const token of (classValue ?? "").split(/\s+/u).filter(Boolean)) {
- const rule = findRule(token, element, sheets);
+ const rule = findRule(token, element, sheets, selector);
for (const property of rule.properties) {
const parameter = topLevelKernel
? parameterFor(topLevelKernel, property.name)
: undefined;
- if (topLevelKernel && !parameter) {
+ const field = !topLevelKernel && ownerKernel
+ ? fieldFor(ownerKernel.children, element.name, property.name)
+ : undefined;
+ if ((topLevelKernel || ownerKernel) && !parameter && !field) {
fail(
"style_unknown_parameter",
`${rule.source} configures unknown public parameter "${property.name}".`,
);
}
- const name = parameter?.name ?? property.name;
- const value = typedValue(property.raw, parameter, rule.source);
+ const schema = parameter ?? (field ? {
+ name: field.name,
+ styleName: field.styleName,
+ type: field.type,
+ } : undefined);
+ const name = schema?.name ?? property.name;
+ const value = typedValue(property.raw, schema, rule.source);
attributes[name] = value;
(parameterSources[name] ??= []).push({
kind: "svs-class",
diff --git a/src/temporal.ts b/src/temporal.ts
new file mode 100644
index 00000000..3e12b189
--- /dev/null
+++ b/src/temporal.ts
@@ -0,0 +1,484 @@
+import { fail } from "./diagnostics.js";
+import type {
+ AlignmentEvidence,
+ ExactSemanticMap,
+ EstimatedSemanticMap,
+ NarrativeIR,
+ ProgramBasis,
+ SemanticAnchorPoint,
+ SemanticMap,
+ TemporalBasisProduction,
+} from "./model.js";
+import { normalizeWord, sha256, stableJson } from "./util.js";
+
+const EPSILON = 1e-6;
+
+export function basisFps(basis: ProgramBasis): number {
+ return basis.frameRate.numerator / basis.frameRate.denominator;
+}
+
+function integer(value: number, label: string): void {
+ if (!Number.isInteger(value)) fail("temporal_integer", `${label} must be an integer.`);
+}
+
+function normalizeFrameRate(fps: number): ProgramBasis["frameRate"] {
+ if (!Number.isFinite(fps) || fps <= 0) {
+ fail("basis_frame_rate", `ProgramBasis fps must be positive and finite; received ${fps}.`);
+ }
+ if (Number.isInteger(fps)) return { numerator: fps, denominator: 1 };
+ const denominator = 1_000_000;
+ const numerator = Math.round(fps * denominator);
+ const divisor = gcd(numerator, denominator);
+ return { numerator: numerator / divisor, denominator: denominator / divisor };
+}
+
+function gcd(left: number, right: number): number {
+ left = Math.abs(left);
+ right = Math.abs(right);
+ while (right) [left, right] = [right, left % right];
+ return left || 1;
+}
+
+export function createProgramBasis(args: {
+ fps: number;
+ durationFrames: number;
+ outcomeDigest: string;
+}): ProgramBasis {
+ integer(args.durationFrames, "ProgramBasis durationFrames");
+ if (args.durationFrames <= 0) {
+ fail("basis_duration", "ProgramBasis durationFrames must be positive.");
+ }
+ const payload = {
+ contract: "svml.program-basis.v1" as const,
+ frameRate: normalizeFrameRate(args.fps),
+ originFrame: 0 as const,
+ durationFrames: args.durationFrames,
+ outcomeDigest: args.outcomeDigest,
+ };
+ return {
+ contract: payload.contract,
+ frameRate: payload.frameRate,
+ originFrame: payload.originFrame,
+ durationFrames: payload.durationFrames,
+ basisDigest: sha256(stableJson(payload)),
+ };
+}
+
+export function validateProgramBasis(basis: ProgramBasis): void {
+ if (basis.contract !== "svml.program-basis.v1") {
+ fail("basis_contract", `Unsupported ProgramBasis contract "${basis.contract}".`);
+ }
+ integer(basis.frameRate.numerator, "ProgramBasis frameRate numerator");
+ integer(basis.frameRate.denominator, "ProgramBasis frameRate denominator");
+ integer(basis.durationFrames, "ProgramBasis durationFrames");
+ if (
+ basis.originFrame !== 0
+ || basis.frameRate.numerator <= 0
+ || basis.frameRate.denominator <= 0
+ || basis.durationFrames <= 0
+ || !/^[a-f0-9]{64}$/u.test(basis.basisDigest)
+ ) {
+ fail("basis_shape", "ProgramBasis has an invalid clock or digest.");
+ }
+}
+
+function canonicalFacet(value: unknown): unknown {
+ if (Array.isArray(value)) return value.map(canonicalFacet);
+ if (!value || typeof value !== "object") return value;
+ return Object.fromEntries(Object.entries(value as Record<string, unknown>)
+ .filter(([key]) => !["source", "absoluteSource", "basisDigest"].includes(key))
+ .map(([key, nested]) => [key, canonicalFacet(nested)]));
+}
+
+export function basisOutcome(production: TemporalBasisProduction): unknown {
+ return {
+ contract: "svml.program-basis-outcome.v1",
+ frameRate: production.basis.frameRate,
+ originFrame: production.basis.originFrame,
+ durationFrames: production.basis.durationFrames,
+ alignmentSubjects: production.alignmentSubjects,
+ sourceMaps: production.sourceMaps,
+ audio: production.audio.map(({ source: _source, ...contribution }) => contribution),
+ facets: canonicalFacet(production.facets),
+ };
+}
+
+function digest(value: string, label: string): void {
+ if (!/^[a-f0-9]{64}$/u.test(value)) {
+ fail("temporal_digest", `${label} must be a lowercase SHA-256 digest.`);
+ }
+}
+
+function uniqueIds(values: Array<{ id: string }>, label: string): void {
+ if (new Set(values.map((value) => value.id)).size !== values.length) {
+ fail("basis_identity_duplicate", `${label} contains duplicate ids.`);
+ }
+}
+
+function validateProgramWindow(
+ start: number,
+ end: number,
+ durationFrames: number,
+ label: string,
+): void {
+ integer(start, `${label} start`);
+ integer(end, `${label} end`);
+ if (start < 0 || end <= start || end > durationFrames) {
+ fail("basis_program_window", `${label} has invalid Program range ${start}..${end}.`);
+ }
+}
+
+function validateFacetAffinity(value: unknown, basisDigest: string, path = "facets"): void {
+ if (Array.isArray(value)) {
+ value.forEach((nested, index) => validateFacetAffinity(nested, basisDigest, `${path}[${index}]`));
+ return;
+ }
+ if (!value || typeof value !== "object") return;
+ const record = value as Record<string, unknown>;
+ if ("basisDigest" in record && record.basisDigest !== basisDigest) {
+ fail("basis_facet_affinity", `${path} is bound to a different ProgramBasis.`);
+ }
+ for (const [key, nested] of Object.entries(record)) {
+ validateFacetAffinity(nested, basisDigest, `${path}.${key}`);
+ }
+}
+
+export function validateTemporalBasisProduction(production: TemporalBasisProduction): void {
+ if (production.contract !== "svml.temporal-basis-production.v1") {
+ fail("basis_production_contract", "Unsupported TemporalBasisProduction contract.");
+ }
+ validateProgramBasis(production.basis);
+ digest(production.productionDigest, "productionDigest");
+ uniqueIds(production.alignmentSubjects, "alignmentSubjects");
+ uniqueIds(production.sourceMaps, "sourceMaps");
+ uniqueIds(production.audio, "Program audio");
+ const maps = new Map(production.sourceMaps.map((map) => [map.id, map]));
+ for (const map of production.sourceMaps) {
+ digest(map.sourceDigest, `Source map "${map.id}" digest`);
+ integer(map.sourceStartFrame, `Source map "${map.id}" source start`);
+ integer(map.sourceEndFrameExclusive, `Source map "${map.id}" source end`);
+ if (map.sourceStartFrame < 0 || map.sourceEndFrameExclusive <= map.sourceStartFrame) {
+ fail("basis_source_window", `Source map "${map.id}" has an invalid source range.`);
+ }
+ validateProgramWindow(
+ map.programStartFrame,
+ map.programEndFrameExclusive,
+ production.basis.durationFrames,
+ `Source map "${map.id}"`,
+ );
+ }
+ for (const subject of production.alignmentSubjects) {
+ digest(subject.sourceDigest, `Alignment subject "${subject.id}" digest`);
+ const map = maps.get(subject.sourceMapId);
+ if (
+ !map
+ || map.sourceId !== subject.sourceId
+ || map.sourceDigest !== subject.sourceDigest
+ ) {
+ fail("basis_alignment_subject", `Alignment subject "${subject.id}" has no matching source map.`);
+ }
+ }
+ for (const contribution of production.audio) {
+ digest(contribution.sourceDigest, `Program audio "${contribution.id}" digest`);
+ const map = maps.get(contribution.sourceMapId);
+ if (
+ !map
+ || map.sourceId !== contribution.sourceId
+ || map.sourceDigest !== contribution.sourceDigest
+ ) {
+ fail("basis_audio_source_map", `Program audio "${contribution.id}" has no matching source map.`);
+ }
+ validateProgramWindow(
+ contribution.programStartFrame,
+ contribution.programEndFrameExclusive,
+ production.basis.durationFrames,
+ `Program audio "${contribution.id}"`,
+ );
+ const duration = contribution.programEndFrameExclusive - contribution.programStartFrame;
+ for (const [name, value] of [
+ ["fadeInFrames", contribution.fadeInFrames ?? 0],
+ ["fadeOutFrames", contribution.fadeOutFrames ?? 0],
+ ] as const) {
+ integer(value, `Program audio "${contribution.id}" ${name}`);
+ if (value < 0 || value > duration) {
+ fail("basis_audio_fade", `Program audio "${contribution.id}" has invalid ${name}.`);
+ }
+ }
+ if (!Number.isFinite(contribution.gain) || contribution.gain < 0) {
+ fail("basis_audio_gain", `Program audio "${contribution.id}" has invalid gain.`);
+ }
+ }
+ validateFacetAffinity(production.facets, production.basis.basisDigest);
+ const expected = sha256(stableJson(basisOutcome(production)));
+ if (production.basis.basisDigest !== expected) {
+ fail(
+ "basis_digest",
+ `ProgramBasis digest ${production.basis.basisDigest} does not match canonical outcome ${expected}.`,
+ );
+ }
+}
+
+function frameAt(seconds: number, basis: ProgramBasis): number {
+ return Math.round(seconds * basisFps(basis));
+}
+
+function semanticMapDigest(map: Omit<SemanticMap, "mapDigest">): string {
+ return sha256(stableJson(map));
+}
+
+function validateAlignment(
+ narrative: NarrativeIR,
+ basis: ProgramBasis,
+ evidence: AlignmentEvidence,
+): void {
+ if (evidence.contract !== "svml.speech-alignment.v1") {
+ fail("locator_evidence_contract", `Unsupported alignment contract "${evidence.contract}".`);
+ }
+ if (
+ !Number.isFinite(evidence.fps)
+ || evidence.fps <= 0
+ || !Number.isFinite(evidence.durationSec)
+ || evidence.durationSec <= 0
+ ) {
+ fail("locator_evidence_clock", "Alignment evidence requires positive fps and duration.");
+ }
+ if (Math.abs(evidence.fps - basisFps(basis)) > EPSILON) {
+ fail(
+ "locator_basis_frame_rate",
+ `Alignment fps ${evidence.fps} does not match ProgramBasis fps ${basisFps(basis)}.`,
+ );
+ }
+ if (frameAt(evidence.durationSec, basis) !== basis.durationFrames) {
+ fail(
+ "locator_basis_duration",
+ `Alignment duration resolves to ${frameAt(evidence.durationSec, basis)} frames; ProgramBasis has ${basis.durationFrames}.`,
+ );
+ }
+ if (evidence.words.length !== narrative.tokens.length) {
+ fail(
+ "locator_word_cardinality",
+ `Script has ${narrative.tokens.length} tokens but alignment has ${evidence.words.length}.`,
+ );
+ }
+ if (evidence.segments.length !== narrative.segments.length) {
+ fail(
+ "locator_segment_cardinality",
+ `Script has ${narrative.segments.length} Segments but alignment has ${evidence.segments.length}.`,
+ );
+ }
+
+ const segments = new Map(evidence.segments.map((segment) => [segment.id, segment]));
+ if (segments.size !== evidence.segments.length) {
+ fail("locator_segment_duplicate", "Alignment evidence repeats a Segment id.");
+ }
+ for (const [index, segment] of narrative.segments.entries()) {
+ const aligned = segments.get(segment.id);
+ if (!aligned) fail("locator_segment_missing", `Alignment is missing Segment "${segment.id}".`);
+ if (
+ !Number.isFinite(aligned.startSec)
+ || !Number.isFinite(aligned.endSec)
+ || aligned.startSec < 0
+ || aligned.endSec < aligned.startSec
+ || aligned.endSec > evidence.durationSec + EPSILON
+ ) {
+ fail(
+ "locator_segment_window",
+ `Segment "${segment.id}" has invalid window ${aligned.startSec}..${aligned.endSec}.`,
+ );
+ }
+ const previousScript = narrative.segments[index - 1];
+ const previous = previousScript ? segments.get(previousScript.id) : undefined;
+ if (
+ previous
+ && (
+ aligned.startSec < previous.startSec - EPSILON
+ || aligned.endSec < previous.endSec - EPSILON
+ )
+ ) {
+ fail(
+ "locator_segment_order",
+ `Segment "${segment.id}" violates source-order start/end monotonicity.`,
+ );
+ }
+ }
+
+ const previousBySegment = new Map<string, AlignmentEvidence["words"][number]>();
+ for (const [index, token] of narrative.tokens.entries()) {
+ const word = evidence.words[index];
+ if (!word) fail("locator_word_missing", `Alignment is missing token ${index + 1}.`);
+ if (normalizeWord(word.text) !== token.normalized || word.segmentId !== token.segmentId) {
+ fail(
+ "locator_word_mismatch",
+ `Token ${index + 1} does not match alignment word "${word.text}" in Segment "${word.segmentId}".`,
+ );
+ }
+ const segment = segments.get(word.segmentId);
+ if (
+ !segment
+ || !Number.isFinite(word.startSec)
+ || !Number.isFinite(word.endSec)
+ || word.startSec < segment.startSec - EPSILON
+ || word.endSec < word.startSec
+ || word.endSec > segment.endSec + EPSILON
+ ) {
+ fail(
+ "locator_word_window",
+ `Alignment token ${index + 1} falls outside Segment "${word.segmentId}".`,
+ );
+ }
+ const previous = previousBySegment.get(word.segmentId);
+ if (previous && word.startSec < previous.endSec - EPSILON) {
+ fail(
+ "locator_word_overlap",
+ `Alignment token ${index + 1} overlaps the preceding token in Segment "${word.segmentId}".`,
+ );
+ }
+ previousBySegment.set(word.segmentId, word);
+ }
+}
+
+export function semanticMapFromAlignment(
+ narrative: NarrativeIR,
+ basis: ProgramBasis,
+ evidence: AlignmentEvidence,
+ options: { precision: "exact"; locatorDigest?: string },
+): ExactSemanticMap;
+export function semanticMapFromAlignment(
+ narrative: NarrativeIR,
+ basis: ProgramBasis,
+ evidence: AlignmentEvidence,
+ options: { precision: "estimated"; locatorDigest?: string },
+): EstimatedSemanticMap;
+export function semanticMapFromAlignment(
+ narrative: NarrativeIR,
+ basis: ProgramBasis,
+ evidence: AlignmentEvidence,
+ options: {
+ precision: "estimated" | "exact";
+ locatorDigest?: string;
+ },
+): EstimatedSemanticMap | ExactSemanticMap {
+ validateProgramBasis(basis);
+ validateAlignment(narrative, basis, evidence);
+ const segmentEvidence = new Map(evidence.segments.map((segment) => [segment.id, segment]));
+ const wordEvidence = new Map(narrative.tokens.map((token, index) => [token.id, evidence.words[index]!]));
+ const quality = options.precision === "exact" ? "measured" : "estimated";
+ const anchors: SemanticAnchorPoint[] = narrative.semanticIndex.anchors.map((anchor) => {
+ const segment = segmentEvidence.get(anchor.segmentId);
+ if (!segment) fail("locator_segment_missing", `Alignment is missing Segment "${anchor.segmentId}".`);
+ const word = anchor.tokenId ? wordEvidence.get(anchor.tokenId) : undefined;
+ const seconds = anchor.kind === "segment-start"
+ ? segment.startSec
+ : anchor.kind === "segment-end"
+ ? segment.endSec
+ : anchor.kind === "token-start"
+ ? word?.startSec
+ : word?.endSec;
+ if (seconds === undefined) {
+ fail("locator_anchor_missing", `No alignment time exists for anchor "${anchor.id}".`);
+ }
+ return {
+ identity: anchor.id,
+ point: { basisDigest: basis.basisDigest, frame: frameAt(seconds, basis) },
+ quality,
+ };
+ });
+ const contract = options.precision === "exact"
+ ? "svml.exact-semantic-map.v1" as const
+ : "svml.estimated-semantic-map.v1" as const;
+ const payload = {
+ contract,
+ semanticIndexDigest: narrative.semanticIndex.digest,
+ basisDigest: basis.basisDigest,
+ anchors,
+ evidenceDigests: [sha256(stableJson(evidence))],
+ locatorDigest: options.locatorDigest ?? sha256(`svml.${options.precision}-speech-locator.v1`),
+ quantizationPolicy: "nearest-frame" as const,
+ ...(evidence.captionCues ? { captionCues: evidence.captionCues } : {}),
+ };
+ const map = { ...payload, mapDigest: semanticMapDigest(payload) };
+ validateSemanticMap(narrative, basis, map);
+ return map;
+}
+
+export function validateSemanticMap(
+ narrative: NarrativeIR,
+ basis: ProgramBasis,
+ map: SemanticMap,
+): void {
+ validateProgramBasis(basis);
+ if (
+ map.contract !== "svml.estimated-semantic-map.v1"
+ && map.contract !== "svml.exact-semantic-map.v1"
+ ) {
+ fail("semantic_map_contract", "Unsupported SemanticMap contract.");
+ }
+ if (map.semanticIndexDigest !== narrative.semanticIndex.digest) {
+ fail("semantic_map_index", "SemanticMap was produced for a different SemanticIndex.");
+ }
+ if (map.basisDigest !== basis.basisDigest) {
+ fail("semantic_map_basis", "SemanticMap was produced for a different ProgramBasis.");
+ }
+ const { mapDigest, ...canonicalMap } = map;
+ if (semanticMapDigest(canonicalMap) !== mapDigest) {
+ fail("semantic_map_digest", "SemanticMap digest does not match its canonical contents.");
+ }
+ if (map.anchors.length !== narrative.semanticIndex.anchors.length) {
+ fail(
+ "semantic_map_cardinality",
+ `SemanticMap has ${map.anchors.length} anchors; expected ${narrative.semanticIndex.anchors.length}.`,
+ );
+ }
+ const pointById = new Map<string, number>();
+ for (const [index, expected] of narrative.semanticIndex.anchors.entries()) {
+ const actual = map.anchors[index];
+ if (!actual || actual.identity !== expected.id) {
+ fail(
+ "semantic_map_identity",
+ `SemanticMap anchor ${index + 1} must be "${expected.id}".`,
+ );
+ }
+ integer(actual.point.frame, `SemanticMap point ${actual.identity}`);
+ if (
+ actual.point.basisDigest !== basis.basisDigest
+ || actual.point.frame < 0
+ || actual.point.frame > basis.durationFrames
+ ) {
+ fail("semantic_map_point", `SemanticMap anchor "${actual.identity}" is outside ProgramBasis.`);
+ }
+ if (map.contract === "svml.exact-semantic-map.v1" && actual.quality === "estimated") {
+ fail("semantic_map_precision", `ExactSemanticMap anchor "${actual.identity}" is estimated.`);
+ }
+ if (pointById.has(actual.identity)) {
+ fail("semantic_map_duplicate", `SemanticMap repeats anchor "${actual.identity}".`);
+ }
+ pointById.set(actual.identity, actual.point.frame);
+ }
+ for (const segment of narrative.segments) {
+ const ordered = [
+ segment.startAnchorId,
+ ...narrative.tokens.slice(segment.tokenStart, segment.tokenEnd)
+ .flatMap((token) => [token.startAnchorId, token.endAnchorId]),
+ segment.endAnchorId,
+ ];
+ for (let index = 1; index < ordered.length; index += 1) {
+ if (pointById.get(ordered[index]!)! < pointById.get(ordered[index - 1]!)!) {
+ fail("semantic_map_segment_order", `SemanticMap runs backwards inside Segment "${segment.id}".`);
+ }
+ }
+ }
+ for (let index = 1; index < narrative.segments.length; index += 1) {
+ const previous = narrative.segments[index - 1]!;
+ const current = narrative.segments[index]!;
+ if (
+ pointById.get(current.startAnchorId)! < pointById.get(previous.startAnchorId)!
+ || pointById.get(current.endAnchorId)! < pointById.get(previous.endAnchorId)!
+ ) {
+ fail("semantic_map_source_order", `SemanticMap reorders Segment "${current.id}".`);
+ }
+ }
+}
+
+export function semanticPoints(map: SemanticMap): ReadonlyMap<string, number> {
+ return new Map(map.anchors.map((anchor) => [anchor.identity, anchor.point.frame]));
+}
diff --git a/stdlib/broll-track.svk b/stdlib/broll-track.svk
index 18995248..f15c0bf8 100644
--- a/stdlib/broll-track.svk
+++ b/stdlib/broll-track.svk
@@ -4,7 +4,7 @@
<port name="track" direction="output" type="Track" cardinality="one"/>
<child name="item" cardinality="one-or-more">
<field name="id" type="id" required="true"/>
- <field name="source" type="Image|Video" required="true"/>
+ <field name="source" type="Image|Video|ProgramBoundImage|ProgramBoundVideo" required="true"/>
<field name="during" type="SelectionSet|ProgramSpan" consume="each"/>
<field name="window" type="string"/>
<field name="playback" type="string"/>
@@ -20,22 +20,23 @@
<field name="height" type="number"/>
<field name="fit" type="string"/>
<field name="position" type="string"/>
- <field name="shape" type="string"/>
+ <field name="shape" style="mask" type="string"/>
<field name="radius" type="number"/>
<field name="opacity" type="number"/>
<field name="background" type="string"/>
+ <field name="z" type="number"/>
<child name="present" cardinality="zero-or-more">
<field name="id" type="id" required="true"/>
<field name="during" type="SelectionSet|ProgramSpan" consume="each"/>
<field name="window" type="string"/>
- <field name="layer" type="number"/>
+ <field name="z" type="number"/>
<field name="x" type="number"/>
<field name="y" type="number"/>
<field name="width" type="number"/>
<field name="height" type="number"/>
<field name="fit" type="string"/>
<field name="position" type="string"/>
- <field name="shape" type="string"/>
+ <field name="shape" style="mask" type="string"/>
<field name="radius" type="number"/>
<field name="opacity" type="number"/>
<field name="background" type="string"/>
diff --git a/stdlib/caption-track.svk b/stdlib/caption-track.svk
index a077aa14..d96c40ea 100644
--- a/stdlib/caption-track.svk
+++ b/stdlib/caption-track.svk
@@ -16,5 +16,15 @@
<param name="strokeColor" style="stroke-color" type="string" default="rgba(0,0,0,.82)"/>
<param name="shadow" type="string" default="0 2px 9px rgba(0,0,0,.5), 0 0 2px rgba(0,0,0,.68)"/>
<port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
+ <port name="semantic" direction="input" type="ExactSemanticMap" cardinality="one"/>
<port name="track" direction="output" type="Track" cardinality="one"/>
+ <child name="style" cardinality="zero-or-more">
+ <field name="id" type="id" required="true"/>
+ <field name="during" type="SelectionSet|ProgramSpan" consume="set" required="true"/>
+ <field name="color" style="fill" type="string"/>
+ <field name="activeColor" style="active-fill" type="string"/>
+ <field name="background" type="string"/>
+ <field name="weight" type="number"/>
+ <field name="scale" type="number"/>
+ </child>
</kernel>
diff --git a/stdlib/film.svk b/stdlib/film.svk
index d2ac17f2..5c91952d 100644
--- a/stdlib/film.svk
+++ b/stdlib/film.svk
@@ -3,7 +3,8 @@
<param name="height" type="number"/>
<param name="fps" style="frame-rate" type="number"/>
<param name="background" type="string" default="#000000"/>
- <port name="speech" direction="input" type="SpeechSpine" cardinality="one"/>
+ <port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
+ <port name="semantic" direction="input" type="ExactSemanticMap" cardinality="one"/>
<port name="track" direction="input" type="Track" cardinality="zero-or-more"/>
<port name="document" direction="output" type="HyperframesDocument" cardinality="one"/>
<child name="track" cardinality="zero-or-more">
diff --git a/stdlib/media-basis.svk b/stdlib/media-basis.svk
new file mode 100644
index 00000000..3a415a82
--- /dev/null
+++ b/stdlib/media-basis.svk
@@ -0,0 +1,9 @@
+<kernel name="media-basis" abi="1" implementation="./runtime/media-basis.mjs">
+ <param name="fps" type="number" default="30"/>
+ <param name="duration" type="duration"/>
+ <param name="sourceStart" type="duration" default="0ms"/>
+ <port name="audio" direction="input" type="Audio" cardinality="one"/>
+ <port name="visual" direction="input" type="Video" cardinality="zero-or-one"/>
+ <port name="production" direction="output" type="TemporalBasisProduction" cardinality="one"/>
+ <port name="facets.visual" direction="output" type="ProgramBoundVideo" cardinality="zero-or-one"/>
+</kernel>
diff --git a/stdlib/media-track.svk b/stdlib/media-track.svk
index 13fdeced..b32a5a17 100644
--- a/stdlib/media-track.svk
+++ b/stdlib/media-track.svk
@@ -4,7 +4,7 @@
<port name="track" direction="output" type="Track" cardinality="one"/>
<child name="item" cardinality="one-or-more">
<field name="id" type="id" required="true"/>
- <field name="source" type="Image|Video" required="true"/>
+ <field name="source" type="Image|Video|ProgramBoundImage|ProgramBoundVideo|ProgramBoundVideoSequence" required="true"/>
<field name="during" type="SelectionSet|ProgramSpan" consume="each"/>
<field name="window" type="string"/>
<field name="playback" type="string"/>
@@ -16,22 +16,23 @@
<field name="height" type="number"/>
<field name="fit" type="string"/>
<field name="position" type="string"/>
- <field name="shape" type="string"/>
+ <field name="shape" style="mask" type="string"/>
<field name="radius" type="number"/>
<field name="opacity" type="number"/>
<field name="background" type="string"/>
+ <field name="z" type="number"/>
<child name="present" cardinality="zero-or-more">
<field name="id" type="id" required="true"/>
<field name="during" type="SelectionSet|ProgramSpan" consume="each"/>
<field name="window" type="string"/>
- <field name="layer" type="number"/>
+ <field name="z" type="number"/>
<field name="x" type="number"/>
<field name="y" type="number"/>
<field name="width" type="number"/>
<field name="height" type="number"/>
<field name="fit" type="string"/>
<field name="position" type="string"/>
- <field name="shape" type="string"/>
+ <field name="shape" style="mask" type="string"/>
<field name="radius" type="number"/>
<field name="opacity" type="number"/>
<field name="background" type="string"/>
diff --git a/stdlib/ranking-column.svk b/stdlib/ranking-column.svk
index 84f66d08..dc528c42 100644
--- a/stdlib/ranking-column.svk
+++ b/stdlib/ranking-column.svk
@@ -12,6 +12,7 @@
<param name="moveGain" style="move-gain" type="number" default="0.1"/>
<port name="appearSound" direction="input" type="Audio" cardinality="zero-or-one"/>
<port name="moveSound" direction="input" type="Audio" cardinality="zero-or-one"/>
+ <port name="at" direction="input" type="MomentSet|ProgramPoint" cardinality="zero-or-one" consume="one"/>
<port name="item" direction="input" type="RankingItem" cardinality="one-or-more"/>
<port name="track" direction="output" type="Track" cardinality="one"/>
<child name="item" cardinality="one-or-more">
diff --git a/stdlib/runtime/audio-track.mjs b/stdlib/runtime/audio-track.mjs
index 77339779..565e4e74 100644
--- a/stdlib/runtime/audio-track.mjs
+++ b/stdlib/runtime/audio-track.mjs
@@ -24,7 +24,7 @@ export default {
return {
startFrame,
endFrameExclusive: Math.min(
- context.located.durationFrames,
+ context.program.durationFrames,
startFrame + durationFrames,
),
};
diff --git a/stdlib/runtime/broll-track.mjs b/stdlib/runtime/broll-track.mjs
index 9c2ed5a5..4e2c50dc 100644
--- a/stdlib/runtime/broll-track.mjs
+++ b/stdlib/runtime/broll-track.mjs
@@ -34,7 +34,7 @@ export default {
if (String(source.type).toLowerCase() !== expectedKind.toLowerCase()) {
throw new Error(`broll item "${id}" expected ${expectedKind}, received ${source.type}`);
}
- const presentations = resolveItemPresentations(context, item, source);
+ const presentations = resolveItemPresentations(context, item, source, z);
for (const [index, presentation] of presentations.entries()) {
const range = presentation.range;
const zoom = numberAttr(item, "zoom", 1);
@@ -55,9 +55,10 @@ export default {
source: source.source,
startFrame: range.startFrame,
endFrameExclusive: range.endFrameExclusive,
- z,
- layer: presentation.layer,
- muted: !(item.attributes.audio === true || item.attributes.audio === "source"),
+ z: presentation.z,
+ // HyperFrames receives source audio as an explicit sibling contribution.
+ // Keep the visual element muted so one source can never play twice.
+ muted: true,
...timing,
style: {
...presentationStyle(presentation.element, item, { fit: "contain" }),
@@ -83,12 +84,14 @@ export default {
presentation.parentRange,
]));
for (const [parentIndex, range] of parents) {
+ const timing = mediaTiming(item, { parentRange: range, range }, source, context.fps);
audios.push({
id: `${context.instance.id}:${id}:${parentIndex}:audio`,
source: source.source,
startFrame: range.startFrame,
endFrameExclusive: range.endFrameExclusive,
- mediaStartSec: numberAttr(item, "mediaStart", 0),
+ mediaStartSec: timing.mediaStartSec,
+ playbackRate: timing.playbackRate,
volume: numberAttr(item, "volume", 1),
bus: "source",
});
@@ -122,7 +125,7 @@ export default {
const cue = incoming.startFrame;
const startFrame = cue - Math.floor(durationFrames * cueRatio);
const endFrameExclusive = startFrame + durationFrames;
- if (startFrame < 0 || endFrameExclusive > context.located.durationFrames) {
+ if (startFrame < 0 || endFrameExclusive > context.program.durationFrames) {
throw new Error(`broll transition ${from} -> ${to} leaves the master timeline`);
}
if (occupiedWindows.some((window) =>
diff --git a/stdlib/runtime/caption-track.mjs b/stdlib/runtime/caption-track.mjs
index 85c9d95b..20658cc1 100644
--- a/stdlib/runtime/caption-track.mjs
+++ b/stdlib/runtime/caption-track.mjs
@@ -1,13 +1,76 @@
-import { html, numberAttr, stringAttr, trackOutput } from "./helpers.mjs";
+import { elements, html, numberAttr, stringAttr, trackOutput } from "./helpers.mjs";
export default {
abiVersion: "1",
project(context) {
const script = context.resolve(context.element.attributes.script);
- const located = script.located;
+ const semantic = context.resolve(context.element.attributes.semantic);
+ if (semantic?.contract !== "svml.exact-semantic-map.v1") {
+ throw new Error("caption-track requires ExactSemanticMap");
+ }
+ if (semantic.basisDigest !== context.program.basisDigest) {
+ throw new Error("caption-track SemanticMap does not match the selected ProgramBasis");
+ }
+ const points = new Map(semantic.anchors.map((anchor) => [anchor.identity, anchor.point.frame]));
+ const frameRange = (startFrame, endFrameExclusive) => ({
+ startFrame,
+ endFrameExclusive,
+ startSec: startFrame / context.fps,
+ endSec: endFrameExclusive / context.fps,
+ });
+ const locatedWords = script.tokens.map((token) => ({
+ ...token,
+ ...frameRange(points.get(token.startAnchorId), points.get(token.endAnchorId)),
+ }));
+ const captionAtoms = script.captionAtoms.map((atom) => ({
+ ...atom,
+ ...frameRange(
+ locatedWords[atom.startWord].startFrame,
+ locatedWords[atom.endWordExclusive - 1].endFrameExclusive,
+ ),
+ }));
+ const captionCues = semantic.captionCues?.map((cue, index) => ({
+ id: cue.id ?? `cue-${index + 1}`,
+ ...cue,
+ ...frameRange(
+ locatedWords[cue.startWord].startFrame,
+ locatedWords[cue.endWordExclusive - 1].endFrameExclusive,
+ ),
+ }));
+ const located = {
+ words: locatedWords,
+ captionAtoms,
+ captionCues,
+ durationFrames: context.program.durationFrames,
+ };
const z = numberAttr(context.element, "z");
const maxWords = Math.max(1, Math.round(numberAttr(context.element, "maxWords", 3)));
const words = located.words;
+ const styledRanges = elements(context, "style").map((style) => ({
+ id: stringAttr(style, "id"),
+ ranges: context.selection(style.attributes.during, "style.during"),
+ color: style.attributes.color,
+ activeColor: style.attributes.activeColor,
+ background: style.attributes.background,
+ weight: style.attributes.weight,
+ scale: style.attributes.scale,
+ }));
+ const inlineStyle = (word, active) => {
+ const declarations = {};
+ for (const rule of styledRanges) {
+ if (!rule.ranges.some((range) =>
+ word.startFrame < range.endFrameExclusive
+ && word.endFrameExclusive > range.startFrame)) continue;
+ if (typeof rule.color === "string") declarations.color = rule.color;
+ if (active && typeof rule.activeColor === "string") declarations.color = rule.activeColor;
+ if (typeof rule.background === "string") declarations.background = rule.background;
+ if (typeof rule.weight === "number") declarations["font-weight"] = rule.weight;
+ if (typeof rule.scale === "number") declarations.transform = `scale(${rule.scale})`;
+ }
+ const body = Object.entries(declarations)
+ .map(([name, value]) => `${name}:${String(value)}`).join(";");
+ return body ? ` style="${html(body)}"` : "";
+ };
const atomsByStart = new Map(located.captionAtoms.map((atom) => [atom.startWord, atom]));
const displayUnits = (startWord, endWordExclusive) => {
const units = [];
@@ -58,8 +121,11 @@ export default {
nextCueStart ?? located.durationFrames,
word.endFrameExclusive + Math.round(numberAttr(context.element, "hold", 0.08) * context.fps),
);
- const body = group.map((item, index) =>
- `<span class="${index === activeIndex ? "active" : ""}">${html(item.text)}</span>`).join("");
+ if (endFrameExclusive <= word.startFrame) continue;
+ const body = group.map((item, index) => {
+ const active = index === activeIndex;
+ return `<span class="${active ? "active" : ""}"${inlineStyle(item, active)}>${html(item.text)}</span>`;
+ }).join("");
const fragmentDuration = Math.max(
1 / context.fps,
(endFrameExclusive - word.startFrame) / context.fps,
diff --git a/stdlib/runtime/film.mjs b/stdlib/runtime/film.mjs
index 80187e75..0aa411bb 100644
--- a/stdlib/runtime/film.mjs
+++ b/stdlib/runtime/film.mjs
@@ -6,16 +6,33 @@ export default {
const visuals = [];
const audios = [];
const styles = [];
- const speech = resolveAttr(context, context.element, "speech");
- const speechTrack = speech.track ?? speech.speech?.track;
- if (speechTrack) {
- visuals.push(...(speechTrack.visuals ?? []));
- audios.push(...(speechTrack.audios ?? []));
- styles.push(...(speechTrack.styles ?? []));
- } else if (speech.speech?.primaryAudio) {
- audios.push(...speech.speech.primaryAudio);
- } else if (speech.primaryAudio) {
- audios.push(...speech.primaryAudio);
+ const production = resolveAttr(context, context.element, "basis");
+ const semantic = resolveAttr(context, context.element, "semantic");
+ if (production?.contract !== "svml.temporal-basis-production.v1") {
+ throw new Error("film requires TemporalBasisProduction");
+ }
+ if (semantic?.contract !== "svml.exact-semantic-map.v1") {
+ throw new Error("film requires ExactSemanticMap");
+ }
+ if (semantic.basisDigest !== production.basis.basisDigest) {
+ throw new Error("film basis and SemanticMap do not share a basisDigest");
+ }
+ const basisFps = production.basis.frameRate.numerator
+ / production.basis.frameRate.denominator;
+ for (const contribution of production.audio ?? []) {
+ audios.push({
+ id: contribution.id,
+ source: contribution.source,
+ startFrame: contribution.programStartFrame,
+ endFrameExclusive: contribution.programEndFrameExclusive,
+ mediaStartSec: contribution.sourceStartFrame / basisFps,
+ playbackRate: (contribution.sourceEndFrameExclusive - contribution.sourceStartFrame)
+ / Math.max(1, contribution.programEndFrameExclusive - contribution.programStartFrame),
+ volume: contribution.gain,
+ ...(contribution.fadeInFrames ? { fadeInSec: contribution.fadeInFrames / basisFps } : {}),
+ ...(contribution.fadeOutFrames ? { fadeOutSec: contribution.fadeOutFrames / basisFps } : {}),
+ bus: "speech",
+ });
}
for (const trackElement of elements(context, "track")) {
const track = resolveAttr(context, trackElement, "ref");
@@ -23,9 +40,9 @@ export default {
audios.push(...(track.audios ?? []));
styles.push(...(track.styles ?? []));
}
- const fps = numberAttr(context.element, "fps", context.fps);
- if (fps !== context.fps) {
- throw new Error(`film frame-rate ${fps} does not match Located clock ${context.fps}`);
+ const fps = numberAttr(context.element, "fps", basisFps);
+ if (fps !== basisFps) {
+ throw new Error(`film frame-rate ${fps} does not match ProgramBasis ${basisFps}`);
}
return {
outputs: {
@@ -35,7 +52,7 @@ export default {
width: numberAttr(context.element, "width", context.width),
height: numberAttr(context.element, "height", context.height),
fps,
- durationFrames: context.located.durationFrames,
+ durationFrames: production.basis.durationFrames,
background: stringAttr(context.element, "background", "#000000"),
visuals,
audios,
diff --git a/stdlib/runtime/helpers.mjs b/stdlib/runtime/helpers.mjs
index 6b44645e..9626e99b 100644
--- a/stdlib/runtime/helpers.mjs
+++ b/stdlib/runtime/helpers.mjs
@@ -30,13 +30,24 @@ export function resolveAttr(context, element, name) {
export function material(value, expected) {
if (!value || typeof value !== "object") throw new Error(`expected ${expected ?? "media"} material`);
- const type = value.type ?? value.kind;
- if (expected && type && String(type).toLowerCase() !== expected.toLowerCase()) {
- throw new Error(`expected ${expected}, received ${String(type)}`);
+ const declaredType = String(value.type ?? value.kind ?? "");
+ const programBound = /^ProgramBound(?:Image|Video|Audio)$/u.test(declaredType);
+ const type = programBound ? declaredType.slice("ProgramBound".length) : declaredType;
+ if (expected && type && type.toLowerCase() !== expected.toLowerCase()) {
+ throw new Error(`expected ${expected}, received ${declaredType}`);
}
const source = value.absoluteSource ?? value.source;
if (typeof source !== "string") throw new Error(`${expected ?? "media"} material has no source`);
- return { ...value, source };
+ return { ...value, type, declaredType, programBound, source };
+}
+
+export function mediaMaterials(value, expected) {
+ if (value?.type === "ProgramBoundVideoSequence") {
+ const segments = Object.values(value.segments ?? {}).map((segment) => material(segment, expected));
+ if (!segments.length) throw new Error("ProgramBoundVideoSequence has no media segments");
+ return segments;
+ }
+ return [material(value, expected)];
}
function unitSeconds(raw) {
@@ -219,7 +230,7 @@ export function presentationAnimation(element, range, fps, id) {
};
}
-export function resolveItemPresentations(context, item, source) {
+export function resolveItemPresentations(context, item, source, defaultZ, options = {}) {
const rawParents = item.attributes.during
? context.selection(item.attributes.during, "item.during")
: source.range
@@ -228,8 +239,10 @@ export function resolveItemPresentations(context, item, source) {
if (!rawParents.length) {
throw new Error(`item "${stringAttr(item, "id")}" has no temporal range`);
}
- const parents = rawParents.map((range) =>
- projectWindow(range, item.attributes.window, context.fps));
+ const parents = rawParents
+ .map((range) => projectWindow(range, item.attributes.window, context.fps))
+ .map((range) => source.range ? intersectRange(range, source.range) : range)
+ .filter(Boolean);
const presents = elements({ element: item }, "present");
const explicit = [];
for (const [presentIndex, present] of presents.entries()) {
@@ -247,11 +260,14 @@ export function resolveItemPresentations(context, item, source) {
parentRange,
range,
element: present,
- layer: numberAttr(present, "layer", presentIndex + 1),
+ z: numberAttr(present, "z", numberAttr(item, "z", defaultZ)),
explicit: true,
});
}
}
+ if (!options.allowEmptyPresent && !explicit.some((value) => value.id === stringAttr(present, "id"))) {
+ throw new Error(`presentation_empty: Present "${stringAttr(present, "id")}" has no intersection with its Item`);
+ }
}
const defaults = [];
for (const [parentIndex, parentRange] of parents.entries()) {
@@ -268,18 +284,43 @@ export function resolveItemPresentations(context, item, source) {
parentRange,
range,
element: undefined,
- layer: 0,
+ z: numberAttr(item, "z", defaultZ),
explicit: false,
});
}
}
+ for (let left = 0; left < explicit.length; left += 1) {
+ for (let right = left + 1; right < explicit.length; right += 1) {
+ const a = explicit[left];
+ const b = explicit[right];
+ if (
+ a.z === b.z
+ && a.range.startFrame < b.range.endFrameExclusive
+ && b.range.startFrame < a.range.endFrameExclusive
+ ) {
+ throw new Error(`presentation_z_ambiguous: Present "${a.id}" and "${b.id}" overlap at z=${a.z}`);
+ }
+ }
+ }
return [...defaults, ...explicit];
}
export function mediaTiming(item, presentation, source, fps) {
- const playback = stringAttr(item, "playback", "sync");
- const offsetFrames = presentation.range.startFrame - presentation.parentRange.startFrame;
- const mediaStart = numberAttr(item, "mediaStart", 0);
+ const playback = stringAttr(item, "playback", source.programBound ? "program-map" : "sync");
+ const mappingRange = source.programBound && source.range
+ ? source.range
+ : presentation.parentRange;
+ const offsetFrames = presentation.range.startFrame - mappingRange.startFrame;
+ const mediaStart = numberAttr(item, "mediaStart", source.mediaStartSec ?? 0);
+ if (playback === "program-map") {
+ if (!source.programBound) {
+ throw new Error(`playback "program-map" requires ProgramBound media`);
+ }
+ return {
+ mediaStartSec: mediaStart + offsetFrames / fps * (source.playbackRate ?? 1),
+ playbackRate: source.playbackRate ?? 1,
+ };
+ }
if (playback === "sync") {
return {
mediaStartSec: mediaStart + offsetFrames / fps,
@@ -307,3 +348,29 @@ export function mediaTiming(item, presentation, source, fps) {
if (playback === "hold" && source.type === "Image") return {};
throw new Error(`unsupported playback "${playback}" for ${source.type}`);
}
+
+export function programBoundAnimation(source, range, fps, id) {
+ if (!source.programBound || (!source.fadeInFrames && !source.fadeOutFrames)) return {};
+ const totalFrames = Math.max(1, range.endFrameExclusive - range.startFrame);
+ const sourceStart = source.range.startFrame;
+ const sourceEnd = source.range.endFrameExclusive;
+ const fadeInEnd = sourceStart + (source.fadeInFrames ?? 0);
+ const fadeOutStart = sourceEnd - (source.fadeOutFrames ?? 0);
+ const points = new Map([[0, 1], [100, 1]]);
+ const percent = (frameValue) => Math.max(0, Math.min(100,
+ (frameValue - range.startFrame) / totalFrames * 100));
+ if (source.fadeInFrames && range.startFrame < fadeInEnd) {
+ points.set(0, Math.max(0, (range.startFrame - sourceStart) / source.fadeInFrames));
+ points.set(percent(Math.min(range.endFrameExclusive, fadeInEnd)), 1);
+ }
+ if (source.fadeOutFrames && range.endFrameExclusive > fadeOutStart) {
+ points.set(percent(Math.max(range.startFrame, fadeOutStart)), 1);
+ points.set(100, Math.max(0, (sourceEnd - range.endFrameExclusive) / source.fadeOutFrames));
+ }
+ const name = `svml-program-bound-${id}`.replaceAll(/[^A-Za-z0-9_-]/gu, "-");
+ return {
+ animation: `${name} ${totalFrames / fps}s linear both`,
+ css: `@keyframes ${name} { ${[...points].sort((a, b) => a[0] - b[0])
+ .map(([at, opacity]) => `${at}% { opacity:${opacity}; }`).join(" ")} }`,
+ };
+}
diff --git a/stdlib/runtime/media-basis.mjs b/stdlib/runtime/media-basis.mjs
new file mode 100644
index 00000000..369bf3b0
--- /dev/null
+++ b/stdlib/runtime/media-basis.mjs
@@ -0,0 +1,133 @@
+import { material, numberAttr, resolveAttr, stringAttr } from "./helpers.mjs";
+
+function seconds(value) {
+ const match = /^(\d+(?:\.\d+)?)(ms|s)$/u.exec(String(value).trim());
+ if (!match) throw new Error(`invalid duration "${String(value)}"`);
+ return Number(match[1]) * (match[2] === "ms" ? 0.001 : 1);
+}
+
+function canonicalFacet(value) {
+ if (Array.isArray(value)) return value.map(canonicalFacet);
+ if (!value || typeof value !== "object") return value;
+ return Object.fromEntries(Object.entries(value)
+ .filter(([key]) => !["source", "absoluteSource", "basisDigest"].includes(key))
+ .map(([key, nested]) => [key, canonicalFacet(nested)]));
+}
+
+function frameRate(fps) {
+ if (!Number.isFinite(fps) || fps <= 0) throw new Error("fps must be positive and finite");
+ if (Number.isInteger(fps)) return { numerator: fps, denominator: 1 };
+ let numerator = Math.round(fps * 1_000_000);
+ let denominator = 1_000_000;
+ let left = numerator;
+ let right = denominator;
+ while (right) [left, right] = [right, left % right];
+ numerator /= left;
+ denominator /= left;
+ return { numerator, denominator };
+}
+
+export default {
+ abiVersion: "1",
+ project(context) {
+ const fps = numberAttr(context.element, "fps", 30);
+ const rationalFps = frameRate(fps);
+ const durationSec = seconds(stringAttr(context.element, "duration"));
+ const sourceStartSec = seconds(stringAttr(context.element, "sourceStart", "0ms"));
+ const durationFrames = Math.round(durationSec * fps);
+ if (durationFrames <= 0) throw new Error("media-basis duration must resolve to positive frames");
+ const sourceStartFrame = Math.round(sourceStartSec * fps);
+ const sourceEndFrameExclusive = sourceStartFrame + durationFrames;
+ const audioValue = material(resolveAttr(context, context.element, "audio"), "audio");
+ const visualValue = context.element.attributes.visual
+ ? material(resolveAttr(context, context.element, "visual"), "video")
+ : undefined;
+ const sourceMaps = [{
+ id: "program:audio-map",
+ sourceId: audioValue.id,
+ sourceDigest: audioValue.contentDigest,
+ sourceStartFrame,
+ sourceEndFrameExclusive,
+ programStartFrame: 0,
+ programEndFrameExclusive: durationFrames,
+ }];
+ if (visualValue) {
+ sourceMaps.push({
+ id: "program:visual-map",
+ sourceId: visualValue.id,
+ sourceDigest: visualValue.contentDigest,
+ sourceStartFrame,
+ sourceEndFrameExclusive,
+ programStartFrame: 0,
+ programEndFrameExclusive: durationFrames,
+ });
+ }
+ const alignmentSubjects = [{
+ id: "program:audio",
+ sourceId: audioValue.id,
+ sourceDigest: audioValue.contentDigest,
+ sourceMapId: "program:audio-map",
+ }];
+ const audio = [{
+ id: "program:audio",
+ sourceId: audioValue.id,
+ sourceDigest: audioValue.contentDigest,
+ source: audioValue.source,
+ sourceMapId: "program:audio-map",
+ sourceStartFrame,
+ sourceEndFrameExclusive,
+ programStartFrame: 0,
+ programEndFrameExclusive: durationFrames,
+ gain: 1,
+ }];
+ const unboundVisual = visualValue ? {
+ type: "ProgramBoundVideo",
+ source: visualValue.source,
+ sourceId: visualValue.id,
+ sourceDigest: visualValue.contentDigest,
+ range: {
+ startFrame: 0,
+ endFrameExclusive: durationFrames,
+ startSec: 0,
+ endSec: durationFrames / fps,
+ },
+ mediaStartSec: sourceStartSec,
+ playbackRate: 1,
+ sourceMapId: "program:visual-map",
+ } : undefined;
+ const outcome = {
+ contract: "svml.program-basis-outcome.v1",
+ frameRate: rationalFps,
+ originFrame: 0,
+ durationFrames,
+ alignmentSubjects,
+ sourceMaps,
+ audio: audio.map(({ source: _source, ...contribution }) => contribution),
+ facets: canonicalFacet({ ...(unboundVisual ? { visual: unboundVisual } : {}) }),
+ };
+ const basisDigest = context.digest(outcome);
+ const basis = {
+ contract: "svml.program-basis.v1",
+ frameRate: outcome.frameRate,
+ originFrame: 0,
+ durationFrames,
+ basisDigest,
+ };
+ const facets = unboundVisual
+ ? { visual: { ...unboundVisual, basisDigest } }
+ : {};
+ const production = {
+ contract: "svml.temporal-basis-production.v1",
+ basis,
+ alignmentSubjects,
+ sourceMaps,
+ audio,
+ facets,
+ productionDigest: context.digest({
+ implementation: context.instance.executionDigest,
+ outcome,
+ }),
+ };
+ return { outputs: { production, facets } };
+ },
+};
diff --git a/stdlib/runtime/media-track.mjs b/stdlib/runtime/media-track.mjs
index f50140e4..ad28359d 100644
--- a/stdlib/runtime/media-track.mjs
+++ b/stdlib/runtime/media-track.mjs
@@ -1,9 +1,10 @@
import {
elements,
- material,
+ mediaMaterials,
mediaTiming,
numberAttr,
presentationAnimation,
+ programBoundAnimation,
presentationStyle,
resolveItemPresentations,
resolveAttr,
@@ -18,9 +19,13 @@ export default {
const visuals = [];
for (const item of elements(context, "item")) {
const id = stringAttr(item, "id");
- const source = material(resolveAttr(context, item, "source"));
- const presentations = resolveItemPresentations(context, item, source);
+ const sources = mediaMaterials(resolveAttr(context, item, "source"));
+ const itemVisuals = [];
+ const resolvedPresentIds = new Set();
+ for (const [sourceIndex, source] of sources.entries()) {
+ const presentations = resolveItemPresentations(context, item, source, z, { allowEmptyPresent: true });
for (const [index, presentation] of presentations.entries()) {
+ if (presentation.explicit) resolvedPresentIds.add(presentation.id);
const range = presentation.range;
const animation = presentationAnimation(
presentation.element,
@@ -28,26 +33,46 @@ export default {
context.fps,
`${context.instance.id}-${id}-${presentation.id}-${index}`,
);
+ const sourceAnimation = programBoundAnimation(
+ source,
+ range,
+ context.fps,
+ `${context.instance.id}-${id}-${sourceIndex}-${presentation.id}-${index}`,
+ );
const timing = source.type === "Video"
? mediaTiming(item, presentation, source, context.fps)
: {};
- visuals.push({
- id: `${context.instance.id}:${id}:${index}`,
+ itemVisuals.push({
+ id: `${context.instance.id}:${id}:${sourceIndex}:${index}`,
kind: source.type === "Image" ? "image" : "video",
source: source.source,
startFrame: range.startFrame,
endFrameExclusive: range.endFrameExclusive,
- z,
- layer: presentation.layer,
+ z: presentation.z,
muted: true,
...timing,
style: {
...presentationStyle(presentation.element, item),
- ...(animation.animation ? { animation: animation.animation } : {}),
+ ...((animation.animation || sourceAnimation.animation) ? {
+ animation: [animation.animation, sourceAnimation.animation].filter(Boolean).join(", "),
+ } : {}),
},
- ...(animation.css ? { css: animation.css } : {}),
+ ...((animation.css || sourceAnimation.css) ? {
+ css: [animation.css, sourceAnimation.css].filter(Boolean).join("\n"),
+ } : {}),
});
}
+ }
+ if (!itemVisuals.length) {
+ throw new Error(`item_empty: Item "${id}" has no intersection with its source`);
+ }
+ for (const present of elements({ element: item }, "present")) {
+ const presentId = stringAttr(present, "id");
+ if (!resolvedPresentIds.has(presentId)) {
+ throw new Error(`presentation_empty: Present "${presentId}" has no intersection with its Item`);
+ }
+ }
+ visuals.push(...itemVisuals);
}
return trackOutput(visuals);
},
diff --git a/stdlib/runtime/ranking-column.mjs b/stdlib/runtime/ranking-column.mjs
index 8117ff54..8b4fe5eb 100644
--- a/stdlib/runtime/ranking-column.mjs
+++ b/stdlib/runtime/ranking-column.mjs
@@ -141,6 +141,9 @@ export default {
const stageSize = Math.round(context.width * numberAttr(context.element, "stageSize", 0.33));
const stageCx = Math.round(context.width * numberAttr(context.element, "stageX", 0.66));
const stageCy = Math.round(context.height * numberAttr(context.element, "stageY", 0.73));
+ const boardStartFrame = context.element.attributes.at
+ ? context.moment(context.element.attributes.at, "at")[0]
+ : 0;
const cellLeft = x + pad + rowH + Math.max(8, Math.round(rowH * 0.16));
const rows = Array.from({ length: total }, (_, index) => {
const top = pad + index * (rowH + gap);
@@ -150,8 +153,8 @@ export default {
const visuals = [{
id: `${context.instance.id}:board`,
kind: "html",
- startFrame: 0,
- endFrameExclusive: context.located.durationFrames,
+ startFrame: boardStartFrame,
+ endFrameExclusive: context.program.durationFrames,
z,
html: rows,
style: {
@@ -178,15 +181,20 @@ export default {
const rank = Math.max(1, Math.min(total, numberAttr(item, "rank")));
const base = context.selection(item.attributes.during, "item.during")[0];
const range = projectWindow(base, item.attributes.window, context.fps);
+ const visibleRange = {
+ ...range,
+ startFrame: Math.max(boardStartFrame, range.startFrame),
+ };
const top = y + pad + (rank - 1) * (rowH + gap);
- const endDuration = (context.located.durationFrames - range.endFrameExclusive) / context.fps;
+ const settledStartFrame = Math.max(boardStartFrame, range.endFrameExclusive);
+ const endDuration = (context.program.durationFrames - settledStartFrame) / context.fps;
if (endDuration > 0) {
visuals.push({
id: `${context.instance.id}:${id}:settled`,
kind: "image",
source: image.source,
- startFrame: range.endFrameExclusive,
- endFrameExclusive: context.located.durationFrames,
+ startFrame: settledStartFrame,
+ endFrameExclusive: context.program.durationFrames,
z: z + 1,
style: {
left: `${cellLeft}px`,
@@ -205,8 +213,9 @@ export default {
},
});
}
+ if (visibleRange.endFrameExclusive <= visibleRange.startFrame) continue;
const keyframes = `svml-rank-${context.instance.id}-${id}`.replaceAll(/[^A-Za-z0-9_-]/gu, "-");
- const durationFrames = Math.max(1, range.endFrameExclusive - range.startFrame);
+ const durationFrames = Math.max(1, visibleRange.endFrameExclusive - visibleRange.startFrame);
const duration = durationFrames / context.fps;
const targetCx = cellLeft + rowH / 2;
const targetCy = top + rowH / 2;
@@ -214,8 +223,8 @@ export default {
id: `${context.instance.id}:${id}:active`,
kind: "image",
source: image.source,
- startFrame: range.startFrame,
- endFrameExclusive: range.endFrameExclusive,
+ startFrame: visibleRange.startFrame,
+ endFrameExclusive: visibleRange.endFrameExclusive,
z: z + 2,
style: {
left: `${stageCx - stageSize / 2}px`,
@@ -254,10 +263,10 @@ ${activeKeyframes({
audios.push({
id: `${context.instance.id}:${id}:appear`,
source: appear.source,
- startFrame: range.startFrame,
+ startFrame: visibleRange.startFrame,
endFrameExclusive: Math.min(
- context.located.durationFrames,
- range.startFrame + Math.round(0.247021 * context.fps),
+ context.program.durationFrames,
+ visibleRange.startFrame + Math.round(0.247021 * context.fps),
),
volume: numberAttr(context.element, "appearGain", 0.09),
bus: "sfx",
@@ -268,8 +277,8 @@ ${activeKeyframes({
audios.push({
id: `${context.instance.id}:${id}:move`,
source: move.source,
- startFrame: Math.max(range.startFrame, range.endFrameExclusive - Math.round(0.55 * context.fps)),
- endFrameExclusive: Math.min(context.located.durationFrames, range.endFrameExclusive - Math.round(0.55 * context.fps) + moveFrames),
+ startFrame: Math.max(visibleRange.startFrame, visibleRange.endFrameExclusive - Math.round(0.55 * context.fps)),
+ endFrameExclusive: Math.min(context.program.durationFrames, visibleRange.endFrameExclusive - Math.round(0.55 * context.fps) + moveFrames),
volume: numberAttr(context.element, "moveGain", 0.1),
bus: "sfx",
});
diff --git a/stdlib/runtime/speech-assemble.mjs b/stdlib/runtime/speech-assemble.mjs
new file mode 100644
index 00000000..7384308b
--- /dev/null
+++ b/stdlib/runtime/speech-assemble.mjs
@@ -0,0 +1,310 @@
+import { elements, material, numberAttr, resolveAttr, stringAttr } from "./helpers.mjs";
+
+function seconds(raw) {
+ const match = /^([+-]?\d+(?:\.\d+)?)(ms|s)$/u.exec(String(raw).trim());
+ if (!match) throw new Error(`invalid duration "${String(raw)}"`);
+ return Number(match[1]) * (match[2] === "ms" ? 0.001 : 1);
+}
+
+function sourceWindow(raw) {
+ const parts = String(raw).split("..");
+ if (parts.length !== 2) throw new Error(`sourceWindow requires start .. end, received "${raw}"`);
+ const startSec = seconds(parts[0]);
+ const endSec = seconds(parts[1]);
+ if (startSec < 0 || endSec <= startSec) throw new Error(`invalid sourceWindow "${raw}"`);
+ return { startSec, endSec, durationSec: endSec - startSec };
+}
+
+function defaultJoin(element) {
+ const kind = stringAttr(element, "defaultJoin", "cut");
+ const duration = seconds(stringAttr(element, "defaultJoinDuration", "0ms"));
+ if (kind === "cut") return { temporal: "cut", durationSec: 0, audio: "cut", visual: "cut" };
+ if (kind === "crossfade") {
+ if (duration <= 0) throw new Error("crossfade defaultJoinDuration must be positive");
+ return { temporal: "overlap", durationSec: duration, audio: "crossfade", visual: "dissolve" };
+ }
+ if (kind === "gap") {
+ if (duration <= 0) throw new Error("gap defaultJoinDuration must be positive");
+ return { temporal: "gap", durationSec: duration, audio: "cut", visual: "cut" };
+ }
+ throw new Error(`unsupported defaultJoin "${kind}"`);
+}
+
+function explicitJoin(element, fallback) {
+ const overlap = element.attributes.overlap;
+ const gap = element.attributes.gap;
+ if (overlap !== undefined && gap !== undefined) {
+ throw new Error(`<join after="${stringAttr(element, "after")}"> cannot declare both overlap and gap`);
+ }
+ const temporal = overlap !== undefined ? "overlap" : gap !== undefined ? "gap" : "cut";
+ const durationSec = overlap !== undefined
+ ? seconds(overlap)
+ : gap !== undefined
+ ? seconds(gap)
+ : 0;
+ if (temporal !== "cut" && durationSec <= 0) throw new Error(`${temporal} duration must be positive`);
+ const audio = stringAttr(element, "audio", temporal === "overlap" ? fallback.audio : "cut");
+ const visual = stringAttr(element, "visual", temporal === "overlap" ? fallback.visual : "cut");
+ if (!["cut", "crossfade"].includes(audio)) throw new Error(`unsupported audio join "${audio}"`);
+ if (!["cut", "dissolve"].includes(visual)) throw new Error(`unsupported visual join "${visual}"`);
+ if (temporal !== "overlap" && (audio === "crossfade" || visual === "dissolve")) {
+ throw new Error(`${audio}/${visual} requires a temporal overlap`);
+ }
+ return {
+ temporal,
+ durationSec,
+ audio,
+ visual,
+ };
+}
+
+function frame(secondsValue, fps) {
+ return Math.round(secondsValue * fps);
+}
+
+function rationalFrameRate(fps) {
+ if (!Number.isFinite(fps) || fps <= 0) throw new Error("fps must be positive and finite");
+ if (Number.isInteger(fps)) return { numerator: fps, denominator: 1 };
+ let numerator = Math.round(fps * 1_000_000);
+ let denominator = 1_000_000;
+ let left = numerator;
+ let right = denominator;
+ while (right) [left, right] = [right, left % right];
+ numerator /= left;
+ denominator /= left;
+ return { numerator, denominator };
+}
+
+function canonicalFacet(value) {
+ if (Array.isArray(value)) return value.map(canonicalFacet);
+ if (!value || typeof value !== "object") return value;
+ return Object.fromEntries(Object.entries(value)
+ .filter(([key]) => !["source", "absoluteSource", "basisDigest"].includes(key))
+ .map(([key, nested]) => [key, canonicalFacet(nested)]));
+}
+
+export default {
+ abiVersion: "1",
+ project(context) {
+ const fps = numberAttr(context.element, "fps", 30);
+ const rationalFps = rationalFrameRate(fps);
+ const segmentElements = elements(context, "segment");
+ const ids = segmentElements.map(segment => stringAttr(segment, "id"));
+ if (new Set(ids).size !== ids.length) throw new Error("speech-assemble Segment ids must be unique");
+ const fallback = defaultJoin(context.element);
+ const overrides = new Map();
+ for (const join of elements(context, "join")) {
+ const after = stringAttr(join, "after");
+ if (overrides.has(after)) throw new Error(`duplicate join after "${after}"`);
+ if (!ids.includes(after) || ids.at(-1) === after) {
+ throw new Error(`join after "${after}" must name a non-final Segment`);
+ }
+ overrides.set(after, explicitJoin(join, fallback));
+ }
+
+ const segments = [];
+ const joins = [];
+ let nextStartSec = 0;
+ for (const [index, element] of segmentElements.entries()) {
+ const id = ids[index];
+ const script = resolveAttr(context, element, "script");
+ if (!script || script.id !== id) throw new Error(`Segment "${id}" does not match its ScriptSegment`);
+ const audio = material(resolveAttr(context, element, "audio"), "audio");
+ const visual = element.attributes.visual
+ ? material(resolveAttr(context, element, "visual"), "video")
+ : undefined;
+ const window = sourceWindow(stringAttr(element, "sourceWindow"));
+ const startSec = nextStartSec;
+ const endSec = startSec + window.durationSec;
+ segments.push({ id, script, audio, visual, window, startSec, endSec });
+ if (index >= segmentElements.length - 1) continue;
+ const join = overrides.get(id) ?? fallback;
+ joins.push({ after: id, ...join });
+ nextStartSec = join.temporal === "overlap"
+ ? endSec - join.durationSec
+ : join.temporal === "gap"
+ ? endSec + join.durationSec
+ : endSec;
+ if (nextStartSec < startSec) throw new Error(`join after "${id}" overlaps more than the Segment duration`);
+ }
+
+ const durationFrames = Math.max(...segments.map(segment => frame(segment.endSec, fps)));
+ const frameRate = rationalFps;
+ const sourceMaps = [];
+ const alignmentSubjects = [];
+ const audio = [];
+ const unboundSegmentOutputs = {};
+ for (const [index, segment] of segments.entries()) {
+ const programStartFrame = frame(segment.startSec, fps);
+ const programEndFrameExclusive = frame(segment.endSec, fps);
+ const sourceStartFrame = frame(segment.window.startSec, fps);
+ const sourceEndFrameExclusive = frame(segment.window.endSec, fps);
+ const previousJoin = joins[index - 1];
+ const nextJoin = joins[index];
+ const audioMap = {
+ id: `segment:${segment.id}:audio-map`,
+ sourceId: segment.audio.id,
+ sourceDigest: segment.audio.contentDigest,
+ sourceStartFrame,
+ sourceEndFrameExclusive,
+ programStartFrame,
+ programEndFrameExclusive,
+ };
+ sourceMaps.push(audioMap);
+ alignmentSubjects.push({
+ id: `segment:${segment.id}:speech`,
+ sourceId: segment.audio.id,
+ sourceDigest: segment.audio.contentDigest,
+ sourceMapId: audioMap.id,
+ });
+
+ let audioEndFrameExclusive = programEndFrameExclusive;
+ if (nextJoin?.temporal === "overlap" && nextJoin.audio === "cut") {
+ audioEndFrameExclusive = frame(segments[index + 1].startSec, fps);
+ }
+ const fadeInFrames = previousJoin?.temporal === "overlap" && previousJoin.audio === "crossfade"
+ ? frame(previousJoin.durationSec, fps)
+ : 0;
+ const fadeOutFrames = nextJoin?.temporal === "overlap" && nextJoin.audio === "crossfade"
+ ? frame(nextJoin.durationSec, fps)
+ : 0;
+ const audioContribution = {
+ id: `segment:${segment.id}:program-audio`,
+ sourceId: segment.audio.id,
+ sourceDigest: segment.audio.contentDigest,
+ source: segment.audio.source,
+ sourceMapId: audioMap.id,
+ sourceStartFrame,
+ sourceEndFrameExclusive: sourceStartFrame + (audioEndFrameExclusive - programStartFrame),
+ programStartFrame,
+ programEndFrameExclusive: audioEndFrameExclusive,
+ gain: 1,
+ ...(fadeInFrames ? { fadeInFrames } : {}),
+ ...(fadeOutFrames ? { fadeOutFrames } : {}),
+ };
+ audio.push(audioContribution);
+
+ const audioRange = {
+ startFrame: programStartFrame,
+ endFrameExclusive: audioEndFrameExclusive,
+ startSec: programStartFrame / fps,
+ endSec: audioEndFrameExclusive / fps,
+ };
+ const output = {
+ audio: {
+ type: "ProgramBoundAudio",
+ source: segment.audio.source,
+ sourceId: segment.audio.id,
+ sourceDigest: segment.audio.contentDigest,
+ range: audioRange,
+ mediaStartSec: segment.window.startSec,
+ playbackRate: 1,
+ sourceMapId: audioMap.id,
+ ...(fadeInFrames ? { fadeInFrames } : {}),
+ ...(fadeOutFrames ? { fadeOutFrames } : {}),
+ },
+ };
+ if (segment.visual) {
+ const visualMap = {
+ id: `segment:${segment.id}:visual-map`,
+ sourceId: segment.visual.id,
+ sourceDigest: segment.visual.contentDigest,
+ sourceStartFrame,
+ sourceEndFrameExclusive,
+ programStartFrame,
+ programEndFrameExclusive,
+ };
+ sourceMaps.push(visualMap);
+ let visualEndFrameExclusive = programEndFrameExclusive;
+ if (nextJoin?.temporal === "overlap" && nextJoin.visual === "cut") {
+ visualEndFrameExclusive = frame(segments[index + 1].startSec, fps);
+ }
+ const visualFadeInFrames = previousJoin?.temporal === "overlap" && previousJoin.visual === "dissolve"
+ ? frame(previousJoin.durationSec, fps)
+ : 0;
+ const visualFadeOutFrames = nextJoin?.temporal === "overlap" && nextJoin.visual === "dissolve"
+ ? frame(nextJoin.durationSec, fps)
+ : 0;
+ output.visual = {
+ type: "ProgramBoundVideo",
+ source: segment.visual.source,
+ sourceId: segment.visual.id,
+ sourceDigest: segment.visual.contentDigest,
+ range: {
+ startFrame: programStartFrame,
+ endFrameExclusive: visualEndFrameExclusive,
+ startSec: programStartFrame / fps,
+ endSec: visualEndFrameExclusive / fps,
+ },
+ mediaStartSec: segment.window.startSec,
+ playbackRate: 1,
+ sourceMapId: visualMap.id,
+ ...(visualFadeInFrames ? { fadeInFrames: visualFadeInFrames } : {}),
+ ...(visualFadeOutFrames ? { fadeOutFrames: visualFadeOutFrames } : {}),
+ };
+ }
+ unboundSegmentOutputs[segment.id] = output;
+ }
+ const unboundFacets = {
+ visual: {
+ type: "ProgramBoundVideoSequence",
+ segments: Object.fromEntries(Object.entries(unboundSegmentOutputs)
+ .filter(([, value]) => value.visual)
+ .map(([id, value]) => [id, value.visual])),
+ },
+ };
+ const outcome = {
+ contract: "svml.program-basis-outcome.v1",
+ frameRate,
+ originFrame: 0,
+ durationFrames,
+ alignmentSubjects,
+ sourceMaps,
+ audio: audio.map(({ source: _source, ...contribution }) => contribution),
+ facets: canonicalFacet(unboundFacets),
+ };
+ const basisDigest = context.digest(outcome);
+ const basis = {
+ contract: "svml.program-basis.v1",
+ frameRate,
+ originFrame: 0,
+ durationFrames,
+ basisDigest,
+ };
+ const segmentOutputs = Object.fromEntries(Object.entries(unboundSegmentOutputs).map(([id, output]) => [
+ id,
+ {
+ audio: { ...output.audio, basisDigest },
+ ...(output.visual ? { visual: { ...output.visual, basisDigest } } : {}),
+ },
+ ]));
+ const facets = {
+ visual: {
+ type: "ProgramBoundVideoSequence",
+ basisDigest,
+ segments: Object.fromEntries(Object.entries(segmentOutputs)
+ .filter(([, value]) => value.visual)
+ .map(([id, value]) => [id, value.visual])),
+ },
+ };
+ const production = {
+ contract: "svml.temporal-basis-production.v1",
+ basis,
+ alignmentSubjects,
+ sourceMaps,
+ audio,
+ facets,
+ productionDigest: context.digest({
+ implementation: context.instance.executionDigest,
+ outcome,
+ }),
+ };
+ return {
+ outputs: {
+ production,
+ facets: production.facets,
+ segment: segmentOutputs,
+ },
+ };
+ },
+};
diff --git a/stdlib/runtime/speech-locator.mjs b/stdlib/runtime/speech-locator.mjs
new file mode 100644
index 00000000..547da910
--- /dev/null
+++ b/stdlib/runtime/speech-locator.mjs
@@ -0,0 +1,94 @@
+function normalizeWord(value) {
+ return String(value)
+ .normalize("NFKC")
+ .toLocaleLowerCase("en")
+ .replace(/[^\p{L}\p{M}\p{N}]+/gu, "");
+}
+
+function fail(message) {
+ throw new Error(message);
+}
+
+export default {
+ abiVersion: "1",
+ project(context) {
+ const narrative = context.element.attributes.script;
+ const production = context.element.attributes.basis;
+ const evidenceValue = context.element.attributes.evidence;
+ const evidence = evidenceValue?.data ?? evidenceValue;
+ if (!narrative?.semanticIndex) fail("speech-locator requires NarrativeIR");
+ if (production?.contract !== "svml.temporal-basis-production.v1") {
+ fail("speech-locator requires TemporalBasisProduction");
+ }
+ if (evidence?.contract !== "svml.speech-alignment.v1") fail("unsupported alignment evidence");
+ const basis = production.basis;
+ const fps = basis.frameRate.numerator / basis.frameRate.denominator;
+ if (Math.round(evidence.durationSec * fps) !== basis.durationFrames) {
+ fail("alignment duration does not match ProgramBasis");
+ }
+ if (evidence.words.length !== narrative.tokens.length) fail("alignment token cardinality mismatch");
+ if (evidence.segments.length !== narrative.segments.length) fail("alignment Segment cardinality mismatch");
+ const segments = new Map(evidence.segments.map(segment => [segment.id, segment]));
+ if (segments.size !== evidence.segments.length) fail("alignment repeats a Segment id");
+ const words = new Map();
+ const previousBySegment = new Map();
+ for (const [index, token] of narrative.tokens.entries()) {
+ const word = evidence.words[index];
+ const segment = segments.get(token.segmentId);
+ if (!word || !segment || word.segmentId !== token.segmentId || normalizeWord(word.text) !== token.normalized) {
+ fail(`alignment token ${index + 1} does not match Script`);
+ }
+ if (word.startSec < segment.startSec || word.endSec < word.startSec || word.endSec > segment.endSec) {
+ fail(`alignment token ${index + 1} falls outside Segment`);
+ }
+ const previous = previousBySegment.get(word.segmentId);
+ if (previous && word.startSec < previous.endSec - 1e-6) fail(`alignment overlaps inside Segment ${word.segmentId}`);
+ previousBySegment.set(word.segmentId, word);
+ words.set(token.id, word);
+ }
+ for (const [index, scriptSegment] of narrative.segments.entries()) {
+ const segment = segments.get(scriptSegment.id);
+ if (!segment) fail(`alignment is missing Segment ${scriptSegment.id}`);
+ const previousScript = narrative.segments[index - 1];
+ const previous = previousScript ? segments.get(previousScript.id) : undefined;
+ if (previous && (segment.startSec < previous.startSec || segment.endSec < previous.endSec)) {
+ fail(`alignment reorders Segment ${scriptSegment.id}`);
+ }
+ }
+ const anchors = narrative.semanticIndex.anchors.map(anchor => {
+ const segment = segments.get(anchor.segmentId);
+ const word = anchor.tokenId ? words.get(anchor.tokenId) : undefined;
+ const seconds = anchor.kind === "segment-start"
+ ? segment?.startSec
+ : anchor.kind === "segment-end"
+ ? segment?.endSec
+ : anchor.kind === "token-start"
+ ? word?.startSec
+ : word?.endSec;
+ if (seconds === undefined) fail(`alignment is missing anchor ${anchor.id}`);
+ return {
+ identity: anchor.id,
+ point: { basisDigest: basis.basisDigest, frame: Math.round(seconds * fps) },
+ quality: "measured",
+ };
+ });
+ const payload = {
+ contract: "svml.exact-semantic-map.v1",
+ semanticIndexDigest: narrative.semanticIndex.digest,
+ basisDigest: basis.basisDigest,
+ anchors,
+ evidenceDigests: [evidenceValue.contentDigest ?? context.digest(evidence)],
+ locatorDigest: context.digest({
+ implementation: context.instance.executionDigest,
+ quantizationPolicy: "nearest-frame",
+ }),
+ quantizationPolicy: "nearest-frame",
+ ...(evidence.captionCues ? { captionCues: evidence.captionCues } : {}),
+ };
+ return {
+ outputs: {
+ map: { ...payload, mapDigest: context.digest(payload) },
+ },
+ };
+ },
+};
diff --git a/stdlib/runtime/speech-spine.mjs b/stdlib/runtime/speech-spine.mjs
deleted file mode 100644
index f98fc712..00000000
--- a/stdlib/runtime/speech-spine.mjs
+++ /dev/null
@@ -1,49 +0,0 @@
-import { elements, material, resolveAttr, stringAttr } from "./helpers.mjs";
-
-export default {
- abiVersion: "1",
- project(context) {
- const segmentOutputs = {};
- const audios = [];
- for (const segment of elements(context, "segment")) {
- const id = stringAttr(segment, "id");
- const located = resolveAttr(context, segment, "script");
- const visual = material(resolveAttr(context, segment, "visual"), "video");
- const audioValue = segment.attributes.audio
- ? material(resolveAttr(context, segment, "audio"), "audio")
- : { source: visual.source };
- segmentOutputs[id] = {
- range: located,
- visual: {
- type: "Video",
- source: visual.source,
- range: located,
- },
- audio: {
- type: "Audio",
- source: audioValue.source,
- range: located,
- },
- };
- audios.push({
- id: `${context.instance.id}:speech:${id}`,
- source: audioValue.source,
- startFrame: located.startFrame,
- endFrameExclusive: located.endFrameExclusive,
- volume: 1,
- bus: "speech",
- });
- }
- return {
- outputs: {
- segment: segmentOutputs,
- speech: {
- segments: segmentOutputs,
- primaryAudio: audios,
- },
- track: { visuals: [], audios, styles: [] },
- },
- audios,
- };
- },
-};
diff --git a/stdlib/speech-assemble.svk b/stdlib/speech-assemble.svk
new file mode 100644
index 00000000..20a5b5c2
--- /dev/null
+++ b/stdlib/speech-assemble.svk
@@ -0,0 +1,23 @@
+<kernel name="speech-assemble" abi="1" implementation="./runtime/speech-assemble.mjs">
+ <param name="fps" type="number" default="30"/>
+ <param name="defaultJoin" style="default-join" type="string" default="cut"/>
+ <param name="defaultJoinDuration" style="default-join-duration" type="string" default="0ms"/>
+ <port name="production" direction="output" type="TemporalBasisProduction" cardinality="one"/>
+ <port name="facets.visual" direction="output" type="ProgramBoundVideoSequence" cardinality="zero-or-one"/>
+ <port name="segment.*.visual" direction="output" type="ProgramBoundVideo" cardinality="zero-or-one"/>
+ <port name="segment.*.audio" direction="output" type="ProgramBoundAudio" cardinality="one"/>
+ <child name="segment" cardinality="one-or-more">
+ <field name="id" type="id" required="true"/>
+ <field name="script" type="ScriptSegment" required="true"/>
+ <field name="visual" type="Video"/>
+ <field name="audio" type="Audio" required="true"/>
+ <field name="sourceWindow" type="string" required="true"/>
+ </child>
+ <child name="join" cardinality="zero-or-more">
+ <field name="after" type="id" required="true"/>
+ <field name="overlap" type="string"/>
+ <field name="gap" type="string"/>
+ <field name="audio" type="string"/>
+ <field name="visual" type="string"/>
+ </child>
+</kernel>
diff --git a/stdlib/speech-locator.svk b/stdlib/speech-locator.svk
new file mode 100644
index 00000000..34b36f0c
--- /dev/null
+++ b/stdlib/speech-locator.svk
@@ -0,0 +1,6 @@
+<kernel name="speech-locator" abi="1" implementation="./runtime/speech-locator.mjs">
+ <port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
+ <port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
+ <port name="evidence" direction="input" type="AlignmentEvidence" cardinality="one"/>
+ <port name="map" direction="output" type="ExactSemanticMap" cardinality="one"/>
+</kernel>
diff --git a/stdlib/speech-spine.svk b/stdlib/speech-spine.svk
deleted file mode 100644
index e2ab8e4c..00000000
--- a/stdlib/speech-spine.svk
+++ /dev/null
@@ -1,14 +0,0 @@
-<kernel name="speech-spine" abi="1" implementation="./runtime/speech-spine.mjs">
- <port name="segment" direction="input" type="SegmentBinding" cardinality="one-or-more"/>
- <port name="segment" direction="output" type="SegmentMaterial" cardinality="one-or-more"/>
- <port name="segment.*.visual" direction="output" type="Video" cardinality="zero-or-one"/>
- <port name="segment.*.audio" direction="output" type="Audio" cardinality="one"/>
- <port name="speech" direction="output" type="SpeechSpine" cardinality="one"/>
- <port name="track" direction="output" type="Track" cardinality="one"/>
- <child name="segment" cardinality="one-or-more">
- <field name="id" type="id" required="true"/>
- <field name="script" type="ScriptSegment" required="true"/>
- <field name="visual" type="Video" required="true"/>
- <field name="audio" type="Audio"/>
- </child>
-</kernel>
diff --git a/stdlib/text-track.svk b/stdlib/text-track.svk
index f2a1a488..e25358b8 100644
--- a/stdlib/text-track.svk
+++ b/stdlib/text-track.svk
@@ -10,12 +10,12 @@
<field name="y" type="number"/>
<field name="width" type="number"/>
<field name="height" type="number"/>
- <field name="font" type="string"/>
- <field name="size" type="number"/>
- <field name="weight" type="number"/>
- <field name="lineHeight" type="number"/>
- <field name="letterSpacing" type="number"/>
- <field name="color" type="string"/>
+ <field name="font" style="font-family" type="string"/>
+ <field name="size" style="font-size" type="number"/>
+ <field name="weight" style="font-weight" type="number"/>
+ <field name="lineHeight" style="line-height" type="number"/>
+ <field name="letterSpacing" style="letter-spacing" type="number"/>
+ <field name="color" style="fill" type="string"/>
<field name="background" type="string"/>
<field name="uppercase" type="boolean"/>
</child>
diff --git a/test/artifacts.test.ts b/test/artifacts.test.ts
index a9174d03..987a4725 100644
--- a/test/artifacts.test.ts
+++ b/test/artifacts.test.ts
@@ -6,7 +6,7 @@ import { join } from "node:path";
import test from "node:test";
import { fileURLToPath, pathToFileURL } from "node:url";
import { checkSource, compileSource } from "../src/compiler.js";
-import { sha256 } from "../src/util.js";
+import { sha256, stableJson } from "../src/util.js";
const fixture = fileURLToPath(new URL("./fixtures/artifacts/", import.meta.url));
const pinnedImage = fileURLToPath(
@@ -27,7 +27,6 @@ test("capability instances require exact verified artifacts and never invoke a p
await assert.rejects(
compileSource({
file,
- evidenceFile: `${fixture}/alignment.json`,
}),
/artifact_file_required/u,
);
@@ -54,16 +53,27 @@ test("capability instances require exact verified artifacts and never invoke a p
const compilation = await compileSource({
file,
- evidenceFile: `${fixture}/alignment.json`,
lockFile: lock,
artifactsFile: artifacts,
outputFile: output,
});
assert.equal(compilation.artifactsVerified, true);
- assert.equal(compilation.reproducible, true);
+ assert.equal(compilation.sourceClosureVerified, true);
+ assert.equal(compilation.temporalEvidenceVerified, false);
+ assert.equal(compilation.reproducible, false);
assert.match(compilation.html, /bound-image/u);
assert.match(compilation.html, /\.\/assets\/[a-f0-9]{16}-image\.png/u);
+ await writeFile(lock, `${JSON.stringify(compilation.lock, null, 2)}\n`, "utf8");
+ const frozen = await compileSource({
+ file,
+ lockFile: lock,
+ artifactsFile: artifacts,
+ });
+ assert.equal(frozen.lockVerified, true);
+ assert.equal(frozen.temporalEvidenceVerified, true);
+ assert.equal(frozen.reproducible, true);
+
const alternateArtifacts = join(directory, "alternate-artifacts.json");
await writeFile(alternateArtifacts, `${JSON.stringify({
contract: "svml.artifacts.v1",
@@ -81,8 +91,6 @@ test("capability instances require exact verified artifacts and never invoke a p
}, null, 2)}\n`, "utf8");
const alternate = await compileSource({
file,
- evidenceFile: `${fixture}/alignment.json`,
- lockFile: lock,
artifactsFile: alternateArtifacts,
});
assert.equal(
@@ -94,3 +102,65 @@ test("capability instances require exact verified artifacts and never invoke a p
alternate.plan.instances.find((instance) => instance.id === "film")?.executionDigest,
);
});
+
+test("a Locator capability returns ExactSemanticMap through the same typed Artifact ABI", async () => {
+ const source = fileURLToPath(
+ new URL("../examples/composite-speech-program/minimal.svml", import.meta.url),
+ );
+ const reference = await compileSource({ file: source });
+ const directory = await mkdtemp(join(tmpdir(), "svml-locator-artifact-"));
+ const svml = join(directory, "main.svml");
+ const assemble = fileURLToPath(new URL("../stdlib/speech-assemble.svk", import.meta.url));
+ const film = fileURLToPath(new URL("../stdlib/film.svk", import.meta.url));
+ const locator = fileURLToPath(
+ new URL("./fixtures/artifacts/pinned-locator.svk", import.meta.url),
+ );
+ const avatar = fileURLToPath(
+ new URL("../examples/regen-ranking/assets/avatar-1.mp4", import.meta.url),
+ );
+ const audio = fileURLToPath(
+ new URL("../examples/regen-ranking/assets/avatar-1.mp3", import.meta.url),
+ );
+ const evidence = fileURLToPath(
+ new URL("../examples/composite-speech-program/alignment.json", import.meta.url),
+ );
+ await writeFile(svml, `<svml version="1">
+ <import from="${assemble}"/>
+ <import from="${locator}"/>
+ <import from="${film}"/>
+ <script>
+ <segment id="intro"><HOST> Hello, this is SVML.</segment>
+ <segment id="close"><HOST> Write the script, get the video.</segment>
+ </script>
+ <video id="host" src="${avatar}"/>
+ <audio id="host-audio" src="${audio}"/>
+ <alignment id="alignment" src="${evidence}"/>
+ <speech-assemble id="voice" fps="30" defaultJoin="cut">
+ <segment id="intro" script={script.segment.intro} visual={host} audio={host-audio} sourceWindow="0s .. 5s"/>
+ <segment id="close" script={script.segment.close} visual={host} audio={host-audio} sourceWindow="5s .. 10s"/>
+ <join after="intro" overlap="500ms" audio="crossfade" visual="dissolve"/>
+ </speech-assemble>
+ <pinned-locator id="location" script={script} basis={voice.production} evidence={alignment}/>
+ <film id="film" basis={voice.production} semantic={location.map}/>
+</svml>\n`, "utf8");
+ const checked = await checkSource(svml);
+ const instance = checked.plan.instances.find((item) => item.id === "location");
+ const artifacts = join(directory, "artifacts.json");
+ await writeFile(artifacts, `${JSON.stringify({
+ contract: "svml.artifacts.v1",
+ bindings: [{
+ instanceIdentity: instance?.identity,
+ executionDigest: instance?.executionDigest,
+ outputs: {
+ map: {
+ type: "ExactSemanticMap",
+ value: reference.semanticMap,
+ contentDigest: sha256(stableJson(reference.semanticMap)),
+ },
+ },
+ }],
+ }, null, 2)}\n`, "utf8");
+ const compilation = await compileSource({ file: svml, artifactsFile: artifacts });
+ assert.equal(compilation.semanticMap.mapDigest, reference.semanticMap.mapDigest);
+ assert.equal(compilation.semanticMap.basisDigest, compilation.basis.basis.basisDigest);
+});
diff --git a/test/composite.test.ts b/test/composite.test.ts
new file mode 100644
index 00000000..a3ea210a
--- /dev/null
+++ b/test/composite.test.ts
@@ -0,0 +1,54 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { fileURLToPath } from "node:url";
+import { checkSource, compileSource } from "../src/compiler.js";
+
+const fixture = fileURLToPath(
+ new URL("../examples/composite-speech-program/minimal.svml", import.meta.url),
+);
+
+test("composite-v1 expands finite internal Basis/Locator instances with audited aliases", async () => {
+ const checked = await checkSource(fixture);
+ assert.deepEqual(
+ checked.plan.instances.map((instance) => instance.id).sort(),
+ ["aroll", "main", "voice::basis", "voice::locator"],
+ );
+ assert.equal(checked.plan.expansions.length, 1);
+ assert.deepEqual(checked.plan.expansions[0]?.internalIds, [
+ "voice::basis",
+ "voice::locator",
+ ]);
+ assert.deepEqual(checked.plan.expansions[0]?.exports, {
+ production: "voice::basis.production",
+ map: "voice::locator.map",
+ "facets.visual": "voice::basis.facets.visual",
+ });
+ assert.equal(checked.lock.expansions[0]?.expansionDigest.length, 64);
+ const internal = checked.plan.instances.find((instance) => instance.id === "voice::basis");
+ assert.match(internal?.sourceMap?.callSite.file ?? "", /minimal\.svml$/u);
+ assert.match(internal?.sourceMap?.definitionSite.file ?? "", /speech-program\.svk$/u);
+
+ const compilation = await compileSource({ file: fixture });
+ assert.equal(compilation.located.durationFrames, 285);
+ assert.equal(compilation.basis.sourceMaps.length, 4);
+ assert.equal(compilation.basis.audio[0]?.fadeOutFrames, 15);
+ assert.equal(compilation.basis.audio[1]?.fadeInFrames, 15);
+
+ const visuals = compilation.projections.get("aroll")?.visuals ?? [];
+ assert.deepEqual(
+ visuals.map((visual) => [visual.startFrame, visual.endFrameExclusive, visual.mediaStartSec]),
+ [[0, 150, 0], [135, 285, 5]],
+ );
+ assert.ok(visuals.every((visual) => visual.z === 100));
+ assert.ok(visuals.every((visual) => /svml-program-bound/u.test(visual.css ?? "")));
+});
+
+test("a precomposed media Basis reuses the same Locator and Composition contracts", async () => {
+ const file = fileURLToPath(new URL("./fixtures/media-basis/main.svml", import.meta.url));
+ const compilation = await compileSource({ file });
+ assert.equal(compilation.basis.basis.durationFrames, 30);
+ assert.equal(compilation.basis.alignmentSubjects.length, 1);
+ assert.equal(compilation.basis.sourceMaps.length, 2);
+ assert.equal(compilation.semanticMap.anchors.length, 4);
+ assert.equal(compilation.target.audios.length, 1);
+});
diff --git a/test/fixtures/artifacts/artifact-film.mjs b/test/fixtures/artifacts/artifact-film.mjs
index 15d72e75..5368690a 100644
--- a/test/fixtures/artifacts/artifact-film.mjs
+++ b/test/fixtures/artifacts/artifact-film.mjs
@@ -2,6 +2,7 @@ export default {
abiVersion: "1",
project(context) {
const image = context.resolve(context.element.attributes.image);
+ const production = context.resolve(context.element.attributes.basis);
return {
outputs: {
document: {
@@ -9,15 +10,15 @@ export default {
id: context.instance.id,
width: 64,
height: 64,
- fps: context.fps,
- durationFrames: context.located.durationFrames,
+ fps: production.basis.frameRate.numerator / production.basis.frameRate.denominator,
+ durationFrames: production.basis.durationFrames,
background: "#000000",
visuals: [{
id: "bound-image",
kind: "image",
source: image.absoluteSource ?? image.source,
startFrame: 0,
- endFrameExclusive: context.located.durationFrames,
+ endFrameExclusive: production.basis.durationFrames,
z: 0,
}],
audios: [],
diff --git a/test/fixtures/artifacts/artifact-film.svk b/test/fixtures/artifacts/artifact-film.svk
index 8f93ad04..b1611e71 100644
--- a/test/fixtures/artifacts/artifact-film.svk
+++ b/test/fixtures/artifacts/artifact-film.svk
@@ -5,5 +5,7 @@
implementation="./artifact-film.mjs"
>
<port name="image" direction="input" type="Image" cardinality="one"/>
+ <port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
+ <port name="semantic" direction="input" type="ExactSemanticMap" cardinality="one"/>
<port name="document" direction="output" type="HyperframesDocument" cardinality="one"/>
</kernel>
diff --git a/test/fixtures/artifacts/main.svml b/test/fixtures/artifacts/main.svml
index 730bb240..e8a49887 100644
--- a/test/fixtures/artifacts/main.svml
+++ b/test/fixtures/artifacts/main.svml
@@ -1,11 +1,20 @@
<svml version="1">
<import from="./pinned-image.svk"/>
<import from="./artifact-film.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<script>
<segment id="main">Hello.</segment>
</script>
+ <audio id="voice-audio" src="./alignment.json"/>
+ <alignment id="alignment" src="./alignment.json"/>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} audio={voice-audio} sourceWindow="0s .. 1s"/>
+ </speech-assemble>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+
<pinned-image id="poster" prompt="A pinned result; never call a provider."/>
- <artifact-film id="film" image={poster.image}/>
+ <artifact-film id="film" image={poster.image} basis={speech.production} semantic={location.map}/>
</svml>
diff --git a/test/fixtures/artifacts/pinned-locator.svk b/test/fixtures/artifacts/pinned-locator.svk
new file mode 100644
index 00000000..6296311c
--- /dev/null
+++ b/test/fixtures/artifacts/pinned-locator.svk
@@ -0,0 +1,6 @@
+<kernel name="pinned-locator" abi="1" profile="capability-v1" capability="semantic-locate">
+ <port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
+ <port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
+ <port name="evidence" direction="input" type="AlignmentEvidence" cardinality="one"/>
+ <port name="map" direction="output" type="ExactSemanticMap" cardinality="one"/>
+</kernel>
diff --git a/test/fixtures/collision/main.svml b/test/fixtures/collision/main.svml
index f813653f..0925685b 100644
--- a/test/fixtures/collision/main.svml
+++ b/test/fixtures/collision/main.svml
@@ -1,5 +1,6 @@
<svml version="1">
- <import from="../../../stdlib/speech-spine.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<import from="../../../stdlib/broll-track.svk"/>
<import from="../../../stdlib/film.svk"/>
@@ -10,12 +11,16 @@
</script>
<video id="avatar" src="./avatar.mp4"/>
+ <audio id="voice-audio" src="./avatar.mp4"/>
+ <alignment id="alignment" src="./alignment.json"/>
<image id="left-image" src="./left/image.png"/>
<image id="right-image" src="./right/image.png"/>
- <speech-spine id="speech">
- <segment id="main" script={script.segment.main} visual={avatar}/>
- </speech-spine>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 2s"/>
+ </speech-assemble>
+
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
<broll-track id="broll" z="100">
<item id="left" source={left-image} during={script.selection.left}/>
@@ -23,7 +28,7 @@
<transition from="left" to="right" kind="crossfade" duration="200ms"/>
</broll-track>
- <film id="collision" speech={speech.speech}>
+ <film id="collision" basis={speech.production} semantic={location.map}>
<track ref={broll.track}/>
</film>
</svml>
diff --git a/test/fixtures/disconnected/captions.svml b/test/fixtures/disconnected/captions.svml
new file mode 100644
index 00000000..9081f409
--- /dev/null
+++ b/test/fixtures/disconnected/captions.svml
@@ -0,0 +1,25 @@
+<svml version="1">
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
+ <import from="../../../stdlib/caption-track.svk"/>
+ <import from="../../../stdlib/film.svk"/>
+
+ <script>
+ <segment id="main">@pick Alpha @/pick gap @pick omega @/pick.</segment>
+ </script>
+
+ <audio id="voice-audio" src="../presentation/long.mp4"/>
+ <alignment id="alignment" src="./alignment.json"/>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} audio={voice-audio} sourceWindow="0s .. 3s"/>
+ </speech-assemble>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+
+ <caption-track id="captions" script={script} semantic={location.map} z="10" maxWords="3">
+ <style id="picked" during={script.selection.pick} color="#ff0000" activeColor="#00ff00" scale="1.1"/>
+ </caption-track>
+
+ <film id="film" basis={speech.production} semantic={location.map}>
+ <track ref={captions.track}/>
+ </film>
+</svml>
diff --git a/test/fixtures/disconnected/each.svml b/test/fixtures/disconnected/each.svml
index ce79c2e5..5cc168df 100644
--- a/test/fixtures/disconnected/each.svml
+++ b/test/fixtures/disconnected/each.svml
@@ -1,5 +1,6 @@
<svml version="1">
- <import from="../../../stdlib/speech-spine.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<import from="../../../stdlib/media-track.svk"/>
<import from="../../../stdlib/film.svk"/>
@@ -8,16 +9,19 @@
</script>
<video id="avatar" src="../presentation/long.mp4"/>
+ <audio id="voice-audio" src="../presentation/long.mp4"/>
+ <alignment id="alignment" src="./alignment.json"/>
- <speech-spine id="speech">
- <segment id="main" script={script.segment.main} visual={avatar}/>
- </speech-spine>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 3s"/>
+ </speech-assemble>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
<media-track id="media" z="1">
<item id="picked" source={avatar} during={script.selection.pick}/>
</media-track>
- <film id="film" speech={speech.speech}>
+ <film id="film" basis={speech.production} semantic={location.map}>
<track ref={media.track}/>
</film>
</svml>
diff --git a/test/fixtures/disconnected/one.svml b/test/fixtures/disconnected/one.svml
index 7f0d83cf..f7e737f4 100644
--- a/test/fixtures/disconnected/one.svml
+++ b/test/fixtures/disconnected/one.svml
@@ -1,5 +1,6 @@
<svml version="1">
- <import from="../../../stdlib/speech-spine.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<import from="../../../stdlib/ranking-column.svk"/>
<import from="../../../stdlib/film.svk"/>
@@ -8,17 +9,20 @@
</script>
<video id="avatar" src="../presentation/long.mp4"/>
+ <audio id="voice-audio" src="../presentation/long.mp4"/>
<image id="icon" src="../collision/left/image.png"/>
+ <alignment id="alignment" src="./alignment.json"/>
- <speech-spine id="speech">
- <segment id="main" script={script.segment.main} visual={avatar}/>
- </speech-spine>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 3s"/>
+ </speech-assemble>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
<ranking-column id="ranking" z="1">
<item id="picked" rank="1" image={icon} during={script.selection.pick}/>
</ranking-column>
- <film id="film" speech={speech.speech}>
+ <film id="film" basis={speech.production} semantic={location.map}>
<track ref={ranking.track}/>
</film>
</svml>
diff --git a/test/fixtures/disconnected/set.svml b/test/fixtures/disconnected/set.svml
index dccdcc62..5f78cd9a 100644
--- a/test/fixtures/disconnected/set.svml
+++ b/test/fixtures/disconnected/set.svml
@@ -1,5 +1,6 @@
<svml version="1">
- <import from="../../../stdlib/speech-spine.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<import from="./selection-set.svk"/>
<import from="../../../stdlib/film.svk"/>
@@ -8,16 +9,19 @@
</script>
<video id="avatar" src="../presentation/long.mp4"/>
+ <audio id="voice-audio" src="../presentation/long.mp4"/>
+ <alignment id="alignment" src="./alignment.json"/>
- <speech-spine id="speech">
- <segment id="main" script={script.segment.main} visual={avatar}/>
- </speech-spine>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 3s"/>
+ </speech-assemble>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
<selection-set id="styles">
<item id="highlight" during={script.selection.pick}/>
</selection-set>
- <film id="film" speech={speech.speech}>
+ <film id="film" basis={speech.production} semantic={location.map}>
<track ref={styles.track}/>
</film>
</svml>
diff --git a/test/fixtures/media-basis/main.svml b/test/fixtures/media-basis/main.svml
new file mode 100644
index 00000000..0bc1f7bc
--- /dev/null
+++ b/test/fixtures/media-basis/main.svml
@@ -0,0 +1,14 @@
+<svml version="1">
+ <import from="../../../stdlib/media-basis.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
+ <import from="../../../stdlib/film.svk"/>
+
+ <script><segment id="main">Hello.</segment></script>
+ <video id="visual" src="../collision/avatar.mp4"/>
+ <audio id="audio" src="../collision/avatar.mp4"/>
+ <alignment id="alignment" src="../artifacts/alignment.json"/>
+
+ <media-basis id="program" visual={visual} audio={audio} duration="1s"/>
+ <speech-locator id="location" script={script} basis={program.production} evidence={alignment}/>
+ <film id="film" basis={program.production} semantic={location.map}/>
+</svml>
diff --git a/test/fixtures/modules/house.svs b/test/fixtures/modules/house.svs
index e06a57ad..ee3945d1 100644
--- a/test/fixtures/modules/house.svs
+++ b/test/fixtures/modules/house.svs
@@ -9,6 +9,12 @@
fill: #fefefe;
}
+ subtitles.launch {
+ z: 510;
+ font-size: 44;
+ fill: #fefefe;
+ }
+
film.vertical {
width: 1080;
height: 1920;
diff --git a/test/fixtures/modules/main-renamed-alias.svml b/test/fixtures/modules/main-renamed-alias.svml
index 42b8d0ef..743ba685 100644
--- a/test/fixtures/modules/main-renamed-alias.svml
+++ b/test/fixtures/modules/main-renamed-alias.svml
@@ -1,7 +1,8 @@
<svml version="1">
- <import from="../../../stdlib/speech-spine.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<import from="../../../stdlib/broll-track.svk"/>
- <import from="../../../stdlib/caption-track.svk"/>
+ <import as="subtitles" from="../../../stdlib/caption-track.svk"/>
<import from="../../../stdlib/film.svk"/>
<import as="assets" from="./brand.svc"/>
<import as="house" from="./house.svs"/>
@@ -13,18 +14,22 @@
</script>
<video id="avatar" src="./avatar.mp4"/>
+ <audio id="voice-audio" src="./avatar.mp4"/>
+ <alignment id="alignment" src="./alignment.json"/>
- <speech-spine id="speech">
- <segment id="main" script={script.segment.main} visual={avatar}/>
- </speech-spine>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 1s"/>
+ </speech-assemble>
+
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
<broll-track id="broll" class="house.full">
<item id="poster" source={assets.poster.image} during={script.selection.poster}/>
</broll-track>
- <caption-track id="captions" script={script} class="house.launch" size="48"/>
+ <subtitles id="captions" script={script} semantic={location.map} class="house.launch" size="48"/>
- <film id="main-film" class="house.vertical" speech={speech.speech}>
+ <film id="main-film" class="house.vertical" basis={speech.production} semantic={location.map}>
<track ref={broll.track}/>
<track ref={captions.track}/>
</film>
diff --git a/test/fixtures/modules/main.svml b/test/fixtures/modules/main.svml
index 671a4afc..178a9e20 100644
--- a/test/fixtures/modules/main.svml
+++ b/test/fixtures/modules/main.svml
@@ -1,5 +1,6 @@
<svml version="1">
- <import from="../../../stdlib/speech-spine.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<import from="../../../stdlib/broll-track.svk"/>
<import from="../../../stdlib/caption-track.svk"/>
<import from="../../../stdlib/film.svk"/>
@@ -13,18 +14,22 @@
</script>
<video id="avatar" src="./avatar.mp4"/>
+ <audio id="voice-audio" src="./avatar.mp4"/>
+ <alignment id="alignment" src="./alignment.json"/>
- <speech-spine id="speech">
- <segment id="main" script={script.segment.main} visual={avatar}/>
- </speech-spine>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 1s"/>
+ </speech-assemble>
+
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
<broll-track id="broll" class="house.full">
<item id="poster" source={brand.poster.image} during={script.selection.poster}/>
</broll-track>
- <caption-track id="captions" script={script} class="house.launch" size="48"/>
+ <caption-track id="captions" script={script} semantic={location.map} class="house.launch" size="48"/>
- <film id="main-film" class="house.vertical" speech={speech.speech}>
+ <film id="main-film" class="house.vertical" basis={speech.production} semantic={location.map}>
<track ref={broll.track}/>
<track ref={captions.track}/>
</film>
diff --git a/test/fixtures/presentation/empty-present.svml b/test/fixtures/presentation/empty-present.svml
new file mode 100644
index 00000000..9b46e3e2
--- /dev/null
+++ b/test/fixtures/presentation/empty-present.svml
@@ -0,0 +1,31 @@
+<svml version="1">
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
+ <import from="../../../stdlib/media-track.svk"/>
+ <import from="../../../stdlib/film.svk"/>
+
+ <script>
+ <segment id="main">
+ One @pop! two three.
+ </segment>
+ </script>
+
+ <video id="long-video" src="./long.mp4"/>
+ <audio id="voice-audio" src="./long.mp4"/>
+ <alignment id="alignment" src="./alignment.json"/>
+
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} visual={long-video} audio={voice-audio} sourceWindow="0s .. 3s"/>
+ </speech-assemble>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+
+ <media-track id="aroll" z="200">
+ <item id="host" source={long-video} during="full">
+ <present id="outside" during="end+1s .. end+2s" z="230"/>
+ </item>
+ </media-track>
+
+ <film id="presentation" basis={speech.production} semantic={location.map}>
+ <track ref={aroll.track}/>
+ </film>
+</svml>
diff --git a/test/fixtures/presentation/main.svml b/test/fixtures/presentation/main.svml
index 3875d687..c41502d1 100644
--- a/test/fixtures/presentation/main.svml
+++ b/test/fixtures/presentation/main.svml
@@ -1,5 +1,6 @@
<svml version="1">
- <import from="../../../stdlib/speech-spine.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<import from="../../../stdlib/media-track.svk"/>
<import from="../../../stdlib/audio-track.svk"/>
<import from="../../../stdlib/film.svk"/>
@@ -11,18 +12,22 @@
</script>
<video id="long-video" src="./long.mp4"/>
+ <audio id="voice-audio" src="./long.mp4"/>
<audio id="pop-sound" src="./pop.wav"/>
+ <alignment id="alignment" src="./alignment.json"/>
- <speech-spine id="speech">
- <segment id="main" script={script.segment.main} visual={long-video}/>
- </speech-spine>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} visual={long-video} audio={voice-audio} sourceWindow="0s .. 3s"/>
+ </speech-assemble>
+
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
<media-track id="aroll" z="200">
<item id="host" source={long-video} during="full" fit="cover">
<present
id="top-circle"
during="start .. start+1.2s"
- layer="10"
+ z="210"
x="0.05"
y="0.05"
width="0.3"
@@ -32,18 +37,13 @@
<present
id="lower-split"
during="start+1s .. start+2s"
- layer="20"
+ z="220"
x="0"
y="0.5"
width="1"
height="0.5"
enter="fade 200ms"
/>
- <present
- id="outside"
- during="end+1s .. end+2s"
- layer="30"
- />
</item>
</media-track>
@@ -51,7 +51,7 @@
<item id="pop" source={pop-sound} at={script.moment.pop} duration="200ms" bus="sfx"/>
</audio-track>
- <film id="presentation" speech={speech.speech}>
+ <film id="presentation" basis={speech.production} semantic={location.map}>
<track ref={aroll.track}/>
<track ref={sfx.track}/>
</film>
diff --git a/test/fixtures/unsafe/main.svml b/test/fixtures/unsafe/main.svml
index 7b384f4d..fb265c2f 100644
--- a/test/fixtures/unsafe/main.svml
+++ b/test/fixtures/unsafe/main.svml
@@ -1,9 +1,18 @@
<svml version="1">
<import from="./unsafe.svk"/>
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
<script>
<segment id="main">Hello.</segment>
</script>
- <unsafe id="main"/>
+ <audio id="voice-audio" src="./alignment.json"/>
+ <alignment id="alignment" src="./alignment.json"/>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} audio={voice-audio} sourceWindow="0s .. 1s"/>
+ </speech-assemble>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+
+ <unsafe id="main" basis={speech.production} semantic={location.map}/>
</svml>
diff --git a/test/fixtures/unsafe/unsafe.svk b/test/fixtures/unsafe/unsafe.svk
index 8e7466e7..dbc423bd 100644
--- a/test/fixtures/unsafe/unsafe.svk
+++ b/test/fixtures/unsafe/unsafe.svk
@@ -1,3 +1,5 @@
<kernel name="unsafe" abi="1" root="true" implementation="./unsafe.mjs">
+ <port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
+ <port name="semantic" direction="input" type="ExactSemanticMap" cardinality="one"/>
<port name="document" direction="output" type="HyperframesDocument" cardinality="one"/>
</kernel>
diff --git a/test/flat-track-launch.test.ts b/test/flat-track-launch.test.ts
new file mode 100644
index 00000000..67211e76
--- /dev/null
+++ b/test/flat-track-launch.test.ts
@@ -0,0 +1,62 @@
+import assert from "node:assert/strict";
+import test from "node:test";
+import { fileURLToPath } from "node:url";
+import { compileSource } from "../src/compiler.js";
+import type { SourceElement } from "../src/model.js";
+
+const example = fileURLToPath(new URL("../examples/flat-track-launch/flat-track-launch.svml", import.meta.url));
+
+function elements(nodes: SourceElement["children"], name: string): SourceElement[] {
+ return nodes.filter((node): node is SourceElement => node.kind === "element" && node.name === name);
+}
+
+test("the flat Track launch fixture closes the complete v1 compilation contract", async () => {
+ const compilation = await compileSource({ file: example });
+
+ assert.equal(compilation.located.durationFrames, 1083);
+ assert.equal(
+ compilation.narrative.semanticIndex.anchors.length,
+ compilation.narrative.tokens.length * 2 + compilation.narrative.segments.length * 2,
+ );
+ assert.equal(compilation.located.selections.emphasis?.ranges.length, 2);
+ assert.equal(compilation.target.width, 1080);
+ assert.equal(compilation.target.height, 1920);
+
+ const arroll = compilation.plan.instances.find((instance) => instance.id === "aroll");
+ const item = elements(arroll?.children ?? [], "item")[0];
+ const openingLeft = elements(item?.children ?? [], "present")
+ .find((present) => present.attributes.id === "opening-left");
+ assert.equal(openingLeft?.attributes.shape, "circle");
+ assert.equal(openingLeft?.attributes.width, 0.3);
+
+ const overlays = compilation.plan.instances.find((instance) => instance.id === "overlays");
+ const title = elements(overlays?.children ?? [], "item")
+ .find((candidate) => candidate.attributes.id === "opening-title");
+ assert.equal(title?.attributes.size, 64);
+ assert.equal(title?.attributes.color, "#FFFFFF");
+
+ const broll = compilation.projections.get("broll");
+ const productVisual = broll?.visuals?.find((fragment) => fragment.id.includes(":product-demo:"));
+ const productAudio = broll?.audios?.find((fragment) => fragment.id.includes(":product-demo:"));
+ assert.equal(productAudio?.playbackRate, productVisual?.playbackRate);
+ assert.equal(productVisual?.muted, true);
+ assert.equal(broll?.audios?.some((fragment) => fragment.bus === "sfx"), true);
+
+ const board = compilation.projections.get("ranking")?.visuals
+ ?.find((fragment) => fragment.id === "ranking:board");
+ assert.equal(board?.startFrame, compilation.located.moments.ranking?.frames[0]);
+
+ const text = compilation.projections.get("overlays")?.visuals ?? [];
+ assert.deepEqual(
+ text.map((fragment) => [fragment.id, fragment.startFrame, fragment.endFrameExclusive]),
+ [
+ ["overlays:opening-title:0", 0, 90],
+ ["overlays:ending-arrow:0", 1023, 1083],
+ ],
+ );
+
+ const captions = compilation.projections.get("captions")?.visuals ?? [];
+ assert.ok(captions.some((fragment) => /#FDE047/u.test(fragment.html ?? "")));
+ assert.ok(captions.some((fragment) => /scale\(1\.08\)/u.test(fragment.html ?? "")));
+ assert.match(compilation.html, /data-composition-id="launch"/u);
+});
diff --git a/test/kernel-contracts.test.ts b/test/kernel-contracts.test.ts
index d4811fe5..495b7ae9 100644
--- a/test/kernel-contracts.test.ts
+++ b/test/kernel-contracts.test.ts
@@ -10,7 +10,7 @@ const brollKernel = fileURLToPath(
new URL("../stdlib/broll-track.svk", import.meta.url),
);
const unsafeKernel = fileURLToPath(
- new URL("./fixtures/unsafe/unsafe.svk", import.meta.url),
+ new URL("./fixtures/artifacts/artifact-film.svk", import.meta.url),
);
async function checkBody(body: string): Promise<ReturnType<typeof checkSource>> {
@@ -22,7 +22,7 @@ async function checkBody(body: string): Promise<ReturnType<typeof checkSource>>
<script><segment id="main">Hello.</segment></script>
<audio id="sound" src="./missing.wav"/>
${body}
- <unsafe id="root"/>
+ <artifact-film id="root" image={sound} basis={sound} semantic={sound}/>
</svml>
`, "utf8");
return checkSource(file);
diff --git a/test/modules.test.ts b/test/modules.test.ts
index 94033472..74df9f57 100644
--- a/test/modules.test.ts
+++ b/test/modules.test.ts
@@ -16,11 +16,11 @@ test(".svc content DAG and .svs parameter classes bind into one source closure",
assert.deepEqual(
checked.plan.instances.map((instance) => instance.id).sort(),
- ["brand.poster", "broll", "captions", "main-film", "speech"],
+ ["brand.poster", "broll", "captions", "location", "main-film", "speech"],
);
assert.deepEqual(
checked.plan.values.map((value) => value.id).sort(),
- ["avatar", "brand.logo"],
+ ["alignment", "avatar", "brand.logo", "voice-audio"],
);
const logo = checked.plan.values.find((value) => value.id === "brand.logo");
assert.equal(logo?.localId, "logo");
@@ -42,7 +42,6 @@ test(".svc content DAG and .svs parameter classes bind into one source closure",
const compilation = await compileSource({
file,
- evidenceFile: `${fixtures}/alignment.json`,
});
assert.match(compilation.html, /data-composition-id="main-film"/u);
assert.match(compilation.html, /#112233/u);
@@ -75,10 +74,14 @@ test("svml.lock verifies the exact source closure and Kernel implementations", a
const verified = await compileSource({
file,
- evidenceFile: `${fixtures}/alignment.json`,
lockFile: lock,
});
- assert.equal(verified.lockVerified, true);
+ assert.equal(verified.sourceClosureVerified, true);
+ assert.equal(verified.lockVerified, false);
+ await writeFile(lock, `${JSON.stringify(verified.lock, null, 2)}\n`, "utf8");
+ const frozen = await compileSource({ file, lockFile: lock });
+ assert.equal(frozen.lockVerified, true);
+ assert.equal(frozen.lock.execution?.htmlDigest.length, 64);
const changed = structuredClone(checked.lock);
changed.kernels[0]!.implementationHash = "0".repeat(64);
@@ -86,7 +89,6 @@ test("svml.lock verifies the exact source closure and Kernel implementations", a
await assert.rejects(
compileSource({
file,
- evidenceFile: `${fixtures}/alignment.json`,
lockFile: lock,
}),
(error: unknown) =>
@@ -109,7 +111,6 @@ test("asset staging uses content-addressed names and never overwrites equal base
await mkdir(directory, { recursive: true });
await compileSource({
file: `${collision}/main.svml`,
- evidenceFile: `${collision}/alignment.json`,
outputFile: output,
});
diff --git a/test/presentation.test.ts b/test/presentation.test.ts
index 9b79882b..60c990b1 100644
--- a/test/presentation.test.ts
+++ b/test/presentation.test.ts
@@ -4,11 +4,10 @@ import { fileURLToPath } from "node:url";
import { compileSource } from "../src/compiler.js";
import type { VisualFragment } from "../src/runtime-contract.js";
-test("Present ranges intersect the Item, overlap by layer, and keep one source-time map", async () => {
+test("Present ranges intersect the Item, overlap by absolute z, and keep one source-time map", async () => {
const fixture = fileURLToPath(new URL("./fixtures/presentation/", import.meta.url));
const compilation = await compileSource({
file: `${fixture}/main.svml`,
- evidenceFile: `${fixture}/alignment.json`,
});
const visuals = compilation.projections.get("aroll")?.visuals ?? [];
assert.equal(visuals.length, 3);
@@ -16,13 +15,13 @@ test("Present ranges intersect the Item, overlap by layer, and keep one source-t
visuals.map((fragment) => [
fragment.startFrame,
fragment.endFrameExclusive,
- fragment.layer,
+ fragment.z,
fragment.mediaStartSec,
]),
[
- [60, 90, 0, 2],
- [0, 36, 10, 0],
- [30, 60, 20, 1],
+ [60, 90, 200, 2],
+ [0, 36, 210, 0],
+ [30, 60, 220, 1],
],
);
assert.equal(visuals[1]?.style?.["border-radius"], "50%");
@@ -38,11 +37,18 @@ test("Present ranges intersect the Item, overlap by layer, and keep one source-t
);
});
+test("a Present with no parent intersection fails instead of disappearing", async () => {
+ const fixture = fileURLToPath(new URL("./fixtures/presentation/", import.meta.url));
+ await assert.rejects(
+ compileSource({ file: `${fixture}/empty-present.svml` }),
+ /presentation_empty/u,
+ );
+});
+
test("B-roll crossfade extends only the visual surfaces and leaves source audio independent", async () => {
const fixture = fileURLToPath(new URL("./fixtures/collision/", import.meta.url));
const compilation = await compileSource({
file: `${fixture}/main.svml`,
- evidenceFile: `${fixture}/alignment.json`,
});
const broll = compilation.projections.get("broll");
const visuals = (broll?.visuals ?? []) as VisualFragment[];
@@ -59,7 +65,6 @@ test("temporal one/each cardinality is declared by the Kernel contract", async (
const fixture = fileURLToPath(new URL("./fixtures/disconnected/", import.meta.url));
const each = await compileSource({
file: `${fixture}/each.svml`,
- evidenceFile: `${fixture}/alignment.json`,
});
assert.equal(each.projections.get("media")?.visuals?.length, 2);
const media = each.plan.kernels.find((kernel) => kernel.name === "media-track");
@@ -67,7 +72,6 @@ test("temporal one/each cardinality is declared by the Kernel contract", async (
const set = await compileSource({
file: `${fixture}/set.svml`,
- evidenceFile: `${fixture}/alignment.json`,
});
assert.deepEqual(set.projections.get("styles")?.diagnostics, [{
code: "selection_set_count",
@@ -77,8 +81,17 @@ test("temporal one/each cardinality is declared by the Kernel contract", async (
await assert.rejects(
compileSource({
file: `${fixture}/one.svml`,
- evidenceFile: `${fixture}/alignment.json`,
}),
/temporal cardinality expected one, received 2/u,
);
});
+
+test("disconnected SelectionSet drives one caption style rule without losing occurrences", async () => {
+ const fixture = fileURLToPath(new URL("./fixtures/disconnected/", import.meta.url));
+ const compilation = await compileSource({ file: `${fixture}/captions.svml` });
+ const captions = compilation.projections.get("captions")?.visuals ?? [];
+ assert.ok(captions.length > 0);
+ assert.ok(captions.some((fragment) => /color:#ff0000/u.test(fragment.html ?? "")));
+ assert.ok(captions.some((fragment) => /color:#00ff00/u.test(fragment.html ?? "")));
+ assert.ok(captions.some((fragment) => /scale\(1\.1\)/u.test(fragment.html ?? "")));
+});
diff --git a/test/regen-ranking.test.ts b/test/regen-ranking.test.ts
index a5a85e03..a6d64b0f 100644
--- a/test/regen-ranking.test.ts
+++ b/test/regen-ranking.test.ts
@@ -20,7 +20,6 @@ test("regen ranking compiles to the frozen production timing contract", async ()
) as Reference;
const compilation = await compileSource({
file: `${example}/regen-ranking.svml`,
- evidenceFile: `${example}/evidence/alignment.json`,
});
assert.equal(compilation.located.fps, reference.clock.fps);
diff --git a/test/runtime-security.test.ts b/test/runtime-security.test.ts
index 1c11f003..be80afba 100644
--- a/test/runtime-security.test.ts
+++ b/test/runtime-security.test.ts
@@ -42,7 +42,6 @@ test("isolated projector ABI rejects Node builtin imports", async () => {
await assert.rejects(
compileSource({
file: `${fixture}/main.svml`,
- evidenceFile: `${fixture}/alignment.json`,
}),
(error: unknown) =>
error instanceof SvmlError
diff --git a/test/script.test.ts b/test/script.test.ts
index fb4a4aa3..6b3271c9 100644
--- a/test/script.test.ts
+++ b/test/script.test.ts
@@ -1,11 +1,28 @@
import assert from "node:assert/strict";
import test from "node:test";
-import type { AlignmentEvidence } from "../src/model.js";
+import type { AlignmentEvidence, NarrativeIR } from "../src/model.js";
import { locateScript } from "../src/script/locate.js";
import { estimateAlignment } from "../src/script/estimate.js";
import { parseScript } from "../src/script/parse.js";
import { formatDocumentScript } from "../src/format.js";
import { parseDocumentSource } from "../src/source/parse-document.js";
+import {
+ createProgramBasis,
+ semanticMapFromAlignment,
+ validateSemanticMap,
+} from "../src/temporal.js";
+import { sha256, stableJson } from "../src/util.js";
+
+function locateAlignment(narrative: NarrativeIR, evidence: AlignmentEvidence) {
+ const basis = createProgramBasis({
+ fps: evidence.fps,
+ durationFrames: Math.round(evidence.durationSec * evidence.fps),
+ outcomeDigest: sha256(stableJson(evidence)),
+ });
+ const map = semanticMapFromAlignment(narrative, basis, evidence, { precision: "exact" });
+ assert.equal(map.contract, "svml.exact-semantic-map.v1");
+ return locateScript(narrative, basis, map);
+}
test("Script Surface projects role, dual text, selections and moments", () => {
const narrative = parseScript("fixture.svml", `
@@ -32,6 +49,130 @@ test("Script Surface projects role, dual text, selections and moments", () => {
);
});
+test("Script v1 builds independent 2M + 2N anchor identities", () => {
+ const narrative = parseScript("fixture.svml", `
+ <segment id="a">Alpha beta.</segment>
+ <segment id="b"/>
+ <segment id="c">Gamma.</segment>
+ `);
+ assert.equal(narrative.tokens.length, 3);
+ assert.equal(narrative.segments.length, 3);
+ assert.equal(narrative.semanticIndex.anchors.length, 2 * 3 + 2 * 3);
+ assert.deepEqual(
+ narrative.semanticIndex.anchors.map((anchor) => anchor.id),
+ [
+ "segment:a:start",
+ "segment:a:token:1:start",
+ "segment:a:token:1:end",
+ "segment:a:token:2:start",
+ "segment:a:token:2:end",
+ "segment:a:end",
+ "segment:b:start",
+ "segment:b:end",
+ "segment:c:start",
+ "segment:c:token:1:start",
+ "segment:c:token:1:end",
+ "segment:c:end",
+ ],
+ );
+ assert.notEqual(
+ narrative.segments[0]?.endAnchorId,
+ narrative.segments[1]?.startAnchorId,
+ );
+ assert.match(narrative.semanticIndex.digest, /^[a-f0-9]{64}$/u);
+});
+
+test("overlapping Segments retain distinct boundary affinity on one ProgramBasis", () => {
+ const narrative = parseScript("fixture.svml", `
+ <segment id="a">Alpha.</segment>
+ ~@left! @right!
+ <segment id="b">Beta.</segment>
+ `);
+ const evidence: AlignmentEvidence = {
+ contract: "svml.speech-alignment.v1",
+ durationSec: 8,
+ fps: 30,
+ words: [
+ { text: "Alpha", startSec: 0.2, endSec: 4.8, segmentId: "a" },
+ { text: "Beta", startSec: 4.6, endSec: 7.8, segmentId: "b" },
+ ],
+ segments: [
+ { id: "a", startSec: 0, endSec: 5 },
+ { id: "b", startSec: 4.5, endSec: 8 },
+ ],
+ };
+ const located = locateAlignment(narrative, evidence);
+ assert.deepEqual(located.moments.left?.frames, [150]);
+ assert.deepEqual(located.moments.right?.frames, [135]);
+ assert.equal(located.segments.a?.endFrameExclusive, 150);
+ assert.equal(located.segments.b?.startFrame, 135);
+});
+
+test("SemanticMap rejects source-order reversal, wrong basis and digest tampering", () => {
+ const narrative = parseScript(
+ "fixture.svml",
+ `<segment id="a">Alpha.</segment><segment id="b">Beta.</segment>`,
+ );
+ const evidence: AlignmentEvidence = {
+ contract: "svml.speech-alignment.v1",
+ durationSec: 5,
+ fps: 30,
+ words: [
+ { text: "Alpha", startSec: 1.1, endSec: 1.8, segmentId: "a" },
+ { text: "Beta", startSec: 0.2, endSec: 0.8, segmentId: "b" },
+ ],
+ segments: [
+ { id: "a", startSec: 1, endSec: 2 },
+ { id: "b", startSec: 0, endSec: 1 },
+ ],
+ };
+ const basis = createProgramBasis({
+ fps: 30,
+ durationFrames: 150,
+ outcomeDigest: sha256("reordered"),
+ });
+ assert.throws(
+ () => semanticMapFromAlignment(narrative, basis, evidence, { precision: "exact" }),
+ /locator_segment_order/u,
+ );
+
+ const validEvidence: AlignmentEvidence = {
+ ...evidence,
+ words: [
+ { text: "Alpha", startSec: 0.1, endSec: 0.8, segmentId: "a" },
+ { text: "Beta", startSec: 1.1, endSec: 1.8, segmentId: "b" },
+ ],
+ segments: [
+ { id: "a", startSec: 0, endSec: 1 },
+ { id: "b", startSec: 1, endSec: 2 },
+ ],
+ };
+ const validMap = semanticMapFromAlignment(
+ narrative,
+ basis,
+ validEvidence,
+ { precision: "exact" },
+ );
+ assert.throws(
+ () => validateSemanticMap(narrative, basis, {
+ ...validMap,
+ anchors: validMap.anchors.map((anchor, index) => index === 0
+ ? { ...anchor, point: { ...anchor.point, frame: anchor.point.frame + 1 } }
+ : anchor),
+ }),
+ /semantic_map_digest/u,
+ );
+ const otherBasis = createProgramBasis({
+ fps: 30,
+ durationFrames: 150,
+ outcomeDigest: sha256("other"),
+ });
+ assert.throws(
+ () => validateSemanticMap(narrative, otherBasis, validMap),
+ /semantic_map_basis/u,
+ );
+});
+
test("Dual Text owns its complete speech span and caption cues cannot split it", () => {
const narrative = parseScript(
"fixture.svml",
@@ -51,7 +192,7 @@ test("Dual Text owns its complete speech span and caption cues cannot split it",
],
segments: [{ id: "one", startSec: 0, endSec: 2 }],
};
- const located = locateScript(narrative, evidence);
+ const located = locateAlignment(narrative, evidence);
assert.deepEqual(
located.captionAtoms.map((atom) => [
atom.display,
@@ -61,7 +202,7 @@ test("Dual Text owns its complete speech span and caption cues cannot split it",
[["lmao", 15, 51]],
);
assert.throws(
- () => locateScript(narrative, {
+ () => locateAlignment(narrative, {
...evidence,
captionCues: [
{ startWord: 0, endWordExclusive: 4 },
@@ -139,7 +280,7 @@ test("caption cue evidence is a complete ordered word partition", () => {
{ id: "payoff", startWord: 1, endWordExclusive: 3 },
],
};
- const located = locateScript(narrative, evidence);
+ const located = locateAlignment(narrative, evidence);
assert.deepEqual(
located.captionCues?.map((cue) => [
cue.id,
@@ -155,7 +296,7 @@ test("caption cue evidence is a complete ordered word partition", () => {
);
assert.throws(
- () => locateScript(narrative, {
+ () => locateAlignment(narrative, {
...evidence,
captionCues: [{ startWord: 1, endWordExclusive: 3 }],
}),
@@ -163,12 +304,12 @@ test("caption cue evidence is a complete ordered word partition", () => {
);
assert.throws(
- () => locateScript(narrative, {
+ () => locateAlignment(narrative, {
...evidence,
words: evidence.words.map((word, index) =>
index === 1 ? { ...word, startSec: 0.2 } : word),
}),
- /locate_alignment_word_overlap/u,
+ /locator_word_overlap/u,
);
});