Skip to content

armgit / hypit

Public Imported from GitHub
0

Clone any viral video with AI agents. Not just a script, the whole workflow: swap the face, the words, the B-roll, ship 100 variants in one command, and get your 100M views.

feat: make Script authoritative for speech timing

kashorin07b2924cc37a
diff --git a/README.md b/README.md
index c322f81c..348d2a56 100644
--- a/README.md
+++ b/README.md
@@ -11,7 +11,7 @@ HyperFrames HTML document.
│
├─ typed Plan IR ─────────────────────────────> Canvas view
├─ Basis Producer ─────> TemporalBasisProduction
- ├─ Locator ────────────> ExactSemanticMap
+ ├─ Locator ────────────> CompleteSemanticMap
├─ Basis + Map ────────> TemporalBinding ─────> Timeline view
└─ flat Track[] ───────> Composition ─────────> HyperFrames HTML
```
@@ -29,8 +29,10 @@ architecture end to end:
Selections and zero-width Moments;
- an exact `SemanticIndex` with independent word and Segment endpoints
(`2M + 2N` identities);
-- replaceable `TemporalBasisProduction` and `ExactSemanticMap` Components,
+- replaceable `TemporalBasisProduction` and `CompleteSemanticMap` Components,
validated and bound before any Track lowering;
+- direct monotonic many-to-many alignment from authoritative Script tokens to
+ noisy `SpeechTimingEvidence`, without a corrected-transcript stage;
- hard cut, gap and audio/visual crossfade joins with distinct source-to-program
maps;
- typed `.svc` content modules and `.svs` classes, including nested Component
@@ -40,6 +42,8 @@ architecture end to end:
- isolated `.svk` projectors, verified offline capability artifacts and finite,
auditable `composite-v1` expansion;
- manifest-enforced temporal `one` / `each` / `set` consumption;
+- a post-locate `CaptionPlan` that may group cues and add typed style annotations
+ but cannot rewrite Script text or timing;
- flat media, ranking, B-roll, caption, text and audio Tracks in one absolute
ProgramSpace;
- Film as the only `Track[]` consumer and HyperFrames HTML as the sole formal
@@ -81,10 +85,10 @@ pnpm svml render build/index.html --out build/video.mp4
```
`--artifacts` is required only when the reachable Plan contains capability
-Components. Exact alignment is an explicit typed value referenced by the
-Locator in source; it is not an ambient CLI argument. `estimate` returns an
-`EstimatedSemanticMap` for preview and cannot be supplied where final HTML
-requires an `ExactSemanticMap`.
+Components. Speech timing is an explicit typed value referenced by the Locator
+in source; it is not an ambient CLI argument. `estimate` produces estimated
+timing evidence and the same structurally complete `CompleteSemanticMap` type;
+per-anchor quality records what was measured, derived or estimated.
`fmt` currently canonicalizes only the Script body and refuses a rewrite if the
reparsed semantic IR differs. `canvas` needs no materialized time evidence.
@@ -108,8 +112,8 @@ Canvas owns only a human-readable view.
- [`examples/regen-ranking/regen-ranking.svml`](examples/regen-ranking/regen-ranking.svml)
reconstructs a pinned production video with five ranking items, five B-roll
- items, captions, title, BGM and sound effects. It compiles to 29 Plan nodes,
- 33 typed edges, 1083 frames, 153 visual fragments and 14 audio fragments.
+ items, captions, title, BGM and sound effects. It compiles to 30 Plan nodes,
+ 35 typed edges and 1083 frames.
- [`examples/flat-track-launch/flat-track-launch.svml`](examples/flat-track-launch/flat-track-launch.svml)
is the comprehensive language fixture: explicit Basis/Locator, nested SVS,
overlapping Presents, manual ProgramSpans, a Moment, B-roll source audio and
diff --git a/examples/composite-speech-program/README.md b/examples/composite-speech-program/README.md
index 1d416eb5..054c5345 100644
--- a/examples/composite-speech-program/README.md
+++ b/examples/composite-speech-program/README.md
@@ -13,7 +13,7 @@ For `<speech-program id="voice">`, expansion creates:
```text
voice::basis → TemporalBasisProduction + ProgramBoundVideoSequence
│
- └──────────────> voice::locator → ExactSemanticMap
+ └──────────────> voice::locator → CompleteSemanticMap
voice.production = voice::basis.production
voice.map = voice::locator.map
diff --git a/examples/composite-speech-program/alignment.json b/examples/composite-speech-program/alignment.json
index 5cc43427..a23de902 100644
--- a/examples/composite-speech-program/alignment.json
+++ b/examples/composite-speech-program/alignment.json
@@ -1,8 +1,9 @@
{
- "contract": "svml.speech-alignment.v1",
+ "contract": "svml.speech-timing-evidence.v1",
"durationSec": 9.5,
"fps": 30,
- "words": [
+ "quality": "measured",
+ "units": [
{ "text": "Hello", "startSec": 0.2, "endSec": 0.8, "segmentId": "intro" },
{ "text": "this", "startSec": 1.0, "endSec": 1.5, "segmentId": "intro" },
{ "text": "is", "startSec": 1.6, "endSec": 1.9, "segmentId": "intro" },
diff --git a/examples/composite-speech-program/minimal.svml b/examples/composite-speech-program/minimal.svml
index a15d9958..70370f95 100644
--- a/examples/composite-speech-program/minimal.svml
+++ b/examples/composite-speech-program/minimal.svml
@@ -16,9 +16,9 @@
<video id="host" src="../regen-ranking/assets/avatar-1.mp4"/>
<audio id="host-audio" src="../regen-ranking/assets/avatar-1.mp3"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
- <speech-program id="voice" script={script} evidence={alignment} class="house.default">
+ <speech-program id="voice" script={script} evidence={timing} class="house.default">
<segment id="intro" visual={host} audio={host-audio} sourceWindow="0s .. 5s"/>
<segment id="close" visual={host} audio={host-audio} sourceWindow="5s .. 10s"/>
<join after="intro" overlap="500ms" audio="crossfade" visual="dissolve"/>
diff --git a/examples/composite-speech-program/speech-program.svk b/examples/composite-speech-program/speech-program.svk
index 0dc9970f..8b1277db 100644
--- a/examples/composite-speech-program/speech-program.svk
+++ b/examples/composite-speech-program/speech-program.svk
@@ -1,8 +1,8 @@
<kernel name="speech-program" abi="1" profile="composite-v1">
<port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
- <port name="evidence" direction="input" type="AlignmentEvidence" cardinality="one"/>
+ <port name="evidence" direction="input" type="SpeechTimingEvidence" cardinality="one"/>
<port name="production" direction="output" type="TemporalBasisProduction" cardinality="one"/>
- <port name="map" direction="output" type="ExactSemanticMap" cardinality="one"/>
+ <port name="map" direction="output" type="CompleteSemanticMap" cardinality="one"/>
<port name="facets.visual" direction="output" type="ProgramBoundVideoSequence" cardinality="one"/>
<param name="join" type="enum(cut,crossfade,gap)" default="cut"/>
diff --git a/examples/flat-track-launch/README.md b/examples/flat-track-launch/README.md
index 4984925a..d6db0113 100644
--- a/examples/flat-track-launch/README.md
+++ b/examples/flat-track-launch/README.md
@@ -5,14 +5,14 @@ providers and exercises the contracts together:
- independent Script Segment endpoints (`2M + 2N`);
- replaceable Basis Producer and Locator Components;
-- one explicit `ProgramBasis` plus `ExactSemanticMap`;
+- one explicit `ProgramBasis` plus `CompleteSemanticMap`;
- flat Tracks and absolute `z`, with Film as the only `Track[]` consumer;
- one continuous A-roll mapping with overlapping Present windows;
- manual `ProgramSpan`, a `Moment`, B-roll audio and a visual crossfade;
- a disconnected SelectionSet driving one caption style rule;
- top-level and nested Component fields configured by `.svs`.
-It reuses the checked-in local artifacts and exact alignment evidence from the
+It reuses the checked-in local artifacts and measured speech timing evidence from the
adjacent `regen-ranking` example. No provider is called.
```sh
diff --git a/examples/flat-track-launch/flat-track-launch.svml b/examples/flat-track-launch/flat-track-launch.svml
index a3d82c5f..2a513e20 100644
--- a/examples/flat-track-launch/flat-track-launch.svml
+++ b/examples/flat-track-launch/flat-track-launch.svml
@@ -6,6 +6,7 @@
<import from="../../stdlib/broll-track.svk"/>
<import from="../../stdlib/ranking-column.svk"/>
<import from="../../stdlib/text-track.svk"/>
+ <import from="../../stdlib/caption-planner.svk"/>
<import from="../../stdlib/caption-track.svk"/>
<import from="../../stdlib/audio-track.svk"/>
<import from="../../stdlib/film.svk"/>
@@ -61,7 +62,7 @@
<audio id="ranking-appear" src="../regen-ranking/assets/ranking-appear.wav"/>
<audio id="ranking-move" src="../regen-ranking/assets/ranking-move.wav"/>
<audio id="music" src="../regen-ranking/assets/background.mp3"/>
- <alignment id="alignment" src="../regen-ranking/evidence/alignment.json"/>
+ <timing id="timing" src="../regen-ranking/evidence/alignment.json"/>
<!-- The Basis Producer and Locator are replaceable Components with typed outputs. -->
<speech-assemble id="voice" fps="30" defaultJoin="cut">
@@ -69,7 +70,7 @@
<segment id="scene-2" script={script.segment.scene-2} visual={avatar-2} audio={avatar-2-audio} sourceWindow="0s .. 12.033333333333333s"/>
<segment id="scene-3" script={script.segment.scene-3} visual={avatar-3} audio={avatar-3-audio} sourceWindow="0s .. 12.033333333333333s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={voice.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={voice.production} evidence={timing}/>
<!-- One source mapping; Present only changes its projection in ProgramSpace. -->
<media-track id="aroll" z="20">
@@ -107,8 +108,10 @@
</text-track>
<!-- The repeated emphasis Selection is disconnected and consumed as one set. -->
- <caption-track id="captions" z="500" script={script} semantic={location.map} class="house.captions">
+ <caption-planner id="caption-plan" script={script} semantic={location.map} maxWords="3"/>
+ <caption-track id="captions" z="500" script={script} semantic={location.map} plan={caption-plan.plan} class="house.captions">
<style id="emphasis" during={script.selection.emphasis} class="house.caption-emphasis"/>
+ <style id="narrator" role="NARRATOR" activeColor="#73fbd3"/>
</caption-track>
<audio-track id="background-music">
diff --git a/examples/regen-ranking/evidence/alignment.json b/examples/regen-ranking/evidence/alignment.json
index b2b45d70..fd7153c2 100644
--- a/examples/regen-ranking/evidence/alignment.json
+++ b/examples/regen-ranking/evidence/alignment.json
@@ -1,35 +1,42 @@
{
- "contract": "svml.speech-alignment.v1",
+ "contract": "svml.speech-timing-evidence.v1",
"durationSec": 36.1,
"fps": 30,
- "words": [
+ "quality": "measured",
+ "units": [
{
- "text": "Photoshop",
+ "text": "Photo shop",
"startSec": 0.08,
"endSec": 1,
"segmentId": "scene-1"
},
{
- "text": "Powerful",
+ "text": "power full but",
"startSec": 1,
- "endSec": 1.53,
+ "endSec": 1.9,
"segmentId": "scene-1"
},
{
- "text": "but",
- "startSec": 1.53,
- "endSec": 1.9,
+ "text": "on",
+ "startSec": 1.9,
+ "endSec": 1.945,
"segmentId": "scene-1"
},
{
- "text": "only",
- "startSec": 1.9,
+ "text": "lee",
+ "startSec": 1.945,
"endSec": 1.99,
"segmentId": "scene-1"
},
{
- "text": "if",
+ "text": "of",
"startSec": 1.99,
+ "endSec": 1.995,
+ "segmentId": "scene-1"
+ },
+ {
+ "text": "uh",
+ "startSec": 1.995,
"endSec": 2,
"segmentId": "scene-1"
},
@@ -832,295 +839,12 @@
}
],
"provenance": {
- "method": "whisper.cpp word timestamps + deterministic intended-script LCS projection",
- "source": "existing pinned avatar audio; no generation call",
- "captionCues": {
- "kind": "reference-recovery",
- "source": "regen ranking Job cmrxqbub800012tjoi2ev2u24",
- "note": "Cue boundaries are evidence emitted by Locate, not authored SVML timing."
- },
+ "method": "pinned speech timing with deliberate recognition noise",
+ "source": "existing avatar audio; wording is measurement evidence, never corrected transcript truth",
"timingOverrides": {
"kind": "reference-recovery",
"source": "scene transitions in regen ranking Job cmrxqbub800012tjoi2ev2u24",
"note": "Only boundaries visible in the frozen reference video are overridden."
}
- },
- "captionCues": [
- {
- "startWord": 0,
- "endWordExclusive": 1
- },
- {
- "startWord": 1,
- "endWordExclusive": 2
- },
- {
- "startWord": 2,
- "endWordExclusive": 5
- },
- {
- "startWord": 5,
- "endWordExclusive": 8
- },
- {
- "startWord": 8,
- "endWordExclusive": 11
- },
- {
- "startWord": 11,
- "endWordExclusive": 12
- },
- {
- "startWord": 12,
- "endWordExclusive": 14
- },
- {
- "startWord": 14,
- "endWordExclusive": 16
- },
- {
- "startWord": 16,
- "endWordExclusive": 18
- },
- {
- "startWord": 18,
- "endWordExclusive": 20
- },
- {
- "startWord": 20,
- "endWordExclusive": 21
- },
- {
- "startWord": 21,
- "endWordExclusive": 22
- },
- {
- "startWord": 22,
- "endWordExclusive": 24
- },
- {
- "startWord": 24,
- "endWordExclusive": 26
- },
- {
- "startWord": 26,
- "endWordExclusive": 27
- },
- {
- "startWord": 27,
- "endWordExclusive": 28
- },
- {
- "startWord": 28,
- "endWordExclusive": 31
- },
- {
- "startWord": 31,
- "endWordExclusive": 34
- },
- {
- "startWord": 34,
- "endWordExclusive": 35
- },
- {
- "startWord": 35,
- "endWordExclusive": 38
- },
- {
- "startWord": 38,
- "endWordExclusive": 40
- },
- {
- "startWord": 40,
- "endWordExclusive": 42
- },
- {
- "startWord": 42,
- "endWordExclusive": 43
- },
- {
- "startWord": 43,
- "endWordExclusive": 44
- },
- {
- "startWord": 44,
- "endWordExclusive": 47
- },
- {
- "startWord": 47,
- "endWordExclusive": 50
- },
- {
- "startWord": 50,
- "endWordExclusive": 53
- },
- {
- "startWord": 53,
- "endWordExclusive": 56
- },
- {
- "startWord": 56,
- "endWordExclusive": 57
- },
- {
- "startWord": 57,
- "endWordExclusive": 59
- },
- {
- "startWord": 59,
- "endWordExclusive": 62
- },
- {
- "startWord": 62,
- "endWordExclusive": 65
- },
- {
- "startWord": 65,
- "endWordExclusive": 67
- },
- {
- "startWord": 67,
- "endWordExclusive": 68
- },
- {
- "startWord": 68,
- "endWordExclusive": 70
- },
- {
- "startWord": 70,
- "endWordExclusive": 73
- },
- {
- "startWord": 73,
- "endWordExclusive": 75
- },
- {
- "startWord": 75,
- "endWordExclusive": 76
- },
- {
- "startWord": 76,
- "endWordExclusive": 77
- },
- {
- "startWord": 77,
- "endWordExclusive": 80
- },
- {
- "startWord": 80,
- "endWordExclusive": 83
- },
- {
- "startWord": 83,
- "endWordExclusive": 86
- },
- {
- "startWord": 86,
- "endWordExclusive": 87
- },
- {
- "startWord": 87,
- "endWordExclusive": 89
- },
- {
- "startWord": 89,
- "endWordExclusive": 90
- },
- {
- "startWord": 90,
- "endWordExclusive": 93
- },
- {
- "startWord": 93,
- "endWordExclusive": 95
- },
- {
- "startWord": 95,
- "endWordExclusive": 98
- },
- {
- "startWord": 98,
- "endWordExclusive": 100
- },
- {
- "startWord": 100,
- "endWordExclusive": 103
- },
- {
- "startWord": 103,
- "endWordExclusive": 105
- },
- {
- "startWord": 105,
- "endWordExclusive": 106
- },
- {
- "startWord": 106,
- "endWordExclusive": 107
- },
- {
- "startWord": 107,
- "endWordExclusive": 108
- },
- {
- "startWord": 108,
- "endWordExclusive": 109
- },
- {
- "startWord": 109,
- "endWordExclusive": 111
- },
- {
- "startWord": 111,
- "endWordExclusive": 112
- },
- {
- "startWord": 112,
- "endWordExclusive": 114
- },
- {
- "startWord": 114,
- "endWordExclusive": 116
- },
- {
- "startWord": 116,
- "endWordExclusive": 118
- },
- {
- "startWord": 118,
- "endWordExclusive": 120
- },
- {
- "startWord": 120,
- "endWordExclusive": 122
- },
- {
- "startWord": 122,
- "endWordExclusive": 123
- },
- {
- "startWord": 123,
- "endWordExclusive": 125
- },
- {
- "startWord": 125,
- "endWordExclusive": 127
- },
- {
- "startWord": 127,
- "endWordExclusive": 129
- },
- {
- "startWord": 129,
- "endWordExclusive": 131
- },
- {
- "startWord": 131,
- "endWordExclusive": 133
- },
- {
- "startWord": 133,
- "endWordExclusive": 135
- }
- ]
+ }
}
diff --git a/examples/regen-ranking/evidence/caption-cues.json b/examples/regen-ranking/evidence/caption-cues.json
deleted file mode 100644
index 232bdad8..00000000
--- a/examples/regen-ranking/evidence/caption-cues.json
+++ /dev/null
@@ -1,81 +0,0 @@
-{
- "contract": "svml.caption-cues.v1",
- "captionCues": [
- { "startWord": 0, "endWordExclusive": 1 },
- { "startWord": 1, "endWordExclusive": 2 },
- { "startWord": 2, "endWordExclusive": 5 },
- { "startWord": 5, "endWordExclusive": 8 },
- { "startWord": 8, "endWordExclusive": 11 },
- { "startWord": 11, "endWordExclusive": 12 },
- { "startWord": 12, "endWordExclusive": 14 },
- { "startWord": 14, "endWordExclusive": 16 },
- { "startWord": 16, "endWordExclusive": 18 },
- { "startWord": 18, "endWordExclusive": 20 },
- { "startWord": 20, "endWordExclusive": 21 },
- { "startWord": 21, "endWordExclusive": 22 },
- { "startWord": 22, "endWordExclusive": 24 },
- { "startWord": 24, "endWordExclusive": 26 },
- { "startWord": 26, "endWordExclusive": 27 },
- { "startWord": 27, "endWordExclusive": 28 },
- { "startWord": 28, "endWordExclusive": 31 },
- { "startWord": 31, "endWordExclusive": 34 },
- { "startWord": 34, "endWordExclusive": 35 },
- { "startWord": 35, "endWordExclusive": 38 },
- { "startWord": 38, "endWordExclusive": 40 },
- { "startWord": 40, "endWordExclusive": 42 },
-
- { "startWord": 42, "endWordExclusive": 43 },
- { "startWord": 43, "endWordExclusive": 44 },
- { "startWord": 44, "endWordExclusive": 47 },
- { "startWord": 47, "endWordExclusive": 50 },
- { "startWord": 50, "endWordExclusive": 53 },
- { "startWord": 53, "endWordExclusive": 56 },
- { "startWord": 56, "endWordExclusive": 57 },
- { "startWord": 57, "endWordExclusive": 59 },
- { "startWord": 59, "endWordExclusive": 62 },
- { "startWord": 62, "endWordExclusive": 65 },
- { "startWord": 65, "endWordExclusive": 67 },
- { "startWord": 67, "endWordExclusive": 68 },
- { "startWord": 68, "endWordExclusive": 70 },
- { "startWord": 70, "endWordExclusive": 73 },
- { "startWord": 73, "endWordExclusive": 75 },
- { "startWord": 75, "endWordExclusive": 76 },
- { "startWord": 76, "endWordExclusive": 77 },
- { "startWord": 77, "endWordExclusive": 80 },
- { "startWord": 80, "endWordExclusive": 83 },
- { "startWord": 83, "endWordExclusive": 86 },
- { "startWord": 86, "endWordExclusive": 87 },
- { "startWord": 87, "endWordExclusive": 89 },
-
- { "startWord": 89, "endWordExclusive": 90 },
- { "startWord": 90, "endWordExclusive": 93 },
- { "startWord": 93, "endWordExclusive": 95 },
- { "startWord": 95, "endWordExclusive": 98 },
- { "startWord": 98, "endWordExclusive": 100 },
- { "startWord": 100, "endWordExclusive": 103 },
- { "startWord": 103, "endWordExclusive": 105 },
- { "startWord": 105, "endWordExclusive": 106 },
- { "startWord": 106, "endWordExclusive": 107 },
- { "startWord": 107, "endWordExclusive": 108 },
- { "startWord": 108, "endWordExclusive": 109 },
- { "startWord": 109, "endWordExclusive": 111 },
- { "startWord": 111, "endWordExclusive": 112 },
- { "startWord": 112, "endWordExclusive": 114 },
- { "startWord": 114, "endWordExclusive": 116 },
- { "startWord": 116, "endWordExclusive": 118 },
- { "startWord": 118, "endWordExclusive": 120 },
- { "startWord": 120, "endWordExclusive": 122 },
- { "startWord": 122, "endWordExclusive": 123 },
- { "startWord": 123, "endWordExclusive": 125 },
- { "startWord": 125, "endWordExclusive": 127 },
- { "startWord": 127, "endWordExclusive": 129 },
- { "startWord": 129, "endWordExclusive": 131 },
- { "startWord": 131, "endWordExclusive": 133 },
- { "startWord": 133, "endWordExclusive": 135 }
- ],
- "provenance": {
- "kind": "reference-recovery",
- "source": "regen ranking Job cmrxqbub800012tjoi2ev2u24",
- "note": "Cue boundaries are evidence emitted by Locate, not authored SVML timing."
- }
-}
diff --git a/examples/regen-ranking/evidence/timing-overrides.json b/examples/regen-ranking/evidence/timing-overrides.json
index 548b1138..405808b8 100644
--- a/examples/regen-ranking/evidence/timing-overrides.json
+++ b/examples/regen-ranking/evidence/timing-overrides.json
@@ -1,6 +1,6 @@
{
- "contract": "svml.alignment-overrides.v1",
- "words": [
+ "contract": "svml.speech-timing-overrides.v1",
+ "units": [
{ "index": 69, "endSec": 20.000 },
{ "index": 70, "startSec": 20.000, "endSec": 20.033 },
{ "index": 71, "startSec": 20.033, "endSec": 20.133 },
diff --git a/examples/regen-ranking/reference.json b/examples/regen-ranking/reference.json
index 9360a94e..cc68f6fe 100644
--- a/examples/regen-ranking/reference.json
+++ b/examples/regen-ranking/reference.json
@@ -15,8 +15,8 @@
"plan": {
"root": "regen-ranking",
"values": 20,
- "instances": 9,
- "edges": 33
+ "instances": 10,
+ "edges": 35
},
"selectionFrames": {
"handsome-1": [[791, 831]],
@@ -26,7 +26,7 @@
"instagram-post": [[977, 1009]]
},
"locatedOutput": {
- "visualFragments": 153,
+ "visualFragments": 154,
"audioFragments": 14
},
"renderComparison": {
diff --git a/examples/regen-ranking/regen-ranking.svml b/examples/regen-ranking/regen-ranking.svml
index 5149f76a..e1a9ce76 100644
--- a/examples/regen-ranking/regen-ranking.svml
+++ b/examples/regen-ranking/regen-ranking.svml
@@ -6,6 +6,7 @@
<import from="../../stdlib/ranking-column.svk"/>
<import from="../../stdlib/broll-track.svk"/>
<import from="../../stdlib/text-track.svk"/>
+ <import from="../../stdlib/caption-planner.svk"/>
<import from="../../stdlib/caption-track.svk"/>
<import from="../../stdlib/audio-track.svk"/>
<import from="../../stdlib/film.svk"/>
@@ -62,7 +63,7 @@
<audio id="ranking-appear" src="./assets/ranking-appear.wav"/>
<audio id="ranking-move" src="./assets/ranking-move.wav"/>
<audio id="background-music" src="./assets/background.mp3"/>
- <alignment id="alignment" src="./evidence/alignment.json"/>
+ <timing id="timing" src="./evidence/alignment.json"/>
<speech-assemble id="voice" fps="30" defaultJoin="cut">
<segment id="scene-1" script={script.segment.scene-1} visual={avatar-1} audio={avatar-1-audio} sourceWindow="0s .. 12.033333333333333s"/>
@@ -70,7 +71,7 @@
<segment id="scene-3" script={script.segment.scene-3} visual={avatar-3} audio={avatar-3-audio} sourceWindow="0s .. 12.033333333333333s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={voice.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={voice.production} evidence={timing}/>
<media-track id="base" z="20">
<item id="scene-1" source={voice.segment.scene-1.visual} fit="cover"/>
@@ -129,15 +130,17 @@
<item id="instagram-post" source={broll-1} during={script.selection.instagram-post} fit="contain" zoom="1.02"/>
</broll-track>
+ <caption-planner id="caption-plan" script={script} semantic={location.map} maxWords="3"/>
+
<caption-track
id="captions"
script={script}
semantic={location.map}
+ plan={caption-plan.plan}
z="500"
x="0.37"
y="0.54"
width="0.64"
- maxWords="3"
hold="0.08"
font="Poppins, Inter, Arial, sans-serif"
size="50"
diff --git a/spec/script-surface-v1.md b/spec/script-surface-v1.md
index 4f44850a..c150c64c 100644
--- a/spec/script-surface-v1.md
+++ b/spec/script-surface-v1.md
@@ -356,9 +356,9 @@ gap B.start > A.end
若未来需要真正重排或并行 speech,应发布显式的非线性叙事模型;v1 不让普通
Range 在不同 Locator 下反向或消失。
-同一份 SelectionSet 可先在 `EstimatedSemanticMap` 上得到预览区间,再在
-`ExactSemanticMap` 上得到成片区间。两者使用完全相同的 identity、不同的精度
-类型。Script 本身不含秒数、帧号或采样点。
+同一份 SelectionSet 在不同 fulfillment 产生的 `CompleteSemanticMap` 上可得到
+预览或成片区间。Map 使用相同 identity;每个 anchor 以 `measured`、`derived` 或
+`estimated` 记录证据质量。Script 本身不含秒数、帧号或采样点。
Script Surface 不绑定帧率、采样率或渲染器。后端一旦选择物理时钟,必须只
量化一次并让所有消费者复用同一整数边界;后端时钟变化不改变本语言表面。
@@ -413,7 +413,7 @@ overlap 或 gap 落在不同点。
Dual Text 的 speech 侧,但均不得切入 v1 speech token。
- Moment 只选择 Semantic Anchor Index 中已有的一个候选点,不新增 anchor
identity;因此 `2M + 2N` 的精确计数不因 Moment 数量改变。
-- MomentSet 与 SelectionSet 复用同一份 Estimated/Exact SemanticMap 和一次性
+- MomentSet 与 SelectionSet 复用同一份 CompleteSemanticMap 和一次性
帧量化;区别只在最终载体是 `Point[]` 而不是 `Range[]`。
消费端保持两个互不相混的类型:
diff --git a/src/cli.ts b/src/cli.ts
index 972b13c0..266dcb85 100644
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -169,7 +169,7 @@ async function run(): Promise<void> {
return;
}
if (args.command === "estimate") {
- if (!file) throw new Error("usage: svml estimate <file.svml> [--out alignment.json]");
+ if (!file) throw new Error("usage: svml estimate <file.svml> [--out timing.json]");
const numeric = (name: string): number | undefined => {
const value = option(args, name);
if (value === undefined) return undefined;
@@ -188,9 +188,6 @@ async function run(): Promise<void> {
...(numeric("empty-segment-duration") !== undefined
? { emptySegmentSec: numeric("empty-segment-duration")! }
: {}),
- ...(numeric("max-caption-words") !== undefined
- ? { maxCaptionWords: numeric("max-caption-words")! }
- : {}),
...(numeric("fps") !== undefined ? { fps: numeric("fps")! } : {}),
});
const payload = {
@@ -288,7 +285,7 @@ usage:
svml script <file.svml> [--out narrative.json]
svml plan <file.svml> [--out plan.json]
svml fmt <file.svml> [--check|--write|--out formatted.svml]
- svml estimate <file.svml> [--out alignment.json] [--fps 30]
+ svml estimate <file.svml> [--out timing.json] [--fps 30]
svml canvas <file.svml> [--out canvas.json]
svml timeline <file.svml> [--out timeline.json] [--lock svml.lock] [--artifacts artifacts.json]
svml compile <file.svml> --out index.html [--lock svml.lock] [--artifacts artifacts.json]
diff --git a/src/compiler.ts b/src/compiler.ts
index 1f753ff8..78cde6dd 100644
--- a/src/compiler.ts
+++ b/src/compiler.ts
@@ -16,9 +16,9 @@ import {
} from "./kernel.js";
import { createLock, freezeLock, verifyLockFile, type SvmlLock } from "./lock.js";
import type {
- AlignmentEvidence,
+ CompleteSemanticMap,
AttributeValue,
- ExactSemanticMap,
+ SpeechTimingEvidence,
LocatedScript,
PlanIR,
SourceDocument,
@@ -34,7 +34,7 @@ import type {
import { projectKernelIsolated } from "./sandbox.js";
import { validateKernelProjection } from "./runtime-validation.js";
import { bindSemanticMap, locateScript } from "./script/locate.js";
-import { estimateAlignment, type EstimateOptions } from "./script/estimate.js";
+import { estimateSpeechTiming, type EstimateOptions } from "./script/estimate.js";
import { parseScript } from "./script/parse.js";
import { loadSourceClosure } from "./source/modules.js";
import { childElements, textContent } from "./source/query.js";
@@ -42,7 +42,7 @@ import { applyStyleSheets } from "./source/styles.js";
import { sha256, stableJson } from "./util.js";
import {
createProgramBasis,
- semanticMapFromAlignment,
+ semanticMapFromTiming,
validateSemanticMap,
validateTemporalBasisProduction,
} from "./temporal.js";
@@ -52,7 +52,7 @@ export type Compilation = {
narrative: ReturnType<typeof parseScript>;
plan: PlanIR;
basis: TemporalBasisProduction;
- semanticMap: ExactSemanticMap;
+ semanticMap: CompleteSemanticMap;
located: LocatedScript;
html: string;
target: HyperframesDocument;
@@ -211,13 +211,13 @@ export async function estimateSource(
file: string,
options: EstimateOptions = {},
): Promise<{
- evidence: AlignmentEvidence;
+ evidence: SpeechTimingEvidence;
basis: ReturnType<typeof createProgramBasis>;
- map: ReturnType<typeof semanticMapFromAlignment>;
+ map: ReturnType<typeof semanticMapFromTiming>;
located: LocatedScript;
}> {
const checked = await checkSource(file);
- const evidence = estimateAlignment(checked.narrative, options);
+ const evidence = estimateSpeechTiming(checked.narrative, options);
const basis = createProgramBasis({
fps: evidence.fps,
durationFrames: Math.round(evidence.durationSec * evidence.fps),
@@ -226,11 +226,10 @@ export async function estimateSource(
evidence,
})),
});
- const map = semanticMapFromAlignment(
+ const map = semanticMapFromTiming(
checked.narrative,
basis,
evidence,
- { precision: "estimated" },
);
return {
evidence,
@@ -264,7 +263,7 @@ export async function compileSource(args: {
const basisPort = rootManifest.ports.find((port) =>
port.direction === "input" && port.type.split("|").includes("TemporalBasisProduction"));
const semanticPort = rootManifest.ports.find((port) =>
- port.direction === "input" && port.type.split("|").includes("ExactSemanticMap"));
+ port.direction === "input" && port.type.split("|").includes("CompleteSemanticMap"));
if (!basisPort || !semanticPort) {
fail("runtime_composition_contract", "Composition does not expose basis/map input ports.");
}
@@ -278,7 +277,7 @@ export async function compileSource(args: {
const absoluteSource = value.source
? pathToFileURL(resolve(dirname(value.module), value.source)).href
: undefined;
- const data = value.type === "AlignmentEvidence" && value.source
+ const data = value.type === "SpeechTimingEvidence" && value.source
? JSON.parse(await readFile(resolve(dirname(value.module), value.source), "utf8"))
: undefined;
return [
@@ -390,7 +389,8 @@ export async function compileSource(args: {
const projection = validateKernelProjection(await projectKernelIsolated(manifest, context));
if (located) {
for (const port of manifest.ports.filter((candidate) =>
- candidate.direction === "output" && candidate.type === "Track")) {
+ candidate.direction === "output"
+ && (candidate.type === "Track" || candidate.type === "CaptionTrack"))) {
validateTrackOutput(instance.id, port.name, projection.outputs[port.name], located.durationFrames);
}
}
@@ -428,12 +428,12 @@ export async function compileSource(args: {
) as TemporalBasisProduction | undefined;
const semanticMap = resolveValue(
rootElement.attributes[semanticPort.name],
- ) as ExactSemanticMap | undefined;
+ ) as CompleteSemanticMap | undefined;
if (production?.contract !== "svml.temporal-basis-production.v1") {
fail("runtime_basis_missing", "Composition root must select one TemporalBasisProduction.");
}
- if (semanticMap?.contract !== "svml.exact-semantic-map.v1") {
- fail("runtime_semantic_map_missing", "Composition root must select one ExactSemanticMap.");
+ if (semanticMap?.contract !== "svml.complete-semantic-map.v1") {
+ fail("runtime_semantic_map_missing", "Composition root must select one CompleteSemanticMap.");
}
validateTemporalBasisProduction(production);
validateSemanticMap(checked.narrative, production.basis, semanticMap);
diff --git a/src/format.ts b/src/format.ts
index 7fec0a8e..480b23b5 100644
--- a/src/format.ts
+++ b/src/format.ts
@@ -48,8 +48,8 @@ function semanticNarrative(narrative: NarrativeIR): unknown {
captionAtoms: narrative.captionAtoms.map((atom) => ({
display: atom.display,
segmentId: atom.segmentId,
- startWord: atom.startWord,
- endWordExclusive: atom.endWordExclusive,
+ startToken: atom.startToken,
+ endTokenExclusive: atom.endTokenExclusive,
})),
projections: narrative.projections,
};
diff --git a/src/model.ts b/src/model.ts
index 27946dee..d84fa62c 100644
--- a/src/model.ts
+++ b/src/model.ts
@@ -64,9 +64,27 @@ export type SourceDocument = {
};
export type ScriptAtom =
- | { kind: "text"; speech: string; caption: string; start: number; end: number }
+ | {
+ kind: "text";
+ speech: string;
+ caption: string;
+ start: number;
+ end: number;
+ tokenStart: number;
+ tokenEndExclusive: number;
+ }
| { kind: "role"; label: string; start: number; end: number };
+export type SpokenTurn = {
+ id: string;
+ segmentId: string;
+ role?: string;
+ tokenStart: number;
+ tokenEndExclusive: number;
+ sourceStart: number;
+ sourceEnd: number;
+};
+
export type ScriptToken = {
id: string;
index: number;
@@ -153,8 +171,8 @@ export type CaptionAtom = {
id: string;
display: string;
segmentId: string;
- startWord: number;
- endWordExclusive: number;
+ startToken: number;
+ endTokenExclusive: number;
sourceStart: number;
sourceEnd: number;
};
@@ -162,6 +180,7 @@ export type CaptionAtom = {
export type NarrativeIR = {
segments: ScriptSegment[];
tokens: ScriptToken[];
+ turns: SpokenTurn[];
selections: Record<string, SelectionOccurrence[]>;
moments: Record<string, MomentOccurrence[]>;
captionAtoms: CaptionAtom[];
@@ -206,19 +225,14 @@ export type SemanticMapBase = {
evidenceDigests: string[];
locatorDigest: string;
quantizationPolicy: "nearest-frame";
- captionCues?: AlignmentCaptionCue[];
mapDigest: string;
};
-export type EstimatedSemanticMap = SemanticMapBase & {
- contract: "svml.estimated-semantic-map.v1";
-};
-
-export type ExactSemanticMap = SemanticMapBase & {
- contract: "svml.exact-semantic-map.v1";
+export type CompleteSemanticMap = SemanticMapBase & {
+ contract: "svml.complete-semantic-map.v1";
};
-export type SemanticMap = EstimatedSemanticMap | ExactSemanticMap;
+export type SemanticMap = CompleteSemanticMap;
export type SourceToProgramMap = {
id: string;
@@ -260,35 +274,53 @@ export type TemporalBasisProduction = {
productionDigest: string;
};
-export type AlignmentWord = {
+export type SpeechTimingUnit = {
text: string;
startSec: number;
endSec: number;
segmentId: string;
};
-export type AlignmentSegment = {
+export type SpeechTimingSegment = {
id: string;
startSec: number;
endSec: number;
};
-export type AlignmentCaptionCue = {
- id?: string;
- startWord: number;
- endWordExclusive: number;
-};
-
-export type AlignmentEvidence = {
- contract: "svml.speech-alignment.v1";
+export type SpeechTimingEvidence = {
+ contract: "svml.speech-timing-evidence.v1";
durationSec: number;
fps: number;
- words: AlignmentWord[];
- segments: AlignmentSegment[];
- captionCues?: AlignmentCaptionCue[];
+ quality: "measured" | "estimated";
+ units: SpeechTimingUnit[];
+ segments: SpeechTimingSegment[];
provenance?: Record<string, unknown>;
};
+export type CaptionCue = {
+ id: string;
+ startToken: number;
+ endTokenExclusive: number;
+};
+
+export type CaptionAnnotation = {
+ id: string;
+ kind: string;
+ startToken: number;
+ endTokenExclusive: number;
+ value?: string;
+};
+
+export type CaptionPlan = {
+ contract: "svml.caption-plan.v1";
+ semanticIndexDigest: string;
+ basisDigest: string;
+ cues: CaptionCue[];
+ annotations: CaptionAnnotation[];
+ plannerDigest: string;
+ planDigest: string;
+};
+
export type ProgramRange = {
startFrame: number;
endFrameExclusive: number;
@@ -306,17 +338,10 @@ export type LocatedMoment = {
frames: number[];
};
-export type LocatedCaptionCue = ProgramRange & {
- id: string;
- startWord: number;
- endWordExclusive: number;
-};
-
export type LocatedCaptionAtom = CaptionAtom & ProgramRange;
export type LocatedScript = {
contract: "svml.temporal-binding.v1";
- precision: "estimated" | "exact";
basisDigest: string;
semanticMapDigest: string;
durationFrames: number;
@@ -327,12 +352,11 @@ export type LocatedScript = {
selections: Record<string, LocatedSelection>;
moments: Record<string, LocatedMoment>;
captionAtoms: LocatedCaptionAtom[];
- captionCues?: LocatedCaptionCue[];
};
export type PlanValue = {
id: string;
- type: "Image" | "Video" | "Audio" | "Text" | "AlignmentEvidence";
+ type: "Image" | "Video" | "Audio" | "Text" | "SpeechTimingEvidence";
source?: string;
value?: string;
identity: string;
diff --git a/src/plan.ts b/src/plan.ts
index fb25fdad..894056b6 100644
--- a/src/plan.ts
+++ b/src/plan.ts
@@ -24,7 +24,7 @@ const VALUE_TYPES: Record<string, PlanValue["type"]> = {
video: "Video",
audio: "Audio",
text: "Text",
- alignment: "AlignmentEvidence",
+ timing: "SpeechTimingEvidence",
};
function idOf(element: SourceElement): string {
@@ -507,7 +507,7 @@ export function buildPlan(
const basisInputs = rootManifest.ports.filter((port) =>
port.direction === "input" && port.type.split("|").includes("TemporalBasisProduction"));
const semanticInputs = rootManifest.ports.filter((port) =>
- port.direction === "input" && port.type.split("|").includes("ExactSemanticMap"));
+ port.direction === "input" && port.type.split("|").includes("CompleteSemanticMap"));
if (
basisInputs.length !== 1
|| basisInputs[0]?.cardinality !== "one"
@@ -516,10 +516,11 @@ export function buildPlan(
) {
fail(
"composition_temporal_contract",
- `Composition Kernel "${rootManifest.name}" must select exactly one TemporalBasisProduction and one ExactSemanticMap.`,
+ `Composition Kernel "${rootManifest.name}" must select exactly one TemporalBasisProduction and one CompleteSemanticMap.`,
);
}
- const trackEdges = reachableEdges.filter((edge) => edge.type === "Track");
+ const trackEdges = reachableEdges.filter((edge) =>
+ edge.type === "Track" || edge.type === "CaptionTrack");
const illegalTrackConsumer = trackEdges.find((edge) => edge.to !== root);
if (illegalTrackConsumer) {
fail(
@@ -535,6 +536,15 @@ export function buildPlan(
`Composition "${root}" references Track "${duplicateTrack.from}" more than once.`,
);
}
+ const captionTracks = reachableInstances.filter((instance) =>
+ kernels.get(instance.kernel)?.ports.some((port) =>
+ port.direction === "output" && port.type === "CaptionTrack"));
+ if (captionTracks.length > 1) {
+ fail(
+ "caption_track_cardinality",
+ `A Composition may contain at most one CaptionTrack; received ${captionTracks.length}.`,
+ );
+ }
const usedKernels = new Set(reachableInstances.map((instance) => instance.kernel));
return {
contract: "svml.plan.v1",
diff --git a/src/script/estimate.ts b/src/script/estimate.ts
index 4f8a7d3c..6e8c1496 100644
--- a/src/script/estimate.ts
+++ b/src/script/estimate.ts
@@ -1,8 +1,7 @@
import type {
- AlignmentCaptionCue,
- AlignmentEvidence,
NarrativeIR,
ScriptToken,
+ SpeechTimingEvidence,
} from "../model.js";
export type EstimateOptions = {
@@ -10,7 +9,6 @@ export type EstimateOptions = {
wordGapSec?: number;
segmentPaddingSec?: number;
emptySegmentSec?: number;
- maxCaptionWords?: number;
fps?: number;
};
@@ -27,33 +25,14 @@ function syllables(token: ScriptToken): number {
return Math.max(1, groups - silentE);
}
-function captionCues(narrative: NarrativeIR, maxWords: number): AlignmentCaptionCue[] {
- const output: AlignmentCaptionCue[] = [];
- for (const segment of narrative.segments) {
- for (
- let startWord = segment.tokenStart;
- startWord < segment.tokenEnd;
- startWord += maxWords
- ) {
- output.push({
- id: `estimate-cue-${output.length + 1}`,
- startWord,
- endWordExclusive: Math.min(segment.tokenEnd, startWord + maxWords),
- });
- }
- }
- return output;
-}
-
-export function estimateAlignment(
+export function estimateSpeechTiming(
narrative: NarrativeIR,
options: EstimateOptions = {},
-): AlignmentEvidence {
+): SpeechTimingEvidence {
const syllablesPerSecond = options.syllablesPerSecond ?? 6.5;
const wordGapSec = options.wordGapSec ?? 0.045;
const segmentPaddingSec = options.segmentPaddingSec ?? 0.12;
const emptySegmentSec = options.emptySegmentSec ?? 1;
- const maxCaptionWords = Math.max(1, Math.floor(options.maxCaptionWords ?? 3));
const fps = Math.max(1, Math.round(options.fps ?? 30));
for (const [name, value] of Object.entries({
syllablesPerSecond,
@@ -67,8 +46,8 @@ export function estimateAlignment(
}
let cursor = 0;
- const words: AlignmentEvidence["words"] = [];
- const segments: AlignmentEvidence["segments"] = [];
+ const units: SpeechTimingEvidence["units"] = [];
+ const segments: SpeechTimingEvidence["segments"] = [];
for (const segment of narrative.segments) {
const segmentTokens = narrative.tokens.slice(segment.tokenStart, segment.tokenEnd);
const startSec = cursor;
@@ -82,7 +61,7 @@ export function estimateAlignment(
for (const [index, token] of segmentTokens.entries()) {
const wordStart = cursor;
cursor += syllables(token) / syllablesPerSecond;
- words.push({
+ units.push({
text: token.text,
startSec: wordStart,
endSec: cursor,
@@ -97,12 +76,12 @@ export function estimateAlignment(
const durationFrames = Math.max(1, Math.ceil(cursor * fps));
const durationSec = durationFrames / fps;
return {
- contract: "svml.speech-alignment.v1",
+ contract: "svml.speech-timing-evidence.v1",
durationSec,
fps,
- words,
+ quality: "estimated",
+ units,
segments,
- captionCues: captionCues(narrative, maxCaptionWords),
provenance: {
method: "svml.syllable-estimate.v1",
measured: false,
@@ -110,7 +89,6 @@ export function estimateAlignment(
wordGapSec,
segmentPaddingSec,
emptySegmentSec,
- maxCaptionWords,
},
};
}
diff --git a/src/script/locate.ts b/src/script/locate.ts
index 40a39211..040acbb4 100644
--- a/src/script/locate.ts
+++ b/src/script/locate.ts
@@ -1,6 +1,6 @@
import { fail } from "../diagnostics.js";
import type {
- ExactSemanticMap,
+ CompleteSemanticMap,
LocatedScript,
MarkerBoundary,
NarrativeIR,
@@ -61,77 +61,13 @@ export function boundaryAnchor(
?? fail("locate_boundary_empty", "Script has no structural boundary candidate.");
}
-function validateCaptionCues(
- narrative: NarrativeIR,
- locatedWords: LocatedScript["words"],
- locatedAtoms: LocatedScript["captionAtoms"],
- map: SemanticMap,
- basis: ProgramBasis,
-): LocatedScript["captionCues"] {
- const cues = map.captionCues;
- if (!cues) return undefined;
- const located = cues.map((cue, index) => {
- if (!Number.isInteger(cue.startWord) || !Number.isInteger(cue.endWordExclusive)) {
- fail("locate_caption_cue_index", `Caption cue ${index + 1} must use integer token indexes.`);
- }
- if (
- cue.startWord < 0
- || cue.endWordExclusive <= cue.startWord
- || cue.endWordExclusive > locatedWords.length
- ) {
- fail(
- "locate_caption_cue_range",
- `Caption cue ${index + 1} has invalid token range ${cue.startWord}..${cue.endWordExclusive}.`,
- );
- }
- const previous = cues[index - 1];
- const expectedStart = previous?.endWordExclusive ?? 0;
- if (cue.startWord !== expectedStart) {
- fail(
- "locate_caption_cue_partition",
- `Caption cue ${index + 1} starts at token ${cue.startWord}; expected ${expectedStart}.`,
- );
- }
- const first = locatedWords[cue.startWord];
- const last = locatedWords[cue.endWordExclusive - 1];
- if (!first || !last || first.segmentId !== last.segmentId) {
- fail("locate_caption_cue_segment", `Caption cue ${index + 1} crosses Script Segments.`);
- }
- for (const atom of locatedAtoms) {
- const overlaps = cue.startWord < atom.endWordExclusive
- && cue.endWordExclusive > atom.startWord;
- const contains = cue.startWord <= atom.startWord
- && cue.endWordExclusive >= atom.endWordExclusive;
- if (overlaps && !contains) {
- fail(
- "locate_caption_partial_dual",
- `Caption cue ${index + 1} cuts through Dual Text atom "${atom.id}".`,
- );
- }
- }
- return {
- id: cue.id ?? `cue-${index + 1}`,
- startWord: cue.startWord,
- endWordExclusive: cue.endWordExclusive,
- ...range(first.startFrame, last.endFrameExclusive, basis, `Caption cue ${index + 1}`),
- };
- });
- if ((located.at(-1)?.endWordExclusive ?? 0) !== narrative.tokens.length) {
- fail(
- "locate_caption_cue_partition",
- `Caption cues end at token ${located.at(-1)?.endWordExclusive ?? 0}; expected ${narrative.tokens.length}.`,
- );
- }
- return located;
-}
-
export function locateScript(
narrative: NarrativeIR,
basis: ProgramBasis,
- map: ExactSemanticMap,
+ map: CompleteSemanticMap,
): LocatedScript {
- if (map.contract !== "svml.exact-semantic-map.v1") {
- fail("locate_exact_required", "Located HTML compilation requires ExactSemanticMap.");
+ if (map.contract !== "svml.complete-semantic-map.v1") {
+ fail("locate_complete_required", "HTML compilation requires CompleteSemanticMap.");
}
return bindSemanticMap(narrative, basis, map);
}
@@ -194,8 +130,8 @@ export function bindSemanticMap(
],
));
const captionAtoms = narrative.captionAtoms.map((atom) => {
- const first = words[atom.startWord];
- const last = words[atom.endWordExclusive - 1];
+ const first = words[atom.startToken];
+ const last = words[atom.endTokenExclusive - 1];
if (!first || !last || first.segmentId !== last.segmentId) {
fail("locate_caption_atom", `Caption atom "${atom.id}" has an invalid speech span.`);
}
@@ -204,10 +140,8 @@ export function bindSemanticMap(
...range(first.startFrame, last.endFrameExclusive, basis, `Caption atom "${atom.id}"`),
};
});
- const captionCues = validateCaptionCues(narrative, words, captionAtoms, map, basis);
return {
contract: "svml.temporal-binding.v1",
- precision: map.contract === "svml.exact-semantic-map.v1" ? "exact" : "estimated",
basisDigest: basis.basisDigest,
semanticMapDigest: map.mapDigest,
durationFrames: basis.durationFrames,
@@ -218,6 +152,5 @@ export function bindSemanticMap(
selections,
moments,
captionAtoms,
- ...(captionCues ? { captionCues } : {}),
};
}
diff --git a/src/script/parse.ts b/src/script/parse.ts
index 4f626dab..09cb6a48 100644
--- a/src/script/parse.ts
+++ b/src/script/parse.ts
@@ -291,13 +291,7 @@ export function parseScript(
}
return;
}
- current.atoms.push({
- kind: "text",
- speech,
- caption,
- start: sourceOffset + start,
- end: sourceOffset + end,
- });
+ const tokenStart = tokens.length;
WORD.lastIndex = 0;
let match: RegExpExecArray | null;
while ((match = WORD.exec(speech))) {
@@ -318,6 +312,15 @@ export function parseScript(
sourceEnd: sourceOffset + start + match.index + match[0].length,
});
}
+ current.atoms.push({
+ kind: "text",
+ speech,
+ caption,
+ start: sourceOffset + start,
+ end: sourceOffset + end,
+ tokenStart,
+ tokenEndExclusive: tokens.length,
+ });
};
const consumeSpeechSide = (
@@ -325,7 +328,7 @@ export function parseScript(
absoluteStart: number,
caption: string,
): void => {
- const startWord = tokens.length;
+ const startToken = tokens.length;
let partStart = 0;
let index = 0;
let emittedCaption = false;
@@ -370,14 +373,14 @@ export function parseScript(
partStart = index;
}
addLiteral(raw.slice(partStart), absoluteStart + partStart);
- const endWordExclusive = tokens.length;
- if (endWordExclusive > startWord) {
+ const endTokenExclusive = tokens.length;
+ if (endTokenExclusive > startToken) {
captionAtoms.push({
id: `caption-atom-${captionAtoms.length + 1}`,
display: cleanProjection(caption),
segmentId: current!.id,
- startWord,
- endWordExclusive,
+ startToken,
+ endTokenExclusive,
sourceStart: sourceOffset + absoluteStart,
sourceEnd: sourceOffset + absoluteStart + raw.length,
});
@@ -553,16 +556,39 @@ export function parseScript(
const speechSegments: string[] = [];
const captionSegments: string[] = [];
const dialogueTurns: string[] = [];
+ const turns: NarrativeIR["turns"] = [];
for (const segment of segments) {
const speechParts: string[] = [];
const captionParts: string[] = [];
let activeRole: string | undefined;
let turnParts: string[] = [];
+ let turnTokenStart: number | undefined;
+ let turnTokenEndExclusive: number | undefined;
+ let turnSourceStart: number | undefined;
+ let turnSourceEnd: number | undefined;
const flushTurn = (): void => {
const body = joinProjection(turnParts);
- if (!body) return;
- dialogueTurns.push(activeRole ? `${activeRole}: ${body}` : body);
+ if (body) dialogueTurns.push(activeRole ? `${activeRole}: ${body}` : body);
+ if (
+ turnTokenStart !== undefined
+ && turnTokenEndExclusive !== undefined
+ && turnTokenEndExclusive > turnTokenStart
+ ) {
+ turns.push({
+ id: `turn-${turns.length + 1}`,
+ segmentId: segment.id,
+ ...(activeRole ? { role: activeRole } : {}),
+ tokenStart: turnTokenStart,
+ tokenEndExclusive: turnTokenEndExclusive,
+ sourceStart: turnSourceStart ?? segment.sourceStart,
+ sourceEnd: turnSourceEnd ?? segment.sourceEnd,
+ });
+ }
turnParts = [];
+ turnTokenStart = undefined;
+ turnTokenEndExclusive = undefined;
+ turnSourceStart = undefined;
+ turnSourceEnd = undefined;
};
for (const atom of segment.atoms) {
if (atom.kind === "role") {
@@ -572,6 +598,12 @@ export function parseScript(
speechParts.push(atom.speech);
captionParts.push(atom.caption);
turnParts.push(atom.speech);
+ if (atom.tokenEndExclusive > atom.tokenStart) {
+ turnTokenStart ??= atom.tokenStart;
+ turnTokenEndExclusive = atom.tokenEndExclusive;
+ turnSourceStart ??= atom.start;
+ turnSourceEnd = atom.end;
+ }
}
}
flushTurn();
@@ -626,6 +658,7 @@ export function parseScript(
return {
segments,
tokens,
+ turns,
selections,
moments,
captionAtoms,
diff --git a/src/temporal.ts b/src/temporal.ts
index 3e12b189..6f1d36a8 100644
--- a/src/temporal.ts
+++ b/src/temporal.ts
@@ -1,12 +1,11 @@
import { fail } from "./diagnostics.js";
import type {
- AlignmentEvidence,
- ExactSemanticMap,
- EstimatedSemanticMap,
+ CompleteSemanticMap,
NarrativeIR,
ProgramBasis,
SemanticAnchorPoint,
SemanticMap,
+ SpeechTimingEvidence,
TemporalBasisProduction,
} from "./model.js";
import { normalizeWord, sha256, stableJson } from "./util.js";
@@ -222,17 +221,17 @@ function frameAt(seconds: number, basis: ProgramBasis): number {
return Math.round(seconds * basisFps(basis));
}
-function semanticMapDigest(map: Omit<SemanticMap, "mapDigest">): string {
+function semanticMapDigest(map: Omit<CompleteSemanticMap, "mapDigest">): string {
return sha256(stableJson(map));
}
-function validateAlignment(
+function validateSpeechTiming(
narrative: NarrativeIR,
basis: ProgramBasis,
- evidence: AlignmentEvidence,
+ evidence: SpeechTimingEvidence,
): void {
- if (evidence.contract !== "svml.speech-alignment.v1") {
- fail("locator_evidence_contract", `Unsupported alignment contract "${evidence.contract}".`);
+ if (evidence.contract !== "svml.speech-timing-evidence.v1") {
+ fail("locator_evidence_contract", `Unsupported speech timing contract "${evidence.contract}".`);
}
if (
!Number.isFinite(evidence.fps)
@@ -240,40 +239,37 @@ function validateAlignment(
|| !Number.isFinite(evidence.durationSec)
|| evidence.durationSec <= 0
) {
- fail("locator_evidence_clock", "Alignment evidence requires positive fps and duration.");
+ fail("locator_evidence_clock", "SpeechTimingEvidence requires positive fps and duration.");
+ }
+ if (evidence.quality !== "measured" && evidence.quality !== "estimated") {
+ fail("locator_evidence_quality", "SpeechTimingEvidence quality must be measured or estimated.");
}
if (Math.abs(evidence.fps - basisFps(basis)) > EPSILON) {
fail(
"locator_basis_frame_rate",
- `Alignment fps ${evidence.fps} does not match ProgramBasis fps ${basisFps(basis)}.`,
+ `Speech timing fps ${evidence.fps} does not match ProgramBasis fps ${basisFps(basis)}.`,
);
}
if (frameAt(evidence.durationSec, basis) !== basis.durationFrames) {
fail(
"locator_basis_duration",
- `Alignment duration resolves to ${frameAt(evidence.durationSec, basis)} frames; ProgramBasis has ${basis.durationFrames}.`,
- );
- }
- if (evidence.words.length !== narrative.tokens.length) {
- fail(
- "locator_word_cardinality",
- `Script has ${narrative.tokens.length} tokens but alignment has ${evidence.words.length}.`,
+ `Speech timing duration resolves to ${frameAt(evidence.durationSec, basis)} frames; ProgramBasis has ${basis.durationFrames}.`,
);
}
if (evidence.segments.length !== narrative.segments.length) {
fail(
"locator_segment_cardinality",
- `Script has ${narrative.segments.length} Segments but alignment has ${evidence.segments.length}.`,
+ `Script has ${narrative.segments.length} Segments but speech timing has ${evidence.segments.length}.`,
);
}
const segments = new Map(evidence.segments.map((segment) => [segment.id, segment]));
if (segments.size !== evidence.segments.length) {
- fail("locator_segment_duplicate", "Alignment evidence repeats a Segment id.");
+ fail("locator_segment_duplicate", "SpeechTimingEvidence repeats a Segment id.");
}
for (const [index, segment] of narrative.segments.entries()) {
const aligned = segments.get(segment.id);
- if (!aligned) fail("locator_segment_missing", `Alignment is missing Segment "${segment.id}".`);
+ if (!aligned) fail("locator_segment_missing", `Speech timing is missing Segment "${segment.id}".`);
if (
!Number.isFinite(aligned.startSec)
|| !Number.isFinite(aligned.endSec)
@@ -302,99 +298,200 @@ function validateAlignment(
}
}
- const previousBySegment = new Map<string, AlignmentEvidence["words"][number]>();
- for (const [index, token] of narrative.tokens.entries()) {
- const word = evidence.words[index];
- if (!word) fail("locator_word_missing", `Alignment is missing token ${index + 1}.`);
- if (normalizeWord(word.text) !== token.normalized || word.segmentId !== token.segmentId) {
- fail(
- "locator_word_mismatch",
- `Token ${index + 1} does not match alignment word "${word.text}" in Segment "${word.segmentId}".`,
- );
- }
- const segment = segments.get(word.segmentId);
+ const previousBySegment = new Map<string, SpeechTimingEvidence["units"][number]>();
+ for (const [index, unit] of evidence.units.entries()) {
+ const segment = segments.get(unit.segmentId);
if (
!segment
- || !Number.isFinite(word.startSec)
- || !Number.isFinite(word.endSec)
- || word.startSec < segment.startSec - EPSILON
- || word.endSec < word.startSec
- || word.endSec > segment.endSec + EPSILON
+ || typeof unit.text !== "string"
+ || !Number.isFinite(unit.startSec)
+ || !Number.isFinite(unit.endSec)
+ || unit.startSec < segment.startSec - EPSILON
+ || unit.endSec < unit.startSec
+ || unit.endSec > segment.endSec + EPSILON
) {
fail(
"locator_word_window",
- `Alignment token ${index + 1} falls outside Segment "${word.segmentId}".`,
+ `Speech timing unit ${index + 1} falls outside Segment "${unit.segmentId}".`,
);
}
- const previous = previousBySegment.get(word.segmentId);
- if (previous && word.startSec < previous.endSec - EPSILON) {
+ const previous = previousBySegment.get(unit.segmentId);
+ if (previous && unit.startSec < previous.endSec - EPSILON) {
fail(
"locator_word_overlap",
- `Alignment token ${index + 1} overlaps the preceding token in Segment "${word.segmentId}".`,
+ `Speech timing unit ${index + 1} overlaps the preceding unit in Segment "${unit.segmentId}".`,
);
}
- previousBySegment.set(word.segmentId, word);
+ previousBySegment.set(unit.segmentId, unit);
}
}
-export function semanticMapFromAlignment(
- narrative: NarrativeIR,
- basis: ProgramBasis,
- evidence: AlignmentEvidence,
- options: { precision: "exact"; locatorDigest?: string },
-): ExactSemanticMap;
-export function semanticMapFromAlignment(
- narrative: NarrativeIR,
+type TokenFrame = {
+ startFrame: number;
+ endFrameExclusive: number;
+ quality: "estimated" | "derived" | "measured";
+};
+
+function lcsMatches(
+ tokens: NarrativeIR["tokens"],
+ units: SpeechTimingEvidence["units"],
+): Array<[number, number]> {
+ const rows = Array.from({ length: tokens.length + 1 }, () =>
+ Array<number>(units.length + 1).fill(0));
+ for (let left = tokens.length - 1; left >= 0; left -= 1) {
+ for (let right = units.length - 1; right >= 0; right -= 1) {
+ rows[left]![right] = tokens[left]!.normalized === normalizeWord(units[right]!.text)
+ ? rows[left + 1]![right + 1]! + 1
+ : Math.max(rows[left + 1]![right]!, rows[left]![right + 1]!);
+ }
+ }
+ const matches: Array<[number, number]> = [];
+ let left = 0;
+ let right = 0;
+ while (left < tokens.length && right < units.length) {
+ if (
+ tokens[left]!.normalized === normalizeWord(units[right]!.text)
+ && rows[left]![right] === rows[left + 1]![right + 1]! + 1
+ ) {
+ matches.push([left, right]);
+ left += 1;
+ right += 1;
+ } else if (rows[left + 1]![right]! >= rows[left]![right + 1]!) {
+ left += 1;
+ } else {
+ right += 1;
+ }
+ }
+ return matches;
+}
+
+function distributeFrames(
+ tokens: NarrativeIR["tokens"],
+ startFrame: number,
+ endFrameExclusive: number,
+ quality: TokenFrame["quality"],
+): TokenFrame[] {
+ if (!tokens.length) return [];
+ const weights = tokens.map((token) => Math.max(1, [...token.normalized].length));
+ const total = weights.reduce((sum, weight) => sum + weight, 0);
+ let accumulated = 0;
+ return tokens.map((_, index) => {
+ const start = startFrame + Math.round((endFrameExclusive - startFrame) * accumulated / total);
+ accumulated += weights[index]!;
+ const end = index === tokens.length - 1
+ ? endFrameExclusive
+ : startFrame + Math.round((endFrameExclusive - startFrame) * accumulated / total);
+ return { startFrame: start, endFrameExclusive: Math.max(start, end), quality };
+ });
+}
+
+function alignSegment(
+ tokens: NarrativeIR["tokens"],
+ units: SpeechTimingEvidence["units"],
+ segmentStartFrame: number,
+ segmentEndFrame: number,
basis: ProgramBasis,
- evidence: AlignmentEvidence,
- options: { precision: "estimated"; locatorDigest?: string },
-): EstimatedSemanticMap;
-export function semanticMapFromAlignment(
+ evidenceQuality: SpeechTimingEvidence["quality"],
+): TokenFrame[] {
+ const output: Array<TokenFrame | undefined> = Array(tokens.length).fill(undefined);
+ const matches = lcsMatches(tokens, units);
+ let tokenCursor = 0;
+ let unitCursor = 0;
+ for (let matchIndex = 0; matchIndex <= matches.length; matchIndex += 1) {
+ const match = matches[matchIndex];
+ const tokenEnd = match?.[0] ?? tokens.length;
+ const unitEnd = match?.[1] ?? units.length;
+ const tokenRun = tokens.slice(tokenCursor, tokenEnd);
+ const unitRun = units.slice(unitCursor, unitEnd);
+ if (tokenRun.length) {
+ const previous = tokenCursor > 0 ? output[tokenCursor - 1] : undefined;
+ const nextUnit = match ? units[unitEnd] : undefined;
+ const startFrame = unitRun.length
+ ? frameAt(unitRun[0]!.startSec, basis)
+ : previous?.endFrameExclusive ?? segmentStartFrame;
+ const endFrame = unitRun.length
+ ? frameAt(unitRun.at(-1)!.endSec, basis)
+ : nextUnit
+ ? frameAt(nextUnit.startSec, basis)
+ : segmentEndFrame;
+ const quality = evidenceQuality === "estimated" || !unitRun.length
+ ? "estimated" as const
+ : "derived" as const;
+ for (const [offset, value] of distributeFrames(
+ tokenRun,
+ startFrame,
+ Math.max(startFrame, endFrame),
+ quality,
+ ).entries()) output[tokenCursor + offset] = value;
+ }
+ if (match) {
+ const [tokenIndex, unitIndex] = match;
+ const unit = units[unitIndex]!;
+ output[tokenIndex] = {
+ startFrame: frameAt(unit.startSec, basis),
+ endFrameExclusive: frameAt(unit.endSec, basis),
+ quality: evidenceQuality === "measured" ? "measured" : "estimated",
+ };
+ tokenCursor = tokenIndex + 1;
+ unitCursor = unitIndex + 1;
+ }
+ }
+ return output.map((value, index) => value
+ ?? fail("locator_token_unmapped", `Script token ${index + 1} could not be aligned.`));
+}
+
+export function semanticMapFromTiming(
narrative: NarrativeIR,
basis: ProgramBasis,
- evidence: AlignmentEvidence,
- options: {
- precision: "estimated" | "exact";
- locatorDigest?: string;
- },
-): EstimatedSemanticMap | ExactSemanticMap {
+ evidence: SpeechTimingEvidence,
+ options: { locatorDigest?: string; evidenceDigest?: string } = {},
+): CompleteSemanticMap {
validateProgramBasis(basis);
- validateAlignment(narrative, basis, evidence);
+ validateSpeechTiming(narrative, basis, evidence);
const segmentEvidence = new Map(evidence.segments.map((segment) => [segment.id, segment]));
- const wordEvidence = new Map(narrative.tokens.map((token, index) => [token.id, evidence.words[index]!]));
- const quality = options.precision === "exact" ? "measured" : "estimated";
+ const tokenFrames = new Map<string, TokenFrame>();
+ for (const segment of narrative.segments) {
+ const measuredSegment = segmentEvidence.get(segment.id)!;
+ const tokens = narrative.tokens.slice(segment.tokenStart, segment.tokenEnd);
+ const units = evidence.units.filter((unit) => unit.segmentId === segment.id);
+ const aligned = alignSegment(
+ tokens,
+ units,
+ frameAt(measuredSegment.startSec, basis),
+ frameAt(measuredSegment.endSec, basis),
+ basis,
+ evidence.quality,
+ );
+ tokens.forEach((token, index) => tokenFrames.set(token.id, aligned[index]!));
+ }
const anchors: SemanticAnchorPoint[] = narrative.semanticIndex.anchors.map((anchor) => {
const segment = segmentEvidence.get(anchor.segmentId);
- if (!segment) fail("locator_segment_missing", `Alignment is missing Segment "${anchor.segmentId}".`);
- const word = anchor.tokenId ? wordEvidence.get(anchor.tokenId) : undefined;
- const seconds = anchor.kind === "segment-start"
- ? segment.startSec
+ if (!segment) fail("locator_segment_missing", `Speech timing is missing Segment "${anchor.segmentId}".`);
+ const token = anchor.tokenId ? tokenFrames.get(anchor.tokenId) : undefined;
+ const frame = anchor.kind === "segment-start"
+ ? frameAt(segment.startSec, basis)
: anchor.kind === "segment-end"
- ? segment.endSec
+ ? frameAt(segment.endSec, basis)
: anchor.kind === "token-start"
- ? word?.startSec
- : word?.endSec;
- if (seconds === undefined) {
- fail("locator_anchor_missing", `No alignment time exists for anchor "${anchor.id}".`);
+ ? token?.startFrame
+ : token?.endFrameExclusive;
+ if (frame === undefined) {
+ fail("locator_anchor_missing", `No speech timing exists for anchor "${anchor.id}".`);
}
return {
identity: anchor.id,
- point: { basisDigest: basis.basisDigest, frame: frameAt(seconds, basis) },
- quality,
+ point: { basisDigest: basis.basisDigest, frame },
+ quality: token?.quality ?? evidence.quality,
};
});
- const contract = options.precision === "exact"
- ? "svml.exact-semantic-map.v1" as const
- : "svml.estimated-semantic-map.v1" as const;
const payload = {
- contract,
+ contract: "svml.complete-semantic-map.v1" as const,
semanticIndexDigest: narrative.semanticIndex.digest,
basisDigest: basis.basisDigest,
anchors,
- evidenceDigests: [sha256(stableJson(evidence))],
- locatorDigest: options.locatorDigest ?? sha256(`svml.${options.precision}-speech-locator.v1`),
+ evidenceDigests: [options.evidenceDigest ?? sha256(stableJson(evidence))],
+ locatorDigest: options.locatorDigest ?? sha256("svml.script-timing-locator.v1"),
quantizationPolicy: "nearest-frame" as const,
- ...(evidence.captionCues ? { captionCues: evidence.captionCues } : {}),
};
const map = { ...payload, mapDigest: semanticMapDigest(payload) };
validateSemanticMap(narrative, basis, map);
@@ -408,8 +505,7 @@ export function validateSemanticMap(
): void {
validateProgramBasis(basis);
if (
- map.contract !== "svml.estimated-semantic-map.v1"
- && map.contract !== "svml.exact-semantic-map.v1"
+ map.contract !== "svml.complete-semantic-map.v1"
) {
fail("semantic_map_contract", "Unsupported SemanticMap contract.");
}
@@ -446,8 +542,8 @@ export function validateSemanticMap(
) {
fail("semantic_map_point", `SemanticMap anchor "${actual.identity}" is outside ProgramBasis.`);
}
- if (map.contract === "svml.exact-semantic-map.v1" && actual.quality === "estimated") {
- fail("semantic_map_precision", `ExactSemanticMap anchor "${actual.identity}" is estimated.`);
+ if (!["measured", "derived", "estimated"].includes(actual.quality)) {
+ fail("semantic_map_quality", `SemanticMap anchor "${actual.identity}" has invalid quality.`);
}
if (pointById.has(actual.identity)) {
fail("semantic_map_duplicate", `SemanticMap repeats anchor "${actual.identity}".`);
diff --git a/src/views.ts b/src/views.ts
index 3753de35..51278874 100644
--- a/src/views.ts
+++ b/src/views.ts
@@ -31,7 +31,6 @@ export type TimelineView = {
segments: Compilation["located"]["segments"];
selections: Compilation["located"]["selections"];
moments: Compilation["located"]["moments"];
- captionCues: Compilation["located"]["captionCues"];
visuals: Compilation["target"]["visuals"];
audios: Compilation["target"]["audios"];
};
@@ -71,7 +70,6 @@ export function projectTimeline(compilation: Compilation): TimelineView {
segments: compilation.located.segments,
selections: compilation.located.selections,
moments: compilation.located.moments,
- captionCues: compilation.located.captionCues,
visuals: compilation.target.visuals,
audios: compilation.target.audios,
};
diff --git a/stdlib/caption-planner.svk b/stdlib/caption-planner.svk
new file mode 100644
index 00000000..934cb4b5
--- /dev/null
+++ b/stdlib/caption-planner.svk
@@ -0,0 +1,13 @@
+<kernel name="caption-planner" abi="1" implementation="./runtime/caption-planner.mjs">
+ <param name="maxWords" style="max-words" type="number" default="3"/>
+ <port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
+ <port name="semantic" direction="input" type="CompleteSemanticMap" cardinality="one"/>
+ <port name="plan" direction="output" type="CaptionPlan" cardinality="one"/>
+ <child name="annotate" cardinality="zero-or-more">
+ <field name="id" type="id" required="true"/>
+ <field name="kind" type="string" required="true"/>
+ <field name="value" type="string"/>
+ <field name="during" type="SelectionSet|ProgramSpan" consume="set"/>
+ <field name="role" type="string"/>
+ </child>
+</kernel>
diff --git a/stdlib/caption-track.svk b/stdlib/caption-track.svk
index d96c40ea..2af1e667 100644
--- a/stdlib/caption-track.svk
+++ b/stdlib/caption-track.svk
@@ -3,7 +3,6 @@
<param name="x" type="number" default="0.37"/>
<param name="y" type="number" default="0.54"/>
<param name="width" type="number" default="0.64"/>
- <param name="maxWords" style="max-words" type="number" default="3"/>
<param name="hold" type="number" default="0.08"/>
<param name="cueFade" style="cue-fade" type="number" default="0.12"/>
<param name="font" style="font-family" type="string" default="Poppins, Inter, Arial, sans-serif"/>
@@ -16,15 +15,24 @@
<param name="strokeColor" style="stroke-color" type="string" default="rgba(0,0,0,.82)"/>
<param name="shadow" type="string" default="0 2px 9px rgba(0,0,0,.5), 0 0 2px rgba(0,0,0,.68)"/>
<port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
- <port name="semantic" direction="input" type="ExactSemanticMap" cardinality="one"/>
- <port name="track" direction="output" type="Track" cardinality="one"/>
+ <port name="semantic" direction="input" type="CompleteSemanticMap" cardinality="one"/>
+ <port name="plan" direction="input" type="CaptionPlan" cardinality="one"/>
+ <port name="track" direction="output" type="CaptionTrack" cardinality="one"/>
<child name="style" cardinality="zero-or-more">
<field name="id" type="id" required="true"/>
- <field name="during" type="SelectionSet|ProgramSpan" consume="set" required="true"/>
+ <field name="during" type="SelectionSet|ProgramSpan" consume="set"/>
+ <field name="role" type="string"/>
+ <field name="annotation" type="string"/>
<field name="color" style="fill" type="string"/>
<field name="activeColor" style="active-fill" type="string"/>
<field name="background" type="string"/>
<field name="weight" type="number"/>
<field name="scale" type="number"/>
</child>
+ <child name="mute" cardinality="zero-or-more">
+ <field name="id" type="id" required="true"/>
+ <field name="during" type="SelectionSet|ProgramSpan" consume="set"/>
+ <field name="role" type="string"/>
+ <field name="annotation" type="string"/>
+ </child>
</kernel>
diff --git a/stdlib/film.svk b/stdlib/film.svk
index 5c91952d..9c36f6ff 100644
--- a/stdlib/film.svk
+++ b/stdlib/film.svk
@@ -4,10 +4,10 @@
<param name="fps" style="frame-rate" type="number"/>
<param name="background" type="string" default="#000000"/>
<port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
- <port name="semantic" direction="input" type="ExactSemanticMap" cardinality="one"/>
- <port name="track" direction="input" type="Track" cardinality="zero-or-more"/>
+ <port name="semantic" direction="input" type="CompleteSemanticMap" cardinality="one"/>
+ <port name="track" direction="input" type="Track|CaptionTrack" cardinality="zero-or-more"/>
<port name="document" direction="output" type="HyperframesDocument" cardinality="one"/>
<child name="track" cardinality="zero-or-more">
- <field name="ref" type="Track" required="true"/>
+ <field name="ref" type="Track|CaptionTrack" required="true"/>
</child>
</kernel>
diff --git a/stdlib/runtime/caption-planner.mjs b/stdlib/runtime/caption-planner.mjs
new file mode 100644
index 00000000..0fb78dbb
--- /dev/null
+++ b/stdlib/runtime/caption-planner.mjs
@@ -0,0 +1,95 @@
+import { elements, numberAttr, stringAttr } from "./helpers.mjs";
+
+function fail(message) {
+ throw new Error(message);
+}
+
+export default {
+ abiVersion: "1",
+ project(context) {
+ const script = context.resolve(context.element.attributes.script);
+ const semantic = context.resolve(context.element.attributes.semantic);
+ if (!script?.semanticIndex) fail("caption-planner requires NarrativeIR");
+ if (semantic?.contract !== "svml.complete-semantic-map.v1") {
+ fail("caption-planner requires CompleteSemanticMap");
+ }
+ if (semantic.semanticIndexDigest !== script.semanticIndex.digest) {
+ fail("caption-planner SemanticMap does not match Script");
+ }
+ const points = new Map(semantic.anchors.map(anchor => [anchor.identity, anchor.point.frame]));
+ const words = script.tokens.map(token => ({
+ ...token,
+ startFrame: points.get(token.startAnchorId),
+ endFrameExclusive: points.get(token.endAnchorId),
+ }));
+ const maxWords = Math.max(1, Math.round(numberAttr(context.element, "maxWords", 3)));
+ const atomsByStart = new Map(script.captionAtoms.map(atom => [atom.startToken, atom]));
+ const cues = [];
+ for (const segment of script.segments) {
+ const units = [];
+ let token = segment.tokenStart;
+ while (token < segment.tokenEnd) {
+ const atom = atomsByStart.get(token);
+ if (atom) {
+ units.push({ startToken: atom.startToken, endTokenExclusive: atom.endTokenExclusive });
+ token = atom.endTokenExclusive;
+ } else {
+ units.push({ startToken: token, endTokenExclusive: token + 1 });
+ token += 1;
+ }
+ }
+ for (let index = 0; index < units.length; index += maxWords) {
+ const group = units.slice(index, index + maxWords);
+ cues.push({
+ id: `cue-${cues.length + 1}`,
+ startToken: group[0].startToken,
+ endTokenExclusive: group.at(-1).endTokenExclusive,
+ });
+ }
+ }
+ const frameRangesForRole = role => script.turns
+ .filter(turn => turn.role === role)
+ .map(turn => ({
+ startFrame: words[turn.tokenStart]?.startFrame,
+ endFrameExclusive: words[turn.tokenEndExclusive - 1]?.endFrameExclusive,
+ }))
+ .filter(range => Number.isInteger(range.startFrame) && Number.isInteger(range.endFrameExclusive));
+ const annotations = [];
+ for (const item of elements(context, "annotate")) {
+ const role = item.attributes.role;
+ const during = item.attributes.during;
+ if ((role === undefined) === (during === undefined)) {
+ fail(`<annotate id="${stringAttr(item, "id")}"> requires exactly one of role or during`);
+ }
+ const ranges = role !== undefined
+ ? frameRangesForRole(stringAttr(item, "role"))
+ : context.selection(during, "annotate.during");
+ for (const [occurrence, range] of ranges.entries()) {
+ const selected = words.filter(word =>
+ word.startFrame < range.endFrameExclusive
+ && word.endFrameExclusive > range.startFrame);
+ if (!selected.length) continue;
+ annotations.push({
+ id: `${stringAttr(item, "id")}:${occurrence + 1}`,
+ kind: stringAttr(item, "kind"),
+ startToken: selected[0].index,
+ endTokenExclusive: selected.at(-1).index + 1,
+ ...(typeof item.attributes.value === "string" ? { value: item.attributes.value } : {}),
+ });
+ }
+ }
+ const plannerDigest = context.digest({
+ implementation: context.instance.executionDigest,
+ maxWords,
+ });
+ const payload = {
+ contract: "svml.caption-plan.v1",
+ semanticIndexDigest: script.semanticIndex.digest,
+ basisDigest: semantic.basisDigest,
+ cues,
+ annotations,
+ plannerDigest,
+ };
+ return { outputs: { plan: { ...payload, planDigest: context.digest(payload) } } };
+ },
+};
diff --git a/stdlib/runtime/caption-track.mjs b/stdlib/runtime/caption-track.mjs
index 20658cc1..d140dc6b 100644
--- a/stdlib/runtime/caption-track.mjs
+++ b/stdlib/runtime/caption-track.mjs
@@ -1,66 +1,101 @@
import { elements, html, numberAttr, stringAttr, trackOutput } from "./helpers.mjs";
+function fail(message) {
+ throw new Error(message);
+}
+
export default {
abiVersion: "1",
project(context) {
const script = context.resolve(context.element.attributes.script);
const semantic = context.resolve(context.element.attributes.semantic);
- if (semantic?.contract !== "svml.exact-semantic-map.v1") {
- throw new Error("caption-track requires ExactSemanticMap");
+ const plan = context.resolve(context.element.attributes.plan);
+ if (semantic?.contract !== "svml.complete-semantic-map.v1") {
+ fail("caption-track requires CompleteSemanticMap");
+ }
+ if (plan?.contract !== "svml.caption-plan.v1") fail("caption-track requires CaptionPlan");
+ if (
+ semantic.basisDigest !== context.program.basisDigest
+ || plan.basisDigest !== semantic.basisDigest
+ || plan.semanticIndexDigest !== script.semanticIndex.digest
+ ) fail("caption-track inputs do not share one Script and ProgramBasis");
+ if (!Array.isArray(plan.cues) || !Array.isArray(plan.annotations)) {
+ fail("CaptionPlan requires cues and annotations arrays");
}
- if (semantic.basisDigest !== context.program.basisDigest) {
- throw new Error("caption-track SemanticMap does not match the selected ProgramBasis");
+ const { planDigest, ...canonicalPlan } = plan;
+ if (context.digest(canonicalPlan) !== planDigest) fail("CaptionPlan digest does not match its contents");
+ for (const [index, annotation] of plan.annotations.entries()) {
+ if (
+ typeof annotation.id !== "string"
+ || typeof annotation.kind !== "string"
+ || !Number.isInteger(annotation.startToken)
+ || !Number.isInteger(annotation.endTokenExclusive)
+ || annotation.startToken < 0
+ || annotation.endTokenExclusive <= annotation.startToken
+ || annotation.endTokenExclusive > script.tokens.length
+ ) fail(`CaptionPlan annotation ${index + 1} has an invalid Script token range`);
}
- const points = new Map(semantic.anchors.map((anchor) => [anchor.identity, anchor.point.frame]));
+ const points = new Map(semantic.anchors.map(anchor => [anchor.identity, anchor.point.frame]));
const frameRange = (startFrame, endFrameExclusive) => ({
startFrame,
endFrameExclusive,
startSec: startFrame / context.fps,
endSec: endFrameExclusive / context.fps,
});
- const locatedWords = script.tokens.map((token) => ({
+ const words = script.tokens.map(token => ({
...token,
...frameRange(points.get(token.startAnchorId), points.get(token.endAnchorId)),
}));
- const captionAtoms = script.captionAtoms.map((atom) => ({
+ const captionAtoms = script.captionAtoms.map(atom => ({
...atom,
...frameRange(
- locatedWords[atom.startWord].startFrame,
- locatedWords[atom.endWordExclusive - 1].endFrameExclusive,
+ words[atom.startToken].startFrame,
+ words[atom.endTokenExclusive - 1].endFrameExclusive,
),
}));
- const captionCues = semantic.captionCues?.map((cue, index) => ({
- id: cue.id ?? `cue-${index + 1}`,
- ...cue,
- ...frameRange(
- locatedWords[cue.startWord].startFrame,
- locatedWords[cue.endWordExclusive - 1].endFrameExclusive,
- ),
- }));
- const located = {
- words: locatedWords,
- captionAtoms,
- captionCues,
- durationFrames: context.program.durationFrames,
+ const roleRanges = role => script.turns
+ .filter(turn => turn.role === role)
+ .map(turn => frameRange(
+ words[turn.tokenStart].startFrame,
+ words[turn.tokenEndExclusive - 1].endFrameExclusive,
+ ));
+ const annotationRanges = kind => plan.annotations
+ .filter(annotation => annotation.kind === kind)
+ .map(annotation => frameRange(
+ words[annotation.startToken].startFrame,
+ words[annotation.endTokenExclusive - 1].endFrameExclusive,
+ ));
+ const selectorRanges = (element, label, globalAllowed) => {
+ const selectors = ["during", "role", "annotation"]
+ .filter(name => element.attributes[name] !== undefined);
+ if (selectors.length > 1 || (!globalAllowed && selectors.length !== 1)) {
+ fail(`<${label} id="${stringAttr(element, "id")}"> requires ${globalAllowed ? "at most" : "exactly"} one selector`);
+ }
+ if (!selectors.length) return [frameRange(0, context.program.durationFrames)];
+ if (selectors[0] === "during") {
+ return context.selection(element.attributes.during, `${label}.during`);
+ }
+ if (selectors[0] === "role") return roleRanges(stringAttr(element, "role"));
+ return annotationRanges(stringAttr(element, "annotation"));
};
- const z = numberAttr(context.element, "z");
- const maxWords = Math.max(1, Math.round(numberAttr(context.element, "maxWords", 3)));
- const words = located.words;
- const styledRanges = elements(context, "style").map((style) => ({
+ const styledRanges = elements(context, "style").map(style => ({
id: stringAttr(style, "id"),
- ranges: context.selection(style.attributes.during, "style.during"),
+ ranges: selectorRanges(style, "style", true),
color: style.attributes.color,
activeColor: style.attributes.activeColor,
background: style.attributes.background,
weight: style.attributes.weight,
scale: style.attributes.scale,
}));
+ const mutedRanges = elements(context, "mute")
+ .flatMap(mute => selectorRanges(mute, "mute", false));
+ const overlaps = (unit, range) =>
+ unit.startFrame < range.endFrameExclusive
+ && unit.endFrameExclusive > range.startFrame;
const inlineStyle = (word, active) => {
const declarations = {};
for (const rule of styledRanges) {
- if (!rule.ranges.some((range) =>
- word.startFrame < range.endFrameExclusive
- && word.endFrameExclusive > range.startFrame)) continue;
+ if (!rule.ranges.some(range => overlaps(word, range))) continue;
if (typeof rule.color === "string") declarations.color = rule.color;
if (active && typeof rule.activeColor === "string") declarations.color = rule.activeColor;
if (typeof rule.background === "string") declarations.background = rule.background;
@@ -71,45 +106,50 @@ export default {
.map(([name, value]) => `${name}:${String(value)}`).join(";");
return body ? ` style="${html(body)}"` : "";
};
- const atomsByStart = new Map(located.captionAtoms.map((atom) => [atom.startWord, atom]));
- const displayUnits = (startWord, endWordExclusive) => {
+ const atomsByStart = new Map(captionAtoms.map(atom => [atom.startToken, atom]));
+ const displayUnits = (startToken, endTokenExclusive) => {
const units = [];
- let index = startWord;
- while (index < endWordExclusive) {
+ let index = startToken;
+ while (index < endTokenExclusive) {
const atom = atomsByStart.get(index);
if (atom) {
- if (atom.endWordExclusive > endWordExclusive) {
- throw new Error(`caption cue cuts through Dual Text atom "${atom.id}"`);
+ if (atom.endTokenExclusive > endTokenExclusive) {
+ fail(`caption cue cuts through Dual Text atom "${atom.id}"`);
}
- if (atom.display) {
+ if (atom.display && !mutedRanges.some(range => overlaps(atom, range))) {
units.push({
text: atom.display,
startFrame: atom.startFrame,
endFrameExclusive: atom.endFrameExclusive,
});
}
- index = atom.endWordExclusive;
+ index = atom.endTokenExclusive;
continue;
}
const word = words[index];
- if (word) units.push(word);
+ if (word && !mutedRanges.some(range => overlaps(word, range))) units.push(word);
index += 1;
}
return units;
};
- const groups = located.captionCues?.map((cue) => ({
- cue,
- words: displayUnits(cue.startWord, cue.endWordExclusive),
- })) ?? [];
- if (groups.length === 0) {
- const units = displayUnits(0, words.length);
- for (let index = 0; index < units.length; index += maxWords) {
- groups.push({
- words: units.slice(index, index + maxWords),
- });
- }
- }
+ let expectedStart = 0;
+ const groups = plan.cues.map((cue, index) => {
+ if (
+ !Number.isInteger(cue.startToken)
+ || !Number.isInteger(cue.endTokenExclusive)
+ || cue.startToken !== expectedStart
+ || cue.endTokenExclusive <= cue.startToken
+ || cue.endTokenExclusive > words.length
+ ) fail(`CaptionPlan cue ${index + 1} is not a complete ordered token partition`);
+ const first = words[cue.startToken];
+ const last = words[cue.endTokenExclusive - 1];
+ if (first.segmentId !== last.segmentId) fail(`CaptionPlan cue ${index + 1} crosses Segments`);
+ expectedStart = cue.endTokenExclusive;
+ return { cue, words: displayUnits(cue.startToken, cue.endTokenExclusive) };
+ }).filter(entry => entry.words.length);
+ if (expectedStart !== words.length) fail("CaptionPlan does not cover every Script token");
const visuals = [];
+ const z = numberAttr(context.element, "z");
for (const [groupIndex, entry] of groups.entries()) {
const group = entry.words;
const nextCueStart = groups[groupIndex + 1]?.words[0]?.startFrame;
@@ -117,8 +157,8 @@ export default {
const next = group[activeIndex + 1];
const endFrameExclusive = next?.startFrame
?? Math.min(
- located.durationFrames,
- nextCueStart ?? located.durationFrames,
+ context.program.durationFrames,
+ nextCueStart ?? context.program.durationFrames,
word.endFrameExclusive + Math.round(numberAttr(context.element, "hold", 0.08) * context.fps),
);
if (endFrameExclusive <= word.startFrame) continue;
@@ -126,14 +166,8 @@ export default {
const active = index === activeIndex;
return `<span class="${active ? "active" : ""}"${inlineStyle(item, active)}>${html(item.text)}</span>`;
}).join("");
- const fragmentDuration = Math.max(
- 1 / context.fps,
- (endFrameExclusive - word.startFrame) / context.fps,
- );
- const fadeSec = Math.min(
- numberAttr(context.element, "cueFade", 0.12),
- fragmentDuration * 0.45,
- );
+ const fragmentDuration = Math.max(1 / context.fps, (endFrameExclusive - word.startFrame) / context.fps);
+ const fadeSec = Math.min(numberAttr(context.element, "cueFade", 0.12), fragmentDuration * 0.45);
const fadePercent = Math.min(45, fadeSec / fragmentDuration * 100);
const animationName = `svml-caption-${context.instance.id}-${groupIndex}-${activeIndex}`
.replaceAll(/[^A-Za-z0-9_-]/gu, "-");
@@ -167,11 +201,7 @@ export default {
"text-align": "center",
"-webkit-text-stroke": `${context.width * numberAttr(context.element, "strokeWidth", 0.002)}px ${stringAttr(context.element, "strokeColor", "rgba(0,0,0,.82)")}`,
"paint-order": "stroke fill",
- "text-shadow": stringAttr(
- context.element,
- "shadow",
- "0 2px 9px rgba(0,0,0,.5), 0 0 2px rgba(0,0,0,.68)",
- ),
+ "text-shadow": stringAttr(context.element, "shadow", "0 2px 9px rgba(0,0,0,.5), 0 0 2px rgba(0,0,0,.68)"),
animation: `${animationName} ${fragmentDuration}s linear both`,
},
css: `@keyframes ${animationName} { ${opacityKeyframes} }`,
@@ -179,13 +209,11 @@ export default {
}
}
const highlight = stringAttr(context.element, "highlight", "#ffd34d");
- const gap = Math.max(
- 12,
- Math.round(
- numberAttr(context.element, "size", 50)
- * (0.34 + Math.max(0, -numberAttr(context.element, "tracking", -1) / numberAttr(context.element, "size", 50))),
- ),
- );
+ const fontSize = numberAttr(context.element, "size", 50);
+ const gap = Math.max(12, Math.round(fontSize * (0.34 + Math.max(
+ 0,
+ -numberAttr(context.element, "tracking", -1) / fontSize,
+ ))));
return trackOutput(visuals, [], [
`.svml-caption-line .active { color: ${highlight}; }`,
`.svml-caption-line {
diff --git a/stdlib/runtime/film.mjs b/stdlib/runtime/film.mjs
index 0aa411bb..5552a7cc 100644
--- a/stdlib/runtime/film.mjs
+++ b/stdlib/runtime/film.mjs
@@ -11,8 +11,8 @@ export default {
if (production?.contract !== "svml.temporal-basis-production.v1") {
throw new Error("film requires TemporalBasisProduction");
}
- if (semantic?.contract !== "svml.exact-semantic-map.v1") {
- throw new Error("film requires ExactSemanticMap");
+ if (semantic?.contract !== "svml.complete-semantic-map.v1") {
+ throw new Error("film requires CompleteSemanticMap");
}
if (semantic.basisDigest !== production.basis.basisDigest) {
throw new Error("film basis and SemanticMap do not share a basisDigest");
diff --git a/stdlib/runtime/speech-locator.mjs b/stdlib/runtime/speech-locator.mjs
index 547da910..427efe95 100644
--- a/stdlib/runtime/speech-locator.mjs
+++ b/stdlib/runtime/speech-locator.mjs
@@ -9,6 +9,150 @@ function fail(message) {
throw new Error(message);
}
+function lcsMatches(tokens, units) {
+ const rows = Array.from({ length: tokens.length + 1 }, () =>
+ Array(units.length + 1).fill(0));
+ for (let left = tokens.length - 1; left >= 0; left -= 1) {
+ for (let right = units.length - 1; right >= 0; right -= 1) {
+ rows[left][right] = tokens[left].normalized === normalizeWord(units[right].text)
+ ? rows[left + 1][right + 1] + 1
+ : Math.max(rows[left + 1][right], rows[left][right + 1]);
+ }
+ }
+ const matches = [];
+ let left = 0;
+ let right = 0;
+ while (left < tokens.length && right < units.length) {
+ if (
+ tokens[left].normalized === normalizeWord(units[right].text)
+ && rows[left][right] === rows[left + 1][right + 1] + 1
+ ) {
+ matches.push([left, right]);
+ left += 1;
+ right += 1;
+ } else if (rows[left + 1][right] >= rows[left][right + 1]) {
+ left += 1;
+ } else {
+ right += 1;
+ }
+ }
+ return matches;
+}
+
+function distributeFrames(tokens, startFrame, endFrameExclusive, quality) {
+ const weights = tokens.map(token => Math.max(1, [...token.normalized].length));
+ const total = weights.reduce((sum, value) => sum + value, 0);
+ let accumulated = 0;
+ return tokens.map((_, index) => {
+ const start = startFrame + Math.round((endFrameExclusive - startFrame) * accumulated / total);
+ accumulated += weights[index];
+ const end = index === tokens.length - 1
+ ? endFrameExclusive
+ : startFrame + Math.round((endFrameExclusive - startFrame) * accumulated / total);
+ return { startFrame: start, endFrameExclusive: Math.max(start, end), quality };
+ });
+}
+
+function alignSegment(tokens, units, segment, fps, evidenceQuality) {
+ const output = Array(tokens.length).fill(undefined);
+ const matches = lcsMatches(tokens, units);
+ let tokenCursor = 0;
+ let unitCursor = 0;
+ for (let matchIndex = 0; matchIndex <= matches.length; matchIndex += 1) {
+ const match = matches[matchIndex];
+ const tokenEnd = match?.[0] ?? tokens.length;
+ const unitEnd = match?.[1] ?? units.length;
+ const tokenRun = tokens.slice(tokenCursor, tokenEnd);
+ const unitRun = units.slice(unitCursor, unitEnd);
+ if (tokenRun.length) {
+ const previous = tokenCursor > 0 ? output[tokenCursor - 1] : undefined;
+ const nextUnit = match ? units[unitEnd] : undefined;
+ const startFrame = unitRun.length
+ ? Math.round(unitRun[0].startSec * fps)
+ : previous?.endFrameExclusive ?? Math.round(segment.startSec * fps);
+ const endFrame = unitRun.length
+ ? Math.round(unitRun.at(-1).endSec * fps)
+ : nextUnit
+ ? Math.round(nextUnit.startSec * fps)
+ : Math.round(segment.endSec * fps);
+ const quality = evidenceQuality === "estimated" || !unitRun.length
+ ? "estimated"
+ : "derived";
+ for (const [offset, value] of distributeFrames(
+ tokenRun,
+ startFrame,
+ Math.max(startFrame, endFrame),
+ quality,
+ ).entries()) output[tokenCursor + offset] = value;
+ }
+ if (match) {
+ const [tokenIndex, unitIndex] = match;
+ const unit = units[unitIndex];
+ output[tokenIndex] = {
+ startFrame: Math.round(unit.startSec * fps),
+ endFrameExclusive: Math.round(unit.endSec * fps),
+ quality: evidenceQuality === "measured" ? "measured" : "estimated",
+ };
+ tokenCursor = tokenIndex + 1;
+ unitCursor = unitIndex + 1;
+ }
+ }
+ if (output.some(value => !value)) fail(`could not align every Script token in Segment ${segment.id}`);
+ return output;
+}
+
+function validate(narrative, basis, evidence) {
+ if (evidence?.contract !== "svml.speech-timing-evidence.v1") {
+ fail("speech-locator requires SpeechTimingEvidence");
+ }
+ if (!['measured', 'estimated'].includes(evidence.quality)) fail("invalid speech timing quality");
+ const fps = basis.frameRate.numerator / basis.frameRate.denominator;
+ if (Math.abs(evidence.fps - fps) > 1e-6) fail("speech timing fps does not match ProgramBasis");
+ if (Math.round(evidence.durationSec * fps) !== basis.durationFrames) {
+ fail("speech timing duration does not match ProgramBasis");
+ }
+ if (evidence.segments.length !== narrative.segments.length) {
+ fail("speech timing Segment cardinality mismatch");
+ }
+ const segments = new Map(evidence.segments.map(segment => [segment.id, segment]));
+ if (segments.size !== evidence.segments.length) fail("speech timing repeats a Segment id");
+ for (const [index, scriptSegment] of narrative.segments.entries()) {
+ const segment = segments.get(scriptSegment.id);
+ if (!segment) fail(`speech timing is missing Segment ${scriptSegment.id}`);
+ if (
+ !Number.isFinite(segment.startSec)
+ || !Number.isFinite(segment.endSec)
+ || segment.startSec < 0
+ || segment.endSec < segment.startSec
+ || segment.endSec > evidence.durationSec + 1e-6
+ ) fail(`invalid speech timing window for Segment ${scriptSegment.id}`);
+ const previousScript = narrative.segments[index - 1];
+ const previous = previousScript ? segments.get(previousScript.id) : undefined;
+ if (previous && (segment.startSec < previous.startSec - 1e-6 || segment.endSec < previous.endSec - 1e-6)) {
+ fail(`speech timing reorders Segment ${scriptSegment.id}`);
+ }
+ }
+ const previousBySegment = new Map();
+ for (const [index, unit] of evidence.units.entries()) {
+ const segment = segments.get(unit.segmentId);
+ if (
+ !segment
+ || typeof unit.text !== "string"
+ || !Number.isFinite(unit.startSec)
+ || !Number.isFinite(unit.endSec)
+ || unit.startSec < segment.startSec - 1e-6
+ || unit.endSec < unit.startSec
+ || unit.endSec > segment.endSec + 1e-6
+ ) fail(`speech timing unit ${index + 1} falls outside its Segment`);
+ const previous = previousBySegment.get(unit.segmentId);
+ if (previous && unit.startSec < previous.endSec - 1e-6) {
+ fail(`speech timing units overlap inside Segment ${unit.segmentId}`);
+ }
+ previousBySegment.set(unit.segmentId, unit);
+ }
+ return { fps, segments };
+}
+
export default {
abiVersion: "1",
project(context) {
@@ -20,60 +164,35 @@ export default {
if (production?.contract !== "svml.temporal-basis-production.v1") {
fail("speech-locator requires TemporalBasisProduction");
}
- if (evidence?.contract !== "svml.speech-alignment.v1") fail("unsupported alignment evidence");
const basis = production.basis;
- const fps = basis.frameRate.numerator / basis.frameRate.denominator;
- if (Math.round(evidence.durationSec * fps) !== basis.durationFrames) {
- fail("alignment duration does not match ProgramBasis");
- }
- if (evidence.words.length !== narrative.tokens.length) fail("alignment token cardinality mismatch");
- if (evidence.segments.length !== narrative.segments.length) fail("alignment Segment cardinality mismatch");
- const segments = new Map(evidence.segments.map(segment => [segment.id, segment]));
- if (segments.size !== evidence.segments.length) fail("alignment repeats a Segment id");
- const words = new Map();
- const previousBySegment = new Map();
- for (const [index, token] of narrative.tokens.entries()) {
- const word = evidence.words[index];
- const segment = segments.get(token.segmentId);
- if (!word || !segment || word.segmentId !== token.segmentId || normalizeWord(word.text) !== token.normalized) {
- fail(`alignment token ${index + 1} does not match Script`);
- }
- if (word.startSec < segment.startSec || word.endSec < word.startSec || word.endSec > segment.endSec) {
- fail(`alignment token ${index + 1} falls outside Segment`);
- }
- const previous = previousBySegment.get(word.segmentId);
- if (previous && word.startSec < previous.endSec - 1e-6) fail(`alignment overlaps inside Segment ${word.segmentId}`);
- previousBySegment.set(word.segmentId, word);
- words.set(token.id, word);
- }
- for (const [index, scriptSegment] of narrative.segments.entries()) {
+ const { fps, segments } = validate(narrative, basis, evidence);
+ const tokenFrames = new Map();
+ for (const scriptSegment of narrative.segments) {
const segment = segments.get(scriptSegment.id);
- if (!segment) fail(`alignment is missing Segment ${scriptSegment.id}`);
- const previousScript = narrative.segments[index - 1];
- const previous = previousScript ? segments.get(previousScript.id) : undefined;
- if (previous && (segment.startSec < previous.startSec || segment.endSec < previous.endSec)) {
- fail(`alignment reorders Segment ${scriptSegment.id}`);
- }
+ const tokens = narrative.tokens.slice(scriptSegment.tokenStart, scriptSegment.tokenEnd);
+ const units = evidence.units.filter(unit => unit.segmentId === scriptSegment.id);
+ const aligned = alignSegment(tokens, units, segment, fps, evidence.quality);
+ tokens.forEach((token, index) => tokenFrames.set(token.id, aligned[index]));
}
const anchors = narrative.semanticIndex.anchors.map(anchor => {
const segment = segments.get(anchor.segmentId);
- const word = anchor.tokenId ? words.get(anchor.tokenId) : undefined;
- const seconds = anchor.kind === "segment-start"
- ? segment?.startSec
+ const token = anchor.tokenId ? tokenFrames.get(anchor.tokenId) : undefined;
+ const frame = anchor.kind === "segment-start"
+ ? Math.round(segment.startSec * fps)
: anchor.kind === "segment-end"
- ? segment?.endSec
+ ? Math.round(segment.endSec * fps)
: anchor.kind === "token-start"
- ? word?.startSec
- : word?.endSec;
- if (seconds === undefined) fail(`alignment is missing anchor ${anchor.id}`);
+ ? token?.startFrame
+ : token?.endFrameExclusive;
+ if (frame === undefined) fail(`speech timing is missing anchor ${anchor.id}`);
return {
identity: anchor.id,
- point: { basisDigest: basis.basisDigest, frame: Math.round(seconds * fps) },
- quality: "measured",
+ point: { basisDigest: basis.basisDigest, frame },
+ quality: token?.quality ?? evidence.quality,
};
});
const payload = {
- contract: "svml.exact-semantic-map.v1",
+ contract: "svml.complete-semantic-map.v1",
semanticIndexDigest: narrative.semanticIndex.digest,
basisDigest: basis.basisDigest,
anchors,
@@ -83,12 +202,7 @@ export default {
quantizationPolicy: "nearest-frame",
}),
quantizationPolicy: "nearest-frame",
- ...(evidence.captionCues ? { captionCues: evidence.captionCues } : {}),
- };
- return {
- outputs: {
- map: { ...payload, mapDigest: context.digest(payload) },
- },
};
+ return { outputs: { map: { ...payload, mapDigest: context.digest(payload) } } };
},
};
diff --git a/stdlib/speech-locator.svk b/stdlib/speech-locator.svk
index 34b36f0c..a7bd5758 100644
--- a/stdlib/speech-locator.svk
+++ b/stdlib/speech-locator.svk
@@ -1,6 +1,6 @@
<kernel name="speech-locator" abi="1" implementation="./runtime/speech-locator.mjs">
<port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
<port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
- <port name="evidence" direction="input" type="AlignmentEvidence" cardinality="one"/>
- <port name="map" direction="output" type="ExactSemanticMap" cardinality="one"/>
+ <port name="evidence" direction="input" type="SpeechTimingEvidence" cardinality="one"/>
+ <port name="map" direction="output" type="CompleteSemanticMap" cardinality="one"/>
</kernel>
diff --git a/test/artifacts.test.ts b/test/artifacts.test.ts
index 987a4725..b7a362b0 100644
--- a/test/artifacts.test.ts
+++ b/test/artifacts.test.ts
@@ -103,7 +103,7 @@ test("capability instances require exact verified artifacts and never invoke a p
);
});
-test("a Locator capability returns ExactSemanticMap through the same typed Artifact ABI", async () => {
+test("a Locator capability returns CompleteSemanticMap through the same typed Artifact ABI", async () => {
const source = fileURLToPath(
new URL("../examples/composite-speech-program/minimal.svml", import.meta.url),
);
@@ -134,13 +134,13 @@ test("a Locator capability returns ExactSemanticMap through the same typed Artif
</script>
<video id="host" src="${avatar}"/>
<audio id="host-audio" src="${audio}"/>
- <alignment id="alignment" src="${evidence}"/>
+ <timing id="timing" src="${evidence}"/>
<speech-assemble id="voice" fps="30" defaultJoin="cut">
<segment id="intro" script={script.segment.intro} visual={host} audio={host-audio} sourceWindow="0s .. 5s"/>
<segment id="close" script={script.segment.close} visual={host} audio={host-audio} sourceWindow="5s .. 10s"/>
<join after="intro" overlap="500ms" audio="crossfade" visual="dissolve"/>
</speech-assemble>
- <pinned-locator id="location" script={script} basis={voice.production} evidence={alignment}/>
+ <pinned-locator id="location" script={script} basis={voice.production} evidence={timing}/>
<film id="film" basis={voice.production} semantic={location.map}/>
</svml>\n`, "utf8");
const checked = await checkSource(svml);
@@ -153,7 +153,7 @@ test("a Locator capability returns ExactSemanticMap through the same typed Artif
executionDigest: instance?.executionDigest,
outputs: {
map: {
- type: "ExactSemanticMap",
+ type: "CompleteSemanticMap",
value: reference.semanticMap,
contentDigest: sha256(stableJson(reference.semanticMap)),
},
diff --git a/test/composite.test.ts b/test/composite.test.ts
index a3ea210a..3c4c7613 100644
--- a/test/composite.test.ts
+++ b/test/composite.test.ts
@@ -29,6 +29,8 @@ test("composite-v1 expands finite internal Basis/Locator instances with audited
assert.match(internal?.sourceMap?.definitionSite.file ?? "", /speech-program\.svk$/u);
const compilation = await compileSource({ file: fixture });
+ assert.equal(compilation.plan.kernels.some((kernel) =>
+ kernel.ports.some((port) => port.type === "CaptionTrack")), false);
assert.equal(compilation.located.durationFrames, 285);
assert.equal(compilation.basis.sourceMaps.length, 4);
assert.equal(compilation.basis.audio[0]?.fadeOutFrames, 15);
diff --git a/test/fixtures/artifacts/alignment.json b/test/fixtures/artifacts/alignment.json
index 2b301f1d..eaf3c0ce 100644
--- a/test/fixtures/artifacts/alignment.json
+++ b/test/fixtures/artifacts/alignment.json
@@ -1,8 +1,9 @@
{
- "contract": "svml.speech-alignment.v1",
+ "contract": "svml.speech-timing-evidence.v1",
"durationSec": 1,
"fps": 30,
- "words": [
+ "quality": "measured",
+ "units": [
{
"text": "Hello",
"startSec": 0,
diff --git a/test/fixtures/artifacts/artifact-film.svk b/test/fixtures/artifacts/artifact-film.svk
index b1611e71..8908a8f4 100644
--- a/test/fixtures/artifacts/artifact-film.svk
+++ b/test/fixtures/artifacts/artifact-film.svk
@@ -6,6 +6,6 @@
>
<port name="image" direction="input" type="Image" cardinality="one"/>
<port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
- <port name="semantic" direction="input" type="ExactSemanticMap" cardinality="one"/>
+ <port name="semantic" direction="input" type="CompleteSemanticMap" cardinality="one"/>
<port name="document" direction="output" type="HyperframesDocument" cardinality="one"/>
</kernel>
diff --git a/test/fixtures/artifacts/main.svml b/test/fixtures/artifacts/main.svml
index e8a49887..219bb0c2 100644
--- a/test/fixtures/artifacts/main.svml
+++ b/test/fixtures/artifacts/main.svml
@@ -9,11 +9,11 @@
</script>
<audio id="voice-audio" src="./alignment.json"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} audio={voice-audio} sourceWindow="0s .. 1s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<pinned-image id="poster" prompt="A pinned result; never call a provider."/>
<artifact-film id="film" image={poster.image} basis={speech.production} semantic={location.map}/>
diff --git a/test/fixtures/artifacts/pinned-locator.svk b/test/fixtures/artifacts/pinned-locator.svk
index 6296311c..855cdabb 100644
--- a/test/fixtures/artifacts/pinned-locator.svk
+++ b/test/fixtures/artifacts/pinned-locator.svk
@@ -1,6 +1,6 @@
<kernel name="pinned-locator" abi="1" profile="capability-v1" capability="semantic-locate">
<port name="script" direction="input" type="NarrativeIR" cardinality="one"/>
<port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
- <port name="evidence" direction="input" type="AlignmentEvidence" cardinality="one"/>
- <port name="map" direction="output" type="ExactSemanticMap" cardinality="one"/>
+ <port name="evidence" direction="input" type="SpeechTimingEvidence" cardinality="one"/>
+ <port name="map" direction="output" type="CompleteSemanticMap" cardinality="one"/>
</kernel>
diff --git a/test/fixtures/collision/alignment.json b/test/fixtures/collision/alignment.json
index 413c7b61..f1dd6006 100644
--- a/test/fixtures/collision/alignment.json
+++ b/test/fixtures/collision/alignment.json
@@ -1,8 +1,9 @@
{
- "contract": "svml.speech-alignment.v1",
+ "contract": "svml.speech-timing-evidence.v1",
"durationSec": 2,
"fps": 30,
- "words": [
+ "quality": "measured",
+ "units": [
{ "text": "Left", "startSec": 0.1, "endSec": 0.5, "segmentId": "main" },
{ "text": "and", "startSec": 0.6, "endSec": 0.9, "segmentId": "main" },
{ "text": "right", "startSec": 1, "endSec": 1.5, "segmentId": "main" }
diff --git a/test/fixtures/collision/main.svml b/test/fixtures/collision/main.svml
index 0925685b..b915620f 100644
--- a/test/fixtures/collision/main.svml
+++ b/test/fixtures/collision/main.svml
@@ -12,7 +12,7 @@
<video id="avatar" src="./avatar.mp4"/>
<audio id="voice-audio" src="./avatar.mp4"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<image id="left-image" src="./left/image.png"/>
<image id="right-image" src="./right/image.png"/>
@@ -20,7 +20,7 @@
<segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 2s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<broll-track id="broll" z="100">
<item id="left" source={left-image} during={script.selection.left}/>
diff --git a/test/fixtures/disconnected/alignment.json b/test/fixtures/disconnected/alignment.json
index a5401de8..5d28c064 100644
--- a/test/fixtures/disconnected/alignment.json
+++ b/test/fixtures/disconnected/alignment.json
@@ -1,8 +1,9 @@
{
- "contract": "svml.speech-alignment.v1",
+ "contract": "svml.speech-timing-evidence.v1",
"durationSec": 3,
"fps": 30,
- "words": [
+ "quality": "measured",
+ "units": [
{ "text": "Alpha", "startSec": 0, "endSec": 0.7, "segmentId": "main" },
{ "text": "gap", "startSec": 1, "endSec": 1.5, "segmentId": "main" },
{ "text": "omega", "startSec": 2, "endSec": 2.7, "segmentId": "main" }
diff --git a/test/fixtures/disconnected/captions.svml b/test/fixtures/disconnected/captions.svml
index 9081f409..eeb0c7d5 100644
--- a/test/fixtures/disconnected/captions.svml
+++ b/test/fixtures/disconnected/captions.svml
@@ -1,22 +1,31 @@
<svml version="1">
<import from="../../../stdlib/speech-assemble.svk"/>
<import from="../../../stdlib/speech-locator.svk"/>
+ <import from="../../../stdlib/caption-planner.svk"/>
<import from="../../../stdlib/caption-track.svk"/>
<import from="../../../stdlib/film.svk"/>
<script>
- <segment id="main">@pick Alpha @/pick gap @pick omega @/pick.</segment>
+ <segment id="main">
+ <A> @pick Alpha @/pick @silent gap @/silent
+ <B> @pick omega @/pick.
+ </segment>
</script>
<audio id="voice-audio" src="../presentation/long.mp4"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} audio={voice-audio} sourceWindow="0s .. 3s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
- <caption-track id="captions" script={script} semantic={location.map} z="10" maxWords="3">
- <style id="picked" during={script.selection.pick} color="#ff0000" activeColor="#00ff00" scale="1.1"/>
+ <caption-planner id="caption-plan" script={script} semantic={location.map} maxWords="3">
+ <annotate id="picked" kind="accent" during={script.selection.pick}/>
+ </caption-planner>
+ <caption-track id="captions" script={script} semantic={location.map} plan={caption-plan.plan} z="10">
+ <style id="speaker-a" role="A" background="#0000ff"/>
+ <style id="picked" annotation="accent" color="#ff0000" activeColor="#00ff00" scale="1.1"/>
+ <mute id="silent" during={script.selection.silent}/>
</caption-track>
<film id="film" basis={speech.production} semantic={location.map}>
diff --git a/test/fixtures/disconnected/duplicate-captions.svml b/test/fixtures/disconnected/duplicate-captions.svml
new file mode 100644
index 00000000..39286f14
--- /dev/null
+++ b/test/fixtures/disconnected/duplicate-captions.svml
@@ -0,0 +1,22 @@
+<svml version="1">
+ <import from="../../../stdlib/speech-assemble.svk"/>
+ <import from="../../../stdlib/speech-locator.svk"/>
+ <import from="../../../stdlib/caption-planner.svk"/>
+ <import from="../../../stdlib/caption-track.svk"/>
+ <import from="../../../stdlib/film.svk"/>
+
+ <script><segment id="main">Alpha omega.</segment></script>
+ <audio id="voice-audio" src="../presentation/long.mp4"/>
+ <timing id="timing" src="./alignment.json"/>
+ <speech-assemble id="speech" fps="30">
+ <segment id="main" script={script.segment.main} audio={voice-audio} sourceWindow="0s .. 3s"/>
+ </speech-assemble>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
+ <caption-planner id="caption-plan" script={script} semantic={location.map}/>
+ <caption-track id="captions-a" script={script} semantic={location.map} plan={caption-plan.plan} z="10"/>
+ <caption-track id="captions-b" script={script} semantic={location.map} plan={caption-plan.plan} z="11"/>
+ <film id="film" basis={speech.production} semantic={location.map}>
+ <track ref={captions-a.track}/>
+ <track ref={captions-b.track}/>
+ </film>
+</svml>
diff --git a/test/fixtures/disconnected/each.svml b/test/fixtures/disconnected/each.svml
index 5cc168df..d2f60278 100644
--- a/test/fixtures/disconnected/each.svml
+++ b/test/fixtures/disconnected/each.svml
@@ -10,12 +10,12 @@
<video id="avatar" src="../presentation/long.mp4"/>
<audio id="voice-audio" src="../presentation/long.mp4"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 3s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<media-track id="media" z="1">
<item id="picked" source={avatar} during={script.selection.pick}/>
diff --git a/test/fixtures/disconnected/one.svml b/test/fixtures/disconnected/one.svml
index f7e737f4..86ceafbf 100644
--- a/test/fixtures/disconnected/one.svml
+++ b/test/fixtures/disconnected/one.svml
@@ -11,12 +11,12 @@
<video id="avatar" src="../presentation/long.mp4"/>
<audio id="voice-audio" src="../presentation/long.mp4"/>
<image id="icon" src="../collision/left/image.png"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 3s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<ranking-column id="ranking" z="1">
<item id="picked" rank="1" image={icon} during={script.selection.pick}/>
diff --git a/test/fixtures/disconnected/set.svml b/test/fixtures/disconnected/set.svml
index 5f78cd9a..ff951c22 100644
--- a/test/fixtures/disconnected/set.svml
+++ b/test/fixtures/disconnected/set.svml
@@ -10,12 +10,12 @@
<video id="avatar" src="../presentation/long.mp4"/>
<audio id="voice-audio" src="../presentation/long.mp4"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 3s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<selection-set id="styles">
<item id="highlight" during={script.selection.pick}/>
diff --git a/test/fixtures/media-basis/main.svml b/test/fixtures/media-basis/main.svml
index 0bc1f7bc..7f0b36d0 100644
--- a/test/fixtures/media-basis/main.svml
+++ b/test/fixtures/media-basis/main.svml
@@ -6,9 +6,9 @@
<script><segment id="main">Hello.</segment></script>
<video id="visual" src="../collision/avatar.mp4"/>
<audio id="audio" src="../collision/avatar.mp4"/>
- <alignment id="alignment" src="../artifacts/alignment.json"/>
+ <timing id="timing" src="../artifacts/alignment.json"/>
<media-basis id="program" visual={visual} audio={audio} duration="1s"/>
- <speech-locator id="location" script={script} basis={program.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={program.production} evidence={timing}/>
<film id="film" basis={program.production} semantic={location.map}/>
</svml>
diff --git a/test/fixtures/modules/alignment.json b/test/fixtures/modules/alignment.json
index 5d086519..02f6b18e 100644
--- a/test/fixtures/modules/alignment.json
+++ b/test/fixtures/modules/alignment.json
@@ -1,8 +1,9 @@
{
- "contract": "svml.speech-alignment.v1",
+ "contract": "svml.speech-timing-evidence.v1",
"durationSec": 1,
"fps": 30,
- "words": [
+ "quality": "measured",
+ "units": [
{
"text": "Hello",
"startSec": 0.1,
diff --git a/test/fixtures/modules/main-renamed-alias.svml b/test/fixtures/modules/main-renamed-alias.svml
index 743ba685..25f8b8d8 100644
--- a/test/fixtures/modules/main-renamed-alias.svml
+++ b/test/fixtures/modules/main-renamed-alias.svml
@@ -2,6 +2,7 @@
<import from="../../../stdlib/speech-assemble.svk"/>
<import from="../../../stdlib/speech-locator.svk"/>
<import from="../../../stdlib/broll-track.svk"/>
+ <import from="../../../stdlib/caption-planner.svk"/>
<import as="subtitles" from="../../../stdlib/caption-track.svk"/>
<import from="../../../stdlib/film.svk"/>
<import as="assets" from="./brand.svc"/>
@@ -15,19 +16,20 @@
<video id="avatar" src="./avatar.mp4"/>
<audio id="voice-audio" src="./avatar.mp4"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 1s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<broll-track id="broll" class="house.full">
<item id="poster" source={assets.poster.image} during={script.selection.poster}/>
</broll-track>
- <subtitles id="captions" script={script} semantic={location.map} class="house.launch" size="48"/>
+ <caption-planner id="caption-plan" script={script} semantic={location.map}/>
+ <subtitles id="captions" script={script} semantic={location.map} plan={caption-plan.plan} class="house.launch" size="48"/>
<film id="main-film" class="house.vertical" basis={speech.production} semantic={location.map}>
<track ref={broll.track}/>
diff --git a/test/fixtures/modules/main.svml b/test/fixtures/modules/main.svml
index 178a9e20..68e0afc4 100644
--- a/test/fixtures/modules/main.svml
+++ b/test/fixtures/modules/main.svml
@@ -2,6 +2,7 @@
<import from="../../../stdlib/speech-assemble.svk"/>
<import from="../../../stdlib/speech-locator.svk"/>
<import from="../../../stdlib/broll-track.svk"/>
+ <import from="../../../stdlib/caption-planner.svk"/>
<import from="../../../stdlib/caption-track.svk"/>
<import from="../../../stdlib/film.svk"/>
<import as="brand" from="./brand.svc"/>
@@ -15,19 +16,20 @@
<video id="avatar" src="./avatar.mp4"/>
<audio id="voice-audio" src="./avatar.mp4"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} visual={avatar} audio={voice-audio} sourceWindow="0s .. 1s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<broll-track id="broll" class="house.full">
<item id="poster" source={brand.poster.image} during={script.selection.poster}/>
</broll-track>
- <caption-track id="captions" script={script} semantic={location.map} class="house.launch" size="48"/>
+ <caption-planner id="caption-plan" script={script} semantic={location.map}/>
+ <caption-track id="captions" script={script} semantic={location.map} plan={caption-plan.plan} class="house.launch" size="48"/>
<film id="main-film" class="house.vertical" basis={speech.production} semantic={location.map}>
<track ref={broll.track}/>
diff --git a/test/fixtures/presentation/alignment.json b/test/fixtures/presentation/alignment.json
index ec4e5903..581dec35 100644
--- a/test/fixtures/presentation/alignment.json
+++ b/test/fixtures/presentation/alignment.json
@@ -1,8 +1,9 @@
{
- "contract": "svml.speech-alignment.v1",
+ "contract": "svml.speech-timing-evidence.v1",
"durationSec": 3,
"fps": 30,
- "words": [
+ "quality": "measured",
+ "units": [
{ "text": "One", "startSec": 0.2, "endSec": 0.6, "segmentId": "main" },
{ "text": "two", "startSec": 1.1, "endSec": 1.5, "segmentId": "main" },
{ "text": "three", "startSec": 2.1, "endSec": 2.7, "segmentId": "main" }
diff --git a/test/fixtures/presentation/empty-present.svml b/test/fixtures/presentation/empty-present.svml
index 9b46e3e2..7644c3e8 100644
--- a/test/fixtures/presentation/empty-present.svml
+++ b/test/fixtures/presentation/empty-present.svml
@@ -12,12 +12,12 @@
<video id="long-video" src="./long.mp4"/>
<audio id="voice-audio" src="./long.mp4"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} visual={long-video} audio={voice-audio} sourceWindow="0s .. 3s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<media-track id="aroll" z="200">
<item id="host" source={long-video} during="full">
diff --git a/test/fixtures/presentation/main.svml b/test/fixtures/presentation/main.svml
index c41502d1..e74783c6 100644
--- a/test/fixtures/presentation/main.svml
+++ b/test/fixtures/presentation/main.svml
@@ -14,13 +14,13 @@
<video id="long-video" src="./long.mp4"/>
<audio id="voice-audio" src="./long.mp4"/>
<audio id="pop-sound" src="./pop.wav"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} visual={long-video} audio={voice-audio} sourceWindow="0s .. 3s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<media-track id="aroll" z="200">
<item id="host" source={long-video} during="full" fit="cover">
diff --git a/test/fixtures/unsafe/alignment.json b/test/fixtures/unsafe/alignment.json
index b6b288ff..568a3efb 100644
--- a/test/fixtures/unsafe/alignment.json
+++ b/test/fixtures/unsafe/alignment.json
@@ -1,8 +1,9 @@
{
- "contract": "svml.speech-alignment.v1",
+ "contract": "svml.speech-timing-evidence.v1",
"durationSec": 1,
"fps": 30,
- "words": [
+ "quality": "measured",
+ "units": [
{ "text": "Hello", "startSec": 0.1, "endSec": 0.8, "segmentId": "main" }
],
"segments": [
diff --git a/test/fixtures/unsafe/main.svml b/test/fixtures/unsafe/main.svml
index fb265c2f..b7f7b962 100644
--- a/test/fixtures/unsafe/main.svml
+++ b/test/fixtures/unsafe/main.svml
@@ -8,11 +8,11 @@
</script>
<audio id="voice-audio" src="./alignment.json"/>
- <alignment id="alignment" src="./alignment.json"/>
+ <timing id="timing" src="./alignment.json"/>
<speech-assemble id="speech" fps="30">
<segment id="main" script={script.segment.main} audio={voice-audio} sourceWindow="0s .. 1s"/>
</speech-assemble>
- <speech-locator id="location" script={script} basis={speech.production} evidence={alignment}/>
+ <speech-locator id="location" script={script} basis={speech.production} evidence={timing}/>
<unsafe id="main" basis={speech.production} semantic={location.map}/>
</svml>
diff --git a/test/fixtures/unsafe/unsafe.svk b/test/fixtures/unsafe/unsafe.svk
index dbc423bd..d7ca9692 100644
--- a/test/fixtures/unsafe/unsafe.svk
+++ b/test/fixtures/unsafe/unsafe.svk
@@ -1,5 +1,5 @@
<kernel name="unsafe" abi="1" root="true" implementation="./unsafe.mjs">
<port name="basis" direction="input" type="TemporalBasisProduction" cardinality="one"/>
- <port name="semantic" direction="input" type="ExactSemanticMap" cardinality="one"/>
+ <port name="semantic" direction="input" type="CompleteSemanticMap" cardinality="one"/>
<port name="document" direction="output" type="HyperframesDocument" cardinality="one"/>
</kernel>
diff --git a/test/modules.test.ts b/test/modules.test.ts
index 74df9f57..0d901df5 100644
--- a/test/modules.test.ts
+++ b/test/modules.test.ts
@@ -16,11 +16,11 @@ test(".svc content DAG and .svs parameter classes bind into one source closure",
assert.deepEqual(
checked.plan.instances.map((instance) => instance.id).sort(),
- ["brand.poster", "broll", "captions", "location", "main-film", "speech"],
+ ["brand.poster", "broll", "caption-plan", "captions", "location", "main-film", "speech"],
);
assert.deepEqual(
checked.plan.values.map((value) => value.id).sort(),
- ["alignment", "avatar", "brand.logo", "voice-audio"],
+ ["avatar", "brand.logo", "timing", "voice-audio"],
);
const logo = checked.plan.values.find((value) => value.id === "brand.logo");
assert.equal(logo?.localId, "logo");
diff --git a/test/presentation.test.ts b/test/presentation.test.ts
index 60c990b1..327d359c 100644
--- a/test/presentation.test.ts
+++ b/test/presentation.test.ts
@@ -94,4 +94,18 @@ test("disconnected SelectionSet drives one caption style rule without losing occ
assert.ok(captions.some((fragment) => /color:#ff0000/u.test(fragment.html ?? "")));
assert.ok(captions.some((fragment) => /color:#00ff00/u.test(fragment.html ?? "")));
assert.ok(captions.some((fragment) => /scale\(1\.1\)/u.test(fragment.html ?? "")));
+ assert.ok(captions.some((fragment) => /background:#0000ff/u.test(fragment.html ?? "")));
+ assert.ok(captions.every((fragment) => !/gap/u.test(fragment.html ?? "")));
+ const plan = compilation.projections.get("caption-plan")?.outputs.plan as {
+ annotations?: Array<{ kind: string }>;
+ };
+ assert.deepEqual(plan.annotations?.map((annotation) => annotation.kind), ["accent", "accent"]);
+});
+
+test("a Composition rejects a second CaptionTrack but permits no CaptionTrack", async () => {
+ const fixture = fileURLToPath(new URL("./fixtures/disconnected/", import.meta.url));
+ await assert.rejects(
+ compileSource({ file: `${fixture}/duplicate-captions.svml` }),
+ /caption_track_cardinality/u,
+ );
});
diff --git a/test/regen-ranking.test.ts b/test/regen-ranking.test.ts
index a6d64b0f..6bbd53d0 100644
--- a/test/regen-ranking.test.ts
+++ b/test/regen-ranking.test.ts
@@ -52,6 +52,9 @@ test("regen ranking compiles to the frozen production timing contract", async ()
assert.equal(document.audios?.length, reference.locatedOutput.audioFragments);
assert.match(compilation.html, /data-composition-id="regen-ranking"/u);
assert.match(compilation.html, /data-duration="36\.1"/u);
+ assert.match(compilation.html, />Powerful</u);
+ assert.doesNotMatch(compilation.html, /power full but|>uh</u);
+ assert.ok(compilation.semanticMap.anchors.some((anchor) => anchor.quality === "derived"));
const canvas = projectCanvas(compilation.plan);
assert.equal(canvas.root, reference.plan.root);
diff --git a/test/script.test.ts b/test/script.test.ts
index 6b3271c9..6fa6bb8c 100644
--- a/test/script.test.ts
+++ b/test/script.test.ts
@@ -1,26 +1,26 @@
import assert from "node:assert/strict";
import test from "node:test";
-import type { AlignmentEvidence, NarrativeIR } from "../src/model.js";
+import type { NarrativeIR, SpeechTimingEvidence } from "../src/model.js";
import { locateScript } from "../src/script/locate.js";
-import { estimateAlignment } from "../src/script/estimate.js";
+import { estimateSpeechTiming } from "../src/script/estimate.js";
import { parseScript } from "../src/script/parse.js";
import { formatDocumentScript } from "../src/format.js";
import { parseDocumentSource } from "../src/source/parse-document.js";
import {
createProgramBasis,
- semanticMapFromAlignment,
+ semanticMapFromTiming,
validateSemanticMap,
} from "../src/temporal.js";
import { sha256, stableJson } from "../src/util.js";
-function locateAlignment(narrative: NarrativeIR, evidence: AlignmentEvidence) {
+function locateTiming(narrative: NarrativeIR, evidence: SpeechTimingEvidence) {
const basis = createProgramBasis({
fps: evidence.fps,
durationFrames: Math.round(evidence.durationSec * evidence.fps),
outcomeDigest: sha256(stableJson(evidence)),
});
- const map = semanticMapFromAlignment(narrative, basis, evidence, { precision: "exact" });
- assert.equal(map.contract, "svml.exact-semantic-map.v1");
+ const map = semanticMapFromTiming(narrative, basis, evidence);
+ assert.equal(map.contract, "svml.complete-semantic-map.v1");
return locateScript(narrative, basis, map);
}
@@ -35,6 +35,9 @@ test("Script Surface projects role, dual text, selections and moments", () => {
assert.equal(narrative.projections.dialogue, "A: I just laughed my ass out.");
assert.equal(narrative.projections.speech, "I just laughed my ass out.");
assert.equal(narrative.projections.caption, "I just lmao.");
+ assert.deepEqual(narrative.turns.map((turn) => [turn.role, turn.tokenStart, turn.tokenEndExclusive]), [
+ ["A", 0, 6],
+ ]);
assert.equal(narrative.selections.whole?.length, 1);
assert.equal(narrative.selections.laugh?.length, 1);
assert.equal(narrative.selections.punch?.length, 1);
@@ -42,8 +45,8 @@ test("Script Surface projects role, dual text, selections and moments", () => {
assert.deepEqual(
narrative.captionAtoms.map((atom) => [
atom.display,
- atom.startWord,
- atom.endWordExclusive,
+ atom.startToken,
+ atom.endTokenExclusive,
]),
[["lmao", 2, 6]],
);
@@ -88,11 +91,12 @@ test("overlapping Segments retain distinct boundary affinity on one ProgramBasis
~@left! @right!
<segment id="b">Beta.</segment>
`);
- const evidence: AlignmentEvidence = {
- contract: "svml.speech-alignment.v1",
+ const evidence: SpeechTimingEvidence = {
+ contract: "svml.speech-timing-evidence.v1",
durationSec: 8,
fps: 30,
- words: [
+ quality: "measured",
+ units: [
{ text: "Alpha", startSec: 0.2, endSec: 4.8, segmentId: "a" },
{ text: "Beta", startSec: 4.6, endSec: 7.8, segmentId: "b" },
],
@@ -101,7 +105,7 @@ test("overlapping Segments retain distinct boundary affinity on one ProgramBasis
{ id: "b", startSec: 4.5, endSec: 8 },
],
};
- const located = locateAlignment(narrative, evidence);
+ const located = locateTiming(narrative, evidence);
assert.deepEqual(located.moments.left?.frames, [150]);
assert.deepEqual(located.moments.right?.frames, [135]);
assert.equal(located.segments.a?.endFrameExclusive, 150);
@@ -113,11 +117,12 @@ test("SemanticMap rejects source-order reversal, wrong basis and digest tamperin
"fixture.svml",
`<segment id="a">Alpha.</segment><segment id="b">Beta.</segment>`,
);
- const evidence: AlignmentEvidence = {
- contract: "svml.speech-alignment.v1",
+ const evidence: SpeechTimingEvidence = {
+ contract: "svml.speech-timing-evidence.v1",
durationSec: 5,
fps: 30,
- words: [
+ quality: "measured",
+ units: [
{ text: "Alpha", startSec: 1.1, endSec: 1.8, segmentId: "a" },
{ text: "Beta", startSec: 0.2, endSec: 0.8, segmentId: "b" },
],
@@ -132,13 +137,13 @@ test("SemanticMap rejects source-order reversal, wrong basis and digest tamperin
outcomeDigest: sha256("reordered"),
});
assert.throws(
- () => semanticMapFromAlignment(narrative, basis, evidence, { precision: "exact" }),
+ () => semanticMapFromTiming(narrative, basis, evidence),
/locator_segment_order/u,
);
- const validEvidence: AlignmentEvidence = {
+ const validEvidence: SpeechTimingEvidence = {
...evidence,
- words: [
+ units: [
{ text: "Alpha", startSec: 0.1, endSec: 0.8, segmentId: "a" },
{ text: "Beta", startSec: 1.1, endSec: 1.8, segmentId: "b" },
],
@@ -147,11 +152,10 @@ test("SemanticMap rejects source-order reversal, wrong basis and digest tamperin
{ id: "b", startSec: 1, endSec: 2 },
],
};
- const validMap = semanticMapFromAlignment(
+ const validMap = semanticMapFromTiming(
narrative,
basis,
validEvidence,
- { precision: "exact" },
);
assert.throws(
() => validateSemanticMap(narrative, basis, {
@@ -173,16 +177,17 @@ test("SemanticMap rejects source-order reversal, wrong basis and digest tamperin
);
});
-test("Dual Text owns its complete speech span and caption cues cannot split it", () => {
+test("Dual Text owns its complete speech span without entering SemanticMap", () => {
const narrative = parseScript(
"fixture.svml",
`<segment id="one">I just <lmao | laughed my ass out>.</segment>`,
);
- const evidence: AlignmentEvidence = {
- contract: "svml.speech-alignment.v1",
+ const evidence: SpeechTimingEvidence = {
+ contract: "svml.speech-timing-evidence.v1",
durationSec: 2,
fps: 30,
- words: [
+ quality: "measured",
+ units: [
{ text: "I", startSec: 0, endSec: 0.1, segmentId: "one" },
{ text: "just", startSec: 0.2, endSec: 0.4, segmentId: "one" },
{ text: "laughed", startSec: 0.5, endSec: 0.8, segmentId: "one" },
@@ -192,7 +197,7 @@ test("Dual Text owns its complete speech span and caption cues cannot split it",
],
segments: [{ id: "one", startSec: 0, endSec: 2 }],
};
- const located = locateAlignment(narrative, evidence);
+ const located = locateTiming(narrative, evidence);
assert.deepEqual(
located.captionAtoms.map((atom) => [
atom.display,
@@ -201,16 +206,6 @@ test("Dual Text owns its complete speech span and caption cues cannot split it",
]),
[["lmao", 15, 51]],
);
- assert.throws(
- () => locateAlignment(narrative, {
- ...evidence,
- captionCues: [
- { startWord: 0, endWordExclusive: 4 },
- { startWord: 4, endWordExclusive: 6 },
- ],
- }),
- /locate_caption_partial_dual/u,
- );
});
test("Slots bind after parsing and escaped reserved syntax stays literal", () => {
@@ -260,57 +255,52 @@ test("selection endpoints must close", () => {
);
});
-test("caption cue evidence is a complete ordered word partition", () => {
+test("direct Script-to-timing alignment handles N:1, 1:N, extras and missing units", () => {
const narrative = parseScript(
"fixture.svml",
- `<segment id="one">Alpha beta gamma.</segment>`,
+ `<segment id="one">We really love hyper frames cannot now.</segment>`,
);
- const evidence: AlignmentEvidence = {
- contract: "svml.speech-alignment.v1",
- durationSec: 1.5,
+ const evidence: SpeechTimingEvidence = {
+ contract: "svml.speech-timing-evidence.v1",
+ durationSec: 2.4,
fps: 30,
- words: [
- { text: "Alpha", startSec: 0, endSec: 0.4, segmentId: "one" },
- { text: "beta", startSec: 0.5, endSec: 0.9, segmentId: "one" },
- { text: "gamma", startSec: 1, endSec: 1.4, segmentId: "one" },
- ],
- segments: [{ id: "one", startSec: 0, endSec: 1.5 }],
- captionCues: [
- { id: "opening", startWord: 0, endWordExclusive: 1 },
- { id: "payoff", startWord: 1, endWordExclusive: 3 },
+ quality: "measured",
+ units: [
+ { text: "We", startSec: 0.1, endSec: 0.2, segmentId: "one" },
+ { text: "reallylovehyper", startSec: 0.4, endSec: 0.9, segmentId: "one" },
+ { text: "frames", startSec: 1, endSec: 1.3, segmentId: "one" },
+ { text: "can", startSec: 1.4, endSec: 1.55, segmentId: "one" },
+ { text: "not", startSec: 1.56, endSec: 1.8, segmentId: "one" },
+ { text: "um", startSec: 1.81, endSec: 1.85, segmentId: "one" },
+ { text: "now", startSec: 1.9, endSec: 2.1, segmentId: "one" },
],
+ segments: [{ id: "one", startSec: 0, endSec: 2.4 }],
};
- const located = locateAlignment(narrative, evidence);
+ const basis = createProgramBasis({
+ fps: evidence.fps,
+ durationFrames: Math.round(evidence.durationSec * evidence.fps),
+ outcomeDigest: sha256(stableJson(evidence)),
+ });
+ const map = semanticMapFromTiming(narrative, basis, evidence);
+ const located = locateScript(narrative, basis, map);
+ assert.deepEqual(located.words.map((word) => word.text),
+ ["We", "really", "love", "hyper", "frames", "cannot", "now"]);
assert.deepEqual(
- located.captionCues?.map((cue) => [
- cue.id,
- cue.startWord,
- cue.endWordExclusive,
- cue.startFrame,
- cue.endFrameExclusive,
- ]),
- [
- ["opening", 0, 1, 0, 12],
- ["payoff", 1, 3, 15, 42],
- ],
+ map.anchors.filter((anchor) => anchor.identity.includes(":token:") && anchor.identity.endsWith(":start"))
+ .map((anchor) => anchor.quality),
+ ["measured", "derived", "derived", "derived", "measured", "derived", "measured"],
);
-
- assert.throws(
- () => locateAlignment(narrative, {
- ...evidence,
- captionCues: [{ startWord: 1, endWordExclusive: 3 }],
- }),
- /locate_caption_cue_partition/u,
+ assert.deepEqual(
+ [located.words[1]?.startFrame, located.words[3]?.endFrameExclusive],
+ [12, 27],
);
- assert.throws(
- () => locateAlignment(narrative, {
- ...evidence,
- words: evidence.words.map((word, index) =>
- index === 1 ? { ...word, startSec: 0.2 } : word),
- }),
- /locator_word_overlap/u,
- );
+ const missing = locateTiming(narrative, {
+ ...evidence,
+ units: [evidence.units[0]!, evidence.units[2]!, evidence.units.at(-1)!],
+ });
+ assert.equal(missing.words[1]?.startFrame, 6);
+ assert.equal(missing.words[3]?.endFrameExclusive, 30);
});
test("Script normalizes newlines and NFC while tokenizing East Asian writing per character", () => {
@@ -340,30 +330,25 @@ test("reserved @ syntax and non-canonical ids fail closed", () => {
);
});
-test("syllable Estimate produces complete deterministic alignment including silence", () => {
+test("syllable Estimate produces deterministic SpeechTimingEvidence including silence", () => {
const narrative = parseScript("fixture.svml", `
<segment id="speech">Hello 中文.</segment>
<segment id="silence"/>
`);
- const first = estimateAlignment(narrative, {
+ const first = estimateSpeechTiming(narrative, {
fps: 30,
syllablesPerSecond: 4,
emptySegmentSec: 2,
- maxCaptionWords: 2,
});
- const second = estimateAlignment(narrative, {
+ const second = estimateSpeechTiming(narrative, {
fps: 30,
syllablesPerSecond: 4,
emptySegmentSec: 2,
- maxCaptionWords: 2,
});
assert.deepEqual(first, second);
- assert.deepEqual(first.words.map((word) => word.text), ["Hello", "中", "文"]);
+ assert.deepEqual(first.units.map((unit) => unit.text), ["Hello", "中", "文"]);
assert.equal(first.segments[1]?.endSec! - first.segments[1]?.startSec!, 2);
- assert.deepEqual(
- first.captionCues?.map((cue) => [cue.startWord, cue.endWordExclusive]),
- [[0, 2], [2, 3]],
- );
+ assert.equal(first.quality, "estimated");
assert.equal(first.provenance?.method, "svml.syllable-estimate.v1");
});
diff --git a/tools/build-speech-timing.mjs b/tools/build-speech-timing.mjs
new file mode 100644
index 00000000..836d21d2
--- /dev/null
+++ b/tools/build-speech-timing.mjs
@@ -0,0 +1,88 @@
+#!/usr/bin/env node
+import { readFile, writeFile } from "node:fs/promises";
+import { resolve } from "node:path";
+
+const args = process.argv.slice(2);
+function takeOption(name) {
+ const index = args.indexOf(name);
+ if (index < 0) return undefined;
+ const value = args[index + 1];
+ if (!value) throw new Error(`${name} requires a value.`);
+ args.splice(index, 2);
+ return value;
+}
+
+const timingOverrideFile = takeOption("--timing-overrides");
+const fps = Number(takeOption("--fps") ?? 30);
+const segmentFrames = Number(takeOption("--segment-frames") ?? 361);
+const [scriptFile, outputFile, ...whisperFiles] = args;
+if (!scriptFile || !outputFile || whisperFiles.length === 0) {
+ throw new Error(
+ "usage: build-speech-timing.mjs <script.json> <output.json> <whisper-1.json> ... "
+ + "[--fps 30] [--segment-frames 361] [--timing-overrides <overrides.json>]",
+ );
+}
+if (!Number.isFinite(fps) || fps <= 0 || !Number.isInteger(segmentFrames) || segmentFrames <= 0) {
+ throw new Error("fps and segment-frames must define a positive clock.");
+}
+
+const script = JSON.parse(await readFile(resolve(scriptFile), "utf8"));
+if (!Array.isArray(script.segments) || script.segments.length !== whisperFiles.length) {
+ throw new Error("Script Segment count must match the supplied Whisper result count.");
+}
+
+const units = [];
+for (const [segmentIndex, segment] of script.segments.entries()) {
+ const whisper = JSON.parse(await readFile(resolve(whisperFiles[segmentIndex]), "utf8"));
+ const offsetSec = segmentIndex * segmentFrames / fps;
+ const segmentEndSec = (segmentIndex + 1) * segmentFrames / fps;
+ for (const entry of whisper.transcription ?? []) {
+ const text = String(entry.text ?? "").trim();
+ if (!text) continue;
+ const startSec = Number(entry.offsets?.from) / 1000 + offsetSec;
+ const endSec = Number(entry.offsets?.to) / 1000 + offsetSec;
+ units.push({
+ text,
+ startSec: Math.max(offsetSec, Math.min(segmentEndSec, startSec)),
+ endSec: Math.max(offsetSec, Math.min(segmentEndSec, endSec)),
+ segmentId: segment.id,
+ });
+ }
+}
+
+const evidence = {
+ contract: "svml.speech-timing-evidence.v1",
+ durationSec: script.segments.length * segmentFrames / fps,
+ fps,
+ quality: "measured",
+ units,
+ segments: script.segments.map((segment, index) => ({
+ id: segment.id,
+ startSec: index * segmentFrames / fps,
+ endSec: (index + 1) * segmentFrames / fps,
+ })),
+ provenance: {
+ method: "whisper.cpp raw word timestamps",
+ source: "measured units are not corrected or projected to Script wording",
+ },
+};
+
+if (timingOverrideFile) {
+ const overrideDocument = JSON.parse(await readFile(resolve(timingOverrideFile), "utf8"));
+ if (!Array.isArray(overrideDocument.units)) {
+ throw new Error(`${timingOverrideFile} must contain a units array.`);
+ }
+ for (const override of overrideDocument.units) {
+ const index = Number(override.index);
+ const unit = evidence.units[index];
+ if (!Number.isInteger(index) || !unit) {
+ throw new Error(`${timingOverrideFile} contains invalid unit index ${String(override.index)}.`);
+ }
+ if (override.startSec !== undefined) unit.startSec = Number(override.startSec);
+ if (override.endSec !== undefined) unit.endSec = Number(override.endSec);
+ }
+ evidence.provenance.timingOverrides = overrideDocument.provenance
+ ?? "manual timing evidence correction";
+}
+
+await writeFile(resolve(outputFile), `${JSON.stringify(evidence, null, 2)}\n`);
diff --git a/tools/build-whisper-evidence.mjs b/tools/build-whisper-evidence.mjs
deleted file mode 100644
index 14f114fc..00000000
--- a/tools/build-whisper-evidence.mjs
+++ /dev/null
@@ -1,168 +0,0 @@
-#!/usr/bin/env node
-import { readFile, writeFile } from "node:fs/promises";
-import { resolve } from "node:path";
-
-const args = process.argv.slice(2);
-function takeOption(name) {
- const index = args.indexOf(name);
- if (index < 0) return undefined;
- const value = args[index + 1];
- if (!value) throw new Error(`${name} requires a file path.`);
- args.splice(index, 2);
- return value;
-}
-const cueFile = takeOption("--caption-cues");
-const timingOverrideFile = takeOption("--timing-overrides");
-const [sourceFile, outputFile, ...whisperFiles] = args;
-if (!sourceFile || !outputFile || whisperFiles.length === 0) {
- throw new Error(
- "usage: build-whisper-evidence.mjs <tokens.json> <output.json> <whisper-1.json> ... "
- + "[--caption-cues <cues.json>] [--timing-overrides <overrides.json>]",
- );
-}
-
-const intended = JSON.parse(await readFile(resolve(sourceFile), "utf8"));
-const normalize = (value) => String(value)
- .normalize("NFKC")
- .toLowerCase()
- .replace(/[^\p{L}\p{M}\p{N}]+/gu, "");
-
-const lcs = (left, right) => {
- const table = Array.from({ length: left.length + 1 }, () =>
- Array(right.length + 1).fill(0));
- for (let leftIndex = left.length - 1; leftIndex >= 0; leftIndex -= 1) {
- for (let rightIndex = right.length - 1; rightIndex >= 0; rightIndex -= 1) {
- table[leftIndex][rightIndex] = left[leftIndex] === right[rightIndex]
- ? 1 + table[leftIndex + 1][rightIndex + 1]
- : Math.max(table[leftIndex + 1][rightIndex], table[leftIndex][rightIndex + 1]);
- }
- }
- const matches = [];
- let leftIndex = 0;
- let rightIndex = 0;
- while (leftIndex < left.length && rightIndex < right.length) {
- if (left[leftIndex] === right[rightIndex]) {
- matches.push([leftIndex, rightIndex]);
- leftIndex += 1;
- rightIndex += 1;
- } else if (table[leftIndex + 1][rightIndex] >= table[leftIndex][rightIndex + 1]) {
- leftIndex += 1;
- } else {
- rightIndex += 1;
- }
- }
- return matches;
-};
-
-const evidenceWords = [];
-const segmentFrames = 361;
-for (let segmentIndex = 0; segmentIndex < intended.segments.length; segmentIndex += 1) {
- const segment = intended.segments[segmentIndex];
- const whisper = JSON.parse(await readFile(resolve(whisperFiles[segmentIndex]), "utf8"));
- const measured = whisper.transcription
- .filter((entry) => String(entry.text).trim())
- .map((entry) => ({
- text: String(entry.text).trim(),
- startSec: Number(entry.offsets.from) / 1000,
- endSec: Number(entry.offsets.to) / 1000,
- }));
- const target = intended.words.filter((word) => word.segmentId === segment.id);
- const matches = lcs(
- target.map((word) => normalize(word.text)),
- measured.map((word) => normalize(word.text)),
- );
- const anchors = [[-1, -1], ...matches, [target.length, measured.length]];
- const located = Array(target.length);
- for (const [targetIndex, measuredIndex] of matches) {
- located[targetIndex] = {
- text: target[targetIndex].text,
- startSec: measured[measuredIndex].startSec,
- endSec: measured[measuredIndex].endSec,
- };
- }
- for (let anchorIndex = 0; anchorIndex < anchors.length - 1; anchorIndex += 1) {
- const [leftTarget, leftMeasured] = anchors[anchorIndex];
- const [rightTarget, rightMeasured] = anchors[anchorIndex + 1];
- const missing = rightTarget - leftTarget - 1;
- if (missing <= 0) continue;
- const measuredSlice = measured.slice(leftMeasured + 1, rightMeasured);
- const startSec = measuredSlice[0]?.startSec
- ?? measured[leftMeasured]?.endSec
- ?? 0;
- const endSec = measuredSlice.at(-1)?.endSec
- ?? measured[rightMeasured]?.startSec
- ?? segmentFrames / 30;
- const weights = target
- .slice(leftTarget + 1, rightTarget)
- .map((word) => Math.max(1, normalize(word.text).length));
- const totalWeight = weights.reduce((sum, value) => sum + value, 0);
- let cursor = startSec;
- for (let index = 0; index < missing; index += 1) {
- const duration = (endSec - startSec) * weights[index] / totalWeight;
- const targetIndex = leftTarget + 1 + index;
- located[targetIndex] = {
- text: target[targetIndex].text,
- startSec: cursor,
- endSec: index === missing - 1 ? endSec : cursor + duration,
- };
- cursor += duration;
- }
- }
- const offsetSec = segmentIndex * segmentFrames / 30;
- const segmentEndSec = (segmentIndex + 1) * segmentFrames / 30;
- for (const word of located) {
- evidenceWords.push({
- text: word.text,
- startSec: Math.max(offsetSec, Math.min(segmentEndSec, word.startSec + offsetSec)),
- endSec: Math.max(offsetSec, Math.min(segmentEndSec, word.endSec + offsetSec)),
- segmentId: segment.id,
- });
- }
-}
-
-const evidence = {
- contract: "svml.speech-alignment.v1",
- durationSec: intended.segments.length * segmentFrames / 30,
- fps: 30,
- words: evidenceWords,
- segments: intended.segments.map((segment, index) => ({
- id: segment.id,
- startSec: index * segmentFrames / 30,
- endSec: (index + 1) * segmentFrames / 30,
- })),
- provenance: {
- method: "whisper.cpp word timestamps + deterministic intended-script LCS projection",
- source: "existing pinned avatar audio; no generation call",
- },
-};
-if (cueFile) {
- const cueDocument = JSON.parse(await readFile(resolve(cueFile), "utf8"));
- evidence.captionCues = Array.isArray(cueDocument)
- ? cueDocument
- : cueDocument.captionCues;
- if (!Array.isArray(evidence.captionCues)) {
- throw new Error(`${cueFile} must contain an array or a captionCues array.`);
- }
- evidence.provenance.captionCues = cueDocument.provenance
- ?? "pinned cue partition from the reference Locate result";
-}
-if (timingOverrideFile) {
- const overrideDocument = JSON.parse(
- await readFile(resolve(timingOverrideFile), "utf8"),
- );
- if (!Array.isArray(overrideDocument.words)) {
- throw new Error(`${timingOverrideFile} must contain a words array.`);
- }
- for (const override of overrideDocument.words) {
- const index = Number(override.index);
- const word = evidence.words[index];
- if (!Number.isInteger(index) || !word) {
- throw new Error(`${timingOverrideFile} contains invalid word index ${String(override.index)}.`);
- }
- if (override.startSec !== undefined) word.startSec = Number(override.startSec);
- if (override.endSec !== undefined) word.endSec = Number(override.endSec);
- }
- evidence.provenance.timingOverrides = overrideDocument.provenance
- ?? "reference transition recovery";
-}
-await writeFile(resolve(outputFile), `${JSON.stringify(evidence, null, 2)}\n`);